2014-06-12 14:45:54.904881 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2704 (0'0,173'2704] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:54.904895 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2704 (0'0,173'2704] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] do_osd_op  write 925696~516096
2014-06-12 14:45:54.904912 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2704 (0'0,173'2704] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:54.904925 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2704 (0'0,173'2704] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:45:54.904938 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2704 (0'0,173'2704] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:54.904951 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2704 (0'0,173'2704] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] finish_ctx 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 0x1a919600 op modify  
2014-06-12 14:45:54.904967 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2704 (0'0,173'2704] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]  set mtime to 2014-06-12 08:44:57.276137
2014-06-12 14:45:54.904988 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2704 (0'0,173'2704] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]  final snapset 0=[]:[]+head in 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.905012 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2704 (0'0,173'2704] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]  zeroing write result code 0
2014-06-12 14:45:54.905023 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2704 (0'0,173'2704] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] new_repop rep_tid 236129 on osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.905039 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2704 (0'0,173'2704] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] issue_repop rep_tid 236129 o 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.905077 7f9d69be4700  5 -- op tracker -- , seq: 747282, time: 2014-06-12 14:45:54.905070, event: waiting for subops from 23, op: osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.905095 7f9d69be4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:54.905116 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2704 (0'0,173'2704] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] append_log log((0'0,173'2704], crt=0'0) [173'2705 (173'2704) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144972 2014-06-12 08:44:57.276137]
2014-06-12 14:45:54.905133 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] add_log_entry 173'2705 (173'2704) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144972 2014-06-12 08:44:57.276137
2014-06-12 14:45:54.905155 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] append_log  adding 1 keys
2014-06-12 14:45:54.905188 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2705, trimmed: 
2014-06-12 14:45:54.905209 7f9d69be4700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.905214 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.905215 7f9d69be4700  5 queue_op 0x7697860 seq 2450 osr(3.54b 0x276db30) 518389 bytes   (queue has 3 ops and 1448671 bytes)
2014-06-12 14:45:54.905223 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] eval_repop repgather(0x1c832e40 173'2705 rep_tid=236129 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.905246 7f9d69be4700 10 osd.2 173 dequeue_op 0x1e3ebc00 finish
2014-06-12 14:45:54.905254 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:45:54.905258 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.905569 7f9d633d7700 10 osd.2 173 dequeue_op 0x1e3ea600 prio 63 cost 516096 latency 0.003602 osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]
2014-06-12 14:45:54.905600 7f9d633d7700  5 -- op tracker -- , seq: 747283, time: 2014-06-12 14:45:54.905594, event: reached_pg, op: osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.905631 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (3 active)
2014-06-12 14:45:54.905609 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:54.905642 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] handle_message: 0x1e3ea600
2014-06-12 14:45:54.905652 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] do_op osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:54.905673 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] get_object_context: found obc in cache: 0x14265600
2014-06-12 14:45:54.905686 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] get_object_context: 0x14265600 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 rwstate(write n=3 w=0) oi: 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3(173'2705 client.4314.0:144972 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2705) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:45:54.905702 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] find_object_context 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 @head oi=89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3(173'2705 client.4314.0:144972 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2705)
2014-06-12 14:45:54.905723 7f9d633d7700  5 -- op tracker -- , seq: 747283, time: 2014-06-12 14:45:54.905718, event: started, op: osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.905732 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] execute_ctx 0x16af3800
2014-06-12 14:45:54.905751 7f9d633d7700  5 -- op tracker -- , seq: 747283, time: 2014-06-12 14:45:54.905746, event: started, op: osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.905758 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] do_op 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] ov 173'2705 av 173'2706 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:54.905775 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] do_osd_op 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096]
2014-06-12 14:45:54.905785 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:54.905796 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] do_osd_op  write 1441792~516096
2014-06-12 14:45:54.905810 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:54.905821 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:45:54.905832 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:54.905843 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] finish_ctx 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 0x16af3800 op modify  
2014-06-12 14:45:54.905858 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]  set mtime to 2014-06-12 08:44:57.282309
2014-06-12 14:45:54.905878 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]  final snapset 0=[]:[]+head in 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.905896 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]  zeroing write result code 0
2014-06-12 14:45:54.905908 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] new_repop rep_tid 236130 on osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.905925 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] issue_repop rep_tid 236130 o 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.905947 7f9d633d7700  5 -- op tracker -- , seq: 747283, time: 2014-06-12 14:45:54.905941, event: waiting for subops from 23, op: osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.905963 7f9d633d7700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:54.905982 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2705 (0'0,173'2705] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] append_log log((0'0,173'2705], crt=0'0) [173'2706 (173'2705) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144973 2014-06-12 08:44:57.282309]
2014-06-12 14:45:54.906003 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] add_log_entry 173'2706 (173'2705) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144973 2014-06-12 08:44:57.282309
2014-06-12 14:45:54.906024 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] append_log  adding 1 keys
2014-06-12 14:45:54.906052 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2706, trimmed: 
2014-06-12 14:45:54.906072 7f9d633d7700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.906076 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.906073 7f9d7d167700 10 _generic_write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 0~409600 = 0
2014-06-12 14:45:54.906078 7f9d633d7700  5 queue_op 0x15cda900 seq 2451 osr(3.54b 0x276db30) 518389 bytes   (queue has 4 ops and 1967060 bytes)
2014-06-12 14:45:54.906086 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (4 active)
2014-06-12 14:45:54.906084 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] eval_repop repgather(0x5147ec0 173'2706 rep_tid=236130 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.906100 7f9d633d7700 10 osd.2 173 dequeue_op 0x1e3ea600 finish
2014-06-12 14:45:54.906106 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.906110 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.906124 7f9d7d167700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.906138 7f9d7d167700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.906143 7f9d7d167700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.906150 7f9d7d167700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.906357 7f9d7d167700 20 set_header setting 17860 cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 parent seq 0
2014-06-12 14:45:54.906418 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:54.906435 7f9d7d167700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:45:54.906445 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:45:54.907971 7f9d7d167700 10 _do_op 0x67a05a0 seq 2448 r = 0, finisher 0x80f6780 0x38997fc0
2014-06-12 14:45:54.907986 7f9d7d167700 10 _finish_op 0x67a05a0 seq 2448 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.907999 7f9d7e96a700  5 _do_op 0x88d2720 seq 2449 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:45:54.908003 7f9d7e96a700 10 _do_transaction on 0x31ee66c0
2014-06-12 14:45:54.907993 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] op_commit: 236127
2014-06-12 14:45:54.908010 7f9d7e96a700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.908019 7f9d7715b700  5 -- op tracker -- , seq: 747276, time: 2014-06-12 14:45:54.908011, event: op_commit, op: osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.908124 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.908154 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.908219 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:54.908327 7f9d7e96a700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.908337 7f9d7e96a700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:45:54.908343 7f9d7e96a700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.908365 7f9d7e96a700 15 _write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 409600~516096
2014-06-12 14:45:54.908482 7f9d7e96a700 10 lookup_strip_header done  cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.908621 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:45:54.909751 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] op_applied: 236127
2014-06-12 14:45:54.909785 7f9d7795c700  5 -- op tracker -- , seq: 747276, time: 2014-06-12 14:45:54.909776, event: op_applied, op: osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.909794 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] op_applied on primary on version 173'2703
2014-06-12 14:45:54.909823 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (3 active)
2014-06-12 14:45:54.909829 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:54.909990 7f9d7e96a700 10 _generic_write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 409600~516096 = 0
2014-06-12 14:45:54.910033 7f9d7e96a700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.910043 7f9d7e96a700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.910050 7f9d7e96a700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.910055 7f9d7e96a700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.910193 7f9d7e96a700 20 set_header setting 17860 cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 parent seq 0
2014-06-12 14:45:54.910256 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:54.910267 7f9d7e96a700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:45:54.910273 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:45:54.911118 7f9d5a16f700  5 -- op tracker -- , seq: 747284, time: 2014-06-12 14:45:54.910328, event: header_read, op: osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.911133 7f9d5a16f700  5 -- op tracker -- , seq: 747284, time: 2014-06-12 14:45:54.910331, event: throttled, op: osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.911142 7f9d5a16f700  5 -- op tracker -- , seq: 747284, time: 2014-06-12 14:45:54.911079, event: all_read, op: osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.911150 7f9d5a16f700  5 -- op tracker -- , seq: 747284, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.911156 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:54.911163 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:45:54.911172 7f9d5a16f700 15 osd.2 173 enqueue_op 0x124e3700 prio 127 cost 287527 latency 0.000844 osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:54.912122 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.912137 7f9d68be2700 10 osd.2 173 dequeue_op 0x124e3700 prio 127 cost 287527 latency 0.001809 osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4e0( v 173'2629 (0'0,173'2629] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2628 active]
2014-06-12 14:45:54.912165 7f9d68be2700  5 -- op tracker -- , seq: 747284, time: 2014-06-12 14:45:54.912160, event: reached_pg, op: osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.912175 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2629 (0'0,173'2629] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2628 active] handle_message: 0x124e3700
2014-06-12 14:45:54.912184 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2629 (0'0,173'2629] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2628 active] sub_op_modify trans 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 v 173'2630 (transaction) 174
2014-06-12 14:45:54.912195 7f9d68be2700  5 -- op tracker -- , seq: 747284, time: 2014-06-12 14:45:54.912192, event: started, op: osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.912211 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2629 (0'0,173'2629] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2628 active] append_log log((0'0,173'2629], crt=0'0) [173'2630 (173'696) modify   6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 by client.4306.0:143773 2014-06-12 08:43:00.237313]
2014-06-12 14:45:54.912232 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2630 (0'0,173'2630] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2628 active] add_log_entry 173'2630 (173'696) modify   6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 by client.4306.0:143773 2014-06-12 08:43:00.237313
2014-06-12 14:45:54.912251 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2630 (0'0,173'2630] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2628 active] append_log  adding 1 keys
2014-06-12 14:45:54.912280 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2630, trimmed: 
2014-06-12 14:45:54.912299 7f9d68be2700  5 -- op tracker -- , seq: 747284, time: 2014-06-12 14:45:54.912296, event: started, op: osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.912308 7f9d68be2700  5 queue_transactions existing osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:45:54.912312 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.912314 7f9d68be2700  5 queue_op 0x6075740 seq 2430 osr(3.4e0 0x276dbd0) 289013 bytes   (queue has 4 ops and 1844180 bytes)
2014-06-12 14:45:54.912325 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2630 (0'0,173'2630] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2628 active] do_sub_op osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:54.912334 7f9d68be2700 10 osd.2 173 dequeue_op 0x124e3700 finish
2014-06-12 14:45:54.912337 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.912338 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.912858 7f9d7e96a700 10 _do_op 0x88d2720 seq 2449 r = 0, finisher 0x63cf9c0 0x1a3f0f00
2014-06-12 14:45:54.912875 7f9d7e96a700 10 _finish_op 0x88d2720 seq 2449 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.912879 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] op_commit: 236128
2014-06-12 14:45:54.912892 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (2 active)
2014-06-12 14:45:54.912900 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5180900 (3 active)
2014-06-12 14:45:54.912902 7f9d7e96a700  5 _do_op 0x6075740 seq 2430 osr(3.4e0 0x276dbd0)/0x276dbd0 start
2014-06-12 14:45:54.912905 7f9d7e96a700 10 _do_transaction on 0x6275eb0
2014-06-12 14:45:54.912910 7f9d7e96a700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:45:54.912908 7f9d7715b700  5 -- op tracker -- , seq: 747280, time: 2014-06-12 14:45:54.912898, event: op_commit, op: osd_op(client.4314.0:144971 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.912923 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] op_applied: 236128
2014-06-12 14:45:54.912942 7f9d7795c700  5 -- op tracker -- , seq: 747280, time: 2014-06-12 14:45:54.912936, event: op_applied, op: osd_op(client.4314.0:144971 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.912950 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] op_applied on primary on version 173'2704
2014-06-12 14:45:54.913032 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:45:54.913060 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.913111 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:54.913118 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:54.913112 7f9d7815d700  5 _do_op 0x7697860 seq 2450 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:45:54.913127 7f9d7815d700 10 _do_transaction on 0x2a12e540
2014-06-12 14:45:54.913133 7f9d7815d700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.913137 7f9d7e96a700 15 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:45:54.913142 7f9d7e96a700 10 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1 r = 0
2014-06-12 14:45:54.913147 7f9d7e96a700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:45:54.913204 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.913205 7f9d7e96a700 15 _write 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 0~286720
2014-06-12 14:45:54.913226 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.913419 7f9d7e96a700 10 lookup_strip_header done  cid 3.4e0_head oid 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:45:54.913510 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:45:54.914378 7f9d5501e700  5 -- op tracker -- , seq: 747285, time: 2014-06-12 14:45:54.912198, event: header_read, op: osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.914390 7f9d5501e700  5 -- op tracker -- , seq: 747285, time: 2014-06-12 14:45:54.912200, event: throttled, op: osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.914399 7f9d5501e700  5 -- op tracker -- , seq: 747285, time: 2014-06-12 14:45:54.914339, event: all_read, op: osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.914408 7f9d5501e700  5 -- op tracker -- , seq: 747285, time: 0.000000, event: dispatched, op: osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.914415 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:54.914422 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:45:54.914432 7f9d5501e700 15 osd.2 173 enqueue_op 0x2519c200 prio 127 cost 1008423 latency 0.002233 osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:54.914482 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.914490 7f9d643d9700 10 osd.2 173 dequeue_op 0x2519c200 prio 127 cost 1008423 latency 0.002292 osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.169( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2860 active]
2014-06-12 14:45:54.914514 7f9d643d9700  5 -- op tracker -- , seq: 747285, time: 2014-06-12 14:45:54.914509, event: reached_pg, op: osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.914523 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2860 active] handle_message: 0x2519c200
2014-06-12 14:45:54.914535 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2860 active] sub_op_modify trans e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 v 173'2862 (transaction) 174
2014-06-12 14:45:54.914548 7f9d7e96a700 10 _generic_write 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 0~286720 = 0
2014-06-12 14:45:54.914553 7f9d643d9700  5 -- op tracker -- , seq: 747285, time: 2014-06-12 14:45:54.914548, event: started, op: osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.914569 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2860 active] append_log log((0'0,173'2861], crt=0'0) [173'2862 (173'816) modify   e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 by client.4296.0:143497 2014-06-12 08:44:57.257141]
2014-06-12 14:45:54.914589 7f9d7e96a700 15 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:45:54.914594 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2860 active] add_log_entry 173'2862 (173'816) modify   e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 by client.4296.0:143497 2014-06-12 08:44:57.257141
2014-06-12 14:45:54.914603 7f9d7e96a700 10 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:45:54.914614 7f9d7e96a700 15 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:45:54.914623 7f9d7e96a700 10 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:45:54.914616 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2860 active] append_log  adding 1 keys
2014-06-12 14:45:54.914650 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2862, trimmed: 
2014-06-12 14:45:54.914675 7f9d643d9700  5 -- op tracker -- , seq: 747285, time: 2014-06-12 14:45:54.914670, event: started, op: osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.914686 7f9d643d9700  5 queue_transactions existing osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:45:54.914690 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.914692 7f9d643d9700  5 queue_op 0x53c2840 seq 2600 osr(3.169 0x276ce50) 1009909 bytes   (queue has 4 ops and 2335700 bytes)
2014-06-12 14:45:54.914700 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.169( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2860 active] do_sub_op osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:54.914714 7f9d643d9700 10 osd.2 173 dequeue_op 0x2519c200 finish
2014-06-12 14:45:54.914718 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.914721 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.914729 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54f1200 (4 active)
2014-06-12 14:45:54.914737 7f9d7d968700  5 _do_op 0x53c2840 seq 2600 osr(3.169 0x276ce50)/0x276ce50 start
2014-06-12 14:45:54.914741 7f9d7d968700 10 _do_transaction on 0x587f0b0
2014-06-12 14:45:54.914746 7f9d7d968700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:54.914790 7f9d7e96a700 20 set_header setting 16555 cid 3.4e0_head oid 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:45:54.914841 7f9d7e96a700 20 set_header setting 2236 cid meta oid 1fe07435/pglog_3.4e0/0//-1 parent seq 0
2014-06-12 14:45:54.914855 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:54.914856 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:54.914864 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:45:54.914876 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.916152 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:54.916183 7f9d7d968700 15 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:54.916184 7f9d7e96a700 10 _do_op 0x6075740 seq 2430 r = 0, finisher 0x698cc00 0
2014-06-12 14:45:54.916189 7f9d7d968700 10 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1 r = 0
2014-06-12 14:45:54.916196 7f9d7d968700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:54.916212 7f9d7d968700 15 _write 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 0~1007616
2014-06-12 14:45:54.916208 7f9d7715b700  5 -- op tracker -- , seq: 747284, time: 2014-06-12 14:45:54.916201, event: commit_sent, op: osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.916218 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2630 (0'0,173'2630] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2628 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:45:54.916241 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:45:54.916282 7f9d7e96a700 10 _finish_op 0x6075740 seq 2430 osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:45:54.916310 7f9d7795c700  5 -- op tracker -- , seq: 747284, time: 2014-06-12 14:45:54.916305, event: sub_op_applied, op: osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.916318 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2630 (0'0,173'2630] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active] sub_op_modify_applied on 0x6275dc0 op osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:54.916333 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2630 (0'0,173'2630] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active] op_applied on primary on version 173'2630
2014-06-12 14:45:54.916343 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2630 (0'0,173'2630] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active] op_applied on replica on version 173'2630
2014-06-12 14:45:54.916377 7f9d7e96a700  5 -- op tracker -- , seq: 747284, time: 2014-06-12 14:45:54.916367, event: done, op: osd_sub_op(client.4306.0:143773 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2630 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.916394 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5180900 (3 active)
2014-06-12 14:45:54.916396 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:54.916519 7f9d7d968700 10 lookup_strip_header done  cid 3.169_head oid e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:54.916700 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:45:54.920668 7f9d7d968700 10 _generic_write 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 0~1007616 = 0
2014-06-12 14:45:54.920788 7f9d7d968700 15 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:54.920808 7f9d7d968700 10 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 = 0
2014-06-12 14:45:54.920818 7f9d7d968700 15 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:54.920827 7f9d7d968700 10 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 = 0
2014-06-12 14:45:54.921130 7f9d7d968700 20 set_header setting 16925 cid 3.169_head oid e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 parent seq 0
2014-06-12 14:45:54.921238 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:54.921255 7f9d7d968700 20 set_header setting 2306 cid meta oid 1fe2acfa/pglog_3.169/0//-1 parent seq 0
2014-06-12 14:45:54.921265 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:45:54.922675 7f9d57543700  5 -- op tracker -- , seq: 747286, time: 2014-06-12 14:45:54.922615, event: header_read, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.922689 7f9d57543700  5 -- op tracker -- , seq: 747286, time: 2014-06-12 14:45:54.922618, event: throttled, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.922697 7f9d57543700  5 -- op tracker -- , seq: 747286, time: 2014-06-12 14:45:54.922632, event: all_read, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.922705 7f9d57543700  5 -- op tracker -- , seq: 747286, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.922711 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:45:54.922717 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:54.922726 7f9d57543700 15 osd.2 173 enqueue_op 0x28fd9c00 prio 196 cost 0 latency 0.000111 osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0) v2
2014-06-12 14:45:54.922785 7f9d57543700  5 -- op tracker -- , seq: 747287, time: 2014-06-12 14:45:54.922751, event: header_read, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.922794 7f9d57543700  5 -- op tracker -- , seq: 747287, time: 2014-06-12 14:45:54.922752, event: throttled, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.922802 7f9d57543700  5 -- op tracker -- , seq: 747287, time: 2014-06-12 14:45:54.922765, event: all_read, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.922810 7f9d57543700  5 -- op tracker -- , seq: 747287, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.922813 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:54.922818 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:54.922823 7f9d57543700 15 osd.2 173 enqueue_op 0x124e2000 prio 196 cost 0 latency 0.000072 osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:54.922847 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.922856 7f9d623d5700 10 osd.2 173 dequeue_op 0x28fd9c00 prio 196 cost 0 latency 0.000241 osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0) v2 pg pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]
2014-06-12 14:45:54.922906 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:45:54.922878 7f9d623d5700  5 -- op tracker -- , seq: 747286, time: 2014-06-12 14:45:54.922875, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.922939 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] handle_message: 0x28fd9c00
2014-06-12 14:45:54.922955 7f9d623d5700  5 -- op tracker -- , seq: 747286, time: 2014-06-12 14:45:54.922951, event: started, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.922960 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] sub_op_modify_reply: tid 236127 op  ack_type 1 from 23
2014-06-12 14:45:54.922979 7f9d623d5700  5 -- op tracker -- , seq: 747276, time: 2014-06-12 14:45:54.922971, event: sub_op_applied_rec, op: osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.922986 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] repop_all_applied: repop tid 236127 all applied 
2014-06-12 14:45:54.922996 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] eval_repop repgather(0x2a12eb40 173'2703 rep_tid=236127 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.923009 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]  sending ack on repgather(0x2a12eb40 173'2703 rep_tid=236127 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x38136780
2014-06-12 14:45:54.923038 7f9d623d5700 10 osd.2 173 dequeue_op 0x28fd9c00 finish
2014-06-12 14:45:54.923046 7f9d623d5700  5 -- op tracker -- , seq: 747286, time: 2014-06-12 14:45:54.923043, event: done, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.923059 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:45:54.923062 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.923093 7f9d61bd4700 10 osd.2 173 dequeue_op 0x124e2000 prio 196 cost 0 latency 0.000342 osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]
2014-06-12 14:45:54.923111 7f9d61bd4700  5 -- op tracker -- , seq: 747287, time: 2014-06-12 14:45:54.923108, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.923126 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] handle_message: 0x124e2000
2014-06-12 14:45:54.923141 7f9d61bd4700  5 -- op tracker -- , seq: 747287, time: 2014-06-12 14:45:54.923137, event: started, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.923151 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] sub_op_modify_reply: tid 236127 op  ack_type 4 from 23
2014-06-12 14:45:54.923168 7f9d61bd4700  5 -- op tracker -- , seq: 747276, time: 2014-06-12 14:45:54.923163, event: sub_op_commit_rec, op: osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.923175 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] repop_all_committed: repop tid 236127 all committed 
2014-06-12 14:45:54.923185 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean] eval_repop repgather(0x2a12eb40 173'2703 rep_tid=236127 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.923201 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean] log_op_stats osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 inb 409879 outb 0 rlat 0.037896 lat 0.038059
2014-06-12 14:45:54.923223 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean] publish_stats_to_osd 173:3596
2014-06-12 14:45:54.923238 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean]  sending commit on repgather(0x2a12eb40 173'2703 rep_tid=236127 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x45dfb80
2014-06-12 14:45:54.923270 7f9d61bd4700  5 -- op tracker -- , seq: 747276, time: 2014-06-12 14:45:54.923265, event: commit_sent, op: osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.923287 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean]  requeue_ops 
2014-06-12 14:45:54.923297 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]  removing repgather(0x2a12eb40 173'2703 rep_tid=236127 committed?=1 applied?=1 op=osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.923314 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]    q front is repgather(0x2a12eb40 173'2703 rep_tid=236127 committed?=1 applied?=1 op=osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.923328 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] remove_repop repgather(0x2a12eb40 173'2703 rep_tid=236127 committed?=1 applied?=1 op=osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.923346 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]  requeue_ops 
2014-06-12 14:45:54.923366 7f9d61bd4700  5 -- op tracker -- , seq: 747276, time: 2014-06-12 14:45:54.923361, event: done, op: osd_op(client.4314.0:144970 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.923384 7f9d61bd4700 10 osd.2 173 dequeue_op 0x124e2000 finish
2014-06-12 14:45:54.923391 7f9d61bd4700  5 -- op tracker -- , seq: 747287, time: 2014-06-12 14:45:54.923388, event: done, op: osd_sub_op_reply(client.4314.0:144970 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.923401 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.923404 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.926184 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:54.926230 7f9d7815d700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.926237 7f9d7815d700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:45:54.926245 7f9d7815d700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.926358 7f9d7815d700 15 _write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 925696~516096
2014-06-12 14:45:54.926516 7f9d7815d700 10 lookup_strip_header done  cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.926516 7f9d425f5700  5 -- op tracker -- , seq: 747288, time: 2014-06-12 14:45:54.925557, event: header_read, op: osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.926533 7f9d425f5700  5 -- op tracker -- , seq: 747288, time: 2014-06-12 14:45:54.925560, event: throttled, op: osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.926543 7f9d425f5700  5 -- op tracker -- , seq: 747288, time: 2014-06-12 14:45:54.926466, event: all_read, op: osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.926553 7f9d425f5700  5 -- op tracker -- , seq: 747288, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.926560 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:45:54.926587 7f9d425f5700 15 osd.2 173 enqueue_op 0x1e3eab00 prio 63 cost 516096 latency 0.001029 osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.926620 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.926639 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:45:54.926626 7f9d673df700 10 osd.2 173 dequeue_op 0x1e3eab00 prio 63 cost 516096 latency 0.001069 osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]
2014-06-12 14:45:54.926651 7f9d673df700  5 -- op tracker -- , seq: 747288, time: 2014-06-12 14:45:54.926646, event: reached_pg, op: osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.926660 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:54.926674 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] handle_message: 0x1e3eab00
2014-06-12 14:45:54.926685 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] do_op osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:54.926709 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] get_object_context: found obc in cache: 0x14265600
2014-06-12 14:45:54.926720 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] get_object_context: 0x14265600 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 rwstate(write n=3 w=0) oi: 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3(173'2706 client.4314.0:144973 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2706) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:45:54.926735 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] find_object_context 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 @head oi=89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3(173'2706 client.4314.0:144973 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2706)
2014-06-12 14:45:54.926762 7f9d673df700  5 -- op tracker -- , seq: 747288, time: 2014-06-12 14:45:54.926756, event: started, op: osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.926769 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] execute_ctx 0x184d4c00
2014-06-12 14:45:54.926787 7f9d673df700  5 -- op tracker -- , seq: 747288, time: 2014-06-12 14:45:54.926782, event: started, op: osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.926794 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] do_op 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] ov 173'2706 av 173'2707 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:54.926808 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] do_osd_op 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096]
2014-06-12 14:45:54.926818 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:54.926832 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] do_osd_op  write 1957888~516096
2014-06-12 14:45:54.926846 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:54.926857 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:45:54.926869 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:54.926879 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] finish_ctx 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 0x184d4c00 op modify  
2014-06-12 14:45:54.926893 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]  set mtime to 2014-06-12 08:44:57.305944
2014-06-12 14:45:54.926914 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]  final snapset 0=[]:[]+head in 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.926932 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]  zeroing write result code 0
2014-06-12 14:45:54.926943 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] new_repop rep_tid 236131 on osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.926959 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] issue_repop rep_tid 236131 o 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.926984 7f9d673df700  5 -- op tracker -- , seq: 747288, time: 2014-06-12 14:45:54.926979, event: waiting for subops from 23, op: osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.927005 7f9d673df700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:54.927026 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2706 (0'0,173'2706] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] append_log log((0'0,173'2706], crt=0'0) [173'2707 (173'2706) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144974 2014-06-12 08:44:57.305944]
2014-06-12 14:45:54.927049 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] add_log_entry 173'2707 (173'2706) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144974 2014-06-12 08:44:57.305944
2014-06-12 14:45:54.927069 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] append_log  adding 1 keys
2014-06-12 14:45:54.927101 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2707, trimmed: 
2014-06-12 14:45:54.927122 7f9d673df700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.927127 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.927129 7f9d673df700  5 queue_op 0x1415e540 seq 2452 osr(3.54b 0x276db30) 518389 bytes   (queue has 4 ops and 2565076 bytes)
2014-06-12 14:45:54.927140 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (4 active)
2014-06-12 14:45:54.927136 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] eval_repop repgather(0x12ae2540 173'2707 rep_tid=236131 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.927155 7f9d673df700 10 osd.2 173 dequeue_op 0x1e3eab00 finish
2014-06-12 14:45:54.927159 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.927162 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.927683 7f9d57543700  5 -- op tracker -- , seq: 747289, time: 2014-06-12 14:45:54.927645, event: header_read, op: osd_sub_op_reply(client.4314.0:144971 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.927697 7f9d57543700  5 -- op tracker -- , seq: 747289, time: 2014-06-12 14:45:54.927649, event: throttled, op: osd_sub_op_reply(client.4314.0:144971 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.927706 7f9d57543700  5 -- op tracker -- , seq: 747289, time: 2014-06-12 14:45:54.927659, event: all_read, op: osd_sub_op_reply(client.4314.0:144971 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.927714 7f9d57543700  5 -- op tracker -- , seq: 747289, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144971 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.927718 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144971 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:54.927724 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:54.927730 7f9d57543700 15 osd.2 173 enqueue_op 0x20e87e00 prio 196 cost 0 latency 0.000085 osd_sub_op_reply(client.4314.0:144971 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:54.927753 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.927759 7f9d663dd700 10 osd.2 173 dequeue_op 0x20e87e00 prio 196 cost 0 latency 0.000114 osd_sub_op_reply(client.4314.0:144971 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]
2014-06-12 14:45:54.927777 7f9d663dd700  5 -- op tracker -- , seq: 747289, time: 2014-06-12 14:45:54.927774, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144971 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.927785 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] handle_message: 0x20e87e00
2014-06-12 14:45:54.927799 7f9d663dd700  5 -- op tracker -- , seq: 747289, time: 2014-06-12 14:45:54.927796, event: started, op: osd_sub_op_reply(client.4314.0:144971 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.927805 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] sub_op_modify_reply: tid 236128 op  ack_type 4 from 23
2014-06-12 14:45:54.927823 7f9d663dd700  5 -- op tracker -- , seq: 747280, time: 2014-06-12 14:45:54.927817, event: sub_op_commit_rec, op: osd_op(client.4314.0:144971 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.927831 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] repop_all_applied: repop tid 236128 all applied 
2014-06-12 14:45:54.927841 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] eval_repop repgather(0x31ee7a40 173'2704 rep_tid=236128 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144971 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.927856 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]  sending ack on repgather(0x31ee7a40 173'2704 rep_tid=236128 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144971 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x1b775400
2014-06-12 14:45:54.927885 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] repop_all_committed: repop tid 236128 all committed 
2014-06-12 14:45:54.927896 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean] eval_repop repgather(0x31ee7a40 173'2704 rep_tid=236128 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144971 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.927912 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean] log_op_stats osd_op(client.4314.0:144971 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.037740 lat 0.037766
2014-06-12 14:45:54.927930 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean] publish_stats_to_osd 173:3597
2014-06-12 14:45:54.927945 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean]  sending commit on repgather(0x31ee7a40 173'2704 rep_tid=236128 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144971 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x56ca500
2014-06-12 14:45:54.927983 7f9d663dd700  5 -- op tracker -- , seq: 747280, time: 2014-06-12 14:45:54.927978, event: commit_sent, op: osd_op(client.4314.0:144971 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.927991 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean]  requeue_ops 
2014-06-12 14:45:54.928001 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]  removing repgather(0x31ee7a40 173'2704 rep_tid=236128 committed?=1 applied?=1 op=osd_op(client.4314.0:144971 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.928016 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]    q front is repgather(0x31ee7a40 173'2704 rep_tid=236128 committed?=1 applied?=1 op=osd_op(client.4314.0:144971 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.928029 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] remove_repop repgather(0x31ee7a40 173'2704 rep_tid=236128 committed?=1 applied?=1 op=osd_op(client.4314.0:144971 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.928042 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]  requeue_ops 
2014-06-12 14:45:54.928061 7f9d663dd700  5 -- op tracker -- , seq: 747280, time: 2014-06-12 14:45:54.928056, event: done, op: osd_op(client.4314.0:144971 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.928077 7f9d663dd700 10 osd.2 173 dequeue_op 0x20e87e00 finish
2014-06-12 14:45:54.928084 7f9d663dd700  5 -- op tracker -- , seq: 747289, time: 2014-06-12 14:45:54.928080, event: done, op: osd_sub_op_reply(client.4314.0:144971 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.928094 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.928097 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.928423 7f9d7815d700 10 _generic_write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 925696~516096 = 0
2014-06-12 14:45:54.928469 7f9d7815d700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.928482 7f9d7815d700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.928492 7f9d7815d700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.928501 7f9d7815d700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.928702 7f9d7815d700 20 set_header setting 17860 cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 parent seq 0
2014-06-12 14:45:54.928773 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:54.928786 7f9d7815d700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:45:54.928796 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:45:54.931478 7f9d7d968700 10 _do_op 0x53c2840 seq 2600 r = 0, finisher 0x28f2880 0
2014-06-12 14:45:54.931521 7f9d7715b700  5 -- op tracker -- , seq: 747285, time: 2014-06-12 14:45:54.931511, event: commit_sent, op: osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.931532 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2860 active] sub_op_modify_commit on op osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:45:54.931558 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:54.931583 7f9d7d968700 10 _finish_op 0x53c2840 seq 2600 osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:45:54.931603 7f9d7795c700  5 -- op tracker -- , seq: 747285, time: 2014-06-12 14:45:54.931598, event: sub_op_applied, op: osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.931610 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active] sub_op_modify_applied on 0x587efc0 op osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:54.931624 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active] op_applied on primary on version 173'2862
2014-06-12 14:45:54.931633 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active] op_applied on replica on version 173'2862
2014-06-12 14:45:54.931668 7f9d7d968700  5 -- op tracker -- , seq: 747285, time: 2014-06-12 14:45:54.931657, event: done, op: osd_sub_op(client.4296.0:143497 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.931698 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54f1200 (3 active)
2014-06-12 14:45:54.931701 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:54.932363 7f9d425f5700  5 -- op tracker -- , seq: 747290, time: 2014-06-12 14:45:54.930886, event: header_read, op: osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.932376 7f9d425f5700  5 -- op tracker -- , seq: 747290, time: 2014-06-12 14:45:54.930890, event: throttled, op: osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.932386 7f9d425f5700  5 -- op tracker -- , seq: 747290, time: 2014-06-12 14:45:54.932282, event: all_read, op: osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.932396 7f9d425f5700  5 -- op tracker -- , seq: 747290, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.932404 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:45:54.932487 7f9d425f5700 15 osd.2 173 enqueue_op 0x1e3ea800 prio 63 cost 516096 latency 0.001600 osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.932522 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.932529 7f9d67be0700 10 osd.2 173 dequeue_op 0x1e3ea800 prio 63 cost 516096 latency 0.001642 osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]
2014-06-12 14:45:54.932552 7f9d67be0700  5 -- op tracker -- , seq: 747290, time: 2014-06-12 14:45:54.932547, event: reached_pg, op: osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.932560 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:54.932574 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] handle_message: 0x1e3ea800
2014-06-12 14:45:54.932606 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] do_op osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:54.932629 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] get_object_context: found obc in cache: 0x14265600
2014-06-12 14:45:54.932640 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] get_object_context: 0x14265600 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 rwstate(write n=3 w=0) oi: 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3(173'2707 client.4314.0:144974 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2707) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:45:54.932656 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] find_object_context 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 @head oi=89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3(173'2707 client.4314.0:144974 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2707)
2014-06-12 14:45:54.932679 7f9d67be0700  5 -- op tracker -- , seq: 747290, time: 2014-06-12 14:45:54.932673, event: started, op: osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.932691 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] execute_ctx 0x5df9000
2014-06-12 14:45:54.932709 7f9d67be0700  5 -- op tracker -- , seq: 747290, time: 2014-06-12 14:45:54.932704, event: started, op: osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.932716 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] do_op 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] ov 173'2707 av 173'2708 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:54.932750 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] do_osd_op 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096]
2014-06-12 14:45:54.932762 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:54.932776 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] do_osd_op  write 2473984~516096
2014-06-12 14:45:54.932790 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:54.932801 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:45:54.932813 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:54.932824 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] finish_ctx 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 0x5df9000 op modify  
2014-06-12 14:45:54.932836 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]  set mtime to 2014-06-12 08:44:57.310843
2014-06-12 14:45:54.932875 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]  final snapset 0=[]:[]+head in 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.932894 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]  zeroing write result code 0
2014-06-12 14:45:54.932904 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] new_repop rep_tid 236132 on osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.932920 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] issue_repop rep_tid 236132 o 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.932943 7f9d67be0700  5 -- op tracker -- , seq: 747290, time: 2014-06-12 14:45:54.932938, event: waiting for subops from 23, op: osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.932963 7f9d67be0700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:54.932985 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2707 (0'0,173'2707] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] append_log log((0'0,173'2707], crt=0'0) [173'2708 (173'2707) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144975 2014-06-12 08:44:57.310843]
2014-06-12 14:45:54.933007 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] add_log_entry 173'2708 (173'2707) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144975 2014-06-12 08:44:57.310843
2014-06-12 14:45:54.933028 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] append_log  adding 1 keys
2014-06-12 14:45:54.933059 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2708, trimmed: 
2014-06-12 14:45:54.933079 7f9d67be0700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.933083 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.933085 7f9d67be0700  5 queue_op 0x6956660 seq 2453 osr(3.54b 0x276db30) 518389 bytes   (queue has 4 ops and 2073556 bytes)
2014-06-12 14:45:54.933092 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] eval_repop repgather(0x1ced2a80 173'2708 rep_tid=236132 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.933111 7f9d67be0700 10 osd.2 173 dequeue_op 0x1e3ea800 finish
2014-06-12 14:45:54.933115 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.933118 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.933126 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (4 active)
2014-06-12 14:45:54.934160 7f9d57543700  5 -- op tracker -- , seq: 747291, time: 2014-06-12 14:45:54.934102, event: header_read, op: osd_sub_op_reply(client.4314.0:144972 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.934170 7f9d57543700  5 -- op tracker -- , seq: 747291, time: 2014-06-12 14:45:54.934106, event: throttled, op: osd_sub_op_reply(client.4314.0:144972 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.934178 7f9d57543700  5 -- op tracker -- , seq: 747291, time: 2014-06-12 14:45:54.934120, event: all_read, op: osd_sub_op_reply(client.4314.0:144972 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.934185 7f9d57543700  5 -- op tracker -- , seq: 747291, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144972 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.934191 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144972 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:54.934197 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:54.934205 7f9d57543700 15 osd.2 173 enqueue_op 0x12bf7e00 prio 196 cost 0 latency 0.000103 osd_sub_op_reply(client.4314.0:144972 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:54.934236 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.934245 7f9d693e3700 10 osd.2 173 dequeue_op 0x12bf7e00 prio 196 cost 0 latency 0.000142 osd_sub_op_reply(client.4314.0:144972 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]
2014-06-12 14:45:54.934266 7f9d693e3700  5 -- op tracker -- , seq: 747291, time: 2014-06-12 14:45:54.934263, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144972 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.934276 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] handle_message: 0x12bf7e00
2014-06-12 14:45:54.934292 7f9d693e3700  5 -- op tracker -- , seq: 747291, time: 2014-06-12 14:45:54.934289, event: started, op: osd_sub_op_reply(client.4314.0:144972 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.934302 7f9d7815d700 10 _do_op 0x7697860 seq 2450 r = 0, finisher 0x960d5c0 0x381e7900
2014-06-12 14:45:54.934298 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] sub_op_modify_reply: tid 236129 op  ack_type 4 from 23
2014-06-12 14:45:54.934319 7f9d693e3700  5 -- op tracker -- , seq: 747282, time: 2014-06-12 14:45:54.934311, event: sub_op_commit_rec, op: osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.934326 7f9d693e3700 10 osd.2 173 dequeue_op 0x12bf7e00 finish
2014-06-12 14:45:54.934327 7f9d7815d700 10 _finish_op 0x7697860 seq 2450 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.934338 7f9d693e3700  5 -- op tracker -- , seq: 747291, time: 2014-06-12 14:45:54.934334, event: done, op: osd_sub_op_reply(client.4314.0:144972 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.934352 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.934341 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] op_commit: 236129
2014-06-12 14:45:54.934356 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.934363 7f9d8116f700  5 _do_op 0x15cda900 seq 2451 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:45:54.934367 7f9d8116f700 10 _do_transaction on 0x96da600
2014-06-12 14:45:54.934364 7f9d7715b700  5 -- op tracker -- , seq: 747282, time: 2014-06-12 14:45:54.934357, event: op_commit, op: osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.934375 7f9d8116f700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.934372 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] repop_all_committed: repop tid 236129 all committed 
2014-06-12 14:45:54.934384 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean] eval_repop repgather(0x1c832e40 173'2705 rep_tid=236129 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.934401 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean] log_op_stats osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.037411
2014-06-12 14:45:54.934418 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean] publish_stats_to_osd 173:3598
2014-06-12 14:45:54.934429 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean]  sending commit on repgather(0x1c832e40 173'2705 rep_tid=236129 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x205c000
2014-06-12 14:45:54.934466 7f9d7715b700  5 -- op tracker -- , seq: 747282, time: 2014-06-12 14:45:54.934460, event: commit_sent, op: osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.934499 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.934490 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean] op_applied: 236129
2014-06-12 14:45:54.934509 7f9d7795c700  5 -- op tracker -- , seq: 747282, time: 2014-06-12 14:45:54.934504, event: op_applied, op: osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.934516 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean] op_applied on primary on version 173'2705
2014-06-12 14:45:54.934524 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.934526 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean] repop_all_applied: repop tid 236129 all applied 
2014-06-12 14:45:54.934540 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean] eval_repop repgather(0x1c832e40 173'2705 rep_tid=236129 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.934555 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean] log_op_stats osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.037566
2014-06-12 14:45:54.934569 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean] publish_stats_to_osd 173:3599
2014-06-12 14:45:54.934577 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:54.934579 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean]  requeue_ops 
2014-06-12 14:45:54.934589 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean]  removing repgather(0x1c832e40 173'2705 rep_tid=236129 committed?=1 applied?=1 op=osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.934600 7f9d8116f700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.934607 7f9d8116f700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:45:54.934613 7f9d8116f700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.934603 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean]    q front is repgather(0x1c832e40 173'2705 rep_tid=236129 committed?=1 applied?=1 op=osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.934616 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean] remove_repop repgather(0x1c832e40 173'2705 rep_tid=236129 committed?=1 applied?=1 op=osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.934629 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean]  requeue_ops 
2014-06-12 14:45:54.934673 7f9d7815d700  5 -- op tracker -- , seq: 747282, time: 2014-06-12 14:45:54.934659, event: done, op: osd_op(client.4314.0:144972 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.934698 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (3 active)
2014-06-12 14:45:54.934701 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:54.934834 7f9d8116f700 15 _write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 1441792~516096
2014-06-12 14:45:54.935015 7f9d8116f700 10 lookup_strip_header done  cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.935385 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:45:54.940733 7f9d8116f700 10 _generic_write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 1441792~516096 = 0
2014-06-12 14:45:54.940818 7f9d8116f700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.940837 7f9d8116f700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.940844 7f9d8116f700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.940853 7f9d8116f700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.941091 7f9d8116f700 20 set_header setting 17860 cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 parent seq 0
2014-06-12 14:45:54.941198 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:54.941220 7f9d8116f700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:45:54.941236 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:45:54.944217 7f9d8116f700 10 _do_op 0x15cda900 seq 2451 r = 0, finisher 0x90719e0 0x2519ab40
2014-06-12 14:45:54.944248 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean] op_commit: 236130
2014-06-12 14:45:54.944286 7f9d7715b700  5 -- op tracker -- , seq: 747283, time: 2014-06-12 14:45:54.944275, event: op_commit, op: osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.944312 7f9d8116f700 10 _finish_op 0x15cda900 seq 2451 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.944328 7f9d79960700  5 _do_op 0x1415e540 seq 2452 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:45:54.944333 7f9d79960700 10 _do_transaction on 0x5236c00
2014-06-12 14:45:54.944340 7f9d79960700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.944461 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.944481 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.944537 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:54.944554 7f9d79960700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.944559 7f9d79960700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:45:54.944566 7f9d79960700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.944580 7f9d79960700 15 _write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 1957888~516096
2014-06-12 14:45:54.944725 7f9d79960700 10 lookup_strip_header done  cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.944836 7f9d79960700 10 file_to_extents done 
2014-06-12 14:45:54.945036 7f9d57543700  5 -- op tracker -- , seq: 747292, time: 2014-06-12 14:45:54.944988, event: header_read, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.945047 7f9d57543700  5 -- op tracker -- , seq: 747292, time: 2014-06-12 14:45:54.944990, event: throttled, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.945055 7f9d57543700  5 -- op tracker -- , seq: 747292, time: 2014-06-12 14:45:54.945005, event: all_read, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.945064 7f9d57543700  5 -- op tracker -- , seq: 747292, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.945070 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:45:54.945076 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:54.945085 7f9d57543700 15 osd.2 173 enqueue_op 0xe20b000 prio 196 cost 0 latency 0.000097 osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0) v2
2014-06-12 14:45:54.945135 7f9d57543700  5 -- op tracker -- , seq: 747293, time: 2014-06-12 14:45:54.945110, event: header_read, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.945143 7f9d57543700  5 -- op tracker -- , seq: 747293, time: 2014-06-12 14:45:54.945111, event: throttled, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.945152 7f9d57543700  5 -- op tracker -- , seq: 747293, time: 2014-06-12 14:45:54.945119, event: all_read, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.945160 7f9d57543700  5 -- op tracker -- , seq: 747293, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.945164 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:54.945169 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:54.945174 7f9d57543700 15 osd.2 173 enqueue_op 0x5289200 prio 196 cost 0 latency 0.000064 osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:54.945248 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.945256 7f9d62bd6700 10 osd.2 173 dequeue_op 0xe20b000 prio 196 cost 0 latency 0.000268 osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0) v2 pg pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean]
2014-06-12 14:45:54.945275 7f9d62bd6700  5 -- op tracker -- , seq: 747292, time: 2014-06-12 14:45:54.945272, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.945286 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean] handle_message: 0xe20b000
2014-06-12 14:45:54.945301 7f9d62bd6700  5 -- op tracker -- , seq: 747292, time: 2014-06-12 14:45:54.945298, event: started, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.945307 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean] sub_op_modify_reply: tid 236130 op  ack_type 1 from 23
2014-06-12 14:45:54.945333 7f9d62bd6700  5 -- op tracker -- , seq: 747283, time: 2014-06-12 14:45:54.945326, event: sub_op_applied_rec, op: osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.945341 7f9d62bd6700 10 osd.2 173 dequeue_op 0xe20b000 finish
2014-06-12 14:45:54.945347 7f9d62bd6700  5 -- op tracker -- , seq: 747292, time: 2014-06-12 14:45:54.945344, event: done, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.945361 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.945367 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.945371 7f9d62bd6700 10 osd.2 173 dequeue_op 0x5289200 prio 196 cost 0 latency 0.000261 osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean]
2014-06-12 14:45:54.945387 7f9d62bd6700  5 -- op tracker -- , seq: 747293, time: 2014-06-12 14:45:54.945384, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.945393 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean] handle_message: 0x5289200
2014-06-12 14:45:54.945406 7f9d62bd6700  5 -- op tracker -- , seq: 747293, time: 2014-06-12 14:45:54.945403, event: started, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.945411 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean] sub_op_modify_reply: tid 236130 op  ack_type 4 from 23
2014-06-12 14:45:54.945427 7f9d62bd6700  5 -- op tracker -- , seq: 747283, time: 2014-06-12 14:45:54.945422, event: sub_op_commit_rec, op: osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.945433 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean] repop_all_committed: repop tid 236130 all committed 
2014-06-12 14:45:54.945443 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean] eval_repop repgather(0x5147ec0 173'2706 rep_tid=236130 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.945459 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean] log_op_stats osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.043492
2014-06-12 14:45:54.945478 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean] publish_stats_to_osd 173:3600
2014-06-12 14:45:54.945491 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean]  sending commit on repgather(0x5147ec0 173'2706 rep_tid=236130 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x2b42000
2014-06-12 14:45:54.945526 7f9d62bd6700  5 -- op tracker -- , seq: 747283, time: 2014-06-12 14:45:54.945521, event: commit_sent, op: osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.945534 7f9d62bd6700 10 osd.2 173 dequeue_op 0x5289200 finish
2014-06-12 14:45:54.945540 7f9d62bd6700  5 -- op tracker -- , seq: 747293, time: 2014-06-12 14:45:54.945537, event: done, op: osd_sub_op_reply(client.4314.0:144973 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.945549 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.945552 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.945562 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.946841 7f9d79960700 10 _generic_write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 1957888~516096 = 0
2014-06-12 14:45:54.946883 7f9d79960700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.946900 7f9d79960700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.946906 7f9d79960700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.946915 7f9d79960700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.947133 7f9d79960700 20 set_header setting 17860 cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 parent seq 0
2014-06-12 14:45:54.947203 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:54.947231 7f9d79960700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:45:54.947242 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:45:54.950430 7f9d79960700 10 _do_op 0x1415e540 seq 2452 r = 0, finisher 0x1f086880 0x352bd4c0
2014-06-12 14:45:54.950456 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean] op_commit: 236131
2014-06-12 14:45:54.950488 7f9d7715b700  5 -- op tracker -- , seq: 747288, time: 2014-06-12 14:45:54.950479, event: op_commit, op: osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.950516 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (2 active)
2014-06-12 14:45:54.950531 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:54.950541 7f9d79960700 10 _finish_op 0x1415e540 seq 2452 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.950530 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean] op_applied: 236130
2014-06-12 14:45:54.950555 7f9d7795c700  5 -- op tracker -- , seq: 747283, time: 2014-06-12 14:45:54.950549, event: op_applied, op: osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.950552 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (1 active)
2014-06-12 14:45:54.950564 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:54.950569 7f9d8016d700  5 _do_op 0x6956660 seq 2453 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:45:54.950564 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean] op_applied on primary on version 173'2706
2014-06-12 14:45:54.950574 7f9d8016d700 10 _do_transaction on 0x2f89f140
2014-06-12 14:45:54.950573 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean] repop_all_applied: repop tid 236130 all applied 
2014-06-12 14:45:54.950584 7f9d8016d700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.950582 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean] eval_repop repgather(0x5147ec0 173'2706 rep_tid=236130 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.950597 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean] log_op_stats osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.048629
2014-06-12 14:45:54.950611 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean] publish_stats_to_osd 173:3601
2014-06-12 14:45:54.950620 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean]  requeue_ops 
2014-06-12 14:45:54.950629 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]  removing repgather(0x5147ec0 173'2706 rep_tid=236130 committed?=1 applied?=1 op=osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.950641 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]    q front is repgather(0x5147ec0 173'2706 rep_tid=236130 committed?=1 applied?=1 op=osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.950653 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] remove_repop repgather(0x5147ec0 173'2706 rep_tid=236130 committed?=1 applied?=1 op=osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.950664 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]  requeue_ops 
2014-06-12 14:45:54.950680 7f9d7795c700  5 -- op tracker -- , seq: 747283, time: 2014-06-12 14:45:54.950676, event: done, op: osd_op(client.4314.0:144973 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.950718 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.950714 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] op_applied: 236131
2014-06-12 14:45:54.950729 7f9d7795c700  5 -- op tracker -- , seq: 747288, time: 2014-06-12 14:45:54.950724, event: op_applied, op: osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.950735 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] op_applied on primary on version 173'2707
2014-06-12 14:45:54.950745 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.950799 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:54.950824 7f9d8016d700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.950830 7f9d8016d700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:45:54.950837 7f9d8016d700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.950855 7f9d8016d700 15 _write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 2473984~516096
2014-06-12 14:45:54.951006 7f9d8016d700 10 lookup_strip_header done  cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.951032 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:45:54.953238 7f9d8016d700 10 _generic_write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 2473984~516096 = 0
2014-06-12 14:45:54.953283 7f9d8016d700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.953302 7f9d8016d700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.953313 7f9d8016d700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.953321 7f9d8016d700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.953531 7f9d8016d700 20 set_header setting 17860 cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 parent seq 0
2014-06-12 14:45:54.953608 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:54.953643 7f9d8016d700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:45:54.953655 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:45:54.953688 7f9d57543700  5 -- op tracker -- , seq: 747294, time: 2014-06-12 14:45:54.953633, event: header_read, op: osd_sub_op_reply(client.4314.0:144974 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.953701 7f9d57543700  5 -- op tracker -- , seq: 747294, time: 2014-06-12 14:45:54.953637, event: throttled, op: osd_sub_op_reply(client.4314.0:144974 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.953710 7f9d57543700  5 -- op tracker -- , seq: 747294, time: 2014-06-12 14:45:54.953651, event: all_read, op: osd_sub_op_reply(client.4314.0:144974 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.953718 7f9d57543700  5 -- op tracker -- , seq: 747294, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144974 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.953724 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144974 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:54.953735 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:54.953743 7f9d57543700 15 osd.2 173 enqueue_op 0x6d11b00 prio 196 cost 0 latency 0.000109 osd_sub_op_reply(client.4314.0:144974 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:54.956524 7f9d8016d700 10 _do_op 0x6956660 seq 2453 r = 0, finisher 0x1e3bc960 0x111fa2c0
2014-06-12 14:45:54.956551 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] op_commit: 236132
2014-06-12 14:45:54.956579 7f9d7715b700  5 -- op tracker -- , seq: 747290, time: 2014-06-12 14:45:54.956571, event: op_commit, op: osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.956594 7f9d8016d700 10 _finish_op 0x6956660 seq 2453 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.956610 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (0 active)
2014-06-12 14:45:54.956620 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:54.956901 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] op_applied: 236132
2014-06-12 14:45:54.956928 7f9d7795c700  5 -- op tracker -- , seq: 747290, time: 2014-06-12 14:45:54.956921, event: op_applied, op: osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.956938 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] op_applied on primary on version 173'2708
2014-06-12 14:45:54.957060 7f9d57543700  5 -- op tracker -- , seq: 747295, time: 2014-06-12 14:45:54.957028, event: header_read, op: osd_sub_op_reply(client.4314.0:144975 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.957069 7f9d57543700  5 -- op tracker -- , seq: 747295, time: 2014-06-12 14:45:54.957030, event: throttled, op: osd_sub_op_reply(client.4314.0:144975 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.957076 7f9d57543700  5 -- op tracker -- , seq: 747295, time: 2014-06-12 14:45:54.957038, event: all_read, op: osd_sub_op_reply(client.4314.0:144975 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.957083 7f9d57543700  5 -- op tracker -- , seq: 747295, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144975 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.957086 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144975 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:54.957091 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:54.957100 7f9d57543700 15 osd.2 173 enqueue_op 0x2910a00 prio 196 cost 0 latency 0.000072 osd_sub_op_reply(client.4314.0:144975 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:54.957125 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.957131 7f9d64bda700 10 osd.2 173 dequeue_op 0x6d11b00 prio 196 cost 0 latency 0.003498 osd_sub_op_reply(client.4314.0:144974 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]
2014-06-12 14:45:54.957147 7f9d64bda700  5 -- op tracker -- , seq: 747294, time: 2014-06-12 14:45:54.957144, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144974 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.957154 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] handle_message: 0x6d11b00
2014-06-12 14:45:54.957167 7f9d64bda700  5 -- op tracker -- , seq: 747294, time: 2014-06-12 14:45:54.957164, event: started, op: osd_sub_op_reply(client.4314.0:144974 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.957171 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] sub_op_modify_reply: tid 236131 op  ack_type 4 from 23
2014-06-12 14:45:54.957191 7f9d64bda700  5 -- op tracker -- , seq: 747288, time: 2014-06-12 14:45:54.957185, event: sub_op_commit_rec, op: osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.957196 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] repop_all_applied: repop tid 236131 all applied 
2014-06-12 14:45:54.957206 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] eval_repop repgather(0x12ae2540 173'2707 rep_tid=236131 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.957219 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]  sending ack on repgather(0x12ae2540 173'2707 rep_tid=236131 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x16ad1400
2014-06-12 14:45:54.957244 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] repop_all_committed: repop tid 236131 all committed 
2014-06-12 14:45:54.957261 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:45:54.957253 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean] eval_repop repgather(0x12ae2540 173'2707 rep_tid=236131 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.957267 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean] log_op_stats osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.031686 lat 0.031709
2014-06-12 14:45:54.957280 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean] publish_stats_to_osd 173:3602
2014-06-12 14:45:54.957291 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean]  sending commit on repgather(0x12ae2540 173'2707 rep_tid=236131 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x26fdef00
2014-06-12 14:45:54.957316 7f9d64bda700  5 -- op tracker -- , seq: 747288, time: 2014-06-12 14:45:54.957311, event: commit_sent, op: osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.957322 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean]  requeue_ops 
2014-06-12 14:45:54.957331 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]  removing repgather(0x12ae2540 173'2707 rep_tid=236131 committed?=1 applied?=1 op=osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.957343 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]    q front is repgather(0x12ae2540 173'2707 rep_tid=236131 committed?=1 applied?=1 op=osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.957357 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] remove_repop repgather(0x12ae2540 173'2707 rep_tid=236131 committed?=1 applied?=1 op=osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.957369 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]  requeue_ops 
2014-06-12 14:45:54.957385 7f9d64bda700  5 -- op tracker -- , seq: 747288, time: 2014-06-12 14:45:54.957380, event: done, op: osd_op(client.4314.0:144974 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.957400 7f9d64bda700 10 osd.2 173 dequeue_op 0x6d11b00 finish
2014-06-12 14:45:54.957409 7f9d64bda700  5 -- op tracker -- , seq: 747294, time: 2014-06-12 14:45:54.957406, event: done, op: osd_sub_op_reply(client.4314.0:144974 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.957418 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:45:54.957420 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.958981 7f9d425f5700  5 -- op tracker -- , seq: 747296, time: 2014-06-12 14:45:54.956645, event: header_read, op: osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.958997 7f9d425f5700  5 -- op tracker -- , seq: 747296, time: 2014-06-12 14:45:54.956647, event: throttled, op: osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.959007 7f9d425f5700  5 -- op tracker -- , seq: 747296, time: 2014-06-12 14:45:54.957374, event: all_read, op: osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.959017 7f9d425f5700  5 -- op tracker -- , seq: 747296, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.959025 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:45:54.959050 7f9d425f5700 15 osd.2 173 enqueue_op 0x1e3ea100 prio 63 cost 516096 latency 0.002405 osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.959077 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:45:54.959084 7f9d683e1700 10 osd.2 173 dequeue_op 0x2910a00 prio 196 cost 0 latency 0.002056 osd_sub_op_reply(client.4314.0:144975 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]
2014-06-12 14:45:54.959105 7f9d683e1700  5 -- op tracker -- , seq: 747295, time: 2014-06-12 14:45:54.959101, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144975 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.959113 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] handle_message: 0x2910a00
2014-06-12 14:45:54.959129 7f9d683e1700  5 -- op tracker -- , seq: 747295, time: 2014-06-12 14:45:54.959125, event: started, op: osd_sub_op_reply(client.4314.0:144975 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.959135 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sub_op_modify_reply: tid 236132 op  ack_type 4 from 23
2014-06-12 14:45:54.959155 7f9d683e1700  5 -- op tracker -- , seq: 747290, time: 2014-06-12 14:45:54.959147, event: sub_op_commit_rec, op: osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.959161 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] repop_all_applied: repop tid 236132 all applied 
2014-06-12 14:45:54.959172 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] eval_repop repgather(0x1ced2a80 173'2708 rep_tid=236132 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.959188 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]  sending ack on repgather(0x1ced2a80 173'2708 rep_tid=236132 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x1ba78c80
2014-06-12 14:45:54.959334 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] repop_all_committed: repop tid 236132 all committed 
2014-06-12 14:45:54.959346 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean] eval_repop repgather(0x1ced2a80 173'2708 rep_tid=236132 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.959362 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean] log_op_stats osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.028447 lat 0.028475
2014-06-12 14:45:54.959379 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean] publish_stats_to_osd 173:3603
2014-06-12 14:45:54.959392 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean]  sending commit on repgather(0x1ced2a80 173'2708 rep_tid=236132 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x1687a000
2014-06-12 14:45:54.959421 7f9d683e1700  5 -- op tracker -- , seq: 747290, time: 2014-06-12 14:45:54.959416, event: commit_sent, op: osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.959429 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean]  requeue_ops 
2014-06-12 14:45:54.959439 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  removing repgather(0x1ced2a80 173'2708 rep_tid=236132 committed?=1 applied?=1 op=osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.959453 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]    q front is repgather(0x1ced2a80 173'2708 rep_tid=236132 committed?=1 applied?=1 op=osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.959467 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] remove_repop repgather(0x1ced2a80 173'2708 rep_tid=236132 committed?=1 applied?=1 op=osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.959480 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  requeue_ops 
2014-06-12 14:45:54.959506 7f9d683e1700  5 -- op tracker -- , seq: 747290, time: 2014-06-12 14:45:54.959501, event: done, op: osd_op(client.4314.0:144975 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.959522 7f9d683e1700 10 osd.2 173 dequeue_op 0x2910a00 finish
2014-06-12 14:45:54.959529 7f9d683e1700  5 -- op tracker -- , seq: 747295, time: 2014-06-12 14:45:54.959526, event: done, op: osd_sub_op_reply(client.4314.0:144975 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.959541 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:45:54.959544 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.959734 7f9d425f5700  5 -- op tracker -- , seq: 747297, time: 2014-06-12 14:45:54.959077, event: header_read, op: osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.959746 7f9d425f5700  5 -- op tracker -- , seq: 747297, time: 2014-06-12 14:45:54.959078, event: throttled, op: osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.959755 7f9d425f5700  5 -- op tracker -- , seq: 747297, time: 2014-06-12 14:45:54.959709, event: all_read, op: osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.959765 7f9d425f5700  5 -- op tracker -- , seq: 747297, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.959770 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:45:54.959786 7f9d425f5700 15 osd.2 173 enqueue_op 0x1e3ea200 prio 63 cost 516096 latency 0.000709 osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.959810 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:45:54.959816 7f9d6abe6700 10 osd.2 173 dequeue_op 0x1e3ea100 prio 63 cost 516096 latency 0.003170 osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]
2014-06-12 14:45:54.959838 7f9d6abe6700  5 -- op tracker -- , seq: 747296, time: 2014-06-12 14:45:54.959833, event: reached_pg, op: osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.959876 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:54.959891 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] handle_message: 0x1e3ea100
2014-06-12 14:45:54.959901 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_op osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:54.959925 7f9d6abe6700 15 getattr 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 '_'
2014-06-12 14:45:54.960020 7f9d425f5700  5 -- op tracker -- , seq: 747298, time: 2014-06-12 14:45:54.959812, event: header_read, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.960031 7f9d425f5700  5 -- op tracker -- , seq: 747298, time: 2014-06-12 14:45:54.959813, event: throttled, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.960041 7f9d425f5700  5 -- op tracker -- , seq: 747298, time: 2014-06-12 14:45:54.959997, event: all_read, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.960051 7f9d425f5700  5 -- op tracker -- , seq: 747298, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.960056 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:45:54.960070 7f9d425f5700 15 osd.2 173 enqueue_op 0x5ee9600 prio 63 cost 172032 latency 0.000258 osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.960142 7f9d6abe6700 10 getattr 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 '_' = 0
2014-06-12 14:45:54.960158 7f9d6abe6700 15 getattr 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 'snapset'
2014-06-12 14:45:54.960249 7f9d6abe6700 10 getattr 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 'snapset' = 0
2014-06-12 14:45:54.960258 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] populate_obc_watchers 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.960270 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.960279 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] get_object_context: creating obc from disk: 0x28012dc0
2014-06-12 14:45:54.960288 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] get_object_context: 0x28012dc0 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 rwstate(none n=0 w=0) oi: 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3(173'2708 client.4314.0:144975 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2708) ssc: 0x2ef8eb60 snapset: 0=[]:[]+head
2014-06-12 14:45:54.960304 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] find_object_context 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 @head oi=89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3(173'2708 client.4314.0:144975 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2708)
2014-06-12 14:45:54.960328 7f9d6abe6700  5 -- op tracker -- , seq: 747296, time: 2014-06-12 14:45:54.960323, event: started, op: osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.960343 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] execute_ctx 0x2963d000
2014-06-12 14:45:54.960360 7f9d6abe6700  5 -- op tracker -- , seq: 747296, time: 2014-06-12 14:45:54.960355, event: started, op: osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.960367 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_op 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] ov 173'2708 av 173'2709 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:54.960381 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096]
2014-06-12 14:45:54.960392 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:54.960410 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op  write 2990080~516096
2014-06-12 14:45:54.960467 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:54.960480 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 snapset=0x2ef8eb98  snapc=0=[]
2014-06-12 14:45:54.960492 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:54.960501 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] finish_ctx 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 0x2963d000 op modify  
2014-06-12 14:45:54.960514 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  set mtime to 2014-06-12 08:44:57.317470
2014-06-12 14:45:54.960540 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  final snapset 0=[]:[]+head in 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.960558 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  zeroing write result code 0
2014-06-12 14:45:54.960568 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] new_repop rep_tid 236133 on osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.960584 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] issue_repop rep_tid 236133 o 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.960608 7f9d6abe6700  5 -- op tracker -- , seq: 747296, time: 2014-06-12 14:45:54.960603, event: waiting for subops from 23, op: osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.960629 7f9d6abe6700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:54.960650 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2708 (0'0,173'2708] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] append_log log((0'0,173'2708], crt=0'0) [173'2709 (173'2708) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144976 2014-06-12 08:44:57.317470]
2014-06-12 14:45:54.960673 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] add_log_entry 173'2709 (173'2708) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144976 2014-06-12 08:44:57.317470
2014-06-12 14:45:54.961174 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] append_log  adding 1 keys
2014-06-12 14:45:54.961213 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2709, trimmed: 
2014-06-12 14:45:54.961244 7f9d6abe6700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.961248 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.961250 7f9d6abe6700  5 queue_op 0x6872300 seq 2454 osr(3.54b 0x276db30) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:45:54.961258 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] eval_repop repgather(0x18225e00 173'2709 rep_tid=236133 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.961285 7f9d6abe6700 10 osd.2 173 dequeue_op 0x1e3ea100 finish
2014-06-12 14:45:54.961290 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:45:54.961298 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:45:54.961302 7f9d6abe6700 10 osd.2 173 dequeue_op 0x1e3ea200 prio 63 cost 516096 latency 0.002225 osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]
2014-06-12 14:45:54.961334 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (1 active)
2014-06-12 14:45:54.961339 7f9d8096e700  5 _do_op 0x6872300 seq 2454 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:45:54.961343 7f9d8096e700 10 _do_transaction on 0xd5e2cc0
2014-06-12 14:45:54.961349 7f9d8096e700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.961561 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.961584 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.961649 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:54.961679 7f9d8096e700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.961688 7f9d8096e700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:45:54.961695 7f9d8096e700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.961712 7f9d8096e700 15 _write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 2990080~516096
2014-06-12 14:45:54.962015 7f9d8096e700 10 lookup_strip_header done  cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.962159 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:45:54.964003 7f9d8096e700 10 _generic_write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 2990080~516096 = 0
2014-06-12 14:45:54.964062 7f9d8096e700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.964086 7f9d8096e700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.964099 7f9d8096e700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.964108 7f9d8096e700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.964321 7f9d8096e700 20 set_header setting 17860 cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 parent seq 0
2014-06-12 14:45:54.964390 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:54.964407 7f9d8096e700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:45:54.964422 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:45:54.965633 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.961328 7f9d6abe6700  5 -- op tracker -- , seq: 747297, time: 2014-06-12 14:45:54.961323, event: reached_pg, op: osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.965679 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:54.965705 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] handle_message: 0x1e3ea200
2014-06-12 14:45:54.965711 7f9d425f5700  5 -- op tracker -- , seq: 747299, time: 2014-06-12 14:45:54.960340, event: header_read, op: osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.965724 7f9d425f5700  5 -- op tracker -- , seq: 747299, time: 2014-06-12 14:45:54.960341, event: throttled, op: osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.965718 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_op osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:54.965735 7f9d425f5700  5 -- op tracker -- , seq: 747299, time: 2014-06-12 14:45:54.961305, event: all_read, op: osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.965744 7f9d425f5700  5 -- op tracker -- , seq: 747299, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.965746 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] get_object_context: found obc in cache: 0x28012dc0
2014-06-12 14:45:54.965753 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:45:54.965757 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] get_object_context: 0x28012dc0 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 rwstate(write n=1 w=0) oi: 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3(173'2709 client.4314.0:144976 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2709) ssc: 0x2ef8eb60 snapset: 0=[]:[]+head
2014-06-12 14:45:54.965773 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] find_object_context 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 @head oi=89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3(173'2709 client.4314.0:144976 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2709)
2014-06-12 14:45:54.965785 7f9d425f5700 15 osd.2 173 enqueue_op 0x5ee8700 prio 63 cost 344064 latency 0.005444 osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.965800 7f9d6abe6700  5 -- op tracker -- , seq: 747297, time: 2014-06-12 14:45:54.965794, event: started, op: osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.965808 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] execute_ctx 0x1474ca00
2014-06-12 14:45:54.965821 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active)
2014-06-12 14:45:54.965826 7f9d6abe6700  5 -- op tracker -- , seq: 747297, time: 2014-06-12 14:45:54.965821, event: started, op: osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.965828 7f9d66bde700 10 osd.2 173 dequeue_op 0x5ee8700 prio 63 cost 344064 latency 0.005488 osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean]
2014-06-12 14:45:54.965833 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_op 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] ov 173'2709 av 173'2710 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:54.965851 7f9d66bde700  5 -- op tracker -- , seq: 747299, time: 2014-06-12 14:45:54.965846, event: reached_pg, op: osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.965848 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096]
2014-06-12 14:45:54.965859 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:54.965859 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:54.965873 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] handle_message: 0x5ee8700
2014-06-12 14:45:54.965874 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op  write 3506176~516096
2014-06-12 14:45:54.965883 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] do_op osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:54.965890 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:54.965904 7f9d66bde700 15 getattr 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 '_'
2014-06-12 14:45:54.965902 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 snapset=0x2ef8eb98  snapc=0=[]
2014-06-12 14:45:54.965915 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:54.965925 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] finish_ctx 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 0x1474ca00 op modify  
2014-06-12 14:45:54.965938 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  set mtime to 2014-06-12 08:44:57.333033
2014-06-12 14:45:54.965965 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  final snapset 0=[]:[]+head in 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.965984 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  zeroing write result code 0
2014-06-12 14:45:54.965996 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] new_repop rep_tid 236134 on osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.966012 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] issue_repop rep_tid 236134 o 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.966038 7f9d6abe6700  5 -- op tracker -- , seq: 747297, time: 2014-06-12 14:45:54.966033, event: waiting for subops from 23, op: osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.966065 7f9d6abe6700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:54.966087 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2709 (0'0,173'2709] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] append_log log((0'0,173'2709], crt=0'0) [173'2710 (173'2709) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144977 2014-06-12 08:44:57.333033]
2014-06-12 14:45:54.966111 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] add_log_entry 173'2710 (173'2709) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144977 2014-06-12 08:44:57.333033
2014-06-12 14:45:54.966136 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] append_log  adding 1 keys
2014-06-12 14:45:54.966146 7f9d66bde700 10 getattr 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 '_' = 0
2014-06-12 14:45:54.966163 7f9d66bde700 15 getattr 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 'snapset'
2014-06-12 14:45:54.966171 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2710, trimmed: 
2014-06-12 14:45:54.966194 7f9d6abe6700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.966199 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.966200 7f9d6abe6700  5 queue_op 0x53c2a20 seq 2455 osr(3.54b 0x276db30) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:45:54.966209 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] eval_repop repgather(0x534ef00 173'2710 rep_tid=236134 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.966228 7f9d6abe6700 10 osd.2 173 dequeue_op 0x1e3ea200 finish
2014-06-12 14:45:54.966235 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active)
2014-06-12 14:45:54.966238 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.966255 7f9d66bde700 10 getattr 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 'snapset' = 0
2014-06-12 14:45:54.966265 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] populate_obc_watchers 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.966277 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.966287 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] get_object_context: creating obc from disk: 0x1be4d080
2014-06-12 14:45:54.966296 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] get_object_context: 0x1be4d080 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 rwstate(none n=0 w=0) oi: 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'1112 client.4314.0:39074 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1112) ssc: 0x72b9420 snapset: 0=[]:[]+head
2014-06-12 14:45:54.966311 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] find_object_context 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 @head oi=61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'1112 client.4314.0:39074 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1112)
2014-06-12 14:45:54.966333 7f9d66bde700  5 -- op tracker -- , seq: 747299, time: 2014-06-12 14:45:54.966327, event: started, op: osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.966341 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] execute_ctx 0x292d000
2014-06-12 14:45:54.966357 7f9d66bde700  5 -- op tracker -- , seq: 747299, time: 2014-06-12 14:45:54.966352, event: started, op: osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.966363 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] do_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] ov 173'1112 av 173'3321 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:54.966376 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] do_osd_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064]
2014-06-12 14:45:54.966386 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:54.966398 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] do_osd_op  write 0~344064
2014-06-12 14:45:54.966412 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:54.966423 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 snapset=0x72b9458  snapc=0=[]
2014-06-12 14:45:54.966434 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:54.966444 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] finish_ctx 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 0x292d000 op modify  
2014-06-12 14:45:54.966455 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean]  set mtime to 2014-06-12 08:44:57.340339
2014-06-12 14:45:54.966474 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean]  final snapset 0=[]:[]+head in 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.966490 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean]  zeroing write result code 0
2014-06-12 14:45:54.966500 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] calc_trim_to 125'319 -> 125'320
2014-06-12 14:45:54.966509 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] new_repop rep_tid 236135 on osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.966523 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] issue_repop rep_tid 236135 o 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.966544 7f9d66bde700  5 -- op tracker -- , seq: 747299, time: 2014-06-12 14:45:54.966539, event: waiting for subops from 20, op: osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.966563 7f9d66bde700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:54.966587 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3320 (125'319,173'3320] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] append_log log((125'319,173'3320], crt=0'0) [173'3321 (173'1112) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144979 2014-06-12 08:44:57.340339]
2014-06-12 14:45:54.966610 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'319,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] add_log_entry 173'3321 (173'1112) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144979 2014-06-12 08:44:57.340339
2014-06-12 14:45:54.966630 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'319,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] append_log  adding 1 keys
2014-06-12 14:45:54.966642 7f9d66bde700 10 trim log((125'319,173'3321], crt=0'0) to 125'320
2014-06-12 14:45:54.966672 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3321, trimmed: 125'320
2014-06-12 14:45:54.966697 7f9d66bde700  5 queue_transactions existing osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:54.966701 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.966703 7f9d66bde700  5 queue_op 0x4063bc0 seq 2978 osr(3.395 0x276d740) 346392 bytes   (queue has 3 ops and 1383170 bytes)
2014-06-12 14:45:54.966711 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] eval_repop repgather(0x2562180 173'3321 rep_tid=236135 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.966729 7f9d66bde700 10 osd.2 173 dequeue_op 0x5ee8700 finish
2014-06-12 14:45:54.966734 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:45:54.966737 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.966747 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (2 active)
2014-06-12 14:45:54.967678 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227e360 (3 active)
2014-06-12 14:45:54.967687 7f9d7915f700  5 _do_op 0x4063bc0 seq 2978 osr(3.395 0x276d740)/0x276d740 start
2014-06-12 14:45:54.967692 7f9d7915f700 10 _do_transaction on 0x51ac6c0
2014-06-12 14:45:54.967700 7f9d7915f700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:54.967842 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:54.967869 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.967932 7f9d63bd8700 10 osd.2 173 dequeue_op 0x5ee9600 prio 63 cost 172032 latency 0.008119 osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]
2014-06-12 14:45:54.967964 7f9d63bd8700  5 -- op tracker -- , seq: 747298, time: 2014-06-12 14:45:54.967958, event: reached_pg, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.967972 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:54.967986 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] handle_message: 0x5ee9600
2014-06-12 14:45:54.967995 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_op osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:54.968017 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] get_object_context: found obc in cache: 0x28012dc0
2014-06-12 14:45:54.968026 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] get_object_context: 0x28012dc0 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 rwstate(write n=2 w=0) oi: 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3(173'2710 client.4314.0:144977 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2710) ssc: 0x2ef8eb60 snapset: 0=[]:[]+head
2014-06-12 14:45:54.968040 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] find_object_context 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 @head oi=89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3(173'2710 client.4314.0:144977 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2710)
2014-06-12 14:45:54.968064 7f9d63bd8700  5 -- op tracker -- , seq: 747298, time: 2014-06-12 14:45:54.968059, event: started, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.968070 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] execute_ctx 0x2e247800
2014-06-12 14:45:54.968085 7f9d63bd8700  5 -- op tracker -- , seq: 747298, time: 2014-06-12 14:45:54.968080, event: started, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.968090 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_op 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] ov 173'2710 av 173'2711 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:54.968102 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032]
2014-06-12 14:45:54.968112 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:54.968123 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op  write 4022272~172032
2014-06-12 14:45:54.968137 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:54.968146 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 snapset=0x2ef8eb98  snapc=0=[]
2014-06-12 14:45:54.968156 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] make_writeable 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:54.968169 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] finish_ctx 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 0x2e247800 op modify  
2014-06-12 14:45:54.968180 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  set mtime to 2014-06-12 08:44:57.340119
2014-06-12 14:45:54.968197 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  final snapset 0=[]:[]+head in 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.968214 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  zeroing write result code 0
2014-06-12 14:45:54.968223 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] new_repop rep_tid 236136 on osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.968237 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] issue_repop rep_tid 236136 o 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.968259 7f9d63bd8700  5 -- op tracker -- , seq: 747298, time: 2014-06-12 14:45:54.968254, event: waiting for subops from 23, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.968277 7f9d63bd8700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:54.968299 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2710 (0'0,173'2710] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] append_log log((0'0,173'2710], crt=0'0) [173'2711 (173'2710) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144978 2014-06-12 08:44:57.340119]
2014-06-12 14:45:54.968313 7f9d425f5700  5 -- op tracker -- , seq: 747300, time: 2014-06-12 14:45:54.966781, event: header_read, op: osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.968328 7f9d425f5700  5 -- op tracker -- , seq: 747300, time: 2014-06-12 14:45:54.966783, event: throttled, op: osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.968326 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] add_log_entry 173'2711 (173'2710) modify   89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 by client.4314.0:144978 2014-06-12 08:44:57.340119
2014-06-12 14:45:54.968337 7f9d425f5700  5 -- op tracker -- , seq: 747300, time: 2014-06-12 14:45:54.968286, event: all_read, op: osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.968347 7f9d425f5700  5 -- op tracker -- , seq: 747300, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.968346 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] append_log  adding 1 keys
2014-06-12 14:45:54.968354 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:45:54.968378 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2711, trimmed: 
2014-06-12 14:45:54.968386 7f9d425f5700 15 osd.2 173 enqueue_op 0x5ee8200 prio 63 cost 516096 latency 0.001604 osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.968400 7f9d63bd8700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.968404 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.968406 7f9d63bd8700  5 queue_op 0xcc97320 seq 2456 osr(3.54b 0x276db30) 174325 bytes   (queue has 4 ops and 1557495 bytes)
2014-06-12 14:45:54.968417 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:45:54.968413 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] eval_repop repgather(0x2335cd80 173'2711 rep_tid=236136 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.968431 7f9d63bd8700 10 osd.2 173 dequeue_op 0x5ee9600 finish
2014-06-12 14:45:54.968423 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x5ee8200 prio 63 cost 516096 latency 0.001642 osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean]
2014-06-12 14:45:54.968436 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:45:54.968439 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.968444 7f9d6a3e5700  5 -- op tracker -- , seq: 747300, time: 2014-06-12 14:45:54.968439, event: reached_pg, op: osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.968452 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:54.968465 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] handle_message: 0x5ee8200
2014-06-12 14:45:54.968475 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] do_op osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:54.968495 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] get_object_context: found obc in cache: 0x1be4d080
2014-06-12 14:45:54.968506 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] get_object_context: 0x1be4d080 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 rwstate(write n=1 w=0) oi: 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3321 client.4314.0:144979 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3321) ssc: 0x72b9420 snapset: 0=[]:[]+head
2014-06-12 14:45:54.968521 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] find_object_context 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 @head oi=61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3321 client.4314.0:144979 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3321)
2014-06-12 14:45:54.968542 7f9d6a3e5700  5 -- op tracker -- , seq: 747300, time: 2014-06-12 14:45:54.968537, event: started, op: osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.968548 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] execute_ctx 0x2a209000
2014-06-12 14:45:54.968564 7f9d6a3e5700  5 -- op tracker -- , seq: 747300, time: 2014-06-12 14:45:54.968559, event: started, op: osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.968570 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] do_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] ov 173'3321 av 173'3322 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:54.968583 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] do_osd_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096]
2014-06-12 14:45:54.968593 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:54.968605 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] do_osd_op  write 344064~516096
2014-06-12 14:45:54.968617 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:54.968630 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 snapset=0x72b9458  snapc=0=[]
2014-06-12 14:45:54.968641 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:54.968651 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] finish_ctx 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 0x2a209000 op modify  
2014-06-12 14:45:54.968662 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean]  set mtime to 2014-06-12 08:44:57.342898
2014-06-12 14:45:54.968494 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (4 active)
2014-06-12 14:45:54.968683 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean]  final snapset 0=[]:[]+head in 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.968703 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean]  zeroing write result code 0
2014-06-12 14:45:54.968712 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] calc_trim_to 125'320 -> 125'321
2014-06-12 14:45:54.968722 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] new_repop rep_tid 236137 on osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.968736 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] issue_repop rep_tid 236137 o 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.968758 7f9d6a3e5700  5 -- op tracker -- , seq: 747300, time: 2014-06-12 14:45:54.968752, event: waiting for subops from 20, op: osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.968778 7f9d6a3e5700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:54.968797 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3321 (125'320,173'3321] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] append_log log((125'320,173'3321], crt=0'0) [173'3322 (173'3321) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144980 2014-06-12 08:44:57.342898]
2014-06-12 14:45:54.968818 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'320,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] add_log_entry 173'3322 (173'3321) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144980 2014-06-12 08:44:57.342898
2014-06-12 14:45:54.968838 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'320,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] append_log  adding 1 keys
2014-06-12 14:45:54.968851 7f9d6a3e5700 10 trim log((125'320,173'3322], crt=0'0) to 125'321
2014-06-12 14:45:54.968874 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3322, trimmed: 125'321
2014-06-12 14:45:54.968897 7f9d6a3e5700  5 queue_transactions existing osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:54.968900 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.968902 7f9d6a3e5700  5 queue_op 0xdcb94a0 seq 2979 osr(3.395 0x276d740) 518424 bytes   (queue has 5 ops and 2075919 bytes)
2014-06-12 14:45:54.968908 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] eval_repop repgather(0x29702d80 173'3322 rep_tid=236137 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.968926 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x5ee8200 finish
2014-06-12 14:45:54.968931 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.968933 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.968942 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227e360 (5 active)
2014-06-12 14:45:54.969052 7f9d8096e700 10 _do_op 0x6872300 seq 2454 r = 0, finisher 0x90711c0 0x198e6bc0
2014-06-12 14:45:54.969064 7f9d8096e700 10 _finish_op 0x6872300 seq 2454 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.969069 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] op_commit: 236133
2014-06-12 14:45:54.969089 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (4 active)
2014-06-12 14:45:54.969092 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:54.969092 7f9d7715b700  5 -- op tracker -- , seq: 747296, time: 2014-06-12 14:45:54.969085, event: op_commit, op: osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.969106 7f9d83173700  5 _do_op 0x53c2a20 seq 2455 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:45:54.969111 7f9d83173700 10 _do_transaction on 0x534e000
2014-06-12 14:45:54.969105 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] op_applied: 236133
2014-06-12 14:45:54.969118 7f9d83173700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.969127 7f9d7795c700  5 -- op tracker -- , seq: 747296, time: 2014-06-12 14:45:54.969120, event: op_applied, op: osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.969136 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] op_applied on primary on version 173'2709
2014-06-12 14:45:54.969220 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.969239 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.969289 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:54.969469 7f9d83173700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.969476 7f9d83173700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:45:54.969483 7f9d83173700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.969498 7f9d83173700 15 _write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 3506176~516096
2014-06-12 14:45:54.969676 7f9d83173700 10 lookup_strip_header done  cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.969915 7f9d83173700 10 file_to_extents done 
2014-06-12 14:45:54.971936 7f9d83173700 10 _generic_write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 3506176~516096 = 0
2014-06-12 14:45:54.971973 7f9d83173700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.971982 7f9d83173700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.971991 7f9d83173700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.971996 7f9d83173700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.972134 7f9d83173700 20 set_header setting 17860 cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 parent seq 0
2014-06-12 14:45:54.972178 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:54.972187 7f9d83173700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:45:54.972192 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:45:54.973016 7f9d57543700  5 -- op tracker -- , seq: 747301, time: 2014-06-12 14:45:54.972966, event: header_read, op: osd_sub_op_reply(client.4314.0:144976 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.973027 7f9d57543700  5 -- op tracker -- , seq: 747301, time: 2014-06-12 14:45:54.972968, event: throttled, op: osd_sub_op_reply(client.4314.0:144976 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.973034 7f9d57543700  5 -- op tracker -- , seq: 747301, time: 2014-06-12 14:45:54.972983, event: all_read, op: osd_sub_op_reply(client.4314.0:144976 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.973041 7f9d57543700  5 -- op tracker -- , seq: 747301, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144976 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.973046 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144976 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:54.973052 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:54.973060 7f9d57543700 15 osd.2 173 enqueue_op 0x2911a00 prio 196 cost 0 latency 0.000093 osd_sub_op_reply(client.4314.0:144976 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:54.973093 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.973101 7f9d653db700 10 osd.2 173 dequeue_op 0x2911a00 prio 196 cost 0 latency 0.000134 osd_sub_op_reply(client.4314.0:144976 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]
2014-06-12 14:45:54.973120 7f9d653db700  5 -- op tracker -- , seq: 747301, time: 2014-06-12 14:45:54.973117, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144976 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.973128 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] handle_message: 0x2911a00
2014-06-12 14:45:54.973143 7f9d653db700  5 -- op tracker -- , seq: 747301, time: 2014-06-12 14:45:54.973140, event: started, op: osd_sub_op_reply(client.4314.0:144976 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.973148 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] sub_op_modify_reply: tid 236133 op  ack_type 4 from 23
2014-06-12 14:45:54.973169 7f9d653db700  5 -- op tracker -- , seq: 747296, time: 2014-06-12 14:45:54.973160, event: sub_op_commit_rec, op: osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.973175 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] repop_all_applied: repop tid 236133 all applied 
2014-06-12 14:45:54.973186 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] eval_repop repgather(0x18225e00 173'2709 rep_tid=236133 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.973201 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  sending ack on repgather(0x18225e00 173'2709 rep_tid=236133 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x1687bb80
2014-06-12 14:45:54.973233 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] repop_all_committed: repop tid 236133 all committed 
2014-06-12 14:45:54.973243 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2707 active+clean] eval_repop repgather(0x18225e00 173'2709 rep_tid=236133 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.973257 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2707 active+clean] log_op_stats osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.016585 lat 0.016611
2014-06-12 14:45:54.973273 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2707 active+clean] publish_stats_to_osd 173:3604
2014-06-12 14:45:54.973285 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2707 active+clean]  sending commit on repgather(0x18225e00 173'2709 rep_tid=236133 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x20237400
2014-06-12 14:45:54.973310 7f9d653db700  5 -- op tracker -- , seq: 747296, time: 2014-06-12 14:45:54.973306, event: commit_sent, op: osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.973317 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2707 active+clean]  requeue_ops 
2014-06-12 14:45:54.973327 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]  removing repgather(0x18225e00 173'2709 rep_tid=236133 committed?=1 applied?=1 op=osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.973340 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]    q front is repgather(0x18225e00 173'2709 rep_tid=236133 committed?=1 applied?=1 op=osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.973353 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] remove_repop repgather(0x18225e00 173'2709 rep_tid=236133 committed?=1 applied?=1 op=osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.973366 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]  requeue_ops 
2014-06-12 14:45:54.973383 7f9d653db700  5 -- op tracker -- , seq: 747296, time: 2014-06-12 14:45:54.973379, event: done, op: osd_op(client.4314.0:144976 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.973400 7f9d653db700 10 osd.2 173 dequeue_op 0x2911a00 finish
2014-06-12 14:45:54.973407 7f9d653db700  5 -- op tracker -- , seq: 747301, time: 2014-06-12 14:45:54.973404, event: done, op: osd_sub_op_reply(client.4314.0:144976 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.973415 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.973418 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.974577 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:54.974608 7f9d7915f700 15 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:54.974617 7f9d7915f700 10 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1 r = 0
2014-06-12 14:45:54.974623 7f9d7915f700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:54.974641 7f9d83173700 10 _do_op 0x53c2a20 seq 2455 r = 0, finisher 0x2ef7cdc0 0x1a3f0500
2014-06-12 14:45:54.974659 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] op_commit: 236134
2014-06-12 14:45:54.974686 7f9d7715b700  5 -- op tracker -- , seq: 747297, time: 2014-06-12 14:45:54.974677, event: op_commit, op: osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.974699 7f9d83173700 10 _finish_op 0x53c2a20 seq 2455 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.974712 7f9d7c165700  5 _do_op 0xcc97320 seq 2456 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:45:54.974716 7f9d7c165700 10 _do_transaction on 0x51b2180
2014-06-12 14:45:54.974721 7f9d7c165700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.974794 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.974808 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.974849 7f9d7915f700 15 _write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 0~344064
2014-06-12 14:45:54.974864 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] op_applied: 236134
2014-06-12 14:45:54.974886 7f9d7795c700  5 -- op tracker -- , seq: 747297, time: 2014-06-12 14:45:54.974879, event: op_applied, op: osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.974893 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] op_applied on primary on version 173'2710
2014-06-12 14:45:54.974923 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (3 active)
2014-06-12 14:45:54.974931 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:54.974979 7f9d7915f700 10 lookup_strip_header done  cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.975146 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:45:54.975967 7f9d7915f700 10 _generic_write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 0~344064 = 0
2014-06-12 14:45:54.975987 7f9d7915f700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.975997 7f9d7915f700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:54.976000 7f9d7915f700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.976005 7f9d7915f700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:54.976114 7f9d7915f700 20 set_header setting 17862 cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 parent seq 0
2014-06-12 14:45:54.976151 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:54.976159 7f9d7915f700 20 set_header setting 2060 cid meta oid 1fe1ea2a/pglog_3.395/0//-1 parent seq 0
2014-06-12 14:45:54.976165 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:45:54.977296 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:54.977298 7f9d7915f700 10 _do_op 0x4063bc0 seq 2978 r = 0, finisher 0xabf2ae0 0x3b72a80
2014-06-12 14:45:54.977310 7f9d7915f700 10 _finish_op 0x4063bc0 seq 2978 osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:54.977322 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227e360 (2 active)
2014-06-12 14:45:54.977330 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:54.977395 7f9d7c165700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.977403 7f9d7c165700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:45:54.977411 7f9d7c165700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:45:54.977430 7f9d7c165700 15 _write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 4022272~172032
2014-06-12 14:45:54.977594 7f9d7c165700 10 lookup_strip_header done  cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.977634 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] op_commit: 236135
2014-06-12 14:45:54.977669 7f9d7715b700  5 -- op tracker -- , seq: 747299, time: 2014-06-12 14:45:54.977657, event: op_commit, op: osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.977690 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] op_applied: 236135
2014-06-12 14:45:54.977712 7f9d7795c700  5 -- op tracker -- , seq: 747299, time: 2014-06-12 14:45:54.977706, event: op_applied, op: osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.977720 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] op_applied on primary on version 173'3321
2014-06-12 14:45:54.977747 7f9d5703e700  5 -- op tracker -- , seq: 747302, time: 2014-06-12 14:45:54.977711, event: header_read, op: osd_sub_op_reply(client.4314.0:144979 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.977758 7f9d5703e700  5 -- op tracker -- , seq: 747302, time: 2014-06-12 14:45:54.977712, event: throttled, op: osd_sub_op_reply(client.4314.0:144979 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.977767 7f9d5703e700  5 -- op tracker -- , seq: 747302, time: 2014-06-12 14:45:54.977721, event: all_read, op: osd_sub_op_reply(client.4314.0:144979 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.977775 7f9d5703e700  5 -- op tracker -- , seq: 747302, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144979 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.977779 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144979 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:54.977785 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:54.977793 7f9d5703e700 15 osd.2 173 enqueue_op 0x31d61700 prio 196 cost 0 latency 0.000081 osd_sub_op_reply(client.4314.0:144979 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:54.977817 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.977823 7f9d69be4700 10 osd.2 173 dequeue_op 0x31d61700 prio 196 cost 0 latency 0.000112 osd_sub_op_reply(client.4314.0:144979 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 pg pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean]
2014-06-12 14:45:54.977840 7f9d69be4700  5 -- op tracker -- , seq: 747302, time: 2014-06-12 14:45:54.977837, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144979 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.977857 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] handle_message: 0x31d61700
2014-06-12 14:45:54.977870 7f9d69be4700  5 -- op tracker -- , seq: 747302, time: 2014-06-12 14:45:54.977867, event: started, op: osd_sub_op_reply(client.4314.0:144979 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.977875 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] sub_op_modify_reply: tid 236135 op  ack_type 4 from 20
2014-06-12 14:45:54.977888 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:45:54.977891 7f9d69be4700  5 -- op tracker -- , seq: 747299, time: 2014-06-12 14:45:54.977886, event: sub_op_commit_rec, op: osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.977898 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] repop_all_applied: repop tid 236135 all applied 
2014-06-12 14:45:54.977908 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] eval_repop repgather(0x2562180 173'3321 rep_tid=236135 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.977923 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean]  sending ack on repgather(0x2562180 173'3321 rep_tid=236135 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) 0x1687aa00
2014-06-12 14:45:54.977952 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3320 crt=0'0 lcod 173'3319 mlcod 173'3319 active+clean] repop_all_committed: repop tid 236135 all committed 
2014-06-12 14:45:54.977962 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3319 active+clean] eval_repop repgather(0x2562180 173'3321 rep_tid=236135 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.977977 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3319 active+clean] log_op_stats osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 344343 outb 0 rlat 0.017611 lat 0.017636
2014-06-12 14:45:54.977993 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3319 active+clean] publish_stats_to_osd 173:4281
2014-06-12 14:45:54.978006 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3319 active+clean]  sending commit on repgather(0x2562180 173'3321 rep_tid=236135 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) 0x24fe780
2014-06-12 14:45:54.978031 7f9d69be4700  5 -- op tracker -- , seq: 747299, time: 2014-06-12 14:45:54.978027, event: commit_sent, op: osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.978038 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3319 active+clean]  requeue_ops 
2014-06-12 14:45:54.978048 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3320 active+clean]  removing repgather(0x2562180 173'3321 rep_tid=236135 committed?=1 applied?=1 op=osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.978062 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3320 active+clean]    q front is repgather(0x2562180 173'3321 rep_tid=236135 committed?=1 applied?=1 op=osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.978076 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3320 active+clean] remove_repop repgather(0x2562180 173'3321 rep_tid=236135 committed?=1 applied?=1 op=osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.978089 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3320 active+clean]  requeue_ops 
2014-06-12 14:45:54.978110 7f9d69be4700  5 -- op tracker -- , seq: 747299, time: 2014-06-12 14:45:54.978105, event: done, op: osd_op(client.4314.0:144979 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.978128 7f9d69be4700 10 osd.2 173 dequeue_op 0x31d61700 finish
2014-06-12 14:45:54.978134 7f9d69be4700  5 -- op tracker -- , seq: 747302, time: 2014-06-12 14:45:54.978131, event: done, op: osd_sub_op_reply(client.4314.0:144979 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.978144 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.978147 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.978493 7f9d7c165700 10 _generic_write 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 4022272~172032 = 0
2014-06-12 14:45:54.978517 7f9d7c165700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.978530 7f9d7c165700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.978536 7f9d7c165700 15 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3
2014-06-12 14:45:54.978545 7f9d7c165700 10 _setattrs 3.54b_head/89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 = 0
2014-06-12 14:45:54.978690 7f9d7c165700 20 set_header setting 17860 cid 3.54b_head oid 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 parent seq 0
2014-06-12 14:45:54.978722 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:54.978735 7f9d7c165700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:45:54.978745 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:45:54.978958 7f9d7a161700  5 _do_op 0xdcb94a0 seq 2979 osr(3.395 0x276d740)/0x276d740 start
2014-06-12 14:45:54.978968 7f9d7a161700 10 _do_transaction on 0x29702780
2014-06-12 14:45:54.978976 7f9d7a161700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:54.979097 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:54.979115 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.979636 7f9d7c165700 10 _do_op 0xcc97320 seq 2456 r = 0, finisher 0x5cca300 0x30d64c0
2014-06-12 14:45:54.979649 7f9d7c165700 10 _finish_op 0xcc97320 seq 2456 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:45:54.979665 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (1 active)
2014-06-12 14:45:54.979669 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:54.979680 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] op_commit: 236136
2014-06-12 14:45:54.979708 7f9d7715b700  5 -- op tracker -- , seq: 747298, time: 2014-06-12 14:45:54.979699, event: op_commit, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.979726 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] op_applied: 236136
2014-06-12 14:45:54.979745 7f9d7795c700  5 -- op tracker -- , seq: 747298, time: 2014-06-12 14:45:54.979740, event: op_applied, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.979753 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] op_applied on primary on version 173'2711
2014-06-12 14:45:54.979831 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:54.979858 7f9d7a161700 15 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:54.979867 7f9d7a161700 10 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1 r = 0
2014-06-12 14:45:54.979874 7f9d7a161700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:54.980014 7f9d7a161700 15 _write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 344064~516096
2014-06-12 14:45:54.980220 7f9d7a161700 10 lookup_strip_header done  cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.980379 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:45:54.982447 7f9d57543700  5 -- op tracker -- , seq: 747303, time: 2014-06-12 14:45:54.982388, event: header_read, op: osd_sub_op_reply(client.4314.0:144977 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.982462 7f9d57543700  5 -- op tracker -- , seq: 747303, time: 2014-06-12 14:45:54.982390, event: throttled, op: osd_sub_op_reply(client.4314.0:144977 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.982469 7f9d57543700  5 -- op tracker -- , seq: 747303, time: 2014-06-12 14:45:54.982405, event: all_read, op: osd_sub_op_reply(client.4314.0:144977 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.982477 7f9d57543700  5 -- op tracker -- , seq: 747303, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144977 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.982483 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144977 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:54.982489 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:54.982499 7f9d57543700 15 osd.2 173 enqueue_op 0x31d61d00 prio 196 cost 0 latency 0.000110 osd_sub_op_reply(client.4314.0:144977 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:54.982547 7f9d57543700  5 -- op tracker -- , seq: 747304, time: 2014-06-12 14:45:54.982523, event: header_read, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.982556 7f9d57543700  5 -- op tracker -- , seq: 747304, time: 2014-06-12 14:45:54.982523, event: throttled, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.982563 7f9d57543700  5 -- op tracker -- , seq: 747304, time: 2014-06-12 14:45:54.982530, event: all_read, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.982570 7f9d57543700  5 -- op tracker -- , seq: 747304, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.982567 7f9d7a161700 10 _generic_write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 344064~516096 = 0
2014-06-12 14:45:54.982573 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:45:54.982577 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:54.982582 7f9d57543700 15 osd.2 173 enqueue_op 0x25e42100 prio 196 cost 0 latency 0.000059 osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0) v2
2014-06-12 14:45:54.982618 7f9d7a161700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.982628 7f9d57543700  5 -- op tracker -- , seq: 747305, time: 2014-06-12 14:45:54.982598, event: header_read, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.982635 7f9d7a161700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:54.982637 7f9d57543700  5 -- op tracker -- , seq: 747305, time: 2014-06-12 14:45:54.982599, event: throttled, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.982642 7f9d7a161700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.982644 7f9d57543700  5 -- op tracker -- , seq: 747305, time: 2014-06-12 14:45:54.982610, event: all_read, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.982650 7f9d7a161700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:54.982651 7f9d57543700  5 -- op tracker -- , seq: 747305, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.982655 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:54.982660 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:54.982665 7f9d57543700 15 osd.2 173 enqueue_op 0x8dbc500 prio 196 cost 0 latency 0.000066 osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:54.982691 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.982700 7f9d633d7700 10 osd.2 173 dequeue_op 0x31d61d00 prio 196 cost 0 latency 0.000311 osd_sub_op_reply(client.4314.0:144977 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]
2014-06-12 14:45:54.982721 7f9d633d7700  5 -- op tracker -- , seq: 747303, time: 2014-06-12 14:45:54.982718, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144977 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.982730 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] handle_message: 0x31d61d00
2014-06-12 14:45:54.982745 7f9d633d7700  5 -- op tracker -- , seq: 747303, time: 2014-06-12 14:45:54.982742, event: started, op: osd_sub_op_reply(client.4314.0:144977 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.982751 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] sub_op_modify_reply: tid 236134 op  ack_type 4 from 23
2014-06-12 14:45:54.982772 7f9d633d7700  5 -- op tracker -- , seq: 747297, time: 2014-06-12 14:45:54.982763, event: sub_op_commit_rec, op: osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.982779 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] repop_all_applied: repop tid 236134 all applied 
2014-06-12 14:45:54.982790 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] eval_repop repgather(0x534ef00 173'2710 rep_tid=236134 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.982805 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]  sending ack on repgather(0x534ef00 173'2710 rep_tid=236134 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x6701680
2014-06-12 14:45:54.982837 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] repop_all_committed: repop tid 236134 all committed 
2014-06-12 14:45:54.982848 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2708 active+clean] eval_repop repgather(0x534ef00 173'2710 rep_tid=236134 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.982864 7f9d7a161700 20 set_header setting 17862 cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 parent seq 0
2014-06-12 14:45:54.982864 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2708 active+clean] log_op_stats osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.023758 lat 0.023786
2014-06-12 14:45:54.982882 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2708 active+clean] publish_stats_to_osd 173:3605
2014-06-12 14:45:54.982894 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2708 active+clean]  sending commit on repgather(0x534ef00 173'2710 rep_tid=236134 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x2b42500
2014-06-12 14:45:54.982921 7f9d633d7700  5 -- op tracker -- , seq: 747297, time: 2014-06-12 14:45:54.982917, event: commit_sent, op: osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.982929 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2708 active+clean]  requeue_ops 
2014-06-12 14:45:54.982938 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:54.982938 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean]  removing repgather(0x534ef00 173'2710 rep_tid=236134 committed?=1 applied?=1 op=osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.982953 7f9d7a161700 20 set_header setting 2060 cid meta oid 1fe1ea2a/pglog_3.395/0//-1 parent seq 0
2014-06-12 14:45:54.982952 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean]    q front is repgather(0x534ef00 173'2710 rep_tid=236134 committed?=1 applied?=1 op=osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.982963 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:45:54.982964 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] remove_repop repgather(0x534ef00 173'2710 rep_tid=236134 committed?=1 applied?=1 op=osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.982976 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean]  requeue_ops 
2014-06-12 14:45:54.982993 7f9d633d7700  5 -- op tracker -- , seq: 747297, time: 2014-06-12 14:45:54.982989, event: done, op: osd_op(client.4314.0:144977 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.983011 7f9d633d7700 10 osd.2 173 dequeue_op 0x31d61d00 finish
2014-06-12 14:45:54.983017 7f9d633d7700  5 -- op tracker -- , seq: 747303, time: 2014-06-12 14:45:54.983014, event: done, op: osd_sub_op_reply(client.4314.0:144977 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.983026 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.983031 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.983034 7f9d633d7700 10 osd.2 173 dequeue_op 0x25e42100 prio 196 cost 0 latency 0.000511 osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0) v2 pg pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean]
2014-06-12 14:45:54.983047 7f9d633d7700  5 -- op tracker -- , seq: 747304, time: 2014-06-12 14:45:54.983045, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.983052 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] handle_message: 0x25e42100
2014-06-12 14:45:54.983064 7f9d633d7700  5 -- op tracker -- , seq: 747304, time: 2014-06-12 14:45:54.983061, event: started, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.983069 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] sub_op_modify_reply: tid 236136 op  ack_type 1 from 23
2014-06-12 14:45:54.983084 7f9d633d7700  5 -- op tracker -- , seq: 747298, time: 2014-06-12 14:45:54.983079, event: sub_op_applied_rec, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.983090 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] repop_all_applied: repop tid 236136 all applied 
2014-06-12 14:45:54.983098 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] eval_repop repgather(0x2335cd80 173'2711 rep_tid=236136 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.983111 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean]  sending ack on repgather(0x2335cd80 173'2711 rep_tid=236136 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x348b7680
2014-06-12 14:45:54.983131 7f9d633d7700 10 osd.2 173 dequeue_op 0x25e42100 finish
2014-06-12 14:45:54.983137 7f9d633d7700  5 -- op tracker -- , seq: 747304, time: 2014-06-12 14:45:54.983134, event: done, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ack, result = 0)
2014-06-12 14:45:54.983146 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.983151 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.983153 7f9d633d7700 10 osd.2 173 dequeue_op 0x8dbc500 prio 196 cost 0 latency 0.000555 osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean]
2014-06-12 14:45:54.983167 7f9d633d7700  5 -- op tracker -- , seq: 747305, time: 2014-06-12 14:45:54.983164, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.983172 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] handle_message: 0x8dbc500
2014-06-12 14:45:54.983185 7f9d633d7700  5 -- op tracker -- , seq: 747305, time: 2014-06-12 14:45:54.983182, event: started, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.983190 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] sub_op_modify_reply: tid 236136 op  ack_type 4 from 23
2014-06-12 14:45:54.983203 7f9d633d7700  5 -- op tracker -- , seq: 747298, time: 2014-06-12 14:45:54.983199, event: sub_op_commit_rec, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.983208 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2710 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] repop_all_committed: repop tid 236136 all committed 
2014-06-12 14:45:54.983216 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2709 active+clean] eval_repop repgather(0x2335cd80 173'2711 rep_tid=236136 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.983229 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2709 active+clean] log_op_stats osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4 inb 172311 outb 0 rlat 0.023318 lat 0.023416
2014-06-12 14:45:54.983241 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2709 active+clean] publish_stats_to_osd 173:3606
2014-06-12 14:45:54.983246 7f9d5501e700  5 -- op tracker -- , seq: 747306, time: 2014-06-12 14:45:54.946033, event: header_read, op: osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.983258 7f9d5501e700  5 -- op tracker -- , seq: 747306, time: 2014-06-12 14:45:54.946034, event: throttled, op: osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.983250 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2709 active+clean]  sending commit on repgather(0x2335cd80 173'2711 rep_tid=236136 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4) 0x6486f00
2014-06-12 14:45:54.983267 7f9d5501e700  5 -- op tracker -- , seq: 747306, time: 2014-06-12 14:45:54.983218, event: all_read, op: osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.983276 7f9d633d7700  5 -- op tracker -- , seq: 747298, time: 2014-06-12 14:45:54.983272, event: commit_sent, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.983286 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2709 active+clean]  requeue_ops 
2014-06-12 14:45:54.983296 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]  removing repgather(0x2335cd80 173'2711 rep_tid=236136 committed?=1 applied?=1 op=osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.983314 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]    q front is repgather(0x2335cd80 173'2711 rep_tid=236136 committed?=1 applied?=1 op=osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.983326 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] remove_repop repgather(0x2335cd80 173'2711 rep_tid=236136 committed?=1 applied?=1 op=osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.983348 7f9d5501e700  5 -- op tracker -- , seq: 747306, time: 0.000000, event: dispatched, op: osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.983344 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]  requeue_ops 
2014-06-12 14:45:54.983355 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:54.983362 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:45:54.983366 7f9d633d7700  5 -- op tracker -- , seq: 747298, time: 2014-06-12 14:45:54.983362, event: done, op: osd_op(client.4314.0:144978 rbd_data.10b16b8b4567.000000000000104a [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.89d0ed4b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.983369 7f9d5501e700 15 osd.2 173 enqueue_op 0x2519d300 prio 127 cost 1032999 latency 0.037336 osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:54.983398 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:45:54.983404 7f9d623d5700 10 osd.2 173 dequeue_op 0x2519d300 prio 127 cost 1032999 latency 0.037370 osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.169( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active]
2014-06-12 14:45:54.983426 7f9d623d5700  5 -- op tracker -- , seq: 747306, time: 2014-06-12 14:45:54.983421, event: reached_pg, op: osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.983436 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active] handle_message: 0x2519d300
2014-06-12 14:45:54.983448 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active] sub_op_modify trans e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 v 173'2863 (transaction) 174
2014-06-12 14:45:54.983463 7f9d633d7700 10 osd.2 173 dequeue_op 0x8dbc500 finish
2014-06-12 14:45:54.983469 7f9d623d5700  5 -- op tracker -- , seq: 747306, time: 2014-06-12 14:45:54.983465, event: started, op: osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.983488 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active] append_log log((0'0,173'2862], crt=0'0) [173'2863 (173'2862) modify   e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 by client.4296.0:143498 2014-06-12 08:44:57.315699]
2014-06-12 14:45:54.983513 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active] add_log_entry 173'2863 (173'2862) modify   e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 by client.4296.0:143498 2014-06-12 08:44:57.315699
2014-06-12 14:45:54.983535 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active] append_log  adding 1 keys
2014-06-12 14:45:54.983570 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2863, trimmed: 
2014-06-12 14:45:54.983596 7f9d623d5700  5 -- op tracker -- , seq: 747306, time: 2014-06-12 14:45:54.983591, event: started, op: osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:54.983607 7f9d623d5700  5 queue_transactions existing osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:45:54.983611 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.983613 7f9d623d5700  5 queue_op 0x15168e40 seq 2601 osr(3.169 0x276ce50) 1034485 bytes   (queue has 2 ops and 1552909 bytes)
2014-06-12 14:45:54.983621 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.169( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active] do_sub_op osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:54.983636 7f9d623d5700 10 osd.2 173 dequeue_op 0x2519d300 finish
2014-06-12 14:45:54.983640 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:45:54.983647 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.983657 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54f1200 (2 active)
2014-06-12 14:45:54.983662 7f9d82972700  5 _do_op 0x15168e40 seq 2601 osr(3.169 0x276ce50)/0x276ce50 start
2014-06-12 14:45:54.983666 7f9d82972700 10 _do_transaction on 0x6319b30
2014-06-12 14:45:54.983672 7f9d82972700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:54.983808 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:54.983831 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.983894 7f9d633d7700  5 -- op tracker -- , seq: 747305, time: 2014-06-12 14:45:54.983467, event: done, op: osd_sub_op_reply(client.4314.0:144978 3.54b 89d0ed4b/rbd_data.10b16b8b4567.000000000000104a/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.983910 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.983913 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.983926 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.983956 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.984374 7f9d5703e700  5 -- op tracker -- , seq: 747307, time: 2014-06-12 14:45:54.984334, event: header_read, op: osd_sub_op_reply(client.4314.0:144980 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.984384 7f9d5703e700  5 -- op tracker -- , seq: 747307, time: 2014-06-12 14:45:54.984336, event: throttled, op: osd_sub_op_reply(client.4314.0:144980 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.984391 7f9d5703e700  5 -- op tracker -- , seq: 747307, time: 2014-06-12 14:45:54.984349, event: all_read, op: osd_sub_op_reply(client.4314.0:144980 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.984399 7f9d5703e700  5 -- op tracker -- , seq: 747307, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144980 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.984404 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144980 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:54.984409 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:54.984417 7f9d5703e700 15 osd.2 173 enqueue_op 0x31d60e00 prio 196 cost 0 latency 0.000082 osd_sub_op_reply(client.4314.0:144980 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:54.984443 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.984450 7f9d61bd4700 10 osd.2 173 dequeue_op 0x31d60e00 prio 196 cost 0 latency 0.000116 osd_sub_op_reply(client.4314.0:144980 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 pg pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3320 active+clean]
2014-06-12 14:45:54.984470 7f9d61bd4700  5 -- op tracker -- , seq: 747307, time: 2014-06-12 14:45:54.984467, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144980 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.984480 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3320 active+clean] handle_message: 0x31d60e00
2014-06-12 14:45:54.984494 7f9d61bd4700  5 -- op tracker -- , seq: 747307, time: 2014-06-12 14:45:54.984491, event: started, op: osd_sub_op_reply(client.4314.0:144980 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.984500 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3320 active+clean] sub_op_modify_reply: tid 236137 op  ack_type 4 from 20
2014-06-12 14:45:54.984521 7f9d61bd4700  5 -- op tracker -- , seq: 747300, time: 2014-06-12 14:45:54.984513, event: sub_op_commit_rec, op: osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.984528 7f9d61bd4700 10 osd.2 173 dequeue_op 0x31d60e00 finish
2014-06-12 14:45:54.984534 7f9d61bd4700  5 -- op tracker -- , seq: 747307, time: 2014-06-12 14:45:54.984531, event: done, op: osd_sub_op_reply(client.4314.0:144980 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:54.984548 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.984550 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.986323 7f9d7a161700 10 _do_op 0xdcb94a0 seq 2979 r = 0, finisher 0x7f57060 0x2fe5ee80
2014-06-12 14:45:54.987061 7f9d425f5700  5 -- op tracker -- , seq: 747308, time: 2014-06-12 14:45:54.986359, event: header_read, op: osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.987074 7f9d425f5700  5 -- op tracker -- , seq: 747308, time: 2014-06-12 14:45:54.986362, event: throttled, op: osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.987083 7f9d425f5700  5 -- op tracker -- , seq: 747308, time: 2014-06-12 14:45:54.987019, event: all_read, op: osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.987092 7f9d425f5700  5 -- op tracker -- , seq: 747308, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.987101 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:45:54.987146 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:54.987153 7f9d425f5700 15 osd.2 173 enqueue_op 0x5ee8600 prio 63 cost 516096 latency 0.000793 osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.987179 7f9d82972700 15 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:54.987185 7f9d82972700 10 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1 r = 0
2014-06-12 14:45:54.987192 7f9d82972700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:54.987211 7f9d82972700 15 _write 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 1007616~1032192
2014-06-12 14:45:54.987201 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3320 active+clean] op_commit: 236137
2014-06-12 14:45:54.987236 7f9d7715b700  5 -- op tracker -- , seq: 747300, time: 2014-06-12 14:45:54.987221, event: op_commit, op: osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.987245 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3321 crt=0'0 lcod 173'3320 mlcod 173'3320 active+clean] repop_all_committed: repop tid 236137 all committed 
2014-06-12 14:45:54.987254 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3320 active+clean] eval_repop repgather(0x29702d80 173'3322 rep_tid=236137 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.987270 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3320 active+clean] log_op_stats osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.020488
2014-06-12 14:45:54.987296 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3320 active+clean] publish_stats_to_osd 173:4282
2014-06-12 14:45:54.987307 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3320 active+clean]  sending commit on repgather(0x29702d80 173'3322 rep_tid=236137 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) 0x38136f00
2014-06-12 14:45:54.987348 7f9d7715b700  5 -- op tracker -- , seq: 747300, time: 2014-06-12 14:45:54.987342, event: commit_sent, op: osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.987362 7f9d7a161700 10 _finish_op 0xdcb94a0 seq 2979 osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:54.987364 7f9d82972700 10 lookup_strip_header done  cid 3.169_head oid e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:54.987378 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3320 active+clean] op_applied: 236137
2014-06-12 14:45:54.987406 7f9d7795c700  5 -- op tracker -- , seq: 747300, time: 2014-06-12 14:45:54.987391, event: op_applied, op: osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.987414 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3320 active+clean] op_applied on primary on version 173'3322
2014-06-12 14:45:54.987423 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3320 active+clean] repop_all_applied: repop tid 236137 all applied 
2014-06-12 14:45:54.987431 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3320 active+clean] eval_repop repgather(0x29702d80 173'3322 rep_tid=236137 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.987445 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3320 active+clean] log_op_stats osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.020664
2014-06-12 14:45:54.987463 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3320 active+clean] publish_stats_to_osd 173:4283
2014-06-12 14:45:54.987474 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3320 active+clean]  requeue_ops 
2014-06-12 14:45:54.987484 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  removing repgather(0x29702d80 173'3322 rep_tid=236137 committed?=1 applied?=1 op=osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.987497 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]    q front is repgather(0x29702d80 173'3322 rep_tid=236137 committed?=1 applied?=1 op=osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.987510 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] remove_repop repgather(0x29702d80 173'3322 rep_tid=236137 committed?=1 applied?=1 op=osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:54.987522 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  requeue_ops 
2014-06-12 14:45:54.987608 7f9d82972700 10 file_to_extents done 
2014-06-12 14:45:54.987583 7f9d7a161700  5 -- op tracker -- , seq: 747300, time: 2014-06-12 14:45:54.987569, event: done, op: osd_op(client.4314.0:144980 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 344064~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.987628 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227e360 (1 active)
2014-06-12 14:45:54.987631 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:54.987650 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.987656 7f9d673df700 10 osd.2 173 dequeue_op 0x5ee8600 prio 63 cost 516096 latency 0.001297 osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]
2014-06-12 14:45:54.987693 7f9d673df700  5 -- op tracker -- , seq: 747308, time: 2014-06-12 14:45:54.987688, event: reached_pg, op: osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.987702 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:54.987715 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] handle_message: 0x5ee8600
2014-06-12 14:45:54.987726 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_op osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:54.987754 7f9d673df700 15 getattr 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 '_'
2014-06-12 14:45:54.987934 7f9d673df700 10 getattr 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 '_' = 0
2014-06-12 14:45:54.987950 7f9d673df700 15 getattr 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 'snapset'
2014-06-12 14:45:54.988041 7f9d673df700 10 getattr 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 'snapset' = 0
2014-06-12 14:45:54.988050 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] populate_obc_watchers 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.988062 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.988072 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] get_object_context: creating obc from disk: 0x29f75340
2014-06-12 14:45:54.988081 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] get_object_context: 0x29f75340 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 rwstate(none n=0 w=0) oi: 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3322 client.4314.0:144980 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3322) ssc: 0x72b88c0 snapset: 0=[]:[]+head
2014-06-12 14:45:54.988097 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] find_object_context 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 @head oi=61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3322 client.4314.0:144980 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3322)
2014-06-12 14:45:54.988121 7f9d673df700  5 -- op tracker -- , seq: 747308, time: 2014-06-12 14:45:54.988116, event: started, op: osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.988129 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] execute_ctx 0x2d2a6600
2014-06-12 14:45:54.988146 7f9d673df700  5 -- op tracker -- , seq: 747308, time: 2014-06-12 14:45:54.988142, event: started, op: osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.988153 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] ov 173'3322 av 173'3323 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:54.988167 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096]
2014-06-12 14:45:54.988178 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:54.988191 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op  write 860160~516096
2014-06-12 14:45:54.988211 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:54.988222 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 snapset=0x72b88f8  snapc=0=[]
2014-06-12 14:45:54.988234 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:54.988244 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] finish_ctx 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 0x2d2a6600 op modify  
2014-06-12 14:45:54.988256 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  set mtime to 2014-06-12 08:44:57.360481
2014-06-12 14:45:54.988276 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  final snapset 0=[]:[]+head in 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.988295 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  zeroing write result code 0
2014-06-12 14:45:54.988305 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] calc_trim_to 125'321 -> 125'322
2014-06-12 14:45:54.988314 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] new_repop rep_tid 236138 on osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.988330 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] issue_repop rep_tid 236138 o 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.988353 7f9d673df700  5 -- op tracker -- , seq: 747308, time: 2014-06-12 14:45:54.988348, event: waiting for subops from 20, op: osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.988373 7f9d673df700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:54.988393 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3322 (125'321,173'3322] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] append_log log((125'321,173'3322], crt=0'0) [173'3323 (173'3322) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144981 2014-06-12 08:44:57.360481]
2014-06-12 14:45:54.988416 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'321,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] add_log_entry 173'3323 (173'3322) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144981 2014-06-12 08:44:57.360481
2014-06-12 14:45:54.988436 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'321,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] append_log  adding 1 keys
2014-06-12 14:45:54.988449 7f9d673df700 10 trim log((125'321,173'3323], crt=0'0) to 125'322
2014-06-12 14:45:54.988476 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3323, trimmed: 125'322
2014-06-12 14:45:54.988499 7f9d673df700  5 queue_transactions existing osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:54.988503 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.988504 7f9d673df700  5 queue_op 0x72c42a0 seq 2980 osr(3.395 0x276d740) 518424 bytes   (queue has 2 ops and 1552909 bytes)
2014-06-12 14:45:54.988512 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] eval_repop repgather(0x15e56300 173'3323 rep_tid=236138 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.988530 7f9d673df700 10 osd.2 173 dequeue_op 0x5ee8600 finish
2014-06-12 14:45:54.988534 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.988537 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.989677 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:54.989684 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:54.989686 7f9d74bfa700 20 osd.2 173 _dispatch 0x53936c0 ping v1
2014-06-12 14:45:54.989690 7f9d74bfa700 10 osd.2 173 ping from client.4292
2014-06-12 14:45:54.989694 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:54.989695 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:54.990875 7f9d425f5700  5 -- op tracker -- , seq: 747309, time: 2014-06-12 14:45:54.989029, event: header_read, op: osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.990891 7f9d425f5700  5 -- op tracker -- , seq: 747309, time: 2014-06-12 14:45:54.989031, event: throttled, op: osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.990900 7f9d425f5700  5 -- op tracker -- , seq: 747309, time: 2014-06-12 14:45:54.990819, event: all_read, op: osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.990911 7f9d425f5700  5 -- op tracker -- , seq: 747309, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.990927 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:45:54.990976 7f9d425f5700 15 osd.2 173 enqueue_op 0x5ee9500 prio 63 cost 1032192 latency 0.001947 osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.991045 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.991060 7f9d663dd700 10 osd.2 173 dequeue_op 0x5ee9500 prio 63 cost 1032192 latency 0.002030 osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]
2014-06-12 14:45:54.991094 7f9d663dd700  5 -- op tracker -- , seq: 747309, time: 2014-06-12 14:45:54.991088, event: reached_pg, op: osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.991103 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:54.991121 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] handle_message: 0x5ee9500
2014-06-12 14:45:54.991132 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_op osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:54.991157 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] get_object_context: found obc in cache: 0x29f75340
2014-06-12 14:45:54.991167 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] get_object_context: 0x29f75340 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 rwstate(write n=1 w=0) oi: 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3323 client.4314.0:144981 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3323) ssc: 0x72b88c0 snapset: 0=[]:[]+head
2014-06-12 14:45:54.991182 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] find_object_context 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 @head oi=61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3323 client.4314.0:144981 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3323)
2014-06-12 14:45:54.991208 7f9d663dd700  5 -- op tracker -- , seq: 747309, time: 2014-06-12 14:45:54.991203, event: started, op: osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.991216 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] execute_ctx 0x1474c400
2014-06-12 14:45:54.991233 7f9d663dd700  5 -- op tracker -- , seq: 747309, time: 2014-06-12 14:45:54.991228, event: started, op: osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.991240 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] ov 173'3323 av 173'3324 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:54.991253 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192]
2014-06-12 14:45:54.991263 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:54.991277 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op  write 1376256~1032192
2014-06-12 14:45:54.991292 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:54.991303 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 snapset=0x72b88f8  snapc=0=[]
2014-06-12 14:45:54.991315 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:54.991325 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] finish_ctx 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 0x1474c400 op modify  
2014-06-12 14:45:54.991338 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  set mtime to 2014-06-12 08:44:57.366647
2014-06-12 14:45:54.991359 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  final snapset 0=[]:[]+head in 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.991382 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  zeroing write result code 0
2014-06-12 14:45:54.991392 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] calc_trim_to 125'322 -> 125'323
2014-06-12 14:45:54.991401 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] new_repop rep_tid 236139 on osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.991416 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] issue_repop rep_tid 236139 o 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.991440 7f9d663dd700  5 -- op tracker -- , seq: 747309, time: 2014-06-12 14:45:54.991435, event: waiting for subops from 20, op: osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.991469 7f9d663dd700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:54.991494 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3323 (125'322,173'3323] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] append_log log((125'322,173'3323], crt=0'0) [173'3324 (173'3323) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144982 2014-06-12 08:44:57.366647]
2014-06-12 14:45:54.991516 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'322,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] add_log_entry 173'3324 (173'3323) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144982 2014-06-12 08:44:57.366647
2014-06-12 14:45:54.991534 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'322,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] append_log  adding 1 keys
2014-06-12 14:45:54.991547 7f9d663dd700 10 trim log((125'322,173'3324], crt=0'0) to 125'323
2014-06-12 14:45:54.991573 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3324, trimmed: 125'323
2014-06-12 14:45:54.991598 7f9d663dd700  5 queue_transactions existing osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:54.991602 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.991604 7f9d663dd700  5 queue_op 0x19ed6d20 seq 2981 osr(3.395 0x276d740) 1034520 bytes   (queue has 3 ops and 2587429 bytes)
2014-06-12 14:45:54.991611 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] eval_repop repgather(0x2dc1f380 173'3324 rep_tid=236139 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.991628 7f9d663dd700 10 osd.2 173 dequeue_op 0x5ee9500 finish
2014-06-12 14:45:54.991632 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.991635 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.991643 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227e360 (2 active)
2014-06-12 14:45:54.991648 7f9d81970700  5 _do_op 0x72c42a0 seq 2980 osr(3.395 0x276d740)/0x276d740 start
2014-06-12 14:45:54.991653 7f9d81970700 10 _do_transaction on 0x2dc1f080
2014-06-12 14:45:54.991660 7f9d81970700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:54.991793 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:54.991816 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:54.992025 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227e360 (3 active)
2014-06-12 14:45:54.998540 7f9d425f5700  5 -- op tracker -- , seq: 747310, time: 2014-06-12 14:45:54.994101, event: header_read, op: osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.998553 7f9d425f5700  5 -- op tracker -- , seq: 747310, time: 2014-06-12 14:45:54.994104, event: throttled, op: osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.998564 7f9d425f5700  5 -- op tracker -- , seq: 747310, time: 2014-06-12 14:45:54.994968, event: all_read, op: osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.998574 7f9d425f5700  5 -- op tracker -- , seq: 747310, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.998582 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:45:54.998609 7f9d425f5700 15 osd.2 173 enqueue_op 0x5ee8d00 prio 63 cost 516096 latency 0.004507 osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.998639 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:54.998645 7f9d67be0700 10 osd.2 173 dequeue_op 0x5ee8d00 prio 63 cost 516096 latency 0.004544 osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]
2014-06-12 14:45:54.998669 7f9d67be0700  5 -- op tracker -- , seq: 747310, time: 2014-06-12 14:45:54.998664, event: reached_pg, op: osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.998678 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:54.998693 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] handle_message: 0x5ee8d00
2014-06-12 14:45:54.998703 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_op osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:54.998728 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] get_object_context: found obc in cache: 0x29f75340
2014-06-12 14:45:54.998738 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] get_object_context: 0x29f75340 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 rwstate(write n=2 w=0) oi: 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3324 client.4314.0:144982 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3324) ssc: 0x72b88c0 snapset: 0=[]:[]+head
2014-06-12 14:45:54.998754 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] find_object_context 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 @head oi=61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3324 client.4314.0:144982 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3324)
2014-06-12 14:45:54.998778 7f9d67be0700  5 -- op tracker -- , seq: 747310, time: 2014-06-12 14:45:54.998773, event: started, op: osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.998785 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] execute_ctx 0x110c2c00
2014-06-12 14:45:54.998803 7f9d67be0700  5 -- op tracker -- , seq: 747310, time: 2014-06-12 14:45:54.998797, event: started, op: osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.998810 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] ov 173'3324 av 173'3325 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:54.998824 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096]
2014-06-12 14:45:54.998835 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:54.998848 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op  write 2408448~516096
2014-06-12 14:45:54.998864 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:54.998875 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 snapset=0x72b88f8  snapc=0=[]
2014-06-12 14:45:54.998887 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:54.998897 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] finish_ctx 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 0x110c2c00 op modify  
2014-06-12 14:45:54.998910 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  set mtime to 2014-06-12 08:44:57.372487
2014-06-12 14:45:54.998931 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  final snapset 0=[]:[]+head in 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.998955 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  zeroing write result code 0
2014-06-12 14:45:54.998966 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] calc_trim_to 125'323 -> 125'324
2014-06-12 14:45:54.998976 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] new_repop rep_tid 236140 on osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:54.998992 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] issue_repop rep_tid 236140 o 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:54.999016 7f9d67be0700  5 -- op tracker -- , seq: 747310, time: 2014-06-12 14:45:54.999011, event: waiting for subops from 20, op: osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:54.999041 7f9d67be0700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:54.999062 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3324 (125'323,173'3324] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] append_log log((125'323,173'3324], crt=0'0) [173'3325 (173'3324) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144983 2014-06-12 08:44:57.372487]
2014-06-12 14:45:54.999084 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'323,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] add_log_entry 173'3325 (173'3324) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144983 2014-06-12 08:44:57.372487
2014-06-12 14:45:54.999105 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'323,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] append_log  adding 1 keys
2014-06-12 14:45:54.999118 7f9d67be0700 10 trim log((125'323,173'3325], crt=0'0) to 125'324
2014-06-12 14:45:54.999143 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3325, trimmed: 125'324
2014-06-12 14:45:54.999166 7f9d67be0700  5 queue_transactions existing osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:54.999170 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:54.999171 7f9d67be0700  5 queue_op 0x1b41800 seq 2982 osr(3.395 0x276d740) 518424 bytes   (queue has 4 ops and 3105853 bytes)
2014-06-12 14:45:54.999179 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] eval_repop repgather(0x18224540 173'3325 rep_tid=236140 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:54.999197 7f9d67be0700 10 osd.2 173 dequeue_op 0x5ee8d00 finish
2014-06-12 14:45:54.999201 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:54.999204 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:45:54.999979 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227e360 (4 active)
2014-06-12 14:45:55.001110 7f9d425f5700  5 -- op tracker -- , seq: 747311, time: 2014-06-12 14:45:55.000003, event: header_read, op: osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.001122 7f9d425f5700  5 -- op tracker -- , seq: 747311, time: 2014-06-12 14:45:55.000005, event: throttled, op: osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.001132 7f9d425f5700  5 -- op tracker -- , seq: 747311, time: 2014-06-12 14:45:55.001061, event: all_read, op: osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.001142 7f9d425f5700  5 -- op tracker -- , seq: 747311, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.001150 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:45:55.001179 7f9d425f5700 15 osd.2 173 enqueue_op 0x5ee9700 prio 63 cost 516096 latency 0.001176 osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.001228 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.001235 7f9d693e3700 10 osd.2 173 dequeue_op 0x5ee9700 prio 63 cost 516096 latency 0.001232 osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]
2014-06-12 14:45:55.001261 7f9d693e3700  5 -- op tracker -- , seq: 747311, time: 2014-06-12 14:45:55.001255, event: reached_pg, op: osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.001270 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:55.001286 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] handle_message: 0x5ee9700
2014-06-12 14:45:55.001296 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_op osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:55.001324 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] get_object_context: found obc in cache: 0x29f75340
2014-06-12 14:45:55.001336 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] get_object_context: 0x29f75340 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 rwstate(write n=3 w=0) oi: 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3325 client.4314.0:144983 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3325) ssc: 0x72b88c0 snapset: 0=[]:[]+head
2014-06-12 14:45:55.001355 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] find_object_context 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 @head oi=61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3325 client.4314.0:144983 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3325)
2014-06-12 14:45:55.001387 7f9d693e3700  5 -- op tracker -- , seq: 747311, time: 2014-06-12 14:45:55.001381, event: started, op: osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.001395 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] execute_ctx 0x55a3800
2014-06-12 14:45:55.001414 7f9d693e3700  5 -- op tracker -- , seq: 747311, time: 2014-06-12 14:45:55.001408, event: started, op: osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.001420 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] ov 173'3325 av 173'3326 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:55.001435 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096]
2014-06-12 14:45:55.001446 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:55.001460 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op  write 2924544~516096
2014-06-12 14:45:55.001476 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:55.001487 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 snapset=0x72b88f8  snapc=0=[]
2014-06-12 14:45:55.001499 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:55.001509 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] finish_ctx 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 0x55a3800 op modify  
2014-06-12 14:45:55.001522 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  set mtime to 2014-06-12 08:44:57.377854
2014-06-12 14:45:55.001544 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  final snapset 0=[]:[]+head in 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.001565 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]  zeroing write result code 0
2014-06-12 14:45:55.001575 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] calc_trim_to 125'324 -> 125'325
2014-06-12 14:45:55.001585 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] new_repop rep_tid 236141 on osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.001601 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] issue_repop rep_tid 236141 o 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.001648 7f9d693e3700  5 -- op tracker -- , seq: 747311, time: 2014-06-12 14:45:55.001642, event: waiting for subops from 20, op: osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.001671 7f9d693e3700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:55.001695 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3325 (125'324,173'3325] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] append_log log((125'324,173'3325], crt=0'0) [173'3326 (173'3325) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144984 2014-06-12 08:44:57.377854]
2014-06-12 14:45:55.001720 7f9d82972700 10 _generic_write 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 1007616~1032192 = 0
2014-06-12 14:45:55.001720 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'324,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] add_log_entry 173'3326 (173'3325) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144984 2014-06-12 08:44:57.377854
2014-06-12 14:45:55.001743 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'324,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] append_log  adding 1 keys
2014-06-12 14:45:55.001758 7f9d693e3700 10 trim log((125'324,173'3326], crt=0'0) to 125'325
2014-06-12 14:45:55.001785 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3326, trimmed: 125'325
2014-06-12 14:45:55.001811 7f9d693e3700  5 queue_transactions existing osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:55.001816 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.001818 7f9d693e3700  5 queue_op 0x5a4c540 seq 2983 osr(3.395 0x276d740) 518424 bytes   (queue has 5 ops and 3624277 bytes)
2014-06-12 14:45:55.001830 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227e360 (5 active)
2014-06-12 14:45:55.001827 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] eval_repop repgather(0x2b62b800 173'3326 rep_tid=236141 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.001846 7f9d693e3700 10 osd.2 173 dequeue_op 0x5ee9700 finish
2014-06-12 14:45:55.001850 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.001853 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.001969 7f9d82972700 15 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:55.001988 7f9d82972700 10 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 = 0
2014-06-12 14:45:55.001994 7f9d82972700 15 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:55.002003 7f9d82972700 10 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 = 0
2014-06-12 14:45:55.002594 7f9d82972700 20 set_header setting 16925 cid 3.169_head oid e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 parent seq 0
2014-06-12 14:45:55.002745 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.002760 7f9d82972700 20 set_header setting 2306 cid meta oid 1fe2acfa/pglog_3.169/0//-1 parent seq 0
2014-06-12 14:45:55.002771 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:45:55.007039 7f9d5703e700  5 -- op tracker -- , seq: 747312, time: 2014-06-12 14:45:55.006985, event: header_read, op: osd_sub_op_reply(client.4314.0:144981 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.007051 7f9d5703e700  5 -- op tracker -- , seq: 747312, time: 2014-06-12 14:45:55.006987, event: throttled, op: osd_sub_op_reply(client.4314.0:144981 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.007058 7f9d5703e700  5 -- op tracker -- , seq: 747312, time: 2014-06-12 14:45:55.007002, event: all_read, op: osd_sub_op_reply(client.4314.0:144981 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.007065 7f9d5703e700  5 -- op tracker -- , seq: 747312, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144981 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.007070 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144981 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:55.007075 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:55.007083 7f9d5703e700 15 osd.2 173 enqueue_op 0x31d61e00 prio 196 cost 0 latency 0.000097 osd_sub_op_reply(client.4314.0:144981 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:55.007192 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.007202 7f9d62bd6700 10 osd.2 173 dequeue_op 0x31d61e00 prio 196 cost 0 latency 0.000217 osd_sub_op_reply(client.4314.0:144981 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 pg pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean]
2014-06-12 14:45:55.007228 7f9d62bd6700  5 -- op tracker -- , seq: 747312, time: 2014-06-12 14:45:55.007224, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144981 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.007236 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] handle_message: 0x31d61e00
2014-06-12 14:45:55.007255 7f9d62bd6700  5 -- op tracker -- , seq: 747312, time: 2014-06-12 14:45:55.007251, event: started, op: osd_sub_op_reply(client.4314.0:144981 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.007261 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] sub_op_modify_reply: tid 236138 op  ack_type 4 from 20
2014-06-12 14:45:55.007288 7f9d62bd6700  5 -- op tracker -- , seq: 747308, time: 2014-06-12 14:45:55.007278, event: sub_op_commit_rec, op: osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.007298 7f9d62bd6700 10 osd.2 173 dequeue_op 0x31d61e00 finish
2014-06-12 14:45:55.007305 7f9d62bd6700  5 -- op tracker -- , seq: 747312, time: 2014-06-12 14:45:55.007302, event: done, op: osd_sub_op_reply(client.4314.0:144981 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.007320 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.007323 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.007764 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.007870 7f9d81970700 15 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.007882 7f9d81970700 10 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1 r = 0
2014-06-12 14:45:55.007890 7f9d81970700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.007910 7f9d81970700 15 _write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 860160~516096
2014-06-12 14:45:55.008028 7f9d81970700 10 lookup_strip_header done  cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.008034 7f9d82972700 10 _do_op 0x15168e40 seq 2601 r = 0, finisher 0x5fba580 0
2014-06-12 14:45:55.008049 7f9d82972700 10 _finish_op 0x15168e40 seq 2601 osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:45:55.008077 7f9d7795c700  5 -- op tracker -- , seq: 747306, time: 2014-06-12 14:45:55.008069, event: sub_op_applied, op: osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.008089 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active] sub_op_modify_applied on 0x6319a40 op osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.008118 7f9d7795c700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:55.008142 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active] op_applied on primary on version 173'2863
2014-06-12 14:45:55.008155 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active] op_applied on replica on version 173'2863
2014-06-12 14:45:55.008180 7f9d7715b700  5 -- op tracker -- , seq: 747306, time: 2014-06-12 14:45:55.008173, event: commit_sent, op: osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.008187 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2861 active] sub_op_modify_commit on op osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:45:55.008206 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:55.008202 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54f1200 (4 active)
2014-06-12 14:45:55.008211 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.008239 7f9d7715b700  5 -- op tracker -- , seq: 747306, time: 2014-06-12 14:45:55.008234, event: done, op: osd_sub_op(client.4296.0:143498 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.008286 7f9d81970700 10 file_to_extents done 
2014-06-12 14:45:55.009661 7f9d81970700 10 _generic_write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 860160~516096 = 0
2014-06-12 14:45:55.009726 7f9d81970700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.009746 7f9d81970700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:55.009752 7f9d81970700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.009759 7f9d81970700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:55.009957 7f9d81970700 20 set_header setting 17862 cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 parent seq 0
2014-06-12 14:45:55.010029 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.010043 7f9d81970700 20 set_header setting 2060 cid meta oid 1fe1ea2a/pglog_3.395/0//-1 parent seq 0
2014-06-12 14:45:55.010053 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:45:55.011999 7f9d81970700 10 _do_op 0x72c42a0 seq 2980 r = 0, finisher 0x59faac0 0x31aed640
2014-06-12 14:45:55.012039 7f9d81970700 10 _finish_op 0x72c42a0 seq 2980 osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:55.012051 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227e360 (3 active)
2014-06-12 14:45:55.012060 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.012044 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] op_commit: 236138
2014-06-12 14:45:55.012079 7f9d7715b700  5 -- op tracker -- , seq: 747308, time: 2014-06-12 14:45:55.012068, event: op_commit, op: osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.012092 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3322 crt=0'0 lcod 173'3321 mlcod 173'3321 active+clean] repop_all_committed: repop tid 236138 all committed 
2014-06-12 14:45:55.012100 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3321 active+clean] eval_repop repgather(0x15e56300 173'3323 rep_tid=236138 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.012111 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3321 active+clean] log_op_stats osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.025751
2014-06-12 14:45:55.012124 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3321 active+clean] publish_stats_to_osd 173:4284
2014-06-12 14:45:55.012131 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3321 active+clean]  sending commit on repgather(0x15e56300 173'3323 rep_tid=236138 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) 0x18418c80
2014-06-12 14:45:55.012160 7f9d7715b700  5 -- op tracker -- , seq: 747308, time: 2014-06-12 14:45:55.012154, event: commit_sent, op: osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.012172 7f9d7c966700  5 _do_op 0x19ed6d20 seq 2981 osr(3.395 0x276d740)/0x276d740 start
2014-06-12 14:45:55.012177 7f9d7c966700 10 _do_transaction on 0x5312c00
2014-06-12 14:45:55.012185 7f9d7c966700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.012296 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.012312 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.012347 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.012430 7f9d7c966700 15 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.012436 7f9d7c966700 10 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1 r = 0
2014-06-12 14:45:55.012440 7f9d7c966700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.012454 7f9d7c966700 15 _write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 1376256~1032192
2014-06-12 14:45:55.012555 7f9d7c966700 10 lookup_strip_header done  cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.012784 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:45:55.013645 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3321 active+clean] op_applied: 236138
2014-06-12 14:45:55.013696 7f9d7795c700  5 -- op tracker -- , seq: 747308, time: 2014-06-12 14:45:55.013681, event: op_applied, op: osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.013713 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3321 active+clean] op_applied on primary on version 173'3323
2014-06-12 14:45:55.013725 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3321 active+clean] repop_all_applied: repop tid 236138 all applied 
2014-06-12 14:45:55.013736 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3321 active+clean] eval_repop repgather(0x15e56300 173'3323 rep_tid=236138 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.013754 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3321 active+clean] log_op_stats osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.027394
2014-06-12 14:45:55.013772 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3321 active+clean] publish_stats_to_osd 173:4285
2014-06-12 14:45:55.013785 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3321 active+clean]  requeue_ops 
2014-06-12 14:45:55.013795 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean]  removing repgather(0x15e56300 173'3323 rep_tid=236138 committed?=1 applied?=1 op=osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.013810 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean]    q front is repgather(0x15e56300 173'3323 rep_tid=236138 committed?=1 applied?=1 op=osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.013824 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] remove_repop repgather(0x15e56300 173'3323 rep_tid=236138 committed?=1 applied?=1 op=osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.013837 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean]  requeue_ops 
2014-06-12 14:45:55.013859 7f9d7795c700  5 -- op tracker -- , seq: 747308, time: 2014-06-12 14:45:55.013854, event: done, op: osd_op(client.4314.0:144981 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.016520 7f9d7c966700 10 _generic_write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 1376256~1032192 = 0
2014-06-12 14:45:55.016610 7f9d7c966700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.016647 7f9d7c966700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:55.016653 7f9d7c966700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.016661 7f9d7c966700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:55.016963 7f9d7c966700 20 set_header setting 17862 cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 parent seq 0
2014-06-12 14:45:55.017074 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.017087 7f9d7c966700 20 set_header setting 2060 cid meta oid 1fe1ea2a/pglog_3.395/0//-1 parent seq 0
2014-06-12 14:45:55.017097 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:45:55.017720 7f9d5703e700  5 -- op tracker -- , seq: 747313, time: 2014-06-12 14:45:55.014210, event: header_read, op: osd_sub_op_reply(client.4314.0:144982 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.017732 7f9d5703e700  5 -- op tracker -- , seq: 747313, time: 2014-06-12 14:45:55.014212, event: throttled, op: osd_sub_op_reply(client.4314.0:144982 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.017740 7f9d5703e700  5 -- op tracker -- , seq: 747313, time: 2014-06-12 14:45:55.014225, event: all_read, op: osd_sub_op_reply(client.4314.0:144982 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.017748 7f9d5703e700  5 -- op tracker -- , seq: 747313, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144982 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.017755 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144982 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:55.017762 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:55.017774 7f9d5703e700 15 osd.2 173 enqueue_op 0x31d61200 prio 196 cost 0 latency 0.003564 osd_sub_op_reply(client.4314.0:144982 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:55.017813 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.017824 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x31d61200 prio 196 cost 0 latency 0.003613 osd_sub_op_reply(client.4314.0:144982 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 pg pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean]
2014-06-12 14:45:55.017844 7f9d6b3e7700  5 -- op tracker -- , seq: 747313, time: 2014-06-12 14:45:55.017841, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144982 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.017853 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] handle_message: 0x31d61200
2014-06-12 14:45:55.017868 7f9d6b3e7700  5 -- op tracker -- , seq: 747313, time: 2014-06-12 14:45:55.017865, event: started, op: osd_sub_op_reply(client.4314.0:144982 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.017874 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] sub_op_modify_reply: tid 236139 op  ack_type 4 from 20
2014-06-12 14:45:55.017899 7f9d6b3e7700  5 -- op tracker -- , seq: 747309, time: 2014-06-12 14:45:55.017893, event: sub_op_commit_rec, op: osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.017906 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x31d61200 finish
2014-06-12 14:45:55.017912 7f9d6b3e7700  5 -- op tracker -- , seq: 747313, time: 2014-06-12 14:45:55.017909, event: done, op: osd_sub_op_reply(client.4314.0:144982 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.017925 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.017928 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.019420 7f9d5703e700  5 -- op tracker -- , seq: 747314, time: 2014-06-12 14:45:55.019348, event: header_read, op: osd_sub_op_reply(client.4314.0:144983 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.019435 7f9d5703e700  5 -- op tracker -- , seq: 747314, time: 2014-06-12 14:45:55.019350, event: throttled, op: osd_sub_op_reply(client.4314.0:144983 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.019443 7f9d5703e700  5 -- op tracker -- , seq: 747314, time: 2014-06-12 14:45:55.019379, event: all_read, op: osd_sub_op_reply(client.4314.0:144983 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.019451 7f9d5703e700  5 -- op tracker -- , seq: 747314, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144983 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.019457 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144983 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:55.019462 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:55.019471 7f9d5703e700 15 osd.2 173 enqueue_op 0x31d60500 prio 196 cost 0 latency 0.000122 osd_sub_op_reply(client.4314.0:144983 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:55.019501 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.019510 7f9d64bda700 10 osd.2 173 dequeue_op 0x31d60500 prio 196 cost 0 latency 0.000162 osd_sub_op_reply(client.4314.0:144983 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 pg pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean]
2014-06-12 14:45:55.019532 7f9d64bda700  5 -- op tracker -- , seq: 747314, time: 2014-06-12 14:45:55.019529, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144983 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.019540 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] handle_message: 0x31d60500
2014-06-12 14:45:55.019560 7f9d64bda700  5 -- op tracker -- , seq: 747314, time: 2014-06-12 14:45:55.019556, event: started, op: osd_sub_op_reply(client.4314.0:144983 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.019566 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] sub_op_modify_reply: tid 236140 op  ack_type 4 from 20
2014-06-12 14:45:55.019591 7f9d64bda700  5 -- op tracker -- , seq: 747310, time: 2014-06-12 14:45:55.019583, event: sub_op_commit_rec, op: osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.019602 7f9d64bda700 10 osd.2 173 dequeue_op 0x31d60500 finish
2014-06-12 14:45:55.019610 7f9d64bda700  5 -- op tracker -- , seq: 747314, time: 2014-06-12 14:45:55.019606, event: done, op: osd_sub_op_reply(client.4314.0:144983 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.019629 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.019632 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.021461 7f9d425f5700  5 -- op tracker -- , seq: 747315, time: 2014-06-12 14:45:55.014605, event: header_read, op: osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.021484 7f9d425f5700  5 -- op tracker -- , seq: 747315, time: 2014-06-12 14:45:55.014609, event: throttled, op: osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.021495 7f9d425f5700  5 -- op tracker -- , seq: 747315, time: 2014-06-12 14:45:55.015469, event: all_read, op: osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.021505 7f9d425f5700  5 -- op tracker -- , seq: 747315, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.021515 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:45:55.021552 7f9d425f5700 15 osd.2 173 enqueue_op 0x5ee8f00 prio 63 cost 516096 latency 0.006947 osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.021593 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.021600 7f9d683e1700 10 osd.2 173 dequeue_op 0x5ee8f00 prio 63 cost 516096 latency 0.006995 osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean]
2014-06-12 14:45:55.021635 7f9d683e1700  5 -- op tracker -- , seq: 747315, time: 2014-06-12 14:45:55.021629, event: reached_pg, op: osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.021648 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:55.021666 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] handle_message: 0x5ee8f00
2014-06-12 14:45:55.021678 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] do_op osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:55.021711 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] get_object_context: found obc in cache: 0x29f75340
2014-06-12 14:45:55.021723 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] get_object_context: 0x29f75340 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 rwstate(write n=3 w=0) oi: 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3326 client.4314.0:144984 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3326) ssc: 0x72b88c0 snapset: 0=[]:[]+head
2014-06-12 14:45:55.021739 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] find_object_context 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 @head oi=61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3326 client.4314.0:144984 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3326)
2014-06-12 14:45:55.021766 7f9d683e1700  5 -- op tracker -- , seq: 747315, time: 2014-06-12 14:45:55.021760, event: started, op: osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.021774 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] execute_ctx 0x5df9000
2014-06-12 14:45:55.021792 7f9d683e1700  5 -- op tracker -- , seq: 747315, time: 2014-06-12 14:45:55.021786, event: started, op: osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.021799 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] do_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] ov 173'3326 av 173'3327 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:55.021814 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] do_osd_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096]
2014-06-12 14:45:55.021829 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:55.021847 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] do_osd_op  write 3440640~516096
2014-06-12 14:45:55.021862 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:55.021873 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 snapset=0x72b88f8  snapc=0=[]
2014-06-12 14:45:55.021885 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:55.021896 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] finish_ctx 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 0x5df9000 op modify  
2014-06-12 14:45:55.021908 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean]  set mtime to 2014-06-12 08:44:57.394828
2014-06-12 14:45:55.021930 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean]  final snapset 0=[]:[]+head in 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.021952 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean]  zeroing write result code 0
2014-06-12 14:45:55.021962 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] calc_trim_to 125'325 -> 125'326
2014-06-12 14:45:55.021971 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] new_repop rep_tid 236142 on osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.021988 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] issue_repop rep_tid 236142 o 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.022013 7f9d683e1700  5 -- op tracker -- , seq: 747315, time: 2014-06-12 14:45:55.022008, event: waiting for subops from 20, op: osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.022040 7f9d683e1700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:55.022063 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3326 (125'325,173'3326] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] append_log log((125'325,173'3326], crt=0'0) [173'3327 (173'3326) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144985 2014-06-12 08:44:57.394828]
2014-06-12 14:45:55.022087 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'325,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] add_log_entry 173'3327 (173'3326) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144985 2014-06-12 08:44:57.394828
2014-06-12 14:45:55.022108 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'325,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] append_log  adding 1 keys
2014-06-12 14:45:55.022121 7f9d683e1700 10 trim log((125'325,173'3327], crt=0'0) to 125'326
2014-06-12 14:45:55.022155 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3327, trimmed: 125'326
2014-06-12 14:45:55.022183 7f9d683e1700  5 queue_transactions existing osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:55.022187 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.022189 7f9d683e1700  5 queue_op 0xdf8a720 seq 2984 osr(3.395 0x276d740) 518424 bytes   (queue has 4 ops and 2589792 bytes)
2014-06-12 14:45:55.022199 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] eval_repop repgather(0x5323200 173'3327 rep_tid=236142 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.022218 7f9d683e1700 10 osd.2 173 dequeue_op 0x5ee8f00 finish
2014-06-12 14:45:55.022223 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.022228 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.023317 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227e360 (4 active)
2014-06-12 14:45:55.025417 7f9d5501e700  5 -- op tracker -- , seq: 747316, time: 2014-06-12 14:45:55.022726, event: header_read, op: osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.025445 7f9d5501e700  5 -- op tracker -- , seq: 747316, time: 2014-06-12 14:45:55.022728, event: throttled, op: osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.025457 7f9d5501e700  5 -- op tracker -- , seq: 747316, time: 2014-06-12 14:45:55.025361, event: all_read, op: osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.025466 7f9d5501e700  5 -- op tracker -- , seq: 747316, time: 0.000000, event: dispatched, op: osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.025476 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:55.025483 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:45:55.025493 7f9d5501e700 15 osd.2 173 enqueue_op 0x2519cb00 prio 127 cost 1032999 latency 0.002767 osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.025544 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.025551 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2519cb00 prio 127 cost 1032999 latency 0.002825 osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.169( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2862 active]
2014-06-12 14:45:55.025574 7f9d65bdc700  5 -- op tracker -- , seq: 747316, time: 2014-06-12 14:45:55.025570, event: reached_pg, op: osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.025584 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2862 active] handle_message: 0x2519cb00
2014-06-12 14:45:55.025595 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2862 active] sub_op_modify trans e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 v 173'2864 (transaction) 174
2014-06-12 14:45:55.025610 7f9d65bdc700  5 -- op tracker -- , seq: 747316, time: 2014-06-12 14:45:55.025607, event: started, op: osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.025650 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2862 active] append_log log((0'0,173'2863], crt=0'0) [173'2864 (173'2863) modify   e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 by client.4296.0:143499 2014-06-12 08:44:57.392242]
2014-06-12 14:45:55.025679 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2862 active] add_log_entry 173'2864 (173'2863) modify   e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 by client.4296.0:143499 2014-06-12 08:44:57.392242
2014-06-12 14:45:55.025709 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2862 active] append_log  adding 1 keys
2014-06-12 14:45:55.025749 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2864, trimmed: 
2014-06-12 14:45:55.025777 7f9d65bdc700  5 -- op tracker -- , seq: 747316, time: 2014-06-12 14:45:55.025772, event: started, op: osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.025790 7f9d65bdc700  5 queue_transactions existing osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:45:55.025795 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.025797 7f9d65bdc700  5 queue_op 0x12451800 seq 2602 osr(3.169 0x276ce50) 1034485 bytes   (queue has 5 ops and 3624277 bytes)
2014-06-12 14:45:55.025805 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.169( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2862 active] do_sub_op osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.025819 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2519cb00 finish
2014-06-12 14:45:55.025822 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.025824 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.025832 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54f1200 (5 active)
2014-06-12 14:45:55.025837 7f9d7a962700  5 _do_op 0x12451800 seq 2602 osr(3.169 0x276ce50)/0x276ce50 start
2014-06-12 14:45:55.025841 7f9d7a962700 10 _do_transaction on 0x16bce9b0
2014-06-12 14:45:55.025848 7f9d7a962700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:55.026109 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:55.026131 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.031219 7f9d5703e700  5 -- op tracker -- , seq: 747317, time: 2014-06-12 14:45:55.031162, event: header_read, op: osd_sub_op_reply(client.4314.0:144984 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.031250 7f9d5703e700  5 -- op tracker -- , seq: 747317, time: 2014-06-12 14:45:55.031164, event: throttled, op: osd_sub_op_reply(client.4314.0:144984 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.031365 7f9d5703e700  5 -- op tracker -- , seq: 747317, time: 2014-06-12 14:45:55.031177, event: all_read, op: osd_sub_op_reply(client.4314.0:144984 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.031414 7f9d5703e700  5 -- op tracker -- , seq: 747317, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144984 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.031447 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144984 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:55.031482 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:55.031592 7f9d5703e700 15 osd.2 173 enqueue_op 0x2519da00 prio 196 cost 0 latency 0.000430 osd_sub_op_reply(client.4314.0:144984 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:55.031684 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.031692 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2519da00 prio 196 cost 0 latency 0.000530 osd_sub_op_reply(client.4314.0:144984 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 pg pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean]
2014-06-12 14:45:55.031715 7f9d6abe6700  5 -- op tracker -- , seq: 747317, time: 2014-06-12 14:45:55.031711, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144984 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.031723 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] handle_message: 0x2519da00
2014-06-12 14:45:55.031739 7f9d6abe6700  5 -- op tracker -- , seq: 747317, time: 2014-06-12 14:45:55.031736, event: started, op: osd_sub_op_reply(client.4314.0:144984 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.031746 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] sub_op_modify_reply: tid 236141 op  ack_type 4 from 20
2014-06-12 14:45:55.031769 7f9d6abe6700  5 -- op tracker -- , seq: 747311, time: 2014-06-12 14:45:55.031760, event: sub_op_commit_rec, op: osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.031777 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2519da00 finish
2014-06-12 14:45:55.031784 7f9d6abe6700  5 -- op tracker -- , seq: 747317, time: 2014-06-12 14:45:55.031780, event: done, op: osd_sub_op_reply(client.4314.0:144984 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.031798 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.031801 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.032634 7f9d7c966700 10 _do_op 0x19ed6d20 seq 2981 r = 0, finisher 0x31d0c520 0x381e6640
2014-06-12 14:45:55.032659 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] op_commit: 236139
2014-06-12 14:45:55.032687 7f9d7715b700  5 -- op tracker -- , seq: 747309, time: 2014-06-12 14:45:55.032677, event: op_commit, op: osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.032696 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3323 crt=0'0 lcod 173'3322 mlcod 173'3322 active+clean] repop_all_committed: repop tid 236139 all committed 
2014-06-12 14:45:55.032708 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3322 active+clean] eval_repop repgather(0x2dc1f380 173'3324 rep_tid=236139 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.032724 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3322 active+clean] log_op_stats osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.000000 lat 0.043694
2014-06-12 14:45:55.032742 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3322 active+clean] publish_stats_to_osd 173:4286
2014-06-12 14:45:55.032753 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3322 active+clean]  sending commit on repgather(0x2dc1f380 173'3324 rep_tid=236139 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) 0x24675180
2014-06-12 14:45:55.032790 7f9d7715b700  5 -- op tracker -- , seq: 747309, time: 2014-06-12 14:45:55.032784, event: commit_sent, op: osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.032831 7f9d7c966700 10 _finish_op 0x19ed6d20 seq 2981 osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:55.032840 7f9d7895e700  5 _do_op 0x1b41800 seq 2982 osr(3.395 0x276d740)/0x276d740 start
2014-06-12 14:45:55.032844 7f9d7895e700 10 _do_transaction on 0xd5e2c00
2014-06-12 14:45:55.032851 7f9d7895e700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.033007 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.033028 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.033049 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3322 active+clean] op_applied: 236139
2014-06-12 14:45:55.033071 7f9d7795c700  5 -- op tracker -- , seq: 747309, time: 2014-06-12 14:45:55.033065, event: op_applied, op: osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.033085 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.033079 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3322 active+clean] op_applied on primary on version 173'3324
2014-06-12 14:45:55.033090 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3322 active+clean] repop_all_applied: repop tid 236139 all applied 
2014-06-12 14:45:55.033100 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3322 active+clean] eval_repop repgather(0x2dc1f380 173'3324 rep_tid=236139 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.033115 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3322 active+clean] log_op_stats osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.000000 lat 0.044085
2014-06-12 14:45:55.033130 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3322 active+clean] publish_stats_to_osd 173:4287
2014-06-12 14:45:55.033141 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3322 active+clean]  requeue_ops 
2014-06-12 14:45:55.033151 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean]  removing repgather(0x2dc1f380 173'3324 rep_tid=236139 committed?=1 applied?=1 op=osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.033165 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean]    q front is repgather(0x2dc1f380 173'3324 rep_tid=236139 committed?=1 applied?=1 op=osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.033186 7f9d7a962700 15 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:55.033179 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] remove_repop repgather(0x2dc1f380 173'3324 rep_tid=236139 committed?=1 applied?=1 op=osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.033194 7f9d7a962700 10 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1 r = 0
2014-06-12 14:45:55.033192 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean]  requeue_ops 
2014-06-12 14:45:55.033201 7f9d7a962700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:55.033221 7f9d7a962700 15 _write 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 2039808~1032192
2014-06-12 14:45:55.033240 7f9d7c966700  5 -- op tracker -- , seq: 747309, time: 2014-06-12 14:45:55.033227, event: done, op: osd_op(client.4314.0:144982 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 1376256~1032192] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.033365 7f9d7a962700 10 lookup_strip_header done  cid 3.169_head oid e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:55.033392 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227e360 (4 active)
2014-06-12 14:45:55.033396 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.033455 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:45:55.035374 7f9d425f5700  5 -- op tracker -- , seq: 747318, time: 2014-06-12 14:45:55.034882, event: header_read, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.035402 7f9d425f5700  5 -- op tracker -- , seq: 747318, time: 2014-06-12 14:45:55.034885, event: throttled, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.035412 7f9d425f5700  5 -- op tracker -- , seq: 747318, time: 2014-06-12 14:45:55.035272, event: all_read, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.035423 7f9d425f5700  5 -- op tracker -- , seq: 747318, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.035435 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:45:55.035473 7f9d425f5700 15 osd.2 173 enqueue_op 0x8dbc700 prio 63 cost 237568 latency 0.000590 osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.035518 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.035527 7f9d66bde700 10 osd.2 173 dequeue_op 0x8dbc700 prio 63 cost 237568 latency 0.000645 osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean]
2014-06-12 14:45:55.035556 7f9d66bde700  5 -- op tracker -- , seq: 747318, time: 2014-06-12 14:45:55.035551, event: reached_pg, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.035566 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:55.035582 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] handle_message: 0x8dbc700
2014-06-12 14:45:55.035594 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] do_op osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:55.035629 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] get_object_context: found obc in cache: 0x29f75340
2014-06-12 14:45:55.035642 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] get_object_context: 0x29f75340 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 rwstate(write n=3 w=0) oi: 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3327 client.4314.0:144985 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3327) ssc: 0x72b88c0 snapset: 0=[]:[]+head
2014-06-12 14:45:55.035663 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] find_object_context 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 @head oi=61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3(173'3327 client.4314.0:144985 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3327)
2014-06-12 14:45:55.035701 7f9d66bde700  5 -- op tracker -- , seq: 747318, time: 2014-06-12 14:45:55.035694, event: started, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.035710 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] execute_ctx 0x292ac00
2014-06-12 14:45:55.035729 7f9d66bde700  5 -- op tracker -- , seq: 747318, time: 2014-06-12 14:45:55.035723, event: started, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.035746 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] do_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] ov 173'3327 av 173'3328 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:55.035763 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] do_osd_op 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568]
2014-06-12 14:45:55.035775 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:55.035791 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] do_osd_op  write 3956736~237568
2014-06-12 14:45:55.035811 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:45:55.035823 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 snapset=0x72b88f8  snapc=0=[]
2014-06-12 14:45:55.035837 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] make_writeable 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:55.035847 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] finish_ctx 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 0x292ac00 op modify  
2014-06-12 14:45:55.035861 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean]  set mtime to 2014-06-12 08:44:57.415671
2014-06-12 14:45:55.035883 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean]  final snapset 0=[]:[]+head in 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.035903 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean]  zeroing write result code 0
2014-06-12 14:45:55.035914 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] calc_trim_to 125'326 -> 125'327
2014-06-12 14:45:55.035923 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] new_repop rep_tid 236143 on osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.035940 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] issue_repop rep_tid 236143 o 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.035966 7f9d66bde700  5 -- op tracker -- , seq: 747318, time: 2014-06-12 14:45:55.035960, event: waiting for subops from 20, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.035991 7f9d66bde700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:55.036018 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3327 (125'326,173'3327] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] append_log log((125'326,173'3327], crt=0'0) [173'3328 (173'3327) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144986 2014-06-12 08:44:57.415671]
2014-06-12 14:45:55.036056 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'326,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] add_log_entry 173'3328 (173'3327) modify   61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 by client.4314.0:144986 2014-06-12 08:44:57.415671
2014-06-12 14:45:55.036077 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'326,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] append_log  adding 1 keys
2014-06-12 14:45:55.036090 7f9d66bde700 10 trim log((125'326,173'3328], crt=0'0) to 125'327
2014-06-12 14:45:55.036118 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3328, trimmed: 125'327
2014-06-12 14:45:55.036169 7f9d66bde700  5 queue_transactions existing osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:55.036174 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.036177 7f9d66bde700  5 queue_op 0xed4da40 seq 2985 osr(3.395 0x276d740) 239896 bytes   (queue has 5 ops and 2829653 bytes)
2014-06-12 14:45:55.036185 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] eval_repop repgather(0x384e20c0 173'3328 rep_tid=236143 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.036204 7f9d66bde700 10 osd.2 173 dequeue_op 0x8dbc700 finish
2014-06-12 14:45:55.036209 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.036214 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.036274 7f9d5703e700  5 -- op tracker -- , seq: 747319, time: 2014-06-12 14:45:55.036233, event: header_read, op: osd_sub_op_reply(client.4314.0:144985 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.036285 7f9d5703e700  5 -- op tracker -- , seq: 747319, time: 2014-06-12 14:45:55.036235, event: throttled, op: osd_sub_op_reply(client.4314.0:144985 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.036293 7f9d5703e700  5 -- op tracker -- , seq: 747319, time: 2014-06-12 14:45:55.036253, event: all_read, op: osd_sub_op_reply(client.4314.0:144985 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.036301 7f9d5703e700  5 -- op tracker -- , seq: 747319, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144985 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.036306 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144985 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:55.036312 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:55.036319 7f9d5703e700 15 osd.2 173 enqueue_op 0x2519dc00 prio 196 cost 0 latency 0.000086 osd_sub_op_reply(client.4314.0:144985 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:55.036358 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.036364 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2519dc00 prio 196 cost 0 latency 0.000131 osd_sub_op_reply(client.4314.0:144985 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 pg pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean]
2014-06-12 14:45:55.036382 7f9d63bd8700  5 -- op tracker -- , seq: 747319, time: 2014-06-12 14:45:55.036379, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144985 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.036390 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] handle_message: 0x2519dc00
2014-06-12 14:45:55.036405 7f9d63bd8700  5 -- op tracker -- , seq: 747319, time: 2014-06-12 14:45:55.036402, event: started, op: osd_sub_op_reply(client.4314.0:144985 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.036411 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] sub_op_modify_reply: tid 236142 op  ack_type 4 from 20
2014-06-12 14:45:55.036430 7f9d63bd8700  5 -- op tracker -- , seq: 747315, time: 2014-06-12 14:45:55.036423, event: sub_op_commit_rec, op: osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.036437 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2519dc00 finish
2014-06-12 14:45:55.036444 7f9d63bd8700  5 -- op tracker -- , seq: 747319, time: 2014-06-12 14:45:55.036441, event: done, op: osd_sub_op_reply(client.4314.0:144985 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.036459 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.036462 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.036468 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227e360 (5 active)
2014-06-12 14:45:55.038076 7f9d7a962700 10 _generic_write 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 2039808~1032192 = 0
2014-06-12 14:45:55.038152 7f9d7a962700 15 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:55.038172 7f9d7a962700 10 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 = 0
2014-06-12 14:45:55.038177 7f9d7a962700 15 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:55.038184 7f9d7a962700 10 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 = 0
2014-06-12 14:45:55.038483 7f9d7a962700 20 set_header setting 16925 cid 3.169_head oid e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 parent seq 0
2014-06-12 14:45:55.038591 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.038605 7f9d7a962700 20 set_header setting 2306 cid meta oid 1fe2acfa/pglog_3.169/0//-1 parent seq 0
2014-06-12 14:45:55.038619 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:45:55.040561 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:45:55.041737 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:45:55.041818 7f9d5703e700  5 -- op tracker -- , seq: 747320, time: 2014-06-12 14:45:55.041785, event: header_read, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ack, result = 0)
2014-06-12 14:45:55.041828 7f9d5703e700  5 -- op tracker -- , seq: 747320, time: 2014-06-12 14:45:55.041786, event: throttled, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ack, result = 0)
2014-06-12 14:45:55.041836 7f9d5703e700  5 -- op tracker -- , seq: 747320, time: 2014-06-12 14:45:55.041794, event: all_read, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ack, result = 0)
2014-06-12 14:45:55.041844 7f9d5703e700  5 -- op tracker -- , seq: 747320, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ack, result = 0)
2014-06-12 14:45:55.041848 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:45:55.041854 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:55.041861 7f9d5703e700 15 osd.2 173 enqueue_op 0x28dd0b00 prio 196 cost 0 latency 0.000075 osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ack, result = 0) v2
2014-06-12 14:45:55.041911 7f9d5703e700  5 -- op tracker -- , seq: 747321, time: 2014-06-12 14:45:55.041881, event: header_read, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.041920 7f9d5703e700  5 -- op tracker -- , seq: 747321, time: 2014-06-12 14:45:55.041882, event: throttled, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.041928 7f9d5703e700  5 -- op tracker -- , seq: 747321, time: 2014-06-12 14:45:55.041892, event: all_read, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.041935 7f9d5703e700  5 -- op tracker -- , seq: 747321, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.041939 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:55.041944 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:55.041949 7f9d5703e700 15 osd.2 173 enqueue_op 0xe20a800 prio 196 cost 0 latency 0.000068 osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:55.041979 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.041986 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x28dd0b00 prio 196 cost 0 latency 0.000201 osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ack, result = 0) v2 pg pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean]
2014-06-12 14:45:55.042007 7f9d6a3e5700  5 -- op tracker -- , seq: 747320, time: 2014-06-12 14:45:55.042003, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ack, result = 0)
2014-06-12 14:45:55.042021 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] handle_message: 0x28dd0b00
2014-06-12 14:45:55.042038 7f9d6a3e5700  5 -- op tracker -- , seq: 747320, time: 2014-06-12 14:45:55.042034, event: started, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ack, result = 0)
2014-06-12 14:45:55.042045 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] sub_op_modify_reply: tid 236143 op  ack_type 1 from 20
2014-06-12 14:45:55.042065 7f9d6a3e5700  5 -- op tracker -- , seq: 747318, time: 2014-06-12 14:45:55.042057, event: sub_op_applied_rec, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.042072 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x28dd0b00 finish
2014-06-12 14:45:55.042079 7f9d6a3e5700  5 -- op tracker -- , seq: 747320, time: 2014-06-12 14:45:55.042076, event: done, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ack, result = 0)
2014-06-12 14:45:55.042095 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.042101 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.042105 7f9d6a3e5700 10 osd.2 173 dequeue_op 0xe20a800 prio 196 cost 0 latency 0.000224 osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0) v2 pg pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean]
2014-06-12 14:45:55.042121 7f9d6a3e5700  5 -- op tracker -- , seq: 747321, time: 2014-06-12 14:45:55.042118, event: reached_pg, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.042128 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] handle_message: 0xe20a800
2014-06-12 14:45:55.042141 7f9d6a3e5700  5 -- op tracker -- , seq: 747321, time: 2014-06-12 14:45:55.042138, event: started, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.042147 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] sub_op_modify_reply: tid 236143 op  ack_type 4 from 20
2014-06-12 14:45:55.042163 7f9d6a3e5700  5 -- op tracker -- , seq: 747318, time: 2014-06-12 14:45:55.042158, event: sub_op_commit_rec, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.042170 7f9d6a3e5700 10 osd.2 173 dequeue_op 0xe20a800 finish
2014-06-12 14:45:55.042176 7f9d6a3e5700  5 -- op tracker -- , seq: 747321, time: 2014-06-12 14:45:55.042172, event: done, op: osd_sub_op_reply(client.4314.0:144986 3.395 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.042185 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.042188 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.042204 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.044242 7f9d7a962700 10 _do_op 0x12451800 seq 2602 r = 0, finisher 0x172fffa0 0
2014-06-12 14:45:55.044282 7f9d7715b700  5 -- op tracker -- , seq: 747316, time: 2014-06-12 14:45:55.044271, event: commit_sent, op: osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.044291 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2862 active] sub_op_modify_commit on op osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:45:55.044321 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:55.044376 7f9d7a962700 10 _finish_op 0x12451800 seq 2602 osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:45:55.044400 7f9d7795c700  5 -- op tracker -- , seq: 747316, time: 2014-06-12 14:45:55.044395, event: sub_op_applied, op: osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.044410 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2863 active] sub_op_modify_applied on 0x16bce8c0 op osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.044426 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2863 active] op_applied on primary on version 173'2864
2014-06-12 14:45:55.044437 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2863 active] op_applied on replica on version 173'2864
2014-06-12 14:45:55.044469 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.044473 7f9d7a962700  5 -- op tracker -- , seq: 747316, time: 2014-06-12 14:45:55.044463, event: done, op: osd_sub_op(client.4296.0:143499 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.044503 7f9d7895e700 15 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.044505 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54f1200 (4 active)
2014-06-12 14:45:55.044509 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.044512 7f9d7895e700 10 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1 r = 0
2014-06-12 14:45:55.044518 7f9d7895e700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.044619 7f9d7895e700 15 _write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 2408448~516096
2014-06-12 14:45:55.044761 7f9d7895e700 10 lookup_strip_header done  cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.044919 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:45:55.047109 7f9d7895e700 10 _generic_write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 2408448~516096 = 0
2014-06-12 14:45:55.047170 7f9d7895e700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.047190 7f9d7895e700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:55.047199 7f9d7895e700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.047207 7f9d7895e700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:55.047430 7f9d7895e700 20 set_header setting 17862 cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 parent seq 0
2014-06-12 14:45:55.047497 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.047513 7f9d7895e700 20 set_header setting 2060 cid meta oid 1fe1ea2a/pglog_3.395/0//-1 parent seq 0
2014-06-12 14:45:55.047526 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:45:55.079422 7f9d5501e700  5 -- op tracker -- , seq: 747322, time: 2014-06-12 14:45:55.068700, event: header_read, op: osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.079439 7f9d5501e700  5 -- op tracker -- , seq: 747322, time: 2014-06-12 14:45:55.068702, event: throttled, op: osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.079448 7f9d5501e700  5 -- op tracker -- , seq: 747322, time: 2014-06-12 14:45:55.079375, event: all_read, op: osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.079457 7f9d5501e700  5 -- op tracker -- , seq: 747322, time: 0.000000, event: dispatched, op: osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.079464 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:55.079471 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:45:55.079481 7f9d5501e700 15 osd.2 173 enqueue_op 0x876f200 prio 127 cost 1032999 latency 0.010781 osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.079592 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.079605 7f9d69be4700 10 osd.2 173 dequeue_op 0x876f200 prio 127 cost 1032999 latency 0.010905 osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.169( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2863 active]
2014-06-12 14:45:55.079634 7f9d69be4700  5 -- op tracker -- , seq: 747322, time: 2014-06-12 14:45:55.079629, event: reached_pg, op: osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.079655 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2863 active] handle_message: 0x876f200
2014-06-12 14:45:55.079669 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2863 active] sub_op_modify trans e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 v 173'2865 (transaction) 174
2014-06-12 14:45:55.079687 7f9d69be4700  5 -- op tracker -- , seq: 747322, time: 2014-06-12 14:45:55.079682, event: started, op: osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.079708 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2863 active] append_log log((0'0,173'2864], crt=0'0) [173'2865 (173'2864) modify   e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 by client.4296.0:143500 2014-06-12 08:44:57.435453]
2014-06-12 14:45:55.079735 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2863 active] add_log_entry 173'2865 (173'2864) modify   e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 by client.4296.0:143500 2014-06-12 08:44:57.435453
2014-06-12 14:45:55.079778 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2863 active] append_log  adding 1 keys
2014-06-12 14:45:55.079815 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2865, trimmed: 
2014-06-12 14:45:55.079844 7f9d69be4700  5 -- op tracker -- , seq: 747322, time: 2014-06-12 14:45:55.079838, event: started, op: osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.079857 7f9d69be4700  5 queue_transactions existing osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:45:55.079861 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.079863 7f9d69be4700  5 queue_op 0x614d6e0 seq 2603 osr(3.169 0x276ce50) 1034485 bytes   (queue has 5 ops and 2829653 bytes)
2014-06-12 14:45:55.079876 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54f1200 (5 active)
2014-06-12 14:45:55.079885 7f9d7e169700  5 _do_op 0x614d6e0 seq 2603 osr(3.169 0x276ce50)/0x276ce50 start
2014-06-12 14:45:55.079871 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.169( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2863 active] do_sub_op osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.079890 7f9d7e169700 10 _do_transaction on 0x13c6c630
2014-06-12 14:45:55.079891 7f9d69be4700 10 osd.2 173 dequeue_op 0x876f200 finish
2014-06-12 14:45:55.079894 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.079897 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.079897 7f9d7e169700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:55.089986 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:55.090055 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.091047 7f9d7895e700 10 _do_op 0x1b41800 seq 2982 r = 0, finisher 0x31388380 0x1a3f1a00
2014-06-12 14:45:55.091102 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] op_commit: 236140
2014-06-12 14:45:55.091134 7f9d7715b700  5 -- op tracker -- , seq: 747310, time: 2014-06-12 14:45:55.091125, event: op_commit, op: osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.091142 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3324 crt=0'0 lcod 173'3323 mlcod 173'3323 active+clean] repop_all_committed: repop tid 236140 all committed 
2014-06-12 14:45:55.091153 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3323 active+clean] eval_repop repgather(0x18224540 173'3325 rep_tid=236140 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.091169 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3323 active+clean] log_op_stats osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.097067
2014-06-12 14:45:55.091186 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3323 active+clean] publish_stats_to_osd 173:4288
2014-06-12 14:45:55.091195 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3323 active+clean]  sending commit on repgather(0x18224540 173'3325 rep_tid=236140 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) 0x5ffb180
2014-06-12 14:45:55.091232 7f9d7715b700  5 -- op tracker -- , seq: 747310, time: 2014-06-12 14:45:55.091227, event: commit_sent, op: osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.091350 7f9d7895e700 10 _finish_op 0x1b41800 seq 2982 osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:55.091381 7f9d7f16b700  5 _do_op 0x5a4c540 seq 2983 osr(3.395 0x276d740)/0x276d740 start
2014-06-12 14:45:55.091390 7f9d7f16b700 10 _do_transaction on 0x384e2000
2014-06-12 14:45:55.091397 7f9d7f16b700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.091390 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3323 active+clean] op_applied: 236140
2014-06-12 14:45:55.091411 7f9d7795c700  5 -- op tracker -- , seq: 747310, time: 2014-06-12 14:45:55.091405, event: op_applied, op: osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.091418 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3323 active+clean] op_applied on primary on version 173'3325
2014-06-12 14:45:55.091428 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3323 active+clean] repop_all_applied: repop tid 236140 all applied 
2014-06-12 14:45:55.091436 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3323 active+clean] eval_repop repgather(0x18224540 173'3325 rep_tid=236140 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.091450 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3323 active+clean] log_op_stats osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.097349
2014-06-12 14:45:55.091464 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3323 active+clean] publish_stats_to_osd 173:4289
2014-06-12 14:45:55.091474 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3323 active+clean]  requeue_ops 
2014-06-12 14:45:55.091483 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3324 active+clean]  removing repgather(0x18224540 173'3325 rep_tid=236140 committed?=1 applied?=1 op=osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.091496 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3324 active+clean]    q front is repgather(0x18224540 173'3325 rep_tid=236140 committed?=1 applied?=1 op=osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.091509 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3324 active+clean] remove_repop repgather(0x18224540 173'3325 rep_tid=236140 committed?=1 applied?=1 op=osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.091518 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.091521 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3324 active+clean]  requeue_ops 
2014-06-12 14:45:55.091544 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.091563 7f9d7895e700  5 -- op tracker -- , seq: 747310, time: 2014-06-12 14:45:55.091551, event: done, op: osd_op(client.4314.0:144983 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.091587 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227e360 (4 active)
2014-06-12 14:45:55.091590 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.091596 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.091685 7f9d7f16b700 15 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.091693 7f9d7f16b700 10 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1 r = 0
2014-06-12 14:45:55.091700 7f9d7f16b700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.091718 7f9d7f16b700 15 _write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 2924544~516096
2014-06-12 14:45:55.091893 7f9d7f16b700 10 lookup_strip_header done  cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.092042 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:45:55.094627 7f9d7f16b700 10 _generic_write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 2924544~516096 = 0
2014-06-12 14:45:55.094708 7f9d7f16b700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.094738 7f9d7f16b700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:55.094759 7f9d7f16b700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.094782 7f9d7f16b700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:55.094992 7f9d7f16b700 20 set_header setting 17862 cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 parent seq 0
2014-06-12 14:45:55.095088 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.095125 7f9d7f16b700 20 set_header setting 2060 cid meta oid 1fe1ea2a/pglog_3.395/0//-1 parent seq 0
2014-06-12 14:45:55.095155 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:45:55.098035 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.098080 7f9d7e169700 15 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:55.098090 7f9d7e169700 10 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1 r = 0
2014-06-12 14:45:55.098097 7f9d7e169700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:55.098117 7f9d7e169700 15 _write 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 3072000~1032192
2014-06-12 14:45:55.098293 7f9d7e169700 10 lookup_strip_header done  cid 3.169_head oid e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:55.098454 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:45:55.098514 7f9d7f16b700 10 _do_op 0x5a4c540 seq 2983 r = 0, finisher 0x8586960 0x1b743780
2014-06-12 14:45:55.098548 7f9d7f16b700 10 _finish_op 0x5a4c540 seq 2983 osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:55.098553 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3324 active+clean] op_commit: 236141
2014-06-12 14:45:55.098590 7f9d7715b700  5 -- op tracker -- , seq: 747311, time: 2014-06-12 14:45:55.098579, event: op_commit, op: osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.098602 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3325 crt=0'0 lcod 173'3324 mlcod 173'3324 active+clean] repop_all_committed: repop tid 236141 all committed 
2014-06-12 14:45:55.098616 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3324 active+clean] eval_repop repgather(0x2b62b800 173'3326 rep_tid=236141 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.098634 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3324 active+clean] log_op_stats osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.098629
2014-06-12 14:45:55.098654 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3324 active+clean] publish_stats_to_osd 173:4290
2014-06-12 14:45:55.098595 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227e360 (3 active)
2014-06-12 14:45:55.098667 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3324 active+clean]  sending commit on repgather(0x2b62b800 173'3326 rep_tid=236141 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) 0x2844aa00
2014-06-12 14:45:55.098703 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.098706 7f9d7715b700  5 -- op tracker -- , seq: 747311, time: 2014-06-12 14:45:55.098701, event: commit_sent, op: osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.098726 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3324 active+clean] op_applied: 236141
2014-06-12 14:45:55.098748 7f9d7795c700  5 -- op tracker -- , seq: 747311, time: 2014-06-12 14:45:55.098743, event: op_applied, op: osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.098756 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3324 active+clean] op_applied on primary on version 173'3326
2014-06-12 14:45:55.098768 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3324 active+clean] repop_all_applied: repop tid 236141 all applied 
2014-06-12 14:45:55.098778 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3324 active+clean] eval_repop repgather(0x2b62b800 173'3326 rep_tid=236141 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.098793 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3324 active+clean] log_op_stats osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.098790
2014-06-12 14:45:55.098808 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3324 active+clean] publish_stats_to_osd 173:4291
2014-06-12 14:45:55.098818 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3324 active+clean]  requeue_ops 
2014-06-12 14:45:55.098830 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3325 active+clean]  removing repgather(0x2b62b800 173'3326 rep_tid=236141 committed?=1 applied?=1 op=osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.098845 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3325 active+clean]    q front is repgather(0x2b62b800 173'3326 rep_tid=236141 committed?=1 applied?=1 op=osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.098861 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3325 active+clean] remove_repop repgather(0x2b62b800 173'3326 rep_tid=236141 committed?=1 applied?=1 op=osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.098881 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3325 active+clean]  requeue_ops 
2014-06-12 14:45:55.098902 7f9d7795c700  5 -- op tracker -- , seq: 747311, time: 2014-06-12 14:45:55.098897, event: done, op: osd_op(client.4314.0:144984 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.099831 7f9d82171700  5 _do_op 0xdf8a720 seq 2984 osr(3.395 0x276d740)/0x276d740 start
2014-06-12 14:45:55.099847 7f9d82171700 10 _do_transaction on 0x2c3c6240
2014-06-12 14:45:55.099857 7f9d82171700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.100003 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.100106 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.103224 7f9d7e169700 10 _generic_write 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 3072000~1032192 = 0
2014-06-12 14:45:55.103347 7f9d7e169700 15 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:55.103364 7f9d7e169700 10 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 = 0
2014-06-12 14:45:55.103370 7f9d7e169700 15 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:55.103383 7f9d7e169700 10 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 = 0
2014-06-12 14:45:55.103706 7f9d7e169700 20 set_header setting 16925 cid 3.169_head oid e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 parent seq 0
2014-06-12 14:45:55.103829 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.103847 7f9d7e169700 20 set_header setting 2306 cid meta oid 1fe2acfa/pglog_3.169/0//-1 parent seq 0
2014-06-12 14:45:55.103864 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:45:55.109299 7f9d7e169700 10 _do_op 0x614d6e0 seq 2603 r = 0, finisher 0x960d0e0 0
2014-06-12 14:45:55.109324 7f9d7e169700 10 _finish_op 0x614d6e0 seq 2603 osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:45:55.109344 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54f1200 (2 active)
2014-06-12 14:45:55.109354 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.109345 7f9d7715b700  5 -- op tracker -- , seq: 747322, time: 2014-06-12 14:45:55.109332, event: commit_sent, op: osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.109363 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2863 active] sub_op_modify_commit on op osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:45:55.109391 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:55.109428 7f9d7795c700  5 -- op tracker -- , seq: 747322, time: 2014-06-12 14:45:55.109421, event: sub_op_applied, op: osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.109436 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2864 active] sub_op_modify_applied on 0x13c6c540 op osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.109453 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2864 active] op_applied on primary on version 173'2865
2014-06-12 14:45:55.109463 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2864 active] op_applied on replica on version 173'2865
2014-06-12 14:45:55.109483 7f9d7795c700  5 -- op tracker -- , seq: 747322, time: 2014-06-12 14:45:55.109478, event: done, op: osd_sub_op(client.4296.0:143500 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.109499 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.109530 7f9d82171700 15 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.109539 7f9d82171700 10 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1 r = 0
2014-06-12 14:45:55.109546 7f9d82171700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.109563 7f9d82171700 15 _write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 3440640~516096
2014-06-12 14:45:55.109723 7f9d82171700 10 lookup_strip_header done  cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.109961 7f9d82171700 10 file_to_extents done 
2014-06-12 14:45:55.111696 7f9d82171700 10 _generic_write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 3440640~516096 = 0
2014-06-12 14:45:55.111759 7f9d82171700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.111775 7f9d82171700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:55.111780 7f9d82171700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.111785 7f9d82171700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:55.111934 7f9d82171700 20 set_header setting 17862 cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 parent seq 0
2014-06-12 14:45:55.111991 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.112001 7f9d82171700 20 set_header setting 2060 cid meta oid 1fe1ea2a/pglog_3.395/0//-1 parent seq 0
2014-06-12 14:45:55.112008 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:45:55.113969 7f9d82171700 10 _do_op 0xdf8a720 seq 2984 r = 0, finisher 0x6ce65e0 0x389968c0
2014-06-12 14:45:55.113982 7f9d82171700 10 _finish_op 0xdf8a720 seq 2984 osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:55.113990 7f9d7d167700  5 _do_op 0xed4da40 seq 2985 osr(3.395 0x276d740)/0x276d740 start
2014-06-12 14:45:55.113998 7f9d7d167700 10 _do_transaction on 0x11c7e840
2014-06-12 14:45:55.114001 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227e360 (1 active)
2014-06-12 14:45:55.113986 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3325 active+clean] op_commit: 236142
2014-06-12 14:45:55.114004 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.114004 7f9d7d167700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.114014 7f9d7715b700  5 -- op tracker -- , seq: 747315, time: 2014-06-12 14:45:55.114006, event: op_commit, op: osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.114024 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3326 crt=0'0 lcod 173'3325 mlcod 173'3325 active+clean] repop_all_committed: repop tid 236142 all committed 
2014-06-12 14:45:55.114035 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3325 active+clean] eval_repop repgather(0x5323200 173'3327 rep_tid=236142 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.114058 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3325 active+clean] log_op_stats osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.099452
2014-06-12 14:45:55.114076 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3325 active+clean] publish_stats_to_osd 173:4292
2014-06-12 14:45:55.114088 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3325 active+clean]  sending commit on repgather(0x5323200 173'3327 rep_tid=236142 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) 0x34e4d180
2014-06-12 14:45:55.114121 7f9d7715b700  5 -- op tracker -- , seq: 747315, time: 2014-06-12 14:45:55.114116, event: commit_sent, op: osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.114137 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.114149 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3325 active+clean] op_applied: 236142
2014-06-12 14:45:55.114161 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.114165 7f9d7795c700  5 -- op tracker -- , seq: 747315, time: 2014-06-12 14:45:55.114161, event: op_applied, op: osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.114171 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3325 active+clean] op_applied on primary on version 173'3327
2014-06-12 14:45:55.114177 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3325 active+clean] repop_all_applied: repop tid 236142 all applied 
2014-06-12 14:45:55.114186 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3325 active+clean] eval_repop repgather(0x5323200 173'3327 rep_tid=236142 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.114199 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3325 active+clean] log_op_stats osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.099594
2014-06-12 14:45:55.114208 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3325 active+clean] publish_stats_to_osd 173:4293
2014-06-12 14:45:55.114212 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.114215 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3325 active+clean]  requeue_ops 
2014-06-12 14:45:55.114221 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3326 active+clean]  removing repgather(0x5323200 173'3327 rep_tid=236142 committed?=1 applied?=1 op=osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.114229 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3326 active+clean]    q front is repgather(0x5323200 173'3327 rep_tid=236142 committed?=1 applied?=1 op=osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.114236 7f9d7d167700 15 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.114237 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3326 active+clean] remove_repop repgather(0x5323200 173'3327 rep_tid=236142 committed?=1 applied?=1 op=osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.114244 7f9d7d167700 10 _omap_rmkeys meta/1fe1ea2a/pglog_3.395/0//-1 r = 0
2014-06-12 14:45:55.114245 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3326 active+clean]  requeue_ops 
2014-06-12 14:45:55.114250 7f9d7d167700 15 _omap_setkeys meta/1fe1ea2a/pglog_3.395/0//-1
2014-06-12 14:45:55.114258 7f9d7795c700  5 -- op tracker -- , seq: 747315, time: 2014-06-12 14:45:55.114255, event: done, op: osd_op(client.4314.0:144985 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~516096] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.114395 7f9d7d167700 15 _write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 3956736~237568
2014-06-12 14:45:55.114540 7f9d7d167700 10 lookup_strip_header done  cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.114641 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:45:55.115392 7f9d7d167700 10 _generic_write 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 3956736~237568 = 0
2014-06-12 14:45:55.115435 7f9d7d167700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.115453 7f9d7d167700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:55.115461 7f9d7d167700 15 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3
2014-06-12 14:45:55.115469 7f9d7d167700 10 _setattrs 3.395_head/61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 = 0
2014-06-12 14:45:55.115609 7f9d7d167700 20 set_header setting 17862 cid 3.395_head oid 61cc1395/rbd_data.10b16b8b4567.000000000000104b/head//3 parent seq 0
2014-06-12 14:45:55.115672 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.115690 7f9d7d167700 20 set_header setting 2060 cid meta oid 1fe1ea2a/pglog_3.395/0//-1 parent seq 0
2014-06-12 14:45:55.115702 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:45:55.116211 7f9d4a3c5700  5 -- op tracker -- , seq: 747323, time: 2014-06-12 14:45:55.113640, event: header_read, op: osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.116225 7f9d4a3c5700  5 -- op tracker -- , seq: 747323, time: 2014-06-12 14:45:55.113643, event: throttled, op: osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.116239 7f9d4a3c5700  5 -- op tracker -- , seq: 747323, time: 2014-06-12 14:45:55.116161, event: all_read, op: osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.116248 7f9d4a3c5700  5 -- op tracker -- , seq: 747323, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.116257 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:45:55.116285 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x25e43700 prio 63 cost 942080 latency 0.002644 osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.116325 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.116328 7f9d5501e700  5 -- op tracker -- , seq: 747324, time: 2014-06-12 14:45:55.116113, event: header_read, op: osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.116339 7f9d5501e700  5 -- op tracker -- , seq: 747324, time: 2014-06-12 14:45:55.116115, event: throttled, op: osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.116332 7f9d623d5700 10 osd.2 173 dequeue_op 0x25e43700 prio 63 cost 942080 latency 0.002692 osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean]
2014-06-12 14:45:55.116348 7f9d5501e700  5 -- op tracker -- , seq: 747324, time: 2014-06-12 14:45:55.116302, event: all_read, op: osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.116359 7f9d5501e700  5 -- op tracker -- , seq: 747324, time: 0.000000, event: dispatched, op: osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.116365 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:55.116371 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:45:55.116370 7f9d623d5700  5 -- op tracker -- , seq: 747323, time: 2014-06-12 14:45:55.116351, event: reached_pg, op: osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.116378 7f9d5501e700 15 osd.2 173 enqueue_op 0x876f100 prio 127 cost 90919 latency 0.000265 osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.116379 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:55.116394 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] handle_message: 0x25e43700
2014-06-12 14:45:55.116404 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] do_op osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:55.116417 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:45:55.116428 7f9d623d5700 15 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 '_'
2014-06-12 14:45:55.116422 7f9d633d7700 10 osd.2 173 dequeue_op 0x876f100 prio 127 cost 90919 latency 0.000309 osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.169( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2864 active]
2014-06-12 14:45:55.116443 7f9d633d7700  5 -- op tracker -- , seq: 747324, time: 2014-06-12 14:45:55.116439, event: reached_pg, op: osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.116450 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2864 active] handle_message: 0x876f100
2014-06-12 14:45:55.116461 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2864 active] sub_op_modify trans e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 v 173'2866 (transaction) 174
2014-06-12 14:45:55.116478 7f9d633d7700  5 -- op tracker -- , seq: 747324, time: 2014-06-12 14:45:55.116474, event: started, op: osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.116495 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2864 active] append_log log((0'0,173'2865], crt=0'0) [173'2866 (173'2865) modify   e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 by client.4296.0:143501 2014-06-12 08:44:57.492767]
2014-06-12 14:45:55.116518 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2866 (0'0,173'2866] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2864 active] add_log_entry 173'2866 (173'2865) modify   e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 by client.4296.0:143501 2014-06-12 08:44:57.492767
2014-06-12 14:45:55.116539 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2866 (0'0,173'2866] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2864 active] append_log  adding 1 keys
2014-06-12 14:45:55.116573 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2866, trimmed: 
2014-06-12 14:45:55.116600 7f9d633d7700  5 -- op tracker -- , seq: 747324, time: 2014-06-12 14:45:55.116595, event: started, op: osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.116610 7f9d633d7700  5 queue_transactions existing osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:45:55.116614 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.116616 7f9d633d7700  5 queue_op 0x34c6d80 seq 2604 osr(3.169 0x276ce50) 92405 bytes   (queue has 2 ops and 332301 bytes)
2014-06-12 14:45:55.116625 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.169( v 173'2866 (0'0,173'2866] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2864 active] do_sub_op osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.116638 7f9d633d7700 10 osd.2 173 dequeue_op 0x876f100 finish
2014-06-12 14:45:55.116642 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:45:55.116644 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.116830 7f9d7d167700 10 _do_op 0xed4da40 seq 2985 r = 0, finisher 0x65dd840 0x12d18ac0
2014-06-12 14:45:55.116840 7f9d7d167700 10 _finish_op 0xed4da40 seq 2985 osr(3.395 0x276d740)/0x276d740
2014-06-12 14:45:55.116853 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227e360 (0 active)
2014-06-12 14:45:55.116845 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3326 active+clean] op_commit: 236143
2014-06-12 14:45:55.116857 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54f1200 (1 active)
2014-06-12 14:45:55.116864 7f9d7d167700  5 _do_op 0x34c6d80 seq 2604 osr(3.169 0x276ce50)/0x276ce50 start
2014-06-12 14:45:55.116871 7f9d7d167700 10 _do_transaction on 0x50e7970
2014-06-12 14:45:55.116876 7f9d7d167700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:55.116872 7f9d7715b700  5 -- op tracker -- , seq: 747318, time: 2014-06-12 14:45:55.116864, event: op_commit, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.116888 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3327 crt=0'0 lcod 173'3326 mlcod 173'3326 active+clean] repop_all_committed: repop tid 236143 all committed 
2014-06-12 14:45:55.116900 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3326 active+clean] eval_repop repgather(0x384e20c0 173'3328 rep_tid=236143 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.116917 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3326 active+clean] log_op_stats osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 237847 outb 0 rlat 0.000000 lat 0.082034
2014-06-12 14:45:55.116932 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3326 active+clean] publish_stats_to_osd 173:4294
2014-06-12 14:45:55.116941 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3326 active+clean]  sending commit on repgather(0x384e20c0 173'3328 rep_tid=236143 committed?=1 applied?=0 lock=0 op=osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) 0x31d25b80
2014-06-12 14:45:55.116962 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:55.116985 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.116978 7f9d7715b700  5 -- op tracker -- , seq: 747318, time: 2014-06-12 14:45:55.116972, event: commit_sent, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.117019 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.117042 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.117065 7f9d7d167700 15 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:55.117072 7f9d7d167700 10 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1 r = 0
2014-06-12 14:45:55.117077 7f9d7d167700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:45:55.117093 7f9d7d167700 15 _write 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 4104192~90112
2014-06-12 14:45:55.117093 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3326 active+clean] op_applied: 236143
2014-06-12 14:45:55.117113 7f9d7795c700  5 -- op tracker -- , seq: 747318, time: 2014-06-12 14:45:55.117107, event: op_applied, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.117121 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3326 active+clean] op_applied on primary on version 173'3328
2014-06-12 14:45:55.117132 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3326 active+clean] repop_all_applied: repop tid 236143 all applied 
2014-06-12 14:45:55.117144 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3326 active+clean] eval_repop repgather(0x384e20c0 173'3328 rep_tid=236143 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.117162 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3326 active+clean] log_op_stats osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4 inb 237847 outb 0 rlat 0.000000 lat 0.082279
2014-06-12 14:45:55.117179 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3326 active+clean] publish_stats_to_osd 173:4295
2014-06-12 14:45:55.117191 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3326 active+clean]  requeue_ops 
2014-06-12 14:45:55.117201 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3327 active+clean]  removing repgather(0x384e20c0 173'3328 rep_tid=236143 committed?=1 applied?=1 op=osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.117217 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3327 active+clean]    q front is repgather(0x384e20c0 173'3328 rep_tid=236143 committed?=1 applied?=1 op=osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.117241 7f9d7d167700 10 lookup_strip_header done  cid 3.169_head oid e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:55.117232 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3327 active+clean] remove_repop repgather(0x384e20c0 173'3328 rep_tid=236143 committed?=1 applied?=1 op=osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.117248 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3327 active+clean]  requeue_ops 
2014-06-12 14:45:55.117257 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:45:55.117275 7f9d7795c700  5 -- op tracker -- , seq: 747318, time: 2014-06-12 14:45:55.117269, event: done, op: osd_op(client.4314.0:144986 rbd_data.10b16b8b4567.000000000000104b [set-alloc-hint object_size 4194304 write_size 4194304,write 3956736~237568] 3.61cc1395 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.117531 7f9d7d167700 10 _generic_write 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 4104192~90112 = 0
2014-06-12 14:45:55.117548 7f9d7d167700 15 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:55.117558 7f9d7d167700 10 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 = 0
2014-06-12 14:45:55.117564 7f9d7d167700 15 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3
2014-06-12 14:45:55.117572 7f9d7d167700 10 _setattrs 3.169_head/e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 = 0
2014-06-12 14:45:55.118558 7f9d7d167700 20 set_header setting 16925 cid 3.169_head oid e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 parent seq 0
2014-06-12 14:45:55.118625 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.118661 7f9d7d167700 20 set_header setting 2306 cid meta oid 1fe2acfa/pglog_3.169/0//-1 parent seq 0
2014-06-12 14:45:55.118695 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:45:55.119688 7f9d7d167700 10 _do_op 0x34c6d80 seq 2604 r = 0, finisher 0x31387220 0
2014-06-12 14:45:55.119726 7f9d7d167700 10 _finish_op 0x34c6d80 seq 2604 osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:45:55.119741 7f9d7715b700  5 -- op tracker -- , seq: 747324, time: 2014-06-12 14:45:55.119732, event: commit_sent, op: osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.119755 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2866 (0'0,173'2866] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2864 active] sub_op_modify_commit on op osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:45:55.119783 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:55.119816 7f9d7795c700  5 -- op tracker -- , seq: 747324, time: 2014-06-12 14:45:55.119809, event: sub_op_applied, op: osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.119825 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2866 (0'0,173'2866] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2865 active] sub_op_modify_applied on 0x50e7880 op osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.119841 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2866 (0'0,173'2866] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2865 active] op_applied on primary on version 173'2866
2014-06-12 14:45:55.119851 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2866 (0'0,173'2866] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2865 active] op_applied on replica on version 173'2866
2014-06-12 14:45:55.119876 7f9d7795c700  5 -- op tracker -- , seq: 747324, time: 2014-06-12 14:45:55.119871, event: done, op: osd_sub_op(client.4296.0:143501 3.169 e887169/rbd_data.109f6b8b4567.00000000000011a2/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.119805 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54f1200 (0 active)
2014-06-12 14:45:55.119977 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.165429 7f9d623d5700 10 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 '_' = 0
2014-06-12 14:45:55.165458 7f9d623d5700 15 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 'snapset'
2014-06-12 14:45:55.165669 7f9d623d5700 10 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 'snapset' = 0
2014-06-12 14:45:55.165679 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] populate_obc_watchers 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.165694 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.165704 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] get_object_context: creating obc from disk: 0x35963600
2014-06-12 14:45:55.165713 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] get_object_context: 0x35963600 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 rwstate(none n=0 w=0) oi: 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3(173'671 client.4296.0:40788 wrlock_by=unknown.0.0:0 dirty s 4194304 uv671) ssc: 0x72b9960 snapset: 0=[]:[]+head
2014-06-12 14:45:55.165728 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] find_object_context 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 @head oi=5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3(173'671 client.4296.0:40788 wrlock_by=unknown.0.0:0 dirty s 4194304 uv671)
2014-06-12 14:45:55.165757 7f9d623d5700  5 -- op tracker -- , seq: 747323, time: 2014-06-12 14:45:55.165750, event: started, op: osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.165767 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] execute_ctx 0x9084c00
2014-06-12 14:45:55.165785 7f9d623d5700  5 -- op tracker -- , seq: 747323, time: 2014-06-12 14:45:55.165780, event: started, op: osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.165792 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] do_op 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] ov 173'671 av 173'2726 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:55.165806 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] do_osd_op 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080]
2014-06-12 14:45:55.165817 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:55.165831 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] do_osd_op  write 0~942080
2014-06-12 14:45:55.165913 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:45:55.165925 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] make_writeable 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 snapset=0x72b9998  snapc=0=[]
2014-06-12 14:45:55.165936 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] make_writeable 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:55.165947 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] finish_ctx 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 0x9084c00 op modify  
2014-06-12 14:45:55.165959 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean]  set mtime to 2014-06-12 08:44:57.493242
2014-06-12 14:45:55.165980 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean]  final snapset 0=[]:[]+head in 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.165998 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean]  zeroing write result code 0
2014-06-12 14:45:55.166008 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] new_repop rep_tid 236144 on osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.166024 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] issue_repop rep_tid 236144 o 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.166048 7f9d623d5700  5 -- op tracker -- , seq: 747323, time: 2014-06-12 14:45:55.166043, event: waiting for subops from 25, op: osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.166073 7f9d623d5700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:45:55.166096 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2725 (0'0,173'2725] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] append_log log((0'0,173'2725], crt=0'0) [173'2726 (173'671) modify   5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 by client.4296.0:143502 2014-06-12 08:44:57.493242]
2014-06-12 14:45:55.166119 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2725 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] add_log_entry 173'2726 (173'671) modify   5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 by client.4296.0:143502 2014-06-12 08:44:57.493242
2014-06-12 14:45:55.166140 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2725 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] append_log  adding 1 keys
2014-06-12 14:45:55.166172 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2726, trimmed: 
2014-06-12 14:45:55.166196 7f9d623d5700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:45:55.166201 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.166202 7f9d623d5700  5 queue_op 0x6ec6240 seq 2538 osr(3.122 0x276cf40) 944373 bytes   (queue has 1 ops and 944373 bytes)
2014-06-12 14:45:55.166209 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2725 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] eval_repop repgather(0x13d23c80 173'2726 rep_tid=236144 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.166227 7f9d623d5700 10 osd.2 173 dequeue_op 0x25e43700 finish
2014-06-12 14:45:55.166232 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.166236 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.166907 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:45:55.166948 7f9d7d968700  5 _do_op 0x6ec6240 seq 2538 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:45:55.166973 7f9d7d968700 10 _do_transaction on 0x13d23680
2014-06-12 14:45:55.166999 7f9d7d968700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.167319 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.167366 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.167454 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.167500 7f9d7d968700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.167534 7f9d7d968700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:45:55.167560 7f9d7d968700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.167593 7f9d7d968700 15 _write 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 0~942080
2014-06-12 14:45:55.167767 7f9d7d968700 10 lookup_strip_header done  cid 3.122_head oid 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.167979 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:45:55.171963 7f9d7d968700 10 _generic_write 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 0~942080 = 0
2014-06-12 14:45:55.172225 7f9d7d968700 15 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.172266 7f9d7d968700 10 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 = 0
2014-06-12 14:45:55.172293 7f9d7d968700 15 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.172327 7f9d7d968700 10 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 = 0
2014-06-12 14:45:55.172693 7f9d7d968700 20 set_header setting 16931 cid 3.122_head oid 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 parent seq 0
2014-06-12 14:45:55.172824 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.172861 7f9d7d968700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:45:55.172891 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:45:55.177824 7f9d5bec4700  5 -- op tracker -- , seq: 747325, time: 2014-06-12 14:45:55.177739, event: header_read, op: osd_sub_op_reply(client.4296.0:143502 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.177840 7f9d5bec4700  5 -- op tracker -- , seq: 747325, time: 2014-06-12 14:45:55.177741, event: throttled, op: osd_sub_op_reply(client.4296.0:143502 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.177848 7f9d5bec4700  5 -- op tracker -- , seq: 747325, time: 2014-06-12 14:45:55.177758, event: all_read, op: osd_sub_op_reply(client.4296.0:143502 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.177856 7f9d5bec4700  5 -- op tracker -- , seq: 747325, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143502 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.177861 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143502 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:55.177868 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:45:55.177877 7f9d5bec4700 15 osd.2 173 enqueue_op 0x2519d900 prio 196 cost 0 latency 0.000138 osd_sub_op_reply(client.4296.0:143502 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:55.178160 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.178176 7f9d643d9700 10 osd.2 173 dequeue_op 0x2519d900 prio 196 cost 0 latency 0.000436 osd_sub_op_reply(client.4296.0:143502 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2725 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean]
2014-06-12 14:45:55.178208 7f9d643d9700  5 -- op tracker -- , seq: 747325, time: 2014-06-12 14:45:55.178204, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143502 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.178217 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2725 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] handle_message: 0x2519d900
2014-06-12 14:45:55.178234 7f9d643d9700  5 -- op tracker -- , seq: 747325, time: 2014-06-12 14:45:55.178230, event: started, op: osd_sub_op_reply(client.4296.0:143502 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.178240 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2725 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] sub_op_modify_reply: tid 236144 op  ack_type 4 from 25
2014-06-12 14:45:55.178278 7f9d643d9700  5 -- op tracker -- , seq: 747323, time: 2014-06-12 14:45:55.178268, event: sub_op_commit_rec, op: osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.178287 7f9d643d9700 10 osd.2 173 dequeue_op 0x2519d900 finish
2014-06-12 14:45:55.178295 7f9d643d9700  5 -- op tracker -- , seq: 747325, time: 2014-06-12 14:45:55.178291, event: done, op: osd_sub_op_reply(client.4296.0:143502 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.178311 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.178314 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.180282 7f9d7d968700 10 _do_op 0x6ec6240 seq 2538 r = 0, finisher 0x89a24c0 0x351a7200
2014-06-12 14:45:55.180322 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2725 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] op_commit: 236144
2014-06-12 14:45:55.180358 7f9d7715b700  5 -- op tracker -- , seq: 747323, time: 2014-06-12 14:45:55.180347, event: op_commit, op: osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.180375 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2725 crt=0'0 lcod 173'2724 mlcod 173'2724 active+clean] repop_all_committed: repop tid 236144 all committed 
2014-06-12 14:45:55.180387 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2724 active+clean] eval_repop repgather(0x13d23c80 173'2726 rep_tid=236144 committed?=1 applied?=0 lock=0 op=osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.180404 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2724 active+clean] log_op_stats osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 inb 942359 outb 0 rlat 0.000000 lat 0.066762
2014-06-12 14:45:55.180421 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2724 active+clean] publish_stats_to_osd 173:3622
2014-06-12 14:45:55.180431 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2724 active+clean]  sending commit on repgather(0x13d23c80 173'2726 rep_tid=236144 committed?=1 applied?=0 lock=0 op=osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) 0x28efbb80
2014-06-12 14:45:55.180468 7f9d7715b700  5 -- op tracker -- , seq: 747323, time: 2014-06-12 14:45:55.180463, event: commit_sent, op: osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.180526 7f9d7d968700 10 _finish_op 0x6ec6240 seq 2538 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:45:55.180544 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2724 active+clean] op_applied: 236144
2014-06-12 14:45:55.180564 7f9d7795c700  5 -- op tracker -- , seq: 747323, time: 2014-06-12 14:45:55.180558, event: op_applied, op: osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.180572 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2724 active+clean] op_applied on primary on version 173'2726
2014-06-12 14:45:55.180581 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2724 active+clean] repop_all_applied: repop tid 236144 all applied 
2014-06-12 14:45:55.180590 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2724 active+clean] eval_repop repgather(0x13d23c80 173'2726 rep_tid=236144 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.180604 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2724 active+clean] log_op_stats osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 inb 942359 outb 0 rlat 0.000000 lat 0.066964
2014-06-12 14:45:55.180618 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2724 active+clean] publish_stats_to_osd 173:3623
2014-06-12 14:45:55.180629 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2724 active+clean]  requeue_ops 
2014-06-12 14:45:55.180639 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean]  removing repgather(0x13d23c80 173'2726 rep_tid=236144 committed?=1 applied?=1 op=osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.180652 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean]    q front is repgather(0x13d23c80 173'2726 rep_tid=236144 committed?=1 applied?=1 op=osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.180665 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] remove_repop repgather(0x13d23c80 173'2726 rep_tid=236144 committed?=1 applied?=1 op=osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.180678 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean]  requeue_ops 
2014-06-12 14:45:55.180747 7f9d7d968700  5 -- op tracker -- , seq: 747323, time: 2014-06-12 14:45:55.180732, event: done, op: osd_op(client.4296.0:143502 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~942080] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.180880 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:45:55.180884 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.186738 7f9d4a3c5700  5 -- op tracker -- , seq: 747326, time: 2014-06-12 14:45:55.184371, event: header_read, op: osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.186758 7f9d4a3c5700  5 -- op tracker -- , seq: 747326, time: 2014-06-12 14:45:55.184374, event: throttled, op: osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.186767 7f9d4a3c5700  5 -- op tracker -- , seq: 747326, time: 2014-06-12 14:45:55.186679, event: all_read, op: osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.186775 7f9d4a3c5700  5 -- op tracker -- , seq: 747326, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.186785 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:45:55.186813 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x25e42c00 prio 63 cost 1032192 latency 0.002442 osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.186863 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.186873 7f9d68be2700 10 osd.2 173 dequeue_op 0x25e42c00 prio 63 cost 1032192 latency 0.002501 osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean]
2014-06-12 14:45:55.186898 7f9d68be2700  5 -- op tracker -- , seq: 747326, time: 2014-06-12 14:45:55.186893, event: reached_pg, op: osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.186906 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:55.186923 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] handle_message: 0x25e42c00
2014-06-12 14:45:55.186932 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] do_op osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:55.186954 7f9d68be2700 15 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 '_'
2014-06-12 14:45:55.187171 7f9d68be2700 10 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 '_' = 0
2014-06-12 14:45:55.187189 7f9d68be2700 15 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 'snapset'
2014-06-12 14:45:55.187440 7f9d68be2700 10 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 'snapset' = 0
2014-06-12 14:45:55.187452 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] populate_obc_watchers 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.187467 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.187477 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] get_object_context: creating obc from disk: 0x2b9fbb80
2014-06-12 14:45:55.187486 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] get_object_context: 0x2b9fbb80 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 rwstate(none n=0 w=0) oi: 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3(173'2726 client.4296.0:143502 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2726) ssc: 0x407e380 snapset: 0=[]:[]+head
2014-06-12 14:45:55.187502 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] find_object_context 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 @head oi=5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3(173'2726 client.4296.0:143502 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2726)
2014-06-12 14:45:55.187526 7f9d68be2700  5 -- op tracker -- , seq: 747326, time: 2014-06-12 14:45:55.187521, event: started, op: osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.187536 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] execute_ctx 0x580ac00
2014-06-12 14:45:55.187553 7f9d68be2700  5 -- op tracker -- , seq: 747326, time: 2014-06-12 14:45:55.187549, event: started, op: osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.187562 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] do_op 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] ov 173'2726 av 173'2727 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:55.187576 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] do_osd_op 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192]
2014-06-12 14:45:55.187587 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:55.187600 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] do_osd_op  write 942080~1032192
2014-06-12 14:45:55.187616 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:45:55.187628 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] make_writeable 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 snapset=0x407e3b8  snapc=0=[]
2014-06-12 14:45:55.187640 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] make_writeable 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:55.187652 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] finish_ctx 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 0x580ac00 op modify  
2014-06-12 14:45:55.187663 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean]  set mtime to 2014-06-12 08:44:57.563765
2014-06-12 14:45:55.187683 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean]  final snapset 0=[]:[]+head in 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.187703 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean]  zeroing write result code 0
2014-06-12 14:45:55.187713 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] new_repop rep_tid 236145 on osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.187728 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] issue_repop rep_tid 236145 o 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.187751 7f9d68be2700  5 -- op tracker -- , seq: 747326, time: 2014-06-12 14:45:55.187746, event: waiting for subops from 25, op: osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.187773 7f9d68be2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:45:55.187795 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2726 (0'0,173'2726] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] append_log log((0'0,173'2726], crt=0'0) [173'2727 (173'2726) modify   5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 by client.4296.0:143503 2014-06-12 08:44:57.563765]
2014-06-12 14:45:55.187819 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2726 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] add_log_entry 173'2727 (173'2726) modify   5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 by client.4296.0:143503 2014-06-12 08:44:57.563765
2014-06-12 14:45:55.187839 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2726 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] append_log  adding 1 keys
2014-06-12 14:45:55.187870 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2727, trimmed: 
2014-06-12 14:45:55.187890 7f9d68be2700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:45:55.187894 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.187895 7f9d68be2700  5 queue_op 0x558e480 seq 2539 osr(3.122 0x276cf40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:45:55.187902 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2726 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] eval_repop repgather(0xbe0c900 173'2727 rep_tid=236145 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.187918 7f9d68be2700 10 osd.2 173 dequeue_op 0x25e42c00 finish
2014-06-12 14:45:55.187922 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.187925 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.187932 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:45:55.187937 7f9d7815d700  5 _do_op 0x558e480 seq 2539 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:45:55.187940 7f9d7815d700 10 _do_transaction on 0x384e3740
2014-06-12 14:45:55.187945 7f9d7815d700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.188044 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.188059 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.188113 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.188262 7f9d7815d700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.188267 7f9d7815d700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:45:55.188273 7f9d7815d700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.188290 7f9d7815d700 15 _write 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 942080~1032192
2014-06-12 14:45:55.188419 7f9d7815d700 10 lookup_strip_header done  cid 3.122_head oid 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.188611 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:45:55.193435 7f9d7815d700 10 _generic_write 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 942080~1032192 = 0
2014-06-12 14:45:55.193594 7f9d7815d700 15 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.193659 7f9d7815d700 10 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 = 0
2014-06-12 14:45:55.193691 7f9d7815d700 15 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.193717 7f9d7815d700 10 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 = 0
2014-06-12 14:45:55.194066 7f9d7815d700 20 set_header setting 16931 cid 3.122_head oid 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 parent seq 0
2014-06-12 14:45:55.194207 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.194241 7f9d7815d700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:45:55.194268 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:45:55.199456 7f9d7815d700 10 _do_op 0x558e480 seq 2539 r = 0, finisher 0x65dd380 0x32272e00
2014-06-12 14:45:55.202924 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2726 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] op_commit: 236145
2014-06-12 14:45:55.202988 7f9d7715b700  5 -- op tracker -- , seq: 747326, time: 2014-06-12 14:45:55.202978, event: op_commit, op: osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.203160 7f9d7815d700 10 _finish_op 0x558e480 seq 2539 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:45:55.203192 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:45:55.203220 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.203261 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2726 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] op_applied: 236145
2014-06-12 14:45:55.203295 7f9d7795c700  5 -- op tracker -- , seq: 747326, time: 2014-06-12 14:45:55.203285, event: op_applied, op: osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.203305 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2726 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] op_applied on primary on version 173'2727
2014-06-12 14:45:55.217336 7f9d5bec4700  5 -- op tracker -- , seq: 747327, time: 2014-06-12 14:45:55.217270, event: header_read, op: osd_sub_op_reply(client.4296.0:143503 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.217360 7f9d5bec4700  5 -- op tracker -- , seq: 747327, time: 2014-06-12 14:45:55.217273, event: throttled, op: osd_sub_op_reply(client.4296.0:143503 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.217369 7f9d5bec4700  5 -- op tracker -- , seq: 747327, time: 2014-06-12 14:45:55.217290, event: all_read, op: osd_sub_op_reply(client.4296.0:143503 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.217377 7f9d5bec4700  5 -- op tracker -- , seq: 747327, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143503 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.217383 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143503 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:55.217389 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:45:55.217398 7f9d5bec4700 15 osd.2 173 enqueue_op 0x6f4b900 prio 196 cost 0 latency 0.000128 osd_sub_op_reply(client.4296.0:143503 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:55.217430 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.217442 7f9d61bd4700 10 osd.2 173 dequeue_op 0x6f4b900 prio 196 cost 0 latency 0.000171 osd_sub_op_reply(client.4296.0:143503 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2726 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean]
2014-06-12 14:45:55.217467 7f9d61bd4700  5 -- op tracker -- , seq: 747327, time: 2014-06-12 14:45:55.217463, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143503 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.217476 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2726 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] handle_message: 0x6f4b900
2014-06-12 14:45:55.217495 7f9d61bd4700  5 -- op tracker -- , seq: 747327, time: 2014-06-12 14:45:55.217492, event: started, op: osd_sub_op_reply(client.4296.0:143503 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.217502 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2726 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] sub_op_modify_reply: tid 236145 op  ack_type 4 from 25
2014-06-12 14:45:55.217528 7f9d61bd4700  5 -- op tracker -- , seq: 747326, time: 2014-06-12 14:45:55.217519, event: sub_op_commit_rec, op: osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.217538 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2726 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] repop_all_applied: repop tid 236145 all applied 
2014-06-12 14:45:55.217553 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2726 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] eval_repop repgather(0xbe0c900 173'2727 rep_tid=236145 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.217572 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2726 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean]  sending ack on repgather(0xbe0c900 173'2727 rep_tid=236145 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) 0x56ca500
2014-06-12 14:45:55.217608 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2726 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] repop_all_committed: repop tid 236145 all committed 
2014-06-12 14:45:55.217625 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2725 active+clean] eval_repop repgather(0xbe0c900 173'2727 rep_tid=236145 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.217643 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2725 active+clean] log_op_stats osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.033235 lat 0.033271
2014-06-12 14:45:55.217664 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2725 active+clean] publish_stats_to_osd 173:3624
2014-06-12 14:45:55.217680 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2725 active+clean]  sending commit on repgather(0xbe0c900 173'2727 rep_tid=236145 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) 0x2fa4c280
2014-06-12 14:45:55.217715 7f9d61bd4700  5 -- op tracker -- , seq: 747326, time: 2014-06-12 14:45:55.217709, event: commit_sent, op: osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.217723 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2725 active+clean]  requeue_ops 
2014-06-12 14:45:55.217734 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]  removing repgather(0xbe0c900 173'2727 rep_tid=236145 committed?=1 applied?=1 op=osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.217751 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]    q front is repgather(0xbe0c900 173'2727 rep_tid=236145 committed?=1 applied?=1 op=osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.217768 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] remove_repop repgather(0xbe0c900 173'2727 rep_tid=236145 committed?=1 applied?=1 op=osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.217785 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]  requeue_ops 
2014-06-12 14:45:55.217813 7f9d61bd4700  5 -- op tracker -- , seq: 747326, time: 2014-06-12 14:45:55.217807, event: done, op: osd_op(client.4296.0:143503 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.217835 7f9d61bd4700 10 osd.2 173 dequeue_op 0x6f4b900 finish
2014-06-12 14:45:55.217844 7f9d61bd4700  5 -- op tracker -- , seq: 747327, time: 2014-06-12 14:45:55.217840, event: done, op: osd_sub_op_reply(client.4296.0:143503 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.217855 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.217858 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.223498 7f9d4a3c5700  5 -- op tracker -- , seq: 747328, time: 2014-06-12 14:45:55.221263, event: header_read, op: osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.223523 7f9d4a3c5700  5 -- op tracker -- , seq: 747328, time: 2014-06-12 14:45:55.221265, event: throttled, op: osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.223540 7f9d4a3c5700  5 -- op tracker -- , seq: 747328, time: 2014-06-12 14:45:55.223437, event: all_read, op: osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.223546 7f9d4a3c5700  5 -- op tracker -- , seq: 747328, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.223555 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:45:55.223579 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x25e42a00 prio 63 cost 1032192 latency 0.002316 osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.223636 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.223649 7f9d673df700 10 osd.2 173 dequeue_op 0x25e42a00 prio 63 cost 1032192 latency 0.002386 osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]
2014-06-12 14:45:55.223683 7f9d673df700  5 -- op tracker -- , seq: 747328, time: 2014-06-12 14:45:55.223677, event: reached_pg, op: osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.223697 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:55.223716 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] handle_message: 0x25e42a00
2014-06-12 14:45:55.223726 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] do_op osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:55.223746 7f9d673df700 15 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 '_'
2014-06-12 14:45:55.224313 7f9d673df700 10 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 '_' = 0
2014-06-12 14:45:55.224346 7f9d673df700 15 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 'snapset'
2014-06-12 14:45:55.224416 7f9d673df700 10 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 'snapset' = 0
2014-06-12 14:45:55.224425 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] populate_obc_watchers 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.224433 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.224439 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] get_object_context: creating obc from disk: 0x15519600
2014-06-12 14:45:55.224445 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] get_object_context: 0x15519600 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 rwstate(none n=0 w=0) oi: 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3(173'2727 client.4296.0:143503 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2727) ssc: 0x5bb8620 snapset: 0=[]:[]+head
2014-06-12 14:45:55.224455 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] find_object_context 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 @head oi=5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3(173'2727 client.4296.0:143503 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2727)
2014-06-12 14:45:55.224472 7f9d673df700  5 -- op tracker -- , seq: 747328, time: 2014-06-12 14:45:55.224468, event: started, op: osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.224489 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] execute_ctx 0x33b2d800
2014-06-12 14:45:55.224501 7f9d673df700  5 -- op tracker -- , seq: 747328, time: 2014-06-12 14:45:55.224498, event: started, op: osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.224508 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] do_op 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] ov 173'2727 av 173'2728 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:55.224517 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] do_osd_op 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192]
2014-06-12 14:45:55.224523 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:55.224532 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] do_osd_op  write 1974272~1032192
2014-06-12 14:45:55.224542 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:45:55.224548 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] make_writeable 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 snapset=0x5bb8658  snapc=0=[]
2014-06-12 14:45:55.224556 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] make_writeable 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:55.224562 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] finish_ctx 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 0x33b2d800 op modify  
2014-06-12 14:45:55.224569 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]  set mtime to 2014-06-12 08:44:57.600751
2014-06-12 14:45:55.224584 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]  final snapset 0=[]:[]+head in 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.224596 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]  zeroing write result code 0
2014-06-12 14:45:55.224602 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] new_repop rep_tid 236146 on osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.224612 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] issue_repop rep_tid 236146 o 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.224628 7f9d673df700  5 -- op tracker -- , seq: 747328, time: 2014-06-12 14:45:55.224625, event: waiting for subops from 25, op: osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.224659 7f9d673df700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:45:55.224676 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2727 (0'0,173'2727] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] append_log log((0'0,173'2727], crt=0'0) [173'2728 (173'2727) modify   5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 by client.4296.0:143504 2014-06-12 08:44:57.600751]
2014-06-12 14:45:55.224698 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] add_log_entry 173'2728 (173'2727) modify   5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 by client.4296.0:143504 2014-06-12 08:44:57.600751
2014-06-12 14:45:55.224719 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] append_log  adding 1 keys
2014-06-12 14:45:55.224748 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2728, trimmed: 
2014-06-12 14:45:55.224766 7f9d673df700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:45:55.224771 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.224772 7f9d673df700  5 queue_op 0x5149740 seq 2540 osr(3.122 0x276cf40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:45:55.224781 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:45:55.224787 7f9d8116f700  5 _do_op 0x5149740 seq 2540 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:45:55.224790 7f9d8116f700 10 _do_transaction on 0x52f5bc0
2014-06-12 14:45:55.224778 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] eval_repop repgather(0x1687d140 173'2728 rep_tid=236146 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.224794 7f9d673df700 10 osd.2 173 dequeue_op 0x25e42a00 finish
2014-06-12 14:45:55.224795 7f9d8116f700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.224798 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.224800 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.224881 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.224905 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.224946 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.225037 7f9d8116f700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.225043 7f9d8116f700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:45:55.225047 7f9d8116f700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.225059 7f9d8116f700 15 _write 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 1974272~1032192
2014-06-12 14:45:55.225146 7f9d8116f700 10 lookup_strip_header done  cid 3.122_head oid 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.225414 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:45:55.228672 7f9d8116f700 10 _generic_write 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 1974272~1032192 = 0
2014-06-12 14:45:55.228815 7f9d8116f700 15 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.228854 7f9d8116f700 10 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 = 0
2014-06-12 14:45:55.228862 7f9d8116f700 15 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.228870 7f9d8116f700 10 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 = 0
2014-06-12 14:45:55.229192 7f9d8116f700 20 set_header setting 16931 cid 3.122_head oid 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 parent seq 0
2014-06-12 14:45:55.229304 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.229320 7f9d8116f700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:45:55.229338 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:45:55.235266 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:55.235269 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:55.235271 7f9d74bfa700 20 osd.2 173 _dispatch 0x6c341a0 ping v1
2014-06-12 14:45:55.235273 7f9d74bfa700 10 osd.2 173 ping from client.4294
2014-06-12 14:45:55.235275 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:55.235276 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:55.235565 7f9d8116f700 10 _do_op 0x5149740 seq 2540 r = 0, finisher 0x29fc0120 0x3cc5bc0
2014-06-12 14:45:55.235579 7f9d8116f700 10 _finish_op 0x5149740 seq 2540 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:45:55.235586 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:45:55.235590 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.235584 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] op_commit: 236146
2014-06-12 14:45:55.235609 7f9d7715b700  5 -- op tracker -- , seq: 747328, time: 2014-06-12 14:45:55.235603, event: op_commit, op: osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.235621 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] op_applied: 236146
2014-06-12 14:45:55.235650 7f9d7795c700  5 -- op tracker -- , seq: 747328, time: 2014-06-12 14:45:55.235642, event: op_applied, op: osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.235661 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] op_applied on primary on version 173'2728
2014-06-12 14:45:55.237056 7f9d5bec4700  5 -- op tracker -- , seq: 747329, time: 2014-06-12 14:45:55.237000, event: header_read, op: osd_sub_op_reply(client.4296.0:143504 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.237073 7f9d5bec4700  5 -- op tracker -- , seq: 747329, time: 2014-06-12 14:45:55.237002, event: throttled, op: osd_sub_op_reply(client.4296.0:143504 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.237082 7f9d5bec4700  5 -- op tracker -- , seq: 747329, time: 2014-06-12 14:45:55.237016, event: all_read, op: osd_sub_op_reply(client.4296.0:143504 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.237096 7f9d5bec4700  5 -- op tracker -- , seq: 747329, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143504 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.237100 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143504 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:55.237106 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:45:55.237114 7f9d5bec4700 15 osd.2 173 enqueue_op 0x124e3f00 prio 196 cost 0 latency 0.000113 osd_sub_op_reply(client.4296.0:143504 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:55.237145 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.237154 7f9d663dd700 10 osd.2 173 dequeue_op 0x124e3f00 prio 196 cost 0 latency 0.000154 osd_sub_op_reply(client.4296.0:143504 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]
2014-06-12 14:45:55.237179 7f9d663dd700  5 -- op tracker -- , seq: 747329, time: 2014-06-12 14:45:55.237175, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143504 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.237192 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] handle_message: 0x124e3f00
2014-06-12 14:45:55.237211 7f9d663dd700  5 -- op tracker -- , seq: 747329, time: 2014-06-12 14:45:55.237207, event: started, op: osd_sub_op_reply(client.4296.0:143504 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.237217 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] sub_op_modify_reply: tid 236146 op  ack_type 4 from 25
2014-06-12 14:45:55.237241 7f9d663dd700  5 -- op tracker -- , seq: 747328, time: 2014-06-12 14:45:55.237233, event: sub_op_commit_rec, op: osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.237251 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] repop_all_applied: repop tid 236146 all applied 
2014-06-12 14:45:55.237264 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] eval_repop repgather(0x1687d140 173'2728 rep_tid=236146 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.237282 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]  sending ack on repgather(0x1687d140 173'2728 rep_tid=236146 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) 0x5ffe280
2014-06-12 14:45:55.237316 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] repop_all_committed: repop tid 236146 all committed 
2014-06-12 14:45:55.237328 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2726 active+clean] eval_repop repgather(0x1687d140 173'2728 rep_tid=236146 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.237344 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2726 active+clean] log_op_stats osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.016052 lat 0.016080
2014-06-12 14:45:55.237360 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2726 active+clean] publish_stats_to_osd 173:3625
2014-06-12 14:45:55.237376 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2726 active+clean]  sending commit on repgather(0x1687d140 173'2728 rep_tid=236146 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) 0x56cb400
2014-06-12 14:45:55.237411 7f9d663dd700  5 -- op tracker -- , seq: 747328, time: 2014-06-12 14:45:55.237406, event: commit_sent, op: osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.237421 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2726 active+clean]  requeue_ops 
2014-06-12 14:45:55.237431 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  removing repgather(0x1687d140 173'2728 rep_tid=236146 committed?=1 applied?=1 op=osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.237447 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]    q front is repgather(0x1687d140 173'2728 rep_tid=236146 committed?=1 applied?=1 op=osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.237463 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] remove_repop repgather(0x1687d140 173'2728 rep_tid=236146 committed?=1 applied?=1 op=osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.237479 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  requeue_ops 
2014-06-12 14:45:55.237506 7f9d663dd700  5 -- op tracker -- , seq: 747328, time: 2014-06-12 14:45:55.237500, event: done, op: osd_op(client.4296.0:143504 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1974272~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.237526 7f9d663dd700 10 osd.2 173 dequeue_op 0x124e3f00 finish
2014-06-12 14:45:55.237536 7f9d663dd700  5 -- op tracker -- , seq: 747329, time: 2014-06-12 14:45:55.237532, event: done, op: osd_sub_op_reply(client.4296.0:143504 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.237546 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.237549 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.243547 7f9d4a3c5700  5 -- op tracker -- , seq: 747330, time: 2014-06-12 14:45:55.241351, event: header_read, op: osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.243565 7f9d4a3c5700  5 -- op tracker -- , seq: 747330, time: 2014-06-12 14:45:55.241354, event: throttled, op: osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.243576 7f9d4a3c5700  5 -- op tracker -- , seq: 747330, time: 2014-06-12 14:45:55.243493, event: all_read, op: osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.243586 7f9d4a3c5700  5 -- op tracker -- , seq: 747330, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.243595 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:45:55.243623 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x25e43b00 prio 63 cost 1032192 latency 0.002272 osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.243660 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.243672 7f9d67be0700 10 osd.2 173 dequeue_op 0x25e43b00 prio 63 cost 1032192 latency 0.002321 osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]
2014-06-12 14:45:55.243703 7f9d67be0700  5 -- op tracker -- , seq: 747330, time: 2014-06-12 14:45:55.243698, event: reached_pg, op: osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.243716 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:55.243734 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] handle_message: 0x25e43b00
2014-06-12 14:45:55.243752 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_op osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:55.243778 7f9d67be0700 15 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 '_'
2014-06-12 14:45:55.249878 7f9d67be0700 10 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 '_' = 0
2014-06-12 14:45:55.249915 7f9d67be0700 15 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 'snapset'
2014-06-12 14:45:55.250015 7f9d67be0700 10 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 'snapset' = 0
2014-06-12 14:45:55.250028 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] populate_obc_watchers 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.250049 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.250060 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] get_object_context: creating obc from disk: 0x23526840
2014-06-12 14:45:55.250070 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] get_object_context: 0x23526840 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 rwstate(none n=0 w=0) oi: 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3(173'2728 client.4296.0:143504 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2728) ssc: 0x72b9a40 snapset: 0=[]:[]+head
2014-06-12 14:45:55.250089 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] find_object_context 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 @head oi=5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3(173'2728 client.4296.0:143504 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2728)
2014-06-12 14:45:55.250121 7f9d67be0700  5 -- op tracker -- , seq: 747330, time: 2014-06-12 14:45:55.250112, event: started, op: osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.250133 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] execute_ctx 0x110c4a00
2014-06-12 14:45:55.250152 7f9d67be0700  5 -- op tracker -- , seq: 747330, time: 2014-06-12 14:45:55.250146, event: started, op: osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.250161 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_op 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] ov 173'2728 av 173'2729 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:55.250178 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_osd_op 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192]
2014-06-12 14:45:55.250190 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:55.250204 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_osd_op  write 3006464~1032192
2014-06-12 14:45:55.250219 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:45:55.250233 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] make_writeable 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 snapset=0x72b9a78  snapc=0=[]
2014-06-12 14:45:55.250251 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] make_writeable 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:55.250265 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] finish_ctx 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 0x110c4a00 op modify  
2014-06-12 14:45:55.250278 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  set mtime to 2014-06-12 08:44:57.620789
2014-06-12 14:45:55.250318 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  final snapset 0=[]:[]+head in 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.250342 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  zeroing write result code 0
2014-06-12 14:45:55.250355 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] new_repop rep_tid 236147 on osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.250374 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] issue_repop rep_tid 236147 o 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.250400 7f9d67be0700  5 -- op tracker -- , seq: 747330, time: 2014-06-12 14:45:55.250395, event: waiting for subops from 25, op: osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.250425 7f9d67be0700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:45:55.250455 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2728 (0'0,173'2728] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] append_log log((0'0,173'2728], crt=0'0) [173'2729 (173'2728) modify   5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 by client.4296.0:143505 2014-06-12 08:44:57.620789]
2014-06-12 14:45:55.250482 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] add_log_entry 173'2729 (173'2728) modify   5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 by client.4296.0:143505 2014-06-12 08:44:57.620789
2014-06-12 14:45:55.250502 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] append_log  adding 1 keys
2014-06-12 14:45:55.250534 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2729, trimmed: 
2014-06-12 14:45:55.250558 7f9d67be0700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:45:55.250562 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.250564 7f9d67be0700  5 queue_op 0x67a1e60 seq 2541 osr(3.122 0x276cf40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:45:55.250576 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:45:55.250582 7f9d79960700  5 _do_op 0x67a1e60 seq 2541 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:45:55.250586 7f9d79960700 10 _do_transaction on 0x1ced3a40
2014-06-12 14:45:55.250572 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] eval_repop repgather(0x1c378d80 173'2729 rep_tid=236147 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.250589 7f9d67be0700 10 osd.2 173 dequeue_op 0x25e43b00 finish
2014-06-12 14:45:55.250592 7f9d79960700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.250594 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.250598 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.250709 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.250799 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.250854 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.250881 7f9d79960700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.250889 7f9d79960700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:45:55.250895 7f9d79960700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.250913 7f9d79960700 15 _write 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 3006464~1032192
2014-06-12 14:45:55.251131 7f9d79960700 10 lookup_strip_header done  cid 3.122_head oid 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.251240 7f9d79960700 10 file_to_extents done 
2014-06-12 14:45:55.254501 7f9d79960700 10 _generic_write 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 3006464~1032192 = 0
2014-06-12 14:45:55.254617 7f9d79960700 15 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.254642 7f9d79960700 10 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 = 0
2014-06-12 14:45:55.254648 7f9d79960700 15 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.254659 7f9d79960700 10 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 = 0
2014-06-12 14:45:55.254950 7f9d79960700 20 set_header setting 16931 cid 3.122_head oid 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 parent seq 0
2014-06-12 14:45:55.255057 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.255071 7f9d79960700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:45:55.255086 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:45:55.264753 7f9d79960700 10 _do_op 0x67a1e60 seq 2541 r = 0, finisher 0x8a00bc0 0x111fb800
2014-06-12 14:45:55.264780 7f9d79960700 10 _finish_op 0x67a1e60 seq 2541 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:45:55.264790 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:45:55.264800 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.264785 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] op_commit: 236147
2014-06-12 14:45:55.264819 7f9d7715b700  5 -- op tracker -- , seq: 747330, time: 2014-06-12 14:45:55.264810, event: op_commit, op: osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.264839 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] op_applied: 236147
2014-06-12 14:45:55.264868 7f9d7795c700  5 -- op tracker -- , seq: 747330, time: 2014-06-12 14:45:55.264861, event: op_applied, op: osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.264880 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] op_applied on primary on version 173'2729
2014-06-12 14:45:55.266274 7f9d5bec4700  5 -- op tracker -- , seq: 747331, time: 2014-06-12 14:45:55.266195, event: header_read, op: osd_sub_op_reply(client.4296.0:143505 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.266296 7f9d5bec4700  5 -- op tracker -- , seq: 747331, time: 2014-06-12 14:45:55.266198, event: throttled, op: osd_sub_op_reply(client.4296.0:143505 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.266305 7f9d5bec4700  5 -- op tracker -- , seq: 747331, time: 2014-06-12 14:45:55.266212, event: all_read, op: osd_sub_op_reply(client.4296.0:143505 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.266314 7f9d5bec4700  5 -- op tracker -- , seq: 747331, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143505 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.266320 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143505 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:55.266327 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:45:55.266337 7f9d5bec4700 15 osd.2 173 enqueue_op 0x1a16fd00 prio 196 cost 0 latency 0.000142 osd_sub_op_reply(client.4296.0:143505 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:55.266404 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.266417 7f9d693e3700 10 osd.2 173 dequeue_op 0x1a16fd00 prio 196 cost 0 latency 0.000222 osd_sub_op_reply(client.4296.0:143505 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]
2014-06-12 14:45:55.266440 7f9d693e3700  5 -- op tracker -- , seq: 747331, time: 2014-06-12 14:45:55.266437, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143505 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.266449 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] handle_message: 0x1a16fd00
2014-06-12 14:45:55.266466 7f9d693e3700  5 -- op tracker -- , seq: 747331, time: 2014-06-12 14:45:55.266463, event: started, op: osd_sub_op_reply(client.4296.0:143505 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.266473 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] sub_op_modify_reply: tid 236147 op  ack_type 4 from 25
2014-06-12 14:45:55.266494 7f9d693e3700  5 -- op tracker -- , seq: 747330, time: 2014-06-12 14:45:55.266486, event: sub_op_commit_rec, op: osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.266501 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] repop_all_applied: repop tid 236147 all applied 
2014-06-12 14:45:55.266512 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] eval_repop repgather(0x1c378d80 173'2729 rep_tid=236147 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.266528 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  sending ack on repgather(0x1c378d80 173'2729 rep_tid=236147 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) 0x16ad0000
2014-06-12 14:45:55.266562 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] repop_all_committed: repop tid 236147 all committed 
2014-06-12 14:45:55.266573 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2727 active+clean] eval_repop repgather(0x1c378d80 173'2729 rep_tid=236147 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.266590 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2727 active+clean] log_op_stats osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.025210 lat 0.025238
2014-06-12 14:45:55.266609 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2727 active+clean] publish_stats_to_osd 173:3626
2014-06-12 14:45:55.266622 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2727 active+clean]  sending commit on repgather(0x1c378d80 173'2729 rep_tid=236147 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) 0x20236f00
2014-06-12 14:45:55.266655 7f9d693e3700  5 -- op tracker -- , seq: 747330, time: 2014-06-12 14:45:55.266650, event: commit_sent, op: osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.266665 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2727 active+clean]  requeue_ops 
2014-06-12 14:45:55.266676 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]  removing repgather(0x1c378d80 173'2729 rep_tid=236147 committed?=1 applied?=1 op=osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.266690 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]    q front is repgather(0x1c378d80 173'2729 rep_tid=236147 committed?=1 applied?=1 op=osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.266704 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] remove_repop repgather(0x1c378d80 173'2729 rep_tid=236147 committed?=1 applied?=1 op=osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.266718 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]  requeue_ops 
2014-06-12 14:45:55.266744 7f9d693e3700  5 -- op tracker -- , seq: 747330, time: 2014-06-12 14:45:55.266739, event: done, op: osd_op(client.4296.0:143505 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3006464~1032192] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.266765 7f9d693e3700 10 osd.2 173 dequeue_op 0x1a16fd00 finish
2014-06-12 14:45:55.266772 7f9d693e3700  5 -- op tracker -- , seq: 747331, time: 2014-06-12 14:45:55.266769, event: done, op: osd_sub_op_reply(client.4296.0:143505 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.266782 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.266785 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.267211 7f9d5a16f700  5 -- op tracker -- , seq: 747332, time: 2014-06-12 14:45:55.267168, event: header_read, op: osd_sub_op_reply(client.4292.0:142324 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.267224 7f9d5a16f700  5 -- op tracker -- , seq: 747332, time: 2014-06-12 14:45:55.267171, event: throttled, op: osd_sub_op_reply(client.4292.0:142324 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.267232 7f9d5a16f700  5 -- op tracker -- , seq: 747332, time: 2014-06-12 14:45:55.267181, event: all_read, op: osd_sub_op_reply(client.4292.0:142324 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.267243 7f9d5a16f700  5 -- op tracker -- , seq: 747332, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4292.0:142324 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.267248 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4292.0:142324 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:55.267254 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:45:55.267261 7f9d5a16f700 15 osd.2 173 enqueue_op 0x124e3400 prio 196 cost 0 latency 0.000093 osd_sub_op_reply(client.4292.0:142324 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:55.267287 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.267294 7f9d62bd6700 10 osd.2 173 dequeue_op 0x124e3400 prio 196 cost 0 latency 0.000126 osd_sub_op_reply(client.4292.0:142324 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0) v2 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2563 crt=0'0 lcod 173'2562 mlcod 173'2562 active+clean]
2014-06-12 14:45:55.267316 7f9d62bd6700  5 -- op tracker -- , seq: 747332, time: 2014-06-12 14:45:55.267312, event: reached_pg, op: osd_sub_op_reply(client.4292.0:142324 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.267325 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2563 crt=0'0 lcod 173'2562 mlcod 173'2562 active+clean] handle_message: 0x124e3400
2014-06-12 14:45:55.267343 7f9d62bd6700  5 -- op tracker -- , seq: 747332, time: 2014-06-12 14:45:55.267339, event: started, op: osd_sub_op_reply(client.4292.0:142324 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.267349 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2563 crt=0'0 lcod 173'2562 mlcod 173'2562 active+clean] sub_op_modify_reply: tid 235895 op  ack_type 4 from 24
2014-06-12 14:45:55.267372 7f9d62bd6700  5 -- op tracker -- , seq: 746594, time: 2014-06-12 14:45:55.267364, event: sub_op_commit_rec, op: osd_op(client.4292.0:142324 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 2932736~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.267381 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2563 crt=0'0 lcod 173'2562 mlcod 173'2562 active+clean] repop_all_applied: repop tid 235895 all applied 
2014-06-12 14:45:55.267393 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2563 crt=0'0 lcod 173'2562 mlcod 173'2562 active+clean] eval_repop repgather(0x2ad98b40 173'2564 rep_tid=235895 committed?=0 applied?=1 lock=0 op=osd_op(client.4292.0:142324 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 2932736~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.267413 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2563 crt=0'0 lcod 173'2562 mlcod 173'2562 active+clean]  sending ack on repgather(0x2ad98b40 173'2564 rep_tid=235895 committed?=0 applied?=1 lock=0 op=osd_op(client.4292.0:142324 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 2932736~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4) 0x20271b80
2014-06-12 14:45:55.267448 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2563 crt=0'0 lcod 173'2562 mlcod 173'2562 active+clean] repop_all_committed: repop tid 235895 all committed 
2014-06-12 14:45:55.267457 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2562 active+clean] eval_repop repgather(0x2ad98b40 173'2564 rep_tid=235895 committed?=1 applied?=1 lock=0 op=osd_op(client.4292.0:142324 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 2932736~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.267475 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2562 active+clean] log_op_stats osd_op(client.4292.0:142324 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 2932736~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4 inb 516373 outb 0 rlat 32.476157 lat 32.476184
2014-06-12 14:45:55.267494 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2562 active+clean] publish_stats_to_osd 173:2745
2014-06-12 14:45:55.267510 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2562 active+clean]  sending commit on repgather(0x2ad98b40 173'2564 rep_tid=235895 committed?=1 applied?=1 lock=0 op=osd_op(client.4292.0:142324 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 2932736~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4) 0xfa08f00
2014-06-12 14:45:55.267544 7f9d62bd6700  5 -- op tracker -- , seq: 746594, time: 2014-06-12 14:45:55.267540, event: commit_sent, op: osd_op(client.4292.0:142324 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 2932736~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.267552 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2562 active+clean]  requeue_ops 
2014-06-12 14:45:55.267562 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean]  removing repgather(0x2ad98b40 173'2564 rep_tid=235895 committed?=1 applied?=1 op=osd_op(client.4292.0:142324 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 2932736~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.267578 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean]    q front is repgather(0x2ad98b40 173'2564 rep_tid=235895 committed?=1 applied?=1 op=osd_op(client.4292.0:142324 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 2932736~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.267594 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean] remove_repop repgather(0x2ad98b40 173'2564 rep_tid=235895 committed?=1 applied?=1 op=osd_op(client.4292.0:142324 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 2932736~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.267610 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean]  requeue_ops 
2014-06-12 14:45:55.267631 7f9d62bd6700  5 -- op tracker -- , seq: 746594, time: 2014-06-12 14:45:55.267626, event: done, op: osd_op(client.4292.0:142324 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 2932736~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.267650 7f9d62bd6700 10 osd.2 173 dequeue_op 0x124e3400 finish
2014-06-12 14:45:55.267659 7f9d62bd6700  5 -- op tracker -- , seq: 747332, time: 2014-06-12 14:45:55.267655, event: done, op: osd_sub_op_reply(client.4292.0:142324 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.267670 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.267673 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.269054 7f9d4a3c5700  5 -- op tracker -- , seq: 747333, time: 2014-06-12 14:45:55.268841, event: header_read, op: osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.269070 7f9d4a3c5700  5 -- op tracker -- , seq: 747333, time: 2014-06-12 14:45:55.268843, event: throttled, op: osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.269080 7f9d4a3c5700  5 -- op tracker -- , seq: 747333, time: 2014-06-12 14:45:55.269017, event: all_read, op: osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.269130 7f9d4a3c5700  5 -- op tracker -- , seq: 747333, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.269158 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:45:55.269236 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x3b42be00 prio 63 cost 155648 latency 0.000394 osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.269307 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.269315 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x3b42be00 prio 63 cost 155648 latency 0.000474 osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]
2014-06-12 14:45:55.269339 7f9d6b3e7700  5 -- op tracker -- , seq: 747333, time: 2014-06-12 14:45:55.269334, event: reached_pg, op: osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.269347 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:55.269360 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] handle_message: 0x3b42be00
2014-06-12 14:45:55.269369 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] do_op osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:55.269392 7f9d6b3e7700 15 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 '_'
2014-06-12 14:45:55.269632 7f9d6b3e7700 10 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 '_' = 0
2014-06-12 14:45:55.269652 7f9d6b3e7700 15 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 'snapset'
2014-06-12 14:45:55.269746 7f9d6b3e7700 10 getattr 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 'snapset' = 0
2014-06-12 14:45:55.269754 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] populate_obc_watchers 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.269766 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.269775 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] get_object_context: creating obc from disk: 0x37e4b8c0
2014-06-12 14:45:55.269783 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] get_object_context: 0x37e4b8c0 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 rwstate(none n=0 w=0) oi: 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3(173'2729 client.4296.0:143505 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2729) ssc: 0x93447e0 snapset: 0=[]:[]+head
2014-06-12 14:45:55.269804 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] find_object_context 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 @head oi=5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3(173'2729 client.4296.0:143505 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2729)
2014-06-12 14:45:55.269829 7f9d6b3e7700  5 -- op tracker -- , seq: 747333, time: 2014-06-12 14:45:55.269824, event: started, op: osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.269840 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] execute_ctx 0x27bc0c00
2014-06-12 14:45:55.269856 7f9d6b3e7700  5 -- op tracker -- , seq: 747333, time: 2014-06-12 14:45:55.269851, event: started, op: osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.269861 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] do_op 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] ov 173'2729 av 173'2730 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:55.269874 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] do_osd_op 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648]
2014-06-12 14:45:55.269884 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:55.269896 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] do_osd_op  write 4038656~155648
2014-06-12 14:45:55.269909 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:45:55.269919 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] make_writeable 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 snapset=0x9344818  snapc=0=[]
2014-06-12 14:45:55.269929 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] make_writeable 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:55.269938 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] finish_ctx 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 0x27bc0c00 op modify  
2014-06-12 14:45:55.269949 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]  set mtime to 2014-06-12 08:44:57.649372
2014-06-12 14:45:55.269968 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]  final snapset 0=[]:[]+head in 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.269984 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]  zeroing write result code 0
2014-06-12 14:45:55.269994 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] new_repop rep_tid 236148 on osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:55.270008 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] issue_repop rep_tid 236148 o 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.270029 7f9d6b3e7700  5 -- op tracker -- , seq: 747333, time: 2014-06-12 14:45:55.270025, event: waiting for subops from 25, op: osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.270047 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:45:55.270066 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2729 (0'0,173'2729] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] append_log log((0'0,173'2729], crt=0'0) [173'2730 (173'2729) modify   5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 by client.4296.0:143506 2014-06-12 08:44:57.649372]
2014-06-12 14:45:55.270087 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] add_log_entry 173'2730 (173'2729) modify   5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 by client.4296.0:143506 2014-06-12 08:44:57.649372
2014-06-12 14:45:55.270105 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] append_log  adding 1 keys
2014-06-12 14:45:55.270135 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2730, trimmed: 
2014-06-12 14:45:55.270155 7f9d6b3e7700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:45:55.270160 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.270161 7f9d6b3e7700  5 queue_op 0x790b560 seq 2542 osr(3.122 0x276cf40) 157941 bytes   (queue has 1 ops and 157941 bytes)
2014-06-12 14:45:55.270168 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] eval_repop repgather(0x1c832780 173'2730 rep_tid=236148 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.270185 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x3b42be00 finish
2014-06-12 14:45:55.270188 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.270195 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.270203 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:45:55.270208 7f9d8016d700  5 _do_op 0x790b560 seq 2542 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:45:55.270212 7f9d8016d700 10 _do_transaction on 0x96db140
2014-06-12 14:45:55.270218 7f9d8016d700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.270320 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.270338 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.270390 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.270491 7f9d8016d700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.270499 7f9d8016d700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:45:55.270504 7f9d8016d700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:45:55.270523 7f9d8016d700 15 _write 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 4038656~155648
2014-06-12 14:45:55.270663 7f9d8016d700 10 lookup_strip_header done  cid 3.122_head oid 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.270826 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:45:55.275336 7f9d8016d700 10 _generic_write 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 4038656~155648 = 0
2014-06-12 14:45:55.275404 7f9d8016d700 15 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.275419 7f9d8016d700 10 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 = 0
2014-06-12 14:45:55.275425 7f9d8016d700 15 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3
2014-06-12 14:45:55.275439 7f9d8016d700 10 _setattrs 3.122_head/5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 = 0
2014-06-12 14:45:55.275584 7f9d8016d700 20 set_header setting 16931 cid 3.122_head oid 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 parent seq 0
2014-06-12 14:45:55.275622 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.275634 7f9d8016d700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:45:55.275643 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:45:55.276674 7f9d8016d700 10 _do_op 0x790b560 seq 2542 r = 0, finisher 0x75b0680 0x33fd4740
2014-06-12 14:45:55.276694 7f9d8016d700 10 _finish_op 0x790b560 seq 2542 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:45:55.276705 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:45:55.276710 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.276701 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] op_commit: 236148
2014-06-12 14:45:55.276733 7f9d7715b700  5 -- op tracker -- , seq: 747333, time: 2014-06-12 14:45:55.276724, event: op_commit, op: osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.277633 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] op_applied: 236148
2014-06-12 14:45:55.277666 7f9d7795c700  5 -- op tracker -- , seq: 747333, time: 2014-06-12 14:45:55.277657, event: op_applied, op: osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.280762 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] op_applied on primary on version 173'2730
2014-06-12 14:45:55.287224 7f9d5bec4700  5 -- op tracker -- , seq: 747334, time: 2014-06-12 14:45:55.287171, event: header_read, op: osd_sub_op_reply(client.4296.0:143506 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.287235 7f9d5bec4700  5 -- op tracker -- , seq: 747334, time: 2014-06-12 14:45:55.287173, event: throttled, op: osd_sub_op_reply(client.4296.0:143506 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.287240 7f9d5bec4700  5 -- op tracker -- , seq: 747334, time: 2014-06-12 14:45:55.287187, event: all_read, op: osd_sub_op_reply(client.4296.0:143506 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.287244 7f9d5bec4700  5 -- op tracker -- , seq: 747334, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143506 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.287248 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143506 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:55.287252 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:45:55.287258 7f9d5bec4700 15 osd.2 173 enqueue_op 0x5289700 prio 196 cost 0 latency 0.000087 osd_sub_op_reply(client.4296.0:143506 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:55.287280 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.287287 7f9d64bda700 10 osd.2 173 dequeue_op 0x5289700 prio 196 cost 0 latency 0.000116 osd_sub_op_reply(client.4296.0:143506 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]
2014-06-12 14:45:55.287328 7f9d64bda700  5 -- op tracker -- , seq: 747334, time: 2014-06-12 14:45:55.287325, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143506 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.287334 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] handle_message: 0x5289700
2014-06-12 14:45:55.287344 7f9d64bda700  5 -- op tracker -- , seq: 747334, time: 2014-06-12 14:45:55.287342, event: started, op: osd_sub_op_reply(client.4296.0:143506 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.287348 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] sub_op_modify_reply: tid 236148 op  ack_type 4 from 25
2014-06-12 14:45:55.287361 7f9d64bda700  5 -- op tracker -- , seq: 747333, time: 2014-06-12 14:45:55.287356, event: sub_op_commit_rec, op: osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.287365 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] repop_all_applied: repop tid 236148 all applied 
2014-06-12 14:45:55.287371 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] eval_repop repgather(0x1c832780 173'2730 rep_tid=236148 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.287381 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]  sending ack on repgather(0x1c832780 173'2730 rep_tid=236148 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) 0x26fdfb80
2014-06-12 14:45:55.287402 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] repop_all_committed: repop tid 236148 all committed 
2014-06-12 14:45:55.287408 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2728 active+clean] eval_repop repgather(0x1c832780 173'2730 rep_tid=236148 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:55.287418 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2728 active+clean] log_op_stats osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4 inb 155927 outb 0 rlat 0.018560 lat 0.018576
2014-06-12 14:45:55.287428 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2728 active+clean] publish_stats_to_osd 173:3627
2014-06-12 14:45:55.287436 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2728 active+clean]  sending commit on repgather(0x1c832780 173'2730 rep_tid=236148 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4) 0x5ffa000
2014-06-12 14:45:55.287455 7f9d64bda700  5 -- op tracker -- , seq: 747333, time: 2014-06-12 14:45:55.287452, event: commit_sent, op: osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.287460 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2728 active+clean]  requeue_ops 
2014-06-12 14:45:55.287467 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean]  removing repgather(0x1c832780 173'2730 rep_tid=236148 committed?=1 applied?=1 op=osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.287475 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean]    q front is repgather(0x1c832780 173'2730 rep_tid=236148 committed?=1 applied?=1 op=osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.287483 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] remove_repop repgather(0x1c832780 173'2730 rep_tid=236148 committed?=1 applied?=1 op=osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:55.287491 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean]  requeue_ops 
2014-06-12 14:45:55.287508 7f9d64bda700  5 -- op tracker -- , seq: 747333, time: 2014-06-12 14:45:55.287505, event: done, op: osd_op(client.4296.0:143506 rbd_data.109f6b8b4567.00000000000011a3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4038656~155648] 3.5e695922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:55.287619 7f9d64bda700 10 osd.2 173 dequeue_op 0x5289700 finish
2014-06-12 14:45:55.287625 7f9d64bda700  5 -- op tracker -- , seq: 747334, time: 2014-06-12 14:45:55.287623, event: done, op: osd_sub_op_reply(client.4296.0:143506 3.122 5e695922/rbd_data.109f6b8b4567.00000000000011a3/head//3 [] ondisk, result = 0)
2014-06-12 14:45:55.287632 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.287634 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.365107 7f9d5a16f700  5 -- op tracker -- , seq: 747335, time: 2014-06-12 14:45:55.362986, event: header_read, op: osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.365119 7f9d5a16f700  5 -- op tracker -- , seq: 747335, time: 2014-06-12 14:45:55.362988, event: throttled, op: osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.365126 7f9d5a16f700  5 -- op tracker -- , seq: 747335, time: 2014-06-12 14:45:55.365071, event: all_read, op: osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.365134 7f9d5a16f700  5 -- op tracker -- , seq: 747335, time: 0.000000, event: dispatched, op: osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.365138 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:55.365144 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:45:55.365151 7f9d5a16f700 15 osd.2 173 enqueue_op 0x124e2200 prio 127 cost 516903 latency 0.002165 osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.365178 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.365184 7f9d683e1700 10 osd.2 173 dequeue_op 0x124e2200 prio 127 cost 516903 latency 0.002198 osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2714 (0'0,173'2714] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2713 active]
2014-06-12 14:45:55.365206 7f9d683e1700  5 -- op tracker -- , seq: 747335, time: 2014-06-12 14:45:55.365202, event: reached_pg, op: osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.365214 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2714 (0'0,173'2714] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2713 active] handle_message: 0x124e2200
2014-06-12 14:45:55.365226 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2714 (0'0,173'2714] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2713 active] sub_op_modify trans e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 v 173'2715 (transaction) 174
2014-06-12 14:45:55.365243 7f9d683e1700  5 -- op tracker -- , seq: 747335, time: 2014-06-12 14:45:55.365239, event: started, op: osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.365258 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2714 (0'0,173'2714] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2713 active] append_log log((0'0,173'2714], crt=0'0) [173'2715 (173'2714) modify   e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 by client.4285.0:143302 2014-06-12 08:42:54.041492]
2014-06-12 14:45:55.365281 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2715 (0'0,173'2715] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2713 active] add_log_entry 173'2715 (173'2714) modify   e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 by client.4285.0:143302 2014-06-12 08:42:54.041492
2014-06-12 14:45:55.365303 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2715 (0'0,173'2715] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2713 active] append_log  adding 1 keys
2014-06-12 14:45:55.365336 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2715, trimmed: 
2014-06-12 14:45:55.365358 7f9d683e1700  5 -- op tracker -- , seq: 747335, time: 2014-06-12 14:45:55.365353, event: started, op: osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.365367 7f9d683e1700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:45:55.365371 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.365373 7f9d683e1700  5 queue_op 0x568e840 seq 2462 osr(3.21c 0x276d020) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:45:55.365380 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2715 (0'0,173'2715] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2713 active] do_sub_op osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.365394 7f9d683e1700 10 osd.2 173 dequeue_op 0x124e2200 finish
2014-06-12 14:45:55.365398 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.365400 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.365408 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (1 active)
2014-06-12 14:45:55.365413 7f9d8096e700  5 _do_op 0x568e840 seq 2462 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:45:55.365417 7f9d8096e700 10 _do_transaction on 0x59a1b30
2014-06-12 14:45:55.365423 7f9d8096e700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:45:55.365558 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:45:55.365574 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.365628 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.365648 7f9d8096e700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:45:55.365653 7f9d8096e700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:45:55.365656 7f9d8096e700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:45:55.365839 7f9d8096e700 15 _write 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 1417216~516096
2014-06-12 14:45:55.365976 7f9d8096e700 10 lookup_strip_header done  cid 3.21c_head oid e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:45:55.366076 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:45:55.366205 7f9d56432700  5 -- op tracker -- , seq: 747336, time: 2014-06-12 14:45:55.365628, event: header_read, op: osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.366217 7f9d56432700  5 -- op tracker -- , seq: 747336, time: 2014-06-12 14:45:55.365631, event: throttled, op: osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.366230 7f9d56432700  5 -- op tracker -- , seq: 747336, time: 2014-06-12 14:45:55.366175, event: all_read, op: osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.366238 7f9d56432700  5 -- op tracker -- , seq: 747336, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.366243 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:55.366248 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:55.366255 7f9d56432700 15 osd.2 173 enqueue_op 0x3b42a300 prio 127 cost 516899 latency 0.000626 osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.366764 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.366775 7f9d65bdc700 10 osd.2 173 dequeue_op 0x3b42a300 prio 127 cost 516899 latency 0.001146 osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2462 (0'0,173'2462] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2461 active]
2014-06-12 14:45:55.366803 7f9d65bdc700  5 -- op tracker -- , seq: 747336, time: 2014-06-12 14:45:55.366799, event: reached_pg, op: osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.366813 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2462 (0'0,173'2462] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2461 active] handle_message: 0x3b42a300
2014-06-12 14:45:55.366824 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2462 (0'0,173'2462] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2461 active] sub_op_modify trans cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 v 173'2463 (transaction) 174
2014-06-12 14:45:55.366841 7f9d65bdc700  5 -- op tracker -- , seq: 747336, time: 2014-06-12 14:45:55.366837, event: started, op: osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.366857 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2462 (0'0,173'2462] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2461 active] append_log log((0'0,173'2462], crt=0'0) [173'2463 (173'2461) modify   cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 by client.4308.0:144509 2014-06-12 08:43:34.091399]
2014-06-12 14:45:55.366883 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2463 (0'0,173'2463] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2461 active] add_log_entry 173'2463 (173'2461) modify   cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 by client.4308.0:144509 2014-06-12 08:43:34.091399
2014-06-12 14:45:55.366905 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2463 (0'0,173'2463] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2461 active] append_log  adding 1 keys
2014-06-12 14:45:55.366938 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2463, trimmed: 
2014-06-12 14:45:55.366959 7f9d65bdc700  5 -- op tracker -- , seq: 747336, time: 2014-06-12 14:45:55.366955, event: started, op: osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.366969 7f9d65bdc700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:45:55.366973 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.366974 7f9d65bdc700  5 queue_op 0x124505a0 seq 2276 osr(3.2b 0x276cd20) 518380 bytes   (queue has 2 ops and 1036769 bytes)
2014-06-12 14:45:55.366982 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2463 (0'0,173'2463] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2461 active] do_sub_op osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.366996 7f9d65bdc700 10 osd.2 173 dequeue_op 0x3b42a300 finish
2014-06-12 14:45:55.366999 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.367002 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.367704 7f9d8096e700 10 _generic_write 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 1417216~516096 = 0
2014-06-12 14:45:55.367748 7f9d8096e700 15 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:45:55.367762 7f9d8096e700 10 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 = 0
2014-06-12 14:45:55.367774 7f9d8096e700 15 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:45:55.367782 7f9d8096e700 10 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 = 0
2014-06-12 14:45:55.367996 7f9d8096e700 20 set_header setting 17494 cid 3.21c_head oid e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 parent seq 0
2014-06-12 14:45:55.368077 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.368093 7f9d8096e700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:45:55.368103 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:45:55.372500 7f9d8096e700 10 _do_op 0x568e840 seq 2462 r = 0, finisher 0x73f88e0 0
2014-06-12 14:45:55.372577 7f9d7715b700  5 -- op tracker -- , seq: 747335, time: 2014-06-12 14:45:55.372567, event: commit_sent, op: osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.372586 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2715 (0'0,173'2715] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2713 active] sub_op_modify_commit on op osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:45:55.372614 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:45:55.372728 7f9d8096e700 10 _finish_op 0x568e840 seq 2462 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:45:55.372773 7f9d7795c700  5 -- op tracker -- , seq: 747335, time: 2014-06-12 14:45:55.372766, event: sub_op_applied, op: osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.372781 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2715 (0'0,173'2715] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2714 active] sub_op_modify_applied on 0x59a1a40 op osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.372798 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2715 (0'0,173'2715] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2714 active] op_applied on primary on version 173'2715
2014-06-12 14:45:55.372808 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2715 (0'0,173'2715] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2714 active] op_applied on replica on version 173'2715
2014-06-12 14:45:55.372883 7f9d8096e700  5 -- op tracker -- , seq: 747335, time: 2014-06-12 14:45:55.372873, event: done, op: osd_sub_op(client.4285.0:143302 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2715 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.372915 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (0 active)
2014-06-12 14:45:55.372918 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:45:55.372925 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.372972 7f9d8096e700  5 _do_op 0x124505a0 seq 2276 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:45:55.372978 7f9d8096e700 10 _do_transaction on 0x2720470
2014-06-12 14:45:55.372983 7f9d8096e700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:55.373176 7f9d8096e700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:55.373197 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.373262 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.373279 7f9d8096e700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:55.373285 7f9d8096e700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:45:55.373292 7f9d8096e700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:55.373306 7f9d8096e700 15 _write 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 1482752~516096
2014-06-12 14:45:55.373479 7f9d8096e700 10 lookup_strip_header done  cid 3.2b_head oid cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3
2014-06-12 14:45:55.373574 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:45:55.375689 7f9d8096e700 10 _generic_write 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 1482752~516096 = 0
2014-06-12 14:45:55.375759 7f9d8096e700 15 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3
2014-06-12 14:45:55.375779 7f9d8096e700 10 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 = 0
2014-06-12 14:45:55.375785 7f9d8096e700 15 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3
2014-06-12 14:45:55.375795 7f9d8096e700 10 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 = 0
2014-06-12 14:45:55.375943 7f9d8096e700 20 set_header setting 16771 cid 3.2b_head oid cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 parent seq 0
2014-06-12 14:45:55.376011 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.376028 7f9d8096e700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:45:55.376043 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:45:55.382056 7f9d8096e700 10 _do_op 0x124505a0 seq 2276 r = 0, finisher 0x6971ae0 0
2014-06-12 14:45:55.382111 7f9d8096e700 10 _finish_op 0x124505a0 seq 2276 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:45:55.382142 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:45:55.382166 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.382255 7f9d7715b700  5 -- op tracker -- , seq: 747336, time: 2014-06-12 14:45:55.382243, event: commit_sent, op: osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.382266 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2463 (0'0,173'2463] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2461 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:55.382297 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:55.382334 7f9d7795c700  5 -- op tracker -- , seq: 747336, time: 2014-06-12 14:45:55.382329, event: sub_op_applied, op: osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.382342 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2463 (0'0,173'2463] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2462 active] sub_op_modify_applied on 0x2720380 op osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.382358 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2463 (0'0,173'2463] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2462 active] op_applied on primary on version 173'2463
2014-06-12 14:45:55.382368 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2463 (0'0,173'2463] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2462 active] op_applied on replica on version 173'2463
2014-06-12 14:45:55.382395 7f9d7795c700  5 -- op tracker -- , seq: 747336, time: 2014-06-12 14:45:55.382391, event: done, op: osd_sub_op(client.4308.0:144509 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2463 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.389154 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:45:55.389477 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:45:55.415095 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:45:55.415437 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:45:55.463769 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:45:55.463796 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:45:55.463800 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:55.463801 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:55.463804 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 44; oldest is 32.922675 seconds old
2014-06-12 14:45:55.463802 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:45:55.536194 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:55.536233 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:55.547160 7f9d56432700  5 -- op tracker -- , seq: 747337, time: 2014-06-12 14:45:55.545298, event: header_read, op: osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.547177 7f9d56432700  5 -- op tracker -- , seq: 747337, time: 2014-06-12 14:45:55.545300, event: throttled, op: osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.547188 7f9d56432700  5 -- op tracker -- , seq: 747337, time: 2014-06-12 14:45:55.547111, event: all_read, op: osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.547198 7f9d56432700  5 -- op tracker -- , seq: 747337, time: 0.000000, event: dispatched, op: osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.547224 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:55.547236 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:55.547252 7f9d56432700 15 osd.2 173 enqueue_op 0x3b42a500 prio 127 cost 1032995 latency 0.001953 osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.547338 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.547350 7f9d6abe6700 10 osd.2 173 dequeue_op 0x3b42a500 prio 127 cost 1032995 latency 0.002051 osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2463 (0'0,173'2463] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2462 active]
2014-06-12 14:45:55.547380 7f9d6abe6700  5 -- op tracker -- , seq: 747337, time: 2014-06-12 14:45:55.547376, event: reached_pg, op: osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.547390 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2463 (0'0,173'2463] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2462 active] handle_message: 0x3b42a500
2014-06-12 14:45:55.547402 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2463 (0'0,173'2463] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2462 active] sub_op_modify trans 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 v 173'2464 (transaction) 174
2014-06-12 14:45:55.547420 7f9d6abe6700  5 -- op tracker -- , seq: 747337, time: 2014-06-12 14:45:55.547416, event: started, op: osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.547437 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2463 (0'0,173'2463] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2462 active] append_log log((0'0,173'2463], crt=0'0) [173'2464 (173'2462) modify   4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 by client.4280.0:139524 2014-06-12 08:43:17.934105]
2014-06-12 14:45:55.547461 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2464 (0'0,173'2464] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2462 active] add_log_entry 173'2464 (173'2462) modify   4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 by client.4280.0:139524 2014-06-12 08:43:17.934105
2014-06-12 14:45:55.547483 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2464 (0'0,173'2464] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2462 active] append_log  adding 1 keys
2014-06-12 14:45:55.547547 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2464, trimmed: 
2014-06-12 14:45:55.547576 7f9d6abe6700  5 -- op tracker -- , seq: 747337, time: 2014-06-12 14:45:55.547571, event: started, op: osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.547591 7f9d6abe6700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:45:55.547595 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.547597 7f9d6abe6700  5 queue_op 0x75c38c0 seq 2277 osr(3.2b 0x276cd20) 1034476 bytes   (queue has 1 ops and 1034476 bytes)
2014-06-12 14:45:55.547604 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2464 (0'0,173'2464] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2462 active] do_sub_op osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.547619 7f9d6abe6700 10 osd.2 173 dequeue_op 0x3b42a500 finish
2014-06-12 14:45:55.547622 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.547625 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.547634 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:45:55.547638 7f9d7915f700  5 _do_op 0x75c38c0 seq 2277 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:45:55.547643 7f9d7915f700 10 _do_transaction on 0x20a175f0
2014-06-12 14:45:55.547654 7f9d7915f700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:55.547833 7f9d7915f700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:55.547856 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.547922 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.547950 7f9d7915f700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:55.547955 7f9d7915f700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:45:55.547961 7f9d7915f700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:55.548077 7f9d7915f700 15 _write 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 385024~1032192
2014-06-12 14:45:55.548250 7f9d7915f700 10 lookup_strip_header done  cid 3.2b_head oid 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:45:55.548428 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:45:55.552705 7f9d7915f700 10 _generic_write 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 385024~1032192 = 0
2014-06-12 14:45:55.552823 7f9d7915f700 15 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:45:55.552851 7f9d7915f700 10 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 = 0
2014-06-12 14:45:55.552861 7f9d7915f700 15 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:45:55.552869 7f9d7915f700 10 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 = 0
2014-06-12 14:45:55.553170 7f9d7915f700 20 set_header setting 15771 cid 3.2b_head oid 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 parent seq 0
2014-06-12 14:45:55.553270 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.553283 7f9d7915f700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:45:55.553292 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:45:55.557392 7f9d7915f700 10 _do_op 0x75c38c0 seq 2277 r = 0, finisher 0x1b2d7ea0 0
2014-06-12 14:45:55.557411 7f9d7915f700 10 _finish_op 0x75c38c0 seq 2277 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:45:55.557416 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:45:55.557423 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.557429 7f9d7715b700  5 -- op tracker -- , seq: 747337, time: 2014-06-12 14:45:55.557419, event: commit_sent, op: osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.557438 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2464 (0'0,173'2464] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2462 active] sub_op_modify_commit on op osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:55.557465 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:55.557497 7f9d7795c700  5 -- op tracker -- , seq: 747337, time: 2014-06-12 14:45:55.557493, event: sub_op_applied, op: osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.557504 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2464 (0'0,173'2464] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2463 active] sub_op_modify_applied on 0x20a17500 op osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.557515 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2464 (0'0,173'2464] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2463 active] op_applied on primary on version 173'2464
2014-06-12 14:45:55.557521 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2464 (0'0,173'2464] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2463 active] op_applied on replica on version 173'2464
2014-06-12 14:45:55.557542 7f9d7795c700  5 -- op tracker -- , seq: 747337, time: 2014-06-12 14:45:55.557539, event: done, op: osd_sub_op(client.4280.0:139524 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2464 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.633398 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:55.633441 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:55.817007 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:55.817010 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:55.817011 7f9d74bfa700 20 osd.2 173 _dispatch 0x17704d00 ping v1
2014-06-12 14:45:55.817014 7f9d74bfa700 10 osd.2 173 ping from client.4303
2014-06-12 14:45:55.817016 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:55.817016 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:55.911698 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19405 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,2,2,3,33,4])
2014-06-12 14:45:55.911715 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19405 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,2,2,3,33,4])
2014-06-12 14:45:55.912118 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:45:55.912145 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:45:55.912161 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:55.912182 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:55.912204 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:45:55.912223 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:55.912242 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:45:55.912258 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:55.912273 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:45:55.912299 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:45:55.912369 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:55.912384 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:55.912424 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:45:55.912435 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:45:55.912446 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:45:55.912474 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:45:55.912555 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:55.912566 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:55.913990 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:55.914010 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:45:55.914028 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:55.914199 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:45:55.995893 7f9d56432700  5 -- op tracker -- , seq: 747338, time: 2014-06-12 14:45:55.995478, event: header_read, op: osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.995906 7f9d56432700  5 -- op tracker -- , seq: 747338, time: 2014-06-12 14:45:55.995482, event: throttled, op: osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.995915 7f9d56432700  5 -- op tracker -- , seq: 747338, time: 2014-06-12 14:45:55.995854, event: all_read, op: osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.995924 7f9d56432700  5 -- op tracker -- , seq: 747338, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.995929 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:55.995936 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:55.995944 7f9d56432700 15 osd.2 173 enqueue_op 0x3b42bd00 prio 127 cost 176935 latency 0.000466 osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.995971 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:55.995979 7f9d66bde700 10 osd.2 173 dequeue_op 0x3b42bd00 prio 127 cost 176935 latency 0.000501 osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.51e( v 173'2674 (0'0,173'2674] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2673 active]
2014-06-12 14:45:55.996003 7f9d66bde700  5 -- op tracker -- , seq: 747338, time: 2014-06-12 14:45:55.995998, event: reached_pg, op: osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.996012 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2674 (0'0,173'2674] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2673 active] handle_message: 0x3b42bd00
2014-06-12 14:45:55.996022 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2674 (0'0,173'2674] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2673 active] sub_op_modify trans 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 v 173'2675 (transaction) 174
2014-06-12 14:45:55.996049 7f9d66bde700  5 -- op tracker -- , seq: 747338, time: 2014-06-12 14:45:55.996044, event: started, op: osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.996064 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2674 (0'0,173'2674] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2673 active] append_log log((0'0,173'2674], crt=0'0) [173'2675 (173'695) modify   55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 by client.4294.0:143951 2014-06-12 08:44:53.123786]
2014-06-12 14:45:55.996089 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2675 (0'0,173'2675] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2673 active] add_log_entry 173'2675 (173'695) modify   55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 by client.4294.0:143951 2014-06-12 08:44:53.123786
2014-06-12 14:45:55.996110 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2675 (0'0,173'2675] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2673 active] append_log  adding 1 keys
2014-06-12 14:45:55.996143 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2675, trimmed: 
2014-06-12 14:45:55.996167 7f9d66bde700  5 -- op tracker -- , seq: 747338, time: 2014-06-12 14:45:55.996162, event: started, op: osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.996176 7f9d66bde700  5 queue_transactions existing osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:45:55.996180 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:55.996182 7f9d66bde700  5 queue_op 0x1295ed80 seq 2511 osr(3.51e 0x276db50) 178421 bytes   (queue has 1 ops and 178421 bytes)
2014-06-12 14:45:55.996189 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.51e( v 173'2675 (0'0,173'2675] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2673 active] do_sub_op osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.996203 7f9d66bde700 10 osd.2 173 dequeue_op 0x3b42bd00 finish
2014-06-12 14:45:55.996207 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:55.996209 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:45:55.996218 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fb00 (1 active)
2014-06-12 14:45:55.996227 7f9d7c165700  5 _do_op 0x1295ed80 seq 2511 osr(3.51e 0x276db50)/0x276db50 start
2014-06-12 14:45:55.996231 7f9d7c165700 10 _do_transaction on 0x668c9b0
2014-06-12 14:45:55.996237 7f9d7c165700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:45:55.996354 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:45:55.996370 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:55.996412 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:55.996431 7f9d7c165700 15 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:45:55.996438 7f9d7c165700 10 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1 r = 0
2014-06-12 14:45:55.996441 7f9d7c165700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:45:55.996450 7f9d7c165700 15 _write 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 0~176128
2014-06-12 14:45:55.996589 7f9d7c165700 10 lookup_strip_header done  cid 3.51e_head oid 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3
2014-06-12 14:45:55.996602 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:45:55.997050 7f9d7c165700 10 _generic_write 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 0~176128 = 0
2014-06-12 14:45:55.997065 7f9d7c165700 15 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3
2014-06-12 14:45:55.997072 7f9d7c165700 10 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 = 0
2014-06-12 14:45:55.997075 7f9d7c165700 15 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3
2014-06-12 14:45:55.997080 7f9d7c165700 10 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 = 0
2014-06-12 14:45:55.997168 7f9d7c165700 20 set_header setting 17621 cid 3.51e_head oid 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 parent seq 0
2014-06-12 14:45:55.997187 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:55.997196 7f9d7c165700 20 set_header setting 2288 cid meta oid 1fe1612b/pglog_3.51e/0//-1 parent seq 0
2014-06-12 14:45:55.997236 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:45:55.997952 7f9d7c165700 10 _do_op 0x1295ed80 seq 2511 r = 0, finisher 0x20aadb20 0
2014-06-12 14:45:55.997970 7f9d7c165700 10 _finish_op 0x1295ed80 seq 2511 osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:45:55.997976 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fb00 (0 active)
2014-06-12 14:45:55.997980 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:55.997993 7f9d7715b700  5 -- op tracker -- , seq: 747338, time: 2014-06-12 14:45:55.997982, event: commit_sent, op: osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.998012 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2675 (0'0,173'2675] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2673 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:55.998043 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:55.998089 7f9d7795c700  5 -- op tracker -- , seq: 747338, time: 2014-06-12 14:45:55.998084, event: sub_op_applied, op: osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:55.998099 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2675 (0'0,173'2675] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active] sub_op_modify_applied on 0x668c8c0 op osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:55.998116 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2675 (0'0,173'2675] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active] op_applied on primary on version 173'2675
2014-06-12 14:45:55.998133 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2675 (0'0,173'2675] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active] op_applied on replica on version 173'2675
2014-06-12 14:45:55.998160 7f9d7795c700  5 -- op tracker -- , seq: 747338, time: 2014-06-12 14:45:55.998156, event: done, op: osd_sub_op(client.4294.0:143951 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.034457 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:56.034520 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:56.106433 7f9d56432700  5 -- op tracker -- , seq: 747339, time: 2014-06-12 14:45:56.100590, event: header_read, op: osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.106453 7f9d56432700  5 -- op tracker -- , seq: 747339, time: 2014-06-12 14:45:56.100593, event: throttled, op: osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.106464 7f9d56432700  5 -- op tracker -- , seq: 747339, time: 2014-06-12 14:45:56.106379, event: all_read, op: osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.106472 7f9d56432700  5 -- op tracker -- , seq: 747339, time: 0.000000, event: dispatched, op: osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.106479 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:56.106486 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:56.106495 7f9d56432700 15 osd.2 173 enqueue_op 0x2d17a100 prio 127 cost 1549095 latency 0.005904 osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.107320 7f9d56432700  5 -- op tracker -- , seq: 747340, time: 2014-06-12 14:45:56.106529, event: header_read, op: osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.107333 7f9d56432700  5 -- op tracker -- , seq: 747340, time: 2014-06-12 14:45:56.106530, event: throttled, op: osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.107342 7f9d56432700  5 -- op tracker -- , seq: 747340, time: 2014-06-12 14:45:56.107291, event: all_read, op: osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.107347 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.107353 7f9d56432700  5 -- op tracker -- , seq: 747340, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.107357 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:56.107362 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:56.107355 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2d17a100 prio 127 cost 1549095 latency 0.006764 osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2982 (0'0,173'2982] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2981 active]
2014-06-12 14:45:56.107368 7f9d56432700 15 osd.2 173 enqueue_op 0x2d17bd00 prio 127 cost 516899 latency 0.000839 osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.107377 7f9d63bd8700  5 -- op tracker -- , seq: 747339, time: 2014-06-12 14:45:56.107372, event: reached_pg, op: osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.107394 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:45:56.107389 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2982 (0'0,173'2982] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2981 active] handle_message: 0x2d17a100
2014-06-12 14:45:56.107402 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2982 (0'0,173'2982] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2981 active] sub_op_modify trans 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 v 173'2983 (transaction) 174
2014-06-12 14:45:56.107398 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2d17bd00 prio 127 cost 516899 latency 0.000869 osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2464 (0'0,173'2464] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2463 active]
2014-06-12 14:45:56.107418 7f9d6a3e5700  5 -- op tracker -- , seq: 747340, time: 2014-06-12 14:45:56.107414, event: reached_pg, op: osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.107427 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2464 (0'0,173'2464] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2463 active] handle_message: 0x2d17bd00
2014-06-12 14:45:56.107440 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2464 (0'0,173'2464] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2463 active] sub_op_modify trans cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 v 173'2465 (transaction) 174
2014-06-12 14:45:56.107457 7f9d6a3e5700  5 -- op tracker -- , seq: 747340, time: 2014-06-12 14:45:56.107453, event: started, op: osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.107479 7f9d63bd8700  5 -- op tracker -- , seq: 747339, time: 2014-06-12 14:45:56.107414, event: started, op: osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.107474 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2464 (0'0,173'2464] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2463 active] append_log log((0'0,173'2464], crt=0'0) [173'2465 (173'2463) modify   cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 by client.4308.0:144510 2014-06-12 08:43:36.349168]
2014-06-12 14:45:56.107495 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2982 (0'0,173'2982] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2981 active] append_log log((0'0,173'2982], crt=0'0) [173'2983 (173'2982) modify   6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 by client.4307.0:146567 2014-06-12 08:44:32.676616]
2014-06-12 14:45:56.107499 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2465 (0'0,173'2465] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2463 active] add_log_entry 173'2465 (173'2463) modify   cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 by client.4308.0:144510 2014-06-12 08:43:36.349168
2014-06-12 14:45:56.107525 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2465 (0'0,173'2465] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2463 active] append_log  adding 1 keys
2014-06-12 14:45:56.107521 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2983 (0'0,173'2983] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2981 active] add_log_entry 173'2983 (173'2982) modify   6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 by client.4307.0:146567 2014-06-12 08:44:32.676616
2014-06-12 14:45:56.107541 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2983 (0'0,173'2983] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2981 active] append_log  adding 1 keys
2014-06-12 14:45:56.107558 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2465, trimmed: 
2014-06-12 14:45:56.107569 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2983, trimmed: 
2014-06-12 14:45:56.107581 7f9d6a3e5700  5 -- op tracker -- , seq: 747340, time: 2014-06-12 14:45:56.107576, event: started, op: osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.107596 7f9d6a3e5700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:45:56.107600 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.107601 7f9d6a3e5700  5 queue_op 0x6ceb4a0 seq 2278 osr(3.2b 0x276cd20) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:45:56.107597 7f9d63bd8700  5 -- op tracker -- , seq: 747339, time: 2014-06-12 14:45:56.107583, event: started, op: osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.107608 7f9d63bd8700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:45:56.107611 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.107612 7f9d63bd8700  5 queue_op 0x177b4ba0 seq 2785 osr(3.458 0x276d960) 1550581 bytes   (queue has 2 ops and 2068961 bytes)
2014-06-12 14:45:56.107608 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2465 (0'0,173'2465] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2463 active] do_sub_op osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.107623 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2d17bd00 finish
2014-06-12 14:45:56.107626 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:45:56.107628 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.107617 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2983 (0'0,173'2983] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2981 active] do_sub_op osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.107630 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2d17a100 finish
2014-06-12 14:45:56.107633 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.107635 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.107661 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:45:56.107666 7f9d82972700  5 _do_op 0x6ceb4a0 seq 2278 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:45:56.107669 7f9d82972700 10 _do_transaction on 0x62949b0
2014-06-12 14:45:56.107673 7f9d82972700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:56.107852 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (2 active)
2014-06-12 14:45:56.107860 7f9d7a161700  5 _do_op 0x177b4ba0 seq 2785 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:45:56.107863 7f9d7a161700 10 _do_transaction on 0x2963eb0
2014-06-12 14:45:56.107869 7f9d7a161700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:45:56.107927 7f9d82972700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:56.107944 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.108032 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:45:56.108050 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.108082 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.108101 7f9d82972700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:56.108106 7f9d82972700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:45:56.108110 7f9d82972700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:56.108122 7f9d82972700 15 _write 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 1998848~516096
2014-06-12 14:45:56.108237 7f9d82972700 10 lookup_strip_header done  cid 3.2b_head oid cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3
2014-06-12 14:45:56.108401 7f9d82972700 10 file_to_extents done 
2014-06-12 14:45:56.110203 7f9d82972700 10 _generic_write 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 1998848~516096 = 0
2014-06-12 14:45:56.110271 7f9d82972700 15 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3
2014-06-12 14:45:56.110287 7f9d82972700 10 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 = 0
2014-06-12 14:45:56.110291 7f9d82972700 15 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3
2014-06-12 14:45:56.110296 7f9d82972700 10 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 = 0
2014-06-12 14:45:56.110449 7f9d82972700 20 set_header setting 16771 cid 3.2b_head oid cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 parent seq 0
2014-06-12 14:45:56.110507 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.110518 7f9d82972700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:45:56.110525 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:45:56.113233 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.113274 7f9d7a161700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:45:56.113274 7f9d82972700 10 _do_op 0x6ceb4a0 seq 2278 r = 0, finisher 0x3769760 0
2014-06-12 14:45:56.113284 7f9d7a161700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:45:56.113289 7f9d82972700 10 _finish_op 0x6ceb4a0 seq 2278 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:45:56.113297 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (1 active)
2014-06-12 14:45:56.113301 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.113304 7f9d7715b700  5 -- op tracker -- , seq: 747340, time: 2014-06-12 14:45:56.113296, event: commit_sent, op: osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.113318 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2465 (0'0,173'2465] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2463 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:56.113343 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:56.113379 7f9d7795c700  5 -- op tracker -- , seq: 747340, time: 2014-06-12 14:45:56.113373, event: sub_op_applied, op: osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.113387 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2465 (0'0,173'2465] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2464 active] sub_op_modify_applied on 0x62948c0 op osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.113402 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2465 (0'0,173'2465] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2464 active] op_applied on primary on version 173'2465
2014-06-12 14:45:56.113412 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2465 (0'0,173'2465] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2464 active] op_applied on replica on version 173'2465
2014-06-12 14:45:56.113434 7f9d7795c700  5 -- op tracker -- , seq: 747340, time: 2014-06-12 14:45:56.113429, event: done, op: osd_sub_op(client.4308.0:144510 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2465 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.113461 7f9d7a161700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:45:56.113562 7f9d7a161700 15 _write 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 233472~1548288
2014-06-12 14:45:56.113737 7f9d7a161700 10 lookup_strip_header done  cid 3.458_head oid 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:45:56.113883 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:45:56.114711 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:45:56.115482 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:45:56.118755 7f9d7a161700 10 _generic_write 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 233472~1548288 = 0
2014-06-12 14:45:56.118849 7f9d7a161700 15 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:45:56.118865 7f9d7a161700 10 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 = 0
2014-06-12 14:45:56.118869 7f9d7a161700 15 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:45:56.118873 7f9d7a161700 10 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 = 0
2014-06-12 14:45:56.119119 7f9d7a161700 20 set_header setting 18434 cid 3.458_head oid 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 parent seq 0
2014-06-12 14:45:56.119210 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.119221 7f9d7a161700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:45:56.119227 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:45:56.125490 7f9d7a161700 10 _do_op 0x177b4ba0 seq 2785 r = 0, finisher 0x89a3880 0
2014-06-12 14:45:56.125516 7f9d7a161700 10 _finish_op 0x177b4ba0 seq 2785 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:45:56.125532 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:45:56.125541 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.125535 7f9d7715b700  5 -- op tracker -- , seq: 747339, time: 2014-06-12 14:45:56.125525, event: commit_sent, op: osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.125545 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2983 (0'0,173'2983] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2981 active] sub_op_modify_commit on op osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:56.125574 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:56.125604 7f9d7795c700  5 -- op tracker -- , seq: 747339, time: 2014-06-12 14:45:56.125597, event: sub_op_applied, op: osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.125619 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2983 (0'0,173'2983] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2982 active] sub_op_modify_applied on 0x2963dc0 op osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.125638 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2983 (0'0,173'2983] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2982 active] op_applied on primary on version 173'2983
2014-06-12 14:45:56.125648 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2983 (0'0,173'2983] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2982 active] op_applied on replica on version 173'2983
2014-06-12 14:45:56.125671 7f9d7795c700  5 -- op tracker -- , seq: 747339, time: 2014-06-12 14:45:56.125667, event: done, op: osd_sub_op(client.4307.0:146567 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2983 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.152680 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:45:56.215586 7f9d4ed71700  5 -- op tracker -- , seq: 747341, time: 2014-06-12 14:45:56.214780, event: header_read, op: osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.215609 7f9d4ed71700  5 -- op tracker -- , seq: 747341, time: 2014-06-12 14:45:56.214783, event: throttled, op: osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.215619 7f9d4ed71700  5 -- op tracker -- , seq: 747341, time: 2014-06-12 14:45:56.215549, event: all_read, op: osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.215629 7f9d4ed71700  5 -- op tracker -- , seq: 747341, time: 0.000000, event: dispatched, op: osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.215636 7f9d4ed71700 20 osd.2 173 should_share_map client.4291 172.16.96.220:0/1241802 173
2014-06-12 14:45:56.215659 7f9d4ed71700 15 osd.2 173 enqueue_op 0x876f800 prio 63 cost 356352 latency 0.000879 osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.215703 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.215713 7f9d653db700 10 osd.2 173 dequeue_op 0x876f800 prio 63 cost 356352 latency 0.000933 osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean]
2014-06-12 14:45:56.215742 7f9d653db700  5 -- op tracker -- , seq: 747341, time: 2014-06-12 14:45:56.215737, event: reached_pg, op: osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.215754 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:56.215768 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] handle_message: 0x876f800
2014-06-12 14:45:56.215779 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] do_op osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:56.215803 7f9d653db700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_'
2014-06-12 14:45:56.216099 7f9d653db700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_' = 0
2014-06-12 14:45:56.216113 7f9d653db700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset'
2014-06-12 14:45:56.216219 7f9d653db700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset' = 0
2014-06-12 14:45:56.216229 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] populate_obc_watchers d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.216242 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.216254 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] get_object_context: creating obc from disk: 0x1b8cb00
2014-06-12 14:45:56.216265 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] get_object_context: 0x1b8cb00 d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 rwstate(none n=0 w=0) oi: d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'576 client.4291.0:36641 wrlock_by=unknown.0.0:0 dirty s 4194304 uv576) ssc: 0x2ef8e2a0 snapset: 0=[]:[]+head
2014-06-12 14:45:56.216281 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] find_object_context d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 @head oi=d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'576 client.4291.0:36641 wrlock_by=unknown.0.0:0 dirty s 4194304 uv576)
2014-06-12 14:45:56.216305 7f9d653db700  5 -- op tracker -- , seq: 747341, time: 2014-06-12 14:45:56.216299, event: started, op: osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.216315 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] execute_ctx 0x2963d000
2014-06-12 14:45:56.216334 7f9d653db700  5 -- op tracker -- , seq: 747341, time: 2014-06-12 14:45:56.216329, event: started, op: osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.216343 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] do_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] ov 173'576 av 173'2451 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:56.216358 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] do_osd_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352]
2014-06-12 14:45:56.216370 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:56.216383 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] do_osd_op  write 0~356352
2014-06-12 14:45:56.216397 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] do_osd_op_effects on session 0xdd06400
2014-06-12 14:45:56.216409 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 snapset=0x2ef8e2d8  snapc=0=[]
2014-06-12 14:45:56.216423 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:56.216434 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] finish_ctx d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 0x2963d000 op modify  
2014-06-12 14:45:56.216448 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean]  set mtime to 2014-06-12 08:44:58.595434
2014-06-12 14:45:56.216469 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean]  final snapset 0=[]:[]+head in d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.216486 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean]  zeroing write result code 0
2014-06-12 14:45:56.216497 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] new_repop rep_tid 236149 on osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.216514 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] issue_repop rep_tid 236149 o d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.216537 7f9d653db700  5 -- op tracker -- , seq: 747341, time: 2014-06-12 14:45:56.216532, event: waiting for subops from 21, op: osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.216558 7f9d653db700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:56.216581 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2450 (0'0,173'2450] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] append_log log((0'0,173'2450], crt=0'0) [173'2451 (173'576) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140615 2014-06-12 08:44:58.595434]
2014-06-12 14:45:56.216605 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] add_log_entry 173'2451 (173'576) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140615 2014-06-12 08:44:58.595434
2014-06-12 14:45:56.216626 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] append_log  adding 1 keys
2014-06-12 14:45:56.216658 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2451, trimmed: 
2014-06-12 14:45:56.216685 7f9d653db700  5 queue_transactions existing osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.216690 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.216692 7f9d653db700  5 queue_op 0x5a73080 seq 2308 osr(3.1bc 0x276d1b0) 358645 bytes   (queue has 1 ops and 358645 bytes)
2014-06-12 14:45:56.216698 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] eval_repop repgather(0x5146e40 173'2451 rep_tid=236149 committed?=0 applied?=0 lock=0 op=osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.216718 7f9d653db700 10 osd.2 173 dequeue_op 0x876f800 finish
2014-06-12 14:45:56.216723 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.216725 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.216735 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519320 (1 active)
2014-06-12 14:45:56.216740 7f9d81970700  5 _do_op 0x5a73080 seq 2308 osr(3.1bc 0x276d1b0)/0x276d1b0 start
2014-06-12 14:45:56.216744 7f9d81970700 10 _do_transaction on 0x5146a80
2014-06-12 14:45:56.216750 7f9d81970700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.216858 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.216882 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.216939 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.216963 7f9d81970700 15 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.216971 7f9d81970700 10 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1 r = 0
2014-06-12 14:45:56.216977 7f9d81970700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.217118 7f9d81970700 15 _write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 0~356352
2014-06-12 14:45:56.217264 7f9d81970700 10 lookup_strip_header done  cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.217401 7f9d81970700 10 file_to_extents done 
2014-06-12 14:45:56.218655 7f9d81970700 10 _generic_write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 0~356352 = 0
2014-06-12 14:45:56.218700 7f9d81970700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.218715 7f9d81970700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.218724 7f9d81970700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.218733 7f9d81970700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.218912 7f9d81970700 20 set_header setting 16575 cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 parent seq 0
2014-06-12 14:45:56.218966 7f9d81970700 20 set_header setting 2006 cid meta oid 1fe11b06/pglog_3.1bc/0//-1 parent seq 0
2014-06-12 14:45:56.218984 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.218993 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:45:56.220933 7f9d81970700 10 _do_op 0x5a73080 seq 2308 r = 0, finisher 0x7eb9f80 0x33fd4800
2014-06-12 14:45:56.220948 7f9d81970700 10 _finish_op 0x5a73080 seq 2308 osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.220959 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519320 (0 active)
2014-06-12 14:45:56.220963 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.220953 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] op_commit: 236149
2014-06-12 14:45:56.220987 7f9d7715b700  5 -- op tracker -- , seq: 747341, time: 2014-06-12 14:45:56.220978, event: op_commit, op: osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.221003 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] op_applied: 236149
2014-06-12 14:45:56.221025 7f9d7795c700  5 -- op tracker -- , seq: 747341, time: 2014-06-12 14:45:56.221018, event: op_applied, op: osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.221037 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] op_applied on primary on version 173'2451
2014-06-12 14:45:56.222781 7f9d44101700  5 -- op tracker -- , seq: 747342, time: 2014-06-12 14:45:56.222732, event: header_read, op: osd_sub_op_reply(client.4291.0:140615 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.222791 7f9d44101700  5 -- op tracker -- , seq: 747342, time: 2014-06-12 14:45:56.222734, event: throttled, op: osd_sub_op_reply(client.4291.0:140615 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.222799 7f9d44101700  5 -- op tracker -- , seq: 747342, time: 2014-06-12 14:45:56.222748, event: all_read, op: osd_sub_op_reply(client.4291.0:140615 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.222806 7f9d44101700  5 -- op tracker -- , seq: 747342, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140615 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.222811 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140615 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.222818 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.222826 7f9d44101700 15 osd.2 173 enqueue_op 0x1e3eb700 prio 196 cost 0 latency 0.000094 osd_sub_op_reply(client.4291.0:140615 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.222858 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.222866 7f9d69be4700 10 osd.2 173 dequeue_op 0x1e3eb700 prio 196 cost 0 latency 0.000134 osd_sub_op_reply(client.4291.0:140615 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 pg pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean]
2014-06-12 14:45:56.222883 7f9d69be4700  5 -- op tracker -- , seq: 747342, time: 2014-06-12 14:45:56.222881, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140615 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.222891 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] handle_message: 0x1e3eb700
2014-06-12 14:45:56.222904 7f9d69be4700  5 -- op tracker -- , seq: 747342, time: 2014-06-12 14:45:56.222902, event: started, op: osd_sub_op_reply(client.4291.0:140615 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.222909 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] sub_op_modify_reply: tid 236149 op  ack_type 4 from 21
2014-06-12 14:45:56.222926 7f9d69be4700  5 -- op tracker -- , seq: 747341, time: 2014-06-12 14:45:56.222920, event: sub_op_commit_rec, op: osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.222932 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] repop_all_applied: repop tid 236149 all applied 
2014-06-12 14:45:56.222941 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] eval_repop repgather(0x5146e40 173'2451 rep_tid=236149 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.222955 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean]  sending ack on repgather(0x5146e40 173'2451 rep_tid=236149 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x25910280
2014-06-12 14:45:56.222982 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] repop_all_committed: repop tid 236149 all committed 
2014-06-12 14:45:56.222992 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2449 active+clean] eval_repop repgather(0x5146e40 173'2451 rep_tid=236149 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.223006 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2449 active+clean] log_op_stats osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 inb 356631 outb 0 rlat 0.008201 lat 0.008225
2014-06-12 14:45:56.223021 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2449 active+clean] publish_stats_to_osd 173:3174
2014-06-12 14:45:56.223032 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2449 active+clean]  sending commit on repgather(0x5146e40 173'2451 rep_tid=236149 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x3371c500
2014-06-12 14:45:56.223056 7f9d69be4700  5 -- op tracker -- , seq: 747341, time: 2014-06-12 14:45:56.223052, event: commit_sent, op: osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.223063 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2449 active+clean]  requeue_ops 
2014-06-12 14:45:56.223072 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]  removing repgather(0x5146e40 173'2451 rep_tid=236149 committed?=1 applied?=1 op=osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.223084 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]    q front is repgather(0x5146e40 173'2451 rep_tid=236149 committed?=1 applied?=1 op=osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.223100 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] remove_repop repgather(0x5146e40 173'2451 rep_tid=236149 committed?=1 applied?=1 op=osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.223111 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]  requeue_ops 
2014-06-12 14:45:56.223134 7f9d69be4700  5 -- op tracker -- , seq: 747341, time: 2014-06-12 14:45:56.223130, event: done, op: osd_op(client.4291.0:140615 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.223149 7f9d69be4700 10 osd.2 173 dequeue_op 0x1e3eb700 finish
2014-06-12 14:45:56.223155 7f9d69be4700  5 -- op tracker -- , seq: 747342, time: 2014-06-12 14:45:56.223152, event: done, op: osd_sub_op_reply(client.4291.0:140615 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.223163 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.223165 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.226901 7f9d4ed71700  5 -- op tracker -- , seq: 747343, time: 2014-06-12 14:45:56.225642, event: header_read, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.226926 7f9d4ed71700  5 -- op tracker -- , seq: 747343, time: 2014-06-12 14:45:56.225645, event: throttled, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.226944 7f9d4ed71700  5 -- op tracker -- , seq: 747343, time: 2014-06-12 14:45:56.226850, event: all_read, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.226965 7f9d4ed71700  5 -- op tracker -- , seq: 747343, time: 0.000000, event: dispatched, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.226974 7f9d4ed71700 20 osd.2 173 should_share_map client.4291 172.16.96.220:0/1241802 173
2014-06-12 14:45:56.227010 7f9d4ed71700 15 osd.2 173 enqueue_op 0x876e200 prio 63 cost 516096 latency 0.001368 osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.227052 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.227061 7f9d633d7700 10 osd.2 173 dequeue_op 0x876e200 prio 63 cost 516096 latency 0.001419 osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]
2014-06-12 14:45:56.227095 7f9d633d7700  5 -- op tracker -- , seq: 747343, time: 2014-06-12 14:45:56.227090, event: reached_pg, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.227112 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:56.227127 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] handle_message: 0x876e200
2014-06-12 14:45:56.227137 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] do_op osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:56.227164 7f9d633d7700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_'
2014-06-12 14:45:56.227405 7f9d633d7700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_' = 0
2014-06-12 14:45:56.227425 7f9d633d7700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset'
2014-06-12 14:45:56.227532 7f9d633d7700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset' = 0
2014-06-12 14:45:56.227544 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] populate_obc_watchers d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.227558 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.227569 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] get_object_context: creating obc from disk: 0x341c9340
2014-06-12 14:45:56.227581 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] get_object_context: 0x341c9340 d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 rwstate(none n=0 w=0) oi: d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2451 client.4291.0:140615 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2451) ssc: 0x679b420 snapset: 0=[]:[]+head
2014-06-12 14:45:56.227599 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] find_object_context d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 @head oi=d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2451 client.4291.0:140615 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2451)
2014-06-12 14:45:56.227628 7f9d633d7700  5 -- op tracker -- , seq: 747343, time: 2014-06-12 14:45:56.227622, event: started, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.227643 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] execute_ctx 0x55a4400
2014-06-12 14:45:56.227663 7f9d633d7700  5 -- op tracker -- , seq: 747343, time: 2014-06-12 14:45:56.227658, event: started, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.227672 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] do_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] ov 173'2451 av 173'2452 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:56.227689 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] do_osd_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096]
2014-06-12 14:45:56.227700 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:56.227714 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] do_osd_op  write 356352~516096
2014-06-12 14:45:56.227731 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] do_osd_op_effects on session 0xdd06400
2014-06-12 14:45:56.227743 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 snapset=0x679b458  snapc=0=[]
2014-06-12 14:45:56.227761 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:56.227774 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] finish_ctx d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 0x55a4400 op modify  
2014-06-12 14:45:56.227788 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]  set mtime to 2014-06-12 08:44:58.605993
2014-06-12 14:45:56.227812 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]  final snapset 0=[]:[]+head in d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.227832 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]  zeroing write result code 0
2014-06-12 14:45:56.227845 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] new_repop rep_tid 236150 on osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.227863 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] issue_repop rep_tid 236150 o d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.227890 7f9d633d7700  5 -- op tracker -- , seq: 747343, time: 2014-06-12 14:45:56.227885, event: waiting for subops from 21, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.227914 7f9d633d7700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:56.227938 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2451 (0'0,173'2451] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] append_log log((0'0,173'2451], crt=0'0) [173'2452 (173'2451) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140616 2014-06-12 08:44:58.605993]
2014-06-12 14:45:56.227964 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] add_log_entry 173'2452 (173'2451) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140616 2014-06-12 08:44:58.605993
2014-06-12 14:45:56.227985 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] append_log  adding 1 keys
2014-06-12 14:45:56.228017 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2452, trimmed: 
2014-06-12 14:45:56.228043 7f9d633d7700  5 queue_transactions existing osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.228047 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.228049 7f9d633d7700  5 queue_op 0x8856240 seq 2309 osr(3.1bc 0x276d1b0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:45:56.228056 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] eval_repop repgather(0x2f89e780 173'2452 rep_tid=236150 committed?=0 applied?=0 lock=0 op=osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.228075 7f9d633d7700 10 osd.2 173 dequeue_op 0x876e200 finish
2014-06-12 14:45:56.228079 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.228082 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.228092 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519320 (1 active)
2014-06-12 14:45:56.228097 7f9d7c966700  5 _do_op 0x8856240 seq 2309 osr(3.1bc 0x276d1b0)/0x276d1b0 start
2014-06-12 14:45:56.228100 7f9d7c966700 10 _do_transaction on 0x1771cfc0
2014-06-12 14:45:56.228107 7f9d7c966700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.228205 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.228225 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.228280 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.228303 7f9d7c966700 15 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.228309 7f9d7c966700 10 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1 r = 0
2014-06-12 14:45:56.228315 7f9d7c966700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.228333 7f9d7c966700 15 _write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 356352~516096
2014-06-12 14:45:56.228471 7f9d7c966700 10 lookup_strip_header done  cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.228630 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:45:56.230600 7f9d7c966700 10 _generic_write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 356352~516096 = 0
2014-06-12 14:45:56.230651 7f9d7c966700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.230667 7f9d7c966700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.230673 7f9d7c966700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.230682 7f9d7c966700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.230896 7f9d7c966700 20 set_header setting 16575 cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 parent seq 0
2014-06-12 14:45:56.230959 7f9d7c966700 20 set_header setting 2006 cid meta oid 1fe11b06/pglog_3.1bc/0//-1 parent seq 0
2014-06-12 14:45:56.230977 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.230986 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:45:56.233369 7f9d7c966700 10 _do_op 0x8856240 seq 2309 r = 0, finisher 0x59fbea0 0x3393e340
2014-06-12 14:45:56.233387 7f9d7c966700 10 _finish_op 0x8856240 seq 2309 osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.233395 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519320 (0 active)
2014-06-12 14:45:56.233404 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.233392 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] op_commit: 236150
2014-06-12 14:45:56.233426 7f9d7715b700  5 -- op tracker -- , seq: 747343, time: 2014-06-12 14:45:56.233416, event: op_commit, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.233445 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] op_applied: 236150
2014-06-12 14:45:56.233468 7f9d7795c700  5 -- op tracker -- , seq: 747343, time: 2014-06-12 14:45:56.233462, event: op_applied, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.233476 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] op_applied on primary on version 173'2452
2014-06-12 14:45:56.238921 7f9d44101700  5 -- op tracker -- , seq: 747344, time: 2014-06-12 14:45:56.238875, event: header_read, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.238933 7f9d44101700  5 -- op tracker -- , seq: 747344, time: 2014-06-12 14:45:56.238876, event: throttled, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.238940 7f9d44101700  5 -- op tracker -- , seq: 747344, time: 2014-06-12 14:45:56.238888, event: all_read, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.238947 7f9d44101700  5 -- op tracker -- , seq: 747344, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.238951 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:45:56.238956 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.238970 7f9d44101700 15 osd.2 173 enqueue_op 0x23e09900 prio 196 cost 0 latency 0.000095 osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0) v2
2014-06-12 14:45:56.238996 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.239002 7f9d623d5700 10 osd.2 173 dequeue_op 0x23e09900 prio 196 cost 0 latency 0.000127 osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0) v2 pg pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]
2014-06-12 14:45:56.239020 7f9d623d5700  5 -- op tracker -- , seq: 747344, time: 2014-06-12 14:45:56.239017, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.239027 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] handle_message: 0x23e09900
2014-06-12 14:45:56.239042 7f9d623d5700  5 -- op tracker -- , seq: 747344, time: 2014-06-12 14:45:56.239039, event: started, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.239047 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] sub_op_modify_reply: tid 236150 op  ack_type 1 from 21
2014-06-12 14:45:56.239071 7f9d623d5700  5 -- op tracker -- , seq: 747343, time: 2014-06-12 14:45:56.239062, event: sub_op_applied_rec, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.239078 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] repop_all_applied: repop tid 236150 all applied 
2014-06-12 14:45:56.239089 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] eval_repop repgather(0x2f89e780 173'2452 rep_tid=236150 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.239104 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]  sending ack on repgather(0x2f89e780 173'2452 rep_tid=236150 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x37812f00
2014-06-12 14:45:56.239133 7f9d623d5700 10 osd.2 173 dequeue_op 0x23e09900 finish
2014-06-12 14:45:56.239140 7f9d623d5700  5 -- op tracker -- , seq: 747344, time: 2014-06-12 14:45:56.239137, event: done, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.239153 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.239156 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.240483 7f9d44101700  5 -- op tracker -- , seq: 747345, time: 2014-06-12 14:45:56.240450, event: header_read, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.240498 7f9d44101700  5 -- op tracker -- , seq: 747345, time: 2014-06-12 14:45:56.240451, event: throttled, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.240505 7f9d44101700  5 -- op tracker -- , seq: 747345, time: 2014-06-12 14:45:56.240462, event: all_read, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.240513 7f9d44101700  5 -- op tracker -- , seq: 747345, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.240518 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.240534 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.240542 7f9d44101700 15 osd.2 173 enqueue_op 0x23e08f00 prio 196 cost 0 latency 0.000092 osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.240573 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.240584 7f9d643d9700 10 osd.2 173 dequeue_op 0x23e08f00 prio 196 cost 0 latency 0.000134 osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 pg pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]
2014-06-12 14:45:56.240608 7f9d643d9700  5 -- op tracker -- , seq: 747345, time: 2014-06-12 14:45:56.240604, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.240619 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] handle_message: 0x23e08f00
2014-06-12 14:45:56.240638 7f9d643d9700  5 -- op tracker -- , seq: 747345, time: 2014-06-12 14:45:56.240634, event: started, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.240646 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] sub_op_modify_reply: tid 236150 op  ack_type 4 from 21
2014-06-12 14:45:56.240668 7f9d643d9700  5 -- op tracker -- , seq: 747343, time: 2014-06-12 14:45:56.240661, event: sub_op_commit_rec, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.240678 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] repop_all_committed: repop tid 236150 all committed 
2014-06-12 14:45:56.240690 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2450 active+clean] eval_repop repgather(0x2f89e780 173'2452 rep_tid=236150 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.240710 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2450 active+clean] log_op_stats osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.013490 lat 0.015067
2014-06-12 14:45:56.240730 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2450 active+clean] publish_stats_to_osd 173:3175
2014-06-12 14:45:56.240745 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2450 active+clean]  sending commit on repgather(0x2f89e780 173'2452 rep_tid=236150 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0xee35900
2014-06-12 14:45:56.240785 7f9d643d9700  5 -- op tracker -- , seq: 747343, time: 2014-06-12 14:45:56.240778, event: commit_sent, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.240796 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2450 active+clean]  requeue_ops 
2014-06-12 14:45:56.240807 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean]  removing repgather(0x2f89e780 173'2452 rep_tid=236150 committed?=1 applied?=1 op=osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.240833 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean]    q front is repgather(0x2f89e780 173'2452 rep_tid=236150 committed?=1 applied?=1 op=osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.240850 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] remove_repop repgather(0x2f89e780 173'2452 rep_tid=236150 committed?=1 applied?=1 op=osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.240867 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean]  requeue_ops 
2014-06-12 14:45:56.240907 7f9d643d9700  5 -- op tracker -- , seq: 747343, time: 2014-06-12 14:45:56.240901, event: done, op: osd_op(client.4291.0:140616 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.240930 7f9d643d9700 10 osd.2 173 dequeue_op 0x23e08f00 finish
2014-06-12 14:45:56.240940 7f9d643d9700  5 -- op tracker -- , seq: 747345, time: 2014-06-12 14:45:56.240936, event: done, op: osd_sub_op_reply(client.4291.0:140616 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.240952 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.240955 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.244227 7f9d4ed71700  5 -- op tracker -- , seq: 747346, time: 2014-06-12 14:45:56.243011, event: header_read, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.244242 7f9d4ed71700  5 -- op tracker -- , seq: 747346, time: 2014-06-12 14:45:56.243015, event: throttled, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.244252 7f9d4ed71700  5 -- op tracker -- , seq: 747346, time: 2014-06-12 14:45:56.244171, event: all_read, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.244262 7f9d4ed71700  5 -- op tracker -- , seq: 747346, time: 0.000000, event: dispatched, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.244270 7f9d4ed71700 20 osd.2 173 should_share_map client.4291 172.16.96.220:0/1241802 173
2014-06-12 14:45:56.244299 7f9d4ed71700 15 osd.2 173 enqueue_op 0x876f500 prio 63 cost 516096 latency 0.001288 osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.244338 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.244347 7f9d68be2700 10 osd.2 173 dequeue_op 0x876f500 prio 63 cost 516096 latency 0.001335 osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean]
2014-06-12 14:45:56.244373 7f9d68be2700  5 -- op tracker -- , seq: 747346, time: 2014-06-12 14:45:56.244368, event: reached_pg, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.244383 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:56.244398 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] handle_message: 0x876f500
2014-06-12 14:45:56.244410 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] do_op osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:56.244436 7f9d68be2700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_'
2014-06-12 14:45:56.244672 7f9d68be2700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_' = 0
2014-06-12 14:45:56.244689 7f9d68be2700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset'
2014-06-12 14:45:56.244792 7f9d68be2700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset' = 0
2014-06-12 14:45:56.244803 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] populate_obc_watchers d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.244815 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.244825 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] get_object_context: creating obc from disk: 0x12d10840
2014-06-12 14:45:56.244835 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] get_object_context: 0x12d10840 d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 rwstate(none n=0 w=0) oi: d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2452 client.4291.0:140616 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2452) ssc: 0x93440e0 snapset: 0=[]:[]+head
2014-06-12 14:45:56.244851 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] find_object_context d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 @head oi=d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2452 client.4291.0:140616 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2452)
2014-06-12 14:45:56.244879 7f9d68be2700  5 -- op tracker -- , seq: 747346, time: 2014-06-12 14:45:56.244873, event: started, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.244892 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] execute_ctx 0x580a000
2014-06-12 14:45:56.244910 7f9d68be2700  5 -- op tracker -- , seq: 747346, time: 2014-06-12 14:45:56.244905, event: started, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.244917 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] do_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] ov 173'2452 av 173'2453 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:56.244931 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] do_osd_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096]
2014-06-12 14:45:56.244941 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:56.244953 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] do_osd_op  write 872448~516096
2014-06-12 14:45:56.244969 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] do_osd_op_effects on session 0xdd06400
2014-06-12 14:45:56.244980 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 snapset=0x9344118  snapc=0=[]
2014-06-12 14:45:56.244992 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:56.245003 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] finish_ctx d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 0x580a000 op modify  
2014-06-12 14:45:56.245015 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean]  set mtime to 2014-06-12 08:44:58.623364
2014-06-12 14:45:56.245037 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean]  final snapset 0=[]:[]+head in d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.245056 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean]  zeroing write result code 0
2014-06-12 14:45:56.245066 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] new_repop rep_tid 236151 on osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.245081 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] issue_repop rep_tid 236151 o d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.245107 7f9d68be2700  5 -- op tracker -- , seq: 747346, time: 2014-06-12 14:45:56.245101, event: waiting for subops from 21, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.245131 7f9d68be2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:56.245154 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2452 (0'0,173'2452] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] append_log log((0'0,173'2452], crt=0'0) [173'2453 (173'2452) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140617 2014-06-12 08:44:58.623364]
2014-06-12 14:45:56.245180 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] add_log_entry 173'2453 (173'2452) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140617 2014-06-12 08:44:58.623364
2014-06-12 14:45:56.245201 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] append_log  adding 1 keys
2014-06-12 14:45:56.245234 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2453, trimmed: 
2014-06-12 14:45:56.245258 7f9d68be2700  5 queue_transactions existing osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.245262 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.245263 7f9d68be2700  5 queue_op 0x694a7e0 seq 2310 osr(3.1bc 0x276d1b0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:45:56.245271 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] eval_repop repgather(0x53609c0 173'2453 rep_tid=236151 committed?=0 applied?=0 lock=0 op=osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.245287 7f9d68be2700 10 osd.2 173 dequeue_op 0x876f500 finish
2014-06-12 14:45:56.245291 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.245294 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.245303 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519320 (1 active)
2014-06-12 14:45:56.245307 7f9d7a962700  5 _do_op 0x694a7e0 seq 2310 osr(3.1bc 0x276d1b0)/0x276d1b0 start
2014-06-12 14:45:56.245311 7f9d7a962700 10 _do_transaction on 0x1c832540
2014-06-12 14:45:56.245316 7f9d7a962700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.245415 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.245432 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.245481 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.245501 7f9d7a962700 15 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.245505 7f9d7a962700 10 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1 r = 0
2014-06-12 14:45:56.245510 7f9d7a962700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.245522 7f9d7a962700 15 _write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 872448~516096
2014-06-12 14:45:56.245658 7f9d7a962700 10 lookup_strip_header done  cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.245849 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:45:56.247804 7f9d7a962700 10 _generic_write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 872448~516096 = 0
2014-06-12 14:45:56.247858 7f9d7a962700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.247874 7f9d7a962700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.247881 7f9d7a962700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.247890 7f9d7a962700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.248107 7f9d7a962700 20 set_header setting 16575 cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 parent seq 0
2014-06-12 14:45:56.248202 7f9d7a962700 20 set_header setting 2006 cid meta oid 1fe11b06/pglog_3.1bc/0//-1 parent seq 0
2014-06-12 14:45:56.248219 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.248228 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:45:56.251128 7f9d7a962700 10 _do_op 0x694a7e0 seq 2310 r = 0, finisher 0x6851820 0x218d1900
2014-06-12 14:45:56.251756 7f9d7a962700 10 _finish_op 0x694a7e0 seq 2310 osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.251759 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] op_commit: 236151
2014-06-12 14:45:56.251797 7f9d7715b700  5 -- op tracker -- , seq: 747346, time: 2014-06-12 14:45:56.251785, event: op_commit, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.251805 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519320 (0 active)
2014-06-12 14:45:56.251810 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.251812 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] op_applied: 236151
2014-06-12 14:45:56.251834 7f9d7795c700  5 -- op tracker -- , seq: 747346, time: 2014-06-12 14:45:56.251827, event: op_applied, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.251842 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] op_applied on primary on version 173'2453
2014-06-12 14:45:56.253236 7f9d44101700  5 -- op tracker -- , seq: 747347, time: 2014-06-12 14:45:56.253176, event: header_read, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.253252 7f9d44101700  5 -- op tracker -- , seq: 747347, time: 2014-06-12 14:45:56.253178, event: throttled, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.253260 7f9d44101700  5 -- op tracker -- , seq: 747347, time: 2014-06-12 14:45:56.253194, event: all_read, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.253267 7f9d44101700  5 -- op tracker -- , seq: 747347, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.253274 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:45:56.253280 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.253291 7f9d44101700 15 osd.2 173 enqueue_op 0x23e08e00 prio 196 cost 0 latency 0.000114 osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0) v2
2014-06-12 14:45:56.253328 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.253337 7f9d61bd4700 10 osd.2 173 dequeue_op 0x23e08e00 prio 196 cost 0 latency 0.000161 osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0) v2 pg pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean]
2014-06-12 14:45:56.253362 7f9d61bd4700  5 -- op tracker -- , seq: 747347, time: 2014-06-12 14:45:56.253358, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.253370 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] handle_message: 0x23e08e00
2014-06-12 14:45:56.253386 7f9d61bd4700  5 -- op tracker -- , seq: 747347, time: 2014-06-12 14:45:56.253383, event: started, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.253392 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] sub_op_modify_reply: tid 236151 op  ack_type 1 from 21
2014-06-12 14:45:56.253414 7f9d61bd4700  5 -- op tracker -- , seq: 747346, time: 2014-06-12 14:45:56.253405, event: sub_op_applied_rec, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.253423 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] repop_all_applied: repop tid 236151 all applied 
2014-06-12 14:45:56.253434 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] eval_repop repgather(0x53609c0 173'2453 rep_tid=236151 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.253451 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean]  sending ack on repgather(0x53609c0 173'2453 rep_tid=236151 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x1b775400
2014-06-12 14:45:56.253483 7f9d61bd4700 10 osd.2 173 dequeue_op 0x23e08e00 finish
2014-06-12 14:45:56.253490 7f9d61bd4700  5 -- op tracker -- , seq: 747347, time: 2014-06-12 14:45:56.253486, event: done, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.253506 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.253510 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.253758 7f9d44101700  5 -- op tracker -- , seq: 747348, time: 2014-06-12 14:45:56.253721, event: header_read, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.253784 7f9d44101700  5 -- op tracker -- , seq: 747348, time: 2014-06-12 14:45:56.253722, event: throttled, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.253792 7f9d44101700  5 -- op tracker -- , seq: 747348, time: 2014-06-12 14:45:56.253735, event: all_read, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.253800 7f9d44101700  5 -- op tracker -- , seq: 747348, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.253807 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.253812 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.253818 7f9d44101700 15 osd.2 173 enqueue_op 0x23e09f00 prio 196 cost 0 latency 0.000097 osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.253844 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.253849 7f9d673df700 10 osd.2 173 dequeue_op 0x23e09f00 prio 196 cost 0 latency 0.000128 osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 pg pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean]
2014-06-12 14:45:56.253869 7f9d673df700  5 -- op tracker -- , seq: 747348, time: 2014-06-12 14:45:56.253864, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.253876 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] handle_message: 0x23e09f00
2014-06-12 14:45:56.253891 7f9d673df700  5 -- op tracker -- , seq: 747348, time: 2014-06-12 14:45:56.253888, event: started, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.253896 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] sub_op_modify_reply: tid 236151 op  ack_type 4 from 21
2014-06-12 14:45:56.253915 7f9d673df700  5 -- op tracker -- , seq: 747346, time: 2014-06-12 14:45:56.253908, event: sub_op_commit_rec, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.253922 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2452 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] repop_all_committed: repop tid 236151 all committed 
2014-06-12 14:45:56.253933 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2451 active+clean] eval_repop repgather(0x53609c0 173'2453 rep_tid=236151 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.253956 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2451 active+clean] log_op_stats osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.010470 lat 0.010943
2014-06-12 14:45:56.253976 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2451 active+clean] publish_stats_to_osd 173:3176
2014-06-12 14:45:56.253994 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2451 active+clean]  sending commit on repgather(0x53609c0 173'2453 rep_tid=236151 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x1b774a00
2014-06-12 14:45:56.254032 7f9d673df700  5 -- op tracker -- , seq: 747346, time: 2014-06-12 14:45:56.254026, event: commit_sent, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.254040 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2451 active+clean]  requeue_ops 
2014-06-12 14:45:56.254052 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean]  removing repgather(0x53609c0 173'2453 rep_tid=236151 committed?=1 applied?=1 op=osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.254067 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean]    q front is repgather(0x53609c0 173'2453 rep_tid=236151 committed?=1 applied?=1 op=osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.254084 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] remove_repop repgather(0x53609c0 173'2453 rep_tid=236151 committed?=1 applied?=1 op=osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.254100 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean]  requeue_ops 
2014-06-12 14:45:56.254143 7f9d673df700  5 -- op tracker -- , seq: 747346, time: 2014-06-12 14:45:56.254138, event: done, op: osd_op(client.4291.0:140617 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.254161 7f9d673df700 10 osd.2 173 dequeue_op 0x23e09f00 finish
2014-06-12 14:45:56.254169 7f9d673df700  5 -- op tracker -- , seq: 747348, time: 2014-06-12 14:45:56.254166, event: done, op: osd_sub_op_reply(client.4291.0:140617 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.254179 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.254182 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.262793 7f9d56432700  5 -- op tracker -- , seq: 747349, time: 2014-06-12 14:45:56.253271, event: header_read, op: osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.262813 7f9d56432700  5 -- op tracker -- , seq: 747349, time: 2014-06-12 14:45:56.253273, event: throttled, op: osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.262822 7f9d56432700  5 -- op tracker -- , seq: 747349, time: 2014-06-12 14:45:56.262735, event: all_read, op: osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.262831 7f9d56432700  5 -- op tracker -- , seq: 747349, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.262838 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:56.262845 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:56.262855 7f9d56432700 15 osd.2 173 enqueue_op 0x2d17be00 prio 127 cost 1172263 latency 0.009584 osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.262895 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.262903 7f9d663dd700 10 osd.2 173 dequeue_op 0x2d17be00 prio 127 cost 1172263 latency 0.009632 osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.582( v 173'2758 (0'0,173'2758] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2757 active]
2014-06-12 14:45:56.262928 7f9d663dd700  5 -- op tracker -- , seq: 747349, time: 2014-06-12 14:45:56.262924, event: reached_pg, op: osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.262937 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2758 (0'0,173'2758] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2757 active] handle_message: 0x2d17be00
2014-06-12 14:45:56.262950 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2758 (0'0,173'2758] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2757 active] sub_op_modify trans c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 v 173'2759 (transaction) 174
2014-06-12 14:45:56.262967 7f9d663dd700  5 -- op tracker -- , seq: 747349, time: 2014-06-12 14:45:56.262963, event: started, op: osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.262984 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2758 (0'0,173'2758] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2757 active] append_log log((0'0,173'2758], crt=0'0) [173'2759 (173'2758) modify   c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 by client.4314.0:144672 2014-06-12 08:43:36.174499]
2014-06-12 14:45:56.263010 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2759 (0'0,173'2759] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2757 active] add_log_entry 173'2759 (173'2758) modify   c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 by client.4314.0:144672 2014-06-12 08:43:36.174499
2014-06-12 14:45:56.263033 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2759 (0'0,173'2759] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2757 active] append_log  adding 1 keys
2014-06-12 14:45:56.263136 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2759, trimmed: 
2014-06-12 14:45:56.263162 7f9d663dd700  5 -- op tracker -- , seq: 747349, time: 2014-06-12 14:45:56.263157, event: started, op: osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.263174 7f9d663dd700  5 queue_transactions existing osr(3.582 0x276da70)/0x276da70
2014-06-12 14:45:56.263178 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.263180 7f9d663dd700  5 queue_op 0x544cfc0 seq 2607 osr(3.582 0x276da70) 1173749 bytes   (queue has 1 ops and 1173749 bytes)
2014-06-12 14:45:56.263188 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.582( v 173'2759 (0'0,173'2759] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2757 active] do_sub_op osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.263204 7f9d663dd700 10 osd.2 173 dequeue_op 0x2d17be00 finish
2014-06-12 14:45:56.263208 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.263211 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.265630 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438900 (1 active)
2014-06-12 14:45:56.265645 7f9d7895e700  5 _do_op 0x544cfc0 seq 2607 osr(3.582 0x276da70)/0x276da70 start
2014-06-12 14:45:56.265651 7f9d7895e700 10 _do_transaction on 0x1ebec2b0
2014-06-12 14:45:56.265661 7f9d7895e700 15 _omap_setkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:45:56.265831 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe177ea/pglog_3.582/0//-1
2014-06-12 14:45:56.265854 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.265916 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.265940 7f9d7895e700 15 _omap_rmkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:45:56.265946 7f9d7895e700 10 _omap_rmkeys meta/1fe177ea/pglog_3.582/0//-1 r = 0
2014-06-12 14:45:56.265953 7f9d7895e700 15 _omap_setkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:45:56.266093 7f9d7895e700 15 _write 3.582_head/c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 3022848~1171456
2014-06-12 14:45:56.266260 7f9d7895e700 10 lookup_strip_header done  cid 3.582_head oid c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3
2014-06-12 14:45:56.266393 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:45:56.270110 7f9d4ed71700  5 -- op tracker -- , seq: 747350, time: 2014-06-12 14:45:56.256131, event: header_read, op: osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.270123 7f9d4ed71700  5 -- op tracker -- , seq: 747350, time: 2014-06-12 14:45:56.256134, event: throttled, op: osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.270133 7f9d4ed71700  5 -- op tracker -- , seq: 747350, time: 2014-06-12 14:45:56.270069, event: all_read, op: osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.270143 7f9d4ed71700  5 -- op tracker -- , seq: 747350, time: 0.000000, event: dispatched, op: osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.270151 7f9d4ed71700 20 osd.2 173 should_share_map client.4291 172.16.96.220:0/1241802 173
2014-06-12 14:45:56.270183 7f9d4ed71700 15 osd.2 173 enqueue_op 0x876fa00 prio 63 cost 516096 latency 0.014051 osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.271655 7f9d7895e700 10 _generic_write 3.582_head/c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 3022848~1171456 = 0
2014-06-12 14:45:56.271755 7f9d7895e700 15 _setattrs 3.582_head/c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3
2014-06-12 14:45:56.271775 7f9d7895e700 10 _setattrs 3.582_head/c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 = 0
2014-06-12 14:45:56.271780 7f9d7895e700 15 _setattrs 3.582_head/c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3
2014-06-12 14:45:56.271788 7f9d7895e700 10 _setattrs 3.582_head/c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 = 0
2014-06-12 14:45:56.272115 7f9d7895e700 20 set_header setting 17813 cid 3.582_head oid c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 parent seq 0
2014-06-12 14:45:56.272238 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.272253 7f9d7895e700 20 set_header setting 2318 cid meta oid 1fe177ea/pglog_3.582/0//-1 parent seq 0
2014-06-12 14:45:56.272263 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:45:56.272884 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.272897 7f9d67be0700 10 osd.2 173 dequeue_op 0x876fa00 prio 63 cost 516096 latency 0.016765 osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean]
2014-06-12 14:45:56.272933 7f9d67be0700  5 -- op tracker -- , seq: 747350, time: 2014-06-12 14:45:56.272926, event: reached_pg, op: osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.272955 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:56.272977 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] handle_message: 0x876fa00
2014-06-12 14:45:56.272990 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] do_op osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:56.273017 7f9d67be0700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_'
2014-06-12 14:45:56.273266 7f9d67be0700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_' = 0
2014-06-12 14:45:56.273283 7f9d67be0700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset'
2014-06-12 14:45:56.273376 7f9d67be0700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset' = 0
2014-06-12 14:45:56.273386 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] populate_obc_watchers d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.273398 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.273407 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] get_object_context: creating obc from disk: 0x6c97080
2014-06-12 14:45:56.273418 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] get_object_context: 0x6c97080 d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 rwstate(none n=0 w=0) oi: d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2453 client.4291.0:140617 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2453) ssc: 0x81ba540 snapset: 0=[]:[]+head
2014-06-12 14:45:56.273435 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] find_object_context d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 @head oi=d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2453 client.4291.0:140617 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2453)
2014-06-12 14:45:56.273460 7f9d67be0700  5 -- op tracker -- , seq: 747350, time: 2014-06-12 14:45:56.273454, event: started, op: osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.273467 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] execute_ctx 0x2963be00
2014-06-12 14:45:56.273484 7f9d67be0700  5 -- op tracker -- , seq: 747350, time: 2014-06-12 14:45:56.273479, event: started, op: osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.273490 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] do_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] ov 173'2453 av 173'2454 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:56.273503 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] do_osd_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096]
2014-06-12 14:45:56.273513 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:56.273526 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] do_osd_op  write 1388544~516096
2014-06-12 14:45:56.273542 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] do_osd_op_effects on session 0xdd06400
2014-06-12 14:45:56.273552 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 snapset=0x81ba578  snapc=0=[]
2014-06-12 14:45:56.273563 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:56.273579 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] finish_ctx d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 0x2963be00 op modify  
2014-06-12 14:45:56.273591 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean]  set mtime to 2014-06-12 08:44:58.636457
2014-06-12 14:45:56.273611 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean]  final snapset 0=[]:[]+head in d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.273649 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean]  zeroing write result code 0
2014-06-12 14:45:56.273659 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] new_repop rep_tid 236152 on osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.273675 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] issue_repop rep_tid 236152 o d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.273699 7f9d67be0700  5 -- op tracker -- , seq: 747350, time: 2014-06-12 14:45:56.273694, event: waiting for subops from 21, op: osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.273727 7f9d67be0700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:56.273752 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2453 (0'0,173'2453] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] append_log log((0'0,173'2453], crt=0'0) [173'2454 (173'2453) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140618 2014-06-12 08:44:58.636457]
2014-06-12 14:45:56.273777 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2453 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] add_log_entry 173'2454 (173'2453) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140618 2014-06-12 08:44:58.636457
2014-06-12 14:45:56.273802 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2453 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] append_log  adding 1 keys
2014-06-12 14:45:56.273836 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2454, trimmed: 
2014-06-12 14:45:56.273867 7f9d67be0700  5 queue_transactions existing osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.273874 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.273875 7f9d67be0700  5 queue_op 0xff14300 seq 2311 osr(3.1bc 0x276d1b0) 518389 bytes   (queue has 2 ops and 1692138 bytes)
2014-06-12 14:45:56.273883 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2453 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] eval_repop repgather(0x1c378000 173'2454 rep_tid=236152 committed?=0 applied?=0 lock=0 op=osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.273903 7f9d67be0700 10 osd.2 173 dequeue_op 0x876fa00 finish
2014-06-12 14:45:56.273907 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.273911 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.277646 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519320 (2 active)
2014-06-12 14:45:56.277661 7f9d7f16b700  5 _do_op 0xff14300 seq 2311 osr(3.1bc 0x276d1b0)/0x276d1b0 start
2014-06-12 14:45:56.277668 7f9d7f16b700 10 _do_transaction on 0x1c379bc0
2014-06-12 14:45:56.277676 7f9d7f16b700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.277857 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.277887 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.283983 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:56.283986 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:56.283987 7f9d74bfa700 20 osd.2 173 _dispatch 0x1e701520 ping v1
2014-06-12 14:45:56.283989 7f9d74bfa700 10 osd.2 173 ping from client.4306
2014-06-12 14:45:56.283991 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:56.283992 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:56.286477 7f9d44101700  5 -- op tracker -- , seq: 747351, time: 2014-06-12 14:45:56.286432, event: header_read, op: osd_sub_op_reply(client.4291.0:140618 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.286488 7f9d44101700  5 -- op tracker -- , seq: 747351, time: 2014-06-12 14:45:56.286433, event: throttled, op: osd_sub_op_reply(client.4291.0:140618 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.286497 7f9d44101700  5 -- op tracker -- , seq: 747351, time: 2014-06-12 14:45:56.286444, event: all_read, op: osd_sub_op_reply(client.4291.0:140618 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.286504 7f9d44101700  5 -- op tracker -- , seq: 747351, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140618 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.286510 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140618 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.286516 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.286524 7f9d44101700 15 osd.2 173 enqueue_op 0x28fd8800 prio 196 cost 0 latency 0.000091 osd_sub_op_reply(client.4291.0:140618 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.286554 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.286563 7f9d693e3700 10 osd.2 173 dequeue_op 0x28fd8800 prio 196 cost 0 latency 0.000131 osd_sub_op_reply(client.4291.0:140618 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 pg pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2453 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean]
2014-06-12 14:45:56.286584 7f9d693e3700  5 -- op tracker -- , seq: 747351, time: 2014-06-12 14:45:56.286581, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140618 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.286594 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2453 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] handle_message: 0x28fd8800
2014-06-12 14:45:56.286609 7f9d693e3700  5 -- op tracker -- , seq: 747351, time: 2014-06-12 14:45:56.286606, event: started, op: osd_sub_op_reply(client.4291.0:140618 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.286616 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2453 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] sub_op_modify_reply: tid 236152 op  ack_type 4 from 21
2014-06-12 14:45:56.286638 7f9d693e3700  5 -- op tracker -- , seq: 747350, time: 2014-06-12 14:45:56.286629, event: sub_op_commit_rec, op: osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.286646 7f9d693e3700 10 osd.2 173 dequeue_op 0x28fd8800 finish
2014-06-12 14:45:56.286653 7f9d693e3700  5 -- op tracker -- , seq: 747351, time: 2014-06-12 14:45:56.286649, event: done, op: osd_sub_op_reply(client.4291.0:140618 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.286665 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.286668 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.290863 7f9d7895e700 10 _do_op 0x544cfc0 seq 2607 r = 0, finisher 0x31d0cbe0 0
2014-06-12 14:45:56.290880 7f9d7895e700 10 _finish_op 0x544cfc0 seq 2607 osr(3.582 0x276da70)/0x276da70
2014-06-12 14:45:56.290897 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438900 (1 active)
2014-06-12 14:45:56.290901 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.290897 7f9d7715b700  5 -- op tracker -- , seq: 747349, time: 2014-06-12 14:45:56.290888, event: commit_sent, op: osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.290907 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2759 (0'0,173'2759] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2757 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:56.290930 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:56.291016 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.291045 7f9d7f16b700 15 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.291051 7f9d7f16b700 10 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1 r = 0
2014-06-12 14:45:56.291056 7f9d7f16b700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.291135 7f9d7f16b700 15 _write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 1388544~516096
2014-06-12 14:45:56.291240 7f9d7f16b700 10 lookup_strip_header done  cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.291361 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:45:56.292532 7f9d7f16b700 10 _generic_write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 1388544~516096 = 0
2014-06-12 14:45:56.292573 7f9d7f16b700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.292583 7f9d7f16b700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.292590 7f9d7f16b700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.292594 7f9d7f16b700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.292723 7f9d7f16b700 20 set_header setting 16575 cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 parent seq 0
2014-06-12 14:45:56.292773 7f9d7f16b700 20 set_header setting 2006 cid meta oid 1fe11b06/pglog_3.1bc/0//-1 parent seq 0
2014-06-12 14:45:56.292784 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.292792 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:45:56.295075 7f9d7f16b700 10 _do_op 0xff14300 seq 2311 r = 0, finisher 0x654caa0 0x3d03bc0
2014-06-12 14:45:56.295117 7f9d7795c700  5 -- op tracker -- , seq: 747349, time: 2014-06-12 14:45:56.295106, event: sub_op_applied, op: osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.295139 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2759 (0'0,173'2759] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2758 active] sub_op_modify_applied on 0x1ebec1c0 op osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.295163 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2759 (0'0,173'2759] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2758 active] op_applied on primary on version 173'2759
2014-06-12 14:45:56.295174 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2759 (0'0,173'2759] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2758 active] op_applied on replica on version 173'2759
2014-06-12 14:45:56.295194 7f9d7795c700  5 -- op tracker -- , seq: 747349, time: 2014-06-12 14:45:56.295190, event: done, op: osd_sub_op(client.4314.0:144672 3.582 c526fd82/rbd_data.10b16b8b4567.0000000000001022/head//3 [] v 173'2759 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.295227 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2453 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] op_commit: 236152
2014-06-12 14:45:56.295255 7f9d7715b700  5 -- op tracker -- , seq: 747350, time: 2014-06-12 14:45:56.295244, event: op_commit, op: osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.295265 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2453 crt=0'0 lcod 173'2452 mlcod 173'2452 active+clean] repop_all_committed: repop tid 236152 all committed 
2014-06-12 14:45:56.295280 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2452 active+clean] eval_repop repgather(0x1c378000 173'2454 rep_tid=236152 committed?=1 applied?=0 lock=0 op=osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.295301 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2452 active+clean] log_op_stats osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.039169
2014-06-12 14:45:56.295320 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2452 active+clean] publish_stats_to_osd 173:3177
2014-06-12 14:45:56.295333 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2452 active+clean]  sending commit on repgather(0x1c378000 173'2454 rep_tid=236152 committed?=1 applied?=0 lock=0 op=osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x564d400
2014-06-12 14:45:56.295374 7f9d7715b700  5 -- op tracker -- , seq: 747350, time: 2014-06-12 14:45:56.295369, event: commit_sent, op: osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.295424 7f9d7f16b700 10 _finish_op 0xff14300 seq 2311 osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.295441 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2452 active+clean] op_applied: 236152
2014-06-12 14:45:56.295461 7f9d7795c700  5 -- op tracker -- , seq: 747350, time: 2014-06-12 14:45:56.295455, event: op_applied, op: osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.295470 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2452 active+clean] op_applied on primary on version 173'2454
2014-06-12 14:45:56.295481 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2452 active+clean] repop_all_applied: repop tid 236152 all applied 
2014-06-12 14:45:56.295492 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2452 active+clean] eval_repop repgather(0x1c378000 173'2454 rep_tid=236152 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.295510 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2452 active+clean] log_op_stats osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.039379
2014-06-12 14:45:56.295526 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2452 active+clean] publish_stats_to_osd 173:3178
2014-06-12 14:45:56.295538 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2452 active+clean]  requeue_ops 
2014-06-12 14:45:56.295548 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean]  removing repgather(0x1c378000 173'2454 rep_tid=236152 committed?=1 applied?=1 op=osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.295564 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean]    q front is repgather(0x1c378000 173'2454 rep_tid=236152 committed?=1 applied?=1 op=osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.295581 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] remove_repop repgather(0x1c378000 173'2454 rep_tid=236152 committed?=1 applied?=1 op=osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.295596 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean]  requeue_ops 
2014-06-12 14:45:56.295642 7f9d7f16b700  5 -- op tracker -- , seq: 747350, time: 2014-06-12 14:45:56.295629, event: done, op: osd_op(client.4291.0:140618 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.295694 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519320 (0 active)
2014-06-12 14:45:56.295699 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.306078 7f9d4ed71700  5 -- op tracker -- , seq: 747352, time: 2014-06-12 14:45:56.297643, event: header_read, op: osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.306100 7f9d4ed71700  5 -- op tracker -- , seq: 747352, time: 2014-06-12 14:45:56.297646, event: throttled, op: osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.306110 7f9d4ed71700  5 -- op tracker -- , seq: 747352, time: 2014-06-12 14:45:56.306023, event: all_read, op: osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.306120 7f9d4ed71700  5 -- op tracker -- , seq: 747352, time: 0.000000, event: dispatched, op: osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.306130 7f9d4ed71700 20 osd.2 173 should_share_map client.4291 172.16.96.220:0/1241802 173
2014-06-12 14:45:56.306159 7f9d4ed71700 15 osd.2 173 enqueue_op 0x876ea00 prio 63 cost 516096 latency 0.008515 osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.306195 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.306206 7f9d62bd6700 10 osd.2 173 dequeue_op 0x876ea00 prio 63 cost 516096 latency 0.008563 osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean]
2014-06-12 14:45:56.306236 7f9d62bd6700  5 -- op tracker -- , seq: 747352, time: 2014-06-12 14:45:56.306231, event: reached_pg, op: osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.306246 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:56.306263 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] handle_message: 0x876ea00
2014-06-12 14:45:56.306274 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] do_op osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:56.306302 7f9d62bd6700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_'
2014-06-12 14:45:56.306525 7f9d62bd6700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_' = 0
2014-06-12 14:45:56.306543 7f9d62bd6700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset'
2014-06-12 14:45:56.306649 7f9d62bd6700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset' = 0
2014-06-12 14:45:56.306661 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] populate_obc_watchers d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.306675 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.306685 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] get_object_context: creating obc from disk: 0xa546840
2014-06-12 14:45:56.306694 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] get_object_context: 0xa546840 d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 rwstate(none n=0 w=0) oi: d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2454 client.4291.0:140618 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2454) ssc: 0x16ee4540 snapset: 0=[]:[]+head
2014-06-12 14:45:56.306712 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] find_object_context d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 @head oi=d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2454 client.4291.0:140618 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2454)
2014-06-12 14:45:56.306738 7f9d62bd6700  5 -- op tracker -- , seq: 747352, time: 2014-06-12 14:45:56.306733, event: started, op: osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.306748 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] execute_ctx 0x2a82c00
2014-06-12 14:45:56.306766 7f9d62bd6700  5 -- op tracker -- , seq: 747352, time: 2014-06-12 14:45:56.306761, event: started, op: osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.306774 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] do_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] ov 173'2454 av 173'2455 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:56.306791 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] do_osd_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096]
2014-06-12 14:45:56.306804 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:56.306818 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] do_osd_op  write 1904640~516096
2014-06-12 14:45:56.306835 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] do_osd_op_effects on session 0xdd06400
2014-06-12 14:45:56.306849 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 snapset=0x16ee4578  snapc=0=[]
2014-06-12 14:45:56.306863 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:56.306876 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] finish_ctx d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 0x2a82c00 op modify  
2014-06-12 14:45:56.306889 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean]  set mtime to 2014-06-12 08:44:58.677910
2014-06-12 14:45:56.306911 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean]  final snapset 0=[]:[]+head in d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.306933 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean]  zeroing write result code 0
2014-06-12 14:45:56.306945 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] new_repop rep_tid 236153 on osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.306963 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] issue_repop rep_tid 236153 o d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.306984 7f9d62bd6700  5 -- op tracker -- , seq: 747352, time: 2014-06-12 14:45:56.306980, event: waiting for subops from 21, op: osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.307006 7f9d62bd6700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:56.307027 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2454 (0'0,173'2454] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] append_log log((0'0,173'2454], crt=0'0) [173'2455 (173'2454) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140619 2014-06-12 08:44:58.677910]
2014-06-12 14:45:56.307051 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2454 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] add_log_entry 173'2455 (173'2454) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140619 2014-06-12 08:44:58.677910
2014-06-12 14:45:56.307070 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2454 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] append_log  adding 1 keys
2014-06-12 14:45:56.307100 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2455, trimmed: 
2014-06-12 14:45:56.307120 7f9d62bd6700  5 queue_transactions existing osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.307125 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.307126 7f9d62bd6700  5 queue_op 0x101ccf60 seq 2312 osr(3.1bc 0x276d1b0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:45:56.307133 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2454 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] eval_repop repgather(0x5214000 173'2455 rep_tid=236153 committed?=0 applied?=0 lock=0 op=osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.307150 7f9d62bd6700 10 osd.2 173 dequeue_op 0x876ea00 finish
2014-06-12 14:45:56.307154 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.307156 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.307165 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519320 (1 active)
2014-06-12 14:45:56.307170 7f9d7e169700  5 _do_op 0x101ccf60 seq 2312 osr(3.1bc 0x276d1b0)/0x276d1b0 start
2014-06-12 14:45:56.307173 7f9d7e169700 10 _do_transaction on 0x2ad98180
2014-06-12 14:45:56.307179 7f9d7e169700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.307287 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.307306 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.307360 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.307383 7f9d7e169700 15 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.307390 7f9d7e169700 10 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1 r = 0
2014-06-12 14:45:56.307396 7f9d7e169700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.307413 7f9d7e169700 15 _write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 1904640~516096
2014-06-12 14:45:56.307719 7f9d7e169700 10 lookup_strip_header done  cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.307857 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:45:56.309658 7f9d7e169700 10 _generic_write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 1904640~516096 = 0
2014-06-12 14:45:56.309705 7f9d7e169700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.309722 7f9d7e169700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.309730 7f9d7e169700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.309738 7f9d7e169700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.309947 7f9d7e169700 20 set_header setting 16575 cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 parent seq 0
2014-06-12 14:45:56.310019 7f9d7e169700 20 set_header setting 2006 cid meta oid 1fe11b06/pglog_3.1bc/0//-1 parent seq 0
2014-06-12 14:45:56.310036 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.310046 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:45:56.311980 7f9d7e169700 10 _do_op 0x101ccf60 seq 2312 r = 0, finisher 0x20aaa240 0x397a8c0
2014-06-12 14:45:56.311999 7f9d7e169700 10 _finish_op 0x101ccf60 seq 2312 osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.312018 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519320 (0 active)
2014-06-12 14:45:56.312005 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2454 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] op_commit: 236153
2014-06-12 14:45:56.312027 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.312040 7f9d7715b700  5 -- op tracker -- , seq: 747352, time: 2014-06-12 14:45:56.312029, event: op_commit, op: osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.312059 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2454 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] op_applied: 236153
2014-06-12 14:45:56.312075 7f9d7795c700  5 -- op tracker -- , seq: 747352, time: 2014-06-12 14:45:56.312071, event: op_applied, op: osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.312081 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2454 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] op_applied on primary on version 173'2455
2014-06-12 14:45:56.316888 7f9d44101700  5 -- op tracker -- , seq: 747353, time: 2014-06-12 14:45:56.316837, event: header_read, op: osd_sub_op_reply(client.4291.0:140619 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.316901 7f9d44101700  5 -- op tracker -- , seq: 747353, time: 2014-06-12 14:45:56.316839, event: throttled, op: osd_sub_op_reply(client.4291.0:140619 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.316916 7f9d44101700  5 -- op tracker -- , seq: 747353, time: 2014-06-12 14:45:56.316852, event: all_read, op: osd_sub_op_reply(client.4291.0:140619 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.316921 7f9d44101700  5 -- op tracker -- , seq: 747353, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140619 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.316925 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140619 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.316929 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.316936 7f9d44101700 15 osd.2 173 enqueue_op 0x28fd8c00 prio 196 cost 0 latency 0.000099 osd_sub_op_reply(client.4291.0:140619 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.316961 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.316968 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x28fd8c00 prio 196 cost 0 latency 0.000130 osd_sub_op_reply(client.4291.0:140619 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 pg pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2454 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean]
2014-06-12 14:45:56.316983 7f9d6b3e7700  5 -- op tracker -- , seq: 747353, time: 2014-06-12 14:45:56.316980, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140619 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.316994 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2454 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] handle_message: 0x28fd8c00
2014-06-12 14:45:56.317004 7f9d6b3e7700  5 -- op tracker -- , seq: 747353, time: 2014-06-12 14:45:56.317002, event: started, op: osd_sub_op_reply(client.4291.0:140619 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.317008 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2454 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] sub_op_modify_reply: tid 236153 op  ack_type 4 from 21
2014-06-12 14:45:56.317021 7f9d6b3e7700  5 -- op tracker -- , seq: 747352, time: 2014-06-12 14:45:56.317015, event: sub_op_commit_rec, op: osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.317026 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2454 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] repop_all_applied: repop tid 236153 all applied 
2014-06-12 14:45:56.317033 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2454 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] eval_repop repgather(0x5214000 173'2455 rep_tid=236153 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.317042 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2454 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean]  sending ack on repgather(0x5214000 173'2455 rep_tid=236153 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x32eef680
2014-06-12 14:45:56.317062 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2454 crt=0'0 lcod 173'2453 mlcod 173'2453 active+clean] repop_all_committed: repop tid 236153 all committed 
2014-06-12 14:45:56.317072 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2453 active+clean] eval_repop repgather(0x5214000 173'2455 rep_tid=236153 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.317087 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2453 active+clean] log_op_stats osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.019418 lat 0.019443
2014-06-12 14:45:56.317103 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2453 active+clean] publish_stats_to_osd 173:3179
2014-06-12 14:45:56.317112 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2453 active+clean]  sending commit on repgather(0x5214000 173'2455 rep_tid=236153 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x6ce8000
2014-06-12 14:45:56.317134 7f9d6b3e7700  5 -- op tracker -- , seq: 747352, time: 2014-06-12 14:45:56.317129, event: commit_sent, op: osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.317144 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2453 active+clean]  requeue_ops 
2014-06-12 14:45:56.317155 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean]  removing repgather(0x5214000 173'2455 rep_tid=236153 committed?=1 applied?=1 op=osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.317168 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean]    q front is repgather(0x5214000 173'2455 rep_tid=236153 committed?=1 applied?=1 op=osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.317177 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] remove_repop repgather(0x5214000 173'2455 rep_tid=236153 committed?=1 applied?=1 op=osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.317185 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean]  requeue_ops 
2014-06-12 14:45:56.317203 7f9d6b3e7700  5 -- op tracker -- , seq: 747352, time: 2014-06-12 14:45:56.317200, event: done, op: osd_op(client.4291.0:140619 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.317218 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x28fd8c00 finish
2014-06-12 14:45:56.317223 7f9d6b3e7700  5 -- op tracker -- , seq: 747353, time: 2014-06-12 14:45:56.317221, event: done, op: osd_sub_op_reply(client.4291.0:140619 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.317229 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.317232 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.320364 7f9d4ed71700  5 -- op tracker -- , seq: 747354, time: 2014-06-12 14:45:56.319406, event: header_read, op: osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.320373 7f9d4ed71700  5 -- op tracker -- , seq: 747354, time: 2014-06-12 14:45:56.319409, event: throttled, op: osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.320389 7f9d4ed71700  5 -- op tracker -- , seq: 747354, time: 2014-06-12 14:45:56.320337, event: all_read, op: osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.320394 7f9d4ed71700  5 -- op tracker -- , seq: 747354, time: 0.000000, event: dispatched, op: osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.320399 7f9d4ed71700 20 osd.2 173 should_share_map client.4291 172.16.96.220:0/1241802 173
2014-06-12 14:45:56.320421 7f9d4ed71700 15 osd.2 173 enqueue_op 0x2a902100 prio 63 cost 516096 latency 0.001015 osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.320455 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.320465 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a902100 prio 63 cost 516096 latency 0.001059 osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean]
2014-06-12 14:45:56.320490 7f9d64bda700  5 -- op tracker -- , seq: 747354, time: 2014-06-12 14:45:56.320485, event: reached_pg, op: osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.320498 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:56.320513 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] handle_message: 0x2a902100
2014-06-12 14:45:56.320523 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] do_op osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:56.320547 7f9d64bda700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_'
2014-06-12 14:45:56.320784 7f9d64bda700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_' = 0
2014-06-12 14:45:56.320801 7f9d64bda700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset'
2014-06-12 14:45:56.320901 7f9d64bda700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset' = 0
2014-06-12 14:45:56.320910 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] populate_obc_watchers d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.320922 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.320931 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] get_object_context: creating obc from disk: 0x1e6c4b00
2014-06-12 14:45:56.320941 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] get_object_context: 0x1e6c4b00 d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 rwstate(none n=0 w=0) oi: d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2455 client.4291.0:140619 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2455) ssc: 0x283d2ee0 snapset: 0=[]:[]+head
2014-06-12 14:45:56.320957 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] find_object_context d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 @head oi=d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2455 client.4291.0:140619 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2455)
2014-06-12 14:45:56.320981 7f9d64bda700  5 -- op tracker -- , seq: 747354, time: 2014-06-12 14:45:56.320976, event: started, op: osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.320989 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] execute_ctx 0x29edf000
2014-06-12 14:45:56.321007 7f9d64bda700  5 -- op tracker -- , seq: 747354, time: 2014-06-12 14:45:56.321002, event: started, op: osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.321013 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] do_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] ov 173'2455 av 173'2456 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:56.321027 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] do_osd_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096]
2014-06-12 14:45:56.321038 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:56.321051 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] do_osd_op  write 2420736~516096
2014-06-12 14:45:56.321066 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] do_osd_op_effects on session 0xdd06400
2014-06-12 14:45:56.321077 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 snapset=0x283d2f18  snapc=0=[]
2014-06-12 14:45:56.321089 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:56.321098 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] finish_ctx d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 0x29edf000 op modify  
2014-06-12 14:45:56.321112 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean]  set mtime to 2014-06-12 08:44:58.699742
2014-06-12 14:45:56.321133 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean]  final snapset 0=[]:[]+head in d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.321150 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean]  zeroing write result code 0
2014-06-12 14:45:56.321160 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] new_repop rep_tid 236154 on osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.321180 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] issue_repop rep_tid 236154 o d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.321204 7f9d64bda700  5 -- op tracker -- , seq: 747354, time: 2014-06-12 14:45:56.321198, event: waiting for subops from 21, op: osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.321223 7f9d64bda700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:56.321244 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2455 (0'0,173'2455] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] append_log log((0'0,173'2455], crt=0'0) [173'2456 (173'2455) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140620 2014-06-12 08:44:58.699742]
2014-06-12 14:45:56.321272 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2455 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] add_log_entry 173'2456 (173'2455) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140620 2014-06-12 08:44:58.699742
2014-06-12 14:45:56.321294 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2455 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] append_log  adding 1 keys
2014-06-12 14:45:56.321327 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2456, trimmed: 
2014-06-12 14:45:56.321349 7f9d64bda700  5 queue_transactions existing osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.321354 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.321355 7f9d64bda700  5 queue_op 0x563c120 seq 2313 osr(3.1bc 0x276d1b0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:45:56.321367 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519320 (1 active)
2014-06-12 14:45:56.321374 7f9d82171700  5 _do_op 0x563c120 seq 2313 osr(3.1bc 0x276d1b0)/0x276d1b0 start
2014-06-12 14:45:56.321378 7f9d82171700 10 _do_transaction on 0x514c840
2014-06-12 14:45:56.321364 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2455 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] eval_repop repgather(0x52d1d40 173'2456 rep_tid=236154 committed?=0 applied?=0 lock=0 op=osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.321382 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a902100 finish
2014-06-12 14:45:56.321385 7f9d82171700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.321387 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.321389 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.321493 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.321517 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.321572 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.321594 7f9d82171700 15 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.321600 7f9d82171700 10 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1 r = 0
2014-06-12 14:45:56.321606 7f9d82171700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.321719 7f9d82171700 15 _write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 2420736~516096
2014-06-12 14:45:56.321895 7f9d82171700 10 lookup_strip_header done  cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.322066 7f9d82171700 10 file_to_extents done 
2014-06-12 14:45:56.323868 7f9d82171700 10 _generic_write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 2420736~516096 = 0
2014-06-12 14:45:56.323923 7f9d82171700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.323940 7f9d82171700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.323946 7f9d82171700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.323955 7f9d82171700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.324165 7f9d82171700 20 set_header setting 16575 cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 parent seq 0
2014-06-12 14:45:56.324241 7f9d82171700 20 set_header setting 2006 cid meta oid 1fe11b06/pglog_3.1bc/0//-1 parent seq 0
2014-06-12 14:45:56.324258 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.324267 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:45:56.327845 7f9d44101700  5 -- op tracker -- , seq: 747355, time: 2014-06-12 14:45:56.327777, event: header_read, op: osd_sub_op_reply(client.4291.0:140620 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.327862 7f9d44101700  5 -- op tracker -- , seq: 747355, time: 2014-06-12 14:45:56.327778, event: throttled, op: osd_sub_op_reply(client.4291.0:140620 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.327871 7f9d44101700  5 -- op tracker -- , seq: 747355, time: 2014-06-12 14:45:56.327795, event: all_read, op: osd_sub_op_reply(client.4291.0:140620 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.327879 7f9d44101700  5 -- op tracker -- , seq: 747355, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140620 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.327885 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140620 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.327891 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.327900 7f9d44101700 15 osd.2 173 enqueue_op 0x28fd9800 prio 196 cost 0 latency 0.000123 osd_sub_op_reply(client.4291.0:140620 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.327935 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.327948 7f9d683e1700 10 osd.2 173 dequeue_op 0x28fd9800 prio 196 cost 0 latency 0.000170 osd_sub_op_reply(client.4291.0:140620 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 pg pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2455 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean]
2014-06-12 14:45:56.327973 7f9d683e1700  5 -- op tracker -- , seq: 747355, time: 2014-06-12 14:45:56.327969, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140620 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.327984 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2455 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] handle_message: 0x28fd9800
2014-06-12 14:45:56.328002 7f9d683e1700  5 -- op tracker -- , seq: 747355, time: 2014-06-12 14:45:56.327999, event: started, op: osd_sub_op_reply(client.4291.0:140620 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.328010 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2455 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] sub_op_modify_reply: tid 236154 op  ack_type 4 from 21
2014-06-12 14:45:56.328033 7f9d683e1700  5 -- op tracker -- , seq: 747354, time: 2014-06-12 14:45:56.328024, event: sub_op_commit_rec, op: osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.328042 7f9d683e1700 10 osd.2 173 dequeue_op 0x28fd9800 finish
2014-06-12 14:45:56.328049 7f9d683e1700  5 -- op tracker -- , seq: 747355, time: 2014-06-12 14:45:56.328046, event: done, op: osd_sub_op_reply(client.4291.0:140620 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.328064 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.328069 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.328161 7f9d82171700 10 _do_op 0x563c120 seq 2313 r = 0, finisher 0x70f52a0 0x2bc6acc0
2014-06-12 14:45:56.328175 7f9d82171700 10 _finish_op 0x563c120 seq 2313 osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.328184 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519320 (0 active)
2014-06-12 14:45:56.328194 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.328181 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2455 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] op_commit: 236154
2014-06-12 14:45:56.328204 7f9d7715b700  5 -- op tracker -- , seq: 747354, time: 2014-06-12 14:45:56.328197, event: op_commit, op: osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.328218 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2455 crt=0'0 lcod 173'2454 mlcod 173'2454 active+clean] repop_all_committed: repop tid 236154 all committed 
2014-06-12 14:45:56.328231 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2454 active+clean] eval_repop repgather(0x52d1d40 173'2456 rep_tid=236154 committed?=1 applied?=0 lock=0 op=osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.328250 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2454 active+clean] log_op_stats osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.008843
2014-06-12 14:45:56.328270 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2454 active+clean] publish_stats_to_osd 173:3180
2014-06-12 14:45:56.328283 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2454 active+clean]  sending commit on repgather(0x52d1d40 173'2456 rep_tid=236154 committed?=1 applied?=0 lock=0 op=osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x564d180
2014-06-12 14:45:56.328323 7f9d7715b700  5 -- op tracker -- , seq: 747354, time: 2014-06-12 14:45:56.328317, event: commit_sent, op: osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.328339 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2454 active+clean] op_applied: 236154
2014-06-12 14:45:56.328362 7f9d7795c700  5 -- op tracker -- , seq: 747354, time: 2014-06-12 14:45:56.328356, event: op_applied, op: osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.328371 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2454 active+clean] op_applied on primary on version 173'2456
2014-06-12 14:45:56.328383 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2454 active+clean] repop_all_applied: repop tid 236154 all applied 
2014-06-12 14:45:56.328394 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2454 active+clean] eval_repop repgather(0x52d1d40 173'2456 rep_tid=236154 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.328411 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2454 active+clean] log_op_stats osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.009005
2014-06-12 14:45:56.328428 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2454 active+clean] publish_stats_to_osd 173:3181
2014-06-12 14:45:56.328441 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2454 active+clean]  requeue_ops 
2014-06-12 14:45:56.328453 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean]  removing repgather(0x52d1d40 173'2456 rep_tid=236154 committed?=1 applied?=1 op=osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.328470 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean]    q front is repgather(0x52d1d40 173'2456 rep_tid=236154 committed?=1 applied?=1 op=osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.328486 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] remove_repop repgather(0x52d1d40 173'2456 rep_tid=236154 committed?=1 applied?=1 op=osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.328501 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean]  requeue_ops 
2014-06-12 14:45:56.328529 7f9d7795c700  5 -- op tracker -- , seq: 747354, time: 2014-06-12 14:45:56.328524, event: done, op: osd_op(client.4291.0:140620 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.331572 7f9d4ed71700  5 -- op tracker -- , seq: 747356, time: 2014-06-12 14:45:56.330479, event: header_read, op: osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.331591 7f9d4ed71700  5 -- op tracker -- , seq: 747356, time: 2014-06-12 14:45:56.330481, event: throttled, op: osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.331601 7f9d4ed71700  5 -- op tracker -- , seq: 747356, time: 2014-06-12 14:45:56.331530, event: all_read, op: osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.331611 7f9d4ed71700  5 -- op tracker -- , seq: 747356, time: 0.000000, event: dispatched, op: osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.331620 7f9d4ed71700 20 osd.2 173 should_share_map client.4291 172.16.96.220:0/1241802 173
2014-06-12 14:45:56.331651 7f9d4ed71700 15 osd.2 173 enqueue_op 0x2a903200 prio 63 cost 516096 latency 0.001172 osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.331684 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.331695 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2a903200 prio 63 cost 516096 latency 0.001215 osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean]
2014-06-12 14:45:56.331723 7f9d65bdc700  5 -- op tracker -- , seq: 747356, time: 2014-06-12 14:45:56.331718, event: reached_pg, op: osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.331735 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:56.331752 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] handle_message: 0x2a903200
2014-06-12 14:45:56.331765 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] do_op osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:56.331793 7f9d65bdc700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_'
2014-06-12 14:45:56.332028 7f9d65bdc700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_' = 0
2014-06-12 14:45:56.332048 7f9d65bdc700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset'
2014-06-12 14:45:56.332156 7f9d65bdc700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset' = 0
2014-06-12 14:45:56.332168 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] populate_obc_watchers d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.332182 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.332194 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] get_object_context: creating obc from disk: 0x212fc000
2014-06-12 14:45:56.332205 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] get_object_context: 0x212fc000 d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 rwstate(none n=0 w=0) oi: d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2456 client.4291.0:140620 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2456) ssc: 0xee5ed20 snapset: 0=[]:[]+head
2014-06-12 14:45:56.332223 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] find_object_context d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 @head oi=d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2456 client.4291.0:140620 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2456)
2014-06-12 14:45:56.332251 7f9d65bdc700  5 -- op tracker -- , seq: 747356, time: 2014-06-12 14:45:56.332245, event: started, op: osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.332262 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] execute_ctx 0x25b1e600
2014-06-12 14:45:56.332281 7f9d65bdc700  5 -- op tracker -- , seq: 747356, time: 2014-06-12 14:45:56.332276, event: started, op: osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.332290 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] do_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] ov 173'2456 av 173'2457 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:56.332306 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] do_osd_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096]
2014-06-12 14:45:56.332319 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:56.332334 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] do_osd_op  write 2936832~516096
2014-06-12 14:45:56.332352 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] do_osd_op_effects on session 0xdd06400
2014-06-12 14:45:56.332365 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 snapset=0xee5ed58  snapc=0=[]
2014-06-12 14:45:56.332379 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:56.332392 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] finish_ctx d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 0x25b1e600 op modify  
2014-06-12 14:45:56.332406 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean]  set mtime to 2014-06-12 08:44:58.710850
2014-06-12 14:45:56.332428 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean]  final snapset 0=[]:[]+head in d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.332449 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean]  zeroing write result code 0
2014-06-12 14:45:56.332462 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] new_repop rep_tid 236155 on osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.332479 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] issue_repop rep_tid 236155 o d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.332505 7f9d65bdc700  5 -- op tracker -- , seq: 747356, time: 2014-06-12 14:45:56.332500, event: waiting for subops from 21, op: osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.332528 7f9d65bdc700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:56.332552 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2456 (0'0,173'2456] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] append_log log((0'0,173'2456], crt=0'0) [173'2457 (173'2456) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140621 2014-06-12 08:44:58.710850]
2014-06-12 14:45:56.332576 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2456 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] add_log_entry 173'2457 (173'2456) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140621 2014-06-12 08:44:58.710850
2014-06-12 14:45:56.332598 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2456 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] append_log  adding 1 keys
2014-06-12 14:45:56.332632 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2457, trimmed: 
2014-06-12 14:45:56.332656 7f9d65bdc700  5 queue_transactions existing osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.332663 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.332664 7f9d65bdc700  5 queue_op 0x101cc7e0 seq 2314 osr(3.1bc 0x276d1b0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:45:56.332677 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519320 (1 active)
2014-06-12 14:45:56.332691 7f9d7e96a700  5 _do_op 0x101cc7e0 seq 2314 osr(3.1bc 0x276d1b0)/0x276d1b0 start
2014-06-12 14:45:56.332672 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2456 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] eval_repop repgather(0x51dde00 173'2457 rep_tid=236155 committed?=0 applied?=0 lock=0 op=osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.332695 7f9d7e96a700 10 _do_transaction on 0x384e2780
2014-06-12 14:45:56.332696 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2a903200 finish
2014-06-12 14:45:56.332701 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.332701 7f9d7e96a700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.332704 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.332802 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.332828 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.332884 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.332903 7f9d7e96a700 15 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.332908 7f9d7e96a700 10 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1 r = 0
2014-06-12 14:45:56.332914 7f9d7e96a700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.332926 7f9d7e96a700 15 _write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 2936832~516096
2014-06-12 14:45:56.333062 7f9d7e96a700 10 lookup_strip_header done  cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.333175 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:45:56.335092 7f9d7e96a700 10 _generic_write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 2936832~516096 = 0
2014-06-12 14:45:56.335152 7f9d7e96a700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.335174 7f9d7e96a700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.335182 7f9d7e96a700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.335190 7f9d7e96a700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.335409 7f9d7e96a700 20 set_header setting 16575 cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 parent seq 0
2014-06-12 14:45:56.335482 7f9d7e96a700 20 set_header setting 2006 cid meta oid 1fe11b06/pglog_3.1bc/0//-1 parent seq 0
2014-06-12 14:45:56.335499 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.335509 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:45:56.337788 7f9d7e96a700 10 _do_op 0x101cc7e0 seq 2314 r = 0, finisher 0x4f61dc0 0x31aec7c0
2014-06-12 14:45:56.337806 7f9d7e96a700 10 _finish_op 0x101cc7e0 seq 2314 osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.337815 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519320 (0 active)
2014-06-12 14:45:56.337819 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.337814 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2456 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] op_commit: 236155
2014-06-12 14:45:56.337850 7f9d7715b700  5 -- op tracker -- , seq: 747356, time: 2014-06-12 14:45:56.337839, event: op_commit, op: osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.337870 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2456 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] op_applied: 236155
2014-06-12 14:45:56.337893 7f9d7795c700  5 -- op tracker -- , seq: 747356, time: 2014-06-12 14:45:56.337887, event: op_applied, op: osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.337903 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2456 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] op_applied on primary on version 173'2457
2014-06-12 14:45:56.342336 7f9d44101700  5 -- op tracker -- , seq: 747357, time: 2014-06-12 14:45:56.342276, event: header_read, op: osd_sub_op_reply(client.4291.0:140621 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.342351 7f9d44101700  5 -- op tracker -- , seq: 747357, time: 2014-06-12 14:45:56.342278, event: throttled, op: osd_sub_op_reply(client.4291.0:140621 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.342364 7f9d44101700  5 -- op tracker -- , seq: 747357, time: 2014-06-12 14:45:56.342293, event: all_read, op: osd_sub_op_reply(client.4291.0:140621 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.342384 7f9d44101700  5 -- op tracker -- , seq: 747357, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140621 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.342390 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140621 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.342396 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.342406 7f9d44101700 15 osd.2 173 enqueue_op 0x59dd000 prio 196 cost 0 latency 0.000129 osd_sub_op_reply(client.4291.0:140621 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.342440 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.342450 7f9d6abe6700 10 osd.2 173 dequeue_op 0x59dd000 prio 196 cost 0 latency 0.000174 osd_sub_op_reply(client.4291.0:140621 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 pg pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2456 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean]
2014-06-12 14:45:56.342474 7f9d6abe6700  5 -- op tracker -- , seq: 747357, time: 2014-06-12 14:45:56.342471, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140621 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.342483 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2456 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] handle_message: 0x59dd000
2014-06-12 14:45:56.342502 7f9d6abe6700  5 -- op tracker -- , seq: 747357, time: 2014-06-12 14:45:56.342498, event: started, op: osd_sub_op_reply(client.4291.0:140621 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.342508 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2456 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] sub_op_modify_reply: tid 236155 op  ack_type 4 from 21
2014-06-12 14:45:56.342533 7f9d6abe6700  5 -- op tracker -- , seq: 747356, time: 2014-06-12 14:45:56.342524, event: sub_op_commit_rec, op: osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.342542 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2456 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] repop_all_applied: repop tid 236155 all applied 
2014-06-12 14:45:56.342556 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2456 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] eval_repop repgather(0x51dde00 173'2457 rep_tid=236155 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.342574 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2456 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean]  sending ack on repgather(0x51dde00 173'2457 rep_tid=236155 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x25914a00
2014-06-12 14:45:56.342609 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2456 crt=0'0 lcod 173'2455 mlcod 173'2455 active+clean] repop_all_committed: repop tid 236155 all committed 
2014-06-12 14:45:56.342619 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2455 active+clean] eval_repop repgather(0x51dde00 173'2457 rep_tid=236155 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.342638 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2455 active+clean] log_op_stats osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.012129 lat 0.012158
2014-06-12 14:45:56.342658 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2455 active+clean] publish_stats_to_osd 173:3182
2014-06-12 14:45:56.342674 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2455 active+clean]  sending commit on repgather(0x51dde00 173'2457 rep_tid=236155 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x28ea3400
2014-06-12 14:45:56.342706 7f9d6abe6700  5 -- op tracker -- , seq: 747356, time: 2014-06-12 14:45:56.342700, event: commit_sent, op: osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.342715 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2455 active+clean]  requeue_ops 
2014-06-12 14:45:56.342726 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean]  removing repgather(0x51dde00 173'2457 rep_tid=236155 committed?=1 applied?=1 op=osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.342742 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean]    q front is repgather(0x51dde00 173'2457 rep_tid=236155 committed?=1 applied?=1 op=osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.342759 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] remove_repop repgather(0x51dde00 173'2457 rep_tid=236155 committed?=1 applied?=1 op=osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.342775 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean]  requeue_ops 
2014-06-12 14:45:56.342803 7f9d6abe6700  5 -- op tracker -- , seq: 747356, time: 2014-06-12 14:45:56.342798, event: done, op: osd_op(client.4291.0:140621 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.342822 7f9d6abe6700 10 osd.2 173 dequeue_op 0x59dd000 finish
2014-06-12 14:45:56.342831 7f9d6abe6700  5 -- op tracker -- , seq: 747357, time: 2014-06-12 14:45:56.342827, event: done, op: osd_sub_op_reply(client.4291.0:140621 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.342842 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.342845 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.346198 7f9d4ed71700  5 -- op tracker -- , seq: 747358, time: 2014-06-12 14:45:56.345153, event: header_read, op: osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.346212 7f9d4ed71700  5 -- op tracker -- , seq: 747358, time: 2014-06-12 14:45:56.345155, event: throttled, op: osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.346222 7f9d4ed71700  5 -- op tracker -- , seq: 747358, time: 2014-06-12 14:45:56.346142, event: all_read, op: osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.346232 7f9d4ed71700  5 -- op tracker -- , seq: 747358, time: 0.000000, event: dispatched, op: osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.346239 7f9d4ed71700 20 osd.2 173 should_share_map client.4291 172.16.96.220:0/1241802 173
2014-06-12 14:45:56.346265 7f9d4ed71700 15 osd.2 173 enqueue_op 0x2a903300 prio 63 cost 516096 latency 0.001112 osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.346297 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.346304 7f9d66bde700 10 osd.2 173 dequeue_op 0x2a903300 prio 63 cost 516096 latency 0.001150 osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean]
2014-06-12 14:45:56.346328 7f9d66bde700  5 -- op tracker -- , seq: 747358, time: 2014-06-12 14:45:56.346323, event: reached_pg, op: osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.346336 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:56.346351 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] handle_message: 0x2a903300
2014-06-12 14:45:56.346361 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] do_op osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:56.346386 7f9d66bde700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_'
2014-06-12 14:45:56.346623 7f9d66bde700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_' = 0
2014-06-12 14:45:56.346640 7f9d66bde700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset'
2014-06-12 14:45:56.346740 7f9d66bde700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset' = 0
2014-06-12 14:45:56.346750 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] populate_obc_watchers d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.346762 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.346771 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] get_object_context: creating obc from disk: 0x5cfb080
2014-06-12 14:45:56.346780 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] get_object_context: 0x5cfb080 d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 rwstate(none n=0 w=0) oi: d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2457 client.4291.0:140621 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2457) ssc: 0x72b87e0 snapset: 0=[]:[]+head
2014-06-12 14:45:56.346796 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] find_object_context d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 @head oi=d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2457 client.4291.0:140621 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2457)
2014-06-12 14:45:56.346821 7f9d66bde700  5 -- op tracker -- , seq: 747358, time: 2014-06-12 14:45:56.346815, event: started, op: osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.346829 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] execute_ctx 0x7ad1200
2014-06-12 14:45:56.346853 7f9d66bde700  5 -- op tracker -- , seq: 747358, time: 2014-06-12 14:45:56.346848, event: started, op: osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.346860 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] do_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] ov 173'2457 av 173'2458 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:56.346873 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] do_osd_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096]
2014-06-12 14:45:56.346884 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:56.346897 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] do_osd_op  write 3452928~516096
2014-06-12 14:45:56.346911 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] do_osd_op_effects on session 0xdd06400
2014-06-12 14:45:56.346922 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 snapset=0x72b8818  snapc=0=[]
2014-06-12 14:45:56.346933 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:56.346943 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] finish_ctx d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 0x7ad1200 op modify  
2014-06-12 14:45:56.346955 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean]  set mtime to 2014-06-12 08:44:58.725517
2014-06-12 14:45:56.346978 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean]  final snapset 0=[]:[]+head in d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.346996 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean]  zeroing write result code 0
2014-06-12 14:45:56.347007 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] new_repop rep_tid 236156 on osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.347023 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] issue_repop rep_tid 236156 o d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.347046 7f9d66bde700  5 -- op tracker -- , seq: 747358, time: 2014-06-12 14:45:56.347041, event: waiting for subops from 21, op: osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.347065 7f9d66bde700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:56.347087 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2457 (0'0,173'2457] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] append_log log((0'0,173'2457], crt=0'0) [173'2458 (173'2457) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140622 2014-06-12 08:44:58.725517]
2014-06-12 14:45:56.347110 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2457 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] add_log_entry 173'2458 (173'2457) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140622 2014-06-12 08:44:58.725517
2014-06-12 14:45:56.347130 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2457 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] append_log  adding 1 keys
2014-06-12 14:45:56.347161 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2458, trimmed: 
2014-06-12 14:45:56.347183 7f9d66bde700  5 queue_transactions existing osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.347187 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.347188 7f9d66bde700  5 queue_op 0x216fe60 seq 2315 osr(3.1bc 0x276d1b0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:45:56.347196 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2457 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] eval_repop repgather(0x2b62a900 173'2458 rep_tid=236156 committed?=0 applied?=0 lock=0 op=osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.347214 7f9d66bde700 10 osd.2 173 dequeue_op 0x2a903300 finish
2014-06-12 14:45:56.347218 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.347220 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.347228 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519320 (1 active)
2014-06-12 14:45:56.347234 7f9d7d167700  5 _do_op 0x216fe60 seq 2315 osr(3.1bc 0x276d1b0)/0x276d1b0 start
2014-06-12 14:45:56.347237 7f9d7d167700 10 _do_transaction on 0x5352c00
2014-06-12 14:45:56.347249 7f9d7d167700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.347351 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.347371 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.347428 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.347445 7f9d7d167700 15 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.347450 7f9d7d167700 10 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1 r = 0
2014-06-12 14:45:56.347456 7f9d7d167700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.347469 7f9d7d167700 15 _write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 3452928~516096
2014-06-12 14:45:56.347603 7f9d7d167700 10 lookup_strip_header done  cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.347734 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:45:56.349579 7f9d7d167700 10 _generic_write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 3452928~516096 = 0
2014-06-12 14:45:56.349651 7f9d7d167700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.349668 7f9d7d167700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.349674 7f9d7d167700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.349686 7f9d7d167700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.349903 7f9d7d167700 20 set_header setting 16575 cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 parent seq 0
2014-06-12 14:45:56.349967 7f9d7d167700 20 set_header setting 2006 cid meta oid 1fe11b06/pglog_3.1bc/0//-1 parent seq 0
2014-06-12 14:45:56.349982 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.349990 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:45:56.351952 7f9d7d167700 10 _do_op 0x216fe60 seq 2315 r = 0, finisher 0x73300e0 0x3373de00
2014-06-12 14:45:56.351984 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2457 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] op_commit: 236156
2014-06-12 14:45:56.352015 7f9d7715b700  5 -- op tracker -- , seq: 747358, time: 2014-06-12 14:45:56.352006, event: op_commit, op: osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.352033 7f9d7d167700 10 _finish_op 0x216fe60 seq 2315 osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.352051 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2457 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] op_applied: 236156
2014-06-12 14:45:56.352087 7f9d7795c700  5 -- op tracker -- , seq: 747358, time: 2014-06-12 14:45:56.352074, event: op_applied, op: osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.352096 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2457 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] op_applied on primary on version 173'2458
2014-06-12 14:45:56.352116 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519320 (0 active)
2014-06-12 14:45:56.352123 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.354380 7f9d44101700  5 -- op tracker -- , seq: 747359, time: 2014-06-12 14:45:56.354328, event: header_read, op: osd_sub_op_reply(client.4291.0:140622 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.354392 7f9d44101700  5 -- op tracker -- , seq: 747359, time: 2014-06-12 14:45:56.354330, event: throttled, op: osd_sub_op_reply(client.4291.0:140622 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.354400 7f9d44101700  5 -- op tracker -- , seq: 747359, time: 2014-06-12 14:45:56.354344, event: all_read, op: osd_sub_op_reply(client.4291.0:140622 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.354408 7f9d44101700  5 -- op tracker -- , seq: 747359, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140622 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.354414 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140622 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.354421 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.354431 7f9d44101700 15 osd.2 173 enqueue_op 0x59dce00 prio 196 cost 0 latency 0.000103 osd_sub_op_reply(client.4291.0:140622 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.354465 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.354474 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x59dce00 prio 196 cost 0 latency 0.000145 osd_sub_op_reply(client.4291.0:140622 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 pg pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2457 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean]
2014-06-12 14:45:56.354495 7f9d6a3e5700  5 -- op tracker -- , seq: 747359, time: 2014-06-12 14:45:56.354492, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140622 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.354503 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2457 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] handle_message: 0x59dce00
2014-06-12 14:45:56.354522 7f9d6a3e5700  5 -- op tracker -- , seq: 747359, time: 2014-06-12 14:45:56.354518, event: started, op: osd_sub_op_reply(client.4291.0:140622 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.354528 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2457 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] sub_op_modify_reply: tid 236156 op  ack_type 4 from 21
2014-06-12 14:45:56.354550 7f9d6a3e5700  5 -- op tracker -- , seq: 747358, time: 2014-06-12 14:45:56.354543, event: sub_op_commit_rec, op: osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.354559 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2457 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] repop_all_applied: repop tid 236156 all applied 
2014-06-12 14:45:56.354574 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2457 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] eval_repop repgather(0x2b62a900 173'2458 rep_tid=236156 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.354592 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2457 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean]  sending ack on repgather(0x2b62a900 173'2458 rep_tid=236156 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x30f6f680
2014-06-12 14:45:56.354627 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2457 crt=0'0 lcod 173'2456 mlcod 173'2456 active+clean] repop_all_committed: repop tid 236156 all committed 
2014-06-12 14:45:56.354636 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2456 active+clean] eval_repop repgather(0x2b62a900 173'2458 rep_tid=236156 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.354653 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2456 active+clean] log_op_stats osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.009473 lat 0.009499
2014-06-12 14:45:56.354673 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2456 active+clean] publish_stats_to_osd 173:3183
2014-06-12 14:45:56.354689 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2456 active+clean]  sending commit on repgather(0x2b62a900 173'2458 rep_tid=236156 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0xee35400
2014-06-12 14:45:56.354723 7f9d6a3e5700  5 -- op tracker -- , seq: 747358, time: 2014-06-12 14:45:56.354718, event: commit_sent, op: osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.354731 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2456 active+clean]  requeue_ops 
2014-06-12 14:45:56.354742 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean]  removing repgather(0x2b62a900 173'2458 rep_tid=236156 committed?=1 applied?=1 op=osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.354758 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean]    q front is repgather(0x2b62a900 173'2458 rep_tid=236156 committed?=1 applied?=1 op=osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.354773 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] remove_repop repgather(0x2b62a900 173'2458 rep_tid=236156 committed?=1 applied?=1 op=osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.354789 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean]  requeue_ops 
2014-06-12 14:45:56.354816 7f9d6a3e5700  5 -- op tracker -- , seq: 747358, time: 2014-06-12 14:45:56.354810, event: done, op: osd_op(client.4291.0:140622 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.354837 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x59dce00 finish
2014-06-12 14:45:56.354846 7f9d6a3e5700  5 -- op tracker -- , seq: 747359, time: 2014-06-12 14:45:56.354842, event: done, op: osd_sub_op_reply(client.4291.0:140622 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.354856 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.354859 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.355659 7f9d4ed71700  5 -- op tracker -- , seq: 747360, time: 2014-06-12 14:45:56.355608, event: header_read, op: osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.355674 7f9d4ed71700  5 -- op tracker -- , seq: 747360, time: 2014-06-12 14:45:56.355611, event: throttled, op: osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.355686 7f9d4ed71700  5 -- op tracker -- , seq: 747360, time: 2014-06-12 14:45:56.355629, event: all_read, op: osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.355697 7f9d4ed71700  5 -- op tracker -- , seq: 747360, time: 0.000000, event: dispatched, op: osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.355706 7f9d4ed71700 20 osd.2 173 should_share_map client.4291 172.16.96.220:0/1241802 173
2014-06-12 14:45:56.355732 7f9d4ed71700 15 osd.2 173 enqueue_op 0x8dbc100 prio 63 cost 4096 latency 0.000124 osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.355756 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.355764 7f9d63bd8700 10 osd.2 173 dequeue_op 0x8dbc100 prio 63 cost 4096 latency 0.000156 osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean]
2014-06-12 14:45:56.355786 7f9d63bd8700  5 -- op tracker -- , seq: 747360, time: 2014-06-12 14:45:56.355782, event: reached_pg, op: osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.355794 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:56.355804 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] handle_message: 0x8dbc100
2014-06-12 14:45:56.355810 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] do_op osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:56.355827 7f9d63bd8700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_'
2014-06-12 14:45:56.355992 7f9d63bd8700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_' = 0
2014-06-12 14:45:56.356005 7f9d63bd8700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset'
2014-06-12 14:45:56.356071 7f9d63bd8700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset' = 0
2014-06-12 14:45:56.356079 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] populate_obc_watchers d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.356088 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.356094 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] get_object_context: creating obc from disk: 0x14655600
2014-06-12 14:45:56.356099 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] get_object_context: 0x14655600 d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 rwstate(none n=0 w=0) oi: d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2458 client.4291.0:140622 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2458) ssc: 0x24d4a9a0 snapset: 0=[]:[]+head
2014-06-12 14:45:56.356111 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] find_object_context d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 @head oi=d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2458 client.4291.0:140622 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2458)
2014-06-12 14:45:56.356129 7f9d63bd8700  5 -- op tracker -- , seq: 747360, time: 2014-06-12 14:45:56.356126, event: started, op: osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.356136 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] execute_ctx 0xd21a000
2014-06-12 14:45:56.356147 7f9d63bd8700  5 -- op tracker -- , seq: 747360, time: 2014-06-12 14:45:56.356144, event: started, op: osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.356152 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] do_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] ov 173'2458 av 173'2459 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:56.356162 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] do_osd_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096]
2014-06-12 14:45:56.356168 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:56.356176 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] do_osd_op  write 3969024~4096
2014-06-12 14:45:56.356187 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] do_osd_op_effects on session 0xdd06400
2014-06-12 14:45:56.356195 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 snapset=0x24d4a9d8  snapc=0=[]
2014-06-12 14:45:56.356203 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:56.356209 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] finish_ctx d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 0xd21a000 op modify  
2014-06-12 14:45:56.356217 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean]  set mtime to 2014-06-12 08:44:58.736927
2014-06-12 14:45:56.356230 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean]  final snapset 0=[]:[]+head in d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.356242 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean]  zeroing write result code 0
2014-06-12 14:45:56.356250 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] new_repop rep_tid 236157 on osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.356261 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] issue_repop rep_tid 236157 o d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.356278 7f9d63bd8700  5 -- op tracker -- , seq: 747360, time: 2014-06-12 14:45:56.356275, event: waiting for subops from 21, op: osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.356292 7f9d63bd8700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:56.356307 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2458 (0'0,173'2458] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] append_log log((0'0,173'2458], crt=0'0) [173'2459 (173'2458) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140623 2014-06-12 08:44:58.736927]
2014-06-12 14:45:56.356324 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2458 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] add_log_entry 173'2459 (173'2458) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140623 2014-06-12 08:44:58.736927
2014-06-12 14:45:56.356337 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2458 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] append_log  adding 1 keys
2014-06-12 14:45:56.356356 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2459, trimmed: 
2014-06-12 14:45:56.356372 7f9d63bd8700  5 queue_transactions existing osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.356377 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.356378 7f9d63bd8700  5 queue_op 0x663faa0 seq 2316 osr(3.1bc 0x276d1b0) 6389 bytes   (queue has 1 ops and 6389 bytes)
2014-06-12 14:45:56.356387 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519320 (1 active)
2014-06-12 14:45:56.356382 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2458 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] eval_repop repgather(0x52f43c0 173'2459 rep_tid=236157 committed?=0 applied?=0 lock=0 op=osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.356393 7f9d63bd8700 10 osd.2 173 dequeue_op 0x8dbc100 finish
2014-06-12 14:45:56.356394 7f9d7d968700  5 _do_op 0x663faa0 seq 2316 osr(3.1bc 0x276d1b0)/0x276d1b0 start
2014-06-12 14:45:56.356396 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.356397 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.356398 7f9d7d968700 10 _do_transaction on 0x2ad99200
2014-06-12 14:45:56.356405 7f9d7d968700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.356505 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.356527 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.356586 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.356611 7f9d7d968700 15 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.356619 7f9d7d968700 10 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1 r = 0
2014-06-12 14:45:56.356625 7f9d7d968700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.356642 7f9d7d968700 15 _write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 3969024~4096
2014-06-12 14:45:56.356918 7f9d7d968700 10 lookup_strip_header done  cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.356932 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:45:56.356955 7f9d7d968700 10 _generic_write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 3969024~4096 = 0
2014-06-12 14:45:56.356966 7f9d7d968700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.356979 7f9d7d968700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.356987 7f9d7d968700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.356995 7f9d7d968700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.357107 7f9d7d968700 20 set_header setting 16575 cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 parent seq 0
2014-06-12 14:45:56.357129 7f9d7d968700 20 set_header setting 2006 cid meta oid 1fe11b06/pglog_3.1bc/0//-1 parent seq 0
2014-06-12 14:45:56.357143 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.357155 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:45:56.357230 7f9d7d968700 10 _do_op 0x663faa0 seq 2316 r = 0, finisher 0x65e6040 0x36ca800
2014-06-12 14:45:56.357240 7f9d7d968700 10 _finish_op 0x663faa0 seq 2316 osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.357243 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2458 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] op_commit: 236157
2014-06-12 14:45:56.357253 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519320 (0 active)
2014-06-12 14:45:56.357256 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.357258 7f9d7715b700  5 -- op tracker -- , seq: 747360, time: 2014-06-12 14:45:56.357254, event: op_commit, op: osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.357267 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2458 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] op_applied: 236157
2014-06-12 14:45:56.357294 7f9d7795c700  5 -- op tracker -- , seq: 747360, time: 2014-06-12 14:45:56.357287, event: op_applied, op: osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.357303 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2458 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] op_applied on primary on version 173'2459
2014-06-12 14:45:56.357503 7f9d44101700  5 -- op tracker -- , seq: 747361, time: 2014-06-12 14:45:56.357481, event: header_read, op: osd_sub_op_reply(client.4291.0:140623 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.357510 7f9d44101700  5 -- op tracker -- , seq: 747361, time: 2014-06-12 14:45:56.357482, event: throttled, op: osd_sub_op_reply(client.4291.0:140623 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.357514 7f9d44101700  5 -- op tracker -- , seq: 747361, time: 2014-06-12 14:45:56.357488, event: all_read, op: osd_sub_op_reply(client.4291.0:140623 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.357518 7f9d44101700  5 -- op tracker -- , seq: 747361, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140623 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.357521 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140623 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.357524 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.357527 7f9d44101700 15 osd.2 173 enqueue_op 0x20355d00 prio 196 cost 0 latency 0.000046 osd_sub_op_reply(client.4291.0:140623 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.357545 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.357549 7f9d653db700 10 osd.2 173 dequeue_op 0x20355d00 prio 196 cost 0 latency 0.000068 osd_sub_op_reply(client.4291.0:140623 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 pg pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2458 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean]
2014-06-12 14:45:56.357564 7f9d653db700  5 -- op tracker -- , seq: 747361, time: 2014-06-12 14:45:56.357561, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140623 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.357569 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2458 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] handle_message: 0x20355d00
2014-06-12 14:45:56.357579 7f9d653db700  5 -- op tracker -- , seq: 747361, time: 2014-06-12 14:45:56.357577, event: started, op: osd_sub_op_reply(client.4291.0:140623 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.357582 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2458 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] sub_op_modify_reply: tid 236157 op  ack_type 4 from 21
2014-06-12 14:45:56.357593 7f9d653db700  5 -- op tracker -- , seq: 747360, time: 2014-06-12 14:45:56.357589, event: sub_op_commit_rec, op: osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.357598 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2458 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] repop_all_applied: repop tid 236157 all applied 
2014-06-12 14:45:56.357605 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2458 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] eval_repop repgather(0x52f43c0 173'2459 rep_tid=236157 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.357635 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2458 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean]  sending ack on repgather(0x52f43c0 173'2459 rep_tid=236157 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x28ea2000
2014-06-12 14:45:56.357657 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2458 crt=0'0 lcod 173'2457 mlcod 173'2457 active+clean] repop_all_committed: repop tid 236157 all committed 
2014-06-12 14:45:56.357668 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2457 active+clean] eval_repop repgather(0x52f43c0 173'2459 rep_tid=236157 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.357685 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2457 active+clean] log_op_stats osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 inb 4375 outb 0 rlat 0.002048 lat 0.002077
2014-06-12 14:45:56.357695 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2457 active+clean] publish_stats_to_osd 173:3184
2014-06-12 14:45:56.357703 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2457 active+clean]  sending commit on repgather(0x52f43c0 173'2459 rep_tid=236157 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0xbc08000
2014-06-12 14:45:56.357727 7f9d653db700  5 -- op tracker -- , seq: 747360, time: 2014-06-12 14:45:56.357721, event: commit_sent, op: osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.357736 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2457 active+clean]  requeue_ops 
2014-06-12 14:45:56.357747 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean]  removing repgather(0x52f43c0 173'2459 rep_tid=236157 committed?=1 applied?=1 op=osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.357758 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean]    q front is repgather(0x52f43c0 173'2459 rep_tid=236157 committed?=1 applied?=1 op=osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.357767 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] remove_repop repgather(0x52f43c0 173'2459 rep_tid=236157 committed?=1 applied?=1 op=osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.357776 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean]  requeue_ops 
2014-06-12 14:45:56.357790 7f9d653db700  5 -- op tracker -- , seq: 747360, time: 2014-06-12 14:45:56.357787, event: done, op: osd_op(client.4291.0:140623 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~4096] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.357800 7f9d653db700 10 osd.2 173 dequeue_op 0x20355d00 finish
2014-06-12 14:45:56.357804 7f9d653db700  5 -- op tracker -- , seq: 747361, time: 2014-06-12 14:45:56.357802, event: done, op: osd_sub_op_reply(client.4291.0:140623 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.357811 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.357813 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.359689 7f9d4ed71700  5 -- op tracker -- , seq: 747362, time: 2014-06-12 14:45:56.359221, event: header_read, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.359704 7f9d4ed71700  5 -- op tracker -- , seq: 747362, time: 2014-06-12 14:45:56.359223, event: throttled, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.359716 7f9d4ed71700  5 -- op tracker -- , seq: 747362, time: 2014-06-12 14:45:56.359651, event: all_read, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.359727 7f9d4ed71700  5 -- op tracker -- , seq: 747362, time: 0.000000, event: dispatched, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.359736 7f9d4ed71700 20 osd.2 173 should_share_map client.4291 172.16.96.220:0/1241802 173
2014-06-12 14:45:56.359759 7f9d4ed71700 15 osd.2 173 enqueue_op 0x2911000 prio 63 cost 221184 latency 0.000538 osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.359786 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.359796 7f9d69be4700 10 osd.2 173 dequeue_op 0x2911000 prio 63 cost 221184 latency 0.000575 osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean]
2014-06-12 14:45:56.359822 7f9d69be4700  5 -- op tracker -- , seq: 747362, time: 2014-06-12 14:45:56.359817, event: reached_pg, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.359833 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:56.359847 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] handle_message: 0x2911000
2014-06-12 14:45:56.359859 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] do_op osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:56.359883 7f9d69be4700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_'
2014-06-12 14:45:56.360041 7f9d69be4700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 '_' = 0
2014-06-12 14:45:56.360057 7f9d69be4700 15 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset'
2014-06-12 14:45:56.360160 7f9d69be4700 10 getattr 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 'snapset' = 0
2014-06-12 14:45:56.360171 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] populate_obc_watchers d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.360184 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.360196 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] get_object_context: creating obc from disk: 0x11f658c0
2014-06-12 14:45:56.360207 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] get_object_context: 0x11f658c0 d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 rwstate(none n=0 w=0) oi: d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2459 client.4291.0:140623 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2459) ssc: 0x2ef8e2a0 snapset: 0=[]:[]+head
2014-06-12 14:45:56.360224 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] find_object_context d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 @head oi=d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3(173'2459 client.4291.0:140623 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2459)
2014-06-12 14:45:56.360250 7f9d69be4700  5 -- op tracker -- , seq: 747362, time: 2014-06-12 14:45:56.360244, event: started, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.360260 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] execute_ctx 0x2963d000
2014-06-12 14:45:56.360279 7f9d69be4700  5 -- op tracker -- , seq: 747362, time: 2014-06-12 14:45:56.360274, event: started, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.360288 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] do_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] ov 173'2459 av 173'2460 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:56.360305 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] do_osd_op d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184]
2014-06-12 14:45:56.360317 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:56.360332 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] do_osd_op  write 3973120~221184
2014-06-12 14:45:56.360348 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] do_osd_op_effects on session 0xdd06400
2014-06-12 14:45:56.360360 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 snapset=0x2ef8e2d8  snapc=0=[]
2014-06-12 14:45:56.360373 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] make_writeable d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:56.360385 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] finish_ctx d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 0x2963d000 op modify  
2014-06-12 14:45:56.360399 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean]  set mtime to 2014-06-12 08:44:58.740099
2014-06-12 14:45:56.360420 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean]  final snapset 0=[]:[]+head in d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.360439 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean]  zeroing write result code 0
2014-06-12 14:45:56.360452 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] new_repop rep_tid 236158 on osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.360468 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] issue_repop rep_tid 236158 o d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.360492 7f9d69be4700  5 -- op tracker -- , seq: 747362, time: 2014-06-12 14:45:56.360487, event: waiting for subops from 21, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.360510 7f9d69be4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:56.360532 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2459 (0'0,173'2459] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] append_log log((0'0,173'2459], crt=0'0) [173'2460 (173'2459) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140624 2014-06-12 08:44:58.740099]
2014-06-12 14:45:56.360555 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] add_log_entry 173'2460 (173'2459) modify   d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 by client.4291.0:140624 2014-06-12 08:44:58.740099
2014-06-12 14:45:56.360577 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] append_log  adding 1 keys
2014-06-12 14:45:56.360609 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2460, trimmed: 
2014-06-12 14:45:56.360629 7f9d69be4700  5 queue_transactions existing osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.360635 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.360636 7f9d69be4700  5 queue_op 0x7fbbc80 seq 2317 osr(3.1bc 0x276d1b0) 223477 bytes   (queue has 1 ops and 223477 bytes)
2014-06-12 14:45:56.360643 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] eval_repop repgather(0x2a37b200 173'2460 rep_tid=236158 committed?=0 applied?=0 lock=0 op=osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.360662 7f9d69be4700 10 osd.2 173 dequeue_op 0x2911000 finish
2014-06-12 14:45:56.360666 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.360669 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.360690 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519320 (1 active)
2014-06-12 14:45:56.360694 7f9d7815d700  5 _do_op 0x7fbbc80 seq 2317 osr(3.1bc 0x276d1b0)/0x276d1b0 start
2014-06-12 14:45:56.360698 7f9d7815d700 10 _do_transaction on 0x51b3c80
2014-06-12 14:45:56.360703 7f9d7815d700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.360799 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.360825 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.360882 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.360901 7f9d7815d700 15 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.360909 7f9d7815d700 10 _omap_rmkeys meta/1fe11b06/pglog_3.1bc/0//-1 r = 0
2014-06-12 14:45:56.360915 7f9d7815d700 15 _omap_setkeys meta/1fe11b06/pglog_3.1bc/0//-1
2014-06-12 14:45:56.360931 7f9d7815d700 15 _write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 3973120~221184
2014-06-12 14:45:56.361070 7f9d7815d700 10 lookup_strip_header done  cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.361262 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:45:56.362049 7f9d7815d700 10 _generic_write 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 3973120~221184 = 0
2014-06-12 14:45:56.362076 7f9d7815d700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.362090 7f9d7815d700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.362097 7f9d7815d700 15 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3
2014-06-12 14:45:56.362106 7f9d7815d700 10 _setattrs 3.1bc_head/d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 = 0
2014-06-12 14:45:56.362257 7f9d7815d700 20 set_header setting 16575 cid 3.1bc_head oid d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 parent seq 0
2014-06-12 14:45:56.362299 7f9d7815d700 20 set_header setting 2006 cid meta oid 1fe11b06/pglog_3.1bc/0//-1 parent seq 0
2014-06-12 14:45:56.362315 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.362326 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:45:56.363220 7f9d7815d700 10 _do_op 0x7fbbc80 seq 2317 r = 0, finisher 0x66b3fc0 0x3907de80
2014-06-12 14:45:56.363231 7f9d7815d700 10 _finish_op 0x7fbbc80 seq 2317 osr(3.1bc 0x276d1b0)/0x276d1b0
2014-06-12 14:45:56.363238 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519320 (0 active)
2014-06-12 14:45:56.363240 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.363244 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] op_applied: 236158
2014-06-12 14:45:56.363272 7f9d7795c700  5 -- op tracker -- , seq: 747362, time: 2014-06-12 14:45:56.363264, event: op_applied, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.363281 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] op_applied on primary on version 173'2460
2014-06-12 14:45:56.363298 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] op_commit: 236158
2014-06-12 14:45:56.363322 7f9d7715b700  5 -- op tracker -- , seq: 747362, time: 2014-06-12 14:45:56.363315, event: op_commit, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.364462 7f9d44101700  5 -- op tracker -- , seq: 747363, time: 2014-06-12 14:45:56.364418, event: header_read, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.364474 7f9d44101700  5 -- op tracker -- , seq: 747363, time: 2014-06-12 14:45:56.364419, event: throttled, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.364482 7f9d44101700  5 -- op tracker -- , seq: 747363, time: 2014-06-12 14:45:56.364433, event: all_read, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.364489 7f9d44101700  5 -- op tracker -- , seq: 747363, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.364500 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:45:56.364507 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.364514 7f9d44101700 15 osd.2 173 enqueue_op 0x20355500 prio 196 cost 0 latency 0.000096 osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0) v2
2014-06-12 14:45:56.364542 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.364550 7f9d633d7700 10 osd.2 173 dequeue_op 0x20355500 prio 196 cost 0 latency 0.000131 osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0) v2 pg pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean]
2014-06-12 14:45:56.364567 7f9d633d7700  5 -- op tracker -- , seq: 747363, time: 2014-06-12 14:45:56.364564, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.364578 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] handle_message: 0x20355500
2014-06-12 14:45:56.364592 7f9d633d7700  5 -- op tracker -- , seq: 747363, time: 2014-06-12 14:45:56.364589, event: started, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.364597 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] sub_op_modify_reply: tid 236158 op  ack_type 1 from 21
2014-06-12 14:45:56.364612 7f9d633d7700  5 -- op tracker -- , seq: 747362, time: 2014-06-12 14:45:56.364607, event: sub_op_applied_rec, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.364618 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] repop_all_applied: repop tid 236158 all applied 
2014-06-12 14:45:56.364627 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] eval_repop repgather(0x2a37b200 173'2460 rep_tid=236158 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.364641 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean]  sending ack on repgather(0x2a37b200 173'2460 rep_tid=236158 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x55a7900
2014-06-12 14:45:56.364667 7f9d633d7700 10 osd.2 173 dequeue_op 0x20355500 finish
2014-06-12 14:45:56.364673 7f9d633d7700  5 -- op tracker -- , seq: 747363, time: 2014-06-12 14:45:56.364670, event: done, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ack, result = 0)
2014-06-12 14:45:56.364686 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.364688 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.364698 7f9d44101700  5 -- op tracker -- , seq: 747364, time: 2014-06-12 14:45:56.364537, event: header_read, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.364705 7f9d44101700  5 -- op tracker -- , seq: 747364, time: 2014-06-12 14:45:56.364537, event: throttled, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.364711 7f9d44101700  5 -- op tracker -- , seq: 747364, time: 2014-06-12 14:45:56.364550, event: all_read, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.364721 7f9d44101700  5 -- op tracker -- , seq: 747364, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.364725 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.364729 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.364733 7f9d44101700 15 osd.2 173 enqueue_op 0x20355000 prio 196 cost 0 latency 0.000196 osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.364754 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.364758 7f9d623d5700 10 osd.2 173 dequeue_op 0x20355000 prio 196 cost 0 latency 0.000221 osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0) v2 pg pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean]
2014-06-12 14:45:56.364778 7f9d623d5700  5 -- op tracker -- , seq: 747364, time: 2014-06-12 14:45:56.364775, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.364784 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] handle_message: 0x20355000
2014-06-12 14:45:56.364796 7f9d623d5700  5 -- op tracker -- , seq: 747364, time: 2014-06-12 14:45:56.364794, event: started, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.364801 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] sub_op_modify_reply: tid 236158 op  ack_type 4 from 21
2014-06-12 14:45:56.364815 7f9d623d5700  5 -- op tracker -- , seq: 747362, time: 2014-06-12 14:45:56.364811, event: sub_op_commit_rec, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.364820 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2459 crt=0'0 lcod 173'2458 mlcod 173'2458 active+clean] repop_all_committed: repop tid 236158 all committed 
2014-06-12 14:45:56.364829 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2458 active+clean] eval_repop repgather(0x2a37b200 173'2460 rep_tid=236158 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.364842 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2458 active+clean] log_op_stats osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4 inb 221463 outb 0 rlat 0.005444 lat 0.005620
2014-06-12 14:45:56.364856 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2458 active+clean] publish_stats_to_osd 173:3185
2014-06-12 14:45:56.364867 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2458 active+clean]  sending commit on repgather(0x2a37b200 173'2460 rep_tid=236158 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4) 0x30d70780
2014-06-12 14:45:56.364895 7f9d623d5700  5 -- op tracker -- , seq: 747362, time: 2014-06-12 14:45:56.364890, event: commit_sent, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.364902 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2458 active+clean]  requeue_ops 
2014-06-12 14:45:56.364910 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2459 active+clean]  removing repgather(0x2a37b200 173'2460 rep_tid=236158 committed?=1 applied?=1 op=osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.364923 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2459 active+clean]    q front is repgather(0x2a37b200 173'2460 rep_tid=236158 committed?=1 applied?=1 op=osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.364934 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2459 active+clean] remove_repop repgather(0x2a37b200 173'2460 rep_tid=236158 committed?=1 applied?=1 op=osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.364945 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2459 active+clean]  requeue_ops 
2014-06-12 14:45:56.364966 7f9d623d5700  5 -- op tracker -- , seq: 747362, time: 2014-06-12 14:45:56.364962, event: done, op: osd_op(client.4291.0:140624 rbd_data.109a6b8b4567.0000000000000f52 [set-alloc-hint object_size 4194304 write_size 4194304,write 3973120~221184] 3.d38cf9bc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.365003 7f9d623d5700 10 osd.2 173 dequeue_op 0x20355000 finish
2014-06-12 14:45:56.365009 7f9d623d5700  5 -- op tracker -- , seq: 747364, time: 2014-06-12 14:45:56.365006, event: done, op: osd_sub_op_reply(client.4291.0:140624 3.1bc d38cf9bc/rbd_data.109a6b8b4567.0000000000000f52/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.365018 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.365020 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.366339 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:45:56.436483 7f9d56432700  5 -- op tracker -- , seq: 747365, time: 2014-06-12 14:45:56.435808, event: header_read, op: osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.436506 7f9d56432700  5 -- op tracker -- , seq: 747365, time: 2014-06-12 14:45:56.435818, event: throttled, op: osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.436524 7f9d56432700  5 -- op tracker -- , seq: 747365, time: 2014-06-12 14:45:56.436408, event: all_read, op: osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.436541 7f9d56432700  5 -- op tracker -- , seq: 747365, time: 0.000000, event: dispatched, op: osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.436547 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:56.436555 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:56.436564 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3eaf00 prio 127 cost 406311 latency 0.000756 osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.436601 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.436612 7f9d643d9700 10 osd.2 173 dequeue_op 0x1e3eaf00 prio 127 cost 406311 latency 0.000804 osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.25a( v 173'2664 (0'0,173'2664] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2663 active]
2014-06-12 14:45:56.436640 7f9d643d9700  5 -- op tracker -- , seq: 747365, time: 2014-06-12 14:45:56.436636, event: reached_pg, op: osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.436650 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2664 (0'0,173'2664] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2663 active] handle_message: 0x1e3eaf00
2014-06-12 14:45:56.436664 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2664 (0'0,173'2664] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2663 active] sub_op_modify trans 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 v 173'2665 (transaction) 174
2014-06-12 14:45:56.436685 7f9d643d9700  5 -- op tracker -- , seq: 747365, time: 2014-06-12 14:45:56.436681, event: started, op: osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.436706 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2664 (0'0,173'2664] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2663 active] append_log log((0'0,173'2664], crt=0'0) [173'2665 (173'706) modify   2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 by client.4311.0:139682 2014-06-12 08:44:55.948764]
2014-06-12 14:45:56.436728 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2665 (0'0,173'2665] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2663 active] add_log_entry 173'2665 (173'706) modify   2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 by client.4311.0:139682 2014-06-12 08:44:55.948764
2014-06-12 14:45:56.436751 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2665 (0'0,173'2665] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2663 active] append_log  adding 1 keys
2014-06-12 14:45:56.436800 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2665, trimmed: 
2014-06-12 14:45:56.436826 7f9d643d9700  5 -- op tracker -- , seq: 747365, time: 2014-06-12 14:45:56.436822, event: started, op: osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.436838 7f9d643d9700  5 queue_transactions existing osr(3.25a 0x276d340)/0x276d340
2014-06-12 14:45:56.436842 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.436843 7f9d643d9700  5 queue_op 0x8376b40 seq 2500 osr(3.25a 0x276d340) 407797 bytes   (queue has 1 ops and 407797 bytes)
2014-06-12 14:45:56.436855 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x543e480 (1 active)
2014-06-12 14:45:56.436862 7f9d8116f700  5 _do_op 0x8376b40 seq 2500 osr(3.25a 0x276d340)/0x276d340 start
2014-06-12 14:45:56.436851 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.25a( v 173'2665 (0'0,173'2665] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2663 active] do_sub_op osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.436867 7f9d8116f700 10 _do_transaction on 0x165e75f0
2014-06-12 14:45:56.436867 7f9d643d9700 10 osd.2 173 dequeue_op 0x1e3eaf00 finish
2014-06-12 14:45:56.436872 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.436873 7f9d8116f700 15 _omap_setkeys meta/1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:45:56.436874 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.437154 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:45:56.437179 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.437242 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.437262 7f9d8116f700 15 _omap_rmkeys meta/1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:45:56.437270 7f9d8116f700 10 _omap_rmkeys meta/1fe2073b/pglog_3.25a/0//-1 r = 0
2014-06-12 14:45:56.437276 7f9d8116f700 15 _omap_setkeys meta/1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:45:56.437289 7f9d8116f700 15 _write 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 0~405504
2014-06-12 14:45:56.463872 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:45:56.463897 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:45:56.463900 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:56.463903 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:56.463905 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 45; oldest is 33.922776 seconds old
2014-06-12 14:45:56.463937 7f9d84175700  0 log [WRN] : 6 slow requests, 1 included below; oldest blocked for > 33.922776 secs
2014-06-12 14:45:56.463945 7f9d84175700  0 log [WRN] : slow request 30.080697 seconds old, received at 2014-06-12 14:45:26.383207: osd_op(client.4311.0:139599 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~86016] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:45:56.463962 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:45:56.476285 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:56.476288 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:56.476289 7f9d74bfa700 20 osd.2 173 _dispatch 0x1621f6c0 ping v1
2014-06-12 14:45:56.476292 7f9d74bfa700 10 osd.2 173 ping from client.4309
2014-06-12 14:45:56.476294 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:56.476295 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:56.509478 7f9d8116f700 10 lookup_strip_header done  cid 3.25a_head oid 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3
2014-06-12 14:45:56.509541 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:45:56.511121 7f9d8116f700 10 _generic_write 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 0~405504 = 0
2014-06-12 14:45:56.511177 7f9d8116f700 15 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3
2014-06-12 14:45:56.511198 7f9d8116f700 10 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:45:56.511207 7f9d8116f700 15 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3
2014-06-12 14:45:56.511216 7f9d8116f700 10 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:45:56.511405 7f9d8116f700 20 set_header setting 16076 cid 3.25a_head oid 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:45:56.511472 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.511488 7f9d8116f700 20 set_header setting 2302 cid meta oid 1fe2073b/pglog_3.25a/0//-1 parent seq 0
2014-06-12 14:45:56.511500 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:45:56.513793 7f9d8116f700 10 _do_op 0x8376b40 seq 2500 r = 0, finisher 0x660bee0 0
2014-06-12 14:45:56.513832 7f9d7715b700  5 -- op tracker -- , seq: 747365, time: 2014-06-12 14:45:56.513823, event: commit_sent, op: osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.513844 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2665 (0'0,173'2665] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2663 active] sub_op_modify_commit on op osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:56.513871 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:56.513921 7f9d8116f700 10 _finish_op 0x8376b40 seq 2500 osr(3.25a 0x276d340)/0x276d340
2014-06-12 14:45:56.513934 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x543e480 (0 active)
2014-06-12 14:45:56.513937 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.513951 7f9d7795c700  5 -- op tracker -- , seq: 747365, time: 2014-06-12 14:45:56.513943, event: sub_op_applied, op: osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.513962 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2665 (0'0,173'2665] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2664 active] sub_op_modify_applied on 0x165e7500 op osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.513981 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2665 (0'0,173'2665] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2664 active] op_applied on primary on version 173'2665
2014-06-12 14:45:56.513993 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2665 (0'0,173'2665] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2664 active] op_applied on replica on version 173'2665
2014-06-12 14:45:56.514013 7f9d7795c700  5 -- op tracker -- , seq: 747365, time: 2014-06-12 14:45:56.514009, event: done, op: osd_sub_op(client.4311.0:139682 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.519622 7f9d56432700  5 -- op tracker -- , seq: 747366, time: 2014-06-12 14:45:56.519577, event: header_read, op: osd_sub_op_reply(client.4294.0:143871 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.519633 7f9d56432700  5 -- op tracker -- , seq: 747366, time: 2014-06-12 14:45:56.519579, event: throttled, op: osd_sub_op_reply(client.4294.0:143871 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.519641 7f9d56432700  5 -- op tracker -- , seq: 747366, time: 2014-06-12 14:45:56.519589, event: all_read, op: osd_sub_op_reply(client.4294.0:143871 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.519649 7f9d56432700  5 -- op tracker -- , seq: 747366, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:143871 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.519654 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:143871 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.519659 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:56.519666 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3eb300 prio 196 cost 0 latency 0.000089 osd_sub_op_reply(client.4294.0:143871 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.519691 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.519699 7f9d68be2700 10 osd.2 173 dequeue_op 0x1e3eb300 prio 196 cost 0 latency 0.000121 osd_sub_op_reply(client.4294.0:143871 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0) v2 pg pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2461 crt=0'0 lcod 173'2460 mlcod 173'2460 active+clean]
2014-06-12 14:45:56.519721 7f9d68be2700  5 -- op tracker -- , seq: 747366, time: 2014-06-12 14:45:56.519717, event: reached_pg, op: osd_sub_op_reply(client.4294.0:143871 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.519730 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2461 crt=0'0 lcod 173'2460 mlcod 173'2460 active+clean] handle_message: 0x1e3eb300
2014-06-12 14:45:56.519752 7f9d68be2700  5 -- op tracker -- , seq: 747366, time: 2014-06-12 14:45:56.519749, event: started, op: osd_sub_op_reply(client.4294.0:143871 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.519758 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2461 crt=0'0 lcod 173'2460 mlcod 173'2460 active+clean] sub_op_modify_reply: tid 235892 op  ack_type 4 from 26
2014-06-12 14:45:56.519778 7f9d68be2700  5 -- op tracker -- , seq: 746578, time: 2014-06-12 14:45:56.519771, event: sub_op_commit_rec, op: osd_op(client.4294.0:143871 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2445312] 3.e176ab42 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.519784 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2461 crt=0'0 lcod 173'2460 mlcod 173'2460 active+clean] repop_all_applied: repop tid 235892 all applied 
2014-06-12 14:45:56.519795 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2461 crt=0'0 lcod 173'2460 mlcod 173'2460 active+clean] eval_repop repgather(0x15ace840 173'2462 rep_tid=235892 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:143871 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2445312] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.519811 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2461 crt=0'0 lcod 173'2460 mlcod 173'2460 active+clean]  sending ack on repgather(0x15ace840 173'2462 rep_tid=235892 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:143871 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2445312] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) 0x5ffa780
2014-06-12 14:45:56.519842 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2461 crt=0'0 lcod 173'2460 mlcod 173'2460 active+clean] repop_all_committed: repop tid 235892 all committed 
2014-06-12 14:45:56.519852 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2460 active+clean] eval_repop repgather(0x15ace840 173'2462 rep_tid=235892 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:143871 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2445312] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.519868 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2460 active+clean] log_op_stats osd_op(client.4294.0:143871 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2445312] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4 inb 2445591 outb 0 rlat 33.978712 lat 33.978739
2014-06-12 14:45:56.519884 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2460 active+clean] publish_stats_to_osd 173:2831
2014-06-12 14:45:56.519897 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2460 active+clean]  sending commit on repgather(0x15ace840 173'2462 rep_tid=235892 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:143871 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2445312] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) 0x45dfb80
2014-06-12 14:45:56.519926 7f9d68be2700  5 -- op tracker -- , seq: 746578, time: 2014-06-12 14:45:56.519921, event: commit_sent, op: osd_op(client.4294.0:143871 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2445312] 3.e176ab42 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.519933 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2460 active+clean]  requeue_ops 
2014-06-12 14:45:56.519943 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2461 active+clean]  removing repgather(0x15ace840 173'2462 rep_tid=235892 committed?=1 applied?=1 op=osd_op(client.4294.0:143871 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2445312] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.519957 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2461 active+clean]    q front is repgather(0x15ace840 173'2462 rep_tid=235892 committed?=1 applied?=1 op=osd_op(client.4294.0:143871 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2445312] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.519971 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2461 active+clean] remove_repop repgather(0x15ace840 173'2462 rep_tid=235892 committed?=1 applied?=1 op=osd_op(client.4294.0:143871 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2445312] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.519985 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2461 active+clean]  requeue_ops 
2014-06-12 14:45:56.520004 7f9d68be2700  5 -- op tracker -- , seq: 746578, time: 2014-06-12 14:45:56.519999, event: done, op: osd_op(client.4294.0:143871 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2445312] 3.e176ab42 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.520021 7f9d68be2700 10 osd.2 173 dequeue_op 0x1e3eb300 finish
2014-06-12 14:45:56.520028 7f9d68be2700  5 -- op tracker -- , seq: 747366, time: 2014-06-12 14:45:56.520025, event: done, op: osd_sub_op_reply(client.4294.0:143871 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.520043 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.520046 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.524864 7f9d56432700  5 -- op tracker -- , seq: 747367, time: 2014-06-12 14:45:56.524793, event: header_read, op: osd_sub_op_reply(client.4294.0:143872 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.524879 7f9d56432700  5 -- op tracker -- , seq: 747367, time: 2014-06-12 14:45:56.524797, event: throttled, op: osd_sub_op_reply(client.4294.0:143872 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.524887 7f9d56432700  5 -- op tracker -- , seq: 747367, time: 2014-06-12 14:45:56.524814, event: all_read, op: osd_sub_op_reply(client.4294.0:143872 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.524894 7f9d56432700  5 -- op tracker -- , seq: 747367, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:143872 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.524899 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:143872 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.524905 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:56.524913 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3eb000 prio 196 cost 0 latency 0.000120 osd_sub_op_reply(client.4294.0:143872 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.524944 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.524950 7f9d61bd4700 10 osd.2 173 dequeue_op 0x1e3eb000 prio 196 cost 0 latency 0.000157 osd_sub_op_reply(client.4294.0:143872 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0) v2 pg pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2461 active+clean]
2014-06-12 14:45:56.524970 7f9d61bd4700  5 -- op tracker -- , seq: 747367, time: 2014-06-12 14:45:56.524967, event: reached_pg, op: osd_sub_op_reply(client.4294.0:143872 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.524979 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2461 active+clean] handle_message: 0x1e3eb000
2014-06-12 14:45:56.524994 7f9d61bd4700  5 -- op tracker -- , seq: 747367, time: 2014-06-12 14:45:56.524991, event: started, op: osd_sub_op_reply(client.4294.0:143872 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.524999 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2461 active+clean] sub_op_modify_reply: tid 235893 op  ack_type 4 from 26
2014-06-12 14:45:56.525016 7f9d61bd4700  5 -- op tracker -- , seq: 746586, time: 2014-06-12 14:45:56.525010, event: sub_op_commit_rec, op: osd_op(client.4294.0:143872 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2445312~516096] 3.e176ab42 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.525023 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2461 active+clean] repop_all_applied: repop tid 235893 all applied 
2014-06-12 14:45:56.525032 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2461 active+clean] eval_repop repgather(0x51ad800 173'2463 rep_tid=235893 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:143872 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2445312~516096] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.525046 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2461 active+clean]  sending ack on repgather(0x51ad800 173'2463 rep_tid=235893 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:143872 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2445312~516096] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) 0x3b82bb80
2014-06-12 14:45:56.525073 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2462 crt=0'0 lcod 173'2461 mlcod 173'2461 active+clean] repop_all_committed: repop tid 235893 all committed 
2014-06-12 14:45:56.525083 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2461 active+clean] eval_repop repgather(0x51ad800 173'2463 rep_tid=235893 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:143872 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2445312~516096] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.525097 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2461 active+clean] log_op_stats osd_op(client.4294.0:143872 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2445312~516096] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 33.893224 lat 33.893248
2014-06-12 14:45:56.525111 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2461 active+clean] publish_stats_to_osd 173:2832
2014-06-12 14:45:56.525122 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2461 active+clean]  sending commit on repgather(0x51ad800 173'2463 rep_tid=235893 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:143872 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2445312~516096] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) 0x6486000
2014-06-12 14:45:56.525147 7f9d61bd4700  5 -- op tracker -- , seq: 746586, time: 2014-06-12 14:45:56.525143, event: commit_sent, op: osd_op(client.4294.0:143872 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2445312~516096] 3.e176ab42 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.525154 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2461 active+clean]  requeue_ops 
2014-06-12 14:45:56.525163 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2462 active+clean]  removing repgather(0x51ad800 173'2463 rep_tid=235893 committed?=1 applied?=1 op=osd_op(client.4294.0:143872 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2445312~516096] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.525175 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2462 active+clean]    q front is repgather(0x51ad800 173'2463 rep_tid=235893 committed?=1 applied?=1 op=osd_op(client.4294.0:143872 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2445312~516096] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.525189 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2462 active+clean] remove_repop repgather(0x51ad800 173'2463 rep_tid=235893 committed?=1 applied?=1 op=osd_op(client.4294.0:143872 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2445312~516096] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.525201 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2462 active+clean]  requeue_ops 
2014-06-12 14:45:56.525219 7f9d61bd4700  5 -- op tracker -- , seq: 746586, time: 2014-06-12 14:45:56.525215, event: done, op: osd_op(client.4294.0:143872 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2445312~516096] 3.e176ab42 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.525240 7f9d61bd4700 10 osd.2 173 dequeue_op 0x1e3eb000 finish
2014-06-12 14:45:56.525243 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.525246 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.525290 7f9d56432700  5 -- op tracker -- , seq: 747367, time: 2014-06-12 14:45:56.525286, event: done, op: osd_sub_op_reply(client.4294.0:143872 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.525684 7f9d56432700  5 -- op tracker -- , seq: 747368, time: 2014-06-12 14:45:56.525651, event: header_read, op: osd_sub_op_reply(client.4294.0:143873 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.525703 7f9d56432700  5 -- op tracker -- , seq: 747368, time: 2014-06-12 14:45:56.525653, event: throttled, op: osd_sub_op_reply(client.4294.0:143873 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.525710 7f9d56432700  5 -- op tracker -- , seq: 747368, time: 2014-06-12 14:45:56.525664, event: all_read, op: osd_sub_op_reply(client.4294.0:143873 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.525716 7f9d56432700  5 -- op tracker -- , seq: 747368, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:143873 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.525720 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:143873 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.525724 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:56.525731 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3eb000 prio 196 cost 0 latency 0.000079 osd_sub_op_reply(client.4294.0:143873 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.525753 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.525760 7f9d673df700 10 osd.2 173 dequeue_op 0x1e3eb000 prio 196 cost 0 latency 0.000109 osd_sub_op_reply(client.4294.0:143873 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0) v2 pg pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2462 active+clean]
2014-06-12 14:45:56.525779 7f9d673df700  5 -- op tracker -- , seq: 747368, time: 2014-06-12 14:45:56.525776, event: reached_pg, op: osd_sub_op_reply(client.4294.0:143873 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.525787 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2462 active+clean] handle_message: 0x1e3eb000
2014-06-12 14:45:56.525814 7f9d673df700  5 -- op tracker -- , seq: 747368, time: 2014-06-12 14:45:56.525810, event: started, op: osd_sub_op_reply(client.4294.0:143873 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.525819 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2462 active+clean] sub_op_modify_reply: tid 235896 op  ack_type 4 from 26
2014-06-12 14:45:56.525844 7f9d673df700  5 -- op tracker -- , seq: 746595, time: 2014-06-12 14:45:56.525837, event: sub_op_commit_rec, op: osd_op(client.4294.0:143873 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2961408~1032192] 3.e176ab42 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.525851 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2462 active+clean] repop_all_applied: repop tid 235896 all applied 
2014-06-12 14:45:56.525862 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2462 active+clean] eval_repop repgather(0x2a32cc00 173'2464 rep_tid=235896 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:143873 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2961408~1032192] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.525878 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2462 active+clean]  sending ack on repgather(0x2a32cc00 173'2464 rep_tid=235896 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:143873 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2961408~1032192] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) 0x25edd400
2014-06-12 14:45:56.525906 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2463 crt=0'0 lcod 173'2462 mlcod 173'2462 active+clean] repop_all_committed: repop tid 235896 all committed 
2014-06-12 14:45:56.525927 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2462 active+clean] eval_repop repgather(0x2a32cc00 173'2464 rep_tid=235896 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:143873 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2961408~1032192] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.525943 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2462 active+clean] log_op_stats osd_op(client.4294.0:143873 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2961408~1032192] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 33.740106 lat 33.740143
2014-06-12 14:45:56.525959 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2462 active+clean] publish_stats_to_osd 173:2833
2014-06-12 14:45:56.525971 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2462 active+clean]  sending commit on repgather(0x2a32cc00 173'2464 rep_tid=235896 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:143873 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2961408~1032192] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) 0x6487680
2014-06-12 14:45:56.526003 7f9d673df700  5 -- op tracker -- , seq: 746595, time: 2014-06-12 14:45:56.525998, event: commit_sent, op: osd_op(client.4294.0:143873 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2961408~1032192] 3.e176ab42 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.526013 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2462 active+clean]  requeue_ops 
2014-06-12 14:45:56.526023 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2463 active+clean]  removing repgather(0x2a32cc00 173'2464 rep_tid=235896 committed?=1 applied?=1 op=osd_op(client.4294.0:143873 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2961408~1032192] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.526048 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2463 active+clean]    q front is repgather(0x2a32cc00 173'2464 rep_tid=235896 committed?=1 applied?=1 op=osd_op(client.4294.0:143873 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2961408~1032192] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.526062 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2463 active+clean] remove_repop repgather(0x2a32cc00 173'2464 rep_tid=235896 committed?=1 applied?=1 op=osd_op(client.4294.0:143873 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2961408~1032192] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.526076 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2463 active+clean]  requeue_ops 
2014-06-12 14:45:56.526092 7f9d673df700  5 -- op tracker -- , seq: 746595, time: 2014-06-12 14:45:56.526088, event: done, op: osd_op(client.4294.0:143873 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2961408~1032192] 3.e176ab42 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.526108 7f9d673df700 10 osd.2 173 dequeue_op 0x1e3eb000 finish
2014-06-12 14:45:56.526115 7f9d673df700  5 -- op tracker -- , seq: 747368, time: 2014-06-12 14:45:56.526111, event: done, op: osd_sub_op_reply(client.4294.0:143873 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.526125 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.526128 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.535172 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:56.535217 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:56.583554 7f9d56432700  5 -- op tracker -- , seq: 747369, time: 2014-06-12 14:45:56.582534, event: header_read, op: osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.583566 7f9d56432700  5 -- op tracker -- , seq: 747369, time: 2014-06-12 14:45:56.582537, event: throttled, op: osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.583575 7f9d56432700  5 -- op tracker -- , seq: 747369, time: 2014-06-12 14:45:56.583513, event: all_read, op: osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.583584 7f9d56432700  5 -- op tracker -- , seq: 747369, time: 0.000000, event: dispatched, op: osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.583602 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:56.583611 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:56.583620 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3eb100 prio 127 cost 516899 latency 0.001086 osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.583655 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.583663 7f9d663dd700 10 osd.2 173 dequeue_op 0x1e3eb100 prio 127 cost 516899 latency 0.001128 osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2465 (0'0,173'2465] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2464 active]
2014-06-12 14:45:56.583686 7f9d663dd700  5 -- op tracker -- , seq: 747369, time: 2014-06-12 14:45:56.583682, event: reached_pg, op: osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.583695 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2465 (0'0,173'2465] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2464 active] handle_message: 0x1e3eb100
2014-06-12 14:45:56.583707 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2465 (0'0,173'2465] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2464 active] sub_op_modify trans 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 v 173'2466 (transaction) 174
2014-06-12 14:45:56.583724 7f9d663dd700  5 -- op tracker -- , seq: 747369, time: 2014-06-12 14:45:56.583720, event: started, op: osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.583741 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2465 (0'0,173'2465] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2464 active] append_log log((0'0,173'2465], crt=0'0) [173'2466 (173'2464) modify   4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 by client.4280.0:139525 2014-06-12 08:43:25.244110]
2014-06-12 14:45:56.583765 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2466 (0'0,173'2466] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2464 active] add_log_entry 173'2466 (173'2464) modify   4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 by client.4280.0:139525 2014-06-12 08:43:25.244110
2014-06-12 14:45:56.583788 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2466 (0'0,173'2466] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2464 active] append_log  adding 1 keys
2014-06-12 14:45:56.583822 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2466, trimmed: 
2014-06-12 14:45:56.583846 7f9d663dd700  5 -- op tracker -- , seq: 747369, time: 2014-06-12 14:45:56.583842, event: started, op: osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.583856 7f9d663dd700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:45:56.583859 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.583861 7f9d663dd700  5 queue_op 0x575e0c0 seq 2279 osr(3.2b 0x276cd20) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:45:56.583869 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2466 (0'0,173'2466] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2464 active] do_sub_op osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.583884 7f9d663dd700 10 osd.2 173 dequeue_op 0x1e3eb100 finish
2014-06-12 14:45:56.583888 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.583891 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.583899 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:45:56.583904 7f9d79960700  5 _do_op 0x575e0c0 seq 2279 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:45:56.583908 7f9d79960700 10 _do_transaction on 0x1a46e630
2014-06-12 14:45:56.583913 7f9d79960700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:56.584105 7f9d79960700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:56.584128 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.584199 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.584386 7f9d79960700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:56.584393 7f9d79960700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:45:56.584421 7f9d79960700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:56.584441 7f9d79960700 15 _write 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 1417216~516096
2014-06-12 14:45:56.584582 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:56.584584 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:56.584585 7f9d74bfa700 20 osd.2 173 _dispatch 0x2c2dc820 ping v1
2014-06-12 14:45:56.584587 7f9d74bfa700 10 osd.2 173 ping from client.4310
2014-06-12 14:45:56.584589 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:56.584590 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:56.584603 7f9d79960700 10 lookup_strip_header done  cid 3.2b_head oid 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:45:56.584742 7f9d79960700 10 file_to_extents done 
2014-06-12 14:45:56.587140 7f9d79960700 10 _generic_write 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 1417216~516096 = 0
2014-06-12 14:45:56.587202 7f9d79960700 15 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:45:56.587222 7f9d79960700 10 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 = 0
2014-06-12 14:45:56.587229 7f9d79960700 15 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:45:56.587241 7f9d79960700 10 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 = 0
2014-06-12 14:45:56.587476 7f9d79960700 20 set_header setting 15771 cid 3.2b_head oid 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 parent seq 0
2014-06-12 14:45:56.587592 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.587611 7f9d79960700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:45:56.587624 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:45:56.590540 7f9d79960700 10 _do_op 0x575e0c0 seq 2279 r = 0, finisher 0x583ec40 0
2014-06-12 14:45:56.590579 7f9d79960700 10 _finish_op 0x575e0c0 seq 2279 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:45:56.590594 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:45:56.590609 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.590709 7f9d7715b700  5 -- op tracker -- , seq: 747369, time: 2014-06-12 14:45:56.590698, event: commit_sent, op: osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.590719 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2466 (0'0,173'2466] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2464 active] sub_op_modify_commit on op osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:56.590747 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:56.594486 7f9d7795c700  5 -- op tracker -- , seq: 747369, time: 2014-06-12 14:45:56.594470, event: sub_op_applied, op: osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.594502 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2466 (0'0,173'2466] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active] sub_op_modify_applied on 0x1a46e540 op osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.594523 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2466 (0'0,173'2466] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active] op_applied on primary on version 173'2466
2014-06-12 14:45:56.594532 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2466 (0'0,173'2466] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active] op_applied on replica on version 173'2466
2014-06-12 14:45:56.594559 7f9d7795c700  5 -- op tracker -- , seq: 747369, time: 2014-06-12 14:45:56.594555, event: done, op: osd_sub_op(client.4280.0:139525 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.632491 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:56.632495 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:56.632497 7f9d74bfa700 20 osd.2 173 _dispatch 0x330a8d00 ping v1
2014-06-12 14:45:56.632499 7f9d74bfa700 10 osd.2 173 ping from client.4311
2014-06-12 14:45:56.632502 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:56.632503 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:56.738349 7f9d44424700  5 -- op tracker -- , seq: 747370, time: 2014-06-12 14:45:56.735044, event: header_read, op: osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.738370 7f9d44424700  5 -- op tracker -- , seq: 747370, time: 2014-06-12 14:45:56.735049, event: throttled, op: osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.738380 7f9d44424700  5 -- op tracker -- , seq: 747370, time: 2014-06-12 14:45:56.738220, event: all_read, op: osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.738391 7f9d44424700  5 -- op tracker -- , seq: 747370, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.738404 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:45:56.738451 7f9d44424700 15 osd.2 173 enqueue_op 0x5ee9800 prio 63 cost 421888 latency 0.003406 osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.738515 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.738528 7f9d67be0700 10 osd.2 173 dequeue_op 0x5ee9800 prio 63 cost 421888 latency 0.003484 osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]
2014-06-12 14:45:56.738555 7f9d67be0700  5 -- op tracker -- , seq: 747370, time: 2014-06-12 14:45:56.738550, event: reached_pg, op: osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.738564 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:56.738578 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] handle_message: 0x5ee9800
2014-06-12 14:45:56.738587 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_op osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:56.738612 7f9d67be0700 15 getattr 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 '_'
2014-06-12 14:45:56.738934 7f9d67be0700 10 getattr 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 '_' = 0
2014-06-12 14:45:56.738959 7f9d67be0700 15 getattr 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 'snapset'
2014-06-12 14:45:56.739073 7f9d67be0700 10 getattr 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 'snapset' = 0
2014-06-12 14:45:56.739083 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] populate_obc_watchers e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:45:56.739095 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:45:56.739105 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] get_object_context: creating obc from disk: 0xa9af340
2014-06-12 14:45:56.739114 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] get_object_context: 0xa9af340 e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 rwstate(none n=0 w=0) oi: e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3(173'629 client.4317.0:38225 wrlock_by=unknown.0.0:0 dirty s 4194304 uv629) ssc: 0x81bbb20 snapset: 0=[]:[]+head
2014-06-12 14:45:56.739129 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] find_object_context e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 @head oi=e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3(173'629 client.4317.0:38225 wrlock_by=unknown.0.0:0 dirty s 4194304 uv629)
2014-06-12 14:45:56.739153 7f9d67be0700  5 -- op tracker -- , seq: 747370, time: 2014-06-12 14:45:56.739148, event: started, op: osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.739161 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] execute_ctx 0x1fc53200
2014-06-12 14:45:56.739176 7f9d67be0700  5 -- op tracker -- , seq: 747370, time: 2014-06-12 14:45:56.739172, event: started, op: osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.739182 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_op e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] ov 173'629 av 173'2507 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:56.739195 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888]
2014-06-12 14:45:56.739204 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:56.739216 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op  write 0~421888
2014-06-12 14:45:56.739230 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:45:56.739240 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] make_writeable e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 snapset=0x81bbb58  snapc=0=[]
2014-06-12 14:45:56.739251 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] make_writeable e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:56.739260 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] finish_ctx e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 0x1fc53200 op modify  
2014-06-12 14:45:56.739272 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  set mtime to 2014-06-12 08:44:59.115135
2014-06-12 14:45:56.739290 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  final snapset 0=[]:[]+head in e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:45:56.739306 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  zeroing write result code 0
2014-06-12 14:45:56.739316 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] new_repop rep_tid 236159 on osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.739330 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] issue_repop rep_tid 236159 o e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:45:56.739352 7f9d67be0700  5 -- op tracker -- , seq: 747370, time: 2014-06-12 14:45:56.739348, event: waiting for subops from 26, op: osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.739373 7f9d67be0700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:56.739395 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2506 (0'0,173'2506] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] append_log log((0'0,173'2506], crt=0'0) [173'2507 (173'629) modify   e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 by client.4317.0:143789 2014-06-12 08:44:59.115135]
2014-06-12 14:45:56.739417 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] add_log_entry 173'2507 (173'629) modify   e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 by client.4317.0:143789 2014-06-12 08:44:59.115135
2014-06-12 14:45:56.739438 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] append_log  adding 1 keys
2014-06-12 14:45:56.739468 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2507, trimmed: 
2014-06-12 14:45:56.739487 7f9d67be0700  5 queue_transactions existing osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:45:56.739491 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.739493 7f9d67be0700  5 queue_op 0x4467800 seq 2323 osr(3.7ae 0x49b4010) 424181 bytes   (queue has 1 ops and 424181 bytes)
2014-06-12 14:45:56.739503 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] eval_repop repgather(0x9909d40 173'2507 rep_tid=236159 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.739523 7f9d67be0700 10 osd.2 173 dequeue_op 0x5ee9800 finish
2014-06-12 14:45:56.739528 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.739532 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.739549 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1c0b440 (1 active)
2014-06-12 14:45:56.739558 7f9d8016d700  5 _do_op 0x4467800 seq 2323 osr(3.7ae 0x49b4010)/0x49b4010 start
2014-06-12 14:45:56.739564 7f9d8016d700 10 _do_transaction on 0x9908240
2014-06-12 14:45:56.739570 7f9d8016d700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:45:56.739709 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:45:56.739735 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.739844 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.739863 7f9d8016d700 15 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:45:56.739870 7f9d8016d700 10 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1 r = 0
2014-06-12 14:45:56.739884 7f9d8016d700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:45:56.739898 7f9d8016d700 15 _write 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 0~421888
2014-06-12 14:45:56.740071 7f9d8016d700 10 lookup_strip_header done  cid 3.7ae_head oid e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:45:56.740094 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:45:56.741439 7f9d56432700  5 -- op tracker -- , seq: 747371, time: 2014-06-12 14:45:56.736020, event: header_read, op: osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.741451 7f9d56432700  5 -- op tracker -- , seq: 747371, time: 2014-06-12 14:45:56.736023, event: throttled, op: osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.741459 7f9d56432700  5 -- op tracker -- , seq: 747371, time: 2014-06-12 14:45:56.741397, event: all_read, op: osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.741467 7f9d56432700  5 -- op tracker -- , seq: 747371, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.741473 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:56.741479 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:56.741488 7f9d56432700 15 osd.2 173 enqueue_op 0x1702fb00 prio 127 cost 1549095 latency 0.005468 osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.741529 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.741537 7f9d693e3700 10 osd.2 173 dequeue_op 0x1702fb00 prio 127 cost 1549095 latency 0.005517 osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.51e( v 173'2675 (0'0,173'2675] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active]
2014-06-12 14:45:56.741563 7f9d693e3700  5 -- op tracker -- , seq: 747371, time: 2014-06-12 14:45:56.741559, event: reached_pg, op: osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.741571 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2675 (0'0,173'2675] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active] handle_message: 0x1702fb00
2014-06-12 14:45:56.741584 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2675 (0'0,173'2675] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active] sub_op_modify trans 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 v 173'2676 (transaction) 174
2014-06-12 14:45:56.741600 7f9d693e3700  5 -- op tracker -- , seq: 747371, time: 2014-06-12 14:45:56.741596, event: started, op: osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.741752 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2675 (0'0,173'2675] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active] append_log log((0'0,173'2675], crt=0'0) [173'2676 (173'2675) modify   55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 by client.4294.0:143952 2014-06-12 08:44:56.016934]
2014-06-12 14:45:56.741782 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2676 (0'0,173'2676] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active] add_log_entry 173'2676 (173'2675) modify   55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 by client.4294.0:143952 2014-06-12 08:44:56.016934
2014-06-12 14:45:56.741806 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2676 (0'0,173'2676] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active] append_log  adding 1 keys
2014-06-12 14:45:56.741848 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2676, trimmed: 
2014-06-12 14:45:56.741876 7f9d693e3700  5 -- op tracker -- , seq: 747371, time: 2014-06-12 14:45:56.741871, event: started, op: osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.741892 7f9d693e3700  5 queue_transactions existing osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:45:56.741897 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.741899 7f9d693e3700  5 queue_op 0x775efc0 seq 2512 osr(3.51e 0x276db50) 1550581 bytes   (queue has 2 ops and 1974762 bytes)
2014-06-12 14:45:56.741907 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.51e( v 173'2676 (0'0,173'2676] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active] do_sub_op osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.741922 7f9d693e3700 10 osd.2 173 dequeue_op 0x1702fb00 finish
2014-06-12 14:45:56.741926 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.741929 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.741938 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fb00 (2 active)
2014-06-12 14:45:56.741944 7f9d83173700  5 _do_op 0x775efc0 seq 2512 osr(3.51e 0x276db50)/0x276db50 start
2014-06-12 14:45:56.741948 7f9d83173700 10 _do_transaction on 0x2f7ce2b0
2014-06-12 14:45:56.741953 7f9d83173700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:45:56.742124 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:45:56.742148 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.742406 7f9d8016d700 10 _generic_write 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 0~421888 = 0
2014-06-12 14:45:56.742449 7f9d8016d700 15 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:45:56.742461 7f9d8016d700 10 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 = 0
2014-06-12 14:45:56.742467 7f9d8016d700 15 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:45:56.742475 7f9d8016d700 10 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 = 0
2014-06-12 14:45:56.742682 7f9d8016d700 20 set_header setting 16740 cid 3.7ae_head oid e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 parent seq 0
2014-06-12 14:45:56.742740 7f9d8016d700 20 set_header setting 2150 cid meta oid 1fe77416/pglog_3.7ae/0//-1 parent seq 0
2014-06-12 14:45:56.742755 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.742766 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:45:56.749887 7f9d56432700  5 -- op tracker -- , seq: 747372, time: 2014-06-12 14:45:56.749806, event: header_read, op: osd_sub_op_reply(client.4294.0:143874 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.749899 7f9d56432700  5 -- op tracker -- , seq: 747372, time: 2014-06-12 14:45:56.749808, event: throttled, op: osd_sub_op_reply(client.4294.0:143874 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.749907 7f9d56432700  5 -- op tracker -- , seq: 747372, time: 2014-06-12 14:45:56.749820, event: all_read, op: osd_sub_op_reply(client.4294.0:143874 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.749915 7f9d56432700  5 -- op tracker -- , seq: 747372, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:143874 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.749921 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:143874 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.749926 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:56.749935 7f9d56432700 15 osd.2 173 enqueue_op 0x5288800 prio 196 cost 0 latency 0.000129 osd_sub_op_reply(client.4294.0:143874 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.749965 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.750043 7f9d62bd6700 10 osd.2 173 dequeue_op 0x5288800 prio 196 cost 0 latency 0.000236 osd_sub_op_reply(client.4294.0:143874 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0) v2 pg pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2463 active+clean]
2014-06-12 14:45:56.750072 7f9d62bd6700  5 -- op tracker -- , seq: 747372, time: 2014-06-12 14:45:56.750068, event: reached_pg, op: osd_sub_op_reply(client.4294.0:143874 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.750146 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2463 active+clean] handle_message: 0x5288800
2014-06-12 14:45:56.750171 7f9d62bd6700  5 -- op tracker -- , seq: 747372, time: 2014-06-12 14:45:56.750167, event: started, op: osd_sub_op_reply(client.4294.0:143874 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.750178 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2463 active+clean] sub_op_modify_reply: tid 235901 op  ack_type 4 from 26
2014-06-12 14:45:56.750207 7f9d62bd6700  5 -- op tracker -- , seq: 746604, time: 2014-06-12 14:45:56.750194, event: sub_op_commit_rec, op: osd_op(client.4294.0:143874 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3993600~200704] 3.e176ab42 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.750218 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2463 active+clean] repop_all_applied: repop tid 235901 all applied 
2014-06-12 14:45:56.750231 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2463 active+clean] eval_repop repgather(0x5113e00 173'2465 rep_tid=235901 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:143874 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3993600~200704] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.750253 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2463 active+clean]  sending ack on repgather(0x5113e00 173'2465 rep_tid=235901 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:143874 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3993600~200704] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) 0x5ffaf00
2014-06-12 14:45:56.750293 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2464 crt=0'0 lcod 173'2463 mlcod 173'2463 active+clean] repop_all_committed: repop tid 235901 all committed 
2014-06-12 14:45:56.750306 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2463 active+clean] eval_repop repgather(0x5113e00 173'2465 rep_tid=235901 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:143874 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3993600~200704] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.750800 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2463 active+clean] log_op_stats osd_op(client.4294.0:143874 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3993600~200704] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4 inb 200983 outb 0 rlat 33.818147 lat 33.818654
2014-06-12 14:45:56.750860 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2463 active+clean] publish_stats_to_osd 173:2834
2014-06-12 14:45:56.750876 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2463 active+clean]  sending commit on repgather(0x5113e00 173'2465 rep_tid=235901 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:143874 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3993600~200704] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4) 0x34e4c280
2014-06-12 14:45:56.750909 7f9d62bd6700  5 -- op tracker -- , seq: 746604, time: 2014-06-12 14:45:56.750904, event: commit_sent, op: osd_op(client.4294.0:143874 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3993600~200704] 3.e176ab42 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.750918 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2463 active+clean]  requeue_ops 
2014-06-12 14:45:56.750961 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2464 active+clean]  removing repgather(0x5113e00 173'2465 rep_tid=235901 committed?=1 applied?=1 op=osd_op(client.4294.0:143874 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3993600~200704] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.750981 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2464 active+clean]    q front is repgather(0x5113e00 173'2465 rep_tid=235901 committed?=1 applied?=1 op=osd_op(client.4294.0:143874 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3993600~200704] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.750992 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2464 active+clean] remove_repop repgather(0x5113e00 173'2465 rep_tid=235901 committed?=1 applied?=1 op=osd_op(client.4294.0:143874 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3993600~200704] 3.e176ab42 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.751003 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2464 active+clean]  requeue_ops 
2014-06-12 14:45:56.751028 7f9d62bd6700  5 -- op tracker -- , seq: 746604, time: 2014-06-12 14:45:56.751023, event: done, op: osd_op(client.4294.0:143874 rbd_data.109d6b8b4567.00000000000010ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3993600~200704] 3.e176ab42 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.751074 7f9d62bd6700 10 osd.2 173 dequeue_op 0x5288800 finish
2014-06-12 14:45:56.751083 7f9d62bd6700  5 -- op tracker -- , seq: 747372, time: 2014-06-12 14:45:56.751079, event: done, op: osd_sub_op_reply(client.4294.0:143874 3.342 e176ab42/rbd_data.109d6b8b4567.00000000000010ff/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.751096 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.751100 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.753050 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.753086 7f9d83173700 15 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:45:56.753094 7f9d83173700 10 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1 r = 0
2014-06-12 14:45:56.753102 7f9d83173700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:45:56.753127 7f9d83173700 15 _write 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 176128~1548288
2014-06-12 14:45:56.753310 7f9d83173700 10 lookup_strip_header done  cid 3.51e_head oid 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3
2014-06-12 14:45:56.753954 7f9d83173700 10 file_to_extents done 
2014-06-12 14:45:56.754445 7f9d8016d700 10 _do_op 0x4467800 seq 2323 r = 0, finisher 0x73fa600 0x1a3f0000
2014-06-12 14:45:56.754480 7f9d8016d700 10 _finish_op 0x4467800 seq 2323 osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:45:56.754491 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1c0b440 (1 active)
2014-06-12 14:45:56.754495 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.754716 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_commit: 236159
2014-06-12 14:45:56.754753 7f9d7715b700  5 -- op tracker -- , seq: 747370, time: 2014-06-12 14:45:56.754744, event: op_commit, op: osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.757636 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_applied: 236159
2014-06-12 14:45:56.757691 7f9d7795c700  5 -- op tracker -- , seq: 747370, time: 2014-06-12 14:45:56.757678, event: op_applied, op: osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.757705 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_applied on primary on version 173'2507
2014-06-12 14:45:56.776560 7f9d5703e700  5 -- op tracker -- , seq: 747373, time: 2014-06-12 14:45:56.773644, event: header_read, op: osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.776572 7f9d5703e700  5 -- op tracker -- , seq: 747373, time: 2014-06-12 14:45:56.773647, event: throttled, op: osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.776580 7f9d5703e700  5 -- op tracker -- , seq: 747373, time: 2014-06-12 14:45:56.776519, event: all_read, op: osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.776587 7f9d5703e700  5 -- op tracker -- , seq: 747373, time: 0.000000, event: dispatched, op: osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.776592 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:56.776598 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:56.776606 7f9d5703e700 15 osd.2 173 enqueue_op 0xe20bc00 prio 127 cost 234279 latency 0.002962 osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.776644 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.776650 7f9d6b3e7700 10 osd.2 173 dequeue_op 0xe20bc00 prio 127 cost 234279 latency 0.003006 osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.237( v 173'2839 (0'0,173'2839] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2838 active]
2014-06-12 14:45:56.776671 7f9d6b3e7700  5 -- op tracker -- , seq: 747373, time: 2014-06-12 14:45:56.776667, event: reached_pg, op: osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.776678 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2839 (0'0,173'2839] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2838 active] handle_message: 0xe20bc00
2014-06-12 14:45:56.776690 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2839 (0'0,173'2839] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2838 active] sub_op_modify trans 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 v 173'2840 (transaction) 174
2014-06-12 14:45:56.776704 7f9d6b3e7700  5 -- op tracker -- , seq: 747373, time: 2014-06-12 14:45:56.776701, event: started, op: osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.776718 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2839 (0'0,173'2839] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2838 active] append_log log((0'0,173'2839], crt=0'0) [173'2840 (173'905) modify   64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 by client.4295.0:143269 2014-06-12 08:44:59.148316]
2014-06-12 14:45:56.776739 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2840 (0'0,173'2840] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2838 active] add_log_entry 173'2840 (173'905) modify   64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 by client.4295.0:143269 2014-06-12 08:44:59.148316
2014-06-12 14:45:56.776762 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2840 (0'0,173'2840] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2838 active] append_log  adding 1 keys
2014-06-12 14:45:56.776792 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2840, trimmed: 
2014-06-12 14:45:56.776814 7f9d6b3e7700  5 -- op tracker -- , seq: 747373, time: 2014-06-12 14:45:56.776809, event: started, op: osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.776823 7f9d6b3e7700  5 queue_transactions existing osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:45:56.776827 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.776829 7f9d6b3e7700  5 queue_op 0x5a737a0 seq 2597 osr(3.237 0x276d3f0) 235765 bytes   (queue has 2 ops and 1786346 bytes)
2014-06-12 14:45:56.776837 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.237( v 173'2840 (0'0,173'2840] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2838 active] do_sub_op osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.776851 7f9d6b3e7700 10 osd.2 173 dequeue_op 0xe20bc00 finish
2014-06-12 14:45:56.776854 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.776857 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.777672 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540eb40 (2 active)
2014-06-12 14:45:56.777682 7f9d8096e700  5 _do_op 0x5a737a0 seq 2597 osr(3.237 0x276d3f0)/0x276d3f0 start
2014-06-12 14:45:56.777689 7f9d8096e700 10 _do_transaction on 0x165e69b0
2014-06-12 14:45:56.777697 7f9d8096e700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:56.777872 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:56.777908 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.780654 7f9d83173700 10 _generic_write 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 176128~1548288 = 0
2014-06-12 14:45:56.780805 7f9d44424700  5 -- op tracker -- , seq: 747374, time: 2014-06-12 14:45:56.770363, event: header_read, op: osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.780818 7f9d44424700  5 -- op tracker -- , seq: 747374, time: 2014-06-12 14:45:56.770367, event: throttled, op: osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.780828 7f9d44424700  5 -- op tracker -- , seq: 747374, time: 2014-06-12 14:45:56.780769, event: all_read, op: osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.780837 7f9d44424700  5 -- op tracker -- , seq: 747374, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.780844 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:45:56.780871 7f9d44424700 15 osd.2 173 enqueue_op 0x5ee8300 prio 63 cost 1032192 latency 0.010507 osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.780898 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.780906 7f9d64bda700 10 osd.2 173 dequeue_op 0x5ee8300 prio 63 cost 1032192 latency 0.010543 osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]
2014-06-12 14:45:56.780930 7f9d64bda700  5 -- op tracker -- , seq: 747374, time: 2014-06-12 14:45:56.780925, event: reached_pg, op: osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.780939 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:56.780963 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] handle_message: 0x5ee8300
2014-06-12 14:45:56.780973 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_op osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:56.780995 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] get_object_context: found obc in cache: 0xa9af340
2014-06-12 14:45:56.781004 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] get_object_context: 0xa9af340 e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 rwstate(write n=1 w=0) oi: e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3(173'2507 client.4317.0:143789 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2507) ssc: 0x81bbb20 snapset: 0=[]:[]+head
2014-06-12 14:45:56.781018 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] find_object_context e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 @head oi=e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3(173'2507 client.4317.0:143789 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2507)
2014-06-12 14:45:56.781040 7f9d64bda700  5 -- op tracker -- , seq: 747374, time: 2014-06-12 14:45:56.781034, event: started, op: osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.781047 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] execute_ctx 0x29edcc00
2014-06-12 14:45:56.781063 7f9d64bda700  5 -- op tracker -- , seq: 747374, time: 2014-06-12 14:45:56.781058, event: started, op: osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.781070 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_op e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] ov 173'2507 av 173'2508 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:56.781083 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192]
2014-06-12 14:45:56.781094 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:56.781107 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op  write 421888~1032192
2014-06-12 14:45:56.781119 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:45:56.781130 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] make_writeable e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 snapset=0x81bbb58  snapc=0=[]
2014-06-12 14:45:56.781142 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] make_writeable e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:56.781152 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] finish_ctx e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 0x29edcc00 op modify  
2014-06-12 14:45:56.781167 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  set mtime to 2014-06-12 08:44:59.149503
2014-06-12 14:45:56.781187 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  final snapset 0=[]:[]+head in e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:45:56.781204 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  zeroing write result code 0
2014-06-12 14:45:56.781214 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] new_repop rep_tid 236160 on osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.781229 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] issue_repop rep_tid 236160 o e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:45:56.781253 7f9d64bda700  5 -- op tracker -- , seq: 747374, time: 2014-06-12 14:45:56.781247, event: waiting for subops from 26, op: osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.781273 7f9d64bda700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:56.781294 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2507 (0'0,173'2507] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] append_log log((0'0,173'2507], crt=0'0) [173'2508 (173'2507) modify   e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 by client.4317.0:143790 2014-06-12 08:44:59.149503]
2014-06-12 14:45:56.781318 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] add_log_entry 173'2508 (173'2507) modify   e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 by client.4317.0:143790 2014-06-12 08:44:59.149503
2014-06-12 14:45:56.781339 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] append_log  adding 1 keys
2014-06-12 14:45:56.781376 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2508, trimmed: 
2014-06-12 14:45:56.781396 7f9d64bda700  5 queue_transactions existing osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:45:56.781400 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.781401 7f9d64bda700  5 queue_op 0x7c886c0 seq 2324 osr(3.7ae 0x49b4010) 1034485 bytes   (queue has 3 ops and 2820831 bytes)
2014-06-12 14:45:56.781409 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] eval_repop repgather(0x31b643c0 173'2508 rep_tid=236160 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.781428 7f9d64bda700 10 osd.2 173 dequeue_op 0x5ee8300 finish
2014-06-12 14:45:56.781433 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.781436 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.781555 7f9d83173700 15 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3
2014-06-12 14:45:56.781577 7f9d83173700 10 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 = 0
2014-06-12 14:45:56.781583 7f9d83173700 15 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3
2014-06-12 14:45:56.781596 7f9d83173700 10 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 = 0
2014-06-12 14:45:56.781626 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1c0b440 (3 active)
2014-06-12 14:45:56.781639 7f9d7915f700  5 _do_op 0x7c886c0 seq 2324 osr(3.7ae 0x49b4010)/0x49b4010 start
2014-06-12 14:45:56.781645 7f9d7915f700 10 _do_transaction on 0x31b64a80
2014-06-12 14:45:56.781654 7f9d7915f700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:45:56.781867 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:45:56.781890 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.782415 7f9d83173700 20 set_header setting 17621 cid 3.51e_head oid 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 parent seq 0
2014-06-12 14:45:56.782876 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.782894 7f9d83173700 20 set_header setting 2288 cid meta oid 1fe1612b/pglog_3.51e/0//-1 parent seq 0
2014-06-12 14:45:56.782909 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:45:56.793255 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.793291 7f9d8096e700 15 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:56.793299 7f9d8096e700 10 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1 r = 0
2014-06-12 14:45:56.793307 7f9d8096e700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:56.793315 7f9d83173700 10 _do_op 0x775efc0 seq 2512 r = 0, finisher 0x8586180 0
2014-06-12 14:45:56.793327 7f9d83173700 10 _finish_op 0x775efc0 seq 2512 osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:45:56.793334 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fb00 (2 active)
2014-06-12 14:45:56.793337 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.793350 7f9d7795c700  5 -- op tracker -- , seq: 747371, time: 2014-06-12 14:45:56.793341, event: sub_op_applied, op: osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.793357 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2676 (0'0,173'2676] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active] sub_op_modify_applied on 0x2f7ce1c0 op osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.793383 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:56.793404 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2676 (0'0,173'2676] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active] op_applied on primary on version 173'2676
2014-06-12 14:45:56.793416 7f9d8096e700 15 _write 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 0~233472
2014-06-12 14:45:56.793415 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2676 (0'0,173'2676] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active] op_applied on replica on version 173'2676
2014-06-12 14:45:56.793638 7f9d8096e700 10 lookup_strip_header done  cid 3.237_head oid 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3
2014-06-12 14:45:56.793643 7f9d7715b700  5 -- op tracker -- , seq: 747371, time: 2014-06-12 14:45:56.793635, event: commit_sent, op: osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.793653 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2676 (0'0,173'2676] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2674 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:56.793674 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:56.793714 7f9d7715b700  5 -- op tracker -- , seq: 747371, time: 2014-06-12 14:45:56.793709, event: done, op: osd_sub_op(client.4294.0:143952 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.793903 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:45:56.794511 7f9d8096e700 10 _generic_write 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 0~233472 = 0
2014-06-12 14:45:56.794563 7f9d8096e700 15 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3
2014-06-12 14:45:56.794578 7f9d8096e700 10 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 = 0
2014-06-12 14:45:56.794585 7f9d8096e700 15 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3
2014-06-12 14:45:56.794594 7f9d8096e700 10 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 = 0
2014-06-12 14:45:56.794723 7f9d8096e700 20 set_header setting 16982 cid 3.237_head oid 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 parent seq 0
2014-06-12 14:45:56.794762 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.794777 7f9d8096e700 20 set_header setting 2190 cid meta oid 1fe26b1a/pglog_3.237/0//-1 parent seq 0
2014-06-12 14:45:56.794790 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:45:56.796191 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.796236 7f9d7915f700 15 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:45:56.796243 7f9d7915f700 10 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1 r = 0
2014-06-12 14:45:56.796250 7f9d7915f700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:45:56.796270 7f9d7915f700 15 _write 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 421888~1032192
2014-06-12 14:45:56.796628 7f9d7915f700 10 lookup_strip_header done  cid 3.7ae_head oid e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:45:56.796808 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:45:56.797198 7f9d8096e700 10 _do_op 0x5a737a0 seq 2597 r = 0, finisher 0x1f0879a0 0
2014-06-12 14:45:56.797209 7f9d8096e700 10 _finish_op 0x5a737a0 seq 2597 osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:45:56.797217 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540eb40 (1 active)
2014-06-12 14:45:56.797221 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.797261 7f9d7795c700  5 -- op tracker -- , seq: 747373, time: 2014-06-12 14:45:56.797252, event: sub_op_applied, op: osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.797270 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2840 (0'0,173'2840] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2838 active] sub_op_modify_applied on 0x165e68c0 op osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:56.797298 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:56.797321 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2840 (0'0,173'2840] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2838 active] op_applied on primary on version 173'2840
2014-06-12 14:45:56.797331 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2840 (0'0,173'2840] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2838 active] op_applied on replica on version 173'2840
2014-06-12 14:45:56.797508 7f9d7715b700  5 -- op tracker -- , seq: 747373, time: 2014-06-12 14:45:56.797501, event: commit_sent, op: osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.797517 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2840 (0'0,173'2840] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2838 active] sub_op_modify_commit on op osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:45:56.797534 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:56.797560 7f9d7715b700  5 -- op tracker -- , seq: 747373, time: 2014-06-12 14:45:56.797556, event: done, op: osd_sub_op(client.4295.0:143269 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2840 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:56.800615 7f9d7915f700 10 _generic_write 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 421888~1032192 = 0
2014-06-12 14:45:56.800772 7f9d7915f700 15 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:45:56.800791 7f9d7915f700 10 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 = 0
2014-06-12 14:45:56.800797 7f9d7915f700 15 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:45:56.800806 7f9d7915f700 10 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 = 0
2014-06-12 14:45:56.801189 7f9d7915f700 20 set_header setting 16740 cid 3.7ae_head oid e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 parent seq 0
2014-06-12 14:45:56.801302 7f9d7915f700 20 set_header setting 2150 cid meta oid 1fe77416/pglog_3.7ae/0//-1 parent seq 0
2014-06-12 14:45:56.801318 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.801328 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:45:56.808335 7f9d7915f700 10 _do_op 0x7c886c0 seq 2324 r = 0, finisher 0x72f79e0 0x187b8500
2014-06-12 14:45:56.808376 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_commit: 236160
2014-06-12 14:45:56.808412 7f9d7715b700  5 -- op tracker -- , seq: 747374, time: 2014-06-12 14:45:56.808401, event: op_commit, op: osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.808440 7f9d7915f700 10 _finish_op 0x7c886c0 seq 2324 osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:45:56.808451 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1c0b440 (0 active)
2014-06-12 14:45:56.808463 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.808458 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_applied: 236160
2014-06-12 14:45:56.808483 7f9d7795c700  5 -- op tracker -- , seq: 747374, time: 2014-06-12 14:45:56.808477, event: op_applied, op: osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.808504 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_applied on primary on version 173'2508
2014-06-12 14:45:56.860310 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:56.861764 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:56.934138 7f9d523da700  5 -- op tracker -- , seq: 747375, time: 2014-06-12 14:45:56.932575, event: header_read, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.934153 7f9d523da700  5 -- op tracker -- , seq: 747375, time: 2014-06-12 14:45:56.932586, event: throttled, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.934163 7f9d523da700  5 -- op tracker -- , seq: 747375, time: 2014-06-12 14:45:56.934092, event: all_read, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.934172 7f9d523da700  5 -- op tracker -- , seq: 747375, time: 0.000000, event: dispatched, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.934181 7f9d523da700 20 osd.2 173 should_share_map client.4284 172.16.96.220:0/1241415 173
2014-06-12 14:45:56.934211 7f9d523da700 15 osd.2 173 enqueue_op 0x6f4a600 prio 63 cost 286720 latency 0.001636 osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.934249 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.934256 7f9d683e1700 10 osd.2 173 dequeue_op 0x6f4a600 prio 63 cost 286720 latency 0.001681 osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]
2014-06-12 14:45:56.934286 7f9d683e1700  5 -- op tracker -- , seq: 747375, time: 2014-06-12 14:45:56.934280, event: reached_pg, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.934296 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:56.934313 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] handle_message: 0x6f4a600
2014-06-12 14:45:56.934327 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] do_op osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:56.934356 7f9d683e1700 15 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 '_'
2014-06-12 14:45:56.934642 7f9d683e1700 10 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 '_' = 0
2014-06-12 14:45:56.934663 7f9d683e1700 15 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 'snapset'
2014-06-12 14:45:56.934744 7f9d683e1700 10 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 'snapset' = 0
2014-06-12 14:45:56.934755 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] populate_obc_watchers fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:56.934770 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:56.934781 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] get_object_context: creating obc from disk: 0x14e7bb80
2014-06-12 14:45:56.934792 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] get_object_context: 0x14e7bb80 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 rwstate(none n=0 w=0) oi: fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3(173'869 client.4284.0:35993 wrlock_by=unknown.0.0:0 dirty s 4194304 uv869) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:45:56.934809 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] find_object_context fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 @head oi=fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3(173'869 client.4284.0:35993 wrlock_by=unknown.0.0:0 dirty s 4194304 uv869)
2014-06-12 14:45:56.934838 7f9d683e1700  5 -- op tracker -- , seq: 747375, time: 2014-06-12 14:45:56.934831, event: started, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.934846 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] execute_ctx 0xa04f800
2014-06-12 14:45:56.934866 7f9d683e1700  5 -- op tracker -- , seq: 747375, time: 2014-06-12 14:45:56.934860, event: started, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.934875 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] do_op fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] ov 173'869 av 173'2857 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:56.934892 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] do_osd_op fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720]
2014-06-12 14:45:56.934904 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:56.934919 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] do_osd_op  write 0~286720
2014-06-12 14:45:56.934937 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] do_osd_op_effects on session 0x5c8ce00
2014-06-12 14:45:56.934952 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] make_writeable fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:45:56.934967 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] make_writeable fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:56.934981 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] finish_ctx fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 0xa04f800 op modify  
2014-06-12 14:45:56.934998 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]  set mtime to 2014-06-12 08:44:59.313313
2014-06-12 14:45:56.935023 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]  final snapset 0=[]:[]+head in fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:56.935044 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]  zeroing write result code 0
2014-06-12 14:45:56.935057 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] new_repop rep_tid 236161 on osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:56.935076 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] issue_repop rep_tid 236161 o fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:56.935105 7f9d683e1700  5 -- op tracker -- , seq: 747375, time: 2014-06-12 14:45:56.935100, event: waiting for subops from 21, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.935130 7f9d683e1700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:56.935158 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2856 (0'0,173'2856] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] append_log log((0'0,173'2856], crt=0'0) [173'2857 (173'869) modify   fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 by client.4284.0:141380 2014-06-12 08:44:59.313313]
2014-06-12 14:45:56.935178 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] add_log_entry 173'2857 (173'869) modify   fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 by client.4284.0:141380 2014-06-12 08:44:59.313313
2014-06-12 14:45:56.935201 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] append_log  adding 1 keys
2014-06-12 14:45:56.935235 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2857, trimmed: 
2014-06-12 14:45:56.935259 7f9d683e1700  5 queue_transactions existing osr(3.131 0x276cee0)/0x276cee0
2014-06-12 14:45:56.935265 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:56.935267 7f9d683e1700  5 queue_op 0xf1e13e0 seq 2581 osr(3.131 0x276cee0) 289013 bytes   (queue has 1 ops and 289013 bytes)
2014-06-12 14:45:56.935275 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] eval_repop repgather(0x2b62b740 173'2857 rep_tid=236161 committed?=0 applied?=0 lock=0 op=osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.935297 7f9d683e1700 10 osd.2 173 dequeue_op 0x6f4a600 finish
2014-06-12 14:45:56.935303 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.935307 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.936154 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b87e0 (1 active)
2014-06-12 14:45:56.936167 7f9d7c165700  5 _do_op 0xf1e13e0 seq 2581 osr(3.131 0x276cee0)/0x276cee0 start
2014-06-12 14:45:56.936174 7f9d7c165700 10 _do_transaction on 0x2ad986c0
2014-06-12 14:45:56.936183 7f9d7c165700 15 _omap_setkeys meta/1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:56.936336 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:56.936361 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:56.936425 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:56.936445 7f9d7c165700 15 _omap_rmkeys meta/1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:56.936452 7f9d7c165700 10 _omap_rmkeys meta/1fe280aa/pglog_3.131/0//-1 r = 0
2014-06-12 14:45:56.936458 7f9d7c165700 15 _omap_setkeys meta/1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:56.936476 7f9d7c165700 15 _write 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 0~286720
2014-06-12 14:45:56.936636 7f9d7c165700 10 lookup_strip_header done  cid 3.131_head oid fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:56.936668 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:45:56.942429 7f9d7c165700 10 _generic_write 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 0~286720 = 0
2014-06-12 14:45:56.942477 7f9d7c165700 15 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:56.942491 7f9d7c165700 10 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:45:56.942497 7f9d7c165700 15 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:56.942504 7f9d7c165700 10 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:45:56.942696 7f9d7c165700 20 set_header setting 17294 cid 3.131_head oid fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 parent seq 0
2014-06-12 14:45:56.942765 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:56.942782 7f9d7c165700 20 set_header setting 1996 cid meta oid 1fe280aa/pglog_3.131/0//-1 parent seq 0
2014-06-12 14:45:56.942793 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:45:56.944427 7f9d7c165700 10 _do_op 0xf1e13e0 seq 2581 r = 0, finisher 0x31d37780 0x2f966740
2014-06-12 14:45:56.944463 7f9d7c165700 10 _finish_op 0xf1e13e0 seq 2581 osr(3.131 0x276cee0)/0x276cee0
2014-06-12 14:45:56.944485 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b87e0 (0 active)
2014-06-12 14:45:56.944492 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:56.944500 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] op_applied: 236161
2014-06-12 14:45:56.944546 7f9d7795c700  5 -- op tracker -- , seq: 747375, time: 2014-06-12 14:45:56.944534, event: op_applied, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.944559 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] op_applied on primary on version 173'2857
2014-06-12 14:45:56.945638 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] op_commit: 236161
2014-06-12 14:45:56.945695 7f9d7715b700  5 -- op tracker -- , seq: 747375, time: 2014-06-12 14:45:56.945683, event: op_commit, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.947929 7f9d44101700  5 -- op tracker -- , seq: 747376, time: 2014-06-12 14:45:56.947875, event: header_read, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:56.947943 7f9d44101700  5 -- op tracker -- , seq: 747376, time: 2014-06-12 14:45:56.947877, event: throttled, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:56.947950 7f9d44101700  5 -- op tracker -- , seq: 747376, time: 2014-06-12 14:45:56.947888, event: all_read, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:56.947958 7f9d44101700  5 -- op tracker -- , seq: 747376, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:56.947963 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:45:56.947969 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.947978 7f9d44101700 15 osd.2 173 enqueue_op 0x20355a00 prio 196 cost 0 latency 0.000103 osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0) v2
2014-06-12 14:45:56.948007 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:56.948016 7f9d65bdc700 10 osd.2 173 dequeue_op 0x20355a00 prio 196 cost 0 latency 0.000140 osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0) v2 pg pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]
2014-06-12 14:45:56.948036 7f9d65bdc700  5 -- op tracker -- , seq: 747376, time: 2014-06-12 14:45:56.948033, event: reached_pg, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:56.948045 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] handle_message: 0x20355a00
2014-06-12 14:45:56.948059 7f9d65bdc700  5 -- op tracker -- , seq: 747376, time: 2014-06-12 14:45:56.948056, event: started, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:56.948065 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] sub_op_modify_reply: tid 236161 op  ack_type 1 from 21
2014-06-12 14:45:56.948085 7f9d65bdc700  5 -- op tracker -- , seq: 747375, time: 2014-06-12 14:45:56.948077, event: sub_op_applied_rec, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.948102 7f9d44101700  5 -- op tracker -- , seq: 747377, time: 2014-06-12 14:45:56.948050, event: header_read, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.948097 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] repop_all_applied: repop tid 236161 all applied 
2014-06-12 14:45:56.948116 7f9d44101700  5 -- op tracker -- , seq: 747377, time: 2014-06-12 14:45:56.948051, event: throttled, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.948124 7f9d44101700  5 -- op tracker -- , seq: 747377, time: 2014-06-12 14:45:56.948058, event: all_read, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.948131 7f9d44101700  5 -- op tracker -- , seq: 747377, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.948134 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:56.948120 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] eval_repop repgather(0x2b62b740 173'2857 rep_tid=236161 committed?=0 applied?=1 lock=0 op=osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.948139 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:56.948144 7f9d44101700 15 osd.2 173 enqueue_op 0x6f4af00 prio 196 cost 0 latency 0.000094 osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:56.948139 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]  sending ack on repgather(0x2b62b740 173'2857 rep_tid=236161 committed?=0 applied?=1 lock=0 op=osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) 0xc210500
2014-06-12 14:45:56.948166 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:45:56.948172 7f9d65bdc700 10 osd.2 173 dequeue_op 0x20355a00 finish
2014-06-12 14:45:56.948184 7f9d65bdc700  5 -- op tracker -- , seq: 747376, time: 2014-06-12 14:45:56.948180, event: done, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:56.948199 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:45:56.948202 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:45:56.953634 7f9d6abe6700 10 osd.2 173 dequeue_op 0x6f4af00 prio 196 cost 0 latency 0.005583 osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0) v2 pg pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]
2014-06-12 14:45:56.953672 7f9d6abe6700  5 -- op tracker -- , seq: 747377, time: 2014-06-12 14:45:56.953668, event: reached_pg, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.953684 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] handle_message: 0x6f4af00
2014-06-12 14:45:56.953699 7f9d6abe6700  5 -- op tracker -- , seq: 747377, time: 2014-06-12 14:45:56.953696, event: started, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.953705 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] sub_op_modify_reply: tid 236161 op  ack_type 4 from 21
2014-06-12 14:45:56.953725 7f9d6abe6700  5 -- op tracker -- , seq: 747375, time: 2014-06-12 14:45:56.953717, event: sub_op_commit_rec, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.953731 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] repop_all_committed: repop tid 236161 all committed 
2014-06-12 14:45:56.953741 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] eval_repop repgather(0x2b62b740 173'2857 rep_tid=236161 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:56.953755 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] log_op_stats osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4 inb 286999 outb 0 rlat 0.015595 lat 0.021179
2014-06-12 14:45:56.953769 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] publish_stats_to_osd 173:3664
2014-06-12 14:45:56.953783 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean]  sending commit on repgather(0x2b62b740 173'2857 rep_tid=236161 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) 0xc211180
2014-06-12 14:45:56.953820 7f9d6abe6700  5 -- op tracker -- , seq: 747375, time: 2014-06-12 14:45:56.953814, event: commit_sent, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.953827 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean]  requeue_ops 
2014-06-12 14:45:56.953837 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  removing repgather(0x2b62b740 173'2857 rep_tid=236161 committed?=1 applied?=1 op=osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.953849 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]    q front is repgather(0x2b62b740 173'2857 rep_tid=236161 committed?=1 applied?=1 op=osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.953861 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] remove_repop repgather(0x2b62b740 173'2857 rep_tid=236161 committed?=1 applied?=1 op=osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:56.953873 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  requeue_ops 
2014-06-12 14:45:56.953898 7f9d6abe6700  5 -- op tracker -- , seq: 747375, time: 2014-06-12 14:45:56.953894, event: done, op: osd_op(client.4284.0:141380 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~286720] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:56.953917 7f9d6abe6700 10 osd.2 173 dequeue_op 0x6f4af00 finish
2014-06-12 14:45:56.953924 7f9d6abe6700  5 -- op tracker -- , seq: 747377, time: 2014-06-12 14:45:56.953922, event: done, op: osd_sub_op_reply(client.4284.0:141380 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:56.953936 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:56.953940 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.169391 7f9d56432700  5 -- op tracker -- , seq: 747378, time: 2014-06-12 14:45:57.168315, event: header_read, op: osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.169408 7f9d56432700  5 -- op tracker -- , seq: 747378, time: 2014-06-12 14:45:57.168319, event: throttled, op: osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.169418 7f9d56432700  5 -- op tracker -- , seq: 747378, time: 2014-06-12 14:45:57.169346, event: all_read, op: osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.169427 7f9d56432700  5 -- op tracker -- , seq: 747378, time: 0.000000, event: dispatched, op: osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.169432 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.169439 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:57.169447 7f9d56432700 15 osd.2 173 enqueue_op 0x5289500 prio 127 cost 516903 latency 0.001132 osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.169510 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.169518 7f9d66bde700 10 osd.2 173 dequeue_op 0x5289500 prio 127 cost 516903 latency 0.001203 osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2983 (0'0,173'2983] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2982 active]
2014-06-12 14:45:57.169547 7f9d66bde700  5 -- op tracker -- , seq: 747378, time: 2014-06-12 14:45:57.169542, event: reached_pg, op: osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.169556 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2983 (0'0,173'2983] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2982 active] handle_message: 0x5289500
2014-06-12 14:45:57.169579 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2983 (0'0,173'2983] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2982 active] sub_op_modify trans 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 v 173'2984 (transaction) 174
2014-06-12 14:45:57.169599 7f9d66bde700  5 -- op tracker -- , seq: 747378, time: 2014-06-12 14:45:57.169595, event: started, op: osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.169621 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2983 (0'0,173'2983] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2982 active] append_log log((0'0,173'2983], crt=0'0) [173'2984 (173'2983) modify   6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 by client.4307.0:146568 2014-06-12 08:44:32.699813]
2014-06-12 14:45:57.169648 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2984 (0'0,173'2984] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2982 active] add_log_entry 173'2984 (173'2983) modify   6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 by client.4307.0:146568 2014-06-12 08:44:32.699813
2014-06-12 14:45:57.169670 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2984 (0'0,173'2984] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2982 active] append_log  adding 1 keys
2014-06-12 14:45:57.169704 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2984, trimmed: 
2014-06-12 14:45:57.169729 7f9d66bde700  5 -- op tracker -- , seq: 747378, time: 2014-06-12 14:45:57.169725, event: started, op: osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.169738 7f9d66bde700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:45:57.169742 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.169744 7f9d66bde700  5 queue_op 0x4f77f20 seq 2786 osr(3.458 0x276d960) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:45:57.169753 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:45:57.169759 7f9d82972700  5 _do_op 0x4f77f20 seq 2786 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:45:57.169764 7f9d82972700 10 _do_transaction on 0x668d270
2014-06-12 14:45:57.169750 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2984 (0'0,173'2984] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2982 active] do_sub_op osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.169765 7f9d66bde700 10 osd.2 173 dequeue_op 0x5289500 finish
2014-06-12 14:45:57.169769 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.169769 7f9d82972700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:45:57.169772 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.169927 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:45:57.169944 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.169991 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.170078 7f9d82972700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:45:57.170085 7f9d82972700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:45:57.170090 7f9d82972700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:45:57.170101 7f9d82972700 15 _write 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 1781760~516096
2014-06-12 14:45:57.170222 7f9d82972700 10 lookup_strip_header done  cid 3.458_head oid 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:45:57.170376 7f9d82972700 10 file_to_extents done 
2014-06-12 14:45:57.172218 7f9d82972700 10 _generic_write 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 1781760~516096 = 0
2014-06-12 14:45:57.172284 7f9d82972700 15 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:45:57.172299 7f9d82972700 10 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 = 0
2014-06-12 14:45:57.172303 7f9d82972700 15 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:45:57.172309 7f9d82972700 10 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 = 0
2014-06-12 14:45:57.172472 7f9d82972700 20 set_header setting 18434 cid 3.458_head oid 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 parent seq 0
2014-06-12 14:45:57.172539 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.172552 7f9d82972700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:45:57.172558 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:45:57.175599 7f9d82972700 10 _do_op 0x4f77f20 seq 2786 r = 0, finisher 0x73f8100 0
2014-06-12 14:45:57.175620 7f9d82972700 10 _finish_op 0x4f77f20 seq 2786 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:45:57.175627 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:45:57.175638 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.175640 7f9d7715b700  5 -- op tracker -- , seq: 747378, time: 2014-06-12 14:45:57.175629, event: commit_sent, op: osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.175652 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2984 (0'0,173'2984] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2982 active] sub_op_modify_commit on op osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:57.175679 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:57.175716 7f9d7795c700  5 -- op tracker -- , seq: 747378, time: 2014-06-12 14:45:57.175709, event: sub_op_applied, op: osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.175727 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2984 (0'0,173'2984] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2983 active] sub_op_modify_applied on 0x668d180 op osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.175745 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2984 (0'0,173'2984] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2983 active] op_applied on primary on version 173'2984
2014-06-12 14:45:57.175757 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2984 (0'0,173'2984] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2983 active] op_applied on replica on version 173'2984
2014-06-12 14:45:57.175789 7f9d7795c700  5 -- op tracker -- , seq: 747378, time: 2014-06-12 14:45:57.175782, event: done, op: osd_sub_op(client.4307.0:146568 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.180171 7f9d56432700  5 -- op tracker -- , seq: 747379, time: 2014-06-12 14:45:57.174767, event: header_read, op: osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.180189 7f9d56432700  5 -- op tracker -- , seq: 747379, time: 2014-06-12 14:45:57.174770, event: throttled, op: osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.180197 7f9d56432700  5 -- op tracker -- , seq: 747379, time: 2014-06-12 14:45:57.180091, event: all_read, op: osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.180205 7f9d56432700  5 -- op tracker -- , seq: 747379, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.180221 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.180229 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:57.180240 7f9d56432700 15 osd.2 173 enqueue_op 0x5288100 prio 127 cost 1549091 latency 0.005472 osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.180277 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.180288 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x5288100 prio 127 cost 1549091 latency 0.005521 osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2466 (0'0,173'2466] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active]
2014-06-12 14:45:57.180314 7f9d6a3e5700  5 -- op tracker -- , seq: 747379, time: 2014-06-12 14:45:57.180309, event: reached_pg, op: osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.180325 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2466 (0'0,173'2466] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active] handle_message: 0x5288100
2014-06-12 14:45:57.180338 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2466 (0'0,173'2466] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active] sub_op_modify trans cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 v 173'2467 (transaction) 174
2014-06-12 14:45:57.180355 7f9d6a3e5700  5 -- op tracker -- , seq: 747379, time: 2014-06-12 14:45:57.180352, event: started, op: osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.180373 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2466 (0'0,173'2466] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active] append_log log((0'0,173'2466], crt=0'0) [173'2467 (173'2465) modify   cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 by client.4308.0:144511 2014-06-12 08:43:36.661341]
2014-06-12 14:45:57.180398 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2467 (0'0,173'2467] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active] add_log_entry 173'2467 (173'2465) modify   cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 by client.4308.0:144511 2014-06-12 08:43:36.661341
2014-06-12 14:45:57.180420 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2467 (0'0,173'2467] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active] append_log  adding 1 keys
2014-06-12 14:45:57.180454 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2467, trimmed: 
2014-06-12 14:45:57.180479 7f9d6a3e5700  5 -- op tracker -- , seq: 747379, time: 2014-06-12 14:45:57.180474, event: started, op: osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.180490 7f9d6a3e5700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:45:57.180494 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.180496 7f9d6a3e5700  5 queue_op 0x6794420 seq 2280 osr(3.2b 0x276cd20) 1550572 bytes   (queue has 1 ops and 1550572 bytes)
2014-06-12 14:45:57.180506 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:45:57.180512 7f9d7a161700  5 _do_op 0x6794420 seq 2280 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:45:57.180502 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2467 (0'0,173'2467] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active] do_sub_op osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.180516 7f9d7a161700 10 _do_transaction on 0x31d08b70
2014-06-12 14:45:57.180517 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x5288100 finish
2014-06-12 14:45:57.180520 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.180520 7f9d7a161700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:57.180522 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.180650 7f9d7a161700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:57.180676 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.180726 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.180741 7f9d7a161700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:57.180746 7f9d7a161700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:45:57.180750 7f9d7a161700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:57.180761 7f9d7a161700 15 _write 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 2514944~1548288
2014-06-12 14:45:57.181052 7f9d7a161700 10 lookup_strip_header done  cid 3.2b_head oid cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3
2014-06-12 14:45:57.181208 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:45:57.186919 7f9d7a161700 10 _generic_write 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 2514944~1548288 = 0
2014-06-12 14:45:57.187069 7f9d7a161700 15 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3
2014-06-12 14:45:57.187088 7f9d7a161700 10 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 = 0
2014-06-12 14:45:57.187096 7f9d7a161700 15 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3
2014-06-12 14:45:57.187101 7f9d7a161700 10 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 = 0
2014-06-12 14:45:57.187481 7f9d7a161700 20 set_header setting 16771 cid 3.2b_head oid cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 parent seq 0
2014-06-12 14:45:57.187591 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.187603 7f9d7a161700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:45:57.187613 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:45:57.195120 7f9d7a161700 10 _do_op 0x6794420 seq 2280 r = 0, finisher 0x66b3460 0
2014-06-12 14:45:57.195153 7f9d7a161700 10 _finish_op 0x6794420 seq 2280 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:45:57.195188 7f9d7795c700  5 -- op tracker -- , seq: 747379, time: 2014-06-12 14:45:57.195175, event: sub_op_applied, op: osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.195202 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2467 (0'0,173'2467] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active] sub_op_modify_applied on 0x31d08a80 op osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.195233 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:57.195254 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2467 (0'0,173'2467] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active] op_applied on primary on version 173'2467
2014-06-12 14:45:57.195266 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2467 (0'0,173'2467] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active] op_applied on replica on version 173'2467
2014-06-12 14:45:57.195281 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:45:57.195290 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.196135 7f9d7715b700  5 -- op tracker -- , seq: 747379, time: 2014-06-12 14:45:57.196125, event: commit_sent, op: osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.196147 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2467 (0'0,173'2467] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2465 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:57.196170 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:57.196204 7f9d7715b700  5 -- op tracker -- , seq: 747379, time: 2014-06-12 14:45:57.196199, event: done, op: osd_sub_op(client.4308.0:144511 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.208157 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:45:57.208841 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:45:57.228596 7f9d56432700  5 -- op tracker -- , seq: 747380, time: 2014-06-12 14:45:57.226578, event: header_read, op: osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.228619 7f9d56432700  5 -- op tracker -- , seq: 747380, time: 2014-06-12 14:45:57.226580, event: throttled, op: osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.228628 7f9d56432700  5 -- op tracker -- , seq: 747380, time: 2014-06-12 14:45:57.228537, event: all_read, op: osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.228643 7f9d56432700  5 -- op tracker -- , seq: 747380, time: 0.000000, event: dispatched, op: osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.228649 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.228663 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:57.228673 7f9d56432700 15 osd.2 173 enqueue_op 0x5288900 prio 127 cost 1032999 latency 0.002094 osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.228706 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.228714 7f9d63bd8700 10 osd.2 173 dequeue_op 0x5288900 prio 127 cost 1032999 latency 0.002135 osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.25a( v 173'2665 (0'0,173'2665] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2664 active]
2014-06-12 14:45:57.228732 7f9d63bd8700  5 -- op tracker -- , seq: 747380, time: 2014-06-12 14:45:57.228729, event: reached_pg, op: osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.228739 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2665 (0'0,173'2665] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2664 active] handle_message: 0x5288900
2014-06-12 14:45:57.228747 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2665 (0'0,173'2665] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2664 active] sub_op_modify trans 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 v 173'2666 (transaction) 174
2014-06-12 14:45:57.228758 7f9d63bd8700  5 -- op tracker -- , seq: 747380, time: 2014-06-12 14:45:57.228755, event: started, op: osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.228805 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2665 (0'0,173'2665] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2664 active] append_log log((0'0,173'2665], crt=0'0) [173'2666 (173'2665) modify   2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 by client.4311.0:139683 2014-06-12 08:44:56.163541]
2014-06-12 14:45:57.228840 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2666 (0'0,173'2666] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2664 active] add_log_entry 173'2666 (173'2665) modify   2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 by client.4311.0:139683 2014-06-12 08:44:56.163541
2014-06-12 14:45:57.228886 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2666 (0'0,173'2666] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2664 active] append_log  adding 1 keys
2014-06-12 14:45:57.228927 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2666, trimmed: 
2014-06-12 14:45:57.228971 7f9d63bd8700  5 -- op tracker -- , seq: 747380, time: 2014-06-12 14:45:57.228965, event: started, op: osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.228985 7f9d63bd8700  5 queue_transactions existing osr(3.25a 0x276d340)/0x276d340
2014-06-12 14:45:57.228989 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.228991 7f9d63bd8700  5 queue_op 0x132b8c60 seq 2501 osr(3.25a 0x276d340) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:45:57.228999 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.25a( v 173'2666 (0'0,173'2666] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2664 active] do_sub_op osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.229014 7f9d63bd8700 10 osd.2 173 dequeue_op 0x5288900 finish
2014-06-12 14:45:57.229017 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.229020 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.229029 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x543e480 (1 active)
2014-06-12 14:45:57.229034 7f9d81970700  5 _do_op 0x132b8c60 seq 2501 osr(3.25a 0x276d340)/0x276d340 start
2014-06-12 14:45:57.229039 7f9d81970700 10 _do_transaction on 0x19d87430
2014-06-12 14:45:57.229052 7f9d81970700 15 _omap_setkeys meta/1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:45:57.229216 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:45:57.229408 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.229503 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.229531 7f9d81970700 15 _omap_rmkeys meta/1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:45:57.229540 7f9d81970700 10 _omap_rmkeys meta/1fe2073b/pglog_3.25a/0//-1 r = 0
2014-06-12 14:45:57.229547 7f9d81970700 15 _omap_setkeys meta/1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:45:57.229564 7f9d81970700 15 _write 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 405504~1032192
2014-06-12 14:45:57.229731 7f9d81970700 10 lookup_strip_header done  cid 3.25a_head oid 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3
2014-06-12 14:45:57.230036 7f9d81970700 10 file_to_extents done 
2014-06-12 14:45:57.235225 7f9d81970700 10 _generic_write 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 405504~1032192 = 0
2014-06-12 14:45:57.235333 7f9d81970700 15 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3
2014-06-12 14:45:57.235363 7f9d81970700 10 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:45:57.235374 7f9d81970700 15 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3
2014-06-12 14:45:57.235382 7f9d81970700 10 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:45:57.235688 7f9d81970700 20 set_header setting 16076 cid 3.25a_head oid 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:45:57.235794 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.235807 7f9d81970700 20 set_header setting 2302 cid meta oid 1fe2073b/pglog_3.25a/0//-1 parent seq 0
2014-06-12 14:45:57.235816 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:45:57.237133 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:57.237804 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:57.254700 7f9d81970700 10 _do_op 0x132b8c60 seq 2501 r = 0, finisher 0x4f542a0 0
2014-06-12 14:45:57.254755 7f9d81970700 10 _finish_op 0x132b8c60 seq 2501 osr(3.25a 0x276d340)/0x276d340
2014-06-12 14:45:57.254783 7f9d7715b700  5 -- op tracker -- , seq: 747380, time: 2014-06-12 14:45:57.254773, event: commit_sent, op: osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.254798 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2666 (0'0,173'2666] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2664 active] sub_op_modify_commit on op osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:57.254822 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:57.254952 7f9d7795c700  5 -- op tracker -- , seq: 747380, time: 2014-06-12 14:45:57.254946, event: sub_op_applied, op: osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.254961 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2666 (0'0,173'2666] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2665 active] sub_op_modify_applied on 0x19d87340 op osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.254977 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2666 (0'0,173'2666] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2665 active] op_applied on primary on version 173'2666
2014-06-12 14:45:57.254986 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2666 (0'0,173'2666] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2665 active] op_applied on replica on version 173'2666
2014-06-12 14:45:57.255017 7f9d7795c700  5 -- op tracker -- , seq: 747380, time: 2014-06-12 14:45:57.255013, event: done, op: osd_sub_op(client.4311.0:139683 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.254784 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x543e480 (0 active)
2014-06-12 14:45:57.256173 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.319611 7f9d523da700  5 -- op tracker -- , seq: 747381, time: 2014-06-12 14:45:57.311673, event: header_read, op: osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.319626 7f9d523da700  5 -- op tracker -- , seq: 747381, time: 2014-06-12 14:45:57.311677, event: throttled, op: osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.319636 7f9d523da700  5 -- op tracker -- , seq: 747381, time: 2014-06-12 14:45:57.319550, event: all_read, op: osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.319644 7f9d523da700  5 -- op tracker -- , seq: 747381, time: 0.000000, event: dispatched, op: osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.319655 7f9d523da700 20 osd.2 173 should_share_map client.4284 172.16.96.220:0/1241415 173
2014-06-12 14:45:57.319690 7f9d523da700 15 osd.2 173 enqueue_op 0x6f4ad00 prio 63 cost 1548288 latency 0.008017 osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:57.319728 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.319737 7f9d653db700 10 osd.2 173 dequeue_op 0x6f4ad00 prio 63 cost 1548288 latency 0.008064 osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]
2014-06-12 14:45:57.319765 7f9d653db700  5 -- op tracker -- , seq: 747381, time: 2014-06-12 14:45:57.319759, event: reached_pg, op: osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.319775 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:57.319790 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] handle_message: 0x6f4ad00
2014-06-12 14:45:57.319800 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_op osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:57.319825 7f9d653db700 15 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 '_'
2014-06-12 14:45:57.320116 7f9d653db700 10 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 '_' = 0
2014-06-12 14:45:57.320135 7f9d653db700 15 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 'snapset'
2014-06-12 14:45:57.320209 7f9d653db700 10 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 'snapset' = 0
2014-06-12 14:45:57.320219 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] populate_obc_watchers fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.320228 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.320234 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] get_object_context: creating obc from disk: 0x34d12b00
2014-06-12 14:45:57.320240 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] get_object_context: 0x34d12b00 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 rwstate(none n=0 w=0) oi: fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3(173'2857 client.4284.0:141380 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2857) ssc: 0x24d4a9a0 snapset: 0=[]:[]+head
2014-06-12 14:45:57.320251 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] find_object_context fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 @head oi=fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3(173'2857 client.4284.0:141380 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2857)
2014-06-12 14:45:57.320269 7f9d653db700  5 -- op tracker -- , seq: 747381, time: 2014-06-12 14:45:57.320265, event: started, op: osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.320280 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] execute_ctx 0xd21a000
2014-06-12 14:45:57.320291 7f9d653db700  5 -- op tracker -- , seq: 747381, time: 2014-06-12 14:45:57.320288, event: started, op: osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.320302 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_op fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] ov 173'2857 av 173'2858 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:57.320311 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288]
2014-06-12 14:45:57.320318 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:57.320326 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op  write 286720~1548288
2014-06-12 14:45:57.320337 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op_effects on session 0x5c8ce00
2014-06-12 14:45:57.320343 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] make_writeable fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 snapset=0x24d4a9d8  snapc=0=[]
2014-06-12 14:45:57.320350 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] make_writeable fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:57.320356 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] finish_ctx fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 0xd21a000 op modify  
2014-06-12 14:45:57.320364 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  set mtime to 2014-06-12 08:44:59.690166
2014-06-12 14:45:57.320377 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  final snapset 0=[]:[]+head in fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.320389 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  zeroing write result code 0
2014-06-12 14:45:57.320396 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] new_repop rep_tid 236162 on osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:57.320406 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] issue_repop rep_tid 236162 o fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.320422 7f9d653db700  5 -- op tracker -- , seq: 747381, time: 2014-06-12 14:45:57.320419, event: waiting for subops from 21, op: osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.320437 7f9d653db700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.320455 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2857 (0'0,173'2857] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] append_log log((0'0,173'2857], crt=0'0) [173'2858 (173'2857) modify   fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 by client.4284.0:141381 2014-06-12 08:44:59.690166]
2014-06-12 14:45:57.320478 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] add_log_entry 173'2858 (173'2857) modify   fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 by client.4284.0:141381 2014-06-12 08:44:59.690166
2014-06-12 14:45:57.320496 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] append_log  adding 1 keys
2014-06-12 14:45:57.320527 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2858, trimmed: 
2014-06-12 14:45:57.320547 7f9d653db700  5 queue_transactions existing osr(3.131 0x276cee0)/0x276cee0
2014-06-12 14:45:57.320551 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.320552 7f9d653db700  5 queue_op 0x1516b740 seq 2582 osr(3.131 0x276cee0) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:45:57.320567 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b87e0 (1 active)
2014-06-12 14:45:57.320574 7f9d7c966700  5 _do_op 0x1516b740 seq 2582 osr(3.131 0x276cee0)/0x276cee0 start
2014-06-12 14:45:57.320579 7f9d7c966700 10 _do_transaction on 0x10450780
2014-06-12 14:45:57.320559 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] eval_repop repgather(0x6591e00 173'2858 rep_tid=236162 committed?=0 applied?=0 lock=0 op=osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:57.320582 7f9d653db700 10 osd.2 173 dequeue_op 0x6f4ad00 finish
2014-06-12 14:45:57.320585 7f9d7c966700 15 _omap_setkeys meta/1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:57.320586 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.320589 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.320718 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:57.320750 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.320816 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.320837 7f9d7c966700 15 _omap_rmkeys meta/1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:57.320842 7f9d7c966700 10 _omap_rmkeys meta/1fe280aa/pglog_3.131/0//-1 r = 0
2014-06-12 14:45:57.320848 7f9d7c966700 15 _omap_setkeys meta/1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:57.320977 7f9d7c966700 15 _write 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 286720~1548288
2014-06-12 14:45:57.321125 7f9d7c966700 10 lookup_strip_header done  cid 3.131_head oid fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.321271 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:45:57.327822 7f9d7c966700 10 _generic_write 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 286720~1548288 = 0
2014-06-12 14:45:57.328041 7f9d7c966700 15 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.328063 7f9d7c966700 10 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:45:57.328068 7f9d7c966700 15 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.328073 7f9d7c966700 10 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:45:57.328415 7f9d7c966700 20 set_header setting 17294 cid 3.131_head oid fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 parent seq 0
2014-06-12 14:45:57.328518 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.328527 7f9d7c966700 20 set_header setting 1996 cid meta oid 1fe280aa/pglog_3.131/0//-1 parent seq 0
2014-06-12 14:45:57.328539 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:45:57.335550 7f9d7c966700 10 _do_op 0x1516b740 seq 2582 r = 0, finisher 0x7ffbf40 0x2a39a680
2014-06-12 14:45:57.335611 7f9d7c966700 10 _finish_op 0x1516b740 seq 2582 osr(3.131 0x276cee0)/0x276cee0
2014-06-12 14:45:57.335627 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_commit: 236162
2014-06-12 14:45:57.335668 7f9d7715b700  5 -- op tracker -- , seq: 747381, time: 2014-06-12 14:45:57.335656, event: op_commit, op: osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.335702 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_applied: 236162
2014-06-12 14:45:57.335734 7f9d7795c700  5 -- op tracker -- , seq: 747381, time: 2014-06-12 14:45:57.335727, event: op_applied, op: osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.335743 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_applied on primary on version 173'2858
2014-06-12 14:45:57.335777 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b87e0 (0 active)
2014-06-12 14:45:57.335786 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.342305 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:45:57.345693 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:45:57.357973 7f9d44101700  5 -- op tracker -- , seq: 747382, time: 2014-06-12 14:45:57.357874, event: header_read, op: osd_sub_op_reply(client.4284.0:141381 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.357986 7f9d44101700  5 -- op tracker -- , seq: 747382, time: 2014-06-12 14:45:57.357876, event: throttled, op: osd_sub_op_reply(client.4284.0:141381 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.357993 7f9d44101700  5 -- op tracker -- , seq: 747382, time: 2014-06-12 14:45:57.357892, event: all_read, op: osd_sub_op_reply(client.4284.0:141381 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.357999 7f9d44101700  5 -- op tracker -- , seq: 747382, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4284.0:141381 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.358017 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4284.0:141381 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:57.358024 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:57.358034 7f9d44101700 15 osd.2 173 enqueue_op 0x34016a00 prio 196 cost 0 latency 0.000160 osd_sub_op_reply(client.4284.0:141381 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:57.358072 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.358089 7f9d69be4700 10 osd.2 173 dequeue_op 0x34016a00 prio 196 cost 0 latency 0.000215 osd_sub_op_reply(client.4284.0:141381 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0) v2 pg pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]
2014-06-12 14:45:57.358119 7f9d69be4700  5 -- op tracker -- , seq: 747382, time: 2014-06-12 14:45:57.358115, event: reached_pg, op: osd_sub_op_reply(client.4284.0:141381 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.358138 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] handle_message: 0x34016a00
2014-06-12 14:45:57.358158 7f9d69be4700  5 -- op tracker -- , seq: 747382, time: 2014-06-12 14:45:57.358154, event: started, op: osd_sub_op_reply(client.4284.0:141381 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.358167 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] sub_op_modify_reply: tid 236162 op  ack_type 4 from 21
2014-06-12 14:45:57.358213 7f9d69be4700  5 -- op tracker -- , seq: 747381, time: 2014-06-12 14:45:57.358200, event: sub_op_commit_rec, op: osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.358226 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] repop_all_applied: repop tid 236162 all applied 
2014-06-12 14:45:57.358239 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] eval_repop repgather(0x6591e00 173'2858 rep_tid=236162 committed?=0 applied?=1 lock=0 op=osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:57.358255 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  sending ack on repgather(0x6591e00 173'2858 rep_tid=236162 committed?=0 applied?=1 lock=0 op=osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) 0x6989180
2014-06-12 14:45:57.358292 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] repop_all_committed: repop tid 236162 all committed 
2014-06-12 14:45:57.358303 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean] eval_repop repgather(0x6591e00 173'2858 rep_tid=236162 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:57.358319 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean] log_op_stats osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 0.046616 lat 0.046645
2014-06-12 14:45:57.358343 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean] publish_stats_to_osd 173:3665
2014-06-12 14:45:57.358357 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean]  sending commit on repgather(0x6591e00 173'2858 rep_tid=236162 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) 0x17f00500
2014-06-12 14:45:57.358386 7f9d69be4700  5 -- op tracker -- , seq: 747381, time: 2014-06-12 14:45:57.358381, event: commit_sent, op: osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.358394 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean]  requeue_ops 
2014-06-12 14:45:57.358404 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  removing repgather(0x6591e00 173'2858 rep_tid=236162 committed?=1 applied?=1 op=osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:57.358418 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]    q front is repgather(0x6591e00 173'2858 rep_tid=236162 committed?=1 applied?=1 op=osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:57.358433 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] remove_repop repgather(0x6591e00 173'2858 rep_tid=236162 committed?=1 applied?=1 op=osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:57.358446 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  requeue_ops 
2014-06-12 14:45:57.358472 7f9d69be4700  5 -- op tracker -- , seq: 747381, time: 2014-06-12 14:45:57.358467, event: done, op: osd_op(client.4284.0:141381 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 286720~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.358491 7f9d69be4700 10 osd.2 173 dequeue_op 0x34016a00 finish
2014-06-12 14:45:57.358498 7f9d69be4700  5 -- op tracker -- , seq: 747382, time: 2014-06-12 14:45:57.358494, event: done, op: osd_sub_op_reply(client.4284.0:141381 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.358508 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.358511 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.371651 7f9d523da700  5 -- op tracker -- , seq: 747383, time: 2014-06-12 14:45:57.365667, event: header_read, op: osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.371669 7f9d523da700  5 -- op tracker -- , seq: 747383, time: 2014-06-12 14:45:57.365670, event: throttled, op: osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.371678 7f9d523da700  5 -- op tracker -- , seq: 747383, time: 2014-06-12 14:45:57.371580, event: all_read, op: osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.371689 7f9d523da700  5 -- op tracker -- , seq: 747383, time: 0.000000, event: dispatched, op: osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.371698 7f9d523da700 20 osd.2 173 should_share_map client.4284 172.16.96.220:0/1241415 173
2014-06-12 14:45:57.371728 7f9d523da700 15 osd.2 173 enqueue_op 0x6f4b100 prio 63 cost 1548288 latency 0.006061 osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:57.371765 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.371772 7f9d633d7700 10 osd.2 173 dequeue_op 0x6f4b100 prio 63 cost 1548288 latency 0.006105 osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]
2014-06-12 14:45:57.371798 7f9d633d7700  5 -- op tracker -- , seq: 747383, time: 2014-06-12 14:45:57.371792, event: reached_pg, op: osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.371806 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:57.371822 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] handle_message: 0x6f4b100
2014-06-12 14:45:57.371833 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_op osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:57.371858 7f9d633d7700 15 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 '_'
2014-06-12 14:45:57.372122 7f9d633d7700 10 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 '_' = 0
2014-06-12 14:45:57.372143 7f9d633d7700 15 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 'snapset'
2014-06-12 14:45:57.372260 7f9d633d7700 10 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 'snapset' = 0
2014-06-12 14:45:57.372275 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] populate_obc_watchers fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.372289 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.372299 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] get_object_context: creating obc from disk: 0x1703a840
2014-06-12 14:45:57.372309 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] get_object_context: 0x1703a840 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 rwstate(none n=0 w=0) oi: fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3(173'2858 client.4284.0:141381 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2858) ssc: 0x81ba460 snapset: 0=[]:[]+head
2014-06-12 14:45:57.372327 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] find_object_context fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 @head oi=fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3(173'2858 client.4284.0:141381 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2858)
2014-06-12 14:45:57.372356 7f9d633d7700  5 -- op tracker -- , seq: 747383, time: 2014-06-12 14:45:57.372350, event: started, op: osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.372366 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] execute_ctx 0x2aad600
2014-06-12 14:45:57.372386 7f9d633d7700  5 -- op tracker -- , seq: 747383, time: 2014-06-12 14:45:57.372381, event: started, op: osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.372392 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_op fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] ov 173'2858 av 173'2859 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:57.372407 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_osd_op fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288]
2014-06-12 14:45:57.372418 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:57.372432 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_osd_op  write 1835008~1548288
2014-06-12 14:45:57.372449 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_osd_op_effects on session 0x5c8ce00
2014-06-12 14:45:57.372461 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] make_writeable fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 snapset=0x81ba498  snapc=0=[]
2014-06-12 14:45:57.372473 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] make_writeable fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:57.372483 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] finish_ctx fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 0x2aad600 op modify  
2014-06-12 14:45:57.372505 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  set mtime to 2014-06-12 08:44:59.742442
2014-06-12 14:45:57.372527 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  final snapset 0=[]:[]+head in fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.372545 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  zeroing write result code 0
2014-06-12 14:45:57.372557 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] new_repop rep_tid 236163 on osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:57.372574 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] issue_repop rep_tid 236163 o fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.372598 7f9d633d7700  5 -- op tracker -- , seq: 747383, time: 2014-06-12 14:45:57.372594, event: waiting for subops from 21, op: osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.372621 7f9d633d7700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.372644 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2858 (0'0,173'2858] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] append_log log((0'0,173'2858], crt=0'0) [173'2859 (173'2858) modify   fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 by client.4284.0:141382 2014-06-12 08:44:59.742442]
2014-06-12 14:45:57.372677 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] add_log_entry 173'2859 (173'2858) modify   fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 by client.4284.0:141382 2014-06-12 08:44:59.742442
2014-06-12 14:45:57.372699 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] append_log  adding 1 keys
2014-06-12 14:45:57.372734 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2859, trimmed: 
2014-06-12 14:45:57.372760 7f9d633d7700  5 queue_transactions existing osr(3.131 0x276cee0)/0x276cee0
2014-06-12 14:45:57.372766 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.372768 7f9d633d7700  5 queue_op 0x6514420 seq 2583 osr(3.131 0x276cee0) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:45:57.372776 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] eval_repop repgather(0xbe0d980 173'2859 rep_tid=236163 committed?=0 applied?=0 lock=0 op=osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:57.372796 7f9d633d7700 10 osd.2 173 dequeue_op 0x6f4b100 finish
2014-06-12 14:45:57.372801 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.372805 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.373434 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b87e0 (1 active)
2014-06-12 14:45:57.373444 7f9d7a962700  5 _do_op 0x6514420 seq 2583 osr(3.131 0x276cee0)/0x276cee0 start
2014-06-12 14:45:57.373449 7f9d7a962700 10 _do_transaction on 0x234c3c0
2014-06-12 14:45:57.373456 7f9d7a962700 15 _omap_setkeys meta/1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:57.373679 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:57.373726 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.373805 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.373829 7f9d7a962700 15 _omap_rmkeys meta/1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:57.373836 7f9d7a962700 10 _omap_rmkeys meta/1fe280aa/pglog_3.131/0//-1 r = 0
2014-06-12 14:45:57.373842 7f9d7a962700 15 _omap_setkeys meta/1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:57.374052 7f9d7a962700 15 _write 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 1835008~1548288
2014-06-12 14:45:57.374230 7f9d7a962700 10 lookup_strip_header done  cid 3.131_head oid fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.374436 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:45:57.387448 7f9d7a962700 10 _generic_write 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 1835008~1548288 = 0
2014-06-12 14:45:57.387768 7f9d7a962700 15 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.387811 7f9d7a962700 10 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:45:57.387826 7f9d7a962700 15 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.387837 7f9d7a962700 10 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:45:57.388422 7f9d7a962700 20 set_header setting 17294 cid 3.131_head oid fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 parent seq 0
2014-06-12 14:45:57.388602 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.388618 7f9d7a962700 20 set_header setting 1996 cid meta oid 1fe280aa/pglog_3.131/0//-1 parent seq 0
2014-06-12 14:45:57.388637 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:45:57.397923 7f9d57543700  5 -- op tracker -- , seq: 747384, time: 2014-06-12 14:45:57.381647, event: header_read, op: osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.397938 7f9d57543700  5 -- op tracker -- , seq: 747384, time: 2014-06-12 14:45:57.381651, event: throttled, op: osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.397953 7f9d57543700  5 -- op tracker -- , seq: 747384, time: 2014-06-12 14:45:57.397876, event: all_read, op: osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.397962 7f9d57543700  5 -- op tracker -- , seq: 747384, time: 0.000000, event: dispatched, op: osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.397968 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.397970 7f9d44101700  5 -- op tracker -- , seq: 747385, time: 2014-06-12 14:45:57.397935, event: header_read, op: osd_sub_op_reply(client.4284.0:141382 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.397976 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:57.397981 7f9d44101700  5 -- op tracker -- , seq: 747385, time: 2014-06-12 14:45:57.397936, event: throttled, op: osd_sub_op_reply(client.4284.0:141382 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.397985 7f9d57543700 15 osd.2 173 enqueue_op 0x5ee8500 prio 127 cost 287527 latency 0.016338 osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.397990 7f9d44101700  5 -- op tracker -- , seq: 747385, time: 2014-06-12 14:45:57.397945, event: all_read, op: osd_sub_op_reply(client.4284.0:141382 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.397998 7f9d44101700  5 -- op tracker -- , seq: 747385, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4284.0:141382 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.398002 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4284.0:141382 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:57.398008 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:57.398014 7f9d44101700 15 osd.2 173 enqueue_op 0x34017900 prio 196 cost 0 latency 0.000079 osd_sub_op_reply(client.4284.0:141382 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:57.398017 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.398029 7f9d623d5700 10 osd.2 173 dequeue_op 0x5ee8500 prio 127 cost 287527 latency 0.016382 osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2d7( v 173'2989 (0'0,173'2989] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2988 active]
2014-06-12 14:45:57.398051 7f9d623d5700  5 -- op tracker -- , seq: 747384, time: 2014-06-12 14:45:57.398047, event: reached_pg, op: osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.398060 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2989 (0'0,173'2989] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2988 active] handle_message: 0x5ee8500
2014-06-12 14:45:57.398071 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2989 (0'0,173'2989] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2988 active] sub_op_modify trans 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 v 173'2990 (transaction) 174
2014-06-12 14:45:57.398086 7f9d623d5700  5 -- op tracker -- , seq: 747384, time: 2014-06-12 14:45:57.398083, event: started, op: osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.398101 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2989 (0'0,173'2989] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2988 active] append_log log((0'0,173'2989], crt=0'0) [173'2990 (173'816) modify   88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 by client.4301.0:145391 2014-06-12 08:44:59.758258]
2014-06-12 14:45:57.398134 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2990 (0'0,173'2990] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2988 active] add_log_entry 173'2990 (173'816) modify   88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 by client.4301.0:145391 2014-06-12 08:44:59.758258
2014-06-12 14:45:57.398159 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2990 (0'0,173'2990] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2988 active] append_log  adding 1 keys
2014-06-12 14:45:57.398194 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2990, trimmed: 
2014-06-12 14:45:57.398221 7f9d623d5700  5 -- op tracker -- , seq: 747384, time: 2014-06-12 14:45:57.398215, event: started, op: osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.398232 7f9d623d5700  5 queue_transactions existing osr(3.2d7 0x276d5c0)/0x276d5c0
2014-06-12 14:45:57.398236 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.398238 7f9d623d5700  5 queue_op 0xbdfdce0 seq 2703 osr(3.2d7 0x276d5c0) 289013 bytes   (queue has 2 ops and 1839594 bytes)
2014-06-12 14:45:57.398245 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2990 (0'0,173'2990] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2988 active] do_sub_op osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.398261 7f9d623d5700 10 osd.2 173 dequeue_op 0x5ee8500 finish
2014-06-12 14:45:57.398264 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.398270 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.398274 7f9d623d5700 10 osd.2 173 dequeue_op 0x34017900 prio 196 cost 0 latency 0.000339 osd_sub_op_reply(client.4284.0:141382 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0) v2 pg pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]
2014-06-12 14:45:57.398290 7f9d623d5700  5 -- op tracker -- , seq: 747385, time: 2014-06-12 14:45:57.398287, event: reached_pg, op: osd_sub_op_reply(client.4284.0:141382 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.398297 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] handle_message: 0x34017900
2014-06-12 14:45:57.398312 7f9d623d5700  5 -- op tracker -- , seq: 747385, time: 2014-06-12 14:45:57.398309, event: started, op: osd_sub_op_reply(client.4284.0:141382 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.398317 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] sub_op_modify_reply: tid 236163 op  ack_type 4 from 21
2014-06-12 14:45:57.398338 7f9d623d5700  5 -- op tracker -- , seq: 747383, time: 2014-06-12 14:45:57.398331, event: sub_op_commit_rec, op: osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.398346 7f9d623d5700 10 osd.2 173 dequeue_op 0x34017900 finish
2014-06-12 14:45:57.398353 7f9d623d5700  5 -- op tracker -- , seq: 747385, time: 2014-06-12 14:45:57.398350, event: done, op: osd_sub_op_reply(client.4284.0:141382 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.398366 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.398369 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.398379 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cbe60 (2 active)
2014-06-12 14:45:57.398384 7f9d7895e700  5 _do_op 0xbdfdce0 seq 2703 osr(3.2d7 0x276d5c0)/0x276d5c0 start
2014-06-12 14:45:57.398388 7f9d7895e700 10 _do_transaction on 0x6671270
2014-06-12 14:45:57.398394 7f9d7895e700 15 _omap_setkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.398406 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.398548 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.398569 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.435389 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.435435 7f9d7895e700 15 _omap_rmkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.435441 7f9d7895e700 10 _omap_rmkeys meta/1fe0df15/pglog_3.2d7/0//-1 r = 0
2014-06-12 14:45:57.435449 7f9d7895e700 15 _omap_setkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.435658 7f9d7a962700 10 _do_op 0x6514420 seq 2583 r = 0, finisher 0x2896d20 0x32115e80
2014-06-12 14:45:57.435661 7f9d7895e700 15 _write 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 0~286720
2014-06-12 14:45:57.435670 7f9d7a962700 10 _finish_op 0x6514420 seq 2583 osr(3.131 0x276cee0)/0x276cee0
2014-06-12 14:45:57.435689 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b87e0 (1 active)
2014-06-12 14:45:57.435697 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.435678 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] op_commit: 236163
2014-06-12 14:45:57.435711 7f9d7715b700  5 -- op tracker -- , seq: 747383, time: 2014-06-12 14:45:57.435701, event: op_commit, op: osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.435727 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] repop_all_committed: repop tid 236163 all committed 
2014-06-12 14:45:57.435735 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] eval_repop repgather(0xbe0d980 173'2859 rep_tid=236163 committed?=1 applied?=0 lock=0 op=osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:57.435747 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] log_op_stats osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 0.000000 lat 0.070079
2014-06-12 14:45:57.435759 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] publish_stats_to_osd 173:3666
2014-06-12 14:45:57.435767 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean]  sending commit on repgather(0xbe0d980 173'2859 rep_tid=236163 committed?=1 applied?=0 lock=0 op=osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) 0x6988c80
2014-06-12 14:45:57.435798 7f9d7715b700  5 -- op tracker -- , seq: 747383, time: 2014-06-12 14:45:57.435793, event: commit_sent, op: osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.435829 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] op_applied: 236163
2014-06-12 14:45:57.435851 7f9d7795c700  5 -- op tracker -- , seq: 747383, time: 2014-06-12 14:45:57.435845, event: op_applied, op: osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.435858 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] op_applied on primary on version 173'2859
2014-06-12 14:45:57.435866 7f9d7895e700 10 lookup_strip_header done  cid 3.2d7_head oid 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.435868 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] repop_all_applied: repop tid 236163 all applied 
2014-06-12 14:45:57.435878 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] eval_repop repgather(0xbe0d980 173'2859 rep_tid=236163 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:57.435894 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] log_op_stats osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 0.000000 lat 0.070227
2014-06-12 14:45:57.435910 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] publish_stats_to_osd 173:3667
2014-06-12 14:45:57.435924 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean]  requeue_ops 
2014-06-12 14:45:57.435935 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  removing repgather(0xbe0d980 173'2859 rep_tid=236163 committed?=1 applied?=1 op=osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:57.435951 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]    q front is repgather(0xbe0d980 173'2859 rep_tid=236163 committed?=1 applied?=1 op=osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:57.435966 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] remove_repop repgather(0xbe0d980 173'2859 rep_tid=236163 committed?=1 applied?=1 op=osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:57.435981 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  requeue_ops 
2014-06-12 14:45:57.436008 7f9d7795c700  5 -- op tracker -- , seq: 747383, time: 2014-06-12 14:45:57.436002, event: done, op: osd_op(client.4284.0:141382 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~1548288] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.436049 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:45:57.437136 7f9d7895e700 10 _generic_write 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 0~286720 = 0
2014-06-12 14:45:57.437175 7f9d7895e700 15 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.437189 7f9d7895e700 10 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 = 0
2014-06-12 14:45:57.437196 7f9d7895e700 15 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.437205 7f9d7895e700 10 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 = 0
2014-06-12 14:45:57.437327 7f9d7895e700 20 set_header setting 17355 cid 3.2d7_head oid 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 parent seq 0
2014-06-12 14:45:57.437359 7f9d7895e700 20 set_header setting 2282 cid meta oid 1fe0df15/pglog_3.2d7/0//-1 parent seq 0
2014-06-12 14:45:57.437370 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.437377 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:45:57.439868 7f9d7895e700 10 _do_op 0xbdfdce0 seq 2703 r = 0, finisher 0x6615cc0 0
2014-06-12 14:45:57.439891 7f9d7895e700 10 _finish_op 0xbdfdce0 seq 2703 osr(3.2d7 0x276d5c0)/0x276d5c0
2014-06-12 14:45:57.439897 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cbe60 (0 active)
2014-06-12 14:45:57.439905 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.439913 7f9d7715b700  5 -- op tracker -- , seq: 747384, time: 2014-06-12 14:45:57.439900, event: commit_sent, op: osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.439923 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2990 (0'0,173'2990] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2988 active] sub_op_modify_commit on op osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:45:57.439953 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:57.439988 7f9d7795c700  5 -- op tracker -- , seq: 747384, time: 2014-06-12 14:45:57.439982, event: sub_op_applied, op: osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.439996 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2990 (0'0,173'2990] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2989 active] sub_op_modify_applied on 0x6671180 op osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.440017 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2990 (0'0,173'2990] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2989 active] op_applied on primary on version 173'2990
2014-06-12 14:45:57.440029 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2990 (0'0,173'2990] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2989 active] op_applied on replica on version 173'2990
2014-06-12 14:45:57.440052 7f9d7795c700  5 -- op tracker -- , seq: 747384, time: 2014-06-12 14:45:57.440047, event: done, op: osd_sub_op(client.4301.0:145391 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.441292 7f9d523da700  5 -- op tracker -- , seq: 747386, time: 2014-06-12 14:45:57.439556, event: header_read, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.441303 7f9d523da700  5 -- op tracker -- , seq: 747386, time: 2014-06-12 14:45:57.439558, event: throttled, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.441309 7f9d523da700  5 -- op tracker -- , seq: 747386, time: 2014-06-12 14:45:57.441258, event: all_read, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.441315 7f9d523da700  5 -- op tracker -- , seq: 747386, time: 0.000000, event: dispatched, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.441321 7f9d523da700 20 osd.2 173 should_share_map client.4284 172.16.96.220:0/1241415 173
2014-06-12 14:45:57.441344 7f9d523da700 15 osd.2 173 enqueue_op 0x6f4ba00 prio 63 cost 811008 latency 0.001788 osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:57.441371 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.441380 7f9d68be2700 10 osd.2 173 dequeue_op 0x6f4ba00 prio 63 cost 811008 latency 0.001824 osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]
2014-06-12 14:45:57.441407 7f9d68be2700  5 -- op tracker -- , seq: 747386, time: 2014-06-12 14:45:57.441402, event: reached_pg, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.441421 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:45:57.441438 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] handle_message: 0x6f4ba00
2014-06-12 14:45:57.441452 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_op osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:45:57.441479 7f9d68be2700 15 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 '_'
2014-06-12 14:45:57.441713 7f9d68be2700 10 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 '_' = 0
2014-06-12 14:45:57.441730 7f9d68be2700 15 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 'snapset'
2014-06-12 14:45:57.441824 7f9d68be2700 10 getattr 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 'snapset' = 0
2014-06-12 14:45:57.441839 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] populate_obc_watchers fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.441852 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.441863 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] get_object_context: creating obc from disk: 0xa9aedc0
2014-06-12 14:45:57.441873 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] get_object_context: 0xa9aedc0 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 rwstate(none n=0 w=0) oi: fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3(173'2859 client.4284.0:141382 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2859) ssc: 0x9345880 snapset: 0=[]:[]+head
2014-06-12 14:45:57.441890 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] find_object_context fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 @head oi=fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3(173'2859 client.4284.0:141382 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2859)
2014-06-12 14:45:57.441916 7f9d68be2700  5 -- op tracker -- , seq: 747386, time: 2014-06-12 14:45:57.441910, event: started, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.441927 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] execute_ctx 0x5ca0600
2014-06-12 14:45:57.441946 7f9d68be2700  5 -- op tracker -- , seq: 747386, time: 2014-06-12 14:45:57.441940, event: started, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.441953 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_op fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] ov 173'2859 av 173'2860 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:45:57.441967 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008]
2014-06-12 14:45:57.441978 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:45:57.441993 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op  write 3383296~811008
2014-06-12 14:45:57.442009 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op_effects on session 0x5c8ce00
2014-06-12 14:45:57.442020 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] make_writeable fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 snapset=0x93458b8  snapc=0=[]
2014-06-12 14:45:57.442033 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] make_writeable fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:45:57.442044 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] finish_ctx fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 0x5ca0600 op modify  
2014-06-12 14:45:57.442058 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  set mtime to 2014-06-12 08:44:59.819331
2014-06-12 14:45:57.442081 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  final snapset 0=[]:[]+head in fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.442107 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  zeroing write result code 0
2014-06-12 14:45:57.442118 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] new_repop rep_tid 236164 on osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:45:57.442135 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] issue_repop rep_tid 236164 o fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.442161 7f9d68be2700  5 -- op tracker -- , seq: 747386, time: 2014-06-12 14:45:57.442156, event: waiting for subops from 21, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.442189 7f9d68be2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.442214 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2859 (0'0,173'2859] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] append_log log((0'0,173'2859], crt=0'0) [173'2860 (173'2859) modify   fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 by client.4284.0:141383 2014-06-12 08:44:59.819331]
2014-06-12 14:45:57.442239 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] add_log_entry 173'2860 (173'2859) modify   fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 by client.4284.0:141383 2014-06-12 08:44:59.819331
2014-06-12 14:45:57.442263 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] append_log  adding 1 keys
2014-06-12 14:45:57.442298 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2860, trimmed: 
2014-06-12 14:45:57.442322 7f9d68be2700  5 queue_transactions existing osr(3.131 0x276cee0)/0x276cee0
2014-06-12 14:45:57.442327 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.442329 7f9d68be2700  5 queue_op 0x683e3c0 seq 2584 osr(3.131 0x276cee0) 813301 bytes   (queue has 1 ops and 813301 bytes)
2014-06-12 14:45:57.442336 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] eval_repop repgather(0x1f067140 173'2860 rep_tid=236164 committed?=0 applied?=0 lock=0 op=osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:57.442356 7f9d68be2700 10 osd.2 173 dequeue_op 0x6f4ba00 finish
2014-06-12 14:45:57.442360 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.442363 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.442371 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b87e0 (1 active)
2014-06-12 14:45:57.442377 7f9d7f16b700  5 _do_op 0x683e3c0 seq 2584 osr(3.131 0x276cee0)/0x276cee0 start
2014-06-12 14:45:57.442381 7f9d7f16b700 10 _do_transaction on 0x10193800
2014-06-12 14:45:57.442387 7f9d7f16b700 15 _omap_setkeys meta/1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:57.442484 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:57.442503 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.442559 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.442585 7f9d7f16b700 15 _omap_rmkeys meta/1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:57.442592 7f9d7f16b700 10 _omap_rmkeys meta/1fe280aa/pglog_3.131/0//-1 r = 0
2014-06-12 14:45:57.442598 7f9d7f16b700 15 _omap_setkeys meta/1fe280aa/pglog_3.131/0//-1
2014-06-12 14:45:57.442617 7f9d7f16b700 15 _write 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 3383296~811008
2014-06-12 14:45:57.442722 7f9d7f16b700 10 lookup_strip_header done  cid 3.131_head oid fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.443002 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:45:57.445303 7f9d7f16b700 10 _generic_write 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 3383296~811008 = 0
2014-06-12 14:45:57.445395 7f9d7f16b700 15 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.445419 7f9d7f16b700 10 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:45:57.445427 7f9d7f16b700 15 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3
2014-06-12 14:45:57.445437 7f9d7f16b700 10 _setattrs 3.131_head/fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:45:57.445657 7f9d7f16b700 20 set_header setting 17294 cid 3.131_head oid fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 parent seq 0
2014-06-12 14:45:57.445744 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.445761 7f9d7f16b700 20 set_header setting 1996 cid meta oid 1fe280aa/pglog_3.131/0//-1 parent seq 0
2014-06-12 14:45:57.445773 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:45:57.450115 7f9d7f16b700 10 _do_op 0x683e3c0 seq 2584 r = 0, finisher 0x29015680 0x38e12700
2014-06-12 14:45:57.450134 7f9d7f16b700 10 _finish_op 0x683e3c0 seq 2584 osr(3.131 0x276cee0)/0x276cee0
2014-06-12 14:45:57.450151 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b87e0 (0 active)
2014-06-12 14:45:57.450155 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.450138 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_commit: 236164
2014-06-12 14:45:57.450166 7f9d7715b700  5 -- op tracker -- , seq: 747386, time: 2014-06-12 14:45:57.450158, event: op_commit, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.450191 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_applied: 236164
2014-06-12 14:45:57.450213 7f9d7795c700  5 -- op tracker -- , seq: 747386, time: 2014-06-12 14:45:57.450206, event: op_applied, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.450226 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_applied on primary on version 173'2860
2014-06-12 14:45:57.455018 7f9d57543700  5 -- op tracker -- , seq: 747387, time: 2014-06-12 14:45:57.448337, event: header_read, op: osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.455058 7f9d57543700  5 -- op tracker -- , seq: 747387, time: 2014-06-12 14:45:57.448340, event: throttled, op: osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.455068 7f9d57543700  5 -- op tracker -- , seq: 747387, time: 2014-06-12 14:45:57.454881, event: all_read, op: osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.455078 7f9d57543700  5 -- op tracker -- , seq: 747387, time: 0.000000, event: dispatched, op: osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.455089 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.455102 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:57.455123 7f9d57543700 15 osd.2 173 enqueue_op 0x39576200 prio 127 cost 1032999 latency 0.006785 osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.455386 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.455409 7f9d61bd4700 10 osd.2 173 dequeue_op 0x39576200 prio 127 cost 1032999 latency 0.007071 osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2d7( v 173'2990 (0'0,173'2990] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2989 active]
2014-06-12 14:45:57.455440 7f9d61bd4700  5 -- op tracker -- , seq: 747387, time: 2014-06-12 14:45:57.455436, event: reached_pg, op: osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.455453 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2990 (0'0,173'2990] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2989 active] handle_message: 0x39576200
2014-06-12 14:45:57.455469 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2990 (0'0,173'2990] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2989 active] sub_op_modify trans 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 v 173'2991 (transaction) 174
2014-06-12 14:45:57.455498 7f9d61bd4700  5 -- op tracker -- , seq: 747387, time: 2014-06-12 14:45:57.455494, event: started, op: osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.455520 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2990 (0'0,173'2990] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2989 active] append_log log((0'0,173'2990], crt=0'0) [173'2991 (173'2990) modify   88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 by client.4301.0:145392 2014-06-12 08:44:59.823633]
2014-06-12 14:45:57.455548 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2991 (0'0,173'2991] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2989 active] add_log_entry 173'2991 (173'2990) modify   88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 by client.4301.0:145392 2014-06-12 08:44:59.823633
2014-06-12 14:45:57.455571 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2991 (0'0,173'2991] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2989 active] append_log  adding 1 keys
2014-06-12 14:45:57.455608 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2991, trimmed: 
2014-06-12 14:45:57.455636 7f9d61bd4700  5 -- op tracker -- , seq: 747387, time: 2014-06-12 14:45:57.455630, event: started, op: osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.455646 7f9d61bd4700  5 queue_transactions existing osr(3.2d7 0x276d5c0)/0x276d5c0
2014-06-12 14:45:57.455657 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.455658 7f9d61bd4700  5 queue_op 0x1b0ba600 seq 2704 osr(3.2d7 0x276d5c0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:45:57.455666 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2991 (0'0,173'2991] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2989 active] do_sub_op osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.455681 7f9d61bd4700 10 osd.2 173 dequeue_op 0x39576200 finish
2014-06-12 14:45:57.455685 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.455687 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.455696 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cbe60 (1 active)
2014-06-12 14:45:57.455701 7f9d7e169700  5 _do_op 0x1b0ba600 seq 2704 osr(3.2d7 0x276d5c0)/0x276d5c0 start
2014-06-12 14:45:57.455705 7f9d7e169700 10 _do_transaction on 0x65cd970
2014-06-12 14:45:57.455711 7f9d7e169700 15 _omap_setkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.455877 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.455906 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.455968 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.455991 7f9d7e169700 15 _omap_rmkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.455997 7f9d7e169700 10 _omap_rmkeys meta/1fe0df15/pglog_3.2d7/0//-1 r = 0
2014-06-12 14:45:57.456002 7f9d7e169700 15 _omap_setkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.456122 7f9d7e169700 15 _write 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 286720~1032192
2014-06-12 14:45:57.456292 7f9d7e169700 10 lookup_strip_header done  cid 3.2d7_head oid 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.456457 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:45:57.458561 7f9d44101700  5 -- op tracker -- , seq: 747388, time: 2014-06-12 14:45:57.458511, event: header_read, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:57.458572 7f9d44101700  5 -- op tracker -- , seq: 747388, time: 2014-06-12 14:45:57.458513, event: throttled, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:57.458629 7f9d44101700  5 -- op tracker -- , seq: 747388, time: 2014-06-12 14:45:57.458525, event: all_read, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:57.458639 7f9d44101700  5 -- op tracker -- , seq: 747388, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:57.458643 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:45:57.458649 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:57.458659 7f9d44101700 15 osd.2 173 enqueue_op 0x34017800 prio 196 cost 0 latency 0.000147 osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0) v2
2014-06-12 14:45:57.458747 7f9d44101700  5 -- op tracker -- , seq: 747389, time: 2014-06-12 14:45:57.458683, event: header_read, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.458755 7f9d44101700  5 -- op tracker -- , seq: 747389, time: 2014-06-12 14:45:57.458683, event: throttled, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.458762 7f9d44101700  5 -- op tracker -- , seq: 747389, time: 2014-06-12 14:45:57.458728, event: all_read, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.458769 7f9d44101700  5 -- op tracker -- , seq: 747389, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.458772 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:45:57.458776 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:57.458782 7f9d44101700 15 osd.2 173 enqueue_op 0x34017b00 prio 196 cost 0 latency 0.000098 osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0) v2
2014-06-12 14:45:57.458825 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.458833 7f9d673df700 10 osd.2 173 dequeue_op 0x34017800 prio 196 cost 0 latency 0.000322 osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0) v2 pg pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]
2014-06-12 14:45:57.458853 7f9d673df700  5 -- op tracker -- , seq: 747388, time: 2014-06-12 14:45:57.458850, event: reached_pg, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:57.458861 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] handle_message: 0x34017800
2014-06-12 14:45:57.458876 7f9d673df700  5 -- op tracker -- , seq: 747388, time: 2014-06-12 14:45:57.458873, event: started, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:57.458882 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] sub_op_modify_reply: tid 236164 op  ack_type 1 from 21
2014-06-12 14:45:57.458901 7f9d673df700  5 -- op tracker -- , seq: 747386, time: 2014-06-12 14:45:57.458894, event: sub_op_applied_rec, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.458937 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] repop_all_applied: repop tid 236164 all applied 
2014-06-12 14:45:57.458949 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] eval_repop repgather(0x1f067140 173'2860 rep_tid=236164 committed?=0 applied?=1 lock=0 op=osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:57.458965 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  sending ack on repgather(0x1f067140 173'2860 rep_tid=236164 committed?=0 applied?=1 lock=0 op=osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) 0x133ff180
2014-06-12 14:45:57.459003 7f9d673df700 10 osd.2 173 dequeue_op 0x34017800 finish
2014-06-12 14:45:57.459012 7f9d673df700  5 -- op tracker -- , seq: 747388, time: 2014-06-12 14:45:57.459007, event: done, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ack, result = 0)
2014-06-12 14:45:57.459046 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.459053 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.459056 7f9d673df700 10 osd.2 173 dequeue_op 0x34017b00 prio 196 cost 0 latency 0.000373 osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0) v2 pg pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]
2014-06-12 14:45:57.459072 7f9d673df700  5 -- op tracker -- , seq: 747389, time: 2014-06-12 14:45:57.459069, event: reached_pg, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.459078 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] handle_message: 0x34017b00
2014-06-12 14:45:57.459091 7f9d673df700  5 -- op tracker -- , seq: 747389, time: 2014-06-12 14:45:57.459088, event: started, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.459096 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] sub_op_modify_reply: tid 236164 op  ack_type 4 from 21
2014-06-12 14:45:57.459113 7f9d673df700  5 -- op tracker -- , seq: 747386, time: 2014-06-12 14:45:57.459108, event: sub_op_commit_rec, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.459119 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] repop_all_committed: repop tid 236164 all committed 
2014-06-12 14:45:57.459169 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean] eval_repop repgather(0x1f067140 173'2860 rep_tid=236164 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:45:57.459185 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean] log_op_stats osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4 inb 811287 outb 0 rlat 0.019446 lat 0.019628
2014-06-12 14:45:57.459203 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean] publish_stats_to_osd 173:3668
2014-06-12 14:45:57.459217 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean]  sending commit on repgather(0x1f067140 173'2860 rep_tid=236164 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4) 0x17432500
2014-06-12 14:45:57.459252 7f9d673df700  5 -- op tracker -- , seq: 747386, time: 2014-06-12 14:45:57.459247, event: commit_sent, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.459261 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean]  requeue_ops 
2014-06-12 14:45:57.459271 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  removing repgather(0x1f067140 173'2860 rep_tid=236164 committed?=1 applied?=1 op=osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:57.459284 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]    q front is repgather(0x1f067140 173'2860 rep_tid=236164 committed?=1 applied?=1 op=osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:57.459298 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] remove_repop repgather(0x1f067140 173'2860 rep_tid=236164 committed?=1 applied?=1 op=osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:45:57.459311 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  requeue_ops 
2014-06-12 14:45:57.459341 7f9d673df700  5 -- op tracker -- , seq: 747386, time: 2014-06-12 14:45:57.459336, event: done, op: osd_op(client.4284.0:141383 rbd_data.10936b8b4567.0000000000000fd1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3383296~811008] 3.fb4d0931 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:45:57.459357 7f9d673df700 10 osd.2 173 dequeue_op 0x34017b00 finish
2014-06-12 14:45:57.459364 7f9d673df700  5 -- op tracker -- , seq: 747389, time: 2014-06-12 14:45:57.459360, event: done, op: osd_sub_op_reply(client.4284.0:141383 3.131 fb4d0931/rbd_data.10936b8b4567.0000000000000fd1/head//3 [] ondisk, result = 0)
2014-06-12 14:45:57.459374 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.459378 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.459389 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.461178 7f9d7e169700 10 _generic_write 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 286720~1032192 = 0
2014-06-12 14:45:57.461385 7f9d7e169700 15 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.461409 7f9d7e169700 10 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 = 0
2014-06-12 14:45:57.461419 7f9d7e169700 15 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.461425 7f9d7e169700 10 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 = 0
2014-06-12 14:45:57.461760 7f9d7e169700 20 set_header setting 17355 cid 3.2d7_head oid 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 parent seq 0
2014-06-12 14:45:57.461852 7f9d7e169700 20 set_header setting 2282 cid meta oid 1fe0df15/pglog_3.2d7/0//-1 parent seq 0
2014-06-12 14:45:57.461867 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.461873 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:45:57.462223 7f9d56432700  5 -- op tracker -- , seq: 747390, time: 2014-06-12 14:45:57.456632, event: header_read, op: osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.462237 7f9d56432700  5 -- op tracker -- , seq: 747390, time: 2014-06-12 14:45:57.456634, event: throttled, op: osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.462245 7f9d56432700  5 -- op tracker -- , seq: 747390, time: 2014-06-12 14:45:57.462186, event: all_read, op: osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.462253 7f9d56432700  5 -- op tracker -- , seq: 747390, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.462258 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.462265 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:57.462273 7f9d56432700 15 osd.2 173 enqueue_op 0x5289800 prio 127 cost 795427 latency 0.005641 osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.462300 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.462307 7f9d67be0700 10 osd.2 173 dequeue_op 0x5289800 prio 127 cost 795427 latency 0.005675 osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2467 (0'0,173'2467] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2466 active]
2014-06-12 14:45:57.462325 7f9d67be0700  5 -- op tracker -- , seq: 747390, time: 2014-06-12 14:45:57.462322, event: reached_pg, op: osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.462332 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2467 (0'0,173'2467] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2466 active] handle_message: 0x5289800
2014-06-12 14:45:57.462340 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2467 (0'0,173'2467] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2466 active] sub_op_modify trans 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 v 173'2468 (transaction) 174
2014-06-12 14:45:57.462350 7f9d67be0700  5 -- op tracker -- , seq: 747390, time: 2014-06-12 14:45:57.462348, event: started, op: osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.462363 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2467 (0'0,173'2467] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2466 active] append_log log((0'0,173'2467], crt=0'0) [173'2468 (173'684) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144987 2014-06-12 08:44:57.416051]
2014-06-12 14:45:57.462386 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2468 (0'0,173'2468] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2466 active] add_log_entry 173'2468 (173'684) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144987 2014-06-12 08:44:57.416051
2014-06-12 14:45:57.462408 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2468 (0'0,173'2468] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2466 active] append_log  adding 1 keys
2014-06-12 14:45:57.462434 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2468, trimmed: 
2014-06-12 14:45:57.462451 7f9d67be0700  5 -- op tracker -- , seq: 747390, time: 2014-06-12 14:45:57.462448, event: started, op: osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.462458 7f9d67be0700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:45:57.462461 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.462462 7f9d67be0700  5 queue_op 0x5541980 seq 2281 osr(3.2b 0x276cd20) 796908 bytes   (queue has 2 ops and 1831393 bytes)
2014-06-12 14:45:57.462471 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (2 active)
2014-06-12 14:45:57.462477 7f9d82171700  5 _do_op 0x5541980 seq 2281 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:45:57.462468 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2468 (0'0,173'2468] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2466 active] do_sub_op osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.462480 7f9d82171700 10 _do_transaction on 0x2e636470
2014-06-12 14:45:57.462480 7f9d67be0700 10 osd.2 173 dequeue_op 0x5289800 finish
2014-06-12 14:45:57.462484 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.462486 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.462485 7f9d82171700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:57.462624 7f9d82171700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:57.462650 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.464622 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:45:57.464640 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:45:57.464643 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:45:57.464647 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:45:57.464651 7f9d84175700 10 osd.2 173 send_pg_stats - 10 pgs updated
2014-06-12 14:45:57.464713 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:45:57.464718 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:57.464719 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:57.464722 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 44; oldest is 34.023068 seconds old
2014-06-12 14:45:57.464763 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:45:57.464771 7f9d84175700  0 log [WRN] : 4 slow requests, 2 included below; oldest blocked for > 34.023068 secs
2014-06-12 14:45:57.464776 7f9d84175700  0 log [WRN] : slow request 30.715177 seconds old, received at 2014-06-12 14:45:26.749545: osd_op(client.4311.0:139600 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 86016~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:45:57.464780 7f9d84175700  0 log [WRN] : slow request 30.667919 seconds old, received at 2014-06-12 14:45:26.796803: osd_op(client.4311.0:139601 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1634304~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:45:57.467234 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.467283 7f9d82171700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:57.467293 7f9d82171700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:45:57.467301 7f9d82171700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:45:57.467435 7f9d82171700 15 _write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 0~794624
2014-06-12 14:45:57.467551 7f9d7e169700 10 _do_op 0x1b0ba600 seq 2704 r = 0, finisher 0x2ae5540 0
2014-06-12 14:45:57.467564 7f9d7e169700 10 _finish_op 0x1b0ba600 seq 2704 osr(3.2d7 0x276d5c0)/0x276d5c0
2014-06-12 14:45:57.467570 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cbe60 (1 active)
2014-06-12 14:45:57.467578 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.467596 7f9d7715b700  5 -- op tracker -- , seq: 747387, time: 2014-06-12 14:45:57.467586, event: commit_sent, op: osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.467604 7f9d82171700 10 lookup_strip_header done  cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:45:57.467617 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2991 (0'0,173'2991] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2989 active] sub_op_modify_commit on op osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:45:57.467650 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:57.467712 7f9d7795c700  5 -- op tracker -- , seq: 747387, time: 2014-06-12 14:45:57.467703, event: sub_op_applied, op: osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.467720 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2991 (0'0,173'2991] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify_applied on 0x65cd880 op osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.467736 7f9d82171700 10 file_to_extents done 
2014-06-12 14:45:57.467737 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2991 (0'0,173'2991] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2990 active] op_applied on primary on version 173'2991
2014-06-12 14:45:57.467748 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2991 (0'0,173'2991] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2990 active] op_applied on replica on version 173'2991
2014-06-12 14:45:57.467772 7f9d7795c700  5 -- op tracker -- , seq: 747387, time: 2014-06-12 14:45:57.467768, event: done, op: osd_sub_op(client.4301.0:145392 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.469761 7f9d5703e700  5 -- op tracker -- , seq: 747391, time: 2014-06-12 14:45:57.468637, event: header_read, op: osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.469776 7f9d5703e700  5 -- op tracker -- , seq: 747391, time: 2014-06-12 14:45:57.468639, event: throttled, op: osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.469786 7f9d5703e700  5 -- op tracker -- , seq: 747391, time: 2014-06-12 14:45:57.469714, event: all_read, op: osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.469796 7f9d5703e700  5 -- op tracker -- , seq: 747391, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.469802 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.469810 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:57.469820 7f9d5703e700 15 osd.2 173 enqueue_op 0xe20a100 prio 127 cost 471847 latency 0.001183 osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.469871 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.469882 7f9d693e3700 10 osd.2 173 dequeue_op 0xe20a100 prio 127 cost 471847 latency 0.001245 osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2821 (0'0,173'2821] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active]
2014-06-12 14:45:57.469907 7f9d693e3700  5 -- op tracker -- , seq: 747391, time: 2014-06-12 14:45:57.469902, event: reached_pg, op: osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.469917 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2821 (0'0,173'2821] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active] handle_message: 0xe20a100
2014-06-12 14:45:57.469930 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2821 (0'0,173'2821] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active] sub_op_modify trans 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 v 173'2822 (transaction) 174
2014-06-12 14:45:57.469945 7f9d693e3700  5 -- op tracker -- , seq: 747391, time: 2014-06-12 14:45:57.469941, event: started, op: osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.469961 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2821 (0'0,173'2821] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active] append_log log((0'0,173'2821], crt=0'0) [173'2822 (173'845) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146003 2014-06-12 08:44:59.833462]
2014-06-12 14:45:57.469986 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2822 (0'0,173'2822] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active] add_log_entry 173'2822 (173'845) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146003 2014-06-12 08:44:59.833462
2014-06-12 14:45:57.470008 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2822 (0'0,173'2822] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active] append_log  adding 1 keys
2014-06-12 14:45:57.470048 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2822, trimmed: 
2014-06-12 14:45:57.470072 7f9d693e3700  5 -- op tracker -- , seq: 747391, time: 2014-06-12 14:45:57.470066, event: started, op: osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.470082 7f9d693e3700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.470086 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.470087 7f9d693e3700  5 queue_op 0x287f1a0 seq 2577 osr(3.39f 0x276d720) 473333 bytes   (queue has 2 ops and 1270241 bytes)
2014-06-12 14:45:57.470095 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2822 (0'0,173'2822] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active] do_sub_op osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.470108 7f9d693e3700 10 osd.2 173 dequeue_op 0xe20a100 finish
2014-06-12 14:45:57.470112 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.470115 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.470941 7f9d82171700 10 _generic_write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 0~794624 = 0
2014-06-12 14:45:57.470950 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (2 active)
2014-06-12 14:45:57.470964 7f9d7e96a700  5 _do_op 0x287f1a0 seq 2577 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:45:57.470969 7f9d7e96a700 10 _do_transaction on 0x27756470
2014-06-12 14:45:57.470976 7f9d7e96a700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.471025 7f9d82171700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:45:57.471046 7f9d82171700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:45:57.471051 7f9d82171700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:45:57.471058 7f9d82171700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:45:57.471215 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.471239 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.471327 7f9d82171700 20 set_header setting 17866 cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 parent seq 0
2014-06-12 14:45:57.471463 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.471477 7f9d82171700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:45:57.471485 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:45:57.477010 7f9d5703e700  5 -- op tracker -- , seq: 747392, time: 2014-06-12 14:45:57.475508, event: header_read, op: osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.477029 7f9d5703e700  5 -- op tracker -- , seq: 747392, time: 2014-06-12 14:45:57.475511, event: throttled, op: osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.477038 7f9d5703e700  5 -- op tracker -- , seq: 747392, time: 2014-06-12 14:45:57.476959, event: all_read, op: osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.477047 7f9d5703e700  5 -- op tracker -- , seq: 747392, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.477053 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.477059 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:57.477068 7f9d5703e700 15 osd.2 173 enqueue_op 0xe20b800 prio 127 cost 516903 latency 0.001559 osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.477096 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.477108 7f9d62bd6700 10 osd.2 173 dequeue_op 0xe20b800 prio 127 cost 516903 latency 0.001599 osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2822 (0'0,173'2822] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active]
2014-06-12 14:45:57.477133 7f9d62bd6700  5 -- op tracker -- , seq: 747392, time: 2014-06-12 14:45:57.477129, event: reached_pg, op: osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.477143 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2822 (0'0,173'2822] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active] handle_message: 0xe20b800
2014-06-12 14:45:57.477155 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2822 (0'0,173'2822] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active] sub_op_modify trans 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 v 173'2823 (transaction) 174
2014-06-12 14:45:57.477173 7f9d62bd6700  5 -- op tracker -- , seq: 747392, time: 2014-06-12 14:45:57.477168, event: started, op: osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.477189 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2822 (0'0,173'2822] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active] append_log log((0'0,173'2822], crt=0'0) [173'2823 (173'2822) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146004 2014-06-12 08:44:59.853288]
2014-06-12 14:45:57.477214 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active] add_log_entry 173'2823 (173'2822) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146004 2014-06-12 08:44:59.853288
2014-06-12 14:45:57.477236 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active] append_log  adding 1 keys
2014-06-12 14:45:57.477272 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2823, trimmed: 
2014-06-12 14:45:57.477297 7f9d62bd6700  5 -- op tracker -- , seq: 747392, time: 2014-06-12 14:45:57.477292, event: started, op: osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.477308 7f9d62bd6700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.477313 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.477315 7f9d62bd6700  5 queue_op 0x565ac60 seq 2578 osr(3.39f 0x276d720) 518389 bytes   (queue has 3 ops and 1788630 bytes)
2014-06-12 14:45:57.477323 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active] do_sub_op osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.477338 7f9d62bd6700 10 osd.2 173 dequeue_op 0xe20b800 finish
2014-06-12 14:45:57.477342 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.477344 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.477498 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.477530 7f9d7e96a700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.477537 7f9d7e96a700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:45:57.477545 7f9d7e96a700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.477565 7f9d7e96a700 15 _write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 0~471040
2014-06-12 14:45:57.477786 7f9d7e96a700 10 lookup_strip_header done  cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.478080 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:45:57.478281 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (3 active)
2014-06-12 14:45:57.479929 7f9d7e96a700 10 _generic_write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 0~471040 = 0
2014-06-12 14:45:57.479994 7f9d7e96a700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.480009 7f9d7e96a700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.480016 7f9d7e96a700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.480025 7f9d7e96a700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.480241 7f9d7e96a700 20 set_header setting 17144 cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 parent seq 0
2014-06-12 14:45:57.480330 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.480346 7f9d7e96a700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:45:57.480358 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:45:57.483799 7f9d7e96a700 10 _do_op 0x287f1a0 seq 2577 r = 0, finisher 0x288c020 0
2014-06-12 14:45:57.483867 7f9d7715b700  5 -- op tracker -- , seq: 747391, time: 2014-06-12 14:45:57.483848, event: commit_sent, op: osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.483925 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2820 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:45:57.483960 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:57.483991 7f9d7e96a700 10 _finish_op 0x287f1a0 seq 2577 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.483999 7f9d7d167700  5 _do_op 0x565ac60 seq 2578 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:45:57.484004 7f9d7d167700 10 _do_transaction on 0x5ba8d30
2014-06-12 14:45:57.484015 7f9d7d167700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.484173 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.484195 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.484259 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.484281 7f9d7d167700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.484286 7f9d7d167700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:45:57.484292 7f9d7d167700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.484377 7f9d82171700 10 _do_op 0x5541980 seq 2281 r = 0, finisher 0x53ca180 0
2014-06-12 14:45:57.484396 7f9d7d167700 15 _write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 471040~516096
2014-06-12 14:45:57.484551 7f9d7d167700 10 lookup_strip_header done  cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.484663 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:45:57.485653 7f9d7715b700  5 -- op tracker -- , seq: 747390, time: 2014-06-12 14:45:57.485642, event: commit_sent, op: osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.485666 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2468 (0'0,173'2468] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2466 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:57.485690 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:57.490192 7f9d7d167700 10 _generic_write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 471040~516096 = 0
2014-06-12 14:45:57.490267 7f9d7d167700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.490283 7f9d7d167700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.490290 7f9d7d167700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.490299 7f9d7d167700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.490538 7f9d7d167700 20 set_header setting 17144 cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 parent seq 0
2014-06-12 14:45:57.490613 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.490651 7f9d7d167700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:45:57.490663 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:45:57.490680 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (2 active)
2014-06-12 14:45:57.490693 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.490697 7f9d7795c700  5 -- op tracker -- , seq: 747391, time: 2014-06-12 14:45:57.490687, event: sub_op_applied, op: osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.490704 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2821 active] sub_op_modify_applied on 0x27756380 op osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.490724 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2821 active] op_applied on primary on version 173'2822
2014-06-12 14:45:57.490739 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2821 active] op_applied on replica on version 173'2822
2014-06-12 14:45:57.490761 7f9d7795c700  5 -- op tracker -- , seq: 747391, time: 2014-06-12 14:45:57.490757, event: done, op: osd_sub_op(client.4302.0:146003 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2822 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.490789 7f9d82171700 10 _finish_op 0x5541980 seq 2281 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:45:57.490808 7f9d7795c700  5 -- op tracker -- , seq: 747390, time: 2014-06-12 14:45:57.490802, event: sub_op_applied, op: osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.490814 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2468 (0'0,173'2468] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2467 active] sub_op_modify_applied on 0x2e636380 op osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.490835 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2468 (0'0,173'2468] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2467 active] op_applied on primary on version 173'2468
2014-06-12 14:45:57.490844 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2468 (0'0,173'2468] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2467 active] op_applied on replica on version 173'2468
2014-06-12 14:45:57.490872 7f9d82171700  5 -- op tracker -- , seq: 747390, time: 2014-06-12 14:45:57.490864, event: done, op: osd_sub_op(client.4314.0:144987 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.490897 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (1 active)
2014-06-12 14:45:57.490900 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.495895 7f9d7d167700 10 _do_op 0x565ac60 seq 2578 r = 0, finisher 0x20aaac20 0
2014-06-12 14:45:57.495948 7f9d7715b700  5 -- op tracker -- , seq: 747392, time: 2014-06-12 14:45:57.495937, event: commit_sent, op: osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.495965 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2821 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:45:57.495996 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:57.496073 7f9d7d167700 10 _finish_op 0x565ac60 seq 2578 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.496083 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (0 active)
2014-06-12 14:45:57.496093 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.496099 7f9d7795c700  5 -- op tracker -- , seq: 747392, time: 2014-06-12 14:45:57.496091, event: sub_op_applied, op: osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.496107 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] sub_op_modify_applied on 0x5ba8c40 op osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.496123 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] op_applied on primary on version 173'2823
2014-06-12 14:45:57.496133 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] op_applied on replica on version 173'2823
2014-06-12 14:45:57.496155 7f9d7795c700  5 -- op tracker -- , seq: 747392, time: 2014-06-12 14:45:57.496151, event: done, op: osd_sub_op(client.4302.0:146004 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2823 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.502017 7f9d57543700  5 -- op tracker -- , seq: 747393, time: 2014-06-12 14:45:57.499730, event: header_read, op: osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.502036 7f9d57543700  5 -- op tracker -- , seq: 747393, time: 2014-06-12 14:45:57.499731, event: throttled, op: osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.502044 7f9d57543700  5 -- op tracker -- , seq: 747393, time: 2014-06-12 14:45:57.501964, event: all_read, op: osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.502053 7f9d57543700  5 -- op tracker -- , seq: 747393, time: 0.000000, event: dispatched, op: osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.502059 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.502066 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:57.502075 7f9d57543700 15 osd.2 173 enqueue_op 0x5ee8c00 prio 127 cost 1032999 latency 0.002344 osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.502108 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.502120 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x5ee8c00 prio 127 cost 1032999 latency 0.002389 osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2d7( v 173'2991 (0'0,173'2991] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2990 active]
2014-06-12 14:45:57.502148 7f9d6b3e7700  5 -- op tracker -- , seq: 747393, time: 2014-06-12 14:45:57.502144, event: reached_pg, op: osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.502166 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2991 (0'0,173'2991] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2990 active] handle_message: 0x5ee8c00
2014-06-12 14:45:57.502179 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2991 (0'0,173'2991] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify trans 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 v 173'2992 (transaction) 174
2014-06-12 14:45:57.502196 7f9d6b3e7700  5 -- op tracker -- , seq: 747393, time: 2014-06-12 14:45:57.502192, event: started, op: osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.502216 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2991 (0'0,173'2991] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2990 active] append_log log((0'0,173'2991], crt=0'0) [173'2992 (173'2991) modify   88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 by client.4301.0:145393 2014-06-12 08:44:59.866645]
2014-06-12 14:45:57.502250 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2992 (0'0,173'2992] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2990 active] add_log_entry 173'2992 (173'2991) modify   88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 by client.4301.0:145393 2014-06-12 08:44:59.866645
2014-06-12 14:45:57.502274 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2992 (0'0,173'2992] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2990 active] append_log  adding 1 keys
2014-06-12 14:45:57.502311 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2992, trimmed: 
2014-06-12 14:45:57.502349 7f9d6b3e7700  5 -- op tracker -- , seq: 747393, time: 2014-06-12 14:45:57.502344, event: started, op: osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.502363 7f9d6b3e7700  5 queue_transactions existing osr(3.2d7 0x276d5c0)/0x276d5c0
2014-06-12 14:45:57.502368 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.502369 7f9d6b3e7700  5 queue_op 0x776c7e0 seq 2705 osr(3.2d7 0x276d5c0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:45:57.502377 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2992 (0'0,173'2992] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2990 active] do_sub_op osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.502392 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x5ee8c00 finish
2014-06-12 14:45:57.502396 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.502399 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.502846 7f9d5703e700  5 -- op tracker -- , seq: 747394, time: 2014-06-12 14:45:57.500032, event: header_read, op: osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.502862 7f9d5703e700  5 -- op tracker -- , seq: 747394, time: 2014-06-12 14:45:57.500033, event: throttled, op: osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.502871 7f9d5703e700  5 -- op tracker -- , seq: 747394, time: 2014-06-12 14:45:57.502812, event: all_read, op: osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.502891 7f9d5703e700  5 -- op tracker -- , seq: 747394, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.502896 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.502902 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:57.502909 7f9d5703e700 15 osd.2 173 enqueue_op 0xe20ad00 prio 127 cost 516903 latency 0.002877 osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.503068 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.503078 7f9d64bda700 10 osd.2 173 dequeue_op 0xe20ad00 prio 127 cost 516903 latency 0.003046 osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active]
2014-06-12 14:45:57.503105 7f9d64bda700  5 -- op tracker -- , seq: 747394, time: 2014-06-12 14:45:57.503100, event: reached_pg, op: osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.503116 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] handle_message: 0xe20ad00
2014-06-12 14:45:57.503130 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] sub_op_modify trans 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 v 173'2824 (transaction) 174
2014-06-12 14:45:57.503146 7f9d64bda700  5 -- op tracker -- , seq: 747394, time: 2014-06-12 14:45:57.503142, event: started, op: osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.503163 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2823 (0'0,173'2823] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] append_log log((0'0,173'2823], crt=0'0) [173'2824 (173'2823) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146005 2014-06-12 08:44:59.875818]
2014-06-12 14:45:57.503189 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2824 (0'0,173'2824] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] add_log_entry 173'2824 (173'2823) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146005 2014-06-12 08:44:59.875818
2014-06-12 14:45:57.503211 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2824 (0'0,173'2824] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] append_log  adding 1 keys
2014-06-12 14:45:57.503245 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2824, trimmed: 
2014-06-12 14:45:57.503275 7f9d64bda700  5 -- op tracker -- , seq: 747394, time: 2014-06-12 14:45:57.503269, event: started, op: osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.503288 7f9d64bda700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.503293 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.503294 7f9d64bda700  5 queue_op 0x683e000 seq 2579 osr(3.39f 0x276d720) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:45:57.503301 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2824 (0'0,173'2824] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] do_sub_op osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.503315 7f9d64bda700 10 osd.2 173 dequeue_op 0xe20ad00 finish
2014-06-12 14:45:57.503318 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.503321 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.503331 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cbe60 (1 active)
2014-06-12 14:45:57.503345 7f9d7815d700  5 _do_op 0x776c7e0 seq 2705 osr(3.2d7 0x276d5c0)/0x276d5c0 start
2014-06-12 14:45:57.503350 7f9d7815d700 10 _do_transaction on 0x1cc29970
2014-06-12 14:45:57.503357 7f9d7815d700 15 _omap_setkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.503516 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.503539 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.503602 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.503628 7f9d7815d700 15 _omap_rmkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.503636 7f9d7815d700 10 _omap_rmkeys meta/1fe0df15/pglog_3.2d7/0//-1 r = 0
2014-06-12 14:45:57.503643 7f9d7815d700 15 _omap_setkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.503657 7f9d7815d700 15 _write 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 1318912~1032192
2014-06-12 14:45:57.503827 7f9d7815d700 10 lookup_strip_header done  cid 3.2d7_head oid 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.503856 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:45:57.508112 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (2 active)
2014-06-12 14:45:57.508129 7f9d7d968700  5 _do_op 0x683e000 seq 2579 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:45:57.508136 7f9d7d968700 10 _do_transaction on 0x587f430
2014-06-12 14:45:57.508144 7f9d7d968700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.508281 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.508301 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.509735 7f9d7815d700 10 _generic_write 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 1318912~1032192 = 0
2014-06-12 14:45:57.509891 7f9d7815d700 15 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.509909 7f9d7815d700 10 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 = 0
2014-06-12 14:45:57.509918 7f9d7815d700 15 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.509931 7f9d7815d700 10 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 = 0
2014-06-12 14:45:57.510138 7f9d5703e700  5 -- op tracker -- , seq: 747395, time: 2014-06-12 14:45:57.502938, event: header_read, op: osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.510152 7f9d5703e700  5 -- op tracker -- , seq: 747395, time: 2014-06-12 14:45:57.502943, event: throttled, op: osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.510162 7f9d5703e700  5 -- op tracker -- , seq: 747395, time: 2014-06-12 14:45:57.510098, event: all_read, op: osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.510173 7f9d5703e700  5 -- op tracker -- , seq: 747395, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.510180 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.510189 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:57.510198 7f9d5703e700 15 osd.2 173 enqueue_op 0xe20a400 prio 127 cost 516903 latency 0.007260 osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.510228 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.510238 7f9d683e1700 10 osd.2 173 dequeue_op 0xe20a400 prio 127 cost 516903 latency 0.007300 osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2824 (0'0,173'2824] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active]
2014-06-12 14:45:57.510264 7f9d683e1700  5 -- op tracker -- , seq: 747395, time: 2014-06-12 14:45:57.510259, event: reached_pg, op: osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.510274 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2824 (0'0,173'2824] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] handle_message: 0xe20a400
2014-06-12 14:45:57.510289 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2824 (0'0,173'2824] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] sub_op_modify trans 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 v 173'2825 (transaction) 174
2014-06-12 14:45:57.510310 7f9d683e1700  5 -- op tracker -- , seq: 747395, time: 2014-06-12 14:45:57.510305, event: started, op: osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.510317 7f9d7815d700 20 set_header setting 17355 cid 3.2d7_head oid 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 parent seq 0
2014-06-12 14:45:57.510327 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2824 (0'0,173'2824] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] append_log log((0'0,173'2824], crt=0'0) [173'2825 (173'2824) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146006 2014-06-12 08:44:59.879724]
2014-06-12 14:45:57.510351 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2825 (0'0,173'2825] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] add_log_entry 173'2825 (173'2824) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146006 2014-06-12 08:44:59.879724
2014-06-12 14:45:57.510376 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2825 (0'0,173'2825] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] append_log  adding 1 keys
2014-06-12 14:45:57.510411 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2825, trimmed: 
2014-06-12 14:45:57.510438 7f9d683e1700  5 -- op tracker -- , seq: 747395, time: 2014-06-12 14:45:57.510434, event: started, op: osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.510442 7f9d7815d700 20 set_header setting 2282 cid meta oid 1fe0df15/pglog_3.2d7/0//-1 parent seq 0
2014-06-12 14:45:57.510451 7f9d683e1700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.510456 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.510458 7f9d683e1700  5 queue_op 0x5395e60 seq 2580 osr(3.39f 0x276d720) 518389 bytes   (queue has 3 ops and 2071263 bytes)
2014-06-12 14:45:57.510458 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.510467 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:45:57.510466 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2825 (0'0,173'2825] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] do_sub_op osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.510483 7f9d683e1700 10 osd.2 173 dequeue_op 0xe20a400 finish
2014-06-12 14:45:57.510487 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.510489 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.510499 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (3 active)
2014-06-12 14:45:57.516308 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.516340 7f9d7d968700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.516346 7f9d7d968700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:45:57.516354 7f9d7d968700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.516369 7f9d7d968700 15 _write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 987136~516096
2014-06-12 14:45:57.516536 7f9d7d968700 10 lookup_strip_header done  cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.516667 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:45:57.516686 7f9d7815d700 10 _do_op 0x776c7e0 seq 2705 r = 0, finisher 0x73eaaa0 0
2014-06-12 14:45:57.516717 7f9d7715b700  5 -- op tracker -- , seq: 747393, time: 2014-06-12 14:45:57.516708, event: commit_sent, op: osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.516729 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2992 (0'0,173'2992] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify_commit on op osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:45:57.516756 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:57.516782 7f9d7815d700 10 _finish_op 0x776c7e0 seq 2705 osr(3.2d7 0x276d5c0)/0x276d5c0
2014-06-12 14:45:57.516802 7f9d7795c700  5 -- op tracker -- , seq: 747393, time: 2014-06-12 14:45:57.516797, event: sub_op_applied, op: osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.516810 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2992 (0'0,173'2992] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2991 active] sub_op_modify_applied on 0x1cc29880 op osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.516825 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2992 (0'0,173'2992] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2991 active] op_applied on primary on version 173'2992
2014-06-12 14:45:57.516835 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2992 (0'0,173'2992] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2991 active] op_applied on replica on version 173'2992
2014-06-12 14:45:57.516862 7f9d7815d700  5 -- op tracker -- , seq: 747393, time: 2014-06-12 14:45:57.516854, event: done, op: osd_sub_op(client.4301.0:145393 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.516887 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cbe60 (2 active)
2014-06-12 14:45:57.516891 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.518662 7f9d7d968700 10 _generic_write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 987136~516096 = 0
2014-06-12 14:45:57.518720 7f9d7d968700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.518738 7f9d7d968700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.518752 7f9d7d968700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.518761 7f9d7d968700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.518982 7f9d7d968700 20 set_header setting 17144 cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 parent seq 0
2014-06-12 14:45:57.519062 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.519078 7f9d7d968700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:45:57.519088 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:45:57.522417 7f9d5703e700  5 -- op tracker -- , seq: 747396, time: 2014-06-12 14:45:57.519296, event: header_read, op: osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.522439 7f9d5703e700  5 -- op tracker -- , seq: 747396, time: 2014-06-12 14:45:57.519298, event: throttled, op: osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.522450 7f9d5703e700  5 -- op tracker -- , seq: 747396, time: 2014-06-12 14:45:57.522360, event: all_read, op: osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.522459 7f9d5703e700  5 -- op tracker -- , seq: 747396, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.522466 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.522474 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:57.522486 7f9d5703e700 15 osd.2 173 enqueue_op 0x24c9c900 prio 127 cost 1549095 latency 0.003189 osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.522520 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.522527 7f9d65bdc700 10 osd.2 173 dequeue_op 0x24c9c900 prio 127 cost 1549095 latency 0.003231 osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2825 (0'0,173'2825] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active]
2014-06-12 14:45:57.522555 7f9d65bdc700  5 -- op tracker -- , seq: 747396, time: 2014-06-12 14:45:57.522550, event: reached_pg, op: osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.522566 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2825 (0'0,173'2825] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] handle_message: 0x24c9c900
2014-06-12 14:45:57.522582 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2825 (0'0,173'2825] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] sub_op_modify trans 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 v 173'2826 (transaction) 174
2014-06-12 14:45:57.522602 7f9d65bdc700  5 -- op tracker -- , seq: 747396, time: 2014-06-12 14:45:57.522598, event: started, op: osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.522624 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2825 (0'0,173'2825] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] append_log log((0'0,173'2825], crt=0'0) [173'2826 (173'2825) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146007 2014-06-12 08:44:59.891391]
2014-06-12 14:45:57.522649 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] add_log_entry 173'2826 (173'2825) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146007 2014-06-12 08:44:59.891391
2014-06-12 14:45:57.522672 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] append_log  adding 1 keys
2014-06-12 14:45:57.522708 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2826, trimmed: 
2014-06-12 14:45:57.522734 7f9d65bdc700  5 -- op tracker -- , seq: 747396, time: 2014-06-12 14:45:57.522729, event: started, op: osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.522744 7f9d65bdc700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.522749 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.522750 7f9d65bdc700  5 queue_op 0x6c4f320 seq 2581 osr(3.39f 0x276d720) 1550581 bytes   (queue has 3 ops and 2587359 bytes)
2014-06-12 14:45:57.522783 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] do_sub_op osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.522799 7f9d65bdc700 10 osd.2 173 dequeue_op 0x24c9c900 finish
2014-06-12 14:45:57.522803 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.522805 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.525386 7f9d7d968700 10 _do_op 0x683e000 seq 2579 r = 0, finisher 0x6bb33a0 0
2014-06-12 14:45:57.525431 7f9d7715b700  5 -- op tracker -- , seq: 747394, time: 2014-06-12 14:45:57.525421, event: commit_sent, op: osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.525444 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2822 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:45:57.525472 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:57.525493 7f9d7d968700 10 _finish_op 0x683e000 seq 2579 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.525502 7f9d8116f700  5 _do_op 0x5395e60 seq 2580 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:45:57.525506 7f9d8116f700 10 _do_transaction on 0x2124d30
2014-06-12 14:45:57.525515 7f9d8116f700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.525699 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.525728 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.525796 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.525822 7f9d8116f700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.525829 7f9d8116f700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:45:57.525835 7f9d8116f700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.525951 7f9d8116f700 15 _write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 1503232~516096
2014-06-12 14:45:57.526115 7f9d8116f700 10 lookup_strip_header done  cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.526218 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:45:57.528145 7f9d8116f700 10 _generic_write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 1503232~516096 = 0
2014-06-12 14:45:57.528207 7f9d8116f700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.528223 7f9d8116f700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.528235 7f9d8116f700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.528244 7f9d8116f700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.528490 7f9d8116f700 20 set_header setting 17144 cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 parent seq 0
2014-06-12 14:45:57.528582 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.528598 7f9d8116f700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:45:57.528609 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:45:57.529249 7f9d7795c700  5 -- op tracker -- , seq: 747394, time: 2014-06-12 14:45:57.529236, event: sub_op_applied, op: osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.529260 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2823 active] sub_op_modify_applied on 0x587f340 op osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.529283 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2823 active] op_applied on primary on version 173'2824
2014-06-12 14:45:57.529293 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2823 active] op_applied on replica on version 173'2824
2014-06-12 14:45:57.529326 7f9d7d968700  5 -- op tracker -- , seq: 747394, time: 2014-06-12 14:45:57.529317, event: done, op: osd_sub_op(client.4302.0:146005 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2824 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.529367 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (1 active)
2014-06-12 14:45:57.529371 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (2 active)
2014-06-12 14:45:57.532015 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.532268 7f9d8116f700 10 _do_op 0x5395e60 seq 2580 r = 0, finisher 0x288c420 0
2014-06-12 14:45:57.532636 7f9d7715b700  5 -- op tracker -- , seq: 747395, time: 2014-06-12 14:45:57.532625, event: commit_sent, op: osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.532646 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2823 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:45:57.532671 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:57.532716 7f9d8116f700 10 _finish_op 0x5395e60 seq 2580 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.532732 7f9d7d968700  5 _do_op 0x6c4f320 seq 2581 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:45:57.532736 7f9d7d968700 10 _do_transaction on 0x61e67f0
2014-06-12 14:45:57.532744 7f9d7d968700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.532918 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.532942 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.533023 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.533044 7f9d7d968700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.533049 7f9d7d968700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:45:57.533055 7f9d7d968700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.533067 7f9d7d968700 15 _write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 2019328~1548288
2014-06-12 14:45:57.533220 7f9d7d968700 10 lookup_strip_header done  cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.533444 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:45:57.533783 7f9d7795c700  5 -- op tracker -- , seq: 747395, time: 2014-06-12 14:45:57.533776, event: sub_op_applied, op: osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.533791 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] sub_op_modify_applied on 0x2124c40 op osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.533806 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] op_applied on primary on version 173'2825
2014-06-12 14:45:57.533814 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] op_applied on replica on version 173'2825
2014-06-12 14:45:57.533852 7f9d8116f700  5 -- op tracker -- , seq: 747395, time: 2014-06-12 14:45:57.533844, event: done, op: osd_sub_op(client.4302.0:146006 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2825 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.533885 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (1 active)
2014-06-12 14:45:57.533888 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.534535 7f9d5703e700  5 -- op tracker -- , seq: 747397, time: 2014-06-12 14:45:57.533946, event: header_read, op: osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.534547 7f9d5703e700  5 -- op tracker -- , seq: 747397, time: 2014-06-12 14:45:57.533948, event: throttled, op: osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.534557 7f9d5703e700  5 -- op tracker -- , seq: 747397, time: 2014-06-12 14:45:57.534500, event: all_read, op: osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.534565 7f9d5703e700  5 -- op tracker -- , seq: 747397, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.534571 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.534578 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:57.534587 7f9d5703e700 15 osd.2 173 enqueue_op 0x20e88c00 prio 127 cost 516903 latency 0.000640 osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.534622 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.534629 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20e88c00 prio 127 cost 516903 latency 0.000683 osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active]
2014-06-12 14:45:57.534650 7f9d6abe6700  5 -- op tracker -- , seq: 747397, time: 2014-06-12 14:45:57.534646, event: reached_pg, op: osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.534659 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] handle_message: 0x20e88c00
2014-06-12 14:45:57.534671 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] sub_op_modify trans 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 v 173'2827 (transaction) 174
2014-06-12 14:45:57.534688 7f9d6abe6700  5 -- op tracker -- , seq: 747397, time: 2014-06-12 14:45:57.534684, event: started, op: osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.534705 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2826 (0'0,173'2826] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] append_log log((0'0,173'2826], crt=0'0) [173'2827 (173'2826) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146008 2014-06-12 08:44:59.910066]
2014-06-12 14:45:57.534734 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2827 (0'0,173'2827] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] add_log_entry 173'2827 (173'2826) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146008 2014-06-12 08:44:59.910066
2014-06-12 14:45:57.534760 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2827 (0'0,173'2827] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] append_log  adding 1 keys
2014-06-12 14:45:57.534796 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2827, trimmed: 
2014-06-12 14:45:57.534819 7f9d6abe6700  5 -- op tracker -- , seq: 747397, time: 2014-06-12 14:45:57.534814, event: started, op: osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.534828 7f9d6abe6700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.534832 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.534833 7f9d6abe6700  5 queue_op 0x142a9260 seq 2582 osr(3.39f 0x276d720) 518389 bytes   (queue has 2 ops and 2068970 bytes)
2014-06-12 14:45:57.534841 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2827 (0'0,173'2827] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] do_sub_op osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.534854 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20e88c00 finish
2014-06-12 14:45:57.534857 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.534859 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.534867 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (2 active)
2014-06-12 14:45:57.538890 7f9d5703e700  5 -- op tracker -- , seq: 747398, time: 2014-06-12 14:45:57.538685, event: header_read, op: osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.538902 7f9d5703e700  5 -- op tracker -- , seq: 747398, time: 2014-06-12 14:45:57.538687, event: throttled, op: osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.538911 7f9d5703e700  5 -- op tracker -- , seq: 747398, time: 2014-06-12 14:45:57.538850, event: all_read, op: osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.538920 7f9d5703e700  5 -- op tracker -- , seq: 747398, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.538926 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.538932 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:57.538941 7f9d5703e700 15 osd.2 173 enqueue_op 0x20e89400 prio 127 cost 111399 latency 0.000256 osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.538975 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.538984 7f9d66bde700 10 osd.2 173 dequeue_op 0x20e89400 prio 127 cost 111399 latency 0.000299 osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2827 (0'0,173'2827] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active]
2014-06-12 14:45:57.539009 7f9d66bde700  5 -- op tracker -- , seq: 747398, time: 2014-06-12 14:45:57.539005, event: reached_pg, op: osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.539021 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2827 (0'0,173'2827] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] handle_message: 0x20e89400
2014-06-12 14:45:57.539034 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2827 (0'0,173'2827] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] sub_op_modify trans 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 v 173'2828 (transaction) 174
2014-06-12 14:45:57.539050 7f9d66bde700  5 -- op tracker -- , seq: 747398, time: 2014-06-12 14:45:57.539046, event: started, op: osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.539067 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2827 (0'0,173'2827] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] append_log log((0'0,173'2827], crt=0'0) [173'2828 (173'2827) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146009 2014-06-12 08:44:59.915893]
2014-06-12 14:45:57.539092 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] add_log_entry 173'2828 (173'2827) modify   1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 by client.4302.0:146009 2014-06-12 08:44:59.915893
2014-06-12 14:45:57.539114 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] append_log  adding 1 keys
2014-06-12 14:45:57.539148 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2828, trimmed: 
2014-06-12 14:45:57.539172 7f9d66bde700  5 -- op tracker -- , seq: 747398, time: 2014-06-12 14:45:57.539167, event: started, op: osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.539182 7f9d66bde700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.539187 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.539188 7f9d66bde700  5 queue_op 0x6de04e0 seq 2583 osr(3.39f 0x276d720) 112885 bytes   (queue has 3 ops and 2181855 bytes)
2014-06-12 14:45:57.539197 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] do_sub_op osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.539212 7f9d66bde700 10 osd.2 173 dequeue_op 0x20e89400 finish
2014-06-12 14:45:57.539216 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.539218 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.539227 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (3 active)
2014-06-12 14:45:57.540652 7f9d7d968700 10 _generic_write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 2019328~1548288 = 0
2014-06-12 14:45:57.540855 7f9d7d968700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.540872 7f9d7d968700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.540878 7f9d7d968700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.540885 7f9d7d968700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.542249 7f9d7d968700 20 set_header setting 17144 cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 parent seq 0
2014-06-12 14:45:57.542848 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.542866 7f9d7d968700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:45:57.542878 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:45:57.550482 7f9d57543700  5 -- op tracker -- , seq: 747399, time: 2014-06-12 14:45:57.539669, event: header_read, op: osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.550501 7f9d57543700  5 -- op tracker -- , seq: 747399, time: 2014-06-12 14:45:57.539670, event: throttled, op: osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.550510 7f9d57543700  5 -- op tracker -- , seq: 747399, time: 2014-06-12 14:45:57.550427, event: all_read, op: osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.550519 7f9d57543700  5 -- op tracker -- , seq: 747399, time: 0.000000, event: dispatched, op: osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.550525 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.550533 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:57.550543 7f9d57543700 15 osd.2 173 enqueue_op 0x39576c00 prio 127 cost 1032999 latency 0.010874 osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.550612 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.550620 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x39576c00 prio 127 cost 1032999 latency 0.010951 osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2d7( v 173'2992 (0'0,173'2992] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2991 active]
2014-06-12 14:45:57.550646 7f9d6a3e5700  5 -- op tracker -- , seq: 747399, time: 2014-06-12 14:45:57.550641, event: reached_pg, op: osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.550656 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2992 (0'0,173'2992] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2991 active] handle_message: 0x39576c00
2014-06-12 14:45:57.550674 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2992 (0'0,173'2992] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2991 active] sub_op_modify trans 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 v 173'2993 (transaction) 174
2014-06-12 14:45:57.550693 7f9d6a3e5700  5 -- op tracker -- , seq: 747399, time: 2014-06-12 14:45:57.550689, event: started, op: osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.550712 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2992 (0'0,173'2992] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2991 active] append_log log((0'0,173'2992], crt=0'0) [173'2993 (173'2992) modify   88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 by client.4301.0:145394 2014-06-12 08:44:59.902903]
2014-06-12 14:45:57.550739 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2993 (0'0,173'2993] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2991 active] add_log_entry 173'2993 (173'2992) modify   88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 by client.4301.0:145394 2014-06-12 08:44:59.902903
2014-06-12 14:45:57.550763 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2993 (0'0,173'2993] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2991 active] append_log  adding 1 keys
2014-06-12 14:45:57.550800 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2993, trimmed: 
2014-06-12 14:45:57.550826 7f9d6a3e5700  5 -- op tracker -- , seq: 747399, time: 2014-06-12 14:45:57.550821, event: started, op: osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.550841 7f9d6a3e5700  5 queue_transactions existing osr(3.2d7 0x276d5c0)/0x276d5c0
2014-06-12 14:45:57.550846 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.550848 7f9d6a3e5700  5 queue_op 0x3b19440 seq 2706 osr(3.2d7 0x276d5c0) 1034485 bytes   (queue has 4 ops and 3216340 bytes)
2014-06-12 14:45:57.550856 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2993 (0'0,173'2993] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2991 active] do_sub_op osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.550871 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x39576c00 finish
2014-06-12 14:45:57.550875 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.550878 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.550887 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cbe60 (4 active)
2014-06-12 14:45:57.550898 7f9d8096e700  5 _do_op 0x3b19440 seq 2706 osr(3.2d7 0x276d5c0)/0x276d5c0 start
2014-06-12 14:45:57.550903 7f9d8096e700 10 _do_transaction on 0x678feb0
2014-06-12 14:45:57.550910 7f9d8096e700 15 _omap_setkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.551098 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.551138 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.555345 7f9d7d968700 10 _do_op 0x6c4f320 seq 2581 r = 0, finisher 0x7ffa840 0
2014-06-12 14:45:57.555371 7f9d7d968700 10 _finish_op 0x6c4f320 seq 2581 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.555382 7f9d8016d700  5 _do_op 0x142a9260 seq 2582 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:45:57.555388 7f9d8016d700 10 _do_transaction on 0x5ea62b0
2014-06-12 14:45:57.555399 7f9d8016d700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.555531 7f9d7715b700  5 -- op tracker -- , seq: 747396, time: 2014-06-12 14:45:57.555519, event: commit_sent, op: osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.555545 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2824 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:45:57.555576 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:57.555619 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.555643 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.555730 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.555757 7f9d8016d700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.555763 7f9d8016d700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:45:57.555770 7f9d8016d700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.555896 7f9d8016d700 15 _write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 3567616~516096
2014-06-12 14:45:57.556094 7f9d8016d700 10 lookup_strip_header done  cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.556227 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:45:57.557563 7f9d8016d700 10 _generic_write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 3567616~516096 = 0
2014-06-12 14:45:57.557632 7f9d8016d700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.557647 7f9d8016d700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.557653 7f9d8016d700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.557650 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (3 active)
2014-06-12 14:45:57.557656 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.557661 7f9d8016d700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.557818 7f9d8016d700 20 set_header setting 17144 cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 parent seq 0
2014-06-12 14:45:57.557880 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.557897 7f9d8016d700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:45:57.557912 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:45:57.558933 7f9d56432700  5 -- op tracker -- , seq: 747400, time: 2014-06-12 14:45:57.549771, event: header_read, op: osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.558947 7f9d56432700  5 -- op tracker -- , seq: 747400, time: 2014-06-12 14:45:57.549773, event: throttled, op: osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.558957 7f9d56432700  5 -- op tracker -- , seq: 747400, time: 2014-06-12 14:45:57.558897, event: all_read, op: osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.558970 7f9d56432700  5 -- op tracker -- , seq: 747400, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.558976 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.558983 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:57.558991 7f9d56432700 15 osd.2 173 enqueue_op 0xe20bd00 prio 127 cost 2065191 latency 0.009220 osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.559023 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.559033 7f9d63bd8700 10 osd.2 173 dequeue_op 0xe20bd00 prio 127 cost 2065191 latency 0.009261 osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.51e( v 173'2676 (0'0,173'2676] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2675 active]
2014-06-12 14:45:57.559058 7f9d63bd8700  5 -- op tracker -- , seq: 747400, time: 2014-06-12 14:45:57.559053, event: reached_pg, op: osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.559069 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2676 (0'0,173'2676] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2675 active] handle_message: 0xe20bd00
2014-06-12 14:45:57.559085 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2676 (0'0,173'2676] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2675 active] sub_op_modify trans 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 v 173'2677 (transaction) 174
2014-06-12 14:45:57.559100 7f9d63bd8700  5 -- op tracker -- , seq: 747400, time: 2014-06-12 14:45:57.559097, event: started, op: osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.559117 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2676 (0'0,173'2676] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2675 active] append_log log((0'0,173'2676], crt=0'0) [173'2677 (173'2676) modify   55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 by client.4294.0:143953 2014-06-12 08:44:56.020124]
2014-06-12 14:45:57.559143 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2677 (0'0,173'2677] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2675 active] add_log_entry 173'2677 (173'2676) modify   55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 by client.4294.0:143953 2014-06-12 08:44:56.020124
2014-06-12 14:45:57.559166 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2677 (0'0,173'2677] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2675 active] append_log  adding 1 keys
2014-06-12 14:45:57.559200 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2677, trimmed: 
2014-06-12 14:45:57.559224 7f9d63bd8700  5 -- op tracker -- , seq: 747400, time: 2014-06-12 14:45:57.559220, event: started, op: osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.559240 7f9d63bd8700  5 queue_transactions existing osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:45:57.559244 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.559246 7f9d63bd8700  5 queue_op 0x9182120 seq 2513 osr(3.51e 0x276db50) 2066677 bytes   (queue has 4 ops and 3732436 bytes)
2014-06-12 14:45:57.559258 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fb00 (4 active)
2014-06-12 14:45:57.559254 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.51e( v 173'2677 (0'0,173'2677] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2675 active] do_sub_op osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.559266 7f9d7915f700  5 _do_op 0x9182120 seq 2513 osr(3.51e 0x276db50)/0x276db50 start
2014-06-12 14:45:57.559268 7f9d63bd8700 10 osd.2 173 dequeue_op 0xe20bd00 finish
2014-06-12 14:45:57.559271 7f9d7915f700 10 _do_transaction on 0x7f875f0
2014-06-12 14:45:57.559272 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.559275 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.559277 7f9d7915f700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:45:57.559557 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:45:57.559583 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.559686 7f9d7795c700  5 -- op tracker -- , seq: 747396, time: 2014-06-12 14:45:57.559678, event: sub_op_applied, op: osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.559694 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2825 active] sub_op_modify_applied on 0x61e6700 op osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.559712 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2825 active] op_applied on primary on version 173'2826
2014-06-12 14:45:57.559723 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2825 active] op_applied on replica on version 173'2826
2014-06-12 14:45:57.559751 7f9d7795c700  5 -- op tracker -- , seq: 747396, time: 2014-06-12 14:45:57.559746, event: done, op: osd_sub_op(client.4302.0:146007 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2826 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.561410 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.561447 7f9d8096e700 15 _omap_rmkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.561449 7f9d8016d700 10 _do_op 0x142a9260 seq 2582 r = 0, finisher 0x70f52a0 0
2014-06-12 14:45:57.561457 7f9d8096e700 10 _omap_rmkeys meta/1fe0df15/pglog_3.2d7/0//-1 r = 0
2014-06-12 14:45:57.561463 7f9d8016d700 10 _finish_op 0x142a9260 seq 2582 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.561465 7f9d8096e700 15 _omap_setkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.561473 7f9d83173700  5 _do_op 0x6de04e0 seq 2583 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:45:57.561479 7f9d83173700 10 _do_transaction on 0x668cb70
2014-06-12 14:45:57.561473 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (3 active)
2014-06-12 14:45:57.561481 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.561483 7f9d8096e700 15 _write 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 2351104~1032192
2014-06-12 14:45:57.561486 7f9d83173700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.561489 7f9d7795c700  5 -- op tracker -- , seq: 747397, time: 2014-06-12 14:45:57.561480, event: sub_op_applied, op: osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.561496 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2825 active] sub_op_modify_applied on 0x5ea61c0 op osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.561529 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:57.561549 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2825 active] op_applied on primary on version 173'2827
2014-06-12 14:45:57.561562 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2825 active] op_applied on replica on version 173'2827
2014-06-12 14:45:57.561696 7f9d8096e700 10 lookup_strip_header done  cid 3.2d7_head oid 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.561722 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:45:57.561805 7f9d7715b700  5 -- op tracker -- , seq: 747397, time: 2014-06-12 14:45:57.561798, event: commit_sent, op: osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.561816 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2825 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:45:57.561834 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:57.561862 7f9d7715b700  5 -- op tracker -- , seq: 747397, time: 2014-06-12 14:45:57.561857, event: done, op: osd_sub_op(client.4302.0:146008 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2827 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.565907 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.565936 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.566510 7f9d8096e700 10 _generic_write 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 2351104~1032192 = 0
2014-06-12 14:45:57.566659 7f9d8096e700 15 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.566676 7f9d8096e700 10 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 = 0
2014-06-12 14:45:57.566681 7f9d8096e700 15 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.566689 7f9d8096e700 10 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 = 0
2014-06-12 14:45:57.567040 7f9d8096e700 20 set_header setting 17355 cid 3.2d7_head oid 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 parent seq 0
2014-06-12 14:45:57.567188 7f9d8096e700 20 set_header setting 2282 cid meta oid 1fe0df15/pglog_3.2d7/0//-1 parent seq 0
2014-06-12 14:45:57.567204 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.567211 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:45:57.573120 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.573154 7f9d7915f700 15 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:45:57.573161 7f9d7915f700 10 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1 r = 0
2014-06-12 14:45:57.573166 7f9d7915f700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:45:57.573179 7f9d7915f700 15 _write 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 1724416~2064384
2014-06-12 14:45:57.573493 7f9d8096e700 10 _do_op 0x3b19440 seq 2706 r = 0, finisher 0xa646aa0 0
2014-06-12 14:45:57.573507 7f9d7915f700 10 lookup_strip_header done  cid 3.51e_head oid 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3
2014-06-12 14:45:57.573530 7f9d7715b700  5 -- op tracker -- , seq: 747399, time: 2014-06-12 14:45:57.573520, event: commit_sent, op: osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.573539 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2993 (0'0,173'2993] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2991 active] sub_op_modify_commit on op osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:45:57.573569 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:57.573622 7f9d8096e700 10 _finish_op 0x3b19440 seq 2706 osr(3.2d7 0x276d5c0)/0x276d5c0
2014-06-12 14:45:57.573642 7f9d7795c700  5 -- op tracker -- , seq: 747399, time: 2014-06-12 14:45:57.573637, event: sub_op_applied, op: osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.573659 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:45:57.573650 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2993 (0'0,173'2993] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active] sub_op_modify_applied on 0x678fdc0 op osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.573664 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2993 (0'0,173'2993] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active] op_applied on primary on version 173'2993
2014-06-12 14:45:57.573672 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2993 (0'0,173'2993] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active] op_applied on replica on version 173'2993
2014-06-12 14:45:57.573704 7f9d8096e700  5 -- op tracker -- , seq: 747399, time: 2014-06-12 14:45:57.573695, event: done, op: osd_sub_op(client.4301.0:145394 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.573781 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cbe60 (2 active)
2014-06-12 14:45:57.573785 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.581375 7f9d7915f700 10 _generic_write 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 1724416~2064384 = 0
2014-06-12 14:45:57.581654 7f9d7915f700 15 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3
2014-06-12 14:45:57.581679 7f9d7915f700 10 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 = 0
2014-06-12 14:45:57.581684 7f9d7915f700 15 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3
2014-06-12 14:45:57.581694 7f9d7915f700 10 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 = 0
2014-06-12 14:45:57.582079 7f9d57543700  5 -- op tracker -- , seq: 747401, time: 2014-06-12 14:45:57.580361, event: header_read, op: osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.582093 7f9d57543700  5 -- op tracker -- , seq: 747401, time: 2014-06-12 14:45:57.580363, event: throttled, op: osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.582103 7f9d57543700  5 -- op tracker -- , seq: 747401, time: 2014-06-12 14:45:57.582043, event: all_read, op: osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.582112 7f9d57543700  5 -- op tracker -- , seq: 747401, time: 0.000000, event: dispatched, op: osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.582117 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.582124 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:57.582133 7f9d57543700 15 osd.2 173 enqueue_op 0x34017500 prio 127 cost 520999 latency 0.001772 osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.582166 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.582175 7f9d653db700 10 osd.2 173 dequeue_op 0x34017500 prio 127 cost 520999 latency 0.001814 osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2d7( v 173'2993 (0'0,173'2993] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active]
2014-06-12 14:45:57.582201 7f9d653db700  5 -- op tracker -- , seq: 747401, time: 2014-06-12 14:45:57.582197, event: reached_pg, op: osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.582211 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2993 (0'0,173'2993] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active] handle_message: 0x34017500
2014-06-12 14:45:57.582224 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2993 (0'0,173'2993] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active] sub_op_modify trans 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 v 173'2994 (transaction) 174
2014-06-12 14:45:57.582240 7f9d653db700  5 -- op tracker -- , seq: 747401, time: 2014-06-12 14:45:57.582236, event: started, op: osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.582256 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2993 (0'0,173'2993] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active] append_log log((0'0,173'2993], crt=0'0) [173'2994 (173'2993) modify   88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 by client.4301.0:145395 2014-06-12 08:44:59.957208]
2014-06-12 14:45:57.582276 7f9d7915f700 20 set_header setting 17621 cid 3.51e_head oid 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 parent seq 0
2014-06-12 14:45:57.582281 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2994 (0'0,173'2994] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active] add_log_entry 173'2994 (173'2993) modify   88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 by client.4301.0:145395 2014-06-12 08:44:59.957208
2014-06-12 14:45:57.582303 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2994 (0'0,173'2994] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active] append_log  adding 1 keys
2014-06-12 14:45:57.582337 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2994, trimmed: 
2014-06-12 14:45:57.582366 7f9d653db700  5 -- op tracker -- , seq: 747401, time: 2014-06-12 14:45:57.582359, event: started, op: osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.582379 7f9d653db700  5 queue_transactions existing osr(3.2d7 0x276d5c0)/0x276d5c0
2014-06-12 14:45:57.582387 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.582390 7f9d653db700  5 queue_op 0x8cabbc0 seq 2707 osr(3.2d7 0x276d5c0) 522485 bytes   (queue has 3 ops and 2702047 bytes)
2014-06-12 14:45:57.582403 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2994 (0'0,173'2994] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active] do_sub_op osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.582419 7f9d653db700 10 osd.2 173 dequeue_op 0x34017500 finish
2014-06-12 14:45:57.582422 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.582434 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.582527 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cbe60 (3 active)
2014-06-12 14:45:57.582535 7f9d7c165700  5 _do_op 0x8cabbc0 seq 2707 osr(3.2d7 0x276d5c0)/0x276d5c0 start
2014-06-12 14:45:57.582539 7f9d7c165700 10 _do_transaction on 0x5f4bcf0
2014-06-12 14:45:57.582545 7f9d7c165700 15 _omap_setkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.582559 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.582574 7f9d7915f700 20 set_header setting 2288 cid meta oid 1fe1612b/pglog_3.51e/0//-1 parent seq 0
2014-06-12 14:45:57.582583 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:45:57.582713 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.582737 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.588890 7f9d5703e700  5 -- op tracker -- , seq: 747402, time: 2014-06-12 14:45:57.585665, event: header_read, op: osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.588903 7f9d5703e700  5 -- op tracker -- , seq: 747402, time: 2014-06-12 14:45:57.585666, event: throttled, op: osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.588912 7f9d5703e700  5 -- op tracker -- , seq: 747402, time: 2014-06-12 14:45:57.588842, event: all_read, op: osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.588922 7f9d5703e700  5 -- op tracker -- , seq: 747402, time: 0.000000, event: dispatched, op: osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.588927 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.588934 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:57.588942 7f9d5703e700 15 osd.2 173 enqueue_op 0x20e88200 prio 127 cost 2065191 latency 0.003277 osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.588973 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.588982 7f9d69be4700 10 osd.2 173 dequeue_op 0x20e88200 prio 127 cost 2065191 latency 0.003317 osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.237( v 173'2840 (0'0,173'2840] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active]
2014-06-12 14:45:57.589006 7f9d69be4700  5 -- op tracker -- , seq: 747402, time: 2014-06-12 14:45:57.589002, event: reached_pg, op: osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.589016 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2840 (0'0,173'2840] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] handle_message: 0x20e88200
2014-06-12 14:45:57.589028 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2840 (0'0,173'2840] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] sub_op_modify trans 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 v 173'2841 (transaction) 174
2014-06-12 14:45:57.589046 7f9d69be4700  5 -- op tracker -- , seq: 747402, time: 2014-06-12 14:45:57.589041, event: started, op: osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.589063 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2840 (0'0,173'2840] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] append_log log((0'0,173'2840], crt=0'0) [173'2841 (173'2840) modify   64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 by client.4295.0:143270 2014-06-12 08:44:59.948703]
2014-06-12 14:45:57.589088 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2841 (0'0,173'2841] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] add_log_entry 173'2841 (173'2840) modify   64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 by client.4295.0:143270 2014-06-12 08:44:59.948703
2014-06-12 14:45:57.589116 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2841 (0'0,173'2841] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] append_log  adding 1 keys
2014-06-12 14:45:57.589153 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2841, trimmed: 
2014-06-12 14:45:57.589180 7f9d69be4700  5 -- op tracker -- , seq: 747402, time: 2014-06-12 14:45:57.589175, event: started, op: osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.589193 7f9d69be4700  5 queue_transactions existing osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:45:57.589197 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.589199 7f9d69be4700  5 queue_op 0x5c74540 seq 2598 osr(3.237 0x276d3f0) 2066677 bytes   (queue has 4 ops and 4768724 bytes)
2014-06-12 14:45:57.589206 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.237( v 173'2841 (0'0,173'2841] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] do_sub_op osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.589221 7f9d69be4700 10 osd.2 173 dequeue_op 0x20e88200 finish
2014-06-12 14:45:57.589225 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.589227 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.589236 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540eb40 (4 active)
2014-06-12 14:45:57.589242 7f9d82972700  5 _do_op 0x5c74540 seq 2598 osr(3.237 0x276d3f0)/0x276d3f0 start
2014-06-12 14:45:57.589246 7f9d82972700 10 _do_transaction on 0x5ba80f0
2014-06-12 14:45:57.589253 7f9d82972700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:57.589418 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:57.589441 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.594534 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.594581 7f9d7c165700 15 _omap_rmkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.594591 7f9d7c165700 10 _omap_rmkeys meta/1fe0df15/pglog_3.2d7/0//-1 r = 0
2014-06-12 14:45:57.594599 7f9d7c165700 15 _omap_setkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.594615 7f9d7c165700 15 _write 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 3383296~520192
2014-06-12 14:45:57.594766 7f9d7c165700 10 lookup_strip_header done  cid 3.2d7_head oid 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.594870 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:45:57.595829 7f9d7915f700 10 _do_op 0x9182120 seq 2513 r = 0, finisher 0x7bd4560 0
2014-06-12 14:45:57.595903 7f9d7715b700  5 -- op tracker -- , seq: 747400, time: 2014-06-12 14:45:57.595892, event: commit_sent, op: osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.595913 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2677 (0'0,173'2677] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2675 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:57.595943 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:57.596027 7f9d7915f700 10 _finish_op 0x9182120 seq 2513 osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:45:57.596066 7f9d7795c700  5 -- op tracker -- , seq: 747400, time: 2014-06-12 14:45:57.596060, event: sub_op_applied, op: osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.596077 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2677 (0'0,173'2677] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2676 active] sub_op_modify_applied on 0x7f87500 op osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.596092 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2677 (0'0,173'2677] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2676 active] op_applied on primary on version 173'2677
2014-06-12 14:45:57.596101 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2677 (0'0,173'2677] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2676 active] op_applied on replica on version 173'2677
2014-06-12 14:45:57.596182 7f9d7915f700  5 -- op tracker -- , seq: 747400, time: 2014-06-12 14:45:57.596172, event: done, op: osd_sub_op(client.4294.0:143953 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.596232 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fb00 (3 active)
2014-06-12 14:45:57.596255 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.596888 7f9d7c165700 10 _generic_write 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 3383296~520192 = 0
2014-06-12 14:45:57.596956 7f9d7c165700 15 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.596977 7f9d7c165700 10 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 = 0
2014-06-12 14:45:57.596982 7f9d7c165700 15 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.596990 7f9d7c165700 10 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 = 0
2014-06-12 14:45:57.597218 7f9d7c165700 20 set_header setting 17355 cid 3.2d7_head oid 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 parent seq 0
2014-06-12 14:45:57.597311 7f9d7c165700 20 set_header setting 2282 cid meta oid 1fe0df15/pglog_3.2d7/0//-1 parent seq 0
2014-06-12 14:45:57.597332 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.597343 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:45:57.610699 7f9d5703e700  5 -- op tracker -- , seq: 747403, time: 2014-06-12 14:45:57.607783, event: header_read, op: osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.610714 7f9d5703e700  5 -- op tracker -- , seq: 747403, time: 2014-06-12 14:45:57.607786, event: throttled, op: osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.610723 7f9d5703e700  5 -- op tracker -- , seq: 747403, time: 2014-06-12 14:45:57.610654, event: all_read, op: osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.610731 7f9d5703e700  5 -- op tracker -- , seq: 747403, time: 0.000000, event: dispatched, op: osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.610738 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.610745 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:57.610754 7f9d5703e700 15 osd.2 173 enqueue_op 0x20e89700 prio 127 cost 1032999 latency 0.002970 osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.610864 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.610874 7f9d633d7700 10 osd.2 173 dequeue_op 0x20e89700 prio 127 cost 1032999 latency 0.003090 osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.237( v 173'2841 (0'0,173'2841] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active]
2014-06-12 14:45:57.610897 7f9d633d7700  5 -- op tracker -- , seq: 747403, time: 2014-06-12 14:45:57.610893, event: reached_pg, op: osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.610906 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2841 (0'0,173'2841] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] handle_message: 0x20e89700
2014-06-12 14:45:57.610920 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2841 (0'0,173'2841] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] sub_op_modify trans 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 v 173'2842 (transaction) 174
2014-06-12 14:45:57.610936 7f9d633d7700  5 -- op tracker -- , seq: 747403, time: 2014-06-12 14:45:57.610932, event: started, op: osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.610954 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2841 (0'0,173'2841] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] append_log log((0'0,173'2841], crt=0'0) [173'2842 (173'2841) modify   64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 by client.4295.0:143271 2014-06-12 08:44:59.979735]
2014-06-12 14:45:57.610980 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2842 (0'0,173'2842] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] add_log_entry 173'2842 (173'2841) modify   64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 by client.4295.0:143271 2014-06-12 08:44:59.979735
2014-06-12 14:45:57.611001 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2842 (0'0,173'2842] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] append_log  adding 1 keys
2014-06-12 14:45:57.611036 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2842, trimmed: 
2014-06-12 14:45:57.611062 7f9d633d7700  5 -- op tracker -- , seq: 747403, time: 2014-06-12 14:45:57.611057, event: started, op: osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.611073 7f9d633d7700  5 queue_transactions existing osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:45:57.611077 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.611079 7f9d633d7700  5 queue_op 0x6f81b00 seq 2599 osr(3.237 0x276d3f0) 1034485 bytes   (queue has 4 ops and 3736532 bytes)
2014-06-12 14:45:57.611085 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.237( v 173'2842 (0'0,173'2842] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] do_sub_op osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.611102 7f9d633d7700 10 osd.2 173 dequeue_op 0x20e89700 finish
2014-06-12 14:45:57.611106 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.611108 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.611154 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540eb40 (4 active)
2014-06-12 14:45:57.641468 7f9d56432700  5 -- op tracker -- , seq: 747404, time: 2014-06-12 14:45:57.640132, event: header_read, op: osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.641483 7f9d56432700  5 -- op tracker -- , seq: 747404, time: 2014-06-12 14:45:57.640134, event: throttled, op: osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.641493 7f9d56432700  5 -- op tracker -- , seq: 747404, time: 2014-06-12 14:45:57.641428, event: all_read, op: osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.641502 7f9d56432700  5 -- op tracker -- , seq: 747404, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.641507 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.641513 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:45:57.641521 7f9d56432700 15 osd.2 173 enqueue_op 0x5ee9200 prio 127 cost 254759 latency 0.001389 osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.641547 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.641555 7f9d623d5700 10 osd.2 173 dequeue_op 0x5ee9200 prio 127 cost 254759 latency 0.001423 osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2984 (0'0,173'2984] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2983 active]
2014-06-12 14:45:57.641573 7f9d623d5700  5 -- op tracker -- , seq: 747404, time: 2014-06-12 14:45:57.641570, event: reached_pg, op: osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.641581 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2984 (0'0,173'2984] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2983 active] handle_message: 0x5ee9200
2014-06-12 14:45:57.641588 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2984 (0'0,173'2984] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2983 active] sub_op_modify trans 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 v 173'2985 (transaction) 174
2014-06-12 14:45:57.641599 7f9d623d5700  5 -- op tracker -- , seq: 747404, time: 2014-06-12 14:45:57.641597, event: started, op: osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.641611 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2984 (0'0,173'2984] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2983 active] append_log log((0'0,173'2984], crt=0'0) [173'2985 (173'733) modify   3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 by client.4298.0:145087 2014-06-12 08:43:25.619184]
2014-06-12 14:45:57.641652 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2985 (0'0,173'2985] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2983 active] add_log_entry 173'2985 (173'733) modify   3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 by client.4298.0:145087 2014-06-12 08:43:25.619184
2014-06-12 14:45:57.641670 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2985 (0'0,173'2985] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2983 active] append_log  adding 1 keys
2014-06-12 14:45:57.641694 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2985, trimmed: 
2014-06-12 14:45:57.641715 7f9d623d5700  5 -- op tracker -- , seq: 747404, time: 2014-06-12 14:45:57.641711, event: started, op: osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.641722 7f9d623d5700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:45:57.641726 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.641727 7f9d623d5700  5 queue_op 0xbdfd5c0 seq 2787 osr(3.458 0x276d960) 256245 bytes   (queue has 5 ops and 3992777 bytes)
2014-06-12 14:45:57.641733 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2985 (0'0,173'2985] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2983 active] do_sub_op osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.641742 7f9d623d5700 10 osd.2 173 dequeue_op 0x5ee9200 finish
2014-06-12 14:45:57.641745 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.641746 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.641983 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (5 active)
2014-06-12 14:45:57.641993 7f9d81970700  5 _do_op 0xbdfd5c0 seq 2787 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:45:57.641998 7f9d81970700 10 _do_transaction on 0x1ebed430
2014-06-12 14:45:57.642012 7f9d81970700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:45:57.654240 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:45:57.654284 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.656410 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.656448 7f9d83173700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.656455 7f9d83173700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:45:57.656462 7f9d83173700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:45:57.656465 7f9d7c165700 10 _do_op 0x8cabbc0 seq 2707 r = 0, finisher 0x1d49bfa0 0
2014-06-12 14:45:57.656477 7f9d7c165700 10 _finish_op 0x8cabbc0 seq 2707 osr(3.2d7 0x276d5c0)/0x276d5c0
2014-06-12 14:45:57.656505 7f9d7795c700  5 -- op tracker -- , seq: 747401, time: 2014-06-12 14:45:57.656495, event: sub_op_applied, op: osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.656523 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2994 (0'0,173'2994] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active] sub_op_modify_applied on 0x5f4bc00 op osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.656554 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:57.656575 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2994 (0'0,173'2994] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active] op_applied on primary on version 173'2994
2014-06-12 14:45:57.656585 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2994 (0'0,173'2994] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active] op_applied on replica on version 173'2994
2014-06-12 14:45:57.656617 7f9d7715b700  5 -- op tracker -- , seq: 747401, time: 2014-06-12 14:45:57.656610, event: commit_sent, op: osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.656626 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2994 (0'0,173'2994] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2992 active] sub_op_modify_commit on op osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:45:57.656645 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:57.656639 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cbe60 (4 active)
2014-06-12 14:45:57.656652 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.656680 7f9d7715b700  5 -- op tracker -- , seq: 747401, time: 2014-06-12 14:45:57.656675, event: done, op: osd_sub_op(client.4301.0:145395 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.656685 7f9d83173700 15 _write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 4083712~110592
2014-06-12 14:45:57.656872 7f9d83173700 10 lookup_strip_header done  cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.657034 7f9d83173700 10 file_to_extents done 
2014-06-12 14:45:57.657419 7f9d83173700 10 _generic_write 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 4083712~110592 = 0
2014-06-12 14:45:57.657440 7f9d83173700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.657451 7f9d83173700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.657457 7f9d83173700 15 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3
2014-06-12 14:45:57.657466 7f9d83173700 10 _setattrs 3.39f_head/1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 = 0
2014-06-12 14:45:57.657597 7f9d83173700 20 set_header setting 17144 cid 3.39f_head oid 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 parent seq 0
2014-06-12 14:45:57.657708 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.657726 7f9d83173700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:45:57.657739 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:45:57.659433 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.659466 7f9d82972700 15 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:57.659475 7f9d82972700 10 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1 r = 0
2014-06-12 14:45:57.659482 7f9d82972700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:57.659623 7f9d82972700 15 _write 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 233472~2064384
2014-06-12 14:45:57.659813 7f9d82972700 10 lookup_strip_header done  cid 3.237_head oid 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3
2014-06-12 14:45:57.659964 7f9d82972700 10 file_to_extents done 
2014-06-12 14:45:57.661759 7f9d83173700 10 _do_op 0x6de04e0 seq 2583 r = 0, finisher 0x8891f00 0
2014-06-12 14:45:57.661772 7f9d83173700 10 _finish_op 0x6de04e0 seq 2583 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:45:57.661779 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (3 active)
2014-06-12 14:45:57.661782 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.661802 7f9d7795c700  5 -- op tracker -- , seq: 747398, time: 2014-06-12 14:45:57.661794, event: sub_op_applied, op: osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.661819 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2826 active] sub_op_modify_applied on 0x668ca80 op osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.661845 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:57.661862 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2826 active] op_applied on primary on version 173'2828
2014-06-12 14:45:57.661873 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2826 active] op_applied on replica on version 173'2828
2014-06-12 14:45:57.663715 7f9d57543700  5 -- op tracker -- , seq: 747405, time: 2014-06-12 14:45:57.661567, event: header_read, op: osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.663732 7f9d57543700  5 -- op tracker -- , seq: 747405, time: 2014-06-12 14:45:57.661569, event: throttled, op: osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.663741 7f9d57543700  5 -- op tracker -- , seq: 747405, time: 2014-06-12 14:45:57.663667, event: all_read, op: osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.663748 7f9d57543700  5 -- op tracker -- , seq: 747405, time: 0.000000, event: dispatched, op: osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.663765 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.663775 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:45:57.663789 7f9d57543700 15 osd.2 173 enqueue_op 0x20355b00 prio 127 cost 291623 latency 0.002222 osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.664148 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.664159 7f9d643d9700 10 osd.2 173 dequeue_op 0x20355b00 prio 127 cost 291623 latency 0.002591 osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2d7( v 173'2994 (0'0,173'2994] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2993 active]
2014-06-12 14:45:57.664184 7f9d643d9700  5 -- op tracker -- , seq: 747405, time: 2014-06-12 14:45:57.664180, event: reached_pg, op: osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.664193 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2994 (0'0,173'2994] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2993 active] handle_message: 0x20355b00
2014-06-12 14:45:57.664206 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2994 (0'0,173'2994] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2993 active] sub_op_modify trans 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 v 173'2995 (transaction) 174
2014-06-12 14:45:57.664222 7f9d643d9700  5 -- op tracker -- , seq: 747405, time: 2014-06-12 14:45:57.664218, event: started, op: osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.664240 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2994 (0'0,173'2994] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2993 active] append_log log((0'0,173'2994], crt=0'0) [173'2995 (173'2994) modify   88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 by client.4301.0:145396 2014-06-12 08:45:00.040195]
2014-06-12 14:45:57.664264 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2995 (0'0,173'2995] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2993 active] add_log_entry 173'2995 (173'2994) modify   88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 by client.4301.0:145396 2014-06-12 08:45:00.040195
2014-06-12 14:45:57.664285 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2995 (0'0,173'2995] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2993 active] append_log  adding 1 keys
2014-06-12 14:45:57.664318 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2995, trimmed: 
2014-06-12 14:45:57.664348 7f9d643d9700  5 -- op tracker -- , seq: 747405, time: 2014-06-12 14:45:57.664344, event: started, op: osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.664361 7f9d643d9700  5 queue_transactions existing osr(3.2d7 0x276d5c0)/0x276d5c0
2014-06-12 14:45:57.664368 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.664370 7f9d643d9700  5 queue_op 0x34c7aa0 seq 2708 osr(3.2d7 0x276d5c0) 293109 bytes   (queue has 4 ops and 3650516 bytes)
2014-06-12 14:45:57.664381 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2995 (0'0,173'2995] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2993 active] do_sub_op osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.664397 7f9d643d9700 10 osd.2 173 dequeue_op 0x20355b00 finish
2014-06-12 14:45:57.664401 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.664405 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.664707 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cbe60 (4 active)
2014-06-12 14:45:57.664716 7f9d7c966700  5 _do_op 0x34c7aa0 seq 2708 osr(3.2d7 0x276d5c0)/0x276d5c0 start
2014-06-12 14:45:57.664721 7f9d7c966700 10 _do_transaction on 0x191d1270
2014-06-12 14:45:57.664726 7f9d7c966700 15 _omap_setkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.664895 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.664911 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.665652 7f9d7715b700  5 -- op tracker -- , seq: 747398, time: 2014-06-12 14:45:57.665640, event: commit_sent, op: osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.665669 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2826 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:45:57.665697 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:57.665754 7f9d7715b700  5 -- op tracker -- , seq: 747398, time: 2014-06-12 14:45:57.665749, event: done, op: osd_sub_op(client.4302.0:146009 3.39f 1da5839f/rbd_data.10a56b8b4567.0000000000001028/head//3 [] v 173'2828 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.669194 7f9d82972700 10 _generic_write 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 233472~2064384 = 0
2014-06-12 14:45:57.669427 7f9d82972700 15 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3
2014-06-12 14:45:57.669443 7f9d82972700 10 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 = 0
2014-06-12 14:45:57.669448 7f9d82972700 15 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3
2014-06-12 14:45:57.669457 7f9d82972700 10 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 = 0
2014-06-12 14:45:57.669989 7f9d82972700 20 set_header setting 16982 cid 3.237_head oid 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 parent seq 0
2014-06-12 14:45:57.670177 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.670189 7f9d82972700 20 set_header setting 2190 cid meta oid 1fe26b1a/pglog_3.237/0//-1 parent seq 0
2014-06-12 14:45:57.670198 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:45:57.680775 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.680897 7f9d7c966700 15 _omap_rmkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.680921 7f9d7c966700 10 _omap_rmkeys meta/1fe0df15/pglog_3.2d7/0//-1 r = 0
2014-06-12 14:45:57.680945 7f9d7c966700 15 _omap_setkeys meta/1fe0df15/pglog_3.2d7/0//-1
2014-06-12 14:45:57.681195 7f9d7c966700 15 _write 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 3903488~290816
2014-06-12 14:45:57.681423 7f9d7c966700 10 lookup_strip_header done  cid 3.2d7_head oid 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.681653 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:45:57.682503 7f9d82972700 10 _do_op 0x5c74540 seq 2598 r = 0, finisher 0x6d63420 0
2014-06-12 14:45:57.682520 7f9d82972700 10 _finish_op 0x5c74540 seq 2598 osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:45:57.682530 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540eb40 (3 active)
2014-06-12 14:45:57.682536 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.682571 7f9d7795c700  5 -- op tracker -- , seq: 747402, time: 2014-06-12 14:45:57.682563, event: sub_op_applied, op: osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.682583 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2842 (0'0,173'2842] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] sub_op_modify_applied on 0x5ba8000 op osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.682610 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:57.682628 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2842 (0'0,173'2842] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] op_applied on primary on version 173'2841
2014-06-12 14:45:57.682641 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2842 (0'0,173'2842] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] op_applied on replica on version 173'2841
2014-06-12 14:45:57.682665 7f9d7715b700  5 -- op tracker -- , seq: 747402, time: 2014-06-12 14:45:57.682658, event: commit_sent, op: osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.682673 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2842 (0'0,173'2842] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2839 active] sub_op_modify_commit on op osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:45:57.682701 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:57.682728 7f9d7a161700  5 _do_op 0x6f81b00 seq 2599 osr(3.237 0x276d3f0)/0x276d3f0 start
2014-06-12 14:45:57.682730 7f9d7715b700  5 -- op tracker -- , seq: 747402, time: 2014-06-12 14:45:57.682725, event: done, op: osd_sub_op(client.4295.0:143270 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2841 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.682736 7f9d7a161700 10 _do_transaction on 0x63ad0b0
2014-06-12 14:45:57.682742 7f9d7a161700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:57.682979 7f9d7c966700 10 _generic_write 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 3903488~290816 = 0
2014-06-12 14:45:57.683005 7f9d7c966700 15 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.683019 7f9d7c966700 10 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 = 0
2014-06-12 14:45:57.683024 7f9d7c966700 15 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3
2014-06-12 14:45:57.683029 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:57.683032 7f9d7c966700 10 _setattrs 3.2d7_head/88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 = 0
2014-06-12 14:45:57.683053 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.683184 7f9d7c966700 20 set_header setting 17355 cid 3.2d7_head oid 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 parent seq 0
2014-06-12 14:45:57.683228 7f9d7c966700 20 set_header setting 2282 cid meta oid 1fe0df15/pglog_3.2d7/0//-1 parent seq 0
2014-06-12 14:45:57.683245 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.683256 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:45:57.685741 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.685946 7f9d81970700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:45:57.685951 7f9d7c966700 10 _do_op 0x34c7aa0 seq 2708 r = 0, finisher 0x50ce0a0 0
2014-06-12 14:45:57.685955 7f9d81970700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:45:57.685962 7f9d81970700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:45:57.685983 7f9d81970700 15 _write 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 0~253952
2014-06-12 14:45:57.685983 7f9d7715b700  5 -- op tracker -- , seq: 747405, time: 2014-06-12 14:45:57.685973, event: commit_sent, op: osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.685993 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2995 (0'0,173'2995] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2993 active] sub_op_modify_commit on op osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:45:57.686023 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:45:57.686050 7f9d7c966700 10 _finish_op 0x34c7aa0 seq 2708 osr(3.2d7 0x276d5c0)/0x276d5c0
2014-06-12 14:45:57.686059 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cbe60 (2 active)
2014-06-12 14:45:57.686062 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.686085 7f9d7795c700  5 -- op tracker -- , seq: 747405, time: 2014-06-12 14:45:57.686078, event: sub_op_applied, op: osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.686093 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2995 (0'0,173'2995] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2994 active] sub_op_modify_applied on 0x191d1180 op osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.686110 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2995 (0'0,173'2995] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2994 active] op_applied on primary on version 173'2995
2014-06-12 14:45:57.686120 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2d7( v 173'2995 (0'0,173'2995] local-les=145 n=305 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2994 active] op_applied on replica on version 173'2995
2014-06-12 14:45:57.686143 7f9d7795c700  5 -- op tracker -- , seq: 747405, time: 2014-06-12 14:45:57.686139, event: done, op: osd_sub_op(client.4301.0:145396 3.2d7 88018ad7/rbd_data.10a46b8b4567.00000000000010a2/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.686150 7f9d81970700 10 lookup_strip_header done  cid 3.458_head oid 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:45:57.686320 7f9d81970700 10 file_to_extents done 
2014-06-12 14:45:57.686710 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:57.686896 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:57.687212 7f9d81970700 10 _generic_write 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 0~253952 = 0
2014-06-12 14:45:57.687237 7f9d81970700 15 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:45:57.687250 7f9d81970700 10 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 = 0
2014-06-12 14:45:57.687255 7f9d81970700 15 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:45:57.687259 7f9d81970700 10 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 = 0
2014-06-12 14:45:57.687459 7f9d81970700 20 set_header setting 16953 cid 3.458_head oid 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 parent seq 0
2014-06-12 14:45:57.687540 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.687558 7f9d81970700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:45:57.687571 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:45:57.689679 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.689683 7f9d81970700 10 _do_op 0xbdfd5c0 seq 2787 r = 0, finisher 0x50cf6a0 0
2014-06-12 14:45:57.689697 7f9d81970700 10 _finish_op 0xbdfd5c0 seq 2787 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:45:57.689707 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (1 active)
2014-06-12 14:45:57.689711 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.689721 7f9d7a161700 15 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:57.689727 7f9d7a161700 10 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1 r = 0
2014-06-12 14:45:57.689733 7f9d7a161700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:57.689729 7f9d7715b700  5 -- op tracker -- , seq: 747404, time: 2014-06-12 14:45:57.689721, event: commit_sent, op: osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.689738 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2985 (0'0,173'2985] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2983 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:45:57.689766 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:57.689811 7f9d7795c700  5 -- op tracker -- , seq: 747404, time: 2014-06-12 14:45:57.689804, event: sub_op_applied, op: osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.689821 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2985 (0'0,173'2985] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active] sub_op_modify_applied on 0x1ebed340 op osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.689843 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2985 (0'0,173'2985] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active] op_applied on primary on version 173'2985
2014-06-12 14:45:57.689853 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2985 (0'0,173'2985] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active] op_applied on replica on version 173'2985
2014-06-12 14:45:57.689881 7f9d7795c700  5 -- op tracker -- , seq: 747404, time: 2014-06-12 14:45:57.689877, event: done, op: osd_sub_op(client.4298.0:145087 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.689956 7f9d7a161700 15 _write 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 2297856~1032192
2014-06-12 14:45:57.690074 7f9d7a161700 10 lookup_strip_header done  cid 3.237_head oid 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3
2014-06-12 14:45:57.690245 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:45:57.693403 7f9d7a161700 10 _generic_write 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 2297856~1032192 = 0
2014-06-12 14:45:57.693481 7f9d7a161700 15 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3
2014-06-12 14:45:57.693496 7f9d7a161700 10 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 = 0
2014-06-12 14:45:57.693501 7f9d7a161700 15 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3
2014-06-12 14:45:57.693506 7f9d7a161700 10 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 = 0
2014-06-12 14:45:57.693647 7f9d5703e700  5 -- op tracker -- , seq: 747406, time: 2014-06-12 14:45:57.691535, event: header_read, op: osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.693660 7f9d5703e700  5 -- op tracker -- , seq: 747406, time: 2014-06-12 14:45:57.691544, event: throttled, op: osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.693670 7f9d5703e700  5 -- op tracker -- , seq: 747406, time: 2014-06-12 14:45:57.692859, event: all_read, op: osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.693682 7f9d5703e700  5 -- op tracker -- , seq: 747406, time: 0.000000, event: dispatched, op: osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.693689 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.693696 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:45:57.693706 7f9d5703e700 15 osd.2 173 enqueue_op 0x1e3eac00 prio 127 cost 865063 latency 0.002171 osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.693737 7f9d7a161700 20 set_header setting 16982 cid 3.237_head oid 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 parent seq 0
2014-06-12 14:45:57.693740 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.693747 7f9d68be2700 10 osd.2 173 dequeue_op 0x1e3eac00 prio 127 cost 865063 latency 0.002212 osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.237( v 173'2842 (0'0,173'2842] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2840 active]
2014-06-12 14:45:57.693775 7f9d68be2700  5 -- op tracker -- , seq: 747406, time: 2014-06-12 14:45:57.693770, event: reached_pg, op: osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.693786 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2842 (0'0,173'2842] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2840 active] handle_message: 0x1e3eac00
2014-06-12 14:45:57.693800 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2842 (0'0,173'2842] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2840 active] sub_op_modify trans 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 v 173'2843 (transaction) 174
2014-06-12 14:45:57.693817 7f9d68be2700  5 -- op tracker -- , seq: 747406, time: 2014-06-12 14:45:57.693813, event: started, op: osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.693839 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.693833 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2842 (0'0,173'2842] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2840 active] append_log log((0'0,173'2842], crt=0'0) [173'2843 (173'2842) modify   64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 by client.4295.0:143272 2014-06-12 08:45:00.067202]
2014-06-12 14:45:57.693856 7f9d7a161700 20 set_header setting 2190 cid meta oid 1fe26b1a/pglog_3.237/0//-1 parent seq 0
2014-06-12 14:45:57.693867 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:45:57.693856 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2840 active] add_log_entry 173'2843 (173'2842) modify   64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 by client.4295.0:143272 2014-06-12 08:45:00.067202
2014-06-12 14:45:57.693900 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2840 active] append_log  adding 1 keys
2014-06-12 14:45:57.693934 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2843, trimmed: 
2014-06-12 14:45:57.693966 7f9d68be2700  5 -- op tracker -- , seq: 747406, time: 2014-06-12 14:45:57.693962, event: started, op: osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.693975 7f9d68be2700  5 queue_transactions existing osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:45:57.693978 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.693980 7f9d68be2700  5 queue_op 0xdf8ad20 seq 2600 osr(3.237 0x276d3f0) 866549 bytes   (queue has 2 ops and 1901034 bytes)
2014-06-12 14:45:57.693988 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540eb40 (2 active)
2014-06-12 14:45:57.693984 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2840 active] do_sub_op osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.694021 7f9d68be2700 10 osd.2 173 dequeue_op 0x1e3eac00 finish
2014-06-12 14:45:57.694025 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.694027 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.700248 7f9d7a161700 10 _do_op 0x6f81b00 seq 2599 r = 0, finisher 0x1d498120 0
2014-06-12 14:45:57.700275 7f9d7a161700 10 _finish_op 0x6f81b00 seq 2599 osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:45:57.700293 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540eb40 (1 active)
2014-06-12 14:45:57.700303 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.700313 7f9d7a962700  5 _do_op 0xdf8ad20 seq 2600 osr(3.237 0x276d3f0)/0x276d3f0 start
2014-06-12 14:45:57.700320 7f9d7a962700 10 _do_transaction on 0x2ca03270
2014-06-12 14:45:57.700316 7f9d7715b700  5 -- op tracker -- , seq: 747403, time: 2014-06-12 14:45:57.700305, event: commit_sent, op: osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.700329 7f9d7a962700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:57.700324 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2840 active] sub_op_modify_commit on op osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:45:57.700347 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:57.700380 7f9d7795c700  5 -- op tracker -- , seq: 747403, time: 2014-06-12 14:45:57.700373, event: sub_op_applied, op: osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.700394 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2841 active] sub_op_modify_applied on 0x63acfc0 op osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.700413 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2841 active] op_applied on primary on version 173'2842
2014-06-12 14:45:57.700424 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2841 active] op_applied on replica on version 173'2842
2014-06-12 14:45:57.700451 7f9d7795c700  5 -- op tracker -- , seq: 747403, time: 2014-06-12 14:45:57.700446, event: done, op: osd_sub_op(client.4295.0:143271 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2842 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.700587 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:57.700615 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.700672 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.700699 7f9d7a962700 15 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:57.700707 7f9d7a962700 10 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1 r = 0
2014-06-12 14:45:57.700715 7f9d7a962700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:45:57.700891 7f9d7a962700 15 _write 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 3330048~864256
2014-06-12 14:45:57.701011 7f9d7a962700 10 lookup_strip_header done  cid 3.237_head oid 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3
2014-06-12 14:45:57.701142 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:45:57.703130 7f9d7a962700 10 _generic_write 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 3330048~864256 = 0
2014-06-12 14:45:57.703209 7f9d7a962700 15 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3
2014-06-12 14:45:57.703230 7f9d7a962700 10 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 = 0
2014-06-12 14:45:57.703236 7f9d7a962700 15 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3
2014-06-12 14:45:57.703245 7f9d7a962700 10 _setattrs 3.237_head/64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 = 0
2014-06-12 14:45:57.703453 7f9d7a962700 20 set_header setting 16982 cid 3.237_head oid 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 parent seq 0
2014-06-12 14:45:57.703547 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.703563 7f9d7a962700 20 set_header setting 2190 cid meta oid 1fe26b1a/pglog_3.237/0//-1 parent seq 0
2014-06-12 14:45:57.703576 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:45:57.707941 7f9d7a962700 10 _do_op 0xdf8ad20 seq 2600 r = 0, finisher 0x8588320 0
2014-06-12 14:45:57.707989 7f9d7715b700  5 -- op tracker -- , seq: 747406, time: 2014-06-12 14:45:57.707978, event: commit_sent, op: osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.708021 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2841 active] sub_op_modify_commit on op osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:45:57.708053 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:45:57.708084 7f9d7a962700 10 _finish_op 0xdf8ad20 seq 2600 osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:45:57.708105 7f9d7795c700  5 -- op tracker -- , seq: 747406, time: 2014-06-12 14:45:57.708100, event: sub_op_applied, op: osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.708113 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active] sub_op_modify_applied on 0x2ca03180 op osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.708131 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active] op_applied on primary on version 173'2843
2014-06-12 14:45:57.708142 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active] op_applied on replica on version 173'2843
2014-06-12 14:45:57.708180 7f9d7a962700  5 -- op tracker -- , seq: 747406, time: 2014-06-12 14:45:57.708173, event: done, op: osd_sub_op(client.4295.0:143272 3.237 64f37237/rbd_data.109e6b8b4567.0000000000000fc0/head//3 [] v 173'2843 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.708238 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540eb40 (0 active)
2014-06-12 14:45:57.708245 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.845359 7f9d44101700  5 -- op tracker -- , seq: 747407, time: 2014-06-12 14:45:57.844913, event: header_read, op: osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.845379 7f9d44101700  5 -- op tracker -- , seq: 747407, time: 2014-06-12 14:45:57.844916, event: throttled, op: osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.845388 7f9d44101700  5 -- op tracker -- , seq: 747407, time: 2014-06-12 14:45:57.845307, event: all_read, op: osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.845402 7f9d44101700  5 -- op tracker -- , seq: 747407, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.845408 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.845424 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:57.845433 7f9d44101700 15 osd.2 173 enqueue_op 0xe20b900 prio 127 cost 156455 latency 0.000520 osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.845465 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.845477 7f9d61bd4700 10 osd.2 173 dequeue_op 0xe20b900 prio 127 cost 156455 latency 0.000564 osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.567( v 173'3083 (79'82,173'3083] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3082 active]
2014-06-12 14:45:57.845506 7f9d61bd4700  5 -- op tracker -- , seq: 747407, time: 2014-06-12 14:45:57.845502, event: reached_pg, op: osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.845517 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3083 (79'82,173'3083] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3082 active] handle_message: 0xe20b900
2014-06-12 14:45:57.845531 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3083 (79'82,173'3083] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3082 active] sub_op_modify trans 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 v 173'3084 (transaction) 174
2014-06-12 14:45:57.845549 7f9d61bd4700  5 -- op tracker -- , seq: 747407, time: 2014-06-12 14:45:57.845545, event: started, op: osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.845567 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3083 (79'82,173'3083] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3082 active] append_log log((79'82,173'3083], crt=0'0) [173'3084 (173'987) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140036 2014-06-12 08:45:00.204363]
2014-06-12 14:45:57.845590 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3084 (79'82,173'3084] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3082 active] add_log_entry 173'3084 (173'987) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140036 2014-06-12 08:45:00.204363
2014-06-12 14:45:57.845625 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3084 (79'82,173'3084] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3082 active] append_log  adding 1 keys
2014-06-12 14:45:57.845642 7f9d61bd4700 10 trim log((79'82,173'3084], crt=0'0) to 79'83
2014-06-12 14:45:57.845671 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3084, trimmed: 79'83
2014-06-12 14:45:57.845698 7f9d61bd4700  5 -- op tracker -- , seq: 747407, time: 2014-06-12 14:45:57.845694, event: started, op: osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.845709 7f9d61bd4700  5 queue_transactions existing osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.845714 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.845716 7f9d61bd4700  5 queue_op 0xdf16600 seq 2735 osr(3.567 0x276db00) 157976 bytes   (queue has 1 ops and 157976 bytes)
2014-06-12 14:45:57.845728 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227ea20 (1 active)
2014-06-12 14:45:57.845735 7f9d7895e700  5 _do_op 0xdf16600 seq 2735 osr(3.567 0x276db00)/0x276db00 start
2014-06-12 14:45:57.845723 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.567( v 173'3084 (79'83,173'3084] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3082 active] do_sub_op osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.845738 7f9d61bd4700 10 osd.2 173 dequeue_op 0xe20b900 finish
2014-06-12 14:45:57.845740 7f9d7895e700 10 _do_transaction on 0x5546470
2014-06-12 14:45:57.845742 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.845745 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.845746 7f9d7895e700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.845951 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.845981 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.846058 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.846082 7f9d7895e700 15 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.846092 7f9d7895e700 10 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1 r = 0
2014-06-12 14:45:57.846099 7f9d7895e700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.846203 7f9d7895e700 15 _write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 0~155648
2014-06-12 14:45:57.846423 7f9d7895e700 10 lookup_strip_header done  cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.846535 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:45:57.847072 7f9d7895e700 10 _generic_write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 0~155648 = 0
2014-06-12 14:45:57.847095 7f9d7895e700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.847110 7f9d7895e700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.847118 7f9d7895e700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.847130 7f9d7895e700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.847270 7f9d7895e700 20 set_header setting 16539 cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:45:57.847303 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.847318 7f9d7895e700 20 set_header setting 2212 cid meta oid 1fe17b5a/pglog_3.567/0//-1 parent seq 0
2014-06-12 14:45:57.847329 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:45:57.849133 7f9d7895e700 10 _do_op 0xdf16600 seq 2735 r = 0, finisher 0x1f24700 0
2014-06-12 14:45:57.849150 7f9d7895e700 10 _finish_op 0xdf16600 seq 2735 osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.849159 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227ea20 (0 active)
2014-06-12 14:45:57.849162 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.849184 7f9d7795c700  5 -- op tracker -- , seq: 747407, time: 2014-06-12 14:45:57.849176, event: sub_op_applied, op: osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.849200 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3084 (79'83,173'3084] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3082 active] sub_op_modify_applied on 0x5546380 op osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.849230 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.849248 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3084 (79'83,173'3084] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3082 active] op_applied on primary on version 173'3084
2014-06-12 14:45:57.849260 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3084 (79'83,173'3084] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3082 active] op_applied on replica on version 173'3084
2014-06-12 14:45:57.849306 7f9d7715b700  5 -- op tracker -- , seq: 747407, time: 2014-06-12 14:45:57.849300, event: commit_sent, op: osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.849318 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3084 (79'83,173'3084] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3082 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:45:57.849336 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.849367 7f9d7715b700  5 -- op tracker -- , seq: 747407, time: 2014-06-12 14:45:57.849362, event: done, op: osd_sub_op(client.4300.0:140036 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3084 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.860161 7f9d44101700  5 -- op tracker -- , seq: 747408, time: 2014-06-12 14:45:57.859334, event: header_read, op: osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.860174 7f9d44101700  5 -- op tracker -- , seq: 747408, time: 2014-06-12 14:45:57.859336, event: throttled, op: osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.860189 7f9d44101700  5 -- op tracker -- , seq: 747408, time: 2014-06-12 14:45:57.860121, event: all_read, op: osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.860195 7f9d44101700  5 -- op tracker -- , seq: 747408, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.860199 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.860204 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:57.860211 7f9d44101700 15 osd.2 173 enqueue_op 0xe20af00 prio 127 cost 516903 latency 0.000877 osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.860239 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.860249 7f9d673df700 10 osd.2 173 dequeue_op 0xe20af00 prio 127 cost 516903 latency 0.000915 osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.567( v 173'3084 (79'83,173'3084] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3083 active]
2014-06-12 14:45:57.860278 7f9d673df700  5 -- op tracker -- , seq: 747408, time: 2014-06-12 14:45:57.860274, event: reached_pg, op: osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.860291 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3084 (79'83,173'3084] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3083 active] handle_message: 0xe20af00
2014-06-12 14:45:57.860306 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3084 (79'83,173'3084] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3083 active] sub_op_modify trans 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 v 173'3085 (transaction) 174
2014-06-12 14:45:57.860334 7f9d673df700  5 -- op tracker -- , seq: 747408, time: 2014-06-12 14:45:57.860329, event: started, op: osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.860351 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3084 (79'83,173'3084] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3083 active] append_log log((79'83,173'3084], crt=0'0) [173'3085 (173'3084) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140037 2014-06-12 08:45:00.232362]
2014-06-12 14:45:57.860378 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3085 (79'83,173'3085] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3083 active] add_log_entry 173'3085 (173'3084) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140037 2014-06-12 08:45:00.232362
2014-06-12 14:45:57.860402 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3085 (79'83,173'3085] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3083 active] append_log  adding 1 keys
2014-06-12 14:45:57.860419 7f9d673df700 10 trim log((79'83,173'3085], crt=0'0) to 79'84
2014-06-12 14:45:57.860453 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3085, trimmed: 79'84
2014-06-12 14:45:57.860485 7f9d673df700  5 -- op tracker -- , seq: 747408, time: 2014-06-12 14:45:57.860480, event: started, op: osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.860498 7f9d673df700  5 queue_transactions existing osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.860505 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.860506 7f9d673df700  5 queue_op 0x33605a0 seq 2736 osr(3.567 0x276db00) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:45:57.860513 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.567( v 173'3085 (79'84,173'3085] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3083 active] do_sub_op osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.860530 7f9d673df700 10 osd.2 173 dequeue_op 0xe20af00 finish
2014-06-12 14:45:57.860534 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.860536 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.860545 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227ea20 (1 active)
2014-06-12 14:45:57.860549 7f9d7f16b700  5 _do_op 0x33605a0 seq 2736 osr(3.567 0x276db00)/0x276db00 start
2014-06-12 14:45:57.860554 7f9d7f16b700 10 _do_transaction on 0x63ad970
2014-06-12 14:45:57.860560 7f9d7f16b700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.860825 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.860857 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.860937 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.860963 7f9d7f16b700 15 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.860974 7f9d7f16b700 10 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1 r = 0
2014-06-12 14:45:57.860982 7f9d7f16b700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.861001 7f9d7f16b700 15 _write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 155648~516096
2014-06-12 14:45:57.861322 7f9d7f16b700 10 lookup_strip_header done  cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.861450 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:45:57.863276 7f9d7f16b700 10 _generic_write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 155648~516096 = 0
2014-06-12 14:45:57.863330 7f9d7f16b700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.863345 7f9d7f16b700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.863351 7f9d7f16b700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.863359 7f9d7f16b700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.863565 7f9d7f16b700 20 set_header setting 16539 cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:45:57.863647 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.863665 7f9d7f16b700 20 set_header setting 2212 cid meta oid 1fe17b5a/pglog_3.567/0//-1 parent seq 0
2014-06-12 14:45:57.863678 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:45:57.866805 7f9d7f16b700 10 _do_op 0x33605a0 seq 2736 r = 0, finisher 0x6d50b00 0
2014-06-12 14:45:57.866824 7f9d7f16b700 10 _finish_op 0x33605a0 seq 2736 osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.866832 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227ea20 (0 active)
2014-06-12 14:45:57.866840 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.866846 7f9d7715b700  5 -- op tracker -- , seq: 747408, time: 2014-06-12 14:45:57.866835, event: commit_sent, op: osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.866861 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3085 (79'84,173'3085] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3083 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:45:57.866889 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.866921 7f9d7795c700  5 -- op tracker -- , seq: 747408, time: 2014-06-12 14:45:57.866916, event: sub_op_applied, op: osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.866930 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3085 (79'84,173'3085] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3084 active] sub_op_modify_applied on 0x63ad880 op osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.866947 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3085 (79'84,173'3085] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3084 active] op_applied on primary on version 173'3085
2014-06-12 14:45:57.866959 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3085 (79'84,173'3085] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3084 active] op_applied on replica on version 173'3085
2014-06-12 14:45:57.866986 7f9d7795c700  5 -- op tracker -- , seq: 747408, time: 2014-06-12 14:45:57.866981, event: done, op: osd_sub_op(client.4300.0:140037 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3085 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.874832 7f9d44101700  5 -- op tracker -- , seq: 747409, time: 2014-06-12 14:45:57.873693, event: header_read, op: osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.874848 7f9d44101700  5 -- op tracker -- , seq: 747409, time: 2014-06-12 14:45:57.873701, event: throttled, op: osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.874857 7f9d44101700  5 -- op tracker -- , seq: 747409, time: 2014-06-12 14:45:57.874793, event: all_read, op: osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.874866 7f9d44101700  5 -- op tracker -- , seq: 747409, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.874872 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.874878 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:57.874888 7f9d44101700 15 osd.2 173 enqueue_op 0xe20aa00 prio 127 cost 516903 latency 0.001195 osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.874919 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.874929 7f9d663dd700 10 osd.2 173 dequeue_op 0xe20aa00 prio 127 cost 516903 latency 0.001236 osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.567( v 173'3085 (79'84,173'3085] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3084 active]
2014-06-12 14:45:57.874955 7f9d663dd700  5 -- op tracker -- , seq: 747409, time: 2014-06-12 14:45:57.874950, event: reached_pg, op: osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.874965 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3085 (79'84,173'3085] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3084 active] handle_message: 0xe20aa00
2014-06-12 14:45:57.874980 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3085 (79'84,173'3085] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3084 active] sub_op_modify trans 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 v 173'3086 (transaction) 174
2014-06-12 14:45:57.874999 7f9d663dd700  5 -- op tracker -- , seq: 747409, time: 2014-06-12 14:45:57.874995, event: started, op: osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.875017 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3085 (79'84,173'3085] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3084 active] append_log log((79'84,173'3085], crt=0'0) [173'3086 (173'3085) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140038 2014-06-12 08:45:00.249778]
2014-06-12 14:45:57.875043 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3086 (79'84,173'3086] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3084 active] add_log_entry 173'3086 (173'3085) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140038 2014-06-12 08:45:00.249778
2014-06-12 14:45:57.875067 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3086 (79'84,173'3086] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3084 active] append_log  adding 1 keys
2014-06-12 14:45:57.875083 7f9d663dd700 10 trim log((79'84,173'3086], crt=0'0) to 79'85
2014-06-12 14:45:57.875111 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3086, trimmed: 79'85
2014-06-12 14:45:57.875149 7f9d663dd700  5 -- op tracker -- , seq: 747409, time: 2014-06-12 14:45:57.875144, event: started, op: osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.875160 7f9d663dd700  5 queue_transactions existing osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.875163 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.875164 7f9d663dd700  5 queue_op 0x6a7bb00 seq 2737 osr(3.567 0x276db00) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:45:57.875173 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227ea20 (1 active)
2014-06-12 14:45:57.875179 7f9d7e169700  5 _do_op 0x6a7bb00 seq 2737 osr(3.567 0x276db00)/0x276db00 start
2014-06-12 14:45:57.875170 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.567( v 173'3086 (79'85,173'3086] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3084 active] do_sub_op osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.875184 7f9d7e169700 10 _do_transaction on 0x8dce9b0
2014-06-12 14:45:57.875184 7f9d663dd700 10 osd.2 173 dequeue_op 0xe20aa00 finish
2014-06-12 14:45:57.875187 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.875190 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.875190 7f9d7e169700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.875374 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.875504 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.875586 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.875613 7f9d7e169700 15 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.875623 7f9d7e169700 10 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1 r = 0
2014-06-12 14:45:57.875631 7f9d7e169700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.875648 7f9d7e169700 15 _write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 671744~516096
2014-06-12 14:45:57.875976 7f9d7e169700 10 lookup_strip_header done  cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.876102 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:45:57.878088 7f9d7e169700 10 _generic_write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 671744~516096 = 0
2014-06-12 14:45:57.878221 7f9d7e169700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.878265 7f9d7e169700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.878291 7f9d7e169700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.878320 7f9d7e169700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.878554 7f9d7e169700 20 set_header setting 16539 cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:45:57.878660 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.878701 7f9d7e169700 20 set_header setting 2212 cid meta oid 1fe17b5a/pglog_3.567/0//-1 parent seq 0
2014-06-12 14:45:57.878733 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:45:57.882350 7f9d7e169700 10 _do_op 0x6a7bb00 seq 2737 r = 0, finisher 0x85880c0 0
2014-06-12 14:45:57.882430 7f9d7e169700 10 _finish_op 0x6a7bb00 seq 2737 osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.882454 7f9d7715b700  5 -- op tracker -- , seq: 747409, time: 2014-06-12 14:45:57.882441, event: commit_sent, op: osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.882457 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227ea20 (0 active)
2014-06-12 14:45:57.882471 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3086 (79'85,173'3086] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3084 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:45:57.882501 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.882520 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.882537 7f9d7795c700  5 -- op tracker -- , seq: 747409, time: 2014-06-12 14:45:57.882532, event: sub_op_applied, op: osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.882548 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3086 (79'85,173'3086] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3085 active] sub_op_modify_applied on 0x8dce8c0 op osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.882569 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3086 (79'85,173'3086] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3085 active] op_applied on primary on version 173'3086
2014-06-12 14:45:57.882580 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3086 (79'85,173'3086] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3085 active] op_applied on replica on version 173'3086
2014-06-12 14:45:57.882602 7f9d7795c700  5 -- op tracker -- , seq: 747409, time: 2014-06-12 14:45:57.882598, event: done, op: osd_sub_op(client.4300.0:140038 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3086 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.888682 7f9d44101700  5 -- op tracker -- , seq: 747410, time: 2014-06-12 14:45:57.887893, event: header_read, op: osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.888694 7f9d44101700  5 -- op tracker -- , seq: 747410, time: 2014-06-12 14:45:57.887894, event: throttled, op: osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.888703 7f9d44101700  5 -- op tracker -- , seq: 747410, time: 2014-06-12 14:45:57.888644, event: all_read, op: osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.888712 7f9d44101700  5 -- op tracker -- , seq: 747410, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.888717 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.888724 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:57.888732 7f9d44101700 15 osd.2 173 enqueue_op 0xe20b600 prio 127 cost 516903 latency 0.000839 osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.888765 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.888772 7f9d67be0700 10 osd.2 173 dequeue_op 0xe20b600 prio 127 cost 516903 latency 0.000879 osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.567( v 173'3086 (79'85,173'3086] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3085 active]
2014-06-12 14:45:57.888793 7f9d67be0700  5 -- op tracker -- , seq: 747410, time: 2014-06-12 14:45:57.888789, event: reached_pg, op: osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.888802 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3086 (79'85,173'3086] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3085 active] handle_message: 0xe20b600
2014-06-12 14:45:57.888814 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3086 (79'85,173'3086] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3085 active] sub_op_modify trans 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 v 173'3087 (transaction) 174
2014-06-12 14:45:57.888831 7f9d67be0700  5 -- op tracker -- , seq: 747410, time: 2014-06-12 14:45:57.888827, event: started, op: osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.888847 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3086 (79'85,173'3086] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3085 active] append_log log((79'85,173'3086], crt=0'0) [173'3087 (173'3086) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140039 2014-06-12 08:45:00.265509]
2014-06-12 14:45:57.888870 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3087 (79'85,173'3087] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3085 active] add_log_entry 173'3087 (173'3086) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140039 2014-06-12 08:45:00.265509
2014-06-12 14:45:57.888899 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3087 (79'85,173'3087] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3085 active] append_log  adding 1 keys
2014-06-12 14:45:57.888913 7f9d67be0700 10 trim log((79'85,173'3087], crt=0'0) to 79'86
2014-06-12 14:45:57.888940 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3087, trimmed: 79'86
2014-06-12 14:45:57.888967 7f9d67be0700  5 -- op tracker -- , seq: 747410, time: 2014-06-12 14:45:57.888962, event: started, op: osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.888977 7f9d67be0700  5 queue_transactions existing osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.888981 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.888983 7f9d67be0700  5 queue_op 0x5e43440 seq 2738 osr(3.567 0x276db00) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:45:57.888990 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.567( v 173'3087 (79'86,173'3087] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3085 active] do_sub_op osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.889011 7f9d67be0700 10 osd.2 173 dequeue_op 0xe20b600 finish
2014-06-12 14:45:57.889015 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.889017 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.889024 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227ea20 (1 active)
2014-06-12 14:45:57.889029 7f9d7e96a700  5 _do_op 0x5e43440 seq 2738 osr(3.567 0x276db00)/0x276db00 start
2014-06-12 14:45:57.889033 7f9d7e96a700 10 _do_transaction on 0x1e78a0f0
2014-06-12 14:45:57.889039 7f9d7e96a700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.889214 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.889243 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.889317 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.889342 7f9d7e96a700 15 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.889350 7f9d7e96a700 10 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1 r = 0
2014-06-12 14:45:57.889356 7f9d7e96a700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.889475 7f9d7e96a700 15 _write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 1187840~516096
2014-06-12 14:45:57.889657 7f9d7e96a700 10 lookup_strip_header done  cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.889773 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:45:57.891575 7f9d7e96a700 10 _generic_write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 1187840~516096 = 0
2014-06-12 14:45:57.891640 7f9d7e96a700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.891658 7f9d7e96a700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.891666 7f9d7e96a700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.891675 7f9d7e96a700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.891890 7f9d7e96a700 20 set_header setting 16539 cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:45:57.891979 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.891997 7f9d7e96a700 20 set_header setting 2212 cid meta oid 1fe17b5a/pglog_3.567/0//-1 parent seq 0
2014-06-12 14:45:57.892008 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:45:57.900266 7f9d7e96a700 10 _do_op 0x5e43440 seq 2738 r = 0, finisher 0x6cfcd80 0
2014-06-12 14:45:57.900346 7f9d7715b700  5 -- op tracker -- , seq: 747410, time: 2014-06-12 14:45:57.900334, event: commit_sent, op: osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.900358 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3087 (79'86,173'3087] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3085 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:45:57.900388 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.900451 7f9d7e96a700 10 _finish_op 0x5e43440 seq 2738 osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.900488 7f9d7795c700  5 -- op tracker -- , seq: 747410, time: 2014-06-12 14:45:57.900482, event: sub_op_applied, op: osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.900497 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3087 (79'86,173'3087] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active] sub_op_modify_applied on 0x1e78a000 op osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.900513 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3087 (79'86,173'3087] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active] op_applied on primary on version 173'3087
2014-06-12 14:45:57.900524 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3087 (79'86,173'3087] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active] op_applied on replica on version 173'3087
2014-06-12 14:45:57.900605 7f9d7e96a700  5 -- op tracker -- , seq: 747410, time: 2014-06-12 14:45:57.900595, event: done, op: osd_sub_op(client.4300.0:140039 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3087 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.900639 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227ea20 (0 active)
2014-06-12 14:45:57.900642 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.906677 7f9d44101700  5 -- op tracker -- , seq: 747411, time: 2014-06-12 14:45:57.905676, event: header_read, op: osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.906698 7f9d44101700  5 -- op tracker -- , seq: 747411, time: 2014-06-12 14:45:57.905678, event: throttled, op: osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.906708 7f9d44101700  5 -- op tracker -- , seq: 747411, time: 2014-06-12 14:45:57.906639, event: all_read, op: osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.906722 7f9d44101700  5 -- op tracker -- , seq: 747411, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.906728 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.906734 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:57.906743 7f9d44101700 15 osd.2 173 enqueue_op 0x2911400 prio 127 cost 516903 latency 0.001067 osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.906776 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.906787 7f9d693e3700 10 osd.2 173 dequeue_op 0x2911400 prio 127 cost 516903 latency 0.001111 osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.567( v 173'3087 (79'86,173'3087] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active]
2014-06-12 14:45:57.906814 7f9d693e3700  5 -- op tracker -- , seq: 747411, time: 2014-06-12 14:45:57.906809, event: reached_pg, op: osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.906826 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3087 (79'86,173'3087] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active] handle_message: 0x2911400
2014-06-12 14:45:57.906840 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3087 (79'86,173'3087] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active] sub_op_modify trans 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 v 173'3088 (transaction) 174
2014-06-12 14:45:57.906859 7f9d693e3700  5 -- op tracker -- , seq: 747411, time: 2014-06-12 14:45:57.906855, event: started, op: osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.906876 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3087 (79'86,173'3087] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active] append_log log((79'86,173'3087], crt=0'0) [173'3088 (173'3087) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140040 2014-06-12 08:45:00.283466]
2014-06-12 14:45:57.906909 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3088 (79'86,173'3088] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active] add_log_entry 173'3088 (173'3087) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140040 2014-06-12 08:45:00.283466
2014-06-12 14:45:57.906933 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3088 (79'86,173'3088] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active] append_log  adding 1 keys
2014-06-12 14:45:57.906949 7f9d693e3700 10 trim log((79'86,173'3088], crt=0'0) to 79'87
2014-06-12 14:45:57.906977 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3088, trimmed: 79'87
2014-06-12 14:45:57.907012 7f9d693e3700  5 -- op tracker -- , seq: 747411, time: 2014-06-12 14:45:57.907007, event: started, op: osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.907026 7f9d693e3700  5 queue_transactions existing osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.907030 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.907032 7f9d693e3700  5 queue_op 0x13c51ce0 seq 2739 osr(3.567 0x276db00) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:45:57.907038 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.567( v 173'3088 (79'87,173'3088] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active] do_sub_op osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.907053 7f9d693e3700 10 osd.2 173 dequeue_op 0x2911400 finish
2014-06-12 14:45:57.907057 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.907059 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.907067 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227ea20 (1 active)
2014-06-12 14:45:57.907072 7f9d82171700  5 _do_op 0x13c51ce0 seq 2739 osr(3.567 0x276db00)/0x276db00 start
2014-06-12 14:45:57.907077 7f9d82171700 10 _do_transaction on 0x30b07eb0
2014-06-12 14:45:57.907083 7f9d82171700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.907260 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.907289 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.907364 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.907387 7f9d82171700 15 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.907398 7f9d82171700 10 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1 r = 0
2014-06-12 14:45:57.907404 7f9d82171700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.907418 7f9d82171700 15 _write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 1703936~516096
2014-06-12 14:45:57.907587 7f9d82171700 10 lookup_strip_header done  cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.907860 7f9d82171700 10 file_to_extents done 
2014-06-12 14:45:57.909636 7f9d82171700 10 _generic_write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 1703936~516096 = 0
2014-06-12 14:45:57.909693 7f9d82171700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.909708 7f9d82171700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.909716 7f9d82171700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.909725 7f9d82171700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.909933 7f9d82171700 20 set_header setting 16539 cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:45:57.910008 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.910024 7f9d82171700 20 set_header setting 2212 cid meta oid 1fe17b5a/pglog_3.567/0//-1 parent seq 0
2014-06-12 14:45:57.910037 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:45:57.913548 7f9d82171700 10 _do_op 0x13c51ce0 seq 2739 r = 0, finisher 0x73ea820 0
2014-06-12 14:45:57.913573 7f9d82171700 10 _finish_op 0x13c51ce0 seq 2739 osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.913582 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227ea20 (0 active)
2014-06-12 14:45:57.913593 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.913602 7f9d7795c700  5 -- op tracker -- , seq: 747411, time: 2014-06-12 14:45:57.913590, event: sub_op_applied, op: osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.913609 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3088 (79'87,173'3088] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active] sub_op_modify_applied on 0x30b07dc0 op osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.913657 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.913677 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3088 (79'87,173'3088] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active] op_applied on primary on version 173'3088
2014-06-12 14:45:57.913688 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3088 (79'87,173'3088] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active] op_applied on replica on version 173'3088
2014-06-12 14:45:57.913729 7f9d7715b700  5 -- op tracker -- , seq: 747411, time: 2014-06-12 14:45:57.913722, event: commit_sent, op: osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.913743 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3088 (79'87,173'3088] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3086 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:45:57.913762 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.913790 7f9d7715b700  5 -- op tracker -- , seq: 747411, time: 2014-06-12 14:45:57.913785, event: done, op: osd_sub_op(client.4300.0:140040 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3088 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.924148 7f9d44101700  5 -- op tracker -- , seq: 747412, time: 2014-06-12 14:45:57.923194, event: header_read, op: osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.924162 7f9d44101700  5 -- op tracker -- , seq: 747412, time: 2014-06-12 14:45:57.923196, event: throttled, op: osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.924172 7f9d44101700  5 -- op tracker -- , seq: 747412, time: 2014-06-12 14:45:57.924098, event: all_read, op: osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.924181 7f9d44101700  5 -- op tracker -- , seq: 747412, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.924186 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.924195 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:57.924204 7f9d44101700 15 osd.2 173 enqueue_op 0x2911300 prio 127 cost 516903 latency 0.001010 osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.924238 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.924247 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2911300 prio 127 cost 516903 latency 0.001053 osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.567( v 173'3088 (79'87,173'3088] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3087 active]
2014-06-12 14:45:57.924273 7f9d62bd6700  5 -- op tracker -- , seq: 747412, time: 2014-06-12 14:45:57.924269, event: reached_pg, op: osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.924283 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3088 (79'87,173'3088] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3087 active] handle_message: 0x2911300
2014-06-12 14:45:57.924300 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3088 (79'87,173'3088] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3087 active] sub_op_modify trans 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 v 173'3089 (transaction) 174
2014-06-12 14:45:57.924317 7f9d62bd6700  5 -- op tracker -- , seq: 747412, time: 2014-06-12 14:45:57.924313, event: started, op: osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.924333 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3088 (79'87,173'3088] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3087 active] append_log log((79'87,173'3088], crt=0'0) [173'3089 (173'3088) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140041 2014-06-12 08:45:00.296710]
2014-06-12 14:45:57.924367 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3089 (79'87,173'3089] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3087 active] add_log_entry 173'3089 (173'3088) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140041 2014-06-12 08:45:00.296710
2014-06-12 14:45:57.924387 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3089 (79'87,173'3089] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3087 active] append_log  adding 1 keys
2014-06-12 14:45:57.924399 7f9d62bd6700 10 trim log((79'87,173'3089], crt=0'0) to 79'88
2014-06-12 14:45:57.924425 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3089, trimmed: 79'88
2014-06-12 14:45:57.924451 7f9d62bd6700  5 -- op tracker -- , seq: 747412, time: 2014-06-12 14:45:57.924446, event: started, op: osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.924460 7f9d62bd6700  5 queue_transactions existing osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.924464 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.924466 7f9d62bd6700  5 queue_op 0x6268240 seq 2740 osr(3.567 0x276db00) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:45:57.924473 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.567( v 173'3089 (79'88,173'3089] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3087 active] do_sub_op osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.924486 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2911300 finish
2014-06-12 14:45:57.924490 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.924492 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.926560 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227ea20 (1 active)
2014-06-12 14:45:57.926572 7f9d7d167700  5 _do_op 0x6268240 seq 2740 osr(3.567 0x276db00)/0x276db00 start
2014-06-12 14:45:57.926577 7f9d7d167700 10 _do_transaction on 0x31d09970
2014-06-12 14:45:57.926584 7f9d7d167700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.926751 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.926781 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.926857 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.926974 7f9d7d167700 15 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.926986 7f9d7d167700 10 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1 r = 0
2014-06-12 14:45:57.926994 7f9d7d167700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.927014 7f9d7d167700 15 _write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 2220032~516096
2014-06-12 14:45:57.927192 7f9d7d167700 10 lookup_strip_header done  cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.927316 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:45:57.929216 7f9d7d167700 10 _generic_write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 2220032~516096 = 0
2014-06-12 14:45:57.929274 7f9d7d167700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.929288 7f9d7d167700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.929297 7f9d7d167700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.929306 7f9d7d167700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.929537 7f9d7d167700 20 set_header setting 16539 cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:45:57.929645 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.929663 7f9d7d167700 20 set_header setting 2212 cid meta oid 1fe17b5a/pglog_3.567/0//-1 parent seq 0
2014-06-12 14:45:57.929677 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:45:57.941301 7f9d7d167700 10 _do_op 0x6268240 seq 2740 r = 0, finisher 0x1f087420 0
2014-06-12 14:45:57.941336 7f9d7d167700 10 _finish_op 0x6268240 seq 2740 osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.941348 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227ea20 (0 active)
2014-06-12 14:45:57.941357 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.941360 7f9d7715b700  5 -- op tracker -- , seq: 747412, time: 2014-06-12 14:45:57.941347, event: commit_sent, op: osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.941371 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3089 (79'88,173'3089] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3087 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:45:57.941403 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.941461 7f9d7795c700  5 -- op tracker -- , seq: 747412, time: 2014-06-12 14:45:57.941453, event: sub_op_applied, op: osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.941471 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3089 (79'88,173'3089] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active] sub_op_modify_applied on 0x31d09880 op osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.941488 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3089 (79'88,173'3089] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active] op_applied on primary on version 173'3089
2014-06-12 14:45:57.941498 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3089 (79'88,173'3089] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active] op_applied on replica on version 173'3089
2014-06-12 14:45:57.941521 7f9d7795c700  5 -- op tracker -- , seq: 747412, time: 2014-06-12 14:45:57.941516, event: done, op: osd_sub_op(client.4300.0:140041 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3089 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.953733 7f9d44101700  5 -- op tracker -- , seq: 747413, time: 2014-06-12 14:45:57.953022, event: header_read, op: osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.953752 7f9d44101700  5 -- op tracker -- , seq: 747413, time: 2014-06-12 14:45:57.953024, event: throttled, op: osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.953760 7f9d44101700  5 -- op tracker -- , seq: 747413, time: 2014-06-12 14:45:57.953686, event: all_read, op: osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.953768 7f9d44101700  5 -- op tracker -- , seq: 747413, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.953774 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.953781 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:57.953791 7f9d44101700 15 osd.2 173 enqueue_op 0x2910300 prio 127 cost 516903 latency 0.000769 osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.953827 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.953835 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2910300 prio 127 cost 516903 latency 0.000813 osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.567( v 173'3089 (79'88,173'3089] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active]
2014-06-12 14:45:57.953859 7f9d6b3e7700  5 -- op tracker -- , seq: 747413, time: 2014-06-12 14:45:57.953855, event: reached_pg, op: osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.953872 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3089 (79'88,173'3089] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active] handle_message: 0x2910300
2014-06-12 14:45:57.953887 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3089 (79'88,173'3089] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active] sub_op_modify trans 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 v 173'3090 (transaction) 174
2014-06-12 14:45:57.953909 7f9d6b3e7700  5 -- op tracker -- , seq: 747413, time: 2014-06-12 14:45:57.953904, event: started, op: osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.953930 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3089 (79'88,173'3089] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active] append_log log((79'88,173'3089], crt=0'0) [173'3090 (173'3089) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140042 2014-06-12 08:45:00.324526]
2014-06-12 14:45:57.953952 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3090 (79'88,173'3090] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active] add_log_entry 173'3090 (173'3089) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140042 2014-06-12 08:45:00.324526
2014-06-12 14:45:57.953975 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3090 (79'88,173'3090] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active] append_log  adding 1 keys
2014-06-12 14:45:57.953992 7f9d6b3e7700 10 trim log((79'88,173'3090], crt=0'0) to 79'89
2014-06-12 14:45:57.954019 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3090, trimmed: 79'89
2014-06-12 14:45:57.954047 7f9d6b3e7700  5 -- op tracker -- , seq: 747413, time: 2014-06-12 14:45:57.954043, event: started, op: osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.954057 7f9d6b3e7700  5 queue_transactions existing osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.954062 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.954064 7f9d6b3e7700  5 queue_op 0x87dc780 seq 2741 osr(3.567 0x276db00) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:45:57.954071 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.567( v 173'3090 (79'89,173'3090] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active] do_sub_op osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.954087 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2910300 finish
2014-06-12 14:45:57.954091 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.954093 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.954132 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227ea20 (1 active)
2014-06-12 14:45:57.954140 7f9d7815d700  5 _do_op 0x87dc780 seq 2741 osr(3.567 0x276db00)/0x276db00 start
2014-06-12 14:45:57.954145 7f9d7815d700 10 _do_transaction on 0x5ab9b30
2014-06-12 14:45:57.954152 7f9d7815d700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.954289 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.954312 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.954374 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.954409 7f9d7815d700 15 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.954419 7f9d7815d700 10 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1 r = 0
2014-06-12 14:45:57.954425 7f9d7815d700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.954440 7f9d7815d700 15 _write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 2736128~516096
2014-06-12 14:45:57.954564 7f9d7815d700 10 lookup_strip_header done  cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.954739 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:45:57.956329 7f9d7815d700 10 _generic_write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 2736128~516096 = 0
2014-06-12 14:45:57.956413 7f9d7815d700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.956448 7f9d7815d700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.956474 7f9d7815d700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.956502 7f9d7815d700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.956740 7f9d7815d700 20 set_header setting 16539 cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:45:57.956844 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.956881 7f9d7815d700 20 set_header setting 2212 cid meta oid 1fe17b5a/pglog_3.567/0//-1 parent seq 0
2014-06-12 14:45:57.956923 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:45:57.960747 7f9d7815d700 10 _do_op 0x87dc780 seq 2741 r = 0, finisher 0x631aa00 0
2014-06-12 14:45:57.960809 7f9d7815d700 10 _finish_op 0x87dc780 seq 2741 osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.960837 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227ea20 (0 active)
2014-06-12 14:45:57.960860 7f9d7795c700  5 -- op tracker -- , seq: 747413, time: 2014-06-12 14:45:57.960848, event: sub_op_applied, op: osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.960872 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.960874 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3090 (79'89,173'3090] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active] sub_op_modify_applied on 0x5ab9a40 op osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.960906 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.960928 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3090 (79'89,173'3090] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active] op_applied on primary on version 173'3090
2014-06-12 14:45:57.960940 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3090 (79'89,173'3090] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active] op_applied on replica on version 173'3090
2014-06-12 14:45:57.960964 7f9d7715b700  5 -- op tracker -- , seq: 747413, time: 2014-06-12 14:45:57.960957, event: commit_sent, op: osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.960973 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3090 (79'89,173'3090] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3088 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:45:57.960992 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.961019 7f9d7715b700  5 -- op tracker -- , seq: 747413, time: 2014-06-12 14:45:57.961014, event: done, op: osd_sub_op(client.4300.0:140042 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3090 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.966843 7f9d44101700  5 -- op tracker -- , seq: 747414, time: 2014-06-12 14:45:57.966047, event: header_read, op: osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.966890 7f9d44101700  5 -- op tracker -- , seq: 747414, time: 2014-06-12 14:45:57.966048, event: throttled, op: osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.966921 7f9d44101700  5 -- op tracker -- , seq: 747414, time: 2014-06-12 14:45:57.966790, event: all_read, op: osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.966950 7f9d44101700  5 -- op tracker -- , seq: 747414, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.966976 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.967004 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:57.967037 7f9d44101700 15 osd.2 173 enqueue_op 0x2911500 prio 127 cost 516903 latency 0.000989 osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.967087 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.967098 7f9d64bda700 10 osd.2 173 dequeue_op 0x2911500 prio 127 cost 516903 latency 0.001051 osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.567( v 173'3090 (79'89,173'3090] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3089 active]
2014-06-12 14:45:57.967121 7f9d64bda700  5 -- op tracker -- , seq: 747414, time: 2014-06-12 14:45:57.967117, event: reached_pg, op: osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.967130 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3090 (79'89,173'3090] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3089 active] handle_message: 0x2911500
2014-06-12 14:45:57.967143 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3090 (79'89,173'3090] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3089 active] sub_op_modify trans 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 v 173'3091 (transaction) 174
2014-06-12 14:45:57.967160 7f9d64bda700  5 -- op tracker -- , seq: 747414, time: 2014-06-12 14:45:57.967156, event: started, op: osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.967177 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3090 (79'89,173'3090] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3089 active] append_log log((79'89,173'3090], crt=0'0) [173'3091 (173'3090) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140043 2014-06-12 08:45:00.344193]
2014-06-12 14:45:57.967205 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3091 (79'89,173'3091] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3089 active] add_log_entry 173'3091 (173'3090) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140043 2014-06-12 08:45:00.344193
2014-06-12 14:45:57.967227 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3091 (79'89,173'3091] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3089 active] append_log  adding 1 keys
2014-06-12 14:45:57.967241 7f9d64bda700 10 trim log((79'89,173'3091], crt=0'0) to 79'90
2014-06-12 14:45:57.967269 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3091, trimmed: 79'90
2014-06-12 14:45:57.967296 7f9d64bda700  5 -- op tracker -- , seq: 747414, time: 2014-06-12 14:45:57.967291, event: started, op: osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.967306 7f9d64bda700  5 queue_transactions existing osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.967311 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.967313 7f9d64bda700  5 queue_op 0x7c88360 seq 2742 osr(3.567 0x276db00) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:45:57.967320 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.567( v 173'3091 (79'90,173'3091] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3089 active] do_sub_op osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.967334 7f9d64bda700 10 osd.2 173 dequeue_op 0x2911500 finish
2014-06-12 14:45:57.967338 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.967340 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.967347 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227ea20 (1 active)
2014-06-12 14:45:57.967352 7f9d79960700  5 _do_op 0x7c88360 seq 2742 osr(3.567 0x276db00)/0x276db00 start
2014-06-12 14:45:57.967356 7f9d79960700 10 _do_transaction on 0x6670d30
2014-06-12 14:45:57.967362 7f9d79960700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.967546 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.967573 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.967647 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.967745 7f9d79960700 15 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.967754 7f9d79960700 10 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1 r = 0
2014-06-12 14:45:57.967760 7f9d79960700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.967779 7f9d79960700 15 _write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 3252224~516096
2014-06-12 14:45:57.967950 7f9d79960700 10 lookup_strip_header done  cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.968101 7f9d79960700 10 file_to_extents done 
2014-06-12 14:45:57.970254 7f9d79960700 10 _generic_write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 3252224~516096 = 0
2014-06-12 14:45:57.970313 7f9d79960700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.970328 7f9d79960700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.970335 7f9d79960700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.970343 7f9d79960700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.970555 7f9d79960700 20 set_header setting 16539 cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:45:57.970631 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.970646 7f9d79960700 20 set_header setting 2212 cid meta oid 1fe17b5a/pglog_3.567/0//-1 parent seq 0
2014-06-12 14:45:57.970655 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:45:57.973875 7f9d79960700 10 _do_op 0x7c88360 seq 2742 r = 0, finisher 0x92ff820 0
2014-06-12 14:45:57.973916 7f9d7715b700  5 -- op tracker -- , seq: 747414, time: 2014-06-12 14:45:57.973906, event: commit_sent, op: osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.973928 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3091 (79'90,173'3091] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3089 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:45:57.973959 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.973988 7f9d79960700 10 _finish_op 0x7c88360 seq 2742 osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.973998 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227ea20 (0 active)
2014-06-12 14:45:57.974005 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.974034 7f9d7795c700  5 -- op tracker -- , seq: 747414, time: 2014-06-12 14:45:57.974027, event: sub_op_applied, op: osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.974042 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3091 (79'90,173'3091] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active] sub_op_modify_applied on 0x6670c40 op osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.974064 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3091 (79'90,173'3091] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active] op_applied on primary on version 173'3091
2014-06-12 14:45:57.974075 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3091 (79'90,173'3091] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active] op_applied on replica on version 173'3091
2014-06-12 14:45:57.974096 7f9d7795c700  5 -- op tracker -- , seq: 747414, time: 2014-06-12 14:45:57.974091, event: done, op: osd_sub_op(client.4300.0:140043 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3091 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.979059 7f9d44101700  5 -- op tracker -- , seq: 747415, time: 2014-06-12 14:45:57.978313, event: header_read, op: osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.979075 7f9d44101700  5 -- op tracker -- , seq: 747415, time: 2014-06-12 14:45:57.978314, event: throttled, op: osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.979083 7f9d44101700  5 -- op tracker -- , seq: 747415, time: 2014-06-12 14:45:57.979017, event: all_read, op: osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.979092 7f9d44101700  5 -- op tracker -- , seq: 747415, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.979098 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:45:57.979106 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:45:57.979115 7f9d44101700 15 osd.2 173 enqueue_op 0x2911100 prio 127 cost 426791 latency 0.000802 osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.979149 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:45:57.979156 7f9d683e1700 10 osd.2 173 dequeue_op 0x2911100 prio 127 cost 426791 latency 0.000843 osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.567( v 173'3091 (79'90,173'3091] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active]
2014-06-12 14:45:57.979181 7f9d683e1700  5 -- op tracker -- , seq: 747415, time: 2014-06-12 14:45:57.979177, event: reached_pg, op: osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.979190 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3091 (79'90,173'3091] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active] handle_message: 0x2911100
2014-06-12 14:45:57.979210 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3091 (79'90,173'3091] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active] sub_op_modify trans 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 v 173'3092 (transaction) 174
2014-06-12 14:45:57.979235 7f9d683e1700  5 -- op tracker -- , seq: 747415, time: 2014-06-12 14:45:57.979230, event: started, op: osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.979251 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3091 (79'90,173'3091] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active] append_log log((79'90,173'3091], crt=0'0) [173'3092 (173'3091) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140044 2014-06-12 08:45:00.357070]
2014-06-12 14:45:57.979275 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3092 (79'90,173'3092] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active] add_log_entry 173'3092 (173'3091) modify   72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 by client.4300.0:140044 2014-06-12 08:45:00.357070
2014-06-12 14:45:57.979302 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3092 (79'90,173'3092] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active] append_log  adding 1 keys
2014-06-12 14:45:57.979318 7f9d683e1700 10 trim log((79'90,173'3092], crt=0'0) to 79'91
2014-06-12 14:45:57.979347 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3092, trimmed: 79'91
2014-06-12 14:45:57.979373 7f9d683e1700  5 -- op tracker -- , seq: 747415, time: 2014-06-12 14:45:57.979369, event: started, op: osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.979383 7f9d683e1700  5 queue_transactions existing osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.979388 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:45:57.979390 7f9d683e1700  5 queue_op 0x7c136e0 seq 2743 osr(3.567 0x276db00) 428312 bytes   (queue has 1 ops and 428312 bytes)
2014-06-12 14:45:57.979400 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227ea20 (1 active)
2014-06-12 14:45:57.979407 7f9d8116f700  5 _do_op 0x7c136e0 seq 2743 osr(3.567 0x276db00)/0x276db00 start
2014-06-12 14:45:57.979410 7f9d8116f700 10 _do_transaction on 0x2125cf0
2014-06-12 14:45:57.979397 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.567( v 173'3092 (79'91,173'3092] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active] do_sub_op osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.979412 7f9d683e1700 10 osd.2 173 dequeue_op 0x2911100 finish
2014-06-12 14:45:57.979416 7f9d8116f700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.979416 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:45:57.979419 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:45:57.979548 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.979570 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:45:57.979620 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:45:57.979698 7f9d8116f700 15 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.979706 7f9d8116f700 10 _omap_rmkeys meta/1fe17b5a/pglog_3.567/0//-1 r = 0
2014-06-12 14:45:57.979710 7f9d8116f700 15 _omap_setkeys meta/1fe17b5a/pglog_3.567/0//-1
2014-06-12 14:45:57.979719 7f9d8116f700 15 _write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 3768320~425984
2014-06-12 14:45:57.979836 7f9d8116f700 10 lookup_strip_header done  cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.979961 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:45:57.980914 7f9d8116f700 10 _generic_write 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 3768320~425984 = 0
2014-06-12 14:45:57.980950 7f9d8116f700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.980960 7f9d8116f700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.980965 7f9d8116f700 15 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3
2014-06-12 14:45:57.980971 7f9d8116f700 10 _setattrs 3.567_head/72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:45:57.981097 7f9d8116f700 20 set_header setting 16539 cid 3.567_head oid 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:45:57.981146 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:45:57.981156 7f9d8116f700 20 set_header setting 2212 cid meta oid 1fe17b5a/pglog_3.567/0//-1 parent seq 0
2014-06-12 14:45:57.981163 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:45:57.984315 7f9d8116f700 10 _do_op 0x7c136e0 seq 2743 r = 0, finisher 0x88a6520 0
2014-06-12 14:45:57.984338 7f9d8116f700 10 _finish_op 0x7c136e0 seq 2743 osr(3.567 0x276db00)/0x276db00
2014-06-12 14:45:57.984346 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227ea20 (0 active)
2014-06-12 14:45:57.984350 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:57.984368 7f9d7795c700  5 -- op tracker -- , seq: 747415, time: 2014-06-12 14:45:57.984356, event: sub_op_applied, op: osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.984375 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3092 (79'91,173'3092] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active] sub_op_modify_applied on 0x2125c00 op osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:45:57.984403 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.984428 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3092 (79'91,173'3092] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active] op_applied on primary on version 173'3092
2014-06-12 14:45:57.984441 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3092 (79'91,173'3092] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active] op_applied on replica on version 173'3092
2014-06-12 14:45:57.984463 7f9d7715b700  5 -- op tracker -- , seq: 747415, time: 2014-06-12 14:45:57.984457, event: commit_sent, op: osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.984484 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.567( v 173'3092 (79'91,173'3092] local-les=139 n=298 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3090 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:45:57.984506 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:57.984553 7f9d7715b700  5 -- op tracker -- , seq: 747415, time: 2014-06-12 14:45:57.984548, event: done, op: osd_sub_op(client.4300.0:140044 3.567 72dd9567/rbd_data.10a36b8b4567.0000000000000ec2/head//3 [] v 173'3092 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:45:57.996145 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:45:57.996263 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:45:58.152752 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:45:58.315541 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:45:58.315805 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:45:58.366404 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:45:58.431887 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:58.431890 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:58.431891 7f9d74bfa700 20 osd.2 173 _dispatch 0x587e380 pg_stats_ack(10 pgs tid 4845) v1
2014-06-12 14:45:58.431894 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:45:58.431904 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:58.431905 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:58.464847 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:45:58.464877 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:45:58.464905 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 12.35 >= max 0.5 = no, load too high
2014-06-12 14:45:58.464921 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:45:58.464925 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92606 < max (604800 seconds)
2014-06-12 14:45:58.464937 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:45:58.464940 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:58.464942 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:58.464944 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 42; oldest is 35.023290 seconds old
2014-06-12 14:45:58.533498 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:58.534008 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:45:59.248197 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:45:59.250471 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:45:59.465011 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:45:59.465036 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:45:59.465040 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:59.465041 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:59.465044 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 42; oldest is 36.023389 seconds old
2014-06-12 14:45:59.465095 7f9d84175700  0 log [WRN] : 7 slow requests, 3 included below; oldest blocked for > 36.023389 secs
2014-06-12 14:45:59.465099 7f9d84175700  0 log [WRN] : slow request 30.960203 seconds old, received at 2014-06-12 14:45:28.504840: osd_op(client.4311.0:139602 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3182592~516096] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:45:59.465104 7f9d84175700  0 log [WRN] : slow request 30.804128 seconds old, received at 2014-06-12 14:45:28.660915: osd_op(client.4311.0:139603 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3698688~495616] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:45:59.465107 7f9d84175700  0 log [WRN] : slow request 30.800350 seconds old, received at 2014-06-12 14:45:28.664693: osd_op(client.4292.0:142326 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3964928~229376] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:45:59.465120 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:45:59.510383 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:45:59.510823 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:45:59.522872 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.534923 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.538485 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:59.538508 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:45:59.539278 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.550942 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.557718 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.559336 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.561544 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.573842 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.582498 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.589294 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.596334 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.611169 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.641810 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.656714 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.661842 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.664474 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.682594 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.686121 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.689771 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.694088 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.698608 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:45:59.700364 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.704655 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:45:59.709095 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.845807 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.849254 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.860598 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.866930 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.875253 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.882599 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.889079 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.900705 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.907135 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.913648 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.924553 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.941424 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.954157 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.960936 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.967405 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.974069 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.979487 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:45:59.984420 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:45:59.986009 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:59.986012 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:59.986014 7f9d74bfa700 20 osd.2 173 _dispatch 0x5392340 ping v1
2014-06-12 14:45:59.986016 7f9d74bfa700 10 osd.2 173 ping from client.4292
2014-06-12 14:45:59.986018 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:45:59.986019 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:45:59.986835 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:45:59.986872 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:00.152829 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:00.360850 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:00.360903 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:00.366481 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:00.465170 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:00.465196 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:00.465199 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:00.465201 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:00.465201 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:00.465203 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 42; oldest is 37.023548 seconds old
2014-06-12 14:46:00.465278 7f9d84175700  0 log [WRN] : 13 slow requests, 6 included below; oldest blocked for > 37.023548 secs
2014-06-12 14:46:00.465287 7f9d84175700  0 log [WRN] : slow request 30.267549 seconds old, received at 2014-06-12 14:45:30.197653: osd_op(client.4310.0:142386 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:46:00.465291 7f9d84175700  0 log [WRN] : slow request 30.255545 seconds old, received at 2014-06-12 14:45:30.209657: osd_op(client.4310.0:142387 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:46:00.465301 7f9d84175700  0 log [WRN] : slow request 30.245279 seconds old, received at 2014-06-12 14:45:30.219923: osd_op(client.4310.0:142388 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:46:00.465304 7f9d84175700  0 log [WRN] : slow request 30.238969 seconds old, received at 2014-06-12 14:45:30.226233: osd_op(client.4310.0:142389 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:46:00.465308 7f9d84175700  0 log [WRN] : slow request 30.222334 seconds old, received at 2014-06-12 14:45:30.242868: osd_op(client.4310.0:142390 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1548288] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:46:00.551146 7f9d5501e700  5 -- op tracker -- , seq: 747416, time: 2014-06-12 14:46:00.550375, event: header_read, op: osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.551158 7f9d5501e700  5 -- op tracker -- , seq: 747416, time: 2014-06-12 14:46:00.550379, event: throttled, op: osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.551166 7f9d5501e700  5 -- op tracker -- , seq: 747416, time: 2014-06-12 14:46:00.551109, event: all_read, op: osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.551173 7f9d5501e700  5 -- op tracker -- , seq: 747416, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.551178 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:00.551184 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:00.551191 7f9d5501e700 15 osd.2 173 enqueue_op 0x876e500 prio 127 cost 365351 latency 0.000816 osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.551216 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.551231 7f9d65bdc700 10 osd.2 173 dequeue_op 0x876e500 prio 127 cost 365351 latency 0.000856 osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2772 (0'0,173'2772] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2771 active]
2014-06-12 14:46:00.551254 7f9d65bdc700  5 -- op tracker -- , seq: 747416, time: 2014-06-12 14:46:00.551250, event: reached_pg, op: osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.551280 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2772 (0'0,173'2772] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2771 active] handle_message: 0x876e500
2014-06-12 14:46:00.551299 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2772 (0'0,173'2772] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2771 active] sub_op_modify trans e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 v 173'2773 (transaction) 174
2014-06-12 14:46:00.551314 7f9d65bdc700  5 -- op tracker -- , seq: 747416, time: 2014-06-12 14:46:00.551310, event: started, op: osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.551327 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2772 (0'0,173'2772] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2771 active] append_log log((0'0,173'2772], crt=0'0) [173'2773 (173'714) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144752 2014-06-12 08:45:02.929238]
2014-06-12 14:46:00.551348 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2773 (0'0,173'2773] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2771 active] add_log_entry 173'2773 (173'714) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144752 2014-06-12 08:45:02.929238
2014-06-12 14:46:00.551374 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2773 (0'0,173'2773] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2771 active] append_log  adding 1 keys
2014-06-12 14:46:00.551404 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2773, trimmed: 
2014-06-12 14:46:00.551429 7f9d65bdc700  5 -- op tracker -- , seq: 747416, time: 2014-06-12 14:46:00.551424, event: started, op: osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.551440 7f9d65bdc700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.551443 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.551445 7f9d65bdc700  5 queue_op 0x139d9bc0 seq 2510 osr(3.6d1 0x276df10) 366837 bytes   (queue has 1 ops and 366837 bytes)
2014-06-12 14:46:00.551451 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2773 (0'0,173'2773] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2771 active] do_sub_op osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.551464 7f9d65bdc700 10 osd.2 173 dequeue_op 0x876e500 finish
2014-06-12 14:46:00.551468 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.551470 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.551478 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (1 active)
2014-06-12 14:46:00.551483 7f9d7d968700  5 _do_op 0x139d9bc0 seq 2510 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:00.551486 7f9d7d968700 10 _do_transaction on 0x601b0b0
2014-06-12 14:46:00.551491 7f9d7d968700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.551636 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.551657 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.551712 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.551735 7f9d7d968700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.551743 7f9d7d968700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:00.551749 7f9d7d968700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.551927 7f9d7d968700 15 _write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 0~364544
2014-06-12 14:46:00.552100 7f9d7d968700 10 lookup_strip_header done  cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.552278 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:00.553577 7f9d7d968700 10 _generic_write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 0~364544 = 0
2014-06-12 14:46:00.553606 7f9d7d968700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.553643 7f9d7d968700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.553649 7f9d7d968700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.553657 7f9d7d968700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.553816 7f9d7d968700 20 set_header setting 17236 cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 parent seq 0
2014-06-12 14:46:00.553870 7f9d7d968700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:00.553887 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.553898 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:00.555532 7f9d7d968700 10 _do_op 0x139d9bc0 seq 2510 r = 0, finisher 0x7232640 0
2014-06-12 14:46:00.555580 7f9d7715b700  5 -- op tracker -- , seq: 747416, time: 2014-06-12 14:46:00.555569, event: commit_sent, op: osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.555593 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2773 (0'0,173'2773] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2771 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:00.555625 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:00.555656 7f9d7d968700 10 _finish_op 0x139d9bc0 seq 2510 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.555679 7f9d7795c700  5 -- op tracker -- , seq: 747416, time: 2014-06-12 14:46:00.555674, event: sub_op_applied, op: osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.555688 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2773 (0'0,173'2773] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2772 active] sub_op_modify_applied on 0x601afc0 op osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.555707 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2773 (0'0,173'2773] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2772 active] op_applied on primary on version 173'2773
2014-06-12 14:46:00.555718 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2773 (0'0,173'2773] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2772 active] op_applied on replica on version 173'2773
2014-06-12 14:46:00.555747 7f9d7d968700  5 -- op tracker -- , seq: 747416, time: 2014-06-12 14:46:00.555737, event: done, op: osd_sub_op(client.4315.0:144752 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2773 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.555777 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:00.555783 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.561272 7f9d5501e700  5 -- op tracker -- , seq: 747417, time: 2014-06-12 14:46:00.560337, event: header_read, op: osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.561285 7f9d5501e700  5 -- op tracker -- , seq: 747417, time: 2014-06-12 14:46:00.560339, event: throttled, op: osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.561301 7f9d5501e700  5 -- op tracker -- , seq: 747417, time: 2014-06-12 14:46:00.561228, event: all_read, op: osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.561307 7f9d5501e700  5 -- op tracker -- , seq: 747417, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.561312 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:00.561317 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:00.561324 7f9d5501e700 15 osd.2 173 enqueue_op 0x124e3d00 prio 127 cost 516903 latency 0.000987 osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.561355 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.561365 7f9d6abe6700 10 osd.2 173 dequeue_op 0x124e3d00 prio 127 cost 516903 latency 0.001027 osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2773 (0'0,173'2773] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2772 active]
2014-06-12 14:46:00.561392 7f9d6abe6700  5 -- op tracker -- , seq: 747417, time: 2014-06-12 14:46:00.561387, event: reached_pg, op: osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.561403 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2773 (0'0,173'2773] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2772 active] handle_message: 0x124e3d00
2014-06-12 14:46:00.561418 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2773 (0'0,173'2773] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2772 active] sub_op_modify trans e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 v 173'2774 (transaction) 174
2014-06-12 14:46:00.561438 7f9d6abe6700  5 -- op tracker -- , seq: 747417, time: 2014-06-12 14:46:00.561434, event: started, op: osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.561456 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2773 (0'0,173'2773] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2772 active] append_log log((0'0,173'2773], crt=0'0) [173'2774 (173'2773) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144753 2014-06-12 08:45:02.938270]
2014-06-12 14:46:00.561484 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2774 (0'0,173'2774] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2772 active] add_log_entry 173'2774 (173'2773) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144753 2014-06-12 08:45:02.938270
2014-06-12 14:46:00.561508 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2774 (0'0,173'2774] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2772 active] append_log  adding 1 keys
2014-06-12 14:46:00.561545 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2774, trimmed: 
2014-06-12 14:46:00.561573 7f9d6abe6700  5 -- op tracker -- , seq: 747417, time: 2014-06-12 14:46:00.561568, event: started, op: osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.561590 7f9d6abe6700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.561595 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.561597 7f9d6abe6700  5 queue_op 0x50c7e00 seq 2511 osr(3.6d1 0x276df10) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.561606 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (1 active)
2014-06-12 14:46:00.561612 7f9d8016d700  5 _do_op 0x50c7e00 seq 2511 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:00.561626 7f9d8016d700 10 _do_transaction on 0x12021b30
2014-06-12 14:46:00.561633 7f9d8016d700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.561604 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2774 (0'0,173'2774] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2772 active] do_sub_op osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.561647 7f9d6abe6700 10 osd.2 173 dequeue_op 0x124e3d00 finish
2014-06-12 14:46:00.561652 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.561654 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.561737 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.561758 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.561799 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.561876 7f9d8016d700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.561882 7f9d8016d700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:00.561886 7f9d8016d700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.561899 7f9d8016d700 15 _write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 364544~516096
2014-06-12 14:46:00.561992 7f9d8016d700 10 lookup_strip_header done  cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.562130 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:00.563381 7f9d8016d700 10 _generic_write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 364544~516096 = 0
2014-06-12 14:46:00.563420 7f9d8016d700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.563432 7f9d8016d700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.563436 7f9d8016d700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.563443 7f9d8016d700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.563582 7f9d8016d700 20 set_header setting 17236 cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 parent seq 0
2014-06-12 14:46:00.563627 7f9d8016d700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:00.563638 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.563643 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:00.565643 7f9d8016d700 10 _do_op 0x50c7e00 seq 2511 r = 0, finisher 0x6688fe0 0
2014-06-12 14:46:00.565659 7f9d8016d700 10 _finish_op 0x50c7e00 seq 2511 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.565665 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:00.565672 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.565678 7f9d7715b700  5 -- op tracker -- , seq: 747417, time: 2014-06-12 14:46:00.565667, event: commit_sent, op: osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.565689 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2774 (0'0,173'2774] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2772 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:00.565719 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:00.565754 7f9d7795c700  5 -- op tracker -- , seq: 747417, time: 2014-06-12 14:46:00.565748, event: sub_op_applied, op: osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.565764 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2774 (0'0,173'2774] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2773 active] sub_op_modify_applied on 0x12021a40 op osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.565782 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2774 (0'0,173'2774] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2773 active] op_applied on primary on version 173'2774
2014-06-12 14:46:00.565793 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2774 (0'0,173'2774] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2773 active] op_applied on replica on version 173'2774
2014-06-12 14:46:00.565822 7f9d7795c700  5 -- op tracker -- , seq: 747417, time: 2014-06-12 14:46:00.565818, event: done, op: osd_sub_op(client.4315.0:144753 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2774 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.571598 7f9d5501e700  5 -- op tracker -- , seq: 747418, time: 2014-06-12 14:46:00.570600, event: header_read, op: osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.571608 7f9d5501e700  5 -- op tracker -- , seq: 747418, time: 2014-06-12 14:46:00.570601, event: throttled, op: osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.571623 7f9d5501e700  5 -- op tracker -- , seq: 747418, time: 2014-06-12 14:46:00.571567, event: all_read, op: osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.571629 7f9d5501e700  5 -- op tracker -- , seq: 747418, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.571632 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:00.571637 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:00.571643 7f9d5501e700 15 osd.2 173 enqueue_op 0x31d60d00 prio 127 cost 516903 latency 0.001043 osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.571668 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.571674 7f9d66bde700 10 osd.2 173 dequeue_op 0x31d60d00 prio 127 cost 516903 latency 0.001074 osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2774 (0'0,173'2774] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2773 active]
2014-06-12 14:46:00.571699 7f9d66bde700  5 -- op tracker -- , seq: 747418, time: 2014-06-12 14:46:00.571697, event: reached_pg, op: osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.571706 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2774 (0'0,173'2774] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2773 active] handle_message: 0x31d60d00
2014-06-12 14:46:00.571714 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2774 (0'0,173'2774] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2773 active] sub_op_modify trans e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 v 173'2775 (transaction) 174
2014-06-12 14:46:00.571724 7f9d66bde700  5 -- op tracker -- , seq: 747418, time: 2014-06-12 14:46:00.571722, event: started, op: osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.571736 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2774 (0'0,173'2774] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2773 active] append_log log((0'0,173'2774], crt=0'0) [173'2775 (173'2774) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144754 2014-06-12 08:45:02.948563]
2014-06-12 14:46:00.571756 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2775 (0'0,173'2775] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2773 active] add_log_entry 173'2775 (173'2774) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144754 2014-06-12 08:45:02.948563
2014-06-12 14:46:00.571772 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2775 (0'0,173'2775] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2773 active] append_log  adding 1 keys
2014-06-12 14:46:00.571796 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2775, trimmed: 
2014-06-12 14:46:00.571818 7f9d66bde700  5 -- op tracker -- , seq: 747418, time: 2014-06-12 14:46:00.571815, event: started, op: osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.571829 7f9d66bde700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.571832 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.571833 7f9d66bde700  5 queue_op 0x663f380 seq 2512 osr(3.6d1 0x276df10) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.571842 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (1 active)
2014-06-12 14:46:00.571848 7f9d8096e700  5 _do_op 0x663f380 seq 2512 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:00.571838 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2775 (0'0,173'2775] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2773 active] do_sub_op osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.571852 7f9d8096e700 10 _do_transaction on 0x277567f0
2014-06-12 14:46:00.571851 7f9d66bde700 10 osd.2 173 dequeue_op 0x31d60d00 finish
2014-06-12 14:46:00.571854 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.571857 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.571858 7f9d8096e700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.571958 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.571976 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.572012 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.572027 7f9d8096e700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.572032 7f9d8096e700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:00.572035 7f9d8096e700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.572043 7f9d8096e700 15 _write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 880640~516096
2014-06-12 14:46:00.572137 7f9d8096e700 10 lookup_strip_header done  cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.572291 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:00.573519 7f9d8096e700 10 _generic_write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 880640~516096 = 0
2014-06-12 14:46:00.573552 7f9d8096e700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.573561 7f9d8096e700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.573565 7f9d8096e700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.573570 7f9d8096e700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.573720 7f9d8096e700 20 set_header setting 17236 cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 parent seq 0
2014-06-12 14:46:00.573777 7f9d8096e700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:00.573788 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.573793 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:00.575474 7f9d8096e700 10 _do_op 0x663f380 seq 2512 r = 0, finisher 0x88cdfc0 0
2014-06-12 14:46:00.575487 7f9d8096e700 10 _finish_op 0x663f380 seq 2512 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.575492 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:00.575498 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.575508 7f9d7715b700  5 -- op tracker -- , seq: 747418, time: 2014-06-12 14:46:00.575497, event: commit_sent, op: osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.575518 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2775 (0'0,173'2775] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2773 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:00.575545 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:00.575579 7f9d7795c700  5 -- op tracker -- , seq: 747418, time: 2014-06-12 14:46:00.575574, event: sub_op_applied, op: osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.575589 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2775 (0'0,173'2775] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2774 active] sub_op_modify_applied on 0x27756700 op osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.575608 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2775 (0'0,173'2775] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2774 active] op_applied on primary on version 173'2775
2014-06-12 14:46:00.575619 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2775 (0'0,173'2775] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2774 active] op_applied on replica on version 173'2775
2014-06-12 14:46:00.575642 7f9d7795c700  5 -- op tracker -- , seq: 747418, time: 2014-06-12 14:46:00.575637, event: done, op: osd_sub_op(client.4315.0:144754 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2775 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.580632 7f9d5501e700  5 -- op tracker -- , seq: 747419, time: 2014-06-12 14:46:00.579293, event: header_read, op: osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.580647 7f9d5501e700  5 -- op tracker -- , seq: 747419, time: 2014-06-12 14:46:00.579294, event: throttled, op: osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.580656 7f9d5501e700  5 -- op tracker -- , seq: 747419, time: 2014-06-12 14:46:00.580591, event: all_read, op: osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.580665 7f9d5501e700  5 -- op tracker -- , seq: 747419, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.580671 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:00.580679 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:00.580688 7f9d5501e700 15 osd.2 173 enqueue_op 0x22b3b200 prio 127 cost 516903 latency 0.001394 osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.580721 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.580732 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x22b3b200 prio 127 cost 516903 latency 0.001439 osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2775 (0'0,173'2775] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2774 active]
2014-06-12 14:46:00.580760 7f9d6a3e5700  5 -- op tracker -- , seq: 747419, time: 2014-06-12 14:46:00.580755, event: reached_pg, op: osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.580769 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2775 (0'0,173'2775] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2774 active] handle_message: 0x22b3b200
2014-06-12 14:46:00.580786 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2775 (0'0,173'2775] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2774 active] sub_op_modify trans e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 v 173'2776 (transaction) 174
2014-06-12 14:46:00.580804 7f9d6a3e5700  5 -- op tracker -- , seq: 747419, time: 2014-06-12 14:46:00.580799, event: started, op: osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.580823 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2775 (0'0,173'2775] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2774 active] append_log log((0'0,173'2775], crt=0'0) [173'2776 (173'2775) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144755 2014-06-12 08:45:02.958309]
2014-06-12 14:46:00.580846 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2776 (0'0,173'2776] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2774 active] add_log_entry 173'2776 (173'2775) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144755 2014-06-12 08:45:02.958309
2014-06-12 14:46:00.580872 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2776 (0'0,173'2776] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2774 active] append_log  adding 1 keys
2014-06-12 14:46:00.580910 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2776, trimmed: 
2014-06-12 14:46:00.580934 7f9d6a3e5700  5 -- op tracker -- , seq: 747419, time: 2014-06-12 14:46:00.580929, event: started, op: osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.580944 7f9d6a3e5700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.580949 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.580950 7f9d6a3e5700  5 queue_op 0x61f3c20 seq 2513 osr(3.6d1 0x276df10) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.580963 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (1 active)
2014-06-12 14:46:00.580970 7f9d7915f700  5 _do_op 0x61f3c20 seq 2513 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:00.580958 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2776 (0'0,173'2776] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2774 active] do_sub_op osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.580974 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x22b3b200 finish
2014-06-12 14:46:00.580975 7f9d7915f700 10 _do_transaction on 0x173cab70
2014-06-12 14:46:00.580978 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.580981 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.580981 7f9d7915f700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.581105 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.581128 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.581182 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.581205 7f9d7915f700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.581213 7f9d7915f700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:00.581219 7f9d7915f700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.581233 7f9d7915f700 15 _write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 1396736~516096
2014-06-12 14:46:00.581380 7f9d7915f700 10 lookup_strip_header done  cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.581636 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:00.583558 7f9d7915f700 10 _generic_write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 1396736~516096 = 0
2014-06-12 14:46:00.583610 7f9d7915f700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.583627 7f9d7915f700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.583636 7f9d7915f700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.583644 7f9d7915f700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.583855 7f9d7915f700 20 set_header setting 17236 cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 parent seq 0
2014-06-12 14:46:00.583920 7f9d7915f700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:00.583937 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.583949 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:00.586950 7f9d7915f700 10 _do_op 0x61f3c20 seq 2513 r = 0, finisher 0xa646c60 0
2014-06-12 14:46:00.586997 7f9d7715b700  5 -- op tracker -- , seq: 747419, time: 2014-06-12 14:46:00.586987, event: commit_sent, op: osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.587010 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2776 (0'0,173'2776] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2774 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:00.587047 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:00.587081 7f9d7915f700 10 _finish_op 0x61f3c20 seq 2513 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.587103 7f9d7795c700  5 -- op tracker -- , seq: 747419, time: 2014-06-12 14:46:00.587098, event: sub_op_applied, op: osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.587113 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2776 (0'0,173'2776] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2775 active] sub_op_modify_applied on 0x173caa80 op osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.587130 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2776 (0'0,173'2776] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2775 active] op_applied on primary on version 173'2776
2014-06-12 14:46:00.587142 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2776 (0'0,173'2776] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2775 active] op_applied on replica on version 173'2776
2014-06-12 14:46:00.587173 7f9d7915f700  5 -- op tracker -- , seq: 747419, time: 2014-06-12 14:46:00.587164, event: done, op: osd_sub_op(client.4315.0:144755 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2776 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.587201 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:00.587207 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.592044 7f9d5501e700  5 -- op tracker -- , seq: 747420, time: 2014-06-12 14:46:00.590982, event: header_read, op: osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.592057 7f9d5501e700  5 -- op tracker -- , seq: 747420, time: 2014-06-12 14:46:00.590983, event: throttled, op: osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.592066 7f9d5501e700  5 -- op tracker -- , seq: 747420, time: 2014-06-12 14:46:00.592003, event: all_read, op: osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.592075 7f9d5501e700  5 -- op tracker -- , seq: 747420, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.592081 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:00.592088 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:00.592095 7f9d5501e700 15 osd.2 173 enqueue_op 0x3b42a800 prio 127 cost 516903 latency 0.001113 osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.592153 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.592164 7f9d63bd8700 10 osd.2 173 dequeue_op 0x3b42a800 prio 127 cost 516903 latency 0.001181 osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2776 (0'0,173'2776] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2775 active]
2014-06-12 14:46:00.592181 7f9d63bd8700  5 -- op tracker -- , seq: 747420, time: 2014-06-12 14:46:00.592178, event: reached_pg, op: osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.592188 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2776 (0'0,173'2776] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2775 active] handle_message: 0x3b42a800
2014-06-12 14:46:00.592195 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2776 (0'0,173'2776] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2775 active] sub_op_modify trans e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 v 173'2777 (transaction) 174
2014-06-12 14:46:00.592205 7f9d63bd8700  5 -- op tracker -- , seq: 747420, time: 2014-06-12 14:46:00.592203, event: started, op: osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.592216 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2776 (0'0,173'2776] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2775 active] append_log log((0'0,173'2776], crt=0'0) [173'2777 (173'2776) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144756 2014-06-12 08:45:02.969887]
2014-06-12 14:46:00.592234 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2775 active] add_log_entry 173'2777 (173'2776) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144756 2014-06-12 08:45:02.969887
2014-06-12 14:46:00.592248 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2775 active] append_log  adding 1 keys
2014-06-12 14:46:00.592273 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2777, trimmed: 
2014-06-12 14:46:00.592289 7f9d63bd8700  5 -- op tracker -- , seq: 747420, time: 2014-06-12 14:46:00.592286, event: started, op: osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.592296 7f9d63bd8700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.592298 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.592299 7f9d63bd8700  5 queue_op 0x69cd140 seq 2514 osr(3.6d1 0x276df10) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.592306 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2775 active] do_sub_op osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.592315 7f9d63bd8700 10 osd.2 173 dequeue_op 0x3b42a800 finish
2014-06-12 14:46:00.592313 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (1 active)
2014-06-12 14:46:00.592318 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.592320 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.592321 7f9d7c165700  5 _do_op 0x69cd140 seq 2514 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:00.592326 7f9d7c165700 10 _do_transaction on 0x594bb30
2014-06-12 14:46:00.592333 7f9d7c165700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.592462 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.592487 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.592539 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.592650 7f9d7c165700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.592660 7f9d7c165700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:00.592667 7f9d7c165700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.592687 7f9d7c165700 15 _write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 1912832~516096
2014-06-12 14:46:00.592838 7f9d7c165700 10 lookup_strip_header done  cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.592989 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:00.594831 7f9d7c165700 10 _generic_write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 1912832~516096 = 0
2014-06-12 14:46:00.594894 7f9d7c165700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.594909 7f9d7c165700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.594915 7f9d7c165700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.594923 7f9d7c165700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.595132 7f9d7c165700 20 set_header setting 17236 cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 parent seq 0
2014-06-12 14:46:00.595211 7f9d7c165700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:00.595225 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.595234 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:00.597566 7f9d7c165700 10 _do_op 0x69cd140 seq 2514 r = 0, finisher 0x1ddf05e0 0
2014-06-12 14:46:00.597580 7f9d7c165700 10 _finish_op 0x69cd140 seq 2514 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.597589 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:00.597598 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.597605 7f9d7715b700  5 -- op tracker -- , seq: 747420, time: 2014-06-12 14:46:00.597594, event: commit_sent, op: osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.597651 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2775 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:00.597681 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:00.597716 7f9d7795c700  5 -- op tracker -- , seq: 747420, time: 2014-06-12 14:46:00.597711, event: sub_op_applied, op: osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.597724 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2776 active] sub_op_modify_applied on 0x594ba40 op osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.597745 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2776 active] op_applied on primary on version 173'2777
2014-06-12 14:46:00.597755 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2776 active] op_applied on replica on version 173'2777
2014-06-12 14:46:00.597778 7f9d7795c700  5 -- op tracker -- , seq: 747420, time: 2014-06-12 14:46:00.597774, event: done, op: osd_sub_op(client.4315.0:144756 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2777 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.607283 7f9d5501e700  5 -- op tracker -- , seq: 747421, time: 2014-06-12 14:46:00.606448, event: header_read, op: osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.607302 7f9d5501e700  5 -- op tracker -- , seq: 747421, time: 2014-06-12 14:46:00.606450, event: throttled, op: osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.607310 7f9d5501e700  5 -- op tracker -- , seq: 747421, time: 2014-06-12 14:46:00.607233, event: all_read, op: osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.607318 7f9d5501e700  5 -- op tracker -- , seq: 747421, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.607324 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:00.607333 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:00.607342 7f9d5501e700 15 osd.2 173 enqueue_op 0x17c2a400 prio 127 cost 516903 latency 0.000893 osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.607377 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.607385 7f9d653db700 10 osd.2 173 dequeue_op 0x17c2a400 prio 127 cost 516903 latency 0.000937 osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2776 active]
2014-06-12 14:46:00.607411 7f9d653db700  5 -- op tracker -- , seq: 747421, time: 2014-06-12 14:46:00.607406, event: reached_pg, op: osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.607420 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2776 active] handle_message: 0x17c2a400
2014-06-12 14:46:00.607433 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2776 active] sub_op_modify trans e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 v 173'2778 (transaction) 174
2014-06-12 14:46:00.607451 7f9d653db700  5 -- op tracker -- , seq: 747421, time: 2014-06-12 14:46:00.607447, event: started, op: osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.607474 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2776 active] append_log log((0'0,173'2777], crt=0'0) [173'2778 (173'2777) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144757 2014-06-12 08:45:02.980880]
2014-06-12 14:46:00.607498 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2778 (0'0,173'2778] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2776 active] add_log_entry 173'2778 (173'2777) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144757 2014-06-12 08:45:02.980880
2014-06-12 14:46:00.607523 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2778 (0'0,173'2778] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2776 active] append_log  adding 1 keys
2014-06-12 14:46:00.607561 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2778, trimmed: 
2014-06-12 14:46:00.607588 7f9d653db700  5 -- op tracker -- , seq: 747421, time: 2014-06-12 14:46:00.607583, event: started, op: osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.607598 7f9d653db700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.607603 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.607605 7f9d653db700  5 queue_op 0x7c6b1a0 seq 2515 osr(3.6d1 0x276df10) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.607616 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (1 active)
2014-06-12 14:46:00.607625 7f9d83173700  5 _do_op 0x7c6b1a0 seq 2515 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:00.607612 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2778 (0'0,173'2778] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2776 active] do_sub_op osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.607630 7f9d83173700 10 _do_transaction on 0x139c5970
2014-06-12 14:46:00.607629 7f9d653db700 10 osd.2 173 dequeue_op 0x17c2a400 finish
2014-06-12 14:46:00.607633 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.607636 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.607637 7f9d83173700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.607760 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.607783 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.607837 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.607855 7f9d83173700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.607864 7f9d83173700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:00.607870 7f9d83173700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.607884 7f9d83173700 15 _write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 2428928~516096
2014-06-12 14:46:00.608028 7f9d83173700 10 lookup_strip_header done  cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.608049 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:00.609981 7f9d83173700 10 _generic_write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 2428928~516096 = 0
2014-06-12 14:46:00.610029 7f9d83173700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.610049 7f9d83173700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.610058 7f9d83173700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.610066 7f9d83173700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.610271 7f9d83173700 20 set_header setting 17236 cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 parent seq 0
2014-06-12 14:46:00.610337 7f9d83173700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:00.610352 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.610364 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:00.612992 7f9d83173700 10 _do_op 0x7c6b1a0 seq 2515 r = 0, finisher 0xa646460 0
2014-06-12 14:46:00.613015 7f9d83173700 10 _finish_op 0x7c6b1a0 seq 2515 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.613024 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:00.613028 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.613033 7f9d7715b700  5 -- op tracker -- , seq: 747421, time: 2014-06-12 14:46:00.613022, event: commit_sent, op: osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.613042 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2778 (0'0,173'2778] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2776 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:00.613071 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:00.613140 7f9d7795c700  5 -- op tracker -- , seq: 747421, time: 2014-06-12 14:46:00.613129, event: sub_op_applied, op: osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.613148 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2778 (0'0,173'2778] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active] sub_op_modify_applied on 0x139c5880 op osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.613168 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2778 (0'0,173'2778] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active] op_applied on primary on version 173'2778
2014-06-12 14:46:00.613179 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2778 (0'0,173'2778] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active] op_applied on replica on version 173'2778
2014-06-12 14:46:00.613211 7f9d7795c700  5 -- op tracker -- , seq: 747421, time: 2014-06-12 14:46:00.613207, event: done, op: osd_sub_op(client.4315.0:144757 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2778 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.620849 7f9d5501e700  5 -- op tracker -- , seq: 747422, time: 2014-06-12 14:46:00.619898, event: header_read, op: osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.620866 7f9d5501e700  5 -- op tracker -- , seq: 747422, time: 2014-06-12 14:46:00.619901, event: throttled, op: osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.620875 7f9d5501e700  5 -- op tracker -- , seq: 747422, time: 2014-06-12 14:46:00.620787, event: all_read, op: osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.620883 7f9d5501e700  5 -- op tracker -- , seq: 747422, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.620889 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:00.620897 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:00.620906 7f9d5501e700 15 osd.2 173 enqueue_op 0xe20ad00 prio 127 cost 516903 latency 0.001008 osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.620941 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.620950 7f9d69be4700 10 osd.2 173 dequeue_op 0xe20ad00 prio 127 cost 516903 latency 0.001052 osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2778 (0'0,173'2778] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active]
2014-06-12 14:46:00.620976 7f9d69be4700  5 -- op tracker -- , seq: 747422, time: 2014-06-12 14:46:00.620972, event: reached_pg, op: osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.620985 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2778 (0'0,173'2778] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active] handle_message: 0xe20ad00
2014-06-12 14:46:00.621000 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2778 (0'0,173'2778] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active] sub_op_modify trans e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 v 173'2779 (transaction) 174
2014-06-12 14:46:00.621019 7f9d69be4700  5 -- op tracker -- , seq: 747422, time: 2014-06-12 14:46:00.621015, event: started, op: osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.621035 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2778 (0'0,173'2778] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active] append_log log((0'0,173'2778], crt=0'0) [173'2779 (173'2778) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144758 2014-06-12 08:45:02.996010]
2014-06-12 14:46:00.621062 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2779 (0'0,173'2779] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active] add_log_entry 173'2779 (173'2778) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144758 2014-06-12 08:45:02.996010
2014-06-12 14:46:00.621087 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2779 (0'0,173'2779] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active] append_log  adding 1 keys
2014-06-12 14:46:00.621127 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2779, trimmed: 
2014-06-12 14:46:00.621155 7f9d69be4700  5 -- op tracker -- , seq: 747422, time: 2014-06-12 14:46:00.621150, event: started, op: osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.621167 7f9d69be4700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.621172 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.621174 7f9d69be4700  5 queue_op 0x6ec73e0 seq 2516 osr(3.6d1 0x276df10) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.621187 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (1 active)
2014-06-12 14:46:00.621182 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2779 (0'0,173'2779] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active] do_sub_op osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.621196 7f9d82972700  5 _do_op 0x6ec73e0 seq 2516 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:00.621197 7f9d69be4700 10 osd.2 173 dequeue_op 0xe20ad00 finish
2014-06-12 14:46:00.621200 7f9d82972700 10 _do_transaction on 0x61a5430
2014-06-12 14:46:00.621201 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.621204 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.621207 7f9d82972700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.626640 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.626684 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.626735 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.626757 7f9d82972700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.626764 7f9d82972700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:00.626772 7f9d82972700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.626964 7f9d82972700 15 _write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 2945024~516096
2014-06-12 14:46:00.627067 7f9d82972700 10 lookup_strip_header done  cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.627231 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:00.628511 7f9d82972700 10 _generic_write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 2945024~516096 = 0
2014-06-12 14:46:00.628543 7f9d82972700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.628552 7f9d82972700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.628556 7f9d82972700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.628560 7f9d82972700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.628697 7f9d82972700 20 set_header setting 17236 cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 parent seq 0
2014-06-12 14:46:00.628743 7f9d82972700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:00.628754 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.628759 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:00.630439 7f9d82972700 10 _do_op 0x6ec73e0 seq 2516 r = 0, finisher 0x62ccc60 0
2014-06-12 14:46:00.630454 7f9d82972700 10 _finish_op 0x6ec73e0 seq 2516 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.630461 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:00.630472 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.630484 7f9d7795c700  5 -- op tracker -- , seq: 747422, time: 2014-06-12 14:46:00.630473, event: sub_op_applied, op: osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.630493 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2779 (0'0,173'2779] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active] sub_op_modify_applied on 0x61a5340 op osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.630522 7f9d7795c700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:00.630538 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2779 (0'0,173'2779] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active] op_applied on primary on version 173'2779
2014-06-12 14:46:00.630550 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2779 (0'0,173'2779] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active] op_applied on replica on version 173'2779
2014-06-12 14:46:00.630587 7f9d7715b700  5 -- op tracker -- , seq: 747422, time: 2014-06-12 14:46:00.630582, event: commit_sent, op: osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.630595 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2779 (0'0,173'2779] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2777 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:00.630606 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:00.630628 7f9d7715b700  5 -- op tracker -- , seq: 747422, time: 2014-06-12 14:46:00.630623, event: done, op: osd_sub_op(client.4315.0:144758 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2779 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.636946 7f9d5501e700  5 -- op tracker -- , seq: 747423, time: 2014-06-12 14:46:00.635844, event: header_read, op: osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.636957 7f9d5501e700  5 -- op tracker -- , seq: 747423, time: 2014-06-12 14:46:00.635846, event: throttled, op: osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.636972 7f9d5501e700  5 -- op tracker -- , seq: 747423, time: 2014-06-12 14:46:00.636916, event: all_read, op: osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.636977 7f9d5501e700  5 -- op tracker -- , seq: 747423, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.636981 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:00.636985 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:00.636991 7f9d5501e700 15 osd.2 173 enqueue_op 0xe20b800 prio 127 cost 516903 latency 0.001146 osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.637053 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.637065 7f9d633d7700 10 osd.2 173 dequeue_op 0xe20b800 prio 127 cost 516903 latency 0.001220 osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2779 (0'0,173'2779] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2778 active]
2014-06-12 14:46:00.637091 7f9d633d7700  5 -- op tracker -- , seq: 747423, time: 2014-06-12 14:46:00.637086, event: reached_pg, op: osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.637100 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2779 (0'0,173'2779] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2778 active] handle_message: 0xe20b800
2014-06-12 14:46:00.637115 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2779 (0'0,173'2779] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2778 active] sub_op_modify trans e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 v 173'2780 (transaction) 174
2014-06-12 14:46:00.637133 7f9d633d7700  5 -- op tracker -- , seq: 747423, time: 2014-06-12 14:46:00.637129, event: started, op: osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.637152 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2779 (0'0,173'2779] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2778 active] append_log log((0'0,173'2779], crt=0'0) [173'2780 (173'2779) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144759 2014-06-12 08:45:03.013581]
2014-06-12 14:46:00.637179 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2780 (0'0,173'2780] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2778 active] add_log_entry 173'2780 (173'2779) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144759 2014-06-12 08:45:03.013581
2014-06-12 14:46:00.637204 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2780 (0'0,173'2780] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2778 active] append_log  adding 1 keys
2014-06-12 14:46:00.637242 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2780, trimmed: 
2014-06-12 14:46:00.637271 7f9d633d7700  5 -- op tracker -- , seq: 747423, time: 2014-06-12 14:46:00.637266, event: started, op: osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.637283 7f9d633d7700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.637288 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.637289 7f9d633d7700  5 queue_op 0x58da120 seq 2517 osr(3.6d1 0x276df10) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.637297 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2780 (0'0,173'2780] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2778 active] do_sub_op osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.637313 7f9d633d7700 10 osd.2 173 dequeue_op 0xe20b800 finish
2014-06-12 14:46:00.637317 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.637320 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.637327 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (1 active)
2014-06-12 14:46:00.637336 7f9d7c966700  5 _do_op 0x58da120 seq 2517 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:00.637340 7f9d7c966700 10 _do_transaction on 0x609f0b0
2014-06-12 14:46:00.637347 7f9d7c966700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.637467 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.637496 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.637548 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.637575 7f9d7c966700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.637583 7f9d7c966700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:00.637589 7f9d7c966700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.637699 7f9d7c966700 15 _write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 3461120~516096
2014-06-12 14:46:00.637867 7f9d7c966700 10 lookup_strip_header done  cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.637995 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:00.639693 7f9d7c966700 10 _generic_write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 3461120~516096 = 0
2014-06-12 14:46:00.639747 7f9d7c966700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.639768 7f9d7c966700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.639782 7f9d7c966700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.639790 7f9d7c966700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.639997 7f9d7c966700 20 set_header setting 17236 cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 parent seq 0
2014-06-12 14:46:00.640074 7f9d7c966700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:00.640089 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.640105 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:00.642998 7f9d7c966700 10 _do_op 0x58da120 seq 2517 r = 0, finisher 0x6d62f40 0
2014-06-12 14:46:00.643014 7f9d7c966700 10 _finish_op 0x58da120 seq 2517 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.643029 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:00.643038 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.643058 7f9d7715b700  5 -- op tracker -- , seq: 747423, time: 2014-06-12 14:46:00.643045, event: commit_sent, op: osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.643079 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2780 (0'0,173'2780] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2778 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:00.643111 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:00.643147 7f9d7795c700  5 -- op tracker -- , seq: 747423, time: 2014-06-12 14:46:00.643140, event: sub_op_applied, op: osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.643159 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2780 (0'0,173'2780] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2779 active] sub_op_modify_applied on 0x609efc0 op osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.643176 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2780 (0'0,173'2780] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2779 active] op_applied on primary on version 173'2780
2014-06-12 14:46:00.643186 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2780 (0'0,173'2780] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2779 active] op_applied on replica on version 173'2780
2014-06-12 14:46:00.643212 7f9d7795c700  5 -- op tracker -- , seq: 747423, time: 2014-06-12 14:46:00.643208, event: done, op: osd_sub_op(client.4315.0:144759 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2780 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.646213 7f9d44000700  5 -- op tracker -- , seq: 747424, time: 2014-06-12 14:46:00.645539, event: header_read, op: osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.646227 7f9d44000700  5 -- op tracker -- , seq: 747424, time: 2014-06-12 14:46:00.645542, event: throttled, op: osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.646243 7f9d44000700  5 -- op tracker -- , seq: 747424, time: 2014-06-12 14:46:00.646181, event: all_read, op: osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.646253 7f9d44000700  5 -- op tracker -- , seq: 747424, time: 0.000000, event: dispatched, op: osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.646261 7f9d44000700 20 osd.2 173 should_share_map client.4315 172.16.96.220:0/1244608 173
2014-06-12 14:46:00.646289 7f9d44000700 15 osd.2 173 enqueue_op 0x3850f500 prio 63 cost 299008 latency 0.000750 osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.646319 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.646327 7f9d623d5700 10 osd.2 173 dequeue_op 0x3850f500 prio 63 cost 299008 latency 0.000788 osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean]
2014-06-12 14:46:00.646347 7f9d623d5700  5 -- op tracker -- , seq: 747424, time: 2014-06-12 14:46:00.646343, event: reached_pg, op: osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.646356 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:00.646367 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] handle_message: 0x3850f500
2014-06-12 14:46:00.646373 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] do_op osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:00.646394 7f9d623d5700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_'
2014-06-12 14:46:00.647187 7f9d5501e700  5 -- op tracker -- , seq: 747425, time: 2014-06-12 14:46:00.646768, event: header_read, op: osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.647199 7f9d5501e700  5 -- op tracker -- , seq: 747425, time: 2014-06-12 14:46:00.646770, event: throttled, op: osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.647208 7f9d5501e700  5 -- op tracker -- , seq: 747425, time: 2014-06-12 14:46:00.647166, event: all_read, op: osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.647225 7f9d5501e700  5 -- op tracker -- , seq: 747425, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.647230 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:00.647237 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:00.647243 7f9d5501e700 15 osd.2 173 enqueue_op 0xe20a100 prio 127 cost 217895 latency 0.000475 osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.647265 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:00.647269 7f9d643d9700 10 osd.2 173 dequeue_op 0xe20a100 prio 127 cost 217895 latency 0.000501 osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2780 (0'0,173'2780] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2779 active]
2014-06-12 14:46:00.647286 7f9d643d9700  5 -- op tracker -- , seq: 747425, time: 2014-06-12 14:46:00.647283, event: reached_pg, op: osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.647299 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2780 (0'0,173'2780] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2779 active] handle_message: 0xe20a100
2014-06-12 14:46:00.647307 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2780 (0'0,173'2780] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2779 active] sub_op_modify trans e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 v 173'2781 (transaction) 174
2014-06-12 14:46:00.647318 7f9d643d9700  5 -- op tracker -- , seq: 747425, time: 2014-06-12 14:46:00.647315, event: started, op: osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.647330 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2780 (0'0,173'2780] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2779 active] append_log log((0'0,173'2780], crt=0'0) [173'2781 (173'2780) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144760 2014-06-12 08:45:03.026028]
2014-06-12 14:46:00.647347 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2781 (0'0,173'2781] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2779 active] add_log_entry 173'2781 (173'2780) modify   e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 by client.4315.0:144760 2014-06-12 08:45:03.026028
2014-06-12 14:46:00.647362 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2781 (0'0,173'2781] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2779 active] append_log  adding 1 keys
2014-06-12 14:46:00.647387 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2781, trimmed: 
2014-06-12 14:46:00.647404 7f9d643d9700  5 -- op tracker -- , seq: 747425, time: 2014-06-12 14:46:00.647400, event: started, op: osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.647411 7f9d643d9700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.647414 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.647415 7f9d643d9700  5 queue_op 0x2272d20 seq 2518 osr(3.6d1 0x276df10) 219381 bytes   (queue has 1 ops and 219381 bytes)
2014-06-12 14:46:00.647423 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (1 active)
2014-06-12 14:46:00.647420 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2781 (0'0,173'2781] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2779 active] do_sub_op osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.647432 7f9d643d9700 10 osd.2 173 dequeue_op 0xe20a100 finish
2014-06-12 14:46:00.647432 7f9d81970700  5 _do_op 0x2272d20 seq 2518 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:00.647436 7f9d81970700 10 _do_transaction on 0x508a9b0
2014-06-12 14:46:00.647435 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:00.647437 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.647441 7f9d81970700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.647507 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.647524 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.647558 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.647571 7f9d81970700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.647576 7f9d81970700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:00.647579 7f9d81970700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:00.647588 7f9d81970700 15 _write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 3977216~217088
2014-06-12 14:46:00.647679 7f9d81970700 10 lookup_strip_header done  cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.647772 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:00.648274 7f9d81970700 10 _generic_write 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 3977216~217088 = 0
2014-06-12 14:46:00.648295 7f9d81970700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.648303 7f9d81970700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.648307 7f9d81970700 15 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3
2014-06-12 14:46:00.648311 7f9d81970700 10 _setattrs 3.6d1_head/e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 = 0
2014-06-12 14:46:00.648401 7f9d81970700 20 set_header setting 17236 cid 3.6d1_head oid e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 parent seq 0
2014-06-12 14:46:00.648424 7f9d81970700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:00.648433 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.648441 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:00.649209 7f9d81970700 10 _do_op 0x2272d20 seq 2518 r = 0, finisher 0x3d19d00 0
2014-06-12 14:46:00.649218 7f9d81970700 10 _finish_op 0x2272d20 seq 2518 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:00.649221 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:00.649223 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.649231 7f9d7715b700  5 -- op tracker -- , seq: 747425, time: 2014-06-12 14:46:00.649224, event: commit_sent, op: osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.649247 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2781 (0'0,173'2781] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2779 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:00.649271 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:00.649305 7f9d7795c700  5 -- op tracker -- , seq: 747425, time: 2014-06-12 14:46:00.649298, event: sub_op_applied, op: osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.649316 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2781 (0'0,173'2781] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2780 active] sub_op_modify_applied on 0x508a8c0 op osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:00.649334 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2781 (0'0,173'2781] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2780 active] op_applied on primary on version 173'2781
2014-06-12 14:46:00.649346 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2781 (0'0,173'2781] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2780 active] op_applied on replica on version 173'2781
2014-06-12 14:46:00.649371 7f9d7795c700  5 -- op tracker -- , seq: 747425, time: 2014-06-12 14:46:00.649366, event: done, op: osd_sub_op(client.4315.0:144760 3.6d1 e5119ed1/rbd_data.10b26b8b4567.000000000000106d/head//3 [] v 173'2781 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:00.656490 7f9d623d5700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_' = 0
2014-06-12 14:46:00.656520 7f9d623d5700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset'
2014-06-12 14:46:00.656651 7f9d623d5700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset' = 0
2014-06-12 14:46:00.656660 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] populate_obc_watchers 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.656679 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.656686 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] get_object_context: creating obc from disk: 0x128c2dc0
2014-06-12 14:46:00.656692 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] get_object_context: 0x128c2dc0 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 rwstate(none n=0 w=0) oi: 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'768 client.4315.0:38879 wrlock_by=unknown.0.0:0 dirty s 4194304 uv768) ssc: 0x1550efc0 snapset: 0=[]:[]+head
2014-06-12 14:46:00.656703 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] find_object_context 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 @head oi=9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'768 client.4315.0:38879 wrlock_by=unknown.0.0:0 dirty s 4194304 uv768)
2014-06-12 14:46:00.656724 7f9d623d5700  5 -- op tracker -- , seq: 747424, time: 2014-06-12 14:46:00.656718, event: started, op: osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.656736 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] execute_ctx 0x26f9f000
2014-06-12 14:46:00.656746 7f9d623d5700  5 -- op tracker -- , seq: 747424, time: 2014-06-12 14:46:00.656743, event: started, op: osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.656751 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] do_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] ov 173'768 av 173'2845 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:00.656760 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] do_osd_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008]
2014-06-12 14:46:00.656766 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:00.656774 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] do_osd_op  write 0~299008
2014-06-12 14:46:00.656785 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] do_osd_op_effects on session 0xdd07800
2014-06-12 14:46:00.656791 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 snapset=0x1550eff8  snapc=0=[]
2014-06-12 14:46:00.656798 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:00.656805 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] finish_ctx 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 0x26f9f000 op modify  
2014-06-12 14:46:00.656813 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean]  set mtime to 2014-06-12 08:45:03.026221
2014-06-12 14:46:00.656826 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean]  final snapset 0=[]:[]+head in 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.656840 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean]  zeroing write result code 0
2014-06-12 14:46:00.656846 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] new_repop rep_tid 236165 on osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.656862 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] issue_repop rep_tid 236165 o 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.656878 7f9d623d5700  5 -- op tracker -- , seq: 747424, time: 2014-06-12 14:46:00.656875, event: waiting for subops from 25, op: osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.656897 7f9d623d5700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:00.656914 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2844 (0'0,173'2844] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] append_log log((0'0,173'2844], crt=0'0) [173'2845 (173'768) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144761 2014-06-12 08:45:03.026221]
2014-06-12 14:46:00.656938 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2844 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] add_log_entry 173'2845 (173'768) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144761 2014-06-12 08:45:03.026221
2014-06-12 14:46:00.656960 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2844 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] append_log  adding 1 keys
2014-06-12 14:46:00.656980 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2845, trimmed: 
2014-06-12 14:46:00.656994 7f9d623d5700  5 queue_transactions existing osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.656997 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.656998 7f9d623d5700  5 queue_op 0x92980c0 seq 2625 osr(3.15d 0x276cea0) 301301 bytes   (queue has 1 ops and 301301 bytes)
2014-06-12 14:46:00.657008 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9d40 (1 active)
2014-06-12 14:46:00.657014 7f9d7a161700  5 _do_op 0x92980c0 seq 2625 osr(3.15d 0x276cea0)/0x276cea0 start
2014-06-12 14:46:00.657003 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2844 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] eval_repop repgather(0x5214f00 173'2845 rep_tid=236165 committed?=0 applied?=0 lock=0 op=osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.657018 7f9d7a161700 10 _do_transaction on 0x6e529c0
2014-06-12 14:46:00.657021 7f9d623d5700 10 osd.2 173 dequeue_op 0x3850f500 finish
2014-06-12 14:46:00.657023 7f9d7a161700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.657025 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.657029 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.657211 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.657241 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.657289 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.657317 7f9d7a161700 15 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.657321 7f9d7a161700 10 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1 r = 0
2014-06-12 14:46:00.657324 7f9d7a161700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.657336 7f9d7a161700 15 _write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 0~299008
2014-06-12 14:46:00.657579 7f9d7a161700 10 lookup_strip_header done  cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.657719 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:00.658382 7f9d7a161700 10 _generic_write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 0~299008 = 0
2014-06-12 14:46:00.658406 7f9d7a161700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.658414 7f9d7a161700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.658418 7f9d7a161700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.658423 7f9d7a161700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.658530 7f9d7a161700 20 set_header setting 17237 cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 parent seq 0
2014-06-12 14:46:00.658564 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.658573 7f9d7a161700 20 set_header setting 1998 cid meta oid 1fe2b61b/pglog_3.15d/0//-1 parent seq 0
2014-06-12 14:46:00.658579 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:00.659426 7f9d7a161700 10 _do_op 0x92980c0 seq 2625 r = 0, finisher 0x87fa9c0 0x28eadd00
2014-06-12 14:46:00.659436 7f9d7a161700 10 _finish_op 0x92980c0 seq 2625 osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.659440 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9d40 (0 active)
2014-06-12 14:46:00.659442 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.659443 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2844 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] op_commit: 236165
2014-06-12 14:46:00.659471 7f9d7715b700  5 -- op tracker -- , seq: 747424, time: 2014-06-12 14:46:00.659464, event: op_commit, op: osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.659485 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2844 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] op_applied: 236165
2014-06-12 14:46:00.659512 7f9d7795c700  5 -- op tracker -- , seq: 747424, time: 2014-06-12 14:46:00.659505, event: op_applied, op: osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.659523 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2844 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] op_applied on primary on version 173'2845
2014-06-12 14:46:00.661889 7f9d5bec4700  5 -- op tracker -- , seq: 747426, time: 2014-06-12 14:46:00.661853, event: header_read, op: osd_sub_op_reply(client.4315.0:144761 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.661899 7f9d5bec4700  5 -- op tracker -- , seq: 747426, time: 2014-06-12 14:46:00.661854, event: throttled, op: osd_sub_op_reply(client.4315.0:144761 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.661912 7f9d5bec4700  5 -- op tracker -- , seq: 747426, time: 2014-06-12 14:46:00.661864, event: all_read, op: osd_sub_op_reply(client.4315.0:144761 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.661916 7f9d5bec4700  5 -- op tracker -- , seq: 747426, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4315.0:144761 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.661920 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4315.0:144761 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:00.661924 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:00.661929 7f9d5bec4700 15 osd.2 173 enqueue_op 0x9face00 prio 196 cost 0 latency 0.000076 osd_sub_op_reply(client.4315.0:144761 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:00.661991 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.662001 7f9d68be2700 10 osd.2 173 dequeue_op 0x9face00 prio 196 cost 0 latency 0.000148 osd_sub_op_reply(client.4315.0:144761 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 pg pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2844 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean]
2014-06-12 14:46:00.662017 7f9d68be2700  5 -- op tracker -- , seq: 747426, time: 2014-06-12 14:46:00.662014, event: reached_pg, op: osd_sub_op_reply(client.4315.0:144761 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.662033 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2844 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] handle_message: 0x9face00
2014-06-12 14:46:00.662044 7f9d68be2700  5 -- op tracker -- , seq: 747426, time: 2014-06-12 14:46:00.662042, event: started, op: osd_sub_op_reply(client.4315.0:144761 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.662049 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2844 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] sub_op_modify_reply: tid 236165 op  ack_type 4 from 25
2014-06-12 14:46:00.662062 7f9d68be2700  5 -- op tracker -- , seq: 747424, time: 2014-06-12 14:46:00.662057, event: sub_op_commit_rec, op: osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.662066 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2844 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] repop_all_applied: repop tid 236165 all applied 
2014-06-12 14:46:00.662073 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2844 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] eval_repop repgather(0x5214f00 173'2845 rep_tid=236165 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.662082 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2844 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean]  sending ack on repgather(0x5214f00 173'2845 rep_tid=236165 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x1285aa00
2014-06-12 14:46:00.662106 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2844 crt=0'0 lcod 173'2843 mlcod 173'2843 active+clean] repop_all_committed: repop tid 236165 all committed 
2014-06-12 14:46:00.662117 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2843 active+clean] eval_repop repgather(0x5214f00 173'2845 rep_tid=236165 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.662133 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2843 active+clean] log_op_stats osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 inb 299287 outb 0 rlat 0.016565 lat 0.016592
2014-06-12 14:46:00.662148 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2843 active+clean] publish_stats_to_osd 173:3708
2014-06-12 14:46:00.662159 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2843 active+clean]  sending commit on repgather(0x5214f00 173'2845 rep_tid=236165 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x38136780
2014-06-12 14:46:00.662182 7f9d68be2700  5 -- op tracker -- , seq: 747424, time: 2014-06-12 14:46:00.662177, event: commit_sent, op: osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.662192 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2843 active+clean]  requeue_ops 
2014-06-12 14:46:00.662205 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean]  removing repgather(0x5214f00 173'2845 rep_tid=236165 committed?=1 applied?=1 op=osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.662215 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean]    q front is repgather(0x5214f00 173'2845 rep_tid=236165 committed?=1 applied?=1 op=osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.662224 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] remove_repop repgather(0x5214f00 173'2845 rep_tid=236165 committed?=1 applied?=1 op=osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.662233 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean]  requeue_ops 
2014-06-12 14:46:00.662250 7f9d68be2700  5 -- op tracker -- , seq: 747424, time: 2014-06-12 14:46:00.662247, event: done, op: osd_op(client.4315.0:144761 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~299008] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.662265 7f9d68be2700 10 osd.2 173 dequeue_op 0x9face00 finish
2014-06-12 14:46:00.662271 7f9d68be2700  5 -- op tracker -- , seq: 747426, time: 2014-06-12 14:46:00.662269, event: done, op: osd_sub_op_reply(client.4315.0:144761 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.662277 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.662279 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.664752 7f9d44000700  5 -- op tracker -- , seq: 747427, time: 2014-06-12 14:46:00.663847, event: header_read, op: osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.664763 7f9d44000700  5 -- op tracker -- , seq: 747427, time: 2014-06-12 14:46:00.663849, event: throttled, op: osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.664774 7f9d44000700  5 -- op tracker -- , seq: 747427, time: 2014-06-12 14:46:00.664725, event: all_read, op: osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.664784 7f9d44000700  5 -- op tracker -- , seq: 747427, time: 0.000000, event: dispatched, op: osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.664790 7f9d44000700 20 osd.2 173 should_share_map client.4315 172.16.96.220:0/1244608 173
2014-06-12 14:46:00.664820 7f9d44000700 15 osd.2 173 enqueue_op 0x124e3700 prio 63 cost 516096 latency 0.000973 osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.664869 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.664878 7f9d61bd4700 10 osd.2 173 dequeue_op 0x124e3700 prio 63 cost 516096 latency 0.001031 osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean]
2014-06-12 14:46:00.664899 7f9d61bd4700  5 -- op tracker -- , seq: 747427, time: 2014-06-12 14:46:00.664894, event: reached_pg, op: osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.664908 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:00.664919 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] handle_message: 0x124e3700
2014-06-12 14:46:00.664928 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] do_op osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:00.664945 7f9d61bd4700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_'
2014-06-12 14:46:00.665081 7f9d61bd4700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_' = 0
2014-06-12 14:46:00.665094 7f9d61bd4700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset'
2014-06-12 14:46:00.665151 7f9d61bd4700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset' = 0
2014-06-12 14:46:00.665159 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] populate_obc_watchers 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.665168 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.665173 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] get_object_context: creating obc from disk: 0x120eab00
2014-06-12 14:46:00.665179 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] get_object_context: 0x120eab00 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 rwstate(none n=0 w=0) oi: 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2845 client.4315.0:144761 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2845) ssc: 0x407e380 snapset: 0=[]:[]+head
2014-06-12 14:46:00.665189 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] find_object_context 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 @head oi=9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2845 client.4315.0:144761 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2845)
2014-06-12 14:46:00.665206 7f9d61bd4700  5 -- op tracker -- , seq: 747427, time: 2014-06-12 14:46:00.665203, event: started, op: osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.665213 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] execute_ctx 0x7ad1e00
2014-06-12 14:46:00.665223 7f9d61bd4700  5 -- op tracker -- , seq: 747427, time: 2014-06-12 14:46:00.665220, event: started, op: osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.665228 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] do_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] ov 173'2845 av 173'2846 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:00.665237 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] do_osd_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096]
2014-06-12 14:46:00.665244 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:00.665252 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] do_osd_op  write 299008~516096
2014-06-12 14:46:00.665262 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] do_osd_op_effects on session 0xdd07800
2014-06-12 14:46:00.665270 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 snapset=0x407e3b8  snapc=0=[]
2014-06-12 14:46:00.665277 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:00.665284 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] finish_ctx 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 0x7ad1e00 op modify  
2014-06-12 14:46:00.665292 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean]  set mtime to 2014-06-12 08:45:03.044583
2014-06-12 14:46:00.665305 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean]  final snapset 0=[]:[]+head in 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.665317 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean]  zeroing write result code 0
2014-06-12 14:46:00.665325 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] new_repop rep_tid 236166 on osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.665342 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] issue_repop rep_tid 236166 o 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.665358 7f9d61bd4700  5 -- op tracker -- , seq: 747427, time: 2014-06-12 14:46:00.665355, event: waiting for subops from 25, op: osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.665372 7f9d61bd4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:00.665386 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2845 (0'0,173'2845] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] append_log log((0'0,173'2845], crt=0'0) [173'2846 (173'2845) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144762 2014-06-12 08:45:03.044583]
2014-06-12 14:46:00.665401 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2845 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] add_log_entry 173'2846 (173'2845) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144762 2014-06-12 08:45:03.044583
2014-06-12 14:46:00.665415 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2845 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] append_log  adding 1 keys
2014-06-12 14:46:00.665435 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2846, trimmed: 
2014-06-12 14:46:00.665450 7f9d61bd4700  5 queue_transactions existing osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.665454 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.665455 7f9d61bd4700  5 queue_op 0x149e4000 seq 2626 osr(3.15d 0x276cea0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.665460 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2845 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] eval_repop repgather(0x2ad99a40 173'2846 rep_tid=236166 committed?=0 applied?=0 lock=0 op=osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.665478 7f9d61bd4700 10 osd.2 173 dequeue_op 0x124e3700 finish
2014-06-12 14:46:00.665481 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.665483 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.665483 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9d40 (1 active)
2014-06-12 14:46:00.665489 7f9d7a962700  5 _do_op 0x149e4000 seq 2626 osr(3.15d 0x276cea0)/0x276cea0 start
2014-06-12 14:46:00.665492 7f9d7a962700 10 _do_transaction on 0x2ad98540
2014-06-12 14:46:00.665497 7f9d7a962700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.665571 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.665587 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.665638 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.665662 7f9d7a962700 15 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.665667 7f9d7a962700 10 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1 r = 0
2014-06-12 14:46:00.665671 7f9d7a962700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.665679 7f9d7a962700 15 _write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 299008~516096
2014-06-12 14:46:00.665759 7f9d7a962700 10 lookup_strip_header done  cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.665896 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:00.667096 7f9d7a962700 10 _generic_write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 299008~516096 = 0
2014-06-12 14:46:00.667129 7f9d7a962700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.667140 7f9d7a962700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.667146 7f9d7a962700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.667163 7f9d7a962700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.667297 7f9d7a962700 20 set_header setting 17237 cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 parent seq 0
2014-06-12 14:46:00.667341 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.667350 7f9d7a962700 20 set_header setting 1998 cid meta oid 1fe2b61b/pglog_3.15d/0//-1 parent seq 0
2014-06-12 14:46:00.667356 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:00.669199 7f9d7a962700 10 _do_op 0x149e4000 seq 2626 r = 0, finisher 0x31388840 0x42cdac0
2014-06-12 14:46:00.669213 7f9d7a962700 10 _finish_op 0x149e4000 seq 2626 osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.669219 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9d40 (0 active)
2014-06-12 14:46:00.669224 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.669219 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2845 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] op_commit: 236166
2014-06-12 14:46:00.669249 7f9d7715b700  5 -- op tracker -- , seq: 747427, time: 2014-06-12 14:46:00.669240, event: op_commit, op: osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.669266 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2845 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] op_applied: 236166
2014-06-12 14:46:00.669290 7f9d7795c700  5 -- op tracker -- , seq: 747427, time: 2014-06-12 14:46:00.669284, event: op_applied, op: osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.669302 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2845 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] op_applied on primary on version 173'2846
2014-06-12 14:46:00.676024 7f9d5bec4700  5 -- op tracker -- , seq: 747428, time: 2014-06-12 14:46:00.675980, event: header_read, op: osd_sub_op_reply(client.4315.0:144762 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.676033 7f9d5bec4700  5 -- op tracker -- , seq: 747428, time: 2014-06-12 14:46:00.675981, event: throttled, op: osd_sub_op_reply(client.4315.0:144762 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.676041 7f9d5bec4700  5 -- op tracker -- , seq: 747428, time: 2014-06-12 14:46:00.675993, event: all_read, op: osd_sub_op_reply(client.4315.0:144762 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.676053 7f9d5bec4700  5 -- op tracker -- , seq: 747428, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4315.0:144762 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.676057 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4315.0:144762 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:00.676061 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:00.676067 7f9d5bec4700 15 osd.2 173 enqueue_op 0x2a903c00 prio 196 cost 0 latency 0.000086 osd_sub_op_reply(client.4315.0:144762 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:00.676146 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.676168 7f9d673df700 10 osd.2 173 dequeue_op 0x2a903c00 prio 196 cost 0 latency 0.000187 osd_sub_op_reply(client.4315.0:144762 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 pg pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2845 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean]
2014-06-12 14:46:00.676201 7f9d673df700  5 -- op tracker -- , seq: 747428, time: 2014-06-12 14:46:00.676195, event: reached_pg, op: osd_sub_op_reply(client.4315.0:144762 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.676216 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2845 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] handle_message: 0x2a903c00
2014-06-12 14:46:00.676237 7f9d673df700  5 -- op tracker -- , seq: 747428, time: 2014-06-12 14:46:00.676232, event: started, op: osd_sub_op_reply(client.4315.0:144762 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.676266 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2845 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] sub_op_modify_reply: tid 236166 op  ack_type 4 from 25
2014-06-12 14:46:00.676293 7f9d673df700  5 -- op tracker -- , seq: 747427, time: 2014-06-12 14:46:00.676284, event: sub_op_commit_rec, op: osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.676313 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2845 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] repop_all_applied: repop tid 236166 all applied 
2014-06-12 14:46:00.676327 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2845 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] eval_repop repgather(0x2ad99a40 173'2846 rep_tid=236166 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.676349 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2845 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean]  sending ack on repgather(0x2ad99a40 173'2846 rep_tid=236166 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x17be8000
2014-06-12 14:46:00.676390 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2845 crt=0'0 lcod 173'2844 mlcod 173'2844 active+clean] repop_all_committed: repop tid 236166 all committed 
2014-06-12 14:46:00.676404 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2844 active+clean] eval_repop repgather(0x2ad99a40 173'2846 rep_tid=236166 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.676429 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2844 active+clean] log_op_stats osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.012542 lat 0.012581
2014-06-12 14:46:00.676456 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2844 active+clean] publish_stats_to_osd 173:3709
2014-06-12 14:46:00.676473 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2844 active+clean]  sending commit on repgather(0x2ad99a40 173'2846 rep_tid=236166 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x1d54e280
2014-06-12 14:46:00.676516 7f9d673df700  5 -- op tracker -- , seq: 747427, time: 2014-06-12 14:46:00.676509, event: commit_sent, op: osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.676526 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2844 active+clean]  requeue_ops 
2014-06-12 14:46:00.676540 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean]  removing repgather(0x2ad99a40 173'2846 rep_tid=236166 committed?=1 applied?=1 op=osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.676559 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean]    q front is repgather(0x2ad99a40 173'2846 rep_tid=236166 committed?=1 applied?=1 op=osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.676577 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] remove_repop repgather(0x2ad99a40 173'2846 rep_tid=236166 committed?=1 applied?=1 op=osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.676598 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean]  requeue_ops 
2014-06-12 14:46:00.676630 7f9d673df700  5 -- op tracker -- , seq: 747427, time: 2014-06-12 14:46:00.676623, event: done, op: osd_op(client.4315.0:144762 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 299008~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.676656 7f9d673df700 10 osd.2 173 dequeue_op 0x2a903c00 finish
2014-06-12 14:46:00.676669 7f9d673df700  5 -- op tracker -- , seq: 747428, time: 2014-06-12 14:46:00.676664, event: done, op: osd_sub_op_reply(client.4315.0:144762 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.676683 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.676687 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.678644 7f9d44000700  5 -- op tracker -- , seq: 747429, time: 2014-06-12 14:46:00.677556, event: header_read, op: osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.678677 7f9d44000700  5 -- op tracker -- , seq: 747429, time: 2014-06-12 14:46:00.677561, event: throttled, op: osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.678699 7f9d44000700  5 -- op tracker -- , seq: 747429, time: 2014-06-12 14:46:00.678589, event: all_read, op: osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.678742 7f9d44000700  5 -- op tracker -- , seq: 747429, time: 0.000000, event: dispatched, op: osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.678757 7f9d44000700 20 osd.2 173 should_share_map client.4315 172.16.96.220:0/1244608 173
2014-06-12 14:46:00.678808 7f9d44000700 15 osd.2 173 enqueue_op 0x124e3c00 prio 63 cost 516096 latency 0.001252 osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.678892 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.678912 7f9d663dd700 10 osd.2 173 dequeue_op 0x124e3c00 prio 63 cost 516096 latency 0.001356 osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean]
2014-06-12 14:46:00.678954 7f9d663dd700  5 -- op tracker -- , seq: 747429, time: 2014-06-12 14:46:00.678946, event: reached_pg, op: osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.678974 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:00.678995 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] handle_message: 0x124e3c00
2014-06-12 14:46:00.679016 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] do_op osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:00.679050 7f9d663dd700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_'
2014-06-12 14:46:00.679291 7f9d663dd700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_' = 0
2014-06-12 14:46:00.679317 7f9d663dd700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset'
2014-06-12 14:46:00.679447 7f9d663dd700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset' = 0
2014-06-12 14:46:00.679463 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] populate_obc_watchers 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.679482 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.679496 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] get_object_context: creating obc from disk: 0x152e4dc0
2014-06-12 14:46:00.679508 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] get_object_context: 0x152e4dc0 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 rwstate(none n=0 w=0) oi: 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2846 client.4315.0:144762 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2846) ssc: 0x407f6c0 snapset: 0=[]:[]+head
2014-06-12 14:46:00.679531 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] find_object_context 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 @head oi=9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2846 client.4315.0:144762 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2846)
2014-06-12 14:46:00.679565 7f9d663dd700  5 -- op tracker -- , seq: 747429, time: 2014-06-12 14:46:00.679557, event: started, op: osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.679579 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] execute_ctx 0x5d20600
2014-06-12 14:46:00.679602 7f9d663dd700  5 -- op tracker -- , seq: 747429, time: 2014-06-12 14:46:00.679596, event: started, op: osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.679613 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] do_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] ov 173'2846 av 173'2847 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:00.679634 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] do_osd_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096]
2014-06-12 14:46:00.679648 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:00.679666 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] do_osd_op  write 815104~516096
2014-06-12 14:46:00.679688 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] do_osd_op_effects on session 0xdd07800
2014-06-12 14:46:00.679705 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 snapset=0x407f6f8  snapc=0=[]
2014-06-12 14:46:00.679721 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:00.679736 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] finish_ctx 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 0x5d20600 op modify  
2014-06-12 14:46:00.679753 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean]  set mtime to 2014-06-12 08:45:03.058444
2014-06-12 14:46:00.679783 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean]  final snapset 0=[]:[]+head in 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.679809 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean]  zeroing write result code 0
2014-06-12 14:46:00.679825 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] new_repop rep_tid 236167 on osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.679848 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] issue_repop rep_tid 236167 o 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.679879 7f9d663dd700  5 -- op tracker -- , seq: 747429, time: 2014-06-12 14:46:00.679873, event: waiting for subops from 25, op: osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.679911 7f9d663dd700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:00.679941 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2846 (0'0,173'2846] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] append_log log((0'0,173'2846], crt=0'0) [173'2847 (173'2846) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144763 2014-06-12 08:45:03.058444]
2014-06-12 14:46:00.679975 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2846 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] add_log_entry 173'2847 (173'2846) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144763 2014-06-12 08:45:03.058444
2014-06-12 14:46:00.680007 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2846 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] append_log  adding 1 keys
2014-06-12 14:46:00.680052 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2847, trimmed: 
2014-06-12 14:46:00.680083 7f9d663dd700  5 queue_transactions existing osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.680090 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.680093 7f9d663dd700  5 queue_op 0xff15080 seq 2627 osr(3.15d 0x276cea0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.680103 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2846 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] eval_repop repgather(0x52f4240 173'2847 rep_tid=236167 committed?=0 applied?=0 lock=0 op=osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.680128 7f9d663dd700 10 osd.2 173 dequeue_op 0x124e3c00 finish
2014-06-12 14:46:00.680123 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9d40 (1 active)
2014-06-12 14:46:00.680133 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.680137 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.680138 7f9d7895e700  5 _do_op 0xff15080 seq 2627 osr(3.15d 0x276cea0)/0x276cea0 start
2014-06-12 14:46:00.680145 7f9d7895e700 10 _do_transaction on 0x65906c0
2014-06-12 14:46:00.680160 7f9d7895e700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.680298 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.680335 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.680411 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.680446 7f9d7895e700 15 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.680457 7f9d7895e700 10 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1 r = 0
2014-06-12 14:46:00.680465 7f9d7895e700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.680487 7f9d7895e700 15 _write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 815104~516096
2014-06-12 14:46:00.680668 7f9d7895e700 10 lookup_strip_header done  cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.680694 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:00.683072 7f9d7895e700 10 _generic_write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 815104~516096 = 0
2014-06-12 14:46:00.683139 7f9d7895e700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.683162 7f9d7895e700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.683173 7f9d7895e700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.683184 7f9d7895e700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.683465 7f9d7895e700 20 set_header setting 17237 cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 parent seq 0
2014-06-12 14:46:00.683559 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.683580 7f9d7895e700 20 set_header setting 1998 cid meta oid 1fe2b61b/pglog_3.15d/0//-1 parent seq 0
2014-06-12 14:46:00.683598 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:00.687562 7f9d7895e700 10 _do_op 0xff15080 seq 2627 r = 0, finisher 0x216fb1a0 0x111fb680
2014-06-12 14:46:00.687596 7f9d7895e700 10 _finish_op 0xff15080 seq 2627 osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.687608 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9d40 (0 active)
2014-06-12 14:46:00.687621 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.687609 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2846 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] op_commit: 236167
2014-06-12 14:46:00.687678 7f9d7715b700  5 -- op tracker -- , seq: 747429, time: 2014-06-12 14:46:00.687658, event: op_commit, op: osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.687718 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2846 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] op_applied: 236167
2014-06-12 14:46:00.687767 7f9d7795c700  5 -- op tracker -- , seq: 747429, time: 2014-06-12 14:46:00.687752, event: op_applied, op: osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.687795 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2846 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] op_applied on primary on version 173'2847
2014-06-12 14:46:00.687815 7f9d5bec4700  5 -- op tracker -- , seq: 747430, time: 2014-06-12 14:46:00.687746, event: header_read, op: osd_sub_op_reply(client.4315.0:144763 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.687827 7f9d5bec4700  5 -- op tracker -- , seq: 747430, time: 2014-06-12 14:46:00.687749, event: throttled, op: osd_sub_op_reply(client.4315.0:144763 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.687841 7f9d5bec4700  5 -- op tracker -- , seq: 747430, time: 2014-06-12 14:46:00.687781, event: all_read, op: osd_sub_op_reply(client.4315.0:144763 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.687858 7f9d5bec4700  5 -- op tracker -- , seq: 747430, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4315.0:144763 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.687866 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4315.0:144763 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:00.687880 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:00.687891 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e86b00 prio 196 cost 0 latency 0.000145 osd_sub_op_reply(client.4315.0:144763 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:00.687934 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.687954 7f9d67be0700 10 osd.2 173 dequeue_op 0x20e86b00 prio 196 cost 0 latency 0.000208 osd_sub_op_reply(client.4315.0:144763 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 pg pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2846 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean]
2014-06-12 14:46:00.688001 7f9d67be0700  5 -- op tracker -- , seq: 747430, time: 2014-06-12 14:46:00.687993, event: reached_pg, op: osd_sub_op_reply(client.4315.0:144763 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.688019 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2846 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] handle_message: 0x20e86b00
2014-06-12 14:46:00.688060 7f9d67be0700  5 -- op tracker -- , seq: 747430, time: 2014-06-12 14:46:00.688051, event: started, op: osd_sub_op_reply(client.4315.0:144763 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.688073 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2846 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] sub_op_modify_reply: tid 236167 op  ack_type 4 from 25
2014-06-12 14:46:00.688120 7f9d67be0700  5 -- op tracker -- , seq: 747429, time: 2014-06-12 14:46:00.688107, event: sub_op_commit_rec, op: osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.688141 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2846 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] repop_all_applied: repop tid 236167 all applied 
2014-06-12 14:46:00.688171 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2846 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] eval_repop repgather(0x52f4240 173'2847 rep_tid=236167 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.688213 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2846 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean]  sending ack on repgather(0x52f4240 173'2847 rep_tid=236167 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x1285aa00
2014-06-12 14:46:00.688286 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2846 crt=0'0 lcod 173'2845 mlcod 173'2845 active+clean] repop_all_committed: repop tid 236167 all committed 
2014-06-12 14:46:00.688306 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2845 active+clean] eval_repop repgather(0x52f4240 173'2847 rep_tid=236167 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.688347 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2845 active+clean] log_op_stats osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.010727 lat 0.010789
2014-06-12 14:46:00.688395 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2845 active+clean] publish_stats_to_osd 173:3710
2014-06-12 14:46:00.688428 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2845 active+clean]  sending commit on repgather(0x52f4240 173'2847 rep_tid=236167 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x6486280
2014-06-12 14:46:00.688503 7f9d67be0700  5 -- op tracker -- , seq: 747429, time: 2014-06-12 14:46:00.688492, event: commit_sent, op: osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.688520 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2845 active+clean]  requeue_ops 
2014-06-12 14:46:00.688544 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]  removing repgather(0x52f4240 173'2847 rep_tid=236167 committed?=1 applied?=1 op=osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.688579 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]    q front is repgather(0x52f4240 173'2847 rep_tid=236167 committed?=1 applied?=1 op=osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.688613 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] remove_repop repgather(0x52f4240 173'2847 rep_tid=236167 committed?=1 applied?=1 op=osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.688648 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]  requeue_ops 
2014-06-12 14:46:00.688702 7f9d67be0700  5 -- op tracker -- , seq: 747429, time: 2014-06-12 14:46:00.688689, event: done, op: osd_op(client.4315.0:144763 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 815104~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.688736 7f9d67be0700 10 osd.2 173 dequeue_op 0x20e86b00 finish
2014-06-12 14:46:00.688757 7f9d67be0700  5 -- op tracker -- , seq: 747430, time: 2014-06-12 14:46:00.688748, event: done, op: osd_sub_op_reply(client.4315.0:144763 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.688780 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.688787 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.689769 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:00.689930 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:00.691452 7f9d44000700  5 -- op tracker -- , seq: 747431, time: 2014-06-12 14:46:00.690466, event: header_read, op: osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.691464 7f9d44000700  5 -- op tracker -- , seq: 747431, time: 2014-06-12 14:46:00.690471, event: throttled, op: osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.691469 7f9d44000700  5 -- op tracker -- , seq: 747431, time: 2014-06-12 14:46:00.691421, event: all_read, op: osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.691475 7f9d44000700  5 -- op tracker -- , seq: 747431, time: 0.000000, event: dispatched, op: osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.691480 7f9d44000700 20 osd.2 173 should_share_map client.4315 172.16.96.220:0/1244608 173
2014-06-12 14:46:00.691503 7f9d44000700 15 osd.2 173 enqueue_op 0x25e42200 prio 63 cost 516096 latency 0.001037 osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.691529 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.691539 7f9d693e3700 10 osd.2 173 dequeue_op 0x25e42200 prio 63 cost 516096 latency 0.001073 osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]
2014-06-12 14:46:00.691568 7f9d693e3700  5 -- op tracker -- , seq: 747431, time: 2014-06-12 14:46:00.691562, event: reached_pg, op: osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.691580 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:00.691597 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] handle_message: 0x25e42200
2014-06-12 14:46:00.691609 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] do_op osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:00.691637 7f9d693e3700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_'
2014-06-12 14:46:00.691844 7f9d693e3700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_' = 0
2014-06-12 14:46:00.691863 7f9d693e3700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset'
2014-06-12 14:46:00.691958 7f9d693e3700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset' = 0
2014-06-12 14:46:00.691970 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] populate_obc_watchers 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.691984 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.691995 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] get_object_context: creating obc from disk: 0x70d9080
2014-06-12 14:46:00.692007 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] get_object_context: 0x70d9080 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 rwstate(none n=0 w=0) oi: 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2847 client.4315.0:144763 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2847) ssc: 0x72b9a40 snapset: 0=[]:[]+head
2014-06-12 14:46:00.692025 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] find_object_context 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 @head oi=9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2847 client.4315.0:144763 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2847)
2014-06-12 14:46:00.692053 7f9d693e3700  5 -- op tracker -- , seq: 747431, time: 2014-06-12 14:46:00.692047, event: started, op: osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.692063 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] execute_ctx 0x110c4a00
2014-06-12 14:46:00.692084 7f9d693e3700  5 -- op tracker -- , seq: 747431, time: 2014-06-12 14:46:00.692079, event: started, op: osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.692093 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] do_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] ov 173'2847 av 173'2848 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:00.692109 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] do_osd_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096]
2014-06-12 14:46:00.692122 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:00.692138 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] do_osd_op  write 1331200~516096
2014-06-12 14:46:00.692155 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] do_osd_op_effects on session 0xdd07800
2014-06-12 14:46:00.692167 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 snapset=0x72b9a78  snapc=0=[]
2014-06-12 14:46:00.692181 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:00.692194 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] finish_ctx 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 0x110c4a00 op modify  
2014-06-12 14:46:00.692208 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]  set mtime to 2014-06-12 08:45:03.070763
2014-06-12 14:46:00.692232 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]  final snapset 0=[]:[]+head in 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.692253 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]  zeroing write result code 0
2014-06-12 14:46:00.692265 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] new_repop rep_tid 236168 on osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.692283 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] issue_repop rep_tid 236168 o 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.692309 7f9d693e3700  5 -- op tracker -- , seq: 747431, time: 2014-06-12 14:46:00.692304, event: waiting for subops from 25, op: osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.692332 7f9d693e3700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:00.692355 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2847 (0'0,173'2847] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] append_log log((0'0,173'2847], crt=0'0) [173'2848 (173'2847) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144764 2014-06-12 08:45:03.070763]
2014-06-12 14:46:00.692381 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] add_log_entry 173'2848 (173'2847) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144764 2014-06-12 08:45:03.070763
2014-06-12 14:46:00.692403 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] append_log  adding 1 keys
2014-06-12 14:46:00.692437 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2848, trimmed: 
2014-06-12 14:46:00.692461 7f9d693e3700  5 queue_transactions existing osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.692467 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.692468 7f9d693e3700  5 queue_op 0x68a5860 seq 2628 osr(3.15d 0x276cea0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.692476 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] eval_repop repgather(0x52f4fc0 173'2848 rep_tid=236168 committed?=0 applied?=0 lock=0 op=osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.692496 7f9d693e3700 10 osd.2 173 dequeue_op 0x25e42200 finish
2014-06-12 14:46:00.692501 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.692504 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.692512 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9d40 (1 active)
2014-06-12 14:46:00.692517 7f9d7f16b700  5 _do_op 0x68a5860 seq 2628 osr(3.15d 0x276cea0)/0x276cea0 start
2014-06-12 14:46:00.692521 7f9d7f16b700 10 _do_transaction on 0x5360300
2014-06-12 14:46:00.692526 7f9d7f16b700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.692626 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.692648 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.692705 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.692795 7f9d7f16b700 15 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.692805 7f9d7f16b700 10 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1 r = 0
2014-06-12 14:46:00.692811 7f9d7f16b700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.692834 7f9d7f16b700 15 _write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 1331200~516096
2014-06-12 14:46:00.692971 7f9d7f16b700 10 lookup_strip_header done  cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.693113 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:00.694883 7f9d7f16b700 10 _generic_write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 1331200~516096 = 0
2014-06-12 14:46:00.694943 7f9d7f16b700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.694958 7f9d7f16b700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.694972 7f9d7f16b700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.694980 7f9d7f16b700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.695188 7f9d7f16b700 20 set_header setting 17237 cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 parent seq 0
2014-06-12 14:46:00.695263 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.695278 7f9d7f16b700 20 set_header setting 1998 cid meta oid 1fe2b61b/pglog_3.15d/0//-1 parent seq 0
2014-06-12 14:46:00.695290 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:00.699011 7f9d7f16b700 10 _do_op 0x68a5860 seq 2628 r = 0, finisher 0x53cb760 0x42cd9c0
2014-06-12 14:46:00.699038 7f9d7f16b700 10 _finish_op 0x68a5860 seq 2628 osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.699047 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9d40 (0 active)
2014-06-12 14:46:00.699059 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.699065 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] op_applied: 236168
2014-06-12 14:46:00.699102 7f9d7795c700  5 -- op tracker -- , seq: 747431, time: 2014-06-12 14:46:00.699091, event: op_applied, op: osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.699111 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] op_applied on primary on version 173'2848
2014-06-12 14:46:00.699129 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] op_commit: 236168
2014-06-12 14:46:00.699143 7f9d5bec4700  5 -- op tracker -- , seq: 747432, time: 2014-06-12 14:46:00.699102, event: header_read, op: osd_sub_op_reply(client.4315.0:144764 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.699156 7f9d5bec4700  5 -- op tracker -- , seq: 747432, time: 2014-06-12 14:46:00.699103, event: throttled, op: osd_sub_op_reply(client.4315.0:144764 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.699165 7f9d5bec4700  5 -- op tracker -- , seq: 747432, time: 2014-06-12 14:46:00.699119, event: all_read, op: osd_sub_op_reply(client.4315.0:144764 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.699170 7f9d5bec4700  5 -- op tracker -- , seq: 747432, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4315.0:144764 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.699175 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4315.0:144764 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:00.699180 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:00.699186 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e86500 prio 196 cost 0 latency 0.000084 osd_sub_op_reply(client.4315.0:144764 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:00.699181 7f9d7715b700  5 -- op tracker -- , seq: 747431, time: 2014-06-12 14:46:00.699148, event: op_commit, op: osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.699210 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.699217 7f9d62bd6700 10 osd.2 173 dequeue_op 0x20e86500 prio 196 cost 0 latency 0.000115 osd_sub_op_reply(client.4315.0:144764 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 pg pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]
2014-06-12 14:46:00.699230 7f9d62bd6700  5 -- op tracker -- , seq: 747432, time: 2014-06-12 14:46:00.699228, event: reached_pg, op: osd_sub_op_reply(client.4315.0:144764 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.699242 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] handle_message: 0x20e86500
2014-06-12 14:46:00.699251 7f9d62bd6700  5 -- op tracker -- , seq: 747432, time: 2014-06-12 14:46:00.699249, event: started, op: osd_sub_op_reply(client.4315.0:144764 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.699254 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] sub_op_modify_reply: tid 236168 op  ack_type 4 from 25
2014-06-12 14:46:00.699265 7f9d62bd6700  5 -- op tracker -- , seq: 747431, time: 2014-06-12 14:46:00.699262, event: sub_op_commit_rec, op: osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.699273 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] repop_all_applied: repop tid 236168 all applied 
2014-06-12 14:46:00.699280 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] eval_repop repgather(0x52f4fc0 173'2848 rep_tid=236168 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.699290 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]  sending ack on repgather(0x52f4fc0 173'2848 rep_tid=236168 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0xc211900
2014-06-12 14:46:00.699311 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] repop_all_committed: repop tid 236168 all committed 
2014-06-12 14:46:00.699320 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2846 active+clean] eval_repop repgather(0x52f4fc0 173'2848 rep_tid=236168 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.699335 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2846 active+clean] log_op_stats osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008844 lat 0.008868
2014-06-12 14:46:00.699352 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2846 active+clean] publish_stats_to_osd 173:3711
2014-06-12 14:46:00.699363 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2846 active+clean]  sending commit on repgather(0x52f4fc0 173'2848 rep_tid=236168 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x2183ac80
2014-06-12 14:46:00.699385 7f9d62bd6700  5 -- op tracker -- , seq: 747431, time: 2014-06-12 14:46:00.699380, event: commit_sent, op: osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.699394 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2846 active+clean]  requeue_ops 
2014-06-12 14:46:00.699407 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]  removing repgather(0x52f4fc0 173'2848 rep_tid=236168 committed?=1 applied?=1 op=osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.699418 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]    q front is repgather(0x52f4fc0 173'2848 rep_tid=236168 committed?=1 applied?=1 op=osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.699427 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] remove_repop repgather(0x52f4fc0 173'2848 rep_tid=236168 committed?=1 applied?=1 op=osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.699435 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]  requeue_ops 
2014-06-12 14:46:00.699452 7f9d62bd6700  5 -- op tracker -- , seq: 747431, time: 2014-06-12 14:46:00.699449, event: done, op: osd_op(client.4315.0:144764 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.699469 7f9d62bd6700 10 osd.2 173 dequeue_op 0x20e86500 finish
2014-06-12 14:46:00.699476 7f9d62bd6700  5 -- op tracker -- , seq: 747432, time: 2014-06-12 14:46:00.699473, event: done, op: osd_sub_op_reply(client.4315.0:144764 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.699488 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.699490 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.702488 7f9d44000700  5 -- op tracker -- , seq: 747433, time: 2014-06-12 14:46:00.701509, event: header_read, op: osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.702520 7f9d44000700  5 -- op tracker -- , seq: 747433, time: 2014-06-12 14:46:00.701512, event: throttled, op: osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.702529 7f9d44000700  5 -- op tracker -- , seq: 747433, time: 2014-06-12 14:46:00.702432, event: all_read, op: osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.702545 7f9d44000700  5 -- op tracker -- , seq: 747433, time: 0.000000, event: dispatched, op: osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.702554 7f9d44000700 20 osd.2 173 should_share_map client.4315 172.16.96.220:0/1244608 173
2014-06-12 14:46:00.702584 7f9d44000700 15 osd.2 173 enqueue_op 0x25e42400 prio 63 cost 516096 latency 0.001075 osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.702629 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.702638 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x25e42400 prio 63 cost 516096 latency 0.001129 osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]
2014-06-12 14:46:00.702671 7f9d6b3e7700  5 -- op tracker -- , seq: 747433, time: 2014-06-12 14:46:00.702666, event: reached_pg, op: osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.702680 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:00.702694 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] handle_message: 0x25e42400
2014-06-12 14:46:00.702704 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] do_op osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:00.702737 7f9d6b3e7700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_'
2014-06-12 14:46:00.702957 7f9d6b3e7700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_' = 0
2014-06-12 14:46:00.702975 7f9d6b3e7700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset'
2014-06-12 14:46:00.703082 7f9d6b3e7700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset' = 0
2014-06-12 14:46:00.703094 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] populate_obc_watchers 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.703113 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.703122 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] get_object_context: creating obc from disk: 0xa0398c0
2014-06-12 14:46:00.703132 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] get_object_context: 0xa0398c0 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 rwstate(none n=0 w=0) oi: 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2848 client.4315.0:144764 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2848) ssc: 0x16ee4540 snapset: 0=[]:[]+head
2014-06-12 14:46:00.703148 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] find_object_context 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 @head oi=9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2848 client.4315.0:144764 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2848)
2014-06-12 14:46:00.703173 7f9d6b3e7700  5 -- op tracker -- , seq: 747433, time: 2014-06-12 14:46:00.703167, event: started, op: osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.703183 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] execute_ctx 0x2a82c00
2014-06-12 14:46:00.703215 7f9d6b3e7700  5 -- op tracker -- , seq: 747433, time: 2014-06-12 14:46:00.703210, event: started, op: osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.703230 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] do_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] ov 173'2848 av 173'2849 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:00.703245 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] do_osd_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096]
2014-06-12 14:46:00.703255 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:00.703268 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] do_osd_op  write 1847296~516096
2014-06-12 14:46:00.703286 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] do_osd_op_effects on session 0xdd07800
2014-06-12 14:46:00.703299 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 snapset=0x16ee4578  snapc=0=[]
2014-06-12 14:46:00.703313 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:00.703325 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] finish_ctx 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 0x2a82c00 op modify  
2014-06-12 14:46:00.703340 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]  set mtime to 2014-06-12 08:45:03.081775
2014-06-12 14:46:00.703363 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]  final snapset 0=[]:[]+head in 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.703391 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]  zeroing write result code 0
2014-06-12 14:46:00.703409 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] new_repop rep_tid 236169 on osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.703424 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] issue_repop rep_tid 236169 o 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.703448 7f9d6b3e7700  5 -- op tracker -- , seq: 747433, time: 2014-06-12 14:46:00.703443, event: waiting for subops from 25, op: osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.703474 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:00.703512 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2848 (0'0,173'2848] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] append_log log((0'0,173'2848], crt=0'0) [173'2849 (173'2848) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144765 2014-06-12 08:45:03.081775]
2014-06-12 14:46:00.703536 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] add_log_entry 173'2849 (173'2848) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144765 2014-06-12 08:45:03.081775
2014-06-12 14:46:00.703573 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] append_log  adding 1 keys
2014-06-12 14:46:00.703607 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2849, trimmed: 
2014-06-12 14:46:00.703631 7f9d6b3e7700  5 queue_transactions existing osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.703637 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.703639 7f9d6b3e7700  5 queue_op 0x82e0840 seq 2629 osr(3.15d 0x276cea0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.703646 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] eval_repop repgather(0x1b72d80 173'2849 rep_tid=236169 committed?=0 applied?=0 lock=0 op=osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.703666 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x25e42400 finish
2014-06-12 14:46:00.703671 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.703673 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.703681 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9d40 (1 active)
2014-06-12 14:46:00.703686 7f9d7e169700  5 _do_op 0x82e0840 seq 2629 osr(3.15d 0x276cea0)/0x276cea0 start
2014-06-12 14:46:00.703690 7f9d7e169700 10 _do_transaction on 0x519d680
2014-06-12 14:46:00.703696 7f9d7e169700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.703800 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.703826 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.703897 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.703989 7f9d7e169700 15 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.703998 7f9d7e169700 10 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1 r = 0
2014-06-12 14:46:00.704004 7f9d7e169700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.704023 7f9d7e169700 15 _write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 1847296~516096
2014-06-12 14:46:00.704159 7f9d7e169700 10 lookup_strip_header done  cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.704301 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:00.706167 7f9d7e169700 10 _generic_write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 1847296~516096 = 0
2014-06-12 14:46:00.706224 7f9d7e169700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.706245 7f9d7e169700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.706255 7f9d7e169700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.706264 7f9d7e169700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.706473 7f9d7e169700 20 set_header setting 17237 cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 parent seq 0
2014-06-12 14:46:00.706540 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.706556 7f9d7e169700 20 set_header setting 1998 cid meta oid 1fe2b61b/pglog_3.15d/0//-1 parent seq 0
2014-06-12 14:46:00.706570 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:00.709165 7f9d7e169700 10 _do_op 0x82e0840 seq 2629 r = 0, finisher 0x7890300 0x42cd940
2014-06-12 14:46:00.709729 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] op_commit: 236169
2014-06-12 14:46:00.709774 7f9d7715b700  5 -- op tracker -- , seq: 747433, time: 2014-06-12 14:46:00.709762, event: op_commit, op: osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.709789 7f9d7e169700 10 _finish_op 0x82e0840 seq 2629 osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.709802 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9d40 (0 active)
2014-06-12 14:46:00.709811 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.709807 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] op_applied: 236169
2014-06-12 14:46:00.709831 7f9d7795c700  5 -- op tracker -- , seq: 747433, time: 2014-06-12 14:46:00.709824, event: op_applied, op: osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.709838 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] op_applied on primary on version 173'2849
2014-06-12 14:46:00.717031 7f9d5bec4700  5 -- op tracker -- , seq: 747434, time: 2014-06-12 14:46:00.716964, event: header_read, op: osd_sub_op_reply(client.4315.0:144765 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.717046 7f9d5bec4700  5 -- op tracker -- , seq: 747434, time: 2014-06-12 14:46:00.716966, event: throttled, op: osd_sub_op_reply(client.4315.0:144765 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.717055 7f9d5bec4700  5 -- op tracker -- , seq: 747434, time: 2014-06-12 14:46:00.716984, event: all_read, op: osd_sub_op_reply(client.4315.0:144765 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.717064 7f9d5bec4700  5 -- op tracker -- , seq: 747434, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4315.0:144765 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.717069 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4315.0:144765 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:00.717076 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:00.717085 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e87600 prio 196 cost 0 latency 0.000121 osd_sub_op_reply(client.4315.0:144765 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:00.717116 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.717125 7f9d64bda700 10 osd.2 173 dequeue_op 0x20e87600 prio 196 cost 0 latency 0.000161 osd_sub_op_reply(client.4315.0:144765 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 pg pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]
2014-06-12 14:46:00.717151 7f9d64bda700  5 -- op tracker -- , seq: 747434, time: 2014-06-12 14:46:00.717147, event: reached_pg, op: osd_sub_op_reply(client.4315.0:144765 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.717161 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] handle_message: 0x20e87600
2014-06-12 14:46:00.717177 7f9d64bda700  5 -- op tracker -- , seq: 747434, time: 2014-06-12 14:46:00.717174, event: started, op: osd_sub_op_reply(client.4315.0:144765 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.717183 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] sub_op_modify_reply: tid 236169 op  ack_type 4 from 25
2014-06-12 14:46:00.717211 7f9d64bda700  5 -- op tracker -- , seq: 747433, time: 2014-06-12 14:46:00.717201, event: sub_op_commit_rec, op: osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.717221 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] repop_all_applied: repop tid 236169 all applied 
2014-06-12 14:46:00.717232 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] eval_repop repgather(0x1b72d80 173'2849 rep_tid=236169 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.717250 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]  sending ack on repgather(0x1b72d80 173'2849 rep_tid=236169 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0xc210280
2014-06-12 14:46:00.717289 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] repop_all_committed: repop tid 236169 all committed 
2014-06-12 14:46:00.717301 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2847 active+clean] eval_repop repgather(0x1b72d80 173'2849 rep_tid=236169 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.717320 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2847 active+clean] log_op_stats osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.015778 lat 0.015810
2014-06-12 14:46:00.717340 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2847 active+clean] publish_stats_to_osd 173:3712
2014-06-12 14:46:00.717353 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2847 active+clean]  sending commit on repgather(0x1b72d80 173'2849 rep_tid=236169 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x34e4c000
2014-06-12 14:46:00.717387 7f9d64bda700  5 -- op tracker -- , seq: 747433, time: 2014-06-12 14:46:00.717381, event: commit_sent, op: osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.717397 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2847 active+clean]  requeue_ops 
2014-06-12 14:46:00.717408 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean]  removing repgather(0x1b72d80 173'2849 rep_tid=236169 committed?=1 applied?=1 op=osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.717424 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean]    q front is repgather(0x1b72d80 173'2849 rep_tid=236169 committed?=1 applied?=1 op=osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.717441 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] remove_repop repgather(0x1b72d80 173'2849 rep_tid=236169 committed?=1 applied?=1 op=osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.717455 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean]  requeue_ops 
2014-06-12 14:46:00.717481 7f9d64bda700  5 -- op tracker -- , seq: 747433, time: 2014-06-12 14:46:00.717476, event: done, op: osd_op(client.4315.0:144765 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.717504 7f9d64bda700 10 osd.2 173 dequeue_op 0x20e87600 finish
2014-06-12 14:46:00.717513 7f9d64bda700  5 -- op tracker -- , seq: 747434, time: 2014-06-12 14:46:00.717510, event: done, op: osd_sub_op_reply(client.4315.0:144765 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.717524 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.717527 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.720290 7f9d44000700  5 -- op tracker -- , seq: 747435, time: 2014-06-12 14:46:00.719303, event: header_read, op: osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.720306 7f9d44000700  5 -- op tracker -- , seq: 747435, time: 2014-06-12 14:46:00.719306, event: throttled, op: osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.720316 7f9d44000700  5 -- op tracker -- , seq: 747435, time: 2014-06-12 14:46:00.720239, event: all_read, op: osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.720326 7f9d44000700  5 -- op tracker -- , seq: 747435, time: 0.000000, event: dispatched, op: osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.720341 7f9d44000700 20 osd.2 173 should_share_map client.4315 172.16.96.220:0/1244608 173
2014-06-12 14:46:00.720375 7f9d44000700 15 osd.2 173 enqueue_op 0x25e42a00 prio 63 cost 516096 latency 0.001072 osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.721645 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.721657 7f9d683e1700 10 osd.2 173 dequeue_op 0x25e42a00 prio 63 cost 516096 latency 0.002354 osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean]
2014-06-12 14:46:00.721694 7f9d683e1700  5 -- op tracker -- , seq: 747435, time: 2014-06-12 14:46:00.721688, event: reached_pg, op: osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.721711 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:00.721728 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] handle_message: 0x25e42a00
2014-06-12 14:46:00.721744 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] do_op osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:00.721772 7f9d683e1700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_'
2014-06-12 14:46:00.721998 7f9d683e1700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_' = 0
2014-06-12 14:46:00.722017 7f9d683e1700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset'
2014-06-12 14:46:00.722126 7f9d683e1700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset' = 0
2014-06-12 14:46:00.722139 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] populate_obc_watchers 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.722152 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.722163 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] get_object_context: creating obc from disk: 0x217f0000
2014-06-12 14:46:00.722173 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] get_object_context: 0x217f0000 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 rwstate(none n=0 w=0) oi: 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2849 client.4315.0:144765 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2849) ssc: 0x16ee47e0 snapset: 0=[]:[]+head
2014-06-12 14:46:00.722190 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] find_object_context 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 @head oi=9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2849 client.4315.0:144765 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2849)
2014-06-12 14:46:00.722218 7f9d683e1700  5 -- op tracker -- , seq: 747435, time: 2014-06-12 14:46:00.722211, event: started, op: osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.722228 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] execute_ctx 0xa051600
2014-06-12 14:46:00.722246 7f9d683e1700  5 -- op tracker -- , seq: 747435, time: 2014-06-12 14:46:00.722241, event: started, op: osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.722254 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] do_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] ov 173'2849 av 173'2850 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:00.722269 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] do_osd_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096]
2014-06-12 14:46:00.722280 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:00.722294 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] do_osd_op  write 2363392~516096
2014-06-12 14:46:00.722311 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] do_osd_op_effects on session 0xdd07800
2014-06-12 14:46:00.722324 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 snapset=0x16ee4818  snapc=0=[]
2014-06-12 14:46:00.722339 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:00.722350 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] finish_ctx 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 0xa051600 op modify  
2014-06-12 14:46:00.722364 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean]  set mtime to 2014-06-12 08:45:03.099787
2014-06-12 14:46:00.722387 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean]  final snapset 0=[]:[]+head in 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.722407 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean]  zeroing write result code 0
2014-06-12 14:46:00.722419 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] new_repop rep_tid 236170 on osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.722437 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] issue_repop rep_tid 236170 o 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.722461 7f9d683e1700  5 -- op tracker -- , seq: 747435, time: 2014-06-12 14:46:00.722456, event: waiting for subops from 25, op: osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.722484 7f9d683e1700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:00.722508 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2849 (0'0,173'2849] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] append_log log((0'0,173'2849], crt=0'0) [173'2850 (173'2849) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144766 2014-06-12 08:45:03.099787]
2014-06-12 14:46:00.722533 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2849 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] add_log_entry 173'2850 (173'2849) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144766 2014-06-12 08:45:03.099787
2014-06-12 14:46:00.722554 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2849 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] append_log  adding 1 keys
2014-06-12 14:46:00.722586 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2850, trimmed: 
2014-06-12 14:46:00.722607 7f9d683e1700  5 queue_transactions existing osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.722611 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.722613 7f9d683e1700  5 queue_op 0x132b1f20 seq 2630 osr(3.15d 0x276cea0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.722621 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2849 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] eval_repop repgather(0x32b69a40 173'2850 rep_tid=236170 committed?=0 applied?=0 lock=0 op=osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.722639 7f9d683e1700 10 osd.2 173 dequeue_op 0x25e42a00 finish
2014-06-12 14:46:00.722643 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.722645 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.722653 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9d40 (1 active)
2014-06-12 14:46:00.722659 7f9d7e96a700  5 _do_op 0x132b1f20 seq 2630 osr(3.15d 0x276cea0)/0x276cea0 start
2014-06-12 14:46:00.722663 7f9d7e96a700 10 _do_transaction on 0x32b68b40
2014-06-12 14:46:00.722669 7f9d7e96a700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.722767 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.722869 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.722931 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.722951 7f9d7e96a700 15 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.722957 7f9d7e96a700 10 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1 r = 0
2014-06-12 14:46:00.722963 7f9d7e96a700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.722977 7f9d7e96a700 15 _write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 2363392~516096
2014-06-12 14:46:00.723112 7f9d7e96a700 10 lookup_strip_header done  cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.723255 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:00.725047 7f9d7e96a700 10 _generic_write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 2363392~516096 = 0
2014-06-12 14:46:00.725103 7f9d7e96a700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.725120 7f9d7e96a700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.725127 7f9d7e96a700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.725136 7f9d7e96a700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.725346 7f9d7e96a700 20 set_header setting 17237 cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 parent seq 0
2014-06-12 14:46:00.725421 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.725435 7f9d7e96a700 20 set_header setting 1998 cid meta oid 1fe2b61b/pglog_3.15d/0//-1 parent seq 0
2014-06-12 14:46:00.725445 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:00.727525 7f9d7e96a700 10 _do_op 0x132b1f20 seq 2630 r = 0, finisher 0x2ef7dd40 0x42cd8c0
2014-06-12 14:46:00.727552 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2849 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] op_commit: 236170
2014-06-12 14:46:00.727583 7f9d7715b700  5 -- op tracker -- , seq: 747435, time: 2014-06-12 14:46:00.727574, event: op_commit, op: osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.727601 7f9d7e96a700 10 _finish_op 0x132b1f20 seq 2630 osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.727613 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9d40 (0 active)
2014-06-12 14:46:00.727622 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.727632 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2849 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] op_applied: 236170
2014-06-12 14:46:00.727651 7f9d7795c700  5 -- op tracker -- , seq: 747435, time: 2014-06-12 14:46:00.727645, event: op_applied, op: osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.727659 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2849 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] op_applied on primary on version 173'2850
2014-06-12 14:46:00.733712 7f9d5bec4700  5 -- op tracker -- , seq: 747436, time: 2014-06-12 14:46:00.733649, event: header_read, op: osd_sub_op_reply(client.4315.0:144766 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.733729 7f9d5bec4700  5 -- op tracker -- , seq: 747436, time: 2014-06-12 14:46:00.733651, event: throttled, op: osd_sub_op_reply(client.4315.0:144766 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.733737 7f9d5bec4700  5 -- op tracker -- , seq: 747436, time: 2014-06-12 14:46:00.733668, event: all_read, op: osd_sub_op_reply(client.4315.0:144766 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.733745 7f9d5bec4700  5 -- op tracker -- , seq: 747436, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4315.0:144766 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.733751 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4315.0:144766 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:00.733758 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:00.733767 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e87500 prio 196 cost 0 latency 0.000118 osd_sub_op_reply(client.4315.0:144766 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:00.733801 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.733809 7f9d65bdc700 10 osd.2 173 dequeue_op 0x20e87500 prio 196 cost 0 latency 0.000160 osd_sub_op_reply(client.4315.0:144766 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 pg pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2849 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean]
2014-06-12 14:46:00.733830 7f9d65bdc700  5 -- op tracker -- , seq: 747436, time: 2014-06-12 14:46:00.733827, event: reached_pg, op: osd_sub_op_reply(client.4315.0:144766 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.733839 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2849 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] handle_message: 0x20e87500
2014-06-12 14:46:00.733855 7f9d65bdc700  5 -- op tracker -- , seq: 747436, time: 2014-06-12 14:46:00.733852, event: started, op: osd_sub_op_reply(client.4315.0:144766 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.733862 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2849 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] sub_op_modify_reply: tid 236170 op  ack_type 4 from 25
2014-06-12 14:46:00.733883 7f9d65bdc700  5 -- op tracker -- , seq: 747435, time: 2014-06-12 14:46:00.733875, event: sub_op_commit_rec, op: osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.733891 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2849 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] repop_all_applied: repop tid 236170 all applied 
2014-06-12 14:46:00.733902 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2849 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] eval_repop repgather(0x32b69a40 173'2850 rep_tid=236170 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.733919 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2849 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean]  sending ack on repgather(0x32b69a40 173'2850 rep_tid=236170 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x26e57680
2014-06-12 14:46:00.733952 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2849 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] repop_all_committed: repop tid 236170 all committed 
2014-06-12 14:46:00.733963 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2848 active+clean] eval_repop repgather(0x32b69a40 173'2850 rep_tid=236170 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.733980 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2848 active+clean] log_op_stats osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.014647 lat 0.014675
2014-06-12 14:46:00.733997 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2848 active+clean] publish_stats_to_osd 173:3713
2014-06-12 14:46:00.734010 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2848 active+clean]  sending commit on repgather(0x32b69a40 173'2850 rep_tid=236170 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x15cd7b80
2014-06-12 14:46:00.734043 7f9d65bdc700  5 -- op tracker -- , seq: 747435, time: 2014-06-12 14:46:00.734037, event: commit_sent, op: osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.734051 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2848 active+clean]  requeue_ops 
2014-06-12 14:46:00.734062 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean]  removing repgather(0x32b69a40 173'2850 rep_tid=236170 committed?=1 applied?=1 op=osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.734076 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean]    q front is repgather(0x32b69a40 173'2850 rep_tid=236170 committed?=1 applied?=1 op=osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.734091 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] remove_repop repgather(0x32b69a40 173'2850 rep_tid=236170 committed?=1 applied?=1 op=osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.734105 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean]  requeue_ops 
2014-06-12 14:46:00.734131 7f9d65bdc700  5 -- op tracker -- , seq: 747435, time: 2014-06-12 14:46:00.734125, event: done, op: osd_op(client.4315.0:144766 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.734151 7f9d65bdc700 10 osd.2 173 dequeue_op 0x20e87500 finish
2014-06-12 14:46:00.734159 7f9d65bdc700  5 -- op tracker -- , seq: 747436, time: 2014-06-12 14:46:00.734156, event: done, op: osd_sub_op_reply(client.4315.0:144766 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.734170 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.734174 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.738218 7f9d44000700  5 -- op tracker -- , seq: 747437, time: 2014-06-12 14:46:00.737644, event: header_read, op: osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.738234 7f9d44000700  5 -- op tracker -- , seq: 747437, time: 2014-06-12 14:46:00.737646, event: throttled, op: osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.738244 7f9d44000700  5 -- op tracker -- , seq: 747437, time: 2014-06-12 14:46:00.738173, event: all_read, op: osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.738254 7f9d44000700  5 -- op tracker -- , seq: 747437, time: 0.000000, event: dispatched, op: osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.738262 7f9d44000700 20 osd.2 173 should_share_map client.4315 172.16.96.220:0/1244608 173
2014-06-12 14:46:00.738292 7f9d44000700 15 osd.2 173 enqueue_op 0x25e43700 prio 63 cost 516096 latency 0.000647 osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.745245 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.745261 7f9d6abe6700 10 osd.2 173 dequeue_op 0x25e43700 prio 63 cost 516096 latency 0.007616 osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean]
2014-06-12 14:46:00.745289 7f9d6abe6700  5 -- op tracker -- , seq: 747437, time: 2014-06-12 14:46:00.745285, event: reached_pg, op: osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.745302 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:00.745318 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] handle_message: 0x25e43700
2014-06-12 14:46:00.745324 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] do_op osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:00.745342 7f9d6abe6700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_'
2014-06-12 14:46:00.745509 7f9d6abe6700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_' = 0
2014-06-12 14:46:00.745523 7f9d6abe6700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset'
2014-06-12 14:46:00.745587 7f9d6abe6700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset' = 0
2014-06-12 14:46:00.745593 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] populate_obc_watchers 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.745601 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.745607 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] get_object_context: creating obc from disk: 0x14654dc0
2014-06-12 14:46:00.745620 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] get_object_context: 0x14654dc0 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 rwstate(none n=0 w=0) oi: 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2850 client.4315.0:144766 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2850) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:46:00.745641 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] find_object_context 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 @head oi=9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2850 client.4315.0:144766 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2850)
2014-06-12 14:46:00.745666 7f9d6abe6700  5 -- op tracker -- , seq: 747437, time: 2014-06-12 14:46:00.745661, event: started, op: osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.745674 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] execute_ctx 0xa04f800
2014-06-12 14:46:00.745691 7f9d6abe6700  5 -- op tracker -- , seq: 747437, time: 2014-06-12 14:46:00.745686, event: started, op: osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.745698 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] do_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] ov 173'2850 av 173'2851 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:00.745712 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] do_osd_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096]
2014-06-12 14:46:00.745723 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:00.745737 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] do_osd_op  write 2879488~516096
2014-06-12 14:46:00.745754 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] do_osd_op_effects on session 0xdd07800
2014-06-12 14:46:00.745765 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:46:00.745776 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:00.745786 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] finish_ctx 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 0xa04f800 op modify  
2014-06-12 14:46:00.745799 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean]  set mtime to 2014-06-12 08:45:03.116200
2014-06-12 14:46:00.745820 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean]  final snapset 0=[]:[]+head in 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.745839 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean]  zeroing write result code 0
2014-06-12 14:46:00.745850 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] new_repop rep_tid 236171 on osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.745866 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] issue_repop rep_tid 236171 o 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.745892 7f9d6abe6700  5 -- op tracker -- , seq: 747437, time: 2014-06-12 14:46:00.745887, event: waiting for subops from 25, op: osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.745915 7f9d6abe6700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:00.745940 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2850 (0'0,173'2850] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] append_log log((0'0,173'2850], crt=0'0) [173'2851 (173'2850) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144767 2014-06-12 08:45:03.116200]
2014-06-12 14:46:00.745965 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2850 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] add_log_entry 173'2851 (173'2850) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144767 2014-06-12 08:45:03.116200
2014-06-12 14:46:00.745983 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2850 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] append_log  adding 1 keys
2014-06-12 14:46:00.746012 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2851, trimmed: 
2014-06-12 14:46:00.746032 7f9d6abe6700  5 queue_transactions existing osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.746036 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.746037 7f9d6abe6700  5 queue_op 0x58db260 seq 2631 osr(3.15d 0x276cea0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.746044 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2850 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] eval_repop repgather(0xbe0d200 173'2851 rep_tid=236171 committed?=0 applied?=0 lock=0 op=osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.746060 7f9d6abe6700 10 osd.2 173 dequeue_op 0x25e43700 finish
2014-06-12 14:46:00.746063 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.746065 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.746080 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9d40 (1 active)
2014-06-12 14:46:00.746085 7f9d82171700  5 _do_op 0x58db260 seq 2631 osr(3.15d 0x276cea0)/0x276cea0 start
2014-06-12 14:46:00.746088 7f9d82171700 10 _do_transaction on 0x1f066180
2014-06-12 14:46:00.746093 7f9d82171700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.746185 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.746208 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.746260 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.746384 7f9d82171700 15 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.746391 7f9d82171700 10 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1 r = 0
2014-06-12 14:46:00.746397 7f9d82171700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.746418 7f9d82171700 15 _write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 2879488~516096
2014-06-12 14:46:00.746562 7f9d82171700 10 lookup_strip_header done  cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.746702 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:00.748175 7f9d82171700 10 _generic_write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 2879488~516096 = 0
2014-06-12 14:46:00.748236 7f9d82171700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.748254 7f9d82171700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.748260 7f9d82171700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.748268 7f9d82171700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.748479 7f9d82171700 20 set_header setting 17237 cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 parent seq 0
2014-06-12 14:46:00.748545 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.748558 7f9d82171700 20 set_header setting 1998 cid meta oid 1fe2b61b/pglog_3.15d/0//-1 parent seq 0
2014-06-12 14:46:00.748568 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:00.751135 7f9d82171700 10 _do_op 0x58db260 seq 2631 r = 0, finisher 0x4f54060 0x21505bc0
2014-06-12 14:46:00.751164 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2850 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] op_commit: 236171
2014-06-12 14:46:00.751198 7f9d7715b700  5 -- op tracker -- , seq: 747437, time: 2014-06-12 14:46:00.751188, event: op_commit, op: osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.751218 7f9d82171700 10 _finish_op 0x58db260 seq 2631 osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.751227 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9d40 (0 active)
2014-06-12 14:46:00.751232 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.751241 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2850 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] op_applied: 236171
2014-06-12 14:46:00.751260 7f9d7795c700  5 -- op tracker -- , seq: 747437, time: 2014-06-12 14:46:00.751255, event: op_applied, op: osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.751268 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2850 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] op_applied on primary on version 173'2851
2014-06-12 14:46:00.751696 7f9d5bec4700  5 -- op tracker -- , seq: 747438, time: 2014-06-12 14:46:00.751652, event: header_read, op: osd_sub_op_reply(client.4315.0:144767 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.751707 7f9d5bec4700  5 -- op tracker -- , seq: 747438, time: 2014-06-12 14:46:00.751653, event: throttled, op: osd_sub_op_reply(client.4315.0:144767 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.751715 7f9d5bec4700  5 -- op tracker -- , seq: 747438, time: 2014-06-12 14:46:00.751667, event: all_read, op: osd_sub_op_reply(client.4315.0:144767 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.751722 7f9d5bec4700  5 -- op tracker -- , seq: 747438, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4315.0:144767 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.751727 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4315.0:144767 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:00.751733 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:00.751741 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e87b00 prio 196 cost 0 latency 0.000089 osd_sub_op_reply(client.4315.0:144767 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:00.751773 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.751780 7f9d66bde700 10 osd.2 173 dequeue_op 0x20e87b00 prio 196 cost 0 latency 0.000127 osd_sub_op_reply(client.4315.0:144767 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 pg pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2850 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean]
2014-06-12 14:46:00.751798 7f9d66bde700  5 -- op tracker -- , seq: 747438, time: 2014-06-12 14:46:00.751794, event: reached_pg, op: osd_sub_op_reply(client.4315.0:144767 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.751806 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2850 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] handle_message: 0x20e87b00
2014-06-12 14:46:00.751820 7f9d66bde700  5 -- op tracker -- , seq: 747438, time: 2014-06-12 14:46:00.751817, event: started, op: osd_sub_op_reply(client.4315.0:144767 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.751826 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2850 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] sub_op_modify_reply: tid 236171 op  ack_type 4 from 25
2014-06-12 14:46:00.751844 7f9d66bde700  5 -- op tracker -- , seq: 747437, time: 2014-06-12 14:46:00.751839, event: sub_op_commit_rec, op: osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.751851 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2850 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] repop_all_applied: repop tid 236171 all applied 
2014-06-12 14:46:00.751862 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2850 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] eval_repop repgather(0xbe0d200 173'2851 rep_tid=236171 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.751877 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2850 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean]  sending ack on repgather(0xbe0d200 173'2851 rep_tid=236171 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x6989900
2014-06-12 14:46:00.751908 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2850 crt=0'0 lcod 173'2849 mlcod 173'2849 active+clean] repop_all_committed: repop tid 236171 all committed 
2014-06-12 14:46:00.751919 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2849 active+clean] eval_repop repgather(0xbe0d200 173'2851 rep_tid=236171 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.751935 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2849 active+clean] log_op_stats osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.014263 lat 0.014290
2014-06-12 14:46:00.751952 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2849 active+clean] publish_stats_to_osd 173:3714
2014-06-12 14:46:00.751965 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2849 active+clean]  sending commit on repgather(0xbe0d200 173'2851 rep_tid=236171 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x8abf680
2014-06-12 14:46:00.751994 7f9d66bde700  5 -- op tracker -- , seq: 747437, time: 2014-06-12 14:46:00.751989, event: commit_sent, op: osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.752001 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2849 active+clean]  requeue_ops 
2014-06-12 14:46:00.752011 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]  removing repgather(0xbe0d200 173'2851 rep_tid=236171 committed?=1 applied?=1 op=osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.752026 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]    q front is repgather(0xbe0d200 173'2851 rep_tid=236171 committed?=1 applied?=1 op=osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.752039 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] remove_repop repgather(0xbe0d200 173'2851 rep_tid=236171 committed?=1 applied?=1 op=osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.752053 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]  requeue_ops 
2014-06-12 14:46:00.752077 7f9d66bde700  5 -- op tracker -- , seq: 747437, time: 2014-06-12 14:46:00.752072, event: done, op: osd_op(client.4315.0:144767 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.752094 7f9d66bde700 10 osd.2 173 dequeue_op 0x20e87b00 finish
2014-06-12 14:46:00.752101 7f9d66bde700  5 -- op tracker -- , seq: 747438, time: 2014-06-12 14:46:00.752098, event: done, op: osd_sub_op_reply(client.4315.0:144767 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.752111 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.752114 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.755420 7f9d44000700  5 -- op tracker -- , seq: 747439, time: 2014-06-12 14:46:00.754363, event: header_read, op: osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.755434 7f9d44000700  5 -- op tracker -- , seq: 747439, time: 2014-06-12 14:46:00.754365, event: throttled, op: osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.755444 7f9d44000700  5 -- op tracker -- , seq: 747439, time: 2014-06-12 14:46:00.755372, event: all_read, op: osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.755454 7f9d44000700  5 -- op tracker -- , seq: 747439, time: 0.000000, event: dispatched, op: osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.755461 7f9d44000700 20 osd.2 173 should_share_map client.4315 172.16.96.220:0/1244608 173
2014-06-12 14:46:00.755489 7f9d44000700 15 osd.2 173 enqueue_op 0x25e42100 prio 63 cost 516096 latency 0.001125 osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.755518 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.755524 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x25e42100 prio 63 cost 516096 latency 0.001161 osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]
2014-06-12 14:46:00.755547 7f9d6a3e5700  5 -- op tracker -- , seq: 747439, time: 2014-06-12 14:46:00.755542, event: reached_pg, op: osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.755556 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:00.755570 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] handle_message: 0x25e42100
2014-06-12 14:46:00.755580 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_op osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:00.755615 7f9d6a3e5700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_'
2014-06-12 14:46:00.755855 7f9d6a3e5700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_' = 0
2014-06-12 14:46:00.755871 7f9d6a3e5700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset'
2014-06-12 14:46:00.755965 7f9d6a3e5700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset' = 0
2014-06-12 14:46:00.755975 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] populate_obc_watchers 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.755986 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.755996 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] get_object_context: creating obc from disk: 0x28013600
2014-06-12 14:46:00.756005 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] get_object_context: 0x28013600 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 rwstate(none n=0 w=0) oi: 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2851 client.4315.0:144767 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2851) ssc: 0x72b87e0 snapset: 0=[]:[]+head
2014-06-12 14:46:00.756020 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] find_object_context 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 @head oi=9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2851 client.4315.0:144767 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2851)
2014-06-12 14:46:00.756045 7f9d6a3e5700  5 -- op tracker -- , seq: 747439, time: 2014-06-12 14:46:00.756039, event: started, op: osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.756053 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] execute_ctx 0x2a84400
2014-06-12 14:46:00.756069 7f9d6a3e5700  5 -- op tracker -- , seq: 747439, time: 2014-06-12 14:46:00.756065, event: started, op: osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.756076 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] ov 173'2851 av 173'2852 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:00.756090 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_osd_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096]
2014-06-12 14:46:00.756101 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:00.756114 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_osd_op  write 3395584~516096
2014-06-12 14:46:00.756129 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_osd_op_effects on session 0xdd07800
2014-06-12 14:46:00.756140 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 snapset=0x72b8818  snapc=0=[]
2014-06-12 14:46:00.756151 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:00.756161 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] finish_ctx 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 0x2a84400 op modify  
2014-06-12 14:46:00.756416 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]  set mtime to 2014-06-12 08:45:03.134647
2014-06-12 14:46:00.756441 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]  final snapset 0=[]:[]+head in 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.756459 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]  zeroing write result code 0
2014-06-12 14:46:00.756470 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] new_repop rep_tid 236172 on osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.756487 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] issue_repop rep_tid 236172 o 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.756510 7f9d6a3e5700  5 -- op tracker -- , seq: 747439, time: 2014-06-12 14:46:00.756505, event: waiting for subops from 25, op: osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.756531 7f9d6a3e5700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:00.756551 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2851 (0'0,173'2851] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] append_log log((0'0,173'2851], crt=0'0) [173'2852 (173'2851) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144768 2014-06-12 08:45:03.134647]
2014-06-12 14:46:00.756575 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] add_log_entry 173'2852 (173'2851) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144768 2014-06-12 08:45:03.134647
2014-06-12 14:46:00.756596 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] append_log  adding 1 keys
2014-06-12 14:46:00.756628 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2852, trimmed: 
2014-06-12 14:46:00.756649 7f9d6a3e5700  5 queue_transactions existing osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.756654 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.756655 7f9d6a3e5700  5 queue_op 0x24648a0 seq 2632 osr(3.15d 0x276cea0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:00.756663 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] eval_repop repgather(0x2d8120c0 173'2852 rep_tid=236172 committed?=0 applied?=0 lock=0 op=osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.756681 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x25e42100 finish
2014-06-12 14:46:00.756691 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.756694 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.758242 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9d40 (1 active)
2014-06-12 14:46:00.758253 7f9d7d167700  5 _do_op 0x24648a0 seq 2632 osr(3.15d 0x276cea0)/0x276cea0 start
2014-06-12 14:46:00.758259 7f9d7d167700 10 _do_transaction on 0x2d812600
2014-06-12 14:46:00.758266 7f9d7d167700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.758386 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.758410 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.758468 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.758495 7f9d7d167700 15 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.758504 7f9d7d167700 10 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1 r = 0
2014-06-12 14:46:00.758510 7f9d7d167700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.758654 7f9d7d167700 15 _write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 3395584~516096
2014-06-12 14:46:00.758802 7f9d7d167700 10 lookup_strip_header done  cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.758911 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:00.760713 7f9d7d167700 10 _generic_write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 3395584~516096 = 0
2014-06-12 14:46:00.760765 7f9d7d167700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.760780 7f9d7d167700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.760786 7f9d7d167700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.760795 7f9d7d167700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.761005 7f9d7d167700 20 set_header setting 17237 cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 parent seq 0
2014-06-12 14:46:00.761073 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.761088 7f9d7d167700 20 set_header setting 1998 cid meta oid 1fe2b61b/pglog_3.15d/0//-1 parent seq 0
2014-06-12 14:46:00.761101 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:00.763440 7f9d7d167700 10 _do_op 0x24648a0 seq 2632 r = 0, finisher 0x1d4992e0 0x1a3f1d00
2014-06-12 14:46:00.763461 7f9d7d167700 10 _finish_op 0x24648a0 seq 2632 osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.763471 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9d40 (0 active)
2014-06-12 14:46:00.763478 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.763467 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] op_commit: 236172
2014-06-12 14:46:00.763501 7f9d7715b700  5 -- op tracker -- , seq: 747439, time: 2014-06-12 14:46:00.763490, event: op_commit, op: osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.763534 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] op_applied: 236172
2014-06-12 14:46:00.763554 7f9d7795c700  5 -- op tracker -- , seq: 747439, time: 2014-06-12 14:46:00.763548, event: op_applied, op: osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.763561 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] op_applied on primary on version 173'2852
2014-06-12 14:46:00.766223 7f9d5bec4700  5 -- op tracker -- , seq: 747440, time: 2014-06-12 14:46:00.766167, event: header_read, op: osd_sub_op_reply(client.4315.0:144768 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.766237 7f9d5bec4700  5 -- op tracker -- , seq: 747440, time: 2014-06-12 14:46:00.766169, event: throttled, op: osd_sub_op_reply(client.4315.0:144768 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.766245 7f9d5bec4700  5 -- op tracker -- , seq: 747440, time: 2014-06-12 14:46:00.766185, event: all_read, op: osd_sub_op_reply(client.4315.0:144768 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.766252 7f9d5bec4700  5 -- op tracker -- , seq: 747440, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4315.0:144768 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.766258 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4315.0:144768 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:00.766264 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:00.766272 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e86f00 prio 196 cost 0 latency 0.000105 osd_sub_op_reply(client.4315.0:144768 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:00.766304 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.766312 7f9d63bd8700 10 osd.2 173 dequeue_op 0x20e86f00 prio 196 cost 0 latency 0.000144 osd_sub_op_reply(client.4315.0:144768 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 pg pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]
2014-06-12 14:46:00.766337 7f9d63bd8700  5 -- op tracker -- , seq: 747440, time: 2014-06-12 14:46:00.766333, event: reached_pg, op: osd_sub_op_reply(client.4315.0:144768 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.766345 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] handle_message: 0x20e86f00
2014-06-12 14:46:00.766361 7f9d63bd8700  5 -- op tracker -- , seq: 747440, time: 2014-06-12 14:46:00.766357, event: started, op: osd_sub_op_reply(client.4315.0:144768 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.766366 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] sub_op_modify_reply: tid 236172 op  ack_type 4 from 25
2014-06-12 14:46:00.766387 7f9d63bd8700  5 -- op tracker -- , seq: 747439, time: 2014-06-12 14:46:00.766379, event: sub_op_commit_rec, op: osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.766394 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] repop_all_applied: repop tid 236172 all applied 
2014-06-12 14:46:00.766405 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] eval_repop repgather(0x2d8120c0 173'2852 rep_tid=236172 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.766422 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]  sending ack on repgather(0x2d8120c0 173'2852 rep_tid=236172 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x6988280
2014-06-12 14:46:00.766452 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] repop_all_committed: repop tid 236172 all committed 
2014-06-12 14:46:00.766463 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2850 active+clean] eval_repop repgather(0x2d8120c0 173'2852 rep_tid=236172 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.766478 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2850 active+clean] log_op_stats osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.012087 lat 0.012114
2014-06-12 14:46:00.766495 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2850 active+clean] publish_stats_to_osd 173:3715
2014-06-12 14:46:00.766508 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2850 active+clean]  sending commit on repgather(0x2d8120c0 173'2852 rep_tid=236172 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x75cdb80
2014-06-12 14:46:00.766536 7f9d63bd8700  5 -- op tracker -- , seq: 747439, time: 2014-06-12 14:46:00.766531, event: commit_sent, op: osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.766544 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2850 active+clean]  requeue_ops 
2014-06-12 14:46:00.766554 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]  removing repgather(0x2d8120c0 173'2852 rep_tid=236172 committed?=1 applied?=1 op=osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.766568 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]    q front is repgather(0x2d8120c0 173'2852 rep_tid=236172 committed?=1 applied?=1 op=osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.766582 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] remove_repop repgather(0x2d8120c0 173'2852 rep_tid=236172 committed?=1 applied?=1 op=osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.766596 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]  requeue_ops 
2014-06-12 14:46:00.766620 7f9d63bd8700  5 -- op tracker -- , seq: 747439, time: 2014-06-12 14:46:00.766615, event: done, op: osd_op(client.4315.0:144768 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.766639 7f9d63bd8700 10 osd.2 173 dequeue_op 0x20e86f00 finish
2014-06-12 14:46:00.766646 7f9d63bd8700  5 -- op tracker -- , seq: 747440, time: 2014-06-12 14:46:00.766642, event: done, op: osd_sub_op_reply(client.4315.0:144768 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.766656 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.766659 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.769107 7f9d44000700  5 -- op tracker -- , seq: 747441, time: 2014-06-12 14:46:00.768578, event: header_read, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.769122 7f9d44000700  5 -- op tracker -- , seq: 747441, time: 2014-06-12 14:46:00.768580, event: throttled, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.769132 7f9d44000700  5 -- op tracker -- , seq: 747441, time: 2014-06-12 14:46:00.769074, event: all_read, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.769142 7f9d44000700  5 -- op tracker -- , seq: 747441, time: 0.000000, event: dispatched, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.769149 7f9d44000700 20 osd.2 173 should_share_map client.4315 172.16.96.220:0/1244608 173
2014-06-12 14:46:00.769177 7f9d44000700 15 osd.2 173 enqueue_op 0x25e42e00 prio 63 cost 282624 latency 0.000599 osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.769209 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.769216 7f9d653db700 10 osd.2 173 dequeue_op 0x25e42e00 prio 63 cost 282624 latency 0.000638 osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]
2014-06-12 14:46:00.769243 7f9d653db700  5 -- op tracker -- , seq: 747441, time: 2014-06-12 14:46:00.769237, event: reached_pg, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.769252 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:00.769268 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] handle_message: 0x25e42e00
2014-06-12 14:46:00.769282 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] do_op osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:00.769309 7f9d653db700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_'
2014-06-12 14:46:00.769476 7f9d653db700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 '_' = 0
2014-06-12 14:46:00.769494 7f9d653db700 15 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset'
2014-06-12 14:46:00.769568 7f9d653db700 10 getattr 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 'snapset' = 0
2014-06-12 14:46:00.769578 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] populate_obc_watchers 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.769593 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.769604 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] get_object_context: creating obc from disk: 0x30ac3340
2014-06-12 14:46:00.769634 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] get_object_context: 0x30ac3340 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 rwstate(none n=0 w=0) oi: 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2852 client.4315.0:144768 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2852) ssc: 0x2ef8fa40 snapset: 0=[]:[]+head
2014-06-12 14:46:00.769654 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] find_object_context 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 @head oi=9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3(173'2852 client.4315.0:144768 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2852)
2014-06-12 14:46:00.769682 7f9d653db700  5 -- op tracker -- , seq: 747441, time: 2014-06-12 14:46:00.769676, event: started, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.769691 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] execute_ctx 0x1474be00
2014-06-12 14:46:00.769710 7f9d653db700  5 -- op tracker -- , seq: 747441, time: 2014-06-12 14:46:00.769705, event: started, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.769718 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] do_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] ov 173'2852 av 173'2853 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:00.769734 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] do_osd_op 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624]
2014-06-12 14:46:00.769748 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:00.769762 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] do_osd_op  write 3911680~282624
2014-06-12 14:46:00.769778 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] do_osd_op_effects on session 0xdd07800
2014-06-12 14:46:00.769791 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 snapset=0x2ef8fa78  snapc=0=[]
2014-06-12 14:46:00.769804 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] make_writeable 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:00.769817 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] finish_ctx 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 0x1474be00 op modify  
2014-06-12 14:46:00.769830 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]  set mtime to 2014-06-12 08:45:03.149282
2014-06-12 14:46:00.769852 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]  final snapset 0=[]:[]+head in 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.769873 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]  zeroing write result code 0
2014-06-12 14:46:00.769886 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] new_repop rep_tid 236173 on osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:00.769905 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] issue_repop rep_tid 236173 o 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.769934 7f9d653db700  5 -- op tracker -- , seq: 747441, time: 2014-06-12 14:46:00.769928, event: waiting for subops from 25, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.769954 7f9d653db700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:00.769976 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2852 (0'0,173'2852] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] append_log log((0'0,173'2852], crt=0'0) [173'2853 (173'2852) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144769 2014-06-12 08:45:03.149282]
2014-06-12 14:46:00.770001 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] add_log_entry 173'2853 (173'2852) modify   9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 by client.4315.0:144769 2014-06-12 08:45:03.149282
2014-06-12 14:46:00.770020 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] append_log  adding 1 keys
2014-06-12 14:46:00.770048 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2853, trimmed: 
2014-06-12 14:46:00.770067 7f9d653db700  5 queue_transactions existing osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.770071 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:00.770072 7f9d653db700  5 queue_op 0x165ae300 seq 2633 osr(3.15d 0x276cea0) 284917 bytes   (queue has 1 ops and 284917 bytes)
2014-06-12 14:46:00.770079 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] eval_repop repgather(0x384e2540 173'2853 rep_tid=236173 committed?=0 applied?=0 lock=0 op=osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.770095 7f9d653db700 10 osd.2 173 dequeue_op 0x25e42e00 finish
2014-06-12 14:46:00.770098 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.770101 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.770108 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9d40 (1 active)
2014-06-12 14:46:00.770112 7f9d7815d700  5 _do_op 0x165ae300 seq 2633 osr(3.15d 0x276cea0)/0x276cea0 start
2014-06-12 14:46:00.770116 7f9d7815d700 10 _do_transaction on 0x96dad80
2014-06-12 14:46:00.770121 7f9d7815d700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.770224 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.770251 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:00.770311 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:00.770416 7f9d7815d700 15 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.770425 7f9d7815d700 10 _omap_rmkeys meta/1fe2b61b/pglog_3.15d/0//-1 r = 0
2014-06-12 14:46:00.770432 7f9d7815d700 15 _omap_setkeys meta/1fe2b61b/pglog_3.15d/0//-1
2014-06-12 14:46:00.770463 7f9d7815d700 15 _write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 3911680~282624
2014-06-12 14:46:00.770560 7f9d7815d700 10 lookup_strip_header done  cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.770697 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:00.771364 7f9d7815d700 10 _generic_write 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 3911680~282624 = 0
2014-06-12 14:46:00.771394 7f9d7815d700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.771407 7f9d7815d700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.771415 7f9d7815d700 15 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3
2014-06-12 14:46:00.771423 7f9d7815d700 10 _setattrs 3.15d_head/9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 = 0
2014-06-12 14:46:00.771527 7f9d7815d700 20 set_header setting 17237 cid 3.15d_head oid 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 parent seq 0
2014-06-12 14:46:00.771566 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:00.771581 7f9d7815d700 20 set_header setting 1998 cid meta oid 1fe2b61b/pglog_3.15d/0//-1 parent seq 0
2014-06-12 14:46:00.771592 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:00.772755 7f9d7815d700 10 _do_op 0x165ae300 seq 2633 r = 0, finisher 0x73fb580 0x4417300
2014-06-12 14:46:00.772770 7f9d7815d700 10 _finish_op 0x165ae300 seq 2633 osr(3.15d 0x276cea0)/0x276cea0
2014-06-12 14:46:00.772779 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9d40 (0 active)
2014-06-12 14:46:00.772786 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:00.772796 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] op_commit: 236173
2014-06-12 14:46:00.772840 7f9d7715b700  5 -- op tracker -- , seq: 747441, time: 2014-06-12 14:46:00.772830, event: op_commit, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.772868 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] op_applied: 236173
2014-06-12 14:46:00.772888 7f9d7795c700  5 -- op tracker -- , seq: 747441, time: 2014-06-12 14:46:00.772883, event: op_applied, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.772893 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] op_applied on primary on version 173'2853
2014-06-12 14:46:00.783049 7f9d5bec4700  5 -- op tracker -- , seq: 747442, time: 2014-06-12 14:46:00.782985, event: header_read, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ack, result = 0)
2014-06-12 14:46:00.783064 7f9d5bec4700  5 -- op tracker -- , seq: 747442, time: 2014-06-12 14:46:00.782987, event: throttled, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ack, result = 0)
2014-06-12 14:46:00.783077 7f9d5bec4700  5 -- op tracker -- , seq: 747442, time: 2014-06-12 14:46:00.783006, event: all_read, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ack, result = 0)
2014-06-12 14:46:00.783082 7f9d5bec4700  5 -- op tracker -- , seq: 747442, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ack, result = 0)
2014-06-12 14:46:00.783087 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:00.783091 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:00.783098 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e86400 prio 196 cost 0 latency 0.000113 osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ack, result = 0) v2
2014-06-12 14:46:00.783125 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:00.783144 7f9d5bec4700  5 -- op tracker -- , seq: 747443, time: 2014-06-12 14:46:00.783116, event: header_read, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.783150 7f9d5bec4700  5 -- op tracker -- , seq: 747443, time: 2014-06-12 14:46:00.783117, event: throttled, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.783155 7f9d5bec4700  5 -- op tracker -- , seq: 747443, time: 2014-06-12 14:46:00.783127, event: all_read, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.783137 7f9d69be4700 10 osd.2 173 dequeue_op 0x20e86400 prio 196 cost 0 latency 0.000152 osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ack, result = 0) v2 pg pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]
2014-06-12 14:46:00.783159 7f9d5bec4700  5 -- op tracker -- , seq: 747443, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.783165 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:00.783169 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:00.783172 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e86600 prio 196 cost 0 latency 0.000056 osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:00.783172 7f9d69be4700  5 -- op tracker -- , seq: 747442, time: 2014-06-12 14:46:00.783159, event: reached_pg, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ack, result = 0)
2014-06-12 14:46:00.783191 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:00.783183 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] handle_message: 0x20e86400
2014-06-12 14:46:00.783201 7f9d69be4700  5 -- op tracker -- , seq: 747442, time: 2014-06-12 14:46:00.783197, event: started, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ack, result = 0)
2014-06-12 14:46:00.783215 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] sub_op_modify_reply: tid 236173 op  ack_type 1 from 25
2014-06-12 14:46:00.783237 7f9d69be4700  5 -- op tracker -- , seq: 747441, time: 2014-06-12 14:46:00.783228, event: sub_op_applied_rec, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.783245 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] repop_all_applied: repop tid 236173 all applied 
2014-06-12 14:46:00.783257 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] eval_repop repgather(0x384e2540 173'2853 rep_tid=236173 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.783274 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]  sending ack on repgather(0x384e2540 173'2853 rep_tid=236173 committed?=0 applied?=1 lock=0 op=osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x959b400
2014-06-12 14:46:00.783306 7f9d69be4700 10 osd.2 173 dequeue_op 0x20e86400 finish
2014-06-12 14:46:00.783324 7f9d69be4700  5 -- op tracker -- , seq: 747442, time: 2014-06-12 14:46:00.783313, event: done, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ack, result = 0)
2014-06-12 14:46:00.783315 7f9d633d7700 10 osd.2 173 dequeue_op 0x20e86600 prio 196 cost 0 latency 0.000199 osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0) v2 pg pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]
2014-06-12 14:46:00.783339 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:00.783341 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.783340 7f9d633d7700  5 -- op tracker -- , seq: 747443, time: 2014-06-12 14:46:00.783337, event: reached_pg, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.783347 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] handle_message: 0x20e86600
2014-06-12 14:46:00.783362 7f9d633d7700  5 -- op tracker -- , seq: 747443, time: 2014-06-12 14:46:00.783357, event: started, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.783366 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] sub_op_modify_reply: tid 236173 op  ack_type 4 from 25
2014-06-12 14:46:00.783386 7f9d633d7700  5 -- op tracker -- , seq: 747441, time: 2014-06-12 14:46:00.783374, event: sub_op_commit_rec, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.783390 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] repop_all_committed: repop tid 236173 all committed 
2014-06-12 14:46:00.783396 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean] eval_repop repgather(0x384e2540 173'2853 rep_tid=236173 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:00.783406 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean] log_op_stats osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4 inb 282903 outb 0 rlat 0.014726 lat 0.014827
2014-06-12 14:46:00.783418 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean] publish_stats_to_osd 173:3716
2014-06-12 14:46:00.783426 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean]  sending commit on repgather(0x384e2540 173'2853 rep_tid=236173 committed?=1 applied?=1 lock=0 op=osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4) 0x2b42a00
2014-06-12 14:46:00.783446 7f9d633d7700  5 -- op tracker -- , seq: 747441, time: 2014-06-12 14:46:00.783442, event: commit_sent, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.783450 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean]  requeue_ops 
2014-06-12 14:46:00.783465 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  removing repgather(0x384e2540 173'2853 rep_tid=236173 committed?=1 applied?=1 op=osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.783473 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]    q front is repgather(0x384e2540 173'2853 rep_tid=236173 committed?=1 applied?=1 op=osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.783482 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] remove_repop repgather(0x384e2540 173'2853 rep_tid=236173 committed?=1 applied?=1 op=osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:00.783489 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  requeue_ops 
2014-06-12 14:46:00.783514 7f9d633d7700  5 -- op tracker -- , seq: 747441, time: 2014-06-12 14:46:00.783503, event: done, op: osd_op(client.4315.0:144769 rbd_data.10b26b8b4567.000000000000106e [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.9983c95d ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:00.783526 7f9d633d7700 10 osd.2 173 dequeue_op 0x20e86600 finish
2014-06-12 14:46:00.783534 7f9d633d7700  5 -- op tracker -- , seq: 747443, time: 2014-06-12 14:46:00.783531, event: done, op: osd_sub_op_reply(client.4315.0:144769 3.15d 9983c95d/rbd_data.10b26b8b4567.000000000000106e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:00.783539 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:00.783541 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:00.796930 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:00.796980 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:00.817465 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:00.817468 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:00.817470 7f9d74bfa700 20 osd.2 173 _dispatch 0x177044e0 ping v1
2014-06-12 14:46:00.817473 7f9d74bfa700 10 osd.2 173 ping from client.4303
2014-06-12 14:46:00.817475 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:00.817477 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:01.212143 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19406 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,4,4,30,4])
2014-06-12 14:46:01.212158 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19406 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,4,4,30,4])
2014-06-12 14:46:01.212622 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:01.212643 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:01.212670 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:01.212696 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:01.212711 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:01.212726 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:01.212742 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:01.212763 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:01.212812 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:01.212825 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:01.212842 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:01.212855 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:01.212888 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:01.212913 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:01.212934 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:01.212946 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:01.212965 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:01.212997 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:01.213014 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:01.213026 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:01.213669 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:01.214227 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:01.235442 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:01.235719 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:01.284009 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:01.284012 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:01.284013 7f9d74bfa700 20 osd.2 173 _dispatch 0x1e700d00 ping v1
2014-06-12 14:46:01.284016 7f9d74bfa700 10 osd.2 173 ping from client.4306
2014-06-12 14:46:01.284018 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:01.284019 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:01.314193 7f9d5a16f700  5 -- op tracker -- , seq: 747444, time: 2014-06-12 14:46:01.314136, event: header_read, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.314205 7f9d5a16f700  5 -- op tracker -- , seq: 747444, time: 2014-06-12 14:46:01.314139, event: throttled, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.314219 7f9d5a16f700  5 -- op tracker -- , seq: 747444, time: 2014-06-12 14:46:01.314154, event: all_read, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.314224 7f9d5a16f700  5 -- op tracker -- , seq: 747444, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.314228 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:01.314233 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:01.314239 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a902d00 prio 196 cost 0 latency 0.000103 osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0) v2
2014-06-12 14:46:01.314267 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.314283 7f9d5a16f700  5 -- op tracker -- , seq: 747445, time: 2014-06-12 14:46:01.314259, event: header_read, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.314279 7f9d643d9700 10 osd.2 173 dequeue_op 0x2a902d00 prio 196 cost 0 latency 0.000142 osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0) v2 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean]
2014-06-12 14:46:01.314293 7f9d5a16f700  5 -- op tracker -- , seq: 747445, time: 2014-06-12 14:46:01.314259, event: throttled, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.314301 7f9d643d9700  5 -- op tracker -- , seq: 747444, time: 2014-06-12 14:46:01.314297, event: reached_pg, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.314312 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean] handle_message: 0x2a902d00
2014-06-12 14:46:01.314316 7f9d5a16f700  5 -- op tracker -- , seq: 747445, time: 2014-06-12 14:46:01.314266, event: all_read, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.314338 7f9d5a16f700  5 -- op tracker -- , seq: 747445, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.314345 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:01.314350 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:01.314355 7f9d5a16f700 15 osd.2 173 enqueue_op 0x8dbcc00 prio 196 cost 0 latency 0.000096 osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:01.314349 7f9d643d9700  5 -- op tracker -- , seq: 747444, time: 2014-06-12 14:46:01.314326, event: started, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.314362 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean] sub_op_modify_reply: tid 235923 op  ack_type 1 from 24
2014-06-12 14:46:01.314379 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:01.314385 7f9d643d9700  5 -- op tracker -- , seq: 746657, time: 2014-06-12 14:46:01.314376, event: sub_op_applied_rec, op: osd_op(client.4292.0:142325 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3448832~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.314394 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean] repop_all_applied: repop tid 235923 all applied 
2014-06-12 14:46:01.314403 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean] eval_repop repgather(0x13d22480 173'2565 rep_tid=235923 committed?=0 applied?=1 lock=0 op=osd_op(client.4292.0:142325 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3448832~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.314415 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean]  sending ack on repgather(0x13d22480 173'2565 rep_tid=235923 committed?=0 applied?=1 lock=0 op=osd_op(client.4292.0:142325 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3448832~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4) 0x30af6500
2014-06-12 14:46:01.314439 7f9d643d9700 10 osd.2 173 dequeue_op 0x2a902d00 finish
2014-06-12 14:46:01.314448 7f9d643d9700  5 -- op tracker -- , seq: 747444, time: 2014-06-12 14:46:01.314445, event: done, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.314448 7f9d623d5700 10 osd.2 173 dequeue_op 0x8dbcc00 prio 196 cost 0 latency 0.000189 osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0) v2 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean]
2014-06-12 14:46:01.314464 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:01.314466 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.314465 7f9d623d5700  5 -- op tracker -- , seq: 747445, time: 2014-06-12 14:46:01.314462, event: reached_pg, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.314472 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean] handle_message: 0x8dbcc00
2014-06-12 14:46:01.314485 7f9d623d5700  5 -- op tracker -- , seq: 747445, time: 2014-06-12 14:46:01.314483, event: started, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.314488 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean] sub_op_modify_reply: tid 235923 op  ack_type 4 from 24
2014-06-12 14:46:01.314499 7f9d623d5700  5 -- op tracker -- , seq: 746657, time: 2014-06-12 14:46:01.314495, event: sub_op_commit_rec, op: osd_op(client.4292.0:142325 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3448832~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.314503 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2564 crt=0'0 lcod 173'2563 mlcod 173'2563 active+clean] repop_all_committed: repop tid 235923 all committed 
2014-06-12 14:46:01.314509 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2563 active+clean] eval_repop repgather(0x13d22480 173'2565 rep_tid=235923 committed?=1 applied?=1 lock=0 op=osd_op(client.4292.0:142325 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3448832~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.314519 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2563 active+clean] log_op_stats osd_op(client.4292.0:142325 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3448832~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4 inb 516373 outb 0 rlat 37.872784 lat 37.872864
2014-06-12 14:46:01.314530 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2563 active+clean] publish_stats_to_osd 173:2746
2014-06-12 14:46:01.314539 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2563 active+clean]  sending commit on repgather(0x13d22480 173'2565 rep_tid=235923 committed?=1 applied?=1 lock=0 op=osd_op(client.4292.0:142325 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3448832~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4) 0x29192000
2014-06-12 14:46:01.314561 7f9d623d5700  5 -- op tracker -- , seq: 746657, time: 2014-06-12 14:46:01.314558, event: commit_sent, op: osd_op(client.4292.0:142325 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3448832~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.314567 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2563 active+clean]  requeue_ops 
2014-06-12 14:46:01.314574 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean]  removing repgather(0x13d22480 173'2565 rep_tid=235923 committed?=1 applied?=1 op=osd_op(client.4292.0:142325 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3448832~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.314583 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean]    q front is repgather(0x13d22480 173'2565 rep_tid=235923 committed?=1 applied?=1 op=osd_op(client.4292.0:142325 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3448832~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.314591 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean] remove_repop repgather(0x13d22480 173'2565 rep_tid=235923 committed?=1 applied?=1 op=osd_op(client.4292.0:142325 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3448832~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.314599 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean]  requeue_ops 
2014-06-12 14:46:01.314612 7f9d623d5700  5 -- op tracker -- , seq: 746657, time: 2014-06-12 14:46:01.314609, event: done, op: osd_op(client.4292.0:142325 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3448832~516096] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.314629 7f9d623d5700 10 osd.2 173 dequeue_op 0x8dbcc00 finish
2014-06-12 14:46:01.314637 7f9d623d5700  5 -- op tracker -- , seq: 747445, time: 2014-06-12 14:46:01.314634, event: done, op: osd_sub_op_reply(client.4292.0:142325 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.314644 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.314647 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.321659 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:01.321671 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:01.321673 7f9d74bfa700 20 osd.2 173 _dispatch 0x1fd709c0 ping v1
2014-06-12 14:46:01.321676 7f9d74bfa700 10 osd.2 173 ping from client.4308
2014-06-12 14:46:01.321678 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:01.321679 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:01.325299 7f9d5a16f700  5 -- op tracker -- , seq: 747446, time: 2014-06-12 14:46:01.325240, event: header_read, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.325312 7f9d5a16f700  5 -- op tracker -- , seq: 747446, time: 2014-06-12 14:46:01.325243, event: throttled, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.325320 7f9d5a16f700  5 -- op tracker -- , seq: 747446, time: 2014-06-12 14:46:01.325259, event: all_read, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.325328 7f9d5a16f700  5 -- op tracker -- , seq: 747446, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.325334 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:01.325340 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:01.325348 7f9d5a16f700 15 osd.2 173 enqueue_op 0x8dbd500 prio 196 cost 0 latency 0.000108 osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0) v2
2014-06-12 14:46:01.325381 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.325391 7f9d68be2700 10 osd.2 173 dequeue_op 0x8dbd500 prio 196 cost 0 latency 0.000150 osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0) v2 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean]
2014-06-12 14:46:01.325408 7f9d5a16f700  5 -- op tracker -- , seq: 747447, time: 2014-06-12 14:46:01.325373, event: header_read, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.325526 7f9d5a16f700  5 -- op tracker -- , seq: 747447, time: 2014-06-12 14:46:01.325373, event: throttled, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.325537 7f9d5a16f700  5 -- op tracker -- , seq: 747447, time: 2014-06-12 14:46:01.325386, event: all_read, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.325548 7f9d5a16f700  5 -- op tracker -- , seq: 747447, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.325554 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:01.325560 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:01.325559 7f9d68be2700  5 -- op tracker -- , seq: 747446, time: 2014-06-12 14:46:01.325408, event: reached_pg, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.325567 7f9d5a16f700 15 osd.2 173 enqueue_op 0x5ee9e00 prio 196 cost 0 latency 0.000193 osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:01.325574 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean] handle_message: 0x8dbd500
2014-06-12 14:46:01.325590 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:01.325590 7f9d68be2700  5 -- op tracker -- , seq: 747446, time: 2014-06-12 14:46:01.325586, event: started, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.325596 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean] sub_op_modify_reply: tid 235983 op  ack_type 1 from 24
2014-06-12 14:46:01.325625 7f9d68be2700  5 -- op tracker -- , seq: 746826, time: 2014-06-12 14:46:01.325607, event: sub_op_applied_rec, op: osd_op(client.4292.0:142326 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3964928~229376] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.325635 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean] repop_all_applied: repop tid 235983 all applied 
2014-06-12 14:46:01.325646 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean] eval_repop repgather(0x2a12fbc0 173'2566 rep_tid=235983 committed?=0 applied?=1 lock=0 op=osd_op(client.4292.0:142326 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3964928~229376] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.325661 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean]  sending ack on repgather(0x2a12fbc0 173'2566 rep_tid=235983 committed?=0 applied?=1 lock=0 op=osd_op(client.4292.0:142326 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3964928~229376] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4) 0x239e2500
2014-06-12 14:46:01.325696 7f9d68be2700 10 osd.2 173 dequeue_op 0x8dbd500 finish
2014-06-12 14:46:01.325705 7f9d68be2700  5 -- op tracker -- , seq: 747446, time: 2014-06-12 14:46:01.325702, event: done, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ack, result = 0)
2014-06-12 14:46:01.325704 7f9d61bd4700 10 osd.2 173 dequeue_op 0x5ee9e00 prio 196 cost 0 latency 0.000331 osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0) v2 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean]
2014-06-12 14:46:01.325720 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:01.325723 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.325722 7f9d61bd4700  5 -- op tracker -- , seq: 747447, time: 2014-06-12 14:46:01.325718, event: reached_pg, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.325729 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean] handle_message: 0x5ee9e00
2014-06-12 14:46:01.325743 7f9d61bd4700  5 -- op tracker -- , seq: 747447, time: 2014-06-12 14:46:01.325740, event: started, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.325751 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean] sub_op_modify_reply: tid 235983 op  ack_type 4 from 24
2014-06-12 14:46:01.325769 7f9d61bd4700  5 -- op tracker -- , seq: 746826, time: 2014-06-12 14:46:01.325763, event: sub_op_commit_rec, op: osd_op(client.4292.0:142326 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3964928~229376] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.325778 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2565 crt=0'0 lcod 173'2564 mlcod 173'2564 active+clean] repop_all_committed: repop tid 235983 all committed 
2014-06-12 14:46:01.325785 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2564 active+clean] eval_repop repgather(0x2a12fbc0 173'2566 rep_tid=235983 committed?=1 applied?=1 lock=0 op=osd_op(client.4292.0:142326 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3964928~229376] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.325796 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2564 active+clean] log_op_stats osd_op(client.4292.0:142326 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3964928~229376] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4 inb 229653 outb 0 rlat 32.661000 lat 32.661102
2014-06-12 14:46:01.325808 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2564 active+clean] publish_stats_to_osd 173:2747
2014-06-12 14:46:01.325817 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2564 active+clean]  sending commit on repgather(0x2a12fbc0 173'2566 rep_tid=235983 committed?=1 applied?=1 lock=0 op=osd_op(client.4292.0:142326 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3964928~229376] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4) 0x9657400
2014-06-12 14:46:01.325840 7f9d61bd4700  5 -- op tracker -- , seq: 746826, time: 2014-06-12 14:46:01.325835, event: commit_sent, op: osd_op(client.4292.0:142326 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3964928~229376] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.325854 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2564 active+clean]  requeue_ops 
2014-06-12 14:46:01.325864 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean]  removing repgather(0x2a12fbc0 173'2566 rep_tid=235983 committed?=1 applied?=1 op=osd_op(client.4292.0:142326 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3964928~229376] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.325873 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean]    q front is repgather(0x2a12fbc0 173'2566 rep_tid=235983 committed?=1 applied?=1 op=osd_op(client.4292.0:142326 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3964928~229376] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.325882 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean] remove_repop repgather(0x2a12fbc0 173'2566 rep_tid=235983 committed?=1 applied?=1 op=osd_op(client.4292.0:142326 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3964928~229376] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.325889 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean]  requeue_ops 
2014-06-12 14:46:01.325908 7f9d61bd4700  5 -- op tracker -- , seq: 746826, time: 2014-06-12 14:46:01.325904, event: done, op: osd_op(client.4292.0:142326 rbd_data.109b6b8b4567.0000000000001039 [set-alloc-hint object_size 4194304 write_size 4194304,write 3964928~229376] 3.8eb5f043 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.325989 7f9d61bd4700 10 osd.2 173 dequeue_op 0x5ee9e00 finish
2014-06-12 14:46:01.325997 7f9d61bd4700  5 -- op tracker -- , seq: 747447, time: 2014-06-12 14:46:01.325994, event: done, op: osd_sub_op_reply(client.4292.0:142326 3.43 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.326029 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.326033 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.326238 7f9d4a3c5700  5 -- op tracker -- , seq: 747448, time: 2014-06-12 14:46:01.325499, event: header_read, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.326252 7f9d4a3c5700  5 -- op tracker -- , seq: 747448, time: 2014-06-12 14:46:01.325502, event: throttled, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.326261 7f9d4a3c5700  5 -- op tracker -- , seq: 747448, time: 2014-06-12 14:46:01.326209, event: all_read, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.326304 7f9d4a3c5700  5 -- op tracker -- , seq: 747448, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.326310 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:46:01.326333 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x3b42a700 prio 63 cost 294912 latency 0.000833 osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.326356 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.326364 7f9d673df700 10 osd.2 173 dequeue_op 0x3b42a700 prio 63 cost 294912 latency 0.000865 osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]
2014-06-12 14:46:01.326382 7f9d673df700  5 -- op tracker -- , seq: 747448, time: 2014-06-12 14:46:01.326378, event: reached_pg, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.326388 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.326398 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] handle_message: 0x3b42a700
2014-06-12 14:46:01.326404 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] do_op osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.326420 7f9d673df700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_'
2014-06-12 14:46:01.326646 7f9d673df700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_' = 0
2014-06-12 14:46:01.326660 7f9d673df700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset'
2014-06-12 14:46:01.326725 7f9d673df700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset' = 0
2014-06-12 14:46:01.326733 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] populate_obc_watchers edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.326741 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.326747 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] get_object_context: creating obc from disk: 0x30f138c0
2014-06-12 14:46:01.326752 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] get_object_context: 0x30f138c0 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 rwstate(none n=0 w=0) oi: edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'990 client.4296.0:40806 wrlock_by=unknown.0.0:0 dirty s 4194304 uv990) ssc: 0x407e380 snapset: 0=[]:[]+head
2014-06-12 14:46:01.326762 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] find_object_context edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 @head oi=edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'990 client.4296.0:40806 wrlock_by=unknown.0.0:0 dirty s 4194304 uv990)
2014-06-12 14:46:01.326779 7f9d673df700  5 -- op tracker -- , seq: 747448, time: 2014-06-12 14:46:01.326775, event: started, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.326784 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] execute_ctx 0x7ad1e00
2014-06-12 14:46:01.326794 7f9d673df700  5 -- op tracker -- , seq: 747448, time: 2014-06-12 14:46:01.326792, event: started, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.326798 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] do_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] ov 173'990 av 173'3041 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.326807 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] do_osd_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912]
2014-06-12 14:46:01.326813 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.326821 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] do_osd_op  write 0~294912
2014-06-12 14:46:01.326831 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:46:01.326838 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 snapset=0x407e3b8  snapc=0=[]
2014-06-12 14:46:01.326845 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.326850 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] finish_ctx edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 0x7ad1e00 op modify  
2014-06-12 14:46:01.326858 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]  set mtime to 2014-06-12 08:45:03.706152
2014-06-12 14:46:01.326870 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]  final snapset 0=[]:[]+head in edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.326881 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.326886 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] calc_trim_to 79'39 -> 79'40
2014-06-12 14:46:01.326892 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] new_repop rep_tid 236174 on osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.326901 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] issue_repop rep_tid 236174 o edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.326915 7f9d673df700  5 -- op tracker -- , seq: 747448, time: 2014-06-12 14:46:01.326913, event: waiting for subops from 21, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.326935 7f9d673df700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:01.326949 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3040 (79'39,173'3040] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] append_log log((79'39,173'3040], crt=0'0) [173'3041 (173'990) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143513 2014-06-12 08:45:03.706152]
2014-06-12 14:46:01.326964 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'39,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] add_log_entry 173'3041 (173'990) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143513 2014-06-12 08:45:03.706152
2014-06-12 14:46:01.326976 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'39,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.326984 7f9d673df700 10 trim log((79'39,173'3041], crt=0'0) to 79'40
2014-06-12 14:46:01.327002 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3041, trimmed: 79'40
2014-06-12 14:46:01.327021 7f9d673df700  5 queue_transactions existing osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.327025 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.327027 7f9d673df700  5 queue_op 0x26f7db00 seq 2732 osr(3.170 0x276ce20) 297240 bytes   (queue has 1 ops and 297240 bytes)
2014-06-12 14:46:01.327037 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cb680 (1 active)
2014-06-12 14:46:01.327043 7f9d79960700  5 _do_op 0x26f7db00 seq 2732 osr(3.170 0x276ce20)/0x276ce20 start
2014-06-12 14:46:01.327046 7f9d79960700 10 _do_transaction on 0x10193ec0
2014-06-12 14:46:01.327032 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] eval_repop repgather(0x15e57440 173'3041 rep_tid=236174 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.327048 7f9d673df700 10 osd.2 173 dequeue_op 0x3b42a700 finish
2014-06-12 14:46:01.327052 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.327052 7f9d79960700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.327054 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.327134 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.327158 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.327203 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.327279 7f9d79960700 15 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.327286 7f9d79960700 10 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1 r = 0
2014-06-12 14:46:01.327291 7f9d79960700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.327303 7f9d79960700 15 _write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 0~294912
2014-06-12 14:46:01.327395 7f9d79960700 10 lookup_strip_header done  cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.327509 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:01.328219 7f9d79960700 10 _generic_write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 0~294912 = 0
2014-06-12 14:46:01.328246 7f9d79960700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.328255 7f9d79960700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.328258 7f9d79960700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.328263 7f9d79960700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.328365 7f9d79960700 20 set_header setting 16954 cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 parent seq 0
2014-06-12 14:46:01.328397 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.328407 7f9d79960700 20 set_header setting 2002 cid meta oid 1fe2ad5a/pglog_3.170/0//-1 parent seq 0
2014-06-12 14:46:01.328413 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:01.329332 7f9d79960700 10 _do_op 0x26f7db00 seq 2732 r = 0, finisher 0x1e9f0b00 0x381e7b40
2014-06-12 14:46:01.329346 7f9d79960700 10 _finish_op 0x26f7db00 seq 2732 osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.329353 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cb680 (0 active)
2014-06-12 14:46:01.329359 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.329352 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] op_commit: 236174
2014-06-12 14:46:01.329380 7f9d7715b700  5 -- op tracker -- , seq: 747448, time: 2014-06-12 14:46:01.329371, event: op_commit, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.329421 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] op_applied: 236174
2014-06-12 14:46:01.329452 7f9d7795c700  5 -- op tracker -- , seq: 747448, time: 2014-06-12 14:46:01.329444, event: op_applied, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.329462 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] op_applied on primary on version 173'3041
2014-06-12 14:46:01.331601 7f9d44101700  5 -- op tracker -- , seq: 747449, time: 2014-06-12 14:46:01.331566, event: header_read, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.331614 7f9d44101700  5 -- op tracker -- , seq: 747449, time: 2014-06-12 14:46:01.331566, event: throttled, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.331622 7f9d44101700  5 -- op tracker -- , seq: 747449, time: 2014-06-12 14:46:01.331575, event: all_read, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.331630 7f9d44101700  5 -- op tracker -- , seq: 747449, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.331635 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:01.331640 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.331649 7f9d44101700 15 osd.2 173 enqueue_op 0x23e09900 prio 196 cost 0 latency 0.000082 osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2
2014-06-12 14:46:01.331676 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.331685 7f9d663dd700 10 osd.2 173 dequeue_op 0x23e09900 prio 196 cost 0 latency 0.000118 osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2 pg pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]
2014-06-12 14:46:01.331705 7f9d663dd700  5 -- op tracker -- , seq: 747449, time: 2014-06-12 14:46:01.331701, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.331713 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] handle_message: 0x23e09900
2014-06-12 14:46:01.331729 7f9d663dd700  5 -- op tracker -- , seq: 747449, time: 2014-06-12 14:46:01.331725, event: started, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.331734 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] sub_op_modify_reply: tid 236174 op  ack_type 1 from 21
2014-06-12 14:46:01.331753 7f9d663dd700  5 -- op tracker -- , seq: 747448, time: 2014-06-12 14:46:01.331747, event: sub_op_applied_rec, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.331760 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] repop_all_applied: repop tid 236174 all applied 
2014-06-12 14:46:01.331771 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] eval_repop repgather(0x15e57440 173'3041 rep_tid=236174 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.331787 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]  sending ack on repgather(0x15e57440 173'3041 rep_tid=236174 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x6824a00
2014-06-12 14:46:01.331913 7f9d663dd700 10 osd.2 173 dequeue_op 0x23e09900 finish
2014-06-12 14:46:01.331923 7f9d663dd700  5 -- op tracker -- , seq: 747449, time: 2014-06-12 14:46:01.331919, event: done, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.331936 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.331939 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.332495 7f9d44101700  5 -- op tracker -- , seq: 747450, time: 2014-06-12 14:46:01.332464, event: header_read, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.332506 7f9d44101700  5 -- op tracker -- , seq: 747450, time: 2014-06-12 14:46:01.332464, event: throttled, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.332514 7f9d44101700  5 -- op tracker -- , seq: 747450, time: 2014-06-12 14:46:01.332473, event: all_read, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.332525 7f9d44101700  5 -- op tracker -- , seq: 747450, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.332530 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:01.332535 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.332540 7f9d44101700 15 osd.2 173 enqueue_op 0x23e08f00 prio 196 cost 0 latency 0.000076 osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:01.332781 7f9d4a3c5700  5 -- op tracker -- , seq: 747451, time: 2014-06-12 14:46:01.331140, event: header_read, op: osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.332795 7f9d4a3c5700  5 -- op tracker -- , seq: 747451, time: 2014-06-12 14:46:01.331143, event: throttled, op: osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.332808 7f9d4a3c5700  5 -- op tracker -- , seq: 747451, time: 2014-06-12 14:46:01.332746, event: all_read, op: osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.332819 7f9d4a3c5700  5 -- op tracker -- , seq: 747451, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.332827 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:46:01.332853 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x3b42a900 prio 63 cost 516096 latency 0.001713 osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.332889 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.332897 7f9d67be0700 10 osd.2 173 dequeue_op 0x23e08f00 prio 196 cost 0 latency 0.000433 osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 pg pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]
2014-06-12 14:46:01.332918 7f9d67be0700  5 -- op tracker -- , seq: 747450, time: 2014-06-12 14:46:01.332915, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.332928 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] handle_message: 0x23e08f00
2014-06-12 14:46:01.332944 7f9d67be0700  5 -- op tracker -- , seq: 747450, time: 2014-06-12 14:46:01.332941, event: started, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.332950 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] sub_op_modify_reply: tid 236174 op  ack_type 4 from 21
2014-06-12 14:46:01.332971 7f9d67be0700  5 -- op tracker -- , seq: 747448, time: 2014-06-12 14:46:01.332965, event: sub_op_commit_rec, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.332981 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] repop_all_committed: repop tid 236174 all committed 
2014-06-12 14:46:01.332995 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3039 active+clean] eval_repop repgather(0x15e57440 173'3041 rep_tid=236174 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.333015 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3039 active+clean] log_op_stats osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 inb 295191 outb 0 rlat 0.006412 lat 0.007514
2014-06-12 14:46:01.333032 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3039 active+clean] publish_stats_to_osd 173:3827
2014-06-12 14:46:01.333045 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3039 active+clean]  sending commit on repgather(0x15e57440 173'3041 rep_tid=236174 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x33770a00
2014-06-12 14:46:01.333080 7f9d67be0700  5 -- op tracker -- , seq: 747448, time: 2014-06-12 14:46:01.333075, event: commit_sent, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.333091 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3039 active+clean]  requeue_ops 
2014-06-12 14:46:01.333102 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]  removing repgather(0x15e57440 173'3041 rep_tid=236174 committed?=1 applied?=1 op=osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.333119 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]    q front is repgather(0x15e57440 173'3041 rep_tid=236174 committed?=1 applied?=1 op=osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.333135 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] remove_repop repgather(0x15e57440 173'3041 rep_tid=236174 committed?=1 applied?=1 op=osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.333150 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]  requeue_ops 
2014-06-12 14:46:01.333177 7f9d67be0700  5 -- op tracker -- , seq: 747448, time: 2014-06-12 14:46:01.333172, event: done, op: osd_op(client.4296.0:143513 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.333199 7f9d67be0700 10 osd.2 173 dequeue_op 0x23e08f00 finish
2014-06-12 14:46:01.333210 7f9d67be0700  5 -- op tracker -- , seq: 747450, time: 2014-06-12 14:46:01.333206, event: done, op: osd_sub_op_reply(client.4296.0:143513 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.333222 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.333231 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.333237 7f9d67be0700 10 osd.2 173 dequeue_op 0x3b42a900 prio 63 cost 516096 latency 0.002097 osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]
2014-06-12 14:46:01.333259 7f9d67be0700  5 -- op tracker -- , seq: 747451, time: 2014-06-12 14:46:01.333254, event: reached_pg, op: osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.333269 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.333284 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] handle_message: 0x3b42a900
2014-06-12 14:46:01.333294 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] do_op osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.333319 7f9d67be0700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_'
2014-06-12 14:46:01.333593 7f9d67be0700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_' = 0
2014-06-12 14:46:01.333622 7f9d67be0700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset'
2014-06-12 14:46:01.333724 7f9d67be0700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset' = 0
2014-06-12 14:46:01.333735 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] populate_obc_watchers edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.333751 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.333763 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] get_object_context: creating obc from disk: 0x21ef0840
2014-06-12 14:46:01.333775 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] get_object_context: 0x21ef0840 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 rwstate(none n=0 w=0) oi: edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3041 client.4296.0:143513 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3041) ssc: 0x407e380 snapset: 0=[]:[]+head
2014-06-12 14:46:01.333792 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] find_object_context edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 @head oi=edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3041 client.4296.0:143513 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3041)
2014-06-12 14:46:01.333818 7f9d67be0700  5 -- op tracker -- , seq: 747451, time: 2014-06-12 14:46:01.333812, event: started, op: osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.333828 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] execute_ctx 0x7ad1e00
2014-06-12 14:46:01.333845 7f9d67be0700  5 -- op tracker -- , seq: 747451, time: 2014-06-12 14:46:01.333840, event: started, op: osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.333854 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] do_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] ov 173'3041 av 173'3042 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.333869 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] do_osd_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096]
2014-06-12 14:46:01.333881 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.333895 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] do_osd_op  write 294912~516096
2014-06-12 14:46:01.333911 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:46:01.333924 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 snapset=0x407e3b8  snapc=0=[]
2014-06-12 14:46:01.333936 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.333947 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] finish_ctx edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 0x7ad1e00 op modify  
2014-06-12 14:46:01.333960 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]  set mtime to 2014-06-12 08:45:03.711388
2014-06-12 14:46:01.333982 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]  final snapset 0=[]:[]+head in edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.334001 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.334014 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] calc_trim_to 79'40 -> 79'41
2014-06-12 14:46:01.334024 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] new_repop rep_tid 236175 on osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.334041 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] issue_repop rep_tid 236175 o edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.334066 7f9d67be0700  5 -- op tracker -- , seq: 747451, time: 2014-06-12 14:46:01.334061, event: waiting for subops from 21, op: osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.334092 7f9d67be0700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:01.334116 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3041 (79'40,173'3041] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] append_log log((79'40,173'3041], crt=0'0) [173'3042 (173'3041) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143514 2014-06-12 08:45:03.711388]
2014-06-12 14:46:01.334141 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'40,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] add_log_entry 173'3042 (173'3041) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143514 2014-06-12 08:45:03.711388
2014-06-12 14:46:01.334163 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'40,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.334178 7f9d67be0700 10 trim log((79'40,173'3042], crt=0'0) to 79'41
2014-06-12 14:46:01.334204 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3042, trimmed: 79'41
2014-06-12 14:46:01.334230 7f9d67be0700  5 queue_transactions existing osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.334235 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.334237 7f9d67be0700  5 queue_op 0x54f8240 seq 2733 osr(3.170 0x276ce20) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:01.334244 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] eval_repop repgather(0x32b68300 173'3042 rep_tid=236175 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.334263 7f9d67be0700 10 osd.2 173 dequeue_op 0x3b42a900 finish
2014-06-12 14:46:01.334268 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.334271 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.334279 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cb680 (1 active)
2014-06-12 14:46:01.334289 7f9d8116f700  5 _do_op 0x54f8240 seq 2733 osr(3.170 0x276ce20)/0x276ce20 start
2014-06-12 14:46:01.334293 7f9d8116f700 10 _do_transaction on 0x10192240
2014-06-12 14:46:01.334299 7f9d8116f700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.334399 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.334421 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.334484 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.334615 7f9d8116f700 15 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.334628 7f9d8116f700 10 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1 r = 0
2014-06-12 14:46:01.334635 7f9d8116f700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.334654 7f9d8116f700 15 _write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 294912~516096
2014-06-12 14:46:01.334805 7f9d8116f700 10 lookup_strip_header done  cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.334952 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:01.337142 7f9d8116f700 10 _generic_write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 294912~516096 = 0
2014-06-12 14:46:01.337205 7f9d8116f700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.337223 7f9d8116f700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.337236 7f9d8116f700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.337247 7f9d8116f700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.337463 7f9d8116f700 20 set_header setting 16954 cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 parent seq 0
2014-06-12 14:46:01.337537 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.337555 7f9d8116f700 20 set_header setting 2002 cid meta oid 1fe2ad5a/pglog_3.170/0//-1 parent seq 0
2014-06-12 14:46:01.337570 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:01.337634 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.340002 7f9d8116f700 10 _do_op 0x54f8240 seq 2733 r = 0, finisher 0x29fc14e0 0x3cb10c0
2014-06-12 14:46:01.340025 7f9d8116f700 10 _finish_op 0x54f8240 seq 2733 osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.340033 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cb680 (0 active)
2014-06-12 14:46:01.340045 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.340031 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] op_commit: 236175
2014-06-12 14:46:01.340068 7f9d7715b700  5 -- op tracker -- , seq: 747451, time: 2014-06-12 14:46:01.340057, event: op_commit, op: osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.340085 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] op_applied: 236175
2014-06-12 14:46:01.340110 7f9d7795c700  5 -- op tracker -- , seq: 747451, time: 2014-06-12 14:46:01.340104, event: op_applied, op: osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.340121 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] op_applied on primary on version 173'3042
2014-06-12 14:46:01.341720 7f9d44101700  5 -- op tracker -- , seq: 747452, time: 2014-06-12 14:46:01.341664, event: header_read, op: osd_sub_op_reply(client.4296.0:143514 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.341732 7f9d44101700  5 -- op tracker -- , seq: 747452, time: 2014-06-12 14:46:01.341666, event: throttled, op: osd_sub_op_reply(client.4296.0:143514 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.341754 7f9d44101700  5 -- op tracker -- , seq: 747452, time: 2014-06-12 14:46:01.341678, event: all_read, op: osd_sub_op_reply(client.4296.0:143514 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.341762 7f9d44101700  5 -- op tracker -- , seq: 747452, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143514 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.341766 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143514 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:01.341771 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.341778 7f9d44101700 15 osd.2 173 enqueue_op 0x23e08400 prio 196 cost 0 latency 0.000114 osd_sub_op_reply(client.4296.0:143514 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:01.341812 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.341820 7f9d62bd6700 10 osd.2 173 dequeue_op 0x23e08400 prio 196 cost 0 latency 0.000155 osd_sub_op_reply(client.4296.0:143514 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 pg pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]
2014-06-12 14:46:01.341847 7f9d62bd6700  5 -- op tracker -- , seq: 747452, time: 2014-06-12 14:46:01.341844, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143514 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.341855 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] handle_message: 0x23e08400
2014-06-12 14:46:01.341873 7f9d62bd6700  5 -- op tracker -- , seq: 747452, time: 2014-06-12 14:46:01.341869, event: started, op: osd_sub_op_reply(client.4296.0:143514 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.341879 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] sub_op_modify_reply: tid 236175 op  ack_type 4 from 21
2014-06-12 14:46:01.341900 7f9d62bd6700  5 -- op tracker -- , seq: 747451, time: 2014-06-12 14:46:01.341894, event: sub_op_commit_rec, op: osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.341908 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] repop_all_applied: repop tid 236175 all applied 
2014-06-12 14:46:01.341935 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] eval_repop repgather(0x32b68300 173'3042 rep_tid=236175 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.341954 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]  sending ack on repgather(0x32b68300 173'3042 rep_tid=236175 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x205c500
2014-06-12 14:46:01.341987 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] repop_all_committed: repop tid 236175 all committed 
2014-06-12 14:46:01.341996 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3040 active+clean] eval_repop repgather(0x32b68300 173'3042 rep_tid=236175 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.342015 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3040 active+clean] log_op_stats osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.010846 lat 0.010874
2014-06-12 14:46:01.342035 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3040 active+clean] publish_stats_to_osd 173:3828
2014-06-12 14:46:01.342058 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3040 active+clean]  sending commit on repgather(0x32b68300 173'3042 rep_tid=236175 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x16cf3680
2014-06-12 14:46:01.342089 7f9d62bd6700  5 -- op tracker -- , seq: 747451, time: 2014-06-12 14:46:01.342083, event: commit_sent, op: osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.342096 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3040 active+clean]  requeue_ops 
2014-06-12 14:46:01.342107 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]  removing repgather(0x32b68300 173'3042 rep_tid=236175 committed?=1 applied?=1 op=osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.342123 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]    q front is repgather(0x32b68300 173'3042 rep_tid=236175 committed?=1 applied?=1 op=osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.342140 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] remove_repop repgather(0x32b68300 173'3042 rep_tid=236175 committed?=1 applied?=1 op=osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.342156 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]  requeue_ops 
2014-06-12 14:46:01.342183 7f9d62bd6700  5 -- op tracker -- , seq: 747451, time: 2014-06-12 14:46:01.342178, event: done, op: osd_op(client.4296.0:143514 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.342198 7f9d62bd6700 10 osd.2 173 dequeue_op 0x23e08400 finish
2014-06-12 14:46:01.342207 7f9d62bd6700  5 -- op tracker -- , seq: 747452, time: 2014-06-12 14:46:01.342203, event: done, op: osd_sub_op_reply(client.4296.0:143514 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.342217 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.342219 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.345593 7f9d4a3c5700  5 -- op tracker -- , seq: 747453, time: 2014-06-12 14:46:01.344694, event: header_read, op: osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.345609 7f9d4a3c5700  5 -- op tracker -- , seq: 747453, time: 2014-06-12 14:46:01.344696, event: throttled, op: osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.345625 7f9d4a3c5700  5 -- op tracker -- , seq: 747453, time: 2014-06-12 14:46:01.345552, event: all_read, op: osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.345636 7f9d4a3c5700  5 -- op tracker -- , seq: 747453, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.345644 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:46:01.345680 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x329ea100 prio 63 cost 516096 latency 0.000986 osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.345739 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.345750 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x329ea100 prio 63 cost 516096 latency 0.001056 osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]
2014-06-12 14:46:01.345778 7f9d6b3e7700  5 -- op tracker -- , seq: 747453, time: 2014-06-12 14:46:01.345774, event: reached_pg, op: osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.345789 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.345801 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] handle_message: 0x329ea100
2014-06-12 14:46:01.345808 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] do_op osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.345827 7f9d6b3e7700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_'
2014-06-12 14:46:01.345990 7f9d6b3e7700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_' = 0
2014-06-12 14:46:01.346004 7f9d6b3e7700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset'
2014-06-12 14:46:01.346067 7f9d6b3e7700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset' = 0
2014-06-12 14:46:01.346076 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] populate_obc_watchers edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.346084 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.346090 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] get_object_context: creating obc from disk: 0x14e7adc0
2014-06-12 14:46:01.346096 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] get_object_context: 0x14e7adc0 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 rwstate(none n=0 w=0) oi: edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3042 client.4296.0:143514 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3042) ssc: 0x16ee50a0 snapset: 0=[]:[]+head
2014-06-12 14:46:01.346111 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] find_object_context edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 @head oi=edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3042 client.4296.0:143514 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3042)
2014-06-12 14:46:01.346129 7f9d6b3e7700  5 -- op tracker -- , seq: 747453, time: 2014-06-12 14:46:01.346125, event: started, op: osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.346135 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] execute_ctx 0x2d2a6000
2014-06-12 14:46:01.346146 7f9d6b3e7700  5 -- op tracker -- , seq: 747453, time: 2014-06-12 14:46:01.346143, event: started, op: osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.346151 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] do_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] ov 173'3042 av 173'3043 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.346160 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] do_osd_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096]
2014-06-12 14:46:01.346166 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.346175 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] do_osd_op  write 811008~516096
2014-06-12 14:46:01.346186 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:46:01.346194 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 snapset=0x16ee50d8  snapc=0=[]
2014-06-12 14:46:01.346202 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.346208 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] finish_ctx edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 0x2d2a6000 op modify  
2014-06-12 14:46:01.346216 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]  set mtime to 2014-06-12 08:45:03.724954
2014-06-12 14:46:01.346232 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]  final snapset 0=[]:[]+head in edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.346245 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.346253 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] calc_trim_to 79'41 -> 79'42
2014-06-12 14:46:01.346259 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] new_repop rep_tid 236176 on osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.346269 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] issue_repop rep_tid 236176 o edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.346294 7f9d6b3e7700  5 -- op tracker -- , seq: 747453, time: 2014-06-12 14:46:01.346289, event: waiting for subops from 21, op: osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.346317 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:01.346342 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3042 (79'41,173'3042] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] append_log log((79'41,173'3042], crt=0'0) [173'3043 (173'3042) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143515 2014-06-12 08:45:03.724954]
2014-06-12 14:46:01.346367 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'41,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] add_log_entry 173'3043 (173'3042) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143515 2014-06-12 08:45:03.724954
2014-06-12 14:46:01.346390 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'41,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.346407 7f9d6b3e7700 10 trim log((79'41,173'3043], crt=0'0) to 79'42
2014-06-12 14:46:01.346436 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3043, trimmed: 79'42
2014-06-12 14:46:01.346464 7f9d6b3e7700  5 queue_transactions existing osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.346486 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.346488 7f9d6b3e7700  5 queue_op 0xb045860 seq 2734 osr(3.170 0x276ce20) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:01.346502 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cb680 (1 active)
2014-06-12 14:46:01.346498 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] eval_repop repgather(0x1696d5c0 173'3043 rep_tid=236176 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.346510 7f9d7d968700  5 _do_op 0xb045860 seq 2734 osr(3.170 0x276ce20)/0x276ce20 start
2014-06-12 14:46:01.346518 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x329ea100 finish
2014-06-12 14:46:01.346520 7f9d7d968700 10 _do_transaction on 0x2a37a540
2014-06-12 14:46:01.346523 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.346525 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.346526 7f9d7d968700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.346628 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.346746 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.346826 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.346856 7f9d7d968700 15 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.346867 7f9d7d968700 10 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1 r = 0
2014-06-12 14:46:01.346890 7f9d7d968700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.346919 7f9d7d968700 15 _write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 811008~516096
2014-06-12 14:46:01.347090 7f9d7d968700 10 lookup_strip_header done  cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.347208 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:01.349009 7f9d7d968700 10 _generic_write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 811008~516096 = 0
2014-06-12 14:46:01.349058 7f9d7d968700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.349075 7f9d7d968700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.349083 7f9d7d968700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.349092 7f9d7d968700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.349302 7f9d7d968700 20 set_header setting 16954 cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 parent seq 0
2014-06-12 14:46:01.349376 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.349392 7f9d7d968700 20 set_header setting 2002 cid meta oid 1fe2ad5a/pglog_3.170/0//-1 parent seq 0
2014-06-12 14:46:01.349405 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:01.352381 7f9d7d968700 10 _do_op 0xb045860 seq 2734 r = 0, finisher 0x7ffad00 0x1e627640
2014-06-12 14:46:01.352402 7f9d7d968700 10 _finish_op 0xb045860 seq 2734 osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.352412 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cb680 (0 active)
2014-06-12 14:46:01.352421 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.352408 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] op_commit: 236176
2014-06-12 14:46:01.352446 7f9d7715b700  5 -- op tracker -- , seq: 747453, time: 2014-06-12 14:46:01.352434, event: op_commit, op: osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.352475 7f9d44101700  5 -- op tracker -- , seq: 747454, time: 2014-06-12 14:46:01.352427, event: header_read, op: osd_sub_op_reply(client.4296.0:143515 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.352505 7f9d44101700  5 -- op tracker -- , seq: 747454, time: 2014-06-12 14:46:01.352429, event: throttled, op: osd_sub_op_reply(client.4296.0:143515 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.352527 7f9d44101700  5 -- op tracker -- , seq: 747454, time: 2014-06-12 14:46:01.352446, event: all_read, op: osd_sub_op_reply(client.4296.0:143515 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.352522 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] op_applied: 236176
2014-06-12 14:46:01.352534 7f9d44101700  5 -- op tracker -- , seq: 747454, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143515 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.352540 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143515 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:01.352546 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.352544 7f9d7795c700  5 -- op tracker -- , seq: 747453, time: 2014-06-12 14:46:01.352537, event: op_applied, op: osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.352554 7f9d44101700 15 osd.2 173 enqueue_op 0x876e200 prio 196 cost 0 latency 0.000126 osd_sub_op_reply(client.4296.0:143515 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:01.352552 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] op_applied on primary on version 173'3043
2014-06-12 14:46:01.352597 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.352603 7f9d64bda700 10 osd.2 173 dequeue_op 0x876e200 prio 196 cost 0 latency 0.000176 osd_sub_op_reply(client.4296.0:143515 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 pg pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]
2014-06-12 14:46:01.352623 7f9d64bda700  5 -- op tracker -- , seq: 747454, time: 2014-06-12 14:46:01.352619, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143515 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.352631 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] handle_message: 0x876e200
2014-06-12 14:46:01.352645 7f9d64bda700  5 -- op tracker -- , seq: 747454, time: 2014-06-12 14:46:01.352642, event: started, op: osd_sub_op_reply(client.4296.0:143515 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.352650 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] sub_op_modify_reply: tid 236176 op  ack_type 4 from 21
2014-06-12 14:46:01.352669 7f9d64bda700  5 -- op tracker -- , seq: 747453, time: 2014-06-12 14:46:01.352662, event: sub_op_commit_rec, op: osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.352682 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] repop_all_applied: repop tid 236176 all applied 
2014-06-12 14:46:01.352693 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] eval_repop repgather(0x1696d5c0 173'3043 rep_tid=236176 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.352708 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]  sending ack on repgather(0x1696d5c0 173'3043 rep_tid=236176 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x1a669180
2014-06-12 14:46:01.352740 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] repop_all_committed: repop tid 236176 all committed 
2014-06-12 14:46:01.352750 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean] eval_repop repgather(0x1696d5c0 173'3043 rep_tid=236176 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.352765 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean] log_op_stats osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008044 lat 0.008070
2014-06-12 14:46:01.352782 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean] publish_stats_to_osd 173:3829
2014-06-12 14:46:01.352795 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean]  sending commit on repgather(0x1696d5c0 173'3043 rep_tid=236176 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x22b25400
2014-06-12 14:46:01.352823 7f9d64bda700  5 -- op tracker -- , seq: 747453, time: 2014-06-12 14:46:01.352818, event: commit_sent, op: osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.352831 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean]  requeue_ops 
2014-06-12 14:46:01.352841 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]  removing repgather(0x1696d5c0 173'3043 rep_tid=236176 committed?=1 applied?=1 op=osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.352864 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]    q front is repgather(0x1696d5c0 173'3043 rep_tid=236176 committed?=1 applied?=1 op=osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.352879 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] remove_repop repgather(0x1696d5c0 173'3043 rep_tid=236176 committed?=1 applied?=1 op=osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.352893 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]  requeue_ops 
2014-06-12 14:46:01.352917 7f9d64bda700  5 -- op tracker -- , seq: 747453, time: 2014-06-12 14:46:01.352912, event: done, op: osd_op(client.4296.0:143515 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 811008~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.352943 7f9d64bda700 10 osd.2 173 dequeue_op 0x876e200 finish
2014-06-12 14:46:01.352953 7f9d64bda700  5 -- op tracker -- , seq: 747454, time: 2014-06-12 14:46:01.352949, event: done, op: osd_sub_op_reply(client.4296.0:143515 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.352962 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.352965 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.353942 7f9d4a3c5700  5 -- op tracker -- , seq: 747455, time: 2014-06-12 14:46:01.353899, event: header_read, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.353954 7f9d4a3c5700  5 -- op tracker -- , seq: 747455, time: 2014-06-12 14:46:01.353901, event: throttled, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.353961 7f9d4a3c5700  5 -- op tracker -- , seq: 747455, time: 2014-06-12 14:46:01.353918, event: all_read, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.353967 7f9d4a3c5700  5 -- op tracker -- , seq: 747455, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.353971 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:46:01.353992 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x329ea700 prio 63 cost 4096 latency 0.000093 osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.354014 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.354033 7f9d683e1700 10 osd.2 173 dequeue_op 0x329ea700 prio 63 cost 4096 latency 0.000134 osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]
2014-06-12 14:46:01.354060 7f9d683e1700  5 -- op tracker -- , seq: 747455, time: 2014-06-12 14:46:01.354054, event: reached_pg, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.354069 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.354086 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] handle_message: 0x329ea700
2014-06-12 14:46:01.354097 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] do_op osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.354121 7f9d683e1700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_'
2014-06-12 14:46:01.354344 7f9d683e1700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_' = 0
2014-06-12 14:46:01.354363 7f9d683e1700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset'
2014-06-12 14:46:01.354465 7f9d683e1700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset' = 0
2014-06-12 14:46:01.354477 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] populate_obc_watchers edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.354490 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.354502 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] get_object_context: creating obc from disk: 0x66bc2c0
2014-06-12 14:46:01.354514 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] get_object_context: 0x66bc2c0 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 rwstate(none n=0 w=0) oi: edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3043 client.4296.0:143515 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3043) ssc: 0x16ee5960 snapset: 0=[]:[]+head
2014-06-12 14:46:01.354531 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] find_object_context edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 @head oi=edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3043 client.4296.0:143515 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3043)
2014-06-12 14:46:01.354560 7f9d683e1700  5 -- op tracker -- , seq: 747455, time: 2014-06-12 14:46:01.354553, event: started, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.354571 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] execute_ctx 0x1b264a00
2014-06-12 14:46:01.354590 7f9d683e1700  5 -- op tracker -- , seq: 747455, time: 2014-06-12 14:46:01.354585, event: started, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.354599 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] do_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] ov 173'3043 av 173'3044 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.354616 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] do_osd_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096]
2014-06-12 14:46:01.354629 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.354644 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] do_osd_op  write 1327104~4096
2014-06-12 14:46:01.354661 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:46:01.354674 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 snapset=0x16ee5998  snapc=0=[]
2014-06-12 14:46:01.354689 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.354702 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] finish_ctx edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 0x1b264a00 op modify  
2014-06-12 14:46:01.354721 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]  set mtime to 2014-06-12 08:45:03.735103
2014-06-12 14:46:01.354743 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]  final snapset 0=[]:[]+head in edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.354763 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.354775 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] calc_trim_to 79'42 -> 79'43
2014-06-12 14:46:01.354787 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] new_repop rep_tid 236177 on osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.354805 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] issue_repop rep_tid 236177 o edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.354831 7f9d683e1700  5 -- op tracker -- , seq: 747455, time: 2014-06-12 14:46:01.354826, event: waiting for subops from 21, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.354852 7f9d683e1700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:01.354876 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3043 (79'42,173'3043] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] append_log log((79'42,173'3043], crt=0'0) [173'3044 (173'3043) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143516 2014-06-12 08:45:03.735103]
2014-06-12 14:46:01.354901 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'42,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] add_log_entry 173'3044 (173'3043) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143516 2014-06-12 08:45:03.735103
2014-06-12 14:46:01.354923 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'42,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.354938 7f9d683e1700 10 trim log((79'42,173'3044], crt=0'0) to 79'43
2014-06-12 14:46:01.354964 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3044, trimmed: 79'43
2014-06-12 14:46:01.354990 7f9d683e1700  5 queue_transactions existing osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.354995 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.354997 7f9d683e1700  5 queue_op 0x7e9a2a0 seq 2735 osr(3.170 0x276ce20) 6424 bytes   (queue has 1 ops and 6424 bytes)
2014-06-12 14:46:01.355006 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cb680 (1 active)
2014-06-12 14:46:01.355013 7f9d8016d700  5 _do_op 0x7e9a2a0 seq 2735 osr(3.170 0x276ce20)/0x276ce20 start
2014-06-12 14:46:01.355017 7f9d8016d700 10 _do_transaction on 0x1771c780
2014-06-12 14:46:01.355003 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] eval_repop repgather(0x51ad080 173'3044 rep_tid=236177 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.355021 7f9d683e1700 10 osd.2 173 dequeue_op 0x329ea700 finish
2014-06-12 14:46:01.355023 7f9d8016d700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.355025 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.355028 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.355124 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.355150 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.355211 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.355298 7f9d8016d700 15 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.355310 7f9d8016d700 10 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1 r = 0
2014-06-12 14:46:01.355318 7f9d8016d700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.355337 7f9d8016d700 15 _write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 1327104~4096
2014-06-12 14:46:01.355481 7f9d8016d700 10 lookup_strip_header done  cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.355492 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:01.355598 7f9d8016d700 10 _generic_write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 1327104~4096 = 0
2014-06-12 14:46:01.355610 7f9d8016d700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.355623 7f9d8016d700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.355632 7f9d8016d700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.355641 7f9d8016d700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.355755 7f9d8016d700 20 set_header setting 16954 cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 parent seq 0
2014-06-12 14:46:01.355778 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.355794 7f9d8016d700 20 set_header setting 2002 cid meta oid 1fe2ad5a/pglog_3.170/0//-1 parent seq 0
2014-06-12 14:46:01.355806 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:01.355920 7f9d8016d700 10 _do_op 0x7e9a2a0 seq 2735 r = 0, finisher 0x98eeac0 0x1e627a40
2014-06-12 14:46:01.355936 7f9d8016d700 10 _finish_op 0x7e9a2a0 seq 2735 osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.355942 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cb680 (0 active)
2014-06-12 14:46:01.355945 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.355940 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] op_commit: 236177
2014-06-12 14:46:01.355963 7f9d7715b700  5 -- op tracker -- , seq: 747455, time: 2014-06-12 14:46:01.355956, event: op_commit, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.355975 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] op_applied: 236177
2014-06-12 14:46:01.355992 7f9d7795c700  5 -- op tracker -- , seq: 747455, time: 2014-06-12 14:46:01.355988, event: op_applied, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.355998 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] op_applied on primary on version 173'3044
2014-06-12 14:46:01.356260 7f9d44101700  5 -- op tracker -- , seq: 747456, time: 2014-06-12 14:46:01.356196, event: header_read, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.356285 7f9d44101700  5 -- op tracker -- , seq: 747456, time: 2014-06-12 14:46:01.356198, event: throttled, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.356303 7f9d44101700  5 -- op tracker -- , seq: 747456, time: 2014-06-12 14:46:01.356211, event: all_read, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.356322 7f9d44101700  5 -- op tracker -- , seq: 747456, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.356331 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:01.356341 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.356355 7f9d44101700 15 osd.2 173 enqueue_op 0x598f500 prio 196 cost 0 latency 0.000158 osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2
2014-06-12 14:46:01.356397 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.356407 7f9d65bdc700 10 osd.2 173 dequeue_op 0x598f500 prio 196 cost 0 latency 0.000210 osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2 pg pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]
2014-06-12 14:46:01.356434 7f9d65bdc700  5 -- op tracker -- , seq: 747456, time: 2014-06-12 14:46:01.356429, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.356445 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] handle_message: 0x598f500
2014-06-12 14:46:01.356467 7f9d44101700  5 -- op tracker -- , seq: 747457, time: 2014-06-12 14:46:01.356394, event: header_read, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.356507 7f9d44101700  5 -- op tracker -- , seq: 747457, time: 2014-06-12 14:46:01.356395, event: throttled, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.356530 7f9d44101700  5 -- op tracker -- , seq: 747457, time: 2014-06-12 14:46:01.356426, event: all_read, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.356553 7f9d44101700  5 -- op tracker -- , seq: 747457, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.356565 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:01.356572 7f9d65bdc700  5 -- op tracker -- , seq: 747456, time: 2014-06-12 14:46:01.356459, event: started, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.356577 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.356582 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] sub_op_modify_reply: tid 236177 op  ack_type 1 from 21
2014-06-12 14:46:01.356591 7f9d44101700 15 osd.2 173 enqueue_op 0x598f900 prio 196 cost 0 latency 0.000196 osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:01.356606 7f9d65bdc700  5 -- op tracker -- , seq: 747455, time: 2014-06-12 14:46:01.356598, event: sub_op_applied_rec, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.356625 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] repop_all_applied: repop tid 236177 all applied 
2014-06-12 14:46:01.356635 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:01.356639 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] eval_repop repgather(0x51ad080 173'3044 rep_tid=236177 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.356659 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]  sending ack on repgather(0x51ad080 173'3044 rep_tid=236177 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x33271b80
2014-06-12 14:46:01.356698 7f9d65bdc700 10 osd.2 173 dequeue_op 0x598f500 finish
2014-06-12 14:46:01.356710 7f9d65bdc700  5 -- op tracker -- , seq: 747456, time: 2014-06-12 14:46:01.356705, event: done, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.356727 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:01.356731 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.356716 7f9d6abe6700 10 osd.2 173 dequeue_op 0x598f900 prio 196 cost 0 latency 0.000322 osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 pg pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]
2014-06-12 14:46:01.356758 7f9d6abe6700  5 -- op tracker -- , seq: 747457, time: 2014-06-12 14:46:01.356752, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.356785 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] handle_message: 0x598f900
2014-06-12 14:46:01.356807 7f9d6abe6700  5 -- op tracker -- , seq: 747457, time: 2014-06-12 14:46:01.356804, event: started, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.356810 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] sub_op_modify_reply: tid 236177 op  ack_type 4 from 21
2014-06-12 14:46:01.356821 7f9d6abe6700  5 -- op tracker -- , seq: 747455, time: 2014-06-12 14:46:01.356817, event: sub_op_commit_rec, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.356824 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] repop_all_committed: repop tid 236177 all committed 
2014-06-12 14:46:01.356830 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3042 active+clean] eval_repop repgather(0x51ad080 173'3044 rep_tid=236177 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.356840 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3042 active+clean] log_op_stats osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 inb 4375 outb 0 rlat 0.002797 lat 0.002940
2014-06-12 14:46:01.356849 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3042 active+clean] publish_stats_to_osd 173:3830
2014-06-12 14:46:01.356857 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3042 active+clean]  sending commit on repgather(0x51ad080 173'3044 rep_tid=236177 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x151e6f00
2014-06-12 14:46:01.356881 7f9d6abe6700  5 -- op tracker -- , seq: 747455, time: 2014-06-12 14:46:01.356876, event: commit_sent, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.356890 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3042 active+clean]  requeue_ops 
2014-06-12 14:46:01.356915 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]  removing repgather(0x51ad080 173'3044 rep_tid=236177 committed?=1 applied?=1 op=osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.356924 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]    q front is repgather(0x51ad080 173'3044 rep_tid=236177 committed?=1 applied?=1 op=osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.356932 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] remove_repop repgather(0x51ad080 173'3044 rep_tid=236177 committed?=1 applied?=1 op=osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.356940 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]  requeue_ops 
2014-06-12 14:46:01.356953 7f9d6abe6700  5 -- op tracker -- , seq: 747455, time: 2014-06-12 14:46:01.356950, event: done, op: osd_op(client.4296.0:143516 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~4096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.356965 7f9d6abe6700 10 osd.2 173 dequeue_op 0x598f900 finish
2014-06-12 14:46:01.356970 7f9d6abe6700  5 -- op tracker -- , seq: 747457, time: 2014-06-12 14:46:01.356967, event: done, op: osd_sub_op_reply(client.4296.0:143516 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.356982 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.356984 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.360143 7f9d4a3c5700  5 -- op tracker -- , seq: 747458, time: 2014-06-12 14:46:01.359053, event: header_read, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.360155 7f9d4a3c5700  5 -- op tracker -- , seq: 747458, time: 2014-06-12 14:46:01.359055, event: throttled, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.360169 7f9d4a3c5700  5 -- op tracker -- , seq: 747458, time: 2014-06-12 14:46:01.360110, event: all_read, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.360175 7f9d4a3c5700  5 -- op tracker -- , seq: 747458, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.360180 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:46:01.360195 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x329ebe00 prio 63 cost 516096 latency 0.001142 osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.360218 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.360224 7f9d66bde700 10 osd.2 173 dequeue_op 0x329ebe00 prio 63 cost 516096 latency 0.001171 osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]
2014-06-12 14:46:01.360248 7f9d66bde700  5 -- op tracker -- , seq: 747458, time: 2014-06-12 14:46:01.360243, event: reached_pg, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.360256 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.360269 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] handle_message: 0x329ebe00
2014-06-12 14:46:01.360421 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] do_op osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.360444 7f9d66bde700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_'
2014-06-12 14:46:01.360651 7f9d66bde700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_' = 0
2014-06-12 14:46:01.360663 7f9d66bde700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset'
2014-06-12 14:46:01.360746 7f9d66bde700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset' = 0
2014-06-12 14:46:01.360752 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] populate_obc_watchers edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.360759 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.360765 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] get_object_context: creating obc from disk: 0x3ee38b00
2014-06-12 14:46:01.360770 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] get_object_context: 0x3ee38b00 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 rwstate(none n=0 w=0) oi: edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3044 client.4296.0:143516 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3044) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:46:01.360779 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] find_object_context edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 @head oi=edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3044 client.4296.0:143516 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3044)
2014-06-12 14:46:01.360794 7f9d66bde700  5 -- op tracker -- , seq: 747458, time: 2014-06-12 14:46:01.360790, event: started, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.360799 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] execute_ctx 0xa04f800
2014-06-12 14:46:01.360809 7f9d66bde700  5 -- op tracker -- , seq: 747458, time: 2014-06-12 14:46:01.360806, event: started, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.360813 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] do_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] ov 173'3044 av 173'3045 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.360821 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] do_osd_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096]
2014-06-12 14:46:01.360827 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.360839 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] do_osd_op  write 1331200~516096
2014-06-12 14:46:01.360846 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:46:01.360852 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:46:01.360858 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.360864 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] finish_ctx edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 0xa04f800 op modify  
2014-06-12 14:46:01.360870 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]  set mtime to 2014-06-12 08:45:03.739267
2014-06-12 14:46:01.360882 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]  final snapset 0=[]:[]+head in edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.360892 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.360898 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] calc_trim_to 79'43 -> 79'44
2014-06-12 14:46:01.360903 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] new_repop rep_tid 236178 on osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.360912 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] issue_repop rep_tid 236178 o edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.360924 7f9d66bde700  5 -- op tracker -- , seq: 747458, time: 2014-06-12 14:46:01.360921, event: waiting for subops from 21, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.360936 7f9d66bde700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:01.360950 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3044 (79'43,173'3044] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] append_log log((79'43,173'3044], crt=0'0) [173'3045 (173'3044) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143517 2014-06-12 08:45:03.739267]
2014-06-12 14:46:01.360964 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'43,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] add_log_entry 173'3045 (173'3044) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143517 2014-06-12 08:45:03.739267
2014-06-12 14:46:01.360976 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'43,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.360983 7f9d66bde700 10 trim log((79'43,173'3045], crt=0'0) to 79'44
2014-06-12 14:46:01.360999 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3045, trimmed: 79'44
2014-06-12 14:46:01.361012 7f9d66bde700  5 queue_transactions existing osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.361015 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.361015 7f9d66bde700  5 queue_op 0x9b6b6e0 seq 2736 osr(3.170 0x276ce20) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:01.361024 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cb680 (1 active)
2014-06-12 14:46:01.361021 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] eval_repop repgather(0x5214600 173'3045 rep_tid=236178 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.361033 7f9d66bde700 10 osd.2 173 dequeue_op 0x329ebe00 finish
2014-06-12 14:46:01.361032 7f9d8096e700  5 _do_op 0x9b6b6e0 seq 2736 osr(3.170 0x276ce20)/0x276ce20 start
2014-06-12 14:46:01.361035 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.361036 7f9d8096e700 10 _do_transaction on 0x5147080
2014-06-12 14:46:01.361037 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.361041 7f9d8096e700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.361216 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.361233 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.361311 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.361329 7f9d8096e700 15 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.361335 7f9d8096e700 10 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1 r = 0
2014-06-12 14:46:01.361353 7f9d8096e700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.361374 7f9d8096e700 15 _write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 1331200~516096
2014-06-12 14:46:01.361539 7f9d8096e700 10 lookup_strip_header done  cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.361870 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:01.363918 7f9d8096e700 10 _generic_write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 1331200~516096 = 0
2014-06-12 14:46:01.363974 7f9d8096e700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.363990 7f9d8096e700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.363998 7f9d8096e700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.364006 7f9d8096e700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.364226 7f9d8096e700 20 set_header setting 16954 cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 parent seq 0
2014-06-12 14:46:01.364307 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.364324 7f9d8096e700 20 set_header setting 2002 cid meta oid 1fe2ad5a/pglog_3.170/0//-1 parent seq 0
2014-06-12 14:46:01.364337 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:01.366996 7f9d8096e700 10 _do_op 0x9b6b6e0 seq 2736 r = 0, finisher 0x8a00a80 0x35f49f80
2014-06-12 14:46:01.367019 7f9d8096e700 10 _finish_op 0x9b6b6e0 seq 2736 osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.367037 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] op_applied: 236178
2014-06-12 14:46:01.367072 7f9d7795c700  5 -- op tracker -- , seq: 747458, time: 2014-06-12 14:46:01.367063, event: op_applied, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.367084 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] op_applied on primary on version 173'3045
2014-06-12 14:46:01.367109 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cb680 (0 active)
2014-06-12 14:46:01.367120 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.367298 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] op_commit: 236178
2014-06-12 14:46:01.367321 7f9d7715b700  5 -- op tracker -- , seq: 747458, time: 2014-06-12 14:46:01.367314, event: op_commit, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.372609 7f9d44101700  5 -- op tracker -- , seq: 747459, time: 2014-06-12 14:46:01.372569, event: header_read, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.372621 7f9d44101700  5 -- op tracker -- , seq: 747459, time: 2014-06-12 14:46:01.372571, event: throttled, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.372634 7f9d44101700  5 -- op tracker -- , seq: 747459, time: 2014-06-12 14:46:01.372579, event: all_read, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.372647 7f9d44101700  5 -- op tracker -- , seq: 747459, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.372653 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:01.372658 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.372667 7f9d44101700 15 osd.2 173 enqueue_op 0x59dca00 prio 196 cost 0 latency 0.000098 osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2
2014-06-12 14:46:01.372693 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.372701 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x59dca00 prio 196 cost 0 latency 0.000132 osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2 pg pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]
2014-06-12 14:46:01.372721 7f9d44101700  5 -- op tracker -- , seq: 747460, time: 2014-06-12 14:46:01.372687, event: header_read, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.372733 7f9d44101700  5 -- op tracker -- , seq: 747460, time: 2014-06-12 14:46:01.372688, event: throttled, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.372744 7f9d44101700  5 -- op tracker -- , seq: 747460, time: 2014-06-12 14:46:01.372698, event: all_read, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.372755 7f9d44101700  5 -- op tracker -- , seq: 747460, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.372760 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:01.372766 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.372766 7f9d6a3e5700  5 -- op tracker -- , seq: 747459, time: 2014-06-12 14:46:01.372719, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.372771 7f9d44101700 15 osd.2 173 enqueue_op 0x6d11b00 prio 196 cost 0 latency 0.000084 osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:01.372776 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] handle_message: 0x59dca00
2014-06-12 14:46:01.372793 7f9d6a3e5700  5 -- op tracker -- , seq: 747459, time: 2014-06-12 14:46:01.372790, event: started, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.372801 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:01.372800 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] sub_op_modify_reply: tid 236178 op  ack_type 1 from 21
2014-06-12 14:46:01.372820 7f9d6a3e5700  5 -- op tracker -- , seq: 747458, time: 2014-06-12 14:46:01.372813, event: sub_op_applied_rec, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.372831 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] repop_all_applied: repop tid 236178 all applied 
2014-06-12 14:46:01.372842 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] eval_repop repgather(0x5214600 173'3045 rep_tid=236178 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.372858 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]  sending ack on repgather(0x5214600 173'3045 rep_tid=236178 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x151e6280
2014-06-12 14:46:01.372908 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x59dca00 finish
2014-06-12 14:46:01.372918 7f9d6a3e5700  5 -- op tracker -- , seq: 747459, time: 2014-06-12 14:46:01.372914, event: done, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.372931 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:01.372934 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.372947 7f9d63bd8700 10 osd.2 173 dequeue_op 0x6d11b00 prio 196 cost 0 latency 0.000260 osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 pg pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]
2014-06-12 14:46:01.372971 7f9d63bd8700  5 -- op tracker -- , seq: 747460, time: 2014-06-12 14:46:01.372967, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.372980 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] handle_message: 0x6d11b00
2014-06-12 14:46:01.372996 7f9d63bd8700  5 -- op tracker -- , seq: 747460, time: 2014-06-12 14:46:01.372993, event: started, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.373002 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] sub_op_modify_reply: tid 236178 op  ack_type 4 from 21
2014-06-12 14:46:01.373024 7f9d63bd8700  5 -- op tracker -- , seq: 747458, time: 2014-06-12 14:46:01.373017, event: sub_op_commit_rec, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.373033 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] repop_all_committed: repop tid 236178 all committed 
2014-06-12 14:46:01.373046 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3043 active+clean] eval_repop repgather(0x5214600 173'3045 rep_tid=236178 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.373065 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3043 active+clean] log_op_stats osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.013845 lat 0.014011
2014-06-12 14:46:01.373085 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3043 active+clean] publish_stats_to_osd 173:3831
2014-06-12 14:46:01.373100 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3043 active+clean]  sending commit on repgather(0x5214600 173'3045 rep_tid=236178 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x27474a00
2014-06-12 14:46:01.373135 7f9d63bd8700  5 -- op tracker -- , seq: 747458, time: 2014-06-12 14:46:01.373129, event: commit_sent, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.373145 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3043 active+clean]  requeue_ops 
2014-06-12 14:46:01.373155 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]  removing repgather(0x5214600 173'3045 rep_tid=236178 committed?=1 applied?=1 op=osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.373168 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]    q front is repgather(0x5214600 173'3045 rep_tid=236178 committed?=1 applied?=1 op=osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.373180 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] remove_repop repgather(0x5214600 173'3045 rep_tid=236178 committed?=1 applied?=1 op=osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.373192 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]  requeue_ops 
2014-06-12 14:46:01.373214 7f9d63bd8700  5 -- op tracker -- , seq: 747458, time: 2014-06-12 14:46:01.373209, event: done, op: osd_op(client.4296.0:143517 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1331200~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.373226 7f9d63bd8700 10 osd.2 173 dequeue_op 0x6d11b00 finish
2014-06-12 14:46:01.373232 7f9d63bd8700  5 -- op tracker -- , seq: 747460, time: 2014-06-12 14:46:01.373229, event: done, op: osd_sub_op_reply(client.4296.0:143517 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.373240 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.373242 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.376477 7f9d4a3c5700  5 -- op tracker -- , seq: 747461, time: 2014-06-12 14:46:01.375389, event: header_read, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.376492 7f9d4a3c5700  5 -- op tracker -- , seq: 747461, time: 2014-06-12 14:46:01.375392, event: throttled, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.376502 7f9d4a3c5700  5 -- op tracker -- , seq: 747461, time: 2014-06-12 14:46:01.376438, event: all_read, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.376518 7f9d4a3c5700  5 -- op tracker -- , seq: 747461, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.376526 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:46:01.376554 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x20e86700 prio 63 cost 516096 latency 0.001165 osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.376587 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.376598 7f9d653db700 10 osd.2 173 dequeue_op 0x20e86700 prio 63 cost 516096 latency 0.001209 osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]
2014-06-12 14:46:01.376628 7f9d653db700  5 -- op tracker -- , seq: 747461, time: 2014-06-12 14:46:01.376623, event: reached_pg, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.376640 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.376658 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] handle_message: 0x20e86700
2014-06-12 14:46:01.376671 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] do_op osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.376698 7f9d653db700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_'
2014-06-12 14:46:01.376923 7f9d653db700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_' = 0
2014-06-12 14:46:01.376941 7f9d653db700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset'
2014-06-12 14:46:01.377042 7f9d653db700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset' = 0
2014-06-12 14:46:01.377054 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] populate_obc_watchers edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.377068 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.377079 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] get_object_context: creating obc from disk: 0x17da4000
2014-06-12 14:46:01.377088 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] get_object_context: 0x17da4000 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 rwstate(none n=0 w=0) oi: edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3045 client.4296.0:143517 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3045) ssc: 0x283d2b60 snapset: 0=[]:[]+head
2014-06-12 14:46:01.377107 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] find_object_context edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 @head oi=edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3045 client.4296.0:143517 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3045)
2014-06-12 14:46:01.377135 7f9d653db700  5 -- op tracker -- , seq: 747461, time: 2014-06-12 14:46:01.377129, event: started, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.377145 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] execute_ctx 0x1a919000
2014-06-12 14:46:01.377164 7f9d653db700  5 -- op tracker -- , seq: 747461, time: 2014-06-12 14:46:01.377158, event: started, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.377172 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] do_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] ov 173'3045 av 173'3046 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.377188 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] do_osd_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096]
2014-06-12 14:46:01.377201 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.377216 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] do_osd_op  write 1847296~516096
2014-06-12 14:46:01.377233 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:46:01.377247 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 snapset=0x283d2b98  snapc=0=[]
2014-06-12 14:46:01.377261 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.377273 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] finish_ctx edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 0x1a919000 op modify  
2014-06-12 14:46:01.377290 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]  set mtime to 2014-06-12 08:45:03.755663
2014-06-12 14:46:01.377314 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]  final snapset 0=[]:[]+head in edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.377334 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.377346 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] calc_trim_to 79'44 -> 79'45
2014-06-12 14:46:01.377358 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] new_repop rep_tid 236179 on osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.377376 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] issue_repop rep_tid 236179 o edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.377403 7f9d653db700  5 -- op tracker -- , seq: 747461, time: 2014-06-12 14:46:01.377398, event: waiting for subops from 21, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.377427 7f9d653db700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:01.377452 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3045 (79'44,173'3045] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] append_log log((79'44,173'3045], crt=0'0) [173'3046 (173'3045) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143518 2014-06-12 08:45:03.755663]
2014-06-12 14:46:01.377477 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'44,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] add_log_entry 173'3046 (173'3045) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143518 2014-06-12 08:45:03.755663
2014-06-12 14:46:01.377500 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'44,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.377516 7f9d653db700 10 trim log((79'44,173'3046], crt=0'0) to 79'45
2014-06-12 14:46:01.377541 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3046, trimmed: 79'45
2014-06-12 14:46:01.377567 7f9d653db700  5 queue_transactions existing osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.377573 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.377575 7f9d653db700  5 queue_op 0x1ebb1020 seq 2737 osr(3.170 0x276ce20) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:01.377587 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cb680 (1 active)
2014-06-12 14:46:01.377594 7f9d7915f700  5 _do_op 0x1ebb1020 seq 2737 osr(3.170 0x276ce20)/0x276ce20 start
2014-06-12 14:46:01.377598 7f9d7915f700 10 _do_transaction on 0x515f440
2014-06-12 14:46:01.377583 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] eval_repop repgather(0x32be3bc0 173'3046 rep_tid=236179 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.377602 7f9d653db700 10 osd.2 173 dequeue_op 0x20e86700 finish
2014-06-12 14:46:01.377605 7f9d7915f700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.377606 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.377609 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.377711 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.377732 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.377792 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.377816 7f9d7915f700 15 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.377827 7f9d7915f700 10 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1 r = 0
2014-06-12 14:46:01.377833 7f9d7915f700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.377847 7f9d7915f700 15 _write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 1847296~516096
2014-06-12 14:46:01.377987 7f9d7915f700 10 lookup_strip_header done  cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.378195 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:01.379944 7f9d7915f700 10 _generic_write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 1847296~516096 = 0
2014-06-12 14:46:01.379992 7f9d7915f700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.380006 7f9d7915f700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.380013 7f9d7915f700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.380020 7f9d7915f700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.380228 7f9d7915f700 20 set_header setting 16954 cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 parent seq 0
2014-06-12 14:46:01.380300 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.380315 7f9d7915f700 20 set_header setting 2002 cid meta oid 1fe2ad5a/pglog_3.170/0//-1 parent seq 0
2014-06-12 14:46:01.380328 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:01.383032 7f9d7915f700 10 _do_op 0x1ebb1020 seq 2737 r = 0, finisher 0xabf39a0 0x328d3c40
2014-06-12 14:46:01.383049 7f9d7915f700 10 _finish_op 0x1ebb1020 seq 2737 osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.383057 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cb680 (0 active)
2014-06-12 14:46:01.383063 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.383054 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] op_commit: 236179
2014-06-12 14:46:01.383083 7f9d7715b700  5 -- op tracker -- , seq: 747461, time: 2014-06-12 14:46:01.383075, event: op_commit, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.383097 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] op_applied: 236179
2014-06-12 14:46:01.383124 7f9d7795c700  5 -- op tracker -- , seq: 747461, time: 2014-06-12 14:46:01.383117, event: op_applied, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.383135 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] op_applied on primary on version 173'3046
2014-06-12 14:46:01.389420 7f9d44101700  5 -- op tracker -- , seq: 747462, time: 2014-06-12 14:46:01.389378, event: header_read, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.389432 7f9d44101700  5 -- op tracker -- , seq: 747462, time: 2014-06-12 14:46:01.389379, event: throttled, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.389439 7f9d44101700  5 -- op tracker -- , seq: 747462, time: 2014-06-12 14:46:01.389388, event: all_read, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.389447 7f9d44101700  5 -- op tracker -- , seq: 747462, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.389453 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:01.389459 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.389467 7f9d44101700 15 osd.2 173 enqueue_op 0x2a902800 prio 196 cost 0 latency 0.000089 osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2
2014-06-12 14:46:01.389508 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.389515 7f9d69be4700 10 osd.2 173 dequeue_op 0x2a902800 prio 196 cost 0 latency 0.000137 osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2 pg pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]
2014-06-12 14:46:01.389538 7f9d69be4700  5 -- op tracker -- , seq: 747462, time: 2014-06-12 14:46:01.389535, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.389546 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] handle_message: 0x2a902800
2014-06-12 14:46:01.389565 7f9d69be4700  5 -- op tracker -- , seq: 747462, time: 2014-06-12 14:46:01.389561, event: started, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.389572 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] sub_op_modify_reply: tid 236179 op  ack_type 1 from 21
2014-06-12 14:46:01.389596 7f9d69be4700  5 -- op tracker -- , seq: 747461, time: 2014-06-12 14:46:01.389587, event: sub_op_applied_rec, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.389606 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] repop_all_applied: repop tid 236179 all applied 
2014-06-12 14:46:01.389623 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] eval_repop repgather(0x32be3bc0 173'3046 rep_tid=236179 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.389640 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]  sending ack on repgather(0x32be3bc0 173'3046 rep_tid=236179 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x959bb80
2014-06-12 14:46:01.389672 7f9d69be4700 10 osd.2 173 dequeue_op 0x2a902800 finish
2014-06-12 14:46:01.389681 7f9d69be4700  5 -- op tracker -- , seq: 747462, time: 2014-06-12 14:46:01.389677, event: done, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.389696 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.389701 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.390179 7f9d44101700  5 -- op tracker -- , seq: 747463, time: 2014-06-12 14:46:01.390150, event: header_read, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.390186 7f9d44101700  5 -- op tracker -- , seq: 747463, time: 2014-06-12 14:46:01.390151, event: throttled, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.390191 7f9d44101700  5 -- op tracker -- , seq: 747463, time: 2014-06-12 14:46:01.390159, event: all_read, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.390199 7f9d44101700  5 -- op tracker -- , seq: 747463, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.390201 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:01.390204 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.390207 7f9d44101700 15 osd.2 173 enqueue_op 0x20e89f00 prio 196 cost 0 latency 0.000057 osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:01.390225 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.390231 7f9d633d7700 10 osd.2 173 dequeue_op 0x20e89f00 prio 196 cost 0 latency 0.000081 osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 pg pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]
2014-06-12 14:46:01.390245 7f9d633d7700  5 -- op tracker -- , seq: 747463, time: 2014-06-12 14:46:01.390242, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.390251 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] handle_message: 0x20e89f00
2014-06-12 14:46:01.390262 7f9d633d7700  5 -- op tracker -- , seq: 747463, time: 2014-06-12 14:46:01.390260, event: started, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.390265 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] sub_op_modify_reply: tid 236179 op  ack_type 4 from 21
2014-06-12 14:46:01.390276 7f9d633d7700  5 -- op tracker -- , seq: 747461, time: 2014-06-12 14:46:01.390272, event: sub_op_commit_rec, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.390280 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] repop_all_committed: repop tid 236179 all committed 
2014-06-12 14:46:01.390286 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3044 active+clean] eval_repop repgather(0x32be3bc0 173'3046 rep_tid=236179 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.390298 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3044 active+clean] log_op_stats osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.014281 lat 0.014908
2014-06-12 14:46:01.390310 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3044 active+clean] publish_stats_to_osd 173:3832
2014-06-12 14:46:01.390319 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3044 active+clean]  sending commit on repgather(0x32be3bc0 173'3046 rep_tid=236179 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x29de7400
2014-06-12 14:46:01.390344 7f9d633d7700  5 -- op tracker -- , seq: 747461, time: 2014-06-12 14:46:01.390338, event: commit_sent, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.390353 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3044 active+clean]  requeue_ops 
2014-06-12 14:46:01.390367 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]  removing repgather(0x32be3bc0 173'3046 rep_tid=236179 committed?=1 applied?=1 op=osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.390378 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]    q front is repgather(0x32be3bc0 173'3046 rep_tid=236179 committed?=1 applied?=1 op=osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.390387 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] remove_repop repgather(0x32be3bc0 173'3046 rep_tid=236179 committed?=1 applied?=1 op=osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.390396 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]  requeue_ops 
2014-06-12 14:46:01.390414 7f9d633d7700  5 -- op tracker -- , seq: 747461, time: 2014-06-12 14:46:01.390411, event: done, op: osd_op(client.4296.0:143518 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1847296~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.390428 7f9d633d7700 10 osd.2 173 dequeue_op 0x20e89f00 finish
2014-06-12 14:46:01.390434 7f9d633d7700  5 -- op tracker -- , seq: 747463, time: 2014-06-12 14:46:01.390431, event: done, op: osd_sub_op_reply(client.4296.0:143518 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.390439 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.390441 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.393711 7f9d4a3c5700  5 -- op tracker -- , seq: 747464, time: 2014-06-12 14:46:01.392724, event: header_read, op: osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.393723 7f9d4a3c5700  5 -- op tracker -- , seq: 747464, time: 2014-06-12 14:46:01.392726, event: throttled, op: osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.393738 7f9d4a3c5700  5 -- op tracker -- , seq: 747464, time: 2014-06-12 14:46:01.393668, event: all_read, op: osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.393744 7f9d4a3c5700  5 -- op tracker -- , seq: 747464, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.393750 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:46:01.393773 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x20e86c00 prio 63 cost 516096 latency 0.001049 osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.393798 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.393806 7f9d643d9700 10 osd.2 173 dequeue_op 0x20e86c00 prio 63 cost 516096 latency 0.001082 osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]
2014-06-12 14:46:01.393825 7f9d643d9700  5 -- op tracker -- , seq: 747464, time: 2014-06-12 14:46:01.393822, event: reached_pg, op: osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.393832 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.393842 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] handle_message: 0x20e86c00
2014-06-12 14:46:01.393848 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] do_op osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.393865 7f9d643d9700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_'
2014-06-12 14:46:01.394049 7f9d643d9700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_' = 0
2014-06-12 14:46:01.394067 7f9d643d9700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset'
2014-06-12 14:46:01.394135 7f9d643d9700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset' = 0
2014-06-12 14:46:01.394144 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] populate_obc_watchers edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.394152 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.394157 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] get_object_context: creating obc from disk: 0x22032000
2014-06-12 14:46:01.394163 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] get_object_context: 0x22032000 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 rwstate(none n=0 w=0) oi: edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3046 client.4296.0:143518 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3046) ssc: 0x679b420 snapset: 0=[]:[]+head
2014-06-12 14:46:01.394173 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] find_object_context edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 @head oi=edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3046 client.4296.0:143518 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3046)
2014-06-12 14:46:01.394190 7f9d643d9700  5 -- op tracker -- , seq: 747464, time: 2014-06-12 14:46:01.394186, event: started, op: osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.394197 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] execute_ctx 0x55a4400
2014-06-12 14:46:01.394207 7f9d643d9700  5 -- op tracker -- , seq: 747464, time: 2014-06-12 14:46:01.394204, event: started, op: osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.394211 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] do_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] ov 173'3046 av 173'3047 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.394220 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] do_osd_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096]
2014-06-12 14:46:01.394227 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.394235 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] do_osd_op  write 2363392~516096
2014-06-12 14:46:01.394248 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:46:01.394256 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 snapset=0x679b458  snapc=0=[]
2014-06-12 14:46:01.394263 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.394269 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] finish_ctx edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 0x55a4400 op modify  
2014-06-12 14:46:01.394277 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]  set mtime to 2014-06-12 08:45:03.772936
2014-06-12 14:46:01.394289 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]  final snapset 0=[]:[]+head in edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.394303 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.394309 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] calc_trim_to 79'45 -> 79'46
2014-06-12 14:46:01.394314 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] new_repop rep_tid 236180 on osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.394324 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] issue_repop rep_tid 236180 o edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.394340 7f9d643d9700  5 -- op tracker -- , seq: 747464, time: 2014-06-12 14:46:01.394337, event: waiting for subops from 21, op: osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.394358 7f9d643d9700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:01.394375 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3046 (79'45,173'3046] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] append_log log((79'45,173'3046], crt=0'0) [173'3047 (173'3046) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143519 2014-06-12 08:45:03.772936]
2014-06-12 14:46:01.394399 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'45,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] add_log_entry 173'3047 (173'3046) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143519 2014-06-12 08:45:03.772936
2014-06-12 14:46:01.394419 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'45,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.394433 7f9d643d9700 10 trim log((79'45,173'3047], crt=0'0) to 79'46
2014-06-12 14:46:01.394457 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3047, trimmed: 79'46
2014-06-12 14:46:01.394480 7f9d643d9700  5 queue_transactions existing osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.394485 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.394486 7f9d643d9700  5 queue_op 0x90ad920 seq 2738 osr(3.170 0x276ce20) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:01.394493 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] eval_repop repgather(0x34d8e600 173'3047 rep_tid=236180 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.394509 7f9d643d9700 10 osd.2 173 dequeue_op 0x20e86c00 finish
2014-06-12 14:46:01.394512 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.394515 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.394535 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cb680 (1 active)
2014-06-12 14:46:01.394540 7f9d7c165700  5 _do_op 0x90ad920 seq 2738 osr(3.170 0x276ce20)/0x276ce20 start
2014-06-12 14:46:01.394543 7f9d7c165700 10 _do_transaction on 0x31ee6240
2014-06-12 14:46:01.394548 7f9d7c165700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.394642 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.394663 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.394725 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.394818 7f9d7c165700 15 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.394828 7f9d7c165700 10 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1 r = 0
2014-06-12 14:46:01.394832 7f9d7c165700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.394847 7f9d7c165700 15 _write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 2363392~516096
2014-06-12 14:46:01.394954 7f9d7c165700 10 lookup_strip_header done  cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.395095 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:01.396417 7f9d7c165700 10 _generic_write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 2363392~516096 = 0
2014-06-12 14:46:01.396456 7f9d7c165700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.396468 7f9d7c165700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.396473 7f9d7c165700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.396478 7f9d7c165700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.396616 7f9d7c165700 20 set_header setting 16954 cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 parent seq 0
2014-06-12 14:46:01.396676 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.396687 7f9d7c165700 20 set_header setting 2002 cid meta oid 1fe2ad5a/pglog_3.170/0//-1 parent seq 0
2014-06-12 14:46:01.396695 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:01.398876 7f9d7c165700 10 _do_op 0x90ad920 seq 2738 r = 0, finisher 0x88cc360 0x3984d600
2014-06-12 14:46:01.398893 7f9d7c165700 10 _finish_op 0x90ad920 seq 2738 osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.398903 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cb680 (0 active)
2014-06-12 14:46:01.398913 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.398900 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] op_commit: 236180
2014-06-12 14:46:01.398927 7f9d7715b700  5 -- op tracker -- , seq: 747464, time: 2014-06-12 14:46:01.398919, event: op_commit, op: osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.398941 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] op_applied: 236180
2014-06-12 14:46:01.398968 7f9d7795c700  5 -- op tracker -- , seq: 747464, time: 2014-06-12 14:46:01.398960, event: op_applied, op: osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.398978 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] op_applied on primary on version 173'3047
2014-06-12 14:46:01.401403 7f9d44101700  5 -- op tracker -- , seq: 747465, time: 2014-06-12 14:46:01.401362, event: header_read, op: osd_sub_op_reply(client.4296.0:143519 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.401415 7f9d44101700  5 -- op tracker -- , seq: 747465, time: 2014-06-12 14:46:01.401364, event: throttled, op: osd_sub_op_reply(client.4296.0:143519 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.401452 7f9d44101700  5 -- op tracker -- , seq: 747465, time: 2014-06-12 14:46:01.401374, event: all_read, op: osd_sub_op_reply(client.4296.0:143519 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.401462 7f9d44101700  5 -- op tracker -- , seq: 747465, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143519 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.401467 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143519 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:01.401472 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.401480 7f9d44101700 15 osd.2 173 enqueue_op 0x2d17b600 prio 196 cost 0 latency 0.000118 osd_sub_op_reply(client.4296.0:143519 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:01.401507 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.401516 7f9d623d5700 10 osd.2 173 dequeue_op 0x2d17b600 prio 196 cost 0 latency 0.000153 osd_sub_op_reply(client.4296.0:143519 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 pg pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]
2014-06-12 14:46:01.401539 7f9d623d5700  5 -- op tracker -- , seq: 747465, time: 2014-06-12 14:46:01.401535, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143519 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.401547 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] handle_message: 0x2d17b600
2014-06-12 14:46:01.401565 7f9d623d5700  5 -- op tracker -- , seq: 747465, time: 2014-06-12 14:46:01.401561, event: started, op: osd_sub_op_reply(client.4296.0:143519 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.401571 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] sub_op_modify_reply: tid 236180 op  ack_type 4 from 21
2014-06-12 14:46:01.401595 7f9d623d5700  5 -- op tracker -- , seq: 747464, time: 2014-06-12 14:46:01.401586, event: sub_op_commit_rec, op: osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.401604 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] repop_all_applied: repop tid 236180 all applied 
2014-06-12 14:46:01.401632 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] eval_repop repgather(0x34d8e600 173'3047 rep_tid=236180 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.401651 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]  sending ack on repgather(0x34d8e600 173'3047 rep_tid=236180 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x258d1400
2014-06-12 14:46:01.401686 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] repop_all_committed: repop tid 236180 all committed 
2014-06-12 14:46:01.401694 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3045 active+clean] eval_repop repgather(0x34d8e600 173'3047 rep_tid=236180 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.401713 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3045 active+clean] log_op_stats osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008960 lat 0.008988
2014-06-12 14:46:01.401734 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3045 active+clean] publish_stats_to_osd 173:3833
2014-06-12 14:46:01.401750 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3045 active+clean]  sending commit on repgather(0x34d8e600 173'3047 rep_tid=236180 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x2844b180
2014-06-12 14:46:01.401785 7f9d623d5700  5 -- op tracker -- , seq: 747464, time: 2014-06-12 14:46:01.401779, event: commit_sent, op: osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.401793 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3045 active+clean]  requeue_ops 
2014-06-12 14:46:01.401804 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean]  removing repgather(0x34d8e600 173'3047 rep_tid=236180 committed?=1 applied?=1 op=osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.401818 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean]    q front is repgather(0x34d8e600 173'3047 rep_tid=236180 committed?=1 applied?=1 op=osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.401834 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] remove_repop repgather(0x34d8e600 173'3047 rep_tid=236180 committed?=1 applied?=1 op=osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.401849 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean]  requeue_ops 
2014-06-12 14:46:01.401875 7f9d623d5700  5 -- op tracker -- , seq: 747464, time: 2014-06-12 14:46:01.401869, event: done, op: osd_op(client.4296.0:143519 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2363392~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.401896 7f9d623d5700 10 osd.2 173 dequeue_op 0x2d17b600 finish
2014-06-12 14:46:01.401906 7f9d623d5700  5 -- op tracker -- , seq: 747465, time: 2014-06-12 14:46:01.401902, event: done, op: osd_sub_op_reply(client.4296.0:143519 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.401916 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.401920 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.404872 7f9d4a3c5700  5 -- op tracker -- , seq: 747466, time: 2014-06-12 14:46:01.403863, event: header_read, op: osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.404882 7f9d4a3c5700  5 -- op tracker -- , seq: 747466, time: 2014-06-12 14:46:01.403865, event: throttled, op: osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.404888 7f9d4a3c5700  5 -- op tracker -- , seq: 747466, time: 2014-06-12 14:46:01.404846, event: all_read, op: osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.404894 7f9d4a3c5700  5 -- op tracker -- , seq: 747466, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.404899 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:46:01.404920 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x2fa9d700 prio 63 cost 516096 latency 0.001057 osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.404943 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.404951 7f9d68be2700 10 osd.2 173 dequeue_op 0x2fa9d700 prio 63 cost 516096 latency 0.001088 osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean]
2014-06-12 14:46:01.404979 7f9d68be2700  5 -- op tracker -- , seq: 747466, time: 2014-06-12 14:46:01.404974, event: reached_pg, op: osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.404988 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.405002 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] handle_message: 0x2fa9d700
2014-06-12 14:46:01.405012 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] do_op osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.405038 7f9d68be2700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_'
2014-06-12 14:46:01.405206 7f9d68be2700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_' = 0
2014-06-12 14:46:01.405222 7f9d68be2700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset'
2014-06-12 14:46:01.405294 7f9d68be2700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset' = 0
2014-06-12 14:46:01.405305 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] populate_obc_watchers edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.405319 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.405330 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] get_object_context: creating obc from disk: 0x20106840
2014-06-12 14:46:01.405340 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] get_object_context: 0x20106840 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 rwstate(none n=0 w=0) oi: edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3047 client.4296.0:143519 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3047) ssc: 0x1550efc0 snapset: 0=[]:[]+head
2014-06-12 14:46:01.405357 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] find_object_context edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 @head oi=edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3047 client.4296.0:143519 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3047)
2014-06-12 14:46:01.405386 7f9d68be2700  5 -- op tracker -- , seq: 747466, time: 2014-06-12 14:46:01.405380, event: started, op: osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.405394 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] execute_ctx 0x26f9f000
2014-06-12 14:46:01.405412 7f9d68be2700  5 -- op tracker -- , seq: 747466, time: 2014-06-12 14:46:01.405407, event: started, op: osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.405419 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] do_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] ov 173'3047 av 173'3048 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.405436 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] do_osd_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096]
2014-06-12 14:46:01.405448 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.405463 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] do_osd_op  write 2879488~516096
2014-06-12 14:46:01.405481 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:46:01.405494 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 snapset=0x1550eff8  snapc=0=[]
2014-06-12 14:46:01.405509 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.405522 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] finish_ctx edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 0x26f9f000 op modify  
2014-06-12 14:46:01.405535 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean]  set mtime to 2014-06-12 08:45:03.784069
2014-06-12 14:46:01.405557 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean]  final snapset 0=[]:[]+head in edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.405577 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.405589 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] calc_trim_to 79'46 -> 79'47
2014-06-12 14:46:01.405599 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] new_repop rep_tid 236181 on osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.405634 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] issue_repop rep_tid 236181 o edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.405664 7f9d68be2700  5 -- op tracker -- , seq: 747466, time: 2014-06-12 14:46:01.405658, event: waiting for subops from 21, op: osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.405686 7f9d68be2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:01.405708 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3047 (79'46,173'3047] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] append_log log((79'46,173'3047], crt=0'0) [173'3048 (173'3047) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143520 2014-06-12 08:45:03.784069]
2014-06-12 14:46:01.405733 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'46,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3047 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] add_log_entry 173'3048 (173'3047) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143520 2014-06-12 08:45:03.784069
2014-06-12 14:46:01.405753 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'46,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3047 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.405766 7f9d68be2700 10 trim log((79'46,173'3048], crt=0'0) to 79'47
2014-06-12 14:46:01.405790 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3048, trimmed: 79'47
2014-06-12 14:46:01.405816 7f9d68be2700  5 queue_transactions existing osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.405821 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.405822 7f9d68be2700  5 queue_op 0x8376240 seq 2739 osr(3.170 0x276ce20) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:01.405832 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cb680 (1 active)
2014-06-12 14:46:01.405840 7f9d83173700  5 _do_op 0x8376240 seq 2739 osr(3.170 0x276ce20)/0x276ce20 start
2014-06-12 14:46:01.405828 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3047 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] eval_repop repgather(0x28fc1980 173'3048 rep_tid=236181 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.405844 7f9d83173700 10 _do_transaction on 0x5352480
2014-06-12 14:46:01.405845 7f9d68be2700 10 osd.2 173 dequeue_op 0x2fa9d700 finish
2014-06-12 14:46:01.405848 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.405850 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.405851 7f9d83173700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.405949 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.406042 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.406102 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.406127 7f9d83173700 15 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.406136 7f9d83173700 10 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1 r = 0
2014-06-12 14:46:01.406143 7f9d83173700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.406161 7f9d83173700 15 _write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 2879488~516096
2014-06-12 14:46:01.406265 7f9d83173700 10 lookup_strip_header done  cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.406454 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:01.407624 7f9d83173700 10 _generic_write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 2879488~516096 = 0
2014-06-12 14:46:01.407664 7f9d83173700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.407679 7f9d83173700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.407685 7f9d83173700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.407694 7f9d83173700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.407833 7f9d83173700 20 set_header setting 16954 cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 parent seq 0
2014-06-12 14:46:01.407890 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.407905 7f9d83173700 20 set_header setting 2002 cid meta oid 1fe2ad5a/pglog_3.170/0//-1 parent seq 0
2014-06-12 14:46:01.407915 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:01.409859 7f9d83173700 10 _do_op 0x8376240 seq 2739 r = 0, finisher 0x7eb08a0 0x38dd28c0
2014-06-12 14:46:01.409876 7f9d83173700 10 _finish_op 0x8376240 seq 2739 osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.409884 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cb680 (0 active)
2014-06-12 14:46:01.409893 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.409880 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3047 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] op_commit: 236181
2014-06-12 14:46:01.409907 7f9d7715b700  5 -- op tracker -- , seq: 747466, time: 2014-06-12 14:46:01.409898, event: op_commit, op: osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.409935 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3047 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] op_applied: 236181
2014-06-12 14:46:01.409960 7f9d7795c700  5 -- op tracker -- , seq: 747466, time: 2014-06-12 14:46:01.409953, event: op_applied, op: osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.409973 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3047 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] op_applied on primary on version 173'3048
2014-06-12 14:46:01.416794 7f9d44101700  5 -- op tracker -- , seq: 747467, time: 2014-06-12 14:46:01.416761, event: header_read, op: osd_sub_op_reply(client.4296.0:143520 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.416803 7f9d44101700  5 -- op tracker -- , seq: 747467, time: 2014-06-12 14:46:01.416762, event: throttled, op: osd_sub_op_reply(client.4296.0:143520 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.416816 7f9d44101700  5 -- op tracker -- , seq: 747467, time: 2014-06-12 14:46:01.416769, event: all_read, op: osd_sub_op_reply(client.4296.0:143520 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.416821 7f9d44101700  5 -- op tracker -- , seq: 747467, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143520 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.416823 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143520 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:01.416827 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.416832 7f9d44101700 15 osd.2 173 enqueue_op 0x2d17b200 prio 196 cost 0 latency 0.000071 osd_sub_op_reply(client.4296.0:143520 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:01.416852 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.416861 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2d17b200 prio 196 cost 0 latency 0.000100 osd_sub_op_reply(client.4296.0:143520 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 pg pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3047 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean]
2014-06-12 14:46:01.416879 7f9d61bd4700  5 -- op tracker -- , seq: 747467, time: 2014-06-12 14:46:01.416877, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143520 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.416885 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3047 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] handle_message: 0x2d17b200
2014-06-12 14:46:01.416896 7f9d61bd4700  5 -- op tracker -- , seq: 747467, time: 2014-06-12 14:46:01.416893, event: started, op: osd_sub_op_reply(client.4296.0:143520 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.416899 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3047 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] sub_op_modify_reply: tid 236181 op  ack_type 4 from 21
2014-06-12 14:46:01.416911 7f9d61bd4700  5 -- op tracker -- , seq: 747466, time: 2014-06-12 14:46:01.416907, event: sub_op_commit_rec, op: osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.416917 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3047 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] repop_all_applied: repop tid 236181 all applied 
2014-06-12 14:46:01.416924 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3047 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] eval_repop repgather(0x28fc1980 173'3048 rep_tid=236181 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.416935 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3047 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean]  sending ack on repgather(0x28fc1980 173'3048 rep_tid=236181 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x282ccf00
2014-06-12 14:46:01.416954 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3047 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] repop_all_committed: repop tid 236181 all committed 
2014-06-12 14:46:01.416965 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3046 active+clean] eval_repop repgather(0x28fc1980 173'3048 rep_tid=236181 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.416983 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3046 active+clean] log_op_stats osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.013090 lat 0.013119
2014-06-12 14:46:01.416997 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3046 active+clean] publish_stats_to_osd 173:3834
2014-06-12 14:46:01.417005 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3046 active+clean]  sending commit on repgather(0x28fc1980 173'3048 rep_tid=236181 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x17f01680
2014-06-12 14:46:01.417033 7f9d61bd4700  5 -- op tracker -- , seq: 747466, time: 2014-06-12 14:46:01.417028, event: commit_sent, op: osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.417043 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3046 active+clean]  requeue_ops 
2014-06-12 14:46:01.417054 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean]  removing repgather(0x28fc1980 173'3048 rep_tid=236181 committed?=1 applied?=1 op=osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.417065 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean]    q front is repgather(0x28fc1980 173'3048 rep_tid=236181 committed?=1 applied?=1 op=osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.417074 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] remove_repop repgather(0x28fc1980 173'3048 rep_tid=236181 committed?=1 applied?=1 op=osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.417082 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean]  requeue_ops 
2014-06-12 14:46:01.417098 7f9d61bd4700  5 -- op tracker -- , seq: 747466, time: 2014-06-12 14:46:01.417095, event: done, op: osd_op(client.4296.0:143520 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2879488~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.417112 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2d17b200 finish
2014-06-12 14:46:01.417117 7f9d61bd4700  5 -- op tracker -- , seq: 747467, time: 2014-06-12 14:46:01.417115, event: done, op: osd_sub_op_reply(client.4296.0:143520 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.417123 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.417125 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.419864 7f9d4a3c5700  5 -- op tracker -- , seq: 747468, time: 2014-06-12 14:46:01.418906, event: header_read, op: osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.419878 7f9d4a3c5700  5 -- op tracker -- , seq: 747468, time: 2014-06-12 14:46:01.418908, event: throttled, op: osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.419888 7f9d4a3c5700  5 -- op tracker -- , seq: 747468, time: 2014-06-12 14:46:01.419826, event: all_read, op: osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.419898 7f9d4a3c5700  5 -- op tracker -- , seq: 747468, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.419907 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:46:01.419936 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x2fa9c300 prio 63 cost 516096 latency 0.001029 osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.420238 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.420250 7f9d673df700 10 osd.2 173 dequeue_op 0x2fa9c300 prio 63 cost 516096 latency 0.001343 osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean]
2014-06-12 14:46:01.420279 7f9d673df700  5 -- op tracker -- , seq: 747468, time: 2014-06-12 14:46:01.420273, event: reached_pg, op: osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.420289 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.420306 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] handle_message: 0x2fa9c300
2014-06-12 14:46:01.420319 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] do_op osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.420347 7f9d673df700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_'
2014-06-12 14:46:01.420587 7f9d673df700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_' = 0
2014-06-12 14:46:01.420605 7f9d673df700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset'
2014-06-12 14:46:01.420720 7f9d673df700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset' = 0
2014-06-12 14:46:01.420735 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] populate_obc_watchers edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.420755 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.420764 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] get_object_context: creating obc from disk: 0x2badc2c0
2014-06-12 14:46:01.420774 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] get_object_context: 0x2badc2c0 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 rwstate(none n=0 w=0) oi: edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3048 client.4296.0:143520 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3048) ssc: 0x9345880 snapset: 0=[]:[]+head
2014-06-12 14:46:01.420789 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] find_object_context edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 @head oi=edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3048 client.4296.0:143520 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3048)
2014-06-12 14:46:01.420814 7f9d673df700  5 -- op tracker -- , seq: 747468, time: 2014-06-12 14:46:01.420808, event: started, op: osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.420825 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] execute_ctx 0x5ca0600
2014-06-12 14:46:01.420857 7f9d673df700  5 -- op tracker -- , seq: 747468, time: 2014-06-12 14:46:01.420852, event: started, op: osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.420878 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] do_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] ov 173'3048 av 173'3049 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.420893 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] do_osd_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096]
2014-06-12 14:46:01.420904 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.420917 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] do_osd_op  write 3395584~516096
2014-06-12 14:46:01.420932 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:46:01.420945 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 snapset=0x93458b8  snapc=0=[]
2014-06-12 14:46:01.420957 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.420970 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] finish_ctx edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 0x5ca0600 op modify  
2014-06-12 14:46:01.420997 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean]  set mtime to 2014-06-12 08:45:03.799251
2014-06-12 14:46:01.421029 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean]  final snapset 0=[]:[]+head in edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.421048 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.421067 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] calc_trim_to 79'47 -> 79'48
2014-06-12 14:46:01.421085 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] new_repop rep_tid 236182 on osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.421100 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] issue_repop rep_tid 236182 o edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.421123 7f9d673df700  5 -- op tracker -- , seq: 747468, time: 2014-06-12 14:46:01.421118, event: waiting for subops from 21, op: osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.421150 7f9d673df700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:01.421189 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3048 (79'47,173'3048] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] append_log log((79'47,173'3048], crt=0'0) [173'3049 (173'3048) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143521 2014-06-12 08:45:03.799251]
2014-06-12 14:46:01.421222 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'47,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3048 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] add_log_entry 173'3049 (173'3048) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143521 2014-06-12 08:45:03.799251
2014-06-12 14:46:01.421250 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'47,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3048 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.421272 7f9d673df700 10 trim log((79'47,173'3049], crt=0'0) to 79'48
2014-06-12 14:46:01.421306 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3049, trimmed: 79'48
2014-06-12 14:46:01.421348 7f9d673df700  5 queue_transactions existing osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.421352 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.421354 7f9d673df700  5 queue_op 0x73a3da0 seq 2740 osr(3.170 0x276ce20) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:01.421366 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cb680 (1 active)
2014-06-12 14:46:01.421361 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3048 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] eval_repop repgather(0x10450540 173'3049 rep_tid=236182 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.421376 7f9d82972700  5 _do_op 0x73a3da0 seq 2740 osr(3.170 0x276ce20)/0x276ce20 start
2014-06-12 14:46:01.421379 7f9d673df700 10 osd.2 173 dequeue_op 0x2fa9c300 finish
2014-06-12 14:46:01.421381 7f9d82972700 10 _do_transaction on 0x2f89f5c0
2014-06-12 14:46:01.421383 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.421386 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.421389 7f9d82972700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.421495 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.421521 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.421575 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.421600 7f9d82972700 15 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.421609 7f9d82972700 10 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1 r = 0
2014-06-12 14:46:01.421632 7f9d82972700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.421763 7f9d82972700 15 _write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 3395584~516096
2014-06-12 14:46:01.421868 7f9d82972700 10 lookup_strip_header done  cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.421974 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:01.423189 7f9d82972700 10 _generic_write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 3395584~516096 = 0
2014-06-12 14:46:01.423231 7f9d82972700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.423245 7f9d82972700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.423252 7f9d82972700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.423260 7f9d82972700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.423405 7f9d82972700 20 set_header setting 16954 cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 parent seq 0
2014-06-12 14:46:01.423465 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.423481 7f9d82972700 20 set_header setting 2002 cid meta oid 1fe2ad5a/pglog_3.170/0//-1 parent seq 0
2014-06-12 14:46:01.423491 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:01.425759 7f9d82972700 10 _do_op 0x73a3da0 seq 2740 r = 0, finisher 0x1ecc6240 0x323f1c0
2014-06-12 14:46:01.425784 7f9d82972700 10 _finish_op 0x73a3da0 seq 2740 osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.425793 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cb680 (0 active)
2014-06-12 14:46:01.425801 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.425789 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3048 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] op_commit: 236182
2014-06-12 14:46:01.425823 7f9d7715b700  5 -- op tracker -- , seq: 747468, time: 2014-06-12 14:46:01.425814, event: op_commit, op: osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.425842 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3048 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] op_applied: 236182
2014-06-12 14:46:01.425866 7f9d7795c700  5 -- op tracker -- , seq: 747468, time: 2014-06-12 14:46:01.425859, event: op_applied, op: osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.425879 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3048 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] op_applied on primary on version 173'3049
2014-06-12 14:46:01.427773 7f9d44101700  5 -- op tracker -- , seq: 747469, time: 2014-06-12 14:46:01.427724, event: header_read, op: osd_sub_op_reply(client.4296.0:143521 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.427785 7f9d44101700  5 -- op tracker -- , seq: 747469, time: 2014-06-12 14:46:01.427726, event: throttled, op: osd_sub_op_reply(client.4296.0:143521 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.427793 7f9d44101700  5 -- op tracker -- , seq: 747469, time: 2014-06-12 14:46:01.427741, event: all_read, op: osd_sub_op_reply(client.4296.0:143521 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.427801 7f9d44101700  5 -- op tracker -- , seq: 747469, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143521 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.427807 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143521 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:01.427813 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.427822 7f9d44101700 15 osd.2 173 enqueue_op 0x1e3ea600 prio 196 cost 0 latency 0.000098 osd_sub_op_reply(client.4296.0:143521 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:01.427857 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.427867 7f9d663dd700 10 osd.2 173 dequeue_op 0x1e3ea600 prio 196 cost 0 latency 0.000143 osd_sub_op_reply(client.4296.0:143521 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 pg pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3048 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean]
2014-06-12 14:46:01.427888 7f9d663dd700  5 -- op tracker -- , seq: 747469, time: 2014-06-12 14:46:01.427885, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143521 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.427901 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3048 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] handle_message: 0x1e3ea600
2014-06-12 14:46:01.427918 7f9d663dd700  5 -- op tracker -- , seq: 747469, time: 2014-06-12 14:46:01.427915, event: started, op: osd_sub_op_reply(client.4296.0:143521 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.427924 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3048 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] sub_op_modify_reply: tid 236182 op  ack_type 4 from 21
2014-06-12 14:46:01.427946 7f9d663dd700  5 -- op tracker -- , seq: 747468, time: 2014-06-12 14:46:01.427939, event: sub_op_commit_rec, op: osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.427956 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3048 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] repop_all_applied: repop tid 236182 all applied 
2014-06-12 14:46:01.427969 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3048 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] eval_repop repgather(0x10450540 173'3049 rep_tid=236182 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.427988 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3048 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean]  sending ack on repgather(0x10450540 173'3049 rep_tid=236182 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x133fec80
2014-06-12 14:46:01.428024 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3048 crt=0'0 lcod 173'3047 mlcod 173'3047 active+clean] repop_all_committed: repop tid 236182 all committed 
2014-06-12 14:46:01.428038 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3047 active+clean] eval_repop repgather(0x10450540 173'3049 rep_tid=236182 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.428057 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3047 active+clean] log_op_stats osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.009115 lat 0.009149
2014-06-12 14:46:01.428089 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3047 active+clean] publish_stats_to_osd 173:3835
2014-06-12 14:46:01.428103 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3047 active+clean]  sending commit on repgather(0x10450540 173'3049 rep_tid=236182 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0xc3fea00
2014-06-12 14:46:01.428136 7f9d663dd700  5 -- op tracker -- , seq: 747468, time: 2014-06-12 14:46:01.428131, event: commit_sent, op: osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.428145 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3047 active+clean]  requeue_ops 
2014-06-12 14:46:01.428155 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean]  removing repgather(0x10450540 173'3049 rep_tid=236182 committed?=1 applied?=1 op=osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.428172 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean]    q front is repgather(0x10450540 173'3049 rep_tid=236182 committed?=1 applied?=1 op=osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.428188 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] remove_repop repgather(0x10450540 173'3049 rep_tid=236182 committed?=1 applied?=1 op=osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.428205 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean]  requeue_ops 
2014-06-12 14:46:01.428233 7f9d663dd700  5 -- op tracker -- , seq: 747468, time: 2014-06-12 14:46:01.428227, event: done, op: osd_op(client.4296.0:143521 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3395584~516096] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.428255 7f9d663dd700 10 osd.2 173 dequeue_op 0x1e3ea600 finish
2014-06-12 14:46:01.428265 7f9d663dd700  5 -- op tracker -- , seq: 747469, time: 2014-06-12 14:46:01.428261, event: done, op: osd_sub_op_reply(client.4296.0:143521 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.428277 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.428281 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.430189 7f9d4a3c5700  5 -- op tracker -- , seq: 747470, time: 2014-06-12 14:46:01.429602, event: header_read, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.430204 7f9d4a3c5700  5 -- op tracker -- , seq: 747470, time: 2014-06-12 14:46:01.429605, event: throttled, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.430212 7f9d4a3c5700  5 -- op tracker -- , seq: 747470, time: 2014-06-12 14:46:01.430134, event: all_read, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.430221 7f9d4a3c5700  5 -- op tracker -- , seq: 747470, time: 0.000000, event: dispatched, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.430229 7f9d4a3c5700 20 osd.2 173 should_share_map client.4296 172.16.96.220:0/1242171 173
2014-06-12 14:46:01.430257 7f9d4a3c5700 15 osd.2 173 enqueue_op 0x23e08100 prio 63 cost 282624 latency 0.000655 osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.430290 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.430296 7f9d67be0700 10 osd.2 173 dequeue_op 0x23e08100 prio 63 cost 282624 latency 0.000694 osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean]
2014-06-12 14:46:01.430320 7f9d67be0700  5 -- op tracker -- , seq: 747470, time: 2014-06-12 14:46:01.430315, event: reached_pg, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.430328 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.430343 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] handle_message: 0x23e08100
2014-06-12 14:46:01.430352 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] do_op osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.430375 7f9d67be0700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_'
2014-06-12 14:46:01.430601 7f9d67be0700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 '_' = 0
2014-06-12 14:46:01.430617 7f9d67be0700 15 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset'
2014-06-12 14:46:01.430711 7f9d67be0700 10 getattr 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 'snapset' = 0
2014-06-12 14:46:01.430719 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] populate_obc_watchers edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.430730 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.430739 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] get_object_context: creating obc from disk: 0x2e7ac2c0
2014-06-12 14:46:01.430747 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] get_object_context: 0x2e7ac2c0 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 rwstate(none n=0 w=0) oi: edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3049 client.4296.0:143521 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3049) ssc: 0x407f6c0 snapset: 0=[]:[]+head
2014-06-12 14:46:01.430765 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] find_object_context edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 @head oi=edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3(173'3049 client.4296.0:143521 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3049)
2014-06-12 14:46:01.430788 7f9d67be0700  5 -- op tracker -- , seq: 747470, time: 2014-06-12 14:46:01.430784, event: started, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.430796 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] execute_ctx 0x5d20600
2014-06-12 14:46:01.430811 7f9d67be0700  5 -- op tracker -- , seq: 747470, time: 2014-06-12 14:46:01.430807, event: started, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.430817 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] do_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] ov 173'3049 av 173'3050 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.430830 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] do_osd_op edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624]
2014-06-12 14:46:01.430840 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.430852 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] do_osd_op  write 3911680~282624
2014-06-12 14:46:01.430866 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] do_osd_op_effects on session 0x1b9b6600
2014-06-12 14:46:01.430875 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 snapset=0x407f6f8  snapc=0=[]
2014-06-12 14:46:01.430885 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] make_writeable edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.430895 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] finish_ctx edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 0x5d20600 op modify  
2014-06-12 14:46:01.430906 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean]  set mtime to 2014-06-12 08:45:03.810298
2014-06-12 14:46:01.430925 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean]  final snapset 0=[]:[]+head in edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.430942 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.430951 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] calc_trim_to 79'48 -> 79'49
2014-06-12 14:46:01.430960 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] new_repop rep_tid 236183 on osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.430975 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] issue_repop rep_tid 236183 o edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.430997 7f9d67be0700  5 -- op tracker -- , seq: 747470, time: 2014-06-12 14:46:01.430993, event: waiting for subops from 21, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.431017 7f9d67be0700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:01.431039 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3049 (79'48,173'3049] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] append_log log((79'48,173'3049], crt=0'0) [173'3050 (173'3049) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143522 2014-06-12 08:45:03.810298]
2014-06-12 14:46:01.431059 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'48,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] add_log_entry 173'3050 (173'3049) modify   edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 by client.4296.0:143522 2014-06-12 08:45:03.810298
2014-06-12 14:46:01.431077 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'48,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.431088 7f9d67be0700 10 trim log((79'48,173'3050], crt=0'0) to 79'49
2014-06-12 14:46:01.431114 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3050, trimmed: 79'49
2014-06-12 14:46:01.431139 7f9d67be0700  5 queue_transactions existing osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.431144 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.431145 7f9d67be0700  5 queue_op 0x5e032c0 seq 2741 osr(3.170 0x276ce20) 284952 bytes   (queue has 1 ops and 284952 bytes)
2014-06-12 14:46:01.431153 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] eval_repop repgather(0x5322b40 173'3050 rep_tid=236183 committed?=0 applied?=0 lock=0 op=osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.431170 7f9d67be0700 10 osd.2 173 dequeue_op 0x23e08100 finish
2014-06-12 14:46:01.431174 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.431176 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.431185 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54cb680 (1 active)
2014-06-12 14:46:01.431189 7f9d7c966700  5 _do_op 0x5e032c0 seq 2741 osr(3.170 0x276ce20)/0x276ce20 start
2014-06-12 14:46:01.431193 7f9d7c966700 10 _do_transaction on 0x53223c0
2014-06-12 14:46:01.431199 7f9d7c966700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.431302 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.431330 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.431383 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.431400 7f9d7c966700 15 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.431407 7f9d7c966700 10 _omap_rmkeys meta/1fe2ad5a/pglog_3.170/0//-1 r = 0
2014-06-12 14:46:01.431412 7f9d7c966700 15 _omap_setkeys meta/1fe2ad5a/pglog_3.170/0//-1
2014-06-12 14:46:01.431531 7f9d7c966700 15 _write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 3911680~282624
2014-06-12 14:46:01.431674 7f9d7c966700 10 lookup_strip_header done  cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.431780 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:01.432671 7f9d7c966700 10 _generic_write 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 3911680~282624 = 0
2014-06-12 14:46:01.432706 7f9d7c966700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.432720 7f9d7c966700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.432727 7f9d7c966700 15 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3
2014-06-12 14:46:01.432734 7f9d7c966700 10 _setattrs 3.170_head/edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 = 0
2014-06-12 14:46:01.432897 7f9d7c966700 20 set_header setting 16954 cid 3.170_head oid edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 parent seq 0
2014-06-12 14:46:01.432949 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.432963 7f9d7c966700 20 set_header setting 2002 cid meta oid 1fe2ad5a/pglog_3.170/0//-1 parent seq 0
2014-06-12 14:46:01.432974 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:01.434219 7f9d7c966700 10 _do_op 0x5e032c0 seq 2741 r = 0, finisher 0x654cb00 0x33d68dc0
2014-06-12 14:46:01.434233 7f9d7c966700 10 _finish_op 0x5e032c0 seq 2741 osr(3.170 0x276ce20)/0x276ce20
2014-06-12 14:46:01.434243 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54cb680 (0 active)
2014-06-12 14:46:01.434246 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.434236 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] op_commit: 236183
2014-06-12 14:46:01.434264 7f9d7715b700  5 -- op tracker -- , seq: 747470, time: 2014-06-12 14:46:01.434256, event: op_commit, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.434277 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] op_applied: 236183
2014-06-12 14:46:01.434303 7f9d7795c700  5 -- op tracker -- , seq: 747470, time: 2014-06-12 14:46:01.434297, event: op_applied, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.434314 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] op_applied on primary on version 173'3050
2014-06-12 14:46:01.436355 7f9d44101700  5 -- op tracker -- , seq: 747471, time: 2014-06-12 14:46:01.436306, event: header_read, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.436375 7f9d44101700  5 -- op tracker -- , seq: 747471, time: 2014-06-12 14:46:01.436307, event: throttled, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.436383 7f9d44101700  5 -- op tracker -- , seq: 747471, time: 2014-06-12 14:46:01.436319, event: all_read, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.436396 7f9d44101700  5 -- op tracker -- , seq: 747471, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.436402 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:01.436410 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.436420 7f9d44101700 15 osd.2 173 enqueue_op 0x1e3ea200 prio 196 cost 0 latency 0.000114 osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2
2014-06-12 14:46:01.436450 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.436460 7f9d693e3700 10 osd.2 173 dequeue_op 0x1e3ea200 prio 196 cost 0 latency 0.000154 osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0) v2 pg pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean]
2014-06-12 14:46:01.436490 7f9d44101700  5 -- op tracker -- , seq: 747472, time: 2014-06-12 14:46:01.436445, event: header_read, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.436509 7f9d44101700  5 -- op tracker -- , seq: 747472, time: 2014-06-12 14:46:01.436446, event: throttled, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.436518 7f9d44101700  5 -- op tracker -- , seq: 747472, time: 2014-06-12 14:46:01.436457, event: all_read, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.436526 7f9d44101700  5 -- op tracker -- , seq: 747472, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.436530 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:01.436535 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:01.436541 7f9d44101700 15 osd.2 173 enqueue_op 0x1702fb00 prio 196 cost 0 latency 0.000095 osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:01.436566 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:01.436598 7f9d693e3700  5 -- op tracker -- , seq: 747471, time: 2014-06-12 14:46:01.436491, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.436607 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] handle_message: 0x1e3ea200
2014-06-12 14:46:01.436621 7f9d693e3700  5 -- op tracker -- , seq: 747471, time: 2014-06-12 14:46:01.436618, event: started, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.436625 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] sub_op_modify_reply: tid 236183 op  ack_type 1 from 21
2014-06-12 14:46:01.436645 7f9d693e3700  5 -- op tracker -- , seq: 747470, time: 2014-06-12 14:46:01.436640, event: sub_op_applied_rec, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.436651 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] repop_all_applied: repop tid 236183 all applied 
2014-06-12 14:46:01.436657 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] eval_repop repgather(0x5322b40 173'3050 rep_tid=236183 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.436667 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean]  sending ack on repgather(0x5322b40 173'3050 rep_tid=236183 committed?=0 applied?=1 lock=0 op=osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x33270a00
2014-06-12 14:46:01.436689 7f9d693e3700 10 osd.2 173 dequeue_op 0x1e3ea200 finish
2014-06-12 14:46:01.436696 7f9d693e3700  5 -- op tracker -- , seq: 747471, time: 2014-06-12 14:46:01.436694, event: done, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ack, result = 0)
2014-06-12 14:46:01.436707 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:01.436708 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.436697 7f9d62bd6700 10 osd.2 173 dequeue_op 0x1702fb00 prio 196 cost 0 latency 0.000252 osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0) v2 pg pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean]
2014-06-12 14:46:01.436716 7f9d62bd6700  5 -- op tracker -- , seq: 747472, time: 2014-06-12 14:46:01.436713, event: reached_pg, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.436724 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] handle_message: 0x1702fb00
2014-06-12 14:46:01.436739 7f9d62bd6700  5 -- op tracker -- , seq: 747472, time: 2014-06-12 14:46:01.436735, event: started, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.436744 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] sub_op_modify_reply: tid 236183 op  ack_type 4 from 21
2014-06-12 14:46:01.436763 7f9d62bd6700  5 -- op tracker -- , seq: 747470, time: 2014-06-12 14:46:01.436757, event: sub_op_commit_rec, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.436769 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3049 crt=0'0 lcod 173'3048 mlcod 173'3048 active+clean] repop_all_committed: repop tid 236183 all committed 
2014-06-12 14:46:01.436780 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3048 active+clean] eval_repop repgather(0x5322b40 173'3050 rep_tid=236183 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.436796 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3048 active+clean] log_op_stats osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4 inb 282903 outb 0 rlat 0.007086 lat 0.007192
2014-06-12 14:46:01.436812 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3048 active+clean] publish_stats_to_osd 173:3836
2014-06-12 14:46:01.436825 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3048 active+clean]  sending commit on repgather(0x5322b40 173'3050 rep_tid=236183 committed?=1 applied?=1 lock=0 op=osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4) 0x1990cf00
2014-06-12 14:46:01.437667 7f9d62bd6700  5 -- op tracker -- , seq: 747470, time: 2014-06-12 14:46:01.437659, event: commit_sent, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.437679 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3048 active+clean]  requeue_ops 
2014-06-12 14:46:01.437692 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3049 active+clean]  removing repgather(0x5322b40 173'3050 rep_tid=236183 committed?=1 applied?=1 op=osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.437706 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3049 active+clean]    q front is repgather(0x5322b40 173'3050 rep_tid=236183 committed?=1 applied?=1 op=osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.437721 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3049 active+clean] remove_repop repgather(0x5322b40 173'3050 rep_tid=236183 committed?=1 applied?=1 op=osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:01.437735 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3049 active+clean]  requeue_ops 
2014-06-12 14:46:01.437767 7f9d62bd6700  5 -- op tracker -- , seq: 747470, time: 2014-06-12 14:46:01.437762, event: done, op: osd_op(client.4296.0:143522 rbd_data.109f6b8b4567.00000000000011a5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3911680~282624] 3.edea970 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.437785 7f9d62bd6700 10 osd.2 173 dequeue_op 0x1702fb00 finish
2014-06-12 14:46:01.437794 7f9d62bd6700  5 -- op tracker -- , seq: 747472, time: 2014-06-12 14:46:01.437791, event: done, op: osd_sub_op_reply(client.4296.0:143522 3.170 edea970/rbd_data.109f6b8b4567.00000000000011a5/head//3 [] ondisk, result = 0)
2014-06-12 14:46:01.437806 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.437809 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.465371 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:01.465393 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:01.465426 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 12.32 >= max 0.5 = no, load too high
2014-06-12 14:46:01.465439 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:01.465441 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92609 < max (604800 seconds)
2014-06-12 14:46:01.465450 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:01.465452 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:01.465453 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:01.465454 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 40; oldest is 35.082247 seconds old
2014-06-12 14:46:01.465475 7f9d84175700  0 log [WRN] : 11 slow requests, 1 included below; oldest blocked for > 35.082247 secs
2014-06-12 14:46:01.465478 7f9d84175700  0 log [WRN] : slow request 31.185553 seconds old, received at 2014-06-12 14:45:30.279901: osd_op(client.4310.0:142391 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~741376] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:46:01.476621 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:01.476624 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:01.476626 7f9d74bfa700 20 osd.2 173 _dispatch 0x1621e4e0 ping v1
2014-06-12 14:46:01.476628 7f9d74bfa700 10 osd.2 173 ping from client.4309
2014-06-12 14:46:01.476630 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:01.476632 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:01.488328 7f9d44424700  5 -- op tracker -- , seq: 747473, time: 2014-06-12 14:46:01.485790, event: header_read, op: osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.488360 7f9d44424700  5 -- op tracker -- , seq: 747473, time: 2014-06-12 14:46:01.485795, event: throttled, op: osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.488377 7f9d44424700  5 -- op tracker -- , seq: 747473, time: 2014-06-12 14:46:01.488285, event: all_read, op: osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.488387 7f9d44424700  5 -- op tracker -- , seq: 747473, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.488397 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:01.488422 7f9d44424700 15 osd.2 173 enqueue_op 0xe20b300 prio 63 cost 1032192 latency 0.002631 osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.488463 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.488473 7f9d6b3e7700 10 osd.2 173 dequeue_op 0xe20b300 prio 63 cost 1032192 latency 0.002682 osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]
2014-06-12 14:46:01.488500 7f9d6b3e7700  5 -- op tracker -- , seq: 747473, time: 2014-06-12 14:46:01.488495, event: reached_pg, op: osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.488639 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.488682 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] handle_message: 0xe20b300
2014-06-12 14:46:01.488696 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_op osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.488719 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] get_object_context: found obc in cache: 0xa9af340
2014-06-12 14:46:01.488728 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] get_object_context: 0xa9af340 e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 rwstate(write n=2 w=0) oi: e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3(173'2508 client.4317.0:143790 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2508) ssc: 0x81bbb20 snapset: 0=[]:[]+head
2014-06-12 14:46:01.488742 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] find_object_context e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 @head oi=e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3(173'2508 client.4317.0:143790 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2508)
2014-06-12 14:46:01.488768 7f9d6b3e7700  5 -- op tracker -- , seq: 747473, time: 2014-06-12 14:46:01.488763, event: started, op: osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.488774 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] execute_ctx 0xfd19e00
2014-06-12 14:46:01.488789 7f9d6b3e7700  5 -- op tracker -- , seq: 747473, time: 2014-06-12 14:46:01.488785, event: started, op: osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.488794 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_op e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] ov 173'2508 av 173'2509 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.488806 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192]
2014-06-12 14:46:01.488816 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.488827 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op  write 1454080~1032192
2014-06-12 14:46:01.488839 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:01.488848 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] make_writeable e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 snapset=0x81bbb58  snapc=0=[]
2014-06-12 14:46:01.488859 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] make_writeable e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.488868 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] finish_ctx e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 0xfd19e00 op modify  
2014-06-12 14:46:01.488878 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  set mtime to 2014-06-12 08:45:03.865831
2014-06-12 14:46:01.488897 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  final snapset 0=[]:[]+head in e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.488912 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.488921 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] new_repop rep_tid 236184 on osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.488935 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] issue_repop rep_tid 236184 o e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.488955 7f9d6b3e7700  5 -- op tracker -- , seq: 747473, time: 2014-06-12 14:46:01.488951, event: waiting for subops from 26, op: osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.488973 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:01.488991 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2508 (0'0,173'2508] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] append_log log((0'0,173'2508], crt=0'0) [173'2509 (173'2508) modify   e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 by client.4317.0:143791 2014-06-12 08:45:03.865831]
2014-06-12 14:46:01.489011 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] add_log_entry 173'2509 (173'2508) modify   e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 by client.4317.0:143791 2014-06-12 08:45:03.865831
2014-06-12 14:46:01.489031 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.489047 7f9d44424700  5 -- op tracker -- , seq: 747474, time: 2014-06-12 14:46:01.488448, event: header_read, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.489057 7f9d44424700  5 -- op tracker -- , seq: 747474, time: 2014-06-12 14:46:01.488450, event: throttled, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.489064 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2509, trimmed: 
2014-06-12 14:46:01.489066 7f9d44424700  5 -- op tracker -- , seq: 747474, time: 2014-06-12 14:46:01.489023, event: all_read, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.489076 7f9d44424700  5 -- op tracker -- , seq: 747474, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.489081 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:01.489085 7f9d6b3e7700  5 queue_transactions existing osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:01.489088 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.489089 7f9d6b3e7700  5 queue_op 0x16a79f80 seq 2325 osr(3.7ae 0x49b4010) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:01.489099 7f9d44424700 15 osd.2 173 enqueue_op 0xe20a900 prio 63 cost 516096 latency 0.000651 osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.489100 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1c0b440 (1 active)
2014-06-12 14:46:01.489098 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] eval_repop repgather(0x523c9c0 173'2509 rep_tid=236184 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.489113 7f9d81970700  5 _do_op 0x16a79f80 seq 2325 osr(3.7ae 0x49b4010)/0x49b4010 start
2014-06-12 14:46:01.489115 7f9d6b3e7700 10 osd.2 173 dequeue_op 0xe20b300 finish
2014-06-12 14:46:01.489119 7f9d81970700 10 _do_transaction on 0x22b79c80
2014-06-12 14:46:01.489119 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.489126 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.489126 7f9d81970700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.489133 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.489130 7f9d6b3e7700 10 osd.2 173 dequeue_op 0xe20a900 prio 63 cost 516096 latency 0.000682 osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]
2014-06-12 14:46:01.489143 7f9d6b3e7700  5 -- op tracker -- , seq: 747474, time: 2014-06-12 14:46:01.489140, event: reached_pg, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.489149 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.489156 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] handle_message: 0xe20a900
2014-06-12 14:46:01.489162 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_op osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.489174 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] get_object_context: found obc in cache: 0xa9af340
2014-06-12 14:46:01.489180 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] get_object_context: 0xa9af340 e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 rwstate(write n=3 w=0) oi: e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3(173'2509 client.4317.0:143791 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2509) ssc: 0x81bbb20 snapset: 0=[]:[]+head
2014-06-12 14:46:01.489188 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] find_object_context e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 @head oi=e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3(173'2509 client.4317.0:143791 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2509)
2014-06-12 14:46:01.489201 7f9d6b3e7700  5 -- op tracker -- , seq: 747474, time: 2014-06-12 14:46:01.489198, event: started, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.489205 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] execute_ctx 0x2e413600
2014-06-12 14:46:01.489214 7f9d6b3e7700  5 -- op tracker -- , seq: 747474, time: 2014-06-12 14:46:01.489211, event: started, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.489218 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_op e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] ov 173'2509 av 173'2510 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.489225 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096]
2014-06-12 14:46:01.489232 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.489239 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op  write 2486272~516096
2014-06-12 14:46:01.489246 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:01.489254 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.489253 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] make_writeable e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 snapset=0x81bbb58  snapc=0=[]
2014-06-12 14:46:01.489259 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] make_writeable e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.489265 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] finish_ctx e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 0x2e413600 op modify  
2014-06-12 14:46:01.489270 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.489272 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  set mtime to 2014-06-12 08:45:03.866223
2014-06-12 14:46:01.489284 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  final snapset 0=[]:[]+head in e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.489294 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.489299 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] new_repop rep_tid 236185 on osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.489308 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] issue_repop rep_tid 236185 o e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.489319 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.489321 7f9d6b3e7700  5 -- op tracker -- , seq: 747474, time: 2014-06-12 14:46:01.489318, event: waiting for subops from 26, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.489332 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:01.489336 7f9d81970700 15 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.489340 7f9d81970700 10 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1 r = 0
2014-06-12 14:46:01.489346 7f9d81970700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.489345 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2509 (0'0,173'2509] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] append_log log((0'0,173'2509], crt=0'0) [173'2510 (173'2509) modify   e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 by client.4317.0:143792 2014-06-12 08:45:03.866223]
2014-06-12 14:46:01.489362 7f9d81970700 15 _write 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 1454080~1032192
2014-06-12 14:46:01.489365 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] add_log_entry 173'2510 (173'2509) modify   e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 by client.4317.0:143792 2014-06-12 08:45:03.866223
2014-06-12 14:46:01.489380 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.489397 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2510, trimmed: 
2014-06-12 14:46:01.489410 7f9d6b3e7700  5 queue_transactions existing osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:01.489412 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.489413 7f9d6b3e7700  5 queue_op 0x5e438c0 seq 2326 osr(3.7ae 0x49b4010) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:46:01.489420 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] eval_repop repgather(0x2a37b980 173'2510 rep_tid=236185 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.489439 7f9d6b3e7700 10 osd.2 173 dequeue_op 0xe20a900 finish
2014-06-12 14:46:01.489442 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.489444 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.489521 7f9d81970700 10 lookup_strip_header done  cid 3.7ae_head oid e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.489552 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:01.489642 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1c0b440 (2 active)
2014-06-12 14:46:01.497932 7f9d81970700 10 _generic_write 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 1454080~1032192 = 0
2014-06-12 14:46:01.498097 7f9d81970700 15 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.498117 7f9d81970700 10 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 = 0
2014-06-12 14:46:01.498122 7f9d81970700 15 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.498127 7f9d81970700 10 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 = 0
2014-06-12 14:46:01.498459 7f9d81970700 20 set_header setting 16740 cid 3.7ae_head oid e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 parent seq 0
2014-06-12 14:46:01.498554 7f9d81970700 20 set_header setting 2150 cid meta oid 1fe77416/pglog_3.7ae/0//-1 parent seq 0
2014-06-12 14:46:01.498570 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.498575 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:01.502652 7f9d81970700 10 _do_op 0x16a79f80 seq 2325 r = 0, finisher 0x6f5e380 0x3444f40
2014-06-12 14:46:01.502671 7f9d81970700 10 _finish_op 0x16a79f80 seq 2325 osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:01.502680 7f9d7a161700  5 _do_op 0x5e438c0 seq 2326 osr(3.7ae 0x49b4010)/0x49b4010 start
2014-06-12 14:46:01.502681 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1c0b440 (1 active)
2014-06-12 14:46:01.502688 7f9d7a161700 10 _do_transaction on 0x31ee78c0
2014-06-12 14:46:01.502689 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.502677 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_commit: 236184
2014-06-12 14:46:01.502695 7f9d7a161700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.502707 7f9d7715b700  5 -- op tracker -- , seq: 747473, time: 2014-06-12 14:46:01.502698, event: op_commit, op: osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.502841 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.502862 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.502918 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.502940 7f9d7a161700 15 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.502948 7f9d7a161700 10 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1 r = 0
2014-06-12 14:46:01.502954 7f9d7a161700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.503041 7f9d7a161700 15 _write 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 2486272~516096
2014-06-12 14:46:01.503150 7f9d7a161700 10 lookup_strip_header done  cid 3.7ae_head oid e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.503276 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:01.504794 7f9d7a161700 10 _generic_write 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 2486272~516096 = 0
2014-06-12 14:46:01.504847 7f9d7a161700 15 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.504862 7f9d7a161700 10 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 = 0
2014-06-12 14:46:01.504871 7f9d7a161700 15 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.504894 7f9d7a161700 10 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 = 0
2014-06-12 14:46:01.505043 7f9d7a161700 20 set_header setting 16740 cid 3.7ae_head oid e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 parent seq 0
2014-06-12 14:46:01.505106 7f9d7a161700 20 set_header setting 2150 cid meta oid 1fe77416/pglog_3.7ae/0//-1 parent seq 0
2014-06-12 14:46:01.505119 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.505125 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:01.505418 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_applied: 236184
2014-06-12 14:46:01.505449 7f9d7795c700  5 -- op tracker -- , seq: 747473, time: 2014-06-12 14:46:01.505439, event: op_applied, op: osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.505458 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_applied on primary on version 173'2509
2014-06-12 14:46:01.508746 7f9d7a161700 10 _do_op 0x5e438c0 seq 2326 r = 0, finisher 0x2057600 0x6be3280
2014-06-12 14:46:01.508765 7f9d7a161700 10 _finish_op 0x5e438c0 seq 2326 osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:01.508772 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_commit: 236185
2014-06-12 14:46:01.508806 7f9d7715b700  5 -- op tracker -- , seq: 747474, time: 2014-06-12 14:46:01.508797, event: op_commit, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.508870 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1c0b440 (0 active)
2014-06-12 14:46:01.508875 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.508877 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_applied: 236185
2014-06-12 14:46:01.508924 7f9d7795c700  5 -- op tracker -- , seq: 747474, time: 2014-06-12 14:46:01.508895, event: op_applied, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.508932 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_applied on primary on version 173'2510
2014-06-12 14:46:01.510304 7f9d44424700  5 -- op tracker -- , seq: 747475, time: 2014-06-12 14:46:01.508099, event: header_read, op: osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.510316 7f9d44424700  5 -- op tracker -- , seq: 747475, time: 2014-06-12 14:46:01.508109, event: throttled, op: osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.510365 7f9d44424700  5 -- op tracker -- , seq: 747475, time: 2014-06-12 14:46:01.510264, event: all_read, op: osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.510376 7f9d44424700  5 -- op tracker -- , seq: 747475, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.510386 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:01.510419 7f9d44424700 15 osd.2 173 enqueue_op 0xe20b400 prio 63 cost 1032192 latency 0.002320 osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.510465 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.510474 7f9d683e1700 10 osd.2 173 dequeue_op 0xe20b400 prio 63 cost 1032192 latency 0.002375 osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]
2014-06-12 14:46:01.510503 7f9d683e1700  5 -- op tracker -- , seq: 747475, time: 2014-06-12 14:46:01.510496, event: reached_pg, op: osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.510516 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.510539 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] handle_message: 0xe20b400
2014-06-12 14:46:01.510552 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_op osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.510590 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] get_object_context: found obc in cache: 0xa9af340
2014-06-12 14:46:01.510604 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] get_object_context: 0xa9af340 e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 rwstate(write n=4 w=0) oi: e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3(173'2510 client.4317.0:143792 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2510) ssc: 0x81bbb20 snapset: 0=[]:[]+head
2014-06-12 14:46:01.510622 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] find_object_context e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 @head oi=e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3(173'2510 client.4317.0:143792 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2510)
2014-06-12 14:46:01.510650 7f9d683e1700  5 -- op tracker -- , seq: 747475, time: 2014-06-12 14:46:01.510644, event: started, op: osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.510659 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] execute_ctx 0x589d000
2014-06-12 14:46:01.510678 7f9d683e1700  5 -- op tracker -- , seq: 747475, time: 2014-06-12 14:46:01.510673, event: started, op: osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.510687 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_op e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] ov 173'2510 av 173'2511 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.510702 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192]
2014-06-12 14:46:01.510715 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.510728 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op  write 3002368~1032192
2014-06-12 14:46:01.510745 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:01.510758 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] make_writeable e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 snapset=0x81bbb58  snapc=0=[]
2014-06-12 14:46:01.510773 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] make_writeable e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.510786 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] finish_ctx e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 0x589d000 op modify  
2014-06-12 14:46:01.510798 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  set mtime to 2014-06-12 08:45:03.887340
2014-06-12 14:46:01.510820 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  final snapset 0=[]:[]+head in e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.510842 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.510853 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] new_repop rep_tid 236186 on osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.510872 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] issue_repop rep_tid 236186 o e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.510901 7f9d683e1700  5 -- op tracker -- , seq: 747475, time: 2014-06-12 14:46:01.510896, event: waiting for subops from 26, op: osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.510922 7f9d683e1700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:01.510947 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2510 (0'0,173'2510] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] append_log log((0'0,173'2510], crt=0'0) [173'2511 (173'2510) modify   e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 by client.4317.0:143793 2014-06-12 08:45:03.887340]
2014-06-12 14:46:01.510972 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] add_log_entry 173'2511 (173'2510) modify   e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 by client.4317.0:143793 2014-06-12 08:45:03.887340
2014-06-12 14:46:01.510996 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.511030 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2511, trimmed: 
2014-06-12 14:46:01.511057 7f9d683e1700  5 queue_transactions existing osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:01.511063 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.511064 7f9d683e1700  5 queue_op 0x19ed68a0 seq 2327 osr(3.7ae 0x49b4010) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:01.511072 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] eval_repop repgather(0x51add40 173'2511 rep_tid=236186 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.511093 7f9d683e1700 10 osd.2 173 dequeue_op 0xe20b400 finish
2014-06-12 14:46:01.511097 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.511100 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.511108 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1c0b440 (1 active)
2014-06-12 14:46:01.511113 7f9d7a962700  5 _do_op 0x19ed68a0 seq 2327 osr(3.7ae 0x49b4010)/0x49b4010 start
2014-06-12 14:46:01.511118 7f9d7a962700 10 _do_transaction on 0x51ad500
2014-06-12 14:46:01.511124 7f9d7a962700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.511274 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.511298 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.511354 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.511376 7f9d7a962700 15 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.511384 7f9d7a962700 10 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1 r = 0
2014-06-12 14:46:01.511389 7f9d7a962700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.511485 7f9d7a962700 15 _write 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 3002368~1032192
2014-06-12 14:46:01.511643 7f9d7a962700 10 lookup_strip_header done  cid 3.7ae_head oid e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.511786 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:01.515792 7f9d7a962700 10 _generic_write 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 3002368~1032192 = 0
2014-06-12 14:46:01.515871 7f9d7a962700 15 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.515884 7f9d7a962700 10 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 = 0
2014-06-12 14:46:01.515888 7f9d7a962700 15 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.515893 7f9d7a962700 10 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 = 0
2014-06-12 14:46:01.516102 7f9d7a962700 20 set_header setting 16740 cid 3.7ae_head oid e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 parent seq 0
2014-06-12 14:46:01.516195 7f9d7a962700 20 set_header setting 2150 cid meta oid 1fe77416/pglog_3.7ae/0//-1 parent seq 0
2014-06-12 14:46:01.516207 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.516212 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:01.520683 7f9d7a962700 10 _do_op 0x19ed68a0 seq 2327 r = 0, finisher 0x73fa680 0x1e627980
2014-06-12 14:46:01.520716 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_commit: 236186
2014-06-12 14:46:01.520751 7f9d7715b700  5 -- op tracker -- , seq: 747475, time: 2014-06-12 14:46:01.520741, event: op_commit, op: osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.520774 7f9d7a962700 10 _finish_op 0x19ed68a0 seq 2327 osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:01.520790 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_applied: 236186
2014-06-12 14:46:01.520807 7f9d7795c700  5 -- op tracker -- , seq: 747475, time: 2014-06-12 14:46:01.520802, event: op_applied, op: osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.520818 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_applied on primary on version 173'2511
2014-06-12 14:46:01.520842 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1c0b440 (0 active)
2014-06-12 14:46:01.520851 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.584987 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:01.584990 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:01.584991 7f9d74bfa700 20 osd.2 173 _dispatch 0x14bc9a00 ping v1
2014-06-12 14:46:01.584993 7f9d74bfa700 10 osd.2 173 ping from client.4310
2014-06-12 14:46:01.584995 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:01.584995 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:01.616388 7f9d44424700  5 -- op tracker -- , seq: 747476, time: 2014-06-12 14:46:01.616018, event: header_read, op: osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.616425 7f9d44424700  5 -- op tracker -- , seq: 747476, time: 2014-06-12 14:46:01.616022, event: throttled, op: osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.616446 7f9d44424700  5 -- op tracker -- , seq: 747476, time: 2014-06-12 14:46:01.616346, event: all_read, op: osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.616468 7f9d44424700  5 -- op tracker -- , seq: 747476, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.616493 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:01.616546 7f9d44424700 15 osd.2 173 enqueue_op 0x59ddc00 prio 63 cost 159744 latency 0.000528 osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.616614 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.616629 7f9d65bdc700 10 osd.2 173 dequeue_op 0x59ddc00 prio 63 cost 159744 latency 0.000611 osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]
2014-06-12 14:46:01.616685 7f9d65bdc700  5 -- op tracker -- , seq: 747476, time: 2014-06-12 14:46:01.616673, event: reached_pg, op: osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.616702 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:01.616737 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] handle_message: 0x59ddc00
2014-06-12 14:46:01.616762 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_op osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:01.616824 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] get_object_context: found obc in cache: 0xa9af340
2014-06-12 14:46:01.616853 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] get_object_context: 0xa9af340 e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 rwstate(write n=5 w=0) oi: e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3(173'2511 client.4317.0:143793 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2511) ssc: 0x81bbb20 snapset: 0=[]:[]+head
2014-06-12 14:46:01.616892 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] find_object_context e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 @head oi=e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3(173'2511 client.4317.0:143793 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2511)
2014-06-12 14:46:01.616954 7f9d65bdc700  5 -- op tracker -- , seq: 747476, time: 2014-06-12 14:46:01.616940, event: started, op: osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.616972 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] execute_ctx 0xa051600
2014-06-12 14:46:01.617014 7f9d65bdc700  5 -- op tracker -- , seq: 747476, time: 2014-06-12 14:46:01.617001, event: started, op: osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.617034 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_op e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] ov 173'2511 av 173'2512 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:01.617069 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744]
2014-06-12 14:46:01.617100 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:01.617132 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op  write 4034560~159744
2014-06-12 14:46:01.617168 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:01.617199 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] make_writeable e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 snapset=0x81bbb58  snapc=0=[]
2014-06-12 14:46:01.617231 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] make_writeable e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:01.617260 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] finish_ctx e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 0xa051600 op modify  
2014-06-12 14:46:01.617291 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  set mtime to 2014-06-12 08:45:03.996792
2014-06-12 14:46:01.617340 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  final snapset 0=[]:[]+head in e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.617388 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  zeroing write result code 0
2014-06-12 14:46:01.617414 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] new_repop rep_tid 236187 on osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:01.617455 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] issue_repop rep_tid 236187 o e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.617510 7f9d65bdc700  5 -- op tracker -- , seq: 747476, time: 2014-06-12 14:46:01.617498, event: waiting for subops from 26, op: osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.617556 7f9d65bdc700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:01.617601 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2511 (0'0,173'2511] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] append_log log((0'0,173'2511], crt=0'0) [173'2512 (173'2511) modify   e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 by client.4317.0:143794 2014-06-12 08:45:03.996792]
2014-06-12 14:46:01.617662 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] add_log_entry 173'2512 (173'2511) modify   e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 by client.4317.0:143794 2014-06-12 08:45:03.996792
2014-06-12 14:46:01.617714 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] append_log  adding 1 keys
2014-06-12 14:46:01.617783 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2512, trimmed: 
2014-06-12 14:46:01.617826 7f9d65bdc700  5 queue_transactions existing osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:01.617838 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.617841 7f9d65bdc700  5 queue_op 0x208be8a0 seq 2328 osr(3.7ae 0x49b4010) 162037 bytes   (queue has 1 ops and 162037 bytes)
2014-06-12 14:46:01.617873 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1c0b440 (1 active)
2014-06-12 14:46:01.617855 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] eval_repop repgather(0x384e3800 173'2512 rep_tid=236187 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:01.617896 7f9d7895e700  5 _do_op 0x208be8a0 seq 2328 osr(3.7ae 0x49b4010)/0x49b4010 start
2014-06-12 14:46:01.617897 7f9d65bdc700 10 osd.2 173 dequeue_op 0x59ddc00 finish
2014-06-12 14:46:01.617903 7f9d7895e700 10 _do_transaction on 0x384e2e40
2014-06-12 14:46:01.617906 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.617913 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.617912 7f9d7895e700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.618087 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.618125 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.618205 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.618239 7f9d7895e700 15 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.618249 7f9d7895e700 10 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1 r = 0
2014-06-12 14:46:01.618258 7f9d7895e700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:01.618382 7f9d7895e700 15 _write 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 4034560~159744
2014-06-12 14:46:01.618588 7f9d7895e700 10 lookup_strip_header done  cid 3.7ae_head oid e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.618778 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:01.619553 7f9d7895e700 10 _generic_write 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 4034560~159744 = 0
2014-06-12 14:46:01.619595 7f9d7895e700 15 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.619612 7f9d7895e700 10 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 = 0
2014-06-12 14:46:01.619624 7f9d7895e700 15 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3
2014-06-12 14:46:01.619635 7f9d7895e700 10 _setattrs 3.7ae_head/e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 = 0
2014-06-12 14:46:01.619882 7f9d7895e700 20 set_header setting 16740 cid 3.7ae_head oid e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 parent seq 0
2014-06-12 14:46:01.619934 7f9d7895e700 20 set_header setting 2150 cid meta oid 1fe77416/pglog_3.7ae/0//-1 parent seq 0
2014-06-12 14:46:01.619956 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.619968 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:01.621082 7f9d7895e700 10 _do_op 0x208be8a0 seq 2328 r = 0, finisher 0x153109a0 0x16945d00
2014-06-12 14:46:01.621104 7f9d7895e700 10 _finish_op 0x208be8a0 seq 2328 osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:01.621119 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1c0b440 (0 active)
2014-06-12 14:46:01.621125 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.621113 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_commit: 236187
2014-06-12 14:46:01.621171 7f9d7715b700  5 -- op tracker -- , seq: 747476, time: 2014-06-12 14:46:01.621152, event: op_commit, op: osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.621203 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_applied: 236187
2014-06-12 14:46:01.621238 7f9d7795c700  5 -- op tracker -- , seq: 747476, time: 2014-06-12 14:46:01.621229, event: op_applied, op: osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:01.621264 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] op_applied on primary on version 173'2512
2014-06-12 14:46:01.632849 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:01.632862 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:01.632864 7f9d74bfa700 20 osd.2 173 _dispatch 0x330a8b60 ping v1
2014-06-12 14:46:01.632866 7f9d74bfa700 10 osd.2 173 ping from client.4311
2014-06-12 14:46:01.632868 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:01.632870 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:01.864736 7f9d5a16f700  5 -- op tracker -- , seq: 747477, time: 2014-06-12 14:46:01.863081, event: header_read, op: osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:01.864750 7f9d5a16f700  5 -- op tracker -- , seq: 747477, time: 2014-06-12 14:46:01.863085, event: throttled, op: osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:01.864758 7f9d5a16f700  5 -- op tracker -- , seq: 747477, time: 2014-06-12 14:46:01.864690, event: all_read, op: osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:01.864767 7f9d5a16f700  5 -- op tracker -- , seq: 747477, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:01.864772 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:01.864779 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:01.864787 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329eb600 prio 127 cost 516903 latency 0.001706 osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:01.864812 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:01.864819 7f9d6abe6700 10 osd.2 173 dequeue_op 0x329eb600 prio 127 cost 516903 latency 0.001738 osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4e0( v 173'2630 (0'0,173'2630] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active]
2014-06-12 14:46:01.864843 7f9d6abe6700  5 -- op tracker -- , seq: 747477, time: 2014-06-12 14:46:01.864838, event: reached_pg, op: osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:01.864851 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2630 (0'0,173'2630] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active] handle_message: 0x329eb600
2014-06-12 14:46:01.864864 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2630 (0'0,173'2630] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active] sub_op_modify trans 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 v 173'2631 (transaction) 174
2014-06-12 14:46:01.864881 7f9d6abe6700  5 -- op tracker -- , seq: 747477, time: 2014-06-12 14:46:01.864876, event: started, op: osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:01.864894 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2630 (0'0,173'2630] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active] append_log log((0'0,173'2630], crt=0'0) [173'2631 (173'2630) modify   6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 by client.4306.0:143774 2014-06-12 08:43:00.251469]
2014-06-12 14:46:01.864916 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2631 (0'0,173'2631] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active] add_log_entry 173'2631 (173'2630) modify   6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 by client.4306.0:143774 2014-06-12 08:43:00.251469
2014-06-12 14:46:01.864939 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2631 (0'0,173'2631] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active] append_log  adding 1 keys
2014-06-12 14:46:01.864972 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2631, trimmed: 
2014-06-12 14:46:01.864997 7f9d6abe6700  5 -- op tracker -- , seq: 747477, time: 2014-06-12 14:46:01.864993, event: started, op: osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:01.865007 7f9d6abe6700  5 queue_transactions existing osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:01.865012 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:01.865014 7f9d6abe6700  5 queue_op 0x8cd3b00 seq 2431 osr(3.4e0 0x276dbd0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:01.865023 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5180900 (1 active)
2014-06-12 14:46:01.865031 7f9d7f16b700  5 _do_op 0x8cd3b00 seq 2431 osr(3.4e0 0x276dbd0)/0x276dbd0 start
2014-06-12 14:46:01.865020 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2631 (0'0,173'2631] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active] do_sub_op osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:01.865035 7f9d7f16b700 10 _do_transaction on 0x2e9b20f0
2014-06-12 14:46:01.865036 7f9d6abe6700 10 osd.2 173 dequeue_op 0x329eb600 finish
2014-06-12 14:46:01.865040 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:01.865042 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:01.865043 7f9d7f16b700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:01.865258 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:01.865287 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:01.865339 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:01.865364 7f9d7f16b700 15 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:01.865372 7f9d7f16b700 10 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1 r = 0
2014-06-12 14:46:01.865378 7f9d7f16b700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:01.865395 7f9d7f16b700 15 _write 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 286720~516096
2014-06-12 14:46:01.865558 7f9d7f16b700 10 lookup_strip_header done  cid 3.4e0_head oid 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:01.865845 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:01.867833 7f9d7f16b700 10 _generic_write 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 286720~516096 = 0
2014-06-12 14:46:01.867895 7f9d7f16b700 15 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:01.867913 7f9d7f16b700 10 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:01.867919 7f9d7f16b700 15 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:01.867927 7f9d7f16b700 10 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:01.868142 7f9d7f16b700 20 set_header setting 16555 cid 3.4e0_head oid 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:01.868227 7f9d7f16b700 20 set_header setting 2236 cid meta oid 1fe07435/pglog_3.4e0/0//-1 parent seq 0
2014-06-12 14:46:01.868242 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:01.868251 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:01.876454 7f9d7f16b700 10 _do_op 0x8cd3b00 seq 2431 r = 0, finisher 0x883d320 0
2014-06-12 14:46:01.876482 7f9d7f16b700 10 _finish_op 0x8cd3b00 seq 2431 osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:01.876510 7f9d7795c700  5 -- op tracker -- , seq: 747477, time: 2014-06-12 14:46:01.876501, event: sub_op_applied, op: osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:01.876522 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2631 (0'0,173'2631] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active] sub_op_modify_applied on 0x2e9b2000 op osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:01.876551 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:01.876579 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2631 (0'0,173'2631] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active] op_applied on primary on version 173'2631
2014-06-12 14:46:01.876593 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2631 (0'0,173'2631] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active] op_applied on replica on version 173'2631
2014-06-12 14:46:01.876608 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5180900 (0 active)
2014-06-12 14:46:01.876624 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:01.877439 7f9d7715b700  5 -- op tracker -- , seq: 747477, time: 2014-06-12 14:46:01.877429, event: commit_sent, op: osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:01.877453 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2631 (0'0,173'2631] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2629 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:01.877478 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:01.877517 7f9d7715b700  5 -- op tracker -- , seq: 747477, time: 2014-06-12 14:46:01.877513, event: done, op: osd_sub_op(client.4306.0:143774 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2631 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.006045 7f9d5a16f700  5 -- op tracker -- , seq: 747478, time: 2014-06-12 14:46:02.005998, event: header_read, op: osd_sub_op_reply(client.4310.0:142386 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.006057 7f9d5a16f700  5 -- op tracker -- , seq: 747478, time: 2014-06-12 14:46:02.006000, event: throttled, op: osd_sub_op_reply(client.4310.0:142386 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.006064 7f9d5a16f700  5 -- op tracker -- , seq: 747478, time: 2014-06-12 14:46:02.006012, event: all_read, op: osd_sub_op_reply(client.4310.0:142386 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.006071 7f9d5a16f700  5 -- op tracker -- , seq: 747478, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142386 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.006075 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142386 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.006081 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:02.006088 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329ebb00 prio 196 cost 0 latency 0.000090 osd_sub_op_reply(client.4310.0:142386 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.006115 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.006121 7f9d66bde700 10 osd.2 173 dequeue_op 0x329ebb00 prio 196 cost 0 latency 0.000123 osd_sub_op_reply(client.4310.0:142386 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2 pg pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]
2014-06-12 14:46:02.006139 7f9d66bde700  5 -- op tracker -- , seq: 747478, time: 2014-06-12 14:46:02.006136, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142386 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.006147 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] handle_message: 0x329ebb00
2014-06-12 14:46:02.006160 7f9d66bde700  5 -- op tracker -- , seq: 747478, time: 2014-06-12 14:46:02.006157, event: started, op: osd_sub_op_reply(client.4310.0:142386 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.006165 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] sub_op_modify_reply: tid 235991 op  ack_type 4 from 24
2014-06-12 14:46:02.006183 7f9d66bde700  5 -- op tracker -- , seq: 746897, time: 2014-06-12 14:46:02.006176, event: sub_op_commit_rec, op: osd_op(client.4310.0:142386 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.006189 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] repop_all_applied: repop tid 235991 all applied 
2014-06-12 14:46:02.006233 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] eval_repop repgather(0xba84f00 173'2698 rep_tid=235991 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142386 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.006248 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]  sending ack on repgather(0xba84f00 173'2698 rep_tid=235991 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142386 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) 0x2b2fcf00
2014-06-12 14:46:02.006274 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] repop_all_committed: repop tid 235991 all committed 
2014-06-12 14:46:02.006283 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2696 active+clean] eval_repop repgather(0xba84f00 173'2698 rep_tid=235991 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142386 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.006296 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2696 active+clean] log_op_stats osd_op(client.4310.0:142386 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4 inb 356631 outb 0 rlat 31.808619 lat 31.808642
2014-06-12 14:46:02.006312 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2696 active+clean] publish_stats_to_osd 173:2881
2014-06-12 14:46:02.006324 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2696 active+clean]  sending commit on repgather(0xba84f00 173'2698 rep_tid=235991 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142386 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) 0x11667180
2014-06-12 14:46:02.006350 7f9d66bde700  5 -- op tracker -- , seq: 746897, time: 2014-06-12 14:46:02.006345, event: commit_sent, op: osd_op(client.4310.0:142386 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.006357 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2696 active+clean]  requeue_ops 
2014-06-12 14:46:02.006366 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]  removing repgather(0xba84f00 173'2698 rep_tid=235991 committed?=1 applied?=1 op=osd_op(client.4310.0:142386 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.006379 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]    q front is repgather(0xba84f00 173'2698 rep_tid=235991 committed?=1 applied?=1 op=osd_op(client.4310.0:142386 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.006393 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] remove_repop repgather(0xba84f00 173'2698 rep_tid=235991 committed?=1 applied?=1 op=osd_op(client.4310.0:142386 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.006405 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]  requeue_ops 
2014-06-12 14:46:02.006423 7f9d66bde700  5 -- op tracker -- , seq: 746897, time: 2014-06-12 14:46:02.006418, event: done, op: osd_op(client.4310.0:142386 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.006439 7f9d66bde700 10 osd.2 173 dequeue_op 0x329ebb00 finish
2014-06-12 14:46:02.006445 7f9d66bde700  5 -- op tracker -- , seq: 747478, time: 2014-06-12 14:46:02.006442, event: done, op: osd_sub_op_reply(client.4310.0:142386 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.006454 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.006456 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.014774 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:02.014847 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:02.021145 7f9d42918700  5 -- op tracker -- , seq: 747479, time: 2014-06-12 14:46:02.018149, event: header_read, op: osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.021161 7f9d42918700  5 -- op tracker -- , seq: 747479, time: 2014-06-12 14:46:02.018152, event: throttled, op: osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.021171 7f9d42918700  5 -- op tracker -- , seq: 747479, time: 2014-06-12 14:46:02.021087, event: all_read, op: osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.021180 7f9d42918700  5 -- op tracker -- , seq: 747479, time: 0.000000, event: dispatched, op: osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.021189 7f9d42918700 20 osd.2 173 should_share_map client.4312 172.16.96.220:0/1244332 173
2014-06-12 14:46:02.021218 7f9d42918700 15 osd.2 173 enqueue_op 0x55c3b00 prio 63 cost 1294336 latency 0.003069 osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.021257 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.021265 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x55c3b00 prio 63 cost 1294336 latency 0.003115 osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean]
2014-06-12 14:46:02.021306 7f9d6a3e5700  5 -- op tracker -- , seq: 747479, time: 2014-06-12 14:46:02.021284, event: reached_pg, op: osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.021315 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:02.021330 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] handle_message: 0x55c3b00
2014-06-12 14:46:02.021340 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] do_op osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:02.021364 7f9d6a3e5700 15 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 '_'
2014-06-12 14:46:02.021647 7f9d6a3e5700 10 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 '_' = 0
2014-06-12 14:46:02.021664 7f9d6a3e5700 15 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 'snapset'
2014-06-12 14:46:02.021742 7f9d6a3e5700 10 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 'snapset' = 0
2014-06-12 14:46:02.021751 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] populate_obc_watchers d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:02.021762 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:02.021771 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] get_object_context: creating obc from disk: 0x25ced8c0
2014-06-12 14:46:02.021780 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] get_object_context: 0x25ced8c0 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 rwstate(none n=0 w=0) oi: d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3(173'686 client.4312.0:38519 wrlock_by=unknown.0.0:0 dirty s 4194304 uv686) ssc: 0x1e15e380 snapset: 0=[]:[]+head
2014-06-12 14:46:02.021811 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] find_object_context d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 @head oi=d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3(173'686 client.4312.0:38519 wrlock_by=unknown.0.0:0 dirty s 4194304 uv686)
2014-06-12 14:46:02.021841 7f9d6a3e5700  5 -- op tracker -- , seq: 747479, time: 2014-06-12 14:46:02.021835, event: started, op: osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.021849 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] execute_ctx 0x2a83200
2014-06-12 14:46:02.021866 7f9d6a3e5700  5 -- op tracker -- , seq: 747479, time: 2014-06-12 14:46:02.021861, event: started, op: osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.021872 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] do_op d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] ov 173'686 av 173'2731 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:02.021904 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] do_osd_op d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336]
2014-06-12 14:46:02.021916 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:02.021929 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] do_osd_op  write 0~1294336
2014-06-12 14:46:02.021945 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] do_osd_op_effects on session 0x18a5e000
2014-06-12 14:46:02.021956 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] make_writeable d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 snapset=0x1e15e3b8  snapc=0=[]
2014-06-12 14:46:02.021968 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] make_writeable d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:02.021977 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] finish_ctx d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 0x2a83200 op modify  
2014-06-12 14:46:02.022003 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean]  set mtime to 2014-06-12 08:45:04.396998
2014-06-12 14:46:02.022060 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean]  final snapset 0=[]:[]+head in d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:02.022094 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean]  zeroing write result code 0
2014-06-12 14:46:02.022108 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] new_repop rep_tid 236188 on osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.022125 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] issue_repop rep_tid 236188 o d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:02.022149 7f9d6a3e5700  5 -- op tracker -- , seq: 747479, time: 2014-06-12 14:46:02.022143, event: waiting for subops from 25, op: osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.022174 7f9d6a3e5700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:02.022240 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2730 (0'0,173'2730] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] append_log log((0'0,173'2730], crt=0'0) [173'2731 (173'686) modify   d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 by client.4312.0:144979 2014-06-12 08:45:04.396998]
2014-06-12 14:46:02.022266 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2730 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] add_log_entry 173'2731 (173'686) modify   d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 by client.4312.0:144979 2014-06-12 08:45:04.396998
2014-06-12 14:46:02.022287 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2730 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] append_log  adding 1 keys
2014-06-12 14:46:02.022320 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2731, trimmed: 
2014-06-12 14:46:02.022341 7f9d6a3e5700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:02.022345 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:02.022347 7f9d6a3e5700  5 queue_op 0x245cd20 seq 2543 osr(3.122 0x276cf40) 1296629 bytes   (queue has 1 ops and 1296629 bytes)
2014-06-12 14:46:02.022359 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:46:02.022367 7f9d7e169700  5 _do_op 0x245cd20 seq 2543 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:46:02.022371 7f9d7e169700 10 _do_transaction on 0x15acfec0
2014-06-12 14:46:02.022354 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2730 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] eval_repop repgather(0x24e8de00 173'2731 rep_tid=236188 committed?=0 applied?=0 lock=0 op=osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.022378 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x55c3b00 finish
2014-06-12 14:46:02.022378 7f9d7e169700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:02.022381 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.022385 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.022474 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:02.022500 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:02.022544 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:02.022564 7f9d7e169700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:02.022570 7f9d7e169700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:46:02.022575 7f9d7e169700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:02.022592 7f9d7e169700 15 _write 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 0~1294336
2014-06-12 14:46:02.022725 7f9d7e169700 10 lookup_strip_header done  cid 3.122_head oid d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:02.022977 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:02.028376 7f9d7e169700 10 _generic_write 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 0~1294336 = 0
2014-06-12 14:46:02.028481 7f9d7e169700 15 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:02.028499 7f9d7e169700 10 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 = 0
2014-06-12 14:46:02.028507 7f9d7e169700 15 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:02.028516 7f9d7e169700 10 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 = 0
2014-06-12 14:46:02.028849 7f9d7e169700 20 set_header setting 17279 cid 3.122_head oid d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 parent seq 0
2014-06-12 14:46:02.028981 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:02.028998 7f9d7e169700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:46:02.029012 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:02.033999 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:02.034047 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:02.035423 7f9d7e169700 10 _do_op 0x245cd20 seq 2543 r = 0, finisher 0x1f086340 0x1a3f0f00
2014-06-12 14:46:02.035445 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2730 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] op_commit: 236188
2014-06-12 14:46:02.035480 7f9d7715b700  5 -- op tracker -- , seq: 747479, time: 2014-06-12 14:46:02.035469, event: op_commit, op: osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.035497 7f9d7e169700 10 _finish_op 0x245cd20 seq 2543 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:02.035511 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:46:02.035519 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.035515 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2730 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] op_applied: 236188
2014-06-12 14:46:02.035538 7f9d7795c700  5 -- op tracker -- , seq: 747479, time: 2014-06-12 14:46:02.035531, event: op_applied, op: osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.035546 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2730 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] op_applied on primary on version 173'2731
2014-06-12 14:46:02.041235 7f9d5bec4700  5 -- op tracker -- , seq: 747480, time: 2014-06-12 14:46:02.041195, event: header_read, op: osd_sub_op_reply(client.4312.0:144979 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.041248 7f9d5bec4700  5 -- op tracker -- , seq: 747480, time: 2014-06-12 14:46:02.041196, event: throttled, op: osd_sub_op_reply(client.4312.0:144979 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.041257 7f9d5bec4700  5 -- op tracker -- , seq: 747480, time: 2014-06-12 14:46:02.041207, event: all_read, op: osd_sub_op_reply(client.4312.0:144979 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.041270 7f9d5bec4700  5 -- op tracker -- , seq: 747480, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4312.0:144979 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.041275 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4312.0:144979 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.041281 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:02.041289 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e87a00 prio 196 cost 0 latency 0.000094 osd_sub_op_reply(client.4312.0:144979 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.041315 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.041323 7f9d63bd8700 10 osd.2 173 dequeue_op 0x20e87a00 prio 196 cost 0 latency 0.000128 osd_sub_op_reply(client.4312.0:144979 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2730 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean]
2014-06-12 14:46:02.041339 7f9d63bd8700  5 -- op tracker -- , seq: 747480, time: 2014-06-12 14:46:02.041336, event: reached_pg, op: osd_sub_op_reply(client.4312.0:144979 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.041346 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2730 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] handle_message: 0x20e87a00
2014-06-12 14:46:02.041357 7f9d63bd8700  5 -- op tracker -- , seq: 747480, time: 2014-06-12 14:46:02.041355, event: started, op: osd_sub_op_reply(client.4312.0:144979 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.041361 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2730 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] sub_op_modify_reply: tid 236188 op  ack_type 4 from 25
2014-06-12 14:46:02.041374 7f9d63bd8700  5 -- op tracker -- , seq: 747479, time: 2014-06-12 14:46:02.041370, event: sub_op_commit_rec, op: osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.041380 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2730 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] repop_all_applied: repop tid 236188 all applied 
2014-06-12 14:46:02.041388 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2730 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] eval_repop repgather(0x24e8de00 173'2731 rep_tid=236188 committed?=0 applied?=1 lock=0 op=osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.041400 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2730 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean]  sending ack on repgather(0x24e8de00 173'2731 rep_tid=236188 committed?=0 applied?=1 lock=0 op=osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) 0x22b52500
2014-06-12 14:46:02.041421 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2730 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] repop_all_committed: repop tid 236188 all committed 
2014-06-12 14:46:02.041428 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2729 active+clean] eval_repop repgather(0x24e8de00 173'2731 rep_tid=236188 committed?=1 applied?=1 lock=0 op=osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.041439 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2729 active+clean] log_op_stats osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4 inb 1294615 outb 0 rlat 0.023271 lat 0.023289
2014-06-12 14:46:02.041451 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2729 active+clean] publish_stats_to_osd 173:3628
2014-06-12 14:46:02.041466 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2729 active+clean]  sending commit on repgather(0x24e8de00 173'2731 rep_tid=236188 committed?=1 applied?=1 lock=0 op=osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) 0x6825680
2014-06-12 14:46:02.041487 7f9d63bd8700  5 -- op tracker -- , seq: 747479, time: 2014-06-12 14:46:02.041483, event: commit_sent, op: osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.041493 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2729 active+clean]  requeue_ops 
2014-06-12 14:46:02.041499 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean]  removing repgather(0x24e8de00 173'2731 rep_tid=236188 committed?=1 applied?=1 op=osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.041513 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean]    q front is repgather(0x24e8de00 173'2731 rep_tid=236188 committed?=1 applied?=1 op=osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.041522 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] remove_repop repgather(0x24e8de00 173'2731 rep_tid=236188 committed?=1 applied?=1 op=osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.041532 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean]  requeue_ops 
2014-06-12 14:46:02.041548 7f9d63bd8700  5 -- op tracker -- , seq: 747479, time: 2014-06-12 14:46:02.041545, event: done, op: osd_op(client.4312.0:144979 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1294336] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.041564 7f9d63bd8700 10 osd.2 173 dequeue_op 0x20e87a00 finish
2014-06-12 14:46:02.041569 7f9d63bd8700  5 -- op tracker -- , seq: 747480, time: 2014-06-12 14:46:02.041567, event: done, op: osd_sub_op_reply(client.4312.0:144979 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.041575 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.041577 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.124568 7f9d5a16f700  5 -- op tracker -- , seq: 747481, time: 2014-06-12 14:46:02.124507, event: header_read, op: osd_sub_op_reply(client.4310.0:142387 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.124583 7f9d5a16f700  5 -- op tracker -- , seq: 747481, time: 2014-06-12 14:46:02.124510, event: throttled, op: osd_sub_op_reply(client.4310.0:142387 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.124591 7f9d5a16f700  5 -- op tracker -- , seq: 747481, time: 2014-06-12 14:46:02.124527, event: all_read, op: osd_sub_op_reply(client.4310.0:142387 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.124599 7f9d5a16f700  5 -- op tracker -- , seq: 747481, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142387 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.124605 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142387 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.124611 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:02.124620 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329eba00 prio 196 cost 0 latency 0.000113 osd_sub_op_reply(client.4310.0:142387 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.124661 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.124671 7f9d653db700 10 osd.2 173 dequeue_op 0x329eba00 prio 196 cost 0 latency 0.000163 osd_sub_op_reply(client.4310.0:142387 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2 pg pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]
2014-06-12 14:46:02.124693 7f9d653db700  5 -- op tracker -- , seq: 747481, time: 2014-06-12 14:46:02.124689, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142387 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.124701 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] handle_message: 0x329eba00
2014-06-12 14:46:02.124717 7f9d653db700  5 -- op tracker -- , seq: 747481, time: 2014-06-12 14:46:02.124714, event: started, op: osd_sub_op_reply(client.4310.0:142387 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.124723 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] sub_op_modify_reply: tid 235992 op  ack_type 4 from 24
2014-06-12 14:46:02.124744 7f9d653db700  5 -- op tracker -- , seq: 746898, time: 2014-06-12 14:46:02.124736, event: sub_op_commit_rec, op: osd_op(client.4310.0:142387 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.124751 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] repop_all_applied: repop tid 235992 all applied 
2014-06-12 14:46:02.124792 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] eval_repop repgather(0x10192fc0 173'2699 rep_tid=235992 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142387 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.124812 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]  sending ack on repgather(0x10192fc0 173'2699 rep_tid=235992 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142387 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) 0x32eeec80
2014-06-12 14:46:02.124843 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] repop_all_committed: repop tid 235992 all committed 
2014-06-12 14:46:02.124854 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2697 active+clean] eval_repop repgather(0x10192fc0 173'2699 rep_tid=235992 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142387 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.124875 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2697 active+clean] log_op_stats osd_op(client.4310.0:142387 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 31.915185 lat 31.915216
2014-06-12 14:46:02.124891 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2697 active+clean] publish_stats_to_osd 173:2882
2014-06-12 14:46:02.124905 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2697 active+clean]  sending commit on repgather(0x10192fc0 173'2699 rep_tid=235992 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142387 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) 0x959aa00
2014-06-12 14:46:02.124934 7f9d653db700  5 -- op tracker -- , seq: 746898, time: 2014-06-12 14:46:02.124929, event: commit_sent, op: osd_op(client.4310.0:142387 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.124942 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2697 active+clean]  requeue_ops 
2014-06-12 14:46:02.124953 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean]  removing repgather(0x10192fc0 173'2699 rep_tid=235992 committed?=1 applied?=1 op=osd_op(client.4310.0:142387 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.124968 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean]    q front is repgather(0x10192fc0 173'2699 rep_tid=235992 committed?=1 applied?=1 op=osd_op(client.4310.0:142387 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.124982 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] remove_repop repgather(0x10192fc0 173'2699 rep_tid=235992 committed?=1 applied?=1 op=osd_op(client.4310.0:142387 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.124995 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean]  requeue_ops 
2014-06-12 14:46:02.125023 7f9d653db700  5 -- op tracker -- , seq: 746898, time: 2014-06-12 14:46:02.125017, event: done, op: osd_op(client.4310.0:142387 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.125041 7f9d653db700 10 osd.2 173 dequeue_op 0x329eba00 finish
2014-06-12 14:46:02.125048 7f9d653db700  5 -- op tracker -- , seq: 747481, time: 2014-06-12 14:46:02.125045, event: done, op: osd_sub_op_reply(client.4310.0:142387 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.125058 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.125061 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.152915 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:02.303835 7f9d5a16f700  5 -- op tracker -- , seq: 747482, time: 2014-06-12 14:46:02.301643, event: header_read, op: osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.303849 7f9d5a16f700  5 -- op tracker -- , seq: 747482, time: 2014-06-12 14:46:02.301646, event: throttled, op: osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.303854 7f9d5a16f700  5 -- op tracker -- , seq: 747482, time: 2014-06-12 14:46:02.303803, event: all_read, op: osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.303859 7f9d5a16f700  5 -- op tracker -- , seq: 747482, time: 0.000000, event: dispatched, op: osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.303862 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:02.303866 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:02.303871 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329eac00 prio 127 cost 1032999 latency 0.002228 osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:02.303894 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.303902 7f9d69be4700 10 osd.2 173 dequeue_op 0x329eac00 prio 127 cost 1032999 latency 0.002259 osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2715 (0'0,173'2715] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2714 active]
2014-06-12 14:46:02.303925 7f9d69be4700  5 -- op tracker -- , seq: 747482, time: 2014-06-12 14:46:02.303921, event: reached_pg, op: osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.303934 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2715 (0'0,173'2715] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2714 active] handle_message: 0x329eac00
2014-06-12 14:46:02.303945 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2715 (0'0,173'2715] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2714 active] sub_op_modify trans e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 v 173'2716 (transaction) 174
2014-06-12 14:46:02.303960 7f9d69be4700  5 -- op tracker -- , seq: 747482, time: 2014-06-12 14:46:02.303956, event: started, op: osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.303972 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2715 (0'0,173'2715] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2714 active] append_log log((0'0,173'2715], crt=0'0) [173'2716 (173'2715) modify   e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 by client.4285.0:143303 2014-06-12 08:42:54.139216]
2014-06-12 14:46:02.303993 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2716 (0'0,173'2716] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2714 active] add_log_entry 173'2716 (173'2715) modify   e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 by client.4285.0:143303 2014-06-12 08:42:54.139216
2014-06-12 14:46:02.304012 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2716 (0'0,173'2716] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2714 active] append_log  adding 1 keys
2014-06-12 14:46:02.304042 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2716, trimmed: 
2014-06-12 14:46:02.304064 7f9d69be4700  5 -- op tracker -- , seq: 747482, time: 2014-06-12 14:46:02.304059, event: started, op: osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.304073 7f9d69be4700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:02.304077 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:02.304078 7f9d69be4700  5 queue_op 0x69583c0 seq 2463 osr(3.21c 0x276d020) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:02.304088 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (1 active)
2014-06-12 14:46:02.304095 7f9d7e96a700  5 _do_op 0x69583c0 seq 2463 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:46:02.304085 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2716 (0'0,173'2716] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2714 active] do_sub_op osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:02.304099 7f9d7e96a700 10 _do_transaction on 0x18566ef0
2014-06-12 14:46:02.304099 7f9d69be4700 10 osd.2 173 dequeue_op 0x329eac00 finish
2014-06-12 14:46:02.304103 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.304105 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.304105 7f9d7e96a700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:02.304354 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:02.304377 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:02.317625 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:02.317676 7f9d7e96a700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:02.317680 7f9d7e96a700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:46:02.317687 7f9d7e96a700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:02.317700 7f9d7e96a700 15 _write 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 1933312~1032192
2014-06-12 14:46:02.317807 7f9d7e96a700 10 lookup_strip_header done  cid 3.21c_head oid e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:46:02.318108 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:02.321665 7f9d7e96a700 10 _generic_write 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 1933312~1032192 = 0
2014-06-12 14:46:02.321807 7f9d7e96a700 15 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:46:02.321834 7f9d7e96a700 10 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 = 0
2014-06-12 14:46:02.321842 7f9d7e96a700 15 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:46:02.321850 7f9d7e96a700 10 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 = 0
2014-06-12 14:46:02.322172 7f9d7e96a700 20 set_header setting 17494 cid 3.21c_head oid e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 parent seq 0
2014-06-12 14:46:02.322254 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:02.322266 7f9d7e96a700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:46:02.322272 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:02.326235 7f9d7e96a700 10 _do_op 0x69583c0 seq 2463 r = 0, finisher 0x31388520 0
2014-06-12 14:46:02.326281 7f9d7715b700  5 -- op tracker -- , seq: 747482, time: 2014-06-12 14:46:02.326269, event: commit_sent, op: osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.326297 7f9d7e96a700 10 _finish_op 0x69583c0 seq 2463 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:02.326294 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2716 (0'0,173'2716] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2714 active] sub_op_modify_commit on op osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:02.326315 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (0 active)
2014-06-12 14:46:02.326320 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.326328 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:02.326402 7f9d7795c700  5 -- op tracker -- , seq: 747482, time: 2014-06-12 14:46:02.326396, event: sub_op_applied, op: osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.326411 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2716 (0'0,173'2716] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active] sub_op_modify_applied on 0x18566e00 op osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:02.326430 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2716 (0'0,173'2716] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active] op_applied on primary on version 173'2716
2014-06-12 14:46:02.326441 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2716 (0'0,173'2716] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active] op_applied on replica on version 173'2716
2014-06-12 14:46:02.326469 7f9d7795c700  5 -- op tracker -- , seq: 747482, time: 2014-06-12 14:46:02.326464, event: done, op: osd_sub_op(client.4285.0:143303 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.366566 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:02.390232 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:02.390354 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:02.465543 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:02.465564 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:46:02.465569 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:46:02.465576 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:46:02.465585 7f9d84175700 10 osd.2 173 send_pg_stats - 5 pgs updated
2014-06-12 14:46:02.465681 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 12.32 >= max 0.5 = no, load too high
2014-06-12 14:46:02.465694 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:02.465697 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92610 < max (604800 seconds)
2014-06-12 14:46:02.465707 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:02.465710 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:02.465711 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:02.465713 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 42; oldest is 36.082506 seconds old
2014-06-12 14:46:02.465821 7f9d84175700  0 log [WRN] : 13 slow requests, 4 included below; oldest blocked for > 36.082506 secs
2014-06-12 14:46:02.465825 7f9d84175700  0 log [WRN] : slow request 30.693786 seconds old, received at 2014-06-12 14:45:31.771927: osd_op(client.4306.0:144044 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:02.465835 7f9d84175700  0 log [WRN] : slow request 30.692459 seconds old, received at 2014-06-12 14:45:31.773254: osd_op(client.4306.0:144045 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:02.465844 7f9d84175700  0 log [WRN] : slow request 30.575773 seconds old, received at 2014-06-12 14:45:31.889940: osd_op(client.4306.0:144046 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 561152~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:02.465847 7f9d84175700  0 log [WRN] : slow request 30.387093 seconds old, received at 2014-06-12 14:45:32.078620: osd_op(client.4306.0:144047 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 1077248~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:02.465873 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:02.634323 7f9d5a16f700  5 -- op tracker -- , seq: 747483, time: 2014-06-12 14:46:02.633363, event: header_read, op: osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.634348 7f9d5a16f700  5 -- op tracker -- , seq: 747483, time: 2014-06-12 14:46:02.633365, event: throttled, op: osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.634356 7f9d5a16f700  5 -- op tracker -- , seq: 747483, time: 2014-06-12 14:46:02.634272, event: all_read, op: osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.634365 7f9d5a16f700  5 -- op tracker -- , seq: 747483, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.634371 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:02.634378 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:02.634387 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329ea400 prio 127 cost 516903 latency 0.001024 osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:02.634429 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.634439 7f9d633d7700 10 osd.2 173 dequeue_op 0x329ea400 prio 127 cost 516903 latency 0.001076 osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4e0( v 173'2631 (0'0,173'2631] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2630 active]
2014-06-12 14:46:02.634460 7f9d633d7700  5 -- op tracker -- , seq: 747483, time: 2014-06-12 14:46:02.634457, event: reached_pg, op: osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.634470 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2631 (0'0,173'2631] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2630 active] handle_message: 0x329ea400
2014-06-12 14:46:02.634479 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2631 (0'0,173'2631] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2630 active] sub_op_modify trans 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 v 173'2632 (transaction) 174
2014-06-12 14:46:02.634490 7f9d633d7700  5 -- op tracker -- , seq: 747483, time: 2014-06-12 14:46:02.634487, event: started, op: osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.634504 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2631 (0'0,173'2631] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2630 active] append_log log((0'0,173'2631], crt=0'0) [173'2632 (173'2631) modify   6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 by client.4306.0:143775 2014-06-12 08:43:01.154680]
2014-06-12 14:46:02.634523 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2632 (0'0,173'2632] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2630 active] add_log_entry 173'2632 (173'2631) modify   6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 by client.4306.0:143775 2014-06-12 08:43:01.154680
2014-06-12 14:46:02.634539 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2632 (0'0,173'2632] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2630 active] append_log  adding 1 keys
2014-06-12 14:46:02.634563 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2632, trimmed: 
2014-06-12 14:46:02.634580 7f9d633d7700  5 -- op tracker -- , seq: 747483, time: 2014-06-12 14:46:02.634577, event: started, op: osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.634588 7f9d633d7700  5 queue_transactions existing osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:02.634592 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:02.634593 7f9d633d7700  5 queue_op 0x6db7860 seq 2432 osr(3.4e0 0x276dbd0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:02.634598 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2632 (0'0,173'2632] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2630 active] do_sub_op osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:02.634607 7f9d633d7700 10 osd.2 173 dequeue_op 0x329ea400 finish
2014-06-12 14:46:02.634609 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.634619 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.634631 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5180900 (1 active)
2014-06-12 14:46:02.634640 7f9d82171700  5 _do_op 0x6db7860 seq 2432 osr(3.4e0 0x276dbd0)/0x276dbd0 start
2014-06-12 14:46:02.634644 7f9d82171700 10 _do_transaction on 0x2e9b35f0
2014-06-12 14:46:02.634650 7f9d82171700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:02.634746 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:02.634770 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:02.634802 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:02.634822 7f9d82171700 15 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:02.634827 7f9d82171700 10 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1 r = 0
2014-06-12 14:46:02.634831 7f9d82171700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:02.634982 7f9d82171700 15 _write 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 802816~516096
2014-06-12 14:46:02.635109 7f9d82171700 10 lookup_strip_header done  cid 3.4e0_head oid 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.635247 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:02.636655 7f9d82171700 10 _generic_write 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 802816~516096 = 0
2014-06-12 14:46:02.636696 7f9d82171700 15 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.636705 7f9d82171700 10 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.636708 7f9d82171700 15 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.636713 7f9d82171700 10 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.636855 7f9d82171700 20 set_header setting 16555 cid 3.4e0_head oid 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:02.636901 7f9d82171700 20 set_header setting 2236 cid meta oid 1fe07435/pglog_3.4e0/0//-1 parent seq 0
2014-06-12 14:46:02.636913 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:02.636918 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:02.639085 7f9d82171700 10 _do_op 0x6db7860 seq 2432 r = 0, finisher 0xa647480 0
2014-06-12 14:46:02.639107 7f9d82171700 10 _finish_op 0x6db7860 seq 2432 osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:02.639115 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5180900 (0 active)
2014-06-12 14:46:02.639126 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.639132 7f9d7715b700  5 -- op tracker -- , seq: 747483, time: 2014-06-12 14:46:02.639119, event: commit_sent, op: osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.639143 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2632 (0'0,173'2632] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2630 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:02.639171 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:02.639230 7f9d7795c700  5 -- op tracker -- , seq: 747483, time: 2014-06-12 14:46:02.639222, event: sub_op_applied, op: osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.639244 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2632 (0'0,173'2632] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active] sub_op_modify_applied on 0x2e9b3500 op osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:02.639262 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2632 (0'0,173'2632] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active] op_applied on primary on version 173'2632
2014-06-12 14:46:02.639274 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2632 (0'0,173'2632] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active] op_applied on replica on version 173'2632
2014-06-12 14:46:02.639297 7f9d7795c700  5 -- op tracker -- , seq: 747483, time: 2014-06-12 14:46:02.639294, event: done, op: osd_sub_op(client.4306.0:143775 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2632 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.693924 7f9d4397a700  5 -- op tracker -- , seq: 747484, time: 2014-06-12 14:46:02.692578, event: header_read, op: osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.693941 7f9d4397a700  5 -- op tracker -- , seq: 747484, time: 2014-06-12 14:46:02.692582, event: throttled, op: osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.693951 7f9d4397a700  5 -- op tracker -- , seq: 747484, time: 2014-06-12 14:46:02.693871, event: all_read, op: osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.693961 7f9d4397a700  5 -- op tracker -- , seq: 747484, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.693970 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:02.694002 7f9d4397a700 15 osd.2 173 enqueue_op 0x329eb200 prio 63 cost 372736 latency 0.001423 osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.694039 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.694047 7f9d643d9700 10 osd.2 173 dequeue_op 0x329eb200 prio 63 cost 372736 latency 0.001468 osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean]
2014-06-12 14:46:02.694070 7f9d643d9700  5 -- op tracker -- , seq: 747484, time: 2014-06-12 14:46:02.694066, event: reached_pg, op: osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.694078 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:02.694092 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] handle_message: 0x329eb200
2014-06-12 14:46:02.694101 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] do_op osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:02.694123 7f9d643d9700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_'
2014-06-12 14:46:02.741816 7f9d643d9700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_' = 0
2014-06-12 14:46:02.741857 7f9d643d9700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset'
2014-06-12 14:46:02.748658 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:02.748710 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:02.763544 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.772847 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.794678 7f9d643d9700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset' = 0
2014-06-12 14:46:02.794702 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] populate_obc_watchers 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.794726 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.794736 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] get_object_context: creating obc from disk: 0xc2de580
2014-06-12 14:46:02.794746 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] get_object_context: 0xc2de580 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 rwstate(none n=0 w=0) oi: 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'770 client.4309.0:37070 wrlock_by=unknown.0.0:0 dirty s 4194304 uv770) ssc: 0x2ef8ed20 snapset: 0=[]:[]+head
2014-06-12 14:46:02.794764 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] find_object_context 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 @head oi=5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'770 client.4309.0:37070 wrlock_by=unknown.0.0:0 dirty s 4194304 uv770)
2014-06-12 14:46:02.794795 7f9d643d9700  5 -- op tracker -- , seq: 747484, time: 2014-06-12 14:46:02.794787, event: started, op: osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.794806 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] execute_ctx 0x5ca1e00
2014-06-12 14:46:02.794824 7f9d643d9700  5 -- op tracker -- , seq: 747484, time: 2014-06-12 14:46:02.794819, event: started, op: osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.794831 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] do_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] ov 173'770 av 173'3002 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:02.794845 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] do_osd_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736]
2014-06-12 14:46:02.794856 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:02.794869 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] do_osd_op  write 0~372736
2014-06-12 14:46:02.794884 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:02.794896 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 snapset=0x2ef8ed58  snapc=0=[]
2014-06-12 14:46:02.794908 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:02.794918 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] finish_ctx 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 0x5ca1e00 op modify  
2014-06-12 14:46:02.794930 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean]  set mtime to 2014-06-12 08:45:05.073065
2014-06-12 14:46:02.794951 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean]  final snapset 0=[]:[]+head in 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.794970 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean]  zeroing write result code 0
2014-06-12 14:46:02.794980 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] calc_trim_to 0'0 -> 79'1
2014-06-12 14:46:02.794990 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] new_repop rep_tid 236189 on osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.795009 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] issue_repop rep_tid 236189 o 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.795035 7f9d643d9700  5 -- op tracker -- , seq: 747484, time: 2014-06-12 14:46:02.795030, event: waiting for subops from 29, op: osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.795061 7f9d643d9700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:02.795089 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3001 (0'0,173'3001] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] append_log log((0'0,173'3001], crt=0'0) [173'3002 (173'770) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141582 2014-06-12 08:45:05.073065]
2014-06-12 14:46:02.795113 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (0'0,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3001 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] add_log_entry 173'3002 (173'770) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141582 2014-06-12 08:45:05.073065
2014-06-12 14:46:02.795134 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (0'0,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3001 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] append_log  adding 1 keys
2014-06-12 14:46:02.795147 7f9d643d9700 10 trim log((0'0,173'3002], crt=0'0) to 79'1
2014-06-12 14:46:02.795174 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3002, trimmed: 79'1
2014-06-12 14:46:02.795198 7f9d643d9700  5 queue_transactions existing osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.795203 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:02.795204 7f9d643d9700  5 queue_op 0x13c50120 seq 2768 osr(3.602 0x276dd30) 375064 bytes   (queue has 1 ops and 375064 bytes)
2014-06-12 14:46:02.795217 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541c6c0 (1 active)
2014-06-12 14:46:02.795225 7f9d79960700  5 _do_op 0x13c50120 seq 2768 osr(3.602 0x276dd30)/0x276dd30 start
2014-06-12 14:46:02.795213 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3001 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] eval_repop repgather(0x224e180 173'3002 rep_tid=236189 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.795229 7f9d79960700 10 _do_transaction on 0x1c833380
2014-06-12 14:46:02.795232 7f9d643d9700 10 osd.2 173 dequeue_op 0x329eb200 finish
2014-06-12 14:46:02.795238 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.795239 7f9d79960700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.795242 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.795340 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.795369 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:02.795415 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:02.795439 7f9d79960700 15 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.795447 7f9d79960700 10 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1 r = 0
2014-06-12 14:46:02.795454 7f9d79960700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.795557 7f9d79960700 15 _write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 0~372736
2014-06-12 14:46:02.795658 7f9d79960700 10 lookup_strip_header done  cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.795771 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:02.796658 7f9d79960700 10 _generic_write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 0~372736 = 0
2014-06-12 14:46:02.796689 7f9d79960700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.796700 7f9d79960700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.796703 7f9d79960700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.796708 7f9d79960700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.796825 7f9d79960700 20 set_header setting 17100 cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:02.796860 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:02.796869 7f9d79960700 20 set_header setting 2102 cid meta oid 1fe11eba/pglog_3.602/0//-1 parent seq 0
2014-06-12 14:46:02.796876 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:02.798112 7f9d79960700 10 _do_op 0x13c50120 seq 2768 r = 0, finisher 0x90ee9e0 0x323f040
2014-06-12 14:46:02.798130 7f9d79960700 10 _finish_op 0x13c50120 seq 2768 osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.798139 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541c6c0 (0 active)
2014-06-12 14:46:02.798147 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.798134 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3001 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] op_commit: 236189
2014-06-12 14:46:02.798166 7f9d7715b700  5 -- op tracker -- , seq: 747484, time: 2014-06-12 14:46:02.798156, event: op_commit, op: osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.798191 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3001 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] op_applied: 236189
2014-06-12 14:46:02.798216 7f9d7795c700  5 -- op tracker -- , seq: 747484, time: 2014-06-12 14:46:02.798209, event: op_applied, op: osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.798230 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3001 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] op_applied on primary on version 173'3002
2014-06-12 14:46:02.799994 7f9d5501e700  5 -- op tracker -- , seq: 747485, time: 2014-06-12 14:46:02.799954, event: header_read, op: osd_sub_op_reply(client.4309.0:141582 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.800006 7f9d5501e700  5 -- op tracker -- , seq: 747485, time: 2014-06-12 14:46:02.799956, event: throttled, op: osd_sub_op_reply(client.4309.0:141582 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.800014 7f9d5501e700  5 -- op tracker -- , seq: 747485, time: 2014-06-12 14:46:02.799965, event: all_read, op: osd_sub_op_reply(client.4309.0:141582 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.800023 7f9d5501e700  5 -- op tracker -- , seq: 747485, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141582 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.800028 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141582 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.800034 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.800042 7f9d5501e700 15 osd.2 173 enqueue_op 0xe20bc00 prio 196 cost 0 latency 0.000088 osd_sub_op_reply(client.4309.0:141582 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.800070 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.800077 7f9d623d5700 10 osd.2 173 dequeue_op 0xe20bc00 prio 196 cost 0 latency 0.000123 osd_sub_op_reply(client.4309.0:141582 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 pg pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3001 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean]
2014-06-12 14:46:02.800091 7f9d623d5700  5 -- op tracker -- , seq: 747485, time: 2014-06-12 14:46:02.800088, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141582 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.800096 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3001 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] handle_message: 0xe20bc00
2014-06-12 14:46:02.800106 7f9d623d5700  5 -- op tracker -- , seq: 747485, time: 2014-06-12 14:46:02.800104, event: started, op: osd_sub_op_reply(client.4309.0:141582 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.800109 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3001 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] sub_op_modify_reply: tid 236189 op  ack_type 4 from 29
2014-06-12 14:46:02.800121 7f9d623d5700  5 -- op tracker -- , seq: 747484, time: 2014-06-12 14:46:02.800117, event: sub_op_commit_rec, op: osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.800126 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3001 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] repop_all_applied: repop tid 236189 all applied 
2014-06-12 14:46:02.800132 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3001 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] eval_repop repgather(0x224e180 173'3002 rep_tid=236189 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.800142 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3001 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean]  sending ack on repgather(0x224e180 173'3002 rep_tid=236189 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x2b2fcf00
2014-06-12 14:46:02.800162 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3001 crt=0'0 lcod 173'3000 mlcod 173'3000 active+clean] repop_all_committed: repop tid 236189 all committed 
2014-06-12 14:46:02.800168 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3000 active+clean] eval_repop repgather(0x224e180 173'3002 rep_tid=236189 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.800178 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3000 active+clean] log_op_stats osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 inb 373015 outb 0 rlat 0.107583 lat 0.107599
2014-06-12 14:46:02.800189 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3000 active+clean] publish_stats_to_osd 173:3962
2014-06-12 14:46:02.800197 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3000 active+clean]  sending commit on repgather(0x224e180 173'3002 rep_tid=236189 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x45dec80
2014-06-12 14:46:02.800215 7f9d623d5700  5 -- op tracker -- , seq: 747484, time: 2014-06-12 14:46:02.800212, event: commit_sent, op: osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.800220 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3000 active+clean]  requeue_ops 
2014-06-12 14:46:02.800227 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean]  removing repgather(0x224e180 173'3002 rep_tid=236189 committed?=1 applied?=1 op=osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.800235 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean]    q front is repgather(0x224e180 173'3002 rep_tid=236189 committed?=1 applied?=1 op=osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.800244 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] remove_repop repgather(0x224e180 173'3002 rep_tid=236189 committed?=1 applied?=1 op=osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.800252 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean]  requeue_ops 
2014-06-12 14:46:02.800269 7f9d623d5700  5 -- op tracker -- , seq: 747484, time: 2014-06-12 14:46:02.800266, event: done, op: osd_op(client.4309.0:141582 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~372736] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.800281 7f9d623d5700 10 osd.2 173 dequeue_op 0xe20bc00 finish
2014-06-12 14:46:02.800285 7f9d623d5700  5 -- op tracker -- , seq: 747485, time: 2014-06-12 14:46:02.800283, event: done, op: osd_sub_op_reply(client.4309.0:141582 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.800294 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.800296 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.805565 7f9d4397a700  5 -- op tracker -- , seq: 747486, time: 2014-06-12 14:46:02.803420, event: header_read, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.805578 7f9d4397a700  5 -- op tracker -- , seq: 747486, time: 2014-06-12 14:46:02.803423, event: throttled, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.805588 7f9d4397a700  5 -- op tracker -- , seq: 747486, time: 2014-06-12 14:46:02.805514, event: all_read, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.805597 7f9d4397a700  5 -- op tracker -- , seq: 747486, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.805605 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:02.805645 7f9d4397a700 15 osd.2 173 enqueue_op 0x329ea800 prio 63 cost 516096 latency 0.002224 osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.805680 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.805688 7f9d68be2700 10 osd.2 173 dequeue_op 0x329ea800 prio 63 cost 516096 latency 0.002267 osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean]
2014-06-12 14:46:02.805711 7f9d68be2700  5 -- op tracker -- , seq: 747486, time: 2014-06-12 14:46:02.805706, event: reached_pg, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.805724 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:02.805738 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] handle_message: 0x329ea800
2014-06-12 14:46:02.805747 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] do_op osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:02.805770 7f9d68be2700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_'
2014-06-12 14:46:02.834623 7f9d68be2700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_' = 0
2014-06-12 14:46:02.834650 7f9d68be2700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset'
2014-06-12 14:46:02.834765 7f9d68be2700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset' = 0
2014-06-12 14:46:02.834776 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] populate_obc_watchers 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.834793 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.834804 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] get_object_context: creating obc from disk: 0x16962580
2014-06-12 14:46:02.834814 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] get_object_context: 0x16962580 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 rwstate(none n=0 w=0) oi: 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3002 client.4309.0:141582 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3002) ssc: 0x9344ee0 snapset: 0=[]:[]+head
2014-06-12 14:46:02.834831 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] find_object_context 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 @head oi=5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3002 client.4309.0:141582 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3002)
2014-06-12 14:46:02.834863 7f9d68be2700  5 -- op tracker -- , seq: 747486, time: 2014-06-12 14:46:02.834855, event: started, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.834873 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] execute_ctx 0x580be00
2014-06-12 14:46:02.834892 7f9d68be2700  5 -- op tracker -- , seq: 747486, time: 2014-06-12 14:46:02.834887, event: started, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.834901 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] do_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] ov 173'3002 av 173'3003 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:02.834917 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] do_osd_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096]
2014-06-12 14:46:02.834931 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:02.834945 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] do_osd_op  write 372736~516096
2014-06-12 14:46:02.834962 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:02.834975 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 snapset=0x9344f18  snapc=0=[]
2014-06-12 14:46:02.834990 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:02.835004 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] finish_ctx 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 0x580be00 op modify  
2014-06-12 14:46:02.835017 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean]  set mtime to 2014-06-12 08:45:05.182797
2014-06-12 14:46:02.835039 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean]  final snapset 0=[]:[]+head in 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.835059 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean]  zeroing write result code 0
2014-06-12 14:46:02.835071 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] calc_trim_to 79'1 -> 79'2
2014-06-12 14:46:02.835082 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] new_repop rep_tid 236190 on osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.835100 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] issue_repop rep_tid 236190 o 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.835130 7f9d68be2700  5 -- op tracker -- , seq: 747486, time: 2014-06-12 14:46:02.835125, event: waiting for subops from 29, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.835151 7f9d68be2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:02.835173 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3002 (79'1,173'3002] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] append_log log((79'1,173'3002], crt=0'0) [173'3003 (173'3002) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141583 2014-06-12 08:45:05.182797]
2014-06-12 14:46:02.835193 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'1,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] add_log_entry 173'3003 (173'3002) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141583 2014-06-12 08:45:05.182797
2014-06-12 14:46:02.835214 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'1,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] append_log  adding 1 keys
2014-06-12 14:46:02.835229 7f9d68be2700 10 trim log((79'1,173'3003], crt=0'0) to 79'2
2014-06-12 14:46:02.835255 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3003, trimmed: 79'2
2014-06-12 14:46:02.835281 7f9d68be2700  5 queue_transactions existing osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.835286 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:02.835288 7f9d68be2700  5 queue_op 0x558e5a0 seq 2769 osr(3.602 0x276dd30) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:02.835296 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] eval_repop repgather(0x5390300 173'3003 rep_tid=236190 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.835320 7f9d68be2700 10 osd.2 173 dequeue_op 0x329ea800 finish
2014-06-12 14:46:02.835326 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.835332 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.835338 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541c6c0 (1 active)
2014-06-12 14:46:02.835344 7f9d8116f700  5 _do_op 0x558e5a0 seq 2769 osr(3.602 0x276dd30)/0x276dd30 start
2014-06-12 14:46:02.835348 7f9d8116f700 10 _do_transaction on 0x228e1b00
2014-06-12 14:46:02.835354 7f9d8116f700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.835425 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.835452 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:02.835494 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:02.835518 7f9d8116f700 15 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.835527 7f9d8116f700 10 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1 r = 0
2014-06-12 14:46:02.835534 7f9d8116f700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.835657 7f9d8116f700 15 _write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 372736~516096
2014-06-12 14:46:02.835752 7f9d8116f700 10 lookup_strip_header done  cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.835877 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:02.837088 7f9d8116f700 10 _generic_write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 372736~516096 = 0
2014-06-12 14:46:02.837137 7f9d8116f700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.837151 7f9d8116f700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.837162 7f9d8116f700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.837172 7f9d8116f700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.837396 7f9d8116f700 20 set_header setting 17100 cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:02.837468 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:02.837484 7f9d8116f700 20 set_header setting 2102 cid meta oid 1fe11eba/pglog_3.602/0//-1 parent seq 0
2014-06-12 14:46:02.837498 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:02.839706 7f9d8116f700 10 _do_op 0x558e5a0 seq 2769 r = 0, finisher 0x1e4fd4e0 0x323f000
2014-06-12 14:46:02.839730 7f9d8116f700 10 _finish_op 0x558e5a0 seq 2769 osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.839739 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541c6c0 (0 active)
2014-06-12 14:46:02.839743 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.839736 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] op_commit: 236190
2014-06-12 14:46:02.839772 7f9d7715b700  5 -- op tracker -- , seq: 747486, time: 2014-06-12 14:46:02.839762, event: op_commit, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.839794 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] op_applied: 236190
2014-06-12 14:46:02.839822 7f9d7795c700  5 -- op tracker -- , seq: 747486, time: 2014-06-12 14:46:02.839815, event: op_applied, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.839836 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] op_applied on primary on version 173'3003
2014-06-12 14:46:02.842984 7f9d5501e700  5 -- op tracker -- , seq: 747487, time: 2014-06-12 14:46:02.842938, event: header_read, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.842994 7f9d5501e700  5 -- op tracker -- , seq: 747487, time: 2014-06-12 14:46:02.842941, event: throttled, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.843003 7f9d5501e700  5 -- op tracker -- , seq: 747487, time: 2014-06-12 14:46:02.842954, event: all_read, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.843007 7f9d5501e700  5 -- op tracker -- , seq: 747487, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.843012 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:02.843016 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.843022 7f9d5501e700 15 osd.2 173 enqueue_op 0xe20a400 prio 196 cost 0 latency 0.000084 osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2
2014-06-12 14:46:02.843049 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.843065 7f9d5501e700  5 -- op tracker -- , seq: 747488, time: 2014-06-12 14:46:02.843039, event: header_read, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.843058 7f9d61bd4700 10 osd.2 173 dequeue_op 0xe20a400 prio 196 cost 0 latency 0.000119 osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2 pg pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean]
2014-06-12 14:46:02.843073 7f9d5501e700  5 -- op tracker -- , seq: 747488, time: 2014-06-12 14:46:02.843040, event: throttled, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.843083 7f9d5501e700  5 -- op tracker -- , seq: 747488, time: 2014-06-12 14:46:02.843047, event: all_read, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.843088 7f9d5501e700  5 -- op tracker -- , seq: 747488, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.843092 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.843095 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.843098 7f9d5501e700 15 osd.2 173 enqueue_op 0xe20bd00 prio 196 cost 0 latency 0.000059 osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.843096 7f9d61bd4700  5 -- op tracker -- , seq: 747487, time: 2014-06-12 14:46:02.843076, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.843115 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:02.843106 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] handle_message: 0xe20a400
2014-06-12 14:46:02.843123 7f9d61bd4700  5 -- op tracker -- , seq: 747487, time: 2014-06-12 14:46:02.843120, event: started, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.843129 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] sub_op_modify_reply: tid 236190 op  ack_type 1 from 29
2014-06-12 14:46:02.843149 7f9d61bd4700  5 -- op tracker -- , seq: 747486, time: 2014-06-12 14:46:02.843142, event: sub_op_applied_rec, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.843157 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] repop_all_applied: repop tid 236190 all applied 
2014-06-12 14:46:02.843173 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] eval_repop repgather(0x5390300 173'3003 rep_tid=236190 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.843190 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean]  sending ack on repgather(0x5390300 173'3003 rep_tid=236190 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x1990d900
2014-06-12 14:46:02.843224 7f9d61bd4700 10 osd.2 173 dequeue_op 0xe20a400 finish
2014-06-12 14:46:02.843233 7f9d61bd4700  5 -- op tracker -- , seq: 747487, time: 2014-06-12 14:46:02.843230, event: done, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.843232 7f9d673df700 10 osd.2 173 dequeue_op 0xe20bd00 prio 196 cost 0 latency 0.000193 osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 pg pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean]
2014-06-12 14:46:02.843255 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:02.843258 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.843258 7f9d673df700  5 -- op tracker -- , seq: 747488, time: 2014-06-12 14:46:02.843247, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.843264 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] handle_message: 0xe20bd00
2014-06-12 14:46:02.843274 7f9d673df700  5 -- op tracker -- , seq: 747488, time: 2014-06-12 14:46:02.843272, event: started, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.843278 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] sub_op_modify_reply: tid 236190 op  ack_type 4 from 29
2014-06-12 14:46:02.843289 7f9d673df700  5 -- op tracker -- , seq: 747486, time: 2014-06-12 14:46:02.843285, event: sub_op_commit_rec, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.843293 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3002 crt=0'0 lcod 173'3001 mlcod 173'3001 active+clean] repop_all_committed: repop tid 236190 all committed 
2014-06-12 14:46:02.843299 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3001 active+clean] eval_repop repgather(0x5390300 173'3003 rep_tid=236190 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.843310 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3001 active+clean] log_op_stats osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.039802 lat 0.039889
2014-06-12 14:46:02.843324 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3001 active+clean] publish_stats_to_osd 173:3963
2014-06-12 14:46:02.843333 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3001 active+clean]  sending commit on repgather(0x5390300 173'3003 rep_tid=236190 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0xfa09b80
2014-06-12 14:46:02.843356 7f9d673df700  5 -- op tracker -- , seq: 747486, time: 2014-06-12 14:46:02.843351, event: commit_sent, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.843366 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3001 active+clean]  requeue_ops 
2014-06-12 14:46:02.843378 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean]  removing repgather(0x5390300 173'3003 rep_tid=236190 committed?=1 applied?=1 op=osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.843392 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean]    q front is repgather(0x5390300 173'3003 rep_tid=236190 committed?=1 applied?=1 op=osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.843401 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] remove_repop repgather(0x5390300 173'3003 rep_tid=236190 committed?=1 applied?=1 op=osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.843410 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean]  requeue_ops 
2014-06-12 14:46:02.843428 7f9d673df700  5 -- op tracker -- , seq: 747486, time: 2014-06-12 14:46:02.843425, event: done, op: osd_op(client.4309.0:141583 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 372736~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.843441 7f9d673df700 10 osd.2 173 dequeue_op 0xe20bd00 finish
2014-06-12 14:46:02.843447 7f9d673df700  5 -- op tracker -- , seq: 747488, time: 2014-06-12 14:46:02.843445, event: done, op: osd_sub_op_reply(client.4309.0:141583 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.843453 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.843455 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.846574 7f9d4397a700  5 -- op tracker -- , seq: 747489, time: 2014-06-12 14:46:02.845537, event: header_read, op: osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.846584 7f9d4397a700  5 -- op tracker -- , seq: 747489, time: 2014-06-12 14:46:02.845538, event: throttled, op: osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.846613 7f9d4397a700  5 -- op tracker -- , seq: 747489, time: 2014-06-12 14:46:02.846528, event: all_read, op: osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.846620 7f9d4397a700  5 -- op tracker -- , seq: 747489, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.846626 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:02.846649 7f9d4397a700 15 osd.2 173 enqueue_op 0x124e2900 prio 63 cost 516096 latency 0.001112 osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.846673 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.846681 7f9d663dd700 10 osd.2 173 dequeue_op 0x124e2900 prio 63 cost 516096 latency 0.001143 osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean]
2014-06-12 14:46:02.846698 7f9d663dd700  5 -- op tracker -- , seq: 747489, time: 2014-06-12 14:46:02.846695, event: reached_pg, op: osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.846704 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:02.846713 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] handle_message: 0x124e2900
2014-06-12 14:46:02.846720 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] do_op osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:02.846738 7f9d663dd700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_'
2014-06-12 14:46:02.846874 7f9d663dd700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_' = 0
2014-06-12 14:46:02.846887 7f9d663dd700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset'
2014-06-12 14:46:02.846936 7f9d663dd700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset' = 0
2014-06-12 14:46:02.846945 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] populate_obc_watchers 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.846953 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.846959 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] get_object_context: creating obc from disk: 0x2cab8b00
2014-06-12 14:46:02.846965 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] get_object_context: 0x2cab8b00 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 rwstate(none n=0 w=0) oi: 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3003 client.4309.0:141583 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3003) ssc: 0x9345880 snapset: 0=[]:[]+head
2014-06-12 14:46:02.846976 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] find_object_context 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 @head oi=5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3003 client.4309.0:141583 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3003)
2014-06-12 14:46:02.846995 7f9d663dd700  5 -- op tracker -- , seq: 747489, time: 2014-06-12 14:46:02.846991, event: started, op: osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.847002 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] execute_ctx 0x5ca0600
2014-06-12 14:46:02.847013 7f9d663dd700  5 -- op tracker -- , seq: 747489, time: 2014-06-12 14:46:02.847010, event: started, op: osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.847018 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] do_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] ov 173'3003 av 173'3004 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:02.847029 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] do_osd_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096]
2014-06-12 14:46:02.847036 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:02.847046 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] do_osd_op  write 888832~516096
2014-06-12 14:46:02.847058 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:02.847066 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 snapset=0x93458b8  snapc=0=[]
2014-06-12 14:46:02.847074 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:02.847082 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] finish_ctx 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 0x5ca0600 op modify  
2014-06-12 14:46:02.847090 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean]  set mtime to 2014-06-12 08:45:05.225807
2014-06-12 14:46:02.847104 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean]  final snapset 0=[]:[]+head in 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.847117 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean]  zeroing write result code 0
2014-06-12 14:46:02.847124 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] calc_trim_to 79'2 -> 79'3
2014-06-12 14:46:02.847130 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] new_repop rep_tid 236191 on osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.847141 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] issue_repop rep_tid 236191 o 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.847159 7f9d663dd700  5 -- op tracker -- , seq: 747489, time: 2014-06-12 14:46:02.847156, event: waiting for subops from 29, op: osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.847177 7f9d663dd700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:02.847194 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3003 (79'2,173'3003] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] append_log log((79'2,173'3003], crt=0'0) [173'3004 (173'3003) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141584 2014-06-12 08:45:05.225807]
2014-06-12 14:46:02.847210 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'2,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3003 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] add_log_entry 173'3004 (173'3003) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141584 2014-06-12 08:45:05.225807
2014-06-12 14:46:02.847224 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'2,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3003 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] append_log  adding 1 keys
2014-06-12 14:46:02.847234 7f9d663dd700 10 trim log((79'2,173'3004], crt=0'0) to 79'3
2014-06-12 14:46:02.847252 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3004, trimmed: 79'3
2014-06-12 14:46:02.847269 7f9d663dd700  5 queue_transactions existing osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.847274 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:02.847275 7f9d663dd700  5 queue_op 0x7c04de0 seq 2770 osr(3.602 0x276dd30) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:02.847284 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541c6c0 (1 active)
2014-06-12 14:46:02.847290 7f9d7d968700  5 _do_op 0x7c04de0 seq 2770 osr(3.602 0x276dd30)/0x276dd30 start
2014-06-12 14:46:02.847294 7f9d7d968700 10 _do_transaction on 0x52f5d40
2014-06-12 14:46:02.847280 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3003 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] eval_repop repgather(0x15e57bc0 173'3004 rep_tid=236191 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.847297 7f9d663dd700 10 osd.2 173 dequeue_op 0x124e2900 finish
2014-06-12 14:46:02.847300 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.847300 7f9d7d968700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.847303 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.847367 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.847456 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:02.847491 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:02.847511 7f9d7d968700 15 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.847517 7f9d7d968700 10 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1 r = 0
2014-06-12 14:46:02.847521 7f9d7d968700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.847534 7f9d7d968700 15 _write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 888832~516096
2014-06-12 14:46:02.847612 7f9d7d968700 10 lookup_strip_header done  cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.847806 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:02.849075 7f9d7d968700 10 _generic_write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 888832~516096 = 0
2014-06-12 14:46:02.849122 7f9d7d968700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.849133 7f9d7d968700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.849137 7f9d7d968700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.849142 7f9d7d968700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.849283 7f9d7d968700 20 set_header setting 17100 cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:02.849337 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:02.849347 7f9d7d968700 20 set_header setting 2102 cid meta oid 1fe11eba/pglog_3.602/0//-1 parent seq 0
2014-06-12 14:46:02.849354 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:02.851132 7f9d7d968700 10 _do_op 0x7c04de0 seq 2770 r = 0, finisher 0x7758100 0x2869f400
2014-06-12 14:46:02.851150 7f9d7d968700 10 _finish_op 0x7c04de0 seq 2770 osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.851157 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541c6c0 (0 active)
2014-06-12 14:46:02.851166 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.851156 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3003 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] op_commit: 236191
2014-06-12 14:46:02.851192 7f9d7715b700  5 -- op tracker -- , seq: 747489, time: 2014-06-12 14:46:02.851181, event: op_commit, op: osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.851212 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3003 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] op_applied: 236191
2014-06-12 14:46:02.851231 7f9d7795c700  5 -- op tracker -- , seq: 747489, time: 2014-06-12 14:46:02.851227, event: op_applied, op: osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.851239 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3003 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] op_applied on primary on version 173'3004
2014-06-12 14:46:02.860887 7f9d5501e700  5 -- op tracker -- , seq: 747490, time: 2014-06-12 14:46:02.860835, event: header_read, op: osd_sub_op_reply(client.4309.0:141584 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.860908 7f9d5501e700  5 -- op tracker -- , seq: 747490, time: 2014-06-12 14:46:02.860838, event: throttled, op: osd_sub_op_reply(client.4309.0:141584 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.860916 7f9d5501e700  5 -- op tracker -- , seq: 747490, time: 2014-06-12 14:46:02.860849, event: all_read, op: osd_sub_op_reply(client.4309.0:141584 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.860935 7f9d5501e700  5 -- op tracker -- , seq: 747490, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141584 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.860941 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141584 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.860948 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.860957 7f9d5501e700 15 osd.2 173 enqueue_op 0x6f4b400 prio 196 cost 0 latency 0.000121 osd_sub_op_reply(client.4309.0:141584 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.860992 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.861003 7f9d67be0700 10 osd.2 173 dequeue_op 0x6f4b400 prio 196 cost 0 latency 0.000168 osd_sub_op_reply(client.4309.0:141584 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 pg pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3003 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean]
2014-06-12 14:46:02.861026 7f9d67be0700  5 -- op tracker -- , seq: 747490, time: 2014-06-12 14:46:02.861022, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141584 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.861035 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3003 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] handle_message: 0x6f4b400
2014-06-12 14:46:02.861053 7f9d67be0700  5 -- op tracker -- , seq: 747490, time: 2014-06-12 14:46:02.861050, event: started, op: osd_sub_op_reply(client.4309.0:141584 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.861060 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3003 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] sub_op_modify_reply: tid 236191 op  ack_type 4 from 29
2014-06-12 14:46:02.861085 7f9d67be0700  5 -- op tracker -- , seq: 747489, time: 2014-06-12 14:46:02.861076, event: sub_op_commit_rec, op: osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.861094 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3003 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] repop_all_applied: repop tid 236191 all applied 
2014-06-12 14:46:02.861108 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3003 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] eval_repop repgather(0x15e57bc0 173'3004 rep_tid=236191 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.861127 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3003 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean]  sending ack on repgather(0x15e57bc0 173'3004 rep_tid=236191 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x38136280
2014-06-12 14:46:02.861160 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3003 crt=0'0 lcod 173'3002 mlcod 173'3002 active+clean] repop_all_committed: repop tid 236191 all committed 
2014-06-12 14:46:02.861169 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3002 active+clean] eval_repop repgather(0x15e57bc0 173'3004 rep_tid=236191 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.861188 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3002 active+clean] log_op_stats osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.015622 lat 0.015650
2014-06-12 14:46:02.861209 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3002 active+clean] publish_stats_to_osd 173:3964
2014-06-12 14:46:02.861223 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3002 active+clean]  sending commit on repgather(0x15e57bc0 173'3004 rep_tid=236191 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x33271680
2014-06-12 14:46:02.861257 7f9d67be0700  5 -- op tracker -- , seq: 747489, time: 2014-06-12 14:46:02.861251, event: commit_sent, op: osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.861265 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3002 active+clean]  requeue_ops 
2014-06-12 14:46:02.861276 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean]  removing repgather(0x15e57bc0 173'3004 rep_tid=236191 committed?=1 applied?=1 op=osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.861293 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean]    q front is repgather(0x15e57bc0 173'3004 rep_tid=236191 committed?=1 applied?=1 op=osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.861310 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] remove_repop repgather(0x15e57bc0 173'3004 rep_tid=236191 committed?=1 applied?=1 op=osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.861325 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean]  requeue_ops 
2014-06-12 14:46:02.861352 7f9d67be0700  5 -- op tracker -- , seq: 747489, time: 2014-06-12 14:46:02.861348, event: done, op: osd_op(client.4309.0:141584 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 888832~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.861370 7f9d67be0700 10 osd.2 173 dequeue_op 0x6f4b400 finish
2014-06-12 14:46:02.861379 7f9d67be0700  5 -- op tracker -- , seq: 747490, time: 2014-06-12 14:46:02.861375, event: done, op: osd_sub_op_reply(client.4309.0:141584 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.861390 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.861393 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.864523 7f9d4397a700  5 -- op tracker -- , seq: 747491, time: 2014-06-12 14:46:02.863528, event: header_read, op: osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.864540 7f9d4397a700  5 -- op tracker -- , seq: 747491, time: 2014-06-12 14:46:02.863530, event: throttled, op: osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.864550 7f9d4397a700  5 -- op tracker -- , seq: 747491, time: 2014-06-12 14:46:02.864493, event: all_read, op: osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.864564 7f9d4397a700  5 -- op tracker -- , seq: 747491, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.864573 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:02.864600 7f9d4397a700 15 osd.2 173 enqueue_op 0x68a7400 prio 63 cost 516096 latency 0.001072 osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.864631 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.864639 7f9d693e3700 10 osd.2 173 dequeue_op 0x68a7400 prio 63 cost 516096 latency 0.001111 osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean]
2014-06-12 14:46:02.864664 7f9d693e3700  5 -- op tracker -- , seq: 747491, time: 2014-06-12 14:46:02.864658, event: reached_pg, op: osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.864672 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:02.864687 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] handle_message: 0x68a7400
2014-06-12 14:46:02.864702 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] do_op osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:02.864729 7f9d693e3700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_'
2014-06-12 14:46:02.864872 7f9d693e3700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_' = 0
2014-06-12 14:46:02.864890 7f9d693e3700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset'
2014-06-12 14:46:02.864951 7f9d693e3700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset' = 0
2014-06-12 14:46:02.864962 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] populate_obc_watchers 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.864975 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.864986 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] get_object_context: creating obc from disk: 0x15574840
2014-06-12 14:46:02.864996 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] get_object_context: 0x15574840 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 rwstate(none n=0 w=0) oi: 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3004 client.4309.0:141584 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3004) ssc: 0x679b0a0 snapset: 0=[]:[]+head
2014-06-12 14:46:02.865014 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] find_object_context 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 @head oi=5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3004 client.4309.0:141584 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3004)
2014-06-12 14:46:02.865043 7f9d693e3700  5 -- op tracker -- , seq: 747491, time: 2014-06-12 14:46:02.865037, event: started, op: osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.865052 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] execute_ctx 0xd21c400
2014-06-12 14:46:02.865070 7f9d693e3700  5 -- op tracker -- , seq: 747491, time: 2014-06-12 14:46:02.865065, event: started, op: osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.865077 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] do_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] ov 173'3004 av 173'3005 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:02.865093 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] do_osd_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096]
2014-06-12 14:46:02.865106 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:02.865121 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] do_osd_op  write 1404928~516096
2014-06-12 14:46:02.865138 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:02.865151 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 snapset=0x679b0d8  snapc=0=[]
2014-06-12 14:46:02.865171 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:02.865184 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] finish_ctx 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 0xd21c400 op modify  
2014-06-12 14:46:02.865200 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean]  set mtime to 2014-06-12 08:45:05.243742
2014-06-12 14:46:02.865222 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean]  final snapset 0=[]:[]+head in 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.865242 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean]  zeroing write result code 0
2014-06-12 14:46:02.865254 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] calc_trim_to 79'3 -> 79'4
2014-06-12 14:46:02.865264 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] new_repop rep_tid 236192 on osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.865280 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] issue_repop rep_tid 236192 o 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.865308 7f9d693e3700  5 -- op tracker -- , seq: 747491, time: 2014-06-12 14:46:02.865303, event: waiting for subops from 29, op: osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.865328 7f9d693e3700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:02.865349 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3004 (79'3,173'3004] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] append_log log((79'3,173'3004], crt=0'0) [173'3005 (173'3004) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141585 2014-06-12 08:45:05.243742]
2014-06-12 14:46:02.865375 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'3,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3004 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] add_log_entry 173'3005 (173'3004) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141585 2014-06-12 08:45:05.243742
2014-06-12 14:46:02.865393 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'3,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3004 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] append_log  adding 1 keys
2014-06-12 14:46:02.865405 7f9d693e3700 10 trim log((79'3,173'3005], crt=0'0) to 79'4
2014-06-12 14:46:02.865429 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3005, trimmed: 79'4
2014-06-12 14:46:02.865450 7f9d693e3700  5 queue_transactions existing osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.865453 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:02.865455 7f9d693e3700  5 queue_op 0x50a1b00 seq 2771 osr(3.602 0x276dd30) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:02.865461 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3004 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] eval_repop repgather(0x1a4449c0 173'3005 rep_tid=236192 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.865477 7f9d693e3700 10 osd.2 173 dequeue_op 0x68a7400 finish
2014-06-12 14:46:02.865480 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.865482 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.865489 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541c6c0 (1 active)
2014-06-12 14:46:02.865493 7f9d8016d700  5 _do_op 0x50a1b00 seq 2771 osr(3.602 0x276dd30)/0x276dd30 start
2014-06-12 14:46:02.865497 7f9d8016d700 10 _do_transaction on 0xc9b8000
2014-06-12 14:46:02.865501 7f9d8016d700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.865565 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.865581 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:02.865622 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:02.865638 7f9d8016d700 15 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.865645 7f9d8016d700 10 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1 r = 0
2014-06-12 14:46:02.865651 7f9d8016d700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.865664 7f9d8016d700 15 _write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 1404928~516096
2014-06-12 14:46:02.865785 7f9d8016d700 10 lookup_strip_header done  cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.865948 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:02.867288 7f9d8016d700 10 _generic_write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 1404928~516096 = 0
2014-06-12 14:46:02.867333 7f9d8016d700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.867347 7f9d8016d700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.867354 7f9d8016d700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.867362 7f9d8016d700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.867510 7f9d8016d700 20 set_header setting 17100 cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:02.867566 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:02.867589 7f9d8016d700 20 set_header setting 2102 cid meta oid 1fe11eba/pglog_3.602/0//-1 parent seq 0
2014-06-12 14:46:02.867602 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:02.869612 7f9d8016d700 10 _do_op 0x50a1b00 seq 2771 r = 0, finisher 0x90f6de0 0x35046a40
2014-06-12 14:46:02.869636 7f9d8016d700 10 _finish_op 0x50a1b00 seq 2771 osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.869645 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541c6c0 (0 active)
2014-06-12 14:46:02.869652 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.869644 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3004 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] op_commit: 236192
2014-06-12 14:46:02.869677 7f9d7715b700  5 -- op tracker -- , seq: 747491, time: 2014-06-12 14:46:02.869667, event: op_commit, op: osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.869695 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3004 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] op_applied: 236192
2014-06-12 14:46:02.869720 7f9d7795c700  5 -- op tracker -- , seq: 747491, time: 2014-06-12 14:46:02.869713, event: op_applied, op: osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.869731 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3004 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] op_applied on primary on version 173'3005
2014-06-12 14:46:02.873012 7f9d5501e700  5 -- op tracker -- , seq: 747492, time: 2014-06-12 14:46:02.872953, event: header_read, op: osd_sub_op_reply(client.4309.0:141585 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.873025 7f9d5501e700  5 -- op tracker -- , seq: 747492, time: 2014-06-12 14:46:02.872956, event: throttled, op: osd_sub_op_reply(client.4309.0:141585 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.873033 7f9d5501e700  5 -- op tracker -- , seq: 747492, time: 2014-06-12 14:46:02.872973, event: all_read, op: osd_sub_op_reply(client.4309.0:141585 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.873041 7f9d5501e700  5 -- op tracker -- , seq: 747492, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141585 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.873047 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141585 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.873055 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.873064 7f9d5501e700 15 osd.2 173 enqueue_op 0x6f4a700 prio 196 cost 0 latency 0.000110 osd_sub_op_reply(client.4309.0:141585 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.873099 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.873107 7f9d62bd6700 10 osd.2 173 dequeue_op 0x6f4a700 prio 196 cost 0 latency 0.000154 osd_sub_op_reply(client.4309.0:141585 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 pg pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3004 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean]
2014-06-12 14:46:02.873128 7f9d62bd6700  5 -- op tracker -- , seq: 747492, time: 2014-06-12 14:46:02.873125, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141585 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.873137 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3004 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] handle_message: 0x6f4a700
2014-06-12 14:46:02.873153 7f9d62bd6700  5 -- op tracker -- , seq: 747492, time: 2014-06-12 14:46:02.873150, event: started, op: osd_sub_op_reply(client.4309.0:141585 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.873159 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3004 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] sub_op_modify_reply: tid 236192 op  ack_type 4 from 29
2014-06-12 14:46:02.873180 7f9d62bd6700  5 -- op tracker -- , seq: 747491, time: 2014-06-12 14:46:02.873172, event: sub_op_commit_rec, op: osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.873188 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3004 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] repop_all_applied: repop tid 236192 all applied 
2014-06-12 14:46:02.873199 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3004 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] eval_repop repgather(0x1a4449c0 173'3005 rep_tid=236192 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.873216 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3004 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean]  sending ack on repgather(0x1a4449c0 173'3005 rep_tid=236192 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x6824500
2014-06-12 14:46:02.873248 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3004 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] repop_all_committed: repop tid 236192 all committed 
2014-06-12 14:46:02.873259 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3003 active+clean] eval_repop repgather(0x1a4449c0 173'3005 rep_tid=236192 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.873275 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3003 active+clean] log_op_stats osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.009719 lat 0.009746
2014-06-12 14:46:02.873291 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3003 active+clean] publish_stats_to_osd 173:3965
2014-06-12 14:46:02.873305 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3003 active+clean]  sending commit on repgather(0x1a4449c0 173'3005 rep_tid=236192 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x30ccc000
2014-06-12 14:46:02.873334 7f9d62bd6700  5 -- op tracker -- , seq: 747491, time: 2014-06-12 14:46:02.873329, event: commit_sent, op: osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.873342 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3003 active+clean]  requeue_ops 
2014-06-12 14:46:02.873353 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean]  removing repgather(0x1a4449c0 173'3005 rep_tid=236192 committed?=1 applied?=1 op=osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.873372 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean]    q front is repgather(0x1a4449c0 173'3005 rep_tid=236192 committed?=1 applied?=1 op=osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.873386 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] remove_repop repgather(0x1a4449c0 173'3005 rep_tid=236192 committed?=1 applied?=1 op=osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.873400 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean]  requeue_ops 
2014-06-12 14:46:02.873426 7f9d62bd6700  5 -- op tracker -- , seq: 747491, time: 2014-06-12 14:46:02.873420, event: done, op: osd_op(client.4309.0:141585 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.873444 7f9d62bd6700 10 osd.2 173 dequeue_op 0x6f4a700 finish
2014-06-12 14:46:02.873451 7f9d62bd6700  5 -- op tracker -- , seq: 747492, time: 2014-06-12 14:46:02.873448, event: done, op: osd_sub_op_reply(client.4309.0:141585 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.873461 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.873464 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.877250 7f9d4397a700  5 -- op tracker -- , seq: 747493, time: 2014-06-12 14:46:02.876223, event: header_read, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.877263 7f9d4397a700  5 -- op tracker -- , seq: 747493, time: 2014-06-12 14:46:02.876226, event: throttled, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.877280 7f9d4397a700  5 -- op tracker -- , seq: 747493, time: 2014-06-12 14:46:02.877208, event: all_read, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.877286 7f9d4397a700  5 -- op tracker -- , seq: 747493, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.877293 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:02.877317 7f9d4397a700 15 osd.2 173 enqueue_op 0x25e42300 prio 63 cost 516096 latency 0.001094 osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.877348 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.877360 7f9d64bda700 10 osd.2 173 dequeue_op 0x25e42300 prio 63 cost 516096 latency 0.001136 osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean]
2014-06-12 14:46:02.877390 7f9d64bda700  5 -- op tracker -- , seq: 747493, time: 2014-06-12 14:46:02.877385, event: reached_pg, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.877402 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:02.877421 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] handle_message: 0x25e42300
2014-06-12 14:46:02.877437 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] do_op osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:02.877465 7f9d64bda700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_'
2014-06-12 14:46:02.877665 7f9d64bda700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_' = 0
2014-06-12 14:46:02.877684 7f9d64bda700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset'
2014-06-12 14:46:02.877764 7f9d64bda700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset' = 0
2014-06-12 14:46:02.877776 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] populate_obc_watchers 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.877790 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.877801 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] get_object_context: creating obc from disk: 0x1f4bf340
2014-06-12 14:46:02.877811 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] get_object_context: 0x1f4bf340 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 rwstate(none n=0 w=0) oi: 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3005 client.4309.0:141585 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3005) ssc: 0x16ee50a0 snapset: 0=[]:[]+head
2014-06-12 14:46:02.877828 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] find_object_context 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 @head oi=5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3005 client.4309.0:141585 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3005)
2014-06-12 14:46:02.877854 7f9d64bda700  5 -- op tracker -- , seq: 747493, time: 2014-06-12 14:46:02.877848, event: started, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.877864 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] execute_ctx 0x2d2a6000
2014-06-12 14:46:02.877883 7f9d64bda700  5 -- op tracker -- , seq: 747493, time: 2014-06-12 14:46:02.877878, event: started, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.877890 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] do_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] ov 173'3005 av 173'3006 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:02.877906 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] do_osd_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096]
2014-06-12 14:46:02.877918 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:02.877934 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] do_osd_op  write 1921024~516096
2014-06-12 14:46:02.877951 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:02.877965 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 snapset=0x16ee50d8  snapc=0=[]
2014-06-12 14:46:02.877979 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:02.877992 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] finish_ctx 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 0x2d2a6000 op modify  
2014-06-12 14:46:02.878006 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean]  set mtime to 2014-06-12 08:45:05.256413
2014-06-12 14:46:02.878029 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean]  final snapset 0=[]:[]+head in 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.878050 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean]  zeroing write result code 0
2014-06-12 14:46:02.878063 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] calc_trim_to 79'4 -> 79'5
2014-06-12 14:46:02.878073 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] new_repop rep_tid 236193 on osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.878089 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] issue_repop rep_tid 236193 o 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.878113 7f9d64bda700  5 -- op tracker -- , seq: 747493, time: 2014-06-12 14:46:02.878108, event: waiting for subops from 29, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.878139 7f9d64bda700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:02.878164 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3005 (79'4,173'3005] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] append_log log((79'4,173'3005], crt=0'0) [173'3006 (173'3005) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141586 2014-06-12 08:45:05.256413]
2014-06-12 14:46:02.878190 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'4,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] add_log_entry 173'3006 (173'3005) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141586 2014-06-12 08:45:05.256413
2014-06-12 14:46:02.878212 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'4,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] append_log  adding 1 keys
2014-06-12 14:46:02.878227 7f9d64bda700 10 trim log((79'4,173'3006], crt=0'0) to 79'5
2014-06-12 14:46:02.878255 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3006, trimmed: 79'5
2014-06-12 14:46:02.878283 7f9d64bda700  5 queue_transactions existing osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.878289 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:02.878291 7f9d64bda700  5 queue_op 0x5bf30e0 seq 2772 osr(3.602 0x276dd30) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:02.878302 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541c6c0 (1 active)
2014-06-12 14:46:02.878310 7f9d8096e700  5 _do_op 0x5bf30e0 seq 2772 osr(3.602 0x276dd30)/0x276dd30 start
2014-06-12 14:46:02.878298 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] eval_repop repgather(0x52d0000 173'3006 rep_tid=236193 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.878315 7f9d8096e700 10 _do_transaction on 0x52d0fc0
2014-06-12 14:46:02.878316 7f9d64bda700 10 osd.2 173 dequeue_op 0x25e42300 finish
2014-06-12 14:46:02.878320 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.878322 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.878322 7f9d8096e700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.878406 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.878428 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:02.878469 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:02.878565 7f9d8096e700 15 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.878576 7f9d8096e700 10 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1 r = 0
2014-06-12 14:46:02.878583 7f9d8096e700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.878603 7f9d8096e700 15 _write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 1921024~516096
2014-06-12 14:46:02.878697 7f9d8096e700 10 lookup_strip_header done  cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.878882 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:02.880118 7f9d8096e700 10 _generic_write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 1921024~516096 = 0
2014-06-12 14:46:02.880162 7f9d8096e700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.880177 7f9d8096e700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.880186 7f9d8096e700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.880196 7f9d8096e700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.880344 7f9d8096e700 20 set_header setting 17100 cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:02.880407 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:02.880422 7f9d8096e700 20 set_header setting 2102 cid meta oid 1fe11eba/pglog_3.602/0//-1 parent seq 0
2014-06-12 14:46:02.880437 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:02.882289 7f9d8096e700 10 _do_op 0x5bf30e0 seq 2772 r = 0, finisher 0x8586c20 0x4417ec0
2014-06-12 14:46:02.882308 7f9d8096e700 10 _finish_op 0x5bf30e0 seq 2772 osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.882317 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541c6c0 (0 active)
2014-06-12 14:46:02.882321 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.882312 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] op_commit: 236193
2014-06-12 14:46:02.882345 7f9d7715b700  5 -- op tracker -- , seq: 747493, time: 2014-06-12 14:46:02.882335, event: op_commit, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.882375 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] op_applied: 236193
2014-06-12 14:46:02.882396 7f9d7795c700  5 -- op tracker -- , seq: 747493, time: 2014-06-12 14:46:02.882389, event: op_applied, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.882408 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] op_applied on primary on version 173'3006
2014-06-12 14:46:02.883925 7f9d5501e700  5 -- op tracker -- , seq: 747494, time: 2014-06-12 14:46:02.883867, event: header_read, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.883940 7f9d5501e700  5 -- op tracker -- , seq: 747494, time: 2014-06-12 14:46:02.883870, event: throttled, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.883948 7f9d5501e700  5 -- op tracker -- , seq: 747494, time: 2014-06-12 14:46:02.883886, event: all_read, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.883959 7f9d5501e700  5 -- op tracker -- , seq: 747494, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.883970 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:02.883979 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.883988 7f9d5501e700 15 osd.2 173 enqueue_op 0x6f4a800 prio 196 cost 0 latency 0.000120 osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2
2014-06-12 14:46:02.884023 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.884032 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x6f4a800 prio 196 cost 0 latency 0.000165 osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2 pg pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean]
2014-06-12 14:46:02.884055 7f9d5501e700  5 -- op tracker -- , seq: 747495, time: 2014-06-12 14:46:02.884014, event: header_read, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.884067 7f9d5501e700  5 -- op tracker -- , seq: 747495, time: 2014-06-12 14:46:02.884015, event: throttled, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.884076 7f9d5501e700  5 -- op tracker -- , seq: 747495, time: 2014-06-12 14:46:02.884030, event: all_read, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.884088 7f9d5501e700  5 -- op tracker -- , seq: 747495, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.884093 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.884098 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.884104 7f9d5501e700 15 osd.2 173 enqueue_op 0x6f4aa00 prio 196 cost 0 latency 0.000090 osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.884100 7f9d6b3e7700  5 -- op tracker -- , seq: 747494, time: 2014-06-12 14:46:02.884051, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.884113 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] handle_message: 0x6f4a800
2014-06-12 14:46:02.884128 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:02.884130 7f9d6b3e7700  5 -- op tracker -- , seq: 747494, time: 2014-06-12 14:46:02.884127, event: started, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.884137 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] sub_op_modify_reply: tid 236193 op  ack_type 1 from 29
2014-06-12 14:46:02.884156 7f9d6b3e7700  5 -- op tracker -- , seq: 747493, time: 2014-06-12 14:46:02.884149, event: sub_op_applied_rec, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.884163 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] repop_all_applied: repop tid 236193 all applied 
2014-06-12 14:46:02.884175 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] eval_repop repgather(0x52d0000 173'3006 rep_tid=236193 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.884193 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean]  sending ack on repgather(0x52d0000 173'3006 rep_tid=236193 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x5f94c80
2014-06-12 14:46:02.884231 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x6f4a800 finish
2014-06-12 14:46:02.884239 7f9d6b3e7700  5 -- op tracker -- , seq: 747494, time: 2014-06-12 14:46:02.884236, event: done, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.884241 7f9d683e1700 10 osd.2 173 dequeue_op 0x6f4aa00 prio 196 cost 0 latency 0.000227 osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 pg pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean]
2014-06-12 14:46:02.884255 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:02.884258 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.884261 7f9d683e1700  5 -- op tracker -- , seq: 747495, time: 2014-06-12 14:46:02.884257, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.884269 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] handle_message: 0x6f4aa00
2014-06-12 14:46:02.884284 7f9d683e1700  5 -- op tracker -- , seq: 747495, time: 2014-06-12 14:46:02.884281, event: started, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.884289 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] sub_op_modify_reply: tid 236193 op  ack_type 4 from 29
2014-06-12 14:46:02.884307 7f9d683e1700  5 -- op tracker -- , seq: 747493, time: 2014-06-12 14:46:02.884301, event: sub_op_commit_rec, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.884313 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3005 crt=0'0 lcod 173'3004 mlcod 173'3004 active+clean] repop_all_committed: repop tid 236193 all committed 
2014-06-12 14:46:02.884324 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3004 active+clean] eval_repop repgather(0x52d0000 173'3006 rep_tid=236193 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.884343 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3004 active+clean] log_op_stats osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008006 lat 0.008119
2014-06-12 14:46:02.884363 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3004 active+clean] publish_stats_to_osd 173:3966
2014-06-12 14:46:02.884379 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3004 active+clean]  sending commit on repgather(0x52d0000 173'3006 rep_tid=236193 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x37813900
2014-06-12 14:46:02.884416 7f9d683e1700  5 -- op tracker -- , seq: 747493, time: 2014-06-12 14:46:02.884410, event: commit_sent, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.884426 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3004 active+clean]  requeue_ops 
2014-06-12 14:46:02.884439 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean]  removing repgather(0x52d0000 173'3006 rep_tid=236193 committed?=1 applied?=1 op=osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.884455 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean]    q front is repgather(0x52d0000 173'3006 rep_tid=236193 committed?=1 applied?=1 op=osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.884471 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] remove_repop repgather(0x52d0000 173'3006 rep_tid=236193 committed?=1 applied?=1 op=osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.884493 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean]  requeue_ops 
2014-06-12 14:46:02.884521 7f9d683e1700  5 -- op tracker -- , seq: 747493, time: 2014-06-12 14:46:02.884516, event: done, op: osd_op(client.4309.0:141586 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 1921024~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.884540 7f9d683e1700 10 osd.2 173 dequeue_op 0x6f4aa00 finish
2014-06-12 14:46:02.884550 7f9d683e1700  5 -- op tracker -- , seq: 747495, time: 2014-06-12 14:46:02.884546, event: done, op: osd_sub_op_reply(client.4309.0:141586 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.884562 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.884565 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.887531 7f9d4397a700  5 -- op tracker -- , seq: 747496, time: 2014-06-12 14:46:02.886618, event: header_read, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.887544 7f9d4397a700  5 -- op tracker -- , seq: 747496, time: 2014-06-12 14:46:02.886620, event: throttled, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.887554 7f9d4397a700  5 -- op tracker -- , seq: 747496, time: 2014-06-12 14:46:02.887498, event: all_read, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.887567 7f9d4397a700  5 -- op tracker -- , seq: 747496, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.887576 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:02.887604 7f9d4397a700 15 osd.2 173 enqueue_op 0x25e42700 prio 63 cost 516096 latency 0.000986 osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.887632 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.887640 7f9d65bdc700 10 osd.2 173 dequeue_op 0x25e42700 prio 63 cost 516096 latency 0.001022 osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean]
2014-06-12 14:46:02.887665 7f9d65bdc700  5 -- op tracker -- , seq: 747496, time: 2014-06-12 14:46:02.887660, event: reached_pg, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.887674 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:02.887689 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] handle_message: 0x25e42700
2014-06-12 14:46:02.887701 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] do_op osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:02.887728 7f9d65bdc700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_'
2014-06-12 14:46:02.887871 7f9d65bdc700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_' = 0
2014-06-12 14:46:02.887889 7f9d65bdc700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset'
2014-06-12 14:46:02.887951 7f9d65bdc700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset' = 0
2014-06-12 14:46:02.887961 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] populate_obc_watchers 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.887975 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.887986 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] get_object_context: creating obc from disk: 0x37e4b8c0
2014-06-12 14:46:02.887996 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] get_object_context: 0x37e4b8c0 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 rwstate(none n=0 w=0) oi: 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3006 client.4309.0:141586 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3006) ssc: 0x16ee47e0 snapset: 0=[]:[]+head
2014-06-12 14:46:02.888016 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] find_object_context 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 @head oi=5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3006 client.4309.0:141586 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3006)
2014-06-12 14:46:02.888045 7f9d65bdc700  5 -- op tracker -- , seq: 747496, time: 2014-06-12 14:46:02.888038, event: started, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.888052 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] execute_ctx 0x184d4600
2014-06-12 14:46:02.888071 7f9d65bdc700  5 -- op tracker -- , seq: 747496, time: 2014-06-12 14:46:02.888066, event: started, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.888078 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] do_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] ov 173'3006 av 173'3007 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:02.888094 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] do_osd_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096]
2014-06-12 14:46:02.888108 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:02.888122 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] do_osd_op  write 2437120~516096
2014-06-12 14:46:02.888139 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:02.888152 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 snapset=0x16ee4818  snapc=0=[]
2014-06-12 14:46:02.888167 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:02.888180 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] finish_ctx 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 0x184d4600 op modify  
2014-06-12 14:46:02.888194 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean]  set mtime to 2014-06-12 08:45:05.266851
2014-06-12 14:46:02.888216 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean]  final snapset 0=[]:[]+head in 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.888235 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean]  zeroing write result code 0
2014-06-12 14:46:02.888247 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] calc_trim_to 79'5 -> 79'6
2014-06-12 14:46:02.888257 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] new_repop rep_tid 236194 on osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.888273 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] issue_repop rep_tid 236194 o 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.888301 7f9d65bdc700  5 -- op tracker -- , seq: 747496, time: 2014-06-12 14:46:02.888295, event: waiting for subops from 29, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.888321 7f9d65bdc700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:02.888344 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3006 (79'5,173'3006] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] append_log log((79'5,173'3006], crt=0'0) [173'3007 (173'3006) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141587 2014-06-12 08:45:05.266851]
2014-06-12 14:46:02.888369 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'5,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3006 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] add_log_entry 173'3007 (173'3006) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141587 2014-06-12 08:45:05.266851
2014-06-12 14:46:02.888387 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'5,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3006 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] append_log  adding 1 keys
2014-06-12 14:46:02.888399 7f9d65bdc700 10 trim log((79'5,173'3007], crt=0'0) to 79'6
2014-06-12 14:46:02.888422 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3007, trimmed: 79'6
2014-06-12 14:46:02.888442 7f9d65bdc700  5 queue_transactions existing osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.888446 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:02.888447 7f9d65bdc700  5 queue_op 0xdf8a780 seq 2773 osr(3.602 0x276dd30) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:02.888459 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541c6c0 (1 active)
2014-06-12 14:46:02.888466 7f9d7915f700  5 _do_op 0xdf8a780 seq 2773 osr(3.602 0x276dd30)/0x276dd30 start
2014-06-12 14:46:02.888471 7f9d7915f700 10 _do_transaction on 0x18224f00
2014-06-12 14:46:02.888454 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3006 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] eval_repop repgather(0x5215440 173'3007 rep_tid=236194 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.888474 7f9d65bdc700 10 osd.2 173 dequeue_op 0x25e42700 finish
2014-06-12 14:46:02.888478 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.888478 7f9d7915f700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.888480 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.888564 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.888588 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:02.888632 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:02.888761 7f9d7915f700 15 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.888772 7f9d7915f700 10 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1 r = 0
2014-06-12 14:46:02.888778 7f9d7915f700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.888796 7f9d7915f700 15 _write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 2437120~516096
2014-06-12 14:46:02.888961 7f9d7915f700 10 lookup_strip_header done  cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.889233 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:02.891010 7f9d7915f700 10 _generic_write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 2437120~516096 = 0
2014-06-12 14:46:02.891067 7f9d7915f700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.891084 7f9d7915f700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.891092 7f9d7915f700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.891101 7f9d7915f700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.891304 7f9d7915f700 20 set_header setting 17100 cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:02.891384 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:02.891400 7f9d7915f700 20 set_header setting 2102 cid meta oid 1fe11eba/pglog_3.602/0//-1 parent seq 0
2014-06-12 14:46:02.891413 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:02.894455 7f9d5501e700  5 -- op tracker -- , seq: 747497, time: 2014-06-12 14:46:02.894397, event: header_read, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.894479 7f9d5501e700  5 -- op tracker -- , seq: 747497, time: 2014-06-12 14:46:02.894399, event: throttled, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.894487 7f9d5501e700  5 -- op tracker -- , seq: 747497, time: 2014-06-12 14:46:02.894414, event: all_read, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.894495 7f9d5501e700  5 -- op tracker -- , seq: 747497, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.894501 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:02.894508 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.894516 7f9d5501e700 15 osd.2 173 enqueue_op 0x6f4b700 prio 196 cost 0 latency 0.000119 osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2
2014-06-12 14:46:02.894581 7f9d5501e700  5 -- op tracker -- , seq: 747498, time: 2014-06-12 14:46:02.894540, event: header_read, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.894592 7f9d5501e700  5 -- op tracker -- , seq: 747498, time: 2014-06-12 14:46:02.894540, event: throttled, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.894600 7f9d5501e700  5 -- op tracker -- , seq: 747498, time: 2014-06-12 14:46:02.894556, event: all_read, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.894612 7f9d5501e700  5 -- op tracker -- , seq: 747498, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.894616 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.894622 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.894627 7f9d5501e700 15 osd.2 173 enqueue_op 0x6f4b100 prio 196 cost 0 latency 0.000087 osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.894654 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.894663 7f9d6abe6700 10 osd.2 173 dequeue_op 0x6f4b700 prio 196 cost 0 latency 0.000265 osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2 pg pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3006 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean]
2014-06-12 14:46:02.894687 7f9d6abe6700  5 -- op tracker -- , seq: 747497, time: 2014-06-12 14:46:02.894683, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.894695 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3006 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] handle_message: 0x6f4b700
2014-06-12 14:46:02.894713 7f9d6abe6700  5 -- op tracker -- , seq: 747497, time: 2014-06-12 14:46:02.894709, event: started, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.894719 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3006 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] sub_op_modify_reply: tid 236194 op  ack_type 1 from 29
2014-06-12 14:46:02.894743 7f9d6abe6700  5 -- op tracker -- , seq: 747496, time: 2014-06-12 14:46:02.894734, event: sub_op_applied_rec, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.894753 7f9d6abe6700 10 osd.2 173 dequeue_op 0x6f4b700 finish
2014-06-12 14:46:02.894760 7f9d6abe6700  5 -- op tracker -- , seq: 747497, time: 2014-06-12 14:46:02.894757, event: done, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.894778 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.894783 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.894787 7f9d6abe6700 10 osd.2 173 dequeue_op 0x6f4b100 prio 196 cost 0 latency 0.000247 osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 pg pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3006 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean]
2014-06-12 14:46:02.894803 7f9d6abe6700  5 -- op tracker -- , seq: 747498, time: 2014-06-12 14:46:02.894800, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.894809 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3006 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] handle_message: 0x6f4b100
2014-06-12 14:46:02.894823 7f9d6abe6700  5 -- op tracker -- , seq: 747498, time: 2014-06-12 14:46:02.894820, event: started, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.894828 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3006 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] sub_op_modify_reply: tid 236194 op  ack_type 4 from 29
2014-06-12 14:46:02.894844 7f9d6abe6700  5 -- op tracker -- , seq: 747496, time: 2014-06-12 14:46:02.894839, event: sub_op_commit_rec, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.894852 7f9d6abe6700 10 osd.2 173 dequeue_op 0x6f4b100 finish
2014-06-12 14:46:02.894857 7f9d6abe6700  5 -- op tracker -- , seq: 747498, time: 2014-06-12 14:46:02.894854, event: done, op: osd_sub_op_reply(client.4309.0:141587 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.894866 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.894869 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.894877 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.895747 7f9d7915f700 10 _do_op 0xdf8a780 seq 2773 r = 0, finisher 0x8a009c0 0xba9fd40
2014-06-12 14:46:02.895766 7f9d7915f700 10 _finish_op 0xdf8a780 seq 2773 osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.895775 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541c6c0 (0 active)
2014-06-12 14:46:02.895783 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.895771 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3006 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] op_commit: 236194
2014-06-12 14:46:02.895796 7f9d7715b700  5 -- op tracker -- , seq: 747496, time: 2014-06-12 14:46:02.895789, event: op_commit, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.895804 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3006 crt=0'0 lcod 173'3005 mlcod 173'3005 active+clean] repop_all_committed: repop tid 236194 all committed 
2014-06-12 14:46:02.895815 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3005 active+clean] eval_repop repgather(0x5215440 173'3007 rep_tid=236194 committed?=1 applied?=0 lock=0 op=osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.895832 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3005 active+clean] log_op_stats osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.009212
2014-06-12 14:46:02.895849 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3005 active+clean] publish_stats_to_osd 173:3967
2014-06-12 14:46:02.895860 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3005 active+clean]  sending commit on repgather(0x5215440 173'3007 rep_tid=236194 committed?=1 applied?=0 lock=0 op=osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x151e6c80
2014-06-12 14:46:02.895896 7f9d7715b700  5 -- op tracker -- , seq: 747496, time: 2014-06-12 14:46:02.895891, event: commit_sent, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.895944 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3005 active+clean] op_applied: 236194
2014-06-12 14:46:02.895966 7f9d7795c700  5 -- op tracker -- , seq: 747496, time: 2014-06-12 14:46:02.895960, event: op_applied, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.895973 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3005 active+clean] op_applied on primary on version 173'3007
2014-06-12 14:46:02.895983 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3005 active+clean] repop_all_applied: repop tid 236194 all applied 
2014-06-12 14:46:02.895991 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3005 active+clean] eval_repop repgather(0x5215440 173'3007 rep_tid=236194 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.896005 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3005 active+clean] log_op_stats osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.009387
2014-06-12 14:46:02.896019 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3005 active+clean] publish_stats_to_osd 173:3968
2014-06-12 14:46:02.896029 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3005 active+clean]  requeue_ops 
2014-06-12 14:46:02.896038 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean]  removing repgather(0x5215440 173'3007 rep_tid=236194 committed?=1 applied?=1 op=osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.896052 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean]    q front is repgather(0x5215440 173'3007 rep_tid=236194 committed?=1 applied?=1 op=osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.896065 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] remove_repop repgather(0x5215440 173'3007 rep_tid=236194 committed?=1 applied?=1 op=osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.896078 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean]  requeue_ops 
2014-06-12 14:46:02.896102 7f9d7795c700  5 -- op tracker -- , seq: 747496, time: 2014-06-12 14:46:02.896098, event: done, op: osd_op(client.4309.0:141587 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2437120~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.899317 7f9d4397a700  5 -- op tracker -- , seq: 747499, time: 2014-06-12 14:46:02.898228, event: header_read, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.899333 7f9d4397a700  5 -- op tracker -- , seq: 747499, time: 2014-06-12 14:46:02.898231, event: throttled, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.899343 7f9d4397a700  5 -- op tracker -- , seq: 747499, time: 2014-06-12 14:46:02.899269, event: all_read, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.899352 7f9d4397a700  5 -- op tracker -- , seq: 747499, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.899360 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:02.899388 7f9d4397a700 15 osd.2 173 enqueue_op 0x20e87400 prio 63 cost 516096 latency 0.001159 osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.899418 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.899426 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x20e87400 prio 63 cost 516096 latency 0.001198 osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean]
2014-06-12 14:46:02.899457 7f9d6a3e5700  5 -- op tracker -- , seq: 747499, time: 2014-06-12 14:46:02.899452, event: reached_pg, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.899472 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:02.899488 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] handle_message: 0x20e87400
2014-06-12 14:46:02.899499 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] do_op osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:02.899523 7f9d6a3e5700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_'
2014-06-12 14:46:02.899706 7f9d6a3e5700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_' = 0
2014-06-12 14:46:02.899724 7f9d6a3e5700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset'
2014-06-12 14:46:02.899786 7f9d6a3e5700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset' = 0
2014-06-12 14:46:02.899797 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] populate_obc_watchers 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.899811 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.899822 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] get_object_context: creating obc from disk: 0xc8da840
2014-06-12 14:46:02.899833 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] get_object_context: 0xc8da840 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 rwstate(none n=0 w=0) oi: 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3007 client.4309.0:141587 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3007) ssc: 0x5bb9880 snapset: 0=[]:[]+head
2014-06-12 14:46:02.899850 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] find_object_context 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 @head oi=5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3007 client.4309.0:141587 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3007)
2014-06-12 14:46:02.899878 7f9d6a3e5700  5 -- op tracker -- , seq: 747499, time: 2014-06-12 14:46:02.899872, event: started, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.899886 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] execute_ctx 0x2a84a00
2014-06-12 14:46:02.899904 7f9d6a3e5700  5 -- op tracker -- , seq: 747499, time: 2014-06-12 14:46:02.899899, event: started, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.899912 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] do_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] ov 173'3007 av 173'3008 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:02.899928 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] do_osd_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096]
2014-06-12 14:46:02.899941 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:02.899956 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] do_osd_op  write 2953216~516096
2014-06-12 14:46:02.899973 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:02.899986 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 snapset=0x5bb98b8  snapc=0=[]
2014-06-12 14:46:02.900000 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:02.900013 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] finish_ctx 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 0x2a84a00 op modify  
2014-06-12 14:46:02.900029 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean]  set mtime to 2014-06-12 08:45:05.278454
2014-06-12 14:46:02.900054 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean]  final snapset 0=[]:[]+head in 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.900072 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean]  zeroing write result code 0
2014-06-12 14:46:02.900084 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] calc_trim_to 79'6 -> 79'7
2014-06-12 14:46:02.900095 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] new_repop rep_tid 236195 on osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.900112 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] issue_repop rep_tid 236195 o 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.900139 7f9d6a3e5700  5 -- op tracker -- , seq: 747499, time: 2014-06-12 14:46:02.900134, event: waiting for subops from 29, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.900161 7f9d6a3e5700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:02.900183 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3007 (79'6,173'3007] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] append_log log((79'6,173'3007], crt=0'0) [173'3008 (173'3007) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141588 2014-06-12 08:45:05.278454]
2014-06-12 14:46:02.900208 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'6,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] add_log_entry 173'3008 (173'3007) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141588 2014-06-12 08:45:05.278454
2014-06-12 14:46:02.900226 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'6,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] append_log  adding 1 keys
2014-06-12 14:46:02.900237 7f9d6a3e5700 10 trim log((79'6,173'3008], crt=0'0) to 79'7
2014-06-12 14:46:02.900260 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3008, trimmed: 79'7
2014-06-12 14:46:02.900281 7f9d6a3e5700  5 queue_transactions existing osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.900284 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:02.900286 7f9d6a3e5700  5 queue_op 0x5978300 seq 2774 osr(3.602 0x276dd30) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:02.900292 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] eval_repop repgather(0x31ee7a40 173'3008 rep_tid=236195 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.900309 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x20e87400 finish
2014-06-12 14:46:02.900312 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.900314 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.900323 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541c6c0 (1 active)
2014-06-12 14:46:02.900327 7f9d7c165700  5 _do_op 0x5978300 seq 2774 osr(3.602 0x276dd30)/0x276dd30 start
2014-06-12 14:46:02.900331 7f9d7c165700 10 _do_transaction on 0xba84780
2014-06-12 14:46:02.900336 7f9d7c165700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.900409 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.900435 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:02.900475 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:02.900579 7f9d7c165700 15 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.900589 7f9d7c165700 10 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1 r = 0
2014-06-12 14:46:02.900596 7f9d7c165700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.900616 7f9d7c165700 15 _write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 2953216~516096
2014-06-12 14:46:02.900757 7f9d7c165700 10 lookup_strip_header done  cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.900882 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:02.902095 7f9d7c165700 10 _generic_write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 2953216~516096 = 0
2014-06-12 14:46:02.902139 7f9d7c165700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.902153 7f9d7c165700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.902160 7f9d7c165700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.902169 7f9d7c165700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.902312 7f9d7c165700 20 set_header setting 17100 cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:02.902372 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:02.902388 7f9d7c165700 20 set_header setting 2102 cid meta oid 1fe11eba/pglog_3.602/0//-1 parent seq 0
2014-06-12 14:46:02.902399 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:02.904243 7f9d7c165700 10 _do_op 0x5978300 seq 2774 r = 0, finisher 0x29fc0b40 0x3b465040
2014-06-12 14:46:02.904261 7f9d7c165700 10 _finish_op 0x5978300 seq 2774 osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.904269 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541c6c0 (0 active)
2014-06-12 14:46:02.904273 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.904266 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] op_commit: 236195
2014-06-12 14:46:02.904297 7f9d7715b700  5 -- op tracker -- , seq: 747499, time: 2014-06-12 14:46:02.904287, event: op_commit, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.904316 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] op_applied: 236195
2014-06-12 14:46:02.904335 7f9d7795c700  5 -- op tracker -- , seq: 747499, time: 2014-06-12 14:46:02.904329, event: op_applied, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.904342 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] op_applied on primary on version 173'3008
2014-06-12 14:46:02.909165 7f9d5501e700  5 -- op tracker -- , seq: 747500, time: 2014-06-12 14:46:02.909093, event: header_read, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.909190 7f9d5501e700  5 -- op tracker -- , seq: 747500, time: 2014-06-12 14:46:02.909095, event: throttled, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.909198 7f9d5501e700  5 -- op tracker -- , seq: 747500, time: 2014-06-12 14:46:02.909111, event: all_read, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.909214 7f9d5501e700  5 -- op tracker -- , seq: 747500, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.909219 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:02.909225 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.909234 7f9d5501e700 15 osd.2 173 enqueue_op 0x5ee8c00 prio 196 cost 0 latency 0.000141 osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2
2014-06-12 14:46:02.909267 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.909278 7f9d63bd8700 10 osd.2 173 dequeue_op 0x5ee8c00 prio 196 cost 0 latency 0.000185 osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2 pg pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean]
2014-06-12 14:46:02.909295 7f9d5501e700  5 -- op tracker -- , seq: 747501, time: 2014-06-12 14:46:02.909257, event: header_read, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.909309 7f9d5501e700  5 -- op tracker -- , seq: 747501, time: 2014-06-12 14:46:02.909258, event: throttled, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.909319 7f9d5501e700  5 -- op tracker -- , seq: 747501, time: 2014-06-12 14:46:02.909273, event: all_read, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.909347 7f9d5501e700  5 -- op tracker -- , seq: 747501, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.909352 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.909357 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.909358 7f9d63bd8700  5 -- op tracker -- , seq: 747500, time: 2014-06-12 14:46:02.909300, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.909362 7f9d5501e700 15 osd.2 173 enqueue_op 0x5ee8500 prio 196 cost 0 latency 0.000105 osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.909368 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] handle_message: 0x5ee8c00
2014-06-12 14:46:02.909386 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:02.909386 7f9d63bd8700  5 -- op tracker -- , seq: 747500, time: 2014-06-12 14:46:02.909383, event: started, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.909393 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] sub_op_modify_reply: tid 236195 op  ack_type 1 from 29
2014-06-12 14:46:02.909415 7f9d63bd8700  5 -- op tracker -- , seq: 747499, time: 2014-06-12 14:46:02.909406, event: sub_op_applied_rec, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.909423 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] repop_all_applied: repop tid 236195 all applied 
2014-06-12 14:46:02.909435 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] eval_repop repgather(0x31ee7a40 173'3008 rep_tid=236195 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.909452 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean]  sending ack on repgather(0x31ee7a40 173'3008 rep_tid=236195 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x28ea3180
2014-06-12 14:46:02.909488 7f9d63bd8700 10 osd.2 173 dequeue_op 0x5ee8c00 finish
2014-06-12 14:46:02.909498 7f9d63bd8700  5 -- op tracker -- , seq: 747500, time: 2014-06-12 14:46:02.909494, event: done, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.909498 7f9d653db700 10 osd.2 173 dequeue_op 0x5ee8500 prio 196 cost 0 latency 0.000240 osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 pg pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean]
2014-06-12 14:46:02.909513 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:02.909517 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.909517 7f9d653db700  5 -- op tracker -- , seq: 747501, time: 2014-06-12 14:46:02.909514, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.909525 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] handle_message: 0x5ee8500
2014-06-12 14:46:02.909540 7f9d653db700  5 -- op tracker -- , seq: 747501, time: 2014-06-12 14:46:02.909536, event: started, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.909545 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] sub_op_modify_reply: tid 236195 op  ack_type 4 from 29
2014-06-12 14:46:02.909564 7f9d653db700  5 -- op tracker -- , seq: 747499, time: 2014-06-12 14:46:02.909557, event: sub_op_commit_rec, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.909573 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3007 crt=0'0 lcod 173'3006 mlcod 173'3006 active+clean] repop_all_committed: repop tid 236195 all committed 
2014-06-12 14:46:02.909584 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3006 active+clean] eval_repop repgather(0x31ee7a40 173'3008 rep_tid=236195 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.909603 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3006 active+clean] log_op_stats osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.011258 lat 0.011374
2014-06-12 14:46:02.909647 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3006 active+clean] publish_stats_to_osd 173:3969
2014-06-12 14:46:02.909668 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3006 active+clean]  sending commit on repgather(0x31ee7a40 173'3008 rep_tid=236195 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x1ba79b80
2014-06-12 14:46:02.909719 7f9d653db700  5 -- op tracker -- , seq: 747499, time: 2014-06-12 14:46:02.909712, event: commit_sent, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.909733 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3006 active+clean]  requeue_ops 
2014-06-12 14:46:02.909743 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean]  removing repgather(0x31ee7a40 173'3008 rep_tid=236195 committed?=1 applied?=1 op=osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.909760 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean]    q front is repgather(0x31ee7a40 173'3008 rep_tid=236195 committed?=1 applied?=1 op=osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.909776 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] remove_repop repgather(0x31ee7a40 173'3008 rep_tid=236195 committed?=1 applied?=1 op=osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.909792 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean]  requeue_ops 
2014-06-12 14:46:02.909818 7f9d653db700  5 -- op tracker -- , seq: 747499, time: 2014-06-12 14:46:02.909813, event: done, op: osd_op(client.4309.0:141588 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 2953216~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.909842 7f9d653db700 10 osd.2 173 dequeue_op 0x5ee8500 finish
2014-06-12 14:46:02.909850 7f9d653db700  5 -- op tracker -- , seq: 747501, time: 2014-06-12 14:46:02.909846, event: done, op: osd_sub_op_reply(client.4309.0:141588 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.909862 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.909865 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.912940 7f9d4397a700  5 -- op tracker -- , seq: 747502, time: 2014-06-12 14:46:02.912027, event: header_read, op: osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.912952 7f9d4397a700  5 -- op tracker -- , seq: 747502, time: 2014-06-12 14:46:02.912029, event: throttled, op: osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.912961 7f9d4397a700  5 -- op tracker -- , seq: 747502, time: 2014-06-12 14:46:02.912910, event: all_read, op: osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.912969 7f9d4397a700  5 -- op tracker -- , seq: 747502, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.912975 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:02.912989 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19406 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [1,0,0,0,0,0,0,0,0,0,0,4,0,2,4,27,5])
2014-06-12 14:46:02.912998 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19406 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [1,0,0,0,0,0,0,0,0,0,0,4,0,2,4,27,5])
2014-06-12 14:46:02.913002 7f9d4397a700 15 osd.2 173 enqueue_op 0x876ef00 prio 63 cost 516096 latency 0.000974 osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.913030 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.913036 7f9d69be4700 10 osd.2 173 dequeue_op 0x876ef00 prio 63 cost 516096 latency 0.001008 osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean]
2014-06-12 14:46:02.913058 7f9d69be4700  5 -- op tracker -- , seq: 747502, time: 2014-06-12 14:46:02.913054, event: reached_pg, op: osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.913067 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:02.913080 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] handle_message: 0x876ef00
2014-06-12 14:46:02.913090 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] do_op osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:02.913112 7f9d69be4700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_'
2014-06-12 14:46:02.913305 7f9d69be4700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_' = 0
2014-06-12 14:46:02.913497 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:02.913525 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:02.913320 7f9d69be4700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset'
2014-06-12 14:46:02.913674 7f9d69be4700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset' = 0
2014-06-12 14:46:02.913687 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] populate_obc_watchers 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.913708 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.913724 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:02.913719 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] get_object_context: creating obc from disk: 0x29c1e580
2014-06-12 14:46:02.913744 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:02.913732 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] get_object_context: 0x29c1e580 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 rwstate(none n=0 w=0) oi: 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3008 client.4309.0:141588 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3008) ssc: 0x24d4a9a0 snapset: 0=[]:[]+head
2014-06-12 14:46:02.913748 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] find_object_context 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 @head oi=5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3008 client.4309.0:141588 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3008)
2014-06-12 14:46:02.913765 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:02.913784 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:02.913785 7f9d69be4700  5 -- op tracker -- , seq: 747502, time: 2014-06-12 14:46:02.913769, event: started, op: osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.913802 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:02.913796 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] execute_ctx 0xd21a000
2014-06-12 14:46:02.913816 7f9d69be4700  5 -- op tracker -- , seq: 747502, time: 2014-06-12 14:46:02.913810, event: started, op: osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.913827 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:02.913826 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] do_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] ov 173'3008 av 173'3009 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:02.913842 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] do_osd_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096]
2014-06-12 14:46:02.913858 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:02.913854 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:02.913875 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:02.913873 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] do_osd_op  write 3469312~516096
2014-06-12 14:46:02.913897 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:02.913890 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:02.913902 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 snapset=0x24d4a9d8  snapc=0=[]
2014-06-12 14:46:02.913925 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:02.913915 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:02.913945 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:02.913935 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] finish_ctx 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 0xd21a000 op modify  
2014-06-12 14:46:02.913962 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:02.913952 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean]  set mtime to 2014-06-12 08:45:05.292228
2014-06-12 14:46:02.913981 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:02.913974 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean]  final snapset 0=[]:[]+head in 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.913994 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean]  zeroing write result code 0
2014-06-12 14:46:02.914002 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:02.914004 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] calc_trim_to 79'7 -> 79'8
2014-06-12 14:46:02.914014 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] new_repop rep_tid 236196 on osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.914034 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] issue_repop rep_tid 236196 o 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.914059 7f9d69be4700  5 -- op tracker -- , seq: 747502, time: 2014-06-12 14:46:02.914054, event: waiting for subops from 29, op: osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.914088 7f9d69be4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:02.914103 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:02.914110 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3008 (79'7,173'3008] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] append_log log((79'7,173'3008], crt=0'0) [173'3009 (173'3008) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141589 2014-06-12 08:45:05.292228]
2014-06-12 14:46:02.914519 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:02.914538 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:02.914524 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'7,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3008 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] add_log_entry 173'3009 (173'3008) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141589 2014-06-12 08:45:05.292228
2014-06-12 14:46:02.914567 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'7,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3008 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] append_log  adding 1 keys
2014-06-12 14:46:02.914578 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:02.914583 7f9d69be4700 10 trim log((79'7,173'3009], crt=0'0) to 79'8
2014-06-12 14:46:02.914612 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3009, trimmed: 79'8
2014-06-12 14:46:02.914637 7f9d69be4700  5 queue_transactions existing osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.914641 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:02.914643 7f9d69be4700  5 queue_op 0x196fa660 seq 2775 osr(3.602 0x276dd30) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:02.914651 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3008 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] eval_repop repgather(0x51f6840 173'3009 rep_tid=236196 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.914670 7f9d69be4700 10 osd.2 173 dequeue_op 0x876ef00 finish
2014-06-12 14:46:02.914675 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.914678 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.914687 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541c6c0 (1 active)
2014-06-12 14:46:02.914692 7f9d83173700  5 _do_op 0x196fa660 seq 2775 osr(3.602 0x276dd30)/0x276dd30 start
2014-06-12 14:46:02.914696 7f9d83173700 10 _do_transaction on 0x18225080
2014-06-12 14:46:02.914702 7f9d83173700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.914788 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.914812 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:02.914858 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:02.914879 7f9d83173700 15 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.914887 7f9d83173700 10 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1 r = 0
2014-06-12 14:46:02.914893 7f9d83173700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.914986 7f9d83173700 15 _write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 3469312~516096
2014-06-12 14:46:02.915104 7f9d83173700 10 lookup_strip_header done  cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.915227 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:02.916436 7f9d83173700 10 _generic_write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 3469312~516096 = 0
2014-06-12 14:46:02.916480 7f9d83173700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.916493 7f9d83173700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.916500 7f9d83173700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.916509 7f9d83173700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.916655 7f9d83173700 20 set_header setting 17100 cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:02.916712 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:02.916727 7f9d83173700 20 set_header setting 2102 cid meta oid 1fe11eba/pglog_3.602/0//-1 parent seq 0
2014-06-12 14:46:02.916739 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:02.917696 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:02.918328 7f9d83173700 10 _do_op 0x196fa660 seq 2775 r = 0, finisher 0x5f76900 0x100a4d40
2014-06-12 14:46:02.918343 7f9d83173700 10 _finish_op 0x196fa660 seq 2775 osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.918352 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541c6c0 (0 active)
2014-06-12 14:46:02.918359 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.918348 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3008 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] op_commit: 236196
2014-06-12 14:46:02.918378 7f9d7715b700  5 -- op tracker -- , seq: 747502, time: 2014-06-12 14:46:02.918369, event: op_commit, op: osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.918393 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3008 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] op_applied: 236196
2014-06-12 14:46:02.918406 7f9d7795c700  5 -- op tracker -- , seq: 747502, time: 2014-06-12 14:46:02.918402, event: op_applied, op: osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.918411 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3008 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] op_applied on primary on version 173'3009
2014-06-12 14:46:02.920337 7f9d5501e700  5 -- op tracker -- , seq: 747503, time: 2014-06-12 14:46:02.920289, event: header_read, op: osd_sub_op_reply(client.4309.0:141589 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.920350 7f9d5501e700  5 -- op tracker -- , seq: 747503, time: 2014-06-12 14:46:02.920290, event: throttled, op: osd_sub_op_reply(client.4309.0:141589 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.920361 7f9d5501e700  5 -- op tracker -- , seq: 747503, time: 2014-06-12 14:46:02.920304, event: all_read, op: osd_sub_op_reply(client.4309.0:141589 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.920372 7f9d5501e700  5 -- op tracker -- , seq: 747503, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141589 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.920377 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141589 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.920383 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.920393 7f9d5501e700 15 osd.2 173 enqueue_op 0x68a7100 prio 196 cost 0 latency 0.000103 osd_sub_op_reply(client.4309.0:141589 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.920427 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.920437 7f9d633d7700 10 osd.2 173 dequeue_op 0x68a7100 prio 196 cost 0 latency 0.000148 osd_sub_op_reply(client.4309.0:141589 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 pg pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3008 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean]
2014-06-12 14:46:02.920460 7f9d633d7700  5 -- op tracker -- , seq: 747503, time: 2014-06-12 14:46:02.920456, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141589 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.920469 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3008 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] handle_message: 0x68a7100
2014-06-12 14:46:02.920484 7f9d633d7700  5 -- op tracker -- , seq: 747503, time: 2014-06-12 14:46:02.920481, event: started, op: osd_sub_op_reply(client.4309.0:141589 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.920494 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3008 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] sub_op_modify_reply: tid 236196 op  ack_type 4 from 29
2014-06-12 14:46:02.920514 7f9d633d7700  5 -- op tracker -- , seq: 747502, time: 2014-06-12 14:46:02.920507, event: sub_op_commit_rec, op: osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.920521 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3008 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] repop_all_applied: repop tid 236196 all applied 
2014-06-12 14:46:02.920531 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3008 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] eval_repop repgather(0x51f6840 173'3009 rep_tid=236196 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.920548 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3008 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean]  sending ack on repgather(0x51f6840 173'3009 rep_tid=236196 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x3371c500
2014-06-12 14:46:02.920579 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3008 crt=0'0 lcod 173'3007 mlcod 173'3007 active+clean] repop_all_committed: repop tid 236196 all committed 
2014-06-12 14:46:02.920590 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3007 active+clean] eval_repop repgather(0x51f6840 173'3009 rep_tid=236196 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.920605 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3007 active+clean] log_op_stats osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008551 lat 0.008577
2014-06-12 14:46:02.920620 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3007 active+clean] publish_stats_to_osd 173:3970
2014-06-12 14:46:02.920632 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3007 active+clean]  sending commit on repgather(0x51f6840 173'3009 rep_tid=236196 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x3371db80
2014-06-12 14:46:02.920663 7f9d633d7700  5 -- op tracker -- , seq: 747502, time: 2014-06-12 14:46:02.920658, event: commit_sent, op: osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.920687 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3007 active+clean]  requeue_ops 
2014-06-12 14:46:02.920698 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean]  removing repgather(0x51f6840 173'3009 rep_tid=236196 committed?=1 applied?=1 op=osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.920713 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean]    q front is repgather(0x51f6840 173'3009 rep_tid=236196 committed?=1 applied?=1 op=osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.920727 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] remove_repop repgather(0x51f6840 173'3009 rep_tid=236196 committed?=1 applied?=1 op=osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.920741 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean]  requeue_ops 
2014-06-12 14:46:02.920766 7f9d633d7700  5 -- op tracker -- , seq: 747502, time: 2014-06-12 14:46:02.920761, event: done, op: osd_op(client.4309.0:141589 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3469312~516096] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.920786 7f9d633d7700 10 osd.2 173 dequeue_op 0x68a7100 finish
2014-06-12 14:46:02.920792 7f9d633d7700  5 -- op tracker -- , seq: 747503, time: 2014-06-12 14:46:02.920789, event: done, op: osd_sub_op_reply(client.4309.0:141589 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.920801 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.920804 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.921692 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:02.922755 7f9d4397a700  5 -- op tracker -- , seq: 747504, time: 2014-06-12 14:46:02.922355, event: header_read, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.922774 7f9d4397a700  5 -- op tracker -- , seq: 747504, time: 2014-06-12 14:46:02.922357, event: throttled, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.922785 7f9d4397a700  5 -- op tracker -- , seq: 747504, time: 2014-06-12 14:46:02.922718, event: all_read, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.922794 7f9d4397a700  5 -- op tracker -- , seq: 747504, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.922801 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:02.922826 7f9d4397a700 15 osd.2 173 enqueue_op 0x876ee00 prio 63 cost 208896 latency 0.000471 osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.922871 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.922876 7f9d643d9700 10 osd.2 173 dequeue_op 0x876ee00 prio 63 cost 208896 latency 0.000521 osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean]
2014-06-12 14:46:02.922895 7f9d643d9700  5 -- op tracker -- , seq: 747504, time: 2014-06-12 14:46:02.922891, event: reached_pg, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.922902 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:02.922912 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] handle_message: 0x876ee00
2014-06-12 14:46:02.922919 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] do_op osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:02.922937 7f9d643d9700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_'
2014-06-12 14:46:02.923087 7f9d643d9700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 '_' = 0
2014-06-12 14:46:02.923101 7f9d643d9700 15 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset'
2014-06-12 14:46:02.923154 7f9d643d9700 10 getattr 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 'snapset' = 0
2014-06-12 14:46:02.923163 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] populate_obc_watchers 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.923170 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.923176 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] get_object_context: creating obc from disk: 0x341c8b00
2014-06-12 14:46:02.923182 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] get_object_context: 0x341c8b00 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 rwstate(none n=0 w=0) oi: 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3009 client.4309.0:141589 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3009) ssc: 0x2ef8fc00 snapset: 0=[]:[]+head
2014-06-12 14:46:02.923192 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] find_object_context 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 @head oi=5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3(173'3009 client.4309.0:141589 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3009)
2014-06-12 14:46:02.923208 7f9d643d9700  5 -- op tracker -- , seq: 747504, time: 2014-06-12 14:46:02.923205, event: started, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.923224 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] execute_ctx 0x5ca3600
2014-06-12 14:46:02.923235 7f9d643d9700  5 -- op tracker -- , seq: 747504, time: 2014-06-12 14:46:02.923232, event: started, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.923245 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] do_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] ov 173'3009 av 173'3010 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:02.923254 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] do_osd_op 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896]
2014-06-12 14:46:02.923261 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:02.923269 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] do_osd_op  write 3985408~208896
2014-06-12 14:46:02.923279 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:02.923286 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 snapset=0x2ef8fc38  snapc=0=[]
2014-06-12 14:46:02.923293 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] make_writeable 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:02.923299 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] finish_ctx 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 0x5ca3600 op modify  
2014-06-12 14:46:02.923307 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean]  set mtime to 2014-06-12 08:45:05.303121
2014-06-12 14:46:02.923319 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean]  final snapset 0=[]:[]+head in 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.923331 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean]  zeroing write result code 0
2014-06-12 14:46:02.923337 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] calc_trim_to 79'8 -> 79'9
2014-06-12 14:46:02.923343 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] new_repop rep_tid 236197 on osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:02.923353 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] issue_repop rep_tid 236197 o 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.923368 7f9d643d9700  5 -- op tracker -- , seq: 747504, time: 2014-06-12 14:46:02.923365, event: waiting for subops from 29, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.923386 7f9d643d9700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:02.923400 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3009 (79'8,173'3009] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] append_log log((79'8,173'3009], crt=0'0) [173'3010 (173'3009) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141590 2014-06-12 08:45:05.303121]
2014-06-12 14:46:02.923607 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'8,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] add_log_entry 173'3010 (173'3009) modify   5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 by client.4309.0:141590 2014-06-12 08:45:05.303121
2014-06-12 14:46:02.923621 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'8,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] append_log  adding 1 keys
2014-06-12 14:46:02.923629 7f9d643d9700 10 trim log((79'8,173'3010], crt=0'0) to 79'9
2014-06-12 14:46:02.923650 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3010, trimmed: 79'9
2014-06-12 14:46:02.923675 7f9d643d9700  5 queue_transactions existing osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.923681 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:02.923683 7f9d643d9700  5 queue_op 0x15cda240 seq 2776 osr(3.602 0x276dd30) 211224 bytes   (queue has 1 ops and 211224 bytes)
2014-06-12 14:46:02.923696 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541c6c0 (1 active)
2014-06-12 14:46:02.923706 7f9d82972700  5 _do_op 0x15cda240 seq 2776 osr(3.602 0x276dd30)/0x276dd30 start
2014-06-12 14:46:02.923692 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] eval_repop repgather(0x11c7f200 173'3010 rep_tid=236197 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.923709 7f9d82972700 10 _do_transaction on 0x11c7ea80
2014-06-12 14:46:02.923710 7f9d643d9700 10 osd.2 173 dequeue_op 0x876ee00 finish
2014-06-12 14:46:02.923713 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.923716 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.923715 7f9d82972700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.923797 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.923819 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:02.923857 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:02.923953 7f9d82972700 15 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.923961 7f9d82972700 10 _omap_rmkeys meta/1fe11eba/pglog_3.602/0//-1 r = 0
2014-06-12 14:46:02.923965 7f9d82972700 15 _omap_setkeys meta/1fe11eba/pglog_3.602/0//-1
2014-06-12 14:46:02.923978 7f9d82972700 15 _write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 3985408~208896
2014-06-12 14:46:02.924058 7f9d82972700 10 lookup_strip_header done  cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.924250 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:02.924674 7f9d82972700 10 _generic_write 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 3985408~208896 = 0
2014-06-12 14:46:02.924697 7f9d82972700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.924705 7f9d82972700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.924709 7f9d82972700 15 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3
2014-06-12 14:46:02.924713 7f9d82972700 10 _setattrs 3.602_head/5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:02.924798 7f9d82972700 20 set_header setting 17100 cid 3.602_head oid 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:02.924830 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:02.924839 7f9d82972700 20 set_header setting 2102 cid meta oid 1fe11eba/pglog_3.602/0//-1 parent seq 0
2014-06-12 14:46:02.924846 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:02.925454 7f9d82972700 10 _do_op 0x15cda240 seq 2776 r = 0, finisher 0x313867c0 0x5e7e440
2014-06-12 14:46:02.925464 7f9d82972700 10 _finish_op 0x15cda240 seq 2776 osr(3.602 0x276dd30)/0x276dd30
2014-06-12 14:46:02.925470 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541c6c0 (0 active)
2014-06-12 14:46:02.925473 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.925469 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] op_commit: 236197
2014-06-12 14:46:02.925496 7f9d7715b700  5 -- op tracker -- , seq: 747504, time: 2014-06-12 14:46:02.925488, event: op_commit, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.925517 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] op_applied: 236197
2014-06-12 14:46:02.925536 7f9d7795c700  5 -- op tracker -- , seq: 747504, time: 2014-06-12 14:46:02.925531, event: op_applied, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.925542 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] op_applied on primary on version 173'3010
2014-06-12 14:46:02.927354 7f9d5501e700  5 -- op tracker -- , seq: 747505, time: 2014-06-12 14:46:02.927322, event: header_read, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.927361 7f9d5501e700  5 -- op tracker -- , seq: 747505, time: 2014-06-12 14:46:02.927323, event: throttled, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.927367 7f9d5501e700  5 -- op tracker -- , seq: 747505, time: 2014-06-12 14:46:02.927333, event: all_read, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.927371 7f9d5501e700  5 -- op tracker -- , seq: 747505, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.927375 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:02.927379 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.927383 7f9d5501e700 15 osd.2 173 enqueue_op 0x68a6600 prio 196 cost 0 latency 0.000061 osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2
2014-06-12 14:46:02.927404 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.927412 7f9d623d5700 10 osd.2 173 dequeue_op 0x68a6600 prio 196 cost 0 latency 0.000089 osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0) v2 pg pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean]
2014-06-12 14:46:02.927431 7f9d623d5700  5 -- op tracker -- , seq: 747505, time: 2014-06-12 14:46:02.927427, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.927443 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] handle_message: 0x68a6600
2014-06-12 14:46:02.927459 7f9d623d5700  5 -- op tracker -- , seq: 747505, time: 2014-06-12 14:46:02.927455, event: started, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.927464 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] sub_op_modify_reply: tid 236197 op  ack_type 1 from 29
2014-06-12 14:46:02.927483 7f9d623d5700  5 -- op tracker -- , seq: 747504, time: 2014-06-12 14:46:02.927477, event: sub_op_applied_rec, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.927500 7f9d5501e700  5 -- op tracker -- , seq: 747506, time: 2014-06-12 14:46:02.927467, event: header_read, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.927495 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] repop_all_applied: repop tid 236197 all applied 
2014-06-12 14:46:02.927506 7f9d5501e700  5 -- op tracker -- , seq: 747506, time: 2014-06-12 14:46:02.927467, event: throttled, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.927511 7f9d5501e700  5 -- op tracker -- , seq: 747506, time: 2014-06-12 14:46:02.927477, event: all_read, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.927515 7f9d5501e700  5 -- op tracker -- , seq: 747506, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.927518 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.927507 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] eval_repop repgather(0x11c7f200 173'3010 rep_tid=236197 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.927521 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:02.927524 7f9d5501e700 15 osd.2 173 enqueue_op 0x24c9c900 prio 196 cost 0 latency 0.000057 osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.927524 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean]  sending ack on repgather(0x11c7f200 173'3010 rep_tid=236197 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x24fec80
2014-06-12 14:46:02.927542 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:02.927554 7f9d623d5700 10 osd.2 173 dequeue_op 0x68a6600 finish
2014-06-12 14:46:02.927563 7f9d623d5700  5 -- op tracker -- , seq: 747505, time: 2014-06-12 14:46:02.927560, event: done, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ack, result = 0)
2014-06-12 14:46:02.927564 7f9d68be2700 10 osd.2 173 dequeue_op 0x24c9c900 prio 196 cost 0 latency 0.000096 osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0) v2 pg pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean]
2014-06-12 14:46:02.927578 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:02.927581 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.927583 7f9d68be2700  5 -- op tracker -- , seq: 747506, time: 2014-06-12 14:46:02.927580, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.927588 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] handle_message: 0x24c9c900
2014-06-12 14:46:02.927598 7f9d68be2700  5 -- op tracker -- , seq: 747506, time: 2014-06-12 14:46:02.927596, event: started, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.927601 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] sub_op_modify_reply: tid 236197 op  ack_type 4 from 29
2014-06-12 14:46:02.927612 7f9d68be2700  5 -- op tracker -- , seq: 747504, time: 2014-06-12 14:46:02.927608, event: sub_op_commit_rec, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.927616 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'3009 crt=0'0 lcod 173'3008 mlcod 173'3008 active+clean] repop_all_committed: repop tid 236197 all committed 
2014-06-12 14:46:02.927622 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3008 active+clean] eval_repop repgather(0x11c7f200 173'3010 rep_tid=236197 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.927632 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3008 active+clean] log_op_stats osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4 inb 209175 outb 0 rlat 0.005197 lat 0.005276
2014-06-12 14:46:02.927643 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3008 active+clean] publish_stats_to_osd 173:3971
2014-06-12 14:46:02.927651 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3008 active+clean]  sending commit on repgather(0x11c7f200 173'3010 rep_tid=236197 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4) 0x31452a00
2014-06-12 14:46:02.927674 7f9d68be2700  5 -- op tracker -- , seq: 747504, time: 2014-06-12 14:46:02.927668, event: commit_sent, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.927681 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3008 active+clean]  requeue_ops 
2014-06-12 14:46:02.927692 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3009 active+clean]  removing repgather(0x11c7f200 173'3010 rep_tid=236197 committed?=1 applied?=1 op=osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.927702 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3009 active+clean]    q front is repgather(0x11c7f200 173'3010 rep_tid=236197 committed?=1 applied?=1 op=osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.927711 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3009 active+clean] remove_repop repgather(0x11c7f200 173'3010 rep_tid=236197 committed?=1 applied?=1 op=osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.927719 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3009 active+clean]  requeue_ops 
2014-06-12 14:46:02.927735 7f9d68be2700  5 -- op tracker -- , seq: 747504, time: 2014-06-12 14:46:02.927732, event: done, op: osd_op(client.4309.0:141590 rbd_data.10ac6b8b4567.0000000000000f62 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.5da2b602 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.927798 7f9d68be2700 10 osd.2 173 dequeue_op 0x24c9c900 finish
2014-06-12 14:46:02.927804 7f9d68be2700  5 -- op tracker -- , seq: 747506, time: 2014-06-12 14:46:02.927801, event: done, op: osd_sub_op_reply(client.4309.0:141590 3.602 5da2b602/rbd_data.10ac6b8b4567.0000000000000f62/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.927811 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.927813 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.960983 7f9d5a16f700  5 -- op tracker -- , seq: 747507, time: 2014-06-12 14:46:02.960075, event: header_read, op: osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.961000 7f9d5a16f700  5 -- op tracker -- , seq: 747507, time: 2014-06-12 14:46:02.960078, event: throttled, op: osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.961023 7f9d5a16f700  5 -- op tracker -- , seq: 747507, time: 2014-06-12 14:46:02.960931, event: all_read, op: osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.961030 7f9d5a16f700  5 -- op tracker -- , seq: 747507, time: 0.000000, event: dispatched, op: osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.961035 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:02.961041 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:02.961049 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329eb100 prio 127 cost 516903 latency 0.000973 osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:02.961079 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.961087 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329eb100 prio 127 cost 516903 latency 0.001012 osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2716 (0'0,173'2716] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active]
2014-06-12 14:46:02.961106 7f9d61bd4700  5 -- op tracker -- , seq: 747507, time: 2014-06-12 14:46:02.961103, event: reached_pg, op: osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.961113 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2716 (0'0,173'2716] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active] handle_message: 0x329eb100
2014-06-12 14:46:02.961121 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2716 (0'0,173'2716] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active] sub_op_modify trans e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 v 173'2717 (transaction) 174
2014-06-12 14:46:02.961132 7f9d61bd4700  5 -- op tracker -- , seq: 747507, time: 2014-06-12 14:46:02.961130, event: started, op: osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.961144 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2716 (0'0,173'2716] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active] append_log log((0'0,173'2716], crt=0'0) [173'2717 (173'2716) modify   e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 by client.4285.0:143304 2014-06-12 08:42:54.298852]
2014-06-12 14:46:02.961164 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2717 (0'0,173'2717] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active] add_log_entry 173'2717 (173'2716) modify   e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 by client.4285.0:143304 2014-06-12 08:42:54.298852
2014-06-12 14:46:02.961181 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2717 (0'0,173'2717] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active] append_log  adding 1 keys
2014-06-12 14:46:02.961209 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2717, trimmed: 
2014-06-12 14:46:02.961231 7f9d61bd4700  5 -- op tracker -- , seq: 747507, time: 2014-06-12 14:46:02.961228, event: started, op: osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.961240 7f9d61bd4700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:02.961243 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:02.961244 7f9d61bd4700  5 queue_op 0x558e3c0 seq 2464 osr(3.21c 0x276d020) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:02.961249 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2717 (0'0,173'2717] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active] do_sub_op osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:02.961259 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329eb100 finish
2014-06-12 14:46:02.961261 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.961256 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (1 active)
2014-06-12 14:46:02.961263 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.961265 7f9d7c966700  5 _do_op 0x558e3c0 seq 2464 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:46:02.961270 7f9d7c966700 10 _do_transaction on 0x66640f0
2014-06-12 14:46:02.961277 7f9d7c966700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:02.961399 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:02.961424 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:02.961467 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:02.961492 7f9d7c966700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:02.961499 7f9d7c966700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:46:02.961504 7f9d7c966700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:02.961519 7f9d7c966700 15 _write 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 2965504~516096
2014-06-12 14:46:02.961658 7f9d7c966700 10 lookup_strip_header done  cid 3.21c_head oid e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:46:02.961865 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:02.963510 7f9d7c966700 10 _generic_write 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 2965504~516096 = 0
2014-06-12 14:46:02.963562 7f9d7c966700 15 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:46:02.963582 7f9d7c966700 10 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 = 0
2014-06-12 14:46:02.963589 7f9d7c966700 15 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:46:02.963596 7f9d7c966700 10 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 = 0
2014-06-12 14:46:02.963793 7f9d7c966700 20 set_header setting 17494 cid 3.21c_head oid e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 parent seq 0
2014-06-12 14:46:02.963863 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:02.963877 7f9d7c966700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:46:02.963885 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:02.966424 7f9d7c966700 10 _do_op 0x558e3c0 seq 2464 r = 0, finisher 0x5f76540 0
2014-06-12 14:46:02.966446 7f9d7c966700 10 _finish_op 0x558e3c0 seq 2464 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:02.966453 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (0 active)
2014-06-12 14:46:02.966465 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:02.966496 7f9d7795c700  5 -- op tracker -- , seq: 747507, time: 2014-06-12 14:46:02.966483, event: sub_op_applied, op: osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.966509 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2717 (0'0,173'2717] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active] sub_op_modify_applied on 0x6664000 op osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:02.966535 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:02.966556 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2717 (0'0,173'2717] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active] op_applied on primary on version 173'2717
2014-06-12 14:46:02.966568 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2717 (0'0,173'2717] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active] op_applied on replica on version 173'2717
2014-06-12 14:46:02.966601 7f9d7715b700  5 -- op tracker -- , seq: 747507, time: 2014-06-12 14:46:02.966596, event: commit_sent, op: osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.966609 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2717 (0'0,173'2717] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2715 active] sub_op_modify_commit on op osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:02.966622 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:02.966652 7f9d7715b700  5 -- op tracker -- , seq: 747507, time: 2014-06-12 14:46:02.966646, event: done, op: osd_sub_op(client.4285.0:143304 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:02.994121 7f9d5a16f700  5 -- op tracker -- , seq: 747508, time: 2014-06-12 14:46:02.994072, event: header_read, op: osd_sub_op_reply(client.4310.0:142388 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.994129 7f9d5a16f700  5 -- op tracker -- , seq: 747508, time: 2014-06-12 14:46:02.994073, event: throttled, op: osd_sub_op_reply(client.4310.0:142388 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.994134 7f9d5a16f700  5 -- op tracker -- , seq: 747508, time: 2014-06-12 14:46:02.994089, event: all_read, op: osd_sub_op_reply(client.4310.0:142388 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.994150 7f9d5a16f700  5 -- op tracker -- , seq: 747508, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142388 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.994155 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142388 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.994161 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:02.994169 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329eb900 prio 196 cost 0 latency 0.000097 osd_sub_op_reply(client.4310.0:142388 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.994233 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.994242 7f9d673df700 10 osd.2 173 dequeue_op 0x329eb900 prio 196 cost 0 latency 0.000170 osd_sub_op_reply(client.4310.0:142388 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2 pg pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean]
2014-06-12 14:46:02.994265 7f9d673df700  5 -- op tracker -- , seq: 747508, time: 2014-06-12 14:46:02.994256, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142388 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.994274 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] handle_message: 0x329eb900
2014-06-12 14:46:02.994285 7f9d673df700  5 -- op tracker -- , seq: 747508, time: 2014-06-12 14:46:02.994283, event: started, op: osd_sub_op_reply(client.4310.0:142388 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.994290 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] sub_op_modify_reply: tid 235993 op  ack_type 4 from 24
2014-06-12 14:46:02.994303 7f9d673df700  5 -- op tracker -- , seq: 746899, time: 2014-06-12 14:46:02.994298, event: sub_op_commit_rec, op: osd_op(client.4310.0:142388 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.994310 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] repop_all_applied: repop tid 235993 all applied 
2014-06-12 14:46:02.994316 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] eval_repop repgather(0x32b678c0 173'2700 rep_tid=235993 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142388 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.994331 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean]  sending ack on repgather(0x32b678c0 173'2700 rep_tid=235993 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142388 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) 0x1d54f400
2014-06-12 14:46:02.994350 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] repop_all_committed: repop tid 235993 all committed 
2014-06-12 14:46:02.994371 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2698 active+clean] eval_repop repgather(0x32b678c0 173'2700 rep_tid=235993 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142388 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.994385 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2698 active+clean] log_op_stats osd_op(client.4310.0:142388 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 32.774426 lat 32.774462
2014-06-12 14:46:02.994397 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2698 active+clean] publish_stats_to_osd 173:2883
2014-06-12 14:46:02.994415 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2698 active+clean]  sending commit on repgather(0x32b678c0 173'2700 rep_tid=235993 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142388 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) 0x16cf2f00
2014-06-12 14:46:02.994443 7f9d673df700  5 -- op tracker -- , seq: 746899, time: 2014-06-12 14:46:02.994438, event: commit_sent, op: osd_op(client.4310.0:142388 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.994451 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2698 active+clean]  requeue_ops 
2014-06-12 14:46:02.994472 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]  removing repgather(0x32b678c0 173'2700 rep_tid=235993 committed?=1 applied?=1 op=osd_op(client.4310.0:142388 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.994482 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]    q front is repgather(0x32b678c0 173'2700 rep_tid=235993 committed?=1 applied?=1 op=osd_op(client.4310.0:142388 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.994490 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] remove_repop repgather(0x32b678c0 173'2700 rep_tid=235993 committed?=1 applied?=1 op=osd_op(client.4310.0:142388 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:02.994498 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]  requeue_ops 
2014-06-12 14:46:02.994511 7f9d673df700  5 -- op tracker -- , seq: 746899, time: 2014-06-12 14:46:02.994508, event: done, op: osd_op(client.4310.0:142388 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 872448~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.994537 7f9d673df700 10 osd.2 173 dequeue_op 0x329eb900 finish
2014-06-12 14:46:02.994542 7f9d673df700  5 -- op tracker -- , seq: 747508, time: 2014-06-12 14:46:02.994539, event: done, op: osd_sub_op_reply(client.4310.0:142388 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.994548 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:02.994550 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:02.998349 7f9d5a16f700  5 -- op tracker -- , seq: 747509, time: 2014-06-12 14:46:02.998254, event: header_read, op: osd_sub_op_reply(client.4310.0:142389 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.998371 7f9d5a16f700  5 -- op tracker -- , seq: 747509, time: 2014-06-12 14:46:02.998255, event: throttled, op: osd_sub_op_reply(client.4310.0:142389 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.998398 7f9d5a16f700  5 -- op tracker -- , seq: 747509, time: 2014-06-12 14:46:02.998273, event: all_read, op: osd_sub_op_reply(client.4310.0:142389 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.998468 7f9d5a16f700  5 -- op tracker -- , seq: 747509, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142389 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.998476 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142389 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:02.998482 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:02.998497 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329ea200 prio 196 cost 0 latency 0.000242 osd_sub_op_reply(client.4310.0:142389 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:02.998708 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:02.998718 7f9d663dd700 10 osd.2 173 dequeue_op 0x329ea200 prio 196 cost 0 latency 0.000464 osd_sub_op_reply(client.4310.0:142389 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2 pg pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]
2014-06-12 14:46:02.998740 7f9d663dd700  5 -- op tracker -- , seq: 747509, time: 2014-06-12 14:46:02.998737, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142389 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.998750 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] handle_message: 0x329ea200
2014-06-12 14:46:02.998768 7f9d663dd700  5 -- op tracker -- , seq: 747509, time: 2014-06-12 14:46:02.998764, event: started, op: osd_sub_op_reply(client.4310.0:142389 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:02.998787 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] sub_op_modify_reply: tid 235994 op  ack_type 4 from 24
2014-06-12 14:46:02.998811 7f9d663dd700  5 -- op tracker -- , seq: 746900, time: 2014-06-12 14:46:02.998802, event: sub_op_commit_rec, op: osd_op(client.4310.0:142389 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:02.998820 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] repop_all_applied: repop tid 235994 all applied 
2014-06-12 14:46:02.998832 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] eval_repop repgather(0x2236840 173'2701 rep_tid=235994 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142389 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.998848 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]  sending ack on repgather(0x2236840 173'2701 rep_tid=235994 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142389 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) 0x564c500
2014-06-12 14:46:02.998881 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] repop_all_committed: repop tid 235994 all committed 
2014-06-12 14:46:02.998892 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2699 active+clean] eval_repop repgather(0x2236840 173'2701 rep_tid=235994 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142389 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:02.999220 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2699 active+clean] log_op_stats osd_op(client.4310.0:142389 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 32.772646 lat 32.772984
2014-06-12 14:46:02.999244 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2699 active+clean] publish_stats_to_osd 173:2884
2014-06-12 14:46:02.999258 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2699 active+clean]  sending commit on repgather(0x2236840 173'2701 rep_tid=235994 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142389 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) 0x11671b80
2014-06-12 14:46:03.000071 7f9d663dd700  5 -- op tracker -- , seq: 746900, time: 2014-06-12 14:46:03.000040, event: commit_sent, op: osd_op(client.4310.0:142389 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.000083 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2699 active+clean]  requeue_ops 
2014-06-12 14:46:03.000095 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]  removing repgather(0x2236840 173'2701 rep_tid=235994 committed?=1 applied?=1 op=osd_op(client.4310.0:142389 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.000111 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]    q front is repgather(0x2236840 173'2701 rep_tid=235994 committed?=1 applied?=1 op=osd_op(client.4310.0:142389 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.000126 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] remove_repop repgather(0x2236840 173'2701 rep_tid=235994 committed?=1 applied?=1 op=osd_op(client.4310.0:142389 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.000140 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]  requeue_ops 
2014-06-12 14:46:03.000162 7f9d663dd700  5 -- op tracker -- , seq: 746900, time: 2014-06-12 14:46:03.000157, event: done, op: osd_op(client.4310.0:142389 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.000221 7f9d663dd700 10 osd.2 173 dequeue_op 0x329ea200 finish
2014-06-12 14:46:03.000231 7f9d663dd700  5 -- op tracker -- , seq: 747509, time: 2014-06-12 14:46:03.000227, event: done, op: osd_sub_op_reply(client.4310.0:142389 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.000248 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.000254 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.004607 7f9d42918700  5 -- op tracker -- , seq: 747510, time: 2014-06-12 14:46:02.997820, event: header_read, op: osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.004621 7f9d42918700  5 -- op tracker -- , seq: 747510, time: 2014-06-12 14:46:02.997823, event: throttled, op: osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.004632 7f9d42918700  5 -- op tracker -- , seq: 747510, time: 2014-06-12 14:46:03.004545, event: all_read, op: osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.004641 7f9d42918700  5 -- op tracker -- , seq: 747510, time: 0.000000, event: dispatched, op: osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.004651 7f9d42918700 20 osd.2 173 should_share_map client.4312 172.16.96.220:0/1244332 173
2014-06-12 14:46:03.004683 7f9d42918700 15 osd.2 173 enqueue_op 0x55c3a00 prio 63 cost 1032192 latency 0.006863 osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.004735 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.004743 7f9d67be0700 10 osd.2 173 dequeue_op 0x55c3a00 prio 63 cost 1032192 latency 0.006923 osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean]
2014-06-12 14:46:03.004770 7f9d67be0700  5 -- op tracker -- , seq: 747510, time: 2014-06-12 14:46:03.004765, event: reached_pg, op: osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.004779 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.004794 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] handle_message: 0x55c3a00
2014-06-12 14:46:03.004804 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] do_op osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.004831 7f9d67be0700 15 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 '_'
2014-06-12 14:46:03.005046 7f9d67be0700 10 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 '_' = 0
2014-06-12 14:46:03.005064 7f9d67be0700 15 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 'snapset'
2014-06-12 14:46:03.005153 7f9d67be0700 10 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 'snapset' = 0
2014-06-12 14:46:03.005162 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] populate_obc_watchers d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.005173 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.005182 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] get_object_context: creating obc from disk: 0x30f32dc0
2014-06-12 14:46:03.005191 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] get_object_context: 0x30f32dc0 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 rwstate(none n=0 w=0) oi: d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3(173'2731 client.4312.0:144979 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2731) ssc: 0x9345880 snapset: 0=[]:[]+head
2014-06-12 14:46:03.005208 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] find_object_context d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 @head oi=d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3(173'2731 client.4312.0:144979 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2731)
2014-06-12 14:46:03.005234 7f9d67be0700  5 -- op tracker -- , seq: 747510, time: 2014-06-12 14:46:03.005229, event: started, op: osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.005242 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] execute_ctx 0x5ca0600
2014-06-12 14:46:03.005258 7f9d67be0700  5 -- op tracker -- , seq: 747510, time: 2014-06-12 14:46:03.005254, event: started, op: osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.005265 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] do_op d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] ov 173'2731 av 173'2732 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.005278 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] do_osd_op d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192]
2014-06-12 14:46:03.005288 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.005301 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] do_osd_op  write 1294336~1032192
2014-06-12 14:46:03.005317 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] do_osd_op_effects on session 0x18a5e000
2014-06-12 14:46:03.005327 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] make_writeable d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 snapset=0x93458b8  snapc=0=[]
2014-06-12 14:46:03.005340 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] make_writeable d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.005350 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] finish_ctx d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 0x5ca0600 op modify  
2014-06-12 14:46:03.005364 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean]  set mtime to 2014-06-12 08:45:05.376837
2014-06-12 14:46:03.005387 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean]  final snapset 0=[]:[]+head in d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.005409 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.005421 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] new_repop rep_tid 236198 on osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.005437 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] issue_repop rep_tid 236198 o d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.005467 7f9d67be0700  5 -- op tracker -- , seq: 747510, time: 2014-06-12 14:46:03.005462, event: waiting for subops from 25, op: osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.005501 7f9d67be0700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:03.005534 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2731 (0'0,173'2731] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] append_log log((0'0,173'2731], crt=0'0) [173'2732 (173'2731) modify   d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 by client.4312.0:144980 2014-06-12 08:45:05.376837]
2014-06-12 14:46:03.005558 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2731 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] add_log_entry 173'2732 (173'2731) modify   d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 by client.4312.0:144980 2014-06-12 08:45:05.376837
2014-06-12 14:46:03.005580 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2731 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.005639 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2732, trimmed: 
2014-06-12 14:46:03.005675 7f9d67be0700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:03.005681 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.005683 7f9d67be0700  5 queue_op 0x10528ae0 seq 2544 osr(3.122 0x276cf40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:03.005694 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2731 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] eval_repop repgather(0x24fa6780 173'2732 rep_tid=236198 committed?=0 applied?=0 lock=0 op=osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.005716 7f9d67be0700 10 osd.2 173 dequeue_op 0x55c3a00 finish
2014-06-12 14:46:03.005722 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.005727 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.007023 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:46:03.007046 7f9d81970700  5 _do_op 0x10528ae0 seq 2544 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:46:03.007053 7f9d81970700 10 _do_transaction on 0x9908900
2014-06-12 14:46:03.007063 7f9d81970700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:03.007417 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:03.007443 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.007491 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.007636 7f9d81970700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:03.007643 7f9d81970700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:46:03.007650 7f9d81970700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:03.007671 7f9d81970700 15 _write 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 1294336~1032192
2014-06-12 14:46:03.007819 7f9d81970700 10 lookup_strip_header done  cid 3.122_head oid d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.008015 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:03.012942 7f9d81970700 10 _generic_write 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 1294336~1032192 = 0
2014-06-12 14:46:03.013053 7f9d81970700 15 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.013070 7f9d81970700 10 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 = 0
2014-06-12 14:46:03.013076 7f9d81970700 15 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.013087 7f9d81970700 10 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 = 0
2014-06-12 14:46:03.013405 7f9d81970700 20 set_header setting 17279 cid 3.122_head oid d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 parent seq 0
2014-06-12 14:46:03.013548 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.013566 7f9d81970700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:46:03.013577 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:03.021161 7f9d81970700 10 _do_op 0x10528ae0 seq 2544 r = 0, finisher 0x100263a0 0x1a3f1e40
2014-06-12 14:46:03.021240 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2731 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] op_commit: 236198
2014-06-12 14:46:03.021274 7f9d7715b700  5 -- op tracker -- , seq: 747510, time: 2014-06-12 14:46:03.021264, event: op_commit, op: osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.021297 7f9d81970700 10 _finish_op 0x10528ae0 seq 2544 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:03.021308 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:46:03.021313 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.021317 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2731 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] op_applied: 236198
2014-06-12 14:46:03.021341 7f9d7795c700  5 -- op tracker -- , seq: 747510, time: 2014-06-12 14:46:03.021335, event: op_applied, op: osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.021355 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2731 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] op_applied on primary on version 173'2732
2014-06-12 14:46:03.025730 7f9d5bec4700  5 -- op tracker -- , seq: 747511, time: 2014-06-12 14:46:03.025673, event: header_read, op: osd_sub_op_reply(client.4312.0:144980 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.025742 7f9d5bec4700  5 -- op tracker -- , seq: 747511, time: 2014-06-12 14:46:03.025676, event: throttled, op: osd_sub_op_reply(client.4312.0:144980 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.025750 7f9d5bec4700  5 -- op tracker -- , seq: 747511, time: 2014-06-12 14:46:03.025690, event: all_read, op: osd_sub_op_reply(client.4312.0:144980 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.025758 7f9d5bec4700  5 -- op tracker -- , seq: 747511, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4312.0:144980 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.025763 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4312.0:144980 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.025769 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:03.025777 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e87d00 prio 196 cost 0 latency 0.000104 osd_sub_op_reply(client.4312.0:144980 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.025807 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.025813 7f9d693e3700 10 osd.2 173 dequeue_op 0x20e87d00 prio 196 cost 0 latency 0.000140 osd_sub_op_reply(client.4312.0:144980 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2731 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean]
2014-06-12 14:46:03.025833 7f9d693e3700  5 -- op tracker -- , seq: 747511, time: 2014-06-12 14:46:03.025830, event: reached_pg, op: osd_sub_op_reply(client.4312.0:144980 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.025841 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2731 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] handle_message: 0x20e87d00
2014-06-12 14:46:03.025856 7f9d693e3700  5 -- op tracker -- , seq: 747511, time: 2014-06-12 14:46:03.025853, event: started, op: osd_sub_op_reply(client.4312.0:144980 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.025862 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2731 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] sub_op_modify_reply: tid 236198 op  ack_type 4 from 25
2014-06-12 14:46:03.025881 7f9d693e3700  5 -- op tracker -- , seq: 747510, time: 2014-06-12 14:46:03.025874, event: sub_op_commit_rec, op: osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.025888 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2731 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] repop_all_applied: repop tid 236198 all applied 
2014-06-12 14:46:03.025898 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2731 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] eval_repop repgather(0x24fa6780 173'2732 rep_tid=236198 committed?=0 applied?=1 lock=0 op=osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.025914 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2731 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean]  sending ack on repgather(0x24fa6780 173'2732 rep_tid=236198 committed?=0 applied?=1 lock=0 op=osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) 0x15ce3680
2014-06-12 14:46:03.025946 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2731 crt=0'0 lcod 173'2730 mlcod 173'2730 active+clean] repop_all_committed: repop tid 236198 all committed 
2014-06-12 14:46:03.025957 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2730 active+clean] eval_repop repgather(0x24fa6780 173'2732 rep_tid=236198 committed?=1 applied?=1 lock=0 op=osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.025973 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2730 active+clean] log_op_stats osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.028124 lat 0.028151
2014-06-12 14:46:03.025990 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2730 active+clean] publish_stats_to_osd 173:3629
2014-06-12 14:46:03.026003 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2730 active+clean]  sending commit on repgather(0x24fa6780 173'2732 rep_tid=236198 committed?=1 applied?=1 lock=0 op=osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) 0x564db80
2014-06-12 14:46:03.026035 7f9d693e3700  5 -- op tracker -- , seq: 747510, time: 2014-06-12 14:46:03.026029, event: commit_sent, op: osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.026043 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2730 active+clean]  requeue_ops 
2014-06-12 14:46:03.026053 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean]  removing repgather(0x24fa6780 173'2732 rep_tid=236198 committed?=1 applied?=1 op=osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.026067 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean]    q front is repgather(0x24fa6780 173'2732 rep_tid=236198 committed?=1 applied?=1 op=osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.026080 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] remove_repop repgather(0x24fa6780 173'2732 rep_tid=236198 committed?=1 applied?=1 op=osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.026093 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean]  requeue_ops 
2014-06-12 14:46:03.026118 7f9d693e3700  5 -- op tracker -- , seq: 747510, time: 2014-06-12 14:46:03.026113, event: done, op: osd_op(client.4312.0:144980 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.026139 7f9d693e3700 10 osd.2 173 dequeue_op 0x20e87d00 finish
2014-06-12 14:46:03.026146 7f9d693e3700  5 -- op tracker -- , seq: 747511, time: 2014-06-12 14:46:03.026143, event: done, op: osd_sub_op_reply(client.4312.0:144980 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.026156 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.026159 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.036199 7f9d42918700  5 -- op tracker -- , seq: 747512, time: 2014-06-12 14:46:03.030270, event: header_read, op: osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.036222 7f9d42918700  5 -- op tracker -- , seq: 747512, time: 2014-06-12 14:46:03.030273, event: throttled, op: osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.036232 7f9d42918700  5 -- op tracker -- , seq: 747512, time: 2014-06-12 14:46:03.036146, event: all_read, op: osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.036240 7f9d42918700  5 -- op tracker -- , seq: 747512, time: 0.000000, event: dispatched, op: osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.036249 7f9d42918700 20 osd.2 173 should_share_map client.4312 172.16.96.220:0/1244332 173
2014-06-12 14:46:03.036277 7f9d42918700 15 osd.2 173 enqueue_op 0x55c3f00 prio 63 cost 1032192 latency 0.006006 osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.036314 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.036324 7f9d62bd6700 10 osd.2 173 dequeue_op 0x55c3f00 prio 63 cost 1032192 latency 0.006053 osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean]
2014-06-12 14:46:03.036352 7f9d62bd6700  5 -- op tracker -- , seq: 747512, time: 2014-06-12 14:46:03.036347, event: reached_pg, op: osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.036363 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.036378 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] handle_message: 0x55c3f00
2014-06-12 14:46:03.036388 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] do_op osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.036413 7f9d62bd6700 15 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 '_'
2014-06-12 14:46:03.036613 7f9d62bd6700 10 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 '_' = 0
2014-06-12 14:46:03.036629 7f9d62bd6700 15 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 'snapset'
2014-06-12 14:46:03.036705 7f9d62bd6700 10 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 'snapset' = 0
2014-06-12 14:46:03.036714 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] populate_obc_watchers d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.036725 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.036734 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] get_object_context: creating obc from disk: 0x6968dc0
2014-06-12 14:46:03.036744 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] get_object_context: 0x6968dc0 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 rwstate(none n=0 w=0) oi: d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3(173'2732 client.4312.0:144980 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2732) ssc: 0x679b0a0 snapset: 0=[]:[]+head
2014-06-12 14:46:03.036759 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] find_object_context d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 @head oi=d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3(173'2732 client.4312.0:144980 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2732)
2014-06-12 14:46:03.036783 7f9d62bd6700  5 -- op tracker -- , seq: 747512, time: 2014-06-12 14:46:03.036777, event: started, op: osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.036791 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] execute_ctx 0xd21c400
2014-06-12 14:46:03.036808 7f9d62bd6700  5 -- op tracker -- , seq: 747512, time: 2014-06-12 14:46:03.036803, event: started, op: osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.036814 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] do_op d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] ov 173'2732 av 173'2733 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.036828 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] do_osd_op d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192]
2014-06-12 14:46:03.036839 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.036853 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] do_osd_op  write 2326528~1032192
2014-06-12 14:46:03.036868 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] do_osd_op_effects on session 0x18a5e000
2014-06-12 14:46:03.036878 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] make_writeable d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 snapset=0x679b0d8  snapc=0=[]
2014-06-12 14:46:03.036889 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] make_writeable d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.036899 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] finish_ctx d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 0xd21c400 op modify  
2014-06-12 14:46:03.036911 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean]  set mtime to 2014-06-12 08:45:05.409294
2014-06-12 14:46:03.036931 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean]  final snapset 0=[]:[]+head in d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.036949 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.036959 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] new_repop rep_tid 236199 on osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.036975 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] issue_repop rep_tid 236199 o d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.036998 7f9d62bd6700  5 -- op tracker -- , seq: 747512, time: 2014-06-12 14:46:03.036993, event: waiting for subops from 25, op: osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.037024 7f9d62bd6700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:03.037048 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2732 (0'0,173'2732] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] append_log log((0'0,173'2732], crt=0'0) [173'2733 (173'2732) modify   d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 by client.4312.0:144981 2014-06-12 08:45:05.409294]
2014-06-12 14:46:03.037071 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2732 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] add_log_entry 173'2733 (173'2732) modify   d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 by client.4312.0:144981 2014-06-12 08:45:05.409294
2014-06-12 14:46:03.037090 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2732 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.037119 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2733, trimmed: 
2014-06-12 14:46:03.037140 7f9d62bd6700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:03.037144 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.037145 7f9d62bd6700  5 queue_op 0x614cd80 seq 2545 osr(3.122 0x276cf40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:03.037159 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2732 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] eval_repop repgather(0x2f89ed80 173'2733 rep_tid=236199 committed?=0 applied?=0 lock=0 op=osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.037175 7f9d62bd6700 10 osd.2 173 dequeue_op 0x55c3f00 finish
2014-06-12 14:46:03.037179 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.037182 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.038364 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:46:03.038376 7f9d7a161700  5 _do_op 0x614cd80 seq 2545 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:46:03.038381 7f9d7a161700 10 _do_transaction on 0x1696d2c0
2014-06-12 14:46:03.038391 7f9d7a161700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:03.038624 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:03.038653 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.038702 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.038726 7f9d7a161700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:03.038733 7f9d7a161700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:46:03.038739 7f9d7a161700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:03.038757 7f9d7a161700 15 _write 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 2326528~1032192
2014-06-12 14:46:03.038888 7f9d7a161700 10 lookup_strip_header done  cid 3.122_head oid d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.039140 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:03.045382 7f9d7a161700 10 _generic_write 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 2326528~1032192 = 0
2014-06-12 14:46:03.045524 7f9d7a161700 15 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.045550 7f9d7a161700 10 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 = 0
2014-06-12 14:46:03.045556 7f9d7a161700 15 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.045564 7f9d7a161700 10 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 = 0
2014-06-12 14:46:03.045896 7f9d7a161700 20 set_header setting 17279 cid 3.122_head oid d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 parent seq 0
2014-06-12 14:46:03.046006 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.046023 7f9d7a161700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:46:03.046033 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:03.047160 7f9d56432700  5 -- op tracker -- , seq: 747513, time: 2014-06-12 14:46:03.047110, event: header_read, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ack, result = 0)
2014-06-12 14:46:03.047176 7f9d56432700  5 -- op tracker -- , seq: 747513, time: 2014-06-12 14:46:03.047113, event: throttled, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ack, result = 0)
2014-06-12 14:46:03.047194 7f9d56432700  5 -- op tracker -- , seq: 747513, time: 2014-06-12 14:46:03.047126, event: all_read, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ack, result = 0)
2014-06-12 14:46:03.047202 7f9d56432700  5 -- op tracker -- , seq: 747513, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ack, result = 0)
2014-06-12 14:46:03.047208 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:03.047213 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:03.047221 7f9d56432700 15 osd.2 173 enqueue_op 0x5ee8a00 prio 196 cost 0 latency 0.000111 osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ack, result = 0) v2
2014-06-12 14:46:03.047279 7f9d56432700  5 -- op tracker -- , seq: 747514, time: 2014-06-12 14:46:03.047244, event: header_read, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.047290 7f9d56432700  5 -- op tracker -- , seq: 747514, time: 2014-06-12 14:46:03.047245, event: throttled, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.047299 7f9d56432700  5 -- op tracker -- , seq: 747514, time: 2014-06-12 14:46:03.047255, event: all_read, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.047307 7f9d56432700  5 -- op tracker -- , seq: 747514, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.047311 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.047316 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:03.047321 7f9d56432700 15 osd.2 173 enqueue_op 0x5ee8b00 prio 196 cost 0 latency 0.000077 osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.047346 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.047357 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:03.047354 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x5ee8a00 prio 196 cost 0 latency 0.000244 osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ack, result = 0) v2 pg pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean]
2014-06-12 14:46:03.047375 7f9d6b3e7700  5 -- op tracker -- , seq: 747513, time: 2014-06-12 14:46:03.047372, event: reached_pg, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ack, result = 0)
2014-06-12 14:46:03.047384 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] handle_message: 0x5ee8a00
2014-06-12 14:46:03.047401 7f9d6b3e7700  5 -- op tracker -- , seq: 747513, time: 2014-06-12 14:46:03.047397, event: started, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ack, result = 0)
2014-06-12 14:46:03.047407 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] sub_op_modify_reply: tid 235971 op  ack_type 1 from 26
2014-06-12 14:46:03.047429 7f9d6b3e7700  5 -- op tracker -- , seq: 746782, time: 2014-06-12 14:46:03.047420, event: sub_op_applied_rec, op: osd_op(client.4311.0:139599 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~86016] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.047436 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] repop_all_applied: repop tid 235971 all applied 
2014-06-12 14:46:03.047447 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] eval_repop repgather(0x5113080 173'2658 rep_tid=235971 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139599 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~86016] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.047462 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean]  sending ack on repgather(0x5113080 173'2658 rep_tid=235971 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139599 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~86016] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) 0x353e9180
2014-06-12 14:46:03.047495 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x5ee8a00 finish
2014-06-12 14:46:03.047504 7f9d6b3e7700  5 -- op tracker -- , seq: 747513, time: 2014-06-12 14:46:03.047501, event: done, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ack, result = 0)
2014-06-12 14:46:03.047518 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:03.047505 7f9d64bda700 10 osd.2 173 dequeue_op 0x5ee8b00 prio 196 cost 0 latency 0.000261 osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2 pg pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean]
2014-06-12 14:46:03.047521 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.047526 7f9d64bda700  5 -- op tracker -- , seq: 747514, time: 2014-06-12 14:46:03.047522, event: reached_pg, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.047534 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] handle_message: 0x5ee8b00
2014-06-12 14:46:03.047550 7f9d64bda700  5 -- op tracker -- , seq: 747514, time: 2014-06-12 14:46:03.047547, event: started, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.047556 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] sub_op_modify_reply: tid 235971 op  ack_type 4 from 26
2014-06-12 14:46:03.047575 7f9d64bda700  5 -- op tracker -- , seq: 746782, time: 2014-06-12 14:46:03.047568, event: sub_op_commit_rec, op: osd_op(client.4311.0:139599 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~86016] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.047582 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] repop_all_committed: repop tid 235971 all committed 
2014-06-12 14:46:03.047592 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2656 active+clean] eval_repop repgather(0x5113080 173'2658 rep_tid=235971 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139599 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~86016] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.047609 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2656 active+clean] log_op_stats osd_op(client.4311.0:139599 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~86016] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4 inb 86295 outb 0 rlat 36.664286 lat 36.664401
2014-06-12 14:46:03.047626 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2656 active+clean] publish_stats_to_osd 173:3023
2014-06-12 14:46:03.047639 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2656 active+clean]  sending commit on repgather(0x5113080 173'2658 rep_tid=235971 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139599 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~86016] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) 0x31453680
2014-06-12 14:46:03.047670 7f9d64bda700  5 -- op tracker -- , seq: 746782, time: 2014-06-12 14:46:03.047665, event: commit_sent, op: osd_op(client.4311.0:139599 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~86016] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.047677 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2656 active+clean]  requeue_ops 
2014-06-12 14:46:03.047695 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]  removing repgather(0x5113080 173'2658 rep_tid=235971 committed?=1 applied?=1 op=osd_op(client.4311.0:139599 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~86016] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.047710 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]    q front is repgather(0x5113080 173'2658 rep_tid=235971 committed?=1 applied?=1 op=osd_op(client.4311.0:139599 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~86016] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.047723 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] remove_repop repgather(0x5113080 173'2658 rep_tid=235971 committed?=1 applied?=1 op=osd_op(client.4311.0:139599 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~86016] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.047737 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]  requeue_ops 
2014-06-12 14:46:03.047756 7f9d64bda700  5 -- op tracker -- , seq: 746782, time: 2014-06-12 14:46:03.047751, event: done, op: osd_op(client.4311.0:139599 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~86016] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.047772 7f9d64bda700 10 osd.2 173 dequeue_op 0x5ee8b00 finish
2014-06-12 14:46:03.047780 7f9d64bda700  5 -- op tracker -- , seq: 747514, time: 2014-06-12 14:46:03.047776, event: done, op: osd_sub_op_reply(client.4311.0:139599 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.047790 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.047793 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.052896 7f9d7a161700 10 _do_op 0x614cd80 seq 2545 r = 0, finisher 0x216fbb00 0x35f48580
2014-06-12 14:46:03.052963 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2732 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] op_commit: 236199
2014-06-12 14:46:03.052995 7f9d7715b700  5 -- op tracker -- , seq: 747512, time: 2014-06-12 14:46:03.052985, event: op_commit, op: osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.053042 7f9d7a161700 10 _finish_op 0x614cd80 seq 2545 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:03.053073 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2732 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] op_applied: 236199
2014-06-12 14:46:03.053096 7f9d7795c700  5 -- op tracker -- , seq: 747512, time: 2014-06-12 14:46:03.053090, event: op_applied, op: osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.053103 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2732 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] op_applied on primary on version 173'2733
2014-06-12 14:46:03.053164 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:46:03.053173 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.053719 7f9d5bec4700  5 -- op tracker -- , seq: 747515, time: 2014-06-12 14:46:03.053671, event: header_read, op: osd_sub_op_reply(client.4312.0:144981 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.053731 7f9d5bec4700  5 -- op tracker -- , seq: 747515, time: 2014-06-12 14:46:03.053673, event: throttled, op: osd_sub_op_reply(client.4312.0:144981 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.053738 7f9d5bec4700  5 -- op tracker -- , seq: 747515, time: 2014-06-12 14:46:03.053686, event: all_read, op: osd_sub_op_reply(client.4312.0:144981 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.053745 7f9d5bec4700  5 -- op tracker -- , seq: 747515, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4312.0:144981 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.053751 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4312.0:144981 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.053757 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:03.053765 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e86a00 prio 196 cost 0 latency 0.000094 osd_sub_op_reply(client.4312.0:144981 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.055041 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.055060 7f9d683e1700 10 osd.2 173 dequeue_op 0x20e86a00 prio 196 cost 0 latency 0.001388 osd_sub_op_reply(client.4312.0:144981 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2732 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean]
2014-06-12 14:46:03.055090 7f9d683e1700  5 -- op tracker -- , seq: 747515, time: 2014-06-12 14:46:03.055086, event: reached_pg, op: osd_sub_op_reply(client.4312.0:144981 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.055101 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2732 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] handle_message: 0x20e86a00
2014-06-12 14:46:03.055120 7f9d683e1700  5 -- op tracker -- , seq: 747515, time: 2014-06-12 14:46:03.055116, event: started, op: osd_sub_op_reply(client.4312.0:144981 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.055126 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2732 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] sub_op_modify_reply: tid 236199 op  ack_type 4 from 25
2014-06-12 14:46:03.055172 7f9d683e1700  5 -- op tracker -- , seq: 747512, time: 2014-06-12 14:46:03.055163, event: sub_op_commit_rec, op: osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.055182 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2732 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] repop_all_applied: repop tid 236199 all applied 
2014-06-12 14:46:03.055196 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2732 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] eval_repop repgather(0x2f89ed80 173'2733 rep_tid=236199 committed?=0 applied?=1 lock=0 op=osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.055215 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2732 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean]  sending ack on repgather(0x2f89ed80 173'2733 rep_tid=236199 committed?=0 applied?=1 lock=0 op=osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) 0x16cf3900
2014-06-12 14:46:03.055253 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2732 crt=0'0 lcod 173'2731 mlcod 173'2731 active+clean] repop_all_committed: repop tid 236199 all committed 
2014-06-12 14:46:03.055265 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2731 active+clean] eval_repop repgather(0x2f89ed80 173'2733 rep_tid=236199 committed?=1 applied?=1 lock=0 op=osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.055282 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2731 active+clean] log_op_stats osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.024981 lat 0.025010
2014-06-12 14:46:03.055302 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2731 active+clean] publish_stats_to_osd 173:3630
2014-06-12 14:46:03.055319 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2731 active+clean]  sending commit on repgather(0x2f89ed80 173'2733 rep_tid=236199 committed?=1 applied?=1 lock=0 op=osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) 0x35715400
2014-06-12 14:46:03.055354 7f9d683e1700  5 -- op tracker -- , seq: 747512, time: 2014-06-12 14:46:03.055349, event: commit_sent, op: osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.055363 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2731 active+clean]  requeue_ops 
2014-06-12 14:46:03.055373 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean]  removing repgather(0x2f89ed80 173'2733 rep_tid=236199 committed?=1 applied?=1 op=osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.055388 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean]    q front is repgather(0x2f89ed80 173'2733 rep_tid=236199 committed?=1 applied?=1 op=osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.055401 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] remove_repop repgather(0x2f89ed80 173'2733 rep_tid=236199 committed?=1 applied?=1 op=osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.055412 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean]  requeue_ops 
2014-06-12 14:46:03.055449 7f9d683e1700  5 -- op tracker -- , seq: 747512, time: 2014-06-12 14:46:03.055443, event: done, op: osd_op(client.4312.0:144981 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~1032192] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.055478 7f9d683e1700 10 osd.2 173 dequeue_op 0x20e86a00 finish
2014-06-12 14:46:03.055487 7f9d683e1700  5 -- op tracker -- , seq: 747515, time: 2014-06-12 14:46:03.055483, event: done, op: osd_sub_op_reply(client.4312.0:144981 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.055505 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.055510 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.060932 7f9d42918700  5 -- op tracker -- , seq: 747516, time: 2014-06-12 14:46:03.058755, event: header_read, op: osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.060957 7f9d42918700  5 -- op tracker -- , seq: 747516, time: 2014-06-12 14:46:03.058765, event: throttled, op: osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.060968 7f9d42918700  5 -- op tracker -- , seq: 747516, time: 2014-06-12 14:46:03.060875, event: all_read, op: osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.060977 7f9d42918700  5 -- op tracker -- , seq: 747516, time: 0.000000, event: dispatched, op: osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.060987 7f9d42918700 20 osd.2 173 should_share_map client.4312 172.16.96.220:0/1244332 173
2014-06-12 14:46:03.061020 7f9d42918700 15 osd.2 173 enqueue_op 0x55c3400 prio 63 cost 835584 latency 0.002264 osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.061054 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.061061 7f9d65bdc700 10 osd.2 173 dequeue_op 0x55c3400 prio 63 cost 835584 latency 0.002305 osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean]
2014-06-12 14:46:03.061085 7f9d65bdc700  5 -- op tracker -- , seq: 747516, time: 2014-06-12 14:46:03.061080, event: reached_pg, op: osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.061094 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.061109 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] handle_message: 0x55c3400
2014-06-12 14:46:03.061119 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] do_op osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.061141 7f9d65bdc700 15 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 '_'
2014-06-12 14:46:03.067956 7f9d65bdc700 10 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 '_' = 0
2014-06-12 14:46:03.067988 7f9d65bdc700 15 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 'snapset'
2014-06-12 14:46:03.068069 7f9d65bdc700 10 getattr 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 'snapset' = 0
2014-06-12 14:46:03.068083 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] populate_obc_watchers d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.068100 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.068110 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] get_object_context: creating obc from disk: 0x299e4580
2014-06-12 14:46:03.068120 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] get_object_context: 0x299e4580 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 rwstate(none n=0 w=0) oi: d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3(173'2733 client.4312.0:144981 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2733) ssc: 0xee5e380 snapset: 0=[]:[]+head
2014-06-12 14:46:03.068136 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] find_object_context d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 @head oi=d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3(173'2733 client.4312.0:144981 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2733)
2014-06-12 14:46:03.068167 7f9d65bdc700  5 -- op tracker -- , seq: 747516, time: 2014-06-12 14:46:03.068159, event: started, op: osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.068181 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] execute_ctx 0x184d5e00
2014-06-12 14:46:03.068198 7f9d65bdc700  5 -- op tracker -- , seq: 747516, time: 2014-06-12 14:46:03.068193, event: started, op: osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.068206 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] do_op d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] ov 173'2733 av 173'2734 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.068227 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] do_osd_op d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584]
2014-06-12 14:46:03.068239 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.068254 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] do_osd_op  write 3358720~835584
2014-06-12 14:46:03.068271 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] do_osd_op_effects on session 0x18a5e000
2014-06-12 14:46:03.068283 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] make_writeable d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 snapset=0xee5e3b8  snapc=0=[]
2014-06-12 14:46:03.068296 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] make_writeable d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.068306 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] finish_ctx d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 0x184d5e00 op modify  
2014-06-12 14:46:03.068319 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean]  set mtime to 2014-06-12 08:45:05.438414
2014-06-12 14:46:03.068342 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean]  final snapset 0=[]:[]+head in d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.068365 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.068383 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] new_repop rep_tid 236200 on osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.068400 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] issue_repop rep_tid 236200 o d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.068429 7f9d65bdc700  5 -- op tracker -- , seq: 747516, time: 2014-06-12 14:46:03.068424, event: waiting for subops from 25, op: osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.068455 7f9d65bdc700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:03.068481 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2733 (0'0,173'2733] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] append_log log((0'0,173'2733], crt=0'0) [173'2734 (173'2733) modify   d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 by client.4312.0:144982 2014-06-12 08:45:05.438414]
2014-06-12 14:46:03.068507 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2733 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] add_log_entry 173'2734 (173'2733) modify   d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 by client.4312.0:144982 2014-06-12 08:45:05.438414
2014-06-12 14:46:03.068528 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2733 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.068594 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2734, trimmed: 
2014-06-12 14:46:03.068617 7f9d65bdc700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:03.068621 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.068623 7f9d65bdc700  5 queue_op 0x5875e00 seq 2546 osr(3.122 0x276cf40) 837877 bytes   (queue has 1 ops and 837877 bytes)
2014-06-12 14:46:03.068632 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2733 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] eval_repop repgather(0x13b43680 173'2734 rep_tid=236200 committed?=0 applied?=0 lock=0 op=osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.068651 7f9d65bdc700 10 osd.2 173 dequeue_op 0x55c3400 finish
2014-06-12 14:46:03.068657 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.068662 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.072220 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:46:03.072237 7f9d7a962700  5 _do_op 0x5875e00 seq 2546 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:46:03.072244 7f9d7a962700 10 _do_transaction on 0x13b420c0
2014-06-12 14:46:03.072253 7f9d7a962700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:03.072389 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:03.072425 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.072478 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.072690 7f9d7a962700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:03.072697 7f9d7a962700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:46:03.072704 7f9d7a962700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:03.072723 7f9d7a962700 15 _write 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 3358720~835584
2014-06-12 14:46:03.072894 7f9d7a962700 10 lookup_strip_header done  cid 3.122_head oid d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.073080 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:03.076526 7f9d7a962700 10 _generic_write 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 3358720~835584 = 0
2014-06-12 14:46:03.076621 7f9d7a962700 15 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.076658 7f9d7a962700 10 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 = 0
2014-06-12 14:46:03.076667 7f9d7a962700 15 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3
2014-06-12 14:46:03.076676 7f9d7a962700 10 _setattrs 3.122_head/d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 = 0
2014-06-12 14:46:03.077040 7f9d7a962700 20 set_header setting 17279 cid 3.122_head oid d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 parent seq 0
2014-06-12 14:46:03.077146 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.077160 7f9d7a962700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:46:03.077171 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:03.080242 7f9d5bec4700  5 -- op tracker -- , seq: 747517, time: 2014-06-12 14:46:03.080180, event: header_read, op: osd_sub_op_reply(client.4312.0:144982 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.080254 7f9d5bec4700  5 -- op tracker -- , seq: 747517, time: 2014-06-12 14:46:03.080182, event: throttled, op: osd_sub_op_reply(client.4312.0:144982 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.080262 7f9d5bec4700  5 -- op tracker -- , seq: 747517, time: 2014-06-12 14:46:03.080202, event: all_read, op: osd_sub_op_reply(client.4312.0:144982 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.080269 7f9d5bec4700  5 -- op tracker -- , seq: 747517, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4312.0:144982 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.080277 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4312.0:144982 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.080284 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:03.080295 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e86200 prio 196 cost 0 latency 0.000114 osd_sub_op_reply(client.4312.0:144982 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.080333 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.080342 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20e86200 prio 196 cost 0 latency 0.000162 osd_sub_op_reply(client.4312.0:144982 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2733 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean]
2014-06-12 14:46:03.080364 7f9d6abe6700  5 -- op tracker -- , seq: 747517, time: 2014-06-12 14:46:03.080361, event: reached_pg, op: osd_sub_op_reply(client.4312.0:144982 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.080373 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2733 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] handle_message: 0x20e86200
2014-06-12 14:46:03.080396 7f9d6abe6700  5 -- op tracker -- , seq: 747517, time: 2014-06-12 14:46:03.080392, event: started, op: osd_sub_op_reply(client.4312.0:144982 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.080402 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2733 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] sub_op_modify_reply: tid 236200 op  ack_type 4 from 25
2014-06-12 14:46:03.080421 7f9d6abe6700  5 -- op tracker -- , seq: 747516, time: 2014-06-12 14:46:03.080414, event: sub_op_commit_rec, op: osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.080429 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20e86200 finish
2014-06-12 14:46:03.080436 7f9d6abe6700  5 -- op tracker -- , seq: 747517, time: 2014-06-12 14:46:03.080432, event: done, op: osd_sub_op_reply(client.4312.0:144982 3.122 d61f2122/rbd_data.10af6b8b4567.0000000000000efb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.080450 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.080453 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.082124 7f9d7a962700 10 _do_op 0x5875e00 seq 2546 r = 0, finisher 0x6643760 0x4537040
2014-06-12 14:46:03.082155 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2733 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] op_commit: 236200
2014-06-12 14:46:03.082186 7f9d7a962700 10 _finish_op 0x5875e00 seq 2546 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:03.082184 7f9d7715b700  5 -- op tracker -- , seq: 747516, time: 2014-06-12 14:46:03.082175, event: op_commit, op: osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.082200 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2733 crt=0'0 lcod 173'2732 mlcod 173'2732 active+clean] repop_all_committed: repop tid 236200 all committed 
2014-06-12 14:46:03.082211 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2732 active+clean] eval_repop repgather(0x13b43680 173'2734 rep_tid=236200 committed?=1 applied?=0 lock=0 op=osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.082229 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2732 active+clean] log_op_stats osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4 inb 835863 outb 0 rlat 0.000000 lat 0.023472
2014-06-12 14:46:03.082248 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2732 active+clean] publish_stats_to_osd 173:3631
2014-06-12 14:46:03.082259 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2732 active+clean]  sending commit on repgather(0x13b43680 173'2734 rep_tid=236200 committed?=1 applied?=0 lock=0 op=osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) 0x15cd6000
2014-06-12 14:46:03.082298 7f9d7715b700  5 -- op tracker -- , seq: 747516, time: 2014-06-12 14:46:03.082293, event: commit_sent, op: osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.082310 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2732 active+clean] op_applied: 236200
2014-06-12 14:46:03.082336 7f9d7795c700  5 -- op tracker -- , seq: 747516, time: 2014-06-12 14:46:03.082331, event: op_applied, op: osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.082344 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2732 active+clean] op_applied on primary on version 173'2734
2014-06-12 14:46:03.082353 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2732 active+clean] repop_all_applied: repop tid 236200 all applied 
2014-06-12 14:46:03.082360 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2732 active+clean] eval_repop repgather(0x13b43680 173'2734 rep_tid=236200 committed?=1 applied?=1 lock=0 op=osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.082374 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2732 active+clean] log_op_stats osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4 inb 835863 outb 0 rlat 0.000000 lat 0.023618
2014-06-12 14:46:03.082387 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2732 active+clean] publish_stats_to_osd 173:3632
2014-06-12 14:46:03.082398 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2732 active+clean]  requeue_ops 
2014-06-12 14:46:03.082410 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean]  removing repgather(0x13b43680 173'2734 rep_tid=236200 committed?=1 applied?=1 op=osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.082422 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean]    q front is repgather(0x13b43680 173'2734 rep_tid=236200 committed?=1 applied?=1 op=osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.082435 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] remove_repop repgather(0x13b43680 173'2734 rep_tid=236200 committed?=1 applied?=1 op=osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.082447 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean]  requeue_ops 
2014-06-12 14:46:03.082499 7f9d7a962700  5 -- op tracker -- , seq: 747516, time: 2014-06-12 14:46:03.082489, event: done, op: osd_op(client.4312.0:144982 rbd_data.10af6b8b4567.0000000000000efb [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~835584] 3.d61f2122 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.082536 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:46:03.082540 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.168160 7f9d56432700  5 -- op tracker -- , seq: 747518, time: 2014-06-12 14:46:03.168100, event: header_read, op: osd_sub_op_reply(client.4311.0:139600 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.168174 7f9d56432700  5 -- op tracker -- , seq: 747518, time: 2014-06-12 14:46:03.168105, event: throttled, op: osd_sub_op_reply(client.4311.0:139600 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.168183 7f9d56432700  5 -- op tracker -- , seq: 747518, time: 2014-06-12 14:46:03.168119, event: all_read, op: osd_sub_op_reply(client.4311.0:139600 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.168190 7f9d56432700  5 -- op tracker -- , seq: 747518, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4311.0:139600 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.168201 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4311.0:139600 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.168210 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:03.168220 7f9d56432700 15 osd.2 173 enqueue_op 0x5ee9900 prio 196 cost 0 latency 0.000119 osd_sub_op_reply(client.4311.0:139600 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.168246 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.168256 7f9d66bde700 10 osd.2 173 dequeue_op 0x5ee9900 prio 196 cost 0 latency 0.000156 osd_sub_op_reply(client.4311.0:139600 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2 pg pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]
2014-06-12 14:46:03.168282 7f9d66bde700  5 -- op tracker -- , seq: 747518, time: 2014-06-12 14:46:03.168278, event: reached_pg, op: osd_sub_op_reply(client.4311.0:139600 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.168290 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] handle_message: 0x5ee9900
2014-06-12 14:46:03.168310 7f9d66bde700  5 -- op tracker -- , seq: 747518, time: 2014-06-12 14:46:03.168306, event: started, op: osd_sub_op_reply(client.4311.0:139600 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.168316 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] sub_op_modify_reply: tid 235972 op  ack_type 4 from 26
2014-06-12 14:46:03.168341 7f9d66bde700  5 -- op tracker -- , seq: 746783, time: 2014-06-12 14:46:03.168332, event: sub_op_commit_rec, op: osd_op(client.4311.0:139600 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 86016~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.168349 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] repop_all_applied: repop tid 235972 all applied 
2014-06-12 14:46:03.168363 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] eval_repop repgather(0x31ee63c0 173'2659 rep_tid=235972 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139600 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 86016~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.168383 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]  sending ack on repgather(0x31ee63c0 173'2659 rep_tid=235972 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139600 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 86016~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) 0x45de780
2014-06-12 14:46:03.168429 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] repop_all_committed: repop tid 235972 all committed 
2014-06-12 14:46:03.168439 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2657 active+clean] eval_repop repgather(0x31ee63c0 173'2659 rep_tid=235972 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139600 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 86016~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.168458 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2657 active+clean] log_op_stats osd_op(client.4311.0:139600 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 86016~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 36.418882 lat 36.418912
2014-06-12 14:46:03.168480 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2657 active+clean] publish_stats_to_osd 173:3024
2014-06-12 14:46:03.168495 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2657 active+clean]  sending commit on repgather(0x31ee63c0 173'2659 rep_tid=235972 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139600 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 86016~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) 0xb954780
2014-06-12 14:46:03.168530 7f9d66bde700  5 -- op tracker -- , seq: 746783, time: 2014-06-12 14:46:03.168524, event: commit_sent, op: osd_op(client.4311.0:139600 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 86016~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.168537 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2657 active+clean]  requeue_ops 
2014-06-12 14:46:03.168548 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean]  removing repgather(0x31ee63c0 173'2659 rep_tid=235972 committed?=1 applied?=1 op=osd_op(client.4311.0:139600 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 86016~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.168564 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean]    q front is repgather(0x31ee63c0 173'2659 rep_tid=235972 committed?=1 applied?=1 op=osd_op(client.4311.0:139600 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 86016~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.168582 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean] remove_repop repgather(0x31ee63c0 173'2659 rep_tid=235972 committed?=1 applied?=1 op=osd_op(client.4311.0:139600 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 86016~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.168598 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean]  requeue_ops 
2014-06-12 14:46:03.168620 7f9d66bde700  5 -- op tracker -- , seq: 746783, time: 2014-06-12 14:46:03.168614, event: done, op: osd_op(client.4311.0:139600 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 86016~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.168642 7f9d66bde700 10 osd.2 173 dequeue_op 0x5ee9900 finish
2014-06-12 14:46:03.168652 7f9d66bde700  5 -- op tracker -- , seq: 747518, time: 2014-06-12 14:46:03.168648, event: done, op: osd_sub_op_reply(client.4311.0:139600 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.168663 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.168667 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.212719 7f9d5a16f700  5 -- op tracker -- , seq: 747519, time: 2014-06-12 14:46:03.212659, event: header_read, op: osd_sub_op_reply(client.4310.0:142390 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.212732 7f9d5a16f700  5 -- op tracker -- , seq: 747519, time: 2014-06-12 14:46:03.212661, event: throttled, op: osd_sub_op_reply(client.4310.0:142390 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.212745 7f9d5a16f700  5 -- op tracker -- , seq: 747519, time: 2014-06-12 14:46:03.212675, event: all_read, op: osd_sub_op_reply(client.4310.0:142390 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.212753 7f9d5a16f700  5 -- op tracker -- , seq: 747519, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142390 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.212758 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142390 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.212764 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:03.212773 7f9d5a16f700 15 osd.2 173 enqueue_op 0x22b3b200 prio 196 cost 0 latency 0.000113 osd_sub_op_reply(client.4310.0:142390 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.212803 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.212812 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x22b3b200 prio 196 cost 0 latency 0.000153 osd_sub_op_reply(client.4310.0:142390 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2 pg pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]
2014-06-12 14:46:03.212832 7f9d6a3e5700  5 -- op tracker -- , seq: 747519, time: 2014-06-12 14:46:03.212829, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142390 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.212846 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] handle_message: 0x22b3b200
2014-06-12 14:46:03.212862 7f9d6a3e5700  5 -- op tracker -- , seq: 747519, time: 2014-06-12 14:46:03.212859, event: started, op: osd_sub_op_reply(client.4310.0:142390 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.212868 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] sub_op_modify_reply: tid 235995 op  ack_type 4 from 24
2014-06-12 14:46:03.212887 7f9d6a3e5700  5 -- op tracker -- , seq: 746901, time: 2014-06-12 14:46:03.212880, event: sub_op_commit_rec, op: osd_op(client.4310.0:142390 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1548288] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.212894 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] repop_all_applied: repop tid 235995 all applied 
2014-06-12 14:46:03.212904 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] eval_repop repgather(0x15e569c0 173'2702 rep_tid=235995 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142390 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1548288] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.212919 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]  sending ack on repgather(0x15e569c0 173'2702 rep_tid=235995 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142390 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1548288] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) 0x32eefb80
2014-06-12 14:46:03.212947 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] repop_all_committed: repop tid 235995 all committed 
2014-06-12 14:46:03.212957 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2700 active+clean] eval_repop repgather(0x15e569c0 173'2702 rep_tid=235995 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142390 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1548288] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.212973 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2700 active+clean] log_op_stats osd_op(client.4310.0:142390 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1548288] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 32.970078 lat 32.970104
2014-06-12 14:46:03.212989 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2700 active+clean] publish_stats_to_osd 173:2885
2014-06-12 14:46:03.213012 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2700 active+clean]  sending commit on repgather(0x15e569c0 173'2702 rep_tid=235995 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142390 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1548288] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) 0x29de7b80
2014-06-12 14:46:03.213040 7f9d6a3e5700  5 -- op tracker -- , seq: 746901, time: 2014-06-12 14:46:03.213035, event: commit_sent, op: osd_op(client.4310.0:142390 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1548288] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.213047 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2700 active+clean]  requeue_ops 
2014-06-12 14:46:03.213057 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]  removing repgather(0x15e569c0 173'2702 rep_tid=235995 committed?=1 applied?=1 op=osd_op(client.4310.0:142390 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1548288] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.213069 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]    q front is repgather(0x15e569c0 173'2702 rep_tid=235995 committed?=1 applied?=1 op=osd_op(client.4310.0:142390 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1548288] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.213082 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] remove_repop repgather(0x15e569c0 173'2702 rep_tid=235995 committed?=1 applied?=1 op=osd_op(client.4310.0:142390 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1548288] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.213094 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]  requeue_ops 
2014-06-12 14:46:03.213112 7f9d6a3e5700  5 -- op tracker -- , seq: 746901, time: 2014-06-12 14:46:03.213108, event: done, op: osd_op(client.4310.0:142390 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1548288] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.213130 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x22b3b200 finish
2014-06-12 14:46:03.213136 7f9d6a3e5700  5 -- op tracker -- , seq: 747519, time: 2014-06-12 14:46:03.213133, event: done, op: osd_sub_op_reply(client.4310.0:142390 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.213144 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.213147 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.241825 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:03.242044 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:03.387842 7f9d4ec14700  5 -- op tracker -- , seq: 747520, time: 2014-06-12 14:46:03.387633, event: header_read, op: osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.387859 7f9d4ec14700  5 -- op tracker -- , seq: 747520, time: 2014-06-12 14:46:03.387637, event: throttled, op: osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.387869 7f9d4ec14700  5 -- op tracker -- , seq: 747520, time: 2014-06-12 14:46:03.387790, event: all_read, op: osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.387879 7f9d4ec14700  5 -- op tracker -- , seq: 747520, time: 0.000000, event: dispatched, op: osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.387888 7f9d4ec14700 20 osd.2 173 should_share_map client.4282 172.16.96.220:0/1241359 173
2014-06-12 14:46:03.387917 7f9d4ec14700 15 osd.2 173 enqueue_op 0x23e08a00 prio 63 cost 45056 latency 0.000284 osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.387956 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.387967 7f9d63bd8700 10 osd.2 173 dequeue_op 0x23e08a00 prio 63 cost 45056 latency 0.000334 osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean]
2014-06-12 14:46:03.387996 7f9d63bd8700  5 -- op tracker -- , seq: 747520, time: 2014-06-12 14:46:03.387991, event: reached_pg, op: osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.388006 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.388021 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] handle_message: 0x23e08a00
2014-06-12 14:46:03.388031 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] do_op osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.388056 7f9d63bd8700 15 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 '_'
2014-06-12 14:46:03.401720 7f9d56432700  5 -- op tracker -- , seq: 747521, time: 2014-06-12 14:46:03.400015, event: header_read, op: osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.401733 7f9d56432700  5 -- op tracker -- , seq: 747521, time: 2014-06-12 14:46:03.400018, event: throttled, op: osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.401743 7f9d56432700  5 -- op tracker -- , seq: 747521, time: 2014-06-12 14:46:03.401669, event: all_read, op: osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.401752 7f9d56432700  5 -- op tracker -- , seq: 747521, time: 0.000000, event: dispatched, op: osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.401757 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:03.401764 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:03.401773 7f9d56432700 15 osd.2 173 enqueue_op 0x5ee8800 prio 127 cost 516899 latency 0.001758 osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.401970 7f9d63bd8700 10 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 '_' = 0
2014-06-12 14:46:03.401988 7f9d63bd8700 15 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 'snapset'
2014-06-12 14:46:03.402124 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:03.402143 7f9d63bd8700 10 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 'snapset' = 0
2014-06-12 14:46:03.402131 7f9d653db700 10 osd.2 173 dequeue_op 0x5ee8800 prio 127 cost 516899 latency 0.002116 osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2468 (0'0,173'2468] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2467 active]
2014-06-12 14:46:03.402152 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] populate_obc_watchers 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:03.402159 7f9d653db700  5 -- op tracker -- , seq: 747521, time: 2014-06-12 14:46:03.402155, event: reached_pg, op: osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.402165 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:03.402174 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] get_object_context: creating obc from disk: 0x2870cb00
2014-06-12 14:46:03.402168 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2468 (0'0,173'2468] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2467 active] handle_message: 0x5ee8800
2014-06-12 14:46:03.402185 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2468 (0'0,173'2468] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2467 active] sub_op_modify trans 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 v 173'2469 (transaction) 174
2014-06-12 14:46:03.402183 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] get_object_context: 0x2870cb00 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 rwstate(none n=0 w=0) oi: 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3(173'1004 client.4282.0:40175 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1004) ssc: 0x1e15e380 snapset: 0=[]:[]+head
2014-06-12 14:46:03.402201 7f9d653db700  5 -- op tracker -- , seq: 747521, time: 2014-06-12 14:46:03.402197, event: started, op: osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.402197 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] find_object_context 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 @head oi=71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3(173'1004 client.4282.0:40175 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1004)
2014-06-12 14:46:03.402220 7f9d63bd8700  5 -- op tracker -- , seq: 747520, time: 2014-06-12 14:46:03.402214, event: started, op: osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.402215 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2468 (0'0,173'2468] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2467 active] append_log log((0'0,173'2468], crt=0'0) [173'2469 (173'2466) modify   4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 by client.4280.0:139526 2014-06-12 08:43:26.683824]
2014-06-12 14:46:03.402229 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] execute_ctx 0x2a83200
2014-06-12 14:46:03.402245 7f9d63bd8700  5 -- op tracker -- , seq: 747520, time: 2014-06-12 14:46:03.402240, event: started, op: osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.402237 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2469 (0'0,173'2469] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2467 active] add_log_entry 173'2469 (173'2466) modify   4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 by client.4280.0:139526 2014-06-12 08:43:26.683824
2014-06-12 14:46:03.402251 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] do_op 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] ov 173'1004 av 173'3149 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.402259 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2469 (0'0,173'2469] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2467 active] append_log  adding 1 keys
2014-06-12 14:46:03.402263 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] do_osd_op 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056]
2014-06-12 14:46:03.402273 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.402289 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2469, trimmed: 
2014-06-12 14:46:03.402285 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] do_osd_op  write 0~45056
2014-06-12 14:46:03.402297 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] do_osd_op_effects on session 0x62d5000
2014-06-12 14:46:03.402309 7f9d653db700  5 -- op tracker -- , seq: 747521, time: 2014-06-12 14:46:03.402305, event: started, op: osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.402307 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] make_writeable 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 snapset=0x1e15e3b8  snapc=0=[]
2014-06-12 14:46:03.402318 7f9d653db700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:03.402321 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.402322 7f9d653db700  5 queue_op 0x1ebb05a0 seq 2282 osr(3.2b 0x276cd20) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:46:03.402318 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] make_writeable 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.402327 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] finish_ctx 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 0x2a83200 op modify  
2014-06-12 14:46:03.402328 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2469 (0'0,173'2469] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2467 active] do_sub_op osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.402342 7f9d653db700 10 osd.2 173 dequeue_op 0x5ee8800 finish
2014-06-12 14:46:03.402345 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:03.402347 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.402338 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean]  set mtime to 2014-06-12 08:45:05.768671
2014-06-12 14:46:03.402357 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean]  final snapset 0=[]:[]+head in 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:03.402376 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.402386 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] calc_trim_to 79'147 -> 79'148
2014-06-12 14:46:03.402396 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] new_repop rep_tid 236201 on osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.402411 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] issue_repop rep_tid 236201 o 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:03.402431 7f9d63bd8700  5 -- op tracker -- , seq: 747520, time: 2014-06-12 14:46:03.402426, event: waiting for subops from 20, op: osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.402449 7f9d63bd8700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:03.402470 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3148 (79'147,173'3148] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] append_log log((79'147,173'3148], crt=0'0) [173'3149 (173'1004) modify   71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 by client.4282.0:145272 2014-06-12 08:45:05.768671]
2014-06-12 14:46:03.402493 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'147,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3148 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] add_log_entry 173'3149 (173'1004) modify   71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 by client.4282.0:145272 2014-06-12 08:45:05.768671
2014-06-12 14:46:03.402512 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'147,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3148 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.402524 7f9d63bd8700 10 trim log((79'147,173'3149], crt=0'0) to 79'148
2014-06-12 14:46:03.402555 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3149, trimmed: 79'148
2014-06-12 14:46:03.402574 7f9d63bd8700  5 queue_transactions existing osr(3.63b 0x276dc70)/0x276dc70
2014-06-12 14:46:03.402577 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.402579 7f9d63bd8700  5 queue_op 0x142a8300 seq 2852 osr(3.63b 0x276dc70) 47384 bytes   (queue has 2 ops and 565764 bytes)
2014-06-12 14:46:03.402585 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3148 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] eval_repop repgather(0x10192300 173'3149 rep_tid=236201 committed?=0 applied?=0 lock=0 op=osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.402602 7f9d63bd8700 10 osd.2 173 dequeue_op 0x23e08a00 finish
2014-06-12 14:46:03.402606 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.402609 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.403207 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:46:03.403218 7f9d7f16b700  5 _do_op 0x1ebb05a0 seq 2282 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:46:03.403223 7f9d7f16b700 10 _do_transaction on 0x63adb30
2014-06-12 14:46:03.403228 7f9d7f16b700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.403303 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5421440 (2 active)
2014-06-12 14:46:03.403308 7f9d7895e700  5 _do_op 0x142a8300 seq 2852 osr(3.63b 0x276dc70)/0x276dc70 start
2014-06-12 14:46:03.403312 7f9d7895e700 10 _do_transaction on 0x104509c0
2014-06-12 14:46:03.403317 7f9d7895e700 15 _omap_setkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:03.403340 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.403441 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.403505 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.403524 7f9d7f16b700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.403530 7f9d7f16b700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:46:03.403535 7f9d7f16b700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.403552 7f9d7f16b700 15 _write 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 1933312~516096
2014-06-12 14:46:03.403793 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:03.403809 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.403886 7f9d7f16b700 10 lookup_strip_header done  cid 3.2b_head oid 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:46:03.403992 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:03.405872 7f9d5703e700  5 -- op tracker -- , seq: 747522, time: 2014-06-12 14:46:03.405830, event: header_read, op: osd_sub_op_reply(client.4282.0:145272 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.405884 7f9d5703e700  5 -- op tracker -- , seq: 747522, time: 2014-06-12 14:46:03.405831, event: throttled, op: osd_sub_op_reply(client.4282.0:145272 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.405893 7f9d5703e700  5 -- op tracker -- , seq: 747522, time: 2014-06-12 14:46:03.405842, event: all_read, op: osd_sub_op_reply(client.4282.0:145272 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.405900 7f9d5703e700  5 -- op tracker -- , seq: 747522, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4282.0:145272 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.405905 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4282.0:145272 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.405907 7f9d7f16b700 10 _generic_write 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 1933312~516096 = 0
2014-06-12 14:46:03.405911 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:03.405918 7f9d5703e700 15 osd.2 173 enqueue_op 0x28fd8e00 prio 196 cost 0 latency 0.000088 osd_sub_op_reply(client.4282.0:145272 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.405945 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.405957 7f9d7f16b700 15 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:46:03.405952 7f9d69be4700 10 osd.2 173 dequeue_op 0x28fd8e00 prio 196 cost 0 latency 0.000122 osd_sub_op_reply(client.4282.0:145272 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2 pg pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3148 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean]
2014-06-12 14:46:03.405971 7f9d7f16b700 10 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 = 0
2014-06-12 14:46:03.405972 7f9d69be4700  5 -- op tracker -- , seq: 747522, time: 2014-06-12 14:46:03.405969, event: reached_pg, op: osd_sub_op_reply(client.4282.0:145272 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.405977 7f9d7f16b700 15 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:46:03.405985 7f9d7f16b700 10 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 = 0
2014-06-12 14:46:03.405980 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3148 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] handle_message: 0x28fd8e00
2014-06-12 14:46:03.405996 7f9d69be4700  5 -- op tracker -- , seq: 747522, time: 2014-06-12 14:46:03.405992, event: started, op: osd_sub_op_reply(client.4282.0:145272 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.406002 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3148 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] sub_op_modify_reply: tid 236201 op  ack_type 4 from 20
2014-06-12 14:46:03.406023 7f9d69be4700  5 -- op tracker -- , seq: 747520, time: 2014-06-12 14:46:03.406015, event: sub_op_commit_rec, op: osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.406031 7f9d69be4700 10 osd.2 173 dequeue_op 0x28fd8e00 finish
2014-06-12 14:46:03.406037 7f9d69be4700  5 -- op tracker -- , seq: 747522, time: 2014-06-12 14:46:03.406034, event: done, op: osd_sub_op_reply(client.4282.0:145272 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.406051 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.406054 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.406182 7f9d7f16b700 20 set_header setting 15771 cid 3.2b_head oid 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 parent seq 0
2014-06-12 14:46:03.406263 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.406278 7f9d7f16b700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:46:03.406287 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:03.408708 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.408734 7f9d7895e700 15 _omap_rmkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:03.408741 7f9d7895e700 10 _omap_rmkeys meta/1fe0c16b/pglog_3.63b/0//-1 r = 0
2014-06-12 14:46:03.408748 7f9d7895e700 15 _omap_setkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:03.408761 7f9d7895e700 15 _write 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 0~45056
2014-06-12 14:46:03.408762 7f9d7f16b700 10 _do_op 0x1ebb05a0 seq 2282 r = 0, finisher 0x5feefc0 0
2014-06-12 14:46:03.408776 7f9d7f16b700 10 _finish_op 0x1ebb05a0 seq 2282 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:03.408792 7f9d7715b700  5 -- op tracker -- , seq: 747521, time: 2014-06-12 14:46:03.408783, event: commit_sent, op: osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.408793 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (1 active)
2014-06-12 14:46:03.408804 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.408800 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2469 (0'0,173'2469] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2467 active] sub_op_modify_commit on op osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:03.408828 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:03.408891 7f9d7895e700 10 lookup_strip_header done  cid 3.63b_head oid 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:03.408908 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:03.408911 7f9d7795c700  5 -- op tracker -- , seq: 747521, time: 2014-06-12 14:46:03.408904, event: sub_op_applied, op: osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.408920 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2469 (0'0,173'2469] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2468 active] sub_op_modify_applied on 0x63ada40 op osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.408936 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2469 (0'0,173'2469] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2468 active] op_applied on primary on version 173'2469
2014-06-12 14:46:03.408947 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2469 (0'0,173'2469] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2468 active] op_applied on replica on version 173'2469
2014-06-12 14:46:03.408977 7f9d7795c700  5 -- op tracker -- , seq: 747521, time: 2014-06-12 14:46:03.408973, event: done, op: osd_sub_op(client.4280.0:139526 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.409093 7f9d7895e700 10 _generic_write 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 0~45056 = 0
2014-06-12 14:46:03.409106 7f9d7895e700 15 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:03.409115 7f9d7895e700 10 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 = 0
2014-06-12 14:46:03.409120 7f9d7895e700 15 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:03.409126 7f9d7895e700 10 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 = 0
2014-06-12 14:46:03.409235 7f9d7895e700 20 set_header setting 18103 cid 3.63b_head oid 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 parent seq 0
2014-06-12 14:46:03.409254 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.409265 7f9d7895e700 20 set_header setting 2110 cid meta oid 1fe0c16b/pglog_3.63b/0//-1 parent seq 0
2014-06-12 14:46:03.409273 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:03.409481 7f9d7895e700 10 _do_op 0x142a8300 seq 2852 r = 0, finisher 0x54a52e0 0xbafdb40
2014-06-12 14:46:03.409490 7f9d7895e700 10 _finish_op 0x142a8300 seq 2852 osr(3.63b 0x276dc70)/0x276dc70
2014-06-12 14:46:03.409497 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5421440 (0 active)
2014-06-12 14:46:03.409500 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.409494 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3148 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] op_commit: 236201
2014-06-12 14:46:03.409520 7f9d7715b700  5 -- op tracker -- , seq: 747520, time: 2014-06-12 14:46:03.409511, event: op_commit, op: osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.409529 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3148 crt=0'0 lcod 173'3147 mlcod 173'3147 active+clean] repop_all_committed: repop tid 236201 all committed 
2014-06-12 14:46:03.409540 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3147 active+clean] eval_repop repgather(0x10192300 173'3149 rep_tid=236201 committed?=1 applied?=0 lock=0 op=osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.409558 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3147 active+clean] log_op_stats osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 inb 45335 outb 0 rlat 0.000000 lat 0.021924
2014-06-12 14:46:03.409578 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3147 active+clean] publish_stats_to_osd 173:4109
2014-06-12 14:46:03.409592 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3147 active+clean]  sending commit on repgather(0x10192300 173'3149 rep_tid=236201 committed?=1 applied?=0 lock=0 op=osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) 0x239e2c80
2014-06-12 14:46:03.409643 7f9d7715b700  5 -- op tracker -- , seq: 747520, time: 2014-06-12 14:46:03.409637, event: commit_sent, op: osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.409688 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3147 active+clean] op_applied: 236201
2014-06-12 14:46:03.409706 7f9d7795c700  5 -- op tracker -- , seq: 747520, time: 2014-06-12 14:46:03.409701, event: op_applied, op: osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.409714 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3147 active+clean] op_applied on primary on version 173'3149
2014-06-12 14:46:03.409725 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3147 active+clean] repop_all_applied: repop tid 236201 all applied 
2014-06-12 14:46:03.409734 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3147 active+clean] eval_repop repgather(0x10192300 173'3149 rep_tid=236201 committed?=1 applied?=1 lock=0 op=osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.409749 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3147 active+clean] log_op_stats osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 inb 45335 outb 0 rlat 0.000000 lat 0.022115
2014-06-12 14:46:03.409764 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3147 active+clean] publish_stats_to_osd 173:4110
2014-06-12 14:46:03.409775 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3147 active+clean]  requeue_ops 
2014-06-12 14:46:03.409785 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean]  removing repgather(0x10192300 173'3149 rep_tid=236201 committed?=1 applied?=1 op=osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.409799 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean]    q front is repgather(0x10192300 173'3149 rep_tid=236201 committed?=1 applied?=1 op=osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.409823 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] remove_repop repgather(0x10192300 173'3149 rep_tid=236201 committed?=1 applied?=1 op=osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.409837 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean]  requeue_ops 
2014-06-12 14:46:03.409863 7f9d7795c700  5 -- op tracker -- , seq: 747520, time: 2014-06-12 14:46:03.409858, event: done, op: osd_op(client.4282.0:145272 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.449035 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:03.449043 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:03.449045 7f9d74bfa700 20 osd.2 173 _dispatch 0x587ec40 pg_stats_ack(5 pgs tid 4846) v1
2014-06-12 14:46:03.449047 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:46:03.449056 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:03.449058 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:03.465915 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:03.465944 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:03.465949 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:03.465951 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:03.465955 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 37; oldest is 36.669151 seconds old
2014-06-12 14:46:03.465979 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:03.490812 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:03.491203 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:03.520212 7f9d56432700  5 -- op tracker -- , seq: 747523, time: 2014-06-12 14:46:03.520147, event: header_read, op: osd_sub_op_reply(client.4311.0:139601 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.520227 7f9d56432700  5 -- op tracker -- , seq: 747523, time: 2014-06-12 14:46:03.520150, event: throttled, op: osd_sub_op_reply(client.4311.0:139601 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.520251 7f9d56432700  5 -- op tracker -- , seq: 747523, time: 2014-06-12 14:46:03.520165, event: all_read, op: osd_sub_op_reply(client.4311.0:139601 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.520260 7f9d56432700  5 -- op tracker -- , seq: 747523, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4311.0:139601 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.520268 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4311.0:139601 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.520276 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:03.520288 7f9d56432700 15 osd.2 173 enqueue_op 0x5ee8900 prio 196 cost 0 latency 0.000140 osd_sub_op_reply(client.4311.0:139601 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.520321 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.520331 7f9d633d7700 10 osd.2 173 dequeue_op 0x5ee8900 prio 196 cost 0 latency 0.000184 osd_sub_op_reply(client.4311.0:139601 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2 pg pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean]
2014-06-12 14:46:03.520361 7f9d633d7700  5 -- op tracker -- , seq: 747523, time: 2014-06-12 14:46:03.520357, event: reached_pg, op: osd_sub_op_reply(client.4311.0:139601 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.520371 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean] handle_message: 0x5ee8900
2014-06-12 14:46:03.520391 7f9d633d7700  5 -- op tracker -- , seq: 747523, time: 2014-06-12 14:46:03.520387, event: started, op: osd_sub_op_reply(client.4311.0:139601 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.520398 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean] sub_op_modify_reply: tid 235973 op  ack_type 4 from 26
2014-06-12 14:46:03.520427 7f9d633d7700  5 -- op tracker -- , seq: 746784, time: 2014-06-12 14:46:03.520417, event: sub_op_commit_rec, op: osd_op(client.4311.0:139601 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1634304~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.520437 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean] repop_all_applied: repop tid 235973 all applied 
2014-06-12 14:46:03.520448 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean] eval_repop repgather(0x1a445d40 173'2660 rep_tid=235973 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139601 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1634304~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.520466 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean]  sending ack on repgather(0x1a445d40 173'2660 rep_tid=235973 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139601 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1634304~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) 0x11458f00
2014-06-12 14:46:03.520503 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2659 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean] repop_all_committed: repop tid 235973 all committed 
2014-06-12 14:46:03.520516 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2658 active+clean] eval_repop repgather(0x1a445d40 173'2660 rep_tid=235973 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139601 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1634304~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.520534 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2658 active+clean] log_op_stats osd_op(client.4311.0:139601 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1634304~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 36.723698 lat 36.723730
2014-06-12 14:46:03.520552 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2658 active+clean] publish_stats_to_osd 173:3025
2014-06-12 14:46:03.520566 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2658 active+clean]  sending commit on repgather(0x1a445d40 173'2660 rep_tid=235973 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139601 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1634304~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) 0xae7e500
2014-06-12 14:46:03.520601 7f9d633d7700  5 -- op tracker -- , seq: 746784, time: 2014-06-12 14:46:03.520595, event: commit_sent, op: osd_op(client.4311.0:139601 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1634304~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.520610 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2658 active+clean]  requeue_ops 
2014-06-12 14:46:03.520620 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2659 active+clean]  removing repgather(0x1a445d40 173'2660 rep_tid=235973 committed?=1 applied?=1 op=osd_op(client.4311.0:139601 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1634304~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.520635 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2659 active+clean]    q front is repgather(0x1a445d40 173'2660 rep_tid=235973 committed?=1 applied?=1 op=osd_op(client.4311.0:139601 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1634304~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.520651 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2659 active+clean] remove_repop repgather(0x1a445d40 173'2660 rep_tid=235973 committed?=1 applied?=1 op=osd_op(client.4311.0:139601 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1634304~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.520666 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2659 active+clean]  requeue_ops 
2014-06-12 14:46:03.520686 7f9d633d7700  5 -- op tracker -- , seq: 746784, time: 2014-06-12 14:46:03.520680, event: done, op: osd_op(client.4311.0:139601 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 1634304~1548288] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.520706 7f9d633d7700 10 osd.2 173 dequeue_op 0x5ee8900 finish
2014-06-12 14:46:03.520716 7f9d633d7700  5 -- op tracker -- , seq: 747523, time: 2014-06-12 14:46:03.520713, event: done, op: osd_sub_op_reply(client.4311.0:139601 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.520727 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.520730 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.536081 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:03.540461 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:03.553740 7f9d3b028700  5 -- op tracker -- , seq: 747524, time: 2014-06-12 14:46:03.552975, event: header_read, op: osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.553757 7f9d3b028700  5 -- op tracker -- , seq: 747524, time: 2014-06-12 14:46:03.552980, event: throttled, op: osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.553786 7f9d3b028700  5 -- op tracker -- , seq: 747524, time: 2014-06-12 14:46:03.553681, event: all_read, op: osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.553794 7f9d3b028700  5 -- op tracker -- , seq: 747524, time: 0.000000, event: dispatched, op: osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.553802 7f9d3b028700 20 osd.2 173 should_share_map client.4308 172.16.96.220:0/1243963 173
2014-06-12 14:46:03.553829 7f9d3b028700 15 osd.2 173 enqueue_op 0x9fad900 prio 63 cost 258048 latency 0.000854 osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.553862 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.553872 7f9d643d9700 10 osd.2 173 dequeue_op 0x9fad900 prio 63 cost 258048 latency 0.000896 osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean]
2014-06-12 14:46:03.553894 7f9d643d9700  5 -- op tracker -- , seq: 747524, time: 2014-06-12 14:46:03.553890, event: reached_pg, op: osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.553901 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.553912 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] handle_message: 0x9fad900
2014-06-12 14:46:03.553919 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] do_op osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.553938 7f9d643d9700 15 getattr 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 '_'
2014-06-12 14:46:03.558137 7f9d56432700  5 -- op tracker -- , seq: 747525, time: 2014-06-12 14:46:03.558080, event: header_read, op: osd_sub_op_reply(client.4311.0:139602 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.558148 7f9d56432700  5 -- op tracker -- , seq: 747525, time: 2014-06-12 14:46:03.558084, event: throttled, op: osd_sub_op_reply(client.4311.0:139602 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.558165 7f9d56432700  5 -- op tracker -- , seq: 747525, time: 2014-06-12 14:46:03.558100, event: all_read, op: osd_sub_op_reply(client.4311.0:139602 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.558169 7f9d56432700  5 -- op tracker -- , seq: 747525, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4311.0:139602 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.558174 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4311.0:139602 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.558179 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:03.558186 7f9d56432700 15 osd.2 173 enqueue_op 0x28fd9600 prio 196 cost 0 latency 0.000106 osd_sub_op_reply(client.4311.0:139602 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.558213 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:03.558223 7f9d56432700  5 -- op tracker -- , seq: 747526, time: 2014-06-12 14:46:03.558205, event: header_read, op: osd_sub_op_reply(client.4311.0:139603 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.558237 7f9d56432700  5 -- op tracker -- , seq: 747526, time: 2014-06-12 14:46:03.558205, event: throttled, op: osd_sub_op_reply(client.4311.0:139603 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.558222 7f9d623d5700 10 osd.2 173 dequeue_op 0x28fd9600 prio 196 cost 0 latency 0.000142 osd_sub_op_reply(client.4311.0:139602 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2 pg pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2659 active+clean]
2014-06-12 14:46:03.558242 7f9d56432700  5 -- op tracker -- , seq: 747526, time: 2014-06-12 14:46:03.558211, event: all_read, op: osd_sub_op_reply(client.4311.0:139603 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.558247 7f9d56432700  5 -- op tracker -- , seq: 747526, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4311.0:139603 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.558251 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4311.0:139603 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.558254 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:03.558257 7f9d56432700 15 osd.2 173 enqueue_op 0x20e89800 prio 196 cost 0 latency 0.000052 osd_sub_op_reply(client.4311.0:139603 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.558257 7f9d623d5700  5 -- op tracker -- , seq: 747525, time: 2014-06-12 14:46:03.558244, event: reached_pg, op: osd_sub_op_reply(client.4311.0:139602 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.558276 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active)
2014-06-12 14:46:03.558267 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2659 active+clean] handle_message: 0x28fd9600
2014-06-12 14:46:03.558284 7f9d623d5700  5 -- op tracker -- , seq: 747525, time: 2014-06-12 14:46:03.558281, event: started, op: osd_sub_op_reply(client.4311.0:139602 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.558290 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2659 active+clean] sub_op_modify_reply: tid 235981 op  ack_type 4 from 26
2014-06-12 14:46:03.558315 7f9d623d5700  5 -- op tracker -- , seq: 746820, time: 2014-06-12 14:46:03.558305, event: sub_op_commit_rec, op: osd_op(client.4311.0:139602 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3182592~516096] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.558322 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2659 active+clean] repop_all_applied: repop tid 235981 all applied 
2014-06-12 14:46:03.558334 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2659 active+clean] eval_repop repgather(0x2a12f200 173'2661 rep_tid=235981 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139602 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3182592~516096] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.558367 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2659 active+clean]  sending ack on repgather(0x2a12f200 173'2661 rep_tid=235981 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139602 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3182592~516096] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) 0x30c41180
2014-06-12 14:46:03.558406 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2660 crt=0'0 lcod 173'2659 mlcod 173'2659 active+clean] repop_all_committed: repop tid 235981 all committed 
2014-06-12 14:46:03.558419 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2659 active+clean] eval_repop repgather(0x2a12f200 173'2661 rep_tid=235981 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139602 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3182592~516096] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.558440 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2659 active+clean] log_op_stats osd_op(client.4311.0:139602 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3182592~516096] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 35.053564 lat 35.053598
2014-06-12 14:46:03.558460 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2659 active+clean] publish_stats_to_osd 173:3026
2014-06-12 14:46:03.558476 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2659 active+clean]  sending commit on repgather(0x2a12f200 173'2661 rep_tid=235981 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139602 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3182592~516096] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) 0x32df1900
2014-06-12 14:46:03.558514 7f9d623d5700  5 -- op tracker -- , seq: 746820, time: 2014-06-12 14:46:03.558508, event: commit_sent, op: osd_op(client.4311.0:139602 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3182592~516096] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.558524 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2659 active+clean]  requeue_ops 
2014-06-12 14:46:03.558536 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2660 active+clean]  removing repgather(0x2a12f200 173'2661 rep_tid=235981 committed?=1 applied?=1 op=osd_op(client.4311.0:139602 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3182592~516096] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.558553 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2660 active+clean]    q front is repgather(0x2a12f200 173'2661 rep_tid=235981 committed?=1 applied?=1 op=osd_op(client.4311.0:139602 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3182592~516096] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.558572 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2660 active+clean] remove_repop repgather(0x2a12f200 173'2661 rep_tid=235981 committed?=1 applied?=1 op=osd_op(client.4311.0:139602 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3182592~516096] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.558589 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2660 active+clean]  requeue_ops 
2014-06-12 14:46:03.558611 7f9d623d5700  5 -- op tracker -- , seq: 746820, time: 2014-06-12 14:46:03.558605, event: done, op: osd_op(client.4311.0:139602 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3182592~516096] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.558634 7f9d623d5700 10 osd.2 173 dequeue_op 0x28fd9600 finish
2014-06-12 14:46:03.558647 7f9d623d5700  5 -- op tracker -- , seq: 747525, time: 2014-06-12 14:46:03.558643, event: done, op: osd_sub_op_reply(client.4311.0:139602 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.558646 7f9d68be2700 10 osd.2 173 dequeue_op 0x20e89800 prio 196 cost 0 latency 0.000441 osd_sub_op_reply(client.4311.0:139603 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0) v2 pg pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2660 active+clean]
2014-06-12 14:46:03.558660 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active)
2014-06-12 14:46:03.558666 7f9d68be2700  5 -- op tracker -- , seq: 747526, time: 2014-06-12 14:46:03.558662, event: reached_pg, op: osd_sub_op_reply(client.4311.0:139603 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.558672 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.558674 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2660 active+clean] handle_message: 0x20e89800
2014-06-12 14:46:03.558689 7f9d68be2700  5 -- op tracker -- , seq: 747526, time: 2014-06-12 14:46:03.558686, event: started, op: osd_sub_op_reply(client.4311.0:139603 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.558694 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2660 active+clean] sub_op_modify_reply: tid 235982 op  ack_type 4 from 26
2014-06-12 14:46:03.558712 7f9d68be2700  5 -- op tracker -- , seq: 746824, time: 2014-06-12 14:46:03.558706, event: sub_op_commit_rec, op: osd_op(client.4311.0:139603 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3698688~495616] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.558726 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2660 active+clean] repop_all_applied: repop tid 235982 all applied 
2014-06-12 14:46:03.558736 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2660 active+clean] eval_repop repgather(0x234ce40 173'2662 rep_tid=235982 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139603 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3698688~495616] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.558753 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2660 active+clean]  sending ack on repgather(0x234ce40 173'2662 rep_tid=235982 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139603 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3698688~495616] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) 0x20237680
2014-06-12 14:46:03.558784 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2661 crt=0'0 lcod 173'2660 mlcod 173'2660 active+clean] repop_all_committed: repop tid 235982 all committed 
2014-06-12 14:46:03.558796 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2660 active+clean] eval_repop repgather(0x234ce40 173'2662 rep_tid=235982 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139603 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3698688~495616] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.558813 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2660 active+clean] log_op_stats osd_op(client.4311.0:139603 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3698688~495616] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4 inb 495895 outb 0 rlat 34.897868 lat 34.897898
2014-06-12 14:46:03.558830 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2660 active+clean] publish_stats_to_osd 173:3027
2014-06-12 14:46:03.558841 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2660 active+clean]  sending commit on repgather(0x234ce40 173'2662 rep_tid=235982 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139603 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3698688~495616] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4) 0x56cb180
2014-06-12 14:46:03.558872 7f9d68be2700  5 -- op tracker -- , seq: 746824, time: 2014-06-12 14:46:03.558867, event: commit_sent, op: osd_op(client.4311.0:139603 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3698688~495616] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.558881 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2660 active+clean]  requeue_ops 
2014-06-12 14:46:03.558891 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2661 active+clean]  removing repgather(0x234ce40 173'2662 rep_tid=235982 committed?=1 applied?=1 op=osd_op(client.4311.0:139603 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3698688~495616] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.558906 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2661 active+clean]    q front is repgather(0x234ce40 173'2662 rep_tid=235982 committed?=1 applied?=1 op=osd_op(client.4311.0:139603 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3698688~495616] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.558922 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2661 active+clean] remove_repop repgather(0x234ce40 173'2662 rep_tid=235982 committed?=1 applied?=1 op=osd_op(client.4311.0:139603 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3698688~495616] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.558937 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2661 active+clean]  requeue_ops 
2014-06-12 14:46:03.558962 7f9d68be2700  5 -- op tracker -- , seq: 746824, time: 2014-06-12 14:46:03.558957, event: done, op: osd_op(client.4311.0:139603 rbd_data.10ae6b8b4567.0000000000000ec1 [set-alloc-hint object_size 4194304 write_size 4194304,write 3698688~495616] 3.abbf4b66 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.558979 7f9d68be2700 10 osd.2 173 dequeue_op 0x20e89800 finish
2014-06-12 14:46:03.558990 7f9d68be2700  5 -- op tracker -- , seq: 747526, time: 2014-06-12 14:46:03.558986, event: done, op: osd_sub_op_reply(client.4311.0:139603 3.366 abbf4b66/rbd_data.10ae6b8b4567.0000000000000ec1/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.559003 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:03.559006 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.569759 7f9d643d9700 10 getattr 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 '_' = 0
2014-06-12 14:46:03.569787 7f9d643d9700 15 getattr 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 'snapset'
2014-06-12 14:46:03.569964 7f9d643d9700 10 getattr 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 'snapset' = 0
2014-06-12 14:46:03.569976 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] populate_obc_watchers 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.569992 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.570002 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] get_object_context: creating obc from disk: 0x15574000
2014-06-12 14:46:03.570011 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] get_object_context: 0x15574000 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 rwstate(none n=0 w=0) oi: 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'771 client.4308.0:37850 wrlock_by=unknown.0.0:0 dirty s 4194304 uv771) ssc: 0x407f420 snapset: 0=[]:[]+head
2014-06-12 14:46:03.570026 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] find_object_context 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 @head oi=27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'771 client.4308.0:37850 wrlock_by=unknown.0.0:0 dirty s 4194304 uv771)
2014-06-12 14:46:03.570059 7f9d643d9700  5 -- op tracker -- , seq: 747524, time: 2014-06-12 14:46:03.570052, event: started, op: osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.570071 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] execute_ctx 0x60aa600
2014-06-12 14:46:03.570089 7f9d643d9700  5 -- op tracker -- , seq: 747524, time: 2014-06-12 14:46:03.570084, event: started, op: osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.570096 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] do_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] ov 173'771 av 173'2925 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.570109 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] do_osd_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048]
2014-06-12 14:46:03.570121 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.570132 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] do_osd_op  write 0~258048
2014-06-12 14:46:03.570146 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] do_osd_op_effects on session 0xa702800
2014-06-12 14:46:03.570156 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 snapset=0x407f458  snapc=0=[]
2014-06-12 14:46:03.570167 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.570176 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] finish_ctx 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 0x60aa600 op modify  
2014-06-12 14:46:03.570187 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean]  set mtime to 2014-06-12 08:45:05.933607
2014-06-12 14:46:03.570234 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean]  final snapset 0=[]:[]+head in 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.570250 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.570261 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] new_repop rep_tid 236202 on osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.570277 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] issue_repop rep_tid 236202 o 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.570298 7f9d643d9700  5 -- op tracker -- , seq: 747524, time: 2014-06-12 14:46:03.570294, event: waiting for subops from 28, op: osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.570320 7f9d643d9700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:03.570401 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2924 (0'0,173'2924] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] append_log log((0'0,173'2924], crt=0'0) [173'2925 (173'771) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145082 2014-06-12 08:45:05.933607]
2014-06-12 14:46:03.570423 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2924 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] add_log_entry 173'2925 (173'771) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145082 2014-06-12 08:45:05.933607
2014-06-12 14:46:03.570441 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2924 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.570471 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2925, trimmed: 
2014-06-12 14:46:03.570506 7f9d643d9700  5 queue_transactions existing osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.570512 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.570514 7f9d643d9700  5 queue_op 0x6064420 seq 2697 osr(3.214 0x276d040) 260341 bytes   (queue has 1 ops and 260341 bytes)
2014-06-12 14:46:03.570521 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2924 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] eval_repop repgather(0x5146900 173'2925 rep_tid=236202 committed?=0 applied?=0 lock=0 op=osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.570540 7f9d643d9700 10 osd.2 173 dequeue_op 0x9fad900 finish
2014-06-12 14:46:03.570544 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.570548 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.570931 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5e60 (1 active)
2014-06-12 14:46:03.570942 7f9d7e169700  5 _do_op 0x6064420 seq 2697 osr(3.214 0x276d040)/0x276d040 start
2014-06-12 14:46:03.570947 7f9d7e169700 10 _do_transaction on 0x234cd80
2014-06-12 14:46:03.570954 7f9d7e169700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.571215 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.571237 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.571288 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.571312 7f9d7e169700 15 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.571318 7f9d7e169700 10 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1 r = 0
2014-06-12 14:46:03.571325 7f9d7e169700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.571342 7f9d7e169700 15 _write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 0~258048
2014-06-12 14:46:03.571670 7f9d7e169700 10 lookup_strip_header done  cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.571811 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:03.572764 7f9d7e169700 10 _generic_write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 0~258048 = 0
2014-06-12 14:46:03.572793 7f9d7e169700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.572806 7f9d7e169700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.572813 7f9d7e169700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.572821 7f9d7e169700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.572974 7f9d7e169700 20 set_header setting 16951 cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 parent seq 0
2014-06-12 14:46:03.573019 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.573036 7f9d7e169700 20 set_header setting 2020 cid meta oid 1fe25aca/pglog_3.214/0//-1 parent seq 0
2014-06-12 14:46:03.573047 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:03.574844 7f9d7e169700 10 _do_op 0x6064420 seq 2697 r = 0, finisher 0x68fd2a0 0x4537f80
2014-06-12 14:46:03.574862 7f9d7e169700 10 _finish_op 0x6064420 seq 2697 osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.574874 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5e60 (0 active)
2014-06-12 14:46:03.574878 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.574868 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2924 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] op_commit: 236202
2014-06-12 14:46:03.574901 7f9d7715b700  5 -- op tracker -- , seq: 747524, time: 2014-06-12 14:46:03.574891, event: op_commit, op: osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.574921 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2924 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] op_applied: 236202
2014-06-12 14:46:03.574941 7f9d7795c700  5 -- op tracker -- , seq: 747524, time: 2014-06-12 14:46:03.574935, event: op_applied, op: osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.574949 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2924 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] op_applied on primary on version 173'2925
2014-06-12 14:46:03.574986 7f9d56432700  5 -- op tracker -- , seq: 747527, time: 2014-06-12 14:46:03.574023, event: header_read, op: osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.574998 7f9d56432700  5 -- op tracker -- , seq: 747527, time: 2014-06-12 14:46:03.574025, event: throttled, op: osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.575007 7f9d56432700  5 -- op tracker -- , seq: 747527, time: 2014-06-12 14:46:03.574958, event: all_read, op: osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.575020 7f9d56432700  5 -- op tracker -- , seq: 747527, time: 0.000000, event: dispatched, op: osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.575025 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:03.575031 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:03.575038 7f9d56432700 15 osd.2 173 enqueue_op 0x24c9c200 prio 127 cost 516903 latency 0.001015 osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.575069 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.575076 7f9d61bd4700 10 osd.2 173 dequeue_op 0x24c9c200 prio 127 cost 516903 latency 0.001052 osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2985 (0'0,173'2985] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active]
2014-06-12 14:46:03.575098 7f9d61bd4700  5 -- op tracker -- , seq: 747527, time: 2014-06-12 14:46:03.575093, event: reached_pg, op: osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.575107 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2985 (0'0,173'2985] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active] handle_message: 0x24c9c200
2014-06-12 14:46:03.575117 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2985 (0'0,173'2985] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active] sub_op_modify trans 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 v 173'2986 (transaction) 174
2014-06-12 14:46:03.575133 7f9d61bd4700  5 -- op tracker -- , seq: 747527, time: 2014-06-12 14:46:03.575129, event: started, op: osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.575145 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2985 (0'0,173'2985] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active] append_log log((0'0,173'2985], crt=0'0) [173'2986 (173'2984) modify   6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 by client.4307.0:146569 2014-06-12 08:44:32.702053]
2014-06-12 14:46:03.575171 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2986 (0'0,173'2986] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active] add_log_entry 173'2986 (173'2984) modify   6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 by client.4307.0:146569 2014-06-12 08:44:32.702053
2014-06-12 14:46:03.575193 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2986 (0'0,173'2986] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active] append_log  adding 1 keys
2014-06-12 14:46:03.575225 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2986, trimmed: 
2014-06-12 14:46:03.575247 7f9d61bd4700  5 -- op tracker -- , seq: 747527, time: 2014-06-12 14:46:03.575242, event: started, op: osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.575256 7f9d61bd4700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:03.575259 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.575260 7f9d61bd4700  5 queue_op 0x6fdbaa0 seq 2788 osr(3.458 0x276d960) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:03.575267 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2986 (0'0,173'2986] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active] do_sub_op osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.575282 7f9d61bd4700 10 osd.2 173 dequeue_op 0x24c9c200 finish
2014-06-12 14:46:03.575286 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.575289 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.575798 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:03.575806 7f9d7e96a700  5 _do_op 0x6fdbaa0 seq 2788 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:03.575810 7f9d7e96a700 10 _do_transaction on 0x1da3e470
2014-06-12 14:46:03.575816 7f9d7e96a700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:03.575936 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:03.575955 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.575997 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.576018 7f9d7e96a700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:03.576023 7f9d7e96a700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:03.576028 7f9d7e96a700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:03.576043 7f9d7e96a700 15 _write 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 2297856~516096
2014-06-12 14:46:03.576350 7f9d7e96a700 10 lookup_strip_header done  cid 3.458_head oid 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:46:03.576504 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:03.578437 7f9d7e96a700 10 _generic_write 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 2297856~516096 = 0
2014-06-12 14:46:03.578523 7f9d7e96a700 15 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:46:03.578543 7f9d7e96a700 10 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 = 0
2014-06-12 14:46:03.578548 7f9d7e96a700 15 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:46:03.578560 7f9d7e96a700 10 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 = 0
2014-06-12 14:46:03.578800 7f9d7e96a700 20 set_header setting 18434 cid 3.458_head oid 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 parent seq 0
2014-06-12 14:46:03.578827 7f9d56432700  5 -- op tracker -- , seq: 747528, time: 2014-06-12 14:46:03.578531, event: header_read, op: osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.578844 7f9d56432700  5 -- op tracker -- , seq: 747528, time: 2014-06-12 14:46:03.578533, event: throttled, op: osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.578853 7f9d56432700  5 -- op tracker -- , seq: 747528, time: 2014-06-12 14:46:03.578785, event: all_read, op: osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.578862 7f9d56432700  5 -- op tracker -- , seq: 747528, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.578868 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:03.578875 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:03.578881 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.578884 7f9d56432700 15 osd.2 173 enqueue_op 0x6d10100 prio 127 cost 131875 latency 0.000353 osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.578894 7f9d7e96a700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:03.578904 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:03.578968 7f9d5b6bc700  5 -- op tracker -- , seq: 747529, time: 2014-06-12 14:46:03.578931, event: header_read, op: osd_sub_op_reply(client.4308.0:145082 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.578979 7f9d5b6bc700  5 -- op tracker -- , seq: 747529, time: 2014-06-12 14:46:03.578932, event: throttled, op: osd_sub_op_reply(client.4308.0:145082 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.578987 7f9d5b6bc700  5 -- op tracker -- , seq: 747529, time: 2014-06-12 14:46:03.578941, event: all_read, op: osd_sub_op_reply(client.4308.0:145082 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.578995 7f9d5b6bc700  5 -- op tracker -- , seq: 747529, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145082 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.578999 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145082 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.579004 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:03.579010 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x124e3a00 prio 196 cost 0 latency 0.000079 osd_sub_op_reply(client.4308.0:145082 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.579042 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.579053 7f9d663dd700 10 osd.2 173 dequeue_op 0x124e3a00 prio 196 cost 0 latency 0.000121 osd_sub_op_reply(client.4308.0:145082 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 pg pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2924 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean]
2014-06-12 14:46:03.579076 7f9d663dd700  5 -- op tracker -- , seq: 747529, time: 2014-06-12 14:46:03.579073, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145082 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.579085 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2924 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] handle_message: 0x124e3a00
2014-06-12 14:46:03.579099 7f9d663dd700  5 -- op tracker -- , seq: 747529, time: 2014-06-12 14:46:03.579096, event: started, op: osd_sub_op_reply(client.4308.0:145082 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.579105 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2924 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] sub_op_modify_reply: tid 236202 op  ack_type 4 from 28
2014-06-12 14:46:03.579130 7f9d663dd700  5 -- op tracker -- , seq: 747524, time: 2014-06-12 14:46:03.579119, event: sub_op_commit_rec, op: osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.579139 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2924 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] repop_all_applied: repop tid 236202 all applied 
2014-06-12 14:46:03.579150 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2924 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] eval_repop repgather(0x5146900 173'2925 rep_tid=236202 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.579168 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2924 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean]  sending ack on repgather(0x5146900 173'2925 rep_tid=236202 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x2844a780
2014-06-12 14:46:03.579202 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2924 crt=0'0 lcod 173'2923 mlcod 173'2923 active+clean] repop_all_committed: repop tid 236202 all committed 
2014-06-12 14:46:03.579212 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2923 active+clean] eval_repop repgather(0x5146900 173'2925 rep_tid=236202 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.579229 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2923 active+clean] log_op_stats osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 inb 258327 outb 0 rlat 0.026225 lat 0.026252
2014-06-12 14:46:03.579246 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2923 active+clean] publish_stats_to_osd 173:3693
2014-06-12 14:46:03.579259 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2923 active+clean]  sending commit on repgather(0x5146900 173'2925 rep_tid=236202 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x25edd680
2014-06-12 14:46:03.579289 7f9d663dd700  5 -- op tracker -- , seq: 747524, time: 2014-06-12 14:46:03.579284, event: commit_sent, op: osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.579296 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2923 active+clean]  requeue_ops 
2014-06-12 14:46:03.579306 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean]  removing repgather(0x5146900 173'2925 rep_tid=236202 committed?=1 applied?=1 op=osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.579319 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean]    q front is repgather(0x5146900 173'2925 rep_tid=236202 committed?=1 applied?=1 op=osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.579335 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] remove_repop repgather(0x5146900 173'2925 rep_tid=236202 committed?=1 applied?=1 op=osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.579347 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean]  requeue_ops 
2014-06-12 14:46:03.579373 7f9d663dd700  5 -- op tracker -- , seq: 747524, time: 2014-06-12 14:46:03.579367, event: done, op: osd_op(client.4308.0:145082 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.579392 7f9d663dd700 10 osd.2 173 dequeue_op 0x124e3a00 finish
2014-06-12 14:46:03.579399 7f9d663dd700  5 -- op tracker -- , seq: 747529, time: 2014-06-12 14:46:03.579396, event: done, op: osd_sub_op_reply(client.4308.0:145082 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.579408 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.579420 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.579424 7f9d663dd700 10 osd.2 173 dequeue_op 0x6d10100 prio 127 cost 131875 latency 0.000893 osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2469 (0'0,173'2469] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2468 active]
2014-06-12 14:46:03.579445 7f9d663dd700  5 -- op tracker -- , seq: 747528, time: 2014-06-12 14:46:03.579441, event: reached_pg, op: osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.579452 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2469 (0'0,173'2469] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2468 active] handle_message: 0x6d10100
2014-06-12 14:46:03.579461 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2469 (0'0,173'2469] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2468 active] sub_op_modify trans cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 v 173'2470 (transaction) 174
2014-06-12 14:46:03.579478 7f9d663dd700  5 -- op tracker -- , seq: 747528, time: 2014-06-12 14:46:03.579473, event: started, op: osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.579494 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2469 (0'0,173'2469] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2468 active] append_log log((0'0,173'2469], crt=0'0) [173'2470 (173'2467) modify   cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 by client.4308.0:144512 2014-06-12 08:43:37.183048]
2014-06-12 14:46:03.579517 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2470 (0'0,173'2470] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2468 active] add_log_entry 173'2470 (173'2467) modify   cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 by client.4308.0:144512 2014-06-12 08:43:37.183048
2014-06-12 14:46:03.579539 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2470 (0'0,173'2470] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2468 active] append_log  adding 1 keys
2014-06-12 14:46:03.579571 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2470, trimmed: 
2014-06-12 14:46:03.579595 7f9d663dd700  5 -- op tracker -- , seq: 747528, time: 2014-06-12 14:46:03.579590, event: started, op: osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.579605 7f9d663dd700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:03.579609 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.579610 7f9d663dd700  5 queue_op 0x76d8ba0 seq 2283 osr(3.2b 0x276cd20) 133356 bytes   (queue has 2 ops and 651745 bytes)
2014-06-12 14:46:03.579618 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2470 (0'0,173'2470] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2468 active] do_sub_op osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.579632 7f9d663dd700 10 osd.2 173 dequeue_op 0x6d10100 finish
2014-06-12 14:46:03.579636 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.579638 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.579663 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (2 active)
2014-06-12 14:46:03.579671 7f9d82171700  5 _do_op 0x76d8ba0 seq 2283 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:46:03.579676 7f9d82171700 10 _do_transaction on 0x90f1970
2014-06-12 14:46:03.579684 7f9d82171700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.579830 7f9d82171700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.579924 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.581343 7f9d7e96a700 10 _do_op 0x6fdbaa0 seq 2788 r = 0, finisher 0x8078740 0
2014-06-12 14:46:03.581361 7f9d7e96a700 10 _finish_op 0x6fdbaa0 seq 2788 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:03.581369 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (1 active)
2014-06-12 14:46:03.581380 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.581390 7f9d7795c700  5 -- op tracker -- , seq: 747527, time: 2014-06-12 14:46:03.581380, event: sub_op_applied, op: osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.581403 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2986 (0'0,173'2986] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active] sub_op_modify_applied on 0x1da3e380 op osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.581432 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:03.581453 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2986 (0'0,173'2986] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active] op_applied on primary on version 173'2986
2014-06-12 14:46:03.581465 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2986 (0'0,173'2986] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active] op_applied on replica on version 173'2986
2014-06-12 14:46:03.581529 7f9d7715b700  5 -- op tracker -- , seq: 747527, time: 2014-06-12 14:46:03.581522, event: commit_sent, op: osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.581537 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2986 (0'0,173'2986] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2984 active] sub_op_modify_commit on op osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:03.581555 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:03.581583 7f9d7715b700  5 -- op tracker -- , seq: 747527, time: 2014-06-12 14:46:03.581578, event: done, op: osd_sub_op(client.4307.0:146569 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.581636 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.581735 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.581768 7f9d82171700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.581775 7f9d82171700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:46:03.581783 7f9d82171700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.581801 7f9d82171700 15 _write 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 4063232~131072
2014-06-12 14:46:03.581967 7f9d82171700 10 lookup_strip_header done  cid 3.2b_head oid cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3
2014-06-12 14:46:03.582103 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:03.582643 7f9d82171700 10 _generic_write 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 4063232~131072 = 0
2014-06-12 14:46:03.582675 7f9d82171700 15 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3
2014-06-12 14:46:03.582691 7f9d82171700 10 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 = 0
2014-06-12 14:46:03.582697 7f9d82171700 15 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3
2014-06-12 14:46:03.582706 7f9d82171700 10 _setattrs 3.2b_head/cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 = 0
2014-06-12 14:46:03.582831 7f9d82171700 20 set_header setting 16771 cid 3.2b_head oid cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 parent seq 0
2014-06-12 14:46:03.582874 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.582888 7f9d82171700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:46:03.582898 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:03.583557 7f9d82171700 10 _do_op 0x76d8ba0 seq 2283 r = 0, finisher 0x80b8020 0
2014-06-12 14:46:03.583613 7f9d7715b700  5 -- op tracker -- , seq: 747528, time: 2014-06-12 14:46:03.583596, event: commit_sent, op: osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.583628 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2470 (0'0,173'2470] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2468 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:03.583662 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:03.583760 7f9d82171700 10 _finish_op 0x76d8ba0 seq 2283 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:03.583789 7f9d7795c700  5 -- op tracker -- , seq: 747528, time: 2014-06-12 14:46:03.583783, event: sub_op_applied, op: osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.583798 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2470 (0'0,173'2470] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active] sub_op_modify_applied on 0x90f1880 op osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.583814 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2470 (0'0,173'2470] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active] op_applied on primary on version 173'2470
2014-06-12 14:46:03.583825 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2470 (0'0,173'2470] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active] op_applied on replica on version 173'2470
2014-06-12 14:46:03.583863 7f9d82171700  5 -- op tracker -- , seq: 747528, time: 2014-06-12 14:46:03.583853, event: done, op: osd_sub_op(client.4308.0:144512 3.2b cef4102b/rbd_data.10ab6b8b4567.0000000000000f36/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.583887 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:46:03.583891 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.612110 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:03.615154 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:03.636979 7f9d3b028700  5 -- op tracker -- , seq: 747530, time: 2014-06-12 14:46:03.633869, event: header_read, op: osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.636994 7f9d3b028700  5 -- op tracker -- , seq: 747530, time: 2014-06-12 14:46:03.633873, event: throttled, op: osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.637003 7f9d3b028700  5 -- op tracker -- , seq: 747530, time: 2014-06-12 14:46:03.636935, event: all_read, op: osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.637011 7f9d3b028700  5 -- op tracker -- , seq: 747530, time: 0.000000, event: dispatched, op: osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.637018 7f9d3b028700 20 osd.2 173 should_share_map client.4308 172.16.96.220:0/1243963 173
2014-06-12 14:46:03.637041 7f9d3b028700 15 osd.2 173 enqueue_op 0x160e5600 prio 63 cost 516096 latency 0.003172 osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.637071 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.637077 7f9d67be0700 10 osd.2 173 dequeue_op 0x160e5600 prio 63 cost 516096 latency 0.003208 osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean]
2014-06-12 14:46:03.637100 7f9d67be0700  5 -- op tracker -- , seq: 747530, time: 2014-06-12 14:46:03.637095, event: reached_pg, op: osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.637108 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.637122 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] handle_message: 0x160e5600
2014-06-12 14:46:03.637131 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] do_op osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.637153 7f9d67be0700 15 getattr 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 '_'
2014-06-12 14:46:03.637357 7f9d67be0700 10 getattr 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 '_' = 0
2014-06-12 14:46:03.637374 7f9d67be0700 15 getattr 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 'snapset'
2014-06-12 14:46:03.637458 7f9d67be0700 10 getattr 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 'snapset' = 0
2014-06-12 14:46:03.637469 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] populate_obc_watchers 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.637482 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.637491 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] get_object_context: creating obc from disk: 0x30f32000
2014-06-12 14:46:03.637500 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] get_object_context: 0x30f32000 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 rwstate(none n=0 w=0) oi: 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2925 client.4308.0:145082 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2925) ssc: 0x81bb880 snapset: 0=[]:[]+head
2014-06-12 14:46:03.637518 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] find_object_context 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 @head oi=27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2925 client.4308.0:145082 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2925)
2014-06-12 14:46:03.637545 7f9d67be0700  5 -- op tracker -- , seq: 747530, time: 2014-06-12 14:46:03.637540, event: started, op: osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.637553 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] execute_ctx 0x2963c400
2014-06-12 14:46:03.637572 7f9d67be0700  5 -- op tracker -- , seq: 747530, time: 2014-06-12 14:46:03.637566, event: started, op: osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.637579 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] do_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] ov 173'2925 av 173'2926 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.637593 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] do_osd_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096]
2014-06-12 14:46:03.637603 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.637637 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] do_osd_op  write 258048~516096
2014-06-12 14:46:03.637655 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] do_osd_op_effects on session 0xa702800
2014-06-12 14:46:03.637667 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 snapset=0x81bb8b8  snapc=0=[]
2014-06-12 14:46:03.637679 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.637688 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] finish_ctx 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 0x2963c400 op modify  
2014-06-12 14:46:03.637700 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean]  set mtime to 2014-06-12 08:45:06.014040
2014-06-12 14:46:03.637724 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean]  final snapset 0=[]:[]+head in 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.637744 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.637754 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] new_repop rep_tid 236203 on osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.637771 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] issue_repop rep_tid 236203 o 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.637793 7f9d67be0700  5 -- op tracker -- , seq: 747530, time: 2014-06-12 14:46:03.637788, event: waiting for subops from 28, op: osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.637821 7f9d67be0700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:03.637845 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2925 (0'0,173'2925] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] append_log log((0'0,173'2925], crt=0'0) [173'2926 (173'2925) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145083 2014-06-12 08:45:06.014040]
2014-06-12 14:46:03.637867 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] add_log_entry 173'2926 (173'2925) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145083 2014-06-12 08:45:06.014040
2014-06-12 14:46:03.637885 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.637917 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2926, trimmed: 
2014-06-12 14:46:03.637943 7f9d67be0700  5 queue_transactions existing osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.637948 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.637949 7f9d67be0700  5 queue_op 0x6a7acc0 seq 2698 osr(3.214 0x276d040) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:03.637961 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5e60 (1 active)
2014-06-12 14:46:03.637968 7f9d7d167700  5 _do_op 0x6a7acc0 seq 2698 osr(3.214 0x276d040)/0x276d040 start
2014-06-12 14:46:03.637971 7f9d7d167700 10 _do_transaction on 0x523d800
2014-06-12 14:46:03.637957 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] eval_repop repgather(0x514d2c0 173'2926 rep_tid=236203 committed?=0 applied?=0 lock=0 op=osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.637974 7f9d67be0700 10 osd.2 173 dequeue_op 0x160e5600 finish
2014-06-12 14:46:03.637978 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.637977 7f9d7d167700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.637981 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.638087 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.638114 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.638167 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.638193 7f9d7d167700 15 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.638199 7f9d7d167700 10 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1 r = 0
2014-06-12 14:46:03.638205 7f9d7d167700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.638396 7f9d7d167700 15 _write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 258048~516096
2014-06-12 14:46:03.638548 7f9d7d167700 10 lookup_strip_header done  cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.638681 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:03.640352 7f9d7d167700 10 _generic_write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 258048~516096 = 0
2014-06-12 14:46:03.640404 7f9d7d167700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.640414 7f9d7d167700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.640418 7f9d7d167700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.640423 7f9d7d167700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.640582 7f9d7d167700 20 set_header setting 16951 cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 parent seq 0
2014-06-12 14:46:03.640666 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.640678 7f9d7d167700 20 set_header setting 2020 cid meta oid 1fe25aca/pglog_3.214/0//-1 parent seq 0
2014-06-12 14:46:03.640684 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:03.643594 7f9d7d167700 10 _do_op 0x6a7acc0 seq 2698 r = 0, finisher 0x4f2fca0 0x25c94ac0
2014-06-12 14:46:03.643619 7f9d7d167700 10 _finish_op 0x6a7acc0 seq 2698 osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.643626 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5e60 (0 active)
2014-06-12 14:46:03.643631 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.643626 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] op_commit: 236203
2014-06-12 14:46:03.643662 7f9d7715b700  5 -- op tracker -- , seq: 747530, time: 2014-06-12 14:46:03.643652, event: op_commit, op: osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.643681 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] op_applied: 236203
2014-06-12 14:46:03.643698 7f9d7795c700  5 -- op tracker -- , seq: 747530, time: 2014-06-12 14:46:03.643693, event: op_applied, op: osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.643705 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] op_applied on primary on version 173'2926
2014-06-12 14:46:03.663361 7f9d3b028700  5 -- op tracker -- , seq: 747531, time: 2014-06-12 14:46:03.662045, event: header_read, op: osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.663376 7f9d3b028700  5 -- op tracker -- , seq: 747531, time: 2014-06-12 14:46:03.662050, event: throttled, op: osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.663386 7f9d3b028700  5 -- op tracker -- , seq: 747531, time: 2014-06-12 14:46:03.663313, event: all_read, op: osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.663397 7f9d3b028700  5 -- op tracker -- , seq: 747531, time: 0.000000, event: dispatched, op: osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.663404 7f9d3b028700 20 osd.2 173 should_share_map client.4308 172.16.96.220:0/1243963 173
2014-06-12 14:46:03.663433 7f9d3b028700 15 osd.2 173 enqueue_op 0x160e4900 prio 63 cost 516096 latency 0.001388 osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.663466 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.663477 7f9d693e3700 10 osd.2 173 dequeue_op 0x160e4900 prio 63 cost 516096 latency 0.001432 osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean]
2014-06-12 14:46:03.663507 7f9d693e3700  5 -- op tracker -- , seq: 747531, time: 2014-06-12 14:46:03.663502, event: reached_pg, op: osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.663520 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.663538 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] handle_message: 0x160e4900
2014-06-12 14:46:03.663549 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] do_op osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.663576 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] get_object_context: found obc in cache: 0x30f32000
2014-06-12 14:46:03.663590 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] get_object_context: 0x30f32000 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 rwstate(write n=1 w=0) oi: 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2926 client.4308.0:145083 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2926) ssc: 0x81bb880 snapset: 0=[]:[]+head
2014-06-12 14:46:03.663609 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] find_object_context 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 @head oi=27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2926 client.4308.0:145083 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2926)
2014-06-12 14:46:03.663639 7f9d693e3700  5 -- op tracker -- , seq: 747531, time: 2014-06-12 14:46:03.663633, event: started, op: osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.663650 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] execute_ctx 0x5ca0600
2014-06-12 14:46:03.663670 7f9d693e3700  5 -- op tracker -- , seq: 747531, time: 2014-06-12 14:46:03.663664, event: started, op: osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.663677 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] do_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] ov 173'2926 av 173'2927 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.663693 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] do_osd_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096]
2014-06-12 14:46:03.663706 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.663721 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] do_osd_op  write 774144~516096
2014-06-12 14:46:03.663739 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] do_osd_op_effects on session 0xa702800
2014-06-12 14:46:03.663753 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 snapset=0x81bb8b8  snapc=0=[]
2014-06-12 14:46:03.663769 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.663783 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] finish_ctx 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 0x5ca0600 op modify  
2014-06-12 14:46:03.663797 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean]  set mtime to 2014-06-12 08:45:06.042261
2014-06-12 14:46:03.663820 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean]  final snapset 0=[]:[]+head in 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.663843 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.663856 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] new_repop rep_tid 236204 on osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.663873 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] issue_repop rep_tid 236204 o 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.663902 7f9d693e3700  5 -- op tracker -- , seq: 747531, time: 2014-06-12 14:46:03.663897, event: waiting for subops from 28, op: osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.663927 7f9d693e3700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:03.663956 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2926 (0'0,173'2926] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] append_log log((0'0,173'2926], crt=0'0) [173'2927 (173'2926) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145084 2014-06-12 08:45:06.042261]
2014-06-12 14:46:03.663983 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] add_log_entry 173'2927 (173'2926) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145084 2014-06-12 08:45:06.042261
2014-06-12 14:46:03.664008 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.664043 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2927, trimmed: 
2014-06-12 14:46:03.664066 7f9d693e3700  5 queue_transactions existing osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.664072 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.664074 7f9d693e3700  5 queue_op 0x77830e0 seq 2699 osr(3.214 0x276d040) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:03.664081 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] eval_repop repgather(0x24e8d080 173'2927 rep_tid=236204 committed?=0 applied?=0 lock=0 op=osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.664100 7f9d693e3700 10 osd.2 173 dequeue_op 0x160e4900 finish
2014-06-12 14:46:03.664104 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.664107 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.664116 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5e60 (1 active)
2014-06-12 14:46:03.664121 7f9d7815d700  5 _do_op 0x77830e0 seq 2699 osr(3.214 0x276d040)/0x276d040 start
2014-06-12 14:46:03.664125 7f9d7815d700 10 _do_transaction on 0x2b62b800
2014-06-12 14:46:03.664131 7f9d7815d700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.664254 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.664374 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.664432 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.664459 7f9d7815d700 15 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.664461 7f9d5b6bc700  5 -- op tracker -- , seq: 747532, time: 2014-06-12 14:46:03.664419, event: header_read, op: osd_sub_op_reply(client.4308.0:145083 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.664468 7f9d7815d700 10 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1 r = 0
2014-06-12 14:46:03.664470 7f9d5b6bc700  5 -- op tracker -- , seq: 747532, time: 2014-06-12 14:46:03.664421, event: throttled, op: osd_sub_op_reply(client.4308.0:145083 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.664474 7f9d7815d700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.664479 7f9d5b6bc700  5 -- op tracker -- , seq: 747532, time: 2014-06-12 14:46:03.664437, event: all_read, op: osd_sub_op_reply(client.4308.0:145083 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.664487 7f9d5b6bc700  5 -- op tracker -- , seq: 747532, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145083 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.664491 7f9d7815d700 15 _write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 774144~516096
2014-06-12 14:46:03.664492 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145083 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.664498 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:03.664504 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x28fd8100 prio 196 cost 0 latency 0.000085 osd_sub_op_reply(client.4308.0:145083 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.664639 7f9d7815d700 10 lookup_strip_header done  cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.664850 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:03.665384 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.665396 7f9d62bd6700 10 osd.2 173 dequeue_op 0x28fd8100 prio 196 cost 0 latency 0.000977 osd_sub_op_reply(client.4308.0:145083 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 pg pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean]
2014-06-12 14:46:03.665422 7f9d62bd6700  5 -- op tracker -- , seq: 747532, time: 2014-06-12 14:46:03.665418, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145083 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.665432 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] handle_message: 0x28fd8100
2014-06-12 14:46:03.665450 7f9d62bd6700  5 -- op tracker -- , seq: 747532, time: 2014-06-12 14:46:03.665446, event: started, op: osd_sub_op_reply(client.4308.0:145083 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.665456 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] sub_op_modify_reply: tid 236203 op  ack_type 4 from 28
2014-06-12 14:46:03.665477 7f9d62bd6700  5 -- op tracker -- , seq: 747530, time: 2014-06-12 14:46:03.665469, event: sub_op_commit_rec, op: osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.665487 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] repop_all_applied: repop tid 236203 all applied 
2014-06-12 14:46:03.665498 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] eval_repop repgather(0x514d2c0 173'2926 rep_tid=236203 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.665513 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean]  sending ack on repgather(0x514d2c0 173'2926 rep_tid=236203 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x1a58c280
2014-06-12 14:46:03.665552 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2925 crt=0'0 lcod 173'2924 mlcod 173'2924 active+clean] repop_all_committed: repop tid 236203 all committed 
2014-06-12 14:46:03.665564 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2924 active+clean] eval_repop repgather(0x514d2c0 173'2926 rep_tid=236203 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.665580 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2924 active+clean] log_op_stats osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.031681 lat 0.031710
2014-06-12 14:46:03.665597 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2924 active+clean] publish_stats_to_osd 173:3694
2014-06-12 14:46:03.665610 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2924 active+clean]  sending commit on repgather(0x514d2c0 173'2926 rep_tid=236203 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x30ccbb80
2014-06-12 14:46:03.665661 7f9d62bd6700  5 -- op tracker -- , seq: 747530, time: 2014-06-12 14:46:03.665655, event: commit_sent, op: osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.665669 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2924 active+clean]  requeue_ops 
2014-06-12 14:46:03.665680 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean]  removing repgather(0x514d2c0 173'2926 rep_tid=236203 committed?=1 applied?=1 op=osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.665694 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean]    q front is repgather(0x514d2c0 173'2926 rep_tid=236203 committed?=1 applied?=1 op=osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.665708 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] remove_repop repgather(0x514d2c0 173'2926 rep_tid=236203 committed?=1 applied?=1 op=osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.665722 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean]  requeue_ops 
2014-06-12 14:46:03.665742 7f9d62bd6700  5 -- op tracker -- , seq: 747530, time: 2014-06-12 14:46:03.665737, event: done, op: osd_op(client.4308.0:145083 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.665764 7f9d62bd6700 10 osd.2 173 dequeue_op 0x28fd8100 finish
2014-06-12 14:46:03.665775 7f9d62bd6700  5 -- op tracker -- , seq: 747532, time: 2014-06-12 14:46:03.665771, event: done, op: osd_sub_op_reply(client.4308.0:145083 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.665787 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.665790 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.666886 7f9d7815d700 10 _generic_write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 774144~516096 = 0
2014-06-12 14:46:03.666941 7f9d7815d700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.666962 7f9d7815d700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.666971 7f9d7815d700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.666980 7f9d7815d700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.667209 7f9d7815d700 20 set_header setting 16951 cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 parent seq 0
2014-06-12 14:46:03.667292 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.667311 7f9d7815d700 20 set_header setting 2020 cid meta oid 1fe25aca/pglog_3.214/0//-1 parent seq 0
2014-06-12 14:46:03.667324 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:03.669310 7f9d3b028700  5 -- op tracker -- , seq: 747533, time: 2014-06-12 14:46:03.668136, event: header_read, op: osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.669332 7f9d3b028700  5 -- op tracker -- , seq: 747533, time: 2014-06-12 14:46:03.668140, event: throttled, op: osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.669340 7f9d3b028700  5 -- op tracker -- , seq: 747533, time: 2014-06-12 14:46:03.669238, event: all_read, op: osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.669348 7f9d3b028700  5 -- op tracker -- , seq: 747533, time: 0.000000, event: dispatched, op: osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.669359 7f9d3b028700 20 osd.2 173 should_share_map client.4308 172.16.96.220:0/1243963 173
2014-06-12 14:46:03.669394 7f9d3b028700 15 osd.2 173 enqueue_op 0x160e5400 prio 63 cost 516096 latency 0.001258 osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.669438 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.669448 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x160e5400 prio 63 cost 516096 latency 0.001312 osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean]
2014-06-12 14:46:03.669478 7f9d6b3e7700  5 -- op tracker -- , seq: 747533, time: 2014-06-12 14:46:03.669472, event: reached_pg, op: osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.669487 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.669502 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] handle_message: 0x160e5400
2014-06-12 14:46:03.669513 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] do_op osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.669542 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] get_object_context: found obc in cache: 0x30f32000
2014-06-12 14:46:03.669553 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] get_object_context: 0x30f32000 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 rwstate(write n=1 w=0) oi: 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2927 client.4308.0:145084 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2927) ssc: 0x81bb880 snapset: 0=[]:[]+head
2014-06-12 14:46:03.669568 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] find_object_context 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 @head oi=27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2927 client.4308.0:145084 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2927)
2014-06-12 14:46:03.669594 7f9d6b3e7700  5 -- op tracker -- , seq: 747533, time: 2014-06-12 14:46:03.669589, event: started, op: osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.669602 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] execute_ctx 0x2e410c00
2014-06-12 14:46:03.669631 7f9d6b3e7700  5 -- op tracker -- , seq: 747533, time: 2014-06-12 14:46:03.669626, event: started, op: osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.669638 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] do_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] ov 173'2927 av 173'2928 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.669651 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] do_osd_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096]
2014-06-12 14:46:03.669662 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.669675 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] do_osd_op  write 1290240~516096
2014-06-12 14:46:03.669689 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] do_osd_op_effects on session 0xa702800
2014-06-12 14:46:03.669700 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 snapset=0x81bb8b8  snapc=0=[]
2014-06-12 14:46:03.669712 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.669723 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] finish_ctx 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 0x2e410c00 op modify  
2014-06-12 14:46:03.669736 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean]  set mtime to 2014-06-12 08:45:06.048280
2014-06-12 14:46:03.669757 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean]  final snapset 0=[]:[]+head in 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.669775 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.669788 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] new_repop rep_tid 236205 on osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.669805 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] issue_repop rep_tid 236205 o 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.669831 7f9d6b3e7700  5 -- op tracker -- , seq: 747533, time: 2014-06-12 14:46:03.669826, event: waiting for subops from 28, op: osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.669859 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:03.669882 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2927 (0'0,173'2927] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] append_log log((0'0,173'2927], crt=0'0) [173'2928 (173'2927) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145085 2014-06-12 08:45:06.048280]
2014-06-12 14:46:03.669906 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] add_log_entry 173'2928 (173'2927) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145085 2014-06-12 08:45:06.048280
2014-06-12 14:46:03.669926 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.669956 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2928, trimmed: 
2014-06-12 14:46:03.669977 7f9d6b3e7700  5 queue_transactions existing osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.669982 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.669983 7f9d6b3e7700  5 queue_op 0x8103aa0 seq 2700 osr(3.214 0x276d040) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:03.669995 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5e60 (2 active)
2014-06-12 14:46:03.669992 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] eval_repop repgather(0x12ae3200 173'2928 rep_tid=236205 committed?=0 applied?=0 lock=0 op=osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.670010 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x160e5400 finish
2014-06-12 14:46:03.670014 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.670017 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.670137 7f9d7815d700 10 _do_op 0x77830e0 seq 2699 r = 0, finisher 0x57235a0 0x42ccac0
2014-06-12 14:46:03.670150 7f9d7815d700 10 _finish_op 0x77830e0 seq 2699 osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.670157 7f9d79960700  5 _do_op 0x8103aa0 seq 2700 osr(3.214 0x276d040)/0x276d040 start
2014-06-12 14:46:03.670163 7f9d79960700 10 _do_transaction on 0x29703a40
2014-06-12 14:46:03.670160 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5e60 (1 active)
2014-06-12 14:46:03.670170 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.670170 7f9d79960700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.670167 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] op_applied: 236204
2014-06-12 14:46:03.670189 7f9d7795c700  5 -- op tracker -- , seq: 747531, time: 2014-06-12 14:46:03.670183, event: op_applied, op: osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.670197 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] op_applied on primary on version 173'2927
2014-06-12 14:46:03.670281 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.670310 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.670359 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.670384 7f9d79960700 15 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.670391 7f9d79960700 10 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1 r = 0
2014-06-12 14:46:03.670397 7f9d79960700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.670475 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] op_commit: 236204
2014-06-12 14:46:03.670497 7f9d7715b700  5 -- op tracker -- , seq: 747531, time: 2014-06-12 14:46:03.670491, event: op_commit, op: osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.670540 7f9d79960700 15 _write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 1290240~516096
2014-06-12 14:46:03.670700 7f9d79960700 10 lookup_strip_header done  cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.670805 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:03.672872 7f9d79960700 10 _generic_write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 1290240~516096 = 0
2014-06-12 14:46:03.672938 7f9d79960700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.672954 7f9d79960700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.672961 7f9d79960700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.672969 7f9d79960700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.673222 7f9d79960700 20 set_header setting 16951 cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 parent seq 0
2014-06-12 14:46:03.673305 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.673320 7f9d79960700 20 set_header setting 2020 cid meta oid 1fe25aca/pglog_3.214/0//-1 parent seq 0
2014-06-12 14:46:03.673331 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:03.678238 7f9d79960700 10 _do_op 0x8103aa0 seq 2700 r = 0, finisher 0x8357bc0 0x218d1ec0
2014-06-12 14:46:03.678268 7f9d79960700 10 _finish_op 0x8103aa0 seq 2700 osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.678271 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] op_commit: 236205
2014-06-12 14:46:03.678289 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5e60 (0 active)
2014-06-12 14:46:03.678299 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.678307 7f9d7715b700  5 -- op tracker -- , seq: 747533, time: 2014-06-12 14:46:03.678296, event: op_commit, op: osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.678324 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] op_applied: 236205
2014-06-12 14:46:03.678346 7f9d7795c700  5 -- op tracker -- , seq: 747533, time: 2014-06-12 14:46:03.678340, event: op_applied, op: osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.678355 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] op_applied on primary on version 173'2928
2014-06-12 14:46:03.690094 7f9d5b6bc700  5 -- op tracker -- , seq: 747534, time: 2014-06-12 14:46:03.690034, event: header_read, op: osd_sub_op_reply(client.4308.0:145084 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.690106 7f9d5b6bc700  5 -- op tracker -- , seq: 747534, time: 2014-06-12 14:46:03.690037, event: throttled, op: osd_sub_op_reply(client.4308.0:145084 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.690113 7f9d5b6bc700  5 -- op tracker -- , seq: 747534, time: 2014-06-12 14:46:03.690053, event: all_read, op: osd_sub_op_reply(client.4308.0:145084 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.690121 7f9d5b6bc700  5 -- op tracker -- , seq: 747534, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145084 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.690126 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145084 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.690133 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:03.690141 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39576f00 prio 196 cost 0 latency 0.000107 osd_sub_op_reply(client.4308.0:145084 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.690171 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.690181 7f9d64bda700 10 osd.2 173 dequeue_op 0x39576f00 prio 196 cost 0 latency 0.000146 osd_sub_op_reply(client.4308.0:145084 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 pg pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean]
2014-06-12 14:46:03.690203 7f9d64bda700  5 -- op tracker -- , seq: 747534, time: 2014-06-12 14:46:03.690199, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145084 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.690213 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] handle_message: 0x39576f00
2014-06-12 14:46:03.690229 7f9d64bda700  5 -- op tracker -- , seq: 747534, time: 2014-06-12 14:46:03.690225, event: started, op: osd_sub_op_reply(client.4308.0:145084 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.690235 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] sub_op_modify_reply: tid 236204 op  ack_type 4 from 28
2014-06-12 14:46:03.690265 7f9d64bda700  5 -- op tracker -- , seq: 747531, time: 2014-06-12 14:46:03.690258, event: sub_op_commit_rec, op: osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.690273 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] repop_all_applied: repop tid 236204 all applied 
2014-06-12 14:46:03.690284 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] eval_repop repgather(0x24e8d080 173'2927 rep_tid=236204 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.690301 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean]  sending ack on repgather(0x24e8d080 173'2927 rep_tid=236204 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x30ee0f00
2014-06-12 14:46:03.690335 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2926 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] repop_all_committed: repop tid 236204 all committed 
2014-06-12 14:46:03.690346 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2925 active+clean] eval_repop repgather(0x24e8d080 173'2927 rep_tid=236204 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.690365 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2925 active+clean] log_op_stats osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.028289 lat 0.028319
2014-06-12 14:46:03.690384 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2925 active+clean] publish_stats_to_osd 173:3695
2014-06-12 14:46:03.690401 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2925 active+clean]  sending commit on repgather(0x24e8d080 173'2927 rep_tid=236204 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x2183b400
2014-06-12 14:46:03.690438 7f9d64bda700  5 -- op tracker -- , seq: 747531, time: 2014-06-12 14:46:03.690432, event: commit_sent, op: osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.690448 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2925 active+clean]  requeue_ops 
2014-06-12 14:46:03.690460 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean]  removing repgather(0x24e8d080 173'2927 rep_tid=236204 committed?=1 applied?=1 op=osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.690476 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean]    q front is repgather(0x24e8d080 173'2927 rep_tid=236204 committed?=1 applied?=1 op=osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.690491 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] remove_repop repgather(0x24e8d080 173'2927 rep_tid=236204 committed?=1 applied?=1 op=osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.690507 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean]  requeue_ops 
2014-06-12 14:46:03.690528 7f9d64bda700  5 -- op tracker -- , seq: 747531, time: 2014-06-12 14:46:03.690522, event: done, op: osd_op(client.4308.0:145084 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.690550 7f9d64bda700 10 osd.2 173 dequeue_op 0x39576f00 finish
2014-06-12 14:46:03.690561 7f9d64bda700  5 -- op tracker -- , seq: 747534, time: 2014-06-12 14:46:03.690557, event: done, op: osd_sub_op_reply(client.4308.0:145084 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.690580 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.690586 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.694409 7f9d3b028700  5 -- op tracker -- , seq: 747535, time: 2014-06-12 14:46:03.693664, event: header_read, op: osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.694424 7f9d3b028700  5 -- op tracker -- , seq: 747535, time: 2014-06-12 14:46:03.693668, event: throttled, op: osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.694432 7f9d3b028700  5 -- op tracker -- , seq: 747535, time: 2014-06-12 14:46:03.694349, event: all_read, op: osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.694446 7f9d3b028700  5 -- op tracker -- , seq: 747535, time: 0.000000, event: dispatched, op: osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.694453 7f9d3b028700 20 osd.2 173 should_share_map client.4308 172.16.96.220:0/1243963 173
2014-06-12 14:46:03.694480 7f9d3b028700 15 osd.2 173 enqueue_op 0x160e4800 prio 63 cost 516096 latency 0.000815 osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.694516 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.694528 7f9d683e1700 10 osd.2 173 dequeue_op 0x160e4800 prio 63 cost 516096 latency 0.000863 osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean]
2014-06-12 14:46:03.694557 7f9d683e1700  5 -- op tracker -- , seq: 747535, time: 2014-06-12 14:46:03.694552, event: reached_pg, op: osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.694569 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.694585 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] handle_message: 0x160e4800
2014-06-12 14:46:03.694597 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] do_op osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.694625 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] get_object_context: found obc in cache: 0x30f32000
2014-06-12 14:46:03.694638 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] get_object_context: 0x30f32000 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 rwstate(write n=1 w=0) oi: 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2928 client.4308.0:145085 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2928) ssc: 0x81bb880 snapset: 0=[]:[]+head
2014-06-12 14:46:03.694656 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] find_object_context 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 @head oi=27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2928 client.4308.0:145085 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2928)
2014-06-12 14:46:03.694684 7f9d683e1700  5 -- op tracker -- , seq: 747535, time: 2014-06-12 14:46:03.694678, event: started, op: osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.694693 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] execute_ctx 0xd21c400
2014-06-12 14:46:03.694712 7f9d683e1700  5 -- op tracker -- , seq: 747535, time: 2014-06-12 14:46:03.694706, event: started, op: osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.694721 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] do_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] ov 173'2928 av 173'2929 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.694737 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] do_osd_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096]
2014-06-12 14:46:03.694750 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.694764 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] do_osd_op  write 1806336~516096
2014-06-12 14:46:03.694782 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] do_osd_op_effects on session 0xa702800
2014-06-12 14:46:03.694796 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 snapset=0x81bb8b8  snapc=0=[]
2014-06-12 14:46:03.694811 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.694823 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] finish_ctx 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 0xd21c400 op modify  
2014-06-12 14:46:03.694836 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean]  set mtime to 2014-06-12 08:45:06.073017
2014-06-12 14:46:03.694860 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean]  final snapset 0=[]:[]+head in 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.694882 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.694895 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] new_repop rep_tid 236206 on osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.694914 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] issue_repop rep_tid 236206 o 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.694944 7f9d683e1700  5 -- op tracker -- , seq: 747535, time: 2014-06-12 14:46:03.694938, event: waiting for subops from 28, op: osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.694966 7f9d683e1700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:03.694989 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2928 (0'0,173'2928] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] append_log log((0'0,173'2928], crt=0'0) [173'2929 (173'2928) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145086 2014-06-12 08:45:06.073017]
2014-06-12 14:46:03.695015 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] add_log_entry 173'2929 (173'2928) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145086 2014-06-12 08:45:06.073017
2014-06-12 14:46:03.695037 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.695069 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2929, trimmed: 
2014-06-12 14:46:03.695093 7f9d683e1700  5 queue_transactions existing osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.695097 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.695099 7f9d683e1700  5 queue_op 0x11344540 seq 2701 osr(3.214 0x276d040) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:03.695107 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] eval_repop repgather(0x534e000 173'2929 rep_tid=236206 committed?=0 applied?=0 lock=0 op=osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.695125 7f9d683e1700 10 osd.2 173 dequeue_op 0x160e4800 finish
2014-06-12 14:46:03.695129 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.695132 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.695141 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5e60 (1 active)
2014-06-12 14:46:03.695146 7f9d8116f700  5 _do_op 0x11344540 seq 2701 osr(3.214 0x276d040)/0x276d040 start
2014-06-12 14:46:03.695151 7f9d8116f700 10 _do_transaction on 0x1ced2000
2014-06-12 14:46:03.695157 7f9d8116f700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.695285 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.695311 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.695362 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.695464 7f9d8116f700 15 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.695472 7f9d8116f700 10 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1 r = 0
2014-06-12 14:46:03.695478 7f9d8116f700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.695498 7f9d8116f700 15 _write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 1806336~516096
2014-06-12 14:46:03.695648 7f9d8116f700 10 lookup_strip_header done  cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.695793 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:03.697052 7f9d5b6bc700  5 -- op tracker -- , seq: 747536, time: 2014-06-12 14:46:03.697006, event: header_read, op: osd_sub_op_reply(client.4308.0:145085 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.697066 7f9d5b6bc700  5 -- op tracker -- , seq: 747536, time: 2014-06-12 14:46:03.697008, event: throttled, op: osd_sub_op_reply(client.4308.0:145085 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.697075 7f9d5b6bc700  5 -- op tracker -- , seq: 747536, time: 2014-06-12 14:46:03.697017, event: all_read, op: osd_sub_op_reply(client.4308.0:145085 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.697083 7f9d5b6bc700  5 -- op tracker -- , seq: 747536, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145085 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.697107 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145085 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.697114 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:03.697124 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39577600 prio 196 cost 0 latency 0.000118 osd_sub_op_reply(client.4308.0:145085 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.697156 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.697167 7f9d65bdc700 10 osd.2 173 dequeue_op 0x39577600 prio 196 cost 0 latency 0.000161 osd_sub_op_reply(client.4308.0:145085 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 pg pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean]
2014-06-12 14:46:03.697191 7f9d65bdc700  5 -- op tracker -- , seq: 747536, time: 2014-06-12 14:46:03.697188, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145085 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.697201 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] handle_message: 0x39577600
2014-06-12 14:46:03.697217 7f9d65bdc700  5 -- op tracker -- , seq: 747536, time: 2014-06-12 14:46:03.697214, event: started, op: osd_sub_op_reply(client.4308.0:145085 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.697224 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] sub_op_modify_reply: tid 236205 op  ack_type 4 from 28
2014-06-12 14:46:03.697246 7f9d65bdc700  5 -- op tracker -- , seq: 747533, time: 2014-06-12 14:46:03.697237, event: sub_op_commit_rec, op: osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.697255 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] repop_all_applied: repop tid 236205 all applied 
2014-06-12 14:46:03.697267 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] eval_repop repgather(0x12ae3200 173'2928 rep_tid=236205 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.697287 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean]  sending ack on repgather(0x12ae3200 173'2928 rep_tid=236205 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x25911180
2014-06-12 14:46:03.697331 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2927 crt=0'0 lcod 173'2926 mlcod 173'2926 active+clean] repop_all_committed: repop tid 236205 all committed 
2014-06-12 14:46:03.697342 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2926 active+clean] eval_repop repgather(0x12ae3200 173'2928 rep_tid=236205 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.697359 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2926 active+clean] log_op_stats osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.029193 lat 0.029222
2014-06-12 14:46:03.697377 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2926 active+clean] publish_stats_to_osd 173:3696
2014-06-12 14:46:03.697391 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2926 active+clean]  sending commit on repgather(0x12ae3200 173'2928 rep_tid=236205 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x24ff900
2014-06-12 14:46:03.697425 7f9d65bdc700  5 -- op tracker -- , seq: 747533, time: 2014-06-12 14:46:03.697419, event: commit_sent, op: osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.697434 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2926 active+clean]  requeue_ops 
2014-06-12 14:46:03.697445 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean]  removing repgather(0x12ae3200 173'2928 rep_tid=236205 committed?=1 applied?=1 op=osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.697460 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean]    q front is repgather(0x12ae3200 173'2928 rep_tid=236205 committed?=1 applied?=1 op=osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.697474 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] remove_repop repgather(0x12ae3200 173'2928 rep_tid=236205 committed?=1 applied?=1 op=osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.697488 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean]  requeue_ops 
2014-06-12 14:46:03.697508 7f9d65bdc700  5 -- op tracker -- , seq: 747533, time: 2014-06-12 14:46:03.697503, event: done, op: osd_op(client.4308.0:145085 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.697526 7f9d65bdc700 10 osd.2 173 dequeue_op 0x39577600 finish
2014-06-12 14:46:03.697534 7f9d65bdc700  5 -- op tracker -- , seq: 747536, time: 2014-06-12 14:46:03.697530, event: done, op: osd_sub_op_reply(client.4308.0:145085 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.697544 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.697547 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.697905 7f9d8116f700 10 _generic_write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 1806336~516096 = 0
2014-06-12 14:46:03.697962 7f9d8116f700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.697978 7f9d8116f700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.697986 7f9d8116f700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.697995 7f9d8116f700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.698208 7f9d8116f700 20 set_header setting 16951 cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 parent seq 0
2014-06-12 14:46:03.698292 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.698309 7f9d8116f700 20 set_header setting 2020 cid meta oid 1fe25aca/pglog_3.214/0//-1 parent seq 0
2014-06-12 14:46:03.698322 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:03.700853 7f9d3b028700  5 -- op tracker -- , seq: 747537, time: 2014-06-12 14:46:03.699812, event: header_read, op: osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.700875 7f9d3b028700  5 -- op tracker -- , seq: 747537, time: 2014-06-12 14:46:03.699817, event: throttled, op: osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.700884 7f9d3b028700  5 -- op tracker -- , seq: 747537, time: 2014-06-12 14:46:03.700794, event: all_read, op: osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.700893 7f9d3b028700  5 -- op tracker -- , seq: 747537, time: 0.000000, event: dispatched, op: osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.700902 7f9d3b028700 20 osd.2 173 should_share_map client.4308 172.16.96.220:0/1243963 173
2014-06-12 14:46:03.700932 7f9d3b028700 15 osd.2 173 enqueue_op 0x160e5100 prio 63 cost 516096 latency 0.001120 osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.700970 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.700983 7f9d6abe6700 10 osd.2 173 dequeue_op 0x160e5100 prio 63 cost 516096 latency 0.001170 osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean]
2014-06-12 14:46:03.701015 7f9d6abe6700  5 -- op tracker -- , seq: 747537, time: 2014-06-12 14:46:03.701009, event: reached_pg, op: osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.701029 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.701046 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] handle_message: 0x160e5100
2014-06-12 14:46:03.701057 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] do_op osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.701089 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] get_object_context: found obc in cache: 0x30f32000
2014-06-12 14:46:03.701103 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] get_object_context: 0x30f32000 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 rwstate(write n=1 w=0) oi: 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2929 client.4308.0:145086 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2929) ssc: 0x81bb880 snapset: 0=[]:[]+head
2014-06-12 14:46:03.701123 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] find_object_context 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 @head oi=27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2929 client.4308.0:145086 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2929)
2014-06-12 14:46:03.701154 7f9d6abe6700  5 -- op tracker -- , seq: 747537, time: 2014-06-12 14:46:03.701148, event: started, op: osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.701165 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] execute_ctx 0x1b264a00
2014-06-12 14:46:03.701184 7f9d6abe6700  5 -- op tracker -- , seq: 747537, time: 2014-06-12 14:46:03.701178, event: started, op: osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.701193 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] do_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] ov 173'2929 av 173'2930 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.701210 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] do_osd_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096]
2014-06-12 14:46:03.701224 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.701240 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] do_osd_op  write 2322432~516096
2014-06-12 14:46:03.701260 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] do_osd_op_effects on session 0xa702800
2014-06-12 14:46:03.701274 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 snapset=0x81bb8b8  snapc=0=[]
2014-06-12 14:46:03.701290 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.701302 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] finish_ctx 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 0x1b264a00 op modify  
2014-06-12 14:46:03.701315 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean]  set mtime to 2014-06-12 08:45:06.079962
2014-06-12 14:46:03.701340 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean]  final snapset 0=[]:[]+head in 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.701361 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.701374 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] new_repop rep_tid 236207 on osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.701394 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] issue_repop rep_tid 236207 o 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.701425 7f9d6abe6700  5 -- op tracker -- , seq: 747537, time: 2014-06-12 14:46:03.701419, event: waiting for subops from 28, op: osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.701451 7f9d6abe6700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:03.701479 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2929 (0'0,173'2929] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] append_log log((0'0,173'2929], crt=0'0) [173'2930 (173'2929) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145087 2014-06-12 08:45:06.079962]
2014-06-12 14:46:03.701507 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] add_log_entry 173'2930 (173'2929) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145087 2014-06-12 08:45:06.079962
2014-06-12 14:46:03.701533 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.701579 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2930, trimmed: 
2014-06-12 14:46:03.701608 7f9d6abe6700  5 queue_transactions existing osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.703376 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.703386 7f9d6abe6700  5 queue_op 0x11004480 seq 2702 osr(3.214 0x276d040) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:03.703402 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5e60 (2 active)
2014-06-12 14:46:03.703402 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] eval_repop repgather(0x11c7e9c0 173'2930 rep_tid=236207 committed?=0 applied?=0 lock=0 op=osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.703432 7f9d6abe6700 10 osd.2 173 dequeue_op 0x160e5100 finish
2014-06-12 14:46:03.703438 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.703442 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.703795 7f9d8116f700 10 _do_op 0x11344540 seq 2701 r = 0, finisher 0x65e1ce0 0x3d91c80
2014-06-12 14:46:03.703807 7f9d8116f700 10 _finish_op 0x11344540 seq 2701 osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.703820 7f9d7d968700  5 _do_op 0x11004480 seq 2702 osr(3.214 0x276d040)/0x276d040 start
2014-06-12 14:46:03.703826 7f9d7d968700 10 _do_transaction on 0x11c7f200
2014-06-12 14:46:03.703825 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] op_applied: 236206
2014-06-12 14:46:03.703836 7f9d7d968700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.703847 7f9d7795c700  5 -- op tracker -- , seq: 747535, time: 2014-06-12 14:46:03.703840, event: op_applied, op: osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.703856 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] op_applied on primary on version 173'2929
2014-06-12 14:46:03.703881 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5e60 (1 active)
2014-06-12 14:46:03.703892 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.703959 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] op_commit: 236206
2014-06-12 14:46:03.703983 7f9d7715b700  5 -- op tracker -- , seq: 747535, time: 2014-06-12 14:46:03.703977, event: op_commit, op: osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.704007 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.704026 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.704078 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.704205 7f9d7d968700 15 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.704212 7f9d7d968700 10 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1 r = 0
2014-06-12 14:46:03.704218 7f9d7d968700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.704243 7f9d7d968700 15 _write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 2322432~516096
2014-06-12 14:46:03.704402 7f9d7d968700 10 lookup_strip_header done  cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.704554 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:03.706994 7f9d7d968700 10 _generic_write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 2322432~516096 = 0
2014-06-12 14:46:03.707093 7f9d7d968700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.707112 7f9d7d968700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.707118 7f9d7d968700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.707126 7f9d7d968700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.707365 7f9d7d968700 20 set_header setting 16951 cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 parent seq 0
2014-06-12 14:46:03.707455 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.707469 7f9d7d968700 20 set_header setting 2020 cid meta oid 1fe25aca/pglog_3.214/0//-1 parent seq 0
2014-06-12 14:46:03.707478 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:03.709446 7f9d56432700  5 -- op tracker -- , seq: 747538, time: 2014-06-12 14:46:03.706587, event: header_read, op: osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.709459 7f9d56432700  5 -- op tracker -- , seq: 747538, time: 2014-06-12 14:46:03.706589, event: throttled, op: osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.709470 7f9d56432700  5 -- op tracker -- , seq: 747538, time: 2014-06-12 14:46:03.709396, event: all_read, op: osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.709479 7f9d56432700  5 -- op tracker -- , seq: 747538, time: 0.000000, event: dispatched, op: osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.709488 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:03.709491 7f9d5b6bc700  5 -- op tracker -- , seq: 747539, time: 2014-06-12 14:46:03.709444, event: header_read, op: osd_sub_op_reply(client.4308.0:145086 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.709497 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:03.709502 7f9d5b6bc700  5 -- op tracker -- , seq: 747539, time: 2014-06-12 14:46:03.709447, event: throttled, op: osd_sub_op_reply(client.4308.0:145086 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.709507 7f9d56432700 15 osd.2 173 enqueue_op 0xe20bd00 prio 127 cost 1549095 latency 0.002919 osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.709510 7f9d5b6bc700  5 -- op tracker -- , seq: 747539, time: 2014-06-12 14:46:03.709456, event: all_read, op: osd_sub_op_reply(client.4308.0:145086 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.709518 7f9d5b6bc700  5 -- op tracker -- , seq: 747539, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145086 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.709532 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145086 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.709537 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:03.709543 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39577a00 prio 196 cost 0 latency 0.000099 osd_sub_op_reply(client.4308.0:145086 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.709548 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.709562 7f9d66bde700 10 osd.2 173 dequeue_op 0xe20bd00 prio 127 cost 1549095 latency 0.002975 osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.25a( v 173'2666 (0'0,173'2666] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2665 active]
2014-06-12 14:46:03.709596 7f9d66bde700  5 -- op tracker -- , seq: 747538, time: 2014-06-12 14:46:03.709591, event: reached_pg, op: osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.709609 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2666 (0'0,173'2666] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2665 active] handle_message: 0xe20bd00
2014-06-12 14:46:03.709631 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2666 (0'0,173'2666] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2665 active] sub_op_modify trans 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 v 173'2667 (transaction) 174
2014-06-12 14:46:03.709646 7f9d66bde700  5 -- op tracker -- , seq: 747538, time: 2014-06-12 14:46:03.709643, event: started, op: osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.709662 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2666 (0'0,173'2666] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2665 active] append_log log((0'0,173'2666], crt=0'0) [173'2667 (173'2666) modify   2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 by client.4311.0:139684 2014-06-12 08:44:56.900304]
2014-06-12 14:46:03.709690 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2667 (0'0,173'2667] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2665 active] add_log_entry 173'2667 (173'2666) modify   2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 by client.4311.0:139684 2014-06-12 08:44:56.900304
2014-06-12 14:46:03.709711 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2667 (0'0,173'2667] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2665 active] append_log  adding 1 keys
2014-06-12 14:46:03.709742 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2667, trimmed: 
2014-06-12 14:46:03.709765 7f9d66bde700  5 -- op tracker -- , seq: 747538, time: 2014-06-12 14:46:03.709760, event: started, op: osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.709774 7f9d66bde700  5 queue_transactions existing osr(3.25a 0x276d340)/0x276d340
2014-06-12 14:46:03.709778 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.709780 7f9d66bde700  5 queue_op 0x782ccc0 seq 2502 osr(3.25a 0x276d340) 1550581 bytes   (queue has 2 ops and 2068970 bytes)
2014-06-12 14:46:03.709790 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x543e480 (2 active)
2014-06-12 14:46:03.709796 7f9d8016d700  5 _do_op 0x782ccc0 seq 2502 osr(3.25a 0x276d340)/0x276d340 start
2014-06-12 14:46:03.709787 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.25a( v 173'2667 (0'0,173'2667] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2665 active] do_sub_op osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.709800 7f9d8016d700 10 _do_transaction on 0x274a7cf0
2014-06-12 14:46:03.709800 7f9d66bde700 10 osd.2 173 dequeue_op 0xe20bd00 finish
2014-06-12 14:46:03.709806 7f9d8016d700 15 _omap_setkeys meta/1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:46:03.709804 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.709811 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.709908 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:46:03.709931 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.709965 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.710085 7f9d8016d700 15 _omap_rmkeys meta/1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:46:03.710093 7f9d8016d700 10 _omap_rmkeys meta/1fe2073b/pglog_3.25a/0//-1 r = 0
2014-06-12 14:46:03.710100 7f9d8016d700 15 _omap_setkeys meta/1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:46:03.710113 7f9d8016d700 15 _write 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 1437696~1548288
2014-06-12 14:46:03.710161 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.710168 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x39577a00 prio 196 cost 0 latency 0.000724 osd_sub_op_reply(client.4308.0:145086 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 pg pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean]
2014-06-12 14:46:03.710189 7f9d6a3e5700  5 -- op tracker -- , seq: 747539, time: 2014-06-12 14:46:03.710185, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145086 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.710197 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] handle_message: 0x39577a00
2014-06-12 14:46:03.710212 7f9d6a3e5700  5 -- op tracker -- , seq: 747539, time: 2014-06-12 14:46:03.710209, event: started, op: osd_sub_op_reply(client.4308.0:145086 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.710217 7f9d8016d700 10 lookup_strip_header done  cid 3.25a_head oid 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:03.710218 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] sub_op_modify_reply: tid 236206 op  ack_type 4 from 28
2014-06-12 14:46:03.710239 7f9d6a3e5700  5 -- op tracker -- , seq: 747535, time: 2014-06-12 14:46:03.710231, event: sub_op_commit_rec, op: osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.710246 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] repop_all_applied: repop tid 236206 all applied 
2014-06-12 14:46:03.710257 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] eval_repop repgather(0x534e000 173'2929 rep_tid=236206 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.710273 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean]  sending ack on repgather(0x534e000 173'2929 rep_tid=236206 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x258d0500
2014-06-12 14:46:03.710306 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2928 crt=0'0 lcod 173'2927 mlcod 173'2927 active+clean] repop_all_committed: repop tid 236206 all committed 
2014-06-12 14:46:03.710317 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2927 active+clean] eval_repop repgather(0x534e000 173'2929 rep_tid=236206 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.710333 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2927 active+clean] log_op_stats osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.016641 lat 0.016667
2014-06-12 14:46:03.710350 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2927 active+clean] publish_stats_to_osd 173:3697
2014-06-12 14:46:03.710364 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2927 active+clean]  sending commit on repgather(0x534e000 173'2929 rep_tid=236206 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x912a000
2014-06-12 14:46:03.710393 7f9d6a3e5700  5 -- op tracker -- , seq: 747535, time: 2014-06-12 14:46:03.710388, event: commit_sent, op: osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.710401 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2927 active+clean]  requeue_ops 
2014-06-12 14:46:03.710423 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:03.710412 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean]  removing repgather(0x534e000 173'2929 rep_tid=236206 committed?=1 applied?=1 op=osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.710430 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean]    q front is repgather(0x534e000 173'2929 rep_tid=236206 committed?=1 applied?=1 op=osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.710448 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] remove_repop repgather(0x534e000 173'2929 rep_tid=236206 committed?=1 applied?=1 op=osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.710462 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean]  requeue_ops 
2014-06-12 14:46:03.710480 7f9d6a3e5700  5 -- op tracker -- , seq: 747535, time: 2014-06-12 14:46:03.710475, event: done, op: osd_op(client.4308.0:145086 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.710497 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x39577a00 finish
2014-06-12 14:46:03.710504 7f9d6a3e5700  5 -- op tracker -- , seq: 747539, time: 2014-06-12 14:46:03.710500, event: done, op: osd_sub_op_reply(client.4308.0:145086 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.710514 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.710516 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.712538 7f9d5b6bc700  5 -- op tracker -- , seq: 747540, time: 2014-06-12 14:46:03.712417, event: header_read, op: osd_sub_op_reply(client.4308.0:145087 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.712556 7f9d5b6bc700  5 -- op tracker -- , seq: 747540, time: 2014-06-12 14:46:03.712422, event: throttled, op: osd_sub_op_reply(client.4308.0:145087 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.712564 7f9d5b6bc700  5 -- op tracker -- , seq: 747540, time: 2014-06-12 14:46:03.712468, event: all_read, op: osd_sub_op_reply(client.4308.0:145087 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.712572 7f9d5b6bc700  5 -- op tracker -- , seq: 747540, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145087 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.712579 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145087 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.712587 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:03.712599 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39577400 prio 196 cost 0 latency 0.000181 osd_sub_op_reply(client.4308.0:145087 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.712648 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.712658 7f9d653db700 10 osd.2 173 dequeue_op 0x39577400 prio 196 cost 0 latency 0.000240 osd_sub_op_reply(client.4308.0:145087 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 pg pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean]
2014-06-12 14:46:03.712685 7f9d653db700  5 -- op tracker -- , seq: 747540, time: 2014-06-12 14:46:03.712681, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145087 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.712694 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] handle_message: 0x39577400
2014-06-12 14:46:03.712712 7f9d653db700  5 -- op tracker -- , seq: 747540, time: 2014-06-12 14:46:03.712708, event: started, op: osd_sub_op_reply(client.4308.0:145087 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.712718 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] sub_op_modify_reply: tid 236207 op  ack_type 4 from 28
2014-06-12 14:46:03.712742 7f9d653db700  5 -- op tracker -- , seq: 747537, time: 2014-06-12 14:46:03.712732, event: sub_op_commit_rec, op: osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.712751 7f9d653db700 10 osd.2 173 dequeue_op 0x39577400 finish
2014-06-12 14:46:03.712758 7f9d653db700  5 -- op tracker -- , seq: 747540, time: 2014-06-12 14:46:03.712754, event: done, op: osd_sub_op_reply(client.4308.0:145087 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.712774 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.712777 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.714274 7f9d3b028700  5 -- op tracker -- , seq: 747541, time: 2014-06-12 14:46:03.713263, event: header_read, op: osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.714288 7f9d3b028700  5 -- op tracker -- , seq: 747541, time: 2014-06-12 14:46:03.713266, event: throttled, op: osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.714298 7f9d3b028700  5 -- op tracker -- , seq: 747541, time: 2014-06-12 14:46:03.714210, event: all_read, op: osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.714308 7f9d3b028700  5 -- op tracker -- , seq: 747541, time: 0.000000, event: dispatched, op: osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.714317 7f9d3b028700 20 osd.2 173 should_share_map client.4308 172.16.96.220:0/1243963 173
2014-06-12 14:46:03.714350 7f9d3b028700 15 osd.2 173 enqueue_op 0x160e4300 prio 63 cost 516096 latency 0.001086 osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.714957 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.714968 7f9d63bd8700 10 osd.2 173 dequeue_op 0x160e4300 prio 63 cost 516096 latency 0.001705 osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean]
2014-06-12 14:46:03.714997 7f9d63bd8700  5 -- op tracker -- , seq: 747541, time: 2014-06-12 14:46:03.714991, event: reached_pg, op: osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.715009 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.715025 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] handle_message: 0x160e4300
2014-06-12 14:46:03.715036 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] do_op osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.715061 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] get_object_context: found obc in cache: 0x30f32000
2014-06-12 14:46:03.715073 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] get_object_context: 0x30f32000 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 rwstate(write n=1 w=0) oi: 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2930 client.4308.0:145087 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2930) ssc: 0x81bb880 snapset: 0=[]:[]+head
2014-06-12 14:46:03.715091 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] find_object_context 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 @head oi=27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2930 client.4308.0:145087 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2930)
2014-06-12 14:46:03.715123 7f9d63bd8700  5 -- op tracker -- , seq: 747541, time: 2014-06-12 14:46:03.715117, event: started, op: osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.715133 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] execute_ctx 0xa04f800
2014-06-12 14:46:03.715150 7f9d63bd8700  5 -- op tracker -- , seq: 747541, time: 2014-06-12 14:46:03.715145, event: started, op: osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.715159 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] do_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] ov 173'2930 av 173'2931 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.715174 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] do_osd_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096]
2014-06-12 14:46:03.715186 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.715198 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] do_osd_op  write 2838528~516096
2014-06-12 14:46:03.715215 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] do_osd_op_effects on session 0xa702800
2014-06-12 14:46:03.715228 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 snapset=0x81bb8b8  snapc=0=[]
2014-06-12 14:46:03.715240 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.715251 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] finish_ctx 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 0xa04f800 op modify  
2014-06-12 14:46:03.715264 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean]  set mtime to 2014-06-12 08:45:06.093232
2014-06-12 14:46:03.715288 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean]  final snapset 0=[]:[]+head in 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.715310 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.715322 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] new_repop rep_tid 236208 on osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.715339 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] issue_repop rep_tid 236208 o 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.715369 7f9d63bd8700  5 -- op tracker -- , seq: 747541, time: 2014-06-12 14:46:03.715364, event: waiting for subops from 28, op: osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.715400 7f9d63bd8700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:03.715424 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2930 (0'0,173'2930] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] append_log log((0'0,173'2930], crt=0'0) [173'2931 (173'2930) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145088 2014-06-12 08:45:06.093232]
2014-06-12 14:46:03.715449 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] add_log_entry 173'2931 (173'2930) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145088 2014-06-12 08:45:06.093232
2014-06-12 14:46:03.715469 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.715504 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2931, trimmed: 
2014-06-12 14:46:03.715525 7f9d63bd8700  5 queue_transactions existing osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.715529 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.715531 7f9d63bd8700  5 queue_op 0x9bac9c0 seq 2703 osr(3.214 0x276d040) 518389 bytes   (queue has 3 ops and 2587359 bytes)
2014-06-12 14:46:03.715541 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] eval_repop repgather(0x2237200 173'2931 rep_tid=236208 committed?=0 applied?=0 lock=0 op=osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.715560 7f9d63bd8700 10 osd.2 173 dequeue_op 0x160e4300 finish
2014-06-12 14:46:03.715564 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.715567 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.717529 7f9d8016d700 10 _generic_write 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 1437696~1548288 = 0
2014-06-12 14:46:03.717840 7f9d7d968700 10 _do_op 0x11004480 seq 2702 r = 0, finisher 0x6f5f560 0xe9d67c0
2014-06-12 14:46:03.717861 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] op_commit: 236207
2014-06-12 14:46:03.717907 7f9d7715b700  5 -- op tracker -- , seq: 747537, time: 2014-06-12 14:46:03.717893, event: op_commit, op: osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.717927 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2929 crt=0'0 lcod 173'2928 mlcod 173'2928 active+clean] repop_all_committed: repop tid 236207 all committed 
2014-06-12 14:46:03.717938 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2928 active+clean] eval_repop repgather(0x11c7e9c0 173'2930 rep_tid=236207 committed?=1 applied?=0 lock=0 op=osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.717973 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2928 active+clean] log_op_stats osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.018159
2014-06-12 14:46:03.717997 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2928 active+clean] publish_stats_to_osd 173:3698
2014-06-12 14:46:03.718007 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2928 active+clean]  sending commit on repgather(0x11c7e9c0 173'2930 rep_tid=236207 committed?=1 applied?=0 lock=0 op=osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x30ccb180
2014-06-12 14:46:03.718051 7f9d7715b700  5 -- op tracker -- , seq: 747537, time: 2014-06-12 14:46:03.718046, event: commit_sent, op: osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.718063 7f9d7d968700 10 _finish_op 0x11004480 seq 2702 osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.718072 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5e60 (1 active)
2014-06-12 14:46:03.718082 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5e60 (2 active)
2014-06-12 14:46:03.718084 7f9d7d968700  5 _do_op 0x9bac9c0 seq 2703 osr(3.214 0x276d040)/0x276d040 start
2014-06-12 14:46:03.718088 7f9d7d968700 10 _do_transaction on 0x5361740
2014-06-12 14:46:03.718094 7f9d7d968700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.718229 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.718247 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.718416 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2928 active+clean] op_applied: 236207
2014-06-12 14:46:03.718440 7f9d7795c700  5 -- op tracker -- , seq: 747537, time: 2014-06-12 14:46:03.718434, event: op_applied, op: osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.718449 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2928 active+clean] op_applied on primary on version 173'2930
2014-06-12 14:46:03.718459 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2928 active+clean] repop_all_applied: repop tid 236207 all applied 
2014-06-12 14:46:03.718468 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2928 active+clean] eval_repop repgather(0x11c7e9c0 173'2930 rep_tid=236207 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.718507 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2928 active+clean] log_op_stats osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.018694
2014-06-12 14:46:03.718523 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2928 active+clean] publish_stats_to_osd 173:3699
2014-06-12 14:46:03.718533 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2928 active+clean]  requeue_ops 
2014-06-12 14:46:03.718542 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean]  removing repgather(0x11c7e9c0 173'2930 rep_tid=236207 committed?=1 applied?=1 op=osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.718557 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean]    q front is repgather(0x11c7e9c0 173'2930 rep_tid=236207 committed?=1 applied?=1 op=osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.718571 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] remove_repop repgather(0x11c7e9c0 173'2930 rep_tid=236207 committed?=1 applied?=1 op=osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.718587 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean]  requeue_ops 
2014-06-12 14:46:03.718613 7f9d7795c700  5 -- op tracker -- , seq: 747537, time: 2014-06-12 14:46:03.718603, event: done, op: osd_op(client.4308.0:145087 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2322432~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.718648 7f9d8016d700 15 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:03.718665 7f9d8016d700 10 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:03.718670 7f9d8016d700 15 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:03.718678 7f9d8016d700 10 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:03.719282 7f9d8016d700 20 set_header setting 16076 cid 3.25a_head oid 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:46:03.719439 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.719452 7f9d8016d700 20 set_header setting 2302 cid meta oid 1fe2073b/pglog_3.25a/0//-1 parent seq 0
2014-06-12 14:46:03.719459 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:03.719575 7f9d3b028700  5 -- op tracker -- , seq: 747542, time: 2014-06-12 14:46:03.719455, event: header_read, op: osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.719588 7f9d3b028700  5 -- op tracker -- , seq: 747542, time: 2014-06-12 14:46:03.719459, event: throttled, op: osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.719598 7f9d3b028700  5 -- op tracker -- , seq: 747542, time: 2014-06-12 14:46:03.719489, event: all_read, op: osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.719606 7f9d3b028700  5 -- op tracker -- , seq: 747542, time: 0.000000, event: dispatched, op: osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.719614 7f9d3b028700 20 osd.2 173 should_share_map client.4308 172.16.96.220:0/1243963 173
2014-06-12 14:46:03.719645 7f9d3b028700 15 osd.2 173 enqueue_op 0x160e5700 prio 63 cost 4096 latency 0.000190 osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.719681 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.719689 7f9d69be4700 10 osd.2 173 dequeue_op 0x160e5700 prio 63 cost 4096 latency 0.000234 osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean]
2014-06-12 14:46:03.719715 7f9d69be4700  5 -- op tracker -- , seq: 747542, time: 2014-06-12 14:46:03.719710, event: reached_pg, op: osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.719723 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.719738 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] handle_message: 0x160e5700
2014-06-12 14:46:03.719748 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] do_op osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.719772 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] get_object_context: found obc in cache: 0x30f32000
2014-06-12 14:46:03.719783 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] get_object_context: 0x30f32000 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 rwstate(write n=1 w=0) oi: 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2931 client.4308.0:145088 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2931) ssc: 0x81bb880 snapset: 0=[]:[]+head
2014-06-12 14:46:03.719798 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] find_object_context 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 @head oi=27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2931 client.4308.0:145088 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2931)
2014-06-12 14:46:03.719824 7f9d69be4700  5 -- op tracker -- , seq: 747542, time: 2014-06-12 14:46:03.719819, event: started, op: osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.719831 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] execute_ctx 0x292d000
2014-06-12 14:46:03.719848 7f9d69be4700  5 -- op tracker -- , seq: 747542, time: 2014-06-12 14:46:03.719843, event: started, op: osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.719855 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] do_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] ov 173'2931 av 173'2932 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.719868 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] do_osd_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096]
2014-06-12 14:46:03.719879 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.719891 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] do_osd_op  write 3354624~4096
2014-06-12 14:46:03.719907 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] do_osd_op_effects on session 0xa702800
2014-06-12 14:46:03.719918 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 snapset=0x81bb8b8  snapc=0=[]
2014-06-12 14:46:03.719930 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.719939 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] finish_ctx 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 0x292d000 op modify  
2014-06-12 14:46:03.719951 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean]  set mtime to 2014-06-12 08:45:06.100510
2014-06-12 14:46:03.719972 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean]  final snapset 0=[]:[]+head in 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.719992 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.720002 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] new_repop rep_tid 236209 on osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.720018 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] issue_repop rep_tid 236209 o 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.720044 7f9d69be4700  5 -- op tracker -- , seq: 747542, time: 2014-06-12 14:46:03.720039, event: waiting for subops from 28, op: osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.720071 7f9d69be4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:03.720094 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2931 (0'0,173'2931] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] append_log log((0'0,173'2931], crt=0'0) [173'2932 (173'2931) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145089 2014-06-12 08:45:06.100510]
2014-06-12 14:46:03.720121 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] add_log_entry 173'2932 (173'2931) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145089 2014-06-12 08:45:06.100510
2014-06-12 14:46:03.720141 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.720172 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2932, trimmed: 
2014-06-12 14:46:03.720196 7f9d69be4700  5 queue_transactions existing osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.720201 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.720203 7f9d69be4700  5 queue_op 0x4b2f1a0 seq 2704 osr(3.214 0x276d040) 6389 bytes   (queue has 3 ops and 2075359 bytes)
2014-06-12 14:46:03.720221 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5e60 (3 active)
2014-06-12 14:46:03.720213 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] eval_repop repgather(0xe832b40 173'2932 rep_tid=236209 committed?=0 applied?=0 lock=0 op=osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.720231 7f9d69be4700 10 osd.2 173 dequeue_op 0x160e5700 finish
2014-06-12 14:46:03.720235 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.720238 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.721628 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.725712 7f9d5b6bc700  5 -- op tracker -- , seq: 747543, time: 2014-06-12 14:46:03.725665, event: header_read, op: osd_sub_op_reply(client.4308.0:145088 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.725723 7f9d5b6bc700  5 -- op tracker -- , seq: 747543, time: 2014-06-12 14:46:03.725667, event: throttled, op: osd_sub_op_reply(client.4308.0:145088 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.725732 7f9d5b6bc700  5 -- op tracker -- , seq: 747543, time: 2014-06-12 14:46:03.725679, event: all_read, op: osd_sub_op_reply(client.4308.0:145088 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.725740 7f9d5b6bc700  5 -- op tracker -- , seq: 747543, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145088 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.725744 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145088 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.725750 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:03.725758 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39576600 prio 196 cost 0 latency 0.000093 osd_sub_op_reply(client.4308.0:145088 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.725811 7f9d5b6bc700  5 -- op tracker -- , seq: 747544, time: 2014-06-12 14:46:03.725779, event: header_read, op: osd_sub_op_reply(client.4308.0:145089 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.725820 7f9d5b6bc700  5 -- op tracker -- , seq: 747544, time: 2014-06-12 14:46:03.725779, event: throttled, op: osd_sub_op_reply(client.4308.0:145089 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.725828 7f9d5b6bc700  5 -- op tracker -- , seq: 747544, time: 2014-06-12 14:46:03.725791, event: all_read, op: osd_sub_op_reply(client.4308.0:145089 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.725836 7f9d5b6bc700  5 -- op tracker -- , seq: 747544, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145089 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.725839 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145089 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.725844 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:03.725849 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39576b00 prio 196 cost 0 latency 0.000070 osd_sub_op_reply(client.4308.0:145089 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.725880 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.725887 7f9d633d7700 10 osd.2 173 dequeue_op 0x39576600 prio 196 cost 0 latency 0.000222 osd_sub_op_reply(client.4308.0:145088 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 pg pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean]
2014-06-12 14:46:03.725908 7f9d633d7700  5 -- op tracker -- , seq: 747543, time: 2014-06-12 14:46:03.725904, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145088 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.725916 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] handle_message: 0x39576600
2014-06-12 14:46:03.725931 7f9d633d7700  5 -- op tracker -- , seq: 747543, time: 2014-06-12 14:46:03.725928, event: started, op: osd_sub_op_reply(client.4308.0:145088 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.725937 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] sub_op_modify_reply: tid 236208 op  ack_type 4 from 28
2014-06-12 14:46:03.725959 7f9d633d7700  5 -- op tracker -- , seq: 747541, time: 2014-06-12 14:46:03.725951, event: sub_op_commit_rec, op: osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.725967 7f9d633d7700 10 osd.2 173 dequeue_op 0x39576600 finish
2014-06-12 14:46:03.725973 7f9d633d7700  5 -- op tracker -- , seq: 747543, time: 2014-06-12 14:46:03.725970, event: done, op: osd_sub_op_reply(client.4308.0:145088 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.725986 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.725991 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.725995 7f9d633d7700 10 osd.2 173 dequeue_op 0x39576b00 prio 196 cost 0 latency 0.000216 osd_sub_op_reply(client.4308.0:145089 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 pg pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean]
2014-06-12 14:46:03.726011 7f9d633d7700  5 -- op tracker -- , seq: 747544, time: 2014-06-12 14:46:03.726007, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145089 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.726016 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] handle_message: 0x39576b00
2014-06-12 14:46:03.726030 7f9d633d7700  5 -- op tracker -- , seq: 747544, time: 2014-06-12 14:46:03.726026, event: started, op: osd_sub_op_reply(client.4308.0:145089 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.726036 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] sub_op_modify_reply: tid 236209 op  ack_type 4 from 28
2014-06-12 14:46:03.726054 7f9d633d7700  5 -- op tracker -- , seq: 747542, time: 2014-06-12 14:46:03.726048, event: sub_op_commit_rec, op: osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.726061 7f9d633d7700 10 osd.2 173 dequeue_op 0x39576b00 finish
2014-06-12 14:46:03.726067 7f9d633d7700  5 -- op tracker -- , seq: 747544, time: 2014-06-12 14:46:03.726064, event: done, op: osd_sub_op_reply(client.4308.0:145089 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.726077 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.726080 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.727813 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.737990 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.738187 7f9d7d968700 15 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.738196 7f9d7d968700 10 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1 r = 0
2014-06-12 14:46:03.738203 7f9d7d968700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.738225 7f9d7d968700 15 _write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 2838528~516096
2014-06-12 14:46:03.738387 7f9d7d968700 10 lookup_strip_header done  cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.738572 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:03.740617 7f9d7d968700 10 _generic_write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 2838528~516096 = 0
2014-06-12 14:46:03.742277 7f9d7d968700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.742301 7f9d7d968700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.742307 7f9d7d968700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.742315 7f9d7d968700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.742599 7f9d7d968700 20 set_header setting 16951 cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 parent seq 0
2014-06-12 14:46:03.742671 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.742685 7f9d7d968700 20 set_header setting 2020 cid meta oid 1fe25aca/pglog_3.214/0//-1 parent seq 0
2014-06-12 14:46:03.742695 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:03.744720 7f9d7d968700 10 _do_op 0x9bac9c0 seq 2703 r = 0, finisher 0x824d360 0x1a3f1340
2014-06-12 14:46:03.745358 7f9d8016d700 10 _do_op 0x782ccc0 seq 2502 r = 0, finisher 0x31386b80 0
2014-06-12 14:46:03.745381 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] op_commit: 236208
2014-06-12 14:46:03.745414 7f9d7715b700  5 -- op tracker -- , seq: 747541, time: 2014-06-12 14:46:03.745405, event: op_commit, op: osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.745423 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2930 crt=0'0 lcod 173'2929 mlcod 173'2929 active+clean] repop_all_committed: repop tid 236208 all committed 
2014-06-12 14:46:03.745435 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2929 active+clean] eval_repop repgather(0x2237200 173'2931 rep_tid=236208 committed?=1 applied?=0 lock=0 op=osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.745453 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2929 active+clean] log_op_stats osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.032189
2014-06-12 14:46:03.745472 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2929 active+clean] publish_stats_to_osd 173:3700
2014-06-12 14:46:03.745483 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2929 active+clean]  sending commit on repgather(0x2237200 173'2931 rep_tid=236208 committed?=1 applied?=0 lock=0 op=osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0xb955680
2014-06-12 14:46:03.745524 7f9d7715b700  5 -- op tracker -- , seq: 747541, time: 2014-06-12 14:46:03.745519, event: commit_sent, op: osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.745555 7f9d7715b700  5 -- op tracker -- , seq: 747538, time: 2014-06-12 14:46:03.745548, event: commit_sent, op: osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.745563 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2667 (0'0,173'2667] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2665 active] sub_op_modify_commit on op osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:03.745588 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:03.745649 7f9d8016d700 10 _finish_op 0x782ccc0 seq 2502 osr(3.25a 0x276d340)/0x276d340
2014-06-12 14:46:03.745676 7f9d7795c700  5 -- op tracker -- , seq: 747538, time: 2014-06-12 14:46:03.745670, event: sub_op_applied, op: osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.745691 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2667 (0'0,173'2667] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2666 active] sub_op_modify_applied on 0x274a7c00 op osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.745709 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2667 (0'0,173'2667] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2666 active] op_applied on primary on version 173'2667
2014-06-12 14:46:03.745721 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2667 (0'0,173'2667] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2666 active] op_applied on replica on version 173'2667
2014-06-12 14:46:03.745755 7f9d8016d700  5 -- op tracker -- , seq: 747538, time: 2014-06-12 14:46:03.745747, event: done, op: osd_sub_op(client.4311.0:139684 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.745790 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x543e480 (2 active)
2014-06-12 14:46:03.745797 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.745805 7f9d7d968700 10 _finish_op 0x9bac9c0 seq 2703 osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.745813 7f9d7915f700  5 _do_op 0x4b2f1a0 seq 2704 osr(3.214 0x276d040)/0x276d040 start
2014-06-12 14:46:03.745821 7f9d7915f700 10 _do_transaction on 0xe833140
2014-06-12 14:46:03.745829 7f9d7915f700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.745819 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2929 active+clean] op_applied: 236208
2014-06-12 14:46:03.745839 7f9d7795c700  5 -- op tracker -- , seq: 747541, time: 2014-06-12 14:46:03.745833, event: op_applied, op: osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.745852 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2929 active+clean] op_applied on primary on version 173'2931
2014-06-12 14:46:03.745863 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2929 active+clean] repop_all_applied: repop tid 236208 all applied 
2014-06-12 14:46:03.745873 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2929 active+clean] eval_repop repgather(0x2237200 173'2931 rep_tid=236208 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.745890 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2929 active+clean] log_op_stats osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.032626
2014-06-12 14:46:03.745906 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2929 active+clean] publish_stats_to_osd 173:3701
2014-06-12 14:46:03.745954 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.745976 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.745918 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2929 active+clean]  requeue_ops 
2014-06-12 14:46:03.745997 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2930 active+clean]  removing repgather(0x2237200 173'2931 rep_tid=236208 committed?=1 applied?=1 op=osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.746034 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.746012 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2930 active+clean]    q front is repgather(0x2237200 173'2931 rep_tid=236208 committed?=1 applied?=1 op=osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.746038 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2930 active+clean] remove_repop repgather(0x2237200 173'2931 rep_tid=236208 committed?=1 applied?=1 op=osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.746053 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2930 active+clean]  requeue_ops 
2014-06-12 14:46:03.746065 7f9d7915f700 15 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.746071 7f9d7915f700 10 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1 r = 0
2014-06-12 14:46:03.746078 7f9d7915f700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.746101 7f9d7d968700  5 -- op tracker -- , seq: 747541, time: 2014-06-12 14:46:03.746088, event: done, op: osd_op(client.4308.0:145088 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.746140 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5e60 (1 active)
2014-06-12 14:46:03.746144 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.746255 7f9d7915f700 15 _write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 3354624~4096
2014-06-12 14:46:03.746402 7f9d7915f700 10 lookup_strip_header done  cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.746415 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:03.746447 7f9d7915f700 10 _generic_write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 3354624~4096 = 0
2014-06-12 14:46:03.746457 7f9d7915f700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.746471 7f9d7915f700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.746476 7f9d7915f700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.746485 7f9d7915f700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.746590 7f9d7915f700 20 set_header setting 16951 cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 parent seq 0
2014-06-12 14:46:03.746614 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.746627 7f9d7915f700 20 set_header setting 2020 cid meta oid 1fe25aca/pglog_3.214/0//-1 parent seq 0
2014-06-12 14:46:03.746637 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:03.746850 7f9d7915f700 10 _do_op 0x4b2f1a0 seq 2704 r = 0, finisher 0x57bc820 0x39f50400
2014-06-12 14:46:03.746861 7f9d7915f700 10 _finish_op 0x4b2f1a0 seq 2704 osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.746873 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5e60 (0 active)
2014-06-12 14:46:03.746876 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.746877 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2930 active+clean] op_commit: 236209
2014-06-12 14:46:03.746907 7f9d7715b700  5 -- op tracker -- , seq: 747542, time: 2014-06-12 14:46:03.746900, event: op_commit, op: osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.746914 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2931 crt=0'0 lcod 173'2930 mlcod 173'2930 active+clean] repop_all_committed: repop tid 236209 all committed 
2014-06-12 14:46:03.746925 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2930 active+clean] eval_repop repgather(0xe832b40 173'2932 rep_tid=236209 committed?=1 applied?=0 lock=0 op=osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.746943 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2930 active+clean] log_op_stats osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 inb 4375 outb 0 rlat 0.000000 lat 0.027486
2014-06-12 14:46:03.746963 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2930 active+clean] publish_stats_to_osd 173:3702
2014-06-12 14:46:03.746974 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2930 active+clean]  sending commit on repgather(0xe832b40 173'2932 rep_tid=236209 committed?=1 applied?=0 lock=0 op=osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0xe185900
2014-06-12 14:46:03.747009 7f9d7715b700  5 -- op tracker -- , seq: 747542, time: 2014-06-12 14:46:03.747003, event: commit_sent, op: osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.747069 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2930 active+clean] op_applied: 236209
2014-06-12 14:46:03.747090 7f9d7795c700  5 -- op tracker -- , seq: 747542, time: 2014-06-12 14:46:03.747084, event: op_applied, op: osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.747099 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2930 active+clean] op_applied on primary on version 173'2932
2014-06-12 14:46:03.747110 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2930 active+clean] repop_all_applied: repop tid 236209 all applied 
2014-06-12 14:46:03.747118 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2930 active+clean] eval_repop repgather(0xe832b40 173'2932 rep_tid=236209 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.747133 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2930 active+clean] log_op_stats osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 inb 4375 outb 0 rlat 0.000000 lat 0.027678
2014-06-12 14:46:03.747144 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2930 active+clean] publish_stats_to_osd 173:3703
2014-06-12 14:46:03.747152 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2930 active+clean]  requeue_ops 
2014-06-12 14:46:03.747162 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean]  removing repgather(0xe832b40 173'2932 rep_tid=236209 committed?=1 applied?=1 op=osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.747176 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean]    q front is repgather(0xe832b40 173'2932 rep_tid=236209 committed?=1 applied?=1 op=osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.747190 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] remove_repop repgather(0xe832b40 173'2932 rep_tid=236209 committed?=1 applied?=1 op=osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.747204 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean]  requeue_ops 
2014-06-12 14:46:03.747226 7f9d7795c700  5 -- op tracker -- , seq: 747542, time: 2014-06-12 14:46:03.747222, event: done, op: osd_op(client.4308.0:145089 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~4096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.749223 7f9d3b028700  5 -- op tracker -- , seq: 747545, time: 2014-06-12 14:46:03.748045, event: header_read, op: osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.749236 7f9d3b028700  5 -- op tracker -- , seq: 747545, time: 2014-06-12 14:46:03.748049, event: throttled, op: osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.749246 7f9d3b028700  5 -- op tracker -- , seq: 747545, time: 2014-06-12 14:46:03.749175, event: all_read, op: osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.749255 7f9d3b028700  5 -- op tracker -- , seq: 747545, time: 0.000000, event: dispatched, op: osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.749262 7f9d3b028700 20 osd.2 173 should_share_map client.4308 172.16.96.220:0/1243963 173
2014-06-12 14:46:03.749290 7f9d3b028700 15 osd.2 173 enqueue_op 0x160e5b00 prio 63 cost 516096 latency 0.001245 osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.749934 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.749949 7f9d68be2700 10 osd.2 173 dequeue_op 0x160e5b00 prio 63 cost 516096 latency 0.001903 osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean]
2014-06-12 14:46:03.749976 7f9d68be2700  5 -- op tracker -- , seq: 747545, time: 2014-06-12 14:46:03.749971, event: reached_pg, op: osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.749987 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.750008 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] handle_message: 0x160e5b00
2014-06-12 14:46:03.750018 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] do_op osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.750039 7f9d68be2700 15 getattr 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 '_'
2014-06-12 14:46:03.750266 7f9d68be2700 10 getattr 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 '_' = 0
2014-06-12 14:46:03.750279 7f9d68be2700 15 getattr 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 'snapset'
2014-06-12 14:46:03.750360 7f9d68be2700 10 getattr 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 'snapset' = 0
2014-06-12 14:46:03.750367 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] populate_obc_watchers 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.750378 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.750387 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] get_object_context: creating obc from disk: 0x16962840
2014-06-12 14:46:03.750395 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] get_object_context: 0x16962840 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 rwstate(none n=0 w=0) oi: 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2932 client.4308.0:145089 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2932) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:46:03.750409 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] find_object_context 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 @head oi=27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2932 client.4308.0:145089 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2932)
2014-06-12 14:46:03.750429 7f9d68be2700  5 -- op tracker -- , seq: 747545, time: 2014-06-12 14:46:03.750424, event: started, op: osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.750437 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] execute_ctx 0x2963b200
2014-06-12 14:46:03.750452 7f9d68be2700  5 -- op tracker -- , seq: 747545, time: 2014-06-12 14:46:03.750448, event: started, op: osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.750461 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] do_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] ov 173'2932 av 173'2933 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.750475 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] do_osd_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096]
2014-06-12 14:46:03.750485 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.750497 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] do_osd_op  write 3358720~516096
2014-06-12 14:46:03.750510 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] do_osd_op_effects on session 0xa702800
2014-06-12 14:46:03.750519 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:46:03.750533 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.750542 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] finish_ctx 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 0x2963b200 op modify  
2014-06-12 14:46:03.750553 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean]  set mtime to 2014-06-12 08:45:06.128218
2014-06-12 14:46:03.750570 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean]  final snapset 0=[]:[]+head in 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.750586 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.750595 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] new_repop rep_tid 236210 on osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.750609 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] issue_repop rep_tid 236210 o 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.750628 7f9d68be2700  5 -- op tracker -- , seq: 747545, time: 2014-06-12 14:46:03.750623, event: waiting for subops from 28, op: osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.750651 7f9d68be2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:03.750670 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2932 (0'0,173'2932] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] append_log log((0'0,173'2932], crt=0'0) [173'2933 (173'2932) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145090 2014-06-12 08:45:06.128218]
2014-06-12 14:46:03.750690 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] add_log_entry 173'2933 (173'2932) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145090 2014-06-12 08:45:06.128218
2014-06-12 14:46:03.750709 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.750738 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2933, trimmed: 
2014-06-12 14:46:03.750762 7f9d68be2700  5 queue_transactions existing osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.750766 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.750767 7f9d68be2700  5 queue_op 0x139d9a40 seq 2705 osr(3.214 0x276d040) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:03.750782 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] eval_repop repgather(0x51dc180 173'2933 rep_tid=236210 committed?=0 applied?=0 lock=0 op=osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.750799 7f9d68be2700 10 osd.2 173 dequeue_op 0x160e5b00 finish
2014-06-12 14:46:03.750803 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.750805 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.750813 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5e60 (1 active)
2014-06-12 14:46:03.750818 7f9d7c165700  5 _do_op 0x139d9a40 seq 2705 osr(3.214 0x276d040)/0x276d040 start
2014-06-12 14:46:03.750822 7f9d7c165700 10 _do_transaction on 0x1ced3680
2014-06-12 14:46:03.750827 7f9d7c165700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.750926 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.750948 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.750998 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.751132 7f9d7c165700 15 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.751138 7f9d7c165700 10 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1 r = 0
2014-06-12 14:46:03.751144 7f9d7c165700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.751161 7f9d7c165700 15 _write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 3358720~516096
2014-06-12 14:46:03.751304 7f9d7c165700 10 lookup_strip_header done  cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.751570 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:03.752455 7f9d3b028700  5 -- op tracker -- , seq: 747546, time: 2014-06-12 14:46:03.749587, event: header_read, op: osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.752472 7f9d3b028700  5 -- op tracker -- , seq: 747546, time: 2014-06-12 14:46:03.749591, event: throttled, op: osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.752482 7f9d3b028700  5 -- op tracker -- , seq: 747546, time: 2014-06-12 14:46:03.750417, event: all_read, op: osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.752491 7f9d3b028700  5 -- op tracker -- , seq: 747546, time: 0.000000, event: dispatched, op: osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.752501 7f9d3b028700 20 osd.2 173 should_share_map client.4308 172.16.96.220:0/1243963 173
2014-06-12 14:46:03.752533 7f9d3b028700 15 osd.2 173 enqueue_op 0x160e4100 prio 63 cost 319488 latency 0.002945 osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.752567 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.752575 7f9d643d9700 10 osd.2 173 dequeue_op 0x160e4100 prio 63 cost 319488 latency 0.002988 osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean]
2014-06-12 14:46:03.752603 7f9d643d9700  5 -- op tracker -- , seq: 747546, time: 2014-06-12 14:46:03.752598, event: reached_pg, op: osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.752614 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:03.752631 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] handle_message: 0x160e4100
2014-06-12 14:46:03.752645 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] do_op osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:03.752677 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] get_object_context: found obc in cache: 0x16962840
2014-06-12 14:46:03.752687 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] get_object_context: 0x16962840 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 rwstate(write n=1 w=0) oi: 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2933 client.4308.0:145090 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2933) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:46:03.752709 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] find_object_context 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 @head oi=27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3(173'2933 client.4308.0:145090 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2933)
2014-06-12 14:46:03.752736 7f9d643d9700  5 -- op tracker -- , seq: 747546, time: 2014-06-12 14:46:03.752731, event: started, op: osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.752743 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] execute_ctx 0x5e4d000
2014-06-12 14:46:03.752763 7f9d643d9700  5 -- op tracker -- , seq: 747546, time: 2014-06-12 14:46:03.752757, event: started, op: osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.752770 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] do_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] ov 173'2933 av 173'2934 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:03.752785 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] do_osd_op 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488]
2014-06-12 14:46:03.752796 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:03.752810 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] do_osd_op  write 3874816~319488
2014-06-12 14:46:03.752828 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] do_osd_op_effects on session 0xa702800
2014-06-12 14:46:03.752840 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:46:03.752853 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] make_writeable 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:03.752864 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] finish_ctx 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 0x5e4d000 op modify  
2014-06-12 14:46:03.752879 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean]  set mtime to 2014-06-12 08:45:06.130094
2014-06-12 14:46:03.752903 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean]  final snapset 0=[]:[]+head in 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.752922 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean]  zeroing write result code 0
2014-06-12 14:46:03.752931 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] new_repop rep_tid 236211 on osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:03.752947 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] issue_repop rep_tid 236211 o 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.752971 7f9d643d9700  5 -- op tracker -- , seq: 747546, time: 2014-06-12 14:46:03.752967, event: waiting for subops from 28, op: osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.752994 7f9d643d9700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:03.753022 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2933 (0'0,173'2933] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] append_log log((0'0,173'2933], crt=0'0) [173'2934 (173'2933) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145091 2014-06-12 08:45:06.130094]
2014-06-12 14:46:03.753046 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] add_log_entry 173'2934 (173'2933) modify   27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 by client.4308.0:145091 2014-06-12 08:45:06.130094
2014-06-12 14:46:03.753076 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] append_log  adding 1 keys
2014-06-12 14:46:03.753112 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2934, trimmed: 
2014-06-12 14:46:03.753138 7f9d643d9700  5 queue_transactions existing osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.753144 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.753145 7f9d643d9700  5 queue_op 0x87f5e60 seq 2706 osr(3.214 0x276d040) 321781 bytes   (queue has 2 ops and 840170 bytes)
2014-06-12 14:46:03.753168 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5e60 (2 active)
2014-06-12 14:46:03.753178 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] eval_repop repgather(0x2335cf00 173'2934 rep_tid=236211 committed?=0 applied?=0 lock=0 op=osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.753200 7f9d643d9700 10 osd.2 173 dequeue_op 0x160e4100 finish
2014-06-12 14:46:03.753205 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.753210 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.753814 7f9d7c165700 10 _generic_write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 3358720~516096 = 0
2014-06-12 14:46:03.753865 7f9d7c165700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.753885 7f9d7c165700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.753891 7f9d7c165700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.753898 7f9d7c165700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.754124 7f9d7c165700 20 set_header setting 16951 cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 parent seq 0
2014-06-12 14:46:03.754204 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.754218 7f9d7c165700 20 set_header setting 2020 cid meta oid 1fe25aca/pglog_3.214/0//-1 parent seq 0
2014-06-12 14:46:03.754230 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:03.757146 7f9d7c165700 10 _do_op 0x139d9a40 seq 2705 r = 0, finisher 0x4eb79e0 0x19b2c980
2014-06-12 14:46:03.757173 7f9d7c165700 10 _finish_op 0x139d9a40 seq 2705 osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.757182 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5e60 (1 active)
2014-06-12 14:46:03.757187 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.758758 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] op_applied: 236210
2014-06-12 14:46:03.758817 7f9d7795c700  5 -- op tracker -- , seq: 747545, time: 2014-06-12 14:46:03.758791, event: op_applied, op: osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.758829 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] op_applied on primary on version 173'2933
2014-06-12 14:46:03.759444 7f9d5b6bc700  5 -- op tracker -- , seq: 747547, time: 2014-06-12 14:46:03.759396, event: header_read, op: osd_sub_op_reply(client.4308.0:145090 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.759454 7f9d5b6bc700  5 -- op tracker -- , seq: 747547, time: 2014-06-12 14:46:03.759404, event: throttled, op: osd_sub_op_reply(client.4308.0:145090 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.759461 7f9d5b6bc700  5 -- op tracker -- , seq: 747547, time: 2014-06-12 14:46:03.759416, event: all_read, op: osd_sub_op_reply(client.4308.0:145090 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.759468 7f9d5b6bc700  5 -- op tracker -- , seq: 747547, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145090 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.759473 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145090 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.759478 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:03.759486 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39577500 prio 196 cost 0 latency 0.000089 osd_sub_op_reply(client.4308.0:145090 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.759515 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.759524 7f9d61bd4700 10 osd.2 173 dequeue_op 0x39577500 prio 196 cost 0 latency 0.000127 osd_sub_op_reply(client.4308.0:145090 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 pg pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean]
2014-06-12 14:46:03.759543 7f9d61bd4700  5 -- op tracker -- , seq: 747547, time: 2014-06-12 14:46:03.759540, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145090 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.759552 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] handle_message: 0x39577500
2014-06-12 14:46:03.759568 7f9d61bd4700  5 -- op tracker -- , seq: 747547, time: 2014-06-12 14:46:03.759565, event: started, op: osd_sub_op_reply(client.4308.0:145090 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.759574 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] sub_op_modify_reply: tid 236210 op  ack_type 4 from 28
2014-06-12 14:46:03.759593 7f9d61bd4700  5 -- op tracker -- , seq: 747545, time: 2014-06-12 14:46:03.759586, event: sub_op_commit_rec, op: osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.759600 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] repop_all_applied: repop tid 236210 all applied 
2014-06-12 14:46:03.759610 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] eval_repop repgather(0x51dc180 173'2933 rep_tid=236210 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.759626 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean]  sending ack on repgather(0x51dc180 173'2933 rep_tid=236210 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x15ce3b80
2014-06-12 14:46:03.759659 7f9d61bd4700 10 osd.2 173 dequeue_op 0x39577500 finish
2014-06-12 14:46:03.759665 7f9d83173700  5 _do_op 0x87f5e60 seq 2706 osr(3.214 0x276d040)/0x276d040 start
2014-06-12 14:46:03.759672 7f9d83173700 10 _do_transaction on 0x224e3c0
2014-06-12 14:46:03.759670 7f9d61bd4700  5 -- op tracker -- , seq: 747547, time: 2014-06-12 14:46:03.759666, event: done, op: osd_sub_op_reply(client.4308.0:145090 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.759679 7f9d83173700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.759683 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.759686 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.759815 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.759833 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.759882 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.759902 7f9d83173700 15 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.759908 7f9d83173700 10 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1 r = 0
2014-06-12 14:46:03.759914 7f9d83173700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:03.760021 7f9d83173700 15 _write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 3874816~319488
2014-06-12 14:46:03.760162 7f9d83173700 10 lookup_strip_header done  cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.760317 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:03.761416 7f9d83173700 10 _generic_write 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 3874816~319488 = 0
2014-06-12 14:46:03.761464 7f9d83173700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.761478 7f9d83173700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.761485 7f9d83173700 15 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3
2014-06-12 14:46:03.761494 7f9d83173700 10 _setattrs 3.214_head/27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:46:03.761640 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] op_commit: 236210
2014-06-12 14:46:03.761681 7f9d7715b700  5 -- op tracker -- , seq: 747545, time: 2014-06-12 14:46:03.761669, event: op_commit, op: osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.761694 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2932 crt=0'0 lcod 173'2931 mlcod 173'2931 active+clean] repop_all_committed: repop tid 236210 all committed 
2014-06-12 14:46:03.761706 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2931 active+clean] eval_repop repgather(0x51dc180 173'2933 rep_tid=236210 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.761726 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2931 active+clean] log_op_stats osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.011612 lat 0.013680
2014-06-12 14:46:03.761748 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2931 active+clean] publish_stats_to_osd 173:3704
2014-06-12 14:46:03.761763 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2931 active+clean]  sending commit on repgather(0x51dc180 173'2933 rep_tid=236210 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0xfa08c80
2014-06-12 14:46:03.761777 7f9d83173700 20 set_header setting 16951 cid 3.214_head oid 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 parent seq 0
2014-06-12 14:46:03.761807 7f9d7715b700  5 -- op tracker -- , seq: 747545, time: 2014-06-12 14:46:03.761800, event: commit_sent, op: osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.761819 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2931 active+clean]  requeue_ops 
2014-06-12 14:46:03.761834 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean]  removing repgather(0x51dc180 173'2933 rep_tid=236210 committed?=1 applied?=1 op=osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.761851 7f9d7715b700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean]    q front is repgather(0x51dc180 173'2933 rep_tid=236210 committed?=1 applied?=1 op=osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.761867 7f9d7715b700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean] remove_repop repgather(0x51dc180 173'2933 rep_tid=236210 committed?=1 applied?=1 op=osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.761887 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.761884 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean]  requeue_ops 
2014-06-12 14:46:03.761903 7f9d83173700 20 set_header setting 2020 cid meta oid 1fe25aca/pglog_3.214/0//-1 parent seq 0
2014-06-12 14:46:03.761906 7f9d7715b700  5 -- op tracker -- , seq: 747545, time: 2014-06-12 14:46:03.761901, event: done, op: osd_op(client.4308.0:145090 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.761914 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:03.763544 7f9d83173700 10 _do_op 0x87f5e60 seq 2706 r = 0, finisher 0x5fbb260 0x3eabb5c0
2014-06-12 14:46:03.763561 7f9d83173700 10 _finish_op 0x87f5e60 seq 2706 osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:03.763574 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5e60 (0 active)
2014-06-12 14:46:03.763577 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.763580 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean] op_applied: 236211
2014-06-12 14:46:03.763613 7f9d7795c700  5 -- op tracker -- , seq: 747546, time: 2014-06-12 14:46:03.763603, event: op_applied, op: osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.763620 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean] op_applied on primary on version 173'2934
2014-06-12 14:46:03.763658 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean] op_commit: 236211
2014-06-12 14:46:03.763684 7f9d7715b700  5 -- op tracker -- , seq: 747546, time: 2014-06-12 14:46:03.763677, event: op_commit, op: osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.771405 7f9d5b6bc700  5 -- op tracker -- , seq: 747548, time: 2014-06-12 14:46:03.771343, event: header_read, op: osd_sub_op_reply(client.4308.0:145091 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.771418 7f9d5b6bc700  5 -- op tracker -- , seq: 747548, time: 2014-06-12 14:46:03.771346, event: throttled, op: osd_sub_op_reply(client.4308.0:145091 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.771426 7f9d5b6bc700  5 -- op tracker -- , seq: 747548, time: 2014-06-12 14:46:03.771364, event: all_read, op: osd_sub_op_reply(client.4308.0:145091 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.771433 7f9d5b6bc700  5 -- op tracker -- , seq: 747548, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145091 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.771438 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145091 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:03.771444 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:03.771451 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39576100 prio 196 cost 0 latency 0.000108 osd_sub_op_reply(client.4308.0:145091 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:03.771483 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.771493 7f9d663dd700 10 osd.2 173 dequeue_op 0x39576100 prio 196 cost 0 latency 0.000149 osd_sub_op_reply(client.4308.0:145091 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0) v2 pg pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean]
2014-06-12 14:46:03.771521 7f9d663dd700  5 -- op tracker -- , seq: 747548, time: 2014-06-12 14:46:03.771517, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145091 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.771529 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean] handle_message: 0x39576100
2014-06-12 14:46:03.771548 7f9d663dd700  5 -- op tracker -- , seq: 747548, time: 2014-06-12 14:46:03.771544, event: started, op: osd_sub_op_reply(client.4308.0:145091 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.771555 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean] sub_op_modify_reply: tid 236211 op  ack_type 4 from 28
2014-06-12 14:46:03.771578 7f9d663dd700  5 -- op tracker -- , seq: 747546, time: 2014-06-12 14:46:03.771568, event: sub_op_commit_rec, op: osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.771588 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean] repop_all_applied: repop tid 236211 all applied 
2014-06-12 14:46:03.771602 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean] eval_repop repgather(0x2335cf00 173'2934 rep_tid=236211 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.771621 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean]  sending ack on repgather(0x2335cf00 173'2934 rep_tid=236211 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x28aadb80
2014-06-12 14:46:03.771665 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2933 crt=0'0 lcod 173'2932 mlcod 173'2932 active+clean] repop_all_committed: repop tid 236211 all committed 
2014-06-12 14:46:03.771674 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2932 active+clean] eval_repop repgather(0x2335cf00 173'2934 rep_tid=236211 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:03.771693 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2932 active+clean] log_op_stats osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4 inb 319767 outb 0 rlat 0.022076 lat 0.022105
2014-06-12 14:46:03.771714 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2932 active+clean] publish_stats_to_osd 173:3705
2014-06-12 14:46:03.771731 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2932 active+clean]  sending commit on repgather(0x2335cf00 173'2934 rep_tid=236211 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4) 0x282cc280
2014-06-12 14:46:03.771767 7f9d663dd700  5 -- op tracker -- , seq: 747546, time: 2014-06-12 14:46:03.771760, event: commit_sent, op: osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.771775 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2932 active+clean]  requeue_ops 
2014-06-12 14:46:03.771786 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean]  removing repgather(0x2335cf00 173'2934 rep_tid=236211 committed?=1 applied?=1 op=osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.771803 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean]    q front is repgather(0x2335cf00 173'2934 rep_tid=236211 committed?=1 applied?=1 op=osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.771819 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] remove_repop repgather(0x2335cf00 173'2934 rep_tid=236211 committed?=1 applied?=1 op=osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:03.771836 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean]  requeue_ops 
2014-06-12 14:46:03.771863 7f9d663dd700  5 -- op tracker -- , seq: 747546, time: 2014-06-12 14:46:03.771857, event: done, op: osd_op(client.4308.0:145091 rbd_data.10ab6b8b4567.0000000000000f84 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.27491214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:03.771882 7f9d663dd700 10 osd.2 173 dequeue_op 0x39576100 finish
2014-06-12 14:46:03.771891 7f9d663dd700  5 -- op tracker -- , seq: 747548, time: 2014-06-12 14:46:03.771888, event: done, op: osd_sub_op_reply(client.4308.0:145091 3.214 27491214/rbd_data.10ab6b8b4567.0000000000000f84/head//3 [] ondisk, result = 0)
2014-06-12 14:46:03.771903 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.771906 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.880045 7f9d56432700  5 -- op tracker -- , seq: 747549, time: 2014-06-12 14:46:03.879260, event: header_read, op: osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.880065 7f9d56432700  5 -- op tracker -- , seq: 747549, time: 2014-06-12 14:46:03.879270, event: throttled, op: osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.880074 7f9d56432700  5 -- op tracker -- , seq: 747549, time: 2014-06-12 14:46:03.880003, event: all_read, op: osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.880082 7f9d56432700  5 -- op tracker -- , seq: 747549, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.880088 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:03.880096 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:03.880105 7f9d56432700 15 osd.2 173 enqueue_op 0x329ea800 prio 127 cost 516899 latency 0.000845 osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.880137 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.880147 7f9d673df700 10 osd.2 173 dequeue_op 0x329ea800 prio 127 cost 516899 latency 0.000886 osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2470 (0'0,173'2470] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active]
2014-06-12 14:46:03.880171 7f9d673df700  5 -- op tracker -- , seq: 747549, time: 2014-06-12 14:46:03.880167, event: reached_pg, op: osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.880181 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2470 (0'0,173'2470] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active] handle_message: 0x329ea800
2014-06-12 14:46:03.880192 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2470 (0'0,173'2470] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active] sub_op_modify trans 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 v 173'2471 (transaction) 174
2014-06-12 14:46:03.880207 7f9d673df700  5 -- op tracker -- , seq: 747549, time: 2014-06-12 14:46:03.880203, event: started, op: osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.880221 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2470 (0'0,173'2470] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active] append_log log((0'0,173'2470], crt=0'0) [173'2471 (173'2468) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144988 2014-06-12 08:44:57.420120]
2014-06-12 14:46:03.880243 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2471 (0'0,173'2471] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active] add_log_entry 173'2471 (173'2468) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144988 2014-06-12 08:44:57.420120
2014-06-12 14:46:03.880264 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2471 (0'0,173'2471] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active] append_log  adding 1 keys
2014-06-12 14:46:03.880296 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2471, trimmed: 
2014-06-12 14:46:03.880319 7f9d673df700  5 -- op tracker -- , seq: 747549, time: 2014-06-12 14:46:03.880314, event: started, op: osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.880329 7f9d673df700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:03.880333 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.880334 7f9d673df700  5 queue_op 0x1b2fe360 seq 2284 osr(3.2b 0x276cd20) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:46:03.880342 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2471 (0'0,173'2471] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active] do_sub_op osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.880355 7f9d673df700 10 osd.2 173 dequeue_op 0x329ea800 finish
2014-06-12 14:46:03.880358 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.880361 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.880369 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:46:03.880381 7f9d82972700  5 _do_op 0x1b2fe360 seq 2284 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:46:03.880385 7f9d82972700 10 _do_transaction on 0x247fd270
2014-06-12 14:46:03.880391 7f9d82972700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.880539 7f9d82972700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.880555 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.880613 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.880633 7f9d82972700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.880638 7f9d82972700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:46:03.880643 7f9d82972700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.880654 7f9d82972700 15 _write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 794624~516096
2014-06-12 14:46:03.880807 7f9d82972700 10 lookup_strip_header done  cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:03.880823 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:03.883229 7f9d82972700 10 _generic_write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 794624~516096 = 0
2014-06-12 14:46:03.883296 7f9d82972700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:03.883320 7f9d82972700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:03.883330 7f9d82972700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:03.883340 7f9d82972700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:03.883561 7f9d82972700 20 set_header setting 17866 cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 parent seq 0
2014-06-12 14:46:03.883691 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.883710 7f9d82972700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:46:03.883723 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:03.886795 7f9d82972700 10 _do_op 0x1b2fe360 seq 2284 r = 0, finisher 0x698c660 0
2014-06-12 14:46:03.886821 7f9d82972700 10 _finish_op 0x1b2fe360 seq 2284 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:03.886829 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:46:03.886839 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.886869 7f9d7795c700  5 -- op tracker -- , seq: 747549, time: 2014-06-12 14:46:03.886856, event: sub_op_applied, op: osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.886904 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2471 (0'0,173'2471] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active] sub_op_modify_applied on 0x247fd180 op osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.886930 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:03.886947 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2471 (0'0,173'2471] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active] op_applied on primary on version 173'2471
2014-06-12 14:46:03.886957 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2471 (0'0,173'2471] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active] op_applied on replica on version 173'2471
2014-06-12 14:46:03.890857 7f9d7715b700  5 -- op tracker -- , seq: 747549, time: 2014-06-12 14:46:03.890842, event: commit_sent, op: osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.890872 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2471 (0'0,173'2471] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2469 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:03.890901 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:03.890939 7f9d7715b700  5 -- op tracker -- , seq: 747549, time: 2014-06-12 14:46:03.890934, event: done, op: osd_sub_op(client.4314.0:144988 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.896179 7f9d56432700  5 -- op tracker -- , seq: 747550, time: 2014-06-12 14:46:03.894775, event: header_read, op: osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.896196 7f9d56432700  5 -- op tracker -- , seq: 747550, time: 2014-06-12 14:46:03.894780, event: throttled, op: osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.896205 7f9d56432700  5 -- op tracker -- , seq: 747550, time: 2014-06-12 14:46:03.896122, event: all_read, op: osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.896213 7f9d56432700  5 -- op tracker -- , seq: 747550, time: 0.000000, event: dispatched, op: osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.896219 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:03.896226 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:03.896236 7f9d56432700 15 osd.2 173 enqueue_op 0x2a903c00 prio 127 cost 664359 latency 0.001460 osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.896272 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.896282 7f9d67be0700 10 osd.2 173 dequeue_op 0x2a903c00 prio 127 cost 664359 latency 0.001506 osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.15c( v 173'2642 (0'0,173'2642] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active]
2014-06-12 14:46:03.896312 7f9d67be0700  5 -- op tracker -- , seq: 747550, time: 2014-06-12 14:46:03.896308, event: reached_pg, op: osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.896323 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2642 (0'0,173'2642] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] handle_message: 0x2a903c00
2014-06-12 14:46:03.896337 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2642 (0'0,173'2642] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] sub_op_modify trans 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 v 173'2643 (transaction) 174
2014-06-12 14:46:03.896356 7f9d67be0700  5 -- op tracker -- , seq: 747550, time: 2014-06-12 14:46:03.896351, event: started, op: osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.896375 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2642 (0'0,173'2642] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] append_log log((0'0,173'2642], crt=0'0) [173'2643 (173'672) modify   805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 by client.4279.0:143440 2014-06-12 08:44:31.043184]
2014-06-12 14:46:03.896404 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2643 (0'0,173'2643] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] add_log_entry 173'2643 (173'672) modify   805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 by client.4279.0:143440 2014-06-12 08:44:31.043184
2014-06-12 14:46:03.896430 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2643 (0'0,173'2643] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] append_log  adding 1 keys
2014-06-12 14:46:03.896470 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2643, trimmed: 
2014-06-12 14:46:03.896497 7f9d67be0700  5 -- op tracker -- , seq: 747550, time: 2014-06-12 14:46:03.896492, event: started, op: osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.896509 7f9d67be0700  5 queue_transactions existing osr(3.15c 0x276ced0)/0x276ced0
2014-06-12 14:46:03.896514 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.896516 7f9d67be0700  5 queue_op 0x8373920 seq 2439 osr(3.15c 0x276ced0) 665845 bytes   (queue has 1 ops and 665845 bytes)
2014-06-12 14:46:03.896524 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.15c( v 173'2643 (0'0,173'2643] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] do_sub_op osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.896541 7f9d67be0700 10 osd.2 173 dequeue_op 0x2a903c00 finish
2014-06-12 14:46:03.896545 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.896547 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.896556 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54bde60 (1 active)
2014-06-12 14:46:03.896562 7f9d7c966700  5 _do_op 0x8373920 seq 2439 osr(3.15c 0x276ced0)/0x276ced0 start
2014-06-12 14:46:03.896566 7f9d7c966700 10 _do_transaction on 0x9b93eb0
2014-06-12 14:46:03.896573 7f9d7c966700 15 _omap_setkeys meta/1fe2b14b/pglog_3.15c/0//-1
2014-06-12 14:46:03.896732 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe2b14b/pglog_3.15c/0//-1
2014-06-12 14:46:03.896763 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.896820 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.896841 7f9d7c966700 15 _omap_rmkeys meta/1fe2b14b/pglog_3.15c/0//-1
2014-06-12 14:46:03.896849 7f9d7c966700 10 _omap_rmkeys meta/1fe2b14b/pglog_3.15c/0//-1 r = 0
2014-06-12 14:46:03.896856 7f9d7c966700 15 _omap_setkeys meta/1fe2b14b/pglog_3.15c/0//-1
2014-06-12 14:46:03.896976 7f9d7c966700 15 _write 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 0~663552
2014-06-12 14:46:03.897169 7f9d7c966700 10 lookup_strip_header done  cid 3.15c_head oid 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3
2014-06-12 14:46:03.897304 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:03.899994 7f9d7c966700 10 _generic_write 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 0~663552 = 0
2014-06-12 14:46:03.900074 7f9d7c966700 15 _setattrs 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3
2014-06-12 14:46:03.900092 7f9d7c966700 10 _setattrs 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:03.900122 7f9d7c966700 15 _setattrs 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3
2014-06-12 14:46:03.900134 7f9d7c966700 10 _setattrs 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:03.900392 7f9d7c966700 20 set_header setting 16993 cid 3.15c_head oid 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 parent seq 0
2014-06-12 14:46:03.900480 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.900499 7f9d7c966700 20 set_header setting 2322 cid meta oid 1fe2b14b/pglog_3.15c/0//-1 parent seq 0
2014-06-12 14:46:03.900513 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:03.904151 7f9d7c966700 10 _do_op 0x8373920 seq 2439 r = 0, finisher 0x654bf80 0
2014-06-12 14:46:03.904205 7f9d7715b700  5 -- op tracker -- , seq: 747550, time: 2014-06-12 14:46:03.904194, event: commit_sent, op: osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.904221 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2643 (0'0,173'2643] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] sub_op_modify_commit on op osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:03.904256 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:03.904280 7f9d7c966700 10 _finish_op 0x8373920 seq 2439 osr(3.15c 0x276ced0)/0x276ced0
2014-06-12 14:46:03.904288 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54bde60 (0 active)
2014-06-12 14:46:03.904297 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:03.904302 7f9d7795c700  5 -- op tracker -- , seq: 747550, time: 2014-06-12 14:46:03.904295, event: sub_op_applied, op: osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.904310 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2643 (0'0,173'2643] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] sub_op_modify_applied on 0x9b93dc0 op osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.904327 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2643 (0'0,173'2643] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] op_applied on primary on version 173'2643
2014-06-12 14:46:03.904337 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2643 (0'0,173'2643] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] op_applied on replica on version 173'2643
2014-06-12 14:46:03.904361 7f9d7795c700  5 -- op tracker -- , seq: 747550, time: 2014-06-12 14:46:03.904357, event: done, op: osd_sub_op(client.4279.0:143440 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.927807 7f9d56432700  5 -- op tracker -- , seq: 747551, time: 2014-06-12 14:46:03.927021, event: header_read, op: osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.927820 7f9d56432700  5 -- op tracker -- , seq: 747551, time: 2014-06-12 14:46:03.927023, event: throttled, op: osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.927828 7f9d56432700  5 -- op tracker -- , seq: 747551, time: 2014-06-12 14:46:03.927764, event: all_read, op: osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.927836 7f9d56432700  5 -- op tracker -- , seq: 747551, time: 0.000000, event: dispatched, op: osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.927843 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:03.927850 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:03.927860 7f9d56432700 15 osd.2 173 enqueue_op 0x124e3700 prio 127 cost 516899 latency 0.000838 osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.927890 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:03.927899 7f9d693e3700 10 osd.2 173 dequeue_op 0x124e3700 prio 127 cost 516899 latency 0.000877 osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2471 (0'0,173'2471] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2470 active]
2014-06-12 14:46:03.927930 7f9d693e3700  5 -- op tracker -- , seq: 747551, time: 2014-06-12 14:46:03.927925, event: reached_pg, op: osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.927946 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2471 (0'0,173'2471] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2470 active] handle_message: 0x124e3700
2014-06-12 14:46:03.927964 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2471 (0'0,173'2471] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2470 active] sub_op_modify trans 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 v 173'2472 (transaction) 174
2014-06-12 14:46:03.927983 7f9d693e3700  5 -- op tracker -- , seq: 747551, time: 2014-06-12 14:46:03.927979, event: started, op: osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.928001 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2471 (0'0,173'2471] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2470 active] append_log log((0'0,173'2471], crt=0'0) [173'2472 (173'2469) modify   4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 by client.4280.0:139527 2014-06-12 08:43:26.697805]
2014-06-12 14:46:03.928032 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2472 (0'0,173'2472] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2470 active] add_log_entry 173'2472 (173'2469) modify   4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 by client.4280.0:139527 2014-06-12 08:43:26.697805
2014-06-12 14:46:03.928058 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2472 (0'0,173'2472] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2470 active] append_log  adding 1 keys
2014-06-12 14:46:03.928098 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2472, trimmed: 
2014-06-12 14:46:03.928126 7f9d693e3700  5 -- op tracker -- , seq: 747551, time: 2014-06-12 14:46:03.928121, event: started, op: osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.928139 7f9d693e3700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:03.928143 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:03.928145 7f9d693e3700  5 queue_op 0x287fa40 seq 2285 osr(3.2b 0x276cd20) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:46:03.928157 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:46:03.928165 7f9d81970700  5 _do_op 0x287fa40 seq 2285 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:46:03.928153 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2472 (0'0,173'2472] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2470 active] do_sub_op osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.928169 7f9d693e3700 10 osd.2 173 dequeue_op 0x124e3700 finish
2014-06-12 14:46:03.928171 7f9d81970700 10 _do_transaction on 0x2e9b2470
2014-06-12 14:46:03.928173 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:03.928175 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:03.928177 7f9d81970700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.928322 7f9d81970700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.928467 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:03.928535 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:03.928561 7f9d81970700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.928569 7f9d81970700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:46:03.928576 7f9d81970700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:03.928593 7f9d81970700 15 _write 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 2449408~516096
2014-06-12 14:46:03.928913 7f9d81970700 10 lookup_strip_header done  cid 3.2b_head oid 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:46:03.929046 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:03.930908 7f9d81970700 10 _generic_write 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 2449408~516096 = 0
2014-06-12 14:46:03.930955 7f9d81970700 15 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:46:03.930975 7f9d81970700 10 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 = 0
2014-06-12 14:46:03.930981 7f9d81970700 15 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:46:03.930989 7f9d81970700 10 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 = 0
2014-06-12 14:46:03.931196 7f9d81970700 20 set_header setting 15771 cid 3.2b_head oid 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 parent seq 0
2014-06-12 14:46:03.931274 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:03.931289 7f9d81970700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:46:03.931299 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:03.934185 7f9d81970700 10 _do_op 0x287fa40 seq 2285 r = 0, finisher 0x87fa440 0
2014-06-12 14:46:03.934229 7f9d7715b700  5 -- op tracker -- , seq: 747551, time: 2014-06-12 14:46:03.934220, event: commit_sent, op: osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.934242 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2472 (0'0,173'2472] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2470 active] sub_op_modify_commit on op osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:03.934272 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:03.934300 7f9d81970700 10 _finish_op 0x287fa40 seq 2285 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:03.934323 7f9d7795c700  5 -- op tracker -- , seq: 747551, time: 2014-06-12 14:46:03.934318, event: sub_op_applied, op: osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.934330 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2472 (0'0,173'2472] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2471 active] sub_op_modify_applied on 0x2e9b2380 op osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:03.934345 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2472 (0'0,173'2472] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2471 active] op_applied on primary on version 173'2472
2014-06-12 14:46:03.934356 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2472 (0'0,173'2472] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2471 active] op_applied on replica on version 173'2472
2014-06-12 14:46:03.934387 7f9d81970700  5 -- op tracker -- , seq: 747551, time: 2014-06-12 14:46:03.934377, event: done, op: osd_sub_op(client.4280.0:139527 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:03.934413 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:46:03.934416 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.013403 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19404 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,4,2,4,23,1])
2014-06-12 14:46:04.013428 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19404 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,4,2,4,23,1])
2014-06-12 14:46:04.013924 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:04.013956 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:04.013983 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:04.014049 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:04.014071 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:04.014096 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:04.014118 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:04.014150 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:04.014168 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:04.014187 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:04.014208 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:04.014224 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:04.014241 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:04.014293 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:04.014312 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:04.014329 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:04.014345 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:04.014362 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:04.014380 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:04.014396 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:04.014521 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:04.014539 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:04.036294 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:04.036538 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:04.055919 7f9d56432700  5 -- op tracker -- , seq: 747552, time: 2014-06-12 14:46:04.055327, event: header_read, op: osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.055938 7f9d56432700  5 -- op tracker -- , seq: 747552, time: 2014-06-12 14:46:04.055329, event: throttled, op: osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.055948 7f9d56432700  5 -- op tracker -- , seq: 747552, time: 2014-06-12 14:46:04.055870, event: all_read, op: osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.055956 7f9d56432700  5 -- op tracker -- , seq: 747552, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.055961 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.055968 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:04.055978 7f9d56432700 15 osd.2 173 enqueue_op 0x34017b00 prio 127 cost 406311 latency 0.000651 osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.056019 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.056028 7f9d62bd6700 10 osd.2 173 dequeue_op 0x34017b00 prio 127 cost 406311 latency 0.000701 osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.51e( v 173'2677 (0'0,173'2677] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2676 active]
2014-06-12 14:46:04.056046 7f9d62bd6700  5 -- op tracker -- , seq: 747552, time: 2014-06-12 14:46:04.056043, event: reached_pg, op: osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.056053 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2677 (0'0,173'2677] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2676 active] handle_message: 0x34017b00
2014-06-12 14:46:04.056061 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2677 (0'0,173'2677] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2676 active] sub_op_modify trans 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 v 173'2678 (transaction) 174
2014-06-12 14:46:04.056071 7f9d62bd6700  5 -- op tracker -- , seq: 747552, time: 2014-06-12 14:46:04.056069, event: started, op: osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.056083 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2677 (0'0,173'2677] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2676 active] append_log log((0'0,173'2677], crt=0'0) [173'2678 (173'2677) modify   55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 by client.4294.0:143954 2014-06-12 08:44:56.720783]
2014-06-12 14:46:04.056106 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2678 (0'0,173'2678] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2676 active] add_log_entry 173'2678 (173'2677) modify   55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 by client.4294.0:143954 2014-06-12 08:44:56.720783
2014-06-12 14:46:04.056121 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2678 (0'0,173'2678] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2676 active] append_log  adding 1 keys
2014-06-12 14:46:04.056143 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2678, trimmed: 
2014-06-12 14:46:04.056160 7f9d62bd6700  5 -- op tracker -- , seq: 747552, time: 2014-06-12 14:46:04.056157, event: started, op: osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.056167 7f9d62bd6700  5 queue_transactions existing osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:04.056170 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.056171 7f9d62bd6700  5 queue_op 0x53e3980 seq 2514 osr(3.51e 0x276db50) 407797 bytes   (queue has 1 ops and 407797 bytes)
2014-06-12 14:46:04.056181 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fb00 (1 active)
2014-06-12 14:46:04.056187 7f9d7a161700  5 _do_op 0x53e3980 seq 2514 osr(3.51e 0x276db50)/0x276db50 start
2014-06-12 14:46:04.056177 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.51e( v 173'2678 (0'0,173'2678] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2676 active] do_sub_op osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.056190 7f9d62bd6700 10 osd.2 173 dequeue_op 0x34017b00 finish
2014-06-12 14:46:04.056191 7f9d7a161700 10 _do_transaction on 0x6177b30
2014-06-12 14:46:04.056193 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.056196 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.056197 7f9d7a161700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:04.056337 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:04.056451 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.056502 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.056528 7f9d7a161700 15 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:04.056536 7f9d7a161700 10 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1 r = 0
2014-06-12 14:46:04.056542 7f9d7a161700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:04.056560 7f9d7a161700 15 _write 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 3788800~405504
2014-06-12 14:46:04.056885 7f9d7a161700 10 lookup_strip_header done  cid 3.51e_head oid 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3
2014-06-12 14:46:04.057009 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:04.057922 7f9d7a161700 10 _generic_write 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 3788800~405504 = 0
2014-06-12 14:46:04.057955 7f9d7a161700 15 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3
2014-06-12 14:46:04.057970 7f9d7a161700 10 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 = 0
2014-06-12 14:46:04.057976 7f9d7a161700 15 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3
2014-06-12 14:46:04.057985 7f9d7a161700 10 _setattrs 3.51e_head/55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 = 0
2014-06-12 14:46:04.058114 7f9d7a161700 20 set_header setting 17621 cid 3.51e_head oid 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 parent seq 0
2014-06-12 14:46:04.058162 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.058177 7f9d7a161700 20 set_header setting 2288 cid meta oid 1fe1612b/pglog_3.51e/0//-1 parent seq 0
2014-06-12 14:46:04.058187 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:04.059645 7f9d7a161700 10 _do_op 0x53e3980 seq 2514 r = 0, finisher 0x7ff8c40 0
2014-06-12 14:46:04.059662 7f9d7a161700 10 _finish_op 0x53e3980 seq 2514 osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:04.059671 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fb00 (0 active)
2014-06-12 14:46:04.059680 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.059682 7f9d7715b700  5 -- op tracker -- , seq: 747552, time: 2014-06-12 14:46:04.059671, event: commit_sent, op: osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.059693 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2678 (0'0,173'2678] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2676 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:04.059723 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:04.059759 7f9d7795c700  5 -- op tracker -- , seq: 747552, time: 2014-06-12 14:46:04.059755, event: sub_op_applied, op: osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.059766 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2678 (0'0,173'2678] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2677 active] sub_op_modify_applied on 0x6177a40 op osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.059789 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2678 (0'0,173'2678] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2677 active] op_applied on primary on version 173'2678
2014-06-12 14:46:04.059800 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2678 (0'0,173'2678] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2677 active] op_applied on replica on version 173'2678
2014-06-12 14:46:04.059821 7f9d7795c700  5 -- op tracker -- , seq: 747552, time: 2014-06-12 14:46:04.059816, event: done, op: osd_sub_op(client.4294.0:143954 3.51e 55843d1e/rbd_data.109d6b8b4567.000000000000110a/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.067627 7f9d56432700  5 -- op tracker -- , seq: 747553, time: 2014-06-12 14:46:04.065114, event: header_read, op: osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.067641 7f9d56432700  5 -- op tracker -- , seq: 747553, time: 2014-06-12 14:46:04.065117, event: throttled, op: osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.067650 7f9d56432700  5 -- op tracker -- , seq: 747553, time: 2014-06-12 14:46:04.067593, event: all_read, op: osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.067660 7f9d56432700  5 -- op tracker -- , seq: 747553, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.067687 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.067694 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:04.067703 7f9d56432700 15 osd.2 173 enqueue_op 0x6f4ba00 prio 127 cost 516903 latency 0.002589 osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.067728 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.067737 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x6f4ba00 prio 127 cost 516903 latency 0.002623 osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2986 (0'0,173'2986] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2985 active]
2014-06-12 14:46:04.067756 7f9d6b3e7700  5 -- op tracker -- , seq: 747553, time: 2014-06-12 14:46:04.067753, event: reached_pg, op: osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.067764 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2986 (0'0,173'2986] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2985 active] handle_message: 0x6f4ba00
2014-06-12 14:46:04.067773 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2986 (0'0,173'2986] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2985 active] sub_op_modify trans 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 v 173'2987 (transaction) 174
2014-06-12 14:46:04.067808 7f9d6b3e7700  5 -- op tracker -- , seq: 747553, time: 2014-06-12 14:46:04.067803, event: started, op: osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.067826 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2986 (0'0,173'2986] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2985 active] append_log log((0'0,173'2986], crt=0'0) [173'2987 (173'2985) modify   3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 by client.4298.0:145088 2014-06-12 08:43:26.672801]
2014-06-12 14:46:04.067847 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2987 (0'0,173'2987] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2985 active] add_log_entry 173'2987 (173'2985) modify   3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 by client.4298.0:145088 2014-06-12 08:43:26.672801
2014-06-12 14:46:04.067862 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2987 (0'0,173'2987] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2985 active] append_log  adding 1 keys
2014-06-12 14:46:04.067888 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2987, trimmed: 
2014-06-12 14:46:04.067947 7f9d6b3e7700  5 -- op tracker -- , seq: 747553, time: 2014-06-12 14:46:04.067942, event: started, op: osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.067961 7f9d6b3e7700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:04.067964 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.067966 7f9d6b3e7700  5 queue_op 0x776d620 seq 2789 osr(3.458 0x276d960) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:04.067970 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2987 (0'0,173'2987] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2985 active] do_sub_op osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.067980 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x6f4ba00 finish
2014-06-12 14:46:04.067983 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.067984 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.068151 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:04.068163 7f9d7a962700  5 _do_op 0x776d620 seq 2789 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:04.068169 7f9d7a962700 10 _do_transaction on 0x1909f970
2014-06-12 14:46:04.068384 7f9d7a962700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.068532 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.068550 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.068612 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.068658 7f9d7a962700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.068665 7f9d7a962700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:04.068670 7f9d7a962700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.068688 7f9d7a962700 15 _write 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 253952~516096
2014-06-12 14:46:04.069075 7f9d7a962700 10 lookup_strip_header done  cid 3.458_head oid 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:46:04.069209 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:04.071978 7f9d7a962700 10 _generic_write 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 253952~516096 = 0
2014-06-12 14:46:04.072088 7f9d7a962700 15 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:46:04.072124 7f9d7a962700 10 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 = 0
2014-06-12 14:46:04.072131 7f9d7a962700 15 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:46:04.072141 7f9d7a962700 10 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 = 0
2014-06-12 14:46:04.072409 7f9d7a962700 20 set_header setting 16953 cid 3.458_head oid 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 parent seq 0
2014-06-12 14:46:04.072481 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.072493 7f9d7a962700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:04.072499 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:04.078106 7f9d7a962700 10 _do_op 0x776d620 seq 2789 r = 0, finisher 0x88a4280 0
2014-06-12 14:46:04.078127 7f9d7a962700 10 _finish_op 0x776d620 seq 2789 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:04.078146 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:04.078147 7f9d7715b700  5 -- op tracker -- , seq: 747553, time: 2014-06-12 14:46:04.078135, event: commit_sent, op: osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.078156 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.078158 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2987 (0'0,173'2987] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2985 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:04.078183 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:04.078217 7f9d7795c700  5 -- op tracker -- , seq: 747553, time: 2014-06-12 14:46:04.078210, event: sub_op_applied, op: osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.078223 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2987 (0'0,173'2987] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2986 active] sub_op_modify_applied on 0x1909f880 op osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.078238 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2987 (0'0,173'2987] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2986 active] op_applied on primary on version 173'2987
2014-06-12 14:46:04.078248 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2987 (0'0,173'2987] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2986 active] op_applied on replica on version 173'2987
2014-06-12 14:46:04.078271 7f9d7795c700  5 -- op tracker -- , seq: 747553, time: 2014-06-12 14:46:04.078267, event: done, op: osd_sub_op(client.4298.0:145088 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.097003 7f9d44101700  5 -- op tracker -- , seq: 747554, time: 2014-06-12 14:46:04.095619, event: header_read, op: osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.097021 7f9d44101700  5 -- op tracker -- , seq: 747554, time: 2014-06-12 14:46:04.095621, event: throttled, op: osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.097030 7f9d44101700  5 -- op tracker -- , seq: 747554, time: 2014-06-12 14:46:04.096950, event: all_read, op: osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.097038 7f9d44101700  5 -- op tracker -- , seq: 747554, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.097045 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.097052 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:04.097063 7f9d44101700 15 osd.2 173 enqueue_op 0x3b42b000 prio 127 cost 533283 latency 0.001444 osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.097103 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.097117 7f9d64bda700 10 osd.2 173 dequeue_op 0x3b42b000 prio 127 cost 533283 latency 0.001497 osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.46( v 173'3093 (79'92,173'3093] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3092 active]
2014-06-12 14:46:04.097145 7f9d64bda700  5 -- op tracker -- , seq: 747554, time: 2014-06-12 14:46:04.097141, event: reached_pg, op: osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.097157 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3093 (79'92,173'3093] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3092 active] handle_message: 0x3b42b000
2014-06-12 14:46:04.097171 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3093 (79'92,173'3093] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3092 active] sub_op_modify trans 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 v 173'3094 (transaction) 174
2014-06-12 14:46:04.097189 7f9d64bda700  5 -- op tracker -- , seq: 747554, time: 2014-06-12 14:46:04.097185, event: started, op: osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.097212 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3093 (79'92,173'3093] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3092 active] append_log log((79'92,173'3093], crt=0'0) [173'3094 (173'776) modify   8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 by client.4302.0:146041 2014-06-12 08:45:06.471204]
2014-06-12 14:46:04.097238 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3094 (79'92,173'3094] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3092 active] add_log_entry 173'3094 (173'776) modify   8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 by client.4302.0:146041 2014-06-12 08:45:06.471204
2014-06-12 14:46:04.097261 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3094 (79'92,173'3094] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3092 active] append_log  adding 1 keys
2014-06-12 14:46:04.097275 7f9d64bda700 10 trim log((79'92,173'3094], crt=0'0) to 79'93
2014-06-12 14:46:04.097307 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3094, trimmed: 79'93
2014-06-12 14:46:04.097337 7f9d64bda700  5 -- op tracker -- , seq: 747554, time: 2014-06-12 14:46:04.097332, event: started, op: osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.097349 7f9d64bda700  5 queue_transactions existing osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:04.097354 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.097356 7f9d64bda700  5 queue_op 0x213fb920 seq 2867 osr(3.46 0x276cc90) 534799 bytes   (queue has 1 ops and 534799 bytes)
2014-06-12 14:46:04.097370 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e97200 (1 active)
2014-06-12 14:46:04.097366 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.46( v 173'3094 (79'93,173'3094] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3092 active] do_sub_op osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.097378 7f9d7f16b700  5 _do_op 0x213fb920 seq 2867 osr(3.46 0x276cc90)/0x276cc90 start
2014-06-12 14:46:04.097381 7f9d64bda700 10 osd.2 173 dequeue_op 0x3b42b000 finish
2014-06-12 14:46:04.097384 7f9d7f16b700 10 _do_transaction on 0x2daa70b0
2014-06-12 14:46:04.097385 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.097387 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.097391 7f9d7f16b700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.097543 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.097571 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.097636 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.097734 7f9d7f16b700 15 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.097743 7f9d7f16b700 10 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1 r = 0
2014-06-12 14:46:04.097750 7f9d7f16b700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.097768 7f9d7f16b700 15 _write 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 0~532480
2014-06-12 14:46:04.097946 7f9d7f16b700 10 lookup_strip_header done  cid 3.46_head oid 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.098103 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:04.100174 7f9d7f16b700 10 _generic_write 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 0~532480 = 0
2014-06-12 14:46:04.100234 7f9d7f16b700 15 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.100256 7f9d7f16b700 10 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 = 0
2014-06-12 14:46:04.100266 7f9d7f16b700 15 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.100276 7f9d7f16b700 10 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 = 0
2014-06-12 14:46:04.100495 7f9d7f16b700 20 set_header setting 17146 cid 3.46_head oid 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 parent seq 0
2014-06-12 14:46:04.100579 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.100598 7f9d7f16b700 20 set_header setting 2172 cid meta oid 51681c0b/pglog_3.46/0//-1 parent seq 0
2014-06-12 14:46:04.100611 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:04.103355 7f9d7f16b700 10 _do_op 0x213fb920 seq 2867 r = 0, finisher 0x73f87c0 0
2014-06-12 14:46:04.103402 7f9d7715b700  5 -- op tracker -- , seq: 747554, time: 2014-06-12 14:46:04.103393, event: commit_sent, op: osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.103420 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3094 (79'93,173'3094] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3092 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:46:04.103445 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:04.103472 7f9d7f16b700 10 _finish_op 0x213fb920 seq 2867 osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:04.103481 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e97200 (0 active)
2014-06-12 14:46:04.103490 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.103496 7f9d7795c700  5 -- op tracker -- , seq: 747554, time: 2014-06-12 14:46:04.103489, event: sub_op_applied, op: osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.103504 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3094 (79'93,173'3094] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3093 active] sub_op_modify_applied on 0x2daa6fc0 op osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.103521 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3094 (79'93,173'3094] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3093 active] op_applied on primary on version 173'3094
2014-06-12 14:46:04.103529 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3094 (79'93,173'3094] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3093 active] op_applied on replica on version 173'3094
2014-06-12 14:46:04.103554 7f9d7795c700  5 -- op tracker -- , seq: 747554, time: 2014-06-12 14:46:04.103550, event: done, op: osd_sub_op(client.4302.0:146041 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3094 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.113001 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:04.113029 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:04.136026 7f9d44101700  5 -- op tracker -- , seq: 747555, time: 2014-06-12 14:46:04.134365, event: header_read, op: osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.136047 7f9d44101700  5 -- op tracker -- , seq: 747555, time: 2014-06-12 14:46:04.134367, event: throttled, op: osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.136056 7f9d44101700  5 -- op tracker -- , seq: 747555, time: 2014-06-12 14:46:04.135975, event: all_read, op: osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.136064 7f9d44101700  5 -- op tracker -- , seq: 747555, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.136070 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.136077 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:04.136086 7f9d44101700 15 osd.2 173 enqueue_op 0x20355200 prio 127 cost 1032995 latency 0.001721 osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.136124 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.136137 7f9d683e1700 10 osd.2 173 dequeue_op 0x20355200 prio 127 cost 1032995 latency 0.001771 osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.46( v 173'3094 (79'93,173'3094] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3093 active]
2014-06-12 14:46:04.136162 7f9d683e1700  5 -- op tracker -- , seq: 747555, time: 2014-06-12 14:46:04.136157, event: reached_pg, op: osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.136172 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3094 (79'93,173'3094] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3093 active] handle_message: 0x20355200
2014-06-12 14:46:04.136192 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3094 (79'93,173'3094] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3093 active] sub_op_modify trans 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 v 173'3095 (transaction) 174
2014-06-12 14:46:04.136216 7f9d683e1700  5 -- op tracker -- , seq: 747555, time: 2014-06-12 14:46:04.136212, event: started, op: osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.136236 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3094 (79'93,173'3094] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3093 active] append_log log((79'93,173'3094], crt=0'0) [173'3095 (173'3094) modify   8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 by client.4302.0:146042 2014-06-12 08:45:06.510195]
2014-06-12 14:46:04.136262 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3095 (79'93,173'3095] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3093 active] add_log_entry 173'3095 (173'3094) modify   8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 by client.4302.0:146042 2014-06-12 08:45:06.510195
2014-06-12 14:46:04.136289 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3095 (79'93,173'3095] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3093 active] append_log  adding 1 keys
2014-06-12 14:46:04.136308 7f9d683e1700 10 trim log((79'93,173'3095], crt=0'0) to 79'94
2014-06-12 14:46:04.136354 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3095, trimmed: 79'94
2014-06-12 14:46:04.136381 7f9d683e1700  5 -- op tracker -- , seq: 747555, time: 2014-06-12 14:46:04.136377, event: started, op: osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.136400 7f9d683e1700  5 queue_transactions existing osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:04.136404 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.136406 7f9d683e1700  5 queue_op 0x581cc60 seq 2868 osr(3.46 0x276cc90) 1034511 bytes   (queue has 1 ops and 1034511 bytes)
2014-06-12 14:46:04.136420 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e97200 (1 active)
2014-06-12 14:46:04.136414 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.46( v 173'3095 (79'94,173'3095] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3093 active] do_sub_op osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.136428 7f9d683e1700 10 osd.2 173 dequeue_op 0x20355200 finish
2014-06-12 14:46:04.136428 7f9d7895e700  5 _do_op 0x581cc60 seq 2868 osr(3.46 0x276cc90)/0x276cc90 start
2014-06-12 14:46:04.136431 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.136434 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.136434 7f9d7895e700 10 _do_transaction on 0x2c8f1270
2014-06-12 14:46:04.136440 7f9d7895e700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.136574 7f9d7895e700 10 lookup_strip_header done  cid meta oid 51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.136597 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.136652 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.136670 7f9d7895e700 15 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.136679 7f9d7895e700 10 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1 r = 0
2014-06-12 14:46:04.136685 7f9d7895e700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.136698 7f9d7895e700 15 _write 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 532480~1032192
2014-06-12 14:46:04.136834 7f9d7895e700 10 lookup_strip_header done  cid 3.46_head oid 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.137013 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:04.140846 7f9d7895e700 10 _generic_write 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 532480~1032192 = 0
2014-06-12 14:46:04.140926 7f9d7895e700 15 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.140948 7f9d7895e700 10 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 = 0
2014-06-12 14:46:04.140956 7f9d7895e700 15 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.140964 7f9d7895e700 10 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 = 0
2014-06-12 14:46:04.141269 7f9d7895e700 20 set_header setting 17146 cid 3.46_head oid 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 parent seq 0
2014-06-12 14:46:04.141374 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.141396 7f9d7895e700 20 set_header setting 2172 cid meta oid 51681c0b/pglog_3.46/0//-1 parent seq 0
2014-06-12 14:46:04.141407 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:04.147940 7f9d7895e700 10 _do_op 0x581cc60 seq 2868 r = 0, finisher 0x57232c0 0
2014-06-12 14:46:04.147988 7f9d7715b700  5 -- op tracker -- , seq: 747555, time: 2014-06-12 14:46:04.147977, event: commit_sent, op: osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.148000 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3095 (79'94,173'3095] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3093 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:46:04.148029 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:04.148059 7f9d7895e700 10 _finish_op 0x581cc60 seq 2868 osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:04.148070 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e97200 (0 active)
2014-06-12 14:46:04.148073 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.148085 7f9d7795c700  5 -- op tracker -- , seq: 747555, time: 2014-06-12 14:46:04.148078, event: sub_op_applied, op: osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.148092 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3095 (79'94,173'3095] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3094 active] sub_op_modify_applied on 0x2c8f1180 op osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.148111 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3095 (79'94,173'3095] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3094 active] op_applied on primary on version 173'3095
2014-06-12 14:46:04.148123 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3095 (79'94,173'3095] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3094 active] op_applied on replica on version 173'3095
2014-06-12 14:46:04.148146 7f9d7795c700  5 -- op tracker -- , seq: 747555, time: 2014-06-12 14:46:04.148142, event: done, op: osd_sub_op(client.4302.0:146042 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3095 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.148570 7f9d44101700  5 -- op tracker -- , seq: 747556, time: 2014-06-12 14:46:04.146873, event: header_read, op: osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.148583 7f9d44101700  5 -- op tracker -- , seq: 747556, time: 2014-06-12 14:46:04.146875, event: throttled, op: osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.148593 7f9d44101700  5 -- op tracker -- , seq: 747556, time: 2014-06-12 14:46:04.148540, event: all_read, op: osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.148602 7f9d44101700  5 -- op tracker -- , seq: 747556, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.148610 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.148619 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:04.148626 7f9d44101700 15 osd.2 173 enqueue_op 0x20354400 prio 127 cost 1032995 latency 0.001753 osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.148654 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.148665 7f9d65bdc700 10 osd.2 173 dequeue_op 0x20354400 prio 127 cost 1032995 latency 0.001791 osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.46( v 173'3095 (79'94,173'3095] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3094 active]
2014-06-12 14:46:04.148689 7f9d65bdc700  5 -- op tracker -- , seq: 747556, time: 2014-06-12 14:46:04.148684, event: reached_pg, op: osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.148697 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3095 (79'94,173'3095] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3094 active] handle_message: 0x20354400
2014-06-12 14:46:04.148711 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3095 (79'94,173'3095] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3094 active] sub_op_modify trans 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 v 173'3096 (transaction) 174
2014-06-12 14:46:04.148731 7f9d65bdc700  5 -- op tracker -- , seq: 747556, time: 2014-06-12 14:46:04.148727, event: started, op: osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.148748 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3095 (79'94,173'3095] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3094 active] append_log log((79'94,173'3095], crt=0'0) [173'3096 (173'3095) modify   8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 by client.4302.0:146043 2014-06-12 08:45:06.522645]
2014-06-12 14:46:04.148771 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3096 (79'94,173'3096] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3094 active] add_log_entry 173'3096 (173'3095) modify   8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 by client.4302.0:146043 2014-06-12 08:45:06.522645
2014-06-12 14:46:04.148793 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3096 (79'94,173'3096] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3094 active] append_log  adding 1 keys
2014-06-12 14:46:04.148810 7f9d65bdc700 10 trim log((79'94,173'3096], crt=0'0) to 79'95
2014-06-12 14:46:04.148839 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3096, trimmed: 79'95
2014-06-12 14:46:04.148866 7f9d65bdc700  5 -- op tracker -- , seq: 747556, time: 2014-06-12 14:46:04.148862, event: started, op: osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.148877 7f9d65bdc700  5 queue_transactions existing osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:04.148881 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.148882 7f9d65bdc700  5 queue_op 0x76e6180 seq 2869 osr(3.46 0x276cc90) 1034511 bytes   (queue has 1 ops and 1034511 bytes)
2014-06-12 14:46:04.148889 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.46( v 173'3096 (79'95,173'3096] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3094 active] do_sub_op osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.148905 7f9d65bdc700 10 osd.2 173 dequeue_op 0x20354400 finish
2014-06-12 14:46:04.148909 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.148912 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.148920 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e97200 (1 active)
2014-06-12 14:46:04.148925 7f9d7e169700  5 _do_op 0x76e6180 seq 2869 osr(3.46 0x276cc90)/0x276cc90 start
2014-06-12 14:46:04.148929 7f9d7e169700 10 _do_transaction on 0x20a17eb0
2014-06-12 14:46:04.148935 7f9d7e169700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.149040 7f9d7e169700 10 lookup_strip_header done  cid meta oid 51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.149062 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.149112 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.149131 7f9d7e169700 15 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.149142 7f9d7e169700 10 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1 r = 0
2014-06-12 14:46:04.149149 7f9d7e169700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.149163 7f9d7e169700 15 _write 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 1564672~1032192
2014-06-12 14:46:04.149268 7f9d7e169700 10 lookup_strip_header done  cid 3.46_head oid 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.149363 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:04.151998 7f9d7e169700 10 _generic_write 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 1564672~1032192 = 0
2014-06-12 14:46:04.152076 7f9d7e169700 15 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.152093 7f9d7e169700 10 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 = 0
2014-06-12 14:46:04.152102 7f9d7e169700 15 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.152111 7f9d7e169700 10 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 = 0
2014-06-12 14:46:04.152319 7f9d7e169700 20 set_header setting 17146 cid 3.46_head oid 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 parent seq 0
2014-06-12 14:46:04.152393 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.152410 7f9d7e169700 20 set_header setting 2172 cid meta oid 51681c0b/pglog_3.46/0//-1 parent seq 0
2014-06-12 14:46:04.152428 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:04.152991 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:04.156441 7f9d7e169700 10 _do_op 0x76e6180 seq 2869 r = 0, finisher 0x6642fe0 0
2014-06-12 14:46:04.156488 7f9d7715b700  5 -- op tracker -- , seq: 747556, time: 2014-06-12 14:46:04.156475, event: commit_sent, op: osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.156507 7f9d7e169700 10 _finish_op 0x76e6180 seq 2869 osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:04.156497 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3096 (79'95,173'3096] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3094 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:46:04.156526 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:04.156532 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e97200 (0 active)
2014-06-12 14:46:04.156540 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.156591 7f9d7795c700  5 -- op tracker -- , seq: 747556, time: 2014-06-12 14:46:04.156583, event: sub_op_applied, op: osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.156602 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3096 (79'95,173'3096] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] sub_op_modify_applied on 0x20a17dc0 op osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.156620 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3096 (79'95,173'3096] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] op_applied on primary on version 173'3096
2014-06-12 14:46:04.156628 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3096 (79'95,173'3096] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] op_applied on replica on version 173'3096
2014-06-12 14:46:04.156647 7f9d7795c700  5 -- op tracker -- , seq: 747556, time: 2014-06-12 14:46:04.156643, event: done, op: osd_sub_op(client.4302.0:146043 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3096 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.157405 7f9d44101700  5 -- op tracker -- , seq: 747557, time: 2014-06-12 14:46:04.155729, event: header_read, op: osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.157426 7f9d44101700  5 -- op tracker -- , seq: 747557, time: 2014-06-12 14:46:04.155731, event: throttled, op: osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.157436 7f9d44101700  5 -- op tracker -- , seq: 747557, time: 2014-06-12 14:46:04.157366, event: all_read, op: osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.157447 7f9d44101700  5 -- op tracker -- , seq: 747557, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.157452 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.157460 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:04.157467 7f9d44101700 15 osd.2 173 enqueue_op 0x20e87700 prio 127 cost 1032995 latency 0.001738 osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.157528 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.157539 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20e87700 prio 127 cost 1032995 latency 0.001810 osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.46( v 173'3096 (79'95,173'3096] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active]
2014-06-12 14:46:04.157558 7f9d6abe6700  5 -- op tracker -- , seq: 747557, time: 2014-06-12 14:46:04.157554, event: reached_pg, op: osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.157568 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3096 (79'95,173'3096] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] handle_message: 0x20e87700
2014-06-12 14:46:04.157578 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3096 (79'95,173'3096] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] sub_op_modify trans 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 v 173'3097 (transaction) 174
2014-06-12 14:46:04.157592 7f9d6abe6700  5 -- op tracker -- , seq: 747557, time: 2014-06-12 14:46:04.157589, event: started, op: osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.157604 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3096 (79'95,173'3096] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] append_log log((79'95,173'3096], crt=0'0) [173'3097 (173'3096) modify   8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 by client.4302.0:146044 2014-06-12 08:45:06.531403]
2014-06-12 14:46:04.157640 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3097 (79'95,173'3097] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] add_log_entry 173'3097 (173'3096) modify   8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 by client.4302.0:146044 2014-06-12 08:45:06.531403
2014-06-12 14:46:04.157657 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3097 (79'95,173'3097] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] append_log  adding 1 keys
2014-06-12 14:46:04.157666 7f9d6abe6700 10 trim log((79'95,173'3097], crt=0'0) to 79'96
2014-06-12 14:46:04.157686 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3097, trimmed: 79'96
2014-06-12 14:46:04.157705 7f9d6abe6700  5 -- op tracker -- , seq: 747557, time: 2014-06-12 14:46:04.157702, event: started, op: osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.157714 7f9d6abe6700  5 queue_transactions existing osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:04.157716 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.157717 7f9d6abe6700  5 queue_op 0x15169800 seq 2870 osr(3.46 0x276cc90) 1034511 bytes   (queue has 1 ops and 1034511 bytes)
2014-06-12 14:46:04.157727 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e97200 (1 active)
2014-06-12 14:46:04.157733 7f9d7e96a700  5 _do_op 0x15169800 seq 2870 osr(3.46 0x276cc90)/0x276cc90 start
2014-06-12 14:46:04.157722 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.46( v 173'3097 (79'96,173'3097] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] do_sub_op osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.157736 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20e87700 finish
2014-06-12 14:46:04.157737 7f9d7e96a700 10 _do_transaction on 0x5ea15f0
2014-06-12 14:46:04.157739 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.157741 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.157742 7f9d7e96a700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.157842 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.157864 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.157901 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.157970 7f9d7e96a700 15 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.157977 7f9d7e96a700 10 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1 r = 0
2014-06-12 14:46:04.157982 7f9d7e96a700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.157994 7f9d7e96a700 15 _write 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 2596864~1032192
2014-06-12 14:46:04.158092 7f9d7e96a700 10 lookup_strip_header done  cid 3.46_head oid 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.158262 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:04.160736 7f9d7e96a700 10 _generic_write 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 2596864~1032192 = 0
2014-06-12 14:46:04.160813 7f9d7e96a700 15 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.160831 7f9d7e96a700 10 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 = 0
2014-06-12 14:46:04.160836 7f9d7e96a700 15 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.160841 7f9d7e96a700 10 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 = 0
2014-06-12 14:46:04.161041 7f9d7e96a700 20 set_header setting 17146 cid 3.46_head oid 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 parent seq 0
2014-06-12 14:46:04.161110 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.161121 7f9d7e96a700 20 set_header setting 2172 cid meta oid 51681c0b/pglog_3.46/0//-1 parent seq 0
2014-06-12 14:46:04.161129 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:04.164156 7f9d44101700  5 -- op tracker -- , seq: 747558, time: 2014-06-12 14:46:04.163317, event: header_read, op: osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.164169 7f9d44101700  5 -- op tracker -- , seq: 747558, time: 2014-06-12 14:46:04.163319, event: throttled, op: osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.164175 7f9d44101700  5 -- op tracker -- , seq: 747558, time: 2014-06-12 14:46:04.164117, event: all_read, op: osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.164180 7f9d44101700  5 -- op tracker -- , seq: 747558, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.164184 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.164188 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:04.164195 7f9d44101700 15 osd.2 173 enqueue_op 0x20e87500 prio 127 cost 566051 latency 0.000878 osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.164221 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.164230 7f9d66bde700 10 osd.2 173 dequeue_op 0x20e87500 prio 127 cost 566051 latency 0.000913 osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.46( v 173'3097 (79'96,173'3097] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active]
2014-06-12 14:46:04.164251 7f9d66bde700  5 -- op tracker -- , seq: 747558, time: 2014-06-12 14:46:04.164248, event: reached_pg, op: osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.164261 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3097 (79'96,173'3097] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] handle_message: 0x20e87500
2014-06-12 14:46:04.164270 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3097 (79'96,173'3097] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] sub_op_modify trans 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 v 173'3098 (transaction) 174
2014-06-12 14:46:04.164283 7f9d66bde700  5 -- op tracker -- , seq: 747558, time: 2014-06-12 14:46:04.164280, event: started, op: osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.164296 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3097 (79'96,173'3097] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] append_log log((79'96,173'3097], crt=0'0) [173'3098 (173'3097) modify   8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 by client.4302.0:146045 2014-06-12 08:45:06.541001]
2014-06-12 14:46:04.164315 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'96,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] add_log_entry 173'3098 (173'3097) modify   8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 by client.4302.0:146045 2014-06-12 08:45:06.541001
2014-06-12 14:46:04.164331 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'96,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] append_log  adding 1 keys
2014-06-12 14:46:04.164340 7f9d66bde700 10 trim log((79'96,173'3098], crt=0'0) to 79'97
2014-06-12 14:46:04.164361 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3098, trimmed: 79'97
2014-06-12 14:46:04.164384 7f9d66bde700  5 -- op tracker -- , seq: 747558, time: 2014-06-12 14:46:04.164380, event: started, op: osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.164392 7f9d66bde700  5 queue_transactions existing osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:04.164395 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.164397 7f9d66bde700  5 queue_op 0x11004900 seq 2871 osr(3.46 0x276cc90) 567567 bytes   (queue has 2 ops and 1602078 bytes)
2014-06-12 14:46:04.164401 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'97,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] do_sub_op osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.164405 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e97200 (2 active)
2014-06-12 14:46:04.164411 7f9d66bde700 10 osd.2 173 dequeue_op 0x20e87500 finish
2014-06-12 14:46:04.164413 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.164415 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.166865 7f9d7e96a700 10 _do_op 0x15169800 seq 2870 r = 0, finisher 0x6bb32a0 0
2014-06-12 14:46:04.166880 7f9d7e96a700 10 _finish_op 0x15169800 seq 2870 osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:04.166886 7f9d82171700  5 _do_op 0x11004900 seq 2871 osr(3.46 0x276cc90)/0x276cc90 start
2014-06-12 14:46:04.166887 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e97200 (1 active)
2014-06-12 14:46:04.166891 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.166891 7f9d82171700 10 _do_transaction on 0x82227f0
2014-06-12 14:46:04.166898 7f9d82171700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.166897 7f9d7715b700  5 -- op tracker -- , seq: 747557, time: 2014-06-12 14:46:04.166889, event: commit_sent, op: osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.166910 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'97,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3095 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:46:04.166932 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:04.166961 7f9d7795c700  5 -- op tracker -- , seq: 747557, time: 2014-06-12 14:46:04.166956, event: sub_op_applied, op: osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.166966 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'97,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3096 active] sub_op_modify_applied on 0x5ea1500 op osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.166977 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'97,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3096 active] op_applied on primary on version 173'3097
2014-06-12 14:46:04.166983 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'97,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3096 active] op_applied on replica on version 173'3097
2014-06-12 14:46:04.166988 7f9d82171700 10 lookup_strip_header done  cid meta oid 51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.166999 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.167000 7f9d7795c700  5 -- op tracker -- , seq: 747557, time: 2014-06-12 14:46:04.166998, event: done, op: osd_sub_op(client.4302.0:146044 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3097 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.167042 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.167055 7f9d82171700 15 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.167061 7f9d82171700 10 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1 r = 0
2014-06-12 14:46:04.167065 7f9d82171700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:04.167073 7f9d82171700 15 _write 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 3629056~565248
2014-06-12 14:46:04.167171 7f9d82171700 10 lookup_strip_header done  cid 3.46_head oid 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.167190 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:04.168364 7f9d82171700 10 _generic_write 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 3629056~565248 = 0
2014-06-12 14:46:04.168390 7f9d82171700 15 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.168399 7f9d82171700 10 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 = 0
2014-06-12 14:46:04.168404 7f9d82171700 15 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3
2014-06-12 14:46:04.168409 7f9d82171700 10 _setattrs 3.46_head/8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 = 0
2014-06-12 14:46:04.168526 7f9d82171700 20 set_header setting 17146 cid 3.46_head oid 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 parent seq 0
2014-06-12 14:46:04.168566 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.168575 7f9d82171700 20 set_header setting 2172 cid meta oid 51681c0b/pglog_3.46/0//-1 parent seq 0
2014-06-12 14:46:04.168584 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:04.171005 7f9d82171700 10 _do_op 0x11004900 seq 2871 r = 0, finisher 0x2ef7dd00 0
2014-06-12 14:46:04.171025 7f9d82171700 10 _finish_op 0x11004900 seq 2871 osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:04.171034 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e97200 (0 active)
2014-06-12 14:46:04.171044 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.171053 7f9d7715b700  5 -- op tracker -- , seq: 747558, time: 2014-06-12 14:46:04.171037, event: commit_sent, op: osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.171070 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'97,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3096 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:46:04.171116 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:04.171184 7f9d7795c700  5 -- op tracker -- , seq: 747558, time: 2014-06-12 14:46:04.171177, event: sub_op_applied, op: osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.171194 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'97,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active] sub_op_modify_applied on 0x8222700 op osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.171207 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'97,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active] op_applied on primary on version 173'3098
2014-06-12 14:46:04.171214 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'97,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active] op_applied on replica on version 173'3098
2014-06-12 14:46:04.171229 7f9d7795c700  5 -- op tracker -- , seq: 747558, time: 2014-06-12 14:46:04.171226, event: done, op: osd_sub_op(client.4302.0:146045 3.46 8384e846/rbd_data.10a56b8b4567.000000000000102f/head//3 [] v 173'3098 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.216049 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:04.216081 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:04.315223 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:04.315418 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:04.366639 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:04.439000 7f9d56432700  5 -- op tracker -- , seq: 747559, time: 2014-06-12 14:46:04.438952, event: header_read, op: osd_sub_op_reply(client.4306.0:144044 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.439009 7f9d56432700  5 -- op tracker -- , seq: 747559, time: 2014-06-12 14:46:04.438954, event: throttled, op: osd_sub_op_reply(client.4306.0:144044 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.439014 7f9d56432700  5 -- op tracker -- , seq: 747559, time: 2014-06-12 14:46:04.438968, event: all_read, op: osd_sub_op_reply(client.4306.0:144044 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.439018 7f9d56432700  5 -- op tracker -- , seq: 747559, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144044 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.439022 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144044 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:04.439026 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:04.439031 7f9d56432700 15 osd.2 173 enqueue_op 0x34017800 prio 196 cost 0 latency 0.000079 osd_sub_op_reply(client.4306.0:144044 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:04.439051 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.439059 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34017800 prio 196 cost 0 latency 0.000106 osd_sub_op_reply(client.4306.0:144044 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3080 crt=0'0 lcod 173'3079 mlcod 173'3079 active+clean]
2014-06-12 14:46:04.439075 7f9d6a3e5700  5 -- op tracker -- , seq: 747559, time: 2014-06-12 14:46:04.439073, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144044 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.439082 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3080 crt=0'0 lcod 173'3079 mlcod 173'3079 active+clean] handle_message: 0x34017800
2014-06-12 14:46:04.439091 7f9d6a3e5700  5 -- op tracker -- , seq: 747559, time: 2014-06-12 14:46:04.439089, event: started, op: osd_sub_op_reply(client.4306.0:144044 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.439096 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3080 crt=0'0 lcod 173'3079 mlcod 173'3079 active+clean] sub_op_modify_reply: tid 236000 op  ack_type 4 from 26
2014-06-12 14:46:04.439110 7f9d6a3e5700  5 -- op tracker -- , seq: 746931, time: 2014-06-12 14:46:04.439104, event: sub_op_commit_rec, op: osd_op(client.4306.0:144044 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:04.439115 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3080 crt=0'0 lcod 173'3079 mlcod 173'3079 active+clean] repop_all_applied: repop tid 236000 all applied 
2014-06-12 14:46:04.439122 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3080 crt=0'0 lcod 173'3079 mlcod 173'3079 active+clean] eval_repop repgather(0x3403a480 173'3081 rep_tid=236000 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144044 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:04.439132 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3080 crt=0'0 lcod 173'3079 mlcod 173'3079 active+clean]  sending ack on repgather(0x3403a480 173'3081 rep_tid=236000 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144044 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x2183a500
2014-06-12 14:46:04.439156 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3080 crt=0'0 lcod 173'3079 mlcod 173'3079 active+clean] repop_all_committed: repop tid 236000 all committed 
2014-06-12 14:46:04.439165 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3079 active+clean] eval_repop repgather(0x3403a480 173'3081 rep_tid=236000 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144044 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:04.439187 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3079 active+clean] log_op_stats osd_op(client.4306.0:144044 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4 inb 45335 outb 0 rlat 32.667227 lat 32.667258
2014-06-12 14:46:04.439203 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3079 active+clean] publish_stats_to_osd 173:3547
2014-06-12 14:46:04.439212 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3079 active+clean]  sending commit on repgather(0x3403a480 173'3081 rep_tid=236000 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144044 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x6700280
2014-06-12 14:46:04.439238 7f9d6a3e5700  5 -- op tracker -- , seq: 746931, time: 2014-06-12 14:46:04.439232, event: commit_sent, op: osd_op(client.4306.0:144044 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:04.439247 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3079 active+clean]  requeue_ops 
2014-06-12 14:46:04.439258 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean]  removing repgather(0x3403a480 173'3081 rep_tid=236000 committed?=1 applied?=1 op=osd_op(client.4306.0:144044 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:04.439267 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean]    q front is repgather(0x3403a480 173'3081 rep_tid=236000 committed?=1 applied?=1 op=osd_op(client.4306.0:144044 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:04.439276 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean] remove_repop repgather(0x3403a480 173'3081 rep_tid=236000 committed?=1 applied?=1 op=osd_op(client.4306.0:144044 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:04.439296 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean]  requeue_ops 
2014-06-12 14:46:04.439311 7f9d6a3e5700  5 -- op tracker -- , seq: 746931, time: 2014-06-12 14:46:04.439308, event: done, op: osd_op(client.4306.0:144044 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:04.439341 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34017800 finish
2014-06-12 14:46:04.439347 7f9d6a3e5700  5 -- op tracker -- , seq: 747559, time: 2014-06-12 14:46:04.439345, event: done, op: osd_sub_op_reply(client.4306.0:144044 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.439362 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.439364 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.449753 7f9d56432700  5 -- op tracker -- , seq: 747560, time: 2014-06-12 14:46:04.447071, event: header_read, op: osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.449768 7f9d56432700  5 -- op tracker -- , seq: 747560, time: 2014-06-12 14:46:04.447076, event: throttled, op: osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.449773 7f9d56432700  5 -- op tracker -- , seq: 747560, time: 2014-06-12 14:46:04.449702, event: all_read, op: osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.449778 7f9d56432700  5 -- op tracker -- , seq: 747560, time: 0.000000, event: dispatched, op: osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.449783 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.449789 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:04.449796 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3eb000 prio 127 cost 1209127 latency 0.002724 osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.449833 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.449842 7f9d653db700 10 osd.2 173 dequeue_op 0x1e3eb000 prio 127 cost 1209127 latency 0.002771 osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.25a( v 173'2667 (0'0,173'2667] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2666 active]
2014-06-12 14:46:04.449861 7f9d653db700  5 -- op tracker -- , seq: 747560, time: 2014-06-12 14:46:04.449858, event: reached_pg, op: osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.449869 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2667 (0'0,173'2667] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2666 active] handle_message: 0x1e3eb000
2014-06-12 14:46:04.449877 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2667 (0'0,173'2667] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2666 active] sub_op_modify trans 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 v 173'2668 (transaction) 174
2014-06-12 14:46:04.449888 7f9d653db700  5 -- op tracker -- , seq: 747560, time: 2014-06-12 14:46:04.449886, event: started, op: osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.449901 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2667 (0'0,173'2667] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2666 active] append_log log((0'0,173'2667], crt=0'0) [173'2668 (173'2667) modify   2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 by client.4311.0:139685 2014-06-12 08:44:59.127356]
2014-06-12 14:46:04.449925 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2668 (0'0,173'2668] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2666 active] add_log_entry 173'2668 (173'2667) modify   2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 by client.4311.0:139685 2014-06-12 08:44:59.127356
2014-06-12 14:46:04.449940 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2668 (0'0,173'2668] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2666 active] append_log  adding 1 keys
2014-06-12 14:46:04.449964 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2668, trimmed: 
2014-06-12 14:46:04.449981 7f9d653db700  5 -- op tracker -- , seq: 747560, time: 2014-06-12 14:46:04.449978, event: started, op: osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.449988 7f9d653db700  5 queue_transactions existing osr(3.25a 0x276d340)/0x276d340
2014-06-12 14:46:04.449991 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.449992 7f9d653db700  5 queue_op 0xa44c300 seq 2503 osr(3.25a 0x276d340) 1210613 bytes   (queue has 1 ops and 1210613 bytes)
2014-06-12 14:46:04.450002 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x543e480 (1 active)
2014-06-12 14:46:04.450008 7f9d7d167700  5 _do_op 0xa44c300 seq 2503 osr(3.25a 0x276d340)/0x276d340 start
2014-06-12 14:46:04.449998 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.25a( v 173'2668 (0'0,173'2668] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2666 active] do_sub_op osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.450011 7f9d653db700 10 osd.2 173 dequeue_op 0x1e3eb000 finish
2014-06-12 14:46:04.450012 7f9d7d167700 10 _do_transaction on 0x6232ef0
2014-06-12 14:46:04.450014 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.450017 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.450019 7f9d7d167700 15 _omap_setkeys meta/1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:46:04.450178 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:46:04.450205 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.450266 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.450372 7f9d7d167700 15 _omap_rmkeys meta/1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:46:04.450382 7f9d7d167700 10 _omap_rmkeys meta/1fe2073b/pglog_3.25a/0//-1 r = 0
2014-06-12 14:46:04.450389 7f9d7d167700 15 _omap_setkeys meta/1fe2073b/pglog_3.25a/0//-1
2014-06-12 14:46:04.450410 7f9d7d167700 15 _write 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 2985984~1208320
2014-06-12 14:46:04.450564 7f9d7d167700 10 lookup_strip_header done  cid 3.25a_head oid 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:04.450729 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:04.454877 7f9d7d167700 10 _generic_write 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 2985984~1208320 = 0
2014-06-12 14:46:04.455014 7f9d7d167700 15 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:04.455032 7f9d7d167700 10 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:04.455038 7f9d7d167700 15 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:04.455046 7f9d7d167700 10 _setattrs 3.25a_head/2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:04.455394 7f9d7d167700 20 set_header setting 16076 cid 3.25a_head oid 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:46:04.455533 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.455553 7f9d7d167700 20 set_header setting 2302 cid meta oid 1fe2073b/pglog_3.25a/0//-1 parent seq 0
2014-06-12 14:46:04.455562 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:04.461440 7f9d5bec4700  5 -- op tracker -- , seq: 747561, time: 2014-06-12 14:46:04.457168, event: header_read, op: osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.461452 7f9d5bec4700  5 -- op tracker -- , seq: 747561, time: 2014-06-12 14:46:04.457170, event: throttled, op: osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.461461 7f9d5bec4700  5 -- op tracker -- , seq: 747561, time: 2014-06-12 14:46:04.461394, event: all_read, op: osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.461470 7f9d5bec4700  5 -- op tracker -- , seq: 747561, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.461475 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.461481 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:04.461491 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e87f00 prio 127 cost 488231 latency 0.004322 osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.461526 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.461534 7f9d63bd8700 10 osd.2 173 dequeue_op 0x20e87f00 prio 127 cost 488231 latency 0.004366 osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2854 (0'0,173'2854] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2853 active]
2014-06-12 14:46:04.461557 7f9d63bd8700  5 -- op tracker -- , seq: 747561, time: 2014-06-12 14:46:04.461553, event: reached_pg, op: osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.461566 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2854 (0'0,173'2854] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2853 active] handle_message: 0x20e87f00
2014-06-12 14:46:04.461580 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2854 (0'0,173'2854] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2853 active] sub_op_modify trans 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 v 173'2855 (transaction) 174
2014-06-12 14:46:04.461598 7f9d63bd8700  5 -- op tracker -- , seq: 747561, time: 2014-06-12 14:46:04.461593, event: started, op: osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.461627 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2854 (0'0,173'2854] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2853 active] append_log log((0'0,173'2854], crt=0'0) [173'2855 (173'834) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144102 2014-06-12 08:45:06.822128]
2014-06-12 14:46:04.461676 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2855 (0'0,173'2855] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2853 active] add_log_entry 173'2855 (173'834) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144102 2014-06-12 08:45:06.822128
2014-06-12 14:46:04.461699 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2855 (0'0,173'2855] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2853 active] append_log  adding 1 keys
2014-06-12 14:46:04.461731 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2855, trimmed: 
2014-06-12 14:46:04.461753 7f9d63bd8700  5 -- op tracker -- , seq: 747561, time: 2014-06-12 14:46:04.461749, event: started, op: osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.461764 7f9d63bd8700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.461768 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.461769 7f9d63bd8700  5 queue_op 0x5e7cf00 seq 2621 osr(3.402 0x276d620) 489717 bytes   (queue has 2 ops and 1700330 bytes)
2014-06-12 14:46:04.461776 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2855 (0'0,173'2855] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2853 active] do_sub_op osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.461790 7f9d63bd8700 10 osd.2 173 dequeue_op 0x20e87f00 finish
2014-06-12 14:46:04.461794 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.461796 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.462310 7f9d7d167700 10 _do_op 0xa44c300 seq 2503 r = 0, finisher 0x2e1ce960 0
2014-06-12 14:46:04.462345 7f9d7715b700  5 -- op tracker -- , seq: 747560, time: 2014-06-12 14:46:04.462338, event: commit_sent, op: osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.462354 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2668 (0'0,173'2668] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2666 active] sub_op_modify_commit on op osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:04.462377 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:04.462421 7f9d7d167700 10 _finish_op 0xa44c300 seq 2503 osr(3.25a 0x276d340)/0x276d340
2014-06-12 14:46:04.462447 7f9d7795c700  5 -- op tracker -- , seq: 747560, time: 2014-06-12 14:46:04.462438, event: sub_op_applied, op: osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.462455 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2668 (0'0,173'2668] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2667 active] sub_op_modify_applied on 0x6232e00 op osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.462470 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2668 (0'0,173'2668] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2667 active] op_applied on primary on version 173'2668
2014-06-12 14:46:04.462480 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.25a( v 173'2668 (0'0,173'2668] local-les=153 n=293 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2667 active] op_applied on replica on version 173'2668
2014-06-12 14:46:04.462511 7f9d7d167700  5 -- op tracker -- , seq: 747560, time: 2014-06-12 14:46:04.462501, event: done, op: osd_sub_op(client.4311.0:139685 3.25a 2a5bb25a/rbd_data.10ae6b8b4567.0000000000000ecc/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.462541 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x543e480 (0 active)
2014-06-12 14:46:04.462544 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:46:04.462546 7f9d7d167700  5 _do_op 0x5e7cf00 seq 2621 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:46:04.462550 7f9d7d167700 10 _do_transaction on 0x5fe90b0
2014-06-12 14:46:04.462556 7f9d7d167700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.462757 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.462775 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.462829 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.462850 7f9d7d167700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.462856 7f9d7d167700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:46:04.462862 7f9d7d167700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.462875 7f9d7d167700 15 _write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 0~487424
2014-06-12 14:46:04.463058 7f9d7d167700 10 lookup_strip_header done  cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.463179 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:04.464931 7f9d7d167700 10 _generic_write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 0~487424 = 0
2014-06-12 14:46:04.464985 7f9d7d167700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.465001 7f9d7d167700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.465012 7f9d7d167700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.465021 7f9d7d167700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.465218 7f9d7d167700 20 set_header setting 16798 cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 parent seq 0
2014-06-12 14:46:04.465377 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.465392 7f9d7d167700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:46:04.465402 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:04.465968 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.467671 7f9d7d167700 10 _do_op 0x5e7cf00 seq 2621 r = 0, finisher 0x80522c0 0
2014-06-12 14:46:04.467768 7f9d7715b700  5 -- op tracker -- , seq: 747561, time: 2014-06-12 14:46:04.467757, event: commit_sent, op: osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.467781 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2855 (0'0,173'2855] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2853 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:04.467809 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:04.467856 7f9d7d167700 10 _finish_op 0x5e7cf00 seq 2621 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.467880 7f9d7795c700  5 -- op tracker -- , seq: 747561, time: 2014-06-12 14:46:04.467875, event: sub_op_applied, op: osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.467888 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2855 (0'0,173'2855] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2854 active] sub_op_modify_applied on 0x5fe8fc0 op osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.467906 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2855 (0'0,173'2855] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2854 active] op_applied on primary on version 173'2855
2014-06-12 14:46:04.467916 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2855 (0'0,173'2855] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2854 active] op_applied on replica on version 173'2855
2014-06-12 14:46:04.467950 7f9d7d167700  5 -- op tracker -- , seq: 747561, time: 2014-06-12 14:46:04.467940, event: done, op: osd_sub_op(client.4306.0:144102 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.467985 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:46:04.467989 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.469634 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:04.469650 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:04.469653 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:04.469655 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:04.469656 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:04.469657 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 33; oldest is 34.189756 seconds old
2014-06-12 14:46:04.585321 7f9d56432700  5 -- op tracker -- , seq: 747562, time: 2014-06-12 14:46:04.584089, event: header_read, op: osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.585334 7f9d56432700  5 -- op tracker -- , seq: 747562, time: 2014-06-12 14:46:04.584091, event: throttled, op: osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.585341 7f9d56432700  5 -- op tracker -- , seq: 747562, time: 2014-06-12 14:46:04.585285, event: all_read, op: osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.585349 7f9d56432700  5 -- op tracker -- , seq: 747562, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.585354 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.585360 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:04.585368 7f9d56432700 15 osd.2 173 enqueue_op 0x25e42f00 prio 127 cost 520995 latency 0.001279 osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.585406 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.585412 7f9d69be4700 10 osd.2 173 dequeue_op 0x25e42f00 prio 127 cost 520995 latency 0.001323 osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2472 (0'0,173'2472] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2471 active]
2014-06-12 14:46:04.585434 7f9d69be4700  5 -- op tracker -- , seq: 747562, time: 2014-06-12 14:46:04.585429, event: reached_pg, op: osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.585442 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2472 (0'0,173'2472] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2471 active] handle_message: 0x25e42f00
2014-06-12 14:46:04.585453 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2472 (0'0,173'2472] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2471 active] sub_op_modify trans 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 v 173'2473 (transaction) 174
2014-06-12 14:46:04.585469 7f9d69be4700  5 -- op tracker -- , seq: 747562, time: 2014-06-12 14:46:04.585465, event: started, op: osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.585481 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2472 (0'0,173'2472] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2471 active] append_log log((0'0,173'2472], crt=0'0) [173'2473 (173'2471) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144989 2014-06-12 08:44:57.431125]
2014-06-12 14:46:04.585502 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2473 (0'0,173'2473] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2471 active] add_log_entry 173'2473 (173'2471) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144989 2014-06-12 08:44:57.431125
2014-06-12 14:46:04.585522 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2473 (0'0,173'2473] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2471 active] append_log  adding 1 keys
2014-06-12 14:46:04.585551 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2473, trimmed: 
2014-06-12 14:46:04.585572 7f9d69be4700  5 -- op tracker -- , seq: 747562, time: 2014-06-12 14:46:04.585567, event: started, op: osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.585580 7f9d69be4700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:04.585585 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.585586 7f9d69be4700  5 queue_op 0x68ebc20 seq 2286 osr(3.2b 0x276cd20) 522476 bytes   (queue has 1 ops and 522476 bytes)
2014-06-12 14:46:04.585594 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2473 (0'0,173'2473] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2471 active] do_sub_op osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.585607 7f9d69be4700 10 osd.2 173 dequeue_op 0x25e42f00 finish
2014-06-12 14:46:04.585610 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.585665 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:46:04.585672 7f9d79960700  5 _do_op 0x68ebc20 seq 2286 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:46:04.585677 7f9d79960700 10 _do_transaction on 0x65aeef0
2014-06-12 14:46:04.585684 7f9d79960700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:04.585839 7f9d79960700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:04.585857 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.585914 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.585938 7f9d79960700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:04.585945 7f9d79960700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:46:04.585951 7f9d79960700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:04.586160 7f9d79960700 15 _write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 1310720~520192
2014-06-12 14:46:04.586333 7f9d79960700 10 lookup_strip_header done  cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:04.586469 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:04.588492 7f9d79960700 10 _generic_write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 1310720~520192 = 0
2014-06-12 14:46:04.588554 7f9d79960700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:04.588571 7f9d79960700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:04.588578 7f9d79960700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:04.588587 7f9d79960700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:04.588803 7f9d79960700 20 set_header setting 17866 cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 parent seq 0
2014-06-12 14:46:04.588870 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.588886 7f9d79960700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:46:04.588895 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:04.589054 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.590698 7f9d56432700  5 -- op tracker -- , seq: 747563, time: 2014-06-12 14:46:04.588855, event: header_read, op: osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.590711 7f9d56432700  5 -- op tracker -- , seq: 747563, time: 2014-06-12 14:46:04.588861, event: throttled, op: osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.590720 7f9d56432700  5 -- op tracker -- , seq: 747563, time: 2014-06-12 14:46:04.590659, event: all_read, op: osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.590729 7f9d56432700  5 -- op tracker -- , seq: 747563, time: 0.000000, event: dispatched, op: osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.590741 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.590748 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:04.590757 7f9d56432700 15 osd.2 173 enqueue_op 0x23e09f00 prio 127 cost 1032999 latency 0.001901 osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.590784 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.590793 7f9d633d7700 10 osd.2 173 dequeue_op 0x23e09f00 prio 127 cost 1032999 latency 0.001937 osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2987 (0'0,173'2987] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2986 active]
2014-06-12 14:46:04.590822 7f9d633d7700  5 -- op tracker -- , seq: 747563, time: 2014-06-12 14:46:04.590817, event: reached_pg, op: osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.590837 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2987 (0'0,173'2987] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2986 active] handle_message: 0x23e09f00
2014-06-12 14:46:04.590849 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2987 (0'0,173'2987] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2986 active] sub_op_modify trans 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 v 173'2988 (transaction) 174
2014-06-12 14:46:04.590866 7f9d633d7700  5 -- op tracker -- , seq: 747563, time: 2014-06-12 14:46:04.590862, event: started, op: osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.590880 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2987 (0'0,173'2987] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2986 active] append_log log((0'0,173'2987], crt=0'0) [173'2988 (173'2986) modify   6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 by client.4307.0:146570 2014-06-12 08:44:32.733307]
2014-06-12 14:46:04.590904 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2988 (0'0,173'2988] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2986 active] add_log_entry 173'2988 (173'2986) modify   6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 by client.4307.0:146570 2014-06-12 08:44:32.733307
2014-06-12 14:46:04.590926 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2988 (0'0,173'2988] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2986 active] append_log  adding 1 keys
2014-06-12 14:46:04.590961 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2988, trimmed: 
2014-06-12 14:46:04.590985 7f9d633d7700  5 -- op tracker -- , seq: 747563, time: 2014-06-12 14:46:04.590980, event: started, op: osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.590994 7f9d633d7700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:04.590998 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.591000 7f9d633d7700  5 queue_op 0x91a22a0 seq 2790 osr(3.458 0x276d960) 1034485 bytes   (queue has 2 ops and 1556961 bytes)
2014-06-12 14:46:04.591008 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2988 (0'0,173'2988] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2986 active] do_sub_op osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.591023 7f9d633d7700 10 osd.2 173 dequeue_op 0x23e09f00 finish
2014-06-12 14:46:04.591027 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.591030 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.591039 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (2 active)
2014-06-12 14:46:04.591045 7f9d8116f700  5 _do_op 0x91a22a0 seq 2790 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:04.591049 7f9d8116f700 10 _do_transaction on 0x274a7eb0
2014-06-12 14:46:04.591055 7f9d8116f700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.591188 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.591204 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.592309 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.592480 7f9d8116f700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.592492 7f9d8116f700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:04.592505 7f9d8116f700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.592529 7f9d8116f700 15 _write 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 2813952~1032192
2014-06-12 14:46:04.592539 7f9d79960700 10 _do_op 0x68ebc20 seq 2286 r = 0, finisher 0x57bdde0 0
2014-06-12 14:46:04.592554 7f9d79960700 10 _finish_op 0x68ebc20 seq 2286 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:04.592563 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (1 active)
2014-06-12 14:46:04.592574 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.592663 7f9d8116f700 10 lookup_strip_header done  cid 3.458_head oid 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:46:04.592805 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:04.594295 7f9d7715b700  5 -- op tracker -- , seq: 747562, time: 2014-06-12 14:46:04.594283, event: commit_sent, op: osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.594308 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2473 (0'0,173'2473] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2471 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:04.594336 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:04.594788 7f9d7795c700  5 -- op tracker -- , seq: 747562, time: 2014-06-12 14:46:04.594781, event: sub_op_applied, op: osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.594795 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2473 (0'0,173'2473] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2472 active] sub_op_modify_applied on 0x65aee00 op osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.594806 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2473 (0'0,173'2473] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2472 active] op_applied on primary on version 173'2473
2014-06-12 14:46:04.594812 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2473 (0'0,173'2473] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2472 active] op_applied on replica on version 173'2473
2014-06-12 14:46:04.594829 7f9d7795c700  5 -- op tracker -- , seq: 747562, time: 2014-06-12 14:46:04.594827, event: done, op: osd_sub_op(client.4314.0:144989 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.596705 7f9d8116f700 10 _generic_write 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 2813952~1032192 = 0
2014-06-12 14:46:04.596809 7f9d8116f700 15 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:46:04.596836 7f9d8116f700 10 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 = 0
2014-06-12 14:46:04.596842 7f9d8116f700 15 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:46:04.596851 7f9d8116f700 10 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 = 0
2014-06-12 14:46:04.597164 7f9d8116f700 20 set_header setting 18434 cid 3.458_head oid 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 parent seq 0
2014-06-12 14:46:04.597286 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.597301 7f9d8116f700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:04.597311 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:04.603747 7f9d8116f700 10 _do_op 0x91a22a0 seq 2790 r = 0, finisher 0x8a01820 0
2014-06-12 14:46:04.603809 7f9d7715b700  5 -- op tracker -- , seq: 747563, time: 2014-06-12 14:46:04.603797, event: commit_sent, op: osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.603828 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2988 (0'0,173'2988] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2986 active] sub_op_modify_commit on op osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:04.603860 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:04.603920 7f9d8116f700 10 _finish_op 0x91a22a0 seq 2790 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:04.603946 7f9d7795c700  5 -- op tracker -- , seq: 747563, time: 2014-06-12 14:46:04.603941, event: sub_op_applied, op: osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.603953 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2988 (0'0,173'2988] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2987 active] sub_op_modify_applied on 0x274a7dc0 op osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.603969 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2988 (0'0,173'2988] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2987 active] op_applied on primary on version 173'2988
2014-06-12 14:46:04.603979 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2988 (0'0,173'2988] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2987 active] op_applied on replica on version 173'2988
2014-06-12 14:46:04.604011 7f9d8116f700  5 -- op tracker -- , seq: 747563, time: 2014-06-12 14:46:04.604003, event: done, op: osd_sub_op(client.4307.0:146570 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.604042 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:04.604045 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.652502 7f9d56432700  5 -- op tracker -- , seq: 747564, time: 2014-06-12 14:46:04.648089, event: header_read, op: osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.652517 7f9d56432700  5 -- op tracker -- , seq: 747564, time: 2014-06-12 14:46:04.648091, event: throttled, op: osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.652527 7f9d56432700  5 -- op tracker -- , seq: 747564, time: 2014-06-12 14:46:04.652453, event: all_read, op: osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.652536 7f9d56432700  5 -- op tracker -- , seq: 747564, time: 0.000000, event: dispatched, op: osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.652541 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.652548 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:04.652557 7f9d56432700 15 osd.2 173 enqueue_op 0x876f500 prio 127 cost 2581287 latency 0.004467 osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.653360 7f9d56432700  5 -- op tracker -- , seq: 747565, time: 2014-06-12 14:46:04.652589, event: header_read, op: osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.653373 7f9d56432700  5 -- op tracker -- , seq: 747565, time: 2014-06-12 14:46:04.652590, event: throttled, op: osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.653382 7f9d56432700  5 -- op tracker -- , seq: 747565, time: 2014-06-12 14:46:04.653333, event: all_read, op: osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.653395 7f9d56432700  5 -- op tracker -- , seq: 747565, time: 0.000000, event: dispatched, op: osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.653399 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.653404 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:04.653410 7f9d56432700 15 osd.2 173 enqueue_op 0x17c2b200 prio 127 cost 516899 latency 0.000821 osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.653437 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.653444 7f9d68be2700 10 osd.2 173 dequeue_op 0x876f500 prio 127 cost 2581287 latency 0.005355 osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.15c( v 173'2643 (0'0,173'2643] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active]
2014-06-12 14:46:04.653470 7f9d68be2700  5 -- op tracker -- , seq: 747564, time: 2014-06-12 14:46:04.653465, event: reached_pg, op: osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.653479 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2643 (0'0,173'2643] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] handle_message: 0x876f500
2014-06-12 14:46:04.653494 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2643 (0'0,173'2643] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] sub_op_modify trans 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 v 173'2644 (transaction) 174
2014-06-12 14:46:04.653514 7f9d68be2700  5 -- op tracker -- , seq: 747564, time: 2014-06-12 14:46:04.653509, event: started, op: osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.653534 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2643 (0'0,173'2643] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] append_log log((0'0,173'2643], crt=0'0) [173'2644 (173'2643) modify   805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 by client.4279.0:143441 2014-06-12 08:44:31.049768]
2014-06-12 14:46:04.653563 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2644 (0'0,173'2644] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] add_log_entry 173'2644 (173'2643) modify   805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 by client.4279.0:143441 2014-06-12 08:44:31.049768
2014-06-12 14:46:04.653591 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2644 (0'0,173'2644] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] append_log  adding 1 keys
2014-06-12 14:46:04.653636 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2644, trimmed: 
2014-06-12 14:46:04.653666 7f9d68be2700  5 -- op tracker -- , seq: 747564, time: 2014-06-12 14:46:04.653660, event: started, op: osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.653679 7f9d68be2700  5 queue_transactions existing osr(3.15c 0x276ced0)/0x276ced0
2014-06-12 14:46:04.653684 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.653685 7f9d68be2700  5 queue_op 0x7305860 seq 2440 osr(3.15c 0x276ced0) 2582773 bytes   (queue has 1 ops and 2582773 bytes)
2014-06-12 14:46:04.653693 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.15c( v 173'2644 (0'0,173'2644] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] do_sub_op osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.653708 7f9d68be2700 10 osd.2 173 dequeue_op 0x876f500 finish
2014-06-12 14:46:04.653712 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.653717 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.653721 7f9d68be2700 10 osd.2 173 dequeue_op 0x17c2b200 prio 127 cost 516899 latency 0.001132 osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2473 (0'0,173'2473] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2472 active]
2014-06-12 14:46:04.653740 7f9d68be2700  5 -- op tracker -- , seq: 747565, time: 2014-06-12 14:46:04.653736, event: reached_pg, op: osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.653750 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2473 (0'0,173'2473] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2472 active] handle_message: 0x17c2b200
2014-06-12 14:46:04.653761 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2473 (0'0,173'2473] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2472 active] sub_op_modify trans 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 v 173'2474 (transaction) 174
2014-06-12 14:46:04.653777 7f9d68be2700  5 -- op tracker -- , seq: 747565, time: 2014-06-12 14:46:04.653773, event: started, op: osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.653790 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2473 (0'0,173'2473] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2472 active] append_log log((0'0,173'2473], crt=0'0) [173'2474 (173'2472) modify   4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 by client.4280.0:139528 2014-06-12 08:43:27.842797]
2014-06-12 14:46:04.653815 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2474 (0'0,173'2474] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2472 active] add_log_entry 173'2474 (173'2472) modify   4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 by client.4280.0:139528 2014-06-12 08:43:27.842797
2014-06-12 14:46:04.653838 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2474 (0'0,173'2474] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2472 active] append_log  adding 1 keys
2014-06-12 14:46:04.653869 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2474, trimmed: 
2014-06-12 14:46:04.653898 7f9d68be2700  5 -- op tracker -- , seq: 747565, time: 2014-06-12 14:46:04.653893, event: started, op: osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.653909 7f9d68be2700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:04.653913 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.653914 7f9d68be2700  5 queue_op 0x5f66120 seq 2287 osr(3.2b 0x276cd20) 518380 bytes   (queue has 2 ops and 3101153 bytes)
2014-06-12 14:46:04.653920 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2474 (0'0,173'2474] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2472 active] do_sub_op osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.653934 7f9d68be2700 10 osd.2 173 dequeue_op 0x17c2b200 finish
2014-06-12 14:46:04.653938 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.653940 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.653948 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54bde60 (1 active)
2014-06-12 14:46:04.653953 7f9d8096e700  5 _do_op 0x7305860 seq 2440 osr(3.15c 0x276ced0)/0x276ced0 start
2014-06-12 14:46:04.653957 7f9d8096e700 10 _do_transaction on 0x1d027b30
2014-06-12 14:46:04.653964 7f9d8096e700 15 _omap_setkeys meta/1fe2b14b/pglog_3.15c/0//-1
2014-06-12 14:46:04.654107 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe2b14b/pglog_3.15c/0//-1
2014-06-12 14:46:04.654133 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.654195 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.654300 7f9d8096e700 15 _omap_rmkeys meta/1fe2b14b/pglog_3.15c/0//-1
2014-06-12 14:46:04.654310 7f9d8096e700 10 _omap_rmkeys meta/1fe2b14b/pglog_3.15c/0//-1 r = 0
2014-06-12 14:46:04.654317 7f9d8096e700 15 _omap_setkeys meta/1fe2b14b/pglog_3.15c/0//-1
2014-06-12 14:46:04.654335 7f9d8096e700 15 _write 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 663552~2580480
2014-06-12 14:46:04.654500 7f9d8096e700 10 lookup_strip_header done  cid 3.15c_head oid 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3
2014-06-12 14:46:04.654765 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:04.657627 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.657624 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (2 active)
2014-06-12 14:46:04.658793 7f9d8016d700  5 _do_op 0x5f66120 seq 2287 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:46:04.658804 7f9d8016d700 10 _do_transaction on 0x6274d30
2014-06-12 14:46:04.658815 7f9d8016d700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:04.659002 7f9d8016d700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:04.659029 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.669956 7f9d8096e700 10 _generic_write 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 663552~2580480 = 0
2014-06-12 14:46:04.670247 7f9d8096e700 15 _setattrs 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3
2014-06-12 14:46:04.670271 7f9d8096e700 10 _setattrs 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:04.670279 7f9d8096e700 15 _setattrs 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3
2014-06-12 14:46:04.670288 7f9d8096e700 10 _setattrs 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:04.670923 7f9d8096e700 20 set_header setting 16993 cid 3.15c_head oid 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 parent seq 0
2014-06-12 14:46:04.671204 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.671230 7f9d8096e700 20 set_header setting 2322 cid meta oid 1fe2b14b/pglog_3.15c/0//-1 parent seq 0
2014-06-12 14:46:04.671244 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:04.685502 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.685540 7f9d8016d700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:04.685546 7f9d8016d700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:46:04.685554 7f9d8016d700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:04.685568 7f9d8016d700 15 _write 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 2965504~516096
2014-06-12 14:46:04.685821 7f9d8016d700 10 lookup_strip_header done  cid 3.2b_head oid 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:46:04.686288 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:04.688476 7f9d8016d700 10 _generic_write 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 2965504~516096 = 0
2014-06-12 14:46:04.688530 7f9d8016d700 15 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:46:04.688545 7f9d8016d700 10 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 = 0
2014-06-12 14:46:04.688551 7f9d8016d700 15 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:46:04.688559 7f9d8016d700 10 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 = 0
2014-06-12 14:46:04.688759 7f9d8016d700 20 set_header setting 15771 cid 3.2b_head oid 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 parent seq 0
2014-06-12 14:46:04.688833 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.688848 7f9d8016d700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:46:04.688859 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:04.690274 7f9d8096e700 10 _do_op 0x7305860 seq 2440 r = 0, finisher 0x57bcf00 0
2014-06-12 14:46:04.690299 7f9d8096e700 10 _finish_op 0x7305860 seq 2440 osr(3.15c 0x276ced0)/0x276ced0
2014-06-12 14:46:04.690307 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54bde60 (1 active)
2014-06-12 14:46:04.690316 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.690319 7f9d7715b700  5 -- op tracker -- , seq: 747564, time: 2014-06-12 14:46:04.690308, event: commit_sent, op: osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.690330 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2644 (0'0,173'2644] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] sub_op_modify_commit on op osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:04.690355 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:04.690388 7f9d7795c700  5 -- op tracker -- , seq: 747564, time: 2014-06-12 14:46:04.690383, event: sub_op_applied, op: osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.690395 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2644 (0'0,173'2644] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] sub_op_modify_applied on 0x1d027a40 op osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.690410 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2644 (0'0,173'2644] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] op_applied on primary on version 173'2644
2014-06-12 14:46:04.690420 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2644 (0'0,173'2644] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] op_applied on replica on version 173'2644
2014-06-12 14:46:04.690450 7f9d7795c700  5 -- op tracker -- , seq: 747564, time: 2014-06-12 14:46:04.690445, event: done, op: osd_sub_op(client.4279.0:143441 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.693289 7f9d8016d700 10 _do_op 0x5f66120 seq 2287 r = 0, finisher 0x8357b60 0
2014-06-12 14:46:04.693332 7f9d7715b700  5 -- op tracker -- , seq: 747565, time: 2014-06-12 14:46:04.693321, event: commit_sent, op: osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.693342 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2474 (0'0,173'2474] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2472 active] sub_op_modify_commit on op osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:04.693374 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:04.693403 7f9d8016d700 10 _finish_op 0x5f66120 seq 2287 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:04.693427 7f9d7795c700  5 -- op tracker -- , seq: 747565, time: 2014-06-12 14:46:04.693421, event: sub_op_applied, op: osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.693434 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2474 (0'0,173'2474] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2473 active] sub_op_modify_applied on 0x6274c40 op osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.693451 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2474 (0'0,173'2474] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2473 active] op_applied on primary on version 173'2474
2014-06-12 14:46:04.693462 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2474 (0'0,173'2474] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2473 active] op_applied on replica on version 173'2474
2014-06-12 14:46:04.693491 7f9d8016d700  5 -- op tracker -- , seq: 747565, time: 2014-06-12 14:46:04.693484, event: done, op: osd_sub_op(client.4280.0:139528 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.693549 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:46:04.693554 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.795200 7f9d56432700  5 -- op tracker -- , seq: 747566, time: 2014-06-12 14:46:04.795150, event: header_read, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ack, result = 0)
2014-06-12 14:46:04.795218 7f9d56432700  5 -- op tracker -- , seq: 747566, time: 2014-06-12 14:46:04.795151, event: throttled, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ack, result = 0)
2014-06-12 14:46:04.795226 7f9d56432700  5 -- op tracker -- , seq: 747566, time: 2014-06-12 14:46:04.795164, event: all_read, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ack, result = 0)
2014-06-12 14:46:04.795234 7f9d56432700  5 -- op tracker -- , seq: 747566, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ack, result = 0)
2014-06-12 14:46:04.795239 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:04.795244 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:04.795259 7f9d56432700 15 osd.2 173 enqueue_op 0x5abe200 prio 196 cost 0 latency 0.000109 osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ack, result = 0) v2
2014-06-12 14:46:04.795288 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.795294 7f9d643d9700 10 osd.2 173 dequeue_op 0x5abe200 prio 196 cost 0 latency 0.000144 osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ack, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean]
2014-06-12 14:46:04.795315 7f9d643d9700  5 -- op tracker -- , seq: 747566, time: 2014-06-12 14:46:04.795311, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ack, result = 0)
2014-06-12 14:46:04.795323 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean] handle_message: 0x5abe200
2014-06-12 14:46:04.795340 7f9d643d9700  5 -- op tracker -- , seq: 747566, time: 2014-06-12 14:46:04.795336, event: started, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ack, result = 0)
2014-06-12 14:46:04.795347 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean] sub_op_modify_reply: tid 236001 op  ack_type 1 from 26
2014-06-12 14:46:04.795367 7f9d643d9700  5 -- op tracker -- , seq: 746932, time: 2014-06-12 14:46:04.795359, event: sub_op_applied_rec, op: osd_op(client.4306.0:144045 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:04.795374 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean] repop_all_applied: repop tid 236001 all applied 
2014-06-12 14:46:04.795385 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean] eval_repop repgather(0xbe0c000 173'3082 rep_tid=236001 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144045 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:04.795402 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean]  sending ack on repgather(0xbe0c000 173'3082 rep_tid=236001 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144045 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x355c0280
2014-06-12 14:46:04.795436 7f9d643d9700 10 osd.2 173 dequeue_op 0x5abe200 finish
2014-06-12 14:46:04.795446 7f9d643d9700  5 -- op tracker -- , seq: 747566, time: 2014-06-12 14:46:04.795442, event: done, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ack, result = 0)
2014-06-12 14:46:04.795481 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.795485 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.800138 7f9d56432700  5 -- op tracker -- , seq: 747567, time: 2014-06-12 14:46:04.800083, event: header_read, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.800149 7f9d56432700  5 -- op tracker -- , seq: 747567, time: 2014-06-12 14:46:04.800085, event: throttled, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.800157 7f9d56432700  5 -- op tracker -- , seq: 747567, time: 2014-06-12 14:46:04.800104, event: all_read, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.800164 7f9d56432700  5 -- op tracker -- , seq: 747567, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.800168 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:04.800173 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:04.800180 7f9d56432700 15 osd.2 173 enqueue_op 0x28dd0000 prio 196 cost 0 latency 0.000096 osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:04.800205 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.800212 7f9d61bd4700 10 osd.2 173 dequeue_op 0x28dd0000 prio 196 cost 0 latency 0.000128 osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean]
2014-06-12 14:46:04.800231 7f9d61bd4700  5 -- op tracker -- , seq: 747567, time: 2014-06-12 14:46:04.800227, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.800238 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean] handle_message: 0x28dd0000
2014-06-12 14:46:04.800252 7f9d61bd4700  5 -- op tracker -- , seq: 747567, time: 2014-06-12 14:46:04.800249, event: started, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.800258 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean] sub_op_modify_reply: tid 236001 op  ack_type 4 from 26
2014-06-12 14:46:04.800276 7f9d61bd4700  5 -- op tracker -- , seq: 746932, time: 2014-06-12 14:46:04.800269, event: sub_op_commit_rec, op: osd_op(client.4306.0:144045 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:04.800282 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3081 crt=0'0 lcod 173'3080 mlcod 173'3080 active+clean] repop_all_committed: repop tid 236001 all committed 
2014-06-12 14:46:04.800292 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3080 active+clean] eval_repop repgather(0xbe0c000 173'3082 rep_tid=236001 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144045 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:04.800307 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3080 active+clean] log_op_stats osd_op(client.4306.0:144045 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 33.022180 lat 33.027052
2014-06-12 14:46:04.800322 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3080 active+clean] publish_stats_to_osd 173:3548
2014-06-12 14:46:04.800333 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3080 active+clean]  sending commit on repgather(0xbe0c000 173'3082 rep_tid=236001 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144045 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x25914280
2014-06-12 14:46:04.800367 7f9d61bd4700  5 -- op tracker -- , seq: 746932, time: 2014-06-12 14:46:04.800362, event: commit_sent, op: osd_op(client.4306.0:144045 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:04.800374 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3080 active+clean]  requeue_ops 
2014-06-12 14:46:04.800384 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3081 active+clean]  removing repgather(0xbe0c000 173'3082 rep_tid=236001 committed?=1 applied?=1 op=osd_op(client.4306.0:144045 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:04.800398 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3081 active+clean]    q front is repgather(0xbe0c000 173'3082 rep_tid=236001 committed?=1 applied?=1 op=osd_op(client.4306.0:144045 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:04.800411 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3081 active+clean] remove_repop repgather(0xbe0c000 173'3082 rep_tid=236001 committed?=1 applied?=1 op=osd_op(client.4306.0:144045 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:04.800423 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3081 active+clean]  requeue_ops 
2014-06-12 14:46:04.800441 7f9d61bd4700  5 -- op tracker -- , seq: 746932, time: 2014-06-12 14:46:04.800437, event: done, op: osd_op(client.4306.0:144045 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:04.800457 7f9d61bd4700 10 osd.2 173 dequeue_op 0x28dd0000 finish
2014-06-12 14:46:04.800463 7f9d61bd4700  5 -- op tracker -- , seq: 747567, time: 2014-06-12 14:46:04.800460, event: done, op: osd_sub_op_reply(client.4306.0:144045 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:04.800472 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.800474 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.807076 7f9d5bec4700  5 -- op tracker -- , seq: 747568, time: 2014-06-12 14:46:04.805871, event: header_read, op: osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.807089 7f9d5bec4700  5 -- op tracker -- , seq: 747568, time: 2014-06-12 14:46:04.805874, event: throttled, op: osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.807096 7f9d5bec4700  5 -- op tracker -- , seq: 747568, time: 2014-06-12 14:46:04.807035, event: all_read, op: osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.807104 7f9d5bec4700  5 -- op tracker -- , seq: 747568, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.807109 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.807115 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:04.807122 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e87e00 prio 127 cost 516903 latency 0.001251 osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.807152 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.807158 7f9d663dd700 10 osd.2 173 dequeue_op 0x20e87e00 prio 127 cost 516903 latency 0.001287 osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2855 (0'0,173'2855] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2854 active]
2014-06-12 14:46:04.807178 7f9d663dd700  5 -- op tracker -- , seq: 747568, time: 2014-06-12 14:46:04.807174, event: reached_pg, op: osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.807186 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2855 (0'0,173'2855] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2854 active] handle_message: 0x20e87e00
2014-06-12 14:46:04.807199 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2855 (0'0,173'2855] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2854 active] sub_op_modify trans 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 v 173'2856 (transaction) 174
2014-06-12 14:46:04.807215 7f9d663dd700  5 -- op tracker -- , seq: 747568, time: 2014-06-12 14:46:04.807211, event: started, op: osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.807230 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2855 (0'0,173'2855] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2854 active] append_log log((0'0,173'2855], crt=0'0) [173'2856 (173'2855) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144103 2014-06-12 08:45:07.183018]
2014-06-12 14:46:04.807251 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2856 (0'0,173'2856] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2854 active] add_log_entry 173'2856 (173'2855) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144103 2014-06-12 08:45:07.183018
2014-06-12 14:46:04.807272 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2856 (0'0,173'2856] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2854 active] append_log  adding 1 keys
2014-06-12 14:46:04.807309 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2856, trimmed: 
2014-06-12 14:46:04.807338 7f9d663dd700  5 -- op tracker -- , seq: 747568, time: 2014-06-12 14:46:04.807333, event: started, op: osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.807347 7f9d663dd700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.807351 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.807352 7f9d663dd700  5 queue_op 0x544c420 seq 2622 osr(3.402 0x276d620) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:04.807365 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:46:04.807360 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2856 (0'0,173'2856] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2854 active] do_sub_op osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.807373 7f9d7d968700  5 _do_op 0x544c420 seq 2622 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:46:04.807375 7f9d663dd700 10 osd.2 173 dequeue_op 0x20e87e00 finish
2014-06-12 14:46:04.807378 7f9d7d968700 10 _do_transaction on 0xe0d67f0
2014-06-12 14:46:04.807378 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.807381 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.807382 7f9d7d968700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.807543 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.807562 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.807617 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.807644 7f9d7d968700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.807650 7f9d7d968700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:46:04.807656 7f9d7d968700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.807896 7f9d7d968700 15 _write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 487424~516096
2014-06-12 14:46:04.808055 7f9d7d968700 10 lookup_strip_header done  cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.808207 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:04.810647 7f9d7d968700 10 _generic_write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 487424~516096 = 0
2014-06-12 14:46:04.810715 7f9d7d968700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.810735 7f9d7d968700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.810745 7f9d7d968700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.810755 7f9d7d968700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.810971 7f9d7d968700 20 set_header setting 16798 cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 parent seq 0
2014-06-12 14:46:04.811042 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.811059 7f9d7d968700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:46:04.811071 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:04.814932 7f9d7d968700 10 _do_op 0x544c420 seq 2622 r = 0, finisher 0x8587100 0
2014-06-12 14:46:04.814973 7f9d7715b700  5 -- op tracker -- , seq: 747568, time: 2014-06-12 14:46:04.814964, event: commit_sent, op: osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.814984 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2856 (0'0,173'2856] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2854 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:04.815011 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:04.815063 7f9d7d968700 10 _finish_op 0x544c420 seq 2622 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.815087 7f9d7795c700  5 -- op tracker -- , seq: 747568, time: 2014-06-12 14:46:04.815081, event: sub_op_applied, op: osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.815096 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2856 (0'0,173'2856] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2855 active] sub_op_modify_applied on 0xe0d6700 op osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.815114 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2856 (0'0,173'2856] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2855 active] op_applied on primary on version 173'2856
2014-06-12 14:46:04.815125 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2856 (0'0,173'2856] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2855 active] op_applied on replica on version 173'2856
2014-06-12 14:46:04.815154 7f9d7d968700  5 -- op tracker -- , seq: 747568, time: 2014-06-12 14:46:04.815146, event: done, op: osd_sub_op(client.4306.0:144103 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.815185 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:46:04.815191 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.816111 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:04.825834 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:04.826093 7f9d5bec4700  5 -- op tracker -- , seq: 747569, time: 2014-06-12 14:46:04.821059, event: header_read, op: osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.826105 7f9d5bec4700  5 -- op tracker -- , seq: 747569, time: 2014-06-12 14:46:04.821062, event: throttled, op: osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.826114 7f9d5bec4700  5 -- op tracker -- , seq: 747569, time: 2014-06-12 14:46:04.826059, event: all_read, op: osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.826123 7f9d5bec4700  5 -- op tracker -- , seq: 747569, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.826129 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.826137 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:04.826146 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e86900 prio 127 cost 516903 latency 0.005087 osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.827886 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.827903 7f9d673df700 10 osd.2 173 dequeue_op 0x20e86900 prio 127 cost 516903 latency 0.006844 osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2856 (0'0,173'2856] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2855 active]
2014-06-12 14:46:04.827938 7f9d673df700  5 -- op tracker -- , seq: 747569, time: 2014-06-12 14:46:04.827933, event: reached_pg, op: osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.827954 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2856 (0'0,173'2856] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2855 active] handle_message: 0x20e86900
2014-06-12 14:46:04.827967 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2856 (0'0,173'2856] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2855 active] sub_op_modify trans 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 v 173'2857 (transaction) 174
2014-06-12 14:46:04.827986 7f9d673df700  5 -- op tracker -- , seq: 747569, time: 2014-06-12 14:46:04.827981, event: started, op: osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.828024 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2856 (0'0,173'2856] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2855 active] append_log log((0'0,173'2856], crt=0'0) [173'2857 (173'2856) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144104 2014-06-12 08:45:07.198787]
2014-06-12 14:46:04.828053 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2857 (0'0,173'2857] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2855 active] add_log_entry 173'2857 (173'2856) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144104 2014-06-12 08:45:07.198787
2014-06-12 14:46:04.828078 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2857 (0'0,173'2857] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2855 active] append_log  adding 1 keys
2014-06-12 14:46:04.828132 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2857, trimmed: 
2014-06-12 14:46:04.828161 7f9d673df700  5 -- op tracker -- , seq: 747569, time: 2014-06-12 14:46:04.828156, event: started, op: osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.828174 7f9d673df700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.828181 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.828184 7f9d673df700  5 queue_op 0x883a240 seq 2623 osr(3.402 0x276d620) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:04.828196 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2857 (0'0,173'2857] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2855 active] do_sub_op osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.828212 7f9d673df700 10 osd.2 173 dequeue_op 0x20e86900 finish
2014-06-12 14:46:04.828216 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.828219 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.829632 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:46:04.829652 7f9d7915f700  5 _do_op 0x883a240 seq 2623 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:46:04.829660 7f9d7915f700 10 _do_transaction on 0x61e2ef0
2014-06-12 14:46:04.829674 7f9d7915f700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.829873 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.829896 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.829955 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.829971 7f9d7915f700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.829976 7f9d7915f700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:46:04.829983 7f9d7915f700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.829998 7f9d7915f700 15 _write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 1003520~516096
2014-06-12 14:46:04.830154 7f9d7915f700 10 lookup_strip_header done  cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.830267 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:04.832734 7f9d7915f700 10 _generic_write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 1003520~516096 = 0
2014-06-12 14:46:04.832817 7f9d7915f700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.832835 7f9d7915f700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.832841 7f9d7915f700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.832851 7f9d7915f700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.833091 7f9d7915f700 20 set_header setting 16798 cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 parent seq 0
2014-06-12 14:46:04.833156 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.833189 7f9d7915f700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:46:04.833198 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:04.836317 7f9d7915f700 10 _do_op 0x883a240 seq 2623 r = 0, finisher 0x83568c0 0
2014-06-12 14:46:04.836970 7f9d7715b700  5 -- op tracker -- , seq: 747569, time: 2014-06-12 14:46:04.836957, event: commit_sent, op: osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.836983 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2857 (0'0,173'2857] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2855 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:04.837021 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:04.837111 7f9d7915f700 10 _finish_op 0x883a240 seq 2623 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.837153 7f9d7795c700  5 -- op tracker -- , seq: 747569, time: 2014-06-12 14:46:04.837147, event: sub_op_applied, op: osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.837161 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2857 (0'0,173'2857] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2856 active] sub_op_modify_applied on 0x61e2e00 op osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.837179 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2857 (0'0,173'2857] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2856 active] op_applied on primary on version 173'2857
2014-06-12 14:46:04.837190 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2857 (0'0,173'2857] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2856 active] op_applied on replica on version 173'2857
2014-06-12 14:46:04.837273 7f9d7915f700  5 -- op tracker -- , seq: 747569, time: 2014-06-12 14:46:04.837263, event: done, op: osd_sub_op(client.4306.0:144104 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.837331 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:46:04.837356 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.842298 7f9d56432700  5 -- op tracker -- , seq: 747570, time: 2014-06-12 14:46:04.838899, event: header_read, op: osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.842311 7f9d56432700  5 -- op tracker -- , seq: 747570, time: 2014-06-12 14:46:04.838901, event: throttled, op: osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.842319 7f9d56432700  5 -- op tracker -- , seq: 747570, time: 2014-06-12 14:46:04.842254, event: all_read, op: osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.842327 7f9d56432700  5 -- op tracker -- , seq: 747570, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.842333 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.842340 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:04.842348 7f9d56432700 15 osd.2 173 enqueue_op 0x23e08c00 prio 127 cost 1549095 latency 0.003449 osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.842381 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.842387 7f9d67be0700 10 osd.2 173 dequeue_op 0x23e08c00 prio 127 cost 1549095 latency 0.003488 osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2988 (0'0,173'2988] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2987 active]
2014-06-12 14:46:04.842408 7f9d67be0700  5 -- op tracker -- , seq: 747570, time: 2014-06-12 14:46:04.842404, event: reached_pg, op: osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.842416 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2988 (0'0,173'2988] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2987 active] handle_message: 0x23e08c00
2014-06-12 14:46:04.842428 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2988 (0'0,173'2988] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2987 active] sub_op_modify trans 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 v 173'2989 (transaction) 174
2014-06-12 14:46:04.842444 7f9d67be0700  5 -- op tracker -- , seq: 747570, time: 2014-06-12 14:46:04.842440, event: started, op: osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.842460 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2988 (0'0,173'2988] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2987 active] append_log log((0'0,173'2988], crt=0'0) [173'2989 (173'2987) modify   3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 by client.4298.0:145089 2014-06-12 08:43:26.678000]
2014-06-12 14:46:04.842484 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2989 (0'0,173'2989] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2987 active] add_log_entry 173'2989 (173'2987) modify   3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 by client.4298.0:145089 2014-06-12 08:43:26.678000
2014-06-12 14:46:04.842506 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2989 (0'0,173'2989] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2987 active] append_log  adding 1 keys
2014-06-12 14:46:04.842539 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2989, trimmed: 
2014-06-12 14:46:04.842562 7f9d67be0700  5 -- op tracker -- , seq: 747570, time: 2014-06-12 14:46:04.842558, event: started, op: osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.842573 7f9d67be0700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:04.842577 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.842584 7f9d67be0700  5 queue_op 0x285a3020 seq 2791 osr(3.458 0x276d960) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:46:04.842591 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2989 (0'0,173'2989] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2987 active] do_sub_op osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.842605 7f9d67be0700 10 osd.2 173 dequeue_op 0x23e08c00 finish
2014-06-12 14:46:04.842608 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.842611 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.842619 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:04.842624 7f9d7c165700  5 _do_op 0x285a3020 seq 2791 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:04.842628 7f9d7c165700 10 _do_transaction on 0x9b92ef0
2014-06-12 14:46:04.842633 7f9d7c165700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.842780 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.842807 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.842871 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.842896 7f9d7c165700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.842901 7f9d7c165700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:04.842906 7f9d7c165700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.843081 7f9d7c165700 15 _write 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 770048~1548288
2014-06-12 14:46:04.843235 7f9d7c165700 10 lookup_strip_header done  cid 3.458_head oid 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:46:04.843375 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:04.853096 7f9d7c165700 10 _generic_write 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 770048~1548288 = 0
2014-06-12 14:46:04.853394 7f9d7c165700 15 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:46:04.853412 7f9d7c165700 10 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 = 0
2014-06-12 14:46:04.853420 7f9d7c165700 15 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:46:04.853428 7f9d7c165700 10 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 = 0
2014-06-12 14:46:04.854110 7f9d7c165700 20 set_header setting 16953 cid 3.458_head oid 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 parent seq 0
2014-06-12 14:46:04.854471 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.854487 7f9d7c165700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:04.854500 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:04.856559 7f9d5bec4700  5 -- op tracker -- , seq: 747571, time: 2014-06-12 14:46:04.849693, event: header_read, op: osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.856574 7f9d5bec4700  5 -- op tracker -- , seq: 747571, time: 2014-06-12 14:46:04.849696, event: throttled, op: osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.856582 7f9d5bec4700  5 -- op tracker -- , seq: 747571, time: 2014-06-12 14:46:04.856517, event: all_read, op: osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.856591 7f9d5bec4700  5 -- op tracker -- , seq: 747571, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.856597 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.856603 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:04.856612 7f9d5bec4700 15 osd.2 173 enqueue_op 0x31d60400 prio 127 cost 516903 latency 0.006919 osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.856659 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.856666 7f9d693e3700 10 osd.2 173 dequeue_op 0x31d60400 prio 127 cost 516903 latency 0.006973 osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2857 (0'0,173'2857] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2856 active]
2014-06-12 14:46:04.856689 7f9d693e3700  5 -- op tracker -- , seq: 747571, time: 2014-06-12 14:46:04.856684, event: reached_pg, op: osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.856697 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2857 (0'0,173'2857] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2856 active] handle_message: 0x31d60400
2014-06-12 14:46:04.856708 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2857 (0'0,173'2857] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2856 active] sub_op_modify trans 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 v 173'2858 (transaction) 174
2014-06-12 14:46:04.856724 7f9d693e3700  5 -- op tracker -- , seq: 747571, time: 2014-06-12 14:46:04.856720, event: started, op: osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.856739 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2857 (0'0,173'2857] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2856 active] append_log log((0'0,173'2857], crt=0'0) [173'2858 (173'2857) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144105 2014-06-12 08:45:07.222281]
2014-06-12 14:46:04.856763 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2858 (0'0,173'2858] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2856 active] add_log_entry 173'2858 (173'2857) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144105 2014-06-12 08:45:07.222281
2014-06-12 14:46:04.856783 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2858 (0'0,173'2858] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2856 active] append_log  adding 1 keys
2014-06-12 14:46:04.856817 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2858, trimmed: 
2014-06-12 14:46:04.856849 7f9d693e3700  5 -- op tracker -- , seq: 747571, time: 2014-06-12 14:46:04.856845, event: started, op: osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.856858 7f9d693e3700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.856863 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.856864 7f9d693e3700  5 queue_op 0x684d680 seq 2624 osr(3.402 0x276d620) 518389 bytes   (queue has 2 ops and 2068970 bytes)
2014-06-12 14:46:04.856873 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2858 (0'0,173'2858] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2856 active] do_sub_op osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.856887 7f9d693e3700 10 osd.2 173 dequeue_op 0x31d60400 finish
2014-06-12 14:46:04.856890 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.856893 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.861628 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (2 active)
2014-06-12 14:46:04.861651 7f9d83173700  5 _do_op 0x684d680 seq 2624 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:46:04.861656 7f9d83173700 10 _do_transaction on 0x24eb0d30
2014-06-12 14:46:04.861665 7f9d83173700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.861803 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.861821 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.863556 7f9d7c165700 10 _do_op 0x285a3020 seq 2791 r = 0, finisher 0x7759e80 0
2014-06-12 14:46:04.863587 7f9d7c165700 10 _finish_op 0x285a3020 seq 2791 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:04.863596 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (1 active)
2014-06-12 14:46:04.863606 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.863613 7f9d7715b700  5 -- op tracker -- , seq: 747570, time: 2014-06-12 14:46:04.863599, event: commit_sent, op: osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.863626 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2989 (0'0,173'2989] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2987 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:04.863657 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:04.863697 7f9d7795c700  5 -- op tracker -- , seq: 747570, time: 2014-06-12 14:46:04.863692, event: sub_op_applied, op: osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.863704 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2989 (0'0,173'2989] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2988 active] sub_op_modify_applied on 0x9b92e00 op osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.863720 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2989 (0'0,173'2989] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2988 active] op_applied on primary on version 173'2989
2014-06-12 14:46:04.863731 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2989 (0'0,173'2989] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2988 active] op_applied on replica on version 173'2989
2014-06-12 14:46:04.863765 7f9d7795c700  5 -- op tracker -- , seq: 747570, time: 2014-06-12 14:46:04.863760, event: done, op: osd_sub_op(client.4298.0:145089 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.864847 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.864898 7f9d83173700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.864906 7f9d83173700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:46:04.864913 7f9d83173700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.865058 7f9d83173700 15 _write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 1519616~516096
2014-06-12 14:46:04.865230 7f9d83173700 10 lookup_strip_header done  cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.865503 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:04.867897 7f9d83173700 10 _generic_write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 1519616~516096 = 0
2014-06-12 14:46:04.867962 7f9d83173700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.867989 7f9d83173700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.867995 7f9d83173700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.868005 7f9d83173700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.868243 7f9d83173700 20 set_header setting 16798 cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 parent seq 0
2014-06-12 14:46:04.868305 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.868319 7f9d83173700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:46:04.868327 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:04.877756 7f9d83173700 10 _do_op 0x684d680 seq 2624 r = 0, finisher 0x8595ca0 0
2014-06-12 14:46:04.877777 7f9d83173700 10 _finish_op 0x684d680 seq 2624 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.877793 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:46:04.877794 7f9d7715b700  5 -- op tracker -- , seq: 747571, time: 2014-06-12 14:46:04.877785, event: commit_sent, op: osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.877804 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.877806 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2858 (0'0,173'2858] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2856 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:04.877828 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:04.877873 7f9d7795c700  5 -- op tracker -- , seq: 747571, time: 2014-06-12 14:46:04.877867, event: sub_op_applied, op: osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.877879 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2858 (0'0,173'2858] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2857 active] sub_op_modify_applied on 0x24eb0c40 op osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.877890 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2858 (0'0,173'2858] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2857 active] op_applied on primary on version 173'2858
2014-06-12 14:46:04.877896 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2858 (0'0,173'2858] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2857 active] op_applied on replica on version 173'2858
2014-06-12 14:46:04.877912 7f9d7795c700  5 -- op tracker -- , seq: 747571, time: 2014-06-12 14:46:04.877909, event: done, op: osd_sub_op(client.4306.0:144105 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.888673 7f9d5bec4700  5 -- op tracker -- , seq: 747572, time: 2014-06-12 14:46:04.883552, event: header_read, op: osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.888687 7f9d5bec4700  5 -- op tracker -- , seq: 747572, time: 2014-06-12 14:46:04.883554, event: throttled, op: osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.888696 7f9d5bec4700  5 -- op tracker -- , seq: 747572, time: 2014-06-12 14:46:04.888630, event: all_read, op: osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.888704 7f9d5bec4700  5 -- op tracker -- , seq: 747572, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.888710 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.888717 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:04.888725 7f9d5bec4700 15 osd.2 173 enqueue_op 0x31d60f00 prio 127 cost 516903 latency 0.005173 osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.888765 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.888772 7f9d62bd6700 10 osd.2 173 dequeue_op 0x31d60f00 prio 127 cost 516903 latency 0.005220 osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2858 (0'0,173'2858] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2857 active]
2014-06-12 14:46:04.888794 7f9d62bd6700  5 -- op tracker -- , seq: 747572, time: 2014-06-12 14:46:04.888790, event: reached_pg, op: osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.888803 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2858 (0'0,173'2858] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2857 active] handle_message: 0x31d60f00
2014-06-12 14:46:04.888815 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2858 (0'0,173'2858] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2857 active] sub_op_modify trans 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 v 173'2859 (transaction) 174
2014-06-12 14:46:04.888831 7f9d62bd6700  5 -- op tracker -- , seq: 747572, time: 2014-06-12 14:46:04.888827, event: started, op: osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.888847 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2858 (0'0,173'2858] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2857 active] append_log log((0'0,173'2858], crt=0'0) [173'2859 (173'2858) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144106 2014-06-12 08:45:07.260578]
2014-06-12 14:46:04.888871 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2859 (0'0,173'2859] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2857 active] add_log_entry 173'2859 (173'2858) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144106 2014-06-12 08:45:07.260578
2014-06-12 14:46:04.888892 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2859 (0'0,173'2859] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2857 active] append_log  adding 1 keys
2014-06-12 14:46:04.888927 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2859, trimmed: 
2014-06-12 14:46:04.888951 7f9d62bd6700  5 -- op tracker -- , seq: 747572, time: 2014-06-12 14:46:04.888946, event: started, op: osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.888961 7f9d62bd6700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.888965 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.888967 7f9d62bd6700  5 queue_op 0x76e7c80 seq 2625 osr(3.402 0x276d620) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:04.888980 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:46:04.888975 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2859 (0'0,173'2859] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2857 active] do_sub_op osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.888989 7f9d62bd6700 10 osd.2 173 dequeue_op 0x31d60f00 finish
2014-06-12 14:46:04.888989 7f9d82972700  5 _do_op 0x76e7c80 seq 2625 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:46:04.888994 7f9d82972700 10 _do_transaction on 0x62c8b70
2014-06-12 14:46:04.888993 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.888999 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.889000 7f9d82972700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.889161 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.889191 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.889248 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.889363 7f9d82972700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.889370 7f9d82972700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:46:04.889377 7f9d82972700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.889400 7f9d82972700 15 _write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 2035712~516096
2014-06-12 14:46:04.889591 7f9d82972700 10 lookup_strip_header done  cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.889792 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:04.892091 7f9d82972700 10 _generic_write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 2035712~516096 = 0
2014-06-12 14:46:04.892168 7f9d82972700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.892184 7f9d82972700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.892190 7f9d82972700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.892198 7f9d82972700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.892421 7f9d82972700 20 set_header setting 16798 cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 parent seq 0
2014-06-12 14:46:04.892493 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.892507 7f9d82972700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:46:04.892518 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:04.895029 7f9d56432700  5 -- op tracker -- , seq: 747573, time: 2014-06-12 14:46:04.894104, event: header_read, op: osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.895041 7f9d56432700  5 -- op tracker -- , seq: 747573, time: 2014-06-12 14:46:04.894106, event: throttled, op: osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.895051 7f9d56432700  5 -- op tracker -- , seq: 747573, time: 2014-06-12 14:46:04.894988, event: all_read, op: osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.895059 7f9d56432700  5 -- op tracker -- , seq: 747573, time: 0.000000, event: dispatched, op: osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.895064 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.895071 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:04.895079 7f9d56432700 15 osd.2 173 enqueue_op 0x876e900 prio 127 cost 348967 latency 0.000975 osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.895106 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.895114 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x876e900 prio 127 cost 348967 latency 0.001010 osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2989 (0'0,173'2989] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2988 active]
2014-06-12 14:46:04.895139 7f9d6b3e7700  5 -- op tracker -- , seq: 747573, time: 2014-06-12 14:46:04.895135, event: reached_pg, op: osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.895154 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2989 (0'0,173'2989] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2988 active] handle_message: 0x876e900
2014-06-12 14:46:04.895168 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2989 (0'0,173'2989] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2988 active] sub_op_modify trans 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 v 173'2990 (transaction) 174
2014-06-12 14:46:04.895186 7f9d6b3e7700  5 -- op tracker -- , seq: 747573, time: 2014-06-12 14:46:04.895182, event: started, op: osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.895209 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2989 (0'0,173'2989] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2988 active] append_log log((0'0,173'2989], crt=0'0) [173'2990 (173'2988) modify   6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 by client.4307.0:146571 2014-06-12 08:44:32.748059]
2014-06-12 14:46:04.895236 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2990 (0'0,173'2990] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2988 active] add_log_entry 173'2990 (173'2988) modify   6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 by client.4307.0:146571 2014-06-12 08:44:32.748059
2014-06-12 14:46:04.895258 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2990 (0'0,173'2990] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2988 active] append_log  adding 1 keys
2014-06-12 14:46:04.895296 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2990, trimmed: 
2014-06-12 14:46:04.895322 7f9d6b3e7700  5 -- op tracker -- , seq: 747573, time: 2014-06-12 14:46:04.895316, event: started, op: osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.895332 7f9d6b3e7700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:04.895336 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.895338 7f9d6b3e7700  5 queue_op 0x283b9da0 seq 2792 osr(3.458 0x276d960) 350453 bytes   (queue has 2 ops and 868842 bytes)
2014-06-12 14:46:04.895353 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (2 active)
2014-06-12 14:46:04.895361 7f9d7c966700  5 _do_op 0x283b9da0 seq 2792 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:04.895348 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2990 (0'0,173'2990] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2988 active] do_sub_op osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.895364 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x876e900 finish
2014-06-12 14:46:04.895365 7f9d7c966700 10 _do_transaction on 0x29637b0
2014-06-12 14:46:04.895368 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.895371 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.895371 7f9d7c966700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.895492 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.895520 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.895572 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.895677 7f9d7c966700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.895683 7f9d7c966700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:04.895682 7f9d82972700 10 _do_op 0x76e7c80 seq 2625 r = 0, finisher 0x1d499080 0
2014-06-12 14:46:04.895689 7f9d7c966700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:04.895707 7f9d7c966700 15 _write 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 3846144~348160
2014-06-12 14:46:04.895712 7f9d7715b700  5 -- op tracker -- , seq: 747572, time: 2014-06-12 14:46:04.895705, event: commit_sent, op: osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.895721 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2859 (0'0,173'2859] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2857 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:04.895743 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:04.895788 7f9d82972700 10 _finish_op 0x76e7c80 seq 2625 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.895812 7f9d7795c700  5 -- op tracker -- , seq: 747572, time: 2014-06-12 14:46:04.895807, event: sub_op_applied, op: osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.895825 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2859 (0'0,173'2859] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2858 active] sub_op_modify_applied on 0x62c8a80 op osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.895841 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2859 (0'0,173'2859] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2858 active] op_applied on primary on version 173'2859
2014-06-12 14:46:04.895850 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2859 (0'0,173'2859] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2858 active] op_applied on replica on version 173'2859
2014-06-12 14:46:04.895858 7f9d7c966700 10 lookup_strip_header done  cid 3.458_head oid 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:46:04.895869 7f9d7795c700  5 -- op tracker -- , seq: 747572, time: 2014-06-12 14:46:04.895865, event: done, op: osd_sub_op(client.4306.0:144106 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.895815 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (1 active)
2014-06-12 14:46:04.895894 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.895992 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:04.896788 7f9d7c966700 10 _generic_write 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 3846144~348160 = 0
2014-06-12 14:46:04.896830 7f9d7c966700 15 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:46:04.896845 7f9d7c966700 10 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 = 0
2014-06-12 14:46:04.896853 7f9d7c966700 15 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3
2014-06-12 14:46:04.896862 7f9d7c966700 10 _setattrs 3.458_head/6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 = 0
2014-06-12 14:46:04.896980 7f9d7c966700 20 set_header setting 18434 cid 3.458_head oid 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 parent seq 0
2014-06-12 14:46:04.897025 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.897040 7f9d7c966700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:04.897052 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:04.898242 7f9d7c966700 10 _do_op 0x283b9da0 seq 2792 r = 0, finisher 0x1d499b40 0
2014-06-12 14:46:04.898259 7f9d7c966700 10 _finish_op 0x283b9da0 seq 2792 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:04.898267 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:04.898276 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.898276 7f9d7715b700  5 -- op tracker -- , seq: 747573, time: 2014-06-12 14:46:04.898266, event: commit_sent, op: osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.898284 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2990 (0'0,173'2990] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2988 active] sub_op_modify_commit on op osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:04.898306 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:04.898335 7f9d7795c700  5 -- op tracker -- , seq: 747573, time: 2014-06-12 14:46:04.898329, event: sub_op_applied, op: osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.898345 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2990 (0'0,173'2990] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2989 active] sub_op_modify_applied on 0x29636c0 op osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.898365 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2990 (0'0,173'2990] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2989 active] op_applied on primary on version 173'2990
2014-06-12 14:46:04.898376 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2990 (0'0,173'2990] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2989 active] op_applied on replica on version 173'2990
2014-06-12 14:46:04.898401 7f9d7795c700  5 -- op tracker -- , seq: 747573, time: 2014-06-12 14:46:04.898396, event: done, op: osd_sub_op(client.4307.0:146571 3.458 6c58b458/rbd_data.10aa6b8b4567.0000000000000fdc/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.901872 7f9d5bec4700  5 -- op tracker -- , seq: 747574, time: 2014-06-12 14:46:04.900637, event: header_read, op: osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.901884 7f9d5bec4700  5 -- op tracker -- , seq: 747574, time: 2014-06-12 14:46:04.900638, event: throttled, op: osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.901890 7f9d5bec4700  5 -- op tracker -- , seq: 747574, time: 2014-06-12 14:46:04.901840, event: all_read, op: osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.901895 7f9d5bec4700  5 -- op tracker -- , seq: 747574, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.901899 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.901905 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:04.901911 7f9d5bec4700 15 osd.2 173 enqueue_op 0x31d61f00 prio 127 cost 516903 latency 0.001274 osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.901940 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.901949 7f9d64bda700 10 osd.2 173 dequeue_op 0x31d61f00 prio 127 cost 516903 latency 0.001311 osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2859 (0'0,173'2859] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2858 active]
2014-06-12 14:46:04.901973 7f9d64bda700  5 -- op tracker -- , seq: 747574, time: 2014-06-12 14:46:04.901969, event: reached_pg, op: osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.901983 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2859 (0'0,173'2859] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2858 active] handle_message: 0x31d61f00
2014-06-12 14:46:04.901995 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2859 (0'0,173'2859] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2858 active] sub_op_modify trans 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 v 173'2860 (transaction) 174
2014-06-12 14:46:04.902018 7f9d64bda700  5 -- op tracker -- , seq: 747574, time: 2014-06-12 14:46:04.902013, event: started, op: osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.902034 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2859 (0'0,173'2859] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2858 active] append_log log((0'0,173'2859], crt=0'0) [173'2860 (173'2859) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144107 2014-06-12 08:45:07.278596]
2014-06-12 14:46:04.902061 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2860 (0'0,173'2860] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2858 active] add_log_entry 173'2860 (173'2859) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144107 2014-06-12 08:45:07.278596
2014-06-12 14:46:04.902086 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2860 (0'0,173'2860] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2858 active] append_log  adding 1 keys
2014-06-12 14:46:04.902125 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2860, trimmed: 
2014-06-12 14:46:04.902152 7f9d64bda700  5 -- op tracker -- , seq: 747574, time: 2014-06-12 14:46:04.902147, event: started, op: osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.902165 7f9d64bda700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.902169 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.902171 7f9d64bda700  5 queue_op 0x652c960 seq 2626 osr(3.402 0x276d620) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:04.902178 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2860 (0'0,173'2860] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2858 active] do_sub_op osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.902192 7f9d64bda700 10 osd.2 173 dequeue_op 0x31d61f00 finish
2014-06-12 14:46:04.902196 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.902199 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.902243 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:46:04.902251 7f9d81970700  5 _do_op 0x652c960 seq 2626 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:46:04.902257 7f9d81970700 10 _do_transaction on 0x95487f0
2014-06-12 14:46:04.902263 7f9d81970700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.902424 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.902520 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.902573 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.902596 7f9d81970700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.902608 7f9d81970700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:46:04.902614 7f9d81970700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.902629 7f9d81970700 15 _write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 2551808~516096
2014-06-12 14:46:04.902742 7f9d81970700 10 lookup_strip_header done  cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.902884 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:04.904361 7f9d81970700 10 _generic_write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 2551808~516096 = 0
2014-06-12 14:46:04.904415 7f9d81970700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.904431 7f9d81970700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.904439 7f9d81970700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.904449 7f9d81970700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.904617 7f9d81970700 20 set_header setting 16798 cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 parent seq 0
2014-06-12 14:46:04.904675 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.904692 7f9d81970700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:46:04.904705 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:04.907196 7f9d81970700 10 _do_op 0x652c960 seq 2626 r = 0, finisher 0x80229c0 0
2014-06-12 14:46:04.907217 7f9d81970700 10 _finish_op 0x652c960 seq 2626 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.907233 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:46:04.907244 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.907236 7f9d7715b700  5 -- op tracker -- , seq: 747574, time: 2014-06-12 14:46:04.907225, event: commit_sent, op: osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.907248 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2860 (0'0,173'2860] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2858 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:04.907273 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:04.907307 7f9d7795c700  5 -- op tracker -- , seq: 747574, time: 2014-06-12 14:46:04.907302, event: sub_op_applied, op: osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.907315 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2860 (0'0,173'2860] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2859 active] sub_op_modify_applied on 0x9548700 op osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.907330 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2860 (0'0,173'2860] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2859 active] op_applied on primary on version 173'2860
2014-06-12 14:46:04.907339 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2860 (0'0,173'2860] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2859 active] op_applied on replica on version 173'2860
2014-06-12 14:46:04.907363 7f9d7795c700  5 -- op tracker -- , seq: 747574, time: 2014-06-12 14:46:04.907359, event: done, op: osd_sub_op(client.4306.0:144107 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2860 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.913576 7f9d5bec4700  5 -- op tracker -- , seq: 747575, time: 2014-06-12 14:46:04.912353, event: header_read, op: osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.913588 7f9d5bec4700  5 -- op tracker -- , seq: 747575, time: 2014-06-12 14:46:04.912355, event: throttled, op: osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.913604 7f9d5bec4700  5 -- op tracker -- , seq: 747575, time: 2014-06-12 14:46:04.913534, event: all_read, op: osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.913609 7f9d5bec4700  5 -- op tracker -- , seq: 747575, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.913623 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.913631 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:04.913640 7f9d5bec4700 15 osd.2 173 enqueue_op 0x31d60b00 prio 127 cost 516903 latency 0.001286 osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.913667 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.913675 7f9d683e1700 10 osd.2 173 dequeue_op 0x31d60b00 prio 127 cost 516903 latency 0.001321 osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2860 (0'0,173'2860] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2859 active]
2014-06-12 14:46:04.913692 7f9d683e1700  5 -- op tracker -- , seq: 747575, time: 2014-06-12 14:46:04.913690, event: reached_pg, op: osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.913700 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2860 (0'0,173'2860] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2859 active] handle_message: 0x31d60b00
2014-06-12 14:46:04.913708 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2860 (0'0,173'2860] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2859 active] sub_op_modify trans 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 v 173'2861 (transaction) 174
2014-06-12 14:46:04.913718 7f9d683e1700  5 -- op tracker -- , seq: 747575, time: 2014-06-12 14:46:04.913716, event: started, op: osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.913737 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2860 (0'0,173'2860] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2859 active] append_log log((0'0,173'2860], crt=0'0) [173'2861 (173'2860) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144108 2014-06-12 08:45:07.290227]
2014-06-12 14:46:04.913756 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2861 (0'0,173'2861] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2859 active] add_log_entry 173'2861 (173'2860) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144108 2014-06-12 08:45:07.290227
2014-06-12 14:46:04.913787 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2861 (0'0,173'2861] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2859 active] append_log  adding 1 keys
2014-06-12 14:46:04.913813 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2861, trimmed: 
2014-06-12 14:46:04.913832 7f9d683e1700  5 -- op tracker -- , seq: 747575, time: 2014-06-12 14:46:04.913829, event: started, op: osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.913841 7f9d683e1700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.913846 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.913847 7f9d683e1700  5 queue_op 0x5bf31a0 seq 2627 osr(3.402 0x276d620) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:04.913859 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:46:04.913865 7f9d7a161700  5 _do_op 0x5bf31a0 seq 2627 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:46:04.913856 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2861 (0'0,173'2861] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2859 active] do_sub_op osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.913869 7f9d7a161700 10 _do_transaction on 0x2e9b27f0
2014-06-12 14:46:04.913870 7f9d683e1700 10 osd.2 173 dequeue_op 0x31d60b00 finish
2014-06-12 14:46:04.913873 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.913875 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.913875 7f9d7a161700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.928316 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.928453 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.928504 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.928522 7f9d7a161700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.928526 7f9d7a161700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:46:04.928532 7f9d7a161700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.928543 7f9d7a161700 15 _write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 3067904~516096
2014-06-12 14:46:04.928649 7f9d7a161700 10 lookup_strip_header done  cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.928737 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:04.930062 7f9d7a161700 10 _generic_write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 3067904~516096 = 0
2014-06-12 14:46:04.930100 7f9d7a161700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.930111 7f9d7a161700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.930117 7f9d7a161700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.930122 7f9d7a161700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.930274 7f9d7a161700 20 set_header setting 16798 cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 parent seq 0
2014-06-12 14:46:04.930324 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.930332 7f9d7a161700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:46:04.930338 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:04.932375 7f9d7a161700 10 _do_op 0x5bf31a0 seq 2627 r = 0, finisher 0x69ba000 0
2014-06-12 14:46:04.932396 7f9d7a161700 10 _finish_op 0x5bf31a0 seq 2627 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.932417 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:46:04.932427 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.932420 7f9d7715b700  5 -- op tracker -- , seq: 747575, time: 2014-06-12 14:46:04.932408, event: commit_sent, op: osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.932430 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2861 (0'0,173'2861] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2859 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:04.932459 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:04.932497 7f9d7795c700  5 -- op tracker -- , seq: 747575, time: 2014-06-12 14:46:04.932491, event: sub_op_applied, op: osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.932507 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2861 (0'0,173'2861] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2860 active] sub_op_modify_applied on 0x2e9b2700 op osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.932523 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2861 (0'0,173'2861] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2860 active] op_applied on primary on version 173'2861
2014-06-12 14:46:04.932530 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2861 (0'0,173'2861] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2860 active] op_applied on replica on version 173'2861
2014-06-12 14:46:04.932544 7f9d7795c700  5 -- op tracker -- , seq: 747575, time: 2014-06-12 14:46:04.932541, event: done, op: osd_sub_op(client.4306.0:144108 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2861 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.938736 7f9d5bec4700  5 -- op tracker -- , seq: 747576, time: 2014-06-12 14:46:04.937466, event: header_read, op: osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.938754 7f9d5bec4700  5 -- op tracker -- , seq: 747576, time: 2014-06-12 14:46:04.937468, event: throttled, op: osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.938779 7f9d5bec4700  5 -- op tracker -- , seq: 747576, time: 2014-06-12 14:46:04.938684, event: all_read, op: osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.938789 7f9d5bec4700  5 -- op tracker -- , seq: 747576, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.938795 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.938803 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:04.938812 7f9d5bec4700 15 osd.2 173 enqueue_op 0x31d61d00 prio 127 cost 516903 latency 0.001346 osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.938847 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.938859 7f9d65bdc700 10 osd.2 173 dequeue_op 0x31d61d00 prio 127 cost 516903 latency 0.001393 osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2861 (0'0,173'2861] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2860 active]
2014-06-12 14:46:04.938885 7f9d65bdc700  5 -- op tracker -- , seq: 747576, time: 2014-06-12 14:46:04.938880, event: reached_pg, op: osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.938896 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2861 (0'0,173'2861] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2860 active] handle_message: 0x31d61d00
2014-06-12 14:46:04.938908 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2861 (0'0,173'2861] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2860 active] sub_op_modify trans 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 v 173'2862 (transaction) 174
2014-06-12 14:46:04.938925 7f9d65bdc700  5 -- op tracker -- , seq: 747576, time: 2014-06-12 14:46:04.938921, event: started, op: osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.938941 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2861 (0'0,173'2861] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2860 active] append_log log((0'0,173'2861], crt=0'0) [173'2862 (173'2861) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144109 2014-06-12 08:45:07.315332]
2014-06-12 14:46:04.938965 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2862 (0'0,173'2862] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2860 active] add_log_entry 173'2862 (173'2861) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144109 2014-06-12 08:45:07.315332
2014-06-12 14:46:04.938988 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2862 (0'0,173'2862] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2860 active] append_log  adding 1 keys
2014-06-12 14:46:04.939024 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2862, trimmed: 
2014-06-12 14:46:04.939049 7f9d65bdc700  5 -- op tracker -- , seq: 747576, time: 2014-06-12 14:46:04.939045, event: started, op: osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.939061 7f9d65bdc700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.939065 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.939067 7f9d65bdc700  5 queue_op 0x2b4a8900 seq 2628 osr(3.402 0x276d620) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:04.939078 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:46:04.939085 7f9d7a962700  5 _do_op 0x2b4a8900 seq 2628 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:46:04.939074 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2862 (0'0,173'2862] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2860 active] do_sub_op osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.939088 7f9d65bdc700 10 osd.2 173 dequeue_op 0x31d61d00 finish
2014-06-12 14:46:04.939090 7f9d7a962700 10 _do_transaction on 0x29eb3270
2014-06-12 14:46:04.939092 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.939095 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.939097 7f9d7a962700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.939237 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.939267 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.939323 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.939416 7f9d7a962700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.939426 7f9d7a962700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:46:04.939434 7f9d7a962700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.939454 7f9d7a962700 15 _write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 3584000~516096
2014-06-12 14:46:04.939608 7f9d7a962700 10 lookup_strip_header done  cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.939749 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:04.941504 7f9d7a962700 10 _generic_write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 3584000~516096 = 0
2014-06-12 14:46:04.941553 7f9d7a962700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.941568 7f9d7a962700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.941576 7f9d7a962700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.941585 7f9d7a962700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.941802 7f9d7a962700 20 set_header setting 16798 cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 parent seq 0
2014-06-12 14:46:04.941868 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.941884 7f9d7a962700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:46:04.941896 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:04.944660 7f9d7a962700 10 _do_op 0x2b4a8900 seq 2628 r = 0, finisher 0x567ef80 0
2014-06-12 14:46:04.944679 7f9d7a962700 10 _finish_op 0x2b4a8900 seq 2628 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.944687 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:46:04.944697 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.944696 7f9d7715b700  5 -- op tracker -- , seq: 747576, time: 2014-06-12 14:46:04.944687, event: commit_sent, op: osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.944703 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2862 (0'0,173'2862] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2860 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:04.944723 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:04.944752 7f9d7795c700  5 -- op tracker -- , seq: 747576, time: 2014-06-12 14:46:04.944746, event: sub_op_applied, op: osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.944761 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2862 (0'0,173'2862] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active] sub_op_modify_applied on 0x29eb3180 op osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.944779 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2862 (0'0,173'2862] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active] op_applied on primary on version 173'2862
2014-06-12 14:46:04.944788 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2862 (0'0,173'2862] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active] op_applied on replica on version 173'2862
2014-06-12 14:46:04.944801 7f9d7795c700  5 -- op tracker -- , seq: 747576, time: 2014-06-12 14:46:04.944798, event: done, op: osd_sub_op(client.4306.0:144109 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2862 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.948151 7f9d5bec4700  5 -- op tracker -- , seq: 747577, time: 2014-06-12 14:46:04.947909, event: header_read, op: osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.948160 7f9d5bec4700  5 -- op tracker -- , seq: 747577, time: 2014-06-12 14:46:04.947910, event: throttled, op: osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.948165 7f9d5bec4700  5 -- op tracker -- , seq: 747577, time: 2014-06-12 14:46:04.948119, event: all_read, op: osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.948170 7f9d5bec4700  5 -- op tracker -- , seq: 747577, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.948173 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:04.948177 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:04.948181 7f9d5bec4700 15 osd.2 173 enqueue_op 0x31d60500 prio 127 cost 95015 latency 0.000272 osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.948201 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:04.948207 7f9d6abe6700 10 osd.2 173 dequeue_op 0x31d60500 prio 127 cost 95015 latency 0.000298 osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2862 (0'0,173'2862] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active]
2014-06-12 14:46:04.948229 7f9d6abe6700  5 -- op tracker -- , seq: 747577, time: 2014-06-12 14:46:04.948225, event: reached_pg, op: osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.948238 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2862 (0'0,173'2862] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active] handle_message: 0x31d60500
2014-06-12 14:46:04.948250 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2862 (0'0,173'2862] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active] sub_op_modify trans 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 v 173'2863 (transaction) 174
2014-06-12 14:46:04.948265 7f9d6abe6700  5 -- op tracker -- , seq: 747577, time: 2014-06-12 14:46:04.948262, event: started, op: osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.948284 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2862 (0'0,173'2862] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active] append_log log((0'0,173'2862], crt=0'0) [173'2863 (173'2862) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144110 2014-06-12 08:45:07.327527]
2014-06-12 14:46:04.948317 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2863 (0'0,173'2863] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active] add_log_entry 173'2863 (173'2862) modify   30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 by client.4306.0:144110 2014-06-12 08:45:07.327527
2014-06-12 14:46:04.948339 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2863 (0'0,173'2863] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active] append_log  adding 1 keys
2014-06-12 14:46:04.948373 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2863, trimmed: 
2014-06-12 14:46:04.948397 7f9d6abe6700  5 -- op tracker -- , seq: 747577, time: 2014-06-12 14:46:04.948392, event: started, op: osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.948406 7f9d6abe6700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.948410 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:04.948411 7f9d6abe6700  5 queue_op 0x69c94a0 seq 2629 osr(3.402 0x276d620) 96501 bytes   (queue has 1 ops and 96501 bytes)
2014-06-12 14:46:04.948423 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:46:04.948418 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2863 (0'0,173'2863] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active] do_sub_op osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.948431 7f9d7f16b700  5 _do_op 0x69c94a0 seq 2629 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:46:04.948433 7f9d6abe6700 10 osd.2 173 dequeue_op 0x31d60500 finish
2014-06-12 14:46:04.948436 7f9d7f16b700 10 _do_transaction on 0x58807f0
2014-06-12 14:46:04.948437 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:04.948440 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:04.948441 7f9d7f16b700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.948569 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.948598 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:04.948654 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:04.948746 7f9d7f16b700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.948755 7f9d7f16b700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:46:04.948762 7f9d7f16b700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:46:04.948787 7f9d7f16b700 15 _write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 4100096~94208
2014-06-12 14:46:04.948947 7f9d7f16b700 10 lookup_strip_header done  cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.949078 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:04.949434 7f9d7f16b700 10 _generic_write 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 4100096~94208 = 0
2014-06-12 14:46:04.949452 7f9d7f16b700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.949468 7f9d7f16b700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.949476 7f9d7f16b700 15 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3
2014-06-12 14:46:04.949485 7f9d7f16b700 10 _setattrs 3.402_head/30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 = 0
2014-06-12 14:46:04.949636 7f9d7f16b700 20 set_header setting 16798 cid 3.402_head oid 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 parent seq 0
2014-06-12 14:46:04.949669 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:04.949685 7f9d7f16b700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:46:04.949696 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:04.950293 7f9d7f16b700 10 _do_op 0x69c94a0 seq 2629 r = 0, finisher 0x9071a20 0
2014-06-12 14:46:04.950308 7f9d7f16b700 10 _finish_op 0x69c94a0 seq 2629 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:46:04.950316 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:46:04.950320 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:04.950335 7f9d7795c700  5 -- op tracker -- , seq: 747577, time: 2014-06-12 14:46:04.950324, event: sub_op_applied, op: osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.950344 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2863 (0'0,173'2863] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active] sub_op_modify_applied on 0x5880700 op osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:04.950373 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:04.950396 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2863 (0'0,173'2863] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active] op_applied on primary on version 173'2863
2014-06-12 14:46:04.950408 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2863 (0'0,173'2863] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active] op_applied on replica on version 173'2863
2014-06-12 14:46:04.950457 7f9d7715b700  5 -- op tracker -- , seq: 747577, time: 2014-06-12 14:46:04.950450, event: commit_sent, op: osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:04.950465 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2863 (0'0,173'2863] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2861 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:04.950487 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:04.950541 7f9d7715b700  5 -- op tracker -- , seq: 747577, time: 2014-06-12 14:46:04.950536, event: done, op: osd_sub_op(client.4306.0:144110 3.402 30f43402/rbd_data.10a96b8b4567.0000000000000f9a/head//3 [] v 173'2863 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:05.044068 7f9d56432700  5 -- op tracker -- , seq: 747578, time: 2014-06-12 14:46:05.042984, event: header_read, op: osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:05.044084 7f9d56432700  5 -- op tracker -- , seq: 747578, time: 2014-06-12 14:46:05.042985, event: throttled, op: osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:05.044100 7f9d56432700  5 -- op tracker -- , seq: 747578, time: 2014-06-12 14:46:05.044019, event: all_read, op: osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:05.044108 7f9d56432700  5 -- op tracker -- , seq: 747578, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:05.044115 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:05.044122 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:05.044131 7f9d56432700 15 osd.2 173 enqueue_op 0x6f4bf00 prio 127 cost 516899 latency 0.001147 osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:05.044166 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:05.044176 7f9d66bde700 10 osd.2 173 dequeue_op 0x6f4bf00 prio 127 cost 516899 latency 0.001191 osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2474 (0'0,173'2474] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2473 active]
2014-06-12 14:46:05.044202 7f9d66bde700  5 -- op tracker -- , seq: 747578, time: 2014-06-12 14:46:05.044198, event: reached_pg, op: osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:05.044212 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2474 (0'0,173'2474] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2473 active] handle_message: 0x6f4bf00
2014-06-12 14:46:05.044225 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2474 (0'0,173'2474] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2473 active] sub_op_modify trans 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 v 173'2475 (transaction) 174
2014-06-12 14:46:05.044242 7f9d66bde700  5 -- op tracker -- , seq: 747578, time: 2014-06-12 14:46:05.044238, event: started, op: osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:05.044258 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2474 (0'0,173'2474] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2473 active] append_log log((0'0,173'2474], crt=0'0) [173'2475 (173'2473) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144990 2014-06-12 08:44:57.475586]
2014-06-12 14:46:05.044283 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2475 (0'0,173'2475] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2473 active] add_log_entry 173'2475 (173'2473) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144990 2014-06-12 08:44:57.475586
2014-06-12 14:46:05.044305 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2475 (0'0,173'2475] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2473 active] append_log  adding 1 keys
2014-06-12 14:46:05.044341 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2475, trimmed: 
2014-06-12 14:46:05.044366 7f9d66bde700  5 -- op tracker -- , seq: 747578, time: 2014-06-12 14:46:05.044361, event: started, op: osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:05.044380 7f9d66bde700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:05.044385 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:05.044387 7f9d66bde700  5 queue_op 0x93479e0 seq 2288 osr(3.2b 0x276cd20) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:46:05.044400 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:46:05.044395 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2475 (0'0,173'2475] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2473 active] do_sub_op osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:05.044409 7f9d66bde700 10 osd.2 173 dequeue_op 0x6f4bf00 finish
2014-06-12 14:46:05.044409 7f9d7895e700  5 _do_op 0x93479e0 seq 2288 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:46:05.044413 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:05.044414 7f9d7895e700 10 _do_transaction on 0x2c4e02b0
2014-06-12 14:46:05.044415 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:05.044420 7f9d7895e700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:05.044573 7f9d7895e700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:05.044603 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:05.044662 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:05.044689 7f9d7895e700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:05.044697 7f9d7895e700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:46:05.044703 7f9d7895e700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:05.044902 7f9d7895e700 15 _write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 1830912~516096
2014-06-12 14:46:05.045072 7f9d7895e700 10 lookup_strip_header done  cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:05.045224 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:05.047333 7f9d7895e700 10 _generic_write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 1830912~516096 = 0
2014-06-12 14:46:05.047426 7f9d7895e700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:05.047458 7f9d7895e700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:05.047467 7f9d7895e700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:05.047478 7f9d7895e700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:05.047708 7f9d7895e700 20 set_header setting 17866 cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 parent seq 0
2014-06-12 14:46:05.047830 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:05.047849 7f9d7895e700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:46:05.047860 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:05.049654 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:05.051165 7f9d7895e700 10 _do_op 0x93479e0 seq 2288 r = 0, finisher 0x73f91c0 0
2014-06-12 14:46:05.051211 7f9d7715b700  5 -- op tracker -- , seq: 747578, time: 2014-06-12 14:46:05.051200, event: commit_sent, op: osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:05.051235 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2475 (0'0,173'2475] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2473 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:05.051264 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:05.051314 7f9d7895e700 10 _finish_op 0x93479e0 seq 2288 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:05.051341 7f9d7795c700  5 -- op tracker -- , seq: 747578, time: 2014-06-12 14:46:05.051335, event: sub_op_applied, op: osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:05.051348 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2475 (0'0,173'2475] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active] sub_op_modify_applied on 0x2c4e01c0 op osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:05.051364 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2475 (0'0,173'2475] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active] op_applied on primary on version 173'2475
2014-06-12 14:46:05.051374 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2475 (0'0,173'2475] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active] op_applied on replica on version 173'2475
2014-06-12 14:46:05.051478 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:05.051405 7f9d7895e700  5 -- op tracker -- , seq: 747578, time: 2014-06-12 14:46:05.051396, event: done, op: osd_sub_op(client.4314.0:144990 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:05.051579 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:46:05.051583 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:05.213039 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:05.213088 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:05.439095 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:05.439153 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:05.469722 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:05.469744 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:05.469747 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:05.469749 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:05.469751 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 32; oldest is 35.189849 seconds old
2014-06-12 14:46:05.469776 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:05.497489 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:05.497527 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:05.736749 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:05.736859 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:05.817889 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:05.817892 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:05.817894 7f9d74bfa700 20 osd.2 173 _dispatch 0x17705380 ping v1
2014-06-12 14:46:05.817896 7f9d74bfa700 10 osd.2 173 ping from client.4303
2014-06-12 14:46:05.817898 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:05.817899 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:05.887501 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:05.887559 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:06.153060 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:06.156597 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.166953 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.171106 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.261056 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:06.261142 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:06.284412 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:06.284418 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:06.284419 7f9d74bfa700 20 osd.2 173 _dispatch 0x1e7011e0 ping v1
2014-06-12 14:46:06.284422 7f9d74bfa700 10 osd.2 173 ping from client.4306
2014-06-12 14:46:06.284424 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:06.284425 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:06.320692 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:06.320695 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:06.320696 7f9d74bfa700 20 osd.2 173 _dispatch 0x1fd71860 ping v1
2014-06-12 14:46:06.320698 7f9d74bfa700 10 osd.2 173 ping from client.4308
2014-06-12 14:46:06.320699 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:06.320700 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:06.366714 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:06.439427 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.450084 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.461858 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.466018 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.468045 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.469814 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:06.469838 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:06.469842 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:06.469844 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:06.469846 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 32; oldest is 36.189945 seconds old
2014-06-12 14:46:06.469872 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:06.477061 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:06.477063 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:06.477064 7f9d74bfa700 20 osd.2 173 _dispatch 0x6b79d40 ping v1
2014-06-12 14:46:06.477066 7f9d74bfa700 10 osd.2 173 ping from client.4309
2014-06-12 14:46:06.477068 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:06.477069 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:06.482938 7f9d5b6bc700  5 -- op tracker -- , seq: 747579, time: 2014-06-12 14:46:06.479432, event: header_read, op: osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.482950 7f9d5b6bc700  5 -- op tracker -- , seq: 747579, time: 2014-06-12 14:46:06.479438, event: throttled, op: osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.482960 7f9d5b6bc700  5 -- op tracker -- , seq: 747579, time: 2014-06-12 14:46:06.482882, event: all_read, op: osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.482988 7f9d5b6bc700  5 -- op tracker -- , seq: 747579, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.482995 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:06.483003 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:06.483014 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39576400 prio 127 cost 1393447 latency 0.003581 osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.483044 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:06.483052 7f9d69be4700 10 osd.2 173 dequeue_op 0x39576400 prio 127 cost 1393447 latency 0.003619 osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.426( v 173'2675 (0'0,173'2675] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active]
2014-06-12 14:46:06.483075 7f9d69be4700  5 -- op tracker -- , seq: 747579, time: 2014-06-12 14:46:06.483072, event: reached_pg, op: osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483082 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2675 (0'0,173'2675] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] handle_message: 0x39576400
2014-06-12 14:46:06.483096 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2675 (0'0,173'2675] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] sub_op_modify trans b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 v 173'2676 (transaction) 174
2014-06-12 14:46:06.483102 7f9d5b6bc700  5 -- op tracker -- , seq: 747580, time: 2014-06-12 14:46:06.483044, event: header_read, op: osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483115 7f9d5b6bc700  5 -- op tracker -- , seq: 747580, time: 2014-06-12 14:46:06.483045, event: throttled, op: osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483177 7f9d5b6bc700  5 -- op tracker -- , seq: 747580, time: 2014-06-12 14:46:06.483078, event: all_read, op: osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483188 7f9d5b6bc700  5 -- op tracker -- , seq: 747580, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483194 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:06.483201 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:06.483201 7f9d69be4700  5 -- op tracker -- , seq: 747579, time: 2014-06-12 14:46:06.483104, event: started, op: osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483213 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39576e00 prio 127 cost 4903 latency 0.000169 osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.483246 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:06.483222 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2675 (0'0,173'2675] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] append_log log((0'0,173'2675], crt=0'0) [173'2676 (173'694) modify   b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 by client.4315.0:144865 2014-06-12 08:45:08.818401]
2014-06-12 14:46:06.483264 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2676 (0'0,173'2676] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] add_log_entry 173'2676 (173'694) modify   b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 by client.4315.0:144865 2014-06-12 08:45:08.818401
2014-06-12 14:46:06.483293 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2676 (0'0,173'2676] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] append_log  adding 1 keys
2014-06-12 14:46:06.483328 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2676, trimmed: 
2014-06-12 14:46:06.483353 7f9d69be4700  5 -- op tracker -- , seq: 747579, time: 2014-06-12 14:46:06.483348, event: started, op: osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483365 7f9d69be4700  5 queue_transactions existing osr(3.426 0x276d9c0)/0x276d9c0
2014-06-12 14:46:06.483369 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:06.483371 7f9d69be4700  5 queue_op 0x6c16e40 seq 2479 osr(3.426 0x276d9c0) 1394933 bytes   (queue has 1 ops and 1394933 bytes)
2014-06-12 14:46:06.483379 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.426( v 173'2676 (0'0,173'2676] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] do_sub_op osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.483394 7f9d69be4700 10 osd.2 173 dequeue_op 0x39576400 finish
2014-06-12 14:46:06.483400 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:06.483403 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.483410 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9e60 (1 active)
2014-06-12 14:46:06.483415 7f9d79960700  5 _do_op 0x6c16e40 seq 2479 osr(3.426 0x276d9c0)/0x276d9c0 start
2014-06-12 14:46:06.483419 7f9d79960700 10 _do_transaction on 0x50e60f0
2014-06-12 14:46:06.483425 7f9d79960700 15 _omap_setkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.483551 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.483571 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:06.483626 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:06.483712 7f9d5b6bc700  5 -- op tracker -- , seq: 747581, time: 2014-06-12 14:46:06.483232, event: header_read, op: osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483723 7f9d5b6bc700  5 -- op tracker -- , seq: 747581, time: 2014-06-12 14:46:06.483232, event: throttled, op: osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483731 7f9d5b6bc700  5 -- op tracker -- , seq: 747581, time: 2014-06-12 14:46:06.483692, event: all_read, op: osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483740 7f9d5b6bc700  5 -- op tracker -- , seq: 747581, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483744 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:06.483747 7f9d79960700 15 _omap_rmkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.483750 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:06.483754 7f9d79960700 10 _omap_rmkeys meta/1fe1f53a/pglog_3.426/0//-1 r = 0
2014-06-12 14:46:06.483760 7f9d79960700 15 _omap_setkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.483756 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39576900 prio 127 cost 516903 latency 0.000524 osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.483778 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:06.483779 7f9d79960700 15 _write 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 0~1392640
2014-06-12 14:46:06.483784 7f9d68be2700 10 osd.2 173 dequeue_op 0x39576e00 prio 127 cost 4903 latency 0.000739 osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.426( v 173'2676 (0'0,173'2676] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active]
2014-06-12 14:46:06.483803 7f9d68be2700  5 -- op tracker -- , seq: 747580, time: 2014-06-12 14:46:06.483799, event: reached_pg, op: osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483811 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2676 (0'0,173'2676] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] handle_message: 0x39576e00
2014-06-12 14:46:06.483822 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2676 (0'0,173'2676] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] sub_op_modify trans b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 v 173'2677 (transaction) 174
2014-06-12 14:46:06.483847 7f9d68be2700  5 -- op tracker -- , seq: 747580, time: 2014-06-12 14:46:06.483843, event: started, op: osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483860 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2676 (0'0,173'2676] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] append_log log((0'0,173'2676], crt=0'0) [173'2677 (173'2676) modify   b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 by client.4315.0:144866 2014-06-12 08:45:08.830358]
2014-06-12 14:46:06.483881 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2677 (0'0,173'2677] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] add_log_entry 173'2677 (173'2676) modify   b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 by client.4315.0:144866 2014-06-12 08:45:08.830358
2014-06-12 14:46:06.483901 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2677 (0'0,173'2677] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] append_log  adding 1 keys
2014-06-12 14:46:06.483929 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2677, trimmed: 
2014-06-12 14:46:06.483952 7f9d68be2700  5 -- op tracker -- , seq: 747580, time: 2014-06-12 14:46:06.483947, event: started, op: osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.483962 7f9d68be2700  5 queue_transactions existing osr(3.426 0x276d9c0)/0x276d9c0
2014-06-12 14:46:06.483965 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:06.483966 7f9d68be2700  5 queue_op 0x2c87980 seq 2480 osr(3.426 0x276d9c0) 6389 bytes   (queue has 2 ops and 1401322 bytes)
2014-06-12 14:46:06.483972 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.426( v 173'2677 (0'0,173'2677] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] do_sub_op osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.483986 7f9d68be2700 10 osd.2 173 dequeue_op 0x39576e00 finish
2014-06-12 14:46:06.483989 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:06.483992 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.483999 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9e60 (2 active)
2014-06-12 14:46:06.484052 7f9d79960700 10 lookup_strip_header done  cid 3.426_head oid b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.484159 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:06.485632 7f9d633d7700 10 osd.2 173 dequeue_op 0x39576900 prio 127 cost 516903 latency 0.002400 osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.426( v 173'2677 (0'0,173'2677] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active]
2014-06-12 14:46:06.485660 7f9d633d7700  5 -- op tracker -- , seq: 747581, time: 2014-06-12 14:46:06.485655, event: reached_pg, op: osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.485670 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2677 (0'0,173'2677] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] handle_message: 0x39576900
2014-06-12 14:46:06.485684 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2677 (0'0,173'2677] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] sub_op_modify trans b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 v 173'2678 (transaction) 174
2014-06-12 14:46:06.485703 7f9d633d7700  5 -- op tracker -- , seq: 747581, time: 2014-06-12 14:46:06.485698, event: started, op: osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.485718 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2677 (0'0,173'2677] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] append_log log((0'0,173'2677], crt=0'0) [173'2678 (173'2677) modify   b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 by client.4315.0:144867 2014-06-12 08:45:08.838008]
2014-06-12 14:46:06.485747 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] add_log_entry 173'2678 (173'2677) modify   b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 by client.4315.0:144867 2014-06-12 08:45:08.838008
2014-06-12 14:46:06.485772 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] append_log  adding 1 keys
2014-06-12 14:46:06.485805 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2678, trimmed: 
2014-06-12 14:46:06.485835 7f9d633d7700  5 -- op tracker -- , seq: 747581, time: 2014-06-12 14:46:06.485830, event: started, op: osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.485846 7f9d633d7700  5 queue_transactions existing osr(3.426 0x276d9c0)/0x276d9c0
2014-06-12 14:46:06.485850 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:06.485851 7f9d633d7700  5 queue_op 0x544dec0 seq 2481 osr(3.426 0x276d9c0) 518389 bytes   (queue has 3 ops and 1919711 bytes)
2014-06-12 14:46:06.485860 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9e60 (3 active)
2014-06-12 14:46:06.485857 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] do_sub_op osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.485876 7f9d633d7700 10 osd.2 173 dequeue_op 0x39576900 finish
2014-06-12 14:46:06.485881 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:06.485885 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.489888 7f9d79960700 10 _generic_write 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 0~1392640 = 0
2014-06-12 14:46:06.490006 7f9d79960700 15 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.490025 7f9d79960700 10 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 = 0
2014-06-12 14:46:06.490033 7f9d79960700 15 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.490044 7f9d79960700 10 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 = 0
2014-06-12 14:46:06.490391 7f9d79960700 20 set_header setting 17254 cid 3.426_head oid b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 parent seq 0
2014-06-12 14:46:06.490523 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:06.490540 7f9d79960700 20 set_header setting 2290 cid meta oid 1fe1f53a/pglog_3.426/0//-1 parent seq 0
2014-06-12 14:46:06.490554 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:06.498070 7f9d79960700 10 _do_op 0x6c16e40 seq 2479 r = 0, finisher 0x52dfd40 0
2014-06-12 14:46:06.498097 7f9d79960700 10 _finish_op 0x6c16e40 seq 2479 osr(3.426 0x276d9c0)/0x276d9c0
2014-06-12 14:46:06.498107 7f9d8116f700  5 _do_op 0x2c87980 seq 2480 osr(3.426 0x276d9c0)/0x276d9c0 start
2014-06-12 14:46:06.498113 7f9d8116f700 10 _do_transaction on 0x61d87f0
2014-06-12 14:46:06.498108 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9e60 (2 active)
2014-06-12 14:46:06.498118 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.498123 7f9d8116f700 15 _omap_setkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.498122 7f9d7715b700  5 -- op tracker -- , seq: 747579, time: 2014-06-12 14:46:06.498109, event: commit_sent, op: osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.498133 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2674 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:06.498164 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:06.498206 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.498204 7f9d7795c700  5 -- op tracker -- , seq: 747579, time: 2014-06-12 14:46:06.498198, event: sub_op_applied, op: osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.498211 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2675 active] sub_op_modify_applied on 0x50e6000 op osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.498226 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:06.498227 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2675 active] op_applied on primary on version 173'2676
2014-06-12 14:46:06.498237 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2675 active] op_applied on replica on version 173'2676
2014-06-12 14:46:06.498252 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:06.498261 7f9d7795c700  5 -- op tracker -- , seq: 747579, time: 2014-06-12 14:46:06.498257, event: done, op: osd_sub_op(client.4315.0:144865 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.498386 7f9d8116f700 15 _omap_rmkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.498393 7f9d8116f700 10 _omap_rmkeys meta/1fe1f53a/pglog_3.426/0//-1 r = 0
2014-06-12 14:46:06.498398 7f9d8116f700 15 _omap_setkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.498412 7f9d8116f700 15 _write 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 1392640~4096
2014-06-12 14:46:06.498557 7f9d8116f700 10 lookup_strip_header done  cid 3.426_head oid b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.498569 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:06.498588 7f9d8116f700 10 _generic_write 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 1392640~4096 = 0
2014-06-12 14:46:06.498599 7f9d8116f700 15 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.498614 7f9d8116f700 10 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 = 0
2014-06-12 14:46:06.498624 7f9d8116f700 15 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.498633 7f9d8116f700 10 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 = 0
2014-06-12 14:46:06.498745 7f9d8116f700 20 set_header setting 17254 cid 3.426_head oid b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 parent seq 0
2014-06-12 14:46:06.498766 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:06.498781 7f9d8116f700 20 set_header setting 2290 cid meta oid 1fe1f53a/pglog_3.426/0//-1 parent seq 0
2014-06-12 14:46:06.498794 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:06.500381 7f9d8116f700 10 _do_op 0x2c87980 seq 2480 r = 0, finisher 0x7811320 0
2014-06-12 14:46:06.500391 7f9d8116f700 10 _finish_op 0x2c87980 seq 2480 osr(3.426 0x276d9c0)/0x276d9c0
2014-06-12 14:46:06.500399 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9e60 (1 active)
2014-06-12 14:46:06.500399 7f9d8096e700  5 _do_op 0x544dec0 seq 2481 osr(3.426 0x276d9c0)/0x276d9c0 start
2014-06-12 14:46:06.500402 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.500403 7f9d8096e700 10 _do_transaction on 0x2bc86630
2014-06-12 14:46:06.500409 7f9d8096e700 15 _omap_setkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.500406 7f9d7715b700  5 -- op tracker -- , seq: 747580, time: 2014-06-12 14:46:06.500398, event: commit_sent, op: osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.500415 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2675 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:06.500437 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:06.500463 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.500478 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:06.500487 7f9d7795c700  5 -- op tracker -- , seq: 747580, time: 2014-06-12 14:46:06.500479, event: sub_op_applied, op: osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.500503 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:06.500495 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2676 active] sub_op_modify_applied on 0x61d8700 op osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.500509 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2676 active] op_applied on primary on version 173'2677
2014-06-12 14:46:06.500519 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2676 active] op_applied on replica on version 173'2677
2014-06-12 14:46:06.500540 7f9d7795c700  5 -- op tracker -- , seq: 747580, time: 2014-06-12 14:46:06.500536, event: done, op: osd_sub_op(client.4315.0:144866 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.500689 7f9d8096e700 15 _omap_rmkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.500696 7f9d8096e700 10 _omap_rmkeys meta/1fe1f53a/pglog_3.426/0//-1 r = 0
2014-06-12 14:46:06.500701 7f9d8096e700 15 _omap_setkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.500714 7f9d8096e700 15 _write 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 1396736~516096
2014-06-12 14:46:06.500800 7f9d8096e700 10 lookup_strip_header done  cid 3.426_head oid b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.500937 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:06.502371 7f9d8096e700 10 _generic_write 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 1396736~516096 = 0
2014-06-12 14:46:06.502407 7f9d8096e700 15 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.502424 7f9d8096e700 10 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 = 0
2014-06-12 14:46:06.502429 7f9d8096e700 15 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.502437 7f9d8096e700 10 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 = 0
2014-06-12 14:46:06.502636 7f9d8096e700 20 set_header setting 17254 cid 3.426_head oid b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 parent seq 0
2014-06-12 14:46:06.502698 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:06.502712 7f9d8096e700 20 set_header setting 2290 cid meta oid 1fe1f53a/pglog_3.426/0//-1 parent seq 0
2014-06-12 14:46:06.502722 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:06.505032 7f9d8096e700 10 _do_op 0x544dec0 seq 2481 r = 0, finisher 0x80c23a0 0
2014-06-12 14:46:06.505051 7f9d8096e700 10 _finish_op 0x544dec0 seq 2481 osr(3.426 0x276d9c0)/0x276d9c0
2014-06-12 14:46:06.505061 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9e60 (0 active)
2014-06-12 14:46:06.505072 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.505074 7f9d7715b700  5 -- op tracker -- , seq: 747581, time: 2014-06-12 14:46:06.505063, event: commit_sent, op: osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.505084 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2676 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:06.505115 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:06.505195 7f9d7795c700  5 -- op tracker -- , seq: 747581, time: 2014-06-12 14:46:06.505189, event: sub_op_applied, op: osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.505205 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] sub_op_modify_applied on 0x2bc86540 op osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.505225 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] op_applied on primary on version 173'2678
2014-06-12 14:46:06.505235 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] op_applied on replica on version 173'2678
2014-06-12 14:46:06.505257 7f9d7795c700  5 -- op tracker -- , seq: 747581, time: 2014-06-12 14:46:06.505252, event: done, op: osd_sub_op(client.4315.0:144867 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.512534 7f9d5b6bc700  5 -- op tracker -- , seq: 747582, time: 2014-06-12 14:46:06.509966, event: header_read, op: osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.512550 7f9d5b6bc700  5 -- op tracker -- , seq: 747582, time: 2014-06-12 14:46:06.509968, event: throttled, op: osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.512559 7f9d5b6bc700  5 -- op tracker -- , seq: 747582, time: 2014-06-12 14:46:06.512493, event: all_read, op: osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.512567 7f9d5b6bc700  5 -- op tracker -- , seq: 747582, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.512572 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:06.512579 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:06.512587 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39576200 prio 127 cost 1549095 latency 0.002621 osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.512644 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:06.512656 7f9d623d5700 10 osd.2 173 dequeue_op 0x39576200 prio 127 cost 1549095 latency 0.002690 osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active]
2014-06-12 14:46:06.512684 7f9d623d5700  5 -- op tracker -- , seq: 747582, time: 2014-06-12 14:46:06.512679, event: reached_pg, op: osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.512693 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] handle_message: 0x39576200
2014-06-12 14:46:06.512707 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] sub_op_modify trans b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 v 173'2679 (transaction) 174
2014-06-12 14:46:06.512724 7f9d623d5700  5 -- op tracker -- , seq: 747582, time: 2014-06-12 14:46:06.512720, event: started, op: osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.512741 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2678 (0'0,173'2678] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] append_log log((0'0,173'2678], crt=0'0) [173'2679 (173'2678) modify   b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 by client.4315.0:144868 2014-06-12 08:45:08.882264]
2014-06-12 14:46:06.512768 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2679 (0'0,173'2679] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] add_log_entry 173'2679 (173'2678) modify   b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 by client.4315.0:144868 2014-06-12 08:45:08.882264
2014-06-12 14:46:06.512791 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2679 (0'0,173'2679] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] append_log  adding 1 keys
2014-06-12 14:46:06.512828 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2679, trimmed: 
2014-06-12 14:46:06.512854 7f9d623d5700  5 -- op tracker -- , seq: 747582, time: 2014-06-12 14:46:06.512849, event: started, op: osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.512871 7f9d623d5700  5 queue_transactions existing osr(3.426 0x276d9c0)/0x276d9c0
2014-06-12 14:46:06.512876 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:06.512877 7f9d623d5700  5 queue_op 0x216ef60 seq 2482 osr(3.426 0x276d9c0) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:46:06.512888 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9e60 (1 active)
2014-06-12 14:46:06.512896 7f9d8016d700  5 _do_op 0x216ef60 seq 2482 osr(3.426 0x276d9c0)/0x276d9c0 start
2014-06-12 14:46:06.512885 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.426( v 173'2679 (0'0,173'2679] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] do_sub_op osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.512901 7f9d8016d700 10 _do_transaction on 0x274a6470
2014-06-12 14:46:06.512900 7f9d623d5700 10 osd.2 173 dequeue_op 0x39576200 finish
2014-06-12 14:46:06.512904 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:06.512907 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.512908 7f9d8016d700 15 _omap_setkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.513021 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.513047 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:06.513091 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:06.513111 7f9d8016d700 15 _omap_rmkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.513117 7f9d8016d700 10 _omap_rmkeys meta/1fe1f53a/pglog_3.426/0//-1 r = 0
2014-06-12 14:46:06.513123 7f9d8016d700 15 _omap_setkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.513137 7f9d8016d700 15 _write 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 1912832~1548288
2014-06-12 14:46:06.513278 7f9d5b6bc700  5 -- op tracker -- , seq: 747583, time: 2014-06-12 14:46:06.512615, event: header_read, op: osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513281 7f9d8016d700 10 lookup_strip_header done  cid 3.426_head oid b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.513297 7f9d5b6bc700  5 -- op tracker -- , seq: 747583, time: 2014-06-12 14:46:06.512616, event: throttled, op: osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513306 7f9d5b6bc700  5 -- op tracker -- , seq: 747583, time: 2014-06-12 14:46:06.513253, event: all_read, op: osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513317 7f9d5b6bc700  5 -- op tracker -- , seq: 747583, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513321 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:06.513327 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:06.513332 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39576c00 prio 127 cost 516903 latency 0.000717 osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.513384 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:06.513392 7f9d643d9700 10 osd.2 173 dequeue_op 0x39576c00 prio 127 cost 516903 latency 0.000777 osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.426( v 173'2679 (0'0,173'2679] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active]
2014-06-12 14:46:06.513413 7f9d643d9700  5 -- op tracker -- , seq: 747583, time: 2014-06-12 14:46:06.513409, event: reached_pg, op: osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513421 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2679 (0'0,173'2679] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] handle_message: 0x39576c00
2014-06-12 14:46:06.513432 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2679 (0'0,173'2679] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] sub_op_modify trans b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 v 173'2680 (transaction) 174
2014-06-12 14:46:06.513449 7f9d643d9700  5 -- op tracker -- , seq: 747583, time: 2014-06-12 14:46:06.513445, event: started, op: osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513460 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2679 (0'0,173'2679] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] append_log log((0'0,173'2679], crt=0'0) [173'2680 (173'2679) modify   b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 by client.4315.0:144869 2014-06-12 08:45:08.888046]
2014-06-12 14:46:06.513482 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:06.513483 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2680 (0'0,173'2680] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] add_log_entry 173'2680 (173'2679) modify   b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 by client.4315.0:144869 2014-06-12 08:45:08.888046
2014-06-12 14:46:06.513504 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2680 (0'0,173'2680] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] append_log  adding 1 keys
2014-06-12 14:46:06.513533 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2680, trimmed: 
2014-06-12 14:46:06.513554 7f9d643d9700  5 -- op tracker -- , seq: 747583, time: 2014-06-12 14:46:06.513549, event: started, op: osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513562 7f9d643d9700  5 queue_transactions existing osr(3.426 0x276d9c0)/0x276d9c0
2014-06-12 14:46:06.513565 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:06.513566 7f9d643d9700  5 queue_op 0xae92a20 seq 2483 osr(3.426 0x276d9c0) 518389 bytes   (queue has 2 ops and 2068970 bytes)
2014-06-12 14:46:06.513572 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.426( v 173'2680 (0'0,173'2680] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] do_sub_op osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.513587 7f9d643d9700 10 osd.2 173 dequeue_op 0x39576c00 finish
2014-06-12 14:46:06.513590 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:06.513593 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.513602 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9e60 (2 active)
2014-06-12 14:46:06.513610 7f9d5b6bc700  5 -- op tracker -- , seq: 747584, time: 2014-06-12 14:46:06.513352, event: header_read, op: osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513627 7f9d5b6bc700  5 -- op tracker -- , seq: 747584, time: 2014-06-12 14:46:06.513353, event: throttled, op: osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513638 7f9d5b6bc700  5 -- op tracker -- , seq: 747584, time: 2014-06-12 14:46:06.513582, event: all_read, op: osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513652 7f9d5b6bc700  5 -- op tracker -- , seq: 747584, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513655 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:06.513660 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:06.513666 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x39577c00 prio 127 cost 217895 latency 0.000313 osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.513688 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:06.513694 7f9d61bd4700 10 osd.2 173 dequeue_op 0x39577c00 prio 127 cost 217895 latency 0.000341 osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.426( v 173'2680 (0'0,173'2680] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active]
2014-06-12 14:46:06.513713 7f9d61bd4700  5 -- op tracker -- , seq: 747584, time: 2014-06-12 14:46:06.513709, event: reached_pg, op: osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513719 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2680 (0'0,173'2680] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] handle_message: 0x39577c00
2014-06-12 14:46:06.513730 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2680 (0'0,173'2680] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] sub_op_modify trans b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 v 173'2681 (transaction) 174
2014-06-12 14:46:06.513745 7f9d61bd4700  5 -- op tracker -- , seq: 747584, time: 2014-06-12 14:46:06.513741, event: started, op: osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513756 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2680 (0'0,173'2680] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] append_log log((0'0,173'2680], crt=0'0) [173'2681 (173'2680) modify   b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 by client.4315.0:144870 2014-06-12 08:45:08.888672]
2014-06-12 14:46:06.513779 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] add_log_entry 173'2681 (173'2680) modify   b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 by client.4315.0:144870 2014-06-12 08:45:08.888672
2014-06-12 14:46:06.513804 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] append_log  adding 1 keys
2014-06-12 14:46:06.513831 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2681, trimmed: 
2014-06-12 14:46:06.513853 7f9d61bd4700  5 -- op tracker -- , seq: 747584, time: 2014-06-12 14:46:06.513848, event: started, op: osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.513861 7f9d61bd4700  5 queue_transactions existing osr(3.426 0x276d9c0)/0x276d9c0
2014-06-12 14:46:06.513864 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:06.513865 7f9d61bd4700  5 queue_op 0x8813da0 seq 2484 osr(3.426 0x276d9c0) 219381 bytes   (queue has 3 ops and 2288351 bytes)
2014-06-12 14:46:06.513871 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] do_sub_op osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.513884 7f9d61bd4700 10 osd.2 173 dequeue_op 0x39577c00 finish
2014-06-12 14:46:06.513887 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:06.513890 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.513898 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b9e60 (3 active)
2014-06-12 14:46:06.519273 7f9d8016d700 10 _generic_write 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 1912832~1548288 = 0
2014-06-12 14:46:06.519399 7f9d8016d700 15 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.519424 7f9d8016d700 10 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 = 0
2014-06-12 14:46:06.519432 7f9d8016d700 15 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.519441 7f9d8016d700 10 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 = 0
2014-06-12 14:46:06.519842 7f9d8016d700 20 set_header setting 17254 cid 3.426_head oid b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 parent seq 0
2014-06-12 14:46:06.519987 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:06.520004 7f9d8016d700 20 set_header setting 2290 cid meta oid 1fe1f53a/pglog_3.426/0//-1 parent seq 0
2014-06-12 14:46:06.520014 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:06.527166 7f9d8016d700 10 _do_op 0x216ef60 seq 2482 r = 0, finisher 0xa647480 0
2014-06-12 14:46:06.527187 7f9d8016d700 10 _finish_op 0x216ef60 seq 2482 osr(3.426 0x276d9c0)/0x276d9c0
2014-06-12 14:46:06.527197 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9e60 (2 active)
2014-06-12 14:46:06.527200 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.527214 7f9d7d968700  5 _do_op 0xae92a20 seq 2483 osr(3.426 0x276d9c0)/0x276d9c0 start
2014-06-12 14:46:06.527207 7f9d7715b700  5 -- op tracker -- , seq: 747582, time: 2014-06-12 14:46:06.527197, event: commit_sent, op: osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.527220 7f9d7d968700 10 _do_transaction on 0x570d270
2014-06-12 14:46:06.527228 7f9d7d968700 15 _omap_setkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.527222 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2677 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:06.527246 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:06.527279 7f9d7795c700  5 -- op tracker -- , seq: 747582, time: 2014-06-12 14:46:06.527273, event: sub_op_applied, op: osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.527287 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2678 active] sub_op_modify_applied on 0x274a6380 op osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.527318 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2678 active] op_applied on primary on version 173'2679
2014-06-12 14:46:06.527333 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2678 active] op_applied on replica on version 173'2679
2014-06-12 14:46:06.527344 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.527361 7f9d7795c700  5 -- op tracker -- , seq: 747582, time: 2014-06-12 14:46:06.527357, event: done, op: osd_sub_op(client.4315.0:144868 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.527366 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:06.527411 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:06.527427 7f9d7d968700 15 _omap_rmkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.527433 7f9d7d968700 10 _omap_rmkeys meta/1fe1f53a/pglog_3.426/0//-1 r = 0
2014-06-12 14:46:06.527439 7f9d7d968700 15 _omap_setkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.527452 7f9d7d968700 15 _write 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 3461120~516096
2014-06-12 14:46:06.527594 7f9d7d968700 10 lookup_strip_header done  cid 3.426_head oid b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.527848 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:06.529668 7f9d7d968700 10 _generic_write 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 3461120~516096 = 0
2014-06-12 14:46:06.529718 7f9d7d968700 15 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.529739 7f9d7d968700 10 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 = 0
2014-06-12 14:46:06.529750 7f9d7d968700 15 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.529757 7f9d7d968700 10 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 = 0
2014-06-12 14:46:06.529955 7f9d7d968700 20 set_header setting 17254 cid 3.426_head oid b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 parent seq 0
2014-06-12 14:46:06.530022 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:06.530044 7f9d7d968700 20 set_header setting 2290 cid meta oid 1fe1f53a/pglog_3.426/0//-1 parent seq 0
2014-06-12 14:46:06.530057 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:06.532153 7f9d7d968700 10 _do_op 0xae92a20 seq 2483 r = 0, finisher 0x5fbb620 0
2014-06-12 14:46:06.532171 7f9d7d968700 10 _finish_op 0xae92a20 seq 2483 osr(3.426 0x276d9c0)/0x276d9c0
2014-06-12 14:46:06.532190 7f9d7915f700  5 _do_op 0x8813da0 seq 2484 osr(3.426 0x276d9c0)/0x276d9c0 start
2014-06-12 14:46:06.532189 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9e60 (1 active)
2014-06-12 14:46:06.532198 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.532199 7f9d7915f700 10 _do_transaction on 0x580eb70
2014-06-12 14:46:06.532207 7f9d7915f700 15 _omap_setkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.532198 7f9d7715b700  5 -- op tracker -- , seq: 747583, time: 2014-06-12 14:46:06.532187, event: commit_sent, op: osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.532279 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2678 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:06.532313 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:06.532314 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.532338 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:06.532357 7f9d7795c700  5 -- op tracker -- , seq: 747583, time: 2014-06-12 14:46:06.532350, event: sub_op_applied, op: osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.532364 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2679 active] sub_op_modify_applied on 0x570d180 op osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.532378 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2679 active] op_applied on primary on version 173'2680
2014-06-12 14:46:06.532395 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:06.532391 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2679 active] op_applied on replica on version 173'2680
2014-06-12 14:46:06.532411 7f9d7795c700  5 -- op tracker -- , seq: 747583, time: 2014-06-12 14:46:06.532408, event: done, op: osd_sub_op(client.4315.0:144869 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.532443 7f9d7915f700 15 _omap_rmkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.532451 7f9d7915f700 10 _omap_rmkeys meta/1fe1f53a/pglog_3.426/0//-1 r = 0
2014-06-12 14:46:06.532457 7f9d7915f700 15 _omap_setkeys meta/1fe1f53a/pglog_3.426/0//-1
2014-06-12 14:46:06.532471 7f9d7915f700 15 _write 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 3977216~217088
2014-06-12 14:46:06.532674 7f9d7915f700 10 lookup_strip_header done  cid 3.426_head oid b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.532910 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:06.533528 7f9d7915f700 10 _generic_write 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 3977216~217088 = 0
2014-06-12 14:46:06.533561 7f9d7915f700 15 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.533577 7f9d7915f700 10 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 = 0
2014-06-12 14:46:06.533584 7f9d7915f700 15 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3
2014-06-12 14:46:06.533591 7f9d7915f700 10 _setattrs 3.426_head/b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 = 0
2014-06-12 14:46:06.533710 7f9d7915f700 20 set_header setting 17254 cid 3.426_head oid b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 parent seq 0
2014-06-12 14:46:06.533749 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:06.533758 7f9d7915f700 20 set_header setting 2290 cid meta oid 1fe1f53a/pglog_3.426/0//-1 parent seq 0
2014-06-12 14:46:06.533778 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:06.534523 7f9d7915f700 10 _do_op 0x8813da0 seq 2484 r = 0, finisher 0x73ea3c0 0
2014-06-12 14:46:06.534532 7f9d7915f700 10 _finish_op 0x8813da0 seq 2484 osr(3.426 0x276d9c0)/0x276d9c0
2014-06-12 14:46:06.534539 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b9e60 (0 active)
2014-06-12 14:46:06.534544 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.534550 7f9d7715b700  5 -- op tracker -- , seq: 747584, time: 2014-06-12 14:46:06.534541, event: commit_sent, op: osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.534559 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2679 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:06.534584 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:06.534618 7f9d7795c700  5 -- op tracker -- , seq: 747584, time: 2014-06-12 14:46:06.534613, event: sub_op_applied, op: osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.534630 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2680 active] sub_op_modify_applied on 0x580ea80 op osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:06.534647 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2680 active] op_applied on primary on version 173'2681
2014-06-12 14:46:06.534658 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.426( v 173'2681 (0'0,173'2681] local-les=157 n=294 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2680 active] op_applied on replica on version 173'2681
2014-06-12 14:46:06.534684 7f9d7795c700  5 -- op tracker -- , seq: 747584, time: 2014-06-12 14:46:06.534679, event: done, op: osd_sub_op(client.4315.0:144870 3.426 b6ea6c26/rbd_data.10b26b8b4567.000000000000107d/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:06.585344 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:06.585347 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:06.585348 7f9d74bfa700 20 osd.2 173 _dispatch 0x14bc91e0 ping v1
2014-06-12 14:46:06.585350 7f9d74bfa700 10 osd.2 173 ping from client.4310
2014-06-12 14:46:06.585352 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:06.585353 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:06.742265 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:06.742323 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:06.807442 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.828309 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.842671 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.856956 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.863666 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.877864 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.889062 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.895433 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.895954 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.898337 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.902258 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.907304 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.913937 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.932485 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.939157 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.944758 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:06.948499 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:06.950378 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:07.040959 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:07.040962 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:07.040963 7f9d74bfa700 20 osd.2 173 _dispatch 0x2db689c0 ping v1
2014-06-12 14:46:07.040965 7f9d74bfa700 10 osd.2 173 ping from client.4317
2014-06-12 14:46:07.040967 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:07.040968 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:07.044478 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:07.051644 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:07.198181 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:07.198205 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:07.469924 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:07.469947 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:46:07.469950 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:46:07.469955 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:46:07.469959 7f9d84175700 10 osd.2 173 send_pg_stats - 7 pgs updated
2014-06-12 14:46:07.470021 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:07.470052 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 13.82 >= max 0.5 = no, load too high
2014-06-12 14:46:07.470062 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:07.470064 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92615 < max (604800 seconds)
2014-06-12 14:46:07.470072 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:07.470074 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:07.470075 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:07.470076 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 32; oldest is 37.190175 seconds old
2014-06-12 14:46:07.513938 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19402 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,4,4,21,3])
2014-06-12 14:46:07.513963 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19402 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,4,4,21,3])
2014-06-12 14:46:07.514373 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:07.514397 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:07.514506 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:07.514538 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:07.514559 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:07.514620 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:07.514664 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:07.514701 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:07.514761 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:07.514782 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:07.514800 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:07.514827 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:07.514847 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:07.514869 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:07.514892 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:07.514913 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:07.514933 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:07.514954 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:07.514975 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:07.514999 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:07.515037 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:07.515060 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:07.698666 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:07.698763 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:07.716249 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:07.716480 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:07.739775 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:07.739951 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:07.934480 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:07.934639 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:08.153143 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:08.156659 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.167025 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.171174 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.188007 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:08.188200 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:08.191356 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:08.191614 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:08.240291 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:08.240325 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:08.316575 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:08.316648 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:08.366800 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:08.439503 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.442763 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:08.442792 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:08.450162 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.461953 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.466113 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.468108 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.470134 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:08.470158 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:08.470172 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 13.82 >= max 0.5 = no, load too high
2014-06-12 14:46:08.470185 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:08.470187 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92616 < max (604800 seconds)
2014-06-12 14:46:08.470196 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:08.470198 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:08.470198 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:08.470200 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 32; oldest is 38.190299 seconds old
2014-06-12 14:46:08.470255 7f9d84175700  0 log [WRN] : 7 slow requests, 4 included below; oldest blocked for > 38.190299 secs
2014-06-12 14:46:08.470258 7f9d84175700  0 log [WRN] : slow request 30.793910 seconds old, received at 2014-06-12 14:45:37.676290: osd_op(client.4306.0:144048 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2109440~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:08.470261 7f9d84175700  0 log [WRN] : slow request 30.792397 seconds old, received at 2014-06-12 14:45:37.677803: osd_op(client.4306.0:144049 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2625536~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:08.470263 7f9d84175700  0 log [WRN] : slow request 30.789027 seconds old, received at 2014-06-12 14:45:37.681173: osd_op(client.4306.0:144050 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 3141632~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:08.470266 7f9d84175700  0 log [WRN] : slow request 30.140887 seconds old, received at 2014-06-12 14:45:38.329313: osd_op(client.4306.0:144051 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 4173824~20480] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:08.473903 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:08.473905 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:08.473906 7f9d74bfa700 20 osd.2 173 _dispatch 0x7435c00 pg_stats_ack(7 pgs tid 4847) v1
2014-06-12 14:46:08.473908 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:46:08.473914 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:08.473915 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:08.483493 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.484053 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.485943 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.498173 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.500461 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.505128 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.512969 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.513659 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.513951 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.527261 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.532255 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.534598 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.561547 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:08.561574 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:08.614453 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19402 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,4,4,21,3])
2014-06-12 14:46:08.614473 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19402 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,4,4,21,3])
2014-06-12 14:46:08.614921 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:08.614964 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:08.614993 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:08.615058 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:08.615076 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:08.615102 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:08.615123 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:08.615141 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:08.615161 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:08.615202 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:08.615223 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:08.615243 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:08.615254 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:08.615266 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:08.615277 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:08.615289 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:08.615300 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:08.615311 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:08.615327 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:08.615340 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:08.615350 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:08.615363 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:08.688560 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:08.688792 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:08.807522 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.828384 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.842741 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.857029 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.863734 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.877931 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.889148 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.895501 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.896019 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.898408 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.901437 7f9d5a16f700  5 -- op tracker -- , seq: 747585, time: 2014-06-12 14:46:08.901375, event: header_read, op: osd_sub_op_reply(client.4310.0:142391 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:08.901453 7f9d5a16f700  5 -- op tracker -- , seq: 747585, time: 2014-06-12 14:46:08.901378, event: throttled, op: osd_sub_op_reply(client.4310.0:142391 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:08.901479 7f9d5a16f700  5 -- op tracker -- , seq: 747585, time: 2014-06-12 14:46:08.901396, event: all_read, op: osd_sub_op_reply(client.4310.0:142391 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:08.901487 7f9d5a16f700  5 -- op tracker -- , seq: 747585, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142391 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:08.901493 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142391 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:08.901500 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:08.901510 7f9d5a16f700 15 osd.2 173 enqueue_op 0x9fac300 prio 196 cost 0 latency 0.000134 osd_sub_op_reply(client.4310.0:142391 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:08.901547 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:08.901562 7f9d64bda700 10 osd.2 173 dequeue_op 0x9fac300 prio 196 cost 0 latency 0.000187 osd_sub_op_reply(client.4310.0:142391 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0) v2 pg pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]
2014-06-12 14:46:08.901606 7f9d64bda700  5 -- op tracker -- , seq: 747585, time: 2014-06-12 14:46:08.901599, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142391 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:08.901634 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] handle_message: 0x9fac300
2014-06-12 14:46:08.901663 7f9d64bda700  5 -- op tracker -- , seq: 747585, time: 2014-06-12 14:46:08.901657, event: started, op: osd_sub_op_reply(client.4310.0:142391 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:08.901685 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] sub_op_modify_reply: tid 235996 op  ack_type 4 from 24
2014-06-12 14:46:08.901723 7f9d64bda700  5 -- op tracker -- , seq: 746902, time: 2014-06-12 14:46:08.901709, event: sub_op_commit_rec, op: osd_op(client.4310.0:142391 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~741376] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:08.901742 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] repop_all_applied: repop tid 235996 all applied 
2014-06-12 14:46:08.901795 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] eval_repop repgather(0x2b618900 173'2703 rep_tid=235996 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142391 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~741376] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:08.901829 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]  sending ack on repgather(0x2b618900 173'2703 rep_tid=235996 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142391 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~741376] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) 0x28ace500
2014-06-12 14:46:08.901895 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] repop_all_committed: repop tid 235996 all committed 
2014-06-12 14:46:08.901917 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean] eval_repop repgather(0x2b618900 173'2703 rep_tid=235996 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142391 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~741376] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:08.901946 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean] log_op_stats osd_op(client.4310.0:142391 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~741376] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4 inb 741655 outb 0 rlat 38.621991 lat 38.622043
2014-06-12 14:46:08.901979 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean] publish_stats_to_osd 173:2886
2014-06-12 14:46:08.902002 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean]  sending commit on repgather(0x2b618900 173'2703 rep_tid=235996 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142391 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~741376] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4) 0x34d91680
2014-06-12 14:46:08.902064 7f9d64bda700  5 -- op tracker -- , seq: 746902, time: 2014-06-12 14:46:08.902054, event: commit_sent, op: osd_op(client.4310.0:142391 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~741376] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:08.902078 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean]  requeue_ops 
2014-06-12 14:46:08.902097 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]  removing repgather(0x2b618900 173'2703 rep_tid=235996 committed?=1 applied?=1 op=osd_op(client.4310.0:142391 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~741376] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:08.902123 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]    q front is repgather(0x2b618900 173'2703 rep_tid=235996 committed?=1 applied?=1 op=osd_op(client.4310.0:142391 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~741376] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:08.902148 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] remove_repop repgather(0x2b618900 173'2703 rep_tid=235996 committed?=1 applied?=1 op=osd_op(client.4310.0:142391 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~741376] 3.be44fe13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:08.902173 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]  requeue_ops 
2014-06-12 14:46:08.902218 7f9d64bda700  5 -- op tracker -- , seq: 746902, time: 2014-06-12 14:46:08.902208, event: done, op: osd_op(client.4310.0:142391 rbd_data.10ad6b8b4567.0000000000000ef3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~741376] 3.be44fe13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:08.902250 7f9d64bda700 10 osd.2 173 dequeue_op 0x9fac300 finish
2014-06-12 14:46:08.902263 7f9d64bda700  5 -- op tracker -- , seq: 747585, time: 2014-06-12 14:46:08.902257, event: done, op: osd_sub_op_reply(client.4310.0:142391 3.613 be44fe13/rbd_data.10ad6b8b4567.0000000000000ef3/head//3 [] ondisk, result = 0)
2014-06-12 14:46:08.902283 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:08.902288 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.907399 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.914012 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.932554 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.939228 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.944822 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:08.948564 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:08.950444 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.044547 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.051717 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.149597 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:09.149950 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:09.152972 7f9d57543700  5 -- op tracker -- , seq: 747586, time: 2014-06-12 14:46:09.152171, event: header_read, op: osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.152992 7f9d57543700  5 -- op tracker -- , seq: 747586, time: 2014-06-12 14:46:09.152174, event: throttled, op: osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.153004 7f9d57543700  5 -- op tracker -- , seq: 747586, time: 2014-06-12 14:46:09.152921, event: all_read, op: osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.153015 7f9d57543700  5 -- op tracker -- , seq: 747586, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.153021 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.153030 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:09.153039 7f9d57543700 15 osd.2 173 enqueue_op 0x24c9cd00 prio 127 cost 332583 latency 0.000868 osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.153079 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.153103 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x24c9cd00 prio 127 cost 332583 latency 0.000932 osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.377( v 173'3045 (79'44,173'3045] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3044 active]
2014-06-12 14:46:09.153136 7f9d6a3e5700  5 -- op tracker -- , seq: 747586, time: 2014-06-12 14:46:09.153130, event: reached_pg, op: osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.153151 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3045 (79'44,173'3045] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3044 active] handle_message: 0x24c9cd00
2014-06-12 14:46:09.153167 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3045 (79'44,173'3045] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3044 active] sub_op_modify trans af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 v 173'3046 (transaction) 174
2014-06-12 14:46:09.153189 7f9d6a3e5700  5 -- op tracker -- , seq: 747586, time: 2014-06-12 14:46:09.153184, event: started, op: osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.153209 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3045 (79'44,173'3045] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3044 active] append_log log((79'44,173'3045], crt=0'0) [173'3046 (173'866) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147013 2014-06-12 08:45:11.511174]
2014-06-12 14:46:09.153240 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3046 (79'44,173'3046] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3044 active] add_log_entry 173'3046 (173'866) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147013 2014-06-12 08:45:11.511174
2014-06-12 14:46:09.153270 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3046 (79'44,173'3046] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3044 active] append_log  adding 1 keys
2014-06-12 14:46:09.153289 7f9d6a3e5700 10 trim log((79'44,173'3046], crt=0'0) to 79'45
2014-06-12 14:46:09.153321 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3046, trimmed: 79'45
2014-06-12 14:46:09.153355 7f9d6a3e5700  5 -- op tracker -- , seq: 747586, time: 2014-06-12 14:46:09.153348, event: started, op: osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.153369 7f9d6a3e5700  5 queue_transactions existing osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.153374 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.153376 7f9d6a3e5700  5 queue_op 0x835b3e0 seq 2742 osr(3.377 0x276d7e0) 334104 bytes   (queue has 1 ops and 334104 bytes)
2014-06-12 14:46:09.153384 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.377( v 173'3046 (79'45,173'3046] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3044 active] do_sub_op osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.153403 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x24c9cd00 finish
2014-06-12 14:46:09.153390 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x51818c0 (1 active)
2014-06-12 14:46:09.153407 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.153411 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.153411 7f9d7e169700  5 _do_op 0x835b3e0 seq 2742 osr(3.377 0x276d7e0)/0x276d7e0 start
2014-06-12 14:46:09.153417 7f9d7e169700 10 _do_transaction on 0x15edc0f0
2014-06-12 14:46:09.153426 7f9d7e169700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.153602 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.153664 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.153744 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.153782 7f9d7e169700 15 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.153796 7f9d7e169700 10 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1 r = 0
2014-06-12 14:46:09.153805 7f9d7e169700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.154039 7f9d7e169700 15 _write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 0~331776
2014-06-12 14:46:09.154279 7f9d7e169700 10 lookup_strip_header done  cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.154442 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:09.156031 7f9d7e169700 10 _generic_write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 0~331776 = 0
2014-06-12 14:46:09.156075 7f9d7e169700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.156095 7f9d7e169700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.156106 7f9d7e169700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.156117 7f9d7e169700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.156343 7f9d7e169700 20 set_header setting 17368 cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 parent seq 0
2014-06-12 14:46:09.156406 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.156426 7f9d7e169700 20 set_header setting 2216 cid meta oid 1fe1d86a/pglog_3.377/0//-1 parent seq 0
2014-06-12 14:46:09.156439 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:09.157832 7f9d5a16f700  5 -- op tracker -- , seq: 747587, time: 2014-06-12 14:46:09.156810, event: header_read, op: osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.157841 7f9d5a16f700  5 -- op tracker -- , seq: 747587, time: 2014-06-12 14:46:09.156811, event: throttled, op: osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.157857 7f9d5a16f700  5 -- op tracker -- , seq: 747587, time: 2014-06-12 14:46:09.157805, event: all_read, op: osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.157862 7f9d5a16f700  5 -- op tracker -- , seq: 747587, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.157866 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.157870 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:09.157876 7f9d5a16f700 15 osd.2 173 enqueue_op 0x9fac200 prio 127 cost 516903 latency 0.001065 osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.157898 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.157904 7f9d653db700 10 osd.2 173 dequeue_op 0x9fac200 prio 127 cost 516903 latency 0.001094 osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4e0( v 173'2632 (0'0,173'2632] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active]
2014-06-12 14:46:09.157920 7f9d653db700  5 -- op tracker -- , seq: 747587, time: 2014-06-12 14:46:09.157917, event: reached_pg, op: osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.157927 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2632 (0'0,173'2632] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active] handle_message: 0x9fac200
2014-06-12 14:46:09.157934 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2632 (0'0,173'2632] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active] sub_op_modify trans 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 v 173'2633 (transaction) 174
2014-06-12 14:46:09.157944 7f9d653db700  5 -- op tracker -- , seq: 747587, time: 2014-06-12 14:46:09.157941, event: started, op: osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.157953 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2632 (0'0,173'2632] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active] append_log log((0'0,173'2632], crt=0'0) [173'2633 (173'2632) modify   6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 by client.4306.0:143776 2014-06-12 08:43:01.306545]
2014-06-12 14:46:09.157967 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2633 (0'0,173'2633] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active] add_log_entry 173'2633 (173'2632) modify   6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 by client.4306.0:143776 2014-06-12 08:43:01.306545
2014-06-12 14:46:09.157982 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2633 (0'0,173'2633] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active] append_log  adding 1 keys
2014-06-12 14:46:09.158004 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2633, trimmed: 
2014-06-12 14:46:09.158019 7f9d653db700  5 -- op tracker -- , seq: 747587, time: 2014-06-12 14:46:09.158016, event: started, op: osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.158026 7f9d653db700  5 queue_transactions existing osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:09.158029 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.158030 7f9d653db700  5 queue_op 0x50c77a0 seq 2433 osr(3.4e0 0x276dbd0) 518389 bytes   (queue has 2 ops and 852493 bytes)
2014-06-12 14:46:09.158038 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5180900 (2 active)
2014-06-12 14:46:09.158034 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2633 (0'0,173'2633] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active] do_sub_op osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.158044 7f9d653db700 10 osd.2 173 dequeue_op 0x9fac200 finish
2014-06-12 14:46:09.158046 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.158045 7f9d7e96a700  5 _do_op 0x50c77a0 seq 2433 osr(3.4e0 0x276dbd0)/0x276dbd0 start
2014-06-12 14:46:09.158048 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.158049 7f9d7e96a700 10 _do_transaction on 0x610eb70
2014-06-12 14:46:09.158056 7f9d7e96a700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:09.158173 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:09.158197 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.158731 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.158802 7f9d7e169700 10 _do_op 0x835b3e0 seq 2742 r = 0, finisher 0x70f45e0 0
2014-06-12 14:46:09.158817 7f9d7e169700 10 _finish_op 0x835b3e0 seq 2742 osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.158824 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x51818c0 (1 active)
2014-06-12 14:46:09.158828 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.158857 7f9d7715b700  5 -- op tracker -- , seq: 747586, time: 2014-06-12 14:46:09.158848, event: commit_sent, op: osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.158864 7f9d7e96a700 15 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:09.158885 7f9d7e96a700 10 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1 r = 0
2014-06-12 14:46:09.158893 7f9d7e96a700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:09.158875 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3046 (79'45,173'3046] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3044 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:09.158905 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:09.158914 7f9d7e96a700 15 _write 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 1318912~516096
2014-06-12 14:46:09.159245 7f9d7e96a700 10 lookup_strip_header done  cid 3.4e0_head oid 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:09.159410 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:09.159573 7f9d7795c700  5 -- op tracker -- , seq: 747586, time: 2014-06-12 14:46:09.159566, event: sub_op_applied, op: osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.159584 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3046 (79'45,173'3046] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3045 active] sub_op_modify_applied on 0x15edc000 op osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.159600 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3046 (79'45,173'3046] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3045 active] op_applied on primary on version 173'3046
2014-06-12 14:46:09.159609 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3046 (79'45,173'3046] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3045 active] op_applied on replica on version 173'3046
2014-06-12 14:46:09.159633 7f9d7795c700  5 -- op tracker -- , seq: 747586, time: 2014-06-12 14:46:09.159628, event: done, op: osd_sub_op(client.4316.0:147013 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3046 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.160792 7f9d7e96a700 10 _generic_write 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 1318912~516096 = 0
2014-06-12 14:46:09.160848 7f9d7e96a700 15 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:09.160866 7f9d7e96a700 10 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:09.160873 7f9d7e96a700 15 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:09.160882 7f9d7e96a700 10 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:09.161023 7f9d7e96a700 20 set_header setting 16555 cid 3.4e0_head oid 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:09.161075 7f9d7e96a700 20 set_header setting 2236 cid meta oid 1fe07435/pglog_3.4e0/0//-1 parent seq 0
2014-06-12 14:46:09.161085 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.161090 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:09.162999 7f9d7e96a700 10 _do_op 0x50c77a0 seq 2433 r = 0, finisher 0x216fc880 0
2014-06-12 14:46:09.163020 7f9d7e96a700 10 _finish_op 0x50c77a0 seq 2433 osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:09.163030 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5180900 (0 active)
2014-06-12 14:46:09.163034 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.163045 7f9d7795c700  5 -- op tracker -- , seq: 747587, time: 2014-06-12 14:46:09.163036, event: sub_op_applied, op: osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.163052 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2633 (0'0,173'2633] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active] sub_op_modify_applied on 0x610ea80 op osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.163071 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:09.163085 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2633 (0'0,173'2633] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active] op_applied on primary on version 173'2633
2014-06-12 14:46:09.163096 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2633 (0'0,173'2633] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active] op_applied on replica on version 173'2633
2014-06-12 14:46:09.163121 7f9d7715b700  5 -- op tracker -- , seq: 747587, time: 2014-06-12 14:46:09.163115, event: commit_sent, op: osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.163129 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2633 (0'0,173'2633] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2631 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:09.163148 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:09.163182 7f9d7715b700  5 -- op tracker -- , seq: 747587, time: 2014-06-12 14:46:09.163177, event: done, op: osd_sub_op(client.4306.0:143776 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2633 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.165288 7f9d57543700  5 -- op tracker -- , seq: 747588, time: 2014-06-12 14:46:09.164382, event: header_read, op: osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.165299 7f9d57543700  5 -- op tracker -- , seq: 747588, time: 2014-06-12 14:46:09.164383, event: throttled, op: osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.165304 7f9d57543700  5 -- op tracker -- , seq: 747588, time: 2014-06-12 14:46:09.165252, event: all_read, op: osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.165309 7f9d57543700  5 -- op tracker -- , seq: 747588, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.165313 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.165318 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:09.165323 7f9d57543700 15 osd.2 173 enqueue_op 0x5abe000 prio 127 cost 516903 latency 0.000941 osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.165343 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.165349 7f9d63bd8700 10 osd.2 173 dequeue_op 0x5abe000 prio 127 cost 516903 latency 0.000967 osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.377( v 173'3046 (79'45,173'3046] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3045 active]
2014-06-12 14:46:09.165366 7f9d63bd8700  5 -- op tracker -- , seq: 747588, time: 2014-06-12 14:46:09.165364, event: reached_pg, op: osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.165374 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3046 (79'45,173'3046] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3045 active] handle_message: 0x5abe000
2014-06-12 14:46:09.165387 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3046 (79'45,173'3046] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3045 active] sub_op_modify trans af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 v 173'3047 (transaction) 174
2014-06-12 14:46:09.165399 7f9d63bd8700  5 -- op tracker -- , seq: 747588, time: 2014-06-12 14:46:09.165397, event: started, op: osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.165417 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3046 (79'45,173'3046] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3045 active] append_log log((79'45,173'3046], crt=0'0) [173'3047 (173'3046) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147014 2014-06-12 08:45:11.542003]
2014-06-12 14:46:09.165436 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3047 (79'45,173'3047] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3045 active] add_log_entry 173'3047 (173'3046) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147014 2014-06-12 08:45:11.542003
2014-06-12 14:46:09.165453 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3047 (79'45,173'3047] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3045 active] append_log  adding 1 keys
2014-06-12 14:46:09.165462 7f9d63bd8700 10 trim log((79'45,173'3047], crt=0'0) to 79'46
2014-06-12 14:46:09.165484 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3047, trimmed: 79'46
2014-06-12 14:46:09.165503 7f9d63bd8700  5 -- op tracker -- , seq: 747588, time: 2014-06-12 14:46:09.165499, event: started, op: osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.165511 7f9d63bd8700  5 queue_transactions existing osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.165514 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.165515 7f9d63bd8700  5 queue_op 0x7c89e00 seq 2743 osr(3.377 0x276d7e0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:09.165525 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x51818c0 (1 active)
2014-06-12 14:46:09.165531 7f9d82171700  5 _do_op 0x7c89e00 seq 2743 osr(3.377 0x276d7e0)/0x276d7e0 start
2014-06-12 14:46:09.165520 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.377( v 173'3047 (79'46,173'3047] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3045 active] do_sub_op osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.165533 7f9d63bd8700 10 osd.2 173 dequeue_op 0x5abe000 finish
2014-06-12 14:46:09.165535 7f9d82171700 10 _do_transaction on 0x65f82b0
2014-06-12 14:46:09.165536 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.165539 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.165541 7f9d82171700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.165648 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.165674 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.165707 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.165811 7f9d82171700 15 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.165819 7f9d82171700 10 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1 r = 0
2014-06-12 14:46:09.165823 7f9d82171700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.165837 7f9d82171700 15 _write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 331776~516096
2014-06-12 14:46:09.165933 7f9d82171700 10 lookup_strip_header done  cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.166060 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:09.167469 7f9d82171700 10 _generic_write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 331776~516096 = 0
2014-06-12 14:46:09.167519 7f9d82171700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.167534 7f9d82171700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.167542 7f9d82171700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.167552 7f9d82171700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.167706 7f9d82171700 20 set_header setting 17368 cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 parent seq 0
2014-06-12 14:46:09.167776 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.167793 7f9d82171700 20 set_header setting 2216 cid meta oid 1fe1d86a/pglog_3.377/0//-1 parent seq 0
2014-06-12 14:46:09.167810 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:09.170381 7f9d82171700 10 _do_op 0x7c89e00 seq 2743 r = 0, finisher 0x1e3bf460 0
2014-06-12 14:46:09.170427 7f9d7715b700  5 -- op tracker -- , seq: 747588, time: 2014-06-12 14:46:09.170415, event: commit_sent, op: osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.170440 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3047 (79'46,173'3047] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3045 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:09.170475 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:09.170508 7f9d82171700 10 _finish_op 0x7c89e00 seq 2743 osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.170529 7f9d7795c700  5 -- op tracker -- , seq: 747588, time: 2014-06-12 14:46:09.170524, event: sub_op_applied, op: osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.170537 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3047 (79'46,173'3047] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3046 active] sub_op_modify_applied on 0x65f81c0 op osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.170552 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3047 (79'46,173'3047] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3046 active] op_applied on primary on version 173'3047
2014-06-12 14:46:09.170562 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3047 (79'46,173'3047] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3046 active] op_applied on replica on version 173'3047
2014-06-12 14:46:09.170591 7f9d82171700  5 -- op tracker -- , seq: 747588, time: 2014-06-12 14:46:09.170585, event: done, op: osd_sub_op(client.4316.0:147014 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3047 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.170664 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x51818c0 (0 active)
2014-06-12 14:46:09.170671 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.176451 7f9d57543700  5 -- op tracker -- , seq: 747589, time: 2014-06-12 14:46:09.175630, event: header_read, op: osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.176463 7f9d57543700  5 -- op tracker -- , seq: 747589, time: 2014-06-12 14:46:09.175631, event: throttled, op: osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.176471 7f9d57543700  5 -- op tracker -- , seq: 747589, time: 2014-06-12 14:46:09.176410, event: all_read, op: osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.176476 7f9d57543700  5 -- op tracker -- , seq: 747589, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.176481 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.176488 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:09.176497 7f9d57543700 15 osd.2 173 enqueue_op 0x34016d00 prio 127 cost 516903 latency 0.000867 osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.176533 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.176545 7f9d69be4700 10 osd.2 173 dequeue_op 0x34016d00 prio 127 cost 516903 latency 0.000914 osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.377( v 173'3047 (79'46,173'3047] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3046 active]
2014-06-12 14:46:09.176569 7f9d69be4700  5 -- op tracker -- , seq: 747589, time: 2014-06-12 14:46:09.176565, event: reached_pg, op: osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.176578 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3047 (79'46,173'3047] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3046 active] handle_message: 0x34016d00
2014-06-12 14:46:09.176590 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3047 (79'46,173'3047] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3046 active] sub_op_modify trans af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 v 173'3048 (transaction) 174
2014-06-12 14:46:09.176606 7f9d69be4700  5 -- op tracker -- , seq: 747589, time: 2014-06-12 14:46:09.176602, event: started, op: osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.176622 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3047 (79'46,173'3047] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3046 active] append_log log((79'46,173'3047], crt=0'0) [173'3048 (173'3047) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147015 2014-06-12 08:45:11.553254]
2014-06-12 14:46:09.176655 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3048 (79'46,173'3048] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3046 active] add_log_entry 173'3048 (173'3047) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147015 2014-06-12 08:45:11.553254
2014-06-12 14:46:09.176683 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3048 (79'46,173'3048] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3046 active] append_log  adding 1 keys
2014-06-12 14:46:09.176699 7f9d69be4700 10 trim log((79'46,173'3048], crt=0'0) to 79'47
2014-06-12 14:46:09.176740 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3048, trimmed: 79'47
2014-06-12 14:46:09.176770 7f9d69be4700  5 -- op tracker -- , seq: 747589, time: 2014-06-12 14:46:09.176764, event: started, op: osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.176784 7f9d69be4700  5 queue_transactions existing osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.176788 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.176790 7f9d69be4700  5 queue_op 0x80b7140 seq 2744 osr(3.377 0x276d7e0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:09.176801 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.377( v 173'3048 (79'47,173'3048] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3046 active] do_sub_op osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.176817 7f9d69be4700 10 osd.2 173 dequeue_op 0x34016d00 finish
2014-06-12 14:46:09.176820 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.176817 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x51818c0 (1 active)
2014-06-12 14:46:09.176823 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.176825 7f9d7815d700  5 _do_op 0x80b7140 seq 2744 osr(3.377 0x276d7e0)/0x276d7e0 start
2014-06-12 14:46:09.176831 7f9d7815d700 10 _do_transaction on 0x1c0b5eb0
2014-06-12 14:46:09.176839 7f9d7815d700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.176955 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.176988 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.177032 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.177061 7f9d7815d700 15 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.177071 7f9d7815d700 10 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1 r = 0
2014-06-12 14:46:09.177078 7f9d7815d700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.177200 7f9d7815d700 15 _write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 847872~516096
2014-06-12 14:46:09.177318 7f9d7815d700 10 lookup_strip_header done  cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.177438 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:09.178716 7f9d7815d700 10 _generic_write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 847872~516096 = 0
2014-06-12 14:46:09.178760 7f9d7815d700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.178774 7f9d7815d700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.178781 7f9d7815d700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.178789 7f9d7815d700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.178933 7f9d7815d700 20 set_header setting 17368 cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 parent seq 0
2014-06-12 14:46:09.178989 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.179005 7f9d7815d700 20 set_header setting 2216 cid meta oid 1fe1d86a/pglog_3.377/0//-1 parent seq 0
2014-06-12 14:46:09.179016 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:09.180601 7f9d7815d700 10 _do_op 0x80b7140 seq 2744 r = 0, finisher 0x8587ee0 0
2014-06-12 14:46:09.180614 7f9d7815d700 10 _finish_op 0x80b7140 seq 2744 osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.180620 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x51818c0 (0 active)
2014-06-12 14:46:09.180627 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.180646 7f9d7715b700  5 -- op tracker -- , seq: 747589, time: 2014-06-12 14:46:09.180638, event: commit_sent, op: osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.180668 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3048 (79'47,173'3048] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3046 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:09.180691 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:09.180733 7f9d7795c700  5 -- op tracker -- , seq: 747589, time: 2014-06-12 14:46:09.180725, event: sub_op_applied, op: osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.180740 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3048 (79'47,173'3048] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3047 active] sub_op_modify_applied on 0x1c0b5dc0 op osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.180750 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3048 (79'47,173'3048] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3047 active] op_applied on primary on version 173'3048
2014-06-12 14:46:09.180756 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3048 (79'47,173'3048] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3047 active] op_applied on replica on version 173'3048
2014-06-12 14:46:09.180775 7f9d7795c700  5 -- op tracker -- , seq: 747589, time: 2014-06-12 14:46:09.180773, event: done, op: osd_sub_op(client.4316.0:147015 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3048 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.186851 7f9d57543700  5 -- op tracker -- , seq: 747590, time: 2014-06-12 14:46:09.186099, event: header_read, op: osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.186863 7f9d57543700  5 -- op tracker -- , seq: 747590, time: 2014-06-12 14:46:09.186100, event: throttled, op: osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.186870 7f9d57543700  5 -- op tracker -- , seq: 747590, time: 2014-06-12 14:46:09.186814, event: all_read, op: osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.186875 7f9d57543700  5 -- op tracker -- , seq: 747590, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.186879 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.186884 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:09.186891 7f9d57543700 15 osd.2 173 enqueue_op 0x28dd0400 prio 127 cost 516903 latency 0.000792 osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.186919 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.186939 7f9d68be2700 10 osd.2 173 dequeue_op 0x28dd0400 prio 127 cost 516903 latency 0.000839 osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.377( v 173'3048 (79'47,173'3048] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3047 active]
2014-06-12 14:46:09.186965 7f9d68be2700  5 -- op tracker -- , seq: 747590, time: 2014-06-12 14:46:09.186961, event: reached_pg, op: osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.186977 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3048 (79'47,173'3048] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3047 active] handle_message: 0x28dd0400
2014-06-12 14:46:09.186991 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3048 (79'47,173'3048] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3047 active] sub_op_modify trans af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 v 173'3049 (transaction) 174
2014-06-12 14:46:09.187017 7f9d68be2700  5 -- op tracker -- , seq: 747590, time: 2014-06-12 14:46:09.187005, event: started, op: osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.187036 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3048 (79'47,173'3048] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3047 active] append_log log((79'47,173'3048], crt=0'0) [173'3049 (173'3048) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147016 2014-06-12 08:45:11.563670]
2014-06-12 14:46:09.187063 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3049 (79'47,173'3049] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3047 active] add_log_entry 173'3049 (173'3048) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147016 2014-06-12 08:45:11.563670
2014-06-12 14:46:09.187089 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3049 (79'47,173'3049] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3047 active] append_log  adding 1 keys
2014-06-12 14:46:09.187108 7f9d68be2700 10 trim log((79'47,173'3049], crt=0'0) to 79'48
2014-06-12 14:46:09.187137 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3049, trimmed: 79'48
2014-06-12 14:46:09.187166 7f9d68be2700  5 -- op tracker -- , seq: 747590, time: 2014-06-12 14:46:09.187161, event: started, op: osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.187178 7f9d68be2700  5 queue_transactions existing osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.187182 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.187184 7f9d68be2700  5 queue_op 0x4878a80 seq 2745 osr(3.377 0x276d7e0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:09.187191 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.377( v 173'3049 (79'48,173'3049] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3047 active] do_sub_op osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.187208 7f9d68be2700 10 osd.2 173 dequeue_op 0x28dd0400 finish
2014-06-12 14:46:09.187207 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x51818c0 (1 active)
2014-06-12 14:46:09.187212 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.187214 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.187215 7f9d7d167700  5 _do_op 0x4878a80 seq 2745 osr(3.377 0x276d7e0)/0x276d7e0 start
2014-06-12 14:46:09.187220 7f9d7d167700 10 _do_transaction on 0x274a77b0
2014-06-12 14:46:09.187228 7f9d7d167700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.187336 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.187353 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.187386 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.187403 7f9d7d167700 15 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.187410 7f9d7d167700 10 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1 r = 0
2014-06-12 14:46:09.187414 7f9d7d167700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.187422 7f9d7d167700 15 _write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 1363968~516096
2014-06-12 14:46:09.187513 7f9d7d167700 10 lookup_strip_header done  cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.187721 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:09.189054 7f9d7d167700 10 _generic_write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 1363968~516096 = 0
2014-06-12 14:46:09.189096 7f9d7d167700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.189108 7f9d7d167700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.189120 7f9d7d167700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.189125 7f9d7d167700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.189270 7f9d7d167700 20 set_header setting 17368 cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 parent seq 0
2014-06-12 14:46:09.189323 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.189341 7f9d7d167700 20 set_header setting 2216 cid meta oid 1fe1d86a/pglog_3.377/0//-1 parent seq 0
2014-06-12 14:46:09.189354 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:09.191408 7f9d7d167700 10 _do_op 0x4878a80 seq 2745 r = 0, finisher 0x8053a00 0
2014-06-12 14:46:09.191423 7f9d7d167700 10 _finish_op 0x4878a80 seq 2745 osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.191429 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x51818c0 (0 active)
2014-06-12 14:46:09.191441 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.191446 7f9d7715b700  5 -- op tracker -- , seq: 747590, time: 2014-06-12 14:46:09.191433, event: commit_sent, op: osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.191455 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3049 (79'48,173'3049] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3047 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:09.191482 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:09.191515 7f9d7795c700  5 -- op tracker -- , seq: 747590, time: 2014-06-12 14:46:09.191509, event: sub_op_applied, op: osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.191526 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3049 (79'48,173'3049] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3048 active] sub_op_modify_applied on 0x274a76c0 op osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.191542 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3049 (79'48,173'3049] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3048 active] op_applied on primary on version 173'3049
2014-06-12 14:46:09.191548 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3049 (79'48,173'3049] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3048 active] op_applied on replica on version 173'3049
2014-06-12 14:46:09.191566 7f9d7795c700  5 -- op tracker -- , seq: 747590, time: 2014-06-12 14:46:09.191563, event: done, op: osd_sub_op(client.4316.0:147016 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3049 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.193794 7f9d57543700  5 -- op tracker -- , seq: 747591, time: 2014-06-12 14:46:09.193720, event: header_read, op: osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.193808 7f9d57543700  5 -- op tracker -- , seq: 747591, time: 2014-06-12 14:46:09.193721, event: throttled, op: osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.193824 7f9d57543700  5 -- op tracker -- , seq: 747591, time: 2014-06-12 14:46:09.193754, event: all_read, op: osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.193840 7f9d57543700  5 -- op tracker -- , seq: 747591, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.193843 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.193848 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:09.193852 7f9d57543700 15 osd.2 173 enqueue_op 0x2d17b400 prio 127 cost 4903 latency 0.000132 osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.193873 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.193878 7f9d633d7700 10 osd.2 173 dequeue_op 0x2d17b400 prio 127 cost 4903 latency 0.000158 osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.377( v 173'3049 (79'48,173'3049] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3048 active]
2014-06-12 14:46:09.193893 7f9d633d7700  5 -- op tracker -- , seq: 747591, time: 2014-06-12 14:46:09.193891, event: reached_pg, op: osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.193899 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3049 (79'48,173'3049] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3048 active] handle_message: 0x2d17b400
2014-06-12 14:46:09.193906 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3049 (79'48,173'3049] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3048 active] sub_op_modify trans af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 v 173'3050 (transaction) 174
2014-06-12 14:46:09.193915 7f9d633d7700  5 -- op tracker -- , seq: 747591, time: 2014-06-12 14:46:09.193913, event: started, op: osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.193924 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3049 (79'48,173'3049] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3048 active] append_log log((79'48,173'3049], crt=0'0) [173'3050 (173'3049) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147017 2014-06-12 08:45:11.573935]
2014-06-12 14:46:09.193940 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3050 (79'48,173'3050] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3048 active] add_log_entry 173'3050 (173'3049) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147017 2014-06-12 08:45:11.573935
2014-06-12 14:46:09.193956 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3050 (79'48,173'3050] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3048 active] append_log  adding 1 keys
2014-06-12 14:46:09.193965 7f9d633d7700 10 trim log((79'48,173'3050], crt=0'0) to 79'49
2014-06-12 14:46:09.193985 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3050, trimmed: 79'49
2014-06-12 14:46:09.194003 7f9d633d7700  5 -- op tracker -- , seq: 747591, time: 2014-06-12 14:46:09.194000, event: started, op: osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.194014 7f9d633d7700  5 queue_transactions existing osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.194018 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.194020 7f9d633d7700  5 queue_op 0x9fe26c0 seq 2746 osr(3.377 0x276d7e0) 6424 bytes   (queue has 1 ops and 6424 bytes)
2014-06-12 14:46:09.194033 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x51818c0 (1 active)
2014-06-12 14:46:09.194027 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.377( v 173'3050 (79'49,173'3050] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3048 active] do_sub_op osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.194042 7f9d79960700  5 _do_op 0x9fe26c0 seq 2746 osr(3.377 0x276d7e0)/0x276d7e0 start
2014-06-12 14:46:09.194044 7f9d633d7700 10 osd.2 173 dequeue_op 0x2d17b400 finish
2014-06-12 14:46:09.194048 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.194051 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.194050 7f9d79960700 10 _do_transaction on 0x23fbeef0
2014-06-12 14:46:09.194059 7f9d79960700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.194145 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.194167 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.194198 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.194351 7f9d79960700 15 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.194359 7f9d79960700 10 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1 r = 0
2014-06-12 14:46:09.194363 7f9d79960700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.194375 7f9d79960700 15 _write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 1880064~4096
2014-06-12 14:46:09.194470 7f9d79960700 10 lookup_strip_header done  cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.194476 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:09.194488 7f9d79960700 10 _generic_write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 1880064~4096 = 0
2014-06-12 14:46:09.194497 7f9d79960700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.194503 7f9d79960700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.194508 7f9d79960700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.194512 7f9d79960700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.194583 7f9d79960700 20 set_header setting 17368 cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 parent seq 0
2014-06-12 14:46:09.194781 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.194805 7f9d79960700 20 set_header setting 2216 cid meta oid 1fe1d86a/pglog_3.377/0//-1 parent seq 0
2014-06-12 14:46:09.194820 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:09.194970 7f9d79960700 10 _do_op 0x9fe26c0 seq 2746 r = 0, finisher 0x15698720 0
2014-06-12 14:46:09.194986 7f9d79960700 10 _finish_op 0x9fe26c0 seq 2746 osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.194995 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x51818c0 (0 active)
2014-06-12 14:46:09.195000 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.195007 7f9d7715b700  5 -- op tracker -- , seq: 747591, time: 2014-06-12 14:46:09.194995, event: commit_sent, op: osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.195025 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3050 (79'49,173'3050] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3048 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:09.195054 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:09.195095 7f9d7795c700  5 -- op tracker -- , seq: 747591, time: 2014-06-12 14:46:09.195091, event: sub_op_applied, op: osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.195103 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3050 (79'49,173'3050] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3049 active] sub_op_modify_applied on 0x23fbee00 op osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.195113 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3050 (79'49,173'3050] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3049 active] op_applied on primary on version 173'3050
2014-06-12 14:46:09.195119 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3050 (79'49,173'3050] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3049 active] op_applied on replica on version 173'3050
2014-06-12 14:46:09.195131 7f9d7795c700  5 -- op tracker -- , seq: 747591, time: 2014-06-12 14:46:09.195129, event: done, op: osd_sub_op(client.4316.0:147017 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3050 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.201150 7f9d57543700  5 -- op tracker -- , seq: 747592, time: 2014-06-12 14:46:09.200327, event: header_read, op: osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.201162 7f9d57543700  5 -- op tracker -- , seq: 747592, time: 2014-06-12 14:46:09.200329, event: throttled, op: osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.201171 7f9d57543700  5 -- op tracker -- , seq: 747592, time: 2014-06-12 14:46:09.201108, event: all_read, op: osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.201176 7f9d57543700  5 -- op tracker -- , seq: 747592, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.201182 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.201189 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:09.201199 7f9d57543700 15 osd.2 173 enqueue_op 0x20e88100 prio 127 cost 516903 latency 0.000872 osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.201227 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.201237 7f9d623d5700 10 osd.2 173 dequeue_op 0x20e88100 prio 127 cost 516903 latency 0.000909 osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.377( v 173'3050 (79'49,173'3050] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3049 active]
2014-06-12 14:46:09.201270 7f9d623d5700  5 -- op tracker -- , seq: 747592, time: 2014-06-12 14:46:09.201265, event: reached_pg, op: osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.201280 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3050 (79'49,173'3050] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3049 active] handle_message: 0x20e88100
2014-06-12 14:46:09.201313 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3050 (79'49,173'3050] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3049 active] sub_op_modify trans af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 v 173'3051 (transaction) 174
2014-06-12 14:46:09.201332 7f9d623d5700  5 -- op tracker -- , seq: 747592, time: 2014-06-12 14:46:09.201328, event: started, op: osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.201352 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3050 (79'49,173'3050] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3049 active] append_log log((79'49,173'3050], crt=0'0) [173'3051 (173'3050) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147018 2014-06-12 08:45:11.577895]
2014-06-12 14:46:09.201378 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3051 (79'49,173'3051] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3049 active] add_log_entry 173'3051 (173'3050) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147018 2014-06-12 08:45:11.577895
2014-06-12 14:46:09.201401 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3051 (79'49,173'3051] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3049 active] append_log  adding 1 keys
2014-06-12 14:46:09.201415 7f9d623d5700 10 trim log((79'49,173'3051], crt=0'0) to 79'50
2014-06-12 14:46:09.201446 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3051, trimmed: 79'50
2014-06-12 14:46:09.201479 7f9d623d5700  5 -- op tracker -- , seq: 747592, time: 2014-06-12 14:46:09.201474, event: started, op: osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.201492 7f9d623d5700  5 queue_transactions existing osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.201497 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.201499 7f9d623d5700  5 queue_op 0x7bd2fc0 seq 2747 osr(3.377 0x276d7e0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:09.201509 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x51818c0 (1 active)
2014-06-12 14:46:09.201517 7f9d8116f700  5 _do_op 0x7bd2fc0 seq 2747 osr(3.377 0x276d7e0)/0x276d7e0 start
2014-06-12 14:46:09.201506 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.377( v 173'3051 (79'50,173'3051] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3049 active] do_sub_op osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.201520 7f9d623d5700 10 osd.2 173 dequeue_op 0x20e88100 finish
2014-06-12 14:46:09.201524 7f9d8116f700 10 _do_transaction on 0x3317d0b0
2014-06-12 14:46:09.201524 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.201527 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.201530 7f9d8116f700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.201645 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.201663 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.201698 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.201709 7f9d8116f700 15 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.201714 7f9d8116f700 10 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1 r = 0
2014-06-12 14:46:09.201718 7f9d8116f700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.201726 7f9d8116f700 15 _write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 1884160~516096
2014-06-12 14:46:09.201819 7f9d8116f700 10 lookup_strip_header done  cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.201969 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:09.203330 7f9d8116f700 10 _generic_write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 1884160~516096 = 0
2014-06-12 14:46:09.203362 7f9d8116f700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.203373 7f9d8116f700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.203377 7f9d8116f700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.203381 7f9d8116f700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.203519 7f9d8116f700 20 set_header setting 17368 cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 parent seq 0
2014-06-12 14:46:09.203563 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.203582 7f9d8116f700 20 set_header setting 2216 cid meta oid 1fe1d86a/pglog_3.377/0//-1 parent seq 0
2014-06-12 14:46:09.203590 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:09.205241 7f9d8116f700 10 _do_op 0x7bd2fc0 seq 2747 r = 0, finisher 0x20aac200 0
2014-06-12 14:46:09.205253 7f9d8116f700 10 _finish_op 0x7bd2fc0 seq 2747 osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.205258 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x51818c0 (0 active)
2014-06-12 14:46:09.205264 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.205267 7f9d7715b700  5 -- op tracker -- , seq: 747592, time: 2014-06-12 14:46:09.205260, event: commit_sent, op: osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.205274 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3051 (79'50,173'3051] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3049 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:09.205292 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:09.205315 7f9d7795c700  5 -- op tracker -- , seq: 747592, time: 2014-06-12 14:46:09.205311, event: sub_op_applied, op: osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.205323 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3051 (79'50,173'3051] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3050 active] sub_op_modify_applied on 0x3317cfc0 op osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.205343 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3051 (79'50,173'3051] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3050 active] op_applied on primary on version 173'3051
2014-06-12 14:46:09.205349 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3051 (79'50,173'3051] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3050 active] op_applied on replica on version 173'3051
2014-06-12 14:46:09.205365 7f9d7795c700  5 -- op tracker -- , seq: 747592, time: 2014-06-12 14:46:09.205363, event: done, op: osd_sub_op(client.4316.0:147018 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3051 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.320727 7f9d57543700  5 -- op tracker -- , seq: 747593, time: 2014-06-12 14:46:09.319975, event: header_read, op: osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.320737 7f9d57543700  5 -- op tracker -- , seq: 747593, time: 2014-06-12 14:46:09.319977, event: throttled, op: osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.320750 7f9d57543700  5 -- op tracker -- , seq: 747593, time: 2014-06-12 14:46:09.320694, event: all_read, op: osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.320756 7f9d57543700  5 -- op tracker -- , seq: 747593, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.320759 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.320764 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:09.320770 7f9d57543700 15 osd.2 173 enqueue_op 0x34016b00 prio 127 cost 516903 latency 0.000795 osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.320793 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.320802 7f9d643d9700 10 osd.2 173 dequeue_op 0x34016b00 prio 127 cost 516903 latency 0.000827 osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.377( v 173'3051 (79'50,173'3051] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3050 active]
2014-06-12 14:46:09.320827 7f9d643d9700  5 -- op tracker -- , seq: 747593, time: 2014-06-12 14:46:09.320823, event: reached_pg, op: osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.320836 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3051 (79'50,173'3051] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3050 active] handle_message: 0x34016b00
2014-06-12 14:46:09.320852 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3051 (79'50,173'3051] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3050 active] sub_op_modify trans af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 v 173'3052 (transaction) 174
2014-06-12 14:46:09.320875 7f9d643d9700  5 -- op tracker -- , seq: 747593, time: 2014-06-12 14:46:09.320870, event: started, op: osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.320891 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3051 (79'50,173'3051] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3050 active] append_log log((79'50,173'3051], crt=0'0) [173'3052 (173'3051) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147020 2014-06-12 08:45:11.697757]
2014-06-12 14:46:09.320913 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3052 (79'50,173'3052] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3050 active] add_log_entry 173'3052 (173'3051) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147020 2014-06-12 08:45:11.697757
2014-06-12 14:46:09.320937 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3052 (79'50,173'3052] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3050 active] append_log  adding 1 keys
2014-06-12 14:46:09.320953 7f9d643d9700 10 trim log((79'50,173'3052], crt=0'0) to 79'51
2014-06-12 14:46:09.320978 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3052, trimmed: 79'51
2014-06-12 14:46:09.321010 7f9d643d9700  5 -- op tracker -- , seq: 747593, time: 2014-06-12 14:46:09.321006, event: started, op: osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.321020 7f9d643d9700  5 queue_transactions existing osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.321024 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.321025 7f9d643d9700  5 queue_op 0x6a8acc0 seq 2748 osr(3.377 0x276d7e0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:09.321034 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x51818c0 (1 active)
2014-06-12 14:46:09.321040 7f9d8096e700  5 _do_op 0x6a8acc0 seq 2748 osr(3.377 0x276d7e0)/0x276d7e0 start
2014-06-12 14:46:09.321043 7f9d8096e700 10 _do_transaction on 0x6176b70
2014-06-12 14:46:09.321032 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.377( v 173'3052 (79'51,173'3052] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3050 active] do_sub_op osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.321047 7f9d8096e700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.321048 7f9d643d9700 10 osd.2 173 dequeue_op 0x34016b00 finish
2014-06-12 14:46:09.321052 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.321054 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.321135 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.321155 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.321193 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.321213 7f9d8096e700 15 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.321219 7f9d8096e700 10 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1 r = 0
2014-06-12 14:46:09.321223 7f9d8096e700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.321335 7f9d8096e700 15 _write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 2400256~516096
2014-06-12 14:46:09.321433 7f9d8096e700 10 lookup_strip_header done  cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.321543 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:09.323067 7f9d8096e700 10 _generic_write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 2400256~516096 = 0
2014-06-12 14:46:09.323104 7f9d8096e700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.323114 7f9d8096e700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.323120 7f9d8096e700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.323125 7f9d8096e700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.323275 7f9d8096e700 20 set_header setting 17368 cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 parent seq 0
2014-06-12 14:46:09.323323 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.323334 7f9d8096e700 20 set_header setting 2216 cid meta oid 1fe1d86a/pglog_3.377/0//-1 parent seq 0
2014-06-12 14:46:09.323341 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:09.325756 7f9d8096e700 10 _do_op 0x6a8acc0 seq 2748 r = 0, finisher 0x73fa140 0
2014-06-12 14:46:09.325773 7f9d8096e700 10 _finish_op 0x6a8acc0 seq 2748 osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.325791 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x51818c0 (0 active)
2014-06-12 14:46:09.325796 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.325793 7f9d7715b700  5 -- op tracker -- , seq: 747593, time: 2014-06-12 14:46:09.325782, event: commit_sent, op: osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.325808 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3052 (79'51,173'3052] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3050 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:09.325835 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:09.325865 7f9d7795c700  5 -- op tracker -- , seq: 747593, time: 2014-06-12 14:46:09.325860, event: sub_op_applied, op: osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.325873 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3052 (79'51,173'3052] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3051 active] sub_op_modify_applied on 0x6176a80 op osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.325888 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3052 (79'51,173'3052] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3051 active] op_applied on primary on version 173'3052
2014-06-12 14:46:09.325897 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3052 (79'51,173'3052] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3051 active] op_applied on replica on version 173'3052
2014-06-12 14:46:09.325922 7f9d7795c700  5 -- op tracker -- , seq: 747593, time: 2014-06-12 14:46:09.325917, event: done, op: osd_sub_op(client.4316.0:147020 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3052 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.343641 7f9d57543700  5 -- op tracker -- , seq: 747594, time: 2014-06-12 14:46:09.342788, event: header_read, op: osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.343652 7f9d57543700  5 -- op tracker -- , seq: 747594, time: 2014-06-12 14:46:09.342790, event: throttled, op: osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.343667 7f9d57543700  5 -- op tracker -- , seq: 747594, time: 2014-06-12 14:46:09.343606, event: all_read, op: osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.343673 7f9d57543700  5 -- op tracker -- , seq: 747594, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.343676 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.343681 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:09.343688 7f9d57543700 15 osd.2 173 enqueue_op 0x2a902200 prio 127 cost 516903 latency 0.000900 osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.343716 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.343728 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2a902200 prio 127 cost 516903 latency 0.000939 osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.377( v 173'3052 (79'51,173'3052] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3051 active]
2014-06-12 14:46:09.343755 7f9d61bd4700  5 -- op tracker -- , seq: 747594, time: 2014-06-12 14:46:09.343750, event: reached_pg, op: osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.343767 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3052 (79'51,173'3052] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3051 active] handle_message: 0x2a902200
2014-06-12 14:46:09.343782 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3052 (79'51,173'3052] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3051 active] sub_op_modify trans af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 v 173'3053 (transaction) 174
2014-06-12 14:46:09.343801 7f9d61bd4700  5 -- op tracker -- , seq: 747594, time: 2014-06-12 14:46:09.343797, event: started, op: osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.343823 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3052 (79'51,173'3052] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3051 active] append_log log((79'51,173'3052], crt=0'0) [173'3053 (173'3052) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147021 2014-06-12 08:45:11.720276]
2014-06-12 14:46:09.343863 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3053 (79'51,173'3053] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3051 active] add_log_entry 173'3053 (173'3052) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147021 2014-06-12 08:45:11.720276
2014-06-12 14:46:09.343888 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3053 (79'51,173'3053] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3051 active] append_log  adding 1 keys
2014-06-12 14:46:09.343903 7f9d61bd4700 10 trim log((79'51,173'3053], crt=0'0) to 79'52
2014-06-12 14:46:09.343931 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3053, trimmed: 79'52
2014-06-12 14:46:09.343958 7f9d61bd4700  5 -- op tracker -- , seq: 747594, time: 2014-06-12 14:46:09.343953, event: started, op: osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.343971 7f9d61bd4700  5 queue_transactions existing osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.343975 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.343976 7f9d61bd4700  5 queue_op 0x30acefc0 seq 2749 osr(3.377 0x276d7e0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:09.343985 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x51818c0 (1 active)
2014-06-12 14:46:09.343990 7f9d8016d700  5 _do_op 0x30acefc0 seq 2749 osr(3.377 0x276d7e0)/0x276d7e0 start
2014-06-12 14:46:09.343993 7f9d8016d700 10 _do_transaction on 0x1da3ed30
2014-06-12 14:46:09.343983 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.377( v 173'3053 (79'52,173'3053] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3051 active] do_sub_op osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.343998 7f9d8016d700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.343998 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2a902200 finish
2014-06-12 14:46:09.344002 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.344004 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.344092 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.344114 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.344146 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.344165 7f9d8016d700 15 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.344172 7f9d8016d700 10 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1 r = 0
2014-06-12 14:46:09.344175 7f9d8016d700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.344352 7f9d8016d700 15 _write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 2916352~516096
2014-06-12 14:46:09.344450 7f9d8016d700 10 lookup_strip_header done  cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.344597 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:09.346208 7f9d8016d700 10 _generic_write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 2916352~516096 = 0
2014-06-12 14:46:09.346251 7f9d8016d700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.346267 7f9d8016d700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.346274 7f9d8016d700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.346282 7f9d8016d700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.346485 7f9d8016d700 20 set_header setting 17368 cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 parent seq 0
2014-06-12 14:46:09.346552 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.346567 7f9d8016d700 20 set_header setting 2216 cid meta oid 1fe1d86a/pglog_3.377/0//-1 parent seq 0
2014-06-12 14:46:09.346579 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:09.349631 7f9d8016d700 10 _do_op 0x30acefc0 seq 2749 r = 0, finisher 0x1ae4ece0 0
2014-06-12 14:46:09.349672 7f9d7715b700  5 -- op tracker -- , seq: 747594, time: 2014-06-12 14:46:09.349663, event: commit_sent, op: osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.349684 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3053 (79'52,173'3053] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3051 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:09.349720 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:09.349745 7f9d8016d700 10 _finish_op 0x30acefc0 seq 2749 osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.349767 7f9d7795c700  5 -- op tracker -- , seq: 747594, time: 2014-06-12 14:46:09.349761, event: sub_op_applied, op: osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.349776 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3053 (79'52,173'3053] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3052 active] sub_op_modify_applied on 0x1da3ec40 op osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.349791 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3053 (79'52,173'3053] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3052 active] op_applied on primary on version 173'3053
2014-06-12 14:46:09.349800 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3053 (79'52,173'3053] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3052 active] op_applied on replica on version 173'3053
2014-06-12 14:46:09.349832 7f9d8016d700  5 -- op tracker -- , seq: 747594, time: 2014-06-12 14:46:09.349822, event: done, op: osd_sub_op(client.4316.0:147021 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3053 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.349864 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x51818c0 (0 active)
2014-06-12 14:46:09.349870 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.355760 7f9d57543700  5 -- op tracker -- , seq: 747595, time: 2014-06-12 14:46:09.354934, event: header_read, op: osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.355772 7f9d57543700  5 -- op tracker -- , seq: 747595, time: 2014-06-12 14:46:09.354935, event: throttled, op: osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.355787 7f9d57543700  5 -- op tracker -- , seq: 747595, time: 2014-06-12 14:46:09.355722, event: all_read, op: osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.355792 7f9d57543700  5 -- op tracker -- , seq: 747595, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.355796 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.355801 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:09.355807 7f9d57543700 15 osd.2 173 enqueue_op 0x1e3eb200 prio 127 cost 516903 latency 0.000873 osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.355832 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.355843 7f9d663dd700 10 osd.2 173 dequeue_op 0x1e3eb200 prio 127 cost 516903 latency 0.000909 osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.377( v 173'3053 (79'52,173'3053] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3052 active]
2014-06-12 14:46:09.355874 7f9d663dd700  5 -- op tracker -- , seq: 747595, time: 2014-06-12 14:46:09.355870, event: reached_pg, op: osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.355885 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3053 (79'52,173'3053] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3052 active] handle_message: 0x1e3eb200
2014-06-12 14:46:09.355898 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3053 (79'52,173'3053] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3052 active] sub_op_modify trans af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 v 173'3054 (transaction) 174
2014-06-12 14:46:09.355914 7f9d663dd700  5 -- op tracker -- , seq: 747595, time: 2014-06-12 14:46:09.355910, event: started, op: osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.355929 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3053 (79'52,173'3053] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3052 active] append_log log((79'52,173'3053], crt=0'0) [173'3054 (173'3053) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147022 2014-06-12 08:45:11.732460]
2014-06-12 14:46:09.355953 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3054 (79'52,173'3054] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3052 active] add_log_entry 173'3054 (173'3053) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147022 2014-06-12 08:45:11.732460
2014-06-12 14:46:09.355980 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3054 (79'52,173'3054] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3052 active] append_log  adding 1 keys
2014-06-12 14:46:09.355994 7f9d663dd700 10 trim log((79'52,173'3054], crt=0'0) to 79'53
2014-06-12 14:46:09.356019 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3054, trimmed: 79'53
2014-06-12 14:46:09.356044 7f9d663dd700  5 -- op tracker -- , seq: 747595, time: 2014-06-12 14:46:09.356040, event: started, op: osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.356055 7f9d663dd700  5 queue_transactions existing osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.356058 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.356060 7f9d663dd700  5 queue_op 0x1697db60 seq 2750 osr(3.377 0x276d7e0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:09.356071 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x51818c0 (1 active)
2014-06-12 14:46:09.356066 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.377( v 173'3054 (79'53,173'3054] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3052 active] do_sub_op osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.356078 7f9d7d968700  5 _do_op 0x1697db60 seq 2750 osr(3.377 0x276d7e0)/0x276d7e0 start
2014-06-12 14:46:09.356080 7f9d663dd700 10 osd.2 173 dequeue_op 0x1e3eb200 finish
2014-06-12 14:46:09.356083 7f9d7d968700 10 _do_transaction on 0xc9b5270
2014-06-12 14:46:09.356083 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.356086 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.356090 7f9d7d968700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.356200 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.356224 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.356281 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.356312 7f9d7d968700 15 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.356321 7f9d7d968700 10 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1 r = 0
2014-06-12 14:46:09.356328 7f9d7d968700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.356436 7f9d7d968700 15 _write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 3432448~516096
2014-06-12 14:46:09.356570 7f9d7d968700 10 lookup_strip_header done  cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.356722 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:09.358049 7f9d7d968700 10 _generic_write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 3432448~516096 = 0
2014-06-12 14:46:09.358097 7f9d7d968700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.358111 7f9d7d968700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.358118 7f9d7d968700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.358126 7f9d7d968700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.358276 7f9d7d968700 20 set_header setting 17368 cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 parent seq 0
2014-06-12 14:46:09.358329 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.358345 7f9d7d968700 20 set_header setting 2216 cid meta oid 1fe1d86a/pglog_3.377/0//-1 parent seq 0
2014-06-12 14:46:09.358361 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:09.360301 7f9d7d968700 10 _do_op 0x1697db60 seq 2750 r = 0, finisher 0x5fbbc20 0
2014-06-12 14:46:09.360319 7f9d7d968700 10 _finish_op 0x1697db60 seq 2750 osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.360326 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x51818c0 (0 active)
2014-06-12 14:46:09.360335 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.360340 7f9d7715b700  5 -- op tracker -- , seq: 747595, time: 2014-06-12 14:46:09.360329, event: commit_sent, op: osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.360349 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3054 (79'53,173'3054] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3052 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:09.360380 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:09.360422 7f9d7795c700  5 -- op tracker -- , seq: 747595, time: 2014-06-12 14:46:09.360416, event: sub_op_applied, op: osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.360431 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3054 (79'53,173'3054] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3053 active] sub_op_modify_applied on 0xc9b5180 op osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.360452 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3054 (79'53,173'3054] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3053 active] op_applied on primary on version 173'3054
2014-06-12 14:46:09.360462 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3054 (79'53,173'3054] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3053 active] op_applied on replica on version 173'3054
2014-06-12 14:46:09.360486 7f9d7795c700  5 -- op tracker -- , seq: 747595, time: 2014-06-12 14:46:09.360481, event: done, op: osd_sub_op(client.4316.0:147022 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3054 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.366905 7f9d57543700  5 -- op tracker -- , seq: 747596, time: 2014-06-12 14:46:09.366312, event: header_read, op: osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.366919 7f9d57543700  5 -- op tracker -- , seq: 747596, time: 2014-06-12 14:46:09.366314, event: throttled, op: osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.366928 7f9d57543700  5 -- op tracker -- , seq: 747596, time: 2014-06-12 14:46:09.366871, event: all_read, op: osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.366936 7f9d57543700  5 -- op tracker -- , seq: 747596, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.366942 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.366949 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:09.366958 7f9d57543700 15 osd.2 173 enqueue_op 0x2d17a500 prio 127 cost 246567 latency 0.000646 osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.366983 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.366989 7f9d673df700 10 osd.2 173 dequeue_op 0x2d17a500 prio 127 cost 246567 latency 0.000677 osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.377( v 173'3054 (79'53,173'3054] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3053 active]
2014-06-12 14:46:09.367008 7f9d673df700  5 -- op tracker -- , seq: 747596, time: 2014-06-12 14:46:09.367005, event: reached_pg, op: osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.367016 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3054 (79'53,173'3054] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3053 active] handle_message: 0x2d17a500
2014-06-12 14:46:09.367030 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3054 (79'53,173'3054] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3053 active] sub_op_modify trans af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 v 173'3055 (transaction) 174
2014-06-12 14:46:09.367041 7f9d673df700  5 -- op tracker -- , seq: 747596, time: 2014-06-12 14:46:09.367038, event: started, op: osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.367052 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3054 (79'53,173'3054] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3053 active] append_log log((79'53,173'3054], crt=0'0) [173'3055 (173'3054) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147023 2014-06-12 08:45:11.744614]
2014-06-12 14:46:09.367068 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3055 (79'53,173'3055] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3053 active] add_log_entry 173'3055 (173'3054) modify   af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 by client.4316.0:147023 2014-06-12 08:45:11.744614
2014-06-12 14:46:09.367083 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3055 (79'53,173'3055] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3053 active] append_log  adding 1 keys
2014-06-12 14:46:09.367092 7f9d673df700 10 trim log((79'53,173'3055], crt=0'0) to 79'54
2014-06-12 14:46:09.367109 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3055, trimmed: 79'54
2014-06-12 14:46:09.367127 7f9d673df700  5 -- op tracker -- , seq: 747596, time: 2014-06-12 14:46:09.367123, event: started, op: osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.367134 7f9d673df700  5 queue_transactions existing osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.367137 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.367138 7f9d673df700  5 queue_op 0x16064e40 seq 2751 osr(3.377 0x276d7e0) 248088 bytes   (queue has 1 ops and 248088 bytes)
2014-06-12 14:46:09.367144 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x51818c0 (1 active)
2014-06-12 14:46:09.367142 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.377( v 173'3055 (79'54,173'3055] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3053 active] do_sub_op osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.367151 7f9d673df700 10 osd.2 173 dequeue_op 0x2d17a500 finish
2014-06-12 14:46:09.367152 7f9d7915f700  5 _do_op 0x16064e40 seq 2751 osr(3.377 0x276d7e0)/0x276d7e0 start
2014-06-12 14:46:09.367154 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.367155 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.367156 7f9d7915f700 10 _do_transaction on 0x128c80f0
2014-06-12 14:46:09.367160 7f9d7915f700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.367245 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.367264 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.367297 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.367318 7f9d7915f700 15 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.367325 7f9d7915f700 10 _omap_rmkeys meta/1fe1d86a/pglog_3.377/0//-1 r = 0
2014-06-12 14:46:09.367329 7f9d7915f700 15 _omap_setkeys meta/1fe1d86a/pglog_3.377/0//-1
2014-06-12 14:46:09.367340 7f9d7915f700 15 _write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 3948544~245760
2014-06-12 14:46:09.367437 7f9d7915f700 10 lookup_strip_header done  cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.367447 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:09.368145 7f9d7915f700 10 _generic_write 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 3948544~245760 = 0
2014-06-12 14:46:09.368171 7f9d7915f700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.368184 7f9d7915f700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.368189 7f9d7915f700 15 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3
2014-06-12 14:46:09.368194 7f9d7915f700 10 _setattrs 3.377_head/af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 = 0
2014-06-12 14:46:09.368297 7f9d7915f700 20 set_header setting 17368 cid 3.377_head oid af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 parent seq 0
2014-06-12 14:46:09.368342 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.368351 7f9d7915f700 20 set_header setting 2216 cid meta oid 1fe1d86a/pglog_3.377/0//-1 parent seq 0
2014-06-12 14:46:09.368359 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:09.369267 7f9d7915f700 10 _do_op 0x16064e40 seq 2751 r = 0, finisher 0x90ee5e0 0
2014-06-12 14:46:09.369277 7f9d7915f700 10 _finish_op 0x16064e40 seq 2751 osr(3.377 0x276d7e0)/0x276d7e0
2014-06-12 14:46:09.369282 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x51818c0 (0 active)
2014-06-12 14:46:09.369285 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.369294 7f9d7715b700  5 -- op tracker -- , seq: 747596, time: 2014-06-12 14:46:09.369285, event: commit_sent, op: osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.369303 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3055 (79'54,173'3055] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3053 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:09.369331 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:09.369365 7f9d7795c700  5 -- op tracker -- , seq: 747596, time: 2014-06-12 14:46:09.369360, event: sub_op_applied, op: osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.369373 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3055 (79'54,173'3055] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3054 active] sub_op_modify_applied on 0x128c8000 op osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.369384 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3055 (79'54,173'3055] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3054 active] op_applied on primary on version 173'3055
2014-06-12 14:46:09.369390 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.377( v 173'3055 (79'54,173'3055] local-les=145 n=316 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'3054 active] op_applied on replica on version 173'3055
2014-06-12 14:46:09.369429 7f9d7795c700  5 -- op tracker -- , seq: 747596, time: 2014-06-12 14:46:09.369425, event: done, op: osd_sub_op(client.4316.0:147023 3.377 af426377/rbd_data.10b36b8b4567.0000000000001115/head//3 [] v 173'3055 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.459379 7f9d5a16f700  5 -- op tracker -- , seq: 747597, time: 2014-06-12 14:46:09.457243, event: header_read, op: osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.459397 7f9d5a16f700  5 -- op tracker -- , seq: 747597, time: 2014-06-12 14:46:09.457246, event: throttled, op: osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.459408 7f9d5a16f700  5 -- op tracker -- , seq: 747597, time: 2014-06-12 14:46:09.459317, event: all_read, op: osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.459451 7f9d5a16f700  5 -- op tracker -- , seq: 747597, time: 0.000000, event: dispatched, op: osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.459460 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.459468 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:09.459483 7f9d5a16f700 15 osd.2 173 enqueue_op 0x9fada00 prio 127 cost 516903 latency 0.002240 osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.460156 7f9d5a16f700  5 -- op tracker -- , seq: 747598, time: 2014-06-12 14:46:09.459530, event: header_read, op: osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.460170 7f9d5a16f700  5 -- op tracker -- , seq: 747598, time: 2014-06-12 14:46:09.459532, event: throttled, op: osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.460179 7f9d5a16f700  5 -- op tracker -- , seq: 747598, time: 2014-06-12 14:46:09.460126, event: all_read, op: osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.460187 7f9d5a16f700  5 -- op tracker -- , seq: 747598, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.460191 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.460196 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:09.460203 7f9d5a16f700 15 osd.2 173 enqueue_op 0x9fac500 prio 127 cost 516903 latency 0.000673 osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.460233 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.460245 7f9d693e3700 10 osd.2 173 dequeue_op 0x9fada00 prio 127 cost 516903 latency 0.003002 osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2717 (0'0,173'2717] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active]
2014-06-12 14:46:09.460275 7f9d693e3700  5 -- op tracker -- , seq: 747597, time: 2014-06-12 14:46:09.460270, event: reached_pg, op: osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.460285 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2717 (0'0,173'2717] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] handle_message: 0x9fada00
2014-06-12 14:46:09.460298 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2717 (0'0,173'2717] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] sub_op_modify trans e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 v 173'2718 (transaction) 174
2014-06-12 14:46:09.460315 7f9d693e3700  5 -- op tracker -- , seq: 747597, time: 2014-06-12 14:46:09.460311, event: started, op: osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.460332 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2717 (0'0,173'2717] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] append_log log((0'0,173'2717], crt=0'0) [173'2718 (173'2717) modify   e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 by client.4285.0:143305 2014-06-12 08:42:54.498250]
2014-06-12 14:46:09.460359 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2718 (0'0,173'2718] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] add_log_entry 173'2718 (173'2717) modify   e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 by client.4285.0:143305 2014-06-12 08:42:54.498250
2014-06-12 14:46:09.460379 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2718 (0'0,173'2718] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] append_log  adding 1 keys
2014-06-12 14:46:09.460412 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2718, trimmed: 
2014-06-12 14:46:09.460437 7f9d693e3700  5 -- op tracker -- , seq: 747597, time: 2014-06-12 14:46:09.460432, event: started, op: osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.460449 7f9d693e3700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:09.460453 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.460455 7f9d693e3700  5 queue_op 0x7c6a660 seq 2465 osr(3.21c 0x276d020) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:09.460463 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2718 (0'0,173'2718] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] do_sub_op osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.460476 7f9d693e3700 10 osd.2 173 dequeue_op 0x9fada00 finish
2014-06-12 14:46:09.460479 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.460484 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.460487 7f9d693e3700 10 osd.2 173 dequeue_op 0x9fac500 prio 127 cost 516903 latency 0.000957 osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2654 (0'0,173'2654] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2653 active]
2014-06-12 14:46:09.460505 7f9d693e3700  5 -- op tracker -- , seq: 747598, time: 2014-06-12 14:46:09.460501, event: reached_pg, op: osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.460512 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2654 (0'0,173'2654] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2653 active] handle_message: 0x9fac500
2014-06-12 14:46:09.460521 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2654 (0'0,173'2654] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2653 active] sub_op_modify trans 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 v 173'2655 (transaction) 174
2014-06-12 14:46:09.460537 7f9d693e3700  5 -- op tracker -- , seq: 747598, time: 2014-06-12 14:46:09.460533, event: started, op: osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.460553 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2654 (0'0,173'2654] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2653 active] append_log log((0'0,173'2654], crt=0'0) [173'2655 (173'747) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141062 2014-06-12 08:43:32.902207]
2014-06-12 14:46:09.460575 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2655 (0'0,173'2655] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2653 active] add_log_entry 173'2655 (173'747) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141062 2014-06-12 08:43:32.902207
2014-06-12 14:46:09.460602 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2655 (0'0,173'2655] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2653 active] append_log  adding 1 keys
2014-06-12 14:46:09.460634 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2655, trimmed: 
2014-06-12 14:46:09.460666 7f9d693e3700  5 -- op tracker -- , seq: 747598, time: 2014-06-12 14:46:09.460662, event: started, op: osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.460676 7f9d693e3700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:09.460679 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.460680 7f9d693e3700  5 queue_op 0x6cea7e0 seq 2465 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:09.460687 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2655 (0'0,173'2655] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2653 active] do_sub_op osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.460701 7f9d693e3700 10 osd.2 173 dequeue_op 0x9fac500 finish
2014-06-12 14:46:09.460704 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.460707 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.460718 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.460722 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (1 active)
2014-06-12 14:46:09.460731 7f9d83173700  5 _do_op 0x7c6a660 seq 2465 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:46:09.460737 7f9d83173700 10 _do_transaction on 0x65f9eb0
2014-06-12 14:46:09.460743 7f9d83173700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:09.460896 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:09.460919 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.460968 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.460991 7f9d83173700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:09.460998 7f9d83173700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:46:09.461004 7f9d83173700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:09.461137 7f9d83173700 15 _write 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 3481600~516096
2014-06-12 14:46:09.461310 7f9d83173700 10 lookup_strip_header done  cid 3.21c_head oid e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:46:09.461431 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:09.463640 7f9d83173700 10 _generic_write 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 3481600~516096 = 0
2014-06-12 14:46:09.463710 7f9d83173700 15 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:46:09.463729 7f9d83173700 10 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 = 0
2014-06-12 14:46:09.463737 7f9d83173700 15 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:46:09.463746 7f9d83173700 10 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 = 0
2014-06-12 14:46:09.463951 7f9d83173700 20 set_header setting 17494 cid 3.21c_head oid e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 parent seq 0
2014-06-12 14:46:09.464001 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.464011 7f9d83173700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:46:09.464017 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:09.465695 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (2 active)
2014-06-12 14:46:09.465729 7f9d7c165700  5 _do_op 0x6cea7e0 seq 2465 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:09.465751 7f9d7c165700 10 _do_transaction on 0x5c99cf0
2014-06-12 14:46:09.465763 7f9d7c165700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:09.470611 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:09.470673 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 13.82 >= max 0.5 = no, load too high
2014-06-12 14:46:09.470688 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:09.470691 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92617 < max (604800 seconds)
2014-06-12 14:46:09.470726 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:09.470730 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:09.470731 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:09.470734 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 33; oldest is 37.580794 seconds old
2014-06-12 14:46:09.470798 7f9d84175700  0 log [WRN] : 12 slow requests, 6 included below; oldest blocked for > 37.580794 secs
2014-06-12 14:46:09.470803 7f9d84175700  0 log [WRN] : slow request 30.816130 seconds old, received at 2014-06-12 14:45:38.654604: osd_op(client.4309.0:141523 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~315392] 3.f029e408 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:09.470822 7f9d84175700  0 log [WRN] : slow request 30.788963 seconds old, received at 2014-06-12 14:45:38.681771: osd_op(client.4309.0:141524 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 315392~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:09.470826 7f9d84175700  0 log [WRN] : slow request 30.724634 seconds old, received at 2014-06-12 14:45:38.746100: osd_op(client.4309.0:141525 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 831488~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:09.470830 7f9d84175700  0 log [WRN] : slow request 30.645154 seconds old, received at 2014-06-12 14:45:38.825580: osd_op(client.4309.0:141526 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 1347584~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:09.470833 7f9d84175700  0 log [WRN] : slow request 30.340420 seconds old, received at 2014-06-12 14:45:39.130314: osd_op(client.4303.0:144622 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~28672] 3.892731eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:09.472607 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:09.472652 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.473630 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:09.476065 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.476150 7f9d7c165700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:09.476164 7f9d7c165700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:09.476171 7f9d7c165700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:09.476427 7f9d7c165700 15 _write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 0~516096
2014-06-12 14:46:09.476610 7f9d7c165700 10 lookup_strip_header done  cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:09.476914 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:09.479302 7f9d83173700 10 _do_op 0x7c6a660 seq 2465 r = 0, finisher 0x7759180 0
2014-06-12 14:46:09.479330 7f9d83173700 10 _finish_op 0x7c6a660 seq 2465 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:09.479361 7f9d7795c700  5 -- op tracker -- , seq: 747597, time: 2014-06-12 14:46:09.479350, event: sub_op_applied, op: osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.479375 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2718 (0'0,173'2718] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] sub_op_modify_applied on 0x65f9dc0 op osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.479415 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:09.479437 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2718 (0'0,173'2718] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] op_applied on primary on version 173'2718
2014-06-12 14:46:09.479448 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2718 (0'0,173'2718] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] op_applied on replica on version 173'2718
2014-06-12 14:46:09.479461 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (1 active)
2014-06-12 14:46:09.479472 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.479817 7f9d7c165700 10 _generic_write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 0~516096 = 0
2014-06-12 14:46:09.479869 7f9d7c165700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:09.479885 7f9d7c165700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:09.479891 7f9d7c165700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:09.479899 7f9d7c165700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:09.480137 7f9d7c165700 20 set_header setting 17215 cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 parent seq 0
2014-06-12 14:46:09.480242 7f9d7c165700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:09.480256 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.480263 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:09.481175 7f9d7715b700  5 -- op tracker -- , seq: 747597, time: 2014-06-12 14:46:09.481163, event: commit_sent, op: osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.481190 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2718 (0'0,173'2718] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] sub_op_modify_commit on op osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:09.481216 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:09.481245 7f9d7715b700  5 -- op tracker -- , seq: 747597, time: 2014-06-12 14:46:09.481241, event: done, op: osd_sub_op(client.4285.0:143305 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.483965 7f9d7c165700 10 _do_op 0x6cea7e0 seq 2465 r = 0, finisher 0x5462120 0
2014-06-12 14:46:09.484006 7f9d7715b700  5 -- op tracker -- , seq: 747598, time: 2014-06-12 14:46:09.483997, event: commit_sent, op: osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.484017 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2655 (0'0,173'2655] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2653 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:09.484042 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:09.484067 7f9d7c165700 10 _finish_op 0x6cea7e0 seq 2465 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:09.484087 7f9d7795c700  5 -- op tracker -- , seq: 747598, time: 2014-06-12 14:46:09.484082, event: sub_op_applied, op: osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.484095 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2655 (0'0,173'2655] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2654 active] sub_op_modify_applied on 0x5c99c00 op osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.484110 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2655 (0'0,173'2655] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2654 active] op_applied on primary on version 173'2655
2014-06-12 14:46:09.484120 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2655 (0'0,173'2655] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2654 active] op_applied on replica on version 173'2655
2014-06-12 14:46:09.484150 7f9d7c165700  5 -- op tracker -- , seq: 747598, time: 2014-06-12 14:46:09.484140, event: done, op: osd_sub_op(client.4284.0:141062 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.484177 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:46:09.484180 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.492986 7f9d4ec14700  5 -- op tracker -- , seq: 747599, time: 2014-06-12 14:46:09.487365, event: header_read, op: osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.493000 7f9d4ec14700  5 -- op tracker -- , seq: 747599, time: 2014-06-12 14:46:09.487371, event: throttled, op: osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.493010 7f9d4ec14700  5 -- op tracker -- , seq: 747599, time: 2014-06-12 14:46:09.492946, event: all_read, op: osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.493019 7f9d4ec14700  5 -- op tracker -- , seq: 747599, time: 0.000000, event: dispatched, op: osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.493027 7f9d4ec14700 20 osd.2 173 should_share_map client.4282 172.16.96.220:0/1241359 173
2014-06-12 14:46:09.493051 7f9d4ec14700 15 osd.2 173 enqueue_op 0x22b3bc00 prio 63 cost 520192 latency 0.005686 osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:09.497843 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.497857 7f9d62bd6700 10 osd.2 173 dequeue_op 0x22b3bc00 prio 63 cost 520192 latency 0.010491 osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean]
2014-06-12 14:46:09.497890 7f9d62bd6700  5 -- op tracker -- , seq: 747599, time: 2014-06-12 14:46:09.497883, event: reached_pg, op: osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.497901 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:09.497916 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] handle_message: 0x22b3bc00
2014-06-12 14:46:09.497926 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] do_op osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:09.497953 7f9d62bd6700 15 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 '_'
2014-06-12 14:46:09.523770 7f9d58c5a700  5 -- op tracker -- , seq: 747600, time: 2014-06-12 14:46:09.519943, event: header_read, op: osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.523785 7f9d58c5a700  5 -- op tracker -- , seq: 747600, time: 2014-06-12 14:46:09.519946, event: throttled, op: osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.523794 7f9d58c5a700  5 -- op tracker -- , seq: 747600, time: 2014-06-12 14:46:09.523720, event: all_read, op: osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.523808 7f9d58c5a700  5 -- op tracker -- , seq: 747600, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.523815 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.523823 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:09.523835 7f9d58c5a700 15 osd.2 173 enqueue_op 0x2b014b00 prio 127 cost 1397543 latency 0.003892 osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.523863 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:09.523872 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2b014b00 prio 127 cost 1397543 latency 0.003928 osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3156 (79'155,173'3156] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3155 active]
2014-06-12 14:46:09.523893 7f9d6b3e7700  5 -- op tracker -- , seq: 747600, time: 2014-06-12 14:46:09.523890, event: reached_pg, op: osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.523901 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3156 (79'155,173'3156] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3155 active] handle_message: 0x2b014b00
2014-06-12 14:46:09.523910 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3156 (79'155,173'3156] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3155 active] sub_op_modify trans a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 v 173'3157 (transaction) 174
2014-06-12 14:46:09.523922 7f9d6b3e7700  5 -- op tracker -- , seq: 747600, time: 2014-06-12 14:46:09.523920, event: started, op: osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.523936 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3156 (79'155,173'3156] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3155 active] append_log log((79'155,173'3156], crt=0'0) [173'3157 (173'895) modify   a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 by client.4294.0:144014 2014-06-12 08:45:11.868436]
2014-06-12 14:46:09.523954 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3157 (79'155,173'3157] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3155 active] add_log_entry 173'3157 (173'895) modify   a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 by client.4294.0:144014 2014-06-12 08:45:11.868436
2014-06-12 14:46:09.523976 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3157 (79'155,173'3157] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3155 active] append_log  adding 1 keys
2014-06-12 14:46:09.523986 7f9d6b3e7700 10 trim log((79'155,173'3157], crt=0'0) to 79'156
2014-06-12 14:46:09.524007 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3157, trimmed: 79'156
2014-06-12 14:46:09.524026 7f9d6b3e7700  5 -- op tracker -- , seq: 747600, time: 2014-06-12 14:46:09.524022, event: started, op: osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.524036 7f9d6b3e7700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:09.524041 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.524042 7f9d6b3e7700  5 queue_op 0x70c5140 seq 2852 osr(3.36c 0x276d400) 1399064 bytes   (queue has 1 ops and 1399064 bytes)
2014-06-12 14:46:09.524049 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3157 (79'156,173'3157] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3155 active] do_sub_op osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.524061 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2b014b00 finish
2014-06-12 14:46:09.524063 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:09.524056 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (1 active)
2014-06-12 14:46:09.524065 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.524070 7f9d82972700  5 _do_op 0x70c5140 seq 2852 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:09.524076 7f9d82972700 10 _do_transaction on 0x2963430
2014-06-12 14:46:09.524082 7f9d82972700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:09.524237 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:09.524280 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.524332 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.524356 7f9d82972700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:09.524367 7f9d82972700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:09.524374 7f9d82972700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:09.524521 7f9d82972700 15 _write 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 0~1396736
2014-06-12 14:46:09.524734 7f9d82972700 10 lookup_strip_header done  cid 3.36c_head oid a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3
2014-06-12 14:46:09.524864 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:09.529828 7f9d82972700 10 _generic_write 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 0~1396736 = 0
2014-06-12 14:46:09.529921 7f9d82972700 15 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3
2014-06-12 14:46:09.529941 7f9d82972700 10 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 = 0
2014-06-12 14:46:09.529946 7f9d82972700 15 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3
2014-06-12 14:46:09.529951 7f9d82972700 10 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 = 0
2014-06-12 14:46:09.530203 7f9d82972700 20 set_header setting 17739 cid 3.36c_head oid a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 parent seq 0
2014-06-12 14:46:09.530303 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.530317 7f9d82972700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:09.530324 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:09.536260 7f9d82972700 10 _do_op 0x70c5140 seq 2852 r = 0, finisher 0x63cf240 0
2014-06-12 14:46:09.536295 7f9d82972700 10 _finish_op 0x70c5140 seq 2852 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:09.536303 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (0 active)
2014-06-12 14:46:09.536313 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.536317 7f9d7715b700  5 -- op tracker -- , seq: 747600, time: 2014-06-12 14:46:09.536305, event: commit_sent, op: osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.536325 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3157 (79'156,173'3157] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3155 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:09.536347 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:09.536382 7f9d7795c700  5 -- op tracker -- , seq: 747600, time: 2014-06-12 14:46:09.536375, event: sub_op_applied, op: osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.536393 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3157 (79'156,173'3157] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active] sub_op_modify_applied on 0x2963340 op osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.536416 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3157 (79'156,173'3157] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active] op_applied on primary on version 173'3157
2014-06-12 14:46:09.536429 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3157 (79'156,173'3157] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active] op_applied on replica on version 173'3157
2014-06-12 14:46:09.536466 7f9d7795c700  5 -- op tracker -- , seq: 747600, time: 2014-06-12 14:46:09.536461, event: done, op: osd_sub_op(client.4294.0:144014 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3157 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.543267 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:09.543332 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:09.593631 7f9d62bd6700 10 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 '_' = 0
2014-06-12 14:46:09.593663 7f9d62bd6700 15 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 'snapset'
2014-06-12 14:46:09.593818 7f9d62bd6700 10 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 'snapset' = 0
2014-06-12 14:46:09.593828 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] populate_obc_watchers 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.593845 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.593852 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] get_object_context: creating obc from disk: 0x16c7d080
2014-06-12 14:46:09.593858 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] get_object_context: 0x16c7d080 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 rwstate(none n=0 w=0) oi: 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3(173'3149 client.4282.0:145272 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3149) ssc: 0x283d37a0 snapset: 0=[]:[]+head
2014-06-12 14:46:09.593869 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] find_object_context 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 @head oi=71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3(173'3149 client.4282.0:145272 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3149)
2014-06-12 14:46:09.593893 7f9d62bd6700  5 -- op tracker -- , seq: 747599, time: 2014-06-12 14:46:09.593886, event: started, op: osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.593904 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] execute_ctx 0x589be00
2014-06-12 14:46:09.593915 7f9d62bd6700  5 -- op tracker -- , seq: 747599, time: 2014-06-12 14:46:09.593912, event: started, op: osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.593920 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] do_op 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] ov 173'3149 av 173'3150 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:09.593929 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] do_osd_op 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192]
2014-06-12 14:46:09.593936 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:09.593944 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] do_osd_op  write 45056~520192
2014-06-12 14:46:09.593955 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] do_osd_op_effects on session 0x62d5000
2014-06-12 14:46:09.593962 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] make_writeable 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 snapset=0x283d37d8  snapc=0=[]
2014-06-12 14:46:09.593969 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] make_writeable 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:09.593975 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] finish_ctx 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 0x589be00 op modify  
2014-06-12 14:46:09.593983 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean]  set mtime to 2014-06-12 08:45:11.867380
2014-06-12 14:46:09.594012 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean]  final snapset 0=[]:[]+head in 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.594041 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean]  zeroing write result code 0
2014-06-12 14:46:09.594052 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] calc_trim_to 79'148 -> 79'149
2014-06-12 14:46:09.594064 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] new_repop rep_tid 236212 on osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:09.594081 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] issue_repop rep_tid 236212 o 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.594107 7f9d62bd6700  5 -- op tracker -- , seq: 747599, time: 2014-06-12 14:46:09.594101, event: waiting for subops from 20, op: osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.594129 7f9d62bd6700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:09.594152 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3149 (79'148,173'3149] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] append_log log((79'148,173'3149], crt=0'0) [173'3150 (173'3149) modify   71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 by client.4282.0:145273 2014-06-12 08:45:11.867380]
2014-06-12 14:46:09.594177 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'148,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3149 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] add_log_entry 173'3150 (173'3149) modify   71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 by client.4282.0:145273 2014-06-12 08:45:11.867380
2014-06-12 14:46:09.594202 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'148,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3149 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] append_log  adding 1 keys
2014-06-12 14:46:09.594215 7f9d62bd6700 10 trim log((79'148,173'3150], crt=0'0) to 79'149
2014-06-12 14:46:09.594242 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3150, trimmed: 79'149
2014-06-12 14:46:09.594266 7f9d62bd6700  5 queue_transactions existing osr(3.63b 0x276dc70)/0x276dc70
2014-06-12 14:46:09.594270 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.594272 7f9d62bd6700  5 queue_op 0x1b90fec0 seq 2853 osr(3.63b 0x276dc70) 522520 bytes   (queue has 1 ops and 522520 bytes)
2014-06-12 14:46:09.594284 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5421440 (1 active)
2014-06-12 14:46:09.594290 7f9d7c966700  5 _do_op 0x1b90fec0 seq 2853 osr(3.63b 0x276dc70)/0x276dc70 start
2014-06-12 14:46:09.594297 7f9d7c966700 10 _do_transaction on 0x384e3080
2014-06-12 14:46:09.594280 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3149 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] eval_repop repgather(0x2a12e900 173'3150 rep_tid=236212 committed?=0 applied?=0 lock=0 op=osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:09.594301 7f9d7c966700 15 _omap_setkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:09.594301 7f9d62bd6700 10 osd.2 173 dequeue_op 0x22b3bc00 finish
2014-06-12 14:46:09.594306 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.594311 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.594367 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:09.594387 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.594418 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.594564 7f9d7c966700 15 _omap_rmkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:09.594571 7f9d7c966700 10 _omap_rmkeys meta/1fe0c16b/pglog_3.63b/0//-1 r = 0
2014-06-12 14:46:09.594575 7f9d7c966700 15 _omap_setkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:09.594591 7f9d7c966700 15 _write 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 45056~520192
2014-06-12 14:46:09.594674 7f9d7c966700 10 lookup_strip_header done  cid 3.63b_head oid 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.594819 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:09.596131 7f9d7c966700 10 _generic_write 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 45056~520192 = 0
2014-06-12 14:46:09.596173 7f9d7c966700 15 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.596183 7f9d7c966700 10 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 = 0
2014-06-12 14:46:09.596187 7f9d7c966700 15 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.596192 7f9d7c966700 10 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 = 0
2014-06-12 14:46:09.596322 7f9d7c966700 20 set_header setting 18103 cid 3.63b_head oid 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 parent seq 0
2014-06-12 14:46:09.596363 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.596373 7f9d7c966700 20 set_header setting 2110 cid meta oid 1fe0c16b/pglog_3.63b/0//-1 parent seq 0
2014-06-12 14:46:09.596380 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:09.598182 7f9d7c966700 10 _do_op 0x1b90fec0 seq 2853 r = 0, finisher 0x65fa160 0x217ef0c0
2014-06-12 14:46:09.598198 7f9d7c966700 10 _finish_op 0x1b90fec0 seq 2853 osr(3.63b 0x276dc70)/0x276dc70
2014-06-12 14:46:09.598205 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5421440 (0 active)
2014-06-12 14:46:09.598211 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.598203 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3149 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] op_commit: 236212
2014-06-12 14:46:09.598231 7f9d7715b700  5 -- op tracker -- , seq: 747599, time: 2014-06-12 14:46:09.598223, event: op_commit, op: osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.598243 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3149 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] op_applied: 236212
2014-06-12 14:46:09.598261 7f9d7795c700  5 -- op tracker -- , seq: 747599, time: 2014-06-12 14:46:09.598256, event: op_applied, op: osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.598267 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3149 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] op_applied on primary on version 173'3150
2014-06-12 14:46:09.600057 7f9d5703e700  5 -- op tracker -- , seq: 747601, time: 2014-06-12 14:46:09.600017, event: header_read, op: osd_sub_op_reply(client.4282.0:145273 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.600065 7f9d5703e700  5 -- op tracker -- , seq: 747601, time: 2014-06-12 14:46:09.600019, event: throttled, op: osd_sub_op_reply(client.4282.0:145273 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.600080 7f9d5703e700  5 -- op tracker -- , seq: 747601, time: 2014-06-12 14:46:09.600030, event: all_read, op: osd_sub_op_reply(client.4282.0:145273 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.600094 7f9d5703e700  5 -- op tracker -- , seq: 747601, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4282.0:145273 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.600098 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4282.0:145273 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:09.600102 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:09.600109 7f9d5703e700 15 osd.2 173 enqueue_op 0x2b015d00 prio 196 cost 0 latency 0.000091 osd_sub_op_reply(client.4282.0:145273 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:09.600136 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.600144 7f9d64bda700 10 osd.2 173 dequeue_op 0x2b015d00 prio 196 cost 0 latency 0.000127 osd_sub_op_reply(client.4282.0:145273 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2 pg pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3149 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean]
2014-06-12 14:46:09.600167 7f9d64bda700  5 -- op tracker -- , seq: 747601, time: 2014-06-12 14:46:09.600163, event: reached_pg, op: osd_sub_op_reply(client.4282.0:145273 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.600176 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3149 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] handle_message: 0x2b015d00
2014-06-12 14:46:09.600194 7f9d64bda700  5 -- op tracker -- , seq: 747601, time: 2014-06-12 14:46:09.600191, event: started, op: osd_sub_op_reply(client.4282.0:145273 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.600201 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3149 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] sub_op_modify_reply: tid 236212 op  ack_type 4 from 20
2014-06-12 14:46:09.600220 7f9d64bda700  5 -- op tracker -- , seq: 747599, time: 2014-06-12 14:46:09.600214, event: sub_op_commit_rec, op: osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.600232 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3149 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] repop_all_applied: repop tid 236212 all applied 
2014-06-12 14:46:09.600244 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3149 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] eval_repop repgather(0x2a12e900 173'3150 rep_tid=236212 committed?=0 applied?=1 lock=0 op=osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:09.600263 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3149 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean]  sending ack on repgather(0x2a12e900 173'3150 rep_tid=236212 committed?=0 applied?=1 lock=0 op=osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) 0x2376c80
2014-06-12 14:46:09.600298 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3149 crt=0'0 lcod 173'3148 mlcod 173'3148 active+clean] repop_all_committed: repop tid 236212 all committed 
2014-06-12 14:46:09.600310 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3148 active+clean] eval_repop repgather(0x2a12e900 173'3150 rep_tid=236212 committed?=1 applied?=1 lock=0 op=osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:09.600328 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3148 active+clean] log_op_stats osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 inb 520471 outb 0 rlat 0.112931 lat 0.112962
2014-06-12 14:46:09.600348 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3148 active+clean] publish_stats_to_osd 173:4111
2014-06-12 14:46:09.600362 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3148 active+clean]  sending commit on repgather(0x2a12e900 173'3150 rep_tid=236212 committed?=1 applied?=1 lock=0 op=osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) 0xb954280
2014-06-12 14:46:09.600395 7f9d64bda700  5 -- op tracker -- , seq: 747599, time: 2014-06-12 14:46:09.600390, event: commit_sent, op: osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.600406 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3148 active+clean]  requeue_ops 
2014-06-12 14:46:09.600417 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean]  removing repgather(0x2a12e900 173'3150 rep_tid=236212 committed?=1 applied?=1 op=osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:09.600433 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean]    q front is repgather(0x2a12e900 173'3150 rep_tid=236212 committed?=1 applied?=1 op=osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:09.600449 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] remove_repop repgather(0x2a12e900 173'3150 rep_tid=236212 committed?=1 applied?=1 op=osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:09.600465 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean]  requeue_ops 
2014-06-12 14:46:09.600490 7f9d64bda700  5 -- op tracker -- , seq: 747599, time: 2014-06-12 14:46:09.600485, event: done, op: osd_op(client.4282.0:145273 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 45056~520192] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.600510 7f9d64bda700 10 osd.2 173 dequeue_op 0x2b015d00 finish
2014-06-12 14:46:09.600520 7f9d64bda700  5 -- op tracker -- , seq: 747601, time: 2014-06-12 14:46:09.600516, event: done, op: osd_sub_op_reply(client.4282.0:145273 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.600530 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.600533 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.603894 7f9d4ec14700  5 -- op tracker -- , seq: 747602, time: 2014-06-12 14:46:09.602857, event: header_read, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.603905 7f9d4ec14700  5 -- op tracker -- , seq: 747602, time: 2014-06-12 14:46:09.602858, event: throttled, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.603926 7f9d4ec14700  5 -- op tracker -- , seq: 747602, time: 2014-06-12 14:46:09.603858, event: all_read, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.603932 7f9d4ec14700  5 -- op tracker -- , seq: 747602, time: 0.000000, event: dispatched, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.603936 7f9d4ec14700 20 osd.2 173 should_share_map client.4282 172.16.96.220:0/1241359 173
2014-06-12 14:46:09.603953 7f9d4ec14700 15 osd.2 173 enqueue_op 0x22b3b400 prio 63 cost 516096 latency 0.001096 osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:09.604005 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.604015 7f9d683e1700 10 osd.2 173 dequeue_op 0x22b3b400 prio 63 cost 516096 latency 0.001158 osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean]
2014-06-12 14:46:09.604039 7f9d683e1700  5 -- op tracker -- , seq: 747602, time: 2014-06-12 14:46:09.604034, event: reached_pg, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.604048 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:09.604061 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] handle_message: 0x22b3b400
2014-06-12 14:46:09.604069 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] do_op osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:09.604086 7f9d683e1700 15 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 '_'
2014-06-12 14:46:09.604227 7f9d683e1700 10 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 '_' = 0
2014-06-12 14:46:09.604241 7f9d683e1700 15 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 'snapset'
2014-06-12 14:46:09.604292 7f9d683e1700 10 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 'snapset' = 0
2014-06-12 14:46:09.604301 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] populate_obc_watchers 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.604308 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.604314 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] get_object_context: creating obc from disk: 0x15575340
2014-06-12 14:46:09.604320 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] get_object_context: 0x15575340 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 rwstate(none n=0 w=0) oi: 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3(173'3150 client.4282.0:145273 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3150) ssc: 0x679b0a0 snapset: 0=[]:[]+head
2014-06-12 14:46:09.604330 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] find_object_context 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 @head oi=71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3(173'3150 client.4282.0:145273 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3150)
2014-06-12 14:46:09.604347 7f9d683e1700  5 -- op tracker -- , seq: 747602, time: 2014-06-12 14:46:09.604343, event: started, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.604355 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] execute_ctx 0x2d2a6000
2014-06-12 14:46:09.604366 7f9d683e1700  5 -- op tracker -- , seq: 747602, time: 2014-06-12 14:46:09.604363, event: started, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.604378 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] do_op 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] ov 173'3150 av 173'3151 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:09.604388 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] do_osd_op 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096]
2014-06-12 14:46:09.604395 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:09.604404 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] do_osd_op  write 565248~516096
2014-06-12 14:46:09.604414 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] do_osd_op_effects on session 0x62d5000
2014-06-12 14:46:09.604420 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] make_writeable 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 snapset=0x679b0d8  snapc=0=[]
2014-06-12 14:46:09.604427 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] make_writeable 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:09.604433 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] finish_ctx 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 0x2d2a6000 op modify  
2014-06-12 14:46:09.604440 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean]  set mtime to 2014-06-12 08:45:11.983024
2014-06-12 14:46:09.604453 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean]  final snapset 0=[]:[]+head in 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.604464 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean]  zeroing write result code 0
2014-06-12 14:46:09.604470 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] calc_trim_to 79'149 -> 79'150
2014-06-12 14:46:09.604476 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] new_repop rep_tid 236213 on osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:09.604486 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] issue_repop rep_tid 236213 o 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.604502 7f9d683e1700  5 -- op tracker -- , seq: 747602, time: 2014-06-12 14:46:09.604499, event: waiting for subops from 20, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.604526 7f9d683e1700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:09.604541 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3150 (79'149,173'3150] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] append_log log((79'149,173'3150], crt=0'0) [173'3151 (173'3150) modify   71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 by client.4282.0:145275 2014-06-12 08:45:11.983024]
2014-06-12 14:46:09.604557 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'149,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] add_log_entry 173'3151 (173'3150) modify   71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 by client.4282.0:145275 2014-06-12 08:45:11.983024
2014-06-12 14:46:09.604572 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'149,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] append_log  adding 1 keys
2014-06-12 14:46:09.604580 7f9d683e1700 10 trim log((79'149,173'3151], crt=0'0) to 79'150
2014-06-12 14:46:09.604600 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3151, trimmed: 79'150
2014-06-12 14:46:09.604616 7f9d683e1700  5 queue_transactions existing osr(3.63b 0x276dc70)/0x276dc70
2014-06-12 14:46:09.604618 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.604620 7f9d683e1700  5 queue_op 0x68eb0e0 seq 2854 osr(3.63b 0x276dc70) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:09.604629 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5421440 (1 active)
2014-06-12 14:46:09.604625 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] eval_repop repgather(0x5390900 173'3151 rep_tid=236213 committed?=0 applied?=0 lock=0 op=osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:09.604635 7f9d81970700  5 _do_op 0x68eb0e0 seq 2854 osr(3.63b 0x276dc70)/0x276dc70 start
2014-06-12 14:46:09.604637 7f9d683e1700 10 osd.2 173 dequeue_op 0x22b3b400 finish
2014-06-12 14:46:09.604639 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.604639 7f9d81970700 10 _do_transaction on 0x6591800
2014-06-12 14:46:09.604641 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.604645 7f9d81970700 15 _omap_setkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:09.604736 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:09.604756 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.604797 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.604821 7f9d81970700 15 _omap_rmkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:09.604831 7f9d81970700 10 _omap_rmkeys meta/1fe0c16b/pglog_3.63b/0//-1 r = 0
2014-06-12 14:46:09.604837 7f9d81970700 15 _omap_setkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:09.604968 7f9d81970700 15 _write 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 565248~516096
2014-06-12 14:46:09.605056 7f9d81970700 10 lookup_strip_header done  cid 3.63b_head oid 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.605152 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:09.606569 7f9d81970700 10 _generic_write 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 565248~516096 = 0
2014-06-12 14:46:09.606620 7f9d81970700 15 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.606631 7f9d81970700 10 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 = 0
2014-06-12 14:46:09.606635 7f9d81970700 15 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.606640 7f9d81970700 10 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 = 0
2014-06-12 14:46:09.606783 7f9d81970700 20 set_header setting 18103 cid 3.63b_head oid 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 parent seq 0
2014-06-12 14:46:09.606842 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.606854 7f9d81970700 20 set_header setting 2110 cid meta oid 1fe0c16b/pglog_3.63b/0//-1 parent seq 0
2014-06-12 14:46:09.606860 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:09.608778 7f9d81970700 10 _do_op 0x68eb0e0 seq 2854 r = 0, finisher 0x733a4c0 0x247cfd80
2014-06-12 14:46:09.608800 7f9d81970700 10 _finish_op 0x68eb0e0 seq 2854 osr(3.63b 0x276dc70)/0x276dc70
2014-06-12 14:46:09.608810 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5421440 (0 active)
2014-06-12 14:46:09.608820 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.608808 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] op_commit: 236213
2014-06-12 14:46:09.608843 7f9d7715b700  5 -- op tracker -- , seq: 747602, time: 2014-06-12 14:46:09.608833, event: op_commit, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.608859 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] op_applied: 236213
2014-06-12 14:46:09.608880 7f9d7795c700  5 -- op tracker -- , seq: 747602, time: 2014-06-12 14:46:09.608875, event: op_applied, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.608888 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] op_applied on primary on version 173'3151
2014-06-12 14:46:09.611386 7f9d5703e700  5 -- op tracker -- , seq: 747603, time: 2014-06-12 14:46:09.611348, event: header_read, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ack, result = 0)
2014-06-12 14:46:09.611399 7f9d5703e700  5 -- op tracker -- , seq: 747603, time: 2014-06-12 14:46:09.611349, event: throttled, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ack, result = 0)
2014-06-12 14:46:09.611407 7f9d5703e700  5 -- op tracker -- , seq: 747603, time: 2014-06-12 14:46:09.611360, event: all_read, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ack, result = 0)
2014-06-12 14:46:09.611415 7f9d5703e700  5 -- op tracker -- , seq: 747603, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ack, result = 0)
2014-06-12 14:46:09.611421 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:09.611427 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:09.611435 7f9d5703e700 15 osd.2 173 enqueue_op 0x5289a00 prio 196 cost 0 latency 0.000087 osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ack, result = 0) v2
2014-06-12 14:46:09.611463 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.611471 7f9d65bdc700 10 osd.2 173 dequeue_op 0x5289a00 prio 196 cost 0 latency 0.000122 osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ack, result = 0) v2 pg pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean]
2014-06-12 14:46:09.611492 7f9d65bdc700  5 -- op tracker -- , seq: 747603, time: 2014-06-12 14:46:09.611490, event: reached_pg, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ack, result = 0)
2014-06-12 14:46:09.611507 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] handle_message: 0x5289a00
2014-06-12 14:46:09.611513 7f9d5703e700  5 -- op tracker -- , seq: 747604, time: 2014-06-12 14:46:09.611460, event: header_read, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.611525 7f9d5703e700  5 -- op tracker -- , seq: 747604, time: 2014-06-12 14:46:09.611461, event: throttled, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.611542 7f9d5703e700  5 -- op tracker -- , seq: 747604, time: 2014-06-12 14:46:09.611472, event: all_read, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.611553 7f9d5703e700  5 -- op tracker -- , seq: 747604, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.611558 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:09.611560 7f9d65bdc700  5 -- op tracker -- , seq: 747603, time: 2014-06-12 14:46:09.611517, event: started, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ack, result = 0)
2014-06-12 14:46:09.611564 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:09.611565 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] sub_op_modify_reply: tid 236213 op  ack_type 1 from 20
2014-06-12 14:46:09.611569 7f9d5703e700 15 osd.2 173 enqueue_op 0x124e3d00 prio 196 cost 0 latency 0.000109 osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:09.611580 7f9d65bdc700  5 -- op tracker -- , seq: 747602, time: 2014-06-12 14:46:09.611574, event: sub_op_applied_rec, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.611588 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:09.611585 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] repop_all_applied: repop tid 236213 all applied 
2014-06-12 14:46:09.611592 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] eval_repop repgather(0x5390900 173'3151 rep_tid=236213 committed?=0 applied?=1 lock=0 op=osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:09.611602 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean]  sending ack on repgather(0x5390900 173'3151 rep_tid=236213 committed?=0 applied?=1 lock=0 op=osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) 0x37812c80
2014-06-12 14:46:09.611622 7f9d65bdc700 10 osd.2 173 dequeue_op 0x5289a00 finish
2014-06-12 14:46:09.611630 7f9d65bdc700  5 -- op tracker -- , seq: 747603, time: 2014-06-12 14:46:09.611627, event: done, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ack, result = 0)
2014-06-12 14:46:09.611629 7f9d6abe6700 10 osd.2 173 dequeue_op 0x124e3d00 prio 196 cost 0 latency 0.000169 osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2 pg pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean]
2014-06-12 14:46:09.611645 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:09.611648 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.611647 7f9d6abe6700  5 -- op tracker -- , seq: 747604, time: 2014-06-12 14:46:09.611640, event: reached_pg, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.611653 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] handle_message: 0x124e3d00
2014-06-12 14:46:09.611663 7f9d6abe6700  5 -- op tracker -- , seq: 747604, time: 2014-06-12 14:46:09.611661, event: started, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.611666 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] sub_op_modify_reply: tid 236213 op  ack_type 4 from 20
2014-06-12 14:46:09.611677 7f9d6abe6700  5 -- op tracker -- , seq: 747602, time: 2014-06-12 14:46:09.611673, event: sub_op_commit_rec, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.611682 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3150 crt=0'0 lcod 173'3149 mlcod 173'3149 active+clean] repop_all_committed: repop tid 236213 all committed 
2014-06-12 14:46:09.611692 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3149 active+clean] eval_repop repgather(0x5390900 173'3151 rep_tid=236213 committed?=1 applied?=1 lock=0 op=osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:09.611702 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3149 active+clean] log_op_stats osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008764 lat 0.008844
2014-06-12 14:46:09.611714 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3149 active+clean] publish_stats_to_osd 173:4112
2014-06-12 14:46:09.611723 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3149 active+clean]  sending commit on repgather(0x5390900 173'3151 rep_tid=236213 committed?=1 applied?=1 lock=0 op=osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) 0x2b166500
2014-06-12 14:46:09.611744 7f9d6abe6700  5 -- op tracker -- , seq: 747602, time: 2014-06-12 14:46:09.611741, event: commit_sent, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.611750 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3149 active+clean]  requeue_ops 
2014-06-12 14:46:09.611757 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean]  removing repgather(0x5390900 173'3151 rep_tid=236213 committed?=1 applied?=1 op=osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:09.611767 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean]    q front is repgather(0x5390900 173'3151 rep_tid=236213 committed?=1 applied?=1 op=osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:09.611776 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] remove_repop repgather(0x5390900 173'3151 rep_tid=236213 committed?=1 applied?=1 op=osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:09.611786 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean]  requeue_ops 
2014-06-12 14:46:09.611804 7f9d6abe6700  5 -- op tracker -- , seq: 747602, time: 2014-06-12 14:46:09.611801, event: done, op: osd_op(client.4282.0:145275 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 565248~516096] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.611819 7f9d6abe6700 10 osd.2 173 dequeue_op 0x124e3d00 finish
2014-06-12 14:46:09.611827 7f9d6abe6700  5 -- op tracker -- , seq: 747604, time: 2014-06-12 14:46:09.611824, event: done, op: osd_sub_op_reply(client.4282.0:145275 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.611834 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.611836 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.635033 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:09.635091 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:09.650165 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:09.650200 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:09.801323 7f9d5a16f700  5 -- op tracker -- , seq: 747605, time: 2014-06-12 14:46:09.797867, event: header_read, op: osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.801342 7f9d5a16f700  5 -- op tracker -- , seq: 747605, time: 2014-06-12 14:46:09.797870, event: throttled, op: osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.801350 7f9d5a16f700  5 -- op tracker -- , seq: 747605, time: 2014-06-12 14:46:09.801268, event: all_read, op: osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.801358 7f9d5a16f700  5 -- op tracker -- , seq: 747605, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.801372 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:09.801380 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:09.801391 7f9d5a16f700 15 osd.2 173 enqueue_op 0x9fade00 prio 127 cost 2065191 latency 0.003523 osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.801672 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.801684 7f9d66bde700 10 osd.2 173 dequeue_op 0x9fade00 prio 127 cost 2065191 latency 0.003817 osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4e0( v 173'2633 (0'0,173'2633] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2632 active]
2014-06-12 14:46:09.801711 7f9d66bde700  5 -- op tracker -- , seq: 747605, time: 2014-06-12 14:46:09.801706, event: reached_pg, op: osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.801722 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2633 (0'0,173'2633] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2632 active] handle_message: 0x9fade00
2014-06-12 14:46:09.801735 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2633 (0'0,173'2633] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2632 active] sub_op_modify trans 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 v 173'2634 (transaction) 174
2014-06-12 14:46:09.801752 7f9d66bde700  5 -- op tracker -- , seq: 747605, time: 2014-06-12 14:46:09.801748, event: started, op: osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.801777 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2633 (0'0,173'2633] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2632 active] append_log log((0'0,173'2633], crt=0'0) [173'2634 (173'2633) modify   6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 by client.4306.0:143777 2014-06-12 08:43:01.347965]
2014-06-12 14:46:09.801804 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2634 (0'0,173'2634] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2632 active] add_log_entry 173'2634 (173'2633) modify   6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 by client.4306.0:143777 2014-06-12 08:43:01.347965
2014-06-12 14:46:09.801827 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2634 (0'0,173'2634] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2632 active] append_log  adding 1 keys
2014-06-12 14:46:09.801863 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2634, trimmed: 
2014-06-12 14:46:09.801888 7f9d66bde700  5 -- op tracker -- , seq: 747605, time: 2014-06-12 14:46:09.801883, event: started, op: osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.801899 7f9d66bde700  5 queue_transactions existing osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:09.801904 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.801906 7f9d66bde700  5 queue_op 0x9287ec0 seq 2434 osr(3.4e0 0x276dbd0) 2066677 bytes   (queue has 1 ops and 2066677 bytes)
2014-06-12 14:46:09.801916 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5180900 (1 active)
2014-06-12 14:46:09.801922 7f9d7a161700  5 _do_op 0x9287ec0 seq 2434 osr(3.4e0 0x276dbd0)/0x276dbd0 start
2014-06-12 14:46:09.801927 7f9d7a161700 10 _do_transaction on 0x5b7a2b0
2014-06-12 14:46:09.801914 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2634 (0'0,173'2634] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2632 active] do_sub_op osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.801929 7f9d66bde700 10 osd.2 173 dequeue_op 0x9fade00 finish
2014-06-12 14:46:09.801931 7f9d7a161700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:09.801933 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.801935 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.802129 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:09.802146 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.802179 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.802197 7f9d7a161700 15 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:09.802201 7f9d7a161700 10 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1 r = 0
2014-06-12 14:46:09.802205 7f9d7a161700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:09.802218 7f9d7a161700 15 _write 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 1835008~2064384
2014-06-12 14:46:09.802330 7f9d7a161700 10 lookup_strip_header done  cid 3.4e0_head oid 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:09.802565 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:09.809369 7f9d7a161700 10 _generic_write 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 1835008~2064384 = 0
2014-06-12 14:46:09.809503 7f9d7a161700 15 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:09.809525 7f9d7a161700 10 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:09.809530 7f9d7a161700 15 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:09.809535 7f9d7a161700 10 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:09.809918 7f9d7a161700 20 set_header setting 16555 cid 3.4e0_head oid 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:09.810082 7f9d7a161700 20 set_header setting 2236 cid meta oid 1fe07435/pglog_3.4e0/0//-1 parent seq 0
2014-06-12 14:46:09.810098 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.810108 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:09.817862 7f9d7a161700 10 _do_op 0x9287ec0 seq 2434 r = 0, finisher 0x6cb9c60 0
2014-06-12 14:46:09.817892 7f9d7a161700 10 _finish_op 0x9287ec0 seq 2434 osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:09.817911 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5180900 (0 active)
2014-06-12 14:46:09.817921 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.817919 7f9d7715b700  5 -- op tracker -- , seq: 747605, time: 2014-06-12 14:46:09.817906, event: commit_sent, op: osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.817929 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2634 (0'0,173'2634] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2632 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:09.817963 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:09.818003 7f9d7795c700  5 -- op tracker -- , seq: 747605, time: 2014-06-12 14:46:09.817997, event: sub_op_applied, op: osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.818014 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2634 (0'0,173'2634] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2633 active] sub_op_modify_applied on 0x5b7a1c0 op osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:09.818030 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2634 (0'0,173'2634] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2633 active] op_applied on primary on version 173'2634
2014-06-12 14:46:09.818040 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2634 (0'0,173'2634] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2633 active] op_applied on replica on version 173'2634
2014-06-12 14:46:09.818066 7f9d7795c700  5 -- op tracker -- , seq: 747605, time: 2014-06-12 14:46:09.818060, event: done, op: osd_sub_op(client.4306.0:143777 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2634 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:09.891961 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:09.892143 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:09.968205 7f9d4ec14700  5 -- op tracker -- , seq: 747606, time: 2014-06-12 14:46:09.963781, event: header_read, op: osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.968222 7f9d4ec14700  5 -- op tracker -- , seq: 747606, time: 2014-06-12 14:46:09.963788, event: throttled, op: osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.968232 7f9d4ec14700  5 -- op tracker -- , seq: 747606, time: 2014-06-12 14:46:09.968133, event: all_read, op: osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.968242 7f9d4ec14700  5 -- op tracker -- , seq: 747606, time: 0.000000, event: dispatched, op: osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.968258 7f9d4ec14700 20 osd.2 173 should_share_map client.4282 172.16.96.220:0/1241359 173
2014-06-12 14:46:09.968295 7f9d4ec14700 15 osd.2 173 enqueue_op 0x22b3a400 prio 63 cost 2064384 latency 0.004513 osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:09.968346 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.968358 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x22b3a400 prio 63 cost 2064384 latency 0.004576 osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean]
2014-06-12 14:46:09.968379 7f9d6a3e5700  5 -- op tracker -- , seq: 747606, time: 2014-06-12 14:46:09.968375, event: reached_pg, op: osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.968387 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:09.968399 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] handle_message: 0x22b3a400
2014-06-12 14:46:09.968406 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] do_op osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:09.968424 7f9d6a3e5700 15 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 '_'
2014-06-12 14:46:09.968586 7f9d6a3e5700 10 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 '_' = 0
2014-06-12 14:46:09.968599 7f9d6a3e5700 15 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 'snapset'
2014-06-12 14:46:09.968652 7f9d6a3e5700 10 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 'snapset' = 0
2014-06-12 14:46:09.968660 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] populate_obc_watchers 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.968673 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.968679 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] get_object_context: creating obc from disk: 0x10475080
2014-06-12 14:46:09.968684 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] get_object_context: 0x10475080 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 rwstate(none n=0 w=0) oi: 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3(173'3151 client.4282.0:145275 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3151) ssc: 0x5bb80e0 snapset: 0=[]:[]+head
2014-06-12 14:46:09.968694 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] find_object_context 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 @head oi=71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3(173'3151 client.4282.0:145275 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3151)
2014-06-12 14:46:09.968711 7f9d6a3e5700  5 -- op tracker -- , seq: 747606, time: 2014-06-12 14:46:09.968707, event: started, op: osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.968716 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] execute_ctx 0x17cfb800
2014-06-12 14:46:09.968728 7f9d6a3e5700  5 -- op tracker -- , seq: 747606, time: 2014-06-12 14:46:09.968724, event: started, op: osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.968735 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] do_op 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] ov 173'3151 av 173'3152 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:09.968748 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] do_osd_op 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384]
2014-06-12 14:46:09.968758 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:09.968768 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] do_osd_op  write 1081344~2064384
2014-06-12 14:46:09.968778 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] do_osd_op_effects on session 0x62d5000
2014-06-12 14:46:09.968785 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] make_writeable 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 snapset=0x5bb8118  snapc=0=[]
2014-06-12 14:46:09.968792 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] make_writeable 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:09.968798 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] finish_ctx 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 0x17cfb800 op modify  
2014-06-12 14:46:09.968806 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean]  set mtime to 2014-06-12 08:45:12.341111
2014-06-12 14:46:09.968853 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean]  final snapset 0=[]:[]+head in 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.968872 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean]  zeroing write result code 0
2014-06-12 14:46:09.968878 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] calc_trim_to 79'150 -> 79'151
2014-06-12 14:46:09.968884 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] new_repop rep_tid 236214 on osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:09.968895 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] issue_repop rep_tid 236214 o 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.968914 7f9d6a3e5700  5 -- op tracker -- , seq: 747606, time: 2014-06-12 14:46:09.968911, event: waiting for subops from 20, op: osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.968934 7f9d6a3e5700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:09.968949 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3151 (79'150,173'3151] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] append_log log((79'150,173'3151], crt=0'0) [173'3152 (173'3151) modify   71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 by client.4282.0:145276 2014-06-12 08:45:12.341111]
2014-06-12 14:46:09.968965 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'150,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3151 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] add_log_entry 173'3152 (173'3151) modify   71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 by client.4282.0:145276 2014-06-12 08:45:12.341111
2014-06-12 14:46:09.968978 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'150,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3151 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] append_log  adding 1 keys
2014-06-12 14:46:09.968986 7f9d6a3e5700 10 trim log((79'150,173'3152], crt=0'0) to 79'151
2014-06-12 14:46:09.969005 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3152, trimmed: 79'151
2014-06-12 14:46:09.969021 7f9d6a3e5700  5 queue_transactions existing osr(3.63b 0x276dc70)/0x276dc70
2014-06-12 14:46:09.969024 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:09.969025 7f9d6a3e5700  5 queue_op 0x7380480 seq 2855 osr(3.63b 0x276dc70) 2066712 bytes   (queue has 1 ops and 2066712 bytes)
2014-06-12 14:46:09.969031 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3151 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] eval_repop repgather(0x2a37bbc0 173'3152 rep_tid=236214 committed?=0 applied?=0 lock=0 op=osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:09.969043 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x22b3a400 finish
2014-06-12 14:46:09.969045 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.969047 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.969057 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5421440 (1 active)
2014-06-12 14:46:09.969060 7f9d7a962700  5 _do_op 0x7380480 seq 2855 osr(3.63b 0x276dc70)/0x276dc70 start
2014-06-12 14:46:09.969063 7f9d7a962700 10 _do_transaction on 0x5361980
2014-06-12 14:46:09.969068 7f9d7a962700 15 _omap_setkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:09.969140 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:09.969161 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:09.969192 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:09.969313 7f9d7a962700 15 _omap_rmkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:09.969322 7f9d7a962700 10 _omap_rmkeys meta/1fe0c16b/pglog_3.63b/0//-1 r = 0
2014-06-12 14:46:09.969327 7f9d7a962700 15 _omap_setkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:09.969340 7f9d7a962700 15 _write 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 1081344~2064384
2014-06-12 14:46:09.969427 7f9d7a962700 10 lookup_strip_header done  cid 3.63b_head oid 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.969665 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:09.976459 7f9d7a962700 10 _generic_write 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 1081344~2064384 = 0
2014-06-12 14:46:09.976681 7f9d7a962700 15 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.976699 7f9d7a962700 10 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 = 0
2014-06-12 14:46:09.976706 7f9d7a962700 15 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.976713 7f9d7a962700 10 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 = 0
2014-06-12 14:46:09.977870 7f9d7a962700 20 set_header setting 18103 cid 3.63b_head oid 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 parent seq 0
2014-06-12 14:46:09.978053 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:09.978065 7f9d7a962700 20 set_header setting 2110 cid meta oid 1fe0c16b/pglog_3.63b/0//-1 parent seq 0
2014-06-12 14:46:09.978071 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:09.988536 7f9d7a962700 10 _do_op 0x7380480 seq 2855 r = 0, finisher 0x31d0fd00 0x111fb640
2014-06-12 14:46:09.988566 7f9d7a962700 10 _finish_op 0x7380480 seq 2855 osr(3.63b 0x276dc70)/0x276dc70
2014-06-12 14:46:09.988588 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5421440 (0 active)
2014-06-12 14:46:09.988573 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3151 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] op_commit: 236214
2014-06-12 14:46:09.988597 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:09.988610 7f9d7715b700  5 -- op tracker -- , seq: 747606, time: 2014-06-12 14:46:09.988600, event: op_commit, op: osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.988623 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3151 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] op_applied: 236214
2014-06-12 14:46:09.988648 7f9d7795c700  5 -- op tracker -- , seq: 747606, time: 2014-06-12 14:46:09.988641, event: op_applied, op: osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.988659 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3151 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] op_applied on primary on version 173'3152
2014-06-12 14:46:09.992069 7f9d5703e700  5 -- op tracker -- , seq: 747607, time: 2014-06-12 14:46:09.992015, event: header_read, op: osd_sub_op_reply(client.4282.0:145276 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.992082 7f9d5703e700  5 -- op tracker -- , seq: 747607, time: 2014-06-12 14:46:09.992017, event: throttled, op: osd_sub_op_reply(client.4282.0:145276 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.992105 7f9d5703e700  5 -- op tracker -- , seq: 747607, time: 2014-06-12 14:46:09.992030, event: all_read, op: osd_sub_op_reply(client.4282.0:145276 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.992111 7f9d5703e700  5 -- op tracker -- , seq: 747607, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4282.0:145276 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.992116 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4282.0:145276 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:09.992121 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:09.992128 7f9d5703e700 15 osd.2 173 enqueue_op 0x31d60e00 prio 196 cost 0 latency 0.000113 osd_sub_op_reply(client.4282.0:145276 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:09.992156 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.992174 7f9d653db700 10 osd.2 173 dequeue_op 0x31d60e00 prio 196 cost 0 latency 0.000151 osd_sub_op_reply(client.4282.0:145276 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2 pg pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3151 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean]
2014-06-12 14:46:09.992197 7f9d653db700  5 -- op tracker -- , seq: 747607, time: 2014-06-12 14:46:09.992195, event: reached_pg, op: osd_sub_op_reply(client.4282.0:145276 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.992208 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3151 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] handle_message: 0x31d60e00
2014-06-12 14:46:09.992217 7f9d653db700  5 -- op tracker -- , seq: 747607, time: 2014-06-12 14:46:09.992215, event: started, op: osd_sub_op_reply(client.4282.0:145276 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.992221 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3151 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] sub_op_modify_reply: tid 236214 op  ack_type 4 from 20
2014-06-12 14:46:09.992233 7f9d653db700  5 -- op tracker -- , seq: 747606, time: 2014-06-12 14:46:09.992229, event: sub_op_commit_rec, op: osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.992244 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3151 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] repop_all_applied: repop tid 236214 all applied 
2014-06-12 14:46:09.992251 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3151 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] eval_repop repgather(0x2a37bbc0 173'3152 rep_tid=236214 committed?=0 applied?=1 lock=0 op=osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:09.992261 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3151 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean]  sending ack on repgather(0x2a37bbc0 173'3152 rep_tid=236214 committed?=0 applied?=1 lock=0 op=osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) 0xae7fb80
2014-06-12 14:46:09.992281 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3151 crt=0'0 lcod 173'3150 mlcod 173'3150 active+clean] repop_all_committed: repop tid 236214 all committed 
2014-06-12 14:46:09.992287 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3150 active+clean] eval_repop repgather(0x2a37bbc0 173'3152 rep_tid=236214 committed?=1 applied?=1 lock=0 op=osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:09.992297 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3150 active+clean] log_op_stats osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 inb 2064663 outb 0 rlat 0.028499 lat 0.028515
2014-06-12 14:46:09.992308 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3150 active+clean] publish_stats_to_osd 173:4113
2014-06-12 14:46:09.992316 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3150 active+clean]  sending commit on repgather(0x2a37bbc0 173'3152 rep_tid=236214 committed?=1 applied?=1 lock=0 op=osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) 0x6700000
2014-06-12 14:46:09.992334 7f9d653db700  5 -- op tracker -- , seq: 747606, time: 2014-06-12 14:46:09.992331, event: commit_sent, op: osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.992340 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3150 active+clean]  requeue_ops 
2014-06-12 14:46:09.992346 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean]  removing repgather(0x2a37bbc0 173'3152 rep_tid=236214 committed?=1 applied?=1 op=osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:09.992354 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean]    q front is repgather(0x2a37bbc0 173'3152 rep_tid=236214 committed?=1 applied?=1 op=osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:09.992363 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] remove_repop repgather(0x2a37bbc0 173'3152 rep_tid=236214 committed?=1 applied?=1 op=osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:09.992372 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean]  requeue_ops 
2014-06-12 14:46:09.992390 7f9d653db700  5 -- op tracker -- , seq: 747606, time: 2014-06-12 14:46:09.992387, event: done, op: osd_op(client.4282.0:145276 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 1081344~2064384] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.992405 7f9d653db700 10 osd.2 173 dequeue_op 0x31d60e00 finish
2014-06-12 14:46:09.992409 7f9d653db700  5 -- op tracker -- , seq: 747607, time: 2014-06-12 14:46:09.992407, event: done, op: osd_sub_op_reply(client.4282.0:145276 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:09.992427 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:09.992429 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:09.999379 7f9d4ec14700  5 -- op tracker -- , seq: 747608, time: 2014-06-12 14:46:09.997152, event: header_read, op: osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.999394 7f9d4ec14700  5 -- op tracker -- , seq: 747608, time: 2014-06-12 14:46:09.997154, event: throttled, op: osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.999404 7f9d4ec14700  5 -- op tracker -- , seq: 747608, time: 2014-06-12 14:46:09.999337, event: all_read, op: osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.999414 7f9d4ec14700  5 -- op tracker -- , seq: 747608, time: 0.000000, event: dispatched, op: osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.999424 7f9d4ec14700 20 osd.2 173 should_share_map client.4282 172.16.96.220:0/1241359 173
2014-06-12 14:46:09.999453 7f9d4ec14700 15 osd.2 173 enqueue_op 0x22b3a000 prio 63 cost 1048576 latency 0.002301 osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:09.999488 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:09.999500 7f9d63bd8700 10 osd.2 173 dequeue_op 0x22b3a000 prio 63 cost 1048576 latency 0.002348 osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean]
2014-06-12 14:46:09.999530 7f9d63bd8700  5 -- op tracker -- , seq: 747608, time: 2014-06-12 14:46:09.999524, event: reached_pg, op: osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:09.999542 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:09.999559 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] handle_message: 0x22b3a000
2014-06-12 14:46:09.999573 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] do_op osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:09.999600 7f9d63bd8700 15 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 '_'
2014-06-12 14:46:09.999797 7f9d63bd8700 10 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 '_' = 0
2014-06-12 14:46:09.999815 7f9d63bd8700 15 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 'snapset'
2014-06-12 14:46:09.999902 7f9d63bd8700 10 getattr 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 'snapset' = 0
2014-06-12 14:46:09.999914 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] populate_obc_watchers 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.999927 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:09.999938 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] get_object_context: creating obc from disk: 0x37e4ab00
2014-06-12 14:46:09.999951 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] get_object_context: 0x37e4ab00 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 rwstate(none n=0 w=0) oi: 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3(173'3152 client.4282.0:145276 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3152) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:46:09.999970 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] find_object_context 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 @head oi=71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3(173'3152 client.4282.0:145276 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3152)
2014-06-12 14:46:09.999997 7f9d63bd8700  5 -- op tracker -- , seq: 747608, time: 2014-06-12 14:46:09.999991, event: started, op: osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.000008 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] execute_ctx 0x2a84400
2014-06-12 14:46:10.000029 7f9d63bd8700  5 -- op tracker -- , seq: 747608, time: 2014-06-12 14:46:10.000023, event: started, op: osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.000040 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] do_op 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] ov 173'3152 av 173'3153 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:10.000058 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] do_osd_op 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576]
2014-06-12 14:46:10.000072 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:10.000087 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] do_osd_op  write 3145728~1048576
2014-06-12 14:46:10.000105 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] do_osd_op_effects on session 0x62d5000
2014-06-12 14:46:10.000118 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] make_writeable 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:46:10.000132 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] make_writeable 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:10.000144 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] finish_ctx 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 0x2a84400 op modify  
2014-06-12 14:46:10.000158 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean]  set mtime to 2014-06-12 08:45:12.376260
2014-06-12 14:46:10.000181 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean]  final snapset 0=[]:[]+head in 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:10.000201 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean]  zeroing write result code 0
2014-06-12 14:46:10.000213 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] calc_trim_to 79'151 -> 79'152
2014-06-12 14:46:10.000225 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] new_repop rep_tid 236215 on osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:10.000248 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] issue_repop rep_tid 236215 o 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:10.000275 7f9d63bd8700  5 -- op tracker -- , seq: 747608, time: 2014-06-12 14:46:10.000270, event: waiting for subops from 20, op: osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.000298 7f9d63bd8700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:10.000324 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3152 (79'151,173'3152] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] append_log log((79'151,173'3152], crt=0'0) [173'3153 (173'3152) modify   71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 by client.4282.0:145277 2014-06-12 08:45:12.376260]
2014-06-12 14:46:10.000349 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'151,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3152 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] add_log_entry 173'3153 (173'3152) modify   71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 by client.4282.0:145277 2014-06-12 08:45:12.376260
2014-06-12 14:46:10.000371 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'151,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3152 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] append_log  adding 1 keys
2014-06-12 14:46:10.000387 7f9d63bd8700 10 trim log((79'151,173'3153], crt=0'0) to 79'152
2014-06-12 14:46:10.000414 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3153, trimmed: 79'152
2014-06-12 14:46:10.000440 7f9d63bd8700  5 queue_transactions existing osr(3.63b 0x276dc70)/0x276dc70
2014-06-12 14:46:10.000446 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.000448 7f9d63bd8700  5 queue_op 0x92b19e0 seq 2856 osr(3.63b 0x276dc70) 1050904 bytes   (queue has 1 ops and 1050904 bytes)
2014-06-12 14:46:10.000455 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3152 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] eval_repop repgather(0x224e600 173'3153 rep_tid=236215 committed?=0 applied?=0 lock=0 op=osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.000475 7f9d63bd8700 10 osd.2 173 dequeue_op 0x22b3a000 finish
2014-06-12 14:46:10.000479 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.000482 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.000491 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5421440 (1 active)
2014-06-12 14:46:10.000496 7f9d7f16b700  5 _do_op 0x92b19e0 seq 2856 osr(3.63b 0x276dc70)/0x276dc70 start
2014-06-12 14:46:10.000500 7f9d7f16b700 10 _do_transaction on 0x32e7ac00
2014-06-12 14:46:10.000506 7f9d7f16b700 15 _omap_setkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:10.000594 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:10.000614 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.000665 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.000684 7f9d7f16b700 15 _omap_rmkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:10.000694 7f9d7f16b700 10 _omap_rmkeys meta/1fe0c16b/pglog_3.63b/0//-1 r = 0
2014-06-12 14:46:10.000701 7f9d7f16b700 15 _omap_setkeys meta/1fe0c16b/pglog_3.63b/0//-1
2014-06-12 14:46:10.000716 7f9d7f16b700 15 _write 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 3145728~1048576
2014-06-12 14:46:10.000852 7f9d7f16b700 10 lookup_strip_header done  cid 3.63b_head oid 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:10.000878 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:10.004820 7f9d7f16b700 10 _generic_write 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 3145728~1048576 = 0
2014-06-12 14:46:10.004899 7f9d7f16b700 15 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:10.004922 7f9d7f16b700 10 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 = 0
2014-06-12 14:46:10.004931 7f9d7f16b700 15 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3
2014-06-12 14:46:10.004939 7f9d7f16b700 10 _setattrs 3.63b_head/71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 = 0
2014-06-12 14:46:10.005218 7f9d7f16b700 20 set_header setting 18103 cid 3.63b_head oid 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 parent seq 0
2014-06-12 14:46:10.005320 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.005339 7f9d7f16b700 20 set_header setting 2110 cid meta oid 1fe0c16b/pglog_3.63b/0//-1 parent seq 0
2014-06-12 14:46:10.005352 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:10.011289 7f9d7f16b700 10 _do_op 0x92b19e0 seq 2856 r = 0, finisher 0x65eca40 0x3bbfa00
2014-06-12 14:46:10.011320 7f9d7f16b700 10 _finish_op 0x92b19e0 seq 2856 osr(3.63b 0x276dc70)/0x276dc70
2014-06-12 14:46:10.011330 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5421440 (0 active)
2014-06-12 14:46:10.011338 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.011343 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3152 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] op_commit: 236215
2014-06-12 14:46:10.011414 7f9d7715b700  5 -- op tracker -- , seq: 747608, time: 2014-06-12 14:46:10.011400, event: op_commit, op: osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.011472 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3152 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] op_applied: 236215
2014-06-12 14:46:10.011493 7f9d7795c700  5 -- op tracker -- , seq: 747608, time: 2014-06-12 14:46:10.011488, event: op_applied, op: osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.011499 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3152 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] op_applied on primary on version 173'3153
2014-06-12 14:46:10.012680 7f9d5703e700  5 -- op tracker -- , seq: 747609, time: 2014-06-12 14:46:10.012647, event: header_read, op: osd_sub_op_reply(client.4282.0:145277 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.012694 7f9d5703e700  5 -- op tracker -- , seq: 747609, time: 2014-06-12 14:46:10.012649, event: throttled, op: osd_sub_op_reply(client.4282.0:145277 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.012703 7f9d5703e700  5 -- op tracker -- , seq: 747609, time: 2014-06-12 14:46:10.012657, event: all_read, op: osd_sub_op_reply(client.4282.0:145277 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.012711 7f9d5703e700  5 -- op tracker -- , seq: 747609, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4282.0:145277 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.012715 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4282.0:145277 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:10.012721 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:10.012730 7f9d5703e700 15 osd.2 173 enqueue_op 0x31d60d00 prio 196 cost 0 latency 0.000083 osd_sub_op_reply(client.4282.0:145277 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:10.012754 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.012762 7f9d69be4700 10 osd.2 173 dequeue_op 0x31d60d00 prio 196 cost 0 latency 0.000115 osd_sub_op_reply(client.4282.0:145277 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0) v2 pg pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3152 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean]
2014-06-12 14:46:10.012785 7f9d69be4700  5 -- op tracker -- , seq: 747609, time: 2014-06-12 14:46:10.012781, event: reached_pg, op: osd_sub_op_reply(client.4282.0:145277 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.012794 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3152 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] handle_message: 0x31d60d00
2014-06-12 14:46:10.012811 7f9d69be4700  5 -- op tracker -- , seq: 747609, time: 2014-06-12 14:46:10.012807, event: started, op: osd_sub_op_reply(client.4282.0:145277 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.012818 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3152 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] sub_op_modify_reply: tid 236215 op  ack_type 4 from 20
2014-06-12 14:46:10.012839 7f9d69be4700  5 -- op tracker -- , seq: 747608, time: 2014-06-12 14:46:10.012832, event: sub_op_commit_rec, op: osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.012848 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3152 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] repop_all_applied: repop tid 236215 all applied 
2014-06-12 14:46:10.012862 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3152 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] eval_repop repgather(0x224e600 173'3153 rep_tid=236215 committed?=0 applied?=1 lock=0 op=osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.012880 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3152 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean]  sending ack on repgather(0x224e600 173'3153 rep_tid=236215 committed?=0 applied?=1 lock=0 op=osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) 0xfa08000
2014-06-12 14:46:10.012915 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 luod=173'3152 crt=0'0 lcod 173'3151 mlcod 173'3151 active+clean] repop_all_committed: repop tid 236215 all committed 
2014-06-12 14:46:10.012926 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3151 active+clean] eval_repop repgather(0x224e600 173'3153 rep_tid=236215 committed?=1 applied?=1 lock=0 op=osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.012944 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3151 active+clean] log_op_stats osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4 inb 1048855 outb 0 rlat 0.015761 lat 0.015790
2014-06-12 14:46:10.012962 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3151 active+clean] publish_stats_to_osd 173:4114
2014-06-12 14:46:10.012975 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3151 active+clean]  sending commit on repgather(0x224e600 173'3153 rep_tid=236215 committed?=1 applied?=1 lock=0 op=osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4) 0x3371ca00
2014-06-12 14:46:10.013007 7f9d69be4700  5 -- op tracker -- , seq: 747608, time: 2014-06-12 14:46:10.013002, event: commit_sent, op: osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.013016 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3151 active+clean]  requeue_ops 
2014-06-12 14:46:10.013027 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean]  removing repgather(0x224e600 173'3153 rep_tid=236215 committed?=1 applied?=1 op=osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.013042 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean]    q front is repgather(0x224e600 173'3153 rep_tid=236215 committed?=1 applied?=1 op=osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.013057 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] remove_repop repgather(0x224e600 173'3153 rep_tid=236215 committed?=1 applied?=1 op=osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.013073 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean]  requeue_ops 
2014-06-12 14:46:10.013097 7f9d69be4700  5 -- op tracker -- , seq: 747608, time: 2014-06-12 14:46:10.013093, event: done, op: osd_op(client.4282.0:145277 rbd_data.10916b8b4567.000000000000108d [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] 3.71ae563b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.013117 7f9d69be4700 10 osd.2 173 dequeue_op 0x31d60d00 finish
2014-06-12 14:46:10.013126 7f9d69be4700  5 -- op tracker -- , seq: 747609, time: 2014-06-12 14:46:10.013122, event: done, op: osd_sub_op_reply(client.4282.0:145277 3.63b 71ae563b/rbd_data.10916b8b4567.000000000000108d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.013136 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.013139 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.017568 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:10.017717 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:10.113876 7f9d494c9700  5 -- op tracker -- , seq: 747610, time: 2014-06-12 14:46:10.112299, event: header_read, op: osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.113889 7f9d494c9700  5 -- op tracker -- , seq: 747610, time: 2014-06-12 14:46:10.112303, event: throttled, op: osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.113898 7f9d494c9700  5 -- op tracker -- , seq: 747610, time: 2014-06-12 14:46:10.113833, event: all_read, op: osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.113908 7f9d494c9700  5 -- op tracker -- , seq: 747610, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.113915 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:46:10.113948 7f9d494c9700 15 osd.2 173 enqueue_op 0x20e86e00 prio 63 cost 421888 latency 0.001648 osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:10.113978 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.113986 7f9d68be2700 10 osd.2 173 dequeue_op 0x20e86e00 prio 63 cost 421888 latency 0.001686 osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean]
2014-06-12 14:46:10.114013 7f9d68be2700  5 -- op tracker -- , seq: 747610, time: 2014-06-12 14:46:10.114008, event: reached_pg, op: osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.114022 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:10.114036 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] handle_message: 0x20e86e00
2014-06-12 14:46:10.114045 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] do_op osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:10.114066 7f9d68be2700 15 getattr 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 '_'
2014-06-12 14:46:10.114340 7f9d68be2700 10 getattr 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 '_' = 0
2014-06-12 14:46:10.114353 7f9d68be2700 15 getattr 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 'snapset'
2014-06-12 14:46:10.114438 7f9d68be2700 10 getattr 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 'snapset' = 0
2014-06-12 14:46:10.114446 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] populate_obc_watchers 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:10.114457 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:10.114465 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] get_object_context: creating obc from disk: 0xa45d080
2014-06-12 14:46:10.114475 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] get_object_context: 0xa45d080 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 rwstate(none n=0 w=0) oi: 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'633 client.4290.0:37871 wrlock_by=unknown.0.0:0 dirty s 4194304 uv633) ssc: 0x9344000 snapset: 0=[]:[]+head
2014-06-12 14:46:10.114490 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] find_object_context 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 @head oi=82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'633 client.4290.0:37871 wrlock_by=unknown.0.0:0 dirty s 4194304 uv633)
2014-06-12 14:46:10.114565 7f9d68be2700  5 -- op tracker -- , seq: 747610, time: 2014-06-12 14:46:10.114510, event: started, op: osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.114577 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] execute_ctx 0x22135800
2014-06-12 14:46:10.114595 7f9d68be2700  5 -- op tracker -- , seq: 747610, time: 2014-06-12 14:46:10.114590, event: started, op: osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.114600 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] do_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] ov 173'633 av 173'2576 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:10.114617 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] do_osd_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888]
2014-06-12 14:46:10.114627 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:10.114638 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] do_osd_op  write 0~421888
2014-06-12 14:46:10.114652 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:46:10.114662 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 snapset=0x9344038  snapc=0=[]
2014-06-12 14:46:10.114672 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:10.114682 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] finish_ctx 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 0x22135800 op modify  
2014-06-12 14:46:10.114692 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean]  set mtime to 2014-06-12 08:45:12.492353
2014-06-12 14:46:10.114710 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean]  final snapset 0=[]:[]+head in 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:10.114727 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean]  zeroing write result code 0
2014-06-12 14:46:10.114736 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] new_repop rep_tid 236216 on osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:10.114750 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] issue_repop rep_tid 236216 o 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:10.114771 7f9d68be2700  5 -- op tracker -- , seq: 747610, time: 2014-06-12 14:46:10.114766, event: waiting for subops from 27, op: osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.114794 7f9d68be2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.114815 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2575 (0'0,173'2575] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] append_log log((0'0,173'2575], crt=0'0) [173'2576 (173'633) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143628 2014-06-12 08:45:12.492353]
2014-06-12 14:46:10.114836 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2575 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] add_log_entry 173'2576 (173'633) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143628 2014-06-12 08:45:12.492353
2014-06-12 14:46:10.114855 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2575 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] append_log  adding 1 keys
2014-06-12 14:46:10.114884 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2576, trimmed: 
2014-06-12 14:46:10.114903 7f9d68be2700  5 queue_transactions existing osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:10.114907 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.114909 7f9d68be2700  5 queue_op 0x5cc4600 seq 2378 osr(3.717 0x276de70) 424181 bytes   (queue has 1 ops and 424181 bytes)
2014-06-12 14:46:10.114919 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a3d40 (1 active)
2014-06-12 14:46:10.114927 7f9d7895e700  5 _do_op 0x5cc4600 seq 2378 osr(3.717 0x276de70)/0x276de70 start
2014-06-12 14:46:10.114931 7f9d7895e700 10 _do_transaction on 0x2b619b00
2014-06-12 14:46:10.114916 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2575 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] eval_repop repgather(0x32e7bec0 173'2576 rep_tid=236216 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.114934 7f9d68be2700 10 osd.2 173 dequeue_op 0x20e86e00 finish
2014-06-12 14:46:10.114936 7f9d7895e700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:10.114938 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.114945 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.115041 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:10.115059 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.115107 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.115124 7f9d7895e700 15 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:10.115130 7f9d7895e700 10 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1 r = 0
2014-06-12 14:46:10.115135 7f9d7895e700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:10.115148 7f9d7895e700 15 _write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 0~421888
2014-06-12 14:46:10.115289 7f9d7895e700 10 lookup_strip_header done  cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:10.115650 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:10.117479 7f9d7895e700 10 _generic_write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 0~421888 = 0
2014-06-12 14:46:10.117566 7f9d7895e700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:10.117590 7f9d7895e700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:10.117597 7f9d7895e700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:10.117605 7f9d7895e700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:10.117848 7f9d7895e700 20 set_header setting 15870 cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 parent seq 0
2014-06-12 14:46:10.117914 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.117930 7f9d7895e700 20 set_header setting 2128 cid meta oid 1fe0f8ca/pglog_3.717/0//-1 parent seq 0
2014-06-12 14:46:10.117943 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:10.119946 7f9d7895e700 10 _do_op 0x5cc4600 seq 2378 r = 0, finisher 0x31bdc700 0x216a0e00
2014-06-12 14:46:10.119962 7f9d7895e700 10 _finish_op 0x5cc4600 seq 2378 osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:10.119967 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2575 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] op_commit: 236216
2014-06-12 14:46:10.119982 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a3d40 (0 active)
2014-06-12 14:46:10.119996 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.120003 7f9d7715b700  5 -- op tracker -- , seq: 747610, time: 2014-06-12 14:46:10.119994, event: op_commit, op: osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.120017 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2575 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] op_applied: 236216
2014-06-12 14:46:10.120041 7f9d7795c700  5 -- op tracker -- , seq: 747610, time: 2014-06-12 14:46:10.120034, event: op_applied, op: osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.120053 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2575 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] op_applied on primary on version 173'2576
2014-06-12 14:46:10.122991 7f9d58c5a700  5 -- op tracker -- , seq: 747611, time: 2014-06-12 14:46:10.122330, event: header_read, op: osd_sub_op_reply(client.4290.0:143628 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.123057 7f9d58c5a700  5 -- op tracker -- , seq: 747611, time: 2014-06-12 14:46:10.122498, event: throttled, op: osd_sub_op_reply(client.4290.0:143628 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.123160 7f9d58c5a700  5 -- op tracker -- , seq: 747611, time: 2014-06-12 14:46:10.122571, event: all_read, op: osd_sub_op_reply(client.4290.0:143628 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.123169 7f9d58c5a700  5 -- op tracker -- , seq: 747611, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143628 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.123176 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143628 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:10.123183 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:10.123193 7f9d58c5a700 15 osd.2 173 enqueue_op 0x2b014f00 prio 196 cost 0 latency 0.000862 osd_sub_op_reply(client.4290.0:143628 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:10.124332 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.124346 7f9d633d7700 10 osd.2 173 dequeue_op 0x2b014f00 prio 196 cost 0 latency 0.002016 osd_sub_op_reply(client.4290.0:143628 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 pg pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2575 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean]
2014-06-12 14:46:10.124373 7f9d633d7700  5 -- op tracker -- , seq: 747611, time: 2014-06-12 14:46:10.124368, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143628 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.124382 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2575 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] handle_message: 0x2b014f00
2014-06-12 14:46:10.124398 7f9d633d7700  5 -- op tracker -- , seq: 747611, time: 2014-06-12 14:46:10.124395, event: started, op: osd_sub_op_reply(client.4290.0:143628 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.124405 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2575 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] sub_op_modify_reply: tid 236216 op  ack_type 4 from 27
2014-06-12 14:46:10.124427 7f9d633d7700  5 -- op tracker -- , seq: 747610, time: 2014-06-12 14:46:10.124419, event: sub_op_commit_rec, op: osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.124435 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2575 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] repop_all_applied: repop tid 236216 all applied 
2014-06-12 14:46:10.124446 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2575 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] eval_repop repgather(0x32e7bec0 173'2576 rep_tid=236216 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.124462 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2575 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean]  sending ack on repgather(0x32e7bec0 173'2576 rep_tid=236216 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x1a58d900
2014-06-12 14:46:10.124495 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2575 crt=0'0 lcod 173'2574 mlcod 173'2574 active+clean] repop_all_committed: repop tid 236216 all committed 
2014-06-12 14:46:10.124506 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2574 active+clean] eval_repop repgather(0x32e7bec0 173'2576 rep_tid=236216 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.124522 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2574 active+clean] log_op_stats osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 inb 422167 outb 0 rlat 0.012194 lat 0.012222
2014-06-12 14:46:10.124540 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2574 active+clean] publish_stats_to_osd 173:3221
2014-06-12 14:46:10.124554 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2574 active+clean]  sending commit on repgather(0x32e7bec0 173'2576 rep_tid=236216 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x2fa4cc80
2014-06-12 14:46:10.124583 7f9d633d7700  5 -- op tracker -- , seq: 747610, time: 2014-06-12 14:46:10.124578, event: commit_sent, op: osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.124591 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2574 active+clean]  requeue_ops 
2014-06-12 14:46:10.124601 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean]  removing repgather(0x32e7bec0 173'2576 rep_tid=236216 committed?=1 applied?=1 op=osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.124615 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean]    q front is repgather(0x32e7bec0 173'2576 rep_tid=236216 committed?=1 applied?=1 op=osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.124629 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] remove_repop repgather(0x32e7bec0 173'2576 rep_tid=236216 committed?=1 applied?=1 op=osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.124643 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean]  requeue_ops 
2014-06-12 14:46:10.124669 7f9d633d7700  5 -- op tracker -- , seq: 747610, time: 2014-06-12 14:46:10.124664, event: done, op: osd_op(client.4290.0:143628 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.124689 7f9d633d7700 10 osd.2 173 dequeue_op 0x2b014f00 finish
2014-06-12 14:46:10.124696 7f9d633d7700  5 -- op tracker -- , seq: 747611, time: 2014-06-12 14:46:10.124692, event: done, op: osd_sub_op_reply(client.4290.0:143628 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.124707 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.124710 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.125859 7f9d5a16f700  5 -- op tracker -- , seq: 747612, time: 2014-06-12 14:46:10.123120, event: header_read, op: osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.125873 7f9d5a16f700  5 -- op tracker -- , seq: 747612, time: 2014-06-12 14:46:10.123122, event: throttled, op: osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.125882 7f9d5a16f700  5 -- op tracker -- , seq: 747612, time: 2014-06-12 14:46:10.125822, event: all_read, op: osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.125891 7f9d5a16f700  5 -- op tracker -- , seq: 747612, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.125898 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.125905 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:10.125915 7f9d5a16f700 15 osd.2 173 enqueue_op 0x9facc00 prio 127 cost 295719 latency 0.002795 osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.125950 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.125959 7f9d623d5700 10 osd.2 173 dequeue_op 0x9facc00 prio 127 cost 295719 latency 0.002839 osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2655 (0'0,173'2655] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2654 active]
2014-06-12 14:46:10.125985 7f9d623d5700  5 -- op tracker -- , seq: 747612, time: 2014-06-12 14:46:10.125981, event: reached_pg, op: osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.125994 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2655 (0'0,173'2655] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2654 active] handle_message: 0x9facc00
2014-06-12 14:46:10.126009 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2655 (0'0,173'2655] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2654 active] sub_op_modify trans f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 v 173'2656 (transaction) 174
2014-06-12 14:46:10.126030 7f9d623d5700  5 -- op tracker -- , seq: 747612, time: 2014-06-12 14:46:10.126025, event: started, op: osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.126046 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2655 (0'0,173'2655] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2654 active] append_log log((0'0,173'2655], crt=0'0) [173'2656 (173'725) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146404 2014-06-12 08:41:29.542776]
2014-06-12 14:46:10.126068 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2656 (0'0,173'2656] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2654 active] add_log_entry 173'2656 (173'725) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146404 2014-06-12 08:41:29.542776
2014-06-12 14:46:10.126092 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2656 (0'0,173'2656] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2654 active] append_log  adding 1 keys
2014-06-12 14:46:10.126132 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2656, trimmed: 
2014-06-12 14:46:10.126156 7f9d623d5700  5 -- op tracker -- , seq: 747612, time: 2014-06-12 14:46:10.126152, event: started, op: osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.126166 7f9d623d5700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:10.126171 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.126173 7f9d623d5700  5 queue_op 0xbdfda40 seq 2466 osr(3.1b7 0x276d1a0) 297205 bytes   (queue has 1 ops and 297205 bytes)
2014-06-12 14:46:10.126179 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2656 (0'0,173'2656] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2654 active] do_sub_op osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.126195 7f9d623d5700 10 osd.2 173 dequeue_op 0x9facc00 finish
2014-06-12 14:46:10.126199 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.126202 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.126211 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:46:10.126217 7f9d7e169700  5 _do_op 0xbdfda40 seq 2466 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:10.126222 7f9d7e169700 10 _do_transaction on 0x2f528b70
2014-06-12 14:46:10.126230 7f9d7e169700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:10.126358 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:10.126383 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.126428 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.126541 7f9d7e169700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:10.126550 7f9d7e169700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:10.126557 7f9d7e169700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:10.126576 7f9d7e169700 15 _write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 0~294912
2014-06-12 14:46:10.126711 7f9d7e169700 10 lookup_strip_header done  cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:10.126950 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:10.127781 7f9d7e169700 10 _generic_write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 0~294912 = 0
2014-06-12 14:46:10.127825 7f9d7e169700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:10.127842 7f9d7e169700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:10.127851 7f9d7e169700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:10.127861 7f9d7e169700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:10.127983 7f9d7e169700 20 set_header setting 16921 cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 parent seq 0
2014-06-12 14:46:10.128073 7f9d7e169700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:10.128092 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.128098 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:10.129719 7f9d7e169700 10 _do_op 0xbdfda40 seq 2466 r = 0, finisher 0x13bb7a80 0
2014-06-12 14:46:10.129737 7f9d7e169700 10 _finish_op 0xbdfda40 seq 2466 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:10.129743 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:46:10.129752 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.129837 7f9d7715b700  5 -- op tracker -- , seq: 747612, time: 2014-06-12 14:46:10.129826, event: commit_sent, op: osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.129849 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2656 (0'0,173'2656] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2654 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:10.129919 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:10.129955 7f9d7795c700  5 -- op tracker -- , seq: 747612, time: 2014-06-12 14:46:10.129950, event: sub_op_applied, op: osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.129994 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2656 (0'0,173'2656] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2655 active] sub_op_modify_applied on 0x2f528a80 op osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.130010 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2656 (0'0,173'2656] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2655 active] op_applied on primary on version 173'2656
2014-06-12 14:46:10.130019 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2656 (0'0,173'2656] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2655 active] op_applied on replica on version 173'2656
2014-06-12 14:46:10.130046 7f9d7795c700  5 -- op tracker -- , seq: 747612, time: 2014-06-12 14:46:10.130042, event: done, op: osd_sub_op(client.4316.0:146404 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2656 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.153219 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:10.184280 7f9d5a16f700  5 -- op tracker -- , seq: 747613, time: 2014-06-12 14:46:10.183685, event: header_read, op: osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.184298 7f9d5a16f700  5 -- op tracker -- , seq: 747613, time: 2014-06-12 14:46:10.183687, event: throttled, op: osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.184307 7f9d5a16f700  5 -- op tracker -- , seq: 747613, time: 2014-06-12 14:46:10.184225, event: all_read, op: osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.184316 7f9d5a16f700  5 -- op tracker -- , seq: 747613, time: 0.000000, event: dispatched, op: osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.184327 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.184338 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:10.184356 7f9d5a16f700 15 osd.2 173 enqueue_op 0x9fac800 prio 127 cost 197415 latency 0.000670 osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.185281 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.185295 7f9d643d9700 10 osd.2 173 dequeue_op 0x9fac800 prio 127 cost 197415 latency 0.001610 osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2718 (0'0,173'2718] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active]
2014-06-12 14:46:10.185326 7f9d643d9700  5 -- op tracker -- , seq: 747613, time: 2014-06-12 14:46:10.185321, event: reached_pg, op: osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.185336 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2718 (0'0,173'2718] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] handle_message: 0x9fac800
2014-06-12 14:46:10.185349 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2718 (0'0,173'2718] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] sub_op_modify trans e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 v 173'2719 (transaction) 174
2014-06-12 14:46:10.185378 7f9d643d9700  5 -- op tracker -- , seq: 747613, time: 2014-06-12 14:46:10.185373, event: started, op: osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.185392 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2718 (0'0,173'2718] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] append_log log((0'0,173'2718], crt=0'0) [173'2719 (173'2718) modify   e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 by client.4285.0:143306 2014-06-12 08:42:57.402920]
2014-06-12 14:46:10.185418 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2719 (0'0,173'2719] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] add_log_entry 173'2719 (173'2718) modify   e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 by client.4285.0:143306 2014-06-12 08:42:57.402920
2014-06-12 14:46:10.185441 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2719 (0'0,173'2719] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] append_log  adding 1 keys
2014-06-12 14:46:10.185473 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2719, trimmed: 
2014-06-12 14:46:10.185496 7f9d643d9700  5 -- op tracker -- , seq: 747613, time: 2014-06-12 14:46:10.185491, event: started, op: osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.185505 7f9d643d9700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:10.185509 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.185511 7f9d643d9700  5 queue_op 0x7c9b440 seq 2466 osr(3.21c 0x276d020) 198901 bytes   (queue has 1 ops and 198901 bytes)
2014-06-12 14:46:10.185522 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (1 active)
2014-06-12 14:46:10.185530 7f9d7e96a700  5 _do_op 0x7c9b440 seq 2466 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:46:10.185518 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2719 (0'0,173'2719] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] do_sub_op osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.185533 7f9d643d9700 10 osd.2 173 dequeue_op 0x9fac800 finish
2014-06-12 14:46:10.185535 7f9d7e96a700 10 _do_transaction on 0x62c82b0
2014-06-12 14:46:10.185537 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.185539 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.185540 7f9d7e96a700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:10.185887 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:10.185911 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.185962 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.185982 7f9d7e96a700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:10.185987 7f9d7e96a700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:46:10.185992 7f9d7e96a700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:10.186007 7f9d7e96a700 15 _write 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 3997696~196608
2014-06-12 14:46:10.186145 7f9d7e96a700 10 lookup_strip_header done  cid 3.21c_head oid e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:46:10.186378 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:10.187105 7f9d7e96a700 10 _generic_write 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 3997696~196608 = 0
2014-06-12 14:46:10.187135 7f9d7e96a700 15 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:46:10.187148 7f9d7e96a700 10 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 = 0
2014-06-12 14:46:10.187160 7f9d7e96a700 15 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3
2014-06-12 14:46:10.187171 7f9d7e96a700 10 _setattrs 3.21c_head/e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 = 0
2014-06-12 14:46:10.187320 7f9d7e96a700 20 set_header setting 17494 cid 3.21c_head oid e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 parent seq 0
2014-06-12 14:46:10.187379 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.187393 7f9d7e96a700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:46:10.187402 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:10.189015 7f9d7e96a700 10 _do_op 0x7c9b440 seq 2466 r = 0, finisher 0x6570d20 0
2014-06-12 14:46:10.189032 7f9d7e96a700 10 _finish_op 0x7c9b440 seq 2466 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:10.189044 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (0 active)
2014-06-12 14:46:10.189055 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.189056 7f9d7715b700  5 -- op tracker -- , seq: 747613, time: 2014-06-12 14:46:10.189042, event: commit_sent, op: osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.189071 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2719 (0'0,173'2719] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] sub_op_modify_commit on op osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:10.189103 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:10.189166 7f9d7795c700  5 -- op tracker -- , seq: 747613, time: 2014-06-12 14:46:10.189160, event: sub_op_applied, op: osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.189175 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2719 (0'0,173'2719] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] sub_op_modify_applied on 0x62c81c0 op osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.189190 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2719 (0'0,173'2719] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] op_applied on primary on version 173'2719
2014-06-12 14:46:10.189200 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2719 (0'0,173'2719] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] op_applied on replica on version 173'2719
2014-06-12 14:46:10.189241 7f9d7795c700  5 -- op tracker -- , seq: 747613, time: 2014-06-12 14:46:10.189237, event: done, op: osd_sub_op(client.4285.0:143306 3.21c e2f25a1c/rbd_data.10946b8b4567.0000000000001104/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.203212 7f9d5a16f700  5 -- op tracker -- , seq: 747614, time: 2014-06-12 14:46:10.202157, event: header_read, op: osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.203224 7f9d5a16f700  5 -- op tracker -- , seq: 747614, time: 2014-06-12 14:46:10.202159, event: throttled, op: osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.203233 7f9d5a16f700  5 -- op tracker -- , seq: 747614, time: 2014-06-12 14:46:10.203159, event: all_read, op: osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.203240 7f9d5a16f700  5 -- op tracker -- , seq: 747614, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.203246 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.203253 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:10.203262 7f9d5a16f700 15 osd.2 173 enqueue_op 0x9fac400 prio 127 cost 516903 latency 0.001105 osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.203299 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.203309 7f9d61bd4700 10 osd.2 173 dequeue_op 0x9fac400 prio 127 cost 516903 latency 0.001151 osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2656 (0'0,173'2656] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2655 active]
2014-06-12 14:46:10.203338 7f9d61bd4700  5 -- op tracker -- , seq: 747614, time: 2014-06-12 14:46:10.203333, event: reached_pg, op: osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.203348 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2656 (0'0,173'2656] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2655 active] handle_message: 0x9fac400
2014-06-12 14:46:10.203365 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2656 (0'0,173'2656] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2655 active] sub_op_modify trans 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 v 173'2657 (transaction) 174
2014-06-12 14:46:10.203383 7f9d61bd4700  5 -- op tracker -- , seq: 747614, time: 2014-06-12 14:46:10.203379, event: started, op: osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.203406 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2656 (0'0,173'2656] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2655 active] append_log log((0'0,173'2656], crt=0'0) [173'2657 (173'2655) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141063 2014-06-12 08:43:32.951361]
2014-06-12 14:46:10.203434 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2657 (0'0,173'2657] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2655 active] add_log_entry 173'2657 (173'2655) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141063 2014-06-12 08:43:32.951361
2014-06-12 14:46:10.203457 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2657 (0'0,173'2657] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2655 active] append_log  adding 1 keys
2014-06-12 14:46:10.203496 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2657, trimmed: 
2014-06-12 14:46:10.203522 7f9d61bd4700  5 -- op tracker -- , seq: 747614, time: 2014-06-12 14:46:10.203517, event: started, op: osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.203533 7f9d61bd4700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:10.203538 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.203539 7f9d61bd4700  5 queue_op 0x5a4d2c0 seq 2467 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:10.203549 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2657 (0'0,173'2657] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2655 active] do_sub_op osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.203565 7f9d61bd4700 10 osd.2 173 dequeue_op 0x9fac400 finish
2014-06-12 14:46:10.203569 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.203571 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.203580 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:46:10.203587 7f9d82171700  5 _do_op 0x5a4d2c0 seq 2467 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:10.203592 7f9d82171700 10 _do_transaction on 0x1da3e630
2014-06-12 14:46:10.203599 7f9d82171700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:10.203734 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:10.203764 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.203819 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.203839 7f9d82171700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:10.203845 7f9d82171700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:10.203851 7f9d82171700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:10.203864 7f9d82171700 15 _write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 516096~516096
2014-06-12 14:46:10.204024 7f9d82171700 10 lookup_strip_header done  cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:10.204205 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:10.206419 7f9d82171700 10 _generic_write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 516096~516096 = 0
2014-06-12 14:46:10.206480 7f9d82171700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:10.206495 7f9d82171700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:10.206501 7f9d82171700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:10.206510 7f9d82171700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:10.206716 7f9d82171700 20 set_header setting 17215 cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 parent seq 0
2014-06-12 14:46:10.206796 7f9d82171700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:10.206811 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.206820 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:10.209457 7f9d82171700 10 _do_op 0x5a4d2c0 seq 2467 r = 0, finisher 0x6f57620 0
2014-06-12 14:46:10.209482 7f9d82171700 10 _finish_op 0x5a4d2c0 seq 2467 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:10.209503 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:46:10.209509 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.209506 7f9d7715b700  5 -- op tracker -- , seq: 747614, time: 2014-06-12 14:46:10.209494, event: commit_sent, op: osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.209525 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2657 (0'0,173'2657] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2655 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:10.209555 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:10.209593 7f9d7795c700  5 -- op tracker -- , seq: 747614, time: 2014-06-12 14:46:10.209586, event: sub_op_applied, op: osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.209601 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2657 (0'0,173'2657] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active] sub_op_modify_applied on 0x1da3e540 op osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.209634 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2657 (0'0,173'2657] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active] op_applied on primary on version 173'2657
2014-06-12 14:46:10.209645 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2657 (0'0,173'2657] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active] op_applied on replica on version 173'2657
2014-06-12 14:46:10.209672 7f9d7795c700  5 -- op tracker -- , seq: 747614, time: 2014-06-12 14:46:10.209667, event: done, op: osd_sub_op(client.4284.0:141063 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2657 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.221381 7f9d56432700  5 -- op tracker -- , seq: 747615, time: 2014-06-12 14:46:10.219062, event: header_read, op: osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.221398 7f9d56432700  5 -- op tracker -- , seq: 747615, time: 2014-06-12 14:46:10.219065, event: throttled, op: osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.221408 7f9d56432700  5 -- op tracker -- , seq: 747615, time: 2014-06-12 14:46:10.221330, event: all_read, op: osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.221416 7f9d56432700  5 -- op tracker -- , seq: 747615, time: 0.000000, event: dispatched, op: osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.221422 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.221429 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:10.221437 7f9d56432700 15 osd.2 173 enqueue_op 0x2afc200 prio 127 cost 951079 latency 0.002374 osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.221474 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.221482 7f9d663dd700 10 osd.2 173 dequeue_op 0x2afc200 prio 127 cost 951079 latency 0.002420 osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.15c( v 173'2644 (0'0,173'2644] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active]
2014-06-12 14:46:10.221507 7f9d663dd700  5 -- op tracker -- , seq: 747615, time: 2014-06-12 14:46:10.221503, event: reached_pg, op: osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.221517 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2644 (0'0,173'2644] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] handle_message: 0x2afc200
2014-06-12 14:46:10.221529 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2644 (0'0,173'2644] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] sub_op_modify trans 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 v 173'2645 (transaction) 174
2014-06-12 14:46:10.221546 7f9d663dd700  5 -- op tracker -- , seq: 747615, time: 2014-06-12 14:46:10.221542, event: started, op: osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.221561 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2644 (0'0,173'2644] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] append_log log((0'0,173'2644], crt=0'0) [173'2645 (173'2644) modify   805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 by client.4279.0:143442 2014-06-12 08:44:31.525158]
2014-06-12 14:46:10.221587 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2645 (0'0,173'2645] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] add_log_entry 173'2645 (173'2644) modify   805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 by client.4279.0:143442 2014-06-12 08:44:31.525158
2014-06-12 14:46:10.221607 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2645 (0'0,173'2645] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] append_log  adding 1 keys
2014-06-12 14:46:10.221653 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2645, trimmed: 
2014-06-12 14:46:10.221681 7f9d663dd700  5 -- op tracker -- , seq: 747615, time: 2014-06-12 14:46:10.221676, event: started, op: osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.221695 7f9d663dd700  5 queue_transactions existing osr(3.15c 0x276ced0)/0x276ced0
2014-06-12 14:46:10.221702 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.221703 7f9d663dd700  5 queue_op 0x90adb00 seq 2441 osr(3.15c 0x276ced0) 952565 bytes   (queue has 1 ops and 952565 bytes)
2014-06-12 14:46:10.221715 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54bde60 (1 active)
2014-06-12 14:46:10.221723 7f9d7815d700  5 _do_op 0x90adb00 seq 2441 osr(3.15c 0x276ced0)/0x276ced0 start
2014-06-12 14:46:10.221727 7f9d7815d700 10 _do_transaction on 0x1ea81970
2014-06-12 14:46:10.221711 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.15c( v 173'2645 (0'0,173'2645] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] do_sub_op osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.221732 7f9d663dd700 10 osd.2 173 dequeue_op 0x2afc200 finish
2014-06-12 14:46:10.221733 7f9d7815d700 15 _omap_setkeys meta/1fe2b14b/pglog_3.15c/0//-1
2014-06-12 14:46:10.221736 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.221738 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.221903 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe2b14b/pglog_3.15c/0//-1
2014-06-12 14:46:10.221928 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.221983 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.222100 7f9d7815d700 15 _omap_rmkeys meta/1fe2b14b/pglog_3.15c/0//-1
2014-06-12 14:46:10.222108 7f9d7815d700 10 _omap_rmkeys meta/1fe2b14b/pglog_3.15c/0//-1 r = 0
2014-06-12 14:46:10.222114 7f9d7815d700 15 _omap_setkeys meta/1fe2b14b/pglog_3.15c/0//-1
2014-06-12 14:46:10.222134 7f9d7815d700 15 _write 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 3244032~950272
2014-06-12 14:46:10.222253 7f9d56432700  5 -- op tracker -- , seq: 747616, time: 2014-06-12 14:46:10.222215, event: header_read, op: osd_sub_op_reply(client.4306.0:144046 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.222262 7f9d56432700  5 -- op tracker -- , seq: 747616, time: 2014-06-12 14:46:10.222217, event: throttled, op: osd_sub_op_reply(client.4306.0:144046 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.222269 7f9d56432700  5 -- op tracker -- , seq: 747616, time: 2014-06-12 14:46:10.222230, event: all_read, op: osd_sub_op_reply(client.4306.0:144046 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.222276 7f9d56432700  5 -- op tracker -- , seq: 747616, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144046 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.222281 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144046 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:10.222286 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:10.222292 7f9d56432700 15 osd.2 173 enqueue_op 0x2afc900 prio 196 cost 0 latency 0.000077 osd_sub_op_reply(client.4306.0:144046 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:10.222428 7f9d7815d700 10 lookup_strip_header done  cid 3.15c_head oid 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3
2014-06-12 14:46:10.222465 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:10.226527 7f9d7815d700 10 _generic_write 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 3244032~950272 = 0
2014-06-12 14:46:10.226688 7f9d7815d700 15 _setattrs 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3
2014-06-12 14:46:10.226705 7f9d7815d700 10 _setattrs 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:10.226711 7f9d7815d700 15 _setattrs 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3
2014-06-12 14:46:10.226720 7f9d7815d700 10 _setattrs 3.15c_head/805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:10.227067 7f9d7815d700 20 set_header setting 16993 cid 3.15c_head oid 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 parent seq 0
2014-06-12 14:46:10.227192 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.227207 7f9d7815d700 20 set_header setting 2322 cid meta oid 1fe2b14b/pglog_3.15c/0//-1 parent seq 0
2014-06-12 14:46:10.227218 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:10.233891 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.233910 7f9d673df700 10 osd.2 173 dequeue_op 0x2afc900 prio 196 cost 0 latency 0.011694 osd_sub_op_reply(client.4306.0:144046 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3081 active+clean]
2014-06-12 14:46:10.233941 7f9d673df700  5 -- op tracker -- , seq: 747616, time: 2014-06-12 14:46:10.233936, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144046 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.233957 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3081 active+clean] handle_message: 0x2afc900
2014-06-12 14:46:10.233976 7f9d673df700  5 -- op tracker -- , seq: 747616, time: 2014-06-12 14:46:10.233972, event: started, op: osd_sub_op_reply(client.4306.0:144046 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.233983 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3081 active+clean] sub_op_modify_reply: tid 236002 op  ack_type 4 from 26
2014-06-12 14:46:10.234006 7f9d673df700  5 -- op tracker -- , seq: 746933, time: 2014-06-12 14:46:10.233996, event: sub_op_commit_rec, op: osd_op(client.4306.0:144046 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 561152~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.234014 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3081 active+clean] repop_all_applied: repop tid 236002 all applied 
2014-06-12 14:46:10.234025 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3081 active+clean] eval_repop repgather(0x32b68d80 173'3083 rep_tid=236002 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144046 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 561152~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.234042 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3081 active+clean]  sending ack on repgather(0x32b68d80 173'3083 rep_tid=236002 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144046 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 561152~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x22b53900
2014-06-12 14:46:10.234080 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3082 crt=0'0 lcod 173'3081 mlcod 173'3081 active+clean] repop_all_committed: repop tid 236002 all committed 
2014-06-12 14:46:10.234091 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3081 active+clean] eval_repop repgather(0x32b68d80 173'3083 rep_tid=236002 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144046 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 561152~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.234107 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3081 active+clean] log_op_stats osd_op(client.4306.0:144046 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 561152~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 38.344138 lat 38.344166
2014-06-12 14:46:10.234124 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3081 active+clean] publish_stats_to_osd 173:3549
2014-06-12 14:46:10.234138 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3081 active+clean]  sending commit on repgather(0x32b68d80 173'3083 rep_tid=236002 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144046 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 561152~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x9656280
2014-06-12 14:46:10.234166 7f9d673df700  5 -- op tracker -- , seq: 746933, time: 2014-06-12 14:46:10.234161, event: commit_sent, op: osd_op(client.4306.0:144046 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 561152~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.234174 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3081 active+clean]  requeue_ops 
2014-06-12 14:46:10.234185 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3082 active+clean]  removing repgather(0x32b68d80 173'3083 rep_tid=236002 committed?=1 applied?=1 op=osd_op(client.4306.0:144046 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 561152~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.234200 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3082 active+clean]    q front is repgather(0x32b68d80 173'3083 rep_tid=236002 committed?=1 applied?=1 op=osd_op(client.4306.0:144046 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 561152~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.234213 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3082 active+clean] remove_repop repgather(0x32b68d80 173'3083 rep_tid=236002 committed?=1 applied?=1 op=osd_op(client.4306.0:144046 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 561152~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.234227 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3082 active+clean]  requeue_ops 
2014-06-12 14:46:10.234248 7f9d673df700  5 -- op tracker -- , seq: 746933, time: 2014-06-12 14:46:10.234243, event: done, op: osd_op(client.4306.0:144046 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 561152~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.234268 7f9d673df700 10 osd.2 173 dequeue_op 0x2afc900 finish
2014-06-12 14:46:10.234275 7f9d673df700  5 -- op tracker -- , seq: 747616, time: 2014-06-12 14:46:10.234272, event: done, op: osd_sub_op_reply(client.4306.0:144046 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.234288 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.234291 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.237571 7f9d7815d700 10 _do_op 0x90adb00 seq 2441 r = 0, finisher 0x7b37a80 0
2014-06-12 14:46:10.237597 7f9d7815d700 10 _finish_op 0x90adb00 seq 2441 osr(3.15c 0x276ced0)/0x276ced0
2014-06-12 14:46:10.237692 7f9d7795c700  5 -- op tracker -- , seq: 747615, time: 2014-06-12 14:46:10.237682, event: sub_op_applied, op: osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.237704 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2645 (0'0,173'2645] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] sub_op_modify_applied on 0x1ea81880 op osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.237734 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:10.237756 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2645 (0'0,173'2645] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] op_applied on primary on version 173'2645
2014-06-12 14:46:10.237774 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2645 (0'0,173'2645] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] op_applied on replica on version 173'2645
2014-06-12 14:46:10.237812 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54bde60 (0 active)
2014-06-12 14:46:10.237827 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.241680 7f9d7715b700  5 -- op tracker -- , seq: 747615, time: 2014-06-12 14:46:10.241656, event: commit_sent, op: osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.241700 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.15c( v 173'2645 (0'0,173'2645] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] sub_op_modify_commit on op osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:10.241737 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:10.241796 7f9d7715b700  5 -- op tracker -- , seq: 747615, time: 2014-06-12 14:46:10.241791, event: done, op: osd_sub_op(client.4279.0:143442 3.15c 805a395c/rbd_data.108e6b8b4567.000000000000102c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.341879 7f9d56432700  5 -- op tracker -- , seq: 747617, time: 2014-06-12 14:46:10.340551, event: header_read, op: osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.341893 7f9d56432700  5 -- op tracker -- , seq: 747617, time: 2014-06-12 14:46:10.340554, event: throttled, op: osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.341915 7f9d56432700  5 -- op tracker -- , seq: 747617, time: 2014-06-12 14:46:10.341835, event: all_read, op: osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.341925 7f9d56432700  5 -- op tracker -- , seq: 747617, time: 0.000000, event: dispatched, op: osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.341931 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.341939 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:10.341949 7f9d56432700 15 osd.2 173 enqueue_op 0x2910000 prio 127 cost 713507 latency 0.001398 osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.341978 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.341984 7f9d693e3700 10 osd.2 173 dequeue_op 0x2910000 prio 127 cost 713507 latency 0.001433 osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2475 (0'0,173'2475] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active]
2014-06-12 14:46:10.342002 7f9d693e3700  5 -- op tracker -- , seq: 747617, time: 2014-06-12 14:46:10.341999, event: reached_pg, op: osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.342009 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2475 (0'0,173'2475] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active] handle_message: 0x2910000
2014-06-12 14:46:10.342024 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2475 (0'0,173'2475] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active] sub_op_modify trans 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 v 173'2476 (transaction) 174
2014-06-12 14:46:10.342036 7f9d693e3700  5 -- op tracker -- , seq: 747617, time: 2014-06-12 14:46:10.342033, event: started, op: osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.342052 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2475 (0'0,173'2475] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active] append_log log((0'0,173'2475], crt=0'0) [173'2476 (173'2474) modify   4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 by client.4280.0:139529 2014-06-12 08:43:28.503470]
2014-06-12 14:46:10.342070 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2476 (0'0,173'2476] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active] add_log_entry 173'2476 (173'2474) modify   4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 by client.4280.0:139529 2014-06-12 08:43:28.503470
2014-06-12 14:46:10.342087 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2476 (0'0,173'2476] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active] append_log  adding 1 keys
2014-06-12 14:46:10.342112 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2476, trimmed: 
2014-06-12 14:46:10.342130 7f9d693e3700  5 -- op tracker -- , seq: 747617, time: 2014-06-12 14:46:10.342127, event: started, op: osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.342138 7f9d693e3700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:10.342142 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.342143 7f9d693e3700  5 queue_op 0x5c75da0 seq 2289 osr(3.2b 0x276cd20) 714988 bytes   (queue has 1 ops and 714988 bytes)
2014-06-12 14:46:10.342154 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:46:10.342149 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2476 (0'0,173'2476] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active] do_sub_op osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.342161 7f9d7d167700  5 _do_op 0x5c75da0 seq 2289 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:46:10.342162 7f9d693e3700 10 osd.2 173 dequeue_op 0x2910000 finish
2014-06-12 14:46:10.342165 7f9d7d167700 10 _do_transaction on 0x12c257b0
2014-06-12 14:46:10.342165 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.342168 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.342172 7f9d7d167700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:10.342365 7f9d7d167700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:10.342383 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.342425 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.342441 7f9d7d167700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:10.342446 7f9d7d167700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:46:10.342450 7f9d7d167700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:10.342462 7f9d7d167700 15 _write 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 3481600~712704
2014-06-12 14:46:10.342693 7f9d7d167700 10 lookup_strip_header done  cid 3.2b_head oid 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:46:10.342797 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:10.344570 7f9d7d167700 10 _generic_write 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 3481600~712704 = 0
2014-06-12 14:46:10.344618 7f9d7d167700 15 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:46:10.344629 7f9d7d167700 10 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 = 0
2014-06-12 14:46:10.344633 7f9d7d167700 15 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3
2014-06-12 14:46:10.344639 7f9d7d167700 10 _setattrs 3.2b_head/4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 = 0
2014-06-12 14:46:10.344809 7f9d7d167700 20 set_header setting 15771 cid 3.2b_head oid 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 parent seq 0
2014-06-12 14:46:10.344876 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.344887 7f9d7d167700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:46:10.344893 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:10.347836 7f9d7d167700 10 _do_op 0x5c75da0 seq 2289 r = 0, finisher 0x65dda40 0
2014-06-12 14:46:10.347861 7f9d7d167700 10 _finish_op 0x5c75da0 seq 2289 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:10.347868 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:46:10.347876 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.347893 7f9d7795c700  5 -- op tracker -- , seq: 747617, time: 2014-06-12 14:46:10.347879, event: sub_op_applied, op: osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.347906 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2476 (0'0,173'2476] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active] sub_op_modify_applied on 0x12c256c0 op osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.347936 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:10.347957 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2476 (0'0,173'2476] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active] op_applied on primary on version 173'2476
2014-06-12 14:46:10.347972 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2476 (0'0,173'2476] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active] op_applied on replica on version 173'2476
2014-06-12 14:46:10.348003 7f9d7715b700  5 -- op tracker -- , seq: 747617, time: 2014-06-12 14:46:10.347997, event: commit_sent, op: osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.348013 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2476 (0'0,173'2476] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2474 active] sub_op_modify_commit on op osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:10.348032 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:10.348063 7f9d7715b700  5 -- op tracker -- , seq: 747617, time: 2014-06-12 14:46:10.348058, event: done, op: osd_sub_op(client.4280.0:139529 3.2b 4f7b282b/rbd_data.108f6b8b4567.0000000000000e5d/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.366878 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:10.392608 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:10.393146 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:10.438026 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:10.438071 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:10.470903 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:10.470932 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:10.470935 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:10.470937 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:10.470939 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 30; oldest is 38.392318 seconds old
2014-06-12 14:46:10.470938 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:10.471006 7f9d84175700  0 log [WRN] : 16 slow requests, 6 included below; oldest blocked for > 38.392318 secs
2014-06-12 14:46:10.471013 7f9d84175700  0 log [WRN] : slow request 31.331960 seconds old, received at 2014-06-12 14:45:39.138978: osd_op(client.4303.0:144623 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 28672~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:10.471017 7f9d84175700  0 log [WRN] : slow request 31.254160 seconds old, received at 2014-06-12 14:45:39.216778: osd_op(client.4303.0:144624 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:10.471021 7f9d84175700  0 log [WRN] : slow request 31.222019 seconds old, received at 2014-06-12 14:45:39.248919: osd_op(client.4303.0:144625 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:10.471024 7f9d84175700  0 log [WRN] : slow request 31.143352 seconds old, received at 2014-06-12 14:45:39.327586: osd_op(client.4303.0:144626 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:10.471027 7f9d84175700  0 log [WRN] : slow request 31.115276 seconds old, received at 2014-06-12 14:45:39.355662: osd_op(client.4303.0:144627 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:10.513573 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:10.513926 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:10.559342 7f9d56432700  5 -- op tracker -- , seq: 747618, time: 2014-06-12 14:46:10.557439, event: header_read, op: osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.559368 7f9d56432700  5 -- op tracker -- , seq: 747618, time: 2014-06-12 14:46:10.557440, event: throttled, op: osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.559377 7f9d56432700  5 -- op tracker -- , seq: 747618, time: 2014-06-12 14:46:10.559305, event: all_read, op: osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.559391 7f9d56432700  5 -- op tracker -- , seq: 747618, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.559395 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.559401 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:10.559408 7f9d56432700 15 osd.2 173 enqueue_op 0x23e08200 prio 127 cost 1032999 latency 0.001969 osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.559438 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.559445 7f9d67be0700 10 osd.2 173 dequeue_op 0x23e08200 prio 127 cost 1032999 latency 0.002006 osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2990 (0'0,173'2990] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2989 active]
2014-06-12 14:46:10.559466 7f9d67be0700  5 -- op tracker -- , seq: 747618, time: 2014-06-12 14:46:10.559462, event: reached_pg, op: osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.559484 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2990 (0'0,173'2990] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2989 active] handle_message: 0x23e08200
2014-06-12 14:46:10.559497 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2990 (0'0,173'2990] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2989 active] sub_op_modify trans 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 v 173'2991 (transaction) 174
2014-06-12 14:46:10.559513 7f9d67be0700  5 -- op tracker -- , seq: 747618, time: 2014-06-12 14:46:10.559509, event: started, op: osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.559525 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2990 (0'0,173'2990] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2989 active] append_log log((0'0,173'2990], crt=0'0) [173'2991 (173'2989) modify   3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 by client.4298.0:145090 2014-06-12 08:43:26.691341]
2014-06-12 14:46:10.559547 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2991 (0'0,173'2991] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2989 active] add_log_entry 173'2991 (173'2989) modify   3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 by client.4298.0:145090 2014-06-12 08:43:26.691341
2014-06-12 14:46:10.559569 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2991 (0'0,173'2991] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2989 active] append_log  adding 1 keys
2014-06-12 14:46:10.559599 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2991, trimmed: 
2014-06-12 14:46:10.559620 7f9d67be0700  5 -- op tracker -- , seq: 747618, time: 2014-06-12 14:46:10.559615, event: started, op: osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.559628 7f9d67be0700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:10.559631 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.559633 7f9d67be0700  5 queue_op 0x285a2b40 seq 2793 osr(3.458 0x276d960) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:10.559640 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2991 (0'0,173'2991] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2989 active] do_sub_op osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.559656 7f9d67be0700 10 osd.2 173 dequeue_op 0x23e08200 finish
2014-06-12 14:46:10.559659 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.559662 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.559670 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:10.559674 7f9d79960700  5 _do_op 0x285a2b40 seq 2793 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:10.559678 7f9d79960700 10 _do_transaction on 0x1cef67f0
2014-06-12 14:46:10.559683 7f9d79960700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:10.559807 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:10.559823 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.559876 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.559891 7f9d79960700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:10.559897 7f9d79960700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:10.559903 7f9d79960700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:10.559916 7f9d79960700 15 _write 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 2318336~1032192
2014-06-12 14:46:10.560109 7f9d79960700 10 lookup_strip_header done  cid 3.458_head oid 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:46:10.560290 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:10.564215 7f9d79960700 10 _generic_write 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 2318336~1032192 = 0
2014-06-12 14:46:10.564281 7f9d79960700 15 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:46:10.564299 7f9d79960700 10 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 = 0
2014-06-12 14:46:10.564308 7f9d79960700 15 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:46:10.564312 7f9d79960700 10 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 = 0
2014-06-12 14:46:10.564520 7f9d79960700 20 set_header setting 16953 cid 3.458_head oid 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 parent seq 0
2014-06-12 14:46:10.564602 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.564612 7f9d79960700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:10.564618 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:10.573465 7f9d79960700 10 _do_op 0x285a2b40 seq 2793 r = 0, finisher 0x82c6e00 0
2014-06-12 14:46:10.573483 7f9d79960700 10 _finish_op 0x285a2b40 seq 2793 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:10.573488 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:10.573494 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.573521 7f9d7715b700  5 -- op tracker -- , seq: 747618, time: 2014-06-12 14:46:10.573512, event: commit_sent, op: osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.573531 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2991 (0'0,173'2991] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2989 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:10.573554 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:10.573770 7f9d7795c700  5 -- op tracker -- , seq: 747618, time: 2014-06-12 14:46:10.573765, event: sub_op_applied, op: osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.573778 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2991 (0'0,173'2991] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify_applied on 0x1cef6700 op osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.573796 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2991 (0'0,173'2991] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2990 active] op_applied on primary on version 173'2991
2014-06-12 14:46:10.573808 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2991 (0'0,173'2991] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2990 active] op_applied on replica on version 173'2991
2014-06-12 14:46:10.573836 7f9d7795c700  5 -- op tracker -- , seq: 747618, time: 2014-06-12 14:46:10.573832, event: done, op: osd_sub_op(client.4298.0:145090 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.687488 7f9d58c5a700  5 -- op tracker -- , seq: 747619, time: 2014-06-12 14:46:10.687308, event: header_read, op: osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.687505 7f9d58c5a700  5 -- op tracker -- , seq: 747619, time: 2014-06-12 14:46:10.687310, event: throttled, op: osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.687548 7f9d58c5a700  5 -- op tracker -- , seq: 747619, time: 2014-06-12 14:46:10.687450, event: all_read, op: osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.687559 7f9d58c5a700  5 -- op tracker -- , seq: 747619, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.687564 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.687570 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:10.687579 7f9d58c5a700 15 osd.2 173 enqueue_op 0x2b014000 prio 127 cost 25383 latency 0.000271 osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.687608 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.687616 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2b014000 prio 127 cost 25383 latency 0.000307 osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3157 (79'156,173'3157] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active]
2014-06-12 14:46:10.687638 7f9d6b3e7700  5 -- op tracker -- , seq: 747619, time: 2014-06-12 14:46:10.687634, event: reached_pg, op: osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.687652 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3157 (79'156,173'3157] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active] handle_message: 0x2b014000
2014-06-12 14:46:10.687663 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3157 (79'156,173'3157] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active] sub_op_modify trans 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 v 173'3158 (transaction) 174
2014-06-12 14:46:10.687677 7f9d6b3e7700  5 -- op tracker -- , seq: 747619, time: 2014-06-12 14:46:10.687674, event: started, op: osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.687689 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3157 (79'156,173'3157] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active] append_log log((79'156,173'3157], crt=0'0) [173'3158 (173'856) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140116 2014-06-12 08:45:13.064086]
2014-06-12 14:46:10.687710 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3158 (79'156,173'3158] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active] add_log_entry 173'3158 (173'856) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140116 2014-06-12 08:45:13.064086
2014-06-12 14:46:10.687729 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3158 (79'156,173'3158] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active] append_log  adding 1 keys
2014-06-12 14:46:10.687740 7f9d6b3e7700 10 trim log((79'156,173'3158], crt=0'0) to 79'157
2014-06-12 14:46:10.687766 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3158, trimmed: 79'157
2014-06-12 14:46:10.687787 7f9d6b3e7700  5 -- op tracker -- , seq: 747619, time: 2014-06-12 14:46:10.687783, event: started, op: osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.687800 7f9d6b3e7700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.687803 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.687805 7f9d6b3e7700  5 queue_op 0x68ea120 seq 2853 osr(3.36c 0x276d400) 26904 bytes   (queue has 1 ops and 26904 bytes)
2014-06-12 14:46:10.687810 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3158 (79'157,173'3158] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active] do_sub_op osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.687823 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2b014000 finish
2014-06-12 14:46:10.687827 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.687829 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.687837 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (1 active)
2014-06-12 14:46:10.687842 7f9d8116f700  5 _do_op 0x68ea120 seq 2853 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:10.687846 7f9d8116f700 10 _do_transaction on 0x5880470
2014-06-12 14:46:10.687850 7f9d8116f700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.687983 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.687998 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.688046 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.688150 7f9d8116f700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.688157 7f9d8116f700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:10.688162 7f9d8116f700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.688180 7f9d8116f700 15 _write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 0~24576
2014-06-12 14:46:10.702444 7f9d56432700  5 -- op tracker -- , seq: 747620, time: 2014-06-12 14:46:10.702390, event: header_read, op: osd_sub_op_reply(client.4306.0:144047 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.702457 7f9d56432700  5 -- op tracker -- , seq: 747620, time: 2014-06-12 14:46:10.702391, event: throttled, op: osd_sub_op_reply(client.4306.0:144047 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.702466 7f9d56432700  5 -- op tracker -- , seq: 747620, time: 2014-06-12 14:46:10.702403, event: all_read, op: osd_sub_op_reply(client.4306.0:144047 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.702473 7f9d56432700  5 -- op tracker -- , seq: 747620, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144047 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.702480 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144047 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:10.702486 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:10.702496 7f9d56432700 15 osd.2 173 enqueue_op 0x6f4b000 prio 196 cost 0 latency 0.000106 osd_sub_op_reply(client.4306.0:144047 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:10.702533 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.702549 7f9d56432700  5 -- op tracker -- , seq: 747621, time: 2014-06-12 14:46:10.702526, event: header_read, op: osd_sub_op_reply(client.4306.0:144048 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.702541 7f9d62bd6700 10 osd.2 173 dequeue_op 0x6f4b000 prio 196 cost 0 latency 0.000150 osd_sub_op_reply(client.4306.0:144047 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3082 active+clean]
2014-06-12 14:46:10.702557 7f9d56432700  5 -- op tracker -- , seq: 747621, time: 2014-06-12 14:46:10.702526, event: throttled, op: osd_sub_op_reply(client.4306.0:144048 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.702567 7f9d56432700  5 -- op tracker -- , seq: 747621, time: 2014-06-12 14:46:10.702534, event: all_read, op: osd_sub_op_reply(client.4306.0:144048 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.702578 7f9d56432700  5 -- op tracker -- , seq: 747621, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144048 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.702583 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144048 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:10.702588 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:10.702587 7f9d62bd6700  5 -- op tracker -- , seq: 747620, time: 2014-06-12 14:46:10.702558, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144047 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.702593 7f9d56432700 15 osd.2 173 enqueue_op 0x28dd0200 prio 196 cost 0 latency 0.000067 osd_sub_op_reply(client.4306.0:144048 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:10.702596 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3082 active+clean] handle_message: 0x6f4b000
2014-06-12 14:46:10.702612 7f9d62bd6700  5 -- op tracker -- , seq: 747620, time: 2014-06-12 14:46:10.702609, event: started, op: osd_sub_op_reply(client.4306.0:144047 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.702620 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3082 active+clean] sub_op_modify_reply: tid 236003 op  ack_type 4 from 26
2014-06-12 14:46:10.702640 7f9d62bd6700  5 -- op tracker -- , seq: 746934, time: 2014-06-12 14:46:10.702633, event: sub_op_commit_rec, op: osd_op(client.4306.0:144047 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 1077248~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.702655 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3082 active+clean] repop_all_applied: repop tid 236003 all applied 
2014-06-12 14:46:10.702667 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3082 active+clean] eval_repop repgather(0x5146300 173'3084 rep_tid=236003 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144047 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 1077248~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.702683 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3082 active+clean]  sending ack on repgather(0x5146300 173'3084 rep_tid=236003 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144047 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 1077248~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x45df680
2014-06-12 14:46:10.702714 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3083 crt=0'0 lcod 173'3082 mlcod 173'3082 active+clean] repop_all_committed: repop tid 236003 all committed 
2014-06-12 14:46:10.702725 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3082 active+clean] eval_repop repgather(0x5146300 173'3084 rep_tid=236003 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144047 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 1077248~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.702741 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3082 active+clean] log_op_stats osd_op(client.4306.0:144047 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 1077248~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 38.624093 lat 38.624120
2014-06-12 14:46:10.702758 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3082 active+clean] publish_stats_to_osd 173:3550
2014-06-12 14:46:10.702770 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3082 active+clean]  sending commit on repgather(0x5146300 173'3084 rep_tid=236003 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144047 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 1077248~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x133fe500
2014-06-12 14:46:10.702799 7f9d62bd6700  5 -- op tracker -- , seq: 746934, time: 2014-06-12 14:46:10.702794, event: commit_sent, op: osd_op(client.4306.0:144047 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 1077248~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.702806 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3082 active+clean]  requeue_ops 
2014-06-12 14:46:10.702817 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3083 active+clean]  removing repgather(0x5146300 173'3084 rep_tid=236003 committed?=1 applied?=1 op=osd_op(client.4306.0:144047 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 1077248~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.702832 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3083 active+clean]    q front is repgather(0x5146300 173'3084 rep_tid=236003 committed?=1 applied?=1 op=osd_op(client.4306.0:144047 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 1077248~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.702847 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3083 active+clean] remove_repop repgather(0x5146300 173'3084 rep_tid=236003 committed?=1 applied?=1 op=osd_op(client.4306.0:144047 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 1077248~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.702860 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3083 active+clean]  requeue_ops 
2014-06-12 14:46:10.702881 7f9d62bd6700  5 -- op tracker -- , seq: 746934, time: 2014-06-12 14:46:10.702876, event: done, op: osd_op(client.4306.0:144047 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 1077248~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.702900 7f9d62bd6700 10 osd.2 173 dequeue_op 0x6f4b000 finish
2014-06-12 14:46:10.702907 7f9d62bd6700  5 -- op tracker -- , seq: 747620, time: 2014-06-12 14:46:10.702904, event: done, op: osd_sub_op_reply(client.4306.0:144047 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.702917 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.702925 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.702929 7f9d62bd6700 10 osd.2 173 dequeue_op 0x28dd0200 prio 196 cost 0 latency 0.000403 osd_sub_op_reply(client.4306.0:144048 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3083 active+clean]
2014-06-12 14:46:10.702945 7f9d62bd6700  5 -- op tracker -- , seq: 747621, time: 2014-06-12 14:46:10.702942, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144048 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.702952 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3083 active+clean] handle_message: 0x28dd0200
2014-06-12 14:46:10.702966 7f9d62bd6700  5 -- op tracker -- , seq: 747621, time: 2014-06-12 14:46:10.702963, event: started, op: osd_sub_op_reply(client.4306.0:144048 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.702972 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3083 active+clean] sub_op_modify_reply: tid 236016 op  ack_type 4 from 26
2014-06-12 14:46:10.702989 7f9d62bd6700  5 -- op tracker -- , seq: 746963, time: 2014-06-12 14:46:10.702984, event: sub_op_commit_rec, op: osd_op(client.4306.0:144048 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2109440~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.702996 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3083 active+clean] repop_all_applied: repop tid 236016 all applied 
2014-06-12 14:46:10.703007 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3083 active+clean] eval_repop repgather(0x15acf080 173'3085 rep_tid=236016 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144048 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2109440~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.703021 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3083 active+clean]  sending ack on repgather(0x15acf080 173'3085 rep_tid=236016 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144048 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2109440~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x151e7900
2014-06-12 14:46:10.703052 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3084 crt=0'0 lcod 173'3083 mlcod 173'3083 active+clean] repop_all_committed: repop tid 236016 all committed 
2014-06-12 14:46:10.703062 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3083 active+clean] eval_repop repgather(0x15acf080 173'3085 rep_tid=236016 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144048 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2109440~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.703077 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3083 active+clean] log_op_stats osd_op(client.4306.0:144048 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2109440~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 33.026761 lat 33.026786
2014-06-12 14:46:10.703092 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3083 active+clean] publish_stats_to_osd 173:3551
2014-06-12 14:46:10.703103 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3083 active+clean]  sending commit on repgather(0x15acf080 173'3085 rep_tid=236016 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144048 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2109440~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x3206a280
2014-06-12 14:46:10.703131 7f9d62bd6700  5 -- op tracker -- , seq: 746963, time: 2014-06-12 14:46:10.703126, event: commit_sent, op: osd_op(client.4306.0:144048 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2109440~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.703138 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3083 active+clean]  requeue_ops 
2014-06-12 14:46:10.703147 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3084 active+clean]  removing repgather(0x15acf080 173'3085 rep_tid=236016 committed?=1 applied?=1 op=osd_op(client.4306.0:144048 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2109440~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.703162 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3084 active+clean]    q front is repgather(0x15acf080 173'3085 rep_tid=236016 committed?=1 applied?=1 op=osd_op(client.4306.0:144048 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2109440~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.703176 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3084 active+clean] remove_repop repgather(0x15acf080 173'3085 rep_tid=236016 committed?=1 applied?=1 op=osd_op(client.4306.0:144048 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2109440~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.703190 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3084 active+clean]  requeue_ops 
2014-06-12 14:46:10.703208 7f9d62bd6700  5 -- op tracker -- , seq: 746963, time: 2014-06-12 14:46:10.703203, event: done, op: osd_op(client.4306.0:144048 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2109440~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.703220 7f9d62bd6700 10 osd.2 173 dequeue_op 0x28dd0200 finish
2014-06-12 14:46:10.703227 7f9d62bd6700  5 -- op tracker -- , seq: 747621, time: 2014-06-12 14:46:10.703224, event: done, op: osd_sub_op_reply(client.4306.0:144048 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.703237 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.703240 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.705021 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.709208 7f9d8116f700 10 lookup_strip_header done  cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.709378 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:10.709476 7f9d8116f700 10 _generic_write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 0~24576 = 0
2014-06-12 14:46:10.709493 7f9d8116f700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.709502 7f9d8116f700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.709507 7f9d8116f700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.709514 7f9d8116f700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.709634 7f9d8116f700 20 set_header setting 16547 cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:46:10.709648 7f9d56432700  5 -- op tracker -- , seq: 747622, time: 2014-06-12 14:46:10.702610, event: header_read, op: osd_sub_op_reply(client.4309.0:141523 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.709664 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.709665 7f9d56432700  5 -- op tracker -- , seq: 747622, time: 2014-06-12 14:46:10.702611, event: throttled, op: osd_sub_op_reply(client.4309.0:141523 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.709679 7f9d8116f700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:10.709673 7f9d56432700  5 -- op tracker -- , seq: 747622, time: 2014-06-12 14:46:10.702624, event: all_read, op: osd_sub_op_reply(client.4309.0:141523 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.709687 7f9d56432700  5 -- op tracker -- , seq: 747622, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141523 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.709690 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:10.709693 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141523 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:10.709699 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:10.709709 7f9d56432700 15 osd.2 173 enqueue_op 0x20e86800 prio 196 cost 0 latency 0.007098 osd_sub_op_reply(client.4309.0:141523 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:10.709936 7f9d8116f700 10 _do_op 0x68ea120 seq 2853 r = 0, finisher 0x73f9a20 0
2014-06-12 14:46:10.709952 7f9d8116f700 10 _finish_op 0x68ea120 seq 2853 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.709959 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (0 active)
2014-06-12 14:46:10.709962 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.711149 7f9d7795c700  5 -- op tracker -- , seq: 747619, time: 2014-06-12 14:46:10.711136, event: sub_op_applied, op: osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.711161 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3158 (79'157,173'3158] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active] sub_op_modify_applied on 0x5880380 op osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.711187 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.711204 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3158 (79'157,173'3158] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active] op_applied on primary on version 173'3158
2014-06-12 14:46:10.711214 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3158 (79'157,173'3158] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active] op_applied on replica on version 173'3158
2014-06-12 14:46:10.711644 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.711652 7f9d683e1700 10 osd.2 173 dequeue_op 0x20e86800 prio 196 cost 0 latency 0.009041 osd_sub_op_reply(client.4309.0:141523 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 pg pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2877 crt=0'0 lcod 173'2876 mlcod 173'2876 active+clean]
2014-06-12 14:46:10.711672 7f9d683e1700  5 -- op tracker -- , seq: 747622, time: 2014-06-12 14:46:10.711669, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141523 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.711681 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2877 crt=0'0 lcod 173'2876 mlcod 173'2876 active+clean] handle_message: 0x20e86800
2014-06-12 14:46:10.711695 7f9d683e1700  5 -- op tracker -- , seq: 747622, time: 2014-06-12 14:46:10.711692, event: started, op: osd_sub_op_reply(client.4309.0:141523 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.711702 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2877 crt=0'0 lcod 173'2876 mlcod 173'2876 active+clean] sub_op_modify_reply: tid 236044 op  ack_type 4 from 26
2014-06-12 14:46:10.711721 7f9d683e1700  5 -- op tracker -- , seq: 747025, time: 2014-06-12 14:46:10.711714, event: sub_op_commit_rec, op: osd_op(client.4309.0:141523 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~315392] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.711729 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2877 crt=0'0 lcod 173'2876 mlcod 173'2876 active+clean] repop_all_applied: repop tid 236044 all applied 
2014-06-12 14:46:10.711739 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2877 crt=0'0 lcod 173'2876 mlcod 173'2876 active+clean] eval_repop repgather(0x2680780 173'2878 rep_tid=236044 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141523 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~315392] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.711755 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2877 crt=0'0 lcod 173'2876 mlcod 173'2876 active+clean]  sending ack on repgather(0x2680780 173'2878 rep_tid=236044 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141523 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~315392] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0x5124a00
2014-06-12 14:46:10.711797 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2877 crt=0'0 lcod 173'2876 mlcod 173'2876 active+clean] repop_all_committed: repop tid 236044 all committed 
2014-06-12 14:46:10.711808 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2876 active+clean] eval_repop repgather(0x2680780 173'2878 rep_tid=236044 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141523 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~315392] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.711823 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2876 active+clean] log_op_stats osd_op(client.4309.0:141523 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~315392] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4 inb 315671 outb 0 rlat 32.057191 lat 32.057218
2014-06-12 14:46:10.711839 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2876 active+clean] publish_stats_to_osd 173:3264
2014-06-12 14:46:10.711853 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2876 active+clean]  sending commit on repgather(0x2680780 173'2878 rep_tid=236044 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141523 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~315392] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0x16ad0f00
2014-06-12 14:46:10.711878 7f9d683e1700  5 -- op tracker -- , seq: 747025, time: 2014-06-12 14:46:10.711875, event: commit_sent, op: osd_op(client.4309.0:141523 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~315392] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.711884 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2876 active+clean]  requeue_ops 
2014-06-12 14:46:10.711890 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2877 active+clean]  removing repgather(0x2680780 173'2878 rep_tid=236044 committed?=1 applied?=1 op=osd_op(client.4309.0:141523 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~315392] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.711899 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2877 active+clean]    q front is repgather(0x2680780 173'2878 rep_tid=236044 committed?=1 applied?=1 op=osd_op(client.4309.0:141523 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~315392] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.711908 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2877 active+clean] remove_repop repgather(0x2680780 173'2878 rep_tid=236044 committed?=1 applied?=1 op=osd_op(client.4309.0:141523 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~315392] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.711915 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2877 active+clean]  requeue_ops 
2014-06-12 14:46:10.711932 7f9d683e1700  5 -- op tracker -- , seq: 747025, time: 2014-06-12 14:46:10.711929, event: done, op: osd_op(client.4309.0:141523 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~315392] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.711952 7f9d683e1700 10 osd.2 173 dequeue_op 0x20e86800 finish
2014-06-12 14:46:10.711956 7f9d683e1700  5 -- op tracker -- , seq: 747622, time: 2014-06-12 14:46:10.711954, event: done, op: osd_sub_op_reply(client.4309.0:141523 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.711964 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.711966 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.713648 7f9d7715b700  5 -- op tracker -- , seq: 747619, time: 2014-06-12 14:46:10.713637, event: commit_sent, op: osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.713658 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3158 (79'157,173'3158] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3156 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:10.713684 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.713726 7f9d7715b700  5 -- op tracker -- , seq: 747619, time: 2014-06-12 14:46:10.713721, event: done, op: osd_sub_op(client.4300.0:140116 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3158 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.720962 7f9d58c5a700  5 -- op tracker -- , seq: 747623, time: 2014-06-12 14:46:10.719421, event: header_read, op: osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.720976 7f9d58c5a700  5 -- op tracker -- , seq: 747623, time: 2014-06-12 14:46:10.719423, event: throttled, op: osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.720985 7f9d58c5a700  5 -- op tracker -- , seq: 747623, time: 2014-06-12 14:46:10.720914, event: all_read, op: osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.720994 7f9d58c5a700  5 -- op tracker -- , seq: 747623, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.721000 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.721007 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:10.721016 7f9d58c5a700 15 osd.2 173 enqueue_op 0x20e88e00 prio 127 cost 516903 latency 0.001595 osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.721051 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.721061 7f9d65bdc700 10 osd.2 173 dequeue_op 0x20e88e00 prio 127 cost 516903 latency 0.001640 osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3158 (79'157,173'3158] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3157 active]
2014-06-12 14:46:10.721086 7f9d65bdc700  5 -- op tracker -- , seq: 747623, time: 2014-06-12 14:46:10.721082, event: reached_pg, op: osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.721096 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3158 (79'157,173'3158] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3157 active] handle_message: 0x20e88e00
2014-06-12 14:46:10.721109 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3158 (79'157,173'3158] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3157 active] sub_op_modify trans 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 v 173'3159 (transaction) 174
2014-06-12 14:46:10.721125 7f9d65bdc700  5 -- op tracker -- , seq: 747623, time: 2014-06-12 14:46:10.721122, event: started, op: osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.721144 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3158 (79'157,173'3158] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3157 active] append_log log((79'157,173'3158], crt=0'0) [173'3159 (173'3158) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140117 2014-06-12 08:45:13.096623]
2014-06-12 14:46:10.721169 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3159 (79'157,173'3159] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3157 active] add_log_entry 173'3159 (173'3158) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140117 2014-06-12 08:45:13.096623
2014-06-12 14:46:10.721190 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3159 (79'157,173'3159] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3157 active] append_log  adding 1 keys
2014-06-12 14:46:10.721203 7f9d65bdc700 10 trim log((79'157,173'3159], crt=0'0) to 79'158
2014-06-12 14:46:10.721235 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3159, trimmed: 79'158
2014-06-12 14:46:10.721261 7f9d65bdc700  5 -- op tracker -- , seq: 747623, time: 2014-06-12 14:46:10.721256, event: started, op: osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.721272 7f9d65bdc700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.721277 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.721279 7f9d65bdc700  5 queue_op 0x69b3f20 seq 2854 osr(3.36c 0x276d400) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:10.721287 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3159 (79'158,173'3159] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3157 active] do_sub_op osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.721302 7f9d65bdc700 10 osd.2 173 dequeue_op 0x20e88e00 finish
2014-06-12 14:46:10.721305 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.721308 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.721530 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (1 active)
2014-06-12 14:46:10.721539 7f9d8096e700  5 _do_op 0x69b3f20 seq 2854 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:10.721545 7f9d8096e700 10 _do_transaction on 0x29eb3cf0
2014-06-12 14:46:10.721552 7f9d8096e700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.721819 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.721845 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.721902 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.721931 7f9d8096e700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.721941 7f9d8096e700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:10.721947 7f9d8096e700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.721965 7f9d8096e700 15 _write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 24576~516096
2014-06-12 14:46:10.722114 7f9d8096e700 10 lookup_strip_header done  cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.722408 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:10.724486 7f9d8096e700 10 _generic_write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 24576~516096 = 0
2014-06-12 14:46:10.724561 7f9d8096e700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.724579 7f9d8096e700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.724586 7f9d8096e700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.724595 7f9d8096e700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.724846 7f9d8096e700 20 set_header setting 16547 cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:46:10.724940 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.724956 7f9d8096e700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:10.724968 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:10.730342 7f9d8096e700 10 _do_op 0x69b3f20 seq 2854 r = 0, finisher 0x6f59400 0
2014-06-12 14:46:10.730365 7f9d8096e700 10 _finish_op 0x69b3f20 seq 2854 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.730372 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (0 active)
2014-06-12 14:46:10.730377 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.730653 7f9d7795c700  5 -- op tracker -- , seq: 747623, time: 2014-06-12 14:46:10.730642, event: sub_op_applied, op: osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.730664 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3159 (79'158,173'3159] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3157 active] sub_op_modify_applied on 0x29eb3c00 op osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.730688 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.730707 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3159 (79'158,173'3159] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3157 active] op_applied on primary on version 173'3159
2014-06-12 14:46:10.730718 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3159 (79'158,173'3159] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3157 active] op_applied on replica on version 173'3159
2014-06-12 14:46:10.731228 7f9d7715b700  5 -- op tracker -- , seq: 747623, time: 2014-06-12 14:46:10.731220, event: commit_sent, op: osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.731261 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3159 (79'158,173'3159] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3157 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:10.731281 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.731314 7f9d7715b700  5 -- op tracker -- , seq: 747623, time: 2014-06-12 14:46:10.731309, event: done, op: osd_sub_op(client.4300.0:140117 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3159 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.742611 7f9d58c5a700  5 -- op tracker -- , seq: 747624, time: 2014-06-12 14:46:10.741686, event: header_read, op: osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.742638 7f9d58c5a700  5 -- op tracker -- , seq: 747624, time: 2014-06-12 14:46:10.741688, event: throttled, op: osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.742691 7f9d58c5a700  5 -- op tracker -- , seq: 747624, time: 2014-06-12 14:46:10.742537, event: all_read, op: osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.742705 7f9d58c5a700  5 -- op tracker -- , seq: 747624, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.742722 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.742737 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:10.742759 7f9d58c5a700 15 osd.2 173 enqueue_op 0x20e89000 prio 127 cost 516903 latency 0.001072 osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.742839 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.742859 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20e89000 prio 127 cost 516903 latency 0.001173 osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3159 (79'158,173'3159] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3158 active]
2014-06-12 14:46:10.742887 7f9d6abe6700  5 -- op tracker -- , seq: 747624, time: 2014-06-12 14:46:10.742883, event: reached_pg, op: osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.742897 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3159 (79'158,173'3159] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3158 active] handle_message: 0x20e89000
2014-06-12 14:46:10.742912 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3159 (79'158,173'3159] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3158 active] sub_op_modify trans 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 v 173'3160 (transaction) 174
2014-06-12 14:46:10.742932 7f9d6abe6700  5 -- op tracker -- , seq: 747624, time: 2014-06-12 14:46:10.742928, event: started, op: osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.742951 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3159 (79'158,173'3159] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3158 active] append_log log((79'158,173'3159], crt=0'0) [173'3160 (173'3159) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140118 2014-06-12 08:45:13.116286]
2014-06-12 14:46:10.742984 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3160 (79'158,173'3160] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3158 active] add_log_entry 173'3160 (173'3159) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140118 2014-06-12 08:45:13.116286
2014-06-12 14:46:10.743009 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3160 (79'158,173'3160] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3158 active] append_log  adding 1 keys
2014-06-12 14:46:10.743025 7f9d6abe6700 10 trim log((79'158,173'3160], crt=0'0) to 79'159
2014-06-12 14:46:10.743054 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3160, trimmed: 79'159
2014-06-12 14:46:10.743083 7f9d6abe6700  5 -- op tracker -- , seq: 747624, time: 2014-06-12 14:46:10.743078, event: started, op: osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.743097 7f9d6abe6700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.743102 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.743103 7f9d6abe6700  5 queue_op 0x6d5a000 seq 2855 osr(3.36c 0x276d400) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:10.743111 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3160 (79'159,173'3160] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3158 active] do_sub_op osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.743125 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20e89000 finish
2014-06-12 14:46:10.743128 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.743130 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.743176 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (1 active)
2014-06-12 14:46:10.743186 7f9d8016d700  5 _do_op 0x6d5a000 seq 2855 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:10.743192 7f9d8016d700 10 _do_transaction on 0x6174630
2014-06-12 14:46:10.743199 7f9d8016d700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.743349 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.743373 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.743429 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.743451 7f9d8016d700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.743459 7f9d8016d700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:10.743465 7f9d8016d700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.743478 7f9d8016d700 15 _write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 540672~516096
2014-06-12 14:46:10.743619 7f9d8016d700 10 lookup_strip_header done  cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.743915 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:10.746389 7f9d8016d700 10 _generic_write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 540672~516096 = 0
2014-06-12 14:46:10.746455 7f9d8016d700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.746473 7f9d8016d700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.746479 7f9d8016d700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.746486 7f9d8016d700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.746706 7f9d8016d700 20 set_header setting 16547 cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:46:10.746796 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.746810 7f9d8016d700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:10.746821 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:10.749953 7f9d8016d700 10 _do_op 0x6d5a000 seq 2855 r = 0, finisher 0x98efe80 0
2014-06-12 14:46:10.749979 7f9d8016d700 10 _finish_op 0x6d5a000 seq 2855 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.749986 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (0 active)
2014-06-12 14:46:10.749999 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.750360 7f9d7715b700  5 -- op tracker -- , seq: 747624, time: 2014-06-12 14:46:10.750350, event: commit_sent, op: osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.750371 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3160 (79'159,173'3160] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3158 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:10.750397 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.750476 7f9d7795c700  5 -- op tracker -- , seq: 747624, time: 2014-06-12 14:46:10.750467, event: sub_op_applied, op: osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.750486 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3160 (79'159,173'3160] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active] sub_op_modify_applied on 0x6174540 op osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.750502 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3160 (79'159,173'3160] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active] op_applied on primary on version 173'3160
2014-06-12 14:46:10.750513 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3160 (79'159,173'3160] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active] op_applied on replica on version 173'3160
2014-06-12 14:46:10.750540 7f9d7795c700  5 -- op tracker -- , seq: 747624, time: 2014-06-12 14:46:10.750536, event: done, op: osd_sub_op(client.4300.0:140118 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3160 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.750752 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:10.750798 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:10.758016 7f9d5a16f700  5 -- op tracker -- , seq: 747625, time: 2014-06-12 14:46:10.757633, event: header_read, op: osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.758029 7f9d5a16f700  5 -- op tracker -- , seq: 747625, time: 2014-06-12 14:46:10.757634, event: throttled, op: osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.758037 7f9d5a16f700  5 -- op tracker -- , seq: 747625, time: 2014-06-12 14:46:10.757979, event: all_read, op: osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.758045 7f9d5a16f700  5 -- op tracker -- , seq: 747625, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.758050 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.758057 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:10.758065 7f9d5a16f700 15 osd.2 173 enqueue_op 0x9fac100 prio 127 cost 295719 latency 0.000432 osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.758796 7f9d58c5a700  5 -- op tracker -- , seq: 747626, time: 2014-06-12 14:46:10.757071, event: header_read, op: osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.758813 7f9d58c5a700  5 -- op tracker -- , seq: 747626, time: 2014-06-12 14:46:10.757073, event: throttled, op: osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.758821 7f9d58c5a700  5 -- op tracker -- , seq: 747626, time: 2014-06-12 14:46:10.758768, event: all_read, op: osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.758830 7f9d58c5a700  5 -- op tracker -- , seq: 747626, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.758835 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.758841 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:10.758847 7f9d58c5a700 15 osd.2 173 enqueue_op 0x1702f000 prio 127 cost 516903 latency 0.001776 osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.758883 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.758890 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x9fac100 prio 127 cost 295719 latency 0.001256 osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4e0( v 173'2634 (0'0,173'2634] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2633 active]
2014-06-12 14:46:10.758911 7f9d6a3e5700  5 -- op tracker -- , seq: 747625, time: 2014-06-12 14:46:10.758907, event: reached_pg, op: osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.758920 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2634 (0'0,173'2634] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2633 active] handle_message: 0x9fac100
2014-06-12 14:46:10.758932 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2634 (0'0,173'2634] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2633 active] sub_op_modify trans 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 v 173'2635 (transaction) 174
2014-06-12 14:46:10.758948 7f9d6a3e5700  5 -- op tracker -- , seq: 747625, time: 2014-06-12 14:46:10.758944, event: started, op: osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.758963 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2634 (0'0,173'2634] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2633 active] append_log log((0'0,173'2634], crt=0'0) [173'2635 (173'2634) modify   6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 by client.4306.0:143778 2014-06-12 08:43:01.481773]
2014-06-12 14:46:10.758986 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2635 (0'0,173'2635] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2633 active] add_log_entry 173'2635 (173'2634) modify   6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 by client.4306.0:143778 2014-06-12 08:43:01.481773
2014-06-12 14:46:10.759007 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2635 (0'0,173'2635] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2633 active] append_log  adding 1 keys
2014-06-12 14:46:10.759038 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2635, trimmed: 
2014-06-12 14:46:10.759060 7f9d6a3e5700  5 -- op tracker -- , seq: 747625, time: 2014-06-12 14:46:10.759055, event: started, op: osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.759074 7f9d6a3e5700  5 queue_transactions existing osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:10.759078 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.759079 7f9d6a3e5700  5 queue_op 0x5a4c000 seq 2435 osr(3.4e0 0x276dbd0) 297205 bytes   (queue has 1 ops and 297205 bytes)
2014-06-12 14:46:10.759097 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:10.759087 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2635 (0'0,173'2635] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2633 active] do_sub_op osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.759102 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x9fac100 finish
2014-06-12 14:46:10.759106 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:10.759108 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.759105 7f9d66bde700 10 osd.2 173 dequeue_op 0x1702f000 prio 127 cost 516903 latency 0.002033 osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3160 (79'159,173'3160] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active]
2014-06-12 14:46:10.759128 7f9d66bde700  5 -- op tracker -- , seq: 747626, time: 2014-06-12 14:46:10.759123, event: reached_pg, op: osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.759137 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3160 (79'159,173'3160] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active] handle_message: 0x1702f000
2014-06-12 14:46:10.759148 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3160 (79'159,173'3160] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active] sub_op_modify trans 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 v 173'3161 (transaction) 174
2014-06-12 14:46:10.759164 7f9d66bde700  5 -- op tracker -- , seq: 747626, time: 2014-06-12 14:46:10.759160, event: started, op: osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.759176 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3160 (79'159,173'3160] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active] append_log log((79'159,173'3160], crt=0'0) [173'3161 (173'3160) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140119 2014-06-12 08:45:13.133736]
2014-06-12 14:46:10.759199 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3161 (79'159,173'3161] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active] add_log_entry 173'3161 (173'3160) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140119 2014-06-12 08:45:13.133736
2014-06-12 14:46:10.759221 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3161 (79'159,173'3161] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active] append_log  adding 1 keys
2014-06-12 14:46:10.759238 7f9d66bde700 10 trim log((79'159,173'3161], crt=0'0) to 79'160
2014-06-12 14:46:10.759264 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3161, trimmed: 79'160
2014-06-12 14:46:10.759292 7f9d66bde700  5 -- op tracker -- , seq: 747626, time: 2014-06-12 14:46:10.759286, event: started, op: osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.759302 7f9d66bde700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.759305 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.759306 7f9d66bde700  5 queue_op 0x7730f60 seq 2856 osr(3.36c 0x276d400) 518424 bytes   (queue has 2 ops and 815629 bytes)
2014-06-12 14:46:10.759318 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5180900 (1 active)
2014-06-12 14:46:10.759325 7f9d7915f700  5 _do_op 0x5a4c000 seq 2435 osr(3.4e0 0x276dbd0)/0x276dbd0 start
2014-06-12 14:46:10.759314 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3161 (79'160,173'3161] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active] do_sub_op osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.759327 7f9d66bde700 10 osd.2 173 dequeue_op 0x1702f000 finish
2014-06-12 14:46:10.759329 7f9d7915f700 10 _do_transaction on 0x2705e9b0
2014-06-12 14:46:10.759331 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.759333 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.759334 7f9d7915f700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:10.759342 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (2 active)
2014-06-12 14:46:10.759346 7f9d7d968700  5 _do_op 0x7730f60 seq 2856 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:10.759349 7f9d7d968700 10 _do_transaction on 0x5b7ab70
2014-06-12 14:46:10.759354 7f9d7d968700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.759484 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:10.759504 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.759558 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.759575 7f9d7915f700 15 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:10.759581 7f9d7915f700 10 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1 r = 0
2014-06-12 14:46:10.759587 7f9d7915f700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:10.759600 7f9d7915f700 15 _write 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 3899392~294912
2014-06-12 14:46:10.759753 7f9d7915f700 10 lookup_strip_header done  cid 3.4e0_head oid 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:10.759879 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.759908 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.759960 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:10.761080 7f9d7915f700 10 _generic_write 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 3899392~294912 = 0
2014-06-12 14:46:10.761133 7f9d7915f700 15 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:10.761151 7f9d7915f700 10 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:10.761156 7f9d7915f700 15 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3
2014-06-12 14:46:10.761165 7f9d7915f700 10 _setattrs 3.4e0_head/6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 = 0
2014-06-12 14:46:10.761338 7f9d7915f700 20 set_header setting 16555 cid 3.4e0_head oid 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 parent seq 0
2014-06-12 14:46:10.761400 7f9d7915f700 20 set_header setting 2236 cid meta oid 1fe07435/pglog_3.4e0/0//-1 parent seq 0
2014-06-12 14:46:10.761416 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.761425 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:10.763146 7f9d7915f700 10 _do_op 0x5a4c000 seq 2435 r = 0, finisher 0x682b0c0 0
2014-06-12 14:46:10.763168 7f9d7915f700 10 _finish_op 0x5a4c000 seq 2435 osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:10.763178 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5180900 (1 active)
2014-06-12 14:46:10.763183 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.765657 7f9d7715b700  5 -- op tracker -- , seq: 747625, time: 2014-06-12 14:46:10.765638, event: commit_sent, op: osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.765673 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2635 (0'0,173'2635] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2633 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:10.765705 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:10.765766 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.765809 7f9d7d968700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.765819 7f9d7d968700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:10.765827 7f9d7d968700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.765935 7f9d7d968700 15 _write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 1056768~516096
2014-06-12 14:46:10.766096 7f9d7d968700 10 lookup_strip_header done  cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.766232 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:10.767189 7f9d7795c700  5 -- op tracker -- , seq: 747625, time: 2014-06-12 14:46:10.767178, event: sub_op_applied, op: osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.767202 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2635 (0'0,173'2635] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2634 active] sub_op_modify_applied on 0x2705e8c0 op osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.767235 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2635 (0'0,173'2635] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2634 active] op_applied on primary on version 173'2635
2014-06-12 14:46:10.767246 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2635 (0'0,173'2635] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2634 active] op_applied on replica on version 173'2635
2014-06-12 14:46:10.767281 7f9d7795c700  5 -- op tracker -- , seq: 747625, time: 2014-06-12 14:46:10.767276, event: done, op: osd_sub_op(client.4306.0:143778 3.4e0 6ae164e0/rbd_data.10a96b8b4567.0000000000000f62/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.768152 7f9d7d968700 10 _generic_write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 1056768~516096 = 0
2014-06-12 14:46:10.768206 7f9d7d968700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.768219 7f9d7d968700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.768225 7f9d7d968700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.768234 7f9d7d968700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.768446 7f9d7d968700 20 set_header setting 16547 cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:46:10.768517 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.768532 7f9d7d968700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:10.768542 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:10.779717 7f9d7d968700 10 _do_op 0x7730f60 seq 2856 r = 0, finisher 0x63d1140 0
2014-06-12 14:46:10.779739 7f9d7d968700 10 _finish_op 0x7730f60 seq 2856 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.779750 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (0 active)
2014-06-12 14:46:10.779755 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.781671 7f9d7795c700  5 -- op tracker -- , seq: 747626, time: 2014-06-12 14:46:10.781650, event: sub_op_applied, op: osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.781684 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3161 (79'160,173'3161] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active] sub_op_modify_applied on 0x5b7aa80 op osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.781727 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.781752 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3161 (79'160,173'3161] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active] op_applied on primary on version 173'3161
2014-06-12 14:46:10.781763 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3161 (79'160,173'3161] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active] op_applied on replica on version 173'3161
2014-06-12 14:46:10.782050 7f9d7715b700  5 -- op tracker -- , seq: 747626, time: 2014-06-12 14:46:10.782043, event: commit_sent, op: osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.782062 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3161 (79'160,173'3161] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3159 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:10.782082 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.782119 7f9d7715b700  5 -- op tracker -- , seq: 747626, time: 2014-06-12 14:46:10.782114, event: done, op: osd_sub_op(client.4300.0:140119 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3161 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.794139 7f9d58c5a700  5 -- op tracker -- , seq: 747627, time: 2014-06-12 14:46:10.788520, event: header_read, op: osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.794157 7f9d58c5a700  5 -- op tracker -- , seq: 747627, time: 2014-06-12 14:46:10.788522, event: throttled, op: osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.794166 7f9d58c5a700  5 -- op tracker -- , seq: 747627, time: 2014-06-12 14:46:10.794081, event: all_read, op: osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.794175 7f9d58c5a700  5 -- op tracker -- , seq: 747627, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.794182 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.794191 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:10.794203 7f9d58c5a700 15 osd.2 173 enqueue_op 0x2afdb00 prio 127 cost 516903 latency 0.005683 osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.794240 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.794250 7f9d653db700 10 osd.2 173 dequeue_op 0x2afdb00 prio 127 cost 516903 latency 0.005730 osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3161 (79'160,173'3161] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3160 active]
2014-06-12 14:46:10.794276 7f9d653db700  5 -- op tracker -- , seq: 747627, time: 2014-06-12 14:46:10.794271, event: reached_pg, op: osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.794285 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3161 (79'160,173'3161] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3160 active] handle_message: 0x2afdb00
2014-06-12 14:46:10.794297 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3161 (79'160,173'3161] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3160 active] sub_op_modify trans 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 v 173'3162 (transaction) 174
2014-06-12 14:46:10.794313 7f9d653db700  5 -- op tracker -- , seq: 747627, time: 2014-06-12 14:46:10.794309, event: started, op: osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.794332 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3161 (79'160,173'3161] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3160 active] append_log log((79'160,173'3161], crt=0'0) [173'3162 (173'3161) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140120 2014-06-12 08:45:13.165707]
2014-06-12 14:46:10.794364 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3162 (79'160,173'3162] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3160 active] add_log_entry 173'3162 (173'3161) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140120 2014-06-12 08:45:13.165707
2014-06-12 14:46:10.794387 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3162 (79'160,173'3162] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3160 active] append_log  adding 1 keys
2014-06-12 14:46:10.794401 7f9d653db700 10 trim log((79'160,173'3162], crt=0'0) to 79'161
2014-06-12 14:46:10.794431 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3162, trimmed: 79'161
2014-06-12 14:46:10.794459 7f9d653db700  5 -- op tracker -- , seq: 747627, time: 2014-06-12 14:46:10.794454, event: started, op: osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.794474 7f9d653db700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.794481 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.794484 7f9d653db700  5 queue_op 0x6e68120 seq 2857 osr(3.36c 0x276d400) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:10.794496 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3162 (79'161,173'3162] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3160 active] do_sub_op osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.794511 7f9d653db700 10 osd.2 173 dequeue_op 0x2afdb00 finish
2014-06-12 14:46:10.794514 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.794516 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.794526 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (1 active)
2014-06-12 14:46:10.794532 7f9d83173700  5 _do_op 0x6e68120 seq 2857 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:10.794536 7f9d83173700 10 _do_transaction on 0x61740f0
2014-06-12 14:46:10.794543 7f9d83173700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.794780 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.794803 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.794854 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.794880 7f9d83173700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.794887 7f9d83173700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:10.794893 7f9d83173700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.794912 7f9d83173700 15 _write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 1572864~516096
2014-06-12 14:46:10.795065 7f9d83173700 10 lookup_strip_header done  cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.795310 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:10.797348 7f9d83173700 10 _generic_write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 1572864~516096 = 0
2014-06-12 14:46:10.797413 7f9d83173700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.797434 7f9d83173700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.797446 7f9d83173700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.797454 7f9d83173700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.797674 7f9d83173700 20 set_header setting 16547 cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:46:10.797764 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.797782 7f9d83173700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:10.797793 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:10.800122 7f9d83173700 10 _do_op 0x6e68120 seq 2857 r = 0, finisher 0x7c91f60 0
2014-06-12 14:46:10.800142 7f9d83173700 10 _finish_op 0x6e68120 seq 2857 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.800150 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (0 active)
2014-06-12 14:46:10.800160 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.800168 7f9d7795c700  5 -- op tracker -- , seq: 747627, time: 2014-06-12 14:46:10.800158, event: sub_op_applied, op: osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.800176 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3162 (79'161,173'3162] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3160 active] sub_op_modify_applied on 0x6174000 op osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.800199 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.800216 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3162 (79'161,173'3162] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3160 active] op_applied on primary on version 173'3162
2014-06-12 14:46:10.800229 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3162 (79'161,173'3162] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3160 active] op_applied on replica on version 173'3162
2014-06-12 14:46:10.800671 7f9d7715b700  5 -- op tracker -- , seq: 747627, time: 2014-06-12 14:46:10.800664, event: commit_sent, op: osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.800684 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3162 (79'161,173'3162] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3160 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:10.800703 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.800737 7f9d7715b700  5 -- op tracker -- , seq: 747627, time: 2014-06-12 14:46:10.800732, event: done, op: osd_sub_op(client.4300.0:140120 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3162 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.807059 7f9d58c5a700  5 -- op tracker -- , seq: 747628, time: 2014-06-12 14:46:10.805785, event: header_read, op: osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.807077 7f9d58c5a700  5 -- op tracker -- , seq: 747628, time: 2014-06-12 14:46:10.805787, event: throttled, op: osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.807086 7f9d58c5a700  5 -- op tracker -- , seq: 747628, time: 2014-06-12 14:46:10.807005, event: all_read, op: osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.807094 7f9d58c5a700  5 -- op tracker -- , seq: 747628, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.807101 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.807109 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:10.807118 7f9d58c5a700 15 osd.2 173 enqueue_op 0x20355100 prio 127 cost 516903 latency 0.001333 osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.807156 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.807165 7f9d63bd8700 10 osd.2 173 dequeue_op 0x20355100 prio 127 cost 516903 latency 0.001379 osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3162 (79'161,173'3162] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3161 active]
2014-06-12 14:46:10.807191 7f9d63bd8700  5 -- op tracker -- , seq: 747628, time: 2014-06-12 14:46:10.807186, event: reached_pg, op: osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.807200 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3162 (79'161,173'3162] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3161 active] handle_message: 0x20355100
2014-06-12 14:46:10.807213 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3162 (79'161,173'3162] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3161 active] sub_op_modify trans 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 v 173'3163 (transaction) 174
2014-06-12 14:46:10.807232 7f9d63bd8700  5 -- op tracker -- , seq: 747628, time: 2014-06-12 14:46:10.807227, event: started, op: osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.807252 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3162 (79'161,173'3162] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3161 active] append_log log((79'161,173'3162], crt=0'0) [173'3163 (173'3162) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140121 2014-06-12 08:45:13.183757]
2014-06-12 14:46:10.807285 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3163 (79'161,173'3163] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3161 active] add_log_entry 173'3163 (173'3162) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140121 2014-06-12 08:45:13.183757
2014-06-12 14:46:10.807312 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3163 (79'161,173'3163] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3161 active] append_log  adding 1 keys
2014-06-12 14:46:10.807329 7f9d63bd8700 10 trim log((79'161,173'3163], crt=0'0) to 79'162
2014-06-12 14:46:10.807359 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3163, trimmed: 79'162
2014-06-12 14:46:10.807391 7f9d63bd8700  5 -- op tracker -- , seq: 747628, time: 2014-06-12 14:46:10.807386, event: started, op: osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.807405 7f9d63bd8700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.807410 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.807412 7f9d63bd8700  5 queue_op 0xcc967e0 seq 2858 osr(3.36c 0x276d400) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:10.807420 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3163 (79'162,173'3163] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3161 active] do_sub_op osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.807436 7f9d63bd8700 10 osd.2 173 dequeue_op 0x20355100 finish
2014-06-12 14:46:10.807440 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.807442 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.807452 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (1 active)
2014-06-12 14:46:10.807457 7f9d7c165700  5 _do_op 0xcc967e0 seq 2858 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:10.807462 7f9d7c165700 10 _do_transaction on 0x5bcf270
2014-06-12 14:46:10.807468 7f9d7c165700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.807610 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.807633 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.807691 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.807712 7f9d7c165700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.807722 7f9d7c165700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:10.807729 7f9d7c165700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.807744 7f9d7c165700 15 _write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 2088960~516096
2014-06-12 14:46:10.807904 7f9d7c165700 10 lookup_strip_header done  cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.808200 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:10.810167 7f9d7c165700 10 _generic_write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 2088960~516096 = 0
2014-06-12 14:46:10.810228 7f9d7c165700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.810245 7f9d7c165700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.810253 7f9d7c165700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.810262 7f9d7c165700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.810470 7f9d7c165700 20 set_header setting 16547 cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:46:10.810550 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.810568 7f9d7c165700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:10.810581 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:10.817038 7f9d7c165700 10 _do_op 0xcc967e0 seq 2858 r = 0, finisher 0x65edc80 0
2014-06-12 14:46:10.817068 7f9d7c165700 10 _finish_op 0xcc967e0 seq 2858 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.817087 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (0 active)
2014-06-12 14:46:10.817109 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.817091 7f9d7715b700  5 -- op tracker -- , seq: 747628, time: 2014-06-12 14:46:10.817079, event: commit_sent, op: osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.817636 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3163 (79'162,173'3163] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3161 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:10.817672 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.817714 7f9d7795c700  5 -- op tracker -- , seq: 747628, time: 2014-06-12 14:46:10.817707, event: sub_op_applied, op: osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.817725 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3163 (79'162,173'3163] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3162 active] sub_op_modify_applied on 0x5bcf180 op osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.817744 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3163 (79'162,173'3163] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3162 active] op_applied on primary on version 173'3163
2014-06-12 14:46:10.817754 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3163 (79'162,173'3163] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3162 active] op_applied on replica on version 173'3163
2014-06-12 14:46:10.817781 7f9d7795c700  5 -- op tracker -- , seq: 747628, time: 2014-06-12 14:46:10.817776, event: done, op: osd_sub_op(client.4300.0:140121 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3163 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.819780 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:10.819783 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:10.819785 7f9d74bfa700 20 osd.2 173 _dispatch 0x8d86000 ping v1
2014-06-12 14:46:10.819787 7f9d74bfa700 10 osd.2 173 ping from client.4303
2014-06-12 14:46:10.819789 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:10.819791 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:10.823293 7f9d58c5a700  5 -- op tracker -- , seq: 747629, time: 2014-06-12 14:46:10.822297, event: header_read, op: osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.823308 7f9d58c5a700  5 -- op tracker -- , seq: 747629, time: 2014-06-12 14:46:10.822299, event: throttled, op: osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.823317 7f9d58c5a700  5 -- op tracker -- , seq: 747629, time: 2014-06-12 14:46:10.823251, event: all_read, op: osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.823325 7f9d58c5a700  5 -- op tracker -- , seq: 747629, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.823330 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.823337 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:10.823347 7f9d58c5a700 15 osd.2 173 enqueue_op 0x20354a00 prio 127 cost 516903 latency 0.001049 osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.823382 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.823396 7f9d69be4700 10 osd.2 173 dequeue_op 0x20354a00 prio 127 cost 516903 latency 0.001098 osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3163 (79'162,173'3163] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3162 active]
2014-06-12 14:46:10.823425 7f9d69be4700  5 -- op tracker -- , seq: 747629, time: 2014-06-12 14:46:10.823420, event: reached_pg, op: osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.823436 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3163 (79'162,173'3163] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3162 active] handle_message: 0x20354a00
2014-06-12 14:46:10.823451 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3163 (79'162,173'3163] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3162 active] sub_op_modify trans 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 v 173'3164 (transaction) 174
2014-06-12 14:46:10.823472 7f9d69be4700  5 -- op tracker -- , seq: 747629, time: 2014-06-12 14:46:10.823467, event: started, op: osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.823490 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3163 (79'162,173'3163] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3162 active] append_log log((79'162,173'3163], crt=0'0) [173'3164 (173'3163) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140122 2014-06-12 08:45:13.200423]
2014-06-12 14:46:10.823519 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3164 (79'162,173'3164] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3162 active] add_log_entry 173'3164 (173'3163) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140122 2014-06-12 08:45:13.200423
2014-06-12 14:46:10.823545 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3164 (79'162,173'3164] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3162 active] append_log  adding 1 keys
2014-06-12 14:46:10.823562 7f9d69be4700 10 trim log((79'162,173'3164], crt=0'0) to 79'163
2014-06-12 14:46:10.823593 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3164, trimmed: 79'163
2014-06-12 14:46:10.823624 7f9d69be4700  5 -- op tracker -- , seq: 747629, time: 2014-06-12 14:46:10.823618, event: started, op: osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.823638 7f9d69be4700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.823643 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.823645 7f9d69be4700  5 queue_op 0x287fe00 seq 2859 osr(3.36c 0x276d400) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:10.823652 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3164 (79'163,173'3164] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3162 active] do_sub_op osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.823668 7f9d69be4700 10 osd.2 173 dequeue_op 0x20354a00 finish
2014-06-12 14:46:10.823671 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.823674 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.823683 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (1 active)
2014-06-12 14:46:10.823688 7f9d82972700  5 _do_op 0x287fe00 seq 2859 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:10.823693 7f9d82972700 10 _do_transaction on 0x6175eb0
2014-06-12 14:46:10.823699 7f9d82972700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.823854 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.823877 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.823933 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.824160 7f9d82972700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.824171 7f9d82972700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:10.824179 7f9d82972700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.824202 7f9d82972700 15 _write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 2605056~516096
2014-06-12 14:46:10.824358 7f9d82972700 10 lookup_strip_header done  cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.824557 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:10.826679 7f9d82972700 10 _generic_write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 2605056~516096 = 0
2014-06-12 14:46:10.826728 7f9d82972700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.826743 7f9d82972700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.826748 7f9d82972700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.826755 7f9d82972700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.826959 7f9d82972700 20 set_header setting 16547 cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:46:10.827032 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.827049 7f9d82972700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:10.827059 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:10.830118 7f9d82972700 10 _do_op 0x287fe00 seq 2859 r = 0, finisher 0x65edc80 0
2014-06-12 14:46:10.830158 7f9d7715b700  5 -- op tracker -- , seq: 747629, time: 2014-06-12 14:46:10.830148, event: commit_sent, op: osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.830169 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3164 (79'163,173'3164] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3162 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:10.830197 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.830253 7f9d82972700 10 _finish_op 0x287fe00 seq 2859 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.830263 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (0 active)
2014-06-12 14:46:10.830276 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.830467 7f9d7795c700  5 -- op tracker -- , seq: 747629, time: 2014-06-12 14:46:10.830460, event: sub_op_applied, op: osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.830476 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3164 (79'163,173'3164] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active] sub_op_modify_applied on 0x6175dc0 op osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.830493 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3164 (79'163,173'3164] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active] op_applied on primary on version 173'3164
2014-06-12 14:46:10.830504 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3164 (79'163,173'3164] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active] op_applied on replica on version 173'3164
2014-06-12 14:46:10.830526 7f9d7795c700  5 -- op tracker -- , seq: 747629, time: 2014-06-12 14:46:10.830521, event: done, op: osd_sub_op(client.4300.0:140122 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3164 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.836713 7f9d4ed71700  5 -- op tracker -- , seq: 747630, time: 2014-06-12 14:46:10.835803, event: header_read, op: osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.836730 7f9d4ed71700  5 -- op tracker -- , seq: 747630, time: 2014-06-12 14:46:10.835806, event: throttled, op: osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.836740 7f9d4ed71700  5 -- op tracker -- , seq: 747630, time: 2014-06-12 14:46:10.836668, event: all_read, op: osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.836750 7f9d4ed71700  5 -- op tracker -- , seq: 747630, time: 0.000000, event: dispatched, op: osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.836759 7f9d4ed71700 20 osd.2 173 should_share_map client.4291 172.16.96.220:0/1241802 173
2014-06-12 14:46:10.836788 7f9d4ed71700 15 osd.2 173 enqueue_op 0x2911700 prio 63 cost 258048 latency 0.000985 osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:10.836821 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.836830 7f9d68be2700 10 osd.2 173 dequeue_op 0x2911700 prio 63 cost 258048 latency 0.001027 osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean]
2014-06-12 14:46:10.836850 7f9d68be2700  5 -- op tracker -- , seq: 747630, time: 2014-06-12 14:46:10.836846, event: reached_pg, op: osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.836858 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:10.836868 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] handle_message: 0x2911700
2014-06-12 14:46:10.836874 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] do_op osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:10.836891 7f9d68be2700 15 getattr 3.214_head/53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 '_'
2014-06-12 14:46:10.837116 7f9d68be2700 10 getattr 3.214_head/53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 '_' = 0
2014-06-12 14:46:10.837150 7f9d68be2700 15 getattr 3.214_head/53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 'snapset'
2014-06-12 14:46:10.837243 7f9d68be2700 10 getattr 3.214_head/53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 'snapset' = 0
2014-06-12 14:46:10.837254 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] populate_obc_watchers 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3
2014-06-12 14:46:10.837267 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3
2014-06-12 14:46:10.837277 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] get_object_context: creating obc from disk: 0x15518840
2014-06-12 14:46:10.837285 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] get_object_context: 0x15518840 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 rwstate(none n=0 w=0) oi: 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3(173'753 client.4291.0:36731 wrlock_by=unknown.0.0:0 dirty s 4194304 uv753) ssc: 0x5bb9a40 snapset: 0=[]:[]+head
2014-06-12 14:46:10.837300 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] find_object_context 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 @head oi=53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3(173'753 client.4291.0:36731 wrlock_by=unknown.0.0:0 dirty s 4194304 uv753)
2014-06-12 14:46:10.837323 7f9d68be2700  5 -- op tracker -- , seq: 747630, time: 2014-06-12 14:46:10.837317, event: started, op: osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.837332 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] execute_ctx 0x5e4ca00
2014-06-12 14:46:10.837349 7f9d68be2700  5 -- op tracker -- , seq: 747630, time: 2014-06-12 14:46:10.837344, event: started, op: osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.837355 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] do_op 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] ov 173'753 av 173'2935 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:10.837370 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] do_osd_op 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048]
2014-06-12 14:46:10.837381 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:10.837393 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] do_osd_op  write 0~258048
2014-06-12 14:46:10.837408 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] do_osd_op_effects on session 0xdd06400
2014-06-12 14:46:10.837420 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] make_writeable 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 snapset=0x5bb9a78  snapc=0=[]
2014-06-12 14:46:10.837445 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] make_writeable 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:10.837463 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] finish_ctx 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 0x5e4ca00 op modify  
2014-06-12 14:46:10.837484 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean]  set mtime to 2014-06-12 08:45:13.216308
2014-06-12 14:46:10.837513 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean]  final snapset 0=[]:[]+head in 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3
2014-06-12 14:46:10.837546 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean]  zeroing write result code 0
2014-06-12 14:46:10.837572 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] new_repop rep_tid 236217 on osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:10.837598 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] issue_repop rep_tid 236217 o 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3
2014-06-12 14:46:10.837640 7f9d68be2700  5 -- op tracker -- , seq: 747630, time: 2014-06-12 14:46:10.837635, event: waiting for subops from 28, op: osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.837673 7f9d68be2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:10.837704 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2934 (0'0,173'2934] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] append_log log((0'0,173'2934], crt=0'0) [173'2935 (173'753) modify   53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 by client.4291.0:140689 2014-06-12 08:45:13.216308]
2014-06-12 14:46:10.837729 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2934 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] add_log_entry 173'2935 (173'753) modify   53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 by client.4291.0:140689 2014-06-12 08:45:13.216308
2014-06-12 14:46:10.837750 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2934 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] append_log  adding 1 keys
2014-06-12 14:46:10.837783 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2935, trimmed: 
2014-06-12 14:46:10.837811 7f9d68be2700  5 queue_transactions existing osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:10.837817 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.837818 7f9d68be2700  5 queue_op 0x50299e0 seq 2707 osr(3.214 0x276d040) 260341 bytes   (queue has 1 ops and 260341 bytes)
2014-06-12 14:46:10.837827 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5e60 (1 active)
2014-06-12 14:46:10.837835 7f9d7c966700  5 _do_op 0x50299e0 seq 2707 osr(3.214 0x276d040)/0x276d040 start
2014-06-12 14:46:10.837839 7f9d7c966700 10 _do_transaction on 0x2a37b2c0
2014-06-12 14:46:10.837824 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2934 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] eval_repop repgather(0x514d380 173'2935 rep_tid=236217 committed?=0 applied?=0 lock=0 op=osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.837843 7f9d68be2700 10 osd.2 173 dequeue_op 0x2911700 finish
2014-06-12 14:46:10.837845 7f9d7c966700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:10.837846 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.837849 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.837981 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:10.838004 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.838064 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.838090 7f9d7c966700 15 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:10.838098 7f9d7c966700 10 _omap_rmkeys meta/1fe25aca/pglog_3.214/0//-1 r = 0
2014-06-12 14:46:10.838105 7f9d7c966700 15 _omap_setkeys meta/1fe25aca/pglog_3.214/0//-1
2014-06-12 14:46:10.838213 7f9d7c966700 15 _write 3.214_head/53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 0~258048
2014-06-12 14:46:10.838362 7f9d7c966700 10 lookup_strip_header done  cid 3.214_head oid 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3
2014-06-12 14:46:10.838518 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:10.839443 7f9d7c966700 10 _generic_write 3.214_head/53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 0~258048 = 0
2014-06-12 14:46:10.839480 7f9d7c966700 15 _setattrs 3.214_head/53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3
2014-06-12 14:46:10.839493 7f9d7c966700 10 _setattrs 3.214_head/53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 = 0
2014-06-12 14:46:10.839500 7f9d7c966700 15 _setattrs 3.214_head/53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3
2014-06-12 14:46:10.839509 7f9d7c966700 10 _setattrs 3.214_head/53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 = 0
2014-06-12 14:46:10.839687 7f9d7c966700 20 set_header setting 16647 cid 3.214_head oid 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 parent seq 0
2014-06-12 14:46:10.839754 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.839769 7f9d7c966700 20 set_header setting 2020 cid meta oid 1fe25aca/pglog_3.214/0//-1 parent seq 0
2014-06-12 14:46:10.839781 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:10.840988 7f9d7c966700 10 _do_op 0x50299e0 seq 2707 r = 0, finisher 0x654d2e0 0x38997e40
2014-06-12 14:46:10.841007 7f9d7c966700 10 _finish_op 0x50299e0 seq 2707 osr(3.214 0x276d040)/0x276d040
2014-06-12 14:46:10.841021 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5e60 (0 active)
2014-06-12 14:46:10.841025 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.841039 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2934 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] op_commit: 236217
2014-06-12 14:46:10.841082 7f9d7715b700  5 -- op tracker -- , seq: 747630, time: 2014-06-12 14:46:10.841067, event: op_commit, op: osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.842106 7f9d5b6bc700  5 -- op tracker -- , seq: 747631, time: 2014-06-12 14:46:10.842057, event: header_read, op: osd_sub_op_reply(client.4291.0:140689 3.214 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.842119 7f9d5b6bc700  5 -- op tracker -- , seq: 747631, time: 2014-06-12 14:46:10.842059, event: throttled, op: osd_sub_op_reply(client.4291.0:140689 3.214 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.842141 7f9d5b6bc700  5 -- op tracker -- , seq: 747631, time: 2014-06-12 14:46:10.842072, event: all_read, op: osd_sub_op_reply(client.4291.0:140689 3.214 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.842150 7f9d5b6bc700  5 -- op tracker -- , seq: 747631, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4291.0:140689 3.214 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.841861 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2934 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] op_applied: 236217
2014-06-12 14:46:10.842158 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4291.0:140689 3.214 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:10.842166 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:10.842178 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x55c3700 prio 196 cost 0 latency 0.000120 osd_sub_op_reply(client.4291.0:140689 3.214 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:10.842183 7f9d7795c700  5 -- op tracker -- , seq: 747630, time: 2014-06-12 14:46:10.842160, event: op_applied, op: osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.842193 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2934 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] op_applied on primary on version 173'2935
2014-06-12 14:46:10.842209 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.842218 7f9d633d7700 10 osd.2 173 dequeue_op 0x55c3700 prio 196 cost 0 latency 0.000161 osd_sub_op_reply(client.4291.0:140689 3.214 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 [] ondisk, result = 0) v2 pg pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2934 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean]
2014-06-12 14:46:10.842238 7f9d633d7700  5 -- op tracker -- , seq: 747631, time: 2014-06-12 14:46:10.842234, event: reached_pg, op: osd_sub_op_reply(client.4291.0:140689 3.214 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.842247 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2934 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] handle_message: 0x55c3700
2014-06-12 14:46:10.842261 7f9d633d7700  5 -- op tracker -- , seq: 747631, time: 2014-06-12 14:46:10.842258, event: started, op: osd_sub_op_reply(client.4291.0:140689 3.214 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.842267 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2934 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] sub_op_modify_reply: tid 236217 op  ack_type 4 from 28
2014-06-12 14:46:10.842285 7f9d633d7700  5 -- op tracker -- , seq: 747630, time: 2014-06-12 14:46:10.842279, event: sub_op_commit_rec, op: osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.842292 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2934 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] repop_all_applied: repop tid 236217 all applied 
2014-06-12 14:46:10.842301 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2934 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] eval_repop repgather(0x514d380 173'2935 rep_tid=236217 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.842316 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2934 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean]  sending ack on repgather(0x514d380 173'2935 rep_tid=236217 committed?=0 applied?=1 lock=0 op=osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173) v4) 0x56ca000
2014-06-12 14:46:10.842347 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2934 crt=0'0 lcod 173'2933 mlcod 173'2933 active+clean] repop_all_committed: repop tid 236217 all committed 
2014-06-12 14:46:10.842357 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2933 active+clean] eval_repop repgather(0x514d380 173'2935 rep_tid=236217 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.842371 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2933 active+clean] log_op_stats osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173) v4 inb 258327 outb 0 rlat 0.006542 lat 0.006567
2014-06-12 14:46:10.842386 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2933 active+clean] publish_stats_to_osd 173:3706
2014-06-12 14:46:10.842398 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2933 active+clean]  sending commit on repgather(0x514d380 173'2935 rep_tid=236217 committed?=1 applied?=1 lock=0 op=osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173) v4) 0x19e0b680
2014-06-12 14:46:10.842427 7f9d633d7700  5 -- op tracker -- , seq: 747630, time: 2014-06-12 14:46:10.842421, event: commit_sent, op: osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.842458 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2933 active+clean]  requeue_ops 
2014-06-12 14:46:10.842469 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2934 active+clean]  removing repgather(0x514d380 173'2935 rep_tid=236217 committed?=1 applied?=1 op=osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.842482 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2934 active+clean]    q front is repgather(0x514d380 173'2935 rep_tid=236217 committed?=1 applied?=1 op=osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.842494 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2934 active+clean] remove_repop repgather(0x514d380 173'2935 rep_tid=236217 committed?=1 applied?=1 op=osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.842506 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2934 active+clean]  requeue_ops 
2014-06-12 14:46:10.842531 7f9d633d7700  5 -- op tracker -- , seq: 747630, time: 2014-06-12 14:46:10.842526, event: done, op: osd_op(client.4291.0:140689 rbd_data.109a6b8b4567.0000000000000f5d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.53a88214 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.842574 7f9d58c5a700  5 -- op tracker -- , seq: 747632, time: 2014-06-12 14:46:10.835263, event: header_read, op: osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.842586 7f9d58c5a700  5 -- op tracker -- , seq: 747632, time: 2014-06-12 14:46:10.835265, event: throttled, op: osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.842594 7f9d58c5a700  5 -- op tracker -- , seq: 747632, time: 2014-06-12 14:46:10.842540, event: all_read, op: osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.842605 7f9d58c5a700  5 -- op tracker -- , seq: 747632, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.842609 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.842615 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:10.842621 7f9d58c5a700 15 osd.2 173 enqueue_op 0x20354b00 prio 127 cost 516903 latency 0.007358 osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.842629 7f9d633d7700 10 osd.2 173 dequeue_op 0x55c3700 finish
2014-06-12 14:46:10.842646 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:10.842650 7f9d623d5700 10 osd.2 173 dequeue_op 0x20354b00 prio 127 cost 516903 latency 0.007387 osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3164 (79'163,173'3164] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active]
2014-06-12 14:46:10.842636 7f9d633d7700  5 -- op tracker -- , seq: 747631, time: 2014-06-12 14:46:10.842633, event: done, op: osd_sub_op_reply(client.4291.0:140689 3.214 53a88214/rbd_data.109a6b8b4567.0000000000000f5d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.842689 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:10.842692 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.842702 7f9d623d5700  5 -- op tracker -- , seq: 747632, time: 2014-06-12 14:46:10.842666, event: reached_pg, op: osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.842709 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3164 (79'163,173'3164] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active] handle_message: 0x20354b00
2014-06-12 14:46:10.842720 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3164 (79'163,173'3164] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active] sub_op_modify trans 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 v 173'3165 (transaction) 174
2014-06-12 14:46:10.842735 7f9d623d5700  5 -- op tracker -- , seq: 747632, time: 2014-06-12 14:46:10.842732, event: started, op: osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.842750 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3164 (79'163,173'3164] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active] append_log log((79'163,173'3164], crt=0'0) [173'3165 (173'3164) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140123 2014-06-12 08:45:13.213304]
2014-06-12 14:46:10.842775 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3165 (79'163,173'3165] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active] add_log_entry 173'3165 (173'3164) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140123 2014-06-12 08:45:13.213304
2014-06-12 14:46:10.842796 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3165 (79'163,173'3165] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active] append_log  adding 1 keys
2014-06-12 14:46:10.842811 7f9d623d5700 10 trim log((79'163,173'3165], crt=0'0) to 79'164
2014-06-12 14:46:10.842837 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3165, trimmed: 79'164
2014-06-12 14:46:10.842862 7f9d623d5700  5 -- op tracker -- , seq: 747632, time: 2014-06-12 14:46:10.842858, event: started, op: osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.842873 7f9d623d5700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.842877 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.842878 7f9d623d5700  5 queue_op 0x6a8bc80 seq 2860 osr(3.36c 0x276d400) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:10.842884 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3165 (79'164,173'3165] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active] do_sub_op osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.842896 7f9d623d5700 10 osd.2 173 dequeue_op 0x20354b00 finish
2014-06-12 14:46:10.842899 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.842901 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.842907 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (1 active)
2014-06-12 14:46:10.842911 7f9d81970700  5 _do_op 0x6a8bc80 seq 2860 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:10.842915 7f9d81970700 10 _do_transaction on 0x3479b0b0
2014-06-12 14:46:10.842921 7f9d81970700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.843112 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.843144 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.843194 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.843219 7f9d81970700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.843229 7f9d81970700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:10.843235 7f9d81970700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.843253 7f9d81970700 15 _write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 3121152~516096
2014-06-12 14:46:10.843359 7f9d81970700 10 lookup_strip_header done  cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.843582 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:10.845014 7f9d81970700 10 _generic_write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 3121152~516096 = 0
2014-06-12 14:46:10.845067 7f9d81970700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.845083 7f9d81970700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.845089 7f9d81970700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.845097 7f9d81970700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.845244 7f9d81970700 20 set_header setting 16547 cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:46:10.845308 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.845325 7f9d81970700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:10.845341 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:10.847094 7f9d81970700 10 _do_op 0x6a8bc80 seq 2860 r = 0, finisher 0x792e520 0
2014-06-12 14:46:10.847111 7f9d81970700 10 _finish_op 0x6a8bc80 seq 2860 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.847118 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (0 active)
2014-06-12 14:46:10.847125 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.847132 7f9d7795c700  5 -- op tracker -- , seq: 747632, time: 2014-06-12 14:46:10.847124, event: sub_op_applied, op: osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.847145 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3165 (79'164,173'3165] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active] sub_op_modify_applied on 0x3479afc0 op osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.847167 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.847180 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3165 (79'164,173'3165] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active] op_applied on primary on version 173'3165
2014-06-12 14:46:10.847196 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3165 (79'164,173'3165] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active] op_applied on replica on version 173'3165
2014-06-12 14:46:10.847233 7f9d7715b700  5 -- op tracker -- , seq: 747632, time: 2014-06-12 14:46:10.847228, event: commit_sent, op: osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.847239 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3165 (79'164,173'3165] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3163 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:10.847251 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.847277 7f9d7715b700  5 -- op tracker -- , seq: 747632, time: 2014-06-12 14:46:10.847272, event: done, op: osd_sub_op(client.4300.0:140123 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3165 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.885851 7f9d56432700  5 -- op tracker -- , seq: 747633, time: 2014-06-12 14:46:10.884577, event: header_read, op: osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.885867 7f9d56432700  5 -- op tracker -- , seq: 747633, time: 2014-06-12 14:46:10.884580, event: throttled, op: osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.885876 7f9d56432700  5 -- op tracker -- , seq: 747633, time: 2014-06-12 14:46:10.885806, event: all_read, op: osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.885885 7f9d56432700  5 -- op tracker -- , seq: 747633, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.885891 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.885899 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:10.885909 7f9d56432700 15 osd.2 173 enqueue_op 0x17c2b600 prio 127 cost 516899 latency 0.001331 osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.885943 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.885954 7f9d643d9700 10 osd.2 173 dequeue_op 0x17c2b600 prio 127 cost 516899 latency 0.001376 osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2476 (0'0,173'2476] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2475 active]
2014-06-12 14:46:10.885981 7f9d643d9700  5 -- op tracker -- , seq: 747633, time: 2014-06-12 14:46:10.885977, event: reached_pg, op: osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.885991 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2476 (0'0,173'2476] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2475 active] handle_message: 0x17c2b600
2014-06-12 14:46:10.886014 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2476 (0'0,173'2476] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2475 active] sub_op_modify trans 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 v 173'2477 (transaction) 174
2014-06-12 14:46:10.886035 7f9d643d9700  5 -- op tracker -- , seq: 747633, time: 2014-06-12 14:46:10.886030, event: started, op: osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.886053 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2476 (0'0,173'2476] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2475 active] append_log log((0'0,173'2476], crt=0'0) [173'2477 (173'2475) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144991 2014-06-12 08:44:57.481413]
2014-06-12 14:46:10.886082 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2477 (0'0,173'2477] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2475 active] add_log_entry 173'2477 (173'2475) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144991 2014-06-12 08:44:57.481413
2014-06-12 14:46:10.886108 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2477 (0'0,173'2477] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2475 active] append_log  adding 1 keys
2014-06-12 14:46:10.886153 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2477, trimmed: 
2014-06-12 14:46:10.886182 7f9d643d9700  5 -- op tracker -- , seq: 747633, time: 2014-06-12 14:46:10.886176, event: started, op: osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.886195 7f9d643d9700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:10.886200 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.886202 7f9d643d9700  5 queue_op 0x7e9a480 seq 2290 osr(3.2b 0x276cd20) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:46:10.886210 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2477 (0'0,173'2477] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2475 active] do_sub_op osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.886226 7f9d643d9700 10 osd.2 173 dequeue_op 0x17c2b600 finish
2014-06-12 14:46:10.886230 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.886233 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.886243 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:46:10.886250 7f9d7a161700  5 _do_op 0x7e9a480 seq 2290 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:46:10.886255 7f9d7a161700 10 _do_transaction on 0x66770b0
2014-06-12 14:46:10.886263 7f9d7a161700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:10.886449 7f9d7a161700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:10.886475 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.886535 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.886560 7f9d7a161700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:10.886570 7f9d7a161700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:46:10.886577 7f9d7a161700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:10.886840 7f9d7a161700 15 _write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 2347008~516096
2014-06-12 14:46:10.887036 7f9d7a161700 10 lookup_strip_header done  cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:10.887360 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:10.890350 7f9d7a161700 10 _generic_write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 2347008~516096 = 0
2014-06-12 14:46:10.890457 7f9d7a161700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:10.890483 7f9d7a161700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:10.890491 7f9d7a161700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:10.890500 7f9d7a161700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:10.890718 7f9d7a161700 20 set_header setting 17866 cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 parent seq 0
2014-06-12 14:46:10.890797 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.890814 7f9d7a161700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:46:10.890823 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:10.893941 7f9d7a161700 10 _do_op 0x7e9a480 seq 2290 r = 0, finisher 0x1e3be340 0
2014-06-12 14:46:10.893968 7f9d7a161700 10 _finish_op 0x7e9a480 seq 2290 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:10.893975 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:46:10.893991 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.893995 7f9d7715b700  5 -- op tracker -- , seq: 747633, time: 2014-06-12 14:46:10.893984, event: commit_sent, op: osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.894008 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2477 (0'0,173'2477] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2475 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:10.894037 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:10.894096 7f9d7795c700  5 -- op tracker -- , seq: 747633, time: 2014-06-12 14:46:10.894089, event: sub_op_applied, op: osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.894107 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2477 (0'0,173'2477] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2476 active] sub_op_modify_applied on 0x6676fc0 op osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.894126 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2477 (0'0,173'2477] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2476 active] op_applied on primary on version 173'2477
2014-06-12 14:46:10.894136 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2477 (0'0,173'2477] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2476 active] op_applied on replica on version 173'2477
2014-06-12 14:46:10.894177 7f9d7795c700  5 -- op tracker -- , seq: 747633, time: 2014-06-12 14:46:10.894172, event: done, op: osd_sub_op(client.4314.0:144991 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.899299 7f9d5b6bc700  5 -- op tracker -- , seq: 747634, time: 2014-06-12 14:46:10.893275, event: header_read, op: osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.899315 7f9d5b6bc700  5 -- op tracker -- , seq: 747634, time: 2014-06-12 14:46:10.893277, event: throttled, op: osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.899324 7f9d5b6bc700  5 -- op tracker -- , seq: 747634, time: 2014-06-12 14:46:10.899246, event: all_read, op: osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.899332 7f9d5b6bc700  5 -- op tracker -- , seq: 747634, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.899339 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.899346 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:10.899355 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x59dca00 prio 127 cost 1434407 latency 0.006079 osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.899386 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.899395 7f9d61bd4700 10 osd.2 173 dequeue_op 0x59dca00 prio 127 cost 1434407 latency 0.006120 osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.7fb( v 173'3019 (79'18,173'3019] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3018 active]
2014-06-12 14:46:10.899420 7f9d61bd4700  5 -- op tracker -- , seq: 747634, time: 2014-06-12 14:46:10.899417, event: reached_pg, op: osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.899430 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3019 (79'18,173'3019] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3018 active] handle_message: 0x59dca00
2014-06-12 14:46:10.899438 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3019 (79'18,173'3019] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3018 active] sub_op_modify trans ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 v 173'3020 (transaction) 174
2014-06-12 14:46:10.899449 7f9d61bd4700  5 -- op tracker -- , seq: 747634, time: 2014-06-12 14:46:10.899447, event: started, op: osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.899486 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3019 (79'18,173'3019] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3018 active] append_log log((79'18,173'3019], crt=0'0) [173'3020 (173'936) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146052 2014-06-12 08:45:13.262414]
2014-06-12 14:46:10.899505 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3020 (79'18,173'3020] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3018 active] add_log_entry 173'3020 (173'936) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146052 2014-06-12 08:45:13.262414
2014-06-12 14:46:10.899521 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3020 (79'18,173'3020] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3018 active] append_log  adding 1 keys
2014-06-12 14:46:10.899529 7f9d61bd4700 10 trim log((79'18,173'3020], crt=0'0) to 79'19
2014-06-12 14:46:10.899551 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3020, trimmed: 79'19
2014-06-12 14:46:10.899571 7f9d61bd4700  5 -- op tracker -- , seq: 747634, time: 2014-06-12 14:46:10.899566, event: started, op: osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.899586 7f9d61bd4700  5 queue_transactions existing osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:10.899591 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.899593 7f9d61bd4700  5 queue_op 0x93061e0 seq 2700 osr(3.7fb 0x49b42f0) 1435928 bytes   (queue has 1 ops and 1435928 bytes)
2014-06-12 14:46:10.899606 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c57a0 (1 active)
2014-06-12 14:46:10.899616 7f9d7a962700  5 _do_op 0x93061e0 seq 2700 osr(3.7fb 0x49b42f0)/0x49b42f0 start
2014-06-12 14:46:10.899601 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3020 (79'19,173'3020] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3018 active] do_sub_op osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.899621 7f9d61bd4700 10 osd.2 173 dequeue_op 0x59dca00 finish
2014-06-12 14:46:10.899624 7f9d7a962700 10 _do_transaction on 0x591d270
2014-06-12 14:46:10.899628 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.899632 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.899634 7f9d7a962700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:10.899804 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:10.899831 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.899877 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.899903 7f9d7a962700 15 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:10.899913 7f9d7a962700 10 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1 r = 0
2014-06-12 14:46:10.899918 7f9d7a962700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:10.900211 7f9d7a962700 15 _write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 0~1433600
2014-06-12 14:46:10.900380 7f9d7a962700 10 lookup_strip_header done  cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:10.900594 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:10.911786 7f9d7a962700 10 _generic_write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 0~1433600 = 0
2014-06-12 14:46:10.912136 7f9d7a962700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:10.912171 7f9d7a962700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:10.912178 7f9d7a962700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:10.912191 7f9d7a962700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:10.912769 7f9d7a962700 20 set_header setting 17147 cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 parent seq 0
2014-06-12 14:46:10.912961 7f9d7a962700 20 set_header setting 2206 cid meta oid 1fe728d6/pglog_3.7fb/0//-1 parent seq 0
2014-06-12 14:46:10.912979 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.912989 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:10.931033 7f9d58c5a700  5 -- op tracker -- , seq: 747635, time: 2014-06-12 14:46:10.927404, event: header_read, op: osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.931055 7f9d58c5a700  5 -- op tracker -- , seq: 747635, time: 2014-06-12 14:46:10.927405, event: throttled, op: osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.931063 7f9d58c5a700  5 -- op tracker -- , seq: 747635, time: 2014-06-12 14:46:10.930972, event: all_read, op: osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.931071 7f9d58c5a700  5 -- op tracker -- , seq: 747635, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.931078 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.931092 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:10.931103 7f9d58c5a700 15 osd.2 173 enqueue_op 0x20355600 prio 127 cost 516903 latency 0.003699 osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.931137 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.931147 7f9d663dd700 10 osd.2 173 dequeue_op 0x20355600 prio 127 cost 516903 latency 0.003742 osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3165 (79'164,173'3165] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3164 active]
2014-06-12 14:46:10.931173 7f9d663dd700  5 -- op tracker -- , seq: 747635, time: 2014-06-12 14:46:10.931169, event: reached_pg, op: osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.931182 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3165 (79'164,173'3165] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3164 active] handle_message: 0x20355600
2014-06-12 14:46:10.931195 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3165 (79'164,173'3165] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3164 active] sub_op_modify trans 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 v 173'3166 (transaction) 174
2014-06-12 14:46:10.931212 7f9d663dd700  5 -- op tracker -- , seq: 747635, time: 2014-06-12 14:46:10.931208, event: started, op: osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.931230 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3165 (79'164,173'3165] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3164 active] append_log log((79'164,173'3165], crt=0'0) [173'3166 (173'3165) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140124 2014-06-12 08:45:13.303394]
2014-06-12 14:46:10.931254 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3166 (79'164,173'3166] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3164 active] add_log_entry 173'3166 (173'3165) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140124 2014-06-12 08:45:13.303394
2014-06-12 14:46:10.931279 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3166 (79'164,173'3166] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3164 active] append_log  adding 1 keys
2014-06-12 14:46:10.931292 7f9d663dd700 10 trim log((79'164,173'3166], crt=0'0) to 79'165
2014-06-12 14:46:10.931321 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3166, trimmed: 79'165
2014-06-12 14:46:10.931354 7f9d663dd700  5 -- op tracker -- , seq: 747635, time: 2014-06-12 14:46:10.931349, event: started, op: osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.931365 7f9d663dd700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.931369 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.931371 7f9d663dd700  5 queue_op 0xa2dac00 seq 2861 osr(3.36c 0x276d400) 518424 bytes   (queue has 2 ops and 1954352 bytes)
2014-06-12 14:46:10.931378 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3166 (79'165,173'3166] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3164 active] do_sub_op osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.931393 7f9d663dd700 10 osd.2 173 dequeue_op 0x20355600 finish
2014-06-12 14:46:10.931397 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.931399 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.931409 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (2 active)
2014-06-12 14:46:10.931416 7f9d7f16b700  5 _do_op 0xa2dac00 seq 2861 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:10.931421 7f9d7f16b700 10 _do_transaction on 0xc9b4d30
2014-06-12 14:46:10.931427 7f9d7f16b700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.931589 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.931618 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.931677 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.931701 7f9d7f16b700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.931708 7f9d7f16b700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:10.931714 7f9d7f16b700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.931947 7f9d7f16b700 15 _write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 3637248~516096
2014-06-12 14:46:10.932104 7f9d7f16b700 10 lookup_strip_header done  cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.932246 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:10.933924 7f9d7a962700 10 _do_op 0x93061e0 seq 2700 r = 0, finisher 0x73be440 0
2014-06-12 14:46:10.933943 7f9d7a962700 10 _finish_op 0x93061e0 seq 2700 osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:10.933951 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c57a0 (1 active)
2014-06-12 14:46:10.933958 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.933961 7f9d7715b700  5 -- op tracker -- , seq: 747634, time: 2014-06-12 14:46:10.933951, event: commit_sent, op: osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.933970 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3020 (79'19,173'3020] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3018 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:10.933993 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:10.934023 7f9d7795c700  5 -- op tracker -- , seq: 747634, time: 2014-06-12 14:46:10.934017, event: sub_op_applied, op: osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.934040 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3020 (79'19,173'3020] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active] sub_op_modify_applied on 0x591d180 op osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.934058 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3020 (79'19,173'3020] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active] op_applied on primary on version 173'3020
2014-06-12 14:46:10.934068 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3020 (79'19,173'3020] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active] op_applied on replica on version 173'3020
2014-06-12 14:46:10.934093 7f9d7795c700  5 -- op tracker -- , seq: 747634, time: 2014-06-12 14:46:10.934089, event: done, op: osd_sub_op(client.4302.0:146052 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3020 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.934414 7f9d7f16b700 10 _generic_write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 3637248~516096 = 0
2014-06-12 14:46:10.934457 7f9d7f16b700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.934472 7f9d7f16b700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.934478 7f9d7f16b700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.934486 7f9d7f16b700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.934666 7f9d7f16b700 20 set_header setting 16547 cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:46:10.934732 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.934746 7f9d7f16b700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:10.934756 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:10.938340 7f9d7f16b700 10 _do_op 0xa2dac00 seq 2861 r = 0, finisher 0x738b240 0
2014-06-12 14:46:10.938391 7f9d7f16b700 10 _finish_op 0xa2dac00 seq 2861 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.938414 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (0 active)
2014-06-12 14:46:10.938450 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.939462 7f9d7795c700  5 -- op tracker -- , seq: 747635, time: 2014-06-12 14:46:10.939447, event: sub_op_applied, op: osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.939477 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3166 (79'165,173'3166] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3164 active] sub_op_modify_applied on 0xc9b4c40 op osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.939512 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.939539 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3166 (79'165,173'3166] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3164 active] op_applied on primary on version 173'3166
2014-06-12 14:46:10.939551 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3166 (79'165,173'3166] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3164 active] op_applied on replica on version 173'3166
2014-06-12 14:46:10.941655 7f9d7715b700  5 -- op tracker -- , seq: 747635, time: 2014-06-12 14:46:10.941639, event: commit_sent, op: osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.941686 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3166 (79'165,173'3166] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3164 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:10.941713 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.941760 7f9d7715b700  5 -- op tracker -- , seq: 747635, time: 2014-06-12 14:46:10.941754, event: done, op: osd_sub_op(client.4300.0:140124 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3166 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.942879 7f9d5a16f700  5 -- op tracker -- , seq: 747636, time: 2014-06-12 14:46:10.938374, event: header_read, op: osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.942894 7f9d5a16f700  5 -- op tracker -- , seq: 747636, time: 2014-06-12 14:46:10.938378, event: throttled, op: osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.942903 7f9d5a16f700  5 -- op tracker -- , seq: 747636, time: 2014-06-12 14:46:10.942829, event: all_read, op: osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.942912 7f9d5a16f700  5 -- op tracker -- , seq: 747636, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.942917 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.942923 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:10.942931 7f9d5a16f700 15 osd.2 173 enqueue_op 0x9facd00 prio 127 cost 516903 latency 0.004557 osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.942974 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.942984 7f9d673df700 10 osd.2 173 dequeue_op 0x9facd00 prio 127 cost 516903 latency 0.004610 osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2657 (0'0,173'2657] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active]
2014-06-12 14:46:10.943013 7f9d673df700  5 -- op tracker -- , seq: 747636, time: 2014-06-12 14:46:10.943008, event: reached_pg, op: osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.943024 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2657 (0'0,173'2657] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active] handle_message: 0x9facd00
2014-06-12 14:46:10.943042 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2657 (0'0,173'2657] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active] sub_op_modify trans f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 v 173'2658 (transaction) 174
2014-06-12 14:46:10.943058 7f9d673df700  5 -- op tracker -- , seq: 747636, time: 2014-06-12 14:46:10.943054, event: started, op: osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.943074 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2657 (0'0,173'2657] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active] append_log log((0'0,173'2657], crt=0'0) [173'2658 (173'2656) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146405 2014-06-12 08:41:29.628194]
2014-06-12 14:46:10.943103 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2658 (0'0,173'2658] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active] add_log_entry 173'2658 (173'2656) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146405 2014-06-12 08:41:29.628194
2014-06-12 14:46:10.943125 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2658 (0'0,173'2658] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active] append_log  adding 1 keys
2014-06-12 14:46:10.943162 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2658, trimmed: 
2014-06-12 14:46:10.943186 7f9d673df700  5 -- op tracker -- , seq: 747636, time: 2014-06-12 14:46:10.943182, event: started, op: osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.943197 7f9d673df700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:10.943202 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.943204 7f9d673df700  5 queue_op 0x3886660 seq 2468 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:10.943211 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2658 (0'0,173'2658] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active] do_sub_op osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.943225 7f9d673df700 10 osd.2 173 dequeue_op 0x9facd00 finish
2014-06-12 14:46:10.943229 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.943231 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.943241 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:46:10.943254 7f9d7895e700  5 _do_op 0x3886660 seq 2468 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:10.943258 7f9d7895e700 10 _do_transaction on 0x17a2dcf0
2014-06-12 14:46:10.943264 7f9d7895e700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:10.943423 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:10.943455 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.943505 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.943528 7f9d7895e700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:10.943533 7f9d7895e700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:10.943538 7f9d7895e700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:10.943720 7f9d7895e700 15 _write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 294912~516096
2014-06-12 14:46:10.943876 7f9d7895e700 10 lookup_strip_header done  cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:10.944020 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:10.946244 7f9d7895e700 10 _generic_write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 294912~516096 = 0
2014-06-12 14:46:10.946319 7f9d7895e700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:10.946337 7f9d7895e700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:10.946342 7f9d7895e700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:10.946357 7f9d7895e700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:10.946559 7f9d7895e700 20 set_header setting 16921 cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 parent seq 0
2014-06-12 14:46:10.946637 7f9d7895e700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:10.946652 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.946659 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:10.949123 7f9d7895e700 10 _do_op 0x3886660 seq 2468 r = 0, finisher 0x98eec80 0
2014-06-12 14:46:10.949142 7f9d7895e700 10 _finish_op 0x3886660 seq 2468 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:10.949147 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:46:10.949154 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.949493 7f9d7795c700  5 -- op tracker -- , seq: 747636, time: 2014-06-12 14:46:10.949482, event: sub_op_applied, op: osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.949510 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2658 (0'0,173'2658] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active] sub_op_modify_applied on 0x17a2dc00 op osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.949535 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:10.949553 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2658 (0'0,173'2658] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active] op_applied on primary on version 173'2658
2014-06-12 14:46:10.949563 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2658 (0'0,173'2658] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active] op_applied on replica on version 173'2658
2014-06-12 14:46:10.950070 7f9d7715b700  5 -- op tracker -- , seq: 747636, time: 2014-06-12 14:46:10.950059, event: commit_sent, op: osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.950083 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2658 (0'0,173'2658] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2656 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:10.950107 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:10.950144 7f9d7715b700  5 -- op tracker -- , seq: 747636, time: 2014-06-12 14:46:10.950139, event: done, op: osd_sub_op(client.4316.0:146405 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2658 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.954270 7f9d56432700  5 -- op tracker -- , seq: 747637, time: 2014-06-12 14:46:10.954215, event: header_read, op: osd_sub_op_reply(client.4309.0:141524 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.954285 7f9d56432700  5 -- op tracker -- , seq: 747637, time: 2014-06-12 14:46:10.954217, event: throttled, op: osd_sub_op_reply(client.4309.0:141524 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.954292 7f9d56432700  5 -- op tracker -- , seq: 747637, time: 2014-06-12 14:46:10.954229, event: all_read, op: osd_sub_op_reply(client.4309.0:141524 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.954298 7f9d56432700  5 -- op tracker -- , seq: 747637, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141524 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.954307 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141524 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:10.954323 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:10.954334 7f9d56432700 15 osd.2 173 enqueue_op 0x12bf7500 prio 196 cost 0 latency 0.000119 osd_sub_op_reply(client.4309.0:141524 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:10.954391 7f9d56432700  5 -- op tracker -- , seq: 747638, time: 2014-06-12 14:46:10.954359, event: header_read, op: osd_sub_op_reply(client.4306.0:144049 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.954399 7f9d56432700  5 -- op tracker -- , seq: 747638, time: 2014-06-12 14:46:10.954360, event: throttled, op: osd_sub_op_reply(client.4306.0:144049 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.954406 7f9d56432700  5 -- op tracker -- , seq: 747638, time: 2014-06-12 14:46:10.954372, event: all_read, op: osd_sub_op_reply(client.4306.0:144049 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.954412 7f9d56432700  5 -- op tracker -- , seq: 747638, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144049 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.954416 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144049 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:10.954421 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:10.954425 7f9d56432700 15 osd.2 173 enqueue_op 0x2910700 prio 196 cost 0 latency 0.000066 osd_sub_op_reply(client.4306.0:144049 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:10.954451 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.954459 7f9d693e3700 10 osd.2 173 dequeue_op 0x12bf7500 prio 196 cost 0 latency 0.000243 osd_sub_op_reply(client.4309.0:141524 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 pg pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2877 active+clean]
2014-06-12 14:46:10.954478 7f9d693e3700  5 -- op tracker -- , seq: 747637, time: 2014-06-12 14:46:10.954475, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141524 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.954489 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2877 active+clean] handle_message: 0x12bf7500
2014-06-12 14:46:10.954505 7f9d693e3700  5 -- op tracker -- , seq: 747637, time: 2014-06-12 14:46:10.954502, event: started, op: osd_sub_op_reply(client.4309.0:141524 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.954511 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2877 active+clean] sub_op_modify_reply: tid 236046 op  ack_type 4 from 26
2014-06-12 14:46:10.954534 7f9d693e3700  5 -- op tracker -- , seq: 747031, time: 2014-06-12 14:46:10.954525, event: sub_op_commit_rec, op: osd_op(client.4309.0:141524 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 315392~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.954541 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2877 active+clean] repop_all_applied: repop tid 236046 all applied 
2014-06-12 14:46:10.954552 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2877 active+clean] eval_repop repgather(0x1771c480 173'2879 rep_tid=236046 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141524 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 315392~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.954568 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2877 active+clean]  sending ack on repgather(0x1771c480 173'2879 rep_tid=236046 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141524 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 315392~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0x19e0aa00
2014-06-12 14:46:10.954599 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2878 crt=0'0 lcod 173'2877 mlcod 173'2877 active+clean] repop_all_committed: repop tid 236046 all committed 
2014-06-12 14:46:10.954609 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2877 active+clean] eval_repop repgather(0x1771c480 173'2879 rep_tid=236046 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141524 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 315392~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.954637 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2877 active+clean] log_op_stats osd_op(client.4309.0:141524 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 315392~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 32.272826 lat 32.272864
2014-06-12 14:46:10.954654 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2877 active+clean] publish_stats_to_osd 173:3265
2014-06-12 14:46:10.954668 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2877 active+clean]  sending commit on repgather(0x1771c480 173'2879 rep_tid=236046 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141524 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 315392~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0x16cd2280
2014-06-12 14:46:10.954695 7f9d693e3700  5 -- op tracker -- , seq: 747031, time: 2014-06-12 14:46:10.954690, event: commit_sent, op: osd_op(client.4309.0:141524 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 315392~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.954705 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2877 active+clean]  requeue_ops 
2014-06-12 14:46:10.954717 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean]  removing repgather(0x1771c480 173'2879 rep_tid=236046 committed?=1 applied?=1 op=osd_op(client.4309.0:141524 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 315392~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.954732 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean]    q front is repgather(0x1771c480 173'2879 rep_tid=236046 committed?=1 applied?=1 op=osd_op(client.4309.0:141524 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 315392~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.954747 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean] remove_repop repgather(0x1771c480 173'2879 rep_tid=236046 committed?=1 applied?=1 op=osd_op(client.4309.0:141524 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 315392~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.954761 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean]  requeue_ops 
2014-06-12 14:46:10.954788 7f9d693e3700  5 -- op tracker -- , seq: 747031, time: 2014-06-12 14:46:10.954783, event: done, op: osd_op(client.4309.0:141524 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 315392~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.954816 7f9d693e3700 10 osd.2 173 dequeue_op 0x12bf7500 finish
2014-06-12 14:46:10.954823 7f9d693e3700  5 -- op tracker -- , seq: 747637, time: 2014-06-12 14:46:10.954820, event: done, op: osd_sub_op_reply(client.4309.0:141524 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.954833 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.954840 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.954843 7f9d693e3700 10 osd.2 173 dequeue_op 0x2910700 prio 196 cost 0 latency 0.000484 osd_sub_op_reply(client.4306.0:144049 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3084 active+clean]
2014-06-12 14:46:10.954861 7f9d693e3700  5 -- op tracker -- , seq: 747638, time: 2014-06-12 14:46:10.954858, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144049 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.954867 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3084 active+clean] handle_message: 0x2910700
2014-06-12 14:46:10.954880 7f9d693e3700  5 -- op tracker -- , seq: 747638, time: 2014-06-12 14:46:10.954877, event: started, op: osd_sub_op_reply(client.4306.0:144049 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.954886 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3084 active+clean] sub_op_modify_reply: tid 236017 op  ack_type 4 from 26
2014-06-12 14:46:10.954905 7f9d693e3700  5 -- op tracker -- , seq: 746964, time: 2014-06-12 14:46:10.954899, event: sub_op_commit_rec, op: osd_op(client.4306.0:144049 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2625536~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.954910 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3084 active+clean] repop_all_applied: repop tid 236017 all applied 
2014-06-12 14:46:10.954920 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3084 active+clean] eval_repop repgather(0x2b6198c0 173'3086 rep_tid=236017 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144049 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2625536~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.954935 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3084 active+clean]  sending ack on repgather(0x2b6198c0 173'3086 rep_tid=236017 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144049 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2625536~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x9657900
2014-06-12 14:46:10.954965 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3085 crt=0'0 lcod 173'3084 mlcod 173'3084 active+clean] repop_all_committed: repop tid 236017 all committed 
2014-06-12 14:46:10.954975 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3084 active+clean] eval_repop repgather(0x2b6198c0 173'3086 rep_tid=236017 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144049 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2625536~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:10.954990 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3084 active+clean] log_op_stats osd_op(client.4306.0:144049 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2625536~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 33.277161 lat 33.277186
2014-06-12 14:46:10.955010 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3084 active+clean] publish_stats_to_osd 173:3552
2014-06-12 14:46:10.955021 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3084 active+clean]  sending commit on repgather(0x2b6198c0 173'3086 rep_tid=236017 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144049 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2625536~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x2b166000
2014-06-12 14:46:10.955047 7f9d693e3700  5 -- op tracker -- , seq: 746964, time: 2014-06-12 14:46:10.955043, event: commit_sent, op: osd_op(client.4306.0:144049 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2625536~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.955054 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3084 active+clean]  requeue_ops 
2014-06-12 14:46:10.955063 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3085 active+clean]  removing repgather(0x2b6198c0 173'3086 rep_tid=236017 committed?=1 applied?=1 op=osd_op(client.4306.0:144049 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2625536~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.955077 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3085 active+clean]    q front is repgather(0x2b6198c0 173'3086 rep_tid=236017 committed?=1 applied?=1 op=osd_op(client.4306.0:144049 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2625536~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.955090 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3085 active+clean] remove_repop repgather(0x2b6198c0 173'3086 rep_tid=236017 committed?=1 applied?=1 op=osd_op(client.4306.0:144049 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2625536~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:10.955102 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3085 active+clean]  requeue_ops 
2014-06-12 14:46:10.955121 7f9d693e3700  5 -- op tracker -- , seq: 746964, time: 2014-06-12 14:46:10.955117, event: done, op: osd_op(client.4306.0:144049 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 2625536~516096] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:10.955135 7f9d693e3700 10 osd.2 173 dequeue_op 0x2910700 finish
2014-06-12 14:46:10.955147 7f9d693e3700  5 -- op tracker -- , seq: 747638, time: 2014-06-12 14:46:10.955144, event: done, op: osd_sub_op_reply(client.4306.0:144049 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:10.955156 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.955159 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.955169 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.970132 7f9d58c5a700  5 -- op tracker -- , seq: 747639, time: 2014-06-12 14:46:10.969989, event: header_read, op: osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.970146 7f9d58c5a700  5 -- op tracker -- , seq: 747639, time: 2014-06-12 14:46:10.969992, event: throttled, op: osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.970155 7f9d58c5a700  5 -- op tracker -- , seq: 747639, time: 2014-06-12 14:46:10.970083, event: all_read, op: osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.970163 7f9d58c5a700  5 -- op tracker -- , seq: 747639, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.970169 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:10.970176 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:10.970186 7f9d58c5a700 15 osd.2 173 enqueue_op 0xe20b100 prio 127 cost 41767 latency 0.000197 osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.970225 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:10.970237 7f9d6b3e7700 10 osd.2 173 dequeue_op 0xe20b100 prio 127 cost 41767 latency 0.000248 osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3166 (79'165,173'3166] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3165 active]
2014-06-12 14:46:10.970265 7f9d6b3e7700  5 -- op tracker -- , seq: 747639, time: 2014-06-12 14:46:10.970260, event: reached_pg, op: osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.970276 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3166 (79'165,173'3166] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3165 active] handle_message: 0xe20b100
2014-06-12 14:46:10.970289 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3166 (79'165,173'3166] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3165 active] sub_op_modify trans 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 v 173'3167 (transaction) 174
2014-06-12 14:46:10.970307 7f9d6b3e7700  5 -- op tracker -- , seq: 747639, time: 2014-06-12 14:46:10.970303, event: started, op: osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.970331 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3166 (79'165,173'3166] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3165 active] append_log log((79'165,173'3166], crt=0'0) [173'3167 (173'3166) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140125 2014-06-12 08:45:13.344736]
2014-06-12 14:46:10.970364 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3167 (79'165,173'3167] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3165 active] add_log_entry 173'3167 (173'3166) modify   58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 by client.4300.0:140125 2014-06-12 08:45:13.344736
2014-06-12 14:46:10.970387 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3167 (79'165,173'3167] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3165 active] append_log  adding 1 keys
2014-06-12 14:46:10.970401 7f9d6b3e7700 10 trim log((79'165,173'3167], crt=0'0) to 79'166
2014-06-12 14:46:10.970434 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3167, trimmed: 79'166
2014-06-12 14:46:10.970464 7f9d6b3e7700  5 -- op tracker -- , seq: 747639, time: 2014-06-12 14:46:10.970459, event: started, op: osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.970475 7f9d6b3e7700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.970480 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:10.970482 7f9d6b3e7700  5 queue_op 0x80770e0 seq 2862 osr(3.36c 0x276d400) 43288 bytes   (queue has 1 ops and 43288 bytes)
2014-06-12 14:46:10.970491 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3167 (79'166,173'3167] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3165 active] do_sub_op osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.970507 7f9d6b3e7700 10 osd.2 173 dequeue_op 0xe20b100 finish
2014-06-12 14:46:10.970512 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:10.970515 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:10.971782 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (1 active)
2014-06-12 14:46:10.971806 7f9d7e169700  5 _do_op 0x80770e0 seq 2862 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:10.971816 7f9d7e169700 10 _do_transaction on 0x5881eb0
2014-06-12 14:46:10.971830 7f9d7e169700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.972001 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.972026 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:10.972084 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:10.972106 7f9d7e169700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.972114 7f9d7e169700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:10.972121 7f9d7e169700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:10.972142 7f9d7e169700 15 _write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 4153344~40960
2014-06-12 14:46:10.972298 7f9d7e169700 10 lookup_strip_header done  cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.972433 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:10.972597 7f9d7e169700 10 _generic_write 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 4153344~40960 = 0
2014-06-12 14:46:10.972609 7f9d7e169700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.972619 7f9d7e169700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.972625 7f9d7e169700 15 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3
2014-06-12 14:46:10.972633 7f9d7e169700 10 _setattrs 3.36c_head/58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 = 0
2014-06-12 14:46:10.972754 7f9d7e169700 20 set_header setting 16547 cid 3.36c_head oid 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 parent seq 0
2014-06-12 14:46:10.972777 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:10.972790 7f9d7e169700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:10.972801 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:10.973128 7f9d7e169700 10 _do_op 0x80770e0 seq 2862 r = 0, finisher 0x80c3800 0
2014-06-12 14:46:10.973142 7f9d7e169700 10 _finish_op 0x80770e0 seq 2862 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:10.973150 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (0 active)
2014-06-12 14:46:10.973153 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:10.973160 7f9d7715b700  5 -- op tracker -- , seq: 747639, time: 2014-06-12 14:46:10.973150, event: commit_sent, op: osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.973170 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3167 (79'166,173'3167] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3165 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:10.973199 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:10.973235 7f9d7795c700  5 -- op tracker -- , seq: 747639, time: 2014-06-12 14:46:10.973228, event: sub_op_applied, op: osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:10.973242 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3167 (79'166,173'3167] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3166 active] sub_op_modify_applied on 0x5881dc0 op osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:10.973259 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3167 (79'166,173'3167] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3166 active] op_applied on primary on version 173'3167
2014-06-12 14:46:10.973269 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3167 (79'166,173'3167] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3166 active] op_applied on replica on version 173'3167
2014-06-12 14:46:10.973296 7f9d7795c700  5 -- op tracker -- , seq: 747639, time: 2014-06-12 14:46:10.973291, event: done, op: osd_sub_op(client.4300.0:140125 3.36c 58ca136c/rbd_data.10a36b8b4567.0000000000000ecc/head//3 [] v 173'3167 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.014960 7f9d58c5a700  5 -- op tracker -- , seq: 747640, time: 2014-06-12 14:46:11.011971, event: header_read, op: osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.014974 7f9d58c5a700  5 -- op tracker -- , seq: 747640, time: 2014-06-12 14:46:11.011973, event: throttled, op: osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.014983 7f9d58c5a700  5 -- op tracker -- , seq: 747640, time: 2014-06-12 14:46:11.014913, event: all_read, op: osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.014991 7f9d58c5a700  5 -- op tracker -- , seq: 747640, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.014996 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:11.015003 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:11.015011 7f9d58c5a700 15 osd.2 173 enqueue_op 0x2d17ae00 prio 127 cost 1032999 latency 0.003040 osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.018735 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.018752 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2d17ae00 prio 127 cost 1032999 latency 0.006780 osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3167 (79'166,173'3167] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3166 active]
2014-06-12 14:46:11.018787 7f9d62bd6700  5 -- op tracker -- , seq: 747640, time: 2014-06-12 14:46:11.018782, event: reached_pg, op: osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.018800 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3167 (79'166,173'3167] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3166 active] handle_message: 0x2d17ae00
2014-06-12 14:46:11.018813 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3167 (79'166,173'3167] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3166 active] sub_op_modify trans a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 v 173'3168 (transaction) 174
2014-06-12 14:46:11.018831 7f9d62bd6700  5 -- op tracker -- , seq: 747640, time: 2014-06-12 14:46:11.018827, event: started, op: osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.018851 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3167 (79'166,173'3167] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3166 active] append_log log((79'166,173'3167], crt=0'0) [173'3168 (173'3157) modify   a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 by client.4294.0:144015 2014-06-12 08:45:13.386194]
2014-06-12 14:46:11.018879 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3168 (79'166,173'3168] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3166 active] add_log_entry 173'3168 (173'3157) modify   a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 by client.4294.0:144015 2014-06-12 08:45:13.386194
2014-06-12 14:46:11.018905 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3168 (79'166,173'3168] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3166 active] append_log  adding 1 keys
2014-06-12 14:46:11.018920 7f9d62bd6700 10 trim log((79'166,173'3168], crt=0'0) to 79'167
2014-06-12 14:46:11.018951 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3168, trimmed: 79'167
2014-06-12 14:46:11.018982 7f9d62bd6700  5 -- op tracker -- , seq: 747640, time: 2014-06-12 14:46:11.018977, event: started, op: osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.018995 7f9d62bd6700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:11.019000 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.019002 7f9d62bd6700  5 queue_op 0x7fbb920 seq 2863 osr(3.36c 0x276d400) 1034520 bytes   (queue has 1 ops and 1034520 bytes)
2014-06-12 14:46:11.019010 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3168 (79'167,173'3168] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3166 active] do_sub_op osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.019024 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2d17ae00 finish
2014-06-12 14:46:11.019027 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.019029 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.019038 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (1 active)
2014-06-12 14:46:11.019043 7f9d7e96a700  5 _do_op 0x7fbb920 seq 2863 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:11.019047 7f9d7e96a700 10 _do_transaction on 0x90f0630
2014-06-12 14:46:11.019054 7f9d7e96a700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:11.019218 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:11.019239 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.019294 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.019312 7f9d7e96a700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:11.019320 7f9d7e96a700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:11.019327 7f9d7e96a700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:11.019341 7f9d7e96a700 15 _write 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 1396736~1032192
2014-06-12 14:46:11.019508 7f9d7e96a700 10 lookup_strip_header done  cid 3.36c_head oid a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3
2014-06-12 14:46:11.019534 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:11.025034 7f9d7e96a700 10 _generic_write 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 1396736~1032192 = 0
2014-06-12 14:46:11.025216 7f9d7e96a700 15 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3
2014-06-12 14:46:11.025235 7f9d7e96a700 10 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 = 0
2014-06-12 14:46:11.025241 7f9d7e96a700 15 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3
2014-06-12 14:46:11.025250 7f9d7e96a700 10 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 = 0
2014-06-12 14:46:11.025655 7f9d7e96a700 20 set_header setting 17739 cid 3.36c_head oid a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 parent seq 0
2014-06-12 14:46:11.025824 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.025847 7f9d7e96a700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:11.025858 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:11.038132 7f9d7e96a700 10 _do_op 0x7fbb920 seq 2863 r = 0, finisher 0x7758220 0
2014-06-12 14:46:11.045234 7f9d7e96a700 10 _finish_op 0x7fbb920 seq 2863 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:11.045247 7f9d7715b700  5 -- op tracker -- , seq: 747640, time: 2014-06-12 14:46:11.045239, event: commit_sent, op: osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.045260 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (0 active)
2014-06-12 14:46:11.045270 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.045263 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3168 (79'167,173'3168] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3166 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:11.045288 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:11.049665 7f9d7795c700  5 -- op tracker -- , seq: 747640, time: 2014-06-12 14:46:11.049648, event: sub_op_applied, op: osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.049681 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3168 (79'167,173'3168] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3167 active] sub_op_modify_applied on 0x90f0540 op osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.049702 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3168 (79'167,173'3168] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3167 active] op_applied on primary on version 173'3168
2014-06-12 14:46:11.049712 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3168 (79'167,173'3168] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3167 active] op_applied on replica on version 173'3168
2014-06-12 14:46:11.049756 7f9d7795c700  5 -- op tracker -- , seq: 747640, time: 2014-06-12 14:46:11.049752, event: done, op: osd_sub_op(client.4294.0:144015 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3168 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.102854 7f9d58c5a700  5 -- op tracker -- , seq: 747641, time: 2014-06-12 14:46:11.100302, event: header_read, op: osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.102867 7f9d58c5a700  5 -- op tracker -- , seq: 747641, time: 2014-06-12 14:46:11.100304, event: throttled, op: osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.103081 7f9d58c5a700  5 -- op tracker -- , seq: 747641, time: 2014-06-12 14:46:11.102621, event: all_read, op: osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.103090 7f9d58c5a700  5 -- op tracker -- , seq: 747641, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.103096 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:11.103103 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:11.103111 7f9d58c5a700 15 osd.2 173 enqueue_op 0x68a6800 prio 127 cost 1032999 latency 0.002809 osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.103771 7f9d56432700  5 -- op tracker -- , seq: 747642, time: 2014-06-12 14:46:11.102396, event: header_read, op: osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.103780 7f9d56432700  5 -- op tracker -- , seq: 747642, time: 2014-06-12 14:46:11.102398, event: throttled, op: osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.103787 7f9d56432700  5 -- op tracker -- , seq: 747642, time: 2014-06-12 14:46:11.103739, event: all_read, op: osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.103794 7f9d56432700  5 -- op tracker -- , seq: 747642, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.103798 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:11.103803 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:11.103808 7f9d56432700 15 osd.2 173 enqueue_op 0x28dd0d00 prio 127 cost 844583 latency 0.001412 osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.103856 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.103862 7f9d683e1700 10 osd.2 173 dequeue_op 0x68a6800 prio 127 cost 1032999 latency 0.003560 osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3168 (79'167,173'3168] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3167 active]
2014-06-12 14:46:11.103884 7f9d683e1700  5 -- op tracker -- , seq: 747641, time: 2014-06-12 14:46:11.103880, event: reached_pg, op: osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.103892 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3168 (79'167,173'3168] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3167 active] handle_message: 0x68a6800
2014-06-12 14:46:11.103904 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3168 (79'167,173'3168] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3167 active] sub_op_modify trans a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 v 173'3169 (transaction) 174
2014-06-12 14:46:11.103920 7f9d683e1700  5 -- op tracker -- , seq: 747641, time: 2014-06-12 14:46:11.103916, event: started, op: osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.103937 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3168 (79'167,173'3168] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3167 active] append_log log((79'167,173'3168], crt=0'0) [173'3169 (173'3168) modify   a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 by client.4294.0:144016 2014-06-12 08:45:13.475058]
2014-06-12 14:46:11.103962 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3169 (79'167,173'3169] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3167 active] add_log_entry 173'3169 (173'3168) modify   a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 by client.4294.0:144016 2014-06-12 08:45:13.475058
2014-06-12 14:46:11.103983 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3169 (79'167,173'3169] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3167 active] append_log  adding 1 keys
2014-06-12 14:46:11.103995 7f9d683e1700 10 trim log((79'167,173'3169], crt=0'0) to 79'168
2014-06-12 14:46:11.104024 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3169, trimmed: 79'168
2014-06-12 14:46:11.104051 7f9d683e1700  5 -- op tracker -- , seq: 747641, time: 2014-06-12 14:46:11.104046, event: started, op: osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.104069 7f9d683e1700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:11.104073 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.104075 7f9d683e1700  5 queue_op 0x5cc4ae0 seq 2864 osr(3.36c 0x276d400) 1034520 bytes   (queue has 1 ops and 1034520 bytes)
2014-06-12 14:46:11.104082 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3169 (79'168,173'3169] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3167 active] do_sub_op osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.104097 7f9d683e1700 10 osd.2 173 dequeue_op 0x68a6800 finish
2014-06-12 14:46:11.104100 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.104106 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.104110 7f9d683e1700 10 osd.2 173 dequeue_op 0x28dd0d00 prio 127 cost 844583 latency 0.001714 osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2991 (0'0,173'2991] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2990 active]
2014-06-12 14:46:11.104129 7f9d683e1700  5 -- op tracker -- , seq: 747642, time: 2014-06-12 14:46:11.104125, event: reached_pg, op: osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.104136 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2991 (0'0,173'2991] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2990 active] handle_message: 0x28dd0d00
2014-06-12 14:46:11.104146 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2991 (0'0,173'2991] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify trans 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 v 173'2992 (transaction) 174
2014-06-12 14:46:11.104161 7f9d683e1700  5 -- op tracker -- , seq: 747642, time: 2014-06-12 14:46:11.104157, event: started, op: osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.104172 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2991 (0'0,173'2991] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2990 active] append_log log((0'0,173'2991], crt=0'0) [173'2992 (173'2991) modify   3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 by client.4298.0:145091 2014-06-12 08:43:28.635501]
2014-06-12 14:46:11.104194 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2992 (0'0,173'2992] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2990 active] add_log_entry 173'2992 (173'2991) modify   3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 by client.4298.0:145091 2014-06-12 08:43:28.635501
2014-06-12 14:46:11.104214 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2992 (0'0,173'2992] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2990 active] append_log  adding 1 keys
2014-06-12 14:46:11.104242 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2992, trimmed: 
2014-06-12 14:46:11.104264 7f9d683e1700  5 -- op tracker -- , seq: 747642, time: 2014-06-12 14:46:11.104258, event: started, op: osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.104272 7f9d683e1700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:11.104275 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.104276 7f9d683e1700  5 queue_op 0x7c23140 seq 2794 osr(3.458 0x276d960) 846069 bytes   (queue has 2 ops and 1880589 bytes)
2014-06-12 14:46:11.104286 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (1 active)
2014-06-12 14:46:11.104282 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2992 (0'0,173'2992] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2990 active] do_sub_op osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.104294 7f9d7815d700  5 _do_op 0x5cc4ae0 seq 2864 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:11.104296 7f9d683e1700 10 osd.2 173 dequeue_op 0x28dd0d00 finish
2014-06-12 14:46:11.104298 7f9d7815d700 10 _do_transaction on 0x65ccb70
2014-06-12 14:46:11.104300 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.104302 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.104305 7f9d7815d700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:11.104310 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.104448 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:11.104474 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.104530 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.104553 7f9d7815d700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:11.104561 7f9d7815d700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:11.104567 7f9d7815d700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:11.104585 7f9d7815d700 15 _write 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 2428928~1032192
2014-06-12 14:46:11.104946 7f9d7815d700 10 lookup_strip_header done  cid 3.36c_head oid a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3
2014-06-12 14:46:11.105093 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:11.105640 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (2 active)
2014-06-12 14:46:11.105651 7f9d82171700  5 _do_op 0x7c23140 seq 2794 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:11.105657 7f9d82171700 10 _do_transaction on 0x62ebcf0
2014-06-12 14:46:11.105668 7f9d82171700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:11.105806 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:11.105836 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.110017 7f9d7815d700 10 _generic_write 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 2428928~1032192 = 0
2014-06-12 14:46:11.110159 7f9d7815d700 15 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3
2014-06-12 14:46:11.110204 7f9d7815d700 10 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 = 0
2014-06-12 14:46:11.110229 7f9d7815d700 15 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3
2014-06-12 14:46:11.110242 7f9d7815d700 10 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 = 0
2014-06-12 14:46:11.110627 7f9d7815d700 20 set_header setting 17739 cid 3.36c_head oid a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 parent seq 0
2014-06-12 14:46:11.110745 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.110762 7f9d7815d700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:11.110774 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:11.122285 7f9d7815d700 10 _do_op 0x5cc4ae0 seq 2864 r = 0, finisher 0x733bdc0 0
2014-06-12 14:46:11.124502 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.124697 7f9d82171700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:11.124705 7f9d82171700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:11.124713 7f9d82171700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:11.124734 7f9d82171700 15 _write 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 3350528~843776
2014-06-12 14:46:11.124899 7f9d82171700 10 lookup_strip_header done  cid 3.458_head oid 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:46:11.125128 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:11.129795 7f9d82171700 10 _generic_write 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 3350528~843776 = 0
2014-06-12 14:46:11.129966 7f9d82171700 15 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:46:11.130009 7f9d82171700 10 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 = 0
2014-06-12 14:46:11.130016 7f9d82171700 15 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3
2014-06-12 14:46:11.130026 7f9d82171700 10 _setattrs 3.458_head/3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 = 0
2014-06-12 14:46:11.130425 7f9d82171700 20 set_header setting 16953 cid 3.458_head oid 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 parent seq 0
2014-06-12 14:46:11.130529 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.130545 7f9d82171700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:11.130556 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:11.137094 7f9d82171700 10 _do_op 0x7c23140 seq 2794 r = 0, finisher 0x32e67d00 0
2014-06-12 14:46:11.137656 7f9d7715b700  5 -- op tracker -- , seq: 747641, time: 2014-06-12 14:46:11.137645, event: commit_sent, op: osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.137670 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3169 (79'168,173'3169] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3167 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:11.137697 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:11.137721 7f9d7815d700 10 _finish_op 0x5cc4ae0 seq 2864 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:11.137729 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (1 active)
2014-06-12 14:46:11.137737 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.140175 7f9d7795c700  5 -- op tracker -- , seq: 747641, time: 2014-06-12 14:46:11.140162, event: sub_op_applied, op: osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.140186 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3169 (79'168,173'3169] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3168 active] sub_op_modify_applied on 0x65cca80 op osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.140205 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3169 (79'168,173'3169] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3168 active] op_applied on primary on version 173'3169
2014-06-12 14:46:11.140214 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3169 (79'168,173'3169] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3168 active] op_applied on replica on version 173'3169
2014-06-12 14:46:11.140253 7f9d7795c700  5 -- op tracker -- , seq: 747641, time: 2014-06-12 14:46:11.140249, event: done, op: osd_sub_op(client.4294.0:144016 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3169 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.141757 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:11.142624 7f9d7715b700  5 -- op tracker -- , seq: 747642, time: 2014-06-12 14:46:11.142614, event: commit_sent, op: osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.142635 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2992 (0'0,173'2992] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:11.142667 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:11.142754 7f9d82171700 10 _finish_op 0x7c23140 seq 2794 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:11.142803 7f9d7795c700  5 -- op tracker -- , seq: 747642, time: 2014-06-12 14:46:11.142795, event: sub_op_applied, op: osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.142815 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2992 (0'0,173'2992] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2991 active] sub_op_modify_applied on 0x62ebc00 op osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.142834 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2992 (0'0,173'2992] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2991 active] op_applied on primary on version 173'2992
2014-06-12 14:46:11.142846 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2992 (0'0,173'2992] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2991 active] op_applied on replica on version 173'2992
2014-06-12 14:46:11.142935 7f9d82171700  5 -- op tracker -- , seq: 747642, time: 2014-06-12 14:46:11.142923, event: done, op: osd_sub_op(client.4298.0:145091 3.458 3db4ac58/rbd_data.10a16b8b4567.0000000000000ffd/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.142967 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:11.142971 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.144215 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:11.153720 7f9d58c5a700  5 -- op tracker -- , seq: 747643, time: 2014-06-12 14:46:11.150242, event: header_read, op: osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.153734 7f9d58c5a700  5 -- op tracker -- , seq: 747643, time: 2014-06-12 14:46:11.150243, event: throttled, op: osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.153742 7f9d58c5a700  5 -- op tracker -- , seq: 747643, time: 2014-06-12 14:46:11.153652, event: all_read, op: osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.153751 7f9d58c5a700  5 -- op tracker -- , seq: 747643, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.153758 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:11.153765 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:11.153777 7f9d58c5a700 15 osd.2 173 enqueue_op 0x68a6f00 prio 127 cost 733991 latency 0.003535 osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.157651 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.157671 7f9d65bdc700 10 osd.2 173 dequeue_op 0x68a6f00 prio 127 cost 733991 latency 0.007428 osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.36c( v 173'3169 (79'168,173'3169] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3168 active]
2014-06-12 14:46:11.157703 7f9d65bdc700  5 -- op tracker -- , seq: 747643, time: 2014-06-12 14:46:11.157698, event: reached_pg, op: osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.157717 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3169 (79'168,173'3169] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3168 active] handle_message: 0x68a6f00
2014-06-12 14:46:11.157733 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3169 (79'168,173'3169] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3168 active] sub_op_modify trans a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 v 173'3170 (transaction) 174
2014-06-12 14:46:11.157750 7f9d65bdc700  5 -- op tracker -- , seq: 747643, time: 2014-06-12 14:46:11.157746, event: started, op: osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.157770 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3169 (79'168,173'3169] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3168 active] append_log log((79'168,173'3169], crt=0'0) [173'3170 (173'3169) modify   a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 by client.4294.0:144017 2014-06-12 08:45:13.525990]
2014-06-12 14:46:11.157797 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3170 (79'168,173'3170] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3168 active] add_log_entry 173'3170 (173'3169) modify   a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 by client.4294.0:144017 2014-06-12 08:45:13.525990
2014-06-12 14:46:11.157822 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3170 (79'168,173'3170] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3168 active] append_log  adding 1 keys
2014-06-12 14:46:11.157836 7f9d65bdc700 10 trim log((79'168,173'3170], crt=0'0) to 79'169
2014-06-12 14:46:11.157868 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3170, trimmed: 79'169
2014-06-12 14:46:11.157898 7f9d65bdc700  5 -- op tracker -- , seq: 747643, time: 2014-06-12 14:46:11.157893, event: started, op: osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.157909 7f9d65bdc700  5 queue_transactions existing osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:11.157915 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.157917 7f9d65bdc700  5 queue_op 0x51cc240 seq 2865 osr(3.36c 0x276d400) 735512 bytes   (queue has 1 ops and 735512 bytes)
2014-06-12 14:46:11.157925 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.36c( v 173'3170 (79'169,173'3170] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3168 active] do_sub_op osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.157943 7f9d65bdc700 10 osd.2 173 dequeue_op 0x68a6f00 finish
2014-06-12 14:46:11.157947 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.157950 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.157959 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a2c60 (1 active)
2014-06-12 14:46:11.157964 7f9d7d167700  5 _do_op 0x51cc240 seq 2865 osr(3.36c 0x276d400)/0x276d400 start
2014-06-12 14:46:11.157969 7f9d7d167700 10 _do_transaction on 0x29eb3b30
2014-06-12 14:46:11.157975 7f9d7d167700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:11.158130 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:11.158152 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.158211 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.158228 7f9d7d167700 15 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:11.158236 7f9d7d167700 10 _omap_rmkeys meta/1fe1e35b/pglog_3.36c/0//-1 r = 0
2014-06-12 14:46:11.158242 7f9d7d167700 15 _omap_setkeys meta/1fe1e35b/pglog_3.36c/0//-1
2014-06-12 14:46:11.158404 7f9d7d167700 15 _write 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 3461120~733184
2014-06-12 14:46:11.158568 7f9d7d167700 10 lookup_strip_header done  cid 3.36c_head oid a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3
2014-06-12 14:46:11.158747 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:11.166694 7f9d7d167700 10 _generic_write 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 3461120~733184 = 0
2014-06-12 14:46:11.166810 7f9d7d167700 15 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3
2014-06-12 14:46:11.166840 7f9d7d167700 10 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 = 0
2014-06-12 14:46:11.166847 7f9d7d167700 15 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3
2014-06-12 14:46:11.166857 7f9d7d167700 10 _setattrs 3.36c_head/a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 = 0
2014-06-12 14:46:11.167223 7f9d7d167700 20 set_header setting 17739 cid 3.36c_head oid a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 parent seq 0
2014-06-12 14:46:11.167369 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.167384 7f9d7d167700 20 set_header setting 2164 cid meta oid 1fe1e35b/pglog_3.36c/0//-1 parent seq 0
2014-06-12 14:46:11.167394 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:11.172797 7f9d7d167700 10 _do_op 0x51cc240 seq 2865 r = 0, finisher 0xc528f20 0
2014-06-12 14:46:11.172822 7f9d7d167700 10 _finish_op 0x51cc240 seq 2865 osr(3.36c 0x276d400)/0x276d400
2014-06-12 14:46:11.172831 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a2c60 (0 active)
2014-06-12 14:46:11.172840 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.172852 7f9d7715b700  5 -- op tracker -- , seq: 747643, time: 2014-06-12 14:46:11.172841, event: commit_sent, op: osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.172871 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3170 (79'169,173'3170] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3168 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:11.172908 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:11.172984 7f9d7795c700  5 -- op tracker -- , seq: 747643, time: 2014-06-12 14:46:11.172978, event: sub_op_applied, op: osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.172998 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3170 (79'169,173'3170] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3169 active] sub_op_modify_applied on 0x29eb3a40 op osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.173015 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3170 (79'169,173'3170] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3169 active] op_applied on primary on version 173'3170
2014-06-12 14:46:11.173025 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.36c( v 173'3170 (79'169,173'3170] local-les=156 n=334 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'3169 active] op_applied on replica on version 173'3170
2014-06-12 14:46:11.173050 7f9d7795c700  5 -- op tracker -- , seq: 747643, time: 2014-06-12 14:46:11.173046, event: done, op: osd_sub_op(client.4294.0:144017 3.36c a18ca36c/rbd_data.109d6b8b4567.0000000000001117/head//3 [] v 173'3170 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.199138 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:11.199169 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:11.271858 7f9d48d28700  5 -- op tracker -- , seq: 747644, time: 2014-06-12 14:46:11.271381, event: header_read, op: osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.271871 7f9d48d28700  5 -- op tracker -- , seq: 747644, time: 2014-06-12 14:46:11.271385, event: throttled, op: osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.271879 7f9d48d28700  5 -- op tracker -- , seq: 747644, time: 2014-06-12 14:46:11.271811, event: all_read, op: osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.271888 7f9d48d28700  5 -- op tracker -- , seq: 747644, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.271895 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:11.271921 7f9d48d28700 15 osd.2 173 enqueue_op 0x160e4600 prio 63 cost 151552 latency 0.000540 osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.271964 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.271971 7f9d6abe6700 10 osd.2 173 dequeue_op 0x160e4600 prio 63 cost 151552 latency 0.000590 osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean]
2014-06-12 14:46:11.271994 7f9d6abe6700  5 -- op tracker -- , seq: 747644, time: 2014-06-12 14:46:11.271990, event: reached_pg, op: osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.272002 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:11.272016 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] handle_message: 0x160e4600
2014-06-12 14:46:11.272025 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] do_op osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:11.272047 7f9d6abe6700 15 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 '_'
2014-06-12 14:46:11.284864 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:11.284867 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:11.284868 7f9d74bfa700 20 osd.2 173 _dispatch 0x1e700ea0 ping v1
2014-06-12 14:46:11.284871 7f9d74bfa700 10 osd.2 173 ping from client.4306
2014-06-12 14:46:11.284873 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:11.284874 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:11.301598 7f9d48d28700  5 -- op tracker -- , seq: 747645, time: 2014-06-12 14:46:11.300240, event: header_read, op: osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.301626 7f9d48d28700  5 -- op tracker -- , seq: 747645, time: 2014-06-12 14:46:11.300250, event: throttled, op: osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.301637 7f9d48d28700  5 -- op tracker -- , seq: 747645, time: 2014-06-12 14:46:11.301542, event: all_read, op: osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.301647 7f9d48d28700  5 -- op tracker -- , seq: 747645, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.301656 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:11.301685 7f9d48d28700 15 osd.2 173 enqueue_op 0x160e4e00 prio 63 cost 516096 latency 0.001445 osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.301739 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:11.311688 7f9d6abe6700 10 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 '_' = 0
2014-06-12 14:46:11.311722 7f9d6abe6700 15 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 'snapset'
2014-06-12 14:46:11.311907 7f9d6abe6700 10 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 'snapset' = 0
2014-06-12 14:46:11.311920 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] populate_obc_watchers 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.311939 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.311950 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] get_object_context: creating obc from disk: 0x30f13600
2014-06-12 14:46:11.311959 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] get_object_context: 0x30f13600 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 rwstate(none n=0 w=0) oi: 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'463 client.4289.0:39200 wrlock_by=unknown.0.0:0 dirty s 4194304 uv463) ssc: 0x82de380 snapset: 0=[]:[]+head
2014-06-12 14:46:11.311976 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] find_object_context 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 @head oi=32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'463 client.4289.0:39200 wrlock_by=unknown.0.0:0 dirty s 4194304 uv463)
2014-06-12 14:46:11.312007 7f9d6abe6700  5 -- op tracker -- , seq: 747644, time: 2014-06-12 14:46:11.311999, event: started, op: osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.312018 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] execute_ctx 0x1a917200
2014-06-12 14:46:11.312035 7f9d6abe6700  5 -- op tracker -- , seq: 747644, time: 2014-06-12 14:46:11.312030, event: started, op: osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.312042 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] do_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] ov 173'463 av 173'2350 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:11.312057 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] do_osd_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552]
2014-06-12 14:46:11.312068 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:11.312080 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] do_osd_op  write 0~151552
2014-06-12 14:46:11.312096 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:11.312108 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 snapset=0x82de3b8  snapc=0=[]
2014-06-12 14:46:11.312118 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:11.312128 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] finish_ctx 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 0x1a917200 op modify  
2014-06-12 14:46:11.312139 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean]  set mtime to 2014-06-12 08:45:13.652132
2014-06-12 14:46:11.312157 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean]  final snapset 0=[]:[]+head in 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.312177 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean]  zeroing write result code 0
2014-06-12 14:46:11.312188 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] new_repop rep_tid 236218 on osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.312204 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] issue_repop rep_tid 236218 o 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.312230 7f9d6abe6700  5 -- op tracker -- , seq: 747644, time: 2014-06-12 14:46:11.312225, event: waiting for subops from 23, op: osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.312253 7f9d6abe6700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:11.312280 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2349 (0'0,173'2349] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] append_log log((0'0,173'2349], crt=0'0) [173'2350 (173'463) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143622 2014-06-12 08:45:13.652132]
2014-06-12 14:46:11.312304 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] add_log_entry 173'2350 (173'463) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143622 2014-06-12 08:45:13.652132
2014-06-12 14:46:11.312324 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] append_log  adding 1 keys
2014-06-12 14:46:11.312358 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2350, trimmed: 
2014-06-12 14:46:11.312384 7f9d6abe6700  5 queue_transactions existing osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.312390 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.312392 7f9d6abe6700  5 queue_op 0x528cea0 seq 2261 osr(3.45f 0x276d940) 153845 bytes   (queue has 1 ops and 153845 bytes)
2014-06-12 14:46:11.312400 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] eval_repop repgather(0x1687cb40 173'2350 rep_tid=236218 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.312418 7f9d6abe6700 10 osd.2 173 dequeue_op 0x160e4600 finish
2014-06-12 14:46:11.312427 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:11.312434 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.312451 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x160e4e00 prio 63 cost 516096 latency 0.012211 osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean]
2014-06-12 14:46:11.312497 7f9d6a3e5700  5 -- op tracker -- , seq: 747645, time: 2014-06-12 14:46:11.312473, event: reached_pg, op: osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.312508 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:11.312524 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] handle_message: 0x160e4e00
2014-06-12 14:46:11.312534 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] do_op osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:11.312556 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] get_object_context: found obc in cache: 0x30f13600
2014-06-12 14:46:11.312567 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] get_object_context: 0x30f13600 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 rwstate(write n=1 w=0) oi: 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2350 client.4289.0:143622 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2350) ssc: 0x82de380 snapset: 0=[]:[]+head
2014-06-12 14:46:11.312594 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] find_object_context 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 @head oi=32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2350 client.4289.0:143622 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2350)
2014-06-12 14:46:11.312620 7f9d6a3e5700  5 -- op tracker -- , seq: 747645, time: 2014-06-12 14:46:11.312614, event: started, op: osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.312629 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] execute_ctx 0x55a5000
2014-06-12 14:46:11.312645 7f9d6a3e5700  5 -- op tracker -- , seq: 747645, time: 2014-06-12 14:46:11.312640, event: started, op: osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.312651 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] do_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] ov 173'2350 av 173'2351 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:11.312663 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] do_osd_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096]
2014-06-12 14:46:11.312696 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:11.312709 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] do_osd_op  write 151552~516096
2014-06-12 14:46:11.312721 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:11.312734 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 snapset=0x82de3b8  snapc=0=[]
2014-06-12 14:46:11.312745 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:11.312755 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] finish_ctx 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 0x55a5000 op modify  
2014-06-12 14:46:11.312766 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean]  set mtime to 2014-06-12 08:45:13.680291
2014-06-12 14:46:11.312792 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean]  final snapset 0=[]:[]+head in 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.312809 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean]  zeroing write result code 0
2014-06-12 14:46:11.312819 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] new_repop rep_tid 236219 on osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.312835 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] issue_repop rep_tid 236219 o 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.312855 7f9d6a3e5700  5 -- op tracker -- , seq: 747645, time: 2014-06-12 14:46:11.312850, event: waiting for subops from 23, op: osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.312884 7f9d6a3e5700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:11.312903 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2350 (0'0,173'2350] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] append_log log((0'0,173'2350], crt=0'0) [173'2351 (173'2350) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143623 2014-06-12 08:45:13.680291]
2014-06-12 14:46:11.312926 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] add_log_entry 173'2351 (173'2350) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143623 2014-06-12 08:45:13.680291
2014-06-12 14:46:11.312946 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] append_log  adding 1 keys
2014-06-12 14:46:11.312974 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2351, trimmed: 
2014-06-12 14:46:11.312992 7f9d6a3e5700  5 queue_transactions existing osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.312996 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.312998 7f9d6a3e5700  5 queue_op 0x69b1980 seq 2262 osr(3.45f 0x276d940) 518389 bytes   (queue has 2 ops and 672234 bytes)
2014-06-12 14:46:11.313003 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] eval_repop repgather(0x228e12c0 173'2351 rep_tid=236219 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.313019 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x160e4e00 finish
2014-06-12 14:46:11.313023 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.313025 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.313032 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1e60 (1 active)
2014-06-12 14:46:11.313037 7f9d8116f700  5 _do_op 0x528cea0 seq 2261 osr(3.45f 0x276d940)/0x276d940 start
2014-06-12 14:46:11.313041 7f9d8116f700 10 _do_transaction on 0x1687c540
2014-06-12 14:46:11.313047 7f9d8116f700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.313165 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.313189 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.313246 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.313264 7f9d8116f700 15 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.313272 7f9d8116f700 10 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1 r = 0
2014-06-12 14:46:11.313278 7f9d8116f700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.313292 7f9d8116f700 15 _write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 0~151552
2014-06-12 14:46:11.313450 7f9d8116f700 10 lookup_strip_header done  cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.313539 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:11.314152 7f9d8116f700 10 _generic_write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 0~151552 = 0
2014-06-12 14:46:11.314207 7f9d8116f700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.314221 7f9d8116f700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.314228 7f9d8116f700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.314237 7f9d8116f700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.314367 7f9d8116f700 20 set_header setting 17449 cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 parent seq 0
2014-06-12 14:46:11.314426 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.314442 7f9d8116f700 20 set_header setting 2074 cid meta oid 1fe1bdeb/pglog_3.45f/0//-1 parent seq 0
2014-06-12 14:46:11.314453 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:11.314758 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1e60 (2 active)
2014-06-12 14:46:11.315148 7f9d8116f700 10 _do_op 0x528cea0 seq 2261 r = 0, finisher 0x87fa340 0x216a1b80
2014-06-12 14:46:11.315161 7f9d8116f700 10 _finish_op 0x528cea0 seq 2261 osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.315168 7f9d79960700  5 _do_op 0x69b1980 seq 2262 osr(3.45f 0x276d940)/0x276d940 start
2014-06-12 14:46:11.315172 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1e60 (1 active)
2014-06-12 14:46:11.315176 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.315176 7f9d79960700 10 _do_transaction on 0x228e1bc0
2014-06-12 14:46:11.315184 7f9d79960700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.315187 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] op_commit: 236218
2014-06-12 14:46:11.315228 7f9d7715b700  5 -- op tracker -- , seq: 747644, time: 2014-06-12 14:46:11.315219, event: op_commit, op: osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.315247 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] op_applied: 236218
2014-06-12 14:46:11.315267 7f9d7795c700  5 -- op tracker -- , seq: 747644, time: 2014-06-12 14:46:11.315261, event: op_applied, op: osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.315288 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.315282 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] op_applied on primary on version 173'2350
2014-06-12 14:46:11.315313 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.315355 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.315376 7f9d79960700 15 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.315384 7f9d79960700 10 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1 r = 0
2014-06-12 14:46:11.315390 7f9d79960700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.315538 7f9d79960700 15 _write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 151552~516096
2014-06-12 14:46:11.315639 7f9d79960700 10 lookup_strip_header done  cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.315794 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:11.317295 7f9d79960700 10 _generic_write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 151552~516096 = 0
2014-06-12 14:46:11.317347 7f9d79960700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.317371 7f9d79960700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.317381 7f9d79960700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.317391 7f9d79960700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.317541 7f9d79960700 20 set_header setting 17449 cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 parent seq 0
2014-06-12 14:46:11.317601 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.317649 7f9d79960700 20 set_header setting 2074 cid meta oid 1fe1bdeb/pglog_3.45f/0//-1 parent seq 0
2014-06-12 14:46:11.317664 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:11.319938 7f9d79960700 10 _do_op 0x69b1980 seq 2262 r = 0, finisher 0x6d78780 0x465b3c0
2014-06-12 14:46:11.319958 7f9d79960700 10 _finish_op 0x69b1980 seq 2262 osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.319968 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1e60 (0 active)
2014-06-12 14:46:11.319976 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.319965 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] op_commit: 236219
2014-06-12 14:46:11.319997 7f9d7715b700  5 -- op tracker -- , seq: 747645, time: 2014-06-12 14:46:11.319988, event: op_commit, op: osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.320025 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] op_applied: 236219
2014-06-12 14:46:11.320045 7f9d7795c700  5 -- op tracker -- , seq: 747645, time: 2014-06-12 14:46:11.320040, event: op_applied, op: osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.320051 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] op_applied on primary on version 173'2351
2014-06-12 14:46:11.321050 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:11.321053 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:11.321055 7f9d74bfa700 20 osd.2 173 _dispatch 0x1fd70680 ping v1
2014-06-12 14:46:11.321057 7f9d74bfa700 10 osd.2 173 ping from client.4308
2014-06-12 14:46:11.321058 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:11.321059 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:11.328460 7f9d57543700  5 -- op tracker -- , seq: 747646, time: 2014-06-12 14:46:11.328389, event: header_read, op: osd_sub_op_reply(client.4289.0:143622 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.328489 7f9d57543700  5 -- op tracker -- , seq: 747646, time: 2014-06-12 14:46:11.328393, event: throttled, op: osd_sub_op_reply(client.4289.0:143622 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.328497 7f9d57543700  5 -- op tracker -- , seq: 747646, time: 2014-06-12 14:46:11.328412, event: all_read, op: osd_sub_op_reply(client.4289.0:143622 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.328515 7f9d57543700  5 -- op tracker -- , seq: 747646, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143622 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.328522 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143622 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.328529 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:11.328539 7f9d57543700 15 osd.2 173 enqueue_op 0x2afce00 prio 196 cost 0 latency 0.000150 osd_sub_op_reply(client.4289.0:143622 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.328572 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.328582 7f9d66bde700 10 osd.2 173 dequeue_op 0x2afce00 prio 196 cost 0 latency 0.000193 osd_sub_op_reply(client.4289.0:143622 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 pg pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean]
2014-06-12 14:46:11.328607 7f9d66bde700  5 -- op tracker -- , seq: 747646, time: 2014-06-12 14:46:11.328604, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143622 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.328619 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] handle_message: 0x2afce00
2014-06-12 14:46:11.328635 7f9d66bde700  5 -- op tracker -- , seq: 747646, time: 2014-06-12 14:46:11.328632, event: started, op: osd_sub_op_reply(client.4289.0:143622 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.328641 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] sub_op_modify_reply: tid 236218 op  ack_type 4 from 23
2014-06-12 14:46:11.328663 7f9d66bde700  5 -- op tracker -- , seq: 747644, time: 2014-06-12 14:46:11.328654, event: sub_op_commit_rec, op: osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.328671 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] repop_all_applied: repop tid 236218 all applied 
2014-06-12 14:46:11.328682 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] eval_repop repgather(0x1687cb40 173'2350 rep_tid=236218 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.328698 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean]  sending ack on repgather(0x1687cb40 173'2350 rep_tid=236218 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x22ef6780
2014-06-12 14:46:11.328733 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2349 crt=0'0 lcod 173'2348 mlcod 173'2348 active+clean] repop_all_committed: repop tid 236218 all committed 
2014-06-12 14:46:11.328744 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2348 active+clean] eval_repop repgather(0x1687cb40 173'2350 rep_tid=236218 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.328759 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2348 active+clean] log_op_stats osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 inb 151831 outb 0 rlat 0.057350 lat 0.057376
2014-06-12 14:46:11.328775 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2348 active+clean] publish_stats_to_osd 173:3078
2014-06-12 14:46:11.328790 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2348 active+clean]  sending commit on repgather(0x1687cb40 173'2350 rep_tid=236218 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x29de6500
2014-06-12 14:46:11.328820 7f9d66bde700  5 -- op tracker -- , seq: 747644, time: 2014-06-12 14:46:11.328814, event: commit_sent, op: osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.328827 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2348 active+clean]  requeue_ops 
2014-06-12 14:46:11.328838 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2349 active+clean]  removing repgather(0x1687cb40 173'2350 rep_tid=236218 committed?=1 applied?=1 op=osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.328850 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2349 active+clean]    q front is repgather(0x1687cb40 173'2350 rep_tid=236218 committed?=1 applied?=1 op=osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.328859 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2349 active+clean] remove_repop repgather(0x1687cb40 173'2350 rep_tid=236218 committed?=1 applied?=1 op=osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.328869 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2349 active+clean]  requeue_ops 
2014-06-12 14:46:11.328881 7f9d66bde700  5 -- op tracker -- , seq: 747644, time: 2014-06-12 14:46:11.328878, event: done, op: osd_op(client.4289.0:143622 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.328975 7f9d66bde700 10 osd.2 173 dequeue_op 0x2afce00 finish
2014-06-12 14:46:11.328980 7f9d66bde700  5 -- op tracker -- , seq: 747646, time: 2014-06-12 14:46:11.328978, event: done, op: osd_sub_op_reply(client.4289.0:143622 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.328987 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.328990 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.358508 7f9d57543700  5 -- op tracker -- , seq: 747647, time: 2014-06-12 14:46:11.358436, event: header_read, op: osd_sub_op_reply(client.4289.0:143623 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.358529 7f9d57543700  5 -- op tracker -- , seq: 747647, time: 2014-06-12 14:46:11.358439, event: throttled, op: osd_sub_op_reply(client.4289.0:143623 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.358560 7f9d57543700  5 -- op tracker -- , seq: 747647, time: 2014-06-12 14:46:11.358457, event: all_read, op: osd_sub_op_reply(client.4289.0:143623 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.358571 7f9d57543700  5 -- op tracker -- , seq: 747647, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143623 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.358578 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143623 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.358585 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:11.358596 7f9d57543700 15 osd.2 173 enqueue_op 0x876ec00 prio 196 cost 0 latency 0.000160 osd_sub_op_reply(client.4289.0:143623 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.358633 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.358647 7f9d653db700 10 osd.2 173 dequeue_op 0x876ec00 prio 196 cost 0 latency 0.000210 osd_sub_op_reply(client.4289.0:143623 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 pg pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2349 active+clean]
2014-06-12 14:46:11.358677 7f9d653db700  5 -- op tracker -- , seq: 747647, time: 2014-06-12 14:46:11.358672, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143623 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.358689 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2349 active+clean] handle_message: 0x876ec00
2014-06-12 14:46:11.358716 7f9d653db700  5 -- op tracker -- , seq: 747647, time: 2014-06-12 14:46:11.358705, event: started, op: osd_sub_op_reply(client.4289.0:143623 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.358720 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2349 active+clean] sub_op_modify_reply: tid 236219 op  ack_type 4 from 23
2014-06-12 14:46:11.358733 7f9d653db700  5 -- op tracker -- , seq: 747645, time: 2014-06-12 14:46:11.358728, event: sub_op_commit_rec, op: osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.358739 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2349 active+clean] repop_all_applied: repop tid 236219 all applied 
2014-06-12 14:46:11.358745 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2349 active+clean] eval_repop repgather(0x228e12c0 173'2351 rep_tid=236219 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.358755 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2349 active+clean]  sending ack on repgather(0x228e12c0 173'2351 rep_tid=236219 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x32df0f00
2014-06-12 14:46:11.358774 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2350 crt=0'0 lcod 173'2349 mlcod 173'2349 active+clean] repop_all_committed: repop tid 236219 all committed 
2014-06-12 14:46:11.358781 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2349 active+clean] eval_repop repgather(0x228e12c0 173'2351 rep_tid=236219 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.358796 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2349 active+clean] log_op_stats osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.058534 lat 0.058555
2014-06-12 14:46:11.358806 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2349 active+clean] publish_stats_to_osd 173:3079
2014-06-12 14:46:11.358813 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2349 active+clean]  sending commit on repgather(0x228e12c0 173'2351 rep_tid=236219 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x5125b80
2014-06-12 14:46:11.358832 7f9d653db700  5 -- op tracker -- , seq: 747645, time: 2014-06-12 14:46:11.358829, event: commit_sent, op: osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.358838 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2349 active+clean]  requeue_ops 
2014-06-12 14:46:11.358844 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean]  removing repgather(0x228e12c0 173'2351 rep_tid=236219 committed?=1 applied?=1 op=osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.358853 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean]    q front is repgather(0x228e12c0 173'2351 rep_tid=236219 committed?=1 applied?=1 op=osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.358861 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] remove_repop repgather(0x228e12c0 173'2351 rep_tid=236219 committed?=1 applied?=1 op=osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.358869 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean]  requeue_ops 
2014-06-12 14:46:11.358883 7f9d653db700  5 -- op tracker -- , seq: 747645, time: 2014-06-12 14:46:11.358880, event: done, op: osd_op(client.4289.0:143623 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 151552~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.358897 7f9d653db700 10 osd.2 173 dequeue_op 0x876ec00 finish
2014-06-12 14:46:11.358901 7f9d653db700  5 -- op tracker -- , seq: 747647, time: 2014-06-12 14:46:11.358899, event: done, op: osd_sub_op_reply(client.4289.0:143623 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.358919 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.358922 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.362521 7f9d48d28700  5 -- op tracker -- , seq: 747648, time: 2014-06-12 14:46:11.361261, event: header_read, op: osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.362540 7f9d48d28700  5 -- op tracker -- , seq: 747648, time: 2014-06-12 14:46:11.361264, event: throttled, op: osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.362567 7f9d48d28700  5 -- op tracker -- , seq: 747648, time: 2014-06-12 14:46:11.362461, event: all_read, op: osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.362578 7f9d48d28700  5 -- op tracker -- , seq: 747648, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.362588 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:11.362623 7f9d48d28700 15 osd.2 173 enqueue_op 0x160e4500 prio 63 cost 516096 latency 0.001361 osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.362674 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.362687 7f9d63bd8700 10 osd.2 173 dequeue_op 0x160e4500 prio 63 cost 516096 latency 0.001426 osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean]
2014-06-12 14:46:11.362722 7f9d63bd8700  5 -- op tracker -- , seq: 747648, time: 2014-06-12 14:46:11.362716, event: reached_pg, op: osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.362734 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:11.362758 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] handle_message: 0x160e4500
2014-06-12 14:46:11.362766 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] do_op osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:11.362785 7f9d63bd8700 15 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 '_'
2014-06-12 14:46:11.362936 7f9d63bd8700 10 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 '_' = 0
2014-06-12 14:46:11.362950 7f9d63bd8700 15 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 'snapset'
2014-06-12 14:46:11.363002 7f9d63bd8700 10 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 'snapset' = 0
2014-06-12 14:46:11.363010 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] populate_obc_watchers 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.363019 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.363025 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] get_object_context: creating obc from disk: 0x2f52b600
2014-06-12 14:46:11.363031 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] get_object_context: 0x2f52b600 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 rwstate(none n=0 w=0) oi: 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2351 client.4289.0:143623 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2351) ssc: 0x72b87e0 snapset: 0=[]:[]+head
2014-06-12 14:46:11.363041 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] find_object_context 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 @head oi=32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2351 client.4289.0:143623 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2351)
2014-06-12 14:46:11.363059 7f9d63bd8700  5 -- op tracker -- , seq: 747648, time: 2014-06-12 14:46:11.363055, event: started, op: osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.363065 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] execute_ctx 0xd21d000
2014-06-12 14:46:11.363076 7f9d63bd8700  5 -- op tracker -- , seq: 747648, time: 2014-06-12 14:46:11.363073, event: started, op: osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.363081 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] do_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] ov 173'2351 av 173'2352 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:11.363091 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] do_osd_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096]
2014-06-12 14:46:11.363098 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:11.363107 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] do_osd_op  write 667648~516096
2014-06-12 14:46:11.363118 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:11.363126 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 snapset=0x72b8818  snapc=0=[]
2014-06-12 14:46:11.363134 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:11.363140 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] finish_ctx 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 0xd21d000 op modify  
2014-06-12 14:46:11.363151 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean]  set mtime to 2014-06-12 08:45:13.741346
2014-06-12 14:46:11.363165 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean]  final snapset 0=[]:[]+head in 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.363183 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean]  zeroing write result code 0
2014-06-12 14:46:11.363191 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] new_repop rep_tid 236220 on osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.363202 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] issue_repop rep_tid 236220 o 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.363218 7f9d63bd8700  5 -- op tracker -- , seq: 747648, time: 2014-06-12 14:46:11.363215, event: waiting for subops from 23, op: osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.363238 7f9d63bd8700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:11.363255 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2351 (0'0,173'2351] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] append_log log((0'0,173'2351], crt=0'0) [173'2352 (173'2351) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143624 2014-06-12 08:45:13.741346]
2014-06-12 14:46:11.363271 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2351 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] add_log_entry 173'2352 (173'2351) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143624 2014-06-12 08:45:13.741346
2014-06-12 14:46:11.363285 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2351 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] append_log  adding 1 keys
2014-06-12 14:46:11.363307 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2352, trimmed: 
2014-06-12 14:46:11.363324 7f9d63bd8700  5 queue_transactions existing osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.363328 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.363330 7f9d63bd8700  5 queue_op 0x6cf7c80 seq 2263 osr(3.45f 0x276d940) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:11.363339 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1e60 (1 active)
2014-06-12 14:46:11.363336 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2351 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] eval_repop repgather(0x10451200 173'2352 rep_tid=236220 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.363346 7f9d8096e700  5 _do_op 0x6cf7c80 seq 2263 osr(3.45f 0x276d940)/0x276d940 start
2014-06-12 14:46:11.363348 7f9d63bd8700 10 osd.2 173 dequeue_op 0x160e4500 finish
2014-06-12 14:46:11.363350 7f9d8096e700 10 _do_transaction on 0x32b68f00
2014-06-12 14:46:11.363350 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.363352 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.363356 7f9d8096e700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.363483 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.363523 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.363564 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.363697 7f9d8096e700 15 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.363713 7f9d8096e700 10 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1 r = 0
2014-06-12 14:46:11.363726 7f9d8096e700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.363763 7f9d8096e700 15 _write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 667648~516096
2014-06-12 14:46:11.364048 7f9d8096e700 10 lookup_strip_header done  cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.364259 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:11.366814 7f9d8096e700 10 _generic_write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 667648~516096 = 0
2014-06-12 14:46:11.366903 7f9d8096e700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.366936 7f9d8096e700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.366952 7f9d8096e700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.366971 7f9d8096e700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.367278 7f9d8096e700 20 set_header setting 17449 cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 parent seq 0
2014-06-12 14:46:11.367401 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.367435 7f9d8096e700 20 set_header setting 2074 cid meta oid 1fe1bdeb/pglog_3.45f/0//-1 parent seq 0
2014-06-12 14:46:11.367460 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:11.371625 7f9d57543700  5 -- op tracker -- , seq: 747649, time: 2014-06-12 14:46:11.371575, event: header_read, op: osd_sub_op_reply(client.4289.0:143624 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.371637 7f9d57543700  5 -- op tracker -- , seq: 747649, time: 2014-06-12 14:46:11.371577, event: throttled, op: osd_sub_op_reply(client.4289.0:143624 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.371645 7f9d57543700  5 -- op tracker -- , seq: 747649, time: 2014-06-12 14:46:11.371589, event: all_read, op: osd_sub_op_reply(client.4289.0:143624 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.371652 7f9d57543700  5 -- op tracker -- , seq: 747649, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143624 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.371657 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143624 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.371663 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:11.371670 7f9d57543700 15 osd.2 173 enqueue_op 0x17c2a300 prio 196 cost 0 latency 0.000095 osd_sub_op_reply(client.4289.0:143624 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.371697 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.371705 7f9d69be4700 10 osd.2 173 dequeue_op 0x17c2a300 prio 196 cost 0 latency 0.000130 osd_sub_op_reply(client.4289.0:143624 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 pg pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2351 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean]
2014-06-12 14:46:11.371726 7f9d69be4700  5 -- op tracker -- , seq: 747649, time: 2014-06-12 14:46:11.371722, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143624 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.371748 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2351 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] handle_message: 0x17c2a300
2014-06-12 14:46:11.371763 7f9d69be4700  5 -- op tracker -- , seq: 747649, time: 2014-06-12 14:46:11.371760, event: started, op: osd_sub_op_reply(client.4289.0:143624 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.371769 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2351 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] sub_op_modify_reply: tid 236220 op  ack_type 4 from 23
2014-06-12 14:46:11.371789 7f9d69be4700  5 -- op tracker -- , seq: 747648, time: 2014-06-12 14:46:11.371781, event: sub_op_commit_rec, op: osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.371801 7f9d69be4700 10 osd.2 173 dequeue_op 0x17c2a300 finish
2014-06-12 14:46:11.371807 7f9d69be4700  5 -- op tracker -- , seq: 747649, time: 2014-06-12 14:46:11.371804, event: done, op: osd_sub_op_reply(client.4289.0:143624 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.371822 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.371825 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.381726 7f9d8096e700 10 _do_op 0x6cf7c80 seq 2263 r = 0, finisher 0x5af6f20 0x3b728c0
2014-06-12 14:46:11.381743 7f9d8096e700 10 _finish_op 0x6cf7c80 seq 2263 osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.381750 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1e60 (0 active)
2014-06-12 14:46:11.381760 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.381748 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2351 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] op_commit: 236220
2014-06-12 14:46:11.381782 7f9d7715b700  5 -- op tracker -- , seq: 747648, time: 2014-06-12 14:46:11.381772, event: op_commit, op: osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.381804 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2351 crt=0'0 lcod 173'2350 mlcod 173'2350 active+clean] repop_all_committed: repop tid 236220 all committed 
2014-06-12 14:46:11.381817 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2350 active+clean] eval_repop repgather(0x10451200 173'2352 rep_tid=236220 committed?=1 applied?=0 lock=0 op=osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.381835 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2350 active+clean] log_op_stats osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.020572
2014-06-12 14:46:11.381851 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2350 active+clean] publish_stats_to_osd 173:3080
2014-06-12 14:46:11.381861 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2350 active+clean]  sending commit on repgather(0x10451200 173'2352 rep_tid=236220 committed?=1 applied?=0 lock=0 op=osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x16cf3680
2014-06-12 14:46:11.381896 7f9d7715b700  5 -- op tracker -- , seq: 747648, time: 2014-06-12 14:46:11.381891, event: commit_sent, op: osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.381911 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2350 active+clean] op_applied: 236220
2014-06-12 14:46:11.381928 7f9d7795c700  5 -- op tracker -- , seq: 747648, time: 2014-06-12 14:46:11.381923, event: op_applied, op: osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.381935 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2350 active+clean] op_applied on primary on version 173'2352
2014-06-12 14:46:11.381945 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2350 active+clean] repop_all_applied: repop tid 236220 all applied 
2014-06-12 14:46:11.381953 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2350 active+clean] eval_repop repgather(0x10451200 173'2352 rep_tid=236220 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.381966 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2350 active+clean] log_op_stats osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.020705
2014-06-12 14:46:11.381980 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2350 active+clean] publish_stats_to_osd 173:3081
2014-06-12 14:46:11.381992 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2350 active+clean]  requeue_ops 
2014-06-12 14:46:11.382004 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean]  removing repgather(0x10451200 173'2352 rep_tid=236220 committed?=1 applied?=1 op=osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.382017 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean]    q front is repgather(0x10451200 173'2352 rep_tid=236220 committed?=1 applied?=1 op=osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.382031 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] remove_repop repgather(0x10451200 173'2352 rep_tid=236220 committed?=1 applied?=1 op=osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.382048 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean]  requeue_ops 
2014-06-12 14:46:11.382074 7f9d7795c700  5 -- op tracker -- , seq: 747648, time: 2014-06-12 14:46:11.382069, event: done, op: osd_op(client.4289.0:143624 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.385391 7f9d48d28700  5 -- op tracker -- , seq: 747650, time: 2014-06-12 14:46:11.384348, event: header_read, op: osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.385408 7f9d48d28700  5 -- op tracker -- , seq: 747650, time: 2014-06-12 14:46:11.384360, event: throttled, op: osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.385419 7f9d48d28700  5 -- op tracker -- , seq: 747650, time: 2014-06-12 14:46:11.385339, event: all_read, op: osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.385429 7f9d48d28700  5 -- op tracker -- , seq: 747650, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.385442 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:11.385474 7f9d48d28700 15 osd.2 173 enqueue_op 0x160e5d00 prio 63 cost 516096 latency 0.001125 osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.385511 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.385523 7f9d68be2700 10 osd.2 173 dequeue_op 0x160e5d00 prio 63 cost 516096 latency 0.001175 osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean]
2014-06-12 14:46:11.385552 7f9d68be2700  5 -- op tracker -- , seq: 747650, time: 2014-06-12 14:46:11.385547, event: reached_pg, op: osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.385563 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:11.385580 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] handle_message: 0x160e5d00
2014-06-12 14:46:11.385591 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] do_op osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:11.385624 7f9d68be2700 15 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 '_'
2014-06-12 14:46:11.385827 7f9d68be2700 10 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 '_' = 0
2014-06-12 14:46:11.385856 7f9d68be2700 15 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 'snapset'
2014-06-12 14:46:11.385943 7f9d68be2700 10 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 'snapset' = 0
2014-06-12 14:46:11.385954 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] populate_obc_watchers 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.385967 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.385978 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] get_object_context: creating obc from disk: 0x172ce2c0
2014-06-12 14:46:11.385989 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] get_object_context: 0x172ce2c0 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 rwstate(none n=0 w=0) oi: 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2352 client.4289.0:143624 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2352) ssc: 0x5bb8d20 snapset: 0=[]:[]+head
2014-06-12 14:46:11.386007 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] find_object_context 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 @head oi=32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2352 client.4289.0:143624 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2352)
2014-06-12 14:46:11.386035 7f9d68be2700  5 -- op tracker -- , seq: 747650, time: 2014-06-12 14:46:11.386029, event: started, op: osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.386043 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] execute_ctx 0x5e4be00
2014-06-12 14:46:11.386060 7f9d68be2700  5 -- op tracker -- , seq: 747650, time: 2014-06-12 14:46:11.386055, event: started, op: osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.386067 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] do_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] ov 173'2352 av 173'2353 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:11.386080 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] do_osd_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096]
2014-06-12 14:46:11.386090 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:11.386104 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] do_osd_op  write 1183744~516096
2014-06-12 14:46:11.386119 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:11.386131 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 snapset=0x5bb8d58  snapc=0=[]
2014-06-12 14:46:11.386144 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:11.386153 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] finish_ctx 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 0x5e4be00 op modify  
2014-06-12 14:46:11.386165 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean]  set mtime to 2014-06-12 08:45:13.764408
2014-06-12 14:46:11.386184 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean]  final snapset 0=[]:[]+head in 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.386205 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean]  zeroing write result code 0
2014-06-12 14:46:11.386216 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] new_repop rep_tid 236221 on osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.386233 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] issue_repop rep_tid 236221 o 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.386258 7f9d68be2700  5 -- op tracker -- , seq: 747650, time: 2014-06-12 14:46:11.386254, event: waiting for subops from 23, op: osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.386283 7f9d68be2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:11.386309 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2352 (0'0,173'2352] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] append_log log((0'0,173'2352], crt=0'0) [173'2353 (173'2352) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143625 2014-06-12 08:45:13.764408]
2014-06-12 14:46:11.386336 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2352 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] add_log_entry 173'2353 (173'2352) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143625 2014-06-12 08:45:13.764408
2014-06-12 14:46:11.386358 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2352 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] append_log  adding 1 keys
2014-06-12 14:46:11.386392 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2353, trimmed: 
2014-06-12 14:46:11.386416 7f9d68be2700  5 queue_transactions existing osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.386422 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.386423 7f9d68be2700  5 queue_op 0x50a0900 seq 2264 osr(3.45f 0x276d940) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:11.386431 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2352 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] eval_repop repgather(0x32be2b40 173'2353 rep_tid=236221 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.386449 7f9d68be2700 10 osd.2 173 dequeue_op 0x160e5d00 finish
2014-06-12 14:46:11.386453 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.386457 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.386903 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1e60 (1 active)
2014-06-12 14:46:11.386917 7f9d8016d700  5 _do_op 0x50a0900 seq 2264 osr(3.45f 0x276d940)/0x276d940 start
2014-06-12 14:46:11.386922 7f9d8016d700 10 _do_transaction on 0x11c7ec00
2014-06-12 14:46:11.386929 7f9d8016d700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.387117 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.387146 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.387197 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.387221 7f9d8016d700 15 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.387230 7f9d8016d700 10 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1 r = 0
2014-06-12 14:46:11.387237 7f9d8016d700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.387346 7f9d8016d700 15 _write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 1183744~516096
2014-06-12 14:46:11.387498 7f9d8016d700 10 lookup_strip_header done  cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.387639 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:11.389648 7f9d8016d700 10 _generic_write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 1183744~516096 = 0
2014-06-12 14:46:11.389706 7f9d8016d700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.389724 7f9d8016d700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.389734 7f9d8016d700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.389743 7f9d8016d700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.389958 7f9d8016d700 20 set_header setting 17449 cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 parent seq 0
2014-06-12 14:46:11.390043 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.390060 7f9d8016d700 20 set_header setting 2074 cid meta oid 1fe1bdeb/pglog_3.45f/0//-1 parent seq 0
2014-06-12 14:46:11.390074 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:11.393909 7f9d8016d700 10 _do_op 0x50a0900 seq 2264 r = 0, finisher 0x59e6400 0x217ef500
2014-06-12 14:46:11.393964 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2352 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] op_commit: 236221
2014-06-12 14:46:11.393995 7f9d7715b700  5 -- op tracker -- , seq: 747650, time: 2014-06-12 14:46:11.393986, event: op_commit, op: osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.394023 7f9d8016d700 10 _finish_op 0x50a0900 seq 2264 osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.394033 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1e60 (0 active)
2014-06-12 14:46:11.394042 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.394169 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2352 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] op_applied: 236221
2014-06-12 14:46:11.394196 7f9d7795c700  5 -- op tracker -- , seq: 747650, time: 2014-06-12 14:46:11.394189, event: op_applied, op: osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.394208 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2352 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] op_applied on primary on version 173'2353
2014-06-12 14:46:11.396536 7f9d57543700  5 -- op tracker -- , seq: 747651, time: 2014-06-12 14:46:11.396487, event: header_read, op: osd_sub_op_reply(client.4289.0:143625 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.396550 7f9d57543700  5 -- op tracker -- , seq: 747651, time: 2014-06-12 14:46:11.396489, event: throttled, op: osd_sub_op_reply(client.4289.0:143625 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.396558 7f9d57543700  5 -- op tracker -- , seq: 747651, time: 2014-06-12 14:46:11.396502, event: all_read, op: osd_sub_op_reply(client.4289.0:143625 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.396566 7f9d57543700  5 -- op tracker -- , seq: 747651, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143625 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.396572 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143625 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.396580 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:11.396588 7f9d57543700 15 osd.2 173 enqueue_op 0x31d61300 prio 196 cost 0 latency 0.000101 osd_sub_op_reply(client.4289.0:143625 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.396625 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.396634 7f9d633d7700 10 osd.2 173 dequeue_op 0x31d61300 prio 196 cost 0 latency 0.000146 osd_sub_op_reply(client.4289.0:143625 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 pg pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2352 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean]
2014-06-12 14:46:11.396658 7f9d633d7700  5 -- op tracker -- , seq: 747651, time: 2014-06-12 14:46:11.396654, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143625 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.396666 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2352 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] handle_message: 0x31d61300
2014-06-12 14:46:11.396693 7f9d633d7700  5 -- op tracker -- , seq: 747651, time: 2014-06-12 14:46:11.396689, event: started, op: osd_sub_op_reply(client.4289.0:143625 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.396700 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2352 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] sub_op_modify_reply: tid 236221 op  ack_type 4 from 23
2014-06-12 14:46:11.396726 7f9d633d7700  5 -- op tracker -- , seq: 747650, time: 2014-06-12 14:46:11.396716, event: sub_op_commit_rec, op: osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.396736 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2352 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] repop_all_applied: repop tid 236221 all applied 
2014-06-12 14:46:11.396750 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2352 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] eval_repop repgather(0x32be2b40 173'2353 rep_tid=236221 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.396774 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2352 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean]  sending ack on repgather(0x32be2b40 173'2353 rep_tid=236221 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x27474500
2014-06-12 14:46:11.396809 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2352 crt=0'0 lcod 173'2351 mlcod 173'2351 active+clean] repop_all_committed: repop tid 236221 all committed 
2014-06-12 14:46:11.396821 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2351 active+clean] eval_repop repgather(0x32be2b40 173'2353 rep_tid=236221 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.396837 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2351 active+clean] log_op_stats osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.012459 lat 0.012488
2014-06-12 14:46:11.396855 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2351 active+clean] publish_stats_to_osd 173:3082
2014-06-12 14:46:11.396870 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2351 active+clean]  sending commit on repgather(0x32be2b40 173'2353 rep_tid=236221 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x5ab0500
2014-06-12 14:46:11.396907 7f9d633d7700  5 -- op tracker -- , seq: 747650, time: 2014-06-12 14:46:11.396901, event: commit_sent, op: osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.396917 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2351 active+clean]  requeue_ops 
2014-06-12 14:46:11.396931 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean]  removing repgather(0x32be2b40 173'2353 rep_tid=236221 committed?=1 applied?=1 op=osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.396947 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean]    q front is repgather(0x32be2b40 173'2353 rep_tid=236221 committed?=1 applied?=1 op=osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.396962 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] remove_repop repgather(0x32be2b40 173'2353 rep_tid=236221 committed?=1 applied?=1 op=osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.396982 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean]  requeue_ops 
2014-06-12 14:46:11.397010 7f9d633d7700  5 -- op tracker -- , seq: 747650, time: 2014-06-12 14:46:11.397006, event: done, op: osd_op(client.4289.0:143625 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1183744~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.397032 7f9d633d7700 10 osd.2 173 dequeue_op 0x31d61300 finish
2014-06-12 14:46:11.397041 7f9d633d7700  5 -- op tracker -- , seq: 747651, time: 2014-06-12 14:46:11.397037, event: done, op: osd_sub_op_reply(client.4289.0:143625 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.397051 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.397054 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.406188 7f9d48d28700  5 -- op tracker -- , seq: 747652, time: 2014-06-12 14:46:11.399315, event: header_read, op: osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.406209 7f9d48d28700  5 -- op tracker -- , seq: 747652, time: 2014-06-12 14:46:11.399318, event: throttled, op: osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.406220 7f9d48d28700  5 -- op tracker -- , seq: 747652, time: 2014-06-12 14:46:11.406136, event: all_read, op: osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.406229 7f9d48d28700  5 -- op tracker -- , seq: 747652, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.406236 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:11.406258 7f9d48d28700 15 osd.2 173 enqueue_op 0x23e08b00 prio 63 cost 516096 latency 0.006943 osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.406287 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.406299 7f9d623d5700 10 osd.2 173 dequeue_op 0x23e08b00 prio 63 cost 516096 latency 0.006984 osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean]
2014-06-12 14:46:11.406330 7f9d623d5700  5 -- op tracker -- , seq: 747652, time: 2014-06-12 14:46:11.406324, event: reached_pg, op: osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.406341 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:11.406359 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] handle_message: 0x23e08b00
2014-06-12 14:46:11.406372 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] do_op osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:11.406397 7f9d623d5700 15 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 '_'
2014-06-12 14:46:11.406590 7f9d623d5700 10 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 '_' = 0
2014-06-12 14:46:11.406606 7f9d623d5700 15 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 'snapset'
2014-06-12 14:46:11.406693 7f9d623d5700 10 getattr 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 'snapset' = 0
2014-06-12 14:46:11.406701 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] populate_obc_watchers 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.406708 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.406714 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] get_object_context: creating obc from disk: 0x6ac0000
2014-06-12 14:46:11.406719 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] get_object_context: 0x6ac0000 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 rwstate(none n=0 w=0) oi: 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2353 client.4289.0:143625 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2353) ssc: 0x2ef8ed20 snapset: 0=[]:[]+head
2014-06-12 14:46:11.406728 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] find_object_context 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 @head oi=32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2353 client.4289.0:143625 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2353)
2014-06-12 14:46:11.406747 7f9d623d5700  5 -- op tracker -- , seq: 747652, time: 2014-06-12 14:46:11.406741, event: started, op: osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.406755 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] execute_ctx 0x27bc1200
2014-06-12 14:46:11.406771 7f9d623d5700  5 -- op tracker -- , seq: 747652, time: 2014-06-12 14:46:11.406767, event: started, op: osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.406775 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] do_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] ov 173'2353 av 173'2354 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:11.406784 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] do_osd_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096]
2014-06-12 14:46:11.406790 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:11.406798 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] do_osd_op  write 1699840~516096
2014-06-12 14:46:11.406806 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:11.406815 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 snapset=0x2ef8ed58  snapc=0=[]
2014-06-12 14:46:11.406827 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:11.406833 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] finish_ctx 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 0x27bc1200 op modify  
2014-06-12 14:46:11.406841 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean]  set mtime to 2014-06-12 08:45:13.779406
2014-06-12 14:46:11.406852 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean]  final snapset 0=[]:[]+head in 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.406863 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean]  zeroing write result code 0
2014-06-12 14:46:11.406869 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] new_repop rep_tid 236222 on osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.406879 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] issue_repop rep_tid 236222 o 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.406895 7f9d623d5700  5 -- op tracker -- , seq: 747652, time: 2014-06-12 14:46:11.406892, event: waiting for subops from 23, op: osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.406914 7f9d623d5700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:11.406929 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2353 (0'0,173'2353] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] append_log log((0'0,173'2353], crt=0'0) [173'2354 (173'2353) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143626 2014-06-12 08:45:13.779406]
2014-06-12 14:46:11.406948 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] add_log_entry 173'2354 (173'2353) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143626 2014-06-12 08:45:13.779406
2014-06-12 14:46:11.406964 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] append_log  adding 1 keys
2014-06-12 14:46:11.406995 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2354, trimmed: 
2014-06-12 14:46:11.407016 7f9d623d5700  5 queue_transactions existing osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.407020 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.407021 7f9d623d5700  5 queue_op 0x11eb2180 seq 2265 osr(3.45f 0x276d940) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:11.407030 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1e60 (1 active)
2014-06-12 14:46:11.407027 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] eval_repop repgather(0x15e57080 173'2354 rep_tid=236222 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.407045 7f9d623d5700 10 osd.2 173 dequeue_op 0x23e08b00 finish
2014-06-12 14:46:11.407049 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.407051 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.407061 7f9d7915f700  5 _do_op 0x11eb2180 seq 2265 osr(3.45f 0x276d940)/0x276d940 start
2014-06-12 14:46:11.407073 7f9d7915f700 10 _do_transaction on 0x15e56000
2014-06-12 14:46:11.407079 7f9d7915f700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.407235 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.407286 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.407348 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.407378 7f9d7915f700 15 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.407383 7f9d7915f700 10 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1 r = 0
2014-06-12 14:46:11.407389 7f9d7915f700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.407407 7f9d7915f700 15 _write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 1699840~516096
2014-06-12 14:46:11.407515 7f9d7915f700 10 lookup_strip_header done  cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.407615 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:11.409111 7f9d7915f700 10 _generic_write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 1699840~516096 = 0
2014-06-12 14:46:11.409164 7f9d7915f700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.409197 7f9d7915f700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.409203 7f9d7915f700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.409210 7f9d7915f700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.409443 7f9d7915f700 20 set_header setting 17449 cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 parent seq 0
2014-06-12 14:46:11.409506 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.409520 7f9d7915f700 20 set_header setting 2074 cid meta oid 1fe1bdeb/pglog_3.45f/0//-1 parent seq 0
2014-06-12 14:46:11.409529 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:11.411845 7f9d48d28700  5 -- op tracker -- , seq: 747653, time: 2014-06-12 14:46:11.410730, event: header_read, op: osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.411857 7f9d48d28700  5 -- op tracker -- , seq: 747653, time: 2014-06-12 14:46:11.410734, event: throttled, op: osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.411865 7f9d48d28700  5 -- op tracker -- , seq: 747653, time: 2014-06-12 14:46:11.411797, event: all_read, op: osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.411873 7f9d48d28700  5 -- op tracker -- , seq: 747653, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.411882 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:11.411910 7f9d48d28700 15 osd.2 173 enqueue_op 0x2fa9d600 prio 63 cost 516096 latency 0.001180 osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.411959 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.411966 7f9d643d9700 10 osd.2 173 dequeue_op 0x2fa9d600 prio 63 cost 516096 latency 0.001236 osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean]
2014-06-12 14:46:11.411989 7f9d643d9700  5 -- op tracker -- , seq: 747653, time: 2014-06-12 14:46:11.411984, event: reached_pg, op: osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.411998 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:11.412013 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] handle_message: 0x2fa9d600
2014-06-12 14:46:11.412023 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] do_op osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:11.412047 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] get_object_context: found obc in cache: 0x6ac0000
2014-06-12 14:46:11.412057 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] get_object_context: 0x6ac0000 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 rwstate(write n=1 w=0) oi: 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2354 client.4289.0:143626 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2354) ssc: 0x2ef8ed20 snapset: 0=[]:[]+head
2014-06-12 14:46:11.412072 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] find_object_context 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 @head oi=32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2354 client.4289.0:143626 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2354)
2014-06-12 14:46:11.412095 7f9d643d9700  5 -- op tracker -- , seq: 747653, time: 2014-06-12 14:46:11.412091, event: started, op: osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.412102 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] execute_ctx 0x5e4a000
2014-06-12 14:46:11.412117 7f9d643d9700  5 -- op tracker -- , seq: 747653, time: 2014-06-12 14:46:11.412113, event: started, op: osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.412123 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] do_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] ov 173'2354 av 173'2355 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:11.412135 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] do_osd_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096]
2014-06-12 14:46:11.412144 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:11.412156 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] do_osd_op  write 2215936~516096
2014-06-12 14:46:11.412170 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:11.412180 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 snapset=0x2ef8ed58  snapc=0=[]
2014-06-12 14:46:11.412191 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:11.412199 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] finish_ctx 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 0x5e4a000 op modify  
2014-06-12 14:46:11.412210 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean]  set mtime to 2014-06-12 08:45:13.790805
2014-06-12 14:46:11.412229 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean]  final snapset 0=[]:[]+head in 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.412238 7f9d7915f700 10 _do_op 0x11eb2180 seq 2265 r = 0, finisher 0x6ce5260 0x3b72840
2014-06-12 14:46:11.412246 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean]  zeroing write result code 0
2014-06-12 14:46:11.412261 7f9d7915f700 10 _finish_op 0x11eb2180 seq 2265 osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.412257 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] new_repop rep_tid 236223 on osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.412271 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1e60 (0 active)
2014-06-12 14:46:11.412280 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.412272 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] issue_repop rep_tid 236223 o 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.412295 7f9d643d9700  5 -- op tracker -- , seq: 747653, time: 2014-06-12 14:46:11.412291, event: waiting for subops from 23, op: osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.412313 7f9d643d9700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:11.412330 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2354 (0'0,173'2354] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] append_log log((0'0,173'2354], crt=0'0) [173'2355 (173'2354) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143627 2014-06-12 08:45:13.790805]
2014-06-12 14:46:11.412353 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] add_log_entry 173'2355 (173'2354) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143627 2014-06-12 08:45:13.790805
2014-06-12 14:46:11.412371 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] append_log  adding 1 keys
2014-06-12 14:46:11.412399 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2355, trimmed: 
2014-06-12 14:46:11.412421 7f9d643d9700  5 queue_transactions existing osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.412426 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.412427 7f9d643d9700  5 queue_op 0x92a54a0 seq 2266 osr(3.45f 0x276d940) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:11.412440 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1e60 (1 active)
2014-06-12 14:46:11.412448 7f9d7d968700  5 _do_op 0x92a54a0 seq 2266 osr(3.45f 0x276d940)/0x276d940 start
2014-06-12 14:46:11.412452 7f9d7d968700 10 _do_transaction on 0x5237b00
2014-06-12 14:46:11.412435 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] eval_repop repgather(0x18225ec0 173'2355 rep_tid=236223 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.412457 7f9d643d9700 10 osd.2 173 dequeue_op 0x2fa9d600 finish
2014-06-12 14:46:11.412459 7f9d7d968700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.412463 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.412465 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.412572 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.412596 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.412643 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.412661 7f9d7d968700 15 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.412667 7f9d7d968700 10 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1 r = 0
2014-06-12 14:46:11.412672 7f9d7d968700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.412773 7f9d7d968700 15 _write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 2215936~516096
2014-06-12 14:46:11.412784 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] op_commit: 236222
2014-06-12 14:46:11.412812 7f9d7715b700  5 -- op tracker -- , seq: 747652, time: 2014-06-12 14:46:11.412805, event: op_commit, op: osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.412827 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] op_applied: 236222
2014-06-12 14:46:11.412852 7f9d7795c700  5 -- op tracker -- , seq: 747652, time: 2014-06-12 14:46:11.412845, event: op_applied, op: osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.412866 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] op_applied on primary on version 173'2354
2014-06-12 14:46:11.412907 7f9d7d968700 10 lookup_strip_header done  cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.413038 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:11.415011 7f9d7d968700 10 _generic_write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 2215936~516096 = 0
2014-06-12 14:46:11.415068 7f9d7d968700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.415082 7f9d7d968700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.415088 7f9d7d968700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.415096 7f9d7d968700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.415310 7f9d7d968700 20 set_header setting 17449 cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 parent seq 0
2014-06-12 14:46:11.415378 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.415396 7f9d7d968700 20 set_header setting 2074 cid meta oid 1fe1bdeb/pglog_3.45f/0//-1 parent seq 0
2014-06-12 14:46:11.415411 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:11.419830 7f9d57543700  5 -- op tracker -- , seq: 747654, time: 2014-06-12 14:46:11.419782, event: header_read, op: osd_sub_op_reply(client.4289.0:143626 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.419842 7f9d57543700  5 -- op tracker -- , seq: 747654, time: 2014-06-12 14:46:11.419783, event: throttled, op: osd_sub_op_reply(client.4289.0:143626 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.419850 7f9d57543700  5 -- op tracker -- , seq: 747654, time: 2014-06-12 14:46:11.419794, event: all_read, op: osd_sub_op_reply(client.4289.0:143626 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.419858 7f9d57543700  5 -- op tracker -- , seq: 747654, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143626 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.419864 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143626 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.419870 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:11.419879 7f9d57543700 15 osd.2 173 enqueue_op 0x39576800 prio 196 cost 0 latency 0.000096 osd_sub_op_reply(client.4289.0:143626 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.419908 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.419915 7f9d61bd4700 10 osd.2 173 dequeue_op 0x39576800 prio 196 cost 0 latency 0.000133 osd_sub_op_reply(client.4289.0:143626 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 pg pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean]
2014-06-12 14:46:11.419938 7f9d61bd4700  5 -- op tracker -- , seq: 747654, time: 2014-06-12 14:46:11.419934, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143626 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.419946 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] handle_message: 0x39576800
2014-06-12 14:46:11.419965 7f9d61bd4700  5 -- op tracker -- , seq: 747654, time: 2014-06-12 14:46:11.419961, event: started, op: osd_sub_op_reply(client.4289.0:143626 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.419972 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] sub_op_modify_reply: tid 236222 op  ack_type 4 from 23
2014-06-12 14:46:11.419998 7f9d61bd4700  5 -- op tracker -- , seq: 747652, time: 2014-06-12 14:46:11.419987, event: sub_op_commit_rec, op: osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.420006 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] repop_all_applied: repop tid 236222 all applied 
2014-06-12 14:46:11.420017 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] eval_repop repgather(0x15e57080 173'2354 rep_tid=236222 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.420036 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean]  sending ack on repgather(0x15e57080 173'2354 rep_tid=236222 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x24675680
2014-06-12 14:46:11.420073 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2353 crt=0'0 lcod 173'2352 mlcod 173'2352 active+clean] repop_all_committed: repop tid 236222 all committed 
2014-06-12 14:46:11.420086 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2352 active+clean] eval_repop repgather(0x15e57080 173'2354 rep_tid=236222 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.420105 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2352 active+clean] log_op_stats osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.020755 lat 0.020789
2014-06-12 14:46:11.420124 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2352 active+clean] publish_stats_to_osd 173:3083
2014-06-12 14:46:11.420138 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2352 active+clean]  sending commit on repgather(0x15e57080 173'2354 rep_tid=236222 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x37812f00
2014-06-12 14:46:11.420177 7f9d61bd4700  5 -- op tracker -- , seq: 747652, time: 2014-06-12 14:46:11.420172, event: commit_sent, op: osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.420186 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2352 active+clean]  requeue_ops 
2014-06-12 14:46:11.420198 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean]  removing repgather(0x15e57080 173'2354 rep_tid=236222 committed?=1 applied?=1 op=osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.420213 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean]    q front is repgather(0x15e57080 173'2354 rep_tid=236222 committed?=1 applied?=1 op=osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.420228 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] remove_repop repgather(0x15e57080 173'2354 rep_tid=236222 committed?=1 applied?=1 op=osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.420242 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean]  requeue_ops 
2014-06-12 14:46:11.420264 7f9d61bd4700  5 -- op tracker -- , seq: 747652, time: 2014-06-12 14:46:11.420258, event: done, op: osd_op(client.4289.0:143626 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 1699840~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.420289 7f9d61bd4700 10 osd.2 173 dequeue_op 0x39576800 finish
2014-06-12 14:46:11.420297 7f9d61bd4700  5 -- op tracker -- , seq: 747654, time: 2014-06-12 14:46:11.420293, event: done, op: osd_sub_op_reply(client.4289.0:143626 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.420308 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.420311 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.423770 7f9d48d28700  5 -- op tracker -- , seq: 747655, time: 2014-06-12 14:46:11.422667, event: header_read, op: osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.423784 7f9d48d28700  5 -- op tracker -- , seq: 747655, time: 2014-06-12 14:46:11.422672, event: throttled, op: osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.423794 7f9d48d28700  5 -- op tracker -- , seq: 747655, time: 2014-06-12 14:46:11.423727, event: all_read, op: osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.423804 7f9d48d28700  5 -- op tracker -- , seq: 747655, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.423811 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:11.423842 7f9d48d28700 15 osd.2 173 enqueue_op 0x2519d800 prio 63 cost 516096 latency 0.001175 osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.423887 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.423894 7f9d663dd700 10 osd.2 173 dequeue_op 0x2519d800 prio 63 cost 516096 latency 0.001227 osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean]
2014-06-12 14:46:11.423919 7f9d663dd700  5 -- op tracker -- , seq: 747655, time: 2014-06-12 14:46:11.423914, event: reached_pg, op: osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.423928 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:11.423943 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] handle_message: 0x2519d800
2014-06-12 14:46:11.423953 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] do_op osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:11.423977 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] get_object_context: found obc in cache: 0x6ac0000
2014-06-12 14:46:11.423988 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] get_object_context: 0x6ac0000 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 rwstate(write n=1 w=0) oi: 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2355 client.4289.0:143627 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2355) ssc: 0x2ef8ed20 snapset: 0=[]:[]+head
2014-06-12 14:46:11.424003 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] find_object_context 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 @head oi=32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2355 client.4289.0:143627 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2355)
2014-06-12 14:46:11.424029 7f9d663dd700  5 -- op tracker -- , seq: 747655, time: 2014-06-12 14:46:11.424022, event: started, op: osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.424037 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] execute_ctx 0x5e4d000
2014-06-12 14:46:11.424054 7f9d663dd700  5 -- op tracker -- , seq: 747655, time: 2014-06-12 14:46:11.424049, event: started, op: osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.424061 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] do_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] ov 173'2355 av 173'2356 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:11.424083 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] do_osd_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096]
2014-06-12 14:46:11.424094 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:11.424111 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] do_osd_op  write 2732032~516096
2014-06-12 14:46:11.424123 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:11.424133 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 snapset=0x2ef8ed58  snapc=0=[]
2014-06-12 14:46:11.424145 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:11.424155 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] finish_ctx 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 0x5e4d000 op modify  
2014-06-12 14:46:11.424167 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean]  set mtime to 2014-06-12 08:45:13.802725
2014-06-12 14:46:11.424187 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean]  final snapset 0=[]:[]+head in 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.424204 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean]  zeroing write result code 0
2014-06-12 14:46:11.424214 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] new_repop rep_tid 236224 on osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.424229 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] issue_repop rep_tid 236224 o 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.424252 7f9d663dd700  5 -- op tracker -- , seq: 747655, time: 2014-06-12 14:46:11.424246, event: waiting for subops from 23, op: osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.424271 7f9d663dd700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:11.424291 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2355 (0'0,173'2355] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] append_log log((0'0,173'2355], crt=0'0) [173'2356 (173'2355) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143628 2014-06-12 08:45:13.802725]
2014-06-12 14:46:11.424312 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] add_log_entry 173'2356 (173'2355) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143628 2014-06-12 08:45:13.802725
2014-06-12 14:46:11.424334 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] append_log  adding 1 keys
2014-06-12 14:46:11.424363 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2356, trimmed: 
2014-06-12 14:46:11.424384 7f9d663dd700  5 queue_transactions existing osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.424388 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.424389 7f9d663dd700  5 queue_op 0x68a5c20 seq 2267 osr(3.45f 0x276d940) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:11.424397 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] eval_repop repgather(0x5214900 173'2356 rep_tid=236224 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.424415 7f9d663dd700 10 osd.2 173 dequeue_op 0x2519d800 finish
2014-06-12 14:46:11.424419 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.424421 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.424429 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1e60 (2 active)
2014-06-12 14:46:11.425155 7f9d57543700  5 -- op tracker -- , seq: 747656, time: 2014-06-12 14:46:11.425091, event: header_read, op: osd_sub_op_reply(client.4289.0:143627 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.425166 7f9d57543700  5 -- op tracker -- , seq: 747656, time: 2014-06-12 14:46:11.425093, event: throttled, op: osd_sub_op_reply(client.4289.0:143627 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.425174 7f9d57543700  5 -- op tracker -- , seq: 747656, time: 2014-06-12 14:46:11.425104, event: all_read, op: osd_sub_op_reply(client.4289.0:143627 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.425181 7f9d57543700  5 -- op tracker -- , seq: 747656, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143627 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.425187 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143627 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.425193 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:11.425200 7f9d57543700 15 osd.2 173 enqueue_op 0x9facc00 prio 196 cost 0 latency 0.000109 osd_sub_op_reply(client.4289.0:143627 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.425236 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.425243 7f9d673df700 10 osd.2 173 dequeue_op 0x9facc00 prio 196 cost 0 latency 0.000152 osd_sub_op_reply(client.4289.0:143627 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 pg pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean]
2014-06-12 14:46:11.425262 7f9d673df700  5 -- op tracker -- , seq: 747656, time: 2014-06-12 14:46:11.425259, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143627 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.425270 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] handle_message: 0x9facc00
2014-06-12 14:46:11.425286 7f9d673df700  5 -- op tracker -- , seq: 747656, time: 2014-06-12 14:46:11.425282, event: started, op: osd_sub_op_reply(client.4289.0:143627 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.425292 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] sub_op_modify_reply: tid 236223 op  ack_type 4 from 23
2014-06-12 14:46:11.425327 7f9d673df700  5 -- op tracker -- , seq: 747653, time: 2014-06-12 14:46:11.425320, event: sub_op_commit_rec, op: osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.425339 7f9d673df700 10 osd.2 173 dequeue_op 0x9facc00 finish
2014-06-12 14:46:11.425346 7f9d673df700  5 -- op tracker -- , seq: 747656, time: 2014-06-12 14:46:11.425343, event: done, op: osd_sub_op_reply(client.4289.0:143627 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.425361 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.425364 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.425718 7f9d7d968700 10 _do_op 0x92a54a0 seq 2266 r = 0, finisher 0x1d4990a0 0x36424a40
2014-06-12 14:46:11.425743 7f9d7d968700 10 _finish_op 0x92a54a0 seq 2266 osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.425750 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] op_commit: 236223
2014-06-12 14:46:11.425764 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1e60 (1 active)
2014-06-12 14:46:11.425769 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.425774 7f9d7715b700  5 -- op tracker -- , seq: 747653, time: 2014-06-12 14:46:11.425766, event: op_commit, op: osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.425784 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2354 crt=0'0 lcod 173'2353 mlcod 173'2353 active+clean] repop_all_committed: repop tid 236223 all committed 
2014-06-12 14:46:11.425798 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2353 active+clean] eval_repop repgather(0x18225ec0 173'2355 rep_tid=236223 committed?=1 applied?=0 lock=0 op=osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.425817 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2353 active+clean] log_op_stats osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.015086
2014-06-12 14:46:11.425835 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2353 active+clean] publish_stats_to_osd 173:3084
2014-06-12 14:46:11.425847 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2353 active+clean]  sending commit on repgather(0x18225ec0 173'2355 rep_tid=236223 committed?=1 applied?=0 lock=0 op=osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x11666280
2014-06-12 14:46:11.425883 7f9d7715b700  5 -- op tracker -- , seq: 747653, time: 2014-06-12 14:46:11.425878, event: commit_sent, op: osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.425899 7f9d83173700  5 _do_op 0x68a5c20 seq 2267 osr(3.45f 0x276d940)/0x276d940 start
2014-06-12 14:46:11.425904 7f9d83173700 10 _do_transaction on 0x15e566c0
2014-06-12 14:46:11.425912 7f9d83173700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.426040 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.426061 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.426119 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.426149 7f9d83173700 15 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.426157 7f9d83173700 10 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1 r = 0
2014-06-12 14:46:11.426164 7f9d83173700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.426236 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2353 active+clean] op_applied: 236223
2014-06-12 14:46:11.426262 7f9d7795c700  5 -- op tracker -- , seq: 747653, time: 2014-06-12 14:46:11.426255, event: op_applied, op: osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.426273 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2353 active+clean] op_applied on primary on version 173'2355
2014-06-12 14:46:11.426284 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2353 active+clean] repop_all_applied: repop tid 236223 all applied 
2014-06-12 14:46:11.426297 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2353 active+clean] eval_repop repgather(0x18225ec0 173'2355 rep_tid=236223 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.426312 7f9d83173700 15 _write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 2732032~516096
2014-06-12 14:46:11.426311 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2353 active+clean] log_op_stats osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.015580
2014-06-12 14:46:11.426325 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2353 active+clean] publish_stats_to_osd 173:3085
2014-06-12 14:46:11.426335 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2353 active+clean]  requeue_ops 
2014-06-12 14:46:11.426343 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2354 active+clean]  removing repgather(0x18225ec0 173'2355 rep_tid=236223 committed?=1 applied?=1 op=osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.426355 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2354 active+clean]    q front is repgather(0x18225ec0 173'2355 rep_tid=236223 committed?=1 applied?=1 op=osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.426367 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2354 active+clean] remove_repop repgather(0x18225ec0 173'2355 rep_tid=236223 committed?=1 applied?=1 op=osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.426381 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2354 active+clean]  requeue_ops 
2014-06-12 14:46:11.426407 7f9d7795c700  5 -- op tracker -- , seq: 747653, time: 2014-06-12 14:46:11.426402, event: done, op: osd_op(client.4289.0:143627 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.426472 7f9d83173700 10 lookup_strip_header done  cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.426624 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:11.428783 7f9d83173700 10 _generic_write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 2732032~516096 = 0
2014-06-12 14:46:11.428852 7f9d83173700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.428873 7f9d83173700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.428881 7f9d83173700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.428891 7f9d83173700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.429111 7f9d83173700 20 set_header setting 17449 cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 parent seq 0
2014-06-12 14:46:11.429202 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.429220 7f9d83173700 20 set_header setting 2074 cid meta oid 1fe1bdeb/pglog_3.45f/0//-1 parent seq 0
2014-06-12 14:46:11.429233 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:11.433255 7f9d48d28700  5 -- op tracker -- , seq: 747657, time: 2014-06-12 14:46:11.432098, event: header_read, op: osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.433285 7f9d48d28700  5 -- op tracker -- , seq: 747657, time: 2014-06-12 14:46:11.432104, event: throttled, op: osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.433296 7f9d48d28700  5 -- op tracker -- , seq: 747657, time: 2014-06-12 14:46:11.433200, event: all_read, op: osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.433306 7f9d48d28700  5 -- op tracker -- , seq: 747657, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.433315 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:11.433343 7f9d48d28700 15 osd.2 173 enqueue_op 0x2519ce00 prio 63 cost 516096 latency 0.001245 osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.433600 7f9d83173700 10 _do_op 0x68a5c20 seq 2267 r = 0, finisher 0x3a4a6a0 0x218d0100
2014-06-12 14:46:11.433634 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2354 active+clean] op_commit: 236224
2014-06-12 14:46:11.433661 7f9d7715b700  5 -- op tracker -- , seq: 747655, time: 2014-06-12 14:46:11.433654, event: op_commit, op: osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.433676 7f9d83173700 10 _finish_op 0x68a5c20 seq 2267 osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.433686 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1e60 (0 active)
2014-06-12 14:46:11.433696 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.434185 7f9d57543700  5 -- op tracker -- , seq: 747658, time: 2014-06-12 14:46:11.434144, event: header_read, op: osd_sub_op_reply(client.4289.0:143628 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.434195 7f9d57543700  5 -- op tracker -- , seq: 747658, time: 2014-06-12 14:46:11.434146, event: throttled, op: osd_sub_op_reply(client.4289.0:143628 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.434203 7f9d57543700  5 -- op tracker -- , seq: 747658, time: 2014-06-12 14:46:11.434157, event: all_read, op: osd_sub_op_reply(client.4289.0:143628 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.434210 7f9d57543700  5 -- op tracker -- , seq: 747658, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143628 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.434215 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143628 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.434222 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:11.434229 7f9d57543700 15 osd.2 173 enqueue_op 0x9fade00 prio 196 cost 0 latency 0.000085 osd_sub_op_reply(client.4289.0:143628 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.434259 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.434276 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:11.434267 7f9d67be0700 10 osd.2 173 dequeue_op 0x9fade00 prio 196 cost 0 latency 0.000122 osd_sub_op_reply(client.4289.0:143628 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 pg pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2354 active+clean]
2014-06-12 14:46:11.434285 7f9d67be0700  5 -- op tracker -- , seq: 747658, time: 2014-06-12 14:46:11.434281, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143628 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.434293 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2354 active+clean] handle_message: 0x9fade00
2014-06-12 14:46:11.434308 7f9d67be0700  5 -- op tracker -- , seq: 747658, time: 2014-06-12 14:46:11.434305, event: started, op: osd_sub_op_reply(client.4289.0:143628 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.434314 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2354 active+clean] sub_op_modify_reply: tid 236224 op  ack_type 4 from 23
2014-06-12 14:46:11.434334 7f9d67be0700  5 -- op tracker -- , seq: 747655, time: 2014-06-12 14:46:11.434327, event: sub_op_commit_rec, op: osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.434352 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2355 crt=0'0 lcod 173'2354 mlcod 173'2354 active+clean] repop_all_committed: repop tid 236224 all committed 
2014-06-12 14:46:11.434363 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] eval_repop repgather(0x5214900 173'2356 rep_tid=236224 committed?=1 applied?=0 lock=0 op=osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.434381 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] log_op_stats osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.011712
2014-06-12 14:46:11.434398 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] publish_stats_to_osd 173:3086
2014-06-12 14:46:11.434408 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean]  sending commit on repgather(0x5214900 173'2356 rep_tid=236224 committed?=1 applied?=0 lock=0 op=osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x10148c80
2014-06-12 14:46:11.434444 7f9d67be0700  5 -- op tracker -- , seq: 747655, time: 2014-06-12 14:46:11.434438, event: commit_sent, op: osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.434453 7f9d67be0700 10 osd.2 173 dequeue_op 0x9fade00 finish
2014-06-12 14:46:11.434462 7f9d67be0700  5 -- op tracker -- , seq: 747658, time: 2014-06-12 14:46:11.434459, event: done, op: osd_sub_op_reply(client.4289.0:143628 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.434480 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:11.434483 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.434490 7f9d693e3700 10 osd.2 173 dequeue_op 0x2519ce00 prio 63 cost 516096 latency 0.002392 osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean]
2014-06-12 14:46:11.434512 7f9d693e3700  5 -- op tracker -- , seq: 747657, time: 2014-06-12 14:46:11.434507, event: reached_pg, op: osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.434520 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:11.434534 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] handle_message: 0x2519ce00
2014-06-12 14:46:11.434544 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] do_op osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:11.434569 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] get_object_context: found obc in cache: 0x6ac0000
2014-06-12 14:46:11.434579 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] get_object_context: 0x6ac0000 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 rwstate(write n=1 w=0) oi: 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2356 client.4289.0:143628 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2356) ssc: 0x2ef8ed20 snapset: 0=[]:[]+head
2014-06-12 14:46:11.434595 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] find_object_context 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 @head oi=32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2356 client.4289.0:143628 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2356)
2014-06-12 14:46:11.434620 7f9d693e3700  5 -- op tracker -- , seq: 747657, time: 2014-06-12 14:46:11.434614, event: started, op: osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.434628 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] execute_ctx 0x26f9c600
2014-06-12 14:46:11.434646 7f9d693e3700  5 -- op tracker -- , seq: 747657, time: 2014-06-12 14:46:11.434641, event: started, op: osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.434653 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] do_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] ov 173'2356 av 173'2357 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:11.434666 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] do_osd_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096]
2014-06-12 14:46:11.434677 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:11.434689 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] do_osd_op  write 3248128~516096
2014-06-12 14:46:11.434704 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:11.434715 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 snapset=0x2ef8ed58  snapc=0=[]
2014-06-12 14:46:11.434732 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:11.434742 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] finish_ctx 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 0x26f9c600 op modify  
2014-06-12 14:46:11.434754 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean]  set mtime to 2014-06-12 08:45:13.812132
2014-06-12 14:46:11.434776 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean]  final snapset 0=[]:[]+head in 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.434795 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean]  zeroing write result code 0
2014-06-12 14:46:11.434806 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] new_repop rep_tid 236225 on osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.434822 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] issue_repop rep_tid 236225 o 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.434848 7f9d693e3700  5 -- op tracker -- , seq: 747657, time: 2014-06-12 14:46:11.434843, event: waiting for subops from 23, op: osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.434868 7f9d693e3700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:11.434889 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2356 (0'0,173'2356] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] append_log log((0'0,173'2356], crt=0'0) [173'2357 (173'2356) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143629 2014-06-12 08:45:13.812132]
2014-06-12 14:46:11.434917 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] add_log_entry 173'2357 (173'2356) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143629 2014-06-12 08:45:13.812132
2014-06-12 14:46:11.434938 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] append_log  adding 1 keys
2014-06-12 14:46:11.434971 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2357, trimmed: 
2014-06-12 14:46:11.434994 7f9d693e3700  5 queue_transactions existing osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.434999 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.435001 7f9d693e3700  5 queue_op 0x9287680 seq 2268 osr(3.45f 0x276d940) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:11.435008 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] eval_repop repgather(0x5322600 173'2357 rep_tid=236225 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.435027 7f9d693e3700 10 osd.2 173 dequeue_op 0x2519ce00 finish
2014-06-12 14:46:11.435033 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.435036 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.435042 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] op_applied: 236224
2014-06-12 14:46:11.435061 7f9d7795c700  5 -- op tracker -- , seq: 747655, time: 2014-06-12 14:46:11.435055, event: op_applied, op: osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.435069 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] op_applied on primary on version 173'2356
2014-06-12 14:46:11.435079 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] repop_all_applied: repop tid 236224 all applied 
2014-06-12 14:46:11.435089 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] eval_repop repgather(0x5214900 173'2356 rep_tid=236224 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.435104 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] log_op_stats osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.012436
2014-06-12 14:46:11.435130 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean] publish_stats_to_osd 173:3087
2014-06-12 14:46:11.435140 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2354 active+clean]  requeue_ops 
2014-06-12 14:46:11.435150 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean]  removing repgather(0x5214900 173'2356 rep_tid=236224 committed?=1 applied?=1 op=osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.435164 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean]    q front is repgather(0x5214900 173'2356 rep_tid=236224 committed?=1 applied?=1 op=osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.435177 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] remove_repop repgather(0x5214900 173'2356 rep_tid=236224 committed?=1 applied?=1 op=osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.435191 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean]  requeue_ops 
2014-06-12 14:46:11.435214 7f9d7795c700  5 -- op tracker -- , seq: 747655, time: 2014-06-12 14:46:11.435209, event: done, op: osd_op(client.4289.0:143628 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.435628 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1e60 (1 active)
2014-06-12 14:46:11.435639 7f9d7c165700  5 _do_op 0x9287680 seq 2268 osr(3.45f 0x276d940)/0x276d940 start
2014-06-12 14:46:11.435644 7f9d7c165700 10 _do_transaction on 0x534f680
2014-06-12 14:46:11.435651 7f9d7c165700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.435785 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.435818 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.435868 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.435894 7f9d7c165700 15 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.435901 7f9d7c165700 10 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1 r = 0
2014-06-12 14:46:11.435907 7f9d7c165700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.435925 7f9d7c165700 15 _write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 3248128~516096
2014-06-12 14:46:11.436066 7f9d7c165700 10 lookup_strip_header done  cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.436319 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:11.437699 7f9d48d28700  5 -- op tracker -- , seq: 747659, time: 2014-06-12 14:46:11.436942, event: header_read, op: osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.437720 7f9d48d28700  5 -- op tracker -- , seq: 747659, time: 2014-06-12 14:46:11.436945, event: throttled, op: osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.437730 7f9d48d28700  5 -- op tracker -- , seq: 747659, time: 2014-06-12 14:46:11.437657, event: all_read, op: osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.437748 7f9d48d28700  5 -- op tracker -- , seq: 747659, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.437755 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:11.437784 7f9d48d28700 15 osd.2 173 enqueue_op 0x2519d500 prio 63 cost 430080 latency 0.000841 osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.437813 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.437820 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2519d500 prio 63 cost 430080 latency 0.000878 osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean]
2014-06-12 14:46:11.437841 7f9d6b3e7700  5 -- op tracker -- , seq: 747659, time: 2014-06-12 14:46:11.437838, event: reached_pg, op: osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.437854 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:11.437871 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] handle_message: 0x2519d500
2014-06-12 14:46:11.437883 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] do_op osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:11.437918 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] get_object_context: found obc in cache: 0x6ac0000
2014-06-12 14:46:11.437959 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] get_object_context: 0x6ac0000 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 rwstate(write n=1 w=0) oi: 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2357 client.4289.0:143629 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2357) ssc: 0x2ef8ed20 snapset: 0=[]:[]+head
2014-06-12 14:46:11.437984 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] find_object_context 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 @head oi=32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3(173'2357 client.4289.0:143629 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2357)
2014-06-12 14:46:11.438013 7f9d6b3e7700  5 -- op tracker -- , seq: 747659, time: 2014-06-12 14:46:11.438007, event: started, op: osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.438026 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] execute_ctx 0x2963a600
2014-06-12 14:46:11.438046 7f9d6b3e7700  5 -- op tracker -- , seq: 747659, time: 2014-06-12 14:46:11.438040, event: started, op: osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.438053 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] do_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] ov 173'2357 av 173'2358 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:11.438068 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] do_osd_op 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080]
2014-06-12 14:46:11.438080 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:11.438095 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] do_osd_op  write 3764224~430080
2014-06-12 14:46:11.438112 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:11.438124 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 snapset=0x2ef8ed58  snapc=0=[]
2014-06-12 14:46:11.438138 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] make_writeable 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:11.438149 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] finish_ctx 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 0x2963a600 op modify  
2014-06-12 14:46:11.438163 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean]  set mtime to 2014-06-12 08:45:13.817093
2014-06-12 14:46:11.438185 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean]  final snapset 0=[]:[]+head in 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.438205 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean]  zeroing write result code 0
2014-06-12 14:46:11.438216 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] new_repop rep_tid 236226 on osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:11.438233 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] issue_repop rep_tid 236226 o 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.438260 7f9d6b3e7700  5 -- op tracker -- , seq: 747659, time: 2014-06-12 14:46:11.438254, event: waiting for subops from 23, op: osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.438281 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:11.438304 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2357 (0'0,173'2357] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] append_log log((0'0,173'2357], crt=0'0) [173'2358 (173'2357) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143630 2014-06-12 08:45:13.817093]
2014-06-12 14:46:11.438329 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] add_log_entry 173'2358 (173'2357) modify   32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 by client.4289.0:143630 2014-06-12 08:45:13.817093
2014-06-12 14:46:11.438351 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] append_log  adding 1 keys
2014-06-12 14:46:11.438387 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2358, trimmed: 
2014-06-12 14:46:11.438409 7f9d6b3e7700  5 queue_transactions existing osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.438414 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.438415 7f9d6b3e7700  5 queue_op 0x69afd40 seq 2269 osr(3.45f 0x276d940) 432373 bytes   (queue has 2 ops and 950762 bytes)
2014-06-12 14:46:11.438423 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] eval_repop repgather(0x384e2180 173'2358 rep_tid=236226 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.438443 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2519d500 finish
2014-06-12 14:46:11.438447 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.438450 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.438460 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1e60 (2 active)
2014-06-12 14:46:11.438487 7f9d7c165700 10 _generic_write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 3248128~516096 = 0
2014-06-12 14:46:11.438539 7f9d7c165700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.438553 7f9d7c165700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.438559 7f9d7c165700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.438567 7f9d7c165700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.438776 7f9d7c165700 20 set_header setting 17449 cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 parent seq 0
2014-06-12 14:46:11.438863 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.439030 7f9d7c165700 20 set_header setting 2074 cid meta oid 1fe1bdeb/pglog_3.45f/0//-1 parent seq 0
2014-06-12 14:46:11.439041 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:11.444200 7f9d57543700  5 -- op tracker -- , seq: 747660, time: 2014-06-12 14:46:11.444137, event: header_read, op: osd_sub_op_reply(client.4289.0:143629 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.444213 7f9d57543700  5 -- op tracker -- , seq: 747660, time: 2014-06-12 14:46:11.444140, event: throttled, op: osd_sub_op_reply(client.4289.0:143629 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.444221 7f9d57543700  5 -- op tracker -- , seq: 747660, time: 2014-06-12 14:46:11.444155, event: all_read, op: osd_sub_op_reply(client.4289.0:143629 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.444229 7f9d57543700  5 -- op tracker -- , seq: 747660, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143629 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.444236 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143629 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.444243 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:11.444252 7f9d57543700 15 osd.2 173 enqueue_op 0x9fada00 prio 196 cost 0 latency 0.000114 osd_sub_op_reply(client.4289.0:143629 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.444287 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.444296 7f9d62bd6700 10 osd.2 173 dequeue_op 0x9fada00 prio 196 cost 0 latency 0.000159 osd_sub_op_reply(client.4289.0:143629 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 pg pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean]
2014-06-12 14:46:11.444334 7f9d62bd6700  5 -- op tracker -- , seq: 747660, time: 2014-06-12 14:46:11.444329, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143629 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.444344 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] handle_message: 0x9fada00
2014-06-12 14:46:11.444359 7f9d62bd6700  5 -- op tracker -- , seq: 747660, time: 2014-06-12 14:46:11.444356, event: started, op: osd_sub_op_reply(client.4289.0:143629 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.444364 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] sub_op_modify_reply: tid 236225 op  ack_type 4 from 23
2014-06-12 14:46:11.444385 7f9d62bd6700  5 -- op tracker -- , seq: 747657, time: 2014-06-12 14:46:11.444377, event: sub_op_commit_rec, op: osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.444394 7f9d62bd6700 10 osd.2 173 dequeue_op 0x9fada00 finish
2014-06-12 14:46:11.444401 7f9d62bd6700  5 -- op tracker -- , seq: 747660, time: 2014-06-12 14:46:11.444397, event: done, op: osd_sub_op_reply(client.4289.0:143629 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.444415 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.444418 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.446361 7f9d7c165700 10 _do_op 0x9287680 seq 2268 r = 0, finisher 0x4f2fea0 0x1556e040
2014-06-12 14:46:11.446396 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] op_commit: 236225
2014-06-12 14:46:11.446433 7f9d7715b700  5 -- op tracker -- , seq: 747657, time: 2014-06-12 14:46:11.446422, event: op_commit, op: osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.446450 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2356 crt=0'0 lcod 173'2355 mlcod 173'2355 active+clean] repop_all_committed: repop tid 236225 all committed 
2014-06-12 14:46:11.446460 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2355 active+clean] eval_repop repgather(0x5322600 173'2357 rep_tid=236225 committed?=1 applied?=0 lock=0 op=osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.446475 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2355 active+clean] log_op_stats osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.014375
2014-06-12 14:46:11.446491 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2355 active+clean] publish_stats_to_osd 173:3088
2014-06-12 14:46:11.446500 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2355 active+clean]  sending commit on repgather(0x5322600 173'2357 rep_tid=236225 committed?=1 applied?=0 lock=0 op=osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x17f00a00
2014-06-12 14:46:11.446538 7f9d7715b700  5 -- op tracker -- , seq: 747657, time: 2014-06-12 14:46:11.446533, event: commit_sent, op: osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.446604 7f9d7c165700 10 _finish_op 0x9287680 seq 2268 osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.446612 7f9d82972700  5 _do_op 0x69afd40 seq 2269 osr(3.45f 0x276d940)/0x276d940 start
2014-06-12 14:46:11.446616 7f9d82972700 10 _do_transaction on 0x384e2c00
2014-06-12 14:46:11.446626 7f9d82972700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.446744 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.446767 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.446814 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.446832 7f9d82972700 15 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.446838 7f9d82972700 10 _omap_rmkeys meta/1fe1bdeb/pglog_3.45f/0//-1 r = 0
2014-06-12 14:46:11.446844 7f9d82972700 15 _omap_setkeys meta/1fe1bdeb/pglog_3.45f/0//-1
2014-06-12 14:46:11.446965 7f9d82972700 15 _write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 3764224~430080
2014-06-12 14:46:11.447116 7f9d82972700 10 lookup_strip_header done  cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.447240 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:11.448453 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2355 active+clean] op_applied: 236225
2014-06-12 14:46:11.448486 7f9d7795c700  5 -- op tracker -- , seq: 747657, time: 2014-06-12 14:46:11.448478, event: op_applied, op: osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.448495 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2355 active+clean] op_applied on primary on version 173'2357
2014-06-12 14:46:11.448506 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2355 active+clean] repop_all_applied: repop tid 236225 all applied 
2014-06-12 14:46:11.448517 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2355 active+clean] eval_repop repgather(0x5322600 173'2357 rep_tid=236225 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.448533 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2355 active+clean] log_op_stats osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.016434
2014-06-12 14:46:11.448550 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2355 active+clean] publish_stats_to_osd 173:3089
2014-06-12 14:46:11.448560 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2355 active+clean]  requeue_ops 
2014-06-12 14:46:11.448570 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean]  removing repgather(0x5322600 173'2357 rep_tid=236225 committed?=1 applied?=1 op=osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.448584 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean]    q front is repgather(0x5322600 173'2357 rep_tid=236225 committed?=1 applied?=1 op=osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.448598 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean] remove_repop repgather(0x5322600 173'2357 rep_tid=236225 committed?=1 applied?=1 op=osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.448611 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean]  requeue_ops 
2014-06-12 14:46:11.448819 7f9d7c165700  5 -- op tracker -- , seq: 747657, time: 2014-06-12 14:46:11.448796, event: done, op: osd_op(client.4289.0:143629 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~516096] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.448863 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1e60 (1 active)
2014-06-12 14:46:11.448866 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.448885 7f9d82972700 10 _generic_write 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 3764224~430080 = 0
2014-06-12 14:46:11.448935 7f9d82972700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.448953 7f9d82972700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.448958 7f9d82972700 15 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3
2014-06-12 14:46:11.448966 7f9d82972700 10 _setattrs 3.45f_head/32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 = 0
2014-06-12 14:46:11.449160 7f9d82972700 20 set_header setting 17449 cid 3.45f_head oid 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 parent seq 0
2014-06-12 14:46:11.449232 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.449248 7f9d82972700 20 set_header setting 2074 cid meta oid 1fe1bdeb/pglog_3.45f/0//-1 parent seq 0
2014-06-12 14:46:11.449258 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:11.452816 7f9d82972700 10 _do_op 0x69afd40 seq 2269 r = 0, finisher 0x7bdc000 0x3b72a00
2014-06-12 14:46:11.452835 7f9d82972700 10 _finish_op 0x69afd40 seq 2269 osr(3.45f 0x276d940)/0x276d940
2014-06-12 14:46:11.452839 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean] op_commit: 236226
2014-06-12 14:46:11.452854 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1e60 (0 active)
2014-06-12 14:46:11.452862 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.452868 7f9d7715b700  5 -- op tracker -- , seq: 747659, time: 2014-06-12 14:46:11.452860, event: op_commit, op: osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.452883 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean] op_applied: 236226
2014-06-12 14:46:11.452899 7f9d7795c700  5 -- op tracker -- , seq: 747659, time: 2014-06-12 14:46:11.452895, event: op_applied, op: osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.452904 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean] op_applied on primary on version 173'2358
2014-06-12 14:46:11.453491 7f9d57543700  5 -- op tracker -- , seq: 747661, time: 2014-06-12 14:46:11.453446, event: header_read, op: osd_sub_op_reply(client.4289.0:143630 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.453497 7f9d57543700  5 -- op tracker -- , seq: 747661, time: 2014-06-12 14:46:11.453448, event: throttled, op: osd_sub_op_reply(client.4289.0:143630 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.453502 7f9d57543700  5 -- op tracker -- , seq: 747661, time: 2014-06-12 14:46:11.453466, event: all_read, op: osd_sub_op_reply(client.4289.0:143630 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.453506 7f9d57543700  5 -- op tracker -- , seq: 747661, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143630 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.453511 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143630 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.453515 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:11.453521 7f9d57543700 15 osd.2 173 enqueue_op 0x9fac200 prio 196 cost 0 latency 0.000075 osd_sub_op_reply(client.4289.0:143630 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.453547 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.453556 7f9d683e1700 10 osd.2 173 dequeue_op 0x9fac200 prio 196 cost 0 latency 0.000110 osd_sub_op_reply(client.4289.0:143630 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0) v2 pg pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean]
2014-06-12 14:46:11.453579 7f9d683e1700  5 -- op tracker -- , seq: 747661, time: 2014-06-12 14:46:11.453576, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143630 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.453590 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean] handle_message: 0x9fac200
2014-06-12 14:46:11.453607 7f9d683e1700  5 -- op tracker -- , seq: 747661, time: 2014-06-12 14:46:11.453603, event: started, op: osd_sub_op_reply(client.4289.0:143630 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.453630 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean] sub_op_modify_reply: tid 236226 op  ack_type 4 from 23
2014-06-12 14:46:11.453651 7f9d683e1700  5 -- op tracker -- , seq: 747659, time: 2014-06-12 14:46:11.453644, event: sub_op_commit_rec, op: osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.453659 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean] repop_all_applied: repop tid 236226 all applied 
2014-06-12 14:46:11.453669 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean] eval_repop repgather(0x384e2180 173'2358 rep_tid=236226 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.453685 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean]  sending ack on repgather(0x384e2180 173'2358 rep_tid=236226 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x35714280
2014-06-12 14:46:11.453717 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2357 crt=0'0 lcod 173'2356 mlcod 173'2356 active+clean] repop_all_committed: repop tid 236226 all committed 
2014-06-12 14:46:11.453727 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2356 active+clean] eval_repop repgather(0x384e2180 173'2358 rep_tid=236226 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.453742 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2356 active+clean] log_op_stats osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4 inb 430359 outb 0 rlat 0.016773 lat 0.016799
2014-06-12 14:46:11.453759 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2356 active+clean] publish_stats_to_osd 173:3090
2014-06-12 14:46:11.453772 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2356 active+clean]  sending commit on repgather(0x384e2180 173'2358 rep_tid=236226 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4) 0x1a58c780
2014-06-12 14:46:11.453804 7f9d683e1700  5 -- op tracker -- , seq: 747659, time: 2014-06-12 14:46:11.453798, event: commit_sent, op: osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.453812 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2356 active+clean]  requeue_ops 
2014-06-12 14:46:11.453823 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2357 active+clean]  removing repgather(0x384e2180 173'2358 rep_tid=236226 committed?=1 applied?=1 op=osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.453839 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2357 active+clean]    q front is repgather(0x384e2180 173'2358 rep_tid=236226 committed?=1 applied?=1 op=osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.453853 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2357 active+clean] remove_repop repgather(0x384e2180 173'2358 rep_tid=236226 committed?=1 applied?=1 op=osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.453866 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2357 active+clean]  requeue_ops 
2014-06-12 14:46:11.453891 7f9d683e1700  5 -- op tracker -- , seq: 747659, time: 2014-06-12 14:46:11.453886, event: done, op: osd_op(client.4289.0:143630 rbd_data.10986b8b4567.000000000000112e [set-alloc-hint object_size 4194304 write_size 4194304,write 3764224~430080] 3.32faac5f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.453908 7f9d683e1700 10 osd.2 173 dequeue_op 0x9fac200 finish
2014-06-12 14:46:11.453915 7f9d683e1700  5 -- op tracker -- , seq: 747661, time: 2014-06-12 14:46:11.453912, event: done, op: osd_sub_op_reply(client.4289.0:143630 3.45f 32faac5f/rbd_data.10986b8b4567.000000000000112e/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.453926 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.453929 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.471095 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:11.471121 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:11.471125 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:11.471126 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:11.471129 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 25; oldest is 33.789955 seconds old
2014-06-12 14:46:11.471131 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:11.471181 7f9d84175700  0 log [WRN] : 13 slow requests, 3 included below; oldest blocked for > 33.789955 secs
2014-06-12 14:46:11.471186 7f9d84175700  0 log [WRN] : slow request 32.027642 seconds old, received at 2014-06-12 14:45:39.443486: osd_op(client.4303.0:144628 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:11.471189 7f9d84175700  0 log [WRN] : slow request 32.011239 seconds old, received at 2014-06-12 14:45:39.459889: osd_op(client.4303.0:144629 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.892731eb ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:11.471196 7f9d84175700  0 log [WRN] : slow request 31.723314 seconds old, received at 2014-06-12 14:45:39.747814: osd_op(client.4309.0:141527 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 2379776~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:11.477437 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:11.477440 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:11.477441 7f9d74bfa700 20 osd.2 173 _dispatch 0x6b78b60 ping v1
2014-06-12 14:46:11.477444 7f9d74bfa700 10 osd.2 173 ping from client.4309
2014-06-12 14:46:11.477446 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:11.477447 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:11.567635 7f9d56432700  5 -- op tracker -- , seq: 747662, time: 2014-06-12 14:46:11.567587, event: header_read, op: osd_sub_op_reply(client.4309.0:141525 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567646 7f9d56432700  5 -- op tracker -- , seq: 747662, time: 2014-06-12 14:46:11.567589, event: throttled, op: osd_sub_op_reply(client.4309.0:141525 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567654 7f9d56432700  5 -- op tracker -- , seq: 747662, time: 2014-06-12 14:46:11.567601, event: all_read, op: osd_sub_op_reply(client.4309.0:141525 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567661 7f9d56432700  5 -- op tracker -- , seq: 747662, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141525 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567668 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141525 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.567674 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:11.567681 7f9d56432700 15 osd.2 173 enqueue_op 0x68a7700 prio 196 cost 0 latency 0.000093 osd_sub_op_reply(client.4309.0:141525 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.567721 7f9d56432700  5 -- op tracker -- , seq: 747663, time: 2014-06-12 14:46:11.567699, event: header_read, op: osd_sub_op_reply(client.4309.0:141526 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567729 7f9d56432700  5 -- op tracker -- , seq: 747663, time: 2014-06-12 14:46:11.567700, event: throttled, op: osd_sub_op_reply(client.4309.0:141526 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567736 7f9d56432700  5 -- op tracker -- , seq: 747663, time: 2014-06-12 14:46:11.567706, event: all_read, op: osd_sub_op_reply(client.4309.0:141526 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567743 7f9d56432700  5 -- op tracker -- , seq: 747663, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141526 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567746 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141526 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.567751 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:11.567756 7f9d56432700 15 osd.2 173 enqueue_op 0x2519d400 prio 196 cost 0 latency 0.000057 osd_sub_op_reply(client.4309.0:141526 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.567778 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.567793 7f9d56432700  5 -- op tracker -- , seq: 747664, time: 2014-06-12 14:46:11.567771, event: header_read, op: osd_sub_op_reply(client.4306.0:144050 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567801 7f9d56432700  5 -- op tracker -- , seq: 747664, time: 2014-06-12 14:46:11.567771, event: throttled, op: osd_sub_op_reply(client.4306.0:144050 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567786 7f9d65bdc700 10 osd.2 173 dequeue_op 0x68a7700 prio 196 cost 0 latency 0.000199 osd_sub_op_reply(client.4309.0:141525 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 pg pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean]
2014-06-12 14:46:11.567809 7f9d56432700  5 -- op tracker -- , seq: 747664, time: 2014-06-12 14:46:11.567778, event: all_read, op: osd_sub_op_reply(client.4306.0:144050 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567819 7f9d65bdc700  5 -- op tracker -- , seq: 747662, time: 2014-06-12 14:46:11.567813, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141525 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567835 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:11.567844 7f9d56432700  5 -- op tracker -- , seq: 747664, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144050 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567835 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean] handle_message: 0x68a7700
2014-06-12 14:46:11.567848 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144050 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.567851 7f9d65bdc700  5 -- op tracker -- , seq: 747662, time: 2014-06-12 14:46:11.567848, event: started, op: osd_sub_op_reply(client.4309.0:141525 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567853 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:11.567858 7f9d56432700 15 osd.2 173 enqueue_op 0x59dc600 prio 196 cost 0 latency 0.000087 osd_sub_op_reply(client.4306.0:144050 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.567857 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean] sub_op_modify_reply: tid 236048 op  ack_type 4 from 26
2014-06-12 14:46:11.567875 7f9d65bdc700  5 -- op tracker -- , seq: 747035, time: 2014-06-12 14:46:11.567869, event: sub_op_commit_rec, op: osd_op(client.4309.0:141525 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 831488~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.567881 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active)
2014-06-12 14:46:11.567881 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean] repop_all_applied: repop tid 236048 all applied 
2014-06-12 14:46:11.567885 7f9d6abe6700 10 osd.2 173 dequeue_op 0x59dc600 prio 196 cost 0 latency 0.000114 osd_sub_op_reply(client.4306.0:144050 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3085 active+clean]
2014-06-12 14:46:11.567892 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean] eval_repop repgather(0x2dc1eb40 173'2880 rep_tid=236048 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141525 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 831488~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.567905 7f9d6abe6700  5 -- op tracker -- , seq: 747664, time: 2014-06-12 14:46:11.567902, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144050 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567912 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3085 active+clean] handle_message: 0x59dc600
2014-06-12 14:46:11.567908 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean]  sending ack on repgather(0x2dc1eb40 173'2880 rep_tid=236048 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141525 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 831488~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0x2027c280
2014-06-12 14:46:11.567926 7f9d6abe6700  5 -- op tracker -- , seq: 747664, time: 2014-06-12 14:46:11.567923, event: started, op: osd_sub_op_reply(client.4306.0:144050 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.567932 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3085 active+clean] sub_op_modify_reply: tid 236018 op  ack_type 4 from 26
2014-06-12 14:46:11.567942 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2879 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean] repop_all_committed: repop tid 236048 all committed 
2014-06-12 14:46:11.567950 7f9d6abe6700  5 -- op tracker -- , seq: 746965, time: 2014-06-12 14:46:11.567944, event: sub_op_commit_rec, op: osd_op(client.4306.0:144050 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 3141632~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.567952 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2878 active+clean] eval_repop repgather(0x2dc1eb40 173'2880 rep_tid=236048 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141525 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 831488~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.567957 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3085 active+clean] repop_all_applied: repop tid 236018 all applied 
2014-06-12 14:46:11.567968 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2878 active+clean] log_op_stats osd_op(client.4309.0:141525 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 831488~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 32.821840 lat 32.821867
2014-06-12 14:46:11.567975 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3085 active+clean] eval_repop repgather(0x15acefc0 173'3087 rep_tid=236018 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144050 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 3141632~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.567985 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2878 active+clean] publish_stats_to_osd 173:3266
2014-06-12 14:46:11.567990 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3085 active+clean]  sending ack on repgather(0x15acefc0 173'3087 rep_tid=236018 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144050 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 3141632~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x45df180
2014-06-12 14:46:11.567998 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2878 active+clean]  sending commit on repgather(0x2dc1eb40 173'2880 rep_tid=236048 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141525 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 831488~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0x31452f00
2014-06-12 14:46:11.568021 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3086 crt=0'0 lcod 173'3085 mlcod 173'3085 active+clean] repop_all_committed: repop tid 236018 all committed 
2014-06-12 14:46:11.568030 7f9d65bdc700  5 -- op tracker -- , seq: 747035, time: 2014-06-12 14:46:11.568025, event: commit_sent, op: osd_op(client.4309.0:141525 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 831488~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.568032 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3085 active+clean] eval_repop repgather(0x15acefc0 173'3087 rep_tid=236018 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144050 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 3141632~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.568038 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2878 active+clean]  requeue_ops 
2014-06-12 14:46:11.568048 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3085 active+clean] log_op_stats osd_op(client.4306.0:144050 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 3141632~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 33.886846 lat 33.886874
2014-06-12 14:46:11.568048 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2879 active+clean]  removing repgather(0x2dc1eb40 173'2880 rep_tid=236048 committed?=1 applied?=1 op=osd_op(client.4309.0:141525 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 831488~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.568064 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3085 active+clean] publish_stats_to_osd 173:3553
2014-06-12 14:46:11.568064 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2879 active+clean]    q front is repgather(0x2dc1eb40 173'2880 rep_tid=236048 committed?=1 applied?=1 op=osd_op(client.4309.0:141525 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 831488~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.568078 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2879 active+clean] remove_repop repgather(0x2dc1eb40 173'2880 rep_tid=236048 committed?=1 applied?=1 op=osd_op(client.4309.0:141525 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 831488~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.568077 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3085 active+clean]  sending commit on repgather(0x15acefc0 173'3087 rep_tid=236018 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144050 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 3141632~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x19e0a780
2014-06-12 14:46:11.568092 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2879 active+clean]  requeue_ops 
2014-06-12 14:46:11.568111 7f9d65bdc700  5 -- op tracker -- , seq: 747035, time: 2014-06-12 14:46:11.568106, event: done, op: osd_op(client.4309.0:141525 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 831488~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.568133 7f9d65bdc700 10 osd.2 173 dequeue_op 0x68a7700 finish
2014-06-12 14:46:11.568143 7f9d65bdc700  5 -- op tracker -- , seq: 747662, time: 2014-06-12 14:46:11.568139, event: done, op: osd_sub_op_reply(client.4309.0:141525 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.568154 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active)
2014-06-12 14:46:11.568143 7f9d64bda700 10 osd.2 173 dequeue_op 0x2519d400 prio 196 cost 0 latency 0.000444 osd_sub_op_reply(client.4309.0:141526 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 pg pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2879 active+clean]
2014-06-12 14:46:11.568158 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.568161 7f9d64bda700  5 -- op tracker -- , seq: 747663, time: 2014-06-12 14:46:11.568158, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141526 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.568167 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2879 active+clean] handle_message: 0x2519d400
2014-06-12 14:46:11.568183 7f9d64bda700  5 -- op tracker -- , seq: 747663, time: 2014-06-12 14:46:11.568180, event: started, op: osd_sub_op_reply(client.4309.0:141526 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.568187 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2879 active+clean] sub_op_modify_reply: tid 236051 op  ack_type 4 from 26
2014-06-12 14:46:11.568204 7f9d64bda700  5 -- op tracker -- , seq: 747041, time: 2014-06-12 14:46:11.568198, event: sub_op_commit_rec, op: osd_op(client.4309.0:141526 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 1347584~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.568210 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2879 active+clean] repop_all_applied: repop tid 236051 all applied 
2014-06-12 14:46:11.568219 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2879 active+clean] eval_repop repgather(0x384e2900 173'2881 rep_tid=236051 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141526 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 1347584~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.568233 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2879 active+clean]  sending ack on repgather(0x384e2900 173'2881 rep_tid=236051 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141526 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 1347584~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0x32eef400
2014-06-12 14:46:11.568258 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2880 crt=0'0 lcod 173'2879 mlcod 173'2879 active+clean] repop_all_committed: repop tid 236051 all committed 
2014-06-12 14:46:11.568268 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2879 active+clean] eval_repop repgather(0x384e2900 173'2881 rep_tid=236051 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141526 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 1347584~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.568281 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2879 active+clean] log_op_stats osd_op(client.4309.0:141526 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 1347584~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 32.742677 lat 32.742700
2014-06-12 14:46:11.568295 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2879 active+clean] publish_stats_to_osd 173:3267
2014-06-12 14:46:11.568306 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2879 active+clean]  sending commit on repgather(0x384e2900 173'2881 rep_tid=236051 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141526 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 1347584~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0x25910280
2014-06-12 14:46:11.568332 7f9d64bda700  5 -- op tracker -- , seq: 747041, time: 2014-06-12 14:46:11.568327, event: commit_sent, op: osd_op(client.4309.0:141526 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 1347584~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.568338 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2879 active+clean]  requeue_ops 
2014-06-12 14:46:11.568347 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2880 active+clean]  removing repgather(0x384e2900 173'2881 rep_tid=236051 committed?=1 applied?=1 op=osd_op(client.4309.0:141526 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 1347584~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.568360 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2880 active+clean]    q front is repgather(0x384e2900 173'2881 rep_tid=236051 committed?=1 applied?=1 op=osd_op(client.4309.0:141526 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 1347584~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.568373 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2880 active+clean] remove_repop repgather(0x384e2900 173'2881 rep_tid=236051 committed?=1 applied?=1 op=osd_op(client.4309.0:141526 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 1347584~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.568386 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2880 active+clean]  requeue_ops 
2014-06-12 14:46:11.568402 7f9d64bda700  5 -- op tracker -- , seq: 747041, time: 2014-06-12 14:46:11.568397, event: done, op: osd_op(client.4309.0:141526 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 1347584~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.568414 7f9d64bda700 10 osd.2 173 dequeue_op 0x2519d400 finish
2014-06-12 14:46:11.568422 7f9d64bda700  5 -- op tracker -- , seq: 747663, time: 2014-06-12 14:46:11.568418, event: done, op: osd_sub_op_reply(client.4309.0:141526 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.568431 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:11.568434 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.568476 7f9d6abe6700  5 -- op tracker -- , seq: 746965, time: 2014-06-12 14:46:11.568108, event: commit_sent, op: osd_op(client.4306.0:144050 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 3141632~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.568484 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3085 active+clean]  requeue_ops 
2014-06-12 14:46:11.568497 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3086 active+clean]  removing repgather(0x15acefc0 173'3087 rep_tid=236018 committed?=1 applied?=1 op=osd_op(client.4306.0:144050 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 3141632~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.568510 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3086 active+clean]    q front is repgather(0x15acefc0 173'3087 rep_tid=236018 committed?=1 applied?=1 op=osd_op(client.4306.0:144050 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 3141632~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.568523 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3086 active+clean] remove_repop repgather(0x15acefc0 173'3087 rep_tid=236018 committed?=1 applied?=1 op=osd_op(client.4306.0:144050 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 3141632~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.568535 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3086 active+clean]  requeue_ops 
2014-06-12 14:46:11.568552 7f9d6abe6700  5 -- op tracker -- , seq: 746965, time: 2014-06-12 14:46:11.568548, event: done, op: osd_op(client.4306.0:144050 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 3141632~1032192] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.568565 7f9d6abe6700 10 osd.2 173 dequeue_op 0x59dc600 finish
2014-06-12 14:46:11.568572 7f9d6abe6700  5 -- op tracker -- , seq: 747664, time: 2014-06-12 14:46:11.568568, event: done, op: osd_sub_op_reply(client.4306.0:144050 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.568583 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.568585 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.578629 7f9d56432700  5 -- op tracker -- , seq: 747665, time: 2014-06-12 14:46:11.572011, event: header_read, op: osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.578649 7f9d56432700  5 -- op tracker -- , seq: 747665, time: 2014-06-12 14:46:11.572013, event: throttled, op: osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.578658 7f9d56432700  5 -- op tracker -- , seq: 747665, time: 2014-06-12 14:46:11.578566, event: all_read, op: osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.578666 7f9d56432700  5 -- op tracker -- , seq: 747665, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.578672 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:11.578680 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:11.578690 7f9d56432700 15 osd.2 173 enqueue_op 0x5abe800 prio 127 cost 516899 latency 0.006679 osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.578744 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.578754 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x5abe800 prio 127 cost 516899 latency 0.006742 osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2477 (0'0,173'2477] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2476 active]
2014-06-12 14:46:11.578778 7f9d6a3e5700  5 -- op tracker -- , seq: 747665, time: 2014-06-12 14:46:11.578774, event: reached_pg, op: osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.578787 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2477 (0'0,173'2477] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2476 active] handle_message: 0x5abe800
2014-06-12 14:46:11.578799 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2477 (0'0,173'2477] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2476 active] sub_op_modify trans 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 v 173'2478 (transaction) 174
2014-06-12 14:46:11.578814 7f9d6a3e5700  5 -- op tracker -- , seq: 747665, time: 2014-06-12 14:46:11.578811, event: started, op: osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.578831 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2477 (0'0,173'2477] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2476 active] append_log log((0'0,173'2477], crt=0'0) [173'2478 (173'2477) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144992 2014-06-12 08:44:57.496771]
2014-06-12 14:46:11.578853 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2478 (0'0,173'2478] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2476 active] add_log_entry 173'2478 (173'2477) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144992 2014-06-12 08:44:57.496771
2014-06-12 14:46:11.578874 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2478 (0'0,173'2478] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2476 active] append_log  adding 1 keys
2014-06-12 14:46:11.578907 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2478, trimmed: 
2014-06-12 14:46:11.578928 7f9d6a3e5700  5 -- op tracker -- , seq: 747665, time: 2014-06-12 14:46:11.578924, event: started, op: osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.578937 7f9d6a3e5700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:11.578941 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:11.578942 7f9d6a3e5700  5 queue_op 0x142a8000 seq 2291 osr(3.2b 0x276cd20) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:46:11.578949 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2478 (0'0,173'2478] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2476 active] do_sub_op osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.578962 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x5abe800 finish
2014-06-12 14:46:11.578965 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.578967 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.578976 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:46:11.578985 7f9d7c966700  5 _do_op 0x142a8000 seq 2291 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:46:11.578989 7f9d7c966700 10 _do_transaction on 0x61cf430
2014-06-12 14:46:11.578995 7f9d7c966700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:11.579114 7f9d7c966700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:11.579131 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:11.579182 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:11.579200 7f9d7c966700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:11.579205 7f9d7c966700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:46:11.579211 7f9d7c966700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:11.579348 7f9d7c966700 15 _write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 2863104~516096
2014-06-12 14:46:11.579531 7f9d7c966700 10 lookup_strip_header done  cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:11.579553 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:11.581802 7f9d7c966700 10 _generic_write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 2863104~516096 = 0
2014-06-12 14:46:11.581866 7f9d7c966700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:11.581889 7f9d7c966700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:11.581897 7f9d7c966700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:11.581905 7f9d7c966700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:11.582112 7f9d7c966700 20 set_header setting 17866 cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 parent seq 0
2014-06-12 14:46:11.582212 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:11.582224 7f9d7c966700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:46:11.582235 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:11.584850 7f9d7c966700 10 _do_op 0x142a8000 seq 2291 r = 0, finisher 0x88d93c0 0
2014-06-12 14:46:11.584868 7f9d7c966700 10 _finish_op 0x142a8000 seq 2291 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:11.584875 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:46:11.584882 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:11.584890 7f9d7715b700  5 -- op tracker -- , seq: 747665, time: 2014-06-12 14:46:11.584879, event: commit_sent, op: osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.585204 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2478 (0'0,173'2478] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2476 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:11.585233 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:11.585637 7f9d7795c700  5 -- op tracker -- , seq: 747665, time: 2014-06-12 14:46:11.585630, event: sub_op_applied, op: osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.585648 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2478 (0'0,173'2478] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2477 active] sub_op_modify_applied on 0x61cf340 op osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:11.585664 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2478 (0'0,173'2478] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2477 active] op_applied on primary on version 173'2478
2014-06-12 14:46:11.585673 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2478 (0'0,173'2478] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2477 active] op_applied on replica on version 173'2478
2014-06-12 14:46:11.585697 7f9d7795c700  5 -- op tracker -- , seq: 747665, time: 2014-06-12 14:46:11.585693, event: done, op: osd_sub_op(client.4314.0:144992 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:11.594604 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:11.596068 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:11.873395 7f9d56432700  5 -- op tracker -- , seq: 747666, time: 2014-06-12 14:46:11.873334, event: header_read, op: osd_sub_op_reply(client.4306.0:144051 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.873409 7f9d56432700  5 -- op tracker -- , seq: 747666, time: 2014-06-12 14:46:11.873336, event: throttled, op: osd_sub_op_reply(client.4306.0:144051 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.873414 7f9d56432700  5 -- op tracker -- , seq: 747666, time: 2014-06-12 14:46:11.873349, event: all_read, op: osd_sub_op_reply(client.4306.0:144051 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.873419 7f9d56432700  5 -- op tracker -- , seq: 747666, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144051 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.873424 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144051 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.873428 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:11.873435 7f9d56432700 15 osd.2 173 enqueue_op 0x5abe600 prio 196 cost 0 latency 0.000101 osd_sub_op_reply(client.4306.0:144051 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.873461 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.873470 7f9d66bde700 10 osd.2 173 dequeue_op 0x5abe600 prio 196 cost 0 latency 0.000136 osd_sub_op_reply(client.4306.0:144051 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3086 active+clean]
2014-06-12 14:46:11.873497 7f9d66bde700  5 -- op tracker -- , seq: 747666, time: 2014-06-12 14:46:11.873493, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144051 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.873506 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3086 active+clean] handle_message: 0x5abe600
2014-06-12 14:46:11.873522 7f9d66bde700  5 -- op tracker -- , seq: 747666, time: 2014-06-12 14:46:11.873519, event: started, op: osd_sub_op_reply(client.4306.0:144051 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.873534 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3086 active+clean] sub_op_modify_reply: tid 236029 op  ack_type 4 from 26
2014-06-12 14:46:11.873560 7f9d66bde700  5 -- op tracker -- , seq: 746994, time: 2014-06-12 14:46:11.873551, event: sub_op_commit_rec, op: osd_op(client.4306.0:144051 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 4173824~20480] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.873570 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3086 active+clean] repop_all_applied: repop tid 236029 all applied 
2014-06-12 14:46:11.873585 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3086 active+clean] eval_repop repgather(0xe8338c0 173'3088 rep_tid=236029 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144051 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 4173824~20480] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.873604 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3086 active+clean]  sending ack on repgather(0xe8338c0 173'3088 rep_tid=236029 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144051 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 4173824~20480] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x3b82b680
2014-06-12 14:46:11.873659 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3087 crt=0'0 lcod 173'3086 mlcod 173'3086 active+clean] repop_all_committed: repop tid 236029 all committed 
2014-06-12 14:46:11.873672 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3086 active+clean] eval_repop repgather(0xe8338c0 173'3088 rep_tid=236029 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144051 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 4173824~20480] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.873698 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3086 active+clean] log_op_stats osd_op(client.4306.0:144051 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 4173824~20480] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4 inb 20759 outb 0 rlat 33.544344 lat 33.544384
2014-06-12 14:46:11.873724 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3086 active+clean] publish_stats_to_osd 173:3554
2014-06-12 14:46:11.873739 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3086 active+clean]  sending commit on repgather(0xe8338c0 173'3088 rep_tid=236029 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144051 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 4173824~20480] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4) 0x31657400
2014-06-12 14:46:11.873787 7f9d66bde700  5 -- op tracker -- , seq: 746994, time: 2014-06-12 14:46:11.873781, event: commit_sent, op: osd_op(client.4306.0:144051 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 4173824~20480] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.873797 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3086 active+clean]  requeue_ops 
2014-06-12 14:46:11.873810 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3087 active+clean]  removing repgather(0xe8338c0 173'3088 rep_tid=236029 committed?=1 applied?=1 op=osd_op(client.4306.0:144051 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 4173824~20480] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.873827 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3087 active+clean]    q front is repgather(0xe8338c0 173'3088 rep_tid=236029 committed?=1 applied?=1 op=osd_op(client.4306.0:144051 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 4173824~20480] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.873843 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3087 active+clean] remove_repop repgather(0xe8338c0 173'3088 rep_tid=236029 committed?=1 applied?=1 op=osd_op(client.4306.0:144051 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 4173824~20480] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.873859 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3087 active+clean]  requeue_ops 
2014-06-12 14:46:11.873897 7f9d66bde700  5 -- op tracker -- , seq: 746994, time: 2014-06-12 14:46:11.873891, event: done, op: osd_op(client.4306.0:144051 rbd_data.10a96b8b4567.0000000000000f91 [set-alloc-hint object_size 4194304 write_size 4194304,write 4173824~20480] 3.67a6e1eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.873925 7f9d66bde700 10 osd.2 173 dequeue_op 0x5abe600 finish
2014-06-12 14:46:11.873934 7f9d66bde700  5 -- op tracker -- , seq: 747666, time: 2014-06-12 14:46:11.873930, event: done, op: osd_sub_op_reply(client.4306.0:144051 3.1eb 67a6e1eb/rbd_data.10a96b8b4567.0000000000000f91/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.873950 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.873955 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.875645 7f9d56432700  5 -- op tracker -- , seq: 747667, time: 2014-06-12 14:46:11.875601, event: header_read, op: osd_sub_op_reply(client.4303.0:144622 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.875655 7f9d56432700  5 -- op tracker -- , seq: 747667, time: 2014-06-12 14:46:11.875603, event: throttled, op: osd_sub_op_reply(client.4303.0:144622 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.875663 7f9d56432700  5 -- op tracker -- , seq: 747667, time: 2014-06-12 14:46:11.875614, event: all_read, op: osd_sub_op_reply(client.4303.0:144622 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.875670 7f9d56432700  5 -- op tracker -- , seq: 747667, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4303.0:144622 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.875675 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4303.0:144622 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.875680 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:11.875687 7f9d56432700 15 osd.2 173 enqueue_op 0x5abfa00 prio 196 cost 0 latency 0.000086 osd_sub_op_reply(client.4303.0:144622 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.875738 7f9d56432700  5 -- op tracker -- , seq: 747668, time: 2014-06-12 14:46:11.875707, event: header_read, op: osd_sub_op_reply(client.4303.0:144623 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.875748 7f9d56432700  5 -- op tracker -- , seq: 747668, time: 2014-06-12 14:46:11.875707, event: throttled, op: osd_sub_op_reply(client.4303.0:144623 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.875756 7f9d56432700  5 -- op tracker -- , seq: 747668, time: 2014-06-12 14:46:11.875715, event: all_read, op: osd_sub_op_reply(client.4303.0:144623 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.875764 7f9d56432700  5 -- op tracker -- , seq: 747668, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4303.0:144623 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.875768 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4303.0:144623 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.875773 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:11.875779 7f9d56432700 15 osd.2 173 enqueue_op 0x5abff00 prio 196 cost 0 latency 0.000072 osd_sub_op_reply(client.4303.0:144623 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.875815 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:11.875826 7f9d653db700 10 osd.2 173 dequeue_op 0x5abfa00 prio 196 cost 0 latency 0.000225 osd_sub_op_reply(client.4303.0:144622 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3087 active+clean]
2014-06-12 14:46:11.875853 7f9d653db700  5 -- op tracker -- , seq: 747667, time: 2014-06-12 14:46:11.875849, event: reached_pg, op: osd_sub_op_reply(client.4303.0:144622 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.875861 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3087 active+clean] handle_message: 0x5abfa00
2014-06-12 14:46:11.875878 7f9d653db700  5 -- op tracker -- , seq: 747667, time: 2014-06-12 14:46:11.875874, event: started, op: osd_sub_op_reply(client.4303.0:144622 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.875885 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3087 active+clean] sub_op_modify_reply: tid 236061 op  ack_type 4 from 26
2014-06-12 14:46:11.875917 7f9d653db700  5 -- op tracker -- , seq: 747068, time: 2014-06-12 14:46:11.875908, event: sub_op_commit_rec, op: osd_op(client.4303.0:144622 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~28672] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.875924 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3087 active+clean] repop_all_applied: repop tid 236061 all applied 
2014-06-12 14:46:11.875936 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3087 active+clean] eval_repop repgather(0x32e7a3c0 173'3089 rep_tid=236061 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144622 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~28672] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.875952 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3087 active+clean]  sending ack on repgather(0x32e7a3c0 173'3089 rep_tid=236061 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144622 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~28672] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x30af6c80
2014-06-12 14:46:11.875982 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3088 crt=0'0 lcod 173'3087 mlcod 173'3087 active+clean] repop_all_committed: repop tid 236061 all committed 
2014-06-12 14:46:11.875994 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3087 active+clean] eval_repop repgather(0x32e7a3c0 173'3089 rep_tid=236061 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144622 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~28672] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.876010 7f9d56432700  5 -- op tracker -- , seq: 747669, time: 2014-06-12 14:46:11.875971, event: header_read, op: osd_sub_op_reply(client.4303.0:144624 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.876010 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3087 active+clean] log_op_stats osd_op(client.4303.0:144622 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~28672] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4 inb 28951 outb 0 rlat 32.745667 lat 32.745695
2014-06-12 14:46:11.876020 7f9d56432700  5 -- op tracker -- , seq: 747669, time: 2014-06-12 14:46:11.875972, event: throttled, op: osd_sub_op_reply(client.4303.0:144624 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.876027 7f9d56432700  5 -- op tracker -- , seq: 747669, time: 2014-06-12 14:46:11.875982, event: all_read, op: osd_sub_op_reply(client.4303.0:144624 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.876027 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3087 active+clean] publish_stats_to_osd 173:3555
2014-06-12 14:46:11.876034 7f9d56432700  5 -- op tracker -- , seq: 747669, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4303.0:144624 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.876038 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4303.0:144624 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:11.876042 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:11.876038 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3087 active+clean]  sending commit on repgather(0x32e7a3c0 173'3089 rep_tid=236061 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144622 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~28672] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x3228c280
2014-06-12 14:46:11.876048 7f9d56432700 15 osd.2 173 enqueue_op 0x5abee00 prio 196 cost 0 latency 0.000077 osd_sub_op_reply(client.4303.0:144624 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:11.876070 7f9d653db700  5 -- op tracker -- , seq: 747068, time: 2014-06-12 14:46:11.876064, event: commit_sent, op: osd_op(client.4303.0:144622 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~28672] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.876073 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:11.876078 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3087 active+clean]  requeue_ops 
2014-06-12 14:46:11.876089 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3088 active+clean]  removing repgather(0x32e7a3c0 173'3089 rep_tid=236061 committed?=1 applied?=1 op=osd_op(client.4303.0:144622 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~28672] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.876104 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3088 active+clean]    q front is repgather(0x32e7a3c0 173'3089 rep_tid=236061 committed?=1 applied?=1 op=osd_op(client.4303.0:144622 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~28672] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.876120 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3088 active+clean] remove_repop repgather(0x32e7a3c0 173'3089 rep_tid=236061 committed?=1 applied?=1 op=osd_op(client.4303.0:144622 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~28672] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.876134 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3088 active+clean]  requeue_ops 
2014-06-12 14:46:11.876156 7f9d653db700  5 -- op tracker -- , seq: 747068, time: 2014-06-12 14:46:11.876150, event: done, op: osd_op(client.4303.0:144622 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~28672] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.876182 7f9d653db700 10 osd.2 173 dequeue_op 0x5abfa00 finish
2014-06-12 14:46:11.876194 7f9d653db700  5 -- op tracker -- , seq: 747667, time: 2014-06-12 14:46:11.876190, event: done, op: osd_sub_op_reply(client.4303.0:144622 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.876342 7f9d69be4700 10 osd.2 173 dequeue_op 0x5abff00 prio 196 cost 0 latency 0.000635 osd_sub_op_reply(client.4303.0:144623 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3088 active+clean]
2014-06-12 14:46:11.877064 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:11.877067 7f9d69be4700  5 -- op tracker -- , seq: 747668, time: 2014-06-12 14:46:11.877063, event: reached_pg, op: osd_sub_op_reply(client.4303.0:144623 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.877073 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:11.877075 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3088 active+clean] handle_message: 0x5abff00
2014-06-12 14:46:11.877090 7f9d69be4700  5 -- op tracker -- , seq: 747668, time: 2014-06-12 14:46:11.877087, event: started, op: osd_sub_op_reply(client.4303.0:144623 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.877096 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3088 active+clean] sub_op_modify_reply: tid 236062 op  ack_type 4 from 26
2014-06-12 14:46:11.877114 7f9d69be4700  5 -- op tracker -- , seq: 747070, time: 2014-06-12 14:46:11.877107, event: sub_op_commit_rec, op: osd_op(client.4303.0:144623 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 28672~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.877121 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3088 active+clean] repop_all_applied: repop tid 236062 all applied 
2014-06-12 14:46:11.877131 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3088 active+clean] eval_repop repgather(0x13b43bc0 173'3090 rep_tid=236062 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144623 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 28672~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.877146 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3088 active+clean]  sending ack on repgather(0x13b43bc0 173'3090 rep_tid=236062 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144623 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 28672~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x1080d400
2014-06-12 14:46:11.877175 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3089 crt=0'0 lcod 173'3088 mlcod 173'3088 active+clean] repop_all_committed: repop tid 236062 all committed 
2014-06-12 14:46:11.877185 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3088 active+clean] eval_repop repgather(0x13b43bc0 173'3090 rep_tid=236062 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144623 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 28672~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.877201 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3088 active+clean] log_op_stats osd_op(client.4303.0:144623 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 28672~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 32.738196 lat 32.738222
2014-06-12 14:46:11.877220 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3088 active+clean] publish_stats_to_osd 173:3556
2014-06-12 14:46:11.877232 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3088 active+clean]  sending commit on repgather(0x13b43bc0 173'3090 rep_tid=236062 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144623 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 28672~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x10149b80
2014-06-12 14:46:11.877261 7f9d69be4700  5 -- op tracker -- , seq: 747070, time: 2014-06-12 14:46:11.877255, event: commit_sent, op: osd_op(client.4303.0:144623 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 28672~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.877268 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3088 active+clean]  requeue_ops 
2014-06-12 14:46:11.877278 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3089 active+clean]  removing repgather(0x13b43bc0 173'3090 rep_tid=236062 committed?=1 applied?=1 op=osd_op(client.4303.0:144623 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 28672~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.877292 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3089 active+clean]    q front is repgather(0x13b43bc0 173'3090 rep_tid=236062 committed?=1 applied?=1 op=osd_op(client.4303.0:144623 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 28672~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.877305 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3089 active+clean] remove_repop repgather(0x13b43bc0 173'3090 rep_tid=236062 committed?=1 applied?=1 op=osd_op(client.4303.0:144623 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 28672~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.877319 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3089 active+clean]  requeue_ops 
2014-06-12 14:46:11.877337 7f9d69be4700  5 -- op tracker -- , seq: 747070, time: 2014-06-12 14:46:11.877332, event: done, op: osd_op(client.4303.0:144623 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 28672~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.877352 7f9d69be4700 10 osd.2 173 dequeue_op 0x5abff00 finish
2014-06-12 14:46:11.877362 7f9d69be4700  5 -- op tracker -- , seq: 747668, time: 2014-06-12 14:46:11.877358, event: done, op: osd_sub_op_reply(client.4303.0:144623 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.877372 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:11.877375 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.877419 7f9d653db700 10 osd.2 173 dequeue_op 0x5abee00 prio 196 cost 0 latency 0.001448 osd_sub_op_reply(client.4303.0:144624 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3089 active+clean]
2014-06-12 14:46:11.877440 7f9d653db700  5 -- op tracker -- , seq: 747669, time: 2014-06-12 14:46:11.877437, event: reached_pg, op: osd_sub_op_reply(client.4303.0:144624 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.877447 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3089 active+clean] handle_message: 0x5abee00
2014-06-12 14:46:11.877461 7f9d653db700  5 -- op tracker -- , seq: 747669, time: 2014-06-12 14:46:11.877458, event: started, op: osd_sub_op_reply(client.4303.0:144624 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.877466 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3089 active+clean] sub_op_modify_reply: tid 236063 op  ack_type 4 from 26
2014-06-12 14:46:11.877485 7f9d653db700  5 -- op tracker -- , seq: 747076, time: 2014-06-12 14:46:11.877479, event: sub_op_commit_rec, op: osd_op(client.4303.0:144624 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.877492 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3089 active+clean] repop_all_applied: repop tid 236063 all applied 
2014-06-12 14:46:11.877502 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3089 active+clean] eval_repop repgather(0x15ace180 173'3091 rep_tid=236063 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144624 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.877518 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3089 active+clean]  sending ack on repgather(0x15ace180 173'3091 rep_tid=236063 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144624 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x1b774500
2014-06-12 14:46:11.877545 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3090 crt=0'0 lcod 173'3089 mlcod 173'3089 active+clean] repop_all_committed: repop tid 236063 all committed 
2014-06-12 14:46:11.877556 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3089 active+clean] eval_repop repgather(0x15ace180 173'3091 rep_tid=236063 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144624 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:11.877573 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3089 active+clean] log_op_stats osd_op(client.4303.0:144624 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 32.660766 lat 32.660794
2014-06-12 14:46:11.877589 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3089 active+clean] publish_stats_to_osd 173:3557
2014-06-12 14:46:11.877601 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3089 active+clean]  sending commit on repgather(0x15ace180 173'3091 rep_tid=236063 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144624 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x35714000
2014-06-12 14:46:11.877640 7f9d653db700  5 -- op tracker -- , seq: 747076, time: 2014-06-12 14:46:11.877635, event: commit_sent, op: osd_op(client.4303.0:144624 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.877647 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3089 active+clean]  requeue_ops 
2014-06-12 14:46:11.877657 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3090 active+clean]  removing repgather(0x15ace180 173'3091 rep_tid=236063 committed?=1 applied?=1 op=osd_op(client.4303.0:144624 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.877672 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3090 active+clean]    q front is repgather(0x15ace180 173'3091 rep_tid=236063 committed?=1 applied?=1 op=osd_op(client.4303.0:144624 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.877685 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3090 active+clean] remove_repop repgather(0x15ace180 173'3091 rep_tid=236063 committed?=1 applied?=1 op=osd_op(client.4303.0:144624 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:11.877699 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3090 active+clean]  requeue_ops 
2014-06-12 14:46:11.877717 7f9d653db700  5 -- op tracker -- , seq: 747076, time: 2014-06-12 14:46:11.877712, event: done, op: osd_op(client.4303.0:144624 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:11.877730 7f9d653db700 10 osd.2 173 dequeue_op 0x5abee00 finish
2014-06-12 14:46:11.877738 7f9d653db700  5 -- op tracker -- , seq: 747669, time: 2014-06-12 14:46:11.877735, event: done, op: osd_sub_op_reply(client.4303.0:144624 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:11.877748 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:11.877751 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:11.877790 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:12.006106 7f9d56432700  5 -- op tracker -- , seq: 747670, time: 2014-06-12 14:46:12.006044, event: header_read, op: osd_sub_op_reply(client.4309.0:141527 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:12.006118 7f9d56432700  5 -- op tracker -- , seq: 747670, time: 2014-06-12 14:46:12.006047, event: throttled, op: osd_sub_op_reply(client.4309.0:141527 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:12.006126 7f9d56432700  5 -- op tracker -- , seq: 747670, time: 2014-06-12 14:46:12.006062, event: all_read, op: osd_sub_op_reply(client.4309.0:141527 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:12.006138 7f9d56432700  5 -- op tracker -- , seq: 747670, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141527 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:12.006145 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141527 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:12.006151 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:12.006160 7f9d56432700 15 osd.2 173 enqueue_op 0x5abe100 prio 196 cost 0 latency 0.000116 osd_sub_op_reply(client.4309.0:141527 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:12.006190 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:12.006199 7f9d68be2700 10 osd.2 173 dequeue_op 0x5abe100 prio 196 cost 0 latency 0.000154 osd_sub_op_reply(client.4309.0:141527 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 pg pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2880 active+clean]
2014-06-12 14:46:12.006220 7f9d68be2700  5 -- op tracker -- , seq: 747670, time: 2014-06-12 14:46:12.006217, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141527 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:12.006229 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2880 active+clean] handle_message: 0x5abe100
2014-06-12 14:46:12.006247 7f9d68be2700  5 -- op tracker -- , seq: 747670, time: 2014-06-12 14:46:12.006244, event: started, op: osd_sub_op_reply(client.4309.0:141527 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:12.006253 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2880 active+clean] sub_op_modify_reply: tid 236071 op  ack_type 4 from 26
2014-06-12 14:46:12.006273 7f9d68be2700  5 -- op tracker -- , seq: 747094, time: 2014-06-12 14:46:12.006265, event: sub_op_commit_rec, op: osd_op(client.4309.0:141527 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 2379776~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:12.006280 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2880 active+clean] repop_all_applied: repop tid 236071 all applied 
2014-06-12 14:46:12.006290 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2880 active+clean] eval_repop repgather(0x22b79d40 173'2882 rep_tid=236071 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141527 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 2379776~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:12.006306 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2880 active+clean]  sending ack on repgather(0x22b79d40 173'2882 rep_tid=236071 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141527 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 2379776~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0xfdb2780
2014-06-12 14:46:12.006338 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2881 crt=0'0 lcod 173'2880 mlcod 173'2880 active+clean] repop_all_committed: repop tid 236071 all committed 
2014-06-12 14:46:12.006348 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2880 active+clean] eval_repop repgather(0x22b79d40 173'2882 rep_tid=236071 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141527 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 2379776~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:12.006363 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2880 active+clean] log_op_stats osd_op(client.4309.0:141527 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 2379776~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 32.258522 lat 32.258548
2014-06-12 14:46:12.006381 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2880 active+clean] publish_stats_to_osd 173:3268
2014-06-12 14:46:12.006398 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2880 active+clean]  sending commit on repgather(0x22b79d40 173'2882 rep_tid=236071 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141527 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 2379776~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0x16cd3900
2014-06-12 14:46:12.006430 7f9d68be2700  5 -- op tracker -- , seq: 747094, time: 2014-06-12 14:46:12.006425, event: commit_sent, op: osd_op(client.4309.0:141527 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 2379776~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:12.006439 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2880 active+clean]  requeue_ops 
2014-06-12 14:46:12.006450 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2881 active+clean]  removing repgather(0x22b79d40 173'2882 rep_tid=236071 committed?=1 applied?=1 op=osd_op(client.4309.0:141527 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 2379776~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:12.006462 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2881 active+clean]    q front is repgather(0x22b79d40 173'2882 rep_tid=236071 committed?=1 applied?=1 op=osd_op(client.4309.0:141527 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 2379776~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:12.006479 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2881 active+clean] remove_repop repgather(0x22b79d40 173'2882 rep_tid=236071 committed?=1 applied?=1 op=osd_op(client.4309.0:141527 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 2379776~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:12.006494 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2881 active+clean]  requeue_ops 
2014-06-12 14:46:12.006511 7f9d68be2700  5 -- op tracker -- , seq: 747094, time: 2014-06-12 14:46:12.006507, event: done, op: osd_op(client.4309.0:141527 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 2379776~1032192] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:12.006538 7f9d68be2700 10 osd.2 173 dequeue_op 0x5abe100 finish
2014-06-12 14:46:12.006547 7f9d68be2700  5 -- op tracker -- , seq: 747670, time: 2014-06-12 14:46:12.006543, event: done, op: osd_sub_op_reply(client.4309.0:141527 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:12.006556 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:12.006559 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:12.021731 7f9d56432700  5 -- op tracker -- , seq: 747671, time: 2014-06-12 14:46:12.020578, event: header_read, op: osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:12.021749 7f9d56432700  5 -- op tracker -- , seq: 747671, time: 2014-06-12 14:46:12.020582, event: throttled, op: osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:12.021758 7f9d56432700  5 -- op tracker -- , seq: 747671, time: 2014-06-12 14:46:12.021686, event: all_read, op: osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:12.021766 7f9d56432700  5 -- op tracker -- , seq: 747671, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:12.021775 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:12.021784 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:12.021795 7f9d56432700 15 osd.2 173 enqueue_op 0x5abe500 prio 127 cost 516899 latency 0.001217 osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:12.023373 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:12.023391 7f9d633d7700 10 osd.2 173 dequeue_op 0x5abe500 prio 127 cost 516899 latency 0.002812 osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2478 (0'0,173'2478] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2477 active]
2014-06-12 14:46:12.023420 7f9d633d7700  5 -- op tracker -- , seq: 747671, time: 2014-06-12 14:46:12.023416, event: reached_pg, op: osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:12.023432 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2478 (0'0,173'2478] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2477 active] handle_message: 0x5abe500
2014-06-12 14:46:12.023446 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2478 (0'0,173'2478] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2477 active] sub_op_modify trans 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 v 173'2479 (transaction) 174
2014-06-12 14:46:12.023462 7f9d633d7700  5 -- op tracker -- , seq: 747671, time: 2014-06-12 14:46:12.023459, event: started, op: osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:12.023475 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2478 (0'0,173'2478] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2477 active] append_log log((0'0,173'2478], crt=0'0) [173'2479 (173'2478) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144993 2014-06-12 08:44:57.650725]
2014-06-12 14:46:12.023491 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2479 (0'0,173'2479] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2477 active] add_log_entry 173'2479 (173'2478) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144993 2014-06-12 08:44:57.650725
2014-06-12 14:46:12.023507 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2479 (0'0,173'2479] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2477 active] append_log  adding 1 keys
2014-06-12 14:46:12.023537 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2479, trimmed: 
2014-06-12 14:46:12.023553 7f9d633d7700  5 -- op tracker -- , seq: 747671, time: 2014-06-12 14:46:12.023550, event: started, op: osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:12.023560 7f9d633d7700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:12.023563 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:12.023564 7f9d633d7700  5 queue_op 0x1ebb14a0 seq 2292 osr(3.2b 0x276cd20) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:46:12.023574 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:46:12.023582 7f9d81970700  5 _do_op 0x1ebb14a0 seq 2292 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:46:12.023570 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2479 (0'0,173'2479] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2477 active] do_sub_op osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:12.023585 7f9d633d7700 10 osd.2 173 dequeue_op 0x5abe500 finish
2014-06-12 14:46:12.023587 7f9d81970700 10 _do_transaction on 0x5cb40f0
2014-06-12 14:46:12.023587 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:12.023590 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:12.023594 7f9d81970700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:12.023712 7f9d81970700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:12.023732 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:12.023781 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:12.023803 7f9d81970700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:12.023809 7f9d81970700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:46:12.023815 7f9d81970700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:12.023928 7f9d81970700 15 _write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 3379200~516096
2014-06-12 14:46:12.024081 7f9d81970700 10 lookup_strip_header done  cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:12.024239 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:12.026516 7f9d81970700 10 _generic_write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 3379200~516096 = 0
2014-06-12 14:46:12.026578 7f9d81970700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:12.026594 7f9d81970700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:12.026601 7f9d81970700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:12.026610 7f9d81970700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:12.026828 7f9d81970700 20 set_header setting 17866 cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 parent seq 0
2014-06-12 14:46:12.026913 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:12.026930 7f9d81970700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:46:12.026941 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:12.030686 7f9d81970700 10 _do_op 0x1ebb14a0 seq 2292 r = 0, finisher 0x6ce5800 0
2014-06-12 14:46:12.030729 7f9d7715b700  5 -- op tracker -- , seq: 747671, time: 2014-06-12 14:46:12.030718, event: commit_sent, op: osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:12.030739 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2479 (0'0,173'2479] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2477 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:12.030769 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:12.030797 7f9d81970700 10 _finish_op 0x1ebb14a0 seq 2292 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:12.030806 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:46:12.030815 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:12.030832 7f9d7795c700  5 -- op tracker -- , seq: 747671, time: 2014-06-12 14:46:12.030826, event: sub_op_applied, op: osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:12.030840 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2479 (0'0,173'2479] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active] sub_op_modify_applied on 0x5cb4000 op osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:12.030855 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2479 (0'0,173'2479] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active] op_applied on primary on version 173'2479
2014-06-12 14:46:12.030863 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2479 (0'0,173'2479] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active] op_applied on replica on version 173'2479
2014-06-12 14:46:12.030889 7f9d7795c700  5 -- op tracker -- , seq: 747671, time: 2014-06-12 14:46:12.030885, event: done, op: osd_sub_op(client.4314.0:144993 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:12.041394 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:12.041397 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:12.041399 7f9d74bfa700 20 osd.2 173 _dispatch 0x8d87040 ping v1
2014-06-12 14:46:12.041401 7f9d74bfa700 10 osd.2 173 ping from client.4317
2014-06-12 14:46:12.041403 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:12.041405 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:12.153294 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:12.366952 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:12.417056 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:12.417556 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:12.446462 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:12.446555 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:12.471362 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:12.471383 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:46:12.471386 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:46:12.471390 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:46:12.471394 7f9d84175700 10 osd.2 173 send_pg_stats - 7 pgs updated
2014-06-12 14:46:12.471448 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:12.471452 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:12.471453 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:12.471457 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 17; oldest is 33.222537 seconds old
2014-06-12 14:46:12.473550 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:12.497769 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:12.497771 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:12.497773 7f9d74bfa700 20 osd.2 173 _dispatch 0x7435180 pg_stats_ack(7 pgs tid 4848) v1
2014-06-12 14:46:12.497797 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:46:12.497805 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:12.497807 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:12.894052 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:12.938109 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:12.939746 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:12.946454 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:12.946484 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:12.949214 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:12.973216 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.045333 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.135418 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:13.135554 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:13.137796 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.143032 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.172900 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.315238 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.320063 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.338683 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:13.339080 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:13.381821 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.394105 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.407143 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.412339 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.412552 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.420374 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.424479 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.425420 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.425828 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.433758 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.434546 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.435095 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.438542 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.444480 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.448923 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.452921 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.453991 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.471523 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:13.471558 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:13.471579 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 13.27 >= max 0.5 = no, load too high
2014-06-12 14:46:13.471594 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:13.471596 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92621 < max (604800 seconds)
2014-06-12 14:46:13.471617 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:13.471619 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:13.471620 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:13.471621 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 17; oldest is 34.222702 seconds old
2014-06-12 14:46:13.517687 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:13.517791 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:13.568217 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.568494 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.568641 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.579030 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.584969 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:13.874020 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.877436 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.877810 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.877852 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:13.893292 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:13.893780 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:13.914999 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19410 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,8,5])
2014-06-12 14:46:13.915023 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19410 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,8,5])
2014-06-12 14:46:13.915483 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:13.915530 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:13.915558 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:13.915581 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:13.915600 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:13.915618 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:13.915636 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:13.915657 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:13.915684 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:13.915718 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:13.915769 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:13.915802 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:13.915821 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:13.915839 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:13.915855 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:13.915872 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:13.915888 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:13.915906 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:13.915922 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:13.915939 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:13.915956 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:13.917716 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:14.006624 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.014108 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:14.014339 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:14.023654 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.030874 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:14.153393 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:14.367052 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:14.462071 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:14.462143 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:14.471694 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:14.471727 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:14.471733 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:14.471737 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:14.471741 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 17; oldest is 35.222821 seconds old
2014-06-12 14:46:14.471784 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:14.670339 7f9d5b6bc700  5 -- op tracker -- , seq: 747672, time: 2014-06-12 14:46:14.667851, event: header_read, op: osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.670355 7f9d5b6bc700  5 -- op tracker -- , seq: 747672, time: 2014-06-12 14:46:14.667854, event: throttled, op: osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.670364 7f9d5b6bc700  5 -- op tracker -- , seq: 747672, time: 2014-06-12 14:46:14.670292, event: all_read, op: osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.670373 7f9d5b6bc700  5 -- op tracker -- , seq: 747672, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.670379 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:14.670387 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:14.670396 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x59dd700 prio 127 cost 602919 latency 0.002544 osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.670430 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.670440 7f9d623d5700 10 osd.2 173 dequeue_op 0x59dd700 prio 127 cost 602919 latency 0.002588 osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.456( v 173'2523 (0'0,173'2523] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2522 active]
2014-06-12 14:46:14.670465 7f9d623d5700  5 -- op tracker -- , seq: 747672, time: 2014-06-12 14:46:14.670461, event: reached_pg, op: osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.670475 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2523 (0'0,173'2523] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2522 active] handle_message: 0x59dd700
2014-06-12 14:46:14.670488 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2523 (0'0,173'2523] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2522 active] sub_op_modify trans b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 v 173'2524 (transaction) 174
2014-06-12 14:46:14.670504 7f9d623d5700  5 -- op tracker -- , seq: 747672, time: 2014-06-12 14:46:14.670500, event: started, op: osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.670526 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2523 (0'0,173'2523] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2522 active] append_log log((0'0,173'2523], crt=0'0) [173'2524 (173'662) modify   b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 by client.4303.0:144715 2014-06-12 08:45:16.984766]
2014-06-12 14:46:14.670552 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2524 (0'0,173'2524] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2522 active] add_log_entry 173'2524 (173'662) modify   b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 by client.4303.0:144715 2014-06-12 08:45:16.984766
2014-06-12 14:46:14.670574 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2524 (0'0,173'2524] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2522 active] append_log  adding 1 keys
2014-06-12 14:46:14.670614 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2524, trimmed: 
2014-06-12 14:46:14.670641 7f9d623d5700  5 -- op tracker -- , seq: 747672, time: 2014-06-12 14:46:14.670636, event: started, op: osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.670659 7f9d623d5700  5 queue_transactions existing osr(3.456 0x276d980)/0x276d980
2014-06-12 14:46:14.670663 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:14.670665 7f9d623d5700  5 queue_op 0x88ca480 seq 2319 osr(3.456 0x276d980) 604405 bytes   (queue has 1 ops and 604405 bytes)
2014-06-12 14:46:14.670677 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519e60 (1 active)
2014-06-12 14:46:14.670672 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.456( v 173'2524 (0'0,173'2524] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2522 active] do_sub_op osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.670686 7f9d623d5700 10 osd.2 173 dequeue_op 0x59dd700 finish
2014-06-12 14:46:14.670685 7f9d7a161700  5 _do_op 0x88ca480 seq 2319 osr(3.456 0x276d980)/0x276d980 start
2014-06-12 14:46:14.670690 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.670691 7f9d7a161700 10 _do_transaction on 0xc9b50b0
2014-06-12 14:46:14.670693 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.670698 7f9d7a161700 15 _omap_setkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.670849 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.670875 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:14.670935 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:14.670957 7f9d7a161700 15 _omap_rmkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.670965 7f9d7a161700 10 _omap_rmkeys meta/1fe19cca/pglog_3.456/0//-1 r = 0
2014-06-12 14:46:14.670971 7f9d7a161700 15 _omap_setkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.671107 7f9d7a161700 15 _write 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 0~602112
2014-06-12 14:46:14.671292 7f9d7a161700 10 lookup_strip_header done  cid 3.456_head oid b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.671432 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:14.673934 7f9d7a161700 10 _generic_write 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 0~602112 = 0
2014-06-12 14:46:14.673993 7f9d7a161700 15 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.674010 7f9d7a161700 10 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 = 0
2014-06-12 14:46:14.674017 7f9d7a161700 15 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.674025 7f9d7a161700 10 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 = 0
2014-06-12 14:46:14.674242 7f9d7a161700 20 set_header setting 17414 cid 3.456_head oid b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 parent seq 0
2014-06-12 14:46:14.674325 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:14.674341 7f9d7a161700 20 set_header setting 2304 cid meta oid 1fe19cca/pglog_3.456/0//-1 parent seq 0
2014-06-12 14:46:14.674353 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:14.677238 7f9d7a161700 10 _do_op 0x88ca480 seq 2319 r = 0, finisher 0x6ac9400 0
2014-06-12 14:46:14.677266 7f9d7a161700 10 _finish_op 0x88ca480 seq 2319 osr(3.456 0x276d980)/0x276d980
2014-06-12 14:46:14.677284 7f9d7715b700  5 -- op tracker -- , seq: 747672, time: 2014-06-12 14:46:14.677275, event: commit_sent, op: osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.677288 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519e60 (0 active)
2014-06-12 14:46:14.677297 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:14.677295 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2524 (0'0,173'2524] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2522 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:14.677316 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:14.677361 7f9d7795c700  5 -- op tracker -- , seq: 747672, time: 2014-06-12 14:46:14.677354, event: sub_op_applied, op: osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.677372 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2524 (0'0,173'2524] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active] sub_op_modify_applied on 0xc9b4fc0 op osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.677391 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2524 (0'0,173'2524] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active] op_applied on primary on version 173'2524
2014-06-12 14:46:14.677402 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2524 (0'0,173'2524] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active] op_applied on replica on version 173'2524
2014-06-12 14:46:14.677436 7f9d7795c700  5 -- op tracker -- , seq: 747672, time: 2014-06-12 14:46:14.677431, event: done, op: osd_sub_op(client.4303.0:144715 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2524 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.699913 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:14.700200 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:14.709271 7f9d5b6bc700  5 -- op tracker -- , seq: 747673, time: 2014-06-12 14:46:14.707477, event: header_read, op: osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.709286 7f9d5b6bc700  5 -- op tracker -- , seq: 747673, time: 2014-06-12 14:46:14.707478, event: throttled, op: osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.709295 7f9d5b6bc700  5 -- op tracker -- , seq: 747673, time: 2014-06-12 14:46:14.709230, event: all_read, op: osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.709305 7f9d5b6bc700  5 -- op tracker -- , seq: 747673, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.709309 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:14.709316 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:14.709324 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x2a903300 prio 127 cost 1032999 latency 0.001847 osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.709352 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.709361 7f9d643d9700 10 osd.2 173 dequeue_op 0x2a903300 prio 127 cost 1032999 latency 0.001884 osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.456( v 173'2524 (0'0,173'2524] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active]
2014-06-12 14:46:14.709387 7f9d643d9700  5 -- op tracker -- , seq: 747673, time: 2014-06-12 14:46:14.709382, event: reached_pg, op: osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.709404 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2524 (0'0,173'2524] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active] handle_message: 0x2a903300
2014-06-12 14:46:14.709429 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2524 (0'0,173'2524] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active] sub_op_modify trans b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 v 173'2525 (transaction) 174
2014-06-12 14:46:14.709452 7f9d643d9700  5 -- op tracker -- , seq: 747673, time: 2014-06-12 14:46:14.709447, event: started, op: osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.709466 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2524 (0'0,173'2524] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active] append_log log((0'0,173'2524], crt=0'0) [173'2525 (173'2524) modify   b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 by client.4303.0:144716 2014-06-12 08:45:17.081967]
2014-06-12 14:46:14.709489 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2525 (0'0,173'2525] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active] add_log_entry 173'2525 (173'2524) modify   b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 by client.4303.0:144716 2014-06-12 08:45:17.081967
2014-06-12 14:46:14.709515 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2525 (0'0,173'2525] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active] append_log  adding 1 keys
2014-06-12 14:46:14.709549 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2525, trimmed: 
2014-06-12 14:46:14.709578 7f9d643d9700  5 -- op tracker -- , seq: 747673, time: 2014-06-12 14:46:14.709574, event: started, op: osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.709588 7f9d643d9700  5 queue_transactions existing osr(3.456 0x276d980)/0x276d980
2014-06-12 14:46:14.709592 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:14.709594 7f9d643d9700  5 queue_op 0x3886120 seq 2320 osr(3.456 0x276d980) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:14.709605 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519e60 (1 active)
2014-06-12 14:46:14.709601 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.456( v 173'2525 (0'0,173'2525] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active] do_sub_op osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.709621 7f9d7a962700  5 _do_op 0x3886120 seq 2320 osr(3.456 0x276d980)/0x276d980 start
2014-06-12 14:46:14.709625 7f9d643d9700 10 osd.2 173 dequeue_op 0x2a903300 finish
2014-06-12 14:46:14.709627 7f9d7a962700 10 _do_transaction on 0x2bc869b0
2014-06-12 14:46:14.709629 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.709632 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.709633 7f9d7a962700 15 _omap_setkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.709750 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.709775 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:14.709825 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:14.709847 7f9d7a962700 15 _omap_rmkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.709854 7f9d7a962700 10 _omap_rmkeys meta/1fe19cca/pglog_3.456/0//-1 r = 0
2014-06-12 14:46:14.709860 7f9d7a962700 15 _omap_setkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.709877 7f9d7a962700 15 _write 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 602112~1032192
2014-06-12 14:46:14.710185 7f9d7a962700 10 lookup_strip_header done  cid 3.456_head oid b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.710345 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:14.714377 7f9d7a962700 10 _generic_write 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 602112~1032192 = 0
2014-06-12 14:46:14.714463 7f9d7a962700 15 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.714479 7f9d7a962700 10 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 = 0
2014-06-12 14:46:14.714485 7f9d7a962700 15 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.714492 7f9d7a962700 10 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 = 0
2014-06-12 14:46:14.714776 7f9d7a962700 20 set_header setting 17414 cid 3.456_head oid b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 parent seq 0
2014-06-12 14:46:14.714894 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:14.714910 7f9d7a962700 20 set_header setting 2304 cid meta oid 1fe19cca/pglog_3.456/0//-1 parent seq 0
2014-06-12 14:46:14.714919 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:14.720366 7f9d7a962700 10 _do_op 0x3886120 seq 2320 r = 0, finisher 0x8e0b560 0
2014-06-12 14:46:14.720396 7f9d7a962700 10 _finish_op 0x3886120 seq 2320 osr(3.456 0x276d980)/0x276d980
2014-06-12 14:46:14.720427 7f9d7795c700  5 -- op tracker -- , seq: 747673, time: 2014-06-12 14:46:14.720418, event: sub_op_applied, op: osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.720441 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2525 (0'0,173'2525] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active] sub_op_modify_applied on 0x2bc868c0 op osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.720468 7f9d7795c700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:14.720489 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2525 (0'0,173'2525] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active] op_applied on primary on version 173'2525
2014-06-12 14:46:14.720501 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2525 (0'0,173'2525] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active] op_applied on replica on version 173'2525
2014-06-12 14:46:14.720515 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519e60 (0 active)
2014-06-12 14:46:14.720521 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:14.720577 7f9d7715b700  5 -- op tracker -- , seq: 747673, time: 2014-06-12 14:46:14.720570, event: commit_sent, op: osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.720590 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2525 (0'0,173'2525] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2523 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:14.720619 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:14.720654 7f9d7715b700  5 -- op tracker -- , seq: 747673, time: 2014-06-12 14:46:14.720649, event: done, op: osd_sub_op(client.4303.0:144716 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2525 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.733705 7f9d5b6bc700  5 -- op tracker -- , seq: 747674, time: 2014-06-12 14:46:14.731931, event: header_read, op: osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.733721 7f9d5b6bc700  5 -- op tracker -- , seq: 747674, time: 2014-06-12 14:46:14.731932, event: throttled, op: osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.733730 7f9d5b6bc700  5 -- op tracker -- , seq: 747674, time: 2014-06-12 14:46:14.733656, event: all_read, op: osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.733739 7f9d5b6bc700  5 -- op tracker -- , seq: 747674, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.733755 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:14.733763 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:14.733776 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x2a902d00 prio 127 cost 1032999 latency 0.001845 osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.733839 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.733852 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2a902d00 prio 127 cost 1032999 latency 0.001921 osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.456( v 173'2525 (0'0,173'2525] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2524 active]
2014-06-12 14:46:14.733881 7f9d61bd4700  5 -- op tracker -- , seq: 747674, time: 2014-06-12 14:46:14.733876, event: reached_pg, op: osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.733893 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2525 (0'0,173'2525] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2524 active] handle_message: 0x2a902d00
2014-06-12 14:46:14.733907 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2525 (0'0,173'2525] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2524 active] sub_op_modify trans b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 v 173'2526 (transaction) 174
2014-06-12 14:46:14.733924 7f9d61bd4700  5 -- op tracker -- , seq: 747674, time: 2014-06-12 14:46:14.733920, event: started, op: osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.733942 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2525 (0'0,173'2525] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2524 active] append_log log((0'0,173'2525], crt=0'0) [173'2526 (173'2525) modify   b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 by client.4303.0:144717 2014-06-12 08:45:17.105841]
2014-06-12 14:46:14.733969 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2526 (0'0,173'2526] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2524 active] add_log_entry 173'2526 (173'2525) modify   b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 by client.4303.0:144717 2014-06-12 08:45:17.105841
2014-06-12 14:46:14.733999 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2526 (0'0,173'2526] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2524 active] append_log  adding 1 keys
2014-06-12 14:46:14.734038 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2526, trimmed: 
2014-06-12 14:46:14.734066 7f9d61bd4700  5 -- op tracker -- , seq: 747674, time: 2014-06-12 14:46:14.734061, event: started, op: osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.734078 7f9d61bd4700  5 queue_transactions existing osr(3.456 0x276d980)/0x276d980
2014-06-12 14:46:14.734082 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:14.734084 7f9d61bd4700  5 queue_op 0x5875080 seq 2321 osr(3.456 0x276d980) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:14.734092 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.456( v 173'2526 (0'0,173'2526] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2524 active] do_sub_op osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.734107 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2a902d00 finish
2014-06-12 14:46:14.734110 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.734112 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.734120 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519e60 (1 active)
2014-06-12 14:46:14.734125 7f9d7f16b700  5 _do_op 0x5875080 seq 2321 osr(3.456 0x276d980)/0x276d980 start
2014-06-12 14:46:14.734130 7f9d7f16b700 10 _do_transaction on 0x1f287b30
2014-06-12 14:46:14.734136 7f9d7f16b700 15 _omap_setkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.734260 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.734285 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:14.734333 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:14.734356 7f9d7f16b700 15 _omap_rmkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.734364 7f9d7f16b700 10 _omap_rmkeys meta/1fe19cca/pglog_3.456/0//-1 r = 0
2014-06-12 14:46:14.734370 7f9d7f16b700 15 _omap_setkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.734386 7f9d7f16b700 15 _write 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 1634304~1032192
2014-06-12 14:46:14.734524 7f9d7f16b700 10 lookup_strip_header done  cid 3.456_head oid b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.734769 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:14.738369 7f9d7f16b700 10 _generic_write 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 1634304~1032192 = 0
2014-06-12 14:46:14.738435 7f9d7f16b700 15 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.738457 7f9d7f16b700 10 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 = 0
2014-06-12 14:46:14.738464 7f9d7f16b700 15 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.738469 7f9d7f16b700 10 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 = 0
2014-06-12 14:46:14.738670 7f9d7f16b700 20 set_header setting 17414 cid 3.456_head oid b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 parent seq 0
2014-06-12 14:46:14.738740 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:14.738749 7f9d7f16b700 20 set_header setting 2304 cid meta oid 1fe19cca/pglog_3.456/0//-1 parent seq 0
2014-06-12 14:46:14.738755 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:14.743734 7f9d7f16b700 10 _do_op 0x5875080 seq 2321 r = 0, finisher 0x8356e60 0
2014-06-12 14:46:14.743760 7f9d7f16b700 10 _finish_op 0x5875080 seq 2321 osr(3.456 0x276d980)/0x276d980
2014-06-12 14:46:14.743777 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519e60 (0 active)
2014-06-12 14:46:14.743783 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:14.743782 7f9d7715b700  5 -- op tracker -- , seq: 747674, time: 2014-06-12 14:46:14.743771, event: commit_sent, op: osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.743792 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2526 (0'0,173'2526] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2524 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:14.743818 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:14.743963 7f9d7795c700  5 -- op tracker -- , seq: 747674, time: 2014-06-12 14:46:14.743946, event: sub_op_applied, op: osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.743974 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2526 (0'0,173'2526] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2525 active] sub_op_modify_applied on 0x1f287a40 op osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.744038 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2526 (0'0,173'2526] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2525 active] op_applied on primary on version 173'2526
2014-06-12 14:46:14.744050 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2526 (0'0,173'2526] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2525 active] op_applied on replica on version 173'2526
2014-06-12 14:46:14.744087 7f9d7795c700  5 -- op tracker -- , seq: 747674, time: 2014-06-12 14:46:14.744083, event: done, op: osd_sub_op(client.4303.0:144717 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2526 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.757755 7f9d5b6bc700  5 -- op tracker -- , seq: 747675, time: 2014-06-12 14:46:14.755966, event: header_read, op: osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.757769 7f9d5b6bc700  5 -- op tracker -- , seq: 747675, time: 2014-06-12 14:46:14.755967, event: throttled, op: osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.757776 7f9d5b6bc700  5 -- op tracker -- , seq: 747675, time: 2014-06-12 14:46:14.757697, event: all_read, op: osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.757784 7f9d5b6bc700  5 -- op tracker -- , seq: 747675, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.757789 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:14.757797 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:14.757807 7f9d5b6bc700 15 osd.2 173 enqueue_op 0xe20a800 prio 127 cost 1032999 latency 0.001841 osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.761554 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.761575 7f9d663dd700 10 osd.2 173 dequeue_op 0xe20a800 prio 127 cost 1032999 latency 0.005608 osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.456( v 173'2526 (0'0,173'2526] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2525 active]
2014-06-12 14:46:14.761599 7f9d663dd700  5 -- op tracker -- , seq: 747675, time: 2014-06-12 14:46:14.761595, event: reached_pg, op: osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.761609 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2526 (0'0,173'2526] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2525 active] handle_message: 0xe20a800
2014-06-12 14:46:14.761627 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2526 (0'0,173'2526] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2525 active] sub_op_modify trans b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 v 173'2527 (transaction) 174
2014-06-12 14:46:14.761644 7f9d663dd700  5 -- op tracker -- , seq: 747675, time: 2014-06-12 14:46:14.761641, event: started, op: osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.761677 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2526 (0'0,173'2526] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2525 active] append_log log((0'0,173'2526], crt=0'0) [173'2527 (173'2526) modify   b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 by client.4303.0:144718 2014-06-12 08:45:17.131108]
2014-06-12 14:46:14.761704 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2527 (0'0,173'2527] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2525 active] add_log_entry 173'2527 (173'2526) modify   b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 by client.4303.0:144718 2014-06-12 08:45:17.131108
2014-06-12 14:46:14.761729 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2527 (0'0,173'2527] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2525 active] append_log  adding 1 keys
2014-06-12 14:46:14.761766 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2527, trimmed: 
2014-06-12 14:46:14.761794 7f9d663dd700  5 -- op tracker -- , seq: 747675, time: 2014-06-12 14:46:14.761789, event: started, op: osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.761805 7f9d663dd700  5 queue_transactions existing osr(3.456 0x276d980)/0x276d980
2014-06-12 14:46:14.761810 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:14.761812 7f9d663dd700  5 queue_op 0x228711a0 seq 2322 osr(3.456 0x276d980) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:14.761819 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.456( v 173'2527 (0'0,173'2527] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2525 active] do_sub_op osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.761835 7f9d663dd700 10 osd.2 173 dequeue_op 0xe20a800 finish
2014-06-12 14:46:14.761839 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.761842 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.761851 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519e60 (1 active)
2014-06-12 14:46:14.761856 7f9d7895e700  5 _do_op 0x228711a0 seq 2322 osr(3.456 0x276d980)/0x276d980 start
2014-06-12 14:46:14.761862 7f9d7895e700 10 _do_transaction on 0x274a67f0
2014-06-12 14:46:14.761867 7f9d7895e700 15 _omap_setkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.761999 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.762032 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:14.762085 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:14.762103 7f9d7895e700 15 _omap_rmkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.762109 7f9d7895e700 10 _omap_rmkeys meta/1fe19cca/pglog_3.456/0//-1 r = 0
2014-06-12 14:46:14.762115 7f9d7895e700 15 _omap_setkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.762228 7f9d7895e700 15 _write 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 2666496~1032192
2014-06-12 14:46:14.762368 7f9d7895e700 10 lookup_strip_header done  cid 3.456_head oid b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.762527 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:14.766003 7f9d7895e700 10 _generic_write 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 2666496~1032192 = 0
2014-06-12 14:46:14.766086 7f9d7895e700 15 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.766101 7f9d7895e700 10 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 = 0
2014-06-12 14:46:14.766109 7f9d7895e700 15 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.766117 7f9d7895e700 10 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 = 0
2014-06-12 14:46:14.766328 7f9d7895e700 20 set_header setting 17414 cid 3.456_head oid b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 parent seq 0
2014-06-12 14:46:14.766423 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:14.766440 7f9d7895e700 20 set_header setting 2304 cid meta oid 1fe19cca/pglog_3.456/0//-1 parent seq 0
2014-06-12 14:46:14.766461 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:14.770391 7f9d7895e700 10 _do_op 0x228711a0 seq 2322 r = 0, finisher 0xabf2580 0
2014-06-12 14:46:14.770411 7f9d7895e700 10 _finish_op 0x228711a0 seq 2322 osr(3.456 0x276d980)/0x276d980
2014-06-12 14:46:14.770429 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519e60 (0 active)
2014-06-12 14:46:14.770437 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:14.770430 7f9d7715b700  5 -- op tracker -- , seq: 747675, time: 2014-06-12 14:46:14.770419, event: commit_sent, op: osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.770445 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2527 (0'0,173'2527] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2525 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:14.770472 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:14.770507 7f9d7795c700  5 -- op tracker -- , seq: 747675, time: 2014-06-12 14:46:14.770501, event: sub_op_applied, op: osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.770518 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2527 (0'0,173'2527] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2526 active] sub_op_modify_applied on 0x274a6700 op osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.770536 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2527 (0'0,173'2527] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2526 active] op_applied on primary on version 173'2527
2014-06-12 14:46:14.770548 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2527 (0'0,173'2527] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2526 active] op_applied on replica on version 173'2527
2014-06-12 14:46:14.770575 7f9d7795c700  5 -- op tracker -- , seq: 747675, time: 2014-06-12 14:46:14.770571, event: done, op: osd_sub_op(client.4303.0:144718 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2527 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.777154 7f9d5b6bc700  5 -- op tracker -- , seq: 747676, time: 2014-06-12 14:46:14.776401, event: header_read, op: osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.777169 7f9d5b6bc700  5 -- op tracker -- , seq: 747676, time: 2014-06-12 14:46:14.776403, event: throttled, op: osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.777178 7f9d5b6bc700  5 -- op tracker -- , seq: 747676, time: 2014-06-12 14:46:14.777103, event: all_read, op: osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.777192 7f9d5b6bc700  5 -- op tracker -- , seq: 747676, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.777197 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:14.777204 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:14.777213 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x28dd0b00 prio 127 cost 496423 latency 0.000812 osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.777651 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.777663 7f9d673df700 10 osd.2 173 dequeue_op 0x28dd0b00 prio 127 cost 496423 latency 0.001262 osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.456( v 173'2527 (0'0,173'2527] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2526 active]
2014-06-12 14:46:14.777687 7f9d673df700  5 -- op tracker -- , seq: 747676, time: 2014-06-12 14:46:14.777683, event: reached_pg, op: osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.777696 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2527 (0'0,173'2527] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2526 active] handle_message: 0x28dd0b00
2014-06-12 14:46:14.777708 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2527 (0'0,173'2527] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2526 active] sub_op_modify trans b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 v 173'2528 (transaction) 174
2014-06-12 14:46:14.777725 7f9d673df700  5 -- op tracker -- , seq: 747676, time: 2014-06-12 14:46:14.777721, event: started, op: osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.777741 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2527 (0'0,173'2527] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2526 active] append_log log((0'0,173'2527], crt=0'0) [173'2528 (173'2527) modify   b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 by client.4303.0:144719 2014-06-12 08:45:17.153714]
2014-06-12 14:46:14.777763 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2528 (0'0,173'2528] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2526 active] add_log_entry 173'2528 (173'2527) modify   b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 by client.4303.0:144719 2014-06-12 08:45:17.153714
2014-06-12 14:46:14.777790 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2528 (0'0,173'2528] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2526 active] append_log  adding 1 keys
2014-06-12 14:46:14.777819 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2528, trimmed: 
2014-06-12 14:46:14.777839 7f9d673df700  5 -- op tracker -- , seq: 747676, time: 2014-06-12 14:46:14.777835, event: started, op: osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.777848 7f9d673df700  5 queue_transactions existing osr(3.456 0x276d980)/0x276d980
2014-06-12 14:46:14.777852 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:14.777853 7f9d673df700  5 queue_op 0x6db7c80 seq 2323 osr(3.456 0x276d980) 497909 bytes   (queue has 1 ops and 497909 bytes)
2014-06-12 14:46:14.777871 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5519e60 (1 active)
2014-06-12 14:46:14.777869 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.456( v 173'2528 (0'0,173'2528] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2526 active] do_sub_op osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.777879 7f9d7e169700  5 _do_op 0x6db7c80 seq 2323 osr(3.456 0x276d980)/0x276d980 start
2014-06-12 14:46:14.777881 7f9d673df700 10 osd.2 173 dequeue_op 0x28dd0b00 finish
2014-06-12 14:46:14.777884 7f9d7e169700 10 _do_transaction on 0x274a67f0
2014-06-12 14:46:14.777885 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.777887 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.777890 7f9d7e169700 15 _omap_setkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.777999 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.778014 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:14.778060 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:14.778074 7f9d7e169700 15 _omap_rmkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.778079 7f9d7e169700 10 _omap_rmkeys meta/1fe19cca/pglog_3.456/0//-1 r = 0
2014-06-12 14:46:14.778083 7f9d7e169700 15 _omap_setkeys meta/1fe19cca/pglog_3.456/0//-1
2014-06-12 14:46:14.778095 7f9d7e169700 15 _write 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 3698688~495616
2014-06-12 14:46:14.778227 7f9d7e169700 10 lookup_strip_header done  cid 3.456_head oid b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.778326 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:14.780705 7f9d7e169700 10 _generic_write 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 3698688~495616 = 0
2014-06-12 14:46:14.780764 7f9d7e169700 15 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.780789 7f9d7e169700 10 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 = 0
2014-06-12 14:46:14.780797 7f9d7e169700 15 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3
2014-06-12 14:46:14.780806 7f9d7e169700 10 _setattrs 3.456_head/b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 = 0
2014-06-12 14:46:14.781005 7f9d7e169700 20 set_header setting 17414 cid 3.456_head oid b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 parent seq 0
2014-06-12 14:46:14.781089 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:14.781106 7f9d7e169700 20 set_header setting 2304 cid meta oid 1fe19cca/pglog_3.456/0//-1 parent seq 0
2014-06-12 14:46:14.781115 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:14.783377 7f9d7e169700 10 _do_op 0x6db7c80 seq 2323 r = 0, finisher 0x28f3e20 0
2014-06-12 14:46:14.783418 7f9d7715b700  5 -- op tracker -- , seq: 747676, time: 2014-06-12 14:46:14.783408, event: commit_sent, op: osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.783429 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2528 (0'0,173'2528] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2526 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:14.783461 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:14.783490 7f9d7e169700 10 _finish_op 0x6db7c80 seq 2323 osr(3.456 0x276d980)/0x276d980
2014-06-12 14:46:14.783498 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5519e60 (0 active)
2014-06-12 14:46:14.783508 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:14.783519 7f9d7795c700  5 -- op tracker -- , seq: 747676, time: 2014-06-12 14:46:14.783511, event: sub_op_applied, op: osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.783533 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2528 (0'0,173'2528] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2527 active] sub_op_modify_applied on 0x274a6700 op osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:14.783552 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2528 (0'0,173'2528] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2527 active] op_applied on primary on version 173'2528
2014-06-12 14:46:14.783563 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.456( v 173'2528 (0'0,173'2528] local-les=157 n=274 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2527 active] op_applied on replica on version 173'2528
2014-06-12 14:46:14.783599 7f9d7795c700  5 -- op tracker -- , seq: 747676, time: 2014-06-12 14:46:14.783594, event: done, op: osd_sub_op(client.4303.0:144719 3.456 b0abb456/rbd_data.10a66b8b4567.0000000000000fea/head//3 [] v 173'2528 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:14.835979 7f9d523da700  5 -- op tracker -- , seq: 747677, time: 2014-06-12 14:46:14.833529, event: header_read, op: osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.835994 7f9d523da700  5 -- op tracker -- , seq: 747677, time: 2014-06-12 14:46:14.833533, event: throttled, op: osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.836005 7f9d523da700  5 -- op tracker -- , seq: 747677, time: 2014-06-12 14:46:14.835925, event: all_read, op: osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.836015 7f9d523da700  5 -- op tracker -- , seq: 747677, time: 0.000000, event: dispatched, op: osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.836025 7f9d523da700 20 osd.2 173 should_share_map client.4284 172.16.96.220:0/1241415 173
2014-06-12 14:46:14.836057 7f9d523da700 15 osd.2 173 enqueue_op 0x329eb000 prio 63 cost 925696 latency 0.002527 osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:14.836094 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.836104 7f9d67be0700 10 osd.2 173 dequeue_op 0x329eb000 prio 63 cost 925696 latency 0.002575 osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]
2014-06-12 14:46:14.836134 7f9d67be0700  5 -- op tracker -- , seq: 747677, time: 2014-06-12 14:46:14.836128, event: reached_pg, op: osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.836145 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:14.836160 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] handle_message: 0x329eb000
2014-06-12 14:46:14.836171 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_op osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:14.836198 7f9d67be0700 15 getattr 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 '_'
2014-06-12 14:46:14.845882 7f9d523da700  5 -- op tracker -- , seq: 747678, time: 2014-06-12 14:46:14.842942, event: header_read, op: osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.845896 7f9d523da700  5 -- op tracker -- , seq: 747678, time: 2014-06-12 14:46:14.842948, event: throttled, op: osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.845907 7f9d523da700  5 -- op tracker -- , seq: 747678, time: 2014-06-12 14:46:14.845830, event: all_read, op: osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.845919 7f9d523da700  5 -- op tracker -- , seq: 747678, time: 0.000000, event: dispatched, op: osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.845927 7f9d523da700 20 osd.2 173 should_share_map client.4284 172.16.96.220:0/1241415 173
2014-06-12 14:46:14.845954 7f9d523da700 15 osd.2 173 enqueue_op 0x329eb300 prio 63 cost 516096 latency 0.003012 osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:14.845993 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:14.930134 7f9d67be0700 10 getattr 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 '_' = 0
2014-06-12 14:46:14.930172 7f9d67be0700 15 getattr 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 'snapset'
2014-06-12 14:46:14.930321 7f9d67be0700 10 getattr 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 'snapset' = 0
2014-06-12 14:46:14.930333 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] populate_obc_watchers 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.930354 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.930365 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] get_object_context: creating obc from disk: 0x341c82c0
2014-06-12 14:46:14.930376 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] get_object_context: 0x341c82c0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 rwstate(none n=0 w=0) oi: 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3(173'771 client.4284.0:36101 wrlock_by=unknown.0.0:0 dirty s 4194304 uv771) ssc: 0x81baee0 snapset: 0=[]:[]+head
2014-06-12 14:46:14.930424 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] find_object_context 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 @head oi=627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3(173'771 client.4284.0:36101 wrlock_by=unknown.0.0:0 dirty s 4194304 uv771)
2014-06-12 14:46:14.930463 7f9d67be0700  5 -- op tracker -- , seq: 747677, time: 2014-06-12 14:46:14.930453, event: started, op: osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.930472 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] execute_ctx 0x17cfbe00
2014-06-12 14:46:14.930491 7f9d67be0700  5 -- op tracker -- , seq: 747677, time: 2014-06-12 14:46:14.930485, event: started, op: osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.930503 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_op 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] ov 173'771 av 173'2852 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:14.930521 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_osd_op 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696]
2014-06-12 14:46:14.930534 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:14.930550 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_osd_op  write 0~925696
2014-06-12 14:46:14.930569 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_osd_op_effects on session 0x5c8ce00
2014-06-12 14:46:14.930583 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] make_writeable 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 snapset=0x81baf18  snapc=0=[]
2014-06-12 14:46:14.930596 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] make_writeable 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:14.930607 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] finish_ctx 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 0x17cfbe00 op modify  
2014-06-12 14:46:14.930620 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]  set mtime to 2014-06-12 08:45:17.212836
2014-06-12 14:46:14.930644 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]  final snapset 0=[]:[]+head in 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.930667 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]  zeroing write result code 0
2014-06-12 14:46:14.930680 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] new_repop rep_tid 236227 on osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:14.930698 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] issue_repop rep_tid 236227 o 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.930726 7f9d67be0700  5 -- op tracker -- , seq: 747677, time: 2014-06-12 14:46:14.930721, event: waiting for subops from 29, op: osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.930750 7f9d67be0700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:14.930779 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2851 (0'0,173'2851] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] append_log log((0'0,173'2851], crt=0'0) [173'2852 (173'771) modify   627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 by client.4284.0:141478 2014-06-12 08:45:17.212836]
2014-06-12 14:46:14.930799 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] add_log_entry 173'2852 (173'771) modify   627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 by client.4284.0:141478 2014-06-12 08:45:17.212836
2014-06-12 14:46:14.930822 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] append_log  adding 1 keys
2014-06-12 14:46:14.930857 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2852, trimmed: 
2014-06-12 14:46:14.930881 7f9d67be0700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:46:14.930886 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:14.930888 7f9d67be0700  5 queue_op 0x1b77e720 seq 2619 osr(3.4f0 0x276db90) 927989 bytes   (queue has 1 ops and 927989 bytes)
2014-06-12 14:46:14.930896 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] eval_repop repgather(0xeaac180 173'2852 rep_tid=236227 committed?=0 applied?=0 lock=0 op=osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.930916 7f9d67be0700 10 osd.2 173 dequeue_op 0x329eb000 finish
2014-06-12 14:46:14.930926 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:14.930931 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.930945 7f9d693e3700 10 osd.2 173 dequeue_op 0x329eb300 prio 63 cost 516096 latency 0.088002 osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]
2014-06-12 14:46:14.930971 7f9d693e3700  5 -- op tracker -- , seq: 747678, time: 2014-06-12 14:46:14.930965, event: reached_pg, op: osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.930981 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:14.930998 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] handle_message: 0x329eb300
2014-06-12 14:46:14.931010 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_op osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:14.931034 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] get_object_context: found obc in cache: 0x341c82c0
2014-06-12 14:46:14.931046 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] get_object_context: 0x341c82c0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 rwstate(write n=1 w=0) oi: 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3(173'2852 client.4284.0:141478 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2852) ssc: 0x81baee0 snapset: 0=[]:[]+head
2014-06-12 14:46:14.931063 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] find_object_context 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 @head oi=627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3(173'2852 client.4284.0:141478 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2852)
2014-06-12 14:46:14.931089 7f9d693e3700  5 -- op tracker -- , seq: 747678, time: 2014-06-12 14:46:14.931083, event: started, op: osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.931098 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] execute_ctx 0x1b265600
2014-06-12 14:46:14.931122 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (1 active)
2014-06-12 14:46:14.931127 7f9d7e96a700  5 _do_op 0x1b77e720 seq 2619 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:46:14.931132 7f9d7e96a700 10 _do_transaction on 0xeaacd80
2014-06-12 14:46:14.931138 7f9d7e96a700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.931303 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.931330 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:14.931381 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:14.931401 7f9d7e96a700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.931409 7f9d7e96a700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:46:14.931415 7f9d7e96a700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.931434 7f9d7e96a700 15 _write 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 0~925696
2014-06-12 14:46:14.931669 7f9d7e96a700 10 lookup_strip_header done  cid 3.4f0_head oid 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.931795 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:14.931116 7f9d693e3700  5 -- op tracker -- , seq: 747678, time: 2014-06-12 14:46:14.931111, event: started, op: osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.933663 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_op 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] ov 173'2852 av 173'2853 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:14.933694 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_osd_op 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096]
2014-06-12 14:46:14.933708 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:14.933725 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_osd_op  write 925696~516096
2014-06-12 14:46:14.933744 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] do_osd_op_effects on session 0x5c8ce00
2014-06-12 14:46:14.933760 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] make_writeable 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 snapset=0x81baf18  snapc=0=[]
2014-06-12 14:46:14.933776 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] make_writeable 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:14.933788 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] finish_ctx 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 0x1b265600 op modify  
2014-06-12 14:46:14.933801 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]  set mtime to 2014-06-12 08:45:17.222964
2014-06-12 14:46:14.933827 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]  final snapset 0=[]:[]+head in 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.933850 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]  zeroing write result code 0
2014-06-12 14:46:14.933864 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] new_repop rep_tid 236228 on osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:14.933887 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] issue_repop rep_tid 236228 o 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.933919 7f9d693e3700  5 -- op tracker -- , seq: 747678, time: 2014-06-12 14:46:14.933912, event: waiting for subops from 29, op: osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.933947 7f9d693e3700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:14.933975 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2852 (0'0,173'2852] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] append_log log((0'0,173'2852], crt=0'0) [173'2853 (173'2852) modify   627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 by client.4284.0:141479 2014-06-12 08:45:17.222964]
2014-06-12 14:46:14.934001 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] add_log_entry 173'2853 (173'2852) modify   627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 by client.4284.0:141479 2014-06-12 08:45:17.222964
2014-06-12 14:46:14.934025 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] append_log  adding 1 keys
2014-06-12 14:46:14.934061 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2853, trimmed: 
2014-06-12 14:46:14.934087 7f9d693e3700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:46:14.934094 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:14.934096 7f9d693e3700  5 queue_op 0xb9b4cc0 seq 2620 osr(3.4f0 0x276db90) 518389 bytes   (queue has 2 ops and 1446378 bytes)
2014-06-12 14:46:14.934103 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] eval_repop repgather(0x1c832180 173'2853 rep_tid=236228 committed?=0 applied?=0 lock=0 op=osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.934124 7f9d693e3700 10 osd.2 173 dequeue_op 0x329eb300 finish
2014-06-12 14:46:14.934131 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.934136 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.934462 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (2 active)
2014-06-12 14:46:14.935497 7f9d7e96a700 10 _generic_write 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 0~925696 = 0
2014-06-12 14:46:14.935571 7f9d7e96a700 15 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.935589 7f9d7e96a700 10 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 = 0
2014-06-12 14:46:14.935599 7f9d7e96a700 15 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.935607 7f9d7e96a700 10 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 = 0
2014-06-12 14:46:14.935915 7f9d7e96a700 20 set_header setting 17378 cid 3.4f0_head oid 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 parent seq 0
2014-06-12 14:46:14.936037 7f9d7e96a700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:46:14.936053 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:14.936062 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:14.940818 7f9d7e96a700 10 _do_op 0x1b77e720 seq 2619 r = 0, finisher 0xabf2260 0x38e13080
2014-06-12 14:46:14.940864 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] op_commit: 236227
2014-06-12 14:46:14.940908 7f9d7715b700  5 -- op tracker -- , seq: 747677, time: 2014-06-12 14:46:14.940897, event: op_commit, op: osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.940924 7f9d7e96a700 10 _finish_op 0x1b77e720 seq 2619 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:46:14.940935 7f9d7815d700  5 _do_op 0xb9b4cc0 seq 2620 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:46:14.940938 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (1 active)
2014-06-12 14:46:14.940942 7f9d7815d700 10 _do_transaction on 0x1c832900
2014-06-12 14:46:14.940944 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:14.940954 7f9d7815d700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.940944 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] op_applied: 236227
2014-06-12 14:46:14.940965 7f9d7795c700  5 -- op tracker -- , seq: 747677, time: 2014-06-12 14:46:14.940958, event: op_applied, op: osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.940973 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] op_applied on primary on version 173'2852
2014-06-12 14:46:14.941185 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.941214 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:14.941266 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:14.941289 7f9d7815d700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.941295 7f9d7815d700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:46:14.941302 7f9d7815d700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.941321 7f9d7815d700 15 _write 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 925696~516096
2014-06-12 14:46:14.941641 7f9d7815d700 10 lookup_strip_header done  cid 3.4f0_head oid 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.941779 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:14.943169 7f9d5501e700  5 -- op tracker -- , seq: 747679, time: 2014-06-12 14:46:14.943121, event: header_read, op: osd_sub_op_reply(client.4284.0:141478 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.943179 7f9d5501e700  5 -- op tracker -- , seq: 747679, time: 2014-06-12 14:46:14.943122, event: throttled, op: osd_sub_op_reply(client.4284.0:141478 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.943187 7f9d5501e700  5 -- op tracker -- , seq: 747679, time: 2014-06-12 14:46:14.943135, event: all_read, op: osd_sub_op_reply(client.4284.0:141478 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.943196 7f9d5501e700  5 -- op tracker -- , seq: 747679, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4284.0:141478 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.943202 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4284.0:141478 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:14.943209 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:14.943217 7f9d5501e700 15 osd.2 173 enqueue_op 0x1e3eb100 prio 196 cost 0 latency 0.000096 osd_sub_op_reply(client.4284.0:141478 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:14.943258 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.943271 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x1e3eb100 prio 196 cost 0 latency 0.000149 osd_sub_op_reply(client.4284.0:141478 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]
2014-06-12 14:46:14.943296 7f9d6b3e7700  5 -- op tracker -- , seq: 747679, time: 2014-06-12 14:46:14.943292, event: reached_pg, op: osd_sub_op_reply(client.4284.0:141478 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.943306 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] handle_message: 0x1e3eb100
2014-06-12 14:46:14.943325 7f9d6b3e7700  5 -- op tracker -- , seq: 747679, time: 2014-06-12 14:46:14.943321, event: started, op: osd_sub_op_reply(client.4284.0:141478 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.943331 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] sub_op_modify_reply: tid 236227 op  ack_type 4 from 29
2014-06-12 14:46:14.943352 7f9d6b3e7700  5 -- op tracker -- , seq: 747677, time: 2014-06-12 14:46:14.943344, event: sub_op_commit_rec, op: osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.943360 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] repop_all_applied: repop tid 236227 all applied 
2014-06-12 14:46:14.943373 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] eval_repop repgather(0xeaac180 173'2852 rep_tid=236227 committed?=0 applied?=1 lock=0 op=osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.943397 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean]  sending ack on repgather(0xeaac180 173'2852 rep_tid=236227 committed?=0 applied?=1 lock=0 op=osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) 0x30ee0500
2014-06-12 14:46:14.943434 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2851 crt=0'0 lcod 173'2850 mlcod 173'2850 active+clean] repop_all_committed: repop tid 236227 all committed 
2014-06-12 14:46:14.943446 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2850 active+clean] eval_repop repgather(0xeaac180 173'2852 rep_tid=236227 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.943461 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2850 active+clean] log_op_stats osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 inb 925975 outb 0 rlat 0.109903 lat 0.109931
2014-06-12 14:46:14.943478 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2850 active+clean] publish_stats_to_osd 173:3706
2014-06-12 14:46:14.943494 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2850 active+clean]  sending commit on repgather(0xeaac180 173'2852 rep_tid=236227 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) 0x8df2280
2014-06-12 14:46:14.943530 7f9d6b3e7700  5 -- op tracker -- , seq: 747677, time: 2014-06-12 14:46:14.943523, event: commit_sent, op: osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.943538 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2850 active+clean]  requeue_ops 
2014-06-12 14:46:14.943548 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]  removing repgather(0xeaac180 173'2852 rep_tid=236227 committed?=1 applied?=1 op=osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.943564 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]    q front is repgather(0xeaac180 173'2852 rep_tid=236227 committed?=1 applied?=1 op=osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.943581 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] remove_repop repgather(0xeaac180 173'2852 rep_tid=236227 committed?=1 applied?=1 op=osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.943607 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]  requeue_ops 
2014-06-12 14:46:14.943628 7f9d6b3e7700  5 -- op tracker -- , seq: 747677, time: 2014-06-12 14:46:14.943623, event: done, op: osd_op(client.4284.0:141478 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.943647 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x1e3eb100 finish
2014-06-12 14:46:14.943655 7f9d6b3e7700  5 -- op tracker -- , seq: 747679, time: 2014-06-12 14:46:14.943651, event: done, op: osd_sub_op_reply(client.4284.0:141478 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.943665 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.943669 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.943785 7f9d7815d700 10 _generic_write 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 925696~516096 = 0
2014-06-12 14:46:14.943834 7f9d7815d700 15 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.943852 7f9d7815d700 10 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 = 0
2014-06-12 14:46:14.943858 7f9d7815d700 15 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.943867 7f9d7815d700 10 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 = 0
2014-06-12 14:46:14.944074 7f9d7815d700 20 set_header setting 17378 cid 3.4f0_head oid 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 parent seq 0
2014-06-12 14:46:14.944148 7f9d7815d700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:46:14.944168 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:14.944179 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:14.946129 7f9d5501e700  5 -- op tracker -- , seq: 747680, time: 2014-06-12 14:46:14.946082, event: header_read, op: osd_sub_op_reply(client.4284.0:141479 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.946140 7f9d5501e700  5 -- op tracker -- , seq: 747680, time: 2014-06-12 14:46:14.946084, event: throttled, op: osd_sub_op_reply(client.4284.0:141479 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.946148 7f9d5501e700  5 -- op tracker -- , seq: 747680, time: 2014-06-12 14:46:14.946099, event: all_read, op: osd_sub_op_reply(client.4284.0:141479 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.946156 7f9d5501e700  5 -- op tracker -- , seq: 747680, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4284.0:141479 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.946161 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4284.0:141479 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:14.946167 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:14.946175 7f9d5501e700 15 osd.2 173 enqueue_op 0x1e3eaf00 prio 196 cost 0 latency 0.000093 osd_sub_op_reply(client.4284.0:141479 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:14.946202 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.946210 7f9d62bd6700 10 osd.2 173 dequeue_op 0x1e3eaf00 prio 196 cost 0 latency 0.000127 osd_sub_op_reply(client.4284.0:141479 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean]
2014-06-12 14:46:14.946224 7f9d62bd6700  5 -- op tracker -- , seq: 747680, time: 2014-06-12 14:46:14.946221, event: reached_pg, op: osd_sub_op_reply(client.4284.0:141479 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.946242 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] handle_message: 0x1e3eaf00
2014-06-12 14:46:14.946260 7f9d62bd6700  5 -- op tracker -- , seq: 747680, time: 2014-06-12 14:46:14.946257, event: started, op: osd_sub_op_reply(client.4284.0:141479 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.946266 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] sub_op_modify_reply: tid 236228 op  ack_type 4 from 29
2014-06-12 14:46:14.946287 7f9d62bd6700  5 -- op tracker -- , seq: 747678, time: 2014-06-12 14:46:14.946280, event: sub_op_commit_rec, op: osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.946295 7f9d62bd6700 10 osd.2 173 dequeue_op 0x1e3eaf00 finish
2014-06-12 14:46:14.946301 7f9d62bd6700  5 -- op tracker -- , seq: 747680, time: 2014-06-12 14:46:14.946298, event: done, op: osd_sub_op_reply(client.4284.0:141479 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.946322 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.946325 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.946989 7f9d7815d700 10 _do_op 0xb9b4cc0 seq 2620 r = 0, finisher 0x13d10fc0 0x389975c0
2014-06-12 14:46:14.947008 7f9d7815d700 10 _finish_op 0xb9b4cc0 seq 2620 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:46:14.947018 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (0 active)
2014-06-12 14:46:14.947021 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:14.947020 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] op_commit: 236228
2014-06-12 14:46:14.947043 7f9d7715b700  5 -- op tracker -- , seq: 747678, time: 2014-06-12 14:46:14.947037, event: op_commit, op: osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.947054 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2852 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] repop_all_committed: repop tid 236228 all committed 
2014-06-12 14:46:14.947066 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean] eval_repop repgather(0x1c832180 173'2853 rep_tid=236228 committed?=1 applied?=0 lock=0 op=osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.947084 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean] log_op_stats osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.104141
2014-06-12 14:46:14.947104 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean] publish_stats_to_osd 173:3707
2014-06-12 14:46:14.947117 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean]  sending commit on repgather(0x1c832180 173'2853 rep_tid=236228 committed?=1 applied?=0 lock=0 op=osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) 0x205c780
2014-06-12 14:46:14.947156 7f9d7715b700  5 -- op tracker -- , seq: 747678, time: 2014-06-12 14:46:14.947150, event: commit_sent, op: osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.947173 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean] op_applied: 236228
2014-06-12 14:46:14.947196 7f9d7795c700  5 -- op tracker -- , seq: 747678, time: 2014-06-12 14:46:14.947189, event: op_applied, op: osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.947203 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean] op_applied on primary on version 173'2853
2014-06-12 14:46:14.947214 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean] repop_all_applied: repop tid 236228 all applied 
2014-06-12 14:46:14.947220 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean] eval_repop repgather(0x1c832180 173'2853 rep_tid=236228 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.947229 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean] log_op_stats osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.104287
2014-06-12 14:46:14.947238 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean] publish_stats_to_osd 173:3708
2014-06-12 14:46:14.947245 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2851 active+clean]  requeue_ops 
2014-06-12 14:46:14.947253 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  removing repgather(0x1c832180 173'2853 rep_tid=236228 committed?=1 applied?=1 op=osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.947262 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]    q front is repgather(0x1c832180 173'2853 rep_tid=236228 committed?=1 applied?=1 op=osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.947269 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] remove_repop repgather(0x1c832180 173'2853 rep_tid=236228 committed?=1 applied?=1 op=osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.947277 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  requeue_ops 
2014-06-12 14:46:14.947292 7f9d7795c700  5 -- op tracker -- , seq: 747678, time: 2014-06-12 14:46:14.947289, event: done, op: osd_op(client.4284.0:141479 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.949969 7f9d523da700  5 -- op tracker -- , seq: 747681, time: 2014-06-12 14:46:14.946976, event: header_read, op: osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.949982 7f9d523da700  5 -- op tracker -- , seq: 747681, time: 2014-06-12 14:46:14.946979, event: throttled, op: osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.949990 7f9d523da700  5 -- op tracker -- , seq: 747681, time: 2014-06-12 14:46:14.949920, event: all_read, op: osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.949998 7f9d523da700  5 -- op tracker -- , seq: 747681, time: 0.000000, event: dispatched, op: osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.950007 7f9d523da700 20 osd.2 173 should_share_map client.4284 172.16.96.220:0/1241415 173
2014-06-12 14:46:14.950036 7f9d523da700 15 osd.2 173 enqueue_op 0x28dd0b00 prio 63 cost 1032192 latency 0.003059 osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:14.950070 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.950079 7f9d683e1700 10 osd.2 173 dequeue_op 0x28dd0b00 prio 63 cost 1032192 latency 0.003103 osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]
2014-06-12 14:46:14.950110 7f9d683e1700  5 -- op tracker -- , seq: 747681, time: 2014-06-12 14:46:14.950105, event: reached_pg, op: osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.950121 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:14.950136 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] handle_message: 0x28dd0b00
2014-06-12 14:46:14.950146 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_op osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:14.950173 7f9d683e1700 15 getattr 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 '_'
2014-06-12 14:46:14.950404 7f9d683e1700 10 getattr 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 '_' = 0
2014-06-12 14:46:14.950424 7f9d683e1700 15 getattr 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 'snapset'
2014-06-12 14:46:14.950513 7f9d683e1700 10 getattr 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 'snapset' = 0
2014-06-12 14:46:14.950528 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] populate_obc_watchers 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.950541 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.950551 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] get_object_context: creating obc from disk: 0x1771f340
2014-06-12 14:46:14.950566 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] get_object_context: 0x1771f340 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 rwstate(none n=0 w=0) oi: 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3(173'2853 client.4284.0:141479 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2853) ssc: 0x2ef8ed20 snapset: 0=[]:[]+head
2014-06-12 14:46:14.950585 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] find_object_context 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 @head oi=627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3(173'2853 client.4284.0:141479 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2853)
2014-06-12 14:46:14.950610 7f9d683e1700  5 -- op tracker -- , seq: 747681, time: 2014-06-12 14:46:14.950605, event: started, op: osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.950620 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] execute_ctx 0x2963a600
2014-06-12 14:46:14.950637 7f9d683e1700  5 -- op tracker -- , seq: 747681, time: 2014-06-12 14:46:14.950632, event: started, op: osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.950646 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_op 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] ov 173'2853 av 173'2854 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:14.950662 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_osd_op 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192]
2014-06-12 14:46:14.950673 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:14.950689 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_osd_op  write 1441792~1032192
2014-06-12 14:46:14.950706 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_osd_op_effects on session 0x5c8ce00
2014-06-12 14:46:14.950717 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] make_writeable 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 snapset=0x2ef8ed58  snapc=0=[]
2014-06-12 14:46:14.950729 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] make_writeable 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:14.950741 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] finish_ctx 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 0x2963a600 op modify  
2014-06-12 14:46:14.950755 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  set mtime to 2014-06-12 08:45:17.326156
2014-06-12 14:46:14.950777 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  final snapset 0=[]:[]+head in 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.950802 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  zeroing write result code 0
2014-06-12 14:46:14.950815 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] new_repop rep_tid 236229 on osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:14.950831 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] issue_repop rep_tid 236229 o 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.950857 7f9d683e1700  5 -- op tracker -- , seq: 747681, time: 2014-06-12 14:46:14.950852, event: waiting for subops from 29, op: osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.950883 7f9d683e1700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:14.950906 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2853 (0'0,173'2853] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] append_log log((0'0,173'2853], crt=0'0) [173'2854 (173'2853) modify   627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 by client.4284.0:141480 2014-06-12 08:45:17.326156]
2014-06-12 14:46:14.950932 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] add_log_entry 173'2854 (173'2853) modify   627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 by client.4284.0:141480 2014-06-12 08:45:17.326156
2014-06-12 14:46:14.950952 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] append_log  adding 1 keys
2014-06-12 14:46:14.950991 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2854, trimmed: 
2014-06-12 14:46:14.951018 7f9d683e1700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:46:14.951023 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:14.951025 7f9d683e1700  5 queue_op 0xff14480 seq 2621 osr(3.4f0 0x276db90) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:14.951033 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] eval_repop repgather(0x51b2c00 173'2854 rep_tid=236229 committed?=0 applied?=0 lock=0 op=osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.951053 7f9d683e1700 10 osd.2 173 dequeue_op 0x28dd0b00 finish
2014-06-12 14:46:14.951058 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.951062 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.951071 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (1 active)
2014-06-12 14:46:14.951077 7f9d82171700  5 _do_op 0xff14480 seq 2621 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:46:14.951082 7f9d82171700 10 _do_transaction on 0x5352000
2014-06-12 14:46:14.951089 7f9d82171700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.951186 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.951211 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:14.951256 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:14.951276 7f9d82171700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.951281 7f9d82171700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:46:14.951287 7f9d82171700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.951299 7f9d82171700 15 _write 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 1441792~1032192
2014-06-12 14:46:14.951426 7f9d82171700 10 lookup_strip_header done  cid 3.4f0_head oid 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.951515 7f9d523da700  5 -- op tracker -- , seq: 747682, time: 2014-06-12 14:46:14.950067, event: header_read, op: osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.951529 7f9d523da700  5 -- op tracker -- , seq: 747682, time: 2014-06-12 14:46:14.950069, event: throttled, op: osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.951537 7f9d523da700  5 -- op tracker -- , seq: 747682, time: 2014-06-12 14:46:14.951479, event: all_read, op: osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.951546 7f9d523da700  5 -- op tracker -- , seq: 747682, time: 0.000000, event: dispatched, op: osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.951554 7f9d523da700 20 osd.2 173 should_share_map client.4284 172.16.96.220:0/1241415 173
2014-06-12 14:46:14.951581 7f9d523da700 15 osd.2 173 enqueue_op 0xe20a800 prio 63 cost 516096 latency 0.001514 osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:14.951618 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.951626 7f9d65bdc700 10 osd.2 173 dequeue_op 0xe20a800 prio 63 cost 516096 latency 0.001559 osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]
2014-06-12 14:46:14.951655 7f9d65bdc700  5 -- op tracker -- , seq: 747682, time: 2014-06-12 14:46:14.951649, event: reached_pg, op: osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.951672 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:14.951689 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] handle_message: 0xe20a800
2014-06-12 14:46:14.951701 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_op osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:14.951725 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] get_object_context: found obc in cache: 0x1771f340
2014-06-12 14:46:14.951737 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] get_object_context: 0x1771f340 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 rwstate(write n=1 w=0) oi: 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3(173'2854 client.4284.0:141480 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2854) ssc: 0x2ef8ed20 snapset: 0=[]:[]+head
2014-06-12 14:46:14.951753 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] find_object_context 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 @head oi=627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3(173'2854 client.4284.0:141480 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2854)
2014-06-12 14:46:14.951788 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:14.951778 7f9d65bdc700  5 -- op tracker -- , seq: 747682, time: 2014-06-12 14:46:14.951773, event: started, op: osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.951793 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] execute_ctx 0x27bc2400
2014-06-12 14:46:14.951811 7f9d65bdc700  5 -- op tracker -- , seq: 747682, time: 2014-06-12 14:46:14.951805, event: started, op: osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.951819 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_op 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] ov 173'2854 av 173'2855 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:14.951847 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_osd_op 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096]
2014-06-12 14:46:14.951859 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:14.951872 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_osd_op  write 2473984~516096
2014-06-12 14:46:14.951887 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_osd_op_effects on session 0x5c8ce00
2014-06-12 14:46:14.951898 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] make_writeable 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 snapset=0x2ef8ed58  snapc=0=[]
2014-06-12 14:46:14.951910 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] make_writeable 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:14.951923 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] finish_ctx 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 0x27bc2400 op modify  
2014-06-12 14:46:14.951936 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  set mtime to 2014-06-12 08:45:17.329952
2014-06-12 14:46:14.951976 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  final snapset 0=[]:[]+head in 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.951995 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  zeroing write result code 0
2014-06-12 14:46:14.952006 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] new_repop rep_tid 236230 on osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:14.952022 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] issue_repop rep_tid 236230 o 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.952046 7f9d65bdc700  5 -- op tracker -- , seq: 747682, time: 2014-06-12 14:46:14.952041, event: waiting for subops from 29, op: osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.952078 7f9d65bdc700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:14.952100 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2854 (0'0,173'2854] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] append_log log((0'0,173'2854], crt=0'0) [173'2855 (173'2854) modify   627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 by client.4284.0:141481 2014-06-12 08:45:17.329952]
2014-06-12 14:46:14.952124 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] add_log_entry 173'2855 (173'2854) modify   627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 by client.4284.0:141481 2014-06-12 08:45:17.329952
2014-06-12 14:46:14.952145 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] append_log  adding 1 keys
2014-06-12 14:46:14.952176 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2855, trimmed: 
2014-06-12 14:46:14.952216 7f9d65bdc700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:46:14.952221 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:14.952222 7f9d65bdc700  5 queue_op 0x802f320 seq 2622 osr(3.4f0 0x276db90) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:46:14.952230 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] eval_repop repgather(0xc9b8780 173'2855 rep_tid=236230 committed?=0 applied?=0 lock=0 op=osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.952249 7f9d65bdc700 10 osd.2 173 dequeue_op 0xe20a800 finish
2014-06-12 14:46:14.952253 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.952256 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.952265 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (2 active)
2014-06-12 14:46:14.955532 7f9d82171700 10 _generic_write 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 1441792~1032192 = 0
2014-06-12 14:46:14.955628 7f9d82171700 15 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.955645 7f9d82171700 10 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 = 0
2014-06-12 14:46:14.955653 7f9d82171700 15 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.955658 7f9d82171700 10 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 = 0
2014-06-12 14:46:14.955879 7f9d82171700 20 set_header setting 17378 cid 3.4f0_head oid 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 parent seq 0
2014-06-12 14:46:14.955976 7f9d82171700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:46:14.955988 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:14.955993 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:14.963852 7f9d5501e700  5 -- op tracker -- , seq: 747683, time: 2014-06-12 14:46:14.963789, event: header_read, op: osd_sub_op_reply(client.4284.0:141480 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.963869 7f9d5501e700  5 -- op tracker -- , seq: 747683, time: 2014-06-12 14:46:14.963791, event: throttled, op: osd_sub_op_reply(client.4284.0:141480 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.963877 7f9d5501e700  5 -- op tracker -- , seq: 747683, time: 2014-06-12 14:46:14.963807, event: all_read, op: osd_sub_op_reply(client.4284.0:141480 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.963885 7f9d5501e700  5 -- op tracker -- , seq: 747683, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4284.0:141480 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.963891 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4284.0:141480 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:14.963898 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:14.963908 7f9d5501e700 15 osd.2 173 enqueue_op 0x2a902100 prio 196 cost 0 latency 0.000119 osd_sub_op_reply(client.4284.0:141480 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:14.963945 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.963956 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a902100 prio 196 cost 0 latency 0.000167 osd_sub_op_reply(client.4284.0:141480 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]
2014-06-12 14:46:14.963980 7f9d64bda700  5 -- op tracker -- , seq: 747683, time: 2014-06-12 14:46:14.963977, event: reached_pg, op: osd_sub_op_reply(client.4284.0:141480 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.963988 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] handle_message: 0x2a902100
2014-06-12 14:46:14.964079 7f9d64bda700  5 -- op tracker -- , seq: 747683, time: 2014-06-12 14:46:14.964075, event: started, op: osd_sub_op_reply(client.4284.0:141480 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.964086 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] sub_op_modify_reply: tid 236229 op  ack_type 4 from 29
2014-06-12 14:46:14.964127 7f9d64bda700  5 -- op tracker -- , seq: 747681, time: 2014-06-12 14:46:14.964119, event: sub_op_commit_rec, op: osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.964136 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a902100 finish
2014-06-12 14:46:14.964144 7f9d64bda700  5 -- op tracker -- , seq: 747683, time: 2014-06-12 14:46:14.964140, event: done, op: osd_sub_op_reply(client.4284.0:141480 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.964161 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.964164 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.967610 7f9d5501e700  5 -- op tracker -- , seq: 747684, time: 2014-06-12 14:46:14.967557, event: header_read, op: osd_sub_op_reply(client.4284.0:141481 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.967622 7f9d5501e700  5 -- op tracker -- , seq: 747684, time: 2014-06-12 14:46:14.967559, event: throttled, op: osd_sub_op_reply(client.4284.0:141481 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.967634 7f9d5501e700  5 -- op tracker -- , seq: 747684, time: 2014-06-12 14:46:14.967575, event: all_read, op: osd_sub_op_reply(client.4284.0:141481 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.967642 7f9d5501e700  5 -- op tracker -- , seq: 747684, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4284.0:141481 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.967652 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4284.0:141481 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:14.967658 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:14.967666 7f9d5501e700 15 osd.2 173 enqueue_op 0x2d17be00 prio 196 cost 0 latency 0.000109 osd_sub_op_reply(client.4284.0:141481 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:14.968559 7f9d82171700 10 _do_op 0xff14480 seq 2621 r = 0, finisher 0x8061360 0x218d18c0
2014-06-12 14:46:14.968656 7f9d82171700 10 _finish_op 0xff14480 seq 2621 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:46:14.968671 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (1 active)
2014-06-12 14:46:14.968675 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:14.968659 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] op_commit: 236229
2014-06-12 14:46:14.968685 7f9d7d167700  5 _do_op 0x802f320 seq 2622 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:46:14.968692 7f9d7d167700 10 _do_transaction on 0xc9b9c80
2014-06-12 14:46:14.968689 7f9d7715b700  5 -- op tracker -- , seq: 747681, time: 2014-06-12 14:46:14.968680, event: op_commit, op: osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.968699 7f9d7d167700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.968699 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] repop_all_committed: repop tid 236229 all committed 
2014-06-12 14:46:14.968712 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean] eval_repop repgather(0x51b2c00 173'2854 rep_tid=236229 committed?=1 applied?=0 lock=0 op=osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.968730 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean] log_op_stats osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.000000 lat 0.021753
2014-06-12 14:46:14.968750 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean] publish_stats_to_osd 173:3709
2014-06-12 14:46:14.968761 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean]  sending commit on repgather(0x51b2c00 173'2854 rep_tid=236229 committed?=1 applied?=0 lock=0 op=osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) 0x22f29400
2014-06-12 14:46:14.968801 7f9d7715b700  5 -- op tracker -- , seq: 747681, time: 2014-06-12 14:46:14.968795, event: commit_sent, op: osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.968841 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean] op_applied: 236229
2014-06-12 14:46:14.968863 7f9d7795c700  5 -- op tracker -- , seq: 747681, time: 2014-06-12 14:46:14.968856, event: op_applied, op: osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.968878 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.968871 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean] op_applied on primary on version 173'2854
2014-06-12 14:46:14.968894 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean] repop_all_applied: repop tid 236229 all applied 
2014-06-12 14:46:14.968908 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:14.968905 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean] eval_repop repgather(0x51b2c00 173'2854 rep_tid=236229 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.968921 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean] log_op_stats osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.000000 lat 0.021945
2014-06-12 14:46:14.968937 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean] publish_stats_to_osd 173:3710
2014-06-12 14:46:14.968949 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean]  requeue_ops 
2014-06-12 14:46:14.968961 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:14.968960 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]  removing repgather(0x51b2c00 173'2854 rep_tid=236229 committed?=1 applied?=1 op=osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.968981 7f9d7d167700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.968987 7f9d7d167700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:46:14.968975 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]    q front is repgather(0x51b2c00 173'2854 rep_tid=236229 committed?=1 applied?=1 op=osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.968993 7f9d7d167700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.968991 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] remove_repop repgather(0x51b2c00 173'2854 rep_tid=236229 committed?=1 applied?=1 op=osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.969010 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]  requeue_ops 
2014-06-12 14:46:14.969031 7f9d7795c700  5 -- op tracker -- , seq: 747681, time: 2014-06-12 14:46:14.969026, event: done, op: osd_op(client.4284.0:141480 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.969188 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.969200 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2d17be00 prio 196 cost 0 latency 0.001643 osd_sub_op_reply(client.4284.0:141481 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]
2014-06-12 14:46:14.969221 7f9d6abe6700  5 -- op tracker -- , seq: 747684, time: 2014-06-12 14:46:14.969218, event: reached_pg, op: osd_sub_op_reply(client.4284.0:141481 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.969230 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] handle_message: 0x2d17be00
2014-06-12 14:46:14.969243 7f9d6abe6700  5 -- op tracker -- , seq: 747684, time: 2014-06-12 14:46:14.969240, event: started, op: osd_sub_op_reply(client.4284.0:141481 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.969249 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] sub_op_modify_reply: tid 236230 op  ack_type 4 from 29
2014-06-12 14:46:14.969267 7f9d6abe6700  5 -- op tracker -- , seq: 747682, time: 2014-06-12 14:46:14.969260, event: sub_op_commit_rec, op: osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.969273 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2d17be00 finish
2014-06-12 14:46:14.969187 7f9d7d167700 15 _write 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 2473984~516096
2014-06-12 14:46:14.969428 7f9d7d167700 10 lookup_strip_header done  cid 3.4f0_head oid 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.969544 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:14.969279 7f9d6abe6700  5 -- op tracker -- , seq: 747684, time: 2014-06-12 14:46:14.969276, event: done, op: osd_sub_op_reply(client.4284.0:141481 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.969658 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.969662 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.971497 7f9d7d167700 10 _generic_write 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 2473984~516096 = 0
2014-06-12 14:46:14.971554 7f9d7d167700 15 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.971570 7f9d7d167700 10 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 = 0
2014-06-12 14:46:14.971577 7f9d7d167700 15 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.971584 7f9d7d167700 10 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 = 0
2014-06-12 14:46:14.971776 7f9d7d167700 20 set_header setting 17378 cid 3.4f0_head oid 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 parent seq 0
2014-06-12 14:46:14.971841 7f9d7d167700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:46:14.971857 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:14.971865 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:14.974212 7f9d7d167700 10 _do_op 0x802f320 seq 2622 r = 0, finisher 0x89e7b20 0x1a3f0d00
2014-06-12 14:46:14.974236 7f9d7d167700 10 _finish_op 0x802f320 seq 2622 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:46:14.974247 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (0 active)
2014-06-12 14:46:14.974256 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:14.974244 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] op_commit: 236230
2014-06-12 14:46:14.974281 7f9d7715b700  5 -- op tracker -- , seq: 747682, time: 2014-06-12 14:46:14.974270, event: op_commit, op: osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.974300 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] repop_all_committed: repop tid 236230 all committed 
2014-06-12 14:46:14.974314 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean] eval_repop repgather(0xc9b8780 173'2855 rep_tid=236230 committed?=1 applied?=0 lock=0 op=osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.974332 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean] log_op_stats osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.024264
2014-06-12 14:46:14.974351 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean] publish_stats_to_osd 173:3711
2014-06-12 14:46:14.974362 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean]  sending commit on repgather(0xc9b8780 173'2855 rep_tid=236230 committed?=1 applied?=0 lock=0 op=osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) 0x151e7900
2014-06-12 14:46:14.974422 7f9d7715b700  5 -- op tracker -- , seq: 747682, time: 2014-06-12 14:46:14.974416, event: commit_sent, op: osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.974437 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean] op_applied: 236230
2014-06-12 14:46:14.974461 7f9d7795c700  5 -- op tracker -- , seq: 747682, time: 2014-06-12 14:46:14.974455, event: op_applied, op: osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.974468 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean] op_applied on primary on version 173'2855
2014-06-12 14:46:14.974478 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean] repop_all_applied: repop tid 236230 all applied 
2014-06-12 14:46:14.974485 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean] eval_repop repgather(0xc9b8780 173'2855 rep_tid=236230 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.974495 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean] log_op_stats osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.024427
2014-06-12 14:46:14.974504 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean] publish_stats_to_osd 173:3712
2014-06-12 14:46:14.974511 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean]  requeue_ops 
2014-06-12 14:46:14.974517 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  removing repgather(0xc9b8780 173'2855 rep_tid=236230 committed?=1 applied?=1 op=osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.974526 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]    q front is repgather(0xc9b8780 173'2855 rep_tid=236230 committed?=1 applied?=1 op=osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.974534 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] remove_repop repgather(0xc9b8780 173'2855 rep_tid=236230 committed?=1 applied?=1 op=osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.974542 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  requeue_ops 
2014-06-12 14:46:14.974561 7f9d7795c700  5 -- op tracker -- , seq: 747682, time: 2014-06-12 14:46:14.974558, event: done, op: osd_op(client.4284.0:141481 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2473984~516096] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.975504 7f9d523da700  5 -- op tracker -- , seq: 747685, time: 2014-06-12 14:46:14.972444, event: header_read, op: osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.975518 7f9d523da700  5 -- op tracker -- , seq: 747685, time: 2014-06-12 14:46:14.972447, event: throttled, op: osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.975528 7f9d523da700  5 -- op tracker -- , seq: 747685, time: 2014-06-12 14:46:14.975466, event: all_read, op: osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.975538 7f9d523da700  5 -- op tracker -- , seq: 747685, time: 0.000000, event: dispatched, op: osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.975547 7f9d523da700 20 osd.2 173 should_share_map client.4284 172.16.96.220:0/1241415 173
2014-06-12 14:46:14.975576 7f9d523da700 15 osd.2 173 enqueue_op 0x2a902d00 prio 63 cost 1032192 latency 0.003132 osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:14.975611 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.975622 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2a902d00 prio 63 cost 1032192 latency 0.003178 osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]
2014-06-12 14:46:14.975651 7f9d6a3e5700  5 -- op tracker -- , seq: 747685, time: 2014-06-12 14:46:14.975645, event: reached_pg, op: osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.975662 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:14.975680 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] handle_message: 0x2a902d00
2014-06-12 14:46:14.975693 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_op osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:14.975720 7f9d6a3e5700 15 getattr 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 '_'
2014-06-12 14:46:14.975942 7f9d6a3e5700 10 getattr 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 '_' = 0
2014-06-12 14:46:14.975961 7f9d6a3e5700 15 getattr 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 'snapset'
2014-06-12 14:46:14.976050 7f9d6a3e5700 10 getattr 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 'snapset' = 0
2014-06-12 14:46:14.976061 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] populate_obc_watchers 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.976075 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.976088 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] get_object_context: creating obc from disk: 0x120ebb80
2014-06-12 14:46:14.976100 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] get_object_context: 0x120ebb80 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 rwstate(none n=0 w=0) oi: 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3(173'2855 client.4284.0:141481 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2855) ssc: 0x5bb89a0 snapset: 0=[]:[]+head
2014-06-12 14:46:14.976118 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] find_object_context 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 @head oi=627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3(173'2855 client.4284.0:141481 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2855)
2014-06-12 14:46:14.976149 7f9d6a3e5700  5 -- op tracker -- , seq: 747685, time: 2014-06-12 14:46:14.976143, event: started, op: osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.976159 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] execute_ctx 0xfd19200
2014-06-12 14:46:14.976179 7f9d6a3e5700  5 -- op tracker -- , seq: 747685, time: 2014-06-12 14:46:14.976173, event: started, op: osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.976187 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_op 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] ov 173'2855 av 173'2856 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:14.976203 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192]
2014-06-12 14:46:14.976217 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:14.976232 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op  write 2990080~1032192
2014-06-12 14:46:14.976250 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op_effects on session 0x5c8ce00
2014-06-12 14:46:14.976263 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] make_writeable 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 snapset=0x5bb89d8  snapc=0=[]
2014-06-12 14:46:14.976275 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] make_writeable 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:14.976287 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] finish_ctx 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 0xfd19200 op modify  
2014-06-12 14:46:14.976302 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  set mtime to 2014-06-12 08:45:17.351541
2014-06-12 14:46:14.976324 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  final snapset 0=[]:[]+head in 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.976345 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  zeroing write result code 0
2014-06-12 14:46:14.976358 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] new_repop rep_tid 236231 on osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:14.976377 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] issue_repop rep_tid 236231 o 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.976403 7f9d6a3e5700  5 -- op tracker -- , seq: 747685, time: 2014-06-12 14:46:14.976398, event: waiting for subops from 29, op: osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.976423 7f9d523da700  5 -- op tracker -- , seq: 747686, time: 2014-06-12 14:46:14.975885, event: header_read, op: osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.976428 7f9d6a3e5700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:14.976436 7f9d523da700  5 -- op tracker -- , seq: 747686, time: 2014-06-12 14:46:14.975887, event: throttled, op: osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.976446 7f9d523da700  5 -- op tracker -- , seq: 747686, time: 2014-06-12 14:46:14.976387, event: all_read, op: osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.976457 7f9d523da700  5 -- op tracker -- , seq: 747686, time: 0.000000, event: dispatched, op: osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.976463 7f9d523da700 20 osd.2 173 should_share_map client.4284 172.16.96.220:0/1241415 173
2014-06-12 14:46:14.976477 7f9d523da700 15 osd.2 173 enqueue_op 0x59dd700 prio 63 cost 172032 latency 0.000592 osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:14.976514 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:14.976465 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2855 (0'0,173'2855] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] append_log log((0'0,173'2855], crt=0'0) [173'2856 (173'2855) modify   627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 by client.4284.0:141482 2014-06-12 08:45:17.351541]
2014-06-12 14:46:14.976529 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] add_log_entry 173'2856 (173'2855) modify   627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 by client.4284.0:141482 2014-06-12 08:45:17.351541
2014-06-12 14:46:14.976553 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] append_log  adding 1 keys
2014-06-12 14:46:14.976584 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2856, trimmed: 
2014-06-12 14:46:14.976610 7f9d6a3e5700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:46:14.976615 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:14.976617 7f9d6a3e5700  5 queue_op 0xa2dad80 seq 2623 osr(3.4f0 0x276db90) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:14.976626 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] eval_repop repgather(0x1a444840 173'2856 rep_tid=236231 committed?=0 applied?=0 lock=0 op=osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.976647 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2a902d00 finish
2014-06-12 14:46:14.976655 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:14.976661 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.976670 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (1 active)
2014-06-12 14:46:14.976675 7f9d8116f700  5 _do_op 0xa2dad80 seq 2623 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:46:14.976660 7f9d66bde700 10 osd.2 173 dequeue_op 0x59dd700 prio 63 cost 172032 latency 0.000775 osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]
2014-06-12 14:46:14.976679 7f9d8116f700 10 _do_transaction on 0x514cd80
2014-06-12 14:46:14.976686 7f9d8116f700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.976687 7f9d66bde700  5 -- op tracker -- , seq: 747686, time: 2014-06-12 14:46:14.976681, event: reached_pg, op: osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.976697 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:14.976711 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] handle_message: 0x59dd700
2014-06-12 14:46:14.976721 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_op osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:14.976742 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] get_object_context: found obc in cache: 0x120ebb80
2014-06-12 14:46:14.976753 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] get_object_context: 0x120ebb80 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 rwstate(write n=1 w=0) oi: 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3(173'2856 client.4284.0:141482 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2856) ssc: 0x5bb89a0 snapset: 0=[]:[]+head
2014-06-12 14:46:14.976776 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.976767 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] find_object_context 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 @head oi=627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3(173'2856 client.4284.0:141482 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2856)
2014-06-12 14:46:14.976789 7f9d66bde700  5 -- op tracker -- , seq: 747686, time: 2014-06-12 14:46:14.976784, event: started, op: osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.976796 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] execute_ctx 0x1a916c00
2014-06-12 14:46:14.976803 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:14.976813 7f9d66bde700  5 -- op tracker -- , seq: 747686, time: 2014-06-12 14:46:14.976808, event: started, op: osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.976820 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_op 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] ov 173'2856 av 173'2857 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:14.976833 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032]
2014-06-12 14:46:14.976844 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:14.976851 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:14.976857 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op  write 4022272~172032
2014-06-12 14:46:14.976873 7f9d8116f700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.976870 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op_effects on session 0x5c8ce00
2014-06-12 14:46:14.976879 7f9d8116f700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:46:14.976885 7f9d8116f700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.976881 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] make_writeable 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 snapset=0x5bb89d8  snapc=0=[]
2014-06-12 14:46:14.976901 7f9d8116f700 15 _write 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 2990080~1032192
2014-06-12 14:46:14.976896 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] make_writeable 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:14.976906 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] finish_ctx 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 0x1a916c00 op modify  
2014-06-12 14:46:14.976918 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  set mtime to 2014-06-12 08:45:17.356541
2014-06-12 14:46:14.976941 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  final snapset 0=[]:[]+head in 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.976958 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  zeroing write result code 0
2014-06-12 14:46:14.976968 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] new_repop rep_tid 236232 on osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:14.976982 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] issue_repop rep_tid 236232 o 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.977004 7f9d66bde700  5 -- op tracker -- , seq: 747686, time: 2014-06-12 14:46:14.976999, event: waiting for subops from 29, op: osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.977024 7f9d66bde700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:14.977036 7f9d8116f700 10 lookup_strip_header done  cid 3.4f0_head oid 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.977040 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2856 (0'0,173'2856] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] append_log log((0'0,173'2856], crt=0'0) [173'2857 (173'2856) modify   627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 by client.4284.0:141483 2014-06-12 08:45:17.356541]
2014-06-12 14:46:14.977060 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:14.977062 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] add_log_entry 173'2857 (173'2856) modify   627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 by client.4284.0:141483 2014-06-12 08:45:17.356541
2014-06-12 14:46:14.977082 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] append_log  adding 1 keys
2014-06-12 14:46:14.977113 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2857, trimmed: 
2014-06-12 14:46:14.977135 7f9d66bde700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:46:14.977139 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:14.977140 7f9d66bde700  5 queue_op 0x5ae60c0 seq 2624 osr(3.4f0 0x276db90) 174325 bytes   (queue has 2 ops and 1208810 bytes)
2014-06-12 14:46:14.977148 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] eval_repop repgather(0xe832840 173'2857 rep_tid=236232 committed?=0 applied?=0 lock=0 op=osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.977166 7f9d66bde700 10 osd.2 173 dequeue_op 0x59dd700 finish
2014-06-12 14:46:14.977171 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.977174 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.977277 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (2 active)
2014-06-12 14:46:14.981594 7f9d8116f700 10 _generic_write 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 2990080~1032192 = 0
2014-06-12 14:46:14.981733 7f9d8116f700 15 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.981751 7f9d8116f700 10 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 = 0
2014-06-12 14:46:14.981760 7f9d8116f700 15 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.981768 7f9d8116f700 10 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 = 0
2014-06-12 14:46:14.982061 7f9d8116f700 20 set_header setting 17378 cid 3.4f0_head oid 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 parent seq 0
2014-06-12 14:46:14.982180 7f9d8116f700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:46:14.982198 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:14.982209 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:14.987651 7f9d8116f700 10 _do_op 0xa2dad80 seq 2623 r = 0, finisher 0x6d36780 0xcb60380
2014-06-12 14:46:14.987670 7f9d8116f700 10 _finish_op 0xa2dad80 seq 2623 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:46:14.987682 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (1 active)
2014-06-12 14:46:14.987699 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:14.987675 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] op_commit: 236231
2014-06-12 14:46:14.987700 7f9d79960700  5 _do_op 0x5ae60c0 seq 2624 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:46:14.987706 7f9d79960700 10 _do_transaction on 0x1b72180
2014-06-12 14:46:14.987714 7f9d79960700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.987712 7f9d7715b700  5 -- op tracker -- , seq: 747685, time: 2014-06-12 14:46:14.987704, event: op_commit, op: osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.987726 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] op_applied: 236231
2014-06-12 14:46:14.987750 7f9d7795c700  5 -- op tracker -- , seq: 747685, time: 2014-06-12 14:46:14.987743, event: op_applied, op: osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.987761 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] op_applied on primary on version 173'2856
2014-06-12 14:46:14.987806 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.987828 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:14.987861 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:14.987877 7f9d79960700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.987882 7f9d79960700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:46:14.987886 7f9d79960700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:46:14.987884 7f9d5501e700  5 -- op tracker -- , seq: 747687, time: 2014-06-12 14:46:14.987855, event: header_read, op: osd_sub_op_reply(client.4284.0:141482 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.987891 7f9d5501e700  5 -- op tracker -- , seq: 747687, time: 2014-06-12 14:46:14.987856, event: throttled, op: osd_sub_op_reply(client.4284.0:141482 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.987896 7f9d5501e700  5 -- op tracker -- , seq: 747687, time: 2014-06-12 14:46:14.987866, event: all_read, op: osd_sub_op_reply(client.4284.0:141482 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.987900 7f9d5501e700  5 -- op tracker -- , seq: 747687, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4284.0:141482 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.987904 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4284.0:141482 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:14.987908 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:14.987914 7f9d5501e700 15 osd.2 173 enqueue_op 0x2d17a100 prio 196 cost 0 latency 0.000058 osd_sub_op_reply(client.4284.0:141482 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:14.987939 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.987948 7f9d69be4700 10 osd.2 173 dequeue_op 0x2d17a100 prio 196 cost 0 latency 0.000093 osd_sub_op_reply(client.4284.0:141482 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]
2014-06-12 14:46:14.987968 7f9d69be4700  5 -- op tracker -- , seq: 747687, time: 2014-06-12 14:46:14.987964, event: reached_pg, op: osd_sub_op_reply(client.4284.0:141482 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.987976 7f9d79960700 15 _write 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 4022272~172032
2014-06-12 14:46:14.987979 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] handle_message: 0x2d17a100
2014-06-12 14:46:14.987997 7f9d69be4700  5 -- op tracker -- , seq: 747687, time: 2014-06-12 14:46:14.987994, event: started, op: osd_sub_op_reply(client.4284.0:141482 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.988005 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] sub_op_modify_reply: tid 236231 op  ack_type 4 from 29
2014-06-12 14:46:14.988024 7f9d69be4700  5 -- op tracker -- , seq: 747685, time: 2014-06-12 14:46:14.988018, event: sub_op_commit_rec, op: osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.988033 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] repop_all_applied: repop tid 236231 all applied 
2014-06-12 14:46:14.988045 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] eval_repop repgather(0x1a444840 173'2856 rep_tid=236231 committed?=0 applied?=1 lock=0 op=osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.988063 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  sending ack on repgather(0x1a444840 173'2856 rep_tid=236231 committed?=0 applied?=1 lock=0 op=osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) 0x2b55b80
2014-06-12 14:46:14.988097 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] repop_all_committed: repop tid 236231 all committed 
2014-06-12 14:46:14.988120 7f9d79960700 10 lookup_strip_header done  cid 3.4f0_head oid 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.988108 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2854 active+clean] eval_repop repgather(0x1a444840 173'2856 rep_tid=236231 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.988132 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2854 active+clean] log_op_stats osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.015651 lat 0.015687
2014-06-12 14:46:14.988150 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2854 active+clean] publish_stats_to_osd 173:3713
2014-06-12 14:46:14.988163 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2854 active+clean]  sending commit on repgather(0x1a444840 173'2856 rep_tid=236231 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) 0xee35900
2014-06-12 14:46:14.988193 7f9d69be4700  5 -- op tracker -- , seq: 747685, time: 2014-06-12 14:46:14.988188, event: commit_sent, op: osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.988202 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2854 active+clean]  requeue_ops 
2014-06-12 14:46:14.988212 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]  removing repgather(0x1a444840 173'2856 rep_tid=236231 committed?=1 applied?=1 op=osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.988226 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]    q front is repgather(0x1a444840 173'2856 rep_tid=236231 committed?=1 applied?=1 op=osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.988243 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:14.988240 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] remove_repop repgather(0x1a444840 173'2856 rep_tid=236231 committed?=1 applied?=1 op=osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.988255 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]  requeue_ops 
2014-06-12 14:46:14.988274 7f9d69be4700  5 -- op tracker -- , seq: 747685, time: 2014-06-12 14:46:14.988269, event: done, op: osd_op(client.4284.0:141482 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~1032192] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.988292 7f9d69be4700 10 osd.2 173 dequeue_op 0x2d17a100 finish
2014-06-12 14:46:14.988301 7f9d69be4700  5 -- op tracker -- , seq: 747687, time: 2014-06-12 14:46:14.988297, event: done, op: osd_sub_op_reply(client.4284.0:141482 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.988312 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.988315 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.988919 7f9d79960700 10 _generic_write 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 4022272~172032 = 0
2014-06-12 14:46:14.988937 7f9d79960700 15 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.988945 7f9d79960700 10 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 = 0
2014-06-12 14:46:14.988951 7f9d79960700 15 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3
2014-06-12 14:46:14.988956 7f9d79960700 10 _setattrs 3.4f0_head/627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 = 0
2014-06-12 14:46:14.989041 7f9d79960700 20 set_header setting 17378 cid 3.4f0_head oid 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 parent seq 0
2014-06-12 14:46:14.989061 7f9d79960700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:46:14.989070 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:14.989076 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:14.989750 7f9d5501e700  5 -- op tracker -- , seq: 747688, time: 2014-06-12 14:46:14.989710, event: header_read, op: osd_sub_op_reply(client.4284.0:141483 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.989760 7f9d5501e700  5 -- op tracker -- , seq: 747688, time: 2014-06-12 14:46:14.989711, event: throttled, op: osd_sub_op_reply(client.4284.0:141483 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.989768 7f9d5501e700  5 -- op tracker -- , seq: 747688, time: 2014-06-12 14:46:14.989724, event: all_read, op: osd_sub_op_reply(client.4284.0:141483 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.989775 7f9d5501e700  5 -- op tracker -- , seq: 747688, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4284.0:141483 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.989780 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4284.0:141483 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:14.989785 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:14.989792 7f9d5501e700 15 osd.2 173 enqueue_op 0x876fa00 prio 196 cost 0 latency 0.000082 osd_sub_op_reply(client.4284.0:141483 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:14.989818 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:14.989825 7f9d653db700 10 osd.2 173 dequeue_op 0x876fa00 prio 196 cost 0 latency 0.000115 osd_sub_op_reply(client.4284.0:141483 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]
2014-06-12 14:46:14.989848 7f9d653db700  5 -- op tracker -- , seq: 747688, time: 2014-06-12 14:46:14.989844, event: reached_pg, op: osd_sub_op_reply(client.4284.0:141483 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.989858 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] handle_message: 0x876fa00
2014-06-12 14:46:14.989875 7f9d653db700  5 -- op tracker -- , seq: 747688, time: 2014-06-12 14:46:14.989871, event: started, op: osd_sub_op_reply(client.4284.0:141483 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.989880 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] sub_op_modify_reply: tid 236232 op  ack_type 4 from 29
2014-06-12 14:46:14.989901 7f9d653db700  5 -- op tracker -- , seq: 747686, time: 2014-06-12 14:46:14.989894, event: sub_op_commit_rec, op: osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.989909 7f9d653db700 10 osd.2 173 dequeue_op 0x876fa00 finish
2014-06-12 14:46:14.989915 7f9d653db700  5 -- op tracker -- , seq: 747688, time: 2014-06-12 14:46:14.989912, event: done, op: osd_sub_op_reply(client.4284.0:141483 3.4f0 627624f0/rbd_data.10936b8b4567.0000000000000fdf/head//3 [] ondisk, result = 0)
2014-06-12 14:46:14.989926 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:14.989930 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:14.990141 7f9d79960700 10 _do_op 0x5ae60c0 seq 2624 r = 0, finisher 0x72f67a0 0x7a76080
2014-06-12 14:46:14.990150 7f9d79960700 10 _finish_op 0x5ae60c0 seq 2624 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:46:14.990156 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (0 active)
2014-06-12 14:46:14.990158 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:14.990154 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] op_commit: 236232
2014-06-12 14:46:14.990174 7f9d7715b700  5 -- op tracker -- , seq: 747686, time: 2014-06-12 14:46:14.990169, event: op_commit, op: osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.990190 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] repop_all_committed: repop tid 236232 all committed 
2014-06-12 14:46:14.990198 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] eval_repop repgather(0xe832840 173'2857 rep_tid=236232 committed?=1 applied?=0 lock=0 op=osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.990210 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] log_op_stats osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 inb 172311 outb 0 rlat 0.000000 lat 0.014324
2014-06-12 14:46:14.990229 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] publish_stats_to_osd 173:3714
2014-06-12 14:46:14.990237 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean]  sending commit on repgather(0xe832840 173'2857 rep_tid=236232 committed?=1 applied?=0 lock=0 op=osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) 0x45df680
2014-06-12 14:46:14.990264 7f9d7715b700  5 -- op tracker -- , seq: 747686, time: 2014-06-12 14:46:14.990259, event: commit_sent, op: osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.990315 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] op_applied: 236232
2014-06-12 14:46:14.990338 7f9d7795c700  5 -- op tracker -- , seq: 747686, time: 2014-06-12 14:46:14.990332, event: op_applied, op: osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:14.990347 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] op_applied on primary on version 173'2857
2014-06-12 14:46:14.990356 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] repop_all_applied: repop tid 236232 all applied 
2014-06-12 14:46:14.990362 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] eval_repop repgather(0xe832840 173'2857 rep_tid=236232 committed?=1 applied?=1 lock=0 op=osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:14.990375 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] log_op_stats osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4 inb 172311 outb 0 rlat 0.000000 lat 0.014489
2014-06-12 14:46:14.990387 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] publish_stats_to_osd 173:3715
2014-06-12 14:46:14.990394 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean]  requeue_ops 
2014-06-12 14:46:14.990400 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  removing repgather(0xe832840 173'2857 rep_tid=236232 committed?=1 applied?=1 op=osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.990409 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]    q front is repgather(0xe832840 173'2857 rep_tid=236232 committed?=1 applied?=1 op=osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.990419 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] remove_repop repgather(0xe832840 173'2857 rep_tid=236232 committed?=1 applied?=1 op=osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:14.990427 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  requeue_ops 
2014-06-12 14:46:14.990448 7f9d7795c700  5 -- op tracker -- , seq: 747686, time: 2014-06-12 14:46:14.990445, event: done, op: osd_op(client.4284.0:141483 rbd_data.10936b8b4567.0000000000000fdf [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.627624f0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:15.381897 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:15.394178 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:15.412414 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:15.425894 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:15.433832 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:15.448987 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:15.452998 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:15.471810 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:15.471838 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:15.471860 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 13.09 >= max 0.5 = no, load too high
2014-06-12 14:46:15.471872 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:15.471875 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92623 < max (604800 seconds)
2014-06-12 14:46:15.471900 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:15.471903 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:15.471905 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:15.471907 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 17; oldest is 36.222988 seconds old
2014-06-12 14:46:15.471935 7f9d84175700  0 log [WRN] : 6 slow requests, 1 included below; oldest blocked for > 36.222988 secs
2014-06-12 14:46:15.471940 7f9d84175700  0 log [WRN] : slow request 30.144081 seconds old, received at 2014-06-12 14:45:45.327826: osd_op(client.4309.0:141528 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:15.585038 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:15.615465 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19409 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,8,5])
2014-06-12 14:46:15.615486 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19409 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,8,5])
2014-06-12 14:46:15.615910 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:15.615944 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:15.615966 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:15.615998 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:15.616064 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:15.616105 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:15.616139 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:15.616161 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:15.616182 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:15.616204 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:15.616224 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:15.616252 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:15.616273 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:15.616293 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:15.616313 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:15.616333 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:15.616361 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:15.616391 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:15.616416 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:15.616436 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:15.616456 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:15.616501 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:15.818692 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:15.818699 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:15.818702 7f9d74bfa700 20 osd.2 173 _dispatch 0x646c820 ping v1
2014-06-12 14:46:15.818707 7f9d74bfa700 10 osd.2 173 ping from client.4303
2014-06-12 14:46:15.818712 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:15.818714 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:15.841127 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:15.841179 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:15.877924 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:15.917482 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:15.917536 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:16.006738 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.023749 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.030945 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:16.051145 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:16.051349 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:16.153490 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:16.294986 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:16.295026 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:16.321380 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:16.321382 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:16.321384 7f9d74bfa700 20 osd.2 173 _dispatch 0x1956d1e0 ping v1
2014-06-12 14:46:16.321385 7f9d74bfa700 10 osd.2 173 ping from client.4308
2014-06-12 14:46:16.321387 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:16.321387 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:16.367158 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:16.472008 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:16.472031 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:16.472033 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:16.472035 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:16.472036 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 17; oldest is 37.223117 seconds old
2014-06-12 14:46:16.472066 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:16.472074 7f9d84175700  0 log [WRN] : 7 slow requests, 1 included below; oldest blocked for > 37.223117 secs
2014-06-12 14:46:16.472078 7f9d84175700  0 log [WRN] : slow request 30.336340 seconds old, received at 2014-06-12 14:45:46.135696: osd_op(client.4309.0:141529 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3928064~266240] 3.f029e408 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:46:16.477912 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:16.477916 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:16.477918 7f9d74bfa700 20 osd.2 173 _dispatch 0x6b78340 ping v1
2014-06-12 14:46:16.477921 7f9d74bfa700 10 osd.2 173 ping from client.4309
2014-06-12 14:46:16.477924 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:16.477925 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:16.670759 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.677356 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:16.709699 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.720580 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:16.734174 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.743842 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:16.761918 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.770508 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:16.777948 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.783567 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:16.931012 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.934213 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.941030 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:16.943766 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.946391 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.947090 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:16.951135 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.952330 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.964230 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.968745 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:16.969728 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.974316 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:16.976734 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.977237 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.987760 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:16.988375 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.989990 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:16.990243 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.041929 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:17.041931 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:17.041932 7f9d74bfa700 20 osd.2 173 _dispatch 0x8d871e0 ping v1
2014-06-12 14:46:17.041934 7f9d74bfa700 10 osd.2 173 ping from client.4317
2014-06-12 14:46:17.041936 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:17.041936 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:17.174425 7f9d5b6bc700  5 -- op tracker -- , seq: 747689, time: 2014-06-12 14:46:17.172770, event: header_read, op: osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174441 7f9d5b6bc700  5 -- op tracker -- , seq: 747689, time: 2014-06-12 14:46:17.172772, event: throttled, op: osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174449 7f9d5b6bc700  5 -- op tracker -- , seq: 747689, time: 2014-06-12 14:46:17.174387, event: all_read, op: osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174458 7f9d5b6bc700  5 -- op tracker -- , seq: 747689, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174462 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:17.174469 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:17.174473 7f9d5a16f700  5 -- op tracker -- , seq: 747690, time: 2014-06-12 14:46:17.172839, event: header_read, op: osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174477 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x876e800 prio 127 cost 516903 latency 0.001706 osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.174484 7f9d5a16f700  5 -- op tracker -- , seq: 747690, time: 2014-06-12 14:46:17.172841, event: throttled, op: osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174494 7f9d5a16f700  5 -- op tracker -- , seq: 747690, time: 2014-06-12 14:46:17.174445, event: all_read, op: osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174502 7f9d5a16f700  5 -- op tracker -- , seq: 747690, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174502 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.174506 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:17.174512 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:17.174519 7f9d5a16f700 15 osd.2 173 enqueue_op 0x9facb00 prio 127 cost 516903 latency 0.001679 osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.174509 7f9d63bd8700 10 osd.2 173 dequeue_op 0x876e800 prio 127 cost 516903 latency 0.001739 osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.7fb( v 173'3020 (79'19,173'3020] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active]
2014-06-12 14:46:17.174533 7f9d63bd8700  5 -- op tracker -- , seq: 747689, time: 2014-06-12 14:46:17.174529, event: reached_pg, op: osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174545 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:17.174542 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3020 (79'19,173'3020] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active] handle_message: 0x876e800
2014-06-12 14:46:17.174553 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3020 (79'19,173'3020] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active] sub_op_modify trans ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 v 173'3021 (transaction) 174
2014-06-12 14:46:17.174550 7f9d68be2700 10 osd.2 173 dequeue_op 0x9facb00 prio 127 cost 516903 latency 0.001711 osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2658 (0'0,173'2658] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2657 active]
2014-06-12 14:46:17.174568 7f9d63bd8700  5 -- op tracker -- , seq: 747689, time: 2014-06-12 14:46:17.174564, event: started, op: osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174582 7f9d68be2700  5 -- op tracker -- , seq: 747690, time: 2014-06-12 14:46:17.174565, event: reached_pg, op: osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174583 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3020 (79'19,173'3020] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active] append_log log((79'19,173'3020], crt=0'0) [173'3021 (173'3020) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146053 2014-06-12 08:45:19.531033]
2014-06-12 14:46:17.174606 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3021 (79'19,173'3021] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active] add_log_entry 173'3021 (173'3020) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146053 2014-06-12 08:45:19.531033
2014-06-12 14:46:17.174590 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2658 (0'0,173'2658] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2657 active] handle_message: 0x9facb00
2014-06-12 14:46:17.174629 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3021 (79'19,173'3021] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active] append_log  adding 1 keys
2014-06-12 14:46:17.174627 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2658 (0'0,173'2658] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2657 active] sub_op_modify trans 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 v 173'2659 (transaction) 174
2014-06-12 14:46:17.174644 7f9d63bd8700 10 trim log((79'19,173'3021], crt=0'0) to 79'20
2014-06-12 14:46:17.174645 7f9d68be2700  5 -- op tracker -- , seq: 747690, time: 2014-06-12 14:46:17.174641, event: started, op: osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174672 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3021, trimmed: 79'20
2014-06-12 14:46:17.174664 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2658 (0'0,173'2658] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2657 active] append_log log((0'0,173'2658], crt=0'0) [173'2659 (173'2657) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141064 2014-06-12 08:43:33.052945]
2014-06-12 14:46:17.174686 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2659 (0'0,173'2659] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2657 active] add_log_entry 173'2659 (173'2657) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141064 2014-06-12 08:43:33.052945
2014-06-12 14:46:17.174699 7f9d63bd8700  5 -- op tracker -- , seq: 747689, time: 2014-06-12 14:46:17.174695, event: started, op: osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174708 7f9d63bd8700  5 queue_transactions existing osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:17.174712 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.174713 7f9d63bd8700  5 queue_op 0x8f0c9c0 seq 2701 osr(3.7fb 0x49b42f0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:17.174706 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2659 (0'0,173'2659] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2657 active] append_log  adding 1 keys
2014-06-12 14:46:17.174720 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3021 (79'20,173'3021] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active] do_sub_op osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.174734 7f9d63bd8700 10 osd.2 173 dequeue_op 0x876e800 finish
2014-06-12 14:46:17.174738 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:17.174741 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.174745 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2659, trimmed: 
2014-06-12 14:46:17.174755 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c57a0 (1 active)
2014-06-12 14:46:17.174760 7f9d8096e700  5 _do_op 0x8f0c9c0 seq 2701 osr(3.7fb 0x49b42f0)/0x49b42f0 start
2014-06-12 14:46:17.174764 7f9d8096e700 10 _do_transaction on 0x6238630
2014-06-12 14:46:17.174769 7f9d8096e700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.174781 7f9d68be2700  5 -- op tracker -- , seq: 747690, time: 2014-06-12 14:46:17.174763, event: started, op: osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.174790 7f9d68be2700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:17.174793 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.174794 7f9d68be2700  5 queue_op 0x2fa6a80 seq 2469 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 2 ops and 1036813 bytes)
2014-06-12 14:46:17.174800 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2659 (0'0,173'2659] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2657 active] do_sub_op osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.174809 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (2 active)
2014-06-12 14:46:17.174815 7f9d68be2700 10 osd.2 173 dequeue_op 0x9facb00 finish
2014-06-12 14:46:17.174818 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.174817 7f9d8016d700  5 _do_op 0x2fa6a80 seq 2469 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:17.174822 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.174824 7f9d8016d700 10 _do_transaction on 0x1cc29270
2014-06-12 14:46:17.174829 7f9d8016d700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:17.174939 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.174978 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.175057 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:17.175109 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.175114 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.175142 7f9d8096e700 15 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.175157 7f9d8096e700 10 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1 r = 0
2014-06-12 14:46:17.175161 7f9d8096e700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.175307 7f9d8096e700 15 _write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 1433600~516096
2014-06-12 14:46:17.175540 7f9d8096e700 10 lookup_strip_header done  cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.175836 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:17.178915 7f9d8096e700 10 _generic_write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 1433600~516096 = 0
2014-06-12 14:46:17.178967 7f9d8096e700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.178979 7f9d8096e700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:17.178985 7f9d8096e700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.178993 7f9d8096e700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:17.179183 7f9d8096e700 20 set_header setting 17147 cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 parent seq 0
2014-06-12 14:46:17.179255 7f9d8096e700 20 set_header setting 2206 cid meta oid 1fe728d6/pglog_3.7fb/0//-1 parent seq 0
2014-06-12 14:46:17.179272 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.179282 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:17.181539 7f9d8096e700 10 _do_op 0x8f0c9c0 seq 2701 r = 0, finisher 0x8060420 0
2014-06-12 14:46:17.181571 7f9d8096e700 10 _finish_op 0x8f0c9c0 seq 2701 osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:17.181585 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c57a0 (1 active)
2014-06-12 14:46:17.181624 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.181805 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.181845 7f9d8016d700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:17.181852 7f9d8016d700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:17.181860 7f9d8016d700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:17.182008 7f9d8016d700 15 _write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 1032192~516096
2014-06-12 14:46:17.182207 7f9d8016d700 10 lookup_strip_header done  cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:17.182386 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:17.184082 7f9d8016d700 10 _generic_write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 1032192~516096 = 0
2014-06-12 14:46:17.184142 7f9d8016d700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:17.184160 7f9d8016d700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:17.184169 7f9d8016d700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:17.184179 7f9d8016d700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:17.184327 7f9d8016d700 20 set_header setting 17215 cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 parent seq 0
2014-06-12 14:46:17.184390 7f9d8016d700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:17.184408 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.184417 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:17.181634 7f9d7795c700  5 -- op tracker -- , seq: 747689, time: 2014-06-12 14:46:17.181622, event: sub_op_applied, op: osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.185662 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3021 (79'20,173'3021] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active] sub_op_modify_applied on 0x6238540 op osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.185697 7f9d7795c700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:17.185717 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3021 (79'20,173'3021] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active] op_applied on primary on version 173'3021
2014-06-12 14:46:17.185727 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3021 (79'20,173'3021] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active] op_applied on replica on version 173'3021
2014-06-12 14:46:17.185793 7f9d7715b700  5 -- op tracker -- , seq: 747689, time: 2014-06-12 14:46:17.185786, event: commit_sent, op: osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.185801 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3021 (79'20,173'3021] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3019 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:17.185818 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:17.185856 7f9d7715b700  5 -- op tracker -- , seq: 747689, time: 2014-06-12 14:46:17.185851, event: done, op: osd_sub_op(client.4302.0:146053 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3021 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.186927 7f9d8016d700 10 _do_op 0x2fa6a80 seq 2469 r = 0, finisher 0x7f43280 0
2014-06-12 14:46:17.186972 7f9d7715b700  5 -- op tracker -- , seq: 747690, time: 2014-06-12 14:46:17.186960, event: commit_sent, op: osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.186984 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2659 (0'0,173'2659] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2657 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:17.187013 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:17.187042 7f9d8016d700 10 _finish_op 0x2fa6a80 seq 2469 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:17.187050 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:46:17.187059 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.187077 7f9d7795c700  5 -- op tracker -- , seq: 747690, time: 2014-06-12 14:46:17.187071, event: sub_op_applied, op: osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.187086 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2659 (0'0,173'2659] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2658 active] sub_op_modify_applied on 0x1cc29180 op osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.187104 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2659 (0'0,173'2659] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2658 active] op_applied on primary on version 173'2659
2014-06-12 14:46:17.187116 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2659 (0'0,173'2659] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2658 active] op_applied on replica on version 173'2659
2014-06-12 14:46:17.187138 7f9d7795c700  5 -- op tracker -- , seq: 747690, time: 2014-06-12 14:46:17.187133, event: done, op: osd_sub_op(client.4284.0:141064 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2659 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.192169 7f9d5b6bc700  5 -- op tracker -- , seq: 747691, time: 2014-06-12 14:46:17.191343, event: header_read, op: osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.192180 7f9d5b6bc700  5 -- op tracker -- , seq: 747691, time: 2014-06-12 14:46:17.191345, event: throttled, op: osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.192186 7f9d5b6bc700  5 -- op tracker -- , seq: 747691, time: 2014-06-12 14:46:17.192134, event: all_read, op: osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.192191 7f9d5b6bc700  5 -- op tracker -- , seq: 747691, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.192195 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:17.192200 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:17.192206 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x876e500 prio 127 cost 516903 latency 0.000863 osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.192232 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.192240 7f9d633d7700 10 osd.2 173 dequeue_op 0x876e500 prio 127 cost 516903 latency 0.000897 osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.7fb( v 173'3021 (79'20,173'3021] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3020 active]
2014-06-12 14:46:17.192259 7f9d633d7700  5 -- op tracker -- , seq: 747691, time: 2014-06-12 14:46:17.192256, event: reached_pg, op: osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.192269 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3021 (79'20,173'3021] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3020 active] handle_message: 0x876e500
2014-06-12 14:46:17.192278 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3021 (79'20,173'3021] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3020 active] sub_op_modify trans ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 v 173'3022 (transaction) 174
2014-06-12 14:46:17.192289 7f9d633d7700  5 -- op tracker -- , seq: 747691, time: 2014-06-12 14:46:17.192287, event: started, op: osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.192302 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3021 (79'20,173'3021] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3020 active] append_log log((79'20,173'3021], crt=0'0) [173'3022 (173'3021) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146054 2014-06-12 08:45:19.568751]
2014-06-12 14:46:17.192321 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3022 (79'20,173'3022] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3020 active] add_log_entry 173'3022 (173'3021) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146054 2014-06-12 08:45:19.568751
2014-06-12 14:46:17.192337 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3022 (79'20,173'3022] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3020 active] append_log  adding 1 keys
2014-06-12 14:46:17.192350 7f9d633d7700 10 trim log((79'20,173'3022], crt=0'0) to 79'21
2014-06-12 14:46:17.192371 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3022, trimmed: 79'21
2014-06-12 14:46:17.192391 7f9d633d7700  5 -- op tracker -- , seq: 747691, time: 2014-06-12 14:46:17.192388, event: started, op: osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.192400 7f9d633d7700  5 queue_transactions existing osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:17.192403 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.192404 7f9d633d7700  5 queue_op 0xda7c6c0 seq 2702 osr(3.7fb 0x49b42f0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:17.192412 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c57a0 (1 active)
2014-06-12 14:46:17.192409 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3022 (79'21,173'3022] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3020 active] do_sub_op osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.192417 7f9d7915f700  5 _do_op 0xda7c6c0 seq 2702 osr(3.7fb 0x49b42f0)/0x49b42f0 start
2014-06-12 14:46:17.192419 7f9d633d7700 10 osd.2 173 dequeue_op 0x876e500 finish
2014-06-12 14:46:17.192421 7f9d7915f700 10 _do_transaction on 0x62ea470
2014-06-12 14:46:17.192421 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.192423 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.192426 7f9d7915f700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.192527 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.192545 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.192576 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.192662 7f9d7915f700 15 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.192669 7f9d7915f700 10 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1 r = 0
2014-06-12 14:46:17.192673 7f9d7915f700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.192687 7f9d7915f700 15 _write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 1949696~516096
2014-06-12 14:46:17.192786 7f9d7915f700 10 lookup_strip_header done  cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.192930 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:17.194196 7f9d7915f700 10 _generic_write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 1949696~516096 = 0
2014-06-12 14:46:17.194240 7f9d7915f700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.194250 7f9d7915f700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:17.194257 7f9d7915f700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.194262 7f9d7915f700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:17.194401 7f9d7915f700 20 set_header setting 17147 cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 parent seq 0
2014-06-12 14:46:17.194452 7f9d7915f700 20 set_header setting 2206 cid meta oid 1fe728d6/pglog_3.7fb/0//-1 parent seq 0
2014-06-12 14:46:17.194462 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.194470 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:17.197498 7f9d7915f700 10 _do_op 0xda7c6c0 seq 2702 r = 0, finisher 0x1f087fc0 0
2014-06-12 14:46:17.197516 7f9d7915f700 10 _finish_op 0xda7c6c0 seq 2702 osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:17.197521 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c57a0 (0 active)
2014-06-12 14:46:17.197524 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.197535 7f9d7715b700  5 -- op tracker -- , seq: 747691, time: 2014-06-12 14:46:17.197524, event: commit_sent, op: osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.197545 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3022 (79'21,173'3022] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3020 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:17.197572 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:17.197608 7f9d7795c700  5 -- op tracker -- , seq: 747691, time: 2014-06-12 14:46:17.197602, event: sub_op_applied, op: osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.197635 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3022 (79'21,173'3022] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3021 active] sub_op_modify_applied on 0x62ea380 op osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.197651 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3022 (79'21,173'3022] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3021 active] op_applied on primary on version 173'3022
2014-06-12 14:46:17.197662 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3022 (79'21,173'3022] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3021 active] op_applied on replica on version 173'3022
2014-06-12 14:46:17.197689 7f9d7795c700  5 -- op tracker -- , seq: 747691, time: 2014-06-12 14:46:17.197684, event: done, op: osd_sub_op(client.4302.0:146054 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3022 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.204234 7f9d5b6bc700  5 -- op tracker -- , seq: 747692, time: 2014-06-12 14:46:17.203145, event: header_read, op: osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.204243 7f9d5b6bc700  5 -- op tracker -- , seq: 747692, time: 2014-06-12 14:46:17.203151, event: throttled, op: osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.204248 7f9d5b6bc700  5 -- op tracker -- , seq: 747692, time: 2014-06-12 14:46:17.204200, event: all_read, op: osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.204253 7f9d5b6bc700  5 -- op tracker -- , seq: 747692, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.204258 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:17.204263 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:17.204269 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x329eae00 prio 127 cost 516903 latency 0.001124 osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.204303 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.204314 7f9d623d5700 10 osd.2 173 dequeue_op 0x329eae00 prio 127 cost 516903 latency 0.001168 osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.7fb( v 173'3022 (79'21,173'3022] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3021 active]
2014-06-12 14:46:17.204339 7f9d623d5700  5 -- op tracker -- , seq: 747692, time: 2014-06-12 14:46:17.204335, event: reached_pg, op: osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.204350 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3022 (79'21,173'3022] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3021 active] handle_message: 0x329eae00
2014-06-12 14:46:17.204363 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3022 (79'21,173'3022] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3021 active] sub_op_modify trans ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 v 173'3023 (transaction) 174
2014-06-12 14:46:17.204380 7f9d623d5700  5 -- op tracker -- , seq: 747692, time: 2014-06-12 14:46:17.204376, event: started, op: osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.204400 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3022 (79'21,173'3022] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3021 active] append_log log((79'21,173'3022], crt=0'0) [173'3023 (173'3022) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146055 2014-06-12 08:45:19.580341]
2014-06-12 14:46:17.204426 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3023 (79'21,173'3023] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3021 active] add_log_entry 173'3023 (173'3022) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146055 2014-06-12 08:45:19.580341
2014-06-12 14:46:17.204467 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3023 (79'21,173'3023] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3021 active] append_log  adding 1 keys
2014-06-12 14:46:17.204482 7f9d623d5700 10 trim log((79'21,173'3023], crt=0'0) to 79'22
2014-06-12 14:46:17.204511 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3023, trimmed: 79'22
2014-06-12 14:46:17.204555 7f9d623d5700  5 -- op tracker -- , seq: 747692, time: 2014-06-12 14:46:17.204542, event: started, op: osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.204568 7f9d623d5700  5 queue_transactions existing osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:17.204573 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.204582 7f9d623d5700  5 queue_op 0xbdfc240 seq 2703 osr(3.7fb 0x49b42f0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:17.204589 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3023 (79'22,173'3023] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3021 active] do_sub_op osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.204604 7f9d623d5700 10 osd.2 173 dequeue_op 0x329eae00 finish
2014-06-12 14:46:17.204607 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.204610 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.204619 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c57a0 (1 active)
2014-06-12 14:46:17.204624 7f9d7d968700  5 _do_op 0xbdfc240 seq 2703 osr(3.7fb 0x49b42f0)/0x49b42f0 start
2014-06-12 14:46:17.204629 7f9d7d968700 10 _do_transaction on 0x7197970
2014-06-12 14:46:17.204634 7f9d7d968700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.204773 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.204793 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.204849 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.204872 7f9d7d968700 15 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.204880 7f9d7d968700 10 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1 r = 0
2014-06-12 14:46:17.204886 7f9d7d968700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.204903 7f9d7d968700 15 _write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 2465792~516096
2014-06-12 14:46:17.205054 7f9d7d968700 10 lookup_strip_header done  cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.205327 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:17.207267 7f9d7d968700 10 _generic_write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 2465792~516096 = 0
2014-06-12 14:46:17.207324 7f9d7d968700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.207340 7f9d7d968700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:17.207349 7f9d7d968700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.207358 7f9d7d968700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:17.207596 7f9d7d968700 20 set_header setting 17147 cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 parent seq 0
2014-06-12 14:46:17.207678 7f9d7d968700 20 set_header setting 2206 cid meta oid 1fe728d6/pglog_3.7fb/0//-1 parent seq 0
2014-06-12 14:46:17.207716 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.207728 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:17.210393 7f9d7d968700 10 _do_op 0xbdfc240 seq 2703 r = 0, finisher 0x15310620 0
2014-06-12 14:46:17.210412 7f9d7d968700 10 _finish_op 0xbdfc240 seq 2703 osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:17.210420 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c57a0 (0 active)
2014-06-12 14:46:17.210428 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.210429 7f9d7715b700  5 -- op tracker -- , seq: 747692, time: 2014-06-12 14:46:17.210420, event: commit_sent, op: osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.210438 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3023 (79'22,173'3023] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3021 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:17.210458 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:17.210483 7f9d7795c700  5 -- op tracker -- , seq: 747692, time: 2014-06-12 14:46:17.210477, event: sub_op_applied, op: osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.210491 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3023 (79'22,173'3023] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3022 active] sub_op_modify_applied on 0x7197880 op osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.210504 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3023 (79'22,173'3023] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3022 active] op_applied on primary on version 173'3023
2014-06-12 14:46:17.210510 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3023 (79'22,173'3023] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3022 active] op_applied on replica on version 173'3023
2014-06-12 14:46:17.210527 7f9d7795c700  5 -- op tracker -- , seq: 747692, time: 2014-06-12 14:46:17.210524, event: done, op: osd_sub_op(client.4302.0:146055 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3023 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.216279 7f9d5b6bc700  5 -- op tracker -- , seq: 747693, time: 2014-06-12 14:46:17.215579, event: header_read, op: osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.216298 7f9d5b6bc700  5 -- op tracker -- , seq: 747693, time: 2014-06-12 14:46:17.215580, event: throttled, op: osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.216303 7f9d5b6bc700  5 -- op tracker -- , seq: 747693, time: 2014-06-12 14:46:17.216243, event: all_read, op: osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.216308 7f9d5b6bc700  5 -- op tracker -- , seq: 747693, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.216312 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:17.216317 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:17.216323 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x329ea900 prio 127 cost 516903 latency 0.000744 osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.216349 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.216356 7f9d643d9700 10 osd.2 173 dequeue_op 0x329ea900 prio 127 cost 516903 latency 0.000776 osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.7fb( v 173'3023 (79'22,173'3023] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3022 active]
2014-06-12 14:46:17.216374 7f9d643d9700  5 -- op tracker -- , seq: 747693, time: 2014-06-12 14:46:17.216371, event: reached_pg, op: osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.216381 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3023 (79'22,173'3023] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3022 active] handle_message: 0x329ea900
2014-06-12 14:46:17.216390 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3023 (79'22,173'3023] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3022 active] sub_op_modify trans ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 v 173'3024 (transaction) 174
2014-06-12 14:46:17.216400 7f9d643d9700  5 -- op tracker -- , seq: 747693, time: 2014-06-12 14:46:17.216398, event: started, op: osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.216412 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3023 (79'22,173'3023] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3022 active] append_log log((79'22,173'3023], crt=0'0) [173'3024 (173'3023) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146056 2014-06-12 08:45:19.593219]
2014-06-12 14:46:17.216447 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3024 (79'22,173'3024] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3022 active] add_log_entry 173'3024 (173'3023) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146056 2014-06-12 08:45:19.593219
2014-06-12 14:46:17.216462 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3024 (79'22,173'3024] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3022 active] append_log  adding 1 keys
2014-06-12 14:46:17.216471 7f9d643d9700 10 trim log((79'22,173'3024], crt=0'0) to 79'23
2014-06-12 14:46:17.216491 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3024, trimmed: 79'23
2014-06-12 14:46:17.216509 7f9d643d9700  5 -- op tracker -- , seq: 747693, time: 2014-06-12 14:46:17.216506, event: started, op: osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.216525 7f9d643d9700  5 queue_transactions existing osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:17.216537 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.216538 7f9d643d9700  5 queue_op 0x5979e60 seq 2704 osr(3.7fb 0x49b42f0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:17.216547 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c57a0 (1 active)
2014-06-12 14:46:17.216553 7f9d83173700  5 _do_op 0x5979e60 seq 2704 osr(3.7fb 0x49b42f0)/0x49b42f0 start
2014-06-12 14:46:17.216543 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3024 (79'23,173'3024] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3022 active] do_sub_op osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.216556 7f9d643d9700 10 osd.2 173 dequeue_op 0x329ea900 finish
2014-06-12 14:46:17.216557 7f9d83173700 10 _do_transaction on 0x628d970
2014-06-12 14:46:17.216559 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.216561 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.216564 7f9d83173700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.216678 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.216708 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.216738 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.216836 7f9d83173700 15 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.216843 7f9d83173700 10 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1 r = 0
2014-06-12 14:46:17.216848 7f9d83173700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.216869 7f9d83173700 15 _write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 2981888~516096
2014-06-12 14:46:17.216978 7f9d83173700 10 lookup_strip_header done  cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.217118 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:17.218378 7f9d83173700 10 _generic_write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 2981888~516096 = 0
2014-06-12 14:46:17.218416 7f9d83173700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.218428 7f9d83173700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:17.218431 7f9d83173700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.218436 7f9d83173700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:17.218573 7f9d83173700 20 set_header setting 17147 cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 parent seq 0
2014-06-12 14:46:17.218621 7f9d83173700 20 set_header setting 2206 cid meta oid 1fe728d6/pglog_3.7fb/0//-1 parent seq 0
2014-06-12 14:46:17.218631 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.218636 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:17.220183 7f9d83173700 10 _do_op 0x5979e60 seq 2704 r = 0, finisher 0x6ce5200 0
2014-06-12 14:46:17.220197 7f9d83173700 10 _finish_op 0x5979e60 seq 2704 osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:17.220205 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c57a0 (0 active)
2014-06-12 14:46:17.220213 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.220216 7f9d7715b700  5 -- op tracker -- , seq: 747693, time: 2014-06-12 14:46:17.220206, event: commit_sent, op: osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.220225 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3024 (79'23,173'3024] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3022 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:17.220249 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:17.220278 7f9d7795c700  5 -- op tracker -- , seq: 747693, time: 2014-06-12 14:46:17.220275, event: sub_op_applied, op: osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.220284 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3024 (79'23,173'3024] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3023 active] sub_op_modify_applied on 0x628d880 op osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.220295 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3024 (79'23,173'3024] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3023 active] op_applied on primary on version 173'3024
2014-06-12 14:46:17.220301 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3024 (79'23,173'3024] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3023 active] op_applied on replica on version 173'3024
2014-06-12 14:46:17.220319 7f9d7795c700  5 -- op tracker -- , seq: 747693, time: 2014-06-12 14:46:17.220316, event: done, op: osd_sub_op(client.4302.0:146056 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3024 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.225954 7f9d5b6bc700  5 -- op tracker -- , seq: 747694, time: 2014-06-12 14:46:17.225177, event: header_read, op: osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.225963 7f9d5b6bc700  5 -- op tracker -- , seq: 747694, time: 2014-06-12 14:46:17.225178, event: throttled, op: osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.225972 7f9d5b6bc700  5 -- op tracker -- , seq: 747694, time: 2014-06-12 14:46:17.225923, event: all_read, op: osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.225977 7f9d5b6bc700  5 -- op tracker -- , seq: 747694, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.225981 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:17.225986 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:17.225991 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x329eb800 prio 127 cost 516903 latency 0.000814 osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.226019 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.226031 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329eb800 prio 127 cost 516903 latency 0.000853 osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.7fb( v 173'3024 (79'23,173'3024] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3023 active]
2014-06-12 14:46:17.226059 7f9d61bd4700  5 -- op tracker -- , seq: 747694, time: 2014-06-12 14:46:17.226054, event: reached_pg, op: osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.226070 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3024 (79'23,173'3024] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3023 active] handle_message: 0x329eb800
2014-06-12 14:46:17.226085 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3024 (79'23,173'3024] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3023 active] sub_op_modify trans ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 v 173'3025 (transaction) 174
2014-06-12 14:46:17.226105 7f9d61bd4700  5 -- op tracker -- , seq: 747694, time: 2014-06-12 14:46:17.226100, event: started, op: osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.226121 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3024 (79'23,173'3024] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3023 active] append_log log((79'23,173'3024], crt=0'0) [173'3025 (173'3024) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146057 2014-06-12 08:45:19.603007]
2014-06-12 14:46:17.226150 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3025 (79'23,173'3025] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3023 active] add_log_entry 173'3025 (173'3024) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146057 2014-06-12 08:45:19.603007
2014-06-12 14:46:17.226174 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3025 (79'23,173'3025] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3023 active] append_log  adding 1 keys
2014-06-12 14:46:17.226190 7f9d61bd4700 10 trim log((79'23,173'3025], crt=0'0) to 79'24
2014-06-12 14:46:17.226218 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3025, trimmed: 79'24
2014-06-12 14:46:17.226252 7f9d61bd4700  5 -- op tracker -- , seq: 747694, time: 2014-06-12 14:46:17.226246, event: started, op: osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.226265 7f9d61bd4700  5 queue_transactions existing osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:17.226269 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.226271 7f9d61bd4700  5 queue_op 0x72c5620 seq 2705 osr(3.7fb 0x49b42f0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:17.226280 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c57a0 (1 active)
2014-06-12 14:46:17.226286 7f9d7c165700  5 _do_op 0x72c5620 seq 2705 osr(3.7fb 0x49b42f0)/0x49b42f0 start
2014-06-12 14:46:17.226289 7f9d7c165700 10 _do_transaction on 0x65f9970
2014-06-12 14:46:17.226278 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3025 (79'24,173'3025] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3023 active] do_sub_op osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.226293 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329eb800 finish
2014-06-12 14:46:17.226294 7f9d7c165700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.226296 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.226299 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.226385 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.226401 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.226432 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.226522 7f9d7c165700 15 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.226529 7f9d7c165700 10 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1 r = 0
2014-06-12 14:46:17.226533 7f9d7c165700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.226546 7f9d7c165700 15 _write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 3497984~516096
2014-06-12 14:46:17.226642 7f9d7c165700 10 lookup_strip_header done  cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.226841 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:17.227991 7f9d7c165700 10 _generic_write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 3497984~516096 = 0
2014-06-12 14:46:17.228024 7f9d7c165700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.228033 7f9d7c165700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:17.228037 7f9d7c165700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.228042 7f9d7c165700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:17.228175 7f9d7c165700 20 set_header setting 17147 cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 parent seq 0
2014-06-12 14:46:17.228225 7f9d7c165700 20 set_header setting 2206 cid meta oid 1fe728d6/pglog_3.7fb/0//-1 parent seq 0
2014-06-12 14:46:17.228235 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.228241 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:17.230016 7f9d7c165700 10 _do_op 0x72c5620 seq 2705 r = 0, finisher 0x6961e60 0
2014-06-12 14:46:17.230030 7f9d7c165700 10 _finish_op 0x72c5620 seq 2705 osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:17.230037 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c57a0 (0 active)
2014-06-12 14:46:17.230049 7f9d7715b700  5 -- op tracker -- , seq: 747694, time: 2014-06-12 14:46:17.230040, event: commit_sent, op: osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.230063 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.230066 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3025 (79'24,173'3025] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3023 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:17.230090 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:17.230120 7f9d7795c700  5 -- op tracker -- , seq: 747694, time: 2014-06-12 14:46:17.230114, event: sub_op_applied, op: osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.230128 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3025 (79'24,173'3025] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active] sub_op_modify_applied on 0x65f9880 op osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.230146 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3025 (79'24,173'3025] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active] op_applied on primary on version 173'3025
2014-06-12 14:46:17.230157 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3025 (79'24,173'3025] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active] op_applied on replica on version 173'3025
2014-06-12 14:46:17.230185 7f9d7795c700  5 -- op tracker -- , seq: 747694, time: 2014-06-12 14:46:17.230180, event: done, op: osd_sub_op(client.4302.0:146057 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3025 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.233402 7f9d4829e700  5 -- op tracker -- , seq: 747695, time: 2014-06-12 14:46:17.232643, event: header_read, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.233418 7f9d4829e700  5 -- op tracker -- , seq: 747695, time: 2014-06-12 14:46:17.232645, event: throttled, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.233424 7f9d4829e700  5 -- op tracker -- , seq: 747695, time: 2014-06-12 14:46:17.233371, event: all_read, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.233429 7f9d4829e700  5 -- op tracker -- , seq: 747695, time: 0.000000, event: dispatched, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.233435 7f9d4829e700 20 osd.2 173 should_share_map client.4302 172.16.96.220:0/1243795 173
2014-06-12 14:46:17.233458 7f9d4829e700 15 osd.2 173 enqueue_op 0x598e700 prio 63 cost 335872 latency 0.000815 osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.233497 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.233509 7f9d663dd700 10 osd.2 173 dequeue_op 0x598e700 prio 63 cost 335872 latency 0.000865 osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean]
2014-06-12 14:46:17.233552 7f9d663dd700  5 -- op tracker -- , seq: 747695, time: 2014-06-12 14:46:17.233532, event: reached_pg, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.233562 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.233579 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] handle_message: 0x598e700
2014-06-12 14:46:17.233590 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] do_op osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.233621 7f9d663dd700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_'
2014-06-12 14:46:17.233783 7f9d5b6bc700  5 -- op tracker -- , seq: 747696, time: 2014-06-12 14:46:17.233484, event: header_read, op: osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.233793 7f9d5b6bc700  5 -- op tracker -- , seq: 747696, time: 2014-06-12 14:46:17.233485, event: throttled, op: osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.233808 7f9d5b6bc700  5 -- op tracker -- , seq: 747696, time: 2014-06-12 14:46:17.233757, event: all_read, op: osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.233818 7f9d5b6bc700  5 -- op tracker -- , seq: 747696, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.233821 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:17.233825 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:17.233829 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x329eaa00 prio 127 cost 181031 latency 0.000344 osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.233846 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:17.233851 7f9d673df700 10 osd.2 173 dequeue_op 0x329eaa00 prio 127 cost 181031 latency 0.000367 osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.7fb( v 173'3025 (79'24,173'3025] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active]
2014-06-12 14:46:17.233865 7f9d673df700  5 -- op tracker -- , seq: 747696, time: 2014-06-12 14:46:17.233862, event: reached_pg, op: osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.233875 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3025 (79'24,173'3025] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active] handle_message: 0x329eaa00
2014-06-12 14:46:17.233883 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3025 (79'24,173'3025] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active] sub_op_modify trans ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 v 173'3026 (transaction) 174
2014-06-12 14:46:17.233894 7f9d673df700  5 -- op tracker -- , seq: 747696, time: 2014-06-12 14:46:17.233891, event: started, op: osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.233901 7f9d663dd700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_' = 0
2014-06-12 14:46:17.233905 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3025 (79'24,173'3025] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active] append_log log((79'24,173'3025], crt=0'0) [173'3026 (173'3025) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146058 2014-06-12 08:45:19.612777]
2014-06-12 14:46:17.233919 7f9d663dd700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset'
2014-06-12 14:46:17.233923 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3026 (79'24,173'3026] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active] add_log_entry 173'3026 (173'3025) modify   ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 by client.4302.0:146058 2014-06-12 08:45:19.612777
2014-06-12 14:46:17.233937 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3026 (79'24,173'3026] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active] append_log  adding 1 keys
2014-06-12 14:46:17.233948 7f9d673df700 10 trim log((79'24,173'3026], crt=0'0) to 79'25
2014-06-12 14:46:17.233966 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3026, trimmed: 79'25
2014-06-12 14:46:17.234004 7f9d673df700  5 -- op tracker -- , seq: 747696, time: 2014-06-12 14:46:17.234000, event: started, op: osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.234019 7f9d673df700  5 queue_transactions existing osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:17.234022 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.234031 7f9d673df700  5 queue_op 0x15e60c60 seq 2706 osr(3.7fb 0x49b42f0) 182552 bytes   (queue has 1 ops and 182552 bytes)
2014-06-12 14:46:17.234039 7f9d663dd700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset' = 0
2014-06-12 14:46:17.234039 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c57a0 (1 active)
2014-06-12 14:46:17.234045 7f9d82972700  5 _do_op 0x15e60c60 seq 2706 osr(3.7fb 0x49b42f0)/0x49b42f0 start
2014-06-12 14:46:17.234036 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3026 (79'25,173'3026] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active] do_sub_op osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.234049 7f9d82972700 10 _do_transaction on 0x1f287b30
2014-06-12 14:46:17.234049 7f9d673df700 10 osd.2 173 dequeue_op 0x329eaa00 finish
2014-06-12 14:46:17.234052 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:17.234055 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.234054 7f9d82972700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.234049 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] populate_obc_watchers 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.234061 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.234071 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] get_object_context: creating obc from disk: 0x10475600
2014-06-12 14:46:17.234080 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] get_object_context: 0x10475600 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 rwstate(none n=0 w=0) oi: 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'679 client.4302.0:38688 wrlock_by=unknown.0.0:0 dirty s 4194304 uv679) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:46:17.234095 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] find_object_context 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 @head oi=25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'679 client.4302.0:38688 wrlock_by=unknown.0.0:0 dirty s 4194304 uv679)
2014-06-12 14:46:17.234122 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.234119 7f9d663dd700  5 -- op tracker -- , seq: 747695, time: 2014-06-12 14:46:17.234114, event: started, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.234128 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] execute_ctx 0x60aa600
2014-06-12 14:46:17.234137 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.234145 7f9d663dd700  5 -- op tracker -- , seq: 747695, time: 2014-06-12 14:46:17.234140, event: started, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.234152 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] do_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] ov 173'679 av 173'2790 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.234165 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.234166 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] do_osd_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872]
2014-06-12 14:46:17.234177 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.234189 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] do_osd_op  write 0~335872
2014-06-12 14:46:17.234206 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] do_osd_op_effects on session 0x5c8d200
2014-06-12 14:46:17.234219 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:46:17.234234 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.234247 7f9d82972700 15 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.234253 7f9d82972700 10 _omap_rmkeys meta/1fe728d6/pglog_3.7fb/0//-1 r = 0
2014-06-12 14:46:17.234246 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] finish_ctx 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 0x60aa600 op modify  
2014-06-12 14:46:17.234257 7f9d82972700 15 _omap_setkeys meta/1fe728d6/pglog_3.7fb/0//-1
2014-06-12 14:46:17.234269 7f9d82972700 15 _write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 4014080~180224
2014-06-12 14:46:17.234262 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean]  set mtime to 2014-06-12 08:45:19.613007
2014-06-12 14:46:17.234335 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean]  final snapset 0=[]:[]+head in 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.234362 7f9d82972700 10 lookup_strip_header done  cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.234355 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.234374 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] new_repop rep_tid 236233 on osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.234398 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] issue_repop rep_tid 236233 o 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.234422 7f9d663dd700  5 -- op tracker -- , seq: 747695, time: 2014-06-12 14:46:17.234418, event: waiting for subops from 29, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.234440 7f9d663dd700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:17.234460 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2789 (0'0,173'2789] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] append_log log((0'0,173'2789], crt=0'0) [173'2790 (173'679) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146059 2014-06-12 08:45:19.613007]
2014-06-12 14:46:17.234488 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:17.234481 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] add_log_entry 173'2790 (173'679) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146059 2014-06-12 08:45:19.613007
2014-06-12 14:46:17.234500 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.234527 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2790, trimmed: 
2014-06-12 14:46:17.234548 7f9d663dd700  5 queue_transactions existing osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.234552 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.234554 7f9d663dd700  5 queue_op 0x7749860 seq 2605 osr(3.330 0x276d4f0) 338165 bytes   (queue has 2 ops and 520717 bytes)
2014-06-12 14:46:17.234559 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] eval_repop repgather(0x5146540 173'2790 rep_tid=236233 committed?=0 applied?=0 lock=0 op=osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.234578 7f9d663dd700 10 osd.2 173 dequeue_op 0x598e700 finish
2014-06-12 14:46:17.234582 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.234585 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.234599 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541ca20 (2 active)
2014-06-12 14:46:17.234603 7f9d7c966700  5 _do_op 0x7749860 seq 2605 osr(3.330 0x276d4f0)/0x276d4f0 start
2014-06-12 14:46:17.234606 7f9d7c966700 10 _do_transaction on 0x5323740
2014-06-12 14:46:17.234610 7f9d7c966700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.234688 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.234708 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.235018 7f9d82972700 10 _generic_write 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 4014080~180224 = 0
2014-06-12 14:46:17.235043 7f9d82972700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.235052 7f9d82972700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:17.235059 7f9d82972700 15 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3
2014-06-12 14:46:17.235066 7f9d82972700 10 _setattrs 3.7fb_head/ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 = 0
2014-06-12 14:46:17.235163 7f9d82972700 20 set_header setting 17147 cid 3.7fb_head oid ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 parent seq 0
2014-06-12 14:46:17.235191 7f9d82972700 20 set_header setting 2206 cid meta oid 1fe728d6/pglog_3.7fb/0//-1 parent seq 0
2014-06-12 14:46:17.235202 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.235208 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:17.235875 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:17.235922 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:17.236231 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.236237 7f9d82972700 10 _do_op 0x15e60c60 seq 2706 r = 0, finisher 0x29fc04a0 0
2014-06-12 14:46:17.236244 7f9d82972700 10 _finish_op 0x15e60c60 seq 2706 osr(3.7fb 0x49b42f0)/0x49b42f0
2014-06-12 14:46:17.236251 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c57a0 (1 active)
2014-06-12 14:46:17.236254 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.236257 7f9d7c966700 15 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.236263 7f9d7c966700 10 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1 r = 0
2014-06-12 14:46:17.236268 7f9d7c966700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.236267 7f9d7795c700  5 -- op tracker -- , seq: 747696, time: 2014-06-12 14:46:17.236258, event: sub_op_applied, op: osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.236278 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3026 (79'25,173'3026] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active] sub_op_modify_applied on 0x1f287a40 op osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.236301 7f9d7795c700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:17.236318 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3026 (79'25,173'3026] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active] op_applied on primary on version 173'3026
2014-06-12 14:46:17.236330 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3026 (79'25,173'3026] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active] op_applied on replica on version 173'3026
2014-06-12 14:46:17.236348 7f9d7c966700 15 _write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 0~335872
2014-06-12 14:46:17.236358 7f9d7715b700  5 -- op tracker -- , seq: 747696, time: 2014-06-12 14:46:17.236353, event: commit_sent, op: osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.236366 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7fb( v 173'3026 (79'25,173'3026] local-les=157 n=308 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'3024 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:17.236387 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:17.236420 7f9d7715b700  5 -- op tracker -- , seq: 747696, time: 2014-06-12 14:46:17.236416, event: done, op: osd_sub_op(client.4302.0:146058 3.7fb ba2b1ffb/rbd_data.10a56b8b4567.0000000000001031/head//3 [] v 173'3026 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.236437 7f9d7c966700 10 lookup_strip_header done  cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.236542 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:17.237251 7f9d7c966700 10 _generic_write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 0~335872 = 0
2014-06-12 14:46:17.237272 7f9d7c966700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.237280 7f9d7c966700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.237285 7f9d7c966700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.237289 7f9d7c966700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.237397 7f9d7c966700 20 set_header setting 17148 cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:17.237426 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.237435 7f9d7c966700 20 set_header setting 2048 cid meta oid 1fe2257a/pglog_3.330/0//-1 parent seq 0
2014-06-12 14:46:17.237441 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:17.238478 7f9d7c966700 10 _do_op 0x7749860 seq 2605 r = 0, finisher 0x73f9740 0x323f400
2014-06-12 14:46:17.238487 7f9d7c966700 10 _finish_op 0x7749860 seq 2605 osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.238492 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541ca20 (0 active)
2014-06-12 14:46:17.238497 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.238491 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] op_commit: 236233
2014-06-12 14:46:17.238519 7f9d7715b700  5 -- op tracker -- , seq: 747695, time: 2014-06-12 14:46:17.238510, event: op_commit, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.238562 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] op_applied: 236233
2014-06-12 14:46:17.238581 7f9d7795c700  5 -- op tracker -- , seq: 747695, time: 2014-06-12 14:46:17.238576, event: op_applied, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.238589 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] op_applied on primary on version 173'2790
2014-06-12 14:46:17.239253 7f9d5501e700  5 -- op tracker -- , seq: 747697, time: 2014-06-12 14:46:17.239223, event: header_read, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.239262 7f9d5501e700  5 -- op tracker -- , seq: 747697, time: 2014-06-12 14:46:17.239224, event: throttled, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.239279 7f9d5501e700  5 -- op tracker -- , seq: 747697, time: 2014-06-12 14:46:17.239233, event: all_read, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.239286 7f9d5501e700  5 -- op tracker -- , seq: 747697, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.239289 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:17.239293 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.239298 7f9d5501e700 15 osd.2 173 enqueue_op 0x8dbcf00 prio 196 cost 0 latency 0.000074 osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2
2014-06-12 14:46:17.239322 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.239337 7f9d5501e700  5 -- op tracker -- , seq: 747698, time: 2014-06-12 14:46:17.239315, event: header_read, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.239332 7f9d67be0700 10 osd.2 173 dequeue_op 0x8dbcf00 prio 196 cost 0 latency 0.000108 osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2 pg pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean]
2014-06-12 14:46:17.239348 7f9d5501e700  5 -- op tracker -- , seq: 747698, time: 2014-06-12 14:46:17.239315, event: throttled, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.239359 7f9d5501e700  5 -- op tracker -- , seq: 747698, time: 2014-06-12 14:46:17.239321, event: all_read, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.239367 7f9d5501e700  5 -- op tracker -- , seq: 747698, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.239374 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.239379 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.239377 7f9d67be0700  5 -- op tracker -- , seq: 747697, time: 2014-06-12 14:46:17.239346, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.239384 7f9d5501e700 15 osd.2 173 enqueue_op 0x28fd8f00 prio 196 cost 0 latency 0.000069 osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.239389 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] handle_message: 0x8dbcf00
2014-06-12 14:46:17.239408 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:17.239404 7f9d67be0700  5 -- op tracker -- , seq: 747697, time: 2014-06-12 14:46:17.239401, event: started, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.239411 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] sub_op_modify_reply: tid 236233 op  ack_type 1 from 29
2014-06-12 14:46:17.239428 7f9d67be0700  5 -- op tracker -- , seq: 747695, time: 2014-06-12 14:46:17.239423, event: sub_op_applied_rec, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.239434 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] repop_all_applied: repop tid 236233 all applied 
2014-06-12 14:46:17.239443 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] eval_repop repgather(0x5146540 173'2790 rep_tid=236233 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.239453 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean]  sending ack on repgather(0x5146540 173'2790 rep_tid=236233 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0xc3fec80
2014-06-12 14:46:17.239475 7f9d67be0700 10 osd.2 173 dequeue_op 0x8dbcf00 finish
2014-06-12 14:46:17.239486 7f9d67be0700  5 -- op tracker -- , seq: 747697, time: 2014-06-12 14:46:17.239482, event: done, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.239500 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:17.239504 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.239500 7f9d693e3700 10 osd.2 173 dequeue_op 0x28fd8f00 prio 196 cost 0 latency 0.000185 osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 pg pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean]
2014-06-12 14:46:17.239518 7f9d693e3700  5 -- op tracker -- , seq: 747698, time: 2014-06-12 14:46:17.239515, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.239525 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] handle_message: 0x28fd8f00
2014-06-12 14:46:17.239536 7f9d693e3700  5 -- op tracker -- , seq: 747698, time: 2014-06-12 14:46:17.239534, event: started, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.239539 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] sub_op_modify_reply: tid 236233 op  ack_type 4 from 29
2014-06-12 14:46:17.239550 7f9d693e3700  5 -- op tracker -- , seq: 747695, time: 2014-06-12 14:46:17.239546, event: sub_op_commit_rec, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.239553 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2789 crt=0'0 lcod 173'2788 mlcod 173'2788 active+clean] repop_all_committed: repop tid 236233 all committed 
2014-06-12 14:46:17.239559 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2788 active+clean] eval_repop repgather(0x5146540 173'2790 rep_tid=236233 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.239570 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2788 active+clean] log_op_stats osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 inb 336151 outb 0 rlat 0.006831 lat 0.006926
2014-06-12 14:46:17.239582 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2788 active+clean] publish_stats_to_osd 173:3595
2014-06-12 14:46:17.239592 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2788 active+clean]  sending commit on repgather(0x5146540 173'2790 rep_tid=236233 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x1f889400
2014-06-12 14:46:17.239616 7f9d693e3700  5 -- op tracker -- , seq: 747695, time: 2014-06-12 14:46:17.239611, event: commit_sent, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.239626 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2788 active+clean]  requeue_ops 
2014-06-12 14:46:17.239639 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean]  removing repgather(0x5146540 173'2790 rep_tid=236233 committed?=1 applied?=1 op=osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.239650 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean]    q front is repgather(0x5146540 173'2790 rep_tid=236233 committed?=1 applied?=1 op=osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.239659 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] remove_repop repgather(0x5146540 173'2790 rep_tid=236233 committed?=1 applied?=1 op=osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.239668 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean]  requeue_ops 
2014-06-12 14:46:17.239684 7f9d693e3700  5 -- op tracker -- , seq: 747695, time: 2014-06-12 14:46:17.239681, event: done, op: osd_op(client.4302.0:146059 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~335872] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.239696 7f9d693e3700 10 osd.2 173 dequeue_op 0x28fd8f00 finish
2014-06-12 14:46:17.239702 7f9d693e3700  5 -- op tracker -- , seq: 747698, time: 2014-06-12 14:46:17.239699, event: done, op: osd_sub_op_reply(client.4302.0:146059 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.239708 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.239709 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.242855 7f9d4829e700  5 -- op tracker -- , seq: 747699, time: 2014-06-12 14:46:17.241797, event: header_read, op: osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.242864 7f9d4829e700  5 -- op tracker -- , seq: 747699, time: 2014-06-12 14:46:17.241798, event: throttled, op: osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.242870 7f9d4829e700  5 -- op tracker -- , seq: 747699, time: 2014-06-12 14:46:17.242830, event: all_read, op: osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.242875 7f9d4829e700  5 -- op tracker -- , seq: 747699, time: 0.000000, event: dispatched, op: osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.242880 7f9d4829e700 20 osd.2 173 should_share_map client.4302 172.16.96.220:0/1243795 173
2014-06-12 14:46:17.242901 7f9d4829e700 15 osd.2 173 enqueue_op 0x598fe00 prio 63 cost 516096 latency 0.001104 osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.242924 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.242936 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x598fe00 prio 63 cost 516096 latency 0.001139 osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean]
2014-06-12 14:46:17.242980 7f9d6b3e7700  5 -- op tracker -- , seq: 747699, time: 2014-06-12 14:46:17.242967, event: reached_pg, op: osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.242994 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.243005 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] handle_message: 0x598fe00
2014-06-12 14:46:17.243011 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] do_op osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.243032 7f9d6b3e7700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_'
2014-06-12 14:46:17.243192 7f9d6b3e7700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_' = 0
2014-06-12 14:46:17.243206 7f9d6b3e7700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset'
2014-06-12 14:46:17.243279 7f9d6b3e7700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset' = 0
2014-06-12 14:46:17.243287 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] populate_obc_watchers 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.243295 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.243309 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] get_object_context: creating obc from disk: 0x10093b80
2014-06-12 14:46:17.243314 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] get_object_context: 0x10093b80 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 rwstate(none n=0 w=0) oi: 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2790 client.4302.0:146059 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2790) ssc: 0x24d4a380 snapset: 0=[]:[]+head
2014-06-12 14:46:17.243324 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] find_object_context 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 @head oi=25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2790 client.4302.0:146059 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2790)
2014-06-12 14:46:17.243342 7f9d6b3e7700  5 -- op tracker -- , seq: 747699, time: 2014-06-12 14:46:17.243339, event: started, op: osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.243348 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] execute_ctx 0x17cfbe00
2014-06-12 14:46:17.243359 7f9d6b3e7700  5 -- op tracker -- , seq: 747699, time: 2014-06-12 14:46:17.243356, event: started, op: osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.243363 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] do_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] ov 173'2790 av 173'2791 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.243371 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] do_osd_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096]
2014-06-12 14:46:17.243378 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.243387 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] do_osd_op  write 335872~516096
2014-06-12 14:46:17.243399 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] do_osd_op_effects on session 0x5c8d200
2014-06-12 14:46:17.243406 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 snapset=0x24d4a3b8  snapc=0=[]
2014-06-12 14:46:17.243413 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.243419 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] finish_ctx 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 0x17cfbe00 op modify  
2014-06-12 14:46:17.243427 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean]  set mtime to 2014-06-12 08:45:19.621788
2014-06-12 14:46:17.243441 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean]  final snapset 0=[]:[]+head in 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.243454 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.243461 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] new_repop rep_tid 236234 on osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.243472 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] issue_repop rep_tid 236234 o 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.243488 7f9d6b3e7700  5 -- op tracker -- , seq: 747699, time: 2014-06-12 14:46:17.243485, event: waiting for subops from 29, op: osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.243504 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:17.243520 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2790 (0'0,173'2790] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] append_log log((0'0,173'2790], crt=0'0) [173'2791 (173'2790) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146060 2014-06-12 08:45:19.621788]
2014-06-12 14:46:17.243537 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2790 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] add_log_entry 173'2791 (173'2790) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146060 2014-06-12 08:45:19.621788
2014-06-12 14:46:17.243550 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2790 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.243574 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2791, trimmed: 
2014-06-12 14:46:17.243592 7f9d6b3e7700  5 queue_transactions existing osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.243597 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.243598 7f9d6b3e7700  5 queue_op 0x51cd740 seq 2606 osr(3.330 0x276d4f0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:17.243610 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541ca20 (1 active)
2014-06-12 14:46:17.243607 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2790 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] eval_repop repgather(0x2237c80 173'2791 rep_tid=236234 committed?=0 applied?=0 lock=0 op=osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.243617 7f9d81970700  5 _do_op 0x51cd740 seq 2606 osr(3.330 0x276d4f0)/0x276d4f0 start
2014-06-12 14:46:17.243619 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x598fe00 finish
2014-06-12 14:46:17.243621 7f9d81970700 10 _do_transaction on 0x2236e40
2014-06-12 14:46:17.243622 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.243624 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.243626 7f9d81970700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.243707 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.243724 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.243758 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.243846 7f9d81970700 15 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.243853 7f9d81970700 10 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1 r = 0
2014-06-12 14:46:17.243857 7f9d81970700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.243870 7f9d81970700 15 _write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 335872~516096
2014-06-12 14:46:17.243956 7f9d81970700 10 lookup_strip_header done  cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.244212 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:17.245415 7f9d81970700 10 _generic_write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 335872~516096 = 0
2014-06-12 14:46:17.245476 7f9d81970700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.245494 7f9d81970700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.245498 7f9d81970700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.245511 7f9d81970700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.245677 7f9d81970700 20 set_header setting 17148 cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:17.245744 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.245755 7f9d81970700 20 set_header setting 2048 cid meta oid 1fe2257a/pglog_3.330/0//-1 parent seq 0
2014-06-12 14:46:17.245772 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:17.247697 7f9d81970700 10 _do_op 0x51cd740 seq 2606 r = 0, finisher 0x7f43980 0x33fd4400
2014-06-12 14:46:17.247709 7f9d81970700 10 _finish_op 0x51cd740 seq 2606 osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.247715 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541ca20 (0 active)
2014-06-12 14:46:17.247719 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.247714 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2790 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] op_commit: 236234
2014-06-12 14:46:17.247757 7f9d7715b700  5 -- op tracker -- , seq: 747699, time: 2014-06-12 14:46:17.247737, event: op_commit, op: osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.247838 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2790 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] op_applied: 236234
2014-06-12 14:46:17.247877 7f9d7795c700  5 -- op tracker -- , seq: 747699, time: 2014-06-12 14:46:17.247867, event: op_applied, op: osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.247896 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2790 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] op_applied on primary on version 173'2791
2014-06-12 14:46:17.248896 7f9d5501e700  5 -- op tracker -- , seq: 747700, time: 2014-06-12 14:46:17.248840, event: header_read, op: osd_sub_op_reply(client.4302.0:146060 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.248911 7f9d5501e700  5 -- op tracker -- , seq: 747700, time: 2014-06-12 14:46:17.248842, event: throttled, op: osd_sub_op_reply(client.4302.0:146060 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.248938 7f9d5501e700  5 -- op tracker -- , seq: 747700, time: 2014-06-12 14:46:17.248859, event: all_read, op: osd_sub_op_reply(client.4302.0:146060 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.248953 7f9d5501e700  5 -- op tracker -- , seq: 747700, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146060 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.248959 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146060 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.248966 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.248976 7f9d5501e700 15 osd.2 173 enqueue_op 0x20e88200 prio 196 cost 0 latency 0.000136 osd_sub_op_reply(client.4302.0:146060 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.249046 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.249068 7f9d62bd6700 10 osd.2 173 dequeue_op 0x20e88200 prio 196 cost 0 latency 0.000228 osd_sub_op_reply(client.4302.0:146060 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 pg pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2790 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean]
2014-06-12 14:46:17.249110 7f9d62bd6700  5 -- op tracker -- , seq: 747700, time: 2014-06-12 14:46:17.249097, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146060 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.249121 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2790 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] handle_message: 0x20e88200
2014-06-12 14:46:17.249131 7f9d62bd6700  5 -- op tracker -- , seq: 747700, time: 2014-06-12 14:46:17.249129, event: started, op: osd_sub_op_reply(client.4302.0:146060 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.249135 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2790 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] sub_op_modify_reply: tid 236234 op  ack_type 4 from 29
2014-06-12 14:46:17.249146 7f9d62bd6700  5 -- op tracker -- , seq: 747699, time: 2014-06-12 14:46:17.249142, event: sub_op_commit_rec, op: osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.249153 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2790 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] repop_all_applied: repop tid 236234 all applied 
2014-06-12 14:46:17.249163 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2790 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] eval_repop repgather(0x2237c80 173'2791 rep_tid=236234 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.249173 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2790 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean]  sending ack on repgather(0x2237c80 173'2791 rep_tid=236234 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x1710af00
2014-06-12 14:46:17.249196 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2790 crt=0'0 lcod 173'2789 mlcod 173'2789 active+clean] repop_all_committed: repop tid 236234 all committed 
2014-06-12 14:46:17.249203 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2789 active+clean] eval_repop repgather(0x2237c80 173'2791 rep_tid=236234 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.249213 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2789 active+clean] log_op_stats osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.007398 lat 0.007416
2014-06-12 14:46:17.249224 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2789 active+clean] publish_stats_to_osd 173:3596
2014-06-12 14:46:17.249232 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2789 active+clean]  sending commit on repgather(0x2237c80 173'2791 rep_tid=236234 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x32df1180
2014-06-12 14:46:17.249255 7f9d62bd6700  5 -- op tracker -- , seq: 747699, time: 2014-06-12 14:46:17.249252, event: commit_sent, op: osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.249261 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2789 active+clean]  requeue_ops 
2014-06-12 14:46:17.249268 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean]  removing repgather(0x2237c80 173'2791 rep_tid=236234 committed?=1 applied?=1 op=osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.249277 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean]    q front is repgather(0x2237c80 173'2791 rep_tid=236234 committed?=1 applied?=1 op=osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.249285 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] remove_repop repgather(0x2237c80 173'2791 rep_tid=236234 committed?=1 applied?=1 op=osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.249294 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean]  requeue_ops 
2014-06-12 14:46:17.249310 7f9d62bd6700  5 -- op tracker -- , seq: 747699, time: 2014-06-12 14:46:17.249307, event: done, op: osd_op(client.4302.0:146060 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 335872~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.249324 7f9d62bd6700 10 osd.2 173 dequeue_op 0x20e88200 finish
2014-06-12 14:46:17.249330 7f9d62bd6700  5 -- op tracker -- , seq: 747700, time: 2014-06-12 14:46:17.249328, event: done, op: osd_sub_op_reply(client.4302.0:146060 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.249336 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.249338 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.252741 7f9d4829e700  5 -- op tracker -- , seq: 747701, time: 2014-06-12 14:46:17.251722, event: header_read, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.252758 7f9d4829e700  5 -- op tracker -- , seq: 747701, time: 2014-06-12 14:46:17.251726, event: throttled, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.252773 7f9d4829e700  5 -- op tracker -- , seq: 747701, time: 2014-06-12 14:46:17.252718, event: all_read, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.252787 7f9d4829e700  5 -- op tracker -- , seq: 747701, time: 0.000000, event: dispatched, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.252792 7f9d4829e700 20 osd.2 173 should_share_map client.4302 172.16.96.220:0/1243795 173
2014-06-12 14:46:17.252811 7f9d4829e700 15 osd.2 173 enqueue_op 0x598f300 prio 63 cost 516096 latency 0.001089 osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.252884 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.252906 7f9d683e1700 10 osd.2 173 dequeue_op 0x598f300 prio 63 cost 516096 latency 0.001183 osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean]
2014-06-12 14:46:17.252950 7f9d683e1700  5 -- op tracker -- , seq: 747701, time: 2014-06-12 14:46:17.252942, event: reached_pg, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.252969 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.252991 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] handle_message: 0x598f300
2014-06-12 14:46:17.253005 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] do_op osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.253038 7f9d683e1700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_'
2014-06-12 14:46:17.253256 7f9d683e1700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_' = 0
2014-06-12 14:46:17.253281 7f9d683e1700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset'
2014-06-12 14:46:17.253409 7f9d683e1700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset' = 0
2014-06-12 14:46:17.253425 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] populate_obc_watchers 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.253444 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.253457 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] get_object_context: creating obc from disk: 0xa45d600
2014-06-12 14:46:17.253469 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] get_object_context: 0xa45d600 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 rwstate(none n=0 w=0) oi: 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2791 client.4302.0:146060 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2791) ssc: 0x16ee50a0 snapset: 0=[]:[]+head
2014-06-12 14:46:17.253496 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] find_object_context 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 @head oi=25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2791 client.4302.0:146060 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2791)
2014-06-12 14:46:17.253533 7f9d683e1700  5 -- op tracker -- , seq: 747701, time: 2014-06-12 14:46:17.253525, event: started, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.253547 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] execute_ctx 0x1b262000
2014-06-12 14:46:17.253570 7f9d683e1700  5 -- op tracker -- , seq: 747701, time: 2014-06-12 14:46:17.253564, event: started, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.253581 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] do_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] ov 173'2791 av 173'2792 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.253603 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] do_osd_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096]
2014-06-12 14:46:17.253661 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.253681 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] do_osd_op  write 851968~516096
2014-06-12 14:46:17.253703 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] do_osd_op_effects on session 0x5c8d200
2014-06-12 14:46:17.253718 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 snapset=0x16ee50d8  snapc=0=[]
2014-06-12 14:46:17.253733 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.253748 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] finish_ctx 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 0x1b262000 op modify  
2014-06-12 14:46:17.253777 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean]  set mtime to 2014-06-12 08:45:19.631637
2014-06-12 14:46:17.253809 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean]  final snapset 0=[]:[]+head in 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.253835 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.253849 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] new_repop rep_tid 236235 on osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.253872 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] issue_repop rep_tid 236235 o 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.253903 7f9d683e1700  5 -- op tracker -- , seq: 747701, time: 2014-06-12 14:46:17.253897, event: waiting for subops from 29, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.253933 7f9d683e1700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:17.253964 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2791 (0'0,173'2791] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] append_log log((0'0,173'2791], crt=0'0) [173'2792 (173'2791) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146061 2014-06-12 08:45:19.631637]
2014-06-12 14:46:17.253998 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] add_log_entry 173'2792 (173'2791) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146061 2014-06-12 08:45:19.631637
2014-06-12 14:46:17.254028 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.254071 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2792, trimmed: 
2014-06-12 14:46:17.254101 7f9d683e1700  5 queue_transactions existing osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.254108 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.254110 7f9d683e1700  5 queue_op 0x7c99f80 seq 2607 osr(3.330 0x276d4f0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:17.254119 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] eval_repop repgather(0x2a12e9c0 173'2792 rep_tid=236235 committed?=0 applied?=0 lock=0 op=osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.254130 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541ca20 (1 active)
2014-06-12 14:46:17.254145 7f9d683e1700 10 osd.2 173 dequeue_op 0x598f300 finish
2014-06-12 14:46:17.254150 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.254150 7f9d7a161700  5 _do_op 0x7c99f80 seq 2607 osr(3.330 0x276d4f0)/0x276d4f0 start
2014-06-12 14:46:17.254162 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.254164 7f9d7a161700 10 _do_transaction on 0x2a12fe00
2014-06-12 14:46:17.254171 7f9d7a161700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.254266 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.254288 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.254337 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.254363 7f9d7a161700 15 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.254370 7f9d7a161700 10 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1 r = 0
2014-06-12 14:46:17.254376 7f9d7a161700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.254482 7f9d7a161700 15 _write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 851968~516096
2014-06-12 14:46:17.254567 7f9d7a161700 10 lookup_strip_header done  cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.254662 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:17.255725 7f9d7a161700 10 _generic_write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 851968~516096 = 0
2014-06-12 14:46:17.255749 7f9d7a161700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.255757 7f9d7a161700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.255761 7f9d7a161700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.255766 7f9d7a161700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.255885 7f9d7a161700 20 set_header setting 17148 cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:17.255925 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.255933 7f9d7a161700 20 set_header setting 2048 cid meta oid 1fe2257a/pglog_3.330/0//-1 parent seq 0
2014-06-12 14:46:17.255940 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:17.257483 7f9d7a161700 10 _do_op 0x7c99f80 seq 2607 r = 0, finisher 0x8941a60 0x25c95bc0
2014-06-12 14:46:17.257491 7f9d7a161700 10 _finish_op 0x7c99f80 seq 2607 osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.257496 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541ca20 (0 active)
2014-06-12 14:46:17.257500 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.257495 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] op_commit: 236235
2014-06-12 14:46:17.257522 7f9d7715b700  5 -- op tracker -- , seq: 747701, time: 2014-06-12 14:46:17.257514, event: op_commit, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.257556 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] op_applied: 236235
2014-06-12 14:46:17.257593 7f9d7795c700  5 -- op tracker -- , seq: 747701, time: 2014-06-12 14:46:17.257584, event: op_applied, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.257609 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] op_applied on primary on version 173'2792
2014-06-12 14:46:17.259984 7f9d5501e700  5 -- op tracker -- , seq: 747702, time: 2014-06-12 14:46:17.259923, event: header_read, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.260001 7f9d5501e700  5 -- op tracker -- , seq: 747702, time: 2014-06-12 14:46:17.259924, event: throttled, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.260011 7f9d5501e700  5 -- op tracker -- , seq: 747702, time: 2014-06-12 14:46:17.259941, event: all_read, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.260021 7f9d5501e700  5 -- op tracker -- , seq: 747702, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.260027 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:17.260034 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.260042 7f9d5501e700 15 osd.2 173 enqueue_op 0x20e88c00 prio 196 cost 0 latency 0.000119 osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2
2014-06-12 14:46:17.260081 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.260096 7f9d65bdc700 10 osd.2 173 dequeue_op 0x20e88c00 prio 196 cost 0 latency 0.000172 osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2 pg pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean]
2014-06-12 14:46:17.260120 7f9d5501e700  5 -- op tracker -- , seq: 747703, time: 2014-06-12 14:46:17.260070, event: header_read, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.260136 7f9d5501e700  5 -- op tracker -- , seq: 747703, time: 2014-06-12 14:46:17.260071, event: throttled, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.260163 7f9d5501e700  5 -- op tracker -- , seq: 747703, time: 2014-06-12 14:46:17.260094, event: all_read, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.260203 7f9d5501e700  5 -- op tracker -- , seq: 747703, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.260218 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.260224 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.260231 7f9d5501e700 15 osd.2 173 enqueue_op 0x20e89400 prio 196 cost 0 latency 0.000161 osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.260247 7f9d65bdc700  5 -- op tracker -- , seq: 747702, time: 2014-06-12 14:46:17.260122, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.260276 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] handle_message: 0x20e88c00
2014-06-12 14:46:17.260289 7f9d65bdc700  5 -- op tracker -- , seq: 747702, time: 2014-06-12 14:46:17.260287, event: started, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.260293 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:17.260294 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] sub_op_modify_reply: tid 236235 op  ack_type 1 from 29
2014-06-12 14:46:17.260306 7f9d65bdc700  5 -- op tracker -- , seq: 747701, time: 2014-06-12 14:46:17.260302, event: sub_op_applied_rec, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.260328 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] repop_all_applied: repop tid 236235 all applied 
2014-06-12 14:46:17.260335 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] eval_repop repgather(0x2a12e9c0 173'2792 rep_tid=236235 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.260345 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean]  sending ack on repgather(0x2a12e9c0 173'2792 rep_tid=236235 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x1f888500
2014-06-12 14:46:17.260370 7f9d65bdc700 10 osd.2 173 dequeue_op 0x20e88c00 finish
2014-06-12 14:46:17.260376 7f9d65bdc700  5 -- op tracker -- , seq: 747702, time: 2014-06-12 14:46:17.260374, event: done, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.260401 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:17.260404 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.260398 7f9d64bda700 10 osd.2 173 dequeue_op 0x20e89400 prio 196 cost 0 latency 0.000328 osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 pg pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean]
2014-06-12 14:46:17.260415 7f9d64bda700  5 -- op tracker -- , seq: 747703, time: 2014-06-12 14:46:17.260412, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.260430 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] handle_message: 0x20e89400
2014-06-12 14:46:17.260440 7f9d64bda700  5 -- op tracker -- , seq: 747703, time: 2014-06-12 14:46:17.260438, event: started, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.260450 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] sub_op_modify_reply: tid 236235 op  ack_type 4 from 29
2014-06-12 14:46:17.260462 7f9d64bda700  5 -- op tracker -- , seq: 747701, time: 2014-06-12 14:46:17.260458, event: sub_op_commit_rec, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.260473 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2791 crt=0'0 lcod 173'2790 mlcod 173'2790 active+clean] repop_all_committed: repop tid 236235 all committed 
2014-06-12 14:46:17.260480 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2790 active+clean] eval_repop repgather(0x2a12e9c0 173'2792 rep_tid=236235 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.260492 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2790 active+clean] log_op_stats osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008647 lat 0.008769
2014-06-12 14:46:17.260503 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2790 active+clean] publish_stats_to_osd 173:3597
2014-06-12 14:46:17.260513 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2790 active+clean]  sending commit on repgather(0x2a12e9c0 173'2792 rep_tid=236235 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x10149900
2014-06-12 14:46:17.260536 7f9d64bda700  5 -- op tracker -- , seq: 747701, time: 2014-06-12 14:46:17.260533, event: commit_sent, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.260542 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2790 active+clean]  requeue_ops 
2014-06-12 14:46:17.260549 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean]  removing repgather(0x2a12e9c0 173'2792 rep_tid=236235 committed?=1 applied?=1 op=osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.260557 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean]    q front is repgather(0x2a12e9c0 173'2792 rep_tid=236235 committed?=1 applied?=1 op=osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.260565 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] remove_repop repgather(0x2a12e9c0 173'2792 rep_tid=236235 committed?=1 applied?=1 op=osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.260574 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean]  requeue_ops 
2014-06-12 14:46:17.260591 7f9d64bda700  5 -- op tracker -- , seq: 747701, time: 2014-06-12 14:46:17.260588, event: done, op: osd_op(client.4302.0:146061 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.260602 7f9d64bda700 10 osd.2 173 dequeue_op 0x20e89400 finish
2014-06-12 14:46:17.260607 7f9d64bda700  5 -- op tracker -- , seq: 747703, time: 2014-06-12 14:46:17.260605, event: done, op: osd_sub_op_reply(client.4302.0:146061 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.260613 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.260615 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.264279 7f9d4829e700  5 -- op tracker -- , seq: 747704, time: 2014-06-12 14:46:17.262865, event: header_read, op: osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.264309 7f9d4829e700  5 -- op tracker -- , seq: 747704, time: 2014-06-12 14:46:17.262868, event: throttled, op: osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.264333 7f9d4829e700  5 -- op tracker -- , seq: 747704, time: 2014-06-12 14:46:17.264226, event: all_read, op: osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.264355 7f9d4829e700  5 -- op tracker -- , seq: 747704, time: 0.000000, event: dispatched, op: osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.264369 7f9d4829e700 20 osd.2 173 should_share_map client.4302 172.16.96.220:0/1243795 173
2014-06-12 14:46:17.264416 7f9d4829e700 15 osd.2 173 enqueue_op 0x2fa9c400 prio 63 cost 516096 latency 0.001551 osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.264498 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.264518 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2fa9c400 prio 63 cost 516096 latency 0.001653 osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean]
2014-06-12 14:46:17.264557 7f9d6abe6700  5 -- op tracker -- , seq: 747704, time: 2014-06-12 14:46:17.264552, event: reached_pg, op: osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.264582 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.264599 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] handle_message: 0x2fa9c400
2014-06-12 14:46:17.264607 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] do_op osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.264631 7f9d6abe6700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_'
2014-06-12 14:46:17.264790 7f9d6abe6700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_' = 0
2014-06-12 14:46:17.264806 7f9d6abe6700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset'
2014-06-12 14:46:17.264881 7f9d6abe6700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset' = 0
2014-06-12 14:46:17.264890 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] populate_obc_watchers 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.264901 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.264908 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] get_object_context: creating obc from disk: 0x1e6c4dc0
2014-06-12 14:46:17.264915 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] get_object_context: 0x1e6c4dc0 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 rwstate(none n=0 w=0) oi: 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2792 client.4302.0:146061 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2792) ssc: 0x82df6c0 snapset: 0=[]:[]+head
2014-06-12 14:46:17.264927 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] find_object_context 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 @head oi=25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2792 client.4302.0:146061 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2792)
2014-06-12 14:46:17.264948 7f9d6abe6700  5 -- op tracker -- , seq: 747704, time: 2014-06-12 14:46:17.264944, event: started, op: osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.264958 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] execute_ctx 0x2b885600
2014-06-12 14:46:17.264972 7f9d6abe6700  5 -- op tracker -- , seq: 747704, time: 2014-06-12 14:46:17.264968, event: started, op: osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.264977 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] do_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] ov 173'2792 av 173'2793 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.264988 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] do_osd_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096]
2014-06-12 14:46:17.264996 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.265006 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] do_osd_op  write 1368064~516096
2014-06-12 14:46:17.265019 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] do_osd_op_effects on session 0x5c8d200
2014-06-12 14:46:17.265029 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 snapset=0x82df6f8  snapc=0=[]
2014-06-12 14:46:17.265038 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.265046 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] finish_ctx 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 0x2b885600 op modify  
2014-06-12 14:46:17.265060 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean]  set mtime to 2014-06-12 08:45:19.642850
2014-06-12 14:46:17.265077 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean]  final snapset 0=[]:[]+head in 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.265092 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.265101 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] new_repop rep_tid 236236 on osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.265113 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] issue_repop rep_tid 236236 o 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.265132 7f9d6abe6700  5 -- op tracker -- , seq: 747704, time: 2014-06-12 14:46:17.265129, event: waiting for subops from 29, op: osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.265149 7f9d6abe6700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:17.265166 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2792 (0'0,173'2792] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] append_log log((0'0,173'2792], crt=0'0) [173'2793 (173'2792) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146062 2014-06-12 08:45:19.642850]
2014-06-12 14:46:17.265187 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2792 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] add_log_entry 173'2793 (173'2792) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146062 2014-06-12 08:45:19.642850
2014-06-12 14:46:17.265204 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2792 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.265230 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2793, trimmed: 
2014-06-12 14:46:17.265248 7f9d6abe6700  5 queue_transactions existing osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.265253 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.265254 7f9d6abe6700  5 queue_op 0x6cfeae0 seq 2608 osr(3.330 0x276d4f0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:17.265260 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2792 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] eval_repop repgather(0x2a12f5c0 173'2793 rep_tid=236236 committed?=0 applied?=0 lock=0 op=osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.265275 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2fa9c400 finish
2014-06-12 14:46:17.265278 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.265280 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.265289 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541ca20 (1 active)
2014-06-12 14:46:17.265299 7f9d7a962700  5 _do_op 0x6cfeae0 seq 2608 osr(3.330 0x276d4f0)/0x276d4f0 start
2014-06-12 14:46:17.265303 7f9d7a962700 10 _do_transaction on 0xc9b8d80
2014-06-12 14:46:17.265310 7f9d7a962700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.265390 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.265406 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.265438 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.265555 7f9d7a962700 15 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.265561 7f9d7a962700 10 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1 r = 0
2014-06-12 14:46:17.265565 7f9d7a962700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.265576 7f9d7a962700 15 _write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 1368064~516096
2014-06-12 14:46:17.265692 7f9d7a962700 10 lookup_strip_header done  cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.265856 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:17.267000 7f9d7a962700 10 _generic_write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 1368064~516096 = 0
2014-06-12 14:46:17.267026 7f9d7a962700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.267034 7f9d7a962700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.267042 7f9d7a962700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.267047 7f9d7a962700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.267167 7f9d7a962700 20 set_header setting 17148 cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:17.267206 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.267215 7f9d7a962700 20 set_header setting 2048 cid meta oid 1fe2257a/pglog_3.330/0//-1 parent seq 0
2014-06-12 14:46:17.267221 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:17.269031 7f9d7a962700 10 _do_op 0x6cfeae0 seq 2608 r = 0, finisher 0x291aaa0 0x3b46d940
2014-06-12 14:46:17.269041 7f9d7a962700 10 _finish_op 0x6cfeae0 seq 2608 osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.269047 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541ca20 (0 active)
2014-06-12 14:46:17.269051 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.269047 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2792 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] op_commit: 236236
2014-06-12 14:46:17.269073 7f9d7715b700  5 -- op tracker -- , seq: 747704, time: 2014-06-12 14:46:17.269065, event: op_commit, op: osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.269159 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2792 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] op_applied: 236236
2014-06-12 14:46:17.269208 7f9d7795c700  5 -- op tracker -- , seq: 747704, time: 2014-06-12 14:46:17.269196, event: op_applied, op: osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.269232 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2792 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] op_applied on primary on version 173'2793
2014-06-12 14:46:17.271102 7f9d5501e700  5 -- op tracker -- , seq: 747705, time: 2014-06-12 14:46:17.271038, event: header_read, op: osd_sub_op_reply(client.4302.0:146062 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.271121 7f9d5501e700  5 -- op tracker -- , seq: 747705, time: 2014-06-12 14:46:17.271040, event: throttled, op: osd_sub_op_reply(client.4302.0:146062 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.271132 7f9d5501e700  5 -- op tracker -- , seq: 747705, time: 2014-06-12 14:46:17.271058, event: all_read, op: osd_sub_op_reply(client.4302.0:146062 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.271141 7f9d5501e700  5 -- op tracker -- , seq: 747705, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146062 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.271148 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146062 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.271156 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.271165 7f9d5501e700 15 osd.2 173 enqueue_op 0x34017500 prio 196 cost 0 latency 0.000127 osd_sub_op_reply(client.4302.0:146062 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.271234 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.271257 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34017500 prio 196 cost 0 latency 0.000218 osd_sub_op_reply(client.4302.0:146062 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 pg pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2792 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean]
2014-06-12 14:46:17.271288 7f9d6a3e5700  5 -- op tracker -- , seq: 747705, time: 2014-06-12 14:46:17.271285, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146062 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.271298 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2792 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] handle_message: 0x34017500
2014-06-12 14:46:17.271310 7f9d6a3e5700  5 -- op tracker -- , seq: 747705, time: 2014-06-12 14:46:17.271307, event: started, op: osd_sub_op_reply(client.4302.0:146062 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.271317 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2792 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] sub_op_modify_reply: tid 236236 op  ack_type 4 from 29
2014-06-12 14:46:17.271332 7f9d6a3e5700  5 -- op tracker -- , seq: 747704, time: 2014-06-12 14:46:17.271327, event: sub_op_commit_rec, op: osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.271338 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2792 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] repop_all_applied: repop tid 236236 all applied 
2014-06-12 14:46:17.271346 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2792 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] eval_repop repgather(0x2a12f5c0 173'2793 rep_tid=236236 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.271360 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2792 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean]  sending ack on repgather(0x2a12f5c0 173'2793 rep_tid=236236 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x20236280
2014-06-12 14:46:17.271390 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2792 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] repop_all_committed: repop tid 236236 all committed 
2014-06-12 14:46:17.271399 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2791 active+clean] eval_repop repgather(0x2a12f5c0 173'2793 rep_tid=236236 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.271412 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2791 active+clean] log_op_stats osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008524 lat 0.008546
2014-06-12 14:46:17.271426 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2791 active+clean] publish_stats_to_osd 173:3598
2014-06-12 14:46:17.271436 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2791 active+clean]  sending commit on repgather(0x2a12f5c0 173'2793 rep_tid=236236 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x10149900
2014-06-12 14:46:17.271469 7f9d6a3e5700  5 -- op tracker -- , seq: 747704, time: 2014-06-12 14:46:17.271457, event: commit_sent, op: osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.271476 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2791 active+clean]  requeue_ops 
2014-06-12 14:46:17.271483 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean]  removing repgather(0x2a12f5c0 173'2793 rep_tid=236236 committed?=1 applied?=1 op=osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.271500 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean]    q front is repgather(0x2a12f5c0 173'2793 rep_tid=236236 committed?=1 applied?=1 op=osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.271523 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] remove_repop repgather(0x2a12f5c0 173'2793 rep_tid=236236 committed?=1 applied?=1 op=osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.271535 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean]  requeue_ops 
2014-06-12 14:46:17.271555 7f9d6a3e5700  5 -- op tracker -- , seq: 747704, time: 2014-06-12 14:46:17.271551, event: done, op: osd_op(client.4302.0:146062 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1368064~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.271570 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34017500 finish
2014-06-12 14:46:17.271577 7f9d6a3e5700  5 -- op tracker -- , seq: 747705, time: 2014-06-12 14:46:17.271574, event: done, op: osd_sub_op_reply(client.4302.0:146062 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.271585 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.271588 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.275098 7f9d4829e700  5 -- op tracker -- , seq: 747706, time: 2014-06-12 14:46:17.273888, event: header_read, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.275108 7f9d4829e700  5 -- op tracker -- , seq: 747706, time: 2014-06-12 14:46:17.273891, event: throttled, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.275114 7f9d4829e700  5 -- op tracker -- , seq: 747706, time: 2014-06-12 14:46:17.275061, event: all_read, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.275128 7f9d4829e700  5 -- op tracker -- , seq: 747706, time: 0.000000, event: dispatched, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.275132 7f9d4829e700 20 osd.2 173 should_share_map client.4302 172.16.96.220:0/1243795 173
2014-06-12 14:46:17.275151 7f9d4829e700 15 osd.2 173 enqueue_op 0x2fa9c900 prio 63 cost 516096 latency 0.001262 osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.275210 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.275219 7f9d66bde700 10 osd.2 173 dequeue_op 0x2fa9c900 prio 63 cost 516096 latency 0.001331 osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean]
2014-06-12 14:46:17.275240 7f9d66bde700  5 -- op tracker -- , seq: 747706, time: 2014-06-12 14:46:17.275236, event: reached_pg, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.275249 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.275260 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] handle_message: 0x2fa9c900
2014-06-12 14:46:17.275268 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] do_op osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.275284 7f9d66bde700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_'
2014-06-12 14:46:17.275425 7f9d66bde700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_' = 0
2014-06-12 14:46:17.275437 7f9d66bde700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset'
2014-06-12 14:46:17.275495 7f9d66bde700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset' = 0
2014-06-12 14:46:17.275503 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] populate_obc_watchers 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.275511 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.275517 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] get_object_context: creating obc from disk: 0x3a08c000
2014-06-12 14:46:17.275523 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] get_object_context: 0x3a08c000 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 rwstate(none n=0 w=0) oi: 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2793 client.4302.0:146062 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2793) ssc: 0x1e15f260 snapset: 0=[]:[]+head
2014-06-12 14:46:17.275533 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] find_object_context 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 @head oi=25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2793 client.4302.0:146062 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2793)
2014-06-12 14:46:17.275550 7f9d66bde700  5 -- op tracker -- , seq: 747706, time: 2014-06-12 14:46:17.275546, event: started, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.275556 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] execute_ctx 0x1fc52000
2014-06-12 14:46:17.275567 7f9d66bde700  5 -- op tracker -- , seq: 747706, time: 2014-06-12 14:46:17.275564, event: started, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.275572 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] do_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] ov 173'2793 av 173'2794 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.275582 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] do_osd_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096]
2014-06-12 14:46:17.275588 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.275597 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] do_osd_op  write 1884160~516096
2014-06-12 14:46:17.275607 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] do_osd_op_effects on session 0x5c8d200
2014-06-12 14:46:17.275616 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 snapset=0x1e15f298  snapc=0=[]
2014-06-12 14:46:17.275625 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.275631 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] finish_ctx 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 0x1fc52000 op modify  
2014-06-12 14:46:17.275639 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean]  set mtime to 2014-06-12 08:45:19.653890
2014-06-12 14:46:17.275652 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean]  final snapset 0=[]:[]+head in 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.275667 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.275676 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] new_repop rep_tid 236237 on osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.275688 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] issue_repop rep_tid 236237 o 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.275708 7f9d66bde700  5 -- op tracker -- , seq: 747706, time: 2014-06-12 14:46:17.275705, event: waiting for subops from 29, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.275723 7f9d66bde700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:17.275737 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2793 (0'0,173'2793] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] append_log log((0'0,173'2793], crt=0'0) [173'2794 (173'2793) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146063 2014-06-12 08:45:19.653890]
2014-06-12 14:46:17.275752 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] add_log_entry 173'2794 (173'2793) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146063 2014-06-12 08:45:19.653890
2014-06-12 14:46:17.275766 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.275784 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2794, trimmed: 
2014-06-12 14:46:17.275798 7f9d66bde700  5 queue_transactions existing osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.275801 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.275802 7f9d66bde700  5 queue_op 0x8857c80 seq 2609 osr(3.330 0x276d4f0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:17.275807 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] eval_repop repgather(0x224e480 173'2794 rep_tid=236237 committed?=0 applied?=0 lock=0 op=osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.275813 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541ca20 (1 active)
2014-06-12 14:46:17.275818 7f9d66bde700 10 osd.2 173 dequeue_op 0x2fa9c900 finish
2014-06-12 14:46:17.275821 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.275823 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.275821 7f9d7f16b700  5 _do_op 0x8857c80 seq 2609 osr(3.330 0x276d4f0)/0x276d4f0 start
2014-06-12 14:46:17.275825 7f9d7f16b700 10 _do_transaction on 0x182246c0
2014-06-12 14:46:17.275831 7f9d7f16b700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.275931 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.275949 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.275996 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.276113 7f9d7f16b700 15 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.276121 7f9d7f16b700 10 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1 r = 0
2014-06-12 14:46:17.276127 7f9d7f16b700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.276142 7f9d7f16b700 15 _write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 1884160~516096
2014-06-12 14:46:17.276269 7f9d7f16b700 10 lookup_strip_header done  cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.276512 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:17.278217 7f9d7f16b700 10 _generic_write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 1884160~516096 = 0
2014-06-12 14:46:17.278246 7f9d7f16b700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.278257 7f9d7f16b700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.278262 7f9d7f16b700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.278267 7f9d7f16b700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.278440 7f9d7f16b700 20 set_header setting 17148 cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:17.278498 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.278508 7f9d7f16b700 20 set_header setting 2048 cid meta oid 1fe2257a/pglog_3.330/0//-1 parent seq 0
2014-06-12 14:46:17.278516 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:17.280078 7f9d7f16b700 10 _do_op 0x8857c80 seq 2609 r = 0, finisher 0x8941d60 0x3b464f80
2014-06-12 14:46:17.280088 7f9d7f16b700 10 _finish_op 0x8857c80 seq 2609 osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.280093 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541ca20 (0 active)
2014-06-12 14:46:17.280098 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.280099 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] op_applied: 236237
2014-06-12 14:46:17.280122 7f9d7795c700  5 -- op tracker -- , seq: 747706, time: 2014-06-12 14:46:17.280116, event: op_applied, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.280137 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] op_applied on primary on version 173'2794
2014-06-12 14:46:17.280224 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] op_commit: 236237
2014-06-12 14:46:17.280276 7f9d7715b700  5 -- op tracker -- , seq: 747706, time: 2014-06-12 14:46:17.280264, event: op_commit, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.285291 7f9d5501e700  5 -- op tracker -- , seq: 747707, time: 2014-06-12 14:46:17.285221, event: header_read, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.285316 7f9d5501e700  5 -- op tracker -- , seq: 747707, time: 2014-06-12 14:46:17.285231, event: throttled, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.285335 7f9d5501e700  5 -- op tracker -- , seq: 747707, time: 2014-06-12 14:46:17.285249, event: all_read, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.285352 7f9d5501e700  5 -- op tracker -- , seq: 747707, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.285365 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:17.285371 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.285378 7f9d5501e700 15 osd.2 173 enqueue_op 0x2910200 prio 196 cost 0 latency 0.000157 osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2
2014-06-12 14:46:17.285429 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.285442 7f9d69be4700 10 osd.2 173 dequeue_op 0x2910200 prio 196 cost 0 latency 0.000220 osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2 pg pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean]
2014-06-12 14:46:17.285470 7f9d69be4700  5 -- op tracker -- , seq: 747707, time: 2014-06-12 14:46:17.285460, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.285478 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] handle_message: 0x2910200
2014-06-12 14:46:17.285483 7f9d5501e700  5 -- op tracker -- , seq: 747708, time: 2014-06-12 14:46:17.285412, event: header_read, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.285493 7f9d5501e700  5 -- op tracker -- , seq: 747708, time: 2014-06-12 14:46:17.285413, event: throttled, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.285502 7f9d5501e700  5 -- op tracker -- , seq: 747708, time: 2014-06-12 14:46:17.285445, event: all_read, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.285508 7f9d5501e700  5 -- op tracker -- , seq: 747708, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.285513 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.285517 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.285517 7f9d69be4700  5 -- op tracker -- , seq: 747707, time: 2014-06-12 14:46:17.285486, event: started, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.285520 7f9d5501e700 15 osd.2 173 enqueue_op 0x329eaa00 prio 196 cost 0 latency 0.000108 osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.285523 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] sub_op_modify_reply: tid 236237 op  ack_type 1 from 29
2014-06-12 14:46:17.285536 7f9d69be4700  5 -- op tracker -- , seq: 747706, time: 2014-06-12 14:46:17.285532, event: sub_op_applied_rec, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.285553 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] repop_all_applied: repop tid 236237 all applied 
2014-06-12 14:46:17.285565 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:17.285561 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] eval_repop repgather(0x224e480 173'2794 rep_tid=236237 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.285572 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean]  sending ack on repgather(0x224e480 173'2794 rep_tid=236237 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x22ef7680
2014-06-12 14:46:17.285603 7f9d69be4700 10 osd.2 173 dequeue_op 0x2910200 finish
2014-06-12 14:46:17.285609 7f9d69be4700  5 -- op tracker -- , seq: 747707, time: 2014-06-12 14:46:17.285607, event: done, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.285610 7f9d653db700 10 osd.2 173 dequeue_op 0x329eaa00 prio 196 cost 0 latency 0.000197 osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 pg pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean]
2014-06-12 14:46:17.285642 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:17.285646 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.285645 7f9d653db700  5 -- op tracker -- , seq: 747708, time: 2014-06-12 14:46:17.285642, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.285650 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] handle_message: 0x329eaa00
2014-06-12 14:46:17.285659 7f9d653db700  5 -- op tracker -- , seq: 747708, time: 2014-06-12 14:46:17.285657, event: started, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.285662 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] sub_op_modify_reply: tid 236237 op  ack_type 4 from 29
2014-06-12 14:46:17.285682 7f9d653db700  5 -- op tracker -- , seq: 747706, time: 2014-06-12 14:46:17.285678, event: sub_op_commit_rec, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.285688 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2793 crt=0'0 lcod 173'2792 mlcod 173'2792 active+clean] repop_all_committed: repop tid 236237 all committed 
2014-06-12 14:46:17.285694 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2792 active+clean] eval_repop repgather(0x224e480 173'2794 rep_tid=236237 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.285704 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2792 active+clean] log_op_stats osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.011714 lat 0.011815
2014-06-12 14:46:17.285714 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2792 active+clean] publish_stats_to_osd 173:3599
2014-06-12 14:46:17.285722 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2792 active+clean]  sending commit on repgather(0x224e480 173'2794 rep_tid=236237 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x15cd6280
2014-06-12 14:46:17.285742 7f9d653db700  5 -- op tracker -- , seq: 747706, time: 2014-06-12 14:46:17.285739, event: commit_sent, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.285756 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2792 active+clean]  requeue_ops 
2014-06-12 14:46:17.285763 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean]  removing repgather(0x224e480 173'2794 rep_tid=236237 committed?=1 applied?=1 op=osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.285771 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean]    q front is repgather(0x224e480 173'2794 rep_tid=236237 committed?=1 applied?=1 op=osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.285779 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] remove_repop repgather(0x224e480 173'2794 rep_tid=236237 committed?=1 applied?=1 op=osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.285787 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean]  requeue_ops 
2014-06-12 14:46:17.285804 7f9d653db700  5 -- op tracker -- , seq: 747706, time: 2014-06-12 14:46:17.285801, event: done, op: osd_op(client.4302.0:146063 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.285815 7f9d653db700 10 osd.2 173 dequeue_op 0x329eaa00 finish
2014-06-12 14:46:17.285819 7f9d653db700  5 -- op tracker -- , seq: 747708, time: 2014-06-12 14:46:17.285817, event: done, op: osd_sub_op_reply(client.4302.0:146063 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.285825 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.285827 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.289531 7f9d4829e700  5 -- op tracker -- , seq: 747709, time: 2014-06-12 14:46:17.288156, event: header_read, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.289568 7f9d4829e700  5 -- op tracker -- , seq: 747709, time: 2014-06-12 14:46:17.288160, event: throttled, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.289591 7f9d4829e700  5 -- op tracker -- , seq: 747709, time: 2014-06-12 14:46:17.289470, event: all_read, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.289612 7f9d4829e700  5 -- op tracker -- , seq: 747709, time: 0.000000, event: dispatched, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.289633 7f9d4829e700 20 osd.2 173 should_share_map client.4302 172.16.96.220:0/1243795 173
2014-06-12 14:46:17.289661 7f9d4829e700 15 osd.2 173 enqueue_op 0x17c2b300 prio 63 cost 516096 latency 0.001505 osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.289722 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.289744 7f9d63bd8700 10 osd.2 173 dequeue_op 0x17c2b300 prio 63 cost 516096 latency 0.001588 osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean]
2014-06-12 14:46:17.289791 7f9d63bd8700  5 -- op tracker -- , seq: 747709, time: 2014-06-12 14:46:17.289782, event: reached_pg, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.289810 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.289835 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] handle_message: 0x17c2b300
2014-06-12 14:46:17.289842 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] do_op osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.289868 7f9d63bd8700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_'
2014-06-12 14:46:17.290113 7f9d63bd8700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_' = 0
2014-06-12 14:46:17.290143 7f9d63bd8700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset'
2014-06-12 14:46:17.290288 7f9d63bd8700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset' = 0
2014-06-12 14:46:17.290305 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] populate_obc_watchers 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.290322 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.290335 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] get_object_context: creating obc from disk: 0x11f642c0
2014-06-12 14:46:17.290347 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] get_object_context: 0x11f642c0 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 rwstate(none n=0 w=0) oi: 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2794 client.4302.0:146063 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2794) ssc: 0x5d5ab60 snapset: 0=[]:[]+head
2014-06-12 14:46:17.290368 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] find_object_context 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 @head oi=25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2794 client.4302.0:146063 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2794)
2014-06-12 14:46:17.290406 7f9d63bd8700  5 -- op tracker -- , seq: 747709, time: 2014-06-12 14:46:17.290398, event: started, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.290418 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] execute_ctx 0x29edf000
2014-06-12 14:46:17.290440 7f9d63bd8700  5 -- op tracker -- , seq: 747709, time: 2014-06-12 14:46:17.290437, event: started, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.290444 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] do_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] ov 173'2794 av 173'2795 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.290452 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] do_osd_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096]
2014-06-12 14:46:17.290458 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.290467 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] do_osd_op  write 2400256~516096
2014-06-12 14:46:17.290476 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] do_osd_op_effects on session 0x5c8d200
2014-06-12 14:46:17.290483 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 snapset=0x5d5ab98  snapc=0=[]
2014-06-12 14:46:17.290490 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.290495 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] finish_ctx 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 0x29edf000 op modify  
2014-06-12 14:46:17.290508 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean]  set mtime to 2014-06-12 08:45:19.668086
2014-06-12 14:46:17.290524 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean]  final snapset 0=[]:[]+head in 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.290557 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.290571 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] new_repop rep_tid 236238 on osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.290590 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] issue_repop rep_tid 236238 o 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.290605 7f9d63bd8700  5 -- op tracker -- , seq: 747709, time: 2014-06-12 14:46:17.290602, event: waiting for subops from 29, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.290619 7f9d63bd8700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:17.290634 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2794 (0'0,173'2794] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] append_log log((0'0,173'2794], crt=0'0) [173'2795 (173'2794) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146064 2014-06-12 08:45:19.668086]
2014-06-12 14:46:17.290657 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] add_log_entry 173'2795 (173'2794) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146064 2014-06-12 08:45:19.668086
2014-06-12 14:46:17.290671 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.290693 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2795, trimmed: 
2014-06-12 14:46:17.290729 7f9d63bd8700  5 queue_transactions existing osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.290734 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.290736 7f9d63bd8700  5 queue_op 0x5e6cc00 seq 2610 osr(3.330 0x276d4f0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:17.290741 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] eval_repop repgather(0x534fc80 173'2795 rep_tid=236238 committed?=0 applied?=0 lock=0 op=osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.290748 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541ca20 (1 active)
2014-06-12 14:46:17.290763 7f9d63bd8700 10 osd.2 173 dequeue_op 0x17c2b300 finish
2014-06-12 14:46:17.290766 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.290767 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.290765 7f9d7895e700  5 _do_op 0x5e6cc00 seq 2610 osr(3.330 0x276d4f0)/0x276d4f0 start
2014-06-12 14:46:17.290771 7f9d7895e700 10 _do_transaction on 0x13b43ec0
2014-06-12 14:46:17.290778 7f9d7895e700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.290889 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.290917 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.290989 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.291136 7f9d7895e700 15 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.291147 7f9d7895e700 10 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1 r = 0
2014-06-12 14:46:17.291156 7f9d7895e700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.291181 7f9d7895e700 15 _write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 2400256~516096
2014-06-12 14:46:17.291366 7f9d7895e700 10 lookup_strip_header done  cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.291561 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:17.293749 7f9d7895e700 10 _generic_write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 2400256~516096 = 0
2014-06-12 14:46:17.293823 7f9d7895e700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.293851 7f9d7895e700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.293865 7f9d7895e700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.293892 7f9d7895e700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.294153 7f9d7895e700 20 set_header setting 17148 cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:17.294262 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.294294 7f9d7895e700 20 set_header setting 2048 cid meta oid 1fe2257a/pglog_3.330/0//-1 parent seq 0
2014-06-12 14:46:17.294319 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:17.297903 7f9d7895e700 10 _do_op 0x5e6cc00 seq 2610 r = 0, finisher 0x4e88ec0 0x4dd9b80
2014-06-12 14:46:17.297915 7f9d7895e700 10 _finish_op 0x5e6cc00 seq 2610 osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.297923 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541ca20 (0 active)
2014-06-12 14:46:17.297936 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.297938 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] op_applied: 236238
2014-06-12 14:46:17.297971 7f9d7795c700  5 -- op tracker -- , seq: 747709, time: 2014-06-12 14:46:17.297956, event: op_applied, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.297981 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] op_applied on primary on version 173'2795
2014-06-12 14:46:17.298030 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] op_commit: 236238
2014-06-12 14:46:17.298054 7f9d7715b700  5 -- op tracker -- , seq: 747709, time: 2014-06-12 14:46:17.298048, event: op_commit, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.299399 7f9d5501e700  5 -- op tracker -- , seq: 747710, time: 2014-06-12 14:46:17.299325, event: header_read, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.299423 7f9d5501e700  5 -- op tracker -- , seq: 747710, time: 2014-06-12 14:46:17.299327, event: throttled, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.299434 7f9d5501e700  5 -- op tracker -- , seq: 747710, time: 2014-06-12 14:46:17.299354, event: all_read, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.299468 7f9d5501e700  5 -- op tracker -- , seq: 747710, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.299483 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:17.299487 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.299493 7f9d5501e700 15 osd.2 173 enqueue_op 0x8dbc100 prio 196 cost 0 latency 0.000168 osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2
2014-06-12 14:46:17.299518 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.299528 7f9d68be2700 10 osd.2 173 dequeue_op 0x8dbc100 prio 196 cost 0 latency 0.000203 osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2 pg pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean]
2014-06-12 14:46:17.299545 7f9d5501e700  5 -- op tracker -- , seq: 747711, time: 2014-06-12 14:46:17.299510, event: header_read, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.299559 7f9d5501e700  5 -- op tracker -- , seq: 747711, time: 2014-06-12 14:46:17.299511, event: throttled, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.299570 7f9d5501e700  5 -- op tracker -- , seq: 747711, time: 2014-06-12 14:46:17.299524, event: all_read, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.299581 7f9d5501e700  5 -- op tracker -- , seq: 747711, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.299586 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.299591 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.299590 7f9d68be2700  5 -- op tracker -- , seq: 747710, time: 2014-06-12 14:46:17.299549, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.299596 7f9d5501e700 15 osd.2 173 enqueue_op 0x8dbdf00 prio 196 cost 0 latency 0.000086 osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.299601 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] handle_message: 0x8dbc100
2014-06-12 14:46:17.299614 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:17.299619 7f9d68be2700  5 -- op tracker -- , seq: 747710, time: 2014-06-12 14:46:17.299614, event: started, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.299623 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] sub_op_modify_reply: tid 236238 op  ack_type 1 from 29
2014-06-12 14:46:17.299637 7f9d68be2700  5 -- op tracker -- , seq: 747709, time: 2014-06-12 14:46:17.299632, event: sub_op_applied_rec, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.299643 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] repop_all_applied: repop tid 236238 all applied 
2014-06-12 14:46:17.299650 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] eval_repop repgather(0x534fc80 173'2795 rep_tid=236238 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.299665 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean]  sending ack on repgather(0x534fc80 173'2795 rep_tid=236238 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x16cf3b80
2014-06-12 14:46:17.299688 7f9d68be2700 10 osd.2 173 dequeue_op 0x8dbc100 finish
2014-06-12 14:46:17.299694 7f9d68be2700  5 -- op tracker -- , seq: 747710, time: 2014-06-12 14:46:17.299692, event: done, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.299694 7f9d633d7700 10 osd.2 173 dequeue_op 0x8dbdf00 prio 196 cost 0 latency 0.000184 osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 pg pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean]
2014-06-12 14:46:17.299706 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:17.299714 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.299714 7f9d633d7700  5 -- op tracker -- , seq: 747711, time: 2014-06-12 14:46:17.299712, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.299720 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] handle_message: 0x8dbdf00
2014-06-12 14:46:17.299730 7f9d633d7700  5 -- op tracker -- , seq: 747711, time: 2014-06-12 14:46:17.299728, event: started, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.299734 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] sub_op_modify_reply: tid 236238 op  ack_type 4 from 29
2014-06-12 14:46:17.299745 7f9d633d7700  5 -- op tracker -- , seq: 747709, time: 2014-06-12 14:46:17.299742, event: sub_op_commit_rec, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.299749 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2794 crt=0'0 lcod 173'2793 mlcod 173'2793 active+clean] repop_all_committed: repop tid 236238 all committed 
2014-06-12 14:46:17.299756 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2793 active+clean] eval_repop repgather(0x534fc80 173'2795 rep_tid=236238 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.299767 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2793 active+clean] log_op_stats osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.011531 lat 0.011610
2014-06-12 14:46:17.299780 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2793 active+clean] publish_stats_to_osd 173:3600
2014-06-12 14:46:17.299789 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2793 active+clean]  sending commit on repgather(0x534fc80 173'2795 rep_tid=236238 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x355c1b80
2014-06-12 14:46:17.299811 7f9d633d7700  5 -- op tracker -- , seq: 747709, time: 2014-06-12 14:46:17.299808, event: commit_sent, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.299817 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2793 active+clean]  requeue_ops 
2014-06-12 14:46:17.299823 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean]  removing repgather(0x534fc80 173'2795 rep_tid=236238 committed?=1 applied?=1 op=osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.299832 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean]    q front is repgather(0x534fc80 173'2795 rep_tid=236238 committed?=1 applied?=1 op=osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.299841 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] remove_repop repgather(0x534fc80 173'2795 rep_tid=236238 committed?=1 applied?=1 op=osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.299850 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean]  requeue_ops 
2014-06-12 14:46:17.299867 7f9d633d7700  5 -- op tracker -- , seq: 747709, time: 2014-06-12 14:46:17.299864, event: done, op: osd_op(client.4302.0:146064 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2400256~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.299879 7f9d633d7700 10 osd.2 173 dequeue_op 0x8dbdf00 finish
2014-06-12 14:46:17.299885 7f9d633d7700  5 -- op tracker -- , seq: 747711, time: 2014-06-12 14:46:17.299883, event: done, op: osd_sub_op_reply(client.4302.0:146064 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.299891 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.299893 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.303263 7f9d4829e700  5 -- op tracker -- , seq: 747712, time: 2014-06-12 14:46:17.302206, event: header_read, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.303274 7f9d4829e700  5 -- op tracker -- , seq: 747712, time: 2014-06-12 14:46:17.302208, event: throttled, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.303280 7f9d4829e700  5 -- op tracker -- , seq: 747712, time: 2014-06-12 14:46:17.303229, event: all_read, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.303293 7f9d4829e700  5 -- op tracker -- , seq: 747712, time: 0.000000, event: dispatched, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.303299 7f9d4829e700 20 osd.2 173 should_share_map client.4302 172.16.96.220:0/1243795 173
2014-06-12 14:46:17.303319 7f9d4829e700 15 osd.2 173 enqueue_op 0x17c2ae00 prio 63 cost 516096 latency 0.001113 osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.303343 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.303350 7f9d623d5700 10 osd.2 173 dequeue_op 0x17c2ae00 prio 63 cost 516096 latency 0.001144 osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean]
2014-06-12 14:46:17.303367 7f9d623d5700  5 -- op tracker -- , seq: 747712, time: 2014-06-12 14:46:17.303364, event: reached_pg, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.303374 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.303383 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] handle_message: 0x17c2ae00
2014-06-12 14:46:17.303389 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] do_op osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.303407 7f9d623d5700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_'
2014-06-12 14:46:17.303546 7f9d623d5700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_' = 0
2014-06-12 14:46:17.303559 7f9d623d5700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset'
2014-06-12 14:46:17.303615 7f9d623d5700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset' = 0
2014-06-12 14:46:17.303622 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] populate_obc_watchers 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.303632 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.303637 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] get_object_context: creating obc from disk: 0x15575b80
2014-06-12 14:46:17.303643 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] get_object_context: 0x15575b80 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 rwstate(none n=0 w=0) oi: 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2795 client.4302.0:146064 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2795) ssc: 0x679b420 snapset: 0=[]:[]+head
2014-06-12 14:46:17.303652 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] find_object_context 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 @head oi=25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2795 client.4302.0:146064 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2795)
2014-06-12 14:46:17.303668 7f9d623d5700  5 -- op tracker -- , seq: 747712, time: 2014-06-12 14:46:17.303665, event: started, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.303674 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] execute_ctx 0x5ca1e00
2014-06-12 14:46:17.303683 7f9d623d5700  5 -- op tracker -- , seq: 747712, time: 2014-06-12 14:46:17.303681, event: started, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.303687 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] do_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] ov 173'2795 av 173'2796 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.303695 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] do_osd_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096]
2014-06-12 14:46:17.303701 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.303709 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] do_osd_op  write 2916352~516096
2014-06-12 14:46:17.303719 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] do_osd_op_effects on session 0x5c8d200
2014-06-12 14:46:17.303727 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 snapset=0x679b458  snapc=0=[]
2014-06-12 14:46:17.303734 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.303740 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] finish_ctx 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 0x5ca1e00 op modify  
2014-06-12 14:46:17.303748 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean]  set mtime to 2014-06-12 08:45:19.682174
2014-06-12 14:46:17.303760 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean]  final snapset 0=[]:[]+head in 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.303771 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.303778 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] new_repop rep_tid 236239 on osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.303788 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] issue_repop rep_tid 236239 o 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.303802 7f9d623d5700  5 -- op tracker -- , seq: 747712, time: 2014-06-12 14:46:17.303800, event: waiting for subops from 29, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.303817 7f9d623d5700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:17.303832 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2795 (0'0,173'2795] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] append_log log((0'0,173'2795], crt=0'0) [173'2796 (173'2795) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146065 2014-06-12 08:45:19.682174]
2014-06-12 14:46:17.303856 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] add_log_entry 173'2796 (173'2795) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146065 2014-06-12 08:45:19.682174
2014-06-12 14:46:17.303874 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.303904 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2796, trimmed: 
2014-06-12 14:46:17.303923 7f9d623d5700  5 queue_transactions existing osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.303928 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.303929 7f9d623d5700  5 queue_op 0x69ccc00 seq 2611 osr(3.330 0x276d4f0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:17.303935 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] eval_repop repgather(0x2562d80 173'2796 rep_tid=236239 committed?=0 applied?=0 lock=0 op=osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.303957 7f9d623d5700 10 osd.2 173 dequeue_op 0x17c2ae00 finish
2014-06-12 14:46:17.303961 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.303963 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.303967 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541ca20 (1 active)
2014-06-12 14:46:17.303974 7f9d7e169700  5 _do_op 0x69ccc00 seq 2611 osr(3.330 0x276d4f0)/0x276d4f0 start
2014-06-12 14:46:17.303978 7f9d7e169700 10 _do_transaction on 0x224e180
2014-06-12 14:46:17.303983 7f9d7e169700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.304064 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.304086 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.304122 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.304142 7f9d7e169700 15 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.304147 7f9d7e169700 10 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1 r = 0
2014-06-12 14:46:17.304151 7f9d7e169700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.304260 7f9d7e169700 15 _write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 2916352~516096
2014-06-12 14:46:17.304353 7f9d7e169700 10 lookup_strip_header done  cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.304475 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:17.305758 7f9d7e169700 10 _generic_write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 2916352~516096 = 0
2014-06-12 14:46:17.305809 7f9d7e169700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.305828 7f9d7e169700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.305841 7f9d7e169700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.305846 7f9d7e169700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.305987 7f9d7e169700 20 set_header setting 17148 cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:17.306039 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.306050 7f9d7e169700 20 set_header setting 2048 cid meta oid 1fe2257a/pglog_3.330/0//-1 parent seq 0
2014-06-12 14:46:17.306056 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:17.307751 7f9d7e169700 10 _do_op 0x69ccc00 seq 2611 r = 0, finisher 0x4f552c0 0x2a164840
2014-06-12 14:46:17.307768 7f9d7e169700 10 _finish_op 0x69ccc00 seq 2611 osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.307783 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541ca20 (0 active)
2014-06-12 14:46:17.307794 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.307790 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] op_commit: 236239
2014-06-12 14:46:17.307835 7f9d7715b700  5 -- op tracker -- , seq: 747712, time: 2014-06-12 14:46:17.307817, event: op_commit, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.307880 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] op_applied: 236239
2014-06-12 14:46:17.307911 7f9d7795c700  5 -- op tracker -- , seq: 747712, time: 2014-06-12 14:46:17.307906, event: op_applied, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.307928 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] op_applied on primary on version 173'2796
2014-06-12 14:46:17.310859 7f9d5501e700  5 -- op tracker -- , seq: 747713, time: 2014-06-12 14:46:17.310804, event: header_read, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.310873 7f9d5501e700  5 -- op tracker -- , seq: 747713, time: 2014-06-12 14:46:17.310806, event: throttled, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.310887 7f9d5501e700  5 -- op tracker -- , seq: 747713, time: 2014-06-12 14:46:17.310821, event: all_read, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.310892 7f9d5501e700  5 -- op tracker -- , seq: 747713, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.310897 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:17.310902 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.310908 7f9d5501e700 15 osd.2 173 enqueue_op 0x2911a00 prio 196 cost 0 latency 0.000104 osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2
2014-06-12 14:46:17.310934 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.310946 7f9d643d9700 10 osd.2 173 dequeue_op 0x2911a00 prio 196 cost 0 latency 0.000141 osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0) v2 pg pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean]
2014-06-12 14:46:17.310966 7f9d5501e700  5 -- op tracker -- , seq: 747714, time: 2014-06-12 14:46:17.310928, event: header_read, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.310986 7f9d5501e700  5 -- op tracker -- , seq: 747714, time: 2014-06-12 14:46:17.310928, event: throttled, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.310997 7f9d5501e700  5 -- op tracker -- , seq: 747714, time: 2014-06-12 14:46:17.310945, event: all_read, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.311007 7f9d5501e700  5 -- op tracker -- , seq: 747714, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.311014 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.311019 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.311018 7f9d643d9700  5 -- op tracker -- , seq: 747713, time: 2014-06-12 14:46:17.310967, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.311029 7f9d5501e700 15 osd.2 173 enqueue_op 0x2910300 prio 196 cost 0 latency 0.000101 osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.311033 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] handle_message: 0x2911a00
2014-06-12 14:46:17.311050 7f9d643d9700  5 -- op tracker -- , seq: 747713, time: 2014-06-12 14:46:17.311046, event: started, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.311054 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:17.311058 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] sub_op_modify_reply: tid 236239 op  ack_type 1 from 29
2014-06-12 14:46:17.311075 7f9d643d9700  5 -- op tracker -- , seq: 747712, time: 2014-06-12 14:46:17.311067, event: sub_op_applied_rec, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.311080 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] repop_all_applied: repop tid 236239 all applied 
2014-06-12 14:46:17.311088 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] eval_repop repgather(0x2562d80 173'2796 rep_tid=236239 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.311101 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean]  sending ack on repgather(0x2562d80 173'2796 rep_tid=236239 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x6486000
2014-06-12 14:46:17.311126 7f9d643d9700 10 osd.2 173 dequeue_op 0x2911a00 finish
2014-06-12 14:46:17.311133 7f9d643d9700  5 -- op tracker -- , seq: 747713, time: 2014-06-12 14:46:17.311130, event: done, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ack, result = 0)
2014-06-12 14:46:17.311135 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2910300 prio 196 cost 0 latency 0.000207 osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 pg pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean]
2014-06-12 14:46:17.311157 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:17.311160 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.311161 7f9d61bd4700  5 -- op tracker -- , seq: 747714, time: 2014-06-12 14:46:17.311151, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.311169 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] handle_message: 0x2910300
2014-06-12 14:46:17.311190 7f9d61bd4700  5 -- op tracker -- , seq: 747714, time: 2014-06-12 14:46:17.311187, event: started, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.311195 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] sub_op_modify_reply: tid 236239 op  ack_type 4 from 29
2014-06-12 14:46:17.311215 7f9d61bd4700  5 -- op tracker -- , seq: 747712, time: 2014-06-12 14:46:17.311209, event: sub_op_commit_rec, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.311222 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2795 crt=0'0 lcod 173'2794 mlcod 173'2794 active+clean] repop_all_committed: repop tid 236239 all committed 
2014-06-12 14:46:17.311233 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2794 active+clean] eval_repop repgather(0x2562d80 173'2796 rep_tid=236239 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.311248 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2794 active+clean] log_op_stats osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008918 lat 0.009041
2014-06-12 14:46:17.311265 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2794 active+clean] publish_stats_to_osd 173:3601
2014-06-12 14:46:17.311278 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2794 active+clean]  sending commit on repgather(0x2562d80 173'2796 rep_tid=236239 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x25911900
2014-06-12 14:46:17.311308 7f9d61bd4700  5 -- op tracker -- , seq: 747712, time: 2014-06-12 14:46:17.311303, event: commit_sent, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.311316 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2794 active+clean]  requeue_ops 
2014-06-12 14:46:17.311326 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean]  removing repgather(0x2562d80 173'2796 rep_tid=236239 committed?=1 applied?=1 op=osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.311339 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean]    q front is repgather(0x2562d80 173'2796 rep_tid=236239 committed?=1 applied?=1 op=osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.311353 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] remove_repop repgather(0x2562d80 173'2796 rep_tid=236239 committed?=1 applied?=1 op=osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.311366 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean]  requeue_ops 
2014-06-12 14:46:17.311388 7f9d61bd4700  5 -- op tracker -- , seq: 747712, time: 2014-06-12 14:46:17.311384, event: done, op: osd_op(client.4302.0:146065 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 2916352~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.311409 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2910300 finish
2014-06-12 14:46:17.311418 7f9d61bd4700  5 -- op tracker -- , seq: 747714, time: 2014-06-12 14:46:17.311414, event: done, op: osd_sub_op_reply(client.4302.0:146065 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.311427 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.311430 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.314630 7f9d4829e700  5 -- op tracker -- , seq: 747715, time: 2014-06-12 14:46:17.313603, event: header_read, op: osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.314641 7f9d4829e700  5 -- op tracker -- , seq: 747715, time: 2014-06-12 14:46:17.313606, event: throttled, op: osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.314660 7f9d4829e700  5 -- op tracker -- , seq: 747715, time: 2014-06-12 14:46:17.314604, event: all_read, op: osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.314666 7f9d4829e700  5 -- op tracker -- , seq: 747715, time: 0.000000, event: dispatched, op: osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.314671 7f9d4829e700 20 osd.2 173 should_share_map client.4302 172.16.96.220:0/1243795 173
2014-06-12 14:46:17.314701 7f9d4829e700 15 osd.2 173 enqueue_op 0x17c2bd00 prio 63 cost 516096 latency 0.001097 osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.314768 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.314777 7f9d673df700 10 osd.2 173 dequeue_op 0x17c2bd00 prio 63 cost 516096 latency 0.001174 osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean]
2014-06-12 14:46:17.314804 7f9d673df700  5 -- op tracker -- , seq: 747715, time: 2014-06-12 14:46:17.314800, event: reached_pg, op: osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.314815 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.314826 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] handle_message: 0x17c2bd00
2014-06-12 14:46:17.314832 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] do_op osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.314850 7f9d673df700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_'
2014-06-12 14:46:17.315389 7f9d673df700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_' = 0
2014-06-12 14:46:17.315422 7f9d673df700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset'
2014-06-12 14:46:17.315490 7f9d673df700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset' = 0
2014-06-12 14:46:17.315499 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] populate_obc_watchers 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.315508 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.315513 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] get_object_context: creating obc from disk: 0x25cec000
2014-06-12 14:46:17.315519 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] get_object_context: 0x25cec000 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 rwstate(none n=0 w=0) oi: 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2796 client.4302.0:146065 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2796) ssc: 0x9344ee0 snapset: 0=[]:[]+head
2014-06-12 14:46:17.315528 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] find_object_context 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 @head oi=25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2796 client.4302.0:146065 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2796)
2014-06-12 14:46:17.315545 7f9d673df700  5 -- op tracker -- , seq: 747715, time: 2014-06-12 14:46:17.315541, event: started, op: osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.315550 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] execute_ctx 0x29ede400
2014-06-12 14:46:17.315560 7f9d673df700  5 -- op tracker -- , seq: 747715, time: 2014-06-12 14:46:17.315557, event: started, op: osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.315564 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] do_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] ov 173'2796 av 173'2797 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.315572 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] do_osd_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096]
2014-06-12 14:46:17.315578 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.315586 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] do_osd_op  write 3432448~516096
2014-06-12 14:46:17.315597 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] do_osd_op_effects on session 0x5c8d200
2014-06-12 14:46:17.315604 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 snapset=0x9344f18  snapc=0=[]
2014-06-12 14:46:17.315610 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.315616 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] finish_ctx 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 0x29ede400 op modify  
2014-06-12 14:46:17.315627 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean]  set mtime to 2014-06-12 08:45:19.693582
2014-06-12 14:46:17.315647 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean]  final snapset 0=[]:[]+head in 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.315659 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.315666 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] new_repop rep_tid 236240 on osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.315680 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] issue_repop rep_tid 236240 o 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.315696 7f9d673df700  5 -- op tracker -- , seq: 747715, time: 2014-06-12 14:46:17.315693, event: waiting for subops from 29, op: osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.315711 7f9d673df700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:17.315727 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2796 (0'0,173'2796] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] append_log log((0'0,173'2796], crt=0'0) [173'2797 (173'2796) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146066 2014-06-12 08:45:19.693582]
2014-06-12 14:46:17.315743 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2796 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] add_log_entry 173'2797 (173'2796) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146066 2014-06-12 08:45:19.693582
2014-06-12 14:46:17.315765 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2796 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.315786 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2797, trimmed: 
2014-06-12 14:46:17.315801 7f9d673df700  5 queue_transactions existing osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.315805 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.315806 7f9d673df700  5 queue_op 0x196faa20 seq 2612 osr(3.330 0x276d4f0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:17.315812 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2796 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] eval_repop repgather(0x2d813500 173'2797 rep_tid=236240 committed?=0 applied?=0 lock=0 op=osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.315817 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541ca20 (1 active)
2014-06-12 14:46:17.315824 7f9d673df700 10 osd.2 173 dequeue_op 0x17c2bd00 finish
2014-06-12 14:46:17.315826 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.315826 7f9d7e96a700  5 _do_op 0x196faa20 seq 2612 osr(3.330 0x276d4f0)/0x276d4f0 start
2014-06-12 14:46:17.315828 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.315830 7f9d7e96a700 10 _do_transaction on 0x2d8120c0
2014-06-12 14:46:17.315836 7f9d7e96a700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.315930 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.315950 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.315996 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.316018 7f9d7e96a700 15 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.316023 7f9d7e96a700 10 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1 r = 0
2014-06-12 14:46:17.316028 7f9d7e96a700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.316178 7f9d7e96a700 15 _write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 3432448~516096
2014-06-12 14:46:17.316284 7f9d7e96a700 10 lookup_strip_header done  cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.316414 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:17.317705 7f9d7e96a700 10 _generic_write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 3432448~516096 = 0
2014-06-12 14:46:17.317739 7f9d7e96a700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.317749 7f9d7e96a700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.317753 7f9d7e96a700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.317757 7f9d7e96a700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.317890 7f9d7e96a700 20 set_header setting 17148 cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:17.317930 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.317939 7f9d7e96a700 20 set_header setting 2048 cid meta oid 1fe2257a/pglog_3.330/0//-1 parent seq 0
2014-06-12 14:46:17.317945 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:17.319882 7f9d7e96a700 10 _do_op 0x196faa20 seq 2612 r = 0, finisher 0x7233e00 0x4dd9a40
2014-06-12 14:46:17.319896 7f9d7e96a700 10 _finish_op 0x196faa20 seq 2612 osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.319902 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541ca20 (0 active)
2014-06-12 14:46:17.319905 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.319902 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2796 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] op_commit: 236240
2014-06-12 14:46:17.319937 7f9d7715b700  5 -- op tracker -- , seq: 747715, time: 2014-06-12 14:46:17.319930, event: op_commit, op: osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.319951 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2796 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] op_applied: 236240
2014-06-12 14:46:17.319970 7f9d7795c700  5 -- op tracker -- , seq: 747715, time: 2014-06-12 14:46:17.319966, event: op_applied, op: osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.319976 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2796 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] op_applied on primary on version 173'2797
2014-06-12 14:46:17.324200 7f9d5501e700  5 -- op tracker -- , seq: 747716, time: 2014-06-12 14:46:17.324155, event: header_read, op: osd_sub_op_reply(client.4302.0:146066 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.324216 7f9d5501e700  5 -- op tracker -- , seq: 747716, time: 2014-06-12 14:46:17.324157, event: throttled, op: osd_sub_op_reply(client.4302.0:146066 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.324224 7f9d5501e700  5 -- op tracker -- , seq: 747716, time: 2014-06-12 14:46:17.324168, event: all_read, op: osd_sub_op_reply(client.4302.0:146066 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.324229 7f9d5501e700  5 -- op tracker -- , seq: 747716, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146066 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.324234 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146066 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.324246 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.324252 7f9d5501e700 15 osd.2 173 enqueue_op 0x1e3ea100 prio 196 cost 0 latency 0.000097 osd_sub_op_reply(client.4302.0:146066 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.324319 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.324329 7f9d663dd700 10 osd.2 173 dequeue_op 0x1e3ea100 prio 196 cost 0 latency 0.000174 osd_sub_op_reply(client.4302.0:146066 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 pg pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2796 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean]
2014-06-12 14:46:17.324352 7f9d663dd700  5 -- op tracker -- , seq: 747716, time: 2014-06-12 14:46:17.324350, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146066 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.324360 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2796 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] handle_message: 0x1e3ea100
2014-06-12 14:46:17.324370 7f9d663dd700  5 -- op tracker -- , seq: 747716, time: 2014-06-12 14:46:17.324368, event: started, op: osd_sub_op_reply(client.4302.0:146066 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.324376 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2796 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] sub_op_modify_reply: tid 236240 op  ack_type 4 from 29
2014-06-12 14:46:17.324392 7f9d663dd700  5 -- op tracker -- , seq: 747715, time: 2014-06-12 14:46:17.324388, event: sub_op_commit_rec, op: osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.324397 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2796 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] repop_all_applied: repop tid 236240 all applied 
2014-06-12 14:46:17.324403 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2796 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] eval_repop repgather(0x2d813500 173'2797 rep_tid=236240 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.324414 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2796 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean]  sending ack on repgather(0x2d813500 173'2797 rep_tid=236240 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0xb955b80
2014-06-12 14:46:17.324436 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2796 crt=0'0 lcod 173'2795 mlcod 173'2795 active+clean] repop_all_committed: repop tid 236240 all committed 
2014-06-12 14:46:17.324443 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2795 active+clean] eval_repop repgather(0x2d813500 173'2797 rep_tid=236240 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.324455 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2795 active+clean] log_op_stats osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.010832 lat 0.010851
2014-06-12 14:46:17.324466 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2795 active+clean] publish_stats_to_osd 173:3602
2014-06-12 14:46:17.324474 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2795 active+clean]  sending commit on repgather(0x2d813500 173'2797 rep_tid=236240 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x16cd2780
2014-06-12 14:46:17.324492 7f9d663dd700  5 -- op tracker -- , seq: 747715, time: 2014-06-12 14:46:17.324489, event: commit_sent, op: osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.324496 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2795 active+clean]  requeue_ops 
2014-06-12 14:46:17.324502 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean]  removing repgather(0x2d813500 173'2797 rep_tid=236240 committed?=1 applied?=1 op=osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.324511 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean]    q front is repgather(0x2d813500 173'2797 rep_tid=236240 committed?=1 applied?=1 op=osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.324519 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] remove_repop repgather(0x2d813500 173'2797 rep_tid=236240 committed?=1 applied?=1 op=osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.324527 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean]  requeue_ops 
2014-06-12 14:46:17.324543 7f9d663dd700  5 -- op tracker -- , seq: 747715, time: 2014-06-12 14:46:17.324540, event: done, op: osd_op(client.4302.0:146066 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3432448~516096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.324557 7f9d663dd700 10 osd.2 173 dequeue_op 0x1e3ea100 finish
2014-06-12 14:46:17.324562 7f9d663dd700  5 -- op tracker -- , seq: 747716, time: 2014-06-12 14:46:17.324560, event: done, op: osd_sub_op_reply(client.4302.0:146066 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.324568 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.324570 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.325669 7f9d4829e700  5 -- op tracker -- , seq: 747717, time: 2014-06-12 14:46:17.325626, event: header_read, op: osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.325679 7f9d4829e700  5 -- op tracker -- , seq: 747717, time: 2014-06-12 14:46:17.325628, event: throttled, op: osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.325685 7f9d4829e700  5 -- op tracker -- , seq: 747717, time: 2014-06-12 14:46:17.325644, event: all_read, op: osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.325690 7f9d4829e700  5 -- op tracker -- , seq: 747717, time: 0.000000, event: dispatched, op: osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.325695 7f9d4829e700 20 osd.2 173 should_share_map client.4302 172.16.96.220:0/1243795 173
2014-06-12 14:46:17.325715 7f9d4829e700 15 osd.2 173 enqueue_op 0x1702e200 prio 63 cost 4096 latency 0.000089 osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.325776 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.325785 7f9d67be0700 10 osd.2 173 dequeue_op 0x1702e200 prio 63 cost 4096 latency 0.000159 osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean]
2014-06-12 14:46:17.325803 7f9d67be0700  5 -- op tracker -- , seq: 747717, time: 2014-06-12 14:46:17.325800, event: reached_pg, op: osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.325812 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.325821 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] handle_message: 0x1702e200
2014-06-12 14:46:17.325828 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] do_op osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.325844 7f9d67be0700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_'
2014-06-12 14:46:17.325971 7f9d67be0700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_' = 0
2014-06-12 14:46:17.325984 7f9d67be0700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset'
2014-06-12 14:46:17.326039 7f9d67be0700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset' = 0
2014-06-12 14:46:17.326046 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] populate_obc_watchers 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.326054 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.326059 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] get_object_context: creating obc from disk: 0x172ce2c0
2014-06-12 14:46:17.326064 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] get_object_context: 0x172ce2c0 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 rwstate(none n=0 w=0) oi: 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2797 client.4302.0:146066 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2797) ssc: 0x81ba8c0 snapset: 0=[]:[]+head
2014-06-12 14:46:17.326073 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] find_object_context 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 @head oi=25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2797 client.4302.0:146066 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2797)
2014-06-12 14:46:17.326089 7f9d67be0700  5 -- op tracker -- , seq: 747717, time: 2014-06-12 14:46:17.326086, event: started, op: osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.326099 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] execute_ctx 0x1fc53e00
2014-06-12 14:46:17.326109 7f9d67be0700  5 -- op tracker -- , seq: 747717, time: 2014-06-12 14:46:17.326106, event: started, op: osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.326113 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] do_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] ov 173'2797 av 173'2798 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.326121 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] do_osd_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096]
2014-06-12 14:46:17.326127 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.326135 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] do_osd_op  write 3948544~4096
2014-06-12 14:46:17.326144 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] do_osd_op_effects on session 0x5c8d200
2014-06-12 14:46:17.326152 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 snapset=0x81ba8f8  snapc=0=[]
2014-06-12 14:46:17.326160 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.326165 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] finish_ctx 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 0x1fc53e00 op modify  
2014-06-12 14:46:17.326172 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean]  set mtime to 2014-06-12 08:45:19.706538
2014-06-12 14:46:17.326184 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean]  final snapset 0=[]:[]+head in 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.326195 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.326202 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] new_repop rep_tid 236241 on osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.326212 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] issue_repop rep_tid 236241 o 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.326227 7f9d67be0700  5 -- op tracker -- , seq: 747717, time: 2014-06-12 14:46:17.326224, event: waiting for subops from 29, op: osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.326240 7f9d67be0700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:17.326254 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2797 (0'0,173'2797] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] append_log log((0'0,173'2797], crt=0'0) [173'2798 (173'2797) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146067 2014-06-12 08:45:19.706538]
2014-06-12 14:46:17.326278 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2797 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] add_log_entry 173'2798 (173'2797) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146067 2014-06-12 08:45:19.706538
2014-06-12 14:46:17.326292 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2797 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.326313 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2798, trimmed: 
2014-06-12 14:46:17.326328 7f9d67be0700  5 queue_transactions existing osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.326332 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.326333 7f9d67be0700  5 queue_op 0x56dc600 seq 2613 osr(3.330 0x276d4f0) 6389 bytes   (queue has 1 ops and 6389 bytes)
2014-06-12 14:46:17.326349 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541ca20 (1 active)
2014-06-12 14:46:17.326355 7f9d7815d700  5 _do_op 0x56dc600 seq 2613 osr(3.330 0x276d4f0)/0x276d4f0 start
2014-06-12 14:46:17.326345 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2797 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] eval_repop repgather(0x2a32cfc0 173'2798 rep_tid=236241 committed?=0 applied?=0 lock=0 op=osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.326358 7f9d7815d700 10 _do_transaction on 0x2a32c540
2014-06-12 14:46:17.326361 7f9d67be0700 10 osd.2 173 dequeue_op 0x1702e200 finish
2014-06-12 14:46:17.326364 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.326364 7f9d7815d700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.326366 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.326482 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.326498 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.326532 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.326549 7f9d7815d700 15 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.326554 7f9d7815d700 10 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1 r = 0
2014-06-12 14:46:17.326558 7f9d7815d700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.326569 7f9d7815d700 15 _write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 3948544~4096
2014-06-12 14:46:17.326645 7f9d7815d700 10 lookup_strip_header done  cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.326742 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:17.326756 7f9d7815d700 10 _generic_write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 3948544~4096 = 0
2014-06-12 14:46:17.326763 7f9d7815d700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.326769 7f9d7815d700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.326773 7f9d7815d700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.326777 7f9d7815d700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.326846 7f9d7815d700 20 set_header setting 17148 cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:17.326859 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.326867 7f9d7815d700 20 set_header setting 2048 cid meta oid 1fe2257a/pglog_3.330/0//-1 parent seq 0
2014-06-12 14:46:17.326873 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:17.326927 7f9d7815d700 10 _do_op 0x56dc600 seq 2613 r = 0, finisher 0x54a4160 0x328d1d40
2014-06-12 14:46:17.326933 7f9d7815d700 10 _finish_op 0x56dc600 seq 2613 osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.326938 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541ca20 (0 active)
2014-06-12 14:46:17.326941 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.326936 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2797 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] op_commit: 236241
2014-06-12 14:46:17.326963 7f9d7715b700  5 -- op tracker -- , seq: 747717, time: 2014-06-12 14:46:17.326950, event: op_commit, op: osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.326971 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2797 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] op_applied: 236241
2014-06-12 14:46:17.327001 7f9d7795c700  5 -- op tracker -- , seq: 747717, time: 2014-06-12 14:46:17.326996, event: op_applied, op: osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.327010 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2797 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] op_applied on primary on version 173'2798
2014-06-12 14:46:17.327617 7f9d5501e700  5 -- op tracker -- , seq: 747718, time: 2014-06-12 14:46:17.327577, event: header_read, op: osd_sub_op_reply(client.4302.0:146067 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.327627 7f9d5501e700  5 -- op tracker -- , seq: 747718, time: 2014-06-12 14:46:17.327578, event: throttled, op: osd_sub_op_reply(client.4302.0:146067 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.327634 7f9d5501e700  5 -- op tracker -- , seq: 747718, time: 2014-06-12 14:46:17.327586, event: all_read, op: osd_sub_op_reply(client.4302.0:146067 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.327648 7f9d5501e700  5 -- op tracker -- , seq: 747718, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146067 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.327652 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146067 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.327661 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.327666 7f9d5501e700 15 osd.2 173 enqueue_op 0x1e3ea600 prio 196 cost 0 latency 0.000089 osd_sub_op_reply(client.4302.0:146067 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.327713 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.327723 7f9d693e3700 10 osd.2 173 dequeue_op 0x1e3ea600 prio 196 cost 0 latency 0.000146 osd_sub_op_reply(client.4302.0:146067 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 pg pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2797 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean]
2014-06-12 14:46:17.327741 7f9d693e3700  5 -- op tracker -- , seq: 747718, time: 2014-06-12 14:46:17.327738, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146067 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.327750 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2797 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] handle_message: 0x1e3ea600
2014-06-12 14:46:17.327763 7f9d693e3700  5 -- op tracker -- , seq: 747718, time: 2014-06-12 14:46:17.327760, event: started, op: osd_sub_op_reply(client.4302.0:146067 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.327767 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2797 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] sub_op_modify_reply: tid 236241 op  ack_type 4 from 29
2014-06-12 14:46:17.327780 7f9d693e3700  5 -- op tracker -- , seq: 747717, time: 2014-06-12 14:46:17.327775, event: sub_op_commit_rec, op: osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.327786 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2797 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] repop_all_applied: repop tid 236241 all applied 
2014-06-12 14:46:17.327792 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2797 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] eval_repop repgather(0x2a32cfc0 173'2798 rep_tid=236241 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.327803 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2797 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean]  sending ack on repgather(0x2a32cfc0 173'2798 rep_tid=236241 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x355c1400
2014-06-12 14:46:17.327824 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2797 crt=0'0 lcod 173'2796 mlcod 173'2796 active+clean] repop_all_committed: repop tid 236241 all committed 
2014-06-12 14:46:17.327831 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2796 active+clean] eval_repop repgather(0x2a32cfc0 173'2798 rep_tid=236241 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.327841 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2796 active+clean] log_op_stats osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 inb 4375 outb 0 rlat 0.002197 lat 0.002215
2014-06-12 14:46:17.327851 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2796 active+clean] publish_stats_to_osd 173:3603
2014-06-12 14:46:17.327858 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2796 active+clean]  sending commit on repgather(0x2a32cfc0 173'2798 rep_tid=236241 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x19e0a280
2014-06-12 14:46:17.327879 7f9d693e3700  5 -- op tracker -- , seq: 747717, time: 2014-06-12 14:46:17.327875, event: commit_sent, op: osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.327884 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2796 active+clean]  requeue_ops 
2014-06-12 14:46:17.327890 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean]  removing repgather(0x2a32cfc0 173'2798 rep_tid=236241 committed?=1 applied?=1 op=osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.327899 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean]    q front is repgather(0x2a32cfc0 173'2798 rep_tid=236241 committed?=1 applied?=1 op=osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.327907 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] remove_repop repgather(0x2a32cfc0 173'2798 rep_tid=236241 committed?=1 applied?=1 op=osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.327916 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean]  requeue_ops 
2014-06-12 14:46:17.327931 7f9d693e3700  5 -- op tracker -- , seq: 747717, time: 2014-06-12 14:46:17.327928, event: done, op: osd_op(client.4302.0:146067 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3948544~4096] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.327951 7f9d693e3700 10 osd.2 173 dequeue_op 0x1e3ea600 finish
2014-06-12 14:46:17.327957 7f9d693e3700  5 -- op tracker -- , seq: 747718, time: 2014-06-12 14:46:17.327955, event: done, op: osd_sub_op_reply(client.4302.0:146067 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.327964 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.327966 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.329982 7f9d4829e700  5 -- op tracker -- , seq: 747719, time: 2014-06-12 14:46:17.329527, event: header_read, op: osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.329992 7f9d4829e700  5 -- op tracker -- , seq: 747719, time: 2014-06-12 14:46:17.329529, event: throttled, op: osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.329997 7f9d4829e700  5 -- op tracker -- , seq: 747719, time: 2014-06-12 14:46:17.329956, event: all_read, op: osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.330003 7f9d4829e700  5 -- op tracker -- , seq: 747719, time: 0.000000, event: dispatched, op: osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.330008 7f9d4829e700 20 osd.2 173 should_share_map client.4302 172.16.96.220:0/1243795 173
2014-06-12 14:46:17.330025 7f9d4829e700 15 osd.2 173 enqueue_op 0x59dd100 prio 63 cost 241664 latency 0.000498 osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.330087 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.330096 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x59dd100 prio 63 cost 241664 latency 0.000569 osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean]
2014-06-12 14:46:17.330117 7f9d6b3e7700  5 -- op tracker -- , seq: 747719, time: 2014-06-12 14:46:17.330113, event: reached_pg, op: osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.330127 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.330139 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] handle_message: 0x59dd100
2014-06-12 14:46:17.330146 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] do_op osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.330162 7f9d6b3e7700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_'
2014-06-12 14:46:17.330272 7f9d6b3e7700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 '_' = 0
2014-06-12 14:46:17.330284 7f9d6b3e7700 15 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset'
2014-06-12 14:46:17.330340 7f9d6b3e7700 10 getattr 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 'snapset' = 0
2014-06-12 14:46:17.330347 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] populate_obc_watchers 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.330356 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.330361 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] get_object_context: creating obc from disk: 0x2b9fb080
2014-06-12 14:46:17.330367 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] get_object_context: 0x2b9fb080 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 rwstate(none n=0 w=0) oi: 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2798 client.4302.0:146067 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2798) ssc: 0x24d4a8c0 snapset: 0=[]:[]+head
2014-06-12 14:46:17.330380 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] find_object_context 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 @head oi=25101330/rbd_data.10a56b8b4567.0000000000001032/head//3(173'2798 client.4302.0:146067 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2798)
2014-06-12 14:46:17.330397 7f9d6b3e7700  5 -- op tracker -- , seq: 747719, time: 2014-06-12 14:46:17.330393, event: started, op: osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.330403 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] execute_ctx 0x2963d000
2014-06-12 14:46:17.330414 7f9d6b3e7700  5 -- op tracker -- , seq: 747719, time: 2014-06-12 14:46:17.330411, event: started, op: osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.330419 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] do_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] ov 173'2798 av 173'2799 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.330427 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] do_osd_op 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664]
2014-06-12 14:46:17.330438 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.330447 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] do_osd_op  write 3952640~241664
2014-06-12 14:46:17.330457 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] do_osd_op_effects on session 0x5c8d200
2014-06-12 14:46:17.330464 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 snapset=0x24d4a8f8  snapc=0=[]
2014-06-12 14:46:17.330471 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] make_writeable 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.330478 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] finish_ctx 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 0x2963d000 op modify  
2014-06-12 14:46:17.330485 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean]  set mtime to 2014-06-12 08:45:19.709988
2014-06-12 14:46:17.330498 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean]  final snapset 0=[]:[]+head in 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.330509 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.330517 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] new_repop rep_tid 236242 on osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.330527 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] issue_repop rep_tid 236242 o 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.330541 7f9d6b3e7700  5 -- op tracker -- , seq: 747719, time: 2014-06-12 14:46:17.330538, event: waiting for subops from 29, op: osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.330558 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:17.330572 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2798 (0'0,173'2798] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] append_log log((0'0,173'2798], crt=0'0) [173'2799 (173'2798) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146068 2014-06-12 08:45:19.709988]
2014-06-12 14:46:17.330595 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2798 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] add_log_entry 173'2799 (173'2798) modify   25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 by client.4302.0:146068 2014-06-12 08:45:19.709988
2014-06-12 14:46:17.330609 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2798 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.330631 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2799, trimmed: 
2014-06-12 14:46:17.330645 7f9d6b3e7700  5 queue_transactions existing osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.330648 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.330649 7f9d6b3e7700  5 queue_op 0x584a900 seq 2614 osr(3.330 0x276d4f0) 243957 bytes   (queue has 1 ops and 243957 bytes)
2014-06-12 14:46:17.330654 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2798 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] eval_repop repgather(0x5328300 173'2799 rep_tid=236242 committed?=0 applied?=0 lock=0 op=osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.330660 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541ca20 (1 active)
2014-06-12 14:46:17.330666 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x59dd100 finish
2014-06-12 14:46:17.330668 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.330670 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.330669 7f9d82171700  5 _do_op 0x584a900 seq 2614 osr(3.330 0x276d4f0)/0x276d4f0 start
2014-06-12 14:46:17.330673 7f9d82171700 10 _do_transaction on 0x5329200
2014-06-12 14:46:17.330678 7f9d82171700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.330770 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.330792 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.330837 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.330854 7f9d82171700 15 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.330858 7f9d82171700 10 _omap_rmkeys meta/1fe2257a/pglog_3.330/0//-1 r = 0
2014-06-12 14:46:17.330862 7f9d82171700 15 _omap_setkeys meta/1fe2257a/pglog_3.330/0//-1
2014-06-12 14:46:17.330873 7f9d82171700 15 _write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 3952640~241664
2014-06-12 14:46:17.331140 7f9d82171700 10 lookup_strip_header done  cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.331285 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:17.331851 7f9d82171700 10 _generic_write 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 3952640~241664 = 0
2014-06-12 14:46:17.331868 7f9d82171700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.331877 7f9d82171700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.331882 7f9d82171700 15 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3
2014-06-12 14:46:17.331887 7f9d82171700 10 _setattrs 3.330_head/25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:17.331983 7f9d82171700 20 set_header setting 17148 cid 3.330_head oid 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:17.332049 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.332058 7f9d82171700 20 set_header setting 2048 cid meta oid 1fe2257a/pglog_3.330/0//-1 parent seq 0
2014-06-12 14:46:17.332066 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:17.332781 7f9d82171700 10 _do_op 0x584a900 seq 2614 r = 0, finisher 0x6b06c00 0x38996f80
2014-06-12 14:46:17.332789 7f9d82171700 10 _finish_op 0x584a900 seq 2614 osr(3.330 0x276d4f0)/0x276d4f0
2014-06-12 14:46:17.332793 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541ca20 (0 active)
2014-06-12 14:46:17.332796 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.332801 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2798 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] op_applied: 236242
2014-06-12 14:46:17.332822 7f9d7795c700  5 -- op tracker -- , seq: 747719, time: 2014-06-12 14:46:17.332817, event: op_applied, op: osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.332831 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2798 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] op_applied on primary on version 173'2799
2014-06-12 14:46:17.332865 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2798 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] op_commit: 236242
2014-06-12 14:46:17.332888 7f9d7715b700  5 -- op tracker -- , seq: 747719, time: 2014-06-12 14:46:17.332883, event: op_commit, op: osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.335068 7f9d5501e700  5 -- op tracker -- , seq: 747720, time: 2014-06-12 14:46:17.335012, event: header_read, op: osd_sub_op_reply(client.4302.0:146068 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.335078 7f9d5501e700  5 -- op tracker -- , seq: 747720, time: 2014-06-12 14:46:17.335021, event: throttled, op: osd_sub_op_reply(client.4302.0:146068 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.335101 7f9d5501e700  5 -- op tracker -- , seq: 747720, time: 2014-06-12 14:46:17.335034, event: all_read, op: osd_sub_op_reply(client.4302.0:146068 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.335106 7f9d5501e700  5 -- op tracker -- , seq: 747720, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146068 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.335110 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146068 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.335114 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:17.335120 7f9d5501e700 15 osd.2 173 enqueue_op 0x3b42bb00 prio 196 cost 0 latency 0.000108 osd_sub_op_reply(client.4302.0:146068 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.335186 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.335197 7f9d62bd6700 10 osd.2 173 dequeue_op 0x3b42bb00 prio 196 cost 0 latency 0.000185 osd_sub_op_reply(client.4302.0:146068 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0) v2 pg pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2798 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean]
2014-06-12 14:46:17.335213 7f9d62bd6700  5 -- op tracker -- , seq: 747720, time: 2014-06-12 14:46:17.335211, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146068 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.335221 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2798 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] handle_message: 0x3b42bb00
2014-06-12 14:46:17.335231 7f9d62bd6700  5 -- op tracker -- , seq: 747720, time: 2014-06-12 14:46:17.335229, event: started, op: osd_sub_op_reply(client.4302.0:146068 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.335235 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2798 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] sub_op_modify_reply: tid 236242 op  ack_type 4 from 29
2014-06-12 14:46:17.335248 7f9d62bd6700  5 -- op tracker -- , seq: 747719, time: 2014-06-12 14:46:17.335243, event: sub_op_commit_rec, op: osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.335253 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2798 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] repop_all_applied: repop tid 236242 all applied 
2014-06-12 14:46:17.335262 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2798 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] eval_repop repgather(0x5328300 173'2799 rep_tid=236242 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.335284 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2798 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean]  sending ack on repgather(0x5328300 173'2799 rep_tid=236242 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x28aac000
2014-06-12 14:46:17.335308 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2798 crt=0'0 lcod 173'2797 mlcod 173'2797 active+clean] repop_all_committed: repop tid 236242 all committed 
2014-06-12 14:46:17.335315 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2797 active+clean] eval_repop repgather(0x5328300 173'2799 rep_tid=236242 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.335327 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2797 active+clean] log_op_stats osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4 inb 241943 outb 0 rlat 0.005780 lat 0.005799
2014-06-12 14:46:17.335338 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2797 active+clean] publish_stats_to_osd 173:3604
2014-06-12 14:46:17.335345 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2797 active+clean]  sending commit on repgather(0x5328300 173'2799 rep_tid=236242 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4) 0x5ffac80
2014-06-12 14:46:17.335363 7f9d62bd6700  5 -- op tracker -- , seq: 747719, time: 2014-06-12 14:46:17.335360, event: commit_sent, op: osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.335368 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2797 active+clean]  requeue_ops 
2014-06-12 14:46:17.335374 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2798 active+clean]  removing repgather(0x5328300 173'2799 rep_tid=236242 committed?=1 applied?=1 op=osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.335382 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2798 active+clean]    q front is repgather(0x5328300 173'2799 rep_tid=236242 committed?=1 applied?=1 op=osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.335390 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2798 active+clean] remove_repop repgather(0x5328300 173'2799 rep_tid=236242 committed?=1 applied?=1 op=osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.335398 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2798 active+clean]  requeue_ops 
2014-06-12 14:46:17.335414 7f9d62bd6700  5 -- op tracker -- , seq: 747719, time: 2014-06-12 14:46:17.335411, event: done, op: osd_op(client.4302.0:146068 rbd_data.10a56b8b4567.0000000000001032 [set-alloc-hint object_size 4194304 write_size 4194304,write 3952640~241664] 3.25101330 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.335485 7f9d62bd6700 10 osd.2 173 dequeue_op 0x3b42bb00 finish
2014-06-12 14:46:17.335491 7f9d62bd6700  5 -- op tracker -- , seq: 747720, time: 2014-06-12 14:46:17.335489, event: done, op: osd_sub_op_reply(client.4302.0:146068 3.330 25101330/rbd_data.10a56b8b4567.0000000000001032/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.335498 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.335510 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.393996 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:17.394103 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:17.468837 7f9d56432700  5 -- op tracker -- , seq: 747721, time: 2014-06-12 14:46:17.468095, event: header_read, op: osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.468858 7f9d56432700  5 -- op tracker -- , seq: 747721, time: 2014-06-12 14:46:17.468100, event: throttled, op: osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.468870 7f9d56432700  5 -- op tracker -- , seq: 747721, time: 2014-06-12 14:46:17.468772, event: all_read, op: osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.468880 7f9d56432700  5 -- op tracker -- , seq: 747721, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.468888 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:17.468901 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:17.468917 7f9d56432700 15 osd.2 173 enqueue_op 0x5abe000 prio 127 cost 299811 latency 0.000821 osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.468970 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.468989 7f9d683e1700 10 osd.2 173 dequeue_op 0x5abe000 prio 127 cost 299811 latency 0.000894 osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2b( v 173'2479 (0'0,173'2479] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active]
2014-06-12 14:46:17.469038 7f9d683e1700  5 -- op tracker -- , seq: 747721, time: 2014-06-12 14:46:17.469029, event: reached_pg, op: osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.469061 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2479 (0'0,173'2479] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active] handle_message: 0x5abe000
2014-06-12 14:46:17.469099 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2479 (0'0,173'2479] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active] sub_op_modify trans 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 v 173'2480 (transaction) 174
2014-06-12 14:46:17.469139 7f9d683e1700  5 -- op tracker -- , seq: 747721, time: 2014-06-12 14:46:17.469130, event: started, op: osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.469170 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2479 (0'0,173'2479] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active] append_log log((0'0,173'2479], crt=0'0) [173'2480 (173'2479) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144994 2014-06-12 08:44:58.266742]
2014-06-12 14:46:17.469219 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2480 (0'0,173'2480] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active] add_log_entry 173'2480 (173'2479) modify   74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 by client.4314.0:144994 2014-06-12 08:44:58.266742
2014-06-12 14:46:17.469290 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2480 (0'0,173'2480] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active] append_log  adding 1 keys
2014-06-12 14:46:17.469363 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2480, trimmed: 
2014-06-12 14:46:17.469413 7f9d683e1700  5 -- op tracker -- , seq: 747721, time: 2014-06-12 14:46:17.469402, event: started, op: osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.469433 7f9d683e1700  5 queue_transactions existing osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:17.469440 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.469443 7f9d683e1700  5 queue_op 0x7748fc0 seq 2293 osr(3.2b 0x276cd20) 301292 bytes   (queue has 1 ops and 301292 bytes)
2014-06-12 14:46:17.469455 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.2b( v 173'2480 (0'0,173'2480] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active] do_sub_op osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.469484 7f9d683e1700 10 osd.2 173 dequeue_op 0x5abe000 finish
2014-06-12 14:46:17.469492 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.469497 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.469490 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5107e60 (1 active)
2014-06-12 14:46:17.469507 7f9d7d167700  5 _do_op 0x7748fc0 seq 2293 osr(3.2b 0x276cd20)/0x276cd20 start
2014-06-12 14:46:17.469514 7f9d7d167700 10 _do_transaction on 0x25e120f0
2014-06-12 14:46:17.469524 7f9d7d167700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:17.469854 7f9d7d167700 10 lookup_strip_header done  cid meta oid 51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:17.469894 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.469969 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.470001 7f9d7d167700 15 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:17.470009 7f9d7d167700 10 _omap_rmkeys meta/51682f4c/pglog_3.2b/0//-1 r = 0
2014-06-12 14:46:17.470017 7f9d7d167700 15 _omap_setkeys meta/51682f4c/pglog_3.2b/0//-1
2014-06-12 14:46:17.470038 7f9d7d167700 15 _write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 3895296~299008
2014-06-12 14:46:17.470383 7f9d7d167700 10 lookup_strip_header done  cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:17.470530 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:17.471773 7f9d7d167700 10 _generic_write 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 3895296~299008 = 0
2014-06-12 14:46:17.471817 7f9d7d167700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:17.471834 7f9d7d167700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:17.471842 7f9d7d167700 15 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3
2014-06-12 14:46:17.471852 7f9d7d167700 10 _setattrs 3.2b_head/74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:17.472072 7f9d7d167700 20 set_header setting 17866 cid 3.2b_head oid 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 parent seq 0
2014-06-12 14:46:17.472136 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.472148 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:17.472156 7f9d7d167700 20 set_header setting 2202 cid meta oid 51682f4c/pglog_3.2b/0//-1 parent seq 0
2014-06-12 14:46:17.472166 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:46:17.472168 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:17.472169 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:46:17.472173 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:46:17.472183 7f9d84175700 10 osd.2 173 send_pg_stats - 2 pgs updated
2014-06-12 14:46:17.472234 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:17.472236 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:17.472237 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:17.472238 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 18; oldest is 38.223319 seconds old
2014-06-12 14:46:17.472249 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:17.474619 7f9d7d167700 10 _do_op 0x7748fc0 seq 2293 r = 0, finisher 0x89f49a0 0
2014-06-12 14:46:17.474641 7f9d7d167700 10 _finish_op 0x7748fc0 seq 2293 osr(3.2b 0x276cd20)/0x276cd20
2014-06-12 14:46:17.474649 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5107e60 (0 active)
2014-06-12 14:46:17.474657 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.474700 7f9d7795c700  5 -- op tracker -- , seq: 747721, time: 2014-06-12 14:46:17.474679, event: sub_op_applied, op: osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.474720 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2480 (0'0,173'2480] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active] sub_op_modify_applied on 0x25e12000 op osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.474773 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:17.474821 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2480 (0'0,173'2480] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active] op_applied on primary on version 173'2480
2014-06-12 14:46:17.474856 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2480 (0'0,173'2480] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active] op_applied on replica on version 173'2480
2014-06-12 14:46:17.474897 7f9d56432700  5 -- op tracker -- , seq: 747722, time: 2014-06-12 14:46:17.474822, event: header_read, op: osd_sub_op_reply(client.4309.0:141528 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.474920 7f9d56432700  5 -- op tracker -- , seq: 747722, time: 2014-06-12 14:46:17.474826, event: throttled, op: osd_sub_op_reply(client.4309.0:141528 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.474969 7f9d56432700  5 -- op tracker -- , seq: 747722, time: 2014-06-12 14:46:17.474860, event: all_read, op: osd_sub_op_reply(client.4309.0:141528 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.474974 7f9d56432700  5 -- op tracker -- , seq: 747722, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141528 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.474995 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141528 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.475009 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:17.475019 7f9d56432700 15 osd.2 173 enqueue_op 0x5abfd00 prio 196 cost 0 latency 0.000197 osd_sub_op_reply(client.4309.0:141528 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.475008 7f9d7715b700  5 -- op tracker -- , seq: 747721, time: 2014-06-12 14:46:17.474979, event: commit_sent, op: osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.475039 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2b( v 173'2480 (0'0,173'2480] local-les=153 n=269 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2478 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:17.475061 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.475073 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:17.475086 7f9d56432700  5 -- op tracker -- , seq: 747723, time: 2014-06-12 14:46:17.475048, event: header_read, op: osd_sub_op_reply(client.4303.0:144625 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.475076 7f9d65bdc700 10 osd.2 173 dequeue_op 0x5abfd00 prio 196 cost 0 latency 0.000253 osd_sub_op_reply(client.4309.0:141528 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 pg pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2881 active+clean]
2014-06-12 14:46:17.475106 7f9d56432700  5 -- op tracker -- , seq: 747723, time: 2014-06-12 14:46:17.475049, event: throttled, op: osd_sub_op_reply(client.4303.0:144625 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.475120 7f9d65bdc700  5 -- op tracker -- , seq: 747722, time: 2014-06-12 14:46:17.475111, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141528 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.475158 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2881 active+clean] handle_message: 0x5abfd00
2014-06-12 14:46:17.475200 7f9d65bdc700  5 -- op tracker -- , seq: 747722, time: 2014-06-12 14:46:17.475186, event: started, op: osd_sub_op_reply(client.4309.0:141528 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.475214 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2881 active+clean] sub_op_modify_reply: tid 236072 op  ack_type 4 from 26
2014-06-12 14:46:17.475235 7f9d56432700  5 -- op tracker -- , seq: 747723, time: 2014-06-12 14:46:17.475061, event: all_read, op: osd_sub_op_reply(client.4303.0:144625 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.475266 7f9d56432700  5 -- op tracker -- , seq: 747723, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4303.0:144625 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.475273 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4303.0:144625 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.475281 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:17.475291 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3eb000 prio 196 cost 0 latency 0.000243 osd_sub_op_reply(client.4303.0:144625 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.475308 7f9d7715b700  5 -- op tracker -- , seq: 747721, time: 2014-06-12 14:46:17.475103, event: done, op: osd_sub_op(client.4314.0:144994 3.2b 74aa002b/rbd_data.10b16b8b4567.000000000000104c/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.475341 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:17.475354 7f9d64bda700 10 osd.2 173 dequeue_op 0x1e3eb000 prio 196 cost 0 latency 0.000305 osd_sub_op_reply(client.4303.0:144625 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3090 active+clean]
2014-06-12 14:46:17.475393 7f9d65bdc700  5 -- op tracker -- , seq: 747106, time: 2014-06-12 14:46:17.475242, event: sub_op_commit_rec, op: osd_op(client.4309.0:141528 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.475426 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2881 active+clean] repop_all_applied: repop tid 236072 all applied 
2014-06-12 14:46:17.475441 7f9d64bda700  5 -- op tracker -- , seq: 747723, time: 2014-06-12 14:46:17.475383, event: reached_pg, op: osd_sub_op_reply(client.4303.0:144625 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.475461 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3090 active+clean] handle_message: 0x1e3eb000
2014-06-12 14:46:17.475454 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2881 active+clean] eval_repop repgather(0x34d8ecc0 173'2883 rep_tid=236072 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141528 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.475482 7f9d64bda700  5 -- op tracker -- , seq: 747723, time: 2014-06-12 14:46:17.475480, event: started, op: osd_sub_op_reply(client.4303.0:144625 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.475488 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3090 active+clean] sub_op_modify_reply: tid 236064 op  ack_type 4 from 26
2014-06-12 14:46:17.475485 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2881 active+clean]  sending ack on repgather(0x34d8ecc0 173'2883 rep_tid=236072 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141528 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0x3b82a780
2014-06-12 14:46:17.475501 7f9d64bda700  5 -- op tracker -- , seq: 747079, time: 2014-06-12 14:46:17.475496, event: sub_op_commit_rec, op: osd_op(client.4303.0:144625 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.475513 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3090 active+clean] repop_all_applied: repop tid 236064 all applied 
2014-06-12 14:46:17.475514 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2882 crt=0'0 lcod 173'2881 mlcod 173'2881 active+clean] repop_all_committed: repop tid 236072 all committed 
2014-06-12 14:46:17.475520 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3090 active+clean] eval_repop repgather(0x3a097800 173'3092 rep_tid=236064 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144625 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.475530 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3090 active+clean]  sending ack on repgather(0x3a097800 173'3092 rep_tid=236064 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144625 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x28676780
2014-06-12 14:46:17.475525 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2881 active+clean] eval_repop repgather(0x34d8ecc0 173'2883 rep_tid=236072 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141528 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.475542 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2881 active+clean] log_op_stats osd_op(client.4309.0:141528 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 32.147687 lat 32.147715
2014-06-12 14:46:17.475550 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3091 crt=0'0 lcod 173'3090 mlcod 173'3090 active+clean] repop_all_committed: repop tid 236064 all committed 
2014-06-12 14:46:17.475560 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2881 active+clean] publish_stats_to_osd 173:3269
2014-06-12 14:46:17.475560 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3090 active+clean] eval_repop repgather(0x3a097800 173'3092 rep_tid=236064 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144625 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.475575 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3090 active+clean] log_op_stats osd_op(client.4303.0:144625 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 38.226630 lat 38.226655
2014-06-12 14:46:17.475574 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2881 active+clean]  sending commit on repgather(0x34d8ecc0 173'2883 rep_tid=236072 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141528 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0x1a58cf00
2014-06-12 14:46:17.475589 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3090 active+clean] publish_stats_to_osd 173:3558
2014-06-12 14:46:17.475596 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3090 active+clean]  sending commit on repgather(0x3a097800 173'3092 rep_tid=236064 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144625 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x28aad400
2014-06-12 14:46:17.475607 7f9d65bdc700  5 -- op tracker -- , seq: 747106, time: 2014-06-12 14:46:17.475601, event: commit_sent, op: osd_op(client.4309.0:141528 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.475614 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2881 active+clean]  requeue_ops 
2014-06-12 14:46:17.475618 7f9d64bda700  5 -- op tracker -- , seq: 747079, time: 2014-06-12 14:46:17.475613, event: commit_sent, op: osd_op(client.4303.0:144625 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.475628 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3090 active+clean]  requeue_ops 
2014-06-12 14:46:17.475625 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2882 active+clean]  removing repgather(0x34d8ecc0 173'2883 rep_tid=236072 committed?=1 applied?=1 op=osd_op(client.4309.0:141528 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.475639 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3091 active+clean]  removing repgather(0x3a097800 173'3092 rep_tid=236064 committed?=1 applied?=1 op=osd_op(client.4303.0:144625 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.475641 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2882 active+clean]    q front is repgather(0x34d8ecc0 173'2883 rep_tid=236072 committed?=1 applied?=1 op=osd_op(client.4309.0:141528 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.475649 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3091 active+clean]    q front is repgather(0x3a097800 173'3092 rep_tid=236064 committed?=1 applied?=1 op=osd_op(client.4303.0:144625 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.475659 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3091 active+clean] remove_repop repgather(0x3a097800 173'3092 rep_tid=236064 committed?=1 applied?=1 op=osd_op(client.4303.0:144625 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.475656 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2882 active+clean] remove_repop repgather(0x34d8ecc0 173'2883 rep_tid=236072 committed?=1 applied?=1 op=osd_op(client.4309.0:141528 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.475667 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3091 active+clean]  requeue_ops 
2014-06-12 14:46:17.475670 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2882 active+clean]  requeue_ops 
2014-06-12 14:46:17.475684 7f9d64bda700  5 -- op tracker -- , seq: 747079, time: 2014-06-12 14:46:17.475681, event: done, op: osd_op(client.4303.0:144625 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.475698 7f9d65bdc700  5 -- op tracker -- , seq: 747106, time: 2014-06-12 14:46:17.475685, event: done, op: osd_op(client.4309.0:141528 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~516096] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.475715 7f9d65bdc700 10 osd.2 173 dequeue_op 0x5abfd00 finish
2014-06-12 14:46:17.475723 7f9d65bdc700  5 -- op tracker -- , seq: 747722, time: 2014-06-12 14:46:17.475719, event: done, op: osd_sub_op_reply(client.4309.0:141528 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.475736 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:17.475739 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.475744 7f9d64bda700 10 osd.2 173 dequeue_op 0x1e3eb000 finish
2014-06-12 14:46:17.475753 7f9d64bda700  5 -- op tracker -- , seq: 747723, time: 2014-06-12 14:46:17.475750, event: done, op: osd_sub_op_reply(client.4303.0:144625 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.475764 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.475766 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.520202 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:17.520204 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:17.520205 7f9d74bfa700 20 osd.2 173 _dispatch 0x5fe8700 pg_stats_ack(2 pgs tid 4849) v1
2014-06-12 14:46:17.520207 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:46:17.520212 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:17.520214 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:17.631132 7f9d56432700  5 -- op tracker -- , seq: 747724, time: 2014-06-12 14:46:17.631077, event: header_read, op: osd_sub_op_reply(client.4303.0:144626 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.631144 7f9d56432700  5 -- op tracker -- , seq: 747724, time: 2014-06-12 14:46:17.631079, event: throttled, op: osd_sub_op_reply(client.4303.0:144626 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.631161 7f9d56432700  5 -- op tracker -- , seq: 747724, time: 2014-06-12 14:46:17.631094, event: all_read, op: osd_sub_op_reply(client.4303.0:144626 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.631168 7f9d56432700  5 -- op tracker -- , seq: 747724, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4303.0:144626 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.631173 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4303.0:144626 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.631186 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:17.631193 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3ebb00 prio 196 cost 0 latency 0.000116 osd_sub_op_reply(client.4303.0:144626 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.631226 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.631235 7f9d6abe6700 10 osd.2 173 dequeue_op 0x1e3ebb00 prio 196 cost 0 latency 0.000158 osd_sub_op_reply(client.4303.0:144626 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3091 active+clean]
2014-06-12 14:46:17.631257 7f9d6abe6700  5 -- op tracker -- , seq: 747724, time: 2014-06-12 14:46:17.631254, event: reached_pg, op: osd_sub_op_reply(client.4303.0:144626 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.631264 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3091 active+clean] handle_message: 0x1e3ebb00
2014-06-12 14:46:17.631280 7f9d6abe6700  5 -- op tracker -- , seq: 747724, time: 2014-06-12 14:46:17.631277, event: started, op: osd_sub_op_reply(client.4303.0:144626 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.631286 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3091 active+clean] sub_op_modify_reply: tid 236065 op  ack_type 4 from 26
2014-06-12 14:46:17.631304 7f9d6abe6700  5 -- op tracker -- , seq: 747083, time: 2014-06-12 14:46:17.631297, event: sub_op_commit_rec, op: osd_op(client.4303.0:144626 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.631310 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3091 active+clean] repop_all_applied: repop tid 236065 all applied 
2014-06-12 14:46:17.631320 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3091 active+clean] eval_repop repgather(0xc9b9680 173'3093 rep_tid=236065 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144626 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.631334 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3091 active+clean]  sending ack on repgather(0xc9b9680 173'3093 rep_tid=236065 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144626 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x2b43180
2014-06-12 14:46:17.631365 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3092 crt=0'0 lcod 173'3091 mlcod 173'3091 active+clean] repop_all_committed: repop tid 236065 all committed 
2014-06-12 14:46:17.631375 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3091 active+clean] eval_repop repgather(0xc9b9680 173'3093 rep_tid=236065 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144626 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.631400 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3091 active+clean] log_op_stats osd_op(client.4303.0:144626 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 38.303777 lat 38.303813
2014-06-12 14:46:17.631416 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3091 active+clean] publish_stats_to_osd 173:3559
2014-06-12 14:46:17.631428 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3091 active+clean]  sending commit on repgather(0xc9b9680 173'3093 rep_tid=236065 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144626 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x1b40ac80
2014-06-12 14:46:17.631460 7f9d6abe6700  5 -- op tracker -- , seq: 747083, time: 2014-06-12 14:46:17.631455, event: commit_sent, op: osd_op(client.4303.0:144626 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.631467 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3091 active+clean]  requeue_ops 
2014-06-12 14:46:17.631478 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3092 active+clean]  removing repgather(0xc9b9680 173'3093 rep_tid=236065 committed?=1 applied?=1 op=osd_op(client.4303.0:144626 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.631492 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3092 active+clean]    q front is repgather(0xc9b9680 173'3093 rep_tid=236065 committed?=1 applied?=1 op=osd_op(client.4303.0:144626 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.631512 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3092 active+clean] remove_repop repgather(0xc9b9680 173'3093 rep_tid=236065 committed?=1 applied?=1 op=osd_op(client.4303.0:144626 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.631525 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3092 active+clean]  requeue_ops 
2014-06-12 14:46:17.631542 7f9d6abe6700  5 -- op tracker -- , seq: 747083, time: 2014-06-12 14:46:17.631538, event: done, op: osd_op(client.4303.0:144626 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.631563 7f9d6abe6700 10 osd.2 173 dequeue_op 0x1e3ebb00 finish
2014-06-12 14:46:17.631569 7f9d6abe6700  5 -- op tracker -- , seq: 747724, time: 2014-06-12 14:46:17.631566, event: done, op: osd_sub_op_reply(client.4303.0:144626 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.631578 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.631581 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.794683 7f9d56432700  5 -- op tracker -- , seq: 747725, time: 2014-06-12 14:46:17.794627, event: header_read, op: osd_sub_op_reply(client.4309.0:141529 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.794693 7f9d56432700  5 -- op tracker -- , seq: 747725, time: 2014-06-12 14:46:17.794631, event: throttled, op: osd_sub_op_reply(client.4309.0:141529 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.794708 7f9d56432700  5 -- op tracker -- , seq: 747725, time: 2014-06-12 14:46:17.794643, event: all_read, op: osd_sub_op_reply(client.4309.0:141529 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.794713 7f9d56432700  5 -- op tracker -- , seq: 747725, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141529 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.794717 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141529 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.794721 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:17.794727 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3eb900 prio 196 cost 0 latency 0.000100 osd_sub_op_reply(client.4309.0:141529 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.794752 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.794760 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x1e3eb900 prio 196 cost 0 latency 0.000133 osd_sub_op_reply(client.4309.0:141529 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0) v2 pg pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2882 active+clean]
2014-06-12 14:46:17.794777 7f9d6a3e5700  5 -- op tracker -- , seq: 747725, time: 2014-06-12 14:46:17.794774, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141529 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.794783 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2882 active+clean] handle_message: 0x1e3eb900
2014-06-12 14:46:17.794801 7f9d6a3e5700  5 -- op tracker -- , seq: 747725, time: 2014-06-12 14:46:17.794799, event: started, op: osd_sub_op_reply(client.4309.0:141529 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.794805 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2882 active+clean] sub_op_modify_reply: tid 236073 op  ack_type 4 from 26
2014-06-12 14:46:17.794818 7f9d6a3e5700  5 -- op tracker -- , seq: 747119, time: 2014-06-12 14:46:17.794812, event: sub_op_commit_rec, op: osd_op(client.4309.0:141529 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3928064~266240] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.794824 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2882 active+clean] repop_all_applied: repop tid 236073 all applied 
2014-06-12 14:46:17.794831 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2882 active+clean] eval_repop repgather(0x6590900 173'2884 rep_tid=236073 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141529 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3928064~266240] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.794842 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2882 active+clean]  sending ack on repgather(0x6590900 173'2884 rep_tid=236073 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141529 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3928064~266240] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0x1990c000
2014-06-12 14:46:17.794864 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2883 crt=0'0 lcod 173'2882 mlcod 173'2882 active+clean] repop_all_committed: repop tid 236073 all committed 
2014-06-12 14:46:17.794872 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2882 active+clean] eval_repop repgather(0x6590900 173'2884 rep_tid=236073 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141529 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3928064~266240] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.794883 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2882 active+clean] log_op_stats osd_op(client.4309.0:141529 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3928064~266240] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4 inb 266519 outb 0 rlat 31.659167 lat 31.659186
2014-06-12 14:46:17.794894 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2882 active+clean] publish_stats_to_osd 173:3270
2014-06-12 14:46:17.794903 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2882 active+clean]  sending commit on repgather(0x6590900 173'2884 rep_tid=236073 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141529 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3928064~266240] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4) 0x2d3c0280
2014-06-12 14:46:17.794924 7f9d6a3e5700  5 -- op tracker -- , seq: 747119, time: 2014-06-12 14:46:17.794920, event: commit_sent, op: osd_op(client.4309.0:141529 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3928064~266240] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.794930 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2882 active+clean]  requeue_ops 
2014-06-12 14:46:17.794936 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2883 active+clean]  removing repgather(0x6590900 173'2884 rep_tid=236073 committed?=1 applied?=1 op=osd_op(client.4309.0:141529 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3928064~266240] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.794946 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2883 active+clean]    q front is repgather(0x6590900 173'2884 rep_tid=236073 committed?=1 applied?=1 op=osd_op(client.4309.0:141529 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3928064~266240] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.794955 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2883 active+clean] remove_repop repgather(0x6590900 173'2884 rep_tid=236073 committed?=1 applied?=1 op=osd_op(client.4309.0:141529 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3928064~266240] 3.f029e408 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.794964 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2883 active+clean]  requeue_ops 
2014-06-12 14:46:17.794981 7f9d6a3e5700  5 -- op tracker -- , seq: 747119, time: 2014-06-12 14:46:17.794978, event: done, op: osd_op(client.4309.0:141529 rbd_data.10ac6b8b4567.0000000000000f5b [set-alloc-hint object_size 4194304 write_size 4194304,write 3928064~266240] 3.f029e408 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.794997 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x1e3eb900 finish
2014-06-12 14:46:17.795003 7f9d6a3e5700  5 -- op tracker -- , seq: 747725, time: 2014-06-12 14:46:17.795000, event: done, op: osd_sub_op_reply(client.4309.0:141529 3.408 f029e408/rbd_data.10ac6b8b4567.0000000000000f5b/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.795009 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.795011 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.868052 7f9d494c9700  5 -- op tracker -- , seq: 747726, time: 2014-06-12 14:46:17.866126, event: header_read, op: osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.868064 7f9d494c9700  5 -- op tracker -- , seq: 747726, time: 2014-06-12 14:46:17.866130, event: throttled, op: osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.868072 7f9d494c9700  5 -- op tracker -- , seq: 747726, time: 2014-06-12 14:46:17.868008, event: all_read, op: osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.868080 7f9d494c9700  5 -- op tracker -- , seq: 747726, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.868087 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:46:17.868109 7f9d494c9700 15 osd.2 173 enqueue_op 0x20e87c00 prio 63 cost 516096 latency 0.001983 osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.868141 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.868150 7f9d66bde700 10 osd.2 173 dequeue_op 0x20e87c00 prio 63 cost 516096 latency 0.002024 osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean]
2014-06-12 14:46:17.868174 7f9d66bde700  5 -- op tracker -- , seq: 747726, time: 2014-06-12 14:46:17.868169, event: reached_pg, op: osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.868184 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.868197 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] handle_message: 0x20e87c00
2014-06-12 14:46:17.868209 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] do_op osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.868233 7f9d66bde700 15 getattr 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 '_'
2014-06-12 14:46:17.868549 7f9d66bde700 10 getattr 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 '_' = 0
2014-06-12 14:46:17.868561 7f9d66bde700 15 getattr 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 'snapset'
2014-06-12 14:46:17.868624 7f9d66bde700 10 getattr 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 'snapset' = 0
2014-06-12 14:46:17.868633 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] populate_obc_watchers 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.868642 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.868647 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] get_object_context: creating obc from disk: 0x212fd340
2014-06-12 14:46:17.868653 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] get_object_context: 0x212fd340 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 rwstate(none n=0 w=0) oi: 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2576 client.4290.0:143628 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2576) ssc: 0x72b8b60 snapset: 0=[]:[]+head
2014-06-12 14:46:17.868662 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] find_object_context 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 @head oi=82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2576 client.4290.0:143628 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2576)
2014-06-12 14:46:17.868680 7f9d66bde700  5 -- op tracker -- , seq: 747726, time: 2014-06-12 14:46:17.868676, event: started, op: osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.868686 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] execute_ctx 0x110c4400
2014-06-12 14:46:17.868696 7f9d66bde700  5 -- op tracker -- , seq: 747726, time: 2014-06-12 14:46:17.868693, event: started, op: osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.868700 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] do_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] ov 173'2576 av 173'2577 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.868711 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] do_osd_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096]
2014-06-12 14:46:17.868717 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.868725 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] do_osd_op  write 421888~516096
2014-06-12 14:46:17.868732 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:46:17.868738 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 snapset=0x72b8b98  snapc=0=[]
2014-06-12 14:46:17.868744 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.868750 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] finish_ctx 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 0x110c4400 op modify  
2014-06-12 14:46:17.868757 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean]  set mtime to 2014-06-12 08:45:20.245970
2014-06-12 14:46:17.868768 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean]  final snapset 0=[]:[]+head in 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.868784 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.868789 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] new_repop rep_tid 236243 on osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.868798 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] issue_repop rep_tid 236243 o 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.868812 7f9d66bde700  5 -- op tracker -- , seq: 747726, time: 2014-06-12 14:46:17.868809, event: waiting for subops from 27, op: osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.868824 7f9d66bde700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:17.868840 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2576 (0'0,173'2576] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] append_log log((0'0,173'2576], crt=0'0) [173'2577 (173'2576) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143629 2014-06-12 08:45:20.245970]
2014-06-12 14:46:17.868860 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] add_log_entry 173'2577 (173'2576) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143629 2014-06-12 08:45:20.245970
2014-06-12 14:46:17.868878 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.868916 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2577, trimmed: 
2014-06-12 14:46:17.868937 7f9d66bde700  5 queue_transactions existing osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.868940 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.868941 7f9d66bde700  5 queue_op 0x69588a0 seq 2379 osr(3.717 0x276de70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:17.868948 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] eval_repop repgather(0x5112a80 173'2577 rep_tid=236243 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.868964 7f9d66bde700 10 osd.2 173 dequeue_op 0x20e87c00 finish
2014-06-12 14:46:17.868968 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.868970 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.868979 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a3d40 (1 active)
2014-06-12 14:46:17.868984 7f9d8116f700  5 _do_op 0x69588a0 seq 2379 osr(3.717 0x276de70)/0x276de70 start
2014-06-12 14:46:17.868987 7f9d8116f700 10 _do_transaction on 0x5112840
2014-06-12 14:46:17.868992 7f9d8116f700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.869085 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.869102 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.869148 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.869297 7f9d8116f700 15 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.869305 7f9d8116f700 10 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1 r = 0
2014-06-12 14:46:17.869311 7f9d8116f700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.869328 7f9d8116f700 15 _write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 421888~516096
2014-06-12 14:46:17.869465 7f9d8116f700 10 lookup_strip_header done  cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.871604 7f9d56432700  5 -- op tracker -- , seq: 747727, time: 2014-06-12 14:46:17.870344, event: header_read, op: osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.871619 7f9d56432700  5 -- op tracker -- , seq: 747727, time: 2014-06-12 14:46:17.870346, event: throttled, op: osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.871627 7f9d56432700  5 -- op tracker -- , seq: 747727, time: 2014-06-12 14:46:17.871561, event: all_read, op: osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.871635 7f9d56432700  5 -- op tracker -- , seq: 747727, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.871641 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:17.871648 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:17.871656 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3eb300 prio 127 cost 459559 latency 0.001312 osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.872359 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:17.872896 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.872905 7f9d494c9700  5 -- op tracker -- , seq: 747728, time: 2014-06-12 14:46:17.870825, event: header_read, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.872914 7f9d494c9700  5 -- op tracker -- , seq: 747728, time: 2014-06-12 14:46:17.870829, event: throttled, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.872920 7f9d494c9700  5 -- op tracker -- , seq: 747728, time: 2014-06-12 14:46:17.872867, event: all_read, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.872905 7f9d69be4700 10 osd.2 173 dequeue_op 0x1e3eb300 prio 127 cost 459559 latency 0.002561 osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2992 (0'0,173'2992] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2991 active]
2014-06-12 14:46:17.872926 7f9d494c9700  5 -- op tracker -- , seq: 747728, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.872968 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:46:17.873007 7f9d494c9700 15 osd.2 173 enqueue_op 0x20e87200 prio 63 cost 516096 latency 0.002182 osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.873030 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:17.873039 7f9d653db700 10 osd.2 173 dequeue_op 0x20e87200 prio 63 cost 516096 latency 0.002214 osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean]
2014-06-12 14:46:17.873070 7f9d653db700  5 -- op tracker -- , seq: 747728, time: 2014-06-12 14:46:17.873064, event: reached_pg, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.873084 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.873101 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] handle_message: 0x20e87200
2014-06-12 14:46:17.873126 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] do_op osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.873157 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] get_object_context: found obc in cache: 0x212fd340
2014-06-12 14:46:17.873171 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] get_object_context: 0x212fd340 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 rwstate(write n=1 w=0) oi: 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2577 client.4290.0:143629 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2577) ssc: 0x72b8b60 snapset: 0=[]:[]+head
2014-06-12 14:46:17.873190 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] find_object_context 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 @head oi=82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2577 client.4290.0:143629 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2577)
2014-06-12 14:46:17.873222 7f9d653db700  5 -- op tracker -- , seq: 747728, time: 2014-06-12 14:46:17.873215, event: started, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.873231 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] execute_ctx 0x1fc52000
2014-06-12 14:46:17.873252 7f9d653db700  5 -- op tracker -- , seq: 747728, time: 2014-06-12 14:46:17.873247, event: started, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.873261 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] do_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] ov 173'2577 av 173'2578 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.873278 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] do_osd_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096]
2014-06-12 14:46:17.873292 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.873307 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] do_osd_op  write 937984~516096
2014-06-12 14:46:17.873328 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:46:17.873343 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 snapset=0x72b8b98  snapc=0=[]
2014-06-12 14:46:17.873359 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.873372 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] finish_ctx 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 0x1fc52000 op modify  
2014-06-12 14:46:17.873388 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean]  set mtime to 2014-06-12 08:45:20.250747
2014-06-12 14:46:17.873414 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean]  final snapset 0=[]:[]+head in 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.873437 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.873451 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] new_repop rep_tid 236244 on osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.873471 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] issue_repop rep_tid 236244 o 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.873506 7f9d653db700  5 -- op tracker -- , seq: 747728, time: 2014-06-12 14:46:17.873499, event: waiting for subops from 27, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.873543 7f9d653db700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:17.873569 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2577 (0'0,173'2577] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] append_log log((0'0,173'2577], crt=0'0) [173'2578 (173'2577) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143630 2014-06-12 08:45:20.250747]
2014-06-12 14:46:17.873595 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] add_log_entry 173'2578 (173'2577) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143630 2014-06-12 08:45:20.250747
2014-06-12 14:46:17.873642 7f9d69be4700  5 -- op tracker -- , seq: 747727, time: 2014-06-12 14:46:17.872932, event: reached_pg, op: osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.873659 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2992 (0'0,173'2992] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2991 active] handle_message: 0x1e3eb300
2014-06-12 14:46:17.873677 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2992 (0'0,173'2992] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2991 active] sub_op_modify trans 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 v 173'2993 (transaction) 174
2014-06-12 14:46:17.873699 7f9d69be4700  5 -- op tracker -- , seq: 747727, time: 2014-06-12 14:46:17.873695, event: started, op: osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.873724 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2992 (0'0,173'2992] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2991 active] append_log log((0'0,173'2992], crt=0'0) [173'2993 (173'728) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145014 2014-06-12 08:44:42.048740]
2014-06-12 14:46:17.873755 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2993 (0'0,173'2993] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2991 active] add_log_entry 173'2993 (173'728) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145014 2014-06-12 08:44:42.048740
2014-06-12 14:46:17.873782 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2993 (0'0,173'2993] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2991 active] append_log  adding 1 keys
2014-06-12 14:46:17.873825 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2993, trimmed: 
2014-06-12 14:46:17.873856 7f9d69be4700  5 -- op tracker -- , seq: 747727, time: 2014-06-12 14:46:17.873850, event: started, op: osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.873870 7f9d69be4700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:17.873876 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.873877 7f9d69be4700  5 queue_op 0x777ac00 seq 2795 osr(3.458 0x276d960) 461045 bytes   (queue has 2 ops and 979434 bytes)
2014-06-12 14:46:17.873885 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2993 (0'0,173'2993] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2991 active] do_sub_op osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.873900 7f9d69be4700 10 osd.2 173 dequeue_op 0x1e3eb300 finish
2014-06-12 14:46:17.873905 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:17.873910 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.873919 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (2 active)
2014-06-12 14:46:17.873928 7f9d79960700  5 _do_op 0x777ac00 seq 2795 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:17.873932 7f9d79960700 10 _do_transaction on 0x2b2ae470
2014-06-12 14:46:17.873938 7f9d79960700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:17.874122 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:17.874143 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.874190 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.874229 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2578, trimmed: 
2014-06-12 14:46:17.874262 7f9d653db700  5 queue_transactions existing osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.874269 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.874270 7f9d653db700  5 queue_op 0x6873560 seq 2380 osr(3.717 0x276de70) 518389 bytes   (queue has 3 ops and 1497823 bytes)
2014-06-12 14:46:17.874280 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] eval_repop repgather(0x18224540 173'2578 rep_tid=236244 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.874312 7f9d653db700 10 osd.2 173 dequeue_op 0x20e87200 finish
2014-06-12 14:46:17.874318 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.874322 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.874335 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a3d40 (3 active)
2014-06-12 14:46:17.879432 7f9d8116f700 10 _generic_write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 421888~516096 = 0
2014-06-12 14:46:17.879586 7f9d8116f700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.879609 7f9d8116f700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.879617 7f9d8116f700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.879625 7f9d8116f700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.879911 7f9d8116f700 20 set_header setting 15870 cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 parent seq 0
2014-06-12 14:46:17.880072 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.880093 7f9d8116f700 20 set_header setting 2128 cid meta oid 1fe0f8ca/pglog_3.717/0//-1 parent seq 0
2014-06-12 14:46:17.880104 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:17.883016 7f9d8116f700 10 _do_op 0x69588a0 seq 2379 r = 0, finisher 0x654c5c0 0x28ab8e80
2014-06-12 14:46:17.883038 7f9d8116f700 10 _finish_op 0x69588a0 seq 2379 osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.883051 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a3d40 (2 active)
2014-06-12 14:46:17.883060 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.883150 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.883179 7f9d79960700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:17.883186 7f9d79960700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:17.883186 7f9d8096e700  5 _do_op 0x6873560 seq 2380 osr(3.717 0x276de70)/0x276de70 start
2014-06-12 14:46:17.883192 7f9d8096e700 10 _do_transaction on 0x18224a80
2014-06-12 14:46:17.883193 7f9d79960700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:17.883199 7f9d8096e700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.883211 7f9d79960700 15 _write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 0~458752
2014-06-12 14:46:17.883295 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.883316 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.884278 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] op_applied: 236243
2014-06-12 14:46:17.884313 7f9d7795c700  5 -- op tracker -- , seq: 747726, time: 2014-06-12 14:46:17.884304, event: op_applied, op: osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.884325 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] op_applied on primary on version 173'2577
2014-06-12 14:46:17.884964 7f9d58c5a700  5 -- op tracker -- , seq: 747729, time: 2014-06-12 14:46:17.884922, event: header_read, op: osd_sub_op_reply(client.4290.0:143629 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.884990 7f9d58c5a700  5 -- op tracker -- , seq: 747729, time: 2014-06-12 14:46:17.884925, event: throttled, op: osd_sub_op_reply(client.4290.0:143629 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.884984 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] op_commit: 236243
2014-06-12 14:46:17.885000 7f9d58c5a700  5 -- op tracker -- , seq: 747729, time: 2014-06-12 14:46:17.884935, event: all_read, op: osd_sub_op_reply(client.4290.0:143629 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.885020 7f9d58c5a700  5 -- op tracker -- , seq: 747729, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143629 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.885025 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143629 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.885030 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:17.885046 7f9d58c5a700 15 osd.2 173 enqueue_op 0x68a7600 prio 196 cost 0 latency 0.000123 osd_sub_op_reply(client.4290.0:143629 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.885103 7f9d58c5a700  5 -- op tracker -- , seq: 747730, time: 2014-06-12 14:46:17.885072, event: header_read, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.885111 7f9d58c5a700  5 -- op tracker -- , seq: 747730, time: 2014-06-12 14:46:17.885073, event: throttled, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.885118 7f9d58c5a700  5 -- op tracker -- , seq: 747730, time: 2014-06-12 14:46:17.885079, event: all_read, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.885125 7f9d58c5a700  5 -- op tracker -- , seq: 747730, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.885128 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:17.885132 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:17.885137 7f9d58c5a700 15 osd.2 173 enqueue_op 0x68a6000 prio 196 cost 0 latency 0.000065 osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0) v2
2014-06-12 14:46:17.885174 7f9d58c5a700  5 -- op tracker -- , seq: 747731, time: 2014-06-12 14:46:17.885152, event: header_read, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.885182 7f9d58c5a700  5 -- op tracker -- , seq: 747731, time: 2014-06-12 14:46:17.885152, event: throttled, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.885189 7f9d58c5a700  5 -- op tracker -- , seq: 747731, time: 2014-06-12 14:46:17.885160, event: all_read, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.885198 7f9d58c5a700  5 -- op tracker -- , seq: 747731, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.885203 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.885208 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:17.885213 7f9d58c5a700 15 osd.2 173 enqueue_op 0x68a7000 prio 196 cost 0 latency 0.000060 osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.885212 7f9d7715b700  5 -- op tracker -- , seq: 747726, time: 2014-06-12 14:46:17.885002, event: op_commit, op: osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.885657 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.885667 7f9d63bd8700 10 osd.2 173 dequeue_op 0x68a7600 prio 196 cost 0 latency 0.000745 osd_sub_op_reply(client.4290.0:143629 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 pg pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean]
2014-06-12 14:46:17.885688 7f9d63bd8700  5 -- op tracker -- , seq: 747729, time: 2014-06-12 14:46:17.885684, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143629 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.885698 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] handle_message: 0x68a7600
2014-06-12 14:46:17.885713 7f9d63bd8700  5 -- op tracker -- , seq: 747729, time: 2014-06-12 14:46:17.885710, event: started, op: osd_sub_op_reply(client.4290.0:143629 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.885723 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] sub_op_modify_reply: tid 236243 op  ack_type 4 from 27
2014-06-12 14:46:17.885743 7f9d63bd8700  5 -- op tracker -- , seq: 747726, time: 2014-06-12 14:46:17.885737, event: sub_op_commit_rec, op: osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.885753 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] repop_all_applied: repop tid 236243 all applied 
2014-06-12 14:46:17.885763 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] eval_repop repgather(0x5112a80 173'2577 rep_tid=236243 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.885781 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean]  sending ack on repgather(0x5112a80 173'2577 rep_tid=236243 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x31d24000
2014-06-12 14:46:17.885815 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2576 crt=0'0 lcod 173'2575 mlcod 173'2575 active+clean] repop_all_committed: repop tid 236243 all committed 
2014-06-12 14:46:17.885825 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2575 active+clean] eval_repop repgather(0x5112a80 173'2577 rep_tid=236243 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.885841 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2575 active+clean] log_op_stats osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.019687 lat 0.019714
2014-06-12 14:46:17.885867 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2575 active+clean] publish_stats_to_osd 173:3222
2014-06-12 14:46:17.885883 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2575 active+clean]  sending commit on repgather(0x5112a80 173'2577 rep_tid=236243 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x205c500
2014-06-12 14:46:17.885922 7f9d63bd8700  5 -- op tracker -- , seq: 747726, time: 2014-06-12 14:46:17.885916, event: commit_sent, op: osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.885930 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2575 active+clean]  requeue_ops 
2014-06-12 14:46:17.885943 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2576 active+clean]  removing repgather(0x5112a80 173'2577 rep_tid=236243 committed?=1 applied?=1 op=osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.885959 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2576 active+clean]    q front is repgather(0x5112a80 173'2577 rep_tid=236243 committed?=1 applied?=1 op=osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.885979 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2576 active+clean] remove_repop repgather(0x5112a80 173'2577 rep_tid=236243 committed?=1 applied?=1 op=osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.885996 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2576 active+clean]  requeue_ops 
2014-06-12 14:46:17.886020 7f9d63bd8700  5 -- op tracker -- , seq: 747726, time: 2014-06-12 14:46:17.886014, event: done, op: osd_op(client.4290.0:143629 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.886053 7f9d63bd8700 10 osd.2 173 dequeue_op 0x68a7600 finish
2014-06-12 14:46:17.886061 7f9d63bd8700  5 -- op tracker -- , seq: 747729, time: 2014-06-12 14:46:17.886057, event: done, op: osd_sub_op_reply(client.4290.0:143629 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.886087 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.886095 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.886099 7f9d63bd8700 10 osd.2 173 dequeue_op 0x68a6000 prio 196 cost 0 latency 0.001027 osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0) v2 pg pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2576 active+clean]
2014-06-12 14:46:17.886118 7f9d63bd8700  5 -- op tracker -- , seq: 747730, time: 2014-06-12 14:46:17.886115, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.886126 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2576 active+clean] handle_message: 0x68a6000
2014-06-12 14:46:17.886142 7f9d63bd8700  5 -- op tracker -- , seq: 747730, time: 2014-06-12 14:46:17.886138, event: started, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.886147 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2576 active+clean] sub_op_modify_reply: tid 236244 op  ack_type 1 from 27
2014-06-12 14:46:17.886168 7f9d63bd8700  5 -- op tracker -- , seq: 747728, time: 2014-06-12 14:46:17.886161, event: sub_op_applied_rec, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.886175 7f9d63bd8700 10 osd.2 173 dequeue_op 0x68a6000 finish
2014-06-12 14:46:17.886182 7f9d63bd8700  5 -- op tracker -- , seq: 747730, time: 2014-06-12 14:46:17.886179, event: done, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.886193 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.886198 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.886202 7f9d63bd8700 10 osd.2 173 dequeue_op 0x68a7000 prio 196 cost 0 latency 0.001050 osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 pg pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2576 active+clean]
2014-06-12 14:46:17.886220 7f9d63bd8700  5 -- op tracker -- , seq: 747731, time: 2014-06-12 14:46:17.886217, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.886226 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2576 active+clean] handle_message: 0x68a7000
2014-06-12 14:46:17.886240 7f9d63bd8700  5 -- op tracker -- , seq: 747731, time: 2014-06-12 14:46:17.886237, event: started, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.886246 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2576 active+clean] sub_op_modify_reply: tid 236244 op  ack_type 4 from 27
2014-06-12 14:46:17.886264 7f9d63bd8700  5 -- op tracker -- , seq: 747728, time: 2014-06-12 14:46:17.886259, event: sub_op_commit_rec, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.886270 7f9d63bd8700 10 osd.2 173 dequeue_op 0x68a7000 finish
2014-06-12 14:46:17.886276 7f9d63bd8700  5 -- op tracker -- , seq: 747731, time: 2014-06-12 14:46:17.886273, event: done, op: osd_sub_op_reply(client.4290.0:143630 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.886286 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.886289 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.886307 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.886396 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.886436 7f9d79960700 10 lookup_strip_header done  cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:17.886486 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:17.889144 7f9d79960700 10 _generic_write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 0~458752 = 0
2014-06-12 14:46:17.889203 7f9d79960700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:17.889220 7f9d79960700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:17.889228 7f9d79960700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:17.889237 7f9d79960700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:17.889470 7f9d79960700 20 set_header setting 16936 cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 parent seq 0
2014-06-12 14:46:17.889537 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.889587 7f9d79960700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:17.889600 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:17.892860 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.892910 7f9d8096e700 15 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.892918 7f9d8096e700 10 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1 r = 0
2014-06-12 14:46:17.892926 7f9d8096e700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.892941 7f9d79960700 10 _do_op 0x777ac00 seq 2795 r = 0, finisher 0x738aca0 0
2014-06-12 14:46:17.892954 7f9d79960700 10 _finish_op 0x777ac00 seq 2795 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:17.892966 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (1 active)
2014-06-12 14:46:17.892974 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.893010 7f9d7715b700  5 -- op tracker -- , seq: 747727, time: 2014-06-12 14:46:17.892997, event: commit_sent, op: osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.893025 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2993 (0'0,173'2993] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2991 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:17.893057 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:17.893185 7f9d8096e700 15 _write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 937984~516096
2014-06-12 14:46:17.893335 7f9d8096e700 10 lookup_strip_header done  cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.893435 7f9d7795c700  5 -- op tracker -- , seq: 747727, time: 2014-06-12 14:46:17.893428, event: sub_op_applied, op: osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.893444 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2993 (0'0,173'2993] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2992 active] sub_op_modify_applied on 0x2b2ae380 op osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.893460 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2993 (0'0,173'2993] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2992 active] op_applied on primary on version 173'2993
2014-06-12 14:46:17.893468 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2993 (0'0,173'2993] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2992 active] op_applied on replica on version 173'2993
2014-06-12 14:46:17.893494 7f9d7795c700  5 -- op tracker -- , seq: 747727, time: 2014-06-12 14:46:17.893491, event: done, op: osd_sub_op(client.4308.0:145014 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.893555 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:17.895939 7f9d8096e700 10 _generic_write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 937984~516096 = 0
2014-06-12 14:46:17.895984 7f9d8096e700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.895995 7f9d8096e700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.896000 7f9d8096e700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.896005 7f9d8096e700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.896155 7f9d8096e700 20 set_header setting 15870 cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 parent seq 0
2014-06-12 14:46:17.896219 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.896229 7f9d8096e700 20 set_header setting 2128 cid meta oid 1fe0f8ca/pglog_3.717/0//-1 parent seq 0
2014-06-12 14:46:17.896235 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:17.900565 7f9d8096e700 10 _do_op 0x6873560 seq 2380 r = 0, finisher 0x895d280 0x32b6d2c0
2014-06-12 14:46:17.900764 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2576 active+clean] op_commit: 236244
2014-06-12 14:46:17.900794 7f9d8096e700 10 _finish_op 0x6873560 seq 2380 osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.900792 7f9d494c9700  5 -- op tracker -- , seq: 747732, time: 2014-06-12 14:46:17.897708, event: header_read, op: osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.900808 7f9d494c9700  5 -- op tracker -- , seq: 747732, time: 2014-06-12 14:46:17.897715, event: throttled, op: osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.900820 7f9d494c9700  5 -- op tracker -- , seq: 747732, time: 2014-06-12 14:46:17.900755, event: all_read, op: osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.900832 7f9d494c9700  5 -- op tracker -- , seq: 747732, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.900823 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a3d40 (0 active)
2014-06-12 14:46:17.900841 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:46:17.900846 7f9d7715b700  5 -- op tracker -- , seq: 747728, time: 2014-06-12 14:46:17.900790, event: op_commit, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.900857 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2577 crt=0'0 lcod 173'2576 mlcod 173'2576 active+clean] repop_all_committed: repop tid 236244 all committed 
2014-06-12 14:46:17.900874 7f9d494c9700 15 osd.2 173 enqueue_op 0x20e86100 prio 63 cost 516096 latency 0.003166 osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.900870 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2576 active+clean] eval_repop repgather(0x18224540 173'2578 rep_tid=236244 committed?=1 applied?=0 lock=0 op=osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.900888 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2576 active+clean] log_op_stats osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.030061
2014-06-12 14:46:17.900908 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.900905 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2576 active+clean] publish_stats_to_osd 173:3223
2014-06-12 14:46:17.900916 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2576 active+clean]  sending commit on repgather(0x18224540 173'2578 rep_tid=236244 committed?=1 applied?=0 lock=0 op=osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0xb954c80
2014-06-12 14:46:17.900955 7f9d7715b700  5 -- op tracker -- , seq: 747728, time: 2014-06-12 14:46:17.900950, event: commit_sent, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.900991 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2576 active+clean] op_applied: 236244
2014-06-12 14:46:17.901020 7f9d7795c700  5 -- op tracker -- , seq: 747728, time: 2014-06-12 14:46:17.901015, event: op_applied, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.901027 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2576 active+clean] op_applied on primary on version 173'2578
2014-06-12 14:46:17.901036 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2576 active+clean] repop_all_applied: repop tid 236244 all applied 
2014-06-12 14:46:17.901045 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2576 active+clean] eval_repop repgather(0x18224540 173'2578 rep_tid=236244 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.901059 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2576 active+clean] log_op_stats osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.030233
2014-06-12 14:46:17.901073 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2576 active+clean] publish_stats_to_osd 173:3224
2014-06-12 14:46:17.901084 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2576 active+clean]  requeue_ops 
2014-06-12 14:46:17.901094 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean]  removing repgather(0x18224540 173'2578 rep_tid=236244 committed?=1 applied?=1 op=osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.901108 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean]    q front is repgather(0x18224540 173'2578 rep_tid=236244 committed?=1 applied?=1 op=osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.901121 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] remove_repop repgather(0x18224540 173'2578 rep_tid=236244 committed?=1 applied?=1 op=osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.901134 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean]  requeue_ops 
2014-06-12 14:46:17.901165 7f9d7795c700  5 -- op tracker -- , seq: 747728, time: 2014-06-12 14:46:17.901160, event: done, op: osd_op(client.4290.0:143630 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 937984~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.900836 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.901197 7f9d623d5700 10 osd.2 173 dequeue_op 0x20e86100 prio 63 cost 516096 latency 0.003489 osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean]
2014-06-12 14:46:17.901223 7f9d623d5700  5 -- op tracker -- , seq: 747732, time: 2014-06-12 14:46:17.901218, event: reached_pg, op: osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.901234 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.901250 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] handle_message: 0x20e86100
2014-06-12 14:46:17.901260 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] do_op osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.901285 7f9d623d5700 15 getattr 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 '_'
2014-06-12 14:46:17.901534 7f9d623d5700 10 getattr 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 '_' = 0
2014-06-12 14:46:17.901559 7f9d623d5700 15 getattr 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 'snapset'
2014-06-12 14:46:17.901664 7f9d623d5700 10 getattr 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 'snapset' = 0
2014-06-12 14:46:17.901674 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] populate_obc_watchers 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.901693 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.901702 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] get_object_context: creating obc from disk: 0xa039b80
2014-06-12 14:46:17.901711 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] get_object_context: 0xa039b80 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 rwstate(none n=0 w=0) oi: 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2578 client.4290.0:143630 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2578) ssc: 0x1550eee0 snapset: 0=[]:[]+head
2014-06-12 14:46:17.901727 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] find_object_context 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 @head oi=82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2578 client.4290.0:143630 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2578)
2014-06-12 14:46:17.901762 7f9d623d5700  5 -- op tracker -- , seq: 747732, time: 2014-06-12 14:46:17.901754, event: started, op: osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.901773 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] execute_ctx 0x55a4400
2014-06-12 14:46:17.901790 7f9d623d5700  5 -- op tracker -- , seq: 747732, time: 2014-06-12 14:46:17.901786, event: started, op: osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.901799 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] do_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] ov 173'2578 av 173'2579 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.901814 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] do_osd_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096]
2014-06-12 14:46:17.901824 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.901837 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] do_osd_op  write 1454080~516096
2014-06-12 14:46:17.901851 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:46:17.901860 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 snapset=0x1550ef18  snapc=0=[]
2014-06-12 14:46:17.901872 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.901881 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] finish_ctx 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 0x55a4400 op modify  
2014-06-12 14:46:17.901893 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean]  set mtime to 2014-06-12 08:45:20.275353
2014-06-12 14:46:17.901912 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean]  final snapset 0=[]:[]+head in 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.901930 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.901939 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] new_repop rep_tid 236245 on osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.901954 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] issue_repop rep_tid 236245 o 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.901977 7f9d623d5700  5 -- op tracker -- , seq: 747732, time: 2014-06-12 14:46:17.901972, event: waiting for subops from 27, op: osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.902000 7f9d623d5700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:17.902022 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2578 (0'0,173'2578] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] append_log log((0'0,173'2578], crt=0'0) [173'2579 (173'2578) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143631 2014-06-12 08:45:20.275353]
2014-06-12 14:46:17.902044 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] add_log_entry 173'2579 (173'2578) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143631 2014-06-12 08:45:20.275353
2014-06-12 14:46:17.902064 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.902094 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2579, trimmed: 
2014-06-12 14:46:17.902116 7f9d623d5700  5 queue_transactions existing osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.902120 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.902121 7f9d623d5700  5 queue_op 0x5f87b00 seq 2381 osr(3.717 0x276de70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:17.902129 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] eval_repop repgather(0x11c7e6c0 173'2579 rep_tid=236245 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.902147 7f9d623d5700 10 osd.2 173 dequeue_op 0x20e86100 finish
2014-06-12 14:46:17.902153 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.902157 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.902166 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a3d40 (1 active)
2014-06-12 14:46:17.902171 7f9d8016d700  5 _do_op 0x5f87b00 seq 2381 osr(3.717 0x276de70)/0x276de70 start
2014-06-12 14:46:17.902175 7f9d8016d700 10 _do_transaction on 0x15e56300
2014-06-12 14:46:17.902180 7f9d8016d700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.902296 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.902317 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.902366 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.902385 7f9d8016d700 15 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.902391 7f9d8016d700 10 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1 r = 0
2014-06-12 14:46:17.902396 7f9d8016d700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.902509 7f9d8016d700 15 _write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 1454080~516096
2014-06-12 14:46:17.902648 7f9d8016d700 10 lookup_strip_header done  cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.902787 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:17.903474 7f9d5a16f700  5 -- op tracker -- , seq: 747733, time: 2014-06-12 14:46:17.902974, event: header_read, op: osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.903483 7f9d5a16f700  5 -- op tracker -- , seq: 747733, time: 2014-06-12 14:46:17.902975, event: throttled, op: osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.903488 7f9d5a16f700  5 -- op tracker -- , seq: 747733, time: 2014-06-12 14:46:17.903442, event: all_read, op: osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.903493 7f9d5a16f700  5 -- op tracker -- , seq: 747733, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.903498 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:17.903503 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:17.903510 7f9d5a16f700 15 osd.2 173 enqueue_op 0xe20b600 prio 127 cost 516903 latency 0.000535 osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.903539 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.903548 7f9d643d9700 10 osd.2 173 dequeue_op 0xe20b600 prio 127 cost 516903 latency 0.000573 osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2659 (0'0,173'2659] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2658 active]
2014-06-12 14:46:17.903571 7f9d643d9700  5 -- op tracker -- , seq: 747733, time: 2014-06-12 14:46:17.903567, event: reached_pg, op: osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.903580 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2659 (0'0,173'2659] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2658 active] handle_message: 0xe20b600
2014-06-12 14:46:17.903594 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2659 (0'0,173'2659] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2658 active] sub_op_modify trans f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 v 173'2660 (transaction) 174
2014-06-12 14:46:17.903610 7f9d643d9700  5 -- op tracker -- , seq: 747733, time: 2014-06-12 14:46:17.903606, event: started, op: osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.903626 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2659 (0'0,173'2659] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2658 active] append_log log((0'0,173'2659], crt=0'0) [173'2660 (173'2658) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146406 2014-06-12 08:41:29.644370]
2014-06-12 14:46:17.903656 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2660 (0'0,173'2660] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2658 active] add_log_entry 173'2660 (173'2658) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146406 2014-06-12 08:41:29.644370
2014-06-12 14:46:17.903678 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2660 (0'0,173'2660] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2658 active] append_log  adding 1 keys
2014-06-12 14:46:17.903714 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2660, trimmed: 
2014-06-12 14:46:17.903738 7f9d643d9700  5 -- op tracker -- , seq: 747733, time: 2014-06-12 14:46:17.903733, event: started, op: osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.903758 7f9d643d9700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:17.903762 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.903763 7f9d643d9700  5 queue_op 0x7730f60 seq 2470 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:17.903769 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2660 (0'0,173'2660] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2658 active] do_sub_op osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.903783 7f9d643d9700 10 osd.2 173 dequeue_op 0xe20b600 finish
2014-06-12 14:46:17.903786 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.903789 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.903799 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (2 active)
2014-06-12 14:46:17.903806 7f9d7915f700  5 _do_op 0x7730f60 seq 2470 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:17.903811 7f9d7915f700 10 _do_transaction on 0x1909feb0
2014-06-12 14:46:17.903819 7f9d7915f700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:17.904161 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:17.904203 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.904624 7f9d494c9700  5 -- op tracker -- , seq: 747734, time: 2014-06-12 14:46:17.903762, event: header_read, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.904641 7f9d494c9700  5 -- op tracker -- , seq: 747734, time: 2014-06-12 14:46:17.903765, event: throttled, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.904650 7f9d494c9700  5 -- op tracker -- , seq: 747734, time: 2014-06-12 14:46:17.904586, event: all_read, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.904659 7f9d494c9700  5 -- op tracker -- , seq: 747734, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.904668 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:46:17.904703 7f9d494c9700 15 osd.2 173 enqueue_op 0x20e87300 prio 63 cost 516096 latency 0.000940 osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.904748 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.904760 7f9d61bd4700 10 osd.2 173 dequeue_op 0x20e87300 prio 63 cost 516096 latency 0.000997 osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean]
2014-06-12 14:46:17.904794 7f9d61bd4700  5 -- op tracker -- , seq: 747734, time: 2014-06-12 14:46:17.904789, event: reached_pg, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.904807 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.904823 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] handle_message: 0x20e87300
2014-06-12 14:46:17.904833 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] do_op osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.904858 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] get_object_context: found obc in cache: 0xa039b80
2014-06-12 14:46:17.904868 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] get_object_context: 0xa039b80 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 rwstate(write n=1 w=0) oi: 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2579 client.4290.0:143631 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2579) ssc: 0x1550eee0 snapset: 0=[]:[]+head
2014-06-12 14:46:17.904884 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] find_object_context 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 @head oi=82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2579 client.4290.0:143631 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2579)
2014-06-12 14:46:17.904908 7f9d61bd4700  5 -- op tracker -- , seq: 747734, time: 2014-06-12 14:46:17.904903, event: started, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.904915 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] execute_ctx 0x5ca1e00
2014-06-12 14:46:17.904931 7f9d61bd4700  5 -- op tracker -- , seq: 747734, time: 2014-06-12 14:46:17.904927, event: started, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.904937 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] do_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] ov 173'2579 av 173'2580 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.904950 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] do_osd_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096]
2014-06-12 14:46:17.904961 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.904973 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] do_osd_op  write 1970176~516096
2014-06-12 14:46:17.904987 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:46:17.904998 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 snapset=0x1550ef18  snapc=0=[]
2014-06-12 14:46:17.905009 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.905019 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] finish_ctx 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 0x5ca1e00 op modify  
2014-06-12 14:46:17.905034 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean]  set mtime to 2014-06-12 08:45:20.283181
2014-06-12 14:46:17.905056 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean]  final snapset 0=[]:[]+head in 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.905073 7f9d8016d700 10 _generic_write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 1454080~516096 = 0
2014-06-12 14:46:17.905075 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.905085 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] new_repop rep_tid 236246 on osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.905100 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] issue_repop rep_tid 236246 o 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.905122 7f9d8016d700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.905126 7f9d61bd4700  5 -- op tracker -- , seq: 747734, time: 2014-06-12 14:46:17.905121, event: waiting for subops from 27, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.905136 7f9d8016d700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.905148 7f9d8016d700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.905150 7f9d61bd4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:17.905156 7f9d8016d700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.905174 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2579 (0'0,173'2579] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] append_log log((0'0,173'2579], crt=0'0) [173'2580 (173'2579) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143632 2014-06-12 08:45:20.283181]
2014-06-12 14:46:17.905197 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] add_log_entry 173'2580 (173'2579) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143632 2014-06-12 08:45:20.283181
2014-06-12 14:46:17.905217 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.905246 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2580, trimmed: 
2014-06-12 14:46:17.905273 7f9d61bd4700  5 queue_transactions existing osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.905277 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.905278 7f9d61bd4700  5 queue_op 0x6c32a20 seq 2382 osr(3.717 0x276de70) 518389 bytes   (queue has 3 ops and 1555167 bytes)
2014-06-12 14:46:17.905286 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] eval_repop repgather(0x15acf500 173'2580 rep_tid=236246 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.905304 7f9d61bd4700 10 osd.2 173 dequeue_op 0x20e87300 finish
2014-06-12 14:46:17.905307 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.905310 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.905319 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a3d40 (3 active)
2014-06-12 14:46:17.905355 7f9d8016d700 20 set_header setting 15870 cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 parent seq 0
2014-06-12 14:46:17.905432 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.905442 7f9d8016d700 20 set_header setting 2128 cid meta oid 1fe0f8ca/pglog_3.717/0//-1 parent seq 0
2014-06-12 14:46:17.905448 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:17.908316 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.908357 7f9d7915f700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:17.908364 7f9d7915f700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:17.908372 7f9d7915f700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:17.908391 7f9d7915f700 15 _write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 811008~516096
2014-06-12 14:46:17.908561 7f9d7915f700 10 lookup_strip_header done  cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:17.908900 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:17.911304 7f9d8016d700 10 _do_op 0x5f87b00 seq 2381 r = 0, finisher 0x722e5c0 0x3eabb700
2014-06-12 14:46:17.911343 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] op_commit: 236245
2014-06-12 14:46:17.911380 7f9d7715b700  5 -- op tracker -- , seq: 747732, time: 2014-06-12 14:46:17.911368, event: op_commit, op: osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.911397 7f9d8016d700 10 _finish_op 0x5f87b00 seq 2381 osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.911406 7f9d7d968700  5 _do_op 0x6c32a20 seq 2382 osr(3.717 0x276de70)/0x276de70 start
2014-06-12 14:46:17.911411 7f9d7d968700 10 _do_transaction on 0x5139380
2014-06-12 14:46:17.911421 7f9d7d968700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.911577 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.911596 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.911658 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] op_applied: 236245
2014-06-12 14:46:17.911679 7f9d7795c700  5 -- op tracker -- , seq: 747732, time: 2014-06-12 14:46:17.911673, event: op_applied, op: osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.911687 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] op_applied on primary on version 173'2579
2014-06-12 14:46:17.911716 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a3d40 (2 active)
2014-06-12 14:46:17.911720 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.911933 7f9d58c5a700  5 -- op tracker -- , seq: 747735, time: 2014-06-12 14:46:17.911890, event: header_read, op: osd_sub_op_reply(client.4290.0:143631 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.911942 7f9d58c5a700  5 -- op tracker -- , seq: 747735, time: 2014-06-12 14:46:17.911891, event: throttled, op: osd_sub_op_reply(client.4290.0:143631 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.911949 7f9d58c5a700  5 -- op tracker -- , seq: 747735, time: 2014-06-12 14:46:17.911906, event: all_read, op: osd_sub_op_reply(client.4290.0:143631 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.911956 7f9d58c5a700  5 -- op tracker -- , seq: 747735, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143631 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.911961 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143631 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.911967 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:17.911975 7f9d58c5a700 15 osd.2 173 enqueue_op 0x68a7b00 prio 196 cost 0 latency 0.000085 osd_sub_op_reply(client.4290.0:143631 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.912348 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.912358 7f9d673df700 10 osd.2 173 dequeue_op 0x68a7b00 prio 196 cost 0 latency 0.000468 osd_sub_op_reply(client.4290.0:143631 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 pg pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean]
2014-06-12 14:46:17.912378 7f9d673df700  5 -- op tracker -- , seq: 747735, time: 2014-06-12 14:46:17.912374, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143631 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.912387 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] handle_message: 0x68a7b00
2014-06-12 14:46:17.912402 7f9d673df700  5 -- op tracker -- , seq: 747735, time: 2014-06-12 14:46:17.912398, event: started, op: osd_sub_op_reply(client.4290.0:143631 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.912408 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] sub_op_modify_reply: tid 236245 op  ack_type 4 from 27
2014-06-12 14:46:17.912428 7f9d673df700  5 -- op tracker -- , seq: 747732, time: 2014-06-12 14:46:17.912420, event: sub_op_commit_rec, op: osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.912435 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] repop_all_applied: repop tid 236245 all applied 
2014-06-12 14:46:17.912445 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] eval_repop repgather(0x11c7e6c0 173'2579 rep_tid=236245 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.912460 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean]  sending ack on repgather(0x11c7e6c0 173'2579 rep_tid=236245 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x27474280
2014-06-12 14:46:17.912492 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2578 crt=0'0 lcod 173'2577 mlcod 173'2577 active+clean] repop_all_committed: repop tid 236245 all committed 
2014-06-12 14:46:17.912502 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2577 active+clean] eval_repop repgather(0x11c7e6c0 173'2579 rep_tid=236245 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.912518 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2577 active+clean] log_op_stats osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.014782 lat 0.014808
2014-06-12 14:46:17.912538 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2577 active+clean] publish_stats_to_osd 173:3225
2014-06-12 14:46:17.912556 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2577 active+clean]  sending commit on repgather(0x11c7e6c0 173'2579 rep_tid=236245 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x191a1680
2014-06-12 14:46:17.912585 7f9d673df700  5 -- op tracker -- , seq: 747732, time: 2014-06-12 14:46:17.912580, event: commit_sent, op: osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.912593 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2577 active+clean]  requeue_ops 
2014-06-12 14:46:17.912602 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean]  removing repgather(0x11c7e6c0 173'2579 rep_tid=236245 committed?=1 applied?=1 op=osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.912615 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean]    q front is repgather(0x11c7e6c0 173'2579 rep_tid=236245 committed?=1 applied?=1 op=osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.912629 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] remove_repop repgather(0x11c7e6c0 173'2579 rep_tid=236245 committed?=1 applied?=1 op=osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.912642 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean]  requeue_ops 
2014-06-12 14:46:17.912661 7f9d673df700  5 -- op tracker -- , seq: 747732, time: 2014-06-12 14:46:17.912656, event: done, op: osd_op(client.4290.0:143631 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.912678 7f9d673df700 10 osd.2 173 dequeue_op 0x68a7b00 finish
2014-06-12 14:46:17.912684 7f9d673df700  5 -- op tracker -- , seq: 747735, time: 2014-06-12 14:46:17.912681, event: done, op: osd_sub_op_reply(client.4290.0:143631 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.912694 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.912697 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.913690 7f9d7915f700 10 _generic_write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 811008~516096 = 0
2014-06-12 14:46:17.913754 7f9d7915f700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:17.913767 7f9d7915f700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:17.913772 7f9d7915f700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:17.913779 7f9d7915f700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:17.913988 7f9d7915f700 20 set_header setting 16921 cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 parent seq 0
2014-06-12 14:46:17.914070 7f9d7915f700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:17.914084 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.914091 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:17.916729 7f9d494c9700  5 -- op tracker -- , seq: 747736, time: 2014-06-12 14:46:17.915811, event: header_read, op: osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.916752 7f9d494c9700  5 -- op tracker -- , seq: 747736, time: 2014-06-12 14:46:17.915814, event: throttled, op: osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.916761 7f9d494c9700  5 -- op tracker -- , seq: 747736, time: 2014-06-12 14:46:17.916646, event: all_read, op: osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.916770 7f9d494c9700  5 -- op tracker -- , seq: 747736, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.916782 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:46:17.916819 7f9d494c9700 15 osd.2 173 enqueue_op 0x12bf6800 prio 63 cost 516096 latency 0.001007 osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.916867 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.916876 7f9d663dd700 10 osd.2 173 dequeue_op 0x12bf6800 prio 63 cost 516096 latency 0.001065 osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean]
2014-06-12 14:46:17.916903 7f9d663dd700  5 -- op tracker -- , seq: 747736, time: 2014-06-12 14:46:17.916898, event: reached_pg, op: osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.916912 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.916928 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] handle_message: 0x12bf6800
2014-06-12 14:46:17.916941 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] do_op osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.916986 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] get_object_context: found obc in cache: 0xa039b80
2014-06-12 14:46:17.916996 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] get_object_context: 0xa039b80 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 rwstate(write n=1 w=0) oi: 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2580 client.4290.0:143632 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2580) ssc: 0x1550eee0 snapset: 0=[]:[]+head
2014-06-12 14:46:17.917012 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] find_object_context 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 @head oi=82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2580 client.4290.0:143632 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2580)
2014-06-12 14:46:17.917039 7f9d663dd700  5 -- op tracker -- , seq: 747736, time: 2014-06-12 14:46:17.917034, event: started, op: osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.917047 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] execute_ctx 0x29ede400
2014-06-12 14:46:17.917065 7f9d663dd700  5 -- op tracker -- , seq: 747736, time: 2014-06-12 14:46:17.917060, event: started, op: osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.917071 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] do_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] ov 173'2580 av 173'2581 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.917085 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] do_osd_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096]
2014-06-12 14:46:17.917096 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.917112 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] do_osd_op  write 2486272~516096
2014-06-12 14:46:17.917131 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:46:17.917146 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 snapset=0x1550ef18  snapc=0=[]
2014-06-12 14:46:17.917160 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.917171 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] finish_ctx 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 0x29ede400 op modify  
2014-06-12 14:46:17.917188 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean]  set mtime to 2014-06-12 08:45:20.295562
2014-06-12 14:46:17.917216 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean]  final snapset 0=[]:[]+head in 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.917245 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.917255 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] new_repop rep_tid 236247 on osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.917271 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] issue_repop rep_tid 236247 o 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.917301 7f9d663dd700  5 -- op tracker -- , seq: 747736, time: 2014-06-12 14:46:17.917296, event: waiting for subops from 27, op: osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.917325 7f9d663dd700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:17.917354 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2580 (0'0,173'2580] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] append_log log((0'0,173'2580], crt=0'0) [173'2581 (173'2580) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143633 2014-06-12 08:45:20.295562]
2014-06-12 14:46:17.917380 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] add_log_entry 173'2581 (173'2580) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143633 2014-06-12 08:45:20.295562
2014-06-12 14:46:17.917400 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.917434 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2581, trimmed: 
2014-06-12 14:46:17.917456 7f9d663dd700  5 queue_transactions existing osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.917461 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.917463 7f9d663dd700  5 queue_op 0x7419380 seq 2383 osr(3.717 0x276de70) 518389 bytes   (queue has 3 ops and 1555167 bytes)
2014-06-12 14:46:17.917472 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] eval_repop repgather(0x24fa7740 173'2581 rep_tid=236247 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.917490 7f9d663dd700 10 osd.2 173 dequeue_op 0x12bf6800 finish
2014-06-12 14:46:17.917494 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.917497 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.917508 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a3d40 (3 active)
2014-06-12 14:46:17.918252 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.918362 7f9d7d968700 15 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.918372 7f9d7d968700 10 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1 r = 0
2014-06-12 14:46:17.918380 7f9d7d968700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.918400 7f9d7d968700 15 _write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 1970176~516096
2014-06-12 14:46:17.918563 7f9d7d968700 10 lookup_strip_header done  cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.918729 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:17.921048 7f9d7d968700 10 _generic_write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 1970176~516096 = 0
2014-06-12 14:46:17.921151 7f9d7d968700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.921175 7f9d7d968700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.921181 7f9d7d968700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.921189 7f9d7d968700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.921477 7f9d7d968700 20 set_header setting 15870 cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 parent seq 0
2014-06-12 14:46:17.921689 7f9d58c5a700  5 -- op tracker -- , seq: 747737, time: 2014-06-12 14:46:17.921647, event: header_read, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.921705 7f9d58c5a700  5 -- op tracker -- , seq: 747737, time: 2014-06-12 14:46:17.921649, event: throttled, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.921713 7f9d58c5a700  5 -- op tracker -- , seq: 747737, time: 2014-06-12 14:46:17.921662, event: all_read, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.921721 7f9d58c5a700  5 -- op tracker -- , seq: 747737, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.921727 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:17.921733 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:17.921743 7f9d58c5a700 15 osd.2 173 enqueue_op 0x28dd1000 prio 196 cost 0 latency 0.000095 osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0) v2
2014-06-12 14:46:17.921793 7f9d58c5a700  5 -- op tracker -- , seq: 747738, time: 2014-06-12 14:46:17.921767, event: header_read, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.921801 7f9d58c5a700  5 -- op tracker -- , seq: 747738, time: 2014-06-12 14:46:17.921767, event: throttled, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.921808 7f9d58c5a700  5 -- op tracker -- , seq: 747738, time: 2014-06-12 14:46:17.921776, event: all_read, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.921814 7f9d58c5a700  5 -- op tracker -- , seq: 747738, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.921818 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.921822 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:17.921827 7f9d58c5a700 15 osd.2 173 enqueue_op 0x28dd0800 prio 196 cost 0 latency 0.000060 osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.922577 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.922598 7f9d7d968700 20 set_header setting 2128 cid meta oid 1fe0f8ca/pglog_3.717/0//-1 parent seq 0
2014-06-12 14:46:17.922608 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:17.923229 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.923245 7f9d67be0700 10 osd.2 173 dequeue_op 0x28dd1000 prio 196 cost 0 latency 0.001597 osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0) v2 pg pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean]
2014-06-12 14:46:17.923276 7f9d67be0700  5 -- op tracker -- , seq: 747737, time: 2014-06-12 14:46:17.923272, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.923289 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] handle_message: 0x28dd1000
2014-06-12 14:46:17.923309 7f9d67be0700  5 -- op tracker -- , seq: 747737, time: 2014-06-12 14:46:17.923306, event: started, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.923317 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] sub_op_modify_reply: tid 236246 op  ack_type 1 from 27
2014-06-12 14:46:17.923340 7f9d67be0700  5 -- op tracker -- , seq: 747734, time: 2014-06-12 14:46:17.923330, event: sub_op_applied_rec, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.923349 7f9d67be0700 10 osd.2 173 dequeue_op 0x28dd1000 finish
2014-06-12 14:46:17.923355 7f9d67be0700  5 -- op tracker -- , seq: 747737, time: 2014-06-12 14:46:17.923352, event: done, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ack, result = 0)
2014-06-12 14:46:17.923375 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.923382 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.923386 7f9d67be0700 10 osd.2 173 dequeue_op 0x28dd0800 prio 196 cost 0 latency 0.001619 osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 pg pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean]
2014-06-12 14:46:17.923402 7f9d67be0700  5 -- op tracker -- , seq: 747738, time: 2014-06-12 14:46:17.923399, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.923408 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] handle_message: 0x28dd0800
2014-06-12 14:46:17.923422 7f9d67be0700  5 -- op tracker -- , seq: 747738, time: 2014-06-12 14:46:17.923419, event: started, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.923427 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] sub_op_modify_reply: tid 236246 op  ack_type 4 from 27
2014-06-12 14:46:17.923443 7f9d67be0700  5 -- op tracker -- , seq: 747734, time: 2014-06-12 14:46:17.923439, event: sub_op_commit_rec, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.923449 7f9d67be0700 10 osd.2 173 dequeue_op 0x28dd0800 finish
2014-06-12 14:46:17.923455 7f9d67be0700  5 -- op tracker -- , seq: 747738, time: 2014-06-12 14:46:17.923452, event: done, op: osd_sub_op_reply(client.4290.0:143632 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.923466 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.923469 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.923956 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.924131 7f9d58c5a700  5 -- op tracker -- , seq: 747739, time: 2014-06-12 14:46:17.924081, event: header_read, op: osd_sub_op_reply(client.4290.0:143633 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.924144 7f9d58c5a700  5 -- op tracker -- , seq: 747739, time: 2014-06-12 14:46:17.924083, event: throttled, op: osd_sub_op_reply(client.4290.0:143633 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.924152 7f9d58c5a700  5 -- op tracker -- , seq: 747739, time: 2014-06-12 14:46:17.924100, event: all_read, op: osd_sub_op_reply(client.4290.0:143633 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.924159 7f9d58c5a700  5 -- op tracker -- , seq: 747739, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143633 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.924164 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143633 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.924171 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:17.924180 7f9d58c5a700 15 osd.2 173 enqueue_op 0x8dbdc00 prio 196 cost 0 latency 0.000098 osd_sub_op_reply(client.4290.0:143633 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.924310 7f9d7915f700 10 _do_op 0x7730f60 seq 2470 r = 0, finisher 0x6d504c0 0
2014-06-12 14:46:17.924366 7f9d7715b700  5 -- op tracker -- , seq: 747733, time: 2014-06-12 14:46:17.924353, event: commit_sent, op: osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.924377 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2660 (0'0,173'2660] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2658 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:17.924401 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:17.924427 7f9d7915f700 10 _finish_op 0x7730f60 seq 2470 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:17.924450 7f9d7795c700  5 -- op tracker -- , seq: 747733, time: 2014-06-12 14:46:17.924444, event: sub_op_applied, op: osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.924459 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2660 (0'0,173'2660] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active] sub_op_modify_applied on 0x1909fdc0 op osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:17.924474 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2660 (0'0,173'2660] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active] op_applied on primary on version 173'2660
2014-06-12 14:46:17.924483 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2660 (0'0,173'2660] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active] op_applied on replica on version 173'2660
2014-06-12 14:46:17.924520 7f9d7915f700  5 -- op tracker -- , seq: 747733, time: 2014-06-12 14:46:17.924508, event: done, op: osd_sub_op(client.4316.0:146406 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2660 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:17.924557 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (2 active)
2014-06-12 14:46:17.924561 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.924848 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.924860 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x8dbdc00 prio 196 cost 0 latency 0.000779 osd_sub_op_reply(client.4290.0:143633 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 pg pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean]
2014-06-12 14:46:17.924886 7f9d6b3e7700  5 -- op tracker -- , seq: 747739, time: 2014-06-12 14:46:17.924882, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143633 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.924895 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] handle_message: 0x8dbdc00
2014-06-12 14:46:17.924911 7f9d6b3e7700  5 -- op tracker -- , seq: 747739, time: 2014-06-12 14:46:17.924907, event: started, op: osd_sub_op_reply(client.4290.0:143633 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.924917 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] sub_op_modify_reply: tid 236247 op  ack_type 4 from 27
2014-06-12 14:46:17.924939 7f9d6b3e7700  5 -- op tracker -- , seq: 747736, time: 2014-06-12 14:46:17.924929, event: sub_op_commit_rec, op: osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.924946 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x8dbdc00 finish
2014-06-12 14:46:17.924952 7f9d6b3e7700  5 -- op tracker -- , seq: 747739, time: 2014-06-12 14:46:17.924949, event: done, op: osd_sub_op_reply(client.4290.0:143633 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.924965 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.924968 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.926165 7f9d7d968700 10 _do_op 0x6c32a20 seq 2382 r = 0, finisher 0x698c740 0x216a1b40
2014-06-12 14:46:17.926183 7f9d7d968700 10 _finish_op 0x6c32a20 seq 2382 osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.926195 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a3d40 (1 active)
2014-06-12 14:46:17.926199 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.926187 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] op_commit: 236246
2014-06-12 14:46:17.926220 7f9d7715b700  5 -- op tracker -- , seq: 747734, time: 2014-06-12 14:46:17.926211, event: op_commit, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.926242 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2579 crt=0'0 lcod 173'2578 mlcod 173'2578 active+clean] repop_all_committed: repop tid 236246 all committed 
2014-06-12 14:46:17.926253 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2578 active+clean] eval_repop repgather(0x15acf500 173'2580 rep_tid=236246 committed?=1 applied?=0 lock=0 op=osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.926270 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2578 active+clean] log_op_stats osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.022506
2014-06-12 14:46:17.926288 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2578 active+clean] publish_stats_to_osd 173:3226
2014-06-12 14:46:17.926298 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2578 active+clean]  sending commit on repgather(0x15acf500 173'2580 rep_tid=236246 committed?=1 applied?=0 lock=0 op=osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x355c1680
2014-06-12 14:46:17.926335 7f9d7715b700  5 -- op tracker -- , seq: 747734, time: 2014-06-12 14:46:17.926330, event: commit_sent, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.926391 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2578 active+clean] op_applied: 236246
2014-06-12 14:46:17.926411 7f9d7795c700  5 -- op tracker -- , seq: 747734, time: 2014-06-12 14:46:17.926405, event: op_applied, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.926418 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2578 active+clean] op_applied on primary on version 173'2580
2014-06-12 14:46:17.926428 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2578 active+clean] repop_all_applied: repop tid 236246 all applied 
2014-06-12 14:46:17.926437 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2578 active+clean] eval_repop repgather(0x15acf500 173'2580 rep_tid=236246 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.926451 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2578 active+clean] log_op_stats osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.022689
2014-06-12 14:46:17.926465 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2578 active+clean] publish_stats_to_osd 173:3227
2014-06-12 14:46:17.926475 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2578 active+clean]  requeue_ops 
2014-06-12 14:46:17.926484 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean]  removing repgather(0x15acf500 173'2580 rep_tid=236246 committed?=1 applied?=1 op=osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.926498 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean]    q front is repgather(0x15acf500 173'2580 rep_tid=236246 committed?=1 applied?=1 op=osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.926512 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] remove_repop repgather(0x15acf500 173'2580 rep_tid=236246 committed?=1 applied?=1 op=osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.926525 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean]  requeue_ops 
2014-06-12 14:46:17.926543 7f9d7795c700  5 -- op tracker -- , seq: 747734, time: 2014-06-12 14:46:17.926539, event: done, op: osd_op(client.4290.0:143632 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 1970176~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.926834 7f9d83173700  5 _do_op 0x7419380 seq 2383 osr(3.717 0x276de70)/0x276de70 start
2014-06-12 14:46:17.926842 7f9d83173700 10 _do_transaction on 0x24fa6180
2014-06-12 14:46:17.926849 7f9d83173700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.927018 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.927037 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.927092 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.927112 7f9d83173700 15 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.927118 7f9d83173700 10 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1 r = 0
2014-06-12 14:46:17.927124 7f9d83173700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.927250 7f9d83173700 15 _write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 2486272~516096
2014-06-12 14:46:17.927409 7f9d83173700 10 lookup_strip_header done  cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.927568 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:17.930149 7f9d83173700 10 _generic_write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 2486272~516096 = 0
2014-06-12 14:46:17.930216 7f9d83173700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.930233 7f9d83173700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.930242 7f9d83173700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.930250 7f9d83173700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.930459 7f9d83173700 20 set_header setting 15870 cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 parent seq 0
2014-06-12 14:46:17.930578 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.930613 7f9d83173700 20 set_header setting 2128 cid meta oid 1fe0f8ca/pglog_3.717/0//-1 parent seq 0
2014-06-12 14:46:17.930634 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:17.930892 7f9d494c9700  5 -- op tracker -- , seq: 747740, time: 2014-06-12 14:46:17.928970, event: header_read, op: osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.930904 7f9d494c9700  5 -- op tracker -- , seq: 747740, time: 2014-06-12 14:46:17.928973, event: throttled, op: osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.930913 7f9d494c9700  5 -- op tracker -- , seq: 747740, time: 2014-06-12 14:46:17.930847, event: all_read, op: osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.930921 7f9d494c9700  5 -- op tracker -- , seq: 747740, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.930930 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:46:17.930960 7f9d494c9700 15 osd.2 173 enqueue_op 0x12bf6500 prio 63 cost 516096 latency 0.001989 osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.930995 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.931001 7f9d62bd6700 10 osd.2 173 dequeue_op 0x12bf6500 prio 63 cost 516096 latency 0.002031 osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean]
2014-06-12 14:46:17.931023 7f9d62bd6700  5 -- op tracker -- , seq: 747740, time: 2014-06-12 14:46:17.931019, event: reached_pg, op: osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.931032 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.931048 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] handle_message: 0x12bf6500
2014-06-12 14:46:17.931058 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] do_op osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.931083 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] get_object_context: found obc in cache: 0xa039b80
2014-06-12 14:46:17.931095 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] get_object_context: 0xa039b80 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 rwstate(write n=1 w=0) oi: 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2581 client.4290.0:143633 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2581) ssc: 0x1550eee0 snapset: 0=[]:[]+head
2014-06-12 14:46:17.931109 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] find_object_context 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 @head oi=82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2581 client.4290.0:143633 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2581)
2014-06-12 14:46:17.931135 7f9d62bd6700  5 -- op tracker -- , seq: 747740, time: 2014-06-12 14:46:17.931130, event: started, op: osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.931141 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] execute_ctx 0x2e412a00
2014-06-12 14:46:17.931157 7f9d62bd6700  5 -- op tracker -- , seq: 747740, time: 2014-06-12 14:46:17.931152, event: started, op: osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.931162 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] do_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] ov 173'2581 av 173'2582 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.931174 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] do_osd_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096]
2014-06-12 14:46:17.931184 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.931198 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] do_osd_op  write 3002368~516096
2014-06-12 14:46:17.931213 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:46:17.931227 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 snapset=0x1550ef18  snapc=0=[]
2014-06-12 14:46:17.931245 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.931254 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] finish_ctx 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 0x2e412a00 op modify  
2014-06-12 14:46:17.931267 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean]  set mtime to 2014-06-12 08:45:20.308766
2014-06-12 14:46:17.931288 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean]  final snapset 0=[]:[]+head in 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.931306 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.931317 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] new_repop rep_tid 236248 on osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.931333 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] issue_repop rep_tid 236248 o 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.931357 7f9d62bd6700  5 -- op tracker -- , seq: 747740, time: 2014-06-12 14:46:17.931353, event: waiting for subops from 27, op: osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.931381 7f9d62bd6700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:17.931403 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2581 (0'0,173'2581] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] append_log log((0'0,173'2581], crt=0'0) [173'2582 (173'2581) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143634 2014-06-12 08:45:20.308766]
2014-06-12 14:46:17.931425 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] add_log_entry 173'2582 (173'2581) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143634 2014-06-12 08:45:20.308766
2014-06-12 14:46:17.931449 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.931490 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2582, trimmed: 
2014-06-12 14:46:17.931512 7f9d62bd6700  5 queue_transactions existing osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.931517 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.931519 7f9d62bd6700  5 queue_op 0x5ff1e60 seq 2384 osr(3.717 0x276de70) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:17.931527 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] eval_repop repgather(0x10192d80 173'2582 rep_tid=236248 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.931546 7f9d62bd6700 10 osd.2 173 dequeue_op 0x12bf6500 finish
2014-06-12 14:46:17.931550 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.931553 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.931591 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a3d40 (2 active)
2014-06-12 14:46:17.933411 7f9d83173700 10 _do_op 0x7419380 seq 2383 r = 0, finisher 0x6dbd560 0x38e12540
2014-06-12 14:46:17.933440 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] op_commit: 236247
2014-06-12 14:46:17.933470 7f9d7715b700  5 -- op tracker -- , seq: 747736, time: 2014-06-12 14:46:17.933461, event: op_commit, op: osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.933482 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2580 crt=0'0 lcod 173'2579 mlcod 173'2579 active+clean] repop_all_committed: repop tid 236247 all committed 
2014-06-12 14:46:17.933494 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2579 active+clean] eval_repop repgather(0x24fa7740 173'2581 rep_tid=236247 committed?=1 applied?=0 lock=0 op=osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.933512 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2579 active+clean] log_op_stats osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.017700
2014-06-12 14:46:17.933528 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2579 active+clean] publish_stats_to_osd 173:3228
2014-06-12 14:46:17.933539 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2579 active+clean]  sending commit on repgather(0x24fa7740 173'2581 rep_tid=236247 committed?=1 applied?=0 lock=0 op=osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x2f89db80
2014-06-12 14:46:17.933574 7f9d7715b700  5 -- op tracker -- , seq: 747736, time: 2014-06-12 14:46:17.933569, event: commit_sent, op: osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.933587 7f9d83173700 10 _finish_op 0x7419380 seq 2383 osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.933597 7f9d7c165700  5 _do_op 0x5ff1e60 seq 2384 osr(3.717 0x276de70)/0x276de70 start
2014-06-12 14:46:17.933603 7f9d7c165700 10 _do_transaction on 0x10192840
2014-06-12 14:46:17.933610 7f9d7c165700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.933605 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2579 active+clean] op_applied: 236247
2014-06-12 14:46:17.933644 7f9d7795c700  5 -- op tracker -- , seq: 747736, time: 2014-06-12 14:46:17.933637, event: op_applied, op: osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.933652 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2579 active+clean] op_applied on primary on version 173'2581
2014-06-12 14:46:17.933661 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2579 active+clean] repop_all_applied: repop tid 236247 all applied 
2014-06-12 14:46:17.933671 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2579 active+clean] eval_repop repgather(0x24fa7740 173'2581 rep_tid=236247 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.933687 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2579 active+clean] log_op_stats osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.017876
2014-06-12 14:46:17.933702 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2579 active+clean] publish_stats_to_osd 173:3229
2014-06-12 14:46:17.933713 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2579 active+clean]  requeue_ops 
2014-06-12 14:46:17.933724 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean]  removing repgather(0x24fa7740 173'2581 rep_tid=236247 committed?=1 applied?=1 op=osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.933739 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean]    q front is repgather(0x24fa7740 173'2581 rep_tid=236247 committed?=1 applied?=1 op=osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.933752 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] remove_repop repgather(0x24fa7740 173'2581 rep_tid=236247 committed?=1 applied?=1 op=osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.933766 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean]  requeue_ops 
2014-06-12 14:46:17.933779 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.933808 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.933811 7f9d83173700  5 -- op tracker -- , seq: 747736, time: 2014-06-12 14:46:17.933797, event: done, op: osd_op(client.4290.0:143633 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.933866 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a3d40 (1 active)
2014-06-12 14:46:17.933870 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.933880 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.933901 7f9d7c165700 15 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.933905 7f9d7c165700 10 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1 r = 0
2014-06-12 14:46:17.933911 7f9d7c165700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.934009 7f9d7c165700 15 _write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 3002368~516096
2014-06-12 14:46:17.934159 7f9d7c165700 10 lookup_strip_header done  cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.934329 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:17.936564 7f9d7c165700 10 _generic_write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 3002368~516096 = 0
2014-06-12 14:46:17.936629 7f9d7c165700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.936647 7f9d7c165700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.936653 7f9d7c165700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.936661 7f9d7c165700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.936850 7f9d7c165700 20 set_header setting 15870 cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 parent seq 0
2014-06-12 14:46:17.936918 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.936931 7f9d7c165700 20 set_header setting 2128 cid meta oid 1fe0f8ca/pglog_3.717/0//-1 parent seq 0
2014-06-12 14:46:17.936938 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:17.937357 7f9d494c9700  5 -- op tracker -- , seq: 747741, time: 2014-06-12 14:46:17.936059, event: header_read, op: osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.937371 7f9d494c9700  5 -- op tracker -- , seq: 747741, time: 2014-06-12 14:46:17.936067, event: throttled, op: osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.937379 7f9d494c9700  5 -- op tracker -- , seq: 747741, time: 2014-06-12 14:46:17.937307, event: all_read, op: osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.937387 7f9d494c9700  5 -- op tracker -- , seq: 747741, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.937395 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:46:17.937420 7f9d494c9700 15 osd.2 173 enqueue_op 0x12bf6400 prio 63 cost 516096 latency 0.001361 osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.937452 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.937458 7f9d683e1700 10 osd.2 173 dequeue_op 0x12bf6400 prio 63 cost 516096 latency 0.001399 osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean]
2014-06-12 14:46:17.937487 7f9d683e1700  5 -- op tracker -- , seq: 747741, time: 2014-06-12 14:46:17.937482, event: reached_pg, op: osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.937498 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.937524 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] handle_message: 0x12bf6400
2014-06-12 14:46:17.937533 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] do_op osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.937556 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] get_object_context: found obc in cache: 0xa039b80
2014-06-12 14:46:17.937567 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] get_object_context: 0xa039b80 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 rwstate(write n=1 w=0) oi: 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2582 client.4290.0:143634 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2582) ssc: 0x1550eee0 snapset: 0=[]:[]+head
2014-06-12 14:46:17.937581 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] find_object_context 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 @head oi=82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2582 client.4290.0:143634 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2582)
2014-06-12 14:46:17.937602 7f9d683e1700  5 -- op tracker -- , seq: 747741, time: 2014-06-12 14:46:17.937597, event: started, op: osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.937608 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] execute_ctx 0x589c400
2014-06-12 14:46:17.937653 7f9d683e1700  5 -- op tracker -- , seq: 747741, time: 2014-06-12 14:46:17.937647, event: started, op: osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.937659 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] do_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] ov 173'2582 av 173'2583 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.937672 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] do_osd_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096]
2014-06-12 14:46:17.937681 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.937692 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] do_osd_op  write 3518464~516096
2014-06-12 14:46:17.937703 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:46:17.937712 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 snapset=0x1550ef18  snapc=0=[]
2014-06-12 14:46:17.937722 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.937731 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] finish_ctx 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 0x589c400 op modify  
2014-06-12 14:46:17.937743 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean]  set mtime to 2014-06-12 08:45:20.315786
2014-06-12 14:46:17.939431 7f9d7c165700 10 _do_op 0x5ff1e60 seq 2384 r = 0, finisher 0x6d36580 0x32115940
2014-06-12 14:46:17.939448 7f9d7c165700 10 _finish_op 0x5ff1e60 seq 2384 osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.939459 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a3d40 (0 active)
2014-06-12 14:46:17.939470 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.941634 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean]  final snapset 0=[]:[]+head in 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.941673 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.941687 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] new_repop rep_tid 236249 on osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.941715 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] issue_repop rep_tid 236249 o 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.941750 7f9d683e1700  5 -- op tracker -- , seq: 747741, time: 2014-06-12 14:46:17.941744, event: waiting for subops from 27, op: osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.941781 7f9d683e1700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:17.941781 7f9d58c5a700  5 -- op tracker -- , seq: 747742, time: 2014-06-12 14:46:17.941718, event: header_read, op: osd_sub_op_reply(client.4290.0:143634 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.941792 7f9d58c5a700  5 -- op tracker -- , seq: 747742, time: 2014-06-12 14:46:17.941719, event: throttled, op: osd_sub_op_reply(client.4290.0:143634 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.941799 7f9d58c5a700  5 -- op tracker -- , seq: 747742, time: 2014-06-12 14:46:17.941740, event: all_read, op: osd_sub_op_reply(client.4290.0:143634 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.941807 7f9d58c5a700  5 -- op tracker -- , seq: 747742, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143634 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.941812 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143634 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.941804 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2582 (0'0,173'2582] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] append_log log((0'0,173'2582], crt=0'0) [173'2583 (173'2582) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143635 2014-06-12 08:45:20.315786]
2014-06-12 14:46:17.941817 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:17.941824 7f9d58c5a700 15 osd.2 173 enqueue_op 0x8dbdb00 prio 196 cost 0 latency 0.000106 osd_sub_op_reply(client.4290.0:143634 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.941827 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] add_log_entry 173'2583 (173'2582) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143635 2014-06-12 08:45:20.315786
2014-06-12 14:46:17.941866 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:17.941871 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.941906 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2583, trimmed: 
2014-06-12 14:46:17.941930 7f9d683e1700  5 queue_transactions existing osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.941936 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.941938 7f9d683e1700  5 queue_op 0x8094240 seq 2385 osr(3.717 0x276de70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:17.941952 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a3d40 (1 active)
2014-06-12 14:46:17.941960 7f9d82972700  5 _do_op 0x8094240 seq 2385 osr(3.717 0x276de70)/0x276de70 start
2014-06-12 14:46:17.941946 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] eval_repop repgather(0x5147800 173'2583 rep_tid=236249 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.941965 7f9d82972700 10 _do_transaction on 0x5146600
2014-06-12 14:46:17.941966 7f9d683e1700 10 osd.2 173 dequeue_op 0x12bf6400 finish
2014-06-12 14:46:17.941972 7f9d82972700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.941973 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:17.941976 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.941976 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] op_commit: 236248
2014-06-12 14:46:17.942005 7f9d7715b700  5 -- op tracker -- , seq: 747740, time: 2014-06-12 14:46:17.941997, event: op_commit, op: osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.942020 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] op_applied: 236248
2014-06-12 14:46:17.942037 7f9d7795c700  5 -- op tracker -- , seq: 747740, time: 2014-06-12 14:46:17.942033, event: op_applied, op: osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.942043 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] op_applied on primary on version 173'2582
2014-06-12 14:46:17.942056 7f9d65bdc700 10 osd.2 173 dequeue_op 0x8dbdb00 prio 196 cost 0 latency 0.000338 osd_sub_op_reply(client.4290.0:143634 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 pg pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean]
2014-06-12 14:46:17.942075 7f9d65bdc700  5 -- op tracker -- , seq: 747742, time: 2014-06-12 14:46:17.942072, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143634 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.942084 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] handle_message: 0x8dbdb00
2014-06-12 14:46:17.942099 7f9d65bdc700  5 -- op tracker -- , seq: 747742, time: 2014-06-12 14:46:17.942096, event: started, op: osd_sub_op_reply(client.4290.0:143634 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.942105 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] sub_op_modify_reply: tid 236248 op  ack_type 4 from 27
2014-06-12 14:46:17.942123 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.942126 7f9d65bdc700  5 -- op tracker -- , seq: 747740, time: 2014-06-12 14:46:17.942119, event: sub_op_commit_rec, op: osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.942134 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] repop_all_applied: repop tid 236248 all applied 
2014-06-12 14:46:17.942150 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.942146 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] eval_repop repgather(0x10192d80 173'2582 rep_tid=236248 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.942172 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean]  sending ack on repgather(0x10192d80 173'2582 rep_tid=236248 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x3206b400
2014-06-12 14:46:17.942213 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2581 crt=0'0 lcod 173'2580 mlcod 173'2580 active+clean] repop_all_committed: repop tid 236248 all committed 
2014-06-12 14:46:17.942223 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.942226 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2580 active+clean] eval_repop repgather(0x10192d80 173'2582 rep_tid=236248 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.942251 7f9d82972700 15 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.942257 7f9d82972700 10 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1 r = 0
2014-06-12 14:46:17.942245 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2580 active+clean] log_op_stats osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.013241 lat 0.013273
2014-06-12 14:46:17.942263 7f9d82972700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.942267 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2580 active+clean] publish_stats_to_osd 173:3230
2014-06-12 14:46:17.942278 7f9d82972700 15 _write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 3518464~516096
2014-06-12 14:46:17.942285 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2580 active+clean]  sending commit on repgather(0x10192d80 173'2582 rep_tid=236248 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x9656280
2014-06-12 14:46:17.942317 7f9d65bdc700  5 -- op tracker -- , seq: 747740, time: 2014-06-12 14:46:17.942312, event: commit_sent, op: osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.942334 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2580 active+clean]  requeue_ops 
2014-06-12 14:46:17.942346 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean]  removing repgather(0x10192d80 173'2582 rep_tid=236248 committed?=1 applied?=1 op=osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.942360 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean]    q front is repgather(0x10192d80 173'2582 rep_tid=236248 committed?=1 applied?=1 op=osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.942373 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] remove_repop repgather(0x10192d80 173'2582 rep_tid=236248 committed?=1 applied?=1 op=osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.942383 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean]  requeue_ops 
2014-06-12 14:46:17.942395 7f9d65bdc700  5 -- op tracker -- , seq: 747740, time: 2014-06-12 14:46:17.942392, event: done, op: osd_op(client.4290.0:143634 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.942415 7f9d65bdc700 10 osd.2 173 dequeue_op 0x8dbdb00 finish
2014-06-12 14:46:17.942422 7f9d65bdc700  5 -- op tracker -- , seq: 747742, time: 2014-06-12 14:46:17.942419, event: done, op: osd_sub_op_reply(client.4290.0:143634 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.942431 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.942434 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.942445 7f9d82972700 10 lookup_strip_header done  cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.942470 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:17.944360 7f9d494c9700  5 -- op tracker -- , seq: 747743, time: 2014-06-12 14:46:17.944075, event: header_read, op: osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.944376 7f9d494c9700  5 -- op tracker -- , seq: 747743, time: 2014-06-12 14:46:17.944080, event: throttled, op: osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.944386 7f9d494c9700  5 -- op tracker -- , seq: 747743, time: 2014-06-12 14:46:17.944318, event: all_read, op: osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.944396 7f9d494c9700  5 -- op tracker -- , seq: 747743, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.944406 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:46:17.944438 7f9d494c9700 15 osd.2 173 enqueue_op 0x6d10b00 prio 63 cost 159744 latency 0.000363 osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.944487 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.944496 7f9d64bda700 10 osd.2 173 dequeue_op 0x6d10b00 prio 63 cost 159744 latency 0.000421 osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean]
2014-06-12 14:46:17.944525 7f9d64bda700  5 -- op tracker -- , seq: 747743, time: 2014-06-12 14:46:17.944519, event: reached_pg, op: osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.944534 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:17.944552 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] handle_message: 0x6d10b00
2014-06-12 14:46:17.944564 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] do_op osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:17.944592 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] get_object_context: found obc in cache: 0xa039b80
2014-06-12 14:46:17.944605 7f9d82972700 10 _generic_write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 3518464~516096 = 0
2014-06-12 14:46:17.944604 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] get_object_context: 0xa039b80 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 rwstate(write n=1 w=0) oi: 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2583 client.4290.0:143635 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2583) ssc: 0x1550eee0 snapset: 0=[]:[]+head
2014-06-12 14:46:17.944627 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] find_object_context 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 @head oi=82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3(173'2583 client.4290.0:143635 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2583)
2014-06-12 14:46:17.944655 7f9d82972700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.944657 7f9d64bda700  5 -- op tracker -- , seq: 747743, time: 2014-06-12 14:46:17.944651, event: started, op: osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.944664 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] execute_ctx 0x16af5600
2014-06-12 14:46:17.944673 7f9d82972700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.944682 7f9d82972700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.944683 7f9d64bda700  5 -- op tracker -- , seq: 747743, time: 2014-06-12 14:46:17.944677, event: started, op: osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.944689 7f9d82972700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.944690 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] do_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] ov 173'2583 av 173'2584 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:17.944706 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] do_osd_op 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744]
2014-06-12 14:46:17.944718 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:17.944733 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] do_osd_op  write 4034560~159744
2014-06-12 14:46:17.944750 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:46:17.944761 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 snapset=0x1550ef18  snapc=0=[]
2014-06-12 14:46:17.944777 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] make_writeable 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:17.944790 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] finish_ctx 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 0x16af5600 op modify  
2014-06-12 14:46:17.944804 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean]  set mtime to 2014-06-12 08:45:20.324606
2014-06-12 14:46:17.944828 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean]  final snapset 0=[]:[]+head in 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.944850 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean]  zeroing write result code 0
2014-06-12 14:46:17.944864 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] new_repop rep_tid 236250 on osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:17.944883 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] issue_repop rep_tid 236250 o 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.944900 7f9d82972700 20 set_header setting 15870 cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 parent seq 0
2014-06-12 14:46:17.944914 7f9d64bda700  5 -- op tracker -- , seq: 747743, time: 2014-06-12 14:46:17.944908, event: waiting for subops from 27, op: osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.944939 7f9d64bda700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:17.944965 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.944963 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2583 (0'0,173'2583] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] append_log log((0'0,173'2583], crt=0'0) [173'2584 (173'2583) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143636 2014-06-12 08:45:20.324606]
2014-06-12 14:46:17.944983 7f9d82972700 20 set_header setting 2128 cid meta oid 1fe0f8ca/pglog_3.717/0//-1 parent seq 0
2014-06-12 14:46:17.944992 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:17.944986 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] add_log_entry 173'2584 (173'2583) modify   82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 by client.4290.0:143636 2014-06-12 08:45:20.324606
2014-06-12 14:46:17.945005 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] append_log  adding 1 keys
2014-06-12 14:46:17.945034 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2584, trimmed: 
2014-06-12 14:46:17.945054 7f9d64bda700  5 queue_transactions existing osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.945059 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:17.945060 7f9d64bda700  5 queue_op 0x6d2ed80 seq 2386 osr(3.717 0x276de70) 162037 bytes   (queue has 2 ops and 680426 bytes)
2014-06-12 14:46:17.945069 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] eval_repop repgather(0x34d8f680 173'2584 rep_tid=236250 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.945089 7f9d64bda700 10 osd.2 173 dequeue_op 0x6d10b00 finish
2014-06-12 14:46:17.945093 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.945096 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.945634 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a3d40 (2 active)
2014-06-12 14:46:17.947523 7f9d82972700 10 _do_op 0x8094240 seq 2385 r = 0, finisher 0x31d0f8a0 0x2869f500
2014-06-12 14:46:17.947538 7f9d82972700 10 _finish_op 0x8094240 seq 2385 osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.947549 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a3d40 (1 active)
2014-06-12 14:46:17.947552 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.947542 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] op_commit: 236249
2014-06-12 14:46:17.947572 7f9d7715b700  5 -- op tracker -- , seq: 747741, time: 2014-06-12 14:46:17.947562, event: op_commit, op: osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.947588 7f9d7c966700  5 _do_op 0x6d2ed80 seq 2386 osr(3.717 0x276de70)/0x276de70 start
2014-06-12 14:46:17.947593 7f9d7c966700 10 _do_transaction on 0x31b64900
2014-06-12 14:46:17.947601 7f9d7c966700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.947587 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] op_applied: 236249
2014-06-12 14:46:17.947612 7f9d7795c700  5 -- op tracker -- , seq: 747741, time: 2014-06-12 14:46:17.947605, event: op_applied, op: osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.947624 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] op_applied on primary on version 173'2583
2014-06-12 14:46:17.947701 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.947726 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:17.947762 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:17.947777 7f9d7c966700 15 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.947782 7f9d7c966700 10 _omap_rmkeys meta/1fe0f8ca/pglog_3.717/0//-1 r = 0
2014-06-12 14:46:17.947786 7f9d7c966700 15 _omap_setkeys meta/1fe0f8ca/pglog_3.717/0//-1
2014-06-12 14:46:17.947859 7f9d7c966700 15 _write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 4034560~159744
2014-06-12 14:46:17.947953 7f9d7c966700 10 lookup_strip_header done  cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.948065 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:17.948439 7f9d7c966700 10 _generic_write 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 4034560~159744 = 0
2014-06-12 14:46:17.948455 7f9d7c966700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.948468 7f9d7c966700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.948473 7f9d7c966700 15 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3
2014-06-12 14:46:17.948478 7f9d7c966700 10 _setattrs 3.717_head/82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 = 0
2014-06-12 14:46:17.948560 7f9d7c966700 20 set_header setting 15870 cid 3.717_head oid 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 parent seq 0
2014-06-12 14:46:17.948581 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:17.948589 7f9d7c966700 20 set_header setting 2128 cid meta oid 1fe0f8ca/pglog_3.717/0//-1 parent seq 0
2014-06-12 14:46:17.948595 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:17.949131 7f9d7c966700 10 _do_op 0x6d2ed80 seq 2386 r = 0, finisher 0x7b36a80 0x3b720c0
2014-06-12 14:46:17.949139 7f9d7c966700 10 _finish_op 0x6d2ed80 seq 2386 osr(3.717 0x276de70)/0x276de70
2014-06-12 14:46:17.949147 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a3d40 (0 active)
2014-06-12 14:46:17.949151 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:17.949144 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] op_commit: 236250
2014-06-12 14:46:17.949166 7f9d7715b700  5 -- op tracker -- , seq: 747743, time: 2014-06-12 14:46:17.949160, event: op_commit, op: osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.949178 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] op_applied: 236250
2014-06-12 14:46:17.949199 7f9d7795c700  5 -- op tracker -- , seq: 747743, time: 2014-06-12 14:46:17.949194, event: op_applied, op: osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.949208 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] op_applied on primary on version 173'2584
2014-06-12 14:46:17.949965 7f9d58c5a700  5 -- op tracker -- , seq: 747744, time: 2014-06-12 14:46:17.949919, event: header_read, op: osd_sub_op_reply(client.4290.0:143635 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.949972 7f9d58c5a700  5 -- op tracker -- , seq: 747744, time: 2014-06-12 14:46:17.949921, event: throttled, op: osd_sub_op_reply(client.4290.0:143635 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.949987 7f9d58c5a700  5 -- op tracker -- , seq: 747744, time: 2014-06-12 14:46:17.949939, event: all_read, op: osd_sub_op_reply(client.4290.0:143635 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.949991 7f9d58c5a700  5 -- op tracker -- , seq: 747744, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143635 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.949995 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143635 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.949999 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:17.950005 7f9d58c5a700 15 osd.2 173 enqueue_op 0x8dbd300 prio 196 cost 0 latency 0.000086 osd_sub_op_reply(client.4290.0:143635 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.950032 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.950042 7f9d6abe6700 10 osd.2 173 dequeue_op 0x8dbd300 prio 196 cost 0 latency 0.000123 osd_sub_op_reply(client.4290.0:143635 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 pg pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean]
2014-06-12 14:46:17.950065 7f9d6abe6700  5 -- op tracker -- , seq: 747744, time: 2014-06-12 14:46:17.950061, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143635 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.950074 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] handle_message: 0x8dbd300
2014-06-12 14:46:17.950090 7f9d6abe6700  5 -- op tracker -- , seq: 747744, time: 2014-06-12 14:46:17.950087, event: started, op: osd_sub_op_reply(client.4290.0:143635 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.950097 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] sub_op_modify_reply: tid 236249 op  ack_type 4 from 27
2014-06-12 14:46:17.950117 7f9d6abe6700  5 -- op tracker -- , seq: 747741, time: 2014-06-12 14:46:17.950110, event: sub_op_commit_rec, op: osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.950124 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] repop_all_applied: repop tid 236249 all applied 
2014-06-12 14:46:17.950136 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] eval_repop repgather(0x5147800 173'2583 rep_tid=236249 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.950159 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean]  sending ack on repgather(0x5147800 173'2583 rep_tid=236249 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x22f29680
2014-06-12 14:46:17.950195 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2582 crt=0'0 lcod 173'2581 mlcod 173'2581 active+clean] repop_all_committed: repop tid 236249 all committed 
2014-06-12 14:46:17.950205 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2581 active+clean] eval_repop repgather(0x5147800 173'2583 rep_tid=236249 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.950224 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2581 active+clean] log_op_stats osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.014134 lat 0.014164
2014-06-12 14:46:17.950246 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2581 active+clean] publish_stats_to_osd 173:3231
2014-06-12 14:46:17.950263 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2581 active+clean]  sending commit on repgather(0x5147800 173'2583 rep_tid=236249 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x2f89ca00
2014-06-12 14:46:17.950300 7f9d6abe6700  5 -- op tracker -- , seq: 747741, time: 2014-06-12 14:46:17.950294, event: commit_sent, op: osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.950307 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2581 active+clean]  requeue_ops 
2014-06-12 14:46:17.950319 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2582 active+clean]  removing repgather(0x5147800 173'2583 rep_tid=236249 committed?=1 applied?=1 op=osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.950335 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2582 active+clean]    q front is repgather(0x5147800 173'2583 rep_tid=236249 committed?=1 applied?=1 op=osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.950352 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2582 active+clean] remove_repop repgather(0x5147800 173'2583 rep_tid=236249 committed?=1 applied?=1 op=osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.950369 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2582 active+clean]  requeue_ops 
2014-06-12 14:46:17.950390 7f9d6abe6700  5 -- op tracker -- , seq: 747741, time: 2014-06-12 14:46:17.950384, event: done, op: osd_op(client.4290.0:143635 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 3518464~516096] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.950410 7f9d6abe6700 10 osd.2 173 dequeue_op 0x8dbd300 finish
2014-06-12 14:46:17.950420 7f9d6abe6700  5 -- op tracker -- , seq: 747744, time: 2014-06-12 14:46:17.950416, event: done, op: osd_sub_op_reply(client.4290.0:143635 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.950431 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.950434 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.951638 7f9d58c5a700  5 -- op tracker -- , seq: 747745, time: 2014-06-12 14:46:17.951600, event: header_read, op: osd_sub_op_reply(client.4290.0:143636 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.951647 7f9d58c5a700  5 -- op tracker -- , seq: 747745, time: 2014-06-12 14:46:17.951601, event: throttled, op: osd_sub_op_reply(client.4290.0:143636 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.951655 7f9d58c5a700  5 -- op tracker -- , seq: 747745, time: 2014-06-12 14:46:17.951612, event: all_read, op: osd_sub_op_reply(client.4290.0:143636 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.951668 7f9d58c5a700  5 -- op tracker -- , seq: 747745, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143636 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.951671 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143636 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.951674 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:17.951678 7f9d58c5a700 15 osd.2 173 enqueue_op 0x8dbca00 prio 196 cost 0 latency 0.000078 osd_sub_op_reply(client.4290.0:143636 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.951696 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.951703 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x8dbca00 prio 196 cost 0 latency 0.000102 osd_sub_op_reply(client.4290.0:143636 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0) v2 pg pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2582 active+clean]
2014-06-12 14:46:17.951722 7f9d6a3e5700  5 -- op tracker -- , seq: 747745, time: 2014-06-12 14:46:17.951718, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143636 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.951729 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2582 active+clean] handle_message: 0x8dbca00
2014-06-12 14:46:17.951743 7f9d6a3e5700  5 -- op tracker -- , seq: 747745, time: 2014-06-12 14:46:17.951740, event: started, op: osd_sub_op_reply(client.4290.0:143636 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.951748 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2582 active+clean] sub_op_modify_reply: tid 236250 op  ack_type 4 from 27
2014-06-12 14:46:17.951765 7f9d6a3e5700  5 -- op tracker -- , seq: 747743, time: 2014-06-12 14:46:17.951759, event: sub_op_commit_rec, op: osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.951770 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2582 active+clean] repop_all_applied: repop tid 236250 all applied 
2014-06-12 14:46:17.951779 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2582 active+clean] eval_repop repgather(0x34d8f680 173'2584 rep_tid=236250 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.951797 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2582 active+clean]  sending ack on repgather(0x34d8f680 173'2584 rep_tid=236250 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x1b40a280
2014-06-12 14:46:17.951821 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2583 crt=0'0 lcod 173'2582 mlcod 173'2582 active+clean] repop_all_committed: repop tid 236250 all committed 
2014-06-12 14:46:17.951831 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2582 active+clean] eval_repop repgather(0x34d8f680 173'2584 rep_tid=236250 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.951844 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2582 active+clean] log_op_stats osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4 inb 160023 outb 0 rlat 0.007745 lat 0.007768
2014-06-12 14:46:17.951857 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2582 active+clean] publish_stats_to_osd 173:3232
2014-06-12 14:46:17.951867 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2582 active+clean]  sending commit on repgather(0x34d8f680 173'2584 rep_tid=236250 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4) 0x5729680
2014-06-12 14:46:17.951895 7f9d6a3e5700  5 -- op tracker -- , seq: 747743, time: 2014-06-12 14:46:17.951890, event: commit_sent, op: osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.951901 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2582 active+clean]  requeue_ops 
2014-06-12 14:46:17.951910 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2583 active+clean]  removing repgather(0x34d8f680 173'2584 rep_tid=236250 committed?=1 applied?=1 op=osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.951922 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2583 active+clean]    q front is repgather(0x34d8f680 173'2584 rep_tid=236250 committed?=1 applied?=1 op=osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.951934 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2583 active+clean] remove_repop repgather(0x34d8f680 173'2584 rep_tid=236250 committed?=1 applied?=1 op=osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.951945 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2583 active+clean]  requeue_ops 
2014-06-12 14:46:17.951968 7f9d6a3e5700  5 -- op tracker -- , seq: 747743, time: 2014-06-12 14:46:17.951964, event: done, op: osd_op(client.4290.0:143636 rbd_data.10996b8b4567.0000000000001027 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.82eb5f17 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.952038 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x8dbca00 finish
2014-06-12 14:46:17.952045 7f9d6a3e5700  5 -- op tracker -- , seq: 747745, time: 2014-06-12 14:46:17.952041, event: done, op: osd_sub_op_reply(client.4290.0:143636 3.717 82eb5f17/rbd_data.10996b8b4567.0000000000001027/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.952054 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.952057 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:17.962534 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:17.962627 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:17.988799 7f9d56432700  5 -- op tracker -- , seq: 747746, time: 2014-06-12 14:46:17.988739, event: header_read, op: osd_sub_op_reply(client.4303.0:144627 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.988812 7f9d56432700  5 -- op tracker -- , seq: 747746, time: 2014-06-12 14:46:17.988742, event: throttled, op: osd_sub_op_reply(client.4303.0:144627 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.988821 7f9d56432700  5 -- op tracker -- , seq: 747746, time: 2014-06-12 14:46:17.988753, event: all_read, op: osd_sub_op_reply(client.4303.0:144627 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.988829 7f9d56432700  5 -- op tracker -- , seq: 747746, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4303.0:144627 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.988834 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4303.0:144627 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:17.988840 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:17.988849 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3eb200 prio 196 cost 0 latency 0.000109 osd_sub_op_reply(client.4303.0:144627 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:17.988877 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:17.988886 7f9d66bde700 10 osd.2 173 dequeue_op 0x1e3eb200 prio 196 cost 0 latency 0.000147 osd_sub_op_reply(client.4303.0:144627 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3092 active+clean]
2014-06-12 14:46:17.988910 7f9d66bde700  5 -- op tracker -- , seq: 747746, time: 2014-06-12 14:46:17.988907, event: reached_pg, op: osd_sub_op_reply(client.4303.0:144627 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.988920 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3092 active+clean] handle_message: 0x1e3eb200
2014-06-12 14:46:17.988936 7f9d66bde700  5 -- op tracker -- , seq: 747746, time: 2014-06-12 14:46:17.988933, event: started, op: osd_sub_op_reply(client.4303.0:144627 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.988943 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3092 active+clean] sub_op_modify_reply: tid 236066 op  ack_type 4 from 26
2014-06-12 14:46:17.988965 7f9d66bde700  5 -- op tracker -- , seq: 747085, time: 2014-06-12 14:46:17.988956, event: sub_op_commit_rec, op: osd_op(client.4303.0:144627 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.988976 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3092 active+clean] repop_all_applied: repop tid 236066 all applied 
2014-06-12 14:46:17.988987 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3092 active+clean] eval_repop repgather(0x2c3c7bc0 173'3094 rep_tid=236066 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144627 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.989002 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3092 active+clean]  sending ack on repgather(0x2c3c7bc0 173'3094 rep_tid=236066 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144627 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x20270f00
2014-06-12 14:46:17.989035 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3093 crt=0'0 lcod 173'3092 mlcod 173'3092 active+clean] repop_all_committed: repop tid 236066 all committed 
2014-06-12 14:46:17.989047 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3092 active+clean] eval_repop repgather(0x2c3c7bc0 173'3094 rep_tid=236066 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144627 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:17.989072 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3092 active+clean] log_op_stats osd_op(client.4303.0:144627 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 38.633371 lat 38.633409
2014-06-12 14:46:17.989090 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3092 active+clean] publish_stats_to_osd 173:3560
2014-06-12 14:46:17.989108 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3092 active+clean]  sending commit on repgather(0x2c3c7bc0 173'3094 rep_tid=236066 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144627 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x17f01400
2014-06-12 14:46:17.989140 7f9d66bde700  5 -- op tracker -- , seq: 747085, time: 2014-06-12 14:46:17.989134, event: commit_sent, op: osd_op(client.4303.0:144627 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.989149 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3092 active+clean]  requeue_ops 
2014-06-12 14:46:17.989159 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3093 active+clean]  removing repgather(0x2c3c7bc0 173'3094 rep_tid=236066 committed?=1 applied?=1 op=osd_op(client.4303.0:144627 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.989175 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3093 active+clean]    q front is repgather(0x2c3c7bc0 173'3094 rep_tid=236066 committed?=1 applied?=1 op=osd_op(client.4303.0:144627 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.989191 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3093 active+clean] remove_repop repgather(0x2c3c7bc0 173'3094 rep_tid=236066 committed?=1 applied?=1 op=osd_op(client.4303.0:144627 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:17.989208 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3093 active+clean]  requeue_ops 
2014-06-12 14:46:17.989229 7f9d66bde700  5 -- op tracker -- , seq: 747085, time: 2014-06-12 14:46:17.989224, event: done, op: osd_op(client.4303.0:144627 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:17.989250 7f9d66bde700 10 osd.2 173 dequeue_op 0x1e3eb200 finish
2014-06-12 14:46:17.989259 7f9d66bde700  5 -- op tracker -- , seq: 747746, time: 2014-06-12 14:46:17.989255, event: done, op: osd_sub_op_reply(client.4303.0:144627 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:17.989269 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:17.989272 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.039117 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:18.040423 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:18.080564 7f9d5703e700  5 -- op tracker -- , seq: 747747, time: 2014-06-12 14:46:18.080357, event: header_read, op: osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.080583 7f9d5703e700  5 -- op tracker -- , seq: 747747, time: 2014-06-12 14:46:18.080360, event: throttled, op: osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.080605 7f9d5703e700  5 -- op tracker -- , seq: 747747, time: 2014-06-12 14:46:18.080500, event: all_read, op: osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.080614 7f9d5703e700  5 -- op tracker -- , seq: 747747, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.080621 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.080629 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:18.080640 7f9d5703e700 15 osd.2 173 enqueue_op 0x31d61d00 prio 127 cost 21287 latency 0.000282 osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.080677 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.080692 7f9d69be4700 10 osd.2 173 dequeue_op 0x31d61d00 prio 127 cost 21287 latency 0.000334 osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2827 active]
2014-06-12 14:46:18.080722 7f9d69be4700  5 -- op tracker -- , seq: 747747, time: 2014-06-12 14:46:18.080718, event: reached_pg, op: osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.080737 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2827 active] handle_message: 0x31d61d00
2014-06-12 14:46:18.080753 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2827 active] sub_op_modify trans feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 v 173'2829 (transaction) 174
2014-06-12 14:46:18.080772 7f9d69be4700  5 -- op tracker -- , seq: 747747, time: 2014-06-12 14:46:18.080768, event: started, op: osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.080792 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2828 (0'0,173'2828] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2827 active] append_log log((0'0,173'2828], crt=0'0) [173'2829 (173'878) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144764 2014-06-12 08:45:20.444056]
2014-06-12 14:46:18.080820 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2829 (0'0,173'2829] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2827 active] add_log_entry 173'2829 (173'878) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144764 2014-06-12 08:45:20.444056
2014-06-12 14:46:18.080844 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2829 (0'0,173'2829] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2827 active] append_log  adding 1 keys
2014-06-12 14:46:18.080882 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2829, trimmed: 
2014-06-12 14:46:18.080909 7f9d69be4700  5 -- op tracker -- , seq: 747747, time: 2014-06-12 14:46:18.080903, event: started, op: osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.080921 7f9d69be4700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.080927 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.080929 7f9d69be4700  5 queue_op 0x6957500 seq 2584 osr(3.39f 0x276d720) 22773 bytes   (queue has 1 ops and 22773 bytes)
2014-06-12 14:46:18.080943 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (1 active)
2014-06-12 14:46:18.080939 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2829 (0'0,173'2829] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2827 active] do_sub_op osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.080951 7f9d81970700  5 _do_op 0x6957500 seq 2584 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:46:18.080954 7f9d69be4700 10 osd.2 173 dequeue_op 0x31d61d00 finish
2014-06-12 14:46:18.080958 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.080958 7f9d81970700 10 _do_transaction on 0x286f3970
2014-06-12 14:46:18.080961 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.080969 7f9d81970700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.081150 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.081176 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.081234 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.081345 7f9d81970700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.081356 7f9d81970700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:46:18.081363 7f9d81970700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.081381 7f9d81970700 15 _write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 0~20480
2014-06-12 14:46:18.081608 7f9d81970700 10 lookup_strip_header done  cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.081802 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:18.081888 7f9d81970700 10 _generic_write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 0~20480 = 0
2014-06-12 14:46:18.081900 7f9d81970700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.081910 7f9d81970700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.081915 7f9d81970700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.081922 7f9d81970700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.082027 7f9d81970700 20 set_header setting 17505 cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 parent seq 0
2014-06-12 14:46:18.082049 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.082065 7f9d81970700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:46:18.082075 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:18.082218 7f9d81970700 10 _do_op 0x6957500 seq 2584 r = 0, finisher 0x693aa20 0
2014-06-12 14:46:18.082230 7f9d81970700 10 _finish_op 0x6957500 seq 2584 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.082238 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (0 active)
2014-06-12 14:46:18.082241 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.082244 7f9d7715b700  5 -- op tracker -- , seq: 747747, time: 2014-06-12 14:46:18.082236, event: commit_sent, op: osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.082252 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2829 (0'0,173'2829] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2827 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:18.082275 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:18.082313 7f9d7795c700  5 -- op tracker -- , seq: 747747, time: 2014-06-12 14:46:18.082306, event: sub_op_applied, op: osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.082322 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2829 (0'0,173'2829] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] sub_op_modify_applied on 0x286f3880 op osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.082338 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2829 (0'0,173'2829] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] op_applied on primary on version 173'2829
2014-06-12 14:46:18.082348 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2829 (0'0,173'2829] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] op_applied on replica on version 173'2829
2014-06-12 14:46:18.082378 7f9d7795c700  5 -- op tracker -- , seq: 747747, time: 2014-06-12 14:46:18.082374, event: done, op: osd_sub_op(client.4303.0:144764 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2829 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.095408 7f9d5703e700  5 -- op tracker -- , seq: 747748, time: 2014-06-12 14:46:18.089660, event: header_read, op: osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.095421 7f9d5703e700  5 -- op tracker -- , seq: 747748, time: 2014-06-12 14:46:18.089662, event: throttled, op: osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.095431 7f9d5703e700  5 -- op tracker -- , seq: 747748, time: 2014-06-12 14:46:18.095362, event: all_read, op: osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.095440 7f9d5703e700  5 -- op tracker -- , seq: 747748, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.095446 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.095453 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:18.095461 7f9d5703e700 15 osd.2 173 enqueue_op 0x31d60b00 prio 127 cost 1032999 latency 0.005801 osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.095491 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.095500 7f9d653db700 10 osd.2 173 dequeue_op 0x31d60b00 prio 127 cost 1032999 latency 0.005839 osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2829 (0'0,173'2829] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active]
2014-06-12 14:46:18.095525 7f9d653db700  5 -- op tracker -- , seq: 747748, time: 2014-06-12 14:46:18.095521, event: reached_pg, op: osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.095535 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2829 (0'0,173'2829] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] handle_message: 0x31d60b00
2014-06-12 14:46:18.095546 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2829 (0'0,173'2829] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] sub_op_modify trans feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 v 173'2830 (transaction) 174
2014-06-12 14:46:18.095561 7f9d653db700  5 -- op tracker -- , seq: 747748, time: 2014-06-12 14:46:18.095558, event: started, op: osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.095578 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2829 (0'0,173'2829] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] append_log log((0'0,173'2829], crt=0'0) [173'2830 (173'2829) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144765 2014-06-12 08:45:20.463210]
2014-06-12 14:46:18.095601 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2830 (0'0,173'2830] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] add_log_entry 173'2830 (173'2829) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144765 2014-06-12 08:45:20.463210
2014-06-12 14:46:18.095620 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2830 (0'0,173'2830] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] append_log  adding 1 keys
2014-06-12 14:46:18.095650 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2830, trimmed: 
2014-06-12 14:46:18.095671 7f9d653db700  5 -- op tracker -- , seq: 747748, time: 2014-06-12 14:46:18.095667, event: started, op: osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.095680 7f9d653db700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.095684 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.095686 7f9d653db700  5 queue_op 0x90acde0 seq 2585 osr(3.39f 0x276d720) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:18.095698 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (1 active)
2014-06-12 14:46:18.095694 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2830 (0'0,173'2830] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] do_sub_op osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.095705 7f9d7a161700  5 _do_op 0x90acde0 seq 2585 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:46:18.095707 7f9d653db700 10 osd.2 173 dequeue_op 0x31d60b00 finish
2014-06-12 14:46:18.095710 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.095710 7f9d7a161700 10 _do_transaction on 0x29eb2b70
2014-06-12 14:46:18.095712 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.095716 7f9d7a161700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.096314 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.096345 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.096415 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.096433 7f9d7a161700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.096440 7f9d7a161700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:46:18.096445 7f9d7a161700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.096585 7f9d7a161700 15 _write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 20480~1032192
2014-06-12 14:46:18.096752 7f9d7a161700 10 lookup_strip_header done  cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.096893 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:18.097304 7f9d5703e700  5 -- op tracker -- , seq: 747749, time: 2014-06-12 14:46:18.095489, event: header_read, op: osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.097315 7f9d5703e700  5 -- op tracker -- , seq: 747749, time: 2014-06-12 14:46:18.095490, event: throttled, op: osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.097325 7f9d5703e700  5 -- op tracker -- , seq: 747749, time: 2014-06-12 14:46:18.097272, event: all_read, op: osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.097338 7f9d5703e700  5 -- op tracker -- , seq: 747749, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.097344 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.097352 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:18.097361 7f9d5703e700 15 osd.2 173 enqueue_op 0x31d61f00 prio 127 cost 516903 latency 0.001871 osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.097396 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.097404 7f9d63bd8700 10 osd.2 173 dequeue_op 0x31d61f00 prio 127 cost 516903 latency 0.001914 osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2830 (0'0,173'2830] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active]
2014-06-12 14:46:18.097431 7f9d63bd8700  5 -- op tracker -- , seq: 747749, time: 2014-06-12 14:46:18.097427, event: reached_pg, op: osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.097440 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2830 (0'0,173'2830] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] handle_message: 0x31d61f00
2014-06-12 14:46:18.097453 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2830 (0'0,173'2830] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] sub_op_modify trans feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 v 173'2831 (transaction) 174
2014-06-12 14:46:18.097473 7f9d63bd8700  5 -- op tracker -- , seq: 747749, time: 2014-06-12 14:46:18.097468, event: started, op: osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.097489 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2830 (0'0,173'2830] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] append_log log((0'0,173'2830], crt=0'0) [173'2831 (173'2830) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144766 2014-06-12 08:45:20.467596]
2014-06-12 14:46:18.097510 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2831 (0'0,173'2831] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] add_log_entry 173'2831 (173'2830) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144766 2014-06-12 08:45:20.467596
2014-06-12 14:46:18.097535 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2831 (0'0,173'2831] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] append_log  adding 1 keys
2014-06-12 14:46:18.097595 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2831, trimmed: 
2014-06-12 14:46:18.097629 7f9d63bd8700  5 -- op tracker -- , seq: 747749, time: 2014-06-12 14:46:18.097624, event: started, op: osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.097647 7f9d63bd8700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.097651 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.097653 7f9d63bd8700  5 queue_op 0x2728b620 seq 2586 osr(3.39f 0x276d720) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:46:18.097661 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2831 (0'0,173'2831] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] do_sub_op osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.097677 7f9d63bd8700 10 osd.2 173 dequeue_op 0x31d61f00 finish
2014-06-12 14:46:18.097681 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.097684 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.099200 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (2 active)
2014-06-12 14:46:18.101450 7f9d7a161700 10 _generic_write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 20480~1032192 = 0
2014-06-12 14:46:18.101578 7f9d7a161700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.101593 7f9d7a161700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.101599 7f9d7a161700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.101606 7f9d7a161700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.102073 7f9d7a161700 20 set_header setting 17505 cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 parent seq 0
2014-06-12 14:46:18.102112 7f9d5703e700  5 -- op tracker -- , seq: 747750, time: 2014-06-12 14:46:18.100645, event: header_read, op: osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.102125 7f9d5703e700  5 -- op tracker -- , seq: 747750, time: 2014-06-12 14:46:18.100647, event: throttled, op: osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.102133 7f9d5703e700  5 -- op tracker -- , seq: 747750, time: 2014-06-12 14:46:18.102072, event: all_read, op: osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.102142 7f9d5703e700  5 -- op tracker -- , seq: 747750, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.102149 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.102155 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:18.102165 7f9d5703e700 15 osd.2 173 enqueue_op 0x31d60f00 prio 127 cost 516903 latency 0.001520 osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.102196 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.102212 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.102206 7f9d68be2700 10 osd.2 173 dequeue_op 0x31d60f00 prio 127 cost 516903 latency 0.001560 osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2831 (0'0,173'2831] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active]
2014-06-12 14:46:18.102226 7f9d7a161700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:46:18.102235 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:18.102231 7f9d68be2700  5 -- op tracker -- , seq: 747750, time: 2014-06-12 14:46:18.102227, event: reached_pg, op: osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.102242 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2831 (0'0,173'2831] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] handle_message: 0x31d60f00
2014-06-12 14:46:18.102255 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2831 (0'0,173'2831] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] sub_op_modify trans feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 v 173'2832 (transaction) 174
2014-06-12 14:46:18.102274 7f9d68be2700  5 -- op tracker -- , seq: 747750, time: 2014-06-12 14:46:18.102270, event: started, op: osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.102296 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2831 (0'0,173'2831] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] append_log log((0'0,173'2831], crt=0'0) [173'2832 (173'2831) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144767 2014-06-12 08:45:20.477086]
2014-06-12 14:46:18.102328 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] add_log_entry 173'2832 (173'2831) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144767 2014-06-12 08:45:20.477086
2014-06-12 14:46:18.102345 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] append_log  adding 1 keys
2014-06-12 14:46:18.102373 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2832, trimmed: 
2014-06-12 14:46:18.102393 7f9d68be2700  5 -- op tracker -- , seq: 747750, time: 2014-06-12 14:46:18.102389, event: started, op: osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.102401 7f9d68be2700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.102405 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.102407 7f9d68be2700  5 queue_op 0x774b440 seq 2587 osr(3.39f 0x276d720) 518389 bytes   (queue has 3 ops and 2071263 bytes)
2014-06-12 14:46:18.102418 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (3 active)
2014-06-12 14:46:18.102413 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] do_sub_op osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.102430 7f9d68be2700 10 osd.2 173 dequeue_op 0x31d60f00 finish
2014-06-12 14:46:18.102434 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.102436 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.110271 7f9d7a161700 10 _do_op 0x90acde0 seq 2585 r = 0, finisher 0x69700e0 0
2014-06-12 14:46:18.110296 7f9d7a161700 10 _finish_op 0x90acde0 seq 2585 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.110307 7f9d7a962700  5 _do_op 0x2728b620 seq 2586 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:46:18.110312 7f9d7a962700 10 _do_transaction on 0x13ec0d30
2014-06-12 14:46:18.110320 7f9d7a962700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.110317 7f9d7715b700  5 -- op tracker -- , seq: 747748, time: 2014-06-12 14:46:18.110305, event: commit_sent, op: osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.110326 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2828 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:18.110355 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:18.110476 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.110504 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.110568 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.110598 7f9d7a962700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.110605 7f9d7a962700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:46:18.110612 7f9d7a962700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.110742 7f9d7a962700 15 _write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 1052672~516096
2014-06-12 14:46:18.110911 7f9d7a962700 10 lookup_strip_header done  cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.111058 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:18.113601 7f9d7a962700 10 _generic_write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 1052672~516096 = 0
2014-06-12 14:46:18.113684 7f9d7a962700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.113703 7f9d7a962700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.113725 7f9d7a962700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.113734 7f9d7a962700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.113960 7f9d7a962700 20 set_header setting 17505 cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 parent seq 0
2014-06-12 14:46:18.114053 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.114072 7f9d7a962700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:46:18.114086 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:18.114858 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:18.114925 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:18.115441 7f9d5a16f700  5 -- op tracker -- , seq: 747751, time: 2014-06-12 14:46:18.109747, event: header_read, op: osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.115454 7f9d5a16f700  5 -- op tracker -- , seq: 747751, time: 2014-06-12 14:46:18.109749, event: throttled, op: osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.115462 7f9d5a16f700  5 -- op tracker -- , seq: 747751, time: 2014-06-12 14:46:18.115406, event: all_read, op: osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.115474 7f9d5a16f700  5 -- op tracker -- , seq: 747751, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.115479 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.115485 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:18.115492 7f9d5a16f700 15 osd.2 173 enqueue_op 0x59dce00 prio 127 cost 1032999 latency 0.005745 osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.115522 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.115530 7f9d633d7700 10 osd.2 173 dequeue_op 0x59dce00 prio 127 cost 1032999 latency 0.005782 osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2660 (0'0,173'2660] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active]
2014-06-12 14:46:18.115555 7f9d633d7700  5 -- op tracker -- , seq: 747751, time: 2014-06-12 14:46:18.115551, event: reached_pg, op: osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.115567 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2660 (0'0,173'2660] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active] handle_message: 0x59dce00
2014-06-12 14:46:18.115580 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2660 (0'0,173'2660] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active] sub_op_modify trans 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 v 173'2661 (transaction) 174
2014-06-12 14:46:18.115598 7f9d633d7700  5 -- op tracker -- , seq: 747751, time: 2014-06-12 14:46:18.115593, event: started, op: osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.115621 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2660 (0'0,173'2660] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active] append_log log((0'0,173'2660], crt=0'0) [173'2661 (173'2659) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141065 2014-06-12 08:43:33.288028]
2014-06-12 14:46:18.115648 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2661 (0'0,173'2661] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active] add_log_entry 173'2661 (173'2659) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141065 2014-06-12 08:43:33.288028
2014-06-12 14:46:18.115672 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2661 (0'0,173'2661] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active] append_log  adding 1 keys
2014-06-12 14:46:18.115709 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2661, trimmed: 
2014-06-12 14:46:18.115736 7f9d633d7700  5 -- op tracker -- , seq: 747751, time: 2014-06-12 14:46:18.115731, event: started, op: osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.115748 7f9d633d7700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:18.115752 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.115754 7f9d633d7700  5 queue_op 0x8377aa0 seq 2471 osr(3.1b7 0x276d1a0) 1034485 bytes   (queue has 4 ops and 3105748 bytes)
2014-06-12 14:46:18.116876 7f9d7a962700 10 _do_op 0x2728b620 seq 2586 r = 0, finisher 0x4e89ca0 0
2014-06-12 14:46:18.116904 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (2 active)
2014-06-12 14:46:18.116909 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.116915 7f9d7715b700  5 -- op tracker -- , seq: 747749, time: 2014-06-12 14:46:18.116904, event: commit_sent, op: osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.116925 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2829 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:18.116947 7f9d7a962700 10 _finish_op 0x2728b620 seq 2586 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.116956 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:18.116956 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (1 active)
2014-06-12 14:46:18.116966 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (2 active)
2014-06-12 14:46:18.116969 7f9d7a962700  5 _do_op 0x8377aa0 seq 2471 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:18.116972 7f9d7a962700 10 _do_transaction on 0x628c470
2014-06-12 14:46:18.116977 7f9d7a962700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:18.116995 7f9d7795c700  5 -- op tracker -- , seq: 747748, time: 2014-06-12 14:46:18.116988, event: sub_op_applied, op: osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.117002 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] sub_op_modify_applied on 0x29eb2a80 op osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.117019 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] op_applied on primary on version 173'2830
2014-06-12 14:46:18.117029 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] op_applied on replica on version 173'2830
2014-06-12 14:46:18.117055 7f9d7795c700  5 -- op tracker -- , seq: 747748, time: 2014-06-12 14:46:18.117050, event: done, op: osd_sub_op(client.4303.0:144765 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2830 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.117081 7f9d7795c700  5 -- op tracker -- , seq: 747749, time: 2014-06-12 14:46:18.117076, event: sub_op_applied, op: osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.117094 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:18.117092 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] sub_op_modify_applied on 0x13ec0c40 op osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.117106 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] op_applied on primary on version 173'2831
2014-06-12 14:46:18.117120 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.117115 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] op_applied on replica on version 173'2831
2014-06-12 14:46:18.117138 7f9d7795c700  5 -- op tracker -- , seq: 747749, time: 2014-06-12 14:46:18.117133, event: done, op: osd_sub_op(client.4303.0:144766 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2831 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.117166 7f9d7f16b700  5 _do_op 0x774b440 seq 2587 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:46:18.117172 7f9d7f16b700 10 _do_transaction on 0xc9b49b0
2014-06-12 14:46:18.117173 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.117179 7f9d7f16b700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.117220 7f9d7a962700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:18.117228 7f9d7a962700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:18.117235 7f9d7a962700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:18.117252 7f9d7a962700 15 _write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 1548288~1032192
2014-06-12 14:46:18.117260 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.117271 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.117298 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.116925 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2661 (0'0,173'2661] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active] do_sub_op osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.117306 7f9d633d7700 10 osd.2 173 dequeue_op 0x59dce00 finish
2014-06-12 14:46:18.117312 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.117316 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.117438 7f9d7a962700 10 lookup_strip_header done  cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:18.117575 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:18.122120 7f9d7a962700 10 _generic_write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 1548288~1032192 = 0
2014-06-12 14:46:18.122268 7f9d7a962700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:18.122295 7f9d7a962700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:18.122301 7f9d7a962700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:18.122308 7f9d7a962700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:18.122644 7f9d7a962700 20 set_header setting 17215 cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 parent seq 0
2014-06-12 14:46:18.122753 7f9d7a962700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:18.122768 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.122777 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:18.126797 7f9d5703e700  5 -- op tracker -- , seq: 747752, time: 2014-06-12 14:46:18.121661, event: header_read, op: osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.126813 7f9d5703e700  5 -- op tracker -- , seq: 747752, time: 2014-06-12 14:46:18.121663, event: throttled, op: osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.126823 7f9d5703e700  5 -- op tracker -- , seq: 747752, time: 2014-06-12 14:46:18.126752, event: all_read, op: osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.126833 7f9d5703e700  5 -- op tracker -- , seq: 747752, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.126838 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.126846 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:18.126854 7f9d5703e700 15 osd.2 173 enqueue_op 0x20e86d00 prio 127 cost 1032999 latency 0.005193 osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.127503 7f9d5703e700  5 -- op tracker -- , seq: 747753, time: 2014-06-12 14:46:18.126880, event: header_read, op: osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.127517 7f9d5703e700  5 -- op tracker -- , seq: 747753, time: 2014-06-12 14:46:18.126881, event: throttled, op: osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.127526 7f9d5703e700  5 -- op tracker -- , seq: 747753, time: 2014-06-12 14:46:18.127477, event: all_read, op: osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.127535 7f9d5703e700  5 -- op tracker -- , seq: 747753, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.127539 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.127546 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:18.127552 7f9d5703e700 15 osd.2 173 enqueue_op 0x20e87e00 prio 127 cost 516903 latency 0.000672 osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.127576 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.127584 7f9d643d9700 10 osd.2 173 dequeue_op 0x20e86d00 prio 127 cost 1032999 latency 0.005922 osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active]
2014-06-12 14:46:18.127606 7f9d643d9700  5 -- op tracker -- , seq: 747752, time: 2014-06-12 14:46:18.127602, event: reached_pg, op: osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.127614 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] handle_message: 0x20e86d00
2014-06-12 14:46:18.127626 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] sub_op_modify trans feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 v 173'2833 (transaction) 174
2014-06-12 14:46:18.127641 7f9d643d9700  5 -- op tracker -- , seq: 747752, time: 2014-06-12 14:46:18.127637, event: started, op: osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.127656 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2832 (0'0,173'2832] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] append_log log((0'0,173'2832], crt=0'0) [173'2833 (173'2832) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144768 2014-06-12 08:45:20.493641]
2014-06-12 14:46:18.127677 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2833 (0'0,173'2833] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] add_log_entry 173'2833 (173'2832) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144768 2014-06-12 08:45:20.493641
2014-06-12 14:46:18.127695 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2833 (0'0,173'2833] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] append_log  adding 1 keys
2014-06-12 14:46:18.127723 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2833, trimmed: 
2014-06-12 14:46:18.127743 7f9d643d9700  5 -- op tracker -- , seq: 747752, time: 2014-06-12 14:46:18.127738, event: started, op: osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.127751 7f9d643d9700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.127754 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.127756 7f9d643d9700  5 queue_op 0x1b41e60 seq 2588 osr(3.39f 0x276d720) 1034485 bytes   (queue has 3 ops and 2587359 bytes)
2014-06-12 14:46:18.127762 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2833 (0'0,173'2833] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] do_sub_op osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.127776 7f9d643d9700 10 osd.2 173 dequeue_op 0x20e86d00 finish
2014-06-12 14:46:18.127779 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.127784 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.127787 7f9d643d9700 10 osd.2 173 dequeue_op 0x20e87e00 prio 127 cost 516903 latency 0.000907 osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2833 (0'0,173'2833] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active]
2014-06-12 14:46:18.127803 7f9d643d9700  5 -- op tracker -- , seq: 747753, time: 2014-06-12 14:46:18.127799, event: reached_pg, op: osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.127810 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2833 (0'0,173'2833] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] handle_message: 0x20e87e00
2014-06-12 14:46:18.127819 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2833 (0'0,173'2833] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] sub_op_modify trans feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 v 173'2834 (transaction) 174
2014-06-12 14:46:18.127834 7f9d643d9700  5 -- op tracker -- , seq: 747753, time: 2014-06-12 14:46:18.127830, event: started, op: osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.127844 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2833 (0'0,173'2833] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] append_log log((0'0,173'2833], crt=0'0) [173'2834 (173'2833) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144769 2014-06-12 08:45:20.499570]
2014-06-12 14:46:18.127862 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2834 (0'0,173'2834] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] add_log_entry 173'2834 (173'2833) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144769 2014-06-12 08:45:20.499570
2014-06-12 14:46:18.127880 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2834 (0'0,173'2834] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] append_log  adding 1 keys
2014-06-12 14:46:18.127907 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2834, trimmed: 
2014-06-12 14:46:18.127925 7f9d643d9700  5 -- op tracker -- , seq: 747753, time: 2014-06-12 14:46:18.127921, event: started, op: osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.127933 7f9d643d9700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.127935 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.127936 7f9d643d9700  5 queue_op 0x7fca540 seq 2589 osr(3.39f 0x276d720) 518389 bytes   (queue has 4 ops and 3105748 bytes)
2014-06-12 14:46:18.127942 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2834 (0'0,173'2834] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] do_sub_op osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.127955 7f9d643d9700 10 osd.2 173 dequeue_op 0x20e87e00 finish
2014-06-12 14:46:18.127958 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.127961 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.127968 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (3 active)
2014-06-12 14:46:18.127976 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (4 active)
2014-06-12 14:46:18.128094 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.128129 7f9d7f16b700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.128137 7f9d7f16b700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:46:18.128144 7f9d7f16b700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.128290 7f9d7f16b700 15 _write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 1568768~516096
2014-06-12 14:46:18.128461 7f9d7f16b700 10 lookup_strip_header done  cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.128507 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.128618 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:18.130260 7f9d7f16b700 10 _generic_write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 1568768~516096 = 0
2014-06-12 14:46:18.130321 7f9d7f16b700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.130336 7f9d7f16b700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.130342 7f9d7f16b700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.130350 7f9d7f16b700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.130562 7f9d7f16b700 20 set_header setting 17505 cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 parent seq 0
2014-06-12 14:46:18.130646 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.130662 7f9d7f16b700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:46:18.130673 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:18.131128 7f9d7a962700 10 _do_op 0x8377aa0 seq 2471 r = 0, finisher 0x7ffd540 0
2014-06-12 14:46:18.131175 7f9d7a962700 10 _finish_op 0x8377aa0 seq 2471 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:18.131219 7f9d7795c700  5 -- op tracker -- , seq: 747751, time: 2014-06-12 14:46:18.131209, event: sub_op_applied, op: osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.131230 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2661 (0'0,173'2661] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active] sub_op_modify_applied on 0x628c380 op osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.131264 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:18.131284 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2661 (0'0,173'2661] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active] op_applied on primary on version 173'2661
2014-06-12 14:46:18.131296 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2661 (0'0,173'2661] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active] op_applied on replica on version 173'2661
2014-06-12 14:46:18.131371 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (3 active)
2014-06-12 14:46:18.131395 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.131798 7f9d7715b700  5 -- op tracker -- , seq: 747751, time: 2014-06-12 14:46:18.131788, event: commit_sent, op: osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.131816 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2661 (0'0,173'2661] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2659 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:18.131837 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:18.131867 7f9d7715b700  5 -- op tracker -- , seq: 747751, time: 2014-06-12 14:46:18.131862, event: done, op: osd_sub_op(client.4284.0:141065 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2661 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.133862 7f9d7f16b700 10 _do_op 0x774b440 seq 2587 r = 0, finisher 0x6bb20a0 0
2014-06-12 14:46:18.133881 7f9d7f16b700 10 _finish_op 0x774b440 seq 2587 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.133890 7f9d7e169700  5 _do_op 0x1b41e60 seq 2588 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:46:18.133894 7f9d7e169700 10 _do_transaction on 0x5ea17b0
2014-06-12 14:46:18.133901 7f9d7e169700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.134034 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.134059 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.134112 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.134134 7f9d7e169700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.134140 7f9d7e169700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:46:18.134145 7f9d7e169700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.134244 7f9d7e169700 15 _write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 2084864~1032192
2014-06-12 14:46:18.134395 7f9d7e169700 10 lookup_strip_header done  cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.134512 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:18.135866 7f9d7715b700  5 -- op tracker -- , seq: 747750, time: 2014-06-12 14:46:18.135847, event: commit_sent, op: osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.135890 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2834 (0'0,173'2834] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2830 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:18.135939 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:18.137795 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (2 active)
2014-06-12 14:46:18.137812 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.137886 7f9d7795c700  5 -- op tracker -- , seq: 747750, time: 2014-06-12 14:46:18.137873, event: sub_op_applied, op: osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.137897 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2834 (0'0,173'2834] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active] sub_op_modify_applied on 0xc9b48c0 op osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.137924 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2834 (0'0,173'2834] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active] op_applied on primary on version 173'2832
2014-06-12 14:46:18.137934 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2834 (0'0,173'2834] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active] op_applied on replica on version 173'2832
2014-06-12 14:46:18.137961 7f9d7795c700  5 -- op tracker -- , seq: 747750, time: 2014-06-12 14:46:18.137957, event: done, op: osd_sub_op(client.4303.0:144767 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2832 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.139491 7f9d7e169700 10 _generic_write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 2084864~1032192 = 0
2014-06-12 14:46:18.139615 7f9d7e169700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.139631 7f9d7e169700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.139641 7f9d7e169700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.139650 7f9d7e169700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.140042 7f9d7e169700 20 set_header setting 17505 cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 parent seq 0
2014-06-12 14:46:18.140183 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.140204 7f9d7e169700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:46:18.140216 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:18.142611 7f9d5703e700  5 -- op tracker -- , seq: 747754, time: 2014-06-12 14:46:18.141718, event: header_read, op: osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.142635 7f9d5703e700  5 -- op tracker -- , seq: 747754, time: 2014-06-12 14:46:18.141720, event: throttled, op: osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.142644 7f9d5703e700  5 -- op tracker -- , seq: 747754, time: 2014-06-12 14:46:18.142557, event: all_read, op: osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.142653 7f9d5703e700  5 -- op tracker -- , seq: 747754, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.142658 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.142666 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:18.142674 7f9d5703e700 15 osd.2 173 enqueue_op 0x20e86900 prio 127 cost 516903 latency 0.000956 osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.142703 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.142710 7f9d61bd4700 10 osd.2 173 dequeue_op 0x20e86900 prio 127 cost 516903 latency 0.000992 osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2834 (0'0,173'2834] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active]
2014-06-12 14:46:18.142736 7f9d61bd4700  5 -- op tracker -- , seq: 747754, time: 2014-06-12 14:46:18.142731, event: reached_pg, op: osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.142745 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2834 (0'0,173'2834] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active] handle_message: 0x20e86900
2014-06-12 14:46:18.142759 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2834 (0'0,173'2834] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active] sub_op_modify trans feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 v 173'2835 (transaction) 174
2014-06-12 14:46:18.142779 7f9d61bd4700  5 -- op tracker -- , seq: 747754, time: 2014-06-12 14:46:18.142774, event: started, op: osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.142795 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2834 (0'0,173'2834] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active] append_log log((0'0,173'2834], crt=0'0) [173'2835 (173'2834) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144770 2014-06-12 08:45:20.519522]
2014-06-12 14:46:18.142817 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2835 (0'0,173'2835] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active] add_log_entry 173'2835 (173'2834) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144770 2014-06-12 08:45:20.519522
2014-06-12 14:46:18.142840 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2835 (0'0,173'2835] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active] append_log  adding 1 keys
2014-06-12 14:46:18.142873 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2835, trimmed: 
2014-06-12 14:46:18.142897 7f9d61bd4700  5 -- op tracker -- , seq: 747754, time: 2014-06-12 14:46:18.142892, event: started, op: osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.142908 7f9d61bd4700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.142913 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.142914 7f9d61bd4700  5 queue_op 0x196fafc0 seq 2590 osr(3.39f 0x276d720) 518389 bytes   (queue has 3 ops and 2071263 bytes)
2014-06-12 14:46:18.142930 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2835 (0'0,173'2835] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active] do_sub_op osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.142946 7f9d61bd4700 10 osd.2 173 dequeue_op 0x20e86900 finish
2014-06-12 14:46:18.142950 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.142952 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.142960 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (3 active)
2014-06-12 14:46:18.146285 7f9d7e169700 10 _do_op 0x1b41e60 seq 2588 r = 0, finisher 0x7bd84a0 0
2014-06-12 14:46:18.146309 7f9d7e169700 10 _finish_op 0x1b41e60 seq 2588 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.146342 7f9d7795c700  5 -- op tracker -- , seq: 747752, time: 2014-06-12 14:46:18.146332, event: sub_op_applied, op: osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.146353 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2835 (0'0,173'2835] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active] sub_op_modify_applied on 0x5ea16c0 op osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.146386 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:18.146407 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2835 (0'0,173'2835] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active] op_applied on primary on version 173'2833
2014-06-12 14:46:18.146420 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2835 (0'0,173'2835] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active] op_applied on replica on version 173'2833
2014-06-12 14:46:18.146438 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (2 active)
2014-06-12 14:46:18.146448 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.146447 7f9d7715b700  5 -- op tracker -- , seq: 747752, time: 2014-06-12 14:46:18.146439, event: commit_sent, op: osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.146455 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2835 (0'0,173'2835] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2831 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:18.146474 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:18.146503 7f9d7715b700  5 -- op tracker -- , seq: 747752, time: 2014-06-12 14:46:18.146498, event: done, op: osd_sub_op(client.4303.0:144768 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2833 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.148119 7f9d7e96a700  5 _do_op 0x7fca540 seq 2589 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:46:18.148133 7f9d7e96a700 10 _do_transaction on 0x13c6c2b0
2014-06-12 14:46:18.148141 7f9d7e96a700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.148255 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.148278 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.148316 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.148338 7f9d7e96a700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.148343 7f9d7e96a700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:46:18.148348 7f9d7e96a700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.148491 7f9d7e96a700 15 _write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 3117056~516096
2014-06-12 14:46:18.148600 7f9d7e96a700 10 lookup_strip_header done  cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.148702 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:18.150106 7f9d7e96a700 10 _generic_write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 3117056~516096 = 0
2014-06-12 14:46:18.150158 7f9d7e96a700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.150169 7f9d7e96a700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.150173 7f9d7e96a700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.150178 7f9d7e96a700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.150323 7f9d7e96a700 20 set_header setting 17505 cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 parent seq 0
2014-06-12 14:46:18.150342 7f9d5703e700  5 -- op tracker -- , seq: 747755, time: 2014-06-12 14:46:18.150208, event: header_read, op: osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.150358 7f9d5703e700  5 -- op tracker -- , seq: 747755, time: 2014-06-12 14:46:18.150210, event: throttled, op: osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.150372 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.150370 7f9d5703e700  5 -- op tracker -- , seq: 747755, time: 2014-06-12 14:46:18.150311, event: all_read, op: osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.150381 7f9d5703e700  5 -- op tracker -- , seq: 747755, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.150383 7f9d7e96a700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:46:18.150389 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:18.150387 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.150393 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:18.150401 7f9d5703e700 15 osd.2 173 enqueue_op 0x20e87f00 prio 127 cost 45863 latency 0.000193 osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.150434 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.150442 7f9d673df700 10 osd.2 173 dequeue_op 0x20e87f00 prio 127 cost 45863 latency 0.000234 osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.39f( v 173'2835 (0'0,173'2835] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active]
2014-06-12 14:46:18.150466 7f9d673df700  5 -- op tracker -- , seq: 747755, time: 2014-06-12 14:46:18.150462, event: reached_pg, op: osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.150476 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2835 (0'0,173'2835] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] handle_message: 0x20e87f00
2014-06-12 14:46:18.150487 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2835 (0'0,173'2835] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] sub_op_modify trans feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 v 173'2836 (transaction) 174
2014-06-12 14:46:18.150505 7f9d673df700  5 -- op tracker -- , seq: 747755, time: 2014-06-12 14:46:18.150501, event: started, op: osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.150520 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2835 (0'0,173'2835] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] append_log log((0'0,173'2835], crt=0'0) [173'2836 (173'2835) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144771 2014-06-12 08:45:20.529527]
2014-06-12 14:46:18.150546 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] add_log_entry 173'2836 (173'2835) modify   feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 by client.4303.0:144771 2014-06-12 08:45:20.529527
2014-06-12 14:46:18.150569 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] append_log  adding 1 keys
2014-06-12 14:46:18.150604 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2836, trimmed: 
2014-06-12 14:46:18.150629 7f9d673df700  5 -- op tracker -- , seq: 747755, time: 2014-06-12 14:46:18.150624, event: started, op: osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.150640 7f9d673df700  5 queue_transactions existing osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.150644 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.150646 7f9d673df700  5 queue_op 0x208be480 seq 2591 osr(3.39f 0x276d720) 47349 bytes   (queue has 3 ops and 1084127 bytes)
2014-06-12 14:46:18.150653 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] do_sub_op osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.150667 7f9d673df700 10 osd.2 173 dequeue_op 0x20e87f00 finish
2014-06-12 14:46:18.150671 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.150681 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.150688 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6fc0 (3 active)
2014-06-12 14:46:18.153571 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:18.154594 7f9d7e96a700 10 _do_op 0x7fca540 seq 2589 r = 0, finisher 0x7331aa0 0
2014-06-12 14:46:18.154654 7f9d7715b700  5 -- op tracker -- , seq: 747753, time: 2014-06-12 14:46:18.154646, event: commit_sent, op: osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.154663 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:18.154687 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:18.154762 7f9d7e96a700 10 _finish_op 0x7fca540 seq 2589 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.154783 7f9d7815d700  5 _do_op 0x196fafc0 seq 2590 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:46:18.154788 7f9d7815d700 10 _do_transaction on 0x1e78b5f0
2014-06-12 14:46:18.154795 7f9d7815d700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.155034 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.155054 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.155109 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.155124 7f9d7815d700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.155130 7f9d7815d700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:46:18.155136 7f9d7815d700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.155149 7f9d7815d700 15 _write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 3633152~516096
2014-06-12 14:46:18.155308 7f9d7815d700 10 lookup_strip_header done  cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.155392 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:18.157257 7f9d7815d700 10 _generic_write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 3633152~516096 = 0
2014-06-12 14:46:18.157311 7f9d7815d700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.157326 7f9d7815d700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.157332 7f9d7815d700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.157340 7f9d7815d700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.157561 7f9d7815d700 20 set_header setting 17505 cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 parent seq 0
2014-06-12 14:46:18.157648 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.157663 7f9d7815d700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:46:18.157673 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:18.160295 7f9d7815d700 10 _do_op 0x196fafc0 seq 2590 r = 0, finisher 0x52df1a0 0
2014-06-12 14:46:18.160326 7f9d7715b700  5 -- op tracker -- , seq: 747754, time: 2014-06-12 14:46:18.160318, event: commit_sent, op: osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.160334 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2833 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:18.160352 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:18.160509 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (2 active)
2014-06-12 14:46:18.160522 7f9d7795c700  5 -- op tracker -- , seq: 747753, time: 2014-06-12 14:46:18.160516, event: sub_op_applied, op: osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.160528 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] sub_op_modify_applied on 0x13c6c1c0 op osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.160540 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] op_applied on primary on version 173'2834
2014-06-12 14:46:18.160546 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] op_applied on replica on version 173'2834
2014-06-12 14:46:18.160562 7f9d7795c700  5 -- op tracker -- , seq: 747753, time: 2014-06-12 14:46:18.160559, event: done, op: osd_sub_op(client.4303.0:144769 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.160580 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.160588 7f9d7815d700 10 _finish_op 0x196fafc0 seq 2590 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.160596 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (1 active)
2014-06-12 14:46:18.160604 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.160606 7f9d7795c700  5 -- op tracker -- , seq: 747754, time: 2014-06-12 14:46:18.160600, event: sub_op_applied, op: osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.160616 7f9d82171700  5 _do_op 0x208be480 seq 2591 osr(3.39f 0x276d720)/0x276d720 start
2014-06-12 14:46:18.160624 7f9d82171700 10 _do_transaction on 0x30d5e9b0
2014-06-12 14:46:18.160613 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] sub_op_modify_applied on 0x1e78b500 op osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.160629 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] op_applied on primary on version 173'2835
2014-06-12 14:46:18.160633 7f9d82171700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.160639 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] op_applied on replica on version 173'2835
2014-06-12 14:46:18.160659 7f9d7795c700  5 -- op tracker -- , seq: 747754, time: 2014-06-12 14:46:18.160654, event: done, op: osd_sub_op(client.4303.0:144770 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.160770 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.160795 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.160854 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.160877 7f9d82171700 15 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.160885 7f9d82171700 10 _omap_rmkeys meta/1fe1881b/pglog_3.39f/0//-1 r = 0
2014-06-12 14:46:18.160892 7f9d82171700 15 _omap_setkeys meta/1fe1881b/pglog_3.39f/0//-1
2014-06-12 14:46:18.160907 7f9d82171700 15 _write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 4149248~45056
2014-06-12 14:46:18.161062 7f9d82171700 10 lookup_strip_header done  cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.161171 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:18.161331 7f9d82171700 10 _generic_write 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 4149248~45056 = 0
2014-06-12 14:46:18.161345 7f9d82171700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.161356 7f9d82171700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.161363 7f9d82171700 15 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3
2014-06-12 14:46:18.161372 7f9d82171700 10 _setattrs 3.39f_head/feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 = 0
2014-06-12 14:46:18.161492 7f9d82171700 20 set_header setting 17505 cid 3.39f_head oid feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 parent seq 0
2014-06-12 14:46:18.161516 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.161532 7f9d82171700 20 set_header setting 2170 cid meta oid 1fe1881b/pglog_3.39f/0//-1 parent seq 0
2014-06-12 14:46:18.161542 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:18.161814 7f9d82171700 10 _do_op 0x208be480 seq 2591 r = 0, finisher 0x72339a0 0
2014-06-12 14:46:18.161855 7f9d7715b700  5 -- op tracker -- , seq: 747755, time: 2014-06-12 14:46:18.161839, event: commit_sent, op: osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.161867 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:18.161889 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:18.161912 7f9d82171700 10 _finish_op 0x208be480 seq 2591 osr(3.39f 0x276d720)/0x276d720
2014-06-12 14:46:18.161919 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6fc0 (0 active)
2014-06-12 14:46:18.161922 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.161939 7f9d7795c700  5 -- op tracker -- , seq: 747755, time: 2014-06-12 14:46:18.161932, event: sub_op_applied, op: osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.161946 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active] sub_op_modify_applied on 0x30d5e8c0 op osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.161962 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active] op_applied on primary on version 173'2836
2014-06-12 14:46:18.161972 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.39f( v 173'2836 (0'0,173'2836] local-les=137 n=284 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active] op_applied on replica on version 173'2836
2014-06-12 14:46:18.161995 7f9d7795c700  5 -- op tracker -- , seq: 747755, time: 2014-06-12 14:46:18.161990, event: done, op: osd_sub_op(client.4303.0:144771 3.39f feb83b9f/rbd_data.10a66b8b4567.0000000000000ff3/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.216795 7f9d56432700  5 -- op tracker -- , seq: 747756, time: 2014-06-12 14:46:18.215850, event: header_read, op: osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.216815 7f9d56432700  5 -- op tracker -- , seq: 747756, time: 2014-06-12 14:46:18.215855, event: throttled, op: osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.216824 7f9d56432700  5 -- op tracker -- , seq: 747756, time: 2014-06-12 14:46:18.216749, event: all_read, op: osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.216832 7f9d56432700  5 -- op tracker -- , seq: 747756, time: 0.000000, event: dispatched, op: osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.216838 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.216846 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:18.216855 7f9d56432700 15 osd.2 173 enqueue_op 0x124e2000 prio 127 cost 336679 latency 0.001005 osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.217465 7f9d58d5b700  5 -- op tracker -- , seq: 747757, time: 2014-06-12 14:46:18.214848, event: header_read, op: osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.217477 7f9d58d5b700  5 -- op tracker -- , seq: 747757, time: 2014-06-12 14:46:18.214851, event: throttled, op: osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.217487 7f9d58d5b700  5 -- op tracker -- , seq: 747757, time: 2014-06-12 14:46:18.217433, event: all_read, op: osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.217500 7f9d58d5b700  5 -- op tracker -- , seq: 747757, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.217510 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.217518 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:18.217524 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016300 prio 127 cost 676647 latency 0.002676 osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.217565 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.217572 7f9d67be0700 10 osd.2 173 dequeue_op 0x124e2000 prio 127 cost 336679 latency 0.001722 osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2634 (0'0,173'2634] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2633 active]
2014-06-12 14:46:18.217604 7f9d67be0700  5 -- op tracker -- , seq: 747756, time: 2014-06-12 14:46:18.217599, event: reached_pg, op: osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.217629 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:18.217624 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2634 (0'0,173'2634] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2633 active] handle_message: 0x124e2000
2014-06-12 14:46:18.217638 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2634 (0'0,173'2634] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2633 active] sub_op_modify trans 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 v 173'2635 (transaction) 174
2014-06-12 14:46:18.217635 7f9d663dd700 10 osd.2 173 dequeue_op 0x34016300 prio 127 cost 676647 latency 0.002787 osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4d0( v 173'2715 (0'0,173'2715] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2714 active]
2014-06-12 14:46:18.217653 7f9d67be0700  5 -- op tracker -- , seq: 747756, time: 2014-06-12 14:46:18.217650, event: started, op: osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.217672 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2634 (0'0,173'2634] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2633 active] append_log log((0'0,173'2634], crt=0'0) [173'2635 (173'664) modify   3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 by client.4286.0:144487 2014-06-12 08:43:09.398358]
2014-06-12 14:46:18.217698 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2635 (0'0,173'2635] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2633 active] add_log_entry 173'2635 (173'664) modify   3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 by client.4286.0:144487 2014-06-12 08:43:09.398358
2014-06-12 14:46:18.217719 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2635 (0'0,173'2635] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2633 active] append_log  adding 1 keys
2014-06-12 14:46:18.217753 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2635, trimmed: 
2014-06-12 14:46:18.217781 7f9d67be0700  5 -- op tracker -- , seq: 747756, time: 2014-06-12 14:46:18.217775, event: started, op: osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.217793 7f9d67be0700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:18.217797 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.217799 7f9d67be0700  5 queue_op 0x21002e40 seq 2455 osr(3.1c7 0x276d170) 338165 bytes   (queue has 1 ops and 338165 bytes)
2014-06-12 14:46:18.217812 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:46:18.217808 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2635 (0'0,173'2635] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2633 active] do_sub_op osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.217820 7f9d7d167700  5 _do_op 0x21002e40 seq 2455 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:46:18.217822 7f9d67be0700 10 osd.2 173 dequeue_op 0x124e2000 finish
2014-06-12 14:46:18.217826 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:18.217828 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.217826 7f9d7d167700 10 _do_transaction on 0x23fbf0b0
2014-06-12 14:46:18.217838 7f9d7d167700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:18.218003 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:18.218021 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.218085 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.218191 7f9d7d167700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:18.218197 7f9d7d167700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:46:18.218203 7f9d7d167700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:18.218221 7f9d7d167700 15 _write 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 0~335872
2014-06-12 14:46:18.218407 7f9d7d167700 10 lookup_strip_header done  cid 3.1c7_head oid 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3
2014-06-12 14:46:18.218657 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:18.219755 7f9d7d167700 10 _generic_write 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 0~335872 = 0
2014-06-12 14:46:18.219797 7f9d7d167700 15 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3
2014-06-12 14:46:18.219807 7f9d7d167700 10 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:18.219811 7f9d7d167700 15 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3
2014-06-12 14:46:18.219816 7f9d7d167700 10 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:18.219981 7f9d7d167700 20 set_header setting 16919 cid 3.1c7_head oid 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 parent seq 0
2014-06-12 14:46:18.220037 7f9d7d167700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:46:18.220051 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.220059 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:18.221676 7f9d663dd700  5 -- op tracker -- , seq: 747757, time: 2014-06-12 14:46:18.217652, event: reached_pg, op: osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.221714 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2715 (0'0,173'2715] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2714 active] handle_message: 0x34016300
2014-06-12 14:46:18.221736 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2715 (0'0,173'2715] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2714 active] sub_op_modify trans b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 v 173'2716 (transaction) 174
2014-06-12 14:46:18.221754 7f9d663dd700  5 -- op tracker -- , seq: 747757, time: 2014-06-12 14:46:18.221751, event: started, op: osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.221793 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2715 (0'0,173'2715] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2714 active] append_log log((0'0,173'2715], crt=0'0) [173'2716 (173'594) modify   b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 by client.4306.0:144129 2014-06-12 08:45:20.578399]
2014-06-12 14:46:18.221827 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2716 (0'0,173'2716] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2714 active] add_log_entry 173'2716 (173'594) modify   b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 by client.4306.0:144129 2014-06-12 08:45:20.578399
2014-06-12 14:46:18.221855 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2716 (0'0,173'2716] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2714 active] append_log  adding 1 keys
2014-06-12 14:46:18.221893 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2716, trimmed: 
2014-06-12 14:46:18.221923 7f9d663dd700  5 -- op tracker -- , seq: 747757, time: 2014-06-12 14:46:18.221917, event: started, op: osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.221941 7f9d663dd700  5 queue_transactions existing osr(3.4d0 0x276d840)/0x276d840
2014-06-12 14:46:18.221948 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.221951 7f9d663dd700  5 queue_op 0x22fe91a0 seq 2561 osr(3.4d0 0x276d840) 678133 bytes   (queue has 2 ops and 1016298 bytes)
2014-06-12 14:46:18.221961 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2716 (0'0,173'2716] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2714 active] do_sub_op osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.221976 7f9d663dd700 10 osd.2 173 dequeue_op 0x34016300 finish
2014-06-12 14:46:18.221984 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.221993 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.222007 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479320 (2 active)
2014-06-12 14:46:18.222015 7f9d8116f700  5 _do_op 0x22fe91a0 seq 2561 osr(3.4d0 0x276d840)/0x276d840 start
2014-06-12 14:46:18.222023 7f9d8116f700 10 _do_transaction on 0x23fbe470
2014-06-12 14:46:18.222032 7f9d8116f700 15 _omap_setkeys meta/1fe07d85/pglog_3.4d0/0//-1
2014-06-12 14:46:18.222250 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe07d85/pglog_3.4d0/0//-1
2014-06-12 14:46:18.222374 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.222856 7f9d7d167700 10 _do_op 0x21002e40 seq 2455 r = 0, finisher 0x65f74c0 0
2014-06-12 14:46:18.222874 7f9d7d167700 10 _finish_op 0x21002e40 seq 2455 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:18.222882 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (1 active)
2014-06-12 14:46:18.222886 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.222907 7f9d7715b700  5 -- op tracker -- , seq: 747756, time: 2014-06-12 14:46:18.222899, event: commit_sent, op: osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.222917 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2635 (0'0,173'2635] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2633 active] sub_op_modify_commit on op osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:18.222944 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:18.223068 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.223097 7f9d8116f700 15 _omap_rmkeys meta/1fe07d85/pglog_3.4d0/0//-1
2014-06-12 14:46:18.223103 7f9d8116f700 10 _omap_rmkeys meta/1fe07d85/pglog_3.4d0/0//-1 r = 0
2014-06-12 14:46:18.223109 7f9d8116f700 15 _omap_setkeys meta/1fe07d85/pglog_3.4d0/0//-1
2014-06-12 14:46:18.223126 7f9d8116f700 15 _write 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 0~675840
2014-06-12 14:46:18.223310 7f9d8116f700 10 lookup_strip_header done  cid 3.4d0_head oid b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3
2014-06-12 14:46:18.223608 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:18.224848 7f9d7795c700  5 -- op tracker -- , seq: 747756, time: 2014-06-12 14:46:18.224836, event: sub_op_applied, op: osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.224862 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2635 (0'0,173'2635] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2634 active] sub_op_modify_applied on 0x23fbefc0 op osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.224885 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2635 (0'0,173'2635] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2634 active] op_applied on primary on version 173'2635
2014-06-12 14:46:18.224894 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2635 (0'0,173'2635] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2634 active] op_applied on replica on version 173'2635
2014-06-12 14:46:18.224917 7f9d7795c700  5 -- op tracker -- , seq: 747756, time: 2014-06-12 14:46:18.224912, event: done, op: osd_sub_op(client.4286.0:144487 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2635 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.226404 7f9d8116f700 10 _generic_write 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 0~675840 = 0
2014-06-12 14:46:18.226514 7f9d8116f700 15 _setattrs 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3
2014-06-12 14:46:18.226534 7f9d8116f700 10 _setattrs 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 = 0
2014-06-12 14:46:18.226542 7f9d8116f700 15 _setattrs 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3
2014-06-12 14:46:18.226551 7f9d8116f700 10 _setattrs 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 = 0
2014-06-12 14:46:18.226843 7f9d8116f700 20 set_header setting 16801 cid 3.4d0_head oid b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 parent seq 0
2014-06-12 14:46:18.226956 7f9d8116f700 20 set_header setting 2294 cid meta oid 1fe07d85/pglog_3.4d0/0//-1 parent seq 0
2014-06-12 14:46:18.226974 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.226984 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:18.234394 7f9d8116f700 10 _do_op 0x22fe91a0 seq 2561 r = 0, finisher 0x682b120 0
2014-06-12 14:46:18.234416 7f9d8116f700 10 _finish_op 0x22fe91a0 seq 2561 osr(3.4d0 0x276d840)/0x276d840
2014-06-12 14:46:18.234437 7f9d7715b700  5 -- op tracker -- , seq: 747757, time: 2014-06-12 14:46:18.234424, event: commit_sent, op: osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.234424 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479320 (0 active)
2014-06-12 14:46:18.234463 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.234461 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2716 (0'0,173'2716] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2714 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:18.234485 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:18.234512 7f9d7795c700  5 -- op tracker -- , seq: 747757, time: 2014-06-12 14:46:18.234506, event: sub_op_applied, op: osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.234519 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2716 (0'0,173'2716] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] sub_op_modify_applied on 0x23fbe380 op osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.234564 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2716 (0'0,173'2716] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] op_applied on primary on version 173'2716
2014-06-12 14:46:18.234575 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2716 (0'0,173'2716] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] op_applied on replica on version 173'2716
2014-06-12 14:46:18.234642 7f9d7795c700  5 -- op tracker -- , seq: 747757, time: 2014-06-12 14:46:18.234636, event: done, op: osd_sub_op(client.4306.0:144129 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2716 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.246974 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:18.247014 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:18.259883 7f9d58d5b700  5 -- op tracker -- , seq: 747758, time: 2014-06-12 14:46:18.253660, event: header_read, op: osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.259903 7f9d58d5b700  5 -- op tracker -- , seq: 747758, time: 2014-06-12 14:46:18.253670, event: throttled, op: osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.259911 7f9d58d5b700  5 -- op tracker -- , seq: 747758, time: 2014-06-12 14:46:18.259824, event: all_read, op: osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.259918 7f9d58d5b700  5 -- op tracker -- , seq: 747758, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.259924 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.259930 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:18.259939 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34017100 prio 127 cost 2581287 latency 0.006278 osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.259970 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.259979 7f9d693e3700 10 osd.2 173 dequeue_op 0x34017100 prio 127 cost 2581287 latency 0.006319 osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4d0( v 173'2716 (0'0,173'2716] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active]
2014-06-12 14:46:18.260004 7f9d693e3700  5 -- op tracker -- , seq: 747758, time: 2014-06-12 14:46:18.260000, event: reached_pg, op: osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.260015 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2716 (0'0,173'2716] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] handle_message: 0x34017100
2014-06-12 14:46:18.260029 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2716 (0'0,173'2716] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] sub_op_modify trans b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 v 173'2717 (transaction) 174
2014-06-12 14:46:18.260047 7f9d693e3700  5 -- op tracker -- , seq: 747758, time: 2014-06-12 14:46:18.260043, event: started, op: osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.260066 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2716 (0'0,173'2716] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] append_log log((0'0,173'2716], crt=0'0) [173'2717 (173'2716) modify   b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 by client.4306.0:144130 2014-06-12 08:45:20.620174]
2014-06-12 14:46:18.260099 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2717 (0'0,173'2717] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] add_log_entry 173'2717 (173'2716) modify   b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 by client.4306.0:144130 2014-06-12 08:45:20.620174
2014-06-12 14:46:18.260122 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2717 (0'0,173'2717] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] append_log  adding 1 keys
2014-06-12 14:46:18.260161 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2717, trimmed: 
2014-06-12 14:46:18.260190 7f9d693e3700  5 -- op tracker -- , seq: 747758, time: 2014-06-12 14:46:18.260184, event: started, op: osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.260201 7f9d693e3700  5 queue_transactions existing osr(3.4d0 0x276d840)/0x276d840
2014-06-12 14:46:18.260206 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.260208 7f9d693e3700  5 queue_op 0x8102f60 seq 2562 osr(3.4d0 0x276d840) 2582773 bytes   (queue has 1 ops and 2582773 bytes)
2014-06-12 14:46:18.260223 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479320 (1 active)
2014-06-12 14:46:18.260231 7f9d79960700  5 _do_op 0x8102f60 seq 2562 osr(3.4d0 0x276d840)/0x276d840 start
2014-06-12 14:46:18.260218 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2717 (0'0,173'2717] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] do_sub_op osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.260234 7f9d693e3700 10 osd.2 173 dequeue_op 0x34017100 finish
2014-06-12 14:46:18.260236 7f9d79960700 10 _do_transaction on 0x66beb70
2014-06-12 14:46:18.260238 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.260241 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.260243 7f9d79960700 15 _omap_setkeys meta/1fe07d85/pglog_3.4d0/0//-1
2014-06-12 14:46:18.261062 7f9d58d5b700  5 -- op tracker -- , seq: 747759, time: 2014-06-12 14:46:18.259968, event: header_read, op: osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.261073 7f9d58d5b700  5 -- op tracker -- , seq: 747759, time: 2014-06-12 14:46:18.259970, event: throttled, op: osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.261081 7f9d58d5b700  5 -- op tracker -- , seq: 747759, time: 2014-06-12 14:46:18.260977, event: all_read, op: osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.261088 7f9d58d5b700  5 -- op tracker -- , seq: 747759, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.261093 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.261099 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:18.261110 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34017700 prio 127 cost 938791 latency 0.001141 osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.261137 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.261143 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x34017700 prio 127 cost 938791 latency 0.001175 osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4d0( v 173'2717 (0'0,173'2717] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active]
2014-06-12 14:46:18.261165 7f9d6b3e7700  5 -- op tracker -- , seq: 747759, time: 2014-06-12 14:46:18.261160, event: reached_pg, op: osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.261173 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2717 (0'0,173'2717] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] handle_message: 0x34017700
2014-06-12 14:46:18.261185 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2717 (0'0,173'2717] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] sub_op_modify trans b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 v 173'2718 (transaction) 174
2014-06-12 14:46:18.261200 7f9d6b3e7700  5 -- op tracker -- , seq: 747759, time: 2014-06-12 14:46:18.261196, event: started, op: osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.261224 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2717 (0'0,173'2717] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] append_log log((0'0,173'2717], crt=0'0) [173'2718 (173'2717) modify   b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 by client.4306.0:144131 2014-06-12 08:45:20.629986]
2014-06-12 14:46:18.261250 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2718 (0'0,173'2718] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] add_log_entry 173'2718 (173'2717) modify   b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 by client.4306.0:144131 2014-06-12 08:45:20.629986
2014-06-12 14:46:18.261271 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2718 (0'0,173'2718] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] append_log  adding 1 keys
2014-06-12 14:46:18.261305 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2718, trimmed: 
2014-06-12 14:46:18.261328 7f9d6b3e7700  5 -- op tracker -- , seq: 747759, time: 2014-06-12 14:46:18.261323, event: started, op: osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.261338 7f9d6b3e7700  5 queue_transactions existing osr(3.4d0 0x276d840)/0x276d840
2014-06-12 14:46:18.261342 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.261344 7f9d6b3e7700  5 queue_op 0x652c240 seq 2563 osr(3.4d0 0x276d840) 940277 bytes   (queue has 2 ops and 3523050 bytes)
2014-06-12 14:46:18.261352 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2718 (0'0,173'2718] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] do_sub_op osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.261367 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x34017700 finish
2014-06-12 14:46:18.261370 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.261372 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.261646 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479320 (2 active)
2014-06-12 14:46:18.275356 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe07d85/pglog_3.4d0/0//-1
2014-06-12 14:46:18.275395 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.275450 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.275604 7f9d79960700 15 _omap_rmkeys meta/1fe07d85/pglog_3.4d0/0//-1
2014-06-12 14:46:18.275611 7f9d79960700 10 _omap_rmkeys meta/1fe07d85/pglog_3.4d0/0//-1 r = 0
2014-06-12 14:46:18.275619 7f9d79960700 15 _omap_setkeys meta/1fe07d85/pglog_3.4d0/0//-1
2014-06-12 14:46:18.275639 7f9d79960700 15 _write 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 675840~2580480
2014-06-12 14:46:18.275800 7f9d79960700 10 lookup_strip_header done  cid 3.4d0_head oid b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3
2014-06-12 14:46:18.276025 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:18.290450 7f9d79960700 10 _generic_write 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 675840~2580480 = 0
2014-06-12 14:46:18.290825 7f9d79960700 15 _setattrs 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3
2014-06-12 14:46:18.290845 7f9d79960700 10 _setattrs 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 = 0
2014-06-12 14:46:18.290851 7f9d79960700 15 _setattrs 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3
2014-06-12 14:46:18.290863 7f9d79960700 10 _setattrs 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 = 0
2014-06-12 14:46:18.291915 7f9d79960700 20 set_header setting 16801 cid 3.4d0_head oid b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 parent seq 0
2014-06-12 14:46:18.292666 7f9d79960700 20 set_header setting 2294 cid meta oid 1fe07d85/pglog_3.4d0/0//-1 parent seq 0
2014-06-12 14:46:18.292687 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.292700 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:18.306036 7f9d79960700 10 _do_op 0x8102f60 seq 2562 r = 0, finisher 0x29fc04a0 0
2014-06-12 14:46:18.306093 7f9d79960700 10 _finish_op 0x8102f60 seq 2562 osr(3.4d0 0x276d840)/0x276d840
2014-06-12 14:46:18.306111 7f9d7715b700  5 -- op tracker -- , seq: 747758, time: 2014-06-12 14:46:18.306100, event: commit_sent, op: osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.306130 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2718 (0'0,173'2718] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2715 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:18.306159 7f9d8096e700  5 _do_op 0x652c240 seq 2563 osr(3.4d0 0x276d840)/0x276d840 start
2014-06-12 14:46:18.306168 7f9d8096e700 10 _do_transaction on 0x296eef0
2014-06-12 14:46:18.306168 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:18.306180 7f9d8096e700 15 _omap_setkeys meta/1fe07d85/pglog_3.4d0/0//-1
2014-06-12 14:46:18.306126 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479320 (1 active)
2014-06-12 14:46:18.306204 7f9d7795c700  5 -- op tracker -- , seq: 747758, time: 2014-06-12 14:46:18.306199, event: sub_op_applied, op: osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.306212 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2718 (0'0,173'2718] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2716 active] sub_op_modify_applied on 0x66bea80 op osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.306228 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2718 (0'0,173'2718] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2716 active] op_applied on primary on version 173'2717
2014-06-12 14:46:18.306238 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2718 (0'0,173'2718] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2716 active] op_applied on replica on version 173'2717
2014-06-12 14:46:18.306261 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.306262 7f9d7795c700  5 -- op tracker -- , seq: 747758, time: 2014-06-12 14:46:18.306258, event: done, op: osd_sub_op(client.4306.0:144130 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2717 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.306384 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe07d85/pglog_3.4d0/0//-1
2014-06-12 14:46:18.306507 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.306559 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.306587 7f9d8096e700 15 _omap_rmkeys meta/1fe07d85/pglog_3.4d0/0//-1
2014-06-12 14:46:18.306595 7f9d8096e700 10 _omap_rmkeys meta/1fe07d85/pglog_3.4d0/0//-1 r = 0
2014-06-12 14:46:18.306601 7f9d8096e700 15 _omap_setkeys meta/1fe07d85/pglog_3.4d0/0//-1
2014-06-12 14:46:18.306620 7f9d8096e700 15 _write 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 3256320~937984
2014-06-12 14:46:18.306765 7f9d8096e700 10 lookup_strip_header done  cid 3.4d0_head oid b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3
2014-06-12 14:46:18.307039 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:18.311054 7f9d8096e700 10 _generic_write 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 3256320~937984 = 0
2014-06-12 14:46:18.311178 7f9d8096e700 15 _setattrs 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3
2014-06-12 14:46:18.311221 7f9d8096e700 10 _setattrs 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 = 0
2014-06-12 14:46:18.311230 7f9d8096e700 15 _setattrs 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3
2014-06-12 14:46:18.311238 7f9d8096e700 10 _setattrs 3.4d0_head/b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 = 0
2014-06-12 14:46:18.311579 7f9d8096e700 20 set_header setting 16801 cid 3.4d0_head oid b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 parent seq 0
2014-06-12 14:46:18.311689 7f9d8096e700 20 set_header setting 2294 cid meta oid 1fe07d85/pglog_3.4d0/0//-1 parent seq 0
2014-06-12 14:46:18.311706 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.311715 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:18.316946 7f9d8096e700 10 _do_op 0x652c240 seq 2563 r = 0, finisher 0x20aad180 0
2014-06-12 14:46:18.316971 7f9d8096e700 10 _finish_op 0x652c240 seq 2563 osr(3.4d0 0x276d840)/0x276d840
2014-06-12 14:46:18.316987 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479320 (0 active)
2014-06-12 14:46:18.317004 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.316991 7f9d7715b700  5 -- op tracker -- , seq: 747759, time: 2014-06-12 14:46:18.316980, event: commit_sent, op: osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.317643 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2718 (0'0,173'2718] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2716 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:18.317698 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:18.317782 7f9d7795c700  5 -- op tracker -- , seq: 747759, time: 2014-06-12 14:46:18.317775, event: sub_op_applied, op: osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.317793 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2718 (0'0,173'2718] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2717 active] sub_op_modify_applied on 0x296ee00 op osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.317809 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2718 (0'0,173'2718] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2717 active] op_applied on primary on version 173'2718
2014-06-12 14:46:18.317819 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d0( v 173'2718 (0'0,173'2718] local-les=142 n=292 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2717 active] op_applied on replica on version 173'2718
2014-06-12 14:46:18.317846 7f9d7795c700  5 -- op tracker -- , seq: 747759, time: 2014-06-12 14:46:18.317842, event: done, op: osd_sub_op(client.4306.0:144131 3.4d0 b0c1a4d0/rbd_data.10a96b8b4567.0000000000000f9f/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.367248 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:18.386908 7f9d56432700  5 -- op tracker -- , seq: 747760, time: 2014-06-12 14:46:18.385702, event: header_read, op: osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.386919 7f9d56432700  5 -- op tracker -- , seq: 747760, time: 2014-06-12 14:46:18.385708, event: throttled, op: osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.386928 7f9d56432700  5 -- op tracker -- , seq: 747760, time: 2014-06-12 14:46:18.386863, event: all_read, op: osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.386936 7f9d56432700  5 -- op tracker -- , seq: 747760, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.386941 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.386948 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:18.386956 7f9d56432700 15 osd.2 173 enqueue_op 0x124e3700 prio 127 cost 516903 latency 0.001253 osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.389656 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.389686 7f9d62bd6700 10 osd.2 173 dequeue_op 0x124e3700 prio 127 cost 516903 latency 0.003982 osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2993 (0'0,173'2993] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2992 active]
2014-06-12 14:46:18.389744 7f9d62bd6700  5 -- op tracker -- , seq: 747760, time: 2014-06-12 14:46:18.389738, event: reached_pg, op: osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.389759 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2993 (0'0,173'2993] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2992 active] handle_message: 0x124e3700
2014-06-12 14:46:18.389773 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2993 (0'0,173'2993] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2992 active] sub_op_modify trans 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 v 173'2994 (transaction) 174
2014-06-12 14:46:18.389799 7f9d62bd6700  5 -- op tracker -- , seq: 747760, time: 2014-06-12 14:46:18.389795, event: started, op: osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.389829 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2993 (0'0,173'2993] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2992 active] append_log log((0'0,173'2993], crt=0'0) [173'2994 (173'2993) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145015 2014-06-12 08:44:42.064322]
2014-06-12 14:46:18.389858 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2994 (0'0,173'2994] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2992 active] add_log_entry 173'2994 (173'2993) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145015 2014-06-12 08:44:42.064322
2014-06-12 14:46:18.389883 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2994 (0'0,173'2994] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2992 active] append_log  adding 1 keys
2014-06-12 14:46:18.389919 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2994, trimmed: 
2014-06-12 14:46:18.389947 7f9d62bd6700  5 -- op tracker -- , seq: 747760, time: 2014-06-12 14:46:18.389942, event: started, op: osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.389958 7f9d62bd6700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:18.389962 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.389964 7f9d62bd6700  5 queue_op 0x69ccf00 seq 2796 osr(3.458 0x276d960) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:18.389970 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2994 (0'0,173'2994] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2992 active] do_sub_op osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.389987 7f9d62bd6700 10 osd.2 173 dequeue_op 0x124e3700 finish
2014-06-12 14:46:18.389991 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.389994 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.393628 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:18.393648 7f9d8016d700  5 _do_op 0x69ccf00 seq 2796 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:18.393656 7f9d8016d700 10 _do_transaction on 0x601a0f0
2014-06-12 14:46:18.393668 7f9d8016d700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:18.393963 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:18.394000 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.394061 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.394087 7f9d8016d700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:18.394097 7f9d8016d700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:18.394105 7f9d8016d700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:18.394124 7f9d8016d700 15 _write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 458752~516096
2014-06-12 14:46:18.394499 7f9d8016d700 10 lookup_strip_header done  cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:18.394637 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:18.396721 7f9d8016d700 10 _generic_write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 458752~516096 = 0
2014-06-12 14:46:18.396801 7f9d8016d700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:18.396823 7f9d8016d700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:18.396832 7f9d8016d700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:18.396842 7f9d8016d700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:18.397090 7f9d8016d700 20 set_header setting 16936 cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 parent seq 0
2014-06-12 14:46:18.397184 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.397201 7f9d8016d700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:18.397216 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:18.402924 7f9d8016d700 10 _do_op 0x69ccf00 seq 2796 r = 0, finisher 0x88d8bc0 0
2014-06-12 14:46:18.403005 7f9d7715b700  5 -- op tracker -- , seq: 747760, time: 2014-06-12 14:46:18.402995, event: commit_sent, op: osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.403017 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2994 (0'0,173'2994] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2992 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:18.403045 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:18.403109 7f9d8016d700 10 _finish_op 0x69ccf00 seq 2796 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:18.403153 7f9d7795c700  5 -- op tracker -- , seq: 747760, time: 2014-06-12 14:46:18.403147, event: sub_op_applied, op: osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.403164 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2994 (0'0,173'2994] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active] sub_op_modify_applied on 0x601a000 op osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.403199 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2994 (0'0,173'2994] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active] op_applied on primary on version 173'2994
2014-06-12 14:46:18.403210 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2994 (0'0,173'2994] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active] op_applied on replica on version 173'2994
2014-06-12 14:46:18.403308 7f9d8016d700  5 -- op tracker -- , seq: 747760, time: 2014-06-12 14:46:18.403293, event: done, op: osd_sub_op(client.4308.0:145015 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.403390 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:18.403418 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.472317 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:18.472347 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:18.472352 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:18.472354 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:18.472356 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:18.472358 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 12; oldest is 39.028871 seconds old
2014-06-12 14:46:18.603042 7f9d56432700  5 -- op tracker -- , seq: 747761, time: 2014-06-12 14:46:18.602968, event: header_read, op: osd_sub_op_reply(client.4303.0:144628 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.603057 7f9d56432700  5 -- op tracker -- , seq: 747761, time: 2014-06-12 14:46:18.602974, event: throttled, op: osd_sub_op_reply(client.4303.0:144628 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.603065 7f9d56432700  5 -- op tracker -- , seq: 747761, time: 2014-06-12 14:46:18.602993, event: all_read, op: osd_sub_op_reply(client.4303.0:144628 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.603072 7f9d56432700  5 -- op tracker -- , seq: 747761, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4303.0:144628 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.603077 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4303.0:144628 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:18.603083 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:18.603091 7f9d56432700 15 osd.2 173 enqueue_op 0x124e3d00 prio 196 cost 0 latency 0.000123 osd_sub_op_reply(client.4303.0:144628 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:18.603331 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.603347 7f9d683e1700 10 osd.2 173 dequeue_op 0x124e3d00 prio 196 cost 0 latency 0.000379 osd_sub_op_reply(client.4303.0:144628 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3093 active+clean]
2014-06-12 14:46:18.603386 7f9d683e1700  5 -- op tracker -- , seq: 747761, time: 2014-06-12 14:46:18.603380, event: reached_pg, op: osd_sub_op_reply(client.4303.0:144628 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.603402 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3093 active+clean] handle_message: 0x124e3d00
2014-06-12 14:46:18.603418 7f9d683e1700  5 -- op tracker -- , seq: 747761, time: 2014-06-12 14:46:18.603415, event: started, op: osd_sub_op_reply(client.4303.0:144628 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.603424 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3093 active+clean] sub_op_modify_reply: tid 236067 op  ack_type 4 from 26
2014-06-12 14:46:18.603443 7f9d683e1700  5 -- op tracker -- , seq: 747086, time: 2014-06-12 14:46:18.603436, event: sub_op_commit_rec, op: osd_op(client.4303.0:144628 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:18.603450 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3093 active+clean] repop_all_applied: repop tid 236067 all applied 
2014-06-12 14:46:18.603460 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3093 active+clean] eval_repop repgather(0x5312180 173'3095 rep_tid=236067 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144628 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:18.603475 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3093 active+clean]  sending ack on repgather(0x5312180 173'3095 rep_tid=236067 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144628 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x2ed84780
2014-06-12 14:46:18.603505 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3094 crt=0'0 lcod 173'3093 mlcod 173'3093 active+clean] repop_all_committed: repop tid 236067 all committed 
2014-06-12 14:46:18.603515 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3093 active+clean] eval_repop repgather(0x5312180 173'3095 rep_tid=236067 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144628 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:18.603530 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3093 active+clean] log_op_stats osd_op(client.4303.0:144628 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 39.160017 lat 39.160043
2014-06-12 14:46:18.603546 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3093 active+clean] publish_stats_to_osd 173:3561
2014-06-12 14:46:18.603557 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3093 active+clean]  sending commit on repgather(0x5312180 173'3095 rep_tid=236067 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144628 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x8df3b80
2014-06-12 14:46:18.603586 7f9d683e1700  5 -- op tracker -- , seq: 747086, time: 2014-06-12 14:46:18.603581, event: commit_sent, op: osd_op(client.4303.0:144628 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:18.603594 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3093 active+clean]  requeue_ops 
2014-06-12 14:46:18.603604 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3094 active+clean]  removing repgather(0x5312180 173'3095 rep_tid=236067 committed?=1 applied?=1 op=osd_op(client.4303.0:144628 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:18.603618 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3094 active+clean]    q front is repgather(0x5312180 173'3095 rep_tid=236067 committed?=1 applied?=1 op=osd_op(client.4303.0:144628 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:18.603630 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3094 active+clean] remove_repop repgather(0x5312180 173'3095 rep_tid=236067 committed?=1 applied?=1 op=osd_op(client.4303.0:144628 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:18.603643 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3094 active+clean]  requeue_ops 
2014-06-12 14:46:18.603662 7f9d683e1700  5 -- op tracker -- , seq: 747086, time: 2014-06-12 14:46:18.603657, event: done, op: osd_op(client.4303.0:144628 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~1032192] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:18.603680 7f9d683e1700 10 osd.2 173 dequeue_op 0x124e3d00 finish
2014-06-12 14:46:18.603686 7f9d683e1700  5 -- op tracker -- , seq: 747761, time: 2014-06-12 14:46:18.603683, event: done, op: osd_sub_op_reply(client.4303.0:144628 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.603701 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.603705 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.615051 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:18.615103 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:18.787181 7f9d56432700  5 -- op tracker -- , seq: 747762, time: 2014-06-12 14:46:18.775414, event: header_read, op: osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.787205 7f9d56432700  5 -- op tracker -- , seq: 747762, time: 2014-06-12 14:46:18.775417, event: throttled, op: osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.787216 7f9d56432700  5 -- op tracker -- , seq: 747762, time: 2014-06-12 14:46:18.787099, event: all_read, op: osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.787226 7f9d56432700  5 -- op tracker -- , seq: 747762, time: 0.000000, event: dispatched, op: osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.787238 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:18.787249 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:18.787268 7f9d56432700 15 osd.2 173 enqueue_op 0xe20bd00 prio 127 cost 3097383 latency 0.011853 osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.787322 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.787340 7f9d65bdc700 10 osd.2 173 dequeue_op 0xe20bd00 prio 127 cost 3097383 latency 0.011926 osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2635 (0'0,173'2635] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2634 active]
2014-06-12 14:46:18.787382 7f9d65bdc700  5 -- op tracker -- , seq: 747762, time: 2014-06-12 14:46:18.787378, event: reached_pg, op: osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.787396 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2635 (0'0,173'2635] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2634 active] handle_message: 0xe20bd00
2014-06-12 14:46:18.787414 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2635 (0'0,173'2635] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2634 active] sub_op_modify trans 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 v 173'2636 (transaction) 174
2014-06-12 14:46:18.787444 7f9d65bdc700  5 -- op tracker -- , seq: 747762, time: 2014-06-12 14:46:18.787440, event: started, op: osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.787485 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2635 (0'0,173'2635] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2634 active] append_log log((0'0,173'2635], crt=0'0) [173'2636 (173'2635) modify   3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 by client.4286.0:144488 2014-06-12 08:43:09.496966]
2014-06-12 14:46:18.787628 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2636 (0'0,173'2636] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2634 active] add_log_entry 173'2636 (173'2635) modify   3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 by client.4286.0:144488 2014-06-12 08:43:09.496966
2014-06-12 14:46:18.787656 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2636 (0'0,173'2636] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2634 active] append_log  adding 1 keys
2014-06-12 14:46:18.787699 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2636, trimmed: 
2014-06-12 14:46:18.787730 7f9d65bdc700  5 -- op tracker -- , seq: 747762, time: 2014-06-12 14:46:18.787724, event: started, op: osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.787745 7f9d65bdc700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:18.787750 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:18.787753 7f9d65bdc700  5 queue_op 0x58f8660 seq 2456 osr(3.1c7 0x276d170) 3098869 bytes   (queue has 1 ops and 3098869 bytes)
2014-06-12 14:46:18.787771 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:46:18.787765 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2636 (0'0,173'2636] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2634 active] do_sub_op osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.787781 7f9d65bdc700 10 osd.2 173 dequeue_op 0xe20bd00 finish
2014-06-12 14:46:18.787781 7f9d7915f700  5 _do_op 0x58f8660 seq 2456 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:46:18.787785 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.787787 7f9d7915f700 10 _do_transaction on 0x678f970
2014-06-12 14:46:18.787788 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:18.787821 7f9d7915f700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:18.788002 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:18.788027 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:18.788093 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:18.788118 7f9d7915f700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:18.788123 7f9d7915f700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:46:18.788129 7f9d7915f700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:18.788143 7f9d7915f700 15 _write 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 335872~3096576
2014-06-12 14:46:18.788438 7f9d7915f700 10 lookup_strip_header done  cid 3.1c7_head oid 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3
2014-06-12 14:46:18.788808 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:18.803095 7f9d7915f700 10 _generic_write 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 335872~3096576 = 0
2014-06-12 14:46:18.803705 7f9d7915f700 15 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3
2014-06-12 14:46:18.803731 7f9d7915f700 10 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:18.803740 7f9d7915f700 15 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3
2014-06-12 14:46:18.803752 7f9d7915f700 10 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:18.805232 7f9d7915f700 20 set_header setting 16919 cid 3.1c7_head oid 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 parent seq 0
2014-06-12 14:46:18.806321 7f9d7915f700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:46:18.806353 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:18.806368 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:18.822344 7f9d7915f700 10 _do_op 0x58f8660 seq 2456 r = 0, finisher 0x96583a0 0
2014-06-12 14:46:18.822368 7f9d7915f700 10 _finish_op 0x58f8660 seq 2456 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:18.822374 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:46:18.822381 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:18.822388 7f9d7715b700  5 -- op tracker -- , seq: 747762, time: 2014-06-12 14:46:18.822377, event: commit_sent, op: osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.822404 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2636 (0'0,173'2636] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2634 active] sub_op_modify_commit on op osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:18.822444 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:18.822481 7f9d7795c700  5 -- op tracker -- , seq: 747762, time: 2014-06-12 14:46:18.822475, event: sub_op_applied, op: osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.822490 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2636 (0'0,173'2636] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2635 active] sub_op_modify_applied on 0x678f880 op osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:18.822507 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2636 (0'0,173'2636] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2635 active] op_applied on primary on version 173'2636
2014-06-12 14:46:18.822523 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2636 (0'0,173'2636] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2635 active] op_applied on replica on version 173'2636
2014-06-12 14:46:18.822551 7f9d7795c700  5 -- op tracker -- , seq: 747762, time: 2014-06-12 14:46:18.822547, event: done, op: osd_sub_op(client.4286.0:144488 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:18.959069 7f9d56432700  5 -- op tracker -- , seq: 747763, time: 2014-06-12 14:46:18.958936, event: header_read, op: osd_sub_op_reply(client.4303.0:144629 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.959094 7f9d56432700  5 -- op tracker -- , seq: 747763, time: 2014-06-12 14:46:18.958941, event: throttled, op: osd_sub_op_reply(client.4303.0:144629 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.959108 7f9d56432700  5 -- op tracker -- , seq: 747763, time: 2014-06-12 14:46:18.958973, event: all_read, op: osd_sub_op_reply(client.4303.0:144629 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.959116 7f9d56432700  5 -- op tracker -- , seq: 747763, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4303.0:144629 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.959123 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4303.0:144629 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:18.959131 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:18.959145 7f9d56432700 15 osd.2 173 enqueue_op 0xe20ad00 prio 196 cost 0 latency 0.000209 osd_sub_op_reply(client.4303.0:144629 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:18.959211 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:18.959226 7f9d64bda700 10 osd.2 173 dequeue_op 0xe20ad00 prio 196 cost 0 latency 0.000289 osd_sub_op_reply(client.4303.0:144629 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0) v2 pg pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3094 active+clean]
2014-06-12 14:46:18.959263 7f9d64bda700  5 -- op tracker -- , seq: 747763, time: 2014-06-12 14:46:18.959258, event: reached_pg, op: osd_sub_op_reply(client.4303.0:144629 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.959277 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3094 active+clean] handle_message: 0xe20ad00
2014-06-12 14:46:18.959299 7f9d64bda700  5 -- op tracker -- , seq: 747763, time: 2014-06-12 14:46:18.959295, event: started, op: osd_sub_op_reply(client.4303.0:144629 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.959306 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3094 active+clean] sub_op_modify_reply: tid 236068 op  ack_type 4 from 26
2014-06-12 14:46:18.959331 7f9d64bda700  5 -- op tracker -- , seq: 747087, time: 2014-06-12 14:46:18.959320, event: sub_op_commit_rec, op: osd_op(client.4303.0:144629 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:18.959339 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3094 active+clean] repop_all_applied: repop tid 236068 all applied 
2014-06-12 14:46:18.959351 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3094 active+clean] eval_repop repgather(0x15e56840 173'3096 rep_tid=236068 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144629 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:18.959368 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3094 active+clean]  sending ack on repgather(0x15e56840 173'3096 rep_tid=236068 committed?=0 applied?=1 lock=0 op=osd_op(client.4303.0:144629 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x2b43680
2014-06-12 14:46:18.959404 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'3095 crt=0'0 lcod 173'3094 mlcod 173'3094 active+clean] repop_all_committed: repop tid 236068 all committed 
2014-06-12 14:46:18.959415 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3094 active+clean] eval_repop repgather(0x15e56840 173'3096 rep_tid=236068 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144629 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:18.959432 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3094 active+clean] log_op_stats osd_op(client.4303.0:144629 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4 inb 37143 outb 0 rlat 39.499513 lat 39.499542
2014-06-12 14:46:18.959452 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3094 active+clean] publish_stats_to_osd 173:3562
2014-06-12 14:46:18.959472 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3094 active+clean]  sending commit on repgather(0x15e56840 173'3096 rep_tid=236068 committed?=1 applied?=1 lock=0 op=osd_op(client.4303.0:144629 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4) 0x34e4d400
2014-06-12 14:46:18.959501 7f9d64bda700  5 -- op tracker -- , seq: 747087, time: 2014-06-12 14:46:18.959496, event: commit_sent, op: osd_op(client.4303.0:144629 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:18.959512 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3094 active+clean]  requeue_ops 
2014-06-12 14:46:18.959524 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3095 active+clean]  removing repgather(0x15e56840 173'3096 rep_tid=236068 committed?=1 applied?=1 op=osd_op(client.4303.0:144629 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:18.959541 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3095 active+clean]    q front is repgather(0x15e56840 173'3096 rep_tid=236068 committed?=1 applied?=1 op=osd_op(client.4303.0:144629 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:18.959557 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3095 active+clean] remove_repop repgather(0x15e56840 173'3096 rep_tid=236068 committed?=1 applied?=1 op=osd_op(client.4303.0:144629 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.892731eb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:18.959571 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3095 active+clean]  requeue_ops 
2014-06-12 14:46:18.959604 7f9d64bda700  5 -- op tracker -- , seq: 747087, time: 2014-06-12 14:46:18.959598, event: done, op: osd_op(client.4303.0:144629 rbd_data.10a66b8b4567.0000000000000fdb [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.892731eb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:18.959626 7f9d64bda700 10 osd.2 173 dequeue_op 0xe20ad00 finish
2014-06-12 14:46:18.959633 7f9d64bda700  5 -- op tracker -- , seq: 747763, time: 2014-06-12 14:46:18.959630, event: done, op: osd_sub_op_reply(client.4303.0:144629 3.1eb 892731eb/rbd_data.10a66b8b4567.0000000000000fdb/head//3 [] ondisk, result = 0)
2014-06-12 14:46:18.959650 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:18.959654 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.007202 7f9d56432700  5 -- op tracker -- , seq: 747764, time: 2014-06-12 14:46:19.004115, event: header_read, op: osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.007217 7f9d56432700  5 -- op tracker -- , seq: 747764, time: 2014-06-12 14:46:19.004122, event: throttled, op: osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.007225 7f9d56432700  5 -- op tracker -- , seq: 747764, time: 2014-06-12 14:46:19.007140, event: all_read, op: osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.007232 7f9d56432700  5 -- op tracker -- , seq: 747764, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.007237 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:19.007244 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:19.007251 7f9d56432700 15 osd.2 173 enqueue_op 0xe20a700 prio 127 cost 1032999 latency 0.003136 osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.007278 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.007284 7f9d6abe6700 10 osd.2 173 dequeue_op 0xe20a700 prio 127 cost 1032999 latency 0.003169 osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2994 (0'0,173'2994] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active]
2014-06-12 14:46:19.007304 7f9d6abe6700  5 -- op tracker -- , seq: 747764, time: 2014-06-12 14:46:19.007301, event: reached_pg, op: osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.007315 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2994 (0'0,173'2994] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active] handle_message: 0xe20a700
2014-06-12 14:46:19.007326 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2994 (0'0,173'2994] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active] sub_op_modify trans 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 v 173'2995 (transaction) 174
2014-06-12 14:46:19.007342 7f9d6abe6700  5 -- op tracker -- , seq: 747764, time: 2014-06-12 14:46:19.007338, event: started, op: osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.007358 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2994 (0'0,173'2994] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active] append_log log((0'0,173'2994], crt=0'0) [173'2995 (173'2994) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145016 2014-06-12 08:44:42.123383]
2014-06-12 14:46:19.007381 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2995 (0'0,173'2995] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active] add_log_entry 173'2995 (173'2994) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145016 2014-06-12 08:44:42.123383
2014-06-12 14:46:19.007401 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2995 (0'0,173'2995] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active] append_log  adding 1 keys
2014-06-12 14:46:19.007436 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2995, trimmed: 
2014-06-12 14:46:19.007458 7f9d6abe6700  5 -- op tracker -- , seq: 747764, time: 2014-06-12 14:46:19.007454, event: started, op: osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.007469 7f9d6abe6700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:19.007473 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:19.007475 7f9d6abe6700  5 queue_op 0x7ca8780 seq 2797 osr(3.458 0x276d960) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:19.007482 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2995 (0'0,173'2995] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active] do_sub_op osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.007496 7f9d6abe6700 10 osd.2 173 dequeue_op 0xe20a700 finish
2014-06-12 14:46:19.007499 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.007502 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.007514 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:19.007522 7f9d7d968700  5 _do_op 0x7ca8780 seq 2797 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:19.007528 7f9d7d968700 10 _do_transaction on 0x2a29b7b0
2014-06-12 14:46:19.007537 7f9d7d968700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:19.007674 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:19.007701 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:19.007746 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:19.007770 7f9d7d968700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:19.007775 7f9d7d968700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:19.007780 7f9d7d968700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:19.007980 7f9d7d968700 15 _write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 974848~1032192
2014-06-12 14:46:19.008140 7f9d7d968700 10 lookup_strip_header done  cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:19.008304 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:19.012873 7f9d7d968700 10 _generic_write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 974848~1032192 = 0
2014-06-12 14:46:19.013017 7f9d7d968700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:19.013037 7f9d7d968700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:19.013042 7f9d7d968700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:19.013050 7f9d7d968700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:19.013442 7f9d7d968700 20 set_header setting 16936 cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 parent seq 0
2014-06-12 14:46:19.013565 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:19.013580 7f9d7d968700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:19.013589 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:19.031556 7f9d7d968700 10 _do_op 0x7ca8780 seq 2797 r = 0, finisher 0xa6495e0 0
2014-06-12 14:46:19.031620 7f9d7d968700 10 _finish_op 0x7ca8780 seq 2797 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:19.031652 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:19.031676 7f9d7795c700  5 -- op tracker -- , seq: 747764, time: 2014-06-12 14:46:19.031665, event: sub_op_applied, op: osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.031685 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:19.031687 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2995 (0'0,173'2995] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active] sub_op_modify_applied on 0x2a29b6c0 op osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.031718 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:19.031742 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2995 (0'0,173'2995] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active] op_applied on primary on version 173'2995
2014-06-12 14:46:19.031753 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2995 (0'0,173'2995] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active] op_applied on replica on version 173'2995
2014-06-12 14:46:19.031810 7f9d7715b700  5 -- op tracker -- , seq: 747764, time: 2014-06-12 14:46:19.031804, event: commit_sent, op: osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.031819 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2995 (0'0,173'2995] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2993 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:19.031836 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:19.031870 7f9d7715b700  5 -- op tracker -- , seq: 747764, time: 2014-06-12 14:46:19.031865, event: done, op: osd_sub_op(client.4308.0:145016 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.057886 7f9d56432700  5 -- op tracker -- , seq: 747765, time: 2014-06-12 14:46:19.057773, event: header_read, op: osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.057902 7f9d56432700  5 -- op tracker -- , seq: 747765, time: 2014-06-12 14:46:19.057778, event: throttled, op: osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.057911 7f9d56432700  5 -- op tracker -- , seq: 747765, time: 2014-06-12 14:46:19.057841, event: all_read, op: osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.057920 7f9d56432700  5 -- op tracker -- , seq: 747765, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.057927 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:19.057934 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:19.057944 7f9d56432700 15 osd.2 173 enqueue_op 0x2b015c00 prio 127 cost 21287 latency 0.000171 osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.057982 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.057996 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2b015c00 prio 127 cost 21287 latency 0.000223 osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.67d( v 173'2507 (0'0,173'2507] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2506 active]
2014-06-12 14:46:19.058024 7f9d6a3e5700  5 -- op tracker -- , seq: 747765, time: 2014-06-12 14:46:19.058020, event: reached_pg, op: osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.058036 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2507 (0'0,173'2507] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2506 active] handle_message: 0x2b015c00
2014-06-12 14:46:19.058048 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2507 (0'0,173'2507] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2506 active] sub_op_modify trans 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 v 173'2508 (transaction) 174
2014-06-12 14:46:19.058068 7f9d6a3e5700  5 -- op tracker -- , seq: 747765, time: 2014-06-12 14:46:19.058064, event: started, op: osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.058088 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2507 (0'0,173'2507] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2506 active] append_log log((0'0,173'2507], crt=0'0) [173'2508 (173'677) modify   6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 by client.4303.0:144814 2014-06-12 08:45:21.084362]
2014-06-12 14:46:19.058117 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2508 (0'0,173'2508] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2506 active] add_log_entry 173'2508 (173'677) modify   6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 by client.4303.0:144814 2014-06-12 08:45:21.084362
2014-06-12 14:46:19.058142 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2508 (0'0,173'2508] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2506 active] append_log  adding 1 keys
2014-06-12 14:46:19.058190 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2508, trimmed: 
2014-06-12 14:46:19.058221 7f9d6a3e5700  5 -- op tracker -- , seq: 747765, time: 2014-06-12 14:46:19.058215, event: started, op: osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.058235 7f9d6a3e5700  5 queue_transactions existing osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:19.058239 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:19.058241 7f9d6a3e5700  5 queue_op 0x829a540 seq 2286 osr(3.67d 0x276df70) 22773 bytes   (queue has 1 ops and 22773 bytes)
2014-06-12 14:46:19.058252 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x524cb40 (1 active)
2014-06-12 14:46:19.058258 7f9d83173700  5 _do_op 0x829a540 seq 2286 osr(3.67d 0x276df70)/0x276df70 start
2014-06-12 14:46:19.058249 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.67d( v 173'2508 (0'0,173'2508] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2506 active] do_sub_op osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.058262 7f9d83173700 10 _do_transaction on 0x1da3f0b0
2014-06-12 14:46:19.058263 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2b015c00 finish
2014-06-12 14:46:19.058266 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.058268 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.058268 7f9d83173700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:19.058836 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:19.058955 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:19.059092 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:19.059172 7f9d83173700 15 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:19.059216 7f9d83173700 10 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1 r = 0
2014-06-12 14:46:19.059224 7f9d83173700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:19.059242 7f9d83173700 15 _write 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 0~20480
2014-06-12 14:46:19.074067 7f9d83173700 10 lookup_strip_header done  cid 3.67d_head oid 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:19.074101 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:19.074193 7f9d83173700 10 _generic_write 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 0~20480 = 0
2014-06-12 14:46:19.074207 7f9d83173700 15 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:19.074230 7f9d83173700 10 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 = 0
2014-06-12 14:46:19.074236 7f9d83173700 15 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:19.074244 7f9d83173700 10 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 = 0
2014-06-12 14:46:19.074362 7f9d83173700 20 set_header setting 17507 cid 3.67d_head oid 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 parent seq 0
2014-06-12 14:46:19.074387 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:19.074401 7f9d83173700 20 set_header setting 2272 cid meta oid 1fe0e10b/pglog_3.67d/0//-1 parent seq 0
2014-06-12 14:46:19.074412 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:19.074625 7f9d83173700 10 _do_op 0x829a540 seq 2286 r = 0, finisher 0x59faba0 0
2014-06-12 14:46:19.074639 7f9d83173700 10 _finish_op 0x829a540 seq 2286 osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:19.074647 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x524cb40 (0 active)
2014-06-12 14:46:19.074650 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:19.074659 7f9d7715b700  5 -- op tracker -- , seq: 747765, time: 2014-06-12 14:46:19.074648, event: commit_sent, op: osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.074675 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2508 (0'0,173'2508] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2506 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:19.074702 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:19.074762 7f9d7795c700  5 -- op tracker -- , seq: 747765, time: 2014-06-12 14:46:19.074756, event: sub_op_applied, op: osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.074769 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2508 (0'0,173'2508] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2507 active] sub_op_modify_applied on 0x1da3efc0 op osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.074784 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2508 (0'0,173'2508] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2507 active] op_applied on primary on version 173'2508
2014-06-12 14:46:19.074793 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2508 (0'0,173'2508] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2507 active] op_applied on replica on version 173'2508
2014-06-12 14:46:19.074820 7f9d7795c700  5 -- op tracker -- , seq: 747765, time: 2014-06-12 14:46:19.074816, event: done, op: osd_sub_op(client.4303.0:144814 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.115627 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:19.115726 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:19.122426 7f9d5a16f700  5 -- op tracker -- , seq: 747766, time: 2014-06-12 14:46:19.121152, event: header_read, op: osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.122438 7f9d5a16f700  5 -- op tracker -- , seq: 747766, time: 2014-06-12 14:46:19.121155, event: throttled, op: osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.122446 7f9d5a16f700  5 -- op tracker -- , seq: 747766, time: 2014-06-12 14:46:19.122368, event: all_read, op: osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.122470 7f9d5a16f700  5 -- op tracker -- , seq: 747766, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.122475 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:19.122482 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:19.122489 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2519cb00 prio 127 cost 516903 latency 0.001337 osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.122567 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.122576 7f9d66bde700 10 osd.2 173 dequeue_op 0x2519cb00 prio 127 cost 516903 latency 0.001423 osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2661 (0'0,173'2661] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2660 active]
2014-06-12 14:46:19.122600 7f9d66bde700  5 -- op tracker -- , seq: 747766, time: 2014-06-12 14:46:19.122596, event: reached_pg, op: osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.122610 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2661 (0'0,173'2661] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2660 active] handle_message: 0x2519cb00
2014-06-12 14:46:19.122622 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2661 (0'0,173'2661] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2660 active] sub_op_modify trans f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 v 173'2662 (transaction) 174
2014-06-12 14:46:19.122639 7f9d66bde700  5 -- op tracker -- , seq: 747766, time: 2014-06-12 14:46:19.122635, event: started, op: osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.122655 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2661 (0'0,173'2661] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2660 active] append_log log((0'0,173'2661], crt=0'0) [173'2662 (173'2660) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146407 2014-06-12 08:41:29.656432]
2014-06-12 14:46:19.122677 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2662 (0'0,173'2662] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2660 active] add_log_entry 173'2662 (173'2660) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146407 2014-06-12 08:41:29.656432
2014-06-12 14:46:19.122699 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2662 (0'0,173'2662] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2660 active] append_log  adding 1 keys
2014-06-12 14:46:19.122731 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2662, trimmed: 
2014-06-12 14:46:19.122753 7f9d66bde700  5 -- op tracker -- , seq: 747766, time: 2014-06-12 14:46:19.122748, event: started, op: osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.122763 7f9d66bde700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:19.122767 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:19.122769 7f9d66bde700  5 queue_op 0x1ab079e0 seq 2472 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:19.122777 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2662 (0'0,173'2662] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2660 active] do_sub_op osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.122792 7f9d66bde700 10 osd.2 173 dequeue_op 0x2519cb00 finish
2014-06-12 14:46:19.122796 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.122798 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.122805 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:46:19.122810 7f9d7c165700  5 _do_op 0x1ab079e0 seq 2472 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:19.122815 7f9d7c165700 10 _do_transaction on 0x2e9b2d30
2014-06-12 14:46:19.122820 7f9d7c165700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:19.123070 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:19.123095 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:19.123143 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:19.123167 7f9d7c165700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:19.123173 7f9d7c165700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:19.123179 7f9d7c165700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:19.123196 7f9d7c165700 15 _write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 1327104~516096
2014-06-12 14:46:19.123535 7f9d7c165700 10 lookup_strip_header done  cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:19.123662 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:19.125764 7f9d7c165700 10 _generic_write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 1327104~516096 = 0
2014-06-12 14:46:19.125833 7f9d7c165700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:19.125852 7f9d7c165700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:19.125859 7f9d7c165700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:19.125868 7f9d7c165700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:19.126087 7f9d7c165700 20 set_header setting 16921 cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 parent seq 0
2014-06-12 14:46:19.126159 7f9d7c165700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:19.126177 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:19.126186 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:19.128729 7f9d7c165700 10 _do_op 0x1ab079e0 seq 2472 r = 0, finisher 0x2a3a680 0
2014-06-12 14:46:19.128798 7f9d7715b700  5 -- op tracker -- , seq: 747766, time: 2014-06-12 14:46:19.128781, event: commit_sent, op: osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.128819 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2662 (0'0,173'2662] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2660 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:19.128859 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:19.128913 7f9d7c165700 10 _finish_op 0x1ab079e0 seq 2472 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:19.128939 7f9d7795c700  5 -- op tracker -- , seq: 747766, time: 2014-06-12 14:46:19.128933, event: sub_op_applied, op: osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.128947 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2662 (0'0,173'2662] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2661 active] sub_op_modify_applied on 0x2e9b2c40 op osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.128962 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2662 (0'0,173'2662] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2661 active] op_applied on primary on version 173'2662
2014-06-12 14:46:19.128973 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2662 (0'0,173'2662] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2661 active] op_applied on replica on version 173'2662
2014-06-12 14:46:19.129005 7f9d7c165700  5 -- op tracker -- , seq: 747766, time: 2014-06-12 14:46:19.128996, event: done, op: osd_sub_op(client.4316.0:146407 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2662 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.129038 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:46:19.129042 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:19.132689 7f9d5501e700  5 -- op tracker -- , seq: 747767, time: 2014-06-12 14:46:19.130886, event: header_read, op: osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.132702 7f9d5501e700  5 -- op tracker -- , seq: 747767, time: 2014-06-12 14:46:19.130891, event: throttled, op: osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.132713 7f9d5501e700  5 -- op tracker -- , seq: 747767, time: 2014-06-12 14:46:19.132638, event: all_read, op: osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.132722 7f9d5501e700  5 -- op tracker -- , seq: 747767, time: 0.000000, event: dispatched, op: osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.132729 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:19.132736 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:19.132746 7f9d5501e700 15 osd.2 173 enqueue_op 0x3b42ae00 prio 127 cost 1061671 latency 0.001859 osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.132784 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.132793 7f9d69be4700 10 osd.2 173 dequeue_op 0x3b42ae00 prio 127 cost 1061671 latency 0.001907 osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.63d( v 173'2927 (0'0,173'2927] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2926 active]
2014-06-12 14:46:19.132817 7f9d69be4700  5 -- op tracker -- , seq: 747767, time: 2014-06-12 14:46:19.132813, event: reached_pg, op: osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.132828 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2927 (0'0,173'2927] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2926 active] handle_message: 0x3b42ae00
2014-06-12 14:46:19.132839 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2927 (0'0,173'2927] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2926 active] sub_op_modify trans adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 v 173'2928 (transaction) 174
2014-06-12 14:46:19.132856 7f9d69be4700  5 -- op tracker -- , seq: 747767, time: 2014-06-12 14:46:19.132852, event: started, op: osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.132879 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2927 (0'0,173'2927] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2926 active] append_log log((0'0,173'2927], crt=0'0) [173'2928 (173'862) modify   adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 by client.4304.0:144140 2014-06-12 08:45:21.504654]
2014-06-12 14:46:19.132904 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2928 (0'0,173'2928] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2926 active] add_log_entry 173'2928 (173'862) modify   adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 by client.4304.0:144140 2014-06-12 08:45:21.504654
2014-06-12 14:46:19.132927 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2928 (0'0,173'2928] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2926 active] append_log  adding 1 keys
2014-06-12 14:46:19.132961 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2928, trimmed: 
2014-06-12 14:46:19.132985 7f9d69be4700  5 -- op tracker -- , seq: 747767, time: 2014-06-12 14:46:19.132980, event: started, op: osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.132996 7f9d69be4700  5 queue_transactions existing osr(3.63d 0x276dc50)/0x276dc50
2014-06-12 14:46:19.133000 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:19.133002 7f9d69be4700  5 queue_op 0x7713560 seq 2662 osr(3.63d 0x276dc50) 1063157 bytes   (queue has 1 ops and 1063157 bytes)
2014-06-12 14:46:19.133011 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.63d( v 173'2928 (0'0,173'2928] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2926 active] do_sub_op osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.133015 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e96d80 (1 active)
2014-06-12 14:46:19.133026 7f9d69be4700 10 osd.2 173 dequeue_op 0x3b42ae00 finish
2014-06-12 14:46:19.133030 7f9d82972700  5 _do_op 0x7713560 seq 2662 osr(3.63d 0x276dc50)/0x276dc50 start
2014-06-12 14:46:19.133035 7f9d82972700 10 _do_transaction on 0xe0d70b0
2014-06-12 14:46:19.133030 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.133040 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.133042 7f9d82972700 15 _omap_setkeys meta/1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.133210 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.133230 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:19.133295 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:19.133317 7f9d82972700 15 _omap_rmkeys meta/1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.133324 7f9d82972700 10 _omap_rmkeys meta/1fe0c7cb/pglog_3.63d/0//-1 r = 0
2014-06-12 14:46:19.133329 7f9d82972700 15 _omap_setkeys meta/1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.133508 7f9d82972700 15 _write 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 0~1060864
2014-06-12 14:46:19.133730 7f9d82972700 10 lookup_strip_header done  cid 3.63d_head oid adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3
2014-06-12 14:46:19.134010 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:19.139003 7f9d82972700 10 _generic_write 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 0~1060864 = 0
2014-06-12 14:46:19.139184 7f9d82972700 15 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3
2014-06-12 14:46:19.139206 7f9d82972700 10 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 = 0
2014-06-12 14:46:19.139213 7f9d82972700 15 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3
2014-06-12 14:46:19.139223 7f9d82972700 10 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 = 0
2014-06-12 14:46:19.139681 7f9d82972700 20 set_header setting 17087 cid 3.63d_head oid adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 parent seq 0
2014-06-12 14:46:19.139849 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:19.139864 7f9d82972700 20 set_header setting 2208 cid meta oid 1fe0c7cb/pglog_3.63d/0//-1 parent seq 0
2014-06-12 14:46:19.139874 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:19.152326 7f9d82972700 10 _do_op 0x7713560 seq 2662 r = 0, finisher 0x2ef7dfc0 0
2014-06-12 14:46:19.152435 7f9d7715b700  5 -- op tracker -- , seq: 747767, time: 2014-06-12 14:46:19.152423, event: commit_sent, op: osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.152447 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2928 (0'0,173'2928] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2926 active] sub_op_modify_commit on op osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:19.152478 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:19.152600 7f9d82972700 10 _finish_op 0x7713560 seq 2662 osr(3.63d 0x276dc50)/0x276dc50
2014-06-12 14:46:19.152661 7f9d7795c700  5 -- op tracker -- , seq: 747767, time: 2014-06-12 14:46:19.152655, event: sub_op_applied, op: osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.152670 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2928 (0'0,173'2928] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] sub_op_modify_applied on 0xe0d6fc0 op osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.152686 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2928 (0'0,173'2928] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] op_applied on primary on version 173'2928
2014-06-12 14:46:19.152696 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2928 (0'0,173'2928] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] op_applied on replica on version 173'2928
2014-06-12 14:46:19.152787 7f9d82972700  5 -- op tracker -- , seq: 747767, time: 2014-06-12 14:46:19.152773, event: done, op: osd_sub_op(client.4304.0:144140 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.152854 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e96d80 (0 active)
2014-06-12 14:46:19.152879 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:19.159193 7f9d5a16f700  5 -- op tracker -- , seq: 747768, time: 2014-06-12 14:46:19.159130, event: header_read, op: osd_sub_op_reply(client.4308.0:145044 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.159208 7f9d5a16f700  5 -- op tracker -- , seq: 747768, time: 2014-06-12 14:46:19.159132, event: throttled, op: osd_sub_op_reply(client.4308.0:145044 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.159217 7f9d5a16f700  5 -- op tracker -- , seq: 747768, time: 2014-06-12 14:46:19.159148, event: all_read, op: osd_sub_op_reply(client.4308.0:145044 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.159224 7f9d5a16f700  5 -- op tracker -- , seq: 747768, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145044 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.159231 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145044 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:19.159239 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:19.159250 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2519da00 prio 196 cost 0 latency 0.000119 osd_sub_op_reply(client.4308.0:145044 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:19.159297 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.159309 7f9d653db700 10 osd.2 173 dequeue_op 0x2519da00 prio 196 cost 0 latency 0.000179 osd_sub_op_reply(client.4308.0:145044 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0) v2 pg pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]
2014-06-12 14:46:19.159333 7f9d653db700  5 -- op tracker -- , seq: 747768, time: 2014-06-12 14:46:19.159330, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145044 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.159342 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] handle_message: 0x2519da00
2014-06-12 14:46:19.159359 7f9d653db700  5 -- op tracker -- , seq: 747768, time: 2014-06-12 14:46:19.159355, event: started, op: osd_sub_op_reply(client.4308.0:145044 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.159364 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] sub_op_modify_reply: tid 236092 op  ack_type 4 from 24
2014-06-12 14:46:19.159385 7f9d653db700  5 -- op tracker -- , seq: 747184, time: 2014-06-12 14:46:19.159376, event: sub_op_commit_rec, op: osd_op(client.4308.0:145044 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:19.159392 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] repop_all_applied: repop tid 236092 all applied 
2014-06-12 14:46:19.159403 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] eval_repop repgather(0xe832000 173'2704 rep_tid=236092 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145044 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:19.159426 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]  sending ack on repgather(0xe832000 173'2704 rep_tid=236092 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145044 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) 0xee35680
2014-06-12 14:46:19.159460 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] repop_all_committed: repop tid 236092 all committed 
2014-06-12 14:46:19.159471 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean] eval_repop repgather(0xe832000 173'2704 rep_tid=236092 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145044 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:19.159487 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean] log_op_stats osd_op(client.4308.0:145044 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4 inb 65815 outb 0 rlat 28.492840 lat 28.492867
2014-06-12 14:46:19.159509 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean] publish_stats_to_osd 173:2887
2014-06-12 14:46:19.159526 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean]  sending commit on repgather(0xe832000 173'2704 rep_tid=236092 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145044 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) 0x1b775400
2014-06-12 14:46:19.159558 7f9d653db700  5 -- op tracker -- , seq: 747184, time: 2014-06-12 14:46:19.159552, event: commit_sent, op: osd_op(client.4308.0:145044 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:19.159566 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean]  requeue_ops 
2014-06-12 14:46:19.159577 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]  removing repgather(0xe832000 173'2704 rep_tid=236092 committed?=1 applied?=1 op=osd_op(client.4308.0:145044 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:19.159592 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]    q front is repgather(0xe832000 173'2704 rep_tid=236092 committed?=1 applied?=1 op=osd_op(client.4308.0:145044 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:19.159606 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] remove_repop repgather(0xe832000 173'2704 rep_tid=236092 committed?=1 applied?=1 op=osd_op(client.4308.0:145044 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:19.159620 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]  requeue_ops 
2014-06-12 14:46:19.159641 7f9d653db700  5 -- op tracker -- , seq: 747184, time: 2014-06-12 14:46:19.159636, event: done, op: osd_op(client.4308.0:145044 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:19.159667 7f9d653db700 10 osd.2 173 dequeue_op 0x2519da00 finish
2014-06-12 14:46:19.159674 7f9d653db700  5 -- op tracker -- , seq: 747768, time: 2014-06-12 14:46:19.159671, event: done, op: osd_sub_op_reply(client.4308.0:145044 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.159684 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.159687 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.200739 7f9d5a16f700  5 -- op tracker -- , seq: 747769, time: 2014-06-12 14:46:19.200668, event: header_read, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ack, result = 0)
2014-06-12 14:46:19.200756 7f9d5a16f700  5 -- op tracker -- , seq: 747769, time: 2014-06-12 14:46:19.200671, event: throttled, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ack, result = 0)
2014-06-12 14:46:19.200769 7f9d5a16f700  5 -- op tracker -- , seq: 747769, time: 2014-06-12 14:46:19.200689, event: all_read, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ack, result = 0)
2014-06-12 14:46:19.200776 7f9d5a16f700  5 -- op tracker -- , seq: 747769, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ack, result = 0)
2014-06-12 14:46:19.200782 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:19.200788 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:19.200797 7f9d5a16f700 15 osd.2 173 enqueue_op 0x20355a00 prio 196 cost 0 latency 0.000129 osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ack, result = 0) v2
2014-06-12 14:46:19.200831 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.200843 7f9d63bd8700 10 osd.2 173 dequeue_op 0x20355a00 prio 196 cost 0 latency 0.000175 osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ack, result = 0) v2 pg pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]
2014-06-12 14:46:19.200867 7f9d63bd8700  5 -- op tracker -- , seq: 747769, time: 2014-06-12 14:46:19.200864, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ack, result = 0)
2014-06-12 14:46:19.200879 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] handle_message: 0x20355a00
2014-06-12 14:46:19.200897 7f9d63bd8700  5 -- op tracker -- , seq: 747769, time: 2014-06-12 14:46:19.200894, event: started, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ack, result = 0)
2014-06-12 14:46:19.200904 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] sub_op_modify_reply: tid 236093 op  ack_type 1 from 24
2014-06-12 14:46:19.200925 7f9d63bd8700  5 -- op tracker -- , seq: 747185, time: 2014-06-12 14:46:19.200918, event: sub_op_applied_rec, op: osd_op(client.4308.0:145045 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 65536~1032192] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:19.200935 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] repop_all_applied: repop tid 236093 all applied 
2014-06-12 14:46:19.200948 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] eval_repop repgather(0x1f066a80 173'2705 rep_tid=236093 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145045 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 65536~1032192] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:19.200970 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]  sending ack on repgather(0x1f066a80 173'2705 rep_tid=236093 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145045 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 65536~1032192] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) 0x3228cc80
2014-06-12 14:46:19.201004 7f9d63bd8700 10 osd.2 173 dequeue_op 0x20355a00 finish
2014-06-12 14:46:19.201013 7f9d63bd8700  5 -- op tracker -- , seq: 747769, time: 2014-06-12 14:46:19.201009, event: done, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ack, result = 0)
2014-06-12 14:46:19.201026 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.201032 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.203779 7f9d5a16f700  5 -- op tracker -- , seq: 747770, time: 2014-06-12 14:46:19.203708, event: header_read, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.203796 7f9d5a16f700  5 -- op tracker -- , seq: 747770, time: 2014-06-12 14:46:19.203712, event: throttled, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.203803 7f9d5a16f700  5 -- op tracker -- , seq: 747770, time: 2014-06-12 14:46:19.203727, event: all_read, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.203810 7f9d5a16f700  5 -- op tracker -- , seq: 747770, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.203820 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:19.203829 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:19.203843 7f9d5a16f700 15 osd.2 173 enqueue_op 0x20355b00 prio 196 cost 0 latency 0.000135 osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:19.203909 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.203924 7f9d68be2700 10 osd.2 173 dequeue_op 0x20355b00 prio 196 cost 0 latency 0.000216 osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0) v2 pg pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]
2014-06-12 14:46:19.203964 7f9d68be2700  5 -- op tracker -- , seq: 747770, time: 2014-06-12 14:46:19.203958, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.203979 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] handle_message: 0x20355b00
2014-06-12 14:46:19.204002 7f9d68be2700  5 -- op tracker -- , seq: 747770, time: 2014-06-12 14:46:19.203998, event: started, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.204010 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] sub_op_modify_reply: tid 236093 op  ack_type 4 from 24
2014-06-12 14:46:19.204042 7f9d68be2700  5 -- op tracker -- , seq: 747185, time: 2014-06-12 14:46:19.204028, event: sub_op_commit_rec, op: osd_op(client.4308.0:145045 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 65536~1032192] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:19.204051 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2704 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] repop_all_committed: repop tid 236093 all committed 
2014-06-12 14:46:19.204063 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean] eval_repop repgather(0x1f066a80 173'2705 rep_tid=236093 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145045 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 65536~1032192] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:19.204095 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean] log_op_stats osd_op(client.4308.0:145045 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 65536~1032192] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 28.508878 lat 28.511968
2014-06-12 14:46:19.204116 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean] publish_stats_to_osd 173:2888
2014-06-12 14:46:19.204135 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean]  sending commit on repgather(0x1f066a80 173'2705 rep_tid=236093 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145045 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 65536~1032192] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) 0xe185180
2014-06-12 14:46:19.204174 7f9d68be2700  5 -- op tracker -- , seq: 747185, time: 2014-06-12 14:46:19.204168, event: commit_sent, op: osd_op(client.4308.0:145045 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 65536~1032192] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:19.204186 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2703 active+clean]  requeue_ops 
2014-06-12 14:46:19.204199 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean]  removing repgather(0x1f066a80 173'2705 rep_tid=236093 committed?=1 applied?=1 op=osd_op(client.4308.0:145045 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 65536~1032192] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:19.204215 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean]    q front is repgather(0x1f066a80 173'2705 rep_tid=236093 committed?=1 applied?=1 op=osd_op(client.4308.0:145045 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 65536~1032192] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:19.204228 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean] remove_repop repgather(0x1f066a80 173'2705 rep_tid=236093 committed?=1 applied?=1 op=osd_op(client.4308.0:145045 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 65536~1032192] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:19.204244 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean]  requeue_ops 
2014-06-12 14:46:19.204266 7f9d68be2700  5 -- op tracker -- , seq: 747185, time: 2014-06-12 14:46:19.204261, event: done, op: osd_op(client.4308.0:145045 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 65536~1032192] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:19.204288 7f9d68be2700 10 osd.2 173 dequeue_op 0x20355b00 finish
2014-06-12 14:46:19.204298 7f9d68be2700  5 -- op tracker -- , seq: 747770, time: 2014-06-12 14:46:19.204294, event: done, op: osd_sub_op_reply(client.4308.0:145045 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:19.204309 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.204312 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.215582 7f9d5501e700  5 -- op tracker -- , seq: 747771, time: 2014-06-12 14:46:19.207269, event: header_read, op: osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.215603 7f9d5501e700  5 -- op tracker -- , seq: 747771, time: 2014-06-12 14:46:19.207271, event: throttled, op: osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.215613 7f9d5501e700  5 -- op tracker -- , seq: 747771, time: 2014-06-12 14:46:19.215522, event: all_read, op: osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.215621 7f9d5501e700  5 -- op tracker -- , seq: 747771, time: 0.000000, event: dispatched, op: osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.215628 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:19.215635 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:19.215646 7f9d5501e700 15 osd.2 173 enqueue_op 0x2fa9c700 prio 127 cost 2581287 latency 0.008376 osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.216410 7f9d5501e700  5 -- op tracker -- , seq: 747772, time: 2014-06-12 14:46:19.215687, event: header_read, op: osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.216423 7f9d5501e700  5 -- op tracker -- , seq: 747772, time: 2014-06-12 14:46:19.215696, event: throttled, op: osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.216432 7f9d5501e700  5 -- op tracker -- , seq: 747772, time: 2014-06-12 14:46:19.216369, event: all_read, op: osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.216441 7f9d5501e700  5 -- op tracker -- , seq: 747772, time: 0.000000, event: dispatched, op: osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.216449 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:19.216458 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:19.216474 7f9d5501e700 15 osd.2 173 enqueue_op 0x2fa9c300 prio 127 cost 516903 latency 0.000786 osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.216633 7f9d5501e700  5 -- op tracker -- , seq: 747773, time: 2014-06-12 14:46:19.216513, event: header_read, op: osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.216644 7f9d5501e700  5 -- op tracker -- , seq: 747773, time: 2014-06-12 14:46:19.216514, event: throttled, op: osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.216652 7f9d5501e700  5 -- op tracker -- , seq: 747773, time: 2014-06-12 14:46:19.216613, event: all_read, op: osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.216661 7f9d5501e700  5 -- op tracker -- , seq: 747773, time: 0.000000, event: dispatched, op: osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.216665 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:19.216671 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:19.216677 7f9d5501e700 15 osd.2 173 enqueue_op 0x17c2af00 prio 127 cost 37671 latency 0.000164 osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.216702 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.216712 7f9d643d9700 10 osd.2 173 dequeue_op 0x2fa9c700 prio 127 cost 2581287 latency 0.009442 osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.63d( v 173'2928 (0'0,173'2928] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active]
2014-06-12 14:46:19.216736 7f9d643d9700  5 -- op tracker -- , seq: 747771, time: 2014-06-12 14:46:19.216731, event: reached_pg, op: osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.216745 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2928 (0'0,173'2928] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] handle_message: 0x2fa9c700
2014-06-12 14:46:19.216757 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2928 (0'0,173'2928] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] sub_op_modify trans adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 v 173'2929 (transaction) 174
2014-06-12 14:46:19.216772 7f9d643d9700  5 -- op tracker -- , seq: 747771, time: 2014-06-12 14:46:19.216768, event: started, op: osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.216791 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2928 (0'0,173'2928] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] append_log log((0'0,173'2928], crt=0'0) [173'2929 (173'2928) modify   adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 by client.4304.0:144141 2014-06-12 08:45:21.573695]
2014-06-12 14:46:19.216814 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2929 (0'0,173'2929] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] add_log_entry 173'2929 (173'2928) modify   adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 by client.4304.0:144141 2014-06-12 08:45:21.573695
2014-06-12 14:46:19.216835 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2929 (0'0,173'2929] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] append_log  adding 1 keys
2014-06-12 14:46:19.216865 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2929, trimmed: 
2014-06-12 14:46:19.216886 7f9d643d9700  5 -- op tracker -- , seq: 747771, time: 2014-06-12 14:46:19.216881, event: started, op: osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.216899 7f9d643d9700  5 queue_transactions existing osr(3.63d 0x276dc50)/0x276dc50
2014-06-12 14:46:19.216902 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:19.216904 7f9d643d9700  5 queue_op 0x7782780 seq 2663 osr(3.63d 0x276dc50) 2582773 bytes   (queue has 1 ops and 2582773 bytes)
2014-06-12 14:46:19.216910 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.63d( v 173'2929 (0'0,173'2929] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] do_sub_op osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.216924 7f9d643d9700 10 osd.2 173 dequeue_op 0x2fa9c700 finish
2014-06-12 14:46:19.216927 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.216932 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.216935 7f9d643d9700 10 osd.2 173 dequeue_op 0x2fa9c300 prio 127 cost 516903 latency 0.001248 osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.63d( v 173'2929 (0'0,173'2929] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active]
2014-06-12 14:46:19.216952 7f9d643d9700  5 -- op tracker -- , seq: 747772, time: 2014-06-12 14:46:19.216948, event: reached_pg, op: osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.216959 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2929 (0'0,173'2929] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] handle_message: 0x2fa9c300
2014-06-12 14:46:19.216970 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:19.216968 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2929 (0'0,173'2929] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] sub_op_modify trans adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 v 173'2930 (transaction) 174
2014-06-12 14:46:19.216983 7f9d643d9700  5 -- op tracker -- , seq: 747772, time: 2014-06-12 14:46:19.216979, event: started, op: osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.217011 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.216993 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2929 (0'0,173'2929] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] append_log log((0'0,173'2929], crt=0'0) [173'2930 (173'2929) modify   adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 by client.4304.0:144142 2014-06-12 08:45:21.581427]
2014-06-12 14:46:19.217024 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2930 (0'0,173'2930] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] add_log_entry 173'2930 (173'2929) modify   adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 by client.4304.0:144142 2014-06-12 08:45:21.581427
2014-06-12 14:46:19.217043 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2930 (0'0,173'2930] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] append_log  adding 1 keys
2014-06-12 14:46:19.217069 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2930, trimmed: 
2014-06-12 14:46:19.217088 7f9d643d9700  5 -- op tracker -- , seq: 747772, time: 2014-06-12 14:46:19.217084, event: started, op: osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.217096 7f9d643d9700  5 queue_transactions existing osr(3.63d 0x276dc50)/0x276dc50
2014-06-12 14:46:19.217099 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:19.217100 7f9d643d9700  5 queue_op 0x7782d20 seq 2664 osr(3.63d 0x276dc50) 518389 bytes   (queue has 2 ops and 3101162 bytes)
2014-06-12 14:46:19.217106 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.63d( v 173'2930 (0'0,173'2930] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] do_sub_op osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.217120 7f9d643d9700 10 osd.2 173 dequeue_op 0x2fa9c300 finish
2014-06-12 14:46:19.217126 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:19.217129 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.217135 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e96d80 (1 active)
2014-06-12 14:46:19.217140 7f9d81970700  5 _do_op 0x7782780 seq 2663 osr(3.63d 0x276dc50)/0x276dc50 start
2014-06-12 14:46:19.217144 7f9d81970700 10 _do_transaction on 0x9b927f0
2014-06-12 14:46:19.217150 7f9d81970700 15 _omap_setkeys meta/1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.217200 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e96d80 (2 active)
2014-06-12 14:46:19.217291 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.217309 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:19.217365 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:19.217383 7f9d81970700 15 _omap_rmkeys meta/1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.217389 7f9d81970700 10 _omap_rmkeys meta/1fe0c7cb/pglog_3.63d/0//-1 r = 0
2014-06-12 14:46:19.217395 7f9d81970700 15 _omap_setkeys meta/1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.217408 7f9d81970700 15 _write 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 1060864~2580480
2014-06-12 14:46:19.217548 7f9d81970700 10 lookup_strip_header done  cid 3.63d_head oid adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3
2014-06-12 14:46:19.217637 7f9d633d7700 10 osd.2 173 dequeue_op 0x17c2af00 prio 127 cost 37671 latency 0.001123 osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.63d( v 173'2930 (0'0,173'2930] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active]
2014-06-12 14:46:19.217665 7f9d633d7700  5 -- op tracker -- , seq: 747773, time: 2014-06-12 14:46:19.217660, event: reached_pg, op: osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.217674 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2930 (0'0,173'2930] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] handle_message: 0x17c2af00
2014-06-12 14:46:19.217686 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2930 (0'0,173'2930] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] sub_op_modify trans adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 v 173'2931 (transaction) 174
2014-06-12 14:46:19.217703 7f9d633d7700  5 -- op tracker -- , seq: 747773, time: 2014-06-12 14:46:19.217698, event: started, op: osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.217718 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2930 (0'0,173'2930] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] append_log log((0'0,173'2930], crt=0'0) [173'2931 (173'2930) modify   adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 by client.4304.0:144143 2014-06-12 08:45:21.592599]
2014-06-12 14:46:19.217740 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] add_log_entry 173'2931 (173'2930) modify   adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 by client.4304.0:144143 2014-06-12 08:45:21.592599
2014-06-12 14:46:19.217767 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] append_log  adding 1 keys
2014-06-12 14:46:19.217797 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:19.217801 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2931, trimmed: 
2014-06-12 14:46:19.217824 7f9d633d7700  5 -- op tracker -- , seq: 747773, time: 2014-06-12 14:46:19.217819, event: started, op: osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.217834 7f9d633d7700  5 queue_transactions existing osr(3.63d 0x276dc50)/0x276dc50
2014-06-12 14:46:19.217837 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:19.217838 7f9d633d7700  5 queue_op 0x6cf6de0 seq 2665 osr(3.63d 0x276dc50) 39157 bytes   (queue has 3 ops and 3140319 bytes)
2014-06-12 14:46:19.217845 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] do_sub_op osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.217861 7f9d633d7700 10 osd.2 173 dequeue_op 0x17c2af00 finish
2014-06-12 14:46:19.217866 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.217869 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.217876 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e96d80 (3 active)
2014-06-12 14:46:19.227693 7f9d81970700 10 _generic_write 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 1060864~2580480 = 0
2014-06-12 14:46:19.227962 7f9d81970700 15 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3
2014-06-12 14:46:19.227977 7f9d81970700 10 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 = 0
2014-06-12 14:46:19.227983 7f9d81970700 15 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3
2014-06-12 14:46:19.227988 7f9d81970700 10 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 = 0
2014-06-12 14:46:19.228658 7f9d81970700 20 set_header setting 17087 cid 3.63d_head oid adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 parent seq 0
2014-06-12 14:46:19.228953 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:19.228972 7f9d81970700 20 set_header setting 2208 cid meta oid 1fe0c7cb/pglog_3.63d/0//-1 parent seq 0
2014-06-12 14:46:19.228982 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:19.238982 7f9d81970700 10 _do_op 0x7782780 seq 2663 r = 0, finisher 0x7809200 0
2014-06-12 14:46:19.239084 7f9d7715b700  5 -- op tracker -- , seq: 747771, time: 2014-06-12 14:46:19.239073, event: commit_sent, op: osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.239097 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2927 active] sub_op_modify_commit on op osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:19.239429 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:19.239512 7f9d81970700 10 _finish_op 0x7782780 seq 2663 osr(3.63d 0x276dc50)/0x276dc50
2014-06-12 14:46:19.239539 7f9d7c966700  5 _do_op 0x7782d20 seq 2664 osr(3.63d 0x276dc50)/0x276dc50 start
2014-06-12 14:46:19.239545 7f9d7c966700 10 _do_transaction on 0x570dcf0
2014-06-12 14:46:19.239552 7f9d7c966700 15 _omap_setkeys meta/1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.239676 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.239694 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:19.239746 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:19.239767 7f9d7c966700 15 _omap_rmkeys meta/1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.239773 7f9d7c966700 10 _omap_rmkeys meta/1fe0c7cb/pglog_3.63d/0//-1 r = 0
2014-06-12 14:46:19.239779 7f9d7c966700 15 _omap_setkeys meta/1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.239794 7f9d7c966700 15 _write 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 3641344~516096
2014-06-12 14:46:19.239941 7f9d7c966700 10 lookup_strip_header done  cid 3.63d_head oid adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3
2014-06-12 14:46:19.239958 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:19.242080 7f9d7c966700 10 _generic_write 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 3641344~516096 = 0
2014-06-12 14:46:19.242138 7f9d7c966700 15 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3
2014-06-12 14:46:19.242156 7f9d7c966700 10 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 = 0
2014-06-12 14:46:19.242163 7f9d7c966700 15 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3
2014-06-12 14:46:19.242172 7f9d7c966700 10 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 = 0
2014-06-12 14:46:19.242381 7f9d7c966700 20 set_header setting 17087 cid 3.63d_head oid adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 parent seq 0
2014-06-12 14:46:19.242456 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:19.242472 7f9d7c966700 20 set_header setting 2208 cid meta oid 1fe0c7cb/pglog_3.63d/0//-1 parent seq 0
2014-06-12 14:46:19.242483 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:19.245052 7f9d7c966700 10 _do_op 0x7782d20 seq 2664 r = 0, finisher 0x65e57c0 0
2014-06-12 14:46:19.245108 7f9d7715b700  5 -- op tracker -- , seq: 747772, time: 2014-06-12 14:46:19.245096, event: commit_sent, op: osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.245127 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2928 active] sub_op_modify_commit on op osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:19.245159 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:19.245251 7f9d7795c700  5 -- op tracker -- , seq: 747771, time: 2014-06-12 14:46:19.245244, event: sub_op_applied, op: osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.245259 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2929 active] sub_op_modify_applied on 0x9b92700 op osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.245274 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2929 active] op_applied on primary on version 173'2929
2014-06-12 14:46:19.245284 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2929 active] op_applied on replica on version 173'2929
2014-06-12 14:46:19.245318 7f9d81970700  5 -- op tracker -- , seq: 747771, time: 2014-06-12 14:46:19.245308, event: done, op: osd_sub_op(client.4304.0:144141 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.245345 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e96d80 (2 active)
2014-06-12 14:46:19.245348 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:19.245355 7f9d7c966700 10 _finish_op 0x7782d20 seq 2664 osr(3.63d 0x276dc50)/0x276dc50
2014-06-12 14:46:19.245363 7f9d7895e700  5 _do_op 0x6cf6de0 seq 2665 osr(3.63d 0x276dc50)/0x276dc50 start
2014-06-12 14:46:19.245368 7f9d7895e700 10 _do_transaction on 0x23fbeb70
2014-06-12 14:46:19.245376 7f9d7895e700 15 _omap_setkeys meta/1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.245495 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.245524 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:19.245578 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:19.245598 7f9d7895e700 15 _omap_rmkeys meta/1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.245605 7f9d7895e700 10 _omap_rmkeys meta/1fe0c7cb/pglog_3.63d/0//-1 r = 0
2014-06-12 14:46:19.245611 7f9d7895e700 15 _omap_setkeys meta/1fe0c7cb/pglog_3.63d/0//-1
2014-06-12 14:46:19.245738 7f9d7895e700 15 _write 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 4157440~36864
2014-06-12 14:46:19.245885 7f9d7895e700 10 lookup_strip_header done  cid 3.63d_head oid adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3
2014-06-12 14:46:19.245906 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:19.246133 7f9d7895e700 10 _generic_write 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 4157440~36864 = 0
2014-06-12 14:46:19.246152 7f9d7895e700 15 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3
2014-06-12 14:46:19.246166 7f9d7895e700 10 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 = 0
2014-06-12 14:46:19.246171 7f9d7895e700 15 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3
2014-06-12 14:46:19.246180 7f9d7895e700 10 _setattrs 3.63d_head/adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 = 0
2014-06-12 14:46:19.246300 7f9d7895e700 20 set_header setting 17087 cid 3.63d_head oid adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 parent seq 0
2014-06-12 14:46:19.246324 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:19.246336 7f9d7895e700 20 set_header setting 2208 cid meta oid 1fe0c7cb/pglog_3.63d/0//-1 parent seq 0
2014-06-12 14:46:19.246346 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:19.246614 7f9d7895e700 10 _do_op 0x6cf6de0 seq 2665 r = 0, finisher 0x7c91660 0
2014-06-12 14:46:19.246644 7f9d7715b700  5 -- op tracker -- , seq: 747773, time: 2014-06-12 14:46:19.246636, event: commit_sent, op: osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.246655 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2929 active] sub_op_modify_commit on op osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:19.246681 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:19.246758 7f9d7795c700  5 -- op tracker -- , seq: 747772, time: 2014-06-12 14:46:19.246751, event: sub_op_applied, op: osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.246766 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2930 active] sub_op_modify_applied on 0x570dc00 op osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.246781 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2930 active] op_applied on primary on version 173'2930
2014-06-12 14:46:19.246792 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2930 active] op_applied on replica on version 173'2930
2014-06-12 14:46:19.246826 7f9d7c966700  5 -- op tracker -- , seq: 747772, time: 2014-06-12 14:46:19.246816, event: done, op: osd_sub_op(client.4304.0:144142 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.246850 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e96d80 (1 active)
2014-06-12 14:46:19.246853 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:19.246864 7f9d7895e700 10 _finish_op 0x6cf6de0 seq 2665 osr(3.63d 0x276dc50)/0x276dc50
2014-06-12 14:46:19.246871 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e96d80 (0 active)
2014-06-12 14:46:19.246873 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:19.246882 7f9d7795c700  5 -- op tracker -- , seq: 747773, time: 2014-06-12 14:46:19.246878, event: sub_op_applied, op: osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.246888 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2930 active] sub_op_modify_applied on 0x23fbea80 op osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.246903 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2930 active] op_applied on primary on version 173'2931
2014-06-12 14:46:19.246917 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63d( v 173'2931 (0'0,173'2931] local-les=161 n=296 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2930 active] op_applied on replica on version 173'2931
2014-06-12 14:46:19.246936 7f9d7795c700  5 -- op tracker -- , seq: 747773, time: 2014-06-12 14:46:19.246932, event: done, op: osd_sub_op(client.4304.0:144143 3.63d adf3e3d/rbd_data.10a76b8b4567.0000000000001018/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.341535 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:19.341561 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:19.400129 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:19.400601 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:19.418119 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:19.418172 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:19.440242 7f9d56432700  5 -- op tracker -- , seq: 747774, time: 2014-06-12 14:46:19.439408, event: header_read, op: osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.440255 7f9d56432700  5 -- op tracker -- , seq: 747774, time: 2014-06-12 14:46:19.439411, event: throttled, op: osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.440270 7f9d56432700  5 -- op tracker -- , seq: 747774, time: 2014-06-12 14:46:19.440204, event: all_read, op: osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.440282 7f9d56432700  5 -- op tracker -- , seq: 747774, time: 0.000000, event: dispatched, op: osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.440286 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:19.440290 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:19.440296 7f9d56432700 15 osd.2 173 enqueue_op 0x2b014b00 prio 127 cost 516903 latency 0.000888 osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.440315 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.440322 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2b014b00 prio 127 cost 516903 latency 0.000914 osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2636 (0'0,173'2636] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2635 active]
2014-06-12 14:46:19.440339 7f9d61bd4700  5 -- op tracker -- , seq: 747774, time: 2014-06-12 14:46:19.440336, event: reached_pg, op: osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.440346 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2636 (0'0,173'2636] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2635 active] handle_message: 0x2b014b00
2014-06-12 14:46:19.440354 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2636 (0'0,173'2636] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2635 active] sub_op_modify trans 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 v 173'2637 (transaction) 174
2014-06-12 14:46:19.440364 7f9d61bd4700  5 -- op tracker -- , seq: 747774, time: 2014-06-12 14:46:19.440361, event: started, op: osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.440373 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2636 (0'0,173'2636] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2635 active] append_log log((0'0,173'2636], crt=0'0) [173'2637 (173'2636) modify   3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 by client.4286.0:144489 2014-06-12 08:43:16.790246]
2014-06-12 14:46:19.440389 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2637 (0'0,173'2637] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2635 active] add_log_entry 173'2637 (173'2636) modify   3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 by client.4286.0:144489 2014-06-12 08:43:16.790246
2014-06-12 14:46:19.440411 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2637 (0'0,173'2637] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2635 active] append_log  adding 1 keys
2014-06-12 14:46:19.440432 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2637, trimmed: 
2014-06-12 14:46:19.440448 7f9d61bd4700  5 -- op tracker -- , seq: 747774, time: 2014-06-12 14:46:19.440445, event: started, op: osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.440457 7f9d61bd4700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:19.440461 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:19.440462 7f9d61bd4700  5 queue_op 0x6b5be60 seq 2457 osr(3.1c7 0x276d170) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:19.440468 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:46:19.440473 7f9d7a161700  5 _do_op 0x6b5be60 seq 2457 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:46:19.440466 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2637 (0'0,173'2637] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2635 active] do_sub_op osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.440475 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2b014b00 finish
2014-06-12 14:46:19.440477 7f9d7a161700 10 _do_transaction on 0x1e78b270
2014-06-12 14:46:19.440477 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.440479 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.440482 7f9d7a161700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:19.440586 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:19.440611 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:19.440651 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:19.440823 7f9d7a161700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:19.440830 7f9d7a161700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:46:19.440834 7f9d7a161700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:19.440846 7f9d7a161700 15 _write 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 3432448~516096
2014-06-12 14:46:19.441113 7f9d7a161700 10 lookup_strip_header done  cid 3.1c7_head oid 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3
2014-06-12 14:46:19.441219 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:19.442573 7f9d7a161700 10 _generic_write 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 3432448~516096 = 0
2014-06-12 14:46:19.442613 7f9d7a161700 15 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3
2014-06-12 14:46:19.442624 7f9d7a161700 10 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:19.442628 7f9d7a161700 15 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3
2014-06-12 14:46:19.442633 7f9d7a161700 10 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:19.442763 7f9d7a161700 20 set_header setting 16919 cid 3.1c7_head oid 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 parent seq 0
2014-06-12 14:46:19.442805 7f9d7a161700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:46:19.442816 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:19.442821 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:19.444703 7f9d7a161700 10 _do_op 0x6b5be60 seq 2457 r = 0, finisher 0x52dfee0 0
2014-06-12 14:46:19.444715 7f9d7a161700 10 _finish_op 0x6b5be60 seq 2457 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:19.444722 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:46:19.444730 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:19.444734 7f9d7715b700  5 -- op tracker -- , seq: 747774, time: 2014-06-12 14:46:19.444724, event: commit_sent, op: osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.444753 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2637 (0'0,173'2637] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2635 active] sub_op_modify_commit on op osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:19.444783 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:19.444821 7f9d7795c700  5 -- op tracker -- , seq: 747774, time: 2014-06-12 14:46:19.444805, event: sub_op_applied, op: osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.444829 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2637 (0'0,173'2637] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2636 active] sub_op_modify_applied on 0x1e78b180 op osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.444841 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2637 (0'0,173'2637] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2636 active] op_applied on primary on version 173'2637
2014-06-12 14:46:19.444848 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2637 (0'0,173'2637] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2636 active] op_applied on replica on version 173'2637
2014-06-12 14:46:19.444863 7f9d7795c700  5 -- op tracker -- , seq: 747774, time: 2014-06-12 14:46:19.444860, event: done, op: osd_sub_op(client.4286.0:144489 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.472442 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:19.472479 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:19.472482 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:19.472484 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:19.472487 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 8; oldest is 25.837043 seconds old
2014-06-12 14:46:19.472502 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:19.490206 7f9d56432700  5 -- op tracker -- , seq: 747775, time: 2014-06-12 14:46:19.488232, event: header_read, op: osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.490238 7f9d56432700  5 -- op tracker -- , seq: 747775, time: 2014-06-12 14:46:19.488236, event: throttled, op: osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.490247 7f9d56432700  5 -- op tracker -- , seq: 747775, time: 2014-06-12 14:46:19.490151, event: all_read, op: osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.490255 7f9d56432700  5 -- op tracker -- , seq: 747775, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.490262 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:19.490272 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:19.490285 7f9d56432700 15 osd.2 173 enqueue_op 0x2b014f00 prio 127 cost 1032999 latency 0.002053 osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.490319 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.490330 7f9d673df700 10 osd.2 173 dequeue_op 0x2b014f00 prio 127 cost 1032999 latency 0.002098 osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.67d( v 173'2508 (0'0,173'2508] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2507 active]
2014-06-12 14:46:19.490357 7f9d673df700  5 -- op tracker -- , seq: 747775, time: 2014-06-12 14:46:19.490353, event: reached_pg, op: osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.490366 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2508 (0'0,173'2508] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2507 active] handle_message: 0x2b014f00
2014-06-12 14:46:19.490382 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2508 (0'0,173'2508] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2507 active] sub_op_modify trans 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 v 173'2509 (transaction) 174
2014-06-12 14:46:19.490401 7f9d673df700  5 -- op tracker -- , seq: 747775, time: 2014-06-12 14:46:19.490397, event: started, op: osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.490418 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2508 (0'0,173'2508] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2507 active] append_log log((0'0,173'2508], crt=0'0) [173'2509 (173'2508) modify   6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 by client.4303.0:144815 2014-06-12 08:45:21.089750]
2014-06-12 14:46:19.490444 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2509 (0'0,173'2509] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2507 active] add_log_entry 173'2509 (173'2508) modify   6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 by client.4303.0:144815 2014-06-12 08:45:21.089750
2014-06-12 14:46:19.490467 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2509 (0'0,173'2509] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2507 active] append_log  adding 1 keys
2014-06-12 14:46:19.490501 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2509, trimmed: 
2014-06-12 14:46:19.490529 7f9d673df700  5 -- op tracker -- , seq: 747775, time: 2014-06-12 14:46:19.490524, event: started, op: osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.490542 7f9d673df700  5 queue_transactions existing osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:19.490547 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:19.490548 7f9d673df700  5 queue_op 0xa2db380 seq 2287 osr(3.67d 0x276df70) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:19.490561 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x524cb40 (1 active)
2014-06-12 14:46:19.490556 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.67d( v 173'2509 (0'0,173'2509] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2507 active] do_sub_op osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.490568 7f9d7a962700  5 _do_op 0xa2db380 seq 2287 osr(3.67d 0x276df70)/0x276df70 start
2014-06-12 14:46:19.490571 7f9d673df700 10 osd.2 173 dequeue_op 0x2b014f00 finish
2014-06-12 14:46:19.490573 7f9d7a962700 10 _do_transaction on 0x61cf430
2014-06-12 14:46:19.490574 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.490576 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.490580 7f9d7a962700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:19.490738 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:19.490886 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:19.490948 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:19.490976 7f9d7a962700 15 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:19.490984 7f9d7a962700 10 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1 r = 0
2014-06-12 14:46:19.490990 7f9d7a962700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:19.491005 7f9d7a962700 15 _write 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 20480~1032192
2014-06-12 14:46:19.491349 7f9d7a962700 10 lookup_strip_header done  cid 3.67d_head oid 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:19.491513 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:19.496651 7f9d7a962700 10 _generic_write 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 20480~1032192 = 0
2014-06-12 14:46:19.496806 7f9d7a962700 15 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:19.496831 7f9d7a962700 10 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 = 0
2014-06-12 14:46:19.496839 7f9d7a962700 15 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:19.496847 7f9d7a962700 10 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 = 0
2014-06-12 14:46:19.497242 7f9d7a962700 20 set_header setting 17507 cid 3.67d_head oid 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 parent seq 0
2014-06-12 14:46:19.497372 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:19.497390 7f9d7a962700 20 set_header setting 2272 cid meta oid 1fe0e10b/pglog_3.67d/0//-1 parent seq 0
2014-06-12 14:46:19.497400 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:19.506500 7f9d7a962700 10 _do_op 0xa2db380 seq 2287 r = 0, finisher 0x52dfec0 0
2014-06-12 14:46:19.506525 7f9d7a962700 10 _finish_op 0xa2db380 seq 2287 osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:19.506534 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x524cb40 (0 active)
2014-06-12 14:46:19.506542 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:19.506548 7f9d7715b700  5 -- op tracker -- , seq: 747775, time: 2014-06-12 14:46:19.506536, event: commit_sent, op: osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.506566 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2509 (0'0,173'2509] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2507 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:19.506596 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:19.507616 7f9d7795c700  5 -- op tracker -- , seq: 747775, time: 2014-06-12 14:46:19.507601, event: sub_op_applied, op: osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.507631 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2509 (0'0,173'2509] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2508 active] sub_op_modify_applied on 0x61cf340 op osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.507659 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2509 (0'0,173'2509] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2508 active] op_applied on primary on version 173'2509
2014-06-12 14:46:19.507683 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2509 (0'0,173'2509] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2508 active] op_applied on replica on version 173'2509
2014-06-12 14:46:19.507711 7f9d7795c700  5 -- op tracker -- , seq: 747775, time: 2014-06-12 14:46:19.507706, event: done, op: osd_sub_op(client.4303.0:144815 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.547139 7f9d56432700  5 -- op tracker -- , seq: 747776, time: 2014-06-12 14:46:19.545078, event: header_read, op: osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.547154 7f9d56432700  5 -- op tracker -- , seq: 747776, time: 2014-06-12 14:46:19.545080, event: throttled, op: osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.547163 7f9d56432700  5 -- op tracker -- , seq: 747776, time: 2014-06-12 14:46:19.547083, event: all_read, op: osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.547172 7f9d56432700  5 -- op tracker -- , seq: 747776, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.547178 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:19.547185 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:19.547194 7f9d56432700 15 osd.2 173 enqueue_op 0x160e4300 prio 127 cost 1032999 latency 0.002115 osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.547225 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.547235 7f9d67be0700 10 osd.2 173 dequeue_op 0x160e4300 prio 127 cost 1032999 latency 0.002156 osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2995 (0'0,173'2995] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2994 active]
2014-06-12 14:46:19.547259 7f9d67be0700  5 -- op tracker -- , seq: 747776, time: 2014-06-12 14:46:19.547254, event: reached_pg, op: osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.547268 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2995 (0'0,173'2995] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2994 active] handle_message: 0x160e4300
2014-06-12 14:46:19.547282 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2995 (0'0,173'2995] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2994 active] sub_op_modify trans 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 v 173'2996 (transaction) 174
2014-06-12 14:46:19.547304 7f9d67be0700  5 -- op tracker -- , seq: 747776, time: 2014-06-12 14:46:19.547299, event: started, op: osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.547322 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2995 (0'0,173'2995] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2994 active] append_log log((0'0,173'2995], crt=0'0) [173'2996 (173'2995) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145017 2014-06-12 08:44:42.126461]
2014-06-12 14:46:19.547342 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2996 (0'0,173'2996] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2994 active] add_log_entry 173'2996 (173'2995) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145017 2014-06-12 08:44:42.126461
2014-06-12 14:46:19.547366 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2996 (0'0,173'2996] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2994 active] append_log  adding 1 keys
2014-06-12 14:46:19.547400 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2996, trimmed: 
2014-06-12 14:46:19.547424 7f9d67be0700  5 -- op tracker -- , seq: 747776, time: 2014-06-12 14:46:19.547420, event: started, op: osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.547434 7f9d67be0700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:19.547438 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:19.547440 7f9d67be0700  5 queue_op 0x70c4ae0 seq 2798 osr(3.458 0x276d960) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:19.547447 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2996 (0'0,173'2996] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2994 active] do_sub_op osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.547463 7f9d67be0700 10 osd.2 173 dequeue_op 0x160e4300 finish
2014-06-12 14:46:19.547467 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.547469 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.547478 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:19.547483 7f9d7f16b700  5 _do_op 0x70c4ae0 seq 2798 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:19.547488 7f9d7f16b700 10 _do_transaction on 0x5afdb30
2014-06-12 14:46:19.547494 7f9d7f16b700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:19.547609 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:19.547640 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:19.547690 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:19.547805 7f9d7f16b700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:19.547813 7f9d7f16b700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:19.547822 7f9d7f16b700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:19.547848 7f9d7f16b700 15 _write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 2007040~1032192
2014-06-12 14:46:19.548094 7f9d7f16b700 10 lookup_strip_header done  cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:19.548213 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:19.551736 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:19.552590 7f9d7f16b700 10 _generic_write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 2007040~1032192 = 0
2014-06-12 14:46:19.552708 7f9d7f16b700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:19.552728 7f9d7f16b700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:19.552735 7f9d7f16b700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:19.552744 7f9d7f16b700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:19.553113 7f9d7f16b700 20 set_header setting 16936 cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 parent seq 0
2014-06-12 14:46:19.553230 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:19.553246 7f9d7f16b700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:19.553260 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:19.553729 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:19.566202 7f9d7f16b700 10 _do_op 0x70c4ae0 seq 2798 r = 0, finisher 0x66ad2e0 0
2014-06-12 14:46:19.566227 7f9d7f16b700 10 _finish_op 0x70c4ae0 seq 2798 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:19.566234 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:19.566242 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:19.566245 7f9d7715b700  5 -- op tracker -- , seq: 747776, time: 2014-06-12 14:46:19.566234, event: commit_sent, op: osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.566257 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2996 (0'0,173'2996] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2994 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:19.566290 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:19.569651 7f9d7795c700  5 -- op tracker -- , seq: 747776, time: 2014-06-12 14:46:19.569637, event: sub_op_applied, op: osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.569662 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2996 (0'0,173'2996] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active] sub_op_modify_applied on 0x5afda40 op osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.569682 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2996 (0'0,173'2996] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active] op_applied on primary on version 173'2996
2014-06-12 14:46:19.569692 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2996 (0'0,173'2996] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active] op_applied on replica on version 173'2996
2014-06-12 14:46:19.569721 7f9d7795c700  5 -- op tracker -- , seq: 747776, time: 2014-06-12 14:46:19.569716, event: done, op: osd_sub_op(client.4308.0:145017 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2996 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.647881 7f9d5b6bc700  5 -- op tracker -- , seq: 747777, time: 2014-06-12 14:46:19.638519, event: header_read, op: osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.647898 7f9d5b6bc700  5 -- op tracker -- , seq: 747777, time: 2014-06-12 14:46:19.638524, event: throttled, op: osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.647906 7f9d5b6bc700  5 -- op tracker -- , seq: 747777, time: 2014-06-12 14:46:19.647799, event: all_read, op: osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.647915 7f9d5b6bc700  5 -- op tracker -- , seq: 747777, time: 0.000000, event: dispatched, op: osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.647922 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:19.647929 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:19.647938 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x329ea600 prio 127 cost 3842855 latency 0.009419 osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.647972 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.647980 7f9d663dd700 10 osd.2 173 dequeue_op 0x329ea600 prio 127 cost 3842855 latency 0.009461 osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.61d( v 173'2804 (0'0,173'2804] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2803 active]
2014-06-12 14:46:19.648006 7f9d663dd700  5 -- op tracker -- , seq: 747777, time: 2014-06-12 14:46:19.648001, event: reached_pg, op: osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.648015 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2804 (0'0,173'2804] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2803 active] handle_message: 0x329ea600
2014-06-12 14:46:19.648030 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2804 (0'0,173'2804] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2803 active] sub_op_modify trans 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 v 173'2805 (transaction) 174
2014-06-12 14:46:19.648047 7f9d663dd700  5 -- op tracker -- , seq: 747777, time: 2014-06-12 14:46:19.648043, event: started, op: osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.648064 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2804 (0'0,173'2804] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2803 active] append_log log((0'0,173'2804], crt=0'0) [173'2805 (173'647) modify   7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 by client.4296.0:143633 2014-06-12 08:45:21.960591]
2014-06-12 14:46:19.648086 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2805 (0'0,173'2805] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2803 active] add_log_entry 173'2805 (173'647) modify   7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 by client.4296.0:143633 2014-06-12 08:45:21.960591
2014-06-12 14:46:19.648107 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2805 (0'0,173'2805] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2803 active] append_log  adding 1 keys
2014-06-12 14:46:19.648148 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2805, trimmed: 
2014-06-12 14:46:19.648170 7f9d663dd700  5 -- op tracker -- , seq: 747777, time: 2014-06-12 14:46:19.648165, event: started, op: osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.648179 7f9d663dd700  5 queue_transactions existing osr(3.61d 0x276dcb0)/0x276dcb0
2014-06-12 14:46:19.648183 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:19.648184 7f9d663dd700  5 queue_op 0x6b41500 seq 2597 osr(3.61d 0x276dcb0) 3844341 bytes   (queue has 1 ops and 3844341 bytes)
2014-06-12 14:46:19.648192 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.61d( v 173'2805 (0'0,173'2805] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2803 active] do_sub_op osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.648205 7f9d663dd700 10 osd.2 173 dequeue_op 0x329ea600 finish
2014-06-12 14:46:19.648208 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.648210 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.648218 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541c5a0 (1 active)
2014-06-12 14:46:19.648223 7f9d7e169700  5 _do_op 0x6b41500 seq 2597 osr(3.61d 0x276dcb0)/0x276dcb0 start
2014-06-12 14:46:19.648227 7f9d7e169700 10 _do_transaction on 0x610f270
2014-06-12 14:46:19.648233 7f9d7e169700 15 _omap_setkeys meta/1fe134ab/pglog_3.61d/0//-1
2014-06-12 14:46:19.648366 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe134ab/pglog_3.61d/0//-1
2014-06-12 14:46:19.648389 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:19.648450 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:19.648472 7f9d7e169700 15 _omap_rmkeys meta/1fe134ab/pglog_3.61d/0//-1
2014-06-12 14:46:19.648478 7f9d7e169700 10 _omap_rmkeys meta/1fe134ab/pglog_3.61d/0//-1 r = 0
2014-06-12 14:46:19.648484 7f9d7e169700 15 _omap_setkeys meta/1fe134ab/pglog_3.61d/0//-1
2014-06-12 14:46:19.648645 7f9d7e169700 15 _write 3.61d_head/7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 0~3842048
2014-06-12 14:46:19.648815 7f9d7e169700 10 lookup_strip_header done  cid 3.61d_head oid 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3
2014-06-12 14:46:19.649101 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:19.663119 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:19.663248 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:19.665253 7f9d7e169700 10 _generic_write 3.61d_head/7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 0~3842048 = 0
2014-06-12 14:46:19.665896 7f9d7e169700 15 _setattrs 3.61d_head/7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3
2014-06-12 14:46:19.665957 7f9d7e169700 10 _setattrs 3.61d_head/7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 = 0
2014-06-12 14:46:19.665964 7f9d7e169700 15 _setattrs 3.61d_head/7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3
2014-06-12 14:46:19.665975 7f9d7e169700 10 _setattrs 3.61d_head/7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 = 0
2014-06-12 14:46:19.667041 7f9d7e169700 20 set_header setting 16992 cid 3.61d_head oid 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 parent seq 0
2014-06-12 14:46:19.667654 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:19.667671 7f9d7e169700 20 set_header setting 2224 cid meta oid 1fe134ab/pglog_3.61d/0//-1 parent seq 0
2014-06-12 14:46:19.667681 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:19.688507 7f9d7e169700 10 _do_op 0x6b41500 seq 2597 r = 0, finisher 0x960d5c0 0
2014-06-12 14:46:19.688533 7f9d7e169700 10 _finish_op 0x6b41500 seq 2597 osr(3.61d 0x276dcb0)/0x276dcb0
2014-06-12 14:46:19.688550 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541c5a0 (0 active)
2014-06-12 14:46:19.688559 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:19.688554 7f9d7715b700  5 -- op tracker -- , seq: 747777, time: 2014-06-12 14:46:19.688543, event: commit_sent, op: osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.688569 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2805 (0'0,173'2805] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2803 active] sub_op_modify_commit on op osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:19.688600 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:19.688630 7f9d7795c700  5 -- op tracker -- , seq: 747777, time: 2014-06-12 14:46:19.688624, event: sub_op_applied, op: osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.688637 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2805 (0'0,173'2805] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2804 active] sub_op_modify_applied on 0x610f180 op osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.688648 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2805 (0'0,173'2805] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2804 active] op_applied on primary on version 173'2805
2014-06-12 14:46:19.688654 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2805 (0'0,173'2805] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2804 active] op_applied on replica on version 173'2805
2014-06-12 14:46:19.688671 7f9d7795c700  5 -- op tracker -- , seq: 747777, time: 2014-06-12 14:46:19.688668, event: done, op: osd_sub_op(client.4296.0:143633 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2805 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.696106 7f9d5b6bc700  5 -- op tracker -- , seq: 747778, time: 2014-06-12 14:46:19.695602, event: header_read, op: osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.696116 7f9d5b6bc700  5 -- op tracker -- , seq: 747778, time: 2014-06-12 14:46:19.695604, event: throttled, op: osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.696121 7f9d5b6bc700  5 -- op tracker -- , seq: 747778, time: 2014-06-12 14:46:19.696071, event: all_read, op: osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.696126 7f9d5b6bc700  5 -- op tracker -- , seq: 747778, time: 0.000000, event: dispatched, op: osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.696129 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:19.696133 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:19.696139 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x17c2aa00 prio 127 cost 353063 latency 0.000537 osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.696161 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:19.696170 7f9d693e3700 10 osd.2 173 dequeue_op 0x17c2aa00 prio 127 cost 353063 latency 0.000568 osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.61d( v 173'2805 (0'0,173'2805] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2804 active]
2014-06-12 14:46:19.696197 7f9d693e3700  5 -- op tracker -- , seq: 747778, time: 2014-06-12 14:46:19.696192, event: reached_pg, op: osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.696212 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2805 (0'0,173'2805] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2804 active] handle_message: 0x17c2aa00
2014-06-12 14:46:19.696231 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2805 (0'0,173'2805] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2804 active] sub_op_modify trans 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 v 173'2806 (transaction) 174
2014-06-12 14:46:19.696247 7f9d693e3700  5 -- op tracker -- , seq: 747778, time: 2014-06-12 14:46:19.696243, event: started, op: osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.696263 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2805 (0'0,173'2805] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2804 active] append_log log((0'0,173'2805], crt=0'0) [173'2806 (173'2805) modify   7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 by client.4296.0:143634 2014-06-12 08:45:22.073885]
2014-06-12 14:46:19.696290 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2806 (0'0,173'2806] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2804 active] add_log_entry 173'2806 (173'2805) modify   7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 by client.4296.0:143634 2014-06-12 08:45:22.073885
2014-06-12 14:46:19.696323 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2806 (0'0,173'2806] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2804 active] append_log  adding 1 keys
2014-06-12 14:46:19.696365 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2806, trimmed: 
2014-06-12 14:46:19.696390 7f9d693e3700  5 -- op tracker -- , seq: 747778, time: 2014-06-12 14:46:19.696385, event: started, op: osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.696408 7f9d693e3700  5 queue_transactions existing osr(3.61d 0x276dcb0)/0x276dcb0
2014-06-12 14:46:19.696412 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:19.696413 7f9d693e3700  5 queue_op 0x142a8660 seq 2598 osr(3.61d 0x276dcb0) 354549 bytes   (queue has 1 ops and 354549 bytes)
2014-06-12 14:46:19.696428 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541c5a0 (1 active)
2014-06-12 14:46:19.696434 7f9d7815d700  5 _do_op 0x142a8660 seq 2598 osr(3.61d 0x276dcb0)/0x276dcb0 start
2014-06-12 14:46:19.696425 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.61d( v 173'2806 (0'0,173'2806] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2804 active] do_sub_op osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.696438 7f9d7815d700 10 _do_transaction on 0x65f95f0
2014-06-12 14:46:19.696439 7f9d693e3700 10 osd.2 173 dequeue_op 0x17c2aa00 finish
2014-06-12 14:46:19.696442 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:19.696443 7f9d7815d700 15 _omap_setkeys meta/1fe134ab/pglog_3.61d/0//-1
2014-06-12 14:46:19.696444 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:19.696557 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe134ab/pglog_3.61d/0//-1
2014-06-12 14:46:19.696584 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:19.696631 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:19.696655 7f9d7815d700 15 _omap_rmkeys meta/1fe134ab/pglog_3.61d/0//-1
2014-06-12 14:46:19.696659 7f9d7815d700 10 _omap_rmkeys meta/1fe134ab/pglog_3.61d/0//-1 r = 0
2014-06-12 14:46:19.696663 7f9d7815d700 15 _omap_setkeys meta/1fe134ab/pglog_3.61d/0//-1
2014-06-12 14:46:19.696675 7f9d7815d700 15 _write 3.61d_head/7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 3842048~352256
2014-06-12 14:46:19.696775 7f9d7815d700 10 lookup_strip_header done  cid 3.61d_head oid 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3
2014-06-12 14:46:19.697038 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:19.697881 7f9d7815d700 10 _generic_write 3.61d_head/7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 3842048~352256 = 0
2014-06-12 14:46:19.697908 7f9d7815d700 15 _setattrs 3.61d_head/7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3
2014-06-12 14:46:19.697917 7f9d7815d700 10 _setattrs 3.61d_head/7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 = 0
2014-06-12 14:46:19.697920 7f9d7815d700 15 _setattrs 3.61d_head/7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3
2014-06-12 14:46:19.697925 7f9d7815d700 10 _setattrs 3.61d_head/7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 = 0
2014-06-12 14:46:19.698033 7f9d7815d700 20 set_header setting 16992 cid 3.61d_head oid 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 parent seq 0
2014-06-12 14:46:19.698062 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:19.698071 7f9d7815d700 20 set_header setting 2224 cid meta oid 1fe134ab/pglog_3.61d/0//-1 parent seq 0
2014-06-12 14:46:19.698078 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:19.699189 7f9d7815d700 10 _do_op 0x142a8660 seq 2598 r = 0, finisher 0x28f79c0 0
2014-06-12 14:46:19.699200 7f9d7815d700 10 _finish_op 0x142a8660 seq 2598 osr(3.61d 0x276dcb0)/0x276dcb0
2014-06-12 14:46:19.699209 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541c5a0 (0 active)
2014-06-12 14:46:19.699215 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:19.699217 7f9d7715b700  5 -- op tracker -- , seq: 747778, time: 2014-06-12 14:46:19.699210, event: commit_sent, op: osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.699240 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2806 (0'0,173'2806] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2804 active] sub_op_modify_commit on op osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:19.699260 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:19.699287 7f9d7795c700  5 -- op tracker -- , seq: 747778, time: 2014-06-12 14:46:19.699281, event: sub_op_applied, op: osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.699294 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2806 (0'0,173'2806] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2805 active] sub_op_modify_applied on 0x65f9500 op osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:19.699306 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2806 (0'0,173'2806] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2805 active] op_applied on primary on version 173'2806
2014-06-12 14:46:19.699312 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.61d( v 173'2806 (0'0,173'2806] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2805 active] op_applied on replica on version 173'2806
2014-06-12 14:46:19.699326 7f9d7795c700  5 -- op tracker -- , seq: 747778, time: 2014-06-12 14:46:19.699323, event: done, op: osd_sub_op(client.4296.0:143634 3.61d 7a952e1d/rbd_data.109f6b8b4567.00000000000011b6/head//3 [] v 173'2806 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:19.795597 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:19.795683 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:19.947463 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:19.947557 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:20.136593 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:20.136647 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:20.153672 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:20.160671 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:20.162008 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:20.222948 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:20.235746 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:20.261440 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:20.306324 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:20.315985 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19421 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8])
2014-06-12 14:46:20.316010 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19421 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8])
2014-06-12 14:46:20.316506 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:20.316537 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:20.316637 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:20.316666 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:20.316685 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:20.316702 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:20.316721 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:20.316739 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:20.316757 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:20.316778 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:20.316814 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:20.316832 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:20.316849 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:20.316866 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:20.316883 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:20.316900 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:20.316923 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:20.316940 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:20.316957 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:20.317025 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:20.317042 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:20.317071 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:20.317078 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:20.367326 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:20.390058 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:20.403498 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:20.472556 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:20.472617 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 13 >= max 0.5 = no, load too high
2014-06-12 14:46:20.472638 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:20.472640 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:20.472641 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92628 < max (604800 seconds)
2014-06-12 14:46:20.472655 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:20.472658 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:20.472660 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:20.472662 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 8; oldest is 26.837219 seconds old
2014-06-12 14:46:20.500734 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:20.500815 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:20.518735 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:20.518833 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:20.603767 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:20.763684 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:20.764024 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:20.787855 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:20.822441 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:20.959715 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.007570 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.031746 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:21.047937 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:21.047965 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:21.058388 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.074709 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:21.122861 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.129100 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:21.133103 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.152960 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:21.159751 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.201098 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.204372 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.217072 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.217186 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.217926 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.245410 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:21.246911 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:21.246921 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:21.321758 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:21.321770 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:21.321771 7f9d74bfa700 20 osd.2 173 _dispatch 0x1e701d40 ping v1
2014-06-12 14:46:21.321774 7f9d74bfa700 10 osd.2 173 ping from client.4308
2014-06-12 14:46:21.321776 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:21.321778 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:21.440544 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.444791 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:21.472759 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:21.472787 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:21.472790 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:21.472790 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:21.472792 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 8; oldest is 27.837349 seconds old
2014-06-12 14:46:21.472793 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:21.490669 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.506606 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:21.547538 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.566329 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:21.648305 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.688620 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:21.696512 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:21.699276 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:21.817541 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:21.817580 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:22.016083 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:22.016233 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:22.042272 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:22.042275 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:22.042276 7f9d74bfa700 20 osd.2 173 _dispatch 0x646da00 ping v1
2014-06-12 14:46:22.042278 7f9d74bfa700 10 osd.2 173 ping from client.4317
2014-06-12 14:46:22.042280 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:22.042281 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:22.094959 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:22.095017 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:22.153748 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:22.160745 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:22.162074 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:22.223023 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:22.235820 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:22.261523 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:22.306428 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:22.317135 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:22.367402 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:22.390140 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:22.403602 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:22.472859 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:22.472881 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:46:22.472884 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:46:22.472894 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:46:22.472898 7f9d84175700 10 osd.2 173 send_pg_stats - 4 pgs updated
2014-06-12 14:46:22.472980 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:22.472985 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 13 >= max 0.5 = no, load too high
2014-06-12 14:46:22.473013 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:22.473015 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92630 < max (604800 seconds)
2014-06-12 14:46:22.473040 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:22.473050 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:22.473051 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:22.473054 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 8; oldest is 28.837611 seconds old
2014-06-12 14:46:22.554922 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:22.554928 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:22.554930 7f9d74bfa700 20 osd.2 173 _dispatch 0x28410700 pg_stats_ack(4 pgs tid 4850) v1
2014-06-12 14:46:22.554933 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:46:22.554939 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:22.554940 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:22.603864 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:22.616503 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19410 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8])
2014-06-12 14:46:22.616527 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19410 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8])
2014-06-12 14:46:22.616985 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:22.617012 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:22.617036 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:22.617057 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:22.617079 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:22.617106 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:22.617132 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:22.617152 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:22.617171 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:22.617191 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:22.617214 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:22.617243 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:22.617266 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:22.617289 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:22.617308 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:22.617324 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:22.617345 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:22.617366 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:22.617387 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:22.617407 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:22.617427 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:22.617448 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:22.787954 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:22.801461 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:22.801581 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:22.822513 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:22.959791 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.007679 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.031818 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:23.036814 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:23.037034 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:23.058463 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.074777 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:23.122938 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.129180 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:23.133201 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.153031 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:23.159826 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.195439 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:23.195491 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:23.201172 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.204462 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.217154 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.217249 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.218003 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.245491 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:23.246993 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:23.247022 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:23.301823 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:23.301892 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:23.440629 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.444861 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:23.473157 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:23.473193 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:23.473198 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:23.473201 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:23.473203 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 8; oldest is 29.837760 seconds old
2014-06-12 14:46:23.473219 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:23.490744 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.506677 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:23.547617 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.566421 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:23.648381 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.652151 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:23.652222 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:23.664213 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:23.664473 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:23.688712 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:23.696597 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:23.699342 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:23.939732 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:23.940261 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:24.118185 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:24.118399 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:24.152895 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:24.153004 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:24.153827 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:24.160827 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.162138 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.164730 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:24.164897 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:24.223099 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.235900 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.261633 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.306527 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.316557 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:24.316697 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:24.317205 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.367486 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:24.390219 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.403703 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.419091 7f9d5a16f700  5 -- op tracker -- , seq: 747779, time: 2014-06-12 14:46:24.417967, event: header_read, op: osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.419108 7f9d5a16f700  5 -- op tracker -- , seq: 747779, time: 2014-06-12 14:46:24.417969, event: throttled, op: osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.419117 7f9d5a16f700  5 -- op tracker -- , seq: 747779, time: 2014-06-12 14:46:24.419040, event: all_read, op: osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.419125 7f9d5a16f700  5 -- op tracker -- , seq: 747779, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.419131 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:24.419140 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:24.419150 7f9d5a16f700 15 osd.2 173 enqueue_op 0x20355300 prio 127 cost 516903 latency 0.001182 osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:24.419185 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.419198 7f9d683e1700 10 osd.2 173 dequeue_op 0x20355300 prio 127 cost 516903 latency 0.001230 osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2662 (0'0,173'2662] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2661 active]
2014-06-12 14:46:24.419234 7f9d683e1700  5 -- op tracker -- , seq: 747779, time: 2014-06-12 14:46:24.419229, event: reached_pg, op: osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.419252 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2662 (0'0,173'2662] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2661 active] handle_message: 0x20355300
2014-06-12 14:46:24.419264 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2662 (0'0,173'2662] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2661 active] sub_op_modify trans 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 v 173'2663 (transaction) 174
2014-06-12 14:46:24.419284 7f9d683e1700  5 -- op tracker -- , seq: 747779, time: 2014-06-12 14:46:24.419279, event: started, op: osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.419300 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2662 (0'0,173'2662] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2661 active] append_log log((0'0,173'2662], crt=0'0) [173'2663 (173'2661) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141066 2014-06-12 08:43:33.413471]
2014-06-12 14:46:24.419334 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2663 (0'0,173'2663] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2661 active] add_log_entry 173'2663 (173'2661) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141066 2014-06-12 08:43:33.413471
2014-06-12 14:46:24.419364 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2663 (0'0,173'2663] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2661 active] append_log  adding 1 keys
2014-06-12 14:46:24.419417 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2663, trimmed: 
2014-06-12 14:46:24.419459 7f9d683e1700  5 -- op tracker -- , seq: 747779, time: 2014-06-12 14:46:24.419454, event: started, op: osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.419472 7f9d683e1700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:24.419477 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:24.419479 7f9d683e1700  5 queue_op 0x813d740 seq 2473 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:24.419486 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2663 (0'0,173'2663] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2661 active] do_sub_op osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:24.419501 7f9d683e1700 10 osd.2 173 dequeue_op 0x20355300 finish
2014-06-12 14:46:24.419505 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.419508 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.419517 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:46:24.419522 7f9d7915f700  5 _do_op 0x813d740 seq 2473 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:24.419527 7f9d7915f700 10 _do_transaction on 0x29dbeb0
2014-06-12 14:46:24.419533 7f9d7915f700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:24.419854 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:24.419877 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:24.419926 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:24.419972 7f9d7915f700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:24.419985 7f9d7915f700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:24.419996 7f9d7915f700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:24.420015 7f9d7915f700 15 _write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 2580480~516096
2014-06-12 14:46:24.420154 7f9d7915f700 10 lookup_strip_header done  cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:24.420476 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:24.422729 7f9d7915f700 10 _generic_write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 2580480~516096 = 0
2014-06-12 14:46:24.422779 7f9d7915f700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:24.422795 7f9d7915f700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:24.422803 7f9d7915f700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:24.422811 7f9d7915f700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:24.423001 7f9d7915f700 20 set_header setting 17215 cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 parent seq 0
2014-06-12 14:46:24.423088 7f9d7915f700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:24.423107 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:24.423456 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:24.427634 7f9d7915f700 10 _do_op 0x813d740 seq 2473 r = 0, finisher 0x7eb8a00 0
2014-06-12 14:46:24.427683 7f9d7715b700  5 -- op tracker -- , seq: 747779, time: 2014-06-12 14:46:24.427672, event: commit_sent, op: osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.427695 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2663 (0'0,173'2663] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2661 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:24.427725 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:24.427754 7f9d7915f700 10 _finish_op 0x813d740 seq 2473 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:24.427762 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:46:24.427770 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.427782 7f9d7795c700  5 -- op tracker -- , seq: 747779, time: 2014-06-12 14:46:24.427774, event: sub_op_applied, op: osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.427798 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2663 (0'0,173'2663] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active] sub_op_modify_applied on 0x29dbdc0 op osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:24.427816 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2663 (0'0,173'2663] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active] op_applied on primary on version 173'2663
2014-06-12 14:46:24.427827 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2663 (0'0,173'2663] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active] op_applied on replica on version 173'2663
2014-06-12 14:46:24.427853 7f9d7795c700  5 -- op tracker -- , seq: 747779, time: 2014-06-12 14:46:24.427848, event: done, op: osd_sub_op(client.4284.0:141066 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2663 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.473393 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:24.473432 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:24.473437 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:24.473440 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:24.473443 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 8; oldest is 30.837999 seconds old
2014-06-12 14:46:24.473443 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:24.473505 7f9d84175700  0 log [WRN] : 2 slow requests, 2 included below; oldest blocked for > 30.837999 secs
2014-06-12 14:46:24.473513 7f9d84175700  0 log [WRN] : slow request 30.837999 seconds old, received at 2014-06-12 14:45:53.635443: osd_op(client.4308.0:145046 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1097728~516096] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:46:24.473521 7f9d84175700  0 log [WRN] : slow request 30.827847 seconds old, received at 2014-06-12 14:45:53.645595: osd_op(client.4308.0:145047 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1613824~2580480] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:46:24.653454 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:24.653502 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:24.788050 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.790259 7f9d5501e700  5 -- op tracker -- , seq: 747780, time: 2014-06-12 14:46:24.786900, event: header_read, op: osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.790280 7f9d5501e700  5 -- op tracker -- , seq: 747780, time: 2014-06-12 14:46:24.786903, event: throttled, op: osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.790288 7f9d5501e700  5 -- op tracker -- , seq: 747780, time: 2014-06-12 14:46:24.790193, event: all_read, op: osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.790297 7f9d5501e700  5 -- op tracker -- , seq: 747780, time: 0.000000, event: dispatched, op: osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.790303 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:24.790311 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:24.790321 7f9d5501e700 15 osd.2 173 enqueue_op 0x17c2a000 prio 127 cost 1950503 latency 0.003420 osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:24.790356 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.790369 7f9d64bda700 10 osd.2 173 dequeue_op 0x17c2a000 prio 127 cost 1950503 latency 0.003468 osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.376( v 173'2855 (0'0,173'2855] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2854 active]
2014-06-12 14:46:24.790404 7f9d64bda700  5 -- op tracker -- , seq: 747780, time: 2014-06-12 14:46:24.790400, event: reached_pg, op: osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.790422 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2855 (0'0,173'2855] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2854 active] handle_message: 0x17c2a000
2014-06-12 14:46:24.790435 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2855 (0'0,173'2855] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2854 active] sub_op_modify trans fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 v 173'2856 (transaction) 174
2014-06-12 14:46:24.790452 7f9d64bda700  5 -- op tracker -- , seq: 747780, time: 2014-06-12 14:46:24.790448, event: started, op: osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.790476 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2855 (0'0,173'2855] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2854 active] append_log log((0'0,173'2855], crt=0'0) [173'2856 (173'768) modify   fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 by client.4286.0:144898 2014-06-12 08:45:27.117384]
2014-06-12 14:46:24.790518 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2856 (0'0,173'2856] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2854 active] add_log_entry 173'2856 (173'768) modify   fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 by client.4286.0:144898 2014-06-12 08:45:27.117384
2014-06-12 14:46:24.790543 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2856 (0'0,173'2856] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2854 active] append_log  adding 1 keys
2014-06-12 14:46:24.790579 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2856, trimmed: 
2014-06-12 14:46:24.790621 7f9d64bda700  5 -- op tracker -- , seq: 747780, time: 2014-06-12 14:46:24.790616, event: started, op: osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.790634 7f9d64bda700  5 queue_transactions existing osr(3.376 0x276d410)/0x276d410
2014-06-12 14:46:24.790639 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:24.790641 7f9d64bda700  5 queue_op 0x88d2000 seq 2589 osr(3.376 0x276d410) 1951989 bytes   (queue has 1 ops and 1951989 bytes)
2014-06-12 14:46:24.790652 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e96c60 (1 active)
2014-06-12 14:46:24.790659 7f9d7d968700  5 _do_op 0x88d2000 seq 2589 osr(3.376 0x276d410)/0x276d410 start
2014-06-12 14:46:24.790648 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.376( v 173'2856 (0'0,173'2856] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2854 active] do_sub_op osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:24.790663 7f9d64bda700 10 osd.2 173 dequeue_op 0x17c2a000 finish
2014-06-12 14:46:24.790665 7f9d7d968700 10 _do_transaction on 0x29db7b0
2014-06-12 14:46:24.790666 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.790669 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.790672 7f9d7d968700 15 _omap_setkeys meta/1fe1d8da/pglog_3.376/0//-1
2014-06-12 14:46:24.790812 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe1d8da/pglog_3.376/0//-1
2014-06-12 14:46:24.790847 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:24.790889 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:24.790923 7f9d7d968700 15 _omap_rmkeys meta/1fe1d8da/pglog_3.376/0//-1
2014-06-12 14:46:24.790932 7f9d7d968700 10 _omap_rmkeys meta/1fe1d8da/pglog_3.376/0//-1 r = 0
2014-06-12 14:46:24.790938 7f9d7d968700 15 _omap_setkeys meta/1fe1d8da/pglog_3.376/0//-1
2014-06-12 14:46:24.791195 7f9d7d968700 15 _write 3.376_head/fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 0~1949696
2014-06-12 14:46:24.791414 7f9d7d968700 10 lookup_strip_header done  cid 3.376_head oid fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3
2014-06-12 14:46:24.791617 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:24.800592 7f9d7d968700 10 _generic_write 3.376_head/fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 0~1949696 = 0
2014-06-12 14:46:24.800721 7f9d7d968700 15 _setattrs 3.376_head/fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3
2014-06-12 14:46:24.800741 7f9d7d968700 10 _setattrs 3.376_head/fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:46:24.800746 7f9d7d968700 15 _setattrs 3.376_head/fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3
2014-06-12 14:46:24.800753 7f9d7d968700 10 _setattrs 3.376_head/fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:46:24.801044 7f9d7d968700 20 set_header setting 17095 cid 3.376_head oid fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 parent seq 0
2014-06-12 14:46:24.801156 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:24.801167 7f9d7d968700 20 set_header setting 2226 cid meta oid 1fe1d8da/pglog_3.376/0//-1 parent seq 0
2014-06-12 14:46:24.801173 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:24.808717 7f9d7d968700 10 _do_op 0x88d2000 seq 2589 r = 0, finisher 0x31d0f400 0
2014-06-12 14:46:24.808735 7f9d7d968700 10 _finish_op 0x88d2000 seq 2589 osr(3.376 0x276d410)/0x276d410
2014-06-12 14:46:24.808740 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e96c60 (0 active)
2014-06-12 14:46:24.808746 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.808766 7f9d7795c700  5 -- op tracker -- , seq: 747780, time: 2014-06-12 14:46:24.808749, event: sub_op_applied, op: osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.808776 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2856 (0'0,173'2856] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2854 active] sub_op_modify_applied on 0x29db6c0 op osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:24.808794 7f9d7795c700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:24.808807 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2856 (0'0,173'2856] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2854 active] op_applied on primary on version 173'2856
2014-06-12 14:46:24.808822 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2856 (0'0,173'2856] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2854 active] op_applied on replica on version 173'2856
2014-06-12 14:46:24.808841 7f9d7715b700  5 -- op tracker -- , seq: 747780, time: 2014-06-12 14:46:24.808834, event: commit_sent, op: osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.808848 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2856 (0'0,173'2856] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2854 active] sub_op_modify_commit on op osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:24.808874 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:24.808895 7f9d7715b700  5 -- op tracker -- , seq: 747780, time: 2014-06-12 14:46:24.808892, event: done, op: osd_sub_op(client.4286.0:144898 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.823972 7f9d3823f700  5 -- op tracker -- , seq: 747781, time: 2014-06-12 14:46:24.823353, event: header_read, op: osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.823994 7f9d3823f700  5 -- op tracker -- , seq: 747781, time: 2014-06-12 14:46:24.823357, event: throttled, op: osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.824000 7f9d3823f700  5 -- op tracker -- , seq: 747781, time: 2014-06-12 14:46:24.823918, event: all_read, op: osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.824014 7f9d3823f700  5 -- op tracker -- , seq: 747781, time: 0.000000, event: dispatched, op: osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.824021 7f9d3823f700 20 osd.2 173 should_share_map client.4307 172.16.96.220:0/1243935 173
2014-06-12 14:46:24.824043 7f9d3823f700 15 osd.2 173 enqueue_op 0x39577100 prio 63 cost 221184 latency 0.000690 osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:24.824072 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.824083 7f9d6abe6700 10 osd.2 173 dequeue_op 0x39577100 prio 63 cost 221184 latency 0.000730 osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean]
2014-06-12 14:46:24.824110 7f9d6abe6700  5 -- op tracker -- , seq: 747781, time: 2014-06-12 14:46:24.824099, event: reached_pg, op: osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.824126 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:24.824137 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] handle_message: 0x39577100
2014-06-12 14:46:24.824143 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] do_op osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:24.824162 7f9d6abe6700 15 getattr 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 '_'
2014-06-12 14:46:24.828858 7f9d5501e700  5 -- op tracker -- , seq: 747782, time: 2014-06-12 14:46:24.824618, event: header_read, op: osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.828875 7f9d5501e700  5 -- op tracker -- , seq: 747782, time: 2014-06-12 14:46:24.824619, event: throttled, op: osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.828885 7f9d5501e700  5 -- op tracker -- , seq: 747782, time: 2014-06-12 14:46:24.828814, event: all_read, op: osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.828894 7f9d5501e700  5 -- op tracker -- , seq: 747782, time: 0.000000, event: dispatched, op: osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.828900 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:24.828907 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:24.828917 7f9d5501e700 15 osd.2 173 enqueue_op 0x34017a00 prio 127 cost 2245415 latency 0.004298 osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:24.828947 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:24.828957 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34017a00 prio 127 cost 2245415 latency 0.004339 osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.376( v 173'2856 (0'0,173'2856] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2855 active]
2014-06-12 14:46:24.828991 7f9d6a3e5700  5 -- op tracker -- , seq: 747782, time: 2014-06-12 14:46:24.828986, event: reached_pg, op: osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.829000 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2856 (0'0,173'2856] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2855 active] handle_message: 0x34017a00
2014-06-12 14:46:24.829015 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2856 (0'0,173'2856] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2855 active] sub_op_modify trans fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 v 173'2857 (transaction) 174
2014-06-12 14:46:24.829037 7f9d6a3e5700  5 -- op tracker -- , seq: 747782, time: 2014-06-12 14:46:24.829033, event: started, op: osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.829054 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2856 (0'0,173'2856] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2855 active] append_log log((0'0,173'2856], crt=0'0) [173'2857 (173'2856) modify   fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 by client.4286.0:144899 2014-06-12 08:45:27.194467]
2014-06-12 14:46:24.829079 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2857 (0'0,173'2857] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2855 active] add_log_entry 173'2857 (173'2856) modify   fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 by client.4286.0:144899 2014-06-12 08:45:27.194467
2014-06-12 14:46:24.829104 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2857 (0'0,173'2857] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2855 active] append_log  adding 1 keys
2014-06-12 14:46:24.829139 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2857, trimmed: 
2014-06-12 14:46:24.829164 7f9d6a3e5700  5 -- op tracker -- , seq: 747782, time: 2014-06-12 14:46:24.829159, event: started, op: osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.829174 7f9d6a3e5700  5 queue_transactions existing osr(3.376 0x276d410)/0x276d410
2014-06-12 14:46:24.829179 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:24.829180 7f9d6a3e5700  5 queue_op 0x5be8ba0 seq 2590 osr(3.376 0x276d410) 2246901 bytes   (queue has 1 ops and 2246901 bytes)
2014-06-12 14:46:24.829192 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e96c60 (1 active)
2014-06-12 14:46:24.829200 7f9d83173700  5 _do_op 0x5be8ba0 seq 2590 osr(3.376 0x276d410)/0x276d410 start
2014-06-12 14:46:24.829188 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.376( v 173'2857 (0'0,173'2857] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2855 active] do_sub_op osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:24.829204 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34017a00 finish
2014-06-12 14:46:24.829205 7f9d83173700 10 _do_transaction on 0x9b92d30
2014-06-12 14:46:24.829208 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:24.829211 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.829211 7f9d83173700 15 _omap_setkeys meta/1fe1d8da/pglog_3.376/0//-1
2014-06-12 14:46:24.829317 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe1d8da/pglog_3.376/0//-1
2014-06-12 14:46:24.829339 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:24.829386 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:24.829404 7f9d83173700 15 _omap_rmkeys meta/1fe1d8da/pglog_3.376/0//-1
2014-06-12 14:46:24.829412 7f9d83173700 10 _omap_rmkeys meta/1fe1d8da/pglog_3.376/0//-1 r = 0
2014-06-12 14:46:24.829419 7f9d83173700 15 _omap_setkeys meta/1fe1d8da/pglog_3.376/0//-1
2014-06-12 14:46:24.829433 7f9d83173700 15 _write 3.376_head/fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 1949696~2244608
2014-06-12 14:46:24.829580 7f9d83173700 10 lookup_strip_header done  cid 3.376_head oid fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3
2014-06-12 14:46:24.829754 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:24.840259 7f9d83173700 10 _generic_write 3.376_head/fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 1949696~2244608 = 0
2014-06-12 14:46:24.840420 7f9d83173700 15 _setattrs 3.376_head/fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3
2014-06-12 14:46:24.840440 7f9d83173700 10 _setattrs 3.376_head/fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:46:24.840447 7f9d83173700 15 _setattrs 3.376_head/fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3
2014-06-12 14:46:24.840460 7f9d83173700 10 _setattrs 3.376_head/fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:46:24.840948 7f9d83173700 20 set_header setting 17095 cid 3.376_head oid fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 parent seq 0
2014-06-12 14:46:24.841180 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:24.841203 7f9d83173700 20 set_header setting 2226 cid meta oid 1fe1d8da/pglog_3.376/0//-1 parent seq 0
2014-06-12 14:46:24.841213 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:24.850832 7f9d83173700 10 _do_op 0x5be8ba0 seq 2590 r = 0, finisher 0x7ffa1a0 0
2014-06-12 14:46:24.850854 7f9d83173700 10 _finish_op 0x5be8ba0 seq 2590 osr(3.376 0x276d410)/0x276d410
2014-06-12 14:46:24.850867 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e96c60 (0 active)
2014-06-12 14:46:24.850876 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.850889 7f9d7715b700  5 -- op tracker -- , seq: 747782, time: 2014-06-12 14:46:24.850875, event: commit_sent, op: osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.850900 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2857 (0'0,173'2857] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2855 active] sub_op_modify_commit on op osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:24.850932 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:24.850968 7f9d7795c700  5 -- op tracker -- , seq: 747782, time: 2014-06-12 14:46:24.850961, event: sub_op_applied, op: osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.850985 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2857 (0'0,173'2857] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2856 active] sub_op_modify_applied on 0x9b92c40 op osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:24.851009 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2857 (0'0,173'2857] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2856 active] op_applied on primary on version 173'2857
2014-06-12 14:46:24.851016 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.376( v 173'2857 (0'0,173'2857] local-les=161 n=300 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2856 active] op_applied on replica on version 173'2857
2014-06-12 14:46:24.851038 7f9d7795c700  5 -- op tracker -- , seq: 747782, time: 2014-06-12 14:46:24.851035, event: done, op: osd_sub_op(client.4286.0:144899 3.376 fa3e2b76/rbd_data.10956b8b4567.0000000000001085/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:24.879199 7f9d6abe6700 10 getattr 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 '_' = 0
2014-06-12 14:46:24.879231 7f9d6abe6700 15 getattr 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 'snapset'
2014-06-12 14:46:24.879346 7f9d6abe6700 10 getattr 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 'snapset' = 0
2014-06-12 14:46:24.879356 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] populate_obc_watchers c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.879376 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.879383 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] get_object_context: creating obc from disk: 0x135b6dc0
2014-06-12 14:46:24.879389 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] get_object_context: 0x135b6dc0 c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 rwstate(none n=0 w=0) oi: c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3(173'820 client.4307.0:40468 wrlock_by=unknown.0.0:0 dirty s 4194304 uv820) ssc: 0x82df420 snapset: 0=[]:[]+head
2014-06-12 14:46:24.879400 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] find_object_context c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 @head oi=c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3(173'820 client.4307.0:40468 wrlock_by=unknown.0.0:0 dirty s 4194304 uv820)
2014-06-12 14:46:24.879425 7f9d6abe6700  5 -- op tracker -- , seq: 747781, time: 2014-06-12 14:46:24.879418, event: started, op: osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.879437 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] execute_ctx 0x17cfca00
2014-06-12 14:46:24.879448 7f9d6abe6700  5 -- op tracker -- , seq: 747781, time: 2014-06-12 14:46:24.879445, event: started, op: osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.879454 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] do_op c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] ov 173'820 av 173'2677 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:24.879464 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] do_osd_op c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184]
2014-06-12 14:46:24.879472 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:24.879481 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] do_osd_op  write 0~221184
2014-06-12 14:46:24.879493 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] do_osd_op_effects on session 0x507d200
2014-06-12 14:46:24.879501 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] make_writeable c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 snapset=0x82df458  snapc=0=[]
2014-06-12 14:46:24.879508 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] make_writeable c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:24.879515 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] finish_ctx c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 0x17cfca00 op modify  
2014-06-12 14:46:24.879534 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean]  set mtime to 2014-06-12 08:45:27.203761
2014-06-12 14:46:24.879565 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean]  final snapset 0=[]:[]+head in c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.879592 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean]  zeroing write result code 0
2014-06-12 14:46:24.879609 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] new_repop rep_tid 236251 on osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:24.879633 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] issue_repop rep_tid 236251 o c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.879667 7f9d6abe6700  5 -- op tracker -- , seq: 747781, time: 2014-06-12 14:46:24.879660, event: waiting for subops from 21, op: osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.879700 7f9d6abe6700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:24.879735 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2676 (0'0,173'2676] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] append_log log((0'0,173'2676], crt=0'0) [173'2677 (173'820) modify   c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 by client.4307.0:146732 2014-06-12 08:45:27.203761]
2014-06-12 14:46:24.879770 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2676 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] add_log_entry 173'2677 (173'820) modify   c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 by client.4307.0:146732 2014-06-12 08:45:27.203761
2014-06-12 14:46:24.879798 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2676 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] append_log  adding 1 keys
2014-06-12 14:46:24.879838 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2677, trimmed: 
2014-06-12 14:46:24.879869 7f9d6abe6700  5 queue_transactions existing osr(3.7db 0x49b4380)/0x49b4380
2014-06-12 14:46:24.879877 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:24.879879 7f9d6abe6700  5 queue_op 0x5e02660 seq 2515 osr(3.7db 0x49b4380) 223477 bytes   (queue has 1 ops and 223477 bytes)
2014-06-12 14:46:24.879899 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e970e0 (1 active)
2014-06-12 14:46:24.879906 7f9d7c165700  5 _do_op 0x5e02660 seq 2515 osr(3.7db 0x49b4380)/0x49b4380 start
2014-06-12 14:46:24.879894 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2676 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] eval_repop repgather(0x52f4840 173'2677 rep_tid=236251 committed?=0 applied?=0 lock=0 op=osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.879910 7f9d7c165700 10 _do_transaction on 0x52f4600
2014-06-12 14:46:24.879911 7f9d6abe6700 10 osd.2 173 dequeue_op 0x39577100 finish
2014-06-12 14:46:24.879916 7f9d7c165700 15 _omap_setkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.879916 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.879921 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.880005 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.880040 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:24.880138 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:24.880193 7f9d7c165700 15 _omap_rmkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.880206 7f9d7c165700 10 _omap_rmkeys meta/1fe719b6/pglog_3.7db/0//-1 r = 0
2014-06-12 14:46:24.880227 7f9d7c165700 15 _omap_setkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.880252 7f9d7c165700 15 _write 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 0~221184
2014-06-12 14:46:24.880466 7f9d7c165700 10 lookup_strip_header done  cid 3.7db_head oid c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.880826 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:24.881764 7f9d7c165700 10 _generic_write 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 0~221184 = 0
2014-06-12 14:46:24.881787 7f9d7c165700 15 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.881799 7f9d7c165700 10 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 = 0
2014-06-12 14:46:24.881804 7f9d7c165700 15 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.881810 7f9d7c165700 10 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 = 0
2014-06-12 14:46:24.881908 7f9d7c165700 20 set_header setting 18475 cid 3.7db_head oid c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 parent seq 0
2014-06-12 14:46:24.881942 7f9d7c165700 20 set_header setting 2156 cid meta oid 1fe719b6/pglog_3.7db/0//-1 parent seq 0
2014-06-12 14:46:24.881952 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:24.881958 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:24.882625 7f9d7c165700 10 _do_op 0x5e02660 seq 2515 r = 0, finisher 0xc52bd80 0x33091a80
2014-06-12 14:46:24.882634 7f9d7c165700 10 _finish_op 0x5e02660 seq 2515 osr(3.7db 0x49b4380)/0x49b4380
2014-06-12 14:46:24.882639 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e970e0 (0 active)
2014-06-12 14:46:24.882640 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.882647 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2676 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] op_applied: 236251
2014-06-12 14:46:24.882676 7f9d7795c700  5 -- op tracker -- , seq: 747781, time: 2014-06-12 14:46:24.882668, event: op_applied, op: osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.882686 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2676 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] op_applied on primary on version 173'2677
2014-06-12 14:46:24.882718 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2676 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] op_commit: 236251
2014-06-12 14:46:24.882741 7f9d7715b700  5 -- op tracker -- , seq: 747781, time: 2014-06-12 14:46:24.882736, event: op_commit, op: osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.910195 7f9d44101700  5 -- op tracker -- , seq: 747783, time: 2014-06-12 14:46:24.910131, event: header_read, op: osd_sub_op_reply(client.4307.0:146732 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.910224 7f9d44101700  5 -- op tracker -- , seq: 747783, time: 2014-06-12 14:46:24.910132, event: throttled, op: osd_sub_op_reply(client.4307.0:146732 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.910232 7f9d44101700  5 -- op tracker -- , seq: 747783, time: 2014-06-12 14:46:24.910144, event: all_read, op: osd_sub_op_reply(client.4307.0:146732 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.910241 7f9d44101700  5 -- op tracker -- , seq: 747783, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4307.0:146732 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.910246 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4307.0:146732 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:24.910251 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:24.910260 7f9d44101700 15 osd.2 173 enqueue_op 0x20e87800 prio 196 cost 0 latency 0.000128 osd_sub_op_reply(client.4307.0:146732 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:24.910288 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.910296 7f9d66bde700 10 osd.2 173 dequeue_op 0x20e87800 prio 196 cost 0 latency 0.000165 osd_sub_op_reply(client.4307.0:146732 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2 pg pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2676 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean]
2014-06-12 14:46:24.910320 7f9d66bde700  5 -- op tracker -- , seq: 747783, time: 2014-06-12 14:46:24.910316, event: reached_pg, op: osd_sub_op_reply(client.4307.0:146732 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.910339 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2676 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] handle_message: 0x20e87800
2014-06-12 14:46:24.910356 7f9d66bde700  5 -- op tracker -- , seq: 747783, time: 2014-06-12 14:46:24.910352, event: started, op: osd_sub_op_reply(client.4307.0:146732 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.910364 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2676 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] sub_op_modify_reply: tid 236251 op  ack_type 4 from 21
2014-06-12 14:46:24.910400 7f9d66bde700  5 -- op tracker -- , seq: 747781, time: 2014-06-12 14:46:24.910392, event: sub_op_commit_rec, op: osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.910410 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2676 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] repop_all_applied: repop tid 236251 all applied 
2014-06-12 14:46:24.910423 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2676 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] eval_repop repgather(0x52f4840 173'2677 rep_tid=236251 committed?=0 applied?=1 lock=0 op=osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.910454 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2676 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean]  sending ack on repgather(0x52f4840 173'2677 rep_tid=236251 committed?=0 applied?=1 lock=0 op=osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) 0x2ed84f00
2014-06-12 14:46:24.910485 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2676 crt=0'0 lcod 173'2675 mlcod 173'2675 active+clean] repop_all_committed: repop tid 236251 all committed 
2014-06-12 14:46:24.910496 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2675 active+clean] eval_repop repgather(0x52f4840 173'2677 rep_tid=236251 committed?=1 applied?=1 lock=0 op=osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.910512 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2675 active+clean] log_op_stats osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 inb 221463 outb 0 rlat 0.087131 lat 0.087158
2014-06-12 14:46:24.910529 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2675 active+clean] publish_stats_to_osd 173:3463
2014-06-12 14:46:24.910542 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2675 active+clean]  sending commit on repgather(0x52f4840 173'2677 rep_tid=236251 committed?=1 applied?=1 lock=0 op=osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) 0x19e0b180
2014-06-12 14:46:24.910573 7f9d66bde700  5 -- op tracker -- , seq: 747781, time: 2014-06-12 14:46:24.910568, event: commit_sent, op: osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.910582 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2675 active+clean]  requeue_ops 
2014-06-12 14:46:24.910592 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean]  removing repgather(0x52f4840 173'2677 rep_tid=236251 committed?=1 applied?=1 op=osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.910606 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean]    q front is repgather(0x52f4840 173'2677 rep_tid=236251 committed?=1 applied?=1 op=osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.910620 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] remove_repop repgather(0x52f4840 173'2677 rep_tid=236251 committed?=1 applied?=1 op=osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.910636 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean]  requeue_ops 
2014-06-12 14:46:24.910665 7f9d66bde700  5 -- op tracker -- , seq: 747781, time: 2014-06-12 14:46:24.910659, event: done, op: osd_op(client.4307.0:146732 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.910754 7f9d66bde700 10 osd.2 173 dequeue_op 0x20e87800 finish
2014-06-12 14:46:24.910765 7f9d66bde700  5 -- op tracker -- , seq: 747783, time: 2014-06-12 14:46:24.910761, event: done, op: osd_sub_op_reply(client.4307.0:146732 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.910779 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.910785 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.917088 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19415 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8])
2014-06-12 14:46:24.917110 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19415 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8])
2014-06-12 14:46:24.917188 7f9d3823f700  5 -- op tracker -- , seq: 747784, time: 2014-06-12 14:46:24.914384, event: header_read, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.917220 7f9d3823f700  5 -- op tracker -- , seq: 747784, time: 2014-06-12 14:46:24.914387, event: throttled, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.917232 7f9d3823f700  5 -- op tracker -- , seq: 747784, time: 2014-06-12 14:46:24.917147, event: all_read, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.917242 7f9d3823f700  5 -- op tracker -- , seq: 747784, time: 0.000000, event: dispatched, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.917252 7f9d3823f700 20 osd.2 173 should_share_map client.4307 172.16.96.220:0/1243935 173
2014-06-12 14:46:24.917282 7f9d3823f700 15 osd.2 173 enqueue_op 0x39577b00 prio 63 cost 1032192 latency 0.002898 osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:24.917327 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.917338 7f9d69be4700 10 osd.2 173 dequeue_op 0x39577b00 prio 63 cost 1032192 latency 0.002954 osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean]
2014-06-12 14:46:24.917367 7f9d69be4700  5 -- op tracker -- , seq: 747784, time: 2014-06-12 14:46:24.917363, event: reached_pg, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.917381 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:24.917398 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] handle_message: 0x39577b00
2014-06-12 14:46:24.917406 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] do_op osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:24.917443 7f9d69be4700 15 getattr 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 '_'
2014-06-12 14:46:24.917628 7f9d69be4700 10 getattr 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 '_' = 0
2014-06-12 14:46:24.917645 7f9d69be4700 15 getattr 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 'snapset'
2014-06-12 14:46:24.917728 7f9d69be4700 10 getattr 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 'snapset' = 0
2014-06-12 14:46:24.917737 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] populate_obc_watchers c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.917754 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.917764 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] get_object_context: creating obc from disk: 0x2c966dc0
2014-06-12 14:46:24.917771 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] get_object_context: 0x2c966dc0 c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 rwstate(none n=0 w=0) oi: c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3(173'2677 client.4307.0:146732 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2677) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:46:24.917782 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] find_object_context c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 @head oi=c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3(173'2677 client.4307.0:146732 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2677)
2014-06-12 14:46:24.917806 7f9d69be4700  5 -- op tracker -- , seq: 747784, time: 2014-06-12 14:46:24.917801, event: started, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.917813 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] execute_ctx 0xfd19200
2014-06-12 14:46:24.917830 7f9d69be4700  5 -- op tracker -- , seq: 747784, time: 2014-06-12 14:46:24.917825, event: started, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.917837 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] do_op c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] ov 173'2677 av 173'2678 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:24.917848 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] do_osd_op c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192]
2014-06-12 14:46:24.917856 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:24.917869 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] do_osd_op  write 221184~1032192
2014-06-12 14:46:24.917881 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] do_osd_op_effects on session 0x507d200
2014-06-12 14:46:24.917888 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] make_writeable c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:46:24.917896 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] make_writeable c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:24.917905 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] finish_ctx c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 0xfd19200 op modify  
2014-06-12 14:46:24.917918 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean]  set mtime to 2014-06-12 08:45:27.293371
2014-06-12 14:46:24.917941 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean]  final snapset 0=[]:[]+head in c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.917960 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean]  zeroing write result code 0
2014-06-12 14:46:24.917971 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] new_repop rep_tid 236252 on osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:24.917988 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] issue_repop rep_tid 236252 o c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.918012 7f9d69be4700  5 -- op tracker -- , seq: 747784, time: 2014-06-12 14:46:24.918008, event: waiting for subops from 21, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.918028 7f9d69be4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:24.918042 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2677 (0'0,173'2677] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] append_log log((0'0,173'2677], crt=0'0) [173'2678 (173'2677) modify   c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 by client.4307.0:146733 2014-06-12 08:45:27.293371]
2014-06-12 14:46:24.918056 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] add_log_entry 173'2678 (173'2677) modify   c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 by client.4307.0:146733 2014-06-12 08:45:27.293371
2014-06-12 14:46:24.918069 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] append_log  adding 1 keys
2014-06-12 14:46:24.918096 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2678, trimmed: 
2014-06-12 14:46:24.918117 7f9d69be4700  5 queue_transactions existing osr(3.7db 0x49b4380)/0x49b4380
2014-06-12 14:46:24.918120 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:24.918121 7f9d69be4700  5 queue_op 0x77122a0 seq 2516 osr(3.7db 0x49b4380) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:24.918127 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] eval_repop repgather(0x5236240 173'2678 rep_tid=236252 committed?=0 applied?=0 lock=0 op=osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.918141 7f9d69be4700 10 osd.2 173 dequeue_op 0x39577b00 finish
2014-06-12 14:46:24.918146 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.918149 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.918795 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e970e0 (1 active)
2014-06-12 14:46:24.918805 7f9d82972700  5 _do_op 0x77122a0 seq 2516 osr(3.7db 0x49b4380)/0x49b4380 start
2014-06-12 14:46:24.918812 7f9d82972700 10 _do_transaction on 0x5312cc0
2014-06-12 14:46:24.918825 7f9d82972700 15 _omap_setkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.918932 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.918955 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:24.918999 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:24.919017 7f9d82972700 15 _omap_rmkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.919024 7f9d82972700 10 _omap_rmkeys meta/1fe719b6/pglog_3.7db/0//-1 r = 0
2014-06-12 14:46:24.919032 7f9d82972700 15 _omap_setkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.919052 7f9d82972700 15 _write 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 221184~1032192
2014-06-12 14:46:24.919110 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:24.919155 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:24.919188 7f9d82972700 10 lookup_strip_header done  cid 3.7db_head oid c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.919606 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:24.921705 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:24.921727 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:24.921763 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:24.921779 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:24.921796 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:24.921815 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:24.921832 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:24.921847 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:24.921867 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:24.921883 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:24.921899 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:24.921917 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:24.921934 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:24.921949 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:24.921969 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:24.921985 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:24.922001 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:24.922017 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:24.922036 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:24.922051 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:24.923846 7f9d82972700 10 _generic_write 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 221184~1032192 = 0
2014-06-12 14:46:24.923989 7f9d82972700 15 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.924019 7f9d82972700 10 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 = 0
2014-06-12 14:46:24.924025 7f9d82972700 15 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.924033 7f9d82972700 10 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 = 0
2014-06-12 14:46:24.924388 7f9d82972700 20 set_header setting 18475 cid 3.7db_head oid c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 parent seq 0
2014-06-12 14:46:24.924514 7f9d82972700 20 set_header setting 2156 cid meta oid 1fe719b6/pglog_3.7db/0//-1 parent seq 0
2014-06-12 14:46:24.924531 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:24.924540 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:24.929571 7f9d82972700 10 _do_op 0x77122a0 seq 2516 r = 0, finisher 0x8022900 0x39f51f80
2014-06-12 14:46:24.929605 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] op_commit: 236252
2014-06-12 14:46:24.929657 7f9d7715b700  5 -- op tracker -- , seq: 747784, time: 2014-06-12 14:46:24.929647, event: op_commit, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.929671 7f9d82972700 10 _finish_op 0x77122a0 seq 2516 osr(3.7db 0x49b4380)/0x49b4380
2014-06-12 14:46:24.929680 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e970e0 (0 active)
2014-06-12 14:46:24.929687 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.929687 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] op_applied: 236252
2014-06-12 14:46:24.929713 7f9d7795c700  5 -- op tracker -- , seq: 747784, time: 2014-06-12 14:46:24.929706, event: op_applied, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.929724 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] op_applied on primary on version 173'2678
2014-06-12 14:46:24.931869 7f9d44101700  5 -- op tracker -- , seq: 747785, time: 2014-06-12 14:46:24.931811, event: header_read, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ack, result = 0)
2014-06-12 14:46:24.931880 7f9d44101700  5 -- op tracker -- , seq: 747785, time: 2014-06-12 14:46:24.931813, event: throttled, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ack, result = 0)
2014-06-12 14:46:24.931885 7f9d44101700  5 -- op tracker -- , seq: 747785, time: 2014-06-12 14:46:24.931835, event: all_read, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ack, result = 0)
2014-06-12 14:46:24.931894 7f9d44101700  5 -- op tracker -- , seq: 747785, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ack, result = 0)
2014-06-12 14:46:24.931899 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:24.931903 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:24.931909 7f9d44101700 15 osd.2 173 enqueue_op 0x160e5a00 prio 196 cost 0 latency 0.000098 osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ack, result = 0) v2
2014-06-12 14:46:24.931936 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.931945 7f9d653db700 10 osd.2 173 dequeue_op 0x160e5a00 prio 196 cost 0 latency 0.000134 osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ack, result = 0) v2 pg pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean]
2014-06-12 14:46:24.931958 7f9d44101700  5 -- op tracker -- , seq: 747786, time: 2014-06-12 14:46:24.931927, event: header_read, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.931971 7f9d653db700  5 -- op tracker -- , seq: 747785, time: 2014-06-12 14:46:24.931967, event: reached_pg, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ack, result = 0)
2014-06-12 14:46:24.931986 7f9d44101700  5 -- op tracker -- , seq: 747786, time: 2014-06-12 14:46:24.931928, event: throttled, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.931984 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] handle_message: 0x160e5a00
2014-06-12 14:46:24.932012 7f9d44101700  5 -- op tracker -- , seq: 747786, time: 2014-06-12 14:46:24.931941, event: all_read, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.932024 7f9d44101700  5 -- op tracker -- , seq: 747786, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.932032 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:24.932037 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:24.932036 7f9d653db700  5 -- op tracker -- , seq: 747785, time: 2014-06-12 14:46:24.932008, event: started, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ack, result = 0)
2014-06-12 14:46:24.932043 7f9d44101700 15 osd.2 173 enqueue_op 0x160e5700 prio 196 cost 0 latency 0.000116 osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:24.932045 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] sub_op_modify_reply: tid 236252 op  ack_type 1 from 21
2014-06-12 14:46:24.932062 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:24.932065 7f9d653db700  5 -- op tracker -- , seq: 747784, time: 2014-06-12 14:46:24.932057, event: sub_op_applied_rec, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.932074 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] repop_all_applied: repop tid 236252 all applied 
2014-06-12 14:46:24.932082 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] eval_repop repgather(0x5236240 173'2678 rep_tid=236252 committed?=0 applied?=1 lock=0 op=osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.932096 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean]  sending ack on repgather(0x5236240 173'2678 rep_tid=236252 committed?=0 applied?=1 lock=0 op=osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) 0x2d88ef00
2014-06-12 14:46:24.932124 7f9d653db700 10 osd.2 173 dequeue_op 0x160e5a00 finish
2014-06-12 14:46:24.932132 7f9d653db700  5 -- op tracker -- , seq: 747785, time: 2014-06-12 14:46:24.932130, event: done, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ack, result = 0)
2014-06-12 14:46:24.932133 7f9d63bd8700 10 osd.2 173 dequeue_op 0x160e5700 prio 196 cost 0 latency 0.000206 osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2 pg pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean]
2014-06-12 14:46:24.932149 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:24.932183 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.932187 7f9d63bd8700  5 -- op tracker -- , seq: 747786, time: 2014-06-12 14:46:24.932183, event: reached_pg, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.932195 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] handle_message: 0x160e5700
2014-06-12 14:46:24.932205 7f9d63bd8700  5 -- op tracker -- , seq: 747786, time: 2014-06-12 14:46:24.932203, event: started, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.932209 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] sub_op_modify_reply: tid 236252 op  ack_type 4 from 21
2014-06-12 14:46:24.932220 7f9d63bd8700  5 -- op tracker -- , seq: 747784, time: 2014-06-12 14:46:24.932216, event: sub_op_commit_rec, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.932226 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2677 crt=0'0 lcod 173'2676 mlcod 173'2676 active+clean] repop_all_committed: repop tid 236252 all committed 
2014-06-12 14:46:24.932237 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2676 active+clean] eval_repop repgather(0x5236240 173'2678 rep_tid=236252 committed?=1 applied?=1 lock=0 op=osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.932250 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2676 active+clean] log_op_stats osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.017738 lat 0.017865
2014-06-12 14:46:24.932263 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2676 active+clean] publish_stats_to_osd 173:3464
2014-06-12 14:46:24.932273 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2676 active+clean]  sending commit on repgather(0x5236240 173'2678 rep_tid=236252 committed?=1 applied?=1 lock=0 op=osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) 0x11667b80
2014-06-12 14:46:24.932298 7f9d63bd8700  5 -- op tracker -- , seq: 747784, time: 2014-06-12 14:46:24.932292, event: commit_sent, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.932308 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2676 active+clean]  requeue_ops 
2014-06-12 14:46:24.932321 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean]  removing repgather(0x5236240 173'2678 rep_tid=236252 committed?=1 applied?=1 op=osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.932333 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean]    q front is repgather(0x5236240 173'2678 rep_tid=236252 committed?=1 applied?=1 op=osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.932342 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] remove_repop repgather(0x5236240 173'2678 rep_tid=236252 committed?=1 applied?=1 op=osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.932352 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean]  requeue_ops 
2014-06-12 14:46:24.932370 7f9d63bd8700  5 -- op tracker -- , seq: 747784, time: 2014-06-12 14:46:24.932367, event: done, op: osd_op(client.4307.0:146733 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.932387 7f9d63bd8700 10 osd.2 173 dequeue_op 0x160e5700 finish
2014-06-12 14:46:24.932393 7f9d63bd8700  5 -- op tracker -- , seq: 747786, time: 2014-06-12 14:46:24.932390, event: done, op: osd_sub_op_reply(client.4307.0:146733 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.932399 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.932401 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.933348 7f9d5a16f700  5 -- op tracker -- , seq: 747787, time: 2014-06-12 14:46:24.933284, event: header_read, op: osd_sub_op_reply(client.4308.0:145046 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.933371 7f9d5a16f700  5 -- op tracker -- , seq: 747787, time: 2014-06-12 14:46:24.933285, event: throttled, op: osd_sub_op_reply(client.4308.0:145046 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.933391 7f9d5a16f700  5 -- op tracker -- , seq: 747787, time: 2014-06-12 14:46:24.933295, event: all_read, op: osd_sub_op_reply(client.4308.0:145046 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.933399 7f9d5a16f700  5 -- op tracker -- , seq: 747787, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145046 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.933404 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145046 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:24.933409 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:24.933416 7f9d5a16f700 15 osd.2 173 enqueue_op 0x25e43c00 prio 196 cost 0 latency 0.000132 osd_sub_op_reply(client.4308.0:145046 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:24.933436 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.933442 7f9d68be2700 10 osd.2 173 dequeue_op 0x25e43c00 prio 196 cost 0 latency 0.000157 osd_sub_op_reply(client.4308.0:145046 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0) v2 pg pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean]
2014-06-12 14:46:24.933457 7f9d68be2700  5 -- op tracker -- , seq: 747787, time: 2014-06-12 14:46:24.933455, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145046 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.933463 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean] handle_message: 0x25e43c00
2014-06-12 14:46:24.933473 7f9d68be2700  5 -- op tracker -- , seq: 747787, time: 2014-06-12 14:46:24.933471, event: started, op: osd_sub_op_reply(client.4308.0:145046 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.933477 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean] sub_op_modify_reply: tid 236106 op  ack_type 4 from 24
2014-06-12 14:46:24.933489 7f9d68be2700  5 -- op tracker -- , seq: 747219, time: 2014-06-12 14:46:24.933485, event: sub_op_commit_rec, op: osd_op(client.4308.0:145046 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1097728~516096] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.933493 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean] repop_all_applied: repop tid 236106 all applied 
2014-06-12 14:46:24.933499 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean] eval_repop repgather(0x2a37a6c0 173'2706 rep_tid=236106 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145046 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1097728~516096] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.933510 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean]  sending ack on repgather(0x2a37a6c0 173'2706 rep_tid=236106 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145046 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1097728~516096] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) 0x2b167180
2014-06-12 14:46:24.933532 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2705 crt=0'0 lcod 173'2704 mlcod 173'2704 active+clean] repop_all_committed: repop tid 236106 all committed 
2014-06-12 14:46:24.933539 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean] eval_repop repgather(0x2a37a6c0 173'2706 rep_tid=236106 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145046 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1097728~516096] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.933549 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean] log_op_stats osd_op(client.4308.0:145046 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1097728~516096] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 31.298088 lat 31.298105
2014-06-12 14:46:24.933560 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean] publish_stats_to_osd 173:2889
2014-06-12 14:46:24.933568 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean]  sending commit on repgather(0x2a37a6c0 173'2706 rep_tid=236106 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145046 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1097728~516096] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) 0x2183aa00
2014-06-12 14:46:24.933588 7f9d68be2700  5 -- op tracker -- , seq: 747219, time: 2014-06-12 14:46:24.933585, event: commit_sent, op: osd_op(client.4308.0:145046 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1097728~516096] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.933594 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2704 active+clean]  requeue_ops 
2014-06-12 14:46:24.933600 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]  removing repgather(0x2a37a6c0 173'2706 rep_tid=236106 committed?=1 applied?=1 op=osd_op(client.4308.0:145046 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1097728~516096] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.933610 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]    q front is repgather(0x2a37a6c0 173'2706 rep_tid=236106 committed?=1 applied?=1 op=osd_op(client.4308.0:145046 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1097728~516096] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.933627 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] remove_repop repgather(0x2a37a6c0 173'2706 rep_tid=236106 committed?=1 applied?=1 op=osd_op(client.4308.0:145046 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1097728~516096] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.933636 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]  requeue_ops 
2014-06-12 14:46:24.933649 7f9d68be2700  5 -- op tracker -- , seq: 747219, time: 2014-06-12 14:46:24.933646, event: done, op: osd_op(client.4308.0:145046 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1097728~516096] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.933664 7f9d68be2700 10 osd.2 173 dequeue_op 0x25e43c00 finish
2014-06-12 14:46:24.933669 7f9d68be2700  5 -- op tracker -- , seq: 747787, time: 2014-06-12 14:46:24.933667, event: done, op: osd_sub_op_reply(client.4308.0:145046 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.933676 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.933678 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.938353 7f9d3823f700  5 -- op tracker -- , seq: 747788, time: 2014-06-12 14:46:24.935795, event: header_read, op: osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.938366 7f9d3823f700  5 -- op tracker -- , seq: 747788, time: 2014-06-12 14:46:24.935797, event: throttled, op: osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.938376 7f9d3823f700  5 -- op tracker -- , seq: 747788, time: 2014-06-12 14:46:24.938306, event: all_read, op: osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.938393 7f9d3823f700  5 -- op tracker -- , seq: 747788, time: 0.000000, event: dispatched, op: osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.938402 7f9d3823f700 20 osd.2 173 should_share_map client.4307 172.16.96.220:0/1243935 173
2014-06-12 14:46:24.938431 7f9d3823f700 15 osd.2 173 enqueue_op 0x20e88500 prio 63 cost 1032192 latency 0.002636 osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:24.938497 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.938505 7f9d623d5700 10 osd.2 173 dequeue_op 0x20e88500 prio 63 cost 1032192 latency 0.002709 osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean]
2014-06-12 14:46:24.938526 7f9d623d5700  5 -- op tracker -- , seq: 747788, time: 2014-06-12 14:46:24.938522, event: reached_pg, op: osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.938539 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:24.938550 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] handle_message: 0x20e88500
2014-06-12 14:46:24.938557 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] do_op osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:24.938577 7f9d623d5700 15 getattr 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 '_'
2014-06-12 14:46:24.938767 7f9d623d5700 10 getattr 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 '_' = 0
2014-06-12 14:46:24.938785 7f9d623d5700 15 getattr 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 'snapset'
2014-06-12 14:46:24.938829 7f9d623d5700 10 getattr 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 'snapset' = 0
2014-06-12 14:46:24.938837 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] populate_obc_watchers c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.938845 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.938851 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] get_object_context: creating obc from disk: 0xa45c840
2014-06-12 14:46:24.938857 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] get_object_context: 0xa45c840 c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 rwstate(none n=0 w=0) oi: c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3(173'2678 client.4307.0:146733 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2678) ssc: 0x1550ee00 snapset: 0=[]:[]+head
2014-06-12 14:46:24.938867 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] find_object_context c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 @head oi=c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3(173'2678 client.4307.0:146733 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2678)
2014-06-12 14:46:24.938890 7f9d623d5700  5 -- op tracker -- , seq: 747788, time: 2014-06-12 14:46:24.938886, event: started, op: osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.938897 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] execute_ctx 0x30f7de00
2014-06-12 14:46:24.938908 7f9d623d5700  5 -- op tracker -- , seq: 747788, time: 2014-06-12 14:46:24.938906, event: started, op: osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.938913 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] do_op c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] ov 173'2678 av 173'2679 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:24.938922 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] do_osd_op c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192]
2014-06-12 14:46:24.938928 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:24.938941 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] do_osd_op  write 1253376~1032192
2014-06-12 14:46:24.938952 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] do_osd_op_effects on session 0x507d200
2014-06-12 14:46:24.938959 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] make_writeable c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 snapset=0x1550ee38  snapc=0=[]
2014-06-12 14:46:24.938966 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] make_writeable c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:24.938972 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] finish_ctx c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 0x30f7de00 op modify  
2014-06-12 14:46:24.938980 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean]  set mtime to 2014-06-12 08:45:27.314787
2014-06-12 14:46:24.938993 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean]  final snapset 0=[]:[]+head in c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.939006 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean]  zeroing write result code 0
2014-06-12 14:46:24.939013 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] new_repop rep_tid 236253 on osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:24.939023 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] issue_repop rep_tid 236253 o c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.939040 7f9d623d5700  5 -- op tracker -- , seq: 747788, time: 2014-06-12 14:46:24.939037, event: waiting for subops from 21, op: osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.939055 7f9d623d5700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:24.939074 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2678 (0'0,173'2678] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] append_log log((0'0,173'2678], crt=0'0) [173'2679 (173'2678) modify   c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 by client.4307.0:146734 2014-06-12 08:45:27.314787]
2014-06-12 14:46:24.939089 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] add_log_entry 173'2679 (173'2678) modify   c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 by client.4307.0:146734 2014-06-12 08:45:27.314787
2014-06-12 14:46:24.939102 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] append_log  adding 1 keys
2014-06-12 14:46:24.939123 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2679, trimmed: 
2014-06-12 14:46:24.939136 7f9d623d5700  5 queue_transactions existing osr(3.7db 0x49b4380)/0x49b4380
2014-06-12 14:46:24.939139 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:24.939140 7f9d623d5700  5 queue_op 0x6c32c00 seq 2517 osr(3.7db 0x49b4380) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:24.939150 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e970e0 (1 active)
2014-06-12 14:46:24.939157 7f9d81970700  5 _do_op 0x6c32c00 seq 2517 osr(3.7db 0x49b4380)/0x49b4380 start
2014-06-12 14:46:24.939146 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] eval_repop repgather(0x5390900 173'2679 rep_tid=236253 committed?=0 applied?=0 lock=0 op=osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.939161 7f9d81970700 10 _do_transaction on 0x2010e480
2014-06-12 14:46:24.939163 7f9d623d5700 10 osd.2 173 dequeue_op 0x20e88500 finish
2014-06-12 14:46:24.939167 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.939168 7f9d81970700 15 _omap_setkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.939170 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.939249 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.939271 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:24.940136 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:24.940161 7f9d81970700 15 _omap_rmkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.940167 7f9d81970700 10 _omap_rmkeys meta/1fe719b6/pglog_3.7db/0//-1 r = 0
2014-06-12 14:46:24.940173 7f9d81970700 15 _omap_setkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.940187 7f9d81970700 15 _write 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 1253376~1032192
2014-06-12 14:46:24.940717 7f9d81970700 10 lookup_strip_header done  cid 3.7db_head oid c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.941172 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:24.946485 7f9d81970700 10 _generic_write 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 1253376~1032192 = 0
2014-06-12 14:46:24.946609 7f9d81970700 15 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.946633 7f9d81970700 10 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 = 0
2014-06-12 14:46:24.946641 7f9d81970700 15 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.946653 7f9d81970700 10 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 = 0
2014-06-12 14:46:24.947034 7f9d81970700 20 set_header setting 18475 cid 3.7db_head oid c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 parent seq 0
2014-06-12 14:46:24.947150 7f9d81970700 20 set_header setting 2156 cid meta oid 1fe719b6/pglog_3.7db/0//-1 parent seq 0
2014-06-12 14:46:24.947169 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:24.947181 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:24.948192 7f9d3823f700  5 -- op tracker -- , seq: 747789, time: 2014-06-12 14:46:24.947055, event: header_read, op: osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.948210 7f9d3823f700  5 -- op tracker -- , seq: 747789, time: 2014-06-12 14:46:24.947060, event: throttled, op: osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.948220 7f9d3823f700  5 -- op tracker -- , seq: 747789, time: 2014-06-12 14:46:24.948138, event: all_read, op: osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.948229 7f9d3823f700  5 -- op tracker -- , seq: 747789, time: 0.000000, event: dispatched, op: osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.948239 7f9d3823f700 20 osd.2 173 should_share_map client.4307 172.16.96.220:0/1243935 173
2014-06-12 14:46:24.948270 7f9d3823f700 15 osd.2 173 enqueue_op 0x20e88b00 prio 63 cost 516096 latency 0.001214 osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:24.948320 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.948328 7f9d643d9700 10 osd.2 173 dequeue_op 0x20e88b00 prio 63 cost 516096 latency 0.001273 osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean]
2014-06-12 14:46:24.948352 7f9d643d9700  5 -- op tracker -- , seq: 747789, time: 2014-06-12 14:46:24.948348, event: reached_pg, op: osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.948363 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:24.948379 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] handle_message: 0x20e88b00
2014-06-12 14:46:24.948391 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] do_op osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:24.948416 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] get_object_context: found obc in cache: 0xa45c840
2014-06-12 14:46:24.948427 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] get_object_context: 0xa45c840 c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 rwstate(write n=1 w=0) oi: c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3(173'2679 client.4307.0:146734 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2679) ssc: 0x1550ee00 snapset: 0=[]:[]+head
2014-06-12 14:46:24.948443 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] find_object_context c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 @head oi=c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3(173'2679 client.4307.0:146734 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2679)
2014-06-12 14:46:24.948468 7f9d643d9700  5 -- op tracker -- , seq: 747789, time: 2014-06-12 14:46:24.948463, event: started, op: osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.948476 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] execute_ctx 0x2cec1800
2014-06-12 14:46:24.948492 7f9d643d9700  5 -- op tracker -- , seq: 747789, time: 2014-06-12 14:46:24.948487, event: started, op: osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.948500 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] do_op c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] ov 173'2679 av 173'2680 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:24.948513 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] do_osd_op c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096]
2014-06-12 14:46:24.948523 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:24.948535 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] do_osd_op  write 2285568~516096
2014-06-12 14:46:24.948551 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] do_osd_op_effects on session 0x507d200
2014-06-12 14:46:24.948563 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] make_writeable c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 snapset=0x1550ee38  snapc=0=[]
2014-06-12 14:46:24.948574 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] make_writeable c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:24.948591 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] finish_ctx c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 0x2cec1800 op modify  
2014-06-12 14:46:24.948603 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean]  set mtime to 2014-06-12 08:45:27.326950
2014-06-12 14:46:24.948624 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean]  final snapset 0=[]:[]+head in c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.948644 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean]  zeroing write result code 0
2014-06-12 14:46:24.948656 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] new_repop rep_tid 236254 on osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:24.948672 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] issue_repop rep_tid 236254 o c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.948698 7f9d643d9700  5 -- op tracker -- , seq: 747789, time: 2014-06-12 14:46:24.948693, event: waiting for subops from 21, op: osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.948726 7f9d643d9700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:24.948749 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2679 (0'0,173'2679] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] append_log log((0'0,173'2679], crt=0'0) [173'2680 (173'2679) modify   c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 by client.4307.0:146735 2014-06-12 08:45:27.326950]
2014-06-12 14:46:24.948772 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] add_log_entry 173'2680 (173'2679) modify   c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 by client.4307.0:146735 2014-06-12 08:45:27.326950
2014-06-12 14:46:24.948793 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] append_log  adding 1 keys
2014-06-12 14:46:24.948826 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2680, trimmed: 
2014-06-12 14:46:24.948850 7f9d643d9700  5 queue_transactions existing osr(3.7db 0x49b4380)/0x49b4380
2014-06-12 14:46:24.948855 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:24.948857 7f9d643d9700  5 queue_op 0x10c18f60 seq 2518 osr(3.7db 0x49b4380) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:46:24.948864 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] eval_repop repgather(0x6591c80 173'2680 rep_tid=236254 committed?=0 applied?=0 lock=0 op=osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.948881 7f9d643d9700 10 osd.2 173 dequeue_op 0x20e88b00 finish
2014-06-12 14:46:24.948885 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.948888 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.949625 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e970e0 (2 active)
2014-06-12 14:46:24.952110 7f9d81970700 10 _do_op 0x6c32c00 seq 2517 r = 0, finisher 0x692e1a0 0x38edfb80
2014-06-12 14:46:24.952145 7f9d81970700 10 _finish_op 0x6c32c00 seq 2517 osr(3.7db 0x49b4380)/0x49b4380
2014-06-12 14:46:24.952160 7f9d7c966700  5 _do_op 0x10c18f60 seq 2518 osr(3.7db 0x49b4380)/0x49b4380 start
2014-06-12 14:46:24.952167 7f9d7c966700 10 _do_transaction on 0x2d812780
2014-06-12 14:46:24.952183 7f9d7c966700 15 _omap_setkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.952157 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] op_commit: 236253
2014-06-12 14:46:24.952207 7f9d7715b700  5 -- op tracker -- , seq: 747788, time: 2014-06-12 14:46:24.952196, event: op_commit, op: osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.952338 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.952371 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:24.952427 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:24.952464 7f9d7c966700 15 _omap_rmkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.952470 7f9d7c966700 10 _omap_rmkeys meta/1fe719b6/pglog_3.7db/0//-1 r = 0
2014-06-12 14:46:24.952479 7f9d7c966700 15 _omap_setkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.952502 7f9d7c966700 15 _write 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 2285568~516096
2014-06-12 14:46:24.952693 7f9d7c966700 10 lookup_strip_header done  cid 3.7db_head oid c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.952908 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:24.953734 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] op_applied: 236253
2014-06-12 14:46:24.953769 7f9d7795c700  5 -- op tracker -- , seq: 747788, time: 2014-06-12 14:46:24.953760, event: op_applied, op: osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.953779 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] op_applied on primary on version 173'2679
2014-06-12 14:46:24.953886 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e970e0 (1 active)
2014-06-12 14:46:24.953892 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.955015 7f9d7c966700 10 _generic_write 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 2285568~516096 = 0
2014-06-12 14:46:24.955073 7f9d7c966700 15 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.955093 7f9d7c966700 10 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 = 0
2014-06-12 14:46:24.955100 7f9d7c966700 15 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.955110 7f9d7c966700 10 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 = 0
2014-06-12 14:46:24.955326 7f9d7c966700 20 set_header setting 18475 cid 3.7db_head oid c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 parent seq 0
2014-06-12 14:46:24.955427 7f9d7c966700 20 set_header setting 2156 cid meta oid 1fe719b6/pglog_3.7db/0//-1 parent seq 0
2014-06-12 14:46:24.955448 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:24.955459 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:24.958054 7f9d44101700  5 -- op tracker -- , seq: 747790, time: 2014-06-12 14:46:24.957961, event: header_read, op: osd_sub_op_reply(client.4307.0:146734 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.958069 7f9d44101700  5 -- op tracker -- , seq: 747790, time: 2014-06-12 14:46:24.957964, event: throttled, op: osd_sub_op_reply(client.4307.0:146734 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.958079 7f9d44101700  5 -- op tracker -- , seq: 747790, time: 2014-06-12 14:46:24.957984, event: all_read, op: osd_sub_op_reply(client.4307.0:146734 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.958087 7f9d44101700  5 -- op tracker -- , seq: 747790, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4307.0:146734 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.958095 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4307.0:146734 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:24.958118 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:24.958130 7f9d44101700 15 osd.2 173 enqueue_op 0x160e5100 prio 196 cost 0 latency 0.000168 osd_sub_op_reply(client.4307.0:146734 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:24.958245 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.958252 7f9d633d7700 10 osd.2 173 dequeue_op 0x160e5100 prio 196 cost 0 latency 0.000290 osd_sub_op_reply(client.4307.0:146734 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2 pg pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean]
2014-06-12 14:46:24.958275 7f9d633d7700  5 -- op tracker -- , seq: 747790, time: 2014-06-12 14:46:24.958271, event: reached_pg, op: osd_sub_op_reply(client.4307.0:146734 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.958284 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] handle_message: 0x160e5100
2014-06-12 14:46:24.958299 7f9d633d7700  5 -- op tracker -- , seq: 747790, time: 2014-06-12 14:46:24.958296, event: started, op: osd_sub_op_reply(client.4307.0:146734 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.958305 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] sub_op_modify_reply: tid 236253 op  ack_type 4 from 21
2014-06-12 14:46:24.958331 7f9d633d7700  5 -- op tracker -- , seq: 747788, time: 2014-06-12 14:46:24.958322, event: sub_op_commit_rec, op: osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.958344 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] repop_all_applied: repop tid 236253 all applied 
2014-06-12 14:46:24.958354 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] eval_repop repgather(0x5390900 173'2679 rep_tid=236253 committed?=0 applied?=1 lock=0 op=osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.958370 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean]  sending ack on repgather(0x5390900 173'2679 rep_tid=236253 committed?=0 applied?=1 lock=0 op=osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) 0x27474f00
2014-06-12 14:46:24.958406 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2678 crt=0'0 lcod 173'2677 mlcod 173'2677 active+clean] repop_all_committed: repop tid 236253 all committed 
2014-06-12 14:46:24.958416 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2677 active+clean] eval_repop repgather(0x5390900 173'2679 rep_tid=236253 committed?=1 applied?=1 lock=0 op=osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.958432 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2677 active+clean] log_op_stats osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.022608 lat 0.022636
2014-06-12 14:46:24.958451 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2677 active+clean] publish_stats_to_osd 173:3465
2014-06-12 14:46:24.958476 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2677 active+clean]  sending commit on repgather(0x5390900 173'2679 rep_tid=236253 committed?=1 applied?=1 lock=0 op=osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) 0x282cc500
2014-06-12 14:46:24.958503 7f9d633d7700  5 -- op tracker -- , seq: 747788, time: 2014-06-12 14:46:24.958498, event: commit_sent, op: osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.958512 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2677 active+clean]  requeue_ops 
2014-06-12 14:46:24.958523 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean]  removing repgather(0x5390900 173'2679 rep_tid=236253 committed?=1 applied?=1 op=osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.958537 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean]    q front is repgather(0x5390900 173'2679 rep_tid=236253 committed?=1 applied?=1 op=osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.958551 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] remove_repop repgather(0x5390900 173'2679 rep_tid=236253 committed?=1 applied?=1 op=osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.958563 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean]  requeue_ops 
2014-06-12 14:46:24.958587 7f9d633d7700  5 -- op tracker -- , seq: 747788, time: 2014-06-12 14:46:24.958581, event: done, op: osd_op(client.4307.0:146734 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 1253376~1032192] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.958620 7f9d633d7700 10 osd.2 173 dequeue_op 0x160e5100 finish
2014-06-12 14:46:24.958628 7f9d633d7700  5 -- op tracker -- , seq: 747790, time: 2014-06-12 14:46:24.958624, event: done, op: osd_sub_op_reply(client.4307.0:146734 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.958638 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.958644 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.958715 7f9d3823f700  5 -- op tracker -- , seq: 747791, time: 2014-06-12 14:46:24.957739, event: header_read, op: osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.958731 7f9d3823f700  5 -- op tracker -- , seq: 747791, time: 2014-06-12 14:46:24.957743, event: throttled, op: osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.958740 7f9d3823f700  5 -- op tracker -- , seq: 747791, time: 2014-06-12 14:46:24.958556, event: all_read, op: osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.958749 7f9d3823f700  5 -- op tracker -- , seq: 747791, time: 0.000000, event: dispatched, op: osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.958761 7f9d3823f700 20 osd.2 173 should_share_map client.4307 172.16.96.220:0/1243935 173
2014-06-12 14:46:24.958797 7f9d3823f700 15 osd.2 173 enqueue_op 0x20e89e00 prio 63 cost 516096 latency 0.001057 osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:24.958826 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.958835 7f9d61bd4700 10 osd.2 173 dequeue_op 0x20e89e00 prio 63 cost 516096 latency 0.001096 osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean]
2014-06-12 14:46:24.958860 7f9d61bd4700  5 -- op tracker -- , seq: 747791, time: 2014-06-12 14:46:24.958855, event: reached_pg, op: osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.958870 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:24.958885 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] handle_message: 0x20e89e00
2014-06-12 14:46:24.958895 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] do_op osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:24.958921 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] get_object_context: found obc in cache: 0xa45c840
2014-06-12 14:46:24.958931 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] get_object_context: 0xa45c840 c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 rwstate(write n=1 w=0) oi: c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3(173'2680 client.4307.0:146735 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2680) ssc: 0x1550ee00 snapset: 0=[]:[]+head
2014-06-12 14:46:24.958947 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] find_object_context c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 @head oi=c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3(173'2680 client.4307.0:146735 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2680)
2014-06-12 14:46:24.958975 7f9d61bd4700  5 -- op tracker -- , seq: 747791, time: 2014-06-12 14:46:24.958969, event: started, op: osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.958982 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] execute_ctx 0x27bc1200
2014-06-12 14:46:24.959000 7f9d61bd4700  5 -- op tracker -- , seq: 747791, time: 2014-06-12 14:46:24.958994, event: started, op: osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.959006 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] do_op c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] ov 173'2680 av 173'2681 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:24.959020 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] do_osd_op c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096]
2014-06-12 14:46:24.959031 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:24.959045 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] do_osd_op  write 2801664~516096
2014-06-12 14:46:24.959060 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] do_osd_op_effects on session 0x507d200
2014-06-12 14:46:24.959071 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] make_writeable c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 snapset=0x1550ee38  snapc=0=[]
2014-06-12 14:46:24.959084 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] make_writeable c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:24.959094 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] finish_ctx c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 0x27bc1200 op modify  
2014-06-12 14:46:24.959107 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean]  set mtime to 2014-06-12 08:45:27.336337
2014-06-12 14:46:24.959128 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean]  final snapset 0=[]:[]+head in c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.959147 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean]  zeroing write result code 0
2014-06-12 14:46:24.959157 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] new_repop rep_tid 236255 on osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:24.959173 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] issue_repop rep_tid 236255 o c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.959199 7f9d61bd4700  5 -- op tracker -- , seq: 747791, time: 2014-06-12 14:46:24.959194, event: waiting for subops from 21, op: osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.959223 7f9d61bd4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:24.959246 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2680 (0'0,173'2680] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] append_log log((0'0,173'2680], crt=0'0) [173'2681 (173'2680) modify   c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 by client.4307.0:146736 2014-06-12 08:45:27.336337]
2014-06-12 14:46:24.959269 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] add_log_entry 173'2681 (173'2680) modify   c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 by client.4307.0:146736 2014-06-12 08:45:27.336337
2014-06-12 14:46:24.959290 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] append_log  adding 1 keys
2014-06-12 14:46:24.959323 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2681, trimmed: 
2014-06-12 14:46:24.959349 7f9d61bd4700  5 queue_transactions existing osr(3.7db 0x49b4380)/0x49b4380
2014-06-12 14:46:24.959354 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:24.959356 7f9d61bd4700  5 queue_op 0x7c66900 seq 2519 osr(3.7db 0x49b4380) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:24.959366 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] eval_repop repgather(0x6e52d80 173'2681 rep_tid=236255 committed?=0 applied?=0 lock=0 op=osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.959385 7f9d61bd4700 10 osd.2 173 dequeue_op 0x20e89e00 finish
2014-06-12 14:46:24.959389 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.959392 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.959406 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e970e0 (2 active)
2014-06-12 14:46:24.959445 7f9d7c966700 10 _do_op 0x10c18f60 seq 2518 r = 0, finisher 0x8053480 0x19b2d540
2014-06-12 14:46:24.959920 7f9d7c966700 10 _finish_op 0x10c18f60 seq 2518 osr(3.7db 0x49b4380)/0x49b4380
2014-06-12 14:46:24.959924 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] op_commit: 236254
2014-06-12 14:46:24.959935 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e970e0 (1 active)
2014-06-12 14:46:24.959946 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.959950 7f9d7715b700  5 -- op tracker -- , seq: 747789, time: 2014-06-12 14:46:24.959942, event: op_commit, op: osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.961321 7f9d7895e700  5 _do_op 0x7c66900 seq 2519 osr(3.7db 0x49b4380)/0x49b4380 start
2014-06-12 14:46:24.961361 7f9d7895e700 10 _do_transaction on 0x5112f00
2014-06-12 14:46:24.961374 7f9d7895e700 15 _omap_setkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.961522 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.961556 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:24.961606 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:24.961645 7f9d7895e700 15 _omap_rmkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.961652 7f9d7895e700 10 _omap_rmkeys meta/1fe719b6/pglog_3.7db/0//-1 r = 0
2014-06-12 14:46:24.961659 7f9d7895e700 15 _omap_setkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.961676 7f9d7895e700 15 _write 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 2801664~516096
2014-06-12 14:46:24.961689 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] op_applied: 236254
2014-06-12 14:46:24.961842 7f9d7895e700 10 lookup_strip_header done  cid 3.7db_head oid c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.961837 7f9d7795c700  5 -- op tracker -- , seq: 747789, time: 2014-06-12 14:46:24.961760, event: op_applied, op: osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.961899 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] op_applied on primary on version 173'2680
2014-06-12 14:46:24.961956 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:24.964401 7f9d3823f700  5 -- op tracker -- , seq: 747792, time: 2014-06-12 14:46:24.962581, event: header_read, op: osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.964420 7f9d3823f700  5 -- op tracker -- , seq: 747792, time: 2014-06-12 14:46:24.962588, event: throttled, op: osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.964429 7f9d3823f700  5 -- op tracker -- , seq: 747792, time: 2014-06-12 14:46:24.964325, event: all_read, op: osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.964429 7f9d7895e700 10 _generic_write 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 2801664~516096 = 0
2014-06-12 14:46:24.964437 7f9d3823f700  5 -- op tracker -- , seq: 747792, time: 0.000000, event: dispatched, op: osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.964448 7f9d3823f700 20 osd.2 173 should_share_map client.4307 172.16.96.220:0/1243935 173
2014-06-12 14:46:24.964476 7f9d7895e700 15 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.964477 7f9d3823f700 15 osd.2 173 enqueue_op 0x20e88a00 prio 63 cost 876544 latency 0.001896 osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:24.964493 7f9d7895e700 10 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 = 0
2014-06-12 14:46:24.964500 7f9d7895e700 15 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.964507 7f9d7895e700 10 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 = 0
2014-06-12 14:46:24.964514 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.964521 7f9d673df700 10 osd.2 173 dequeue_op 0x20e88a00 prio 63 cost 876544 latency 0.001940 osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean]
2014-06-12 14:46:24.964544 7f9d673df700  5 -- op tracker -- , seq: 747792, time: 2014-06-12 14:46:24.964540, event: reached_pg, op: osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.964554 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:24.964569 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] handle_message: 0x20e88a00
2014-06-12 14:46:24.964579 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] do_op osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:24.964604 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] get_object_context: found obc in cache: 0xa45c840
2014-06-12 14:46:24.964615 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] get_object_context: 0xa45c840 c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 rwstate(write n=2 w=0) oi: c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3(173'2681 client.4307.0:146736 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2681) ssc: 0x1550ee00 snapset: 0=[]:[]+head
2014-06-12 14:46:24.964631 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] find_object_context c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 @head oi=c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3(173'2681 client.4307.0:146736 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2681)
2014-06-12 14:46:24.964657 7f9d673df700  5 -- op tracker -- , seq: 747792, time: 2014-06-12 14:46:24.964651, event: started, op: osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.964688 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] execute_ctx 0x55a4400
2014-06-12 14:46:24.964707 7f9d673df700  5 -- op tracker -- , seq: 747792, time: 2014-06-12 14:46:24.964702, event: started, op: osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.964715 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] do_op c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] ov 173'2681 av 173'2682 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:24.964729 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] do_osd_op c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544]
2014-06-12 14:46:24.964738 7f9d7895e700 20 set_header setting 18475 cid 3.7db_head oid c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 parent seq 0
2014-06-12 14:46:24.964740 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:24.964759 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] do_osd_op  write 3317760~876544
2014-06-12 14:46:24.964775 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] do_osd_op_effects on session 0x507d200
2014-06-12 14:46:24.964787 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] make_writeable c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 snapset=0x1550ee38  snapc=0=[]
2014-06-12 14:46:24.964798 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] make_writeable c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:24.964808 7f9d7895e700 20 set_header setting 2156 cid meta oid 1fe719b6/pglog_3.7db/0//-1 parent seq 0
2014-06-12 14:46:24.964808 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] finish_ctx c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 0x55a4400 op modify  
2014-06-12 14:46:24.964828 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:24.964821 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean]  set mtime to 2014-06-12 08:45:27.341459
2014-06-12 14:46:24.964837 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:24.964845 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean]  final snapset 0=[]:[]+head in c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.965703 7f9d44101700  5 -- op tracker -- , seq: 747793, time: 2014-06-12 14:46:24.965657, event: header_read, op: osd_sub_op_reply(client.4307.0:146735 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.965712 7f9d44101700  5 -- op tracker -- , seq: 747793, time: 2014-06-12 14:46:24.965659, event: throttled, op: osd_sub_op_reply(client.4307.0:146735 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.965718 7f9d44101700  5 -- op tracker -- , seq: 747793, time: 2014-06-12 14:46:24.965669, event: all_read, op: osd_sub_op_reply(client.4307.0:146735 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.965731 7f9d44101700  5 -- op tracker -- , seq: 747793, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4307.0:146735 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.965735 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4307.0:146735 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:24.965740 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:24.965749 7f9d44101700 15 osd.2 173 enqueue_op 0x8dbd500 prio 196 cost 0 latency 0.000091 osd_sub_op_reply(client.4307.0:146735 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:24.964871 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean]  zeroing write result code 0
2014-06-12 14:46:24.965851 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] new_repop rep_tid 236256 on osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:24.965871 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] issue_repop rep_tid 236256 o c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.965897 7f9d673df700  5 -- op tracker -- , seq: 747792, time: 2014-06-12 14:46:24.965892, event: waiting for subops from 21, op: osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.965927 7f9d673df700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:24.965948 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2681 (0'0,173'2681] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] append_log log((0'0,173'2681], crt=0'0) [173'2682 (173'2681) modify   c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 by client.4307.0:146737 2014-06-12 08:45:27.341459]
2014-06-12 14:46:24.965971 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] add_log_entry 173'2682 (173'2681) modify   c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 by client.4307.0:146737 2014-06-12 08:45:27.341459
2014-06-12 14:46:24.965989 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] append_log  adding 1 keys
2014-06-12 14:46:24.966018 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2682, trimmed: 
2014-06-12 14:46:24.966044 7f9d673df700  5 queue_transactions existing osr(3.7db 0x49b4380)/0x49b4380
2014-06-12 14:46:24.966048 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:24.966050 7f9d673df700  5 queue_op 0x6db18c0 seq 2520 osr(3.7db 0x49b4380) 878837 bytes   (queue has 2 ops and 1397226 bytes)
2014-06-12 14:46:24.966059 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] eval_repop repgather(0x10450780 173'2682 rep_tid=236256 committed?=0 applied?=0 lock=0 op=osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.966078 7f9d673df700 10 osd.2 173 dequeue_op 0x20e88a00 finish
2014-06-12 14:46:24.966083 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.966090 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.966123 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.966130 7f9d67be0700 10 osd.2 173 dequeue_op 0x8dbd500 prio 196 cost 0 latency 0.000473 osd_sub_op_reply(client.4307.0:146735 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2 pg pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean]
2014-06-12 14:46:24.966149 7f9d67be0700  5 -- op tracker -- , seq: 747793, time: 2014-06-12 14:46:24.966146, event: reached_pg, op: osd_sub_op_reply(client.4307.0:146735 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.966157 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] handle_message: 0x8dbd500
2014-06-12 14:46:24.966171 7f9d67be0700  5 -- op tracker -- , seq: 747793, time: 2014-06-12 14:46:24.966168, event: started, op: osd_sub_op_reply(client.4307.0:146735 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.966177 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] sub_op_modify_reply: tid 236254 op  ack_type 4 from 21
2014-06-12 14:46:24.966195 7f9d67be0700  5 -- op tracker -- , seq: 747789, time: 2014-06-12 14:46:24.966188, event: sub_op_commit_rec, op: osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.966201 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] repop_all_applied: repop tid 236254 all applied 
2014-06-12 14:46:24.966211 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] eval_repop repgather(0x6591c80 173'2680 rep_tid=236254 committed?=0 applied?=1 lock=0 op=osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.966225 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean]  sending ack on repgather(0x6591c80 173'2680 rep_tid=236254 committed?=0 applied?=1 lock=0 op=osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) 0x353e8280
2014-06-12 14:46:24.966253 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2679 crt=0'0 lcod 173'2678 mlcod 173'2678 active+clean] repop_all_committed: repop tid 236254 all committed 
2014-06-12 14:46:24.966268 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2678 active+clean] eval_repop repgather(0x6591c80 173'2680 rep_tid=236254 committed?=1 applied?=1 lock=0 op=osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.966283 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2678 active+clean] log_op_stats osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.019197 lat 0.019227
2014-06-12 14:46:24.966299 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2678 active+clean] publish_stats_to_osd 173:3466
2014-06-12 14:46:24.966311 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2678 active+clean]  sending commit on repgather(0x6591c80 173'2680 rep_tid=236254 committed?=1 applied?=1 lock=0 op=osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) 0x29de6000
2014-06-12 14:46:24.966336 7f9d67be0700  5 -- op tracker -- , seq: 747789, time: 2014-06-12 14:46:24.966332, event: commit_sent, op: osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.966343 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2678 active+clean]  requeue_ops 
2014-06-12 14:46:24.966353 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean]  removing repgather(0x6591c80 173'2680 rep_tid=236254 committed?=1 applied?=1 op=osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.966366 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean]    q front is repgather(0x6591c80 173'2680 rep_tid=236254 committed?=1 applied?=1 op=osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.966385 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean] remove_repop repgather(0x6591c80 173'2680 rep_tid=236254 committed?=1 applied?=1 op=osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.966398 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean]  requeue_ops 
2014-06-12 14:46:24.966417 7f9d67be0700  5 -- op tracker -- , seq: 747789, time: 2014-06-12 14:46:24.966412, event: done, op: osd_op(client.4307.0:146735 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.966445 7f9d67be0700 10 osd.2 173 dequeue_op 0x8dbd500 finish
2014-06-12 14:46:24.966452 7f9d67be0700  5 -- op tracker -- , seq: 747793, time: 2014-06-12 14:46:24.966449, event: done, op: osd_sub_op_reply(client.4307.0:146735 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.966465 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.966470 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.966578 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e970e0 (2 active)
2014-06-12 14:46:24.967450 7f9d7895e700 10 _do_op 0x7c66900 seq 2519 r = 0, finisher 0x65e4840 0x1556f5c0
2014-06-12 14:46:24.967467 7f9d7895e700 10 _finish_op 0x7c66900 seq 2519 osr(3.7db 0x49b4380)/0x49b4380
2014-06-12 14:46:24.967475 7f9d7a161700  5 _do_op 0x6db18c0 seq 2520 osr(3.7db 0x49b4380)/0x49b4380 start
2014-06-12 14:46:24.967478 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e970e0 (1 active)
2014-06-12 14:46:24.967482 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.967482 7f9d7a161700 10 _do_transaction on 0x5312300
2014-06-12 14:46:24.967471 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean] op_commit: 236255
2014-06-12 14:46:24.967489 7f9d7a161700 15 _omap_setkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.967499 7f9d7715b700  5 -- op tracker -- , seq: 747791, time: 2014-06-12 14:46:24.967491, event: op_commit, op: osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.967518 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean] op_applied: 236255
2014-06-12 14:46:24.967543 7f9d7795c700  5 -- op tracker -- , seq: 747791, time: 2014-06-12 14:46:24.967537, event: op_applied, op: osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.967553 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean] op_applied on primary on version 173'2681
2014-06-12 14:46:24.967597 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.967621 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:24.967649 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:24.967686 7f9d7a161700 15 _omap_rmkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.967690 7f9d7a161700 10 _omap_rmkeys meta/1fe719b6/pglog_3.7db/0//-1 r = 0
2014-06-12 14:46:24.967694 7f9d7a161700 15 _omap_setkeys meta/1fe719b6/pglog_3.7db/0//-1
2014-06-12 14:46:24.967787 7f9d7a161700 15 _write 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 3317760~876544
2014-06-12 14:46:24.967885 7f9d7a161700 10 lookup_strip_header done  cid 3.7db_head oid c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.968023 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:24.968793 7f9d44101700  5 -- op tracker -- , seq: 747794, time: 2014-06-12 14:46:24.968758, event: header_read, op: osd_sub_op_reply(client.4307.0:146736 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.968805 7f9d44101700  5 -- op tracker -- , seq: 747794, time: 2014-06-12 14:46:24.968759, event: throttled, op: osd_sub_op_reply(client.4307.0:146736 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.968813 7f9d44101700  5 -- op tracker -- , seq: 747794, time: 2014-06-12 14:46:24.968766, event: all_read, op: osd_sub_op_reply(client.4307.0:146736 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.968821 7f9d44101700  5 -- op tracker -- , seq: 747794, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4307.0:146736 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.968825 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4307.0:146736 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:24.968830 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:24.968838 7f9d44101700 15 osd.2 173 enqueue_op 0x9face00 prio 196 cost 0 latency 0.000079 osd_sub_op_reply(client.4307.0:146736 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:24.968866 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.968872 7f9d663dd700 10 osd.2 173 dequeue_op 0x9face00 prio 196 cost 0 latency 0.000114 osd_sub_op_reply(client.4307.0:146736 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2 pg pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean]
2014-06-12 14:46:24.968892 7f9d663dd700  5 -- op tracker -- , seq: 747794, time: 2014-06-12 14:46:24.968889, event: reached_pg, op: osd_sub_op_reply(client.4307.0:146736 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.968899 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean] handle_message: 0x9face00
2014-06-12 14:46:24.968916 7f9d663dd700  5 -- op tracker -- , seq: 747794, time: 2014-06-12 14:46:24.968912, event: started, op: osd_sub_op_reply(client.4307.0:146736 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.968925 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean] sub_op_modify_reply: tid 236255 op  ack_type 4 from 21
2014-06-12 14:46:24.968945 7f9d663dd700  5 -- op tracker -- , seq: 747791, time: 2014-06-12 14:46:24.968938, event: sub_op_commit_rec, op: osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.968961 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean] repop_all_applied: repop tid 236255 all applied 
2014-06-12 14:46:24.968970 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean] eval_repop repgather(0x6e52d80 173'2681 rep_tid=236255 committed?=0 applied?=1 lock=0 op=osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.968981 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean]  sending ack on repgather(0x6e52d80 173'2681 rep_tid=236255 committed?=0 applied?=1 lock=0 op=osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) 0x6487680
2014-06-12 14:46:24.969002 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2680 crt=0'0 lcod 173'2679 mlcod 173'2679 active+clean] repop_all_committed: repop tid 236255 all committed 
2014-06-12 14:46:24.969012 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2679 active+clean] eval_repop repgather(0x6e52d80 173'2681 rep_tid=236255 committed?=1 applied?=1 lock=0 op=osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.969028 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2679 active+clean] log_op_stats osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.011262 lat 0.011288
2014-06-12 14:46:24.969044 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2679 active+clean] publish_stats_to_osd 173:3467
2014-06-12 14:46:24.969053 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2679 active+clean]  sending commit on repgather(0x6e52d80 173'2681 rep_tid=236255 committed?=1 applied?=1 lock=0 op=osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) 0x15cd6500
2014-06-12 14:46:24.969076 7f9d663dd700  5 -- op tracker -- , seq: 747791, time: 2014-06-12 14:46:24.969070, event: commit_sent, op: osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.969085 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2679 active+clean]  requeue_ops 
2014-06-12 14:46:24.969094 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2680 active+clean]  removing repgather(0x6e52d80 173'2681 rep_tid=236255 committed?=1 applied?=1 op=osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.969103 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2680 active+clean]    q front is repgather(0x6e52d80 173'2681 rep_tid=236255 committed?=1 applied?=1 op=osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.969113 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2680 active+clean] remove_repop repgather(0x6e52d80 173'2681 rep_tid=236255 committed?=1 applied?=1 op=osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.969123 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2680 active+clean]  requeue_ops 
2014-06-12 14:46:24.969135 7f9d663dd700  5 -- op tracker -- , seq: 747791, time: 2014-06-12 14:46:24.969132, event: done, op: osd_op(client.4307.0:146736 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~516096] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.969149 7f9d663dd700 10 osd.2 173 dequeue_op 0x9face00 finish
2014-06-12 14:46:24.969154 7f9d663dd700  5 -- op tracker -- , seq: 747794, time: 2014-06-12 14:46:24.969152, event: done, op: osd_sub_op_reply(client.4307.0:146736 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.969160 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.969162 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.975044 7f9d7a161700 10 _generic_write 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 3317760~876544 = 0
2014-06-12 14:46:24.975141 7f9d7a161700 15 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.975159 7f9d7a161700 10 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 = 0
2014-06-12 14:46:24.975166 7f9d7a161700 15 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3
2014-06-12 14:46:24.975175 7f9d7a161700 10 _setattrs 3.7db_head/c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 = 0
2014-06-12 14:46:24.975709 7f9d7a161700 20 set_header setting 18475 cid 3.7db_head oid c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 parent seq 0
2014-06-12 14:46:24.975978 7f9d7a161700 20 set_header setting 2156 cid meta oid 1fe719b6/pglog_3.7db/0//-1 parent seq 0
2014-06-12 14:46:24.975996 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:24.976005 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:24.977076 7f9d44101700  5 -- op tracker -- , seq: 747795, time: 2014-06-12 14:46:24.977013, event: header_read, op: osd_sub_op_reply(client.4307.0:146737 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.977088 7f9d44101700  5 -- op tracker -- , seq: 747795, time: 2014-06-12 14:46:24.977017, event: throttled, op: osd_sub_op_reply(client.4307.0:146737 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.977097 7f9d44101700  5 -- op tracker -- , seq: 747795, time: 2014-06-12 14:46:24.977040, event: all_read, op: osd_sub_op_reply(client.4307.0:146737 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.977104 7f9d44101700  5 -- op tracker -- , seq: 747795, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4307.0:146737 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.977111 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4307.0:146737 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:24.977118 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:24.977126 7f9d44101700 15 osd.2 173 enqueue_op 0x9fac900 prio 196 cost 0 latency 0.000113 osd_sub_op_reply(client.4307.0:146737 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:24.977156 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:24.977163 7f9d693e3700 10 osd.2 173 dequeue_op 0x9fac900 prio 196 cost 0 latency 0.000150 osd_sub_op_reply(client.4307.0:146737 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0) v2 pg pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2680 active+clean]
2014-06-12 14:46:24.977184 7f9d693e3700  5 -- op tracker -- , seq: 747795, time: 2014-06-12 14:46:24.977180, event: reached_pg, op: osd_sub_op_reply(client.4307.0:146737 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.977192 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2680 active+clean] handle_message: 0x9fac900
2014-06-12 14:46:24.977209 7f9d693e3700  5 -- op tracker -- , seq: 747795, time: 2014-06-12 14:46:24.977205, event: started, op: osd_sub_op_reply(client.4307.0:146737 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.977215 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2680 active+clean] sub_op_modify_reply: tid 236256 op  ack_type 4 from 21
2014-06-12 14:46:24.977236 7f9d693e3700  5 -- op tracker -- , seq: 747792, time: 2014-06-12 14:46:24.977230, event: sub_op_commit_rec, op: osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.977246 7f9d693e3700 10 osd.2 173 dequeue_op 0x9fac900 finish
2014-06-12 14:46:24.977252 7f9d693e3700  5 -- op tracker -- , seq: 747795, time: 2014-06-12 14:46:24.977249, event: done, op: osd_sub_op_reply(client.4307.0:146737 3.7db c3aabfdb/rbd_data.10aa6b8b4567.0000000000000ffc/head//3 [] ondisk, result = 0)
2014-06-12 14:46:24.977271 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:24.977273 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:24.985138 7f9d7a161700 10 _do_op 0x6db18c0 seq 2520 r = 0, finisher 0x66b5fe0 0x38dd3e40
2014-06-12 14:46:24.985172 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2680 active+clean] op_commit: 236256
2014-06-12 14:46:24.985211 7f9d7715b700  5 -- op tracker -- , seq: 747792, time: 2014-06-12 14:46:24.985199, event: op_commit, op: osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.985235 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2681 crt=0'0 lcod 173'2680 mlcod 173'2680 active+clean] repop_all_committed: repop tid 236256 all committed 
2014-06-12 14:46:24.985266 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2680 active+clean] eval_repop repgather(0x10450780 173'2682 rep_tid=236256 committed?=1 applied?=0 lock=0 op=osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.985288 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2680 active+clean] log_op_stats osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 inb 876823 outb 0 rlat 0.000000 lat 0.022705
2014-06-12 14:46:24.985311 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2680 active+clean] publish_stats_to_osd 173:3468
2014-06-12 14:46:24.985322 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2680 active+clean]  sending commit on repgather(0x10450780 173'2682 rep_tid=236256 committed?=1 applied?=0 lock=0 op=osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) 0x2f4d7180
2014-06-12 14:46:24.985375 7f9d7715b700  5 -- op tracker -- , seq: 747792, time: 2014-06-12 14:46:24.985368, event: commit_sent, op: osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.985473 7f9d7a161700 10 _finish_op 0x6db18c0 seq 2520 osr(3.7db 0x49b4380)/0x49b4380
2014-06-12 14:46:24.985885 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e970e0 (0 active)
2014-06-12 14:46:24.985909 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:24.985890 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2680 active+clean] op_applied: 236256
2014-06-12 14:46:24.985924 7f9d7795c700  5 -- op tracker -- , seq: 747792, time: 2014-06-12 14:46:24.985916, event: op_applied, op: osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:24.985933 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2680 active+clean] op_applied on primary on version 173'2682
2014-06-12 14:46:24.985941 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2680 active+clean] repop_all_applied: repop tid 236256 all applied 
2014-06-12 14:46:24.985947 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2680 active+clean] eval_repop repgather(0x10450780 173'2682 rep_tid=236256 committed?=1 applied?=1 lock=0 op=osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:24.985958 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2680 active+clean] log_op_stats osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4 inb 876823 outb 0 rlat 0.000000 lat 0.023376
2014-06-12 14:46:24.985973 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2680 active+clean] publish_stats_to_osd 173:3469
2014-06-12 14:46:24.985981 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2680 active+clean]  requeue_ops 
2014-06-12 14:46:24.985987 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2681 active+clean]  removing repgather(0x10450780 173'2682 rep_tid=236256 committed?=1 applied?=1 op=osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.985996 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2681 active+clean]    q front is repgather(0x10450780 173'2682 rep_tid=236256 committed?=1 applied?=1 op=osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.986004 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2681 active+clean] remove_repop repgather(0x10450780 173'2682 rep_tid=236256 committed?=1 applied?=1 op=osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:24.986012 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2681 active+clean]  requeue_ops 
2014-06-12 14:46:24.986033 7f9d7795c700  5 -- op tracker -- , seq: 747792, time: 2014-06-12 14:46:24.986030, event: done, op: osd_op(client.4307.0:146737 rbd_data.10aa6b8b4567.0000000000000ffc [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~876544] 3.c3aabfdb ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.042560 7f9d50844700  5 -- op tracker -- , seq: 747796, time: 2014-06-12 14:46:25.038316, event: header_read, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.042585 7f9d50844700  5 -- op tracker -- , seq: 747796, time: 2014-06-12 14:46:25.038320, event: throttled, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.042594 7f9d50844700  5 -- op tracker -- , seq: 747796, time: 2014-06-12 14:46:25.042507, event: all_read, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.042605 7f9d50844700  5 -- op tracker -- , seq: 747796, time: 0.000000, event: dispatched, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.042615 7f9d50844700 20 osd.2 173 should_share_map client.4294 172.16.96.220:0/1242079 173
2014-06-12 14:46:25.042644 7f9d50844700 15 osd.2 173 enqueue_op 0x24c9dc00 prio 63 cost 1720320 latency 0.004328 osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:25.042705 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:25.042717 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x24c9dc00 prio 63 cost 1720320 latency 0.004401 osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]
2014-06-12 14:46:25.042742 7f9d6b3e7700  5 -- op tracker -- , seq: 747796, time: 2014-06-12 14:46:25.042738, event: reached_pg, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.042753 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:25.042766 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] handle_message: 0x24c9dc00
2014-06-12 14:46:25.042775 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] do_op osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:25.042795 7f9d6b3e7700 15 getattr 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 '_'
2014-06-12 14:46:25.068886 7f9d6b3e7700 10 getattr 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 '_' = 0
2014-06-12 14:46:25.068913 7f9d6b3e7700 15 getattr 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 'snapset'
2014-06-12 14:46:25.069045 7f9d6b3e7700 10 getattr 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 'snapset' = 0
2014-06-12 14:46:25.069054 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] populate_obc_watchers e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.069065 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.069072 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] get_object_context: creating obc from disk: 0x6ac0580
2014-06-12 14:46:25.069078 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] get_object_context: 0x6ac0580 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 rwstate(none n=0 w=0) oi: e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3(173'735 client.4294.0:40033 wrlock_by=unknown.0.0:0 dirty s 4194304 uv735) ssc: 0x24d4b880 snapset: 0=[]:[]+head
2014-06-12 14:46:25.069088 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] find_object_context e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 @head oi=e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3(173'735 client.4294.0:40033 wrlock_by=unknown.0.0:0 dirty s 4194304 uv735)
2014-06-12 14:46:25.069107 7f9d6b3e7700  5 -- op tracker -- , seq: 747796, time: 2014-06-12 14:46:25.069102, event: started, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.069118 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] execute_ctx 0x5d21e00
2014-06-12 14:46:25.069128 7f9d6b3e7700  5 -- op tracker -- , seq: 747796, time: 2014-06-12 14:46:25.069125, event: started, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.069134 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] do_op e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] ov 173'735 av 173'2848 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:25.069148 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] do_osd_op e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320]
2014-06-12 14:46:25.069155 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:25.069163 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] do_osd_op  write 0~1720320
2014-06-12 14:46:25.069175 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] do_osd_op_effects on session 0x50a3000
2014-06-12 14:46:25.069183 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] make_writeable e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 snapset=0x24d4b8b8  snapc=0=[]
2014-06-12 14:46:25.069191 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] make_writeable e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:25.069197 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] finish_ctx e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 0x5d21e00 op modify  
2014-06-12 14:46:25.069205 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]  set mtime to 2014-06-12 08:45:27.415862
2014-06-12 14:46:25.069219 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]  final snapset 0=[]:[]+head in e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.069232 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]  zeroing write result code 0
2014-06-12 14:46:25.069240 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] new_repop rep_tid 236257 on osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:25.069251 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] issue_repop rep_tid 236257 o e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.069269 7f9d6b3e7700  5 -- op tracker -- , seq: 747796, time: 2014-06-12 14:46:25.069266, event: waiting for subops from 23, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.069285 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:25.069311 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2847 (0'0,173'2847] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] append_log log((0'0,173'2847], crt=0'0) [173'2848 (173'735) modify   e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 by client.4294.0:144029 2014-06-12 08:45:27.415862]
2014-06-12 14:46:25.069334 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] add_log_entry 173'2848 (173'735) modify   e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 by client.4294.0:144029 2014-06-12 08:45:27.415862
2014-06-12 14:46:25.069354 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] append_log  adding 1 keys
2014-06-12 14:46:25.069382 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2848, trimmed: 
2014-06-12 14:46:25.069400 7f9d6b3e7700  5 queue_transactions existing osr(3.605 0x276dd10)/0x276dd10
2014-06-12 14:46:25.069405 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:25.069406 7f9d6b3e7700  5 queue_op 0x5cc56e0 seq 2638 osr(3.605 0x276dd10) 1722613 bytes   (queue has 1 ops and 1722613 bytes)
2014-06-12 14:46:25.069415 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5420a20 (1 active)
2014-06-12 14:46:25.069422 7f9d7a962700  5 _do_op 0x5cc56e0 seq 2638 osr(3.605 0x276dd10)/0x276dd10 start
2014-06-12 14:46:25.069426 7f9d7a962700 10 _do_transaction on 0x32ff4b40
2014-06-12 14:46:25.069412 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] eval_repop repgather(0x51f6480 173'2848 rep_tid=236257 committed?=0 applied?=0 lock=0 op=osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:25.069429 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x24c9dc00 finish
2014-06-12 14:46:25.069431 7f9d7a962700 15 _omap_setkeys meta/1fe11caa/pglog_3.605/0//-1
2014-06-12 14:46:25.069432 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:25.069436 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:25.069642 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe11caa/pglog_3.605/0//-1
2014-06-12 14:46:25.069667 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:25.069713 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:25.069735 7f9d7a962700 15 _omap_rmkeys meta/1fe11caa/pglog_3.605/0//-1
2014-06-12 14:46:25.069742 7f9d7a962700 10 _omap_rmkeys meta/1fe11caa/pglog_3.605/0//-1 r = 0
2014-06-12 14:46:25.069748 7f9d7a962700 15 _omap_setkeys meta/1fe11caa/pglog_3.605/0//-1
2014-06-12 14:46:25.069764 7f9d7a962700 15 _write 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 0~1720320
2014-06-12 14:46:25.070055 7f9d7a962700 10 lookup_strip_header done  cid 3.605_head oid e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.070209 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:25.077386 7f9d7a962700 10 _generic_write 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 0~1720320 = 0
2014-06-12 14:46:25.077513 7f9d7a962700 15 _setattrs 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.077528 7f9d7a962700 10 _setattrs 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 = 0
2014-06-12 14:46:25.077533 7f9d7a962700 15 _setattrs 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.077538 7f9d7a962700 10 _setattrs 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 = 0
2014-06-12 14:46:25.077891 7f9d7a962700 20 set_header setting 17751 cid 3.605_head oid e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 parent seq 0
2014-06-12 14:46:25.078015 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:25.078027 7f9d7a962700 20 set_header setting 2104 cid meta oid 1fe11caa/pglog_3.605/0//-1 parent seq 0
2014-06-12 14:46:25.078041 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:25.085764 7f9d7a962700 10 _do_op 0x5cc56e0 seq 2638 r = 0, finisher 0x5db7540 0x328d2f80
2014-06-12 14:46:25.085788 7f9d7a962700 10 _finish_op 0x5cc56e0 seq 2638 osr(3.605 0x276dd10)/0x276dd10
2014-06-12 14:46:25.085797 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5420a20 (0 active)
2014-06-12 14:46:25.085804 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:25.085795 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] op_commit: 236257
2014-06-12 14:46:25.085825 7f9d7715b700  5 -- op tracker -- , seq: 747796, time: 2014-06-12 14:46:25.085817, event: op_commit, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.085879 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] op_applied: 236257
2014-06-12 14:46:25.085907 7f9d7795c700  5 -- op tracker -- , seq: 747796, time: 2014-06-12 14:46:25.085900, event: op_applied, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.085917 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] op_applied on primary on version 173'2848
2014-06-12 14:46:25.089903 7f9d57543700  5 -- op tracker -- , seq: 747797, time: 2014-06-12 14:46:25.089848, event: header_read, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ack, result = 0)
2014-06-12 14:46:25.089916 7f9d57543700  5 -- op tracker -- , seq: 747797, time: 2014-06-12 14:46:25.089850, event: throttled, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ack, result = 0)
2014-06-12 14:46:25.089924 7f9d57543700  5 -- op tracker -- , seq: 747797, time: 2014-06-12 14:46:25.089868, event: all_read, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ack, result = 0)
2014-06-12 14:46:25.089931 7f9d57543700  5 -- op tracker -- , seq: 747797, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ack, result = 0)
2014-06-12 14:46:25.089935 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:25.089941 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:25.089948 7f9d57543700 15 osd.2 173 enqueue_op 0x9fac500 prio 196 cost 0 latency 0.000100 osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ack, result = 0) v2
2014-06-12 14:46:25.090002 7f9d57543700  5 -- op tracker -- , seq: 747798, time: 2014-06-12 14:46:25.089967, event: header_read, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.090012 7f9d57543700  5 -- op tracker -- , seq: 747798, time: 2014-06-12 14:46:25.089967, event: throttled, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.090019 7f9d57543700  5 -- op tracker -- , seq: 747798, time: 2014-06-12 14:46:25.089984, event: all_read, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.090027 7f9d57543700  5 -- op tracker -- , seq: 747798, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.090031 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:25.090035 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:25.090039 7f9d57543700 15 osd.2 173 enqueue_op 0x2d17a500 prio 196 cost 0 latency 0.000072 osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:25.090062 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:25.090069 7f9d62bd6700 10 osd.2 173 dequeue_op 0x9fac500 prio 196 cost 0 latency 0.000221 osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ack, result = 0) v2 pg pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]
2014-06-12 14:46:25.090088 7f9d62bd6700  5 -- op tracker -- , seq: 747797, time: 2014-06-12 14:46:25.090086, event: reached_pg, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ack, result = 0)
2014-06-12 14:46:25.090102 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] handle_message: 0x9fac500
2014-06-12 14:46:25.090119 7f9d62bd6700  5 -- op tracker -- , seq: 747797, time: 2014-06-12 14:46:25.090116, event: started, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ack, result = 0)
2014-06-12 14:46:25.090126 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] sub_op_modify_reply: tid 236257 op  ack_type 1 from 23
2014-06-12 14:46:25.090147 7f9d62bd6700  5 -- op tracker -- , seq: 747796, time: 2014-06-12 14:46:25.090139, event: sub_op_applied_rec, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.090155 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] repop_all_applied: repop tid 236257 all applied 
2014-06-12 14:46:25.090168 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] eval_repop repgather(0x51f6480 173'2848 rep_tid=236257 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:25.090184 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]  sending ack on repgather(0x51f6480 173'2848 rep_tid=236257 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4) 0x2f89cc80
2014-06-12 14:46:25.090216 7f9d62bd6700 10 osd.2 173 dequeue_op 0x9fac500 finish
2014-06-12 14:46:25.090236 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:25.090240 7f9d683e1700 10 osd.2 173 dequeue_op 0x2d17a500 prio 196 cost 0 latency 0.000273 osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0) v2 pg pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]
2014-06-12 14:46:25.090224 7f9d62bd6700  5 -- op tracker -- , seq: 747797, time: 2014-06-12 14:46:25.090221, event: done, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ack, result = 0)
2014-06-12 14:46:25.090279 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:25.090283 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:25.090291 7f9d683e1700  5 -- op tracker -- , seq: 747798, time: 2014-06-12 14:46:25.090259, event: reached_pg, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.090299 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] handle_message: 0x2d17a500
2014-06-12 14:46:25.090314 7f9d683e1700  5 -- op tracker -- , seq: 747798, time: 2014-06-12 14:46:25.090311, event: started, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.090319 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] sub_op_modify_reply: tid 236257 op  ack_type 4 from 23
2014-06-12 14:46:25.090337 7f9d683e1700  5 -- op tracker -- , seq: 747796, time: 2014-06-12 14:46:25.090331, event: sub_op_commit_rec, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.090344 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2847 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] repop_all_committed: repop tid 236257 all committed 
2014-06-12 14:46:25.090355 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2846 active+clean] eval_repop repgather(0x51f6480 173'2848 rep_tid=236257 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:25.090371 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2846 active+clean] log_op_stats osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4 inb 1720599 outb 0 rlat 0.051898 lat 0.052054
2014-06-12 14:46:25.090389 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2846 active+clean] publish_stats_to_osd 173:3665
2014-06-12 14:46:25.090403 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2846 active+clean]  sending commit on repgather(0x51f6480 173'2848 rep_tid=236257 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4) 0x32df0000
2014-06-12 14:46:25.090435 7f9d683e1700  5 -- op tracker -- , seq: 747796, time: 2014-06-12 14:46:25.090429, event: commit_sent, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.090443 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2846 active+clean]  requeue_ops 
2014-06-12 14:46:25.090461 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]  removing repgather(0x51f6480 173'2848 rep_tid=236257 committed?=1 applied?=1 op=osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:25.090480 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]    q front is repgather(0x51f6480 173'2848 rep_tid=236257 committed?=1 applied?=1 op=osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:25.090494 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] remove_repop repgather(0x51f6480 173'2848 rep_tid=236257 committed?=1 applied?=1 op=osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:25.090508 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]  requeue_ops 
2014-06-12 14:46:25.090533 7f9d683e1700  5 -- op tracker -- , seq: 747796, time: 2014-06-12 14:46:25.090528, event: done, op: osd_op(client.4294.0:144029 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1720320] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.090554 7f9d683e1700 10 osd.2 173 dequeue_op 0x2d17a500 finish
2014-06-12 14:46:25.090561 7f9d683e1700  5 -- op tracker -- , seq: 747798, time: 2014-06-12 14:46:25.090558, event: done, op: osd_sub_op_reply(client.4294.0:144029 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.090571 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:25.090574 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:25.097253 7f9d5a16f700  5 -- op tracker -- , seq: 747799, time: 2014-06-12 14:46:25.096063, event: header_read, op: osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.097267 7f9d5a16f700  5 -- op tracker -- , seq: 747799, time: 2014-06-12 14:46:25.096066, event: throttled, op: osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.097277 7f9d5a16f700  5 -- op tracker -- , seq: 747799, time: 2014-06-12 14:46:25.097202, event: all_read, op: osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.097282 7f9d5a16f700  5 -- op tracker -- , seq: 747799, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.097287 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:25.097293 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:25.097300 7f9d5a16f700 15 osd.2 173 enqueue_op 0x25e42c00 prio 127 cost 516903 latency 0.001237 osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.097328 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:25.097335 7f9d65bdc700 10 osd.2 173 dequeue_op 0x25e42c00 prio 127 cost 516903 latency 0.001272 osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2663 (0'0,173'2663] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active]
2014-06-12 14:46:25.097353 7f9d65bdc700  5 -- op tracker -- , seq: 747799, time: 2014-06-12 14:46:25.097351, event: reached_pg, op: osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.097366 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2663 (0'0,173'2663] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active] handle_message: 0x25e42c00
2014-06-12 14:46:25.097381 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2663 (0'0,173'2663] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active] sub_op_modify trans f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 v 173'2664 (transaction) 174
2014-06-12 14:46:25.097392 7f9d65bdc700  5 -- op tracker -- , seq: 747799, time: 2014-06-12 14:46:25.097390, event: started, op: osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.097405 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2663 (0'0,173'2663] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active] append_log log((0'0,173'2663], crt=0'0) [173'2664 (173'2662) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146408 2014-06-12 08:41:29.696960]
2014-06-12 14:46:25.097423 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2664 (0'0,173'2664] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active] add_log_entry 173'2664 (173'2662) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146408 2014-06-12 08:41:29.696960
2014-06-12 14:46:25.097439 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2664 (0'0,173'2664] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active] append_log  adding 1 keys
2014-06-12 14:46:25.097467 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2664, trimmed: 
2014-06-12 14:46:25.097485 7f9d65bdc700  5 -- op tracker -- , seq: 747799, time: 2014-06-12 14:46:25.097482, event: started, op: osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.097493 7f9d65bdc700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:25.097497 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:25.097498 7f9d65bdc700  5 queue_op 0xbdfd9e0 seq 2474 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:25.097507 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:46:25.097513 7f9d7f16b700  5 _do_op 0xbdfd9e0 seq 2474 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:25.097503 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2664 (0'0,173'2664] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active] do_sub_op osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.097516 7f9d65bdc700 10 osd.2 173 dequeue_op 0x25e42c00 finish
2014-06-12 14:46:25.097518 7f9d7f16b700 10 _do_transaction on 0x17687430
2014-06-12 14:46:25.097519 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:25.097521 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:25.097524 7f9d7f16b700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:25.097662 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:25.097686 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:25.097725 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:25.097909 7f9d7f16b700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:25.097916 7f9d7f16b700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:25.097920 7f9d7f16b700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:25.097934 7f9d7f16b700 15 _write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 1843200~516096
2014-06-12 14:46:25.098041 7f9d7f16b700 10 lookup_strip_header done  cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:25.098317 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:25.100006 7f9d7f16b700 10 _generic_write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 1843200~516096 = 0
2014-06-12 14:46:25.100062 7f9d7f16b700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:25.100084 7f9d7f16b700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:25.100096 7f9d7f16b700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:25.100105 7f9d7f16b700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:25.100344 7f9d7f16b700 20 set_header setting 16921 cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 parent seq 0
2014-06-12 14:46:25.100695 7f9d7f16b700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:25.100710 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:25.100716 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:25.103866 7f9d7f16b700 10 _do_op 0xbdfd9e0 seq 2474 r = 0, finisher 0x65ddc20 0
2014-06-12 14:46:25.103897 7f9d7f16b700 10 _finish_op 0xbdfd9e0 seq 2474 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:25.103906 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:46:25.103914 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:25.103933 7f9d7795c700  5 -- op tracker -- , seq: 747799, time: 2014-06-12 14:46:25.103919, event: sub_op_applied, op: osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.103942 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2664 (0'0,173'2664] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active] sub_op_modify_applied on 0x17687340 op osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.103972 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:25.103997 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2664 (0'0,173'2664] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active] op_applied on primary on version 173'2664
2014-06-12 14:46:25.104011 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2664 (0'0,173'2664] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active] op_applied on replica on version 173'2664
2014-06-12 14:46:25.104452 7f9d7715b700  5 -- op tracker -- , seq: 747799, time: 2014-06-12 14:46:25.104446, event: commit_sent, op: osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.104462 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2664 (0'0,173'2664] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2662 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:25.104476 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:25.104499 7f9d7715b700  5 -- op tracker -- , seq: 747799, time: 2014-06-12 14:46:25.104494, event: done, op: osd_sub_op(client.4316.0:146408 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2664 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.105386 7f9d50844700  5 -- op tracker -- , seq: 747800, time: 2014-06-12 14:46:25.098488, event: header_read, op: osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.105404 7f9d50844700  5 -- op tracker -- , seq: 747800, time: 2014-06-12 14:46:25.098492, event: throttled, op: osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.105414 7f9d50844700  5 -- op tracker -- , seq: 747800, time: 2014-06-12 14:46:25.105335, event: all_read, op: osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.105423 7f9d50844700  5 -- op tracker -- , seq: 747800, time: 0.000000, event: dispatched, op: osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.105433 7f9d50844700 20 osd.2 173 should_share_map client.4294 172.16.96.220:0/1242079 173
2014-06-12 14:46:25.105463 7f9d50844700 15 osd.2 173 enqueue_op 0x24c9c300 prio 63 cost 2473984 latency 0.006975 osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:25.105502 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:25.105514 7f9d64bda700 10 osd.2 173 dequeue_op 0x24c9c300 prio 63 cost 2473984 latency 0.007026 osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]
2014-06-12 14:46:25.105543 7f9d64bda700  5 -- op tracker -- , seq: 747800, time: 2014-06-12 14:46:25.105538, event: reached_pg, op: osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.105554 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:25.105571 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] handle_message: 0x24c9c300
2014-06-12 14:46:25.105583 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] do_op osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:25.105611 7f9d64bda700 15 getattr 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 '_'
2014-06-12 14:46:25.105828 7f9d64bda700 10 getattr 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 '_' = 0
2014-06-12 14:46:25.105849 7f9d64bda700 15 getattr 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 'snapset'
2014-06-12 14:46:25.105930 7f9d64bda700 10 getattr 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 'snapset' = 0
2014-06-12 14:46:25.105942 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] populate_obc_watchers e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.105956 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.105969 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] get_object_context: creating obc from disk: 0x6c978c0
2014-06-12 14:46:25.105980 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] get_object_context: 0x6c978c0 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 rwstate(none n=0 w=0) oi: e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3(173'2848 client.4294.0:144029 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2848) ssc: 0x16ee4700 snapset: 0=[]:[]+head
2014-06-12 14:46:25.105999 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] find_object_context e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 @head oi=e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3(173'2848 client.4294.0:144029 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2848)
2014-06-12 14:46:25.106027 7f9d64bda700  5 -- op tracker -- , seq: 747800, time: 2014-06-12 14:46:25.106021, event: started, op: osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.106039 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] execute_ctx 0xfd19800
2014-06-12 14:46:25.106059 7f9d64bda700  5 -- op tracker -- , seq: 747800, time: 2014-06-12 14:46:25.106054, event: started, op: osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.106067 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] do_op e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] ov 173'2848 av 173'2849 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:25.106084 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] do_osd_op e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984]
2014-06-12 14:46:25.106098 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:25.106116 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] do_osd_op  write 1720320~2473984
2014-06-12 14:46:25.106138 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] do_osd_op_effects on session 0x50a3000
2014-06-12 14:46:25.106152 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] make_writeable e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 snapset=0x16ee4738  snapc=0=[]
2014-06-12 14:46:25.106169 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] make_writeable e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:25.106182 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] finish_ctx e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 0xfd19800 op modify  
2014-06-12 14:46:25.106203 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]  set mtime to 2014-06-12 08:45:27.474843
2014-06-12 14:46:25.106225 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]  final snapset 0=[]:[]+head in e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.106247 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]  zeroing write result code 0
2014-06-12 14:46:25.106259 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] new_repop rep_tid 236258 on osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:25.106276 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] issue_repop rep_tid 236258 o e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.106303 7f9d64bda700  5 -- op tracker -- , seq: 747800, time: 2014-06-12 14:46:25.106298, event: waiting for subops from 23, op: osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.106325 7f9d64bda700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:25.106352 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2848 (0'0,173'2848] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] append_log log((0'0,173'2848], crt=0'0) [173'2849 (173'2848) modify   e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 by client.4294.0:144030 2014-06-12 08:45:27.474843]
2014-06-12 14:46:25.106378 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] add_log_entry 173'2849 (173'2848) modify   e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 by client.4294.0:144030 2014-06-12 08:45:27.474843
2014-06-12 14:46:25.106400 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] append_log  adding 1 keys
2014-06-12 14:46:25.106434 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2849, trimmed: 
2014-06-12 14:46:25.106460 7f9d64bda700  5 queue_transactions existing osr(3.605 0x276dd10)/0x276dd10
2014-06-12 14:46:25.106467 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:25.106468 7f9d64bda700  5 queue_op 0x7418a20 seq 2639 osr(3.605 0x276dd10) 2476277 bytes   (queue has 1 ops and 2476277 bytes)
2014-06-12 14:46:25.106476 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] eval_repop repgather(0x5390600 173'2849 rep_tid=236258 committed?=0 applied?=0 lock=0 op=osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:25.106495 7f9d64bda700 10 osd.2 173 dequeue_op 0x24c9c300 finish
2014-06-12 14:46:25.106500 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:25.106505 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:25.108003 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5420a20 (1 active)
2014-06-12 14:46:25.108050 7f9d7e169700  5 _do_op 0x7418a20 seq 2639 osr(3.605 0x276dd10)/0x276dd10 start
2014-06-12 14:46:25.108061 7f9d7e169700 10 _do_transaction on 0x28fc1140
2014-06-12 14:46:25.108072 7f9d7e169700 15 _omap_setkeys meta/1fe11caa/pglog_3.605/0//-1
2014-06-12 14:46:25.108205 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe11caa/pglog_3.605/0//-1
2014-06-12 14:46:25.108608 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:25.108691 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:25.108721 7f9d7e169700 15 _omap_rmkeys meta/1fe11caa/pglog_3.605/0//-1
2014-06-12 14:46:25.108728 7f9d7e169700 10 _omap_rmkeys meta/1fe11caa/pglog_3.605/0//-1 r = 0
2014-06-12 14:46:25.108736 7f9d7e169700 15 _omap_setkeys meta/1fe11caa/pglog_3.605/0//-1
2014-06-12 14:46:25.108756 7f9d7e169700 15 _write 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 1720320~2473984
2014-06-12 14:46:25.109113 7f9d7e169700 10 lookup_strip_header done  cid 3.605_head oid e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.109283 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:25.117714 7f9d7e169700 10 _generic_write 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 1720320~2473984 = 0
2014-06-12 14:46:25.117887 7f9d7e169700 15 _setattrs 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.117911 7f9d7e169700 10 _setattrs 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 = 0
2014-06-12 14:46:25.117920 7f9d7e169700 15 _setattrs 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3
2014-06-12 14:46:25.117929 7f9d7e169700 10 _setattrs 3.605_head/e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 = 0
2014-06-12 14:46:25.118367 7f9d7e169700 20 set_header setting 17751 cid 3.605_head oid e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 parent seq 0
2014-06-12 14:46:25.118580 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:25.118602 7f9d7e169700 20 set_header setting 2104 cid meta oid 1fe11caa/pglog_3.605/0//-1 parent seq 0
2014-06-12 14:46:25.118615 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:25.127978 7f9d7e169700 10 _do_op 0x7418a20 seq 2639 r = 0, finisher 0x89e6c60 0x3984de40
2014-06-12 14:46:25.128003 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] op_commit: 236258
2014-06-12 14:46:25.128034 7f9d7715b700  5 -- op tracker -- , seq: 747800, time: 2014-06-12 14:46:25.128027, event: op_commit, op: osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.128063 7f9d7e169700 10 _finish_op 0x7418a20 seq 2639 osr(3.605 0x276dd10)/0x276dd10
2014-06-12 14:46:25.128074 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5420a20 (0 active)
2014-06-12 14:46:25.128082 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:25.128077 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] op_applied: 236258
2014-06-12 14:46:25.128091 7f9d7795c700  5 -- op tracker -- , seq: 747800, time: 2014-06-12 14:46:25.128087, event: op_applied, op: osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.128099 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] op_applied on primary on version 173'2849
2014-06-12 14:46:25.139646 7f9d57543700  5 -- op tracker -- , seq: 747801, time: 2014-06-12 14:46:25.139581, event: header_read, op: osd_sub_op_reply(client.4294.0:144030 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.139658 7f9d57543700  5 -- op tracker -- , seq: 747801, time: 2014-06-12 14:46:25.139583, event: throttled, op: osd_sub_op_reply(client.4294.0:144030 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.139666 7f9d57543700  5 -- op tracker -- , seq: 747801, time: 2014-06-12 14:46:25.139601, event: all_read, op: osd_sub_op_reply(client.4294.0:144030 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.139675 7f9d57543700  5 -- op tracker -- , seq: 747801, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:144030 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.139680 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:144030 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:25.139686 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:25.139694 7f9d57543700 15 osd.2 173 enqueue_op 0x2d17b400 prio 196 cost 0 latency 0.000113 osd_sub_op_reply(client.4294.0:144030 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:25.139720 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:25.139727 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2d17b400 prio 196 cost 0 latency 0.000146 osd_sub_op_reply(client.4294.0:144030 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0) v2 pg pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]
2014-06-12 14:46:25.139755 7f9d6a3e5700  5 -- op tracker -- , seq: 747801, time: 2014-06-12 14:46:25.139752, event: reached_pg, op: osd_sub_op_reply(client.4294.0:144030 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.139764 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] handle_message: 0x2d17b400
2014-06-12 14:46:25.139779 7f9d6a3e5700  5 -- op tracker -- , seq: 747801, time: 2014-06-12 14:46:25.139776, event: started, op: osd_sub_op_reply(client.4294.0:144030 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.139784 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] sub_op_modify_reply: tid 236258 op  ack_type 4 from 23
2014-06-12 14:46:25.139803 7f9d6a3e5700  5 -- op tracker -- , seq: 747800, time: 2014-06-12 14:46:25.139796, event: sub_op_commit_rec, op: osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.139810 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] repop_all_applied: repop tid 236258 all applied 
2014-06-12 14:46:25.139820 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] eval_repop repgather(0x5390600 173'2849 rep_tid=236258 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:25.139836 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean]  sending ack on repgather(0x5390600 173'2849 rep_tid=236258 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4) 0x32df0280
2014-06-12 14:46:25.139867 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2848 crt=0'0 lcod 173'2847 mlcod 173'2847 active+clean] repop_all_committed: repop tid 236258 all committed 
2014-06-12 14:46:25.139879 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2847 active+clean] eval_repop repgather(0x5390600 173'2849 rep_tid=236258 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:25.139897 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2847 active+clean] log_op_stats osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4 inb 2474263 outb 0 rlat 0.041378 lat 0.041408
2014-06-12 14:46:25.139918 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2847 active+clean] publish_stats_to_osd 173:3666
2014-06-12 14:46:25.139931 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2847 active+clean]  sending commit on repgather(0x5390600 173'2849 rep_tid=236258 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4) 0x6700500
2014-06-12 14:46:25.139960 7f9d6a3e5700  5 -- op tracker -- , seq: 747800, time: 2014-06-12 14:46:25.139955, event: commit_sent, op: osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.139968 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2847 active+clean]  requeue_ops 
2014-06-12 14:46:25.139979 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean]  removing repgather(0x5390600 173'2849 rep_tid=236258 committed?=1 applied?=1 op=osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:25.139994 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean]    q front is repgather(0x5390600 173'2849 rep_tid=236258 committed?=1 applied?=1 op=osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:25.140008 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] remove_repop repgather(0x5390600 173'2849 rep_tid=236258 committed?=1 applied?=1 op=osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:25.140021 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean]  requeue_ops 
2014-06-12 14:46:25.140049 7f9d6a3e5700  5 -- op tracker -- , seq: 747800, time: 2014-06-12 14:46:25.140043, event: done, op: osd_op(client.4294.0:144030 rbd_data.109d6b8b4567.000000000000111a [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~2473984] 3.e4fcfe05 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.140068 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2d17b400 finish
2014-06-12 14:46:25.140076 7f9d6a3e5700  5 -- op tracker -- , seq: 747801, time: 2014-06-12 14:46:25.140072, event: done, op: osd_sub_op_reply(client.4294.0:144030 3.605 e4fcfe05/rbd_data.109d6b8b4567.000000000000111a/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.140086 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:25.140089 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:25.149343 7f9d56432700  5 -- op tracker -- , seq: 747802, time: 2014-06-12 14:46:25.148026, event: header_read, op: osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.149363 7f9d56432700  5 -- op tracker -- , seq: 747802, time: 2014-06-12 14:46:25.148031, event: throttled, op: osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.149371 7f9d56432700  5 -- op tracker -- , seq: 747802, time: 2014-06-12 14:46:25.149278, event: all_read, op: osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.149379 7f9d56432700  5 -- op tracker -- , seq: 747802, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.149386 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:25.149393 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:25.149415 7f9d56432700 15 osd.2 173 enqueue_op 0x160e4800 prio 127 cost 348967 latency 0.001389 osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.149500 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:25.149510 7f9d6abe6700 10 osd.2 173 dequeue_op 0x160e4800 prio 127 cost 348967 latency 0.001483 osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.67d( v 173'2509 (0'0,173'2509] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2508 active]
2014-06-12 14:46:25.149565 7f9d6abe6700  5 -- op tracker -- , seq: 747802, time: 2014-06-12 14:46:25.149561, event: reached_pg, op: osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.149690 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2509 (0'0,173'2509] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2508 active] handle_message: 0x160e4800
2014-06-12 14:46:25.149776 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2509 (0'0,173'2509] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2508 active] sub_op_modify trans a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 v 173'2510 (transaction) 174
2014-06-12 14:46:25.149798 7f9d6abe6700  5 -- op tracker -- , seq: 747802, time: 2014-06-12 14:46:25.149794, event: started, op: osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.149826 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2509 (0'0,173'2509] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2508 active] append_log log((0'0,173'2509], crt=0'0) [173'2510 (173'669) modify   a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 by client.4313.0:146357 2014-06-12 08:44:44.839802]
2014-06-12 14:46:25.149855 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2510 (0'0,173'2510] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2508 active] add_log_entry 173'2510 (173'669) modify   a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 by client.4313.0:146357 2014-06-12 08:44:44.839802
2014-06-12 14:46:25.149884 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2510 (0'0,173'2510] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2508 active] append_log  adding 1 keys
2014-06-12 14:46:25.149961 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2510, trimmed: 
2014-06-12 14:46:25.149998 7f9d6abe6700  5 -- op tracker -- , seq: 747802, time: 2014-06-12 14:46:25.149992, event: started, op: osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.150011 7f9d6abe6700  5 queue_transactions existing osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:25.150018 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:25.150020 7f9d6abe6700  5 queue_op 0x223c720 seq 2288 osr(3.67d 0x276df70) 350453 bytes   (queue has 1 ops and 350453 bytes)
2014-06-12 14:46:25.150039 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x524cb40 (1 active)
2014-06-12 14:46:25.150032 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.67d( v 173'2510 (0'0,173'2510] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2508 active] do_sub_op osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.150047 7f9d7815d700  5 _do_op 0x223c720 seq 2288 osr(3.67d 0x276df70)/0x276df70 start
2014-06-12 14:46:25.150049 7f9d6abe6700 10 osd.2 173 dequeue_op 0x160e4800 finish
2014-06-12 14:46:25.150054 7f9d7815d700 10 _do_transaction on 0x1cc282b0
2014-06-12 14:46:25.150053 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:25.150057 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:25.150061 7f9d7815d700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:25.150238 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:25.150264 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:25.150315 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:25.150516 7f9d7815d700 15 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:25.150526 7f9d7815d700 10 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1 r = 0
2014-06-12 14:46:25.150533 7f9d7815d700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:25.150563 7f9d7815d700 15 _write 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 0~348160
2014-06-12 14:46:25.150825 7f9d7815d700 10 lookup_strip_header done  cid 3.67d_head oid a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:25.151145 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:25.152414 7f9d7815d700 10 _generic_write 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 0~348160 = 0
2014-06-12 14:46:25.152479 7f9d7815d700 15 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:25.152495 7f9d7815d700 10 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 = 0
2014-06-12 14:46:25.152509 7f9d7815d700 15 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:25.152514 7f9d7815d700 10 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 = 0
2014-06-12 14:46:25.152656 7f9d7815d700 20 set_header setting 17386 cid 3.67d_head oid a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 parent seq 0
2014-06-12 14:46:25.152695 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:25.152704 7f9d7815d700 20 set_header setting 2272 cid meta oid 1fe0e10b/pglog_3.67d/0//-1 parent seq 0
2014-06-12 14:46:25.152711 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:25.154489 7f9d7815d700 10 _do_op 0x223c720 seq 2288 r = 0, finisher 0x82c7b00 0
2014-06-12 14:46:25.154512 7f9d7815d700 10 _finish_op 0x223c720 seq 2288 osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:25.154520 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x524cb40 (0 active)
2014-06-12 14:46:25.154529 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:25.154531 7f9d7715b700  5 -- op tracker -- , seq: 747802, time: 2014-06-12 14:46:25.154521, event: commit_sent, op: osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.154541 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2510 (0'0,173'2510] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2508 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:25.154571 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:25.154604 7f9d7795c700  5 -- op tracker -- , seq: 747802, time: 2014-06-12 14:46:25.154599, event: sub_op_applied, op: osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.154613 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2510 (0'0,173'2510] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2509 active] sub_op_modify_applied on 0x1cc281c0 op osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.154630 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2510 (0'0,173'2510] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2509 active] op_applied on primary on version 173'2510
2014-06-12 14:46:25.154640 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2510 (0'0,173'2510] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2509 active] op_applied on replica on version 173'2510
2014-06-12 14:46:25.154666 7f9d7795c700  5 -- op tracker -- , seq: 747802, time: 2014-06-12 14:46:25.154662, event: done, op: osd_sub_op(client.4313.0:146357 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.241998 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:25.242062 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:25.473594 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:25.473633 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:25.473637 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:25.473639 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:25.473641 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 7; oldest is 31.828046 seconds old
2014-06-12 14:46:25.473653 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:25.481934 7f9d56432700  5 -- op tracker -- , seq: 747803, time: 2014-06-12 14:46:25.479245, event: header_read, op: osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.481947 7f9d56432700  5 -- op tracker -- , seq: 747803, time: 2014-06-12 14:46:25.479249, event: throttled, op: osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.481957 7f9d56432700  5 -- op tracker -- , seq: 747803, time: 2014-06-12 14:46:25.481892, event: all_read, op: osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.481965 7f9d56432700  5 -- op tracker -- , seq: 747803, time: 0.000000, event: dispatched, op: osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.481971 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:25.481979 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:25.481987 7f9d56432700 15 osd.2 173 enqueue_op 0x160e5b00 prio 127 cost 246567 latency 0.002742 osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.485180 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:25.485197 7f9d66bde700 10 osd.2 173 dequeue_op 0x160e5b00 prio 127 cost 246567 latency 0.005951 osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2637 (0'0,173'2637] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2636 active]
2014-06-12 14:46:25.485226 7f9d66bde700  5 -- op tracker -- , seq: 747803, time: 2014-06-12 14:46:25.485221, event: reached_pg, op: osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.485241 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2637 (0'0,173'2637] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2636 active] handle_message: 0x160e5b00
2014-06-12 14:46:25.485253 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2637 (0'0,173'2637] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2636 active] sub_op_modify trans 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 v 173'2638 (transaction) 174
2014-06-12 14:46:25.485270 7f9d66bde700  5 -- op tracker -- , seq: 747803, time: 2014-06-12 14:46:25.485265, event: started, op: osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.485285 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2637 (0'0,173'2637] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2636 active] append_log log((0'0,173'2637], crt=0'0) [173'2638 (173'2637) modify   3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 by client.4286.0:144490 2014-06-12 08:43:17.199677]
2014-06-12 14:46:25.485315 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2638 (0'0,173'2638] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2636 active] add_log_entry 173'2638 (173'2637) modify   3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 by client.4286.0:144490 2014-06-12 08:43:17.199677
2014-06-12 14:46:25.485341 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2638 (0'0,173'2638] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2636 active] append_log  adding 1 keys
2014-06-12 14:46:25.485374 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2638, trimmed: 
2014-06-12 14:46:25.485398 7f9d66bde700  5 -- op tracker -- , seq: 747803, time: 2014-06-12 14:46:25.485393, event: started, op: osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.485409 7f9d66bde700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:25.485413 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:25.485414 7f9d66bde700  5 queue_op 0x4b2ee40 seq 2458 osr(3.1c7 0x276d170) 248053 bytes   (queue has 1 ops and 248053 bytes)
2014-06-12 14:46:25.485427 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:46:25.485433 7f9d7e96a700  5 _do_op 0x4b2ee40 seq 2458 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:46:25.485422 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2638 (0'0,173'2638] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2636 active] do_sub_op osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.485436 7f9d66bde700 10 osd.2 173 dequeue_op 0x160e5b00 finish
2014-06-12 14:46:25.485437 7f9d7e96a700 10 _do_transaction on 0x65987f0
2014-06-12 14:46:25.485439 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:25.485441 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:25.485443 7f9d7e96a700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:25.485574 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:25.485597 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:25.485661 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:25.485687 7f9d7e96a700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:25.485694 7f9d7e96a700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:46:25.485699 7f9d7e96a700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:25.485859 7f9d7e96a700 15 _write 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 3948544~245760
2014-06-12 14:46:25.486022 7f9d7e96a700 10 lookup_strip_header done  cid 3.1c7_head oid 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3
2014-06-12 14:46:25.486139 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:25.487101 7f9d7e96a700 10 _generic_write 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 3948544~245760 = 0
2014-06-12 14:46:25.487138 7f9d7e96a700 15 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3
2014-06-12 14:46:25.487152 7f9d7e96a700 10 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:25.487159 7f9d7e96a700 15 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3
2014-06-12 14:46:25.487167 7f9d7e96a700 10 _setattrs 3.1c7_head/3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:25.487324 7f9d7e96a700 20 set_header setting 16919 cid 3.1c7_head oid 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 parent seq 0
2014-06-12 14:46:25.487375 7f9d7e96a700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:46:25.487391 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:25.487401 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:25.488513 7f9d7e96a700 10 _do_op 0x4b2ee40 seq 2458 r = 0, finisher 0xa648e00 0
2014-06-12 14:46:25.488529 7f9d7e96a700 10 _finish_op 0x4b2ee40 seq 2458 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:25.488539 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:46:25.488543 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:25.488544 7f9d7715b700  5 -- op tracker -- , seq: 747803, time: 2014-06-12 14:46:25.488535, event: commit_sent, op: osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.488555 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2638 (0'0,173'2638] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2636 active] sub_op_modify_commit on op osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:25.488575 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:25.488605 7f9d7795c700  5 -- op tracker -- , seq: 747803, time: 2014-06-12 14:46:25.488599, event: sub_op_applied, op: osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.488615 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2638 (0'0,173'2638] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2637 active] sub_op_modify_applied on 0x6598700 op osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.488632 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2638 (0'0,173'2638] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2637 active] op_applied on primary on version 173'2638
2014-06-12 14:46:25.488640 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2638 (0'0,173'2638] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2637 active] op_applied on replica on version 173'2638
2014-06-12 14:46:25.488660 7f9d7795c700  5 -- op tracker -- , seq: 747803, time: 2014-06-12 14:46:25.488656, event: done, op: osd_sub_op(client.4286.0:144490 3.1c7 3d9581c7/rbd_data.10956b8b4567.000000000000103f/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.496695 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:25.498967 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:25.504174 7f9d5a16f700  5 -- op tracker -- , seq: 747804, time: 2014-06-12 14:46:25.498822, event: header_read, op: osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.504187 7f9d5a16f700  5 -- op tracker -- , seq: 747804, time: 2014-06-12 14:46:25.498824, event: throttled, op: osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.504197 7f9d5a16f700  5 -- op tracker -- , seq: 747804, time: 2014-06-12 14:46:25.504124, event: all_read, op: osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.504206 7f9d5a16f700  5 -- op tracker -- , seq: 747804, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.504212 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:25.504225 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:25.504234 7f9d5a16f700 15 osd.2 173 enqueue_op 0x23e08e00 prio 127 cost 516903 latency 0.005412 osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.504261 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:25.504270 7f9d69be4700 10 osd.2 173 dequeue_op 0x23e08e00 prio 127 cost 516903 latency 0.005447 osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2664 (0'0,173'2664] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2663 active]
2014-06-12 14:46:25.504293 7f9d69be4700  5 -- op tracker -- , seq: 747804, time: 2014-06-12 14:46:25.504288, event: reached_pg, op: osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.504301 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2664 (0'0,173'2664] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2663 active] handle_message: 0x23e08e00
2014-06-12 14:46:25.504312 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2664 (0'0,173'2664] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2663 active] sub_op_modify trans 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 v 173'2665 (transaction) 174
2014-06-12 14:46:25.504328 7f9d69be4700  5 -- op tracker -- , seq: 747804, time: 2014-06-12 14:46:25.504324, event: started, op: osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.504340 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2664 (0'0,173'2664] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2663 active] append_log log((0'0,173'2664], crt=0'0) [173'2665 (173'2663) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141067 2014-06-12 08:43:33.677996]
2014-06-12 14:46:25.504361 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2665 (0'0,173'2665] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2663 active] add_log_entry 173'2665 (173'2663) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141067 2014-06-12 08:43:33.677996
2014-06-12 14:46:25.504387 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2665 (0'0,173'2665] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2663 active] append_log  adding 1 keys
2014-06-12 14:46:25.504417 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2665, trimmed: 
2014-06-12 14:46:25.504440 7f9d69be4700  5 -- op tracker -- , seq: 747804, time: 2014-06-12 14:46:25.504435, event: started, op: osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.504449 7f9d69be4700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:25.504452 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:25.504454 7f9d69be4700  5 queue_op 0x6d05260 seq 2475 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:25.504467 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:46:25.504462 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2665 (0'0,173'2665] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2663 active] do_sub_op osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.504476 7f9d69be4700 10 osd.2 173 dequeue_op 0x23e08e00 finish
2014-06-12 14:46:25.504476 7f9d82171700  5 _do_op 0x6d05260 seq 2475 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:25.504480 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:25.504481 7f9d82171700 10 _do_transaction on 0x25e127f0
2014-06-12 14:46:25.504482 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:25.504488 7f9d82171700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:25.504610 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:25.504632 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:25.504677 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:25.504698 7f9d82171700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:25.504704 7f9d82171700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:25.504710 7f9d82171700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:25.504723 7f9d82171700 15 _write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 3096576~516096
2014-06-12 14:46:25.504873 7f9d82171700 10 lookup_strip_header done  cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:25.504965 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:25.507293 7f9d82171700 10 _generic_write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 3096576~516096 = 0
2014-06-12 14:46:25.507365 7f9d82171700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:25.507385 7f9d82171700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:25.507393 7f9d82171700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:25.507402 7f9d82171700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:25.507613 7f9d82171700 20 set_header setting 17215 cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 parent seq 0
2014-06-12 14:46:25.507682 7f9d82171700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:25.507698 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:25.507706 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:25.510550 7f9d82171700 10 _do_op 0x6d05260 seq 2475 r = 0, finisher 0x66acb40 0
2014-06-12 14:46:25.510575 7f9d82171700 10 _finish_op 0x6d05260 seq 2475 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:25.510607 7f9d7795c700  5 -- op tracker -- , seq: 747804, time: 2014-06-12 14:46:25.510597, event: sub_op_applied, op: osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.510618 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2665 (0'0,173'2665] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2663 active] sub_op_modify_applied on 0x25e12700 op osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.510648 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:25.510681 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2665 (0'0,173'2665] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2663 active] op_applied on primary on version 173'2665
2014-06-12 14:46:25.510689 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2665 (0'0,173'2665] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2663 active] op_applied on replica on version 173'2665
2014-06-12 14:46:25.510699 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:46:25.510707 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:25.510710 7f9d7715b700  5 -- op tracker -- , seq: 747804, time: 2014-06-12 14:46:25.510703, event: commit_sent, op: osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.510736 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2665 (0'0,173'2665] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2663 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:25.510756 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:25.510786 7f9d7715b700  5 -- op tracker -- , seq: 747804, time: 2014-06-12 14:46:25.510781, event: done, op: osd_sub_op(client.4284.0:141067 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2665 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.640286 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:25.640719 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:25.696034 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:25.696358 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:25.748836 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:25.751391 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:25.819061 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:25.819190 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:25.832564 7f9d56432700  5 -- op tracker -- , seq: 747805, time: 2014-06-12 14:46:25.829553, event: header_read, op: osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.832578 7f9d56432700  5 -- op tracker -- , seq: 747805, time: 2014-06-12 14:46:25.829565, event: throttled, op: osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.832586 7f9d56432700  5 -- op tracker -- , seq: 747805, time: 2014-06-12 14:46:25.832519, event: all_read, op: osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.832595 7f9d56432700  5 -- op tracker -- , seq: 747805, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.832600 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:25.832614 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:25.832625 7f9d56432700 15 osd.2 173 enqueue_op 0x23e09f00 prio 127 cost 1032999 latency 0.003071 osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.832691 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:25.832700 7f9d653db700 10 osd.2 173 dequeue_op 0x23e09f00 prio 127 cost 1032999 latency 0.003146 osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.67d( v 173'2510 (0'0,173'2510] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2509 active]
2014-06-12 14:46:25.832726 7f9d653db700  5 -- op tracker -- , seq: 747805, time: 2014-06-12 14:46:25.832722, event: reached_pg, op: osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.832736 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2510 (0'0,173'2510] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2509 active] handle_message: 0x23e09f00
2014-06-12 14:46:25.832750 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2510 (0'0,173'2510] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2509 active] sub_op_modify trans 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 v 173'2511 (transaction) 174
2014-06-12 14:46:25.832767 7f9d653db700  5 -- op tracker -- , seq: 747805, time: 2014-06-12 14:46:25.832763, event: started, op: osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.832785 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2510 (0'0,173'2510] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2509 active] append_log log((0'0,173'2510], crt=0'0) [173'2511 (173'2509) modify   6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 by client.4303.0:144816 2014-06-12 08:45:21.200451]
2014-06-12 14:46:25.832810 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2511 (0'0,173'2511] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2509 active] add_log_entry 173'2511 (173'2509) modify   6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 by client.4303.0:144816 2014-06-12 08:45:21.200451
2014-06-12 14:46:25.832833 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2511 (0'0,173'2511] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2509 active] append_log  adding 1 keys
2014-06-12 14:46:25.832870 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2511, trimmed: 
2014-06-12 14:46:25.832894 7f9d653db700  5 -- op tracker -- , seq: 747805, time: 2014-06-12 14:46:25.832889, event: started, op: osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.832905 7f9d653db700  5 queue_transactions existing osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:25.832909 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:25.832910 7f9d653db700  5 queue_op 0x54290e0 seq 2289 osr(3.67d 0x276df70) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:25.832924 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x524cb40 (1 active)
2014-06-12 14:46:25.832919 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.67d( v 173'2511 (0'0,173'2511] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2509 active] do_sub_op osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.832933 7f9d7d167700  5 _do_op 0x54290e0 seq 2289 osr(3.67d 0x276df70)/0x276df70 start
2014-06-12 14:46:25.832934 7f9d653db700 10 osd.2 173 dequeue_op 0x23e09f00 finish
2014-06-12 14:46:25.832937 7f9d7d167700 10 _do_transaction on 0x30b077b0
2014-06-12 14:46:25.832937 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:25.832940 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:25.832943 7f9d7d167700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:25.833079 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:25.833100 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:25.833151 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:25.833179 7f9d7d167700 15 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:25.833186 7f9d7d167700 10 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1 r = 0
2014-06-12 14:46:25.833192 7f9d7d167700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:25.833303 7f9d7d167700 15 _write 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 1052672~1032192
2014-06-12 14:46:25.833469 7f9d7d167700 10 lookup_strip_header done  cid 3.67d_head oid 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:25.833642 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:25.841246 7f9d7d167700 10 _generic_write 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 1052672~1032192 = 0
2014-06-12 14:46:25.841454 7f9d7d167700 15 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:25.841474 7f9d7d167700 10 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 = 0
2014-06-12 14:46:25.841480 7f9d7d167700 15 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:25.841489 7f9d7d167700 10 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 = 0
2014-06-12 14:46:25.846094 7f9d7d167700 20 set_header setting 17507 cid 3.67d_head oid 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 parent seq 0
2014-06-12 14:46:25.846458 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:25.846477 7f9d7d167700 20 set_header setting 2272 cid meta oid 1fe0e10b/pglog_3.67d/0//-1 parent seq 0
2014-06-12 14:46:25.846488 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:25.852064 7f9d7d167700 10 _do_op 0x54290e0 seq 2289 r = 0, finisher 0x5db7b40 0
2014-06-12 14:46:25.852112 7f9d7715b700  5 -- op tracker -- , seq: 747805, time: 2014-06-12 14:46:25.852102, event: commit_sent, op: osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.852122 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2511 (0'0,173'2511] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2509 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:25.852150 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:25.852179 7f9d7d167700 10 _finish_op 0x54290e0 seq 2289 osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:25.852187 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x524cb40 (0 active)
2014-06-12 14:46:25.852194 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:25.852203 7f9d7795c700  5 -- op tracker -- , seq: 747805, time: 2014-06-12 14:46:25.852195, event: sub_op_applied, op: osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.852211 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2511 (0'0,173'2511] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2510 active] sub_op_modify_applied on 0x30b076c0 op osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.852231 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2511 (0'0,173'2511] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2510 active] op_applied on primary on version 173'2511
2014-06-12 14:46:25.852240 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2511 (0'0,173'2511] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2510 active] op_applied on replica on version 173'2511
2014-06-12 14:46:25.852265 7f9d7795c700  5 -- op tracker -- , seq: 747805, time: 2014-06-12 14:46:25.852261, event: done, op: osd_sub_op(client.4303.0:144816 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.865387 7f9d56432700  5 -- op tracker -- , seq: 747806, time: 2014-06-12 14:46:25.865332, event: header_read, op: osd_sub_op_reply(client.4317.0:143789 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.865399 7f9d56432700  5 -- op tracker -- , seq: 747806, time: 2014-06-12 14:46:25.865335, event: throttled, op: osd_sub_op_reply(client.4317.0:143789 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.865407 7f9d56432700  5 -- op tracker -- , seq: 747806, time: 2014-06-12 14:46:25.865349, event: all_read, op: osd_sub_op_reply(client.4317.0:143789 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.865414 7f9d56432700  5 -- op tracker -- , seq: 747806, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143789 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.865419 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143789 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:25.865426 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:25.865436 7f9d56432700 15 osd.2 173 enqueue_op 0x34016b00 prio 196 cost 0 latency 0.000104 osd_sub_op_reply(client.4317.0:143789 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:25.865642 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:25.865653 7f9d63bd8700 10 osd.2 173 dequeue_op 0x34016b00 prio 196 cost 0 latency 0.000320 osd_sub_op_reply(client.4317.0:143789 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2 pg pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]
2014-06-12 14:46:25.865676 7f9d63bd8700  5 -- op tracker -- , seq: 747806, time: 2014-06-12 14:46:25.865673, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143789 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.865686 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] handle_message: 0x34016b00
2014-06-12 14:46:25.865702 7f9d63bd8700  5 -- op tracker -- , seq: 747806, time: 2014-06-12 14:46:25.865699, event: started, op: osd_sub_op_reply(client.4317.0:143789 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.865708 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] sub_op_modify_reply: tid 236159 op  ack_type 4 from 26
2014-06-12 14:46:25.865730 7f9d63bd8700  5 -- op tracker -- , seq: 747370, time: 2014-06-12 14:46:25.865722, event: sub_op_commit_rec, op: osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.865737 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] repop_all_applied: repop tid 236159 all applied 
2014-06-12 14:46:25.865747 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] eval_repop repgather(0x9909d40 173'2507 rep_tid=236159 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:25.865763 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean]  sending ack on repgather(0x9909d40 173'2507 rep_tid=236159 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) 0x6486a00
2014-06-12 14:46:25.865795 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2506 crt=0'0 lcod 173'2505 mlcod 173'2505 active+clean] repop_all_committed: repop tid 236159 all committed 
2014-06-12 14:46:25.865806 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2505 active+clean] eval_repop repgather(0x9909d40 173'2507 rep_tid=236159 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:25.865821 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2505 active+clean] log_op_stats osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 inb 422167 outb 0 rlat 29.130749 lat 29.130775
2014-06-12 14:46:25.865838 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2505 active+clean] publish_stats_to_osd 173:2877
2014-06-12 14:46:25.865850 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2505 active+clean]  sending commit on repgather(0x9909d40 173'2507 rep_tid=236159 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) 0x45df400
2014-06-12 14:46:25.865877 7f9d63bd8700  5 -- op tracker -- , seq: 747370, time: 2014-06-12 14:46:25.865872, event: commit_sent, op: osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.865885 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2505 active+clean]  requeue_ops 
2014-06-12 14:46:25.865895 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2506 active+clean]  removing repgather(0x9909d40 173'2507 rep_tid=236159 committed?=1 applied?=1 op=osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:25.865944 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2506 active+clean]    q front is repgather(0x9909d40 173'2507 rep_tid=236159 committed?=1 applied?=1 op=osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:25.865959 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2506 active+clean] remove_repop repgather(0x9909d40 173'2507 rep_tid=236159 committed?=1 applied?=1 op=osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:25.865972 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2506 active+clean]  requeue_ops 
2014-06-12 14:46:25.865993 7f9d63bd8700  5 -- op tracker -- , seq: 747370, time: 2014-06-12 14:46:25.865988, event: done, op: osd_op(client.4317.0:143789 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~421888] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.866012 7f9d63bd8700 10 osd.2 173 dequeue_op 0x34016b00 finish
2014-06-12 14:46:25.866019 7f9d63bd8700  5 -- op tracker -- , seq: 747806, time: 2014-06-12 14:46:25.866015, event: done, op: osd_sub_op_reply(client.4317.0:143789 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.866030 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:25.866033 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:25.869969 7f9d56432700  5 -- op tracker -- , seq: 747807, time: 2014-06-12 14:46:25.869115, event: header_read, op: osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.869984 7f9d56432700  5 -- op tracker -- , seq: 747807, time: 2014-06-12 14:46:25.869120, event: throttled, op: osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.869993 7f9d56432700  5 -- op tracker -- , seq: 747807, time: 2014-06-12 14:46:25.869928, event: all_read, op: osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.870002 7f9d56432700  5 -- op tracker -- , seq: 747807, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.870008 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:25.870014 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:25.870024 7f9d56432700 15 osd.2 173 enqueue_op 0x34017800 prio 127 cost 516903 latency 0.000908 osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.870768 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:25.870782 7f9d68be2700 10 osd.2 173 dequeue_op 0x34017800 prio 127 cost 516903 latency 0.001666 osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2996 (0'0,173'2996] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active]
2014-06-12 14:46:25.870809 7f9d68be2700  5 -- op tracker -- , seq: 747807, time: 2014-06-12 14:46:25.870805, event: reached_pg, op: osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.870820 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2996 (0'0,173'2996] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active] handle_message: 0x34017800
2014-06-12 14:46:25.870831 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2996 (0'0,173'2996] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active] sub_op_modify trans 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 v 173'2997 (transaction) 174
2014-06-12 14:46:25.870847 7f9d68be2700  5 -- op tracker -- , seq: 747807, time: 2014-06-12 14:46:25.870844, event: started, op: osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.870869 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2996 (0'0,173'2996] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active] append_log log((0'0,173'2996], crt=0'0) [173'2997 (173'2996) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145018 2014-06-12 08:44:48.218299]
2014-06-12 14:46:25.870901 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2997 (0'0,173'2997] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active] add_log_entry 173'2997 (173'2996) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145018 2014-06-12 08:44:48.218299
2014-06-12 14:46:25.870930 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2997 (0'0,173'2997] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active] append_log  adding 1 keys
2014-06-12 14:46:25.870969 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2997, trimmed: 
2014-06-12 14:46:25.870998 7f9d68be2700  5 -- op tracker -- , seq: 747807, time: 2014-06-12 14:46:25.870993, event: started, op: osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.871009 7f9d68be2700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:25.871013 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:25.871015 7f9d68be2700  5 queue_op 0x1ab07800 seq 2799 osr(3.458 0x276d960) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:25.871029 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2997 (0'0,173'2997] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active] do_sub_op osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.871043 7f9d68be2700 10 osd.2 173 dequeue_op 0x34017800 finish
2014-06-12 14:46:25.871046 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:25.871049 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:25.871056 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:25.871061 7f9d8116f700  5 _do_op 0x1ab07800 seq 2799 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:25.871065 7f9d8116f700 10 _do_transaction on 0x122f7430
2014-06-12 14:46:25.871070 7f9d8116f700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:25.871204 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:25.871227 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:25.871277 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:25.871303 7f9d8116f700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:25.871308 7f9d8116f700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:25.871313 7f9d8116f700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:25.871330 7f9d8116f700 15 _write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 3039232~516096
2014-06-12 14:46:25.871662 7f9d8116f700 10 lookup_strip_header done  cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:25.871834 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:25.874649 7f9d8116f700 10 _generic_write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 3039232~516096 = 0
2014-06-12 14:46:25.874727 7f9d8116f700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:25.874754 7f9d8116f700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:25.874759 7f9d8116f700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:25.874767 7f9d8116f700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:25.875031 7f9d8116f700 20 set_header setting 16936 cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 parent seq 0
2014-06-12 14:46:25.875114 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:25.875128 7f9d8116f700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:25.875139 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:25.878348 7f9d8116f700 10 _do_op 0x1ab07800 seq 2799 r = 0, finisher 0x5af6820 0
2014-06-12 14:46:25.878374 7f9d8116f700 10 _finish_op 0x1ab07800 seq 2799 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:25.878382 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:25.878387 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:25.878403 7f9d7795c700  5 -- op tracker -- , seq: 747807, time: 2014-06-12 14:46:25.878392, event: sub_op_applied, op: osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.878411 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2997 (0'0,173'2997] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active] sub_op_modify_applied on 0x122f7340 op osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:25.878443 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:25.878466 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2997 (0'0,173'2997] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active] op_applied on primary on version 173'2997
2014-06-12 14:46:25.878477 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2997 (0'0,173'2997] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active] op_applied on replica on version 173'2997
2014-06-12 14:46:25.878502 7f9d7715b700  5 -- op tracker -- , seq: 747807, time: 2014-06-12 14:46:25.878495, event: commit_sent, op: osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.878533 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2997 (0'0,173'2997] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2995 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:25.878550 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:25.878588 7f9d7715b700  5 -- op tracker -- , seq: 747807, time: 2014-06-12 14:46:25.878584, event: done, op: osd_sub_op(client.4308.0:145018 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2997 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:25.949888 7f9d5a16f700  5 -- op tracker -- , seq: 747808, time: 2014-06-12 14:46:25.949817, event: header_read, op: osd_sub_op_reply(client.4308.0:145047 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.949911 7f9d5a16f700  5 -- op tracker -- , seq: 747808, time: 2014-06-12 14:46:25.949820, event: throttled, op: osd_sub_op_reply(client.4308.0:145047 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.949920 7f9d5a16f700  5 -- op tracker -- , seq: 747808, time: 2014-06-12 14:46:25.949836, event: all_read, op: osd_sub_op_reply(client.4308.0:145047 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.949927 7f9d5a16f700  5 -- op tracker -- , seq: 747808, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4308.0:145047 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.949932 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4308.0:145047 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:25.949938 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:25.949946 7f9d5a16f700 15 osd.2 173 enqueue_op 0x6f4b900 prio 196 cost 0 latency 0.000129 osd_sub_op_reply(client.4308.0:145047 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:25.950005 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:25.950018 7f9d623d5700 10 osd.2 173 dequeue_op 0x6f4b900 prio 196 cost 0 latency 0.000201 osd_sub_op_reply(client.4308.0:145047 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0) v2 pg pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]
2014-06-12 14:46:25.950039 7f9d623d5700  5 -- op tracker -- , seq: 747808, time: 2014-06-12 14:46:25.950037, event: reached_pg, op: osd_sub_op_reply(client.4308.0:145047 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.950051 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] handle_message: 0x6f4b900
2014-06-12 14:46:25.950064 7f9d623d5700  5 -- op tracker -- , seq: 747808, time: 2014-06-12 14:46:25.950062, event: started, op: osd_sub_op_reply(client.4308.0:145047 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.950069 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] sub_op_modify_reply: tid 236107 op  ack_type 4 from 24
2014-06-12 14:46:25.950084 7f9d623d5700  5 -- op tracker -- , seq: 747222, time: 2014-06-12 14:46:25.950077, event: sub_op_commit_rec, op: osd_op(client.4308.0:145047 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1613824~2580480] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.950093 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] repop_all_applied: repop tid 236107 all applied 
2014-06-12 14:46:25.950101 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] eval_repop repgather(0x534eb40 173'2707 rep_tid=236107 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145047 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1613824~2580480] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:25.950112 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]  sending ack on repgather(0x534eb40 173'2707 rep_tid=236107 committed?=0 applied?=1 lock=0 op=osd_op(client.4308.0:145047 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1613824~2580480] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) 0x168ad400
2014-06-12 14:46:25.950139 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] repop_all_committed: repop tid 236107 all committed 
2014-06-12 14:46:25.950151 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean] eval_repop repgather(0x534eb40 173'2707 rep_tid=236107 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145047 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1613824~2580480] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:25.950169 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean] log_op_stats osd_op(client.4308.0:145047 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1613824~2580480] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4 inb 2580759 outb 0 rlat 32.304543 lat 32.304572
2014-06-12 14:46:25.950187 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean] publish_stats_to_osd 173:2890
2014-06-12 14:46:25.950198 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean]  sending commit on repgather(0x534eb40 173'2707 rep_tid=236107 committed?=1 applied?=1 lock=0 op=osd_op(client.4308.0:145047 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1613824~2580480] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4) 0x1990c500
2014-06-12 14:46:25.950227 7f9d623d5700  5 -- op tracker -- , seq: 747222, time: 2014-06-12 14:46:25.950222, event: commit_sent, op: osd_op(client.4308.0:145047 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1613824~2580480] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.950237 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean]  requeue_ops 
2014-06-12 14:46:25.950250 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]  removing repgather(0x534eb40 173'2707 rep_tid=236107 committed?=1 applied?=1 op=osd_op(client.4308.0:145047 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1613824~2580480] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:25.950261 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]    q front is repgather(0x534eb40 173'2707 rep_tid=236107 committed?=1 applied?=1 op=osd_op(client.4308.0:145047 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1613824~2580480] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:25.950271 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] remove_repop repgather(0x534eb40 173'2707 rep_tid=236107 committed?=1 applied?=1 op=osd_op(client.4308.0:145047 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1613824~2580480] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:25.950280 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]  requeue_ops 
2014-06-12 14:46:25.950299 7f9d623d5700  5 -- op tracker -- , seq: 747222, time: 2014-06-12 14:46:25.950296, event: done, op: osd_op(client.4308.0:145047 rbd_data.10ab6b8b4567.0000000000000f7f [set-alloc-hint object_size 4194304 write_size 4194304,write 1613824~2580480] 3.7bd84e13 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:25.950317 7f9d623d5700 10 osd.2 173 dequeue_op 0x6f4b900 finish
2014-06-12 14:46:25.950322 7f9d623d5700  5 -- op tracker -- , seq: 747808, time: 2014-06-12 14:46:25.950320, event: done, op: osd_sub_op_reply(client.4308.0:145047 3.613 7bd84e13/rbd_data.10ab6b8b4567.0000000000000f7f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:25.950329 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:25.950331 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.005960 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:26.006159 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:26.045972 7f9d56432700  5 -- op tracker -- , seq: 747809, time: 2014-06-12 14:46:26.044915, event: header_read, op: osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.045988 7f9d56432700  5 -- op tracker -- , seq: 747809, time: 2014-06-12 14:46:26.044919, event: throttled, op: osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.045998 7f9d56432700  5 -- op tracker -- , seq: 747809, time: 2014-06-12 14:46:26.045928, event: all_read, op: osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.046007 7f9d56432700  5 -- op tracker -- , seq: 747809, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.046012 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.046019 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:26.046028 7f9d56432700 15 osd.2 173 enqueue_op 0x34016d00 prio 127 cost 516903 latency 0.001112 osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.046437 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.046449 7f9d643d9700 10 osd.2 173 dequeue_op 0x34016d00 prio 127 cost 516903 latency 0.001534 osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.67d( v 173'2511 (0'0,173'2511] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2510 active]
2014-06-12 14:46:26.046479 7f9d643d9700  5 -- op tracker -- , seq: 747809, time: 2014-06-12 14:46:26.046475, event: reached_pg, op: osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.046490 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2511 (0'0,173'2511] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2510 active] handle_message: 0x34016d00
2014-06-12 14:46:26.046507 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2511 (0'0,173'2511] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2510 active] sub_op_modify trans a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 v 173'2512 (transaction) 174
2014-06-12 14:46:26.046528 7f9d643d9700  5 -- op tracker -- , seq: 747809, time: 2014-06-12 14:46:26.046524, event: started, op: osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.046546 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2511 (0'0,173'2511] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2510 active] append_log log((0'0,173'2511], crt=0'0) [173'2512 (173'2510) modify   a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 by client.4313.0:146358 2014-06-12 08:44:48.219344]
2014-06-12 14:46:26.046581 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2512 (0'0,173'2512] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2510 active] add_log_entry 173'2512 (173'2510) modify   a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 by client.4313.0:146358 2014-06-12 08:44:48.219344
2014-06-12 14:46:26.046607 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2512 (0'0,173'2512] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2510 active] append_log  adding 1 keys
2014-06-12 14:46:26.046650 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2512, trimmed: 
2014-06-12 14:46:26.046686 7f9d643d9700  5 -- op tracker -- , seq: 747809, time: 2014-06-12 14:46:26.046681, event: started, op: osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.046701 7f9d643d9700  5 queue_transactions existing osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:26.046706 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.046708 7f9d643d9700  5 queue_op 0x177b58c0 seq 2290 osr(3.67d 0x276df70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:26.046716 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.67d( v 173'2512 (0'0,173'2512] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2510 active] do_sub_op osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.046732 7f9d643d9700 10 osd.2 173 dequeue_op 0x34016d00 finish
2014-06-12 14:46:26.046736 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.046739 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.046750 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x524cb40 (1 active)
2014-06-12 14:46:26.046756 7f9d79960700  5 _do_op 0x177b58c0 seq 2290 osr(3.67d 0x276df70)/0x276df70 start
2014-06-12 14:46:26.046762 7f9d79960700 10 _do_transaction on 0x2daa6630
2014-06-12 14:46:26.046769 7f9d79960700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:26.046945 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:26.046982 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.047046 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.047358 7f9d79960700 15 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:26.047371 7f9d79960700 10 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1 r = 0
2014-06-12 14:46:26.047379 7f9d79960700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:26.047401 7f9d79960700 15 _write 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 348160~516096
2014-06-12 14:46:26.047604 7f9d79960700 10 lookup_strip_header done  cid 3.67d_head oid a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:26.048002 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:26.050533 7f9d79960700 10 _generic_write 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 348160~516096 = 0
2014-06-12 14:46:26.050591 7f9d79960700 15 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:26.050626 7f9d79960700 10 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 = 0
2014-06-12 14:46:26.050633 7f9d79960700 15 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:26.050642 7f9d79960700 10 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 = 0
2014-06-12 14:46:26.050931 7f9d79960700 20 set_header setting 17386 cid 3.67d_head oid a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 parent seq 0
2014-06-12 14:46:26.051035 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.051051 7f9d79960700 20 set_header setting 2272 cid meta oid 1fe0e10b/pglog_3.67d/0//-1 parent seq 0
2014-06-12 14:46:26.051061 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:26.055514 7f9d79960700 10 _do_op 0x177b58c0 seq 2290 r = 0, finisher 0x7f438e0 0
2014-06-12 14:46:26.055537 7f9d79960700 10 _finish_op 0x177b58c0 seq 2290 osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:26.055545 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x524cb40 (0 active)
2014-06-12 14:46:26.055554 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.055565 7f9d7715b700  5 -- op tracker -- , seq: 747809, time: 2014-06-12 14:46:26.055554, event: commit_sent, op: osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.055579 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2512 (0'0,173'2512] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2510 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:26.055610 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:26.055645 7f9d7795c700  5 -- op tracker -- , seq: 747809, time: 2014-06-12 14:46:26.055640, event: sub_op_applied, op: osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.055676 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2512 (0'0,173'2512] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2511 active] sub_op_modify_applied on 0x2daa6540 op osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.055692 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2512 (0'0,173'2512] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2511 active] op_applied on primary on version 173'2512
2014-06-12 14:46:26.055702 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2512 (0'0,173'2512] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2511 active] op_applied on replica on version 173'2512
2014-06-12 14:46:26.055727 7f9d7795c700  5 -- op tracker -- , seq: 747809, time: 2014-06-12 14:46:26.055722, event: done, op: osd_sub_op(client.4313.0:146358 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.144835 7f9d4829e700  5 -- op tracker -- , seq: 747810, time: 2014-06-12 14:46:26.144723, event: header_read, op: osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.144859 7f9d4829e700  5 -- op tracker -- , seq: 747810, time: 2014-06-12 14:46:26.144729, event: throttled, op: osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.144869 7f9d4829e700  5 -- op tracker -- , seq: 747810, time: 2014-06-12 14:46:26.144773, event: all_read, op: osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.144886 7f9d4829e700  5 -- op tracker -- , seq: 747810, time: 0.000000, event: dispatched, op: osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.144895 7f9d4829e700 20 osd.2 173 should_share_map client.4302 172.16.96.220:0/1243795 173
2014-06-12 14:46:26.144926 7f9d4829e700 15 osd.2 173 enqueue_op 0x68a7d00 prio 63 cost 12288 latency 0.000203 osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.144971 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.144984 7f9d633d7700 10 osd.2 173 dequeue_op 0x68a7d00 prio 63 cost 12288 latency 0.000261 osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]
2014-06-12 14:46:26.145015 7f9d633d7700  5 -- op tracker -- , seq: 747810, time: 2014-06-12 14:46:26.145010, event: reached_pg, op: osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.145025 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:26.145041 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] handle_message: 0x68a7d00
2014-06-12 14:46:26.145054 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] do_op osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:26.145082 7f9d633d7700 15 getattr 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 '_'
2014-06-12 14:46:26.153897 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:26.164859 7f9d633d7700 10 getattr 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 '_' = 0
2014-06-12 14:46:26.164883 7f9d633d7700 15 getattr 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 'snapset'
2014-06-12 14:46:26.164960 7f9d633d7700 10 getattr 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 'snapset' = 0
2014-06-12 14:46:26.164969 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] populate_obc_watchers 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.164982 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.164988 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] get_object_context: creating obc from disk: 0x217f0dc0
2014-06-12 14:46:26.164994 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] get_object_context: 0x217f0dc0 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 rwstate(none n=0 w=0) oi: 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3(173'663 client.4302.0:38781 wrlock_by=unknown.0.0:0 dirty s 4194304 uv663) ssc: 0x5d5ab60 snapset: 0=[]:[]+head
2014-06-12 14:46:26.165004 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] find_object_context 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 @head oi=67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3(173'663 client.4302.0:38781 wrlock_by=unknown.0.0:0 dirty s 4194304 uv663)
2014-06-12 14:46:26.165026 7f9d633d7700  5 -- op tracker -- , seq: 747810, time: 2014-06-12 14:46:26.165020, event: started, op: osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.165034 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] execute_ctx 0x30f7de00
2014-06-12 14:46:26.165045 7f9d633d7700  5 -- op tracker -- , seq: 747810, time: 2014-06-12 14:46:26.165042, event: started, op: osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.165050 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] do_op 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] ov 173'663 av 173'2728 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:26.165059 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] do_osd_op 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288]
2014-06-12 14:46:26.165065 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:26.165073 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] do_osd_op  write 0~12288
2014-06-12 14:46:26.165084 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] do_osd_op_effects on session 0x5c8d200
2014-06-12 14:46:26.165090 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] make_writeable 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 snapset=0x5d5ab98  snapc=0=[]
2014-06-12 14:46:26.165097 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] make_writeable 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:26.165103 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] finish_ctx 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 0x30f7de00 op modify  
2014-06-12 14:46:26.165111 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]  set mtime to 2014-06-12 08:45:28.525450
2014-06-12 14:46:26.165124 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]  final snapset 0=[]:[]+head in 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.165138 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]  zeroing write result code 0
2014-06-12 14:46:26.165146 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] new_repop rep_tid 236259 on osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.165157 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] issue_repop rep_tid 236259 o 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.165174 7f9d633d7700  5 -- op tracker -- , seq: 747810, time: 2014-06-12 14:46:26.165171, event: waiting for subops from 25, op: osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.165195 7f9d633d7700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:26.165213 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2727 (0'0,173'2727] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] append_log log((0'0,173'2727], crt=0'0) [173'2728 (173'663) modify   67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 by client.4302.0:146134 2014-06-12 08:45:28.525450]
2014-06-12 14:46:26.165236 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] add_log_entry 173'2728 (173'663) modify   67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 by client.4302.0:146134 2014-06-12 08:45:28.525450
2014-06-12 14:46:26.165256 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] append_log  adding 1 keys
2014-06-12 14:46:26.165288 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2728, trimmed: 
2014-06-12 14:46:26.165306 7f9d633d7700  5 queue_transactions existing osr(3.23f 0x276d3c0)/0x276d3c0
2014-06-12 14:46:26.165312 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.165314 7f9d633d7700  5 queue_op 0x5149200 seq 2554 osr(3.23f 0x276d3c0) 14581 bytes   (queue has 1 ops and 14581 bytes)
2014-06-12 14:46:26.165321 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] eval_repop repgather(0x53295c0 173'2728 rep_tid=236259 committed?=0 applied?=0 lock=0 op=osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.165335 7f9d633d7700 10 osd.2 173 dequeue_op 0x68a7d00 finish
2014-06-12 14:46:26.165340 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.165344 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.165353 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540e7e0 (1 active)
2014-06-12 14:46:26.165360 7f9d8096e700  5 _do_op 0x5149200 seq 2554 osr(3.23f 0x276d3c0)/0x276d3c0 start
2014-06-12 14:46:26.165364 7f9d8096e700 10 _do_transaction on 0x32e7bd40
2014-06-12 14:46:26.165370 7f9d8096e700 15 _omap_setkeys meta/1fe20b6b/pglog_3.23f/0//-1
2014-06-12 14:46:26.165478 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe20b6b/pglog_3.23f/0//-1
2014-06-12 14:46:26.165607 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.165676 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.165701 7f9d8096e700 15 _omap_rmkeys meta/1fe20b6b/pglog_3.23f/0//-1
2014-06-12 14:46:26.165707 7f9d8096e700 10 _omap_rmkeys meta/1fe20b6b/pglog_3.23f/0//-1 r = 0
2014-06-12 14:46:26.165713 7f9d8096e700 15 _omap_setkeys meta/1fe20b6b/pglog_3.23f/0//-1
2014-06-12 14:46:26.165730 7f9d8096e700 15 _write 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 0~12288
2014-06-12 14:46:26.165864 7f9d8096e700 10 lookup_strip_header done  cid 3.23f_head oid 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.166013 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:26.166064 7f9d8096e700 10 _generic_write 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 0~12288 = 0
2014-06-12 14:46:26.166078 7f9d8096e700 15 _setattrs 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.166088 7f9d8096e700 10 _setattrs 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:26.166093 7f9d8096e700 15 _setattrs 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.166100 7f9d8096e700 10 _setattrs 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:26.166220 7f9d8096e700 20 set_header setting 17168 cid 3.23f_head oid 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 parent seq 0
2014-06-12 14:46:26.166243 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.166255 7f9d8096e700 20 set_header setting 2024 cid meta oid 1fe20b6b/pglog_3.23f/0//-1 parent seq 0
2014-06-12 14:46:26.166265 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:26.166430 7f9d8096e700 10 _do_op 0x5149200 seq 2554 r = 0, finisher 0x53cb780 0x3dba300
2014-06-12 14:46:26.166441 7f9d8096e700 10 _finish_op 0x5149200 seq 2554 osr(3.23f 0x276d3c0)/0x276d3c0
2014-06-12 14:46:26.166448 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540e7e0 (0 active)
2014-06-12 14:46:26.166451 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.166462 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] op_applied: 236259
2014-06-12 14:46:26.166486 7f9d7795c700  5 -- op tracker -- , seq: 747810, time: 2014-06-12 14:46:26.166479, event: op_applied, op: osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.166495 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] op_applied on primary on version 173'2728
2014-06-12 14:46:26.166551 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] op_commit: 236259
2014-06-12 14:46:26.166574 7f9d7715b700  5 -- op tracker -- , seq: 747810, time: 2014-06-12 14:46:26.166568, event: op_commit, op: osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.167916 7f9d5bec4700  5 -- op tracker -- , seq: 747811, time: 2014-06-12 14:46:26.167872, event: header_read, op: osd_sub_op_reply(client.4302.0:146134 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.167927 7f9d5bec4700  5 -- op tracker -- , seq: 747811, time: 2014-06-12 14:46:26.167874, event: throttled, op: osd_sub_op_reply(client.4302.0:146134 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.167935 7f9d5bec4700  5 -- op tracker -- , seq: 747811, time: 2014-06-12 14:46:26.167883, event: all_read, op: osd_sub_op_reply(client.4302.0:146134 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.167943 7f9d5bec4700  5 -- op tracker -- , seq: 747811, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146134 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.167948 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146134 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:26.167954 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:26.167964 7f9d5bec4700 15 osd.2 173 enqueue_op 0x31d61400 prio 196 cost 0 latency 0.000091 osd_sub_op_reply(client.4302.0:146134 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:26.168013 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.168034 7f9d61bd4700 10 osd.2 173 dequeue_op 0x31d61400 prio 196 cost 0 latency 0.000162 osd_sub_op_reply(client.4302.0:146134 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0) v2 pg pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]
2014-06-12 14:46:26.168069 7f9d61bd4700  5 -- op tracker -- , seq: 747811, time: 2014-06-12 14:46:26.168066, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146134 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.168078 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] handle_message: 0x31d61400
2014-06-12 14:46:26.168110 7f9d61bd4700  5 -- op tracker -- , seq: 747811, time: 2014-06-12 14:46:26.168107, event: started, op: osd_sub_op_reply(client.4302.0:146134 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.168117 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] sub_op_modify_reply: tid 236259 op  ack_type 4 from 25
2014-06-12 14:46:26.168140 7f9d61bd4700  5 -- op tracker -- , seq: 747810, time: 2014-06-12 14:46:26.168132, event: sub_op_commit_rec, op: osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.168147 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] repop_all_applied: repop tid 236259 all applied 
2014-06-12 14:46:26.168158 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] eval_repop repgather(0x53295c0 173'2728 rep_tid=236259 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.168174 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean]  sending ack on repgather(0x53295c0 173'2728 rep_tid=236259 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) 0x45dfb80
2014-06-12 14:46:26.168210 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2727 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] repop_all_committed: repop tid 236259 all committed 
2014-06-12 14:46:26.168220 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2726 active+clean] eval_repop repgather(0x53295c0 173'2728 rep_tid=236259 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.168236 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2726 active+clean] log_op_stats osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4 inb 12567 outb 0 rlat 0.023485 lat 0.023512
2014-06-12 14:46:26.168251 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2726 active+clean] publish_stats_to_osd 173:3549
2014-06-12 14:46:26.168265 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2726 active+clean]  sending commit on repgather(0x53295c0 173'2728 rep_tid=236259 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) 0xe184280
2014-06-12 14:46:26.168295 7f9d61bd4700  5 -- op tracker -- , seq: 747810, time: 2014-06-12 14:46:26.168290, event: commit_sent, op: osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.168303 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2726 active+clean]  requeue_ops 
2014-06-12 14:46:26.168312 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  removing repgather(0x53295c0 173'2728 rep_tid=236259 committed?=1 applied?=1 op=osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.168330 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]    q front is repgather(0x53295c0 173'2728 rep_tid=236259 committed?=1 applied?=1 op=osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.168342 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] remove_repop repgather(0x53295c0 173'2728 rep_tid=236259 committed?=1 applied?=1 op=osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.168355 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  requeue_ops 
2014-06-12 14:46:26.168381 7f9d61bd4700  5 -- op tracker -- , seq: 747810, time: 2014-06-12 14:46:26.168376, event: done, op: osd_op(client.4302.0:146134 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~12288] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.168399 7f9d61bd4700 10 osd.2 173 dequeue_op 0x31d61400 finish
2014-06-12 14:46:26.168406 7f9d61bd4700  5 -- op tracker -- , seq: 747811, time: 2014-06-12 14:46:26.168403, event: done, op: osd_sub_op_reply(client.4302.0:146134 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.168415 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.168418 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.208031 7f9d5b6bc700  5 -- op tracker -- , seq: 747812, time: 2014-06-12 14:46:26.207050, event: header_read, op: osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.208047 7f9d5b6bc700  5 -- op tracker -- , seq: 747812, time: 2014-06-12 14:46:26.207053, event: throttled, op: osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.208064 7f9d5b6bc700  5 -- op tracker -- , seq: 747812, time: 2014-06-12 14:46:26.207982, event: all_read, op: osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.208077 7f9d5b6bc700  5 -- op tracker -- , seq: 747812, time: 0.000000, event: dispatched, op: osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.208084 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.208092 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:26.208102 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x17c2be00 prio 127 cost 279335 latency 0.001051 osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.208134 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.208145 7f9d673df700 10 osd.2 173 dequeue_op 0x17c2be00 prio 127 cost 279335 latency 0.001095 osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.322( v 173'2987 (0'0,173'2987] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active]
2014-06-12 14:46:26.208172 7f9d673df700  5 -- op tracker -- , seq: 747812, time: 2014-06-12 14:46:26.208168, event: reached_pg, op: osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.208182 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2987 (0'0,173'2987] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] handle_message: 0x17c2be00
2014-06-12 14:46:26.208198 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2987 (0'0,173'2987] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] sub_op_modify trans 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 v 173'2988 (transaction) 174
2014-06-12 14:46:26.208217 7f9d673df700  5 -- op tracker -- , seq: 747812, time: 2014-06-12 14:46:26.208213, event: started, op: osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.208239 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2987 (0'0,173'2987] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] append_log log((0'0,173'2987], crt=0'0) [173'2988 (173'881) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143667 2014-06-12 08:45:28.544934]
2014-06-12 14:46:26.208264 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2988 (0'0,173'2988] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] add_log_entry 173'2988 (173'881) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143667 2014-06-12 08:45:28.544934
2014-06-12 14:46:26.208287 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2988 (0'0,173'2988] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] append_log  adding 1 keys
2014-06-12 14:46:26.208325 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2988, trimmed: 
2014-06-12 14:46:26.208353 7f9d673df700  5 -- op tracker -- , seq: 747812, time: 2014-06-12 14:46:26.208349, event: started, op: osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.208364 7f9d673df700  5 queue_transactions existing osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.208370 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.208372 7f9d673df700  5 queue_op 0x774b560 seq 2742 osr(3.322 0x276d4e0) 280821 bytes   (queue has 1 ops and 280821 bytes)
2014-06-12 14:46:26.208386 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439200 (1 active)
2014-06-12 14:46:26.208381 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.322( v 173'2988 (0'0,173'2988] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] do_sub_op osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.208395 7f9d8016d700  5 _do_op 0x774b560 seq 2742 osr(3.322 0x276d4e0)/0x276d4e0 start
2014-06-12 14:46:26.208397 7f9d673df700 10 osd.2 173 dequeue_op 0x17c2be00 finish
2014-06-12 14:46:26.208401 7f9d8016d700 10 _do_transaction on 0x328a69b0
2014-06-12 14:46:26.208401 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.208404 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.208407 7f9d8016d700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.208580 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.208606 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.208659 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.208678 7f9d8016d700 15 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.208683 7f9d8016d700 10 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1 r = 0
2014-06-12 14:46:26.208688 7f9d8016d700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.208699 7f9d8016d700 15 _write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 0~278528
2014-06-12 14:46:26.208884 7f9d8016d700 10 lookup_strip_header done  cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.209301 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:26.209969 7f9d5b6bc700  5 -- op tracker -- , seq: 747813, time: 2014-06-12 14:46:26.208377, event: header_read, op: osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.209982 7f9d5b6bc700  5 -- op tracker -- , seq: 747813, time: 2014-06-12 14:46:26.208379, event: throttled, op: osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.209991 7f9d5b6bc700  5 -- op tracker -- , seq: 747813, time: 2014-06-12 14:46:26.209920, event: all_read, op: osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.209999 7f9d5b6bc700  5 -- op tracker -- , seq: 747813, time: 0.000000, event: dispatched, op: osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.210005 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.210012 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:26.210022 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x12bf6300 prio 127 cost 516903 latency 0.001644 osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.210075 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.210083 7f9d67be0700 10 osd.2 173 dequeue_op 0x12bf6300 prio 127 cost 516903 latency 0.001706 osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.322( v 173'2988 (0'0,173'2988] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active]
2014-06-12 14:46:26.210108 7f9d67be0700  5 -- op tracker -- , seq: 747813, time: 2014-06-12 14:46:26.210104, event: reached_pg, op: osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.210117 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2988 (0'0,173'2988] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] handle_message: 0x12bf6300
2014-06-12 14:46:26.210128 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2988 (0'0,173'2988] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] sub_op_modify trans 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 v 173'2989 (transaction) 174
2014-06-12 14:46:26.210144 7f9d67be0700  5 -- op tracker -- , seq: 747813, time: 2014-06-12 14:46:26.210140, event: started, op: osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.210161 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2988 (0'0,173'2988] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] append_log log((0'0,173'2988], crt=0'0) [173'2989 (173'2988) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143668 2014-06-12 08:45:28.565849]
2014-06-12 14:46:26.210185 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] add_log_entry 173'2989 (173'2988) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143668 2014-06-12 08:45:28.565849
2014-06-12 14:46:26.210206 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] append_log  adding 1 keys
2014-06-12 14:46:26.210259 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2989, trimmed: 
2014-06-12 14:46:26.210295 7f9d67be0700  5 -- op tracker -- , seq: 747813, time: 2014-06-12 14:46:26.210290, event: started, op: osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.210308 7f9d67be0700  5 queue_transactions existing osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.210316 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.210319 7f9d67be0700  5 queue_op 0x8372ba0 seq 2743 osr(3.322 0x276d4e0) 518389 bytes   (queue has 2 ops and 799210 bytes)
2014-06-12 14:46:26.210329 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] do_sub_op osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.210344 7f9d67be0700 10 osd.2 173 dequeue_op 0x12bf6300 finish
2014-06-12 14:46:26.210347 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.210350 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.210622 7f9d8016d700 10 _generic_write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 0~278528 = 0
2014-06-12 14:46:26.210664 7f9d8016d700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.210682 7f9d8016d700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.210688 7f9d8016d700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.210696 7f9d8016d700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.210877 7f9d8016d700 20 set_header setting 17454 cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 parent seq 0
2014-06-12 14:46:26.210935 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.210949 7f9d8016d700 20 set_header setting 2186 cid meta oid 1fe2232a/pglog_3.322/0//-1 parent seq 0
2014-06-12 14:46:26.210961 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:26.211202 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439200 (2 active)
2014-06-12 14:46:26.212567 7f9d8016d700 10 _do_op 0x774b560 seq 2742 r = 0, finisher 0x1e1a1480 0
2014-06-12 14:46:26.212591 7f9d8016d700 10 _finish_op 0x774b560 seq 2742 osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.212603 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439200 (1 active)
2014-06-12 14:46:26.212603 7f9d7915f700  5 _do_op 0x8372ba0 seq 2743 osr(3.322 0x276d4e0)/0x276d4e0 start
2014-06-12 14:46:26.212608 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.212610 7f9d7915f700 10 _do_transaction on 0x5b7a470
2014-06-12 14:46:26.212608 7f9d7715b700  5 -- op tracker -- , seq: 747812, time: 2014-06-12 14:46:26.212597, event: commit_sent, op: osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.212618 7f9d7915f700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.212616 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] sub_op_modify_commit on op osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:26.212641 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:26.212700 7f9d7795c700  5 -- op tracker -- , seq: 747812, time: 2014-06-12 14:46:26.212695, event: sub_op_applied, op: osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.212710 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] sub_op_modify_applied on 0x328a68c0 op osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.212731 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] op_applied on primary on version 173'2988
2014-06-12 14:46:26.212741 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] op_applied on replica on version 173'2988
2014-06-12 14:46:26.212748 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.212768 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.212827 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.212916 7f9d7915f700 15 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.212926 7f9d7915f700 10 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1 r = 0
2014-06-12 14:46:26.212934 7f9d7915f700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.212955 7f9d7915f700 15 _write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 278528~516096
2014-06-12 14:46:26.213088 7f9d7915f700 10 lookup_strip_header done  cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.213237 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:26.212765 7f9d7795c700  5 -- op tracker -- , seq: 747812, time: 2014-06-12 14:46:26.212762, event: done, op: osd_sub_op(client.4289.0:143667 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.215320 7f9d7915f700 10 _generic_write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 278528~516096 = 0
2014-06-12 14:46:26.215394 7f9d7915f700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.215412 7f9d7915f700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.215418 7f9d7915f700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.215427 7f9d7915f700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.215648 7f9d7915f700 20 set_header setting 17454 cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 parent seq 0
2014-06-12 14:46:26.215733 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.215748 7f9d7915f700 20 set_header setting 2186 cid meta oid 1fe2232a/pglog_3.322/0//-1 parent seq 0
2014-06-12 14:46:26.215759 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:26.222781 7f9d7915f700 10 _do_op 0x8372ba0 seq 2743 r = 0, finisher 0x6d390a0 0
2014-06-12 14:46:26.222836 7f9d7715b700  5 -- op tracker -- , seq: 747813, time: 2014-06-12 14:46:26.222825, event: commit_sent, op: osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.222850 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] sub_op_modify_commit on op osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:26.222884 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:26.222981 7f9d7915f700 10 _finish_op 0x8372ba0 seq 2743 osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.223015 7f9d7795c700  5 -- op tracker -- , seq: 747813, time: 2014-06-12 14:46:26.223008, event: sub_op_applied, op: osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.223027 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] sub_op_modify_applied on 0x5b7a380 op osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.223045 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] op_applied on primary on version 173'2989
2014-06-12 14:46:26.223054 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] op_applied on replica on version 173'2989
2014-06-12 14:46:26.223104 7f9d7915f700  5 -- op tracker -- , seq: 747813, time: 2014-06-12 14:46:26.223089, event: done, op: osd_sub_op(client.4289.0:143668 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.223169 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439200 (0 active)
2014-06-12 14:46:26.223173 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.239424 7f9d5b6bc700  5 -- op tracker -- , seq: 747814, time: 2014-06-12 14:46:26.238517, event: header_read, op: osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.239436 7f9d5b6bc700  5 -- op tracker -- , seq: 747814, time: 2014-06-12 14:46:26.238519, event: throttled, op: osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.239444 7f9d5b6bc700  5 -- op tracker -- , seq: 747814, time: 2014-06-12 14:46:26.239376, event: all_read, op: osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.239453 7f9d5b6bc700  5 -- op tracker -- , seq: 747814, time: 0.000000, event: dispatched, op: osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.239459 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.239466 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:26.239476 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x5ee8400 prio 127 cost 516903 latency 0.000958 osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.239519 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.239528 7f9d663dd700 10 osd.2 173 dequeue_op 0x5ee8400 prio 127 cost 516903 latency 0.001010 osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active]
2014-06-12 14:46:26.239552 7f9d663dd700  5 -- op tracker -- , seq: 747814, time: 2014-06-12 14:46:26.239548, event: reached_pg, op: osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.239562 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] handle_message: 0x5ee8400
2014-06-12 14:46:26.239578 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] sub_op_modify trans 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 v 173'2990 (transaction) 174
2014-06-12 14:46:26.239594 7f9d663dd700  5 -- op tracker -- , seq: 747814, time: 2014-06-12 14:46:26.239590, event: started, op: osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.239616 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2989 (0'0,173'2989] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] append_log log((0'0,173'2989], crt=0'0) [173'2990 (173'2989) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143669 2014-06-12 08:45:28.611425]
2014-06-12 14:46:26.239643 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2990 (0'0,173'2990] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] add_log_entry 173'2990 (173'2989) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143669 2014-06-12 08:45:28.611425
2014-06-12 14:46:26.239669 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2990 (0'0,173'2990] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] append_log  adding 1 keys
2014-06-12 14:46:26.239727 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2990, trimmed: 
2014-06-12 14:46:26.239762 7f9d663dd700  5 -- op tracker -- , seq: 747814, time: 2014-06-12 14:46:26.239757, event: started, op: osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.239777 7f9d663dd700  5 queue_transactions existing osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.239782 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.239784 7f9d663dd700  5 queue_op 0x2e7cc00 seq 2744 osr(3.322 0x276d4e0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:26.239799 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439200 (1 active)
2014-06-12 14:46:26.239793 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.322( v 173'2990 (0'0,173'2990] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] do_sub_op osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.239808 7f9d663dd700 10 osd.2 173 dequeue_op 0x5ee8400 finish
2014-06-12 14:46:26.239809 7f9d7d968700  5 _do_op 0x2e7cc00 seq 2744 osr(3.322 0x276d4e0)/0x276d4e0 start
2014-06-12 14:46:26.239811 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.239814 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.239815 7f9d7d968700 10 _do_transaction on 0x243bcf0
2014-06-12 14:46:26.239822 7f9d7d968700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.239947 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.239974 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.240020 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.240040 7f9d7d968700 15 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.240045 7f9d7d968700 10 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1 r = 0
2014-06-12 14:46:26.240050 7f9d7d968700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.240065 7f9d7d968700 15 _write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 794624~516096
2014-06-12 14:46:26.240442 7f9d7d968700 10 lookup_strip_header done  cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.240663 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:26.242587 7f9d5b6bc700  5 -- op tracker -- , seq: 747815, time: 2014-06-12 14:46:26.241587, event: header_read, op: osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.242617 7f9d5b6bc700  5 -- op tracker -- , seq: 747815, time: 2014-06-12 14:46:26.241591, event: throttled, op: osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.242626 7f9d5b6bc700  5 -- op tracker -- , seq: 747815, time: 2014-06-12 14:46:26.242519, event: all_read, op: osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.242635 7f9d5b6bc700  5 -- op tracker -- , seq: 747815, time: 0.000000, event: dispatched, op: osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.242649 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.242661 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:26.242708 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x160e4800 prio 127 cost 516903 latency 0.001120 osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.242776 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.242796 7f9d693e3700 10 osd.2 173 dequeue_op 0x160e4800 prio 127 cost 516903 latency 0.001208 osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.322( v 173'2990 (0'0,173'2990] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active]
2014-06-12 14:46:26.242826 7f9d693e3700  5 -- op tracker -- , seq: 747815, time: 2014-06-12 14:46:26.242823, event: reached_pg, op: osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.242847 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2990 (0'0,173'2990] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] handle_message: 0x160e4800
2014-06-12 14:46:26.242862 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2990 (0'0,173'2990] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] sub_op_modify trans 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 v 173'2991 (transaction) 174
2014-06-12 14:46:26.242879 7f9d693e3700  5 -- op tracker -- , seq: 747815, time: 2014-06-12 14:46:26.242875, event: started, op: osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.242901 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2990 (0'0,173'2990] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] append_log log((0'0,173'2990], crt=0'0) [173'2991 (173'2990) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143670 2014-06-12 08:45:28.618848]
2014-06-12 14:46:26.242928 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2991 (0'0,173'2991] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] add_log_entry 173'2991 (173'2990) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143670 2014-06-12 08:45:28.618848
2014-06-12 14:46:26.242953 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2991 (0'0,173'2991] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] append_log  adding 1 keys
2014-06-12 14:46:26.242995 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2991, trimmed: 
2014-06-12 14:46:26.243030 7f9d693e3700  5 -- op tracker -- , seq: 747815, time: 2014-06-12 14:46:26.243024, event: started, op: osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.243047 7f9d693e3700  5 queue_transactions existing osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.243054 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.243057 7f9d693e3700  5 queue_op 0x663f560 seq 2745 osr(3.322 0x276d4e0) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:26.243067 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.322( v 173'2991 (0'0,173'2991] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] do_sub_op osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.243083 7f9d693e3700 10 osd.2 173 dequeue_op 0x160e4800 finish
2014-06-12 14:46:26.243086 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.243089 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.243486 7f9d7d968700 10 _generic_write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 794624~516096 = 0
2014-06-12 14:46:26.243550 7f9d7d968700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.243622 7f9d7d968700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.243633 7f9d7d968700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.243652 7f9d7d968700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.243682 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439200 (2 active)
2014-06-12 14:46:26.243988 7f9d7d968700 20 set_header setting 17454 cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 parent seq 0
2014-06-12 14:46:26.244144 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.244165 7f9d7d968700 20 set_header setting 2186 cid meta oid 1fe2232a/pglog_3.322/0//-1 parent seq 0
2014-06-12 14:46:26.244175 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:26.248629 7f9d57543700  5 -- op tracker -- , seq: 747816, time: 2014-06-12 14:46:26.247710, event: header_read, op: osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.248642 7f9d57543700  5 -- op tracker -- , seq: 747816, time: 2014-06-12 14:46:26.247713, event: throttled, op: osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.248650 7f9d57543700  5 -- op tracker -- , seq: 747816, time: 2014-06-12 14:46:26.248579, event: all_read, op: osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.248658 7f9d57543700  5 -- op tracker -- , seq: 747816, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.248663 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.248670 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:26.248678 7f9d57543700 15 osd.2 173 enqueue_op 0x6f4bf00 prio 127 cost 340775 latency 0.000967 osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.248910 7f9d7d968700 10 _do_op 0x2e7cc00 seq 2744 r = 0, finisher 0x960c500 0
2014-06-12 14:46:26.248939 7f9d7715b700  5 -- op tracker -- , seq: 747814, time: 2014-06-12 14:46:26.248933, event: commit_sent, op: osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.248947 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2991 (0'0,173'2991] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] sub_op_modify_commit on op osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:26.248971 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:26.248994 7f9d7d968700 10 _finish_op 0x2e7cc00 seq 2744 osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.249006 7f9d83173700  5 _do_op 0x663f560 seq 2745 osr(3.322 0x276d4e0)/0x276d4e0 start
2014-06-12 14:46:26.249014 7f9d83173700 10 _do_transaction on 0x5e9a7f0
2014-06-12 14:46:26.249018 7f9d7795c700  5 -- op tracker -- , seq: 747814, time: 2014-06-12 14:46:26.249013, event: sub_op_applied, op: osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.249023 7f9d83173700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.249026 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2991 (0'0,173'2991] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] sub_op_modify_applied on 0x243bc00 op osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.249046 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2991 (0'0,173'2991] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] op_applied on primary on version 173'2990
2014-06-12 14:46:26.249055 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2991 (0'0,173'2991] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] op_applied on replica on version 173'2990
2014-06-12 14:46:26.249085 7f9d7d968700  5 -- op tracker -- , seq: 747814, time: 2014-06-12 14:46:26.249076, event: done, op: osd_sub_op(client.4289.0:143669 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.249119 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439200 (1 active)
2014-06-12 14:46:26.249123 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.249162 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.249194 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.249243 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.249455 7f9d83173700 15 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.249467 7f9d83173700 10 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1 r = 0
2014-06-12 14:46:26.249475 7f9d83173700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.249495 7f9d83173700 15 _write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 1310720~516096
2014-06-12 14:46:26.249582 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.249592 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x6f4bf00 prio 127 cost 340775 latency 0.001882 osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.247( v 173'2724 (0'0,173'2724] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2723 active]
2014-06-12 14:46:26.249625 7f9d6b3e7700  5 -- op tracker -- , seq: 747816, time: 2014-06-12 14:46:26.249620, event: reached_pg, op: osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.249644 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2724 (0'0,173'2724] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2723 active] handle_message: 0x6f4bf00
2014-06-12 14:46:26.249655 7f9d83173700 10 lookup_strip_header done  cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.249656 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2724 (0'0,173'2724] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2723 active] sub_op_modify trans 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 v 173'2725 (transaction) 174
2014-06-12 14:46:26.249671 7f9d6b3e7700  5 -- op tracker -- , seq: 747816, time: 2014-06-12 14:46:26.249667, event: started, op: osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.249686 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2724 (0'0,173'2724] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2723 active] append_log log((0'0,173'2724], crt=0'0) [173'2725 (173'764) modify   8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 by client.4313.0:146449 2014-06-12 08:45:28.623443]
2014-06-12 14:46:26.249707 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2725 (0'0,173'2725] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2723 active] add_log_entry 173'2725 (173'764) modify   8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 by client.4313.0:146449 2014-06-12 08:45:28.623443
2014-06-12 14:46:26.249730 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2725 (0'0,173'2725] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2723 active] append_log  adding 1 keys
2014-06-12 14:46:26.249759 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2725, trimmed: 
2014-06-12 14:46:26.249780 7f9d6b3e7700  5 -- op tracker -- , seq: 747816, time: 2014-06-12 14:46:26.249776, event: started, op: osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.249789 7f9d6b3e7700  5 queue_transactions existing osr(3.247 0x276d380)/0x276d380
2014-06-12 14:46:26.249792 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.249794 7f9d6b3e7700  5 queue_op 0x5be98c0 seq 2494 osr(3.247 0x276d380) 342261 bytes   (queue has 2 ops and 860650 bytes)
2014-06-12 14:46:26.249804 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x543e6c0 (2 active)
2014-06-12 14:46:26.249801 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.247( v 173'2725 (0'0,173'2725] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2723 active] do_sub_op osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.249814 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x6f4bf00 finish
2014-06-12 14:46:26.249815 7f9d7c165700  5 _do_op 0x5be98c0 seq 2494 osr(3.247 0x276d380)/0x276d380 start
2014-06-12 14:46:26.249818 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.249819 7f9d7c165700 10 _do_transaction on 0x508bcf0
2014-06-12 14:46:26.249820 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.249824 7f9d7c165700 15 _omap_setkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.249882 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:26.249908 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.249923 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.252099 7f9d83173700 10 _generic_write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 1310720~516096 = 0
2014-06-12 14:46:26.252160 7f9d83173700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.252177 7f9d83173700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.252185 7f9d83173700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.252193 7f9d83173700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.252394 7f9d83173700 20 set_header setting 17454 cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 parent seq 0
2014-06-12 14:46:26.252464 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.252481 7f9d83173700 20 set_header setting 2186 cid meta oid 1fe2232a/pglog_3.322/0//-1 parent seq 0
2014-06-12 14:46:26.252497 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:26.256685 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.256718 7f9d7c165700 15 _omap_rmkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.256724 7f9d7c165700 10 _omap_rmkeys meta/1fe2638a/pglog_3.247/0//-1 r = 0
2014-06-12 14:46:26.256727 7f9d83173700 10 _do_op 0x663f560 seq 2745 r = 0, finisher 0x6d38f80 0
2014-06-12 14:46:26.256730 7f9d7c165700 15 _omap_setkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.256749 7f9d7c165700 15 _write 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 0~339968
2014-06-12 14:46:26.257010 7f9d7c165700 10 lookup_strip_header done  cid 3.247_head oid 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.257033 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:26.258878 7f9d7c165700 10 _generic_write 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 0~339968 = 0
2014-06-12 14:46:26.258944 7f9d7c165700 15 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.258969 7f9d7c165700 10 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 = 0
2014-06-12 14:46:26.258977 7f9d7c165700 15 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.258986 7f9d7c165700 10 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 = 0
2014-06-12 14:46:26.259162 7f9d7c165700 20 set_header setting 17396 cid 3.247_head oid 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 parent seq 0
2014-06-12 14:46:26.259225 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.259245 7f9d7c165700 20 set_header setting 2250 cid meta oid 1fe2638a/pglog_3.247/0//-1 parent seq 0
2014-06-12 14:46:26.259260 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:26.261083 7f9d7c165700 10 _do_op 0x5be98c0 seq 2494 r = 0, finisher 0x8145660 0
2014-06-12 14:46:26.262779 7f9d5b6bc700  5 -- op tracker -- , seq: 747817, time: 2014-06-12 14:46:26.261461, event: header_read, op: osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.262792 7f9d5b6bc700  5 -- op tracker -- , seq: 747817, time: 2014-06-12 14:46:26.261463, event: throttled, op: osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.262801 7f9d5b6bc700  5 -- op tracker -- , seq: 747817, time: 2014-06-12 14:46:26.262739, event: all_read, op: osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.262810 7f9d5b6bc700  5 -- op tracker -- , seq: 747817, time: 0.000000, event: dispatched, op: osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.262814 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.262820 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:26.262829 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x20e88a00 prio 127 cost 516903 latency 0.001367 osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.262884 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.262891 7f9d62bd6700 10 osd.2 173 dequeue_op 0x20e88a00 prio 127 cost 516903 latency 0.001430 osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.322( v 173'2991 (0'0,173'2991] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active]
2014-06-12 14:46:26.262913 7f9d62bd6700  5 -- op tracker -- , seq: 747817, time: 2014-06-12 14:46:26.262909, event: reached_pg, op: osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.262922 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2991 (0'0,173'2991] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] handle_message: 0x20e88a00
2014-06-12 14:46:26.262934 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2991 (0'0,173'2991] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] sub_op_modify trans 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 v 173'2992 (transaction) 174
2014-06-12 14:46:26.262949 7f9d62bd6700  5 -- op tracker -- , seq: 747817, time: 2014-06-12 14:46:26.262945, event: started, op: osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.263779 7f9d7c165700 10 _finish_op 0x5be98c0 seq 2494 osr(3.247 0x276d380)/0x276d380
2014-06-12 14:46:26.263792 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x543e6c0 (1 active)
2014-06-12 14:46:26.263796 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.263803 7f9d83173700 10 _finish_op 0x663f560 seq 2745 osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.263811 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439200 (0 active)
2014-06-12 14:46:26.263824 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.265364 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2991 (0'0,173'2991] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] append_log log((0'0,173'2991], crt=0'0) [173'2992 (173'2991) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143671 2014-06-12 08:45:28.635478]
2014-06-12 14:46:26.265411 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] add_log_entry 173'2992 (173'2991) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143671 2014-06-12 08:45:28.635478
2014-06-12 14:46:26.265445 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] append_log  adding 1 keys
2014-06-12 14:46:26.265491 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2992, trimmed: 
2014-06-12 14:46:26.265550 7f9d62bd6700  5 -- op tracker -- , seq: 747817, time: 2014-06-12 14:46:26.265513, event: started, op: osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.265564 7f9d62bd6700  5 queue_transactions existing osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.265570 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.265572 7f9d62bd6700  5 queue_op 0x69b3020 seq 2746 osr(3.322 0x276d4e0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:26.265581 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] do_sub_op osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.265596 7f9d62bd6700 10 osd.2 173 dequeue_op 0x20e88a00 finish
2014-06-12 14:46:26.265604 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.265610 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.265651 7f9d7715b700  5 -- op tracker -- , seq: 747815, time: 2014-06-12 14:46:26.265645, event: commit_sent, op: osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.265659 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] sub_op_modify_commit on op osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:26.265706 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:26.265717 7f9d7795c700  5 -- op tracker -- , seq: 747816, time: 2014-06-12 14:46:26.265707, event: sub_op_applied, op: osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.265726 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2725 (0'0,173'2725] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2723 active] sub_op_modify_applied on 0x508bc00 op osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.265749 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:26.265757 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439200 (1 active)
2014-06-12 14:46:26.265764 7f9d82972700  5 _do_op 0x69b3020 seq 2746 osr(3.322 0x276d4e0)/0x276d4e0 start
2014-06-12 14:46:26.265768 7f9d82972700 10 _do_transaction on 0x601a9b0
2014-06-12 14:46:26.265775 7f9d82972700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.265767 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2725 (0'0,173'2725] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2723 active] op_applied on primary on version 173'2725
2014-06-12 14:46:26.265778 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2725 (0'0,173'2725] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2723 active] op_applied on replica on version 173'2725
2014-06-12 14:46:26.265796 7f9d7795c700  5 -- op tracker -- , seq: 747815, time: 2014-06-12 14:46:26.265792, event: sub_op_applied, op: osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.265811 7f9d7715b700  5 -- op tracker -- , seq: 747816, time: 2014-06-12 14:46:26.265793, event: commit_sent, op: osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.265805 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify_applied on 0x5e9a700 op osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.265819 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] op_applied on primary on version 173'2991
2014-06-12 14:46:26.265819 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2725 (0'0,173'2725] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2723 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:26.265828 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] op_applied on replica on version 173'2991
2014-06-12 14:46:26.265837 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:26.265849 7f9d7795c700  5 -- op tracker -- , seq: 747815, time: 2014-06-12 14:46:26.265845, event: done, op: osd_sub_op(client.4289.0:143670 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.265862 7f9d7715b700  5 -- op tracker -- , seq: 747816, time: 2014-06-12 14:46:26.265857, event: done, op: osd_sub_op(client.4313.0:146449 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.266034 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.266212 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.266276 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.266298 7f9d82972700 15 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.266303 7f9d82972700 10 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1 r = 0
2014-06-12 14:46:26.266309 7f9d82972700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.266326 7f9d82972700 15 _write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 1826816~516096
2014-06-12 14:46:26.266470 7f9d82972700 10 lookup_strip_header done  cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.266748 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:26.269446 7f9d82972700 10 _generic_write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 1826816~516096 = 0
2014-06-12 14:46:26.269517 7f9d82972700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.269534 7f9d82972700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.269541 7f9d82972700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.269549 7f9d82972700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.269826 7f9d82972700 20 set_header setting 17454 cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 parent seq 0
2014-06-12 14:46:26.269910 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.269926 7f9d82972700 20 set_header setting 2186 cid meta oid 1fe2232a/pglog_3.322/0//-1 parent seq 0
2014-06-12 14:46:26.269935 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:26.273173 7f9d82972700 10 _do_op 0x69b3020 seq 2746 r = 0, finisher 0x6ce7020 0
2014-06-12 14:46:26.273314 7f9d7715b700  5 -- op tracker -- , seq: 747817, time: 2014-06-12 14:46:26.273303, event: commit_sent, op: osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.273327 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify_commit on op osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:26.273352 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:26.273430 7f9d82972700 10 _finish_op 0x69b3020 seq 2746 osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.273468 7f9d7795c700  5 -- op tracker -- , seq: 747817, time: 2014-06-12 14:46:26.273463, event: sub_op_applied, op: osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.273493 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] sub_op_modify_applied on 0x601a8c0 op osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.273508 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] op_applied on primary on version 173'2992
2014-06-12 14:46:26.273518 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] op_applied on replica on version 173'2992
2014-06-12 14:46:26.273594 7f9d82972700  5 -- op tracker -- , seq: 747817, time: 2014-06-12 14:46:26.273583, event: done, op: osd_sub_op(client.4289.0:143671 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.273652 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439200 (0 active)
2014-06-12 14:46:26.273659 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.281447 7f9d57543700  5 -- op tracker -- , seq: 747818, time: 2014-06-12 14:46:26.280230, event: header_read, op: osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.281476 7f9d57543700  5 -- op tracker -- , seq: 747818, time: 2014-06-12 14:46:26.280231, event: throttled, op: osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.281487 7f9d57543700  5 -- op tracker -- , seq: 747818, time: 2014-06-12 14:46:26.281397, event: all_read, op: osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.281496 7f9d57543700  5 -- op tracker -- , seq: 747818, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.281502 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.281510 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:26.281520 7f9d57543700 15 osd.2 173 enqueue_op 0x6f4b100 prio 127 cost 516903 latency 0.001290 osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.281563 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.281573 7f9d683e1700 10 osd.2 173 dequeue_op 0x6f4b100 prio 127 cost 516903 latency 0.001342 osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.247( v 173'2725 (0'0,173'2725] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2724 active]
2014-06-12 14:46:26.281600 7f9d683e1700  5 -- op tracker -- , seq: 747818, time: 2014-06-12 14:46:26.281596, event: reached_pg, op: osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.281610 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2725 (0'0,173'2725] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2724 active] handle_message: 0x6f4b100
2014-06-12 14:46:26.281631 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2725 (0'0,173'2725] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2724 active] sub_op_modify trans 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 v 173'2726 (transaction) 174
2014-06-12 14:46:26.281656 7f9d683e1700  5 -- op tracker -- , seq: 747818, time: 2014-06-12 14:46:26.281651, event: started, op: osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.281674 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2725 (0'0,173'2725] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2724 active] append_log log((0'0,173'2725], crt=0'0) [173'2726 (173'2725) modify   8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 by client.4313.0:146450 2014-06-12 08:45:28.657248]
2014-06-12 14:46:26.281698 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2726 (0'0,173'2726] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2724 active] add_log_entry 173'2726 (173'2725) modify   8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 by client.4313.0:146450 2014-06-12 08:45:28.657248
2014-06-12 14:46:26.281721 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2726 (0'0,173'2726] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2724 active] append_log  adding 1 keys
2014-06-12 14:46:26.281758 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2726, trimmed: 
2014-06-12 14:46:26.281785 7f9d683e1700  5 -- op tracker -- , seq: 747818, time: 2014-06-12 14:46:26.281779, event: started, op: osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.281797 7f9d683e1700  5 queue_transactions existing osr(3.247 0x276d380)/0x276d380
2014-06-12 14:46:26.281801 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.281803 7f9d683e1700  5 queue_op 0x6a7bb00 seq 2495 osr(3.247 0x276d380) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:26.281811 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.247( v 173'2726 (0'0,173'2726] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2724 active] do_sub_op osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.281826 7f9d683e1700 10 osd.2 173 dequeue_op 0x6f4b100 finish
2014-06-12 14:46:26.281830 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.281833 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.281844 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x543e6c0 (1 active)
2014-06-12 14:46:26.281849 7f9d81970700  5 _do_op 0x6a7bb00 seq 2495 osr(3.247 0x276d380)/0x276d380 start
2014-06-12 14:46:26.281853 7f9d81970700 10 _do_transaction on 0x5a43270
2014-06-12 14:46:26.281859 7f9d81970700 15 _omap_setkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.282044 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.282228 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.282301 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.282326 7f9d81970700 15 _omap_rmkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.282335 7f9d81970700 10 _omap_rmkeys meta/1fe2638a/pglog_3.247/0//-1 r = 0
2014-06-12 14:46:26.282341 7f9d81970700 15 _omap_setkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.282361 7f9d81970700 15 _write 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 339968~516096
2014-06-12 14:46:26.282528 7f9d81970700 10 lookup_strip_header done  cid 3.247_head oid 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.282818 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:26.285249 7f9d81970700 10 _generic_write 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 339968~516096 = 0
2014-06-12 14:46:26.285322 7f9d81970700 15 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.285342 7f9d81970700 10 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 = 0
2014-06-12 14:46:26.285348 7f9d81970700 15 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.285356 7f9d81970700 10 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 = 0
2014-06-12 14:46:26.285575 7f9d81970700 20 set_header setting 17396 cid 3.247_head oid 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 parent seq 0
2014-06-12 14:46:26.285693 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.285709 7f9d81970700 20 set_header setting 2250 cid meta oid 1fe2638a/pglog_3.247/0//-1 parent seq 0
2014-06-12 14:46:26.285723 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:26.289525 7f9d81970700 10 _do_op 0x6a7bb00 seq 2495 r = 0, finisher 0x31d374c0 0
2014-06-12 14:46:26.289598 7f9d7715b700  5 -- op tracker -- , seq: 747818, time: 2014-06-12 14:46:26.289586, event: commit_sent, op: osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.289609 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2726 (0'0,173'2726] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2724 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:26.289652 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:26.289745 7f9d81970700 10 _finish_op 0x6a7bb00 seq 2495 osr(3.247 0x276d380)/0x276d380
2014-06-12 14:46:26.289784 7f9d7795c700  5 -- op tracker -- , seq: 747818, time: 2014-06-12 14:46:26.289778, event: sub_op_applied, op: osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.289794 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2726 (0'0,173'2726] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active] sub_op_modify_applied on 0x5a43180 op osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.289810 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2726 (0'0,173'2726] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active] op_applied on primary on version 173'2726
2014-06-12 14:46:26.289821 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2726 (0'0,173'2726] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active] op_applied on replica on version 173'2726
2014-06-12 14:46:26.289905 7f9d81970700  5 -- op tracker -- , seq: 747818, time: 2014-06-12 14:46:26.289894, event: done, op: osd_sub_op(client.4313.0:146450 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.289943 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x543e6c0 (0 active)
2014-06-12 14:46:26.289948 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.296219 7f9d57543700  5 -- op tracker -- , seq: 747819, time: 2014-06-12 14:46:26.294909, event: header_read, op: osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.296243 7f9d57543700  5 -- op tracker -- , seq: 747819, time: 2014-06-12 14:46:26.294911, event: throttled, op: osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.296252 7f9d57543700  5 -- op tracker -- , seq: 747819, time: 2014-06-12 14:46:26.296108, event: all_read, op: osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.296261 7f9d57543700  5 -- op tracker -- , seq: 747819, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.296271 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.296283 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:26.296301 7f9d57543700 15 osd.2 173 enqueue_op 0x6f4b700 prio 127 cost 516903 latency 0.001391 osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.296350 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.296364 7f9d65bdc700 10 osd.2 173 dequeue_op 0x6f4b700 prio 127 cost 516903 latency 0.001455 osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.247( v 173'2726 (0'0,173'2726] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active]
2014-06-12 14:46:26.296403 7f9d65bdc700  5 -- op tracker -- , seq: 747819, time: 2014-06-12 14:46:26.296398, event: reached_pg, op: osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.296418 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2726 (0'0,173'2726] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active] handle_message: 0x6f4b700
2014-06-12 14:46:26.296432 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2726 (0'0,173'2726] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active] sub_op_modify trans 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 v 173'2727 (transaction) 174
2014-06-12 14:46:26.296448 7f9d65bdc700  5 -- op tracker -- , seq: 747819, time: 2014-06-12 14:46:26.296444, event: started, op: osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.296477 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2726 (0'0,173'2726] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active] append_log log((0'0,173'2726], crt=0'0) [173'2727 (173'2726) modify   8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 by client.4313.0:146451 2014-06-12 08:45:28.672462]
2014-06-12 14:46:26.296511 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2727 (0'0,173'2727] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active] add_log_entry 173'2727 (173'2726) modify   8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 by client.4313.0:146451 2014-06-12 08:45:28.672462
2014-06-12 14:46:26.296538 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2727 (0'0,173'2727] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active] append_log  adding 1 keys
2014-06-12 14:46:26.296577 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2727, trimmed: 
2014-06-12 14:46:26.296605 7f9d65bdc700  5 -- op tracker -- , seq: 747819, time: 2014-06-12 14:46:26.296600, event: started, op: osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.296617 7f9d65bdc700  5 queue_transactions existing osr(3.247 0x276d380)/0x276d380
2014-06-12 14:46:26.296622 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.296624 7f9d65bdc700  5 queue_op 0x3b18480 seq 2496 osr(3.247 0x276d380) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:26.296641 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x543e6c0 (1 active)
2014-06-12 14:46:26.296635 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.247( v 173'2727 (0'0,173'2727] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active] do_sub_op osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.296650 7f9d65bdc700 10 osd.2 173 dequeue_op 0x6f4b700 finish
2014-06-12 14:46:26.296652 7f9d7c966700  5 _do_op 0x3b18480 seq 2496 osr(3.247 0x276d380)/0x276d380 start
2014-06-12 14:46:26.296654 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.296657 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.296660 7f9d7c966700 10 _do_transaction on 0x628db30
2014-06-12 14:46:26.296670 7f9d7c966700 15 _omap_setkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.296878 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.296937 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.297000 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.297143 7f9d7c966700 15 _omap_rmkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.297155 7f9d7c966700 10 _omap_rmkeys meta/1fe2638a/pglog_3.247/0//-1 r = 0
2014-06-12 14:46:26.297164 7f9d7c966700 15 _omap_setkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.297187 7f9d7c966700 15 _write 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 856064~516096
2014-06-12 14:46:26.297403 7f9d7c966700 10 lookup_strip_header done  cid 3.247_head oid 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.297609 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:26.299863 7f9d7c966700 10 _generic_write 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 856064~516096 = 0
2014-06-12 14:46:26.299928 7f9d7c966700 15 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.299950 7f9d7c966700 10 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 = 0
2014-06-12 14:46:26.299955 7f9d7c966700 15 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.299963 7f9d7c966700 10 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 = 0
2014-06-12 14:46:26.300187 7f9d7c966700 20 set_header setting 17396 cid 3.247_head oid 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 parent seq 0
2014-06-12 14:46:26.300276 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.300292 7f9d7c966700 20 set_header setting 2250 cid meta oid 1fe2638a/pglog_3.247/0//-1 parent seq 0
2014-06-12 14:46:26.300302 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:26.303198 7f9d7c966700 10 _do_op 0x3b18480 seq 2496 r = 0, finisher 0x70c7fe0 0
2014-06-12 14:46:26.303226 7f9d7c966700 10 _finish_op 0x3b18480 seq 2496 osr(3.247 0x276d380)/0x276d380
2014-06-12 14:46:26.303265 7f9d7795c700  5 -- op tracker -- , seq: 747819, time: 2014-06-12 14:46:26.303249, event: sub_op_applied, op: osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.303288 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2727 (0'0,173'2727] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active] sub_op_modify_applied on 0x628da40 op osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.303319 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:26.303341 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2727 (0'0,173'2727] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active] op_applied on primary on version 173'2727
2014-06-12 14:46:26.303352 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2727 (0'0,173'2727] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active] op_applied on replica on version 173'2727
2014-06-12 14:46:26.303417 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x543e6c0 (0 active)
2014-06-12 14:46:26.303428 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.303501 7f9d7715b700  5 -- op tracker -- , seq: 747819, time: 2014-06-12 14:46:26.303495, event: commit_sent, op: osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.303508 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2727 (0'0,173'2727] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2725 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:26.303524 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:26.303553 7f9d7715b700  5 -- op tracker -- , seq: 747819, time: 2014-06-12 14:46:26.303548, event: done, op: osd_sub_op(client.4313.0:146451 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.315852 7f9d57543700  5 -- op tracker -- , seq: 747820, time: 2014-06-12 14:46:26.312406, event: header_read, op: osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.315868 7f9d57543700  5 -- op tracker -- , seq: 747820, time: 2014-06-12 14:46:26.312409, event: throttled, op: osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.315876 7f9d57543700  5 -- op tracker -- , seq: 747820, time: 2014-06-12 14:46:26.315806, event: all_read, op: osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.315885 7f9d57543700  5 -- op tracker -- , seq: 747820, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.315890 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.315897 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:26.315913 7f9d57543700 15 osd.2 173 enqueue_op 0x6f4af00 prio 127 cost 1549095 latency 0.003507 osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.316457 7f9d42e4f700  5 -- op tracker -- , seq: 747821, time: 2014-06-12 14:46:26.313593, event: header_read, op: osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.316471 7f9d42e4f700  5 -- op tracker -- , seq: 747821, time: 2014-06-12 14:46:26.313598, event: throttled, op: osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.316481 7f9d42e4f700  5 -- op tracker -- , seq: 747821, time: 2014-06-12 14:46:26.316418, event: all_read, op: osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.316490 7f9d42e4f700  5 -- op tracker -- , seq: 747821, time: 0.000000, event: dispatched, op: osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.316498 7f9d42e4f700 20 osd.2 173 should_share_map client.4313 172.16.96.220:0/1244424 173
2014-06-12 14:46:26.316525 7f9d42e4f700 15 osd.2 173 enqueue_op 0x2911d00 prio 63 cost 274432 latency 0.002931 osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.316544 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.316544 7f9d57543700  5 -- op tracker -- , seq: 747822, time: 2014-06-12 14:46:26.315939, event: header_read, op: osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.316554 7f9d57543700  5 -- op tracker -- , seq: 747822, time: 2014-06-12 14:46:26.315940, event: throttled, op: osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.316562 7f9d57543700  5 -- op tracker -- , seq: 747822, time: 2014-06-12 14:46:26.316518, event: all_read, op: osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.316553 7f9d64bda700 10 osd.2 173 dequeue_op 0x6f4af00 prio 127 cost 1549095 latency 0.004147 osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.247( v 173'2727 (0'0,173'2727] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active]
2014-06-12 14:46:26.316570 7f9d57543700  5 -- op tracker -- , seq: 747822, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.316573 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:26.316574 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.316577 7f9d64bda700  5 -- op tracker -- , seq: 747820, time: 2014-06-12 14:46:26.316573, event: reached_pg, op: osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.316579 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:26.316586 7f9d57543700 15 osd.2 173 enqueue_op 0x6f4a600 prio 127 cost 516903 latency 0.000647 osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.316578 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2911d00 prio 63 cost 274432 latency 0.002985 osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]
2014-06-12 14:46:26.316586 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2727 (0'0,173'2727] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] handle_message: 0x6f4af00
2014-06-12 14:46:26.316602 7f9d6a3e5700  5 -- op tracker -- , seq: 747821, time: 2014-06-12 14:46:26.316597, event: reached_pg, op: osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.316610 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active)
2014-06-12 14:46:26.316602 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2727 (0'0,173'2727] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] sub_op_modify trans 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 v 173'2728 (transaction) 174
2014-06-12 14:46:26.316621 7f9d64bda700  5 -- op tracker -- , seq: 747820, time: 2014-06-12 14:46:26.316617, event: started, op: osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.316615 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:26.316629 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] handle_message: 0x2911d00
2014-06-12 14:46:26.316639 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_op osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:26.316644 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2727 (0'0,173'2727] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] append_log log((0'0,173'2727], crt=0'0) [173'2728 (173'2727) modify   8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 by client.4313.0:146452 2014-06-12 08:45:28.685160]
2014-06-12 14:46:26.316671 7f9d6a3e5700 15 getattr 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 '_'
2014-06-12 14:46:26.316679 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2728 (0'0,173'2728] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] add_log_entry 173'2728 (173'2727) modify   8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 by client.4313.0:146452 2014-06-12 08:45:28.685160
2014-06-12 14:46:26.316701 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2728 (0'0,173'2728] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] append_log  adding 1 keys
2014-06-12 14:46:26.316738 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2728, trimmed: 
2014-06-12 14:46:26.316763 7f9d64bda700  5 -- op tracker -- , seq: 747820, time: 2014-06-12 14:46:26.316758, event: started, op: osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.316782 7f9d64bda700  5 queue_transactions existing osr(3.247 0x276d380)/0x276d380
2014-06-12 14:46:26.316787 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.316788 7f9d64bda700  5 queue_op 0x71d6f60 seq 2497 osr(3.247 0x276d380) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:46:26.316800 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x543e6c0 (1 active)
2014-06-12 14:46:26.316807 7f9d7895e700  5 _do_op 0x71d6f60 seq 2497 osr(3.247 0x276d380)/0x276d380 start
2014-06-12 14:46:26.316796 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.247( v 173'2728 (0'0,173'2728] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] do_sub_op osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.316810 7f9d64bda700 10 osd.2 173 dequeue_op 0x6f4af00 finish
2014-06-12 14:46:26.316811 7f9d7895e700 10 _do_transaction on 0x19d870b0
2014-06-12 14:46:26.316817 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active)
2014-06-12 14:46:26.316818 7f9d7895e700 15 _omap_setkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.316821 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.316931 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.316957 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.317017 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.317046 7f9d7895e700 15 _omap_rmkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.317053 7f9d7895e700 10 _omap_rmkeys meta/1fe2638a/pglog_3.247/0//-1 r = 0
2014-06-12 14:46:26.317059 7f9d7895e700 15 _omap_setkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.317205 7f9d7895e700 15 _write 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 1372160~1548288
2014-06-12 14:46:26.317352 7f9d7895e700 10 lookup_strip_header done  cid 3.247_head oid 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.317486 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:26.317759 7f9d6abe6700 10 osd.2 173 dequeue_op 0x6f4a600 prio 127 cost 516903 latency 0.001820 osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.247( v 173'2728 (0'0,173'2728] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active]
2014-06-12 14:46:26.317786 7f9d6abe6700  5 -- op tracker -- , seq: 747822, time: 2014-06-12 14:46:26.317781, event: reached_pg, op: osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.317795 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2728 (0'0,173'2728] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] handle_message: 0x6f4a600
2014-06-12 14:46:26.317807 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2728 (0'0,173'2728] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] sub_op_modify trans 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 v 173'2729 (transaction) 174
2014-06-12 14:46:26.317823 7f9d6abe6700  5 -- op tracker -- , seq: 747822, time: 2014-06-12 14:46:26.317819, event: started, op: osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.317838 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2728 (0'0,173'2728] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] append_log log((0'0,173'2728], crt=0'0) [173'2729 (173'2728) modify   8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 by client.4313.0:146453 2014-06-12 08:45:28.689130]
2014-06-12 14:46:26.317860 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2729 (0'0,173'2729] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] add_log_entry 173'2729 (173'2728) modify   8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 by client.4313.0:146453 2014-06-12 08:45:28.689130
2014-06-12 14:46:26.317881 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2729 (0'0,173'2729] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] append_log  adding 1 keys
2014-06-12 14:46:26.317913 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2729, trimmed: 
2014-06-12 14:46:26.317935 7f9d6abe6700  5 -- op tracker -- , seq: 747822, time: 2014-06-12 14:46:26.317930, event: started, op: osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.317945 7f9d6abe6700  5 queue_transactions existing osr(3.247 0x276d380)/0x276d380
2014-06-12 14:46:26.317949 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.317950 7f9d6abe6700  5 queue_op 0xe6f5260 seq 2498 osr(3.247 0x276d380) 518389 bytes   (queue has 2 ops and 2068970 bytes)
2014-06-12 14:46:26.317959 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.247( v 173'2729 (0'0,173'2729] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] do_sub_op osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.317973 7f9d6abe6700 10 osd.2 173 dequeue_op 0x6f4a600 finish
2014-06-12 14:46:26.317978 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:26.317982 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.318014 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x543e6c0 (2 active)
2014-06-12 14:46:26.321038 7f9d5a16f700  5 -- op tracker -- , seq: 747823, time: 2014-06-12 14:46:26.319125, event: header_read, op: osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.321056 7f9d5a16f700  5 -- op tracker -- , seq: 747823, time: 2014-06-12 14:46:26.319126, event: throttled, op: osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.321065 7f9d5a16f700  5 -- op tracker -- , seq: 747823, time: 2014-06-12 14:46:26.320971, event: all_read, op: osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.321073 7f9d5a16f700  5 -- op tracker -- , seq: 747823, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.321094 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.321103 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:26.321116 7f9d5a16f700 15 osd.2 173 enqueue_op 0x6f4bc00 prio 127 cost 516903 latency 0.001990 osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.321213 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:26.321230 7f9d66bde700 10 osd.2 173 dequeue_op 0x6f4bc00 prio 127 cost 516903 latency 0.002104 osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2665 (0'0,173'2665] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2664 active]
2014-06-12 14:46:26.321264 7f9d66bde700  5 -- op tracker -- , seq: 747823, time: 2014-06-12 14:46:26.321257, event: reached_pg, op: osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.321279 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2665 (0'0,173'2665] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2664 active] handle_message: 0x6f4bc00
2014-06-12 14:46:26.321295 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2665 (0'0,173'2665] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2664 active] sub_op_modify trans f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 v 173'2666 (transaction) 174
2014-06-12 14:46:26.321313 7f9d66bde700  5 -- op tracker -- , seq: 747823, time: 2014-06-12 14:46:26.321309, event: started, op: osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.321370 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2665 (0'0,173'2665] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2664 active] append_log log((0'0,173'2665], crt=0'0) [173'2666 (173'2664) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146409 2014-06-12 08:41:29.785884]
2014-06-12 14:46:26.321411 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2666 (0'0,173'2666] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2664 active] add_log_entry 173'2666 (173'2664) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146409 2014-06-12 08:41:29.785884
2014-06-12 14:46:26.321455 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2666 (0'0,173'2666] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2664 active] append_log  adding 1 keys
2014-06-12 14:46:26.321496 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2666, trimmed: 
2014-06-12 14:46:26.321534 7f9d66bde700  5 -- op tracker -- , seq: 747823, time: 2014-06-12 14:46:26.321529, event: started, op: osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.321550 7f9d66bde700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:26.321558 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.321569 7f9d66bde700  5 queue_op 0x565a3c0 seq 2476 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 3 ops and 2587359 bytes)
2014-06-12 14:46:26.321566 7f9d57543700  5 -- op tracker -- , seq: 747824, time: 2014-06-12 14:46:26.318668, event: header_read, op: osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.321577 7f9d57543700  5 -- op tracker -- , seq: 747824, time: 2014-06-12 14:46:26.318670, event: throttled, op: osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.321586 7f9d57543700  5 -- op tracker -- , seq: 747824, time: 2014-06-12 14:46:26.321528, event: all_read, op: osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.321578 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2666 (0'0,173'2666] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2664 active] do_sub_op osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.321594 7f9d66bde700 10 osd.2 173 dequeue_op 0x6f4bc00 finish
2014-06-12 14:46:26.321595 7f9d57543700  5 -- op tracker -- , seq: 747824, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.321598 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:26.321602 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.321602 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.321632 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (3 active)
2014-06-12 14:46:26.321642 7f9d7a962700  5 _do_op 0x565a3c0 seq 2476 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:26.321652 7f9d7a962700 10 _do_transaction on 0x6671970
2014-06-12 14:46:26.321662 7f9d7a962700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:26.321917 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:26.321609 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:26.321938 7f9d57543700 15 osd.2 173 enqueue_op 0x28dd0400 prio 127 cost 758567 latency 0.003270 osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.321950 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.321975 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:26.321984 7f9d69be4700 10 osd.2 173 dequeue_op 0x28dd0400 prio 127 cost 758567 latency 0.003316 osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.247( v 173'2729 (0'0,173'2729] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active]
2014-06-12 14:46:26.322014 7f9d69be4700  5 -- op tracker -- , seq: 747824, time: 2014-06-12 14:46:26.322009, event: reached_pg, op: osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.322027 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2729 (0'0,173'2729] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] handle_message: 0x28dd0400
2014-06-12 14:46:26.322040 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2729 (0'0,173'2729] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] sub_op_modify trans 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 v 173'2730 (transaction) 174
2014-06-12 14:46:26.322085 7f9d69be4700  5 -- op tracker -- , seq: 747824, time: 2014-06-12 14:46:26.322081, event: started, op: osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.322109 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2729 (0'0,173'2729] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] append_log log((0'0,173'2729], crt=0'0) [173'2730 (173'2729) modify   8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 by client.4313.0:146454 2014-06-12 08:45:28.693393]
2014-06-12 14:46:26.322136 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] add_log_entry 173'2730 (173'2729) modify   8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 by client.4313.0:146454 2014-06-12 08:45:28.693393
2014-06-12 14:46:26.322163 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] append_log  adding 1 keys
2014-06-12 14:46:26.322261 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2730, trimmed: 
2014-06-12 14:46:26.322357 7f9d69be4700  5 -- op tracker -- , seq: 747824, time: 2014-06-12 14:46:26.322348, event: started, op: osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.322375 7f9d69be4700  5 queue_transactions existing osr(3.247 0x276d380)/0x276d380
2014-06-12 14:46:26.322381 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.322383 7f9d69be4700  5 queue_op 0x6e75e60 seq 2499 osr(3.247 0x276d380) 760053 bytes   (queue has 4 ops and 3347412 bytes)
2014-06-12 14:46:26.322393 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] do_sub_op osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.322408 7f9d69be4700 10 osd.2 173 dequeue_op 0x28dd0400 finish
2014-06-12 14:46:26.322412 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:26.322415 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.322458 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x543e6c0 (4 active)
2014-06-12 14:46:26.326037 7f9d7895e700 10 _generic_write 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 1372160~1548288 = 0
2014-06-12 14:46:26.326354 7f9d7895e700 15 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.326380 7f9d7895e700 10 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 = 0
2014-06-12 14:46:26.326386 7f9d7895e700 15 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.326395 7f9d7895e700 10 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 = 0
2014-06-12 14:46:26.327071 7f9d7895e700 20 set_header setting 17396 cid 3.247_head oid 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 parent seq 0
2014-06-12 14:46:26.327361 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.327379 7f9d7895e700 20 set_header setting 2250 cid meta oid 1fe2638a/pglog_3.247/0//-1 parent seq 0
2014-06-12 14:46:26.327393 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:26.335810 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.335851 7f9d7a962700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:26.335857 7f9d7a962700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:26.335864 7f9d7a962700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:26.335882 7f9d7a962700 15 _write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 2359296~516096
2014-06-12 14:46:26.336276 7f9d7a962700 10 lookup_strip_header done  cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:26.336468 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:26.338545 7f9d7895e700 10 _do_op 0x71d6f60 seq 2497 r = 0, finisher 0x96590a0 0
2014-06-12 14:46:26.338539 7f9d7a962700 10 _generic_write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 2359296~516096 = 0
2014-06-12 14:46:26.338613 7f9d7a962700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:26.338632 7f9d7a962700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:26.338639 7f9d7a962700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:26.338647 7f9d7a962700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:26.338823 7f9d7a962700 20 set_header setting 16921 cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 parent seq 0
2014-06-12 14:46:26.338907 7f9d7a962700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:26.338921 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.338930 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:26.340797 7f9d7a962700 10 _do_op 0x565a3c0 seq 2476 r = 0, finisher 0x7ff98c0 0
2014-06-12 14:46:26.340826 7f9d7a962700 10 _finish_op 0x565a3c0 seq 2476 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:26.340835 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (3 active)
2014-06-12 14:46:26.340849 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.340996 7f9d7895e700 10 _finish_op 0x71d6f60 seq 2497 osr(3.247 0x276d380)/0x276d380
2014-06-12 14:46:26.341005 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x543e6c0 (2 active)
2014-06-12 14:46:26.341012 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.341068 7f9d7a161700  5 _do_op 0xe6f5260 seq 2498 osr(3.247 0x276d380)/0x276d380 start
2014-06-12 14:46:26.341079 7f9d7a161700 10 _do_transaction on 0x52c8470
2014-06-12 14:46:26.341096 7f9d7a161700 15 _omap_setkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.341267 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.341298 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.341351 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.341379 7f9d7a161700 15 _omap_rmkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.341388 7f9d7a161700 10 _omap_rmkeys meta/1fe2638a/pglog_3.247/0//-1 r = 0
2014-06-12 14:46:26.341395 7f9d7a161700 15 _omap_setkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.341411 7f9d7a161700 15 _write 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 2920448~516096
2014-06-12 14:46:26.341751 7f9d7a161700 10 lookup_strip_header done  cid 3.247_head oid 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.341897 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:26.340859 7f9d7715b700  5 -- op tracker -- , seq: 747823, time: 2014-06-12 14:46:26.340848, event: commit_sent, op: osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.342528 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2666 (0'0,173'2666] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2664 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:26.342560 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:26.342594 7f9d7715b700  5 -- op tracker -- , seq: 747820, time: 2014-06-12 14:46:26.342589, event: commit_sent, op: osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.342601 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2726 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:26.342619 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:26.342647 7f9d7795c700  5 -- op tracker -- , seq: 747823, time: 2014-06-12 14:46:26.342641, event: sub_op_applied, op: osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.342656 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2666 (0'0,173'2666] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2665 active] sub_op_modify_applied on 0x6671880 op osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.342671 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2666 (0'0,173'2666] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2665 active] op_applied on primary on version 173'2666
2014-06-12 14:46:26.342680 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2666 (0'0,173'2666] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2665 active] op_applied on replica on version 173'2666
2014-06-12 14:46:26.342702 7f9d7795c700  5 -- op tracker -- , seq: 747823, time: 2014-06-12 14:46:26.342698, event: done, op: osd_sub_op(client.4316.0:146409 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2666 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.342736 7f9d7795c700  5 -- op tracker -- , seq: 747820, time: 2014-06-12 14:46:26.342731, event: sub_op_applied, op: osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.342741 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2727 active] sub_op_modify_applied on 0x19d86fc0 op osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.342753 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2727 active] op_applied on primary on version 173'2728
2014-06-12 14:46:26.342761 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2727 active] op_applied on replica on version 173'2728
2014-06-12 14:46:26.342782 7f9d7795c700  5 -- op tracker -- , seq: 747820, time: 2014-06-12 14:46:26.342778, event: done, op: osd_sub_op(client.4313.0:146452 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.343814 7f9d7a161700 10 _generic_write 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 2920448~516096 = 0
2014-06-12 14:46:26.343866 7f9d7a161700 15 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.343884 7f9d7a161700 10 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 = 0
2014-06-12 14:46:26.343893 7f9d7a161700 15 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.343902 7f9d7a161700 10 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 = 0
2014-06-12 14:46:26.344104 7f9d7a161700 20 set_header setting 17396 cid 3.247_head oid 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 parent seq 0
2014-06-12 14:46:26.344185 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.344200 7f9d7a161700 20 set_header setting 2250 cid meta oid 1fe2638a/pglog_3.247/0//-1 parent seq 0
2014-06-12 14:46:26.344209 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:26.350313 7f9d7a161700 10 _do_op 0xe6f5260 seq 2498 r = 0, finisher 0x7c90ae0 0
2014-06-12 14:46:26.350374 7f9d7a161700 10 _finish_op 0xe6f5260 seq 2498 osr(3.247 0x276d380)/0x276d380
2014-06-12 14:46:26.350399 7f9d7f16b700  5 _do_op 0x6e75e60 seq 2499 osr(3.247 0x276d380)/0x276d380 start
2014-06-12 14:46:26.350406 7f9d7f16b700 10 _do_transaction on 0x25e130b0
2014-06-12 14:46:26.350417 7f9d7f16b700 15 _omap_setkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.350552 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.350577 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.350605 7f9d42e4f700  5 -- op tracker -- , seq: 747825, time: 2014-06-12 14:46:26.348947, event: header_read, op: osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.350617 7f9d42e4f700  5 -- op tracker -- , seq: 747825, time: 2014-06-12 14:46:26.348954, event: throttled, op: osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.350626 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.350626 7f9d42e4f700  5 -- op tracker -- , seq: 747825, time: 2014-06-12 14:46:26.350567, event: all_read, op: osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.350634 7f9d42e4f700  5 -- op tracker -- , seq: 747825, time: 0.000000, event: dispatched, op: osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.350642 7f9d42e4f700 20 osd.2 173 should_share_map client.4313 172.16.96.220:0/1244424 173
2014-06-12 14:46:26.350646 7f9d7f16b700 15 _omap_rmkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.350652 7f9d7f16b700 10 _omap_rmkeys meta/1fe2638a/pglog_3.247/0//-1 r = 0
2014-06-12 14:46:26.350657 7f9d7f16b700 15 _omap_setkeys meta/1fe2638a/pglog_3.247/0//-1
2014-06-12 14:46:26.350672 7f9d42e4f700 15 osd.2 173 enqueue_op 0x2910100 prio 63 cost 1036288 latency 0.001725 osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.350713 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:26.350776 7f9d7f16b700 15 _write 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 3436544~757760
2014-06-12 14:46:26.350915 7f9d7f16b700 10 lookup_strip_header done  cid 3.247_head oid 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.351044 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:26.351635 7f9d7715b700  5 -- op tracker -- , seq: 747822, time: 2014-06-12 14:46:26.351622, event: commit_sent, op: osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.351650 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2727 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:26.351694 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:26.355133 7f9d7f16b700 10 _generic_write 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 3436544~757760 = 0
2014-06-12 14:46:26.355360 7f9d7f16b700 15 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.355399 7f9d7f16b700 10 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 = 0
2014-06-12 14:46:26.355422 7f9d7f16b700 15 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3
2014-06-12 14:46:26.355446 7f9d7f16b700 10 _setattrs 3.247_head/8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 = 0
2014-06-12 14:46:26.355722 7f9d7f16b700 20 set_header setting 17396 cid 3.247_head oid 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 parent seq 0
2014-06-12 14:46:26.355841 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.355874 7f9d7f16b700 20 set_header setting 2250 cid meta oid 1fe2638a/pglog_3.247/0//-1 parent seq 0
2014-06-12 14:46:26.355901 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:26.357687 7f9d7795c700  5 -- op tracker -- , seq: 747822, time: 2014-06-12 14:46:26.357669, event: sub_op_applied, op: osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.357765 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2728 active] sub_op_modify_applied on 0x52c8380 op osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.357791 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2728 active] op_applied on primary on version 173'2729
2014-06-12 14:46:26.357801 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2728 active] op_applied on replica on version 173'2729
2014-06-12 14:46:26.357890 7f9d7a161700  5 -- op tracker -- , seq: 747822, time: 2014-06-12 14:46:26.357879, event: done, op: osd_sub_op(client.4313.0:146453 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.357958 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x543e6c0 (1 active)
2014-06-12 14:46:26.357982 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.358821 7f9d42e4f700  5 -- op tracker -- , seq: 747826, time: 2014-06-12 14:46:26.358171, event: header_read, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.358837 7f9d42e4f700  5 -- op tracker -- , seq: 747826, time: 2014-06-12 14:46:26.358175, event: throttled, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.358846 7f9d42e4f700  5 -- op tracker -- , seq: 747826, time: 2014-06-12 14:46:26.358775, event: all_read, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.358858 7f9d42e4f700  5 -- op tracker -- , seq: 747826, time: 0.000000, event: dispatched, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.358867 7f9d42e4f700 20 osd.2 173 should_share_map client.4313 172.16.96.220:0/1244424 173
2014-06-12 14:46:26.358906 7f9d42e4f700 15 osd.2 173 enqueue_op 0x2910b00 prio 63 cost 516096 latency 0.000734 osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.359185 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active)
2014-06-12 14:46:26.360881 7f9d7f16b700 10 _do_op 0x6e75e60 seq 2499 r = 0, finisher 0x65e44c0 0
2014-06-12 14:46:26.360927 7f9d7f16b700 10 _finish_op 0x6e75e60 seq 2499 osr(3.247 0x276d380)/0x276d380
2014-06-12 14:46:26.360922 7f9d7715b700  5 -- op tracker -- , seq: 747824, time: 2014-06-12 14:46:26.360911, event: commit_sent, op: osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.360945 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x543e6c0 (0 active)
2014-06-12 14:46:26.360936 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2728 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.23
2014-06-12 14:46:26.360957 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.360967 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:26.361034 7f9d7795c700  5 -- op tracker -- , seq: 747824, time: 2014-06-12 14:46:26.361028, event: sub_op_applied, op: osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.361045 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2729 active] sub_op_modify_applied on 0x25e12fc0 op osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.361061 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2729 active] op_applied on primary on version 173'2730
2014-06-12 14:46:26.361069 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.247( v 173'2730 (0'0,173'2730] local-les=145 n=280 ec=23 les/c 145/145 144/144/144) [23,2] r=1 lpr=144 pi=23-143/12 luod=0'0 crt=0'0 lcod 173'2729 active] op_applied on replica on version 173'2730
2014-06-12 14:46:26.361089 7f9d7795c700  5 -- op tracker -- , seq: 747824, time: 2014-06-12 14:46:26.361085, event: done, op: osd_sub_op(client.4313.0:146454 3.247 8db4e247/rbd_data.10b06b8b4567.0000000000000fb7/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.367566 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:26.389033 7f9d6a3e5700 10 getattr 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 '_' = 0
2014-06-12 14:46:26.389078 7f9d6a3e5700 15 getattr 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 'snapset'
2014-06-12 14:46:26.389344 7f9d6a3e5700 10 getattr 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 'snapset' = 0
2014-06-12 14:46:26.389362 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] populate_obc_watchers 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.389384 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.389394 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] get_object_context: creating obc from disk: 0xac04580
2014-06-12 14:46:26.389405 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] get_object_context: 0xac04580 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 rwstate(none n=0 w=0) oi: 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3(173'795 client.4313.0:38268 wrlock_by=unknown.0.0:0 dirty s 4194304 uv795) ssc: 0x81bb340 snapset: 0=[]:[]+head
2014-06-12 14:46:26.389422 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] find_object_context 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 @head oi=4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3(173'795 client.4313.0:38268 wrlock_by=unknown.0.0:0 dirty s 4194304 uv795)
2014-06-12 14:46:26.389454 7f9d6a3e5700  5 -- op tracker -- , seq: 747821, time: 2014-06-12 14:46:26.389446, event: started, op: osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.389463 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] execute_ctx 0x6245600
2014-06-12 14:46:26.389475 7f9d6a3e5700  5 -- op tracker -- , seq: 747821, time: 2014-06-12 14:46:26.389471, event: started, op: osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.389479 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_op 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] ov 173'795 av 173'3030 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:26.389488 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_osd_op 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432]
2014-06-12 14:46:26.389495 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:26.389504 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_osd_op  write 0~274432
2014-06-12 14:46:26.389515 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_osd_op_effects on session 0x1b56400
2014-06-12 14:46:26.389522 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] make_writeable 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 snapset=0x81bb378  snapc=0=[]
2014-06-12 14:46:26.389536 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] make_writeable 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:26.389542 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] finish_ctx 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 0x6245600 op modify  
2014-06-12 14:46:26.389551 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]  set mtime to 2014-06-12 08:45:28.693594
2014-06-12 14:46:26.389571 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]  final snapset 0=[]:[]+head in 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.389590 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]  zeroing write result code 0
2014-06-12 14:46:26.389599 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] calc_trim_to 79'28 -> 79'29
2014-06-12 14:46:26.389608 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] new_repop rep_tid 236260 on osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.389638 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] issue_repop rep_tid 236260 o 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.389664 7f9d6a3e5700  5 -- op tracker -- , seq: 747821, time: 2014-06-12 14:46:26.389659, event: waiting for subops from 27, op: osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.389694 7f9d6a3e5700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:26.389723 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3029 (79'28,173'3029] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] append_log log((79'28,173'3029], crt=0'0) [173'3030 (173'795) modify   4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 by client.4313.0:146455 2014-06-12 08:45:28.693594]
2014-06-12 14:46:26.389748 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'28,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] add_log_entry 173'3030 (173'795) modify   4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 by client.4313.0:146455 2014-06-12 08:45:28.693594
2014-06-12 14:46:26.389768 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'28,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] append_log  adding 1 keys
2014-06-12 14:46:26.389781 7f9d6a3e5700 10 trim log((79'28,173'3030], crt=0'0) to 79'29
2014-06-12 14:46:26.389809 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3030, trimmed: 79'29
2014-06-12 14:46:26.389841 7f9d6a3e5700  5 queue_transactions existing osr(3.1e8 0x276d100)/0x276d100
2014-06-12 14:46:26.389845 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.389847 7f9d6a3e5700  5 queue_op 0x90e56e0 seq 2815 osr(3.1e8 0x276d100) 276760 bytes   (queue has 1 ops and 276760 bytes)
2014-06-12 14:46:26.389855 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] eval_repop repgather(0x3403b800 173'3030 rep_tid=236260 committed?=0 applied?=0 lock=0 op=osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.389875 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2911d00 finish
2014-06-12 14:46:26.389883 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active)
2014-06-12 14:46:26.389888 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.389901 7f9d653db700 10 osd.2 173 dequeue_op 0x2910100 prio 63 cost 1036288 latency 0.040954 osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]
2014-06-12 14:46:26.389925 7f9d653db700  5 -- op tracker -- , seq: 747825, time: 2014-06-12 14:46:26.389920, event: reached_pg, op: osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.389936 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:26.389954 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] handle_message: 0x2910100
2014-06-12 14:46:26.389966 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_op osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:26.389992 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] get_object_context: found obc in cache: 0xac04580
2014-06-12 14:46:26.390004 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] get_object_context: 0xac04580 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 rwstate(write n=1 w=0) oi: 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3(173'3030 client.4313.0:146455 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3030) ssc: 0x81bb340 snapset: 0=[]:[]+head
2014-06-12 14:46:26.390019 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] find_object_context 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 @head oi=4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3(173'3030 client.4313.0:146455 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3030)
2014-06-12 14:46:26.390042 7f9d653db700  5 -- op tracker -- , seq: 747825, time: 2014-06-12 14:46:26.390037, event: started, op: osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.390050 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] execute_ctx 0x2e410000
2014-06-12 14:46:26.390066 7f9d653db700  5 -- op tracker -- , seq: 747825, time: 2014-06-12 14:46:26.390061, event: started, op: osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.390074 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_op 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] ov 173'3030 av 173'3031 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:26.390088 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_osd_op 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288]
2014-06-12 14:46:26.390099 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:26.390117 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_osd_op  write 274432~1036288
2014-06-12 14:46:26.390129 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_osd_op_effects on session 0x1b56400
2014-06-12 14:46:26.390140 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] make_writeable 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 snapset=0x81bb378  snapc=0=[]
2014-06-12 14:46:26.390151 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] make_writeable 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:26.390161 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] finish_ctx 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 0x2e410000 op modify  
2014-06-12 14:46:26.390173 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]  set mtime to 2014-06-12 08:45:28.726384
2014-06-12 14:46:26.390193 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]  final snapset 0=[]:[]+head in 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.390213 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]  zeroing write result code 0
2014-06-12 14:46:26.390224 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] calc_trim_to 79'29 -> 79'30
2014-06-12 14:46:26.390233 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] new_repop rep_tid 236261 on osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.390248 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] issue_repop rep_tid 236261 o 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.390271 7f9d653db700  5 -- op tracker -- , seq: 747825, time: 2014-06-12 14:46:26.390266, event: waiting for subops from 27, op: osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.390291 7f9d653db700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:26.390310 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3030 (79'29,173'3030] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] append_log log((79'29,173'3030], crt=0'0) [173'3031 (173'3030) modify   4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 by client.4313.0:146456 2014-06-12 08:45:28.726384]
2014-06-12 14:46:26.390331 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'29,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] add_log_entry 173'3031 (173'3030) modify   4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 by client.4313.0:146456 2014-06-12 08:45:28.726384
2014-06-12 14:46:26.390358 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'29,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] append_log  adding 1 keys
2014-06-12 14:46:26.390372 7f9d653db700 10 trim log((79'29,173'3031], crt=0'0) to 79'30
2014-06-12 14:46:26.390394 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3031, trimmed: 79'30
2014-06-12 14:46:26.390415 7f9d653db700  5 queue_transactions existing osr(3.1e8 0x276d100)/0x276d100
2014-06-12 14:46:26.390419 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.390420 7f9d653db700  5 queue_op 0x15169aa0 seq 2816 osr(3.1e8 0x276d100) 1038616 bytes   (queue has 2 ops and 1315376 bytes)
2014-06-12 14:46:26.390426 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] eval_repop repgather(0x65909c0 173'3031 rep_tid=236261 committed?=0 applied?=0 lock=0 op=osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.390444 7f9d653db700 10 osd.2 173 dequeue_op 0x2910100 finish
2014-06-12 14:46:26.390451 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:26.390454 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.390452 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1560 (1 active)
2014-06-12 14:46:26.390460 7f9d7e169700  5 _do_op 0x90e56e0 seq 2815 osr(3.1e8 0x276d100)/0x276d100 start
2014-06-12 14:46:26.390465 7f9d7e169700 10 _do_transaction on 0x3403a240
2014-06-12 14:46:26.390470 7f9d7e169700 15 _omap_setkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.390465 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2910b00 prio 63 cost 516096 latency 0.032293 osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]
2014-06-12 14:46:26.390500 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1560 (2 active)
2014-06-12 14:46:26.390491 7f9d63bd8700  5 -- op tracker -- , seq: 747826, time: 2014-06-12 14:46:26.390485, event: reached_pg, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.390513 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:26.390526 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] handle_message: 0x2910b00
2014-06-12 14:46:26.390537 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_op osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:26.390572 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] get_object_context: found obc in cache: 0xac04580
2014-06-12 14:46:26.390586 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.390583 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] get_object_context: 0xac04580 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 rwstate(write n=2 w=0) oi: 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3(173'3031 client.4313.0:146456 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3031) ssc: 0x81bb340 snapset: 0=[]:[]+head
2014-06-12 14:46:26.390607 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.390607 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] find_object_context 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 @head oi=4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3(173'3031 client.4313.0:146456 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3031)
2014-06-12 14:46:26.390654 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.390673 7f9d7e169700 15 _omap_rmkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.390680 7f9d7e169700 10 _omap_rmkeys meta/1fe10005/pglog_3.1e8/0//-1 r = 0
2014-06-12 14:46:26.390685 7f9d7e169700 15 _omap_setkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.390701 7f9d7e169700 15 _write 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 0~274432
2014-06-12 14:46:26.391147 7f9d7e169700 10 lookup_strip_header done  cid 3.1e8_head oid 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.390632 7f9d63bd8700  5 -- op tracker -- , seq: 747826, time: 2014-06-12 14:46:26.390626, event: started, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.391287 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] execute_ctx 0x1fc53200
2014-06-12 14:46:26.391316 7f9d63bd8700  5 -- op tracker -- , seq: 747826, time: 2014-06-12 14:46:26.391309, event: started, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.391323 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_op 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] ov 173'3031 av 173'3032 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:26.391336 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_osd_op 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096]
2014-06-12 14:46:26.391352 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:26.391348 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:26.391361 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_osd_op  write 1310720~516096
2014-06-12 14:46:26.391376 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] do_osd_op_effects on session 0x1b56400
2014-06-12 14:46:26.391386 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] make_writeable 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 snapset=0x81bb378  snapc=0=[]
2014-06-12 14:46:26.391396 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] make_writeable 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:26.391405 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] finish_ctx 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 0x1fc53200 op modify  
2014-06-12 14:46:26.391416 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]  set mtime to 2014-06-12 08:45:28.735915
2014-06-12 14:46:26.391435 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]  final snapset 0=[]:[]+head in 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.391451 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]  zeroing write result code 0
2014-06-12 14:46:26.391460 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] calc_trim_to 79'30 -> 79'31
2014-06-12 14:46:26.391468 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] new_repop rep_tid 236262 on osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.391483 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] issue_repop rep_tid 236262 o 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.391506 7f9d63bd8700  5 -- op tracker -- , seq: 747826, time: 2014-06-12 14:46:26.391501, event: waiting for subops from 27, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.391525 7f9d63bd8700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:26.391545 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3031 (79'30,173'3031] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] append_log log((79'30,173'3031], crt=0'0) [173'3032 (173'3031) modify   4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 by client.4313.0:146457 2014-06-12 08:45:28.735915]
2014-06-12 14:46:26.391566 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'30,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] add_log_entry 173'3032 (173'3031) modify   4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 by client.4313.0:146457 2014-06-12 08:45:28.735915
2014-06-12 14:46:26.391583 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'30,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] append_log  adding 1 keys
2014-06-12 14:46:26.391594 7f9d63bd8700 10 trim log((79'30,173'3032], crt=0'0) to 79'31
2014-06-12 14:46:26.391618 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3032, trimmed: 79'31
2014-06-12 14:46:26.391641 7f9d63bd8700  5 queue_transactions existing osr(3.1e8 0x276d100)/0x276d100
2014-06-12 14:46:26.391645 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.391647 7f9d63bd8700  5 queue_op 0x6a7a7e0 seq 2817 osr(3.1e8 0x276d100) 518424 bytes   (queue has 3 ops and 1833800 bytes)
2014-06-12 14:46:26.391655 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] eval_repop repgather(0x29703980 173'3032 rep_tid=236262 committed?=0 applied?=0 lock=0 op=osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.391673 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2910b00 finish
2014-06-12 14:46:26.391679 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.391683 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.391698 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1560 (3 active)
2014-06-12 14:46:26.392655 7f9d7e169700 10 _generic_write 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 0~274432 = 0
2014-06-12 14:46:26.392691 7f9d7e169700 15 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.392706 7f9d7e169700 10 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:26.392712 7f9d7e169700 15 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.392719 7f9d7e169700 10 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:26.392878 7f9d7e169700 20 set_header setting 17397 cid 3.1e8_head oid 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 parent seq 0
2014-06-12 14:46:26.392928 7f9d7e169700 20 set_header setting 2010 cid meta oid 1fe10005/pglog_3.1e8/0//-1 parent seq 0
2014-06-12 14:46:26.392945 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.392955 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:26.395968 7f9d7e169700 10 _do_op 0x90e56e0 seq 2815 r = 0, finisher 0x858a700 0x1f116c00
2014-06-12 14:46:26.395997 7f9d7e169700 10 _finish_op 0x90e56e0 seq 2815 osr(3.1e8 0x276d100)/0x276d100
2014-06-12 14:46:26.396013 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1560 (2 active)
2014-06-12 14:46:26.396023 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.396004 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] op_commit: 236260
2014-06-12 14:46:26.396044 7f9d7715b700  5 -- op tracker -- , seq: 747821, time: 2014-06-12 14:46:26.396033, event: op_commit, op: osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.396064 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] op_applied: 236260
2014-06-12 14:46:26.396090 7f9d7795c700  5 -- op tracker -- , seq: 747821, time: 2014-06-12 14:46:26.396082, event: op_applied, op: osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.396102 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] op_applied on primary on version 173'3030
2014-06-12 14:46:26.396718 7f9d58c5a700  5 -- op tracker -- , seq: 747827, time: 2014-06-12 14:46:26.396675, event: header_read, op: osd_sub_op_reply(client.4313.0:146455 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.396728 7f9d58c5a700  5 -- op tracker -- , seq: 747827, time: 2014-06-12 14:46:26.396677, event: throttled, op: osd_sub_op_reply(client.4313.0:146455 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.396736 7f9d58c5a700  5 -- op tracker -- , seq: 747827, time: 2014-06-12 14:46:26.396690, event: all_read, op: osd_sub_op_reply(client.4313.0:146455 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.396743 7f9d58c5a700  5 -- op tracker -- , seq: 747827, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4313.0:146455 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.396749 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4313.0:146455 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:26.396754 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:26.396762 7f9d58c5a700 15 osd.2 173 enqueue_op 0x8dbd100 prio 196 cost 0 latency 0.000087 osd_sub_op_reply(client.4313.0:146455 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:26.396812 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.396820 7f9d68be2700 10 osd.2 173 dequeue_op 0x8dbd100 prio 196 cost 0 latency 0.000145 osd_sub_op_reply(client.4313.0:146455 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2 pg pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]
2014-06-12 14:46:26.396841 7f9d68be2700  5 -- op tracker -- , seq: 747827, time: 2014-06-12 14:46:26.396838, event: reached_pg, op: osd_sub_op_reply(client.4313.0:146455 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.396851 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] handle_message: 0x8dbd100
2014-06-12 14:46:26.396867 7f9d68be2700  5 -- op tracker -- , seq: 747827, time: 2014-06-12 14:46:26.396863, event: started, op: osd_sub_op_reply(client.4313.0:146455 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.396872 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] sub_op_modify_reply: tid 236260 op  ack_type 4 from 27
2014-06-12 14:46:26.396893 7f9d68be2700  5 -- op tracker -- , seq: 747821, time: 2014-06-12 14:46:26.396885, event: sub_op_commit_rec, op: osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.396900 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] repop_all_applied: repop tid 236260 all applied 
2014-06-12 14:46:26.396910 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] eval_repop repgather(0x3403b800 173'3030 rep_tid=236260 committed?=0 applied?=1 lock=0 op=osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.396926 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean]  sending ack on repgather(0x3403b800 173'3030 rep_tid=236260 committed?=0 applied?=1 lock=0 op=osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) 0x1f888c80
2014-06-12 14:46:26.396956 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3029 crt=0'0 lcod 173'3028 mlcod 173'3028 active+clean] repop_all_committed: repop tid 236260 all committed 
2014-06-12 14:46:26.396972 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3028 active+clean] eval_repop repgather(0x3403b800 173'3030 rep_tid=236260 committed?=1 applied?=1 lock=0 op=osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.396988 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3028 active+clean] log_op_stats osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 inb 274711 outb 0 rlat 0.083361 lat 0.083394
2014-06-12 14:46:26.397004 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3028 active+clean] publish_stats_to_osd 173:3798
2014-06-12 14:46:26.397017 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3028 active+clean]  sending commit on repgather(0x3403b800 173'3030 rep_tid=236260 committed?=1 applied?=1 lock=0 op=osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) 0x2481c500
2014-06-12 14:46:26.397053 7f9d68be2700  5 -- op tracker -- , seq: 747821, time: 2014-06-12 14:46:26.397046, event: commit_sent, op: osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.397066 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3028 active+clean]  requeue_ops 
2014-06-12 14:46:26.397075 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean]  removing repgather(0x3403b800 173'3030 rep_tid=236260 committed?=1 applied?=1 op=osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.397088 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean]    q front is repgather(0x3403b800 173'3030 rep_tid=236260 committed?=1 applied?=1 op=osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.397101 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean] remove_repop repgather(0x3403b800 173'3030 rep_tid=236260 committed?=1 applied?=1 op=osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.397114 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean]  requeue_ops 
2014-06-12 14:46:26.397133 7f9d68be2700  5 -- op tracker -- , seq: 747821, time: 2014-06-12 14:46:26.397128, event: done, op: osd_op(client.4313.0:146455 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~274432] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.397153 7f9d68be2700 10 osd.2 173 dequeue_op 0x8dbd100 finish
2014-06-12 14:46:26.397160 7f9d68be2700  5 -- op tracker -- , seq: 747827, time: 2014-06-12 14:46:26.397157, event: done, op: osd_sub_op_reply(client.4313.0:146455 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.397170 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.397173 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.397625 7f9d7815d700  5 _do_op 0x15169aa0 seq 2816 osr(3.1e8 0x276d100)/0x276d100 start
2014-06-12 14:46:26.397636 7f9d7815d700 10 _do_transaction on 0x1771dec0
2014-06-12 14:46:26.397646 7f9d7815d700 15 _omap_setkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.397833 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.397860 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.397913 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.397934 7f9d7815d700 15 _omap_rmkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.397942 7f9d7815d700 10 _omap_rmkeys meta/1fe10005/pglog_3.1e8/0//-1 r = 0
2014-06-12 14:46:26.397949 7f9d7815d700 15 _omap_setkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.398088 7f9d7815d700 15 _write 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 274432~1036288
2014-06-12 14:46:26.398235 7f9d7815d700 10 lookup_strip_header done  cid 3.1e8_head oid 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.398494 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:26.404017 7f9d7815d700 10 _generic_write 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 274432~1036288 = 0
2014-06-12 14:46:26.404658 7f9d7815d700 15 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.404743 7f9d7815d700 10 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:26.404771 7f9d7815d700 15 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.404839 7f9d7815d700 10 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:26.406832 7f9d5b6bc700  5 -- op tracker -- , seq: 747828, time: 2014-06-12 14:46:26.399582, event: header_read, op: osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.406855 7f9d5b6bc700  5 -- op tracker -- , seq: 747828, time: 2014-06-12 14:46:26.399585, event: throttled, op: osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.406864 7f9d5b6bc700  5 -- op tracker -- , seq: 747828, time: 2014-06-12 14:46:26.406765, event: all_read, op: osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.406872 7f9d5b6bc700  5 -- op tracker -- , seq: 747828, time: 0.000000, event: dispatched, op: osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.406881 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.406890 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:26.406899 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x34017a00 prio 127 cost 1549095 latency 0.007317 osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.407111 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.407121 7f9d623d5700 10 osd.2 173 dequeue_op 0x34017a00 prio 127 cost 1549095 latency 0.007539 osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active]
2014-06-12 14:46:26.407148 7f9d623d5700  5 -- op tracker -- , seq: 747828, time: 2014-06-12 14:46:26.407144, event: reached_pg, op: osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.407158 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] handle_message: 0x34017a00
2014-06-12 14:46:26.407170 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] sub_op_modify trans 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 v 173'2993 (transaction) 174
2014-06-12 14:46:26.407187 7f9d623d5700  5 -- op tracker -- , seq: 747828, time: 2014-06-12 14:46:26.407183, event: started, op: osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.407209 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2992 (0'0,173'2992] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] append_log log((0'0,173'2992], crt=0'0) [173'2993 (173'2992) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143672 2014-06-12 08:45:28.770003]
2014-06-12 14:46:26.407233 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2993 (0'0,173'2993] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] add_log_entry 173'2993 (173'2992) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143672 2014-06-12 08:45:28.770003
2014-06-12 14:46:26.407257 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2993 (0'0,173'2993] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] append_log  adding 1 keys
2014-06-12 14:46:26.407298 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2993, trimmed: 
2014-06-12 14:46:26.407321 7f9d623d5700  5 -- op tracker -- , seq: 747828, time: 2014-06-12 14:46:26.407316, event: started, op: osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.407334 7f9d623d5700  5 queue_transactions existing osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.407339 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.407341 7f9d623d5700  5 queue_op 0x5c75560 seq 2747 osr(3.322 0x276d4e0) 1550581 bytes   (queue has 3 ops and 3107621 bytes)
2014-06-12 14:46:26.407350 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.322( v 173'2993 (0'0,173'2993] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] do_sub_op osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.407364 7f9d623d5700 10 osd.2 173 dequeue_op 0x34017a00 finish
2014-06-12 14:46:26.407368 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.407370 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.407948 7f9d5b6bc700  5 -- op tracker -- , seq: 747829, time: 2014-06-12 14:46:26.406941, event: header_read, op: osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.407961 7f9d5b6bc700  5 -- op tracker -- , seq: 747829, time: 2014-06-12 14:46:26.406942, event: throttled, op: osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.407970 7f9d5b6bc700  5 -- op tracker -- , seq: 747829, time: 2014-06-12 14:46:26.407903, event: all_read, op: osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.407979 7f9d5b6bc700  5 -- op tracker -- , seq: 747829, time: 0.000000, event: dispatched, op: osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.407985 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.407992 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:26.408002 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x20355300 prio 127 cost 303911 latency 0.001061 osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.408129 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439200 (3 active)
2014-06-12 14:46:26.408138 7f9d82171700  5 _do_op 0x5c75560 seq 2747 osr(3.322 0x276d4e0)/0x276d4e0 start
2014-06-12 14:46:26.408144 7f9d82171700 10 _do_transaction on 0x67920f0
2014-06-12 14:46:26.408151 7f9d82171700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.408203 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.408212 7f9d643d9700 10 osd.2 173 dequeue_op 0x20355300 prio 127 cost 303911 latency 0.001271 osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.322( v 173'2993 (0'0,173'2993] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active]
2014-06-12 14:46:26.408238 7f9d643d9700  5 -- op tracker -- , seq: 747829, time: 2014-06-12 14:46:26.408233, event: reached_pg, op: osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.408248 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2993 (0'0,173'2993] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] handle_message: 0x20355300
2014-06-12 14:46:26.408261 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2993 (0'0,173'2993] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] sub_op_modify trans 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 v 173'2994 (transaction) 174
2014-06-12 14:46:26.408279 7f9d643d9700  5 -- op tracker -- , seq: 747829, time: 2014-06-12 14:46:26.408274, event: started, op: osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.408291 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.408316 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.408303 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2993 (0'0,173'2993] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] append_log log((0'0,173'2993], crt=0'0) [173'2994 (173'2993) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143673 2014-06-12 08:45:28.782942]
2014-06-12 14:46:26.408329 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2994 (0'0,173'2994] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] add_log_entry 173'2994 (173'2993) modify   206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 by client.4289.0:143673 2014-06-12 08:45:28.782942
2014-06-12 14:46:26.408352 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2994 (0'0,173'2994] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] append_log  adding 1 keys
2014-06-12 14:46:26.408387 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2994, trimmed: 
2014-06-12 14:46:26.408410 7f9d643d9700  5 -- op tracker -- , seq: 747829, time: 2014-06-12 14:46:26.408405, event: started, op: osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.408420 7f9d643d9700  5 queue_transactions existing osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.408424 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.408426 7f9d643d9700  5 queue_op 0x177b42a0 seq 2748 osr(3.322 0x276d4e0) 305397 bytes   (queue has 4 ops and 3413018 bytes)
2014-06-12 14:46:26.408435 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.322( v 173'2994 (0'0,173'2994] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] do_sub_op osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.408450 7f9d643d9700 10 osd.2 173 dequeue_op 0x20355300 finish
2014-06-12 14:46:26.408454 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.408456 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.408466 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439200 (4 active)
2014-06-12 14:46:26.408575 7f9d7815d700 20 set_header setting 17397 cid 3.1e8_head oid 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 parent seq 0
2014-06-12 14:46:26.408978 7f9d7815d700 20 set_header setting 2010 cid meta oid 1fe10005/pglog_3.1e8/0//-1 parent seq 0
2014-06-12 14:46:26.409033 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.409064 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:26.414941 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.414996 7f9d82171700 15 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.415004 7f9d82171700 10 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1 r = 0
2014-06-12 14:46:26.415011 7f9d82171700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.415135 7f9d82171700 15 _write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 2342912~1548288
2014-06-12 14:46:26.415298 7f9d82171700 10 lookup_strip_header done  cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.415579 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:26.415758 7f9d58c5a700  5 -- op tracker -- , seq: 747830, time: 2014-06-12 14:46:26.415703, event: header_read, op: osd_sub_op_reply(client.4313.0:146456 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.415771 7f9d58c5a700  5 -- op tracker -- , seq: 747830, time: 2014-06-12 14:46:26.415705, event: throttled, op: osd_sub_op_reply(client.4313.0:146456 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.415779 7f9d58c5a700  5 -- op tracker -- , seq: 747830, time: 2014-06-12 14:46:26.415719, event: all_read, op: osd_sub_op_reply(client.4313.0:146456 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.415787 7f9d58c5a700  5 -- op tracker -- , seq: 747830, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4313.0:146456 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.415792 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4313.0:146456 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:26.415798 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:26.415806 7f9d58c5a700 15 osd.2 173 enqueue_op 0x24c9d000 prio 196 cost 0 latency 0.000103 osd_sub_op_reply(client.4313.0:146456 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:26.415838 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.415845 7f9d633d7700 10 osd.2 173 dequeue_op 0x24c9d000 prio 196 cost 0 latency 0.000142 osd_sub_op_reply(client.4313.0:146456 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2 pg pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean]
2014-06-12 14:46:26.415866 7f9d633d7700  5 -- op tracker -- , seq: 747830, time: 2014-06-12 14:46:26.415863, event: reached_pg, op: osd_sub_op_reply(client.4313.0:146456 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.415875 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean] handle_message: 0x24c9d000
2014-06-12 14:46:26.415891 7f9d633d7700  5 -- op tracker -- , seq: 747830, time: 2014-06-12 14:46:26.415888, event: started, op: osd_sub_op_reply(client.4313.0:146456 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.415898 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean] sub_op_modify_reply: tid 236261 op  ack_type 4 from 27
2014-06-12 14:46:26.415923 7f9d633d7700  5 -- op tracker -- , seq: 747825, time: 2014-06-12 14:46:26.415913, event: sub_op_commit_rec, op: osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.415938 7f9d633d7700 10 osd.2 173 dequeue_op 0x24c9d000 finish
2014-06-12 14:46:26.415946 7f9d633d7700  5 -- op tracker -- , seq: 747830, time: 2014-06-12 14:46:26.415942, event: done, op: osd_sub_op_reply(client.4313.0:146456 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.415967 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.415972 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.417295 7f9d58c5a700  5 -- op tracker -- , seq: 747831, time: 2014-06-12 14:46:26.417233, event: header_read, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ack, result = 0)
2014-06-12 14:46:26.417306 7f9d58c5a700  5 -- op tracker -- , seq: 747831, time: 2014-06-12 14:46:26.417235, event: throttled, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ack, result = 0)
2014-06-12 14:46:26.417314 7f9d58c5a700  5 -- op tracker -- , seq: 747831, time: 2014-06-12 14:46:26.417248, event: all_read, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ack, result = 0)
2014-06-12 14:46:26.417321 7f9d58c5a700  5 -- op tracker -- , seq: 747831, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ack, result = 0)
2014-06-12 14:46:26.417327 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:26.417332 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:26.417341 7f9d58c5a700 15 osd.2 173 enqueue_op 0x24c9d300 prio 196 cost 0 latency 0.000107 osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ack, result = 0) v2
2014-06-12 14:46:26.417376 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.417382 7f9d61bd4700 10 osd.2 173 dequeue_op 0x24c9d300 prio 196 cost 0 latency 0.000149 osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ack, result = 0) v2 pg pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean]
2014-06-12 14:46:26.417402 7f9d61bd4700  5 -- op tracker -- , seq: 747831, time: 2014-06-12 14:46:26.417399, event: reached_pg, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ack, result = 0)
2014-06-12 14:46:26.417411 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean] handle_message: 0x24c9d300
2014-06-12 14:46:26.417426 7f9d61bd4700  5 -- op tracker -- , seq: 747831, time: 2014-06-12 14:46:26.417423, event: started, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ack, result = 0)
2014-06-12 14:46:26.417433 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean] sub_op_modify_reply: tid 236262 op  ack_type 1 from 27
2014-06-12 14:46:26.417455 7f9d61bd4700  5 -- op tracker -- , seq: 747826, time: 2014-06-12 14:46:26.417446, event: sub_op_applied_rec, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.417465 7f9d61bd4700 10 osd.2 173 dequeue_op 0x24c9d300 finish
2014-06-12 14:46:26.417471 7f9d61bd4700  5 -- op tracker -- , seq: 747831, time: 2014-06-12 14:46:26.417468, event: done, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ack, result = 0)
2014-06-12 14:46:26.417489 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.417491 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.418544 7f9d58c5a700  5 -- op tracker -- , seq: 747832, time: 2014-06-12 14:46:26.418482, event: header_read, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.418555 7f9d58c5a700  5 -- op tracker -- , seq: 747832, time: 2014-06-12 14:46:26.418485, event: throttled, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.418563 7f9d58c5a700  5 -- op tracker -- , seq: 747832, time: 2014-06-12 14:46:26.418500, event: all_read, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.418570 7f9d58c5a700  5 -- op tracker -- , seq: 747832, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.418577 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:26.418584 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:26.418593 7f9d58c5a700 15 osd.2 173 enqueue_op 0x17c2a100 prio 196 cost 0 latency 0.000111 osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:26.420880 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.420898 7f9d673df700 10 osd.2 173 dequeue_op 0x17c2a100 prio 196 cost 0 latency 0.002415 osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2 pg pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean]
2014-06-12 14:46:26.420930 7f9d673df700  5 -- op tracker -- , seq: 747832, time: 2014-06-12 14:46:26.420924, event: reached_pg, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.420943 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean] handle_message: 0x17c2a100
2014-06-12 14:46:26.420960 7f9d673df700  5 -- op tracker -- , seq: 747832, time: 2014-06-12 14:46:26.420956, event: started, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.420971 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean] sub_op_modify_reply: tid 236262 op  ack_type 4 from 27
2014-06-12 14:46:26.421002 7f9d673df700  5 -- op tracker -- , seq: 747826, time: 2014-06-12 14:46:26.420992, event: sub_op_commit_rec, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.421010 7f9d673df700 10 osd.2 173 dequeue_op 0x17c2a100 finish
2014-06-12 14:46:26.421017 7f9d673df700  5 -- op tracker -- , seq: 747832, time: 2014-06-12 14:46:26.421014, event: done, op: osd_sub_op_reply(client.4313.0:146457 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.421032 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.421035 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.424275 7f9d82171700 10 _generic_write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 2342912~1548288 = 0
2014-06-12 14:46:26.424686 7f9d82171700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.424709 7f9d82171700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.424715 7f9d82171700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.424725 7f9d82171700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.425391 7f9d82171700 20 set_header setting 17454 cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 parent seq 0
2014-06-12 14:46:26.425584 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.425601 7f9d82171700 20 set_header setting 2186 cid meta oid 1fe2232a/pglog_3.322/0//-1 parent seq 0
2014-06-12 14:46:26.425610 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:26.433414 7f9d7815d700 10 _do_op 0x15169aa0 seq 2816 r = 0, finisher 0xa647f80 0x14d35ec0
2014-06-12 14:46:26.433441 7f9d7815d700 10 _finish_op 0x15169aa0 seq 2816 osr(3.1e8 0x276d100)/0x276d100
2014-06-12 14:46:26.433451 7f9d7e96a700  5 _do_op 0x6a7a7e0 seq 2817 osr(3.1e8 0x276d100)/0x276d100 start
2014-06-12 14:46:26.433457 7f9d7e96a700 10 _do_transaction on 0x7414c00
2014-06-12 14:46:26.433467 7f9d7e96a700 15 _omap_setkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.433554 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean] op_commit: 236261
2014-06-12 14:46:26.433591 7f9d7715b700  5 -- op tracker -- , seq: 747825, time: 2014-06-12 14:46:26.433579, event: op_commit, op: osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.433605 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.433605 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3030 crt=0'0 lcod 173'3029 mlcod 173'3029 active+clean] repop_all_committed: repop tid 236261 all committed 
2014-06-12 14:46:26.433634 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.433625 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3029 active+clean] eval_repop repgather(0x65909c0 173'3031 rep_tid=236261 committed?=1 applied?=0 lock=0 op=osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.433642 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3029 active+clean] log_op_stats osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 inb 1036567 outb 0 rlat 0.000000 lat 0.084693
2014-06-12 14:46:26.433678 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3029 active+clean] publish_stats_to_osd 173:3799
2014-06-12 14:46:26.433690 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3029 active+clean]  sending commit on repgather(0x65909c0 173'3031 rep_tid=236261 committed?=1 applied?=0 lock=0 op=osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) 0x6ce8000
2014-06-12 14:46:26.433731 7f9d7715b700  5 -- op tracker -- , seq: 747825, time: 2014-06-12 14:46:26.433725, event: commit_sent, op: osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.434967 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3029 active+clean] op_applied: 236261
2014-06-12 14:46:26.435003 7f9d7795c700  5 -- op tracker -- , seq: 747825, time: 2014-06-12 14:46:26.434994, event: op_applied, op: osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.435012 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3029 active+clean] op_applied on primary on version 173'3031
2014-06-12 14:46:26.435026 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3029 active+clean] repop_all_applied: repop tid 236261 all applied 
2014-06-12 14:46:26.435037 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3029 active+clean] eval_repop repgather(0x65909c0 173'3031 rep_tid=236261 committed?=1 applied?=1 lock=0 op=osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.435057 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3029 active+clean] log_op_stats osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 inb 1036567 outb 0 rlat 0.000000 lat 0.086109
2014-06-12 14:46:26.435074 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3029 active+clean] publish_stats_to_osd 173:3800
2014-06-12 14:46:26.435087 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3029 active+clean]  requeue_ops 
2014-06-12 14:46:26.435096 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean]  removing repgather(0x65909c0 173'3031 rep_tid=236261 committed?=1 applied?=1 op=osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.435111 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean]    q front is repgather(0x65909c0 173'3031 rep_tid=236261 committed?=1 applied?=1 op=osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.435124 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] remove_repop repgather(0x65909c0 173'3031 rep_tid=236261 committed?=1 applied?=1 op=osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.435139 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean]  requeue_ops 
2014-06-12 14:46:26.435185 7f9d7815d700  5 -- op tracker -- , seq: 747825, time: 2014-06-12 14:46:26.435171, event: done, op: osd_op(client.4313.0:146456 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~1036288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.435226 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1560 (3 active)
2014-06-12 14:46:26.435231 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.447990 7f9d42e4f700  5 -- op tracker -- , seq: 747833, time: 2014-06-12 14:46:26.439402, event: header_read, op: osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.448002 7f9d42e4f700  5 -- op tracker -- , seq: 747833, time: 2014-06-12 14:46:26.439405, event: throttled, op: osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.448008 7f9d42e4f700  5 -- op tracker -- , seq: 747833, time: 2014-06-12 14:46:26.447948, event: all_read, op: osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.448013 7f9d42e4f700  5 -- op tracker -- , seq: 747833, time: 0.000000, event: dispatched, op: osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.448019 7f9d42e4f700 20 osd.2 173 should_share_map client.4313 172.16.96.220:0/1244424 173
2014-06-12 14:46:26.448042 7f9d42e4f700 15 osd.2 173 enqueue_op 0x2910900 prio 63 cost 1548288 latency 0.008639 osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.448068 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.448078 7f9d67be0700 10 osd.2 173 dequeue_op 0x2910900 prio 63 cost 1548288 latency 0.008676 osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean]
2014-06-12 14:46:26.448107 7f9d67be0700  5 -- op tracker -- , seq: 747833, time: 2014-06-12 14:46:26.448102, event: reached_pg, op: osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.448371 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:26.448390 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] handle_message: 0x2910900
2014-06-12 14:46:26.448400 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] do_op osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:26.448424 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] get_object_context: found obc in cache: 0xac04580
2014-06-12 14:46:26.448433 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] get_object_context: 0xac04580 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 rwstate(write n=1 w=0) oi: 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3(173'3032 client.4313.0:146457 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3032) ssc: 0x81bb340 snapset: 0=[]:[]+head
2014-06-12 14:46:26.448448 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] find_object_context 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 @head oi=4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3(173'3032 client.4313.0:146457 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3032)
2014-06-12 14:46:26.448472 7f9d67be0700  5 -- op tracker -- , seq: 747833, time: 2014-06-12 14:46:26.448467, event: started, op: osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.448483 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] execute_ctx 0x2cec1800
2014-06-12 14:46:26.448499 7f9d67be0700  5 -- op tracker -- , seq: 747833, time: 2014-06-12 14:46:26.448494, event: started, op: osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.448504 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] do_op 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] ov 173'3032 av 173'3033 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:26.448896 7f9d42e4f700  5 -- op tracker -- , seq: 747834, time: 2014-06-12 14:46:26.448069, event: header_read, op: osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.448909 7f9d42e4f700  5 -- op tracker -- , seq: 747834, time: 2014-06-12 14:46:26.448071, event: throttled, op: osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.448918 7f9d42e4f700  5 -- op tracker -- , seq: 747834, time: 2014-06-12 14:46:26.448867, event: all_read, op: osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.448927 7f9d42e4f700  5 -- op tracker -- , seq: 747834, time: 0.000000, event: dispatched, op: osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.448934 7f9d42e4f700 20 osd.2 173 should_share_map client.4313 172.16.96.220:0/1244424 173
2014-06-12 14:46:26.448957 7f9d42e4f700 15 osd.2 173 enqueue_op 0x2910500 prio 63 cost 819200 latency 0.000887 osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.448992 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.449048 7f9d82171700 10 _do_op 0x5c75560 seq 2747 r = 0, finisher 0x66881e0 0
2014-06-12 14:46:26.449079 7f9d82171700 10 _finish_op 0x5c75560 seq 2747 osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.449088 7f9d7d167700  5 _do_op 0x177b42a0 seq 2748 osr(3.322 0x276d4e0)/0x276d4e0 start
2014-06-12 14:46:26.449093 7f9d7d167700 10 _do_transaction on 0x6206ef0
2014-06-12 14:46:26.449092 7f9d7715b700  5 -- op tracker -- , seq: 747828, time: 2014-06-12 14:46:26.449083, event: commit_sent, op: osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.449100 7f9d7d167700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.449100 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2994 (0'0,173'2994] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] sub_op_modify_commit on op osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:26.449124 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:26.449163 7f9d7e96a700 15 _omap_rmkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.449178 7f9d7e96a700 10 _omap_rmkeys meta/1fe10005/pglog_3.1e8/0//-1 r = 0
2014-06-12 14:46:26.449185 7f9d7e96a700 15 _omap_setkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.449202 7f9d7e96a700 15 _write 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 1310720~516096
2014-06-12 14:46:26.449217 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:26.448517 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] do_osd_op 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288]
2014-06-12 14:46:26.449239 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:26.449252 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] do_osd_op  write 1826816~1548288
2014-06-12 14:46:26.449266 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] do_osd_op_effects on session 0x1b56400
2014-06-12 14:46:26.449276 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] make_writeable 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 snapset=0x81bb378  snapc=0=[]
2014-06-12 14:46:26.449287 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] make_writeable 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:26.449296 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] finish_ctx 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 0x2cec1800 op modify  
2014-06-12 14:46:26.449307 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean]  set mtime to 2014-06-12 08:45:28.817391
2014-06-12 14:46:26.449321 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.449327 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean]  final snapset 0=[]:[]+head in 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.449345 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.449343 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean]  zeroing write result code 0
2014-06-12 14:46:26.449352 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] calc_trim_to 79'31 -> 79'32
2014-06-12 14:46:26.449361 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] new_repop rep_tid 236263 on osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.449376 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] issue_repop rep_tid 236263 o 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.449398 7f9d67be0700  5 -- op tracker -- , seq: 747833, time: 2014-06-12 14:46:26.449393, event: waiting for subops from 27, op: osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.449418 7f9d67be0700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:26.449425 7f9d7795c700  5 -- op tracker -- , seq: 747828, time: 2014-06-12 14:46:26.449418, event: sub_op_applied, op: osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.449433 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2994 (0'0,173'2994] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2992 active] sub_op_modify_applied on 0x6792000 op osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.449438 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3032 (79'31,173'3032] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] append_log log((79'31,173'3032], crt=0'0) [173'3033 (173'3032) modify   4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 by client.4313.0:146458 2014-06-12 08:45:28.817391]
2014-06-12 14:46:26.449451 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2994 (0'0,173'2994] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2992 active] op_applied on primary on version 173'2993
2014-06-12 14:46:26.449461 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2994 (0'0,173'2994] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2992 active] op_applied on replica on version 173'2993
2014-06-12 14:46:26.449462 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'31,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] add_log_entry 173'3033 (173'3032) modify   4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 by client.4313.0:146458 2014-06-12 08:45:28.817391
2014-06-12 14:46:26.449481 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'31,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] append_log  adding 1 keys
2014-06-12 14:46:26.449492 7f9d67be0700 10 trim log((79'31,173'3033], crt=0'0) to 79'32
2014-06-12 14:46:26.449516 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3033, trimmed: 79'32
2014-06-12 14:46:26.449539 7f9d67be0700  5 queue_transactions existing osr(3.1e8 0x276d100)/0x276d100
2014-06-12 14:46:26.449543 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.449545 7f9d67be0700  5 queue_op 0x88312c0 seq 2818 osr(3.1e8 0x276d100) 1550616 bytes   (queue has 3 ops and 2374437 bytes)
2014-06-12 14:46:26.449714 7f9d7e96a700 10 lookup_strip_header done  cid 3.1e8_head oid 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.449737 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:26.450794 7f9d82171700  5 -- op tracker -- , seq: 747828, time: 2014-06-12 14:46:26.450775, event: done, op: osd_sub_op(client.4289.0:143672 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.450841 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439200 (2 active)
2014-06-12 14:46:26.450846 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.451897 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] eval_repop repgather(0xe833e00 173'3033 rep_tid=236263 committed?=0 applied?=0 lock=0 op=osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.451940 7f9d67be0700 10 osd.2 173 dequeue_op 0x2910900 finish
2014-06-12 14:46:26.451949 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:26.451955 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.451964 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1560 (3 active)
2014-06-12 14:46:26.451975 7f9d663dd700 10 osd.2 173 dequeue_op 0x2910500 prio 63 cost 819200 latency 0.003906 osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean]
2014-06-12 14:46:26.452001 7f9d663dd700  5 -- op tracker -- , seq: 747834, time: 2014-06-12 14:46:26.451995, event: reached_pg, op: osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.452011 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:26.452026 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] handle_message: 0x2910500
2014-06-12 14:46:26.452037 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] do_op osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:26.452061 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] get_object_context: found obc in cache: 0xac04580
2014-06-12 14:46:26.452072 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] get_object_context: 0xac04580 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 rwstate(write n=2 w=0) oi: 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3(173'3033 client.4313.0:146458 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3033) ssc: 0x81bb340 snapset: 0=[]:[]+head
2014-06-12 14:46:26.452087 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] find_object_context 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 @head oi=4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3(173'3033 client.4313.0:146458 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3033)
2014-06-12 14:46:26.452109 7f9d663dd700  5 -- op tracker -- , seq: 747834, time: 2014-06-12 14:46:26.452104, event: started, op: osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.452116 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] execute_ctx 0x27bc1200
2014-06-12 14:46:26.452133 7f9d663dd700  5 -- op tracker -- , seq: 747834, time: 2014-06-12 14:46:26.452128, event: started, op: osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.452140 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] do_op 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] ov 173'3033 av 173'3034 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:26.452409 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] do_osd_op 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200]
2014-06-12 14:46:26.452423 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:26.452435 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] do_osd_op  write 3375104~819200
2014-06-12 14:46:26.452448 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] do_osd_op_effects on session 0x1b56400
2014-06-12 14:46:26.452462 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] make_writeable 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 snapset=0x81bb378  snapc=0=[]
2014-06-12 14:46:26.452476 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] make_writeable 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:26.452490 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] finish_ctx 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 0x27bc1200 op modify  
2014-06-12 14:46:26.452502 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean]  set mtime to 2014-06-12 08:45:28.820737
2014-06-12 14:46:26.452529 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean]  final snapset 0=[]:[]+head in 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.452549 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean]  zeroing write result code 0
2014-06-12 14:46:26.452560 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] calc_trim_to 79'32 -> 79'33
2014-06-12 14:46:26.452570 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] new_repop rep_tid 236264 on osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.452606 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] issue_repop rep_tid 236264 o 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.452639 7f9d663dd700  5 -- op tracker -- , seq: 747834, time: 2014-06-12 14:46:26.452632, event: waiting for subops from 27, op: osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.452687 7f9d663dd700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:26.452714 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3033 (79'32,173'3033] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] append_log log((79'32,173'3033], crt=0'0) [173'3034 (173'3033) modify   4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 by client.4313.0:146459 2014-06-12 08:45:28.820737]
2014-06-12 14:46:26.452746 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'32,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] add_log_entry 173'3034 (173'3033) modify   4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 by client.4313.0:146459 2014-06-12 08:45:28.820737
2014-06-12 14:46:26.452777 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'32,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] append_log  adding 1 keys
2014-06-12 14:46:26.452794 7f9d663dd700 10 trim log((79'32,173'3034], crt=0'0) to 79'33
2014-06-12 14:46:26.452824 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3034, trimmed: 79'33
2014-06-12 14:46:26.452850 7f9d663dd700  5 queue_transactions existing osr(3.1e8 0x276d100)/0x276d100
2014-06-12 14:46:26.452855 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.452856 7f9d663dd700  5 queue_op 0x69b2420 seq 2819 osr(3.1e8 0x276d100) 821528 bytes   (queue has 4 ops and 3195965 bytes)
2014-06-12 14:46:26.452870 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d1560 (4 active)
2014-06-12 14:46:26.452865 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] eval_repop repgather(0x15e563c0 173'3034 rep_tid=236264 committed?=0 applied?=0 lock=0 op=osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.452884 7f9d663dd700 10 osd.2 173 dequeue_op 0x2910500 finish
2014-06-12 14:46:26.452889 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.452893 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.453267 7f9d7e96a700 10 _generic_write 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 1310720~516096 = 0
2014-06-12 14:46:26.453331 7f9d7e96a700 15 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.453346 7f9d7e96a700 10 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:26.453352 7f9d7e96a700 15 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.453361 7f9d7e96a700 10 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:26.453572 7f9d7e96a700 20 set_header setting 17397 cid 3.1e8_head oid 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 parent seq 0
2014-06-12 14:46:26.455306 7f9d7e96a700 20 set_header setting 2010 cid meta oid 1fe10005/pglog_3.1e8/0//-1 parent seq 0
2014-06-12 14:46:26.455335 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.455347 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:26.460313 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.460394 7f9d7d167700 15 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.460424 7f9d7d167700 10 _omap_rmkeys meta/1fe2232a/pglog_3.322/0//-1 r = 0
2014-06-12 14:46:26.460449 7f9d7d167700 15 _omap_setkeys meta/1fe2232a/pglog_3.322/0//-1
2014-06-12 14:46:26.460485 7f9d7d167700 15 _write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 3891200~303104
2014-06-12 14:46:26.460492 7f9d7e96a700 10 _do_op 0x6a7a7e0 seq 2817 r = 0, finisher 0x9fdf640 0x28ead880
2014-06-12 14:46:26.460514 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] op_commit: 236262
2014-06-12 14:46:26.460552 7f9d7715b700  5 -- op tracker -- , seq: 747826, time: 2014-06-12 14:46:26.460541, event: op_commit, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.460562 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3031 crt=0'0 lcod 173'3030 mlcod 173'3030 active+clean] repop_all_committed: repop tid 236262 all committed 
2014-06-12 14:46:26.460579 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3030 active+clean] eval_repop repgather(0x29703980 173'3032 rep_tid=236262 committed?=1 applied?=0 lock=0 op=osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.460597 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3030 active+clean] log_op_stats osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.102424
2014-06-12 14:46:26.460615 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3030 active+clean] publish_stats_to_osd 173:3801
2014-06-12 14:46:26.460627 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3030 active+clean]  sending commit on repgather(0x29703980 173'3032 rep_tid=236262 committed?=1 applied?=0 lock=0 op=osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) 0x19e0af00
2014-06-12 14:46:26.460668 7f9d7715b700  5 -- op tracker -- , seq: 747826, time: 2014-06-12 14:46:26.460662, event: commit_sent, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.460803 7f9d7e96a700 10 _finish_op 0x6a7a7e0 seq 2817 osr(3.1e8 0x276d100)/0x276d100
2014-06-12 14:46:26.460826 7f9d7d167700 10 lookup_strip_header done  cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.460830 7f9d8116f700  5 _do_op 0x88312c0 seq 2818 osr(3.1e8 0x276d100)/0x276d100 start
2014-06-12 14:46:26.460835 7f9d8116f700 10 _do_transaction on 0x2a37a240
2014-06-12 14:46:26.460841 7f9d8116f700 15 _omap_setkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.460921 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.460942 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.460977 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:26.461074 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3030 active+clean] op_applied: 236262
2014-06-12 14:46:26.461097 7f9d7795c700  5 -- op tracker -- , seq: 747826, time: 2014-06-12 14:46:26.461091, event: op_applied, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.461105 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3030 active+clean] op_applied on primary on version 173'3032
2014-06-12 14:46:26.461116 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3030 active+clean] repop_all_applied: repop tid 236262 all applied 
2014-06-12 14:46:26.461126 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3030 active+clean] eval_repop repgather(0x29703980 173'3032 rep_tid=236262 committed?=1 applied?=1 lock=0 op=osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.461141 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3030 active+clean] log_op_stats osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.102970
2014-06-12 14:46:26.461159 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3030 active+clean] publish_stats_to_osd 173:3802
2014-06-12 14:46:26.461171 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3030 active+clean]  requeue_ops 
2014-06-12 14:46:26.461181 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3031 active+clean]  removing repgather(0x29703980 173'3032 rep_tid=236262 committed?=1 applied?=1 op=osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.461195 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3031 active+clean]    q front is repgather(0x29703980 173'3032 rep_tid=236262 committed?=1 applied?=1 op=osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.461210 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3031 active+clean] remove_repop repgather(0x29703980 173'3032 rep_tid=236262 committed?=1 applied?=1 op=osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.461223 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3031 active+clean]  requeue_ops 
2014-06-12 14:46:26.461444 7f9d7e96a700  5 -- op tracker -- , seq: 747826, time: 2014-06-12 14:46:26.461375, event: done, op: osd_op(client.4313.0:146457 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1310720~516096] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.461501 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1560 (3 active)
2014-06-12 14:46:26.461524 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.462115 7f9d7d167700 10 _generic_write 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 3891200~303104 = 0
2014-06-12 14:46:26.462156 7f9d7d167700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.462173 7f9d7d167700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.462182 7f9d7d167700 15 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3
2014-06-12 14:46:26.462192 7f9d7d167700 10 _setattrs 3.322_head/206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 = 0
2014-06-12 14:46:26.462310 7f9d56432700  5 -- op tracker -- , seq: 747835, time: 2014-06-12 14:46:26.462259, event: header_read, op: osd_sub_op_reply(client.4317.0:143790 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.462322 7f9d56432700  5 -- op tracker -- , seq: 747835, time: 2014-06-12 14:46:26.462262, event: throttled, op: osd_sub_op_reply(client.4317.0:143790 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.462335 7f9d56432700  5 -- op tracker -- , seq: 747835, time: 2014-06-12 14:46:26.462275, event: all_read, op: osd_sub_op_reply(client.4317.0:143790 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.462342 7f9d56432700  5 -- op tracker -- , seq: 747835, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143790 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.462348 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143790 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:26.462351 7f9d7d167700 20 set_header setting 17454 cid 3.322_head oid 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 parent seq 0
2014-06-12 14:46:26.462354 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:26.462362 7f9d56432700 15 osd.2 173 enqueue_op 0x20e89800 prio 196 cost 0 latency 0.000103 osd_sub_op_reply(client.4317.0:143790 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:26.462397 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.462400 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.462415 7f9d7d167700 20 set_header setting 2186 cid meta oid 1fe2232a/pglog_3.322/0//-1 parent seq 0
2014-06-12 14:46:26.462405 7f9d693e3700 10 osd.2 173 dequeue_op 0x20e89800 prio 196 cost 0 latency 0.000145 osd_sub_op_reply(client.4317.0:143790 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2 pg pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2506 active+clean]
2014-06-12 14:46:26.462424 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:26.462426 7f9d693e3700  5 -- op tracker -- , seq: 747835, time: 2014-06-12 14:46:26.462423, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143790 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.462435 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2506 active+clean] handle_message: 0x20e89800
2014-06-12 14:46:26.462450 7f9d693e3700  5 -- op tracker -- , seq: 747835, time: 2014-06-12 14:46:26.462447, event: started, op: osd_sub_op_reply(client.4317.0:143790 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.462455 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2506 active+clean] sub_op_modify_reply: tid 236160 op  ack_type 4 from 26
2014-06-12 14:46:26.462477 7f9d693e3700  5 -- op tracker -- , seq: 747374, time: 2014-06-12 14:46:26.462468, event: sub_op_commit_rec, op: osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.462485 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2506 active+clean] repop_all_applied: repop tid 236160 all applied 
2014-06-12 14:46:26.462497 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2506 active+clean] eval_repop repgather(0x31b643c0 173'2508 rep_tid=236160 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.462512 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2506 active+clean]  sending ack on repgather(0x31b643c0 173'2508 rep_tid=236160 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) 0x1b775900
2014-06-12 14:46:26.462545 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2507 crt=0'0 lcod 173'2506 mlcod 173'2506 active+clean] repop_all_committed: repop tid 236160 all committed 
2014-06-12 14:46:26.462557 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2506 active+clean] eval_repop repgather(0x31b643c0 173'2508 rep_tid=236160 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.462575 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2506 active+clean] log_op_stats osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 29.692180 lat 29.692210
2014-06-12 14:46:26.462595 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2506 active+clean] publish_stats_to_osd 173:2878
2014-06-12 14:46:26.462612 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2506 active+clean]  sending commit on repgather(0x31b643c0 173'2508 rep_tid=236160 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) 0x30cccf00
2014-06-12 14:46:26.462642 7f9d693e3700  5 -- op tracker -- , seq: 747374, time: 2014-06-12 14:46:26.462637, event: commit_sent, op: osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.462651 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2506 active+clean]  requeue_ops 
2014-06-12 14:46:26.462662 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2507 active+clean]  removing repgather(0x31b643c0 173'2508 rep_tid=236160 committed?=1 applied?=1 op=osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.462677 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2507 active+clean]    q front is repgather(0x31b643c0 173'2508 rep_tid=236160 committed?=1 applied?=1 op=osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.462694 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2507 active+clean] remove_repop repgather(0x31b643c0 173'2508 rep_tid=236160 committed?=1 applied?=1 op=osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.462709 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2507 active+clean]  requeue_ops 
2014-06-12 14:46:26.462727 7f9d693e3700  5 -- op tracker -- , seq: 747374, time: 2014-06-12 14:46:26.462722, event: done, op: osd_op(client.4317.0:143790 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 421888~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.462746 7f9d693e3700 10 osd.2 173 dequeue_op 0x20e89800 finish
2014-06-12 14:46:26.462756 7f9d693e3700  5 -- op tracker -- , seq: 747835, time: 2014-06-12 14:46:26.462751, event: done, op: osd_sub_op_reply(client.4317.0:143790 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.462766 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.462769 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.463781 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.463792 7f9d7d167700 10 _do_op 0x177b42a0 seq 2748 r = 0, finisher 0x76af660 0
2014-06-12 14:46:26.463806 7f9d7d167700 10 _finish_op 0x177b42a0 seq 2748 osr(3.322 0x276d4e0)/0x276d4e0
2014-06-12 14:46:26.463813 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439200 (2 active)
2014-06-12 14:46:26.463817 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.463817 7f9d8116f700 15 _omap_rmkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.463825 7f9d8116f700 10 _omap_rmkeys meta/1fe10005/pglog_3.1e8/0//-1 r = 0
2014-06-12 14:46:26.463832 7f9d8116f700 15 _omap_setkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.463934 7f9d8116f700 15 _write 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 1826816~1548288
2014-06-12 14:46:26.464097 7f9d8116f700 10 lookup_strip_header done  cid 3.1e8_head oid 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.464227 7f9d7715b700  5 -- op tracker -- , seq: 747829, time: 2014-06-12 14:46:26.464218, event: commit_sent, op: osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.464237 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2994 (0'0,173'2994] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2992 active] sub_op_modify_commit on op osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:26.464261 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:26.464374 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:26.470278 7f9d7795c700  5 -- op tracker -- , seq: 747829, time: 2014-06-12 14:46:26.470259, event: sub_op_applied, op: osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.470296 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2994 (0'0,173'2994] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2993 active] sub_op_modify_applied on 0x6206e00 op osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.470323 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2994 (0'0,173'2994] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2993 active] op_applied on primary on version 173'2994
2014-06-12 14:46:26.470333 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.322( v 173'2994 (0'0,173'2994] local-les=157 n=304 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2993 active] op_applied on replica on version 173'2994
2014-06-12 14:46:26.470360 7f9d7795c700  5 -- op tracker -- , seq: 747829, time: 2014-06-12 14:46:26.470356, event: done, op: osd_sub_op(client.4289.0:143673 3.322 206a3322/rbd_data.10986b8b4567.0000000000001134/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.474229 7f9d58c5a700  5 -- op tracker -- , seq: 747836, time: 2014-06-12 14:46:26.474186, event: header_read, op: osd_sub_op_reply(client.4313.0:146458 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.474240 7f9d58c5a700  5 -- op tracker -- , seq: 747836, time: 2014-06-12 14:46:26.474188, event: throttled, op: osd_sub_op_reply(client.4313.0:146458 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.474248 7f9d58c5a700  5 -- op tracker -- , seq: 747836, time: 2014-06-12 14:46:26.474198, event: all_read, op: osd_sub_op_reply(client.4313.0:146458 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.474255 7f9d58c5a700  5 -- op tracker -- , seq: 747836, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4313.0:146458 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.474260 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4313.0:146458 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:26.474265 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:26.474272 7f9d58c5a700 15 osd.2 173 enqueue_op 0x598fa00 prio 196 cost 0 latency 0.000086 osd_sub_op_reply(client.4313.0:146458 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:26.474305 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.474313 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x598fa00 prio 196 cost 0 latency 0.000126 osd_sub_op_reply(client.4313.0:146458 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2 pg pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3031 active+clean]
2014-06-12 14:46:26.474332 7f9d6b3e7700  5 -- op tracker -- , seq: 747836, time: 2014-06-12 14:46:26.474329, event: reached_pg, op: osd_sub_op_reply(client.4313.0:146458 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.474340 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3031 active+clean] handle_message: 0x598fa00
2014-06-12 14:46:26.474355 7f9d6b3e7700  5 -- op tracker -- , seq: 747836, time: 2014-06-12 14:46:26.474352, event: started, op: osd_sub_op_reply(client.4313.0:146458 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.474360 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3031 active+clean] sub_op_modify_reply: tid 236263 op  ack_type 4 from 27
2014-06-12 14:46:26.474381 7f9d6b3e7700  5 -- op tracker -- , seq: 747833, time: 2014-06-12 14:46:26.474373, event: sub_op_commit_rec, op: osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.474389 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x598fa00 finish
2014-06-12 14:46:26.474395 7f9d6b3e7700  5 -- op tracker -- , seq: 747836, time: 2014-06-12 14:46:26.474392, event: done, op: osd_sub_op_reply(client.4313.0:146458 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.474408 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.474411 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.475758 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:26.475814 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 12.52 >= max 0.5 = no, load too high
2014-06-12 14:46:26.475828 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:26.475831 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92634 < max (604800 seconds)
2014-06-12 14:46:26.475844 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:26.475847 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:26.475849 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:26.475851 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 6; oldest is 24.990061 seconds old
2014-06-12 14:46:26.475864 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:26.490719 7f9d8116f700 10 _generic_write 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 1826816~1548288 = 0
2014-06-12 14:46:26.491023 7f9d8116f700 15 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.491132 7f9d8116f700 10 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:26.491169 7f9d8116f700 15 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.491202 7f9d8116f700 10 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:26.491820 7f9d8116f700 20 set_header setting 17397 cid 3.1e8_head oid 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 parent seq 0
2014-06-12 14:46:26.492095 7f9d8116f700 20 set_header setting 2010 cid meta oid 1fe10005/pglog_3.1e8/0//-1 parent seq 0
2014-06-12 14:46:26.492136 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.492167 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:26.500568 7f9d8116f700 10 _do_op 0x88312c0 seq 2818 r = 0, finisher 0x8331880 0x32ff0b40
2014-06-12 14:46:26.500687 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3031 active+clean] op_commit: 236263
2014-06-12 14:46:26.500720 7f9d7715b700  5 -- op tracker -- , seq: 747833, time: 2014-06-12 14:46:26.500711, event: op_commit, op: osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.500731 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3032 crt=0'0 lcod 173'3031 mlcod 173'3031 active+clean] repop_all_committed: repop tid 236263 all committed 
2014-06-12 14:46:26.500745 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean] eval_repop repgather(0xe833e00 173'3033 rep_tid=236263 committed?=1 applied?=0 lock=0 op=osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.500764 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean] log_op_stats osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 0.000000 lat 0.061361
2014-06-12 14:46:26.500783 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean] publish_stats_to_osd 173:3803
2014-06-12 14:46:26.500795 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean]  sending commit on repgather(0xe833e00 173'3033 rep_tid=236263 committed?=1 applied?=0 lock=0 op=osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) 0x3b82bb80
2014-06-12 14:46:26.500840 7f9d7715b700  5 -- op tracker -- , seq: 747833, time: 2014-06-12 14:46:26.500834, event: commit_sent, op: osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.500854 7f9d8116f700 10 _finish_op 0x88312c0 seq 2818 osr(3.1e8 0x276d100)/0x276d100
2014-06-12 14:46:26.500862 7f9d79960700  5 _do_op 0x69b2420 seq 2819 osr(3.1e8 0x276d100)/0x276d100 start
2014-06-12 14:46:26.500866 7f9d79960700 10 _do_transaction on 0x3a096a80
2014-06-12 14:46:26.500874 7f9d79960700 15 _omap_setkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.500925 7f9d58c5a700  5 -- op tracker -- , seq: 747837, time: 2014-06-12 14:46:26.500873, event: header_read, op: osd_sub_op_reply(client.4313.0:146459 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.500938 7f9d58c5a700  5 -- op tracker -- , seq: 747837, time: 2014-06-12 14:46:26.500875, event: throttled, op: osd_sub_op_reply(client.4313.0:146459 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.500946 7f9d58c5a700  5 -- op tracker -- , seq: 747837, time: 2014-06-12 14:46:26.500895, event: all_read, op: osd_sub_op_reply(client.4313.0:146459 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.500962 7f9d58c5a700  5 -- op tracker -- , seq: 747837, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4313.0:146459 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.500967 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4313.0:146459 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:26.500973 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:26.500981 7f9d58c5a700 15 osd.2 173 enqueue_op 0x8dbcb00 prio 196 cost 0 latency 0.000108 osd_sub_op_reply(client.4313.0:146459 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:26.501008 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.501016 7f9d62bd6700 10 osd.2 173 dequeue_op 0x8dbcb00 prio 196 cost 0 latency 0.000143 osd_sub_op_reply(client.4313.0:146459 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0) v2 pg pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean]
2014-06-12 14:46:26.501036 7f9d62bd6700  5 -- op tracker -- , seq: 747837, time: 2014-06-12 14:46:26.501032, event: reached_pg, op: osd_sub_op_reply(client.4313.0:146459 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.501045 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean] handle_message: 0x8dbcb00
2014-06-12 14:46:26.501061 7f9d62bd6700  5 -- op tracker -- , seq: 747837, time: 2014-06-12 14:46:26.501057, event: started, op: osd_sub_op_reply(client.4313.0:146459 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.501067 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean] sub_op_modify_reply: tid 236264 op  ack_type 4 from 27
2014-06-12 14:46:26.501090 7f9d62bd6700  5 -- op tracker -- , seq: 747834, time: 2014-06-12 14:46:26.501083, event: sub_op_commit_rec, op: osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.501100 7f9d62bd6700 10 osd.2 173 dequeue_op 0x8dbcb00 finish
2014-06-12 14:46:26.501108 7f9d62bd6700  5 -- op tracker -- , seq: 747837, time: 2014-06-12 14:46:26.501104, event: done, op: osd_sub_op_reply(client.4313.0:146459 3.1e8 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.501127 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.501133 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.501149 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.501168 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.501216 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.501322 7f9d79960700 15 _omap_rmkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.501333 7f9d79960700 10 _omap_rmkeys meta/1fe10005/pglog_3.1e8/0//-1 r = 0
2014-06-12 14:46:26.501341 7f9d79960700 15 _omap_setkeys meta/1fe10005/pglog_3.1e8/0//-1
2014-06-12 14:46:26.501360 7f9d79960700 15 _write 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 3375104~819200
2014-06-12 14:46:26.501468 7f9d79960700 10 lookup_strip_header done  cid 3.1e8_head oid 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.501607 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:26.503908 7f9d79960700 10 _generic_write 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 3375104~819200 = 0
2014-06-12 14:46:26.503985 7f9d79960700 15 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.504004 7f9d79960700 10 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:26.504013 7f9d79960700 15 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:26.504021 7f9d79960700 10 _setattrs 3.1e8_head/4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:26.504251 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1560 (1 active)
2014-06-12 14:46:26.504257 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean] op_applied: 236263
2014-06-12 14:46:26.504287 7f9d7795c700  5 -- op tracker -- , seq: 747833, time: 2014-06-12 14:46:26.504277, event: op_applied, op: osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.504296 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean] op_applied on primary on version 173'3033
2014-06-12 14:46:26.504312 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean] repop_all_applied: repop tid 236263 all applied 
2014-06-12 14:46:26.504307 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.504330 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean] eval_repop repgather(0xe833e00 173'3033 rep_tid=236263 committed?=1 applied?=1 lock=0 op=osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.504348 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean] log_op_stats osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 0.000000 lat 0.064945
2014-06-12 14:46:26.504371 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean] publish_stats_to_osd 173:3804
2014-06-12 14:46:26.504386 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3031 active+clean]  requeue_ops 
2014-06-12 14:46:26.504401 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3032 active+clean]  removing repgather(0xe833e00 173'3033 rep_tid=236263 committed?=1 applied?=1 op=osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.504421 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3032 active+clean]    q front is repgather(0xe833e00 173'3033 rep_tid=236263 committed?=1 applied?=1 op=osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.504435 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3032 active+clean] remove_repop repgather(0xe833e00 173'3033 rep_tid=236263 committed?=1 applied?=1 op=osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.504447 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3032 active+clean]  requeue_ops 
2014-06-12 14:46:26.504468 7f9d7795c700  5 -- op tracker -- , seq: 747833, time: 2014-06-12 14:46:26.504463, event: done, op: osd_op(client.4313.0:146458 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1826816~1548288] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.504547 7f9d79960700 20 set_header setting 17397 cid 3.1e8_head oid 4b6d69e8/rbd_data.10b06b8b4567.0000000000000fb8/head//3 parent seq 0
2014-06-12 14:46:26.504636 7f9d79960700 20 set_header setting 2010 cid meta oid 1fe10005/pglog_3.1e8/0//-1 parent seq 0
2014-06-12 14:46:26.504654 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.504668 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:26.508127 7f9d79960700 10 _do_op 0x69b2420 seq 2819 r = 0, finisher 0x13bb7280 0x2db2ec40
2014-06-12 14:46:26.508165 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3032 active+clean] op_commit: 236264
2014-06-12 14:46:26.508214 7f9d7715b700  5 -- op tracker -- , seq: 747834, time: 2014-06-12 14:46:26.508202, event: op_commit, op: osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.508235 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'3033 crt=0'0 lcod 173'3032 mlcod 173'3032 active+clean] repop_all_committed: repop tid 236264 all committed 
2014-06-12 14:46:26.508250 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3032 active+clean] eval_repop repgather(0x15e563c0 173'3034 rep_tid=236264 committed?=1 applied?=0 lock=0 op=osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.508271 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3032 active+clean] log_op_stats osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 inb 819479 outb 0 rlat 0.000000 lat 0.060200
2014-06-12 14:46:26.508295 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3032 active+clean] publish_stats_to_osd 173:3805
2014-06-12 14:46:26.508311 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3032 active+clean]  sending commit on repgather(0x15e563c0 173'3034 rep_tid=236264 committed?=1 applied?=0 lock=0 op=osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) 0x30ccc780
2014-06-12 14:46:26.508355 7f9d7715b700  5 -- op tracker -- , seq: 747834, time: 2014-06-12 14:46:26.508350, event: commit_sent, op: osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.508371 7f9d79960700 10 _finish_op 0x69b2420 seq 2819 osr(3.1e8 0x276d100)/0x276d100
2014-06-12 14:46:26.508387 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d1560 (0 active)
2014-06-12 14:46:26.508398 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.508389 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3032 active+clean] op_applied: 236264
2014-06-12 14:46:26.508416 7f9d7795c700  5 -- op tracker -- , seq: 747834, time: 2014-06-12 14:46:26.508410, event: op_applied, op: osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.508423 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3032 active+clean] op_applied on primary on version 173'3034
2014-06-12 14:46:26.508430 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3032 active+clean] repop_all_applied: repop tid 236264 all applied 
2014-06-12 14:46:26.508434 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3032 active+clean] eval_repop repgather(0x15e563c0 173'3034 rep_tid=236264 committed?=1 applied?=1 lock=0 op=osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.508443 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3032 active+clean] log_op_stats osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4 inb 819479 outb 0 rlat 0.000000 lat 0.060374
2014-06-12 14:46:26.508452 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3032 active+clean] publish_stats_to_osd 173:3806
2014-06-12 14:46:26.508459 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3032 active+clean]  requeue_ops 
2014-06-12 14:46:26.508465 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3033 active+clean]  removing repgather(0x15e563c0 173'3034 rep_tid=236264 committed?=1 applied?=1 op=osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.508474 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3033 active+clean]    q front is repgather(0x15e563c0 173'3034 rep_tid=236264 committed?=1 applied?=1 op=osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.508482 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3033 active+clean] remove_repop repgather(0x15e563c0 173'3034 rep_tid=236264 committed?=1 applied?=1 op=osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.508490 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3033 active+clean]  requeue_ops 
2014-06-12 14:46:26.508510 7f9d7795c700  5 -- op tracker -- , seq: 747834, time: 2014-06-12 14:46:26.508507, event: done, op: osd_op(client.4313.0:146459 rbd_data.10b06b8b4567.0000000000000fb8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3375104~819200] 3.4b6d69e8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.559928 7f9d56432700  5 -- op tracker -- , seq: 747838, time: 2014-06-12 14:46:26.551612, event: header_read, op: osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.559941 7f9d56432700  5 -- op tracker -- , seq: 747838, time: 2014-06-12 14:46:26.551618, event: throttled, op: osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.559956 7f9d56432700  5 -- op tracker -- , seq: 747838, time: 2014-06-12 14:46:26.559880, event: all_read, op: osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.559961 7f9d56432700  5 -- op tracker -- , seq: 747838, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.559965 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.559970 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:26.559976 7f9d56432700 15 osd.2 173 enqueue_op 0x20e88100 prio 127 cost 2065191 latency 0.008364 osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.560031 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.560043 7f9d683e1700 10 osd.2 173 dequeue_op 0x20e88100 prio 127 cost 2065191 latency 0.008430 osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.67d( v 173'2512 (0'0,173'2512] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2511 active]
2014-06-12 14:46:26.560065 7f9d683e1700  5 -- op tracker -- , seq: 747838, time: 2014-06-12 14:46:26.560061, event: reached_pg, op: osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.560074 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2512 (0'0,173'2512] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2511 active] handle_message: 0x20e88100
2014-06-12 14:46:26.560084 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2512 (0'0,173'2512] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2511 active] sub_op_modify trans 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 v 173'2513 (transaction) 174
2014-06-12 14:46:26.560099 7f9d683e1700  5 -- op tracker -- , seq: 747838, time: 2014-06-12 14:46:26.560095, event: started, op: osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.560114 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2512 (0'0,173'2512] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2511 active] append_log log((0'0,173'2512], crt=0'0) [173'2513 (173'2511) modify   6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 by client.4303.0:144817 2014-06-12 08:45:21.222985]
2014-06-12 14:46:26.560143 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2513 (0'0,173'2513] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2511 active] add_log_entry 173'2513 (173'2511) modify   6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 by client.4303.0:144817 2014-06-12 08:45:21.222985
2014-06-12 14:46:26.560163 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2513 (0'0,173'2513] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2511 active] append_log  adding 1 keys
2014-06-12 14:46:26.560195 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2513, trimmed: 
2014-06-12 14:46:26.560217 7f9d683e1700  5 -- op tracker -- , seq: 747838, time: 2014-06-12 14:46:26.560213, event: started, op: osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.560226 7f9d683e1700  5 queue_transactions existing osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:26.560230 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.560231 7f9d683e1700  5 queue_op 0x14bc6ea0 seq 2291 osr(3.67d 0x276df70) 2066677 bytes   (queue has 1 ops and 2066677 bytes)
2014-06-12 14:46:26.560239 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.67d( v 173'2513 (0'0,173'2513] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2511 active] do_sub_op osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.560251 7f9d683e1700 10 osd.2 173 dequeue_op 0x20e88100 finish
2014-06-12 14:46:26.560254 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.560256 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.560263 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x524cb40 (1 active)
2014-06-12 14:46:26.560267 7f9d8096e700  5 _do_op 0x14bc6ea0 seq 2291 osr(3.67d 0x276df70)/0x276df70 start
2014-06-12 14:46:26.560272 7f9d8096e700 10 _do_transaction on 0x61e6b70
2014-06-12 14:46:26.560277 7f9d8096e700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:26.560408 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:26.560426 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.560476 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.560491 7f9d8096e700 15 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:26.560496 7f9d8096e700 10 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1 r = 0
2014-06-12 14:46:26.560501 7f9d8096e700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:26.560512 7f9d8096e700 15 _write 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 2084864~2064384
2014-06-12 14:46:26.560644 7f9d8096e700 10 lookup_strip_header done  cid 3.67d_head oid 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:26.560760 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:26.575118 7f9d8096e700 10 _generic_write 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 2084864~2064384 = 0
2014-06-12 14:46:26.575459 7f9d8096e700 15 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:26.575498 7f9d8096e700 10 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 = 0
2014-06-12 14:46:26.575525 7f9d8096e700 15 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:26.575559 7f9d8096e700 10 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 = 0
2014-06-12 14:46:26.576312 7f9d8096e700 20 set_header setting 17507 cid 3.67d_head oid 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 parent seq 0
2014-06-12 14:46:26.576669 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.576709 7f9d8096e700 20 set_header setting 2272 cid meta oid 1fe0e10b/pglog_3.67d/0//-1 parent seq 0
2014-06-12 14:46:26.576739 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:26.586753 7f9d8096e700 10 _do_op 0x14bc6ea0 seq 2291 r = 0, finisher 0xb754a60 0
2014-06-12 14:46:26.586821 7f9d8096e700 10 _finish_op 0x14bc6ea0 seq 2291 osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:26.586842 7f9d7715b700  5 -- op tracker -- , seq: 747838, time: 2014-06-12 14:46:26.586829, event: commit_sent, op: osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.586853 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x524cb40 (0 active)
2014-06-12 14:46:26.586857 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2513 (0'0,173'2513] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2511 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:26.586884 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.586895 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:26.586931 7f9d7795c700  5 -- op tracker -- , seq: 747838, time: 2014-06-12 14:46:26.586925, event: sub_op_applied, op: osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.586941 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2513 (0'0,173'2513] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active] sub_op_modify_applied on 0x61e6a80 op osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.586961 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2513 (0'0,173'2513] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active] op_applied on primary on version 173'2513
2014-06-12 14:46:26.586973 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2513 (0'0,173'2513] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active] op_applied on replica on version 173'2513
2014-06-12 14:46:26.587004 7f9d7795c700  5 -- op tracker -- , seq: 747838, time: 2014-06-12 14:46:26.586999, event: done, op: osd_sub_op(client.4303.0:144817 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.617160 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:26.617189 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:26.620261 7f9d56432700  5 -- op tracker -- , seq: 747839, time: 2014-06-12 14:46:26.619323, event: header_read, op: osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.620276 7f9d56432700  5 -- op tracker -- , seq: 747839, time: 2014-06-12 14:46:26.619325, event: throttled, op: osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.620284 7f9d56432700  5 -- op tracker -- , seq: 747839, time: 2014-06-12 14:46:26.620213, event: all_read, op: osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.620292 7f9d56432700  5 -- op tracker -- , seq: 747839, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.620297 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.620303 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:26.620312 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3eb900 prio 127 cost 516903 latency 0.000988 osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.622834 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.622850 7f9d65bdc700 10 osd.2 173 dequeue_op 0x1e3eb900 prio 127 cost 516903 latency 0.003526 osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2997 (0'0,173'2997] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2996 active]
2014-06-12 14:46:26.622884 7f9d65bdc700  5 -- op tracker -- , seq: 747839, time: 2014-06-12 14:46:26.622880, event: reached_pg, op: osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.622895 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2997 (0'0,173'2997] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2996 active] handle_message: 0x1e3eb900
2014-06-12 14:46:26.622905 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2997 (0'0,173'2997] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2996 active] sub_op_modify trans 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 v 173'2998 (transaction) 174
2014-06-12 14:46:26.622919 7f9d65bdc700  5 -- op tracker -- , seq: 747839, time: 2014-06-12 14:46:26.622916, event: started, op: osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.622941 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2997 (0'0,173'2997] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2996 active] append_log log((0'0,173'2997], crt=0'0) [173'2998 (173'2997) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145019 2014-06-12 08:44:48.310976]
2014-06-12 14:46:26.622966 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2998 (0'0,173'2998] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2996 active] add_log_entry 173'2998 (173'2997) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145019 2014-06-12 08:44:48.310976
2014-06-12 14:46:26.622987 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2998 (0'0,173'2998] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2996 active] append_log  adding 1 keys
2014-06-12 14:46:26.623019 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2998, trimmed: 
2014-06-12 14:46:26.623042 7f9d65bdc700  5 -- op tracker -- , seq: 747839, time: 2014-06-12 14:46:26.623037, event: started, op: osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.623052 7f9d65bdc700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:26.623055 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.623057 7f9d65bdc700  5 queue_op 0xfa42a80 seq 2800 osr(3.458 0x276d960) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:26.623064 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2998 (0'0,173'2998] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2996 active] do_sub_op osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.623077 7f9d65bdc700 10 osd.2 173 dequeue_op 0x1e3eb900 finish
2014-06-12 14:46:26.623080 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.623082 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.623541 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:26.623553 7f9d8016d700  5 _do_op 0xfa42a80 seq 2800 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:26.623558 7f9d8016d700 10 _do_transaction on 0x62eb0b0
2014-06-12 14:46:26.623564 7f9d8016d700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:26.623953 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:26.623978 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.624035 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.624057 7f9d8016d700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:26.624062 7f9d8016d700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:26.624069 7f9d8016d700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:26.624086 7f9d8016d700 15 _write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 3555328~516096
2014-06-12 14:46:26.624252 7f9d8016d700 10 lookup_strip_header done  cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:26.624652 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:26.627521 7f9d8016d700 10 _generic_write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 3555328~516096 = 0
2014-06-12 14:46:26.627582 7f9d8016d700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:26.627603 7f9d8016d700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:26.627609 7f9d8016d700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:26.627616 7f9d8016d700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:26.627835 7f9d8016d700 20 set_header setting 16936 cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 parent seq 0
2014-06-12 14:46:26.627915 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.627929 7f9d8016d700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:26.627938 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:26.634544 7f9d8016d700 10 _do_op 0xfa42a80 seq 2800 r = 0, finisher 0x66b0ae0 0
2014-06-12 14:46:26.634586 7f9d7715b700  5 -- op tracker -- , seq: 747839, time: 2014-06-12 14:46:26.634577, event: commit_sent, op: osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.634597 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2998 (0'0,173'2998] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2996 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:26.634623 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:26.634648 7f9d8016d700 10 _finish_op 0xfa42a80 seq 2800 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:26.634669 7f9d7795c700  5 -- op tracker -- , seq: 747839, time: 2014-06-12 14:46:26.634663, event: sub_op_applied, op: osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.634676 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2998 (0'0,173'2998] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2997 active] sub_op_modify_applied on 0x62eafc0 op osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.634691 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2998 (0'0,173'2998] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2997 active] op_applied on primary on version 173'2998
2014-06-12 14:46:26.634699 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2998 (0'0,173'2998] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2997 active] op_applied on replica on version 173'2998
2014-06-12 14:46:26.634726 7f9d8016d700  5 -- op tracker -- , seq: 747839, time: 2014-06-12 14:46:26.634720, event: done, op: osd_sub_op(client.4308.0:145019 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2998 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.634760 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:26.634763 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.673731 7f9d56432700  5 -- op tracker -- , seq: 747840, time: 2014-06-12 14:46:26.673686, event: header_read, op: osd_sub_op_reply(client.4317.0:143791 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.673741 7f9d56432700  5 -- op tracker -- , seq: 747840, time: 2014-06-12 14:46:26.673688, event: throttled, op: osd_sub_op_reply(client.4317.0:143791 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.673749 7f9d56432700  5 -- op tracker -- , seq: 747840, time: 2014-06-12 14:46:26.673697, event: all_read, op: osd_sub_op_reply(client.4317.0:143791 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.673757 7f9d56432700  5 -- op tracker -- , seq: 747840, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143791 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.673761 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143791 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:26.673767 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:26.673775 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3ebb00 prio 196 cost 0 latency 0.000088 osd_sub_op_reply(client.4317.0:143791 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:26.673802 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.673810 7f9d64bda700 10 osd.2 173 dequeue_op 0x1e3ebb00 prio 196 cost 0 latency 0.000124 osd_sub_op_reply(client.4317.0:143791 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2 pg pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2507 active+clean]
2014-06-12 14:46:26.673831 7f9d64bda700  5 -- op tracker -- , seq: 747840, time: 2014-06-12 14:46:26.673827, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143791 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.673847 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2507 active+clean] handle_message: 0x1e3ebb00
2014-06-12 14:46:26.673864 7f9d64bda700  5 -- op tracker -- , seq: 747840, time: 2014-06-12 14:46:26.673860, event: started, op: osd_sub_op_reply(client.4317.0:143791 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.673871 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2507 active+clean] sub_op_modify_reply: tid 236184 op  ack_type 4 from 26
2014-06-12 14:46:26.673897 7f9d64bda700  5 -- op tracker -- , seq: 747473, time: 2014-06-12 14:46:26.673886, event: sub_op_commit_rec, op: osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.673906 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2507 active+clean] repop_all_applied: repop tid 236184 all applied 
2014-06-12 14:46:26.673917 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2507 active+clean] eval_repop repgather(0x523c9c0 173'2509 rep_tid=236184 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.673934 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2507 active+clean]  sending ack on repgather(0x523c9c0 173'2509 rep_tid=236184 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) 0x258d0a00
2014-06-12 14:46:26.673969 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2508 crt=0'0 lcod 173'2507 mlcod 173'2507 active+clean] repop_all_committed: repop tid 236184 all committed 
2014-06-12 14:46:26.673981 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2507 active+clean] eval_repop repgather(0x523c9c0 173'2509 rep_tid=236184 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.674001 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2507 active+clean] log_op_stats osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 25.188177 lat 25.188209
2014-06-12 14:46:26.674021 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2507 active+clean] publish_stats_to_osd 173:2879
2014-06-12 14:46:26.674036 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2507 active+clean]  sending commit on repgather(0x523c9c0 173'2509 rep_tid=236184 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) 0x30f6ec80
2014-06-12 14:46:26.674068 7f9d64bda700  5 -- op tracker -- , seq: 747473, time: 2014-06-12 14:46:26.674063, event: commit_sent, op: osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.674076 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2507 active+clean]  requeue_ops 
2014-06-12 14:46:26.674089 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean]  removing repgather(0x523c9c0 173'2509 rep_tid=236184 committed?=1 applied?=1 op=osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.674105 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean]    q front is repgather(0x523c9c0 173'2509 rep_tid=236184 committed?=1 applied?=1 op=osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.674118 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean] remove_repop repgather(0x523c9c0 173'2509 rep_tid=236184 committed?=1 applied?=1 op=osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.674132 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean]  requeue_ops 
2014-06-12 14:46:26.674153 7f9d64bda700  5 -- op tracker -- , seq: 747473, time: 2014-06-12 14:46:26.674148, event: done, op: osd_op(client.4317.0:143791 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 1454080~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.674176 7f9d64bda700 10 osd.2 173 dequeue_op 0x1e3ebb00 finish
2014-06-12 14:46:26.674187 7f9d64bda700  5 -- op tracker -- , seq: 747840, time: 2014-06-12 14:46:26.674183, event: done, op: osd_sub_op_reply(client.4317.0:143791 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.674197 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.674200 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.744363 7f9d44ece700  5 -- op tracker -- , seq: 747841, time: 2014-06-12 14:46:26.744281, event: header_read, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.744384 7f9d44ece700  5 -- op tracker -- , seq: 747841, time: 2014-06-12 14:46:26.744284, event: throttled, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.744394 7f9d44ece700  5 -- op tracker -- , seq: 747841, time: 2014-06-12 14:46:26.744306, event: all_read, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.744404 7f9d44ece700  5 -- op tracker -- , seq: 747841, time: 0.000000, event: dispatched, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.744413 7f9d44ece700 20 osd.2 173 should_share_map client.4311 172.16.96.220:0/1244240 173
2014-06-12 14:46:26.744442 7f9d44ece700 15 osd.2 173 enqueue_op 0x5ee9f00 prio 63 cost 4096 latency 0.000160 osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.744480 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.744489 7f9d6abe6700 10 osd.2 173 dequeue_op 0x5ee9f00 prio 63 cost 4096 latency 0.000207 osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean]
2014-06-12 14:46:26.744516 7f9d6abe6700  5 -- op tracker -- , seq: 747841, time: 2014-06-12 14:46:26.744510, event: reached_pg, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.744525 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:26.744540 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] handle_message: 0x5ee9f00
2014-06-12 14:46:26.744551 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] do_op osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:26.744583 7f9d6abe6700 15 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 '_'
2014-06-12 14:46:26.756490 7f9d6abe6700 10 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 '_' = 0
2014-06-12 14:46:26.756520 7f9d6abe6700 15 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 'snapset'
2014-06-12 14:46:26.756726 7f9d6abe6700 10 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 'snapset' = 0
2014-06-12 14:46:26.756737 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] populate_obc_watchers aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.756753 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.756761 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] get_object_context: creating obc from disk: 0x14655b80
2014-06-12 14:46:26.756770 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] get_object_context: 0x14655b80 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 rwstate(none n=0 w=0) oi: aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3(173'630 client.4311.0:35556 wrlock_by=unknown.0.0:0 dirty s 4194304 uv630) ssc: 0x93442a0 snapset: 0=[]:[]+head
2014-06-12 14:46:26.756784 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] find_object_context aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 @head oi=aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3(173'630 client.4311.0:35556 wrlock_by=unknown.0.0:0 dirty s 4194304 uv630)
2014-06-12 14:46:26.756810 7f9d6abe6700  5 -- op tracker -- , seq: 747841, time: 2014-06-12 14:46:26.756803, event: started, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.756820 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] execute_ctx 0xcfcf600
2014-06-12 14:46:26.756835 7f9d6abe6700  5 -- op tracker -- , seq: 747841, time: 2014-06-12 14:46:26.756831, event: started, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.756842 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] do_op aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] ov 173'630 av 173'2735 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:26.756854 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] do_osd_op aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096]
2014-06-12 14:46:26.756864 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:26.756876 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] do_osd_op  write 0~4096
2014-06-12 14:46:26.756890 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] do_osd_op_effects on session 0x5a1d800
2014-06-12 14:46:26.756900 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] make_writeable aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 snapset=0x93442d8  snapc=0=[]
2014-06-12 14:46:26.756910 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] make_writeable aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:26.756919 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] finish_ctx aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 0xcfcf600 op modify  
2014-06-12 14:46:26.756929 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean]  set mtime to 2014-06-12 08:45:29.125065
2014-06-12 14:46:26.756947 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean]  final snapset 0=[]:[]+head in aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.756964 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean]  zeroing write result code 0
2014-06-12 14:46:26.756973 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] new_repop rep_tid 236265 on osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.756986 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] issue_repop rep_tid 236265 o aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.757010 7f9d6abe6700  5 -- op tracker -- , seq: 747841, time: 2014-06-12 14:46:26.757005, event: waiting for subops from 25, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.757037 7f9d6abe6700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:26.757125 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2734 (0'0,173'2734] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] append_log log((0'0,173'2734], crt=0'0) [173'2735 (173'630) modify   aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 by client.4311.0:139831 2014-06-12 08:45:29.125065]
2014-06-12 14:46:26.757149 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] add_log_entry 173'2735 (173'630) modify   aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 by client.4311.0:139831 2014-06-12 08:45:29.125065
2014-06-12 14:46:26.757172 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] append_log  adding 1 keys
2014-06-12 14:46:26.757204 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2735, trimmed: 
2014-06-12 14:46:26.757230 7f9d6abe6700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:26.757235 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.757237 7f9d6abe6700  5 queue_op 0xdf16c00 seq 2547 osr(3.122 0x276cf40) 6389 bytes   (queue has 1 ops and 6389 bytes)
2014-06-12 14:46:26.757252 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:46:26.757247 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] eval_repop repgather(0x7415bc0 173'2735 rep_tid=236265 committed?=0 applied?=0 lock=0 op=osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.757262 7f9d7915f700  5 _do_op 0xdf16c00 seq 2547 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:46:26.757267 7f9d7915f700 10 _do_transaction on 0x2237ec0
2014-06-12 14:46:26.757267 7f9d6abe6700 10 osd.2 173 dequeue_op 0x5ee9f00 finish
2014-06-12 14:46:26.757272 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.757273 7f9d7915f700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:26.757277 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.757405 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:26.757424 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.757488 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.757511 7f9d7915f700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:26.757518 7f9d7915f700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:46:26.757524 7f9d7915f700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:26.757539 7f9d7915f700 15 _write 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 0~4096
2014-06-12 14:46:26.757694 7f9d7915f700 10 lookup_strip_header done  cid 3.122_head oid aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.757837 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:26.757869 7f9d7915f700 10 _generic_write 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 0~4096 = 0
2014-06-12 14:46:26.757883 7f9d7915f700 15 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.757894 7f9d7915f700 10 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:26.757900 7f9d7915f700 15 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.757908 7f9d7915f700 10 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:26.758021 7f9d7915f700 20 set_header setting 16189 cid 3.122_head oid aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 parent seq 0
2014-06-12 14:46:26.758047 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.758062 7f9d7915f700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:46:26.758076 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:26.758958 7f9d7915f700 10 _do_op 0xdf16c00 seq 2547 r = 0, finisher 0x63ce500 0x2a164840
2014-06-12 14:46:26.758995 7f9d7915f700 10 _finish_op 0xdf16c00 seq 2547 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:26.759013 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:46:26.759021 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.759081 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] op_applied: 236265
2014-06-12 14:46:26.759131 7f9d7795c700  5 -- op tracker -- , seq: 747841, time: 2014-06-12 14:46:26.759115, event: op_applied, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.759142 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] op_applied on primary on version 173'2735
2014-06-12 14:46:26.759173 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] op_commit: 236265
2014-06-12 14:46:26.759198 7f9d7715b700  5 -- op tracker -- , seq: 747841, time: 2014-06-12 14:46:26.759192, event: op_commit, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.763954 7f9d5bec4700  5 -- op tracker -- , seq: 747842, time: 2014-06-12 14:46:26.763895, event: header_read, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:46:26.763966 7f9d5bec4700  5 -- op tracker -- , seq: 747842, time: 2014-06-12 14:46:26.763897, event: throttled, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:46:26.763978 7f9d5bec4700  5 -- op tracker -- , seq: 747842, time: 2014-06-12 14:46:26.763913, event: all_read, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:46:26.763985 7f9d5bec4700  5 -- op tracker -- , seq: 747842, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:46:26.763991 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:26.764006 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:26.764015 7f9d5bec4700 15 osd.2 173 enqueue_op 0x12bf7300 prio 196 cost 0 latency 0.000119 osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ack, result = 0) v2
2014-06-12 14:46:26.764048 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.764058 7f9d66bde700 10 osd.2 173 dequeue_op 0x12bf7300 prio 196 cost 0 latency 0.000163 osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ack, result = 0) v2 pg pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean]
2014-06-12 14:46:26.764081 7f9d66bde700  5 -- op tracker -- , seq: 747842, time: 2014-06-12 14:46:26.764077, event: reached_pg, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:46:26.764090 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] handle_message: 0x12bf7300
2014-06-12 14:46:26.764107 7f9d66bde700  5 -- op tracker -- , seq: 747842, time: 2014-06-12 14:46:26.764103, event: started, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:46:26.764116 7f9d5bec4700  5 -- op tracker -- , seq: 747843, time: 2014-06-12 14:46:26.764042, event: header_read, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.764114 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] sub_op_modify_reply: tid 236265 op  ack_type 1 from 25
2014-06-12 14:46:26.764125 7f9d5bec4700  5 -- op tracker -- , seq: 747843, time: 2014-06-12 14:46:26.764042, event: throttled, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.764134 7f9d5bec4700  5 -- op tracker -- , seq: 747843, time: 2014-06-12 14:46:26.764096, event: all_read, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.764142 7f9d5bec4700  5 -- op tracker -- , seq: 747843, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.764147 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:26.764151 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:26.764156 7f9d5bec4700 15 osd.2 173 enqueue_op 0x12bf7f00 prio 196 cost 0 latency 0.000114 osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:26.764157 7f9d66bde700  5 -- op tracker -- , seq: 747841, time: 2014-06-12 14:46:26.764126, event: sub_op_applied_rec, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.764166 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] repop_all_applied: repop tid 236265 all applied 
2014-06-12 14:46:26.764177 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:26.764178 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] eval_repop repgather(0x7415bc0 173'2735 rep_tid=236265 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.764201 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean]  sending ack on repgather(0x7415bc0 173'2735 rep_tid=236265 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) 0x11666f00
2014-06-12 14:46:26.764237 7f9d66bde700 10 osd.2 173 dequeue_op 0x12bf7300 finish
2014-06-12 14:46:26.764247 7f9d66bde700  5 -- op tracker -- , seq: 747842, time: 2014-06-12 14:46:26.764244, event: done, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:46:26.764246 7f9d69be4700 10 osd.2 173 dequeue_op 0x12bf7f00 prio 196 cost 0 latency 0.000204 osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean]
2014-06-12 14:46:26.764264 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:26.764268 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.764266 7f9d69be4700  5 -- op tracker -- , seq: 747843, time: 2014-06-12 14:46:26.764262, event: reached_pg, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.764275 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] handle_message: 0x12bf7f00
2014-06-12 14:46:26.764289 7f9d69be4700  5 -- op tracker -- , seq: 747843, time: 2014-06-12 14:46:26.764286, event: started, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.764294 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] sub_op_modify_reply: tid 236265 op  ack_type 4 from 25
2014-06-12 14:46:26.764310 7f9d69be4700  5 -- op tracker -- , seq: 747841, time: 2014-06-12 14:46:26.764304, event: sub_op_commit_rec, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.764315 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2734 crt=0'0 lcod 173'2733 mlcod 173'2733 active+clean] repop_all_committed: repop tid 236265 all committed 
2014-06-12 14:46:26.764324 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2733 active+clean] eval_repop repgather(0x7415bc0 173'2735 rep_tid=236265 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.764343 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2733 active+clean] log_op_stats osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 inb 4375 outb 0 rlat 0.019954 lat 0.020061
2014-06-12 14:46:26.764359 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2733 active+clean] publish_stats_to_osd 173:3633
2014-06-12 14:46:26.764371 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2733 active+clean]  sending commit on repgather(0x7415bc0 173'2735 rep_tid=236265 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) 0x10096c80
2014-06-12 14:46:26.764404 7f9d69be4700  5 -- op tracker -- , seq: 747841, time: 2014-06-12 14:46:26.764399, event: commit_sent, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.764412 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2733 active+clean]  requeue_ops 
2014-06-12 14:46:26.764421 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean]  removing repgather(0x7415bc0 173'2735 rep_tid=236265 committed?=1 applied?=1 op=osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.764435 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean]    q front is repgather(0x7415bc0 173'2735 rep_tid=236265 committed?=1 applied?=1 op=osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.764448 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] remove_repop repgather(0x7415bc0 173'2735 rep_tid=236265 committed?=1 applied?=1 op=osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.764461 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean]  requeue_ops 
2014-06-12 14:46:26.764488 7f9d69be4700  5 -- op tracker -- , seq: 747841, time: 2014-06-12 14:46:26.764482, event: done, op: osd_op(client.4311.0:139831 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4096] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.764509 7f9d69be4700 10 osd.2 173 dequeue_op 0x12bf7f00 finish
2014-06-12 14:46:26.764517 7f9d69be4700  5 -- op tracker -- , seq: 747843, time: 2014-06-12 14:46:26.764514, event: done, op: osd_sub_op_reply(client.4311.0:139831 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.764527 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.764530 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.777906 7f9d44ece700  5 -- op tracker -- , seq: 747844, time: 2014-06-12 14:46:26.774677, event: header_read, op: osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.777916 7f9d44ece700  5 -- op tracker -- , seq: 747844, time: 2014-06-12 14:46:26.774680, event: throttled, op: osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.777923 7f9d44ece700  5 -- op tracker -- , seq: 747844, time: 2014-06-12 14:46:26.777869, event: all_read, op: osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.777928 7f9d44ece700  5 -- op tracker -- , seq: 747844, time: 0.000000, event: dispatched, op: osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.777935 7f9d44ece700 20 osd.2 173 should_share_map client.4311 172.16.96.220:0/1244240 173
2014-06-12 14:46:26.777959 7f9d44ece700 15 osd.2 173 enqueue_op 0x1702fe00 prio 63 cost 1032192 latency 0.003282 osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.777987 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.777997 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x1702fe00 prio 63 cost 1032192 latency 0.003320 osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean]
2014-06-12 14:46:26.778026 7f9d6a3e5700  5 -- op tracker -- , seq: 747844, time: 2014-06-12 14:46:26.778020, event: reached_pg, op: osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.778037 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:26.778054 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] handle_message: 0x1702fe00
2014-06-12 14:46:26.778068 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] do_op osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:26.778094 7f9d6a3e5700 15 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 '_'
2014-06-12 14:46:26.778245 7f9d6a3e5700 10 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 '_' = 0
2014-06-12 14:46:26.778263 7f9d6a3e5700 15 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 'snapset'
2014-06-12 14:46:26.778328 7f9d6a3e5700 10 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 'snapset' = 0
2014-06-12 14:46:26.778339 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] populate_obc_watchers aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.778353 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.778365 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] get_object_context: creating obc from disk: 0x28013b80
2014-06-12 14:46:26.778375 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] get_object_context: 0x28013b80 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 rwstate(none n=0 w=0) oi: aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3(173'2735 client.4311.0:139831 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2735) ssc: 0x283d2ee0 snapset: 0=[]:[]+head
2014-06-12 14:46:26.778392 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] find_object_context aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 @head oi=aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3(173'2735 client.4311.0:139831 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2735)
2014-06-12 14:46:26.778422 7f9d6a3e5700  5 -- op tracker -- , seq: 747844, time: 2014-06-12 14:46:26.778415, event: started, op: osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.778429 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] execute_ctx 0x5e4ca00
2014-06-12 14:46:26.778448 7f9d6a3e5700  5 -- op tracker -- , seq: 747844, time: 2014-06-12 14:46:26.778443, event: started, op: osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.778456 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] do_op aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] ov 173'2735 av 173'2736 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:26.778473 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] do_osd_op aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192]
2014-06-12 14:46:26.778486 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:26.778501 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] do_osd_op  write 4096~1032192
2014-06-12 14:46:26.778518 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] do_osd_op_effects on session 0x5a1d800
2014-06-12 14:46:26.778531 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] make_writeable aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 snapset=0x283d2f18  snapc=0=[]
2014-06-12 14:46:26.778545 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] make_writeable aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:26.778558 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] finish_ctx aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 0x5e4ca00 op modify  
2014-06-12 14:46:26.778571 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean]  set mtime to 2014-06-12 08:45:29.153607
2014-06-12 14:46:26.778592 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean]  final snapset 0=[]:[]+head in aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.778619 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean]  zeroing write result code 0
2014-06-12 14:46:26.778629 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] new_repop rep_tid 236266 on osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.778646 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] issue_repop rep_tid 236266 o aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.778674 7f9d6a3e5700  5 -- op tracker -- , seq: 747844, time: 2014-06-12 14:46:26.778669, event: waiting for subops from 25, op: osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.778695 7f9d6a3e5700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:26.778718 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2735 (0'0,173'2735] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] append_log log((0'0,173'2735], crt=0'0) [173'2736 (173'2735) modify   aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 by client.4311.0:139832 2014-06-12 08:45:29.153607]
2014-06-12 14:46:26.778743 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2735 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] add_log_entry 173'2736 (173'2735) modify   aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 by client.4311.0:139832 2014-06-12 08:45:29.153607
2014-06-12 14:46:26.778761 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2735 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] append_log  adding 1 keys
2014-06-12 14:46:26.778790 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2736, trimmed: 
2014-06-12 14:46:26.778813 7f9d6a3e5700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:26.778817 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.778819 7f9d6a3e5700  5 queue_op 0x8caaa20 seq 2548 osr(3.122 0x276cf40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:26.778829 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:46:26.778837 7f9d7d968700  5 _do_op 0x8caaa20 seq 2548 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:46:26.778826 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2735 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] eval_repop repgather(0x7414480 173'2736 rep_tid=236266 committed?=0 applied?=0 lock=0 op=osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.778841 7f9d7d968700 10 _do_transaction on 0x52f5d40
2014-06-12 14:46:26.778842 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x1702fe00 finish
2014-06-12 14:46:26.778845 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.778847 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.778846 7f9d7d968700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:26.778938 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:26.778953 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.779000 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.779016 7f9d7d968700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:26.779021 7f9d7d968700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:46:26.779026 7f9d7d968700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:26.779039 7f9d7d968700 15 _write 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 4096~1032192
2014-06-12 14:46:26.779164 7f9d7d968700 10 lookup_strip_header done  cid 3.122_head oid aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.779433 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:26.783782 7f9d7d968700 10 _generic_write 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 4096~1032192 = 0
2014-06-12 14:46:26.783875 7f9d7d968700 15 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.783894 7f9d7d968700 10 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:26.783902 7f9d7d968700 15 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.783910 7f9d7d968700 10 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:26.784215 7f9d7d968700 20 set_header setting 16189 cid 3.122_head oid aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 parent seq 0
2014-06-12 14:46:26.784365 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.784381 7f9d7d968700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:46:26.784723 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:26.789703 7f9d7d968700 10 _do_op 0x8caaa20 seq 2548 r = 0, finisher 0xa015be0 0x33a4a40
2014-06-12 14:46:26.789722 7f9d7d968700 10 _finish_op 0x8caaa20 seq 2548 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:26.789730 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:46:26.789740 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.789739 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2735 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] op_applied: 236266
2014-06-12 14:46:26.789772 7f9d7795c700  5 -- op tracker -- , seq: 747844, time: 2014-06-12 14:46:26.789762, event: op_applied, op: osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.789783 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2735 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] op_applied on primary on version 173'2736
2014-06-12 14:46:26.790056 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2735 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] op_commit: 236266
2014-06-12 14:46:26.790074 7f9d7715b700  5 -- op tracker -- , seq: 747844, time: 2014-06-12 14:46:26.790070, event: op_commit, op: osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.804222 7f9d56432700  5 -- op tracker -- , seq: 747845, time: 2014-06-12 14:46:26.801525, event: header_read, op: osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.804236 7f9d56432700  5 -- op tracker -- , seq: 747845, time: 2014-06-12 14:46:26.801528, event: throttled, op: osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.804245 7f9d56432700  5 -- op tracker -- , seq: 747845, time: 2014-06-12 14:46:26.804169, event: all_read, op: osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.804253 7f9d56432700  5 -- op tracker -- , seq: 747845, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.804258 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.804264 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:26.804272 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3eb200 prio 127 cost 1553191 latency 0.002747 osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.804300 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.804309 7f9d653db700 10 osd.2 173 dequeue_op 0x1e3eb200 prio 127 cost 1553191 latency 0.002784 osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.67d( v 173'2513 (0'0,173'2513] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active]
2014-06-12 14:46:26.804337 7f9d653db700  5 -- op tracker -- , seq: 747845, time: 2014-06-12 14:46:26.804332, event: reached_pg, op: osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.804347 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2513 (0'0,173'2513] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active] handle_message: 0x1e3eb200
2014-06-12 14:46:26.804362 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2513 (0'0,173'2513] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active] sub_op_modify trans a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 v 173'2514 (transaction) 174
2014-06-12 14:46:26.804380 7f9d653db700  5 -- op tracker -- , seq: 747845, time: 2014-06-12 14:46:26.804375, event: started, op: osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.804394 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2513 (0'0,173'2513] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active] append_log log((0'0,173'2513], crt=0'0) [173'2514 (173'2512) modify   a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 by client.4313.0:146359 2014-06-12 08:44:48.544695]
2014-06-12 14:46:26.804419 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2514 (0'0,173'2514] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active] add_log_entry 173'2514 (173'2512) modify   a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 by client.4313.0:146359 2014-06-12 08:44:48.544695
2014-06-12 14:46:26.804444 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2514 (0'0,173'2514] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active] append_log  adding 1 keys
2014-06-12 14:46:26.804477 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2514, trimmed: 
2014-06-12 14:46:26.804500 7f9d653db700  5 -- op tracker -- , seq: 747845, time: 2014-06-12 14:46:26.804495, event: started, op: osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.804510 7f9d653db700  5 queue_transactions existing osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:26.804514 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.804516 7f9d653db700  5 queue_op 0x5f67c20 seq 2292 osr(3.67d 0x276df70) 1554677 bytes   (queue has 1 ops and 1554677 bytes)
2014-06-12 14:46:26.804523 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.67d( v 173'2514 (0'0,173'2514] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active] do_sub_op osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.804538 7f9d653db700 10 osd.2 173 dequeue_op 0x1e3eb200 finish
2014-06-12 14:46:26.804542 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.804545 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.804584 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x524cb40 (1 active)
2014-06-12 14:46:26.804593 7f9d7c165700  5 _do_op 0x5f67c20 seq 2292 osr(3.67d 0x276df70)/0x276df70 start
2014-06-12 14:46:26.804597 7f9d7c165700 10 _do_transaction on 0x15edcef0
2014-06-12 14:46:26.804603 7f9d7c165700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:26.804738 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:26.804755 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.804814 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.804834 7f9d7c165700 15 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:26.804840 7f9d7c165700 10 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1 r = 0
2014-06-12 14:46:26.804846 7f9d7c165700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:26.804859 7f9d7c165700 15 _write 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 864256~1552384
2014-06-12 14:46:26.805024 7f9d7c165700 10 lookup_strip_header done  cid 3.67d_head oid a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:26.805176 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:26.805309 7f9d4829e700  5 -- op tracker -- , seq: 747846, time: 2014-06-12 14:46:26.804045, event: header_read, op: osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.805319 7f9d4829e700  5 -- op tracker -- , seq: 747846, time: 2014-06-12 14:46:26.804049, event: throttled, op: osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.805330 7f9d4829e700  5 -- op tracker -- , seq: 747846, time: 2014-06-12 14:46:26.805264, event: all_read, op: osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.805335 7f9d4829e700  5 -- op tracker -- , seq: 747846, time: 0.000000, event: dispatched, op: osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.805340 7f9d4829e700 20 osd.2 173 should_share_map client.4302 172.16.96.220:0/1243795 173
2014-06-12 14:46:26.805360 7f9d4829e700 15 osd.2 173 enqueue_op 0x68a6d00 prio 63 cost 516096 latency 0.001315 osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.805388 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.805395 7f9d63bd8700 10 osd.2 173 dequeue_op 0x68a6d00 prio 63 cost 516096 latency 0.001349 osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]
2014-06-12 14:46:26.805421 7f9d63bd8700  5 -- op tracker -- , seq: 747846, time: 2014-06-12 14:46:26.805416, event: reached_pg, op: osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.805430 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:26.805444 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] handle_message: 0x68a6d00
2014-06-12 14:46:26.805453 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_op osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:26.805481 7f9d63bd8700 15 getattr 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 '_'
2014-06-12 14:46:26.805678 7f9d63bd8700 10 getattr 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 '_' = 0
2014-06-12 14:46:26.805694 7f9d63bd8700 15 getattr 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 'snapset'
2014-06-12 14:46:26.805779 7f9d63bd8700 10 getattr 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 'snapset' = 0
2014-06-12 14:46:26.805788 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] populate_obc_watchers 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.805799 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.805808 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] get_object_context: creating obc from disk: 0x22ead600
2014-06-12 14:46:26.805818 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] get_object_context: 0x22ead600 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 rwstate(none n=0 w=0) oi: 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3(173'2728 client.4302.0:146134 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2728) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:46:26.805833 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] find_object_context 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 @head oi=67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3(173'2728 client.4302.0:146134 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2728)
2014-06-12 14:46:26.805862 7f9d63bd8700  5 -- op tracker -- , seq: 747846, time: 2014-06-12 14:46:26.805857, event: started, op: osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.805871 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] execute_ctx 0xfd19200
2014-06-12 14:46:26.805889 7f9d63bd8700  5 -- op tracker -- , seq: 747846, time: 2014-06-12 14:46:26.805884, event: started, op: osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.805895 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_op 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] ov 173'2728 av 173'2729 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:26.805908 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_osd_op 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096]
2014-06-12 14:46:26.805918 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:26.805930 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_osd_op  write 12288~516096
2014-06-12 14:46:26.805950 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_osd_op_effects on session 0x5c8d200
2014-06-12 14:46:26.805959 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] make_writeable 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:46:26.805970 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] make_writeable 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:26.805979 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] finish_ctx 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 0xfd19200 op modify  
2014-06-12 14:46:26.805991 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  set mtime to 2014-06-12 08:45:29.183876
2014-06-12 14:46:26.806009 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  final snapset 0=[]:[]+head in 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.806025 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  zeroing write result code 0
2014-06-12 14:46:26.806035 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] new_repop rep_tid 236267 on osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.806050 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] issue_repop rep_tid 236267 o 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.806071 7f9d63bd8700  5 -- op tracker -- , seq: 747846, time: 2014-06-12 14:46:26.806066, event: waiting for subops from 25, op: osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.806090 7f9d63bd8700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:26.806109 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2728 (0'0,173'2728] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] append_log log((0'0,173'2728], crt=0'0) [173'2729 (173'2728) modify   67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 by client.4302.0:146135 2014-06-12 08:45:29.183876]
2014-06-12 14:46:26.806130 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] add_log_entry 173'2729 (173'2728) modify   67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 by client.4302.0:146135 2014-06-12 08:45:29.183876
2014-06-12 14:46:26.806148 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] append_log  adding 1 keys
2014-06-12 14:46:26.806176 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2729, trimmed: 
2014-06-12 14:46:26.806196 7f9d63bd8700  5 queue_transactions existing osr(3.23f 0x276d3c0)/0x276d3c0
2014-06-12 14:46:26.806200 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.806201 7f9d63bd8700  5 queue_op 0x1ca3f20 seq 2555 osr(3.23f 0x276d3c0) 518389 bytes   (queue has 2 ops and 2073066 bytes)
2014-06-12 14:46:26.806209 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] eval_repop repgather(0x2a37a480 173'2729 rep_tid=236267 committed?=0 applied?=0 lock=0 op=osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.806225 7f9d63bd8700 10 osd.2 173 dequeue_op 0x68a6d00 finish
2014-06-12 14:46:26.806229 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.806231 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.808531 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540e7e0 (2 active)
2014-06-12 14:46:26.808544 7f9d83173700  5 _do_op 0x1ca3f20 seq 2555 osr(3.23f 0x276d3c0)/0x276d3c0 start
2014-06-12 14:46:26.808552 7f9d83173700 10 _do_transaction on 0x32e7b980
2014-06-12 14:46:26.808561 7f9d83173700 15 _omap_setkeys meta/1fe20b6b/pglog_3.23f/0//-1
2014-06-12 14:46:26.808728 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe20b6b/pglog_3.23f/0//-1
2014-06-12 14:46:26.808750 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.818400 7f9d7c165700 10 _generic_write 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 864256~1552384 = 0
2014-06-12 14:46:26.818676 7f9d7c165700 15 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:26.818694 7f9d7c165700 10 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 = 0
2014-06-12 14:46:26.818701 7f9d7c165700 15 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:26.818709 7f9d7c165700 10 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 = 0
2014-06-12 14:46:26.819297 7f9d7c165700 20 set_header setting 17386 cid 3.67d_head oid a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 parent seq 0
2014-06-12 14:46:26.819487 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.819501 7f9d7c165700 20 set_header setting 2272 cid meta oid 1fe0e10b/pglog_3.67d/0//-1 parent seq 0
2014-06-12 14:46:26.819510 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:26.832365 7f9d7c165700 10 _do_op 0x5f67c20 seq 2292 r = 0, finisher 0x660ac80 0
2014-06-12 14:46:26.832387 7f9d7c165700 10 _finish_op 0x5f67c20 seq 2292 osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:26.832414 7f9d7795c700  5 -- op tracker -- , seq: 747845, time: 2014-06-12 14:46:26.832405, event: sub_op_applied, op: osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.832425 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2514 (0'0,173'2514] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active] sub_op_modify_applied on 0x15edce00 op osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.832450 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:26.832468 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2514 (0'0,173'2514] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active] op_applied on primary on version 173'2514
2014-06-12 14:46:26.832484 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2514 (0'0,173'2514] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active] op_applied on replica on version 173'2514
2014-06-12 14:46:26.832495 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.832525 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x524cb40 (1 active)
2014-06-12 14:46:26.832553 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.832601 7f9d83173700 15 _omap_rmkeys meta/1fe20b6b/pglog_3.23f/0//-1
2014-06-12 14:46:26.832608 7f9d83173700 10 _omap_rmkeys meta/1fe20b6b/pglog_3.23f/0//-1 r = 0
2014-06-12 14:46:26.832615 7f9d83173700 15 _omap_setkeys meta/1fe20b6b/pglog_3.23f/0//-1
2014-06-12 14:46:26.832632 7f9d83173700 15 _write 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 12288~516096
2014-06-12 14:46:26.832764 7f9d83173700 10 lookup_strip_header done  cid 3.23f_head oid 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.832920 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:26.835237 7f9d83173700 10 _generic_write 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 12288~516096 = 0
2014-06-12 14:46:26.835329 7f9d83173700 15 _setattrs 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.835357 7f9d83173700 10 _setattrs 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:26.835363 7f9d83173700 15 _setattrs 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.835371 7f9d83173700 10 _setattrs 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:26.835625 7f9d83173700 20 set_header setting 17168 cid 3.23f_head oid 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 parent seq 0
2014-06-12 14:46:26.835696 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.835712 7f9d83173700 20 set_header setting 2024 cid meta oid 1fe20b6b/pglog_3.23f/0//-1 parent seq 0
2014-06-12 14:46:26.835722 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:26.836113 7f9d7715b700  5 -- op tracker -- , seq: 747845, time: 2014-06-12 14:46:26.836099, event: commit_sent, op: osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.836127 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2514 (0'0,173'2514] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2512 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:26.836160 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:26.836208 7f9d7715b700  5 -- op tracker -- , seq: 747845, time: 2014-06-12 14:46:26.836202, event: done, op: osd_sub_op(client.4313.0:146359 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.848332 7f9d83173700 10 _do_op 0x1ca3f20 seq 2555 r = 0, finisher 0x90eee40 0x33a49c0
2014-06-12 14:46:26.848355 7f9d83173700 10 _finish_op 0x1ca3f20 seq 2555 osr(3.23f 0x276d3c0)/0x276d3c0
2014-06-12 14:46:26.848364 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540e7e0 (0 active)
2014-06-12 14:46:26.848372 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.848359 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] op_commit: 236267
2014-06-12 14:46:26.848399 7f9d7715b700  5 -- op tracker -- , seq: 747846, time: 2014-06-12 14:46:26.848386, event: op_commit, op: osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.849013 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] op_applied: 236267
2014-06-12 14:46:26.849060 7f9d7795c700  5 -- op tracker -- , seq: 747846, time: 2014-06-12 14:46:26.849048, event: op_applied, op: osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.849072 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] op_applied on primary on version 173'2729
2014-06-12 14:46:26.864486 7f9d4829e700  5 -- op tracker -- , seq: 747847, time: 2014-06-12 14:46:26.861435, event: header_read, op: osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.864506 7f9d4829e700  5 -- op tracker -- , seq: 747847, time: 2014-06-12 14:46:26.861440, event: throttled, op: osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.864515 7f9d4829e700  5 -- op tracker -- , seq: 747847, time: 2014-06-12 14:46:26.864401, event: all_read, op: osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.864525 7f9d4829e700  5 -- op tracker -- , seq: 747847, time: 0.000000, event: dispatched, op: osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.864536 7f9d4829e700 20 osd.2 173 should_share_map client.4302 172.16.96.220:0/1243795 173
2014-06-12 14:46:26.864572 7f9d4829e700 15 osd.2 173 enqueue_op 0x28fd8200 prio 63 cost 1249280 latency 0.003137 osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.864614 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.864624 7f9d68be2700 10 osd.2 173 dequeue_op 0x28fd8200 prio 63 cost 1249280 latency 0.003189 osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]
2014-06-12 14:46:26.864654 7f9d68be2700  5 -- op tracker -- , seq: 747847, time: 2014-06-12 14:46:26.864649, event: reached_pg, op: osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.864665 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:26.864679 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] handle_message: 0x28fd8200
2014-06-12 14:46:26.864689 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_op osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:26.864716 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] get_object_context: found obc in cache: 0x22ead600
2014-06-12 14:46:26.864725 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] get_object_context: 0x22ead600 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 rwstate(write n=1 w=0) oi: 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3(173'2729 client.4302.0:146135 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2729) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:46:26.864740 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] find_object_context 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 @head oi=67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3(173'2729 client.4302.0:146135 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2729)
2014-06-12 14:46:26.864766 7f9d68be2700  5 -- op tracker -- , seq: 747847, time: 2014-06-12 14:46:26.864760, event: started, op: osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.864773 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] execute_ctx 0x6245600
2014-06-12 14:46:26.864789 7f9d68be2700  5 -- op tracker -- , seq: 747847, time: 2014-06-12 14:46:26.864785, event: started, op: osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.864796 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_op 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] ov 173'2729 av 173'2730 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:26.864812 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_osd_op 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280]
2014-06-12 14:46:26.864823 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:26.864838 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_osd_op  write 528384~1249280
2014-06-12 14:46:26.864855 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] do_osd_op_effects on session 0x5c8d200
2014-06-12 14:46:26.864868 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] make_writeable 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:46:26.864879 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] make_writeable 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:26.864891 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] finish_ctx 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 0x6245600 op modify  
2014-06-12 14:46:26.864903 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  set mtime to 2014-06-12 08:45:29.237722
2014-06-12 14:46:26.864924 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  final snapset 0=[]:[]+head in 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.864944 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  zeroing write result code 0
2014-06-12 14:46:26.864956 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] new_repop rep_tid 236268 on osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.864972 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] issue_repop rep_tid 236268 o 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.864999 7f9d68be2700  5 -- op tracker -- , seq: 747847, time: 2014-06-12 14:46:26.864994, event: waiting for subops from 25, op: osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.865023 7f9d68be2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:26.865048 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2729 (0'0,173'2729] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] append_log log((0'0,173'2729], crt=0'0) [173'2730 (173'2729) modify   67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 by client.4302.0:146136 2014-06-12 08:45:29.237722]
2014-06-12 14:46:26.865073 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] add_log_entry 173'2730 (173'2729) modify   67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 by client.4302.0:146136 2014-06-12 08:45:29.237722
2014-06-12 14:46:26.865097 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] append_log  adding 1 keys
2014-06-12 14:46:26.865131 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2730, trimmed: 
2014-06-12 14:46:26.865166 7f9d68be2700  5 queue_transactions existing osr(3.23f 0x276d3c0)/0x276d3c0
2014-06-12 14:46:26.865172 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.865174 7f9d68be2700  5 queue_op 0xae92300 seq 2556 osr(3.23f 0x276d3c0) 1251573 bytes   (queue has 1 ops and 1251573 bytes)
2014-06-12 14:46:26.865181 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] eval_repop repgather(0x514ce40 173'2730 rep_tid=236268 committed?=0 applied?=0 lock=0 op=osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.865197 7f9d68be2700 10 osd.2 173 dequeue_op 0x28fd8200 finish
2014-06-12 14:46:26.865201 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.865204 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.865213 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540e7e0 (1 active)
2014-06-12 14:46:26.865218 7f9d82972700  5 _do_op 0xae92300 seq 2556 osr(3.23f 0x276d3c0)/0x276d3c0 start
2014-06-12 14:46:26.865223 7f9d82972700 10 _do_transaction on 0x514d440
2014-06-12 14:46:26.865229 7f9d82972700 15 _omap_setkeys meta/1fe20b6b/pglog_3.23f/0//-1
2014-06-12 14:46:26.865386 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe20b6b/pglog_3.23f/0//-1
2014-06-12 14:46:26.865410 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.865474 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.865497 7f9d82972700 15 _omap_rmkeys meta/1fe20b6b/pglog_3.23f/0//-1
2014-06-12 14:46:26.865506 7f9d82972700 10 _omap_rmkeys meta/1fe20b6b/pglog_3.23f/0//-1 r = 0
2014-06-12 14:46:26.865513 7f9d82972700 15 _omap_setkeys meta/1fe20b6b/pglog_3.23f/0//-1
2014-06-12 14:46:26.865526 7f9d82972700 15 _write 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 528384~1249280
2014-06-12 14:46:26.865729 7f9d82972700 10 lookup_strip_header done  cid 3.23f_head oid 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.866035 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:26.871325 7f9d82972700 10 _generic_write 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 528384~1249280 = 0
2014-06-12 14:46:26.871483 7f9d82972700 15 _setattrs 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.871502 7f9d82972700 10 _setattrs 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:26.871507 7f9d82972700 15 _setattrs 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3
2014-06-12 14:46:26.871514 7f9d82972700 10 _setattrs 3.23f_head/67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 = 0
2014-06-12 14:46:26.871899 7f9d82972700 20 set_header setting 17168 cid 3.23f_head oid 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 parent seq 0
2014-06-12 14:46:26.872029 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.872042 7f9d82972700 20 set_header setting 2024 cid meta oid 1fe20b6b/pglog_3.23f/0//-1 parent seq 0
2014-06-12 14:46:26.872051 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:26.878284 7f9d5bec4700  5 -- op tracker -- , seq: 747848, time: 2014-06-12 14:46:26.878236, event: header_read, op: osd_sub_op_reply(client.4302.0:146135 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.878295 7f9d5bec4700  5 -- op tracker -- , seq: 747848, time: 2014-06-12 14:46:26.878237, event: throttled, op: osd_sub_op_reply(client.4302.0:146135 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.878303 7f9d5bec4700  5 -- op tracker -- , seq: 747848, time: 2014-06-12 14:46:26.878248, event: all_read, op: osd_sub_op_reply(client.4302.0:146135 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.878311 7f9d5bec4700  5 -- op tracker -- , seq: 747848, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146135 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.878315 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146135 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:26.878321 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:26.878319 7f9d82972700 10 _do_op 0xae92300 seq 2556 r = 0, finisher 0x17726360 0x2a368d00
2014-06-12 14:46:26.878328 7f9d5bec4700 15 osd.2 173 enqueue_op 0x12bf6100 prio 196 cost 0 latency 0.000092 osd_sub_op_reply(client.4302.0:146135 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:26.878336 7f9d82972700 10 _finish_op 0xae92300 seq 2556 osr(3.23f 0x276d3c0)/0x276d3c0
2014-06-12 14:46:26.878344 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540e7e0 (0 active)
2014-06-12 14:46:26.878352 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.878356 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.878363 7f9d623d5700 10 osd.2 173 dequeue_op 0x12bf6100 prio 196 cost 0 latency 0.000127 osd_sub_op_reply(client.4302.0:146135 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0) v2 pg pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]
2014-06-12 14:46:26.878383 7f9d623d5700  5 -- op tracker -- , seq: 747848, time: 2014-06-12 14:46:26.878379, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146135 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.878391 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] handle_message: 0x12bf6100
2014-06-12 14:46:26.878407 7f9d623d5700  5 -- op tracker -- , seq: 747848, time: 2014-06-12 14:46:26.878403, event: started, op: osd_sub_op_reply(client.4302.0:146135 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.878412 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] sub_op_modify_reply: tid 236267 op  ack_type 4 from 25
2014-06-12 14:46:26.878431 7f9d623d5700  5 -- op tracker -- , seq: 747846, time: 2014-06-12 14:46:26.878424, event: sub_op_commit_rec, op: osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.878438 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] repop_all_applied: repop tid 236267 all applied 
2014-06-12 14:46:26.878449 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] eval_repop repgather(0x2a37a480 173'2729 rep_tid=236267 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.878464 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean]  sending ack on repgather(0x2a37a480 173'2729 rep_tid=236267 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) 0x9657900
2014-06-12 14:46:26.878498 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2728 crt=0'0 lcod 173'2727 mlcod 173'2727 active+clean] repop_all_committed: repop tid 236267 all committed 
2014-06-12 14:46:26.878509 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2727 active+clean] eval_repop repgather(0x2a37a480 173'2729 rep_tid=236267 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.878524 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2727 active+clean] log_op_stats osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.074452 lat 0.074478
2014-06-12 14:46:26.878540 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2727 active+clean] publish_stats_to_osd 173:3550
2014-06-12 14:46:26.878556 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2727 active+clean]  sending commit on repgather(0x2a37a480 173'2729 rep_tid=236267 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) 0x19e0b400
2014-06-12 14:46:26.878589 7f9d623d5700  5 -- op tracker -- , seq: 747846, time: 2014-06-12 14:46:26.878584, event: commit_sent, op: osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.878596 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2727 active+clean]  requeue_ops 
2014-06-12 14:46:26.878606 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]  removing repgather(0x2a37a480 173'2729 rep_tid=236267 committed?=1 applied?=1 op=osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.878621 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]    q front is repgather(0x2a37a480 173'2729 rep_tid=236267 committed?=1 applied?=1 op=osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.878634 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] remove_repop repgather(0x2a37a480 173'2729 rep_tid=236267 committed?=1 applied?=1 op=osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.878648 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]  requeue_ops 
2014-06-12 14:46:26.878666 7f9d623d5700  5 -- op tracker -- , seq: 747846, time: 2014-06-12 14:46:26.878661, event: done, op: osd_op(client.4302.0:146135 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 12288~516096] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.878687 7f9d623d5700 10 osd.2 173 dequeue_op 0x12bf6100 finish
2014-06-12 14:46:26.878694 7f9d623d5700  5 -- op tracker -- , seq: 747848, time: 2014-06-12 14:46:26.878691, event: done, op: osd_sub_op_reply(client.4302.0:146135 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.878704 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.878707 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.878720 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] op_commit: 236268
2014-06-12 14:46:26.878743 7f9d7715b700  5 -- op tracker -- , seq: 747847, time: 2014-06-12 14:46:26.878736, event: op_commit, op: osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.879357 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] op_applied: 236268
2014-06-12 14:46:26.879387 7f9d7795c700  5 -- op tracker -- , seq: 747847, time: 2014-06-12 14:46:26.879379, event: op_applied, op: osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.879396 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] op_applied on primary on version 173'2730
2014-06-12 14:46:26.905677 7f9d5bec4700  5 -- op tracker -- , seq: 747849, time: 2014-06-12 14:46:26.905604, event: header_read, op: osd_sub_op_reply(client.4311.0:139832 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.905688 7f9d5bec4700  5 -- op tracker -- , seq: 747849, time: 2014-06-12 14:46:26.905606, event: throttled, op: osd_sub_op_reply(client.4311.0:139832 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.905694 7f9d5bec4700  5 -- op tracker -- , seq: 747849, time: 2014-06-12 14:46:26.905640, event: all_read, op: osd_sub_op_reply(client.4311.0:139832 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.905701 7f9d5bec4700  5 -- op tracker -- , seq: 747849, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4311.0:139832 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.905706 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4311.0:139832 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:26.905712 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:26.905720 7f9d5bec4700 15 osd.2 173 enqueue_op 0x12bf7100 prio 196 cost 0 latency 0.000115 osd_sub_op_reply(client.4311.0:139832 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:26.905750 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.905760 7f9d643d9700 10 osd.2 173 dequeue_op 0x12bf7100 prio 196 cost 0 latency 0.000156 osd_sub_op_reply(client.4311.0:139832 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2735 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean]
2014-06-12 14:46:26.905785 7f9d643d9700  5 -- op tracker -- , seq: 747849, time: 2014-06-12 14:46:26.905781, event: reached_pg, op: osd_sub_op_reply(client.4311.0:139832 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.905801 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2735 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] handle_message: 0x12bf7100
2014-06-12 14:46:26.905818 7f9d643d9700  5 -- op tracker -- , seq: 747849, time: 2014-06-12 14:46:26.905815, event: started, op: osd_sub_op_reply(client.4311.0:139832 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.905825 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2735 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] sub_op_modify_reply: tid 236266 op  ack_type 4 from 25
2014-06-12 14:46:26.905846 7f9d643d9700  5 -- op tracker -- , seq: 747844, time: 2014-06-12 14:46:26.905839, event: sub_op_commit_rec, op: osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.905854 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2735 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] repop_all_applied: repop tid 236266 all applied 
2014-06-12 14:46:26.905865 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2735 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] eval_repop repgather(0x7414480 173'2736 rep_tid=236266 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.905882 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2735 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean]  sending ack on repgather(0x7414480 173'2736 rep_tid=236266 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) 0x959a000
2014-06-12 14:46:26.905918 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2735 crt=0'0 lcod 173'2734 mlcod 173'2734 active+clean] repop_all_committed: repop tid 236266 all committed 
2014-06-12 14:46:26.905931 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2734 active+clean] eval_repop repgather(0x7414480 173'2736 rep_tid=236266 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.905951 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2734 active+clean] log_op_stats osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.131240 lat 0.131272
2014-06-12 14:46:26.905971 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2734 active+clean] publish_stats_to_osd 173:3634
2014-06-12 14:46:26.905987 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2734 active+clean]  sending commit on repgather(0x7414480 173'2736 rep_tid=236266 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) 0x15ce2000
2014-06-12 14:46:26.906025 7f9d643d9700  5 -- op tracker -- , seq: 747844, time: 2014-06-12 14:46:26.906019, event: commit_sent, op: osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.906036 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2734 active+clean]  requeue_ops 
2014-06-12 14:46:26.906049 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean]  removing repgather(0x7414480 173'2736 rep_tid=236266 committed?=1 applied?=1 op=osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.906067 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean]    q front is repgather(0x7414480 173'2736 rep_tid=236266 committed?=1 applied?=1 op=osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.906084 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] remove_repop repgather(0x7414480 173'2736 rep_tid=236266 committed?=1 applied?=1 op=osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.906100 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean]  requeue_ops 
2014-06-12 14:46:26.906129 7f9d643d9700  5 -- op tracker -- , seq: 747844, time: 2014-06-12 14:46:26.906124, event: done, op: osd_op(client.4311.0:139832 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.906153 7f9d643d9700 10 osd.2 173 dequeue_op 0x12bf7100 finish
2014-06-12 14:46:26.906164 7f9d643d9700  5 -- op tracker -- , seq: 747849, time: 2014-06-12 14:46:26.906160, event: done, op: osd_sub_op_reply(client.4311.0:139832 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.906177 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.906181 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.914222 7f9d44ece700  5 -- op tracker -- , seq: 747850, time: 2014-06-12 14:46:26.912895, event: header_read, op: osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.914238 7f9d44ece700  5 -- op tracker -- , seq: 747850, time: 2014-06-12 14:46:26.912900, event: throttled, op: osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.914247 7f9d44ece700  5 -- op tracker -- , seq: 747850, time: 2014-06-12 14:46:26.914159, event: all_read, op: osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.914256 7f9d44ece700  5 -- op tracker -- , seq: 747850, time: 0.000000, event: dispatched, op: osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.914264 7f9d44ece700 20 osd.2 173 should_share_map client.4311 172.16.96.220:0/1244240 173
2014-06-12 14:46:26.914292 7f9d44ece700 15 osd.2 173 enqueue_op 0x1702f600 prio 63 cost 1032192 latency 0.001397 osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.914324 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.914336 7f9d633d7700 10 osd.2 173 dequeue_op 0x1702f600 prio 63 cost 1032192 latency 0.001440 osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean]
2014-06-12 14:46:26.914362 7f9d633d7700  5 -- op tracker -- , seq: 747850, time: 2014-06-12 14:46:26.914357, event: reached_pg, op: osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.914372 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:26.914386 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] handle_message: 0x1702f600
2014-06-12 14:46:26.914396 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] do_op osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:26.914418 7f9d633d7700 15 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 '_'
2014-06-12 14:46:26.914627 7f9d633d7700 10 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 '_' = 0
2014-06-12 14:46:26.914645 7f9d633d7700 15 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 'snapset'
2014-06-12 14:46:26.914726 7f9d633d7700 10 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 'snapset' = 0
2014-06-12 14:46:26.914737 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] populate_obc_watchers aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.914750 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.914759 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] get_object_context: creating obc from disk: 0x1771fb80
2014-06-12 14:46:26.914767 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] get_object_context: 0x1771fb80 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 rwstate(none n=0 w=0) oi: aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3(173'2736 client.4311.0:139832 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2736) ssc: 0x5bb8d20 snapset: 0=[]:[]+head
2014-06-12 14:46:26.914782 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] find_object_context aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 @head oi=aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3(173'2736 client.4311.0:139832 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2736)
2014-06-12 14:46:26.914806 7f9d633d7700  5 -- op tracker -- , seq: 747850, time: 2014-06-12 14:46:26.914801, event: started, op: osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.914815 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] execute_ctx 0x29edf000
2014-06-12 14:46:26.914830 7f9d633d7700  5 -- op tracker -- , seq: 747850, time: 2014-06-12 14:46:26.914826, event: started, op: osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.914836 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] do_op aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] ov 173'2736 av 173'2737 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:26.914850 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] do_osd_op aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192]
2014-06-12 14:46:26.914859 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:26.914871 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] do_osd_op  write 1036288~1032192
2014-06-12 14:46:26.914884 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] do_osd_op_effects on session 0x5a1d800
2014-06-12 14:46:26.914895 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] make_writeable aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 snapset=0x5bb8d58  snapc=0=[]
2014-06-12 14:46:26.914906 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] make_writeable aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:26.914914 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] finish_ctx aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 0x29edf000 op modify  
2014-06-12 14:46:26.914925 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean]  set mtime to 2014-06-12 08:45:29.289046
2014-06-12 14:46:26.914945 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean]  final snapset 0=[]:[]+head in aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.914962 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean]  zeroing write result code 0
2014-06-12 14:46:26.914972 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] new_repop rep_tid 236269 on osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:26.914987 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] issue_repop rep_tid 236269 o aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.915008 7f9d633d7700  5 -- op tracker -- , seq: 747850, time: 2014-06-12 14:46:26.915003, event: waiting for subops from 25, op: osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.915030 7f9d633d7700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:26.915051 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2736 (0'0,173'2736] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] append_log log((0'0,173'2736], crt=0'0) [173'2737 (173'2736) modify   aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 by client.4311.0:139833 2014-06-12 08:45:29.289046]
2014-06-12 14:46:26.915072 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2736 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] add_log_entry 173'2737 (173'2736) modify   aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 by client.4311.0:139833 2014-06-12 08:45:29.289046
2014-06-12 14:46:26.915091 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2736 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] append_log  adding 1 keys
2014-06-12 14:46:26.915120 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2737, trimmed: 
2014-06-12 14:46:26.915141 7f9d633d7700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:26.915145 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.915146 7f9d633d7700  5 queue_op 0x9fe2de0 seq 2549 osr(3.122 0x276cf40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:26.915153 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2736 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] eval_repop repgather(0x5237740 173'2737 rep_tid=236269 committed?=0 applied?=0 lock=0 op=osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.915170 7f9d633d7700 10 osd.2 173 dequeue_op 0x1702f600 finish
2014-06-12 14:46:26.915174 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.915176 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.915959 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:46:26.915972 7f9d81970700  5 _do_op 0x9fe2de0 seq 2549 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:46:26.915979 7f9d81970700 10 _do_transaction on 0x5214fc0
2014-06-12 14:46:26.915990 7f9d81970700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:26.916175 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:26.916195 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:26.916250 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:26.916267 7f9d81970700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:26.916274 7f9d81970700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:46:26.916280 7f9d81970700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:26.916293 7f9d81970700 15 _write 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 1036288~1032192
2014-06-12 14:46:26.916442 7f9d81970700 10 lookup_strip_header done  cid 3.122_head oid aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.916716 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:26.921432 7f9d81970700 10 _generic_write 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 1036288~1032192 = 0
2014-06-12 14:46:26.921588 7f9d81970700 15 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.921608 7f9d81970700 10 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:26.921642 7f9d81970700 15 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:26.921653 7f9d81970700 10 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:26.922035 7f9d81970700 20 set_header setting 16189 cid 3.122_head oid aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 parent seq 0
2014-06-12 14:46:26.922170 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:26.922187 7f9d81970700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:46:26.922198 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:26.926296 7f9d56432700  5 -- op tracker -- , seq: 747851, time: 2014-06-12 14:46:26.926202, event: header_read, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ack, result = 0)
2014-06-12 14:46:26.926311 7f9d56432700  5 -- op tracker -- , seq: 747851, time: 2014-06-12 14:46:26.926205, event: throttled, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ack, result = 0)
2014-06-12 14:46:26.926318 7f9d56432700  5 -- op tracker -- , seq: 747851, time: 2014-06-12 14:46:26.926230, event: all_read, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ack, result = 0)
2014-06-12 14:46:26.926325 7f9d56432700  5 -- op tracker -- , seq: 747851, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ack, result = 0)
2014-06-12 14:46:26.926331 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:26.926337 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:26.926346 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3eb100 prio 196 cost 0 latency 0.000144 osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ack, result = 0) v2
2014-06-12 14:46:26.926486 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.926496 7f9d61bd4700 10 osd.2 173 dequeue_op 0x1e3eb100 prio 196 cost 0 latency 0.000294 osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ack, result = 0) v2 pg pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean]
2014-06-12 14:46:26.926518 7f9d61bd4700  5 -- op tracker -- , seq: 747851, time: 2014-06-12 14:46:26.926515, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ack, result = 0)
2014-06-12 14:46:26.926527 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean] handle_message: 0x1e3eb100
2014-06-12 14:46:26.926542 7f9d61bd4700  5 -- op tracker -- , seq: 747851, time: 2014-06-12 14:46:26.926539, event: started, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ack, result = 0)
2014-06-12 14:46:26.926548 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean] sub_op_modify_reply: tid 236185 op  ack_type 1 from 26
2014-06-12 14:46:26.926569 7f9d61bd4700  5 -- op tracker -- , seq: 747474, time: 2014-06-12 14:46:26.926560, event: sub_op_applied_rec, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.926577 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean] repop_all_applied: repop tid 236185 all applied 
2014-06-12 14:46:26.926587 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean] eval_repop repgather(0x2a37b980 173'2510 rep_tid=236185 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.926601 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean]  sending ack on repgather(0x2a37b980 173'2510 rep_tid=236185 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) 0x890b180
2014-06-12 14:46:26.926631 7f9d61bd4700 10 osd.2 173 dequeue_op 0x1e3eb100 finish
2014-06-12 14:46:26.926638 7f9d61bd4700  5 -- op tracker -- , seq: 747851, time: 2014-06-12 14:46:26.926635, event: done, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ack, result = 0)
2014-06-12 14:46:26.926653 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.926655 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.927058 7f9d81970700 10 _do_op 0x9fe2de0 seq 2549 r = 0, finisher 0x65f7420 0x2869f800
2014-06-12 14:46:26.927084 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2736 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] op_commit: 236269
2014-06-12 14:46:26.927112 7f9d7715b700  5 -- op tracker -- , seq: 747850, time: 2014-06-12 14:46:26.927104, event: op_commit, op: osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.927127 7f9d81970700 10 _finish_op 0x9fe2de0 seq 2549 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:26.927145 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2736 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] op_applied: 236269
2014-06-12 14:46:26.927166 7f9d7795c700  5 -- op tracker -- , seq: 747850, time: 2014-06-12 14:46:26.927160, event: op_applied, op: osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.927175 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2736 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] op_applied on primary on version 173'2737
2014-06-12 14:46:26.927209 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:46:26.927217 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:26.929686 7f9d56432700  5 -- op tracker -- , seq: 747852, time: 2014-06-12 14:46:26.929633, event: header_read, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.929695 7f9d56432700  5 -- op tracker -- , seq: 747852, time: 2014-06-12 14:46:26.929636, event: throttled, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.929699 7f9d56432700  5 -- op tracker -- , seq: 747852, time: 2014-06-12 14:46:26.929650, event: all_read, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.929704 7f9d56432700  5 -- op tracker -- , seq: 747852, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.929708 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:26.929713 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:26.929719 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3ea800 prio 196 cost 0 latency 0.000086 osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:26.929879 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.929887 7f9d673df700 10 osd.2 173 dequeue_op 0x1e3ea800 prio 196 cost 0 latency 0.000254 osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2 pg pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean]
2014-06-12 14:46:26.929908 7f9d673df700  5 -- op tracker -- , seq: 747852, time: 2014-06-12 14:46:26.929905, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.929918 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean] handle_message: 0x1e3ea800
2014-06-12 14:46:26.929933 7f9d673df700  5 -- op tracker -- , seq: 747852, time: 2014-06-12 14:46:26.929930, event: started, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.929939 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean] sub_op_modify_reply: tid 236185 op  ack_type 4 from 26
2014-06-12 14:46:26.929959 7f9d673df700  5 -- op tracker -- , seq: 747474, time: 2014-06-12 14:46:26.929951, event: sub_op_commit_rec, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.929966 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2509 crt=0'0 lcod 173'2508 mlcod 173'2508 active+clean] repop_all_committed: repop tid 236185 all committed 
2014-06-12 14:46:26.929976 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2508 active+clean] eval_repop repgather(0x2a37b980 173'2510 rep_tid=236185 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.929991 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2508 active+clean] log_op_stats osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 25.438181 lat 25.441542
2014-06-12 14:46:26.930007 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2508 active+clean] publish_stats_to_osd 173:2880
2014-06-12 14:46:26.930020 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2508 active+clean]  sending commit on repgather(0x2a37b980 173'2510 rep_tid=236185 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) 0x353e9400
2014-06-12 14:46:26.930056 7f9d673df700  5 -- op tracker -- , seq: 747474, time: 2014-06-12 14:46:26.930050, event: commit_sent, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.930063 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2508 active+clean]  requeue_ops 
2014-06-12 14:46:26.930073 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2509 active+clean]  removing repgather(0x2a37b980 173'2510 rep_tid=236185 committed?=1 applied?=1 op=osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.930086 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2509 active+clean]    q front is repgather(0x2a37b980 173'2510 rep_tid=236185 committed?=1 applied?=1 op=osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.930101 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2509 active+clean] remove_repop repgather(0x2a37b980 173'2510 rep_tid=236185 committed?=1 applied?=1 op=osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.930114 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2509 active+clean]  requeue_ops 
2014-06-12 14:46:26.930132 7f9d673df700  5 -- op tracker -- , seq: 747474, time: 2014-06-12 14:46:26.930127, event: done, op: osd_op(client.4317.0:143792 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 2486272~516096] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.930149 7f9d673df700 10 osd.2 173 dequeue_op 0x1e3ea800 finish
2014-06-12 14:46:26.930156 7f9d673df700  5 -- op tracker -- , seq: 747852, time: 2014-06-12 14:46:26.930153, event: done, op: osd_sub_op_reply(client.4317.0:143792 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.930166 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.930169 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.981425 7f9d5bec4700  5 -- op tracker -- , seq: 747853, time: 2014-06-12 14:46:26.981366, event: header_read, op: osd_sub_op_reply(client.4302.0:146136 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.981434 7f9d5bec4700  5 -- op tracker -- , seq: 747853, time: 2014-06-12 14:46:26.981368, event: throttled, op: osd_sub_op_reply(client.4302.0:146136 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.981448 7f9d5bec4700  5 -- op tracker -- , seq: 747853, time: 2014-06-12 14:46:26.981384, event: all_read, op: osd_sub_op_reply(client.4302.0:146136 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.981452 7f9d5bec4700  5 -- op tracker -- , seq: 747853, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4302.0:146136 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.981456 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4302.0:146136 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:26.981460 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:26.981464 7f9d5bec4700 15 osd.2 173 enqueue_op 0x124e2500 prio 196 cost 0 latency 0.000098 osd_sub_op_reply(client.4302.0:146136 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:26.981485 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.981493 7f9d67be0700 10 osd.2 173 dequeue_op 0x124e2500 prio 196 cost 0 latency 0.000127 osd_sub_op_reply(client.4302.0:146136 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0) v2 pg pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]
2014-06-12 14:46:26.981528 7f9d67be0700  5 -- op tracker -- , seq: 747853, time: 2014-06-12 14:46:26.981525, event: reached_pg, op: osd_sub_op_reply(client.4302.0:146136 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.981545 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] handle_message: 0x124e2500
2014-06-12 14:46:26.981556 7f9d67be0700  5 -- op tracker -- , seq: 747853, time: 2014-06-12 14:46:26.981554, event: started, op: osd_sub_op_reply(client.4302.0:146136 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.981560 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] sub_op_modify_reply: tid 236268 op  ack_type 4 from 25
2014-06-12 14:46:26.981581 7f9d67be0700  5 -- op tracker -- , seq: 747847, time: 2014-06-12 14:46:26.981576, event: sub_op_commit_rec, op: osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.981586 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] repop_all_applied: repop tid 236268 all applied 
2014-06-12 14:46:26.981593 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] eval_repop repgather(0x514ce40 173'2730 rep_tid=236268 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.981603 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean]  sending ack on repgather(0x514ce40 173'2730 rep_tid=236268 committed?=0 applied?=1 lock=0 op=osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) 0x6700c80
2014-06-12 14:46:26.981640 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2729 crt=0'0 lcod 173'2728 mlcod 173'2728 active+clean] repop_all_committed: repop tid 236268 all committed 
2014-06-12 14:46:26.981652 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2728 active+clean] eval_repop repgather(0x514ce40 173'2730 rep_tid=236268 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:26.981679 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2728 active+clean] log_op_stats osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4 inb 1249559 outb 0 rlat 0.120203 lat 0.120242
2014-06-12 14:46:26.981697 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2728 active+clean] publish_stats_to_osd 173:3551
2014-06-12 14:46:26.981709 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2728 active+clean]  sending commit on repgather(0x514ce40 173'2730 rep_tid=236268 committed?=1 applied?=1 lock=0 op=osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4) 0x1b40a500
2014-06-12 14:46:26.981742 7f9d67be0700  5 -- op tracker -- , seq: 747847, time: 2014-06-12 14:46:26.981737, event: commit_sent, op: osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.981749 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2728 active+clean]  requeue_ops 
2014-06-12 14:46:26.981759 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean]  removing repgather(0x514ce40 173'2730 rep_tid=236268 committed?=1 applied?=1 op=osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.981773 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean]    q front is repgather(0x514ce40 173'2730 rep_tid=236268 committed?=1 applied?=1 op=osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.981788 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] remove_repop repgather(0x514ce40 173'2730 rep_tid=236268 committed?=1 applied?=1 op=osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:26.981804 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean]  requeue_ops 
2014-06-12 14:46:26.981832 7f9d67be0700  5 -- op tracker -- , seq: 747847, time: 2014-06-12 14:46:26.981826, event: done, op: osd_op(client.4302.0:146136 rbd_data.10a56b8b4567.000000000000103f [set-alloc-hint object_size 4194304 write_size 4194304,write 528384~1249280] 3.67e2b23f ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:26.981852 7f9d67be0700 10 osd.2 173 dequeue_op 0x124e2500 finish
2014-06-12 14:46:26.981859 7f9d67be0700  5 -- op tracker -- , seq: 747853, time: 2014-06-12 14:46:26.981856, event: done, op: osd_sub_op_reply(client.4302.0:146136 3.23f 67e2b23f/rbd_data.10a56b8b4567.000000000000103f/head//3 [] ondisk, result = 0)
2014-06-12 14:46:26.981869 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.981872 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:26.998376 7f9d56432700  5 -- op tracker -- , seq: 747854, time: 2014-06-12 14:46:26.998226, event: header_read, op: osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.998398 7f9d56432700  5 -- op tracker -- , seq: 747854, time: 2014-06-12 14:46:26.998228, event: throttled, op: osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.998414 7f9d56432700  5 -- op tracker -- , seq: 747854, time: 2014-06-12 14:46:26.998330, event: all_read, op: osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.998421 7f9d56432700  5 -- op tracker -- , seq: 747854, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.998428 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:26.998434 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:26.998442 7f9d56432700 15 osd.2 173 enqueue_op 0x1e3ebf00 prio 127 cost 45863 latency 0.000216 osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.998474 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:26.998480 7f9d663dd700 10 osd.2 173 dequeue_op 0x1e3ebf00 prio 127 cost 45863 latency 0.000254 osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.67d( v 173'2514 (0'0,173'2514] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2513 active]
2014-06-12 14:46:26.998504 7f9d663dd700  5 -- op tracker -- , seq: 747854, time: 2014-06-12 14:46:26.998500, event: reached_pg, op: osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.998524 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2514 (0'0,173'2514] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2513 active] handle_message: 0x1e3ebf00
2014-06-12 14:46:26.998536 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2514 (0'0,173'2514] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2513 active] sub_op_modify trans 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 v 173'2515 (transaction) 174
2014-06-12 14:46:26.998553 7f9d663dd700  5 -- op tracker -- , seq: 747854, time: 2014-06-12 14:46:26.998550, event: started, op: osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.998590 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2514 (0'0,173'2514] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2513 active] append_log log((0'0,173'2514], crt=0'0) [173'2515 (173'2513) modify   6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 by client.4303.0:144818 2014-06-12 08:45:21.246215]
2014-06-12 14:46:26.998618 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2515 (0'0,173'2515] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2513 active] add_log_entry 173'2515 (173'2513) modify   6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 by client.4303.0:144818 2014-06-12 08:45:21.246215
2014-06-12 14:46:26.998651 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2515 (0'0,173'2515] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2513 active] append_log  adding 1 keys
2014-06-12 14:46:26.998686 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2515, trimmed: 
2014-06-12 14:46:26.998731 7f9d663dd700  5 -- op tracker -- , seq: 747854, time: 2014-06-12 14:46:26.998726, event: started, op: osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:26.998744 7f9d663dd700  5 queue_transactions existing osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:26.998749 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:26.998751 7f9d663dd700  5 queue_op 0xdf177a0 seq 2293 osr(3.67d 0x276df70) 47349 bytes   (queue has 1 ops and 47349 bytes)
2014-06-12 14:46:26.998761 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x524cb40 (1 active)
2014-06-12 14:46:26.998770 7f9d7c966700  5 _do_op 0xdf177a0 seq 2293 osr(3.67d 0x276df70)/0x276df70 start
2014-06-12 14:46:26.998775 7f9d7c966700 10 _do_transaction on 0x4fe77b0
2014-06-12 14:46:26.998756 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.67d( v 173'2515 (0'0,173'2515] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2513 active] do_sub_op osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:26.998781 7f9d663dd700 10 osd.2 173 dequeue_op 0x1e3ebf00 finish
2014-06-12 14:46:26.998782 7f9d7c966700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:26.998784 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:26.998786 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:27.008905 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:27.008942 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:27.009001 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:27.009164 7f9d7c966700 15 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:27.009171 7f9d7c966700 10 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1 r = 0
2014-06-12 14:46:27.009178 7f9d7c966700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:27.009197 7f9d7c966700 15 _write 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 4149248~45056
2014-06-12 14:46:27.009361 7f9d7c966700 10 lookup_strip_header done  cid 3.67d_head oid 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:27.009480 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:27.009648 7f9d7c966700 10 _generic_write 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 4149248~45056 = 0
2014-06-12 14:46:27.009660 7f9d7c966700 15 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:27.009674 7f9d7c966700 10 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 = 0
2014-06-12 14:46:27.009679 7f9d7c966700 15 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3
2014-06-12 14:46:27.009687 7f9d7c966700 10 _setattrs 3.67d_head/6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 = 0
2014-06-12 14:46:27.009810 7f9d7c966700 20 set_header setting 17507 cid 3.67d_head oid 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 parent seq 0
2014-06-12 14:46:27.009836 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:27.009850 7f9d7c966700 20 set_header setting 2272 cid meta oid 1fe0e10b/pglog_3.67d/0//-1 parent seq 0
2014-06-12 14:46:27.009860 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:27.010128 7f9d7c966700 10 _do_op 0xdf177a0 seq 2293 r = 0, finisher 0x690e240 0
2014-06-12 14:46:27.010164 7f9d7715b700  5 -- op tracker -- , seq: 747854, time: 2014-06-12 14:46:27.010153, event: commit_sent, op: osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.010176 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2515 (0'0,173'2515] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2513 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:27.010204 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:27.010261 7f9d7c966700 10 _finish_op 0xdf177a0 seq 2293 osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:27.010293 7f9d7795c700  5 -- op tracker -- , seq: 747854, time: 2014-06-12 14:46:27.010286, event: sub_op_applied, op: osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.010300 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2515 (0'0,173'2515] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2514 active] sub_op_modify_applied on 0x4fe76c0 op osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:27.010321 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2515 (0'0,173'2515] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2514 active] op_applied on primary on version 173'2515
2014-06-12 14:46:27.010331 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2515 (0'0,173'2515] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2514 active] op_applied on replica on version 173'2515
2014-06-12 14:46:27.010367 7f9d7c966700  5 -- op tracker -- , seq: 747854, time: 2014-06-12 14:46:27.010357, event: done, op: osd_sub_op(client.4303.0:144818 3.67d 6c93ee7d/rbd_data.10a66b8b4567.0000000000000ffb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.010441 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x524cb40 (0 active)
2014-06-12 14:46:27.010446 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:27.038953 7f9d56432700  5 -- op tracker -- , seq: 747855, time: 2014-06-12 14:46:27.038672, event: header_read, op: osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.038979 7f9d56432700  5 -- op tracker -- , seq: 747855, time: 2014-06-12 14:46:27.038676, event: throttled, op: osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.038989 7f9d56432700  5 -- op tracker -- , seq: 747855, time: 2014-06-12 14:46:27.038897, event: all_read, op: osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.038998 7f9d56432700  5 -- op tracker -- , seq: 747855, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.039005 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:27.039012 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:27.039038 7f9d56432700 15 osd.2 173 enqueue_op 0x2519d400 prio 127 cost 123687 latency 0.000365 osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:27.039085 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:27.039095 7f9d693e3700 10 osd.2 173 dequeue_op 0x2519d400 prio 127 cost 123687 latency 0.000422 osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2998 (0'0,173'2998] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2997 active]
2014-06-12 14:46:27.039121 7f9d693e3700  5 -- op tracker -- , seq: 747855, time: 2014-06-12 14:46:27.039117, event: reached_pg, op: osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.039144 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2998 (0'0,173'2998] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2997 active] handle_message: 0x2519d400
2014-06-12 14:46:27.039159 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2998 (0'0,173'2998] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2997 active] sub_op_modify trans 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 v 173'2999 (transaction) 174
2014-06-12 14:46:27.039178 7f9d693e3700  5 -- op tracker -- , seq: 747855, time: 2014-06-12 14:46:27.039174, event: started, op: osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.039210 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2998 (0'0,173'2998] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2997 active] append_log log((0'0,173'2998], crt=0'0) [173'2999 (173'2998) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145020 2014-06-12 08:44:48.517750]
2014-06-12 14:46:27.039237 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2999 (0'0,173'2999] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2997 active] add_log_entry 173'2999 (173'2998) modify   491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 by client.4308.0:145020 2014-06-12 08:44:48.517750
2014-06-12 14:46:27.039276 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2999 (0'0,173'2999] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2997 active] append_log  adding 1 keys
2014-06-12 14:46:27.039326 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2999, trimmed: 
2014-06-12 14:46:27.039354 7f9d693e3700  5 -- op tracker -- , seq: 747855, time: 2014-06-12 14:46:27.039349, event: started, op: osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.039366 7f9d693e3700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:27.039371 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:27.039372 7f9d693e3700  5 queue_op 0x2223020 seq 2801 osr(3.458 0x276d960) 125173 bytes   (queue has 1 ops and 125173 bytes)
2014-06-12 14:46:27.039385 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'2999 (0'0,173'2999] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2997 active] do_sub_op osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:27.039399 7f9d693e3700 10 osd.2 173 dequeue_op 0x2519d400 finish
2014-06-12 14:46:27.039402 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:27.039405 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:27.040013 7f9d56432700  5 -- op tracker -- , seq: 747856, time: 2014-06-12 14:46:27.039422, event: header_read, op: osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.040026 7f9d56432700  5 -- op tracker -- , seq: 747856, time: 2014-06-12 14:46:27.039423, event: throttled, op: osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.040034 7f9d56432700  5 -- op tracker -- , seq: 747856, time: 2014-06-12 14:46:27.039982, event: all_read, op: osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.040047 7f9d56432700  5 -- op tracker -- , seq: 747856, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.040052 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:27.040057 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:27.040064 7f9d56432700 15 osd.2 173 enqueue_op 0x2519dc00 prio 127 cost 516903 latency 0.000641 osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:27.040083 7f9d5bec4700  5 -- op tracker -- , seq: 747857, time: 2014-06-12 14:46:27.040046, event: header_read, op: osd_sub_op_reply(client.4311.0:139833 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.040090 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:27.040093 7f9d5bec4700  5 -- op tracker -- , seq: 747857, time: 2014-06-12 14:46:27.040048, event: throttled, op: osd_sub_op_reply(client.4311.0:139833 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.040102 7f9d5bec4700  5 -- op tracker -- , seq: 747857, time: 2014-06-12 14:46:27.040058, event: all_read, op: osd_sub_op_reply(client.4311.0:139833 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.040095 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2519dc00 prio 127 cost 516903 latency 0.000673 osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.67d( v 173'2515 (0'0,173'2515] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2514 active]
2014-06-12 14:46:27.040110 7f9d5bec4700  5 -- op tracker -- , seq: 747857, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4311.0:139833 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.040114 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4311.0:139833 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:27.040116 7f9d6b3e7700  5 -- op tracker -- , seq: 747856, time: 2014-06-12 14:46:27.040112, event: reached_pg, op: osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.040120 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:27.040126 7f9d5bec4700 15 osd.2 173 enqueue_op 0x124e3500 prio 196 cost 0 latency 0.000079 osd_sub_op_reply(client.4311.0:139833 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:27.040123 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2515 (0'0,173'2515] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2514 active] handle_message: 0x2519dc00
2014-06-12 14:46:27.040134 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2515 (0'0,173'2515] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2514 active] sub_op_modify trans a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 v 173'2516 (transaction) 174
2014-06-12 14:46:27.040149 7f9d6b3e7700  5 -- op tracker -- , seq: 747856, time: 2014-06-12 14:46:27.040146, event: started, op: osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.040161 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2515 (0'0,173'2515] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2514 active] append_log log((0'0,173'2515], crt=0'0) [173'2516 (173'2514) modify   a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 by client.4313.0:146360 2014-06-12 08:44:48.915605]
2014-06-12 14:46:27.040226 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2516 (0'0,173'2516] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2514 active] add_log_entry 173'2516 (173'2514) modify   a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 by client.4313.0:146360 2014-06-12 08:44:48.915605
2014-06-12 14:46:27.040252 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2516 (0'0,173'2516] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2514 active] append_log  adding 1 keys
2014-06-12 14:46:27.040282 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2516, trimmed: 
2014-06-12 14:46:27.040303 7f9d6b3e7700  5 -- op tracker -- , seq: 747856, time: 2014-06-12 14:46:27.040299, event: started, op: osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.040312 7f9d6b3e7700  5 queue_transactions existing osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:27.040315 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:27.040317 7f9d6b3e7700  5 queue_op 0x14bc61e0 seq 2294 osr(3.67d 0x276df70) 518389 bytes   (queue has 2 ops and 643562 bytes)
2014-06-12 14:46:27.040322 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.67d( v 173'2516 (0'0,173'2516] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2514 active] do_sub_op osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:27.040336 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2519dc00 finish
2014-06-12 14:46:27.040339 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:27.040345 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:27.040349 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x124e3500 prio 196 cost 0 latency 0.000303 osd_sub_op_reply(client.4311.0:139833 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2736 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean]
2014-06-12 14:46:27.040365 7f9d6b3e7700  5 -- op tracker -- , seq: 747857, time: 2014-06-12 14:46:27.040362, event: reached_pg, op: osd_sub_op_reply(client.4311.0:139833 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.040371 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2736 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] handle_message: 0x124e3500
2014-06-12 14:46:27.040384 7f9d6b3e7700  5 -- op tracker -- , seq: 747857, time: 2014-06-12 14:46:27.040381, event: started, op: osd_sub_op_reply(client.4311.0:139833 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.040389 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2736 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] sub_op_modify_reply: tid 236269 op  ack_type 4 from 25
2014-06-12 14:46:27.040409 7f9d6b3e7700  5 -- op tracker -- , seq: 747850, time: 2014-06-12 14:46:27.040401, event: sub_op_commit_rec, op: osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.040416 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2736 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] repop_all_applied: repop tid 236269 all applied 
2014-06-12 14:46:27.040426 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2736 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] eval_repop repgather(0x5237740 173'2737 rep_tid=236269 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:27.040448 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2736 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean]  sending ack on repgather(0x5237740 173'2737 rep_tid=236269 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) 0x353e8c80
2014-06-12 14:46:27.040479 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2736 crt=0'0 lcod 173'2735 mlcod 173'2735 active+clean] repop_all_committed: repop tid 236269 all committed 
2014-06-12 14:46:27.040489 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2735 active+clean] eval_repop repgather(0x5237740 173'2737 rep_tid=236269 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:27.040503 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2735 active+clean] log_op_stats osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.127583 lat 0.127607
2014-06-12 14:46:27.040518 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2735 active+clean] publish_stats_to_osd 173:3635
2014-06-12 14:46:27.040530 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2735 active+clean]  sending commit on repgather(0x5237740 173'2737 rep_tid=236269 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) 0x75cca00
2014-06-12 14:46:27.040555 7f9d6b3e7700  5 -- op tracker -- , seq: 747850, time: 2014-06-12 14:46:27.040551, event: commit_sent, op: osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.040562 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2735 active+clean]  requeue_ops 
2014-06-12 14:46:27.040571 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean]  removing repgather(0x5237740 173'2737 rep_tid=236269 committed?=1 applied?=1 op=osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:27.040584 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean]    q front is repgather(0x5237740 173'2737 rep_tid=236269 committed?=1 applied?=1 op=osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:27.040596 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] remove_repop repgather(0x5237740 173'2737 rep_tid=236269 committed?=1 applied?=1 op=osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:27.040608 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean]  requeue_ops 
2014-06-12 14:46:27.040636 7f9d6b3e7700  5 -- op tracker -- , seq: 747850, time: 2014-06-12 14:46:27.040631, event: done, op: osd_op(client.4311.0:139833 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 1036288~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.040651 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x124e3500 finish
2014-06-12 14:46:27.040657 7f9d6b3e7700  5 -- op tracker -- , seq: 747857, time: 2014-06-12 14:46:27.040654, event: done, op: osd_sub_op_reply(client.4311.0:139833 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.040666 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:27.040669 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:27.040965 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:27.040972 7f9d7a962700  5 _do_op 0x2223020 seq 2801 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:27.040977 7f9d7a962700 10 _do_transaction on 0x26839b30
2014-06-12 14:46:27.040984 7f9d7a962700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:27.041047 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x524cb40 (2 active)
2014-06-12 14:46:27.041054 7f9d7895e700  5 _do_op 0x14bc61e0 seq 2294 osr(3.67d 0x276df70)/0x276df70 start
2014-06-12 14:46:27.041058 7f9d7895e700 10 _do_transaction on 0x508b430
2014-06-12 14:46:27.041063 7f9d7895e700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:27.041135 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:27.041160 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:27.041216 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:27.041238 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:27.041289 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:27.041311 7f9d7895e700 15 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:27.041316 7f9d7895e700 10 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1 r = 0
2014-06-12 14:46:27.041321 7f9d7895e700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:27.041336 7f9d7895e700 15 _write 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 2416640~516096
2014-06-12 14:46:27.041636 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:27.042015 7f9d7895e700 10 lookup_strip_header done  cid 3.67d_head oid a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:27.042184 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:27.043206 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:27.043209 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:27.043210 7f9d74bfa700 20 osd.2 173 _dispatch 0x646d860 ping v1
2014-06-12 14:46:27.043213 7f9d74bfa700 10 osd.2 173 ping from client.4317
2014-06-12 14:46:27.043215 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:27.043216 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:27.044054 7f9d7895e700 10 _generic_write 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 2416640~516096 = 0
2014-06-12 14:46:27.044097 7f9d7895e700 15 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:27.044113 7f9d7895e700 10 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 = 0
2014-06-12 14:46:27.044119 7f9d7895e700 15 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:27.044126 7f9d7895e700 10 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 = 0
2014-06-12 14:46:27.044321 7f9d7895e700 20 set_header setting 17386 cid 3.67d_head oid a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 parent seq 0
2014-06-12 14:46:27.044395 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:27.044408 7f9d7895e700 20 set_header setting 2272 cid meta oid 1fe0e10b/pglog_3.67d/0//-1 parent seq 0
2014-06-12 14:46:27.044417 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:27.047527 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:27.047561 7f9d7a962700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:27.047567 7f9d7a962700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:27.047574 7f9d7a962700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:27.047683 7f9d7a962700 15 _write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 4071424~122880
2014-06-12 14:46:27.047861 7f9d7a962700 10 lookup_strip_header done  cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:27.047997 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:27.048426 7f9d7a962700 10 _generic_write 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 4071424~122880 = 0
2014-06-12 14:46:27.048448 7f9d7a962700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:27.048460 7f9d7a962700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:27.048466 7f9d7a962700 15 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3
2014-06-12 14:46:27.048474 7f9d7a962700 10 _setattrs 3.458_head/491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 = 0
2014-06-12 14:46:27.048601 7f9d7a962700 20 set_header setting 16936 cid 3.458_head oid 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 parent seq 0
2014-06-12 14:46:27.048651 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:27.048664 7f9d7a962700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:27.048675 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:27.049281 7f9d7a962700 10 _do_op 0x2223020 seq 2801 r = 0, finisher 0x330652e0 0
2014-06-12 14:46:27.049298 7f9d7a962700 10 _finish_op 0x2223020 seq 2801 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:27.049307 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (1 active)
2014-06-12 14:46:27.049311 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:27.049336 7f9d7715b700  5 -- op tracker -- , seq: 747855, time: 2014-06-12 14:46:27.049326, event: commit_sent, op: osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.049351 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2999 (0'0,173'2999] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2997 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:27.049381 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:27.049416 7f9d7795c700  5 -- op tracker -- , seq: 747855, time: 2014-06-12 14:46:27.049411, event: sub_op_applied, op: osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.049423 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2999 (0'0,173'2999] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2998 active] sub_op_modify_applied on 0x26839a40 op osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:27.049439 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2999 (0'0,173'2999] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2998 active] op_applied on primary on version 173'2999
2014-06-12 14:46:27.049449 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2999 (0'0,173'2999] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2998 active] op_applied on replica on version 173'2999
2014-06-12 14:46:27.049475 7f9d7795c700  5 -- op tracker -- , seq: 747855, time: 2014-06-12 14:46:27.049470, event: done, op: osd_sub_op(client.4308.0:145020 3.458 491dbc58/rbd_data.10ab6b8b4567.0000000000000f79/head//3 [] v 173'2999 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.049998 7f9d7895e700 10 _do_op 0x14bc61e0 seq 2294 r = 0, finisher 0x7bd5620 0
2014-06-12 14:46:27.050045 7f9d7715b700  5 -- op tracker -- , seq: 747856, time: 2014-06-12 14:46:27.050037, event: commit_sent, op: osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.050054 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2516 (0'0,173'2516] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2514 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:27.050074 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:27.050148 7f9d7895e700 10 _finish_op 0x14bc61e0 seq 2294 osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:27.050184 7f9d7795c700  5 -- op tracker -- , seq: 747856, time: 2014-06-12 14:46:27.050178, event: sub_op_applied, op: osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.050191 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2516 (0'0,173'2516] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2515 active] sub_op_modify_applied on 0x508b340 op osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:27.050206 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2516 (0'0,173'2516] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2515 active] op_applied on primary on version 173'2516
2014-06-12 14:46:27.050215 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2516 (0'0,173'2516] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2515 active] op_applied on replica on version 173'2516
2014-06-12 14:46:27.050291 7f9d7895e700  5 -- op tracker -- , seq: 747856, time: 2014-06-12 14:46:27.050280, event: done, op: osd_sub_op(client.4313.0:146360 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:27.050335 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x524cb40 (0 active)
2014-06-12 14:46:27.050357 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:27.054935 7f9d44ece700  5 -- op tracker -- , seq: 747858, time: 2014-06-12 14:46:27.046752, event: header_read, op: osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.054949 7f9d44ece700  5 -- op tracker -- , seq: 747858, time: 2014-06-12 14:46:27.046755, event: throttled, op: osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.054959 7f9d44ece700  5 -- op tracker -- , seq: 747858, time: 2014-06-12 14:46:27.054890, event: all_read, op: osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.054968 7f9d44ece700  5 -- op tracker -- , seq: 747858, time: 0.000000, event: dispatched, op: osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.054979 7f9d44ece700 20 osd.2 173 should_share_map client.4311 172.16.96.220:0/1244240 173
2014-06-12 14:46:27.055025 7f9d44ece700 15 osd.2 173 enqueue_op 0x1702e700 prio 63 cost 1032192 latency 0.008273 osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:27.057654 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:27.057669 7f9d683e1700 10 osd.2 173 dequeue_op 0x1702e700 prio 63 cost 1032192 latency 0.010917 osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean]
2014-06-12 14:46:27.057710 7f9d683e1700  5 -- op tracker -- , seq: 747858, time: 2014-06-12 14:46:27.057704, event: reached_pg, op: osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.057724 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:27.057740 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] handle_message: 0x1702e700
2014-06-12 14:46:27.057750 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] do_op osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:27.057776 7f9d683e1700 15 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 '_'
2014-06-12 14:46:27.058036 7f9d683e1700 10 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 '_' = 0
2014-06-12 14:46:27.058052 7f9d683e1700 15 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 'snapset'
2014-06-12 14:46:27.058138 7f9d683e1700 10 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 'snapset' = 0
2014-06-12 14:46:27.058146 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] populate_obc_watchers aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.058157 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.058166 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] get_object_context: creating obc from disk: 0x14e7b8c0
2014-06-12 14:46:27.058175 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] get_object_context: 0x14e7b8c0 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 rwstate(none n=0 w=0) oi: aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3(173'2737 client.4311.0:139833 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2737) ssc: 0x24d4b880 snapset: 0=[]:[]+head
2014-06-12 14:46:27.058189 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] find_object_context aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 @head oi=aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3(173'2737 client.4311.0:139833 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2737)
2014-06-12 14:46:27.058213 7f9d683e1700  5 -- op tracker -- , seq: 747858, time: 2014-06-12 14:46:27.058207, event: started, op: osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.058220 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] execute_ctx 0x5d21e00
2014-06-12 14:46:27.058236 7f9d683e1700  5 -- op tracker -- , seq: 747858, time: 2014-06-12 14:46:27.058231, event: started, op: osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.058242 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] do_op aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] ov 173'2737 av 173'2738 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:27.058255 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] do_osd_op aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192]
2014-06-12 14:46:27.058265 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:27.058277 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] do_osd_op  write 2068480~1032192
2014-06-12 14:46:27.058292 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] do_osd_op_effects on session 0x5a1d800
2014-06-12 14:46:27.058302 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] make_writeable aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 snapset=0x24d4b8b8  snapc=0=[]
2014-06-12 14:46:27.058313 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] make_writeable aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:27.058322 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] finish_ctx aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 0x5d21e00 op modify  
2014-06-12 14:46:27.058333 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean]  set mtime to 2014-06-12 08:45:29.423744
2014-06-12 14:46:27.058352 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean]  final snapset 0=[]:[]+head in aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.058375 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean]  zeroing write result code 0
2014-06-12 14:46:27.058385 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] new_repop rep_tid 236270 on osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:27.058400 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] issue_repop rep_tid 236270 o aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.058423 7f9d683e1700  5 -- op tracker -- , seq: 747858, time: 2014-06-12 14:46:27.058418, event: waiting for subops from 25, op: osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.058444 7f9d683e1700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:27.058464 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2737 (0'0,173'2737] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] append_log log((0'0,173'2737], crt=0'0) [173'2738 (173'2737) modify   aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 by client.4311.0:139834 2014-06-12 08:45:29.423744]
2014-06-12 14:46:27.058486 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2737 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] add_log_entry 173'2738 (173'2737) modify   aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 by client.4311.0:139834 2014-06-12 08:45:29.423744
2014-06-12 14:46:27.058505 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2737 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] append_log  adding 1 keys
2014-06-12 14:46:27.058537 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2738, trimmed: 
2014-06-12 14:46:27.058565 7f9d683e1700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:27.058569 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:27.058570 7f9d683e1700  5 queue_op 0x48789c0 seq 2550 osr(3.122 0x276cf40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:27.058580 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2737 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] eval_repop repgather(0x519c480 173'2738 rep_tid=236270 committed?=0 applied?=0 lock=0 op=osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:27.058597 7f9d683e1700 10 osd.2 173 dequeue_op 0x1702e700 finish
2014-06-12 14:46:27.058602 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:27.058606 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:27.061628 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:46:27.061641 7f9d7a161700  5 _do_op 0x48789c0 seq 2550 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:46:27.061647 7f9d7a161700 10 _do_transaction on 0x519ca80
2014-06-12 14:46:27.061655 7f9d7a161700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:27.061804 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:27.061828 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:27.061888 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:27.061913 7f9d7a161700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:27.061921 7f9d7a161700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:46:27.061929 7f9d7a161700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:27.061950 7f9d7a161700 15 _write 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 2068480~1032192
2014-06-12 14:46:27.062108 7f9d7a161700 10 lookup_strip_header done  cid 3.122_head oid aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.062299 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:27.086559 7f9d7a161700 10 _generic_write 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 2068480~1032192 = 0
2014-06-12 14:46:27.086764 7f9d7a161700 15 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.086804 7f9d7a161700 10 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:27.086832 7f9d7a161700 15 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.086865 7f9d7a161700 10 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:27.087246 7f9d7a161700 20 set_header setting 16189 cid 3.122_head oid aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 parent seq 0
2014-06-12 14:46:27.087372 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:27.087408 7f9d7a161700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:46:27.087439 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:27.097863 7f9d7a161700 10 _do_op 0x48789c0 seq 2550 r = 0, finisher 0x2a3d640 0x4e2a780
2014-06-12 14:46:27.097886 7f9d7a161700 10 _finish_op 0x48789c0 seq 2550 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:27.097896 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:46:27.097906 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:27.097934 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2737 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] op_applied: 236270
2014-06-12 14:46:27.097969 7f9d7795c700  5 -- op tracker -- , seq: 747858, time: 2014-06-12 14:46:27.097958, event: op_applied, op: osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.097981 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2737 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] op_applied on primary on version 173'2738
2014-06-12 14:46:27.098009 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2737 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] op_commit: 236270
2014-06-12 14:46:27.098033 7f9d7715b700  5 -- op tracker -- , seq: 747858, time: 2014-06-12 14:46:27.098028, event: op_commit, op: osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.174025 7f9d5bec4700  5 -- op tracker -- , seq: 747859, time: 2014-06-12 14:46:27.173977, event: header_read, op: osd_sub_op_reply(client.4311.0:139834 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.174040 7f9d5bec4700  5 -- op tracker -- , seq: 747859, time: 2014-06-12 14:46:27.173979, event: throttled, op: osd_sub_op_reply(client.4311.0:139834 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.174048 7f9d5bec4700  5 -- op tracker -- , seq: 747859, time: 2014-06-12 14:46:27.173990, event: all_read, op: osd_sub_op_reply(client.4311.0:139834 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.174057 7f9d5bec4700  5 -- op tracker -- , seq: 747859, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4311.0:139834 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.174062 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4311.0:139834 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:27.174068 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:27.174076 7f9d5bec4700 15 osd.2 173 enqueue_op 0x124e2200 prio 196 cost 0 latency 0.000099 osd_sub_op_reply(client.4311.0:139834 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:27.177645 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:27.177664 7f9d65bdc700 10 osd.2 173 dequeue_op 0x124e2200 prio 196 cost 0 latency 0.003687 osd_sub_op_reply(client.4311.0:139834 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2737 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean]
2014-06-12 14:46:27.177696 7f9d65bdc700  5 -- op tracker -- , seq: 747859, time: 2014-06-12 14:46:27.177692, event: reached_pg, op: osd_sub_op_reply(client.4311.0:139834 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.177709 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2737 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] handle_message: 0x124e2200
2014-06-12 14:46:27.177728 7f9d65bdc700  5 -- op tracker -- , seq: 747859, time: 2014-06-12 14:46:27.177724, event: started, op: osd_sub_op_reply(client.4311.0:139834 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.177735 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2737 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] sub_op_modify_reply: tid 236270 op  ack_type 4 from 25
2014-06-12 14:46:27.177761 7f9d65bdc700  5 -- op tracker -- , seq: 747858, time: 2014-06-12 14:46:27.177751, event: sub_op_commit_rec, op: osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.177772 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2737 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] repop_all_applied: repop tid 236270 all applied 
2014-06-12 14:46:27.177783 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2737 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] eval_repop repgather(0x519c480 173'2738 rep_tid=236270 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:27.177802 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2737 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean]  sending ack on repgather(0x519c480 173'2738 rep_tid=236270 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) 0x31452500
2014-06-12 14:46:27.177841 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2737 crt=0'0 lcod 173'2736 mlcod 173'2736 active+clean] repop_all_committed: repop tid 236270 all committed 
2014-06-12 14:46:27.177854 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2736 active+clean] eval_repop repgather(0x519c480 173'2738 rep_tid=236270 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:27.177871 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2736 active+clean] log_op_stats osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.131087 lat 0.131117
2014-06-12 14:46:27.177888 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2736 active+clean] publish_stats_to_osd 173:3636
2014-06-12 14:46:27.177908 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2736 active+clean]  sending commit on repgather(0x519c480 173'2738 rep_tid=236270 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) 0x34d90c80
2014-06-12 14:46:27.177941 7f9d65bdc700  5 -- op tracker -- , seq: 747858, time: 2014-06-12 14:46:27.177936, event: commit_sent, op: osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.177949 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2736 active+clean]  requeue_ops 
2014-06-12 14:46:27.177960 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean]  removing repgather(0x519c480 173'2738 rep_tid=236270 committed?=1 applied?=1 op=osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:27.177975 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean]    q front is repgather(0x519c480 173'2738 rep_tid=236270 committed?=1 applied?=1 op=osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:27.177989 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] remove_repop repgather(0x519c480 173'2738 rep_tid=236270 committed?=1 applied?=1 op=osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:27.178003 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean]  requeue_ops 
2014-06-12 14:46:27.178030 7f9d65bdc700  5 -- op tracker -- , seq: 747858, time: 2014-06-12 14:46:27.178025, event: done, op: osd_op(client.4311.0:139834 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2068480~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.178061 7f9d65bdc700 10 osd.2 173 dequeue_op 0x124e2200 finish
2014-06-12 14:46:27.178069 7f9d65bdc700  5 -- op tracker -- , seq: 747859, time: 2014-06-12 14:46:27.178065, event: done, op: osd_sub_op_reply(client.4311.0:139834 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.178080 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:27.178084 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:27.185761 7f9d44ece700  5 -- op tracker -- , seq: 747860, time: 2014-06-12 14:46:27.184558, event: header_read, op: osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.185780 7f9d44ece700  5 -- op tracker -- , seq: 747860, time: 2014-06-12 14:46:27.184561, event: throttled, op: osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.185789 7f9d44ece700  5 -- op tracker -- , seq: 747860, time: 2014-06-12 14:46:27.185701, event: all_read, op: osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.185799 7f9d44ece700  5 -- op tracker -- , seq: 747860, time: 0.000000, event: dispatched, op: osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.185807 7f9d44ece700 20 osd.2 173 should_share_map client.4311 172.16.96.220:0/1244240 173
2014-06-12 14:46:27.185835 7f9d44ece700 15 osd.2 173 enqueue_op 0x1702eb00 prio 63 cost 1032192 latency 0.001277 osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:27.189665 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:27.189690 7f9d64bda700 10 osd.2 173 dequeue_op 0x1702eb00 prio 63 cost 1032192 latency 0.005131 osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean]
2014-06-12 14:46:27.189731 7f9d64bda700  5 -- op tracker -- , seq: 747860, time: 2014-06-12 14:46:27.189724, event: reached_pg, op: osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.189749 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:27.189766 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] handle_message: 0x1702eb00
2014-06-12 14:46:27.189777 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] do_op osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:27.189814 7f9d64bda700 15 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 '_'
2014-06-12 14:46:27.190086 7f9d64bda700 10 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 '_' = 0
2014-06-12 14:46:27.190106 7f9d64bda700 15 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 'snapset'
2014-06-12 14:46:27.190205 7f9d64bda700 10 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 'snapset' = 0
2014-06-12 14:46:27.190220 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] populate_obc_watchers aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.190234 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.190258 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] get_object_context: creating obc from disk: 0x216a9600
2014-06-12 14:46:27.190269 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] get_object_context: 0x216a9600 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 rwstate(none n=0 w=0) oi: aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3(173'2738 client.4311.0:139834 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2738) ssc: 0x16ee50a0 snapset: 0=[]:[]+head
2014-06-12 14:46:27.190289 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] find_object_context aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 @head oi=aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3(173'2738 client.4311.0:139834 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2738)
2014-06-12 14:46:27.190325 7f9d64bda700  5 -- op tracker -- , seq: 747860, time: 2014-06-12 14:46:27.190318, event: started, op: osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.190334 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] execute_ctx 0xfd19e00
2014-06-12 14:46:27.190352 7f9d64bda700  5 -- op tracker -- , seq: 747860, time: 2014-06-12 14:46:27.190347, event: started, op: osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.190359 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] do_op aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] ov 173'2738 av 173'2739 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:27.190373 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] do_osd_op aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192]
2014-06-12 14:46:27.190384 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:27.190398 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] do_osd_op  write 3100672~1032192
2014-06-12 14:46:27.190419 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] do_osd_op_effects on session 0x5a1d800
2014-06-12 14:46:27.190432 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] make_writeable aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 snapset=0x16ee50d8  snapc=0=[]
2014-06-12 14:46:27.190447 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] make_writeable aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:27.190458 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] finish_ctx aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 0xfd19e00 op modify  
2014-06-12 14:46:27.190474 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean]  set mtime to 2014-06-12 08:45:29.560835
2014-06-12 14:46:27.190512 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean]  final snapset 0=[]:[]+head in aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.190532 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean]  zeroing write result code 0
2014-06-12 14:46:27.190543 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] new_repop rep_tid 236271 on osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:27.190560 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] issue_repop rep_tid 236271 o aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.190586 7f9d64bda700  5 -- op tracker -- , seq: 747860, time: 2014-06-12 14:46:27.190581, event: waiting for subops from 25, op: osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.190608 7f9d64bda700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:27.190637 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2738 (0'0,173'2738] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] append_log log((0'0,173'2738], crt=0'0) [173'2739 (173'2738) modify   aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 by client.4311.0:139835 2014-06-12 08:45:29.560835]
2014-06-12 14:46:27.190664 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2738 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] add_log_entry 173'2739 (173'2738) modify   aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 by client.4311.0:139835 2014-06-12 08:45:29.560835
2014-06-12 14:46:27.190691 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2738 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] append_log  adding 1 keys
2014-06-12 14:46:27.190738 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2739, trimmed: 
2014-06-12 14:46:27.190774 7f9d64bda700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:27.190780 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:27.190783 7f9d64bda700  5 queue_op 0x581d680 seq 2551 osr(3.122 0x276cf40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:27.190796 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2738 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] eval_repop repgather(0x12ae2540 173'2739 rep_tid=236271 committed?=0 applied?=0 lock=0 op=osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:27.190820 7f9d64bda700 10 osd.2 173 dequeue_op 0x1702eb00 finish
2014-06-12 14:46:27.190824 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:27.190829 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:27.191901 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:46:27.191938 7f9d7f16b700  5 _do_op 0x581d680 seq 2551 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:46:27.191960 7f9d7f16b700 10 _do_transaction on 0x2c3c6000
2014-06-12 14:46:27.191983 7f9d7f16b700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:27.192144 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:27.192185 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:27.192264 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:27.192308 7f9d7f16b700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:27.192333 7f9d7f16b700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:46:27.192356 7f9d7f16b700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:27.192395 7f9d7f16b700 15 _write 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 3100672~1032192
2014-06-12 14:46:27.192701 7f9d7f16b700 10 lookup_strip_header done  cid 3.122_head oid aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.192845 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:27.197024 7f9d7f16b700 10 _generic_write 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 3100672~1032192 = 0
2014-06-12 14:46:27.197161 7f9d7f16b700 15 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.197218 7f9d7f16b700 10 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:27.197243 7f9d7f16b700 15 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.197266 7f9d7f16b700 10 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:27.197569 7f9d7f16b700 20 set_header setting 16189 cid 3.122_head oid aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 parent seq 0
2014-06-12 14:46:27.204569 7f9d5bec4700  5 -- op tracker -- , seq: 747861, time: 2014-06-12 14:46:27.204502, event: header_read, op: osd_sub_op_reply(client.4311.0:139835 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.204589 7f9d5bec4700  5 -- op tracker -- , seq: 747861, time: 2014-06-12 14:46:27.204505, event: throttled, op: osd_sub_op_reply(client.4311.0:139835 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.204602 7f9d5bec4700  5 -- op tracker -- , seq: 747861, time: 2014-06-12 14:46:27.204525, event: all_read, op: osd_sub_op_reply(client.4311.0:139835 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.204614 7f9d5bec4700  5 -- op tracker -- , seq: 747861, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4311.0:139835 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.204620 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4311.0:139835 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:27.204627 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:27.204636 7f9d5bec4700 15 osd.2 173 enqueue_op 0x124e2c00 prio 196 cost 0 latency 0.000134 osd_sub_op_reply(client.4311.0:139835 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:27.204742 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:27.204752 7f9d6abe6700 10 osd.2 173 dequeue_op 0x124e2c00 prio 196 cost 0 latency 0.000249 osd_sub_op_reply(client.4311.0:139835 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2738 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean]
2014-06-12 14:46:27.204778 7f9d6abe6700  5 -- op tracker -- , seq: 747861, time: 2014-06-12 14:46:27.204774, event: reached_pg, op: osd_sub_op_reply(client.4311.0:139835 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.204789 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2738 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] handle_message: 0x124e2c00
2014-06-12 14:46:27.204808 7f9d6abe6700  5 -- op tracker -- , seq: 747861, time: 2014-06-12 14:46:27.204805, event: started, op: osd_sub_op_reply(client.4311.0:139835 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.204814 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2738 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] sub_op_modify_reply: tid 236271 op  ack_type 4 from 25
2014-06-12 14:46:27.204836 7f9d6abe6700  5 -- op tracker -- , seq: 747860, time: 2014-06-12 14:46:27.204828, event: sub_op_commit_rec, op: osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.204843 7f9d6abe6700 10 osd.2 173 dequeue_op 0x124e2c00 finish
2014-06-12 14:46:27.204848 7f9d6abe6700  5 -- op tracker -- , seq: 747861, time: 2014-06-12 14:46:27.204846, event: done, op: osd_sub_op_reply(client.4311.0:139835 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.204861 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:27.204864 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:27.205415 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:27.205444 7f9d7f16b700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:46:27.205458 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:27.211509 7f9d7f16b700 10 _do_op 0x581d680 seq 2551 r = 0, finisher 0x5956940 0x2d175640
2014-06-12 14:46:27.211562 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2738 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] op_commit: 236271
2014-06-12 14:46:27.211603 7f9d7715b700  5 -- op tracker -- , seq: 747860, time: 2014-06-12 14:46:27.211591, event: op_commit, op: osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.211642 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2738 crt=0'0 lcod 173'2737 mlcod 173'2737 active+clean] repop_all_committed: repop tid 236271 all committed 
2014-06-12 14:46:27.211656 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2737 active+clean] eval_repop repgather(0x12ae2540 173'2739 rep_tid=236271 committed?=1 applied?=0 lock=0 op=osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:27.211682 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2737 active+clean] log_op_stats osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.000000 lat 0.027121
2014-06-12 14:46:27.211709 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2737 active+clean] publish_stats_to_osd 173:3637
2014-06-12 14:46:27.211726 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2737 active+clean]  sending commit on repgather(0x12ae2540 173'2739 rep_tid=236271 committed?=1 applied?=0 lock=0 op=osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) 0x151e7400
2014-06-12 14:46:27.211777 7f9d7715b700  5 -- op tracker -- , seq: 747860, time: 2014-06-12 14:46:27.211771, event: commit_sent, op: osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.211908 7f9d7f16b700 10 _finish_op 0x581d680 seq 2551 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:27.211918 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:46:27.211922 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:27.211923 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2737 active+clean] op_applied: 236271
2014-06-12 14:46:27.211951 7f9d7795c700  5 -- op tracker -- , seq: 747860, time: 2014-06-12 14:46:27.211942, event: op_applied, op: osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.211960 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2737 active+clean] op_applied on primary on version 173'2739
2014-06-12 14:46:27.211969 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2737 active+clean] repop_all_applied: repop tid 236271 all applied 
2014-06-12 14:46:27.211977 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2737 active+clean] eval_repop repgather(0x12ae2540 173'2739 rep_tid=236271 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:27.211992 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2737 active+clean] log_op_stats osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.000000 lat 0.027433
2014-06-12 14:46:27.212005 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2737 active+clean] publish_stats_to_osd 173:3638
2014-06-12 14:46:27.212015 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2737 active+clean]  requeue_ops 
2014-06-12 14:46:27.212029 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]  removing repgather(0x12ae2540 173'2739 rep_tid=236271 committed?=1 applied?=1 op=osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:27.212042 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]    q front is repgather(0x12ae2540 173'2739 rep_tid=236271 committed?=1 applied?=1 op=osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:27.212055 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] remove_repop repgather(0x12ae2540 173'2739 rep_tid=236271 committed?=1 applied?=1 op=osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:27.212067 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]  requeue_ops 
2014-06-12 14:46:27.212092 7f9d7795c700  5 -- op tracker -- , seq: 747860, time: 2014-06-12 14:46:27.212088, event: done, op: osd_op(client.4311.0:139835 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3100672~1032192] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.213776 7f9d44ece700  5 -- op tracker -- , seq: 747862, time: 2014-06-12 14:46:27.213649, event: header_read, op: osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.213790 7f9d44ece700  5 -- op tracker -- , seq: 747862, time: 2014-06-12 14:46:27.213651, event: throttled, op: osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.213798 7f9d44ece700  5 -- op tracker -- , seq: 747862, time: 2014-06-12 14:46:27.213731, event: all_read, op: osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.213806 7f9d44ece700  5 -- op tracker -- , seq: 747862, time: 0.000000, event: dispatched, op: osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.213813 7f9d44ece700 20 osd.2 173 should_share_map client.4311 172.16.96.220:0/1244240 173
2014-06-12 14:46:27.213841 7f9d44ece700 15 osd.2 173 enqueue_op 0x1702f800 prio 63 cost 61440 latency 0.000191 osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:27.213873 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:27.213880 7f9d66bde700 10 osd.2 173 dequeue_op 0x1702f800 prio 63 cost 61440 latency 0.000231 osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]
2014-06-12 14:46:27.213904 7f9d66bde700  5 -- op tracker -- , seq: 747862, time: 2014-06-12 14:46:27.213899, event: reached_pg, op: osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.213912 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:27.213926 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] handle_message: 0x1702f800
2014-06-12 14:46:27.213935 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_op osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:27.213959 7f9d66bde700 15 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 '_'
2014-06-12 14:46:27.214195 7f9d66bde700 10 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 '_' = 0
2014-06-12 14:46:27.214212 7f9d66bde700 15 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 'snapset'
2014-06-12 14:46:27.214301 7f9d66bde700 10 getattr 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 'snapset' = 0
2014-06-12 14:46:27.214310 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] populate_obc_watchers aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.214320 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.214330 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] get_object_context: creating obc from disk: 0x29c1e580
2014-06-12 14:46:27.214347 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] get_object_context: 0x29c1e580 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 rwstate(none n=0 w=0) oi: aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3(173'2739 client.4311.0:139835 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2739) ssc: 0x16ee4d20 snapset: 0=[]:[]+head
2014-06-12 14:46:27.214361 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] find_object_context aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 @head oi=aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3(173'2739 client.4311.0:139835 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2739)
2014-06-12 14:46:27.214384 7f9d66bde700  5 -- op tracker -- , seq: 747862, time: 2014-06-12 14:46:27.214379, event: started, op: osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.214392 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] execute_ctx 0xf3f8a00
2014-06-12 14:46:27.214408 7f9d66bde700  5 -- op tracker -- , seq: 747862, time: 2014-06-12 14:46:27.214403, event: started, op: osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.214414 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_op aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] ov 173'2739 av 173'2740 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:27.214426 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_osd_op aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440]
2014-06-12 14:46:27.214436 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:27.214450 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_osd_op  write 4132864~61440
2014-06-12 14:46:27.214464 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_osd_op_effects on session 0x5a1d800
2014-06-12 14:46:27.214474 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] make_writeable aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 snapset=0x16ee4d58  snapc=0=[]
2014-06-12 14:46:27.214485 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] make_writeable aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:27.214494 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] finish_ctx aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 0xf3f8a00 op modify  
2014-06-12 14:46:27.214505 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]  set mtime to 2014-06-12 08:45:29.594069
2014-06-12 14:46:27.214524 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]  final snapset 0=[]:[]+head in aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.214541 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]  zeroing write result code 0
2014-06-12 14:46:27.214551 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] new_repop rep_tid 236272 on osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:27.214566 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] issue_repop rep_tid 236272 o aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.214590 7f9d66bde700  5 -- op tracker -- , seq: 747862, time: 2014-06-12 14:46:27.214585, event: waiting for subops from 25, op: osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.214614 7f9d66bde700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:27.214635 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2739 (0'0,173'2739] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] append_log log((0'0,173'2739], crt=0'0) [173'2740 (173'2739) modify   aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 by client.4311.0:139836 2014-06-12 08:45:29.594069]
2014-06-12 14:46:27.214657 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] add_log_entry 173'2740 (173'2739) modify   aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 by client.4311.0:139836 2014-06-12 08:45:29.594069
2014-06-12 14:46:27.214675 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] append_log  adding 1 keys
2014-06-12 14:46:27.214707 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2740, trimmed: 
2014-06-12 14:46:27.214729 7f9d66bde700  5 queue_transactions existing osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:27.214733 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:27.214734 7f9d66bde700  5 queue_op 0x568e840 seq 2552 osr(3.122 0x276cf40) 63733 bytes   (queue has 1 ops and 63733 bytes)
2014-06-12 14:46:27.214741 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] eval_repop repgather(0x15e57440 173'2740 rep_tid=236272 committed?=0 applied?=0 lock=0 op=osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:27.214757 7f9d66bde700 10 osd.2 173 dequeue_op 0x1702f800 finish
2014-06-12 14:46:27.214761 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:27.214764 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:27.217584 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5464d80 (1 active)
2014-06-12 14:46:27.217607 7f9d7e169700  5 _do_op 0x568e840 seq 2552 osr(3.122 0x276cf40)/0x276cf40 start
2014-06-12 14:46:27.217633 7f9d7e169700 10 _do_transaction on 0x2a12f2c0
2014-06-12 14:46:27.217642 7f9d7e169700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:27.217818 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:27.217836 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:27.217889 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:27.217999 7f9d7e169700 15 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:27.218010 7f9d7e169700 10 _omap_rmkeys meta/1fe289ca/pglog_3.122/0//-1 r = 0
2014-06-12 14:46:27.218016 7f9d7e169700 15 _omap_setkeys meta/1fe289ca/pglog_3.122/0//-1
2014-06-12 14:46:27.218030 7f9d7e169700 15 _write 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 4132864~61440
2014-06-12 14:46:27.218183 7f9d7e169700 10 lookup_strip_header done  cid 3.122_head oid aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.218195 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:27.218396 7f9d7e169700 10 _generic_write 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 4132864~61440 = 0
2014-06-12 14:46:27.218408 7f9d7e169700 15 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.218416 7f9d7e169700 10 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:27.218421 7f9d7e169700 15 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:27.218428 7f9d7e169700 10 _setattrs 3.122_head/aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:27.218553 7f9d7e169700 20 set_header setting 16189 cid 3.122_head oid aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 parent seq 0
2014-06-12 14:46:27.218575 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:27.218585 7f9d7e169700 20 set_header setting 1992 cid meta oid 1fe289ca/pglog_3.122/0//-1 parent seq 0
2014-06-12 14:46:27.218596 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:27.218989 7f9d7e169700 10 _do_op 0x568e840 seq 2552 r = 0, finisher 0x8356ea0 0x201c2200
2014-06-12 14:46:27.219001 7f9d7e169700 10 _finish_op 0x568e840 seq 2552 osr(3.122 0x276cf40)/0x276cf40
2014-06-12 14:46:27.219010 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5464d80 (0 active)
2014-06-12 14:46:27.219013 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:27.219017 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] op_commit: 236272
2014-06-12 14:46:27.219052 7f9d7715b700  5 -- op tracker -- , seq: 747862, time: 2014-06-12 14:46:27.219043, event: op_commit, op: osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.219131 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] op_applied: 236272
2014-06-12 14:46:27.219154 7f9d7795c700  5 -- op tracker -- , seq: 747862, time: 2014-06-12 14:46:27.219148, event: op_applied, op: osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.219164 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] op_applied on primary on version 173'2740
2014-06-12 14:46:27.221754 7f9d5bec4700  5 -- op tracker -- , seq: 747863, time: 2014-06-12 14:46:27.221690, event: header_read, op: osd_sub_op_reply(client.4311.0:139836 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.221768 7f9d5bec4700  5 -- op tracker -- , seq: 747863, time: 2014-06-12 14:46:27.221692, event: throttled, op: osd_sub_op_reply(client.4311.0:139836 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.221775 7f9d5bec4700  5 -- op tracker -- , seq: 747863, time: 2014-06-12 14:46:27.221716, event: all_read, op: osd_sub_op_reply(client.4311.0:139836 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.221781 7f9d5bec4700  5 -- op tracker -- , seq: 747863, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4311.0:139836 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.221785 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4311.0:139836 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:27.221791 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:27.221799 7f9d5bec4700 15 osd.2 173 enqueue_op 0x124e3f00 prio 196 cost 0 latency 0.000108 osd_sub_op_reply(client.4311.0:139836 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:27.224251 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:27.224267 7f9d69be4700 10 osd.2 173 dequeue_op 0x124e3f00 prio 196 cost 0 latency 0.002576 osd_sub_op_reply(client.4311.0:139836 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 pg pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]
2014-06-12 14:46:27.224291 7f9d69be4700  5 -- op tracker -- , seq: 747863, time: 2014-06-12 14:46:27.224287, event: reached_pg, op: osd_sub_op_reply(client.4311.0:139836 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.224302 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] handle_message: 0x124e3f00
2014-06-12 14:46:27.224318 7f9d69be4700  5 -- op tracker -- , seq: 747863, time: 2014-06-12 14:46:27.224315, event: started, op: osd_sub_op_reply(client.4311.0:139836 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.224328 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] sub_op_modify_reply: tid 236272 op  ack_type 4 from 25
2014-06-12 14:46:27.224348 7f9d69be4700  5 -- op tracker -- , seq: 747862, time: 2014-06-12 14:46:27.224341, event: sub_op_commit_rec, op: osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.224357 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] repop_all_applied: repop tid 236272 all applied 
2014-06-12 14:46:27.224369 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] eval_repop repgather(0x15e57440 173'2740 rep_tid=236272 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:27.224385 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]  sending ack on repgather(0x15e57440 173'2740 rep_tid=236272 committed?=0 applied?=1 lock=0 op=osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) 0x355c0500
2014-06-12 14:46:27.224420 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] repop_all_committed: repop tid 236272 all committed 
2014-06-12 14:46:27.224432 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2738 active+clean] eval_repop repgather(0x15e57440 173'2740 rep_tid=236272 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:27.224448 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2738 active+clean] log_op_stats osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4 inb 61719 outb 0 rlat 0.010769 lat 0.010798
2014-06-12 14:46:27.224465 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2738 active+clean] publish_stats_to_osd 173:3639
2014-06-12 14:46:27.224480 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2738 active+clean]  sending commit on repgather(0x15e57440 173'2740 rep_tid=236272 committed?=1 applied?=1 lock=0 op=osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4) 0x24fea00
2014-06-12 14:46:27.224513 7f9d69be4700  5 -- op tracker -- , seq: 747862, time: 2014-06-12 14:46:27.224507, event: commit_sent, op: osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.224522 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2738 active+clean]  requeue_ops 
2014-06-12 14:46:27.224533 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean]  removing repgather(0x15e57440 173'2740 rep_tid=236272 committed?=1 applied?=1 op=osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:27.224548 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean]    q front is repgather(0x15e57440 173'2740 rep_tid=236272 committed?=1 applied?=1 op=osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:27.224563 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] remove_repop repgather(0x15e57440 173'2740 rep_tid=236272 committed?=1 applied?=1 op=osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:27.224577 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean]  requeue_ops 
2014-06-12 14:46:27.224604 7f9d69be4700  5 -- op tracker -- , seq: 747862, time: 2014-06-12 14:46:27.224599, event: done, op: osd_op(client.4311.0:139836 rbd_data.10ae6b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 4132864~61440] 3.aa354922 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:27.224641 7f9d69be4700 10 osd.2 173 dequeue_op 0x124e3f00 finish
2014-06-12 14:46:27.224650 7f9d69be4700  5 -- op tracker -- , seq: 747863, time: 2014-06-12 14:46:27.224646, event: done, op: osd_sub_op_reply(client.4311.0:139836 3.122 aa354922/rbd_data.10ae6b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:46:27.224662 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:27.224665 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:27.449303 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:27.449427 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:27.475922 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:27.475950 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:46:27.475954 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:46:27.475959 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:46:27.475966 7f9d84175700 10 osd.2 173 send_pg_stats - 7 pgs updated
2014-06-12 14:46:27.476075 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 12.52 >= max 0.5 = no, load too high
2014-06-12 14:46:27.476099 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:27.476102 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92635 < max (604800 seconds)
2014-06-12 14:46:27.476115 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:27.476115 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:27.476118 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:27.476119 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:27.476122 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 2; oldest is 25.968022 seconds old
2014-06-12 14:46:27.542505 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:27.542682 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:27.553984 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:27.554120 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:27.577361 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:27.577364 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:27.577366 7f9d74bfa700 20 osd.2 173 _dispatch 0x28410a80 pg_stats_ack(7 pgs tid 4851) v1
2014-06-12 14:46:27.577368 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:46:27.577377 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:27.577379 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:27.717690 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:27.718110 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:27.949808 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:27.950774 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:27.996625 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:27.996763 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:28.003711 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:28.003752 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:28.153975 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:28.337297 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:28.337483 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:28.367638 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:28.435295 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.450913 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.461604 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.463875 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.476188 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:28.476231 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:28.476234 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:28.476235 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:28.476237 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 2; oldest is 26.968137 seconds old
2014-06-12 14:46:28.476239 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:28.504424 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.508458 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.540708 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:28.540776 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:28.586988 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.634860 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.719592 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:28.719636 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:28.759086 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.778939 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:28.789802 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.804613 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:28.806298 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:28.832612 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.848432 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.865280 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:28.878413 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.878773 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:28.906245 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:28.915270 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:28.926721 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:28.927305 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:28.930257 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:28.981949 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:28.998850 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:29.010505 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:29.039470 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:29.040758 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:29.041727 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:29.049399 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:29.050460 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:29.058672 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:29.097965 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:29.178148 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:29.190922 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:29.204928 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:29.212012 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:29.214828 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:29.219071 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:29.224744 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:29.465118 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:29.465348 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:29.476305 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:29.476363 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 12.52 >= max 0.5 = no, load too high
2014-06-12 14:46:29.476374 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:29.476381 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:29.476385 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92637 < max (604800 seconds)
2014-06-12 14:46:29.476416 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:29.476419 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:29.476421 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:29.476431 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 2; oldest is 27.968331 seconds old
2014-06-12 14:46:29.618371 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19413 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2])
2014-06-12 14:46:29.618394 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19413 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2])
2014-06-12 14:46:29.618840 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:29.618902 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:29.618929 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:29.619030 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:29.619058 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:29.619083 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:29.619106 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:29.619140 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:29.619157 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:29.619176 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:29.619188 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:29.619198 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:29.619208 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:29.619220 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:29.619238 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:29.619265 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:29.619274 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:29.619284 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:29.619294 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:29.619303 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:29.619312 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:29.621826 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:29.854482 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:29.854521 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:30.018571 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:30.018637 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:30.154053 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:30.304369 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:30.304430 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:30.367713 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:30.435373 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.450990 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.461698 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.463950 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.476498 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:30.476524 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:30.476535 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:30.476536 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:30.476538 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 2; oldest is 28.968439 seconds old
2014-06-12 14:46:30.476539 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:30.504498 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.508523 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.587061 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.634965 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.707000 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:30.707198 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:30.759175 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.779027 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:30.789874 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.804710 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:30.804802 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:30.804894 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:30.806369 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:30.832703 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.848498 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.865373 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:30.878504 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.878851 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:30.906318 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:30.915344 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:30.926795 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:30.927396 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:30.930349 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:30.982039 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:30.998926 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:31.010578 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:31.039545 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:31.040824 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:31.041787 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:31.049472 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:31.050524 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:31.058754 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:31.098038 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:31.165694 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:31.165766 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:31.178222 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:31.190997 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:31.204999 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:31.212107 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:31.214899 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:31.219137 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:31.224836 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:31.450354 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:31.450411 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:31.476614 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:31.476643 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:31.476648 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:31.476650 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:31.476652 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 2; oldest is 29.968553 seconds old
2014-06-12 14:46:31.476679 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:31.817413 7f9d56432700  5 -- op tracker -- , seq: 747864, time: 2014-06-12 14:46:31.817347, event: header_read, op: osd_sub_op_reply(client.4317.0:143793 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:31.817424 7f9d56432700  5 -- op tracker -- , seq: 747864, time: 2014-06-12 14:46:31.817350, event: throttled, op: osd_sub_op_reply(client.4317.0:143793 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:31.817450 7f9d56432700  5 -- op tracker -- , seq: 747864, time: 2014-06-12 14:46:31.817368, event: all_read, op: osd_sub_op_reply(client.4317.0:143793 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:31.817455 7f9d56432700  5 -- op tracker -- , seq: 747864, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143793 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:31.817460 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143793 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:31.817464 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:31.817471 7f9d56432700 15 osd.2 173 enqueue_op 0x2519c600 prio 196 cost 0 latency 0.000123 osd_sub_op_reply(client.4317.0:143793 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:31.817496 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:31.817503 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2519c600 prio 196 cost 0 latency 0.000156 osd_sub_op_reply(client.4317.0:143793 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2 pg pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2509 active+clean]
2014-06-12 14:46:31.817520 7f9d6a3e5700  5 -- op tracker -- , seq: 747864, time: 2014-06-12 14:46:31.817518, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143793 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:31.817527 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2509 active+clean] handle_message: 0x2519c600
2014-06-12 14:46:31.817537 7f9d6a3e5700  5 -- op tracker -- , seq: 747864, time: 2014-06-12 14:46:31.817535, event: started, op: osd_sub_op_reply(client.4317.0:143793 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:31.817540 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2509 active+clean] sub_op_modify_reply: tid 236186 op  ack_type 4 from 26
2014-06-12 14:46:31.817555 7f9d6a3e5700  5 -- op tracker -- , seq: 747475, time: 2014-06-12 14:46:31.817549, event: sub_op_commit_rec, op: osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:31.817560 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2509 active+clean] repop_all_applied: repop tid 236186 all applied 
2014-06-12 14:46:31.817567 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2509 active+clean] eval_repop repgather(0x51add40 173'2511 rep_tid=236186 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:31.817576 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2509 active+clean]  sending ack on repgather(0x51add40 173'2511 rep_tid=236186 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) 0x38137180
2014-06-12 14:46:31.817597 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2510 crt=0'0 lcod 173'2509 mlcod 173'2509 active+clean] repop_all_committed: repop tid 236186 all committed 
2014-06-12 14:46:31.817606 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2509 active+clean] eval_repop repgather(0x51add40 173'2511 rep_tid=236186 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:31.817636 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2509 active+clean] log_op_stats osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 30.309496 lat 30.309529
2014-06-12 14:46:31.817653 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2509 active+clean] publish_stats_to_osd 173:2881
2014-06-12 14:46:31.817675 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2509 active+clean]  sending commit on repgather(0x51add40 173'2511 rep_tid=236186 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) 0x8090000
2014-06-12 14:46:31.817706 7f9d6a3e5700  5 -- op tracker -- , seq: 747475, time: 2014-06-12 14:46:31.817701, event: commit_sent, op: osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:31.817714 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2509 active+clean]  requeue_ops 
2014-06-12 14:46:31.817725 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2510 active+clean]  removing repgather(0x51add40 173'2511 rep_tid=236186 committed?=1 applied?=1 op=osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:31.817735 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2510 active+clean]    q front is repgather(0x51add40 173'2511 rep_tid=236186 committed?=1 applied?=1 op=osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:31.817744 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2510 active+clean] remove_repop repgather(0x51add40 173'2511 rep_tid=236186 committed?=1 applied?=1 op=osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:31.817756 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2510 active+clean]  requeue_ops 
2014-06-12 14:46:31.817773 7f9d6a3e5700  5 -- op tracker -- , seq: 747475, time: 2014-06-12 14:46:31.817768, event: done, op: osd_op(client.4317.0:143793 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 3002368~1032192] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:31.817790 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2519c600 finish
2014-06-12 14:46:31.817796 7f9d6a3e5700  5 -- op tracker -- , seq: 747864, time: 2014-06-12 14:46:31.817794, event: done, op: osd_sub_op_reply(client.4317.0:143793 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:31.817830 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:31.817838 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:32.043091 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:32.043094 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:32.043096 7f9d74bfa700 20 osd.2 173 _dispatch 0x646dd40 ping v1
2014-06-12 14:46:32.043098 7f9d74bfa700 10 osd.2 173 ping from client.4317
2014-06-12 14:46:32.043100 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:32.043101 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:32.154130 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:32.155074 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:32.155105 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:32.220207 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:32.220235 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:32.232609 7f9d56432700  5 -- op tracker -- , seq: 747865, time: 2014-06-12 14:46:32.231049, event: header_read, op: osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.232626 7f9d56432700  5 -- op tracker -- , seq: 747865, time: 2014-06-12 14:46:32.231061, event: throttled, op: osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.232643 7f9d56432700  5 -- op tracker -- , seq: 747865, time: 2014-06-12 14:46:32.232576, event: all_read, op: osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.232652 7f9d56432700  5 -- op tracker -- , seq: 747865, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.232659 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:32.232666 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:32.232689 7f9d56432700 15 osd.2 173 enqueue_op 0x2519cb00 prio 127 cost 516903 latency 0.001639 osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.232718 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:32.232725 7f9d653db700 10 osd.2 173 dequeue_op 0x2519cb00 prio 127 cost 516903 latency 0.001676 osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.67d( v 173'2516 (0'0,173'2516] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2515 active]
2014-06-12 14:46:32.232770 7f9d653db700  5 -- op tracker -- , seq: 747865, time: 2014-06-12 14:46:32.232760, event: reached_pg, op: osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.232788 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2516 (0'0,173'2516] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2515 active] handle_message: 0x2519cb00
2014-06-12 14:46:32.232819 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2516 (0'0,173'2516] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2515 active] sub_op_modify trans a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 v 173'2517 (transaction) 174
2014-06-12 14:46:32.232858 7f9d653db700  5 -- op tracker -- , seq: 747865, time: 2014-06-12 14:46:32.232848, event: started, op: osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.232893 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2516 (0'0,173'2516] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2515 active] append_log log((0'0,173'2516], crt=0'0) [173'2517 (173'2516) modify   a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 by client.4313.0:146362 2014-06-12 08:44:55.809092]
2014-06-12 14:46:32.232944 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2517 (0'0,173'2517] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2515 active] add_log_entry 173'2517 (173'2516) modify   a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 by client.4313.0:146362 2014-06-12 08:44:55.809092
2014-06-12 14:46:32.232996 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2517 (0'0,173'2517] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2515 active] append_log  adding 1 keys
2014-06-12 14:46:32.233061 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2517, trimmed: 
2014-06-12 14:46:32.233106 7f9d653db700  5 -- op tracker -- , seq: 747865, time: 2014-06-12 14:46:32.233102, event: started, op: osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.233118 7f9d653db700  5 queue_transactions existing osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:32.233122 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:32.233123 7f9d653db700  5 queue_op 0xfa42ba0 seq 2295 osr(3.67d 0x276df70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:32.233132 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x524cb40 (1 active)
2014-06-12 14:46:32.233139 7f9d7815d700  5 _do_op 0xfa42ba0 seq 2295 osr(3.67d 0x276df70)/0x276df70 start
2014-06-12 14:46:32.233142 7f9d7815d700 10 _do_transaction on 0x146737b0
2014-06-12 14:46:32.233129 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.67d( v 173'2517 (0'0,173'2517] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2515 active] do_sub_op osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.233143 7f9d653db700 10 osd.2 173 dequeue_op 0x2519cb00 finish
2014-06-12 14:46:32.233146 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:32.233147 7f9d7815d700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:32.233149 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:32.233268 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:32.233301 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:32.233361 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:32.233410 7f9d7815d700 15 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:32.233419 7f9d7815d700 10 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1 r = 0
2014-06-12 14:46:32.233425 7f9d7815d700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:32.233646 7f9d7815d700 15 _write 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 2932736~516096
2014-06-12 14:46:32.233860 7f9d7815d700 10 lookup_strip_header done  cid 3.67d_head oid a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:32.234072 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:32.236535 7f9d7815d700 10 _generic_write 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 2932736~516096 = 0
2014-06-12 14:46:32.236616 7f9d7815d700 15 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:32.236650 7f9d7815d700 10 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 = 0
2014-06-12 14:46:32.236664 7f9d7815d700 15 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:32.236683 7f9d7815d700 10 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 = 0
2014-06-12 14:46:32.236990 7f9d7815d700 20 set_header setting 17386 cid 3.67d_head oid a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 parent seq 0
2014-06-12 14:46:32.237135 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:32.237161 7f9d7815d700 20 set_header setting 2272 cid meta oid 1fe0e10b/pglog_3.67d/0//-1 parent seq 0
2014-06-12 14:46:32.237175 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:32.239868 7f9d7815d700 10 _do_op 0xfa42ba0 seq 2295 r = 0, finisher 0x85e0be0 0
2014-06-12 14:46:32.239900 7f9d7815d700 10 _finish_op 0xfa42ba0 seq 2295 osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:32.239920 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x524cb40 (0 active)
2014-06-12 14:46:32.239924 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.239928 7f9d7715b700  5 -- op tracker -- , seq: 747865, time: 2014-06-12 14:46:32.239920, event: commit_sent, op: osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.239935 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2517 (0'0,173'2517] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2515 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:32.239954 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:32.239978 7f9d7795c700  5 -- op tracker -- , seq: 747865, time: 2014-06-12 14:46:32.239973, event: sub_op_applied, op: osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.239985 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2517 (0'0,173'2517] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2516 active] sub_op_modify_applied on 0x146736c0 op osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.239997 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2517 (0'0,173'2517] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2516 active] op_applied on primary on version 173'2517
2014-06-12 14:46:32.240003 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2517 (0'0,173'2517] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2516 active] op_applied on replica on version 173'2517
2014-06-12 14:46:32.240016 7f9d7795c700  5 -- op tracker -- , seq: 747865, time: 2014-06-12 14:46:32.240013, event: done, op: osd_sub_op(client.4313.0:146362 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.242543 7f9d56432700  5 -- op tracker -- , seq: 747866, time: 2014-06-12 14:46:32.240124, event: header_read, op: osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.242563 7f9d56432700  5 -- op tracker -- , seq: 747866, time: 2014-06-12 14:46:32.240128, event: throttled, op: osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.242583 7f9d56432700  5 -- op tracker -- , seq: 747866, time: 2014-06-12 14:46:32.242482, event: all_read, op: osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.242588 7f9d56432700  5 -- op tracker -- , seq: 747866, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.242592 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:32.242597 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:32.242602 7f9d56432700 15 osd.2 173 enqueue_op 0x2519d500 prio 127 cost 1032999 latency 0.002478 osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.242624 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:32.242630 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2519d500 prio 127 cost 1032999 latency 0.002505 osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.51e( v 173'2678 (0'0,173'2678] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2677 active]
2014-06-12 14:46:32.242643 7f9d63bd8700  5 -- op tracker -- , seq: 747866, time: 2014-06-12 14:46:32.242641, event: reached_pg, op: osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.242649 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2678 (0'0,173'2678] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2677 active] handle_message: 0x2519d500
2014-06-12 14:46:32.242656 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2678 (0'0,173'2678] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2677 active] sub_op_modify trans 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 v 173'2679 (transaction) 174
2014-06-12 14:46:32.242666 7f9d63bd8700  5 -- op tracker -- , seq: 747866, time: 2014-06-12 14:46:32.242664, event: started, op: osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.242682 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2678 (0'0,173'2678] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2677 active] append_log log((0'0,173'2678], crt=0'0) [173'2679 (173'640) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145048 2014-06-12 08:44:56.025441]
2014-06-12 14:46:32.242708 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2679 (0'0,173'2679] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2677 active] add_log_entry 173'2679 (173'640) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145048 2014-06-12 08:44:56.025441
2014-06-12 14:46:32.242724 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2679 (0'0,173'2679] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2677 active] append_log  adding 1 keys
2014-06-12 14:46:32.242747 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2679, trimmed: 
2014-06-12 14:46:32.242769 7f9d63bd8700  5 -- op tracker -- , seq: 747866, time: 2014-06-12 14:46:32.242766, event: started, op: osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.242775 7f9d63bd8700  5 queue_transactions existing osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:32.242778 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:32.242779 7f9d63bd8700  5 queue_op 0x5876f60 seq 2515 osr(3.51e 0x276db50) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:32.242783 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.51e( v 173'2679 (0'0,173'2679] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2677 active] do_sub_op osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.242792 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2519d500 finish
2014-06-12 14:46:32.242794 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:32.242796 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:32.242810 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fb00 (1 active)
2014-06-12 14:46:32.242817 7f9d82171700  5 _do_op 0x5876f60 seq 2515 osr(3.51e 0x276db50)/0x276db50 start
2014-06-12 14:46:32.242821 7f9d82171700 10 _do_transaction on 0x191d1430
2014-06-12 14:46:32.242826 7f9d82171700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:32.242939 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:32.242976 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:32.243034 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:32.243072 7f9d82171700 15 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:32.243078 7f9d82171700 10 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1 r = 0
2014-06-12 14:46:32.243081 7f9d82171700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:32.243089 7f9d82171700 15 _write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 0~1032192
2014-06-12 14:46:32.243291 7f9d82171700 10 lookup_strip_header done  cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:32.243678 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:32.247724 7f9d82171700 10 _generic_write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 0~1032192 = 0
2014-06-12 14:46:32.247786 7f9d82171700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:32.247797 7f9d82171700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:32.247815 7f9d82171700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:32.247829 7f9d82171700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:32.248032 7f9d82171700 20 set_header setting 16949 cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 parent seq 0
2014-06-12 14:46:32.248108 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:32.248119 7f9d82171700 20 set_header setting 2288 cid meta oid 1fe1612b/pglog_3.51e/0//-1 parent seq 0
2014-06-12 14:46:32.248133 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:32.252157 7f9d82171700 10 _do_op 0x5876f60 seq 2515 r = 0, finisher 0x69e72c0 0
2014-06-12 14:46:32.252178 7f9d82171700 10 _finish_op 0x5876f60 seq 2515 osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:32.252184 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fb00 (0 active)
2014-06-12 14:46:32.252193 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.252206 7f9d7715b700  5 -- op tracker -- , seq: 747866, time: 2014-06-12 14:46:32.252187, event: commit_sent, op: osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.252218 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2679 (0'0,173'2679] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2677 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:32.252260 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:32.252347 7f9d7795c700  5 -- op tracker -- , seq: 747866, time: 2014-06-12 14:46:32.252333, event: sub_op_applied, op: osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.252377 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2679 (0'0,173'2679] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2678 active] sub_op_modify_applied on 0x191d1340 op osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.252414 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2679 (0'0,173'2679] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2678 active] op_applied on primary on version 173'2679
2014-06-12 14:46:32.252428 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2679 (0'0,173'2679] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2678 active] op_applied on replica on version 173'2679
2014-06-12 14:46:32.252458 7f9d7795c700  5 -- op tracker -- , seq: 747866, time: 2014-06-12 14:46:32.252452, event: done, op: osd_sub_op(client.4308.0:145048 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.266095 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:32.266128 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:32.367792 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:32.391094 7f9d56432700  5 -- op tracker -- , seq: 747867, time: 2014-06-12 14:46:32.391049, event: header_read, op: osd_sub_op_reply(client.4317.0:143794 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:32.391110 7f9d56432700  5 -- op tracker -- , seq: 747867, time: 2014-06-12 14:46:32.391052, event: throttled, op: osd_sub_op_reply(client.4317.0:143794 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:32.391118 7f9d56432700  5 -- op tracker -- , seq: 747867, time: 2014-06-12 14:46:32.391061, event: all_read, op: osd_sub_op_reply(client.4317.0:143794 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:32.391126 7f9d56432700  5 -- op tracker -- , seq: 747867, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143794 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:32.391131 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143794 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:32.391137 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:32.391145 7f9d56432700 15 osd.2 173 enqueue_op 0x2519c000 prio 196 cost 0 latency 0.000096 osd_sub_op_reply(client.4317.0:143794 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:32.391174 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:32.391180 7f9d68be2700 10 osd.2 173 dequeue_op 0x2519c000 prio 196 cost 0 latency 0.000131 osd_sub_op_reply(client.4317.0:143794 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0) v2 pg pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2510 active+clean]
2014-06-12 14:46:32.391208 7f9d68be2700  5 -- op tracker -- , seq: 747867, time: 2014-06-12 14:46:32.391205, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143794 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:32.391225 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2510 active+clean] handle_message: 0x2519c000
2014-06-12 14:46:32.391248 7f9d68be2700  5 -- op tracker -- , seq: 747867, time: 2014-06-12 14:46:32.391245, event: started, op: osd_sub_op_reply(client.4317.0:143794 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:32.391255 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2510 active+clean] sub_op_modify_reply: tid 236187 op  ack_type 4 from 26
2014-06-12 14:46:32.391275 7f9d68be2700  5 -- op tracker -- , seq: 747476, time: 2014-06-12 14:46:32.391268, event: sub_op_commit_rec, op: osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:32.391285 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2510 active+clean] repop_all_applied: repop tid 236187 all applied 
2014-06-12 14:46:32.391292 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2510 active+clean] eval_repop repgather(0x384e3800 173'2512 rep_tid=236187 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:32.391302 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2510 active+clean]  sending ack on repgather(0x384e3800 173'2512 rep_tid=236187 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) 0x15571b80
2014-06-12 14:46:32.391321 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2511 crt=0'0 lcod 173'2510 mlcod 173'2510 active+clean] repop_all_committed: repop tid 236187 all committed 
2014-06-12 14:46:32.391327 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2510 active+clean] eval_repop repgather(0x384e3800 173'2512 rep_tid=236187 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:32.391337 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2510 active+clean] log_op_stats osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4 inb 160023 outb 0 rlat 30.775302 lat 30.775319
2014-06-12 14:46:32.391349 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2510 active+clean] publish_stats_to_osd 173:2882
2014-06-12 14:46:32.391356 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2510 active+clean]  sending commit on repgather(0x384e3800 173'2512 rep_tid=236187 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4) 0x564d400
2014-06-12 14:46:32.391376 7f9d68be2700  5 -- op tracker -- , seq: 747476, time: 2014-06-12 14:46:32.391373, event: commit_sent, op: osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:32.391381 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2510 active+clean]  requeue_ops 
2014-06-12 14:46:32.391387 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  removing repgather(0x384e3800 173'2512 rep_tid=236187 committed?=1 applied?=1 op=osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:32.391400 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]    q front is repgather(0x384e3800 173'2512 rep_tid=236187 committed?=1 applied?=1 op=osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:32.391413 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] remove_repop repgather(0x384e3800 173'2512 rep_tid=236187 committed?=1 applied?=1 op=osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:32.391428 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  requeue_ops 
2014-06-12 14:46:32.391467 7f9d68be2700  5 -- op tracker -- , seq: 747476, time: 2014-06-12 14:46:32.391462, event: done, op: osd_op(client.4317.0:143794 rbd_data.10b46b8b4567.0000000000001035 [set-alloc-hint object_size 4194304 write_size 4194304,write 4034560~159744] 3.e3c37fae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:32.391559 7f9d68be2700 10 osd.2 173 dequeue_op 0x2519c000 finish
2014-06-12 14:46:32.391568 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:32.391571 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:32.391585 7f9d56432700  5 -- op tracker -- , seq: 747867, time: 2014-06-12 14:46:32.391580, event: done, op: osd_sub_op_reply(client.4317.0:143794 3.7ae e3c37fae/rbd_data.10b46b8b4567.0000000000001035/head//3 [] ondisk, result = 0)
2014-06-12 14:46:32.418266 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:32.421767 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:32.427205 7f9d5703e700  5 -- op tracker -- , seq: 747868, time: 2014-06-12 14:46:32.426732, event: header_read, op: osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.427218 7f9d5703e700  5 -- op tracker -- , seq: 747868, time: 2014-06-12 14:46:32.426734, event: throttled, op: osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.427227 7f9d5703e700  5 -- op tracker -- , seq: 747868, time: 2014-06-12 14:46:32.427148, event: all_read, op: osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.427235 7f9d5703e700  5 -- op tracker -- , seq: 747868, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.427241 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:32.427248 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:32.427257 7f9d5703e700 15 osd.2 173 enqueue_op 0x20e86500 prio 127 cost 213799 latency 0.000525 osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.427411 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:32.427423 7f9d623d5700 10 osd.2 173 dequeue_op 0x20e86500 prio 127 cost 213799 latency 0.000690 osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c2( v 173'2833 (0'0,173'2833] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active]
2014-06-12 14:46:32.427447 7f9d623d5700  5 -- op tracker -- , seq: 747868, time: 2014-06-12 14:46:32.427443, event: reached_pg, op: osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.427457 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2833 (0'0,173'2833] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] handle_message: 0x20e86500
2014-06-12 14:46:32.427469 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2833 (0'0,173'2833] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] sub_op_modify trans 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 v 173'2834 (transaction) 174
2014-06-12 14:46:32.427486 7f9d623d5700  5 -- op tracker -- , seq: 747868, time: 2014-06-12 14:46:32.427482, event: started, op: osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.427502 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2833 (0'0,173'2833] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] append_log log((0'0,173'2833], crt=0'0) [173'2834 (173'687) modify   1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 by client.4303.0:144842 2014-06-12 08:45:34.773916]
2014-06-12 14:46:32.427530 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2834 (0'0,173'2834] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] add_log_entry 173'2834 (173'687) modify   1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 by client.4303.0:144842 2014-06-12 08:45:34.773916
2014-06-12 14:46:32.427553 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2834 (0'0,173'2834] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] append_log  adding 1 keys
2014-06-12 14:46:32.427589 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2834, trimmed: 
2014-06-12 14:46:32.427612 7f9d623d5700  5 -- op tracker -- , seq: 747868, time: 2014-06-12 14:46:32.427608, event: started, op: osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.427622 7f9d623d5700  5 queue_transactions existing osr(3.1c2 0x276d160)/0x276d160
2014-06-12 14:46:32.427626 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:32.427627 7f9d623d5700  5 queue_op 0x35fe000 seq 2612 osr(3.1c2 0x276d160) 215285 bytes   (queue has 1 ops and 215285 bytes)
2014-06-12 14:46:32.427635 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2834 (0'0,173'2834] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] do_sub_op osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.427650 7f9d623d5700 10 osd.2 173 dequeue_op 0x20e86500 finish
2014-06-12 14:46:32.427653 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:32.427656 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:32.427664 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x55190e0 (1 active)
2014-06-12 14:46:32.427670 7f9d7e96a700  5 _do_op 0x35fe000 seq 2612 osr(3.1c2 0x276d160)/0x276d160 start
2014-06-12 14:46:32.427675 7f9d7e96a700 10 _do_transaction on 0x232b270
2014-06-12 14:46:32.427681 7f9d7e96a700 15 _omap_setkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.427827 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.427847 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:32.427891 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:32.427909 7f9d7e96a700 15 _omap_rmkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.427916 7f9d7e96a700 10 _omap_rmkeys meta/1fe17dc5/pglog_3.1c2/0//-1 r = 0
2014-06-12 14:46:32.427922 7f9d7e96a700 15 _omap_setkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.427939 7f9d7e96a700 15 _write 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 0~212992
2014-06-12 14:46:32.428125 7f9d7e96a700 10 lookup_strip_header done  cid 3.1c2_head oid 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.428454 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:32.429267 7f9d7e96a700 10 _generic_write 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 0~212992 = 0
2014-06-12 14:46:32.429327 7f9d7e96a700 15 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.429346 7f9d7e96a700 10 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 = 0
2014-06-12 14:46:32.429355 7f9d7e96a700 15 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.429365 7f9d7e96a700 10 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 = 0
2014-06-12 14:46:32.429551 7f9d7e96a700 20 set_header setting 17508 cid 3.1c2_head oid 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 parent seq 0
2014-06-12 14:46:32.429888 7f9d7e96a700 20 set_header setting 2228 cid meta oid 1fe17dc5/pglog_3.1c2/0//-1 parent seq 0
2014-06-12 14:46:32.429913 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:32.429923 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:32.431425 7f9d7e96a700 10 _do_op 0x35fe000 seq 2612 r = 0, finisher 0x76aee40 0
2014-06-12 14:46:32.431451 7f9d7e96a700 10 _finish_op 0x35fe000 seq 2612 osr(3.1c2 0x276d160)/0x276d160
2014-06-12 14:46:32.431462 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x55190e0 (0 active)
2014-06-12 14:46:32.431473 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.432689 7f9d7715b700  5 -- op tracker -- , seq: 747868, time: 2014-06-12 14:46:32.432675, event: commit_sent, op: osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.432705 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2834 (0'0,173'2834] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2832 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:32.432737 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:32.434047 7f9d7795c700  5 -- op tracker -- , seq: 747868, time: 2014-06-12 14:46:32.434034, event: sub_op_applied, op: osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.434061 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2834 (0'0,173'2834] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2833 active] sub_op_modify_applied on 0x232b180 op osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.434082 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2834 (0'0,173'2834] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2833 active] op_applied on primary on version 173'2834
2014-06-12 14:46:32.434093 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2834 (0'0,173'2834] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2833 active] op_applied on replica on version 173'2834
2014-06-12 14:46:32.434133 7f9d7795c700  5 -- op tracker -- , seq: 747868, time: 2014-06-12 14:46:32.434129, event: done, op: osd_sub_op(client.4303.0:144842 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2834 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.464024 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.476724 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:32.476751 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:46:32.476755 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:46:32.476759 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:46:32.476764 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:46:32.476810 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:32.476814 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:32.476815 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:32.476859 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:32.504584 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.505494 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:32.505556 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:32.508592 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.575188 7f9d56432700  5 -- op tracker -- , seq: 747869, time: 2014-06-12 14:46:32.575064, event: header_read, op: osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.575201 7f9d56432700  5 -- op tracker -- , seq: 747869, time: 2014-06-12 14:46:32.575068, event: throttled, op: osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.575209 7f9d56432700  5 -- op tracker -- , seq: 747869, time: 2014-06-12 14:46:32.575124, event: all_read, op: osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.575217 7f9d56432700  5 -- op tracker -- , seq: 747869, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.575223 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:32.575231 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:32.575240 7f9d56432700 15 osd.2 173 enqueue_op 0x2519c000 prio 127 cost 4903 latency 0.000175 osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.575274 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:32.575284 7f9d643d9700 10 osd.2 173 dequeue_op 0x2519c000 prio 127 cost 4903 latency 0.000220 osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.51e( v 173'2679 (0'0,173'2679] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2678 active]
2014-06-12 14:46:32.575308 7f9d643d9700  5 -- op tracker -- , seq: 747869, time: 2014-06-12 14:46:32.575304, event: reached_pg, op: osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.575326 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2679 (0'0,173'2679] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2678 active] handle_message: 0x2519c000
2014-06-12 14:46:32.575339 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2679 (0'0,173'2679] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2678 active] sub_op_modify trans 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 v 173'2680 (transaction) 174
2014-06-12 14:46:32.575356 7f9d643d9700  5 -- op tracker -- , seq: 747869, time: 2014-06-12 14:46:32.575352, event: started, op: osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.575375 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2679 (0'0,173'2679] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2678 active] append_log log((0'0,173'2679], crt=0'0) [173'2680 (173'2679) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145049 2014-06-12 08:44:56.228096]
2014-06-12 14:46:32.575409 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2680 (0'0,173'2680] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2678 active] add_log_entry 173'2680 (173'2679) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145049 2014-06-12 08:44:56.228096
2014-06-12 14:46:32.575437 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2680 (0'0,173'2680] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2678 active] append_log  adding 1 keys
2014-06-12 14:46:32.575475 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2680, trimmed: 
2014-06-12 14:46:32.575503 7f9d643d9700  5 -- op tracker -- , seq: 747869, time: 2014-06-12 14:46:32.575497, event: started, op: osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.575517 7f9d643d9700  5 queue_transactions existing osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:32.575521 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:32.575522 7f9d643d9700  5 queue_op 0x6e6e480 seq 2516 osr(3.51e 0x276db50) 6389 bytes   (queue has 1 ops and 6389 bytes)
2014-06-12 14:46:32.575531 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fb00 (1 active)
2014-06-12 14:46:32.575527 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.51e( v 173'2680 (0'0,173'2680] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2678 active] do_sub_op osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.575538 7f9d643d9700 10 osd.2 173 dequeue_op 0x2519c000 finish
2014-06-12 14:46:32.575540 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:32.575539 7f9d8096e700  5 _do_op 0x6e6e480 seq 2516 osr(3.51e 0x276db50)/0x276db50 start
2014-06-12 14:46:32.575542 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:32.575544 7f9d8096e700 10 _do_transaction on 0x5ea77b0
2014-06-12 14:46:32.575550 7f9d8096e700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:32.575681 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:32.575896 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:32.575952 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:32.575978 7f9d8096e700 15 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:32.575985 7f9d8096e700 10 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1 r = 0
2014-06-12 14:46:32.575991 7f9d8096e700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:32.576010 7f9d8096e700 15 _write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 1032192~4096
2014-06-12 14:46:32.576166 7f9d8096e700 10 lookup_strip_header done  cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:32.576179 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:32.576424 7f9d8096e700 10 _generic_write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 1032192~4096 = 0
2014-06-12 14:46:32.576439 7f9d8096e700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:32.576451 7f9d8096e700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:32.576456 7f9d8096e700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:32.576464 7f9d8096e700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:32.576570 7f9d8096e700 20 set_header setting 16949 cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 parent seq 0
2014-06-12 14:46:32.576595 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:32.576610 7f9d8096e700 20 set_header setting 2288 cid meta oid 1fe1612b/pglog_3.51e/0//-1 parent seq 0
2014-06-12 14:46:32.576619 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:32.578514 7f9d8096e700 10 _do_op 0x6e6e480 seq 2516 r = 0, finisher 0x7bd4280 0
2014-06-12 14:46:32.578536 7f9d8096e700 10 _finish_op 0x6e6e480 seq 2516 osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:32.578543 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fb00 (0 active)
2014-06-12 14:46:32.578547 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.578554 7f9d7715b700  5 -- op tracker -- , seq: 747869, time: 2014-06-12 14:46:32.578544, event: commit_sent, op: osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.578564 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2680 (0'0,173'2680] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2678 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:32.578592 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:32.578628 7f9d7795c700  5 -- op tracker -- , seq: 747869, time: 2014-06-12 14:46:32.578620, event: sub_op_applied, op: osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.578636 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2680 (0'0,173'2680] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active] sub_op_modify_applied on 0x5ea76c0 op osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.578652 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2680 (0'0,173'2680] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active] op_applied on primary on version 173'2680
2014-06-12 14:46:32.578662 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2680 (0'0,173'2680] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active] op_applied on replica on version 173'2680
2014-06-12 14:46:32.578728 7f9d7795c700  5 -- op tracker -- , seq: 747869, time: 2014-06-12 14:46:32.578723, event: done, op: osd_sub_op(client.4308.0:145049 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.601674 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:32.601684 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:32.601685 7f9d74bfa700 20 osd.2 173 _dispatch 0x28411c00 pg_stats_ack(1 pgs tid 4852) v1
2014-06-12 14:46:32.601688 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:46:32.601692 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:32.601693 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:32.635048 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.679686 7f9d56432700  5 -- op tracker -- , seq: 747870, time: 2014-06-12 14:46:32.678418, event: header_read, op: osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.679698 7f9d56432700  5 -- op tracker -- , seq: 747870, time: 2014-06-12 14:46:32.678421, event: throttled, op: osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.679714 7f9d56432700  5 -- op tracker -- , seq: 747870, time: 2014-06-12 14:46:32.679652, event: all_read, op: osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.679719 7f9d56432700  5 -- op tracker -- , seq: 747870, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.679722 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:32.679727 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:32.679732 7f9d56432700 15 osd.2 173 enqueue_op 0x23e08000 prio 127 cost 746279 latency 0.001314 osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.679844 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:32.679855 7f9d633d7700 10 osd.2 173 dequeue_op 0x23e08000 prio 127 cost 746279 latency 0.001437 osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.67d( v 173'2517 (0'0,173'2517] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2516 active]
2014-06-12 14:46:32.679883 7f9d633d7700  5 -- op tracker -- , seq: 747870, time: 2014-06-12 14:46:32.679879, event: reached_pg, op: osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.679894 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2517 (0'0,173'2517] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2516 active] handle_message: 0x23e08000
2014-06-12 14:46:32.679909 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2517 (0'0,173'2517] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2516 active] sub_op_modify trans a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 v 173'2518 (transaction) 174
2014-06-12 14:46:32.679928 7f9d633d7700  5 -- op tracker -- , seq: 747870, time: 2014-06-12 14:46:32.679924, event: started, op: osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.679945 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2517 (0'0,173'2517] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2516 active] append_log log((0'0,173'2517], crt=0'0) [173'2518 (173'2517) modify   a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 by client.4313.0:146363 2014-06-12 08:44:56.140200]
2014-06-12 14:46:32.679970 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2518 (0'0,173'2518] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2516 active] add_log_entry 173'2518 (173'2517) modify   a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 by client.4313.0:146363 2014-06-12 08:44:56.140200
2014-06-12 14:46:32.679998 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2518 (0'0,173'2518] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2516 active] append_log  adding 1 keys
2014-06-12 14:46:32.680032 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2518, trimmed: 
2014-06-12 14:46:32.680056 7f9d633d7700  5 -- op tracker -- , seq: 747870, time: 2014-06-12 14:46:32.680050, event: started, op: osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.680066 7f9d633d7700  5 queue_transactions existing osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:32.680070 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:32.680071 7f9d633d7700  5 queue_op 0x782d440 seq 2296 osr(3.67d 0x276df70) 747765 bytes   (queue has 1 ops and 747765 bytes)
2014-06-12 14:46:32.680078 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.67d( v 173'2518 (0'0,173'2518] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2516 active] do_sub_op osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.680094 7f9d633d7700 10 osd.2 173 dequeue_op 0x23e08000 finish
2014-06-12 14:46:32.680098 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:32.680101 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:32.680109 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x524cb40 (1 active)
2014-06-12 14:46:32.680114 7f9d7915f700  5 _do_op 0x782d440 seq 2296 osr(3.67d 0x276df70)/0x276df70 start
2014-06-12 14:46:32.680119 7f9d7915f700 10 _do_transaction on 0x1da3f430
2014-06-12 14:46:32.680125 7f9d7915f700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:32.680218 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:32.680238 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:32.680282 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:32.680301 7f9d7915f700 15 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:32.680310 7f9d7915f700 10 _omap_rmkeys meta/1fe0e10b/pglog_3.67d/0//-1 r = 0
2014-06-12 14:46:32.680316 7f9d7915f700 15 _omap_setkeys meta/1fe0e10b/pglog_3.67d/0//-1
2014-06-12 14:46:32.680330 7f9d7915f700 15 _write 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 3448832~745472
2014-06-12 14:46:32.680430 7f9d7915f700 10 lookup_strip_header done  cid 3.67d_head oid a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:32.680534 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:32.683199 7f9d7915f700 10 _generic_write 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 3448832~745472 = 0
2014-06-12 14:46:32.683272 7f9d7915f700 15 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:32.683286 7f9d7915f700 10 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 = 0
2014-06-12 14:46:32.683291 7f9d7915f700 15 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3
2014-06-12 14:46:32.683298 7f9d7915f700 10 _setattrs 3.67d_head/a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 = 0
2014-06-12 14:46:32.683563 7f9d7915f700 20 set_header setting 17386 cid 3.67d_head oid a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 parent seq 0
2014-06-12 14:46:32.683646 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:32.683679 7f9d7915f700 20 set_header setting 2272 cid meta oid 1fe0e10b/pglog_3.67d/0//-1 parent seq 0
2014-06-12 14:46:32.683692 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:32.688954 7f9d7915f700 10 _do_op 0x782d440 seq 2296 r = 0, finisher 0x5d41ac0 0
2014-06-12 14:46:32.688975 7f9d7915f700 10 _finish_op 0x782d440 seq 2296 osr(3.67d 0x276df70)/0x276df70
2014-06-12 14:46:32.688983 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x524cb40 (0 active)
2014-06-12 14:46:32.688991 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.688991 7f9d7715b700  5 -- op tracker -- , seq: 747870, time: 2014-06-12 14:46:32.688982, event: commit_sent, op: osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.689000 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2518 (0'0,173'2518] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2516 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:32.689022 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:32.689071 7f9d7795c700  5 -- op tracker -- , seq: 747870, time: 2014-06-12 14:46:32.689063, event: sub_op_applied, op: osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.689083 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2518 (0'0,173'2518] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2517 active] sub_op_modify_applied on 0x1da3f340 op osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.689100 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2518 (0'0,173'2518] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2517 active] op_applied on primary on version 173'2518
2014-06-12 14:46:32.689109 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.67d( v 173'2518 (0'0,173'2518] local-les=153 n=277 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2517 active] op_applied on replica on version 173'2518
2014-06-12 14:46:32.689128 7f9d7795c700  5 -- op tracker -- , seq: 747870, time: 2014-06-12 14:46:32.689124, event: done, op: osd_sub_op(client.4313.0:146363 3.67d a2694e7d/rbd_data.10b06b8b4567.0000000000000fa7/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.721556 7f9d5703e700  5 -- op tracker -- , seq: 747871, time: 2014-06-12 14:46:32.713976, event: header_read, op: osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.721575 7f9d5703e700  5 -- op tracker -- , seq: 747871, time: 2014-06-12 14:46:32.713979, event: throttled, op: osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.721585 7f9d5703e700  5 -- op tracker -- , seq: 747871, time: 2014-06-12 14:46:32.721491, event: all_read, op: osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.721593 7f9d5703e700  5 -- op tracker -- , seq: 747871, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.721599 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:32.721608 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:32.721629 7f9d5703e700 15 osd.2 173 enqueue_op 0x6d10100 prio 127 cost 1032999 latency 0.007653 osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.721669 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:32.721676 7f9d61bd4700 10 osd.2 173 dequeue_op 0x6d10100 prio 127 cost 1032999 latency 0.007700 osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c2( v 173'2834 (0'0,173'2834] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2833 active]
2014-06-12 14:46:32.721700 7f9d61bd4700  5 -- op tracker -- , seq: 747871, time: 2014-06-12 14:46:32.721696, event: reached_pg, op: osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.721709 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2834 (0'0,173'2834] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2833 active] handle_message: 0x6d10100
2014-06-12 14:46:32.721723 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2834 (0'0,173'2834] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2833 active] sub_op_modify trans 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 v 173'2835 (transaction) 174
2014-06-12 14:46:32.721745 7f9d61bd4700  5 -- op tracker -- , seq: 747871, time: 2014-06-12 14:46:32.721741, event: started, op: osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.721763 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2834 (0'0,173'2834] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2833 active] append_log log((0'0,173'2834], crt=0'0) [173'2835 (173'2834) modify   1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 by client.4303.0:144843 2014-06-12 08:45:35.087653]
2014-06-12 14:46:32.721791 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2835 (0'0,173'2835] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2833 active] add_log_entry 173'2835 (173'2834) modify   1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 by client.4303.0:144843 2014-06-12 08:45:35.087653
2014-06-12 14:46:32.721815 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2835 (0'0,173'2835] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2833 active] append_log  adding 1 keys
2014-06-12 14:46:32.721856 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2835, trimmed: 
2014-06-12 14:46:32.721888 7f9d61bd4700  5 -- op tracker -- , seq: 747871, time: 2014-06-12 14:46:32.721882, event: started, op: osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.721902 7f9d61bd4700  5 queue_transactions existing osr(3.1c2 0x276d160)/0x276d160
2014-06-12 14:46:32.721907 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:32.721909 7f9d61bd4700  5 queue_op 0x15664c00 seq 2613 osr(3.1c2 0x276d160) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:32.721919 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x55190e0 (1 active)
2014-06-12 14:46:32.721927 7f9d7d968700  5 _do_op 0x15664c00 seq 2613 osr(3.1c2 0x276d160)/0x276d160 start
2014-06-12 14:46:32.721916 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2835 (0'0,173'2835] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2833 active] do_sub_op osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.721930 7f9d61bd4700 10 osd.2 173 dequeue_op 0x6d10100 finish
2014-06-12 14:46:32.721932 7f9d7d968700 10 _do_transaction on 0x232a7f0
2014-06-12 14:46:32.721934 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:32.721937 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:32.721939 7f9d7d968700 15 _omap_setkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.722081 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.722106 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:32.722167 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:32.722422 7f9d7d968700 15 _omap_rmkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.722430 7f9d7d968700 10 _omap_rmkeys meta/1fe17dc5/pglog_3.1c2/0//-1 r = 0
2014-06-12 14:46:32.722436 7f9d7d968700 15 _omap_setkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.722455 7f9d7d968700 15 _write 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 212992~1032192
2014-06-12 14:46:32.722614 7f9d7d968700 10 lookup_strip_header done  cid 3.1c2_head oid 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.722852 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:32.727698 7f9d7d968700 10 _generic_write 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 212992~1032192 = 0
2014-06-12 14:46:32.727811 7f9d7d968700 15 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.727827 7f9d7d968700 10 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 = 0
2014-06-12 14:46:32.727834 7f9d7d968700 15 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.727842 7f9d7d968700 10 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 = 0
2014-06-12 14:46:32.728151 7f9d7d968700 20 set_header setting 17508 cid 3.1c2_head oid 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 parent seq 0
2014-06-12 14:46:32.728256 7f9d7d968700 20 set_header setting 2228 cid meta oid 1fe17dc5/pglog_3.1c2/0//-1 parent seq 0
2014-06-12 14:46:32.728271 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:32.728279 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:32.739702 7f9d7d968700 10 _do_op 0x15664c00 seq 2613 r = 0, finisher 0x66b3d40 0
2014-06-12 14:46:32.739727 7f9d7d968700 10 _finish_op 0x15664c00 seq 2613 osr(3.1c2 0x276d160)/0x276d160
2014-06-12 14:46:32.739733 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x55190e0 (0 active)
2014-06-12 14:46:32.739746 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.739755 7f9d7715b700  5 -- op tracker -- , seq: 747871, time: 2014-06-12 14:46:32.739740, event: commit_sent, op: osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.739764 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2835 (0'0,173'2835] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2833 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:32.739796 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:32.739846 7f9d7795c700  5 -- op tracker -- , seq: 747871, time: 2014-06-12 14:46:32.739839, event: sub_op_applied, op: osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.739859 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2835 (0'0,173'2835] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] sub_op_modify_applied on 0x232a700 op osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.739878 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2835 (0'0,173'2835] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] op_applied on primary on version 173'2835
2014-06-12 14:46:32.739885 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2835 (0'0,173'2835] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] op_applied on replica on version 173'2835
2014-06-12 14:46:32.739904 7f9d7795c700  5 -- op tracker -- , seq: 747871, time: 2014-06-12 14:46:32.739901, event: done, op: osd_sub_op(client.4303.0:144843 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2835 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.750700 7f9d5703e700  5 -- op tracker -- , seq: 747872, time: 2014-06-12 14:46:32.748903, event: header_read, op: osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.750714 7f9d5703e700  5 -- op tracker -- , seq: 747872, time: 2014-06-12 14:46:32.748904, event: throttled, op: osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.750758 7f9d5703e700  5 -- op tracker -- , seq: 747872, time: 2014-06-12 14:46:32.750660, event: all_read, op: osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.750764 7f9d5703e700  5 -- op tracker -- , seq: 747872, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.750768 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:32.750774 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:32.750782 7f9d5703e700 15 osd.2 173 enqueue_op 0x3b42a500 prio 127 cost 1032999 latency 0.001879 osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.750851 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:32.750870 7f9d673df700 10 osd.2 173 dequeue_op 0x3b42a500 prio 127 cost 1032999 latency 0.001967 osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c2( v 173'2835 (0'0,173'2835] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active]
2014-06-12 14:46:32.750901 7f9d673df700  5 -- op tracker -- , seq: 747872, time: 2014-06-12 14:46:32.750898, event: reached_pg, op: osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.750911 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2835 (0'0,173'2835] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] handle_message: 0x3b42a500
2014-06-12 14:46:32.750919 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2835 (0'0,173'2835] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] sub_op_modify trans 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 v 173'2836 (transaction) 174
2014-06-12 14:46:32.750930 7f9d673df700  5 -- op tracker -- , seq: 747872, time: 2014-06-12 14:46:32.750927, event: started, op: osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.750942 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2835 (0'0,173'2835] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] append_log log((0'0,173'2835], crt=0'0) [173'2836 (173'2835) modify   1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 by client.4303.0:144844 2014-06-12 08:45:35.123020]
2014-06-12 14:46:32.750961 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2836 (0'0,173'2836] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] add_log_entry 173'2836 (173'2835) modify   1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 by client.4303.0:144844 2014-06-12 08:45:35.123020
2014-06-12 14:46:32.750977 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2836 (0'0,173'2836] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] append_log  adding 1 keys
2014-06-12 14:46:32.751002 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2836, trimmed: 
2014-06-12 14:46:32.751020 7f9d673df700  5 -- op tracker -- , seq: 747872, time: 2014-06-12 14:46:32.751016, event: started, op: osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.751028 7f9d673df700  5 queue_transactions existing osr(3.1c2 0x276d160)/0x276d160
2014-06-12 14:46:32.751032 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:32.751033 7f9d673df700  5 queue_op 0x6c33ec0 seq 2614 osr(3.1c2 0x276d160) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:32.751037 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2836 (0'0,173'2836] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] do_sub_op osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.751046 7f9d673df700 10 osd.2 173 dequeue_op 0x3b42a500 finish
2014-06-12 14:46:32.751056 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:32.751058 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:32.751069 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x55190e0 (1 active)
2014-06-12 14:46:32.751077 7f9d7c165700  5 _do_op 0x6c33ec0 seq 2614 osr(3.1c2 0x276d160)/0x276d160 start
2014-06-12 14:46:32.751089 7f9d7c165700 10 _do_transaction on 0x151f2470
2014-06-12 14:46:32.751095 7f9d7c165700 15 _omap_setkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.751206 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.751423 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:32.751478 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:32.751496 7f9d7c165700 15 _omap_rmkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.751501 7f9d7c165700 10 _omap_rmkeys meta/1fe17dc5/pglog_3.1c2/0//-1 r = 0
2014-06-12 14:46:32.751505 7f9d7c165700 15 _omap_setkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.751515 7f9d7c165700 15 _write 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 1245184~1032192
2014-06-12 14:46:32.751635 7f9d7c165700 10 lookup_strip_header done  cid 3.1c2_head oid 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.751817 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:32.755644 7f9d7c165700 10 _generic_write 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 1245184~1032192 = 0
2014-06-12 14:46:32.755750 7f9d7c165700 15 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.755816 7f9d7c165700 10 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 = 0
2014-06-12 14:46:32.755826 7f9d7c165700 15 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.755836 7f9d7c165700 10 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 = 0
2014-06-12 14:46:32.756142 7f9d7c165700 20 set_header setting 17508 cid 3.1c2_head oid 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 parent seq 0
2014-06-12 14:46:32.756217 7f9d7c165700 20 set_header setting 2228 cid meta oid 1fe17dc5/pglog_3.1c2/0//-1 parent seq 0
2014-06-12 14:46:32.756229 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:32.756234 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:32.760474 7f9d7c165700 10 _do_op 0x6c33ec0 seq 2614 r = 0, finisher 0x216fa680 0
2014-06-12 14:46:32.760500 7f9d7c165700 10 _finish_op 0x6c33ec0 seq 2614 osr(3.1c2 0x276d160)/0x276d160
2014-06-12 14:46:32.760508 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x55190e0 (0 active)
2014-06-12 14:46:32.760513 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.760523 7f9d7715b700  5 -- op tracker -- , seq: 747872, time: 2014-06-12 14:46:32.760512, event: commit_sent, op: osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.760531 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2836 (0'0,173'2836] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2834 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:32.760555 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:32.760625 7f9d7795c700  5 -- op tracker -- , seq: 747872, time: 2014-06-12 14:46:32.760617, event: sub_op_applied, op: osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.760640 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2836 (0'0,173'2836] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active] sub_op_modify_applied on 0x151f2380 op osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.760659 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2836 (0'0,173'2836] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active] op_applied on primary on version 173'2836
2014-06-12 14:46:32.760670 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2836 (0'0,173'2836] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active] op_applied on replica on version 173'2836
2014-06-12 14:46:32.760698 7f9d7795c700  5 -- op tracker -- , seq: 747872, time: 2014-06-12 14:46:32.760694, event: done, op: osd_sub_op(client.4303.0:144844 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2836 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.770132 7f9d5703e700  5 -- op tracker -- , seq: 747873, time: 2014-06-12 14:46:32.768605, event: header_read, op: osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.770150 7f9d5703e700  5 -- op tracker -- , seq: 747873, time: 2014-06-12 14:46:32.768607, event: throttled, op: osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.770156 7f9d5703e700  5 -- op tracker -- , seq: 747873, time: 2014-06-12 14:46:32.770092, event: all_read, op: osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.770161 7f9d5703e700  5 -- op tracker -- , seq: 747873, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.770166 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:32.770171 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:32.770178 7f9d5703e700 15 osd.2 173 enqueue_op 0x3b42a300 prio 127 cost 1032999 latency 0.001573 osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.770207 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:32.770215 7f9d67be0700 10 osd.2 173 dequeue_op 0x3b42a300 prio 127 cost 1032999 latency 0.001610 osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c2( v 173'2836 (0'0,173'2836] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active]
2014-06-12 14:46:32.770235 7f9d67be0700  5 -- op tracker -- , seq: 747873, time: 2014-06-12 14:46:32.770232, event: reached_pg, op: osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.770248 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2836 (0'0,173'2836] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active] handle_message: 0x3b42a300
2014-06-12 14:46:32.770262 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2836 (0'0,173'2836] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active] sub_op_modify trans 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 v 173'2837 (transaction) 174
2014-06-12 14:46:32.770273 7f9d67be0700  5 -- op tracker -- , seq: 747873, time: 2014-06-12 14:46:32.770271, event: started, op: osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.770288 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2836 (0'0,173'2836] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active] append_log log((0'0,173'2836], crt=0'0) [173'2837 (173'2836) modify   1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 by client.4303.0:144845 2014-06-12 08:45:35.143383]
2014-06-12 14:46:32.770314 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2837 (0'0,173'2837] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active] add_log_entry 173'2837 (173'2836) modify   1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 by client.4303.0:144845 2014-06-12 08:45:35.143383
2014-06-12 14:46:32.770337 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2837 (0'0,173'2837] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active] append_log  adding 1 keys
2014-06-12 14:46:32.770363 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2837, trimmed: 
2014-06-12 14:46:32.770381 7f9d67be0700  5 -- op tracker -- , seq: 747873, time: 2014-06-12 14:46:32.770377, event: started, op: osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.770388 7f9d67be0700  5 queue_transactions existing osr(3.1c2 0x276d160)/0x276d160
2014-06-12 14:46:32.770391 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:32.770392 7f9d67be0700  5 queue_op 0x80c5da0 seq 2615 osr(3.1c2 0x276d160) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:32.770398 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2837 (0'0,173'2837] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active] do_sub_op osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.770407 7f9d67be0700 10 osd.2 173 dequeue_op 0x3b42a300 finish
2014-06-12 14:46:32.770410 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:32.770411 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:32.770406 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x55190e0 (1 active)
2014-06-12 14:46:32.770416 7f9d83173700  5 _do_op 0x80c5da0 seq 2615 osr(3.1c2 0x276d160)/0x276d160 start
2014-06-12 14:46:32.770422 7f9d83173700 10 _do_transaction on 0x232a470
2014-06-12 14:46:32.770429 7f9d83173700 15 _omap_setkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.770542 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.770568 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:32.770612 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:32.770719 7f9d83173700 15 _omap_rmkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.770727 7f9d83173700 10 _omap_rmkeys meta/1fe17dc5/pglog_3.1c2/0//-1 r = 0
2014-06-12 14:46:32.770735 7f9d83173700 15 _omap_setkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.770753 7f9d83173700 15 _write 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 2277376~1032192
2014-06-12 14:46:32.770862 7f9d83173700 10 lookup_strip_header done  cid 3.1c2_head oid 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.771024 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:32.774249 7f9d83173700 10 _generic_write 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 2277376~1032192 = 0
2014-06-12 14:46:32.774321 7f9d83173700 15 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.774339 7f9d83173700 10 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 = 0
2014-06-12 14:46:32.774347 7f9d83173700 15 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.774356 7f9d83173700 10 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 = 0
2014-06-12 14:46:32.774566 7f9d83173700 20 set_header setting 17508 cid 3.1c2_head oid 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 parent seq 0
2014-06-12 14:46:32.774652 7f9d83173700 20 set_header setting 2228 cid meta oid 1fe17dc5/pglog_3.1c2/0//-1 parent seq 0
2014-06-12 14:46:32.774670 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:32.774682 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:32.779064 7f9d83173700 10 _do_op 0x80c5da0 seq 2615 r = 0, finisher 0x69609c0 0
2014-06-12 14:46:32.779086 7f9d83173700 10 _finish_op 0x80c5da0 seq 2615 osr(3.1c2 0x276d160)/0x276d160
2014-06-12 14:46:32.779093 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x55190e0 (0 active)
2014-06-12 14:46:32.779101 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.779100 7f9d7715b700  5 -- op tracker -- , seq: 747873, time: 2014-06-12 14:46:32.779092, event: commit_sent, op: osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.779128 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2837 (0'0,173'2837] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2835 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:32.779156 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:32.779224 7f9d7795c700  5 -- op tracker -- , seq: 747873, time: 2014-06-12 14:46:32.779216, event: sub_op_applied, op: osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.779238 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2837 (0'0,173'2837] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active] sub_op_modify_applied on 0x232a380 op osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.779264 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2837 (0'0,173'2837] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active] op_applied on primary on version 173'2837
2014-06-12 14:46:32.779272 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2837 (0'0,173'2837] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active] op_applied on replica on version 173'2837
2014-06-12 14:46:32.779287 7f9d7795c700  5 -- op tracker -- , seq: 747873, time: 2014-06-12 14:46:32.779284, event: done, op: osd_sub_op(client.4303.0:144845 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2837 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.791298 7f9d5703e700  5 -- op tracker -- , seq: 747874, time: 2014-06-12 14:46:32.789790, event: header_read, op: osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.791313 7f9d5703e700  5 -- op tracker -- , seq: 747874, time: 2014-06-12 14:46:32.789792, event: throttled, op: osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.791332 7f9d5703e700  5 -- op tracker -- , seq: 747874, time: 2014-06-12 14:46:32.791256, event: all_read, op: osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.791340 7f9d5703e700  5 -- op tracker -- , seq: 747874, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.791345 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:32.791358 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:32.791365 7f9d5703e700 15 osd.2 173 enqueue_op 0x3b42a900 prio 127 cost 885543 latency 0.001575 osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.791394 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:32.791404 7f9d663dd700 10 osd.2 173 dequeue_op 0x3b42a900 prio 127 cost 885543 latency 0.001614 osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c2( v 173'2837 (0'0,173'2837] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active]
2014-06-12 14:46:32.791433 7f9d663dd700  5 -- op tracker -- , seq: 747874, time: 2014-06-12 14:46:32.791428, event: reached_pg, op: osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.791442 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2837 (0'0,173'2837] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active] handle_message: 0x3b42a900
2014-06-12 14:46:32.791459 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2837 (0'0,173'2837] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active] sub_op_modify trans 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 v 173'2838 (transaction) 174
2014-06-12 14:46:32.791481 7f9d663dd700  5 -- op tracker -- , seq: 747874, time: 2014-06-12 14:46:32.791477, event: started, op: osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.791499 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2837 (0'0,173'2837] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active] append_log log((0'0,173'2837], crt=0'0) [173'2838 (173'2837) modify   1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 by client.4303.0:144846 2014-06-12 08:45:35.165674]
2014-06-12 14:46:32.791534 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2838 (0'0,173'2838] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active] add_log_entry 173'2838 (173'2837) modify   1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 by client.4303.0:144846 2014-06-12 08:45:35.165674
2014-06-12 14:46:32.791573 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2838 (0'0,173'2838] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active] append_log  adding 1 keys
2014-06-12 14:46:32.791607 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2838, trimmed: 
2014-06-12 14:46:32.791657 7f9d663dd700  5 -- op tracker -- , seq: 747874, time: 2014-06-12 14:46:32.791653, event: started, op: osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.791667 7f9d663dd700  5 queue_transactions existing osr(3.1c2 0x276d160)/0x276d160
2014-06-12 14:46:32.791672 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:32.791674 7f9d663dd700  5 queue_op 0x56b6060 seq 2616 osr(3.1c2 0x276d160) 887029 bytes   (queue has 1 ops and 887029 bytes)
2014-06-12 14:46:32.791689 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x55190e0 (1 active)
2014-06-12 14:46:32.791681 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2838 (0'0,173'2838] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active] do_sub_op osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.791697 7f9d663dd700 10 osd.2 173 dequeue_op 0x3b42a900 finish
2014-06-12 14:46:32.791697 7f9d82972700  5 _do_op 0x56b6060 seq 2616 osr(3.1c2 0x276d160)/0x276d160 start
2014-06-12 14:46:32.791701 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:32.791702 7f9d82972700 10 _do_transaction on 0x8223970
2014-06-12 14:46:32.791703 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:32.791708 7f9d82972700 15 _omap_setkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.791952 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.791980 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:32.792031 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:32.792050 7f9d82972700 15 _omap_rmkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.792057 7f9d82972700 10 _omap_rmkeys meta/1fe17dc5/pglog_3.1c2/0//-1 r = 0
2014-06-12 14:46:32.792063 7f9d82972700 15 _omap_setkeys meta/1fe17dc5/pglog_3.1c2/0//-1
2014-06-12 14:46:32.792077 7f9d82972700 15 _write 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 3309568~884736
2014-06-12 14:46:32.792220 7f9d82972700 10 lookup_strip_header done  cid 3.1c2_head oid 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.792410 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:32.794996 7f9d82972700 10 _generic_write 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 3309568~884736 = 0
2014-06-12 14:46:32.795044 7f9d82972700 15 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.795060 7f9d82972700 10 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 = 0
2014-06-12 14:46:32.795063 7f9d82972700 15 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3
2014-06-12 14:46:32.795068 7f9d82972700 10 _setattrs 3.1c2_head/1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 = 0
2014-06-12 14:46:32.795258 7f9d82972700 20 set_header setting 17508 cid 3.1c2_head oid 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 parent seq 0
2014-06-12 14:46:32.795327 7f9d82972700 20 set_header setting 2228 cid meta oid 1fe17dc5/pglog_3.1c2/0//-1 parent seq 0
2014-06-12 14:46:32.795339 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:32.795344 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:32.798667 7f9d82972700 10 _do_op 0x56b6060 seq 2616 r = 0, finisher 0x89e6fa0 0
2014-06-12 14:46:32.798684 7f9d82972700 10 _finish_op 0x56b6060 seq 2616 osr(3.1c2 0x276d160)/0x276d160
2014-06-12 14:46:32.798690 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x55190e0 (0 active)
2014-06-12 14:46:32.798697 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.798727 7f9d7795c700  5 -- op tracker -- , seq: 747874, time: 2014-06-12 14:46:32.798717, event: sub_op_applied, op: osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.798760 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2838 (0'0,173'2838] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active] sub_op_modify_applied on 0x8223880 op osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.798826 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:32.798874 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2838 (0'0,173'2838] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active] op_applied on primary on version 173'2838
2014-06-12 14:46:32.798886 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2838 (0'0,173'2838] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active] op_applied on replica on version 173'2838
2014-06-12 14:46:32.798936 7f9d7715b700  5 -- op tracker -- , seq: 747874, time: 2014-06-12 14:46:32.798927, event: commit_sent, op: osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.798956 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c2( v 173'2838 (0'0,173'2838] local-les=137 n=300 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2836 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:32.799001 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:32.799045 7f9d7715b700  5 -- op tracker -- , seq: 747874, time: 2014-06-12 14:46:32.799041, event: done, op: osd_sub_op(client.4303.0:144846 3.1c2 1f19a1c2/rbd_data.10a66b8b4567.0000000000001000/head//3 [] v 173'2838 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.881354 7f9d56432700  5 -- op tracker -- , seq: 747875, time: 2014-06-12 14:46:32.880559, event: header_read, op: osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.881368 7f9d56432700  5 -- op tracker -- , seq: 747875, time: 2014-06-12 14:46:32.880562, event: throttled, op: osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.881376 7f9d56432700  5 -- op tracker -- , seq: 747875, time: 2014-06-12 14:46:32.881304, event: all_read, op: osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.881384 7f9d56432700  5 -- op tracker -- , seq: 747875, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.881391 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:32.881399 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:32.881408 7f9d56432700 15 osd.2 173 enqueue_op 0x23e08f00 prio 127 cost 516903 latency 0.000849 osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.881462 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:32.881472 7f9d693e3700 10 osd.2 173 dequeue_op 0x23e08f00 prio 127 cost 516903 latency 0.000913 osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.51e( v 173'2680 (0'0,173'2680] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active]
2014-06-12 14:46:32.881498 7f9d693e3700  5 -- op tracker -- , seq: 747875, time: 2014-06-12 14:46:32.881494, event: reached_pg, op: osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.881508 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2680 (0'0,173'2680] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active] handle_message: 0x23e08f00
2014-06-12 14:46:32.881523 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2680 (0'0,173'2680] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active] sub_op_modify trans 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 v 173'2681 (transaction) 174
2014-06-12 14:46:32.881542 7f9d693e3700  5 -- op tracker -- , seq: 747875, time: 2014-06-12 14:46:32.881538, event: started, op: osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.881566 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2680 (0'0,173'2680] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active] append_log log((0'0,173'2680], crt=0'0) [173'2681 (173'2680) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145050 2014-06-12 08:44:56.391541]
2014-06-12 14:46:32.881596 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2681 (0'0,173'2681] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active] add_log_entry 173'2681 (173'2680) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145050 2014-06-12 08:44:56.391541
2014-06-12 14:46:32.881634 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2681 (0'0,173'2681] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active] append_log  adding 1 keys
2014-06-12 14:46:32.881672 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2681, trimmed: 
2014-06-12 14:46:32.881700 7f9d693e3700  5 -- op tracker -- , seq: 747875, time: 2014-06-12 14:46:32.881694, event: started, op: osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.881715 7f9d693e3700  5 queue_transactions existing osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:32.881720 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:32.881722 7f9d693e3700  5 queue_op 0x139d82a0 seq 2517 osr(3.51e 0x276db50) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:32.881730 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.51e( v 173'2681 (0'0,173'2681] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active] do_sub_op osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.881744 7f9d693e3700 10 osd.2 173 dequeue_op 0x23e08f00 finish
2014-06-12 14:46:32.881748 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:32.881750 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:32.881795 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fb00 (1 active)
2014-06-12 14:46:32.881805 7f9d81970700  5 _do_op 0x139d82a0 seq 2517 osr(3.51e 0x276db50)/0x276db50 start
2014-06-12 14:46:32.881811 7f9d81970700 10 _do_transaction on 0x2300b7b0
2014-06-12 14:46:32.881818 7f9d81970700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:32.881941 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:32.881962 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:32.881997 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:32.882014 7f9d81970700 15 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:32.882018 7f9d81970700 10 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1 r = 0
2014-06-12 14:46:32.882021 7f9d81970700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:32.882030 7f9d81970700 15 _write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 1036288~516096
2014-06-12 14:46:32.882133 7f9d81970700 10 lookup_strip_header done  cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:32.882429 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:32.883831 7f9d81970700 10 _generic_write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 1036288~516096 = 0
2014-06-12 14:46:32.883869 7f9d81970700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:32.883881 7f9d81970700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:32.883890 7f9d81970700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:32.883896 7f9d81970700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:32.884038 7f9d81970700 20 set_header setting 16949 cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 parent seq 0
2014-06-12 14:46:32.884097 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:32.884118 7f9d81970700 20 set_header setting 2288 cid meta oid 1fe1612b/pglog_3.51e/0//-1 parent seq 0
2014-06-12 14:46:32.884126 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:32.886288 7f9d81970700 10 _do_op 0x139d82a0 seq 2517 r = 0, finisher 0x48397a0 0
2014-06-12 14:46:32.886308 7f9d81970700 10 _finish_op 0x139d82a0 seq 2517 osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:32.886315 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fb00 (0 active)
2014-06-12 14:46:32.886331 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:32.886345 7f9d7795c700  5 -- op tracker -- , seq: 747875, time: 2014-06-12 14:46:32.886336, event: sub_op_applied, op: osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.886354 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2681 (0'0,173'2681] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active] sub_op_modify_applied on 0x2300b6c0 op osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:32.886376 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:32.886391 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2681 (0'0,173'2681] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active] op_applied on primary on version 173'2681
2014-06-12 14:46:32.886400 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2681 (0'0,173'2681] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active] op_applied on replica on version 173'2681
2014-06-12 14:46:32.886421 7f9d7715b700  5 -- op tracker -- , seq: 747875, time: 2014-06-12 14:46:32.886415, event: commit_sent, op: osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:32.886429 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2681 (0'0,173'2681] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2679 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:32.886441 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:32.886463 7f9d7715b700  5 -- op tracker -- , seq: 747875, time: 2014-06-12 14:46:32.886460, event: done, op: osd_sub_op(client.4308.0:145050 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.007611 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:33.007981 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:33.010660 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:33.040918 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.041716 7f9d461e0700  5 -- op tracker -- , seq: 747876, time: 2014-06-12 14:46:33.041444, event: header_read, op: osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.041728 7f9d461e0700  5 -- op tracker -- , seq: 747876, time: 2014-06-12 14:46:33.041448, event: throttled, op: osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.041739 7f9d461e0700  5 -- op tracker -- , seq: 747876, time: 2014-06-12 14:46:33.041679, event: all_read, op: osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.041744 7f9d461e0700  5 -- op tracker -- , seq: 747876, time: 0.000000, event: dispatched, op: osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.041752 7f9d461e0700 20 osd.2 173 should_share_map client.4293 172.16.96.220:0/1241966 173
2014-06-12 14:46:33.041777 7f9d461e0700 15 osd.2 173 enqueue_op 0x2519cc00 prio 63 cost 110592 latency 0.000333 osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.041805 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:33.041815 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2519cc00 prio 63 cost 110592 latency 0.000370 osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]
2014-06-12 14:46:33.041844 7f9d62bd6700  5 -- op tracker -- , seq: 747876, time: 2014-06-12 14:46:33.041838, event: reached_pg, op: osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.041854 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:33.041870 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] handle_message: 0x2519cc00
2014-06-12 14:46:33.041882 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_op osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:33.041915 7f9d62bd6700 15 getattr 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 '_'
2014-06-12 14:46:33.049541 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:33.050607 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:33.058835 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.093456 7f9d62bd6700 10 getattr 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 '_' = 0
2014-06-12 14:46:33.093481 7f9d62bd6700 15 getattr 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 'snapset'
2014-06-12 14:46:33.093593 7f9d62bd6700 10 getattr 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 'snapset' = 0
2014-06-12 14:46:33.093602 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] populate_obc_watchers 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.093638 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.093649 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] get_object_context: creating obc from disk: 0x34d12580
2014-06-12 14:46:33.093659 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] get_object_context: 0x34d12580 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 rwstate(none n=0 w=0) oi: 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3(173'862 client.4293.0:38289 wrlock_by=unknown.0.0:0 dirty s 4194304 uv862) ssc: 0x283d28c0 snapset: 0=[]:[]+head
2014-06-12 14:46:33.093690 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] find_object_context 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 @head oi=1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3(173'862 client.4293.0:38289 wrlock_by=unknown.0.0:0 dirty s 4194304 uv862)
2014-06-12 14:46:33.093713 7f9d62bd6700  5 -- op tracker -- , seq: 747876, time: 2014-06-12 14:46:33.093707, event: started, op: osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.093724 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] execute_ctx 0x60aac00
2014-06-12 14:46:33.093743 7f9d62bd6700  5 -- op tracker -- , seq: 747876, time: 2014-06-12 14:46:33.093740, event: started, op: osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.093747 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_op 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] ov 173'862 av 173'2854 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:33.093756 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_osd_op 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592]
2014-06-12 14:46:33.093765 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:33.093779 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_osd_op  write 0~110592
2014-06-12 14:46:33.093791 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] do_osd_op_effects on session 0x5c8cc00
2014-06-12 14:46:33.093797 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] make_writeable 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 snapset=0x283d28f8  snapc=0=[]
2014-06-12 14:46:33.093805 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] make_writeable 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:33.093810 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] finish_ctx 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 0x60aac00 op modify  
2014-06-12 14:46:33.093818 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  set mtime to 2014-06-12 08:45:35.421858
2014-06-12 14:46:33.093870 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  final snapset 0=[]:[]+head in 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.093891 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  zeroing write result code 0
2014-06-12 14:46:33.093901 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] new_repop rep_tid 236273 on osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.093916 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] issue_repop rep_tid 236273 o 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.093936 7f9d62bd6700  5 -- op tracker -- , seq: 747876, time: 2014-06-12 14:46:33.093932, event: waiting for subops from 23, op: osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.093956 7f9d62bd6700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:33.093975 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2853 (0'0,173'2853] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] append_log log((0'0,173'2853], crt=0'0) [173'2854 (173'862) modify   1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 by client.4293.0:144170 2014-06-12 08:45:35.421858]
2014-06-12 14:46:33.093991 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] add_log_entry 173'2854 (173'862) modify   1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 by client.4293.0:144170 2014-06-12 08:45:35.421858
2014-06-12 14:46:33.094004 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] append_log  adding 1 keys
2014-06-12 14:46:33.094025 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2854, trimmed: 
2014-06-12 14:46:33.094041 7f9d62bd6700  5 queue_transactions existing osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:46:33.094045 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:33.094045 7f9d62bd6700  5 queue_op 0x6f81980 seq 2599 osr(3.650 0x276dff0) 112885 bytes   (queue has 1 ops and 112885 bytes)
2014-06-12 14:46:33.094058 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5440 (1 active)
2014-06-12 14:46:33.094064 7f9d7a161700  5 _do_op 0x6f81980 seq 2599 osr(3.650 0x276dff0)/0x276dff0 start
2014-06-12 14:46:33.094051 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] eval_repop repgather(0x10193800 173'2854 rep_tid=236273 committed?=0 applied?=0 lock=0 op=osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.094068 7f9d7a161700 10 _do_transaction on 0x51b2600
2014-06-12 14:46:33.094069 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2519cc00 finish
2014-06-12 14:46:33.094074 7f9d7a161700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.094074 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:33.094079 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.094153 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.094178 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:33.094213 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:33.094233 7f9d7a161700 15 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.094238 7f9d7a161700 10 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1 r = 0
2014-06-12 14:46:33.094241 7f9d7a161700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.094252 7f9d7a161700 15 _write 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 0~110592
2014-06-12 14:46:33.094335 7f9d7a161700 10 lookup_strip_header done  cid 3.650_head oid 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.094548 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:33.094934 7f9d7a161700 10 _generic_write 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 0~110592 = 0
2014-06-12 14:46:33.094954 7f9d7a161700 15 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.094962 7f9d7a161700 10 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 = 0
2014-06-12 14:46:33.094966 7f9d7a161700 15 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.094971 7f9d7a161700 10 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 = 0
2014-06-12 14:46:33.095061 7f9d7a161700 20 set_header setting 17926 cid 3.650_head oid 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 parent seq 0
2014-06-12 14:46:33.095104 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:33.095113 7f9d7a161700 20 set_header setting 2114 cid meta oid 1fe1320a/pglog_3.650/0//-1 parent seq 0
2014-06-12 14:46:33.095119 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:33.095497 7f9d7a161700 10 _do_op 0x6f81980 seq 2599 r = 0, finisher 0x8957120 0x3786f8c0
2014-06-12 14:46:33.095508 7f9d7a161700 10 _finish_op 0x6f81980 seq 2599 osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:46:33.095517 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5440 (0 active)
2014-06-12 14:46:33.095519 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:33.095514 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] op_commit: 236273
2014-06-12 14:46:33.095539 7f9d7715b700  5 -- op tracker -- , seq: 747876, time: 2014-06-12 14:46:33.095532, event: op_commit, op: osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.095552 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] op_applied: 236273
2014-06-12 14:46:33.095572 7f9d7795c700  5 -- op tracker -- , seq: 747876, time: 2014-06-12 14:46:33.095567, event: op_applied, op: osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.095579 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] op_applied on primary on version 173'2854
2014-06-12 14:46:33.096930 7f9d57543700  5 -- op tracker -- , seq: 747877, time: 2014-06-12 14:46:33.096874, event: header_read, op: osd_sub_op_reply(client.4293.0:144170 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.096940 7f9d57543700  5 -- op tracker -- , seq: 747877, time: 2014-06-12 14:46:33.096876, event: throttled, op: osd_sub_op_reply(client.4293.0:144170 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.096946 7f9d57543700  5 -- op tracker -- , seq: 747877, time: 2014-06-12 14:46:33.096888, event: all_read, op: osd_sub_op_reply(client.4293.0:144170 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.096953 7f9d57543700  5 -- op tracker -- , seq: 747877, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144170 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.096958 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144170 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:33.096963 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:33.096972 7f9d57543700 15 osd.2 173 enqueue_op 0x2910000 prio 196 cost 0 latency 0.000097 osd_sub_op_reply(client.4293.0:144170 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:33.097009 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:33.097017 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2910000 prio 196 cost 0 latency 0.000143 osd_sub_op_reply(client.4293.0:144170 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0) v2 pg pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]
2014-06-12 14:46:33.097040 7f9d65bdc700  5 -- op tracker -- , seq: 747877, time: 2014-06-12 14:46:33.097036, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144170 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.097055 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] handle_message: 0x2910000
2014-06-12 14:46:33.097070 7f9d65bdc700  5 -- op tracker -- , seq: 747877, time: 2014-06-12 14:46:33.097067, event: started, op: osd_sub_op_reply(client.4293.0:144170 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.097076 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] sub_op_modify_reply: tid 236273 op  ack_type 4 from 23
2014-06-12 14:46:33.097095 7f9d65bdc700  5 -- op tracker -- , seq: 747876, time: 2014-06-12 14:46:33.097087, event: sub_op_commit_rec, op: osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.097107 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] repop_all_applied: repop tid 236273 all applied 
2014-06-12 14:46:33.097117 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] eval_repop repgather(0x10193800 173'2854 rep_tid=236273 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.097132 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean]  sending ack on repgather(0x10193800 173'2854 rep_tid=236273 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) 0x25edcc80
2014-06-12 14:46:33.097162 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2853 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] repop_all_committed: repop tid 236273 all committed 
2014-06-12 14:46:33.097179 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean] eval_repop repgather(0x10193800 173'2854 rep_tid=236273 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.097197 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean] log_op_stats osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4 inb 110871 outb 0 rlat 0.055716 lat 0.055751
2014-06-12 14:46:33.097222 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean] publish_stats_to_osd 173:3713
2014-06-12 14:46:33.097234 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean]  sending commit on repgather(0x10193800 173'2854 rep_tid=236273 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) 0x2027c780
2014-06-12 14:46:33.097264 7f9d65bdc700  5 -- op tracker -- , seq: 747876, time: 2014-06-12 14:46:33.097260, event: commit_sent, op: osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.097281 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2852 active+clean]  requeue_ops 
2014-06-12 14:46:33.097292 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]  removing repgather(0x10193800 173'2854 rep_tid=236273 committed?=1 applied?=1 op=osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:33.097327 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]    q front is repgather(0x10193800 173'2854 rep_tid=236273 committed?=1 applied?=1 op=osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:33.097348 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] remove_repop repgather(0x10193800 173'2854 rep_tid=236273 committed?=1 applied?=1 op=osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:33.097360 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]  requeue_ops 
2014-06-12 14:46:33.097385 7f9d65bdc700  5 -- op tracker -- , seq: 747876, time: 2014-06-12 14:46:33.097380, event: done, op: osd_op(client.4293.0:144170 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~110592] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.097465 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2910000 finish
2014-06-12 14:46:33.097475 7f9d65bdc700  5 -- op tracker -- , seq: 747877, time: 2014-06-12 14:46:33.097471, event: done, op: osd_sub_op_reply(client.4293.0:144170 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.097486 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:33.097488 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.108661 7f9d461e0700  5 -- op tracker -- , seq: 747878, time: 2014-06-12 14:46:33.103786, event: header_read, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.108679 7f9d461e0700  5 -- op tracker -- , seq: 747878, time: 2014-06-12 14:46:33.103789, event: throttled, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.108689 7f9d461e0700  5 -- op tracker -- , seq: 747878, time: 2014-06-12 14:46:33.108616, event: all_read, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.108700 7f9d461e0700  5 -- op tracker -- , seq: 747878, time: 0.000000, event: dispatched, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.108709 7f9d461e0700 20 osd.2 173 should_share_map client.4293 172.16.96.220:0/1241966 173
2014-06-12 14:46:33.108740 7f9d461e0700 15 osd.2 173 enqueue_op 0x2519cd00 prio 63 cost 2064384 latency 0.004954 osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.108808 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:33.108828 7f9d64bda700 10 osd.2 173 dequeue_op 0x2519cd00 prio 63 cost 2064384 latency 0.005042 osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]
2014-06-12 14:46:33.108871 7f9d64bda700  5 -- op tracker -- , seq: 747878, time: 2014-06-12 14:46:33.108863, event: reached_pg, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.108888 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:33.108908 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] handle_message: 0x2519cd00
2014-06-12 14:46:33.108920 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] do_op osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:33.108950 7f9d64bda700 15 getattr 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 '_'
2014-06-12 14:46:33.109137 7f9d64bda700 10 getattr 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 '_' = 0
2014-06-12 14:46:33.109160 7f9d64bda700 15 getattr 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 'snapset'
2014-06-12 14:46:33.109256 7f9d64bda700 10 getattr 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 'snapset' = 0
2014-06-12 14:46:33.109283 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] populate_obc_watchers 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.109291 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.109297 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] get_object_context: creating obc from disk: 0x172ceb00
2014-06-12 14:46:33.109303 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] get_object_context: 0x172ceb00 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 rwstate(none n=0 w=0) oi: 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3(173'2854 client.4293.0:144170 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2854) ssc: 0x283d37a0 snapset: 0=[]:[]+head
2014-06-12 14:46:33.109322 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] find_object_context 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 @head oi=1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3(173'2854 client.4293.0:144170 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2854)
2014-06-12 14:46:33.109338 7f9d64bda700  5 -- op tracker -- , seq: 747878, time: 2014-06-12 14:46:33.109335, event: started, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.109354 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] execute_ctx 0x1b262000
2014-06-12 14:46:33.109365 7f9d64bda700  5 -- op tracker -- , seq: 747878, time: 2014-06-12 14:46:33.109362, event: started, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.109377 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] do_op 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] ov 173'2854 av 173'2855 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:33.109387 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] do_osd_op 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384]
2014-06-12 14:46:33.109394 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:33.109402 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] do_osd_op  write 110592~2064384
2014-06-12 14:46:33.109413 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] do_osd_op_effects on session 0x5c8cc00
2014-06-12 14:46:33.109419 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] make_writeable 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 snapset=0x283d37d8  snapc=0=[]
2014-06-12 14:46:33.109426 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] make_writeable 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:33.109432 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] finish_ctx 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 0x1b262000 op modify  
2014-06-12 14:46:33.109439 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]  set mtime to 2014-06-12 08:45:35.480765
2014-06-12 14:46:33.109452 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]  final snapset 0=[]:[]+head in 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.109464 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]  zeroing write result code 0
2014-06-12 14:46:33.109470 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] new_repop rep_tid 236274 on osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.109479 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] issue_repop rep_tid 236274 o 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.109495 7f9d64bda700  5 -- op tracker -- , seq: 747878, time: 2014-06-12 14:46:33.109492, event: waiting for subops from 23, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.109512 7f9d64bda700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:33.109527 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2854 (0'0,173'2854] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] append_log log((0'0,173'2854], crt=0'0) [173'2855 (173'2854) modify   1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 by client.4293.0:144171 2014-06-12 08:45:35.480765]
2014-06-12 14:46:33.109548 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] add_log_entry 173'2855 (173'2854) modify   1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 by client.4293.0:144171 2014-06-12 08:45:35.480765
2014-06-12 14:46:33.109561 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] append_log  adding 1 keys
2014-06-12 14:46:33.109582 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2855, trimmed: 
2014-06-12 14:46:33.109598 7f9d64bda700  5 queue_transactions existing osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:46:33.109602 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:33.109603 7f9d64bda700  5 queue_op 0x213fa660 seq 2600 osr(3.650 0x276dff0) 2066677 bytes   (queue has 1 ops and 2066677 bytes)
2014-06-12 14:46:33.109622 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5440 (1 active)
2014-06-12 14:46:33.109609 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] eval_repop repgather(0x1696d380 173'2855 rep_tid=236274 committed?=0 applied?=0 lock=0 op=osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.109629 7f9d7f16b700  5 _do_op 0x213fa660 seq 2600 osr(3.650 0x276dff0)/0x276dff0 start
2014-06-12 14:46:33.109630 7f9d64bda700 10 osd.2 173 dequeue_op 0x2519cd00 finish
2014-06-12 14:46:33.109632 7f9d7f16b700 10 _do_transaction on 0x12ae3d40
2014-06-12 14:46:33.109633 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:33.109635 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.109638 7f9d7f16b700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.109709 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.109726 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:33.109760 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:33.109773 7f9d7f16b700 15 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.109786 7f9d7f16b700 10 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1 r = 0
2014-06-12 14:46:33.109791 7f9d7f16b700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.109803 7f9d7f16b700 15 _write 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 110592~2064384
2014-06-12 14:46:33.110203 7f9d7f16b700 10 lookup_strip_header done  cid 3.650_head oid 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.110405 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:33.118906 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19413 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,1])
2014-06-12 14:46:33.118921 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19413 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,1])
2014-06-12 14:46:33.119242 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:33.119268 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:33.119483 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:33.119537 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:33.119568 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:33.121560 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:33.121605 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:33.121675 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:33.121708 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:33.121735 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:33.121760 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:33.121786 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:33.121777 7f9d7f16b700 10 _generic_write 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 110592~2064384 = 0
2014-06-12 14:46:33.121812 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:33.121839 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:33.121862 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:33.121882 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:33.121903 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:33.121924 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:33.121954 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:33.121977 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:33.122002 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:33.122163 7f9d7f16b700 15 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.122199 7f9d7f16b700 10 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 = 0
2014-06-12 14:46:33.122213 7f9d7f16b700 15 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.122230 7f9d7f16b700 10 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 = 0
2014-06-12 14:46:33.123195 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:33.123361 7f9d7f16b700 20 set_header setting 17926 cid 3.650_head oid 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 parent seq 0
2014-06-12 14:46:33.123799 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:33.123833 7f9d7f16b700 20 set_header setting 2114 cid meta oid 1fe1320a/pglog_3.650/0//-1 parent seq 0
2014-06-12 14:46:33.123858 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:33.134224 7f9d7f16b700 10 _do_op 0x213fa660 seq 2600 r = 0, finisher 0x6f13c40 0xe9d6180
2014-06-12 14:46:33.134241 7f9d7f16b700 10 _finish_op 0x213fa660 seq 2600 osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:46:33.134246 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5440 (0 active)
2014-06-12 14:46:33.134249 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:33.134272 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] op_commit: 236274
2014-06-12 14:46:33.134324 7f9d7715b700  5 -- op tracker -- , seq: 747878, time: 2014-06-12 14:46:33.134311, event: op_commit, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.134356 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] op_applied: 236274
2014-06-12 14:46:33.134415 7f9d7795c700  5 -- op tracker -- , seq: 747878, time: 2014-06-12 14:46:33.134399, event: op_applied, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.134433 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] op_applied on primary on version 173'2855
2014-06-12 14:46:33.138900 7f9d57543700  5 -- op tracker -- , seq: 747879, time: 2014-06-12 14:46:33.138813, event: header_read, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ack, result = 0)
2014-06-12 14:46:33.138927 7f9d57543700  5 -- op tracker -- , seq: 747879, time: 2014-06-12 14:46:33.138816, event: throttled, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ack, result = 0)
2014-06-12 14:46:33.138945 7f9d57543700  5 -- op tracker -- , seq: 747879, time: 2014-06-12 14:46:33.138836, event: all_read, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ack, result = 0)
2014-06-12 14:46:33.138966 7f9d57543700  5 -- op tracker -- , seq: 747879, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ack, result = 0)
2014-06-12 14:46:33.138971 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:33.138976 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:33.138993 7f9d57543700 15 osd.2 173 enqueue_op 0x2911100 prio 196 cost 0 latency 0.000180 osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ack, result = 0) v2
2014-06-12 14:46:33.139100 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:33.139118 7f9d57543700  5 -- op tracker -- , seq: 747880, time: 2014-06-12 14:46:33.139036, event: header_read, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.139146 7f9d57543700  5 -- op tracker -- , seq: 747880, time: 2014-06-12 14:46:33.139037, event: throttled, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.139128 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2911100 prio 196 cost 0 latency 0.000315 osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ack, result = 0) v2 pg pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]
2014-06-12 14:46:33.139170 7f9d57543700  5 -- op tracker -- , seq: 747880, time: 2014-06-12 14:46:33.139068, event: all_read, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.139183 7f9d57543700  5 -- op tracker -- , seq: 747880, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.139189 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:33.139194 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:33.139194 7f9d6abe6700  5 -- op tracker -- , seq: 747879, time: 2014-06-12 14:46:33.139159, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ack, result = 0)
2014-06-12 14:46:33.139200 7f9d57543700 15 osd.2 173 enqueue_op 0x2afc200 prio 196 cost 0 latency 0.000164 osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:33.139203 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] handle_message: 0x2911100
2014-06-12 14:46:33.139221 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:33.139216 7f9d6abe6700  5 -- op tracker -- , seq: 747879, time: 2014-06-12 14:46:33.139213, event: started, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ack, result = 0)
2014-06-12 14:46:33.139226 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] sub_op_modify_reply: tid 236274 op  ack_type 1 from 23
2014-06-12 14:46:33.139247 7f9d6abe6700  5 -- op tracker -- , seq: 747878, time: 2014-06-12 14:46:33.139241, event: sub_op_applied_rec, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.139253 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] repop_all_applied: repop tid 236274 all applied 
2014-06-12 14:46:33.139260 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] eval_repop repgather(0x1696d380 173'2855 rep_tid=236274 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.139270 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]  sending ack on repgather(0x1696d380 173'2855 rep_tid=236274 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) 0x31656a00
2014-06-12 14:46:33.139291 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2911100 finish
2014-06-12 14:46:33.139299 7f9d6abe6700  5 -- op tracker -- , seq: 747879, time: 2014-06-12 14:46:33.139296, event: done, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ack, result = 0)
2014-06-12 14:46:33.139300 7f9d66bde700 10 osd.2 173 dequeue_op 0x2afc200 prio 196 cost 0 latency 0.000264 osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0) v2 pg pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean]
2014-06-12 14:46:33.139325 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:33.139327 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.139327 7f9d66bde700  5 -- op tracker -- , seq: 747880, time: 2014-06-12 14:46:33.139319, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.139337 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] handle_message: 0x2afc200
2014-06-12 14:46:33.139371 7f9d66bde700  5 -- op tracker -- , seq: 747880, time: 2014-06-12 14:46:33.139356, event: started, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.139393 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] sub_op_modify_reply: tid 236274 op  ack_type 4 from 23
2014-06-12 14:46:33.139431 7f9d66bde700  5 -- op tracker -- , seq: 747878, time: 2014-06-12 14:46:33.139418, event: sub_op_commit_rec, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.139451 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2854 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] repop_all_committed: repop tid 236274 all committed 
2014-06-12 14:46:33.139471 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean] eval_repop repgather(0x1696d380 173'2855 rep_tid=236274 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.139496 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean] log_op_stats osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4 inb 2064663 outb 0 rlat 0.035504 lat 0.035709
2014-06-12 14:46:33.139521 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean] publish_stats_to_osd 173:3714
2014-06-12 14:46:33.139546 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean]  sending commit on repgather(0x1696d380 173'2855 rep_tid=236274 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) 0x6ce9900
2014-06-12 14:46:33.139569 7f9d66bde700  5 -- op tracker -- , seq: 747878, time: 2014-06-12 14:46:33.139565, event: commit_sent, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.139575 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2853 active+clean]  requeue_ops 
2014-06-12 14:46:33.139581 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  removing repgather(0x1696d380 173'2855 rep_tid=236274 committed?=1 applied?=1 op=osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:33.139591 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]    q front is repgather(0x1696d380 173'2855 rep_tid=236274 committed?=1 applied?=1 op=osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:33.139601 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] remove_repop repgather(0x1696d380 173'2855 rep_tid=236274 committed?=1 applied?=1 op=osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:33.139609 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  requeue_ops 
2014-06-12 14:46:33.139626 7f9d66bde700  5 -- op tracker -- , seq: 747878, time: 2014-06-12 14:46:33.139623, event: done, op: osd_op(client.4293.0:144171 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 110592~2064384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.139638 7f9d66bde700 10 osd.2 173 dequeue_op 0x2afc200 finish
2014-06-12 14:46:33.139644 7f9d66bde700  5 -- op tracker -- , seq: 747880, time: 2014-06-12 14:46:33.139642, event: done, op: osd_sub_op_reply(client.4293.0:144171 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.139650 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:33.139653 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.148871 7f9d461e0700  5 -- op tracker -- , seq: 747881, time: 2014-06-12 14:46:33.145036, event: header_read, op: osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.148892 7f9d461e0700  5 -- op tracker -- , seq: 747881, time: 2014-06-12 14:46:33.145039, event: throttled, op: osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.148904 7f9d461e0700  5 -- op tracker -- , seq: 747881, time: 2014-06-12 14:46:33.148808, event: all_read, op: osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.148916 7f9d461e0700  5 -- op tracker -- , seq: 747881, time: 0.000000, event: dispatched, op: osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.148927 7f9d461e0700 20 osd.2 173 should_share_map client.4293 172.16.96.220:0/1241966 173
2014-06-12 14:46:33.148964 7f9d461e0700 15 osd.2 173 enqueue_op 0x2519cf00 prio 63 cost 1552384 latency 0.003927 osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.149043 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:33.149066 7f9d69be4700 10 osd.2 173 dequeue_op 0x2519cf00 prio 63 cost 1552384 latency 0.004029 osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]
2014-06-12 14:46:33.149112 7f9d69be4700  5 -- op tracker -- , seq: 747881, time: 2014-06-12 14:46:33.149108, event: reached_pg, op: osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.149121 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:33.149132 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] handle_message: 0x2519cf00
2014-06-12 14:46:33.149138 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_op osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:33.149157 7f9d69be4700 15 getattr 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 '_'
2014-06-12 14:46:33.149319 7f9d69be4700 10 getattr 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 '_' = 0
2014-06-12 14:46:33.149334 7f9d69be4700 15 getattr 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 'snapset'
2014-06-12 14:46:33.149393 7f9d69be4700 10 getattr 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 'snapset' = 0
2014-06-12 14:46:33.149410 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] populate_obc_watchers 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.149425 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.149431 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] get_object_context: creating obc from disk: 0x129b4840
2014-06-12 14:46:33.149437 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] get_object_context: 0x129b4840 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 rwstate(none n=0 w=0) oi: 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3(173'2855 client.4293.0:144171 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2855) ssc: 0x81bbce0 snapset: 0=[]:[]+head
2014-06-12 14:46:33.149448 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] find_object_context 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 @head oi=1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3(173'2855 client.4293.0:144171 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2855)
2014-06-12 14:46:33.149465 7f9d69be4700  5 -- op tracker -- , seq: 747881, time: 2014-06-12 14:46:33.149461, event: started, op: osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.149475 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] execute_ctx 0x30f7e400
2014-06-12 14:46:33.149486 7f9d69be4700  5 -- op tracker -- , seq: 747881, time: 2014-06-12 14:46:33.149483, event: started, op: osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.149490 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_op 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] ov 173'2855 av 173'2856 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:33.149498 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384]
2014-06-12 14:46:33.149505 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:33.149515 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op  write 2174976~1552384
2014-06-12 14:46:33.149525 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op_effects on session 0x5c8cc00
2014-06-12 14:46:33.149531 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] make_writeable 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 snapset=0x81bbd18  snapc=0=[]
2014-06-12 14:46:33.149538 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] make_writeable 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:33.149545 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] finish_ctx 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 0x30f7e400 op modify  
2014-06-12 14:46:33.149552 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  set mtime to 2014-06-12 08:45:35.522884
2014-06-12 14:46:33.149570 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  final snapset 0=[]:[]+head in 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.149583 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  zeroing write result code 0
2014-06-12 14:46:33.149590 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] new_repop rep_tid 236275 on osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.149601 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] issue_repop rep_tid 236275 o 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.149627 7f9d69be4700  5 -- op tracker -- , seq: 747881, time: 2014-06-12 14:46:33.149624, event: waiting for subops from 23, op: osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.149661 7f9d69be4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:33.149681 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2855 (0'0,173'2855] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] append_log log((0'0,173'2855], crt=0'0) [173'2856 (173'2855) modify   1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 by client.4293.0:144172 2014-06-12 08:45:35.522884]
2014-06-12 14:46:33.149715 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] add_log_entry 173'2856 (173'2855) modify   1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 by client.4293.0:144172 2014-06-12 08:45:35.522884
2014-06-12 14:46:33.149736 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] append_log  adding 1 keys
2014-06-12 14:46:33.149766 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2856, trimmed: 
2014-06-12 14:46:33.149787 7f9d69be4700  5 queue_transactions existing osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:46:33.149792 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:33.149793 7f9d69be4700  5 queue_op 0x7aea240 seq 2601 osr(3.650 0x276dff0) 1554677 bytes   (queue has 1 ops and 1554677 bytes)
2014-06-12 14:46:33.149800 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] eval_repop repgather(0x5312a80 173'2856 rep_tid=236275 committed?=0 applied?=0 lock=0 op=osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.149817 7f9d69be4700 10 osd.2 173 dequeue_op 0x2519cf00 finish
2014-06-12 14:46:33.149821 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:33.149823 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.149829 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5440 (1 active)
2014-06-12 14:46:33.149834 7f9d7e169700  5 _do_op 0x7aea240 seq 2601 osr(3.650 0x276dff0)/0x276dff0 start
2014-06-12 14:46:33.149837 7f9d7e169700 10 _do_transaction on 0x5313500
2014-06-12 14:46:33.149843 7f9d7e169700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.149914 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.149936 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:33.149973 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:33.149990 7f9d7e169700 15 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.149995 7f9d7e169700 10 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1 r = 0
2014-06-12 14:46:33.150000 7f9d7e169700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.150014 7f9d7e169700 15 _write 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 2174976~1552384
2014-06-12 14:46:33.150137 7f9d7e169700 10 lookup_strip_header done  cid 3.650_head oid 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.150221 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:33.154358 7f9d7e169700 10 _generic_write 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 2174976~1552384 = 0
2014-06-12 14:46:33.154448 7f9d7e169700 15 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.154462 7f9d7e169700 10 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 = 0
2014-06-12 14:46:33.154466 7f9d7e169700 15 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.154471 7f9d7e169700 10 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 = 0
2014-06-12 14:46:33.154719 7f9d7e169700 20 set_header setting 17926 cid 3.650_head oid 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 parent seq 0
2014-06-12 14:46:33.154816 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:33.154827 7f9d7e169700 20 set_header setting 2114 cid meta oid 1fe1320a/pglog_3.650/0//-1 parent seq 0
2014-06-12 14:46:33.154841 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:33.160372 7f9d461e0700  5 -- op tracker -- , seq: 747882, time: 2014-06-12 14:46:33.159091, event: header_read, op: osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.160386 7f9d461e0700  5 -- op tracker -- , seq: 747882, time: 2014-06-12 14:46:33.159094, event: throttled, op: osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.160396 7f9d461e0700  5 -- op tracker -- , seq: 747882, time: 2014-06-12 14:46:33.160329, event: all_read, op: osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.160405 7f9d461e0700  5 -- op tracker -- , seq: 747882, time: 0.000000, event: dispatched, op: osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.160414 7f9d461e0700 20 osd.2 173 should_share_map client.4293 172.16.96.220:0/1241966 173
2014-06-12 14:46:33.160441 7f9d461e0700 15 osd.2 173 enqueue_op 0x2519db00 prio 63 cost 466944 latency 0.001349 osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.160473 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:33.160479 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2519db00 prio 63 cost 466944 latency 0.001388 osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]
2014-06-12 14:46:33.160503 7f9d6a3e5700  5 -- op tracker -- , seq: 747882, time: 2014-06-12 14:46:33.160498, event: reached_pg, op: osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.160511 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:33.160537 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] handle_message: 0x2519db00
2014-06-12 14:46:33.160551 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_op osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:33.160711 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] get_object_context: found obc in cache: 0x129b4840
2014-06-12 14:46:33.160848 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] get_object_context: 0x129b4840 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 rwstate(write n=1 w=0) oi: 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3(173'2856 client.4293.0:144172 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2856) ssc: 0x81bbce0 snapset: 0=[]:[]+head
2014-06-12 14:46:33.160871 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] find_object_context 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 @head oi=1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3(173'2856 client.4293.0:144172 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2856)
2014-06-12 14:46:33.160900 7f9d6a3e5700  5 -- op tracker -- , seq: 747882, time: 2014-06-12 14:46:33.160893, event: started, op: osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.160909 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] execute_ctx 0x589d000
2014-06-12 14:46:33.160927 7f9d6a3e5700  5 -- op tracker -- , seq: 747882, time: 2014-06-12 14:46:33.160922, event: started, op: osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.160935 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_op 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] ov 173'2856 av 173'2857 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:33.160950 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944]
2014-06-12 14:46:33.160961 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:33.160974 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op  write 3727360~466944
2014-06-12 14:46:33.160990 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] do_osd_op_effects on session 0x5c8cc00
2014-06-12 14:46:33.161002 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] make_writeable 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 snapset=0x81bbd18  snapc=0=[]
2014-06-12 14:46:33.161016 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] make_writeable 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:33.161028 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] finish_ctx 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 0x589d000 op modify  
2014-06-12 14:46:33.161042 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  set mtime to 2014-06-12 08:45:35.538887
2014-06-12 14:46:33.161064 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  final snapset 0=[]:[]+head in 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.161085 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  zeroing write result code 0
2014-06-12 14:46:33.161098 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] new_repop rep_tid 236276 on osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.161116 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] issue_repop rep_tid 236276 o 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.161144 7f9d6a3e5700  5 -- op tracker -- , seq: 747882, time: 2014-06-12 14:46:33.161138, event: waiting for subops from 23, op: osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.161169 7f9d6a3e5700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:33.161190 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2856 (0'0,173'2856] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] append_log log((0'0,173'2856], crt=0'0) [173'2857 (173'2856) modify   1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 by client.4293.0:144173 2014-06-12 08:45:35.538887]
2014-06-12 14:46:33.161215 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] add_log_entry 173'2857 (173'2856) modify   1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 by client.4293.0:144173 2014-06-12 08:45:35.538887
2014-06-12 14:46:33.161235 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] append_log  adding 1 keys
2014-06-12 14:46:33.161265 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2857, trimmed: 
2014-06-12 14:46:33.161287 7f9d6a3e5700  5 queue_transactions existing osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:46:33.161292 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:33.161294 7f9d6a3e5700  5 queue_op 0x731a8a0 seq 2602 osr(3.650 0x276dff0) 469237 bytes   (queue has 2 ops and 2023914 bytes)
2014-06-12 14:46:33.161303 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5440 (2 active)
2014-06-12 14:46:33.161301 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] eval_repop repgather(0x13d226c0 173'2857 rep_tid=236276 committed?=0 applied?=0 lock=0 op=osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.161319 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2519db00 finish
2014-06-12 14:46:33.161323 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:33.161325 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.161433 7f9d461e0700  5 -- op tracker -- , seq: 747883, time: 2014-06-12 14:46:33.161344, event: header_read, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.161444 7f9d461e0700  5 -- op tracker -- , seq: 747883, time: 2014-06-12 14:46:33.161346, event: throttled, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.161454 7f9d461e0700  5 -- op tracker -- , seq: 747883, time: 2014-06-12 14:46:33.161409, event: all_read, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.161463 7f9d461e0700  5 -- op tracker -- , seq: 747883, time: 0.000000, event: dispatched, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.161473 7f9d461e0700 20 osd.2 173 should_share_map client.4293 172.16.96.220:0/1241966 173
2014-06-12 14:46:33.161492 7f9d461e0700 15 osd.2 173 enqueue_op 0x54ae800 prio 63 cost 49152 latency 0.000148 osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.161517 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:33.161525 7f9d653db700 10 osd.2 173 dequeue_op 0x54ae800 prio 63 cost 49152 latency 0.000180 osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]
2014-06-12 14:46:33.161547 7f9d653db700  5 -- op tracker -- , seq: 747883, time: 2014-06-12 14:46:33.161542, event: reached_pg, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.161562 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:33.161576 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] handle_message: 0x54ae800
2014-06-12 14:46:33.161587 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_op osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:33.161608 7f9d653db700 15 getattr 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 '_'
2014-06-12 14:46:33.161856 7f9d653db700 10 getattr 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 '_' = 0
2014-06-12 14:46:33.161871 7f9d653db700 15 getattr 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 'snapset'
2014-06-12 14:46:33.161981 7f9d653db700 10 getattr 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 'snapset' = 0
2014-06-12 14:46:33.161987 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] populate_obc_watchers 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.161995 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.162000 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] get_object_context: creating obc from disk: 0x104742c0
2014-06-12 14:46:33.162007 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] get_object_context: 0x104742c0 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 rwstate(none n=0 w=0) oi: 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3(173'799 client.4293.0:38298 wrlock_by=unknown.0.0:0 dirty s 4194304 uv799) ssc: 0x1e15f260 snapset: 0=[]:[]+head
2014-06-12 14:46:33.162023 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] find_object_context 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 @head oi=67092105/rbd_data.109c6b8b4567.0000000000001058/head//3(173'799 client.4293.0:38298 wrlock_by=unknown.0.0:0 dirty s 4194304 uv799)
2014-06-12 14:46:33.162046 7f9d653db700  5 -- op tracker -- , seq: 747883, time: 2014-06-12 14:46:33.162040, event: started, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.162055 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] execute_ctx 0x589a000
2014-06-12 14:46:33.162069 7f9d653db700  5 -- op tracker -- , seq: 747883, time: 2014-06-12 14:46:33.162066, event: started, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.162074 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_op 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] ov 173'799 av 173'2815 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:33.162082 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152]
2014-06-12 14:46:33.162088 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:33.162097 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op  write 0~49152
2014-06-12 14:46:33.162110 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op_effects on session 0x5c8cc00
2014-06-12 14:46:33.162116 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] make_writeable 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 snapset=0x1e15f298  snapc=0=[]
2014-06-12 14:46:33.162114 7f9d7e169700 10 _do_op 0x7aea240 seq 2601 r = 0, finisher 0x29014420 0x28f9c180
2014-06-12 14:46:33.162123 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] make_writeable 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:33.162129 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] finish_ctx 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 0x589a000 op modify  
2014-06-12 14:46:33.162138 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]  set mtime to 2014-06-12 08:45:35.539017
2014-06-12 14:46:33.162138 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] op_commit: 236275
2014-06-12 14:46:33.162155 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]  final snapset 0=[]:[]+head in 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.162160 7f9d7715b700  5 -- op tracker -- , seq: 747881, time: 2014-06-12 14:46:33.162154, event: op_commit, op: osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.162173 7f9d7e169700 10 _finish_op 0x7aea240 seq 2601 osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:46:33.162173 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]  zeroing write result code 0
2014-06-12 14:46:33.162180 7f9d7815d700  5 _do_op 0x731a8a0 seq 2602 osr(3.650 0x276dff0)/0x276dff0 start
2014-06-12 14:46:33.162183 7f9d7815d700 10 _do_transaction on 0x51f7ec0
2014-06-12 14:46:33.162188 7f9d7815d700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.162182 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] new_repop rep_tid 236277 on osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.162192 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] issue_repop rep_tid 236277 o 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.162211 7f9d653db700  5 -- op tracker -- , seq: 747883, time: 2014-06-12 14:46:33.162207, event: waiting for subops from 26, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.162226 7f9d653db700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:33.162244 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2814 (0'0,173'2814] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] append_log log((0'0,173'2814], crt=0'0) [173'2815 (173'799) modify   67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 by client.4293.0:144174 2014-06-12 08:45:35.539017]
2014-06-12 14:46:33.162275 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.162270 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] add_log_entry 173'2815 (173'799) modify   67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 by client.4293.0:144174 2014-06-12 08:45:35.539017
2014-06-12 14:46:33.162299 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:33.162293 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] append_log  adding 1 keys
2014-06-12 14:46:33.162337 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2815, trimmed: 
2014-06-12 14:46:33.162361 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:33.162364 7f9d653db700  5 queue_transactions existing osr(3.105 0x276cf80)/0x276cf80
2014-06-12 14:46:33.162368 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:33.162370 7f9d653db700  5 queue_op 0x7704480 seq 2564 osr(3.105 0x276cf80) 51445 bytes   (queue has 3 ops and 2075359 bytes)
2014-06-12 14:46:33.162471 7f9d7815d700 15 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.162478 7f9d7815d700 10 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1 r = 0
2014-06-12 14:46:33.162484 7f9d7815d700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:46:33.162501 7f9d7815d700 15 _write 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 3727360~466944
2014-06-12 14:46:33.162646 7f9d7815d700 10 lookup_strip_header done  cid 3.650_head oid 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.162811 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:33.164748 7f9d7815d700 10 _generic_write 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 3727360~466944 = 0
2014-06-12 14:46:33.164808 7f9d7815d700 15 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.164828 7f9d7815d700 10 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 = 0
2014-06-12 14:46:33.164836 7f9d7815d700 15 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3
2014-06-12 14:46:33.164846 7f9d7815d700 10 _setattrs 3.650_head/1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 = 0
2014-06-12 14:46:33.164994 7f9d7815d700 20 set_header setting 17926 cid 3.650_head oid 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 parent seq 0
2014-06-12 14:46:33.165044 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:33.165061 7f9d7815d700 20 set_header setting 2114 cid meta oid 1fe1320a/pglog_3.650/0//-1 parent seq 0
2014-06-12 14:46:33.165074 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:33.165753 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5440 (1 active)
2014-06-12 14:46:33.165763 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:33.165759 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] op_applied: 236275
2014-06-12 14:46:33.165792 7f9d7795c700  5 -- op tracker -- , seq: 747881, time: 2014-06-12 14:46:33.165781, event: op_applied, op: osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.165804 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] op_applied on primary on version 173'2856
2014-06-12 14:46:33.165838 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54758c0 (2 active)
2014-06-12 14:46:33.165844 7f9d82171700  5 _do_op 0x7704480 seq 2564 osr(3.105 0x276cf80)/0x276cf80 start
2014-06-12 14:46:33.165850 7f9d82171700 10 _do_transaction on 0x2a32c540
2014-06-12 14:46:33.165856 7f9d82171700 15 _omap_setkeys meta/1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.165839 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] eval_repop repgather(0x2237200 173'2815 rep_tid=236277 committed?=0 applied?=0 lock=0 op=osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.165865 7f9d653db700 10 osd.2 173 dequeue_op 0x54ae800 finish
2014-06-12 14:46:33.165870 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:33.165874 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.165983 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.166095 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:33.166457 7f9d57543700  5 -- op tracker -- , seq: 747884, time: 2014-06-12 14:46:33.166430, event: header_read, op: osd_sub_op_reply(client.4293.0:144172 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.166465 7f9d57543700  5 -- op tracker -- , seq: 747884, time: 2014-06-12 14:46:33.166431, event: throttled, op: osd_sub_op_reply(client.4293.0:144172 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.166470 7f9d57543700  5 -- op tracker -- , seq: 747884, time: 2014-06-12 14:46:33.166438, event: all_read, op: osd_sub_op_reply(client.4293.0:144172 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.166474 7f9d57543700  5 -- op tracker -- , seq: 747884, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144172 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.166477 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144172 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:33.166481 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:33.166486 7f9d57543700 15 osd.2 173 enqueue_op 0x31d60500 prio 196 cost 0 latency 0.000056 osd_sub_op_reply(client.4293.0:144172 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:33.166509 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:33.166515 7f9d63bd8700 10 osd.2 173 dequeue_op 0x31d60500 prio 196 cost 0 latency 0.000085 osd_sub_op_reply(client.4293.0:144172 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0) v2 pg pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]
2014-06-12 14:46:33.166529 7f9d63bd8700  5 -- op tracker -- , seq: 747884, time: 2014-06-12 14:46:33.166527, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144172 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.166535 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] handle_message: 0x31d60500
2014-06-12 14:46:33.166546 7f9d63bd8700  5 -- op tracker -- , seq: 747884, time: 2014-06-12 14:46:33.166543, event: started, op: osd_sub_op_reply(client.4293.0:144172 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.166549 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] sub_op_modify_reply: tid 236275 op  ack_type 4 from 23
2014-06-12 14:46:33.166561 7f9d63bd8700  5 -- op tracker -- , seq: 747881, time: 2014-06-12 14:46:33.166556, event: sub_op_commit_rec, op: osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.166566 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] repop_all_applied: repop tid 236275 all applied 
2014-06-12 14:46:33.166573 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] eval_repop repgather(0x5312a80 173'2856 rep_tid=236275 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.166583 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean]  sending ack on repgather(0x5312a80 173'2856 rep_tid=236275 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) 0x1b774a00
2014-06-12 14:46:33.166604 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2855 crt=0'0 lcod 173'2854 mlcod 173'2854 active+clean] repop_all_committed: repop tid 236275 all committed 
2014-06-12 14:46:33.166617 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2854 active+clean] eval_repop repgather(0x5312a80 173'2856 rep_tid=236275 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.166637 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2854 active+clean] log_op_stats osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4 inb 1552663 outb 0 rlat 0.021567 lat 0.021600
2014-06-12 14:46:33.166649 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2854 active+clean] publish_stats_to_osd 173:3715
2014-06-12 14:46:33.166658 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2854 active+clean]  sending commit on repgather(0x5312a80 173'2856 rep_tid=236275 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) 0x3206a280
2014-06-12 14:46:33.166682 7f9d63bd8700  5 -- op tracker -- , seq: 747881, time: 2014-06-12 14:46:33.166677, event: commit_sent, op: osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.166692 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2854 active+clean]  requeue_ops 
2014-06-12 14:46:33.166701 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]  removing repgather(0x5312a80 173'2856 rep_tid=236275 committed?=1 applied?=1 op=osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:33.166713 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]    q front is repgather(0x5312a80 173'2856 rep_tid=236275 committed?=1 applied?=1 op=osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:33.166723 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] remove_repop repgather(0x5312a80 173'2856 rep_tid=236275 committed?=1 applied?=1 op=osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:33.166733 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]  requeue_ops 
2014-06-12 14:46:33.166745 7f9d63bd8700  5 -- op tracker -- , seq: 747881, time: 2014-06-12 14:46:33.166742, event: done, op: osd_op(client.4293.0:144172 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 2174976~1552384] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.166762 7f9d63bd8700 10 osd.2 173 dequeue_op 0x31d60500 finish
2014-06-12 14:46:33.166769 7f9d63bd8700  5 -- op tracker -- , seq: 747884, time: 2014-06-12 14:46:33.166766, event: done, op: osd_sub_op_reply(client.4293.0:144172 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.166780 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:33.166783 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.172377 7f9d7815d700 10 _do_op 0x731a8a0 seq 2602 r = 0, finisher 0x5d412c0 0x26c4fa40
2014-06-12 14:46:33.172402 7f9d7815d700 10 _finish_op 0x731a8a0 seq 2602 osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:46:33.172413 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5440 (1 active)
2014-06-12 14:46:33.172423 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:33.172422 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] op_applied: 236276
2014-06-12 14:46:33.172461 7f9d7795c700  5 -- op tracker -- , seq: 747882, time: 2014-06-12 14:46:33.172450, event: op_applied, op: osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.172480 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] op_applied on primary on version 173'2857
2014-06-12 14:46:33.172748 7f9d57543700  5 -- op tracker -- , seq: 747885, time: 2014-06-12 14:46:33.172688, event: header_read, op: osd_sub_op_reply(client.4293.0:144173 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.172760 7f9d57543700  5 -- op tracker -- , seq: 747885, time: 2014-06-12 14:46:33.172690, event: throttled, op: osd_sub_op_reply(client.4293.0:144173 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.172769 7f9d57543700  5 -- op tracker -- , seq: 747885, time: 2014-06-12 14:46:33.172713, event: all_read, op: osd_sub_op_reply(client.4293.0:144173 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.172775 7f9d57543700  5 -- op tracker -- , seq: 747885, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144173 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.172782 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144173 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:33.172789 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:33.172799 7f9d57543700 15 osd.2 173 enqueue_op 0x329ea800 prio 196 cost 0 latency 0.000110 osd_sub_op_reply(client.4293.0:144173 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:33.172965 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] op_commit: 236276
2014-06-12 14:46:33.173000 7f9d7715b700  5 -- op tracker -- , seq: 747882, time: 2014-06-12 14:46:33.172992, event: op_commit, op: osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.173221 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:33.173249 7f9d82171700 15 _omap_rmkeys meta/1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.173257 7f9d82171700 10 _omap_rmkeys meta/1fe2fcda/pglog_3.105/0//-1 r = 0
2014-06-12 14:46:33.173264 7f9d82171700 15 _omap_setkeys meta/1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.173281 7f9d82171700 15 _write 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 0~49152
2014-06-12 14:46:33.173427 7f9d82171700 10 lookup_strip_header done  cid 3.105_head oid 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.173593 7f9d56432700  5 -- op tracker -- , seq: 747886, time: 2014-06-12 14:46:33.172726, event: header_read, op: osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.173605 7f9d56432700  5 -- op tracker -- , seq: 747886, time: 2014-06-12 14:46:33.172727, event: throttled, op: osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.173653 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:33.173657 7f9d56432700  5 -- op tracker -- , seq: 747886, time: 2014-06-12 14:46:33.173553, event: all_read, op: osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.173668 7f9d56432700  5 -- op tracker -- , seq: 747886, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.173678 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:33.173685 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:33.173669 7f9d68be2700 10 osd.2 173 dequeue_op 0x329ea800 prio 196 cost 0 latency 0.000980 osd_sub_op_reply(client.4293.0:144173 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0) v2 pg pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]
2014-06-12 14:46:33.173699 7f9d56432700 15 osd.2 173 enqueue_op 0x23e09800 prio 127 cost 516903 latency 0.000972 osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:33.173703 7f9d68be2700  5 -- op tracker -- , seq: 747885, time: 2014-06-12 14:46:33.173698, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144173 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.173718 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] handle_message: 0x329ea800
2014-06-12 14:46:33.173737 7f9d68be2700  5 -- op tracker -- , seq: 747885, time: 2014-06-12 14:46:33.173734, event: started, op: osd_sub_op_reply(client.4293.0:144173 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.173744 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] sub_op_modify_reply: tid 236276 op  ack_type 4 from 23
2014-06-12 14:46:33.173768 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:33.173772 7f9d68be2700  5 -- op tracker -- , seq: 747882, time: 2014-06-12 14:46:33.173760, event: sub_op_commit_rec, op: osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.173775 7f9d623d5700 10 osd.2 173 dequeue_op 0x23e09800 prio 127 cost 516903 latency 0.001048 osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.51e( v 173'2681 (0'0,173'2681] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2680 active]
2014-06-12 14:46:33.173801 7f9d623d5700  5 -- op tracker -- , seq: 747886, time: 2014-06-12 14:46:33.173797, event: reached_pg, op: osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.173797 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] repop_all_applied: repop tid 236276 all applied 
2014-06-12 14:46:33.173809 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2681 (0'0,173'2681] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2680 active] handle_message: 0x23e09800
2014-06-12 14:46:33.173816 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:33.173808 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] eval_repop repgather(0x13d226c0 173'2857 rep_tid=236276 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.173819 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2681 (0'0,173'2681] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2680 active] sub_op_modify trans 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 v 173'2682 (transaction) 174
2014-06-12 14:46:33.173824 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean]  sending ack on repgather(0x13d226c0 173'2857 rep_tid=236276 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) 0x22b24f00
2014-06-12 14:46:33.173835 7f9d623d5700  5 -- op tracker -- , seq: 747886, time: 2014-06-12 14:46:33.173832, event: started, op: osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.173855 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2856 crt=0'0 lcod 173'2855 mlcod 173'2855 active+clean] repop_all_committed: repop tid 236276 all committed 
2014-06-12 14:46:33.173852 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2681 (0'0,173'2681] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2680 active] append_log log((0'0,173'2681], crt=0'0) [173'2682 (173'2681) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145051 2014-06-12 08:44:57.414475]
2014-06-12 14:46:33.173864 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] eval_repop repgather(0x13d226c0 173'2857 rep_tid=236276 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.173877 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2682 (0'0,173'2682] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2680 active] add_log_entry 173'2682 (173'2681) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145051 2014-06-12 08:44:57.414475
2014-06-12 14:46:33.173880 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] log_op_stats osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4 inb 467223 outb 0 rlat 0.014762 lat 0.014787
2014-06-12 14:46:33.173896 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean] publish_stats_to_osd 173:3716
2014-06-12 14:46:33.173900 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2682 (0'0,173'2682] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2680 active] append_log  adding 1 keys
2014-06-12 14:46:33.173909 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean]  sending commit on repgather(0x13d226c0 173'2857 rep_tid=236276 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4) 0x28ea3b80
2014-06-12 14:46:33.173935 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2682, trimmed: 
2014-06-12 14:46:33.173936 7f9d68be2700  5 -- op tracker -- , seq: 747882, time: 2014-06-12 14:46:33.173931, event: commit_sent, op: osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.173944 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2855 active+clean]  requeue_ops 
2014-06-12 14:46:33.173959 7f9d623d5700  5 -- op tracker -- , seq: 747886, time: 2014-06-12 14:46:33.173954, event: started, op: osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.173955 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  removing repgather(0x13d226c0 173'2857 rep_tid=236276 committed?=1 applied?=1 op=osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:33.173968 7f9d623d5700  5 queue_transactions existing osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:33.173973 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:33.173974 7f9d623d5700  5 queue_op 0xbdfcc00 seq 2518 osr(3.51e 0x276db50) 518389 bytes   (queue has 2 ops and 569834 bytes)
2014-06-12 14:46:33.173968 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]    q front is repgather(0x13d226c0 173'2857 rep_tid=236276 committed?=1 applied?=1 op=osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:33.173981 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.51e( v 173'2682 (0'0,173'2682] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2680 active] do_sub_op osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:33.173981 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] remove_repop repgather(0x13d226c0 173'2857 rep_tid=236276 committed?=1 applied?=1 op=osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:33.173994 7f9d623d5700 10 osd.2 173 dequeue_op 0x23e09800 finish
2014-06-12 14:46:33.173997 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:33.174000 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.173994 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  requeue_ops 
2014-06-12 14:46:33.174005 7f9d82171700 10 _generic_write 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 0~49152 = 0
2014-06-12 14:46:33.174008 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fb00 (2 active)
2014-06-12 14:46:33.174014 7f9d7e96a700  5 _do_op 0xbdfcc00 seq 2518 osr(3.51e 0x276db50)/0x276db50 start
2014-06-12 14:46:33.174018 7f9d7e96a700 10 _do_transaction on 0x62750b0
2014-06-12 14:46:33.174023 7f9d7e96a700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:33.174024 7f9d82171700 15 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.174021 7f9d68be2700  5 -- op tracker -- , seq: 747882, time: 2014-06-12 14:46:33.174016, event: done, op: osd_op(client.4293.0:144173 rbd_data.109c6b8b4567.0000000000001057 [set-alloc-hint object_size 4194304 write_size 4194304,write 3727360~466944] 3.1a8be50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.174037 7f9d82171700 10 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:46:33.174044 7f9d82171700 15 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.174045 7f9d68be2700 10 osd.2 173 dequeue_op 0x329ea800 finish
2014-06-12 14:46:33.174052 7f9d82171700 10 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:46:33.174052 7f9d68be2700  5 -- op tracker -- , seq: 747885, time: 2014-06-12 14:46:33.174049, event: done, op: osd_sub_op_reply(client.4293.0:144173 3.650 1a8be50/rbd_data.109c6b8b4567.0000000000001057/head//3 [] ondisk, result = 0)
2014-06-12 14:46:33.174066 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:33.174068 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.174160 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:33.174188 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:33.174187 7f9d82171700 20 set_header setting 17927 cid 3.105_head oid 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 parent seq 0
2014-06-12 14:46:33.174211 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:33.174224 7f9d82171700 20 set_header setting 1990 cid meta oid 1fe2fcda/pglog_3.105/0//-1 parent seq 0
2014-06-12 14:46:33.174233 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:33.174651 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:33.174655 7f9d82171700 10 _do_op 0x7704480 seq 2564 r = 0, finisher 0x73c1e20 0x111fb080
2014-06-12 14:46:33.174666 7f9d82171700 10 _finish_op 0x7704480 seq 2564 osr(3.105 0x276cf80)/0x276cf80
2014-06-12 14:46:33.174674 7f9d7e96a700 15 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:33.174676 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54758c0 (1 active)
2014-06-12 14:46:33.174679 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:33.174679 7f9d7e96a700 10 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1 r = 0
2014-06-12 14:46:33.174684 7f9d7e96a700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:33.174682 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_applied: 236277
2014-06-12 14:46:33.174713 7f9d7e96a700 15 _write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 1552384~516096
2014-06-12 14:46:33.174713 7f9d7795c700  5 -- op tracker -- , seq: 747883, time: 2014-06-12 14:46:33.174703, event: op_applied, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.174721 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_applied on primary on version 173'2815
2014-06-12 14:46:33.174873 7f9d7e96a700 10 lookup_strip_header done  cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:33.174889 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:33.176010 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_commit: 236277
2014-06-12 14:46:33.176045 7f9d7715b700  5 -- op tracker -- , seq: 747883, time: 2014-06-12 14:46:33.176036, event: op_commit, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.176983 7f9d7e96a700 10 _generic_write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 1552384~516096 = 0
2014-06-12 14:46:33.177031 7f9d7e96a700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:33.177046 7f9d7e96a700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:33.177054 7f9d7e96a700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:33.177063 7f9d7e96a700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:33.177276 7f9d7e96a700 20 set_header setting 16949 cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 parent seq 0
2014-06-12 14:46:33.177344 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:33.177361 7f9d7e96a700 20 set_header setting 2288 cid meta oid 1fe1612b/pglog_3.51e/0//-1 parent seq 0
2014-06-12 14:46:33.177372 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:33.180142 7f9d7e96a700 10 _do_op 0xbdfcc00 seq 2518 r = 0, finisher 0xabf2c60 0
2014-06-12 14:46:33.180161 7f9d7e96a700 10 _finish_op 0xbdfcc00 seq 2518 osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:33.180180 7f9d7715b700  5 -- op tracker -- , seq: 747886, time: 2014-06-12 14:46:33.180170, event: commit_sent, op: osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.180191 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2682 (0'0,173'2682] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2680 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:33.180217 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:33.180380 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fb00 (0 active)
2014-06-12 14:46:33.180385 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:33.180425 7f9d7795c700  5 -- op tracker -- , seq: 747886, time: 2014-06-12 14:46:33.180416, event: sub_op_applied, op: osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.180435 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2682 (0'0,173'2682] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2681 active] sub_op_modify_applied on 0x6274fc0 op osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:33.180455 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2682 (0'0,173'2682] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2681 active] op_applied on primary on version 173'2682
2014-06-12 14:46:33.180468 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2682 (0'0,173'2682] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2681 active] op_applied on replica on version 173'2682
2014-06-12 14:46:33.180488 7f9d7795c700  5 -- op tracker -- , seq: 747886, time: 2014-06-12 14:46:33.180485, event: done, op: osd_sub_op(client.4308.0:145051 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.184093 7f9d461e0700  5 -- op tracker -- , seq: 747887, time: 2014-06-12 14:46:33.173325, event: header_read, op: osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.184109 7f9d461e0700  5 -- op tracker -- , seq: 747887, time: 2014-06-12 14:46:33.173327, event: throttled, op: osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.184118 7f9d461e0700  5 -- op tracker -- , seq: 747887, time: 2014-06-12 14:46:33.184042, event: all_read, op: osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.184126 7f9d461e0700  5 -- op tracker -- , seq: 747887, time: 0.000000, event: dispatched, op: osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.184135 7f9d461e0700 20 osd.2 173 should_share_map client.4293 172.16.96.220:0/1241966 173
2014-06-12 14:46:33.184169 7f9d461e0700 15 osd.2 173 enqueue_op 0x54aec00 prio 63 cost 2064384 latency 0.010843 osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.184203 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:33.184216 7f9d643d9700 10 osd.2 173 dequeue_op 0x54aec00 prio 63 cost 2064384 latency 0.010890 osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]
2014-06-12 14:46:33.184250 7f9d643d9700  5 -- op tracker -- , seq: 747887, time: 2014-06-12 14:46:33.184244, event: reached_pg, op: osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.184260 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:33.184276 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] handle_message: 0x54aec00
2014-06-12 14:46:33.184288 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_op osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:33.184313 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] get_object_context: found obc in cache: 0x104742c0
2014-06-12 14:46:33.184323 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] get_object_context: 0x104742c0 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 rwstate(write n=1 w=0) oi: 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3(173'2815 client.4293.0:144174 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2815) ssc: 0x1e15f260 snapset: 0=[]:[]+head
2014-06-12 14:46:33.184338 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] find_object_context 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 @head oi=67092105/rbd_data.109c6b8b4567.0000000000001058/head//3(173'2815 client.4293.0:144174 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2815)
2014-06-12 14:46:33.184363 7f9d643d9700  5 -- op tracker -- , seq: 747887, time: 2014-06-12 14:46:33.184358, event: started, op: osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.184371 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] execute_ctx 0x5e4ca00
2014-06-12 14:46:33.184389 7f9d643d9700  5 -- op tracker -- , seq: 747887, time: 2014-06-12 14:46:33.184385, event: started, op: osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.184396 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_op 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] ov 173'2815 av 173'2816 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:33.184410 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384]
2014-06-12 14:46:33.184420 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:33.184434 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op  write 49152~2064384
2014-06-12 14:46:33.184451 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op_effects on session 0x5c8cc00
2014-06-12 14:46:33.184462 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] make_writeable 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 snapset=0x1e15f298  snapc=0=[]
2014-06-12 14:46:33.184474 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] make_writeable 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:33.184485 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] finish_ctx 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 0x5e4ca00 op modify  
2014-06-12 14:46:33.184497 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]  set mtime to 2014-06-12 08:45:35.550315
2014-06-12 14:46:33.184519 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]  final snapset 0=[]:[]+head in 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.184538 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]  zeroing write result code 0
2014-06-12 14:46:33.184550 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] new_repop rep_tid 236278 on osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.184568 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] issue_repop rep_tid 236278 o 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.184597 7f9d643d9700  5 -- op tracker -- , seq: 747887, time: 2014-06-12 14:46:33.184592, event: waiting for subops from 26, op: osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.184620 7f9d643d9700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:33.184642 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2815 (0'0,173'2815] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] append_log log((0'0,173'2815], crt=0'0) [173'2816 (173'2815) modify   67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 by client.4293.0:144175 2014-06-12 08:45:35.550315]
2014-06-12 14:46:33.184668 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] add_log_entry 173'2816 (173'2815) modify   67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 by client.4293.0:144175 2014-06-12 08:45:35.550315
2014-06-12 14:46:33.184689 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] append_log  adding 1 keys
2014-06-12 14:46:33.184720 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2816, trimmed: 
2014-06-12 14:46:33.184742 7f9d643d9700  5 queue_transactions existing osr(3.105 0x276cf80)/0x276cf80
2014-06-12 14:46:33.184746 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:33.184747 7f9d643d9700  5 queue_op 0x92cb500 seq 2565 osr(3.105 0x276cf80) 2066677 bytes   (queue has 1 ops and 2066677 bytes)
2014-06-12 14:46:33.184755 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] eval_repop repgather(0x1a445200 173'2816 rep_tid=236278 committed?=0 applied?=0 lock=0 op=osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.184773 7f9d643d9700 10 osd.2 173 dequeue_op 0x54aec00 finish
2014-06-12 14:46:33.184777 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:33.184780 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.185633 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54758c0 (1 active)
2014-06-12 14:46:33.185645 7f9d7d167700  5 _do_op 0x92cb500 seq 2565 osr(3.105 0x276cf80)/0x276cf80 start
2014-06-12 14:46:33.185652 7f9d7d167700 10 _do_transaction on 0x5147c80
2014-06-12 14:46:33.185660 7f9d7d167700 15 _omap_setkeys meta/1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.185788 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.185817 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:33.185867 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:33.185885 7f9d7d167700 15 _omap_rmkeys meta/1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.185891 7f9d7d167700 10 _omap_rmkeys meta/1fe2fcda/pglog_3.105/0//-1 r = 0
2014-06-12 14:46:33.185897 7f9d7d167700 15 _omap_setkeys meta/1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.185917 7f9d7d167700 15 _write 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 49152~2064384
2014-06-12 14:46:33.186596 7f9d7d167700 10 lookup_strip_header done  cid 3.105_head oid 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.186739 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:33.196694 7f9d7d167700 10 _generic_write 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 49152~2064384 = 0
2014-06-12 14:46:33.197035 7f9d7d167700 15 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.197056 7f9d7d167700 10 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:46:33.197064 7f9d7d167700 15 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.197071 7f9d7d167700 10 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:46:33.197749 7f9d7d167700 20 set_header setting 17927 cid 3.105_head oid 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 parent seq 0
2014-06-12 14:46:33.197942 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:33.197957 7f9d7d167700 20 set_header setting 1990 cid meta oid 1fe2fcda/pglog_3.105/0//-1 parent seq 0
2014-06-12 14:46:33.197966 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:33.207416 7f9d7d167700 10 _do_op 0x92cb500 seq 2565 r = 0, finisher 0x7be7700 0x1e5ca340
2014-06-12 14:46:33.207436 7f9d7d167700 10 _finish_op 0x92cb500 seq 2565 osr(3.105 0x276cf80)/0x276cf80
2014-06-12 14:46:33.207445 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54758c0 (0 active)
2014-06-12 14:46:33.207453 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:33.207440 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_commit: 236278
2014-06-12 14:46:33.207470 7f9d7715b700  5 -- op tracker -- , seq: 747887, time: 2014-06-12 14:46:33.207462, event: op_commit, op: osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.207484 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_applied: 236278
2014-06-12 14:46:33.207497 7f9d7795c700  5 -- op tracker -- , seq: 747887, time: 2014-06-12 14:46:33.207493, event: op_applied, op: osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.207507 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_applied on primary on version 173'2816
2014-06-12 14:46:33.255910 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:33.256053 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:33.442823 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:33.442948 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:33.476876 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:33.476932 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:33.476932 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 11.84 >= max 0.5 = no, load too high
2014-06-12 14:46:33.476944 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:33.476947 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92641 < max (604800 seconds)
2014-06-12 14:46:33.476964 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:33.476966 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:33.476967 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:33.476968 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 2; oldest is 0.315624 seconds old
2014-06-12 14:46:33.508253 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:33.508587 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:33.519001 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:33.519163 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:33.559153 7f9d56432700  5 -- op tracker -- , seq: 747888, time: 2014-06-12 14:46:33.558157, event: header_read, op: osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.559183 7f9d56432700  5 -- op tracker -- , seq: 747888, time: 2014-06-12 14:46:33.558162, event: throttled, op: osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.559192 7f9d56432700  5 -- op tracker -- , seq: 747888, time: 2014-06-12 14:46:33.559098, event: all_read, op: osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.559220 7f9d56432700  5 -- op tracker -- , seq: 747888, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.559233 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:33.559241 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:33.559252 7f9d56432700 15 osd.2 173 enqueue_op 0x23e09a00 prio 127 cost 516903 latency 0.001095 osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:33.559286 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:33.559302 7f9d633d7700 10 osd.2 173 dequeue_op 0x23e09a00 prio 127 cost 516903 latency 0.001145 osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.51e( v 173'2682 (0'0,173'2682] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2681 active]
2014-06-12 14:46:33.559336 7f9d633d7700  5 -- op tracker -- , seq: 747888, time: 2014-06-12 14:46:33.559332, event: reached_pg, op: osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.559356 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2682 (0'0,173'2682] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2681 active] handle_message: 0x23e09a00
2014-06-12 14:46:33.559380 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2682 (0'0,173'2682] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2681 active] sub_op_modify trans 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 v 173'2683 (transaction) 174
2014-06-12 14:46:33.559400 7f9d633d7700  5 -- op tracker -- , seq: 747888, time: 2014-06-12 14:46:33.559396, event: started, op: osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.559449 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2682 (0'0,173'2682] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2681 active] append_log log((0'0,173'2682], crt=0'0) [173'2683 (173'2682) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145052 2014-06-12 08:44:59.124783]
2014-06-12 14:46:33.559485 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2683 (0'0,173'2683] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2681 active] add_log_entry 173'2683 (173'2682) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145052 2014-06-12 08:44:59.124783
2014-06-12 14:46:33.559523 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2683 (0'0,173'2683] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2681 active] append_log  adding 1 keys
2014-06-12 14:46:33.559587 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2683, trimmed: 
2014-06-12 14:46:33.559639 7f9d633d7700  5 -- op tracker -- , seq: 747888, time: 2014-06-12 14:46:33.559626, event: started, op: osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.559652 7f9d633d7700  5 queue_transactions existing osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:33.559657 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:33.559659 7f9d633d7700  5 queue_op 0x24c66d20 seq 2519 osr(3.51e 0x276db50) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:33.559667 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.51e( v 173'2683 (0'0,173'2683] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2681 active] do_sub_op osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:33.559700 7f9d633d7700 10 osd.2 173 dequeue_op 0x23e09a00 finish
2014-06-12 14:46:33.559705 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:33.559708 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.559690 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fb00 (1 active)
2014-06-12 14:46:33.559716 7f9d8116f700  5 _do_op 0x24c66d20 seq 2519 osr(3.51e 0x276db50)/0x276db50 start
2014-06-12 14:46:33.559728 7f9d8116f700 10 _do_transaction on 0x5cb4470
2014-06-12 14:46:33.559734 7f9d8116f700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:33.559949 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:33.559979 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:33.560042 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:33.560395 7f9d8116f700 15 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:33.560404 7f9d8116f700 10 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1 r = 0
2014-06-12 14:46:33.560415 7f9d8116f700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:33.560430 7f9d8116f700 15 _write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 2068480~516096
2014-06-12 14:46:33.560553 7f9d8116f700 10 lookup_strip_header done  cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:33.560686 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:33.562215 7f9d8116f700 10 _generic_write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 2068480~516096 = 0
2014-06-12 14:46:33.562253 7f9d8116f700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:33.562288 7f9d8116f700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:33.562293 7f9d8116f700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:33.562298 7f9d8116f700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:33.562439 7f9d8116f700 20 set_header setting 16949 cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 parent seq 0
2014-06-12 14:46:33.562480 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:33.562490 7f9d8116f700 20 set_header setting 2288 cid meta oid 1fe1612b/pglog_3.51e/0//-1 parent seq 0
2014-06-12 14:46:33.562497 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:33.565171 7f9d8116f700 10 _do_op 0x24c66d20 seq 2519 r = 0, finisher 0x66b3da0 0
2014-06-12 14:46:33.565200 7f9d8116f700 10 _finish_op 0x24c66d20 seq 2519 osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:33.565210 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fb00 (0 active)
2014-06-12 14:46:33.565216 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:33.565243 7f9d7715b700  5 -- op tracker -- , seq: 747888, time: 2014-06-12 14:46:33.565222, event: commit_sent, op: osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.565289 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2683 (0'0,173'2683] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2681 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:33.565326 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:33.565402 7f9d7795c700  5 -- op tracker -- , seq: 747888, time: 2014-06-12 14:46:33.565390, event: sub_op_applied, op: osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.565422 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2683 (0'0,173'2683] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2682 active] sub_op_modify_applied on 0x5cb4380 op osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:33.565449 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2683 (0'0,173'2683] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2682 active] op_applied on primary on version 173'2683
2014-06-12 14:46:33.565462 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2683 (0'0,173'2683] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2682 active] op_applied on replica on version 173'2683
2014-06-12 14:46:33.565493 7f9d7795c700  5 -- op tracker -- , seq: 747888, time: 2014-06-12 14:46:33.565487, event: done, op: osd_sub_op(client.4308.0:145052 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:33.637578 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:33.637646 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:33.669748 7f9d461e0700  5 -- op tracker -- , seq: 747889, time: 2014-06-12 14:46:33.665868, event: header_read, op: osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.669768 7f9d461e0700  5 -- op tracker -- , seq: 747889, time: 2014-06-12 14:46:33.665871, event: throttled, op: osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.669791 7f9d461e0700  5 -- op tracker -- , seq: 747889, time: 2014-06-12 14:46:33.669696, event: all_read, op: osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.669808 7f9d461e0700  5 -- op tracker -- , seq: 747889, time: 0.000000, event: dispatched, op: osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.669817 7f9d461e0700 20 osd.2 173 should_share_map client.4293 172.16.96.220:0/1241966 173
2014-06-12 14:46:33.669847 7f9d461e0700 15 osd.2 173 enqueue_op 0x330c6400 prio 63 cost 1548288 latency 0.003979 osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.669885 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:33.669897 7f9d61bd4700 10 osd.2 173 dequeue_op 0x330c6400 prio 63 cost 1548288 latency 0.004029 osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]
2014-06-12 14:46:33.669920 7f9d61bd4700  5 -- op tracker -- , seq: 747889, time: 2014-06-12 14:46:33.669916, event: reached_pg, op: osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.669928 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:33.669939 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] handle_message: 0x330c6400
2014-06-12 14:46:33.669946 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_op osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:33.669964 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] get_object_context: found obc in cache: 0x104742c0
2014-06-12 14:46:33.669973 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] get_object_context: 0x104742c0 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 rwstate(write n=2 w=0) oi: 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3(173'2816 client.4293.0:144175 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2816) ssc: 0x1e15f260 snapset: 0=[]:[]+head
2014-06-12 14:46:33.669987 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] find_object_context 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 @head oi=67092105/rbd_data.109c6b8b4567.0000000000001058/head//3(173'2816 client.4293.0:144175 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2816)
2014-06-12 14:46:33.670005 7f9d61bd4700  5 -- op tracker -- , seq: 747889, time: 2014-06-12 14:46:33.670002, event: started, op: osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.670010 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] execute_ctx 0x30f7de00
2014-06-12 14:46:33.670022 7f9d61bd4700  5 -- op tracker -- , seq: 747889, time: 2014-06-12 14:46:33.670018, event: started, op: osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.670026 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_op 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] ov 173'2816 av 173'2817 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:33.670035 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288]
2014-06-12 14:46:33.670041 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:33.670050 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op  write 2113536~1548288
2014-06-12 14:46:33.670061 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op_effects on session 0x5c8cc00
2014-06-12 14:46:33.670068 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] make_writeable 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 snapset=0x1e15f298  snapc=0=[]
2014-06-12 14:46:33.670075 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] make_writeable 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:33.670081 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] finish_ctx 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 0x30f7de00 op modify  
2014-06-12 14:46:33.670089 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]  set mtime to 2014-06-12 08:45:36.043687
2014-06-12 14:46:33.670105 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]  final snapset 0=[]:[]+head in 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.670117 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]  zeroing write result code 0
2014-06-12 14:46:33.670126 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] new_repop rep_tid 236279 on osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.670138 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] issue_repop rep_tid 236279 o 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.670156 7f9d61bd4700  5 -- op tracker -- , seq: 747889, time: 2014-06-12 14:46:33.670153, event: waiting for subops from 26, op: osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.670173 7f9d61bd4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:33.670189 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2816 (0'0,173'2816] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] append_log log((0'0,173'2816], crt=0'0) [173'2817 (173'2816) modify   67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 by client.4293.0:144176 2014-06-12 08:45:36.043687]
2014-06-12 14:46:33.670207 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] add_log_entry 173'2817 (173'2816) modify   67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 by client.4293.0:144176 2014-06-12 08:45:36.043687
2014-06-12 14:46:33.670221 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] append_log  adding 1 keys
2014-06-12 14:46:33.670244 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2817, trimmed: 
2014-06-12 14:46:33.670261 7f9d61bd4700  5 queue_transactions existing osr(3.105 0x276cf80)/0x276cf80
2014-06-12 14:46:33.670265 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:33.670266 7f9d61bd4700  5 queue_op 0x80561e0 seq 2566 osr(3.105 0x276cf80) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:46:33.670274 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54758c0 (1 active)
2014-06-12 14:46:33.670271 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] eval_repop repgather(0x10192240 173'2817 rep_tid=236279 committed?=0 applied?=0 lock=0 op=osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.670283 7f9d61bd4700 10 osd.2 173 dequeue_op 0x330c6400 finish
2014-06-12 14:46:33.670282 7f9d79960700  5 _do_op 0x80561e0 seq 2566 osr(3.105 0x276cf80)/0x276cf80 start
2014-06-12 14:46:33.670285 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:33.670287 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.670287 7f9d79960700 10 _do_transaction on 0x15acf800
2014-06-12 14:46:33.670294 7f9d79960700 15 _omap_setkeys meta/1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.670435 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.670461 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:33.670511 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:33.670630 7f9d79960700 15 _omap_rmkeys meta/1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.670638 7f9d79960700 10 _omap_rmkeys meta/1fe2fcda/pglog_3.105/0//-1 r = 0
2014-06-12 14:46:33.670644 7f9d79960700 15 _omap_setkeys meta/1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.670660 7f9d79960700 15 _write 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 2113536~1548288
2014-06-12 14:46:33.670805 7f9d79960700 10 lookup_strip_header done  cid 3.105_head oid 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.671188 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:33.677168 7f9d79960700 10 _generic_write 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 2113536~1548288 = 0
2014-06-12 14:46:33.677335 7f9d79960700 15 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.677360 7f9d79960700 10 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:46:33.677369 7f9d79960700 15 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.677379 7f9d79960700 10 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:46:33.677810 7f9d79960700 20 set_header setting 17927 cid 3.105_head oid 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 parent seq 0
2014-06-12 14:46:33.677912 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:33.677923 7f9d79960700 20 set_header setting 1990 cid meta oid 1fe2fcda/pglog_3.105/0//-1 parent seq 0
2014-06-12 14:46:33.677930 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:33.683574 7f9d79960700 10 _do_op 0x80561e0 seq 2566 r = 0, finisher 0x662da20 0x4417d40
2014-06-12 14:46:33.683591 7f9d79960700 10 _finish_op 0x80561e0 seq 2566 osr(3.105 0x276cf80)/0x276cf80
2014-06-12 14:46:33.683597 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54758c0 (0 active)
2014-06-12 14:46:33.683609 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:33.683611 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_applied: 236279
2014-06-12 14:46:33.683652 7f9d7795c700  5 -- op tracker -- , seq: 747889, time: 2014-06-12 14:46:33.683638, event: op_applied, op: osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.683662 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_applied on primary on version 173'2817
2014-06-12 14:46:33.683706 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_commit: 236279
2014-06-12 14:46:33.683761 7f9d7715b700  5 -- op tracker -- , seq: 747889, time: 2014-06-12 14:46:33.683750, event: op_commit, op: osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.714053 7f9d461e0700  5 -- op tracker -- , seq: 747890, time: 2014-06-12 14:46:33.712585, event: header_read, op: osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.714070 7f9d461e0700  5 -- op tracker -- , seq: 747890, time: 2014-06-12 14:46:33.712589, event: throttled, op: osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.714080 7f9d461e0700  5 -- op tracker -- , seq: 747890, time: 2014-06-12 14:46:33.713992, event: all_read, op: osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.714089 7f9d461e0700  5 -- op tracker -- , seq: 747890, time: 0.000000, event: dispatched, op: osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.714099 7f9d461e0700 20 osd.2 173 should_share_map client.4293 172.16.96.220:0/1241966 173
2014-06-12 14:46:33.714133 7f9d461e0700 15 osd.2 173 enqueue_op 0x330c6c00 prio 63 cost 532480 latency 0.001548 osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.714169 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:33.714178 7f9d673df700 10 osd.2 173 dequeue_op 0x330c6c00 prio 63 cost 532480 latency 0.001593 osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]
2014-06-12 14:46:33.714214 7f9d673df700  5 -- op tracker -- , seq: 747890, time: 2014-06-12 14:46:33.714209, event: reached_pg, op: osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.714225 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:33.714240 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] handle_message: 0x330c6c00
2014-06-12 14:46:33.714252 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_op osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:33.714277 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] get_object_context: found obc in cache: 0x104742c0
2014-06-12 14:46:33.714288 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] get_object_context: 0x104742c0 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 rwstate(write n=3 w=0) oi: 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3(173'2817 client.4293.0:144176 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2817) ssc: 0x1e15f260 snapset: 0=[]:[]+head
2014-06-12 14:46:33.714303 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] find_object_context 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 @head oi=67092105/rbd_data.109c6b8b4567.0000000000001058/head//3(173'2817 client.4293.0:144176 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2817)
2014-06-12 14:46:33.714333 7f9d673df700  5 -- op tracker -- , seq: 747890, time: 2014-06-12 14:46:33.714328, event: started, op: osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.714352 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] execute_ctx 0x2e413600
2014-06-12 14:46:33.714373 7f9d673df700  5 -- op tracker -- , seq: 747890, time: 2014-06-12 14:46:33.714368, event: started, op: osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.714381 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_op 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] ov 173'2817 av 173'2818 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:33.714396 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480]
2014-06-12 14:46:33.714407 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:33.714421 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op  write 3661824~532480
2014-06-12 14:46:33.714436 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] do_osd_op_effects on session 0x5c8cc00
2014-06-12 14:46:33.714448 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] make_writeable 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 snapset=0x1e15f298  snapc=0=[]
2014-06-12 14:46:33.714460 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] make_writeable 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:33.714470 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] finish_ctx 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 0x2e413600 op modify  
2014-06-12 14:46:33.714482 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]  set mtime to 2014-06-12 08:45:36.092126
2014-06-12 14:46:33.714502 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]  final snapset 0=[]:[]+head in 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.714520 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]  zeroing write result code 0
2014-06-12 14:46:33.714533 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] new_repop rep_tid 236280 on osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:33.714551 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] issue_repop rep_tid 236280 o 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.714577 7f9d673df700  5 -- op tracker -- , seq: 747890, time: 2014-06-12 14:46:33.714572, event: waiting for subops from 26, op: osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.714603 7f9d673df700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:33.714626 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2817 (0'0,173'2817] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] append_log log((0'0,173'2817], crt=0'0) [173'2818 (173'2817) modify   67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 by client.4293.0:144177 2014-06-12 08:45:36.092126]
2014-06-12 14:46:33.714652 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] add_log_entry 173'2818 (173'2817) modify   67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 by client.4293.0:144177 2014-06-12 08:45:36.092126
2014-06-12 14:46:33.714672 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] append_log  adding 1 keys
2014-06-12 14:46:33.714707 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2818, trimmed: 
2014-06-12 14:46:33.714728 7f9d673df700  5 queue_transactions existing osr(3.105 0x276cf80)/0x276cf80
2014-06-12 14:46:33.714733 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:33.714734 7f9d673df700  5 queue_op 0x70c5620 seq 2567 osr(3.105 0x276cf80) 534773 bytes   (queue has 1 ops and 534773 bytes)
2014-06-12 14:46:33.714746 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54758c0 (1 active)
2014-06-12 14:46:33.714754 7f9d8096e700  5 _do_op 0x70c5620 seq 2567 osr(3.105 0x276cf80)/0x276cf80 start
2014-06-12 14:46:33.714742 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] eval_repop repgather(0x2dc1ee40 173'2818 rep_tid=236280 committed?=0 applied?=0 lock=0 op=osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:33.714759 7f9d8096e700 10 _do_transaction on 0x3a097440
2014-06-12 14:46:33.714760 7f9d673df700 10 osd.2 173 dequeue_op 0x330c6c00 finish
2014-06-12 14:46:33.714764 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:33.714765 7f9d8096e700 15 _omap_setkeys meta/1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.714767 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:33.714895 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.714912 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:33.714966 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:33.714981 7f9d8096e700 15 _omap_rmkeys meta/1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.714987 7f9d8096e700 10 _omap_rmkeys meta/1fe2fcda/pglog_3.105/0//-1 r = 0
2014-06-12 14:46:33.714992 7f9d8096e700 15 _omap_setkeys meta/1fe2fcda/pglog_3.105/0//-1
2014-06-12 14:46:33.715004 7f9d8096e700 15 _write 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 3661824~532480
2014-06-12 14:46:33.715147 7f9d8096e700 10 lookup_strip_header done  cid 3.105_head oid 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.715281 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:33.717629 7f9d8096e700 10 _generic_write 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 3661824~532480 = 0
2014-06-12 14:46:33.717703 7f9d8096e700 15 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.717734 7f9d8096e700 10 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:46:33.717745 7f9d8096e700 15 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3
2014-06-12 14:46:33.717755 7f9d8096e700 10 _setattrs 3.105_head/67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:46:33.717982 7f9d8096e700 20 set_header setting 17927 cid 3.105_head oid 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 parent seq 0
2014-06-12 14:46:33.718049 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:33.718066 7f9d8096e700 20 set_header setting 1990 cid meta oid 1fe2fcda/pglog_3.105/0//-1 parent seq 0
2014-06-12 14:46:33.718078 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:33.720967 7f9d8096e700 10 _do_op 0x70c5620 seq 2567 r = 0, finisher 0x89f5260 0x7a77080
2014-06-12 14:46:33.720988 7f9d8096e700 10 _finish_op 0x70c5620 seq 2567 osr(3.105 0x276cf80)/0x276cf80
2014-06-12 14:46:33.720995 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54758c0 (0 active)
2014-06-12 14:46:33.721004 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:33.720994 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_commit: 236280
2014-06-12 14:46:33.721029 7f9d7715b700  5 -- op tracker -- , seq: 747890, time: 2014-06-12 14:46:33.721019, event: op_commit, op: osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.721044 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_applied: 236280
2014-06-12 14:46:33.721074 7f9d7795c700  5 -- op tracker -- , seq: 747890, time: 2014-06-12 14:46:33.721067, event: op_applied, op: osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:33.721091 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] op_applied on primary on version 173'2818
2014-06-12 14:46:33.841156 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:33.841237 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:33.897120 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:33.897201 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:34.154203 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:34.367866 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:34.477036 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:34.477100 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:34.477103 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 11.84 >= max 0.5 = no, load too high
2014-06-12 14:46:34.477117 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:34.477127 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92642 < max (604800 seconds)
2014-06-12 14:46:34.477146 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:34.477149 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:34.477150 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:34.477152 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 4; oldest is 1.315808 seconds old
2014-06-12 14:46:34.635135 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:34.689078 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:34.739819 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:34.760577 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:34.770475 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:34.779163 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:34.791770 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:34.798759 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:34.881834 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:34.886422 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:34.950888 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:34.950924 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:35.010742 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.040998 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.049611 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.050702 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.058911 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.094141 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.095578 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.097549 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.109712 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.134344 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.139393 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.139708 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.149887 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.163944 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.165824 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.165937 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.166842 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.172483 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.174065 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.174127 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.174739 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.180448 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.184844 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.207517 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.208820 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:35.209434 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:35.219447 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:35.219498 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:35.249244 7f9d5501e700  5 -- op tracker -- , seq: 747891, time: 2014-06-12 14:46:35.249167, event: header_read, op: osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.249257 7f9d5501e700  5 -- op tracker -- , seq: 747891, time: 2014-06-12 14:46:35.249170, event: throttled, op: osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.249265 7f9d5501e700  5 -- op tracker -- , seq: 747891, time: 2014-06-12 14:46:35.249209, event: all_read, op: osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.249272 7f9d5501e700  5 -- op tracker -- , seq: 747891, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.249277 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:35.249283 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:35.249291 7f9d5501e700 15 osd.2 173 enqueue_op 0x34017000 prio 127 cost 4903 latency 0.000123 osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.249319 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.249331 7f9d633d7700 10 osd.2 173 dequeue_op 0x34017000 prio 127 cost 4903 latency 0.000164 osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.591( v 173'3246 (125'245,173'3246] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3245 active]
2014-06-12 14:46:35.249360 7f9d633d7700  5 -- op tracker -- , seq: 747891, time: 2014-06-12 14:46:35.249355, event: reached_pg, op: osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.249373 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3246 (125'245,173'3246] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3245 active] handle_message: 0x34017000
2014-06-12 14:46:35.249387 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3246 (125'245,173'3246] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3245 active] sub_op_modify trans be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 v 173'3247 (transaction) 174
2014-06-12 14:46:35.249407 7f9d633d7700  5 -- op tracker -- , seq: 747891, time: 2014-06-12 14:46:35.249402, event: started, op: osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.249439 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3246 (125'245,173'3246] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3245 active] append_log log((125'245,173'3246], crt=0'0) [173'3247 (173'998) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145550 2014-06-12 08:45:37.618077]
2014-06-12 14:46:35.249490 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3247 (125'245,173'3247] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3245 active] add_log_entry 173'3247 (173'998) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145550 2014-06-12 08:45:37.618077
2014-06-12 14:46:35.249543 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3247 (125'245,173'3247] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3245 active] append_log  adding 1 keys
2014-06-12 14:46:35.249573 7f9d633d7700 10 trim log((125'245,173'3247], crt=0'0) to 125'246
2014-06-12 14:46:35.249644 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3247, trimmed: 125'246
2014-06-12 14:46:35.249704 7f9d633d7700  5 -- op tracker -- , seq: 747891, time: 2014-06-12 14:46:35.249692, event: started, op: osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.249728 7f9d633d7700  5 queue_transactions existing osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.249742 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.249743 7f9d633d7700  5 queue_op 0x18550d20 seq 2899 osr(3.591 0x276da10) 6424 bytes   (queue has 1 ops and 6424 bytes)
2014-06-12 14:46:35.249749 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.591( v 173'3247 (125'246,173'3247] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3245 active] do_sub_op osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.249763 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438120 (1 active)
2014-06-12 14:46:35.249766 7f9d633d7700 10 osd.2 173 dequeue_op 0x34017000 finish
2014-06-12 14:46:35.249770 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.249770 7f9d8116f700  5 _do_op 0x18550d20 seq 2899 osr(3.591 0x276da10)/0x276da10 start
2014-06-12 14:46:35.249772 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.249773 7f9d8116f700 10 _do_transaction on 0x1451db30
2014-06-12 14:46:35.249778 7f9d8116f700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.249894 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.249911 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.249954 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.249973 7f9d8116f700 15 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.249980 7f9d8116f700 10 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1 r = 0
2014-06-12 14:46:35.249984 7f9d8116f700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.249997 7f9d8116f700 15 _write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 0~4096
2014-06-12 14:46:35.250233 7f9d8116f700 10 lookup_strip_header done  cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.250242 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:35.250257 7f9d8116f700 10 _generic_write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 0~4096 = 0
2014-06-12 14:46:35.250263 7f9d8116f700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.250272 7f9d8116f700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.250275 7f9d8116f700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.250280 7f9d8116f700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.250362 7f9d8116f700 20 set_header setting 17486 cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 parent seq 0
2014-06-12 14:46:35.250380 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.250389 7f9d8116f700 20 set_header setting 2292 cid meta oid 1fe10eca/pglog_3.591/0//-1 parent seq 0
2014-06-12 14:46:35.250397 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:35.250527 7f9d8116f700 10 _do_op 0x18550d20 seq 2899 r = 0, finisher 0x31d0efc0 0
2014-06-12 14:46:35.250534 7f9d8116f700 10 _finish_op 0x18550d20 seq 2899 osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.250538 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438120 (0 active)
2014-06-12 14:46:35.250541 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.250554 7f9d7715b700  5 -- op tracker -- , seq: 747891, time: 2014-06-12 14:46:35.250547, event: commit_sent, op: osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.250564 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3247 (125'246,173'3247] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3245 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:35.250583 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:35.250615 7f9d7795c700  5 -- op tracker -- , seq: 747891, time: 2014-06-12 14:46:35.250609, event: sub_op_applied, op: osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.250626 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3247 (125'246,173'3247] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3246 active] sub_op_modify_applied on 0x1451da40 op osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.250639 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3247 (125'246,173'3247] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3246 active] op_applied on primary on version 173'3247
2014-06-12 14:46:35.250646 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3247 (125'246,173'3247] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3246 active] op_applied on replica on version 173'3247
2014-06-12 14:46:35.250664 7f9d7795c700  5 -- op tracker -- , seq: 747891, time: 2014-06-12 14:46:35.250662, event: done, op: osd_sub_op(client.4298.0:145550 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3247 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.268821 7f9d5501e700  5 -- op tracker -- , seq: 747892, time: 2014-06-12 14:46:35.266142, event: header_read, op: osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.268853 7f9d5501e700  5 -- op tracker -- , seq: 747892, time: 2014-06-12 14:46:35.266144, event: throttled, op: osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.268863 7f9d5501e700  5 -- op tracker -- , seq: 747892, time: 2014-06-12 14:46:35.268766, event: all_read, op: osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.268884 7f9d5501e700  5 -- op tracker -- , seq: 747892, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.268890 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:35.268903 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:35.268912 7f9d5501e700 15 osd.2 173 enqueue_op 0x34016400 prio 127 cost 1032999 latency 0.002770 osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.268945 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.268958 7f9d61bd4700 10 osd.2 173 dequeue_op 0x34016400 prio 127 cost 1032999 latency 0.002816 osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.591( v 173'3247 (125'246,173'3247] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3246 active]
2014-06-12 14:46:35.268986 7f9d61bd4700  5 -- op tracker -- , seq: 747892, time: 2014-06-12 14:46:35.268981, event: reached_pg, op: osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.268996 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3247 (125'246,173'3247] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3246 active] handle_message: 0x34016400
2014-06-12 14:46:35.269011 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3247 (125'246,173'3247] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3246 active] sub_op_modify trans be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 v 173'3248 (transaction) 174
2014-06-12 14:46:35.269033 7f9d61bd4700  5 -- op tracker -- , seq: 747892, time: 2014-06-12 14:46:35.269029, event: started, op: osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.269056 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3247 (125'246,173'3247] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3246 active] append_log log((125'246,173'3247], crt=0'0) [173'3248 (173'3247) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145551 2014-06-12 08:45:37.640418]
2014-06-12 14:46:35.269083 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3248 (125'246,173'3248] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3246 active] add_log_entry 173'3248 (173'3247) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145551 2014-06-12 08:45:37.640418
2014-06-12 14:46:35.269107 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3248 (125'246,173'3248] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3246 active] append_log  adding 1 keys
2014-06-12 14:46:35.269123 7f9d61bd4700 10 trim log((125'246,173'3248], crt=0'0) to 125'247
2014-06-12 14:46:35.269151 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3248, trimmed: 125'247
2014-06-12 14:46:35.269178 7f9d61bd4700  5 -- op tracker -- , seq: 747892, time: 2014-06-12 14:46:35.269174, event: started, op: osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.269189 7f9d61bd4700  5 queue_transactions existing osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.269194 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.269196 7f9d61bd4700  5 queue_op 0x54cc2a0 seq 2900 osr(3.591 0x276da10) 1034520 bytes   (queue has 1 ops and 1034520 bytes)
2014-06-12 14:46:35.269211 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438120 (1 active)
2014-06-12 14:46:35.269204 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.591( v 173'3248 (125'247,173'3248] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3246 active] do_sub_op osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.269219 7f9d79960700  5 _do_op 0x54cc2a0 seq 2900 osr(3.591 0x276da10)/0x276da10 start
2014-06-12 14:46:35.269220 7f9d61bd4700 10 osd.2 173 dequeue_op 0x34016400 finish
2014-06-12 14:46:35.269224 7f9d79960700 10 _do_transaction on 0x62cfeb0
2014-06-12 14:46:35.269224 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.269227 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.269230 7f9d79960700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.269366 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.269391 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.269466 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.269494 7f9d79960700 15 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.269503 7f9d79960700 10 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1 r = 0
2014-06-12 14:46:35.269509 7f9d79960700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.269738 7f9d79960700 15 _write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 4096~1032192
2014-06-12 14:46:35.269844 7f9d79960700 10 lookup_strip_header done  cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.270009 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:35.273002 7f9d79960700 10 _generic_write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 4096~1032192 = 0
2014-06-12 14:46:35.273071 7f9d79960700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.273084 7f9d79960700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.273089 7f9d79960700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.273094 7f9d79960700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.273289 7f9d79960700 20 set_header setting 17486 cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 parent seq 0
2014-06-12 14:46:35.273363 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.273374 7f9d79960700 20 set_header setting 2292 cid meta oid 1fe10eca/pglog_3.591/0//-1 parent seq 0
2014-06-12 14:46:35.273381 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:35.277097 7f9d79960700 10 _do_op 0x54cc2a0 seq 2900 r = 0, finisher 0x80bf5a0 0
2014-06-12 14:46:35.277118 7f9d79960700 10 _finish_op 0x54cc2a0 seq 2900 osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.277124 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438120 (0 active)
2014-06-12 14:46:35.277127 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.277138 7f9d7715b700  5 -- op tracker -- , seq: 747892, time: 2014-06-12 14:46:35.277128, event: commit_sent, op: osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.277172 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3248 (125'247,173'3248] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3246 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:35.277199 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:35.277233 7f9d7795c700  5 -- op tracker -- , seq: 747892, time: 2014-06-12 14:46:35.277226, event: sub_op_applied, op: osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.277242 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3248 (125'247,173'3248] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3247 active] sub_op_modify_applied on 0x62cfdc0 op osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.277259 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3248 (125'247,173'3248] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3247 active] op_applied on primary on version 173'3248
2014-06-12 14:46:35.277269 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3248 (125'247,173'3248] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3247 active] op_applied on replica on version 173'3248
2014-06-12 14:46:35.277293 7f9d7795c700  5 -- op tracker -- , seq: 747892, time: 2014-06-12 14:46:35.277288, event: done, op: osd_sub_op(client.4298.0:145551 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3248 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.287503 7f9d5501e700  5 -- op tracker -- , seq: 747893, time: 2014-06-12 14:46:35.284792, event: header_read, op: osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.287514 7f9d5501e700  5 -- op tracker -- , seq: 747893, time: 2014-06-12 14:46:35.284793, event: throttled, op: osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.287530 7f9d5501e700  5 -- op tracker -- , seq: 747893, time: 2014-06-12 14:46:35.287462, event: all_read, op: osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.287535 7f9d5501e700  5 -- op tracker -- , seq: 747893, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.287541 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:35.287547 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:35.287553 7f9d5501e700 15 osd.2 173 enqueue_op 0x2b014500 prio 127 cost 1032999 latency 0.002761 osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.287576 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.287584 7f9d673df700 10 osd.2 173 dequeue_op 0x2b014500 prio 127 cost 1032999 latency 0.002792 osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.591( v 173'3248 (125'247,173'3248] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3247 active]
2014-06-12 14:46:35.287613 7f9d673df700  5 -- op tracker -- , seq: 747893, time: 2014-06-12 14:46:35.287609, event: reached_pg, op: osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.287623 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3248 (125'247,173'3248] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3247 active] handle_message: 0x2b014500
2014-06-12 14:46:35.287637 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3248 (125'247,173'3248] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3247 active] sub_op_modify trans be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 v 173'3249 (transaction) 174
2014-06-12 14:46:35.287653 7f9d673df700  5 -- op tracker -- , seq: 747893, time: 2014-06-12 14:46:35.287649, event: started, op: osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.287668 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3248 (125'247,173'3248] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3247 active] append_log log((125'247,173'3248], crt=0'0) [173'3249 (173'3248) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145552 2014-06-12 08:45:37.659914]
2014-06-12 14:46:35.287691 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3249 (125'247,173'3249] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3247 active] add_log_entry 173'3249 (173'3248) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145552 2014-06-12 08:45:37.659914
2014-06-12 14:46:35.287714 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3249 (125'247,173'3249] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3247 active] append_log  adding 1 keys
2014-06-12 14:46:35.287729 7f9d673df700 10 trim log((125'247,173'3249], crt=0'0) to 125'248
2014-06-12 14:46:35.287754 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3249, trimmed: 125'248
2014-06-12 14:46:35.287778 7f9d673df700  5 -- op tracker -- , seq: 747893, time: 2014-06-12 14:46:35.287773, event: started, op: osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.287788 7f9d673df700  5 queue_transactions existing osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.287792 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.287794 7f9d673df700  5 queue_op 0x9319920 seq 2901 osr(3.591 0x276da10) 1034520 bytes   (queue has 1 ops and 1034520 bytes)
2014-06-12 14:46:35.287802 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438120 (1 active)
2014-06-12 14:46:35.287809 7f9d8096e700  5 _do_op 0x9319920 seq 2901 osr(3.591 0x276da10)/0x276da10 start
2014-06-12 14:46:35.287812 7f9d8096e700 10 _do_transaction on 0x62ebcf0
2014-06-12 14:46:35.287800 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.591( v 173'3249 (125'248,173'3249] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3247 active] do_sub_op osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.287814 7f9d673df700 10 osd.2 173 dequeue_op 0x2b014500 finish
2014-06-12 14:46:35.287816 7f9d8096e700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.287818 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.287820 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.287917 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.287933 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.287967 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.287985 7f9d8096e700 15 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.287991 7f9d8096e700 10 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1 r = 0
2014-06-12 14:46:35.287995 7f9d8096e700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.288095 7f9d8096e700 15 _write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 1036288~1032192
2014-06-12 14:46:35.288197 7f9d8096e700 10 lookup_strip_header done  cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.288339 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:35.291474 7f9d8096e700 10 _generic_write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 1036288~1032192 = 0
2014-06-12 14:46:35.291559 7f9d8096e700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.291573 7f9d8096e700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.291576 7f9d8096e700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.291581 7f9d8096e700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.291813 7f9d8096e700 20 set_header setting 17486 cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 parent seq 0
2014-06-12 14:46:35.291914 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.291926 7f9d8096e700 20 set_header setting 2292 cid meta oid 1fe10eca/pglog_3.591/0//-1 parent seq 0
2014-06-12 14:46:35.291940 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:35.297120 7f9d8096e700 10 _do_op 0x9319920 seq 2901 r = 0, finisher 0x89f41c0 0
2014-06-12 14:46:35.297140 7f9d8096e700 10 _finish_op 0x9319920 seq 2901 osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.297147 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438120 (0 active)
2014-06-12 14:46:35.297156 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.297160 7f9d7715b700  5 -- op tracker -- , seq: 747893, time: 2014-06-12 14:46:35.297149, event: commit_sent, op: osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.297169 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3249 (125'248,173'3249] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3247 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:35.297193 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:35.297228 7f9d7795c700  5 -- op tracker -- , seq: 747893, time: 2014-06-12 14:46:35.297220, event: sub_op_applied, op: osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.297241 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3249 (125'248,173'3249] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3248 active] sub_op_modify_applied on 0x62ebc00 op osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.297259 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3249 (125'248,173'3249] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3248 active] op_applied on primary on version 173'3249
2014-06-12 14:46:35.297270 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3249 (125'248,173'3249] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3248 active] op_applied on replica on version 173'3249
2014-06-12 14:46:35.297299 7f9d7795c700  5 -- op tracker -- , seq: 747893, time: 2014-06-12 14:46:35.297294, event: done, op: osd_sub_op(client.4298.0:145552 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3249 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.310830 7f9d5501e700  5 -- op tracker -- , seq: 747894, time: 2014-06-12 14:46:35.308332, event: header_read, op: osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.310841 7f9d5501e700  5 -- op tracker -- , seq: 747894, time: 2014-06-12 14:46:35.308333, event: throttled, op: osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.310854 7f9d5501e700  5 -- op tracker -- , seq: 747894, time: 2014-06-12 14:46:35.310795, event: all_read, op: osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.310863 7f9d5501e700  5 -- op tracker -- , seq: 747894, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.310867 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:35.310872 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:35.310878 7f9d5501e700 15 osd.2 173 enqueue_op 0x2b014000 prio 127 cost 1032999 latency 0.002546 osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.310905 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.310914 7f9d67be0700 10 osd.2 173 dequeue_op 0x2b014000 prio 127 cost 1032999 latency 0.002582 osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.591( v 173'3249 (125'248,173'3249] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3248 active]
2014-06-12 14:46:35.310940 7f9d67be0700  5 -- op tracker -- , seq: 747894, time: 2014-06-12 14:46:35.310935, event: reached_pg, op: osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.310949 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3249 (125'248,173'3249] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3248 active] handle_message: 0x2b014000
2014-06-12 14:46:35.310961 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3249 (125'248,173'3249] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3248 active] sub_op_modify trans be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 v 173'3250 (transaction) 174
2014-06-12 14:46:35.310976 7f9d67be0700  5 -- op tracker -- , seq: 747894, time: 2014-06-12 14:46:35.310973, event: started, op: osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.310993 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3249 (125'248,173'3249] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3248 active] append_log log((125'248,173'3249], crt=0'0) [173'3250 (173'3249) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145553 2014-06-12 08:45:37.679856]
2014-06-12 14:46:35.311016 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3250 (125'248,173'3250] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3248 active] add_log_entry 173'3250 (173'3249) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145553 2014-06-12 08:45:37.679856
2014-06-12 14:46:35.311037 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3250 (125'248,173'3250] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3248 active] append_log  adding 1 keys
2014-06-12 14:46:35.311051 7f9d67be0700 10 trim log((125'248,173'3250], crt=0'0) to 125'249
2014-06-12 14:46:35.311078 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3250, trimmed: 125'249
2014-06-12 14:46:35.311104 7f9d67be0700  5 -- op tracker -- , seq: 747894, time: 2014-06-12 14:46:35.311099, event: started, op: osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.311114 7f9d67be0700  5 queue_transactions existing osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.311118 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.311119 7f9d67be0700  5 queue_op 0x18586480 seq 2902 osr(3.591 0x276da10) 1034520 bytes   (queue has 1 ops and 1034520 bytes)
2014-06-12 14:46:35.311129 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438120 (1 active)
2014-06-12 14:46:35.311134 7f9d8016d700  5 _do_op 0x18586480 seq 2902 osr(3.591 0x276da10)/0x276da10 start
2014-06-12 14:46:35.311138 7f9d8016d700 10 _do_transaction on 0x668d7b0
2014-06-12 14:46:35.311127 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.591( v 173'3250 (125'249,173'3250] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3248 active] do_sub_op osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.311141 7f9d67be0700 10 osd.2 173 dequeue_op 0x2b014000 finish
2014-06-12 14:46:35.311143 7f9d8016d700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.311145 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.311147 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.311341 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.311354 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.311392 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.311410 7f9d8016d700 15 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.311416 7f9d8016d700 10 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1 r = 0
2014-06-12 14:46:35.311420 7f9d8016d700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.311429 7f9d8016d700 15 _write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 2068480~1032192
2014-06-12 14:46:35.311524 7f9d8016d700 10 lookup_strip_header done  cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.311648 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:35.314409 7f9d8016d700 10 _generic_write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 2068480~1032192 = 0
2014-06-12 14:46:35.314495 7f9d8016d700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.314512 7f9d8016d700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.314517 7f9d8016d700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.314522 7f9d8016d700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.314717 7f9d8016d700 20 set_header setting 17486 cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 parent seq 0
2014-06-12 14:46:35.314802 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.314815 7f9d8016d700 20 set_header setting 2292 cid meta oid 1fe10eca/pglog_3.591/0//-1 parent seq 0
2014-06-12 14:46:35.314821 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:35.319473 7f9d8016d700 10 _do_op 0x18586480 seq 2902 r = 0, finisher 0x4f61ca0 0
2014-06-12 14:46:35.319495 7f9d8016d700 10 _finish_op 0x18586480 seq 2902 osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.319503 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438120 (0 active)
2014-06-12 14:46:35.319512 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.319513 7f9d7715b700  5 -- op tracker -- , seq: 747894, time: 2014-06-12 14:46:35.319504, event: commit_sent, op: osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.319526 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3250 (125'249,173'3250] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3248 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:35.319550 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:35.319580 7f9d7795c700  5 -- op tracker -- , seq: 747894, time: 2014-06-12 14:46:35.319574, event: sub_op_applied, op: osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.319588 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3250 (125'249,173'3250] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] sub_op_modify_applied on 0x668d6c0 op osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.319600 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3250 (125'249,173'3250] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] op_applied on primary on version 173'3250
2014-06-12 14:46:35.319608 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3250 (125'249,173'3250] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] op_applied on replica on version 173'3250
2014-06-12 14:46:35.319625 7f9d7795c700  5 -- op tracker -- , seq: 747894, time: 2014-06-12 14:46:35.319622, event: done, op: osd_sub_op(client.4298.0:145553 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3250 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.331243 7f9d5501e700  5 -- op tracker -- , seq: 747895, time: 2014-06-12 14:46:35.327237, event: header_read, op: osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.331263 7f9d5501e700  5 -- op tracker -- , seq: 747895, time: 2014-06-12 14:46:35.327242, event: throttled, op: osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.331300 7f9d5501e700  5 -- op tracker -- , seq: 747895, time: 2014-06-12 14:46:35.331188, event: all_read, op: osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.331310 7f9d5501e700  5 -- op tracker -- , seq: 747895, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.331317 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:35.331328 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:35.331338 7f9d5501e700 15 osd.2 173 enqueue_op 0x9fad700 prio 127 cost 516903 latency 0.004100 osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.331372 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.331381 7f9d663dd700 10 osd.2 173 dequeue_op 0x9fad700 prio 127 cost 516903 latency 0.004144 osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.591( v 173'3250 (125'249,173'3250] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active]
2014-06-12 14:46:35.331421 7f9d663dd700  5 -- op tracker -- , seq: 747895, time: 2014-06-12 14:46:35.331417, event: reached_pg, op: osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.331430 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3250 (125'249,173'3250] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] handle_message: 0x9fad700
2014-06-12 14:46:35.331444 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3250 (125'249,173'3250] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] sub_op_modify trans be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 v 173'3251 (transaction) 174
2014-06-12 14:46:35.331463 7f9d663dd700  5 -- op tracker -- , seq: 747895, time: 2014-06-12 14:46:35.331459, event: started, op: osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.331479 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3250 (125'249,173'3250] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] append_log log((125'249,173'3250], crt=0'0) [173'3251 (173'3250) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145554 2014-06-12 08:45:37.704310]
2014-06-12 14:46:35.331524 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3251 (125'249,173'3251] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] add_log_entry 173'3251 (173'3250) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145554 2014-06-12 08:45:37.704310
2014-06-12 14:46:35.331550 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3251 (125'249,173'3251] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] append_log  adding 1 keys
2014-06-12 14:46:35.331564 7f9d663dd700 10 trim log((125'249,173'3251], crt=0'0) to 125'250
2014-06-12 14:46:35.331595 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3251, trimmed: 125'250
2014-06-12 14:46:35.331627 7f9d663dd700  5 -- op tracker -- , seq: 747895, time: 2014-06-12 14:46:35.331619, event: started, op: osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.331639 7f9d663dd700  5 queue_transactions existing osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.331644 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.331645 7f9d663dd700  5 queue_op 0x72c5680 seq 2903 osr(3.591 0x276da10) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:35.331659 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438120 (1 active)
2014-06-12 14:46:35.331653 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.591( v 173'3251 (125'250,173'3251] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] do_sub_op osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.331668 7f9d663dd700 10 osd.2 173 dequeue_op 0x9fad700 finish
2014-06-12 14:46:35.331667 7f9d7915f700  5 _do_op 0x72c5680 seq 2903 osr(3.591 0x276da10)/0x276da10 start
2014-06-12 14:46:35.331671 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.331673 7f9d7915f700 10 _do_transaction on 0x8222470
2014-06-12 14:46:35.331674 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.331680 7f9d7915f700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.331825 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.331857 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.331904 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.331925 7f9d7915f700 15 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.331934 7f9d7915f700 10 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1 r = 0
2014-06-12 14:46:35.331941 7f9d7915f700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.332081 7f9d7915f700 15 _write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 3100672~516096
2014-06-12 14:46:35.332190 7f9d7915f700 10 lookup_strip_header done  cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.332327 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:35.332419 7f9d5501e700  5 -- op tracker -- , seq: 747896, time: 2014-06-12 14:46:35.331690, event: header_read, op: osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.332432 7f9d5501e700  5 -- op tracker -- , seq: 747896, time: 2014-06-12 14:46:35.331691, event: throttled, op: osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.332440 7f9d5501e700  5 -- op tracker -- , seq: 747896, time: 2014-06-12 14:46:35.332392, event: all_read, op: osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.332448 7f9d5501e700  5 -- op tracker -- , seq: 747896, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.332452 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:35.332458 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:35.332464 7f9d5501e700 15 osd.2 173 enqueue_op 0x9fade00 prio 127 cost 516903 latency 0.000773 osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.332488 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.332494 7f9d693e3700 10 osd.2 173 dequeue_op 0x9fade00 prio 127 cost 516903 latency 0.000804 osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.591( v 173'3251 (125'250,173'3251] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active]
2014-06-12 14:46:35.332515 7f9d693e3700  5 -- op tracker -- , seq: 747896, time: 2014-06-12 14:46:35.332511, event: reached_pg, op: osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.332524 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3251 (125'250,173'3251] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] handle_message: 0x9fade00
2014-06-12 14:46:35.332536 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3251 (125'250,173'3251] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] sub_op_modify trans be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 v 173'3252 (transaction) 174
2014-06-12 14:46:35.332553 7f9d693e3700  5 -- op tracker -- , seq: 747896, time: 2014-06-12 14:46:35.332549, event: started, op: osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.332565 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3251 (125'250,173'3251] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] append_log log((125'250,173'3251], crt=0'0) [173'3252 (173'3251) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145555 2014-06-12 08:45:37.704911]
2014-06-12 14:46:35.332586 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'250,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] add_log_entry 173'3252 (173'3251) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145555 2014-06-12 08:45:37.704911
2014-06-12 14:46:35.332607 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'250,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] append_log  adding 1 keys
2014-06-12 14:46:35.332621 7f9d693e3700 10 trim log((125'250,173'3252], crt=0'0) to 125'251
2014-06-12 14:46:35.332643 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3252, trimmed: 125'251
2014-06-12 14:46:35.332667 7f9d693e3700  5 -- op tracker -- , seq: 747896, time: 2014-06-12 14:46:35.332662, event: started, op: osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.332679 7f9d693e3700  5 queue_transactions existing osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.332682 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.332683 7f9d693e3700  5 queue_op 0x5e6c120 seq 2904 osr(3.591 0x276da10) 518424 bytes   (queue has 2 ops and 1036848 bytes)
2014-06-12 14:46:35.332694 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438120 (2 active)
2014-06-12 14:46:35.332689 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'251,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] do_sub_op osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.332703 7f9d693e3700 10 osd.2 173 dequeue_op 0x9fade00 finish
2014-06-12 14:46:35.332707 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.332709 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.333767 7f9d7915f700 10 _generic_write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 3100672~516096 = 0
2014-06-12 14:46:35.333812 7f9d7915f700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.333828 7f9d7915f700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.333836 7f9d7915f700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.333846 7f9d7915f700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.333992 7f9d7915f700 20 set_header setting 17486 cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 parent seq 0
2014-06-12 14:46:35.334054 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.334070 7f9d7915f700 20 set_header setting 2292 cid meta oid 1fe10eca/pglog_3.591/0//-1 parent seq 0
2014-06-12 14:46:35.334081 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:35.335752 7f9d7915f700 10 _do_op 0x72c5680 seq 2903 r = 0, finisher 0x567e180 0
2014-06-12 14:46:35.335767 7f9d7915f700 10 _finish_op 0x72c5680 seq 2903 osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.335777 7f9d7d968700  5 _do_op 0x5e6c120 seq 2904 osr(3.591 0x276da10)/0x276da10 start
2014-06-12 14:46:35.335777 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438120 (1 active)
2014-06-12 14:46:35.335784 7f9d7d968700 10 _do_transaction on 0x1ebed970
2014-06-12 14:46:35.335785 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.335781 7f9d7715b700  5 -- op tracker -- , seq: 747895, time: 2014-06-12 14:46:35.335773, event: commit_sent, op: osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.335791 7f9d7d968700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.335790 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'251,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3249 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:35.335808 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:35.335850 7f9d7795c700  5 -- op tracker -- , seq: 747895, time: 2014-06-12 14:46:35.335845, event: sub_op_applied, op: osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.335858 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'251,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3250 active] sub_op_modify_applied on 0x8222380 op osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.335870 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'251,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3250 active] op_applied on primary on version 173'3251
2014-06-12 14:46:35.335876 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'251,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3250 active] op_applied on replica on version 173'3251
2014-06-12 14:46:35.335890 7f9d7795c700  5 -- op tracker -- , seq: 747895, time: 2014-06-12 14:46:35.335887, event: done, op: osd_sub_op(client.4298.0:145554 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3251 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.335914 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.335939 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.335992 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.336016 7f9d7d968700 15 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.336026 7f9d7d968700 10 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1 r = 0
2014-06-12 14:46:35.336035 7f9d7d968700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.336052 7f9d7d968700 15 _write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 3620864~516096
2014-06-12 14:46:35.336302 7f9d7d968700 10 lookup_strip_header done  cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.336431 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:35.338057 7f9d7d968700 10 _generic_write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 3620864~516096 = 0
2014-06-12 14:46:35.338096 7f9d7d968700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.338111 7f9d7d968700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.338119 7f9d7d968700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.338128 7f9d7d968700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.338310 7f9d7d968700 20 set_header setting 17486 cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 parent seq 0
2014-06-12 14:46:35.338372 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.338388 7f9d7d968700 20 set_header setting 2292 cid meta oid 1fe10eca/pglog_3.591/0//-1 parent seq 0
2014-06-12 14:46:35.338400 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:35.340934 7f9d7d968700 10 _do_op 0x5e6c120 seq 2904 r = 0, finisher 0x6d9d180 0
2014-06-12 14:46:35.340953 7f9d7d968700 10 _finish_op 0x5e6c120 seq 2904 osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.340962 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438120 (0 active)
2014-06-12 14:46:35.340975 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.340970 7f9d7715b700  5 -- op tracker -- , seq: 747896, time: 2014-06-12 14:46:35.340960, event: commit_sent, op: osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.340979 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'251,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3250 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:35.341007 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:35.341045 7f9d7795c700  5 -- op tracker -- , seq: 747896, time: 2014-06-12 14:46:35.341038, event: sub_op_applied, op: osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.341058 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'251,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active] sub_op_modify_applied on 0x1ebed880 op osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.341078 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'251,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active] op_applied on primary on version 173'3252
2014-06-12 14:46:35.341085 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'251,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active] op_applied on replica on version 173'3252
2014-06-12 14:46:35.341108 7f9d7795c700  5 -- op tracker -- , seq: 747896, time: 2014-06-12 14:46:35.341104, event: done, op: osd_sub_op(client.4298.0:145555 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3252 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.347227 7f9d5501e700  5 -- op tracker -- , seq: 747897, time: 2014-06-12 14:46:35.347080, event: header_read, op: osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.347244 7f9d5501e700  5 -- op tracker -- , seq: 747897, time: 2014-06-12 14:46:35.347082, event: throttled, op: osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.347254 7f9d5501e700  5 -- op tracker -- , seq: 747897, time: 2014-06-12 14:46:35.347183, event: all_read, op: osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.347263 7f9d5501e700  5 -- op tracker -- , seq: 747897, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.347268 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:35.347275 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:35.347284 7f9d5501e700 15 osd.2 173 enqueue_op 0x9fad900 prio 127 cost 58151 latency 0.000203 osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.347309 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.347317 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x9fad900 prio 127 cost 58151 latency 0.000237 osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.591( v 173'3252 (125'251,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active]
2014-06-12 14:46:35.347337 7f9d6b3e7700  5 -- op tracker -- , seq: 747897, time: 2014-06-12 14:46:35.347334, event: reached_pg, op: osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.347345 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'251,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active] handle_message: 0x9fad900
2014-06-12 14:46:35.347353 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'251,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active] sub_op_modify trans be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 v 173'3253 (transaction) 174
2014-06-12 14:46:35.347363 7f9d6b3e7700  5 -- op tracker -- , seq: 747897, time: 2014-06-12 14:46:35.347360, event: started, op: osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.347374 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3252 (125'251,173'3252] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active] append_log log((125'251,173'3252], crt=0'0) [173'3253 (173'3252) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145556 2014-06-12 08:45:37.726558]
2014-06-12 14:46:35.347391 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3253 (125'251,173'3253] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active] add_log_entry 173'3253 (173'3252) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145556 2014-06-12 08:45:37.726558
2014-06-12 14:46:35.347408 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3253 (125'251,173'3253] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active] append_log  adding 1 keys
2014-06-12 14:46:35.347419 7f9d6b3e7700 10 trim log((125'251,173'3253], crt=0'0) to 125'252
2014-06-12 14:46:35.347448 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3253, trimmed: 125'252
2014-06-12 14:46:35.347475 7f9d6b3e7700  5 -- op tracker -- , seq: 747897, time: 2014-06-12 14:46:35.347471, event: started, op: osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.347516 7f9d6b3e7700  5 queue_transactions existing osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.347523 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.347525 7f9d6b3e7700  5 queue_op 0x4ecd860 seq 2905 osr(3.591 0x276da10) 59672 bytes   (queue has 1 ops and 59672 bytes)
2014-06-12 14:46:35.347532 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.591( v 173'3253 (125'252,173'3253] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active] do_sub_op osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.347546 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x9fad900 finish
2014-06-12 14:46:35.347550 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.347552 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.347560 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438120 (1 active)
2014-06-12 14:46:35.347566 7f9d7c165700  5 _do_op 0x4ecd860 seq 2905 osr(3.591 0x276da10)/0x276da10 start
2014-06-12 14:46:35.347570 7f9d7c165700 10 _do_transaction on 0x5ea7970
2014-06-12 14:46:35.347575 7f9d7c165700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.347696 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.347714 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.347752 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.347772 7f9d7c165700 15 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.347778 7f9d7c165700 10 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1 r = 0
2014-06-12 14:46:35.347782 7f9d7c165700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:35.347793 7f9d7c165700 15 _write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 4136960~57344
2014-06-12 14:46:35.348019 7f9d7c165700 10 lookup_strip_header done  cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.348039 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:35.348175 7f9d7c165700 10 _generic_write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 4136960~57344 = 0
2014-06-12 14:46:35.348187 7f9d7c165700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.348193 7f9d7c165700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.348197 7f9d7c165700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:35.348201 7f9d7c165700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:35.348279 7f9d7c165700 20 set_header setting 17486 cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 parent seq 0
2014-06-12 14:46:35.348377 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.348391 7f9d7c165700 20 set_header setting 2292 cid meta oid 1fe10eca/pglog_3.591/0//-1 parent seq 0
2014-06-12 14:46:35.348403 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:35.348680 7f9d7c165700 10 _do_op 0x4ecd860 seq 2905 r = 0, finisher 0x7bd46e0 0
2014-06-12 14:46:35.348691 7f9d7c165700 10 _finish_op 0x4ecd860 seq 2905 osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:35.348698 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438120 (0 active)
2014-06-12 14:46:35.348701 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.348716 7f9d7795c700  5 -- op tracker -- , seq: 747897, time: 2014-06-12 14:46:35.348708, event: sub_op_applied, op: osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.348727 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3253 (125'252,173'3253] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active] sub_op_modify_applied on 0x5ea7880 op osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.348752 7f9d7795c700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:35.348788 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3253 (125'252,173'3253] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active] op_applied on primary on version 173'3253
2014-06-12 14:46:35.348800 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3253 (125'252,173'3253] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active] op_applied on replica on version 173'3253
2014-06-12 14:46:35.348829 7f9d7715b700  5 -- op tracker -- , seq: 747897, time: 2014-06-12 14:46:35.348822, event: commit_sent, op: osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.348845 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3253 (125'252,173'3253] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3251 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:35.348866 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:35.348905 7f9d7715b700  5 -- op tracker -- , seq: 747897, time: 2014-06-12 14:46:35.348899, event: done, op: osd_sub_op(client.4298.0:145556 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3253 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.421100 7f9d5a16f700  5 -- op tracker -- , seq: 747898, time: 2014-06-12 14:46:35.419968, event: header_read, op: osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.421118 7f9d5a16f700  5 -- op tracker -- , seq: 747898, time: 2014-06-12 14:46:35.419971, event: throttled, op: osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.421127 7f9d5a16f700  5 -- op tracker -- , seq: 747898, time: 2014-06-12 14:46:35.421042, event: all_read, op: osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.421136 7f9d5a16f700  5 -- op tracker -- , seq: 747898, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.421142 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:35.421150 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:35.421159 7f9d5a16f700 15 osd.2 173 enqueue_op 0x55c2000 prio 127 cost 582439 latency 0.001190 osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.421191 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.421201 7f9d683e1700 10 osd.2 173 dequeue_op 0x55c2000 prio 127 cost 582439 latency 0.001233 osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2666 (0'0,173'2666] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2665 active]
2014-06-12 14:46:35.421230 7f9d683e1700  5 -- op tracker -- , seq: 747898, time: 2014-06-12 14:46:35.421225, event: reached_pg, op: osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.421240 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2666 (0'0,173'2666] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2665 active] handle_message: 0x55c2000
2014-06-12 14:46:35.421254 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2666 (0'0,173'2666] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2665 active] sub_op_modify trans 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 v 173'2667 (transaction) 174
2014-06-12 14:46:35.421276 7f9d683e1700  5 -- op tracker -- , seq: 747898, time: 2014-06-12 14:46:35.421272, event: started, op: osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.421297 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2666 (0'0,173'2666] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2665 active] append_log log((0'0,173'2666], crt=0'0) [173'2667 (173'2665) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141068 2014-06-12 08:43:34.251520]
2014-06-12 14:46:35.421321 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2667 (0'0,173'2667] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2665 active] add_log_entry 173'2667 (173'2665) modify   582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 by client.4284.0:141068 2014-06-12 08:43:34.251520
2014-06-12 14:46:35.421345 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2667 (0'0,173'2667] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2665 active] append_log  adding 1 keys
2014-06-12 14:46:35.421379 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2667, trimmed: 
2014-06-12 14:46:35.421405 7f9d683e1700  5 -- op tracker -- , seq: 747898, time: 2014-06-12 14:46:35.421401, event: started, op: osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.421415 7f9d683e1700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:35.421420 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.421422 7f9d683e1700  5 queue_op 0xe2e70e0 seq 2477 osr(3.1b7 0x276d1a0) 583925 bytes   (queue has 1 ops and 583925 bytes)
2014-06-12 14:46:35.421431 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:46:35.421440 7f9d83173700  5 _do_op 0xe2e70e0 seq 2477 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:35.421428 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2667 (0'0,173'2667] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2665 active] do_sub_op osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.421444 7f9d683e1700 10 osd.2 173 dequeue_op 0x55c2000 finish
2014-06-12 14:46:35.421446 7f9d83173700 10 _do_transaction on 0x1ba9c2b0
2014-06-12 14:46:35.421448 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.421451 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.421453 7f9d83173700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:35.421581 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:35.421608 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.421669 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.421695 7f9d83173700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:35.421703 7f9d83173700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:35.421710 7f9d83173700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:35.421723 7f9d83173700 15 _write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 3612672~581632
2014-06-12 14:46:35.421884 7f9d83173700 10 lookup_strip_header done  cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:35.422206 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:35.424238 7f9d83173700 10 _generic_write 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 3612672~581632 = 0
2014-06-12 14:46:35.424297 7f9d83173700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:35.424313 7f9d83173700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:35.424322 7f9d83173700 15 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3
2014-06-12 14:46:35.424332 7f9d83173700 10 _setattrs 3.1b7_head/582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 = 0
2014-06-12 14:46:35.424607 7f9d83173700 20 set_header setting 17215 cid 3.1b7_head oid 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 parent seq 0
2014-06-12 14:46:35.424726 7f9d83173700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:35.424755 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.424777 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:35.427697 7f9d83173700 10 _do_op 0xe2e70e0 seq 2477 r = 0, finisher 0x1e9f1880 0
2014-06-12 14:46:35.427830 7f9d7715b700  5 -- op tracker -- , seq: 747898, time: 2014-06-12 14:46:35.427803, event: commit_sent, op: osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.427860 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2667 (0'0,173'2667] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2665 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:35.428172 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:35.428292 7f9d83173700 10 _finish_op 0xe2e70e0 seq 2477 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:35.428303 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:46:35.428310 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.428833 7f9d7795c700  5 -- op tracker -- , seq: 747898, time: 2014-06-12 14:46:35.428826, event: sub_op_applied, op: osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.428845 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2667 (0'0,173'2667] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2666 active] sub_op_modify_applied on 0x1ba9c1c0 op osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.428864 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2667 (0'0,173'2667] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2666 active] op_applied on primary on version 173'2667
2014-06-12 14:46:35.428874 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2667 (0'0,173'2667] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2666 active] op_applied on replica on version 173'2667
2014-06-12 14:46:35.428897 7f9d7795c700  5 -- op tracker -- , seq: 747898, time: 2014-06-12 14:46:35.428893, event: done, op: osd_sub_op(client.4284.0:141068 3.1b7 582b59b7/rbd_data.10936b8b4567.0000000000000f9e/head//3 [] v 173'2667 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.477220 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:35.477243 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:35.477245 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:35.477246 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:35.477248 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 4; oldest is 2.315903 seconds old
2014-06-12 14:46:35.477308 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:35.567570 7f9d48d28700  5 -- op tracker -- , seq: 747899, time: 2014-06-12 14:46:35.566461, event: header_read, op: osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.567588 7f9d48d28700  5 -- op tracker -- , seq: 747899, time: 2014-06-12 14:46:35.566466, event: throttled, op: osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.567594 7f9d48d28700  5 -- op tracker -- , seq: 747899, time: 2014-06-12 14:46:35.567530, event: all_read, op: osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.567605 7f9d48d28700  5 -- op tracker -- , seq: 747899, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.567610 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:35.567631 7f9d48d28700 15 osd.2 173 enqueue_op 0x2519c300 prio 63 cost 471040 latency 0.001170 osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:35.567674 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.567686 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2519c300 prio 63 cost 471040 latency 0.001224 osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean]
2014-06-12 14:46:35.567717 7f9d62bd6700  5 -- op tracker -- , seq: 747899, time: 2014-06-12 14:46:35.567710, event: reached_pg, op: osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.567730 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:35.567746 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] handle_message: 0x2519c300
2014-06-12 14:46:35.567753 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] do_op osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:35.567772 7f9d62bd6700 15 getattr 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 '_'
2014-06-12 14:46:35.585896 7f9d5a16f700  5 -- op tracker -- , seq: 747900, time: 2014-06-12 14:46:35.584998, event: header_read, op: osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.585908 7f9d5a16f700  5 -- op tracker -- , seq: 747900, time: 2014-06-12 14:46:35.585000, event: throttled, op: osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.585917 7f9d5a16f700  5 -- op tracker -- , seq: 747900, time: 2014-06-12 14:46:35.585855, event: all_read, op: osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.585926 7f9d5a16f700  5 -- op tracker -- , seq: 747900, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.585932 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:35.585939 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:35.585947 7f9d5a16f700 15 osd.2 173 enqueue_op 0x55c3d00 prio 127 cost 557863 latency 0.000949 osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.585976 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:35.585984 7f9d65bdc700 10 osd.2 173 dequeue_op 0x55c3d00 prio 127 cost 557863 latency 0.000986 osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2465 (0'0,173'2465] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2464 active]
2014-06-12 14:46:35.586003 7f9d65bdc700  5 -- op tracker -- , seq: 747900, time: 2014-06-12 14:46:35.586000, event: reached_pg, op: osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.586020 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2465 (0'0,173'2465] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2464 active] handle_message: 0x55c3d00
2014-06-12 14:46:35.586035 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2465 (0'0,173'2465] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2464 active] sub_op_modify trans 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 v 173'2466 (transaction) 174
2014-06-12 14:46:35.586065 7f9d65bdc700  5 -- op tracker -- , seq: 747900, time: 2014-06-12 14:46:35.586060, event: started, op: osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.586080 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2465 (0'0,173'2465] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2464 active] append_log log((0'0,173'2465], crt=0'0) [173'2466 (173'628) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143625 2014-06-12 08:43:17.711594]
2014-06-12 14:46:35.586099 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2466 (0'0,173'2466] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2464 active] add_log_entry 173'2466 (173'628) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143625 2014-06-12 08:43:17.711594
2014-06-12 14:46:35.586117 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2466 (0'0,173'2466] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2464 active] append_log  adding 1 keys
2014-06-12 14:46:35.586150 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2466, trimmed: 
2014-06-12 14:46:35.586173 7f9d65bdc700  5 -- op tracker -- , seq: 747900, time: 2014-06-12 14:46:35.586169, event: started, op: osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.586213 7f9d65bdc700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:35.586220 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.586222 7f9d65bdc700  5 queue_op 0x61d6c00 seq 2299 osr(3.4f4 0x276db70) 559349 bytes   (queue has 1 ops and 559349 bytes)
2014-06-12 14:46:35.586237 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:46:35.586248 7f9d82972700  5 _do_op 0x61d6c00 seq 2299 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:46:35.586255 7f9d82972700 10 _do_transaction on 0x8c1f0b0
2014-06-12 14:46:35.586231 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2466 (0'0,173'2466] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2464 active] do_sub_op osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.586267 7f9d65bdc700 10 osd.2 173 dequeue_op 0x55c3d00 finish
2014-06-12 14:46:35.586270 7f9d82972700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:35.586271 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:35.586274 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.591600 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:35.591624 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.591664 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.591679 7f9d82972700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:35.591683 7f9d82972700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:46:35.591688 7f9d82972700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:35.591698 7f9d82972700 15 _write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 0~557056
2014-06-12 14:46:35.591992 7f9d82972700 10 lookup_strip_header done  cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:35.592020 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:35.593965 7f9d82972700 10 _generic_write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 0~557056 = 0
2014-06-12 14:46:35.594007 7f9d82972700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:35.594026 7f9d82972700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:35.594032 7f9d82972700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:35.594038 7f9d82972700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:35.594253 7f9d82972700 20 set_header setting 17388 cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 parent seq 0
2014-06-12 14:46:35.594370 7f9d82972700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:46:35.594384 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.594391 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:35.594923 7f9d48d28700  5 -- op tracker -- , seq: 747901, time: 2014-06-12 14:46:35.592615, event: header_read, op: osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.594934 7f9d48d28700  5 -- op tracker -- , seq: 747901, time: 2014-06-12 14:46:35.592624, event: throttled, op: osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.594940 7f9d48d28700  5 -- op tracker -- , seq: 747901, time: 2014-06-12 14:46:35.594886, event: all_read, op: osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.594946 7f9d48d28700  5 -- op tracker -- , seq: 747901, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.594954 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:35.594986 7f9d48d28700 15 osd.2 173 enqueue_op 0x68a6100 prio 63 cost 1032192 latency 0.002370 osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:35.595027 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:35.596872 7f9d82972700 10 _do_op 0x61d6c00 seq 2299 r = 0, finisher 0x66b07c0 0
2014-06-12 14:46:35.596887 7f9d82972700 10 _finish_op 0x61d6c00 seq 2299 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:35.596896 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:46:35.596906 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.596902 7f9d7715b700  5 -- op tracker -- , seq: 747900, time: 2014-06-12 14:46:35.596894, event: commit_sent, op: osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.596909 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2466 (0'0,173'2466] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2464 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:35.596932 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:35.597003 7f9d7795c700  5 -- op tracker -- , seq: 747900, time: 2014-06-12 14:46:35.596987, event: sub_op_applied, op: osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.597032 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2466 (0'0,173'2466] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active] sub_op_modify_applied on 0x8c1efc0 op osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.597063 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2466 (0'0,173'2466] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active] op_applied on primary on version 173'2466
2014-06-12 14:46:35.597078 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2466 (0'0,173'2466] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active] op_applied on replica on version 173'2466
2014-06-12 14:46:35.597110 7f9d7795c700  5 -- op tracker -- , seq: 747900, time: 2014-06-12 14:46:35.597107, event: done, op: osd_sub_op(client.4294.0:143625 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2466 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.651207 7f9d62bd6700 10 getattr 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 '_' = 0
2014-06-12 14:46:35.651239 7f9d62bd6700 15 getattr 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 'snapset'
2014-06-12 14:46:35.651394 7f9d62bd6700 10 getattr 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 'snapset' = 0
2014-06-12 14:46:35.651403 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] populate_obc_watchers 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.651422 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.651428 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] get_object_context: creating obc from disk: 0x1551db80
2014-06-12 14:46:35.651434 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] get_object_context: 0x1551db80 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 rwstate(none n=0 w=0) oi: 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3(173'791 client.4289.0:39291 wrlock_by=unknown.0.0:0 dirty s 4194304 uv791) ssc: 0x24d4a620 snapset: 0=[]:[]+head
2014-06-12 14:46:35.651445 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] find_object_context 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 @head oi=1263d312/rbd_data.10986b8b4567.0000000000001139/head//3(173'791 client.4289.0:39291 wrlock_by=unknown.0.0:0 dirty s 4194304 uv791)
2014-06-12 14:46:35.651502 7f9d62bd6700  5 -- op tracker -- , seq: 747899, time: 2014-06-12 14:46:35.651465, event: started, op: osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.651511 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] execute_ctx 0x2963a600
2014-06-12 14:46:35.651523 7f9d62bd6700  5 -- op tracker -- , seq: 747899, time: 2014-06-12 14:46:35.651519, event: started, op: osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.651527 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] do_op 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] ov 173'791 av 173'2768 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:35.651537 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] do_osd_op 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040]
2014-06-12 14:46:35.651546 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:35.651555 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] do_osd_op  write 0~471040
2014-06-12 14:46:35.651567 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:35.651575 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] make_writeable 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 snapset=0x24d4a658  snapc=0=[]
2014-06-12 14:46:35.651584 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] make_writeable 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:35.651592 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] finish_ctx 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 0x2963a600 op modify  
2014-06-12 14:46:35.651599 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean]  set mtime to 2014-06-12 08:45:37.946243
2014-06-12 14:46:35.651614 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean]  final snapset 0=[]:[]+head in 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.651630 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean]  zeroing write result code 0
2014-06-12 14:46:35.651638 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] new_repop rep_tid 236281 on osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:35.651654 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] issue_repop rep_tid 236281 o 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.651670 7f9d62bd6700  5 -- op tracker -- , seq: 747899, time: 2014-06-12 14:46:35.651667, event: waiting for subops from 27, op: osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.651687 7f9d62bd6700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:35.651703 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2767 (0'0,173'2767] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] append_log log((0'0,173'2767], crt=0'0) [173'2768 (173'791) modify   1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 by client.4289.0:143700 2014-06-12 08:45:37.946243]
2014-06-12 14:46:35.651720 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] add_log_entry 173'2768 (173'791) modify   1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 by client.4289.0:143700 2014-06-12 08:45:37.946243
2014-06-12 14:46:35.651733 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] append_log  adding 1 keys
2014-06-12 14:46:35.651756 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2768, trimmed: 
2014-06-12 14:46:35.651772 7f9d62bd6700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:35.651776 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.651777 7f9d62bd6700  5 queue_op 0x9346480 seq 2483 osr(3.312 0x276d540) 473333 bytes   (queue has 1 ops and 473333 bytes)
2014-06-12 14:46:35.651789 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (1 active)
2014-06-12 14:46:35.651785 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] eval_repop repgather(0x2ad98a80 173'2768 rep_tid=236281 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.651799 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2519c300 finish
2014-06-12 14:46:35.651799 7f9d81970700  5 _do_op 0x9346480 seq 2483 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:35.651803 7f9d81970700 10 _do_transaction on 0x34d8e0c0
2014-06-12 14:46:35.651804 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:35.651809 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.651809 7f9d81970700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.651811 7f9d64bda700 10 osd.2 173 dequeue_op 0x68a6100 prio 63 cost 1032192 latency 0.059195 osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean]
2014-06-12 14:46:35.651831 7f9d64bda700  5 -- op tracker -- , seq: 747901, time: 2014-06-12 14:46:35.651828, event: reached_pg, op: osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.651838 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:35.651846 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] handle_message: 0x68a6100
2014-06-12 14:46:35.651852 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] do_op osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:35.651866 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] get_object_context: found obc in cache: 0x1551db80
2014-06-12 14:46:35.651874 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] get_object_context: 0x1551db80 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 rwstate(write n=1 w=0) oi: 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3(173'2768 client.4289.0:143700 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2768) ssc: 0x24d4a620 snapset: 0=[]:[]+head
2014-06-12 14:46:35.651884 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] find_object_context 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 @head oi=1263d312/rbd_data.10986b8b4567.0000000000001139/head//3(173'2768 client.4289.0:143700 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2768)
2014-06-12 14:46:35.651899 7f9d64bda700  5 -- op tracker -- , seq: 747901, time: 2014-06-12 14:46:35.651896, event: started, op: osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.651906 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.651904 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] execute_ctx 0xfd18c00
2014-06-12 14:46:35.651915 7f9d64bda700  5 -- op tracker -- , seq: 747901, time: 2014-06-12 14:46:35.651912, event: started, op: osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.651923 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.651923 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] do_op 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] ov 173'2768 av 173'2769 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:35.651931 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] do_osd_op 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192]
2014-06-12 14:46:35.651937 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:35.651945 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] do_osd_op  write 471040~1032192
2014-06-12 14:46:35.651952 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:35.651959 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] make_writeable 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 snapset=0x24d4a658  snapc=0=[]
2014-06-12 14:46:35.651965 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] make_writeable 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:35.651971 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.651971 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] finish_ctx 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 0xfd18c00 op modify  
2014-06-12 14:46:35.651979 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean]  set mtime to 2014-06-12 08:45:37.971319
2014-06-12 14:46:35.651991 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean]  final snapset 0=[]:[]+head in 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.652003 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean]  zeroing write result code 0
2014-06-12 14:46:35.652010 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] new_repop rep_tid 236282 on osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:35.652021 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] issue_repop rep_tid 236282 o 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.652035 7f9d64bda700  5 -- op tracker -- , seq: 747901, time: 2014-06-12 14:46:35.652031, event: waiting for subops from 27, op: osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.652049 7f9d64bda700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:35.652067 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2768 (0'0,173'2768] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] append_log log((0'0,173'2768], crt=0'0) [173'2769 (173'2768) modify   1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 by client.4289.0:143701 2014-06-12 08:45:37.971319]
2014-06-12 14:46:35.652081 7f9d81970700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.652082 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] add_log_entry 173'2769 (173'2768) modify   1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 by client.4289.0:143701 2014-06-12 08:45:37.971319
2014-06-12 14:46:35.652089 7f9d81970700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:35.652095 7f9d81970700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.652099 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] append_log  adding 1 keys
2014-06-12 14:46:35.652111 7f9d81970700 15 _write 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 0~471040
2014-06-12 14:46:35.652116 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2769, trimmed: 
2014-06-12 14:46:35.652127 7f9d64bda700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:35.652129 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.652130 7f9d64bda700  5 queue_op 0x581d500 seq 2484 osr(3.312 0x276d540) 1034485 bytes   (queue has 2 ops and 1507818 bytes)
2014-06-12 14:46:35.652134 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] eval_repop repgather(0x5146b40 173'2769 rep_tid=236282 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.652148 7f9d64bda700 10 osd.2 173 dequeue_op 0x68a6100 finish
2014-06-12 14:46:35.652153 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.652156 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.652265 7f9d81970700 10 lookup_strip_header done  cid 3.312_head oid 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.652420 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:35.653589 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (2 active)
2014-06-12 14:46:35.653927 7f9d81970700 10 _generic_write 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 0~471040 = 0
2014-06-12 14:46:35.653974 7f9d81970700 15 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.653986 7f9d81970700 10 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 = 0
2014-06-12 14:46:35.653992 7f9d81970700 15 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.653998 7f9d81970700 10 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 = 0
2014-06-12 14:46:35.654195 7f9d81970700 20 set_header setting 17458 cid 3.312_head oid 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 parent seq 0
2014-06-12 14:46:35.654262 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.654278 7f9d81970700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:35.654289 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:35.655800 7f9d81970700 10 _do_op 0x9346480 seq 2483 r = 0, finisher 0x26867100 0x19b2c500
2014-06-12 14:46:35.655813 7f9d81970700 10 _finish_op 0x9346480 seq 2483 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:35.655820 7f9d7c966700  5 _do_op 0x581d500 seq 2484 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:35.655822 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (1 active)
2014-06-12 14:46:35.655827 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.655827 7f9d7c966700 10 _do_transaction on 0x31b65bc0
2014-06-12 14:46:35.655834 7f9d7c966700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.655821 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] op_commit: 236281
2014-06-12 14:46:35.655850 7f9d7715b700  5 -- op tracker -- , seq: 747899, time: 2014-06-12 14:46:35.655841, event: op_commit, op: osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.655865 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] op_applied: 236281
2014-06-12 14:46:35.655890 7f9d7795c700  5 -- op tracker -- , seq: 747899, time: 2014-06-12 14:46:35.655883, event: op_applied, op: osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.655900 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] op_applied on primary on version 173'2768
2014-06-12 14:46:35.655929 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.655943 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.655978 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.655988 7f9d7c966700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.655992 7f9d7c966700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:35.656000 7f9d7c966700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.656008 7f9d7c966700 15 _write 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 471040~1032192
2014-06-12 14:46:35.656099 7f9d7c966700 10 lookup_strip_header done  cid 3.312_head oid 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.656115 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:35.658930 7f9d7c966700 10 _generic_write 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 471040~1032192 = 0
2014-06-12 14:46:35.659020 7f9d7c966700 15 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.659050 7f9d7c966700 10 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 = 0
2014-06-12 14:46:35.659058 7f9d7c966700 15 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.659067 7f9d7c966700 10 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 = 0
2014-06-12 14:46:35.659339 7f9d7c966700 20 set_header setting 17458 cid 3.312_head oid 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 parent seq 0
2014-06-12 14:46:35.659424 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.659435 7f9d7c966700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:35.659448 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:35.661960 7f9d58c5a700  5 -- op tracker -- , seq: 747902, time: 2014-06-12 14:46:35.661901, event: header_read, op: osd_sub_op_reply(client.4289.0:143700 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.661984 7f9d58c5a700  5 -- op tracker -- , seq: 747902, time: 2014-06-12 14:46:35.661903, event: throttled, op: osd_sub_op_reply(client.4289.0:143700 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.661992 7f9d58c5a700  5 -- op tracker -- , seq: 747902, time: 2014-06-12 14:46:35.661916, event: all_read, op: osd_sub_op_reply(client.4289.0:143700 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.662000 7f9d58c5a700  5 -- op tracker -- , seq: 747902, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143700 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.662006 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143700 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:35.662013 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:35.662023 7f9d58c5a700 15 osd.2 173 enqueue_op 0x8dbca00 prio 196 cost 0 latency 0.000121 osd_sub_op_reply(client.4289.0:143700 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:35.662088 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.662099 7f9d6abe6700 10 osd.2 173 dequeue_op 0x8dbca00 prio 196 cost 0 latency 0.000198 osd_sub_op_reply(client.4289.0:143700 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean]
2014-06-12 14:46:35.662125 7f9d6abe6700  5 -- op tracker -- , seq: 747902, time: 2014-06-12 14:46:35.662123, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143700 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.662147 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] handle_message: 0x8dbca00
2014-06-12 14:46:35.662159 7f9d6abe6700  5 -- op tracker -- , seq: 747902, time: 2014-06-12 14:46:35.662157, event: started, op: osd_sub_op_reply(client.4289.0:143700 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.662163 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] sub_op_modify_reply: tid 236281 op  ack_type 4 from 27
2014-06-12 14:46:35.662177 7f9d6abe6700  5 -- op tracker -- , seq: 747899, time: 2014-06-12 14:46:35.662171, event: sub_op_commit_rec, op: osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.662182 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] repop_all_applied: repop tid 236281 all applied 
2014-06-12 14:46:35.662189 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] eval_repop repgather(0x2ad98a80 173'2768 rep_tid=236281 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.662199 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean]  sending ack on repgather(0x2ad98a80 173'2768 rep_tid=236281 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) 0x11670f00
2014-06-12 14:46:35.662221 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2767 crt=0'0 lcod 173'2766 mlcod 173'2766 active+clean] repop_all_committed: repop tid 236281 all committed 
2014-06-12 14:46:35.662232 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2766 active+clean] eval_repop repgather(0x2ad98a80 173'2768 rep_tid=236281 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.662262 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2766 active+clean] log_op_stats osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 inb 471319 outb 0 rlat 0.095759 lat 0.095800
2014-06-12 14:46:35.662279 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2766 active+clean] publish_stats_to_osd 173:3455
2014-06-12 14:46:35.662289 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2766 active+clean]  sending commit on repgather(0x2ad98a80 173'2768 rep_tid=236281 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) 0x1285ac80
2014-06-12 14:46:35.662312 7f9d6abe6700  5 -- op tracker -- , seq: 747899, time: 2014-06-12 14:46:35.662307, event: commit_sent, op: osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.662322 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2766 active+clean]  requeue_ops 
2014-06-12 14:46:35.662348 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean]  removing repgather(0x2ad98a80 173'2768 rep_tid=236281 committed?=1 applied?=1 op=osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.662358 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean]    q front is repgather(0x2ad98a80 173'2768 rep_tid=236281 committed?=1 applied?=1 op=osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.662367 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] remove_repop repgather(0x2ad98a80 173'2768 rep_tid=236281 committed?=1 applied?=1 op=osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.662375 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean]  requeue_ops 
2014-06-12 14:46:35.662388 7f9d6abe6700  5 -- op tracker -- , seq: 747899, time: 2014-06-12 14:46:35.662385, event: done, op: osd_op(client.4289.0:143700 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~471040] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.662409 7f9d6abe6700 10 osd.2 173 dequeue_op 0x8dbca00 finish
2014-06-12 14:46:35.662413 7f9d6abe6700  5 -- op tracker -- , seq: 747902, time: 2014-06-12 14:46:35.662411, event: done, op: osd_sub_op_reply(client.4289.0:143700 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.662420 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.662422 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.663326 7f9d7c966700 10 _do_op 0x581d500 seq 2484 r = 0, finisher 0x3fd9820 0x3eabb100
2014-06-12 14:46:35.663341 7f9d7c966700 10 _finish_op 0x581d500 seq 2484 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:35.663348 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:35.663355 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.663346 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] op_commit: 236282
2014-06-12 14:46:35.663368 7f9d7715b700  5 -- op tracker -- , seq: 747901, time: 2014-06-12 14:46:35.663361, event: op_commit, op: osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.663382 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] op_applied: 236282
2014-06-12 14:46:35.663403 7f9d7795c700  5 -- op tracker -- , seq: 747901, time: 2014-06-12 14:46:35.663397, event: op_applied, op: osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.663416 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] op_applied on primary on version 173'2769
2014-06-12 14:46:35.665755 7f9d48d28700  5 -- op tracker -- , seq: 747903, time: 2014-06-12 14:46:35.664823, event: header_read, op: osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.665767 7f9d48d28700  5 -- op tracker -- , seq: 747903, time: 2014-06-12 14:46:35.664825, event: throttled, op: osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.665783 7f9d48d28700  5 -- op tracker -- , seq: 747903, time: 2014-06-12 14:46:35.665732, event: all_read, op: osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.665788 7f9d48d28700  5 -- op tracker -- , seq: 747903, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.665794 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:35.665813 7f9d48d28700 15 osd.2 173 enqueue_op 0x124e3e00 prio 63 cost 516096 latency 0.000990 osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:35.665833 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.665841 7f9d66bde700 10 osd.2 173 dequeue_op 0x124e3e00 prio 63 cost 516096 latency 0.001018 osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean]
2014-06-12 14:46:35.665858 7f9d66bde700  5 -- op tracker -- , seq: 747903, time: 2014-06-12 14:46:35.665855, event: reached_pg, op: osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.665865 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:35.665874 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] handle_message: 0x124e3e00
2014-06-12 14:46:35.665880 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] do_op osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:35.665897 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] get_object_context: found obc in cache: 0x1551db80
2014-06-12 14:46:35.665903 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] get_object_context: 0x1551db80 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 rwstate(write n=1 w=0) oi: 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3(173'2769 client.4289.0:143701 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2769) ssc: 0x24d4a620 snapset: 0=[]:[]+head
2014-06-12 14:46:35.665914 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] find_object_context 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 @head oi=1263d312/rbd_data.10986b8b4567.0000000000001139/head//3(173'2769 client.4289.0:143701 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2769)
2014-06-12 14:46:35.665932 7f9d66bde700  5 -- op tracker -- , seq: 747903, time: 2014-06-12 14:46:35.665928, event: started, op: osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.665937 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] execute_ctx 0x1b262000
2014-06-12 14:46:35.665948 7f9d66bde700  5 -- op tracker -- , seq: 747903, time: 2014-06-12 14:46:35.665945, event: started, op: osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.665952 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] do_op 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] ov 173'2769 av 173'2770 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:35.665960 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] do_osd_op 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096]
2014-06-12 14:46:35.665966 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:35.665975 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] do_osd_op  write 1503232~516096
2014-06-12 14:46:35.665985 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:35.665992 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] make_writeable 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 snapset=0x24d4a658  snapc=0=[]
2014-06-12 14:46:35.665999 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] make_writeable 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:35.666005 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] finish_ctx 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 0x1b262000 op modify  
2014-06-12 14:46:35.666013 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean]  set mtime to 2014-06-12 08:45:38.044472
2014-06-12 14:46:35.666026 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean]  final snapset 0=[]:[]+head in 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.666038 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean]  zeroing write result code 0
2014-06-12 14:46:35.666046 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] new_repop rep_tid 236283 on osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:35.666062 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] issue_repop rep_tid 236283 o 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.666086 7f9d66bde700  5 -- op tracker -- , seq: 747903, time: 2014-06-12 14:46:35.666083, event: waiting for subops from 27, op: osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.666104 7f9d66bde700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:35.666118 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2769 (0'0,173'2769] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] append_log log((0'0,173'2769], crt=0'0) [173'2770 (173'2769) modify   1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 by client.4289.0:143702 2014-06-12 08:45:38.044472]
2014-06-12 14:46:35.666135 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] add_log_entry 173'2770 (173'2769) modify   1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 by client.4289.0:143702 2014-06-12 08:45:38.044472
2014-06-12 14:46:35.666156 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] append_log  adding 1 keys
2014-06-12 14:46:35.666179 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2770, trimmed: 
2014-06-12 14:46:35.666197 7f9d66bde700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:35.666199 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.666200 7f9d66bde700  5 queue_op 0x28e7620 seq 2485 osr(3.312 0x276d540) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:35.666207 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (1 active)
2014-06-12 14:46:35.666213 7f9d7a962700  5 _do_op 0x28e7620 seq 2485 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:35.666216 7f9d7a962700 10 _do_transaction on 0x2d812000
2014-06-12 14:46:35.666205 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] eval_repop repgather(0x2d812840 173'2770 rep_tid=236283 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.666222 7f9d66bde700 10 osd.2 173 dequeue_op 0x124e3e00 finish
2014-06-12 14:46:35.666222 7f9d7a962700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.666225 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.666227 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.666318 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.666335 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.666372 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.666383 7f9d7a962700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.666386 7f9d7a962700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:35.666389 7f9d7a962700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.666398 7f9d7a962700 15 _write 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 1503232~516096
2014-06-12 14:46:35.666513 7f9d7a962700 10 lookup_strip_header done  cid 3.312_head oid 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.666532 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:35.667805 7f9d7a962700 10 _generic_write 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 1503232~516096 = 0
2014-06-12 14:46:35.667844 7f9d7a962700 15 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.667855 7f9d7a962700 10 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 = 0
2014-06-12 14:46:35.667860 7f9d7a962700 15 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.667867 7f9d7a962700 10 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 = 0
2014-06-12 14:46:35.668070 7f9d7a962700 20 set_header setting 17458 cid 3.312_head oid 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 parent seq 0
2014-06-12 14:46:35.668133 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.668145 7f9d7a962700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:35.668154 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:35.670586 7f9d7a962700 10 _do_op 0x28e7620 seq 2485 r = 0, finisher 0x6f13c20 0xd9452c0
2014-06-12 14:46:35.670602 7f9d7a962700 10 _finish_op 0x28e7620 seq 2485 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:35.670612 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:35.670615 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.670626 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] op_commit: 236283
2014-06-12 14:46:35.670652 7f9d7715b700  5 -- op tracker -- , seq: 747903, time: 2014-06-12 14:46:35.670644, event: op_commit, op: osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.670674 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] op_applied: 236283
2014-06-12 14:46:35.670699 7f9d7795c700  5 -- op tracker -- , seq: 747903, time: 2014-06-12 14:46:35.670692, event: op_applied, op: osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.670707 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] op_applied on primary on version 173'2770
2014-06-12 14:46:35.672986 7f9d58c5a700  5 -- op tracker -- , seq: 747904, time: 2014-06-12 14:46:35.672941, event: header_read, op: osd_sub_op_reply(client.4289.0:143701 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.673001 7f9d58c5a700  5 -- op tracker -- , seq: 747904, time: 2014-06-12 14:46:35.672942, event: throttled, op: osd_sub_op_reply(client.4289.0:143701 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.673009 7f9d58c5a700  5 -- op tracker -- , seq: 747904, time: 2014-06-12 14:46:35.672954, event: all_read, op: osd_sub_op_reply(client.4289.0:143701 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.673017 7f9d58c5a700  5 -- op tracker -- , seq: 747904, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143701 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.673025 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143701 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:35.673031 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:35.673039 7f9d58c5a700 15 osd.2 173 enqueue_op 0x2fa9d800 prio 196 cost 0 latency 0.000098 osd_sub_op_reply(client.4289.0:143701 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:35.673069 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.673080 7f9d69be4700 10 osd.2 173 dequeue_op 0x2fa9d800 prio 196 cost 0 latency 0.000139 osd_sub_op_reply(client.4289.0:143701 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean]
2014-06-12 14:46:35.673102 7f9d69be4700  5 -- op tracker -- , seq: 747904, time: 2014-06-12 14:46:35.673098, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143701 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.673110 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] handle_message: 0x2fa9d800
2014-06-12 14:46:35.673130 7f9d69be4700  5 -- op tracker -- , seq: 747904, time: 2014-06-12 14:46:35.673126, event: started, op: osd_sub_op_reply(client.4289.0:143701 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.673136 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] sub_op_modify_reply: tid 236282 op  ack_type 4 from 27
2014-06-12 14:46:35.673157 7f9d69be4700  5 -- op tracker -- , seq: 747901, time: 2014-06-12 14:46:35.673150, event: sub_op_commit_rec, op: osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.673165 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] repop_all_applied: repop tid 236282 all applied 
2014-06-12 14:46:35.673187 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] eval_repop repgather(0x5146b40 173'2769 rep_tid=236282 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.673205 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean]  sending ack on repgather(0x5146b40 173'2769 rep_tid=236282 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) 0x25915b80
2014-06-12 14:46:35.673240 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2768 crt=0'0 lcod 173'2767 mlcod 173'2767 active+clean] repop_all_committed: repop tid 236282 all committed 
2014-06-12 14:46:35.673252 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2767 active+clean] eval_repop repgather(0x5146b40 173'2769 rep_tid=236282 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.673271 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2767 active+clean] log_op_stats osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.080624 lat 0.080655
2014-06-12 14:46:35.673291 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2767 active+clean] publish_stats_to_osd 173:3456
2014-06-12 14:46:35.673304 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2767 active+clean]  sending commit on repgather(0x5146b40 173'2769 rep_tid=236282 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) 0x1080cc80
2014-06-12 14:46:35.673336 7f9d69be4700  5 -- op tracker -- , seq: 747901, time: 2014-06-12 14:46:35.673331, event: commit_sent, op: osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.673345 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2767 active+clean]  requeue_ops 
2014-06-12 14:46:35.673355 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2768 active+clean]  removing repgather(0x5146b40 173'2769 rep_tid=236282 committed?=1 applied?=1 op=osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.673372 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2768 active+clean]    q front is repgather(0x5146b40 173'2769 rep_tid=236282 committed?=1 applied?=1 op=osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.673388 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2768 active+clean] remove_repop repgather(0x5146b40 173'2769 rep_tid=236282 committed?=1 applied?=1 op=osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.673405 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2768 active+clean]  requeue_ops 
2014-06-12 14:46:35.673425 7f9d69be4700  5 -- op tracker -- , seq: 747901, time: 2014-06-12 14:46:35.673419, event: done, op: osd_op(client.4289.0:143701 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 471040~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.673445 7f9d69be4700 10 osd.2 173 dequeue_op 0x2fa9d800 finish
2014-06-12 14:46:35.673454 7f9d69be4700  5 -- op tracker -- , seq: 747904, time: 2014-06-12 14:46:35.673450, event: done, op: osd_sub_op_reply(client.4289.0:143701 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.673465 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.673468 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.679800 7f9d48d28700  5 -- op tracker -- , seq: 747905, time: 2014-06-12 14:46:35.677396, event: header_read, op: osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.679815 7f9d48d28700  5 -- op tracker -- , seq: 747905, time: 2014-06-12 14:46:35.677400, event: throttled, op: osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.679825 7f9d48d28700  5 -- op tracker -- , seq: 747905, time: 2014-06-12 14:46:35.679748, event: all_read, op: osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.679834 7f9d48d28700  5 -- op tracker -- , seq: 747905, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.679843 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:35.679844 7f9d58c5a700  5 -- op tracker -- , seq: 747906, time: 2014-06-12 14:46:35.679797, event: header_read, op: osd_sub_op_reply(client.4289.0:143702 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.679853 7f9d58c5a700  5 -- op tracker -- , seq: 747906, time: 2014-06-12 14:46:35.679799, event: throttled, op: osd_sub_op_reply(client.4289.0:143702 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.679860 7f9d58c5a700  5 -- op tracker -- , seq: 747906, time: 2014-06-12 14:46:35.679818, event: all_read, op: osd_sub_op_reply(client.4289.0:143702 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.679867 7f9d58c5a700  5 -- op tracker -- , seq: 747906, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143702 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.679871 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143702 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:35.679876 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:35.679872 7f9d48d28700 15 osd.2 173 enqueue_op 0x876f000 prio 63 cost 1032192 latency 0.002476 osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:35.679883 7f9d58c5a700 15 osd.2 173 enqueue_op 0x2fa9c700 prio 196 cost 0 latency 0.000086 osd_sub_op_reply(client.4289.0:143702 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:35.679916 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.679935 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:35.679932 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2fa9c700 prio 196 cost 0 latency 0.000135 osd_sub_op_reply(client.4289.0:143702 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2768 active+clean]
2014-06-12 14:46:35.679953 7f9d6a3e5700  5 -- op tracker -- , seq: 747906, time: 2014-06-12 14:46:35.679949, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143702 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.679962 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2768 active+clean] handle_message: 0x2fa9c700
2014-06-12 14:46:35.679978 7f9d6a3e5700  5 -- op tracker -- , seq: 747906, time: 2014-06-12 14:46:35.679975, event: started, op: osd_sub_op_reply(client.4289.0:143702 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.679984 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2768 active+clean] sub_op_modify_reply: tid 236283 op  ack_type 4 from 27
2014-06-12 14:46:35.680004 7f9d6a3e5700  5 -- op tracker -- , seq: 747903, time: 2014-06-12 14:46:35.679997, event: sub_op_commit_rec, op: osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.680012 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2768 active+clean] repop_all_applied: repop tid 236283 all applied 
2014-06-12 14:46:35.680024 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2768 active+clean] eval_repop repgather(0x2d812840 173'2770 rep_tid=236283 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.680040 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2768 active+clean]  sending ack on repgather(0x2d812840 173'2770 rep_tid=236283 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) 0x1285a000
2014-06-12 14:46:35.680073 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2769 crt=0'0 lcod 173'2768 mlcod 173'2768 active+clean] repop_all_committed: repop tid 236283 all committed 
2014-06-12 14:46:35.680083 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2768 active+clean] eval_repop repgather(0x2d812840 173'2770 rep_tid=236283 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.680099 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2768 active+clean] log_op_stats osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.015248 lat 0.015275
2014-06-12 14:46:35.680117 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2768 active+clean] publish_stats_to_osd 173:3457
2014-06-12 14:46:35.680132 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2768 active+clean]  sending commit on repgather(0x2d812840 173'2770 rep_tid=236283 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) 0x32df0a00
2014-06-12 14:46:35.680164 7f9d6a3e5700  5 -- op tracker -- , seq: 747903, time: 2014-06-12 14:46:35.680158, event: commit_sent, op: osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.680171 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2768 active+clean]  requeue_ops 
2014-06-12 14:46:35.680181 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean]  removing repgather(0x2d812840 173'2770 rep_tid=236283 committed?=1 applied?=1 op=osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.680197 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean]    q front is repgather(0x2d812840 173'2770 rep_tid=236283 committed?=1 applied?=1 op=osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.680212 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] remove_repop repgather(0x2d812840 173'2770 rep_tid=236283 committed?=1 applied?=1 op=osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.680226 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean]  requeue_ops 
2014-06-12 14:46:35.680255 7f9d6a3e5700  5 -- op tracker -- , seq: 747903, time: 2014-06-12 14:46:35.680249, event: done, op: osd_op(client.4289.0:143702 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 1503232~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.680272 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2fa9c700 finish
2014-06-12 14:46:35.680283 7f9d6a3e5700  5 -- op tracker -- , seq: 747906, time: 2014-06-12 14:46:35.680280, event: done, op: osd_sub_op_reply(client.4289.0:143702 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.680282 7f9d653db700 10 osd.2 173 dequeue_op 0x876f000 prio 63 cost 1032192 latency 0.002886 osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean]
2014-06-12 14:46:35.680341 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:35.680344 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.680346 7f9d653db700  5 -- op tracker -- , seq: 747905, time: 2014-06-12 14:46:35.680336, event: reached_pg, op: osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.680361 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:35.680372 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] handle_message: 0x876f000
2014-06-12 14:46:35.680378 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] do_op osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:35.680395 7f9d653db700 15 getattr 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 '_'
2014-06-12 14:46:35.680557 7f9d653db700 10 getattr 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 '_' = 0
2014-06-12 14:46:35.680568 7f9d653db700 15 getattr 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 'snapset'
2014-06-12 14:46:35.680628 7f9d653db700 10 getattr 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 'snapset' = 0
2014-06-12 14:46:35.680634 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] populate_obc_watchers 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.680642 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.680648 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] get_object_context: creating obc from disk: 0x2fc00dc0
2014-06-12 14:46:35.680653 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] get_object_context: 0x2fc00dc0 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 rwstate(none n=0 w=0) oi: 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3(173'2770 client.4289.0:143702 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2770) ssc: 0x82de380 snapset: 0=[]:[]+head
2014-06-12 14:46:35.680663 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] find_object_context 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 @head oi=1263d312/rbd_data.10986b8b4567.0000000000001139/head//3(173'2770 client.4289.0:143702 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2770)
2014-06-12 14:46:35.680679 7f9d653db700  5 -- op tracker -- , seq: 747905, time: 2014-06-12 14:46:35.680676, event: started, op: osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.680685 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] execute_ctx 0x27bc3000
2014-06-12 14:46:35.680694 7f9d653db700  5 -- op tracker -- , seq: 747905, time: 2014-06-12 14:46:35.680692, event: started, op: osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.680698 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] do_op 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] ov 173'2770 av 173'2771 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:35.680707 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] do_osd_op 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192]
2014-06-12 14:46:35.680713 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:35.680722 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] do_osd_op  write 2019328~1032192
2014-06-12 14:46:35.680732 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:35.680738 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] make_writeable 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 snapset=0x82de3b8  snapc=0=[]
2014-06-12 14:46:35.680744 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] make_writeable 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:35.680750 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] finish_ctx 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 0x27bc3000 op modify  
2014-06-12 14:46:35.680760 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean]  set mtime to 2014-06-12 08:45:38.056141
2014-06-12 14:46:35.680773 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean]  final snapset 0=[]:[]+head in 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.680783 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean]  zeroing write result code 0
2014-06-12 14:46:35.680790 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] new_repop rep_tid 236284 on osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:35.680800 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] issue_repop rep_tid 236284 o 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.680814 7f9d653db700  5 -- op tracker -- , seq: 747905, time: 2014-06-12 14:46:35.680811, event: waiting for subops from 27, op: osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.680826 7f9d653db700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:35.680840 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2770 (0'0,173'2770] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] append_log log((0'0,173'2770], crt=0'0) [173'2771 (173'2770) modify   1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 by client.4289.0:143703 2014-06-12 08:45:38.056141]
2014-06-12 14:46:35.680862 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] add_log_entry 173'2771 (173'2770) modify   1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 by client.4289.0:143703 2014-06-12 08:45:38.056141
2014-06-12 14:46:35.680880 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] append_log  adding 1 keys
2014-06-12 14:46:35.680907 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2771, trimmed: 
2014-06-12 14:46:35.680926 7f9d653db700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:35.680929 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.680930 7f9d653db700  5 queue_op 0x641d560 seq 2486 osr(3.312 0x276d540) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:35.680937 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] eval_repop repgather(0x12ae3bc0 173'2771 rep_tid=236284 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.680952 7f9d653db700 10 osd.2 173 dequeue_op 0x876f000 finish
2014-06-12 14:46:35.680956 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.680958 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.680963 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (1 active)
2014-06-12 14:46:35.680967 7f9d7895e700  5 _do_op 0x641d560 seq 2486 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:35.680971 7f9d7895e700 10 _do_transaction on 0x12ae3a40
2014-06-12 14:46:35.680976 7f9d7895e700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.681137 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.681155 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.681198 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.681211 7f9d7895e700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.681216 7f9d7895e700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:35.681221 7f9d7895e700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.681231 7f9d7895e700 15 _write 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 2019328~1032192
2014-06-12 14:46:35.681356 7f9d7895e700 10 lookup_strip_header done  cid 3.312_head oid 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.681516 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:35.683925 7f9d48d28700  5 -- op tracker -- , seq: 747907, time: 2014-06-12 14:46:35.682806, event: header_read, op: osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.683937 7f9d48d28700  5 -- op tracker -- , seq: 747907, time: 2014-06-12 14:46:35.682810, event: throttled, op: osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.683946 7f9d48d28700  5 -- op tracker -- , seq: 747907, time: 2014-06-12 14:46:35.683887, event: all_read, op: osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.683971 7f9d48d28700  5 -- op tracker -- , seq: 747907, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.683978 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:35.683999 7f9d48d28700 15 osd.2 173 enqueue_op 0x598f200 prio 63 cost 516096 latency 0.001193 osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:35.684028 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.684034 7f9d63bd8700 10 osd.2 173 dequeue_op 0x598f200 prio 63 cost 516096 latency 0.001228 osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean]
2014-06-12 14:46:35.684057 7f9d63bd8700  5 -- op tracker -- , seq: 747907, time: 2014-06-12 14:46:35.684052, event: reached_pg, op: osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.684064 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:35.684077 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] handle_message: 0x598f200
2014-06-12 14:46:35.684086 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] do_op osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:35.684106 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] get_object_context: found obc in cache: 0x2fc00dc0
2014-06-12 14:46:35.684116 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] get_object_context: 0x2fc00dc0 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 rwstate(write n=1 w=0) oi: 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3(173'2771 client.4289.0:143703 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2771) ssc: 0x82de380 snapset: 0=[]:[]+head
2014-06-12 14:46:35.684131 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] find_object_context 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 @head oi=1263d312/rbd_data.10986b8b4567.0000000000001139/head//3(173'2771 client.4289.0:143703 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2771)
2014-06-12 14:46:35.684156 7f9d63bd8700  5 -- op tracker -- , seq: 747907, time: 2014-06-12 14:46:35.684151, event: started, op: osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.684162 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] execute_ctx 0x30f7e400
2014-06-12 14:46:35.684177 7f9d63bd8700  5 -- op tracker -- , seq: 747907, time: 2014-06-12 14:46:35.684173, event: started, op: osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.684183 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] do_op 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] ov 173'2771 av 173'2772 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:35.684195 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] do_osd_op 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096]
2014-06-12 14:46:35.684205 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:35.684218 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] do_osd_op  write 3051520~516096
2014-06-12 14:46:35.684232 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:35.684242 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] make_writeable 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 snapset=0x82de3b8  snapc=0=[]
2014-06-12 14:46:35.684253 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] make_writeable 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:35.684262 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] finish_ctx 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 0x30f7e400 op modify  
2014-06-12 14:46:35.684274 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean]  set mtime to 2014-06-12 08:45:38.062426
2014-06-12 14:46:35.684293 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean]  final snapset 0=[]:[]+head in 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.684310 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean]  zeroing write result code 0
2014-06-12 14:46:35.684320 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] new_repop rep_tid 236285 on osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:35.684334 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] issue_repop rep_tid 236285 o 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.684357 7f9d63bd8700  5 -- op tracker -- , seq: 747907, time: 2014-06-12 14:46:35.684353, event: waiting for subops from 27, op: osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.684378 7f9d63bd8700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:35.684399 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2771 (0'0,173'2771] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] append_log log((0'0,173'2771], crt=0'0) [173'2772 (173'2771) modify   1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 by client.4289.0:143704 2014-06-12 08:45:38.062426]
2014-06-12 14:46:35.684421 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] add_log_entry 173'2772 (173'2771) modify   1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 by client.4289.0:143704 2014-06-12 08:45:38.062426
2014-06-12 14:46:35.684868 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] append_log  adding 1 keys
2014-06-12 14:46:35.684899 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2772, trimmed: 
2014-06-12 14:46:35.684919 7f9d63bd8700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:35.684917 7f9d7895e700 10 _generic_write 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 2019328~1032192 = 0
2014-06-12 14:46:35.684924 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.684926 7f9d63bd8700  5 queue_op 0x614d080 seq 2487 osr(3.312 0x276d540) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:46:35.684932 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] eval_repop repgather(0x12ae2240 173'2772 rep_tid=236285 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.684950 7f9d63bd8700 10 osd.2 173 dequeue_op 0x598f200 finish
2014-06-12 14:46:35.684953 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.684956 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.684964 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (2 active)
2014-06-12 14:46:35.685006 7f9d7895e700 15 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.685020 7f9d7895e700 10 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 = 0
2014-06-12 14:46:35.685025 7f9d7895e700 15 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.685029 7f9d7895e700 10 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 = 0
2014-06-12 14:46:35.685239 7f9d7895e700 20 set_header setting 17458 cid 3.312_head oid 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 parent seq 0
2014-06-12 14:46:35.685308 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.685319 7f9d7895e700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:35.685325 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:35.689208 7f9d7895e700 10 _do_op 0x641d560 seq 2486 r = 0, finisher 0x654d960 0x4537080
2014-06-12 14:46:35.689228 7f9d7895e700 10 _finish_op 0x641d560 seq 2486 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:35.689240 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (1 active)
2014-06-12 14:46:35.689249 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.689235 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] op_commit: 236284
2014-06-12 14:46:35.689263 7f9d7715b700  5 -- op tracker -- , seq: 747905, time: 2014-06-12 14:46:35.689255, event: op_commit, op: osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.689271 7f9d7a161700  5 _do_op 0x614d080 seq 2487 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:35.689276 7f9d7a161700 10 _do_transaction on 0x12ae2fc0
2014-06-12 14:46:35.689284 7f9d7a161700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.689277 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] op_applied: 236284
2014-06-12 14:46:35.689305 7f9d7795c700  5 -- op tracker -- , seq: 747905, time: 2014-06-12 14:46:35.689297, event: op_applied, op: osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.689314 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] op_applied on primary on version 173'2771
2014-06-12 14:46:35.689374 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.689392 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.689426 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.689438 7f9d7a161700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.689443 7f9d7a161700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:35.689447 7f9d7a161700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.689456 7f9d7a161700 15 _write 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 3051520~516096
2014-06-12 14:46:35.689547 7f9d7a161700 10 lookup_strip_header done  cid 3.312_head oid 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.689560 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:35.690765 7f9d7a161700 10 _generic_write 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 3051520~516096 = 0
2014-06-12 14:46:35.690801 7f9d7a161700 15 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.690815 7f9d7a161700 10 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 = 0
2014-06-12 14:46:35.690819 7f9d7a161700 15 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.690824 7f9d7a161700 10 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 = 0
2014-06-12 14:46:35.690943 7f9d7a161700 20 set_header setting 17458 cid 3.312_head oid 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 parent seq 0
2014-06-12 14:46:35.690984 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.690994 7f9d7a161700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:35.691000 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:35.692553 7f9d58c5a700  5 -- op tracker -- , seq: 747908, time: 2014-06-12 14:46:35.692508, event: header_read, op: osd_sub_op_reply(client.4289.0:143703 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.692566 7f9d58c5a700  5 -- op tracker -- , seq: 747908, time: 2014-06-12 14:46:35.692509, event: throttled, op: osd_sub_op_reply(client.4289.0:143703 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.692575 7f9d58c5a700  5 -- op tracker -- , seq: 747908, time: 2014-06-12 14:46:35.692524, event: all_read, op: osd_sub_op_reply(client.4289.0:143703 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.692582 7f9d58c5a700  5 -- op tracker -- , seq: 747908, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143703 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.692586 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143703 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:35.692595 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:35.692605 7f9d58c5a700 15 osd.2 173 enqueue_op 0x2fa9c300 prio 196 cost 0 latency 0.000097 osd_sub_op_reply(client.4289.0:143703 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:35.692633 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.692645 7f9d623d5700 10 osd.2 173 dequeue_op 0x2fa9c300 prio 196 cost 0 latency 0.000136 osd_sub_op_reply(client.4289.0:143703 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean]
2014-06-12 14:46:35.692669 7f9d623d5700  5 -- op tracker -- , seq: 747908, time: 2014-06-12 14:46:35.692665, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143703 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.692681 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] handle_message: 0x2fa9c300
2014-06-12 14:46:35.692696 7f9d623d5700  5 -- op tracker -- , seq: 747908, time: 2014-06-12 14:46:35.692693, event: started, op: osd_sub_op_reply(client.4289.0:143703 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.692706 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] sub_op_modify_reply: tid 236284 op  ack_type 4 from 27
2014-06-12 14:46:35.692728 7f9d623d5700  5 -- op tracker -- , seq: 747905, time: 2014-06-12 14:46:35.692721, event: sub_op_commit_rec, op: osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.692738 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] repop_all_applied: repop tid 236284 all applied 
2014-06-12 14:46:35.692751 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] eval_repop repgather(0x12ae3bc0 173'2771 rep_tid=236284 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.692769 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean]  sending ack on repgather(0x12ae3bc0 173'2771 rep_tid=236284 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) 0x22b52c80
2014-06-12 14:46:35.692803 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2770 crt=0'0 lcod 173'2769 mlcod 173'2769 active+clean] repop_all_committed: repop tid 236284 all committed 
2014-06-12 14:46:35.692815 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2769 active+clean] eval_repop repgather(0x12ae3bc0 173'2771 rep_tid=236284 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.692833 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2769 active+clean] log_op_stats osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.015406 lat 0.015437
2014-06-12 14:46:35.692852 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2769 active+clean] publish_stats_to_osd 173:3458
2014-06-12 14:46:35.692866 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2769 active+clean]  sending commit on repgather(0x12ae3bc0 173'2771 rep_tid=236284 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) 0x1080cf00
2014-06-12 14:46:35.692906 7f9d623d5700  5 -- op tracker -- , seq: 747905, time: 2014-06-12 14:46:35.692895, event: commit_sent, op: osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.692916 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2769 active+clean]  requeue_ops 
2014-06-12 14:46:35.692928 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean]  removing repgather(0x12ae3bc0 173'2771 rep_tid=236284 committed?=1 applied?=1 op=osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.692945 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean]    q front is repgather(0x12ae3bc0 173'2771 rep_tid=236284 committed?=1 applied?=1 op=osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.692961 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] remove_repop repgather(0x12ae3bc0 173'2771 rep_tid=236284 committed?=1 applied?=1 op=osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.692977 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean]  requeue_ops 
2014-06-12 14:46:35.692997 7f9d623d5700  5 -- op tracker -- , seq: 747905, time: 2014-06-12 14:46:35.692992, event: done, op: osd_op(client.4289.0:143703 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 2019328~1032192] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.693017 7f9d623d5700 10 osd.2 173 dequeue_op 0x2fa9c300 finish
2014-06-12 14:46:35.693027 7f9d623d5700  5 -- op tracker -- , seq: 747908, time: 2014-06-12 14:46:35.693023, event: done, op: osd_sub_op_reply(client.4289.0:143703 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.693039 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.693044 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.693792 7f9d7a161700 10 _do_op 0x614d080 seq 2487 r = 0, finisher 0x69e6a00 0x38996fc0
2014-06-12 14:46:35.693802 7f9d7a161700 10 _finish_op 0x614d080 seq 2487 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:35.693810 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:35.693816 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.693808 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] op_commit: 236285
2014-06-12 14:46:35.693831 7f9d7715b700  5 -- op tracker -- , seq: 747907, time: 2014-06-12 14:46:35.693823, event: op_commit, op: osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.693849 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] op_applied: 236285
2014-06-12 14:46:35.693875 7f9d7795c700  5 -- op tracker -- , seq: 747907, time: 2014-06-12 14:46:35.693867, event: op_applied, op: osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.693884 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] op_applied on primary on version 173'2772
2014-06-12 14:46:35.697515 7f9d48d28700  5 -- op tracker -- , seq: 747909, time: 2014-06-12 14:46:35.695970, event: header_read, op: osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.697541 7f9d48d28700  5 -- op tracker -- , seq: 747909, time: 2014-06-12 14:46:35.695973, event: throttled, op: osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.697557 7f9d48d28700  5 -- op tracker -- , seq: 747909, time: 2014-06-12 14:46:35.697456, event: all_read, op: osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.697565 7f9d48d28700  5 -- op tracker -- , seq: 747909, time: 0.000000, event: dispatched, op: osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.697573 7f9d48d28700 20 osd.2 173 should_share_map client.4289 172.16.96.220:0/1241619 173
2014-06-12 14:46:35.697600 7f9d48d28700 15 osd.2 173 enqueue_op 0x12bf6000 prio 63 cost 626688 latency 0.001629 osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:35.697652 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.697664 7f9d68be2700 10 osd.2 173 dequeue_op 0x12bf6000 prio 63 cost 626688 latency 0.001693 osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean]
2014-06-12 14:46:35.697701 7f9d68be2700  5 -- op tracker -- , seq: 747909, time: 2014-06-12 14:46:35.697695, event: reached_pg, op: osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.697724 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:35.697740 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] handle_message: 0x12bf6000
2014-06-12 14:46:35.697751 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] do_op osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:35.697777 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] get_object_context: found obc in cache: 0x2fc00dc0
2014-06-12 14:46:35.697791 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] get_object_context: 0x2fc00dc0 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 rwstate(write n=1 w=0) oi: 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3(173'2772 client.4289.0:143704 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2772) ssc: 0x82de380 snapset: 0=[]:[]+head
2014-06-12 14:46:35.697811 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] find_object_context 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 @head oi=1263d312/rbd_data.10986b8b4567.0000000000001139/head//3(173'2772 client.4289.0:143704 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2772)
2014-06-12 14:46:35.697840 7f9d68be2700  5 -- op tracker -- , seq: 747909, time: 2014-06-12 14:46:35.697833, event: started, op: osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.697856 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] execute_ctx 0x589d000
2014-06-12 14:46:35.697882 7f9d68be2700  5 -- op tracker -- , seq: 747909, time: 2014-06-12 14:46:35.697876, event: started, op: osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.697889 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] do_op 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] ov 173'2772 av 173'2773 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:35.697903 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] do_osd_op 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688]
2014-06-12 14:46:35.697914 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:35.697928 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] do_osd_op  write 3567616~626688
2014-06-12 14:46:35.697946 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] do_osd_op_effects on session 0x1b56a00
2014-06-12 14:46:35.697958 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] make_writeable 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 snapset=0x82de3b8  snapc=0=[]
2014-06-12 14:46:35.697972 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] make_writeable 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:35.697997 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] finish_ctx 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 0x589d000 op modify  
2014-06-12 14:46:35.698010 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean]  set mtime to 2014-06-12 08:45:38.075461
2014-06-12 14:46:35.698035 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean]  final snapset 0=[]:[]+head in 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.698058 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean]  zeroing write result code 0
2014-06-12 14:46:35.698071 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] new_repop rep_tid 236286 on osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:35.698089 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] issue_repop rep_tid 236286 o 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.698118 7f9d68be2700  5 -- op tracker -- , seq: 747909, time: 2014-06-12 14:46:35.698112, event: waiting for subops from 27, op: osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.698141 7f9d68be2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:35.698166 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2772 (0'0,173'2772] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] append_log log((0'0,173'2772], crt=0'0) [173'2773 (173'2772) modify   1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 by client.4289.0:143705 2014-06-12 08:45:38.075461]
2014-06-12 14:46:35.698192 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] add_log_entry 173'2773 (173'2772) modify   1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 by client.4289.0:143705 2014-06-12 08:45:38.075461
2014-06-12 14:46:35.698219 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] append_log  adding 1 keys
2014-06-12 14:46:35.698257 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2773, trimmed: 
2014-06-12 14:46:35.698283 7f9d68be2700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:35.698290 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.698292 7f9d68be2700  5 queue_op 0x830cd20 seq 2488 osr(3.312 0x276d540) 628981 bytes   (queue has 1 ops and 628981 bytes)
2014-06-12 14:46:35.698300 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] eval_repop repgather(0x1f066cc0 173'2773 rep_tid=236286 committed?=0 applied?=0 lock=0 op=osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.698321 7f9d68be2700 10 osd.2 173 dequeue_op 0x12bf6000 finish
2014-06-12 14:46:35.698325 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.698328 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.698336 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (1 active)
2014-06-12 14:46:35.698341 7f9d7f16b700  5 _do_op 0x830cd20 seq 2488 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:35.698346 7f9d7f16b700 10 _do_transaction on 0x15aced80
2014-06-12 14:46:35.698353 7f9d7f16b700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.698496 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.698525 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.698555 7f9d58c5a700  5 -- op tracker -- , seq: 747910, time: 2014-06-12 14:46:35.698390, event: header_read, op: osd_sub_op_reply(client.4289.0:143704 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.698567 7f9d58c5a700  5 -- op tracker -- , seq: 747910, time: 2014-06-12 14:46:35.698392, event: throttled, op: osd_sub_op_reply(client.4289.0:143704 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.698582 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.698605 7f9d7f16b700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.698613 7f9d7f16b700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:35.698620 7f9d7f16b700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:35.698637 7f9d7f16b700 15 _write 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 3567616~626688
2014-06-12 14:46:35.698574 7f9d58c5a700  5 -- op tracker -- , seq: 747910, time: 2014-06-12 14:46:35.698401, event: all_read, op: osd_sub_op_reply(client.4289.0:143704 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.698669 7f9d58c5a700  5 -- op tracker -- , seq: 747910, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143704 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.698673 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143704 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:35.698678 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:35.698684 7f9d58c5a700 15 osd.2 173 enqueue_op 0x28dd0200 prio 196 cost 0 latency 0.000294 osd_sub_op_reply(client.4289.0:143704 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:35.698704 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.698712 7f9d643d9700 10 osd.2 173 dequeue_op 0x28dd0200 prio 196 cost 0 latency 0.000322 osd_sub_op_reply(client.4289.0:143704 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean]
2014-06-12 14:46:35.698752 7f9d643d9700  5 -- op tracker -- , seq: 747910, time: 2014-06-12 14:46:35.698747, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143704 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.698761 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] handle_message: 0x28dd0200
2014-06-12 14:46:35.698780 7f9d643d9700  5 -- op tracker -- , seq: 747910, time: 2014-06-12 14:46:35.698777, event: started, op: osd_sub_op_reply(client.4289.0:143704 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.698787 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] sub_op_modify_reply: tid 236285 op  ack_type 4 from 27
2014-06-12 14:46:35.698813 7f9d643d9700  5 -- op tracker -- , seq: 747907, time: 2014-06-12 14:46:35.698805, event: sub_op_commit_rec, op: osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.698832 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] repop_all_applied: repop tid 236285 all applied 
2014-06-12 14:46:35.698848 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] eval_repop repgather(0x12ae2240 173'2772 rep_tid=236285 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.698869 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean]  sending ack on repgather(0x12ae2240 173'2772 rep_tid=236285 committed?=0 applied?=1 lock=0 op=osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) 0x11666280
2014-06-12 14:46:35.698902 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2771 crt=0'0 lcod 173'2770 mlcod 173'2770 active+clean] repop_all_committed: repop tid 236285 all committed 
2014-06-12 14:46:35.698913 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2770 active+clean] eval_repop repgather(0x12ae2240 173'2772 rep_tid=236285 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.698932 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2770 active+clean] log_op_stats osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.016095 lat 0.016125
2014-06-12 14:46:35.698951 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2770 active+clean] publish_stats_to_osd 173:3459
2014-06-12 14:46:35.698967 7f9d7f16b700 10 lookup_strip_header done  cid 3.312_head oid 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.698967 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2770 active+clean]  sending commit on repgather(0x12ae2240 173'2772 rep_tid=236285 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) 0xfdb2a00
2014-06-12 14:46:35.699016 7f9d643d9700  5 -- op tracker -- , seq: 747907, time: 2014-06-12 14:46:35.699011, event: commit_sent, op: osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.699025 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2770 active+clean]  requeue_ops 
2014-06-12 14:46:35.699037 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2771 active+clean]  removing repgather(0x12ae2240 173'2772 rep_tid=236285 committed?=1 applied?=1 op=osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.699055 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2771 active+clean]    q front is repgather(0x12ae2240 173'2772 rep_tid=236285 committed?=1 applied?=1 op=osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.699072 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2771 active+clean] remove_repop repgather(0x12ae2240 173'2772 rep_tid=236285 committed?=1 applied?=1 op=osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.699100 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2771 active+clean]  requeue_ops 
2014-06-12 14:46:35.699123 7f9d643d9700  5 -- op tracker -- , seq: 747907, time: 2014-06-12 14:46:35.699115, event: done, op: osd_op(client.4289.0:143704 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~516096] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.699141 7f9d643d9700 10 osd.2 173 dequeue_op 0x28dd0200 finish
2014-06-12 14:46:35.699154 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:35.699152 7f9d643d9700  5 -- op tracker -- , seq: 747910, time: 2014-06-12 14:46:35.699147, event: done, op: osd_sub_op_reply(client.4289.0:143704 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.699165 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.699169 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.701343 7f9d7f16b700 10 _generic_write 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 3567616~626688 = 0
2014-06-12 14:46:35.701421 7f9d7f16b700 15 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.701449 7f9d7f16b700 10 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 = 0
2014-06-12 14:46:35.701459 7f9d7f16b700 15 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3
2014-06-12 14:46:35.701469 7f9d7f16b700 10 _setattrs 3.312_head/1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 = 0
2014-06-12 14:46:35.701712 7f9d7f16b700 20 set_header setting 17458 cid 3.312_head oid 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 parent seq 0
2014-06-12 14:46:35.701812 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.701829 7f9d7f16b700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:35.701842 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:35.704783 7f9d58c5a700  5 -- op tracker -- , seq: 747911, time: 2014-06-12 14:46:35.704718, event: header_read, op: osd_sub_op_reply(client.4289.0:143705 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.704800 7f9d58c5a700  5 -- op tracker -- , seq: 747911, time: 2014-06-12 14:46:35.704720, event: throttled, op: osd_sub_op_reply(client.4289.0:143705 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.704807 7f9d58c5a700  5 -- op tracker -- , seq: 747911, time: 2014-06-12 14:46:35.704734, event: all_read, op: osd_sub_op_reply(client.4289.0:143705 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.704814 7f9d58c5a700  5 -- op tracker -- , seq: 747911, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4289.0:143705 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.704820 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4289.0:143705 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:35.704827 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:35.704837 7f9d58c5a700 15 osd.2 173 enqueue_op 0x28dd1c00 prio 196 cost 0 latency 0.000119 osd_sub_op_reply(client.4289.0:143705 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:35.704872 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.704881 7f9d633d7700 10 osd.2 173 dequeue_op 0x28dd1c00 prio 196 cost 0 latency 0.000163 osd_sub_op_reply(client.4289.0:143705 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2771 active+clean]
2014-06-12 14:46:35.704904 7f9d633d7700  5 -- op tracker -- , seq: 747911, time: 2014-06-12 14:46:35.704900, event: reached_pg, op: osd_sub_op_reply(client.4289.0:143705 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.704912 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2771 active+clean] handle_message: 0x28dd1c00
2014-06-12 14:46:35.704931 7f9d633d7700  5 -- op tracker -- , seq: 747911, time: 2014-06-12 14:46:35.704927, event: started, op: osd_sub_op_reply(client.4289.0:143705 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.704937 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2771 active+clean] sub_op_modify_reply: tid 236286 op  ack_type 4 from 27
2014-06-12 14:46:35.704963 7f9d633d7700  5 -- op tracker -- , seq: 747909, time: 2014-06-12 14:46:35.704953, event: sub_op_commit_rec, op: osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.704973 7f9d633d7700 10 osd.2 173 dequeue_op 0x28dd1c00 finish
2014-06-12 14:46:35.704981 7f9d633d7700  5 -- op tracker -- , seq: 747911, time: 2014-06-12 14:46:35.704977, event: done, op: osd_sub_op_reply(client.4289.0:143705 3.312 1263d312/rbd_data.10986b8b4567.0000000000001139/head//3 [] ondisk, result = 0)
2014-06-12 14:46:35.705005 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.705009 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.705078 7f9d7f16b700 10 _do_op 0x830cd20 seq 2488 r = 0, finisher 0x73c1960 0x352bc000
2014-06-12 14:46:35.705094 7f9d7f16b700 10 _finish_op 0x830cd20 seq 2488 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:35.705106 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:35.705116 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:35.705101 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2771 active+clean] op_commit: 236286
2014-06-12 14:46:35.705125 7f9d7715b700  5 -- op tracker -- , seq: 747909, time: 2014-06-12 14:46:35.705118, event: op_commit, op: osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.705137 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2772 crt=0'0 lcod 173'2771 mlcod 173'2771 active+clean] repop_all_committed: repop tid 236286 all committed 
2014-06-12 14:46:35.705155 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2771 active+clean] eval_repop repgather(0x1f066cc0 173'2773 rep_tid=236286 committed?=1 applied?=0 lock=0 op=osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.705173 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2771 active+clean] log_op_stats osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 inb 626967 outb 0 rlat 0.000000 lat 0.009202
2014-06-12 14:46:35.705192 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2771 active+clean] publish_stats_to_osd 173:3460
2014-06-12 14:46:35.705204 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2771 active+clean]  sending commit on repgather(0x1f066cc0 173'2773 rep_tid=236286 committed?=1 applied?=0 lock=0 op=osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) 0x15ce3680
2014-06-12 14:46:35.705241 7f9d7715b700  5 -- op tracker -- , seq: 747909, time: 2014-06-12 14:46:35.705235, event: commit_sent, op: osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.705267 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2771 active+clean] op_applied: 236286
2014-06-12 14:46:35.705289 7f9d7795c700  5 -- op tracker -- , seq: 747909, time: 2014-06-12 14:46:35.705282, event: op_applied, op: osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.705298 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2771 active+clean] op_applied on primary on version 173'2773
2014-06-12 14:46:35.705311 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2771 active+clean] repop_all_applied: repop tid 236286 all applied 
2014-06-12 14:46:35.705322 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2771 active+clean] eval_repop repgather(0x1f066cc0 173'2773 rep_tid=236286 committed?=1 applied?=1 lock=0 op=osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:35.705339 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2771 active+clean] log_op_stats osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4 inb 626967 outb 0 rlat 0.000000 lat 0.009368
2014-06-12 14:46:35.705356 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2771 active+clean] publish_stats_to_osd 173:3461
2014-06-12 14:46:35.705369 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2771 active+clean]  requeue_ops 
2014-06-12 14:46:35.705379 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean]  removing repgather(0x1f066cc0 173'2773 rep_tid=236286 committed?=1 applied?=1 op=osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.705395 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean]    q front is repgather(0x1f066cc0 173'2773 rep_tid=236286 committed?=1 applied?=1 op=osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.705412 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] remove_repop repgather(0x1f066cc0 173'2773 rep_tid=236286 committed?=1 applied?=1 op=osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:35.705428 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean]  requeue_ops 
2014-06-12 14:46:35.705454 7f9d7795c700  5 -- op tracker -- , seq: 747909, time: 2014-06-12 14:46:35.705450, event: done, op: osd_op(client.4289.0:143705 rbd_data.10986b8b4567.0000000000001139 [set-alloc-hint object_size 4194304 write_size 4194304,write 3567616~626688] 3.1263d312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:35.987873 7f9d5a16f700  5 -- op tracker -- , seq: 747912, time: 2014-06-12 14:46:35.986625, event: header_read, op: osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.987887 7f9d5a16f700  5 -- op tracker -- , seq: 747912, time: 2014-06-12 14:46:35.986628, event: throttled, op: osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.987895 7f9d5a16f700  5 -- op tracker -- , seq: 747912, time: 2014-06-12 14:46:35.987830, event: all_read, op: osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.987903 7f9d5a16f700  5 -- op tracker -- , seq: 747912, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.987909 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:35.987916 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:35.987925 7f9d5a16f700 15 osd.2 173 enqueue_op 0x55c3f00 prio 127 cost 516903 latency 0.001299 osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.987965 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:35.987973 7f9d61bd4700 10 osd.2 173 dequeue_op 0x55c3f00 prio 127 cost 516903 latency 0.001347 osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2667 (0'0,173'2667] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2666 active]
2014-06-12 14:46:35.987996 7f9d61bd4700  5 -- op tracker -- , seq: 747912, time: 2014-06-12 14:46:35.987991, event: reached_pg, op: osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.988005 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2667 (0'0,173'2667] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2666 active] handle_message: 0x55c3f00
2014-06-12 14:46:35.988017 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2667 (0'0,173'2667] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2666 active] sub_op_modify trans f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 v 173'2668 (transaction) 174
2014-06-12 14:46:35.988041 7f9d61bd4700  5 -- op tracker -- , seq: 747912, time: 2014-06-12 14:46:35.988037, event: started, op: osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.988057 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2667 (0'0,173'2667] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2666 active] append_log log((0'0,173'2667], crt=0'0) [173'2668 (173'2666) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146410 2014-06-12 08:41:30.058947]
2014-06-12 14:46:35.988081 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2668 (0'0,173'2668] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2666 active] add_log_entry 173'2668 (173'2666) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146410 2014-06-12 08:41:30.058947
2014-06-12 14:46:35.988102 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2668 (0'0,173'2668] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2666 active] append_log  adding 1 keys
2014-06-12 14:46:35.988137 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2668, trimmed: 
2014-06-12 14:46:35.988161 7f9d61bd4700  5 -- op tracker -- , seq: 747912, time: 2014-06-12 14:46:35.988156, event: started, op: osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.988171 7f9d61bd4700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:35.988176 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:35.988177 7f9d61bd4700  5 queue_op 0x2847aa20 seq 2478 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:35.988188 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:46:35.988194 7f9d7e169700  5 _do_op 0x2847aa20 seq 2478 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:35.988185 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2668 (0'0,173'2668] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2666 active] do_sub_op osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.988198 7f9d61bd4700 10 osd.2 173 dequeue_op 0x55c3f00 finish
2014-06-12 14:46:35.988199 7f9d7e169700 10 _do_transaction on 0x5b24630
2014-06-12 14:46:35.988202 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:35.988204 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:35.988206 7f9d7e169700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:35.988340 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:35.988363 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:35.988405 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:35.988421 7f9d7e169700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:35.988424 7f9d7e169700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:35.988428 7f9d7e169700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:35.988439 7f9d7e169700 15 _write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 2875392~516096
2014-06-12 14:46:35.988766 7f9d7e169700 10 lookup_strip_header done  cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:35.988916 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:35.990590 7f9d7e169700 10 _generic_write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 2875392~516096 = 0
2014-06-12 14:46:35.990645 7f9d7e169700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:35.990661 7f9d7e169700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:35.990667 7f9d7e169700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:35.990675 7f9d7e169700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:35.990884 7f9d7e169700 20 set_header setting 16921 cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 parent seq 0
2014-06-12 14:46:35.990975 7f9d7e169700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:35.990993 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:35.991004 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:35.993460 7f9d7e169700 10 _do_op 0x2847aa20 seq 2478 r = 0, finisher 0x7503860 0
2014-06-12 14:46:35.993506 7f9d7715b700  5 -- op tracker -- , seq: 747912, time: 2014-06-12 14:46:35.993496, event: commit_sent, op: osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.993518 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2668 (0'0,173'2668] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2666 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:35.993542 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:35.993566 7f9d7e169700 10 _finish_op 0x2847aa20 seq 2478 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:35.993588 7f9d7795c700  5 -- op tracker -- , seq: 747912, time: 2014-06-12 14:46:35.993583, event: sub_op_applied, op: osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.993595 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2668 (0'0,173'2668] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2667 active] sub_op_modify_applied on 0x5b24540 op osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:35.993610 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2668 (0'0,173'2668] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2667 active] op_applied on primary on version 173'2668
2014-06-12 14:46:35.993640 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2668 (0'0,173'2668] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2667 active] op_applied on replica on version 173'2668
2014-06-12 14:46:35.993678 7f9d7e169700  5 -- op tracker -- , seq: 747912, time: 2014-06-12 14:46:35.993666, event: done, op: osd_sub_op(client.4316.0:146410 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2668 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:35.993742 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:46:35.993748 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:36.147626 7f9d5a16f700  5 -- op tracker -- , seq: 747913, time: 2014-06-12 14:46:36.147160, event: header_read, op: osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.147646 7f9d5a16f700  5 -- op tracker -- , seq: 747913, time: 2014-06-12 14:46:36.147162, event: throttled, op: osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.147660 7f9d5a16f700  5 -- op tracker -- , seq: 747913, time: 2014-06-12 14:46:36.147573, event: all_read, op: osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.147665 7f9d5a16f700  5 -- op tracker -- , seq: 747913, time: 0.000000, event: dispatched, op: osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.147674 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:36.147683 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:36.147695 7f9d5a16f700 15 osd.2 173 enqueue_op 0x28dd1e00 prio 127 cost 123687 latency 0.000535 osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:36.147742 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:36.147754 7f9d673df700 10 osd.2 173 dequeue_op 0x28dd1e00 prio 127 cost 123687 latency 0.000593 osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2719 (0'0,173'2719] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active]
2014-06-12 14:46:36.147781 7f9d673df700  5 -- op tracker -- , seq: 747913, time: 2014-06-12 14:46:36.147778, event: reached_pg, op: osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.147789 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2719 (0'0,173'2719] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] handle_message: 0x28dd1e00
2014-06-12 14:46:36.147797 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2719 (0'0,173'2719] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] sub_op_modify trans cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 v 173'2720 (transaction) 174
2014-06-12 14:46:36.147808 7f9d673df700  5 -- op tracker -- , seq: 747913, time: 2014-06-12 14:46:36.147805, event: started, op: osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.147824 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2719 (0'0,173'2719] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] append_log log((0'0,173'2719], crt=0'0) [173'2720 (173'804) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143142 2014-06-12 08:43:30.466750]
2014-06-12 14:46:36.147843 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2720 (0'0,173'2720] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] add_log_entry 173'2720 (173'804) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143142 2014-06-12 08:43:30.466750
2014-06-12 14:46:36.147860 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2720 (0'0,173'2720] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] append_log  adding 1 keys
2014-06-12 14:46:36.147886 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2720, trimmed: 
2014-06-12 14:46:36.147903 7f9d673df700  5 -- op tracker -- , seq: 747913, time: 2014-06-12 14:46:36.147900, event: started, op: osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.147912 7f9d673df700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:36.147915 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:36.147916 7f9d673df700  5 queue_op 0x142a8480 seq 2467 osr(3.21c 0x276d020) 125173 bytes   (queue has 1 ops and 125173 bytes)
2014-06-12 14:46:36.147926 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (1 active)
2014-06-12 14:46:36.147922 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2720 (0'0,173'2720] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] do_sub_op osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:36.147935 7f9d673df700 10 osd.2 173 dequeue_op 0x28dd1e00 finish
2014-06-12 14:46:36.147935 7f9d7815d700  5 _do_op 0x142a8480 seq 2467 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:46:36.147937 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:36.147939 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:36.147940 7f9d7815d700 10 _do_transaction on 0x65ccb70
2014-06-12 14:46:36.147947 7f9d7815d700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:36.148159 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:36.148187 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:36.148244 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:36.148350 7f9d7815d700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:36.148358 7f9d7815d700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:46:36.148364 7f9d7815d700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:36.148381 7f9d7815d700 15 _write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 0~122880
2014-06-12 14:46:36.148570 7f9d7815d700 10 lookup_strip_header done  cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:36.148717 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:36.149184 7f9d7815d700 10 _generic_write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 0~122880 = 0
2014-06-12 14:46:36.149209 7f9d7815d700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:36.149223 7f9d7815d700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:36.149228 7f9d7815d700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:36.149235 7f9d7815d700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:36.149360 7f9d7815d700 20 set_header setting 16774 cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 parent seq 0
2014-06-12 14:46:36.149418 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:36.149430 7f9d7815d700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:46:36.149439 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:36.149984 7f9d7815d700 10 _do_op 0x142a8480 seq 2467 r = 0, finisher 0x87fa8a0 0
2014-06-12 14:46:36.149996 7f9d7815d700 10 _finish_op 0x142a8480 seq 2467 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:36.150002 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (0 active)
2014-06-12 14:46:36.150005 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:36.150023 7f9d7715b700  5 -- op tracker -- , seq: 747913, time: 2014-06-12 14:46:36.150016, event: commit_sent, op: osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.150033 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2720 (0'0,173'2720] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] sub_op_modify_commit on op osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:36.150056 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:36.150087 7f9d7795c700  5 -- op tracker -- , seq: 747913, time: 2014-06-12 14:46:36.150083, event: sub_op_applied, op: osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.150096 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2720 (0'0,173'2720] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] sub_op_modify_applied on 0x65cca80 op osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:36.150112 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2720 (0'0,173'2720] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] op_applied on primary on version 173'2720
2014-06-12 14:46:36.150123 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2720 (0'0,173'2720] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] op_applied on replica on version 173'2720
2014-06-12 14:46:36.150145 7f9d7795c700  5 -- op tracker -- , seq: 747913, time: 2014-06-12 14:46:36.150141, event: done, op: osd_sub_op(client.4279.0:143142 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.154277 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:36.321707 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:36.321776 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:36.367937 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:36.477311 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:36.477334 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:36.477337 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:36.477339 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:36.477338 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:36.477341 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 4; oldest is 3.315997 seconds old
2014-06-12 14:46:36.510048 7f9d5a16f700  5 -- op tracker -- , seq: 747914, time: 2014-06-12 14:46:36.509045, event: header_read, op: osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.510063 7f9d5a16f700  5 -- op tracker -- , seq: 747914, time: 2014-06-12 14:46:36.509048, event: throttled, op: osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.510073 7f9d5a16f700  5 -- op tracker -- , seq: 747914, time: 2014-06-12 14:46:36.510009, event: all_read, op: osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.510080 7f9d5a16f700  5 -- op tracker -- , seq: 747914, time: 0.000000, event: dispatched, op: osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.510086 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:36.510093 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:36.510102 7f9d5a16f700 15 osd.2 173 enqueue_op 0x12bf6300 prio 127 cost 451367 latency 0.001056 osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:36.513404 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:36.513423 7f9d67be0700 10 osd.2 173 dequeue_op 0x12bf6300 prio 127 cost 451367 latency 0.004378 osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4e0( v 173'2635 (0'0,173'2635] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2634 active]
2014-06-12 14:46:36.513454 7f9d67be0700  5 -- op tracker -- , seq: 747914, time: 2014-06-12 14:46:36.513449, event: reached_pg, op: osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.513466 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2635 (0'0,173'2635] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2634 active] handle_message: 0x12bf6300
2014-06-12 14:46:36.513478 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2635 (0'0,173'2635] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2634 active] sub_op_modify trans 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 v 173'2636 (transaction) 174
2014-06-12 14:46:36.513495 7f9d67be0700  5 -- op tracker -- , seq: 747914, time: 2014-06-12 14:46:36.513491, event: started, op: osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.513514 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2635 (0'0,173'2635] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2634 active] append_log log((0'0,173'2635], crt=0'0) [173'2636 (173'728) modify   79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 by client.4293.0:143960 2014-06-12 08:43:56.445789]
2014-06-12 14:46:36.513539 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2636 (0'0,173'2636] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2634 active] add_log_entry 173'2636 (173'728) modify   79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 by client.4293.0:143960 2014-06-12 08:43:56.445789
2014-06-12 14:46:36.513563 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2636 (0'0,173'2636] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2634 active] append_log  adding 1 keys
2014-06-12 14:46:36.513598 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2636, trimmed: 
2014-06-12 14:46:36.513637 7f9d67be0700  5 -- op tracker -- , seq: 747914, time: 2014-06-12 14:46:36.513632, event: started, op: osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.513649 7f9d67be0700  5 queue_transactions existing osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:36.513653 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:36.513655 7f9d67be0700  5 queue_op 0x91abf20 seq 2436 osr(3.4e0 0x276dbd0) 452853 bytes   (queue has 1 ops and 452853 bytes)
2014-06-12 14:46:36.513665 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5180900 (1 active)
2014-06-12 14:46:36.513673 7f9d82171700  5 _do_op 0x91abf20 seq 2436 osr(3.4e0 0x276dbd0)/0x276dbd0 start
2014-06-12 14:46:36.513663 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2636 (0'0,173'2636] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2634 active] do_sub_op osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:36.513678 7f9d82171700 10 _do_transaction on 0x5b7ad30
2014-06-12 14:46:36.513677 7f9d67be0700 10 osd.2 173 dequeue_op 0x12bf6300 finish
2014-06-12 14:46:36.513681 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:36.513684 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:36.513683 7f9d82171700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:36.513804 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:36.513822 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:36.513869 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:36.513887 7f9d82171700 15 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:36.513892 7f9d82171700 10 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1 r = 0
2014-06-12 14:46:36.513898 7f9d82171700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:36.513920 7f9d82171700 15 _write 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 0~450560
2014-06-12 14:46:36.514152 7f9d82171700 10 lookup_strip_header done  cid 3.4e0_head oid 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3
2014-06-12 14:46:36.514171 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:36.515710 7f9d82171700 10 _generic_write 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 0~450560 = 0
2014-06-12 14:46:36.515748 7f9d82171700 15 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3
2014-06-12 14:46:36.515759 7f9d82171700 10 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:36.515768 7f9d82171700 15 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3
2014-06-12 14:46:36.515773 7f9d82171700 10 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:36.515902 7f9d82171700 20 set_header setting 17854 cid 3.4e0_head oid 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:36.515945 7f9d82171700 20 set_header setting 2236 cid meta oid 1fe07435/pglog_3.4e0/0//-1 parent seq 0
2014-06-12 14:46:36.515956 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:36.515962 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:36.517963 7f9d82171700 10 _do_op 0x91abf20 seq 2436 r = 0, finisher 0x72291e0 0
2014-06-12 14:46:36.517981 7f9d82171700 10 _finish_op 0x91abf20 seq 2436 osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:36.517986 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5180900 (0 active)
2014-06-12 14:46:36.517989 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:36.518006 7f9d7715b700  5 -- op tracker -- , seq: 747914, time: 2014-06-12 14:46:36.517996, event: commit_sent, op: osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.518049 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2636 (0'0,173'2636] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2634 active] sub_op_modify_commit on op osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:36.518083 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:36.518155 7f9d7795c700  5 -- op tracker -- , seq: 747914, time: 2014-06-12 14:46:36.518149, event: sub_op_applied, op: osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.518164 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2636 (0'0,173'2636] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2635 active] sub_op_modify_applied on 0x5b7ac40 op osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:36.518180 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2636 (0'0,173'2636] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2635 active] op_applied on primary on version 173'2636
2014-06-12 14:46:36.518190 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2636 (0'0,173'2636] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2635 active] op_applied on replica on version 173'2636
2014-06-12 14:46:36.518209 7f9d7795c700  5 -- op tracker -- , seq: 747914, time: 2014-06-12 14:46:36.518204, event: done, op: osd_sub_op(client.4293.0:143960 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2636 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.518706 7f9d5a16f700  5 -- op tracker -- , seq: 747915, time: 2014-06-12 14:46:36.514459, event: header_read, op: osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.518719 7f9d5a16f700  5 -- op tracker -- , seq: 747915, time: 2014-06-12 14:46:36.514461, event: throttled, op: osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.518727 7f9d5a16f700  5 -- op tracker -- , seq: 747915, time: 2014-06-12 14:46:36.518680, event: all_read, op: osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.518735 7f9d5a16f700  5 -- op tracker -- , seq: 747915, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.518739 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:36.518745 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:36.518751 7f9d5a16f700 15 osd.2 173 enqueue_op 0x12bf7f00 prio 127 cost 516903 latency 0.004292 osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:36.518814 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:36.518824 7f9d663dd700 10 osd.2 173 dequeue_op 0x12bf7f00 prio 127 cost 516903 latency 0.004364 osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2466 (0'0,173'2466] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active]
2014-06-12 14:46:36.518846 7f9d663dd700  5 -- op tracker -- , seq: 747915, time: 2014-06-12 14:46:36.518842, event: reached_pg, op: osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.518854 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2466 (0'0,173'2466] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active] handle_message: 0x12bf7f00
2014-06-12 14:46:36.518866 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2466 (0'0,173'2466] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active] sub_op_modify trans 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 v 173'2467 (transaction) 174
2014-06-12 14:46:36.518882 7f9d663dd700  5 -- op tracker -- , seq: 747915, time: 2014-06-12 14:46:36.518878, event: started, op: osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.518903 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2466 (0'0,173'2466] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active] append_log log((0'0,173'2466], crt=0'0) [173'2467 (173'2466) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143626 2014-06-12 08:43:17.714204]
2014-06-12 14:46:36.518926 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2467 (0'0,173'2467] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active] add_log_entry 173'2467 (173'2466) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143626 2014-06-12 08:43:17.714204
2014-06-12 14:46:36.518947 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2467 (0'0,173'2467] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active] append_log  adding 1 keys
2014-06-12 14:46:36.518979 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2467, trimmed: 
2014-06-12 14:46:36.519002 7f9d663dd700  5 -- op tracker -- , seq: 747915, time: 2014-06-12 14:46:36.518997, event: started, op: osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.519011 7f9d663dd700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:36.519015 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:36.519016 7f9d663dd700  5 queue_op 0x177b41e0 seq 2300 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:36.519023 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2467 (0'0,173'2467] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active] do_sub_op osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:36.519037 7f9d663dd700 10 osd.2 173 dequeue_op 0x12bf7f00 finish
2014-06-12 14:46:36.519041 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:36.519043 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:36.519487 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:46:36.519504 7f9d7e96a700  5 _do_op 0x177b41e0 seq 2300 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:46:36.519524 7f9d7e96a700 10 _do_transaction on 0x82229b0
2014-06-12 14:46:36.519532 7f9d7e96a700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:36.519778 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:36.519815 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:36.519854 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:36.519873 7f9d7e96a700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:36.519878 7f9d7e96a700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:46:36.519882 7f9d7e96a700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:36.519892 7f9d7e96a700 15 _write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 557056~516096
2014-06-12 14:46:36.520021 7f9d7e96a700 10 lookup_strip_header done  cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:36.520234 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:36.521561 7f9d7e96a700 10 _generic_write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 557056~516096 = 0
2014-06-12 14:46:36.521609 7f9d7e96a700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:36.521652 7f9d7e96a700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:36.521661 7f9d7e96a700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:36.521670 7f9d7e96a700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:36.521892 7f9d7e96a700 20 set_header setting 17388 cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 parent seq 0
2014-06-12 14:46:36.521970 7f9d7e96a700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:46:36.521986 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:36.521993 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:36.523961 7f9d7e96a700 10 _do_op 0x177b41e0 seq 2300 r = 0, finisher 0x6ce6ba0 0
2014-06-12 14:46:36.523981 7f9d7e96a700 10 _finish_op 0x177b41e0 seq 2300 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:36.523986 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:46:36.523996 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:36.524021 7f9d7795c700  5 -- op tracker -- , seq: 747915, time: 2014-06-12 14:46:36.524014, event: sub_op_applied, op: osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.524030 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2467 (0'0,173'2467] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active] sub_op_modify_applied on 0x82228c0 op osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:36.524054 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:36.524070 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2467 (0'0,173'2467] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active] op_applied on primary on version 173'2467
2014-06-12 14:46:36.524078 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2467 (0'0,173'2467] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active] op_applied on replica on version 173'2467
2014-06-12 14:46:36.524101 7f9d7715b700  5 -- op tracker -- , seq: 747915, time: 2014-06-12 14:46:36.524094, event: commit_sent, op: osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.524110 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2467 (0'0,173'2467] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2465 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:36.524142 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:36.524199 7f9d7715b700  5 -- op tracker -- , seq: 747915, time: 2014-06-12 14:46:36.524194, event: done, op: osd_sub_op(client.4294.0:143626 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2467 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:36.741557 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:36.741735 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:36.921068 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:36.921119 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:36.943200 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:36.943270 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:36.966713 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:36.966754 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:37.206205 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:37.206268 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:37.207592 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.250601 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.277192 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.297220 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.319572 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.332775 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.335845 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.341035 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.347625 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.348760 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.397679 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:37.397705 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:37.421514 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.428369 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.477408 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:37.477427 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:46:37.477430 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:46:37.477432 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:46:37.477435 7f9d84175700 10 osd.2 173 send_pg_stats - 2 pgs updated
2014-06-12 14:46:37.477532 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 11.37 >= max 0.5 = no, load too high
2014-06-12 14:46:37.477550 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:37.477553 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92645 < max (604800 seconds)
2014-06-12 14:46:37.477565 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:37.477568 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:37.477570 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:37.477572 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 4; oldest is 4.316228 seconds old
2014-06-12 14:46:37.477586 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:37.586339 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.601635 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.623870 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:37.623877 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:37.623879 7f9d74bfa700 20 osd.2 173 _dispatch 0x65f8380 pg_stats_ack(2 pgs tid 4853) v1
2014-06-12 14:46:37.623882 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:46:37.623887 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:37.623889 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:37.651892 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.652214 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.655882 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.662486 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.663411 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.666282 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.670702 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.673529 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.680410 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.681019 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.685019 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.689311 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.693136 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.693873 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.698418 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.699257 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.705073 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.705206 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.718743 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:37.718998 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:37.795808 7f9d5b6bc700  5 -- op tracker -- , seq: 747916, time: 2014-06-12 14:46:37.794714, event: header_read, op: osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.795822 7f9d5b6bc700  5 -- op tracker -- , seq: 747916, time: 2014-06-12 14:46:37.794719, event: throttled, op: osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.795830 7f9d5b6bc700  5 -- op tracker -- , seq: 747916, time: 2014-06-12 14:46:37.795756, event: all_read, op: osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.795839 7f9d5b6bc700  5 -- op tracker -- , seq: 747916, time: 0.000000, event: dispatched, op: osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.795845 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:37.795853 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:37.795863 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x17c2aa00 prio 127 cost 594727 latency 0.001148 osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:37.795898 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:37.795909 7f9d61bd4700 10 osd.2 173 dequeue_op 0x17c2aa00 prio 127 cost 594727 latency 0.001195 osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.228( v 173'2983 (0'0,173'2983] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2982 active]
2014-06-12 14:46:37.795941 7f9d61bd4700  5 -- op tracker -- , seq: 747916, time: 2014-06-12 14:46:37.795936, event: reached_pg, op: osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.795952 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2983 (0'0,173'2983] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2982 active] handle_message: 0x17c2aa00
2014-06-12 14:46:37.795968 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2983 (0'0,173'2983] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2982 active] sub_op_modify trans 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 v 173'2984 (transaction) 174
2014-06-12 14:46:37.795986 7f9d61bd4700  5 -- op tracker -- , seq: 747916, time: 2014-06-12 14:46:37.795981, event: started, op: osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.796007 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2983 (0'0,173'2983] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2982 active] append_log log((0'0,173'2983], crt=0'0) [173'2984 (173'797) modify   4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 by client.4286.0:144964 2014-06-12 08:45:40.029071]
2014-06-12 14:46:37.796036 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2984 (0'0,173'2984] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2982 active] add_log_entry 173'2984 (173'797) modify   4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 by client.4286.0:144964 2014-06-12 08:45:40.029071
2014-06-12 14:46:37.796062 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2984 (0'0,173'2984] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2982 active] append_log  adding 1 keys
2014-06-12 14:46:37.796102 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2984, trimmed: 
2014-06-12 14:46:37.796129 7f9d61bd4700  5 -- op tracker -- , seq: 747916, time: 2014-06-12 14:46:37.796124, event: started, op: osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.796142 7f9d61bd4700  5 queue_transactions existing osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:46:37.796147 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:37.796149 7f9d61bd4700  5 queue_op 0x13c50840 seq 2771 osr(3.228 0x276d3e0) 596213 bytes   (queue has 1 ops and 596213 bytes)
2014-06-12 14:46:37.796157 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.228( v 173'2984 (0'0,173'2984] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2982 active] do_sub_op osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:37.796174 7f9d61bd4700 10 osd.2 173 dequeue_op 0x17c2aa00 finish
2014-06-12 14:46:37.796178 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:37.796181 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.796190 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533d560 (1 active)
2014-06-12 14:46:37.796195 7f9d7e169700  5 _do_op 0x13c50840 seq 2771 osr(3.228 0x276d3e0)/0x276d3e0 start
2014-06-12 14:46:37.796199 7f9d7e169700 10 _do_transaction on 0x5b25cf0
2014-06-12 14:46:37.796206 7f9d7e169700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.796425 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.796448 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:37.796492 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:37.796514 7f9d7e169700 15 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.796522 7f9d7e169700 10 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1 r = 0
2014-06-12 14:46:37.796528 7f9d7e169700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.796541 7f9d7e169700 15 _write 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 0~593920
2014-06-12 14:46:37.796705 7f9d7e169700 10 lookup_strip_header done  cid 3.228_head oid 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3
2014-06-12 14:46:37.796841 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:37.798996 7f9d7e169700 10 _generic_write 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 0~593920 = 0
2014-06-12 14:46:37.799081 7f9d7e169700 15 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3
2014-06-12 14:46:37.799118 7f9d7e169700 10 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 = 0
2014-06-12 14:46:37.799125 7f9d7e169700 15 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3
2014-06-12 14:46:37.799132 7f9d7e169700 10 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 = 0
2014-06-12 14:46:37.799362 7f9d7e169700 20 set_header setting 17134 cid 3.228_head oid 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 parent seq 0
2014-06-12 14:46:37.799460 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:37.799476 7f9d7e169700 20 set_header setting 2274 cid meta oid 1fe2503a/pglog_3.228/0//-1 parent seq 0
2014-06-12 14:46:37.799494 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:37.802608 7f9d7e169700 10 _do_op 0x13c50840 seq 2771 r = 0, finisher 0x2966e60 0
2014-06-12 14:46:37.802665 7f9d7715b700  5 -- op tracker -- , seq: 747916, time: 2014-06-12 14:46:37.802654, event: commit_sent, op: osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.802680 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2984 (0'0,173'2984] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2982 active] sub_op_modify_commit on op osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:37.802719 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:37.802756 7f9d7e169700 10 _finish_op 0x13c50840 seq 2771 osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:46:37.802778 7f9d7795c700  5 -- op tracker -- , seq: 747916, time: 2014-06-12 14:46:37.802772, event: sub_op_applied, op: osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.802790 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2984 (0'0,173'2984] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2983 active] sub_op_modify_applied on 0x5b25c00 op osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:37.802809 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2984 (0'0,173'2984] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2983 active] op_applied on primary on version 173'2984
2014-06-12 14:46:37.802820 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2984 (0'0,173'2984] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2983 active] op_applied on replica on version 173'2984
2014-06-12 14:46:37.802850 7f9d7e169700  5 -- op tracker -- , seq: 747916, time: 2014-06-12 14:46:37.802841, event: done, op: osd_sub_op(client.4286.0:144964 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2984 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.802894 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533d560 (0 active)
2014-06-12 14:46:37.802898 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.812238 7f9d5b6bc700  5 -- op tracker -- , seq: 747917, time: 2014-06-12 14:46:37.810550, event: header_read, op: osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.812253 7f9d5b6bc700  5 -- op tracker -- , seq: 747917, time: 2014-06-12 14:46:37.810553, event: throttled, op: osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.812262 7f9d5b6bc700  5 -- op tracker -- , seq: 747917, time: 2014-06-12 14:46:37.812182, event: all_read, op: osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.812271 7f9d5b6bc700  5 -- op tracker -- , seq: 747917, time: 0.000000, event: dispatched, op: osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.812277 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:37.812285 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:37.812295 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x329ea600 prio 127 cost 1032999 latency 0.001744 osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:37.812331 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:37.812340 7f9d673df700 10 osd.2 173 dequeue_op 0x329ea600 prio 127 cost 1032999 latency 0.001789 osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.228( v 173'2984 (0'0,173'2984] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2983 active]
2014-06-12 14:46:37.812364 7f9d673df700  5 -- op tracker -- , seq: 747917, time: 2014-06-12 14:46:37.812360, event: reached_pg, op: osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.812374 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2984 (0'0,173'2984] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2983 active] handle_message: 0x329ea600
2014-06-12 14:46:37.812387 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2984 (0'0,173'2984] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2983 active] sub_op_modify trans 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 v 173'2985 (transaction) 174
2014-06-12 14:46:37.812404 7f9d673df700  5 -- op tracker -- , seq: 747917, time: 2014-06-12 14:46:37.812400, event: started, op: osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.812422 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2984 (0'0,173'2984] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2983 active] append_log log((0'0,173'2984], crt=0'0) [173'2985 (173'2984) modify   4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 by client.4286.0:144965 2014-06-12 08:45:40.185841]
2014-06-12 14:46:37.812454 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2985 (0'0,173'2985] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2983 active] add_log_entry 173'2985 (173'2984) modify   4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 by client.4286.0:144965 2014-06-12 08:45:40.185841
2014-06-12 14:46:37.812478 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2985 (0'0,173'2985] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2983 active] append_log  adding 1 keys
2014-06-12 14:46:37.812516 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2985, trimmed: 
2014-06-12 14:46:37.812540 7f9d673df700  5 -- op tracker -- , seq: 747917, time: 2014-06-12 14:46:37.812535, event: started, op: osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.812551 7f9d673df700  5 queue_transactions existing osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:46:37.812555 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:37.812557 7f9d673df700  5 queue_op 0x1516a8a0 seq 2772 osr(3.228 0x276d3e0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:37.812565 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.228( v 173'2985 (0'0,173'2985] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2983 active] do_sub_op osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:37.812580 7f9d673df700 10 osd.2 173 dequeue_op 0x329ea600 finish
2014-06-12 14:46:37.812584 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:37.812587 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.812596 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533d560 (1 active)
2014-06-12 14:46:37.812600 7f9d7815d700  5 _do_op 0x1516a8a0 seq 2772 osr(3.228 0x276d3e0)/0x276d3e0 start
2014-06-12 14:46:37.812605 7f9d7815d700 10 _do_transaction on 0x90f0630
2014-06-12 14:46:37.812612 7f9d7815d700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.812738 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.812758 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:37.812807 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:37.812823 7f9d7815d700 15 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.812829 7f9d7815d700 10 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1 r = 0
2014-06-12 14:46:37.812835 7f9d7815d700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.812848 7f9d7815d700 15 _write 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 593920~1032192
2014-06-12 14:46:37.812996 7f9d7815d700 10 lookup_strip_header done  cid 3.228_head oid 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3
2014-06-12 14:46:37.813018 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:37.817013 7f9d7815d700 10 _generic_write 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 593920~1032192 = 0
2014-06-12 14:46:37.817098 7f9d7815d700 15 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3
2014-06-12 14:46:37.817122 7f9d7815d700 10 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 = 0
2014-06-12 14:46:37.817129 7f9d7815d700 15 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3
2014-06-12 14:46:37.817140 7f9d7815d700 10 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 = 0
2014-06-12 14:46:37.817442 7f9d7815d700 20 set_header setting 17134 cid 3.228_head oid 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 parent seq 0
2014-06-12 14:46:37.817552 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:37.817566 7f9d7815d700 20 set_header setting 2274 cid meta oid 1fe2503a/pglog_3.228/0//-1 parent seq 0
2014-06-12 14:46:37.817576 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:37.823367 7f9d7815d700 10 _do_op 0x1516a8a0 seq 2772 r = 0, finisher 0x5db3f60 0
2014-06-12 14:46:37.823396 7f9d7815d700 10 _finish_op 0x1516a8a0 seq 2772 osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:46:37.823404 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533d560 (0 active)
2014-06-12 14:46:37.823412 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.823418 7f9d7715b700  5 -- op tracker -- , seq: 747917, time: 2014-06-12 14:46:37.823404, event: commit_sent, op: osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.823428 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2985 (0'0,173'2985] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2983 active] sub_op_modify_commit on op osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:37.823459 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:37.823495 7f9d7795c700  5 -- op tracker -- , seq: 747917, time: 2014-06-12 14:46:37.823490, event: sub_op_applied, op: osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.823503 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2985 (0'0,173'2985] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active] sub_op_modify_applied on 0x90f0540 op osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:37.823523 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2985 (0'0,173'2985] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active] op_applied on primary on version 173'2985
2014-06-12 14:46:37.823534 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2985 (0'0,173'2985] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active] op_applied on replica on version 173'2985
2014-06-12 14:46:37.823563 7f9d7795c700  5 -- op tracker -- , seq: 747917, time: 2014-06-12 14:46:37.823558, event: done, op: osd_sub_op(client.4286.0:144965 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2985 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.829342 7f9d5b6bc700  5 -- op tracker -- , seq: 747918, time: 2014-06-12 14:46:37.828446, event: header_read, op: osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.829359 7f9d5b6bc700  5 -- op tracker -- , seq: 747918, time: 2014-06-12 14:46:37.828448, event: throttled, op: osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.829368 7f9d5b6bc700  5 -- op tracker -- , seq: 747918, time: 2014-06-12 14:46:37.829296, event: all_read, op: osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.829377 7f9d5b6bc700  5 -- op tracker -- , seq: 747918, time: 0.000000, event: dispatched, op: osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.829383 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:37.829390 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:37.829400 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x160e4300 prio 127 cost 520999 latency 0.000953 osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:37.829435 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:37.829443 7f9d67be0700 10 osd.2 173 dequeue_op 0x160e4300 prio 127 cost 520999 latency 0.000997 osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.228( v 173'2985 (0'0,173'2985] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active]
2014-06-12 14:46:37.829469 7f9d67be0700  5 -- op tracker -- , seq: 747918, time: 2014-06-12 14:46:37.829464, event: reached_pg, op: osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.829478 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2985 (0'0,173'2985] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active] handle_message: 0x160e4300
2014-06-12 14:46:37.829492 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2985 (0'0,173'2985] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active] sub_op_modify trans 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 v 173'2986 (transaction) 174
2014-06-12 14:46:37.829512 7f9d67be0700  5 -- op tracker -- , seq: 747918, time: 2014-06-12 14:46:37.829508, event: started, op: osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.829529 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2985 (0'0,173'2985] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active] append_log log((0'0,173'2985], crt=0'0) [173'2986 (173'2985) modify   4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 by client.4286.0:144966 2014-06-12 08:45:40.206075]
2014-06-12 14:46:37.829557 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2986 (0'0,173'2986] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active] add_log_entry 173'2986 (173'2985) modify   4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 by client.4286.0:144966 2014-06-12 08:45:40.206075
2014-06-12 14:46:37.829584 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2986 (0'0,173'2986] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active] append_log  adding 1 keys
2014-06-12 14:46:37.829649 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2986, trimmed: 
2014-06-12 14:46:37.829684 7f9d67be0700  5 -- op tracker -- , seq: 747918, time: 2014-06-12 14:46:37.829678, event: started, op: osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.829697 7f9d67be0700  5 queue_transactions existing osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:46:37.829702 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:37.829704 7f9d67be0700  5 queue_op 0x73a0180 seq 2773 osr(3.228 0x276d3e0) 522485 bytes   (queue has 1 ops and 522485 bytes)
2014-06-12 14:46:37.829711 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.228( v 173'2986 (0'0,173'2986] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active] do_sub_op osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:37.829726 7f9d67be0700 10 osd.2 173 dequeue_op 0x160e4300 finish
2014-06-12 14:46:37.829729 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:37.829732 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.829740 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533d560 (1 active)
2014-06-12 14:46:37.829753 7f9d82171700  5 _do_op 0x73a0180 seq 2773 osr(3.228 0x276d3e0)/0x276d3e0 start
2014-06-12 14:46:37.829758 7f9d82171700 10 _do_transaction on 0x601b7b0
2014-06-12 14:46:37.829763 7f9d82171700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.842503 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.842552 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:37.842599 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:37.842710 7f9d82171700 15 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.842720 7f9d82171700 10 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1 r = 0
2014-06-12 14:46:37.842727 7f9d82171700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.842747 7f9d82171700 15 _write 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 1626112~520192
2014-06-12 14:46:37.842894 7f9d82171700 10 lookup_strip_header done  cid 3.228_head oid 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3
2014-06-12 14:46:37.843046 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:37.844963 7f9d82171700 10 _generic_write 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 1626112~520192 = 0
2014-06-12 14:46:37.845026 7f9d82171700 15 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3
2014-06-12 14:46:37.845044 7f9d82171700 10 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 = 0
2014-06-12 14:46:37.845057 7f9d82171700 15 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3
2014-06-12 14:46:37.845066 7f9d82171700 10 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 = 0
2014-06-12 14:46:37.845278 7f9d82171700 20 set_header setting 17134 cid 3.228_head oid 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 parent seq 0
2014-06-12 14:46:37.845352 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:37.845368 7f9d82171700 20 set_header setting 2274 cid meta oid 1fe2503a/pglog_3.228/0//-1 parent seq 0
2014-06-12 14:46:37.845379 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:37.848124 7f9d82171700 10 _do_op 0x73a0180 seq 2773 r = 0, finisher 0x65b8540 0
2014-06-12 14:46:37.848149 7f9d82171700 10 _finish_op 0x73a0180 seq 2773 osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:46:37.848183 7f9d7795c700  5 -- op tracker -- , seq: 747918, time: 2014-06-12 14:46:37.848172, event: sub_op_applied, op: osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.848196 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2986 (0'0,173'2986] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active] sub_op_modify_applied on 0x601b6c0 op osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:37.848227 7f9d7795c700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:37.848252 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2986 (0'0,173'2986] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active] op_applied on primary on version 173'2986
2014-06-12 14:46:37.848266 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2986 (0'0,173'2986] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active] op_applied on replica on version 173'2986
2014-06-12 14:46:37.848299 7f9d7715b700  5 -- op tracker -- , seq: 747918, time: 2014-06-12 14:46:37.848291, event: commit_sent, op: osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.848310 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2986 (0'0,173'2986] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2984 active] sub_op_modify_commit on op osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:37.848330 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:37.848321 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533d560 (0 active)
2014-06-12 14:46:37.848338 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.848362 7f9d7715b700  5 -- op tracker -- , seq: 747918, time: 2014-06-12 14:46:37.848357, event: done, op: osd_sub_op(client.4286.0:144966 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2986 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.854279 7f9d5b6bc700  5 -- op tracker -- , seq: 747919, time: 2014-06-12 14:46:37.853355, event: header_read, op: osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.854295 7f9d5b6bc700  5 -- op tracker -- , seq: 747919, time: 2014-06-12 14:46:37.853358, event: throttled, op: osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.854311 7f9d5b6bc700  5 -- op tracker -- , seq: 747919, time: 2014-06-12 14:46:37.854235, event: all_read, op: osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.854316 7f9d5b6bc700  5 -- op tracker -- , seq: 747919, time: 0.000000, event: dispatched, op: osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.854321 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:37.854326 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:37.854333 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x2b014f00 prio 127 cost 529191 latency 0.000977 osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:37.854365 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:37.854372 7f9d663dd700 10 osd.2 173 dequeue_op 0x2b014f00 prio 127 cost 529191 latency 0.001017 osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.228( v 173'2986 (0'0,173'2986] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2985 active]
2014-06-12 14:46:37.854390 7f9d663dd700  5 -- op tracker -- , seq: 747919, time: 2014-06-12 14:46:37.854388, event: reached_pg, op: osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.854397 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2986 (0'0,173'2986] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2985 active] handle_message: 0x2b014f00
2014-06-12 14:46:37.854406 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2986 (0'0,173'2986] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2985 active] sub_op_modify trans 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 v 173'2987 (transaction) 174
2014-06-12 14:46:37.854416 7f9d663dd700  5 -- op tracker -- , seq: 747919, time: 2014-06-12 14:46:37.854414, event: started, op: osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.854428 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2986 (0'0,173'2986] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2985 active] append_log log((0'0,173'2986], crt=0'0) [173'2987 (173'2986) modify   4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 by client.4286.0:144967 2014-06-12 08:45:40.230956]
2014-06-12 14:46:37.854446 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2987 (0'0,173'2987] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2985 active] add_log_entry 173'2987 (173'2986) modify   4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 by client.4286.0:144967 2014-06-12 08:45:40.230956
2014-06-12 14:46:37.854467 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2987 (0'0,173'2987] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2985 active] append_log  adding 1 keys
2014-06-12 14:46:37.854490 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2987, trimmed: 
2014-06-12 14:46:37.854508 7f9d663dd700  5 -- op tracker -- , seq: 747919, time: 2014-06-12 14:46:37.854505, event: started, op: osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.854516 7f9d663dd700  5 queue_transactions existing osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:46:37.854520 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:37.854521 7f9d663dd700  5 queue_op 0x45bed20 seq 2774 osr(3.228 0x276d3e0) 530677 bytes   (queue has 1 ops and 530677 bytes)
2014-06-12 14:46:37.854527 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.228( v 173'2987 (0'0,173'2987] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2985 active] do_sub_op osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:37.854536 7f9d663dd700 10 osd.2 173 dequeue_op 0x2b014f00 finish
2014-06-12 14:46:37.854539 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:37.854540 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:37.854598 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533d560 (1 active)
2014-06-12 14:46:37.854607 7f9d7e96a700  5 _do_op 0x45bed20 seq 2774 osr(3.228 0x276d3e0)/0x276d3e0 start
2014-06-12 14:46:37.854612 7f9d7e96a700 10 _do_transaction on 0x30b07b30
2014-06-12 14:46:37.854626 7f9d7e96a700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.854757 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.854796 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:37.854853 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:37.854885 7f9d7e96a700 15 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.854891 7f9d7e96a700 10 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1 r = 0
2014-06-12 14:46:37.854898 7f9d7e96a700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:46:37.854999 7f9d7e96a700 15 _write 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 2146304~528384
2014-06-12 14:46:37.855160 7f9d7e96a700 10 lookup_strip_header done  cid 3.228_head oid 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3
2014-06-12 14:46:37.855318 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:37.857149 7f9d7e96a700 10 _generic_write 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 2146304~528384 = 0
2014-06-12 14:46:37.857205 7f9d7e96a700 15 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3
2014-06-12 14:46:37.857219 7f9d7e96a700 10 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 = 0
2014-06-12 14:46:37.857225 7f9d7e96a700 15 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3
2014-06-12 14:46:37.857234 7f9d7e96a700 10 _setattrs 3.228_head/4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 = 0
2014-06-12 14:46:37.857444 7f9d7e96a700 20 set_header setting 17134 cid 3.228_head oid 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 parent seq 0
2014-06-12 14:46:37.857518 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:37.857534 7f9d7e96a700 20 set_header setting 2274 cid meta oid 1fe2503a/pglog_3.228/0//-1 parent seq 0
2014-06-12 14:46:37.857544 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:37.859753 7f9d7e96a700 10 _do_op 0x45bed20 seq 2774 r = 0, finisher 0x88954a0 0
2014-06-12 14:46:37.859799 7f9d7715b700  5 -- op tracker -- , seq: 747919, time: 2014-06-12 14:46:37.859789, event: commit_sent, op: osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.859810 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2987 (0'0,173'2987] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2985 active] sub_op_modify_commit on op osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:46:37.859840 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:37.859870 7f9d7e96a700 10 _finish_op 0x45bed20 seq 2774 osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:46:37.859885 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533d560 (0 active)
2014-06-12 14:46:37.859889 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:37.859917 7f9d7795c700  5 -- op tracker -- , seq: 747919, time: 2014-06-12 14:46:37.859912, event: sub_op_applied, op: osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.859925 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2987 (0'0,173'2987] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] sub_op_modify_applied on 0x30b07a40 op osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:37.859941 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2987 (0'0,173'2987] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] op_applied on primary on version 173'2987
2014-06-12 14:46:37.859951 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2987 (0'0,173'2987] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] op_applied on replica on version 173'2987
2014-06-12 14:46:37.859970 7f9d7795c700  5 -- op tracker -- , seq: 747919, time: 2014-06-12 14:46:37.859965, event: done, op: osd_sub_op(client.4286.0:144967 3.228 4b76c228/rbd_data.10956b8b4567.0000000000001092/head//3 [] v 173'2987 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:37.956347 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:37.956534 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:38.021448 7f9d56432700  5 -- op tracker -- , seq: 747920, time: 2014-06-12 14:46:38.020409, event: header_read, op: osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.021464 7f9d56432700  5 -- op tracker -- , seq: 747920, time: 2014-06-12 14:46:38.020414, event: throttled, op: osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.021473 7f9d56432700  5 -- op tracker -- , seq: 747920, time: 2014-06-12 14:46:38.021401, event: all_read, op: osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.021481 7f9d56432700  5 -- op tracker -- , seq: 747920, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.021487 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:38.021503 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:38.021512 7f9d56432700 15 osd.2 173 enqueue_op 0x23e09e00 prio 127 cost 516903 latency 0.001102 osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.021546 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:38.021557 7f9d693e3700 10 osd.2 173 dequeue_op 0x23e09e00 prio 127 cost 516903 latency 0.001148 osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.51e( v 173'2683 (0'0,173'2683] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2682 active]
2014-06-12 14:46:38.021586 7f9d693e3700  5 -- op tracker -- , seq: 747920, time: 2014-06-12 14:46:38.021581, event: reached_pg, op: osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.021597 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2683 (0'0,173'2683] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2682 active] handle_message: 0x23e09e00
2014-06-12 14:46:38.021622 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2683 (0'0,173'2683] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2682 active] sub_op_modify trans 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 v 173'2684 (transaction) 174
2014-06-12 14:46:38.021641 7f9d693e3700  5 -- op tracker -- , seq: 747920, time: 2014-06-12 14:46:38.021636, event: started, op: osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.021658 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2683 (0'0,173'2683] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2682 active] append_log log((0'0,173'2683], crt=0'0) [173'2684 (173'2683) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145053 2014-06-12 08:44:59.312490]
2014-06-12 14:46:38.021686 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2684 (0'0,173'2684] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2682 active] add_log_entry 173'2684 (173'2683) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145053 2014-06-12 08:44:59.312490
2014-06-12 14:46:38.021711 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2684 (0'0,173'2684] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2682 active] append_log  adding 1 keys
2014-06-12 14:46:38.021739 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2684, trimmed: 
2014-06-12 14:46:38.021759 7f9d693e3700  5 -- op tracker -- , seq: 747920, time: 2014-06-12 14:46:38.021755, event: started, op: osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.021768 7f9d693e3700  5 queue_transactions existing osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:38.021771 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:38.021772 7f9d693e3700  5 queue_op 0x7ca8f60 seq 2520 osr(3.51e 0x276db50) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:38.021777 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.51e( v 173'2684 (0'0,173'2684] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2682 active] do_sub_op osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.021781 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fb00 (1 active)
2014-06-12 14:46:38.021786 7f9d693e3700 10 osd.2 173 dequeue_op 0x23e09e00 finish
2014-06-12 14:46:38.021789 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:38.021790 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:38.021789 7f9d7d167700  5 _do_op 0x7ca8f60 seq 2520 osr(3.51e 0x276db50)/0x276db50 start
2014-06-12 14:46:38.021794 7f9d7d167700 10 _do_transaction on 0x2e9b3b30
2014-06-12 14:46:38.021800 7f9d7d167700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:38.021931 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:38.021956 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:38.022002 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:38.022024 7f9d7d167700 15 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:38.022030 7f9d7d167700 10 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1 r = 0
2014-06-12 14:46:38.022036 7f9d7d167700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:38.022268 7f9d7d167700 15 _write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 2584576~516096
2014-06-12 14:46:38.022411 7f9d7d167700 10 lookup_strip_header done  cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:38.022580 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:38.024595 7f9d7d167700 10 _generic_write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 2584576~516096 = 0
2014-06-12 14:46:38.024629 7f9d7d167700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:38.024645 7f9d7d167700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:38.024649 7f9d7d167700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:38.024654 7f9d7d167700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:38.024794 7f9d7d167700 20 set_header setting 16949 cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 parent seq 0
2014-06-12 14:46:38.024844 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:38.024855 7f9d7d167700 20 set_header setting 2288 cid meta oid 1fe1612b/pglog_3.51e/0//-1 parent seq 0
2014-06-12 14:46:38.024861 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:38.027027 7f9d7d167700 10 _do_op 0x7ca8f60 seq 2520 r = 0, finisher 0x1b2d6980 0
2014-06-12 14:46:38.027048 7f9d7d167700 10 _finish_op 0x7ca8f60 seq 2520 osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:38.027065 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fb00 (0 active)
2014-06-12 14:46:38.027076 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:38.027066 7f9d7715b700  5 -- op tracker -- , seq: 747920, time: 2014-06-12 14:46:38.027055, event: commit_sent, op: osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.027080 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2684 (0'0,173'2684] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2682 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:38.027107 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:38.027148 7f9d7795c700  5 -- op tracker -- , seq: 747920, time: 2014-06-12 14:46:38.027140, event: sub_op_applied, op: osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.027159 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2684 (0'0,173'2684] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2683 active] sub_op_modify_applied on 0x2e9b3a40 op osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.027172 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2684 (0'0,173'2684] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2683 active] op_applied on primary on version 173'2684
2014-06-12 14:46:38.027179 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2684 (0'0,173'2684] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2683 active] op_applied on replica on version 173'2684
2014-06-12 14:46:38.027196 7f9d7795c700  5 -- op tracker -- , seq: 747920, time: 2014-06-12 14:46:38.027194, event: done, op: osd_sub_op(client.4308.0:145053 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.152812 7f9d56432700  5 -- op tracker -- , seq: 747921, time: 2014-06-12 14:46:38.152040, event: header_read, op: osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.152826 7f9d56432700  5 -- op tracker -- , seq: 747921, time: 2014-06-12 14:46:38.152042, event: throttled, op: osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.152835 7f9d56432700  5 -- op tracker -- , seq: 747921, time: 2014-06-12 14:46:38.152770, event: all_read, op: osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.152844 7f9d56432700  5 -- op tracker -- , seq: 747921, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.152849 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:38.152857 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:38.152866 7f9d56432700 15 osd.2 173 enqueue_op 0x23e08d00 prio 127 cost 496423 latency 0.000826 osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.152901 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:38.152909 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x23e08d00 prio 127 cost 496423 latency 0.000869 osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3ae( v 173'2952 (0'0,173'2952] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2951 active]
2014-06-12 14:46:38.152932 7f9d6b3e7700  5 -- op tracker -- , seq: 747921, time: 2014-06-12 14:46:38.152928, event: reached_pg, op: osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.152942 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2952 (0'0,173'2952] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2951 active] handle_message: 0x23e08d00
2014-06-12 14:46:38.152954 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2952 (0'0,173'2952] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2951 active] sub_op_modify trans b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 v 173'2953 (transaction) 174
2014-06-12 14:46:38.152979 7f9d6b3e7700  5 -- op tracker -- , seq: 747921, time: 2014-06-12 14:46:38.152975, event: started, op: osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.152997 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2952 (0'0,173'2952] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2951 active] append_log log((0'0,173'2952], crt=0'0) [173'2953 (173'781) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143731 2014-06-12 08:44:42.276088]
2014-06-12 14:46:38.153026 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2953 (0'0,173'2953] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2951 active] add_log_entry 173'2953 (173'781) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143731 2014-06-12 08:44:42.276088
2014-06-12 14:46:38.153048 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2953 (0'0,173'2953] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2951 active] append_log  adding 1 keys
2014-06-12 14:46:38.153084 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2953, trimmed: 
2014-06-12 14:46:38.153110 7f9d6b3e7700  5 -- op tracker -- , seq: 747921, time: 2014-06-12 14:46:38.153105, event: started, op: osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.153121 7f9d6b3e7700  5 queue_transactions existing osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:38.153126 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:38.153128 7f9d6b3e7700  5 queue_op 0x49d3860 seq 2681 osr(3.3ae 0x276d6c0) 497909 bytes   (queue has 1 ops and 497909 bytes)
2014-06-12 14:46:38.153141 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478a20 (1 active)
2014-06-12 14:46:38.153136 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2953 (0'0,173'2953] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2951 active] do_sub_op osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.153150 7f9d8116f700  5 _do_op 0x49d3860 seq 2681 osr(3.3ae 0x276d6c0)/0x276d6c0 start
2014-06-12 14:46:38.153151 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x23e08d00 finish
2014-06-12 14:46:38.153155 7f9d8116f700 10 _do_transaction on 0x66760f0
2014-06-12 14:46:38.153154 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:38.153158 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:38.153161 7f9d8116f700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:38.153267 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:38.153284 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:38.153326 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:38.153342 7f9d8116f700 15 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:38.153347 7f9d8116f700 10 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1 r = 0
2014-06-12 14:46:38.153353 7f9d8116f700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:38.153366 7f9d8116f700 15 _write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 0~495616
2014-06-12 14:46:38.153528 7f9d8116f700 10 lookup_strip_header done  cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:38.153650 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:38.154355 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:38.155781 7f9d8116f700 10 _generic_write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 0~495616 = 0
2014-06-12 14:46:38.155839 7f9d8116f700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:38.155855 7f9d8116f700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:38.155861 7f9d8116f700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:38.155869 7f9d8116f700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:38.156080 7f9d8116f700 20 set_header setting 16695 cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 parent seq 0
2014-06-12 14:46:38.156146 7f9d8116f700 20 set_header setting 2256 cid meta oid 1fe0bb56/pglog_3.3ae/0//-1 parent seq 0
2014-06-12 14:46:38.156182 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:38.156192 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:38.158502 7f9d8116f700 10 _do_op 0x49d3860 seq 2681 r = 0, finisher 0x6d9c200 0
2014-06-12 14:46:38.158522 7f9d8116f700 10 _finish_op 0x49d3860 seq 2681 osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:38.158530 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478a20 (0 active)
2014-06-12 14:46:38.158539 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:38.158540 7f9d7715b700  5 -- op tracker -- , seq: 747921, time: 2014-06-12 14:46:38.158529, event: commit_sent, op: osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.158668 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2953 (0'0,173'2953] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2951 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:38.158693 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:38.158722 7f9d7795c700  5 -- op tracker -- , seq: 747921, time: 2014-06-12 14:46:38.158716, event: sub_op_applied, op: osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.158731 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2953 (0'0,173'2953] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2952 active] sub_op_modify_applied on 0x6676000 op osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.158746 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2953 (0'0,173'2953] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2952 active] op_applied on primary on version 173'2953
2014-06-12 14:46:38.158757 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2953 (0'0,173'2953] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2952 active] op_applied on replica on version 173'2953
2014-06-12 14:46:38.158782 7f9d7795c700  5 -- op tracker -- , seq: 747921, time: 2014-06-12 14:46:38.158777, event: done, op: osd_sub_op(client.4317.0:143731 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2953 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.351058 7f9d56432700  5 -- op tracker -- , seq: 747922, time: 2014-06-12 14:46:38.350083, event: header_read, op: osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.351078 7f9d56432700  5 -- op tracker -- , seq: 747922, time: 2014-06-12 14:46:38.350086, event: throttled, op: osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.351086 7f9d56432700  5 -- op tracker -- , seq: 747922, time: 2014-06-12 14:46:38.350996, event: all_read, op: osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.351094 7f9d56432700  5 -- op tracker -- , seq: 747922, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.351101 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:38.351108 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:38.351118 7f9d56432700 15 osd.2 173 enqueue_op 0x2b015d00 prio 127 cost 516903 latency 0.001035 osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.351155 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:38.351166 7f9d683e1700 10 osd.2 173 dequeue_op 0x2b015d00 prio 127 cost 516903 latency 0.001082 osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.51e( v 173'2684 (0'0,173'2684] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2683 active]
2014-06-12 14:46:38.351193 7f9d683e1700  5 -- op tracker -- , seq: 747922, time: 2014-06-12 14:46:38.351188, event: reached_pg, op: osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.351207 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2684 (0'0,173'2684] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2683 active] handle_message: 0x2b015d00
2014-06-12 14:46:38.351221 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2684 (0'0,173'2684] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2683 active] sub_op_modify trans 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 v 173'2685 (transaction) 174
2014-06-12 14:46:38.351238 7f9d683e1700  5 -- op tracker -- , seq: 747922, time: 2014-06-12 14:46:38.351234, event: started, op: osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.351256 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2684 (0'0,173'2684] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2683 active] append_log log((0'0,173'2684], crt=0'0) [173'2685 (173'2684) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145054 2014-06-12 08:44:59.572056]
2014-06-12 14:46:38.351282 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2685 (0'0,173'2685] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2683 active] add_log_entry 173'2685 (173'2684) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145054 2014-06-12 08:44:59.572056
2014-06-12 14:46:38.351308 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2685 (0'0,173'2685] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2683 active] append_log  adding 1 keys
2014-06-12 14:46:38.351350 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2685, trimmed: 
2014-06-12 14:46:38.351380 7f9d683e1700  5 -- op tracker -- , seq: 747922, time: 2014-06-12 14:46:38.351375, event: started, op: osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.351394 7f9d683e1700  5 queue_transactions existing osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:38.351399 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:38.351400 7f9d683e1700  5 queue_op 0x30e00c0 seq 2521 osr(3.51e 0x276db50) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:38.351407 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.51e( v 173'2685 (0'0,173'2685] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2683 active] do_sub_op osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.351422 7f9d683e1700 10 osd.2 173 dequeue_op 0x2b015d00 finish
2014-06-12 14:46:38.351426 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:38.351429 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:38.351438 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fb00 (1 active)
2014-06-12 14:46:38.351444 7f9d79960700  5 _do_op 0x30e00c0 seq 2521 osr(3.51e 0x276db50)/0x276db50 start
2014-06-12 14:46:38.351449 7f9d79960700 10 _do_transaction on 0x52c82b0
2014-06-12 14:46:38.351454 7f9d79960700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:38.351586 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:38.351607 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:38.351658 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:38.351822 7f9d79960700 15 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:38.351833 7f9d79960700 10 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1 r = 0
2014-06-12 14:46:38.351840 7f9d79960700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:38.351861 7f9d79960700 15 _write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 3100672~516096
2014-06-12 14:46:38.352014 7f9d79960700 10 lookup_strip_header done  cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:38.352187 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:38.354258 7f9d79960700 10 _generic_write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 3100672~516096 = 0
2014-06-12 14:46:38.354333 7f9d79960700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:38.354355 7f9d79960700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:38.354361 7f9d79960700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:38.354376 7f9d79960700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:38.354609 7f9d79960700 20 set_header setting 16949 cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 parent seq 0
2014-06-12 14:46:38.354709 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:38.354728 7f9d79960700 20 set_header setting 2288 cid meta oid 1fe1612b/pglog_3.51e/0//-1 parent seq 0
2014-06-12 14:46:38.354740 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:38.357996 7f9d79960700 10 _do_op 0x30e00c0 seq 2521 r = 0, finisher 0x7f56f40 0
2014-06-12 14:46:38.358019 7f9d79960700 10 _finish_op 0x30e00c0 seq 2521 osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:38.358027 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fb00 (0 active)
2014-06-12 14:46:38.358038 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:38.358042 7f9d7715b700  5 -- op tracker -- , seq: 747922, time: 2014-06-12 14:46:38.358030, event: commit_sent, op: osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.358055 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2685 (0'0,173'2685] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2683 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:38.358081 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:38.358113 7f9d7795c700  5 -- op tracker -- , seq: 747922, time: 2014-06-12 14:46:38.358106, event: sub_op_applied, op: osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.358124 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2685 (0'0,173'2685] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2684 active] sub_op_modify_applied on 0x52c81c0 op osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.358143 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2685 (0'0,173'2685] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2684 active] op_applied on primary on version 173'2685
2014-06-12 14:46:38.358150 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2685 (0'0,173'2685] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2684 active] op_applied on replica on version 173'2685
2014-06-12 14:46:38.358164 7f9d7795c700  5 -- op tracker -- , seq: 747922, time: 2014-06-12 14:46:38.358161, event: done, op: osd_sub_op(client.4308.0:145054 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.368008 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:38.419327 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19416 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,4])
2014-06-12 14:46:38.419353 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19416 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,4])
2014-06-12 14:46:38.419725 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:38.419781 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:38.419821 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:38.419840 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:38.419859 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:38.419879 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:38.419897 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:38.419926 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:38.419943 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:38.419968 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:38.420001 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:38.420019 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:38.420035 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:38.420070 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:38.420173 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:38.420196 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:38.420217 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:38.420238 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:38.420260 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:38.420280 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:38.420301 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:38.421684 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:38.477651 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:38.477675 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:38.477678 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:38.477679 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:38.477681 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 4; oldest is 5.316336 seconds old
2014-06-12 14:46:38.477709 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:38.620047 7f9d56432700  5 -- op tracker -- , seq: 747923, time: 2014-06-12 14:46:38.619161, event: header_read, op: osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.620061 7f9d56432700  5 -- op tracker -- , seq: 747923, time: 2014-06-12 14:46:38.619165, event: throttled, op: osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.620071 7f9d56432700  5 -- op tracker -- , seq: 747923, time: 2014-06-12 14:46:38.620007, event: all_read, op: osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.620080 7f9d56432700  5 -- op tracker -- , seq: 747923, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.620084 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:38.620091 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:38.620099 7f9d56432700 15 osd.2 173 enqueue_op 0x2b015f00 prio 127 cost 516903 latency 0.000938 osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.620129 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:38.620136 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2b015f00 prio 127 cost 516903 latency 0.000975 osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3ae( v 173'2953 (0'0,173'2953] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2952 active]
2014-06-12 14:46:38.620159 7f9d65bdc700  5 -- op tracker -- , seq: 747923, time: 2014-06-12 14:46:38.620154, event: reached_pg, op: osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.620168 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2953 (0'0,173'2953] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2952 active] handle_message: 0x2b015f00
2014-06-12 14:46:38.620179 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2953 (0'0,173'2953] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2952 active] sub_op_modify trans b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 v 173'2954 (transaction) 174
2014-06-12 14:46:38.620196 7f9d65bdc700  5 -- op tracker -- , seq: 747923, time: 2014-06-12 14:46:38.620192, event: started, op: osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.620209 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2953 (0'0,173'2953] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2952 active] append_log log((0'0,173'2953], crt=0'0) [173'2954 (173'2953) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143732 2014-06-12 08:44:42.288546]
2014-06-12 14:46:38.620233 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2954 (0'0,173'2954] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2952 active] add_log_entry 173'2954 (173'2953) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143732 2014-06-12 08:44:42.288546
2014-06-12 14:46:38.620254 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2954 (0'0,173'2954] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2952 active] append_log  adding 1 keys
2014-06-12 14:46:38.620290 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2954, trimmed: 
2014-06-12 14:46:38.620313 7f9d65bdc700  5 -- op tracker -- , seq: 747923, time: 2014-06-12 14:46:38.620308, event: started, op: osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.620323 7f9d65bdc700  5 queue_transactions existing osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:38.620327 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:38.620328 7f9d65bdc700  5 queue_op 0x139d8540 seq 2682 osr(3.3ae 0x276d6c0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:38.620339 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478a20 (1 active)
2014-06-12 14:46:38.620348 7f9d8096e700  5 _do_op 0x139d8540 seq 2682 osr(3.3ae 0x276d6c0)/0x276d6c0 start
2014-06-12 14:46:38.620336 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2954 (0'0,173'2954] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2952 active] do_sub_op osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.620351 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2b015f00 finish
2014-06-12 14:46:38.620352 7f9d8096e700 10 _do_transaction on 0x1926ab70
2014-06-12 14:46:38.620355 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:38.620357 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:38.620358 7f9d8096e700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:38.620468 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:38.620485 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:38.620529 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:38.620552 7f9d8096e700 15 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:38.620557 7f9d8096e700 10 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1 r = 0
2014-06-12 14:46:38.620562 7f9d8096e700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:38.620731 7f9d8096e700 15 _write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 495616~516096
2014-06-12 14:46:38.620879 7f9d8096e700 10 lookup_strip_header done  cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:38.621061 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:38.622998 7f9d8096e700 10 _generic_write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 495616~516096 = 0
2014-06-12 14:46:38.623052 7f9d8096e700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:38.623067 7f9d8096e700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:38.623072 7f9d8096e700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:38.623080 7f9d8096e700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:38.623279 7f9d8096e700 20 set_header setting 16695 cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 parent seq 0
2014-06-12 14:46:38.623348 7f9d8096e700 20 set_header setting 2256 cid meta oid 1fe0bb56/pglog_3.3ae/0//-1 parent seq 0
2014-06-12 14:46:38.623369 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:38.623379 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:38.625848 7f9d8096e700 10 _do_op 0x139d8540 seq 2682 r = 0, finisher 0x10026fe0 0
2014-06-12 14:46:38.625864 7f9d8096e700 10 _finish_op 0x139d8540 seq 2682 osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:38.625868 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478a20 (0 active)
2014-06-12 14:46:38.625871 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:38.625878 7f9d7715b700  5 -- op tracker -- , seq: 747923, time: 2014-06-12 14:46:38.625870, event: commit_sent, op: osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.625891 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2954 (0'0,173'2954] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2952 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:38.625909 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:38.625936 7f9d7795c700  5 -- op tracker -- , seq: 747923, time: 2014-06-12 14:46:38.625929, event: sub_op_applied, op: osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.625950 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2954 (0'0,173'2954] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active] sub_op_modify_applied on 0x1926aa80 op osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.625966 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2954 (0'0,173'2954] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active] op_applied on primary on version 173'2954
2014-06-12 14:46:38.625976 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2954 (0'0,173'2954] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active] op_applied on replica on version 173'2954
2014-06-12 14:46:38.626001 7f9d7795c700  5 -- op tracker -- , seq: 747923, time: 2014-06-12 14:46:38.625996, event: done, op: osd_sub_op(client.4317.0:143732 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2954 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.643599 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:38.643812 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:38.667156 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:38.667206 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:38.774032 7f9d56432700  5 -- op tracker -- , seq: 747924, time: 2014-06-12 14:46:38.772816, event: header_read, op: osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.774049 7f9d56432700  5 -- op tracker -- , seq: 747924, time: 2014-06-12 14:46:38.772819, event: throttled, op: osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.774058 7f9d56432700  5 -- op tracker -- , seq: 747924, time: 2014-06-12 14:46:38.773988, event: all_read, op: osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.774067 7f9d56432700  5 -- op tracker -- , seq: 747924, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.774074 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:38.774082 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:38.774093 7f9d56432700 15 osd.2 173 enqueue_op 0x9fac000 prio 127 cost 578343 latency 0.001276 osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.774130 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:38.774140 7f9d62bd6700 10 osd.2 173 dequeue_op 0x9fac000 prio 127 cost 578343 latency 0.001324 osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.51e( v 173'2685 (0'0,173'2685] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2684 active]
2014-06-12 14:46:38.774167 7f9d62bd6700  5 -- op tracker -- , seq: 747924, time: 2014-06-12 14:46:38.774162, event: reached_pg, op: osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.774189 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2685 (0'0,173'2685] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2684 active] handle_message: 0x9fac000
2014-06-12 14:46:38.774202 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2685 (0'0,173'2685] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2684 active] sub_op_modify trans 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 v 173'2686 (transaction) 174
2014-06-12 14:46:38.774218 7f9d62bd6700  5 -- op tracker -- , seq: 747924, time: 2014-06-12 14:46:38.774214, event: started, op: osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.774232 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2685 (0'0,173'2685] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2684 active] append_log log((0'0,173'2685], crt=0'0) [173'2686 (173'2685) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145055 2014-06-12 08:45:04.374891]
2014-06-12 14:46:38.774255 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2686 (0'0,173'2686] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2684 active] add_log_entry 173'2686 (173'2685) modify   18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 by client.4308.0:145055 2014-06-12 08:45:04.374891
2014-06-12 14:46:38.774281 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2686 (0'0,173'2686] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2684 active] append_log  adding 1 keys
2014-06-12 14:46:38.774316 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2686, trimmed: 
2014-06-12 14:46:38.774339 7f9d62bd6700  5 -- op tracker -- , seq: 747924, time: 2014-06-12 14:46:38.774334, event: started, op: osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.774370 7f9d62bd6700  5 queue_transactions existing osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:38.774375 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:38.774376 7f9d62bd6700  5 queue_op 0x16909200 seq 2522 osr(3.51e 0x276db50) 579829 bytes   (queue has 1 ops and 579829 bytes)
2014-06-12 14:46:38.774382 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.51e( v 173'2686 (0'0,173'2686] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2684 active] do_sub_op osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.774396 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fb00 (1 active)
2014-06-12 14:46:38.774403 7f9d62bd6700 10 osd.2 173 dequeue_op 0x9fac000 finish
2014-06-12 14:46:38.774406 7f9d8016d700  5 _do_op 0x16909200 seq 2522 osr(3.51e 0x276db50)/0x276db50 start
2014-06-12 14:46:38.774407 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:38.774410 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:38.774411 7f9d8016d700 10 _do_transaction on 0x95480f0
2014-06-12 14:46:38.774417 7f9d8016d700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:38.774683 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:38.774719 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:38.774773 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:38.774804 7f9d8016d700 15 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:38.774813 7f9d8016d700 10 _omap_rmkeys meta/1fe1612b/pglog_3.51e/0//-1 r = 0
2014-06-12 14:46:38.774819 7f9d8016d700 15 _omap_setkeys meta/1fe1612b/pglog_3.51e/0//-1
2014-06-12 14:46:38.774838 7f9d8016d700 15 _write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 3616768~577536
2014-06-12 14:46:38.775282 7f9d8016d700 10 lookup_strip_header done  cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:38.775485 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:38.778514 7f9d8016d700 10 _generic_write 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 3616768~577536 = 0
2014-06-12 14:46:38.778590 7f9d8016d700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:38.778610 7f9d8016d700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:38.778616 7f9d8016d700 15 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3
2014-06-12 14:46:38.778625 7f9d8016d700 10 _setattrs 3.51e_head/18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 = 0
2014-06-12 14:46:38.778912 7f9d8016d700 20 set_header setting 16949 cid 3.51e_head oid 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 parent seq 0
2014-06-12 14:46:38.779010 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:38.779027 7f9d8016d700 20 set_header setting 2288 cid meta oid 1fe1612b/pglog_3.51e/0//-1 parent seq 0
2014-06-12 14:46:38.779039 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:38.785682 7f9d8016d700 10 _do_op 0x16909200 seq 2522 r = 0, finisher 0x7f570a0 0
2014-06-12 14:46:38.785750 7f9d7715b700  5 -- op tracker -- , seq: 747924, time: 2014-06-12 14:46:38.785740, event: commit_sent, op: osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.785761 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2686 (0'0,173'2686] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2684 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:38.785788 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:38.785846 7f9d8016d700 10 _finish_op 0x16909200 seq 2522 osr(3.51e 0x276db50)/0x276db50
2014-06-12 14:46:38.785863 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fb00 (0 active)
2014-06-12 14:46:38.785867 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:38.785876 7f9d7795c700  5 -- op tracker -- , seq: 747924, time: 2014-06-12 14:46:38.785870, event: sub_op_applied, op: osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.785884 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2686 (0'0,173'2686] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2685 active] sub_op_modify_applied on 0x9548000 op osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.785907 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2686 (0'0,173'2686] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2685 active] op_applied on primary on version 173'2686
2014-06-12 14:46:38.785918 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.51e( v 173'2686 (0'0,173'2686] local-les=153 n=300 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2685 active] op_applied on replica on version 173'2686
2014-06-12 14:46:38.785943 7f9d7795c700  5 -- op tracker -- , seq: 747924, time: 2014-06-12 14:46:38.785939, event: done, op: osd_sub_op(client.4308.0:145055 3.51e 18e53d1e/rbd_data.10ab6b8b4567.0000000000000f80/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.919490 7f9d56432700  5 -- op tracker -- , seq: 747925, time: 2014-06-12 14:46:38.918751, event: header_read, op: osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.919505 7f9d56432700  5 -- op tracker -- , seq: 747925, time: 2014-06-12 14:46:38.918753, event: throttled, op: osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.919511 7f9d56432700  5 -- op tracker -- , seq: 747925, time: 2014-06-12 14:46:38.919443, event: all_read, op: osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.919516 7f9d56432700  5 -- op tracker -- , seq: 747925, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.919521 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:38.919526 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:38.919533 7f9d56432700 15 osd.2 173 enqueue_op 0x9fada00 prio 127 cost 516903 latency 0.000782 osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.919564 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:38.919574 7f9d64bda700 10 osd.2 173 dequeue_op 0x9fada00 prio 127 cost 516903 latency 0.000822 osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3ae( v 173'2954 (0'0,173'2954] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active]
2014-06-12 14:46:38.919601 7f9d64bda700  5 -- op tracker -- , seq: 747925, time: 2014-06-12 14:46:38.919596, event: reached_pg, op: osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.919611 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2954 (0'0,173'2954] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active] handle_message: 0x9fada00
2014-06-12 14:46:38.919626 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2954 (0'0,173'2954] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active] sub_op_modify trans b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 v 173'2955 (transaction) 174
2014-06-12 14:46:38.919648 7f9d64bda700  5 -- op tracker -- , seq: 747925, time: 2014-06-12 14:46:38.919644, event: started, op: osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.919666 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2954 (0'0,173'2954] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active] append_log log((0'0,173'2954], crt=0'0) [173'2955 (173'2954) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143733 2014-06-12 08:44:42.317778]
2014-06-12 14:46:38.919693 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2955 (0'0,173'2955] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active] add_log_entry 173'2955 (173'2954) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143733 2014-06-12 08:44:42.317778
2014-06-12 14:46:38.919717 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2955 (0'0,173'2955] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active] append_log  adding 1 keys
2014-06-12 14:46:38.919753 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2955, trimmed: 
2014-06-12 14:46:38.919781 7f9d64bda700  5 -- op tracker -- , seq: 747925, time: 2014-06-12 14:46:38.919777, event: started, op: osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.919791 7f9d64bda700  5 queue_transactions existing osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:38.919796 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:38.919798 7f9d64bda700  5 queue_op 0x56dcea0 seq 2683 osr(3.3ae 0x276d6c0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:38.919809 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478a20 (1 active)
2014-06-12 14:46:38.919817 7f9d7915f700  5 _do_op 0x56dcea0 seq 2683 osr(3.3ae 0x276d6c0)/0x276d6c0 start
2014-06-12 14:46:38.919805 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2955 (0'0,173'2955] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active] do_sub_op osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.919822 7f9d7915f700 10 _do_transaction on 0x5ea62b0
2014-06-12 14:46:38.919821 7f9d64bda700 10 osd.2 173 dequeue_op 0x9fada00 finish
2014-06-12 14:46:38.919825 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:38.919828 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:38.919828 7f9d7915f700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:38.919941 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:38.919974 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:38.920022 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:38.920140 7f9d7915f700 15 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:38.920149 7f9d7915f700 10 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1 r = 0
2014-06-12 14:46:38.920155 7f9d7915f700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:38.920173 7f9d7915f700 15 _write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 1011712~516096
2014-06-12 14:46:38.920303 7f9d7915f700 10 lookup_strip_header done  cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:38.920447 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:38.922088 7f9d7915f700 10 _generic_write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 1011712~516096 = 0
2014-06-12 14:46:38.922143 7f9d7915f700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:38.922156 7f9d7915f700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:38.922165 7f9d7915f700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:38.922173 7f9d7915f700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:38.922394 7f9d7915f700 20 set_header setting 16695 cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 parent seq 0
2014-06-12 14:46:38.922461 7f9d7915f700 20 set_header setting 2256 cid meta oid 1fe0bb56/pglog_3.3ae/0//-1 parent seq 0
2014-06-12 14:46:38.924338 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:38.924356 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:38.928099 7f9d7915f700 10 _do_op 0x56dcea0 seq 2683 r = 0, finisher 0x7a8b400 0
2014-06-12 14:46:38.928119 7f9d7915f700 10 _finish_op 0x56dcea0 seq 2683 osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:38.928127 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478a20 (0 active)
2014-06-12 14:46:38.928137 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:38.928186 7f9d7795c700  5 -- op tracker -- , seq: 747925, time: 2014-06-12 14:46:38.928166, event: sub_op_applied, op: osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.928228 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2955 (0'0,173'2955] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active] sub_op_modify_applied on 0x5ea61c0 op osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:38.928265 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:38.928290 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2955 (0'0,173'2955] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active] op_applied on primary on version 173'2955
2014-06-12 14:46:38.928307 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2955 (0'0,173'2955] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active] op_applied on replica on version 173'2955
2014-06-12 14:46:38.928345 7f9d7715b700  5 -- op tracker -- , seq: 747925, time: 2014-06-12 14:46:38.928333, event: commit_sent, op: osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:38.928383 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2955 (0'0,173'2955] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2953 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:38.928415 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:38.928455 7f9d7715b700  5 -- op tracker -- , seq: 747925, time: 2014-06-12 14:46:38.928449, event: done, op: osd_sub_op(client.4317.0:143733 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2955 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.022668 7f9d56432700  5 -- op tracker -- , seq: 747926, time: 2014-06-12 14:46:39.019025, event: header_read, op: osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.022696 7f9d56432700  5 -- op tracker -- , seq: 747926, time: 2014-06-12 14:46:39.019028, event: throttled, op: osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.022712 7f9d56432700  5 -- op tracker -- , seq: 747926, time: 2014-06-12 14:46:39.022604, event: all_read, op: osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.022724 7f9d56432700  5 -- op tracker -- , seq: 747926, time: 0.000000, event: dispatched, op: osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.022730 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:39.022738 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:39.022747 7f9d56432700 15 osd.2 173 enqueue_op 0x9fad400 prio 127 cost 2229031 latency 0.003721 osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.022783 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.022793 7f9d6abe6700 10 osd.2 173 dequeue_op 0x9fad400 prio 127 cost 2229031 latency 0.003767 osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.60e( v 173'2558 (0'0,173'2558] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2557 active]
2014-06-12 14:46:39.022823 7f9d6abe6700  5 -- op tracker -- , seq: 747926, time: 2014-06-12 14:46:39.022818, event: reached_pg, op: osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.022836 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2558 (0'0,173'2558] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2557 active] handle_message: 0x9fad400
2014-06-12 14:46:39.022851 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2558 (0'0,173'2558] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2557 active] sub_op_modify trans ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 v 173'2559 (transaction) 174
2014-06-12 14:46:39.022871 7f9d6abe6700  5 -- op tracker -- , seq: 747926, time: 2014-06-12 14:46:39.022866, event: started, op: osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.022890 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2558 (0'0,173'2558] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2557 active] append_log log((0'0,173'2558], crt=0'0) [173'2559 (173'642) modify   ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 by client.4296.0:143671 2014-06-12 08:45:41.174532]
2014-06-12 14:46:39.022918 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2559 (0'0,173'2559] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2557 active] add_log_entry 173'2559 (173'642) modify   ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 by client.4296.0:143671 2014-06-12 08:45:41.174532
2014-06-12 14:46:39.022944 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2559 (0'0,173'2559] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2557 active] append_log  adding 1 keys
2014-06-12 14:46:39.022984 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2559, trimmed: 
2014-06-12 14:46:39.023012 7f9d6abe6700  5 -- op tracker -- , seq: 747926, time: 2014-06-12 14:46:39.023006, event: started, op: osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.023024 7f9d6abe6700  5 queue_transactions existing osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:39.023029 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.023031 7f9d6abe6700  5 queue_op 0x6136fc0 seq 2401 osr(3.60e 0x276dcf0) 2230517 bytes   (queue has 1 ops and 2230517 bytes)
2014-06-12 14:46:39.023038 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.60e( v 173'2559 (0'0,173'2559] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2557 active] do_sub_op osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.023054 7f9d6abe6700 10 osd.2 173 dequeue_op 0x9fad400 finish
2014-06-12 14:46:39.023058 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.023061 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.023069 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533cd80 (1 active)
2014-06-12 14:46:39.023074 7f9d7d968700  5 _do_op 0x6136fc0 seq 2401 osr(3.60e 0x276dcf0)/0x276dcf0 start
2014-06-12 14:46:39.023079 7f9d7d968700 10 _do_transaction on 0x33f3ad30
2014-06-12 14:46:39.023086 7f9d7d968700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:39.023219 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:39.023254 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.023317 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.023419 7f9d7d968700 15 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:39.023429 7f9d7d968700 10 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1 r = 0
2014-06-12 14:46:39.023437 7f9d7d968700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:39.023459 7f9d7d968700 15 _write 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 0~2228224
2014-06-12 14:46:39.023741 7f9d7d968700 10 lookup_strip_header done  cid 3.60e_head oid ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:39.023859 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:39.032217 7f9d7d968700 10 _generic_write 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 0~2228224 = 0
2014-06-12 14:46:39.032352 7f9d7d968700 15 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:39.032368 7f9d7d968700 10 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 = 0
2014-06-12 14:46:39.032373 7f9d7d968700 15 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:39.032377 7f9d7d968700 10 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 = 0
2014-06-12 14:46:39.032716 7f9d7d968700 20 set_header setting 17046 cid 3.60e_head oid ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 parent seq 0
2014-06-12 14:46:39.032875 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.032890 7f9d7d968700 20 set_header setting 2240 cid meta oid 1fe13dcb/pglog_3.60e/0//-1 parent seq 0
2014-06-12 14:46:39.032899 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:39.042380 7f9d7d968700 10 _do_op 0x6136fc0 seq 2401 r = 0, finisher 0x63ce540 0
2014-06-12 14:46:39.042404 7f9d7d968700 10 _finish_op 0x6136fc0 seq 2401 osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:39.042420 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533cd80 (0 active)
2014-06-12 14:46:39.042421 7f9d7715b700  5 -- op tracker -- , seq: 747926, time: 2014-06-12 14:46:39.042411, event: commit_sent, op: osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.042436 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.042438 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2559 (0'0,173'2559] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2557 active] sub_op_modify_commit on op osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:39.042466 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:39.042497 7f9d7795c700  5 -- op tracker -- , seq: 747926, time: 2014-06-12 14:46:39.042492, event: sub_op_applied, op: osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.042508 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2559 (0'0,173'2559] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2558 active] sub_op_modify_applied on 0x33f3ac40 op osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.042526 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2559 (0'0,173'2559] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2558 active] op_applied on primary on version 173'2559
2014-06-12 14:46:39.042538 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2559 (0'0,173'2559] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2558 active] op_applied on replica on version 173'2559
2014-06-12 14:46:39.042563 7f9d7795c700  5 -- op tracker -- , seq: 747926, time: 2014-06-12 14:46:39.042558, event: done, op: osd_sub_op(client.4296.0:143671 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2559 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.094611 7f9d56432700  5 -- op tracker -- , seq: 747927, time: 2014-06-12 14:46:39.093768, event: header_read, op: osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.094626 7f9d56432700  5 -- op tracker -- , seq: 747927, time: 2014-06-12 14:46:39.093771, event: throttled, op: osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.094635 7f9d56432700  5 -- op tracker -- , seq: 747927, time: 2014-06-12 14:46:39.094557, event: all_read, op: osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.094644 7f9d56432700  5 -- op tracker -- , seq: 747927, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.094648 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:39.094656 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:39.094673 7f9d56432700 15 osd.2 173 enqueue_op 0x9fac300 prio 127 cost 516903 latency 0.000904 osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.094746 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.094755 7f9d66bde700 10 osd.2 173 dequeue_op 0x9fac300 prio 127 cost 516903 latency 0.000986 osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3ae( v 173'2955 (0'0,173'2955] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2954 active]
2014-06-12 14:46:39.094777 7f9d66bde700  5 -- op tracker -- , seq: 747927, time: 2014-06-12 14:46:39.094773, event: reached_pg, op: osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.094786 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2955 (0'0,173'2955] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2954 active] handle_message: 0x9fac300
2014-06-12 14:46:39.094799 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2955 (0'0,173'2955] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2954 active] sub_op_modify trans b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 v 173'2956 (transaction) 174
2014-06-12 14:46:39.094816 7f9d66bde700  5 -- op tracker -- , seq: 747927, time: 2014-06-12 14:46:39.094812, event: started, op: osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.094859 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2955 (0'0,173'2955] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2954 active] append_log log((0'0,173'2955], crt=0'0) [173'2956 (173'2955) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143734 2014-06-12 08:44:42.331187]
2014-06-12 14:46:39.094888 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2956 (0'0,173'2956] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2954 active] add_log_entry 173'2956 (173'2955) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143734 2014-06-12 08:44:42.331187
2014-06-12 14:46:39.094912 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2956 (0'0,173'2956] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2954 active] append_log  adding 1 keys
2014-06-12 14:46:39.094949 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2956, trimmed: 
2014-06-12 14:46:39.094975 7f9d66bde700  5 -- op tracker -- , seq: 747927, time: 2014-06-12 14:46:39.094971, event: started, op: osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.094986 7f9d66bde700  5 queue_transactions existing osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:39.094990 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.094992 7f9d66bde700  5 queue_op 0x70c53e0 seq 2684 osr(3.3ae 0x276d6c0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:39.094999 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2956 (0'0,173'2956] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2954 active] do_sub_op osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.095014 7f9d66bde700 10 osd.2 173 dequeue_op 0x9fac300 finish
2014-06-12 14:46:39.095018 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.095021 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.095028 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478a20 (1 active)
2014-06-12 14:46:39.095033 7f9d7c165700  5 _do_op 0x70c53e0 seq 2684 osr(3.3ae 0x276d6c0)/0x276d6c0 start
2014-06-12 14:46:39.095038 7f9d7c165700 10 _do_transaction on 0x20c42ef0
2014-06-12 14:46:39.095044 7f9d7c165700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.095144 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.095165 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.095212 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.095229 7f9d7c165700 15 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.095237 7f9d7c165700 10 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1 r = 0
2014-06-12 14:46:39.095244 7f9d7c165700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.095258 7f9d7c165700 15 _write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 1527808~516096
2014-06-12 14:46:39.095382 7f9d7c165700 10 lookup_strip_header done  cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:39.095471 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:39.097013 7f9d7c165700 10 _generic_write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 1527808~516096 = 0
2014-06-12 14:46:39.097064 7f9d7c165700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:39.097083 7f9d7c165700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:39.097091 7f9d7c165700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:39.097100 7f9d7c165700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:39.097282 7f9d7c165700 20 set_header setting 16695 cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 parent seq 0
2014-06-12 14:46:39.097352 7f9d7c165700 20 set_header setting 2256 cid meta oid 1fe0bb56/pglog_3.3ae/0//-1 parent seq 0
2014-06-12 14:46:39.097369 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.097381 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:39.099948 7f9d7c165700 10 _do_op 0x70c53e0 seq 2684 r = 0, finisher 0x73f8a00 0
2014-06-12 14:46:39.100004 7f9d7715b700  5 -- op tracker -- , seq: 747927, time: 2014-06-12 14:46:39.099991, event: commit_sent, op: osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.100029 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2956 (0'0,173'2956] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2954 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:39.100062 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:39.100090 7f9d7c165700 10 _finish_op 0x70c53e0 seq 2684 osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:39.100099 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478a20 (0 active)
2014-06-12 14:46:39.100109 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.100134 7f9d7795c700  5 -- op tracker -- , seq: 747927, time: 2014-06-12 14:46:39.100128, event: sub_op_applied, op: osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.100144 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2956 (0'0,173'2956] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2955 active] sub_op_modify_applied on 0x20c42e00 op osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.100163 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2956 (0'0,173'2956] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2955 active] op_applied on primary on version 173'2956
2014-06-12 14:46:39.100174 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2956 (0'0,173'2956] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2955 active] op_applied on replica on version 173'2956
2014-06-12 14:46:39.100201 7f9d7795c700  5 -- op tracker -- , seq: 747927, time: 2014-06-12 14:46:39.100196, event: done, op: osd_sub_op(client.4317.0:143734 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2956 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.146997 7f9d56432700  5 -- op tracker -- , seq: 747928, time: 2014-06-12 14:46:39.146436, event: header_read, op: osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.147015 7f9d56432700  5 -- op tracker -- , seq: 747928, time: 2014-06-12 14:46:39.146439, event: throttled, op: osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.147028 7f9d56432700  5 -- op tracker -- , seq: 747928, time: 2014-06-12 14:46:39.146950, event: all_read, op: osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.147041 7f9d56432700  5 -- op tracker -- , seq: 747928, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.147048 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:39.147059 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:39.147070 7f9d56432700 15 osd.2 173 enqueue_op 0x9fac200 prio 127 cost 250663 latency 0.000634 osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.147114 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.147124 7f9d69be4700 10 osd.2 173 dequeue_op 0x9fac200 prio 127 cost 250663 latency 0.000688 osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.71f( v 173'2693 (0'0,173'2693] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2692 active]
2014-06-12 14:46:39.147157 7f9d69be4700  5 -- op tracker -- , seq: 747928, time: 2014-06-12 14:46:39.147151, event: reached_pg, op: osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.147170 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2693 (0'0,173'2693] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2692 active] handle_message: 0x9fac200
2014-06-12 14:46:39.147189 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2693 (0'0,173'2693] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2692 active] sub_op_modify trans c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 v 173'2694 (transaction) 174
2014-06-12 14:46:39.147218 7f9d69be4700  5 -- op tracker -- , seq: 747928, time: 2014-06-12 14:46:39.147212, event: started, op: osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.147240 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2693 (0'0,173'2693] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2692 active] append_log log((0'0,173'2693], crt=0'0) [173'2694 (173'622) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145281 2014-06-12 08:45:22.163402]
2014-06-12 14:46:39.147270 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2694 (0'0,173'2694] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2692 active] add_log_entry 173'2694 (173'622) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145281 2014-06-12 08:45:22.163402
2014-06-12 14:46:39.147305 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2694 (0'0,173'2694] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2692 active] append_log  adding 1 keys
2014-06-12 14:46:39.147352 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2694, trimmed: 
2014-06-12 14:46:39.147389 7f9d69be4700  5 -- op tracker -- , seq: 747928, time: 2014-06-12 14:46:39.147384, event: started, op: osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.147402 7f9d69be4700  5 queue_transactions existing osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:39.147408 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.147411 7f9d69be4700  5 queue_op 0x56dc900 seq 2526 osr(3.71f 0x276de50) 252149 bytes   (queue has 1 ops and 252149 bytes)
2014-06-12 14:46:39.147422 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.71f( v 173'2694 (0'0,173'2694] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2692 active] do_sub_op osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.147446 7f9d69be4700 10 osd.2 173 dequeue_op 0x9fac200 finish
2014-06-12 14:46:39.147452 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.147456 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.147471 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d0240 (1 active)
2014-06-12 14:46:39.147480 7f9d83173700  5 _do_op 0x56dc900 seq 2526 osr(3.71f 0x276de50)/0x276de50 start
2014-06-12 14:46:39.147489 7f9d83173700 10 _do_transaction on 0x2353dcf0
2014-06-12 14:46:39.147499 7f9d83173700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.147767 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.147804 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.147852 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.147876 7f9d83173700 15 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.147884 7f9d83173700 10 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1 r = 0
2014-06-12 14:46:39.147891 7f9d83173700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.147912 7f9d83173700 15 _write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 0~249856
2014-06-12 14:46:39.148047 7f9d83173700 10 lookup_strip_header done  cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:39.148328 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:39.149306 7f9d83173700 10 _generic_write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 0~249856 = 0
2014-06-12 14:46:39.149363 7f9d83173700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:39.149386 7f9d83173700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:39.149397 7f9d83173700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:39.149411 7f9d83173700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:39.149586 7f9d83173700 20 set_header setting 17162 cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 parent seq 0
2014-06-12 14:46:39.149724 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.149750 7f9d83173700 20 set_header setting 2330 cid meta oid 1fe0985b/pglog_3.71f/0//-1 parent seq 0
2014-06-12 14:46:39.149768 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:39.150809 7f9d83173700 10 _do_op 0x56dc900 seq 2526 r = 0, finisher 0x82c6e60 0
2014-06-12 14:46:39.150828 7f9d83173700 10 _finish_op 0x56dc900 seq 2526 osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:39.150835 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d0240 (0 active)
2014-06-12 14:46:39.150838 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.150843 7f9d7715b700  5 -- op tracker -- , seq: 747928, time: 2014-06-12 14:46:39.150835, event: commit_sent, op: osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.150852 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2694 (0'0,173'2694] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2692 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:39.150875 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:39.150903 7f9d7795c700  5 -- op tracker -- , seq: 747928, time: 2014-06-12 14:46:39.150898, event: sub_op_applied, op: osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.150911 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2694 (0'0,173'2694] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2693 active] sub_op_modify_applied on 0x2353dc00 op osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.150926 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2694 (0'0,173'2694] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2693 active] op_applied on primary on version 173'2694
2014-06-12 14:46:39.150932 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2694 (0'0,173'2694] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2693 active] op_applied on replica on version 173'2694
2014-06-12 14:46:39.150950 7f9d7795c700  5 -- op tracker -- , seq: 747928, time: 2014-06-12 14:46:39.150947, event: done, op: osd_sub_op(client.4308.0:145281 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.167672 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:39.167749 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:39.220428 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:39.220480 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:39.347958 7f9d56432700  5 -- op tracker -- , seq: 747929, time: 2014-06-12 14:46:39.347159, event: header_read, op: osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.347985 7f9d56432700  5 -- op tracker -- , seq: 747929, time: 2014-06-12 14:46:39.347162, event: throttled, op: osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.347994 7f9d56432700  5 -- op tracker -- , seq: 747929, time: 2014-06-12 14:46:39.347905, event: all_read, op: osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.348003 7f9d56432700  5 -- op tracker -- , seq: 747929, time: 0.000000, event: dispatched, op: osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.348009 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:39.348017 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:39.348027 7f9d56432700 15 osd.2 173 enqueue_op 0x9fad600 prio 127 cost 516903 latency 0.000867 osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.348086 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.348098 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x9fad600 prio 127 cost 516903 latency 0.000938 osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.60e( v 173'2559 (0'0,173'2559] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2558 active]
2014-06-12 14:46:39.348120 7f9d6a3e5700  5 -- op tracker -- , seq: 747929, time: 2014-06-12 14:46:39.348116, event: reached_pg, op: osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.348129 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2559 (0'0,173'2559] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2558 active] handle_message: 0x9fad600
2014-06-12 14:46:39.348138 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2559 (0'0,173'2559] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2558 active] sub_op_modify trans ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 v 173'2560 (transaction) 174
2014-06-12 14:46:39.348156 7f9d6a3e5700  5 -- op tracker -- , seq: 747929, time: 2014-06-12 14:46:39.348153, event: started, op: osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.348170 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2559 (0'0,173'2559] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2558 active] append_log log((0'0,173'2559], crt=0'0) [173'2560 (173'2559) modify   ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 by client.4296.0:143672 2014-06-12 08:45:41.179395]
2014-06-12 14:46:39.348193 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2560 (0'0,173'2560] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2558 active] add_log_entry 173'2560 (173'2559) modify   ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 by client.4296.0:143672 2014-06-12 08:45:41.179395
2014-06-12 14:46:39.348212 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2560 (0'0,173'2560] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2558 active] append_log  adding 1 keys
2014-06-12 14:46:39.348240 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2560, trimmed: 
2014-06-12 14:46:39.348258 7f9d6a3e5700  5 -- op tracker -- , seq: 747929, time: 2014-06-12 14:46:39.348255, event: started, op: osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.348266 7f9d6a3e5700  5 queue_transactions existing osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:39.348269 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.348270 7f9d6a3e5700  5 queue_op 0x142a9ce0 seq 2402 osr(3.60e 0x276dcf0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:39.348282 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533cd80 (1 active)
2014-06-12 14:46:39.348276 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.60e( v 173'2560 (0'0,173'2560] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2558 active] do_sub_op osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.348288 7f9d82972700  5 _do_op 0x142a9ce0 seq 2402 osr(3.60e 0x276dcf0)/0x276dcf0 start
2014-06-12 14:46:39.348289 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x9fad600 finish
2014-06-12 14:46:39.348293 7f9d82972700 10 _do_transaction on 0x25e122b0
2014-06-12 14:46:39.348292 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.348295 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.348299 7f9d82972700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:39.348391 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:39.348416 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.348457 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.348542 7f9d82972700 15 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:39.348548 7f9d82972700 10 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1 r = 0
2014-06-12 14:46:39.348552 7f9d82972700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:39.348565 7f9d82972700 15 _write 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 2228224~516096
2014-06-12 14:46:39.348656 7f9d82972700 10 lookup_strip_header done  cid 3.60e_head oid ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:39.348852 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:39.350206 7f9d82972700 10 _generic_write 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 2228224~516096 = 0
2014-06-12 14:46:39.350257 7f9d82972700 15 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:39.350272 7f9d82972700 10 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 = 0
2014-06-12 14:46:39.350279 7f9d82972700 15 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:39.350288 7f9d82972700 10 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 = 0
2014-06-12 14:46:39.350496 7f9d82972700 20 set_header setting 17046 cid 3.60e_head oid ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 parent seq 0
2014-06-12 14:46:39.350574 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.350591 7f9d82972700 20 set_header setting 2240 cid meta oid 1fe13dcb/pglog_3.60e/0//-1 parent seq 0
2014-06-12 14:46:39.350601 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:39.352664 7f9d82972700 10 _do_op 0x142a9ce0 seq 2402 r = 0, finisher 0x5d40080 0
2014-06-12 14:46:39.352701 7f9d7715b700  5 -- op tracker -- , seq: 747929, time: 2014-06-12 14:46:39.352693, event: commit_sent, op: osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.352712 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2560 (0'0,173'2560] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2558 active] sub_op_modify_commit on op osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:39.352737 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:39.352763 7f9d82972700 10 _finish_op 0x142a9ce0 seq 2402 osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:39.352785 7f9d7795c700  5 -- op tracker -- , seq: 747929, time: 2014-06-12 14:46:39.352780, event: sub_op_applied, op: osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.352796 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2560 (0'0,173'2560] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2559 active] sub_op_modify_applied on 0x25e121c0 op osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.352809 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2560 (0'0,173'2560] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2559 active] op_applied on primary on version 173'2560
2014-06-12 14:46:39.352817 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2560 (0'0,173'2560] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2559 active] op_applied on replica on version 173'2560
2014-06-12 14:46:39.352846 7f9d82972700  5 -- op tracker -- , seq: 747929, time: 2014-06-12 14:46:39.352838, event: done, op: osd_sub_op(client.4296.0:143672 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2560 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.352875 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533cd80 (0 active)
2014-06-12 14:46:39.352880 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.368376 7f9d56432700  5 -- op tracker -- , seq: 747930, time: 2014-06-12 14:46:39.367437, event: header_read, op: osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.368385 7f9d56432700  5 -- op tracker -- , seq: 747930, time: 2014-06-12 14:46:39.367440, event: throttled, op: osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.368401 7f9d56432700  5 -- op tracker -- , seq: 747930, time: 2014-06-12 14:46:39.368337, event: all_read, op: osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.368407 7f9d56432700  5 -- op tracker -- , seq: 747930, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.368411 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:39.368417 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:39.368423 7f9d56432700 15 osd.2 173 enqueue_op 0x6d11f00 prio 127 cost 516903 latency 0.000986 osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.368449 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.368456 7f9d653db700 10 osd.2 173 dequeue_op 0x6d11f00 prio 127 cost 516903 latency 0.001019 osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.71f( v 173'2694 (0'0,173'2694] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2693 active]
2014-06-12 14:46:39.368473 7f9d653db700  5 -- op tracker -- , seq: 747930, time: 2014-06-12 14:46:39.368470, event: reached_pg, op: osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.368480 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2694 (0'0,173'2694] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2693 active] handle_message: 0x6d11f00
2014-06-12 14:46:39.368489 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2694 (0'0,173'2694] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2693 active] sub_op_modify trans c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 v 173'2695 (transaction) 174
2014-06-12 14:46:39.368500 7f9d653db700  5 -- op tracker -- , seq: 747930, time: 2014-06-12 14:46:39.368497, event: started, op: osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.368512 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2694 (0'0,173'2694] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2693 active] append_log log((0'0,173'2694], crt=0'0) [173'2695 (173'2694) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145282 2014-06-12 08:45:26.743781]
2014-06-12 14:46:39.368529 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2695 (0'0,173'2695] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2693 active] add_log_entry 173'2695 (173'2694) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145282 2014-06-12 08:45:26.743781
2014-06-12 14:46:39.368546 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2695 (0'0,173'2695] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2693 active] append_log  adding 1 keys
2014-06-12 14:46:39.368570 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2695, trimmed: 
2014-06-12 14:46:39.368589 7f9d653db700  5 -- op tracker -- , seq: 747930, time: 2014-06-12 14:46:39.368586, event: started, op: osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.368597 7f9d653db700  5 queue_transactions existing osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:39.368600 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.368601 7f9d653db700  5 queue_op 0x6821f80 seq 2527 osr(3.71f 0x276de50) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:39.368609 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d0240 (1 active)
2014-06-12 14:46:39.368615 7f9d81970700  5 _do_op 0x6821f80 seq 2527 osr(3.71f 0x276de50)/0x276de50 start
2014-06-12 14:46:39.368606 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.71f( v 173'2695 (0'0,173'2695] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2693 active] do_sub_op osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.368619 7f9d653db700 10 osd.2 173 dequeue_op 0x6d11f00 finish
2014-06-12 14:46:39.368620 7f9d81970700 10 _do_transaction on 0x62322b0
2014-06-12 14:46:39.368622 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.368625 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.368626 7f9d81970700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.368721 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.368746 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.368783 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.368804 7f9d81970700 15 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.368809 7f9d81970700 10 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1 r = 0
2014-06-12 14:46:39.368813 7f9d81970700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.368910 7f9d81970700 15 _write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 249856~516096
2014-06-12 14:46:39.369034 7f9d81970700 10 lookup_strip_header done  cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:39.369162 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:39.370564 7f9d81970700 10 _generic_write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 249856~516096 = 0
2014-06-12 14:46:39.370618 7f9d81970700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:39.370631 7f9d81970700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:39.370635 7f9d81970700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:39.370639 7f9d81970700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:39.370790 7f9d81970700 20 set_header setting 17162 cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 parent seq 0
2014-06-12 14:46:39.370845 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.370856 7f9d81970700 20 set_header setting 2330 cid meta oid 1fe0985b/pglog_3.71f/0//-1 parent seq 0
2014-06-12 14:46:39.370862 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:39.373005 7f9d81970700 10 _do_op 0x6821f80 seq 2527 r = 0, finisher 0x90f68c0 0
2014-06-12 14:46:39.373021 7f9d81970700 10 _finish_op 0x6821f80 seq 2527 osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:39.373028 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d0240 (0 active)
2014-06-12 14:46:39.373038 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.373039 7f9d7715b700  5 -- op tracker -- , seq: 747930, time: 2014-06-12 14:46:39.373028, event: commit_sent, op: osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.373058 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2695 (0'0,173'2695] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2693 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:39.373077 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:39.373120 7f9d7795c700  5 -- op tracker -- , seq: 747930, time: 2014-06-12 14:46:39.373113, event: sub_op_applied, op: osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.373128 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2695 (0'0,173'2695] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2694 active] sub_op_modify_applied on 0x62321c0 op osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.373138 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2695 (0'0,173'2695] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2694 active] op_applied on primary on version 173'2695
2014-06-12 14:46:39.373144 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2695 (0'0,173'2695] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2694 active] op_applied on replica on version 173'2695
2014-06-12 14:46:39.373161 7f9d7795c700  5 -- op tracker -- , seq: 747930, time: 2014-06-12 14:46:39.373159, event: done, op: osd_sub_op(client.4308.0:145282 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.477773 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:39.477796 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:39.477798 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:39.477800 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:39.477799 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:39.477801 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 4; oldest is 6.316457 seconds old
2014-06-12 14:46:39.499975 7f9d56432700  5 -- op tracker -- , seq: 747931, time: 2014-06-12 14:46:39.499266, event: header_read, op: osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.499986 7f9d56432700  5 -- op tracker -- , seq: 747931, time: 2014-06-12 14:46:39.499268, event: throttled, op: osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.499996 7f9d56432700  5 -- op tracker -- , seq: 747931, time: 2014-06-12 14:46:39.499942, event: all_read, op: osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.500000 7f9d56432700  5 -- op tracker -- , seq: 747931, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.500005 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:39.500010 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:39.500015 7f9d56432700 15 osd.2 173 enqueue_op 0x6d10700 prio 127 cost 516903 latency 0.000749 osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.500068 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.500080 7f9d63bd8700 10 osd.2 173 dequeue_op 0x6d10700 prio 127 cost 516903 latency 0.000813 osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3ae( v 173'2956 (0'0,173'2956] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2955 active]
2014-06-12 14:46:39.500101 7f9d63bd8700  5 -- op tracker -- , seq: 747931, time: 2014-06-12 14:46:39.500097, event: reached_pg, op: osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.500111 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2956 (0'0,173'2956] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2955 active] handle_message: 0x6d10700
2014-06-12 14:46:39.500122 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2956 (0'0,173'2956] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2955 active] sub_op_modify trans b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 v 173'2957 (transaction) 174
2014-06-12 14:46:39.500135 7f9d63bd8700  5 -- op tracker -- , seq: 747931, time: 2014-06-12 14:46:39.500133, event: started, op: osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.500149 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2956 (0'0,173'2956] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2955 active] append_log log((0'0,173'2956], crt=0'0) [173'2957 (173'2956) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143735 2014-06-12 08:44:47.310183]
2014-06-12 14:46:39.500168 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2957 (0'0,173'2957] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2955 active] add_log_entry 173'2957 (173'2956) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143735 2014-06-12 08:44:47.310183
2014-06-12 14:46:39.500184 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2957 (0'0,173'2957] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2955 active] append_log  adding 1 keys
2014-06-12 14:46:39.500208 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2957, trimmed: 
2014-06-12 14:46:39.500226 7f9d63bd8700  5 -- op tracker -- , seq: 747931, time: 2014-06-12 14:46:39.500223, event: started, op: osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.500235 7f9d63bd8700  5 queue_transactions existing osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:39.500238 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.500239 7f9d63bd8700  5 queue_op 0x5e7dd40 seq 2685 osr(3.3ae 0x276d6c0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:39.500248 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478a20 (1 active)
2014-06-12 14:46:39.500254 7f9d7c966700  5 _do_op 0x5e7dd40 seq 2685 osr(3.3ae 0x276d6c0)/0x276d6c0 start
2014-06-12 14:46:39.500244 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2957 (0'0,173'2957] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2955 active] do_sub_op osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.500257 7f9d63bd8700 10 osd.2 173 dequeue_op 0x6d10700 finish
2014-06-12 14:46:39.500258 7f9d7c966700 10 _do_transaction on 0x268380f0
2014-06-12 14:46:39.500261 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.500263 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.500265 7f9d7c966700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.500359 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.500381 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.500417 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.500436 7f9d7c966700 15 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.500441 7f9d7c966700 10 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1 r = 0
2014-06-12 14:46:39.500445 7f9d7c966700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.500598 7f9d7c966700 15 _write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 2043904~516096
2014-06-12 14:46:39.500696 7f9d7c966700 10 lookup_strip_header done  cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:39.500840 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:39.502040 7f9d7c966700 10 _generic_write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 2043904~516096 = 0
2014-06-12 14:46:39.502083 7f9d7c966700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:39.502091 7f9d7c966700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:39.502095 7f9d7c966700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:39.502100 7f9d7c966700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:39.502241 7f9d7c966700 20 set_header setting 16695 cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 parent seq 0
2014-06-12 14:46:39.502291 7f9d7c966700 20 set_header setting 2256 cid meta oid 1fe0bb56/pglog_3.3ae/0//-1 parent seq 0
2014-06-12 14:46:39.502303 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.502308 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:39.505234 7f9d7c966700 10 _do_op 0x5e7dd40 seq 2685 r = 0, finisher 0x73f8a00 0
2014-06-12 14:46:39.505255 7f9d7c966700 10 _finish_op 0x5e7dd40 seq 2685 osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:39.505263 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478a20 (0 active)
2014-06-12 14:46:39.505274 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.505289 7f9d7715b700  5 -- op tracker -- , seq: 747931, time: 2014-06-12 14:46:39.505268, event: commit_sent, op: osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.505306 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2957 (0'0,173'2957] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2955 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:39.505339 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:39.505420 7f9d7795c700  5 -- op tracker -- , seq: 747931, time: 2014-06-12 14:46:39.505414, event: sub_op_applied, op: osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.505431 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2957 (0'0,173'2957] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2956 active] sub_op_modify_applied on 0x26838000 op osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.505451 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2957 (0'0,173'2957] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2956 active] op_applied on primary on version 173'2957
2014-06-12 14:46:39.505458 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2957 (0'0,173'2957] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2956 active] op_applied on replica on version 173'2957
2014-06-12 14:46:39.505478 7f9d7795c700  5 -- op tracker -- , seq: 747931, time: 2014-06-12 14:46:39.505475, event: done, op: osd_sub_op(client.4317.0:143735 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2957 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.538337 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:39.539899 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:39.547810 7f9d50844700  5 -- op tracker -- , seq: 747932, time: 2014-06-12 14:46:39.545294, event: header_read, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.547830 7f9d50844700  5 -- op tracker -- , seq: 747932, time: 2014-06-12 14:46:39.545299, event: throttled, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.547841 7f9d50844700  5 -- op tracker -- , seq: 747932, time: 2014-06-12 14:46:39.547747, event: all_read, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.547851 7f9d50844700  5 -- op tracker -- , seq: 747932, time: 0.000000, event: dispatched, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.547861 7f9d50844700 20 osd.2 173 should_share_map client.4294 172.16.96.220:0/1242079 173
2014-06-12 14:46:39.547893 7f9d50844700 15 osd.2 173 enqueue_op 0x24c9db00 prio 63 cost 1007616 latency 0.002598 osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:39.547932 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.547942 7f9d623d5700 10 osd.2 173 dequeue_op 0x24c9db00 prio 63 cost 1007616 latency 0.002648 osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean]
2014-06-12 14:46:39.547969 7f9d623d5700  5 -- op tracker -- , seq: 747932, time: 2014-06-12 14:46:39.547964, event: reached_pg, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.547979 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:39.547994 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] handle_message: 0x24c9db00
2014-06-12 14:46:39.548003 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] do_op osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:39.548027 7f9d623d5700 15 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 '_'
2014-06-12 14:46:39.559044 7f9d56432700  5 -- op tracker -- , seq: 747933, time: 2014-06-12 14:46:39.557882, event: header_read, op: osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.559059 7f9d56432700  5 -- op tracker -- , seq: 747933, time: 2014-06-12 14:46:39.557884, event: throttled, op: osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.559073 7f9d56432700  5 -- op tracker -- , seq: 747933, time: 2014-06-12 14:46:39.558996, event: all_read, op: osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.559079 7f9d56432700  5 -- op tracker -- , seq: 747933, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.559083 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:39.559088 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:39.559095 7f9d56432700 15 osd.2 173 enqueue_op 0x6d11b00 prio 127 cost 516903 latency 0.001213 osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.559123 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:39.559130 7f9d68be2700 10 osd.2 173 dequeue_op 0x6d11b00 prio 127 cost 516903 latency 0.001248 osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.71f( v 173'2695 (0'0,173'2695] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2694 active]
2014-06-12 14:46:39.559147 7f9d68be2700  5 -- op tracker -- , seq: 747933, time: 2014-06-12 14:46:39.559145, event: reached_pg, op: osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.559155 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2695 (0'0,173'2695] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2694 active] handle_message: 0x6d11b00
2014-06-12 14:46:39.559164 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2695 (0'0,173'2695] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2694 active] sub_op_modify trans c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 v 173'2696 (transaction) 174
2014-06-12 14:46:39.559175 7f9d68be2700  5 -- op tracker -- , seq: 747933, time: 2014-06-12 14:46:39.559173, event: started, op: osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.559193 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2695 (0'0,173'2695] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2694 active] append_log log((0'0,173'2695], crt=0'0) [173'2696 (173'2695) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145283 2014-06-12 08:45:26.947205]
2014-06-12 14:46:39.559216 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2696 (0'0,173'2696] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2694 active] add_log_entry 173'2696 (173'2695) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145283 2014-06-12 08:45:26.947205
2014-06-12 14:46:39.559241 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2696 (0'0,173'2696] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2694 active] append_log  adding 1 keys
2014-06-12 14:46:39.559277 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2696, trimmed: 
2014-06-12 14:46:39.559303 7f9d68be2700  5 -- op tracker -- , seq: 747933, time: 2014-06-12 14:46:39.559299, event: started, op: osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.559313 7f9d68be2700  5 queue_transactions existing osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:39.559318 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.559320 7f9d68be2700  5 queue_op 0x53e3620 seq 2528 osr(3.71f 0x276de50) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:39.559327 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.71f( v 173'2696 (0'0,173'2696] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2694 active] do_sub_op osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.559343 7f9d68be2700 10 osd.2 173 dequeue_op 0x6d11b00 finish
2014-06-12 14:46:39.559346 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:39.559349 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.559358 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d0240 (1 active)
2014-06-12 14:46:39.559363 7f9d7a962700  5 _do_op 0x53e3620 seq 2528 osr(3.71f 0x276de50)/0x276de50 start
2014-06-12 14:46:39.559368 7f9d7a962700 10 _do_transaction on 0x1f286b70
2014-06-12 14:46:39.559375 7f9d7a962700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.559483 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.559508 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.559555 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.559583 7f9d7a962700 15 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.559590 7f9d7a962700 10 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1 r = 0
2014-06-12 14:46:39.559597 7f9d7a962700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.559749 7f9d7a962700 15 _write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 765952~516096
2014-06-12 14:46:39.559854 7f9d7a962700 10 lookup_strip_header done  cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:39.559977 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:39.561396 7f9d7a962700 10 _generic_write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 765952~516096 = 0
2014-06-12 14:46:39.561449 7f9d7a962700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:39.561465 7f9d7a962700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:39.561473 7f9d7a962700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:39.561483 7f9d7a962700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:39.561667 7f9d7a962700 20 set_header setting 17162 cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 parent seq 0
2014-06-12 14:46:39.561738 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.561755 7f9d7a962700 20 set_header setting 2330 cid meta oid 1fe0985b/pglog_3.71f/0//-1 parent seq 0
2014-06-12 14:46:39.561768 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:39.563958 7f9d7a962700 10 _do_op 0x53e3620 seq 2528 r = 0, finisher 0x69e6da0 0
2014-06-12 14:46:39.563981 7f9d7a962700 10 _finish_op 0x53e3620 seq 2528 osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:39.563992 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d0240 (0 active)
2014-06-12 14:46:39.563996 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.564004 7f9d7715b700  5 -- op tracker -- , seq: 747933, time: 2014-06-12 14:46:39.563991, event: commit_sent, op: osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.564026 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2696 (0'0,173'2696] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2694 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:39.564066 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:39.564110 7f9d7795c700  5 -- op tracker -- , seq: 747933, time: 2014-06-12 14:46:39.564103, event: sub_op_applied, op: osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.564122 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2696 (0'0,173'2696] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2695 active] sub_op_modify_applied on 0x1f286a80 op osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.564144 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2696 (0'0,173'2696] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2695 active] op_applied on primary on version 173'2696
2014-06-12 14:46:39.564158 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2696 (0'0,173'2696] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2695 active] op_applied on replica on version 173'2696
2014-06-12 14:46:39.564182 7f9d7795c700  5 -- op tracker -- , seq: 747933, time: 2014-06-12 14:46:39.564178, event: done, op: osd_sub_op(client.4308.0:145283 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.610317 7f9d623d5700 10 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 '_' = 0
2014-06-12 14:46:39.610358 7f9d623d5700 15 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 'snapset'
2014-06-12 14:46:39.610495 7f9d623d5700 10 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 'snapset' = 0
2014-06-12 14:46:39.610510 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] populate_obc_watchers 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.610537 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.610543 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] get_object_context: creating obc from disk: 0xc8dbb80
2014-06-12 14:46:39.610553 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] get_object_context: 0xc8dbb80 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 rwstate(none n=0 w=0) oi: 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3(173'928 client.4294.0:40143 wrlock_by=unknown.0.0:0 dirty s 4194304 uv928) ssc: 0x1550e540 snapset: 0=[]:[]+head
2014-06-12 14:46:39.610569 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] find_object_context 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 @head oi=39913266/rbd_data.109d6b8b4567.000000000000112d/head//3(173'928 client.4294.0:40143 wrlock_by=unknown.0.0:0 dirty s 4194304 uv928)
2014-06-12 14:46:39.610592 7f9d623d5700  5 -- op tracker -- , seq: 747932, time: 2014-06-12 14:46:39.610585, event: started, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.610600 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] execute_ctx 0x5ca3000
2014-06-12 14:46:39.610611 7f9d623d5700  5 -- op tracker -- , seq: 747932, time: 2014-06-12 14:46:39.610608, event: started, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.610615 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] do_op 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] ov 173'928 av 173'2821 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:39.610628 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] do_osd_op 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616]
2014-06-12 14:46:39.610635 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:39.610643 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] do_osd_op  write 0~1007616
2014-06-12 14:46:39.610653 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] do_osd_op_effects on session 0x50a3000
2014-06-12 14:46:39.610660 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] make_writeable 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 snapset=0x1550e578  snapc=0=[]
2014-06-12 14:46:39.610667 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] make_writeable 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:39.610672 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] finish_ctx 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 0x5ca3000 op modify  
2014-06-12 14:46:39.610680 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean]  set mtime to 2014-06-12 08:45:41.924011
2014-06-12 14:46:39.610696 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean]  final snapset 0=[]:[]+head in 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.610714 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean]  zeroing write result code 0
2014-06-12 14:46:39.610724 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] new_repop rep_tid 236287 on osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:39.610734 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] issue_repop rep_tid 236287 o 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.610751 7f9d623d5700  5 -- op tracker -- , seq: 747932, time: 2014-06-12 14:46:39.610748, event: waiting for subops from 27, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.610771 7f9d623d5700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:39.610822 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2820 (0'0,173'2820] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] append_log log((0'0,173'2820], crt=0'0) [173'2821 (173'928) modify   39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 by client.4294.0:144146 2014-06-12 08:45:41.924011]
2014-06-12 14:46:39.610847 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] add_log_entry 173'2821 (173'928) modify   39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 by client.4294.0:144146 2014-06-12 08:45:41.924011
2014-06-12 14:46:39.610860 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] append_log  adding 1 keys
2014-06-12 14:46:39.610881 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2821, trimmed: 
2014-06-12 14:46:39.610899 7f9d623d5700  5 queue_transactions existing osr(3.266 0x276d310)/0x276d310
2014-06-12 14:46:39.610903 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.610904 7f9d623d5700  5 queue_op 0x8097e00 seq 2596 osr(3.266 0x276d310) 1009909 bytes   (queue has 1 ops and 1009909 bytes)
2014-06-12 14:46:39.610931 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f78c0 (1 active)
2014-06-12 14:46:39.610937 7f9d7895e700  5 _do_op 0x8097e00 seq 2596 osr(3.266 0x276d310)/0x276d310 start
2014-06-12 14:46:39.610924 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] eval_repop repgather(0x5390540 173'2821 rep_tid=236287 committed?=0 applied?=0 lock=0 op=osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.610941 7f9d7895e700 10 _do_transaction on 0x2010f740
2014-06-12 14:46:39.610942 7f9d623d5700 10 osd.2 173 dequeue_op 0x24c9db00 finish
2014-06-12 14:46:39.610947 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.610947 7f9d7895e700 15 _omap_setkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.610951 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.611015 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.611123 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.611174 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.611194 7f9d7895e700 15 _omap_rmkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.611207 7f9d7895e700 10 _omap_rmkeys meta/1fe27d1a/pglog_3.266/0//-1 r = 0
2014-06-12 14:46:39.611218 7f9d7895e700 15 _omap_setkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.611231 7f9d7895e700 15 _write 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 0~1007616
2014-06-12 14:46:39.611443 7f9d7895e700 10 lookup_strip_header done  cid 3.266_head oid 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.611605 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:39.614129 7f9d7895e700 10 _generic_write 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 0~1007616 = 0
2014-06-12 14:46:39.614188 7f9d7895e700 15 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.614200 7f9d7895e700 10 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 = 0
2014-06-12 14:46:39.614206 7f9d7895e700 15 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.614210 7f9d7895e700 10 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 = 0
2014-06-12 14:46:39.614412 7f9d7895e700 20 set_header setting 17844 cid 3.266_head oid 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 parent seq 0
2014-06-12 14:46:39.614479 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.614489 7f9d7895e700 20 set_header setting 2030 cid meta oid 1fe27d1a/pglog_3.266/0//-1 parent seq 0
2014-06-12 14:46:39.614495 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:39.618104 7f9d7895e700 10 _do_op 0x8097e00 seq 2596 r = 0, finisher 0x50cf2c0 0x19b2d740
2014-06-12 14:46:39.618129 7f9d7895e700 10 _finish_op 0x8097e00 seq 2596 osr(3.266 0x276d310)/0x276d310
2014-06-12 14:46:39.618138 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f78c0 (0 active)
2014-06-12 14:46:39.618146 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.618139 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] op_commit: 236287
2014-06-12 14:46:39.618178 7f9d7715b700  5 -- op tracker -- , seq: 747932, time: 2014-06-12 14:46:39.618167, event: op_commit, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.618202 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] op_applied: 236287
2014-06-12 14:46:39.618232 7f9d7795c700  5 -- op tracker -- , seq: 747932, time: 2014-06-12 14:46:39.618224, event: op_applied, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.618245 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] op_applied on primary on version 173'2821
2014-06-12 14:46:39.621058 7f9d58c5a700  5 -- op tracker -- , seq: 747934, time: 2014-06-12 14:46:39.621018, event: header_read, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.621074 7f9d58c5a700  5 -- op tracker -- , seq: 747934, time: 2014-06-12 14:46:39.621020, event: throttled, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.621079 7f9d58c5a700  5 -- op tracker -- , seq: 747934, time: 2014-06-12 14:46:39.621033, event: all_read, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.621083 7f9d58c5a700  5 -- op tracker -- , seq: 747934, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.621091 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:39.621094 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:39.621101 7f9d58c5a700 15 osd.2 173 enqueue_op 0x28dd1100 prio 196 cost 0 latency 0.000083 osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0) v2
2014-06-12 14:46:39.621126 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.621134 7f9d643d9700 10 osd.2 173 dequeue_op 0x28dd1100 prio 196 cost 0 latency 0.000116 osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0) v2 pg pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean]
2014-06-12 14:46:39.621145 7f9d58c5a700  5 -- op tracker -- , seq: 747935, time: 2014-06-12 14:46:39.621120, event: header_read, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.621158 7f9d58c5a700  5 -- op tracker -- , seq: 747935, time: 2014-06-12 14:46:39.621120, event: throttled, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.621168 7f9d58c5a700  5 -- op tracker -- , seq: 747935, time: 2014-06-12 14:46:39.621126, event: all_read, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.621181 7f9d58c5a700  5 -- op tracker -- , seq: 747935, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.621185 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:39.621190 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:39.621189 7f9d643d9700  5 -- op tracker -- , seq: 747934, time: 2014-06-12 14:46:39.621148, event: reached_pg, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.621195 7f9d58c5a700 15 osd.2 173 enqueue_op 0x28dd0900 prio 196 cost 0 latency 0.000075 osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:39.621197 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] handle_message: 0x28dd1100
2014-06-12 14:46:39.621217 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:39.621212 7f9d643d9700  5 -- op tracker -- , seq: 747934, time: 2014-06-12 14:46:39.621209, event: started, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.621221 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] sub_op_modify_reply: tid 236287 op  ack_type 1 from 27
2014-06-12 14:46:39.621236 7f9d643d9700  5 -- op tracker -- , seq: 747932, time: 2014-06-12 14:46:39.621232, event: sub_op_applied_rec, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.621245 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] repop_all_applied: repop tid 236287 all applied 
2014-06-12 14:46:39.621252 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] eval_repop repgather(0x5390540 173'2821 rep_tid=236287 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.621264 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean]  sending ack on repgather(0x5390540 173'2821 rep_tid=236287 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) 0x1687af00
2014-06-12 14:46:39.621289 7f9d643d9700 10 osd.2 173 dequeue_op 0x28dd1100 finish
2014-06-12 14:46:39.621297 7f9d643d9700  5 -- op tracker -- , seq: 747934, time: 2014-06-12 14:46:39.621294, event: done, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.621296 7f9d633d7700 10 osd.2 173 dequeue_op 0x28dd0900 prio 196 cost 0 latency 0.000176 osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2 pg pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean]
2014-06-12 14:46:39.621315 7f9d633d7700  5 -- op tracker -- , seq: 747935, time: 2014-06-12 14:46:39.621311, event: reached_pg, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.621323 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] handle_message: 0x28dd0900
2014-06-12 14:46:39.621331 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:39.621334 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.621337 7f9d633d7700  5 -- op tracker -- , seq: 747935, time: 2014-06-12 14:46:39.621334, event: started, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.621343 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] sub_op_modify_reply: tid 236287 op  ack_type 4 from 27
2014-06-12 14:46:39.621354 7f9d633d7700  5 -- op tracker -- , seq: 747932, time: 2014-06-12 14:46:39.621350, event: sub_op_commit_rec, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.621362 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2820 crt=0'0 lcod 173'2819 mlcod 173'2819 active+clean] repop_all_committed: repop tid 236287 all committed 
2014-06-12 14:46:39.621368 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2819 active+clean] eval_repop repgather(0x5390540 173'2821 rep_tid=236287 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.621378 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2819 active+clean] log_op_stats osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 inb 1007895 outb 0 rlat 0.075994 lat 0.076083
2014-06-12 14:46:39.621389 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2819 active+clean] publish_stats_to_osd 173:3465
2014-06-12 14:46:39.621399 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2819 active+clean]  sending commit on repgather(0x5390540 173'2821 rep_tid=236287 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) 0x10096500
2014-06-12 14:46:39.621421 7f9d633d7700  5 -- op tracker -- , seq: 747932, time: 2014-06-12 14:46:39.621417, event: commit_sent, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.621427 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2819 active+clean]  requeue_ops 
2014-06-12 14:46:39.621434 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean]  removing repgather(0x5390540 173'2821 rep_tid=236287 committed?=1 applied?=1 op=osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.621442 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean]    q front is repgather(0x5390540 173'2821 rep_tid=236287 committed?=1 applied?=1 op=osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.621451 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] remove_repop repgather(0x5390540 173'2821 rep_tid=236287 committed?=1 applied?=1 op=osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.621459 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean]  requeue_ops 
2014-06-12 14:46:39.621477 7f9d633d7700  5 -- op tracker -- , seq: 747932, time: 2014-06-12 14:46:39.621474, event: done, op: osd_op(client.4294.0:144146 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1007616] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.621489 7f9d633d7700 10 osd.2 173 dequeue_op 0x28dd0900 finish
2014-06-12 14:46:39.621493 7f9d633d7700  5 -- op tracker -- , seq: 747935, time: 2014-06-12 14:46:39.621491, event: done, op: osd_sub_op_reply(client.4294.0:144146 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.621499 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.621501 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.627071 7f9d50844700  5 -- op tracker -- , seq: 747936, time: 2014-06-12 14:46:39.624973, event: header_read, op: osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.627087 7f9d50844700  5 -- op tracker -- , seq: 747936, time: 2014-06-12 14:46:39.624975, event: throttled, op: osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.627097 7f9d50844700  5 -- op tracker -- , seq: 747936, time: 2014-06-12 14:46:39.627034, event: all_read, op: osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.627107 7f9d50844700  5 -- op tracker -- , seq: 747936, time: 0.000000, event: dispatched, op: osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.627115 7f9d50844700 20 osd.2 173 should_share_map client.4294 172.16.96.220:0/1242079 173
2014-06-12 14:46:39.627146 7f9d50844700 15 osd.2 173 enqueue_op 0x24c9da00 prio 63 cost 1032192 latency 0.002172 osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:39.627175 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.627185 7f9d61bd4700 10 osd.2 173 dequeue_op 0x24c9da00 prio 63 cost 1032192 latency 0.002212 osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean]
2014-06-12 14:46:39.627205 7f9d61bd4700  5 -- op tracker -- , seq: 747936, time: 2014-06-12 14:46:39.627201, event: reached_pg, op: osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.627212 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:39.627222 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] handle_message: 0x24c9da00
2014-06-12 14:46:39.627229 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] do_op osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:39.627247 7f9d61bd4700 15 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 '_'
2014-06-12 14:46:39.627387 7f9d61bd4700 10 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 '_' = 0
2014-06-12 14:46:39.627402 7f9d61bd4700 15 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 'snapset'
2014-06-12 14:46:39.627453 7f9d61bd4700 10 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 'snapset' = 0
2014-06-12 14:46:39.627461 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] populate_obc_watchers 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.627469 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.627475 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] get_object_context: creating obc from disk: 0x58fd080
2014-06-12 14:46:39.627481 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] get_object_context: 0x58fd080 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 rwstate(none n=0 w=0) oi: 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3(173'2821 client.4294.0:144146 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2821) ssc: 0x5d5ab60 snapset: 0=[]:[]+head
2014-06-12 14:46:39.627491 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] find_object_context 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 @head oi=39913266/rbd_data.109d6b8b4567.000000000000112d/head//3(173'2821 client.4294.0:144146 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2821)
2014-06-12 14:46:39.627508 7f9d61bd4700  5 -- op tracker -- , seq: 747936, time: 2014-06-12 14:46:39.627505, event: started, op: osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.627516 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] execute_ctx 0xfd1a400
2014-06-12 14:46:39.627533 7f9d61bd4700  5 -- op tracker -- , seq: 747936, time: 2014-06-12 14:46:39.627528, event: started, op: osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.627540 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] do_op 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] ov 173'2821 av 173'2822 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:39.627550 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] do_osd_op 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192]
2014-06-12 14:46:39.627556 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:39.627566 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] do_osd_op  write 1007616~1032192
2014-06-12 14:46:39.627577 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] do_osd_op_effects on session 0x50a3000
2014-06-12 14:46:39.627586 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] make_writeable 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 snapset=0x5d5ab98  snapc=0=[]
2014-06-12 14:46:39.627595 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] make_writeable 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:39.627603 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] finish_ctx 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 0xfd1a400 op modify  
2014-06-12 14:46:39.627614 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean]  set mtime to 2014-06-12 08:45:42.003683
2014-06-12 14:46:39.627634 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean]  final snapset 0=[]:[]+head in 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.627648 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean]  zeroing write result code 0
2014-06-12 14:46:39.627656 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] new_repop rep_tid 236288 on osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:39.627671 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] issue_repop rep_tid 236288 o 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.627688 7f9d61bd4700  5 -- op tracker -- , seq: 747936, time: 2014-06-12 14:46:39.627685, event: waiting for subops from 27, op: osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.627703 7f9d61bd4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:39.627720 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2821 (0'0,173'2821] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] append_log log((0'0,173'2821], crt=0'0) [173'2822 (173'2821) modify   39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 by client.4294.0:144147 2014-06-12 08:45:42.003683]
2014-06-12 14:46:39.627737 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2821 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] add_log_entry 173'2822 (173'2821) modify   39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 by client.4294.0:144147 2014-06-12 08:45:42.003683
2014-06-12 14:46:39.627750 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2821 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] append_log  adding 1 keys
2014-06-12 14:46:39.627770 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2822, trimmed: 
2014-06-12 14:46:39.627785 7f9d61bd4700  5 queue_transactions existing osr(3.266 0x276d310)/0x276d310
2014-06-12 14:46:39.627788 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.627789 7f9d61bd4700  5 queue_op 0x56b6420 seq 2597 osr(3.266 0x276d310) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:39.627794 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2821 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] eval_repop repgather(0x5360e40 173'2822 rep_tid=236288 committed?=0 applied?=0 lock=0 op=osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.627806 7f9d61bd4700 10 osd.2 173 dequeue_op 0x24c9da00 finish
2014-06-12 14:46:39.627809 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.627811 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.627814 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f78c0 (1 active)
2014-06-12 14:46:39.627822 7f9d7a161700  5 _do_op 0x56b6420 seq 2597 osr(3.266 0x276d310)/0x276d310 start
2014-06-12 14:46:39.627825 7f9d7a161700 10 _do_transaction on 0x3403a600
2014-06-12 14:46:39.627831 7f9d7a161700 15 _omap_setkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.627906 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.627931 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.627963 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.628050 7f9d7a161700 15 _omap_rmkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.628056 7f9d7a161700 10 _omap_rmkeys meta/1fe27d1a/pglog_3.266/0//-1 r = 0
2014-06-12 14:46:39.628060 7f9d7a161700 15 _omap_setkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.628072 7f9d7a161700 15 _write 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 1007616~1032192
2014-06-12 14:46:39.628155 7f9d7a161700 10 lookup_strip_header done  cid 3.266_head oid 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.628386 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:39.631020 7f9d7a161700 10 _generic_write 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 1007616~1032192 = 0
2014-06-12 14:46:39.631088 7f9d7a161700 15 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.631101 7f9d7a161700 10 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 = 0
2014-06-12 14:46:39.631108 7f9d7a161700 15 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.631112 7f9d7a161700 10 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 = 0
2014-06-12 14:46:39.631306 7f9d7a161700 20 set_header setting 17844 cid 3.266_head oid 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 parent seq 0
2014-06-12 14:46:39.631378 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.631388 7f9d7a161700 20 set_header setting 2030 cid meta oid 1fe27d1a/pglog_3.266/0//-1 parent seq 0
2014-06-12 14:46:39.631396 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:39.637125 7f9d7a161700 10 _do_op 0x56b6420 seq 2597 r = 0, finisher 0x4efcba0 0x76c3480
2014-06-12 14:46:39.637187 7f9d7a161700 10 _finish_op 0x56b6420 seq 2597 osr(3.266 0x276d310)/0x276d310
2014-06-12 14:46:39.637160 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2821 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] op_commit: 236288
2014-06-12 14:46:39.637197 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f78c0 (0 active)
2014-06-12 14:46:39.637205 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.637206 7f9d7715b700  5 -- op tracker -- , seq: 747936, time: 2014-06-12 14:46:39.637194, event: op_commit, op: osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.637232 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2821 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] op_applied: 236288
2014-06-12 14:46:39.637264 7f9d7795c700  5 -- op tracker -- , seq: 747936, time: 2014-06-12 14:46:39.637258, event: op_applied, op: osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.637271 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2821 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] op_applied on primary on version 173'2822
2014-06-12 14:46:39.643984 7f9d58c5a700  5 -- op tracker -- , seq: 747937, time: 2014-06-12 14:46:39.643911, event: header_read, op: osd_sub_op_reply(client.4294.0:144147 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.643996 7f9d58c5a700  5 -- op tracker -- , seq: 747937, time: 2014-06-12 14:46:39.643914, event: throttled, op: osd_sub_op_reply(client.4294.0:144147 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.644005 7f9d58c5a700  5 -- op tracker -- , seq: 747937, time: 2014-06-12 14:46:39.643932, event: all_read, op: osd_sub_op_reply(client.4294.0:144147 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.644023 7f9d58c5a700  5 -- op tracker -- , seq: 747937, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:144147 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.644030 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:144147 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:39.644038 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:39.644049 7f9d58c5a700 15 osd.2 173 enqueue_op 0x6f4b000 prio 196 cost 0 latency 0.000137 osd_sub_op_reply(client.4294.0:144147 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:39.644120 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.644143 7f9d673df700 10 osd.2 173 dequeue_op 0x6f4b000 prio 196 cost 0 latency 0.000231 osd_sub_op_reply(client.4294.0:144147 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2 pg pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2821 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean]
2014-06-12 14:46:39.644181 7f9d673df700  5 -- op tracker -- , seq: 747937, time: 2014-06-12 14:46:39.644176, event: reached_pg, op: osd_sub_op_reply(client.4294.0:144147 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.644197 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2821 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] handle_message: 0x6f4b000
2014-06-12 14:46:39.644217 7f9d673df700  5 -- op tracker -- , seq: 747937, time: 2014-06-12 14:46:39.644213, event: started, op: osd_sub_op_reply(client.4294.0:144147 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.644224 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2821 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] sub_op_modify_reply: tid 236288 op  ack_type 4 from 27
2014-06-12 14:46:39.644249 7f9d673df700  5 -- op tracker -- , seq: 747936, time: 2014-06-12 14:46:39.644241, event: sub_op_commit_rec, op: osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.644259 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2821 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] repop_all_applied: repop tid 236288 all applied 
2014-06-12 14:46:39.644275 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2821 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] eval_repop repgather(0x5360e40 173'2822 rep_tid=236288 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.644295 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2821 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean]  sending ack on repgather(0x5360e40 173'2822 rep_tid=236288 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) 0x1b774280
2014-06-12 14:46:39.644332 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2821 crt=0'0 lcod 173'2820 mlcod 173'2820 active+clean] repop_all_committed: repop tid 236288 all committed 
2014-06-12 14:46:39.644346 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2820 active+clean] eval_repop repgather(0x5360e40 173'2822 rep_tid=236288 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.644364 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2820 active+clean] log_op_stats osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.019357 lat 0.019390
2014-06-12 14:46:39.644383 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2820 active+clean] publish_stats_to_osd 173:3466
2014-06-12 14:46:39.644397 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2820 active+clean]  sending commit on repgather(0x5360e40 173'2822 rep_tid=236288 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) 0x35715680
2014-06-12 14:46:39.644437 7f9d673df700  5 -- op tracker -- , seq: 747936, time: 2014-06-12 14:46:39.644431, event: commit_sent, op: osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.644446 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2820 active+clean]  requeue_ops 
2014-06-12 14:46:39.644457 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean]  removing repgather(0x5360e40 173'2822 rep_tid=236288 committed?=1 applied?=1 op=osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.644475 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean]    q front is repgather(0x5360e40 173'2822 rep_tid=236288 committed?=1 applied?=1 op=osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.644491 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] remove_repop repgather(0x5360e40 173'2822 rep_tid=236288 committed?=1 applied?=1 op=osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.644508 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean]  requeue_ops 
2014-06-12 14:46:39.644544 7f9d673df700  5 -- op tracker -- , seq: 747936, time: 2014-06-12 14:46:39.644531, event: done, op: osd_op(client.4294.0:144147 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 1007616~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.644593 7f9d673df700 10 osd.2 173 dequeue_op 0x6f4b000 finish
2014-06-12 14:46:39.644602 7f9d673df700  5 -- op tracker -- , seq: 747937, time: 2014-06-12 14:46:39.644598, event: done, op: osd_sub_op_reply(client.4294.0:144147 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.644613 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.644617 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.650401 7f9d50844700  5 -- op tracker -- , seq: 747938, time: 2014-06-12 14:46:39.648025, event: header_read, op: osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.650419 7f9d50844700  5 -- op tracker -- , seq: 747938, time: 2014-06-12 14:46:39.648027, event: throttled, op: osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.650425 7f9d50844700  5 -- op tracker -- , seq: 747938, time: 2014-06-12 14:46:39.650369, event: all_read, op: osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.650438 7f9d50844700  5 -- op tracker -- , seq: 747938, time: 0.000000, event: dispatched, op: osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.650443 7f9d50844700 20 osd.2 173 should_share_map client.4294 172.16.96.220:0/1242079 173
2014-06-12 14:46:39.650464 7f9d50844700 15 osd.2 173 enqueue_op 0x2afd400 prio 63 cost 1032192 latency 0.002439 osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:39.650503 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.650512 7f9d67be0700 10 osd.2 173 dequeue_op 0x2afd400 prio 63 cost 1032192 latency 0.002487 osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean]
2014-06-12 14:46:39.650534 7f9d67be0700  5 -- op tracker -- , seq: 747938, time: 2014-06-12 14:46:39.650530, event: reached_pg, op: osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.650542 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:39.650554 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] handle_message: 0x2afd400
2014-06-12 14:46:39.650561 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] do_op osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:39.650579 7f9d67be0700 15 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 '_'
2014-06-12 14:46:39.650720 7f9d67be0700 10 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 '_' = 0
2014-06-12 14:46:39.650733 7f9d67be0700 15 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 'snapset'
2014-06-12 14:46:39.650800 7f9d67be0700 10 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 'snapset' = 0
2014-06-12 14:46:39.650808 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] populate_obc_watchers 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.650816 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.650830 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] get_object_context: creating obc from disk: 0x1e6c4dc0
2014-06-12 14:46:39.650835 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] get_object_context: 0x1e6c4dc0 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 rwstate(none n=0 w=0) oi: 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3(173'2822 client.4294.0:144147 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2822) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:46:39.650845 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] find_object_context 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 @head oi=39913266/rbd_data.109d6b8b4567.000000000000112d/head//3(173'2822 client.4294.0:144147 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2822)
2014-06-12 14:46:39.650861 7f9d67be0700  5 -- op tracker -- , seq: 747938, time: 2014-06-12 14:46:39.650857, event: started, op: osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.650870 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] execute_ctx 0x6245600
2014-06-12 14:46:39.650880 7f9d67be0700  5 -- op tracker -- , seq: 747938, time: 2014-06-12 14:46:39.650877, event: started, op: osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.650884 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] do_op 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] ov 173'2822 av 173'2823 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:39.650897 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] do_osd_op 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192]
2014-06-12 14:46:39.650903 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:39.650912 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] do_osd_op  write 2039808~1032192
2014-06-12 14:46:39.650921 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] do_osd_op_effects on session 0x50a3000
2014-06-12 14:46:39.650928 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] make_writeable 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:46:39.650935 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] make_writeable 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:39.650941 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] finish_ctx 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 0x6245600 op modify  
2014-06-12 14:46:39.650948 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean]  set mtime to 2014-06-12 08:45:42.026741
2014-06-12 14:46:39.650961 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean]  final snapset 0=[]:[]+head in 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.650977 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean]  zeroing write result code 0
2014-06-12 14:46:39.650984 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] new_repop rep_tid 236289 on osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:39.650995 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] issue_repop rep_tid 236289 o 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.651010 7f9d67be0700  5 -- op tracker -- , seq: 747938, time: 2014-06-12 14:46:39.651007, event: waiting for subops from 27, op: osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.651024 7f9d67be0700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:39.651038 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2822 (0'0,173'2822] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] append_log log((0'0,173'2822], crt=0'0) [173'2823 (173'2822) modify   39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 by client.4294.0:144148 2014-06-12 08:45:42.026741]
2014-06-12 14:46:39.651054 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2822 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] add_log_entry 173'2823 (173'2822) modify   39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 by client.4294.0:144148 2014-06-12 08:45:42.026741
2014-06-12 14:46:39.651068 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2822 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] append_log  adding 1 keys
2014-06-12 14:46:39.651114 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2823, trimmed: 
2014-06-12 14:46:39.651131 7f9d67be0700  5 queue_transactions existing osr(3.266 0x276d310)/0x276d310
2014-06-12 14:46:39.651135 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.651136 7f9d67be0700  5 queue_op 0x77048a0 seq 2598 osr(3.266 0x276d310) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:39.651146 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f78c0 (1 active)
2014-06-12 14:46:39.651152 7f9d7f16b700  5 _do_op 0x77048a0 seq 2598 osr(3.266 0x276d310)/0x276d310 start
2014-06-12 14:46:39.651140 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2822 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] eval_repop repgather(0xd5e3380 173'2823 rep_tid=236289 committed?=0 applied?=0 lock=0 op=osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.651156 7f9d7f16b700 10 _do_transaction on 0x3403b080
2014-06-12 14:46:39.651157 7f9d67be0700 10 osd.2 173 dequeue_op 0x2afd400 finish
2014-06-12 14:46:39.651161 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.651163 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.651162 7f9d7f16b700 15 _omap_setkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.651307 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:39.651311 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.651327 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.651346 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:39.651369 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.651395 7f9d7f16b700 15 _omap_rmkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.651404 7f9d7f16b700 10 _omap_rmkeys meta/1fe27d1a/pglog_3.266/0//-1 r = 0
2014-06-12 14:46:39.651409 7f9d7f16b700 15 _omap_setkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.651523 7f9d7f16b700 15 _write 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 2039808~1032192
2014-06-12 14:46:39.651616 7f9d7f16b700 10 lookup_strip_header done  cid 3.266_head oid 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.651758 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:39.654399 7f9d7f16b700 10 _generic_write 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 2039808~1032192 = 0
2014-06-12 14:46:39.654453 7f9d7f16b700 15 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.654465 7f9d7f16b700 10 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 = 0
2014-06-12 14:46:39.654471 7f9d7f16b700 15 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.654476 7f9d7f16b700 10 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 = 0
2014-06-12 14:46:39.654676 7f9d7f16b700 20 set_header setting 17844 cid 3.266_head oid 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 parent seq 0
2014-06-12 14:46:39.654750 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.654761 7f9d7f16b700 20 set_header setting 2030 cid meta oid 1fe27d1a/pglog_3.266/0//-1 parent seq 0
2014-06-12 14:46:39.654768 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:39.659542 7f9d7f16b700 10 _do_op 0x77048a0 seq 2598 r = 0, finisher 0x32e66460 0x1a3f1100
2014-06-12 14:46:39.659577 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2822 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] op_commit: 236289
2014-06-12 14:46:39.659634 7f9d7715b700  5 -- op tracker -- , seq: 747938, time: 2014-06-12 14:46:39.659622, event: op_commit, op: osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.659601 7f9d7f16b700 10 _finish_op 0x77048a0 seq 2598 osr(3.266 0x276d310)/0x276d310
2014-06-12 14:46:39.659656 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f78c0 (0 active)
2014-06-12 14:46:39.659661 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.659662 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2822 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] op_applied: 236289
2014-06-12 14:46:39.659682 7f9d7795c700  5 -- op tracker -- , seq: 747938, time: 2014-06-12 14:46:39.659677, event: op_applied, op: osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.659693 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2822 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] op_applied on primary on version 173'2823
2014-06-12 14:46:39.665380 7f9d58c5a700  5 -- op tracker -- , seq: 747939, time: 2014-06-12 14:46:39.665306, event: header_read, op: osd_sub_op_reply(client.4294.0:144148 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.665407 7f9d58c5a700  5 -- op tracker -- , seq: 747939, time: 2014-06-12 14:46:39.665308, event: throttled, op: osd_sub_op_reply(client.4294.0:144148 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.665416 7f9d58c5a700  5 -- op tracker -- , seq: 747939, time: 2014-06-12 14:46:39.665323, event: all_read, op: osd_sub_op_reply(client.4294.0:144148 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.665424 7f9d58c5a700  5 -- op tracker -- , seq: 747939, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:144148 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.665429 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:144148 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:39.665436 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:39.665446 7f9d58c5a700 15 osd.2 173 enqueue_op 0x6f4a900 prio 196 cost 0 latency 0.000139 osd_sub_op_reply(client.4294.0:144148 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:39.665503 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.665515 7f9d663dd700 10 osd.2 173 dequeue_op 0x6f4a900 prio 196 cost 0 latency 0.000209 osd_sub_op_reply(client.4294.0:144148 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2 pg pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2822 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean]
2014-06-12 14:46:39.665536 7f9d663dd700  5 -- op tracker -- , seq: 747939, time: 2014-06-12 14:46:39.665533, event: reached_pg, op: osd_sub_op_reply(client.4294.0:144148 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.665544 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2822 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] handle_message: 0x6f4a900
2014-06-12 14:46:39.665556 7f9d663dd700  5 -- op tracker -- , seq: 747939, time: 2014-06-12 14:46:39.665553, event: started, op: osd_sub_op_reply(client.4294.0:144148 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.665560 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2822 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] sub_op_modify_reply: tid 236289 op  ack_type 4 from 27
2014-06-12 14:46:39.665575 7f9d663dd700  5 -- op tracker -- , seq: 747938, time: 2014-06-12 14:46:39.665569, event: sub_op_commit_rec, op: osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.665580 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2822 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] repop_all_applied: repop tid 236289 all applied 
2014-06-12 14:46:39.665587 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2822 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] eval_repop repgather(0xd5e3380 173'2823 rep_tid=236289 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.665597 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2822 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean]  sending ack on repgather(0xd5e3380 173'2823 rep_tid=236289 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) 0x35714500
2014-06-12 14:46:39.665628 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2822 crt=0'0 lcod 173'2821 mlcod 173'2821 active+clean] repop_all_committed: repop tid 236289 all committed 
2014-06-12 14:46:39.665640 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2821 active+clean] eval_repop repgather(0xd5e3380 173'2823 rep_tid=236289 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.665661 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2821 active+clean] log_op_stats osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.017602 lat 0.017635
2014-06-12 14:46:39.665684 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2821 active+clean] publish_stats_to_osd 173:3467
2014-06-12 14:46:39.665694 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2821 active+clean]  sending commit on repgather(0xd5e3380 173'2823 rep_tid=236289 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) 0x6824280
2014-06-12 14:46:39.665717 7f9d663dd700  5 -- op tracker -- , seq: 747938, time: 2014-06-12 14:46:39.665712, event: commit_sent, op: osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.665727 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2821 active+clean]  requeue_ops 
2014-06-12 14:46:39.665738 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean]  removing repgather(0xd5e3380 173'2823 rep_tid=236289 committed?=1 applied?=1 op=osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.665749 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean]    q front is repgather(0xd5e3380 173'2823 rep_tid=236289 committed?=1 applied?=1 op=osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.665758 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] remove_repop repgather(0xd5e3380 173'2823 rep_tid=236289 committed?=1 applied?=1 op=osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.665766 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean]  requeue_ops 
2014-06-12 14:46:39.665784 7f9d663dd700  5 -- op tracker -- , seq: 747938, time: 2014-06-12 14:46:39.665780, event: done, op: osd_op(client.4294.0:144148 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 2039808~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.665799 7f9d663dd700 10 osd.2 173 dequeue_op 0x6f4a900 finish
2014-06-12 14:46:39.665804 7f9d663dd700  5 -- op tracker -- , seq: 747939, time: 2014-06-12 14:46:39.665802, event: done, op: osd_sub_op_reply(client.4294.0:144148 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.665825 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.665829 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.672402 7f9d50844700  5 -- op tracker -- , seq: 747940, time: 2014-06-12 14:46:39.669397, event: header_read, op: osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.672426 7f9d50844700  5 -- op tracker -- , seq: 747940, time: 2014-06-12 14:46:39.669398, event: throttled, op: osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.672480 7f9d50844700  5 -- op tracker -- , seq: 747940, time: 2014-06-12 14:46:39.672316, event: all_read, op: osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.672511 7f9d50844700  5 -- op tracker -- , seq: 747940, time: 0.000000, event: dispatched, op: osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.672524 7f9d50844700 20 osd.2 173 should_share_map client.4294 172.16.96.220:0/1242079 173
2014-06-12 14:46:39.672550 7f9d50844700 15 osd.2 173 enqueue_op 0x2afd700 prio 63 cost 1032192 latency 0.003153 osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:39.672621 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.672636 7f9d693e3700 10 osd.2 173 dequeue_op 0x2afd700 prio 63 cost 1032192 latency 0.003238 osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean]
2014-06-12 14:46:39.672682 7f9d693e3700  5 -- op tracker -- , seq: 747940, time: 2014-06-12 14:46:39.672674, event: reached_pg, op: osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.672698 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:39.672718 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] handle_message: 0x2afd700
2014-06-12 14:46:39.672740 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] do_op osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:39.672762 7f9d693e3700 15 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 '_'
2014-06-12 14:46:39.672981 7f9d693e3700 10 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 '_' = 0
2014-06-12 14:46:39.673010 7f9d693e3700 15 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 'snapset'
2014-06-12 14:46:39.673149 7f9d693e3700 10 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 'snapset' = 0
2014-06-12 14:46:39.673166 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] populate_obc_watchers 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.673183 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.673196 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] get_object_context: creating obc from disk: 0x30ac3340
2014-06-12 14:46:39.673208 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] get_object_context: 0x30ac3340 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 rwstate(none n=0 w=0) oi: 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3(173'2823 client.4294.0:144148 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2823) ssc: 0x81ba8c0 snapset: 0=[]:[]+head
2014-06-12 14:46:39.673228 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] find_object_context 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 @head oi=39913266/rbd_data.109d6b8b4567.000000000000112d/head//3(173'2823 client.4294.0:144148 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2823)
2014-06-12 14:46:39.673261 7f9d693e3700  5 -- op tracker -- , seq: 747940, time: 2014-06-12 14:46:39.673253, event: started, op: osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.673285 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] execute_ctx 0x29edcc00
2014-06-12 14:46:39.673296 7f9d693e3700  5 -- op tracker -- , seq: 747940, time: 2014-06-12 14:46:39.673293, event: started, op: osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.673313 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] do_op 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] ov 173'2823 av 173'2824 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:39.673323 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] do_osd_op 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192]
2014-06-12 14:46:39.673329 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:39.673338 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] do_osd_op  write 3072000~1032192
2014-06-12 14:46:39.673348 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] do_osd_op_effects on session 0x50a3000
2014-06-12 14:46:39.673355 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] make_writeable 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 snapset=0x81ba8f8  snapc=0=[]
2014-06-12 14:46:39.673362 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] make_writeable 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:39.673368 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] finish_ctx 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 0x29edcc00 op modify  
2014-06-12 14:46:39.673375 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean]  set mtime to 2014-06-12 08:45:42.048000
2014-06-12 14:46:39.673389 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean]  final snapset 0=[]:[]+head in 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.673401 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean]  zeroing write result code 0
2014-06-12 14:46:39.673409 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] new_repop rep_tid 236290 on osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:39.673419 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] issue_repop rep_tid 236290 o 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.673436 7f9d693e3700  5 -- op tracker -- , seq: 747940, time: 2014-06-12 14:46:39.673433, event: waiting for subops from 27, op: osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.673456 7f9d693e3700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:39.673472 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2823 (0'0,173'2823] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] append_log log((0'0,173'2823], crt=0'0) [173'2824 (173'2823) modify   39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 by client.4294.0:144149 2014-06-12 08:45:42.048000]
2014-06-12 14:46:39.673496 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2823 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] add_log_entry 173'2824 (173'2823) modify   39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 by client.4294.0:144149 2014-06-12 08:45:42.048000
2014-06-12 14:46:39.673511 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2823 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] append_log  adding 1 keys
2014-06-12 14:46:39.673546 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2824, trimmed: 
2014-06-12 14:46:39.673578 7f9d693e3700  5 queue_transactions existing osr(3.266 0x276d310)/0x276d310
2014-06-12 14:46:39.673586 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.673588 7f9d693e3700  5 queue_op 0x6d2e660 seq 2599 osr(3.266 0x276d310) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:39.673618 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f78c0 (1 active)
2014-06-12 14:46:39.673625 7f9d7e169700  5 _do_op 0x6d2e660 seq 2599 osr(3.266 0x276d310)/0x276d310 start
2014-06-12 14:46:39.673597 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2823 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] eval_repop repgather(0x5312780 173'2824 rep_tid=236290 committed?=0 applied?=0 lock=0 op=osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.673630 7f9d7e169700 10 _do_transaction on 0x5313380
2014-06-12 14:46:39.673630 7f9d693e3700 10 osd.2 173 dequeue_op 0x2afd700 finish
2014-06-12 14:46:39.673635 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.673636 7f9d7e169700 15 _omap_setkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.673637 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.673760 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.673799 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.673890 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.673926 7f9d7e169700 15 _omap_rmkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.673940 7f9d7e169700 10 _omap_rmkeys meta/1fe27d1a/pglog_3.266/0//-1 r = 0
2014-06-12 14:46:39.673951 7f9d7e169700 15 _omap_setkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.673975 7f9d7e169700 15 _write 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 3072000~1032192
2014-06-12 14:46:39.674269 7f9d7e169700 10 lookup_strip_header done  cid 3.266_head oid 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.674510 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:39.677598 7f9d7e169700 10 _generic_write 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 3072000~1032192 = 0
2014-06-12 14:46:39.677711 7f9d7e169700 15 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.677731 7f9d7e169700 10 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 = 0
2014-06-12 14:46:39.677736 7f9d7e169700 15 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.677745 7f9d7e169700 10 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 = 0
2014-06-12 14:46:39.677931 7f9d7e169700 20 set_header setting 17844 cid 3.266_head oid 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 parent seq 0
2014-06-12 14:46:39.677995 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.678006 7f9d7e169700 20 set_header setting 2030 cid meta oid 1fe27d1a/pglog_3.266/0//-1 parent seq 0
2014-06-12 14:46:39.678013 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:39.681418 7f9d7e169700 10 _do_op 0x6d2e660 seq 2599 r = 0, finisher 0x29f7df40 0xd949040
2014-06-12 14:46:39.681440 7f9d7e169700 10 _finish_op 0x6d2e660 seq 2599 osr(3.266 0x276d310)/0x276d310
2014-06-12 14:46:39.681459 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f78c0 (0 active)
2014-06-12 14:46:39.681447 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2823 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] op_commit: 236290
2014-06-12 14:46:39.681476 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.681490 7f9d7715b700  5 -- op tracker -- , seq: 747940, time: 2014-06-12 14:46:39.681479, event: op_commit, op: osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.681505 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2823 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] op_applied: 236290
2014-06-12 14:46:39.681524 7f9d7795c700  5 -- op tracker -- , seq: 747940, time: 2014-06-12 14:46:39.681520, event: op_applied, op: osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.681531 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2823 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] op_applied on primary on version 173'2824
2014-06-12 14:46:39.690336 7f9d58c5a700  5 -- op tracker -- , seq: 747941, time: 2014-06-12 14:46:39.690268, event: header_read, op: osd_sub_op_reply(client.4294.0:144149 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.690353 7f9d58c5a700  5 -- op tracker -- , seq: 747941, time: 2014-06-12 14:46:39.690270, event: throttled, op: osd_sub_op_reply(client.4294.0:144149 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.690360 7f9d58c5a700  5 -- op tracker -- , seq: 747941, time: 2014-06-12 14:46:39.690285, event: all_read, op: osd_sub_op_reply(client.4294.0:144149 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.690367 7f9d58c5a700  5 -- op tracker -- , seq: 747941, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:144149 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.690373 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:144149 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:39.690380 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:39.690390 7f9d58c5a700 15 osd.2 173 enqueue_op 0x6f4a800 prio 196 cost 0 latency 0.000121 osd_sub_op_reply(client.4294.0:144149 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:39.690426 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.690436 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x6f4a800 prio 196 cost 0 latency 0.000168 osd_sub_op_reply(client.4294.0:144149 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2 pg pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2823 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean]
2014-06-12 14:46:39.690469 7f9d6b3e7700  5 -- op tracker -- , seq: 747941, time: 2014-06-12 14:46:39.690459, event: reached_pg, op: osd_sub_op_reply(client.4294.0:144149 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.690487 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2823 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] handle_message: 0x6f4a800
2014-06-12 14:46:39.690501 7f9d6b3e7700  5 -- op tracker -- , seq: 747941, time: 2014-06-12 14:46:39.690498, event: started, op: osd_sub_op_reply(client.4294.0:144149 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.690507 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2823 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] sub_op_modify_reply: tid 236290 op  ack_type 4 from 27
2014-06-12 14:46:39.690527 7f9d6b3e7700  5 -- op tracker -- , seq: 747940, time: 2014-06-12 14:46:39.690519, event: sub_op_commit_rec, op: osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.690537 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2823 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] repop_all_applied: repop tid 236290 all applied 
2014-06-12 14:46:39.690549 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2823 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] eval_repop repgather(0x5312780 173'2824 rep_tid=236290 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.690566 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2823 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean]  sending ack on repgather(0x5312780 173'2824 rep_tid=236290 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) 0x25915680
2014-06-12 14:46:39.690596 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2823 crt=0'0 lcod 173'2822 mlcod 173'2822 active+clean] repop_all_committed: repop tid 236290 all committed 
2014-06-12 14:46:39.690608 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2822 active+clean] eval_repop repgather(0x5312780 173'2824 rep_tid=236290 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.690623 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2822 active+clean] log_op_stats osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.021197 lat 0.021224
2014-06-12 14:46:39.690639 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2822 active+clean] publish_stats_to_osd 173:3468
2014-06-12 14:46:39.690652 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2822 active+clean]  sending commit on repgather(0x5312780 173'2824 rep_tid=236290 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) 0x35714780
2014-06-12 14:46:39.690681 7f9d6b3e7700  5 -- op tracker -- , seq: 747940, time: 2014-06-12 14:46:39.690676, event: commit_sent, op: osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.690688 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2822 active+clean]  requeue_ops 
2014-06-12 14:46:39.690698 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean]  removing repgather(0x5312780 173'2824 rep_tid=236290 committed?=1 applied?=1 op=osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.690714 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean]    q front is repgather(0x5312780 173'2824 rep_tid=236290 committed?=1 applied?=1 op=osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.690729 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] remove_repop repgather(0x5312780 173'2824 rep_tid=236290 committed?=1 applied?=1 op=osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.690742 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean]  requeue_ops 
2014-06-12 14:46:39.690770 7f9d6b3e7700  5 -- op tracker -- , seq: 747940, time: 2014-06-12 14:46:39.690765, event: done, op: osd_op(client.4294.0:144149 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 3072000~1032192] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.690790 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x6f4a800 finish
2014-06-12 14:46:39.690798 7f9d6b3e7700  5 -- op tracker -- , seq: 747941, time: 2014-06-12 14:46:39.690795, event: done, op: osd_sub_op_reply(client.4294.0:144149 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.690809 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.690812 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.692677 7f9d50844700  5 -- op tracker -- , seq: 747942, time: 2014-06-12 14:46:39.692459, event: header_read, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.692689 7f9d50844700  5 -- op tracker -- , seq: 747942, time: 2014-06-12 14:46:39.692462, event: throttled, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.692699 7f9d50844700  5 -- op tracker -- , seq: 747942, time: 2014-06-12 14:46:39.692643, event: all_read, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.692713 7f9d50844700  5 -- op tracker -- , seq: 747942, time: 0.000000, event: dispatched, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.692720 7f9d50844700 20 osd.2 173 should_share_map client.4294 172.16.96.220:0/1242079 173
2014-06-12 14:46:39.692753 7f9d50844700 15 osd.2 173 enqueue_op 0x2afd500 prio 63 cost 90112 latency 0.000294 osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:39.692778 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.692785 7f9d683e1700 10 osd.2 173 dequeue_op 0x2afd500 prio 63 cost 90112 latency 0.000326 osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean]
2014-06-12 14:46:39.692802 7f9d683e1700  5 -- op tracker -- , seq: 747942, time: 2014-06-12 14:46:39.692799, event: reached_pg, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.692808 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:39.692822 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] handle_message: 0x2afd500
2014-06-12 14:46:39.692828 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] do_op osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:39.692845 7f9d683e1700 15 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 '_'
2014-06-12 14:46:39.692992 7f9d683e1700 10 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 '_' = 0
2014-06-12 14:46:39.693004 7f9d683e1700 15 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 'snapset'
2014-06-12 14:46:39.693056 7f9d683e1700 10 getattr 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 'snapset' = 0
2014-06-12 14:46:39.693063 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] populate_obc_watchers 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.693071 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.693077 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] get_object_context: creating obc from disk: 0x281278c0
2014-06-12 14:46:39.693082 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] get_object_context: 0x281278c0 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 rwstate(none n=0 w=0) oi: 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3(173'2824 client.4294.0:144149 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2824) ssc: 0x16ee42a0 snapset: 0=[]:[]+head
2014-06-12 14:46:39.693093 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] find_object_context 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 @head oi=39913266/rbd_data.109d6b8b4567.000000000000112d/head//3(173'2824 client.4294.0:144149 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2824)
2014-06-12 14:46:39.693110 7f9d683e1700  5 -- op tracker -- , seq: 747942, time: 2014-06-12 14:46:39.693106, event: started, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.693115 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] execute_ctx 0xb571800
2014-06-12 14:46:39.693126 7f9d683e1700  5 -- op tracker -- , seq: 747942, time: 2014-06-12 14:46:39.693123, event: started, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.693130 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] do_op 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] ov 173'2824 av 173'2825 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:39.693138 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] do_osd_op 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112]
2014-06-12 14:46:39.693145 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:39.693154 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] do_osd_op  write 4104192~90112
2014-06-12 14:46:39.693164 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] do_osd_op_effects on session 0x50a3000
2014-06-12 14:46:39.693170 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] make_writeable 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 snapset=0x16ee42d8  snapc=0=[]
2014-06-12 14:46:39.693178 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] make_writeable 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:39.693183 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] finish_ctx 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 0xb571800 op modify  
2014-06-12 14:46:39.693191 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean]  set mtime to 2014-06-12 08:45:42.072819
2014-06-12 14:46:39.693204 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean]  final snapset 0=[]:[]+head in 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.693222 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean]  zeroing write result code 0
2014-06-12 14:46:39.693228 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] new_repop rep_tid 236291 on osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:39.693238 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] issue_repop rep_tid 236291 o 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.693253 7f9d683e1700  5 -- op tracker -- , seq: 747942, time: 2014-06-12 14:46:39.693251, event: waiting for subops from 27, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.693270 7f9d683e1700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:39.693287 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2824 (0'0,173'2824] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] append_log log((0'0,173'2824], crt=0'0) [173'2825 (173'2824) modify   39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 by client.4294.0:144150 2014-06-12 08:45:42.072819]
2014-06-12 14:46:39.693314 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] add_log_entry 173'2825 (173'2824) modify   39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 by client.4294.0:144150 2014-06-12 08:45:42.072819
2014-06-12 14:46:39.693331 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] append_log  adding 1 keys
2014-06-12 14:46:39.693360 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2825, trimmed: 
2014-06-12 14:46:39.693382 7f9d683e1700  5 queue_transactions existing osr(3.266 0x276d310)/0x276d310
2014-06-12 14:46:39.693387 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.693389 7f9d683e1700  5 queue_op 0x8057c20 seq 2600 osr(3.266 0x276d310) 92405 bytes   (queue has 1 ops and 92405 bytes)
2014-06-12 14:46:39.693409 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f78c0 (1 active)
2014-06-12 14:46:39.693397 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] eval_repop repgather(0x519db00 173'2825 rep_tid=236291 committed?=0 applied?=0 lock=0 op=osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.693417 7f9d683e1700 10 osd.2 173 dequeue_op 0x2afd500 finish
2014-06-12 14:46:39.693417 7f9d7815d700  5 _do_op 0x8057c20 seq 2600 osr(3.266 0x276d310)/0x276d310 start
2014-06-12 14:46:39.693420 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.693422 7f9d7815d700 10 _do_transaction on 0x2a37b500
2014-06-12 14:46:39.693423 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.693431 7f9d7815d700 15 _omap_setkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.693537 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.693568 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.693641 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.693673 7f9d7815d700 15 _omap_rmkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.693681 7f9d7815d700 10 _omap_rmkeys meta/1fe27d1a/pglog_3.266/0//-1 r = 0
2014-06-12 14:46:39.693688 7f9d7815d700 15 _omap_setkeys meta/1fe27d1a/pglog_3.266/0//-1
2014-06-12 14:46:39.693912 7f9d7815d700 15 _write 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 4104192~90112
2014-06-12 14:46:39.694014 7f9d7815d700 10 lookup_strip_header done  cid 3.266_head oid 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.694141 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:39.694396 7f9d7815d700 10 _generic_write 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 4104192~90112 = 0
2014-06-12 14:46:39.694415 7f9d7815d700 15 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.694430 7f9d7815d700 10 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 = 0
2014-06-12 14:46:39.694437 7f9d7815d700 15 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3
2014-06-12 14:46:39.694447 7f9d7815d700 10 _setattrs 3.266_head/39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 = 0
2014-06-12 14:46:39.694536 7f9d7815d700 20 set_header setting 17844 cid 3.266_head oid 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 parent seq 0
2014-06-12 14:46:39.694561 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.694576 7f9d7815d700 20 set_header setting 2030 cid meta oid 1fe27d1a/pglog_3.266/0//-1 parent seq 0
2014-06-12 14:46:39.694587 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:39.694901 7f9d7815d700 10 _do_op 0x8057c20 seq 2600 r = 0, finisher 0x70c6340 0x36cab80
2014-06-12 14:46:39.694921 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] op_commit: 236291
2014-06-12 14:46:39.694949 7f9d7715b700  5 -- op tracker -- , seq: 747942, time: 2014-06-12 14:46:39.694939, event: op_commit, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.694967 7f9d7815d700 10 _finish_op 0x8057c20 seq 2600 osr(3.266 0x276d310)/0x276d310
2014-06-12 14:46:39.694982 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] op_applied: 236291
2014-06-12 14:46:39.695002 7f9d7795c700  5 -- op tracker -- , seq: 747942, time: 2014-06-12 14:46:39.694996, event: op_applied, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.695011 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] op_applied on primary on version 173'2825
2014-06-12 14:46:39.695035 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f78c0 (0 active)
2014-06-12 14:46:39.695040 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.698801 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:39.698927 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:39.700759 7f9d58c5a700  5 -- op tracker -- , seq: 747943, time: 2014-06-12 14:46:39.700697, event: header_read, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.700785 7f9d58c5a700  5 -- op tracker -- , seq: 747943, time: 2014-06-12 14:46:39.700697, event: throttled, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.700793 7f9d58c5a700  5 -- op tracker -- , seq: 747943, time: 2014-06-12 14:46:39.700708, event: all_read, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.700801 7f9d58c5a700  5 -- op tracker -- , seq: 747943, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.700807 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:39.700814 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:39.700823 7f9d58c5a700 15 osd.2 173 enqueue_op 0x6f4aa00 prio 196 cost 0 latency 0.000126 osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0) v2
2014-06-12 14:46:39.700871 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.700880 7f9d65bdc700 10 osd.2 173 dequeue_op 0x6f4aa00 prio 196 cost 0 latency 0.000183 osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0) v2 pg pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean]
2014-06-12 14:46:39.700913 7f9d65bdc700  5 -- op tracker -- , seq: 747943, time: 2014-06-12 14:46:39.700910, event: reached_pg, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.700927 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] handle_message: 0x6f4aa00
2014-06-12 14:46:39.700947 7f9d65bdc700  5 -- op tracker -- , seq: 747943, time: 2014-06-12 14:46:39.700944, event: started, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.700966 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] sub_op_modify_reply: tid 236291 op  ack_type 1 from 27
2014-06-12 14:46:39.700993 7f9d65bdc700  5 -- op tracker -- , seq: 747942, time: 2014-06-12 14:46:39.700983, event: sub_op_applied_rec, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.701003 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] repop_all_applied: repop tid 236291 all applied 
2014-06-12 14:46:39.701027 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] eval_repop repgather(0x519db00 173'2825 rep_tid=236291 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.701048 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean]  sending ack on repgather(0x519db00 173'2825 rep_tid=236291 committed?=0 applied?=1 lock=0 op=osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) 0x5728c80
2014-06-12 14:46:39.701084 7f9d65bdc700 10 osd.2 173 dequeue_op 0x6f4aa00 finish
2014-06-12 14:46:39.701107 7f9d65bdc700  5 -- op tracker -- , seq: 747943, time: 2014-06-12 14:46:39.701103, event: done, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ack, result = 0)
2014-06-12 14:46:39.701128 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.701132 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.701197 7f9d58c5a700  5 -- op tracker -- , seq: 747944, time: 2014-06-12 14:46:39.701158, event: header_read, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.701207 7f9d58c5a700  5 -- op tracker -- , seq: 747944, time: 2014-06-12 14:46:39.701159, event: throttled, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.701216 7f9d58c5a700  5 -- op tracker -- , seq: 747944, time: 2014-06-12 14:46:39.701177, event: all_read, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.701226 7f9d58c5a700  5 -- op tracker -- , seq: 747944, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.701231 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:39.701249 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:39.701257 7f9d58c5a700 15 osd.2 173 enqueue_op 0x6f4b300 prio 196 cost 0 latency 0.000099 osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:39.701281 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.701289 7f9d62bd6700 10 osd.2 173 dequeue_op 0x6f4b300 prio 196 cost 0 latency 0.000131 osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0) v2 pg pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean]
2014-06-12 14:46:39.701309 7f9d62bd6700  5 -- op tracker -- , seq: 747944, time: 2014-06-12 14:46:39.701305, event: reached_pg, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.701328 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] handle_message: 0x6f4b300
2014-06-12 14:46:39.701347 7f9d62bd6700  5 -- op tracker -- , seq: 747944, time: 2014-06-12 14:46:39.701343, event: started, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.701353 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] sub_op_modify_reply: tid 236291 op  ack_type 4 from 27
2014-06-12 14:46:39.701374 7f9d62bd6700  5 -- op tracker -- , seq: 747942, time: 2014-06-12 14:46:39.701367, event: sub_op_commit_rec, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.701383 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2824 crt=0'0 lcod 173'2823 mlcod 173'2823 active+clean] repop_all_committed: repop tid 236291 all committed 
2014-06-12 14:46:39.701396 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2823 active+clean] eval_repop repgather(0x519db00 173'2825 rep_tid=236291 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:39.701414 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2823 active+clean] log_op_stats osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4 inb 90391 outb 0 rlat 0.008623 lat 0.008954
2014-06-12 14:46:39.701434 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2823 active+clean] publish_stats_to_osd 173:3469
2014-06-12 14:46:39.701450 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2823 active+clean]  sending commit on repgather(0x519db00 173'2825 rep_tid=236291 committed?=1 applied?=1 lock=0 op=osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4) 0x11670a00
2014-06-12 14:46:39.701485 7f9d62bd6700  5 -- op tracker -- , seq: 747942, time: 2014-06-12 14:46:39.701479, event: commit_sent, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.701494 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2823 active+clean]  requeue_ops 
2014-06-12 14:46:39.701505 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2824 active+clean]  removing repgather(0x519db00 173'2825 rep_tid=236291 committed?=1 applied?=1 op=osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.701522 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2824 active+clean]    q front is repgather(0x519db00 173'2825 rep_tid=236291 committed?=1 applied?=1 op=osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.701540 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2824 active+clean] remove_repop repgather(0x519db00 173'2825 rep_tid=236291 committed?=1 applied?=1 op=osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:39.701557 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2824 active+clean]  requeue_ops 
2014-06-12 14:46:39.701584 7f9d62bd6700  5 -- op tracker -- , seq: 747942, time: 2014-06-12 14:46:39.701579, event: done, op: osd_op(client.4294.0:144150 rbd_data.109d6b8b4567.000000000000112d [set-alloc-hint object_size 4194304 write_size 4194304,write 4104192~90112] 3.39913266 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:39.701600 7f9d62bd6700 10 osd.2 173 dequeue_op 0x6f4b300 finish
2014-06-12 14:46:39.701608 7f9d62bd6700  5 -- op tracker -- , seq: 747944, time: 2014-06-12 14:46:39.701605, event: done, op: osd_sub_op_reply(client.4294.0:144150 3.266 39913266/rbd_data.109d6b8b4567.000000000000112d/head//3 [] ondisk, result = 0)
2014-06-12 14:46:39.701638 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.701641 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.702769 7f9d56432700  5 -- op tracker -- , seq: 747945, time: 2014-06-12 14:46:39.699616, event: header_read, op: osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.702781 7f9d56432700  5 -- op tracker -- , seq: 747945, time: 2014-06-12 14:46:39.699617, event: throttled, op: osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.702788 7f9d56432700  5 -- op tracker -- , seq: 747945, time: 2014-06-12 14:46:39.702726, event: all_read, op: osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.702796 7f9d56432700  5 -- op tracker -- , seq: 747945, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.702800 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:39.702806 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:39.702813 7f9d56432700 15 osd.2 173 enqueue_op 0x6d10900 prio 127 cost 1032999 latency 0.003197 osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.702839 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.702847 7f9d64bda700 10 osd.2 173 dequeue_op 0x6d10900 prio 127 cost 1032999 latency 0.003231 osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3ae( v 173'2957 (0'0,173'2957] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2956 active]
2014-06-12 14:46:39.702872 7f9d64bda700  5 -- op tracker -- , seq: 747945, time: 2014-06-12 14:46:39.702867, event: reached_pg, op: osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.702904 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2957 (0'0,173'2957] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2956 active] handle_message: 0x6d10900
2014-06-12 14:46:39.702917 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2957 (0'0,173'2957] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2956 active] sub_op_modify trans b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 v 173'2958 (transaction) 174
2014-06-12 14:46:39.702934 7f9d64bda700  5 -- op tracker -- , seq: 747945, time: 2014-06-12 14:46:39.702929, event: started, op: osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.702956 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2957 (0'0,173'2957] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2956 active] append_log log((0'0,173'2957], crt=0'0) [173'2958 (173'2957) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143736 2014-06-12 08:44:47.648700]
2014-06-12 14:46:39.702987 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2958 (0'0,173'2958] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2956 active] add_log_entry 173'2958 (173'2957) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143736 2014-06-12 08:44:47.648700
2014-06-12 14:46:39.703012 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2958 (0'0,173'2958] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2956 active] append_log  adding 1 keys
2014-06-12 14:46:39.703046 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2958, trimmed: 
2014-06-12 14:46:39.703070 7f9d64bda700  5 -- op tracker -- , seq: 747945, time: 2014-06-12 14:46:39.703065, event: started, op: osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.703082 7f9d64bda700  5 queue_transactions existing osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:39.703085 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.703087 7f9d64bda700  5 queue_op 0x69f0600 seq 2686 osr(3.3ae 0x276d6c0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:39.703094 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2958 (0'0,173'2958] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2956 active] do_sub_op osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.703108 7f9d64bda700 10 osd.2 173 dequeue_op 0x6d10900 finish
2014-06-12 14:46:39.703112 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.703114 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.703122 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478a20 (1 active)
2014-06-12 14:46:39.703127 7f9d82171700  5 _do_op 0x69f0600 seq 2686 osr(3.3ae 0x276d6c0)/0x276d6c0 start
2014-06-12 14:46:39.703131 7f9d82171700 10 _do_transaction on 0x5ea67f0
2014-06-12 14:46:39.703136 7f9d82171700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.703252 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.703281 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.703331 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.703454 7f9d82171700 15 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.703464 7f9d82171700 10 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1 r = 0
2014-06-12 14:46:39.703470 7f9d82171700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.703489 7f9d82171700 15 _write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 2560000~1032192
2014-06-12 14:46:39.703647 7f9d82171700 10 lookup_strip_header done  cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:39.703820 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:39.708163 7f9d82171700 10 _generic_write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 2560000~1032192 = 0
2014-06-12 14:46:39.708283 7f9d82171700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:39.708302 7f9d82171700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:39.708308 7f9d82171700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:39.708315 7f9d82171700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:39.712051 7f9d82171700 20 set_header setting 16695 cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 parent seq 0
2014-06-12 14:46:39.712488 7f9d82171700 20 set_header setting 2256 cid meta oid 1fe0bb56/pglog_3.3ae/0//-1 parent seq 0
2014-06-12 14:46:39.712513 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.712521 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:39.716970 7f9d82171700 10 _do_op 0x69f0600 seq 2686 r = 0, finisher 0x8145fe0 0
2014-06-12 14:46:39.716995 7f9d82171700 10 _finish_op 0x69f0600 seq 2686 osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:39.717000 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478a20 (0 active)
2014-06-12 14:46:39.717004 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.717016 7f9d7715b700  5 -- op tracker -- , seq: 747945, time: 2014-06-12 14:46:39.717004, event: commit_sent, op: osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.717030 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2958 (0'0,173'2958] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2956 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:39.717058 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:39.717092 7f9d7795c700  5 -- op tracker -- , seq: 747945, time: 2014-06-12 14:46:39.717086, event: sub_op_applied, op: osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.717099 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2958 (0'0,173'2958] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2957 active] sub_op_modify_applied on 0x5ea6700 op osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.717113 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2958 (0'0,173'2958] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2957 active] op_applied on primary on version 173'2958
2014-06-12 14:46:39.717123 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2958 (0'0,173'2958] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2957 active] op_applied on replica on version 173'2958
2014-06-12 14:46:39.717143 7f9d7795c700  5 -- op tracker -- , seq: 747945, time: 2014-06-12 14:46:39.717139, event: done, op: osd_sub_op(client.4317.0:143736 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2958 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.721118 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:39.721312 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:39.744070 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:39.744434 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:39.855685 7f9d56432700  5 -- op tracker -- , seq: 747946, time: 2014-06-12 14:46:39.854459, event: header_read, op: osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.855707 7f9d56432700  5 -- op tracker -- , seq: 747946, time: 2014-06-12 14:46:39.854461, event: throttled, op: osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.855717 7f9d56432700  5 -- op tracker -- , seq: 747946, time: 2014-06-12 14:46:39.855603, event: all_read, op: osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.855725 7f9d56432700  5 -- op tracker -- , seq: 747946, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.855732 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:39.855740 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:39.855753 7f9d56432700 15 osd.2 173 enqueue_op 0x6d10b00 prio 127 cost 516903 latency 0.001293 osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.855791 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.855802 7f9d6abe6700 10 osd.2 173 dequeue_op 0x6d10b00 prio 127 cost 516903 latency 0.001342 osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.71f( v 173'2696 (0'0,173'2696] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2695 active]
2014-06-12 14:46:39.855828 7f9d6abe6700  5 -- op tracker -- , seq: 747946, time: 2014-06-12 14:46:39.855823, event: reached_pg, op: osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.855840 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2696 (0'0,173'2696] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2695 active] handle_message: 0x6d10b00
2014-06-12 14:46:39.855853 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2696 (0'0,173'2696] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2695 active] sub_op_modify trans c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 v 173'2697 (transaction) 174
2014-06-12 14:46:39.855869 7f9d6abe6700  5 -- op tracker -- , seq: 747946, time: 2014-06-12 14:46:39.855865, event: started, op: osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.855891 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2696 (0'0,173'2696] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2695 active] append_log log((0'0,173'2696], crt=0'0) [173'2697 (173'2696) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145284 2014-06-12 08:45:27.315953]
2014-06-12 14:46:39.855915 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2697 (0'0,173'2697] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2695 active] add_log_entry 173'2697 (173'2696) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145284 2014-06-12 08:45:27.315953
2014-06-12 14:46:39.855941 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2697 (0'0,173'2697] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2695 active] append_log  adding 1 keys
2014-06-12 14:46:39.855978 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2697, trimmed: 
2014-06-12 14:46:39.856005 7f9d6abe6700  5 -- op tracker -- , seq: 747946, time: 2014-06-12 14:46:39.856000, event: started, op: osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.856014 7f9d6abe6700  5 queue_transactions existing osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:39.856017 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.856019 7f9d6abe6700  5 queue_op 0x5604780 seq 2529 osr(3.71f 0x276de50) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:39.856026 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.71f( v 173'2697 (0'0,173'2697] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2695 active] do_sub_op osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.856041 7f9d6abe6700 10 osd.2 173 dequeue_op 0x6d10b00 finish
2014-06-12 14:46:39.856045 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.856048 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.856056 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d0240 (1 active)
2014-06-12 14:46:39.856062 7f9d7e96a700  5 _do_op 0x5604780 seq 2529 osr(3.71f 0x276de50)/0x276de50 start
2014-06-12 14:46:39.856066 7f9d7e96a700 10 _do_transaction on 0x8dce630
2014-06-12 14:46:39.856071 7f9d7e96a700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.856227 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.856333 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.856400 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.856434 7f9d7e96a700 15 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.856439 7f9d7e96a700 10 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1 r = 0
2014-06-12 14:46:39.856445 7f9d7e96a700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:39.856460 7f9d7e96a700 15 _write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 1282048~516096
2014-06-12 14:46:39.856794 7f9d7e96a700 10 lookup_strip_header done  cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:39.856913 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:39.858697 7f9d7e96a700 10 _generic_write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 1282048~516096 = 0
2014-06-12 14:46:39.858746 7f9d7e96a700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:39.858758 7f9d7e96a700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:39.858762 7f9d7e96a700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:39.858767 7f9d7e96a700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:39.858906 7f9d7e96a700 20 set_header setting 17162 cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 parent seq 0
2014-06-12 14:46:39.858962 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.858973 7f9d7e96a700 20 set_header setting 2330 cid meta oid 1fe0985b/pglog_3.71f/0//-1 parent seq 0
2014-06-12 14:46:39.858979 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:39.861188 7f9d7e96a700 10 _do_op 0x5604780 seq 2529 r = 0, finisher 0x73f9540 0
2014-06-12 14:46:39.861207 7f9d7e96a700 10 _finish_op 0x5604780 seq 2529 osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:39.861225 7f9d7715b700  5 -- op tracker -- , seq: 747946, time: 2014-06-12 14:46:39.861215, event: commit_sent, op: osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.861229 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d0240 (0 active)
2014-06-12 14:46:39.861238 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.861236 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2697 (0'0,173'2697] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2695 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:39.861261 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:39.861307 7f9d7795c700  5 -- op tracker -- , seq: 747946, time: 2014-06-12 14:46:39.861301, event: sub_op_applied, op: osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.861315 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2697 (0'0,173'2697] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2696 active] sub_op_modify_applied on 0x8dce540 op osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.861332 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2697 (0'0,173'2697] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2696 active] op_applied on primary on version 173'2697
2014-06-12 14:46:39.861342 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2697 (0'0,173'2697] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2696 active] op_applied on replica on version 173'2697
2014-06-12 14:46:39.861363 7f9d7795c700  5 -- op tracker -- , seq: 747946, time: 2014-06-12 14:46:39.861358, event: done, op: osd_sub_op(client.4308.0:145284 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.881064 7f9d56432700  5 -- op tracker -- , seq: 747947, time: 2014-06-12 14:46:39.880026, event: header_read, op: osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.881076 7f9d56432700  5 -- op tracker -- , seq: 747947, time: 2014-06-12 14:46:39.880029, event: throttled, op: osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.881081 7f9d56432700  5 -- op tracker -- , seq: 747947, time: 2014-06-12 14:46:39.881007, event: all_read, op: osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.881086 7f9d56432700  5 -- op tracker -- , seq: 747947, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.881091 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:39.881096 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:39.881103 7f9d56432700 15 osd.2 173 enqueue_op 0x3b42be00 prio 127 cost 602919 latency 0.001077 osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.881131 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:39.881138 7f9d66bde700 10 osd.2 173 dequeue_op 0x3b42be00 prio 127 cost 602919 latency 0.001112 osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3ae( v 173'2958 (0'0,173'2958] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2957 active]
2014-06-12 14:46:39.881157 7f9d66bde700  5 -- op tracker -- , seq: 747947, time: 2014-06-12 14:46:39.881154, event: reached_pg, op: osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.881164 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2958 (0'0,173'2958] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2957 active] handle_message: 0x3b42be00
2014-06-12 14:46:39.881172 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2958 (0'0,173'2958] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2957 active] sub_op_modify trans b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 v 173'2959 (transaction) 174
2014-06-12 14:46:39.881183 7f9d66bde700  5 -- op tracker -- , seq: 747947, time: 2014-06-12 14:46:39.881180, event: started, op: osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.881195 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2958 (0'0,173'2958] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2957 active] append_log log((0'0,173'2958], crt=0'0) [173'2959 (173'2958) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143737 2014-06-12 08:44:48.032735]
2014-06-12 14:46:39.881213 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2959 (0'0,173'2959] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2957 active] add_log_entry 173'2959 (173'2958) modify   b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 by client.4317.0:143737 2014-06-12 08:44:48.032735
2014-06-12 14:46:39.881229 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2959 (0'0,173'2959] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2957 active] append_log  adding 1 keys
2014-06-12 14:46:39.881252 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2959, trimmed: 
2014-06-12 14:46:39.881269 7f9d66bde700  5 -- op tracker -- , seq: 747947, time: 2014-06-12 14:46:39.881266, event: started, op: osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.881276 7f9d66bde700  5 queue_transactions existing osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:39.881279 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:39.881281 7f9d66bde700  5 queue_op 0x4c77380 seq 2687 osr(3.3ae 0x276d6c0) 604405 bytes   (queue has 1 ops and 604405 bytes)
2014-06-12 14:46:39.881290 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478a20 (1 active)
2014-06-12 14:46:39.881297 7f9d7d167700  5 _do_op 0x4c77380 seq 2687 osr(3.3ae 0x276d6c0)/0x276d6c0 start
2014-06-12 14:46:39.881301 7f9d7d167700 10 _do_transaction on 0x20c43270
2014-06-12 14:46:39.881286 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2959 (0'0,173'2959] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2957 active] do_sub_op osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.881306 7f9d7d167700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.881307 7f9d66bde700 10 osd.2 173 dequeue_op 0x3b42be00 finish
2014-06-12 14:46:39.881311 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:39.881314 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:39.881439 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.881466 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:39.881523 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:39.881651 7f9d7d167700 15 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.881660 7f9d7d167700 10 _omap_rmkeys meta/1fe0bb56/pglog_3.3ae/0//-1 r = 0
2014-06-12 14:46:39.881667 7f9d7d167700 15 _omap_setkeys meta/1fe0bb56/pglog_3.3ae/0//-1
2014-06-12 14:46:39.881687 7f9d7d167700 15 _write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 3592192~602112
2014-06-12 14:46:39.881807 7f9d7d167700 10 lookup_strip_header done  cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:39.881919 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:39.883422 7f9d7d167700 10 _generic_write 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 3592192~602112 = 0
2014-06-12 14:46:39.883470 7f9d7d167700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:39.883485 7f9d7d167700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:39.883497 7f9d7d167700 15 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3
2014-06-12 14:46:39.883506 7f9d7d167700 10 _setattrs 3.3ae_head/b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 = 0
2014-06-12 14:46:39.883669 7f9d7d167700 20 set_header setting 16695 cid 3.3ae_head oid b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 parent seq 0
2014-06-12 14:46:39.883730 7f9d7d167700 20 set_header setting 2256 cid meta oid 1fe0bb56/pglog_3.3ae/0//-1 parent seq 0
2014-06-12 14:46:39.883747 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:39.883759 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:39.886206 7f9d7d167700 10 _do_op 0x4c77380 seq 2687 r = 0, finisher 0x82c8c20 0
2014-06-12 14:46:39.886230 7f9d7d167700 10 _finish_op 0x4c77380 seq 2687 osr(3.3ae 0x276d6c0)/0x276d6c0
2014-06-12 14:46:39.886251 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478a20 (0 active)
2014-06-12 14:46:39.886255 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:39.886256 7f9d7715b700  5 -- op tracker -- , seq: 747947, time: 2014-06-12 14:46:39.886243, event: commit_sent, op: osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.886272 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2959 (0'0,173'2959] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2957 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:39.886311 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:39.886349 7f9d7795c700  5 -- op tracker -- , seq: 747947, time: 2014-06-12 14:46:39.886343, event: sub_op_applied, op: osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.886359 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2959 (0'0,173'2959] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2958 active] sub_op_modify_applied on 0x20c43180 op osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:39.886375 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2959 (0'0,173'2959] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2958 active] op_applied on primary on version 173'2959
2014-06-12 14:46:39.886384 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3ae( v 173'2959 (0'0,173'2959] local-les=153 n=325 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2958 active] op_applied on replica on version 173'2959
2014-06-12 14:46:39.886408 7f9d7795c700  5 -- op tracker -- , seq: 747947, time: 2014-06-12 14:46:39.886403, event: done, op: osd_sub_op(client.4317.0:143737 3.3ae b0453ae/rbd_data.10b46b8b4567.000000000000102c/head//3 [] v 173'2959 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:39.909377 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:39.909457 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:40.019569 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:40.019791 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:40.019973 7f9d56432700  5 -- op tracker -- , seq: 747948, time: 2014-06-12 14:46:40.018946, event: header_read, op: osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.019984 7f9d56432700  5 -- op tracker -- , seq: 747948, time: 2014-06-12 14:46:40.018949, event: throttled, op: osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.019994 7f9d56432700  5 -- op tracker -- , seq: 747948, time: 2014-06-12 14:46:40.019939, event: all_read, op: osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.020006 7f9d56432700  5 -- op tracker -- , seq: 747948, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.020012 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:40.020019 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:40.020026 7f9d56432700 15 osd.2 173 enqueue_op 0x55c2000 prio 127 cost 516903 latency 0.001079 osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:40.020053 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:40.020062 7f9d69be4700 10 osd.2 173 dequeue_op 0x55c2000 prio 127 cost 516903 latency 0.001116 osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.71f( v 173'2697 (0'0,173'2697] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2696 active]
2014-06-12 14:46:40.020088 7f9d69be4700  5 -- op tracker -- , seq: 747948, time: 2014-06-12 14:46:40.020085, event: reached_pg, op: osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.020097 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2697 (0'0,173'2697] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2696 active] handle_message: 0x55c2000
2014-06-12 14:46:40.020105 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2697 (0'0,173'2697] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2696 active] sub_op_modify trans c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 v 173'2698 (transaction) 174
2014-06-12 14:46:40.020115 7f9d69be4700  5 -- op tracker -- , seq: 747948, time: 2014-06-12 14:46:40.020113, event: started, op: osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.020133 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2697 (0'0,173'2697] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2696 active] append_log log((0'0,173'2697], crt=0'0) [173'2698 (173'2697) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145285 2014-06-12 08:45:27.351269]
2014-06-12 14:46:40.020152 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2698 (0'0,173'2698] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2696 active] add_log_entry 173'2698 (173'2697) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145285 2014-06-12 08:45:27.351269
2014-06-12 14:46:40.020171 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2698 (0'0,173'2698] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2696 active] append_log  adding 1 keys
2014-06-12 14:46:40.020195 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2698, trimmed: 
2014-06-12 14:46:40.020212 7f9d69be4700  5 -- op tracker -- , seq: 747948, time: 2014-06-12 14:46:40.020209, event: started, op: osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.020221 7f9d69be4700  5 queue_transactions existing osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:40.020224 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:40.020225 7f9d69be4700  5 queue_op 0x56dcf00 seq 2530 osr(3.71f 0x276de50) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:40.020233 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d0240 (1 active)
2014-06-12 14:46:40.020238 7f9d8116f700  5 _do_op 0x56dcf00 seq 2530 osr(3.71f 0x276de50)/0x276de50 start
2014-06-12 14:46:40.020229 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.71f( v 173'2698 (0'0,173'2698] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2696 active] do_sub_op osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:40.020242 7f9d69be4700 10 osd.2 173 dequeue_op 0x55c2000 finish
2014-06-12 14:46:40.020243 7f9d8116f700 10 _do_transaction on 0x59c07f0
2014-06-12 14:46:40.020245 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:40.020247 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:40.020249 7f9d8116f700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:40.020359 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:40.020380 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:40.020419 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:40.020434 7f9d8116f700 15 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:40.020438 7f9d8116f700 10 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1 r = 0
2014-06-12 14:46:40.020442 7f9d8116f700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:40.020536 7f9d8116f700 15 _write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 1798144~516096
2014-06-12 14:46:40.020635 7f9d8116f700 10 lookup_strip_header done  cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:40.020786 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:40.022063 7f9d8116f700 10 _generic_write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 1798144~516096 = 0
2014-06-12 14:46:40.022097 7f9d8116f700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:40.022108 7f9d8116f700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:40.022112 7f9d8116f700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:40.022116 7f9d8116f700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:40.022251 7f9d8116f700 20 set_header setting 17162 cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 parent seq 0
2014-06-12 14:46:40.022297 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:40.022307 7f9d8116f700 20 set_header setting 2330 cid meta oid 1fe0985b/pglog_3.71f/0//-1 parent seq 0
2014-06-12 14:46:40.022314 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:40.024428 7f9d8116f700 10 _do_op 0x56dcf00 seq 2530 r = 0, finisher 0x7eb1d00 0
2014-06-12 14:46:40.024453 7f9d8116f700 10 _finish_op 0x56dcf00 seq 2530 osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:40.024459 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d0240 (0 active)
2014-06-12 14:46:40.024468 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:40.024477 7f9d7715b700  5 -- op tracker -- , seq: 747948, time: 2014-06-12 14:46:40.024464, event: commit_sent, op: osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.024488 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2698 (0'0,173'2698] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2696 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:40.024532 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:40.024598 7f9d7795c700  5 -- op tracker -- , seq: 747948, time: 2014-06-12 14:46:40.024589, event: sub_op_applied, op: osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.024611 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2698 (0'0,173'2698] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2697 active] sub_op_modify_applied on 0x59c0700 op osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:40.024633 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2698 (0'0,173'2698] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2697 active] op_applied on primary on version 173'2698
2014-06-12 14:46:40.024642 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2698 (0'0,173'2698] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2697 active] op_applied on replica on version 173'2698
2014-06-12 14:46:40.024660 7f9d7795c700  5 -- op tracker -- , seq: 747948, time: 2014-06-12 14:46:40.024657, event: done, op: osd_sub_op(client.4308.0:145285 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.038826 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:40.039103 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:40.106585 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:40.107028 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:40.120625 7f9d56432700  5 -- op tracker -- , seq: 747949, time: 2014-06-12 14:46:40.118405, event: header_read, op: osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.120635 7f9d56432700  5 -- op tracker -- , seq: 747949, time: 2014-06-12 14:46:40.118421, event: throttled, op: osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.120642 7f9d56432700  5 -- op tracker -- , seq: 747949, time: 2014-06-12 14:46:40.120594, event: all_read, op: osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.120647 7f9d56432700  5 -- op tracker -- , seq: 747949, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.120651 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:40.120656 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:40.120662 7f9d56432700 15 osd.2 173 enqueue_op 0x9fac300 prio 127 cost 1520423 latency 0.002257 osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:40.120685 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:40.120704 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x9fac300 prio 127 cost 1520423 latency 0.002299 osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.582( v 173'2759 (0'0,173'2759] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2758 active]
2014-06-12 14:46:40.120737 7f9d6a3e5700  5 -- op tracker -- , seq: 747949, time: 2014-06-12 14:46:40.120731, event: reached_pg, op: osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.120752 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2759 (0'0,173'2759] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2758 active] handle_message: 0x9fac300
2014-06-12 14:46:40.120772 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2759 (0'0,173'2759] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2758 active] sub_op_modify trans 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 v 173'2760 (transaction) 174
2014-06-12 14:46:40.120796 7f9d6a3e5700  5 -- op tracker -- , seq: 747949, time: 2014-06-12 14:46:40.120791, event: started, op: osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.120817 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2759 (0'0,173'2759] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2758 active] append_log log((0'0,173'2759], crt=0'0) [173'2760 (173'633) modify   264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 by client.4317.0:143784 2014-06-12 08:44:56.682692]
2014-06-12 14:46:40.120848 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2760 (0'0,173'2760] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2758 active] add_log_entry 173'2760 (173'633) modify   264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 by client.4317.0:143784 2014-06-12 08:44:56.682692
2014-06-12 14:46:40.120879 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2760 (0'0,173'2760] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2758 active] append_log  adding 1 keys
2014-06-12 14:46:40.120921 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2760, trimmed: 
2014-06-12 14:46:40.120957 7f9d6a3e5700  5 -- op tracker -- , seq: 747949, time: 2014-06-12 14:46:40.120950, event: started, op: osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.120972 7f9d6a3e5700  5 queue_transactions existing osr(3.582 0x276da70)/0x276da70
2014-06-12 14:46:40.120977 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:40.120979 7f9d6a3e5700  5 queue_op 0x17dd80c0 seq 2608 osr(3.582 0x276da70) 1521909 bytes   (queue has 1 ops and 1521909 bytes)
2014-06-12 14:46:40.120990 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438900 (1 active)
2014-06-12 14:46:40.120996 7f9d79960700  5 _do_op 0x17dd80c0 seq 2608 osr(3.582 0x276da70)/0x276da70 start
2014-06-12 14:46:40.121000 7f9d79960700 10 _do_transaction on 0x62c9eb0
2014-06-12 14:46:40.120987 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.582( v 173'2760 (0'0,173'2760] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2758 active] do_sub_op osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:40.121005 7f9d79960700 15 _omap_setkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:40.121006 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x9fac300 finish
2014-06-12 14:46:40.121011 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:40.121014 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:40.121127 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:40.121150 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:40.121188 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:40.121279 7f9d79960700 15 _omap_rmkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:40.121284 7f9d79960700 10 _omap_rmkeys meta/1fe177ea/pglog_3.582/0//-1 r = 0
2014-06-12 14:46:40.121289 7f9d79960700 15 _omap_setkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:40.121302 7f9d79960700 15 _write 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 0~1519616
2014-06-12 14:46:40.121436 7f9d79960700 10 lookup_strip_header done  cid 3.582_head oid 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:40.121673 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:40.126270 7f9d79960700 10 _generic_write 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 0~1519616 = 0
2014-06-12 14:46:40.126418 7f9d79960700 15 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:40.126450 7f9d79960700 10 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:40.126456 7f9d79960700 15 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:40.126463 7f9d79960700 10 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:40.126863 7f9d79960700 20 set_header setting 16739 cid 3.582_head oid 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 parent seq 0
2014-06-12 14:46:40.127023 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:40.127038 7f9d79960700 20 set_header setting 2318 cid meta oid 1fe177ea/pglog_3.582/0//-1 parent seq 0
2014-06-12 14:46:40.127054 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:40.133560 7f9d79960700 10 _do_op 0x17dd80c0 seq 2608 r = 0, finisher 0x73f9540 0
2014-06-12 14:46:40.133600 7f9d79960700 10 _finish_op 0x17dd80c0 seq 2608 osr(3.582 0x276da70)/0x276da70
2014-06-12 14:46:40.133607 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438900 (0 active)
2014-06-12 14:46:40.133612 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:40.133634 7f9d7715b700  5 -- op tracker -- , seq: 747949, time: 2014-06-12 14:46:40.133607, event: commit_sent, op: osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.133644 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2760 (0'0,173'2760] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2758 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:40.133679 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:40.133723 7f9d7795c700  5 -- op tracker -- , seq: 747949, time: 2014-06-12 14:46:40.133708, event: sub_op_applied, op: osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.133735 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2760 (0'0,173'2760] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2759 active] sub_op_modify_applied on 0x62c9dc0 op osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:40.133755 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2760 (0'0,173'2760] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2759 active] op_applied on primary on version 173'2760
2014-06-12 14:46:40.133766 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2760 (0'0,173'2760] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2759 active] op_applied on replica on version 173'2760
2014-06-12 14:46:40.133790 7f9d7795c700  5 -- op tracker -- , seq: 747949, time: 2014-06-12 14:46:40.133785, event: done, op: osd_sub_op(client.4317.0:143784 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2760 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.154431 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:40.171040 7f9d56432700  5 -- op tracker -- , seq: 747950, time: 2014-06-12 14:46:40.166871, event: header_read, op: osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.171051 7f9d56432700  5 -- op tracker -- , seq: 747950, time: 2014-06-12 14:46:40.166873, event: throttled, op: osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.171070 7f9d56432700  5 -- op tracker -- , seq: 747950, time: 2014-06-12 14:46:40.171007, event: all_read, op: osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.171081 7f9d56432700  5 -- op tracker -- , seq: 747950, time: 0.000000, event: dispatched, op: osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.171084 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:40.171088 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:40.171094 7f9d56432700 15 osd.2 173 enqueue_op 0x9fac600 prio 127 cost 1700647 latency 0.004223 osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:40.171118 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:40.171134 7f9d653db700 10 osd.2 173 dequeue_op 0x9fac600 prio 127 cost 1700647 latency 0.004263 osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2638 (0'0,173'2638] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2637 active]
2014-06-12 14:46:40.171158 7f9d653db700  5 -- op tracker -- , seq: 747950, time: 2014-06-12 14:46:40.171153, event: reached_pg, op: osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.171171 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2638 (0'0,173'2638] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2637 active] handle_message: 0x9fac600
2014-06-12 14:46:40.171185 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2638 (0'0,173'2638] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2637 active] sub_op_modify trans e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 v 173'2639 (transaction) 174
2014-06-12 14:46:40.171201 7f9d653db700  5 -- op tracker -- , seq: 747950, time: 2014-06-12 14:46:40.171197, event: started, op: osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.171217 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2638 (0'0,173'2638] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2637 active] append_log log((0'0,173'2638], crt=0'0) [173'2639 (173'704) modify   e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 by client.4292.0:142423 2014-06-12 08:44:57.983313]
2014-06-12 14:46:40.171242 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2639 (0'0,173'2639] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2637 active] add_log_entry 173'2639 (173'704) modify   e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 by client.4292.0:142423 2014-06-12 08:44:57.983313
2014-06-12 14:46:40.171266 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2639 (0'0,173'2639] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2637 active] append_log  adding 1 keys
2014-06-12 14:46:40.171303 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2639, trimmed: 
2014-06-12 14:46:40.171327 7f9d653db700  5 -- op tracker -- , seq: 747950, time: 2014-06-12 14:46:40.171322, event: started, op: osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.171337 7f9d653db700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:40.171341 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:40.171343 7f9d653db700  5 queue_op 0x7fba420 seq 2459 osr(3.1c7 0x276d170) 1702133 bytes   (queue has 1 ops and 1702133 bytes)
2014-06-12 14:46:40.171349 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2639 (0'0,173'2639] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2637 active] do_sub_op osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:40.171373 7f9d653db700 10 osd.2 173 dequeue_op 0x9fac600 finish
2014-06-12 14:46:40.171373 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:46:40.171377 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:40.171380 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:40.171380 7f9d8096e700  5 _do_op 0x7fba420 seq 2459 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:46:40.171384 7f9d8096e700 10 _do_transaction on 0x5733cf0
2014-06-12 14:46:40.171388 7f9d8096e700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:40.171491 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:40.171517 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:40.171557 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:40.171574 7f9d8096e700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:40.171579 7f9d8096e700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:46:40.171582 7f9d8096e700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:40.171593 7f9d8096e700 15 _write 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 0~1699840
2014-06-12 14:46:40.171852 7f9d8096e700 10 lookup_strip_header done  cid 3.1c7_head oid e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:40.171958 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:40.176825 7f9d8096e700 10 _generic_write 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 0~1699840 = 0
2014-06-12 14:46:40.176935 7f9d8096e700 15 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:40.176960 7f9d8096e700 10 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:40.176968 7f9d8096e700 15 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:40.176977 7f9d8096e700 10 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:40.177252 7f9d8096e700 20 set_header setting 18233 cid 3.1c7_head oid e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 parent seq 0
2014-06-12 14:46:40.177366 7f9d8096e700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:46:40.177384 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:40.177397 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:40.185102 7f9d8096e700 10 _do_op 0x7fba420 seq 2459 r = 0, finisher 0x6f12f20 0
2014-06-12 14:46:40.185126 7f9d8096e700 10 _finish_op 0x7fba420 seq 2459 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:40.185134 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:46:40.185140 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:40.185150 7f9d7715b700  5 -- op tracker -- , seq: 747950, time: 2014-06-12 14:46:40.185138, event: commit_sent, op: osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.185164 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2639 (0'0,173'2639] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2637 active] sub_op_modify_commit on op osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:40.185200 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:40.185241 7f9d7795c700  5 -- op tracker -- , seq: 747950, time: 2014-06-12 14:46:40.185234, event: sub_op_applied, op: osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.185251 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2639 (0'0,173'2639] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active] sub_op_modify_applied on 0x5733c00 op osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:40.185278 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2639 (0'0,173'2639] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active] op_applied on primary on version 173'2639
2014-06-12 14:46:40.185287 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2639 (0'0,173'2639] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active] op_applied on replica on version 173'2639
2014-06-12 14:46:40.185305 7f9d7795c700  5 -- op tracker -- , seq: 747950, time: 2014-06-12 14:46:40.185302, event: done, op: osd_sub_op(client.4292.0:142423 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:40.242360 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:40.242448 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:40.308806 7f9d425f5700  5 -- op tracker -- , seq: 747951, time: 2014-06-12 14:46:40.308442, event: header_read, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:40.308817 7f9d425f5700  5 -- op tracker -- , seq: 747951, time: 2014-06-12 14:46:40.308446, event: throttled, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:40.308823 7f9d425f5700  5 -- op tracker -- , seq: 747951, time: 2014-06-12 14:46:40.308771, event: all_read, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:40.308828 7f9d425f5700  5 -- op tracker -- , seq: 747951, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:40.308835 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:46:40.308858 7f9d425f5700 15 osd.2 173 enqueue_op 0x876ed00 prio 63 cost 200704 latency 0.000416 osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:40.308922 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:40.308946 7f9d63bd8700 10 osd.2 173 dequeue_op 0x876ed00 prio 63 cost 200704 latency 0.000503 osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]
2014-06-12 14:46:40.308992 7f9d63bd8700  5 -- op tracker -- , seq: 747951, time: 2014-06-12 14:46:40.308984, event: reached_pg, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:40.309012 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:40.309041 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] handle_message: 0x876ed00
2014-06-12 14:46:40.309056 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:40.309090 7f9d63bd8700 15 getattr 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 '_'
2014-06-12 14:46:40.335379 7f9d63bd8700 10 getattr 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 '_' = 0
2014-06-12 14:46:40.335420 7f9d63bd8700 15 getattr 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 'snapset'
2014-06-12 14:46:40.335562 7f9d63bd8700 10 getattr 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 'snapset' = 0
2014-06-12 14:46:40.335571 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] populate_obc_watchers fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:40.335592 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:40.335598 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: creating obc from disk: 0x22eacb00
2014-06-12 14:46:40.335605 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: 0x22eacb00 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 rwstate(none n=0 w=0) oi: fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3(173'596 client.4314.0:39262 wrlock_by=unknown.0.0:0 dirty s 4194304 uv596) ssc: 0x93449a0 snapset: 0=[]:[]+head
2014-06-12 14:46:40.335615 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] find_object_context fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 @head oi=fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3(173'596 client.4314.0:39262 wrlock_by=unknown.0.0:0 dirty s 4194304 uv596)
2014-06-12 14:46:40.335637 7f9d63bd8700  5 -- op tracker -- , seq: 747951, time: 2014-06-12 14:46:40.335631, event: started, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:40.335648 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] execute_ctx 0xfd18600
2014-06-12 14:46:40.335659 7f9d63bd8700  5 -- op tracker -- , seq: 747951, time: 2014-06-12 14:46:40.335656, event: started, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:40.335669 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] ov 173'596 av 173'2444 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:40.335679 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704]
2014-06-12 14:46:40.335685 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:40.335695 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  write 0~200704
2014-06-12 14:46:40.335705 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:46:40.335713 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 snapset=0x93449d8  snapc=0=[]
2014-06-12 14:46:40.335720 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:40.335726 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] finish_ctx fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 0xfd18600 op modify  
2014-06-12 14:46:40.335733 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  set mtime to 2014-06-12 08:45:42.688472
2014-06-12 14:46:40.335749 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  final snapset 0=[]:[]+head in fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:40.335762 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  zeroing write result code 0
2014-06-12 14:46:40.335769 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] new_repop rep_tid 236292 on osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:40.335780 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] issue_repop rep_tid 236292 o fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:40.335796 7f9d63bd8700  5 -- op tracker -- , seq: 747951, time: 2014-06-12 14:46:40.335793, event: waiting for subops from 26, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:40.335830 7f9d63bd8700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:40.335848 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2443 (0'0,173'2443] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log log((0'0,173'2443], crt=0'0) [173'2444 (173'596) modify   fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 by client.4314.0:145115 2014-06-12 08:45:42.688472]
2014-06-12 14:46:40.335865 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] add_log_entry 173'2444 (173'596) modify   fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 by client.4314.0:145115 2014-06-12 08:45:42.688472
2014-06-12 14:46:40.335878 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log  adding 1 keys
2014-06-12 14:46:40.335898 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2444, trimmed: 
2014-06-12 14:46:40.335915 7f9d63bd8700  5 queue_transactions existing osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:40.335919 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:40.335920 7f9d63bd8700  5 queue_op 0x65445a0 seq 2261 osr(3.251 0x276d390) 202997 bytes   (queue has 1 ops and 202997 bytes)
2014-06-12 14:46:40.335928 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fc20 (1 active)
2014-06-12 14:46:40.335925 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] eval_repop repgather(0x2fe22480 173'2444 rep_tid=236292 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:40.335937 7f9d8016d700  5 _do_op 0x65445a0 seq 2261 osr(3.251 0x276d390)/0x276d390 start
2014-06-12 14:46:40.335938 7f9d63bd8700 10 osd.2 173 dequeue_op 0x876ed00 finish
2014-06-12 14:46:40.335940 7f9d8016d700 10 _do_transaction on 0x2ad99ec0
2014-06-12 14:46:40.335941 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:40.335945 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:40.335946 7f9d8016d700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:40.336036 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:40.336062 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:40.336098 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:40.336113 7f9d8016d700 15 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:40.336118 7f9d8016d700 10 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1 r = 0
2014-06-12 14:46:40.336121 7f9d8016d700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:40.336261 7f9d8016d700 15 _write 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 0~200704
2014-06-12 14:46:40.336345 7f9d8016d700 10 lookup_strip_header done  cid 3.251_head oid fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:40.336443 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:40.336893 7f9d8016d700 10 _generic_write 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 0~200704 = 0
2014-06-12 14:46:40.336911 7f9d8016d700 15 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:40.336919 7f9d8016d700 10 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 = 0
2014-06-12 14:46:40.336924 7f9d8016d700 15 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:40.336929 7f9d8016d700 10 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 = 0
2014-06-12 14:46:40.337022 7f9d8016d700 20 set_header setting 17915 cid 3.251_head oid fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 parent seq 0
2014-06-12 14:46:40.337053 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:40.337061 7f9d8016d700 20 set_header setting 2026 cid meta oid 1fe2661a/pglog_3.251/0//-1 parent seq 0
2014-06-12 14:46:40.337069 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:40.338842 7f9d8016d700 10 _do_op 0x65445a0 seq 2261 r = 0, finisher 0x6ce6120 0x31af0e80
2014-06-12 14:46:40.338857 7f9d8016d700 10 _finish_op 0x65445a0 seq 2261 osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:40.338873 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fc20 (0 active)
2014-06-12 14:46:40.338876 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:40.338871 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_commit: 236292
2014-06-12 14:46:40.338908 7f9d7715b700  5 -- op tracker -- , seq: 747951, time: 2014-06-12 14:46:40.338891, event: op_commit, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:40.338928 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied: 236292
2014-06-12 14:46:40.338956 7f9d7795c700  5 -- op tracker -- , seq: 747951, time: 2014-06-12 14:46:40.338949, event: op_applied, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:40.338965 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied on primary on version 173'2444
2014-06-12 14:46:40.368081 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:40.477870 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:40.477926 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:40.477924 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 11.1 >= max 0.5 = no, load too high
2014-06-12 14:46:40.477936 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:40.477938 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92648 < max (604800 seconds)
2014-06-12 14:46:40.477956 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:40.477957 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:40.477958 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:40.477960 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 5; oldest is 7.316616 seconds old
2014-06-12 14:46:40.539240 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:40.539270 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:40.751918 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:40.751965 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:40.928197 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.042498 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.100181 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.150897 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.319807 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19415 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,1,0,0,4])
2014-06-12 14:46:41.319826 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19415 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,1,0,0,4])
2014-06-12 14:46:41.320316 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:41.320355 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:41.320388 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:41.320421 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:41.320444 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:41.320465 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:41.320502 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:41.320536 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:41.320551 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:41.320562 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:41.320578 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:41.320590 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:41.320606 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:41.320617 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:41.320626 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:41.320636 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:41.320647 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:41.320655 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:41.320665 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:41.320681 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:41.320701 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:41.320742 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:41.352940 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.373106 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.478038 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:41.478121 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:41.478116 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 11.1 >= max 0.5 = no, load too high
2014-06-12 14:46:41.478128 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:41.478131 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92649 < max (604800 seconds)
2014-06-12 14:46:41.478140 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:41.478141 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:41.478142 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:41.478143 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 5; oldest is 8.316799 seconds old
2014-06-12 14:46:41.505335 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.559412 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.564054 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.611054 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.618216 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.621402 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.621443 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:41.621583 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.621773 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:41.627898 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.637266 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.644707 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.651253 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.659748 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.665894 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.673711 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.681562 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.690886 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.693503 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.695127 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.701222 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.701705 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.703187 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.717100 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.856114 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.861300 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.881387 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:41.886342 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:41.942843 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:41.942880 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:42.020339 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:42.024530 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:42.121077 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:42.133716 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:42.154510 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:42.171454 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:42.185203 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:42.319751 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:42.320057 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:42.336009 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:42.338936 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:42.368156 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:42.478215 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:42.478240 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:46:42.478244 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:46:42.478249 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:46:42.478253 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:46:42.478295 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:42.478299 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:42.478308 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:42.478304 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:42.478310 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 5; oldest is 9.316966 seconds old
2014-06-12 14:46:42.644758 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:42.644841 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:42.645159 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:42.645162 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:42.645163 7f9d74bfa700 20 osd.2 173 _dispatch 0x5e9bdc0 pg_stats_ack(1 pgs tid 4854) v1
2014-06-12 14:46:42.645165 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:46:42.645170 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:42.645171 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:42.668262 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:42.668481 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:42.810089 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:42.810322 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:42.839788 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:42.839945 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:42.928267 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.042568 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.043088 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:43.043182 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:43.052573 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:43.052629 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:43.100257 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.145023 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:43.145093 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:43.150968 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.256797 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:43.256847 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:43.353017 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.373178 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.477519 7f9d5a16f700  5 -- op tracker -- , seq: 747952, time: 2014-06-12 14:46:43.476664, event: header_read, op: osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.477533 7f9d5a16f700  5 -- op tracker -- , seq: 747952, time: 2014-06-12 14:46:43.476667, event: throttled, op: osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.477549 7f9d5a16f700  5 -- op tracker -- , seq: 747952, time: 2014-06-12 14:46:43.477479, event: all_read, op: osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.477555 7f9d5a16f700  5 -- op tracker -- , seq: 747952, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.477559 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:43.477564 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:43.477570 7f9d5a16f700 15 osd.2 173 enqueue_op 0x6f4a600 prio 127 cost 516903 latency 0.000906 osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:43.477597 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:43.477606 7f9d68be2700 10 osd.2 173 dequeue_op 0x6f4a600 prio 127 cost 516903 latency 0.000941 osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2668 (0'0,173'2668] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2667 active]
2014-06-12 14:46:43.477638 7f9d68be2700  5 -- op tracker -- , seq: 747952, time: 2014-06-12 14:46:43.477634, event: reached_pg, op: osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.477654 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2668 (0'0,173'2668] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2667 active] handle_message: 0x6f4a600
2014-06-12 14:46:43.477667 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2668 (0'0,173'2668] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2667 active] sub_op_modify trans f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 v 173'2669 (transaction) 174
2014-06-12 14:46:43.477684 7f9d68be2700  5 -- op tracker -- , seq: 747952, time: 2014-06-12 14:46:43.477680, event: started, op: osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.477700 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2668 (0'0,173'2668] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2667 active] append_log log((0'0,173'2668], crt=0'0) [173'2669 (173'2668) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146411 2014-06-12 08:41:30.242095]
2014-06-12 14:46:43.477724 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2669 (0'0,173'2669] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2667 active] add_log_entry 173'2669 (173'2668) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146411 2014-06-12 08:41:30.242095
2014-06-12 14:46:43.477748 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2669 (0'0,173'2669] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2667 active] append_log  adding 1 keys
2014-06-12 14:46:43.477783 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2669, trimmed: 
2014-06-12 14:46:43.477808 7f9d68be2700  5 -- op tracker -- , seq: 747952, time: 2014-06-12 14:46:43.477803, event: started, op: osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.477819 7f9d68be2700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:43.477823 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:43.477824 7f9d68be2700  5 queue_op 0x9af4f00 seq 2479 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:43.477833 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:46:43.477839 7f9d7c966700  5 _do_op 0x9af4f00 seq 2479 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:43.477842 7f9d7c966700 10 _do_transaction on 0x1f2869b0
2014-06-12 14:46:43.477831 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2669 (0'0,173'2669] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2667 active] do_sub_op osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:43.477845 7f9d68be2700 10 osd.2 173 dequeue_op 0x6f4a600 finish
2014-06-12 14:46:43.477847 7f9d7c966700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:43.477848 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:43.477851 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.477938 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:43.477955 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:43.477988 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:43.478007 7f9d7c966700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:43.478012 7f9d7c966700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:43.478016 7f9d7c966700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:43.478122 7f9d7c966700 15 _write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 3391488~516096
2014-06-12 14:46:43.478225 7f9d7c966700 10 lookup_strip_header done  cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:43.478357 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:43.478374 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:43.478389 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:43.478394 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:43.478396 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:43.478398 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 6; oldest is 10.317054 seconds old
2014-06-12 14:46:43.478411 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:43.479812 7f9d7c966700 10 _generic_write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 3391488~516096 = 0
2014-06-12 14:46:43.479859 7f9d7c966700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:43.479873 7f9d7c966700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:43.479878 7f9d7c966700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:43.479884 7f9d7c966700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:43.480019 7f9d7c966700 20 set_header setting 16921 cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 parent seq 0
2014-06-12 14:46:43.480075 7f9d7c966700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:43.480087 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:43.480094 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:43.482070 7f9d7c966700 10 _do_op 0x9af4f00 seq 2479 r = 0, finisher 0x810c700 0
2014-06-12 14:46:43.482096 7f9d7c966700 10 _finish_op 0x9af4f00 seq 2479 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:43.482105 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:46:43.482117 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.482121 7f9d7715b700  5 -- op tracker -- , seq: 747952, time: 2014-06-12 14:46:43.482109, event: commit_sent, op: osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.482136 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2669 (0'0,173'2669] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2667 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:43.482161 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:43.482196 7f9d7795c700  5 -- op tracker -- , seq: 747952, time: 2014-06-12 14:46:43.482190, event: sub_op_applied, op: osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.482203 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2669 (0'0,173'2669] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2668 active] sub_op_modify_applied on 0x1f2868c0 op osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:43.482229 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2669 (0'0,173'2669] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2668 active] op_applied on primary on version 173'2669
2014-06-12 14:46:43.482241 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2669 (0'0,173'2669] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2668 active] op_applied on replica on version 173'2669
2014-06-12 14:46:43.482266 7f9d7795c700  5 -- op tracker -- , seq: 747952, time: 2014-06-12 14:46:43.482262, event: done, op: osd_sub_op(client.4316.0:146411 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2669 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.553232 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:43.553269 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:43.564121 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.611188 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.616833 7f9d5a16f700  5 -- op tracker -- , seq: 747953, time: 2014-06-12 14:46:43.615704, event: header_read, op: osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.616852 7f9d5a16f700  5 -- op tracker -- , seq: 747953, time: 2014-06-12 14:46:43.615707, event: throttled, op: osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.616864 7f9d5a16f700  5 -- op tracker -- , seq: 747953, time: 2014-06-12 14:46:43.616782, event: all_read, op: osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.616896 7f9d5a16f700  5 -- op tracker -- , seq: 747953, time: 0.000000, event: dispatched, op: osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.616903 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:43.616912 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:43.616922 7f9d5a16f700 15 osd.2 173 enqueue_op 0x6f4ad00 prio 127 cost 516903 latency 0.001218 osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:43.616970 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:43.616989 7f9d643d9700 10 osd.2 173 dequeue_op 0x6f4ad00 prio 127 cost 516903 latency 0.001284 osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2720 (0'0,173'2720] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active]
2014-06-12 14:46:43.617051 7f9d643d9700  5 -- op tracker -- , seq: 747953, time: 2014-06-12 14:46:43.617041, event: reached_pg, op: osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.617071 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2720 (0'0,173'2720] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] handle_message: 0x6f4ad00
2014-06-12 14:46:43.617101 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2720 (0'0,173'2720] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] sub_op_modify trans cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 v 173'2721 (transaction) 174
2014-06-12 14:46:43.617141 7f9d643d9700  5 -- op tracker -- , seq: 747953, time: 2014-06-12 14:46:43.617131, event: started, op: osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.617179 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2720 (0'0,173'2720] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] append_log log((0'0,173'2720], crt=0'0) [173'2721 (173'2720) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143143 2014-06-12 08:43:30.475842]
2014-06-12 14:46:43.617230 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2721 (0'0,173'2721] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] add_log_entry 173'2721 (173'2720) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143143 2014-06-12 08:43:30.475842
2014-06-12 14:46:43.617293 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2721 (0'0,173'2721] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] append_log  adding 1 keys
2014-06-12 14:46:43.617368 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2721, trimmed: 
2014-06-12 14:46:43.617419 7f9d643d9700  5 -- op tracker -- , seq: 747953, time: 2014-06-12 14:46:43.617407, event: started, op: osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.617439 7f9d643d9700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:43.617447 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:43.617450 7f9d643d9700  5 queue_op 0x6d2ee40 seq 2468 osr(3.21c 0x276d020) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:43.617463 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2721 (0'0,173'2721] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] do_sub_op osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:43.617497 7f9d643d9700 10 osd.2 173 dequeue_op 0x6f4ad00 finish
2014-06-12 14:46:43.617505 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:43.617499 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (1 active)
2014-06-12 14:46:43.617512 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.617516 7f9d7895e700  5 _do_op 0x6d2ee40 seq 2468 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:46:43.617523 7f9d7895e700 10 _do_transaction on 0x6c562b0
2014-06-12 14:46:43.617532 7f9d7895e700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:43.617678 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:43.617821 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:43.617888 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:43.617916 7f9d7895e700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:43.617927 7f9d7895e700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:46:43.617935 7f9d7895e700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:43.617958 7f9d7895e700 15 _write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 122880~516096
2014-06-12 14:46:43.618351 7f9d7895e700 10 lookup_strip_header done  cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:43.618532 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:43.621138 7f9d7895e700 10 _generic_write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 122880~516096 = 0
2014-06-12 14:46:43.621224 7f9d7895e700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:43.621251 7f9d7895e700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:43.621265 7f9d7895e700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:43.621280 7f9d7895e700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:43.621659 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.621696 7f9d7895e700 20 set_header setting 16774 cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 parent seq 0
2014-06-12 14:46:43.621837 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:43.621863 7f9d7895e700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:46:43.621877 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:43.626456 7f9d7895e700 10 _do_op 0x6d2ee40 seq 2468 r = 0, finisher 0x4dbbbe0 0
2014-06-12 14:46:43.626475 7f9d7895e700 10 _finish_op 0x6d2ee40 seq 2468 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:43.626492 7f9d7715b700  5 -- op tracker -- , seq: 747953, time: 2014-06-12 14:46:43.626483, event: commit_sent, op: osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.626491 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (0 active)
2014-06-12 14:46:43.626501 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.626501 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2721 (0'0,173'2721] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] sub_op_modify_commit on op osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:43.626525 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:43.626591 7f9d7795c700  5 -- op tracker -- , seq: 747953, time: 2014-06-12 14:46:43.626580, event: sub_op_applied, op: osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.626608 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2721 (0'0,173'2721] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] sub_op_modify_applied on 0x6c561c0 op osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:43.626631 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2721 (0'0,173'2721] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] op_applied on primary on version 173'2721
2014-06-12 14:46:43.626644 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2721 (0'0,173'2721] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] op_applied on replica on version 173'2721
2014-06-12 14:46:43.626673 7f9d7795c700  5 -- op tracker -- , seq: 747953, time: 2014-06-12 14:46:43.626667, event: done, op: osd_sub_op(client.4279.0:143143 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.627965 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.631329 7f9d425f5700  5 -- op tracker -- , seq: 747954, time: 2014-06-12 14:46:43.628040, event: header_read, op: osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:43.631362 7f9d425f5700  5 -- op tracker -- , seq: 747954, time: 2014-06-12 14:46:43.628046, event: throttled, op: osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:43.631376 7f9d425f5700  5 -- op tracker -- , seq: 747954, time: 2014-06-12 14:46:43.631256, event: all_read, op: osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:43.631388 7f9d425f5700  5 -- op tracker -- , seq: 747954, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:43.631400 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:46:43.631439 7f9d425f5700 15 osd.2 173 enqueue_op 0x876fd00 prio 63 cost 1032192 latency 0.003398 osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:43.631493 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:43.631511 7f9d673df700 10 osd.2 173 dequeue_op 0x876fd00 prio 63 cost 1032192 latency 0.003470 osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]
2014-06-12 14:46:43.631570 7f9d673df700  5 -- op tracker -- , seq: 747954, time: 2014-06-12 14:46:43.631558, event: reached_pg, op: osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:43.631590 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:43.631620 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] handle_message: 0x876fd00
2014-06-12 14:46:43.631641 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:43.631698 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: found obc in cache: 0x22eacb00
2014-06-12 14:46:43.631726 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: 0x22eacb00 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 rwstate(write n=1 w=0) oi: fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3(173'2444 client.4314.0:145115 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2444) ssc: 0x93449a0 snapset: 0=[]:[]+head
2014-06-12 14:46:43.631762 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] find_object_context fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 @head oi=fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3(173'2444 client.4314.0:145115 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2444)
2014-06-12 14:46:43.631816 7f9d673df700  5 -- op tracker -- , seq: 747954, time: 2014-06-12 14:46:43.631804, event: started, op: osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:43.631834 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] execute_ctx 0xfd1a400
2014-06-12 14:46:43.631875 7f9d673df700  5 -- op tracker -- , seq: 747954, time: 2014-06-12 14:46:43.631864, event: started, op: osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:43.631893 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] ov 173'2444 av 173'2445 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:43.631923 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192]
2014-06-12 14:46:43.631945 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:43.631972 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  write 200704~1032192
2014-06-12 14:46:43.632005 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:46:43.632032 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 snapset=0x93449d8  snapc=0=[]
2014-06-12 14:46:43.632056 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:43.632081 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] finish_ctx fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 0xfd1a400 op modify  
2014-06-12 14:46:43.632107 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  set mtime to 2014-06-12 08:45:46.006677
2014-06-12 14:46:43.632158 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  final snapset 0=[]:[]+head in fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:43.632197 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  zeroing write result code 0
2014-06-12 14:46:43.632223 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] new_repop rep_tid 236293 on osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:43.632258 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] issue_repop rep_tid 236293 o fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:43.632315 7f9d673df700  5 -- op tracker -- , seq: 747954, time: 2014-06-12 14:46:43.632303, event: waiting for subops from 26, op: osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:43.632359 7f9d673df700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:43.632406 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2444 (0'0,173'2444] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log log((0'0,173'2444], crt=0'0) [173'2445 (173'2444) modify   fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 by client.4314.0:145116 2014-06-12 08:45:46.006677]
2014-06-12 14:46:43.632458 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] add_log_entry 173'2445 (173'2444) modify   fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 by client.4314.0:145116 2014-06-12 08:45:46.006677
2014-06-12 14:46:43.632505 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log  adding 1 keys
2014-06-12 14:46:43.632575 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2445, trimmed: 
2014-06-12 14:46:43.632625 7f9d673df700  5 queue_transactions existing osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:43.632635 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:43.632638 7f9d673df700  5 queue_op 0x61d75c0 seq 2262 osr(3.251 0x276d390) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:43.632657 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fc20 (1 active)
2014-06-12 14:46:43.632672 7f9d7a161700  5 _do_op 0x61d75c0 seq 2262 osr(3.251 0x276d390)/0x276d390 start
2014-06-12 14:46:43.632680 7f9d7a161700 10 _do_transaction on 0x22363c0
2014-06-12 14:46:43.632651 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] eval_repop repgather(0x2236240 173'2445 rep_tid=236293 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:43.632689 7f9d673df700 10 osd.2 173 dequeue_op 0x876fd00 finish
2014-06-12 14:46:43.632692 7f9d7a161700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:43.632697 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:43.632702 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.632854 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:43.632902 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:43.632988 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:43.633035 7f9d7a161700 15 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:43.633051 7f9d7a161700 10 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1 r = 0
2014-06-12 14:46:43.633062 7f9d7a161700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:43.633094 7f9d7a161700 15 _write 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 200704~1032192
2014-06-12 14:46:43.633636 7f9d7a161700 10 lookup_strip_header done  cid 3.251_head oid fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:43.633905 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:43.639648 7f9d7a161700 10 _generic_write 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 200704~1032192 = 0
2014-06-12 14:46:43.639771 7f9d7a161700 15 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:43.639793 7f9d7a161700 10 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 = 0
2014-06-12 14:46:43.639802 7f9d7a161700 15 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:43.639812 7f9d7a161700 10 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 = 0
2014-06-12 14:46:43.640213 7f9d7a161700 20 set_header setting 17915 cid 3.251_head oid fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 parent seq 0
2014-06-12 14:46:43.640360 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:43.640383 7f9d7a161700 20 set_header setting 2026 cid meta oid 1fe2661a/pglog_3.251/0//-1 parent seq 0
2014-06-12 14:46:43.640396 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:43.648268 7f9d7a161700 10 _do_op 0x61d75c0 seq 2262 r = 0, finisher 0x692ee60 0x3786eb00
2014-06-12 14:46:43.648298 7f9d7a161700 10 _finish_op 0x61d75c0 seq 2262 osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:43.648311 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fc20 (0 active)
2014-06-12 14:46:43.648316 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.648314 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_commit: 236293
2014-06-12 14:46:43.648377 7f9d7715b700  5 -- op tracker -- , seq: 747954, time: 2014-06-12 14:46:43.648358, event: op_commit, op: osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:43.648455 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied: 236293
2014-06-12 14:46:43.648509 7f9d7795c700  5 -- op tracker -- , seq: 747954, time: 2014-06-12 14:46:43.648497, event: op_applied, op: osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:43.648534 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied on primary on version 173'2445
2014-06-12 14:46:43.651323 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.659853 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.665967 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.673789 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.681642 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.690958 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.693586 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.695209 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.701304 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.701774 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.703271 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.717187 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.798913 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:43.798957 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:43.856204 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.861382 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.881471 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.886418 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:43.988297 7f9d5a16f700  5 -- op tracker -- , seq: 747955, time: 2014-06-12 14:46:43.984653, event: header_read, op: osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.988323 7f9d5a16f700  5 -- op tracker -- , seq: 747955, time: 2014-06-12 14:46:43.984658, event: throttled, op: osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.988356 7f9d5a16f700  5 -- op tracker -- , seq: 747955, time: 2014-06-12 14:46:43.988237, event: all_read, op: osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.988373 7f9d5a16f700  5 -- op tracker -- , seq: 747955, time: 0.000000, event: dispatched, op: osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.988380 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:43.988390 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:43.988400 7f9d5a16f700 15 osd.2 173 enqueue_op 0x6f4b100 prio 127 cost 1549095 latency 0.003747 osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:43.988472 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:43.988489 7f9d69be4700 10 osd.2 173 dequeue_op 0x6f4b100 prio 127 cost 1549095 latency 0.003835 osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4e0( v 173'2636 (0'0,173'2636] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2635 active]
2014-06-12 14:46:43.988536 7f9d69be4700  5 -- op tracker -- , seq: 747955, time: 2014-06-12 14:46:43.988529, event: reached_pg, op: osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.988553 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2636 (0'0,173'2636] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2635 active] handle_message: 0x6f4b100
2014-06-12 14:46:43.988569 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2636 (0'0,173'2636] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2635 active] sub_op_modify trans 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 v 173'2637 (transaction) 174
2014-06-12 14:46:43.988590 7f9d69be4700  5 -- op tracker -- , seq: 747955, time: 2014-06-12 14:46:43.988585, event: started, op: osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.988612 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2636 (0'0,173'2636] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2635 active] append_log log((0'0,173'2636], crt=0'0) [173'2637 (173'2636) modify   79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 by client.4293.0:143961 2014-06-12 08:43:56.450144]
2014-06-12 14:46:43.988647 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2637 (0'0,173'2637] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2635 active] add_log_entry 173'2637 (173'2636) modify   79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 by client.4293.0:143961 2014-06-12 08:43:56.450144
2014-06-12 14:46:43.988679 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2637 (0'0,173'2637] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2635 active] append_log  adding 1 keys
2014-06-12 14:46:43.988731 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2637, trimmed: 
2014-06-12 14:46:43.988765 7f9d69be4700  5 -- op tracker -- , seq: 747955, time: 2014-06-12 14:46:43.988758, event: started, op: osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.988786 7f9d69be4700  5 queue_transactions existing osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:43.988792 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:43.988794 7f9d69be4700  5 queue_op 0x6e69980 seq 2437 osr(3.4e0 0x276dbd0) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:46:43.988803 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2637 (0'0,173'2637] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2635 active] do_sub_op osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:43.988823 7f9d69be4700 10 osd.2 173 dequeue_op 0x6f4b100 finish
2014-06-12 14:46:43.988828 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:43.988831 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.988839 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5180900 (1 active)
2014-06-12 14:46:43.988855 7f9d8116f700  5 _do_op 0x6e69980 seq 2437 osr(3.4e0 0x276dbd0)/0x276dbd0 start
2014-06-12 14:46:43.988862 7f9d8116f700 10 _do_transaction on 0x206ec2b0
2014-06-12 14:46:43.988871 7f9d8116f700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:43.988995 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:43.989030 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:43.989088 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:43.989206 7f9d8116f700 15 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:43.989220 7f9d8116f700 10 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1 r = 0
2014-06-12 14:46:43.989228 7f9d8116f700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:43.989251 7f9d8116f700 15 _write 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 450560~1548288
2014-06-12 14:46:43.989454 7f9d8116f700 10 lookup_strip_header done  cid 3.4e0_head oid 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3
2014-06-12 14:46:43.989691 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:43.995913 7f9d5a16f700  5 -- op tracker -- , seq: 747956, time: 2014-06-12 14:46:43.994263, event: header_read, op: osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.995938 7f9d5a16f700  5 -- op tracker -- , seq: 747956, time: 2014-06-12 14:46:43.994267, event: throttled, op: osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.995956 7f9d5a16f700  5 -- op tracker -- , seq: 747956, time: 2014-06-12 14:46:43.995819, event: all_read, op: osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.995975 7f9d5a16f700  5 -- op tracker -- , seq: 747956, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.995984 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:43.995997 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:43.996013 7f9d5a16f700 15 osd.2 173 enqueue_op 0x34016000 prio 127 cost 516903 latency 0.001749 osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:43.996067 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:43.996079 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34016000 prio 127 cost 516903 latency 0.001816 osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2467 (0'0,173'2467] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2466 active]
2014-06-12 14:46:43.996123 7f9d6a3e5700  5 -- op tracker -- , seq: 747956, time: 2014-06-12 14:46:43.996113, event: reached_pg, op: osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.996138 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2467 (0'0,173'2467] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2466 active] handle_message: 0x34016000
2014-06-12 14:46:43.996162 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2467 (0'0,173'2467] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2466 active] sub_op_modify trans 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 v 173'2468 (transaction) 174
2014-06-12 14:46:43.996196 7f9d6a3e5700  5 -- op tracker -- , seq: 747956, time: 2014-06-12 14:46:43.996187, event: started, op: osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.996225 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2467 (0'0,173'2467] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2466 active] append_log log((0'0,173'2467], crt=0'0) [173'2468 (173'2467) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143627 2014-06-12 08:43:18.124172]
2014-06-12 14:46:43.996271 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2468 (0'0,173'2468] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2466 active] add_log_entry 173'2468 (173'2467) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143627 2014-06-12 08:43:18.124172
2014-06-12 14:46:43.996318 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2468 (0'0,173'2468] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2466 active] append_log  adding 1 keys
2014-06-12 14:46:43.996382 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2468, trimmed: 
2014-06-12 14:46:43.996426 7f9d6a3e5700  5 -- op tracker -- , seq: 747956, time: 2014-06-12 14:46:43.996416, event: started, op: osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:43.996444 7f9d6a3e5700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:43.996452 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:43.996454 7f9d6a3e5700  5 queue_op 0x6de0ea0 seq 2301 osr(3.4f4 0x276db70) 518389 bytes   (queue has 2 ops and 2068970 bytes)
2014-06-12 14:46:43.996468 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2468 (0'0,173'2468] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2466 active] do_sub_op osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:43.996499 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34016000 finish
2014-06-12 14:46:43.996507 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:43.996512 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:43.996526 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (2 active)
2014-06-12 14:46:43.996534 7f9d79960700  5 _do_op 0x6de0ea0 seq 2301 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:46:43.996542 7f9d79960700 10 _do_transaction on 0xc9a5970
2014-06-12 14:46:43.996551 7f9d79960700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:43.996717 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:43.996767 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:43.997792 7f9d8116f700 10 _generic_write 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 450560~1548288 = 0
2014-06-12 14:46:43.997980 7f9d8116f700 15 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3
2014-06-12 14:46:43.998003 7f9d8116f700 10 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:43.998011 7f9d8116f700 15 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3
2014-06-12 14:46:43.998019 7f9d8116f700 10 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:43.998413 7f9d8116f700 20 set_header setting 17854 cid 3.4e0_head oid 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:43.998530 7f9d8116f700 20 set_header setting 2236 cid meta oid 1fe07435/pglog_3.4e0/0//-1 parent seq 0
2014-06-12 14:46:43.998543 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:43.998548 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:44.004383 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.004436 7f9d79960700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:44.004444 7f9d79960700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:46:44.004449 7f9d79960700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:44.004463 7f9d79960700 15 _write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 1073152~516096
2014-06-12 14:46:44.004734 7f9d8116f700 10 _do_op 0x6e69980 seq 2437 r = 0, finisher 0x73c1480 0
2014-06-12 14:46:44.004747 7f9d8116f700 10 _finish_op 0x6e69980 seq 2437 osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:44.004760 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5180900 (1 active)
2014-06-12 14:46:44.004766 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.004782 7f9d7715b700  5 -- op tracker -- , seq: 747955, time: 2014-06-12 14:46:44.004770, event: commit_sent, op: osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.004798 7f9d79960700 10 lookup_strip_header done  cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:44.004804 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2637 (0'0,173'2637] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2635 active] sub_op_modify_commit on op osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:44.004834 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:44.004881 7f9d7795c700  5 -- op tracker -- , seq: 747955, time: 2014-06-12 14:46:44.004874, event: sub_op_applied, op: osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.004894 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2637 (0'0,173'2637] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2636 active] sub_op_modify_applied on 0x206ec1c0 op osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.004916 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2637 (0'0,173'2637] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2636 active] op_applied on primary on version 173'2637
2014-06-12 14:46:44.004930 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2637 (0'0,173'2637] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2636 active] op_applied on replica on version 173'2637
2014-06-12 14:46:44.004945 7f9d7795c700  5 -- op tracker -- , seq: 747955, time: 2014-06-12 14:46:44.004942, event: done, op: osd_sub_op(client.4293.0:143961 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2637 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.004956 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:44.006586 7f9d79960700 10 _generic_write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 1073152~516096 = 0
2014-06-12 14:46:44.006628 7f9d79960700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:44.006642 7f9d79960700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:44.006649 7f9d79960700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:44.006656 7f9d79960700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:44.006861 7f9d79960700 20 set_header setting 17388 cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 parent seq 0
2014-06-12 14:46:44.006935 7f9d79960700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:46:44.006951 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.006962 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:44.009838 7f9d79960700 10 _do_op 0x6de0ea0 seq 2301 r = 0, finisher 0x28af4e0 0
2014-06-12 14:46:44.009866 7f9d79960700 10 _finish_op 0x6de0ea0 seq 2301 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:44.009873 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:46:44.009888 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.009920 7f9d7795c700  5 -- op tracker -- , seq: 747956, time: 2014-06-12 14:46:44.009907, event: sub_op_applied, op: osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.009931 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2468 (0'0,173'2468] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2466 active] sub_op_modify_applied on 0xc9a5880 op osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.009957 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:44.009975 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2468 (0'0,173'2468] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2466 active] op_applied on primary on version 173'2468
2014-06-12 14:46:44.009983 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2468 (0'0,173'2468] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2466 active] op_applied on replica on version 173'2468
2014-06-12 14:46:44.010003 7f9d7715b700  5 -- op tracker -- , seq: 747956, time: 2014-06-12 14:46:44.009996, event: commit_sent, op: osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.010015 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2468 (0'0,173'2468] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2466 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:44.010031 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:44.010049 7f9d7715b700  5 -- op tracker -- , seq: 747956, time: 2014-06-12 14:46:44.010046, event: done, op: osd_sub_op(client.4294.0:143627 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2468 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.154595 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:44.171567 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.185277 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.299629 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.299681 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.326362 7f9d56432700  5 -- op tracker -- , seq: 747957, time: 2014-06-12 14:46:44.325415, event: header_read, op: osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.326381 7f9d56432700  5 -- op tracker -- , seq: 747957, time: 2014-06-12 14:46:44.325419, event: throttled, op: osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.326395 7f9d56432700  5 -- op tracker -- , seq: 747957, time: 2014-06-12 14:46:44.326313, event: all_read, op: osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.326401 7f9d56432700  5 -- op tracker -- , seq: 747957, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.326404 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:44.326408 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:44.326413 7f9d56432700 15 osd.2 173 enqueue_op 0x9fad000 prio 127 cost 516903 latency 0.000998 osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.326464 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.326478 7f9d63bd8700 10 osd.2 173 dequeue_op 0x9fad000 prio 127 cost 516903 latency 0.001062 osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.71f( v 173'2698 (0'0,173'2698] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2697 active]
2014-06-12 14:46:44.326515 7f9d63bd8700  5 -- op tracker -- , seq: 747957, time: 2014-06-12 14:46:44.326510, event: reached_pg, op: osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.326534 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2698 (0'0,173'2698] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2697 active] handle_message: 0x9fad000
2014-06-12 14:46:44.326546 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2698 (0'0,173'2698] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2697 active] sub_op_modify trans c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 v 173'2699 (transaction) 174
2014-06-12 14:46:44.326564 7f9d63bd8700  5 -- op tracker -- , seq: 747957, time: 2014-06-12 14:46:44.326559, event: started, op: osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.326578 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2698 (0'0,173'2698] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2697 active] append_log log((0'0,173'2698], crt=0'0) [173'2699 (173'2698) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145286 2014-06-12 08:45:27.519486]
2014-06-12 14:46:44.326609 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2699 (0'0,173'2699] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2697 active] add_log_entry 173'2699 (173'2698) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145286 2014-06-12 08:45:27.519486
2014-06-12 14:46:44.326632 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2699 (0'0,173'2699] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2697 active] append_log  adding 1 keys
2014-06-12 14:46:44.326652 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2699, trimmed: 
2014-06-12 14:46:44.326666 7f9d63bd8700  5 -- op tracker -- , seq: 747957, time: 2014-06-12 14:46:44.326663, event: started, op: osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.326673 7f9d63bd8700  5 queue_transactions existing osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:44.326676 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.326677 7f9d63bd8700  5 queue_op 0x684d500 seq 2531 osr(3.71f 0x276de50) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:44.326681 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.71f( v 173'2699 (0'0,173'2699] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2697 active] do_sub_op osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.326690 7f9d63bd8700 10 osd.2 173 dequeue_op 0x9fad000 finish
2014-06-12 14:46:44.326693 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.326694 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.326687 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d0240 (1 active)
2014-06-12 14:46:44.326701 7f9d8016d700  5 _do_op 0x684d500 seq 2531 osr(3.71f 0x276de50)/0x276de50 start
2014-06-12 14:46:44.326705 7f9d8016d700 10 _do_transaction on 0x2e637430
2014-06-12 14:46:44.326712 7f9d8016d700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:44.326886 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:44.326940 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.327023 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.327129 7f9d8016d700 15 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:44.327147 7f9d8016d700 10 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1 r = 0
2014-06-12 14:46:44.327162 7f9d8016d700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:44.327201 7f9d8016d700 15 _write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 2314240~516096
2014-06-12 14:46:44.327422 7f9d8016d700 10 lookup_strip_header done  cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:44.327470 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:44.329879 7f9d8016d700 10 _generic_write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 2314240~516096 = 0
2014-06-12 14:46:44.329961 7f9d8016d700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:44.329990 7f9d8016d700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:44.330005 7f9d8016d700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:44.330023 7f9d8016d700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:44.330326 7f9d8016d700 20 set_header setting 17162 cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 parent seq 0
2014-06-12 14:46:44.330435 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.330468 7f9d8016d700 20 set_header setting 2330 cid meta oid 1fe0985b/pglog_3.71f/0//-1 parent seq 0
2014-06-12 14:46:44.330491 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:44.333965 7f9d8016d700 10 _do_op 0x684d500 seq 2531 r = 0, finisher 0x6640140 0
2014-06-12 14:46:44.333980 7f9d8016d700 10 _finish_op 0x684d500 seq 2531 osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:44.333986 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d0240 (0 active)
2014-06-12 14:46:44.333993 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.333995 7f9d7715b700  5 -- op tracker -- , seq: 747957, time: 2014-06-12 14:46:44.333986, event: commit_sent, op: osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.334004 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2699 (0'0,173'2699] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2697 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:44.334021 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.334048 7f9d7795c700  5 -- op tracker -- , seq: 747957, time: 2014-06-12 14:46:44.334042, event: sub_op_applied, op: osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.334055 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2699 (0'0,173'2699] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active] sub_op_modify_applied on 0x2e637340 op osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.334066 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2699 (0'0,173'2699] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active] op_applied on primary on version 173'2699
2014-06-12 14:46:44.334071 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2699 (0'0,173'2699] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active] op_applied on replica on version 173'2699
2014-06-12 14:46:44.334101 7f9d7795c700  5 -- op tracker -- , seq: 747957, time: 2014-06-12 14:46:44.334092, event: done, op: osd_sub_op(client.4308.0:145286 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.346543 7f9d56432700  5 -- op tracker -- , seq: 747958, time: 2014-06-12 14:46:44.345127, event: header_read, op: osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.346574 7f9d56432700  5 -- op tracker -- , seq: 747958, time: 2014-06-12 14:46:44.345131, event: throttled, op: osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.346593 7f9d56432700  5 -- op tracker -- , seq: 747958, time: 2014-06-12 14:46:44.346471, event: all_read, op: osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.346620 7f9d56432700  5 -- op tracker -- , seq: 747958, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.346630 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:44.346643 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:44.346658 7f9d56432700 15 osd.2 173 enqueue_op 0x9fac500 prio 127 cost 516903 latency 0.001531 osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.346747 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.346761 7f9d68be2700 10 osd.2 173 dequeue_op 0x9fac500 prio 127 cost 516903 latency 0.001634 osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.582( v 173'2760 (0'0,173'2760] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2759 active]
2014-06-12 14:46:44.346785 7f9d68be2700  5 -- op tracker -- , seq: 747958, time: 2014-06-12 14:46:44.346781, event: reached_pg, op: osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.346826 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2760 (0'0,173'2760] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2759 active] handle_message: 0x9fac500
2014-06-12 14:46:44.346845 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2760 (0'0,173'2760] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2759 active] sub_op_modify trans 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 v 173'2761 (transaction) 174
2014-06-12 14:46:44.346868 7f9d68be2700  5 -- op tracker -- , seq: 747958, time: 2014-06-12 14:46:44.346862, event: started, op: osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.346890 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2760 (0'0,173'2760] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2759 active] append_log log((0'0,173'2760], crt=0'0) [173'2761 (173'2760) modify   264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 by client.4317.0:143785 2014-06-12 08:44:56.691375]
2014-06-12 14:46:44.346930 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2761 (0'0,173'2761] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2759 active] add_log_entry 173'2761 (173'2760) modify   264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 by client.4317.0:143785 2014-06-12 08:44:56.691375
2014-06-12 14:46:44.346964 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2761 (0'0,173'2761] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2759 active] append_log  adding 1 keys
2014-06-12 14:46:44.347007 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2761, trimmed: 
2014-06-12 14:46:44.347047 7f9d68be2700  5 -- op tracker -- , seq: 747958, time: 2014-06-12 14:46:44.347034, event: started, op: osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.347058 7f9d68be2700  5 queue_transactions existing osr(3.582 0x276da70)/0x276da70
2014-06-12 14:46:44.347061 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.347063 7f9d68be2700  5 queue_op 0x8097ec0 seq 2609 osr(3.582 0x276da70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:44.347074 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438900 (1 active)
2014-06-12 14:46:44.347081 7f9d7915f700  5 _do_op 0x8097ec0 seq 2609 osr(3.582 0x276da70)/0x276da70 start
2014-06-12 14:46:44.347068 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.582( v 173'2761 (0'0,173'2761] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2759 active] do_sub_op osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.347084 7f9d68be2700 10 osd.2 173 dequeue_op 0x9fac500 finish
2014-06-12 14:46:44.347086 7f9d7915f700 10 _do_transaction on 0xc9b4d30
2014-06-12 14:46:44.347088 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.347091 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.347093 7f9d7915f700 15 _omap_setkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:44.347191 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:44.347212 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.347245 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.347265 7f9d7915f700 15 _omap_rmkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:44.347271 7f9d7915f700 10 _omap_rmkeys meta/1fe177ea/pglog_3.582/0//-1 r = 0
2014-06-12 14:46:44.347275 7f9d7915f700 15 _omap_setkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:44.347289 7f9d7915f700 15 _write 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 1519616~516096
2014-06-12 14:46:44.347521 7f9d7915f700 10 lookup_strip_header done  cid 3.582_head oid 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:44.347623 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:44.349858 7f9d7915f700 10 _generic_write 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 1519616~516096 = 0
2014-06-12 14:46:44.349914 7f9d7915f700 15 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:44.349942 7f9d7915f700 10 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:44.349950 7f9d7915f700 15 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:44.349960 7f9d7915f700 10 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:44.350171 7f9d7915f700 20 set_header setting 16739 cid 3.582_head oid 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 parent seq 0
2014-06-12 14:46:44.350265 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.350289 7f9d7915f700 20 set_header setting 2318 cid meta oid 1fe177ea/pglog_3.582/0//-1 parent seq 0
2014-06-12 14:46:44.350308 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:44.352457 7f9d7915f700 10 _do_op 0x8097ec0 seq 2609 r = 0, finisher 0x76af360 0
2014-06-12 14:46:44.352484 7f9d7915f700 10 _finish_op 0x8097ec0 seq 2609 osr(3.582 0x276da70)/0x276da70
2014-06-12 14:46:44.352494 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438900 (0 active)
2014-06-12 14:46:44.352504 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.352520 7f9d7715b700  5 -- op tracker -- , seq: 747958, time: 2014-06-12 14:46:44.352502, event: commit_sent, op: osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.352541 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2761 (0'0,173'2761] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2759 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:44.352585 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.352664 7f9d7795c700  5 -- op tracker -- , seq: 747958, time: 2014-06-12 14:46:44.352648, event: sub_op_applied, op: osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.352684 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2761 (0'0,173'2761] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2760 active] sub_op_modify_applied on 0xc9b4c40 op osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.352724 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2761 (0'0,173'2761] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2760 active] op_applied on primary on version 173'2761
2014-06-12 14:46:44.352737 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2761 (0'0,173'2761] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2760 active] op_applied on replica on version 173'2761
2014-06-12 14:46:44.352760 7f9d7795c700  5 -- op tracker -- , seq: 747958, time: 2014-06-12 14:46:44.352756, event: done, op: osd_sub_op(client.4317.0:143785 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2761 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.368236 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:44.370088 7f9d5a16f700  5 -- op tracker -- , seq: 747959, time: 2014-06-12 14:46:44.369575, event: header_read, op: osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.370099 7f9d5a16f700  5 -- op tracker -- , seq: 747959, time: 2014-06-12 14:46:44.369580, event: throttled, op: osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.370112 7f9d5a16f700  5 -- op tracker -- , seq: 747959, time: 2014-06-12 14:46:44.370043, event: all_read, op: osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.370118 7f9d5a16f700  5 -- op tracker -- , seq: 747959, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.370124 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:44.370129 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:44.370136 7f9d5a16f700 15 osd.2 173 enqueue_op 0x34017200 prio 127 cost 287527 latency 0.000561 osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.370167 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.370180 7f9d623d5700 10 osd.2 173 dequeue_op 0x34017200 prio 127 cost 287527 latency 0.000604 osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2669 (0'0,173'2669] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2668 active]
2014-06-12 14:46:44.370200 7f9d623d5700  5 -- op tracker -- , seq: 747959, time: 2014-06-12 14:46:44.370198, event: reached_pg, op: osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.370208 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2669 (0'0,173'2669] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2668 active] handle_message: 0x34017200
2014-06-12 14:46:44.370217 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2669 (0'0,173'2669] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2668 active] sub_op_modify trans f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 v 173'2670 (transaction) 174
2014-06-12 14:46:44.370228 7f9d623d5700  5 -- op tracker -- , seq: 747959, time: 2014-06-12 14:46:44.370226, event: started, op: osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.370242 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2669 (0'0,173'2669] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2668 active] append_log log((0'0,173'2669], crt=0'0) [173'2670 (173'2669) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146412 2014-06-12 08:41:30.500259]
2014-06-12 14:46:44.370261 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2670 (0'0,173'2670] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2668 active] add_log_entry 173'2670 (173'2669) modify   f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 by client.4316.0:146412 2014-06-12 08:41:30.500259
2014-06-12 14:46:44.370277 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2670 (0'0,173'2670] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2668 active] append_log  adding 1 keys
2014-06-12 14:46:44.370303 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2670, trimmed: 
2014-06-12 14:46:44.370322 7f9d623d5700  5 -- op tracker -- , seq: 747959, time: 2014-06-12 14:46:44.370319, event: started, op: osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.370332 7f9d623d5700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:44.370335 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.370337 7f9d623d5700  5 queue_op 0x9bac480 seq 2480 osr(3.1b7 0x276d1a0) 289013 bytes   (queue has 1 ops and 289013 bytes)
2014-06-12 14:46:44.370342 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2670 (0'0,173'2670] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2668 active] do_sub_op osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.370352 7f9d623d5700 10 osd.2 173 dequeue_op 0x34017200 finish
2014-06-12 14:46:44.370348 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:46:44.370354 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.370356 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.370356 7f9d7d968700  5 _do_op 0x9bac480 seq 2480 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:46:44.370362 7f9d7d968700 10 _do_transaction on 0x29eb30b0
2014-06-12 14:46:44.370369 7f9d7d968700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:44.370496 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:44.370525 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.370568 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.370596 7f9d7d968700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:44.370604 7f9d7d968700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:46:44.370609 7f9d7d968700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:46:44.370628 7f9d7d968700 15 _write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 3907584~286720
2014-06-12 14:46:44.370981 7f9d7d968700 10 lookup_strip_header done  cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:44.371177 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:44.372285 7f9d7d968700 10 _generic_write 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 3907584~286720 = 0
2014-06-12 14:46:44.372322 7f9d7d968700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:44.372338 7f9d7d968700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:44.372346 7f9d7d968700 15 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3
2014-06-12 14:46:44.372354 7f9d7d968700 10 _setattrs 3.1b7_head/f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 = 0
2014-06-12 14:46:44.372514 7f9d7d968700 20 set_header setting 16921 cid 3.1b7_head oid f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 parent seq 0
2014-06-12 14:46:44.372557 7f9d7d968700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:46:44.372575 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.372586 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:44.373608 7f9d7d968700 10 _do_op 0x9bac480 seq 2480 r = 0, finisher 0x895d300 0
2014-06-12 14:46:44.373634 7f9d7d968700 10 _finish_op 0x9bac480 seq 2480 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:46:44.373645 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:46:44.373653 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.373650 7f9d7715b700  5 -- op tracker -- , seq: 747959, time: 2014-06-12 14:46:44.373641, event: commit_sent, op: osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.373672 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2670 (0'0,173'2670] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2668 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:44.373694 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:44.373723 7f9d7795c700  5 -- op tracker -- , seq: 747959, time: 2014-06-12 14:46:44.373718, event: sub_op_applied, op: osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.373731 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2670 (0'0,173'2670] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2669 active] sub_op_modify_applied on 0x29eb2fc0 op osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.373741 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2670 (0'0,173'2670] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2669 active] op_applied on primary on version 173'2670
2014-06-12 14:46:44.373748 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2670 (0'0,173'2670] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2669 active] op_applied on replica on version 173'2670
2014-06-12 14:46:44.373762 7f9d7795c700  5 -- op tracker -- , seq: 747959, time: 2014-06-12 14:46:44.373759, event: done, op: osd_sub_op(client.4316.0:146412 3.1b7 f503c9b7/rbd_data.10b36b8b4567.00000000000010bb/head//3 [] v 173'2670 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.387071 7f9d56432700  5 -- op tracker -- , seq: 747960, time: 2014-06-12 14:46:44.385010, event: header_read, op: osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.387097 7f9d56432700  5 -- op tracker -- , seq: 747960, time: 2014-06-12 14:46:44.385013, event: throttled, op: osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.387113 7f9d56432700  5 -- op tracker -- , seq: 747960, time: 2014-06-12 14:46:44.387008, event: all_read, op: osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.387127 7f9d56432700  5 -- op tracker -- , seq: 747960, time: 0.000000, event: dispatched, op: osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.387134 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:44.387145 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:44.387157 7f9d56432700 15 osd.2 173 enqueue_op 0x9fad400 prio 127 cost 1032999 latency 0.002147 osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.387199 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.387212 7f9d643d9700 10 osd.2 173 dequeue_op 0x9fad400 prio 127 cost 1032999 latency 0.002201 osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2639 (0'0,173'2639] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active]
2014-06-12 14:46:44.387255 7f9d643d9700  5 -- op tracker -- , seq: 747960, time: 2014-06-12 14:46:44.387246, event: reached_pg, op: osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.387270 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2639 (0'0,173'2639] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active] handle_message: 0x9fad400
2014-06-12 14:46:44.387291 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2639 (0'0,173'2639] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active] sub_op_modify trans e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 v 173'2640 (transaction) 174
2014-06-12 14:46:44.387308 7f9d643d9700  5 -- op tracker -- , seq: 747960, time: 2014-06-12 14:46:44.387304, event: started, op: osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.387321 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2639 (0'0,173'2639] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active] append_log log((0'0,173'2639], crt=0'0) [173'2640 (173'2639) modify   e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 by client.4292.0:142424 2014-06-12 08:44:58.004013]
2014-06-12 14:46:44.387352 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2640 (0'0,173'2640] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active] add_log_entry 173'2640 (173'2639) modify   e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 by client.4292.0:142424 2014-06-12 08:44:58.004013
2014-06-12 14:46:44.387375 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2640 (0'0,173'2640] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active] append_log  adding 1 keys
2014-06-12 14:46:44.387400 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2640, trimmed: 
2014-06-12 14:46:44.387414 7f9d643d9700  5 -- op tracker -- , seq: 747960, time: 2014-06-12 14:46:44.387411, event: started, op: osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.387421 7f9d643d9700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:44.387424 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.387424 7f9d643d9700  5 queue_op 0x69b1680 seq 2460 osr(3.1c7 0x276d170) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:44.387433 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:46:44.387430 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2640 (0'0,173'2640] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active] do_sub_op osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.387439 7f9d643d9700 10 osd.2 173 dequeue_op 0x9fad400 finish
2014-06-12 14:46:44.387439 7f9d7c165700  5 _do_op 0x69b1680 seq 2460 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:46:44.387441 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.387443 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.387444 7f9d7c165700 10 _do_transaction on 0x2e636470
2014-06-12 14:46:44.387449 7f9d7c165700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:44.387545 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:44.387569 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.387610 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.387701 7f9d7c165700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:44.387710 7f9d7c165700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:46:44.387716 7f9d7c165700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:44.387735 7f9d7c165700 15 _write 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 1699840~1032192
2014-06-12 14:46:44.387875 7f9d7c165700 10 lookup_strip_header done  cid 3.1c7_head oid e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:44.388041 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:44.388880 7f9d56432700  5 -- op tracker -- , seq: 747961, time: 2014-06-12 14:46:44.387975, event: header_read, op: osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.388893 7f9d56432700  5 -- op tracker -- , seq: 747961, time: 2014-06-12 14:46:44.387977, event: throttled, op: osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.388900 7f9d56432700  5 -- op tracker -- , seq: 747961, time: 2014-06-12 14:46:44.388841, event: all_read, op: osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.388907 7f9d56432700  5 -- op tracker -- , seq: 747961, time: 0.000000, event: dispatched, op: osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.388912 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:44.388918 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:44.388925 7f9d56432700 15 osd.2 173 enqueue_op 0x9fac000 prio 127 cost 516903 latency 0.000950 osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.388958 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.388965 7f9d633d7700 10 osd.2 173 dequeue_op 0x9fac000 prio 127 cost 516903 latency 0.000990 osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.60e( v 173'2560 (0'0,173'2560] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2559 active]
2014-06-12 14:46:44.388990 7f9d633d7700  5 -- op tracker -- , seq: 747961, time: 2014-06-12 14:46:44.388986, event: reached_pg, op: osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.389003 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2560 (0'0,173'2560] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2559 active] handle_message: 0x9fac000
2014-06-12 14:46:44.389017 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2560 (0'0,173'2560] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2559 active] sub_op_modify trans ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 v 173'2561 (transaction) 174
2014-06-12 14:46:44.389032 7f9d633d7700  5 -- op tracker -- , seq: 747961, time: 2014-06-12 14:46:44.389029, event: started, op: osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.389047 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2560 (0'0,173'2560] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2559 active] append_log log((0'0,173'2560], crt=0'0) [173'2561 (173'2560) modify   ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 by client.4296.0:143673 2014-06-12 08:45:46.343574]
2014-06-12 14:46:44.389069 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2561 (0'0,173'2561] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2559 active] add_log_entry 173'2561 (173'2560) modify   ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 by client.4296.0:143673 2014-06-12 08:45:46.343574
2014-06-12 14:46:44.389091 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2561 (0'0,173'2561] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2559 active] append_log  adding 1 keys
2014-06-12 14:46:44.389127 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2561, trimmed: 
2014-06-12 14:46:44.389153 7f9d633d7700  5 -- op tracker -- , seq: 747961, time: 2014-06-12 14:46:44.389147, event: started, op: osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.389169 7f9d633d7700  5 queue_transactions existing osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:44.389174 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.389176 7f9d633d7700  5 queue_op 0x1a59b8c0 seq 2403 osr(3.60e 0x276dcf0) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:46:44.389188 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533cd80 (2 active)
2014-06-12 14:46:44.389183 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.60e( v 173'2561 (0'0,173'2561] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2559 active] do_sub_op osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.389196 7f9d83173700  5 _do_op 0x1a59b8c0 seq 2403 osr(3.60e 0x276dcf0)/0x276dcf0 start
2014-06-12 14:46:44.389200 7f9d633d7700 10 osd.2 173 dequeue_op 0x9fac000 finish
2014-06-12 14:46:44.389202 7f9d83173700 10 _do_transaction on 0x5a435f0
2014-06-12 14:46:44.389204 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.389207 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.389208 7f9d83173700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:44.389301 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:44.389321 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.392784 7f9d7c165700 10 _generic_write 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 1699840~1032192 = 0
2014-06-12 14:46:44.392888 7f9d7c165700 15 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:44.392924 7f9d7c165700 10 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:44.392928 7f9d7c165700 15 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:44.392933 7f9d7c165700 10 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:44.393166 7f9d7c165700 20 set_header setting 18233 cid 3.1c7_head oid e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 parent seq 0
2014-06-12 14:46:44.393261 7f9d7c165700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:46:44.393273 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.393286 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:44.397437 7f9d5a16f700  5 -- op tracker -- , seq: 747962, time: 2014-06-12 14:46:44.395125, event: header_read, op: osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.397451 7f9d5a16f700  5 -- op tracker -- , seq: 747962, time: 2014-06-12 14:46:44.395128, event: throttled, op: osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.397459 7f9d5a16f700  5 -- op tracker -- , seq: 747962, time: 2014-06-12 14:46:44.397385, event: all_read, op: osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.397466 7f9d5a16f700  5 -- op tracker -- , seq: 747962, time: 0.000000, event: dispatched, op: osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.397470 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:44.397477 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:44.397483 7f9d5a16f700 15 osd.2 173 enqueue_op 0x34016a00 prio 127 cost 1032999 latency 0.002358 osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.397529 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.397549 7f9d61bd4700 10 osd.2 173 dequeue_op 0x34016a00 prio 127 cost 1032999 latency 0.002424 osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2721 (0'0,173'2721] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active]
2014-06-12 14:46:44.397569 7f9d61bd4700  5 -- op tracker -- , seq: 747962, time: 2014-06-12 14:46:44.397566, event: reached_pg, op: osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.397576 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2721 (0'0,173'2721] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] handle_message: 0x34016a00
2014-06-12 14:46:44.397584 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2721 (0'0,173'2721] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] sub_op_modify trans cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 v 173'2722 (transaction) 174
2014-06-12 14:46:44.397593 7f9d61bd4700  5 -- op tracker -- , seq: 747962, time: 2014-06-12 14:46:44.397591, event: started, op: osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.397626 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2721 (0'0,173'2721] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] append_log log((0'0,173'2721], crt=0'0) [173'2722 (173'2721) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143144 2014-06-12 08:43:32.902565]
2014-06-12 14:46:44.397651 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2722 (0'0,173'2722] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] add_log_entry 173'2722 (173'2721) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143144 2014-06-12 08:43:32.902565
2014-06-12 14:46:44.397669 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2722 (0'0,173'2722] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] append_log  adding 1 keys
2014-06-12 14:46:44.397690 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2722, trimmed: 
2014-06-12 14:46:44.397704 7f9d61bd4700  5 -- op tracker -- , seq: 747962, time: 2014-06-12 14:46:44.397701, event: started, op: osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.397727 7f9d61bd4700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:44.397731 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.397733 7f9d61bd4700  5 queue_op 0x15168e40 seq 2469 osr(3.21c 0x276d020) 1034485 bytes   (queue has 3 ops and 2587359 bytes)
2014-06-12 14:46:44.397741 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2722 (0'0,173'2722] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] do_sub_op osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.397755 7f9d61bd4700 10 osd.2 173 dequeue_op 0x34016a00 finish
2014-06-12 14:46:44.397758 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.397759 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.397765 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (3 active)
2014-06-12 14:46:44.397769 7f9d82972700  5 _do_op 0x15168e40 seq 2469 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:46:44.397772 7f9d82972700 10 _do_transaction on 0x288c4d30
2014-06-12 14:46:44.397776 7f9d82972700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:44.397883 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:44.397897 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.398842 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.398879 7f9d83173700 15 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:44.398887 7f9d83173700 10 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1 r = 0
2014-06-12 14:46:44.398894 7f9d83173700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:44.399038 7f9d83173700 15 _write 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 2744320~516096
2014-06-12 14:46:44.399041 7f9d7c165700 10 _do_op 0x69b1680 seq 2460 r = 0, finisher 0x6615d40 0
2014-06-12 14:46:44.399056 7f9d7c165700 10 _finish_op 0x69b1680 seq 2460 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:44.399063 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (2 active)
2014-06-12 14:46:44.399066 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.399205 7f9d83173700 10 lookup_strip_header done  cid 3.60e_head oid ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:44.399320 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:44.399527 7f9d7795c700  5 -- op tracker -- , seq: 747960, time: 2014-06-12 14:46:44.399518, event: sub_op_applied, op: osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.399538 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2640 (0'0,173'2640] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active] sub_op_modify_applied on 0x2e636380 op osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.399582 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.399611 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2640 (0'0,173'2640] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active] op_applied on primary on version 173'2640
2014-06-12 14:46:44.399628 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2640 (0'0,173'2640] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active] op_applied on replica on version 173'2640
2014-06-12 14:46:44.402061 7f9d83173700 10 _generic_write 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 2744320~516096 = 0
2014-06-12 14:46:44.402128 7f9d83173700 15 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:44.402148 7f9d83173700 10 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 = 0
2014-06-12 14:46:44.402155 7f9d83173700 15 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:44.402171 7f9d83173700 10 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 = 0
2014-06-12 14:46:44.402397 7f9d83173700 20 set_header setting 17046 cid 3.60e_head oid ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 parent seq 0
2014-06-12 14:46:44.402496 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.402511 7f9d83173700 20 set_header setting 2240 cid meta oid 1fe13dcb/pglog_3.60e/0//-1 parent seq 0
2014-06-12 14:46:44.402522 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:44.403358 7f9d7715b700  5 -- op tracker -- , seq: 747960, time: 2014-06-12 14:46:44.403340, event: commit_sent, op: osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.403376 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2640 (0'0,173'2640] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2638 active] sub_op_modify_commit on op osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:44.403414 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.403470 7f9d7715b700  5 -- op tracker -- , seq: 747960, time: 2014-06-12 14:46:44.403465, event: done, op: osd_sub_op(client.4292.0:142424 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2640 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.405772 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.405839 7f9d82972700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:44.405847 7f9d82972700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:46:44.405857 7f9d82972700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:44.405877 7f9d82972700 15 _write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 638976~1032192
2014-06-12 14:46:44.405963 7f9d83173700 10 _do_op 0x1a59b8c0 seq 2403 r = 0, finisher 0x8595ac0 0
2014-06-12 14:46:44.405979 7f9d83173700 10 _finish_op 0x1a59b8c0 seq 2403 osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:44.405987 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533cd80 (1 active)
2014-06-12 14:46:44.405998 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.406003 7f9d7715b700  5 -- op tracker -- , seq: 747961, time: 2014-06-12 14:46:44.405994, event: commit_sent, op: osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.406029 7f9d82972700 10 lookup_strip_header done  cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:44.406024 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2561 (0'0,173'2561] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2559 active] sub_op_modify_commit on op osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:44.406054 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.406089 7f9d7795c700  5 -- op tracker -- , seq: 747961, time: 2014-06-12 14:46:44.406082, event: sub_op_applied, op: osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.406095 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2561 (0'0,173'2561] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2560 active] sub_op_modify_applied on 0x5a43500 op osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.406111 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2561 (0'0,173'2561] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2560 active] op_applied on primary on version 173'2561
2014-06-12 14:46:44.406121 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2561 (0'0,173'2561] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2560 active] op_applied on replica on version 173'2561
2014-06-12 14:46:44.406143 7f9d7795c700  5 -- op tracker -- , seq: 747961, time: 2014-06-12 14:46:44.406139, event: done, op: osd_sub_op(client.4296.0:143673 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2561 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.406264 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:44.410132 7f9d82972700 10 _generic_write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 638976~1032192 = 0
2014-06-12 14:46:44.410216 7f9d82972700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:44.410234 7f9d82972700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:44.410239 7f9d82972700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:44.410244 7f9d82972700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:44.410530 7f9d82972700 20 set_header setting 16774 cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 parent seq 0
2014-06-12 14:46:44.410644 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.410657 7f9d82972700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:46:44.410664 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:44.414802 7f9d82972700 10 _do_op 0x15168e40 seq 2469 r = 0, finisher 0x20aac4a0 0
2014-06-12 14:46:44.414820 7f9d82972700 10 _finish_op 0x15168e40 seq 2469 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:44.414827 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (0 active)
2014-06-12 14:46:44.414836 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.414843 7f9d7715b700  5 -- op tracker -- , seq: 747962, time: 2014-06-12 14:46:44.414831, event: commit_sent, op: osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.414859 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2722 (0'0,173'2722] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] sub_op_modify_commit on op osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:44.414888 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:44.414924 7f9d7795c700  5 -- op tracker -- , seq: 747962, time: 2014-06-12 14:46:44.414917, event: sub_op_applied, op: osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.414933 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2722 (0'0,173'2722] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] sub_op_modify_applied on 0x288c4c40 op osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.414949 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2722 (0'0,173'2722] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] op_applied on primary on version 173'2722
2014-06-12 14:46:44.414957 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2722 (0'0,173'2722] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] op_applied on replica on version 173'2722
2014-06-12 14:46:44.414976 7f9d7795c700  5 -- op tracker -- , seq: 747962, time: 2014-06-12 14:46:44.414973, event: done, op: osd_sub_op(client.4279.0:143144 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.478467 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:44.478493 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:44.478496 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:44.478497 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:44.478500 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 6; oldest is 11.317155 seconds old
2014-06-12 14:46:44.478512 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:44.565084 7f9d56432700  5 -- op tracker -- , seq: 747963, time: 2014-06-12 14:46:44.563862, event: header_read, op: osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.565133 7f9d56432700  5 -- op tracker -- , seq: 747963, time: 2014-06-12 14:46:44.563867, event: throttled, op: osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.565154 7f9d56432700  5 -- op tracker -- , seq: 747963, time: 2014-06-12 14:46:44.564989, event: all_read, op: osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.565199 7f9d56432700  5 -- op tracker -- , seq: 747963, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.565210 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:44.565225 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:44.565248 7f9d56432700 15 osd.2 173 enqueue_op 0x9fade00 prio 127 cost 516903 latency 0.001385 osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.565293 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.565305 7f9d673df700 10 osd.2 173 dequeue_op 0x9fade00 prio 127 cost 516903 latency 0.001443 osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.582( v 173'2761 (0'0,173'2761] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2760 active]
2014-06-12 14:46:44.565345 7f9d673df700  5 -- op tracker -- , seq: 747963, time: 2014-06-12 14:46:44.565336, event: reached_pg, op: osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.565363 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2761 (0'0,173'2761] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2760 active] handle_message: 0x9fade00
2014-06-12 14:46:44.565389 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2761 (0'0,173'2761] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2760 active] sub_op_modify trans 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 v 173'2762 (transaction) 174
2014-06-12 14:46:44.565434 7f9d673df700  5 -- op tracker -- , seq: 747963, time: 2014-06-12 14:46:44.565423, event: started, op: osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.565467 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2761 (0'0,173'2761] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2760 active] append_log log((0'0,173'2761], crt=0'0) [173'2762 (173'2761) modify   264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 by client.4317.0:143786 2014-06-12 08:44:57.670267]
2014-06-12 14:46:44.565490 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2762 (0'0,173'2762] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2760 active] add_log_entry 173'2762 (173'2761) modify   264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 by client.4317.0:143786 2014-06-12 08:44:57.670267
2014-06-12 14:46:44.565512 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2762 (0'0,173'2762] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2760 active] append_log  adding 1 keys
2014-06-12 14:46:44.565559 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2762, trimmed: 
2014-06-12 14:46:44.565611 7f9d673df700  5 -- op tracker -- , seq: 747963, time: 2014-06-12 14:46:44.565607, event: started, op: osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.565629 7f9d673df700  5 queue_transactions existing osr(3.582 0x276da70)/0x276da70
2014-06-12 14:46:44.565634 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.565636 7f9d673df700  5 queue_op 0x558f9e0 seq 2610 osr(3.582 0x276da70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:44.565643 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.582( v 173'2762 (0'0,173'2762] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2760 active] do_sub_op osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.565659 7f9d673df700 10 osd.2 173 dequeue_op 0x9fade00 finish
2014-06-12 14:46:44.565663 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.565666 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.565684 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438900 (1 active)
2014-06-12 14:46:44.565694 7f9d81970700  5 _do_op 0x558f9e0 seq 2610 osr(3.582 0x276da70)/0x276da70 start
2014-06-12 14:46:44.565699 7f9d81970700 10 _do_transaction on 0x591d270
2014-06-12 14:46:44.565705 7f9d81970700 15 _omap_setkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:44.565875 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:44.565930 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.566020 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.566183 7f9d81970700 15 _omap_rmkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:44.566196 7f9d81970700 10 _omap_rmkeys meta/1fe177ea/pglog_3.582/0//-1 r = 0
2014-06-12 14:46:44.566208 7f9d81970700 15 _omap_setkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:44.566241 7f9d81970700 15 _write 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 2035712~516096
2014-06-12 14:46:44.566518 7f9d81970700 10 lookup_strip_header done  cid 3.582_head oid 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:44.566755 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:44.570506 7f9d81970700 10 _generic_write 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 2035712~516096 = 0
2014-06-12 14:46:44.570642 7f9d81970700 15 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:44.570674 7f9d81970700 10 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:44.570687 7f9d81970700 15 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:44.570705 7f9d81970700 10 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:44.571210 7f9d81970700 20 set_header setting 16739 cid 3.582_head oid 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 parent seq 0
2014-06-12 14:46:44.571387 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.571417 7f9d81970700 20 set_header setting 2318 cid meta oid 1fe177ea/pglog_3.582/0//-1 parent seq 0
2014-06-12 14:46:44.571439 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:44.571802 7f9d56432700  5 -- op tracker -- , seq: 747964, time: 2014-06-12 14:46:44.570393, event: header_read, op: osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.571829 7f9d56432700  5 -- op tracker -- , seq: 747964, time: 2014-06-12 14:46:44.570398, event: throttled, op: osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.571847 7f9d56432700  5 -- op tracker -- , seq: 747964, time: 2014-06-12 14:46:44.571712, event: all_read, op: osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.571865 7f9d56432700  5 -- op tracker -- , seq: 747964, time: 0.000000, event: dispatched, op: osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.571876 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:44.571889 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:44.571904 7f9d56432700 15 osd.2 173 enqueue_op 0x9fad700 prio 127 cost 516903 latency 0.001510 osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.571991 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.572007 7f9d67be0700 10 osd.2 173 dequeue_op 0x9fad700 prio 127 cost 516903 latency 0.001614 osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2640 (0'0,173'2640] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2639 active]
2014-06-12 14:46:44.572056 7f9d67be0700  5 -- op tracker -- , seq: 747964, time: 2014-06-12 14:46:44.572046, event: reached_pg, op: osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.572075 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2640 (0'0,173'2640] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2639 active] handle_message: 0x9fad700
2014-06-12 14:46:44.572099 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2640 (0'0,173'2640] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2639 active] sub_op_modify trans e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 v 173'2641 (transaction) 174
2014-06-12 14:46:44.572133 7f9d67be0700  5 -- op tracker -- , seq: 747964, time: 2014-06-12 14:46:44.572125, event: started, op: osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.572162 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2640 (0'0,173'2640] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2639 active] append_log log((0'0,173'2640], crt=0'0) [173'2641 (173'2640) modify   e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 by client.4292.0:142425 2014-06-12 08:44:58.016630]
2014-06-12 14:46:44.572208 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2641 (0'0,173'2641] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2639 active] add_log_entry 173'2641 (173'2640) modify   e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 by client.4292.0:142425 2014-06-12 08:44:58.016630
2014-06-12 14:46:44.572255 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2641 (0'0,173'2641] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2639 active] append_log  adding 1 keys
2014-06-12 14:46:44.572321 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2641, trimmed: 
2014-06-12 14:46:44.572367 7f9d67be0700  5 -- op tracker -- , seq: 747964, time: 2014-06-12 14:46:44.572356, event: started, op: osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.572385 7f9d67be0700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:44.572392 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.572395 7f9d67be0700  5 queue_op 0x5bf3da0 seq 2461 osr(3.1c7 0x276d170) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:44.572409 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2641 (0'0,173'2641] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2639 active] do_sub_op osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.572439 7f9d67be0700 10 osd.2 173 dequeue_op 0x9fad700 finish
2014-06-12 14:46:44.572447 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.572460 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.572474 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (2 active)
2014-06-12 14:46:44.572482 7f9d7c966700  5 _do_op 0x5bf3da0 seq 2461 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:46:44.572489 7f9d7c966700 10 _do_transaction on 0x284109b0
2014-06-12 14:46:44.572499 7f9d7c966700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:44.574624 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:44.574670 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.579127 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.579171 7f9d7c966700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:44.579180 7f9d7c966700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:46:44.579188 7f9d7c966700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:44.579207 7f9d7c966700 15 _write 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 2732032~516096
2014-06-12 14:46:44.579288 7f9d81970700 10 _do_op 0x558f9e0 seq 2610 r = 0, finisher 0x69e7340 0
2014-06-12 14:46:44.579316 7f9d7c966700 10 lookup_strip_header done  cid 3.1c7_head oid e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:44.579320 7f9d7715b700  5 -- op tracker -- , seq: 747963, time: 2014-06-12 14:46:44.579312, event: commit_sent, op: osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.579341 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2762 (0'0,173'2762] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2760 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:44.579368 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.579325 7f9d81970700 10 _finish_op 0x558f9e0 seq 2610 osr(3.582 0x276da70)/0x276da70
2014-06-12 14:46:44.579426 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438900 (1 active)
2014-06-12 14:46:44.579433 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.579440 7f9d7795c700  5 -- op tracker -- , seq: 747963, time: 2014-06-12 14:46:44.579433, event: sub_op_applied, op: osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.579451 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2762 (0'0,173'2762] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2761 active] sub_op_modify_applied on 0x591d180 op osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.579470 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2762 (0'0,173'2762] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2761 active] op_applied on primary on version 173'2762
2014-06-12 14:46:44.579481 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2762 (0'0,173'2762] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2761 active] op_applied on replica on version 173'2762
2014-06-12 14:46:44.579505 7f9d7795c700  5 -- op tracker -- , seq: 747963, time: 2014-06-12 14:46:44.579500, event: done, op: osd_sub_op(client.4317.0:143786 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2762 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.579555 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:44.580989 7f9d7c966700 10 _generic_write 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 2732032~516096 = 0
2014-06-12 14:46:44.581045 7f9d7c966700 15 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:44.581070 7f9d7c966700 10 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:44.581078 7f9d7c966700 15 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:44.581088 7f9d7c966700 10 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:44.581288 7f9d7c966700 20 set_header setting 18233 cid 3.1c7_head oid e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 parent seq 0
2014-06-12 14:46:44.581379 7f9d7c966700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:46:44.581399 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.581410 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:44.583721 7f9d7c966700 10 _do_op 0x5bf3da0 seq 2461 r = 0, finisher 0x80c13a0 0
2014-06-12 14:46:44.583765 7f9d7715b700  5 -- op tracker -- , seq: 747964, time: 2014-06-12 14:46:44.583755, event: commit_sent, op: osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.583776 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2641 (0'0,173'2641] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2639 active] sub_op_modify_commit on op osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:44.583811 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.583859 7f9d7c966700 10 _finish_op 0x5bf3da0 seq 2461 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:44.583885 7f9d7795c700  5 -- op tracker -- , seq: 747964, time: 2014-06-12 14:46:44.583879, event: sub_op_applied, op: osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.583892 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2641 (0'0,173'2641] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2640 active] sub_op_modify_applied on 0x284108c0 op osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.583910 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2641 (0'0,173'2641] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2640 active] op_applied on primary on version 173'2641
2014-06-12 14:46:44.583922 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2641 (0'0,173'2641] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2640 active] op_applied on replica on version 173'2641
2014-06-12 14:46:44.583951 7f9d7c966700  5 -- op tracker -- , seq: 747964, time: 2014-06-12 14:46:44.583946, event: done, op: osd_sub_op(client.4292.0:142425 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2641 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.583978 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:46:44.583983 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.621223 7f9d5a16f700  5 -- op tracker -- , seq: 747965, time: 2014-06-12 14:46:44.618401, event: header_read, op: osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.621234 7f9d5a16f700  5 -- op tracker -- , seq: 747965, time: 2014-06-12 14:46:44.618408, event: throttled, op: osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.621249 7f9d5a16f700  5 -- op tracker -- , seq: 747965, time: 2014-06-12 14:46:44.621189, event: all_read, op: osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.621255 7f9d5a16f700  5 -- op tracker -- , seq: 747965, time: 0.000000, event: dispatched, op: osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.621258 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:44.621263 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:44.621269 7f9d5a16f700 15 osd.2 173 enqueue_op 0x34017500 prio 127 cost 1549095 latency 0.002868 osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.621292 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.621300 7f9d663dd700 10 osd.2 173 dequeue_op 0x34017500 prio 127 cost 1549095 latency 0.002899 osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4e0( v 173'2637 (0'0,173'2637] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2636 active]
2014-06-12 14:46:44.621323 7f9d663dd700  5 -- op tracker -- , seq: 747965, time: 2014-06-12 14:46:44.621319, event: reached_pg, op: osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.621336 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2637 (0'0,173'2637] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2636 active] handle_message: 0x34017500
2014-06-12 14:46:44.621348 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2637 (0'0,173'2637] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2636 active] sub_op_modify trans 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 v 173'2638 (transaction) 174
2014-06-12 14:46:44.621364 7f9d663dd700  5 -- op tracker -- , seq: 747965, time: 2014-06-12 14:46:44.621360, event: started, op: osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.621379 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2637 (0'0,173'2637] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2636 active] append_log log((0'0,173'2637], crt=0'0) [173'2638 (173'2637) modify   79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 by client.4293.0:143962 2014-06-12 08:43:56.508768]
2014-06-12 14:46:44.621404 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2638 (0'0,173'2638] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2636 active] add_log_entry 173'2638 (173'2637) modify   79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 by client.4293.0:143962 2014-06-12 08:43:56.508768
2014-06-12 14:46:44.621425 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2638 (0'0,173'2638] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2636 active] append_log  adding 1 keys
2014-06-12 14:46:44.621458 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2638, trimmed: 
2014-06-12 14:46:44.621488 7f9d663dd700  5 -- op tracker -- , seq: 747965, time: 2014-06-12 14:46:44.621483, event: started, op: osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.621499 7f9d663dd700  5 queue_transactions existing osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:44.621503 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.621504 7f9d663dd700  5 queue_op 0x5a73320 seq 2438 osr(3.4e0 0x276dbd0) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:46:44.621512 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2638 (0'0,173'2638] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2636 active] do_sub_op osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.621527 7f9d663dd700 10 osd.2 173 dequeue_op 0x34017500 finish
2014-06-12 14:46:44.621531 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.621533 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.621540 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5180900 (1 active)
2014-06-12 14:46:44.621544 7f9d7a962700  5 _do_op 0x5a73320 seq 2438 osr(3.4e0 0x276dbd0)/0x276dbd0 start
2014-06-12 14:46:44.621547 7f9d7a962700 10 _do_transaction on 0x7f87430
2014-06-12 14:46:44.621552 7f9d7a962700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:44.621670 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:44.621697 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.621738 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.621850 7f9d7a962700 15 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:44.621857 7f9d7a962700 10 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1 r = 0
2014-06-12 14:46:44.621862 7f9d7a962700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:44.621876 7f9d7a962700 15 _write 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 1998848~1548288
2014-06-12 14:46:44.621975 7f9d5a16f700  5 -- op tracker -- , seq: 747966, time: 2014-06-12 14:46:44.621297, event: header_read, op: osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.621985 7f9d5a16f700  5 -- op tracker -- , seq: 747966, time: 2014-06-12 14:46:44.621298, event: throttled, op: osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.621993 7f9d5a16f700  5 -- op tracker -- , seq: 747966, time: 2014-06-12 14:46:44.621949, event: all_read, op: osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.622001 7f9d5a16f700  5 -- op tracker -- , seq: 747966, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.622005 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:44.622014 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:44.622020 7f9d5a16f700 15 osd.2 173 enqueue_op 0x34017e00 prio 127 cost 516903 latency 0.000723 osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.622027 7f9d7a962700 10 lookup_strip_header done  cid 3.4e0_head oid 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3
2014-06-12 14:46:44.622058 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.622067 7f9d693e3700 10 osd.2 173 dequeue_op 0x34017e00 prio 127 cost 516903 latency 0.000769 osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2468 (0'0,173'2468] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2467 active]
2014-06-12 14:46:44.622089 7f9d693e3700  5 -- op tracker -- , seq: 747966, time: 2014-06-12 14:46:44.622085, event: reached_pg, op: osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.622098 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2468 (0'0,173'2468] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2467 active] handle_message: 0x34017e00
2014-06-12 14:46:44.622109 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2468 (0'0,173'2468] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2467 active] sub_op_modify trans 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 v 173'2469 (transaction) 174
2014-06-12 14:46:44.622123 7f9d693e3700  5 -- op tracker -- , seq: 747966, time: 2014-06-12 14:46:44.622120, event: started, op: osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.622137 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2468 (0'0,173'2468] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2467 active] append_log log((0'0,173'2468], crt=0'0) [173'2469 (173'2468) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143628 2014-06-12 08:43:18.124820]
2014-06-12 14:46:44.622161 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2469 (0'0,173'2469] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2467 active] add_log_entry 173'2469 (173'2468) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143628 2014-06-12 08:43:18.124820
2014-06-12 14:46:44.622187 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2469 (0'0,173'2469] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2467 active] append_log  adding 1 keys
2014-06-12 14:46:44.622206 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:44.622218 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2469, trimmed: 
2014-06-12 14:46:44.622238 7f9d693e3700  5 -- op tracker -- , seq: 747966, time: 2014-06-12 14:46:44.622233, event: started, op: osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.622246 7f9d693e3700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:44.622254 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.622255 7f9d693e3700  5 queue_op 0x92a4f00 seq 2302 osr(3.4f4 0x276db70) 518389 bytes   (queue has 2 ops and 2068970 bytes)
2014-06-12 14:46:44.622267 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (2 active)
2014-06-12 14:46:44.622263 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2469 (0'0,173'2469] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2467 active] do_sub_op osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.622274 7f9d7895e700  5 _do_op 0x92a4f00 seq 2302 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:46:44.622276 7f9d693e3700 10 osd.2 173 dequeue_op 0x34017e00 finish
2014-06-12 14:46:44.622278 7f9d7895e700 10 _do_transaction on 0x13ec15f0
2014-06-12 14:46:44.622279 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.622281 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.622284 7f9d7895e700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:44.622360 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:44.622383 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.627803 7f9d7a962700 10 _generic_write 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 1998848~1548288 = 0
2014-06-12 14:46:44.628073 7f9d7a962700 15 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3
2014-06-12 14:46:44.628103 7f9d7a962700 10 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:44.628114 7f9d7a962700 15 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3
2014-06-12 14:46:44.628125 7f9d7a962700 10 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:44.628583 7f9d7a962700 20 set_header setting 17854 cid 3.4e0_head oid 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:44.628789 7f9d7a962700 20 set_header setting 2236 cid meta oid 1fe07435/pglog_3.4e0/0//-1 parent seq 0
2014-06-12 14:46:44.628808 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.628819 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:44.636232 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.636378 7f9d7895e700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:44.636388 7f9d7895e700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:46:44.636397 7f9d7895e700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:44.636420 7f9d7895e700 15 _write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 1589248~516096
2014-06-12 14:46:44.636538 7f9d7895e700 10 lookup_strip_header done  cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:44.636723 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:44.636993 7f9d7a962700 10 _do_op 0x5a73320 seq 2438 r = 0, finisher 0x31d0d820 0
2014-06-12 14:46:44.637013 7f9d7a962700 10 _finish_op 0x5a73320 seq 2438 osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:44.637021 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5180900 (1 active)
2014-06-12 14:46:44.637031 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.637034 7f9d7715b700  5 -- op tracker -- , seq: 747965, time: 2014-06-12 14:46:44.637021, event: commit_sent, op: osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.637044 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2638 (0'0,173'2638] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2636 active] sub_op_modify_commit on op osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:44.637082 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:44.637147 7f9d7795c700  5 -- op tracker -- , seq: 747965, time: 2014-06-12 14:46:44.637140, event: sub_op_applied, op: osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.637158 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2638 (0'0,173'2638] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2637 active] sub_op_modify_applied on 0x7f87340 op osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.637175 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2638 (0'0,173'2638] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2637 active] op_applied on primary on version 173'2638
2014-06-12 14:46:44.637185 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2638 (0'0,173'2638] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2637 active] op_applied on replica on version 173'2638
2014-06-12 14:46:44.637213 7f9d7795c700  5 -- op tracker -- , seq: 747965, time: 2014-06-12 14:46:44.637208, event: done, op: osd_sub_op(client.4293.0:143962 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2638 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.638626 7f9d7895e700 10 _generic_write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 1589248~516096 = 0
2014-06-12 14:46:44.638689 7f9d7895e700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:44.638704 7f9d7895e700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:44.638713 7f9d7895e700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:44.638723 7f9d7895e700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:44.638880 7f9d7895e700 20 set_header setting 17388 cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 parent seq 0
2014-06-12 14:46:44.638944 7f9d7895e700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:46:44.638962 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.638974 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:44.641458 7f9d7895e700 10 _do_op 0x92a4f00 seq 2302 r = 0, finisher 0x4eb6420 0
2014-06-12 14:46:44.641480 7f9d7895e700 10 _finish_op 0x92a4f00 seq 2302 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:44.641499 7f9d7715b700  5 -- op tracker -- , seq: 747966, time: 2014-06-12 14:46:44.641488, event: commit_sent, op: osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.641503 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:46:44.641512 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.641506 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2469 (0'0,173'2469] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2467 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:44.641527 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:44.641589 7f9d7795c700  5 -- op tracker -- , seq: 747966, time: 2014-06-12 14:46:44.641582, event: sub_op_applied, op: osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.641602 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2469 (0'0,173'2469] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2468 active] sub_op_modify_applied on 0x13ec1500 op osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.641628 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2469 (0'0,173'2469] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2468 active] op_applied on primary on version 173'2469
2014-06-12 14:46:44.641640 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2469 (0'0,173'2469] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2468 active] op_applied on replica on version 173'2469
2014-06-12 14:46:44.641668 7f9d7795c700  5 -- op tracker -- , seq: 747966, time: 2014-06-12 14:46:44.641663, event: done, op: osd_sub_op(client.4294.0:143628 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2469 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.768389 7f9d56432700  5 -- op tracker -- , seq: 747967, time: 2014-06-12 14:46:44.767246, event: header_read, op: osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.768400 7f9d56432700  5 -- op tracker -- , seq: 747967, time: 2014-06-12 14:46:44.767249, event: throttled, op: osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.768408 7f9d56432700  5 -- op tracker -- , seq: 747967, time: 2014-06-12 14:46:44.768350, event: all_read, op: osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.768416 7f9d56432700  5 -- op tracker -- , seq: 747967, time: 0.000000, event: dispatched, op: osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.768427 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:44.768438 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:44.768446 7f9d56432700 15 osd.2 173 enqueue_op 0x34016800 prio 127 cost 516903 latency 0.001200 osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.768473 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.768481 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x34016800 prio 127 cost 516903 latency 0.001235 osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2641 (0'0,173'2641] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2640 active]
2014-06-12 14:46:44.768505 7f9d6b3e7700  5 -- op tracker -- , seq: 747967, time: 2014-06-12 14:46:44.768501, event: reached_pg, op: osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.768514 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2641 (0'0,173'2641] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2640 active] handle_message: 0x34016800
2014-06-12 14:46:44.768524 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2641 (0'0,173'2641] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2640 active] sub_op_modify trans e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 v 173'2642 (transaction) 174
2014-06-12 14:46:44.768540 7f9d6b3e7700  5 -- op tracker -- , seq: 747967, time: 2014-06-12 14:46:44.768536, event: started, op: osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.768555 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2641 (0'0,173'2641] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2640 active] append_log log((0'0,173'2641], crt=0'0) [173'2642 (173'2641) modify   e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 by client.4292.0:142426 2014-06-12 08:44:58.031099]
2014-06-12 14:46:44.768578 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2642 (0'0,173'2642] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2640 active] add_log_entry 173'2642 (173'2641) modify   e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 by client.4292.0:142426 2014-06-12 08:44:58.031099
2014-06-12 14:46:44.768598 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2642 (0'0,173'2642] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2640 active] append_log  adding 1 keys
2014-06-12 14:46:44.768628 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2642, trimmed: 
2014-06-12 14:46:44.768651 7f9d6b3e7700  5 -- op tracker -- , seq: 747967, time: 2014-06-12 14:46:44.768647, event: started, op: osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.768662 7f9d6b3e7700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:44.768665 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.768667 7f9d6b3e7700  5 queue_op 0x9299800 seq 2462 osr(3.1c7 0x276d170) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:44.768678 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:46:44.768684 7f9d7a161700  5 _do_op 0x9299800 seq 2462 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:46:44.768674 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2642 (0'0,173'2642] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2640 active] do_sub_op osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.768687 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x34016800 finish
2014-06-12 14:46:44.768688 7f9d7a161700 10 _do_transaction on 0x7f87eb0
2014-06-12 14:46:44.768690 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.768693 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.768693 7f9d7a161700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:44.768920 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:44.768942 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.768990 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.769012 7f9d7a161700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:44.769017 7f9d7a161700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:46:44.769022 7f9d7a161700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:44.769039 7f9d7a161700 15 _write 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 3248128~516096
2014-06-12 14:46:44.769377 7f9d7a161700 10 lookup_strip_header done  cid 3.1c7_head oid e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:44.769552 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:44.772078 7f9d7a161700 10 _generic_write 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 3248128~516096 = 0
2014-06-12 14:46:44.772140 7f9d7a161700 15 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:44.772163 7f9d7a161700 10 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:44.772171 7f9d7a161700 15 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:44.772181 7f9d7a161700 10 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:44.772402 7f9d7a161700 20 set_header setting 18233 cid 3.1c7_head oid e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 parent seq 0
2014-06-12 14:46:44.772489 7f9d7a161700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:46:44.772507 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.772515 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:44.772647 7f9d47c58700  5 -- op tracker -- , seq: 747968, time: 2014-06-12 14:46:44.772422, event: header_read, op: osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.772661 7f9d47c58700  5 -- op tracker -- , seq: 747968, time: 2014-06-12 14:46:44.772428, event: throttled, op: osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.772669 7f9d47c58700  5 -- op tracker -- , seq: 747968, time: 2014-06-12 14:46:44.772610, event: all_read, op: osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.772678 7f9d47c58700  5 -- op tracker -- , seq: 747968, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.772691 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:44.772737 7f9d47c58700 15 osd.2 173 enqueue_op 0xe20a600 prio 63 cost 69632 latency 0.000314 osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.772811 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.772819 7f9d683e1700 10 osd.2 173 dequeue_op 0xe20a600 prio 63 cost 69632 latency 0.000397 osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]
2014-06-12 14:46:44.772850 7f9d683e1700  5 -- op tracker -- , seq: 747968, time: 2014-06-12 14:46:44.772845, event: reached_pg, op: osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.772858 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.772872 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] handle_message: 0xe20a600
2014-06-12 14:46:44.772881 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_op osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.772905 7f9d683e1700 15 getattr 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 '_'
2014-06-12 14:46:44.773899 7f9d425f5700  5 -- op tracker -- , seq: 747969, time: 2014-06-12 14:46:44.768955, event: header_read, op: osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.773912 7f9d425f5700  5 -- op tracker -- , seq: 747969, time: 2014-06-12 14:46:44.768958, event: throttled, op: osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.773921 7f9d425f5700  5 -- op tracker -- , seq: 747969, time: 2014-06-12 14:46:44.773860, event: all_read, op: osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.773929 7f9d425f5700  5 -- op tracker -- , seq: 747969, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.773937 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:46:44.773965 7f9d425f5700 15 osd.2 173 enqueue_op 0x6d10800 prio 63 cost 516096 latency 0.005010 osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.774001 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:44.774009 7f9d65bdc700 10 osd.2 173 dequeue_op 0x6d10800 prio 63 cost 516096 latency 0.005054 osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]
2014-06-12 14:46:44.774038 7f9d65bdc700  5 -- op tracker -- , seq: 747969, time: 2014-06-12 14:46:44.774033, event: reached_pg, op: osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.774124 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.774140 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] handle_message: 0x6d10800
2014-06-12 14:46:44.774151 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.774177 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: found obc in cache: 0x22eacb00
2014-06-12 14:46:44.774187 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: 0x22eacb00 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 rwstate(write n=2 w=0) oi: fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3(173'2445 client.4314.0:145116 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2445) ssc: 0x93449a0 snapset: 0=[]:[]+head
2014-06-12 14:46:44.774203 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] find_object_context fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 @head oi=fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3(173'2445 client.4314.0:145116 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2445)
2014-06-12 14:46:44.774229 7f9d65bdc700  5 -- op tracker -- , seq: 747969, time: 2014-06-12 14:46:44.774223, event: started, op: osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.774236 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] execute_ctx 0x33b2f000
2014-06-12 14:46:44.774253 7f9d65bdc700  5 -- op tracker -- , seq: 747969, time: 2014-06-12 14:46:44.774248, event: started, op: osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.774260 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] ov 173'2445 av 173'2446 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.774273 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096]
2014-06-12 14:46:44.774284 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.774297 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  write 1232896~516096
2014-06-12 14:46:44.774312 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:46:44.774323 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 snapset=0x93449d8  snapc=0=[]
2014-06-12 14:46:44.774342 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.774352 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] finish_ctx fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 0x33b2f000 op modify  
2014-06-12 14:46:44.774366 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  set mtime to 2014-06-12 08:45:47.148355
2014-06-12 14:46:44.774388 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  final snapset 0=[]:[]+head in fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:44.774411 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.774421 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] new_repop rep_tid 236294 on osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.774444 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] issue_repop rep_tid 236294 o fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:44.774474 7f9d65bdc700  5 -- op tracker -- , seq: 747969, time: 2014-06-12 14:46:44.774465, event: waiting for subops from 26, op: osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.774500 7f9d65bdc700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.774527 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2445 (0'0,173'2445] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log log((0'0,173'2445], crt=0'0) [173'2446 (173'2445) modify   fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 by client.4314.0:145117 2014-06-12 08:45:47.148355]
2014-06-12 14:46:44.774552 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] add_log_entry 173'2446 (173'2445) modify   fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 by client.4314.0:145117 2014-06-12 08:45:47.148355
2014-06-12 14:46:44.774575 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.774610 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2446, trimmed: 
2014-06-12 14:46:44.774664 7f9d65bdc700  5 queue_transactions existing osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:44.774668 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.774670 7f9d65bdc700  5 queue_op 0x5948ea0 seq 2263 osr(3.251 0x276d390) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:44.774682 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fc20 (2 active)
2014-06-12 14:46:44.774688 7f9d7f16b700  5 _do_op 0x5948ea0 seq 2263 osr(3.251 0x276d390)/0x276d390 start
2014-06-12 14:46:44.774692 7f9d7f16b700 10 _do_transaction on 0x53535c0
2014-06-12 14:46:44.774678 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] eval_repop repgather(0x5352240 173'2446 rep_tid=236294 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.774697 7f9d65bdc700 10 osd.2 173 dequeue_op 0x6d10800 finish
2014-06-12 14:46:44.774698 7f9d7f16b700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:44.774701 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:44.774706 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.774809 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:44.774841 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.775250 7f9d425f5700  5 -- op tracker -- , seq: 747970, time: 2014-06-12 14:46:44.773999, event: header_read, op: osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.775267 7f9d425f5700  5 -- op tracker -- , seq: 747970, time: 2014-06-12 14:46:44.774001, event: throttled, op: osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.775277 7f9d425f5700  5 -- op tracker -- , seq: 747970, time: 2014-06-12 14:46:44.775210, event: all_read, op: osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.775286 7f9d425f5700  5 -- op tracker -- , seq: 747970, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.775294 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:46:44.775317 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.775322 7f9d425f5700 15 osd.2 173 enqueue_op 0x6d11e00 prio 63 cost 516096 latency 0.001323 osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.775343 7f9d7f16b700 15 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:44.775350 7f9d7f16b700 10 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1 r = 0
2014-06-12 14:46:44.775357 7f9d7f16b700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:44.775357 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:44.775378 7f9d7f16b700 15 _write 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 1232896~516096
2014-06-12 14:46:44.775362 7f9d62bd6700 10 osd.2 173 dequeue_op 0x6d11e00 prio 63 cost 516096 latency 0.001363 osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]
2014-06-12 14:46:44.775390 7f9d62bd6700  5 -- op tracker -- , seq: 747970, time: 2014-06-12 14:46:44.775384, event: reached_pg, op: osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.775400 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.775414 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] handle_message: 0x6d11e00
2014-06-12 14:46:44.775424 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.775448 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: found obc in cache: 0x22eacb00
2014-06-12 14:46:44.775460 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: 0x22eacb00 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 rwstate(write n=3 w=0) oi: fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3(173'2446 client.4314.0:145117 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2446) ssc: 0x93449a0 snapset: 0=[]:[]+head
2014-06-12 14:46:44.775475 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] find_object_context fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 @head oi=fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3(173'2446 client.4314.0:145117 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2446)
2014-06-12 14:46:44.775499 7f9d62bd6700  5 -- op tracker -- , seq: 747970, time: 2014-06-12 14:46:44.775494, event: started, op: osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.775506 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] execute_ctx 0xb571800
2014-06-12 14:46:44.775521 7f9d62bd6700  5 -- op tracker -- , seq: 747970, time: 2014-06-12 14:46:44.775517, event: started, op: osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.775527 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] ov 173'2446 av 173'2447 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.775540 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096]
2014-06-12 14:46:44.775550 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.775563 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  write 1748992~516096
2014-06-12 14:46:44.775581 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:46:44.775592 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 snapset=0x93449d8  snapc=0=[]
2014-06-12 14:46:44.775604 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.775613 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] finish_ctx fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 0xb571800 op modify  
2014-06-12 14:46:44.775625 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  set mtime to 2014-06-12 08:45:47.152368
2014-06-12 14:46:44.775647 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  final snapset 0=[]:[]+head in fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:44.775673 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.775683 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] new_repop rep_tid 236295 on osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.775697 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] issue_repop rep_tid 236295 o fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:44.775702 7f9d7a161700 10 _do_op 0x9299800 seq 2462 r = 0, finisher 0x73f8420 0
2014-06-12 14:46:44.775719 7f9d62bd6700  5 -- op tracker -- , seq: 747970, time: 2014-06-12 14:46:44.775715, event: waiting for subops from 26, op: osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.775732 7f9d7715b700  5 -- op tracker -- , seq: 747967, time: 2014-06-12 14:46:44.775725, event: commit_sent, op: osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.775738 7f9d62bd6700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.775740 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2642 (0'0,173'2642] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2640 active] sub_op_modify_commit on op osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:44.775772 7f9d7f16b700 10 lookup_strip_header done  cid 3.251_head oid fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:44.775760 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2446 (0'0,173'2446] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log log((0'0,173'2446], crt=0'0) [173'2447 (173'2446) modify   fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 by client.4314.0:145118 2014-06-12 08:45:47.152368]
2014-06-12 14:46:44.775775 7f9d7a161700 10 _finish_op 0x9299800 seq 2462 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:44.775781 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] add_log_entry 173'2447 (173'2446) modify   fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 by client.4314.0:145118 2014-06-12 08:45:47.152368
2014-06-12 14:46:44.775791 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (1 active)
2014-06-12 14:46:44.775804 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.775804 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.775832 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2447, trimmed: 
2014-06-12 14:46:44.775853 7f9d62bd6700  5 queue_transactions existing osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:44.775857 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.775858 7f9d62bd6700  5 queue_op 0x179f5f80 seq 2264 osr(3.251 0x276d390) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:44.775865 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] eval_repop repgather(0x5237c80 173'2447 rep_tid=236295 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.775881 7f9d62bd6700 10 osd.2 173 dequeue_op 0x6d11e00 finish
2014-06-12 14:46:44.775885 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:44.775892 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.775899 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fc20 (2 active)
2014-06-12 14:46:44.775944 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:44.776046 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.776077 7f9d7795c700  5 -- op tracker -- , seq: 747967, time: 2014-06-12 14:46:44.776072, event: sub_op_applied, op: osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.776085 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2642 (0'0,173'2642] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] sub_op_modify_applied on 0x7f87dc0 op osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.776104 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2642 (0'0,173'2642] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] op_applied on primary on version 173'2642
2014-06-12 14:46:44.776112 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2642 (0'0,173'2642] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] op_applied on replica on version 173'2642
2014-06-12 14:46:44.776140 7f9d7795c700  5 -- op tracker -- , seq: 747967, time: 2014-06-12 14:46:44.776136, event: done, op: osd_sub_op(client.4292.0:142426 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2642 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.777540 7f9d7f16b700 10 _generic_write 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 1232896~516096 = 0
2014-06-12 14:46:44.777579 7f9d7f16b700 15 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:44.777599 7f9d7f16b700 10 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 = 0
2014-06-12 14:46:44.777604 7f9d7f16b700 15 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:44.777634 7f9d7f16b700 10 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 = 0
2014-06-12 14:46:44.777861 7f9d7f16b700 20 set_header setting 17915 cid 3.251_head oid fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 parent seq 0
2014-06-12 14:46:44.777954 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.777970 7f9d7f16b700 20 set_header setting 2026 cid meta oid 1fe2661a/pglog_3.251/0//-1 parent seq 0
2014-06-12 14:46:44.777981 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:44.779403 7f9d47c58700  5 -- op tracker -- , seq: 747971, time: 2014-06-12 14:46:44.776621, event: header_read, op: osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.779417 7f9d47c58700  5 -- op tracker -- , seq: 747971, time: 2014-06-12 14:46:44.776625, event: throttled, op: osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.779427 7f9d47c58700  5 -- op tracker -- , seq: 747971, time: 2014-06-12 14:46:44.779361, event: all_read, op: osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.779436 7f9d47c58700  5 -- op tracker -- , seq: 747971, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.779444 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:44.779484 7f9d47c58700 15 osd.2 173 enqueue_op 0xe20a200 prio 63 cost 516096 latency 0.002862 osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.779515 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:44.781072 7f9d7f16b700 10 _do_op 0x5948ea0 seq 2263 r = 0, finisher 0x98f4080 0x38997480
2014-06-12 14:46:44.781091 7f9d7f16b700 10 _finish_op 0x5948ea0 seq 2263 osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:44.781100 7f9d7e169700  5 _do_op 0x179f5f80 seq 2264 osr(3.251 0x276d390)/0x276d390 start
2014-06-12 14:46:44.781105 7f9d7e169700 10 _do_transaction on 0x52366c0
2014-06-12 14:46:44.781114 7f9d7e169700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:44.781096 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_commit: 236294
2014-06-12 14:46:44.781128 7f9d7715b700  5 -- op tracker -- , seq: 747969, time: 2014-06-12 14:46:44.781119, event: op_commit, op: osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.781340 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:44.781367 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.781409 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.781431 7f9d7e169700 15 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:44.781435 7f9d7e169700 10 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1 r = 0
2014-06-12 14:46:44.781440 7f9d7e169700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:44.781455 7f9d7e169700 15 _write 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 1748992~516096
2014-06-12 14:46:44.781607 7f9d47c58700  5 -- op tracker -- , seq: 747972, time: 2014-06-12 14:46:44.780757, event: header_read, op: osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.781626 7f9d47c58700  5 -- op tracker -- , seq: 747972, time: 2014-06-12 14:46:44.780761, event: throttled, op: osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.781636 7f9d47c58700  5 -- op tracker -- , seq: 747972, time: 2014-06-12 14:46:44.781578, event: all_read, op: osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.781646 7f9d47c58700  5 -- op tracker -- , seq: 747972, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.781655 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:44.781683 7f9d47c58700 15 osd.2 173 enqueue_op 0xe20a500 prio 63 cost 516096 latency 0.000926 osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.781724 7f9d7e169700 10 lookup_strip_header done  cid 3.251_head oid fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:44.781732 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active)
2014-06-12 14:46:44.781925 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:44.783644 7f9d7e169700 10 _generic_write 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 1748992~516096 = 0
2014-06-12 14:46:44.783684 7f9d7e169700 15 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:44.783695 7f9d7e169700 10 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 = 0
2014-06-12 14:46:44.783699 7f9d7e169700 15 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:44.783704 7f9d7e169700 10 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 = 0
2014-06-12 14:46:44.783852 7f9d7e169700 20 set_header setting 17915 cid 3.251_head oid fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 parent seq 0
2014-06-12 14:46:44.783899 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.783910 7f9d7e169700 20 set_header setting 2026 cid meta oid 1fe2661a/pglog_3.251/0//-1 parent seq 0
2014-06-12 14:46:44.783917 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:44.784043 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fc20 (1 active)
2014-06-12 14:46:44.784050 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.784066 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied: 236294
2014-06-12 14:46:44.784098 7f9d7795c700  5 -- op tracker -- , seq: 747969, time: 2014-06-12 14:46:44.784088, event: op_applied, op: osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.784109 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied on primary on version 173'2446
2014-06-12 14:46:44.787222 7f9d7e169700 10 _do_op 0x179f5f80 seq 2264 r = 0, finisher 0x690f600 0x389963c0
2014-06-12 14:46:44.787305 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_commit: 236295
2014-06-12 14:46:44.787381 7f9d7715b700  5 -- op tracker -- , seq: 747970, time: 2014-06-12 14:46:44.787358, event: op_commit, op: osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.787405 7f9d7e169700 10 _finish_op 0x179f5f80 seq 2264 osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:44.787431 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied: 236295
2014-06-12 14:46:44.787456 7f9d7795c700  5 -- op tracker -- , seq: 747970, time: 2014-06-12 14:46:44.787448, event: op_applied, op: osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.787467 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied on primary on version 173'2447
2014-06-12 14:46:44.787504 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fc20 (0 active)
2014-06-12 14:46:44.787516 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.803762 7f9d56432700  5 -- op tracker -- , seq: 747973, time: 2014-06-12 14:46:44.802784, event: header_read, op: osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.803778 7f9d56432700  5 -- op tracker -- , seq: 747973, time: 2014-06-12 14:46:44.802785, event: throttled, op: osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.803787 7f9d56432700  5 -- op tracker -- , seq: 747973, time: 2014-06-12 14:46:44.803721, event: all_read, op: osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.803797 7f9d56432700  5 -- op tracker -- , seq: 747973, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.803803 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:44.803810 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:44.803817 7f9d56432700 15 osd.2 173 enqueue_op 0x34016400 prio 127 cost 516903 latency 0.001032 osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.803855 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (4 active)
2014-06-12 14:46:44.803867 7f9d66bde700 10 osd.2 173 dequeue_op 0x34016400 prio 127 cost 516903 latency 0.001083 osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.71f( v 173'2699 (0'0,173'2699] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active]
2014-06-12 14:46:44.803903 7f9d66bde700  5 -- op tracker -- , seq: 747973, time: 2014-06-12 14:46:44.803886, event: reached_pg, op: osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.803912 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2699 (0'0,173'2699] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active] handle_message: 0x34016400
2014-06-12 14:46:44.803925 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2699 (0'0,173'2699] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active] sub_op_modify trans c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 v 173'2700 (transaction) 174
2014-06-12 14:46:44.803942 7f9d66bde700  5 -- op tracker -- , seq: 747973, time: 2014-06-12 14:46:44.803937, event: started, op: osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.803956 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2699 (0'0,173'2699] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active] append_log log((0'0,173'2699], crt=0'0) [173'2700 (173'2699) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145287 2014-06-12 08:45:27.603415]
2014-06-12 14:46:44.803991 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2700 (0'0,173'2700] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active] add_log_entry 173'2700 (173'2699) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145287 2014-06-12 08:45:27.603415
2014-06-12 14:46:44.804012 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2700 (0'0,173'2700] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active] append_log  adding 1 keys
2014-06-12 14:46:44.804043 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2700, trimmed: 
2014-06-12 14:46:44.804087 7f9d66bde700  5 -- op tracker -- , seq: 747973, time: 2014-06-12 14:46:44.804081, event: started, op: osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.804096 7f9d66bde700  5 queue_transactions existing osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:44.804100 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.804101 7f9d66bde700  5 queue_op 0x5be9ce0 seq 2532 osr(3.71f 0x276de50) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:44.804111 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d0240 (1 active)
2014-06-12 14:46:44.804118 7f9d7815d700  5 _do_op 0x5be9ce0 seq 2532 osr(3.71f 0x276de50)/0x276de50 start
2014-06-12 14:46:44.804121 7f9d7815d700 10 _do_transaction on 0x20c42630
2014-06-12 14:46:44.804108 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.71f( v 173'2700 (0'0,173'2700] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active] do_sub_op osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.804123 7f9d66bde700 10 osd.2 173 dequeue_op 0x34016400 finish
2014-06-12 14:46:44.804125 7f9d7815d700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:44.804127 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (3 active)
2014-06-12 14:46:44.804129 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.804221 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:44.804233 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.804265 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.804275 7f9d7815d700 15 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:44.804278 7f9d7815d700 10 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1 r = 0
2014-06-12 14:46:44.804282 7f9d7815d700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:44.804290 7f9d7815d700 15 _write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 2830336~516096
2014-06-12 14:46:44.804392 7f9d7815d700 10 lookup_strip_header done  cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:44.804406 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:44.806038 7f9d7815d700 10 _generic_write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 2830336~516096 = 0
2014-06-12 14:46:44.806076 7f9d7815d700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:44.806087 7f9d7815d700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:44.806091 7f9d7815d700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:44.806097 7f9d7815d700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:44.806301 7f9d7815d700 20 set_header setting 17162 cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 parent seq 0
2014-06-12 14:46:44.806365 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.806380 7f9d7815d700 20 set_header setting 2330 cid meta oid 1fe0985b/pglog_3.71f/0//-1 parent seq 0
2014-06-12 14:46:44.806390 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:44.807355 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:44.807411 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:44.808787 7f9d7815d700 10 _do_op 0x5be9ce0 seq 2532 r = 0, finisher 0x6704e00 0
2014-06-12 14:46:44.808805 7f9d7815d700 10 _finish_op 0x5be9ce0 seq 2532 osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:44.808810 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d0240 (0 active)
2014-06-12 14:46:44.808813 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.809439 7f9d7795c700  5 -- op tracker -- , seq: 747973, time: 2014-06-12 14:46:44.809429, event: sub_op_applied, op: osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.809449 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2700 (0'0,173'2700] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active] sub_op_modify_applied on 0x20c42540 op osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.809476 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.809493 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2700 (0'0,173'2700] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active] op_applied on primary on version 173'2700
2014-06-12 14:46:44.809505 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2700 (0'0,173'2700] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active] op_applied on replica on version 173'2700
2014-06-12 14:46:44.809527 7f9d7715b700  5 -- op tracker -- , seq: 747973, time: 2014-06-12 14:46:44.809522, event: commit_sent, op: osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.809535 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2700 (0'0,173'2700] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2698 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:44.809551 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.809582 7f9d7715b700  5 -- op tracker -- , seq: 747973, time: 2014-06-12 14:46:44.809578, event: done, op: osd_sub_op(client.4308.0:145287 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.819550 7f9d683e1700 10 getattr 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 '_' = 0
2014-06-12 14:46:44.819595 7f9d683e1700 15 getattr 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 'snapset'
2014-06-12 14:46:44.819765 7f9d683e1700 10 getattr 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 'snapset' = 0
2014-06-12 14:46:44.819775 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] populate_obc_watchers eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.819794 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.819803 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] get_object_context: creating obc from disk: 0x2b9fa580
2014-06-12 14:46:44.819813 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] get_object_context: 0x2b9fa580 eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 rwstate(none n=0 w=0) oi: eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'776 client.4316.0:41683 wrlock_by=unknown.0.0:0 dirty s 4194304 uv776) ssc: 0x1e15e460 snapset: 0=[]:[]+head
2014-06-12 14:46:44.819829 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] find_object_context eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 @head oi=eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'776 client.4316.0:41683 wrlock_by=unknown.0.0:0 dirty s 4194304 uv776)
2014-06-12 14:46:44.819860 7f9d683e1700  5 -- op tracker -- , seq: 747968, time: 2014-06-12 14:46:44.819851, event: started, op: osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.819878 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] execute_ctx 0x1fc54400
2014-06-12 14:46:44.819895 7f9d683e1700  5 -- op tracker -- , seq: 747968, time: 2014-06-12 14:46:44.819890, event: started, op: osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.819901 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] ov 173'776 av 173'2695 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.819915 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_osd_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632]
2014-06-12 14:46:44.819924 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.819937 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_osd_op  write 0~69632
2014-06-12 14:46:44.819953 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:44.819963 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 snapset=0x1e15e498  snapc=0=[]
2014-06-12 14:46:44.819974 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.819984 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] finish_ctx eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 0x1fc54400 op modify  
2014-06-12 14:46:44.819996 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]  set mtime to 2014-06-12 08:45:47.152719
2014-06-12 14:46:44.820021 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]  final snapset 0=[]:[]+head in eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.820048 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.820058 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] new_repop rep_tid 236296 on osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.820074 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] issue_repop rep_tid 236296 o eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.820100 7f9d683e1700  5 -- op tracker -- , seq: 747968, time: 2014-06-12 14:46:44.820096, event: waiting for subops from 23, op: osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.820131 7f9d683e1700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:44.820171 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2694 (0'0,173'2694] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] append_log log((0'0,173'2694], crt=0'0) [173'2695 (173'776) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147225 2014-06-12 08:45:47.152719]
2014-06-12 14:46:44.820203 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] add_log_entry 173'2695 (173'776) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147225 2014-06-12 08:45:47.152719
2014-06-12 14:46:44.820226 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.820267 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2695, trimmed: 
2014-06-12 14:46:44.820303 7f9d683e1700  5 queue_transactions existing osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.820311 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.820314 7f9d683e1700  5 queue_op 0x205f1b00 seq 2462 osr(3.7cc 0x49b43c0) 71925 bytes   (queue has 1 ops and 71925 bytes)
2014-06-12 14:46:44.820341 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f7200 (1 active)
2014-06-12 14:46:44.820330 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] eval_repop repgather(0x51ada40 173'2695 rep_tid=236296 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.820354 7f9d82171700  5 _do_op 0x205f1b00 seq 2462 osr(3.7cc 0x49b43c0)/0x49b43c0 start
2014-06-12 14:46:44.820358 7f9d683e1700 10 osd.2 173 dequeue_op 0xe20a600 finish
2014-06-12 14:46:44.820363 7f9d82171700 10 _do_transaction on 0x12ae20c0
2014-06-12 14:46:44.820373 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active)
2014-06-12 14:46:44.820373 7f9d82171700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.820379 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.820381 7f9d64bda700 10 osd.2 173 dequeue_op 0xe20a200 prio 63 cost 516096 latency 0.043759 osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]
2014-06-12 14:46:44.820405 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19417 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,5,0,0,0,0,1,0,1,4])
2014-06-12 14:46:44.820411 7f9d64bda700  5 -- op tracker -- , seq: 747971, time: 2014-06-12 14:46:44.820406, event: reached_pg, op: osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.820415 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19417 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,5,0,0,0,0,1,0,1,4])
2014-06-12 14:46:44.820421 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.820436 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] handle_message: 0xe20a200
2014-06-12 14:46:44.820446 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_op osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.820469 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] get_object_context: found obc in cache: 0x2b9fa580
2014-06-12 14:46:44.820477 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.820479 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] get_object_context: 0x2b9fa580 eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 rwstate(write n=1 w=0) oi: eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2695 client.4316.0:147225 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2695) ssc: 0x1e15e460 snapset: 0=[]:[]+head
2014-06-12 14:46:44.820493 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] find_object_context eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 @head oi=eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2695 client.4316.0:147225 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2695)
2014-06-12 14:46:44.820506 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.820516 7f9d64bda700  5 -- op tracker -- , seq: 747971, time: 2014-06-12 14:46:44.820510, event: started, op: osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.820522 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] execute_ctx 0x589be00
2014-06-12 14:46:44.820539 7f9d64bda700  5 -- op tracker -- , seq: 747971, time: 2014-06-12 14:46:44.820534, event: started, op: osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.820554 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.820545 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] ov 173'2695 av 173'2696 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.820559 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_osd_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096]
2014-06-12 14:46:44.820569 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.820582 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_osd_op  write 69632~516096
2014-06-12 14:46:44.820595 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:44.820605 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 snapset=0x1e15e498  snapc=0=[]
2014-06-12 14:46:44.820616 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.820626 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] finish_ctx eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 0x589be00 op modify  
2014-06-12 14:46:44.820637 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]  set mtime to 2014-06-12 08:45:47.156306
2014-06-12 14:46:44.820657 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]  final snapset 0=[]:[]+head in eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.820673 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.820692 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] new_repop rep_tid 236297 on osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.820709 7f9d82171700 15 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.820706 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] issue_repop rep_tid 236297 o eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.820716 7f9d82171700 10 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1 r = 0
2014-06-12 14:46:44.820730 7f9d82171700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.820727 7f9d64bda700  5 -- op tracker -- , seq: 747971, time: 2014-06-12 14:46:44.820722, event: waiting for subops from 23, op: osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.820745 7f9d64bda700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:44.820750 7f9d82171700 15 _write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 0~69632
2014-06-12 14:46:44.820764 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2695 (0'0,173'2695] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] append_log log((0'0,173'2695], crt=0'0) [173'2696 (173'2695) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147226 2014-06-12 08:45:47.156306]
2014-06-12 14:46:44.820789 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] add_log_entry 173'2696 (173'2695) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147226 2014-06-12 08:45:47.156306
2014-06-12 14:46:44.820809 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.820837 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2696, trimmed: 
2014-06-12 14:46:44.820869 7f9d64bda700  5 queue_transactions existing osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.820875 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.820876 7f9d64bda700  5 queue_op 0x7c899e0 seq 2463 osr(3.7cc 0x49b43c0) 518389 bytes   (queue has 2 ops and 590314 bytes)
2014-06-12 14:46:44.820891 7f9d82171700 10 lookup_strip_header done  cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.820883 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] eval_repop repgather(0x101938c0 173'2696 rep_tid=236297 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.820900 7f9d64bda700 10 osd.2 173 dequeue_op 0xe20a200 finish
2014-06-12 14:46:44.820907 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:44.820910 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.820933 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:44.821064 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:44.821161 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:44.821241 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:44.821275 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.821296 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:44.821272 7f9d6abe6700 10 osd.2 173 dequeue_op 0xe20a500 prio 63 cost 516096 latency 0.040514 osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]
2014-06-12 14:46:44.821309 7f9d6abe6700  5 -- op tracker -- , seq: 747972, time: 2014-06-12 14:46:44.821302, event: reached_pg, op: osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.821317 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:44.821320 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.821343 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:44.821338 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] handle_message: 0xe20a500
2014-06-12 14:46:44.821346 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_op osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.821360 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:44.821366 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] get_object_context: found obc in cache: 0x2b9fa580
2014-06-12 14:46:44.821377 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:44.821375 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] get_object_context: 0x2b9fa580 eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 rwstate(write n=2 w=0) oi: eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2696 client.4316.0:147226 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2696) ssc: 0x1e15e460 snapset: 0=[]:[]+head
2014-06-12 14:46:44.821395 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:44.821388 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] find_object_context eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 @head oi=eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2696 client.4316.0:147226 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2696)
2014-06-12 14:46:44.821412 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:44.821409 7f9d6abe6700  5 -- op tracker -- , seq: 747972, time: 2014-06-12 14:46:44.821404, event: started, op: osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.821416 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] execute_ctx 0x2963a600
2014-06-12 14:46:44.821432 7f9d6abe6700  5 -- op tracker -- , seq: 747972, time: 2014-06-12 14:46:44.821427, event: started, op: osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.821440 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] ov 173'2696 av 173'2697 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.821453 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_osd_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096]
2014-06-12 14:46:44.821463 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.821471 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:44.821472 7f9d82171700 10 _generic_write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 0~69632 = 0
2014-06-12 14:46:44.821476 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_osd_op  write 585728~516096
2014-06-12 14:46:44.821491 7f9d82171700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.821489 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:44.821503 7f9d82171700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.821499 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 snapset=0x1e15e498  snapc=0=[]
2014-06-12 14:46:44.821508 7f9d82171700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.821520 7f9d82171700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.821510 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.821536 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] finish_ctx eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 0x2963a600 op modify  
2014-06-12 14:46:44.821550 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]  set mtime to 2014-06-12 08:45:47.159982
2014-06-12 14:46:44.821569 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]  final snapset 0=[]:[]+head in eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.821585 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.821600 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] new_repop rep_tid 236298 on osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.821624 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] issue_repop rep_tid 236298 o eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.821647 7f9d6abe6700  5 -- op tracker -- , seq: 747972, time: 2014-06-12 14:46:44.821642, event: waiting for subops from 23, op: osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.821649 7f9d82171700 20 set_header setting 17435 cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 parent seq 0
2014-06-12 14:46:44.821669 7f9d6abe6700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:44.821676 7f9d82171700 20 set_header setting 2154 cid meta oid 1fe706d6/pglog_3.7cc/0//-1 parent seq 0
2014-06-12 14:46:44.821689 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.821688 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2696 (0'0,173'2696] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] append_log log((0'0,173'2696], crt=0'0) [173'2697 (173'2696) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147227 2014-06-12 08:45:47.159982]
2014-06-12 14:46:44.821709 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:44.821710 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] add_log_entry 173'2697 (173'2696) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147227 2014-06-12 08:45:47.159982
2014-06-12 14:46:44.821729 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.821746 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:44.821756 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2697, trimmed: 
2014-06-12 14:46:44.821774 7f9d6abe6700  5 queue_transactions existing osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.821777 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.821778 7f9d6abe6700  5 queue_op 0x62c7260 seq 2464 osr(3.7cc 0x49b43c0) 518389 bytes   (queue has 3 ops and 1108703 bytes)
2014-06-12 14:46:44.821785 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] eval_repop repgather(0x6e538c0 173'2697 rep_tid=236298 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.821804 7f9d6abe6700 10 osd.2 173 dequeue_op 0xe20a500 finish
2014-06-12 14:46:44.821810 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.821814 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:44.821816 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.821824 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f7200 (2 active)
2014-06-12 14:46:44.821839 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f7200 (3 active)
2014-06-12 14:46:44.822091 7f9d82171700 10 _do_op 0x205f1b00 seq 2462 r = 0, finisher 0x2e1cfb20 0x4c4c000
2014-06-12 14:46:44.822102 7f9d82171700 10 _finish_op 0x205f1b00 seq 2462 osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.822110 7f9d7e96a700  5 _do_op 0x7c899e0 seq 2463 osr(3.7cc 0x49b43c0)/0x49b43c0 start
2014-06-12 14:46:44.822114 7f9d7e96a700 10 _do_transaction on 0x52d0d80
2014-06-12 14:46:44.822113 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f7200 (2 active)
2014-06-12 14:46:44.822116 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.822120 7f9d7e96a700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.822220 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.822233 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.822223 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] op_applied: 236296
2014-06-12 14:46:44.822244 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.822252 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:44.822249 7f9d7795c700  5 -- op tracker -- , seq: 747968, time: 2014-06-12 14:46:44.822241, event: op_applied, op: osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.822258 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] op_applied on primary on version 173'2695
2014-06-12 14:46:44.822286 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.822370 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:44.822382 7f9d7e96a700 15 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.822390 7f9d7e96a700 10 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1 r = 0
2014-06-12 14:46:44.822397 7f9d7e96a700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.822415 7f9d7e96a700 15 _write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 69632~516096
2014-06-12 14:46:44.822471 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:44.822543 7f9d7e96a700 10 lookup_strip_header done  cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.822554 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] op_commit: 236296
2014-06-12 14:46:44.822577 7f9d7715b700  5 -- op tracker -- , seq: 747968, time: 2014-06-12 14:46:44.822571, event: op_commit, op: osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.822727 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:44.824870 7f9d7e96a700 10 _generic_write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 69632~516096 = 0
2014-06-12 14:46:44.824931 7f9d7e96a700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.824951 7f9d7e96a700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.824957 7f9d7e96a700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.824965 7f9d7e96a700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.825185 7f9d7e96a700 20 set_header setting 17435 cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 parent seq 0
2014-06-12 14:46:44.825253 7f9d7e96a700 20 set_header setting 2154 cid meta oid 1fe706d6/pglog_3.7cc/0//-1 parent seq 0
2014-06-12 14:46:44.825268 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.825277 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:44.825912 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:44.826166 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:44.827945 7f9d7e96a700 10 _do_op 0x7c899e0 seq 2463 r = 0, finisher 0x5b76240 0x2869fb00
2014-06-12 14:46:44.827966 7f9d7e96a700 10 _finish_op 0x7c899e0 seq 2463 osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.827977 7f9d7d167700  5 _do_op 0x62c7260 seq 2464 osr(3.7cc 0x49b43c0)/0x49b43c0 start
2014-06-12 14:46:44.827982 7f9d7d167700 10 _do_transaction on 0x6e53d40
2014-06-12 14:46:44.827991 7f9d7d167700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.827975 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] op_commit: 236297
2014-06-12 14:46:44.828011 7f9d7715b700  5 -- op tracker -- , seq: 747971, time: 2014-06-12 14:46:44.828000, event: op_commit, op: osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.828107 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.828133 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.828178 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.828201 7f9d7d167700 15 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.828206 7f9d7d167700 10 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1 r = 0
2014-06-12 14:46:44.828212 7f9d7d167700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.828341 7f9d7d167700 15 _write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 585728~516096
2014-06-12 14:46:44.828461 7f9d7d167700 10 lookup_strip_header done  cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.828596 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:44.828598 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:44.828764 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] op_applied: 236297
2014-06-12 14:46:44.828787 7f9d7795c700  5 -- op tracker -- , seq: 747971, time: 2014-06-12 14:46:44.828781, event: op_applied, op: osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.828796 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] op_applied on primary on version 173'2696
2014-06-12 14:46:44.828819 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f7200 (1 active)
2014-06-12 14:46:44.828830 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.828937 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:44.830567 7f9d7d167700 10 _generic_write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 585728~516096 = 0
2014-06-12 14:46:44.830632 7f9d7d167700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.830647 7f9d7d167700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.830654 7f9d7d167700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.830662 7f9d7d167700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.830852 7f9d7d167700 20 set_header setting 17435 cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 parent seq 0
2014-06-12 14:46:44.830944 7f9d7d167700 20 set_header setting 2154 cid meta oid 1fe706d6/pglog_3.7cc/0//-1 parent seq 0
2014-06-12 14:46:44.830965 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.830974 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:44.833703 7f9d7d167700 10 _do_op 0x62c7260 seq 2464 r = 0, finisher 0x57bd980 0x3786e880
2014-06-12 14:46:44.833725 7f9d7d167700 10 _finish_op 0x62c7260 seq 2464 osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.833736 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f7200 (0 active)
2014-06-12 14:46:44.833740 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.833742 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] op_applied: 236298
2014-06-12 14:46:44.833771 7f9d7795c700  5 -- op tracker -- , seq: 747972, time: 2014-06-12 14:46:44.833762, event: op_applied, op: osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.833781 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] op_applied on primary on version 173'2697
2014-06-12 14:46:44.834191 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] op_commit: 236298
2014-06-12 14:46:44.834214 7f9d7715b700  5 -- op tracker -- , seq: 747972, time: 2014-06-12 14:46:44.834207, event: op_commit, op: osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.857827 7f9d57543700  5 -- op tracker -- , seq: 747974, time: 2014-06-12 14:46:44.857762, event: header_read, op: osd_sub_op_reply(client.4316.0:147225 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.857842 7f9d57543700  5 -- op tracker -- , seq: 747974, time: 2014-06-12 14:46:44.857764, event: throttled, op: osd_sub_op_reply(client.4316.0:147225 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.857865 7f9d57543700  5 -- op tracker -- , seq: 747974, time: 2014-06-12 14:46:44.857779, event: all_read, op: osd_sub_op_reply(client.4316.0:147225 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.857871 7f9d57543700  5 -- op tracker -- , seq: 747974, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147225 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.857876 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147225 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:44.857881 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:44.857889 7f9d57543700 15 osd.2 173 enqueue_op 0x329eb100 prio 196 cost 0 latency 0.000126 osd_sub_op_reply(client.4316.0:147225 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:44.857918 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.857926 7f9d69be4700 10 osd.2 173 dequeue_op 0x329eb100 prio 196 cost 0 latency 0.000164 osd_sub_op_reply(client.4316.0:147225 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 pg pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]
2014-06-12 14:46:44.857945 7f9d69be4700  5 -- op tracker -- , seq: 747974, time: 2014-06-12 14:46:44.857943, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147225 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.857952 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] handle_message: 0x329eb100
2014-06-12 14:46:44.857963 7f9d69be4700  5 -- op tracker -- , seq: 747974, time: 2014-06-12 14:46:44.857961, event: started, op: osd_sub_op_reply(client.4316.0:147225 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.857967 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] sub_op_modify_reply: tid 236296 op  ack_type 4 from 23
2014-06-12 14:46:44.857983 7f9d69be4700  5 -- op tracker -- , seq: 747968, time: 2014-06-12 14:46:44.857976, event: sub_op_commit_rec, op: osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.857988 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] repop_all_applied: repop tid 236296 all applied 
2014-06-12 14:46:44.857995 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] eval_repop repgather(0x51ada40 173'2695 rep_tid=236296 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.858006 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean]  sending ack on repgather(0x51ada40 173'2695 rep_tid=236296 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x30c41b80
2014-06-12 14:46:44.858028 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2694 crt=0'0 lcod 173'2693 mlcod 173'2693 active+clean] repop_all_committed: repop tid 236296 all committed 
2014-06-12 14:46:44.858038 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2693 active+clean] eval_repop repgather(0x51ada40 173'2695 rep_tid=236296 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.858054 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2693 active+clean] log_op_stats osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 inb 69911 outb 0 rlat 0.085604 lat 0.085630
2014-06-12 14:46:44.858070 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2693 active+clean] publish_stats_to_osd 173:3513
2014-06-12 14:46:44.858080 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2693 active+clean]  sending commit on repgather(0x51ada40 173'2695 rep_tid=236296 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x3206b680
2014-06-12 14:46:44.858103 7f9d69be4700  5 -- op tracker -- , seq: 747968, time: 2014-06-12 14:46:44.858097, event: commit_sent, op: osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.858111 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2693 active+clean]  requeue_ops 
2014-06-12 14:46:44.858122 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2694 active+clean]  removing repgather(0x51ada40 173'2695 rep_tid=236296 committed?=1 applied?=1 op=osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.858133 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2694 active+clean]    q front is repgather(0x51ada40 173'2695 rep_tid=236296 committed?=1 applied?=1 op=osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.858143 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2694 active+clean] remove_repop repgather(0x51ada40 173'2695 rep_tid=236296 committed?=1 applied?=1 op=osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.858151 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2694 active+clean]  requeue_ops 
2014-06-12 14:46:44.858167 7f9d69be4700  5 -- op tracker -- , seq: 747968, time: 2014-06-12 14:46:44.858164, event: done, op: osd_op(client.4316.0:147225 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~69632] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.858289 7f9d69be4700 10 osd.2 173 dequeue_op 0x329eb100 finish
2014-06-12 14:46:44.858297 7f9d69be4700  5 -- op tracker -- , seq: 747974, time: 2014-06-12 14:46:44.858294, event: done, op: osd_sub_op_reply(client.4316.0:147225 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.858307 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.858310 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.865456 7f9d57543700  5 -- op tracker -- , seq: 747975, time: 2014-06-12 14:46:44.865409, event: header_read, op: osd_sub_op_reply(client.4316.0:147226 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.865466 7f9d57543700  5 -- op tracker -- , seq: 747975, time: 2014-06-12 14:46:44.865411, event: throttled, op: osd_sub_op_reply(client.4316.0:147226 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.865480 7f9d57543700  5 -- op tracker -- , seq: 747975, time: 2014-06-12 14:46:44.865426, event: all_read, op: osd_sub_op_reply(client.4316.0:147226 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.865484 7f9d57543700  5 -- op tracker -- , seq: 747975, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147226 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.865488 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147226 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:44.865492 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:44.865500 7f9d57543700 15 osd.2 173 enqueue_op 0x329eb600 prio 196 cost 0 latency 0.000091 osd_sub_op_reply(client.4316.0:147226 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:44.865522 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.865533 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x329eb600 prio 196 cost 0 latency 0.000123 osd_sub_op_reply(client.4316.0:147226 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 pg pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2694 active+clean]
2014-06-12 14:46:44.865557 7f9d6a3e5700  5 -- op tracker -- , seq: 747975, time: 2014-06-12 14:46:44.865553, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147226 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.865567 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2694 active+clean] handle_message: 0x329eb600
2014-06-12 14:46:44.865585 7f9d6a3e5700  5 -- op tracker -- , seq: 747975, time: 2014-06-12 14:46:44.865582, event: started, op: osd_sub_op_reply(client.4316.0:147226 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.865593 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2694 active+clean] sub_op_modify_reply: tid 236297 op  ack_type 4 from 23
2014-06-12 14:46:44.865623 7f9d6a3e5700  5 -- op tracker -- , seq: 747971, time: 2014-06-12 14:46:44.865607, event: sub_op_commit_rec, op: osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.865634 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2694 active+clean] repop_all_applied: repop tid 236297 all applied 
2014-06-12 14:46:44.865645 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2694 active+clean] eval_repop repgather(0x101938c0 173'2696 rep_tid=236297 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.865662 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2694 active+clean]  sending ack on repgather(0x101938c0 173'2696 rep_tid=236297 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x54b0500
2014-06-12 14:46:44.865696 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2695 crt=0'0 lcod 173'2694 mlcod 173'2694 active+clean] repop_all_committed: repop tid 236297 all committed 
2014-06-12 14:46:44.865708 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2694 active+clean] eval_repop repgather(0x101938c0 173'2696 rep_tid=236297 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.865726 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2694 active+clean] log_op_stats osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.089073 lat 0.089104
2014-06-12 14:46:44.865760 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2694 active+clean] publish_stats_to_osd 173:3514
2014-06-12 14:46:44.865776 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2694 active+clean]  sending commit on repgather(0x101938c0 173'2696 rep_tid=236297 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x1a58d680
2014-06-12 14:46:44.865811 7f9d6a3e5700  5 -- op tracker -- , seq: 747971, time: 2014-06-12 14:46:44.865806, event: commit_sent, op: osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.865820 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2694 active+clean]  requeue_ops 
2014-06-12 14:46:44.865832 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean]  removing repgather(0x101938c0 173'2696 rep_tid=236297 committed?=1 applied?=1 op=osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.865856 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean]    q front is repgather(0x101938c0 173'2696 rep_tid=236297 committed?=1 applied?=1 op=osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.865878 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean] remove_repop repgather(0x101938c0 173'2696 rep_tid=236297 committed?=1 applied?=1 op=osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.865893 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean]  requeue_ops 
2014-06-12 14:46:44.865911 7f9d6a3e5700  5 -- op tracker -- , seq: 747971, time: 2014-06-12 14:46:44.865906, event: done, op: osd_op(client.4316.0:147226 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 69632~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.865932 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x329eb600 finish
2014-06-12 14:46:44.865941 7f9d6a3e5700  5 -- op tracker -- , seq: 747975, time: 2014-06-12 14:46:44.865937, event: done, op: osd_sub_op_reply(client.4316.0:147226 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.865951 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.865956 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.869162 7f9d57543700  5 -- op tracker -- , seq: 747976, time: 2014-06-12 14:46:44.869089, event: header_read, op: osd_sub_op_reply(client.4316.0:147227 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.869175 7f9d57543700  5 -- op tracker -- , seq: 747976, time: 2014-06-12 14:46:44.869091, event: throttled, op: osd_sub_op_reply(client.4316.0:147227 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.869185 7f9d57543700  5 -- op tracker -- , seq: 747976, time: 2014-06-12 14:46:44.869105, event: all_read, op: osd_sub_op_reply(client.4316.0:147227 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.869190 7f9d57543700  5 -- op tracker -- , seq: 747976, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147227 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.869197 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147227 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:44.869204 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:44.869227 7f9d57543700 15 osd.2 173 enqueue_op 0x329ea700 prio 196 cost 0 latency 0.000137 osd_sub_op_reply(client.4316.0:147227 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:44.869263 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.869252 7f9d47c58700  5 -- op tracker -- , seq: 747977, time: 2014-06-12 14:46:44.860661, event: header_read, op: osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.869274 7f9d47c58700  5 -- op tracker -- , seq: 747977, time: 2014-06-12 14:46:44.860672, event: throttled, op: osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.869280 7f9d47c58700  5 -- op tracker -- , seq: 747977, time: 2014-06-12 14:46:44.869217, event: all_read, op: osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.869272 7f9d653db700 10 osd.2 173 dequeue_op 0x329ea700 prio 196 cost 0 latency 0.000183 osd_sub_op_reply(client.4316.0:147227 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 pg pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean]
2014-06-12 14:46:44.869292 7f9d47c58700  5 -- op tracker -- , seq: 747977, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.869300 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:44.869301 7f9d653db700  5 -- op tracker -- , seq: 747976, time: 2014-06-12 14:46:44.869291, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147227 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.869317 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean] handle_message: 0x329ea700
2014-06-12 14:46:44.869329 7f9d653db700  5 -- op tracker -- , seq: 747976, time: 2014-06-12 14:46:44.869327, event: started, op: osd_sub_op_reply(client.4316.0:147227 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.869328 7f9d47c58700 15 osd.2 173 enqueue_op 0x24c9c500 prio 63 cost 516096 latency 0.008667 osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.869333 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean] sub_op_modify_reply: tid 236298 op  ack_type 4 from 23
2014-06-12 14:46:44.869348 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:44.869347 7f9d653db700  5 -- op tracker -- , seq: 747972, time: 2014-06-12 14:46:44.869342, event: sub_op_commit_rec, op: osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.869360 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean] repop_all_applied: repop tid 236298 all applied 
2014-06-12 14:46:44.869369 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean] eval_repop repgather(0x6e538c0 173'2697 rep_tid=236298 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.869382 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean]  sending ack on repgather(0x6e538c0 173'2697 rep_tid=236298 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x353e9b80
2014-06-12 14:46:44.869498 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2696 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean] repop_all_committed: repop tid 236298 all committed 
2014-06-12 14:46:44.869507 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2695 active+clean] eval_repop repgather(0x6e538c0 173'2697 rep_tid=236298 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.869517 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2695 active+clean] log_op_stats osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.088740 lat 0.088760
2014-06-12 14:46:44.869528 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2695 active+clean] publish_stats_to_osd 173:3515
2014-06-12 14:46:44.869537 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2695 active+clean]  sending commit on repgather(0x6e538c0 173'2697 rep_tid=236298 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x959b180
2014-06-12 14:46:44.869556 7f9d653db700  5 -- op tracker -- , seq: 747972, time: 2014-06-12 14:46:44.869553, event: commit_sent, op: osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.869562 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2695 active+clean]  requeue_ops 
2014-06-12 14:46:44.869569 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]  removing repgather(0x6e538c0 173'2697 rep_tid=236298 committed?=1 applied?=1 op=osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.869578 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]    q front is repgather(0x6e538c0 173'2697 rep_tid=236298 committed?=1 applied?=1 op=osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.869586 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] remove_repop repgather(0x6e538c0 173'2697 rep_tid=236298 committed?=1 applied?=1 op=osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.869599 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]  requeue_ops 
2014-06-12 14:46:44.869629 7f9d653db700  5 -- op tracker -- , seq: 747972, time: 2014-06-12 14:46:44.869624, event: done, op: osd_op(client.4316.0:147227 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 585728~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.869650 7f9d653db700 10 osd.2 173 dequeue_op 0x329ea700 finish
2014-06-12 14:46:44.869663 7f9d653db700  5 -- op tracker -- , seq: 747976, time: 2014-06-12 14:46:44.869661, event: done, op: osd_sub_op_reply(client.4316.0:147227 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.869671 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:44.869673 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.869665 7f9d63bd8700 10 osd.2 173 dequeue_op 0x24c9c500 prio 63 cost 516096 latency 0.009003 osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]
2014-06-12 14:46:44.869695 7f9d63bd8700  5 -- op tracker -- , seq: 747977, time: 2014-06-12 14:46:44.869689, event: reached_pg, op: osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.869704 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.869720 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] handle_message: 0x24c9c500
2014-06-12 14:46:44.869732 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_op osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.869757 7f9d63bd8700 15 getattr 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 '_'
2014-06-12 14:46:44.869816 7f9d47c58700  5 -- op tracker -- , seq: 747978, time: 2014-06-12 14:46:44.869346, event: header_read, op: osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.869825 7f9d47c58700  5 -- op tracker -- , seq: 747978, time: 2014-06-12 14:46:44.869347, event: throttled, op: osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.869831 7f9d47c58700  5 -- op tracker -- , seq: 747978, time: 2014-06-12 14:46:44.869795, event: all_read, op: osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.869837 7f9d47c58700  5 -- op tracker -- , seq: 747978, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.869840 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:44.869851 7f9d47c58700 15 osd.2 173 enqueue_op 0x24c9d200 prio 63 cost 516096 latency 0.000505 osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.869868 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:44.869946 7f9d63bd8700 10 getattr 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 '_' = 0
2014-06-12 14:46:44.869963 7f9d63bd8700 15 getattr 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 'snapset'
2014-06-12 14:46:44.870022 7f9d63bd8700 10 getattr 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 'snapset' = 0
2014-06-12 14:46:44.870032 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] populate_obc_watchers eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.870046 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.870057 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] get_object_context: creating obc from disk: 0x10092b00
2014-06-12 14:46:44.870068 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] get_object_context: 0x10092b00 eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 rwstate(none n=0 w=0) oi: eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2697 client.4316.0:147227 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2697) ssc: 0x1bc1ea0 snapset: 0=[]:[]+head
2014-06-12 14:46:44.870085 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] find_object_context eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 @head oi=eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2697 client.4316.0:147227 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2697)
2014-06-12 14:46:44.870113 7f9d63bd8700  5 -- op tracker -- , seq: 747977, time: 2014-06-12 14:46:44.870107, event: started, op: osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.870125 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] execute_ctx 0xfd18000
2014-06-12 14:46:44.870144 7f9d63bd8700  5 -- op tracker -- , seq: 747977, time: 2014-06-12 14:46:44.870138, event: started, op: osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.870152 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] ov 173'2697 av 173'2698 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.870169 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_osd_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096]
2014-06-12 14:46:44.870181 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.870194 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_osd_op  write 1101824~516096
2014-06-12 14:46:44.870212 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:44.870226 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 snapset=0x1bc1ed8  snapc=0=[]
2014-06-12 14:46:44.870239 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.870252 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] finish_ctx eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 0xfd18000 op modify  
2014-06-12 14:46:44.870266 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]  set mtime to 2014-06-12 08:45:47.240114
2014-06-12 14:46:44.870287 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]  final snapset 0=[]:[]+head in eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.870307 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.870320 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] new_repop rep_tid 236299 on osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.870338 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] issue_repop rep_tid 236299 o eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.870364 7f9d63bd8700  5 -- op tracker -- , seq: 747977, time: 2014-06-12 14:46:44.870359, event: waiting for subops from 23, op: osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.870390 7f9d63bd8700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:44.870412 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2697 (0'0,173'2697] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] append_log log((0'0,173'2697], crt=0'0) [173'2698 (173'2697) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147228 2014-06-12 08:45:47.240114]
2014-06-12 14:46:44.870440 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] add_log_entry 173'2698 (173'2697) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147228 2014-06-12 08:45:47.240114
2014-06-12 14:46:44.870462 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.870493 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2698, trimmed: 
2014-06-12 14:46:44.870494 7f9d47c58700  5 -- op tracker -- , seq: 747979, time: 2014-06-12 14:46:44.870461, event: header_read, op: osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.870503 7f9d47c58700  5 -- op tracker -- , seq: 747979, time: 2014-06-12 14:46:44.870463, event: throttled, op: osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.870509 7f9d47c58700  5 -- op tracker -- , seq: 747979, time: 2014-06-12 14:46:44.870476, event: all_read, op: osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.870514 7f9d47c58700  5 -- op tracker -- , seq: 747979, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.870515 7f9d63bd8700  5 queue_transactions existing osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.870518 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.870518 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:44.870520 7f9d63bd8700  5 queue_op 0x283b9ce0 seq 2465 osr(3.7cc 0x49b43c0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:44.870531 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f7200 (1 active)
2014-06-12 14:46:44.870532 7f9d47c58700 15 osd.2 173 enqueue_op 0x24c9cf00 prio 63 cost 4096 latency 0.000071 osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.870538 7f9d8116f700  5 _do_op 0x283b9ce0 seq 2465 osr(3.7cc 0x49b43c0)/0x49b43c0 start
2014-06-12 14:46:44.870527 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] eval_repop repgather(0x514c840 173'2698 rep_tid=236299 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.870542 7f9d8116f700 10 _do_transaction on 0x15ace840
2014-06-12 14:46:44.870543 7f9d63bd8700 10 osd.2 173 dequeue_op 0x24c9c500 finish
2014-06-12 14:46:44.870548 7f9d8116f700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.870559 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active)
2014-06-12 14:46:44.870556 7f9d68be2700 10 osd.2 173 dequeue_op 0x24c9d200 prio 63 cost 516096 latency 0.001210 osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]
2014-06-12 14:46:44.870568 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active)
2014-06-12 14:46:44.870571 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.870572 7f9d68be2700  5 -- op tracker -- , seq: 747978, time: 2014-06-12 14:46:44.870568, event: reached_pg, op: osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.870583 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.870592 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] handle_message: 0x24c9d200
2014-06-12 14:46:44.870597 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_op osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.870611 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] get_object_context: found obc in cache: 0x10092b00
2014-06-12 14:46:44.870617 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] get_object_context: 0x10092b00 eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 rwstate(write n=1 w=0) oi: eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2698 client.4316.0:147228 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2698) ssc: 0x1bc1ea0 snapset: 0=[]:[]+head
2014-06-12 14:46:44.870625 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] find_object_context eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 @head oi=eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2698 client.4316.0:147228 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2698)
2014-06-12 14:46:44.870634 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.870640 7f9d68be2700  5 -- op tracker -- , seq: 747978, time: 2014-06-12 14:46:44.870636, event: started, op: osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.870644 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] execute_ctx 0x580ca00
2014-06-12 14:46:44.870652 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.870654 7f9d68be2700  5 -- op tracker -- , seq: 747978, time: 2014-06-12 14:46:44.870651, event: started, op: osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.870662 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] ov 173'2698 av 173'2699 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.870671 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_osd_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096]
2014-06-12 14:46:44.870677 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.870685 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_osd_op  write 1617920~516096
2014-06-12 14:46:44.870691 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.870693 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:44.870699 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 snapset=0x1bc1ed8  snapc=0=[]
2014-06-12 14:46:44.870706 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.870713 7f9d8116f700 15 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.870712 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] finish_ctx eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 0x580ca00 op modify  
2014-06-12 14:46:44.870719 7f9d8116f700 10 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1 r = 0
2014-06-12 14:46:44.870724 7f9d8116f700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.870719 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]  set mtime to 2014-06-12 08:45:47.247672
2014-06-12 14:46:44.870732 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]  final snapset 0=[]:[]+head in eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.870743 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.870749 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] new_repop rep_tid 236300 on osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.870759 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] issue_repop rep_tid 236300 o eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.870773 7f9d68be2700  5 -- op tracker -- , seq: 747978, time: 2014-06-12 14:46:44.870769, event: waiting for subops from 23, op: osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.870786 7f9d68be2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:44.870796 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2698 (0'0,173'2698] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] append_log log((0'0,173'2698], crt=0'0) [173'2699 (173'2698) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147229 2014-06-12 08:45:47.247672]
2014-06-12 14:46:44.870815 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] add_log_entry 173'2699 (173'2698) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147229 2014-06-12 08:45:47.247672
2014-06-12 14:46:44.870829 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.870851 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2699, trimmed: 
2014-06-12 14:46:44.870865 7f9d68be2700  5 queue_transactions existing osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.870868 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.870869 7f9d68be2700  5 queue_op 0x1b44dbc0 seq 2466 osr(3.7cc 0x49b43c0) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:44.870876 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f7200 (2 active)
2014-06-12 14:46:44.870873 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] eval_repop repgather(0x1696c300 173'2699 rep_tid=236300 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.870885 7f9d68be2700 10 osd.2 173 dequeue_op 0x24c9d200 finish
2014-06-12 14:46:44.870902 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:44.870904 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.870905 7f9d623d5700 10 osd.2 173 dequeue_op 0x24c9cf00 prio 63 cost 4096 latency 0.000444 osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]
2014-06-12 14:46:44.870927 7f9d623d5700  5 -- op tracker -- , seq: 747979, time: 2014-06-12 14:46:44.870924, event: reached_pg, op: osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.870932 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.870941 7f9d8116f700 15 _write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 1101824~516096
2014-06-12 14:46:44.870941 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] handle_message: 0x24c9cf00
2014-06-12 14:46:44.870946 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_op osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.870960 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] get_object_context: found obc in cache: 0x10092b00
2014-06-12 14:46:44.870966 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] get_object_context: 0x10092b00 eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 rwstate(write n=2 w=0) oi: eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2699 client.4316.0:147229 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2699) ssc: 0x1bc1ea0 snapset: 0=[]:[]+head
2014-06-12 14:46:44.870975 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] find_object_context eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 @head oi=eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2699 client.4316.0:147229 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2699)
2014-06-12 14:46:44.870988 7f9d623d5700  5 -- op tracker -- , seq: 747979, time: 2014-06-12 14:46:44.870985, event: started, op: osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.870992 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] execute_ctx 0x55a3200
2014-06-12 14:46:44.871002 7f9d623d5700  5 -- op tracker -- , seq: 747979, time: 2014-06-12 14:46:44.870999, event: started, op: osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.871006 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] ov 173'2699 av 173'2700 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.871013 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_osd_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096]
2014-06-12 14:46:44.871020 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.871027 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_osd_op  write 2134016~4096
2014-06-12 14:46:44.871035 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:44.871041 7f9d8116f700 10 lookup_strip_header done  cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.871041 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 snapset=0x1bc1ed8  snapc=0=[]
2014-06-12 14:46:44.871047 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.871053 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] finish_ctx eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 0x55a3200 op modify  
2014-06-12 14:46:44.871060 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]  set mtime to 2014-06-12 08:45:47.250976
2014-06-12 14:46:44.871072 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]  final snapset 0=[]:[]+head in eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.871081 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.871087 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] new_repop rep_tid 236301 on osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.871096 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] issue_repop rep_tid 236301 o eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.871109 7f9d623d5700  5 -- op tracker -- , seq: 747979, time: 2014-06-12 14:46:44.871106, event: waiting for subops from 23, op: osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.871119 7f9d623d5700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:44.871133 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2699 (0'0,173'2699] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] append_log log((0'0,173'2699], crt=0'0) [173'2700 (173'2699) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147230 2014-06-12 08:45:47.250976]
2014-06-12 14:46:44.871151 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] add_log_entry 173'2700 (173'2699) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147230 2014-06-12 08:45:47.250976
2014-06-12 14:46:44.871166 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.871183 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2700, trimmed: 
2014-06-12 14:46:44.871196 7f9d623d5700  5 queue_transactions existing osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.871198 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.871199 7f9d623d5700  5 queue_op 0x6b403c0 seq 2467 osr(3.7cc 0x49b43c0) 6389 bytes   (queue has 3 ops and 1043167 bytes)
2014-06-12 14:46:44.871204 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:44.871207 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f7200 (3 active)
2014-06-12 14:46:44.871204 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] eval_repop repgather(0xd5e3680 173'2700 rep_tid=236301 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.871220 7f9d623d5700 10 osd.2 173 dequeue_op 0x24c9cf00 finish
2014-06-12 14:46:44.871225 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.871228 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.872508 7f9d8116f700 10 _generic_write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 1101824~516096 = 0
2014-06-12 14:46:44.872542 7f9d8116f700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.872553 7f9d8116f700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.872558 7f9d8116f700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.872564 7f9d8116f700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.872703 7f9d8116f700 20 set_header setting 17435 cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 parent seq 0
2014-06-12 14:46:44.872756 7f9d8116f700 20 set_header setting 2154 cid meta oid 1fe706d6/pglog_3.7cc/0//-1 parent seq 0
2014-06-12 14:46:44.872767 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.872775 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:44.875277 7f9d8116f700 10 _do_op 0x283b9ce0 seq 2465 r = 0, finisher 0x630d3a0 0x253ba300
2014-06-12 14:46:44.875297 7f9d8116f700 10 _finish_op 0x283b9ce0 seq 2465 osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.875307 7f9d79960700  5 _do_op 0x1b44dbc0 seq 2466 osr(3.7cc 0x49b43c0)/0x49b43c0 start
2014-06-12 14:46:44.875309 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f7200 (2 active)
2014-06-12 14:46:44.875312 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.875313 7f9d79960700 10 _do_transaction on 0xba85080
2014-06-12 14:46:44.875302 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] op_commit: 236299
2014-06-12 14:46:44.875323 7f9d79960700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.875328 7f9d7715b700  5 -- op tracker -- , seq: 747977, time: 2014-06-12 14:46:44.875320, event: op_commit, op: osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.875339 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] op_applied: 236299
2014-06-12 14:46:44.875357 7f9d7795c700  5 -- op tracker -- , seq: 747977, time: 2014-06-12 14:46:44.875352, event: op_applied, op: osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.875363 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] op_applied on primary on version 173'2698
2014-06-12 14:46:44.875432 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.875459 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.875500 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.875584 7f9d79960700 15 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.875592 7f9d79960700 10 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1 r = 0
2014-06-12 14:46:44.875600 7f9d79960700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.875619 7f9d79960700 15 _write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 1617920~516096
2014-06-12 14:46:44.875719 7f9d79960700 10 lookup_strip_header done  cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.875831 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:44.875867 7f9d57543700  5 -- op tracker -- , seq: 747980, time: 2014-06-12 14:46:44.875828, event: header_read, op: osd_sub_op_reply(client.4316.0:147228 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.875878 7f9d57543700  5 -- op tracker -- , seq: 747980, time: 2014-06-12 14:46:44.875829, event: throttled, op: osd_sub_op_reply(client.4316.0:147228 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.875885 7f9d57543700  5 -- op tracker -- , seq: 747980, time: 2014-06-12 14:46:44.875844, event: all_read, op: osd_sub_op_reply(client.4316.0:147228 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.875900 7f9d57543700  5 -- op tracker -- , seq: 747980, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147228 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.875904 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147228 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:44.875911 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:44.875917 7f9d57543700 15 osd.2 173 enqueue_op 0x329ebb00 prio 196 cost 0 latency 0.000089 osd_sub_op_reply(client.4316.0:147228 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:44.875942 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.875951 7f9d643d9700 10 osd.2 173 dequeue_op 0x329ebb00 prio 196 cost 0 latency 0.000123 osd_sub_op_reply(client.4316.0:147228 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 pg pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]
2014-06-12 14:46:44.875972 7f9d643d9700  5 -- op tracker -- , seq: 747980, time: 2014-06-12 14:46:44.875969, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147228 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.875983 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] handle_message: 0x329ebb00
2014-06-12 14:46:44.875997 7f9d643d9700  5 -- op tracker -- , seq: 747980, time: 2014-06-12 14:46:44.875995, event: started, op: osd_sub_op_reply(client.4316.0:147228 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.876001 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] sub_op_modify_reply: tid 236299 op  ack_type 4 from 23
2014-06-12 14:46:44.876017 7f9d643d9700  5 -- op tracker -- , seq: 747977, time: 2014-06-12 14:46:44.876011, event: sub_op_commit_rec, op: osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.876026 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] repop_all_applied: repop tid 236299 all applied 
2014-06-12 14:46:44.876038 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] eval_repop repgather(0x514c840 173'2698 rep_tid=236299 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.876056 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean]  sending ack on repgather(0x514c840 173'2698 rep_tid=236299 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x22b52500
2014-06-12 14:46:44.876087 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2697 crt=0'0 lcod 173'2696 mlcod 173'2696 active+clean] repop_all_committed: repop tid 236299 all committed 
2014-06-12 14:46:44.876099 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2696 active+clean] eval_repop repgather(0x514c840 173'2698 rep_tid=236299 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.876116 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2696 active+clean] log_op_stats osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.015425 lat 0.015454
2014-06-12 14:46:44.876129 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2696 active+clean] publish_stats_to_osd 173:3516
2014-06-12 14:46:44.876138 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2696 active+clean]  sending commit on repgather(0x514c840 173'2698 rep_tid=236299 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x10097680
2014-06-12 14:46:44.876159 7f9d643d9700  5 -- op tracker -- , seq: 747977, time: 2014-06-12 14:46:44.876155, event: commit_sent, op: osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.876165 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2696 active+clean]  requeue_ops 
2014-06-12 14:46:44.876172 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]  removing repgather(0x514c840 173'2698 rep_tid=236299 committed?=1 applied?=1 op=osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.876181 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]    q front is repgather(0x514c840 173'2698 rep_tid=236299 committed?=1 applied?=1 op=osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.876189 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] remove_repop repgather(0x514c840 173'2698 rep_tid=236299 committed?=1 applied?=1 op=osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.876197 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]  requeue_ops 
2014-06-12 14:46:44.876209 7f9d643d9700  5 -- op tracker -- , seq: 747977, time: 2014-06-12 14:46:44.876206, event: done, op: osd_op(client.4316.0:147228 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.876221 7f9d643d9700 10 osd.2 173 dequeue_op 0x329ebb00 finish
2014-06-12 14:46:44.876225 7f9d643d9700  5 -- op tracker -- , seq: 747980, time: 2014-06-12 14:46:44.876223, event: done, op: osd_sub_op_reply(client.4316.0:147228 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.876231 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.876233 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.877132 7f9d79960700 10 _generic_write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 1617920~516096 = 0
2014-06-12 14:46:44.877167 7f9d79960700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.877188 7f9d79960700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.877196 7f9d79960700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.877205 7f9d79960700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.877334 7f9d79960700 20 set_header setting 17435 cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 parent seq 0
2014-06-12 14:46:44.877388 7f9d79960700 20 set_header setting 2154 cid meta oid 1fe706d6/pglog_3.7cc/0//-1 parent seq 0
2014-06-12 14:46:44.877404 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.877416 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:44.879023 7f9d79960700 10 _do_op 0x1b44dbc0 seq 2466 r = 0, finisher 0x28dd4fa0 0x381e7ac0
2014-06-12 14:46:44.879037 7f9d79960700 10 _finish_op 0x1b44dbc0 seq 2466 osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.879048 7f9d8096e700  5 _do_op 0x6b403c0 seq 2467 osr(3.7cc 0x49b43c0)/0x49b43c0 start
2014-06-12 14:46:44.879041 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] op_commit: 236300
2014-06-12 14:46:44.879048 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f7200 (1 active)
2014-06-12 14:46:44.879055 7f9d8096e700 10 _do_transaction on 0x32b69e00
2014-06-12 14:46:44.879056 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.879061 7f9d8096e700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.879062 7f9d7715b700  5 -- op tracker -- , seq: 747978, time: 2014-06-12 14:46:44.879056, event: op_commit, op: osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.879073 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] op_applied: 236300
2014-06-12 14:46:44.879093 7f9d7795c700  5 -- op tracker -- , seq: 747978, time: 2014-06-12 14:46:44.879087, event: op_applied, op: osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.879101 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] op_applied on primary on version 173'2699
2014-06-12 14:46:44.879160 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.879180 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.879213 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.879387 7f9d8096e700 15 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.879397 7f9d8096e700 10 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1 r = 0
2014-06-12 14:46:44.879405 7f9d8096e700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.879426 7f9d8096e700 15 _write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 2134016~4096
2014-06-12 14:46:44.879535 7f9d8096e700 10 lookup_strip_header done  cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.879545 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:44.879737 7f9d8096e700 10 _generic_write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 2134016~4096 = 0
2014-06-12 14:46:44.879747 7f9d8096e700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.879756 7f9d8096e700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.879761 7f9d8096e700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.879768 7f9d8096e700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.879886 7f9d8096e700 20 set_header setting 17435 cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 parent seq 0
2014-06-12 14:46:44.879905 7f9d8096e700 20 set_header setting 2154 cid meta oid 1fe706d6/pglog_3.7cc/0//-1 parent seq 0
2014-06-12 14:46:44.879917 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.879924 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:44.879975 7f9d47c58700  5 -- op tracker -- , seq: 747981, time: 2014-06-12 14:46:44.878631, event: header_read, op: osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.879988 7f9d47c58700  5 -- op tracker -- , seq: 747981, time: 2014-06-12 14:46:44.878633, event: throttled, op: osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.880000 7f9d47c58700  5 -- op tracker -- , seq: 747981, time: 2014-06-12 14:46:44.879949, event: all_read, op: osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.880009 7f9d47c58700  5 -- op tracker -- , seq: 747981, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.880016 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:44.880049 7f9d47c58700 15 osd.2 173 enqueue_op 0x24c9d800 prio 63 cost 516096 latency 0.001418 osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.880074 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.880083 7f9d633d7700 10 osd.2 173 dequeue_op 0x24c9d800 prio 63 cost 516096 latency 0.001452 osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]
2014-06-12 14:46:44.880101 7f9d633d7700  5 -- op tracker -- , seq: 747981, time: 2014-06-12 14:46:44.880098, event: reached_pg, op: osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.880108 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.880118 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] handle_message: 0x24c9d800
2014-06-12 14:46:44.880125 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] do_op osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.880144 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] get_object_context: found obc in cache: 0x10092b00
2014-06-12 14:46:44.880152 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] get_object_context: 0x10092b00 eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 rwstate(write n=2 w=0) oi: eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2700 client.4316.0:147230 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2700) ssc: 0x1bc1ea0 snapset: 0=[]:[]+head
2014-06-12 14:46:44.880162 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] find_object_context eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 @head oi=eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2700 client.4316.0:147230 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2700)
2014-06-12 14:46:44.880181 7f9d633d7700  5 -- op tracker -- , seq: 747981, time: 2014-06-12 14:46:44.880177, event: started, op: osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.880186 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] execute_ctx 0x5ca3000
2014-06-12 14:46:44.880197 7f9d633d7700  5 -- op tracker -- , seq: 747981, time: 2014-06-12 14:46:44.880194, event: started, op: osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.880201 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] do_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] ov 173'2700 av 173'2701 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.880210 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] do_osd_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096]
2014-06-12 14:46:44.880217 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.880225 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] do_osd_op  write 2138112~516096
2014-06-12 14:46:44.880235 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:44.880242 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 snapset=0x1bc1ed8  snapc=0=[]
2014-06-12 14:46:44.880249 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.880255 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] finish_ctx eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 0x5ca3000 op modify  
2014-06-12 14:46:44.880263 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]  set mtime to 2014-06-12 08:45:47.258200
2014-06-12 14:46:44.880281 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]  final snapset 0=[]:[]+head in eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.880295 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.880303 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] new_repop rep_tid 236302 on osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.880315 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] issue_repop rep_tid 236302 o eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.880333 7f9d633d7700  5 -- op tracker -- , seq: 747981, time: 2014-06-12 14:46:44.880329, event: waiting for subops from 23, op: osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.880348 7f9d633d7700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:44.880366 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2700 (0'0,173'2700] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] append_log log((0'0,173'2700], crt=0'0) [173'2701 (173'2700) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147231 2014-06-12 08:45:47.258200]
2014-06-12 14:46:44.880381 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] add_log_entry 173'2701 (173'2700) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147231 2014-06-12 08:45:47.258200
2014-06-12 14:46:44.880397 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.880417 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2701, trimmed: 
2014-06-12 14:46:44.880430 7f9d633d7700  5 queue_transactions existing osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.880432 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.880433 7f9d633d7700  5 queue_op 0x803f4a0 seq 2468 osr(3.7cc 0x49b43c0) 518389 bytes   (queue has 2 ops and 524778 bytes)
2014-06-12 14:46:44.880441 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] eval_repop repgather(0x2a37acc0 173'2701 rep_tid=236302 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.880453 7f9d633d7700 10 osd.2 173 dequeue_op 0x24c9d800 finish
2014-06-12 14:46:44.880455 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.880457 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.880671 7f9d57543700  5 -- op tracker -- , seq: 747982, time: 2014-06-12 14:46:44.880501, event: header_read, op: osd_sub_op_reply(client.4316.0:147229 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.880681 7f9d57543700  5 -- op tracker -- , seq: 747982, time: 2014-06-12 14:46:44.880502, event: throttled, op: osd_sub_op_reply(client.4316.0:147229 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.880692 7f9d57543700  5 -- op tracker -- , seq: 747982, time: 2014-06-12 14:46:44.880509, event: all_read, op: osd_sub_op_reply(client.4316.0:147229 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.880700 7f9d57543700  5 -- op tracker -- , seq: 747982, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147229 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.880704 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147229 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:44.880709 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:44.880714 7f9d57543700 15 osd.2 173 enqueue_op 0x329ead00 prio 196 cost 0 latency 0.000213 osd_sub_op_reply(client.4316.0:147229 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:44.880810 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.880819 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329ead00 prio 196 cost 0 latency 0.000318 osd_sub_op_reply(client.4316.0:147229 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 pg pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]
2014-06-12 14:46:44.880838 7f9d61bd4700  5 -- op tracker -- , seq: 747982, time: 2014-06-12 14:46:44.880835, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147229 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.880846 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f7200 (2 active)
2014-06-12 14:46:44.880855 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] handle_message: 0x329ead00
2014-06-12 14:46:44.880869 7f9d61bd4700  5 -- op tracker -- , seq: 747982, time: 2014-06-12 14:46:44.880866, event: started, op: osd_sub_op_reply(client.4316.0:147229 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.880875 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] sub_op_modify_reply: tid 236300 op  ack_type 4 from 23
2014-06-12 14:46:44.880890 7f9d61bd4700  5 -- op tracker -- , seq: 747978, time: 2014-06-12 14:46:44.880886, event: sub_op_commit_rec, op: osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.880895 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] repop_all_applied: repop tid 236300 all applied 
2014-06-12 14:46:44.880902 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] eval_repop repgather(0x1696c300 173'2699 rep_tid=236300 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.880912 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean]  sending ack on repgather(0x1696c300 173'2699 rep_tid=236300 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x1ba79680
2014-06-12 14:46:44.880931 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2698 crt=0'0 lcod 173'2697 mlcod 173'2697 active+clean] repop_all_committed: repop tid 236300 all committed 
2014-06-12 14:46:44.880937 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2697 active+clean] eval_repop repgather(0x1696c300 173'2699 rep_tid=236300 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.880946 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2697 active+clean] log_op_stats osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.011584 lat 0.011600
2014-06-12 14:46:44.880957 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2697 active+clean] publish_stats_to_osd 173:3517
2014-06-12 14:46:44.880964 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2697 active+clean]  sending commit on repgather(0x1696c300 173'2699 rep_tid=236300 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x26e57180
2014-06-12 14:46:44.880987 7f9d61bd4700  5 -- op tracker -- , seq: 747978, time: 2014-06-12 14:46:44.880980, event: commit_sent, op: osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.881008 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2697 active+clean]  requeue_ops 
2014-06-12 14:46:44.881017 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean]  removing repgather(0x1696c300 173'2699 rep_tid=236300 committed?=1 applied?=1 op=osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.881027 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean]    q front is repgather(0x1696c300 173'2699 rep_tid=236300 committed?=1 applied?=1 op=osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.881037 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] remove_repop repgather(0x1696c300 173'2699 rep_tid=236300 committed?=1 applied?=1 op=osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.881045 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean]  requeue_ops 
2014-06-12 14:46:44.881056 7f9d61bd4700  5 -- op tracker -- , seq: 747978, time: 2014-06-12 14:46:44.881053, event: done, op: osd_op(client.4316.0:147229 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 1617920~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.881071 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329ead00 finish
2014-06-12 14:46:44.881076 7f9d61bd4700  5 -- op tracker -- , seq: 747982, time: 2014-06-12 14:46:44.881074, event: done, op: osd_sub_op_reply(client.4316.0:147229 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.881082 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.881084 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.881181 7f9d57543700  5 -- op tracker -- , seq: 747983, time: 2014-06-12 14:46:44.881135, event: header_read, op: osd_sub_op_reply(client.4316.0:147230 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.881191 7f9d57543700  5 -- op tracker -- , seq: 747983, time: 2014-06-12 14:46:44.881136, event: throttled, op: osd_sub_op_reply(client.4316.0:147230 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.881201 7f9d57543700  5 -- op tracker -- , seq: 747983, time: 2014-06-12 14:46:44.881144, event: all_read, op: osd_sub_op_reply(client.4316.0:147230 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.881209 7f9d57543700  5 -- op tracker -- , seq: 747983, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147230 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.881213 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147230 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:44.881217 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:44.881224 7f9d57543700 15 osd.2 173 enqueue_op 0x876ee00 prio 196 cost 0 latency 0.000088 osd_sub_op_reply(client.4316.0:147230 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:44.881228 7f9d8096e700 10 _do_op 0x6b403c0 seq 2467 r = 0, finisher 0x7b37200 0x4537100
2014-06-12 14:46:44.881236 7f9d8096e700 10 _finish_op 0x6b403c0 seq 2467 osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.881241 7f9d8016d700  5 _do_op 0x803f4a0 seq 2468 osr(3.7cc 0x49b43c0)/0x49b43c0 start
2014-06-12 14:46:44.881244 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f7200 (1 active)
2014-06-12 14:46:44.881246 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.881246 7f9d8016d700 10 _do_transaction on 0x1771dec0
2014-06-12 14:46:44.881252 7f9d8016d700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.881244 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] op_commit: 236301
2014-06-12 14:46:44.881263 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.881266 7f9d7715b700  5 -- op tracker -- , seq: 747979, time: 2014-06-12 14:46:44.881260, event: op_commit, op: osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.881279 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] op_applied: 236301
2014-06-12 14:46:44.881299 7f9d7795c700  5 -- op tracker -- , seq: 747979, time: 2014-06-12 14:46:44.881294, event: op_applied, op: osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.881307 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] op_applied on primary on version 173'2700
2014-06-12 14:46:44.881349 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.881341 7f9d673df700 10 osd.2 173 dequeue_op 0x876ee00 prio 196 cost 0 latency 0.000205 osd_sub_op_reply(client.4316.0:147230 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 pg pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean]
2014-06-12 14:46:44.881358 7f9d673df700  5 -- op tracker -- , seq: 747983, time: 2014-06-12 14:46:44.881355, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147230 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.881371 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.881368 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] handle_message: 0x876ee00
2014-06-12 14:46:44.881383 7f9d673df700  5 -- op tracker -- , seq: 747983, time: 2014-06-12 14:46:44.881379, event: started, op: osd_sub_op_reply(client.4316.0:147230 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.881391 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] sub_op_modify_reply: tid 236301 op  ack_type 4 from 23
2014-06-12 14:46:44.881408 7f9d673df700  5 -- op tracker -- , seq: 747979, time: 2014-06-12 14:46:44.881403, event: sub_op_commit_rec, op: osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.881415 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.881416 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] repop_all_applied: repop tid 236301 all applied 
2014-06-12 14:46:44.881426 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] eval_repop repgather(0xd5e3680 173'2700 rep_tid=236301 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.881442 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean]  sending ack on repgather(0xd5e3680 173'2700 rep_tid=236301 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x17432000
2014-06-12 14:46:44.881469 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2699 crt=0'0 lcod 173'2698 mlcod 173'2698 active+clean] repop_all_committed: repop tid 236301 all committed 
2014-06-12 14:46:44.881479 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2698 active+clean] eval_repop repgather(0xd5e3680 173'2700 rep_tid=236301 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.881491 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2698 active+clean] log_op_stats osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 inb 4375 outb 0 rlat 0.011007 lat 0.011029
2014-06-12 14:46:44.881504 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2698 active+clean] publish_stats_to_osd 173:3518
2014-06-12 14:46:44.881514 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2698 active+clean]  sending commit on repgather(0xd5e3680 173'2700 rep_tid=236301 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x5ffaa00
2014-06-12 14:46:44.881538 7f9d8016d700 15 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.881545 7f9d8016d700 10 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1 r = 0
2014-06-12 14:46:44.881551 7f9d8016d700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.881553 7f9d673df700  5 -- op tracker -- , seq: 747979, time: 2014-06-12 14:46:44.881537, event: commit_sent, op: osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.881560 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2698 active+clean]  requeue_ops 
2014-06-12 14:46:44.881570 7f9d8016d700 15 _write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 2138112~516096
2014-06-12 14:46:44.881570 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]  removing repgather(0xd5e3680 173'2700 rep_tid=236301 committed?=1 applied?=1 op=osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.881584 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]    q front is repgather(0xd5e3680 173'2700 rep_tid=236301 committed?=1 applied?=1 op=osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.881598 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] remove_repop repgather(0xd5e3680 173'2700 rep_tid=236301 committed?=1 applied?=1 op=osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.881610 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]  requeue_ops 
2014-06-12 14:46:44.881644 7f9d673df700  5 -- op tracker -- , seq: 747979, time: 2014-06-12 14:46:44.881639, event: done, op: osd_op(client.4316.0:147230 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~4096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.881664 7f9d673df700 10 osd.2 173 dequeue_op 0x876ee00 finish
2014-06-12 14:46:44.881672 7f9d673df700  5 -- op tracker -- , seq: 747983, time: 2014-06-12 14:46:44.881668, event: done, op: osd_sub_op_reply(client.4316.0:147230 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.881682 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.881684 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.881711 7f9d8016d700 10 lookup_strip_header done  cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.881862 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:44.883865 7f9d8016d700 10 _generic_write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 2138112~516096 = 0
2014-06-12 14:46:44.883928 7f9d8016d700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.883939 7f9d8016d700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.883943 7f9d8016d700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.883948 7f9d8016d700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.884157 7f9d8016d700 20 set_header setting 17435 cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 parent seq 0
2014-06-12 14:46:44.884247 7f9d8016d700 20 set_header setting 2154 cid meta oid 1fe706d6/pglog_3.7cc/0//-1 parent seq 0
2014-06-12 14:46:44.884260 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.884273 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:44.884743 7f9d47c58700  5 -- op tracker -- , seq: 747984, time: 2014-06-12 14:46:44.883316, event: header_read, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.884756 7f9d47c58700  5 -- op tracker -- , seq: 747984, time: 2014-06-12 14:46:44.883320, event: throttled, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.884766 7f9d47c58700  5 -- op tracker -- , seq: 747984, time: 2014-06-12 14:46:44.884708, event: all_read, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.884775 7f9d47c58700  5 -- op tracker -- , seq: 747984, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.884783 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:44.884811 7f9d47c58700 15 osd.2 173 enqueue_op 0x24c9ce00 prio 63 cost 516096 latency 0.001495 osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.884859 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.884867 7f9d67be0700 10 osd.2 173 dequeue_op 0x24c9ce00 prio 63 cost 516096 latency 0.001551 osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]
2014-06-12 14:46:44.884888 7f9d67be0700  5 -- op tracker -- , seq: 747984, time: 2014-06-12 14:46:44.884884, event: reached_pg, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.884895 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.884906 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] handle_message: 0x24c9ce00
2014-06-12 14:46:44.884912 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] do_op osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.884929 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] get_object_context: found obc in cache: 0x10092b00
2014-06-12 14:46:44.884936 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] get_object_context: 0x10092b00 eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 rwstate(write n=1 w=0) oi: eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2701 client.4316.0:147231 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2701) ssc: 0x1bc1ea0 snapset: 0=[]:[]+head
2014-06-12 14:46:44.884946 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] find_object_context eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 @head oi=eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2701 client.4316.0:147231 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2701)
2014-06-12 14:46:44.884971 7f9d67be0700  5 -- op tracker -- , seq: 747984, time: 2014-06-12 14:46:44.884967, event: started, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.884977 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] execute_ctx 0xa050400
2014-06-12 14:46:44.884988 7f9d67be0700  5 -- op tracker -- , seq: 747984, time: 2014-06-12 14:46:44.884985, event: started, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.884993 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] do_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] ov 173'2701 av 173'2702 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.885001 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] do_osd_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096]
2014-06-12 14:46:44.885008 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.885016 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] do_osd_op  write 2654208~516096
2014-06-12 14:46:44.885026 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:44.885033 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 snapset=0x1bc1ed8  snapc=0=[]
2014-06-12 14:46:44.885041 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.885046 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] finish_ctx eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 0xa050400 op modify  
2014-06-12 14:46:44.885061 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]  set mtime to 2014-06-12 08:45:47.262849
2014-06-12 14:46:44.885076 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]  final snapset 0=[]:[]+head in eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.885090 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.885098 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] new_repop rep_tid 236303 on osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.885114 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] issue_repop rep_tid 236303 o eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.885132 7f9d67be0700  5 -- op tracker -- , seq: 747984, time: 2014-06-12 14:46:44.885129, event: waiting for subops from 23, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.885151 7f9d67be0700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:44.885170 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2701 (0'0,173'2701] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] append_log log((0'0,173'2701], crt=0'0) [173'2702 (173'2701) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147232 2014-06-12 08:45:47.262849]
2014-06-12 14:46:44.885187 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] add_log_entry 173'2702 (173'2701) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147232 2014-06-12 08:45:47.262849
2014-06-12 14:46:44.885199 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.885224 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2702, trimmed: 
2014-06-12 14:46:44.885240 7f9d67be0700  5 queue_transactions existing osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.885243 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.885244 7f9d67be0700  5 queue_op 0x18586540 seq 2469 osr(3.7cc 0x49b43c0) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:44.885251 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f7200 (2 active)
2014-06-12 14:46:44.885249 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] eval_repop repgather(0x28df23c0 173'2702 rep_tid=236303 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.885261 7f9d67be0700 10 osd.2 173 dequeue_op 0x24c9ce00 finish
2014-06-12 14:46:44.885263 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.885265 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.886347 7f9d47c58700  5 -- op tracker -- , seq: 747985, time: 2014-06-12 14:46:44.884863, event: header_read, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.886360 7f9d47c58700  5 -- op tracker -- , seq: 747985, time: 2014-06-12 14:46:44.884864, event: throttled, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.886368 7f9d47c58700  5 -- op tracker -- , seq: 747985, time: 2014-06-12 14:46:44.886316, event: all_read, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.886377 7f9d47c58700  5 -- op tracker -- , seq: 747985, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.886386 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:44.886412 7f9d47c58700 15 osd.2 173 enqueue_op 0x24c9c800 prio 63 cost 516096 latency 0.001549 osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.886441 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.886451 7f9d663dd700 10 osd.2 173 dequeue_op 0x24c9c800 prio 63 cost 516096 latency 0.001587 osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]
2014-06-12 14:46:44.886479 7f9d663dd700  5 -- op tracker -- , seq: 747985, time: 2014-06-12 14:46:44.886473, event: reached_pg, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.886490 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.886507 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] handle_message: 0x24c9c800
2014-06-12 14:46:44.886519 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] do_op osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.886546 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] get_object_context: found obc in cache: 0x10092b00
2014-06-12 14:46:44.886560 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] get_object_context: 0x10092b00 eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 rwstate(write n=2 w=0) oi: eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2702 client.4316.0:147232 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2702) ssc: 0x1bc1ea0 snapset: 0=[]:[]+head
2014-06-12 14:46:44.886580 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] find_object_context eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 @head oi=eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2702 client.4316.0:147232 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2702)
2014-06-12 14:46:44.886616 7f9d663dd700  5 -- op tracker -- , seq: 747985, time: 2014-06-12 14:46:44.886611, event: started, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.886623 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] execute_ctx 0x6245600
2014-06-12 14:46:44.886642 7f9d663dd700  5 -- op tracker -- , seq: 747985, time: 2014-06-12 14:46:44.886637, event: started, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.886651 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] do_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] ov 173'2702 av 173'2703 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.886668 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] do_osd_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096]
2014-06-12 14:46:44.886683 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.886698 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] do_osd_op  write 3170304~516096
2014-06-12 14:46:44.886714 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:44.886727 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 snapset=0x1bc1ed8  snapc=0=[]
2014-06-12 14:46:44.886741 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.886752 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] finish_ctx eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 0x6245600 op modify  
2014-06-12 14:46:44.886767 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]  set mtime to 2014-06-12 08:45:47.263470
2014-06-12 14:46:44.886794 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]  final snapset 0=[]:[]+head in eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.886819 7f9d8016d700 10 _do_op 0x803f4a0 seq 2468 r = 0, finisher 0x76ae680 0x31af15c0
2014-06-12 14:46:44.886814 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.886835 7f9d8016d700 10 _finish_op 0x803f4a0 seq 2468 osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.886830 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] new_repop rep_tid 236304 on osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.886843 7f9d7915f700  5 _do_op 0x18586540 seq 2469 osr(3.7cc 0x49b43c0)/0x49b43c0 start
2014-06-12 14:46:44.886845 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f7200 (1 active)
2014-06-12 14:46:44.886848 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.886849 7f9d7915f700 10 _do_transaction on 0x34d8fe00
2014-06-12 14:46:44.886848 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] issue_repop rep_tid 236304 o eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.886858 7f9d7915f700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.886876 7f9d663dd700  5 -- op tracker -- , seq: 747985, time: 2014-06-12 14:46:44.886870, event: waiting for subops from 23, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.886901 7f9d663dd700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:44.886926 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2702 (0'0,173'2702] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] append_log log((0'0,173'2702], crt=0'0) [173'2703 (173'2702) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147233 2014-06-12 08:45:47.263470]
2014-06-12 14:46:44.886949 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] add_log_entry 173'2703 (173'2702) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147233 2014-06-12 08:45:47.263470
2014-06-12 14:46:44.886970 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.886980 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.886998 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2703, trimmed: 
2014-06-12 14:46:44.887007 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.887016 7f9d663dd700  5 queue_transactions existing osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.887019 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.887021 7f9d663dd700  5 queue_op 0x16064f60 seq 2470 osr(3.7cc 0x49b43c0) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:44.887027 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] eval_repop repgather(0x31b65d40 173'2703 rep_tid=236304 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.887044 7f9d663dd700 10 osd.2 173 dequeue_op 0x24c9c800 finish
2014-06-12 14:46:44.887050 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.887053 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.887058 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.887061 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f7200 (2 active)
2014-06-12 14:46:44.887153 7f9d7915f700 15 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.887159 7f9d7915f700 10 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1 r = 0
2014-06-12 14:46:44.887164 7f9d7915f700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.887176 7f9d7915f700 15 _write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 2654208~516096
2014-06-12 14:46:44.887277 7f9d7915f700 10 lookup_strip_header done  cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.887321 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] op_commit: 236302
2014-06-12 14:46:44.887342 7f9d7715b700  5 -- op tracker -- , seq: 747981, time: 2014-06-12 14:46:44.887337, event: op_commit, op: osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.887354 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] op_applied: 236302
2014-06-12 14:46:44.887364 7f9d57543700  5 -- op tracker -- , seq: 747986, time: 2014-06-12 14:46:44.887328, event: header_read, op: osd_sub_op_reply(client.4316.0:147231 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.887377 7f9d57543700  5 -- op tracker -- , seq: 747986, time: 2014-06-12 14:46:44.887329, event: throttled, op: osd_sub_op_reply(client.4316.0:147231 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.887385 7f9d57543700  5 -- op tracker -- , seq: 747986, time: 2014-06-12 14:46:44.887343, event: all_read, op: osd_sub_op_reply(client.4316.0:147231 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.887393 7f9d57543700  5 -- op tracker -- , seq: 747986, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147231 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.887398 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147231 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:44.887403 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:44.887404 7f9d7795c700  5 -- op tracker -- , seq: 747981, time: 2014-06-12 14:46:44.887367, event: op_applied, op: osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.887409 7f9d57543700 15 osd.2 173 enqueue_op 0x876ef00 prio 196 cost 0 latency 0.000081 osd_sub_op_reply(client.4316.0:147231 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:44.887417 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:44.887413 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] op_applied on primary on version 173'2701
2014-06-12 14:46:44.887435 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.887441 7f9d693e3700 10 osd.2 173 dequeue_op 0x876ef00 prio 196 cost 0 latency 0.000113 osd_sub_op_reply(client.4316.0:147231 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 pg pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]
2014-06-12 14:46:44.887457 7f9d693e3700  5 -- op tracker -- , seq: 747986, time: 2014-06-12 14:46:44.887454, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147231 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.887464 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] handle_message: 0x876ef00
2014-06-12 14:46:44.887478 7f9d693e3700  5 -- op tracker -- , seq: 747986, time: 2014-06-12 14:46:44.887475, event: started, op: osd_sub_op_reply(client.4316.0:147231 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.887485 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] sub_op_modify_reply: tid 236302 op  ack_type 4 from 23
2014-06-12 14:46:44.887502 7f9d693e3700  5 -- op tracker -- , seq: 747981, time: 2014-06-12 14:46:44.887496, event: sub_op_commit_rec, op: osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.887508 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] repop_all_applied: repop tid 236302 all applied 
2014-06-12 14:46:44.887518 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] eval_repop repgather(0x2a37acc0 173'2701 rep_tid=236302 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.887531 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean]  sending ack on repgather(0x2a37acc0 173'2701 rep_tid=236302 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x1f888f00
2014-06-12 14:46:44.887557 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2700 crt=0'0 lcod 173'2699 mlcod 173'2699 active+clean] repop_all_committed: repop tid 236302 all committed 
2014-06-12 14:46:44.887567 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2699 active+clean] eval_repop repgather(0x2a37acc0 173'2701 rep_tid=236302 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.887581 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2699 active+clean] log_op_stats osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008925 lat 0.008949
2014-06-12 14:46:44.887595 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2699 active+clean] publish_stats_to_osd 173:3519
2014-06-12 14:46:44.887606 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2699 active+clean]  sending commit on repgather(0x2a37acc0 173'2701 rep_tid=236302 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x16971400
2014-06-12 14:46:44.887634 7f9d693e3700  5 -- op tracker -- , seq: 747981, time: 2014-06-12 14:46:44.887629, event: commit_sent, op: osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.887644 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2699 active+clean]  requeue_ops 
2014-06-12 14:46:44.887654 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]  removing repgather(0x2a37acc0 173'2701 rep_tid=236302 committed?=1 applied?=1 op=osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.887668 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]    q front is repgather(0x2a37acc0 173'2701 rep_tid=236302 committed?=1 applied?=1 op=osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.887681 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] remove_repop repgather(0x2a37acc0 173'2701 rep_tid=236302 committed?=1 applied?=1 op=osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.887694 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]  requeue_ops 
2014-06-12 14:46:44.887711 7f9d693e3700  5 -- op tracker -- , seq: 747981, time: 2014-06-12 14:46:44.887706, event: done, op: osd_op(client.4316.0:147231 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2138112~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.887729 7f9d693e3700 10 osd.2 173 dequeue_op 0x876ef00 finish
2014-06-12 14:46:44.887736 7f9d693e3700  5 -- op tracker -- , seq: 747986, time: 2014-06-12 14:46:44.887732, event: done, op: osd_sub_op_reply(client.4316.0:147231 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.887750 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.887754 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.888802 7f9d7915f700 10 _generic_write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 2654208~516096 = 0
2014-06-12 14:46:44.888839 7f9d7915f700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.888852 7f9d7915f700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.888857 7f9d7915f700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.888862 7f9d7915f700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.888999 7f9d7915f700 20 set_header setting 17435 cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 parent seq 0
2014-06-12 14:46:44.889057 7f9d7915f700 20 set_header setting 2154 cid meta oid 1fe706d6/pglog_3.7cc/0//-1 parent seq 0
2014-06-12 14:46:44.889068 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.889075 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:44.891419 7f9d47c58700  5 -- op tracker -- , seq: 747987, time: 2014-06-12 14:46:44.890074, event: header_read, op: osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.891433 7f9d47c58700  5 -- op tracker -- , seq: 747987, time: 2014-06-12 14:46:44.890078, event: throttled, op: osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.891439 7f9d47c58700  5 -- op tracker -- , seq: 747987, time: 2014-06-12 14:46:44.891374, event: all_read, op: osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.891444 7f9d47c58700  5 -- op tracker -- , seq: 747987, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.891451 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:44.891475 7f9d47c58700 15 osd.2 173 enqueue_op 0x1a16ee00 prio 63 cost 507904 latency 0.001401 osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.891507 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.891515 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x1a16ee00 prio 63 cost 507904 latency 0.001440 osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]
2014-06-12 14:46:44.891538 7f9d6b3e7700  5 -- op tracker -- , seq: 747987, time: 2014-06-12 14:46:44.891534, event: reached_pg, op: osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.891546 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.891559 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] handle_message: 0x1a16ee00
2014-06-12 14:46:44.891568 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] do_op osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.891590 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] get_object_context: found obc in cache: 0x10092b00
2014-06-12 14:46:44.891606 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] get_object_context: 0x10092b00 eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 rwstate(write n=2 w=0) oi: eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2703 client.4316.0:147233 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2703) ssc: 0x1bc1ea0 snapset: 0=[]:[]+head
2014-06-12 14:46:44.891620 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] find_object_context eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 @head oi=eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3(173'2703 client.4316.0:147233 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2703)
2014-06-12 14:46:44.891644 7f9d6b3e7700  5 -- op tracker -- , seq: 747987, time: 2014-06-12 14:46:44.891639, event: started, op: osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.891650 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] execute_ctx 0x29edcc00
2014-06-12 14:46:44.891665 7f9d6b3e7700  5 -- op tracker -- , seq: 747987, time: 2014-06-12 14:46:44.891661, event: started, op: osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.891671 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] do_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] ov 173'2703 av 173'2704 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.891683 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] do_osd_op eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904]
2014-06-12 14:46:44.891693 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.891705 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] do_osd_op  write 3686400~507904
2014-06-12 14:46:44.891719 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:44.891728 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 snapset=0x1bc1ed8  snapc=0=[]
2014-06-12 14:46:44.891739 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] make_writeable eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.891748 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] finish_ctx eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 0x29edcc00 op modify  
2014-06-12 14:46:44.891758 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]  set mtime to 2014-06-12 08:45:47.269499
2014-06-12 14:46:44.891781 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]  final snapset 0=[]:[]+head in eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.891801 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.891811 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] new_repop rep_tid 236305 on osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.891827 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] issue_repop rep_tid 236305 o eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.891855 7f9d6b3e7700  5 -- op tracker -- , seq: 747987, time: 2014-06-12 14:46:44.891850, event: waiting for subops from 23, op: osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.891880 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:44.891902 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2703 (0'0,173'2703] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] append_log log((0'0,173'2703], crt=0'0) [173'2704 (173'2703) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147234 2014-06-12 08:45:47.269499]
2014-06-12 14:46:44.891927 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] add_log_entry 173'2704 (173'2703) modify   eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 by client.4316.0:147234 2014-06-12 08:45:47.269499
2014-06-12 14:46:44.891946 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.891976 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2704, trimmed: 
2014-06-12 14:46:44.891996 7f9d6b3e7700  5 queue_transactions existing osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.892002 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.892004 7f9d6b3e7700  5 queue_op 0x68995c0 seq 2471 osr(3.7cc 0x49b43c0) 510197 bytes   (queue has 3 ops and 1546975 bytes)
2014-06-12 14:46:44.892010 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] eval_repop repgather(0x1ced2900 173'2704 rep_tid=236305 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.892028 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x1a16ee00 finish
2014-06-12 14:46:44.892032 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.892037 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.892506 7f9d7915f700 10 _do_op 0x18586540 seq 2469 r = 0, finisher 0x53a8380 0x1a3f17c0
2014-06-12 14:46:44.892526 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] op_commit: 236303
2014-06-12 14:46:44.892558 7f9d7715b700  5 -- op tracker -- , seq: 747984, time: 2014-06-12 14:46:44.892550, event: op_commit, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.892506 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53f7200 (3 active)
2014-06-12 14:46:44.892791 7f9d7915f700 10 _finish_op 0x18586540 seq 2469 osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.892802 7f9d7d968700  5 _do_op 0x16064f60 seq 2470 osr(3.7cc 0x49b43c0)/0x49b43c0 start
2014-06-12 14:46:44.892811 7f9d7d968700 10 _do_transaction on 0x31b65a40
2014-06-12 14:46:44.892810 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] op_applied: 236303
2014-06-12 14:46:44.892819 7f9d7d968700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.892832 7f9d7795c700  5 -- op tracker -- , seq: 747984, time: 2014-06-12 14:46:44.892826, event: op_applied, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.892840 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] op_applied on primary on version 173'2702
2014-06-12 14:46:44.892873 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f7200 (2 active)
2014-06-12 14:46:44.892886 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.892947 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.892970 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.893033 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.893052 7f9d7d968700 15 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.893057 7f9d7d968700 10 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1 r = 0
2014-06-12 14:46:44.893063 7f9d7d968700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.893081 7f9d7d968700 15 _write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 3170304~516096
2014-06-12 14:46:44.893210 7f9d7d968700 10 lookup_strip_header done  cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.893306 7f9d57543700  5 -- op tracker -- , seq: 747988, time: 2014-06-12 14:46:44.893274, event: header_read, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.893315 7f9d57543700  5 -- op tracker -- , seq: 747988, time: 2014-06-12 14:46:44.893275, event: throttled, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.893322 7f9d57543700  5 -- op tracker -- , seq: 747988, time: 2014-06-12 14:46:44.893283, event: all_read, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.893329 7f9d57543700  5 -- op tracker -- , seq: 747988, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.893334 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:44.893339 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:44.893346 7f9d57543700 15 osd.2 173 enqueue_op 0x20e86e00 prio 196 cost 0 latency 0.000072 osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0) v2
2014-06-12 14:46:44.893372 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.893378 7f9d65bdc700 10 osd.2 173 dequeue_op 0x20e86e00 prio 196 cost 0 latency 0.000103 osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0) v2 pg pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]
2014-06-12 14:46:44.893395 7f9d65bdc700  5 -- op tracker -- , seq: 747988, time: 2014-06-12 14:46:44.893391, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.893401 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:44.893402 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] handle_message: 0x20e86e00
2014-06-12 14:46:44.893416 7f9d65bdc700  5 -- op tracker -- , seq: 747988, time: 2014-06-12 14:46:44.893413, event: started, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.893421 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] sub_op_modify_reply: tid 236303 op  ack_type 1 from 23
2014-06-12 14:46:44.893440 7f9d65bdc700  5 -- op tracker -- , seq: 747984, time: 2014-06-12 14:46:44.893434, event: sub_op_applied_rec, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.893448 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] repop_all_applied: repop tid 236303 all applied 
2014-06-12 14:46:44.893459 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] eval_repop repgather(0x28df23c0 173'2702 rep_tid=236303 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.893476 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]  sending ack on repgather(0x28df23c0 173'2702 rep_tid=236303 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x1710bb80
2014-06-12 14:46:44.893508 7f9d65bdc700 10 osd.2 173 dequeue_op 0x20e86e00 finish
2014-06-12 14:46:44.893516 7f9d65bdc700  5 -- op tracker -- , seq: 747988, time: 2014-06-12 14:46:44.893512, event: done, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.893531 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.893534 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.893631 7f9d57543700  5 -- op tracker -- , seq: 747989, time: 2014-06-12 14:46:44.893588, event: header_read, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.893640 7f9d57543700  5 -- op tracker -- , seq: 747989, time: 2014-06-12 14:46:44.893589, event: throttled, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.893648 7f9d57543700  5 -- op tracker -- , seq: 747989, time: 2014-06-12 14:46:44.893600, event: all_read, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.893656 7f9d57543700  5 -- op tracker -- , seq: 747989, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.893660 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:44.893665 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:44.893671 7f9d57543700 15 osd.2 173 enqueue_op 0x20e86700 prio 196 cost 0 latency 0.000083 osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:44.893696 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.893702 7f9d62bd6700 10 osd.2 173 dequeue_op 0x20e86700 prio 196 cost 0 latency 0.000114 osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 pg pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean]
2014-06-12 14:46:44.893720 7f9d62bd6700  5 -- op tracker -- , seq: 747989, time: 2014-06-12 14:46:44.893717, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.893727 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] handle_message: 0x20e86700
2014-06-12 14:46:44.893742 7f9d62bd6700  5 -- op tracker -- , seq: 747989, time: 2014-06-12 14:46:44.893739, event: started, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.893748 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] sub_op_modify_reply: tid 236303 op  ack_type 4 from 23
2014-06-12 14:46:44.893765 7f9d62bd6700  5 -- op tracker -- , seq: 747984, time: 2014-06-12 14:46:44.893759, event: sub_op_commit_rec, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.893771 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2701 crt=0'0 lcod 173'2700 mlcod 173'2700 active+clean] repop_all_committed: repop tid 236303 all committed 
2014-06-12 14:46:44.893781 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2700 active+clean] eval_repop repgather(0x28df23c0 173'2702 rep_tid=236303 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.893796 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2700 active+clean] log_op_stats osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.010190 lat 0.010479
2014-06-12 14:46:44.893813 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2700 active+clean] publish_stats_to_osd 173:3520
2014-06-12 14:46:44.893826 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2700 active+clean]  sending commit on repgather(0x28df23c0 173'2702 rep_tid=236303 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x17f00a00
2014-06-12 14:46:44.893859 7f9d62bd6700  5 -- op tracker -- , seq: 747984, time: 2014-06-12 14:46:44.893853, event: commit_sent, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.893867 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2700 active+clean]  requeue_ops 
2014-06-12 14:46:44.893878 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]  removing repgather(0x28df23c0 173'2702 rep_tid=236303 committed?=1 applied?=1 op=osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.893893 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]    q front is repgather(0x28df23c0 173'2702 rep_tid=236303 committed?=1 applied?=1 op=osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.893908 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] remove_repop repgather(0x28df23c0 173'2702 rep_tid=236303 committed?=1 applied?=1 op=osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.893923 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]  requeue_ops 
2014-06-12 14:46:44.893943 7f9d62bd6700  5 -- op tracker -- , seq: 747984, time: 2014-06-12 14:46:44.893938, event: done, op: osd_op(client.4316.0:147232 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 2654208~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.893958 7f9d62bd6700 10 osd.2 173 dequeue_op 0x20e86700 finish
2014-06-12 14:46:44.893966 7f9d62bd6700  5 -- op tracker -- , seq: 747989, time: 2014-06-12 14:46:44.893962, event: done, op: osd_sub_op_reply(client.4316.0:147232 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.893976 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.893979 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.895414 7f9d7d968700 10 _generic_write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 3170304~516096 = 0
2014-06-12 14:46:44.895472 7f9d7d968700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.895513 7f9d7d968700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.895518 7f9d7d968700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.895523 7f9d7d968700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.895679 7f9d7d968700 20 set_header setting 17435 cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 parent seq 0
2014-06-12 14:46:44.895761 7f9d7d968700 20 set_header setting 2154 cid meta oid 1fe706d6/pglog_3.7cc/0//-1 parent seq 0
2014-06-12 14:46:44.895772 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.895779 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:44.897579 7f9d7d968700 10 _do_op 0x16064f60 seq 2470 r = 0, finisher 0x7854e60 0x1a3f1d00
2014-06-12 14:46:44.897598 7f9d7d968700 10 _finish_op 0x16064f60 seq 2470 osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.897609 7f9d7c165700  5 _do_op 0x68995c0 seq 2471 osr(3.7cc 0x49b43c0)/0x49b43c0 start
2014-06-12 14:46:44.897632 7f9d7c165700 10 _do_transaction on 0x5214000
2014-06-12 14:46:44.897642 7f9d7c165700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.897607 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] op_commit: 236304
2014-06-12 14:46:44.897611 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f7200 (1 active)
2014-06-12 14:46:44.897661 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.897665 7f9d7715b700  5 -- op tracker -- , seq: 747985, time: 2014-06-12 14:46:44.897652, event: op_commit, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.897681 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] op_applied: 236304
2014-06-12 14:46:44.897710 7f9d7795c700  5 -- op tracker -- , seq: 747985, time: 2014-06-12 14:46:44.897702, event: op_applied, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.897720 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] op_applied on primary on version 173'2703
2014-06-12 14:46:44.897781 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.897900 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.897955 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.897979 7f9d7c165700 15 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.897994 7f9d7c165700 10 _omap_rmkeys meta/1fe706d6/pglog_3.7cc/0//-1 r = 0
2014-06-12 14:46:44.898008 7f9d7c165700 15 _omap_setkeys meta/1fe706d6/pglog_3.7cc/0//-1
2014-06-12 14:46:44.898023 7f9d7c165700 15 _write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 3686400~507904
2014-06-12 14:46:44.898154 7f9d7c165700 10 lookup_strip_header done  cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.898333 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:44.900217 7f9d7c165700 10 _generic_write 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 3686400~507904 = 0
2014-06-12 14:46:44.900268 7f9d7c165700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.900286 7f9d7c165700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.900294 7f9d7c165700 15 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3
2014-06-12 14:46:44.900301 7f9d7c165700 10 _setattrs 3.7cc_head/eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 = 0
2014-06-12 14:46:44.900737 7f9d7c165700 20 set_header setting 17435 cid 3.7cc_head oid eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 parent seq 0
2014-06-12 14:46:44.900832 7f9d7c165700 20 set_header setting 2154 cid meta oid 1fe706d6/pglog_3.7cc/0//-1 parent seq 0
2014-06-12 14:46:44.900847 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.900855 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:44.903840 7f9d7c165700 10 _do_op 0x68995c0 seq 2471 r = 0, finisher 0x63cfde0 0x1cbed4c0
2014-06-12 14:46:44.903865 7f9d7c165700 10 _finish_op 0x68995c0 seq 2471 osr(3.7cc 0x49b43c0)/0x49b43c0
2014-06-12 14:46:44.903875 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53f7200 (0 active)
2014-06-12 14:46:44.903883 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.903881 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] op_applied: 236305
2014-06-12 14:46:44.903910 7f9d7795c700  5 -- op tracker -- , seq: 747987, time: 2014-06-12 14:46:44.903901, event: op_applied, op: osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.903922 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] op_applied on primary on version 173'2704
2014-06-12 14:46:44.904007 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] op_commit: 236305
2014-06-12 14:46:44.904024 7f9d7715b700  5 -- op tracker -- , seq: 747987, time: 2014-06-12 14:46:44.904019, event: op_commit, op: osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.913710 7f9d57543700  5 -- op tracker -- , seq: 747990, time: 2014-06-12 14:46:44.913642, event: header_read, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.913726 7f9d57543700  5 -- op tracker -- , seq: 747990, time: 2014-06-12 14:46:44.913644, event: throttled, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.913748 7f9d57543700  5 -- op tracker -- , seq: 747990, time: 2014-06-12 14:46:44.913661, event: all_read, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.913754 7f9d57543700  5 -- op tracker -- , seq: 747990, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.913760 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:44.913766 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:44.913774 7f9d57543700 15 osd.2 173 enqueue_op 0x20e86600 prio 196 cost 0 latency 0.000132 osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0) v2
2014-06-12 14:46:44.913817 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.913828 7f9d66bde700 10 osd.2 173 dequeue_op 0x20e86600 prio 196 cost 0 latency 0.000186 osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0) v2 pg pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]
2014-06-12 14:46:44.913849 7f9d57543700  5 -- op tracker -- , seq: 747991, time: 2014-06-12 14:46:44.913811, event: header_read, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.913860 7f9d57543700  5 -- op tracker -- , seq: 747991, time: 2014-06-12 14:46:44.913811, event: throttled, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.913870 7f9d57543700  5 -- op tracker -- , seq: 747991, time: 2014-06-12 14:46:44.913828, event: all_read, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.913879 7f9d57543700  5 -- op tracker -- , seq: 747991, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.913885 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:44.913888 7f9d66bde700  5 -- op tracker -- , seq: 747990, time: 2014-06-12 14:46:44.913846, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.913890 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:44.913898 7f9d57543700 15 osd.2 173 enqueue_op 0x20e86200 prio 196 cost 0 latency 0.000087 osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:44.913896 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] handle_message: 0x20e86600
2014-06-12 14:46:44.913907 7f9d66bde700  5 -- op tracker -- , seq: 747990, time: 2014-06-12 14:46:44.913905, event: started, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.913912 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] sub_op_modify_reply: tid 236304 op  ack_type 1 from 23
2014-06-12 14:46:44.913925 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:44.913926 7f9d66bde700  5 -- op tracker -- , seq: 747985, time: 2014-06-12 14:46:44.913920, event: sub_op_applied_rec, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.913932 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] repop_all_applied: repop tid 236304 all applied 
2014-06-12 14:46:44.913939 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] eval_repop repgather(0x31b65d40 173'2703 rep_tid=236304 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.913949 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]  sending ack on repgather(0x31b65d40 173'2703 rep_tid=236304 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x11667400
2014-06-12 14:46:44.913973 7f9d66bde700 10 osd.2 173 dequeue_op 0x20e86600 finish
2014-06-12 14:46:44.913985 7f9d683e1700 10 osd.2 173 dequeue_op 0x20e86200 prio 196 cost 0 latency 0.000174 osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 pg pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean]
2014-06-12 14:46:44.913984 7f9d66bde700  5 -- op tracker -- , seq: 747990, time: 2014-06-12 14:46:44.913982, event: done, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ack, result = 0)
2014-06-12 14:46:44.914021 7f9d683e1700  5 -- op tracker -- , seq: 747991, time: 2014-06-12 14:46:44.914002, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.914050 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:44.914041 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] handle_message: 0x20e86200
2014-06-12 14:46:44.914053 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.914057 7f9d683e1700  5 -- op tracker -- , seq: 747991, time: 2014-06-12 14:46:44.914053, event: started, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.914063 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] sub_op_modify_reply: tid 236304 op  ack_type 4 from 23
2014-06-12 14:46:44.914081 7f9d683e1700  5 -- op tracker -- , seq: 747985, time: 2014-06-12 14:46:44.914074, event: sub_op_commit_rec, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.914088 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2702 crt=0'0 lcod 173'2701 mlcod 173'2701 active+clean] repop_all_committed: repop tid 236304 all committed 
2014-06-12 14:46:44.914099 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean] eval_repop repgather(0x31b65d40 173'2703 rep_tid=236304 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.914115 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean] log_op_stats osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.029108 lat 0.029251
2014-06-12 14:46:44.914134 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean] publish_stats_to_osd 173:3521
2014-06-12 14:46:44.914147 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean]  sending commit on repgather(0x31b65d40 173'2703 rep_tid=236304 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x21ed9180
2014-06-12 14:46:44.914181 7f9d683e1700  5 -- op tracker -- , seq: 747985, time: 2014-06-12 14:46:44.914175, event: commit_sent, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.914188 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2701 active+clean]  requeue_ops 
2014-06-12 14:46:44.914199 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]  removing repgather(0x31b65d40 173'2703 rep_tid=236304 committed?=1 applied?=1 op=osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.914214 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]    q front is repgather(0x31b65d40 173'2703 rep_tid=236304 committed?=1 applied?=1 op=osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.914228 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] remove_repop repgather(0x31b65d40 173'2703 rep_tid=236304 committed?=1 applied?=1 op=osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.914242 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]  requeue_ops 
2014-06-12 14:46:44.914261 7f9d683e1700  5 -- op tracker -- , seq: 747985, time: 2014-06-12 14:46:44.914256, event: done, op: osd_op(client.4316.0:147233 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3170304~516096] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.914290 7f9d683e1700 10 osd.2 173 dequeue_op 0x20e86200 finish
2014-06-12 14:46:44.914301 7f9d683e1700  5 -- op tracker -- , seq: 747991, time: 2014-06-12 14:46:44.914297, event: done, op: osd_sub_op_reply(client.4316.0:147233 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.914311 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.914314 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.917653 7f9d57543700  5 -- op tracker -- , seq: 747992, time: 2014-06-12 14:46:44.917591, event: header_read, op: osd_sub_op_reply(client.4316.0:147234 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.917666 7f9d57543700  5 -- op tracker -- , seq: 747992, time: 2014-06-12 14:46:44.917594, event: throttled, op: osd_sub_op_reply(client.4316.0:147234 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.917682 7f9d57543700  5 -- op tracker -- , seq: 747992, time: 2014-06-12 14:46:44.917618, event: all_read, op: osd_sub_op_reply(client.4316.0:147234 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.917687 7f9d57543700  5 -- op tracker -- , seq: 747992, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147234 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.917695 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147234 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:44.917699 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:44.917706 7f9d57543700 15 osd.2 173 enqueue_op 0x68a7100 prio 196 cost 0 latency 0.000115 osd_sub_op_reply(client.4316.0:147234 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:44.917734 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.917741 7f9d64bda700 10 osd.2 173 dequeue_op 0x68a7100 prio 196 cost 0 latency 0.000150 osd_sub_op_reply(client.4316.0:147234 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0) v2 pg pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]
2014-06-12 14:46:44.917756 7f9d64bda700  5 -- op tracker -- , seq: 747992, time: 2014-06-12 14:46:44.917754, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147234 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.917763 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] handle_message: 0x68a7100
2014-06-12 14:46:44.917772 7f9d64bda700  5 -- op tracker -- , seq: 747992, time: 2014-06-12 14:46:44.917771, event: started, op: osd_sub_op_reply(client.4316.0:147234 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.917776 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] sub_op_modify_reply: tid 236305 op  ack_type 4 from 23
2014-06-12 14:46:44.917790 7f9d64bda700  5 -- op tracker -- , seq: 747987, time: 2014-06-12 14:46:44.917784, event: sub_op_commit_rec, op: osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.917795 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] repop_all_applied: repop tid 236305 all applied 
2014-06-12 14:46:44.917802 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] eval_repop repgather(0x1ced2900 173'2704 rep_tid=236305 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.917812 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean]  sending ack on repgather(0x1ced2900 173'2704 rep_tid=236305 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x26e56a00
2014-06-12 14:46:44.917833 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2703 crt=0'0 lcod 173'2702 mlcod 173'2702 active+clean] repop_all_committed: repop tid 236305 all committed 
2014-06-12 14:46:44.917840 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean] eval_repop repgather(0x1ced2900 173'2704 rep_tid=236305 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.917853 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean] log_op_stats osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4 inb 508183 outb 0 rlat 0.027758 lat 0.027777
2014-06-12 14:46:44.917864 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean] publish_stats_to_osd 173:3522
2014-06-12 14:46:44.917873 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean]  sending commit on repgather(0x1ced2900 173'2704 rep_tid=236305 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4) 0x33b16000
2014-06-12 14:46:44.917892 7f9d64bda700  5 -- op tracker -- , seq: 747987, time: 2014-06-12 14:46:44.917889, event: commit_sent, op: osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.917898 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2702 active+clean]  requeue_ops 
2014-06-12 14:46:44.917905 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]  removing repgather(0x1ced2900 173'2704 rep_tid=236305 committed?=1 applied?=1 op=osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.917914 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]    q front is repgather(0x1ced2900 173'2704 rep_tid=236305 committed?=1 applied?=1 op=osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.917922 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] remove_repop repgather(0x1ced2900 173'2704 rep_tid=236305 committed?=1 applied?=1 op=osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:44.917930 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean]  requeue_ops 
2014-06-12 14:46:44.917948 7f9d64bda700  5 -- op tracker -- , seq: 747987, time: 2014-06-12 14:46:44.917945, event: done, op: osd_op(client.4316.0:147234 rbd_data.10b36b8b4567.0000000000001131 [set-alloc-hint object_size 4194304 write_size 4194304,write 3686400~507904] 3.eb2517cc ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.917968 7f9d64bda700 10 osd.2 173 dequeue_op 0x68a7100 finish
2014-06-12 14:46:44.917972 7f9d64bda700  5 -- op tracker -- , seq: 747992, time: 2014-06-12 14:46:44.917970, event: done, op: osd_sub_op_reply(client.4316.0:147234 3.7cc eb2517cc/rbd_data.10b36b8b4567.0000000000001131/head//3 [] ondisk, result = 0)
2014-06-12 14:46:44.917979 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.917981 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.958617 7f9d461e0700  5 -- op tracker -- , seq: 747993, time: 2014-06-12 14:46:44.958110, event: header_read, op: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.958631 7f9d461e0700  5 -- op tracker -- , seq: 747993, time: 2014-06-12 14:46:44.958114, event: throttled, op: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.958640 7f9d461e0700  5 -- op tracker -- , seq: 747993, time: 2014-06-12 14:46:44.958572, event: all_read, op: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.958648 7f9d461e0700  5 -- op tracker -- , seq: 747993, time: 0.000000, event: dispatched, op: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.958657 7f9d461e0700 20 osd.2 173 should_share_map client.4293 172.16.96.220:0/1241966 173
2014-06-12 14:46:44.958685 7f9d461e0700 15 osd.2 173 enqueue_op 0x876ef00 prio 63 cost 172032 latency 0.000575 osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.958718 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.958727 7f9d6abe6700 10 osd.2 173 dequeue_op 0x876ef00 prio 63 cost 172032 latency 0.000616 osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]
2014-06-12 14:46:44.958754 7f9d6abe6700  5 -- op tracker -- , seq: 747993, time: 2014-06-12 14:46:44.958749, event: reached_pg, op: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.958763 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.958779 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] handle_message: 0x876ef00
2014-06-12 14:46:44.958790 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_op osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.958819 7f9d6abe6700 15 getattr 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 '_'
2014-06-12 14:46:44.959149 7f9d6abe6700 10 getattr 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 '_' = 0
2014-06-12 14:46:44.959173 7f9d6abe6700 15 getattr 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 'snapset'
2014-06-12 14:46:44.959260 7f9d6abe6700 10 getattr 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 'snapset' = 0
2014-06-12 14:46:44.959270 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] populate_obc_watchers e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:44.959283 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:44.959292 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] get_object_context: creating obc from disk: 0x20107600
2014-06-12 14:46:44.959301 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] get_object_context: 0x20107600 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 rwstate(none n=0 w=0) oi: e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3(173'784 client.4293.0:38345 wrlock_by=unknown.0.0:0 dirty s 4194304 uv784) ssc: 0x72b8d20 snapset: 0=[]:[]+head
2014-06-12 14:46:44.959317 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] find_object_context e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 @head oi=e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3(173'784 client.4293.0:38345 wrlock_by=unknown.0.0:0 dirty s 4194304 uv784)
2014-06-12 14:46:44.959352 7f9d6abe6700  5 -- op tracker -- , seq: 747993, time: 2014-06-12 14:46:44.959346, event: started, op: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.959360 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] execute_ctx 0x29edc000
2014-06-12 14:46:44.959376 7f9d6abe6700  5 -- op tracker -- , seq: 747993, time: 2014-06-12 14:46:44.959372, event: started, op: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.959383 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_op e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] ov 173'784 av 173'2591 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.959397 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032]
2014-06-12 14:46:44.959408 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.959421 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op  write 0~172032
2014-06-12 14:46:44.959438 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op_effects on session 0x5c8cc00
2014-06-12 14:46:44.959452 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] make_writeable e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 snapset=0x72b8d58  snapc=0=[]
2014-06-12 14:46:44.959467 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] make_writeable e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.959478 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] finish_ctx e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 0x29edc000 op modify  
2014-06-12 14:46:44.959496 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  set mtime to 2014-06-12 08:45:47.338430
2014-06-12 14:46:44.959520 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  final snapset 0=[]:[]+head in e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:44.959541 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.959552 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] new_repop rep_tid 236306 on osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.959568 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] issue_repop rep_tid 236306 o e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:44.959595 7f9d6abe6700  5 -- op tracker -- , seq: 747993, time: 2014-06-12 14:46:44.959590, event: waiting for subops from 24, op: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.959618 7f9d6abe6700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:44.959643 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2590 (0'0,173'2590] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] append_log log((0'0,173'2590], crt=0'0) [173'2591 (173'784) modify   e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 by client.4293.0:144208 2014-06-12 08:45:47.338430]
2014-06-12 14:46:44.959668 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] add_log_entry 173'2591 (173'784) modify   e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 by client.4293.0:144208 2014-06-12 08:45:47.338430
2014-06-12 14:46:44.959696 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.959733 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2591, trimmed: 
2014-06-12 14:46:44.959757 7f9d6abe6700  5 queue_transactions existing osr(3.4a7 0x276d8c0)/0x276d8c0
2014-06-12 14:46:44.959761 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.959763 7f9d6abe6700  5 queue_op 0x138cd4a0 seq 2337 osr(3.4a7 0x276d8c0) 174325 bytes   (queue has 1 ops and 174325 bytes)
2014-06-12 14:46:44.959772 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] eval_repop repgather(0xd5e2540 173'2591 rep_tid=236306 committed?=0 applied?=0 lock=0 op=osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.959793 7f9d6abe6700 10 osd.2 173 dequeue_op 0x876ef00 finish
2014-06-12 14:46:44.959797 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.959801 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.959812 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5417b00 (1 active)
2014-06-12 14:46:44.959818 7f9d83173700  5 _do_op 0x138cd4a0 seq 2337 osr(3.4a7 0x276d8c0)/0x276d8c0 start
2014-06-12 14:46:44.959823 7f9d83173700 10 _do_transaction on 0xd5e3ec0
2014-06-12 14:46:44.959830 7f9d83173700 15 _omap_setkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:44.959933 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:44.959957 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.960010 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.960040 7f9d83173700 15 _omap_rmkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:44.960048 7f9d83173700 10 _omap_rmkeys meta/1fe06fa5/pglog_3.4a7/0//-1 r = 0
2014-06-12 14:46:44.960055 7f9d83173700 15 _omap_setkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:44.960222 7f9d83173700 15 _write 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 0~172032
2014-06-12 14:46:44.960401 7f9d83173700 10 lookup_strip_header done  cid 3.4a7_head oid e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:44.960566 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:44.961287 7f9d83173700 10 _generic_write 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 0~172032 = 0
2014-06-12 14:46:44.961316 7f9d83173700 15 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:44.961331 7f9d83173700 10 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 = 0
2014-06-12 14:46:44.961337 7f9d83173700 15 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:44.961346 7f9d83173700 10 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 = 0
2014-06-12 14:46:44.961498 7f9d83173700 20 set_header setting 17968 cid 3.4a7_head oid e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 parent seq 0
2014-06-12 14:46:44.961539 7f9d83173700 20 set_header setting 2078 cid meta oid 1fe06fa5/pglog_3.4a7/0//-1 parent seq 0
2014-06-12 14:46:44.961554 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.961563 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:44.962549 7f9d83173700 10 _do_op 0x138cd4a0 seq 2337 r = 0, finisher 0x89f5420 0x29144ac0
2014-06-12 14:46:44.962572 7f9d83173700 10 _finish_op 0x138cd4a0 seq 2337 osr(3.4a7 0x276d8c0)/0x276d8c0
2014-06-12 14:46:44.962581 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5417b00 (0 active)
2014-06-12 14:46:44.962585 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.962580 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_commit: 236306
2014-06-12 14:46:44.962616 7f9d7715b700  5 -- op tracker -- , seq: 747993, time: 2014-06-12 14:46:44.962605, event: op_commit, op: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.962658 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_applied: 236306
2014-06-12 14:46:44.962684 7f9d7795c700  5 -- op tracker -- , seq: 747993, time: 2014-06-12 14:46:44.962673, event: op_applied, op: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.962692 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_applied on primary on version 173'2591
2014-06-12 14:46:44.972931 7f9d461e0700  5 -- op tracker -- , seq: 747994, time: 2014-06-12 14:46:44.965781, event: header_read, op: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.972968 7f9d461e0700  5 -- op tracker -- , seq: 747994, time: 2014-06-12 14:46:44.965785, event: throttled, op: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.972979 7f9d461e0700  5 -- op tracker -- , seq: 747994, time: 2014-06-12 14:46:44.972821, event: all_read, op: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.973022 7f9d461e0700  5 -- op tracker -- , seq: 747994, time: 0.000000, event: dispatched, op: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.973039 7f9d461e0700 20 osd.2 173 should_share_map client.4293 172.16.96.220:0/1241966 173
2014-06-12 14:46:44.973100 7f9d461e0700 15 osd.2 173 enqueue_op 0x24c9d800 prio 63 cost 2580480 latency 0.007319 osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.975305 7f9d56432700  5 -- op tracker -- , seq: 747995, time: 2014-06-12 14:46:44.972831, event: header_read, op: osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.975325 7f9d56432700  5 -- op tracker -- , seq: 747995, time: 2014-06-12 14:46:44.972835, event: throttled, op: osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.975333 7f9d56432700  5 -- op tracker -- , seq: 747995, time: 2014-06-12 14:46:44.975247, event: all_read, op: osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.975342 7f9d56432700  5 -- op tracker -- , seq: 747995, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.975348 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:44.975356 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:44.975366 7f9d56432700 15 osd.2 173 enqueue_op 0x34017000 prio 127 cost 1549095 latency 0.002535 osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.975412 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.975422 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34017000 prio 127 cost 1549095 latency 0.002591 osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.582( v 173'2762 (0'0,173'2762] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2761 active]
2014-06-12 14:46:44.975448 7f9d6a3e5700  5 -- op tracker -- , seq: 747995, time: 2014-06-12 14:46:44.975443, event: reached_pg, op: osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.975458 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2762 (0'0,173'2762] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2761 active] handle_message: 0x34017000
2014-06-12 14:46:44.975473 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2762 (0'0,173'2762] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2761 active] sub_op_modify trans 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 v 173'2763 (transaction) 174
2014-06-12 14:46:44.975494 7f9d6a3e5700  5 -- op tracker -- , seq: 747995, time: 2014-06-12 14:46:44.975489, event: started, op: osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.975512 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2762 (0'0,173'2762] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2761 active] append_log log((0'0,173'2762], crt=0'0) [173'2763 (173'2762) modify   264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 by client.4317.0:143787 2014-06-12 08:44:58.904725]
2014-06-12 14:46:44.975535 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2763 (0'0,173'2763] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2761 active] add_log_entry 173'2763 (173'2762) modify   264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 by client.4317.0:143787 2014-06-12 08:44:58.904725
2014-06-12 14:46:44.975560 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2763 (0'0,173'2763] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2761 active] append_log  adding 1 keys
2014-06-12 14:46:44.975597 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2763, trimmed: 
2014-06-12 14:46:44.975625 7f9d6a3e5700  5 -- op tracker -- , seq: 747995, time: 2014-06-12 14:46:44.975621, event: started, op: osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.975636 7f9d6a3e5700  5 queue_transactions existing osr(3.582 0x276da70)/0x276da70
2014-06-12 14:46:44.975640 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.975642 7f9d6a3e5700  5 queue_op 0x6cdfda0 seq 2611 osr(3.582 0x276da70) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:46:44.975650 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.582( v 173'2763 (0'0,173'2763] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2761 active] do_sub_op osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.975666 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34017000 finish
2014-06-12 14:46:44.975670 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.975677 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.975682 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x24c9d800 prio 63 cost 2580480 latency 0.009900 osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]
2014-06-12 14:46:44.975708 7f9d6a3e5700  5 -- op tracker -- , seq: 747994, time: 2014-06-12 14:46:44.975703, event: reached_pg, op: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.975716 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.975732 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] handle_message: 0x24c9d800
2014-06-12 14:46:44.975744 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_op osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.975772 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] get_object_context: found obc in cache: 0x20107600
2014-06-12 14:46:44.975782 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] get_object_context: 0x20107600 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 rwstate(write n=1 w=0) oi: e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3(173'2591 client.4293.0:144208 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2591) ssc: 0x72b8d20 snapset: 0=[]:[]+head
2014-06-12 14:46:44.975800 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] find_object_context e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 @head oi=e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3(173'2591 client.4293.0:144208 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2591)
2014-06-12 14:46:44.975830 7f9d6a3e5700  5 -- op tracker -- , seq: 747994, time: 2014-06-12 14:46:44.975824, event: started, op: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.975837 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] execute_ctx 0x589be00
2014-06-12 14:46:44.975856 7f9d6a3e5700  5 -- op tracker -- , seq: 747994, time: 2014-06-12 14:46:44.975850, event: started, op: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.975865 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_op e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] ov 173'2591 av 173'2592 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.975881 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480]
2014-06-12 14:46:44.975895 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.975910 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op  write 172032~2580480
2014-06-12 14:46:44.975929 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op_effects on session 0x5c8cc00
2014-06-12 14:46:44.975943 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] make_writeable e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 snapset=0x72b8d58  snapc=0=[]
2014-06-12 14:46:44.975958 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] make_writeable e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.975971 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] finish_ctx e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 0x589be00 op modify  
2014-06-12 14:46:44.975987 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  set mtime to 2014-06-12 08:45:47.341607
2014-06-12 14:46:44.976010 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  final snapset 0=[]:[]+head in e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:44.976029 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.976042 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] new_repop rep_tid 236307 on osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.976061 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] issue_repop rep_tid 236307 o e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:44.976092 7f9d6a3e5700  5 -- op tracker -- , seq: 747994, time: 2014-06-12 14:46:44.976086, event: waiting for subops from 24, op: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.976245 7f9d6a3e5700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:44.976273 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2591 (0'0,173'2591] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] append_log log((0'0,173'2591], crt=0'0) [173'2592 (173'2591) modify   e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 by client.4293.0:144209 2014-06-12 08:45:47.341607]
2014-06-12 14:46:44.976299 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] add_log_entry 173'2592 (173'2591) modify   e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 by client.4293.0:144209 2014-06-12 08:45:47.341607
2014-06-12 14:46:44.976318 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.976346 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2592, trimmed: 
2014-06-12 14:46:44.976363 7f9d6a3e5700  5 queue_transactions existing osr(3.4a7 0x276d8c0)/0x276d8c0
2014-06-12 14:46:44.976366 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.976367 7f9d6a3e5700  5 queue_op 0x5008540 seq 2338 osr(3.4a7 0x276d8c0) 2582773 bytes   (queue has 2 ops and 4133354 bytes)
2014-06-12 14:46:44.976373 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] eval_repop repgather(0x2f89f140 173'2592 rep_tid=236307 committed?=0 applied?=0 lock=0 op=osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.976390 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x24c9d800 finish
2014-06-12 14:46:44.976394 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.976397 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.976405 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438900 (1 active)
2014-06-12 14:46:44.976410 7f9d82972700  5 _do_op 0x6cdfda0 seq 2611 osr(3.582 0x276da70)/0x276da70 start
2014-06-12 14:46:44.976414 7f9d82972700 10 _do_transaction on 0x2721270
2014-06-12 14:46:44.976420 7f9d82972700 15 _omap_setkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:44.976546 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:44.976574 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.976619 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.976633 7f9d82972700 15 _omap_rmkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:44.976638 7f9d82972700 10 _omap_rmkeys meta/1fe177ea/pglog_3.582/0//-1 r = 0
2014-06-12 14:46:44.976643 7f9d82972700 15 _omap_setkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:44.976837 7f9d82972700 15 _write 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 2551808~1548288
2014-06-12 14:46:44.976994 7f9d82972700 10 lookup_strip_header done  cid 3.582_head oid 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:44.977200 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:44.977637 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.977974 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5417b00 (2 active)
2014-06-12 14:46:44.977984 7f9d81970700  5 _do_op 0x5008540 seq 2338 osr(3.4a7 0x276d8c0)/0x276d8c0 start
2014-06-12 14:46:44.977990 7f9d81970700 10 _do_transaction on 0x26f0d80
2014-06-12 14:46:44.977997 7f9d81970700 15 _omap_setkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:44.978109 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:44.978138 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.983713 7f9d82972700 10 _generic_write 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 2551808~1548288 = 0
2014-06-12 14:46:44.983885 7f9d82972700 15 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:44.983911 7f9d82972700 10 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:44.983920 7f9d82972700 15 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:44.983929 7f9d82972700 10 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:44.984297 7f9d82972700 20 set_header setting 16739 cid 3.582_head oid 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 parent seq 0
2014-06-12 14:46:44.984418 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.984436 7f9d82972700 20 set_header setting 2318 cid meta oid 1fe177ea/pglog_3.582/0//-1 parent seq 0
2014-06-12 14:46:44.984446 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:44.988481 7f9d47c58700  5 -- op tracker -- , seq: 747996, time: 2014-06-12 14:46:44.987377, event: header_read, op: osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.988500 7f9d47c58700  5 -- op tracker -- , seq: 747996, time: 2014-06-12 14:46:44.987380, event: throttled, op: osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.988510 7f9d47c58700  5 -- op tracker -- , seq: 747996, time: 2014-06-12 14:46:44.988426, event: all_read, op: osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.988516 7f9d47c58700  5 -- op tracker -- , seq: 747996, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.988525 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:44.988556 7f9d47c58700 15 osd.2 173 enqueue_op 0x1a16e900 prio 63 cost 458752 latency 0.001179 osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.988602 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:44.988619 7f9d653db700 10 osd.2 173 dequeue_op 0x1a16e900 prio 63 cost 458752 latency 0.001241 osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]
2014-06-12 14:46:44.988653 7f9d653db700  5 -- op tracker -- , seq: 747996, time: 2014-06-12 14:46:44.988649, event: reached_pg, op: osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.988661 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:44.988674 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] handle_message: 0x1a16e900
2014-06-12 14:46:44.988682 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_op osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:44.988704 7f9d653db700 15 getattr 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 '_'
2014-06-12 14:46:44.988920 7f9d653db700 10 getattr 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 '_' = 0
2014-06-12 14:46:44.988940 7f9d653db700 15 getattr 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 'snapset'
2014-06-12 14:46:44.988994 7f9d653db700 10 getattr 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 'snapset' = 0
2014-06-12 14:46:44.989004 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] populate_obc_watchers 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:44.989013 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:44.989019 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] get_object_context: creating obc from disk: 0x120e98c0
2014-06-12 14:46:44.989025 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] get_object_context: 0x120e98c0 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 rwstate(none n=0 w=0) oi: 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'757 client.4316.0:41698 wrlock_by=unknown.0.0:0 dirty s 4194304 uv757) ssc: 0x1e15e460 snapset: 0=[]:[]+head
2014-06-12 14:46:44.989036 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] find_object_context 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 @head oi=91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'757 client.4316.0:41698 wrlock_by=unknown.0.0:0 dirty s 4194304 uv757)
2014-06-12 14:46:44.989055 7f9d653db700  5 -- op tracker -- , seq: 747996, time: 2014-06-12 14:46:44.989051, event: started, op: osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.989062 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] execute_ctx 0x2963a600
2014-06-12 14:46:44.989074 7f9d653db700  5 -- op tracker -- , seq: 747996, time: 2014-06-12 14:46:44.989071, event: started, op: osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.989079 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] ov 173'757 av 173'2712 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:44.989089 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_osd_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752]
2014-06-12 14:46:44.989098 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:44.989111 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_osd_op  write 0~458752
2014-06-12 14:46:44.989128 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:44.989136 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 snapset=0x1e15e498  snapc=0=[]
2014-06-12 14:46:44.989146 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:44.989152 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] finish_ctx 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 0x2963a600 op modify  
2014-06-12 14:46:44.989162 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]  set mtime to 2014-06-12 08:45:47.367029
2014-06-12 14:46:44.989178 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]  final snapset 0=[]:[]+head in 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:44.989193 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]  zeroing write result code 0
2014-06-12 14:46:44.989201 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] new_repop rep_tid 236308 on osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:44.989215 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] issue_repop rep_tid 236308 o 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:44.989241 7f9d653db700  5 -- op tracker -- , seq: 747996, time: 2014-06-12 14:46:44.989237, event: waiting for subops from 23, op: osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.989269 7f9d653db700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:44.989296 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2711 (0'0,173'2711] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] append_log log((0'0,173'2711], crt=0'0) [173'2712 (173'757) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147245 2014-06-12 08:45:47.367029]
2014-06-12 14:46:44.989324 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] add_log_entry 173'2712 (173'757) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147245 2014-06-12 08:45:47.367029
2014-06-12 14:46:44.989349 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] append_log  adding 1 keys
2014-06-12 14:46:44.989389 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2712, trimmed: 
2014-06-12 14:46:44.989414 7f9d653db700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:44.989421 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:44.989423 7f9d653db700  5 queue_op 0x6de00c0 seq 2457 osr(3.54b 0x276db30) 461045 bytes   (queue has 3 ops and 4594399 bytes)
2014-06-12 14:46:44.989430 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] eval_repop repgather(0x2010e480 173'2712 rep_tid=236308 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:44.989449 7f9d653db700 10 osd.2 173 dequeue_op 0x1a16e900 finish
2014-06-12 14:46:44.989453 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:44.989457 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:44.989957 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (3 active)
2014-06-12 14:46:44.989968 7f9d7c966700  5 _do_op 0x6de00c0 seq 2457 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:46:44.989973 7f9d7c966700 10 _do_transaction on 0x514db00
2014-06-12 14:46:44.989979 7f9d7c966700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:44.990134 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:44.990159 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:44.992167 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.992212 7f9d7c966700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:44.992223 7f9d7c966700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:46:44.992232 7f9d7c966700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:44.992250 7f9d7c966700 15 _write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 0~458752
2014-06-12 14:46:44.992394 7f9d7c966700 10 lookup_strip_header done  cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:44.992562 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:44.992766 7f9d82972700 10 _do_op 0x6cdfda0 seq 2611 r = 0, finisher 0x66b76e0 0
2014-06-12 14:46:44.992795 7f9d7715b700  5 -- op tracker -- , seq: 747995, time: 2014-06-12 14:46:44.992787, event: commit_sent, op: osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.992805 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2763 (0'0,173'2763] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2761 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:44.992830 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:44.992880 7f9d82972700 10 _finish_op 0x6cdfda0 seq 2611 osr(3.582 0x276da70)/0x276da70
2014-06-12 14:46:44.992891 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438900 (2 active)
2014-06-12 14:46:44.992896 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.992909 7f9d7795c700  5 -- op tracker -- , seq: 747995, time: 2014-06-12 14:46:44.992902, event: sub_op_applied, op: osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.992919 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2763 (0'0,173'2763] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2762 active] sub_op_modify_applied on 0x2721180 op osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:44.992941 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2763 (0'0,173'2763] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2762 active] op_applied on primary on version 173'2763
2014-06-12 14:46:44.992948 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2763 (0'0,173'2763] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2762 active] op_applied on replica on version 173'2763
2014-06-12 14:46:44.992970 7f9d7795c700  5 -- op tracker -- , seq: 747995, time: 2014-06-12 14:46:44.992968, event: done, op: osd_sub_op(client.4317.0:143787 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2763 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:44.994551 7f9d7c966700 10 _generic_write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 0~458752 = 0
2014-06-12 14:46:44.994623 7f9d7c966700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:44.994644 7f9d7c966700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:44.994652 7f9d7c966700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:44.994661 7f9d7c966700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:44.994905 7f9d7c966700 20 set_header setting 17437 cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 parent seq 0
2014-06-12 14:46:44.994956 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:44.994968 7f9d7c966700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:46:44.994974 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:44.997200 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:44.997202 7f9d7c966700 10 _do_op 0x6de00c0 seq 2457 r = 0, finisher 0x2095980 0x8820a40
2014-06-12 14:46:44.997224 7f9d7c966700 10 _finish_op 0x6de00c0 seq 2457 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:44.997236 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (1 active)
2014-06-12 14:46:44.997244 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:44.997233 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] op_commit: 236308
2014-06-12 14:46:44.997255 7f9d81970700 15 _omap_rmkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:44.997262 7f9d81970700 10 _omap_rmkeys meta/1fe06fa5/pglog_3.4a7/0//-1 r = 0
2014-06-12 14:46:44.997269 7f9d81970700 15 _omap_setkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:44.997264 7f9d7715b700  5 -- op tracker -- , seq: 747996, time: 2014-06-12 14:46:44.997254, event: op_commit, op: osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.997289 7f9d81970700 15 _write 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 172032~2580480
2014-06-12 14:46:44.997285 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] op_applied: 236308
2014-06-12 14:46:44.997304 7f9d7795c700  5 -- op tracker -- , seq: 747996, time: 2014-06-12 14:46:44.997299, event: op_applied, op: osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:44.997312 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] op_applied on primary on version 173'2712
2014-06-12 14:46:44.997453 7f9d81970700 10 lookup_strip_header done  cid 3.4a7_head oid e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:44.997739 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:45.001478 7f9d47c58700  5 -- op tracker -- , seq: 747997, time: 2014-06-12 14:46:44.999721, event: header_read, op: osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.001493 7f9d47c58700  5 -- op tracker -- , seq: 747997, time: 2014-06-12 14:46:44.999728, event: throttled, op: osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.001502 7f9d47c58700  5 -- op tracker -- , seq: 747997, time: 2014-06-12 14:46:45.001425, event: all_read, op: osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.001510 7f9d47c58700  5 -- op tracker -- , seq: 747997, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.001520 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:45.001554 7f9d47c58700 15 osd.2 173 enqueue_op 0x1a16e800 prio 63 cost 516096 latency 0.001833 osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.001602 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.001612 7f9d63bd8700 10 osd.2 173 dequeue_op 0x1a16e800 prio 63 cost 516096 latency 0.001890 osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]
2014-06-12 14:46:45.001650 7f9d63bd8700  5 -- op tracker -- , seq: 747997, time: 2014-06-12 14:46:45.001645, event: reached_pg, op: osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.001658 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.001673 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] handle_message: 0x1a16e800
2014-06-12 14:46:45.001683 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_op osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.001709 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] get_object_context: found obc in cache: 0x120e98c0
2014-06-12 14:46:45.001719 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] get_object_context: 0x120e98c0 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 rwstate(write n=1 w=0) oi: 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2712 client.4316.0:147245 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2712) ssc: 0x1e15e460 snapset: 0=[]:[]+head
2014-06-12 14:46:45.001735 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] find_object_context 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 @head oi=91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2712 client.4316.0:147245 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2712)
2014-06-12 14:46:45.001759 7f9d63bd8700  5 -- op tracker -- , seq: 747997, time: 2014-06-12 14:46:45.001754, event: started, op: osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.001767 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] execute_ctx 0xfd1aa00
2014-06-12 14:46:45.001784 7f9d63bd8700  5 -- op tracker -- , seq: 747997, time: 2014-06-12 14:46:45.001779, event: started, op: osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.001790 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] ov 173'2712 av 173'2713 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.001804 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_osd_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096]
2014-06-12 14:46:45.001815 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.001831 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_osd_op  write 458752~516096
2014-06-12 14:46:45.001847 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:45.001859 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 snapset=0x1e15e498  snapc=0=[]
2014-06-12 14:46:45.001872 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.001882 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] finish_ctx 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 0xfd1aa00 op modify  
2014-06-12 14:46:45.001895 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]  set mtime to 2014-06-12 08:45:47.379235
2014-06-12 14:46:45.001920 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]  final snapset 0=[]:[]+head in 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.001941 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.001953 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] new_repop rep_tid 236309 on osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.001971 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] issue_repop rep_tid 236309 o 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.002000 7f9d63bd8700  5 -- op tracker -- , seq: 747997, time: 2014-06-12 14:46:45.001995, event: waiting for subops from 23, op: osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.002027 7f9d63bd8700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:45.002055 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2712 (0'0,173'2712] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] append_log log((0'0,173'2712], crt=0'0) [173'2713 (173'2712) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147246 2014-06-12 08:45:47.379235]
2014-06-12 14:46:45.002083 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] add_log_entry 173'2713 (173'2712) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147246 2014-06-12 08:45:47.379235
2014-06-12 14:46:45.002105 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.002138 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2713, trimmed: 
2014-06-12 14:46:45.002164 7f9d63bd8700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.002172 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.002174 7f9d63bd8700  5 queue_op 0x283b9e00 seq 2458 osr(3.54b 0x276db30) 518389 bytes   (queue has 2 ops and 3101162 bytes)
2014-06-12 14:46:45.002186 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] eval_repop repgather(0xd5e3bc0 173'2713 rep_tid=236309 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.002206 7f9d63bd8700 10 osd.2 173 dequeue_op 0x1a16e800 finish
2014-06-12 14:46:45.002210 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.002214 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.005642 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (2 active)
2014-06-12 14:46:45.005671 7f9d7a962700  5 _do_op 0x283b9e00 seq 2458 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:46:45.005680 7f9d7a962700 10 _do_transaction on 0x2680480
2014-06-12 14:46:45.005690 7f9d7a962700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.005882 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.005923 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.010883 7f9d81970700 10 _generic_write 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 172032~2580480 = 0
2014-06-12 14:46:45.011169 7f9d81970700 15 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.011184 7f9d81970700 10 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 = 0
2014-06-12 14:46:45.011189 7f9d81970700 15 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.011198 7f9d81970700 10 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 = 0
2014-06-12 14:46:45.011871 7f9d81970700 20 set_header setting 17968 cid 3.4a7_head oid e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 parent seq 0
2014-06-12 14:46:45.012195 7f9d81970700 20 set_header setting 2078 cid meta oid 1fe06fa5/pglog_3.4a7/0//-1 parent seq 0
2014-06-12 14:46:45.012213 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.012221 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:45.012978 7f9d47c58700  5 -- op tracker -- , seq: 747998, time: 2014-06-12 14:46:45.011408, event: header_read, op: osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.012992 7f9d47c58700  5 -- op tracker -- , seq: 747998, time: 2014-06-12 14:46:45.011412, event: throttled, op: osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.013002 7f9d47c58700  5 -- op tracker -- , seq: 747998, time: 2014-06-12 14:46:45.012926, event: all_read, op: osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.013011 7f9d47c58700  5 -- op tracker -- , seq: 747998, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.013022 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:45.013056 7f9d47c58700 15 osd.2 173 enqueue_op 0x3850ea00 prio 63 cost 516096 latency 0.001648 osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.013097 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.013105 7f9d68be2700 10 osd.2 173 dequeue_op 0x3850ea00 prio 63 cost 516096 latency 0.001696 osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]
2014-06-12 14:46:45.013131 7f9d68be2700  5 -- op tracker -- , seq: 747998, time: 2014-06-12 14:46:45.013126, event: reached_pg, op: osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.013140 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.013158 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] handle_message: 0x3850ea00
2014-06-12 14:46:45.013169 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_op osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.013195 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] get_object_context: found obc in cache: 0x120e98c0
2014-06-12 14:46:45.013205 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] get_object_context: 0x120e98c0 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 rwstate(write n=2 w=0) oi: 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2713 client.4316.0:147246 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2713) ssc: 0x1e15e460 snapset: 0=[]:[]+head
2014-06-12 14:46:45.013222 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] find_object_context 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 @head oi=91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2713 client.4316.0:147246 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2713)
2014-06-12 14:46:45.013249 7f9d68be2700  5 -- op tracker -- , seq: 747998, time: 2014-06-12 14:46:45.013244, event: started, op: osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.013256 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] execute_ctx 0x580d000
2014-06-12 14:46:45.013273 7f9d68be2700  5 -- op tracker -- , seq: 747998, time: 2014-06-12 14:46:45.013268, event: started, op: osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.013286 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] ov 173'2713 av 173'2714 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.013300 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_osd_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096]
2014-06-12 14:46:45.013310 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.013323 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_osd_op  write 974848~516096
2014-06-12 14:46:45.013337 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:45.013348 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 snapset=0x1e15e498  snapc=0=[]
2014-06-12 14:46:45.013360 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.013370 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] finish_ctx 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 0x580d000 op modify  
2014-06-12 14:46:45.013383 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]  set mtime to 2014-06-12 08:45:47.390955
2014-06-12 14:46:45.013408 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]  final snapset 0=[]:[]+head in 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.013437 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.013454 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] new_repop rep_tid 236310 on osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.013482 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] issue_repop rep_tid 236310 o 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.013529 7f9d68be2700  5 -- op tracker -- , seq: 747998, time: 2014-06-12 14:46:45.013520, event: waiting for subops from 23, op: osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.013567 7f9d68be2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:45.013595 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2713 (0'0,173'2713] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] append_log log((0'0,173'2713], crt=0'0) [173'2714 (173'2713) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147247 2014-06-12 08:45:47.390955]
2014-06-12 14:46:45.013632 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] add_log_entry 173'2714 (173'2713) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147247 2014-06-12 08:45:47.390955
2014-06-12 14:46:45.013655 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.013690 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2714, trimmed: 
2014-06-12 14:46:45.013713 7f9d68be2700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.013719 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.013727 7f9d68be2700  5 queue_op 0x70c45a0 seq 2459 osr(3.54b 0x276db30) 518389 bytes   (queue has 3 ops and 3619551 bytes)
2014-06-12 14:46:45.013737 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] eval_repop repgather(0x1ced2b40 173'2714 rep_tid=236310 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.013757 7f9d68be2700 10 osd.2 173 dequeue_op 0x3850ea00 finish
2014-06-12 14:46:45.013761 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.013765 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.013790 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (3 active)
2014-06-12 14:46:45.021401 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:45.021513 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:45.027992 7f9d57543700  5 -- op tracker -- , seq: 747999, time: 2014-06-12 14:46:45.027925, event: header_read, op: osd_sub_op_reply(client.4316.0:147245 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.028011 7f9d57543700  5 -- op tracker -- , seq: 747999, time: 2014-06-12 14:46:45.027927, event: throttled, op: osd_sub_op_reply(client.4316.0:147245 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.028020 7f9d57543700  5 -- op tracker -- , seq: 747999, time: 2014-06-12 14:46:45.027944, event: all_read, op: osd_sub_op_reply(client.4316.0:147245 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.028028 7f9d57543700  5 -- op tracker -- , seq: 747999, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147245 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.028034 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147245 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:45.028041 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:45.028051 7f9d57543700 15 osd.2 173 enqueue_op 0x22b3b400 prio 196 cost 0 latency 0.000125 osd_sub_op_reply(client.4316.0:147245 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:45.028349 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.028361 7f9d623d5700 10 osd.2 173 dequeue_op 0x22b3b400 prio 196 cost 0 latency 0.000435 osd_sub_op_reply(client.4316.0:147245 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]
2014-06-12 14:46:45.028383 7f9d623d5700  5 -- op tracker -- , seq: 747999, time: 2014-06-12 14:46:45.028379, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147245 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.028392 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] handle_message: 0x22b3b400
2014-06-12 14:46:45.028407 7f9d623d5700  5 -- op tracker -- , seq: 747999, time: 2014-06-12 14:46:45.028404, event: started, op: osd_sub_op_reply(client.4316.0:147245 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.028414 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] sub_op_modify_reply: tid 236308 op  ack_type 4 from 23
2014-06-12 14:46:45.028434 7f9d623d5700  5 -- op tracker -- , seq: 747996, time: 2014-06-12 14:46:45.028427, event: sub_op_commit_rec, op: osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.028442 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] repop_all_applied: repop tid 236308 all applied 
2014-06-12 14:46:45.028452 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] eval_repop repgather(0x2010e480 173'2712 rep_tid=236308 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.028470 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean]  sending ack on repgather(0x2010e480 173'2712 rep_tid=236308 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) 0x6d1d680
2014-06-12 14:46:45.028503 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2711 crt=0'0 lcod 173'2710 mlcod 173'2710 active+clean] repop_all_committed: repop tid 236308 all committed 
2014-06-12 14:46:45.028514 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2710 active+clean] eval_repop repgather(0x2010e480 173'2712 rep_tid=236308 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.028531 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2710 active+clean] log_op_stats osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 459031 outb 0 rlat 0.041123 lat 0.041152
2014-06-12 14:46:45.028549 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2710 active+clean] publish_stats_to_osd 173:3607
2014-06-12 14:46:45.028563 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2710 active+clean]  sending commit on repgather(0x2010e480 173'2712 rep_tid=236308 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) 0x17433b80
2014-06-12 14:46:45.028593 7f9d623d5700  5 -- op tracker -- , seq: 747996, time: 2014-06-12 14:46:45.028588, event: commit_sent, op: osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.028601 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2710 active+clean]  requeue_ops 
2014-06-12 14:46:45.028611 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2711 active+clean]  removing repgather(0x2010e480 173'2712 rep_tid=236308 committed?=1 applied?=1 op=osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.028626 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2711 active+clean]    q front is repgather(0x2010e480 173'2712 rep_tid=236308 committed?=1 applied?=1 op=osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.028640 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2711 active+clean] remove_repop repgather(0x2010e480 173'2712 rep_tid=236308 committed?=1 applied?=1 op=osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.028654 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2711 active+clean]  requeue_ops 
2014-06-12 14:46:45.028673 7f9d623d5700  5 -- op tracker -- , seq: 747996, time: 2014-06-12 14:46:45.028668, event: done, op: osd_op(client.4316.0:147245 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.028694 7f9d623d5700 10 osd.2 173 dequeue_op 0x22b3b400 finish
2014-06-12 14:46:45.028701 7f9d623d5700  5 -- op tracker -- , seq: 747999, time: 2014-06-12 14:46:45.028698, event: done, op: osd_sub_op_reply(client.4316.0:147245 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.028712 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.028715 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.032634 7f9d57543700  5 -- op tracker -- , seq: 748000, time: 2014-06-12 14:46:45.032577, event: header_read, op: osd_sub_op_reply(client.4316.0:147246 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.032644 7f9d57543700  5 -- op tracker -- , seq: 748000, time: 2014-06-12 14:46:45.032579, event: throttled, op: osd_sub_op_reply(client.4316.0:147246 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.032653 7f9d57543700  5 -- op tracker -- , seq: 748000, time: 2014-06-12 14:46:45.032598, event: all_read, op: osd_sub_op_reply(client.4316.0:147246 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.032661 7f9d57543700  5 -- op tracker -- , seq: 748000, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147246 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.032665 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147246 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:45.032671 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:45.032679 7f9d57543700 15 osd.2 173 enqueue_op 0x2a902200 prio 196 cost 0 latency 0.000102 osd_sub_op_reply(client.4316.0:147246 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:45.032705 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.032713 7f9d643d9700 10 osd.2 173 dequeue_op 0x2a902200 prio 196 cost 0 latency 0.000136 osd_sub_op_reply(client.4316.0:147246 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2711 active+clean]
2014-06-12 14:46:45.032740 7f9d643d9700  5 -- op tracker -- , seq: 748000, time: 2014-06-12 14:46:45.032736, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147246 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.032748 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2711 active+clean] handle_message: 0x2a902200
2014-06-12 14:46:45.032764 7f9d643d9700  5 -- op tracker -- , seq: 748000, time: 2014-06-12 14:46:45.032761, event: started, op: osd_sub_op_reply(client.4316.0:147246 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.032773 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2711 active+clean] sub_op_modify_reply: tid 236309 op  ack_type 4 from 23
2014-06-12 14:46:45.032793 7f9d643d9700  5 -- op tracker -- , seq: 747997, time: 2014-06-12 14:46:45.032786, event: sub_op_commit_rec, op: osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.032801 7f9d643d9700 10 osd.2 173 dequeue_op 0x2a902200 finish
2014-06-12 14:46:45.032807 7f9d643d9700  5 -- op tracker -- , seq: 748000, time: 2014-06-12 14:46:45.032804, event: done, op: osd_sub_op_reply(client.4316.0:147246 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.032819 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.032822 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.034915 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.034955 7f9d7a962700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.034959 7f9d7a962700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:46:45.034965 7f9d7a962700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.034978 7f9d7a962700 15 _write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 458752~516096
2014-06-12 14:46:45.035511 7f9d7a962700 10 lookup_strip_header done  cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.035542 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:45.037188 7f9d7a962700 10 _generic_write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 458752~516096 = 0
2014-06-12 14:46:45.037226 7f9d7a962700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.037247 7f9d7a962700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.037251 7f9d7a962700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.037256 7f9d7a962700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.037300 7f9d57543700  5 -- op tracker -- , seq: 748001, time: 2014-06-12 14:46:45.037249, event: header_read, op: osd_sub_op_reply(client.4316.0:147247 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.037312 7f9d57543700  5 -- op tracker -- , seq: 748001, time: 2014-06-12 14:46:45.037251, event: throttled, op: osd_sub_op_reply(client.4316.0:147247 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.037322 7f9d57543700  5 -- op tracker -- , seq: 748001, time: 2014-06-12 14:46:45.037265, event: all_read, op: osd_sub_op_reply(client.4316.0:147247 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.037332 7f9d57543700  5 -- op tracker -- , seq: 748001, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147247 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.037342 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147247 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:45.037347 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:45.037355 7f9d57543700 15 osd.2 173 enqueue_op 0x8dbd000 prio 196 cost 0 latency 0.000105 osd_sub_op_reply(client.4316.0:147247 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:45.037387 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.037404 7f9d7a962700 20 set_header setting 17437 cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 parent seq 0
2014-06-12 14:46:45.037396 7f9d633d7700 10 osd.2 173 dequeue_op 0x8dbd000 prio 196 cost 0 latency 0.000146 osd_sub_op_reply(client.4316.0:147247 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2711 active+clean]
2014-06-12 14:46:45.037419 7f9d633d7700  5 -- op tracker -- , seq: 748001, time: 2014-06-12 14:46:45.037415, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147247 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.037442 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.037451 7f9d7a962700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:46:45.037446 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2711 active+clean] handle_message: 0x8dbd000
2014-06-12 14:46:45.037457 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:45.037462 7f9d633d7700  5 -- op tracker -- , seq: 748001, time: 2014-06-12 14:46:45.037459, event: started, op: osd_sub_op_reply(client.4316.0:147247 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.037468 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2711 active+clean] sub_op_modify_reply: tid 236310 op  ack_type 4 from 23
2014-06-12 14:46:45.037489 7f9d633d7700  5 -- op tracker -- , seq: 747998, time: 2014-06-12 14:46:45.037481, event: sub_op_commit_rec, op: osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.037496 7f9d633d7700 10 osd.2 173 dequeue_op 0x8dbd000 finish
2014-06-12 14:46:45.037503 7f9d633d7700  5 -- op tracker -- , seq: 748001, time: 2014-06-12 14:46:45.037500, event: done, op: osd_sub_op_reply(client.4316.0:147247 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.037516 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.037519 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.037968 7f9d81970700 10 _do_op 0x5008540 seq 2338 r = 0, finisher 0x73303a0 0x169648c0
2014-06-12 14:46:45.037985 7f9d81970700 10 _finish_op 0x5008540 seq 2338 osr(3.4a7 0x276d8c0)/0x276d8c0
2014-06-12 14:46:45.037993 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5417b00 (2 active)
2014-06-12 14:46:45.038002 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.038000 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_commit: 236307
2014-06-12 14:46:45.038027 7f9d7715b700  5 -- op tracker -- , seq: 747994, time: 2014-06-12 14:46:45.038019, event: op_commit, op: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.038044 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_applied: 236307
2014-06-12 14:46:45.038065 7f9d7795c700  5 -- op tracker -- , seq: 747994, time: 2014-06-12 14:46:45.038059, event: op_applied, op: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.038080 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_applied on primary on version 173'2592
2014-06-12 14:46:45.039467 7f9d7a962700 10 _do_op 0x283b9e00 seq 2458 r = 0, finisher 0x660b640 0x187b98c0
2014-06-12 14:46:45.039489 7f9d7a962700 10 _finish_op 0x283b9e00 seq 2458 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.039506 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (1 active)
2014-06-12 14:46:45.039514 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.039513 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2711 active+clean] op_commit: 236309
2014-06-12 14:46:45.039545 7f9d7715b700  5 -- op tracker -- , seq: 747997, time: 2014-06-12 14:46:45.039535, event: op_commit, op: osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.039561 7f9d7895e700  5 _do_op 0x70c45a0 seq 2459 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:46:45.039568 7f9d7895e700 10 _do_transaction on 0xeaadd40
2014-06-12 14:46:45.039564 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2712 crt=0'0 lcod 173'2711 mlcod 173'2711 active+clean] repop_all_committed: repop tid 236309 all committed 
2014-06-12 14:46:45.039576 7f9d7895e700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.039577 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2711 active+clean] eval_repop repgather(0xd5e3bc0 173'2713 rep_tid=236309 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.039595 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2711 active+clean] log_op_stats osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.039873
2014-06-12 14:46:45.039614 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2711 active+clean] publish_stats_to_osd 173:3608
2014-06-12 14:46:45.039626 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2711 active+clean]  sending commit on repgather(0xd5e3bc0 173'2713 rep_tid=236309 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) 0x21ebb400
2014-06-12 14:46:45.039665 7f9d7715b700  5 -- op tracker -- , seq: 747997, time: 2014-06-12 14:46:45.039660, event: commit_sent, op: osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.039686 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2711 active+clean] op_applied: 236309
2014-06-12 14:46:45.039706 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.039704 7f9d7795c700  5 -- op tracker -- , seq: 747997, time: 2014-06-12 14:46:45.039699, event: op_applied, op: osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.039711 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2711 active+clean] op_applied on primary on version 173'2713
2014-06-12 14:46:45.039725 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.039721 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2711 active+clean] repop_all_applied: repop tid 236309 all applied 
2014-06-12 14:46:45.039731 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2711 active+clean] eval_repop repgather(0xd5e3bc0 173'2713 rep_tid=236309 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.039746 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2711 active+clean] log_op_stats osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.040024
2014-06-12 14:46:45.039765 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2711 active+clean] publish_stats_to_osd 173:3609
2014-06-12 14:46:45.039776 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.039776 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2711 active+clean]  requeue_ops 
2014-06-12 14:46:45.039796 7f9d7895e700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.039786 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean]  removing repgather(0xd5e3bc0 173'2713 rep_tid=236309 committed?=1 applied?=1 op=osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.039803 7f9d7895e700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:46:45.039809 7f9d7895e700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.039802 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean]    q front is repgather(0xd5e3bc0 173'2713 rep_tid=236309 committed?=1 applied?=1 op=osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.039824 7f9d7895e700 15 _write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 974848~516096
2014-06-12 14:46:45.039816 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] remove_repop repgather(0xd5e3bc0 173'2713 rep_tid=236309 committed?=1 applied?=1 op=osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.039829 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean]  requeue_ops 
2014-06-12 14:46:45.039849 7f9d7795c700  5 -- op tracker -- , seq: 747997, time: 2014-06-12 14:46:45.039844, event: done, op: osd_op(client.4316.0:147246 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.039955 7f9d7895e700 10 lookup_strip_header done  cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.040136 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:45.041567 7f9d7895e700 10 _generic_write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 974848~516096 = 0
2014-06-12 14:46:45.041893 7f9d7895e700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.041915 7f9d7895e700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.041923 7f9d7895e700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.041931 7f9d7895e700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.042111 7f9d7895e700 20 set_header setting 17437 cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 parent seq 0
2014-06-12 14:46:45.042173 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.042183 7f9d7895e700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:46:45.042189 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:45.043215 7f9d47c58700  5 -- op tracker -- , seq: 748002, time: 2014-06-12 14:46:45.041897, event: header_read, op: osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.043229 7f9d47c58700  5 -- op tracker -- , seq: 748002, time: 2014-06-12 14:46:45.041901, event: throttled, op: osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.043238 7f9d47c58700  5 -- op tracker -- , seq: 748002, time: 2014-06-12 14:46:45.043168, event: all_read, op: osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.043247 7f9d47c58700  5 -- op tracker -- , seq: 748002, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.043256 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:45.043284 7f9d47c58700 15 osd.2 173 enqueue_op 0x3850f600 prio 63 cost 516096 latency 0.001386 osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.043324 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.043335 7f9d61bd4700 10 osd.2 173 dequeue_op 0x3850f600 prio 63 cost 516096 latency 0.001437 osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean]
2014-06-12 14:46:45.043363 7f9d61bd4700  5 -- op tracker -- , seq: 748002, time: 2014-06-12 14:46:45.043358, event: reached_pg, op: osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.043374 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.043390 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] handle_message: 0x3850f600
2014-06-12 14:46:45.043401 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] do_op osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.043428 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] get_object_context: found obc in cache: 0x120e98c0
2014-06-12 14:46:45.043440 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] get_object_context: 0x120e98c0 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 rwstate(write n=1 w=0) oi: 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2714 client.4316.0:147247 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2714) ssc: 0x1e15e460 snapset: 0=[]:[]+head
2014-06-12 14:46:45.043455 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] find_object_context 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 @head oi=91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2714 client.4316.0:147247 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2714)
2014-06-12 14:46:45.043489 7f9d61bd4700  5 -- op tracker -- , seq: 748002, time: 2014-06-12 14:46:45.043483, event: started, op: osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.043502 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] execute_ctx 0x580ca00
2014-06-12 14:46:45.043519 7f9d61bd4700  5 -- op tracker -- , seq: 748002, time: 2014-06-12 14:46:45.043514, event: started, op: osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.043526 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] do_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] ov 173'2714 av 173'2715 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.043540 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] do_osd_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096]
2014-06-12 14:46:45.043551 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.043565 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] do_osd_op  write 1490944~516096
2014-06-12 14:46:45.043580 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:45.043591 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 snapset=0x1e15e498  snapc=0=[]
2014-06-12 14:46:45.043604 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.043614 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] finish_ctx 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 0x580ca00 op modify  
2014-06-12 14:46:45.043627 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean]  set mtime to 2014-06-12 08:45:47.421364
2014-06-12 14:46:45.043649 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean]  final snapset 0=[]:[]+head in 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.043667 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.043681 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] new_repop rep_tid 236311 on osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.043703 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] issue_repop rep_tid 236311 o 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.043731 7f9d61bd4700  5 -- op tracker -- , seq: 748002, time: 2014-06-12 14:46:45.043725, event: waiting for subops from 23, op: osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.043763 7f9d61bd4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:45.043795 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2714 (0'0,173'2714] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] append_log log((0'0,173'2714], crt=0'0) [173'2715 (173'2714) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147248 2014-06-12 08:45:47.421364]
2014-06-12 14:46:45.043826 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] add_log_entry 173'2715 (173'2714) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147248 2014-06-12 08:45:47.421364
2014-06-12 14:46:45.043849 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.043888 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2715, trimmed: 
2014-06-12 14:46:45.043923 7f9d61bd4700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.043930 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.043933 7f9d61bd4700  5 queue_op 0x24c67080 seq 2460 osr(3.54b 0x276db30) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:45.043947 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] eval_repop repgather(0x32ff4600 173'2715 rep_tid=236311 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.043970 7f9d61bd4700 10 osd.2 173 dequeue_op 0x3850f600 finish
2014-06-12 14:46:45.043976 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.043980 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.043997 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (2 active)
2014-06-12 14:46:45.045713 7f9d7895e700 10 _do_op 0x70c45a0 seq 2459 r = 0, finisher 0x8d89ee0 0x12d187c0
2014-06-12 14:46:45.045731 7f9d7895e700 10 _finish_op 0x70c45a0 seq 2459 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.045736 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] op_commit: 236310
2014-06-12 14:46:45.045750 7f9d7a161700  5 _do_op 0x24c67080 seq 2460 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:46:45.045758 7f9d7a161700 10 _do_transaction on 0x32ff5ec0
2014-06-12 14:46:45.045768 7f9d7a161700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.045766 7f9d7715b700  5 -- op tracker -- , seq: 747998, time: 2014-06-12 14:46:45.045757, event: op_commit, op: osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.045775 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2713 crt=0'0 lcod 173'2712 mlcod 173'2712 active+clean] repop_all_committed: repop tid 236310 all committed 
2014-06-12 14:46:45.045787 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2712 active+clean] eval_repop repgather(0x1ced2b40 173'2714 rep_tid=236310 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.045806 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2712 active+clean] log_op_stats osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.034396
2014-06-12 14:46:45.045825 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2712 active+clean] publish_stats_to_osd 173:3610
2014-06-12 14:46:45.045837 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2712 active+clean]  sending commit on repgather(0x1ced2b40 173'2714 rep_tid=236310 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) 0x1ba79900
2014-06-12 14:46:45.045882 7f9d7715b700  5 -- op tracker -- , seq: 747998, time: 2014-06-12 14:46:45.045876, event: commit_sent, op: osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.045897 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.045925 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.045977 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.045998 7f9d7a161700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.046004 7f9d7a161700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:46:45.046010 7f9d7a161700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.046110 7f9d7a161700 15 _write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 1490944~516096
2014-06-12 14:46:45.046182 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (1 active)
2014-06-12 14:46:45.046187 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2712 active+clean] op_applied: 236310
2014-06-12 14:46:45.046208 7f9d7795c700  5 -- op tracker -- , seq: 747998, time: 2014-06-12 14:46:45.046202, event: op_applied, op: osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.046222 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2712 active+clean] op_applied on primary on version 173'2714
2014-06-12 14:46:45.046234 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2712 active+clean] repop_all_applied: repop tid 236310 all applied 
2014-06-12 14:46:45.046220 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.046247 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2712 active+clean] eval_repop repgather(0x1ced2b40 173'2714 rep_tid=236310 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.046261 7f9d7a161700 10 lookup_strip_header done  cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.046265 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2712 active+clean] log_op_stats osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.034856
2014-06-12 14:46:45.046282 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2712 active+clean] publish_stats_to_osd 173:3611
2014-06-12 14:46:45.046293 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2712 active+clean]  requeue_ops 
2014-06-12 14:46:45.046304 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean]  removing repgather(0x1ced2b40 173'2714 rep_tid=236310 committed?=1 applied?=1 op=osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.046321 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean]    q front is repgather(0x1ced2b40 173'2714 rep_tid=236310 committed?=1 applied?=1 op=osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.046339 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] remove_repop repgather(0x1ced2b40 173'2714 rep_tid=236310 committed?=1 applied?=1 op=osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.046352 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean]  requeue_ops 
2014-06-12 14:46:45.046372 7f9d7795c700  5 -- op tracker -- , seq: 747998, time: 2014-06-12 14:46:45.046366, event: done, op: osd_op(client.4316.0:147247 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 974848~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.046395 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:45.048194 7f9d7a161700 10 _generic_write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 1490944~516096 = 0
2014-06-12 14:46:45.048262 7f9d7a161700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.048280 7f9d7a161700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.048288 7f9d7a161700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.048296 7f9d7a161700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.048552 7f9d7a161700 20 set_header setting 17437 cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 parent seq 0
2014-06-12 14:46:45.048635 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.048653 7f9d7a161700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:46:45.048663 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:45.051349 7f9d7a161700 10 _do_op 0x24c67080 seq 2460 r = 0, finisher 0x29f7da40 0x29eaee40
2014-06-12 14:46:45.051371 7f9d7a161700 10 _finish_op 0x24c67080 seq 2460 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.051379 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] op_commit: 236311
2014-06-12 14:46:45.051399 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (0 active)
2014-06-12 14:46:45.051409 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.051410 7f9d7715b700  5 -- op tracker -- , seq: 748002, time: 2014-06-12 14:46:45.051400, event: op_commit, op: osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.051445 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] op_applied: 236311
2014-06-12 14:46:45.051468 7f9d7795c700  5 -- op tracker -- , seq: 748002, time: 2014-06-12 14:46:45.051462, event: op_applied, op: osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.051477 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] op_applied on primary on version 173'2715
2014-06-12 14:46:45.051503 7f9d57543700  5 -- op tracker -- , seq: 748003, time: 2014-06-12 14:46:45.051462, event: header_read, op: osd_sub_op_reply(client.4316.0:147248 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.051522 7f9d57543700  5 -- op tracker -- , seq: 748003, time: 2014-06-12 14:46:45.051463, event: throttled, op: osd_sub_op_reply(client.4316.0:147248 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.051531 7f9d57543700  5 -- op tracker -- , seq: 748003, time: 2014-06-12 14:46:45.051481, event: all_read, op: osd_sub_op_reply(client.4316.0:147248 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.051538 7f9d57543700  5 -- op tracker -- , seq: 748003, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147248 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.051542 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147248 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:45.051548 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:45.051555 7f9d57543700 15 osd.2 173 enqueue_op 0x8dbdb00 prio 196 cost 0 latency 0.000093 osd_sub_op_reply(client.4316.0:147248 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:45.051588 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.051595 7f9d673df700 10 osd.2 173 dequeue_op 0x8dbdb00 prio 196 cost 0 latency 0.000133 osd_sub_op_reply(client.4316.0:147248 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean]
2014-06-12 14:46:45.051614 7f9d673df700  5 -- op tracker -- , seq: 748003, time: 2014-06-12 14:46:45.051610, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147248 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.051627 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] handle_message: 0x8dbdb00
2014-06-12 14:46:45.051649 7f9d673df700  5 -- op tracker -- , seq: 748003, time: 2014-06-12 14:46:45.051645, event: started, op: osd_sub_op_reply(client.4316.0:147248 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.051655 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] sub_op_modify_reply: tid 236311 op  ack_type 4 from 23
2014-06-12 14:46:45.051674 7f9d673df700  5 -- op tracker -- , seq: 748002, time: 2014-06-12 14:46:45.051669, event: sub_op_commit_rec, op: osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.051682 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] repop_all_applied: repop tid 236311 all applied 
2014-06-12 14:46:45.051692 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] eval_repop repgather(0x32ff4600 173'2715 rep_tid=236311 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.051709 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean]  sending ack on repgather(0x32ff4600 173'2715 rep_tid=236311 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) 0x2183b680
2014-06-12 14:46:45.051748 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2714 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] repop_all_committed: repop tid 236311 all committed 
2014-06-12 14:46:45.051759 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2713 active+clean] eval_repop repgather(0x32ff4600 173'2715 rep_tid=236311 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.051776 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2713 active+clean] log_op_stats osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.009849 lat 0.009877
2014-06-12 14:46:45.051793 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2713 active+clean] publish_stats_to_osd 173:3612
2014-06-12 14:46:45.051807 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2713 active+clean]  sending commit on repgather(0x32ff4600 173'2715 rep_tid=236311 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) 0x2b54a00
2014-06-12 14:46:45.051842 7f9d673df700  5 -- op tracker -- , seq: 748002, time: 2014-06-12 14:46:45.051836, event: commit_sent, op: osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.051850 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2713 active+clean]  requeue_ops 
2014-06-12 14:46:45.051861 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]  removing repgather(0x32ff4600 173'2715 rep_tid=236311 committed?=1 applied?=1 op=osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.051876 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]    q front is repgather(0x32ff4600 173'2715 rep_tid=236311 committed?=1 applied?=1 op=osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.051892 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] remove_repop repgather(0x32ff4600 173'2715 rep_tid=236311 committed?=1 applied?=1 op=osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.051907 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]  requeue_ops 
2014-06-12 14:46:45.051933 7f9d673df700  5 -- op tracker -- , seq: 748002, time: 2014-06-12 14:46:45.051928, event: done, op: osd_op(client.4316.0:147248 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 1490944~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.051952 7f9d673df700 10 osd.2 173 dequeue_op 0x8dbdb00 finish
2014-06-12 14:46:45.051958 7f9d673df700  5 -- op tracker -- , seq: 748003, time: 2014-06-12 14:46:45.051955, event: done, op: osd_sub_op_reply(client.4316.0:147248 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.051972 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.051975 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.053132 7f9d47c58700  5 -- op tracker -- , seq: 748004, time: 2014-06-12 14:46:45.051628, event: header_read, op: osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.053144 7f9d47c58700  5 -- op tracker -- , seq: 748004, time: 2014-06-12 14:46:45.051633, event: throttled, op: osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.053153 7f9d47c58700  5 -- op tracker -- , seq: 748004, time: 2014-06-12 14:46:45.053087, event: all_read, op: osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.053162 7f9d47c58700  5 -- op tracker -- , seq: 748004, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.053169 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:45.053194 7f9d47c58700 15 osd.2 173 enqueue_op 0x3850eb00 prio 63 cost 516096 latency 0.001566 osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.053363 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.053371 7f9d67be0700 10 osd.2 173 dequeue_op 0x3850eb00 prio 63 cost 516096 latency 0.001743 osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]
2014-06-12 14:46:45.053394 7f9d67be0700  5 -- op tracker -- , seq: 748004, time: 2014-06-12 14:46:45.053389, event: reached_pg, op: osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.053403 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.053418 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] handle_message: 0x3850eb00
2014-06-12 14:46:45.053428 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_op osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.053468 7f9d67be0700 15 getattr 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 '_'
2014-06-12 14:46:45.053719 7f9d67be0700 10 getattr 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 '_' = 0
2014-06-12 14:46:45.053736 7f9d67be0700 15 getattr 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 'snapset'
2014-06-12 14:46:45.053822 7f9d67be0700 10 getattr 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 'snapset' = 0
2014-06-12 14:46:45.053832 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] populate_obc_watchers 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.053844 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.053852 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] get_object_context: creating obc from disk: 0x212fd600
2014-06-12 14:46:45.053861 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] get_object_context: 0x212fd600 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 rwstate(none n=0 w=0) oi: 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2715 client.4316.0:147248 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2715) ssc: 0x16ee5ce0 snapset: 0=[]:[]+head
2014-06-12 14:46:45.053876 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] find_object_context 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 @head oi=91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2715 client.4316.0:147248 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2715)
2014-06-12 14:46:45.053902 7f9d67be0700  5 -- op tracker -- , seq: 748004, time: 2014-06-12 14:46:45.053896, event: started, op: osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.053911 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] execute_ctx 0xa04f800
2014-06-12 14:46:45.053927 7f9d67be0700  5 -- op tracker -- , seq: 748004, time: 2014-06-12 14:46:45.053922, event: started, op: osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.053933 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] ov 173'2715 av 173'2716 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.053946 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_osd_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096]
2014-06-12 14:46:45.053955 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.053968 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_osd_op  write 2007040~516096
2014-06-12 14:46:45.053982 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:45.053992 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 snapset=0x16ee5d18  snapc=0=[]
2014-06-12 14:46:45.054003 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.054011 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] finish_ctx 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 0xa04f800 op modify  
2014-06-12 14:46:45.054022 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]  set mtime to 2014-06-12 08:45:47.430258
2014-06-12 14:46:45.054040 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]  final snapset 0=[]:[]+head in 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.054057 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.054066 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] new_repop rep_tid 236312 on osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.054087 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] issue_repop rep_tid 236312 o 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.054109 7f9d67be0700  5 -- op tracker -- , seq: 748004, time: 2014-06-12 14:46:45.054105, event: waiting for subops from 23, op: osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.054132 7f9d67be0700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:45.054157 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2715 (0'0,173'2715] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] append_log log((0'0,173'2715], crt=0'0) [173'2716 (173'2715) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147249 2014-06-12 08:45:47.430258]
2014-06-12 14:46:45.054182 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] add_log_entry 173'2716 (173'2715) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147249 2014-06-12 08:45:47.430258
2014-06-12 14:46:45.054201 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.054231 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2716, trimmed: 
2014-06-12 14:46:45.054253 7f9d67be0700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.054269 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.054271 7f9d67be0700  5 queue_op 0x6e75740 seq 2461 osr(3.54b 0x276db30) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:45.054278 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] eval_repop repgather(0x534f980 173'2716 rep_tid=236312 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.054295 7f9d67be0700 10 osd.2 173 dequeue_op 0x3850eb00 finish
2014-06-12 14:46:45.054299 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.054303 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.055355 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (1 active)
2014-06-12 14:46:45.055366 7f9d7f16b700  5 _do_op 0x6e75740 seq 2461 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:46:45.055371 7f9d7f16b700 10 _do_transaction on 0x534fb00
2014-06-12 14:46:45.055379 7f9d7f16b700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.055515 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.055564 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.055630 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.055650 7f9d7f16b700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.055656 7f9d7f16b700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:46:45.055662 7f9d7f16b700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.055772 7f9d7f16b700 15 _write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 2007040~516096
2014-06-12 14:46:45.055943 7f9d7f16b700 10 lookup_strip_header done  cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.056109 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:45.058209 7f9d7f16b700 10 _generic_write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 2007040~516096 = 0
2014-06-12 14:46:45.058218 7f9d47c58700  5 -- op tracker -- , seq: 748005, time: 2014-06-12 14:46:45.053220, event: header_read, op: osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.058241 7f9d47c58700  5 -- op tracker -- , seq: 748005, time: 2014-06-12 14:46:45.053222, event: throttled, op: osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.058252 7f9d47c58700  5 -- op tracker -- , seq: 748005, time: 2014-06-12 14:46:45.058139, event: all_read, op: osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.058262 7f9d47c58700  5 -- op tracker -- , seq: 748005, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.058273 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:45.058305 7f9d7f16b700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.058304 7f9d47c58700 15 osd.2 173 enqueue_op 0x3850f000 prio 63 cost 516096 latency 0.005083 osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.058322 7f9d7f16b700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.058329 7f9d7f16b700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.058340 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.058348 7f9d663dd700 10 osd.2 173 dequeue_op 0x3850f000 prio 63 cost 516096 latency 0.005127 osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]
2014-06-12 14:46:45.058368 7f9d7f16b700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.058374 7f9d663dd700  5 -- op tracker -- , seq: 748005, time: 2014-06-12 14:46:45.058368, event: reached_pg, op: osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.058384 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.058401 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] handle_message: 0x3850f000
2014-06-12 14:46:45.058412 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_op osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.058440 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] get_object_context: found obc in cache: 0x212fd600
2014-06-12 14:46:45.058452 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] get_object_context: 0x212fd600 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 rwstate(write n=1 w=0) oi: 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2716 client.4316.0:147249 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2716) ssc: 0x16ee5ce0 snapset: 0=[]:[]+head
2014-06-12 14:46:45.058469 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] find_object_context 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 @head oi=91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2716 client.4316.0:147249 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2716)
2014-06-12 14:46:45.058496 7f9d663dd700  5 -- op tracker -- , seq: 748005, time: 2014-06-12 14:46:45.058491, event: started, op: osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.058505 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] execute_ctx 0x2e411200
2014-06-12 14:46:45.058523 7f9d663dd700  5 -- op tracker -- , seq: 748005, time: 2014-06-12 14:46:45.058518, event: started, op: osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.058530 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] ov 173'2716 av 173'2717 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.058544 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_osd_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096]
2014-06-12 14:46:45.058555 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.058569 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_osd_op  write 2523136~516096
2014-06-12 14:46:45.058587 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:45.058599 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 snapset=0x16ee5d18  snapc=0=[]
2014-06-12 14:46:45.058612 7f9d7f16b700 20 set_header setting 17437 cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 parent seq 0
2014-06-12 14:46:45.058611 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.058622 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] finish_ctx 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 0x2e411200 op modify  
2014-06-12 14:46:45.058634 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]  set mtime to 2014-06-12 08:45:47.431034
2014-06-12 14:46:45.058656 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]  final snapset 0=[]:[]+head in 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.058675 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.058682 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.058685 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] new_repop rep_tid 236313 on osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.058702 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] issue_repop rep_tid 236313 o 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.058710 7f9d7f16b700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:46:45.058721 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:45.058732 7f9d663dd700  5 -- op tracker -- , seq: 748005, time: 2014-06-12 14:46:45.058723, event: waiting for subops from 23, op: osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.058756 7f9d663dd700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:45.058779 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2716 (0'0,173'2716] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] append_log log((0'0,173'2716], crt=0'0) [173'2717 (173'2716) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147250 2014-06-12 08:45:47.431034]
2014-06-12 14:46:45.058803 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] add_log_entry 173'2717 (173'2716) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147250 2014-06-12 08:45:47.431034
2014-06-12 14:46:45.058823 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.058860 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2717, trimmed: 
2014-06-12 14:46:45.058881 7f9d663dd700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.058886 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.058888 7f9d663dd700  5 queue_op 0x6268c60 seq 2462 osr(3.54b 0x276db30) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:45.058896 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] eval_repop repgather(0x1ced3d40 173'2717 rep_tid=236313 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.058915 7f9d663dd700 10 osd.2 173 dequeue_op 0x3850f000 finish
2014-06-12 14:46:45.058919 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.058922 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.058931 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (2 active)
2014-06-12 14:46:45.059550 7f9d47c58700  5 -- op tracker -- , seq: 748006, time: 2014-06-12 14:46:45.058960, event: header_read, op: osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.059563 7f9d47c58700  5 -- op tracker -- , seq: 748006, time: 2014-06-12 14:46:45.058962, event: throttled, op: osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.059572 7f9d47c58700  5 -- op tracker -- , seq: 748006, time: 2014-06-12 14:46:45.059511, event: all_read, op: osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.059582 7f9d47c58700  5 -- op tracker -- , seq: 748006, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.059589 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:45.059609 7f9d47c58700 15 osd.2 173 enqueue_op 0x2519d100 prio 63 cost 516096 latency 0.000649 osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.059642 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.059649 7f9d693e3700 10 osd.2 173 dequeue_op 0x2519d100 prio 63 cost 516096 latency 0.000689 osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]
2014-06-12 14:46:45.059673 7f9d693e3700  5 -- op tracker -- , seq: 748006, time: 2014-06-12 14:46:45.059667, event: reached_pg, op: osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.059681 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.059696 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] handle_message: 0x2519d100
2014-06-12 14:46:45.059706 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_op osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.059729 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] get_object_context: found obc in cache: 0x212fd600
2014-06-12 14:46:45.059739 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] get_object_context: 0x212fd600 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 rwstate(write n=2 w=0) oi: 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2717 client.4316.0:147250 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2717) ssc: 0x16ee5ce0 snapset: 0=[]:[]+head
2014-06-12 14:46:45.059755 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] find_object_context 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 @head oi=91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2717 client.4316.0:147250 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2717)
2014-06-12 14:46:45.059780 7f9d693e3700  5 -- op tracker -- , seq: 748006, time: 2014-06-12 14:46:45.059774, event: started, op: osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.059787 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] execute_ctx 0x5ca3000
2014-06-12 14:46:45.059805 7f9d693e3700  5 -- op tracker -- , seq: 748006, time: 2014-06-12 14:46:45.059800, event: started, op: osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.059812 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] ov 173'2717 av 173'2718 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.059826 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_osd_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096]
2014-06-12 14:46:45.059837 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.059851 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_osd_op  write 3039232~516096
2014-06-12 14:46:45.059865 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:45.059880 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 snapset=0x16ee5d18  snapc=0=[]
2014-06-12 14:46:45.059893 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.059903 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] finish_ctx 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 0x5ca3000 op modify  
2014-06-12 14:46:45.059916 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]  set mtime to 2014-06-12 08:45:47.434396
2014-06-12 14:46:45.059937 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]  final snapset 0=[]:[]+head in 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.059956 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.059967 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] new_repop rep_tid 236314 on osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.059983 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] issue_repop rep_tid 236314 o 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.060008 7f9d693e3700  5 -- op tracker -- , seq: 748006, time: 2014-06-12 14:46:45.060002, event: waiting for subops from 23, op: osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.060035 7f9d693e3700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:45.060057 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2717 (0'0,173'2717] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] append_log log((0'0,173'2717], crt=0'0) [173'2718 (173'2717) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147251 2014-06-12 08:45:47.434396]
2014-06-12 14:46:45.060080 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] add_log_entry 173'2718 (173'2717) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147251 2014-06-12 08:45:47.434396
2014-06-12 14:46:45.060100 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.060131 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2718, trimmed: 
2014-06-12 14:46:45.060154 7f9d693e3700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.060158 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.060159 7f9d693e3700  5 queue_op 0x2465140 seq 2463 osr(3.54b 0x276db30) 518389 bytes   (queue has 3 ops and 1555167 bytes)
2014-06-12 14:46:45.060167 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] eval_repop repgather(0x297020c0 173'2718 rep_tid=236314 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.060186 7f9d693e3700 10 osd.2 173 dequeue_op 0x2519d100 finish
2014-06-12 14:46:45.060190 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.060193 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.060202 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (3 active)
2014-06-12 14:46:45.061267 7f9d57543700  5 -- op tracker -- , seq: 748007, time: 2014-06-12 14:46:45.061119, event: header_read, op: osd_sub_op_reply(client.4316.0:147249 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.061278 7f9d57543700  5 -- op tracker -- , seq: 748007, time: 2014-06-12 14:46:45.061213, event: throttled, op: osd_sub_op_reply(client.4316.0:147249 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.061286 7f9d57543700  5 -- op tracker -- , seq: 748007, time: 2014-06-12 14:46:45.061230, event: all_read, op: osd_sub_op_reply(client.4316.0:147249 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.061294 7f9d57543700  5 -- op tracker -- , seq: 748007, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147249 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.061299 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147249 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:45.061305 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:45.061314 7f9d57543700 15 osd.2 173 enqueue_op 0x8dbd300 prio 196 cost 0 latency 0.000194 osd_sub_op_reply(client.4316.0:147249 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:45.061342 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.061350 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x8dbd300 prio 196 cost 0 latency 0.000231 osd_sub_op_reply(client.4316.0:147249 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean]
2014-06-12 14:46:45.061372 7f9d6b3e7700  5 -- op tracker -- , seq: 748007, time: 2014-06-12 14:46:45.061368, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147249 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.061382 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] handle_message: 0x8dbd300
2014-06-12 14:46:45.061399 7f9d6b3e7700  5 -- op tracker -- , seq: 748007, time: 2014-06-12 14:46:45.061396, event: started, op: osd_sub_op_reply(client.4316.0:147249 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.061405 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] sub_op_modify_reply: tid 236312 op  ack_type 4 from 23
2014-06-12 14:46:45.061429 7f9d6b3e7700  5 -- op tracker -- , seq: 748004, time: 2014-06-12 14:46:45.061419, event: sub_op_commit_rec, op: osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.061436 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x8dbd300 finish
2014-06-12 14:46:45.061444 7f9d6b3e7700  5 -- op tracker -- , seq: 748007, time: 2014-06-12 14:46:45.061440, event: done, op: osd_sub_op_reply(client.4316.0:147249 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.061458 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.061461 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.063405 7f9d7f16b700 10 _do_op 0x6e75740 seq 2461 r = 0, finisher 0x960c560 0x3786f1c0
2014-06-12 14:46:45.063434 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] op_commit: 236312
2014-06-12 14:46:45.063472 7f9d7715b700  5 -- op tracker -- , seq: 748004, time: 2014-06-12 14:46:45.063461, event: op_commit, op: osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.063482 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2715 crt=0'0 lcod 173'2714 mlcod 173'2714 active+clean] repop_all_committed: repop tid 236312 all committed 
2014-06-12 14:46:45.063496 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2714 active+clean] eval_repop repgather(0x534f980 173'2716 rep_tid=236312 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.063516 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2714 active+clean] log_op_stats osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.011886
2014-06-12 14:46:45.063537 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2714 active+clean] publish_stats_to_osd 173:3613
2014-06-12 14:46:45.063552 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2714 active+clean]  sending commit on repgather(0x534f980 173'2716 rep_tid=236312 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) 0x1710a780
2014-06-12 14:46:45.063590 7f9d7715b700  5 -- op tracker -- , seq: 748004, time: 2014-06-12 14:46:45.063585, event: commit_sent, op: osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.063600 7f9d7f16b700 10 _finish_op 0x6e75740 seq 2461 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.063647 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (2 active)
2014-06-12 14:46:45.063657 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.063666 7f9d7e169700  5 _do_op 0x6268c60 seq 2462 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:46:45.063654 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2714 active+clean] op_applied: 236312
2014-06-12 14:46:45.063671 7f9d7e169700 10 _do_transaction on 0x51adb00
2014-06-12 14:46:45.063683 7f9d7e169700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.063718 7f9d47c58700  5 -- op tracker -- , seq: 748008, time: 2014-06-12 14:46:45.060229, event: header_read, op: osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.063750 7f9d47c58700  5 -- op tracker -- , seq: 748008, time: 2014-06-12 14:46:45.060231, event: throttled, op: osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.063777 7f9d47c58700  5 -- op tracker -- , seq: 748008, time: 2014-06-12 14:46:45.060881, event: all_read, op: osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.063804 7f9d47c58700  5 -- op tracker -- , seq: 748008, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.063833 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.063835 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:45.063838 7f9d7795c700  5 -- op tracker -- , seq: 748004, time: 2014-06-12 14:46:45.063672, event: op_applied, op: osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.063857 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.063854 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2714 active+clean] op_applied on primary on version 173'2716
2014-06-12 14:46:45.063867 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2714 active+clean] repop_all_applied: repop tid 236312 all applied 
2014-06-12 14:46:45.063877 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2714 active+clean] eval_repop repgather(0x534f980 173'2716 rep_tid=236312 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.063893 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2714 active+clean] log_op_stats osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.012264
2014-06-12 14:46:45.063909 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2714 active+clean] publish_stats_to_osd 173:3614
2014-06-12 14:46:45.063917 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.063920 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2714 active+clean]  requeue_ops 
2014-06-12 14:46:45.063935 7f9d7e169700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.063941 7f9d7e169700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:46:45.063931 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean]  removing repgather(0x534f980 173'2716 rep_tid=236312 committed?=1 applied?=1 op=osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.063948 7f9d7e169700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.063950 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean]    q front is repgather(0x534f980 173'2716 rep_tid=236312 committed?=1 applied?=1 op=osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.063964 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] remove_repop repgather(0x534f980 173'2716 rep_tid=236312 committed?=1 applied?=1 op=osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.063978 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean]  requeue_ops 
2014-06-12 14:46:45.063999 7f9d7795c700  5 -- op tracker -- , seq: 748004, time: 2014-06-12 14:46:45.063994, event: done, op: osd_op(client.4316.0:147249 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2007040~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.064033 7f9d47c58700 15 osd.2 173 enqueue_op 0x2519de00 prio 63 cost 516096 latency 0.003803 osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.064041 7f9d7e169700 15 _write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 2523136~516096
2014-06-12 14:46:45.064094 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.064102 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2519de00 prio 63 cost 516096 latency 0.003873 osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean]
2014-06-12 14:46:45.064125 7f9d65bdc700  5 -- op tracker -- , seq: 748008, time: 2014-06-12 14:46:45.064120, event: reached_pg, op: osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.064133 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.064147 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] handle_message: 0x2519de00
2014-06-12 14:46:45.064156 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] do_op osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.064178 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] get_object_context: found obc in cache: 0x212fd600
2014-06-12 14:46:45.064196 7f9d7e169700 10 lookup_strip_header done  cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.064188 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] get_object_context: 0x212fd600 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 rwstate(write n=2 w=0) oi: 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2718 client.4316.0:147251 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2718) ssc: 0x16ee5ce0 snapset: 0=[]:[]+head
2014-06-12 14:46:45.064202 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] find_object_context 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 @head oi=91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2718 client.4316.0:147251 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2718)
2014-06-12 14:46:45.064225 7f9d65bdc700  5 -- op tracker -- , seq: 748008, time: 2014-06-12 14:46:45.064221, event: started, op: osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.064231 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] execute_ctx 0x33b2e400
2014-06-12 14:46:45.064246 7f9d65bdc700  5 -- op tracker -- , seq: 748008, time: 2014-06-12 14:46:45.064242, event: started, op: osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.064251 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] do_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] ov 173'2718 av 173'2719 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.064269 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] do_osd_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096]
2014-06-12 14:46:45.064278 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.064289 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] do_osd_op  write 3555328~516096
2014-06-12 14:46:45.064302 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:45.064315 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 snapset=0x16ee5d18  snapc=0=[]
2014-06-12 14:46:45.064323 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:45.064326 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.064335 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] finish_ctx 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 0x33b2e400 op modify  
2014-06-12 14:46:45.064347 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean]  set mtime to 2014-06-12 08:45:47.435245
2014-06-12 14:46:45.064367 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean]  final snapset 0=[]:[]+head in 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.064384 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.064394 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] new_repop rep_tid 236315 on osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.064410 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] issue_repop rep_tid 236315 o 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.064432 7f9d65bdc700  5 -- op tracker -- , seq: 748008, time: 2014-06-12 14:46:45.064427, event: waiting for subops from 23, op: osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.064454 7f9d65bdc700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:45.064474 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2718 (0'0,173'2718] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] append_log log((0'0,173'2718], crt=0'0) [173'2719 (173'2718) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147252 2014-06-12 08:45:47.435245]
2014-06-12 14:46:45.064495 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] add_log_entry 173'2719 (173'2718) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147252 2014-06-12 08:45:47.435245
2014-06-12 14:46:45.064513 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.064543 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2719, trimmed: 
2014-06-12 14:46:45.064562 7f9d65bdc700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.064565 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.064567 7f9d65bdc700  5 queue_op 0x6c326c0 seq 2464 osr(3.54b 0x276db30) 518389 bytes   (queue has 3 ops and 1555167 bytes)
2014-06-12 14:46:45.064577 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (3 active)
2014-06-12 14:46:45.064573 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] eval_repop repgather(0x1c378b40 173'2719 rep_tid=236315 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.064594 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2519de00 finish
2014-06-12 14:46:45.064597 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.064600 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.065363 7f9d47c58700  5 -- op tracker -- , seq: 748009, time: 2014-06-12 14:46:45.065051, event: header_read, op: osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.065399 7f9d47c58700  5 -- op tracker -- , seq: 748009, time: 2014-06-12 14:46:45.065053, event: throttled, op: osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.065427 7f9d47c58700  5 -- op tracker -- , seq: 748009, time: 2014-06-12 14:46:45.065303, event: all_read, op: osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.065456 7f9d47c58700  5 -- op tracker -- , seq: 748009, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.065481 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:45.065526 7f9d47c58700 15 osd.2 173 enqueue_op 0x124e2a00 prio 63 cost 122880 latency 0.000475 osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.065566 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.065573 7f9d62bd6700 10 osd.2 173 dequeue_op 0x124e2a00 prio 63 cost 122880 latency 0.000521 osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean]
2014-06-12 14:46:45.065535 7f9d57543700  5 -- op tracker -- , seq: 748010, time: 2014-06-12 14:46:45.065502, event: header_read, op: osd_sub_op_reply(client.4316.0:147250 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.065643 7f9d57543700  5 -- op tracker -- , seq: 748010, time: 2014-06-12 14:46:45.065503, event: throttled, op: osd_sub_op_reply(client.4316.0:147250 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.065653 7f9d57543700  5 -- op tracker -- , seq: 748010, time: 2014-06-12 14:46:45.065512, event: all_read, op: osd_sub_op_reply(client.4316.0:147250 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.065661 7f9d57543700  5 -- op tracker -- , seq: 748010, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147250 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.065665 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147250 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:45.065671 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:45.065678 7f9d57543700 15 osd.2 173 enqueue_op 0x2d17be00 prio 196 cost 0 latency 0.000176 osd_sub_op_reply(client.4316.0:147250 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:45.065698 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:45.065700 7f9d62bd6700  5 -- op tracker -- , seq: 748009, time: 2014-06-12 14:46:45.065590, event: reached_pg, op: osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.065709 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.065720 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] handle_message: 0x124e2a00
2014-06-12 14:46:45.065726 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] do_op osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.065743 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] get_object_context: found obc in cache: 0x212fd600
2014-06-12 14:46:45.065749 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] get_object_context: 0x212fd600 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 rwstate(write n=3 w=0) oi: 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2719 client.4316.0:147252 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2719) ssc: 0x16ee5ce0 snapset: 0=[]:[]+head
2014-06-12 14:46:45.065759 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] find_object_context 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 @head oi=91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3(173'2719 client.4316.0:147252 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2719)
2014-06-12 14:46:45.065780 7f9d62bd6700  5 -- op tracker -- , seq: 748009, time: 2014-06-12 14:46:45.065776, event: started, op: osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.065784 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] execute_ctx 0xa050400
2014-06-12 14:46:45.065794 7f9d62bd6700  5 -- op tracker -- , seq: 748009, time: 2014-06-12 14:46:45.065791, event: started, op: osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.065798 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] do_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] ov 173'2719 av 173'2720 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.065807 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] do_osd_op 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880]
2014-06-12 14:46:45.065813 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.065821 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] do_osd_op  write 4071424~122880
2014-06-12 14:46:45.065830 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:45.065837 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 snapset=0x16ee5d18  snapc=0=[]
2014-06-12 14:46:45.065844 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] make_writeable 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.065850 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] finish_ctx 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 0xa050400 op modify  
2014-06-12 14:46:45.065857 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean]  set mtime to 2014-06-12 08:45:47.445288
2014-06-12 14:46:45.065872 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean]  final snapset 0=[]:[]+head in 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.065883 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.065889 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] new_repop rep_tid 236316 on osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.065902 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] issue_repop rep_tid 236316 o 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.065919 7f9d62bd6700  5 -- op tracker -- , seq: 748009, time: 2014-06-12 14:46:45.065916, event: waiting for subops from 23, op: osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.066147 7f9d62bd6700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:45.066167 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2719 (0'0,173'2719] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] append_log log((0'0,173'2719], crt=0'0) [173'2720 (173'2719) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147253 2014-06-12 08:45:47.445288]
2014-06-12 14:46:45.066197 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] add_log_entry 173'2720 (173'2719) modify   91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 by client.4316.0:147253 2014-06-12 08:45:47.445288
2014-06-12 14:46:45.066218 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.066249 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2720, trimmed: 
2014-06-12 14:46:45.066273 7f9d62bd6700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.066278 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.066280 7f9d62bd6700  5 queue_op 0x77481e0 seq 2465 osr(3.54b 0x276db30) 125173 bytes   (queue has 4 ops and 1680340 bytes)
2014-06-12 14:46:45.066295 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (4 active)
2014-06-12 14:46:45.066289 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] eval_repop repgather(0x2a37bec0 173'2720 rep_tid=236316 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.066307 7f9d62bd6700 10 osd.2 173 dequeue_op 0x124e2a00 finish
2014-06-12 14:46:45.066314 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:45.066317 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.066325 7f9d66bde700 10 osd.2 173 dequeue_op 0x2d17be00 prio 196 cost 0 latency 0.000823 osd_sub_op_reply(client.4316.0:147250 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean]
2014-06-12 14:46:45.066343 7f9d66bde700  5 -- op tracker -- , seq: 748010, time: 2014-06-12 14:46:45.066339, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147250 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.066349 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] handle_message: 0x2d17be00
2014-06-12 14:46:45.066362 7f9d66bde700  5 -- op tracker -- , seq: 748010, time: 2014-06-12 14:46:45.066359, event: started, op: osd_sub_op_reply(client.4316.0:147250 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.066367 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] sub_op_modify_reply: tid 236313 op  ack_type 4 from 23
2014-06-12 14:46:45.066384 7f9d66bde700  5 -- op tracker -- , seq: 748005, time: 2014-06-12 14:46:45.066378, event: sub_op_commit_rec, op: osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.066389 7f9d66bde700 10 osd.2 173 dequeue_op 0x2d17be00 finish
2014-06-12 14:46:45.066395 7f9d66bde700  5 -- op tracker -- , seq: 748010, time: 2014-06-12 14:46:45.066392, event: done, op: osd_sub_op_reply(client.4316.0:147250 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.066408 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.066410 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.067457 7f9d7e169700 10 _generic_write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 2523136~516096 = 0
2014-06-12 14:46:45.067502 7f9d7e169700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.067518 7f9d7e169700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.067526 7f9d7e169700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.067536 7f9d7e169700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.067717 7f9d7e169700 20 set_header setting 17437 cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 parent seq 0
2014-06-12 14:46:45.067799 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.067815 7f9d7e169700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:46:45.067824 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:45.070361 7f9d7e169700 10 _do_op 0x6268c60 seq 2462 r = 0, finisher 0x7f21b80 0x4dd9e80
2014-06-12 14:46:45.070391 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] op_commit: 236313
2014-06-12 14:46:45.070422 7f9d7715b700  5 -- op tracker -- , seq: 748005, time: 2014-06-12 14:46:45.070411, event: op_commit, op: osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.070432 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2716 crt=0'0 lcod 173'2715 mlcod 173'2715 active+clean] repop_all_committed: repop tid 236313 all committed 
2014-06-12 14:46:45.070446 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean] eval_repop repgather(0x1ced3d40 173'2717 rep_tid=236313 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.070466 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean] log_op_stats osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.017244
2014-06-12 14:46:45.070486 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean] publish_stats_to_osd 173:3615
2014-06-12 14:46:45.070500 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean]  sending commit on repgather(0x1ced3d40 173'2717 rep_tid=236313 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) 0x26e56780
2014-06-12 14:46:45.070543 7f9d7715b700  5 -- op tracker -- , seq: 748005, time: 2014-06-12 14:46:45.070537, event: commit_sent, op: osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.070557 7f9d7e169700 10 _finish_op 0x6268c60 seq 2462 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.070576 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (3 active)
2014-06-12 14:46:45.070581 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.071352 7f9d7815d700  5 _do_op 0x2465140 seq 2463 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:46:45.071364 7f9d7815d700 10 _do_transaction on 0x29703980
2014-06-12 14:46:45.071372 7f9d7815d700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.071559 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.071578 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.071616 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.071634 7f9d7815d700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.071639 7f9d7815d700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:46:45.071644 7f9d7815d700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.071657 7f9d7815d700 15 _write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 3039232~516096
2014-06-12 14:46:45.071669 7f9d57543700  5 -- op tracker -- , seq: 748011, time: 2014-06-12 14:46:45.071616, event: header_read, op: osd_sub_op_reply(client.4316.0:147251 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.071679 7f9d57543700  5 -- op tracker -- , seq: 748011, time: 2014-06-12 14:46:45.071618, event: throttled, op: osd_sub_op_reply(client.4316.0:147251 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.071688 7f9d57543700  5 -- op tracker -- , seq: 748011, time: 2014-06-12 14:46:45.071631, event: all_read, op: osd_sub_op_reply(client.4316.0:147251 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.071696 7f9d57543700  5 -- op tracker -- , seq: 748011, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147251 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.071702 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147251 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:45.071709 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:45.071719 7f9d57543700 15 osd.2 173 enqueue_op 0x2d17a300 prio 196 cost 0 latency 0.000102 osd_sub_op_reply(client.4316.0:147251 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:45.071753 7f9d7815d700 10 lookup_strip_header done  cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.071762 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.071771 7f9d683e1700 10 osd.2 173 dequeue_op 0x2d17a300 prio 196 cost 0 latency 0.000155 osd_sub_op_reply(client.4316.0:147251 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean]
2014-06-12 14:46:45.071793 7f9d683e1700  5 -- op tracker -- , seq: 748011, time: 2014-06-12 14:46:45.071789, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147251 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.071801 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean] handle_message: 0x2d17a300
2014-06-12 14:46:45.071819 7f9d683e1700  5 -- op tracker -- , seq: 748011, time: 2014-06-12 14:46:45.071815, event: started, op: osd_sub_op_reply(client.4316.0:147251 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.071826 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean] sub_op_modify_reply: tid 236314 op  ack_type 4 from 23
2014-06-12 14:46:45.071868 7f9d683e1700  5 -- op tracker -- , seq: 748006, time: 2014-06-12 14:46:45.071860, event: sub_op_commit_rec, op: osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.071878 7f9d683e1700 10 osd.2 173 dequeue_op 0x2d17a300 finish
2014-06-12 14:46:45.071893 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:45.071886 7f9d683e1700  5 -- op tracker -- , seq: 748011, time: 2014-06-12 14:46:45.071883, event: done, op: osd_sub_op_reply(client.4316.0:147251 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.071914 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.071917 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.072482 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean] op_applied: 236313
2014-06-12 14:46:45.072512 7f9d7795c700  5 -- op tracker -- , seq: 748005, time: 2014-06-12 14:46:45.072503, event: op_applied, op: osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.072523 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean] op_applied on primary on version 173'2717
2014-06-12 14:46:45.072534 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean] repop_all_applied: repop tid 236313 all applied 
2014-06-12 14:46:45.072544 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean] eval_repop repgather(0x1ced3d40 173'2717 rep_tid=236313 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.072564 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean] log_op_stats osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.019343
2014-06-12 14:46:45.072583 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean] publish_stats_to_osd 173:3616
2014-06-12 14:46:45.072597 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2715 active+clean]  requeue_ops 
2014-06-12 14:46:45.072609 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2716 active+clean]  removing repgather(0x1ced3d40 173'2717 rep_tid=236313 committed?=1 applied?=1 op=osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.072627 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2716 active+clean]    q front is repgather(0x1ced3d40 173'2717 rep_tid=236313 committed?=1 applied?=1 op=osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.072642 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2716 active+clean] remove_repop repgather(0x1ced3d40 173'2717 rep_tid=236313 committed?=1 applied?=1 op=osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.072658 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2716 active+clean]  requeue_ops 
2014-06-12 14:46:45.072681 7f9d7795c700  5 -- op tracker -- , seq: 748005, time: 2014-06-12 14:46:45.072676, event: done, op: osd_op(client.4316.0:147250 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 2523136~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.074533 7f9d7815d700 10 _generic_write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 3039232~516096 = 0
2014-06-12 14:46:45.074586 7f9d7815d700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.074605 7f9d7815d700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.074613 7f9d7815d700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.074622 7f9d7815d700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.074775 7f9d7815d700 20 set_header setting 17437 cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 parent seq 0
2014-06-12 14:46:45.074849 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.074887 7f9d7815d700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:46:45.074899 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:45.078016 7f9d7815d700 10 _do_op 0x2465140 seq 2463 r = 0, finisher 0x66b7660 0xcb60cc0
2014-06-12 14:46:45.078040 7f9d7815d700 10 _finish_op 0x2465140 seq 2463 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.078057 7f9d82171700  5 _do_op 0x6c326c0 seq 2464 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:46:45.078063 7f9d82171700 10 _do_transaction on 0x1c378fc0
2014-06-12 14:46:45.078071 7f9d82171700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.078118 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2716 active+clean] op_commit: 236314
2014-06-12 14:46:45.078158 7f9d7715b700  5 -- op tracker -- , seq: 748006, time: 2014-06-12 14:46:45.078145, event: op_commit, op: osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.078167 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2717 crt=0'0 lcod 173'2716 mlcod 173'2716 active+clean] repop_all_committed: repop tid 236314 all committed 
2014-06-12 14:46:45.078210 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.078196 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2716 active+clean] eval_repop repgather(0x297020c0 173'2718 rep_tid=236314 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.078243 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.078226 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2716 active+clean] log_op_stats osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.019263
2014-06-12 14:46:45.078252 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2716 active+clean] publish_stats_to_osd 173:3617
2014-06-12 14:46:45.078302 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2716 active+clean]  sending commit on repgather(0x297020c0 173'2718 rep_tid=236314 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) 0x355c0a00
2014-06-12 14:46:45.078358 7f9d7715b700  5 -- op tracker -- , seq: 748006, time: 2014-06-12 14:46:45.078352, event: commit_sent, op: osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.079784 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.079819 7f9d82171700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.079825 7f9d82171700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:46:45.079833 7f9d82171700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.079846 7f9d82171700 15 _write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 3555328~516096
2014-06-12 14:46:45.079832 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2716 active+clean] op_applied: 236314
2014-06-12 14:46:45.079862 7f9d7795c700  5 -- op tracker -- , seq: 748006, time: 2014-06-12 14:46:45.079853, event: op_applied, op: osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.079873 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2716 active+clean] op_applied on primary on version 173'2718
2014-06-12 14:46:45.079883 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2716 active+clean] repop_all_applied: repop tid 236314 all applied 
2014-06-12 14:46:45.079892 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2716 active+clean] eval_repop repgather(0x297020c0 173'2718 rep_tid=236314 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.079909 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2716 active+clean] log_op_stats osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.020947
2014-06-12 14:46:45.079945 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2716 active+clean] publish_stats_to_osd 173:3618
2014-06-12 14:46:45.079957 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2716 active+clean]  requeue_ops 
2014-06-12 14:46:45.079967 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2717 active+clean]  removing repgather(0x297020c0 173'2718 rep_tid=236314 committed?=1 applied?=1 op=osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.079981 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2717 active+clean]    q front is repgather(0x297020c0 173'2718 rep_tid=236314 committed?=1 applied?=1 op=osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.080001 7f9d82171700 10 lookup_strip_header done  cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.079995 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2717 active+clean] remove_repop repgather(0x297020c0 173'2718 rep_tid=236314 committed?=1 applied?=1 op=osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.080008 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2717 active+clean]  requeue_ops 
2014-06-12 14:46:45.080229 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:45.080084 7f9d7815d700  5 -- op tracker -- , seq: 748006, time: 2014-06-12 14:46:45.080055, event: done, op: osd_op(client.4316.0:147251 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3039232~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.081730 7f9d57543700  5 -- op tracker -- , seq: 748012, time: 2014-06-12 14:46:45.081213, event: header_read, op: osd_sub_op_reply(client.4316.0:147252 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.081744 7f9d57543700  5 -- op tracker -- , seq: 748012, time: 2014-06-12 14:46:45.081214, event: throttled, op: osd_sub_op_reply(client.4316.0:147252 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.081756 7f9d57543700  5 -- op tracker -- , seq: 748012, time: 2014-06-12 14:46:45.081227, event: all_read, op: osd_sub_op_reply(client.4316.0:147252 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.081765 7f9d57543700  5 -- op tracker -- , seq: 748012, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147252 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.081771 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147252 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:45.081777 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:45.081783 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (2 active)
2014-06-12 14:46:45.081785 7f9d57543700 15 osd.2 173 enqueue_op 0x2d17a100 prio 196 cost 0 latency 0.000572 osd_sub_op_reply(client.4316.0:147252 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:45.081789 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.081810 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.081816 7f9d64bda700 10 osd.2 173 dequeue_op 0x2d17a100 prio 196 cost 0 latency 0.000603 osd_sub_op_reply(client.4316.0:147252 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2717 active+clean]
2014-06-12 14:46:45.081838 7f9d64bda700  5 -- op tracker -- , seq: 748012, time: 2014-06-12 14:46:45.081834, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147252 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.081849 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2717 active+clean] handle_message: 0x2d17a100
2014-06-12 14:46:45.081864 7f9d64bda700  5 -- op tracker -- , seq: 748012, time: 2014-06-12 14:46:45.081861, event: started, op: osd_sub_op_reply(client.4316.0:147252 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.081871 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2717 active+clean] sub_op_modify_reply: tid 236315 op  ack_type 4 from 23
2014-06-12 14:46:45.081892 7f9d64bda700  5 -- op tracker -- , seq: 748008, time: 2014-06-12 14:46:45.081884, event: sub_op_commit_rec, op: osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.081906 7f9d64bda700 10 osd.2 173 dequeue_op 0x2d17a100 finish
2014-06-12 14:46:45.081914 7f9d64bda700  5 -- op tracker -- , seq: 748012, time: 2014-06-12 14:46:45.081910, event: done, op: osd_sub_op_reply(client.4316.0:147252 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.081926 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.081929 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.082627 7f9d82171700 10 _generic_write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 3555328~516096 = 0
2014-06-12 14:46:45.082680 7f9d82171700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.082704 7f9d82171700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.082713 7f9d82171700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.082721 7f9d82171700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.082954 7f9d82171700 20 set_header setting 17437 cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 parent seq 0
2014-06-12 14:46:45.083020 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.083036 7f9d82171700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:46:45.083048 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:45.083184 7f9d57543700  5 -- op tracker -- , seq: 748013, time: 2014-06-12 14:46:45.083131, event: header_read, op: osd_sub_op_reply(client.4316.0:147253 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.083193 7f9d57543700  5 -- op tracker -- , seq: 748013, time: 2014-06-12 14:46:45.083133, event: throttled, op: osd_sub_op_reply(client.4316.0:147253 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.083200 7f9d57543700  5 -- op tracker -- , seq: 748013, time: 2014-06-12 14:46:45.083156, event: all_read, op: osd_sub_op_reply(client.4316.0:147253 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.083206 7f9d57543700  5 -- op tracker -- , seq: 748013, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147253 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.083212 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147253 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:45.083217 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:45.083224 7f9d57543700 15 osd.2 173 enqueue_op 0x2d17a800 prio 196 cost 0 latency 0.000093 osd_sub_op_reply(client.4316.0:147253 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:45.083251 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.083258 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2d17a800 prio 196 cost 0 latency 0.000127 osd_sub_op_reply(client.4316.0:147253 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2717 active+clean]
2014-06-12 14:46:45.083280 7f9d6abe6700  5 -- op tracker -- , seq: 748013, time: 2014-06-12 14:46:45.083277, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147253 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.083289 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2717 active+clean] handle_message: 0x2d17a800
2014-06-12 14:46:45.083304 7f9d6abe6700  5 -- op tracker -- , seq: 748013, time: 2014-06-12 14:46:45.083300, event: started, op: osd_sub_op_reply(client.4316.0:147253 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.083310 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2717 active+clean] sub_op_modify_reply: tid 236316 op  ack_type 4 from 23
2014-06-12 14:46:45.083329 7f9d6abe6700  5 -- op tracker -- , seq: 748009, time: 2014-06-12 14:46:45.083322, event: sub_op_commit_rec, op: osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.083342 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2d17a800 finish
2014-06-12 14:46:45.083348 7f9d6abe6700  5 -- op tracker -- , seq: 748013, time: 2014-06-12 14:46:45.083345, event: done, op: osd_sub_op_reply(client.4316.0:147253 3.54b 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.083360 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.083363 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.085954 7f9d82171700 10 _do_op 0x6c326c0 seq 2464 r = 0, finisher 0x9658e80 0x25319b80
2014-06-12 14:46:45.085978 7f9d82171700 10 _finish_op 0x6c326c0 seq 2464 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.085994 7f9d7e96a700  5 _do_op 0x77481e0 seq 2465 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:46:45.086000 7f9d7e96a700 10 _do_transaction on 0x2a37a3c0
2014-06-12 14:46:45.085983 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2717 active+clean] op_commit: 236315
2014-06-12 14:46:45.086008 7f9d7e96a700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.086005 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (1 active)
2014-06-12 14:46:45.086013 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.086017 7f9d7715b700  5 -- op tracker -- , seq: 748008, time: 2014-06-12 14:46:45.086006, event: op_commit, op: osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.086035 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2718 crt=0'0 lcod 173'2717 mlcod 173'2717 active+clean] repop_all_committed: repop tid 236315 all committed 
2014-06-12 14:46:45.086046 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2717 active+clean] eval_repop repgather(0x1c378b40 173'2719 rep_tid=236315 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.086062 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2717 active+clean] log_op_stats osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.025831
2014-06-12 14:46:45.086078 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2717 active+clean] publish_stats_to_osd 173:3619
2014-06-12 14:46:45.086088 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2717 active+clean]  sending commit on repgather(0x1c378b40 173'2719 rep_tid=236315 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) 0x2481d400
2014-06-12 14:46:45.086123 7f9d7715b700  5 -- op tracker -- , seq: 748008, time: 2014-06-12 14:46:45.086118, event: commit_sent, op: osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.086126 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.086135 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2717 active+clean] op_applied: 236315
2014-06-12 14:46:45.086150 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.086158 7f9d7795c700  5 -- op tracker -- , seq: 748008, time: 2014-06-12 14:46:45.086151, event: op_applied, op: osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.086167 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2717 active+clean] op_applied on primary on version 173'2719
2014-06-12 14:46:45.086180 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2717 active+clean] repop_all_applied: repop tid 236315 all applied 
2014-06-12 14:46:45.086189 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2717 active+clean] eval_repop repgather(0x1c378b40 173'2719 rep_tid=236315 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.086202 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.086203 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2717 active+clean] log_op_stats osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.025973
2014-06-12 14:46:45.086219 7f9d7e96a700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.086217 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2717 active+clean] publish_stats_to_osd 173:3620
2014-06-12 14:46:45.086224 7f9d7e96a700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:46:45.086230 7f9d7e96a700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.086227 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2717 active+clean]  requeue_ops 
2014-06-12 14:46:45.086267 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2718 active+clean]  removing repgather(0x1c378b40 173'2719 rep_tid=236315 committed?=1 applied?=1 op=osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.086281 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2718 active+clean]    q front is repgather(0x1c378b40 173'2719 rep_tid=236315 committed?=1 applied?=1 op=osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.086293 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2718 active+clean] remove_repop repgather(0x1c378b40 173'2719 rep_tid=236315 committed?=1 applied?=1 op=osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.086304 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2718 active+clean]  requeue_ops 
2014-06-12 14:46:45.086323 7f9d7795c700  5 -- op tracker -- , seq: 748008, time: 2014-06-12 14:46:45.086318, event: done, op: osd_op(client.4316.0:147252 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 3555328~516096] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.086349 7f9d7e96a700 15 _write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 4071424~122880
2014-06-12 14:46:45.086489 7f9d7e96a700 10 lookup_strip_header done  cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.086519 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:45.087113 7f9d7e96a700 10 _generic_write 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 4071424~122880 = 0
2014-06-12 14:46:45.087165 7f9d7e96a700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.087176 7f9d7e96a700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.087181 7f9d7e96a700 15 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3
2014-06-12 14:46:45.087188 7f9d7e96a700 10 _setattrs 3.54b_head/91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 = 0
2014-06-12 14:46:45.087305 7f9d7e96a700 20 set_header setting 17437 cid 3.54b_head oid 91c8954b/rbd_data.10b36b8b4567.0000000000001133/head//3 parent seq 0
2014-06-12 14:46:45.087355 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.087368 7f9d7e96a700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:46:45.087376 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:45.087921 7f9d7e96a700 10 _do_op 0x77481e0 seq 2465 r = 0, finisher 0x288cda0 0x30aeea00
2014-06-12 14:46:45.087932 7f9d7e96a700 10 _finish_op 0x77481e0 seq 2465 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.087949 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (0 active)
2014-06-12 14:46:45.087938 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2718 active+clean] op_commit: 236316
2014-06-12 14:46:45.087954 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.087964 7f9d7715b700  5 -- op tracker -- , seq: 748009, time: 2014-06-12 14:46:45.087956, event: op_commit, op: osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.087974 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2719 crt=0'0 lcod 173'2718 mlcod 173'2718 active+clean] repop_all_committed: repop tid 236316 all committed 
2014-06-12 14:46:45.087985 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2718 active+clean] eval_repop repgather(0x2a37bec0 173'2720 rep_tid=236316 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.088006 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2718 active+clean] log_op_stats osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 123159 outb 0 rlat 0.000000 lat 0.022954
2014-06-12 14:46:45.088020 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2718 active+clean] publish_stats_to_osd 173:3621
2014-06-12 14:46:45.088031 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2718 active+clean]  sending commit on repgather(0x2a37bec0 173'2720 rep_tid=236316 committed?=1 applied?=0 lock=0 op=osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) 0x2e73c500
2014-06-12 14:46:45.088058 7f9d7715b700  5 -- op tracker -- , seq: 748009, time: 2014-06-12 14:46:45.088055, event: commit_sent, op: osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.088126 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2718 active+clean] op_applied: 236316
2014-06-12 14:46:45.088147 7f9d7795c700  5 -- op tracker -- , seq: 748009, time: 2014-06-12 14:46:45.088141, event: op_applied, op: osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.088156 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2718 active+clean] op_applied on primary on version 173'2720
2014-06-12 14:46:45.088169 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2718 active+clean] repop_all_applied: repop tid 236316 all applied 
2014-06-12 14:46:45.088179 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2718 active+clean] eval_repop repgather(0x2a37bec0 173'2720 rep_tid=236316 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.088197 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2718 active+clean] log_op_stats osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4 inb 123159 outb 0 rlat 0.000000 lat 0.023145
2014-06-12 14:46:45.088215 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2718 active+clean] publish_stats_to_osd 173:3622
2014-06-12 14:46:45.088227 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2718 active+clean]  requeue_ops 
2014-06-12 14:46:45.088240 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]  removing repgather(0x2a37bec0 173'2720 rep_tid=236316 committed?=1 applied?=1 op=osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.088256 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]    q front is repgather(0x2a37bec0 173'2720 rep_tid=236316 committed?=1 applied?=1 op=osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.088272 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] remove_repop repgather(0x2a37bec0 173'2720 rep_tid=236316 committed?=1 applied?=1 op=osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.088288 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]  requeue_ops 
2014-06-12 14:46:45.088315 7f9d7795c700  5 -- op tracker -- , seq: 748009, time: 2014-06-12 14:46:45.088311, event: done, op: osd_op(client.4316.0:147253 rbd_data.10b36b8b4567.0000000000001133 [set-alloc-hint object_size 4194304 write_size 4194304,write 4071424~122880] 3.91c8954b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.097965 7f9d5501e700  5 -- op tracker -- , seq: 748014, time: 2014-06-12 14:46:45.097893, event: header_read, op: osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.097981 7f9d5501e700  5 -- op tracker -- , seq: 748014, time: 2014-06-12 14:46:45.097898, event: throttled, op: osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.097990 7f9d5501e700  5 -- op tracker -- , seq: 748014, time: 2014-06-12 14:46:45.097930, event: all_read, op: osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.097999 7f9d5501e700  5 -- op tracker -- , seq: 748014, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.098005 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.098012 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:45.098022 7f9d5501e700 15 osd.2 173 enqueue_op 0x9facd00 prio 127 cost 4903 latency 0.000128 osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.098049 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.098056 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x9facd00 prio 127 cost 4903 latency 0.000162 osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.591( v 173'3253 (125'252,173'3253] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3252 active]
2014-06-12 14:46:45.098075 7f9d6a3e5700  5 -- op tracker -- , seq: 748014, time: 2014-06-12 14:46:45.098072, event: reached_pg, op: osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.098090 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3253 (125'252,173'3253] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3252 active] handle_message: 0x9facd00
2014-06-12 14:46:45.098101 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3253 (125'252,173'3253] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3252 active] sub_op_modify trans be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 v 173'3254 (transaction) 174
2014-06-12 14:46:45.098112 7f9d6a3e5700  5 -- op tracker -- , seq: 748014, time: 2014-06-12 14:46:45.098109, event: started, op: osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.098121 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3253 (125'252,173'3253] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3252 active] append_log log((125'252,173'3253], crt=0'0) [173'3254 (173'3253) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145586 2014-06-12 08:45:47.477054]
2014-06-12 14:46:45.098138 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3254 (125'252,173'3254] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3252 active] add_log_entry 173'3254 (173'3253) modify   be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 by client.4298.0:145586 2014-06-12 08:45:47.477054
2014-06-12 14:46:45.098153 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3254 (125'252,173'3254] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3252 active] append_log  adding 1 keys
2014-06-12 14:46:45.098162 7f9d6a3e5700 10 trim log((125'252,173'3254], crt=0'0) to 125'253
2014-06-12 14:46:45.098179 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3254, trimmed: 125'253
2014-06-12 14:46:45.098214 7f9d6a3e5700  5 -- op tracker -- , seq: 748014, time: 2014-06-12 14:46:45.098210, event: started, op: osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.098220 7f9d6a3e5700  5 queue_transactions existing osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:45.098223 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.098224 7f9d6a3e5700  5 queue_op 0x2728a5a0 seq 2906 osr(3.591 0x276da10) 6424 bytes   (queue has 1 ops and 6424 bytes)
2014-06-12 14:46:45.098232 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438120 (1 active)
2014-06-12 14:46:45.098229 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.591( v 173'3254 (125'253,173'3254] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3252 active] do_sub_op osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.098238 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x9facd00 finish
2014-06-12 14:46:45.098238 7f9d7d167700  5 _do_op 0x2728a5a0 seq 2906 osr(3.591 0x276da10)/0x276da10 start
2014-06-12 14:46:45.098240 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.098241 7f9d7d167700 10 _do_transaction on 0x30b06d30
2014-06-12 14:46:45.098242 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.098246 7f9d7d167700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:45.098352 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:45.098382 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.098419 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.098515 7f9d7d167700 15 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:45.098522 7f9d7d167700 10 _omap_rmkeys meta/1fe10eca/pglog_3.591/0//-1 r = 0
2014-06-12 14:46:45.098527 7f9d7d167700 15 _omap_setkeys meta/1fe10eca/pglog_3.591/0//-1
2014-06-12 14:46:45.098540 7f9d7d167700 15 _write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 3616768~4096
2014-06-12 14:46:45.098650 7f9d7d167700 10 lookup_strip_header done  cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:45.098783 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:45.098800 7f9d7d167700 10 _generic_write 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 3616768~4096 = 0
2014-06-12 14:46:45.098807 7f9d7d167700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:45.098813 7f9d7d167700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:45.098816 7f9d7d167700 15 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3
2014-06-12 14:46:45.098821 7f9d7d167700 10 _setattrs 3.591_head/be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 = 0
2014-06-12 14:46:45.098892 7f9d7d167700 20 set_header setting 17486 cid 3.591_head oid be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 parent seq 0
2014-06-12 14:46:45.098905 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.098915 7f9d7d167700 20 set_header setting 2292 cid meta oid 1fe10eca/pglog_3.591/0//-1 parent seq 0
2014-06-12 14:46:45.098923 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:45.098987 7f9d7d167700 10 _do_op 0x2728a5a0 seq 2906 r = 0, finisher 0x7811120 0
2014-06-12 14:46:45.098994 7f9d7d167700 10 _finish_op 0x2728a5a0 seq 2906 osr(3.591 0x276da10)/0x276da10
2014-06-12 14:46:45.098998 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438120 (0 active)
2014-06-12 14:46:45.098999 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.099004 7f9d7715b700  5 -- op tracker -- , seq: 748014, time: 2014-06-12 14:46:45.098999, event: commit_sent, op: osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.099009 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3254 (125'253,173'3254] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3252 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:45.099022 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:45.099045 7f9d7795c700  5 -- op tracker -- , seq: 748014, time: 2014-06-12 14:46:45.099040, event: sub_op_applied, op: osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.099052 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3254 (125'253,173'3254] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3253 active] sub_op_modify_applied on 0x30b06c40 op osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.099070 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3254 (125'253,173'3254] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3253 active] op_applied on primary on version 173'3254
2014-06-12 14:46:45.099081 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.591( v 173'3254 (125'253,173'3254] local-les=161 n=325 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3253 active] op_applied on replica on version 173'3254
2014-06-12 14:46:45.099105 7f9d7795c700  5 -- op tracker -- , seq: 748014, time: 2014-06-12 14:46:45.099101, event: done, op: osd_sub_op(client.4298.0:145586 3.591 be7f2591/rbd_data.10a16b8b4567.0000000000001048/head//3 [] v 173'3254 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.114066 7f9d56432700  5 -- op tracker -- , seq: 748015, time: 2014-06-12 14:46:45.112932, event: header_read, op: osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.114083 7f9d56432700  5 -- op tracker -- , seq: 748015, time: 2014-06-12 14:46:45.112935, event: throttled, op: osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.114093 7f9d56432700  5 -- op tracker -- , seq: 748015, time: 2014-06-12 14:46:45.113997, event: all_read, op: osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.114101 7f9d56432700  5 -- op tracker -- , seq: 748015, time: 0.000000, event: dispatched, op: osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.114107 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.114115 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:45.114126 7f9d56432700 15 osd.2 173 enqueue_op 0x34016d00 prio 127 cost 430887 latency 0.001193 osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.114284 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.114294 7f9d69be4700 10 osd.2 173 dequeue_op 0x34016d00 prio 127 cost 430887 latency 0.001362 osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2642 (0'0,173'2642] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active]
2014-06-12 14:46:45.114317 7f9d69be4700  5 -- op tracker -- , seq: 748015, time: 2014-06-12 14:46:45.114313, event: reached_pg, op: osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.114327 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2642 (0'0,173'2642] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] handle_message: 0x34016d00
2014-06-12 14:46:45.114341 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2642 (0'0,173'2642] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] sub_op_modify trans e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 v 173'2643 (transaction) 174
2014-06-12 14:46:45.114358 7f9d69be4700  5 -- op tracker -- , seq: 748015, time: 2014-06-12 14:46:45.114354, event: started, op: osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.114377 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2642 (0'0,173'2642] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] append_log log((0'0,173'2642], crt=0'0) [173'2643 (173'2642) modify   e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 by client.4292.0:142427 2014-06-12 08:44:58.267097]
2014-06-12 14:46:45.114405 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2643 (0'0,173'2643] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] add_log_entry 173'2643 (173'2642) modify   e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 by client.4292.0:142427 2014-06-12 08:44:58.267097
2014-06-12 14:46:45.114430 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2643 (0'0,173'2643] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] append_log  adding 1 keys
2014-06-12 14:46:45.114477 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2643, trimmed: 
2014-06-12 14:46:45.114510 7f9d69be4700  5 -- op tracker -- , seq: 748015, time: 2014-06-12 14:46:45.114503, event: started, op: osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.114527 7f9d69be4700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:45.114536 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.114539 7f9d69be4700  5 queue_op 0x5207bc0 seq 2463 osr(3.1c7 0x276d170) 432373 bytes   (queue has 1 ops and 432373 bytes)
2014-06-12 14:46:45.114554 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2643 (0'0,173'2643] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] do_sub_op osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.114573 7f9d69be4700 10 osd.2 173 dequeue_op 0x34016d00 finish
2014-06-12 14:46:45.114577 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.114582 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.114597 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:46:45.114607 7f9d8116f700  5 _do_op 0x5207bc0 seq 2463 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:46:45.114618 7f9d8116f700 10 _do_transaction on 0x5ba80f0
2014-06-12 14:46:45.114633 7f9d8116f700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:45.114818 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:45.114846 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.114898 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.115059 7f9d8116f700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:45.115067 7f9d8116f700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:46:45.115073 7f9d8116f700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:45.115097 7f9d8116f700 15 _write 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 3764224~430080
2014-06-12 14:46:45.115247 7f9d8116f700 10 lookup_strip_header done  cid 3.1c7_head oid e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:45.115267 7f9d461e0700  5 -- op tracker -- , seq: 748016, time: 2014-06-12 14:46:45.113909, event: header_read, op: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.115296 7f9d461e0700  5 -- op tracker -- , seq: 748016, time: 2014-06-12 14:46:45.113914, event: throttled, op: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.115305 7f9d461e0700  5 -- op tracker -- , seq: 748016, time: 2014-06-12 14:46:45.115233, event: all_read, op: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.115313 7f9d461e0700  5 -- op tracker -- , seq: 748016, time: 0.000000, event: dispatched, op: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.115319 7f9d461e0700 20 osd.2 173 should_share_map client.4293 172.16.96.220:0/1241966 173
2014-06-12 14:46:45.115340 7f9d461e0700 15 osd.2 173 enqueue_op 0x20e89800 prio 63 cost 516096 latency 0.001431 osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.115370 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.115378 7f9d653db700 10 osd.2 173 dequeue_op 0x20e89800 prio 63 cost 516096 latency 0.001469 osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]
2014-06-12 14:46:45.115403 7f9d653db700  5 -- op tracker -- , seq: 748016, time: 2014-06-12 14:46:45.115400, event: reached_pg, op: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.115410 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.115421 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] handle_message: 0x20e89800
2014-06-12 14:46:45.115429 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:45.115427 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_op osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.115445 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] get_object_context: found obc in cache: 0x20107600
2014-06-12 14:46:45.115452 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] get_object_context: 0x20107600 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 rwstate(write n=2 w=0) oi: e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3(173'2592 client.4293.0:144209 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2592) ssc: 0x72b8d20 snapset: 0=[]:[]+head
2014-06-12 14:46:45.115467 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] find_object_context e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 @head oi=e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3(173'2592 client.4293.0:144209 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2592)
2014-06-12 14:46:45.115486 7f9d653db700  5 -- op tracker -- , seq: 748016, time: 2014-06-12 14:46:45.115483, event: started, op: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.115491 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] execute_ctx 0x55a5000
2014-06-12 14:46:45.115502 7f9d653db700  5 -- op tracker -- , seq: 748016, time: 2014-06-12 14:46:45.115499, event: started, op: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.115505 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_op e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] ov 173'2592 av 173'2593 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.115513 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096]
2014-06-12 14:46:45.115520 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.115528 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op  write 2752512~516096
2014-06-12 14:46:45.115538 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op_effects on session 0x5c8cc00
2014-06-12 14:46:45.115545 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] make_writeable e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 snapset=0x72b8d58  snapc=0=[]
2014-06-12 14:46:45.115552 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] make_writeable e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.115558 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] finish_ctx e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 0x55a5000 op modify  
2014-06-12 14:46:45.115565 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  set mtime to 2014-06-12 08:45:47.492877
2014-06-12 14:46:45.115578 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  final snapset 0=[]:[]+head in e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.115589 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.115598 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] new_repop rep_tid 236317 on osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.115615 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] issue_repop rep_tid 236317 o e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.115634 7f9d653db700  5 -- op tracker -- , seq: 748016, time: 2014-06-12 14:46:45.115630, event: waiting for subops from 24, op: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.115652 7f9d653db700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:45.115670 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2592 (0'0,173'2592] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] append_log log((0'0,173'2592], crt=0'0) [173'2593 (173'2592) modify   e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 by client.4293.0:144210 2014-06-12 08:45:47.492877]
2014-06-12 14:46:45.115692 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] add_log_entry 173'2593 (173'2592) modify   e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 by client.4293.0:144210 2014-06-12 08:45:47.492877
2014-06-12 14:46:45.115710 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.115737 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2593, trimmed: 
2014-06-12 14:46:45.115756 7f9d653db700  5 queue_transactions existing osr(3.4a7 0x276d8c0)/0x276d8c0
2014-06-12 14:46:45.115760 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.115761 7f9d653db700  5 queue_op 0x287fe60 seq 2339 osr(3.4a7 0x276d8c0) 518389 bytes   (queue has 2 ops and 950762 bytes)
2014-06-12 14:46:45.115770 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] eval_repop repgather(0x224fe00 173'2593 rep_tid=236317 committed?=0 applied?=0 lock=0 op=osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.115786 7f9d653db700 10 osd.2 173 dequeue_op 0x20e89800 finish
2014-06-12 14:46:45.115789 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.115792 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.117421 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5417b00 (2 active)
2014-06-12 14:46:45.117419 7f9d8116f700 10 _generic_write 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 3764224~430080 = 0
2014-06-12 14:46:45.117437 7f9d79960700  5 _do_op 0x287fe60 seq 2339 osr(3.4a7 0x276d8c0)/0x276d8c0 start
2014-06-12 14:46:45.117443 7f9d79960700 10 _do_transaction on 0x1a445c80
2014-06-12 14:46:45.117451 7f9d79960700 15 _omap_setkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:45.117488 7f9d8116f700 15 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:45.117503 7f9d8116f700 10 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:45.117521 7f9d8116f700 15 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3
2014-06-12 14:46:45.117530 7f9d8116f700 10 _setattrs 3.1c7_head/e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 = 0
2014-06-12 14:46:45.117660 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:45.117689 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.117815 7f9d8116f700 20 set_header setting 18233 cid 3.1c7_head oid e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 parent seq 0
2014-06-12 14:46:45.117904 7f9d8116f700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:46:45.117919 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.117927 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:45.118257 7f9d411d2700  5 -- op tracker -- , seq: 748017, time: 2014-06-12 14:46:45.113991, event: header_read, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.118272 7f9d411d2700  5 -- op tracker -- , seq: 748017, time: 2014-06-12 14:46:45.113993, event: throttled, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.118283 7f9d411d2700  5 -- op tracker -- , seq: 748017, time: 2014-06-12 14:46:45.118219, event: all_read, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.118292 7f9d411d2700  5 -- op tracker -- , seq: 748017, time: 0.000000, event: dispatched, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.118298 7f9d411d2700 20 osd.2 173 should_share_map client.4306 172.16.96.220:0/1243907 173
2014-06-12 14:46:45.118327 7f9d411d2700 15 osd.2 173 enqueue_op 0x59dd400 prio 63 cost 479232 latency 0.004336 osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.118373 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.118384 7f9d63bd8700 10 osd.2 173 dequeue_op 0x59dd400 prio 63 cost 479232 latency 0.004393 osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]
2014-06-12 14:46:45.118414 7f9d63bd8700  5 -- op tracker -- , seq: 748017, time: 2014-06-12 14:46:45.118408, event: reached_pg, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.118424 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.118435 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] handle_message: 0x59dd400
2014-06-12 14:46:45.118442 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.118460 7f9d63bd8700 15 getattr 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 '_'
2014-06-12 14:46:45.118782 7f9d63bd8700 10 getattr 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 '_' = 0
2014-06-12 14:46:45.118802 7f9d63bd8700 15 getattr 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 'snapset'
2014-06-12 14:46:45.118880 7f9d63bd8700 10 getattr 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 'snapset' = 0
2014-06-12 14:46:45.118887 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] populate_obc_watchers 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.118895 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.118901 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: creating obc from disk: 0x3bce9600
2014-06-12 14:46:45.118907 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: 0x3bce9600 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 rwstate(none n=0 w=0) oi: 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3(173'553 client.4306.0:38549 wrlock_by=unknown.0.0:0 dirty s 4194304 uv553) ssc: 0x407ea80 snapset: 0=[]:[]+head
2014-06-12 14:46:45.118921 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] find_object_context 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 @head oi=3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3(173'553 client.4306.0:38549 wrlock_by=unknown.0.0:0 dirty s 4194304 uv553)
2014-06-12 14:46:45.118939 7f9d63bd8700  5 -- op tracker -- , seq: 748017, time: 2014-06-12 14:46:45.118935, event: started, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.118948 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] execute_ctx 0x60aa000
2014-06-12 14:46:45.118966 7f9d63bd8700  5 -- op tracker -- , seq: 748017, time: 2014-06-12 14:46:45.118960, event: started, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.118973 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] ov 173'553 av 173'2448 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.118988 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232]
2014-06-12 14:46:45.119003 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.119012 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  write 0~479232
2014-06-12 14:46:45.119023 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op_effects on session 0x53ac600
2014-06-12 14:46:45.119030 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 snapset=0x407eab8  snapc=0=[]
2014-06-12 14:46:45.119037 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.119043 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] finish_ctx 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 0x60aa000 op modify  
2014-06-12 14:46:45.119052 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  set mtime to 2014-06-12 08:45:47.493211
2014-06-12 14:46:45.119065 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  final snapset 0=[]:[]+head in 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.119078 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.119084 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] new_repop rep_tid 236318 on osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.119095 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] issue_repop rep_tid 236318 o 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.119112 7f9d63bd8700  5 -- op tracker -- , seq: 748017, time: 2014-06-12 14:46:45.119109, event: waiting for subops from 26, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.119127 7f9d63bd8700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.119145 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2447 (0'0,173'2447] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log log((0'0,173'2447], crt=0'0) [173'2448 (173'553) modify   3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 by client.4306.0:144193 2014-06-12 08:45:47.493211]
2014-06-12 14:46:45.119167 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] add_log_entry 173'2448 (173'553) modify   3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 by client.4306.0:144193 2014-06-12 08:45:47.493211
2014-06-12 14:46:45.119185 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.119214 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2448, trimmed: 
2014-06-12 14:46:45.119233 7f9d63bd8700  5 queue_transactions existing osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:45.119237 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.119238 7f9d63bd8700  5 queue_op 0x886f740 seq 2265 osr(3.251 0x276d390) 481525 bytes   (queue has 3 ops and 1432287 bytes)
2014-06-12 14:46:45.119245 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] eval_repop repgather(0x52d1c80 173'2448 rep_tid=236318 committed?=0 applied?=0 lock=0 op=osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.119262 7f9d63bd8700 10 osd.2 173 dequeue_op 0x59dd400 finish
2014-06-12 14:46:45.119266 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.119269 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.119277 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fc20 (3 active)
2014-06-12 14:46:45.119281 7f9d8096e700  5 _do_op 0x886f740 seq 2265 osr(3.251 0x276d390)/0x276d390 start
2014-06-12 14:46:45.119284 7f9d8096e700 10 _do_transaction on 0x5322780
2014-06-12 14:46:45.119289 7f9d8096e700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.119370 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.119384 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.119503 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.119534 7f9d79960700 15 _omap_rmkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:45.119540 7f9d79960700 10 _omap_rmkeys meta/1fe06fa5/pglog_3.4a7/0//-1 r = 0
2014-06-12 14:46:45.119547 7f9d79960700 15 _omap_setkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:45.119660 7f9d79960700 15 _write 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 2752512~516096
2014-06-12 14:46:45.119821 7f9d79960700 10 lookup_strip_header done  cid 3.4a7_head oid e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.119954 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:45.120218 7f9d56432700  5 -- op tracker -- , seq: 748018, time: 2014-06-12 14:46:45.114153, event: header_read, op: osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.120227 7f9d56432700  5 -- op tracker -- , seq: 748018, time: 2014-06-12 14:46:45.114153, event: throttled, op: osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.120234 7f9d56432700  5 -- op tracker -- , seq: 748018, time: 2014-06-12 14:46:45.120186, event: all_read, op: osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.120245 7f9d56432700  5 -- op tracker -- , seq: 748018, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.120250 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.120256 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:45.120263 7f9d56432700 15 osd.2 173 enqueue_op 0x2fa9c300 prio 127 cost 516903 latency 0.006109 osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.120292 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.120299 7f9d68be2700 10 osd.2 173 dequeue_op 0x2fa9c300 prio 127 cost 516903 latency 0.006146 osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.71f( v 173'2700 (0'0,173'2700] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2699 active]
2014-06-12 14:46:45.120319 7f9d68be2700  5 -- op tracker -- , seq: 748018, time: 2014-06-12 14:46:45.120315, event: reached_pg, op: osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.120551 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2700 (0'0,173'2700] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2699 active] handle_message: 0x2fa9c300
2014-06-12 14:46:45.120571 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2700 (0'0,173'2700] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2699 active] sub_op_modify trans c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 v 173'2701 (transaction) 174
2014-06-12 14:46:45.120588 7f9d68be2700  5 -- op tracker -- , seq: 748018, time: 2014-06-12 14:46:45.120584, event: started, op: osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.121154 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2700 (0'0,173'2700] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2699 active] append_log log((0'0,173'2700], crt=0'0) [173'2701 (173'2700) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145288 2014-06-12 08:45:27.694428]
2014-06-12 14:46:45.121180 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2701 (0'0,173'2701] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2699 active] add_log_entry 173'2701 (173'2700) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145288 2014-06-12 08:45:27.694428
2014-06-12 14:46:45.121200 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2701 (0'0,173'2701] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2699 active] append_log  adding 1 keys
2014-06-12 14:46:45.121228 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2701, trimmed: 
2014-06-12 14:46:45.121251 7f9d68be2700  5 -- op tracker -- , seq: 748018, time: 2014-06-12 14:46:45.121246, event: started, op: osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.121260 7f9d68be2700  5 queue_transactions existing osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:45.121264 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.121265 7f9d68be2700  5 queue_op 0x53d3140 seq 2533 osr(3.71f 0x276de50) 518389 bytes   (queue has 4 ops and 1950676 bytes)
2014-06-12 14:46:45.121272 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.71f( v 173'2701 (0'0,173'2701] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2699 active] do_sub_op osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.121287 7f9d68be2700 10 osd.2 173 dequeue_op 0x2fa9c300 finish
2014-06-12 14:46:45.121290 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.121294 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.121319 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d0240 (4 active)
2014-06-12 14:46:45.121326 7f9d8016d700  5 _do_op 0x53d3140 seq 2533 osr(3.71f 0x276de50)/0x276de50 start
2014-06-12 14:46:45.121331 7f9d8016d700 10 _do_transaction on 0x1b39c2b0
2014-06-12 14:46:45.121338 7f9d8016d700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:45.121474 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:45.121492 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.121628 7f9d8116f700 10 _do_op 0x5207bc0 seq 2463 r = 0, finisher 0x82c9940 0
2014-06-12 14:46:45.121639 7f9d8116f700 10 _finish_op 0x5207bc0 seq 2463 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:45.121645 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (3 active)
2014-06-12 14:46:45.121648 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.121664 7f9d7715b700  5 -- op tracker -- , seq: 748015, time: 2014-06-12 14:46:45.121657, event: commit_sent, op: osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.121677 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2643 (0'0,173'2643] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2641 active] sub_op_modify_commit on op osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:45.121697 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.121896 7f9d79960700 10 _generic_write 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 2752512~516096 = 0
2014-06-12 14:46:45.121940 7f9d79960700 15 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.121952 7f9d79960700 10 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 = 0
2014-06-12 14:46:45.121958 7f9d79960700 15 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.121964 7f9d79960700 10 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 = 0
2014-06-12 14:46:45.123342 7f9d79960700 20 set_header setting 17968 cid 3.4a7_head oid e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 parent seq 0
2014-06-12 14:46:45.123589 7f9d79960700 20 set_header setting 2078 cid meta oid 1fe06fa5/pglog_3.4a7/0//-1 parent seq 0
2014-06-12 14:46:45.123608 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.123618 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:45.125653 7f9d7795c700  5 -- op tracker -- , seq: 748015, time: 2014-06-12 14:46:45.125641, event: sub_op_applied, op: osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.125672 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2643 (0'0,173'2643] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] sub_op_modify_applied on 0x5ba8000 op osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.125701 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2643 (0'0,173'2643] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] op_applied on primary on version 173'2643
2014-06-12 14:46:45.125712 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2643 (0'0,173'2643] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] op_applied on replica on version 173'2643
2014-06-12 14:46:45.125757 7f9d7795c700  5 -- op tracker -- , seq: 748015, time: 2014-06-12 14:46:45.125752, event: done, op: osd_sub_op(client.4292.0:142427 3.1c7 e9a3d1c7/rbd_data.109b6b8b4567.000000000000104c/head//3 [] v 173'2643 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.125815 7f9d56432700  5 -- op tracker -- , seq: 748019, time: 2014-06-12 14:46:45.120515, event: header_read, op: osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.125827 7f9d56432700  5 -- op tracker -- , seq: 748019, time: 2014-06-12 14:46:45.120517, event: throttled, op: osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.125836 7f9d56432700  5 -- op tracker -- , seq: 748019, time: 2014-06-12 14:46:45.125573, event: all_read, op: osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.125845 7f9d56432700  5 -- op tracker -- , seq: 748019, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.125853 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.125860 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:45.125875 7f9d56432700 15 osd.2 173 enqueue_op 0x12bf6100 prio 127 cost 95015 latency 0.005359 osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.125915 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.125923 7f9d623d5700 10 osd.2 173 dequeue_op 0x12bf6100 prio 127 cost 95015 latency 0.005407 osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.582( v 173'2763 (0'0,173'2763] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2762 active]
2014-06-12 14:46:45.125945 7f9d623d5700  5 -- op tracker -- , seq: 748019, time: 2014-06-12 14:46:45.125940, event: reached_pg, op: osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.125957 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2763 (0'0,173'2763] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2762 active] handle_message: 0x12bf6100
2014-06-12 14:46:45.125972 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2763 (0'0,173'2763] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2762 active] sub_op_modify trans 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 v 173'2764 (transaction) 174
2014-06-12 14:46:45.125991 7f9d623d5700  5 -- op tracker -- , seq: 748019, time: 2014-06-12 14:46:45.125987, event: started, op: osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.126008 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2763 (0'0,173'2763] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2762 active] append_log log((0'0,173'2763], crt=0'0) [173'2764 (173'2763) modify   264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 by client.4317.0:143788 2014-06-12 08:44:59.114995]
2014-06-12 14:46:45.126035 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2764 (0'0,173'2764] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2762 active] add_log_entry 173'2764 (173'2763) modify   264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 by client.4317.0:143788 2014-06-12 08:44:59.114995
2014-06-12 14:46:45.126062 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2764 (0'0,173'2764] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2762 active] append_log  adding 1 keys
2014-06-12 14:46:45.126113 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2764, trimmed: 
2014-06-12 14:46:45.126139 7f9d623d5700  5 -- op tracker -- , seq: 748019, time: 2014-06-12 14:46:45.126134, event: started, op: osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.126149 7f9d623d5700  5 queue_transactions existing osr(3.582 0x276da70)/0x276da70
2014-06-12 14:46:45.126154 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.126155 7f9d623d5700  5 queue_op 0x64788a0 seq 2612 osr(3.582 0x276da70) 96501 bytes   (queue has 4 ops and 1614804 bytes)
2014-06-12 14:46:45.126163 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.582( v 173'2764 (0'0,173'2764] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2762 active] do_sub_op osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.126178 7f9d623d5700 10 osd.2 173 dequeue_op 0x12bf6100 finish
2014-06-12 14:46:45.126182 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.126184 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.126193 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438900 (4 active)
2014-06-12 14:46:45.126199 7f9d7915f700  5 _do_op 0x64788a0 seq 2612 osr(3.582 0x276da70)/0x276da70 start
2014-06-12 14:46:45.126203 7f9d7915f700 10 _do_transaction on 0x1cef7b30
2014-06-12 14:46:45.126209 7f9d7915f700 15 _omap_setkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:45.126392 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:45.126413 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.135713 7f9d79960700 10 _do_op 0x287fe60 seq 2339 r = 0, finisher 0x824a260 0x28eace00
2014-06-12 14:46:45.135745 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_commit: 236317
2014-06-12 14:46:45.135775 7f9d7715b700  5 -- op tracker -- , seq: 748016, time: 2014-06-12 14:46:45.135767, event: op_commit, op: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.135791 7f9d79960700 10 _finish_op 0x287fe60 seq 2339 osr(3.4a7 0x276d8c0)/0x276d8c0
2014-06-12 14:46:45.135801 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5417b00 (3 active)
2014-06-12 14:46:45.135805 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.135808 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_applied: 236317
2014-06-12 14:46:45.135833 7f9d7795c700  5 -- op tracker -- , seq: 748016, time: 2014-06-12 14:46:45.135826, event: op_applied, op: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.135842 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_applied on primary on version 173'2593
2014-06-12 14:46:45.137783 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.137826 7f9d8016d700 15 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:45.137845 7f9d8016d700 10 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1 r = 0
2014-06-12 14:46:45.137852 7f9d8016d700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:45.137871 7f9d8016d700 15 _write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 3346432~516096
2014-06-12 14:46:45.138007 7f9d8016d700 10 lookup_strip_header done  cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:45.138268 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:45.140362 7f9d8016d700 10 _generic_write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 3346432~516096 = 0
2014-06-12 14:46:45.140453 7f9d8016d700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:45.140470 7f9d8016d700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:45.140477 7f9d8016d700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:45.140485 7f9d8016d700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:45.140697 7f9d8016d700 20 set_header setting 17162 cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 parent seq 0
2014-06-12 14:46:45.140789 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.140806 7f9d8016d700 20 set_header setting 2330 cid meta oid 1fe0985b/pglog_3.71f/0//-1 parent seq 0
2014-06-12 14:46:45.140817 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:45.142550 7f9d5a16f700  5 -- op tracker -- , seq: 748020, time: 2014-06-12 14:46:45.142436, event: header_read, op: osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.142563 7f9d5a16f700  5 -- op tracker -- , seq: 748020, time: 2014-06-12 14:46:45.142438, event: throttled, op: osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.142571 7f9d5a16f700  5 -- op tracker -- , seq: 748020, time: 2014-06-12 14:46:45.142506, event: all_read, op: osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.142579 7f9d5a16f700  5 -- op tracker -- , seq: 748020, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.142585 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.142591 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:45.142599 7f9d5a16f700 15 osd.2 173 enqueue_op 0x34017a00 prio 127 cost 33575 latency 0.000163 osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.142638 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.142647 7f9d643d9700 10 osd.2 173 dequeue_op 0x34017a00 prio 127 cost 33575 latency 0.000210 osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2503 (0'0,173'2503] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2502 active]
2014-06-12 14:46:45.142670 7f9d643d9700  5 -- op tracker -- , seq: 748020, time: 2014-06-12 14:46:45.142666, event: reached_pg, op: osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.142680 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2503 (0'0,173'2503] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2502 active] handle_message: 0x34017a00
2014-06-12 14:46:45.142692 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2503 (0'0,173'2503] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2502 active] sub_op_modify trans a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 v 173'2504 (transaction) 174
2014-06-12 14:46:45.142720 7f9d643d9700  5 -- op tracker -- , seq: 748020, time: 2014-06-12 14:46:45.142715, event: started, op: osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.142736 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2503 (0'0,173'2503] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2502 active] append_log log((0'0,173'2503], crt=0'0) [173'2504 (173'569) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146439 2014-06-12 08:41:36.550120]
2014-06-12 14:46:45.142759 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2504 (0'0,173'2504] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2502 active] add_log_entry 173'2504 (173'569) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146439 2014-06-12 08:41:36.550120
2014-06-12 14:46:45.142786 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2504 (0'0,173'2504] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2502 active] append_log  adding 1 keys
2014-06-12 14:46:45.142820 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2504, trimmed: 
2014-06-12 14:46:45.142842 7f9d643d9700  5 -- op tracker -- , seq: 748020, time: 2014-06-12 14:46:45.142837, event: started, op: osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.142852 7f9d643d9700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:46:45.142856 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.142858 7f9d643d9700  5 queue_op 0x69b0ae0 seq 2350 osr(3.168 0x276ce40) 35061 bytes   (queue has 4 ops and 1131476 bytes)
2014-06-12 14:46:45.142871 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (4 active)
2014-06-12 14:46:45.142867 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2504 (0'0,173'2504] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2502 active] do_sub_op osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.142880 7f9d7d968700  5 _do_op 0x69b0ae0 seq 2350 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:46:45.142881 7f9d643d9700 10 osd.2 173 dequeue_op 0x34017a00 finish
2014-06-12 14:46:45.142885 7f9d7d968700 10 _do_transaction on 0x5a422b0
2014-06-12 14:46:45.142885 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.142888 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.142892 7f9d7d968700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:45.143058 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:45.143083 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.143629 7f9d8016d700 10 _do_op 0x53d3140 seq 2533 r = 0, finisher 0x8d88100 0
2014-06-12 14:46:45.143659 7f9d7715b700  5 -- op tracker -- , seq: 748018, time: 2014-06-12 14:46:45.143652, event: commit_sent, op: osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.143669 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2701 (0'0,173'2701] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2699 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:45.143693 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.143734 7f9d8016d700 10 _finish_op 0x53d3140 seq 2533 osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:45.143759 7f9d7795c700  5 -- op tracker -- , seq: 748018, time: 2014-06-12 14:46:45.143754, event: sub_op_applied, op: osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.143768 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2701 (0'0,173'2701] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2700 active] sub_op_modify_applied on 0x1b39c1c0 op osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.143785 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2701 (0'0,173'2701] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2700 active] op_applied on primary on version 173'2701
2014-06-12 14:46:45.143795 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2701 (0'0,173'2701] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2700 active] op_applied on replica on version 173'2701
2014-06-12 14:46:45.143826 7f9d8016d700  5 -- op tracker -- , seq: 748018, time: 2014-06-12 14:46:45.143817, event: done, op: osd_sub_op(client.4308.0:145288 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.143862 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d0240 (3 active)
2014-06-12 14:46:45.143865 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.144037 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.144125 7f9d7915f700 15 _omap_rmkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:45.144132 7f9d7915f700 10 _omap_rmkeys meta/1fe177ea/pglog_3.582/0//-1 r = 0
2014-06-12 14:46:45.144138 7f9d7915f700 15 _omap_setkeys meta/1fe177ea/pglog_3.582/0//-1
2014-06-12 14:46:45.144156 7f9d7915f700 15 _write 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 4100096~94208
2014-06-12 14:46:45.144299 7f9d7915f700 10 lookup_strip_header done  cid 3.582_head oid 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:45.144433 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:45.144734 7f9d7915f700 10 _generic_write 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 4100096~94208 = 0
2014-06-12 14:46:45.144747 7f9d7915f700 15 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:45.144761 7f9d7915f700 10 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:45.144766 7f9d7915f700 15 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3
2014-06-12 14:46:45.144774 7f9d7915f700 10 _setattrs 3.582_head/264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:45.145028 7f9d7915f700 20 set_header setting 16739 cid 3.582_head oid 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 parent seq 0
2014-06-12 14:46:45.145096 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.145109 7f9d7915f700 20 set_header setting 2318 cid meta oid 1fe177ea/pglog_3.582/0//-1 parent seq 0
2014-06-12 14:46:45.145118 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:45.146288 7f9d7915f700 10 _do_op 0x64788a0 seq 2612 r = 0, finisher 0x76884a0 0
2014-06-12 14:46:45.146304 7f9d7915f700 10 _finish_op 0x64788a0 seq 2612 osr(3.582 0x276da70)/0x276da70
2014-06-12 14:46:45.146316 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438900 (2 active)
2014-06-12 14:46:45.146322 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.146359 7f9d7715b700  5 -- op tracker -- , seq: 748019, time: 2014-06-12 14:46:45.146346, event: commit_sent, op: osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.146370 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2764 (0'0,173'2764] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2762 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:45.146397 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.146466 7f9d7795c700  5 -- op tracker -- , seq: 748019, time: 2014-06-12 14:46:45.146460, event: sub_op_applied, op: osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.146473 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2764 (0'0,173'2764] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2763 active] sub_op_modify_applied on 0x1cef7a40 op osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.146489 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2764 (0'0,173'2764] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2763 active] op_applied on primary on version 173'2764
2014-06-12 14:46:45.146498 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.582( v 173'2764 (0'0,173'2764] local-les=153 n=305 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2763 active] op_applied on replica on version 173'2764
2014-06-12 14:46:45.146527 7f9d7795c700  5 -- op tracker -- , seq: 748019, time: 2014-06-12 14:46:45.146522, event: done, op: osd_sub_op(client.4317.0:143788 3.582 264b7d82/rbd_data.10b46b8b4567.0000000000001034/head//3 [] v 173'2764 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.148045 7f9d56432700  5 -- op tracker -- , seq: 748021, time: 2014-06-12 14:46:45.142262, event: header_read, op: osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.148060 7f9d56432700  5 -- op tracker -- , seq: 748021, time: 2014-06-12 14:46:45.142265, event: throttled, op: osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.148068 7f9d56432700  5 -- op tracker -- , seq: 748021, time: 2014-06-12 14:46:45.147997, event: all_read, op: osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.148076 7f9d56432700  5 -- op tracker -- , seq: 748021, time: 0.000000, event: dispatched, op: osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.148082 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.148094 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:45.148104 7f9d56432700 15 osd.2 173 enqueue_op 0x12bf6300 prio 127 cost 516903 latency 0.005842 osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.148657 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.148664 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.148668 7f9d633d7700 10 osd.2 173 dequeue_op 0x12bf6300 prio 127 cost 516903 latency 0.006405 osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.60e( v 173'2561 (0'0,173'2561] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2560 active]
2014-06-12 14:46:45.148695 7f9d633d7700  5 -- op tracker -- , seq: 748021, time: 2014-06-12 14:46:45.148691, event: reached_pg, op: osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.148708 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2561 (0'0,173'2561] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2560 active] handle_message: 0x12bf6300
2014-06-12 14:46:45.148720 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2561 (0'0,173'2561] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2560 active] sub_op_modify trans ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 v 173'2562 (transaction) 174
2014-06-12 14:46:45.148735 7f9d633d7700  5 -- op tracker -- , seq: 748021, time: 2014-06-12 14:46:45.148732, event: started, op: osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.148754 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2561 (0'0,173'2561] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2560 active] append_log log((0'0,173'2561], crt=0'0) [173'2562 (173'2561) modify   ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 by client.4296.0:143674 2014-06-12 08:45:47.337209]
2014-06-12 14:46:45.148778 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2562 (0'0,173'2562] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2560 active] add_log_entry 173'2562 (173'2561) modify   ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 by client.4296.0:143674 2014-06-12 08:45:47.337209
2014-06-12 14:46:45.148800 7f9d7d968700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:45.148808 7f9d7d968700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:46:45.148804 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2562 (0'0,173'2562] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2560 active] append_log  adding 1 keys
2014-06-12 14:46:45.148816 7f9d7d968700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:45.148836 7f9d7d968700 15 _write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 0~32768
2014-06-12 14:46:45.148841 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2562, trimmed: 
2014-06-12 14:46:45.148862 7f9d633d7700  5 -- op tracker -- , seq: 748021, time: 2014-06-12 14:46:45.148858, event: started, op: osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.148872 7f9d633d7700  5 queue_transactions existing osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:45.148876 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.148877 7f9d633d7700  5 queue_op 0x6957560 seq 2404 osr(3.60e 0x276dcf0) 518389 bytes   (queue has 3 ops and 1034975 bytes)
2014-06-12 14:46:45.148885 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.60e( v 173'2562 (0'0,173'2562] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2560 active] do_sub_op osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.148899 7f9d633d7700 10 osd.2 173 dequeue_op 0x12bf6300 finish
2014-06-12 14:46:45.148903 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.148905 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.148912 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533cd80 (3 active)
2014-06-12 14:46:45.148918 7f9d7c165700  5 _do_op 0x6957560 seq 2404 osr(3.60e 0x276dcf0)/0x276dcf0 start
2014-06-12 14:46:45.148922 7f9d7c165700 10 _do_transaction on 0x2720ef0
2014-06-12 14:46:45.148927 7f9d7c165700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.149037 7f9d7d968700 10 lookup_strip_header done  cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:46:45.149040 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.149055 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:45.149063 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.149346 7f9d7d968700 10 _generic_write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 0~32768 = 0
2014-06-12 14:46:45.149360 7f9d7d968700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:46:45.149375 7f9d7d968700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:46:45.149381 7f9d7d968700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:46:45.149389 7f9d7d968700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:46:45.149507 7f9d7d968700 20 set_header setting 16934 cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 parent seq 0
2014-06-12 14:46:45.149545 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.149557 7f9d7d968700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:46:45.149567 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:45.149959 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.149982 7f9d8096e700 15 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.149988 7f9d8096e700 10 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1 r = 0
2014-06-12 14:46:45.149995 7f9d8096e700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.150009 7f9d8096e700 15 _write 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 0~479232
2014-06-12 14:46:45.150154 7f9d8096e700 10 lookup_strip_header done  cid 3.251_head oid 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.150281 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:45.152361 7f9d8096e700 10 _generic_write 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 0~479232 = 0
2014-06-12 14:46:45.152406 7f9d8096e700 15 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.152422 7f9d8096e700 10 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 = 0
2014-06-12 14:46:45.152429 7f9d8096e700 15 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.152438 7f9d8096e700 10 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 = 0
2014-06-12 14:46:45.152681 7f9d8096e700 20 set_header setting 16808 cid 3.251_head oid 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 parent seq 0
2014-06-12 14:46:45.152754 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.152771 7f9d8096e700 20 set_header setting 2026 cid meta oid 1fe2661a/pglog_3.251/0//-1 parent seq 0
2014-06-12 14:46:45.152810 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:45.153309 7f9d7d968700 10 _do_op 0x69b0ae0 seq 2350 r = 0, finisher 0x69606c0 0
2014-06-12 14:46:45.153327 7f9d7d968700 10 _finish_op 0x69b0ae0 seq 2350 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:46:45.153337 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (2 active)
2014-06-12 14:46:45.153341 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.153369 7f9d7715b700  5 -- op tracker -- , seq: 748020, time: 2014-06-12 14:46:45.153358, event: commit_sent, op: osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.153391 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2504 (0'0,173'2504] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2502 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:45.153421 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:45.154078 7f9d7795c700  5 -- op tracker -- , seq: 748020, time: 2014-06-12 14:46:45.154067, event: sub_op_applied, op: osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.154093 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2504 (0'0,173'2504] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2503 active] sub_op_modify_applied on 0x5a421c0 op osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.154112 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2504 (0'0,173'2504] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2503 active] op_applied on primary on version 173'2504
2014-06-12 14:46:45.154122 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2504 (0'0,173'2504] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2503 active] op_applied on replica on version 173'2504
2014-06-12 14:46:45.154165 7f9d7795c700  5 -- op tracker -- , seq: 748020, time: 2014-06-12 14:46:45.154160, event: done, op: osd_sub_op(client.4316.0:146439 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2504 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.156319 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.156371 7f9d7c165700 15 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.156379 7f9d7c165700 10 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1 r = 0
2014-06-12 14:46:45.156389 7f9d7c165700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.156565 7f9d7c165700 15 _write 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 3260416~516096
2014-06-12 14:46:45.156719 7f9d7c165700 10 lookup_strip_header done  cid 3.60e_head oid ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:45.159979 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:45.159982 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:45.159984 7f9d74bfa700 20 osd.2 173 _dispatch 0xa24f860 ping v1
2014-06-12 14:46:45.159986 7f9d74bfa700 10 osd.2 173 ping from client.4293
2014-06-12 14:46:45.159988 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:45.159989 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:45.160568 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:45.162053 7f9d8096e700 10 _do_op 0x886f740 seq 2265 r = 0, finisher 0x6cb9520 0xcb61100
2014-06-12 14:46:45.162083 7f9d8096e700 10 _finish_op 0x886f740 seq 2265 osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:45.162105 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied: 236318
2014-06-12 14:46:45.162142 7f9d7795c700  5 -- op tracker -- , seq: 748017, time: 2014-06-12 14:46:45.162130, event: op_applied, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.162155 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied on primary on version 173'2448
2014-06-12 14:46:45.162205 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fc20 (1 active)
2014-06-12 14:46:45.162199 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_commit: 236318
2014-06-12 14:46:45.162215 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.162224 7f9d7715b700  5 -- op tracker -- , seq: 748017, time: 2014-06-12 14:46:45.162217, event: op_commit, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.163598 7f9d7c165700 10 _generic_write 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 3260416~516096 = 0
2014-06-12 14:46:45.163676 7f9d7c165700 15 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:45.163697 7f9d7c165700 10 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 = 0
2014-06-12 14:46:45.163705 7f9d7c165700 15 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:45.163716 7f9d7c165700 10 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 = 0
2014-06-12 14:46:45.164045 7f9d7c165700 20 set_header setting 17046 cid 3.60e_head oid ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 parent seq 0
2014-06-12 14:46:45.164177 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.164191 7f9d7c165700 20 set_header setting 2240 cid meta oid 1fe13dcb/pglog_3.60e/0//-1 parent seq 0
2014-06-12 14:46:45.164200 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:45.164914 7f9d411d2700  5 -- op tracker -- , seq: 748022, time: 2014-06-12 14:46:45.159780, event: header_read, op: osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.164933 7f9d411d2700  5 -- op tracker -- , seq: 748022, time: 2014-06-12 14:46:45.159787, event: throttled, op: osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.164943 7f9d411d2700  5 -- op tracker -- , seq: 748022, time: 2014-06-12 14:46:45.164874, event: all_read, op: osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.164952 7f9d411d2700  5 -- op tracker -- , seq: 748022, time: 0.000000, event: dispatched, op: osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.164965 7f9d411d2700 20 osd.2 173 should_share_map client.4306 172.16.96.220:0/1243907 173
2014-06-12 14:46:45.164996 7f9d411d2700 15 osd.2 173 enqueue_op 0x59dc200 prio 63 cost 1032192 latency 0.005215 osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.165145 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.165153 7f9d61bd4700 10 osd.2 173 dequeue_op 0x59dc200 prio 63 cost 1032192 latency 0.005373 osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]
2014-06-12 14:46:45.165180 7f9d61bd4700  5 -- op tracker -- , seq: 748022, time: 2014-06-12 14:46:45.165175, event: reached_pg, op: osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.165189 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.165205 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] handle_message: 0x59dc200
2014-06-12 14:46:45.165215 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.165239 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: found obc in cache: 0x3bce9600
2014-06-12 14:46:45.165250 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: 0x3bce9600 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 rwstate(write n=1 w=0) oi: 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3(173'2448 client.4306.0:144193 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2448) ssc: 0x407ea80 snapset: 0=[]:[]+head
2014-06-12 14:46:45.165286 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] find_object_context 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 @head oi=3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3(173'2448 client.4306.0:144193 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2448)
2014-06-12 14:46:45.165312 7f9d61bd4700  5 -- op tracker -- , seq: 748022, time: 2014-06-12 14:46:45.165306, event: started, op: osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.165320 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] execute_ctx 0x26f9f000
2014-06-12 14:46:45.165336 7f9d61bd4700  5 -- op tracker -- , seq: 748022, time: 2014-06-12 14:46:45.165331, event: started, op: osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.165343 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] ov 173'2448 av 173'2449 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.165356 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192]
2014-06-12 14:46:45.165366 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.165381 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  write 479232~1032192
2014-06-12 14:46:45.165395 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op_effects on session 0x53ac600
2014-06-12 14:46:45.165405 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 snapset=0x407eab8  snapc=0=[]
2014-06-12 14:46:45.165417 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.165426 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] finish_ctx 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 0x26f9f000 op modify  
2014-06-12 14:46:45.165439 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  set mtime to 2014-06-12 08:45:47.536528
2014-06-12 14:46:45.165459 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  final snapset 0=[]:[]+head in 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.165477 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.165486 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] new_repop rep_tid 236319 on osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.165501 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] issue_repop rep_tid 236319 o 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.165534 7f9d61bd4700  5 -- op tracker -- , seq: 748022, time: 2014-06-12 14:46:45.165528, event: waiting for subops from 26, op: osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.165561 7f9d61bd4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.167275 7f9d461e0700  5 -- op tracker -- , seq: 748023, time: 2014-06-12 14:46:45.165066, event: header_read, op: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.167289 7f9d461e0700  5 -- op tracker -- , seq: 748023, time: 2014-06-12 14:46:45.165069, event: throttled, op: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.167297 7f9d461e0700  5 -- op tracker -- , seq: 748023, time: 2014-06-12 14:46:45.167215, event: all_read, op: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.167305 7f9d461e0700  5 -- op tracker -- , seq: 748023, time: 0.000000, event: dispatched, op: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.167317 7f9d461e0700 20 osd.2 173 should_share_map client.4293 172.16.96.220:0/1241966 173
2014-06-12 14:46:45.167355 7f9d461e0700 15 osd.2 173 enqueue_op 0x5ee8300 prio 63 cost 516096 latency 0.002289 osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.167393 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:45.165599 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2448 (0'0,173'2448] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log log((0'0,173'2448], crt=0'0) [173'2449 (173'2448) modify   3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 by client.4306.0:144194 2014-06-12 08:45:47.536528]
2014-06-12 14:46:45.167402 7f9d673df700 10 osd.2 173 dequeue_op 0x5ee8300 prio 63 cost 516096 latency 0.002336 osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]
2014-06-12 14:46:45.167432 7f9d673df700  5 -- op tracker -- , seq: 748023, time: 2014-06-12 14:46:45.167427, event: reached_pg, op: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.167423 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] add_log_entry 173'2449 (173'2448) modify   3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 by client.4306.0:144194 2014-06-12 08:45:47.536528
2014-06-12 14:46:45.167442 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.167456 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] handle_message: 0x5ee8300
2014-06-12 14:46:45.167462 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.167466 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_op osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.167491 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] get_object_context: found obc in cache: 0x20107600
2014-06-12 14:46:45.167499 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2449, trimmed: 
2014-06-12 14:46:45.167502 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] get_object_context: 0x20107600 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 rwstate(write n=3 w=0) oi: e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3(173'2593 client.4293.0:144210 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2593) ssc: 0x72b8d20 snapset: 0=[]:[]+head
2014-06-12 14:46:45.167524 7f9d61bd4700  5 queue_transactions existing osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:45.167529 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.167520 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] find_object_context e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 @head oi=e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3(173'2593 client.4293.0:144210 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2593)
2014-06-12 14:46:45.167531 7f9d61bd4700  5 queue_op 0x544d020 seq 2266 osr(3.251 0x276d390) 1034485 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:46:45.167543 7f9d673df700  5 -- op tracker -- , seq: 748023, time: 2014-06-12 14:46:45.167537, event: started, op: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.167551 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] execute_ctx 0x580ca00
2014-06-12 14:46:45.167541 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] eval_repop repgather(0x2b62b680 173'2449 rep_tid=236319 committed?=0 applied?=0 lock=0 op=osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.167563 7f9d61bd4700 10 osd.2 173 dequeue_op 0x59dc200 finish
2014-06-12 14:46:45.167569 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:45.167567 7f9d673df700  5 -- op tracker -- , seq: 748023, time: 2014-06-12 14:46:45.167562, event: started, op: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.167572 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.167573 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_op e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] ov 173'2593 av 173'2594 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.167591 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fc20 (2 active)
2014-06-12 14:46:45.167586 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096]
2014-06-12 14:46:45.167598 7f9d83173700  5 _do_op 0x544d020 seq 2266 osr(3.251 0x276d390)/0x276d390 start
2014-06-12 14:46:45.167603 7f9d83173700 10 _do_transaction on 0x2b62b2c0
2014-06-12 14:46:45.167597 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.167609 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op  write 3268608~516096
2014-06-12 14:46:45.167612 7f9d83173700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.167622 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op_effects on session 0x5c8cc00
2014-06-12 14:46:45.167633 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] make_writeable e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 snapset=0x72b8d58  snapc=0=[]
2014-06-12 14:46:45.167643 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] make_writeable e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.167652 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] finish_ctx e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 0x580ca00 op modify  
2014-06-12 14:46:45.167663 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  set mtime to 2014-06-12 08:45:47.543916
2014-06-12 14:46:45.167680 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  final snapset 0=[]:[]+head in e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.167695 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.167704 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] new_repop rep_tid 236320 on osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.167722 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] issue_repop rep_tid 236320 o e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.167743 7f9d673df700  5 -- op tracker -- , seq: 748023, time: 2014-06-12 14:46:45.167738, event: waiting for subops from 24, op: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.167761 7f9d673df700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:45.167770 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.167789 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.167780 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2593 (0'0,173'2593] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] append_log log((0'0,173'2593], crt=0'0) [173'2594 (173'2593) modify   e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 by client.4293.0:144211 2014-06-12 08:45:47.543916]
2014-06-12 14:46:45.167799 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] add_log_entry 173'2594 (173'2593) modify   e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 by client.4293.0:144211 2014-06-12 08:45:47.543916
2014-06-12 14:46:45.167817 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.167850 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2594, trimmed: 
2014-06-12 14:46:45.167869 7f9d673df700  5 queue_transactions existing osr(3.4a7 0x276d8c0)/0x276d8c0
2014-06-12 14:46:45.167872 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.167873 7f9d673df700  5 queue_op 0xa44d380 seq 2340 osr(3.4a7 0x276d8c0) 518389 bytes   (queue has 3 ops and 2071263 bytes)
2014-06-12 14:46:45.167881 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] eval_repop repgather(0x32b68b40 173'2594 rep_tid=236320 committed?=0 applied?=0 lock=0 op=osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.167898 7f9d673df700 10 osd.2 173 dequeue_op 0x5ee8300 finish
2014-06-12 14:46:45.167901 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.167904 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.170223 7f9d5a16f700  5 -- op tracker -- , seq: 748024, time: 2014-06-12 14:46:45.168461, event: header_read, op: osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.170238 7f9d5a16f700  5 -- op tracker -- , seq: 748024, time: 2014-06-12 14:46:45.168464, event: throttled, op: osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.170247 7f9d5a16f700  5 -- op tracker -- , seq: 748024, time: 2014-06-12 14:46:45.170175, event: all_read, op: osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.170256 7f9d5a16f700  5 -- op tracker -- , seq: 748024, time: 0.000000, event: dispatched, op: osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.170265 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.170274 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:45.170285 7f9d5a16f700 15 osd.2 173 enqueue_op 0x31d60e00 prio 127 cost 516903 latency 0.001824 osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.171732 7f9d425f5700  5 -- op tracker -- , seq: 748025, time: 2014-06-12 14:46:45.160135, event: header_read, op: osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.171746 7f9d425f5700  5 -- op tracker -- , seq: 748025, time: 2014-06-12 14:46:45.160137, event: throttled, op: osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.171765 7f9d425f5700  5 -- op tracker -- , seq: 748025, time: 2014-06-12 14:46:45.171685, event: all_read, op: osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.171775 7f9d425f5700  5 -- op tracker -- , seq: 748025, time: 0.000000, event: dispatched, op: osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.171783 7f9d425f5700 20 osd.2 173 should_share_map client.4314 172.16.96.220:0/1244501 173
2014-06-12 14:46:45.171807 7f9d425f5700 15 osd.2 173 enqueue_op 0x6d10d00 prio 63 cost 1929216 latency 0.011672 osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.171853 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.171860 7f9d663dd700 10 osd.2 173 dequeue_op 0x31d60e00 prio 127 cost 516903 latency 0.003398 osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2722 (0'0,173'2722] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active]
2014-06-12 14:46:45.171894 7f9d663dd700  5 -- op tracker -- , seq: 748024, time: 2014-06-12 14:46:45.171889, event: reached_pg, op: osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.171912 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2722 (0'0,173'2722] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] handle_message: 0x31d60e00
2014-06-12 14:46:45.171928 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2722 (0'0,173'2722] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] sub_op_modify trans cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 v 173'2723 (transaction) 174
2014-06-12 14:46:45.171944 7f9d663dd700  5 -- op tracker -- , seq: 748024, time: 2014-06-12 14:46:45.171940, event: started, op: osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.171963 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2722 (0'0,173'2722] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] append_log log((0'0,173'2722], crt=0'0) [173'2723 (173'2722) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143145 2014-06-12 08:43:33.840756]
2014-06-12 14:46:45.171988 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2723 (0'0,173'2723] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] add_log_entry 173'2723 (173'2722) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143145 2014-06-12 08:43:33.840756
2014-06-12 14:46:45.172013 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2723 (0'0,173'2723] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] append_log  adding 1 keys
2014-06-12 14:46:45.172057 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2723, trimmed: 
2014-06-12 14:46:45.172079 7f9d663dd700  5 -- op tracker -- , seq: 748024, time: 2014-06-12 14:46:45.172074, event: started, op: osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.172090 7f9d663dd700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:45.172094 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.172096 7f9d663dd700  5 queue_op 0x2b4a83c0 seq 2470 osr(3.21c 0x276d020) 518389 bytes   (queue has 4 ops and 2589652 bytes)
2014-06-12 14:46:45.172107 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2723 (0'0,173'2723] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] do_sub_op osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.172133 7f9d663dd700 10 osd.2 173 dequeue_op 0x31d60e00 finish
2014-06-12 14:46:45.172137 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.172143 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.172147 7f9d663dd700 10 osd.2 173 dequeue_op 0x6d10d00 prio 63 cost 1929216 latency 0.012012 osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]
2014-06-12 14:46:45.172168 7f9d663dd700  5 -- op tracker -- , seq: 748025, time: 2014-06-12 14:46:45.172164, event: reached_pg, op: osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.172176 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.172190 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] handle_message: 0x6d10d00
2014-06-12 14:46:45.172199 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.172223 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: found obc in cache: 0x22eacb00
2014-06-12 14:46:45.172234 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: 0x22eacb00 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 rwstate(write n=4 w=0) oi: fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3(173'2447 client.4314.0:145118 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2447) ssc: 0x93449a0 snapset: 0=[]:[]+head
2014-06-12 14:46:45.172250 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] find_object_context fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 @head oi=fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3(173'2447 client.4314.0:145118 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2447)
2014-06-12 14:46:45.172274 7f9d663dd700  5 -- op tracker -- , seq: 748025, time: 2014-06-12 14:46:45.172269, event: started, op: osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.172282 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] execute_ctx 0x2e413000
2014-06-12 14:46:45.172299 7f9d663dd700  5 -- op tracker -- , seq: 748025, time: 2014-06-12 14:46:45.172294, event: started, op: osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.172306 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] ov 173'2447 av 173'2450 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.172320 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216]
2014-06-12 14:46:45.172332 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.172357 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  write 2265088~1929216
2014-06-12 14:46:45.172374 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op_effects on session 0x53ac800
2014-06-12 14:46:45.172387 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 snapset=0x93449d8  snapc=0=[]
2014-06-12 14:46:45.172400 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.172410 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] finish_ctx fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 0x2e413000 op modify  
2014-06-12 14:46:45.172423 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  set mtime to 2014-06-12 08:45:47.536172
2014-06-12 14:46:45.172444 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  final snapset 0=[]:[]+head in fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:45.172463 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.172473 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] new_repop rep_tid 236321 on osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.172488 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] issue_repop rep_tid 236321 o fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:45.172502 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5417b00 (3 active)
2014-06-12 14:46:45.172515 7f9d7c966700  5 _do_op 0xa44d380 seq 2340 osr(3.4a7 0x276d8c0)/0x276d8c0 start
2014-06-12 14:46:45.172520 7f9d7c966700 10 _do_transaction on 0x9908d80
2014-06-12 14:46:45.172527 7f9d7c966700 15 _omap_setkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:45.172586 7f9d663dd700  5 -- op tracker -- , seq: 748025, time: 2014-06-12 14:46:45.172579, event: waiting for subops from 26, op: osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.172613 7f9d663dd700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.172646 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2449 (0'0,173'2449] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log log((0'0,173'2449], crt=0'0) [173'2450 (173'2447) modify   fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 by client.4314.0:145119 2014-06-12 08:45:47.536172]
2014-06-12 14:46:45.172665 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:45.172673 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] add_log_entry 173'2450 (173'2447) modify   fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 by client.4314.0:145119 2014-06-12 08:45:47.536172
2014-06-12 14:46:45.172687 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.172694 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.172723 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2450, trimmed: 
2014-06-12 14:46:45.172753 7f9d663dd700  5 queue_transactions existing osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:45.172757 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.172759 7f9d663dd700  5 queue_op 0x5ced6e0 seq 2267 osr(3.251 0x276d390) 1931509 bytes   (queue has 5 ops and 4521161 bytes)
2014-06-12 14:46:45.172768 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] eval_repop repgather(0x3a096480 173'2450 rep_tid=236321 committed?=0 applied?=0 lock=0 op=osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.172786 7f9d663dd700 10 osd.2 173 dequeue_op 0x6d10d00 finish
2014-06-12 14:46:45.172791 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.172795 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.173642 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.173633 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (4 active)
2014-06-12 14:46:45.173650 7f9d81970700  5 _do_op 0x2b4a83c0 seq 2470 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:46:45.173659 7f9d81970700 10 _do_transaction on 0x12020b70
2014-06-12 14:46:45.173672 7f9d81970700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:45.174229 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:45.174262 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.176283 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fc20 (5 active)
2014-06-12 14:46:45.176620 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.176657 7f9d81970700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:45.176663 7f9d81970700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:46:45.176672 7f9d81970700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:45.176694 7f9d81970700 15 _write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 1671168~516096
2014-06-12 14:46:45.176794 7f9d7c165700 10 _do_op 0x6957560 seq 2404 r = 0, finisher 0x4f61840 0
2014-06-12 14:46:45.176826 7f9d7715b700  5 -- op tracker -- , seq: 748021, time: 2014-06-12 14:46:45.176818, event: commit_sent, op: osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.176835 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2562 (0'0,173'2562] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2560 active] sub_op_modify_commit on op osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:45.176860 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.176882 7f9d7c165700 10 _finish_op 0x6957560 seq 2404 osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:45.176893 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533cd80 (4 active)
2014-06-12 14:46:45.176897 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.176915 7f9d7795c700  5 -- op tracker -- , seq: 748021, time: 2014-06-12 14:46:45.176906, event: sub_op_applied, op: osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.176926 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2562 (0'0,173'2562] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2561 active] sub_op_modify_applied on 0x2720e00 op osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.176944 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2562 (0'0,173'2562] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2561 active] op_applied on primary on version 173'2562
2014-06-12 14:46:45.176954 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2562 (0'0,173'2562] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2561 active] op_applied on replica on version 173'2562
2014-06-12 14:46:45.176983 7f9d7795c700  5 -- op tracker -- , seq: 748021, time: 2014-06-12 14:46:45.176979, event: done, op: osd_sub_op(client.4296.0:143674 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2562 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.177168 7f9d81970700 10 lookup_strip_header done  cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:45.177446 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:45.179340 7f9d81970700 10 _generic_write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 1671168~516096 = 0
2014-06-12 14:46:45.179392 7f9d81970700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:45.179408 7f9d81970700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:45.179414 7f9d81970700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:45.179423 7f9d81970700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:45.179620 7f9d81970700 20 set_header setting 16774 cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 parent seq 0
2014-06-12 14:46:45.179697 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.179708 7f9d81970700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:46:45.179715 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:45.182103 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.182140 7f9d81970700 10 _do_op 0x2b4a83c0 seq 2470 r = 0, finisher 0x897d260 0
2014-06-12 14:46:45.182177 7f9d7715b700  5 -- op tracker -- , seq: 748024, time: 2014-06-12 14:46:45.182166, event: commit_sent, op: osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.182187 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2723 (0'0,173'2723] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] sub_op_modify_commit on op osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:45.182219 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:45.182244 7f9d7c966700 15 _omap_rmkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:45.182254 7f9d7c966700 10 _omap_rmkeys meta/1fe06fa5/pglog_3.4a7/0//-1 r = 0
2014-06-12 14:46:45.182264 7f9d7c966700 15 _omap_setkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:45.182292 7f9d7c966700 15 _write 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 3268608~516096
2014-06-12 14:46:45.182294 7f9d81970700 10 _finish_op 0x2b4a83c0 seq 2470 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:45.182307 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (3 active)
2014-06-12 14:46:45.182324 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.183180 7f9d7c966700 10 lookup_strip_header done  cid 3.4a7_head oid e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.183345 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:45.184780 7f9d7795c700  5 -- op tracker -- , seq: 748024, time: 2014-06-12 14:46:45.184755, event: sub_op_applied, op: osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.184808 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2723 (0'0,173'2723] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] sub_op_modify_applied on 0x12020a80 op osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.184832 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2723 (0'0,173'2723] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] op_applied on primary on version 173'2723
2014-06-12 14:46:45.184842 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2723 (0'0,173'2723] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] op_applied on replica on version 173'2723
2014-06-12 14:46:45.184865 7f9d7795c700  5 -- op tracker -- , seq: 748024, time: 2014-06-12 14:46:45.184862, event: done, op: osd_sub_op(client.4279.0:143145 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.185361 7f9d7c966700 10 _generic_write 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 3268608~516096 = 0
2014-06-12 14:46:45.185422 7f9d7c966700 15 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.185440 7f9d7c966700 10 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 = 0
2014-06-12 14:46:45.185452 7f9d7c966700 15 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.185461 7f9d7c966700 10 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 = 0
2014-06-12 14:46:45.185786 7f9d7c966700 20 set_header setting 17968 cid 3.4a7_head oid e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 parent seq 0
2014-06-12 14:46:45.185880 7f9d7c966700 20 set_header setting 2078 cid meta oid 1fe06fa5/pglog_3.4a7/0//-1 parent seq 0
2014-06-12 14:46:45.185895 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.185903 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:45.189921 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.189962 7f9d83173700 15 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.189964 7f9d7c966700 10 _do_op 0xa44d380 seq 2340 r = 0, finisher 0x6907760 0x3373d2c0
2014-06-12 14:46:45.189969 7f9d83173700 10 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1 r = 0
2014-06-12 14:46:45.189977 7f9d83173700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.189993 7f9d83173700 15 _write 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 479232~1032192
2014-06-12 14:46:45.189982 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_commit: 236320
2014-06-12 14:46:45.190010 7f9d7715b700  5 -- op tracker -- , seq: 748023, time: 2014-06-12 14:46:45.190002, event: op_commit, op: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.190027 7f9d7c966700 10 _finish_op 0xa44d380 seq 2340 osr(3.4a7 0x276d8c0)/0x276d8c0
2014-06-12 14:46:45.190044 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_applied: 236320
2014-06-12 14:46:45.190064 7f9d7795c700  5 -- op tracker -- , seq: 748023, time: 2014-06-12 14:46:45.190057, event: op_applied, op: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.190072 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_applied on primary on version 173'2594
2014-06-12 14:46:45.190105 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5417b00 (2 active)
2014-06-12 14:46:45.190114 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.190139 7f9d83173700 10 lookup_strip_header done  cid 3.251_head oid 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.190167 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:45.190435 7f9d535e6700  5 -- op tracker -- , seq: 748026, time: 2014-06-12 14:46:45.189456, event: header_read, op: osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.190448 7f9d535e6700  5 -- op tracker -- , seq: 748026, time: 2014-06-12 14:46:45.189461, event: throttled, op: osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.190458 7f9d535e6700  5 -- op tracker -- , seq: 748026, time: 2014-06-12 14:46:45.190405, event: all_read, op: osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.190468 7f9d535e6700  5 -- op tracker -- , seq: 748026, time: 0.000000, event: dispatched, op: osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.190476 7f9d535e6700 20 osd.2 173 should_share_map client.4280 172.16.96.220:0/1241303 173
2014-06-12 14:46:45.190502 7f9d535e6700 15 osd.2 173 enqueue_op 0x598e800 prio 63 cost 679936 latency 0.001046 osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.190550 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.190557 7f9d693e3700 10 osd.2 173 dequeue_op 0x598e800 prio 63 cost 679936 latency 0.001101 osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]
2014-06-12 14:46:45.190582 7f9d693e3700  5 -- op tracker -- , seq: 748026, time: 2014-06-12 14:46:45.190577, event: reached_pg, op: osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.190591 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.190608 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] handle_message: 0x598e800
2014-06-12 14:46:45.190620 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_op osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.190654 7f9d693e3700 15 getattr 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 '_'
2014-06-12 14:46:45.205905 7f9d83173700 10 _generic_write 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 479232~1032192 = 0
2014-06-12 14:46:45.206073 7f9d83173700 15 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.206094 7f9d83173700 10 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 = 0
2014-06-12 14:46:45.206101 7f9d83173700 15 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.206110 7f9d83173700 10 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 = 0
2014-06-12 14:46:45.206542 7f9d83173700 20 set_header setting 16808 cid 3.251_head oid 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 parent seq 0
2014-06-12 14:46:45.206664 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.206680 7f9d83173700 20 set_header setting 2026 cid meta oid 1fe2661a/pglog_3.251/0//-1 parent seq 0
2014-06-12 14:46:45.206692 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:45.213474 7f9d535e6700  5 -- op tracker -- , seq: 748027, time: 2014-06-12 14:46:45.210027, event: header_read, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.213483 7f9d535e6700  5 -- op tracker -- , seq: 748027, time: 2014-06-12 14:46:45.210031, event: throttled, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.213489 7f9d535e6700  5 -- op tracker -- , seq: 748027, time: 2014-06-12 14:46:45.213437, event: all_read, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.213494 7f9d535e6700  5 -- op tracker -- , seq: 748027, time: 0.000000, event: dispatched, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.213499 7f9d535e6700 20 osd.2 173 should_share_map client.4280 172.16.96.220:0/1241303 173
2014-06-12 14:46:45.213521 7f9d535e6700 15 osd.2 173 enqueue_op 0x598ee00 prio 63 cost 516096 latency 0.003494 osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.214268 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:45.217653 7f9d411d2700  5 -- op tracker -- , seq: 748028, time: 2014-06-12 14:46:45.165025, event: header_read, op: osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.217668 7f9d411d2700  5 -- op tracker -- , seq: 748028, time: 2014-06-12 14:46:45.165028, event: throttled, op: osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.217676 7f9d411d2700  5 -- op tracker -- , seq: 748028, time: 2014-06-12 14:46:45.217567, event: all_read, op: osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.217684 7f9d411d2700  5 -- op tracker -- , seq: 748028, time: 0.000000, event: dispatched, op: osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.217694 7f9d411d2700 20 osd.2 173 should_share_map client.4306 172.16.96.220:0/1243907 173
2014-06-12 14:46:45.217721 7f9d411d2700 15 osd.2 173 enqueue_op 0x59dd800 prio 63 cost 2580480 latency 0.052696 osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.217755 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active)
2014-06-12 14:46:45.217764 7f9d65bdc700 10 osd.2 173 dequeue_op 0x59dd800 prio 63 cost 2580480 latency 0.052739 osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]
2014-06-12 14:46:45.217795 7f9d65bdc700  5 -- op tracker -- , seq: 748028, time: 2014-06-12 14:46:45.217789, event: reached_pg, op: osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.217819 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.217836 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] handle_message: 0x59dd800
2014-06-12 14:46:45.217846 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.217870 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: found obc in cache: 0x3bce9600
2014-06-12 14:46:45.217880 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: 0x3bce9600 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 rwstate(write n=2 w=0) oi: 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3(173'2449 client.4306.0:144194 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2449) ssc: 0x407ea80 snapset: 0=[]:[]+head
2014-06-12 14:46:45.217895 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] find_object_context 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 @head oi=3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3(173'2449 client.4306.0:144194 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2449)
2014-06-12 14:46:45.217920 7f9d65bdc700  5 -- op tracker -- , seq: 748028, time: 2014-06-12 14:46:45.217915, event: started, op: osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.217930 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] execute_ctx 0x33b2ea00
2014-06-12 14:46:45.217948 7f9d65bdc700  5 -- op tracker -- , seq: 748028, time: 2014-06-12 14:46:45.217943, event: started, op: osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.217955 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] ov 173'2449 av 173'2451 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.217968 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480]
2014-06-12 14:46:45.217978 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.217991 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  write 1511424~2580480
2014-06-12 14:46:45.218005 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op_effects on session 0x53ac600
2014-06-12 14:46:45.218016 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 snapset=0x407eab8  snapc=0=[]
2014-06-12 14:46:45.218027 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.218036 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] finish_ctx 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 0x33b2ea00 op modify  
2014-06-12 14:46:45.218049 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  set mtime to 2014-06-12 08:45:47.540558
2014-06-12 14:46:45.218073 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  final snapset 0=[]:[]+head in 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.218091 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.218102 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] new_repop rep_tid 236322 on osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.218117 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] issue_repop rep_tid 236322 o 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.218142 7f9d65bdc700  5 -- op tracker -- , seq: 748028, time: 2014-06-12 14:46:45.218137, event: waiting for subops from 26, op: osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.218163 7f9d65bdc700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.218185 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2450 (0'0,173'2450] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log log((0'0,173'2450], crt=0'0) [173'2451 (173'2449) modify   3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 by client.4306.0:144195 2014-06-12 08:45:47.540558]
2014-06-12 14:46:45.218208 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] add_log_entry 173'2451 (173'2449) modify   3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 by client.4306.0:144195 2014-06-12 08:45:47.540558
2014-06-12 14:46:45.218229 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.218259 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2451, trimmed: 
2014-06-12 14:46:45.218272 7f9d411d2700  5 -- op tracker -- , seq: 748029, time: 2014-06-12 14:46:45.217760, event: header_read, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.218282 7f9d65bdc700  5 queue_transactions existing osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:45.218286 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.218283 7f9d411d2700  5 -- op tracker -- , seq: 748029, time: 2014-06-12 14:46:45.217762, event: throttled, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.218287 7f9d65bdc700  5 queue_op 0x803db00 seq 2268 osr(3.251 0x276d390) 2582773 bytes   (queue has 3 ops and 5548767 bytes)
2014-06-12 14:46:45.218292 7f9d411d2700  5 -- op tracker -- , seq: 748029, time: 2014-06-12 14:46:45.217894, event: all_read, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.218300 7f9d411d2700  5 -- op tracker -- , seq: 748029, time: 0.000000, event: dispatched, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.218306 7f9d411d2700 20 osd.2 173 should_share_map client.4306 172.16.96.220:0/1243907 173
2014-06-12 14:46:45.218295 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] eval_repop repgather(0x34d8eb40 173'2451 rep_tid=236322 committed?=0 applied?=0 lock=0 op=osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.218313 7f9d65bdc700 10 osd.2 173 dequeue_op 0x59dd800 finish
2014-06-12 14:46:45.218316 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active)
2014-06-12 14:46:45.218323 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.218321 7f9d411d2700 15 osd.2 173 enqueue_op 0x8dbd900 prio 63 cost 102400 latency 0.000560 osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.218344 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active)
2014-06-12 14:46:45.218349 7f9d62bd6700 10 osd.2 173 dequeue_op 0x8dbd900 prio 63 cost 102400 latency 0.000589 osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]
2014-06-12 14:46:45.218369 7f9d62bd6700  5 -- op tracker -- , seq: 748029, time: 2014-06-12 14:46:45.218365, event: reached_pg, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.218377 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.218388 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] handle_message: 0x8dbd900
2014-06-12 14:46:45.218397 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.218413 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: found obc in cache: 0x3bce9600
2014-06-12 14:46:45.218422 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] get_object_context: 0x3bce9600 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 rwstate(write n=3 w=0) oi: 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3(173'2451 client.4306.0:144195 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2451) ssc: 0x407ea80 snapset: 0=[]:[]+head
2014-06-12 14:46:45.218434 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] find_object_context 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 @head oi=3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3(173'2451 client.4306.0:144195 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2451)
2014-06-12 14:46:45.218453 7f9d62bd6700  5 -- op tracker -- , seq: 748029, time: 2014-06-12 14:46:45.218448, event: started, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.218459 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] execute_ctx 0x2963b200
2014-06-12 14:46:45.218473 7f9d62bd6700  5 -- op tracker -- , seq: 748029, time: 2014-06-12 14:46:45.218469, event: started, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.218479 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_op 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] ov 173'2451 av 173'2452 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.218490 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400]
2014-06-12 14:46:45.218499 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.218513 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op  write 4091904~102400
2014-06-12 14:46:45.218524 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] do_osd_op_effects on session 0x53ac600
2014-06-12 14:46:45.218533 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 snapset=0x407eab8  snapc=0=[]
2014-06-12 14:46:45.218543 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] make_writeable 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.218552 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] finish_ctx 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 0x2963b200 op modify  
2014-06-12 14:46:45.218562 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  set mtime to 2014-06-12 08:45:47.541534
2014-06-12 14:46:45.218579 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  final snapset 0=[]:[]+head in 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.218599 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.218626 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] new_repop rep_tid 236323 on osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.218639 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] issue_repop rep_tid 236323 o 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.218659 7f9d62bd6700  5 -- op tracker -- , seq: 748029, time: 2014-06-12 14:46:45.218654, event: waiting for subops from 26, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.219054 7f9d535e6700  5 -- op tracker -- , seq: 748030, time: 2014-06-12 14:46:45.218965, event: header_read, op: osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.219064 7f9d535e6700  5 -- op tracker -- , seq: 748030, time: 2014-06-12 14:46:45.218968, event: throttled, op: osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.219072 7f9d535e6700  5 -- op tracker -- , seq: 748030, time: 2014-06-12 14:46:45.219028, event: all_read, op: osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.219079 7f9d535e6700  5 -- op tracker -- , seq: 748030, time: 0.000000, event: dispatched, op: osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.219084 7f9d535e6700 20 osd.2 173 should_share_map client.4280 172.16.96.220:0/1241303 173
2014-06-12 14:46:45.219102 7f9d535e6700 15 osd.2 173 enqueue_op 0x598e400 prio 63 cost 28672 latency 0.000136 osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.219137 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (4 active)
2014-06-12 14:46:45.219530 7f9d83173700 10 _do_op 0x544d020 seq 2266 r = 0, finisher 0x32e67420 0x2d174000
2014-06-12 14:46:45.219558 7f9d83173700 10 _finish_op 0x544d020 seq 2266 osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:45.219571 7f9d82972700  5 _do_op 0x5ced6e0 seq 2267 osr(3.251 0x276d390)/0x276d390 start
2014-06-12 14:46:45.219576 7f9d82972700 10 _do_transaction on 0x3a097ec0
2014-06-12 14:46:45.219588 7f9d82972700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.219765 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.219797 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.219853 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.219882 7f9d82972700 15 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.219888 7f9d82972700 10 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1 r = 0
2014-06-12 14:46:45.219895 7f9d82972700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.220162 7f9d82972700 15 _write 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 2265088~1929216
2014-06-12 14:46:45.220342 7f9d82972700 10 lookup_strip_header done  cid 3.251_head oid fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:45.220519 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:45.220957 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fc20 (1 active)
2014-06-12 14:46:45.220966 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fc20 (2 active)
2014-06-12 14:46:45.220974 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.225671 7f9d62bd6700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.225776 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2451 (0'0,173'2451] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log log((0'0,173'2451], crt=0'0) [173'2452 (173'2451) modify   3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 by client.4306.0:144196 2014-06-12 08:45:47.541534]
2014-06-12 14:46:45.225818 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] add_log_entry 173'2452 (173'2451) modify   3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 by client.4306.0:144196 2014-06-12 08:45:47.541534
2014-06-12 14:46:45.225846 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.225883 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2452, trimmed: 
2014-06-12 14:46:45.225918 7f9d62bd6700  5 queue_transactions existing osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:45.225926 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.225930 7f9d62bd6700  5 queue_op 0x73a2de0 seq 2269 osr(3.251 0x276d390) 104693 bytes   (queue has 3 ops and 4618975 bytes)
2014-06-12 14:46:45.225942 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] eval_repop repgather(0x52f5ec0 173'2452 rep_tid=236323 committed?=0 applied?=0 lock=0 op=osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.225983 7f9d62bd6700 10 osd.2 173 dequeue_op 0x8dbd900 finish
2014-06-12 14:46:45.225992 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (3 active)
2014-06-12 14:46:45.225998 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.225994 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_commit: 236319
2014-06-12 14:46:45.226021 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:45.226034 7f9d7715b700  5 -- op tracker -- , seq: 748022, time: 2014-06-12 14:46:45.226021, event: op_commit, op: osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.226065 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540fc20 (3 active)
2014-06-12 14:46:45.226062 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied: 236319
2014-06-12 14:46:45.226083 7f9d7795c700  5 -- op tracker -- , seq: 748022, time: 2014-06-12 14:46:45.226078, event: op_applied, op: osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.226089 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied on primary on version 173'2449
2014-06-12 14:46:45.227984 7f9d82972700 10 _generic_write 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 2265088~1929216 = 0
2014-06-12 14:46:45.228149 7f9d82972700 15 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:45.228173 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:45.228179 7f9d82972700 10 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 = 0
2014-06-12 14:46:45.228186 7f9d82972700 15 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3
2014-06-12 14:46:45.228191 7f9d82972700 10 _setattrs 3.251_head/fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 = 0
2014-06-12 14:46:45.228662 7f9d82972700 20 set_header setting 17915 cid 3.251_head oid fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 parent seq 0
2014-06-12 14:46:45.228808 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.228821 7f9d82972700 20 set_header setting 2026 cid meta oid 1fe2661a/pglog_3.251/0//-1 parent seq 0
2014-06-12 14:46:45.228828 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:45.247381 7f9d82972700 10 _do_op 0x5ced6e0 seq 2267 r = 0, finisher 0x98ef2a0 0x1e626b40
2014-06-12 14:46:45.247546 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_commit: 236321
2014-06-12 14:46:45.247576 7f9d7715b700  5 -- op tracker -- , seq: 748025, time: 2014-06-12 14:46:45.247568, event: op_commit, op: osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.247598 7f9d82972700 10 _finish_op 0x5ced6e0 seq 2267 osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:45.247605 7f9d83173700  5 _do_op 0x803db00 seq 2268 osr(3.251 0x276d390)/0x276d390 start
2014-06-12 14:46:45.247609 7f9d83173700 10 _do_transaction on 0x1687ccc0
2014-06-12 14:46:45.247617 7f9d83173700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.247754 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.247779 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.247828 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.248015 7f9d83173700 15 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.248022 7f9d83173700 10 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1 r = 0
2014-06-12 14:46:45.248029 7f9d83173700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.248047 7f9d83173700 15 _write 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 1511424~2580480
2014-06-12 14:46:45.248199 7f9d83173700 10 lookup_strip_header done  cid 3.251_head oid 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.248544 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:45.253792 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fc20 (2 active)
2014-06-12 14:46:45.253809 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.253799 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied: 236321
2014-06-12 14:46:45.253842 7f9d7795c700  5 -- op tracker -- , seq: 748025, time: 2014-06-12 14:46:45.253830, event: op_applied, op: osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.253858 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied on primary on version 173'2450
2014-06-12 14:46:45.257674 7f9d83173700 10 _generic_write 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 1511424~2580480 = 0
2014-06-12 14:46:45.257891 7f9d83173700 15 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.257910 7f9d83173700 10 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 = 0
2014-06-12 14:46:45.257917 7f9d83173700 15 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.257921 7f9d83173700 10 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 = 0
2014-06-12 14:46:45.258541 7f9d83173700 20 set_header setting 16808 cid 3.251_head oid 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 parent seq 0
2014-06-12 14:46:45.258910 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.258930 7f9d83173700 20 set_header setting 2026 cid meta oid 1fe2661a/pglog_3.251/0//-1 parent seq 0
2014-06-12 14:46:45.258939 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:45.269699 7f9d83173700 10 _do_op 0x803db00 seq 2268 r = 0, finisher 0x7b42a00 0x25318cc0
2014-06-12 14:46:45.269721 7f9d83173700 10 _finish_op 0x803db00 seq 2268 osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:45.269729 7f9d7895e700  5 _do_op 0x73a2de0 seq 2269 osr(3.251 0x276d390)/0x276d390 start
2014-06-12 14:46:45.269738 7f9d7895e700 10 _do_transaction on 0x2335cc00
2014-06-12 14:46:45.269731 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fc20 (1 active)
2014-06-12 14:46:45.269741 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.269725 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_commit: 236322
2014-06-12 14:46:45.269746 7f9d7895e700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.269757 7f9d7715b700  5 -- op tracker -- , seq: 748028, time: 2014-06-12 14:46:45.269746, event: op_commit, op: osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.269882 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.269998 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.270062 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.270088 7f9d7895e700 15 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.270095 7f9d7895e700 10 _omap_rmkeys meta/1fe2661a/pglog_3.251/0//-1 r = 0
2014-06-12 14:46:45.270101 7f9d7895e700 15 _omap_setkeys meta/1fe2661a/pglog_3.251/0//-1
2014-06-12 14:46:45.270118 7f9d7895e700 15 _write 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 4091904~102400
2014-06-12 14:46:45.270269 7f9d7895e700 10 lookup_strip_header done  cid 3.251_head oid 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.270432 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:45.270691 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied: 236322
2014-06-12 14:46:45.270727 7f9d7795c700  5 -- op tracker -- , seq: 748028, time: 2014-06-12 14:46:45.270716, event: op_applied, op: osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.270739 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied on primary on version 173'2451
2014-06-12 14:46:45.270849 7f9d7895e700 10 _generic_write 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 4091904~102400 = 0
2014-06-12 14:46:45.270874 7f9d7895e700 15 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.270887 7f9d7895e700 10 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 = 0
2014-06-12 14:46:45.270894 7f9d7895e700 15 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3
2014-06-12 14:46:45.270903 7f9d7895e700 10 _setattrs 3.251_head/3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 = 0
2014-06-12 14:46:45.271020 7f9d7895e700 20 set_header setting 16808 cid 3.251_head oid 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 parent seq 0
2014-06-12 14:46:45.271051 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.271066 7f9d7895e700 20 set_header setting 2026 cid meta oid 1fe2661a/pglog_3.251/0//-1 parent seq 0
2014-06-12 14:46:45.271076 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:45.271636 7f9d7895e700 10 _do_op 0x73a2de0 seq 2269 r = 0, finisher 0x172ff740 0x4dd9a40
2014-06-12 14:46:45.271650 7f9d7895e700 10 _finish_op 0x73a2de0 seq 2269 osr(3.251 0x276d390)/0x276d390
2014-06-12 14:46:45.271659 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540fc20 (0 active)
2014-06-12 14:46:45.271663 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.271655 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_commit: 236323
2014-06-12 14:46:45.271687 7f9d7715b700  5 -- op tracker -- , seq: 748029, time: 2014-06-12 14:46:45.271677, event: op_commit, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.271701 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied: 236323
2014-06-12 14:46:45.271724 7f9d7795c700  5 -- op tracker -- , seq: 748029, time: 2014-06-12 14:46:45.271718, event: op_applied, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.271735 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] op_applied on primary on version 173'2452
2014-06-12 14:46:45.306814 7f9d56432700  5 -- op tracker -- , seq: 748031, time: 2014-06-12 14:46:45.306291, event: header_read, op: osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.306827 7f9d56432700  5 -- op tracker -- , seq: 748031, time: 2014-06-12 14:46:45.306293, event: throttled, op: osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.306836 7f9d56432700  5 -- op tracker -- , seq: 748031, time: 2014-06-12 14:46:45.306774, event: all_read, op: osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.306844 7f9d56432700  5 -- op tracker -- , seq: 748031, time: 0.000000, event: dispatched, op: osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.306849 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.306863 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:45.306871 7f9d56432700 15 osd.2 173 enqueue_op 0x12bf6000 prio 127 cost 332583 latency 0.000579 osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.306900 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (4 active)
2014-06-12 14:46:45.306909 7f9d683e1700 10 osd.2 173 dequeue_op 0x12bf6000 prio 127 cost 332583 latency 0.000618 osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.71f( v 173'2701 (0'0,173'2701] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2700 active]
2014-06-12 14:46:45.306939 7f9d683e1700  5 -- op tracker -- , seq: 748031, time: 2014-06-12 14:46:45.306934, event: reached_pg, op: osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.306949 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2701 (0'0,173'2701] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2700 active] handle_message: 0x12bf6000
2014-06-12 14:46:45.306961 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2701 (0'0,173'2701] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2700 active] sub_op_modify trans c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 v 173'2702 (transaction) 174
2014-06-12 14:46:45.306977 7f9d683e1700  5 -- op tracker -- , seq: 748031, time: 2014-06-12 14:46:45.306973, event: started, op: osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.306994 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2701 (0'0,173'2701] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2700 active] append_log log((0'0,173'2701], crt=0'0) [173'2702 (173'2701) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145289 2014-06-12 08:45:28.228755]
2014-06-12 14:46:45.307020 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2702 (0'0,173'2702] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2700 active] add_log_entry 173'2702 (173'2701) modify   c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 by client.4308.0:145289 2014-06-12 08:45:28.228755
2014-06-12 14:46:45.307041 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2702 (0'0,173'2702] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2700 active] append_log  adding 1 keys
2014-06-12 14:46:45.307076 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2702, trimmed: 
2014-06-12 14:46:45.307102 7f9d683e1700  5 -- op tracker -- , seq: 748031, time: 2014-06-12 14:46:45.307097, event: started, op: osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.307113 7f9d683e1700  5 queue_transactions existing osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:45.307117 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.307118 7f9d683e1700  5 queue_op 0x9797f20 seq 2534 osr(3.71f 0x276de50) 334069 bytes   (queue has 1 ops and 334069 bytes)
2014-06-12 14:46:45.307130 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d0240 (1 active)
2014-06-12 14:46:45.307137 7f9d7a161700  5 _do_op 0x9797f20 seq 2534 osr(3.71f 0x276de50)/0x276de50 start
2014-06-12 14:46:45.307126 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.71f( v 173'2702 (0'0,173'2702] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2700 active] do_sub_op osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.307141 7f9d683e1700 10 osd.2 173 dequeue_op 0x12bf6000 finish
2014-06-12 14:46:45.307142 7f9d7a161700 10 _do_transaction on 0x62c0630
2014-06-12 14:46:45.307144 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (3 active)
2014-06-12 14:46:45.307147 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.307148 7f9d7a161700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:45.307281 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:45.307424 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.307492 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.307516 7f9d7a161700 15 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:45.307521 7f9d7a161700 10 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1 r = 0
2014-06-12 14:46:45.307526 7f9d7a161700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:45.307544 7f9d7a161700 15 _write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 3862528~331776
2014-06-12 14:46:45.307700 7f9d7a161700 10 lookup_strip_header done  cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:45.307912 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:45.309728 7f9d7a161700 10 _generic_write 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 3862528~331776 = 0
2014-06-12 14:46:45.309839 7f9d7a161700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:45.309876 7f9d7a161700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:45.309899 7f9d7a161700 15 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3
2014-06-12 14:46:45.309924 7f9d7a161700 10 _setattrs 3.71f_head/c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 = 0
2014-06-12 14:46:45.310109 7f9d7a161700 20 set_header setting 17162 cid 3.71f_head oid c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 parent seq 0
2014-06-12 14:46:45.310180 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.310214 7f9d7a161700 20 set_header setting 2330 cid meta oid 1fe0985b/pglog_3.71f/0//-1 parent seq 0
2014-06-12 14:46:45.310242 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:45.311024 7f9d693e3700 10 getattr 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 '_' = 0
2014-06-12 14:46:45.311053 7f9d693e3700 15 getattr 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 'snapset'
2014-06-12 14:46:45.311261 7f9d693e3700 10 getattr 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 'snapset' = 0
2014-06-12 14:46:45.311275 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] populate_obc_watchers 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.311291 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.311301 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] get_object_context: creating obc from disk: 0x37e4b600
2014-06-12 14:46:45.311310 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] get_object_context: 0x37e4b600 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 rwstate(none n=0 w=0) oi: 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3(173'700 client.4280.0:35126 wrlock_by=unknown.0.0:0 dirty s 4194304 uv700) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:46:45.311326 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] find_object_context 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 @head oi=3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3(173'700 client.4280.0:35126 wrlock_by=unknown.0.0:0 dirty s 4194304 uv700)
2014-06-12 14:46:45.311355 7f9d693e3700  5 -- op tracker -- , seq: 748026, time: 2014-06-12 14:46:45.311347, event: started, op: osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.311363 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] execute_ctx 0x1fc53e00
2014-06-12 14:46:45.311381 7f9d693e3700  5 -- op tracker -- , seq: 748026, time: 2014-06-12 14:46:45.311376, event: started, op: osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.311388 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_op 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] ov 173'700 av 173'2721 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.311403 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_osd_op 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936]
2014-06-12 14:46:45.311414 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.311428 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_osd_op  write 0~679936
2014-06-12 14:46:45.311443 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_osd_op_effects on session 0xa702000
2014-06-12 14:46:45.311453 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] make_writeable 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:46:45.311464 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] make_writeable 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.311476 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] finish_ctx 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 0x1fc53e00 op modify  
2014-06-12 14:46:45.311488 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]  set mtime to 2014-06-12 08:45:47.567748
2014-06-12 14:46:45.311507 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]  final snapset 0=[]:[]+head in 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.311525 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.311536 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] new_repop rep_tid 236324 on osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.311560 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] issue_repop rep_tid 236324 o 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.311583 7f9d693e3700  5 -- op tracker -- , seq: 748026, time: 2014-06-12 14:46:45.311577, event: waiting for subops from 23, op: osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.311605 7f9d693e3700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:45.311629 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2720 (0'0,173'2720] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] append_log log((0'0,173'2720], crt=0'0) [173'2721 (173'700) modify   3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 by client.4280.0:139925 2014-06-12 08:45:47.567748]
2014-06-12 14:46:45.311652 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] add_log_entry 173'2721 (173'700) modify   3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 by client.4280.0:139925 2014-06-12 08:45:47.567748
2014-06-12 14:46:45.311670 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.311700 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2721, trimmed: 
2014-06-12 14:46:45.311718 7f9d693e3700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.311723 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.311724 7f9d693e3700  5 queue_op 0x11eb2480 seq 2466 osr(3.54b 0x276db30) 682229 bytes   (queue has 2 ops and 1016298 bytes)
2014-06-12 14:46:45.311731 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] eval_repop repgather(0x384e35c0 173'2721 rep_tid=236324 committed?=0 applied?=0 lock=0 op=osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.311748 7f9d693e3700 10 osd.2 173 dequeue_op 0x598e800 finish
2014-06-12 14:46:45.311756 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active)
2014-06-12 14:46:45.311761 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.311770 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (2 active)
2014-06-12 14:46:45.311774 7f9d7f16b700  5 _do_op 0x11eb2480 seq 2466 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:46:45.311778 7f9d7f16b700 10 _do_transaction on 0x384e2d80
2014-06-12 14:46:45.311783 7f9d7f16b700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.311764 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x598ee00 prio 63 cost 516096 latency 0.101736 osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]
2014-06-12 14:46:45.311792 7f9d6b3e7700  5 -- op tracker -- , seq: 748027, time: 2014-06-12 14:46:45.311786, event: reached_pg, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.311802 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.311817 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] handle_message: 0x598ee00
2014-06-12 14:46:45.311826 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_op osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.311846 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] get_object_context: found obc in cache: 0x37e4b600
2014-06-12 14:46:45.311854 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] get_object_context: 0x37e4b600 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 rwstate(write n=1 w=0) oi: 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3(173'2721 client.4280.0:139925 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2721) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:46:45.311867 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] find_object_context 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 @head oi=3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3(173'2721 client.4280.0:139925 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2721)
2014-06-12 14:46:45.311886 7f9d6b3e7700  5 -- op tracker -- , seq: 748027, time: 2014-06-12 14:46:45.311881, event: started, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.311896 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.311893 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] execute_ctx 0x29edf000
2014-06-12 14:46:45.311908 7f9d6b3e7700  5 -- op tracker -- , seq: 748027, time: 2014-06-12 14:46:45.311904, event: started, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.311913 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.311913 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_op 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] ov 173'2721 av 173'2722 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.311924 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_osd_op 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096]
2014-06-12 14:46:45.311933 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.311944 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_osd_op  write 679936~516096
2014-06-12 14:46:45.311956 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_osd_op_effects on session 0xa702000
2014-06-12 14:46:45.311971 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] make_writeable 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:46:45.311983 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] make_writeable 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.311993 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] finish_ctx 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 0x29edf000 op modify  
2014-06-12 14:46:45.312005 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]  set mtime to 2014-06-12 08:45:47.589416
2014-06-12 14:46:45.312042 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]  final snapset 0=[]:[]+head in 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.312060 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.312077 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] new_repop rep_tid 236325 on osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.312092 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] issue_repop rep_tid 236325 o 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.312117 7f9d6b3e7700  5 -- op tracker -- , seq: 748027, time: 2014-06-12 14:46:45.312112, event: waiting for subops from 23, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.312144 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:45.312161 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2721 (0'0,173'2721] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] append_log log((0'0,173'2721], crt=0'0) [173'2722 (173'2721) modify   3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 by client.4280.0:139926 2014-06-12 08:45:47.589416]
2014-06-12 14:46:45.312180 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] add_log_entry 173'2722 (173'2721) modify   3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 by client.4280.0:139926 2014-06-12 08:45:47.589416
2014-06-12 14:46:45.312204 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.312235 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2722, trimmed: 
2014-06-12 14:46:45.312240 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.312253 7f9d6b3e7700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.312256 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.312257 7f9d6b3e7700  5 queue_op 0x89dbec0 seq 2467 osr(3.54b 0x276db30) 518389 bytes   (queue has 3 ops and 1534687 bytes)
2014-06-12 14:46:45.312263 7f9d7f16b700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.312267 7f9d7f16b700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:46:45.312273 7f9d7f16b700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.312264 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] eval_repop repgather(0x6590240 173'2722 rep_tid=236325 committed?=0 applied?=0 lock=0 op=osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.312282 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x598ee00 finish
2014-06-12 14:46:45.312289 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:45.312292 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.312300 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (3 active)
2014-06-12 14:46:45.312322 7f9d7a161700 10 _do_op 0x9797f20 seq 2534 r = 0, finisher 0x2e1cece0 0
2014-06-12 14:46:45.312358 7f9d7a161700 10 _finish_op 0x9797f20 seq 2534 osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:45.312366 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d0240 (2 active)
2014-06-12 14:46:45.312374 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.312402 7f9d7f16b700 15 _write 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 0~679936
2014-06-12 14:46:45.312558 7f9d7f16b700 10 lookup_strip_header done  cid 3.54b_head oid 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.312767 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:45.312353 7f9d7715b700  5 -- op tracker -- , seq: 748031, time: 2014-06-12 14:46:45.312346, event: commit_sent, op: osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.312957 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2702 (0'0,173'2702] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2700 active] sub_op_modify_commit on op osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:45.312981 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.315651 7f9d7f16b700 10 _generic_write 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 0~679936 = 0
2014-06-12 14:46:45.315736 7f9d7f16b700 15 _setattrs 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.315756 7f9d7f16b700 10 _setattrs 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 = 0
2014-06-12 14:46:45.315793 7f9d7f16b700 15 _setattrs 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.315803 7f9d7f16b700 10 _setattrs 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 = 0
2014-06-12 14:46:45.316084 7f9d7f16b700 20 set_header setting 15954 cid 3.54b_head oid 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 parent seq 0
2014-06-12 14:46:45.316220 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.316234 7f9d7f16b700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:46:45.316244 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:45.316754 7f9d66bde700 10 osd.2 173 dequeue_op 0x598e400 prio 63 cost 28672 latency 0.097788 osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]
2014-06-12 14:46:45.316880 7f9d66bde700  5 -- op tracker -- , seq: 748030, time: 2014-06-12 14:46:45.316874, event: reached_pg, op: osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.316891 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.316905 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] handle_message: 0x598e400
2014-06-12 14:46:45.316915 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_op osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.316940 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] get_object_context: found obc in cache: 0x37e4b600
2014-06-12 14:46:45.316949 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] get_object_context: 0x37e4b600 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 rwstate(write n=2 w=0) oi: 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3(173'2722 client.4280.0:139926 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2722) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:46:45.316964 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] find_object_context 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 @head oi=3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3(173'2722 client.4280.0:139926 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2722)
2014-06-12 14:46:45.316987 7f9d66bde700  5 -- op tracker -- , seq: 748030, time: 2014-06-12 14:46:45.316982, event: started, op: osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.317000 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] execute_ctx 0x5d20600
2014-06-12 14:46:45.317015 7f9d66bde700  5 -- op tracker -- , seq: 748030, time: 2014-06-12 14:46:45.317011, event: started, op: osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.317021 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_op 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] ov 173'2722 av 173'2723 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.317033 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_osd_op 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672]
2014-06-12 14:46:45.317043 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.317055 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_osd_op  write 1196032~28672
2014-06-12 14:46:45.317069 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] do_osd_op_effects on session 0xa702000
2014-06-12 14:46:45.317078 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] make_writeable 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:46:45.317089 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] make_writeable 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.317097 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] finish_ctx 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 0x5d20600 op modify  
2014-06-12 14:46:45.317108 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]  set mtime to 2014-06-12 08:45:47.595409
2014-06-12 14:46:45.317130 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]  final snapset 0=[]:[]+head in 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.317147 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.317157 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] new_repop rep_tid 236326 on osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.317171 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] issue_repop rep_tid 236326 o 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.317195 7f9d66bde700  5 -- op tracker -- , seq: 748030, time: 2014-06-12 14:46:45.317190, event: waiting for subops from 23, op: osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.317214 7f9d66bde700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:45.317234 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2722 (0'0,173'2722] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] append_log log((0'0,173'2722], crt=0'0) [173'2723 (173'2722) modify   3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 by client.4280.0:139927 2014-06-12 08:45:47.595409]
2014-06-12 14:46:45.317253 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] add_log_entry 173'2723 (173'2722) modify   3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 by client.4280.0:139927 2014-06-12 08:45:47.595409
2014-06-12 14:46:45.317271 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.317301 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2723, trimmed: 
2014-06-12 14:46:45.317316 7f9d66bde700  5 queue_transactions existing osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.317320 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.317321 7f9d66bde700  5 queue_op 0x118a44e0 seq 2468 osr(3.54b 0x276db30) 30965 bytes   (queue has 3 ops and 1231583 bytes)
2014-06-12 14:46:45.317333 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7440 (3 active)
2014-06-12 14:46:45.317328 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] eval_repop repgather(0x2d813ec0 173'2723 rep_tid=236326 committed?=0 applied?=0 lock=0 op=osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.317345 7f9d66bde700 10 osd.2 173 dequeue_op 0x598e400 finish
2014-06-12 14:46:45.317351 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.317355 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.317375 7f9d7795c700  5 -- op tracker -- , seq: 748031, time: 2014-06-12 14:46:45.317369, event: sub_op_applied, op: osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.317382 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2702 (0'0,173'2702] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2701 active] sub_op_modify_applied on 0x62c0540 op osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.317396 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2702 (0'0,173'2702] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2701 active] op_applied on primary on version 173'2702
2014-06-12 14:46:45.317404 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2702 (0'0,173'2702] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2701 active] op_applied on replica on version 173'2702
2014-06-12 14:46:45.317425 7f9d7795c700  5 -- op tracker -- , seq: 748031, time: 2014-06-12 14:46:45.317422, event: done, op: osd_sub_op(client.4308.0:145289 3.71f c4edbf1f/rbd_data.10ab6b8b4567.0000000000000f9c/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.320588 7f9d7f16b700 10 _do_op 0x11eb2480 seq 2466 r = 0, finisher 0x7eb94c0 0x314fdf00
2014-06-12 14:46:45.320624 7f9d7f16b700 10 _finish_op 0x11eb2480 seq 2466 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.320646 7f9d7e169700  5 _do_op 0x89dbec0 seq 2467 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:46:45.320652 7f9d7e169700 10 _do_transaction on 0x51dcf00
2014-06-12 14:46:45.320667 7f9d7e169700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.320823 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.320843 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.323073 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] op_commit: 236324
2014-06-12 14:46:45.323122 7f9d7715b700  5 -- op tracker -- , seq: 748026, time: 2014-06-12 14:46:45.323112, event: op_commit, op: osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.325830 7f9d57543700  5 -- op tracker -- , seq: 748032, time: 2014-06-12 14:46:45.325716, event: header_read, op: osd_sub_op_reply(client.4280.0:139925 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.325855 7f9d57543700  5 -- op tracker -- , seq: 748032, time: 2014-06-12 14:46:45.325720, event: throttled, op: osd_sub_op_reply(client.4280.0:139925 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.325863 7f9d57543700  5 -- op tracker -- , seq: 748032, time: 2014-06-12 14:46:45.325747, event: all_read, op: osd_sub_op_reply(client.4280.0:139925 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.325870 7f9d57543700  5 -- op tracker -- , seq: 748032, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4280.0:139925 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.325881 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.325881 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4280.0:139925 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:45.325891 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:45.325906 7f9d57543700 15 osd.2 173 enqueue_op 0x2d17b600 prio 196 cost 0 latency 0.000189 osd_sub_op_reply(client.4280.0:139925 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:45.325960 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.325973 7f9d64bda700 10 osd.2 173 dequeue_op 0x2d17b600 prio 196 cost 0 latency 0.000256 osd_sub_op_reply(client.4280.0:139925 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean]
2014-06-12 14:46:45.326011 7f9d64bda700  5 -- op tracker -- , seq: 748032, time: 2014-06-12 14:46:45.326007, event: reached_pg, op: osd_sub_op_reply(client.4280.0:139925 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.326025 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] handle_message: 0x2d17b600
2014-06-12 14:46:45.326044 7f9d64bda700  5 -- op tracker -- , seq: 748032, time: 2014-06-12 14:46:45.326041, event: started, op: osd_sub_op_reply(client.4280.0:139925 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.326051 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] sub_op_modify_reply: tid 236324 op  ack_type 4 from 23
2014-06-12 14:46:45.326079 7f9d64bda700  5 -- op tracker -- , seq: 748026, time: 2014-06-12 14:46:45.326066, event: sub_op_commit_rec, op: osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.326088 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2720 crt=0'0 lcod 173'2719 mlcod 173'2719 active+clean] repop_all_committed: repop tid 236324 all committed 
2014-06-12 14:46:45.326099 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2719 active+clean] eval_repop repgather(0x384e35c0 173'2721 rep_tid=236324 committed?=1 applied?=0 lock=0 op=osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.326115 7f9d7e169700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.326123 7f9d7e169700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:46:45.326117 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2719 active+clean] log_op_stats osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4 inb 680215 outb 0 rlat 0.000000 lat 0.136659
2014-06-12 14:46:45.326132 7f9d7e169700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.326139 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2719 active+clean] publish_stats_to_osd 173:3623
2014-06-12 14:46:45.326152 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2719 active+clean]  sending commit on repgather(0x384e35c0 173'2721 rep_tid=236324 committed?=1 applied?=0 lock=0 op=osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) 0x1f889b80
2014-06-12 14:46:45.326167 7f9d7e169700 15 _write 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 679936~516096
2014-06-12 14:46:45.326203 7f9d64bda700  5 -- op tracker -- , seq: 748026, time: 2014-06-12 14:46:45.326196, event: commit_sent, op: osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.326211 7f9d64bda700 10 osd.2 173 dequeue_op 0x2d17b600 finish
2014-06-12 14:46:45.326218 7f9d64bda700  5 -- op tracker -- , seq: 748032, time: 2014-06-12 14:46:45.326215, event: done, op: osd_sub_op_reply(client.4280.0:139925 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.326235 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.326238 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.326405 7f9d7e169700 10 lookup_strip_header done  cid 3.54b_head oid 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.326688 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:45.329412 7f9d7e169700 10 _generic_write 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 679936~516096 = 0
2014-06-12 14:46:45.329482 7f9d7e169700 15 _setattrs 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.329497 7f9d7e169700 10 _setattrs 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 = 0
2014-06-12 14:46:45.329509 7f9d7e169700 15 _setattrs 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.329517 7f9d7e169700 10 _setattrs 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 = 0
2014-06-12 14:46:45.329777 7f9d7e169700 20 set_header setting 15954 cid 3.54b_head oid 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 parent seq 0
2014-06-12 14:46:45.329868 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.329885 7f9d7e169700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:46:45.329896 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:45.333010 7f9d7e169700 10 _do_op 0x89dbec0 seq 2467 r = 0, finisher 0x889aa00 0x1e627880
2014-06-12 14:46:45.333065 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2719 active+clean] op_commit: 236325
2014-06-12 14:46:45.333101 7f9d7715b700  5 -- op tracker -- , seq: 748027, time: 2014-06-12 14:46:45.333092, event: op_commit, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.333133 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2719 active+clean] op_applied: 236324
2014-06-12 14:46:45.333155 7f9d7795c700  5 -- op tracker -- , seq: 748026, time: 2014-06-12 14:46:45.333148, event: op_applied, op: osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.333163 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2719 active+clean] op_applied on primary on version 173'2721
2014-06-12 14:46:45.333172 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2719 active+clean] repop_all_applied: repop tid 236324 all applied 
2014-06-12 14:46:45.333188 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2719 active+clean] eval_repop repgather(0x384e35c0 173'2721 rep_tid=236324 committed?=1 applied?=1 lock=0 op=osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.333203 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2719 active+clean] log_op_stats osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4 inb 680215 outb 0 rlat 0.000000 lat 0.143745
2014-06-12 14:46:45.333219 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2719 active+clean] publish_stats_to_osd 173:3624
2014-06-12 14:46:45.333231 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2719 active+clean]  requeue_ops 
2014-06-12 14:46:45.333240 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean]  removing repgather(0x384e35c0 173'2721 rep_tid=236324 committed?=1 applied?=1 op=osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.333253 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean]    q front is repgather(0x384e35c0 173'2721 rep_tid=236324 committed?=1 applied?=1 op=osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.333265 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean] remove_repop repgather(0x384e35c0 173'2721 rep_tid=236324 committed?=1 applied?=1 op=osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.333276 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean]  requeue_ops 
2014-06-12 14:46:45.333331 7f9d7f16b700  5 -- op tracker -- , seq: 748026, time: 2014-06-12 14:46:45.333315, event: done, op: osd_op(client.4280.0:139925 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.333366 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (2 active)
2014-06-12 14:46:45.333370 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.333380 7f9d7e169700 10 _finish_op 0x89dbec0 seq 2467 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.333400 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean] op_applied: 236325
2014-06-12 14:46:45.333417 7f9d7795c700  5 -- op tracker -- , seq: 748027, time: 2014-06-12 14:46:45.333412, event: op_applied, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.333425 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean] op_applied on primary on version 173'2722
2014-06-12 14:46:45.333448 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (1 active)
2014-06-12 14:46:45.333459 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.333898 7f9d7815d700  5 _do_op 0x118a44e0 seq 2468 osr(3.54b 0x276db30)/0x276db30 start
2014-06-12 14:46:45.333907 7f9d7815d700 10 _do_transaction on 0x2d813800
2014-06-12 14:46:45.333915 7f9d7815d700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.334054 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.334074 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.334136 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.334163 7f9d7815d700 15 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.334169 7f9d7815d700 10 _omap_rmkeys meta/1fe10acb/pglog_3.54b/0//-1 r = 0
2014-06-12 14:46:45.334176 7f9d7815d700 15 _omap_setkeys meta/1fe10acb/pglog_3.54b/0//-1
2014-06-12 14:46:45.334360 7f9d7815d700 15 _write 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 1196032~28672
2014-06-12 14:46:45.334518 7f9d7815d700 10 lookup_strip_header done  cid 3.54b_head oid 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.334534 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:45.334642 7f9d7815d700 10 _generic_write 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 1196032~28672 = 0
2014-06-12 14:46:45.334655 7f9d7815d700 15 _setattrs 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.334670 7f9d7815d700 10 _setattrs 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 = 0
2014-06-12 14:46:45.334676 7f9d7815d700 15 _setattrs 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3
2014-06-12 14:46:45.334690 7f9d7815d700 10 _setattrs 3.54b_head/3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 = 0
2014-06-12 14:46:45.334807 7f9d7815d700 20 set_header setting 15954 cid 3.54b_head oid 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 parent seq 0
2014-06-12 14:46:45.334832 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.334844 7f9d7815d700 20 set_header setting 2088 cid meta oid 1fe10acb/pglog_3.54b/0//-1 parent seq 0
2014-06-12 14:46:45.334957 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:45.335234 7f9d7815d700 10 _do_op 0x118a44e0 seq 2468 r = 0, finisher 0x8357ac0 0x2628fbc0
2014-06-12 14:46:45.335245 7f9d7815d700 10 _finish_op 0x118a44e0 seq 2468 osr(3.54b 0x276db30)/0x276db30
2014-06-12 14:46:45.335269 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean] op_applied: 236326
2014-06-12 14:46:45.335294 7f9d7795c700  5 -- op tracker -- , seq: 748030, time: 2014-06-12 14:46:45.335286, event: op_applied, op: osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.335304 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean] op_applied on primary on version 173'2723
2014-06-12 14:46:45.335333 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7440 (0 active)
2014-06-12 14:46:45.335337 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.336107 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean] op_commit: 236326
2014-06-12 14:46:45.336145 7f9d7715b700  5 -- op tracker -- , seq: 748030, time: 2014-06-12 14:46:45.336136, event: op_commit, op: osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.344318 7f9d58d5b700  5 -- op tracker -- , seq: 748033, time: 2014-06-12 14:46:45.344170, event: header_read, op: osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.344331 7f9d58d5b700  5 -- op tracker -- , seq: 748033, time: 2014-06-12 14:46:45.344174, event: throttled, op: osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.344340 7f9d58d5b700  5 -- op tracker -- , seq: 748033, time: 2014-06-12 14:46:45.344282, event: all_read, op: osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.344349 7f9d58d5b700  5 -- op tracker -- , seq: 748033, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.344354 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.344361 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:45.344370 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016500 prio 127 cost 70439 latency 0.000199 osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.344401 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.344408 7f9d6abe6700 10 osd.2 173 dequeue_op 0x34016500 prio 127 cost 70439 latency 0.000237 osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.5c5( v 173'2846 (0'0,173'2846] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2845 active]
2014-06-12 14:46:45.344432 7f9d6abe6700  5 -- op tracker -- , seq: 748033, time: 2014-06-12 14:46:45.344427, event: reached_pg, op: osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.344441 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2846 (0'0,173'2846] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2845 active] handle_message: 0x34016500
2014-06-12 14:46:45.344456 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2846 (0'0,173'2846] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2845 active] sub_op_modify trans f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 v 173'2847 (transaction) 174
2014-06-12 14:46:45.344476 7f9d6abe6700  5 -- op tracker -- , seq: 748033, time: 2014-06-12 14:46:45.344472, event: started, op: osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.344492 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2846 (0'0,173'2846] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2845 active] append_log log((0'0,173'2846], crt=0'0) [173'2847 (173'797) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145129 2014-06-12 08:45:47.687267]
2014-06-12 14:46:45.344513 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2847 (0'0,173'2847] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2845 active] add_log_entry 173'2847 (173'797) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145129 2014-06-12 08:45:47.687267
2014-06-12 14:46:45.344579 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2847 (0'0,173'2847] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2845 active] append_log  adding 1 keys
2014-06-12 14:46:45.344610 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2847, trimmed: 
2014-06-12 14:46:45.344634 7f9d6abe6700  5 -- op tracker -- , seq: 748033, time: 2014-06-12 14:46:45.344629, event: started, op: osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.344644 7f9d6abe6700  5 queue_transactions existing osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.344648 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.344650 7f9d6abe6700  5 queue_op 0x69c99e0 seq 2601 osr(3.5c5 0x276dde0) 71925 bytes   (queue has 1 ops and 71925 bytes)
2014-06-12 14:46:45.344660 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439d40 (1 active)
2014-06-12 14:46:45.344668 7f9d82171700  5 _do_op 0x69c99e0 seq 2601 osr(3.5c5 0x276dde0)/0x276dde0 start
2014-06-12 14:46:45.344671 7f9d82171700 10 _do_transaction on 0x191d1b30
2014-06-12 14:46:45.344657 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2847 (0'0,173'2847] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2845 active] do_sub_op osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.344673 7f9d6abe6700 10 osd.2 173 dequeue_op 0x34016500 finish
2014-06-12 14:46:45.344676 7f9d82171700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.344676 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.344679 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.344848 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.344874 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.344931 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.344960 7f9d82171700 15 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.344968 7f9d82171700 10 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1 r = 0
2014-06-12 14:46:45.344975 7f9d82171700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.344993 7f9d82171700 15 _write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 0~69632
2014-06-12 14:46:45.345139 7f9d82171700 10 lookup_strip_header done  cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.345321 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:45.345689 7f9d82171700 10 _generic_write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 0~69632 = 0
2014-06-12 14:46:45.345699 7f9d57543700  5 -- op tracker -- , seq: 748034, time: 2014-06-12 14:46:45.345636, event: header_read, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ack, result = 0)
2014-06-12 14:46:45.345707 7f9d82171700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.345711 7f9d57543700  5 -- op tracker -- , seq: 748034, time: 2014-06-12 14:46:45.345638, event: throttled, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ack, result = 0)
2014-06-12 14:46:45.345719 7f9d82171700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.345719 7f9d57543700  5 -- op tracker -- , seq: 748034, time: 2014-06-12 14:46:45.345659, event: all_read, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ack, result = 0)
2014-06-12 14:46:45.345724 7f9d82171700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.345728 7f9d57543700  5 -- op tracker -- , seq: 748034, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ack, result = 0)
2014-06-12 14:46:45.345732 7f9d82171700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.345734 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:45.345741 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:45.345749 7f9d57543700 15 osd.2 173 enqueue_op 0x2d17a500 prio 196 cost 0 latency 0.000113 osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ack, result = 0) v2
2014-06-12 14:46:45.345780 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.345788 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2d17a500 prio 196 cost 0 latency 0.000152 osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ack, result = 0) v2 pg pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean]
2014-06-12 14:46:45.345811 7f9d6a3e5700  5 -- op tracker -- , seq: 748034, time: 2014-06-12 14:46:45.345808, event: reached_pg, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ack, result = 0)
2014-06-12 14:46:45.345828 7f9d57543700  5 -- op tracker -- , seq: 748035, time: 2014-06-12 14:46:45.345774, event: header_read, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.345823 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean] handle_message: 0x2d17a500
2014-06-12 14:46:45.345838 7f9d57543700  5 -- op tracker -- , seq: 748035, time: 2014-06-12 14:46:45.345775, event: throttled, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.345848 7f9d57543700  5 -- op tracker -- , seq: 748035, time: 2014-06-12 14:46:45.345782, event: all_read, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.345858 7f9d57543700  5 -- op tracker -- , seq: 748035, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.345859 7f9d82171700 20 set_header setting 17924 cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 parent seq 0
2014-06-12 14:46:45.345864 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:45.345869 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:45.345876 7f9d57543700 15 osd.2 173 enqueue_op 0x2519d700 prio 196 cost 0 latency 0.000102 osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:45.345870 7f9d6a3e5700  5 -- op tracker -- , seq: 748034, time: 2014-06-12 14:46:45.345840, event: started, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ack, result = 0)
2014-06-12 14:46:45.345887 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean] sub_op_modify_reply: tid 236325 op  ack_type 1 from 23
2014-06-12 14:46:45.345901 7f9d82171700 20 set_header setting 2264 cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1 parent seq 0
2014-06-12 14:46:45.345911 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:45.345919 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.345921 7f9d6a3e5700  5 -- op tracker -- , seq: 748027, time: 2014-06-12 14:46:45.345906, event: sub_op_applied_rec, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.345928 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:45.345932 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean] repop_all_applied: repop tid 236325 all applied 
2014-06-12 14:46:45.345947 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean] eval_repop repgather(0x6590240 173'2722 rep_tid=236325 committed?=0 applied?=1 lock=0 op=osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.345967 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean]  sending ack on repgather(0x6590240 173'2722 rep_tid=236325 committed?=0 applied?=1 lock=0 op=osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) 0x31656500
2014-06-12 14:46:45.346007 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2d17a500 finish
2014-06-12 14:46:45.346017 7f9d6a3e5700  5 -- op tracker -- , seq: 748034, time: 2014-06-12 14:46:45.346013, event: done, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ack, result = 0)
2014-06-12 14:46:45.346018 7f9d69be4700 10 osd.2 173 dequeue_op 0x2519d700 prio 196 cost 0 latency 0.000243 osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean]
2014-06-12 14:46:45.346034 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:45.346038 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.346037 7f9d69be4700  5 -- op tracker -- , seq: 748035, time: 2014-06-12 14:46:45.346034, event: reached_pg, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.346045 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean] handle_message: 0x2519d700
2014-06-12 14:46:45.346060 7f9d69be4700  5 -- op tracker -- , seq: 748035, time: 2014-06-12 14:46:45.346057, event: started, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.346066 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean] sub_op_modify_reply: tid 236325 op  ack_type 4 from 23
2014-06-12 14:46:45.346084 7f9d69be4700  5 -- op tracker -- , seq: 748027, time: 2014-06-12 14:46:45.346078, event: sub_op_commit_rec, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.346090 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2721 crt=0'0 lcod 173'2720 mlcod 173'2720 active+clean] repop_all_committed: repop tid 236325 all committed 
2014-06-12 14:46:45.346099 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2720 active+clean] eval_repop repgather(0x6590240 173'2722 rep_tid=236325 committed?=1 applied?=1 lock=0 op=osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.346115 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2720 active+clean] log_op_stats osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.135978 lat 0.136087
2014-06-12 14:46:45.346132 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2720 active+clean] publish_stats_to_osd 173:3625
2014-06-12 14:46:45.346146 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2720 active+clean]  sending commit on repgather(0x6590240 173'2722 rep_tid=236325 committed?=1 applied?=1 lock=0 op=osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) 0x11666500
2014-06-12 14:46:45.346179 7f9d69be4700  5 -- op tracker -- , seq: 748027, time: 2014-06-12 14:46:45.346174, event: commit_sent, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.346187 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2720 active+clean]  requeue_ops 
2014-06-12 14:46:45.346197 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2721 active+clean]  removing repgather(0x6590240 173'2722 rep_tid=236325 committed?=1 applied?=1 op=osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.346211 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2721 active+clean]    q front is repgather(0x6590240 173'2722 rep_tid=236325 committed?=1 applied?=1 op=osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.346225 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2721 active+clean] remove_repop repgather(0x6590240 173'2722 rep_tid=236325 committed?=1 applied?=1 op=osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.346237 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2721 active+clean]  requeue_ops 
2014-06-12 14:46:45.346255 7f9d69be4700  5 -- op tracker -- , seq: 748027, time: 2014-06-12 14:46:45.346251, event: done, op: osd_op(client.4280.0:139926 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~516096] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.346270 7f9d69be4700 10 osd.2 173 dequeue_op 0x2519d700 finish
2014-06-12 14:46:45.346277 7f9d69be4700  5 -- op tracker -- , seq: 748035, time: 2014-06-12 14:46:45.346274, event: done, op: osd_sub_op_reply(client.4280.0:139926 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.346287 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.346290 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.346303 7f9d82171700 10 _do_op 0x69c99e0 seq 2601 r = 0, finisher 0x89f4500 0
2014-06-12 14:46:45.346313 7f9d82171700 10 _finish_op 0x69c99e0 seq 2601 osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.346321 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439d40 (0 active)
2014-06-12 14:46:45.346325 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.346340 7f9d7715b700  5 -- op tracker -- , seq: 748033, time: 2014-06-12 14:46:45.346335, event: commit_sent, op: osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.346356 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2847 (0'0,173'2847] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2845 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:45.346380 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.346402 7f9d7795c700  5 -- op tracker -- , seq: 748033, time: 2014-06-12 14:46:45.346397, event: sub_op_applied, op: osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.346408 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2847 (0'0,173'2847] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2846 active] sub_op_modify_applied on 0x191d1a40 op osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.346418 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2847 (0'0,173'2847] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2846 active] op_applied on primary on version 173'2847
2014-06-12 14:46:45.346428 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2847 (0'0,173'2847] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2846 active] op_applied on replica on version 173'2847
2014-06-12 14:46:45.346444 7f9d7795c700  5 -- op tracker -- , seq: 748033, time: 2014-06-12 14:46:45.346441, event: done, op: osd_sub_op(client.4314.0:145129 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.350637 7f9d57543700  5 -- op tracker -- , seq: 748036, time: 2014-06-12 14:46:45.350572, event: header_read, op: osd_sub_op_reply(client.4280.0:139927 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.350651 7f9d57543700  5 -- op tracker -- , seq: 748036, time: 2014-06-12 14:46:45.350575, event: throttled, op: osd_sub_op_reply(client.4280.0:139927 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.350658 7f9d57543700  5 -- op tracker -- , seq: 748036, time: 2014-06-12 14:46:45.350598, event: all_read, op: osd_sub_op_reply(client.4280.0:139927 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.350667 7f9d57543700  5 -- op tracker -- , seq: 748036, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4280.0:139927 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.350672 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4280.0:139927 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:45.350678 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:46:45.350687 7f9d57543700 15 osd.2 173 enqueue_op 0x2519d500 prio 196 cost 0 latency 0.000115 osd_sub_op_reply(client.4280.0:139927 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:45.350719 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.350726 7f9d653db700 10 osd.2 173 dequeue_op 0x2519d500 prio 196 cost 0 latency 0.000154 osd_sub_op_reply(client.4280.0:139927 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0) v2 pg pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2721 active+clean]
2014-06-12 14:46:45.350744 7f9d653db700  5 -- op tracker -- , seq: 748036, time: 2014-06-12 14:46:45.350741, event: reached_pg, op: osd_sub_op_reply(client.4280.0:139927 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.350752 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2721 active+clean] handle_message: 0x2519d500
2014-06-12 14:46:45.350766 7f9d653db700  5 -- op tracker -- , seq: 748036, time: 2014-06-12 14:46:45.350763, event: started, op: osd_sub_op_reply(client.4280.0:139927 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.350771 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2721 active+clean] sub_op_modify_reply: tid 236326 op  ack_type 4 from 23
2014-06-12 14:46:45.350789 7f9d653db700  5 -- op tracker -- , seq: 748030, time: 2014-06-12 14:46:45.350783, event: sub_op_commit_rec, op: osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.350795 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2721 active+clean] repop_all_applied: repop tid 236326 all applied 
2014-06-12 14:46:45.350805 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2721 active+clean] eval_repop repgather(0x2d813ec0 173'2723 rep_tid=236326 committed?=0 applied?=1 lock=0 op=osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.350819 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2721 active+clean]  sending ack on repgather(0x2d813ec0 173'2723 rep_tid=236326 committed?=0 applied?=1 lock=0 op=osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) 0x31657180
2014-06-12 14:46:45.350847 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2722 crt=0'0 lcod 173'2721 mlcod 173'2721 active+clean] repop_all_committed: repop tid 236326 all committed 
2014-06-12 14:46:45.350857 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2721 active+clean] eval_repop repgather(0x2d813ec0 173'2723 rep_tid=236326 committed?=1 applied?=1 lock=0 op=osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.350872 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2721 active+clean] log_op_stats osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4 inb 28951 outb 0 rlat 0.131880 lat 0.131906
2014-06-12 14:46:45.350887 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2721 active+clean] publish_stats_to_osd 173:3626
2014-06-12 14:46:45.350898 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2721 active+clean]  sending commit on repgather(0x2d813ec0 173'2723 rep_tid=236326 committed?=1 applied?=1 lock=0 op=osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4) 0x5ffe780
2014-06-12 14:46:45.350931 7f9d653db700  5 -- op tracker -- , seq: 748030, time: 2014-06-12 14:46:45.350924, event: commit_sent, op: osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.350938 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2721 active+clean]  requeue_ops 
2014-06-12 14:46:45.350947 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2722 active+clean]  removing repgather(0x2d813ec0 173'2723 rep_tid=236326 committed?=1 applied?=1 op=osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.350963 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2722 active+clean]    q front is repgather(0x2d813ec0 173'2723 rep_tid=236326 committed?=1 applied?=1 op=osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.350976 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2722 active+clean] remove_repop repgather(0x2d813ec0 173'2723 rep_tid=236326 committed?=1 applied?=1 op=osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:45.350988 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2722 active+clean]  requeue_ops 
2014-06-12 14:46:45.351019 7f9d653db700  5 -- op tracker -- , seq: 748030, time: 2014-06-12 14:46:45.351014, event: done, op: osd_op(client.4280.0:139927 rbd_data.108f6b8b4567.0000000000000e9c [set-alloc-hint object_size 4194304 write_size 4194304,write 1196032~28672] 3.3268754b ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.351034 7f9d653db700 10 osd.2 173 dequeue_op 0x2519d500 finish
2014-06-12 14:46:45.351040 7f9d653db700  5 -- op tracker -- , seq: 748036, time: 2014-06-12 14:46:45.351037, event: done, op: osd_sub_op_reply(client.4280.0:139927 3.54b 3268754b/rbd_data.108f6b8b4567.0000000000000e9c/head//3 [] ondisk, result = 0)
2014-06-12 14:46:45.351050 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.351052 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.360770 7f9d461e0700  5 -- op tracker -- , seq: 748037, time: 2014-06-12 14:46:45.360243, event: header_read, op: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.360794 7f9d461e0700  5 -- op tracker -- , seq: 748037, time: 2014-06-12 14:46:45.360250, event: throttled, op: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.360804 7f9d461e0700  5 -- op tracker -- , seq: 748037, time: 2014-06-12 14:46:45.360718, event: all_read, op: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.360813 7f9d461e0700  5 -- op tracker -- , seq: 748037, time: 0.000000, event: dispatched, op: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.360822 7f9d461e0700 20 osd.2 173 should_share_map client.4293 172.16.96.220:0/1241966 173
2014-06-12 14:46:45.360854 7f9d461e0700 15 osd.2 173 enqueue_op 0x9fac900 prio 63 cost 409600 latency 0.000611 osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.360893 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.360906 7f9d63bd8700 10 osd.2 173 dequeue_op 0x9fac900 prio 63 cost 409600 latency 0.000662 osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]
2014-06-12 14:46:45.360936 7f9d63bd8700  5 -- op tracker -- , seq: 748037, time: 2014-06-12 14:46:45.360931, event: reached_pg, op: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.360948 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:45.360963 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] handle_message: 0x9fac900
2014-06-12 14:46:45.360975 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_op osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:45.361003 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] get_object_context: found obc in cache: 0x20107600
2014-06-12 14:46:45.361016 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] get_object_context: 0x20107600 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 rwstate(write n=4 w=0) oi: e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3(173'2594 client.4293.0:144211 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2594) ssc: 0x72b8d20 snapset: 0=[]:[]+head
2014-06-12 14:46:45.361035 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] find_object_context e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 @head oi=e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3(173'2594 client.4293.0:144211 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2594)
2014-06-12 14:46:45.361063 7f9d63bd8700  5 -- op tracker -- , seq: 748037, time: 2014-06-12 14:46:45.361057, event: started, op: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.361072 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] execute_ctx 0x60ad600
2014-06-12 14:46:45.361091 7f9d63bd8700  5 -- op tracker -- , seq: 748037, time: 2014-06-12 14:46:45.361086, event: started, op: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.361099 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_op e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] ov 173'2594 av 173'2595 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:45.361116 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600]
2014-06-12 14:46:45.361129 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:45.361143 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op  write 3784704~409600
2014-06-12 14:46:45.361158 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] do_osd_op_effects on session 0x5c8cc00
2014-06-12 14:46:45.361171 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] make_writeable e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 snapset=0x72b8d58  snapc=0=[]
2014-06-12 14:46:45.361185 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] make_writeable e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:45.361196 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] finish_ctx e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 0x60ad600 op modify  
2014-06-12 14:46:45.361209 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  set mtime to 2014-06-12 08:45:47.738293
2014-06-12 14:46:45.361233 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  final snapset 0=[]:[]+head in e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.361253 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  zeroing write result code 0
2014-06-12 14:46:45.361265 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] new_repop rep_tid 236327 on osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:45.361290 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] issue_repop rep_tid 236327 o e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.361317 7f9d63bd8700  5 -- op tracker -- , seq: 748037, time: 2014-06-12 14:46:45.361311, event: waiting for subops from 24, op: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.361339 7f9d63bd8700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:45.361361 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2594 (0'0,173'2594] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] append_log log((0'0,173'2594], crt=0'0) [173'2595 (173'2594) modify   e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 by client.4293.0:144212 2014-06-12 08:45:47.738293]
2014-06-12 14:46:45.361386 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] add_log_entry 173'2595 (173'2594) modify   e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 by client.4293.0:144212 2014-06-12 08:45:47.738293
2014-06-12 14:46:45.361408 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] append_log  adding 1 keys
2014-06-12 14:46:45.361441 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2595, trimmed: 
2014-06-12 14:46:45.361465 7f9d63bd8700  5 queue_transactions existing osr(3.4a7 0x276d8c0)/0x276d8c0
2014-06-12 14:46:45.361471 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.361472 7f9d63bd8700  5 queue_op 0x16064420 seq 2341 osr(3.4a7 0x276d8c0) 411893 bytes   (queue has 1 ops and 411893 bytes)
2014-06-12 14:46:45.361480 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] eval_repop repgather(0x31ee7bc0 173'2595 rep_tid=236327 committed?=0 applied?=0 lock=0 op=osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:45.361497 7f9d63bd8700 10 osd.2 173 dequeue_op 0x9fac900 finish
2014-06-12 14:46:45.361501 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.361504 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.361668 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5417b00 (1 active)
2014-06-12 14:46:45.361677 7f9d7e96a700  5 _do_op 0x16064420 seq 2341 osr(3.4a7 0x276d8c0)/0x276d8c0 start
2014-06-12 14:46:45.361681 7f9d7e96a700 10 _do_transaction on 0x9909d40
2014-06-12 14:46:45.361687 7f9d7e96a700 15 _omap_setkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:45.361828 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:45.361849 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.361903 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.361931 7f9d7e96a700 15 _omap_rmkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:45.361940 7f9d7e96a700 10 _omap_rmkeys meta/1fe06fa5/pglog_3.4a7/0//-1 r = 0
2014-06-12 14:46:45.361946 7f9d7e96a700 15 _omap_setkeys meta/1fe06fa5/pglog_3.4a7/0//-1
2014-06-12 14:46:45.361961 7f9d7e96a700 15 _write 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 3784704~409600
2014-06-12 14:46:45.362089 7f9d7e96a700 10 lookup_strip_header done  cid 3.4a7_head oid e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.362110 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:45.363597 7f9d7e96a700 10 _generic_write 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 3784704~409600 = 0
2014-06-12 14:46:45.363663 7f9d7e96a700 15 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.363688 7f9d7e96a700 10 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 = 0
2014-06-12 14:46:45.363695 7f9d7e96a700 15 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3
2014-06-12 14:46:45.363704 7f9d7e96a700 10 _setattrs 3.4a7_head/e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 = 0
2014-06-12 14:46:45.363913 7f9d7e96a700 20 set_header setting 17968 cid 3.4a7_head oid e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 parent seq 0
2014-06-12 14:46:45.363973 7f9d7e96a700 20 set_header setting 2078 cid meta oid 1fe06fa5/pglog_3.4a7/0//-1 parent seq 0
2014-06-12 14:46:45.363991 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.364002 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:45.365858 7f9d56432700  5 -- op tracker -- , seq: 748038, time: 2014-06-12 14:46:45.364949, event: header_read, op: osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.365872 7f9d56432700  5 -- op tracker -- , seq: 748038, time: 2014-06-12 14:46:45.364952, event: throttled, op: osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.365881 7f9d56432700  5 -- op tracker -- , seq: 748038, time: 2014-06-12 14:46:45.365807, event: all_read, op: osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.365889 7f9d56432700  5 -- op tracker -- , seq: 748038, time: 0.000000, event: dispatched, op: osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.365894 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.365901 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:45.365909 7f9d56432700 15 osd.2 173 enqueue_op 0x68a6d00 prio 127 cost 418599 latency 0.000960 osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.365940 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.365948 7f9d68be2700 10 osd.2 173 dequeue_op 0x68a6d00 prio 127 cost 418599 latency 0.000999 osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.60e( v 173'2562 (0'0,173'2562] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2561 active]
2014-06-12 14:46:45.365966 7f9d68be2700  5 -- op tracker -- , seq: 748038, time: 2014-06-12 14:46:45.365963, event: reached_pg, op: osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.365985 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2562 (0'0,173'2562] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2561 active] handle_message: 0x68a6d00
2014-06-12 14:46:45.365998 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2562 (0'0,173'2562] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2561 active] sub_op_modify trans ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 v 173'2563 (transaction) 174
2014-06-12 14:46:45.366014 7f9d68be2700  5 -- op tracker -- , seq: 748038, time: 2014-06-12 14:46:45.366010, event: started, op: osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.366033 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2562 (0'0,173'2562] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2561 active] append_log log((0'0,173'2562], crt=0'0) [173'2563 (173'2562) modify   ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 by client.4296.0:143675 2014-06-12 08:45:47.488508]
2014-06-12 14:46:45.366056 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2563 (0'0,173'2563] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2561 active] add_log_entry 173'2563 (173'2562) modify   ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 by client.4296.0:143675 2014-06-12 08:45:47.488508
2014-06-12 14:46:45.366077 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2563 (0'0,173'2563] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2561 active] append_log  adding 1 keys
2014-06-12 14:46:45.366110 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2563, trimmed: 
2014-06-12 14:46:45.366133 7f9d68be2700  5 -- op tracker -- , seq: 748038, time: 2014-06-12 14:46:45.366127, event: started, op: osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.366144 7f9d68be2700  5 queue_transactions existing osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:45.366148 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.366150 7f9d68be2700  5 queue_op 0x5d51da0 seq 2405 osr(3.60e 0x276dcf0) 420085 bytes   (queue has 2 ops and 831978 bytes)
2014-06-12 14:46:45.366157 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.60e( v 173'2563 (0'0,173'2563] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2561 active] do_sub_op osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.366171 7f9d68be2700 10 osd.2 173 dequeue_op 0x68a6d00 finish
2014-06-12 14:46:45.366175 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.366178 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.366187 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533cd80 (2 active)
2014-06-12 14:46:45.366192 7f9d7d167700  5 _do_op 0x5d51da0 seq 2405 osr(3.60e 0x276dcf0)/0x276dcf0 start
2014-06-12 14:46:45.366197 7f9d7d167700 10 _do_transaction on 0x6295eb0
2014-06-12 14:46:45.366203 7f9d7d167700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.366428 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.366450 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.366528 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.366547 7f9d7d167700 15 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.366554 7f9d7d167700 10 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1 r = 0
2014-06-12 14:46:45.366560 7f9d7d167700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.366573 7f9d7d167700 15 _write 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 3776512~417792
2014-06-12 14:46:45.366681 7f9d7e96a700 10 _do_op 0x16064420 seq 2341 r = 0, finisher 0x8357640 0x1e627780
2014-06-12 14:46:45.366700 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_commit: 236327
2014-06-12 14:46:45.366730 7f9d7715b700  5 -- op tracker -- , seq: 748037, time: 2014-06-12 14:46:45.366720, event: op_commit, op: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.366735 7f9d7d167700 10 lookup_strip_header done  cid 3.60e_head oid ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:45.366744 7f9d7e96a700 10 _finish_op 0x16064420 seq 2341 osr(3.4a7 0x276d8c0)/0x276d8c0
2014-06-12 14:46:45.366754 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5417b00 (1 active)
2014-06-12 14:46:45.366757 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.366759 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_applied: 236327
2014-06-12 14:46:45.366783 7f9d7795c700  5 -- op tracker -- , seq: 748037, time: 2014-06-12 14:46:45.366776, event: op_applied, op: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:45.366801 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] op_applied on primary on version 173'2595
2014-06-12 14:46:45.366862 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:45.368948 7f9d58d5b700  5 -- op tracker -- , seq: 748039, time: 2014-06-12 14:46:45.367151, event: header_read, op: osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.368961 7f9d58d5b700  5 -- op tracker -- , seq: 748039, time: 2014-06-12 14:46:45.367152, event: throttled, op: osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.368970 7f9d58d5b700  5 -- op tracker -- , seq: 748039, time: 2014-06-12 14:46:45.368903, event: all_read, op: osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.368978 7f9d58d5b700  5 -- op tracker -- , seq: 748039, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.368985 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.368992 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:45.369002 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016600 prio 127 cost 516903 latency 0.001850 osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.369014 7f9d7d167700 10 _generic_write 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 3776512~417792 = 0
2014-06-12 14:46:45.369039 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.369047 7f9d623d5700 10 osd.2 173 dequeue_op 0x34016600 prio 127 cost 516903 latency 0.001895 osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.5c5( v 173'2847 (0'0,173'2847] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2846 active]
2014-06-12 14:46:45.369066 7f9d7d167700 15 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:45.369070 7f9d623d5700  5 -- op tracker -- , seq: 748039, time: 2014-06-12 14:46:45.369066, event: reached_pg, op: osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.369080 7f9d7d167700 10 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 = 0
2014-06-12 14:46:45.369086 7f9d7d167700 15 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3
2014-06-12 14:46:45.369079 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2847 (0'0,173'2847] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2846 active] handle_message: 0x34016600
2014-06-12 14:46:45.369095 7f9d7d167700 10 _setattrs 3.60e_head/ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 = 0
2014-06-12 14:46:45.369094 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2847 (0'0,173'2847] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2846 active] sub_op_modify trans f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 v 173'2848 (transaction) 174
2014-06-12 14:46:45.369114 7f9d623d5700  5 -- op tracker -- , seq: 748039, time: 2014-06-12 14:46:45.369109, event: started, op: osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.369143 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2847 (0'0,173'2847] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2846 active] append_log log((0'0,173'2847], crt=0'0) [173'2848 (173'2847) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145130 2014-06-12 08:45:47.743652]
2014-06-12 14:46:45.369177 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2848 (0'0,173'2848] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2846 active] add_log_entry 173'2848 (173'2847) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145130 2014-06-12 08:45:47.743652
2014-06-12 14:46:45.369205 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2848 (0'0,173'2848] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2846 active] append_log  adding 1 keys
2014-06-12 14:46:45.369238 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2848, trimmed: 
2014-06-12 14:46:45.369260 7f9d623d5700  5 -- op tracker -- , seq: 748039, time: 2014-06-12 14:46:45.369255, event: started, op: osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.369270 7f9d623d5700  5 queue_transactions existing osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.369274 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.369276 7f9d623d5700  5 queue_op 0x61ca300 seq 2602 osr(3.5c5 0x276dde0) 518389 bytes   (queue has 2 ops and 938474 bytes)
2014-06-12 14:46:45.369284 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2848 (0'0,173'2848] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2846 active] do_sub_op osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.369298 7f9d623d5700 10 osd.2 173 dequeue_op 0x34016600 finish
2014-06-12 14:46:45.369302 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.369304 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.369351 7f9d7d167700 20 set_header setting 17046 cid 3.60e_head oid ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 parent seq 0
2014-06-12 14:46:45.369413 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.369427 7f9d7d167700 20 set_header setting 2240 cid meta oid 1fe13dcb/pglog_3.60e/0//-1 parent seq 0
2014-06-12 14:46:45.369438 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:45.372556 7f9d7d167700 10 _do_op 0x5d51da0 seq 2405 r = 0, finisher 0x6ab9540 0
2014-06-12 14:46:45.372580 7f9d7d167700 10 _finish_op 0x5d51da0 seq 2405 osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:45.372592 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533cd80 (0 active)
2014-06-12 14:46:45.372598 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439d40 (1 active)
2014-06-12 14:46:45.372607 7f9d7d167700  5 _do_op 0x61ca300 seq 2602 osr(3.5c5 0x276dde0)/0x276dde0 start
2014-06-12 14:46:45.372611 7f9d7d167700 10 _do_transaction on 0x122f62b0
2014-06-12 14:46:45.372618 7f9d7d167700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.372604 7f9d7715b700  5 -- op tracker -- , seq: 748038, time: 2014-06-12 14:46:45.372590, event: commit_sent, op: osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.372624 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2563 (0'0,173'2563] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2561 active] sub_op_modify_commit on op osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:45.372659 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.372764 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.372783 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.372832 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.372858 7f9d7d167700 15 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.372865 7f9d7d167700 10 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1 r = 0
2014-06-12 14:46:45.372871 7f9d7d167700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.372956 7f9d7d167700 15 _write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 69632~516096
2014-06-12 14:46:45.373121 7f9d7d167700 10 lookup_strip_header done  cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.373253 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:45.373638 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.375608 7f9d7d167700 10 _generic_write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 69632~516096 = 0
2014-06-12 14:46:45.375676 7f9d7d167700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.375693 7f9d7d167700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.375698 7f9d7d167700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.375712 7f9d7d167700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.375808 7f9d7795c700  5 -- op tracker -- , seq: 748038, time: 2014-06-12 14:46:45.375798, event: sub_op_applied, op: osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.375818 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2563 (0'0,173'2563] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2562 active] sub_op_modify_applied on 0x6295dc0 op osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.375838 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2563 (0'0,173'2563] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2562 active] op_applied on primary on version 173'2563
2014-06-12 14:46:45.375848 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2563 (0'0,173'2563] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2562 active] op_applied on replica on version 173'2563
2014-06-12 14:46:45.375871 7f9d7795c700  5 -- op tracker -- , seq: 748038, time: 2014-06-12 14:46:45.375867, event: done, op: osd_sub_op(client.4296.0:143675 3.60e ddfa0e0e/rbd_data.109f6b8b4567.00000000000011bf/head//3 [] v 173'2563 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.375926 7f9d7d167700 20 set_header setting 17924 cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 parent seq 0
2014-06-12 14:46:45.376014 7f9d7d167700 20 set_header setting 2264 cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1 parent seq 0
2014-06-12 14:46:45.376029 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.376037 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:45.378677 7f9d7d167700 10 _do_op 0x61ca300 seq 2602 r = 0, finisher 0x5d40ba0 0
2014-06-12 14:46:45.378695 7f9d7d167700 10 _finish_op 0x61ca300 seq 2602 osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.378702 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439d40 (0 active)
2014-06-12 14:46:45.378710 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.378708 7f9d7715b700  5 -- op tracker -- , seq: 748039, time: 2014-06-12 14:46:45.378701, event: commit_sent, op: osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.378719 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2848 (0'0,173'2848] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2846 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:45.378736 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.378760 7f9d7795c700  5 -- op tracker -- , seq: 748039, time: 2014-06-12 14:46:45.378754, event: sub_op_applied, op: osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.378771 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2848 (0'0,173'2848] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2847 active] sub_op_modify_applied on 0x122f61c0 op osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.378793 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2848 (0'0,173'2848] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2847 active] op_applied on primary on version 173'2848
2014-06-12 14:46:45.378811 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2848 (0'0,173'2848] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2847 active] op_applied on replica on version 173'2848
2014-06-12 14:46:45.378836 7f9d7795c700  5 -- op tracker -- , seq: 748039, time: 2014-06-12 14:46:45.378832, event: done, op: osd_sub_op(client.4314.0:145130 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.388196 7f9d58d5b700  5 -- op tracker -- , seq: 748040, time: 2014-06-12 14:46:45.387404, event: header_read, op: osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.388210 7f9d58d5b700  5 -- op tracker -- , seq: 748040, time: 2014-06-12 14:46:45.387405, event: throttled, op: osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.388220 7f9d58d5b700  5 -- op tracker -- , seq: 748040, time: 2014-06-12 14:46:45.388126, event: all_read, op: osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.388225 7f9d58d5b700  5 -- op tracker -- , seq: 748040, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.388229 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.388235 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:45.388241 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34017d00 prio 127 cost 516903 latency 0.000837 osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.388266 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.388273 7f9d643d9700 10 osd.2 173 dequeue_op 0x34017d00 prio 127 cost 516903 latency 0.000869 osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.5c5( v 173'2848 (0'0,173'2848] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2847 active]
2014-06-12 14:46:45.388290 7f9d643d9700  5 -- op tracker -- , seq: 748040, time: 2014-06-12 14:46:45.388288, event: reached_pg, op: osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.388297 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2848 (0'0,173'2848] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2847 active] handle_message: 0x34017d00
2014-06-12 14:46:45.388305 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2848 (0'0,173'2848] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2847 active] sub_op_modify trans f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 v 173'2849 (transaction) 174
2014-06-12 14:46:45.388318 7f9d643d9700  5 -- op tracker -- , seq: 748040, time: 2014-06-12 14:46:45.388316, event: started, op: osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.388331 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2848 (0'0,173'2848] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2847 active] append_log log((0'0,173'2848], crt=0'0) [173'2849 (173'2848) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145131 2014-06-12 08:45:47.763926]
2014-06-12 14:46:45.388357 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2849 (0'0,173'2849] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2847 active] add_log_entry 173'2849 (173'2848) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145131 2014-06-12 08:45:47.763926
2014-06-12 14:46:45.388380 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2849 (0'0,173'2849] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2847 active] append_log  adding 1 keys
2014-06-12 14:46:45.388404 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2849, trimmed: 
2014-06-12 14:46:45.388438 7f9d643d9700  5 -- op tracker -- , seq: 748040, time: 2014-06-12 14:46:45.388435, event: started, op: osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.388447 7f9d643d9700  5 queue_transactions existing osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.388451 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.388452 7f9d643d9700  5 queue_op 0x69b1b00 seq 2603 osr(3.5c5 0x276dde0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:45.388461 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439d40 (1 active)
2014-06-12 14:46:45.388467 7f9d79960700  5 _do_op 0x69b1b00 seq 2603 osr(3.5c5 0x276dde0)/0x276dde0 start
2014-06-12 14:46:45.388457 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2849 (0'0,173'2849] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2847 active] do_sub_op osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.388471 7f9d643d9700 10 osd.2 173 dequeue_op 0x34017d00 finish
2014-06-12 14:46:45.388472 7f9d79960700 10 _do_transaction on 0x2e6369b0
2014-06-12 14:46:45.388474 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.388476 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.388478 7f9d79960700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.388598 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.388633 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.388694 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.388804 7f9d79960700 15 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.388811 7f9d79960700 10 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1 r = 0
2014-06-12 14:46:45.388815 7f9d79960700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.388835 7f9d79960700 15 _write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 585728~516096
2014-06-12 14:46:45.388944 7f9d79960700 10 lookup_strip_header done  cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.389083 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:45.390389 7f9d79960700 10 _generic_write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 585728~516096 = 0
2014-06-12 14:46:45.390422 7f9d79960700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.390430 7f9d79960700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.390434 7f9d79960700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.390439 7f9d79960700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.390574 7f9d79960700 20 set_header setting 17924 cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 parent seq 0
2014-06-12 14:46:45.390618 7f9d79960700 20 set_header setting 2264 cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1 parent seq 0
2014-06-12 14:46:45.390628 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.390634 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:45.400275 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.400384 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.429206 7f9d79960700 10 _do_op 0x69b1b00 seq 2603 r = 0, finisher 0x28e9240 0
2014-06-12 14:46:45.429251 7f9d7715b700  5 -- op tracker -- , seq: 748040, time: 2014-06-12 14:46:45.429240, event: commit_sent, op: osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.429263 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2849 (0'0,173'2849] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2847 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:45.429293 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.429321 7f9d79960700 10 _finish_op 0x69b1b00 seq 2603 osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.429332 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439d40 (0 active)
2014-06-12 14:46:45.429340 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.429347 7f9d7795c700  5 -- op tracker -- , seq: 748040, time: 2014-06-12 14:46:45.429339, event: sub_op_applied, op: osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.429356 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2849 (0'0,173'2849] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active] sub_op_modify_applied on 0x2e6368c0 op osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.429373 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2849 (0'0,173'2849] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active] op_applied on primary on version 173'2849
2014-06-12 14:46:45.429384 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2849 (0'0,173'2849] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active] op_applied on replica on version 173'2849
2014-06-12 14:46:45.429405 7f9d7795c700  5 -- op tracker -- , seq: 748040, time: 2014-06-12 14:46:45.429401, event: done, op: osd_sub_op(client.4314.0:145131 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.435738 7f9d58d5b700  5 -- op tracker -- , seq: 748041, time: 2014-06-12 14:46:45.434960, event: header_read, op: osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.435753 7f9d58d5b700  5 -- op tracker -- , seq: 748041, time: 2014-06-12 14:46:45.434968, event: throttled, op: osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.435765 7f9d58d5b700  5 -- op tracker -- , seq: 748041, time: 2014-06-12 14:46:45.435699, event: all_read, op: osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.435774 7f9d58d5b700  5 -- op tracker -- , seq: 748041, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.435780 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.435788 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:45.435798 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016f00 prio 127 cost 516903 latency 0.000838 osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.435829 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.435839 7f9d633d7700 10 osd.2 173 dequeue_op 0x34016f00 prio 127 cost 516903 latency 0.000879 osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.5c5( v 173'2849 (0'0,173'2849] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active]
2014-06-12 14:46:45.435860 7f9d633d7700  5 -- op tracker -- , seq: 748041, time: 2014-06-12 14:46:45.435856, event: reached_pg, op: osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.435868 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2849 (0'0,173'2849] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active] handle_message: 0x34016f00
2014-06-12 14:46:45.435877 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2849 (0'0,173'2849] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active] sub_op_modify trans f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 v 173'2850 (transaction) 174
2014-06-12 14:46:45.435888 7f9d633d7700  5 -- op tracker -- , seq: 748041, time: 2014-06-12 14:46:45.435885, event: started, op: osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.435901 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2849 (0'0,173'2849] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active] append_log log((0'0,173'2849], crt=0'0) [173'2850 (173'2849) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145132 2014-06-12 08:45:47.811882]
2014-06-12 14:46:45.435921 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2850 (0'0,173'2850] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active] add_log_entry 173'2850 (173'2849) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145132 2014-06-12 08:45:47.811882
2014-06-12 14:46:45.435938 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2850 (0'0,173'2850] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active] append_log  adding 1 keys
2014-06-12 14:46:45.435963 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2850, trimmed: 
2014-06-12 14:46:45.435981 7f9d633d7700  5 -- op tracker -- , seq: 748041, time: 2014-06-12 14:46:45.435978, event: started, op: osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.435990 7f9d633d7700  5 queue_transactions existing osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.435993 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.435995 7f9d633d7700  5 queue_op 0x76d81e0 seq 2604 osr(3.5c5 0x276dde0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:45.436008 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439d40 (1 active)
2014-06-12 14:46:45.436000 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2850 (0'0,173'2850] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active] do_sub_op osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.436014 7f9d633d7700 10 osd.2 173 dequeue_op 0x34016f00 finish
2014-06-12 14:46:45.436015 7f9d8016d700  5 _do_op 0x76d81e0 seq 2604 osr(3.5c5 0x276dde0)/0x276dde0 start
2014-06-12 14:46:45.436017 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.436020 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.436021 7f9d8016d700 10 _do_transaction on 0x2bc87270
2014-06-12 14:46:45.436030 7f9d8016d700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.436153 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.436169 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.436208 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.436225 7f9d8016d700 15 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.436230 7f9d8016d700 10 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1 r = 0
2014-06-12 14:46:45.436233 7f9d8016d700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.436242 7f9d8016d700 15 _write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 1101824~516096
2014-06-12 14:46:45.436508 7f9d8016d700 10 lookup_strip_header done  cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.436633 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:45.438082 7f9d8016d700 10 _generic_write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 1101824~516096 = 0
2014-06-12 14:46:45.438125 7f9d8016d700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.438137 7f9d8016d700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.438143 7f9d8016d700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.438148 7f9d8016d700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.438285 7f9d8016d700 20 set_header setting 17924 cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 parent seq 0
2014-06-12 14:46:45.438343 7f9d8016d700 20 set_header setting 2264 cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1 parent seq 0
2014-06-12 14:46:45.438354 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.438361 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:45.440318 7f9d8016d700 10 _do_op 0x76d81e0 seq 2604 r = 0, finisher 0x6640140 0
2014-06-12 14:46:45.440331 7f9d8016d700 10 _finish_op 0x76d81e0 seq 2604 osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.440335 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439d40 (0 active)
2014-06-12 14:46:45.440344 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.440378 7f9d7795c700  5 -- op tracker -- , seq: 748041, time: 2014-06-12 14:46:45.440368, event: sub_op_applied, op: osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.440398 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2850 (0'0,173'2850] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active] sub_op_modify_applied on 0x2bc87180 op osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.440429 7f9d7795c700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.440446 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2850 (0'0,173'2850] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active] op_applied on primary on version 173'2850
2014-06-12 14:46:45.440456 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2850 (0'0,173'2850] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active] op_applied on replica on version 173'2850
2014-06-12 14:46:45.440475 7f9d7715b700  5 -- op tracker -- , seq: 748041, time: 2014-06-12 14:46:45.440469, event: commit_sent, op: osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.440484 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2850 (0'0,173'2850] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2848 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:45.440502 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.440522 7f9d7715b700  5 -- op tracker -- , seq: 748041, time: 2014-06-12 14:46:45.440519, event: done, op: osd_sub_op(client.4314.0:145132 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.446664 7f9d58d5b700  5 -- op tracker -- , seq: 748042, time: 2014-06-12 14:46:45.445895, event: header_read, op: osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.446675 7f9d58d5b700  5 -- op tracker -- , seq: 748042, time: 2014-06-12 14:46:45.445897, event: throttled, op: osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.446681 7f9d58d5b700  5 -- op tracker -- , seq: 748042, time: 2014-06-12 14:46:45.446635, event: all_read, op: osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.446686 7f9d58d5b700  5 -- op tracker -- , seq: 748042, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.446690 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.446695 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:45.446701 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016e00 prio 127 cost 516903 latency 0.000806 osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.446743 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.446752 7f9d61bd4700 10 osd.2 173 dequeue_op 0x34016e00 prio 127 cost 516903 latency 0.000857 osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.5c5( v 173'2850 (0'0,173'2850] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2849 active]
2014-06-12 14:46:45.446772 7f9d61bd4700  5 -- op tracker -- , seq: 748042, time: 2014-06-12 14:46:45.446769, event: reached_pg, op: osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.446783 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2850 (0'0,173'2850] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2849 active] handle_message: 0x34016e00
2014-06-12 14:46:45.446793 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2850 (0'0,173'2850] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2849 active] sub_op_modify trans f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 v 173'2851 (transaction) 174
2014-06-12 14:46:45.446808 7f9d61bd4700  5 -- op tracker -- , seq: 748042, time: 2014-06-12 14:46:45.446805, event: started, op: osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.446823 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2850 (0'0,173'2850] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2849 active] append_log log((0'0,173'2850], crt=0'0) [173'2851 (173'2850) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145133 2014-06-12 08:45:47.822955]
2014-06-12 14:46:45.446841 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2851 (0'0,173'2851] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2849 active] add_log_entry 173'2851 (173'2850) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145133 2014-06-12 08:45:47.822955
2014-06-12 14:46:45.446858 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2851 (0'0,173'2851] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2849 active] append_log  adding 1 keys
2014-06-12 14:46:45.446883 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2851, trimmed: 
2014-06-12 14:46:45.446902 7f9d61bd4700  5 -- op tracker -- , seq: 748042, time: 2014-06-12 14:46:45.446898, event: started, op: osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.446915 7f9d61bd4700  5 queue_transactions existing osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.446919 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.446920 7f9d61bd4700  5 queue_op 0x24c670e0 seq 2605 osr(3.5c5 0x276dde0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:45.446927 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439d40 (1 active)
2014-06-12 14:46:45.446925 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2851 (0'0,173'2851] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2849 active] do_sub_op osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.446933 7f9d7915f700  5 _do_op 0x24c670e0 seq 2605 osr(3.5c5 0x276dde0)/0x276dde0 start
2014-06-12 14:46:45.446934 7f9d61bd4700 10 osd.2 173 dequeue_op 0x34016e00 finish
2014-06-12 14:46:45.446937 7f9d7915f700 10 _do_transaction on 0x610eb70
2014-06-12 14:46:45.446936 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.446939 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.446942 7f9d7915f700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.447049 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.447065 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.447103 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.447115 7f9d7915f700 15 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.447120 7f9d7915f700 10 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1 r = 0
2014-06-12 14:46:45.447124 7f9d7915f700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.447133 7f9d7915f700 15 _write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 1617920~516096
2014-06-12 14:46:45.447231 7f9d7915f700 10 lookup_strip_header done  cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.447245 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:45.448566 7f9d7915f700 10 _generic_write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 1617920~516096 = 0
2014-06-12 14:46:45.448599 7f9d7915f700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.448608 7f9d7915f700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.448615 7f9d7915f700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.448620 7f9d7915f700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.448762 7f9d7915f700 20 set_header setting 17924 cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 parent seq 0
2014-06-12 14:46:45.448801 7f9d7915f700 20 set_header setting 2264 cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1 parent seq 0
2014-06-12 14:46:45.448812 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.448819 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:45.450448 7f9d7915f700 10 _do_op 0x24c670e0 seq 2605 r = 0, finisher 0x28e9240 0
2014-06-12 14:46:45.450466 7f9d7915f700 10 _finish_op 0x24c670e0 seq 2605 osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.450471 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439d40 (0 active)
2014-06-12 14:46:45.450474 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.450503 7f9d7795c700  5 -- op tracker -- , seq: 748042, time: 2014-06-12 14:46:45.450492, event: sub_op_applied, op: osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.450515 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2851 (0'0,173'2851] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2849 active] sub_op_modify_applied on 0x610ea80 op osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.450541 7f9d7795c700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.450561 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2851 (0'0,173'2851] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2849 active] op_applied on primary on version 173'2851
2014-06-12 14:46:45.450570 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2851 (0'0,173'2851] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2849 active] op_applied on replica on version 173'2851
2014-06-12 14:46:45.450589 7f9d7715b700  5 -- op tracker -- , seq: 748042, time: 2014-06-12 14:46:45.450583, event: commit_sent, op: osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.450598 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2851 (0'0,173'2851] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2849 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:45.450611 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.450629 7f9d7715b700  5 -- op tracker -- , seq: 748042, time: 2014-06-12 14:46:45.450626, event: done, op: osd_sub_op(client.4314.0:145133 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.456154 7f9d58d5b700  5 -- op tracker -- , seq: 748043, time: 2014-06-12 14:46:45.455394, event: header_read, op: osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.456167 7f9d58d5b700  5 -- op tracker -- , seq: 748043, time: 2014-06-12 14:46:45.455396, event: throttled, op: osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.456177 7f9d58d5b700  5 -- op tracker -- , seq: 748043, time: 2014-06-12 14:46:45.456105, event: all_read, op: osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.456185 7f9d58d5b700  5 -- op tracker -- , seq: 748043, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.456192 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.456199 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:45.456210 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34017600 prio 127 cost 516903 latency 0.000815 osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.456251 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.456259 7f9d673df700 10 osd.2 173 dequeue_op 0x34017600 prio 127 cost 516903 latency 0.000865 osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.5c5( v 173'2851 (0'0,173'2851] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2850 active]
2014-06-12 14:46:45.456284 7f9d673df700  5 -- op tracker -- , seq: 748043, time: 2014-06-12 14:46:45.456280, event: reached_pg, op: osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.456295 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2851 (0'0,173'2851] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2850 active] handle_message: 0x34017600
2014-06-12 14:46:45.456308 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2851 (0'0,173'2851] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2850 active] sub_op_modify trans f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 v 173'2852 (transaction) 174
2014-06-12 14:46:45.456326 7f9d673df700  5 -- op tracker -- , seq: 748043, time: 2014-06-12 14:46:45.456321, event: started, op: osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.456344 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2851 (0'0,173'2851] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2850 active] append_log log((0'0,173'2851], crt=0'0) [173'2852 (173'2851) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145134 2014-06-12 08:45:47.832913]
2014-06-12 14:46:45.456369 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2852 (0'0,173'2852] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2850 active] add_log_entry 173'2852 (173'2851) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145134 2014-06-12 08:45:47.832913
2014-06-12 14:46:45.456392 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2852 (0'0,173'2852] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2850 active] append_log  adding 1 keys
2014-06-12 14:46:45.456427 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2852, trimmed: 
2014-06-12 14:46:45.456453 7f9d673df700  5 -- op tracker -- , seq: 748043, time: 2014-06-12 14:46:45.456448, event: started, op: osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.456468 7f9d673df700  5 queue_transactions existing osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.456473 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.456474 7f9d673df700  5 queue_op 0x24390e0 seq 2606 osr(3.5c5 0x276dde0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:45.456485 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439d40 (1 active)
2014-06-12 14:46:45.456492 7f9d7d968700  5 _do_op 0x24390e0 seq 2606 osr(3.5c5 0x276dde0)/0x276dde0 start
2014-06-12 14:46:45.456482 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2852 (0'0,173'2852] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2850 active] do_sub_op osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.456496 7f9d673df700 10 osd.2 173 dequeue_op 0x34017600 finish
2014-06-12 14:46:45.456497 7f9d7d968700 10 _do_transaction on 0x6175eb0
2014-06-12 14:46:45.456499 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.456502 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.456502 7f9d7d968700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.456629 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.456647 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.456700 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.456715 7f9d7d968700 15 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.456720 7f9d7d968700 10 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1 r = 0
2014-06-12 14:46:45.456725 7f9d7d968700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.456736 7f9d7d968700 15 _write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 2134016~516096
2014-06-12 14:46:45.456891 7f9d7d968700 10 lookup_strip_header done  cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.456912 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:45.459552 7f9d7d968700 10 _generic_write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 2134016~516096 = 0
2014-06-12 14:46:45.459606 7f9d7d968700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.459621 7f9d7d968700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.459627 7f9d7d968700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.459634 7f9d7d968700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.459836 7f9d7d968700 20 set_header setting 17924 cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 parent seq 0
2014-06-12 14:46:45.459908 7f9d7d968700 20 set_header setting 2264 cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1 parent seq 0
2014-06-12 14:46:45.459926 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.459937 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:45.463157 7f9d7d968700 10 _do_op 0x24390e0 seq 2606 r = 0, finisher 0x10027ec0 0
2014-06-12 14:46:45.463192 7f9d7d968700 10 _finish_op 0x24390e0 seq 2606 osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.463207 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439d40 (0 active)
2014-06-12 14:46:45.463217 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.463815 7f9d7715b700  5 -- op tracker -- , seq: 748043, time: 2014-06-12 14:46:45.463805, event: commit_sent, op: osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.463828 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2852 (0'0,173'2852] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2850 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:45.463856 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.464226 7f9d7795c700  5 -- op tracker -- , seq: 748043, time: 2014-06-12 14:46:45.464219, event: sub_op_applied, op: osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.464237 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2852 (0'0,173'2852] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2851 active] sub_op_modify_applied on 0x6175dc0 op osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.464254 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2852 (0'0,173'2852] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2851 active] op_applied on primary on version 173'2852
2014-06-12 14:46:45.464264 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2852 (0'0,173'2852] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2851 active] op_applied on replica on version 173'2852
2014-06-12 14:46:45.464286 7f9d7795c700  5 -- op tracker -- , seq: 748043, time: 2014-06-12 14:46:45.464281, event: done, op: osd_sub_op(client.4314.0:145134 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.475178 7f9d58d5b700  5 -- op tracker -- , seq: 748044, time: 2014-06-12 14:46:45.475084, event: header_read, op: osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.475192 7f9d58d5b700  5 -- op tracker -- , seq: 748044, time: 2014-06-12 14:46:45.475086, event: throttled, op: osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.475202 7f9d58d5b700  5 -- op tracker -- , seq: 748044, time: 2014-06-12 14:46:45.475129, event: all_read, op: osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.475211 7f9d58d5b700  5 -- op tracker -- , seq: 748044, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.475218 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.475226 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:45.475236 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34017f00 prio 127 cost 4903 latency 0.000152 osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.475270 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.475278 7f9d663dd700 10 osd.2 173 dequeue_op 0x34017f00 prio 127 cost 4903 latency 0.000193 osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.5c5( v 173'2852 (0'0,173'2852] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2851 active]
2014-06-12 14:46:45.475303 7f9d663dd700  5 -- op tracker -- , seq: 748044, time: 2014-06-12 14:46:45.475299, event: reached_pg, op: osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.475313 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2852 (0'0,173'2852] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2851 active] handle_message: 0x34017f00
2014-06-12 14:46:45.475327 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2852 (0'0,173'2852] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2851 active] sub_op_modify trans f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 v 173'2853 (transaction) 174
2014-06-12 14:46:45.475347 7f9d663dd700  5 -- op tracker -- , seq: 748044, time: 2014-06-12 14:46:45.475343, event: started, op: osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.475366 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2852 (0'0,173'2852] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2851 active] append_log log((0'0,173'2852], crt=0'0) [173'2853 (173'2852) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145135 2014-06-12 08:45:47.850896]
2014-06-12 14:46:45.475388 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2853 (0'0,173'2853] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2851 active] add_log_entry 173'2853 (173'2852) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145135 2014-06-12 08:45:47.850896
2014-06-12 14:46:45.475404 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2853 (0'0,173'2853] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2851 active] append_log  adding 1 keys
2014-06-12 14:46:45.475430 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2853, trimmed: 
2014-06-12 14:46:45.475453 7f9d663dd700  5 -- op tracker -- , seq: 748044, time: 2014-06-12 14:46:45.475450, event: started, op: osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.475460 7f9d663dd700  5 queue_transactions existing osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.475463 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.475464 7f9d663dd700  5 queue_op 0xe834f60 seq 2607 osr(3.5c5 0x276dde0) 6389 bytes   (queue has 1 ops and 6389 bytes)
2014-06-12 14:46:45.475474 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439d40 (1 active)
2014-06-12 14:46:45.475481 7f9d8096e700  5 _do_op 0xe834f60 seq 2607 osr(3.5c5 0x276dde0)/0x276dde0 start
2014-06-12 14:46:45.475471 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2853 (0'0,173'2853] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2851 active] do_sub_op osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.475484 7f9d663dd700 10 osd.2 173 dequeue_op 0x34017f00 finish
2014-06-12 14:46:45.475485 7f9d8096e700 10 _do_transaction on 0x61ced30
2014-06-12 14:46:45.475487 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.475489 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.475491 7f9d8096e700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.475660 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.475692 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.475757 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.475785 7f9d8096e700 15 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.475794 7f9d8096e700 10 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1 r = 0
2014-06-12 14:46:45.475800 7f9d8096e700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.475930 7f9d8096e700 15 _write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 2650112~4096
2014-06-12 14:46:45.476081 7f9d8096e700 10 lookup_strip_header done  cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.476091 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:45.476111 7f9d8096e700 10 _generic_write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 2650112~4096 = 0
2014-06-12 14:46:45.476120 7f9d8096e700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.476133 7f9d8096e700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.476139 7f9d8096e700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.476148 7f9d8096e700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.476240 7f9d8096e700 20 set_header setting 17924 cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 parent seq 0
2014-06-12 14:46:45.476261 7f9d8096e700 20 set_header setting 2264 cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1 parent seq 0
2014-06-12 14:46:45.476278 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.476289 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:45.476421 7f9d8096e700 10 _do_op 0xe834f60 seq 2607 r = 0, finisher 0x98eef60 0
2014-06-12 14:46:45.476431 7f9d8096e700 10 _finish_op 0xe834f60 seq 2607 osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.476438 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439d40 (0 active)
2014-06-12 14:46:45.476441 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.476448 7f9d7715b700  5 -- op tracker -- , seq: 748044, time: 2014-06-12 14:46:45.476439, event: commit_sent, op: osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.476458 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2853 (0'0,173'2853] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2851 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:45.476483 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.476517 7f9d7795c700  5 -- op tracker -- , seq: 748044, time: 2014-06-12 14:46:45.476511, event: sub_op_applied, op: osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.476526 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2853 (0'0,173'2853] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active] sub_op_modify_applied on 0x61cec40 op osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.476542 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2853 (0'0,173'2853] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active] op_applied on primary on version 173'2853
2014-06-12 14:46:45.476553 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2853 (0'0,173'2853] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active] op_applied on replica on version 173'2853
2014-06-12 14:46:45.476588 7f9d7795c700  5 -- op tracker -- , seq: 748044, time: 2014-06-12 14:46:45.476583, event: done, op: osd_sub_op(client.4314.0:145135 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2853 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.478564 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:45.478591 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:45.478626 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 11.57 >= max 0.5 = no, load too high
2014-06-12 14:46:45.478638 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:45.478640 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92653 < max (604800 seconds)
2014-06-12 14:46:45.478649 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:45.478651 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:45.478652 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:45.478655 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 18; oldest is 12.317310 seconds old
2014-06-12 14:46:45.485507 7f9d5501e700  5 -- op tracker -- , seq: 748045, time: 2014-06-12 14:46:45.484716, event: header_read, op: osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.485523 7f9d5501e700  5 -- op tracker -- , seq: 748045, time: 2014-06-12 14:46:45.484719, event: throttled, op: osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.485533 7f9d5501e700  5 -- op tracker -- , seq: 748045, time: 2014-06-12 14:46:45.485462, event: all_read, op: osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.485542 7f9d5501e700  5 -- op tracker -- , seq: 748045, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.485548 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.485559 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:45.485569 7f9d5501e700 15 osd.2 173 enqueue_op 0x28dd1f00 prio 127 cost 299815 latency 0.000852 osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.485699 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.485715 7f9d67be0700 10 osd.2 173 dequeue_op 0x28dd1f00 prio 127 cost 299815 latency 0.000998 osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2781 (0'0,173'2781] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2780 active]
2014-06-12 14:46:45.485744 7f9d67be0700  5 -- op tracker -- , seq: 748045, time: 2014-06-12 14:46:45.485740, event: reached_pg, op: osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.485756 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2781 (0'0,173'2781] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2780 active] handle_message: 0x28dd1f00
2014-06-12 14:46:45.485770 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2781 (0'0,173'2781] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2780 active] sub_op_modify trans 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 v 173'2782 (transaction) 174
2014-06-12 14:46:45.485790 7f9d67be0700  5 -- op tracker -- , seq: 748045, time: 2014-06-12 14:46:45.485786, event: started, op: osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.485810 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2781 (0'0,173'2781] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2780 active] append_log log((0'0,173'2781], crt=0'0) [173'2782 (173'681) modify   1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 by client.4305.0:142477 2014-06-12 08:45:47.858983]
2014-06-12 14:46:45.485839 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2782 (0'0,173'2782] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2780 active] add_log_entry 173'2782 (173'681) modify   1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 by client.4305.0:142477 2014-06-12 08:45:47.858983
2014-06-12 14:46:45.485863 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2782 (0'0,173'2782] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2780 active] append_log  adding 1 keys
2014-06-12 14:46:45.485899 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2782, trimmed: 
2014-06-12 14:46:45.485930 7f9d67be0700  5 -- op tracker -- , seq: 748045, time: 2014-06-12 14:46:45.485925, event: started, op: osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.485940 7f9d67be0700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:45.485945 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.485947 7f9d67be0700  5 queue_op 0x614c780 seq 2519 osr(3.6d1 0x276df10) 301301 bytes   (queue has 1 ops and 301301 bytes)
2014-06-12 14:46:45.485955 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2782 (0'0,173'2782] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2780 active] do_sub_op osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.485969 7f9d67be0700 10 osd.2 173 dequeue_op 0x28dd1f00 finish
2014-06-12 14:46:45.485973 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.485975 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.485984 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (1 active)
2014-06-12 14:46:45.485990 7f9d7c165700  5 _do_op 0x614c780 seq 2519 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:45.485994 7f9d7c165700 10 _do_transaction on 0x328a7270
2014-06-12 14:46:45.486000 7f9d7c165700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:45.486140 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:45.486168 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.486222 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.486334 7f9d7c165700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:45.486341 7f9d7c165700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:45.486347 7f9d7c165700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:45.486364 7f9d7c165700 15 _write 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 0~299008
2014-06-12 14:46:45.486605 7f9d7c165700 10 lookup_strip_header done  cid 3.6d1_head oid 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:45.486791 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:45.488144 7f9d58d5b700  5 -- op tracker -- , seq: 748046, time: 2014-06-12 14:46:45.487201, event: header_read, op: osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488156 7f9d58d5b700  5 -- op tracker -- , seq: 748046, time: 2014-06-12 14:46:45.487205, event: throttled, op: osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488165 7f9d58d5b700  5 -- op tracker -- , seq: 748046, time: 2014-06-12 14:46:45.488109, event: all_read, op: osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488172 7f9d58d5b700  5 -- op tracker -- , seq: 748046, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488177 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.488184 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:45.488193 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016200 prio 127 cost 516903 latency 0.000991 osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.488222 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.488228 7f9d65bdc700 10 osd.2 173 dequeue_op 0x34016200 prio 127 cost 516903 latency 0.001027 osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.5c5( v 173'2853 (0'0,173'2853] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active]
2014-06-12 14:46:45.488249 7f9d65bdc700  5 -- op tracker -- , seq: 748046, time: 2014-06-12 14:46:45.488245, event: reached_pg, op: osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488257 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2853 (0'0,173'2853] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active] handle_message: 0x34016200
2014-06-12 14:46:45.488268 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2853 (0'0,173'2853] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active] sub_op_modify trans f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 v 173'2854 (transaction) 174
2014-06-12 14:46:45.488289 7f9d65bdc700  5 -- op tracker -- , seq: 748046, time: 2014-06-12 14:46:45.488285, event: started, op: osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488307 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2853 (0'0,173'2853] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active] append_log log((0'0,173'2853], crt=0'0) [173'2854 (173'2853) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145136 2014-06-12 08:45:47.859858]
2014-06-12 14:46:45.488320 7f9d7c165700 10 _generic_write 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 0~299008 = 0
2014-06-12 14:46:45.488328 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2854 (0'0,173'2854] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active] add_log_entry 173'2854 (173'2853) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145136 2014-06-12 08:45:47.859858
2014-06-12 14:46:45.488348 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2854 (0'0,173'2854] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active] append_log  adding 1 keys
2014-06-12 14:46:45.488358 7f9d7c165700 15 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:45.488370 7f9d7c165700 10 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 = 0
2014-06-12 14:46:45.488376 7f9d7c165700 15 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:45.488376 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2854, trimmed: 
2014-06-12 14:46:45.488384 7f9d7c165700 10 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 = 0
2014-06-12 14:46:45.488396 7f9d65bdc700  5 -- op tracker -- , seq: 748046, time: 2014-06-12 14:46:45.488392, event: started, op: osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488405 7f9d65bdc700  5 queue_transactions existing osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.488408 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.488409 7f9d65bdc700  5 queue_op 0x88d3d40 seq 2608 osr(3.5c5 0x276dde0) 518389 bytes   (queue has 2 ops and 819690 bytes)
2014-06-12 14:46:45.488420 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439d40 (2 active)
2014-06-12 14:46:45.488417 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2854 (0'0,173'2854] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active] do_sub_op osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.488429 7f9d81970700  5 _do_op 0x88d3d40 seq 2608 osr(3.5c5 0x276dde0)/0x276dde0 start
2014-06-12 14:46:45.488431 7f9d65bdc700 10 osd.2 173 dequeue_op 0x34016200 finish
2014-06-12 14:46:45.488434 7f9d81970700 10 _do_transaction on 0x62215f0
2014-06-12 14:46:45.488435 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.488437 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.488440 7f9d81970700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.488556 7f9d7c165700 20 set_header setting 16429 cid 3.6d1_head oid 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 parent seq 0
2014-06-12 14:46:45.488663 7f9d7c165700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:45.488684 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.488694 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:45.488699 7f9d44101700  5 -- op tracker -- , seq: 748047, time: 2014-06-12 14:46:45.488278, event: header_read, op: osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488716 7f9d44101700  5 -- op tracker -- , seq: 748047, time: 2014-06-12 14:46:45.488280, event: throttled, op: osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488726 7f9d44101700  5 -- op tracker -- , seq: 748047, time: 2014-06-12 14:46:45.488664, event: all_read, op: osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488735 7f9d44101700  5 -- op tracker -- , seq: 748047, time: 0.000000, event: dispatched, op: osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488739 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.488745 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:45.488752 7f9d44101700 15 osd.2 173 enqueue_op 0x25e43f00 prio 127 cost 103203 latency 0.000474 osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.488781 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.488790 7f9d62bd6700 10 osd.2 173 dequeue_op 0x25e43f00 prio 127 cost 103203 latency 0.000511 osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.46( v 173'3098 (79'97,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active]
2014-06-12 14:46:45.488814 7f9d62bd6700  5 -- op tracker -- , seq: 748047, time: 2014-06-12 14:46:45.488810, event: reached_pg, op: osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488824 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'97,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active] handle_message: 0x25e43f00
2014-06-12 14:46:45.488836 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'97,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active] sub_op_modify trans 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 v 173'3099 (transaction) 174
2014-06-12 14:46:45.488854 7f9d62bd6700  5 -- op tracker -- , seq: 748047, time: 2014-06-12 14:46:45.488849, event: started, op: osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488868 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3098 (79'97,173'3098] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active] append_log log((79'97,173'3098], crt=0'0) [173'3099 (173'822) modify   26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 by client.4282.0:145396 2014-06-12 08:45:47.850503]
2014-06-12 14:46:45.488885 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.488892 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3099 (79'97,173'3099] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active] add_log_entry 173'3099 (173'822) modify   26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 by client.4282.0:145396 2014-06-12 08:45:47.850503
2014-06-12 14:46:45.488914 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.488915 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3099 (79'97,173'3099] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active] append_log  adding 1 keys
2014-06-12 14:46:45.488930 7f9d62bd6700 10 trim log((79'97,173'3099], crt=0'0) to 79'98
2014-06-12 14:46:45.488954 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3099, trimmed: 79'98
2014-06-12 14:46:45.488979 7f9d62bd6700  5 -- op tracker -- , seq: 748047, time: 2014-06-12 14:46:45.488973, event: started, op: osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.488988 7f9d62bd6700  5 queue_transactions existing osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:45.488993 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.488995 7f9d62bd6700  5 queue_op 0x53e2ba0 seq 2872 osr(3.46 0x276cc90) 104719 bytes   (queue has 3 ops and 924409 bytes)
2014-06-12 14:46:45.489003 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.46( v 173'3099 (79'98,173'3099] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active] do_sub_op osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.489019 7f9d62bd6700 10 osd.2 173 dequeue_op 0x25e43f00 finish
2014-06-12 14:46:45.489023 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.489026 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.489035 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e97200 (3 active)
2014-06-12 14:46:45.489039 7f9d7c966700  5 _do_op 0x53e2ba0 seq 2872 osr(3.46 0x276cc90)/0x276cc90 start
2014-06-12 14:46:45.489042 7f9d7c966700 10 _do_transaction on 0x1cc28b70
2014-06-12 14:46:45.489047 7f9d7c966700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.489129 7f9d7c966700 10 lookup_strip_header done  cid meta oid 51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.489149 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.490182 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.490214 7f9d81970700 15 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.490221 7f9d81970700 10 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1 r = 0
2014-06-12 14:46:45.490228 7f9d81970700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.490337 7f9d7c165700 10 _do_op 0x614c780 seq 2519 r = 0, finisher 0x69074c0 0
2014-06-12 14:46:45.490342 7f9d81970700 15 _write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 2654208~516096
2014-06-12 14:46:45.490350 7f9d7c165700 10 _finish_op 0x614c780 seq 2519 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:45.490357 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (2 active)
2014-06-12 14:46:45.490365 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.490386 7f9d7715b700  5 -- op tracker -- , seq: 748045, time: 2014-06-12 14:46:45.490377, event: commit_sent, op: osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.490397 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2782 (0'0,173'2782] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2780 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:45.490423 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:45.490485 7f9d7795c700  5 -- op tracker -- , seq: 748045, time: 2014-06-12 14:46:45.490480, event: sub_op_applied, op: osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.490492 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2782 (0'0,173'2782] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2781 active] sub_op_modify_applied on 0x328a7180 op osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.490498 7f9d81970700 10 lookup_strip_header done  cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.490506 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2782 (0'0,173'2782] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2781 active] op_applied on primary on version 173'2782
2014-06-12 14:46:45.490519 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2782 (0'0,173'2782] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2781 active] op_applied on replica on version 173'2782
2014-06-12 14:46:45.490549 7f9d7795c700  5 -- op tracker -- , seq: 748045, time: 2014-06-12 14:46:45.490545, event: done, op: osd_sub_op(client.4305.0:142477 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2782 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.490658 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:45.492919 7f9d81970700 10 _generic_write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 2654208~516096 = 0
2014-06-12 14:46:45.493032 7f9d81970700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.493067 7f9d81970700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.493076 7f9d81970700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.493088 7f9d81970700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.493339 7f9d81970700 20 set_header setting 17924 cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 parent seq 0
2014-06-12 14:46:45.493456 7f9d81970700 20 set_header setting 2264 cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1 parent seq 0
2014-06-12 14:46:45.493475 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.493485 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:45.496716 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.496787 7f9d7c966700 15 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.496802 7f9d7c966700 10 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1 r = 0
2014-06-12 14:46:45.496815 7f9d7c966700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.496843 7f9d7c966700 15 _write 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 0~102400
2014-06-12 14:46:45.497318 7f9d7c966700 10 lookup_strip_header done  cid 3.46_head oid 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3
2014-06-12 14:46:45.497347 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:45.497961 7f9d7c966700 10 _generic_write 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 0~102400 = 0
2014-06-12 14:46:45.498014 7f9d7c966700 15 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3
2014-06-12 14:46:45.498051 7f9d7c966700 10 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 = 0
2014-06-12 14:46:45.498060 7f9d7c966700 15 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3
2014-06-12 14:46:45.498071 7f9d7c966700 10 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 = 0
2014-06-12 14:46:45.498223 7f9d7c966700 20 set_header setting 18271 cid 3.46_head oid 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 parent seq 0
2014-06-12 14:46:45.498269 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.498284 7f9d7c966700 20 set_header setting 2172 cid meta oid 51681c0b/pglog_3.46/0//-1 parent seq 0
2014-06-12 14:46:45.498296 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:45.498927 7f9d7c966700 10 _do_op 0x53e2ba0 seq 2872 r = 0, finisher 0x172fe2c0 0
2014-06-12 14:46:45.498940 7f9d7c966700 10 _finish_op 0x53e2ba0 seq 2872 osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:45.498949 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e97200 (1 active)
2014-06-12 14:46:45.498952 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.498975 7f9d7795c700  5 -- op tracker -- , seq: 748047, time: 2014-06-12 14:46:45.498966, event: sub_op_applied, op: osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.498987 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3099 (79'98,173'3099] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active] sub_op_modify_applied on 0x1cc28a80 op osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.499016 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:45.499042 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3099 (79'98,173'3099] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active] op_applied on primary on version 173'3099
2014-06-12 14:46:45.499054 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3099 (79'98,173'3099] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active] op_applied on replica on version 173'3099
2014-06-12 14:46:45.499587 7f9d81970700 10 _do_op 0x88d3d40 seq 2608 r = 0, finisher 0x5db7900 0
2014-06-12 14:46:45.499600 7f9d81970700 10 _finish_op 0x88d3d40 seq 2608 osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.499647 7f9d7795c700  5 -- op tracker -- , seq: 748046, time: 2014-06-12 14:46:45.499639, event: sub_op_applied, op: osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.499655 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2854 (0'0,173'2854] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active] sub_op_modify_applied on 0x6221500 op osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.499686 7f9d7795c700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.499705 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2854 (0'0,173'2854] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active] op_applied on primary on version 173'2854
2014-06-12 14:46:45.499717 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2854 (0'0,173'2854] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active] op_applied on replica on version 173'2854
2014-06-12 14:46:45.499783 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439d40 (0 active)
2014-06-12 14:46:45.499789 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.501873 7f9d7715b700  5 -- op tracker -- , seq: 748047, time: 2014-06-12 14:46:45.501844, event: commit_sent, op: osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.501909 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3099 (79'98,173'3099] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3097 active] sub_op_modify_commit on op osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:46:45.501954 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:45.502016 7f9d7715b700  5 -- op tracker -- , seq: 748047, time: 2014-06-12 14:46:45.502011, event: done, op: osd_sub_op(client.4282.0:145396 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3099 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.502114 7f9d7715b700  5 -- op tracker -- , seq: 748046, time: 2014-06-12 14:46:45.502108, event: commit_sent, op: osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.502121 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2854 (0'0,173'2854] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2852 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:45.502144 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.502172 7f9d7715b700  5 -- op tracker -- , seq: 748046, time: 2014-06-12 14:46:45.502168, event: done, op: osd_sub_op(client.4314.0:145136 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2854 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.516809 7f9d44101700  5 -- op tracker -- , seq: 748048, time: 2014-06-12 14:46:45.497659, event: header_read, op: osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.516825 7f9d44101700  5 -- op tracker -- , seq: 748048, time: 2014-06-12 14:46:45.497662, event: throttled, op: osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.516834 7f9d44101700  5 -- op tracker -- , seq: 748048, time: 2014-06-12 14:46:45.516766, event: all_read, op: osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.516842 7f9d44101700  5 -- op tracker -- , seq: 748048, time: 0.000000, event: dispatched, op: osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.516847 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.516854 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:45.516863 7f9d44101700 15 osd.2 173 enqueue_op 0x25e42a00 prio 127 cost 1549091 latency 0.019204 osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.516891 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.516899 7f9d683e1700 10 osd.2 173 dequeue_op 0x25e42a00 prio 127 cost 1549091 latency 0.019240 osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.46( v 173'3099 (79'98,173'3099] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3098 active]
2014-06-12 14:46:45.516921 7f9d683e1700  5 -- op tracker -- , seq: 748048, time: 2014-06-12 14:46:45.516917, event: reached_pg, op: osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.516931 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3099 (79'98,173'3099] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3098 active] handle_message: 0x25e42a00
2014-06-12 14:46:45.516942 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3099 (79'98,173'3099] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3098 active] sub_op_modify trans 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 v 173'3100 (transaction) 174
2014-06-12 14:46:45.516958 7f9d683e1700  5 -- op tracker -- , seq: 748048, time: 2014-06-12 14:46:45.516954, event: started, op: osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.516972 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3099 (79'98,173'3099] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3098 active] append_log log((79'98,173'3099], crt=0'0) [173'3100 (173'3099) modify   26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 by client.4282.0:145397 2014-06-12 08:45:47.854858]
2014-06-12 14:46:45.516997 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3100 (79'98,173'3100] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3098 active] add_log_entry 173'3100 (173'3099) modify   26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 by client.4282.0:145397 2014-06-12 08:45:47.854858
2014-06-12 14:46:45.517025 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3100 (79'98,173'3100] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3098 active] append_log  adding 1 keys
2014-06-12 14:46:45.517039 7f9d683e1700 10 trim log((79'98,173'3100], crt=0'0) to 79'99
2014-06-12 14:46:45.517071 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3100, trimmed: 79'99
2014-06-12 14:46:45.517102 7f9d683e1700  5 -- op tracker -- , seq: 748048, time: 2014-06-12 14:46:45.517098, event: started, op: osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.517114 7f9d683e1700  5 queue_transactions existing osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:45.517118 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.517120 7f9d683e1700  5 queue_op 0x8344b40 seq 2873 osr(3.46 0x276cc90) 1550607 bytes   (queue has 1 ops and 1550607 bytes)
2014-06-12 14:46:45.517128 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.46( v 173'3100 (79'99,173'3100] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3098 active] do_sub_op osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.517143 7f9d683e1700 10 osd.2 173 dequeue_op 0x25e42a00 finish
2014-06-12 14:46:45.517146 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.517148 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.517154 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e97200 (1 active)
2014-06-12 14:46:45.517159 7f9d7a962700  5 _do_op 0x8344b40 seq 2873 osr(3.46 0x276cc90)/0x276cc90 start
2014-06-12 14:46:45.517164 7f9d7a962700 10 _do_transaction on 0x5a960f0
2014-06-12 14:46:45.517169 7f9d7a962700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.517325 7f9d7a962700 10 lookup_strip_header done  cid meta oid 51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.517352 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.517416 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.517560 7f9d7a962700 15 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.517570 7f9d7a962700 10 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1 r = 0
2014-06-12 14:46:45.517576 7f9d7a962700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.517593 7f9d7a962700 15 _write 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 102400~1548288
2014-06-12 14:46:45.517780 7f9d7a962700 10 lookup_strip_header done  cid 3.46_head oid 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3
2014-06-12 14:46:45.517960 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:45.528093 7f9d7a962700 10 _generic_write 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 102400~1548288 = 0
2014-06-12 14:46:45.528339 7f9d7a962700 15 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3
2014-06-12 14:46:45.528361 7f9d7a962700 10 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 = 0
2014-06-12 14:46:45.528367 7f9d7a962700 15 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3
2014-06-12 14:46:45.528379 7f9d7a962700 10 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 = 0
2014-06-12 14:46:45.528918 7f9d7a962700 20 set_header setting 18271 cid 3.46_head oid 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 parent seq 0
2014-06-12 14:46:45.529105 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.529128 7f9d7a962700 20 set_header setting 2172 cid meta oid 51681c0b/pglog_3.46/0//-1 parent seq 0
2014-06-12 14:46:45.529137 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:45.530962 7f9d5a16f700  5 -- op tracker -- , seq: 748049, time: 2014-06-12 14:46:45.522115, event: header_read, op: osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.530975 7f9d5a16f700  5 -- op tracker -- , seq: 748049, time: 2014-06-12 14:46:45.522118, event: throttled, op: osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.530983 7f9d5a16f700  5 -- op tracker -- , seq: 748049, time: 2014-06-12 14:46:45.530915, event: all_read, op: osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.530992 7f9d5a16f700  5 -- op tracker -- , seq: 748049, time: 0.000000, event: dispatched, op: osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.530999 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.531006 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:45.531015 7f9d5a16f700 15 osd.2 173 enqueue_op 0x31d60d00 prio 127 cost 647975 latency 0.008899 osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.532092 7f9d58d5b700  5 -- op tracker -- , seq: 748050, time: 2014-06-12 14:46:45.522956, event: header_read, op: osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.532104 7f9d58d5b700  5 -- op tracker -- , seq: 748050, time: 2014-06-12 14:46:45.522957, event: throttled, op: osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.532112 7f9d58d5b700  5 -- op tracker -- , seq: 748050, time: 2014-06-12 14:46:45.532052, event: all_read, op: osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.532119 7f9d58d5b700  5 -- op tracker -- , seq: 748050, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.532125 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.532132 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:45.532140 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016c00 prio 127 cost 516903 latency 0.009183 osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.532264 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.532272 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x31d60d00 prio 127 cost 647975 latency 0.010156 osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4e0( v 173'2638 (0'0,173'2638] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2637 active]
2014-06-12 14:46:45.532297 7f9d6b3e7700  5 -- op tracker -- , seq: 748049, time: 2014-06-12 14:46:45.532292, event: reached_pg, op: osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.532307 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2638 (0'0,173'2638] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2637 active] handle_message: 0x31d60d00
2014-06-12 14:46:45.532319 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2638 (0'0,173'2638] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2637 active] sub_op_modify trans 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 v 173'2639 (transaction) 174
2014-06-12 14:46:45.532336 7f9d6b3e7700  5 -- op tracker -- , seq: 748049, time: 2014-06-12 14:46:45.532332, event: started, op: osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.532354 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2638 (0'0,173'2638] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2637 active] append_log log((0'0,173'2638], crt=0'0) [173'2639 (173'2638) modify   79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 by client.4293.0:143963 2014-06-12 08:43:57.139594]
2014-06-12 14:46:45.532402 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2639 (0'0,173'2639] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2637 active] add_log_entry 173'2639 (173'2638) modify   79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 by client.4293.0:143963 2014-06-12 08:43:57.139594
2014-06-12 14:46:45.532426 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2639 (0'0,173'2639] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2637 active] append_log  adding 1 keys
2014-06-12 14:46:45.532462 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2639, trimmed: 
2014-06-12 14:46:45.532489 7f9d6b3e7700  5 -- op tracker -- , seq: 748049, time: 2014-06-12 14:46:45.532484, event: started, op: osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.532509 7f9d6b3e7700  5 queue_transactions existing osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:45.532514 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.532516 7f9d6b3e7700  5 queue_op 0xa2da4e0 seq 2439 osr(3.4e0 0x276dbd0) 649461 bytes   (queue has 2 ops and 2200068 bytes)
2014-06-12 14:46:45.532524 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2639 (0'0,173'2639] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2637 active] do_sub_op osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.532538 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x31d60d00 finish
2014-06-12 14:46:45.532542 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.532555 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.532558 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x34016c00 prio 127 cost 516903 latency 0.009602 osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.5c5( v 173'2854 (0'0,173'2854] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2853 active]
2014-06-12 14:46:45.532579 7f9d6b3e7700  5 -- op tracker -- , seq: 748050, time: 2014-06-12 14:46:45.532575, event: reached_pg, op: osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.532587 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2854 (0'0,173'2854] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2853 active] handle_message: 0x34016c00
2014-06-12 14:46:45.532597 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2854 (0'0,173'2854] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2853 active] sub_op_modify trans f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 v 173'2855 (transaction) 174
2014-06-12 14:46:45.532614 7f9d6b3e7700  5 -- op tracker -- , seq: 748050, time: 2014-06-12 14:46:45.532610, event: started, op: osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.532625 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2854 (0'0,173'2854] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2853 active] append_log log((0'0,173'2854], crt=0'0) [173'2855 (173'2854) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145137 2014-06-12 08:45:47.891057]
2014-06-12 14:46:45.532647 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2855 (0'0,173'2855] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2853 active] add_log_entry 173'2855 (173'2854) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145137 2014-06-12 08:45:47.891057
2014-06-12 14:46:45.532666 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2855 (0'0,173'2855] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2853 active] append_log  adding 1 keys
2014-06-12 14:46:45.532696 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2855, trimmed: 
2014-06-12 14:46:45.532720 7f9d6b3e7700  5 -- op tracker -- , seq: 748050, time: 2014-06-12 14:46:45.532715, event: started, op: osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.532727 7f9d6b3e7700  5 queue_transactions existing osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.532730 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.532732 7f9d6b3e7700  5 queue_op 0x91a2a80 seq 2609 osr(3.5c5 0x276dde0) 518389 bytes   (queue has 3 ops and 2718457 bytes)
2014-06-12 14:46:45.532739 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2855 (0'0,173'2855] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2853 active] do_sub_op osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.532753 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x34016c00 finish
2014-06-12 14:46:45.532757 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.532759 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.533308 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.533571 7f9d5a16f700  5 -- op tracker -- , seq: 748051, time: 2014-06-12 14:46:45.531046, event: header_read, op: osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.533582 7f9d5a16f700  5 -- op tracker -- , seq: 748051, time: 2014-06-12 14:46:45.531047, event: throttled, op: osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.533589 7f9d5a16f700  5 -- op tracker -- , seq: 748051, time: 2014-06-12 14:46:45.533539, event: all_read, op: osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.533596 7f9d5a16f700  5 -- op tracker -- , seq: 748051, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.533603 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.533610 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:45.533634 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5180900 (2 active)
2014-06-12 14:46:45.533638 7f9d5a16f700 15 osd.2 173 enqueue_op 0x31d61700 prio 127 cost 1032999 latency 0.002592 osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.533645 7f9d83173700  5 _do_op 0xa2da4e0 seq 2439 osr(3.4e0 0x276dbd0)/0x276dbd0 start
2014-06-12 14:46:45.533653 7f9d83173700 10 _do_transaction on 0x662a630
2014-06-12 14:46:45.533662 7f9d83173700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:45.534846 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439d40 (3 active)
2014-06-12 14:46:45.534858 7f9d82972700  5 _do_op 0x91a2a80 seq 2609 osr(3.5c5 0x276dde0)/0x276dde0 start
2014-06-12 14:46:45.534864 7f9d82972700 10 _do_transaction on 0x5ab82b0
2014-06-12 14:46:45.534874 7f9d82972700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.535868 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.535883 7f9d66bde700 10 osd.2 173 dequeue_op 0x31d61700 prio 127 cost 1032999 latency 0.004836 osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2469 (0'0,173'2469] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2468 active]
2014-06-12 14:46:45.535909 7f9d66bde700  5 -- op tracker -- , seq: 748051, time: 2014-06-12 14:46:45.535904, event: reached_pg, op: osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.535933 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2469 (0'0,173'2469] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2468 active] handle_message: 0x31d61700
2014-06-12 14:46:45.535945 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2469 (0'0,173'2469] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2468 active] sub_op_modify trans 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 v 173'2470 (transaction) 174
2014-06-12 14:46:45.535960 7f9d66bde700  5 -- op tracker -- , seq: 748051, time: 2014-06-12 14:46:45.535956, event: started, op: osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.535976 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2469 (0'0,173'2469] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2468 active] append_log log((0'0,173'2469], crt=0'0) [173'2470 (173'2469) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143629 2014-06-12 08:43:18.502181]
2014-06-12 14:46:45.536009 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2470 (0'0,173'2470] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2468 active] add_log_entry 173'2470 (173'2469) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143629 2014-06-12 08:43:18.502181
2014-06-12 14:46:45.536037 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2470 (0'0,173'2470] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2468 active] append_log  adding 1 keys
2014-06-12 14:46:45.536068 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2470, trimmed: 
2014-06-12 14:46:45.536088 7f9d66bde700  5 -- op tracker -- , seq: 748051, time: 2014-06-12 14:46:45.536083, event: started, op: osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.536096 7f9d66bde700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:45.536100 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.536101 7f9d66bde700  5 queue_op 0x1b90efc0 seq 2303 osr(3.4f4 0x276db70) 1034485 bytes   (queue has 4 ops and 3752942 bytes)
2014-06-12 14:46:45.536109 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2470 (0'0,173'2470] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2468 active] do_sub_op osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.536124 7f9d66bde700 10 osd.2 173 dequeue_op 0x31d61700 finish
2014-06-12 14:46:45.536127 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.536130 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.536136 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (4 active)
2014-06-12 14:46:45.536141 7f9d7895e700  5 _do_op 0x1b90efc0 seq 2303 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:46:45.536145 7f9d7895e700 10 _do_transaction on 0x20c43cf0
2014-06-12 14:46:45.536150 7f9d7895e700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:45.539050 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.539087 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.539219 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:45.539237 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.539344 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:45.539375 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.541169 7f9d7a962700 10 _do_op 0x8344b40 seq 2873 r = 0, finisher 0x20959a0 0
2014-06-12 14:46:45.541196 7f9d7a962700 10 _finish_op 0x8344b40 seq 2873 osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:45.541205 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e97200 (3 active)
2014-06-12 14:46:45.541218 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.541222 7f9d7715b700  5 -- op tracker -- , seq: 748048, time: 2014-06-12 14:46:45.541206, event: commit_sent, op: osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.541252 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3100 (79'99,173'3100] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3098 active] sub_op_modify_commit on op osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:46:45.541300 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:45.541503 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.541543 7f9d82972700 15 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.541550 7f9d82972700 10 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1 r = 0
2014-06-12 14:46:45.541557 7f9d82972700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.541578 7f9d82972700 15 _write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 3170304~516096
2014-06-12 14:46:45.541933 7f9d7795c700  5 -- op tracker -- , seq: 748048, time: 2014-06-12 14:46:45.541922, event: sub_op_applied, op: osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.541949 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3100 (79'99,173'3100] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3099 active] sub_op_modify_applied on 0x5a96000 op osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.541970 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3100 (79'99,173'3100] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3099 active] op_applied on primary on version 173'3100
2014-06-12 14:46:45.541980 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3100 (79'99,173'3100] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3099 active] op_applied on replica on version 173'3100
2014-06-12 14:46:45.542016 7f9d7795c700  5 -- op tracker -- , seq: 748048, time: 2014-06-12 14:46:45.542011, event: done, op: osd_sub_op(client.4282.0:145397 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3100 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.542166 7f9d82972700 10 lookup_strip_header done  cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.542196 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:45.544716 7f9d82972700 10 _generic_write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 3170304~516096 = 0
2014-06-12 14:46:45.544777 7f9d82972700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.544799 7f9d82972700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.544805 7f9d82972700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.544812 7f9d82972700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.545014 7f9d82972700 20 set_header setting 17924 cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 parent seq 0
2014-06-12 14:46:45.545102 7f9d82972700 20 set_header setting 2264 cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1 parent seq 0
2014-06-12 14:46:45.545116 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.545125 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:45.548253 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.548293 7f9d7895e700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:45.548300 7f9d7895e700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:46:45.548307 7f9d7895e700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:45.548323 7f9d7895e700 15 _write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 2105344~1032192
2014-06-12 14:46:45.548499 7f9d7895e700 10 lookup_strip_header done  cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:45.548521 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:45.555941 7f9d7895e700 10 _generic_write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 2105344~1032192 = 0
2014-06-12 14:46:45.556021 7f9d7895e700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:45.556047 7f9d7895e700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:45.556053 7f9d7895e700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:45.556061 7f9d7895e700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:45.556353 7f9d7895e700 20 set_header setting 17388 cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 parent seq 0
2014-06-12 14:46:45.556462 7f9d7895e700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:46:45.556477 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.556486 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:45.557839 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:45.558149 7f9d82972700 10 _do_op 0x91a2a80 seq 2609 r = 0, finisher 0x82c75c0 0
2014-06-12 14:46:45.558163 7f9d82972700 10 _finish_op 0x91a2a80 seq 2609 osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.558189 7f9d7795c700  5 -- op tracker -- , seq: 748050, time: 2014-06-12 14:46:45.558181, event: sub_op_applied, op: osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.558197 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2855 (0'0,173'2855] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2853 active] sub_op_modify_applied on 0x5ab81c0 op osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.558221 7f9d7795c700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.558238 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2855 (0'0,173'2855] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2853 active] op_applied on primary on version 173'2855
2014-06-12 14:46:45.558250 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2855 (0'0,173'2855] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2853 active] op_applied on replica on version 173'2855
2014-06-12 14:46:45.558264 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439d40 (2 active)
2014-06-12 14:46:45.558278 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.561118 7f9d7715b700  5 -- op tracker -- , seq: 748050, time: 2014-06-12 14:46:45.561103, event: commit_sent, op: osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.561132 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2855 (0'0,173'2855] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2853 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:45.561161 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.561194 7f9d7715b700  5 -- op tracker -- , seq: 748050, time: 2014-06-12 14:46:45.561189, event: done, op: osd_sub_op(client.4314.0:145137 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2855 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.561674 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:45.562834 7f9d7895e700 10 _do_op 0x1b90efc0 seq 2303 r = 0, finisher 0xabf3180 0
2014-06-12 14:46:45.562860 7f9d7895e700 10 _finish_op 0x1b90efc0 seq 2303 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:45.562880 7f9d7715b700  5 -- op tracker -- , seq: 748051, time: 2014-06-12 14:46:45.562868, event: commit_sent, op: osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.562890 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2470 (0'0,173'2470] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2468 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:45.562918 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:45.563358 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.563394 7f9d83173700 15 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:45.563401 7f9d83173700 10 _omap_rmkeys meta/1fe07435/pglog_3.4e0/0//-1 r = 0
2014-06-12 14:46:45.563408 7f9d83173700 15 _omap_setkeys meta/1fe07435/pglog_3.4e0/0//-1
2014-06-12 14:46:45.563426 7f9d83173700 15 _write 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 3547136~647168
2014-06-12 14:46:45.563768 7f9d83173700 10 lookup_strip_header done  cid 3.4e0_head oid 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3
2014-06-12 14:46:45.563920 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:45.566209 7f9d7795c700  5 -- op tracker -- , seq: 748051, time: 2014-06-12 14:46:45.566197, event: sub_op_applied, op: osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.566221 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2470 (0'0,173'2470] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2469 active] sub_op_modify_applied on 0x20c43c00 op osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.566241 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2470 (0'0,173'2470] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2469 active] op_applied on primary on version 173'2470
2014-06-12 14:46:45.566251 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2470 (0'0,173'2470] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2469 active] op_applied on replica on version 173'2470
2014-06-12 14:46:45.566299 7f9d7895e700  5 -- op tracker -- , seq: 748051, time: 2014-06-12 14:46:45.566283, event: done, op: osd_sub_op(client.4294.0:143629 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2470 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.566327 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (1 active)
2014-06-12 14:46:45.566331 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.566440 7f9d83173700 10 _generic_write 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 3547136~647168 = 0
2014-06-12 14:46:45.566492 7f9d83173700 15 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3
2014-06-12 14:46:45.566507 7f9d83173700 10 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:45.566513 7f9d83173700 15 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3
2014-06-12 14:46:45.566520 7f9d83173700 10 _setattrs 3.4e0_head/79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 = 0
2014-06-12 14:46:45.566741 7f9d83173700 20 set_header setting 17854 cid 3.4e0_head oid 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 parent seq 0
2014-06-12 14:46:45.566827 7f9d83173700 20 set_header setting 2236 cid meta oid 1fe07435/pglog_3.4e0/0//-1 parent seq 0
2014-06-12 14:46:45.566842 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.566851 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:45.567288 7f9d44101700  5 -- op tracker -- , seq: 748052, time: 2014-06-12 14:46:45.557504, event: header_read, op: osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.567300 7f9d44101700  5 -- op tracker -- , seq: 748052, time: 2014-06-12 14:46:45.557506, event: throttled, op: osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.567309 7f9d44101700  5 -- op tracker -- , seq: 748052, time: 2014-06-12 14:46:45.567255, event: all_read, op: osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.567318 7f9d44101700  5 -- op tracker -- , seq: 748052, time: 0.000000, event: dispatched, op: osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.567323 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.567335 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:45.567344 7f9d44101700 15 osd.2 173 enqueue_op 0x25e42700 prio 127 cost 1549091 latency 0.009840 osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.567374 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.567382 7f9d64bda700 10 osd.2 173 dequeue_op 0x25e42700 prio 127 cost 1549091 latency 0.009878 osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.46( v 173'3100 (79'99,173'3100] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3099 active]
2014-06-12 14:46:45.567407 7f9d64bda700  5 -- op tracker -- , seq: 748052, time: 2014-06-12 14:46:45.567403, event: reached_pg, op: osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.567417 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3100 (79'99,173'3100] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3099 active] handle_message: 0x25e42700
2014-06-12 14:46:45.567429 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3100 (79'99,173'3100] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3099 active] sub_op_modify trans 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 v 173'3101 (transaction) 174
2014-06-12 14:46:45.567446 7f9d64bda700  5 -- op tracker -- , seq: 748052, time: 2014-06-12 14:46:45.567442, event: started, op: osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.567463 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3100 (79'99,173'3100] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3099 active] append_log log((79'99,173'3100], crt=0'0) [173'3101 (173'3100) modify   26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 by client.4282.0:145398 2014-06-12 08:45:47.925131]
2014-06-12 14:46:45.567487 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3101 (79'99,173'3101] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3099 active] add_log_entry 173'3101 (173'3100) modify   26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 by client.4282.0:145398 2014-06-12 08:45:47.925131
2014-06-12 14:46:45.567508 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3101 (79'99,173'3101] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3099 active] append_log  adding 1 keys
2014-06-12 14:46:45.567522 7f9d64bda700 10 trim log((79'99,173'3101], crt=0'0) to 79'100
2014-06-12 14:46:45.567550 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3101, trimmed: 79'100
2014-06-12 14:46:45.567575 7f9d64bda700  5 -- op tracker -- , seq: 748052, time: 2014-06-12 14:46:45.567570, event: started, op: osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.567586 7f9d64bda700  5 queue_transactions existing osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:45.567590 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.567591 7f9d64bda700  5 queue_op 0x185514a0 seq 2874 osr(3.46 0x276cc90) 1550607 bytes   (queue has 2 ops and 2200068 bytes)
2014-06-12 14:46:45.567603 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e97200 (2 active)
2014-06-12 14:46:45.567599 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.46( v 173'3101 (79'100,173'3101] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3099 active] do_sub_op osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.567611 7f9d7a161700  5 _do_op 0x185514a0 seq 2874 osr(3.46 0x276cc90)/0x276cc90 start
2014-06-12 14:46:45.567613 7f9d64bda700 10 osd.2 173 dequeue_op 0x25e42700 finish
2014-06-12 14:46:45.567617 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.567620 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.567615 7f9d7a161700 10 _do_transaction on 0x8222b70
2014-06-12 14:46:45.567626 7f9d7a161700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.567766 7f9d7a161700 10 lookup_strip_header done  cid meta oid 51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.567768 7f9d58d5b700  5 -- op tracker -- , seq: 748053, time: 2014-06-12 14:46:45.566423, event: header_read, op: osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.567779 7f9d58d5b700  5 -- op tracker -- , seq: 748053, time: 2014-06-12 14:46:45.566427, event: throttled, op: osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.567787 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.567788 7f9d58d5b700  5 -- op tracker -- , seq: 748053, time: 2014-06-12 14:46:45.567726, event: all_read, op: osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.567797 7f9d58d5b700  5 -- op tracker -- , seq: 748053, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.567801 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.567807 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:45.567813 7f9d58d5b700 15 osd.2 173 enqueue_op 0x2fa9d800 prio 127 cost 508711 latency 0.001390 osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.567836 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.567843 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2fa9d800 prio 127 cost 508711 latency 0.001419 osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.5c5( v 173'2855 (0'0,173'2855] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2854 active]
2014-06-12 14:46:45.567865 7f9d6abe6700  5 -- op tracker -- , seq: 748053, time: 2014-06-12 14:46:45.567861, event: reached_pg, op: osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.567873 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2855 (0'0,173'2855] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2854 active] handle_message: 0x2fa9d800
2014-06-12 14:46:45.567885 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2855 (0'0,173'2855] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2854 active] sub_op_modify trans f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 v 173'2856 (transaction) 174
2014-06-12 14:46:45.567900 7f9d6abe6700  5 -- op tracker -- , seq: 748053, time: 2014-06-12 14:46:45.567896, event: started, op: osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.567913 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2855 (0'0,173'2855] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2854 active] append_log log((0'0,173'2855], crt=0'0) [173'2856 (173'2855) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145138 2014-06-12 08:45:47.943645]
2014-06-12 14:46:45.567937 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2856 (0'0,173'2856] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2854 active] add_log_entry 173'2856 (173'2855) modify   f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 by client.4314.0:145138 2014-06-12 08:45:47.943645
2014-06-12 14:46:45.567955 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2856 (0'0,173'2856] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2854 active] append_log  adding 1 keys
2014-06-12 14:46:45.567990 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2856, trimmed: 
2014-06-12 14:46:45.568012 7f9d6abe6700  5 -- op tracker -- , seq: 748053, time: 2014-06-12 14:46:45.568006, event: started, op: osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.568019 7f9d6abe6700  5 queue_transactions existing osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.568022 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.568023 7f9d6abe6700  5 queue_op 0xdf17da0 seq 2610 osr(3.5c5 0x276dde0) 510197 bytes   (queue has 3 ops and 2710265 bytes)
2014-06-12 14:46:45.568029 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2856 (0'0,173'2856] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2854 active] do_sub_op osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.568043 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2fa9d800 finish
2014-06-12 14:46:45.568046 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.568049 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.568057 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439d40 (3 active)
2014-06-12 14:46:45.568061 7f9d7f16b700  5 _do_op 0xdf17da0 seq 2610 osr(3.5c5 0x276dde0)/0x276dde0 start
2014-06-12 14:46:45.568065 7f9d7f16b700 10 _do_transaction on 0xd9e60f0
2014-06-12 14:46:45.568069 7f9d7f16b700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.568154 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.568170 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.569726 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.569763 7f9d7a161700 15 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.569772 7f9d7a161700 10 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1 r = 0
2014-06-12 14:46:45.569780 7f9d7a161700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.569994 7f9d7a161700 15 _write 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 1650688~1548288
2014-06-12 14:46:45.570169 7f9d7a161700 10 lookup_strip_header done  cid 3.46_head oid 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3
2014-06-12 14:46:45.570289 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:45.576352 7f9d7a161700 10 _generic_write 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 1650688~1548288 = 0
2014-06-12 14:46:45.576522 7f9d7a161700 15 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3
2014-06-12 14:46:45.576555 7f9d7a161700 10 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 = 0
2014-06-12 14:46:45.576581 7f9d7a161700 15 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3
2014-06-12 14:46:45.576606 7f9d7a161700 10 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 = 0
2014-06-12 14:46:45.577068 7f9d7a161700 20 set_header setting 18271 cid 3.46_head oid 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 parent seq 0
2014-06-12 14:46:45.577236 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.577271 7f9d7a161700 20 set_header setting 2172 cid meta oid 51681c0b/pglog_3.46/0//-1 parent seq 0
2014-06-12 14:46:45.577301 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:45.577635 7f9d83173700 10 _do_op 0xa2da4e0 seq 2439 r = 0, finisher 0x82c65c0 0
2014-06-12 14:46:45.577691 7f9d7715b700  5 -- op tracker -- , seq: 748049, time: 2014-06-12 14:46:45.577682, event: commit_sent, op: osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.577702 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2639 (0'0,173'2639] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2637 active] sub_op_modify_commit on op osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:45.577728 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:45.577810 7f9d83173700 10 _finish_op 0xa2da4e0 seq 2439 osr(3.4e0 0x276dbd0)/0x276dbd0
2014-06-12 14:46:45.577849 7f9d7795c700  5 -- op tracker -- , seq: 748049, time: 2014-06-12 14:46:45.577843, event: sub_op_applied, op: osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.577858 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2639 (0'0,173'2639] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2638 active] sub_op_modify_applied on 0x662a540 op osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.577886 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2639 (0'0,173'2639] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2638 active] op_applied on primary on version 173'2639
2014-06-12 14:46:45.577898 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4e0( v 173'2639 (0'0,173'2639] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2638 active] op_applied on replica on version 173'2639
2014-06-12 14:46:45.577972 7f9d83173700  5 -- op tracker -- , seq: 748049, time: 2014-06-12 14:46:45.577963, event: done, op: osd_sub_op(client.4293.0:143963 3.4e0 79c3cce0/rbd_data.109c6b8b4567.0000000000001032/head//3 [] v 173'2639 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.578015 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5180900 (2 active)
2014-06-12 14:46:45.578019 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.596063 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.596108 7f9d7f16b700 15 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.596115 7f9d7f16b700 10 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1 r = 0
2014-06-12 14:46:45.596123 7f9d7f16b700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:46:45.596143 7f9d7f16b700 15 _write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 3686400~507904
2014-06-12 14:46:45.596308 7f9d7f16b700 10 lookup_strip_header done  cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.596438 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:45.598331 7f9d7f16b700 10 _generic_write 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 3686400~507904 = 0
2014-06-12 14:46:45.598388 7f9d7f16b700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.598403 7f9d7f16b700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.598409 7f9d7f16b700 15 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3
2014-06-12 14:46:45.598415 7f9d7f16b700 10 _setattrs 3.5c5_head/f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:46:45.598631 7f9d7f16b700 20 set_header setting 17924 cid 3.5c5_head oid f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 parent seq 0
2014-06-12 14:46:45.598702 7f9d7f16b700 20 set_header setting 2264 cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1 parent seq 0
2014-06-12 14:46:45.598715 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.598723 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:45.602600 7f9d7a161700 10 _do_op 0x185514a0 seq 2874 r = 0, finisher 0x5b77d60 0
2014-06-12 14:46:45.602684 7f9d7715b700  5 -- op tracker -- , seq: 748052, time: 2014-06-12 14:46:45.602674, event: commit_sent, op: osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.602695 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3101 (79'100,173'3101] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3099 active] sub_op_modify_commit on op osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:46:45.602728 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:45.602795 7f9d7a161700 10 _finish_op 0x185514a0 seq 2874 osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:45.602845 7f9d7795c700  5 -- op tracker -- , seq: 748052, time: 2014-06-12 14:46:45.602838, event: sub_op_applied, op: osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.602854 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3101 (79'100,173'3101] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active] sub_op_modify_applied on 0x8222a80 op osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.602875 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3101 (79'100,173'3101] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active] op_applied on primary on version 173'3101
2014-06-12 14:46:45.602886 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3101 (79'100,173'3101] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active] op_applied on replica on version 173'3101
2014-06-12 14:46:45.602976 7f9d7a161700  5 -- op tracker -- , seq: 748052, time: 2014-06-12 14:46:45.602967, event: done, op: osd_sub_op(client.4282.0:145398 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3101 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.603041 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e97200 (1 active)
2014-06-12 14:46:45.603068 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.609735 7f9d7f16b700 10 _do_op 0xdf17da0 seq 2610 r = 0, finisher 0x1b33ba0 0
2014-06-12 14:46:45.609812 7f9d7715b700  5 -- op tracker -- , seq: 748053, time: 2014-06-12 14:46:45.609801, event: commit_sent, op: osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.609823 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2856 (0'0,173'2856] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2854 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:45.609859 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:45.609921 7f9d7f16b700 10 _finish_op 0xdf17da0 seq 2610 osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:46:45.609964 7f9d7795c700  5 -- op tracker -- , seq: 748053, time: 2014-06-12 14:46:45.609959, event: sub_op_applied, op: osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.609973 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2856 (0'0,173'2856] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2855 active] sub_op_modify_applied on 0xd9e6000 op osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.609990 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2856 (0'0,173'2856] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2855 active] op_applied on primary on version 173'2856
2014-06-12 14:46:45.610000 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2856 (0'0,173'2856] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2855 active] op_applied on replica on version 173'2856
2014-06-12 14:46:45.610078 7f9d7f16b700  5 -- op tracker -- , seq: 748053, time: 2014-06-12 14:46:45.610069, event: done, op: osd_sub_op(client.4314.0:145138 3.5c5 f1779dc5/rbd_data.10b16b8b4567.0000000000001067/head//3 [] v 173'2856 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.610133 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439d40 (0 active)
2014-06-12 14:46:45.610158 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.618417 7f9d44101700  5 -- op tracker -- , seq: 748054, time: 2014-06-12 14:46:45.616129, event: header_read, op: osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.618432 7f9d44101700  5 -- op tracker -- , seq: 748054, time: 2014-06-12 14:46:45.616132, event: throttled, op: osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.618442 7f9d44101700  5 -- op tracker -- , seq: 748054, time: 2014-06-12 14:46:45.618359, event: all_read, op: osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.618451 7f9d44101700  5 -- op tracker -- , seq: 748054, time: 0.000000, event: dispatched, op: osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.618456 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.618462 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:45.618478 7f9d44101700 15 osd.2 173 enqueue_op 0x55c3400 prio 127 cost 996131 latency 0.002348 osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.618508 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.618519 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x55c3400 prio 127 cost 996131 latency 0.002390 osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.46( v 173'3101 (79'100,173'3101] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active]
2014-06-12 14:46:45.618546 7f9d6a3e5700  5 -- op tracker -- , seq: 748054, time: 2014-06-12 14:46:45.618541, event: reached_pg, op: osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.618554 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3101 (79'100,173'3101] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active] handle_message: 0x55c3400
2014-06-12 14:46:45.618569 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3101 (79'100,173'3101] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active] sub_op_modify trans 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 v 173'3102 (transaction) 174
2014-06-12 14:46:45.618589 7f9d6a3e5700  5 -- op tracker -- , seq: 748054, time: 2014-06-12 14:46:45.618584, event: started, op: osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.618604 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3101 (79'100,173'3101] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active] append_log log((79'100,173'3101], crt=0'0) [173'3102 (173'3101) modify   26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 by client.4282.0:145399 2014-06-12 08:45:47.989360]
2014-06-12 14:46:45.618633 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3102 (79'100,173'3102] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active] add_log_entry 173'3102 (173'3101) modify   26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 by client.4282.0:145399 2014-06-12 08:45:47.989360
2014-06-12 14:46:45.618664 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3102 (79'100,173'3102] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active] append_log  adding 1 keys
2014-06-12 14:46:45.618681 7f9d6a3e5700 10 trim log((79'100,173'3102], crt=0'0) to 79'101
2014-06-12 14:46:45.618712 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3102, trimmed: 79'101
2014-06-12 14:46:45.618741 7f9d6a3e5700  5 -- op tracker -- , seq: 748054, time: 2014-06-12 14:46:45.618736, event: started, op: osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.618753 7f9d6a3e5700  5 queue_transactions existing osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:45.618758 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.618759 7f9d6a3e5700  5 queue_op 0x227429c0 seq 2875 osr(3.46 0x276cc90) 997647 bytes   (queue has 1 ops and 997647 bytes)
2014-06-12 14:46:45.618766 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.46( v 173'3102 (79'101,173'3102] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active] do_sub_op osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.618781 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x55c3400 finish
2014-06-12 14:46:45.618785 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.618788 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.621343 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1e97200 (1 active)
2014-06-12 14:46:45.621362 7f9d7e169700  5 _do_op 0x227429c0 seq 2875 osr(3.46 0x276cc90)/0x276cc90 start
2014-06-12 14:46:45.621369 7f9d7e169700 10 _do_transaction on 0x62942b0
2014-06-12 14:46:45.621378 7f9d7e169700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.621530 7f9d7e169700 10 lookup_strip_header done  cid meta oid 51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.621557 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.621649 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.621693 7f9d7e169700 15 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.621705 7f9d7e169700 10 _omap_rmkeys meta/51681c0b/pglog_3.46/0//-1 r = 0
2014-06-12 14:46:45.621714 7f9d7e169700 15 _omap_setkeys meta/51681c0b/pglog_3.46/0//-1
2014-06-12 14:46:45.621894 7f9d7e169700 15 _write 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 3198976~995328
2014-06-12 14:46:45.622071 7f9d7e169700 10 lookup_strip_header done  cid 3.46_head oid 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3
2014-06-12 14:46:45.622229 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:45.626561 7f9d7e169700 10 _generic_write 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 3198976~995328 = 0
2014-06-12 14:46:45.626688 7f9d7e169700 15 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3
2014-06-12 14:46:45.626705 7f9d7e169700 10 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 = 0
2014-06-12 14:46:45.626711 7f9d7e169700 15 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3
2014-06-12 14:46:45.626722 7f9d7e169700 10 _setattrs 3.46_head/26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 = 0
2014-06-12 14:46:45.627038 7f9d7e169700 20 set_header setting 18271 cid 3.46_head oid 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 parent seq 0
2014-06-12 14:46:45.627145 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.627161 7f9d7e169700 20 set_header setting 2172 cid meta oid 51681c0b/pglog_3.46/0//-1 parent seq 0
2014-06-12 14:46:45.627171 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:45.637974 7f9d7e169700 10 _do_op 0x227429c0 seq 2875 r = 0, finisher 0x177287e0 0
2014-06-12 14:46:45.637998 7f9d7e169700 10 _finish_op 0x227429c0 seq 2875 osr(3.46 0x276cc90)/0x276cc90
2014-06-12 14:46:45.638026 7f9d7795c700  5 -- op tracker -- , seq: 748054, time: 2014-06-12 14:46:45.638017, event: sub_op_applied, op: osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.638039 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3102 (79'101,173'3102] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active] sub_op_modify_applied on 0x62941c0 op osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.638070 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:45.638095 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3102 (79'101,173'3102] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active] op_applied on primary on version 173'3102
2014-06-12 14:46:45.638108 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3102 (79'101,173'3102] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active] op_applied on replica on version 173'3102
2014-06-12 14:46:45.638125 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1e97200 (0 active)
2014-06-12 14:46:45.638144 7f9d7715b700  5 -- op tracker -- , seq: 748054, time: 2014-06-12 14:46:45.638137, event: commit_sent, op: osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.638152 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.46( v 173'3102 (79'101,173'3102] local-les=139 n=333 ec=23 les/c 139/139 138/138/138) [21,2] r=1 lpr=138 pi=23-137/12 luod=0'0 crt=0'0 lcod 173'3100 active] sub_op_modify_commit on op osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.21
2014-06-12 14:46:45.638183 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:45.638168 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.638218 7f9d7715b700  5 -- op tracker -- , seq: 748054, time: 2014-06-12 14:46:45.638212, event: done, op: osd_sub_op(client.4282.0:145399 3.46 26d5c046/rbd_data.10916b8b4567.00000000000010a4/head//3 [] v 173'3102 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.683500 7f9d56432700  5 -- op tracker -- , seq: 748055, time: 2014-06-12 14:46:45.682800, event: header_read, op: osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.683509 7f9d56432700  5 -- op tracker -- , seq: 748055, time: 2014-06-12 14:46:45.682804, event: throttled, op: osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.683514 7f9d56432700  5 -- op tracker -- , seq: 748055, time: 2014-06-12 14:46:45.683466, event: all_read, op: osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.683520 7f9d56432700  5 -- op tracker -- , seq: 748055, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.683523 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.683527 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:45.683533 7f9d56432700 15 osd.2 173 enqueue_op 0x68a6600 prio 127 cost 332583 latency 0.000733 osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.683883 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.683891 7f9d69be4700 10 osd.2 173 dequeue_op 0x68a6600 prio 127 cost 332583 latency 0.001091 osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.60e( v 173'2563 (0'0,173'2563] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2562 active]
2014-06-12 14:46:45.683916 7f9d69be4700  5 -- op tracker -- , seq: 748055, time: 2014-06-12 14:46:45.683912, event: reached_pg, op: osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.683926 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2563 (0'0,173'2563] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2562 active] handle_message: 0x68a6600
2014-06-12 14:46:45.683939 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2563 (0'0,173'2563] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2562 active] sub_op_modify trans 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 v 173'2564 (transaction) 174
2014-06-12 14:46:45.683957 7f9d69be4700  5 -- op tracker -- , seq: 748055, time: 2014-06-12 14:46:45.683952, event: started, op: osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.683977 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2563 (0'0,173'2563] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2562 active] append_log log((0'0,173'2563], crt=0'0) [173'2564 (173'666) modify   8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 by client.4298.0:145522 2014-06-12 08:45:20.005110]
2014-06-12 14:46:45.684007 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2564 (0'0,173'2564] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2562 active] add_log_entry 173'2564 (173'666) modify   8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 by client.4298.0:145522 2014-06-12 08:45:20.005110
2014-06-12 14:46:45.684035 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2564 (0'0,173'2564] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2562 active] append_log  adding 1 keys
2014-06-12 14:46:45.684078 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2564, trimmed: 
2014-06-12 14:46:45.684116 7f9d69be4700  5 -- op tracker -- , seq: 748055, time: 2014-06-12 14:46:45.684110, event: started, op: osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.684133 7f9d69be4700  5 queue_transactions existing osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:45.684138 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.684139 7f9d69be4700  5 queue_op 0x1b39eba0 seq 2406 osr(3.60e 0x276dcf0) 334069 bytes   (queue has 1 ops and 334069 bytes)
2014-06-12 14:46:45.684151 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533cd80 (1 active)
2014-06-12 14:46:45.684158 7f9d7815d700  5 _do_op 0x1b39eba0 seq 2406 osr(3.60e 0x276dcf0)/0x276dcf0 start
2014-06-12 14:46:45.684162 7f9d7815d700 10 _do_transaction on 0x2daa7eb0
2014-06-12 14:46:45.684148 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.60e( v 173'2564 (0'0,173'2564] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2562 active] do_sub_op osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.684163 7f9d69be4700 10 osd.2 173 dequeue_op 0x68a6600 finish
2014-06-12 14:46:45.684167 7f9d7815d700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.684167 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.684171 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.684290 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.684304 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.684351 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.684362 7f9d7815d700 15 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.684366 7f9d7815d700 10 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1 r = 0
2014-06-12 14:46:45.684369 7f9d7815d700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.684378 7f9d7815d700 15 _write 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 0~331776
2014-06-12 14:46:45.684508 7f9d7815d700 10 lookup_strip_header done  cid 3.60e_head oid 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:45.684527 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:45.685624 7f9d7815d700 10 _generic_write 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 0~331776 = 0
2014-06-12 14:46:45.685674 7f9d7815d700 15 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:45.685690 7f9d7815d700 10 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 = 0
2014-06-12 14:46:45.685697 7f9d7815d700 15 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:45.685707 7f9d7815d700 10 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 = 0
2014-06-12 14:46:45.685844 7f9d7815d700 20 set_header setting 17483 cid 3.60e_head oid 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 parent seq 0
2014-06-12 14:46:45.685901 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.685920 7f9d7815d700 20 set_header setting 2240 cid meta oid 1fe13dcb/pglog_3.60e/0//-1 parent seq 0
2014-06-12 14:46:45.685932 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:45.687474 7f9d7815d700 10 _do_op 0x1b39eba0 seq 2406 r = 0, finisher 0x65e1820 0
2014-06-12 14:46:45.687496 7f9d7815d700 10 _finish_op 0x1b39eba0 seq 2406 osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:45.687505 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533cd80 (0 active)
2014-06-12 14:46:45.687510 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.687516 7f9d7715b700  5 -- op tracker -- , seq: 748055, time: 2014-06-12 14:46:45.687507, event: commit_sent, op: osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.687528 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2564 (0'0,173'2564] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2562 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:45.687558 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.689651 7f9d7795c700  5 -- op tracker -- , seq: 748055, time: 2014-06-12 14:46:45.689639, event: sub_op_applied, op: osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.689665 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2564 (0'0,173'2564] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2563 active] sub_op_modify_applied on 0x2daa7dc0 op osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.689684 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2564 (0'0,173'2564] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2563 active] op_applied on primary on version 173'2564
2014-06-12 14:46:45.689698 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2564 (0'0,173'2564] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2563 active] op_applied on replica on version 173'2564
2014-06-12 14:46:45.689726 7f9d7795c700  5 -- op tracker -- , seq: 748055, time: 2014-06-12 14:46:45.689722, event: done, op: osd_sub_op(client.4298.0:145522 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2564 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.835617 7f9d56432700  5 -- op tracker -- , seq: 748056, time: 2014-06-12 14:46:45.833752, event: header_read, op: osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.835626 7f9d56432700  5 -- op tracker -- , seq: 748056, time: 2014-06-12 14:46:45.833754, event: throttled, op: osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.835631 7f9d56432700  5 -- op tracker -- , seq: 748056, time: 2014-06-12 14:46:45.835568, event: all_read, op: osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.835646 7f9d56432700  5 -- op tracker -- , seq: 748056, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.835651 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.835656 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:45.835661 7f9d56432700 15 osd.2 173 enqueue_op 0x6f4b900 prio 127 cost 1032999 latency 0.001909 osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.835688 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.835696 7f9d653db700 10 osd.2 173 dequeue_op 0x6f4b900 prio 127 cost 1032999 latency 0.001944 osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.60e( v 173'2564 (0'0,173'2564] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2563 active]
2014-06-12 14:46:45.835724 7f9d653db700  5 -- op tracker -- , seq: 748056, time: 2014-06-12 14:46:45.835719, event: reached_pg, op: osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.835735 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2564 (0'0,173'2564] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2563 active] handle_message: 0x6f4b900
2014-06-12 14:46:45.835748 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2564 (0'0,173'2564] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2563 active] sub_op_modify trans 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 v 173'2565 (transaction) 174
2014-06-12 14:46:45.835765 7f9d653db700  5 -- op tracker -- , seq: 748056, time: 2014-06-12 14:46:45.835761, event: started, op: osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.835786 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2564 (0'0,173'2564] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2563 active] append_log log((0'0,173'2564], crt=0'0) [173'2565 (173'2564) modify   8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 by client.4298.0:145523 2014-06-12 08:45:20.584636]
2014-06-12 14:46:45.835820 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2565 (0'0,173'2565] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2563 active] add_log_entry 173'2565 (173'2564) modify   8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 by client.4298.0:145523 2014-06-12 08:45:20.584636
2014-06-12 14:46:45.835851 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2565 (0'0,173'2565] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2563 active] append_log  adding 1 keys
2014-06-12 14:46:45.835884 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2565, trimmed: 
2014-06-12 14:46:45.835926 7f9d653db700  5 -- op tracker -- , seq: 748056, time: 2014-06-12 14:46:45.835921, event: started, op: osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.835937 7f9d653db700  5 queue_transactions existing osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:45.835940 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.835942 7f9d653db700  5 queue_op 0x3886d80 seq 2407 osr(3.60e 0x276dcf0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:45.835953 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533cd80 (1 active)
2014-06-12 14:46:45.835958 7f9d82171700  5 _do_op 0x3886d80 seq 2407 osr(3.60e 0x276dcf0)/0x276dcf0 start
2014-06-12 14:46:45.835962 7f9d82171700 10 _do_transaction on 0x2daa69b0
2014-06-12 14:46:45.835950 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.60e( v 173'2565 (0'0,173'2565] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2563 active] do_sub_op osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.835965 7f9d653db700 10 osd.2 173 dequeue_op 0x6f4b900 finish
2014-06-12 14:46:45.835967 7f9d82171700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.835969 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.835971 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.836092 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.836117 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.836175 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.836188 7f9d82171700 15 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.836200 7f9d82171700 10 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1 r = 0
2014-06-12 14:46:45.836204 7f9d82171700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:45.836212 7f9d82171700 15 _write 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 331776~1032192
2014-06-12 14:46:45.836325 7f9d82171700 10 lookup_strip_header done  cid 3.60e_head oid 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:45.836350 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:45.839544 7f9d82171700 10 _generic_write 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 331776~1032192 = 0
2014-06-12 14:46:45.839637 7f9d82171700 15 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:45.839649 7f9d82171700 10 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 = 0
2014-06-12 14:46:45.839661 7f9d82171700 15 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:45.839666 7f9d82171700 10 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 = 0
2014-06-12 14:46:45.839874 7f9d82171700 20 set_header setting 17483 cid 3.60e_head oid 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 parent seq 0
2014-06-12 14:46:45.839953 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.839970 7f9d82171700 20 set_header setting 2240 cid meta oid 1fe13dcb/pglog_3.60e/0//-1 parent seq 0
2014-06-12 14:46:45.839988 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:45.849792 7f9d82171700 10 _do_op 0x3886d80 seq 2407 r = 0, finisher 0x63869a0 0
2014-06-12 14:46:45.849865 7f9d7715b700  5 -- op tracker -- , seq: 748056, time: 2014-06-12 14:46:45.849854, event: commit_sent, op: osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.849876 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2565 (0'0,173'2565] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2563 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:45.849904 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.849966 7f9d82171700 10 _finish_op 0x3886d80 seq 2407 osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:45.850005 7f9d7795c700  5 -- op tracker -- , seq: 748056, time: 2014-06-12 14:46:45.850000, event: sub_op_applied, op: osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.850015 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2565 (0'0,173'2565] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2564 active] sub_op_modify_applied on 0x2daa68c0 op osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.850031 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2565 (0'0,173'2565] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2564 active] op_applied on primary on version 173'2565
2014-06-12 14:46:45.850041 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2565 (0'0,173'2565] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2564 active] op_applied on replica on version 173'2565
2014-06-12 14:46:45.850112 7f9d82171700  5 -- op tracker -- , seq: 748056, time: 2014-06-12 14:46:45.850102, event: done, op: osd_sub_op(client.4298.0:145523 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2565 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.850161 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533cd80 (0 active)
2014-06-12 14:46:45.850185 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.907819 7f9d56432700  5 -- op tracker -- , seq: 748057, time: 2014-06-12 14:46:45.902813, event: header_read, op: osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.907851 7f9d56432700  5 -- op tracker -- , seq: 748057, time: 2014-06-12 14:46:45.902815, event: throttled, op: osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.907859 7f9d56432700  5 -- op tracker -- , seq: 748057, time: 2014-06-12 14:46:45.907757, event: all_read, op: osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.907871 7f9d56432700  5 -- op tracker -- , seq: 748057, time: 0.000000, event: dispatched, op: osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.907877 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:45.907885 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:45.907895 7f9d56432700 15 osd.2 173 enqueue_op 0x6f4bc00 prio 127 cost 2425639 latency 0.005082 osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.907942 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:45.907953 7f9d63bd8700 10 osd.2 173 dequeue_op 0x6f4bc00 prio 127 cost 2425639 latency 0.005140 osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.71f( v 173'2702 (0'0,173'2702] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2701 active]
2014-06-12 14:46:45.907980 7f9d63bd8700  5 -- op tracker -- , seq: 748057, time: 2014-06-12 14:46:45.907976, event: reached_pg, op: osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.907989 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2702 (0'0,173'2702] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2701 active] handle_message: 0x6f4bc00
2014-06-12 14:46:45.908004 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2702 (0'0,173'2702] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2701 active] sub_op_modify trans 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 v 173'2703 (transaction) 174
2014-06-12 14:46:45.908024 7f9d63bd8700  5 -- op tracker -- , seq: 748057, time: 2014-06-12 14:46:45.908020, event: started, op: osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.908042 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2702 (0'0,173'2702] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2701 active] append_log log((0'0,173'2702], crt=0'0) [173'2703 (173'583) modify   4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 by client.4280.0:139905 2014-06-12 08:45:40.401355]
2014-06-12 14:46:45.908064 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2703 (0'0,173'2703] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2701 active] add_log_entry 173'2703 (173'583) modify   4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 by client.4280.0:139905 2014-06-12 08:45:40.401355
2014-06-12 14:46:45.908088 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2703 (0'0,173'2703] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2701 active] append_log  adding 1 keys
2014-06-12 14:46:45.908124 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2703, trimmed: 
2014-06-12 14:46:45.908150 7f9d63bd8700  5 -- op tracker -- , seq: 748057, time: 2014-06-12 14:46:45.908146, event: started, op: osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.908161 7f9d63bd8700  5 queue_transactions existing osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:45.908165 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:45.908166 7f9d63bd8700  5 queue_op 0x1acc6d80 seq 2535 osr(3.71f 0x276de50) 2427125 bytes   (queue has 1 ops and 2427125 bytes)
2014-06-12 14:46:45.908174 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.71f( v 173'2703 (0'0,173'2703] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2701 active] do_sub_op osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.908190 7f9d63bd8700 10 osd.2 173 dequeue_op 0x6f4bc00 finish
2014-06-12 14:46:45.908193 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:45.908196 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:45.908205 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d0240 (1 active)
2014-06-12 14:46:45.908210 7f9d7e96a700  5 _do_op 0x1acc6d80 seq 2535 osr(3.71f 0x276de50)/0x276de50 start
2014-06-12 14:46:45.908215 7f9d7e96a700 10 _do_transaction on 0x624d5f0
2014-06-12 14:46:45.908222 7f9d7e96a700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:45.908448 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:45.908468 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:45.908525 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:45.908544 7f9d7e96a700 15 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:45.908553 7f9d7e96a700 10 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1 r = 0
2014-06-12 14:46:45.908559 7f9d7e96a700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:45.908579 7f9d7e96a700 15 _write 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 0~2424832
2014-06-12 14:46:45.908712 7f9d7e96a700 10 lookup_strip_header done  cid 3.71f_head oid 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3
2014-06-12 14:46:45.908874 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:45.918202 7f9d7e96a700 10 _generic_write 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 0~2424832 = 0
2014-06-12 14:46:45.918650 7f9d7e96a700 15 _setattrs 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3
2014-06-12 14:46:45.918682 7f9d7e96a700 10 _setattrs 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 = 0
2014-06-12 14:46:45.918690 7f9d7e96a700 15 _setattrs 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3
2014-06-12 14:46:45.918702 7f9d7e96a700 10 _setattrs 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 = 0
2014-06-12 14:46:45.919537 7f9d7e96a700 20 set_header setting 15950 cid 3.71f_head oid 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 parent seq 0
2014-06-12 14:46:45.919947 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:45.919964 7f9d7e96a700 20 set_header setting 2330 cid meta oid 1fe0985b/pglog_3.71f/0//-1 parent seq 0
2014-06-12 14:46:45.919978 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:45.933453 7f9d7e96a700 10 _do_op 0x1acc6d80 seq 2535 r = 0, finisher 0x1e3bdfe0 0
2014-06-12 14:46:45.933479 7f9d7e96a700 10 _finish_op 0x1acc6d80 seq 2535 osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:45.933497 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d0240 (0 active)
2014-06-12 14:46:45.933505 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:45.933500 7f9d7715b700  5 -- op tracker -- , seq: 748057, time: 2014-06-12 14:46:45.933489, event: commit_sent, op: osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.933511 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2703 (0'0,173'2703] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2701 active] sub_op_modify_commit on op osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:45.933539 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:45.933608 7f9d7795c700  5 -- op tracker -- , seq: 748057, time: 2014-06-12 14:46:45.933602, event: sub_op_applied, op: osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:45.933637 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2703 (0'0,173'2703] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2702 active] sub_op_modify_applied on 0x624d500 op osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:45.933653 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2703 (0'0,173'2703] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2702 active] op_applied on primary on version 173'2703
2014-06-12 14:46:45.933662 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2703 (0'0,173'2703] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2702 active] op_applied on replica on version 173'2703
2014-06-12 14:46:45.933689 7f9d7795c700  5 -- op tracker -- , seq: 748057, time: 2014-06-12 14:46:45.933684, event: done, op: osd_sub_op(client.4280.0:139905 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.072352 7f9d58d5b700  5 -- op tracker -- , seq: 748058, time: 2014-06-12 14:46:46.071719, event: header_read, op: osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.072365 7f9d58d5b700  5 -- op tracker -- , seq: 748058, time: 2014-06-12 14:46:46.071722, event: throttled, op: osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.072375 7f9d58d5b700  5 -- op tracker -- , seq: 748058, time: 2014-06-12 14:46:46.072302, event: all_read, op: osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.072383 7f9d58d5b700  5 -- op tracker -- , seq: 748058, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.072390 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:46.072397 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:46.072406 7f9d58d5b700 15 osd.2 173 enqueue_op 0x2fa9c300 prio 127 cost 209703 latency 0.000687 osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.072438 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.072446 7f9d68be2700 10 osd.2 173 dequeue_op 0x2fa9c300 prio 127 cost 209703 latency 0.000727 osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.263( v 173'3181 (79'180,173'3181] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3180 active]
2014-06-12 14:46:46.072463 7f9d68be2700  5 -- op tracker -- , seq: 748058, time: 2014-06-12 14:46:46.072460, event: reached_pg, op: osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.072471 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3181 (79'180,173'3181] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3180 active] handle_message: 0x2fa9c300
2014-06-12 14:46:46.072479 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3181 (79'180,173'3181] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3180 active] sub_op_modify trans 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 v 173'3182 (transaction) 174
2014-06-12 14:46:46.072489 7f9d68be2700  5 -- op tracker -- , seq: 748058, time: 2014-06-12 14:46:46.072487, event: started, op: osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.072505 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3181 (79'180,173'3181] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3180 active] append_log log((79'180,173'3181], crt=0'0) [173'3182 (173'985) modify   44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 by client.4303.0:144884 2014-06-12 08:45:48.346929]
2014-06-12 14:46:46.072525 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3182 (79'180,173'3182] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3180 active] add_log_entry 173'3182 (173'985) modify   44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 by client.4303.0:144884 2014-06-12 08:45:48.346929
2014-06-12 14:46:46.072546 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3182 (79'180,173'3182] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3180 active] append_log  adding 1 keys
2014-06-12 14:46:46.072562 7f9d68be2700 10 trim log((79'180,173'3182], crt=0'0) to 79'181
2014-06-12 14:46:46.072585 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3182, trimmed: 79'181
2014-06-12 14:46:46.072605 7f9d68be2700  5 -- op tracker -- , seq: 748058, time: 2014-06-12 14:46:46.072602, event: started, op: osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.072614 7f9d68be2700  5 queue_transactions existing osr(3.263 0x276d300)/0x276d300
2014-06-12 14:46:46.072618 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.072620 7f9d68be2700  5 queue_op 0x565a4e0 seq 2898 osr(3.263 0x276d300) 211224 bytes   (queue has 1 ops and 211224 bytes)
2014-06-12 14:46:46.072625 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.263( v 173'3182 (79'181,173'3182] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3180 active] do_sub_op osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.072635 7f9d68be2700 10 osd.2 173 dequeue_op 0x2fa9c300 finish
2014-06-12 14:46:46.072637 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.072639 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.073105 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c47e0 (1 active)
2014-06-12 14:46:46.073115 7f9d8116f700  5 _do_op 0x565a4e0 seq 2898 osr(3.263 0x276d300)/0x276d300 start
2014-06-12 14:46:46.073120 7f9d8116f700 10 _do_transaction on 0x6294b70
2014-06-12 14:46:46.073127 7f9d8116f700 15 _omap_setkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.073284 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.073309 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.073378 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.073561 7f9d8116f700 15 _omap_rmkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.073574 7f9d8116f700 10 _omap_rmkeys meta/1fe27fea/pglog_3.263/0//-1 r = 0
2014-06-12 14:46:46.073582 7f9d8116f700 15 _omap_setkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.073603 7f9d8116f700 15 _write 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 0~208896
2014-06-12 14:46:46.073843 7f9d8116f700 10 lookup_strip_header done  cid 3.263_head oid 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.074033 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:46.074839 7f9d8116f700 10 _generic_write 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 0~208896 = 0
2014-06-12 14:46:46.074894 7f9d8116f700 15 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.074908 7f9d8116f700 10 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 = 0
2014-06-12 14:46:46.074914 7f9d8116f700 15 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.074923 7f9d8116f700 10 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 = 0
2014-06-12 14:46:46.075077 7f9d8116f700 20 set_header setting 17535 cid 3.263_head oid 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 parent seq 0
2014-06-12 14:46:46.075115 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.075130 7f9d8116f700 20 set_header setting 2232 cid meta oid 1fe27fea/pglog_3.263/0//-1 parent seq 0
2014-06-12 14:46:46.075140 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:46.076165 7f9d8116f700 10 _do_op 0x565a4e0 seq 2898 r = 0, finisher 0x6dbdfe0 0
2014-06-12 14:46:46.076178 7f9d8116f700 10 _finish_op 0x565a4e0 seq 2898 osr(3.263 0x276d300)/0x276d300
2014-06-12 14:46:46.076189 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c47e0 (0 active)
2014-06-12 14:46:46.076193 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.076248 7f9d7715b700  5 -- op tracker -- , seq: 748058, time: 2014-06-12 14:46:46.076238, event: commit_sent, op: osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.076259 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3182 (79'181,173'3182] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3180 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:46.076286 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:46.076377 7f9d7795c700  5 -- op tracker -- , seq: 748058, time: 2014-06-12 14:46:46.076370, event: sub_op_applied, op: osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.076385 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3182 (79'181,173'3182] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] sub_op_modify_applied on 0x6294a80 op osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.076400 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3182 (79'181,173'3182] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] op_applied on primary on version 173'3182
2014-06-12 14:46:46.076414 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3182 (79'181,173'3182] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] op_applied on replica on version 173'3182
2014-06-12 14:46:46.076440 7f9d7795c700  5 -- op tracker -- , seq: 748058, time: 2014-06-12 14:46:46.076435, event: done, op: osd_sub_op(client.4303.0:144884 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3182 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.078469 7f9d58d5b700  5 -- op tracker -- , seq: 748059, time: 2014-06-12 14:46:46.072536, event: header_read, op: osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.078483 7f9d58d5b700  5 -- op tracker -- , seq: 748059, time: 2014-06-12 14:46:46.072537, event: throttled, op: osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.078496 7f9d58d5b700  5 -- op tracker -- , seq: 748059, time: 2014-06-12 14:46:46.078434, event: all_read, op: osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.080966 7f9d58d5b700  5 -- op tracker -- , seq: 748059, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.080974 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:46.080981 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:46.081030 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34017a00 prio 127 cost 1032999 latency 0.008494 osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.081070 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.081083 7f9d623d5700 10 osd.2 173 dequeue_op 0x34017a00 prio 127 cost 1032999 latency 0.008546 osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.263( v 173'3182 (79'181,173'3182] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active]
2014-06-12 14:46:46.081113 7f9d623d5700  5 -- op tracker -- , seq: 748059, time: 2014-06-12 14:46:46.081108, event: reached_pg, op: osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.081123 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3182 (79'181,173'3182] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] handle_message: 0x34017a00
2014-06-12 14:46:46.081144 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3182 (79'181,173'3182] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] sub_op_modify trans 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 v 173'3183 (transaction) 174
2014-06-12 14:46:46.081161 7f9d623d5700  5 -- op tracker -- , seq: 748059, time: 2014-06-12 14:46:46.081157, event: started, op: osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.081182 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3182 (79'181,173'3182] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] append_log log((79'181,173'3182], crt=0'0) [173'3183 (173'3182) modify   44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 by client.4303.0:144885 2014-06-12 08:45:48.438321]
2014-06-12 14:46:46.081209 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3183 (79'181,173'3183] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] add_log_entry 173'3183 (173'3182) modify   44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 by client.4303.0:144885 2014-06-12 08:45:48.438321
2014-06-12 14:46:46.081237 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3183 (79'181,173'3183] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] append_log  adding 1 keys
2014-06-12 14:46:46.081256 7f9d623d5700 10 trim log((79'181,173'3183], crt=0'0) to 79'182
2014-06-12 14:46:46.081287 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3183, trimmed: 79'182
2014-06-12 14:46:46.081319 7f9d623d5700  5 -- op tracker -- , seq: 748059, time: 2014-06-12 14:46:46.081314, event: started, op: osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.081333 7f9d623d5700  5 queue_transactions existing osr(3.263 0x276d300)/0x276d300
2014-06-12 14:46:46.081339 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.081341 7f9d623d5700  5 queue_op 0x7c9b7a0 seq 2899 osr(3.263 0x276d300) 1034520 bytes   (queue has 1 ops and 1034520 bytes)
2014-06-12 14:46:46.081350 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.263( v 173'3183 (79'182,173'3183] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] do_sub_op osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.081365 7f9d623d5700 10 osd.2 173 dequeue_op 0x34017a00 finish
2014-06-12 14:46:46.081368 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.081371 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.081623 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c47e0 (1 active)
2014-06-12 14:46:46.081635 7f9d7d167700  5 _do_op 0x7c9b7a0 seq 2899 osr(3.263 0x276d300)/0x276d300 start
2014-06-12 14:46:46.081640 7f9d7d167700 10 _do_transaction on 0x62210b0
2014-06-12 14:46:46.081646 7f9d7d167700 15 _omap_setkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.081865 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.081904 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.081989 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.082025 7f9d7d167700 15 _omap_rmkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.082033 7f9d7d167700 10 _omap_rmkeys meta/1fe27fea/pglog_3.263/0//-1 r = 0
2014-06-12 14:46:46.082040 7f9d7d167700 15 _omap_setkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.082239 7f9d7d167700 15 _write 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 208896~1032192
2014-06-12 14:46:46.082419 7f9d7d167700 10 lookup_strip_header done  cid 3.263_head oid 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.082607 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:46.088428 7f9d58d5b700  5 -- op tracker -- , seq: 748060, time: 2014-06-12 14:46:46.081067, event: header_read, op: osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.088441 7f9d58d5b700  5 -- op tracker -- , seq: 748060, time: 2014-06-12 14:46:46.081068, event: throttled, op: osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.088450 7f9d58d5b700  5 -- op tracker -- , seq: 748060, time: 2014-06-12 14:46:46.088393, event: all_read, op: osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.088459 7f9d58d5b700  5 -- op tracker -- , seq: 748060, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.088464 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:46.088471 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:46.088479 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016d00 prio 127 cost 1032999 latency 0.007412 osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.088508 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.088516 7f9d643d9700 10 osd.2 173 dequeue_op 0x34016d00 prio 127 cost 1032999 latency 0.007449 osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.263( v 173'3183 (79'182,173'3183] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active]
2014-06-12 14:46:46.088543 7f9d643d9700  5 -- op tracker -- , seq: 748060, time: 2014-06-12 14:46:46.088538, event: reached_pg, op: osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.088555 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3183 (79'182,173'3183] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] handle_message: 0x34016d00
2014-06-12 14:46:46.088568 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3183 (79'182,173'3183] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] sub_op_modify trans 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 v 173'3184 (transaction) 174
2014-06-12 14:46:46.088585 7f9d643d9700  5 -- op tracker -- , seq: 748060, time: 2014-06-12 14:46:46.088581, event: started, op: osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.088601 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3183 (79'182,173'3183] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] append_log log((79'182,173'3183], crt=0'0) [173'3184 (173'3183) modify   44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 by client.4303.0:144886 2014-06-12 08:45:48.444880]
2014-06-12 14:46:46.088627 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3184 (79'182,173'3184] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] add_log_entry 173'3184 (173'3183) modify   44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 by client.4303.0:144886 2014-06-12 08:45:48.444880
2014-06-12 14:46:46.088650 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3184 (79'182,173'3184] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] append_log  adding 1 keys
2014-06-12 14:46:46.088665 7f9d643d9700 10 trim log((79'182,173'3184], crt=0'0) to 79'183
2014-06-12 14:46:46.088693 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3184, trimmed: 79'183
2014-06-12 14:46:46.088720 7f9d643d9700  5 -- op tracker -- , seq: 748060, time: 2014-06-12 14:46:46.088715, event: started, op: osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.088730 7f9d643d9700  5 queue_transactions existing osr(3.263 0x276d300)/0x276d300
2014-06-12 14:46:46.088734 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.088736 7f9d643d9700  5 queue_op 0xa2dade0 seq 2900 osr(3.263 0x276d300) 1034520 bytes   (queue has 2 ops and 2069040 bytes)
2014-06-12 14:46:46.088748 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c47e0 (2 active)
2014-06-12 14:46:46.088743 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.263( v 173'3184 (79'183,173'3184] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] do_sub_op osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.088757 7f9d643d9700 10 osd.2 173 dequeue_op 0x34016d00 finish
2014-06-12 14:46:46.088760 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.088763 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.088901 7f9d7d167700 10 _generic_write 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 208896~1032192 = 0
2014-06-12 14:46:46.089005 7f9d7d167700 15 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.089039 7f9d7d167700 10 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 = 0
2014-06-12 14:46:46.089052 7f9d7d167700 15 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.089060 7f9d7d167700 10 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 = 0
2014-06-12 14:46:46.089365 7f9d7d167700 20 set_header setting 17535 cid 3.263_head oid 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 parent seq 0
2014-06-12 14:46:46.089467 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.089483 7f9d7d167700 20 set_header setting 2232 cid meta oid 1fe27fea/pglog_3.263/0//-1 parent seq 0
2014-06-12 14:46:46.089494 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:46.096538 7f9d7d167700 10 _do_op 0x7c9b7a0 seq 2899 r = 0, finisher 0x8890100 0
2014-06-12 14:46:46.096590 7f9d7715b700  5 -- op tracker -- , seq: 748059, time: 2014-06-12 14:46:46.096579, event: commit_sent, op: osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.096604 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3184 (79'183,173'3184] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3181 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:46.096641 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:46.096709 7f9d7d167700 10 _finish_op 0x7c9b7a0 seq 2899 osr(3.263 0x276d300)/0x276d300
2014-06-12 14:46:46.096724 7f9d79960700  5 _do_op 0xa2dade0 seq 2900 osr(3.263 0x276d300)/0x276d300 start
2014-06-12 14:46:46.096735 7f9d79960700 10 _do_transaction on 0x90f0b70
2014-06-12 14:46:46.096735 7f9d7795c700  5 -- op tracker -- , seq: 748059, time: 2014-06-12 14:46:46.096730, event: sub_op_applied, op: osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.096744 7f9d79960700 15 _omap_setkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.096743 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3184 (79'183,173'3184] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3182 active] sub_op_modify_applied on 0x6220fc0 op osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.096757 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3184 (79'183,173'3184] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3182 active] op_applied on primary on version 173'3183
2014-06-12 14:46:46.096767 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3184 (79'183,173'3184] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3182 active] op_applied on replica on version 173'3183
2014-06-12 14:46:46.096802 7f9d7d167700  5 -- op tracker -- , seq: 748059, time: 2014-06-12 14:46:46.096794, event: done, op: osd_sub_op(client.4303.0:144885 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3183 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.096884 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c47e0 (1 active)
2014-06-12 14:46:46.096890 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.096907 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.096933 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.096998 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.097022 7f9d79960700 15 _omap_rmkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.097034 7f9d79960700 10 _omap_rmkeys meta/1fe27fea/pglog_3.263/0//-1 r = 0
2014-06-12 14:46:46.097040 7f9d79960700 15 _omap_setkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.097194 7f9d79960700 15 _write 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 1241088~1032192
2014-06-12 14:46:46.097353 7f9d79960700 10 lookup_strip_header done  cid 3.263_head oid 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.097471 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:46.102032 7f9d79960700 10 _generic_write 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 1241088~1032192 = 0
2014-06-12 14:46:46.102166 7f9d79960700 15 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.102191 7f9d79960700 10 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 = 0
2014-06-12 14:46:46.102200 7f9d79960700 15 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.102209 7f9d79960700 10 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 = 0
2014-06-12 14:46:46.102532 7f9d79960700 20 set_header setting 17535 cid 3.263_head oid 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 parent seq 0
2014-06-12 14:46:46.102640 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.102657 7f9d79960700 20 set_header setting 2232 cid meta oid 1fe27fea/pglog_3.263/0//-1 parent seq 0
2014-06-12 14:46:46.102670 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:46.106704 7f9d58d5b700  5 -- op tracker -- , seq: 748061, time: 2014-06-12 14:46:46.103620, event: header_read, op: osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.106724 7f9d58d5b700  5 -- op tracker -- , seq: 748061, time: 2014-06-12 14:46:46.103622, event: throttled, op: osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.106733 7f9d58d5b700  5 -- op tracker -- , seq: 748061, time: 2014-06-12 14:46:46.106646, event: all_read, op: osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.106741 7f9d58d5b700  5 -- op tracker -- , seq: 748061, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.106747 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:46.106754 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:46.106764 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34017000 prio 127 cost 1032999 latency 0.003143 osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.106796 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.106805 7f9d633d7700 10 osd.2 173 dequeue_op 0x34017000 prio 127 cost 1032999 latency 0.003184 osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.263( v 173'3184 (79'183,173'3184] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3182 active]
2014-06-12 14:46:46.106831 7f9d633d7700  5 -- op tracker -- , seq: 748061, time: 2014-06-12 14:46:46.106827, event: reached_pg, op: osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.106852 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3184 (79'183,173'3184] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3182 active] handle_message: 0x34017000
2014-06-12 14:46:46.106865 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3184 (79'183,173'3184] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3182 active] sub_op_modify trans 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 v 173'3185 (transaction) 174
2014-06-12 14:46:46.106881 7f9d633d7700  5 -- op tracker -- , seq: 748061, time: 2014-06-12 14:46:46.106878, event: started, op: osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.106898 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3184 (79'183,173'3184] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3182 active] append_log log((79'183,173'3184], crt=0'0) [173'3185 (173'3184) modify   44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 by client.4303.0:144887 2014-06-12 08:45:48.478015]
2014-06-12 14:46:46.106921 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3185 (79'183,173'3185] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3182 active] add_log_entry 173'3185 (173'3184) modify   44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 by client.4303.0:144887 2014-06-12 08:45:48.478015
2014-06-12 14:46:46.106942 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3185 (79'183,173'3185] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3182 active] append_log  adding 1 keys
2014-06-12 14:46:46.106953 7f9d633d7700 10 trim log((79'183,173'3185], crt=0'0) to 79'184
2014-06-12 14:46:46.106983 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3185, trimmed: 79'184
2014-06-12 14:46:46.107006 7f9d633d7700  5 -- op tracker -- , seq: 748061, time: 2014-06-12 14:46:46.107001, event: started, op: osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.107015 7f9d633d7700  5 queue_transactions existing osr(3.263 0x276d300)/0x276d300
2014-06-12 14:46:46.107020 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.107021 7f9d633d7700  5 queue_op 0xbc5f260 seq 2901 osr(3.263 0x276d300) 1034520 bytes   (queue has 2 ops and 2069040 bytes)
2014-06-12 14:46:46.107028 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.263( v 173'3185 (79'184,173'3185] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3182 active] do_sub_op osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.107042 7f9d633d7700 10 osd.2 173 dequeue_op 0x34017000 finish
2014-06-12 14:46:46.107045 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.107047 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.107212 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c47e0 (2 active)
2014-06-12 14:46:46.108167 7f9d79960700 10 _do_op 0xa2dade0 seq 2900 r = 0, finisher 0x29792a00 0
2014-06-12 14:46:46.108185 7f9d79960700 10 _finish_op 0xa2dade0 seq 2900 osr(3.263 0x276d300)/0x276d300
2014-06-12 14:46:46.108192 7f9d8016d700  5 _do_op 0xbc5f260 seq 2901 osr(3.263 0x276d300)/0x276d300 start
2014-06-12 14:46:46.108196 7f9d8016d700 10 _do_transaction on 0x2f3c10b0
2014-06-12 14:46:46.108202 7f9d8016d700 15 _omap_setkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.108344 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.108439 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.108504 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.108525 7f9d8016d700 15 _omap_rmkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.108533 7f9d8016d700 10 _omap_rmkeys meta/1fe27fea/pglog_3.263/0//-1 r = 0
2014-06-12 14:46:46.108539 7f9d8016d700 15 _omap_setkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.108557 7f9d8016d700 15 _write 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 2273280~1032192
2014-06-12 14:46:46.108712 7f9d8016d700 10 lookup_strip_header done  cid 3.263_head oid 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.108878 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:46.109746 7f9d7715b700  5 -- op tracker -- , seq: 748060, time: 2014-06-12 14:46:46.109717, event: commit_sent, op: osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.109765 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3185 (79'184,173'3185] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3182 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:46.109806 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:46.111166 7f9d58d5b700  5 -- op tracker -- , seq: 748062, time: 2014-06-12 14:46:46.106795, event: header_read, op: osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.111186 7f9d58d5b700  5 -- op tracker -- , seq: 748062, time: 2014-06-12 14:46:46.106796, event: throttled, op: osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.111194 7f9d58d5b700  5 -- op tracker -- , seq: 748062, time: 2014-06-12 14:46:46.111104, event: all_read, op: osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.111201 7f9d58d5b700  5 -- op tracker -- , seq: 748062, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.111208 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:46.111216 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:46:46.111227 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016800 prio 127 cost 889639 latency 0.004432 osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.111270 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.111278 7f9d61bd4700 10 osd.2 173 dequeue_op 0x34016800 prio 127 cost 889639 latency 0.004483 osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.263( v 173'3185 (79'184,173'3185] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3183 active]
2014-06-12 14:46:46.111309 7f9d61bd4700  5 -- op tracker -- , seq: 748062, time: 2014-06-12 14:46:46.111304, event: reached_pg, op: osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.111319 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3185 (79'184,173'3185] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3183 active] handle_message: 0x34016800
2014-06-12 14:46:46.111335 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3185 (79'184,173'3185] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3183 active] sub_op_modify trans 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 v 173'3186 (transaction) 174
2014-06-12 14:46:46.111358 7f9d61bd4700  5 -- op tracker -- , seq: 748062, time: 2014-06-12 14:46:46.111354, event: started, op: osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.111380 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3185 (79'184,173'3185] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3183 active] append_log log((79'184,173'3185], crt=0'0) [173'3186 (173'3185) modify   44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 by client.4303.0:144888 2014-06-12 08:45:48.479216]
2014-06-12 14:46:46.111405 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'184,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3183 active] add_log_entry 173'3186 (173'3185) modify   44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 by client.4303.0:144888 2014-06-12 08:45:48.479216
2014-06-12 14:46:46.111432 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'184,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3183 active] append_log  adding 1 keys
2014-06-12 14:46:46.111449 7f9d61bd4700 10 trim log((79'184,173'3186], crt=0'0) to 79'185
2014-06-12 14:46:46.111482 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3186, trimmed: 79'185
2014-06-12 14:46:46.111518 7f9d61bd4700  5 -- op tracker -- , seq: 748062, time: 2014-06-12 14:46:46.111514, event: started, op: osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.111540 7f9d61bd4700  5 queue_transactions existing osr(3.263 0x276d300)/0x276d300
2014-06-12 14:46:46.111550 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.111554 7f9d61bd4700  5 queue_op 0x6f9fe00 seq 2902 osr(3.263 0x276d300) 891160 bytes   (queue has 3 ops and 2960200 bytes)
2014-06-12 14:46:46.113965 7f9d8016d700 10 _generic_write 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 2273280~1032192 = 0
2014-06-12 14:46:46.114137 7f9d8016d700 15 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.114156 7f9d8016d700 10 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 = 0
2014-06-12 14:46:46.114165 7f9d8016d700 15 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.114173 7f9d8016d700 10 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 = 0
2014-06-12 14:46:46.114505 7f9d8016d700 20 set_header setting 17535 cid 3.263_head oid 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 parent seq 0
2014-06-12 14:46:46.114644 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.114665 7f9d8016d700 20 set_header setting 2232 cid meta oid 1fe27fea/pglog_3.263/0//-1 parent seq 0
2014-06-12 14:46:46.114676 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:46.114968 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c47e0 (1 active)
2014-06-12 14:46:46.114997 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.115044 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'185,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3183 active] do_sub_op osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.115070 7f9d61bd4700 10 osd.2 173 dequeue_op 0x34016800 finish
2014-06-12 14:46:46.115078 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.115082 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.115088 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c47e0 (2 active)
2014-06-12 14:46:46.115113 7f9d7795c700  5 -- op tracker -- , seq: 748060, time: 2014-06-12 14:46:46.115106, event: sub_op_applied, op: osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.115119 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'185,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3183 active] sub_op_modify_applied on 0x90f0a80 op osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.115135 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'185,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3183 active] op_applied on primary on version 173'3184
2014-06-12 14:46:46.115144 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'185,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3183 active] op_applied on replica on version 173'3184
2014-06-12 14:46:46.115165 7f9d7795c700  5 -- op tracker -- , seq: 748060, time: 2014-06-12 14:46:46.115161, event: done, op: osd_sub_op(client.4303.0:144886 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3184 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.125050 7f9d8016d700 10 _do_op 0xbc5f260 seq 2901 r = 0, finisher 0x66b1500 0
2014-06-12 14:46:46.125075 7f9d8016d700 10 _finish_op 0xbc5f260 seq 2901 osr(3.263 0x276d300)/0x276d300
2014-06-12 14:46:46.125086 7f9d7915f700  5 _do_op 0x6f9fe00 seq 2902 osr(3.263 0x276d300)/0x276d300 start
2014-06-12 14:46:46.125093 7f9d7915f700 10 _do_transaction on 0x58807f0
2014-06-12 14:46:46.125086 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c47e0 (1 active)
2014-06-12 14:46:46.125095 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.125091 7f9d7715b700  5 -- op tracker -- , seq: 748061, time: 2014-06-12 14:46:46.125082, event: commit_sent, op: osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.125102 7f9d7915f700 15 _omap_setkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.125098 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'185,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3183 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:46.125121 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:46.125171 7f9d7795c700  5 -- op tracker -- , seq: 748061, time: 2014-06-12 14:46:46.125165, event: sub_op_applied, op: osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.125181 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'185,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3184 active] sub_op_modify_applied on 0x2f3c0fc0 op osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.125198 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'185,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3184 active] op_applied on primary on version 173'3185
2014-06-12 14:46:46.125206 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'185,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3184 active] op_applied on replica on version 173'3185
2014-06-12 14:46:46.125221 7f9d7795c700  5 -- op tracker -- , seq: 748061, time: 2014-06-12 14:46:46.125219, event: done, op: osd_sub_op(client.4303.0:144887 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3185 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.125297 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.125323 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.125386 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.125477 7f9d7915f700 15 _omap_rmkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.125487 7f9d7915f700 10 _omap_rmkeys meta/1fe27fea/pglog_3.263/0//-1 r = 0
2014-06-12 14:46:46.125494 7f9d7915f700 15 _omap_setkeys meta/1fe27fea/pglog_3.263/0//-1
2014-06-12 14:46:46.125514 7f9d7915f700 15 _write 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 3305472~888832
2014-06-12 14:46:46.125687 7f9d7915f700 10 lookup_strip_header done  cid 3.263_head oid 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.125860 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:46.128558 7f9d7915f700 10 _generic_write 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 3305472~888832 = 0
2014-06-12 14:46:46.128624 7f9d7915f700 15 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.128638 7f9d7915f700 10 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 = 0
2014-06-12 14:46:46.128644 7f9d7915f700 15 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3
2014-06-12 14:46:46.128649 7f9d7915f700 10 _setattrs 3.263_head/44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 = 0
2014-06-12 14:46:46.128852 7f9d7915f700 20 set_header setting 17535 cid 3.263_head oid 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 parent seq 0
2014-06-12 14:46:46.128932 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.128945 7f9d7915f700 20 set_header setting 2232 cid meta oid 1fe27fea/pglog_3.263/0//-1 parent seq 0
2014-06-12 14:46:46.128953 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:46.132459 7f9d7915f700 10 _do_op 0x6f9fe00 seq 2902 r = 0, finisher 0x72f6940 0
2014-06-12 14:46:46.132490 7f9d7915f700 10 _finish_op 0x6f9fe00 seq 2902 osr(3.263 0x276d300)/0x276d300
2014-06-12 14:46:46.132498 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c47e0 (0 active)
2014-06-12 14:46:46.132509 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.132522 7f9d7795c700  5 -- op tracker -- , seq: 748062, time: 2014-06-12 14:46:46.132509, event: sub_op_applied, op: osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.132546 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'185,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3184 active] sub_op_modify_applied on 0x5880700 op osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.132573 7f9d7795c700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:46.132592 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'185,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3184 active] op_applied on primary on version 173'3186
2014-06-12 14:46:46.132601 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'185,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3184 active] op_applied on replica on version 173'3186
2014-06-12 14:46:46.132623 7f9d7715b700  5 -- op tracker -- , seq: 748062, time: 2014-06-12 14:46:46.132616, event: commit_sent, op: osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.132639 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.263( v 173'3186 (79'185,173'3186] local-les=142 n=320 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'3184 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:46:46.132657 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:46.132680 7f9d7715b700  5 -- op tracker -- , seq: 748062, time: 2014-06-12 14:46:46.132675, event: done, op: osd_sub_op(client.4303.0:144888 3.263 44e06a63/rbd_data.10a66b8b4567.0000000000001008/head//3 [] v 173'3186 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.154679 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:46.169150 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:46.169193 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:46.175635 7f9d47c58700  5 -- op tracker -- , seq: 748063, time: 2014-06-12 14:46:46.171229, event: header_read, op: osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.175649 7f9d47c58700  5 -- op tracker -- , seq: 748063, time: 2014-06-12 14:46:46.171233, event: throttled, op: osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.175660 7f9d47c58700  5 -- op tracker -- , seq: 748063, time: 2014-06-12 14:46:46.175600, event: all_read, op: osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.175670 7f9d47c58700  5 -- op tracker -- , seq: 748063, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.175678 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:46.175706 7f9d47c58700 15 osd.2 173 enqueue_op 0x124e3600 prio 63 cost 778240 latency 0.004477 osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.175738 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.175746 7f9d673df700 10 osd.2 173 dequeue_op 0x124e3600 prio 63 cost 778240 latency 0.004516 osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean]
2014-06-12 14:46:46.175771 7f9d673df700  5 -- op tracker -- , seq: 748063, time: 2014-06-12 14:46:46.175765, event: reached_pg, op: osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.175780 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:46.175796 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] handle_message: 0x124e3600
2014-06-12 14:46:46.175807 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] do_op osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:46.175835 7f9d673df700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_'
2014-06-12 14:46:46.198611 7f9d47c58700  5 -- op tracker -- , seq: 748064, time: 2014-06-12 14:46:46.196922, event: header_read, op: osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.198631 7f9d47c58700  5 -- op tracker -- , seq: 748064, time: 2014-06-12 14:46:46.196926, event: throttled, op: osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.198642 7f9d47c58700  5 -- op tracker -- , seq: 748064, time: 2014-06-12 14:46:46.198567, event: all_read, op: osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.198652 7f9d47c58700  5 -- op tracker -- , seq: 748064, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.198668 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:46.198698 7f9d47c58700 15 osd.2 173 enqueue_op 0x124e2600 prio 63 cost 516096 latency 0.001775 osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.198741 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:46.229221 7f9d56432700  5 -- op tracker -- , seq: 748065, time: 2014-06-12 14:46:46.226923, event: header_read, op: osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.229237 7f9d56432700  5 -- op tracker -- , seq: 748065, time: 2014-06-12 14:46:46.226926, event: throttled, op: osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.229246 7f9d56432700  5 -- op tracker -- , seq: 748065, time: 2014-06-12 14:46:46.229175, event: all_read, op: osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.229255 7f9d56432700  5 -- op tracker -- , seq: 748065, time: 0.000000, event: dispatched, op: osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.229260 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:46.229267 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:46.229275 7f9d56432700 15 osd.2 173 enqueue_op 0x28fd9600 prio 127 cost 1032999 latency 0.002352 osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.229307 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active)
2014-06-12 14:46:46.229313 7f9d67be0700 10 osd.2 173 dequeue_op 0x28fd9600 prio 127 cost 1032999 latency 0.002390 osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.71f( v 173'2703 (0'0,173'2703] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2702 active]
2014-06-12 14:46:46.229335 7f9d67be0700  5 -- op tracker -- , seq: 748065, time: 2014-06-12 14:46:46.229331, event: reached_pg, op: osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.229345 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2703 (0'0,173'2703] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2702 active] handle_message: 0x28fd9600
2014-06-12 14:46:46.229357 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2703 (0'0,173'2703] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2702 active] sub_op_modify trans 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 v 173'2704 (transaction) 174
2014-06-12 14:46:46.229381 7f9d67be0700  5 -- op tracker -- , seq: 748065, time: 2014-06-12 14:46:46.229377, event: started, op: osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.229398 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2703 (0'0,173'2703] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2702 active] append_log log((0'0,173'2703], crt=0'0) [173'2704 (173'2703) modify   4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 by client.4280.0:139906 2014-06-12 08:45:41.124662]
2014-06-12 14:46:46.229424 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2704 (0'0,173'2704] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2702 active] add_log_entry 173'2704 (173'2703) modify   4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 by client.4280.0:139906 2014-06-12 08:45:41.124662
2014-06-12 14:46:46.229446 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2704 (0'0,173'2704] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2702 active] append_log  adding 1 keys
2014-06-12 14:46:46.229482 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2704, trimmed: 
2014-06-12 14:46:46.229507 7f9d67be0700  5 -- op tracker -- , seq: 748065, time: 2014-06-12 14:46:46.229502, event: started, op: osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.229518 7f9d67be0700  5 queue_transactions existing osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:46.229522 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.229524 7f9d67be0700  5 queue_op 0x5d51140 seq 2536 osr(3.71f 0x276de50) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:46.229536 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d0240 (1 active)
2014-06-12 14:46:46.229532 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.71f( v 173'2704 (0'0,173'2704] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2702 active] do_sub_op osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.229544 7f9d7d968700  5 _do_op 0x5d51140 seq 2536 osr(3.71f 0x276de50)/0x276de50 start
2014-06-12 14:46:46.229547 7f9d67be0700 10 osd.2 173 dequeue_op 0x28fd9600 finish
2014-06-12 14:46:46.229550 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active)
2014-06-12 14:46:46.229551 7f9d7d968700 10 _do_transaction on 0x328a6630
2014-06-12 14:46:46.229553 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.229558 7f9d7d968700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:46.238882 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:46.238929 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.239004 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.239036 7f9d7d968700 15 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:46.239045 7f9d7d968700 10 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1 r = 0
2014-06-12 14:46:46.239052 7f9d7d968700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:46.239236 7f9d7d968700 15 _write 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 2424832~1032192
2014-06-12 14:46:46.239396 7f9d7d968700 10 lookup_strip_header done  cid 3.71f_head oid 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3
2014-06-12 14:46:46.239531 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:46.243891 7f9d7d968700 10 _generic_write 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 2424832~1032192 = 0
2014-06-12 14:46:46.244029 7f9d7d968700 15 _setattrs 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3
2014-06-12 14:46:46.244050 7f9d7d968700 10 _setattrs 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 = 0
2014-06-12 14:46:46.244059 7f9d7d968700 15 _setattrs 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3
2014-06-12 14:46:46.244071 7f9d7d968700 10 _setattrs 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 = 0
2014-06-12 14:46:46.244379 7f9d7d968700 20 set_header setting 15950 cid 3.71f_head oid 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 parent seq 0
2014-06-12 14:46:46.244490 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.244508 7f9d7d968700 20 set_header setting 2330 cid meta oid 1fe0985b/pglog_3.71f/0//-1 parent seq 0
2014-06-12 14:46:46.244523 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:46.249637 7f9d7d968700 10 _do_op 0x5d51140 seq 2536 r = 0, finisher 0x65e6960 0
2014-06-12 14:46:46.249669 7f9d7d968700 10 _finish_op 0x5d51140 seq 2536 osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:46.249688 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d0240 (0 active)
2014-06-12 14:46:46.249697 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.249692 7f9d7715b700  5 -- op tracker -- , seq: 748065, time: 2014-06-12 14:46:46.249679, event: commit_sent, op: osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.249704 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2704 (0'0,173'2704] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2702 active] sub_op_modify_commit on op osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:46.249734 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:46.249774 7f9d7795c700  5 -- op tracker -- , seq: 748065, time: 2014-06-12 14:46:46.249767, event: sub_op_applied, op: osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.249786 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2704 (0'0,173'2704] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2703 active] sub_op_modify_applied on 0x328a6540 op osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:46.249806 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2704 (0'0,173'2704] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2703 active] op_applied on primary on version 173'2704
2014-06-12 14:46:46.249817 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2704 (0'0,173'2704] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2703 active] op_applied on replica on version 173'2704
2014-06-12 14:46:46.249842 7f9d7795c700  5 -- op tracker -- , seq: 748065, time: 2014-06-12 14:46:46.249837, event: done, op: osd_sub_op(client.4280.0:139906 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:46.260843 7f9d673df700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_' = 0
2014-06-12 14:46:46.260869 7f9d673df700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset'
2014-06-12 14:46:46.261096 7f9d673df700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset' = 0
2014-06-12 14:46:46.261108 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] populate_obc_watchers 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.261126 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.261137 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] get_object_context: creating obc from disk: 0x29c1f080
2014-06-12 14:46:46.261147 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] get_object_context: 0x29c1f080 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 rwstate(none n=0 w=0) oi: 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'680 client.4316.0:41723 wrlock_by=unknown.0.0:0 dirty s 4194304 uv680) ssc: 0x1e15e460 snapset: 0=[]:[]+head
2014-06-12 14:46:46.261165 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] find_object_context 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 @head oi=267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'680 client.4316.0:41723 wrlock_by=unknown.0.0:0 dirty s 4194304 uv680)
2014-06-12 14:46:46.261190 7f9d673df700  5 -- op tracker -- , seq: 748063, time: 2014-06-12 14:46:46.261184, event: started, op: osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.261200 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] execute_ctx 0x55a3200
2014-06-12 14:46:46.261219 7f9d673df700  5 -- op tracker -- , seq: 748063, time: 2014-06-12 14:46:46.261214, event: started, op: osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.261228 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] do_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] ov 173'680 av 173'2652 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:46.261245 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] do_osd_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240]
2014-06-12 14:46:46.261256 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:46.261269 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] do_osd_op  write 0~778240
2014-06-12 14:46:46.261283 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:46.261294 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 snapset=0x1e15e498  snapc=0=[]
2014-06-12 14:46:46.261308 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:46.261319 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] finish_ctx 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 0x55a3200 op modify  
2014-06-12 14:46:46.261334 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean]  set mtime to 2014-06-12 08:45:48.549982
2014-06-12 14:46:46.261358 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean]  final snapset 0=[]:[]+head in 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.261378 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean]  zeroing write result code 0
2014-06-12 14:46:46.261392 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] new_repop rep_tid 236328 on osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.261410 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] issue_repop rep_tid 236328 o 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.261436 7f9d673df700  5 -- op tracker -- , seq: 748063, time: 2014-06-12 14:46:46.261430, event: waiting for subops from 29, op: osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.261457 7f9d673df700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:46.261480 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2651 (0'0,173'2651] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] append_log log((0'0,173'2651], crt=0'0) [173'2652 (173'680) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147271 2014-06-12 08:45:48.549982]
2014-06-12 14:46:46.261506 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] add_log_entry 173'2652 (173'680) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147271 2014-06-12 08:45:48.549982
2014-06-12 14:46:46.261530 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] append_log  adding 1 keys
2014-06-12 14:46:46.261565 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2652, trimmed: 
2014-06-12 14:46:46.261588 7f9d673df700  5 queue_transactions existing osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.261594 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.261596 7f9d673df700  5 queue_op 0x2b3e240 seq 2464 osr(3.7f8 0x49b4340) 780533 bytes   (queue has 1 ops and 780533 bytes)
2014-06-12 14:46:46.261602 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] eval_repop repgather(0x1ced32c0 173'2652 rep_tid=236328 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.261630 7f9d673df700 10 osd.2 173 dequeue_op 0x124e3600 finish
2014-06-12 14:46:46.261639 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:46.261643 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.261645 7f9d663dd700 10 osd.2 173 dequeue_op 0x124e2600 prio 63 cost 516096 latency 0.064723 osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean]
2014-06-12 14:46:46.261670 7f9d663dd700  5 -- op tracker -- , seq: 748064, time: 2014-06-12 14:46:46.261665, event: reached_pg, op: osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.261677 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:46.261689 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] handle_message: 0x124e2600
2014-06-12 14:46:46.261699 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] do_op osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:46.261720 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] get_object_context: found obc in cache: 0x29c1f080
2014-06-12 14:46:46.261730 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] get_object_context: 0x29c1f080 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 rwstate(write n=1 w=0) oi: 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2652 client.4316.0:147271 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2652) ssc: 0x1e15e460 snapset: 0=[]:[]+head
2014-06-12 14:46:46.261743 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] find_object_context 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 @head oi=267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2652 client.4316.0:147271 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2652)
2014-06-12 14:46:46.261771 7f9d663dd700  5 -- op tracker -- , seq: 748064, time: 2014-06-12 14:46:46.261766, event: started, op: osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.261776 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] execute_ctx 0x2e411e00
2014-06-12 14:46:46.261790 7f9d663dd700  5 -- op tracker -- , seq: 748064, time: 2014-06-12 14:46:46.261786, event: started, op: osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.261795 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] do_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] ov 173'2652 av 173'2653 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:46.261806 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] do_osd_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096]
2014-06-12 14:46:46.261816 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:46.261827 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] do_osd_op  write 778240~516096
2014-06-12 14:46:46.261839 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:46.261848 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 snapset=0x1e15e498  snapc=0=[]
2014-06-12 14:46:46.261858 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:46.261867 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] finish_ctx 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 0x2e411e00 op modify  
2014-06-12 14:46:46.261878 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean]  set mtime to 2014-06-12 08:45:48.576229
2014-06-12 14:46:46.261896 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean]  final snapset 0=[]:[]+head in 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.261911 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean]  zeroing write result code 0
2014-06-12 14:46:46.261919 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] new_repop rep_tid 236329 on osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.261933 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] issue_repop rep_tid 236329 o 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.261952 7f9d663dd700  5 -- op tracker -- , seq: 748064, time: 2014-06-12 14:46:46.261947, event: waiting for subops from 29, op: osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.262053 7f9d663dd700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:46.262070 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2652 (0'0,173'2652] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] append_log log((0'0,173'2652], crt=0'0) [173'2653 (173'2652) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147272 2014-06-12 08:45:48.576229]
2014-06-12 14:46:46.262091 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] add_log_entry 173'2653 (173'2652) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147272 2014-06-12 08:45:48.576229
2014-06-12 14:46:46.262110 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] append_log  adding 1 keys
2014-06-12 14:46:46.262138 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2653, trimmed: 
2014-06-12 14:46:46.262157 7f9d663dd700  5 queue_transactions existing osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.262161 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.262162 7f9d663dd700  5 queue_op 0x6e61a40 seq 2465 osr(3.7f8 0x49b4340) 518389 bytes   (queue has 2 ops and 1298922 bytes)
2014-06-12 14:46:46.262169 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] eval_repop repgather(0x29702240 173'2653 rep_tid=236329 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.262187 7f9d663dd700 10 osd.2 173 dequeue_op 0x124e2600 finish
2014-06-12 14:46:46.262191 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.262194 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.262205 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541cb40 (1 active)
2014-06-12 14:46:46.262216 7f9d7c165700  5 _do_op 0x2b3e240 seq 2464 osr(3.7f8 0x49b4340)/0x49b4340 start
2014-06-12 14:46:46.262221 7f9d7c165700 10 _do_transaction on 0x52378c0
2014-06-12 14:46:46.262227 7f9d7c165700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.262363 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.262387 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.262445 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.262463 7f9d7c165700 15 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.262468 7f9d7c165700 10 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1 r = 0
2014-06-12 14:46:46.262473 7f9d7c165700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.262586 7f9d7c165700 15 _write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 0~778240
2014-06-12 14:46:46.262733 7f9d7c165700 10 lookup_strip_header done  cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.262728 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541cb40 (2 active)
2014-06-12 14:46:46.262852 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:46.265938 7f9d7c165700 10 _generic_write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 0~778240 = 0
2014-06-12 14:46:46.266030 7f9d7c165700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.266051 7f9d7c165700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.266062 7f9d7c165700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.266070 7f9d7c165700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.266326 7f9d7c165700 20 set_header setting 17472 cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 parent seq 0
2014-06-12 14:46:46.266412 7f9d7c165700 20 set_header setting 2158 cid meta oid 1fe70fd5/pglog_3.7f8/0//-1 parent seq 0
2014-06-12 14:46:46.266428 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.266437 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:46.270335 7f9d7c165700 10 _do_op 0x2b3e240 seq 2464 r = 0, finisher 0x6614480 0x1a3f05c0
2014-06-12 14:46:46.270362 7f9d7c165700 10 _finish_op 0x2b3e240 seq 2464 osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.270373 7f9d8096e700  5 _do_op 0x6e61a40 seq 2465 osr(3.7f8 0x49b4340)/0x49b4340 start
2014-06-12 14:46:46.270379 7f9d8096e700 10 _do_transaction on 0x234c780
2014-06-12 14:46:46.270376 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541cb40 (1 active)
2014-06-12 14:46:46.270385 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.270389 7f9d8096e700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.270369 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] op_commit: 236328
2014-06-12 14:46:46.270405 7f9d7715b700  5 -- op tracker -- , seq: 748063, time: 2014-06-12 14:46:46.270394, event: op_commit, op: osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.270438 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] op_applied: 236328
2014-06-12 14:46:46.270461 7f9d7795c700  5 -- op tracker -- , seq: 748063, time: 2014-06-12 14:46:46.270454, event: op_applied, op: osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.270480 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] op_applied on primary on version 173'2652
2014-06-12 14:46:46.270537 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.270556 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.270615 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.270632 7f9d8096e700 15 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.270637 7f9d8096e700 10 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1 r = 0
2014-06-12 14:46:46.270644 7f9d8096e700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.270658 7f9d8096e700 15 _write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 778240~516096
2014-06-12 14:46:46.270815 7f9d8096e700 10 lookup_strip_header done  cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.270906 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:46.272419 7f9d8096e700 10 _generic_write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 778240~516096 = 0
2014-06-12 14:46:46.272467 7f9d8096e700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.272483 7f9d8096e700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.272492 7f9d8096e700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.272502 7f9d8096e700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.272650 7f9d8096e700 20 set_header setting 17472 cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 parent seq 0
2014-06-12 14:46:46.272706 7f9d8096e700 20 set_header setting 2158 cid meta oid 1fe70fd5/pglog_3.7f8/0//-1 parent seq 0
2014-06-12 14:46:46.272733 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.272744 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:46.273414 7f9d5501e700  5 -- op tracker -- , seq: 748066, time: 2014-06-12 14:46:46.273364, event: header_read, op: osd_sub_op_reply(client.4316.0:147271 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.273427 7f9d5501e700  5 -- op tracker -- , seq: 748066, time: 2014-06-12 14:46:46.273366, event: throttled, op: osd_sub_op_reply(client.4316.0:147271 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.273434 7f9d5501e700  5 -- op tracker -- , seq: 748066, time: 2014-06-12 14:46:46.273379, event: all_read, op: osd_sub_op_reply(client.4316.0:147271 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.273442 7f9d5501e700  5 -- op tracker -- , seq: 748066, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147271 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.273447 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147271 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:46.273453 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:46.273461 7f9d5501e700 15 osd.2 173 enqueue_op 0x28dd0d00 prio 196 cost 0 latency 0.000096 osd_sub_op_reply(client.4316.0:147271 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:46.273500 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.273508 7f9d65bdc700 10 osd.2 173 dequeue_op 0x28dd0d00 prio 196 cost 0 latency 0.000144 osd_sub_op_reply(client.4316.0:147271 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 pg pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean]
2014-06-12 14:46:46.273530 7f9d65bdc700  5 -- op tracker -- , seq: 748066, time: 2014-06-12 14:46:46.273527, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147271 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.273538 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] handle_message: 0x28dd0d00
2014-06-12 14:46:46.273555 7f9d65bdc700  5 -- op tracker -- , seq: 748066, time: 2014-06-12 14:46:46.273552, event: started, op: osd_sub_op_reply(client.4316.0:147271 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.273561 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] sub_op_modify_reply: tid 236328 op  ack_type 4 from 29
2014-06-12 14:46:46.273580 7f9d65bdc700  5 -- op tracker -- , seq: 748063, time: 2014-06-12 14:46:46.273573, event: sub_op_commit_rec, op: osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.273590 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] repop_all_applied: repop tid 236328 all applied 
2014-06-12 14:46:46.273602 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] eval_repop repgather(0x1ced32c0 173'2652 rep_tid=236328 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.273627 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean]  sending ack on repgather(0x1ced32c0 173'2652 rep_tid=236328 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x191a0000
2014-06-12 14:46:46.273656 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2651 crt=0'0 lcod 173'2650 mlcod 173'2650 active+clean] repop_all_committed: repop tid 236328 all committed 
2014-06-12 14:46:46.273668 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2650 active+clean] eval_repop repgather(0x1ced32c0 173'2652 rep_tid=236328 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.273684 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2650 active+clean] log_op_stats osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 inb 778519 outb 0 rlat 0.102426 lat 0.102454
2014-06-12 14:46:46.273703 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2650 active+clean] publish_stats_to_osd 173:3427
2014-06-12 14:46:46.273722 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2650 active+clean]  sending commit on repgather(0x1ced32c0 173'2652 rep_tid=236328 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x30af6780
2014-06-12 14:46:46.273755 7f9d65bdc700  5 -- op tracker -- , seq: 748063, time: 2014-06-12 14:46:46.273750, event: commit_sent, op: osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.273766 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2650 active+clean]  requeue_ops 
2014-06-12 14:46:46.273779 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean]  removing repgather(0x1ced32c0 173'2652 rep_tid=236328 committed?=1 applied?=1 op=osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.273796 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean]    q front is repgather(0x1ced32c0 173'2652 rep_tid=236328 committed?=1 applied?=1 op=osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.273811 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean] remove_repop repgather(0x1ced32c0 173'2652 rep_tid=236328 committed?=1 applied?=1 op=osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.273831 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean]  requeue_ops 
2014-06-12 14:46:46.273850 7f9d65bdc700  5 -- op tracker -- , seq: 748063, time: 2014-06-12 14:46:46.273845, event: done, op: osd_op(client.4316.0:147271 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~778240] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.273871 7f9d65bdc700 10 osd.2 173 dequeue_op 0x28dd0d00 finish
2014-06-12 14:46:46.273880 7f9d65bdc700  5 -- op tracker -- , seq: 748066, time: 2014-06-12 14:46:46.273876, event: done, op: osd_sub_op_reply(client.4316.0:147271 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.273889 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.273892 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.275089 7f9d8096e700 10 _do_op 0x6e61a40 seq 2465 r = 0, finisher 0x6640a60 0x1e6266c0
2014-06-12 14:46:46.275106 7f9d8096e700 10 _finish_op 0x6e61a40 seq 2465 osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.275116 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541cb40 (0 active)
2014-06-12 14:46:46.275124 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.275111 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean] op_commit: 236329
2014-06-12 14:46:46.275138 7f9d7715b700  5 -- op tracker -- , seq: 748064, time: 2014-06-12 14:46:46.275130, event: op_commit, op: osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.275154 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean] op_applied: 236329
2014-06-12 14:46:46.275181 7f9d7795c700  5 -- op tracker -- , seq: 748064, time: 2014-06-12 14:46:46.275173, event: op_applied, op: osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.275192 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean] op_applied on primary on version 173'2653
2014-06-12 14:46:46.279538 7f9d5501e700  5 -- op tracker -- , seq: 748067, time: 2014-06-12 14:46:46.279482, event: header_read, op: osd_sub_op_reply(client.4316.0:147272 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.279548 7f9d5501e700  5 -- op tracker -- , seq: 748067, time: 2014-06-12 14:46:46.279485, event: throttled, op: osd_sub_op_reply(client.4316.0:147272 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.279555 7f9d5501e700  5 -- op tracker -- , seq: 748067, time: 2014-06-12 14:46:46.279500, event: all_read, op: osd_sub_op_reply(client.4316.0:147272 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.279563 7f9d5501e700  5 -- op tracker -- , seq: 748067, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147272 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.279569 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147272 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:46.279576 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:46.279586 7f9d5501e700 15 osd.2 173 enqueue_op 0xe20ba00 prio 196 cost 0 latency 0.000103 osd_sub_op_reply(client.4316.0:147272 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:46.279619 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.279629 7f9d62bd6700 10 osd.2 173 dequeue_op 0xe20ba00 prio 196 cost 0 latency 0.000146 osd_sub_op_reply(client.4316.0:147272 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 pg pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean]
2014-06-12 14:46:46.279650 7f9d62bd6700  5 -- op tracker -- , seq: 748067, time: 2014-06-12 14:46:46.279646, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147272 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.279664 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean] handle_message: 0xe20ba00
2014-06-12 14:46:46.279680 7f9d62bd6700  5 -- op tracker -- , seq: 748067, time: 2014-06-12 14:46:46.279677, event: started, op: osd_sub_op_reply(client.4316.0:147272 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.279690 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean] sub_op_modify_reply: tid 236329 op  ack_type 4 from 29
2014-06-12 14:46:46.279712 7f9d62bd6700  5 -- op tracker -- , seq: 748064, time: 2014-06-12 14:46:46.279704, event: sub_op_commit_rec, op: osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.279719 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean] repop_all_applied: repop tid 236329 all applied 
2014-06-12 14:46:46.279736 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean] eval_repop repgather(0x29702240 173'2653 rep_tid=236329 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.279758 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean]  sending ack on repgather(0x29702240 173'2653 rep_tid=236329 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x16cd3400
2014-06-12 14:46:46.279793 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2652 crt=0'0 lcod 173'2651 mlcod 173'2651 active+clean] repop_all_committed: repop tid 236329 all committed 
2014-06-12 14:46:46.279803 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2651 active+clean] eval_repop repgather(0x29702240 173'2653 rep_tid=236329 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.279819 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2651 active+clean] log_op_stats osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.082869 lat 0.082896
2014-06-12 14:46:46.279835 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2651 active+clean] publish_stats_to_osd 173:3428
2014-06-12 14:46:46.279847 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2651 active+clean]  sending commit on repgather(0x29702240 173'2653 rep_tid=236329 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x1b775900
2014-06-12 14:46:46.279884 7f9d62bd6700  5 -- op tracker -- , seq: 748064, time: 2014-06-12 14:46:46.279878, event: commit_sent, op: osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.279892 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2651 active+clean]  requeue_ops 
2014-06-12 14:46:46.279902 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean]  removing repgather(0x29702240 173'2653 rep_tid=236329 committed?=1 applied?=1 op=osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.279917 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean]    q front is repgather(0x29702240 173'2653 rep_tid=236329 committed?=1 applied?=1 op=osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.279930 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] remove_repop repgather(0x29702240 173'2653 rep_tid=236329 committed?=1 applied?=1 op=osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.279944 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean]  requeue_ops 
2014-06-12 14:46:46.279969 7f9d62bd6700  5 -- op tracker -- , seq: 748064, time: 2014-06-12 14:46:46.279964, event: done, op: osd_op(client.4316.0:147272 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 778240~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.279987 7f9d62bd6700 10 osd.2 173 dequeue_op 0xe20ba00 finish
2014-06-12 14:46:46.279995 7f9d62bd6700  5 -- op tracker -- , seq: 748067, time: 2014-06-12 14:46:46.279991, event: done, op: osd_sub_op_reply(client.4316.0:147272 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.280005 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.280008 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.283575 7f9d47c58700  5 -- op tracker -- , seq: 748068, time: 2014-06-12 14:46:46.282240, event: header_read, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.283588 7f9d47c58700  5 -- op tracker -- , seq: 748068, time: 2014-06-12 14:46:46.282242, event: throttled, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.283599 7f9d47c58700  5 -- op tracker -- , seq: 748068, time: 2014-06-12 14:46:46.283547, event: all_read, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.283609 7f9d47c58700  5 -- op tracker -- , seq: 748068, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.283617 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:46.283643 7f9d47c58700 15 osd.2 173 enqueue_op 0x17c2b100 prio 63 cost 516096 latency 0.001402 osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.283672 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.283679 7f9d683e1700 10 osd.2 173 dequeue_op 0x17c2b100 prio 63 cost 516096 latency 0.001439 osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean]
2014-06-12 14:46:46.283704 7f9d683e1700  5 -- op tracker -- , seq: 748068, time: 2014-06-12 14:46:46.283699, event: reached_pg, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.283714 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:46.283730 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] handle_message: 0x17c2b100
2014-06-12 14:46:46.283741 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] do_op osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:46.283767 7f9d683e1700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_'
2014-06-12 14:46:46.283955 7f9d683e1700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_' = 0
2014-06-12 14:46:46.283973 7f9d683e1700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset'
2014-06-12 14:46:46.284055 7f9d683e1700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset' = 0
2014-06-12 14:46:46.284066 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] populate_obc_watchers 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.284081 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.284091 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] get_object_context: creating obc from disk: 0xa546000
2014-06-12 14:46:46.284103 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] get_object_context: 0xa546000 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 rwstate(none n=0 w=0) oi: 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2653 client.4316.0:147272 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2653) ssc: 0x1e15efc0 snapset: 0=[]:[]+head
2014-06-12 14:46:46.284121 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] find_object_context 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 @head oi=267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2653 client.4316.0:147272 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2653)
2014-06-12 14:46:46.284150 7f9d683e1700  5 -- op tracker -- , seq: 748068, time: 2014-06-12 14:46:46.284143, event: started, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.284157 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] execute_ctx 0x6245600
2014-06-12 14:46:46.284176 7f9d683e1700  5 -- op tracker -- , seq: 748068, time: 2014-06-12 14:46:46.284170, event: started, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.284185 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] do_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] ov 173'2653 av 173'2654 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:46.284201 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] do_osd_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096]
2014-06-12 14:46:46.284214 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:46.284229 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] do_osd_op  write 1294336~516096
2014-06-12 14:46:46.284247 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:46.284260 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 snapset=0x1e15eff8  snapc=0=[]
2014-06-12 14:46:46.284275 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:46.284288 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] finish_ctx 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 0x6245600 op modify  
2014-06-12 14:46:46.284302 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean]  set mtime to 2014-06-12 08:45:48.661768
2014-06-12 14:46:46.284323 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean]  final snapset 0=[]:[]+head in 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.284342 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean]  zeroing write result code 0
2014-06-12 14:46:46.284355 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] new_repop rep_tid 236330 on osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.284373 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] issue_repop rep_tid 236330 o 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.284400 7f9d683e1700  5 -- op tracker -- , seq: 748068, time: 2014-06-12 14:46:46.284394, event: waiting for subops from 29, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.284421 7f9d683e1700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:46.284442 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2653 (0'0,173'2653] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] append_log log((0'0,173'2653], crt=0'0) [173'2654 (173'2653) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147273 2014-06-12 08:45:48.661768]
2014-06-12 14:46:46.284462 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] add_log_entry 173'2654 (173'2653) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147273 2014-06-12 08:45:48.661768
2014-06-12 14:46:46.284484 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] append_log  adding 1 keys
2014-06-12 14:46:46.284517 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2654, trimmed: 
2014-06-12 14:46:46.284540 7f9d683e1700  5 queue_transactions existing osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.284545 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.284547 7f9d683e1700  5 queue_op 0x6835620 seq 2466 osr(3.7f8 0x49b4340) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:46.284558 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] eval_repop repgather(0xd5e32c0 173'2654 rep_tid=236330 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.284578 7f9d683e1700 10 osd.2 173 dequeue_op 0x17c2b100 finish
2014-06-12 14:46:46.284582 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.284585 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.284870 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541cb40 (1 active)
2014-06-12 14:46:46.284880 7f9d7c966700  5 _do_op 0x6835620 seq 2466 osr(3.7f8 0x49b4340)/0x49b4340 start
2014-06-12 14:46:46.284884 7f9d7c966700 10 _do_transaction on 0x5390d80
2014-06-12 14:46:46.284890 7f9d7c966700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.284990 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.285011 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.285073 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.285092 7f9d7c966700 15 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.285100 7f9d7c966700 10 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1 r = 0
2014-06-12 14:46:46.285106 7f9d7c966700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.285119 7f9d7c966700 15 _write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 1294336~516096
2014-06-12 14:46:46.285262 7f9d7c966700 10 lookup_strip_header done  cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.285284 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:46.287341 7f9d7c966700 10 _generic_write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 1294336~516096 = 0
2014-06-12 14:46:46.287393 7f9d7c966700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.287410 7f9d7c966700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.287418 7f9d7c966700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.287428 7f9d7c966700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.287622 7f9d7c966700 20 set_header setting 17472 cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 parent seq 0
2014-06-12 14:46:46.287686 7f9d7c966700 20 set_header setting 2158 cid meta oid 1fe70fd5/pglog_3.7f8/0//-1 parent seq 0
2014-06-12 14:46:46.287703 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.287714 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:46.290169 7f9d7c966700 10 _do_op 0x6835620 seq 2466 r = 0, finisher 0x2e1cf080 0x2ea2400
2014-06-12 14:46:46.290189 7f9d7c966700 10 _finish_op 0x6835620 seq 2466 osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.290198 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541cb40 (0 active)
2014-06-12 14:46:46.290202 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.290195 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] op_commit: 236330
2014-06-12 14:46:46.290225 7f9d7715b700  5 -- op tracker -- , seq: 748068, time: 2014-06-12 14:46:46.290217, event: op_commit, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.290244 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] op_applied: 236330
2014-06-12 14:46:46.290268 7f9d7795c700  5 -- op tracker -- , seq: 748068, time: 2014-06-12 14:46:46.290261, event: op_applied, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.290278 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] op_applied on primary on version 173'2654
2014-06-12 14:46:46.297235 7f9d5501e700  5 -- op tracker -- , seq: 748069, time: 2014-06-12 14:46:46.297164, event: header_read, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.297247 7f9d5501e700  5 -- op tracker -- , seq: 748069, time: 2014-06-12 14:46:46.297167, event: throttled, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.297254 7f9d5501e700  5 -- op tracker -- , seq: 748069, time: 2014-06-12 14:46:46.297185, event: all_read, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.297261 7f9d5501e700  5 -- op tracker -- , seq: 748069, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.297267 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:46.297273 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:46.297282 7f9d5501e700 15 osd.2 173 enqueue_op 0x6f4b400 prio 196 cost 0 latency 0.000118 osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0) v2
2014-06-12 14:46:46.297320 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.297330 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x6f4b400 prio 196 cost 0 latency 0.000166 osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0) v2 pg pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean]
2014-06-12 14:46:46.297354 7f9d6b3e7700  5 -- op tracker -- , seq: 748069, time: 2014-06-12 14:46:46.297351, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.297369 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] handle_message: 0x6f4b400
2014-06-12 14:46:46.297385 7f9d6b3e7700  5 -- op tracker -- , seq: 748069, time: 2014-06-12 14:46:46.297382, event: started, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.297390 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] sub_op_modify_reply: tid 236330 op  ack_type 1 from 29
2014-06-12 14:46:46.297411 7f9d6b3e7700  5 -- op tracker -- , seq: 748068, time: 2014-06-12 14:46:46.297402, event: sub_op_applied_rec, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.297419 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] repop_all_applied: repop tid 236330 all applied 
2014-06-12 14:46:46.297429 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] eval_repop repgather(0xd5e32c0 173'2654 rep_tid=236330 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.297445 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean]  sending ack on repgather(0xd5e32c0 173'2654 rep_tid=236330 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x26e56280
2014-06-12 14:46:46.297477 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x6f4b400 finish
2014-06-12 14:46:46.297484 7f9d6b3e7700  5 -- op tracker -- , seq: 748069, time: 2014-06-12 14:46:46.297480, event: done, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.297498 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.297501 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.298682 7f9d5501e700  5 -- op tracker -- , seq: 748070, time: 2014-06-12 14:46:46.298626, event: header_read, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.298696 7f9d5501e700  5 -- op tracker -- , seq: 748070, time: 2014-06-12 14:46:46.298628, event: throttled, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.298704 7f9d5501e700  5 -- op tracker -- , seq: 748070, time: 2014-06-12 14:46:46.298646, event: all_read, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.298711 7f9d5501e700  5 -- op tracker -- , seq: 748070, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.298717 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:46.298722 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:46.298731 7f9d5501e700 15 osd.2 173 enqueue_op 0x2519ce00 prio 196 cost 0 latency 0.000105 osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:46.298774 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.298782 7f9d693e3700 10 osd.2 173 dequeue_op 0x2519ce00 prio 196 cost 0 latency 0.000156 osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 pg pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean]
2014-06-12 14:46:46.298803 7f9d693e3700  5 -- op tracker -- , seq: 748070, time: 2014-06-12 14:46:46.298800, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.298812 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] handle_message: 0x2519ce00
2014-06-12 14:46:46.298828 7f9d693e3700  5 -- op tracker -- , seq: 748070, time: 2014-06-12 14:46:46.298825, event: started, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.298834 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] sub_op_modify_reply: tid 236330 op  ack_type 4 from 29
2014-06-12 14:46:46.298856 7f9d693e3700  5 -- op tracker -- , seq: 748068, time: 2014-06-12 14:46:46.298847, event: sub_op_commit_rec, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.298864 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2653 crt=0'0 lcod 173'2652 mlcod 173'2652 active+clean] repop_all_committed: repop tid 236330 all committed 
2014-06-12 14:46:46.298875 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2652 active+clean] eval_repop repgather(0xd5e32c0 173'2654 rep_tid=236330 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.298892 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2652 active+clean] log_op_stats osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.015235 lat 0.016651
2014-06-12 14:46:46.298910 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2652 active+clean] publish_stats_to_osd 173:3429
2014-06-12 14:46:46.298923 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2652 active+clean]  sending commit on repgather(0xd5e32c0 173'2654 rep_tid=236330 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x18419900
2014-06-12 14:46:46.298959 7f9d693e3700  5 -- op tracker -- , seq: 748068, time: 2014-06-12 14:46:46.298953, event: commit_sent, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.298967 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2652 active+clean]  requeue_ops 
2014-06-12 14:46:46.298979 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean]  removing repgather(0xd5e32c0 173'2654 rep_tid=236330 committed?=1 applied?=1 op=osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.298994 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean]    q front is repgather(0xd5e32c0 173'2654 rep_tid=236330 committed?=1 applied?=1 op=osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.299008 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] remove_repop repgather(0xd5e32c0 173'2654 rep_tid=236330 committed?=1 applied?=1 op=osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.299021 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean]  requeue_ops 
2014-06-12 14:46:46.299049 7f9d693e3700  5 -- op tracker -- , seq: 748068, time: 2014-06-12 14:46:46.299044, event: done, op: osd_op(client.4316.0:147273 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1294336~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.299070 7f9d693e3700 10 osd.2 173 dequeue_op 0x2519ce00 finish
2014-06-12 14:46:46.299077 7f9d693e3700  5 -- op tracker -- , seq: 748070, time: 2014-06-12 14:46:46.299073, event: done, op: osd_sub_op_reply(client.4316.0:147273 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.299088 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.299091 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.302428 7f9d47c58700  5 -- op tracker -- , seq: 748071, time: 2014-06-12 14:46:46.301218, event: header_read, op: osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.302443 7f9d47c58700  5 -- op tracker -- , seq: 748071, time: 2014-06-12 14:46:46.301221, event: throttled, op: osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.302453 7f9d47c58700  5 -- op tracker -- , seq: 748071, time: 2014-06-12 14:46:46.302384, event: all_read, op: osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.302464 7f9d47c58700  5 -- op tracker -- , seq: 748071, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.302473 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:46.302505 7f9d47c58700 15 osd.2 173 enqueue_op 0x17c2a800 prio 63 cost 516096 latency 0.001287 osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.302539 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.302551 7f9d66bde700 10 osd.2 173 dequeue_op 0x17c2a800 prio 63 cost 516096 latency 0.001333 osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean]
2014-06-12 14:46:46.302581 7f9d66bde700  5 -- op tracker -- , seq: 748071, time: 2014-06-12 14:46:46.302575, event: reached_pg, op: osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.302593 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:46.302611 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] handle_message: 0x17c2a800
2014-06-12 14:46:46.302623 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] do_op osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:46.302649 7f9d66bde700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_'
2014-06-12 14:46:46.306819 7f9d66bde700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_' = 0
2014-06-12 14:46:46.306845 7f9d66bde700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset'
2014-06-12 14:46:46.306939 7f9d66bde700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset' = 0
2014-06-12 14:46:46.306950 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] populate_obc_watchers 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.306967 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.306978 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] get_object_context: creating obc from disk: 0x2060edc0
2014-06-12 14:46:46.306989 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] get_object_context: 0x2060edc0 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 rwstate(none n=0 w=0) oi: 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2654 client.4316.0:147273 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2654) ssc: 0x16ee5180 snapset: 0=[]:[]+head
2014-06-12 14:46:46.307006 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] find_object_context 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 @head oi=267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2654 client.4316.0:147273 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2654)
2014-06-12 14:46:46.307041 7f9d66bde700  5 -- op tracker -- , seq: 748071, time: 2014-06-12 14:46:46.307032, event: started, op: osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.307049 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] execute_ctx 0x2a83200
2014-06-12 14:46:46.307068 7f9d66bde700  5 -- op tracker -- , seq: 748071, time: 2014-06-12 14:46:46.307062, event: started, op: osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.307076 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] do_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] ov 173'2654 av 173'2655 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:46.307093 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] do_osd_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096]
2014-06-12 14:46:46.307107 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:46.307121 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] do_osd_op  write 1810432~516096
2014-06-12 14:46:46.307139 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:46.307153 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 snapset=0x16ee51b8  snapc=0=[]
2014-06-12 14:46:46.307167 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:46.307184 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] finish_ctx 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 0x2a83200 op modify  
2014-06-12 14:46:46.307198 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean]  set mtime to 2014-06-12 08:45:48.680745
2014-06-12 14:46:46.307220 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean]  final snapset 0=[]:[]+head in 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.307240 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean]  zeroing write result code 0
2014-06-12 14:46:46.307253 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] new_repop rep_tid 236331 on osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.307271 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] issue_repop rep_tid 236331 o 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.307298 7f9d66bde700  5 -- op tracker -- , seq: 748071, time: 2014-06-12 14:46:46.307293, event: waiting for subops from 29, op: osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.307319 7f9d66bde700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:46.307340 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2654 (0'0,173'2654] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] append_log log((0'0,173'2654], crt=0'0) [173'2655 (173'2654) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147274 2014-06-12 08:45:48.680745]
2014-06-12 14:46:46.307360 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2654 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] add_log_entry 173'2655 (173'2654) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147274 2014-06-12 08:45:48.680745
2014-06-12 14:46:46.307381 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2654 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] append_log  adding 1 keys
2014-06-12 14:46:46.307415 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2655, trimmed: 
2014-06-12 14:46:46.307437 7f9d66bde700  5 queue_transactions existing osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.307442 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.307444 7f9d66bde700  5 queue_op 0x216fe60 seq 2467 osr(3.7f8 0x49b4340) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:46.307451 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2654 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] eval_repop repgather(0x2010e3c0 173'2655 rep_tid=236331 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.307474 7f9d66bde700 10 osd.2 173 dequeue_op 0x17c2a800 finish
2014-06-12 14:46:46.307480 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.307484 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.307493 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541cb40 (1 active)
2014-06-12 14:46:46.307498 7f9d81970700  5 _do_op 0x216fe60 seq 2467 osr(3.7f8 0x49b4340)/0x49b4340 start
2014-06-12 14:46:46.307501 7f9d81970700 10 _do_transaction on 0x13b43740
2014-06-12 14:46:46.307507 7f9d81970700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.307589 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.307616 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.307669 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.307772 7f9d81970700 15 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.307780 7f9d81970700 10 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1 r = 0
2014-06-12 14:46:46.307787 7f9d81970700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.307804 7f9d81970700 15 _write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 1810432~516096
2014-06-12 14:46:46.307944 7f9d81970700 10 lookup_strip_header done  cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.308079 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:46.309387 7f9d81970700 10 _generic_write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 1810432~516096 = 0
2014-06-12 14:46:46.309432 7f9d81970700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.309445 7f9d81970700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.309452 7f9d81970700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.309460 7f9d81970700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.309606 7f9d81970700 20 set_header setting 17472 cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 parent seq 0
2014-06-12 14:46:46.309672 7f9d81970700 20 set_header setting 2158 cid meta oid 1fe70fd5/pglog_3.7f8/0//-1 parent seq 0
2014-06-12 14:46:46.309689 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.309699 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:46.311658 7f9d81970700 10 _do_op 0x216fe60 seq 2467 r = 0, finisher 0x98f4ba0 0xba9eac0
2014-06-12 14:46:46.311678 7f9d81970700 10 _finish_op 0x216fe60 seq 2467 osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.311686 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541cb40 (0 active)
2014-06-12 14:46:46.311690 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.311701 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2654 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] op_commit: 236331
2014-06-12 14:46:46.311733 7f9d7715b700  5 -- op tracker -- , seq: 748071, time: 2014-06-12 14:46:46.311722, event: op_commit, op: osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.311750 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2654 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] op_applied: 236331
2014-06-12 14:46:46.311771 7f9d7795c700  5 -- op tracker -- , seq: 748071, time: 2014-06-12 14:46:46.311765, event: op_applied, op: osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.311779 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2654 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] op_applied on primary on version 173'2655
2014-06-12 14:46:46.319433 7f9d5501e700  5 -- op tracker -- , seq: 748072, time: 2014-06-12 14:46:46.319374, event: header_read, op: osd_sub_op_reply(client.4316.0:147274 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.319448 7f9d5501e700  5 -- op tracker -- , seq: 748072, time: 2014-06-12 14:46:46.319376, event: throttled, op: osd_sub_op_reply(client.4316.0:147274 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.319456 7f9d5501e700  5 -- op tracker -- , seq: 748072, time: 2014-06-12 14:46:46.319391, event: all_read, op: osd_sub_op_reply(client.4316.0:147274 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.319464 7f9d5501e700  5 -- op tracker -- , seq: 748072, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147274 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.319470 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147274 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:46.319477 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:46.319485 7f9d5501e700 15 osd.2 173 enqueue_op 0x160e4100 prio 196 cost 0 latency 0.000111 osd_sub_op_reply(client.4316.0:147274 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:46.319515 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.319526 7f9d64bda700 10 osd.2 173 dequeue_op 0x160e4100 prio 196 cost 0 latency 0.000152 osd_sub_op_reply(client.4316.0:147274 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 pg pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2654 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean]
2014-06-12 14:46:46.319549 7f9d64bda700  5 -- op tracker -- , seq: 748072, time: 2014-06-12 14:46:46.319546, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147274 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.319558 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2654 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] handle_message: 0x160e4100
2014-06-12 14:46:46.319573 7f9d64bda700  5 -- op tracker -- , seq: 748072, time: 2014-06-12 14:46:46.319570, event: started, op: osd_sub_op_reply(client.4316.0:147274 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.319579 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2654 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] sub_op_modify_reply: tid 236331 op  ack_type 4 from 29
2014-06-12 14:46:46.319605 7f9d64bda700  5 -- op tracker -- , seq: 748071, time: 2014-06-12 14:46:46.319595, event: sub_op_commit_rec, op: osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.319613 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2654 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] repop_all_applied: repop tid 236331 all applied 
2014-06-12 14:46:46.319630 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2654 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] eval_repop repgather(0x2010e3c0 173'2655 rep_tid=236331 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.319649 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2654 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean]  sending ack on repgather(0x2010e3c0 173'2655 rep_tid=236331 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x30ccd180
2014-06-12 14:46:46.319681 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2654 crt=0'0 lcod 173'2653 mlcod 173'2653 active+clean] repop_all_committed: repop tid 236331 all committed 
2014-06-12 14:46:46.319691 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2653 active+clean] eval_repop repgather(0x2010e3c0 173'2655 rep_tid=236331 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.319706 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2653 active+clean] log_op_stats osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.018461 lat 0.018487
2014-06-12 14:46:46.319723 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2653 active+clean] publish_stats_to_osd 173:3430
2014-06-12 14:46:46.319736 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2653 active+clean]  sending commit on repgather(0x2010e3c0 173'2655 rep_tid=236331 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x3228d680
2014-06-12 14:46:46.319767 7f9d64bda700  5 -- op tracker -- , seq: 748071, time: 2014-06-12 14:46:46.319761, event: commit_sent, op: osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.319774 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2653 active+clean]  requeue_ops 
2014-06-12 14:46:46.319784 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean]  removing repgather(0x2010e3c0 173'2655 rep_tid=236331 committed?=1 applied?=1 op=osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.319797 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean]    q front is repgather(0x2010e3c0 173'2655 rep_tid=236331 committed?=1 applied?=1 op=osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.319811 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] remove_repop repgather(0x2010e3c0 173'2655 rep_tid=236331 committed?=1 applied?=1 op=osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.319824 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean]  requeue_ops 
2014-06-12 14:46:46.319848 7f9d64bda700  5 -- op tracker -- , seq: 748071, time: 2014-06-12 14:46:46.319842, event: done, op: osd_op(client.4316.0:147274 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 1810432~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.319867 7f9d64bda700 10 osd.2 173 dequeue_op 0x160e4100 finish
2014-06-12 14:46:46.319874 7f9d64bda700  5 -- op tracker -- , seq: 748072, time: 2014-06-12 14:46:46.319870, event: done, op: osd_sub_op_reply(client.4316.0:147274 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.319883 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.319885 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.323279 7f9d47c58700  5 -- op tracker -- , seq: 748073, time: 2014-06-12 14:46:46.322093, event: header_read, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.323294 7f9d47c58700  5 -- op tracker -- , seq: 748073, time: 2014-06-12 14:46:46.322097, event: throttled, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.323304 7f9d47c58700  5 -- op tracker -- , seq: 748073, time: 2014-06-12 14:46:46.323241, event: all_read, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.323317 7f9d47c58700  5 -- op tracker -- , seq: 748073, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.323327 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:46.323358 7f9d47c58700 15 osd.2 173 enqueue_op 0x17c2bc00 prio 63 cost 516096 latency 0.001264 osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.323394 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.323403 7f9d6abe6700 10 osd.2 173 dequeue_op 0x17c2bc00 prio 63 cost 516096 latency 0.001309 osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean]
2014-06-12 14:46:46.323429 7f9d6abe6700  5 -- op tracker -- , seq: 748073, time: 2014-06-12 14:46:46.323424, event: reached_pg, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.323441 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:46.323457 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] handle_message: 0x17c2bc00
2014-06-12 14:46:46.323470 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] do_op osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:46.323498 7f9d6abe6700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_'
2014-06-12 14:46:46.323714 7f9d6abe6700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_' = 0
2014-06-12 14:46:46.323733 7f9d6abe6700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset'
2014-06-12 14:46:46.323839 7f9d6abe6700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset' = 0
2014-06-12 14:46:46.323852 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] populate_obc_watchers 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.323866 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.323877 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] get_object_context: creating obc from disk: 0x34d12580
2014-06-12 14:46:46.323888 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] get_object_context: 0x34d12580 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 rwstate(none n=0 w=0) oi: 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2655 client.4316.0:147274 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2655) ssc: 0x72b9260 snapset: 0=[]:[]+head
2014-06-12 14:46:46.323905 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] find_object_context 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 @head oi=267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2655 client.4316.0:147274 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2655)
2014-06-12 14:46:46.323932 7f9d6abe6700  5 -- op tracker -- , seq: 748073, time: 2014-06-12 14:46:46.323926, event: started, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.323940 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] execute_ctx 0x33b2c000
2014-06-12 14:46:46.323959 7f9d6abe6700  5 -- op tracker -- , seq: 748073, time: 2014-06-12 14:46:46.323953, event: started, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.323967 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] do_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] ov 173'2655 av 173'2656 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:46.323983 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] do_osd_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096]
2014-06-12 14:46:46.323997 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:46.324012 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] do_osd_op  write 2326528~516096
2014-06-12 14:46:46.324030 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:46.324043 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 snapset=0x72b9298  snapc=0=[]
2014-06-12 14:46:46.324057 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:46.324069 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] finish_ctx 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 0x33b2c000 op modify  
2014-06-12 14:46:46.324085 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean]  set mtime to 2014-06-12 08:45:48.701617
2014-06-12 14:46:46.324107 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean]  final snapset 0=[]:[]+head in 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.324127 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean]  zeroing write result code 0
2014-06-12 14:46:46.324140 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] new_repop rep_tid 236332 on osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.324154 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] issue_repop rep_tid 236332 o 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.324174 7f9d6abe6700  5 -- op tracker -- , seq: 748073, time: 2014-06-12 14:46:46.324171, event: waiting for subops from 29, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.324191 7f9d6abe6700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:46.324206 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2655 (0'0,173'2655] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] append_log log((0'0,173'2655], crt=0'0) [173'2656 (173'2655) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147275 2014-06-12 08:45:48.701617]
2014-06-12 14:46:46.324229 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] add_log_entry 173'2656 (173'2655) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147275 2014-06-12 08:45:48.701617
2014-06-12 14:46:46.324252 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] append_log  adding 1 keys
2014-06-12 14:46:46.324281 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2656, trimmed: 
2014-06-12 14:46:46.324302 7f9d6abe6700  5 queue_transactions existing osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.324305 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.324307 7f9d6abe6700  5 queue_op 0x7304540 seq 2468 osr(3.7f8 0x49b4340) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:46.324313 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] eval_repop repgather(0x5360b40 173'2656 rep_tid=236332 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.324330 7f9d6abe6700 10 osd.2 173 dequeue_op 0x17c2bc00 finish
2014-06-12 14:46:46.324334 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.324336 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.325633 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541cb40 (1 active)
2014-06-12 14:46:46.325646 7f9d7a962700  5 _do_op 0x7304540 seq 2468 osr(3.7f8 0x49b4340)/0x49b4340 start
2014-06-12 14:46:46.325651 7f9d7a962700 10 _do_transaction on 0x32b69800
2014-06-12 14:46:46.325659 7f9d7a962700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.325836 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.325862 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.325919 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.325944 7f9d7a962700 15 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.325950 7f9d7a962700 10 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1 r = 0
2014-06-12 14:46:46.325956 7f9d7a962700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.326153 7f9d7a962700 15 _write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 2326528~516096
2014-06-12 14:46:46.326299 7f9d7a962700 10 lookup_strip_header done  cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.326442 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:46.328418 7f9d7a962700 10 _generic_write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 2326528~516096 = 0
2014-06-12 14:46:46.328466 7f9d7a962700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.328503 7f9d7a962700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.328510 7f9d7a962700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.328518 7f9d7a962700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.328722 7f9d7a962700 20 set_header setting 17472 cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 parent seq 0
2014-06-12 14:46:46.328797 7f9d7a962700 20 set_header setting 2158 cid meta oid 1fe70fd5/pglog_3.7f8/0//-1 parent seq 0
2014-06-12 14:46:46.328811 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.328819 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:46.331950 7f9d5501e700  5 -- op tracker -- , seq: 748074, time: 2014-06-12 14:46:46.331900, event: header_read, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.331963 7f9d5501e700  5 -- op tracker -- , seq: 748074, time: 2014-06-12 14:46:46.331901, event: throttled, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.331977 7f9d5501e700  5 -- op tracker -- , seq: 748074, time: 2014-06-12 14:46:46.331916, event: all_read, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.331984 7f9d5501e700  5 -- op tracker -- , seq: 748074, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.331990 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:46.331996 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:46.332004 7f9d5501e700 15 osd.2 173 enqueue_op 0x124e3a00 prio 196 cost 0 latency 0.000104 osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0) v2
2014-06-12 14:46:46.332067 7f9d5501e700  5 -- op tracker -- , seq: 748075, time: 2014-06-12 14:46:46.332026, event: header_read, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.332076 7f9d5501e700  5 -- op tracker -- , seq: 748075, time: 2014-06-12 14:46:46.332026, event: throttled, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.332084 7f9d5501e700  5 -- op tracker -- , seq: 748075, time: 2014-06-12 14:46:46.332042, event: all_read, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.332091 7f9d5501e700  5 -- op tracker -- , seq: 748075, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.332094 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:46.332099 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:46.332104 7f9d5501e700 15 osd.2 173 enqueue_op 0x6d11900 prio 196 cost 0 latency 0.000077 osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:46.332128 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.332136 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x124e3a00 prio 196 cost 0 latency 0.000236 osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0) v2 pg pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean]
2014-06-12 14:46:46.332157 7f9d6a3e5700  5 -- op tracker -- , seq: 748074, time: 2014-06-12 14:46:46.332154, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.332165 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] handle_message: 0x124e3a00
2014-06-12 14:46:46.332181 7f9d6a3e5700  5 -- op tracker -- , seq: 748074, time: 2014-06-12 14:46:46.332178, event: started, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.332186 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] sub_op_modify_reply: tid 236332 op  ack_type 1 from 29
2014-06-12 14:46:46.332216 7f9d6a3e5700  5 -- op tracker -- , seq: 748073, time: 2014-06-12 14:46:46.332206, event: sub_op_applied_rec, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.332223 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x124e3a00 finish
2014-06-12 14:46:46.332230 7f9d6a3e5700  5 -- op tracker -- , seq: 748074, time: 2014-06-12 14:46:46.332227, event: done, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ack, result = 0)
2014-06-12 14:46:46.332245 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.332250 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.332254 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x6d11900 prio 196 cost 0 latency 0.000228 osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 pg pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean]
2014-06-12 14:46:46.332268 7f9d6a3e5700  5 -- op tracker -- , seq: 748075, time: 2014-06-12 14:46:46.332265, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.332274 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] handle_message: 0x6d11900
2014-06-12 14:46:46.332287 7f9d6a3e5700  5 -- op tracker -- , seq: 748075, time: 2014-06-12 14:46:46.332284, event: started, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.332292 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] sub_op_modify_reply: tid 236332 op  ack_type 4 from 29
2014-06-12 14:46:46.332308 7f9d6a3e5700  5 -- op tracker -- , seq: 748073, time: 2014-06-12 14:46:46.332303, event: sub_op_commit_rec, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.332314 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x6d11900 finish
2014-06-12 14:46:46.332319 7f9d6a3e5700  5 -- op tracker -- , seq: 748075, time: 2014-06-12 14:46:46.332316, event: done, op: osd_sub_op_reply(client.4316.0:147275 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.332328 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.332330 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.332339 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.339621 7f9d7a962700 10 _do_op 0x7304540 seq 2468 r = 0, finisher 0x89e6200 0x3b720c0
2014-06-12 14:46:46.339653 7f9d7a962700 10 _finish_op 0x7304540 seq 2468 osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.339675 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] op_applied: 236332
2014-06-12 14:46:46.339710 7f9d7795c700  5 -- op tracker -- , seq: 748073, time: 2014-06-12 14:46:46.339699, event: op_applied, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.339734 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] op_applied on primary on version 173'2656
2014-06-12 14:46:46.339746 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] repop_all_applied: repop tid 236332 all applied 
2014-06-12 14:46:46.339756 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] eval_repop repgather(0x5360b40 173'2656 rep_tid=236332 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.339773 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean]  sending ack on repgather(0x5360b40 173'2656 rep_tid=236332 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x2183a500
2014-06-12 14:46:46.339875 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541cb40 (0 active)
2014-06-12 14:46:46.339884 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.340075 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] op_commit: 236332
2014-06-12 14:46:46.340100 7f9d7715b700  5 -- op tracker -- , seq: 748073, time: 2014-06-12 14:46:46.340093, event: op_commit, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.340109 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2655 crt=0'0 lcod 173'2654 mlcod 173'2654 active+clean] repop_all_committed: repop tid 236332 all committed 
2014-06-12 14:46:46.340120 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2654 active+clean] eval_repop repgather(0x5360b40 173'2656 rep_tid=236332 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.340137 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2654 active+clean] log_op_stats osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.017723 lat 0.018043
2014-06-12 14:46:46.340155 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2654 active+clean] publish_stats_to_osd 173:3431
2014-06-12 14:46:46.340169 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2654 active+clean]  sending commit on repgather(0x5360b40 173'2656 rep_tid=236332 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x3b82af00
2014-06-12 14:46:46.340204 7f9d7715b700  5 -- op tracker -- , seq: 748073, time: 2014-06-12 14:46:46.340198, event: commit_sent, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.340212 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2654 active+clean]  requeue_ops 
2014-06-12 14:46:46.340223 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean]  removing repgather(0x5360b40 173'2656 rep_tid=236332 committed?=1 applied?=1 op=osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.340238 7f9d7715b700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean]    q front is repgather(0x5360b40 173'2656 rep_tid=236332 committed?=1 applied?=1 op=osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.340252 7f9d7715b700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] remove_repop repgather(0x5360b40 173'2656 rep_tid=236332 committed?=1 applied?=1 op=osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.340266 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean]  requeue_ops 
2014-06-12 14:46:46.340292 7f9d7715b700  5 -- op tracker -- , seq: 748073, time: 2014-06-12 14:46:46.340287, event: done, op: osd_op(client.4316.0:147275 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2326528~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.356932 7f9d47c58700  5 -- op tracker -- , seq: 748076, time: 2014-06-12 14:46:46.344014, event: header_read, op: osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.356952 7f9d47c58700  5 -- op tracker -- , seq: 748076, time: 2014-06-12 14:46:46.344017, event: throttled, op: osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.356961 7f9d47c58700  5 -- op tracker -- , seq: 748076, time: 2014-06-12 14:46:46.356869, event: all_read, op: osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.356969 7f9d47c58700  5 -- op tracker -- , seq: 748076, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.356978 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:46.357010 7f9d47c58700 15 osd.2 173 enqueue_op 0x3b42a100 prio 63 cost 516096 latency 0.012996 osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.357047 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.357060 7f9d653db700 10 osd.2 173 dequeue_op 0x3b42a100 prio 63 cost 516096 latency 0.013046 osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean]
2014-06-12 14:46:46.357088 7f9d653db700  5 -- op tracker -- , seq: 748076, time: 2014-06-12 14:46:46.357083, event: reached_pg, op: osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.357099 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:46.357114 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] handle_message: 0x3b42a100
2014-06-12 14:46:46.357125 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] do_op osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:46.357149 7f9d653db700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_'
2014-06-12 14:46:46.357397 7f9d653db700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_' = 0
2014-06-12 14:46:46.357417 7f9d653db700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset'
2014-06-12 14:46:46.357511 7f9d653db700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset' = 0
2014-06-12 14:46:46.357523 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] populate_obc_watchers 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.357538 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.357550 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] get_object_context: creating obc from disk: 0x2c967340
2014-06-12 14:46:46.357563 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] get_object_context: 0x2c967340 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 rwstate(none n=0 w=0) oi: 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2656 client.4316.0:147275 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2656) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:46:46.357582 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] find_object_context 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 @head oi=267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2656 client.4316.0:147275 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2656)
2014-06-12 14:46:46.357611 7f9d653db700  5 -- op tracker -- , seq: 748076, time: 2014-06-12 14:46:46.357605, event: started, op: osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.357629 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] execute_ctx 0x5d20600
2014-06-12 14:46:46.357648 7f9d653db700  5 -- op tracker -- , seq: 748076, time: 2014-06-12 14:46:46.357643, event: started, op: osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.357661 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] do_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] ov 173'2656 av 173'2657 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:46.357676 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] do_osd_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096]
2014-06-12 14:46:46.357687 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:46.357700 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] do_osd_op  write 2842624~516096
2014-06-12 14:46:46.357715 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:46.357726 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:46:46.357739 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:46.357748 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] finish_ctx 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 0x5d20600 op modify  
2014-06-12 14:46:46.357761 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean]  set mtime to 2014-06-12 08:45:48.721955
2014-06-12 14:46:46.357781 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean]  final snapset 0=[]:[]+head in 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.357799 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean]  zeroing write result code 0
2014-06-12 14:46:46.357811 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] new_repop rep_tid 236333 on osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.357834 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] issue_repop rep_tid 236333 o 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.357860 7f9d653db700  5 -- op tracker -- , seq: 748076, time: 2014-06-12 14:46:46.357855, event: waiting for subops from 29, op: osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.357884 7f9d653db700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:46.357907 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2656 (0'0,173'2656] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] append_log log((0'0,173'2656], crt=0'0) [173'2657 (173'2656) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147276 2014-06-12 08:45:48.721955]
2014-06-12 14:46:46.357932 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2656 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] add_log_entry 173'2657 (173'2656) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147276 2014-06-12 08:45:48.721955
2014-06-12 14:46:46.357955 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2656 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] append_log  adding 1 keys
2014-06-12 14:46:46.357988 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2657, trimmed: 
2014-06-12 14:46:46.358014 7f9d653db700  5 queue_transactions existing osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.358020 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.358022 7f9d653db700  5 queue_op 0x13033bc0 seq 2469 osr(3.7f8 0x49b4340) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:46.358030 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2656 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] eval_repop repgather(0x2236780 173'2657 rep_tid=236333 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.358051 7f9d653db700 10 osd.2 173 dequeue_op 0x3b42a100 finish
2014-06-12 14:46:46.358056 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.358059 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.358068 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541cb40 (1 active)
2014-06-12 14:46:46.358073 7f9d82972700  5 _do_op 0x13033bc0 seq 2469 osr(3.7f8 0x49b4340)/0x49b4340 start
2014-06-12 14:46:46.358077 7f9d82972700 10 _do_transaction on 0x2237740
2014-06-12 14:46:46.358084 7f9d82972700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.358176 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.358203 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.358254 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.358279 7f9d82972700 15 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.358287 7f9d82972700 10 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1 r = 0
2014-06-12 14:46:46.358293 7f9d82972700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.358413 7f9d82972700 15 _write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 2842624~516096
2014-06-12 14:46:46.358522 7f9d82972700 10 lookup_strip_header done  cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.358628 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:46.360294 7f9d82972700 10 _generic_write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 2842624~516096 = 0
2014-06-12 14:46:46.360353 7f9d82972700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.360368 7f9d82972700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.360374 7f9d82972700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.360383 7f9d82972700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.360589 7f9d82972700 20 set_header setting 17472 cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 parent seq 0
2014-06-12 14:46:46.360670 7f9d82972700 20 set_header setting 2158 cid meta oid 1fe70fd5/pglog_3.7f8/0//-1 parent seq 0
2014-06-12 14:46:46.360684 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.360692 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:46.364077 7f9d82972700 10 _do_op 0x13033bc0 seq 2469 r = 0, finisher 0x85e1440 0x16944600
2014-06-12 14:46:46.364107 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2656 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] op_commit: 236333
2014-06-12 14:46:46.364135 7f9d7715b700  5 -- op tracker -- , seq: 748076, time: 2014-06-12 14:46:46.364127, event: op_commit, op: osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.364153 7f9d82972700 10 _finish_op 0x13033bc0 seq 2469 osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.364161 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541cb40 (0 active)
2014-06-12 14:46:46.364165 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.365641 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2656 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] op_applied: 236333
2014-06-12 14:46:46.365673 7f9d7795c700  5 -- op tracker -- , seq: 748076, time: 2014-06-12 14:46:46.365665, event: op_applied, op: osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.365683 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2656 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] op_applied on primary on version 173'2657
2014-06-12 14:46:46.369360 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:46.369702 7f9d5501e700  5 -- op tracker -- , seq: 748077, time: 2014-06-12 14:46:46.369657, event: header_read, op: osd_sub_op_reply(client.4316.0:147276 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.369715 7f9d5501e700  5 -- op tracker -- , seq: 748077, time: 2014-06-12 14:46:46.369658, event: throttled, op: osd_sub_op_reply(client.4316.0:147276 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.369723 7f9d5501e700  5 -- op tracker -- , seq: 748077, time: 2014-06-12 14:46:46.369670, event: all_read, op: osd_sub_op_reply(client.4316.0:147276 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.369732 7f9d5501e700  5 -- op tracker -- , seq: 748077, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147276 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.369738 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147276 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:46.369744 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:46.369752 7f9d5501e700 15 osd.2 173 enqueue_op 0x2afdb00 prio 196 cost 0 latency 0.000095 osd_sub_op_reply(client.4316.0:147276 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:46.373645 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.373669 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2afdb00 prio 196 cost 0 latency 0.004011 osd_sub_op_reply(client.4316.0:147276 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 pg pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2656 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean]
2014-06-12 14:46:46.373703 7f9d63bd8700  5 -- op tracker -- , seq: 748077, time: 2014-06-12 14:46:46.373699, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147276 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.373716 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2656 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] handle_message: 0x2afdb00
2014-06-12 14:46:46.373734 7f9d63bd8700  5 -- op tracker -- , seq: 748077, time: 2014-06-12 14:46:46.373730, event: started, op: osd_sub_op_reply(client.4316.0:147276 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.373742 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2656 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] sub_op_modify_reply: tid 236333 op  ack_type 4 from 29
2014-06-12 14:46:46.373767 7f9d63bd8700  5 -- op tracker -- , seq: 748076, time: 2014-06-12 14:46:46.373757, event: sub_op_commit_rec, op: osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.373776 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2656 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] repop_all_applied: repop tid 236333 all applied 
2014-06-12 14:46:46.373790 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2656 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] eval_repop repgather(0x2236780 173'2657 rep_tid=236333 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.373809 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2656 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean]  sending ack on repgather(0x2236780 173'2657 rep_tid=236333 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x55a6a00
2014-06-12 14:46:46.373846 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2656 crt=0'0 lcod 173'2655 mlcod 173'2655 active+clean] repop_all_committed: repop tid 236333 all committed 
2014-06-12 14:46:46.373859 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2655 active+clean] eval_repop repgather(0x2236780 173'2657 rep_tid=236333 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.373878 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2655 active+clean] log_op_stats osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.029830 lat 0.029863
2014-06-12 14:46:46.373896 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2655 active+clean] publish_stats_to_osd 173:3432
2014-06-12 14:46:46.373910 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2655 active+clean]  sending commit on repgather(0x2236780 173'2657 rep_tid=236333 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x5f95180
2014-06-12 14:46:46.373946 7f9d63bd8700  5 -- op tracker -- , seq: 748076, time: 2014-06-12 14:46:46.373940, event: commit_sent, op: osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.373956 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2655 active+clean]  requeue_ops 
2014-06-12 14:46:46.373967 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean]  removing repgather(0x2236780 173'2657 rep_tid=236333 committed?=1 applied?=1 op=osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.373983 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean]    q front is repgather(0x2236780 173'2657 rep_tid=236333 committed?=1 applied?=1 op=osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.373999 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] remove_repop repgather(0x2236780 173'2657 rep_tid=236333 committed?=1 applied?=1 op=osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.374015 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean]  requeue_ops 
2014-06-12 14:46:46.374044 7f9d63bd8700  5 -- op tracker -- , seq: 748076, time: 2014-06-12 14:46:46.374038, event: done, op: osd_op(client.4316.0:147276 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 2842624~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.374065 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2afdb00 finish
2014-06-12 14:46:46.374076 7f9d63bd8700  5 -- op tracker -- , seq: 748077, time: 2014-06-12 14:46:46.374072, event: done, op: osd_sub_op_reply(client.4316.0:147276 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.374090 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.374095 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.382842 7f9d47c58700  5 -- op tracker -- , seq: 748078, time: 2014-06-12 14:46:46.376113, event: header_read, op: osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.382857 7f9d47c58700  5 -- op tracker -- , seq: 748078, time: 2014-06-12 14:46:46.376115, event: throttled, op: osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.382867 7f9d47c58700  5 -- op tracker -- , seq: 748078, time: 2014-06-12 14:46:46.382791, event: all_read, op: osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.382877 7f9d47c58700  5 -- op tracker -- , seq: 748078, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.382885 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:46.382921 7f9d47c58700 15 osd.2 173 enqueue_op 0x3b42bb00 prio 63 cost 516096 latency 0.006808 osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.385547 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.385561 7f9d68be2700 10 osd.2 173 dequeue_op 0x3b42bb00 prio 63 cost 516096 latency 0.009447 osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean]
2014-06-12 14:46:46.385589 7f9d68be2700  5 -- op tracker -- , seq: 748078, time: 2014-06-12 14:46:46.385584, event: reached_pg, op: osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.385599 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:46.385624 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] handle_message: 0x3b42bb00
2014-06-12 14:46:46.385634 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] do_op osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:46.385656 7f9d68be2700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_'
2014-06-12 14:46:46.385904 7f9d68be2700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_' = 0
2014-06-12 14:46:46.385921 7f9d68be2700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset'
2014-06-12 14:46:46.386039 7f9d68be2700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset' = 0
2014-06-12 14:46:46.386047 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] populate_obc_watchers 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.386059 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.386067 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] get_object_context: creating obc from disk: 0x1e6feb00
2014-06-12 14:46:46.386076 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] get_object_context: 0x1e6feb00 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 rwstate(none n=0 w=0) oi: 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2657 client.4316.0:147276 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2657) ssc: 0x81bbce0 snapset: 0=[]:[]+head
2014-06-12 14:46:46.386092 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] find_object_context 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 @head oi=267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2657 client.4316.0:147276 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2657)
2014-06-12 14:46:46.386115 7f9d68be2700  5 -- op tracker -- , seq: 748078, time: 2014-06-12 14:46:46.386109, event: started, op: osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.386123 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] execute_ctx 0x1b264a00
2014-06-12 14:46:46.386139 7f9d68be2700  5 -- op tracker -- , seq: 748078, time: 2014-06-12 14:46:46.386135, event: started, op: osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.386145 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] do_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] ov 173'2657 av 173'2658 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:46.386157 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] do_osd_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096]
2014-06-12 14:46:46.386175 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:46.386186 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] do_osd_op  write 3358720~516096
2014-06-12 14:46:46.386198 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:46.386207 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 snapset=0x81bbd18  snapc=0=[]
2014-06-12 14:46:46.386225 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:46.386233 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] finish_ctx 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 0x1b264a00 op modify  
2014-06-12 14:46:46.386243 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean]  set mtime to 2014-06-12 08:45:48.755647
2014-06-12 14:46:46.386260 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean]  final snapset 0=[]:[]+head in 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.386274 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean]  zeroing write result code 0
2014-06-12 14:46:46.386283 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] new_repop rep_tid 236334 on osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.386295 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] issue_repop rep_tid 236334 o 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.386314 7f9d68be2700  5 -- op tracker -- , seq: 748078, time: 2014-06-12 14:46:46.386310, event: waiting for subops from 29, op: osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.386332 7f9d68be2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:46.386350 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2657 (0'0,173'2657] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] append_log log((0'0,173'2657], crt=0'0) [173'2658 (173'2657) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147277 2014-06-12 08:45:48.755647]
2014-06-12 14:46:46.386369 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] add_log_entry 173'2658 (173'2657) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147277 2014-06-12 08:45:48.755647
2014-06-12 14:46:46.386386 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] append_log  adding 1 keys
2014-06-12 14:46:46.386413 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2658, trimmed: 
2014-06-12 14:46:46.386430 7f9d68be2700  5 queue_transactions existing osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.386433 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.386439 7f9d68be2700  5 queue_op 0x68dcd80 seq 2470 osr(3.7f8 0x49b4340) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:46.386449 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541cb40 (1 active)
2014-06-12 14:46:46.386457 7f9d7895e700  5 _do_op 0x68dcd80 seq 2470 osr(3.7f8 0x49b4340)/0x49b4340 start
2014-06-12 14:46:46.386446 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] eval_repop repgather(0x1f067740 173'2658 rep_tid=236334 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.386461 7f9d7895e700 10 _do_transaction on 0x1687cd80
2014-06-12 14:46:46.386462 7f9d68be2700 10 osd.2 173 dequeue_op 0x3b42bb00 finish
2014-06-12 14:46:46.386466 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.386466 7f9d7895e700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.386468 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.386569 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.386588 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.386659 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.386681 7f9d7895e700 15 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.386686 7f9d7895e700 10 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1 r = 0
2014-06-12 14:46:46.386691 7f9d7895e700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.386704 7f9d7895e700 15 _write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 3358720~516096
2014-06-12 14:46:46.386975 7f9d7895e700 10 lookup_strip_header done  cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.387004 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:46.388440 7f9d7895e700 10 _generic_write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 3358720~516096 = 0
2014-06-12 14:46:46.388483 7f9d7895e700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.388499 7f9d7895e700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.388504 7f9d7895e700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.388509 7f9d7895e700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.388664 7f9d7895e700 20 set_header setting 17472 cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 parent seq 0
2014-06-12 14:46:46.388744 7f9d7895e700 20 set_header setting 2158 cid meta oid 1fe70fd5/pglog_3.7f8/0//-1 parent seq 0
2014-06-12 14:46:46.388757 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.388762 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:46.391135 7f9d7895e700 10 _do_op 0x68dcd80 seq 2470 r = 0, finisher 0x66416a0 0x28eb08c0
2014-06-12 14:46:46.391153 7f9d7895e700 10 _finish_op 0x68dcd80 seq 2470 osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.391160 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541cb40 (0 active)
2014-06-12 14:46:46.391169 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.391160 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] op_commit: 236334
2014-06-12 14:46:46.391197 7f9d7715b700  5 -- op tracker -- , seq: 748078, time: 2014-06-12 14:46:46.391185, event: op_commit, op: osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.391216 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] op_applied: 236334
2014-06-12 14:46:46.391242 7f9d7795c700  5 -- op tracker -- , seq: 748078, time: 2014-06-12 14:46:46.391235, event: op_applied, op: osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.391253 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] op_applied on primary on version 173'2658
2014-06-12 14:46:46.392518 7f9d5501e700  5 -- op tracker -- , seq: 748079, time: 2014-06-12 14:46:46.392472, event: header_read, op: osd_sub_op_reply(client.4316.0:147277 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.392531 7f9d5501e700  5 -- op tracker -- , seq: 748079, time: 2014-06-12 14:46:46.392473, event: throttled, op: osd_sub_op_reply(client.4316.0:147277 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.392540 7f9d5501e700  5 -- op tracker -- , seq: 748079, time: 2014-06-12 14:46:46.392488, event: all_read, op: osd_sub_op_reply(client.4316.0:147277 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.392549 7f9d5501e700  5 -- op tracker -- , seq: 748079, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147277 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.392555 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147277 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:46.392561 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:46.392570 7f9d5501e700 15 osd.2 173 enqueue_op 0x1702f000 prio 196 cost 0 latency 0.000098 osd_sub_op_reply(client.4316.0:147277 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:46.392606 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.392615 7f9d623d5700 10 osd.2 173 dequeue_op 0x1702f000 prio 196 cost 0 latency 0.000143 osd_sub_op_reply(client.4316.0:147277 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 pg pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean]
2014-06-12 14:46:46.392636 7f9d623d5700  5 -- op tracker -- , seq: 748079, time: 2014-06-12 14:46:46.392632, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147277 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.392647 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] handle_message: 0x1702f000
2014-06-12 14:46:46.392664 7f9d623d5700  5 -- op tracker -- , seq: 748079, time: 2014-06-12 14:46:46.392661, event: started, op: osd_sub_op_reply(client.4316.0:147277 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.392672 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] sub_op_modify_reply: tid 236334 op  ack_type 4 from 29
2014-06-12 14:46:46.392694 7f9d623d5700  5 -- op tracker -- , seq: 748078, time: 2014-06-12 14:46:46.392687, event: sub_op_commit_rec, op: osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.392703 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] repop_all_applied: repop tid 236334 all applied 
2014-06-12 14:46:46.392716 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] eval_repop repgather(0x1f067740 173'2658 rep_tid=236334 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.392734 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean]  sending ack on repgather(0x1f067740 173'2658 rep_tid=236334 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x38136a00
2014-06-12 14:46:46.392770 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2657 crt=0'0 lcod 173'2656 mlcod 173'2656 active+clean] repop_all_committed: repop tid 236334 all committed 
2014-06-12 14:46:46.392782 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2656 active+clean] eval_repop repgather(0x1f067740 173'2658 rep_tid=236334 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.392798 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2656 active+clean] log_op_stats osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.016655 lat 0.016684
2014-06-12 14:46:46.392817 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2656 active+clean] publish_stats_to_osd 173:3433
2014-06-12 14:46:46.392838 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2656 active+clean]  sending commit on repgather(0x1f067740 173'2658 rep_tid=236334 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x191a0a00
2014-06-12 14:46:46.392875 7f9d623d5700  5 -- op tracker -- , seq: 748078, time: 2014-06-12 14:46:46.392870, event: commit_sent, op: osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.392885 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2656 active+clean]  requeue_ops 
2014-06-12 14:46:46.392898 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]  removing repgather(0x1f067740 173'2658 rep_tid=236334 committed?=1 applied?=1 op=osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.392915 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]    q front is repgather(0x1f067740 173'2658 rep_tid=236334 committed?=1 applied?=1 op=osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.392931 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] remove_repop repgather(0x1f067740 173'2658 rep_tid=236334 committed?=1 applied?=1 op=osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.392947 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]  requeue_ops 
2014-06-12 14:46:46.392975 7f9d623d5700  5 -- op tracker -- , seq: 748078, time: 2014-06-12 14:46:46.392969, event: done, op: osd_op(client.4316.0:147277 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3358720~516096] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.392996 7f9d623d5700 10 osd.2 173 dequeue_op 0x1702f000 finish
2014-06-12 14:46:46.393006 7f9d623d5700  5 -- op tracker -- , seq: 748079, time: 2014-06-12 14:46:46.393002, event: done, op: osd_sub_op_reply(client.4316.0:147277 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.393018 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.393021 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.395365 7f9d47c58700  5 -- op tracker -- , seq: 748080, time: 2014-06-12 14:46:46.394718, event: header_read, op: osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.395381 7f9d47c58700  5 -- op tracker -- , seq: 748080, time: 2014-06-12 14:46:46.394721, event: throttled, op: osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.395389 7f9d47c58700  5 -- op tracker -- , seq: 748080, time: 2014-06-12 14:46:46.395319, event: all_read, op: osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.395400 7f9d47c58700  5 -- op tracker -- , seq: 748080, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.395408 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:46:46.395436 7f9d47c58700 15 osd.2 173 enqueue_op 0x3b42a300 prio 63 cost 319488 latency 0.000718 osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.395466 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.395473 7f9d643d9700 10 osd.2 173 dequeue_op 0x3b42a300 prio 63 cost 319488 latency 0.000754 osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]
2014-06-12 14:46:46.395490 7f9d643d9700  5 -- op tracker -- , seq: 748080, time: 2014-06-12 14:46:46.395486, event: reached_pg, op: osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.395496 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:46.395505 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] handle_message: 0x3b42a300
2014-06-12 14:46:46.395511 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] do_op osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:46.395534 7f9d643d9700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_'
2014-06-12 14:46:46.395724 7f9d643d9700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 '_' = 0
2014-06-12 14:46:46.395738 7f9d643d9700 15 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset'
2014-06-12 14:46:46.395811 7f9d643d9700 10 getattr 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 'snapset' = 0
2014-06-12 14:46:46.395819 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] populate_obc_watchers 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.395826 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.395832 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] get_object_context: creating obc from disk: 0x638edc0
2014-06-12 14:46:46.395837 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] get_object_context: 0x638edc0 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 rwstate(none n=0 w=0) oi: 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2658 client.4316.0:147277 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2658) ssc: 0x283d2ee0 snapset: 0=[]:[]+head
2014-06-12 14:46:46.395847 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] find_object_context 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 @head oi=267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3(173'2658 client.4316.0:147277 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2658)
2014-06-12 14:46:46.395864 7f9d643d9700  5 -- op tracker -- , seq: 748080, time: 2014-06-12 14:46:46.395861, event: started, op: osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.395870 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] execute_ctx 0xfd18000
2014-06-12 14:46:46.395880 7f9d643d9700  5 -- op tracker -- , seq: 748080, time: 2014-06-12 14:46:46.395877, event: started, op: osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.395883 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] do_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] ov 173'2658 av 173'2659 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:46.395892 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] do_osd_op 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488]
2014-06-12 14:46:46.395898 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:46.395906 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] do_osd_op  write 3874816~319488
2014-06-12 14:46:46.395916 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:46:46.395923 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 snapset=0x283d2f18  snapc=0=[]
2014-06-12 14:46:46.395930 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] make_writeable 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:46.395935 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] finish_ctx 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 0xfd18000 op modify  
2014-06-12 14:46:46.395943 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]  set mtime to 2014-06-12 08:45:48.774601
2014-06-12 14:46:46.395956 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]  final snapset 0=[]:[]+head in 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.395968 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]  zeroing write result code 0
2014-06-12 14:46:46.395974 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] new_repop rep_tid 236335 on osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.395984 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] issue_repop rep_tid 236335 o 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.396000 7f9d643d9700  5 -- op tracker -- , seq: 748080, time: 2014-06-12 14:46:46.395997, event: waiting for subops from 29, op: osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.396014 7f9d643d9700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:46.396031 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2658 (0'0,173'2658] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] append_log log((0'0,173'2658], crt=0'0) [173'2659 (173'2658) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147278 2014-06-12 08:45:48.774601]
2014-06-12 14:46:46.396054 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] add_log_entry 173'2659 (173'2658) modify   267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 by client.4316.0:147278 2014-06-12 08:45:48.774601
2014-06-12 14:46:46.396071 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] append_log  adding 1 keys
2014-06-12 14:46:46.396101 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2659, trimmed: 
2014-06-12 14:46:46.396121 7f9d643d9700  5 queue_transactions existing osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.396124 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:46.396126 7f9d643d9700  5 queue_op 0x6268180 seq 2471 osr(3.7f8 0x49b4340) 321781 bytes   (queue has 1 ops and 321781 bytes)
2014-06-12 14:46:46.396133 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] eval_repop repgather(0xeaac000 173'2659 rep_tid=236335 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.396154 7f9d643d9700 10 osd.2 173 dequeue_op 0x3b42a300 finish
2014-06-12 14:46:46.396157 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.396160 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.396168 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541cb40 (1 active)
2014-06-12 14:46:46.396172 7f9d83173700  5 _do_op 0x6268180 seq 2471 osr(3.7f8 0x49b4340)/0x49b4340 start
2014-06-12 14:46:46.396176 7f9d83173700 10 _do_transaction on 0xe832240
2014-06-12 14:46:46.396181 7f9d83173700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.396286 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.396387 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:46.396434 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:46.396448 7f9d83173700 15 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.396452 7f9d83173700 10 _omap_rmkeys meta/1fe70fd5/pglog_3.7f8/0//-1 r = 0
2014-06-12 14:46:46.396455 7f9d83173700 15 _omap_setkeys meta/1fe70fd5/pglog_3.7f8/0//-1
2014-06-12 14:46:46.396464 7f9d83173700 15 _write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 3874816~319488
2014-06-12 14:46:46.396550 7f9d83173700 10 lookup_strip_header done  cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.396668 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:46.397398 7f9d83173700 10 _generic_write 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 3874816~319488 = 0
2014-06-12 14:46:46.397423 7f9d83173700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.397430 7f9d83173700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.397437 7f9d83173700 15 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3
2014-06-12 14:46:46.397442 7f9d83173700 10 _setattrs 3.7f8_head/267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 = 0
2014-06-12 14:46:46.397574 7f9d83173700 20 set_header setting 17472 cid 3.7f8_head oid 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 parent seq 0
2014-06-12 14:46:46.397646 7f9d83173700 20 set_header setting 2158 cid meta oid 1fe70fd5/pglog_3.7f8/0//-1 parent seq 0
2014-06-12 14:46:46.397662 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:46.397672 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:46.398839 7f9d83173700 10 _do_op 0x6268180 seq 2471 r = 0, finisher 0x7fbf380 0x36cb0c0
2014-06-12 14:46:46.398851 7f9d83173700 10 _finish_op 0x6268180 seq 2471 osr(3.7f8 0x49b4340)/0x49b4340
2014-06-12 14:46:46.398860 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541cb40 (0 active)
2014-06-12 14:46:46.398864 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:46.398857 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] op_commit: 236335
2014-06-12 14:46:46.398886 7f9d7715b700  5 -- op tracker -- , seq: 748080, time: 2014-06-12 14:46:46.398878, event: op_commit, op: osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.398904 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] op_applied: 236335
2014-06-12 14:46:46.398930 7f9d7795c700  5 -- op tracker -- , seq: 748080, time: 2014-06-12 14:46:46.398923, event: op_applied, op: osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.398941 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] op_applied on primary on version 173'2659
2014-06-12 14:46:46.400335 7f9d5501e700  5 -- op tracker -- , seq: 748081, time: 2014-06-12 14:46:46.400294, event: header_read, op: osd_sub_op_reply(client.4316.0:147278 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.400348 7f9d5501e700  5 -- op tracker -- , seq: 748081, time: 2014-06-12 14:46:46.400295, event: throttled, op: osd_sub_op_reply(client.4316.0:147278 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.400357 7f9d5501e700  5 -- op tracker -- , seq: 748081, time: 2014-06-12 14:46:46.400304, event: all_read, op: osd_sub_op_reply(client.4316.0:147278 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.400365 7f9d5501e700  5 -- op tracker -- , seq: 748081, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147278 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.400369 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147278 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:46.400375 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:46.400381 7f9d5501e700 15 osd.2 173 enqueue_op 0x124e2200 prio 196 cost 0 latency 0.000087 osd_sub_op_reply(client.4316.0:147278 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:46.400405 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.400412 7f9d633d7700 10 osd.2 173 dequeue_op 0x124e2200 prio 196 cost 0 latency 0.000118 osd_sub_op_reply(client.4316.0:147278 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0) v2 pg pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]
2014-06-12 14:46:46.400426 7f9d633d7700  5 -- op tracker -- , seq: 748081, time: 2014-06-12 14:46:46.400424, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147278 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.400432 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] handle_message: 0x124e2200
2014-06-12 14:46:46.400441 7f9d633d7700  5 -- op tracker -- , seq: 748081, time: 2014-06-12 14:46:46.400439, event: started, op: osd_sub_op_reply(client.4316.0:147278 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.400444 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] sub_op_modify_reply: tid 236335 op  ack_type 4 from 29
2014-06-12 14:46:46.400455 7f9d633d7700  5 -- op tracker -- , seq: 748080, time: 2014-06-12 14:46:46.400451, event: sub_op_commit_rec, op: osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.400459 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] repop_all_applied: repop tid 236335 all applied 
2014-06-12 14:46:46.400465 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] eval_repop repgather(0xeaac000 173'2659 rep_tid=236335 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.400474 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean]  sending ack on repgather(0xeaac000 173'2659 rep_tid=236335 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x133fef00
2014-06-12 14:46:46.400492 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2658 crt=0'0 lcod 173'2657 mlcod 173'2657 active+clean] repop_all_committed: repop tid 236335 all committed 
2014-06-12 14:46:46.400501 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2657 active+clean] eval_repop repgather(0xeaac000 173'2659 rep_tid=236335 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:46.400511 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2657 active+clean] log_op_stats osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4 inb 319767 outb 0 rlat 0.005774 lat 0.005792
2014-06-12 14:46:46.400520 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2657 active+clean] publish_stats_to_osd 173:3434
2014-06-12 14:46:46.400528 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2657 active+clean]  sending commit on repgather(0xeaac000 173'2659 rep_tid=236335 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4) 0x191a1400
2014-06-12 14:46:46.400546 7f9d633d7700  5 -- op tracker -- , seq: 748080, time: 2014-06-12 14:46:46.400543, event: commit_sent, op: osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.400556 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2657 active+clean]  requeue_ops 
2014-06-12 14:46:46.400562 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean]  removing repgather(0xeaac000 173'2659 rep_tid=236335 committed?=1 applied?=1 op=osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.400570 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean]    q front is repgather(0xeaac000 173'2659 rep_tid=236335 committed?=1 applied?=1 op=osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.400578 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean] remove_repop repgather(0xeaac000 173'2659 rep_tid=236335 committed?=1 applied?=1 op=osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:46.400586 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean]  requeue_ops 
2014-06-12 14:46:46.400599 7f9d633d7700  5 -- op tracker -- , seq: 748080, time: 2014-06-12 14:46:46.400596, event: done, op: osd_op(client.4316.0:147278 rbd_data.10b36b8b4567.0000000000001136 [set-alloc-hint object_size 4194304 write_size 4194304,write 3874816~319488] 3.267bdff8 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.400621 7f9d633d7700 10 osd.2 173 dequeue_op 0x124e2200 finish
2014-06-12 14:46:46.400628 7f9d633d7700  5 -- op tracker -- , seq: 748081, time: 2014-06-12 14:46:46.400625, event: done, op: osd_sub_op_reply(client.4316.0:147278 3.7f8 267bdff8/rbd_data.10b36b8b4567.0000000000001136/head//3 [] ondisk, result = 0)
2014-06-12 14:46:46.400640 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:46.400649 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:46.478720 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:46.478792 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:46.478781 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 11.57 >= max 0.5 = no, load too high
2014-06-12 14:46:46.478814 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:46.478817 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92654 < max (604800 seconds)
2014-06-12 14:46:46.478831 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:46.478834 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:46.478836 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:46.478838 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 18; oldest is 13.317494 seconds old
2014-06-12 14:46:46.721975 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:46.722379 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:46.971936 7f9d494c9700  5 -- op tracker -- , seq: 748082, time: 2014-06-12 14:46:46.971597, event: header_read, op: osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.971954 7f9d494c9700  5 -- op tracker -- , seq: 748082, time: 2014-06-12 14:46:46.971601, event: throttled, op: osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.971964 7f9d494c9700  5 -- op tracker -- , seq: 748082, time: 2014-06-12 14:46:46.971886, event: all_read, op: osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.971974 7f9d494c9700  5 -- op tracker -- , seq: 748082, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.971982 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:46:46.972012 7f9d494c9700 15 osd.2 173 enqueue_op 0x6d10600 prio 63 cost 98304 latency 0.000415 osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:46.972051 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:46.972062 7f9d61bd4700 10 osd.2 173 dequeue_op 0x6d10600 prio 63 cost 98304 latency 0.000465 osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]
2014-06-12 14:46:46.972089 7f9d61bd4700  5 -- op tracker -- , seq: 748082, time: 2014-06-12 14:46:46.972083, event: reached_pg, op: osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:46.972098 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:46.972115 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] handle_message: 0x6d10600
2014-06-12 14:46:46.972126 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_op osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:46.972153 7f9d61bd4700 15 getattr 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 '_'
2014-06-12 14:46:47.047356 7f9d61bd4700 10 getattr 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 '_' = 0
2014-06-12 14:46:47.047390 7f9d61bd4700 15 getattr 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 'snapset'
2014-06-12 14:46:47.047563 7f9d61bd4700 10 getattr 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 'snapset' = 0
2014-06-12 14:46:47.047576 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] populate_obc_watchers d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:47.047597 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:47.047608 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] get_object_context: creating obc from disk: 0x244de000
2014-06-12 14:46:47.047618 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] get_object_context: 0x244de000 d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 rwstate(none n=0 w=0) oi: d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3(173'532 client.4290.0:37980 wrlock_by=unknown.0.0:0 dirty s 4194304 uv532) ssc: 0x679b420 snapset: 0=[]:[]+head
2014-06-12 14:46:47.047636 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] find_object_context d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 @head oi=d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3(173'532 client.4290.0:37980 wrlock_by=unknown.0.0:0 dirty s 4194304 uv532)
2014-06-12 14:46:47.047669 7f9d61bd4700  5 -- op tracker -- , seq: 748082, time: 2014-06-12 14:46:47.047660, event: started, op: osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:47.047681 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] execute_ctx 0x7ad3600
2014-06-12 14:46:47.047699 7f9d61bd4700  5 -- op tracker -- , seq: 748082, time: 2014-06-12 14:46:47.047694, event: started, op: osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:47.047709 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_op d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] ov 173'532 av 173'2513 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:47.047726 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304]
2014-06-12 14:46:47.047738 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:47.047753 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op  write 0~98304
2014-06-12 14:46:47.047770 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:46:47.047783 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] make_writeable d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 snapset=0x679b458  snapc=0=[]
2014-06-12 14:46:47.047796 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] make_writeable d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:47.047808 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] finish_ctx d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 0x7ad3600 op modify  
2014-06-12 14:46:47.047821 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  set mtime to 2014-06-12 08:45:49.351848
2014-06-12 14:46:47.047846 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  final snapset 0=[]:[]+head in d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:47.047867 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  zeroing write result code 0
2014-06-12 14:46:47.047880 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] new_repop rep_tid 236336 on osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:47.047898 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] issue_repop rep_tid 236336 o d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:47.047925 7f9d61bd4700  5 -- op tracker -- , seq: 748082, time: 2014-06-12 14:46:47.047919, event: waiting for subops from 26, op: osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:47.047949 7f9d61bd4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:47.047976 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2512 (0'0,173'2512] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] append_log log((0'0,173'2512], crt=0'0) [173'2513 (173'532) modify   d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 by client.4290.0:143736 2014-06-12 08:45:49.351848]
2014-06-12 14:46:47.048002 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] add_log_entry 173'2513 (173'532) modify   d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 by client.4290.0:143736 2014-06-12 08:45:49.351848
2014-06-12 14:46:47.048025 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] append_log  adding 1 keys
2014-06-12 14:46:47.048061 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2513, trimmed: 
2014-06-12 14:46:47.048086 7f9d61bd4700  5 queue_transactions existing osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:47.048093 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:47.048095 7f9d61bd4700  5 queue_op 0x6d6bda0 seq 2329 osr(3.7ae 0x49b4010) 100597 bytes   (queue has 1 ops and 100597 bytes)
2014-06-12 14:46:47.048113 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1c0b440 (1 active)
2014-06-12 14:46:47.048102 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] eval_repop repgather(0x2f89f980 173'2513 rep_tid=236336 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:47.048122 7f9d7a161700  5 _do_op 0x6d6bda0 seq 2329 osr(3.7ae 0x49b4010)/0x49b4010 start
2014-06-12 14:46:47.048124 7f9d61bd4700 10 osd.2 173 dequeue_op 0x6d10600 finish
2014-06-12 14:46:47.048126 7f9d7a161700 10 _do_transaction on 0x1f067200
2014-06-12 14:46:47.048130 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:47.048133 7f9d7a161700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:47.048135 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:47.048266 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:47.048304 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:47.048358 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:47.048380 7f9d7a161700 15 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:47.048388 7f9d7a161700 10 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1 r = 0
2014-06-12 14:46:47.048394 7f9d7a161700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:47.048408 7f9d7a161700 15 _write 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 0~98304
2014-06-12 14:46:47.048545 7f9d7a161700 10 lookup_strip_header done  cid 3.7ae_head oid d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:47.048744 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:47.049164 7f9d7a161700 10 _generic_write 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 0~98304 = 0
2014-06-12 14:46:47.049190 7f9d7a161700 15 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:47.049203 7f9d7a161700 10 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:47.049209 7f9d7a161700 15 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:47.049218 7f9d7a161700 10 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:47.049348 7f9d7a161700 20 set_header setting 15929 cid 3.7ae_head oid d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 parent seq 0
2014-06-12 14:46:47.049377 7f9d7a161700 20 set_header setting 2150 cid meta oid 1fe77416/pglog_3.7ae/0//-1 parent seq 0
2014-06-12 14:46:47.049393 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:47.049403 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:47.049858 7f9d7a161700 10 _do_op 0x6d6bda0 seq 2329 r = 0, finisher 0x288daa0 0x36cb040
2014-06-12 14:46:47.049870 7f9d7a161700 10 _finish_op 0x6d6bda0 seq 2329 osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:47.049877 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1c0b440 (0 active)
2014-06-12 14:46:47.049880 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:47.049890 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_commit: 236336
2014-06-12 14:46:47.049914 7f9d7715b700  5 -- op tracker -- , seq: 748082, time: 2014-06-12 14:46:47.049906, event: op_commit, op: osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:47.049937 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_applied: 236336
2014-06-12 14:46:47.049965 7f9d7795c700  5 -- op tracker -- , seq: 748082, time: 2014-06-12 14:46:47.049957, event: op_applied, op: osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:47.049973 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_applied on primary on version 173'2513
2014-06-12 14:46:47.478901 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:47.478919 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:46:47.478921 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:46:47.478923 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:46:47.478941 7f9d84175700 10 osd.2 173 send_pg_stats - 3 pgs updated
2014-06-12 14:46:47.478984 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:47.479008 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 11.57 >= max 0.5 = no, load too high
2014-06-12 14:46:47.479021 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:47.479024 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92655 < max (604800 seconds)
2014-06-12 14:46:47.479033 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:47.479034 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:47.479035 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:47.479037 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 19; oldest is 14.317692 seconds old
2014-06-12 14:46:47.510668 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:47.510813 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:47.522210 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:47.522452 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:47.610240 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:47.638309 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:47.663401 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:47.663404 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:47.663406 7f9d74bfa700 20 osd.2 173 _dispatch 0x6793c00 pg_stats_ack(3 pgs tid 4855) v1
2014-06-12 14:46:47.663408 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:46:47.663414 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:47.663422 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:47.687573 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:47.743683 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:47.743872 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:47.822522 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:47.822583 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:47.850262 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:47.858215 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:47.858445 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:47.933566 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.076258 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.096950 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.115073 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.125154 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.132579 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.140387 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:48.140520 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:48.154766 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:48.229632 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.249759 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.253727 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:48.253760 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:48.261702 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.262250 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.270445 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.273955 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.275195 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.280081 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.284675 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.290272 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.297590 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.299164 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.307544 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.311760 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.319974 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.324425 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.332420 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.332454 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.339953 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.358127 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.364241 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.369393 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:48.374158 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.386542 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.391258 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.393090 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.396232 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.398934 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:48.400722 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:48.445622 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:48.445671 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:48.479099 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:48.479115 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:48.479118 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:48.479127 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:48.479129 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 19; oldest is 15.317785 seconds old
2014-06-12 14:46:48.479129 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:48.920878 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19422 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,1,10,3,5])
2014-06-12 14:46:48.920907 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19422 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,1,10,3,5])
2014-06-12 14:46:48.921635 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:48.921675 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:48.921714 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:48.921735 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:48.921749 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:48.921784 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:48.921808 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:48.921840 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:48.921861 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:48.921871 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:48.921882 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:48.921891 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:48.921906 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:48.921921 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:48.921932 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:48.921942 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:48.921953 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:48.921964 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:48.921975 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:48.921987 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:48.921999 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:48.922010 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:49.048229 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:49.049936 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:49.479194 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:49.479214 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:49.479226 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:49.479227 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:49.479228 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:49.479229 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 19; oldest is 16.317885 seconds old
2014-06-12 14:46:49.610326 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:49.638388 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:49.687646 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:49.850347 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:49.933637 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.076361 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.097021 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.115155 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.125232 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.132651 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.146061 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:50.146089 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:50.154850 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:50.158843 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:50.158851 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:50.158855 7f9d74bfa700 20 osd.2 173 _dispatch 0xa24f040 ping v1
2014-06-12 14:46:50.158859 7f9d74bfa700 10 osd.2 173 ping from client.4293
2014-06-12 14:46:50.158864 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:50.158867 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:50.229719 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.249826 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.261782 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.262324 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.270536 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.274048 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.275276 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.280167 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.284759 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.290353 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.297678 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.299250 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.307622 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.311833 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.320056 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.324523 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.332492 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.332537 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.340023 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.358218 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.364317 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.369495 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:50.374253 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.386633 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.391359 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.393189 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.396332 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.399016 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.400803 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.479295 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:50.479315 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:50.479318 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:50.479327 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:50.479328 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:50.479329 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 19; oldest is 17.317985 seconds old
2014-06-12 14:46:50.554386 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:50.554426 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:50.644292 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:50.644423 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:50.700896 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:50.700920 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:50.707551 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:50.707610 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:50.749404 7f9d56432700  5 -- op tracker -- , seq: 748083, time: 2014-06-12 14:46:50.747726, event: header_read, op: osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.749436 7f9d56432700  5 -- op tracker -- , seq: 748083, time: 2014-06-12 14:46:50.747731, event: throttled, op: osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.749455 7f9d56432700  5 -- op tracker -- , seq: 748083, time: 2014-06-12 14:46:50.749338, event: all_read, op: osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.749474 7f9d56432700  5 -- op tracker -- , seq: 748083, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.749485 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:50.749501 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:50.749518 7f9d56432700 15 osd.2 173 enqueue_op 0x28fd8200 prio 127 cost 516903 latency 0.001791 osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:50.749564 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:50.749578 7f9d61bd4700 10 osd.2 173 dequeue_op 0x28fd8200 prio 127 cost 516903 latency 0.001851 osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.60e( v 173'2565 (0'0,173'2565] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2564 active]
2014-06-12 14:46:50.749642 7f9d61bd4700  5 -- op tracker -- , seq: 748083, time: 2014-06-12 14:46:50.749627, event: reached_pg, op: osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.749652 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2565 (0'0,173'2565] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2564 active] handle_message: 0x28fd8200
2014-06-12 14:46:50.749666 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2565 (0'0,173'2565] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2564 active] sub_op_modify trans 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 v 173'2566 (transaction) 174
2014-06-12 14:46:50.749678 7f9d61bd4700  5 -- op tracker -- , seq: 748083, time: 2014-06-12 14:46:50.749675, event: started, op: osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.749691 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2565 (0'0,173'2565] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2564 active] append_log log((0'0,173'2565], crt=0'0) [173'2566 (173'2565) modify   8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 by client.4298.0:145524 2014-06-12 08:45:20.901711]
2014-06-12 14:46:50.749736 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2566 (0'0,173'2566] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2564 active] add_log_entry 173'2566 (173'2565) modify   8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 by client.4298.0:145524 2014-06-12 08:45:20.901711
2014-06-12 14:46:50.749814 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2566 (0'0,173'2566] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2564 active] append_log  adding 1 keys
2014-06-12 14:46:50.749902 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2566, trimmed: 
2014-06-12 14:46:50.749942 7f9d61bd4700  5 -- op tracker -- , seq: 748083, time: 2014-06-12 14:46:50.749932, event: started, op: osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.749959 7f9d61bd4700  5 queue_transactions existing osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:50.749966 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:50.749968 7f9d61bd4700  5 queue_op 0x41a8e40 seq 2408 osr(3.60e 0x276dcf0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:50.749996 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533cd80 (1 active)
2014-06-12 14:46:50.749979 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.60e( v 173'2566 (0'0,173'2566] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2564 active] do_sub_op osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:50.750004 7f9d61bd4700 10 osd.2 173 dequeue_op 0x28fd8200 finish
2014-06-12 14:46:50.750006 7f9d7a161700  5 _do_op 0x41a8e40 seq 2408 osr(3.60e 0x276dcf0)/0x276dcf0 start
2014-06-12 14:46:50.750010 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:50.750013 7f9d7a161700 10 _do_transaction on 0x66640f0
2014-06-12 14:46:50.750014 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.750023 7f9d7a161700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:50.750295 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:50.750332 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:50.750423 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:50.750461 7f9d7a161700 15 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:50.750470 7f9d7a161700 10 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1 r = 0
2014-06-12 14:46:50.750481 7f9d7a161700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:50.750507 7f9d7a161700 15 _write 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 1363968~516096
2014-06-12 14:46:50.750724 7f9d7a161700 10 lookup_strip_header done  cid 3.60e_head oid 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:50.750810 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:50.752969 7f9d7a161700 10 _generic_write 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 1363968~516096 = 0
2014-06-12 14:46:50.753016 7f9d7a161700 15 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:50.753032 7f9d7a161700 10 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 = 0
2014-06-12 14:46:50.753037 7f9d7a161700 15 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:50.753044 7f9d7a161700 10 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 = 0
2014-06-12 14:46:50.753260 7f9d7a161700 20 set_header setting 17483 cid 3.60e_head oid 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 parent seq 0
2014-06-12 14:46:50.753338 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:50.753351 7f9d7a161700 20 set_header setting 2240 cid meta oid 1fe13dcb/pglog_3.60e/0//-1 parent seq 0
2014-06-12 14:46:50.753359 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:50.756045 7f9d7a161700 10 _do_op 0x41a8e40 seq 2408 r = 0, finisher 0x456e0c0 0
2014-06-12 14:46:50.756064 7f9d7a161700 10 _finish_op 0x41a8e40 seq 2408 osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:50.756076 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533cd80 (0 active)
2014-06-12 14:46:50.756085 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.756091 7f9d7715b700  5 -- op tracker -- , seq: 748083, time: 2014-06-12 14:46:50.756080, event: commit_sent, op: osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.756100 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2566 (0'0,173'2566] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2564 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:50.756142 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:50.756211 7f9d7795c700  5 -- op tracker -- , seq: 748083, time: 2014-06-12 14:46:50.756195, event: sub_op_applied, op: osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.756232 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2566 (0'0,173'2566] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active] sub_op_modify_applied on 0x6664000 op osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:50.756273 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2566 (0'0,173'2566] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active] op_applied on primary on version 173'2566
2014-06-12 14:46:50.756298 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2566 (0'0,173'2566] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active] op_applied on replica on version 173'2566
2014-06-12 14:46:50.756357 7f9d7795c700  5 -- op tracker -- , seq: 748083, time: 2014-06-12 14:46:50.756345, event: done, op: osd_sub_op(client.4298.0:145524 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2566 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.759012 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:50.759073 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:50.773281 7f9d56432700  5 -- op tracker -- , seq: 748084, time: 2014-06-12 14:46:50.771088, event: header_read, op: osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.773305 7f9d56432700  5 -- op tracker -- , seq: 748084, time: 2014-06-12 14:46:50.771091, event: throttled, op: osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.773320 7f9d56432700  5 -- op tracker -- , seq: 748084, time: 2014-06-12 14:46:50.773220, event: all_read, op: osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.773335 7f9d56432700  5 -- op tracker -- , seq: 748084, time: 0.000000, event: dispatched, op: osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.773343 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:50.773353 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:50.773367 7f9d56432700 15 osd.2 173 enqueue_op 0x28dd0800 prio 127 cost 738087 latency 0.002278 osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:50.773407 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:50.773416 7f9d67be0700 10 osd.2 173 dequeue_op 0x28dd0800 prio 127 cost 738087 latency 0.002328 osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.71f( v 173'2704 (0'0,173'2704] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2703 active]
2014-06-12 14:46:50.773460 7f9d67be0700  5 -- op tracker -- , seq: 748084, time: 2014-06-12 14:46:50.773453, event: reached_pg, op: osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.773473 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2704 (0'0,173'2704] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2703 active] handle_message: 0x28dd0800
2014-06-12 14:46:50.773495 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2704 (0'0,173'2704] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2703 active] sub_op_modify trans 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 v 173'2705 (transaction) 174
2014-06-12 14:46:50.773522 7f9d67be0700  5 -- op tracker -- , seq: 748084, time: 2014-06-12 14:46:50.773516, event: started, op: osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.773543 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2704 (0'0,173'2704] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2703 active] append_log log((0'0,173'2704], crt=0'0) [173'2705 (173'2704) modify   4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 by client.4280.0:139907 2014-06-12 08:45:46.526116]
2014-06-12 14:46:50.773580 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2705 (0'0,173'2705] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2703 active] add_log_entry 173'2705 (173'2704) modify   4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 by client.4280.0:139907 2014-06-12 08:45:46.526116
2014-06-12 14:46:50.773660 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2705 (0'0,173'2705] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2703 active] append_log  adding 1 keys
2014-06-12 14:46:50.773711 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2705, trimmed: 
2014-06-12 14:46:50.773758 7f9d67be0700  5 -- op tracker -- , seq: 748084, time: 2014-06-12 14:46:50.773746, event: started, op: osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.773773 7f9d67be0700  5 queue_transactions existing osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:50.773779 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:50.773781 7f9d67be0700  5 queue_op 0x39cbda0 seq 2537 osr(3.71f 0x276de50) 739573 bytes   (queue has 1 ops and 739573 bytes)
2014-06-12 14:46:50.773791 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.71f( v 173'2705 (0'0,173'2705] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2703 active] do_sub_op osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:50.773815 7f9d67be0700 10 osd.2 173 dequeue_op 0x28dd0800 finish
2014-06-12 14:46:50.773820 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:50.773824 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.773836 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54d0240 (1 active)
2014-06-12 14:46:50.773842 7f9d7f16b700  5 _do_op 0x39cbda0 seq 2537 osr(3.71f 0x276de50)/0x276de50 start
2014-06-12 14:46:50.773848 7f9d7f16b700 10 _do_transaction on 0x328a69b0
2014-06-12 14:46:50.773856 7f9d7f16b700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:50.774026 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:50.774054 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:50.774137 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:50.774168 7f9d7f16b700 15 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:50.774178 7f9d7f16b700 10 _omap_rmkeys meta/1fe0985b/pglog_3.71f/0//-1 r = 0
2014-06-12 14:46:50.774188 7f9d7f16b700 15 _omap_setkeys meta/1fe0985b/pglog_3.71f/0//-1
2014-06-12 14:46:50.774209 7f9d7f16b700 15 _write 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 3457024~737280
2014-06-12 14:46:50.774455 7f9d7f16b700 10 lookup_strip_header done  cid 3.71f_head oid 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3
2014-06-12 14:46:50.774851 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:50.779146 7f9d7f16b700 10 _generic_write 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 3457024~737280 = 0
2014-06-12 14:46:50.779244 7f9d7f16b700 15 _setattrs 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3
2014-06-12 14:46:50.779275 7f9d7f16b700 10 _setattrs 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 = 0
2014-06-12 14:46:50.779283 7f9d7f16b700 15 _setattrs 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3
2014-06-12 14:46:50.779291 7f9d7f16b700 10 _setattrs 3.71f_head/4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 = 0
2014-06-12 14:46:50.779561 7f9d7f16b700 20 set_header setting 15950 cid 3.71f_head oid 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 parent seq 0
2014-06-12 14:46:50.779640 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:50.779656 7f9d7f16b700 20 set_header setting 2330 cid meta oid 1fe0985b/pglog_3.71f/0//-1 parent seq 0
2014-06-12 14:46:50.779667 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:50.785348 7f9d7f16b700 10 _do_op 0x39cbda0 seq 2537 r = 0, finisher 0x5fee3c0 0
2014-06-12 14:46:50.785384 7f9d7f16b700 10 _finish_op 0x39cbda0 seq 2537 osr(3.71f 0x276de50)/0x276de50
2014-06-12 14:46:50.785394 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54d0240 (0 active)
2014-06-12 14:46:50.785402 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.785404 7f9d7715b700  5 -- op tracker -- , seq: 748084, time: 2014-06-12 14:46:50.785392, event: commit_sent, op: osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.785418 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2705 (0'0,173'2705] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2703 active] sub_op_modify_commit on op osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:50.785446 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:50.785481 7f9d7795c700  5 -- op tracker -- , seq: 748084, time: 2014-06-12 14:46:50.785474, event: sub_op_applied, op: osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.785490 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2705 (0'0,173'2705] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2704 active] sub_op_modify_applied on 0x328a68c0 op osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:50.785513 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2705 (0'0,173'2705] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2704 active] op_applied on primary on version 173'2705
2014-06-12 14:46:50.785523 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.71f( v 173'2705 (0'0,173'2705] local-les=153 n=302 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2704 active] op_applied on replica on version 173'2705
2014-06-12 14:46:50.785543 7f9d7795c700  5 -- op tracker -- , seq: 748084, time: 2014-06-12 14:46:50.785539, event: done, op: osd_sub_op(client.4280.0:139907 3.71f 4b257f1f/rbd_data.108f6b8b4567.0000000000000e99/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.869329 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:50.869447 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:50.903485 7f9d56432700  5 -- op tracker -- , seq: 748085, time: 2014-06-12 14:46:50.901086, event: header_read, op: osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.903513 7f9d56432700  5 -- op tracker -- , seq: 748085, time: 2014-06-12 14:46:50.901090, event: throttled, op: osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.903540 7f9d56432700  5 -- op tracker -- , seq: 748085, time: 2014-06-12 14:46:50.903432, event: all_read, op: osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.903550 7f9d56432700  5 -- op tracker -- , seq: 748085, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.903554 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:50.903559 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:50.903565 7f9d56432700 15 osd.2 173 enqueue_op 0x28dd1e00 prio 127 cost 1032999 latency 0.002479 osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:50.903589 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:50.903597 7f9d673df700 10 osd.2 173 dequeue_op 0x28dd1e00 prio 127 cost 1032999 latency 0.002511 osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.60e( v 173'2566 (0'0,173'2566] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active]
2014-06-12 14:46:50.903618 7f9d673df700  5 -- op tracker -- , seq: 748085, time: 2014-06-12 14:46:50.903615, event: reached_pg, op: osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.903629 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2566 (0'0,173'2566] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active] handle_message: 0x28dd1e00
2014-06-12 14:46:50.903640 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2566 (0'0,173'2566] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active] sub_op_modify trans 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 v 173'2567 (transaction) 174
2014-06-12 14:46:50.903652 7f9d673df700  5 -- op tracker -- , seq: 748085, time: 2014-06-12 14:46:50.903649, event: started, op: osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.903664 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2566 (0'0,173'2566] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active] append_log log((0'0,173'2566], crt=0'0) [173'2567 (173'2566) modify   8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 by client.4298.0:145525 2014-06-12 08:45:21.541606]
2014-06-12 14:46:50.903695 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2567 (0'0,173'2567] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active] add_log_entry 173'2567 (173'2566) modify   8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 by client.4298.0:145525 2014-06-12 08:45:21.541606
2014-06-12 14:46:50.903731 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2567 (0'0,173'2567] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active] append_log  adding 1 keys
2014-06-12 14:46:50.903756 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2567, trimmed: 
2014-06-12 14:46:50.903773 7f9d673df700  5 -- op tracker -- , seq: 748085, time: 2014-06-12 14:46:50.903770, event: started, op: osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.903782 7f9d673df700  5 queue_transactions existing osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:50.903785 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:50.903786 7f9d673df700  5 queue_op 0x6d5b260 seq 2409 osr(3.60e 0x276dcf0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:50.903799 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533cd80 (1 active)
2014-06-12 14:46:50.903794 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.60e( v 173'2567 (0'0,173'2567] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active] do_sub_op osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:50.903810 7f9d673df700 10 osd.2 173 dequeue_op 0x28dd1e00 finish
2014-06-12 14:46:50.903809 7f9d7e169700  5 _do_op 0x6d5b260 seq 2409 osr(3.60e 0x276dcf0)/0x276dcf0 start
2014-06-12 14:46:50.903814 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:50.903815 7f9d7e169700 10 _do_transaction on 0x2705e9b0
2014-06-12 14:46:50.903817 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:50.903822 7f9d7e169700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:50.904002 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:50.904046 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:50.904154 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:50.904360 7f9d7e169700 15 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:50.904379 7f9d7e169700 10 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1 r = 0
2014-06-12 14:46:50.904394 7f9d7e169700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:50.904433 7f9d7e169700 15 _write 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 1880064~1032192
2014-06-12 14:46:50.904652 7f9d7e169700 10 lookup_strip_header done  cid 3.60e_head oid 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:50.904900 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:50.909840 7f9d7e169700 10 _generic_write 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 1880064~1032192 = 0
2014-06-12 14:46:50.909925 7f9d7e169700 15 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:50.909946 7f9d7e169700 10 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 = 0
2014-06-12 14:46:50.909953 7f9d7e169700 15 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:50.909962 7f9d7e169700 10 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 = 0
2014-06-12 14:46:50.910208 7f9d7e169700 20 set_header setting 17483 cid 3.60e_head oid 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 parent seq 0
2014-06-12 14:46:50.910306 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:50.910323 7f9d7e169700 20 set_header setting 2240 cid meta oid 1fe13dcb/pglog_3.60e/0//-1 parent seq 0
2014-06-12 14:46:50.910334 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:50.913621 7f9d7e169700 10 _do_op 0x6d5b260 seq 2409 r = 0, finisher 0x1ecc9e60 0
2014-06-12 14:46:50.913660 7f9d7e169700 10 _finish_op 0x6d5b260 seq 2409 osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:50.913675 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533cd80 (0 active)
2014-06-12 14:46:50.913684 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:50.913697 7f9d7795c700  5 -- op tracker -- , seq: 748085, time: 2014-06-12 14:46:50.913687, event: sub_op_applied, op: osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.913713 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2567 (0'0,173'2567] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active] sub_op_modify_applied on 0x2705e8c0 op osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:50.913738 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:50.913757 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2567 (0'0,173'2567] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active] op_applied on primary on version 173'2567
2014-06-12 14:46:50.913766 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2567 (0'0,173'2567] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active] op_applied on replica on version 173'2567
2014-06-12 14:46:50.913815 7f9d7715b700  5 -- op tracker -- , seq: 748085, time: 2014-06-12 14:46:50.913800, event: commit_sent, op: osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:50.913825 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2567 (0'0,173'2567] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2565 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:50.913846 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:50.913877 7f9d7715b700  5 -- op tracker -- , seq: 748085, time: 2014-06-12 14:46:50.913872, event: done, op: osd_sub_op(client.4298.0:145525 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2567 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.022802 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:51.022889 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:51.054902 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:51.054958 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:51.068602 7f9d56432700  5 -- op tracker -- , seq: 748086, time: 2014-06-12 14:46:51.067742, event: header_read, op: osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.068617 7f9d56432700  5 -- op tracker -- , seq: 748086, time: 2014-06-12 14:46:51.067743, event: throttled, op: osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.068626 7f9d56432700  5 -- op tracker -- , seq: 748086, time: 2014-06-12 14:46:51.068562, event: all_read, op: osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.068635 7f9d56432700  5 -- op tracker -- , seq: 748086, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.068640 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:51.068647 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:51.068655 7f9d56432700 15 osd.2 173 enqueue_op 0x28dd0400 prio 127 cost 516903 latency 0.000913 osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:51.068687 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.068695 7f9d663dd700 10 osd.2 173 dequeue_op 0x28dd0400 prio 127 cost 516903 latency 0.000953 osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.60e( v 173'2567 (0'0,173'2567] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2566 active]
2014-06-12 14:46:51.068720 7f9d663dd700  5 -- op tracker -- , seq: 748086, time: 2014-06-12 14:46:51.068715, event: reached_pg, op: osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.068730 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2567 (0'0,173'2567] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2566 active] handle_message: 0x28dd0400
2014-06-12 14:46:51.068744 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2567 (0'0,173'2567] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2566 active] sub_op_modify trans 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 v 173'2568 (transaction) 174
2014-06-12 14:46:51.068765 7f9d663dd700  5 -- op tracker -- , seq: 748086, time: 2014-06-12 14:46:51.068760, event: started, op: osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.068780 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2567 (0'0,173'2567] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2566 active] append_log log((0'0,173'2567], crt=0'0) [173'2568 (173'2567) modify   8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 by client.4298.0:145526 2014-06-12 08:45:27.289779]
2014-06-12 14:46:51.068805 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2568 (0'0,173'2568] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2566 active] add_log_entry 173'2568 (173'2567) modify   8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 by client.4298.0:145526 2014-06-12 08:45:27.289779
2014-06-12 14:46:51.068829 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2568 (0'0,173'2568] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2566 active] append_log  adding 1 keys
2014-06-12 14:46:51.068865 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2568, trimmed: 
2014-06-12 14:46:51.068889 7f9d663dd700  5 -- op tracker -- , seq: 748086, time: 2014-06-12 14:46:51.068884, event: started, op: osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.068898 7f9d663dd700  5 queue_transactions existing osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:51.068902 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:51.068904 7f9d663dd700  5 queue_op 0x4063ec0 seq 2410 osr(3.60e 0x276dcf0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:51.068913 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533cd80 (1 active)
2014-06-12 14:46:51.068921 7f9d7815d700  5 _do_op 0x4063ec0 seq 2410 osr(3.60e 0x276dcf0)/0x276dcf0 start
2014-06-12 14:46:51.068910 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.60e( v 173'2568 (0'0,173'2568] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2566 active] do_sub_op osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:51.068924 7f9d663dd700 10 osd.2 173 dequeue_op 0x28dd0400 finish
2014-06-12 14:46:51.068925 7f9d7815d700 10 _do_transaction on 0x1c0b4470
2014-06-12 14:46:51.068928 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.068931 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.068932 7f9d7815d700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:51.069063 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:51.069087 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:51.069144 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:51.069174 7f9d7815d700 15 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:51.069183 7f9d7815d700 10 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1 r = 0
2014-06-12 14:46:51.069190 7f9d7815d700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:51.069290 7f9d7815d700 15 _write 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 2912256~516096
2014-06-12 14:46:51.069407 7f9d7815d700 10 lookup_strip_header done  cid 3.60e_head oid 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:51.069511 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:51.070951 7f9d7815d700 10 _generic_write 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 2912256~516096 = 0
2014-06-12 14:46:51.071012 7f9d7815d700 15 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:51.071028 7f9d7815d700 10 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 = 0
2014-06-12 14:46:51.071037 7f9d7815d700 15 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:51.071047 7f9d7815d700 10 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 = 0
2014-06-12 14:46:51.071194 7f9d7815d700 20 set_header setting 17483 cid 3.60e_head oid 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 parent seq 0
2014-06-12 14:46:51.071258 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:51.071275 7f9d7815d700 20 set_header setting 2240 cid meta oid 1fe13dcb/pglog_3.60e/0//-1 parent seq 0
2014-06-12 14:46:51.071290 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:51.073950 7f9d7815d700 10 _do_op 0x4063ec0 seq 2410 r = 0, finisher 0x8078f80 0
2014-06-12 14:46:51.073969 7f9d7815d700 10 _finish_op 0x4063ec0 seq 2410 osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:51.073976 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533cd80 (0 active)
2014-06-12 14:46:51.073983 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:51.073987 7f9d7715b700  5 -- op tracker -- , seq: 748086, time: 2014-06-12 14:46:51.073976, event: commit_sent, op: osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.073995 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2568 (0'0,173'2568] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2566 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:51.074028 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:51.074063 7f9d7795c700  5 -- op tracker -- , seq: 748086, time: 2014-06-12 14:46:51.074056, event: sub_op_applied, op: osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.074074 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2568 (0'0,173'2568] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2567 active] sub_op_modify_applied on 0x1c0b4380 op osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:51.074089 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2568 (0'0,173'2568] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2567 active] op_applied on primary on version 173'2568
2014-06-12 14:46:51.074095 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2568 (0'0,173'2568] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2567 active] op_applied on replica on version 173'2568
2014-06-12 14:46:51.074108 7f9d7795c700  5 -- op tracker -- , seq: 748086, time: 2014-06-12 14:46:51.074105, event: done, op: osd_sub_op(client.4298.0:145526 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2568 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.120803 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:51.120857 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:51.144726 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:51.144803 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:51.225325 7f9d56432700  5 -- op tracker -- , seq: 748087, time: 2014-06-12 14:46:51.223914, event: header_read, op: osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.225341 7f9d56432700  5 -- op tracker -- , seq: 748087, time: 2014-06-12 14:46:51.223918, event: throttled, op: osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.225357 7f9d56432700  5 -- op tracker -- , seq: 748087, time: 2014-06-12 14:46:51.225276, event: all_read, op: osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.225362 7f9d56432700  5 -- op tracker -- , seq: 748087, time: 0.000000, event: dispatched, op: osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.225367 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:51.225372 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:51.225380 7f9d56432700 15 osd.2 173 enqueue_op 0x2b015f00 prio 127 cost 766759 latency 0.001465 osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:51.225407 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.225414 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2b015f00 prio 127 cost 766759 latency 0.001500 osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.60e( v 173'2568 (0'0,173'2568] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2567 active]
2014-06-12 14:46:51.225432 7f9d65bdc700  5 -- op tracker -- , seq: 748087, time: 2014-06-12 14:46:51.225430, event: reached_pg, op: osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.225440 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2568 (0'0,173'2568] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2567 active] handle_message: 0x2b015f00
2014-06-12 14:46:51.225448 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2568 (0'0,173'2568] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2567 active] sub_op_modify trans 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 v 173'2569 (transaction) 174
2014-06-12 14:46:51.225459 7f9d65bdc700  5 -- op tracker -- , seq: 748087, time: 2014-06-12 14:46:51.225457, event: started, op: osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.225471 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2568 (0'0,173'2568] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2567 active] append_log log((0'0,173'2568], crt=0'0) [173'2569 (173'2568) modify   8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 by client.4298.0:145527 2014-06-12 08:45:27.309212]
2014-06-12 14:46:51.225489 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2569 (0'0,173'2569] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2567 active] add_log_entry 173'2569 (173'2568) modify   8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 by client.4298.0:145527 2014-06-12 08:45:27.309212
2014-06-12 14:46:51.225505 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2569 (0'0,173'2569] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2567 active] append_log  adding 1 keys
2014-06-12 14:46:51.225529 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2569, trimmed: 
2014-06-12 14:46:51.225549 7f9d65bdc700  5 -- op tracker -- , seq: 748087, time: 2014-06-12 14:46:51.225546, event: started, op: osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.225558 7f9d65bdc700  5 queue_transactions existing osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:51.225561 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:51.225562 7f9d65bdc700  5 queue_op 0x6e6e720 seq 2411 osr(3.60e 0x276dcf0) 768245 bytes   (queue has 1 ops and 768245 bytes)
2014-06-12 14:46:51.225568 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.60e( v 173'2569 (0'0,173'2569] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2567 active] do_sub_op osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:51.225581 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2b015f00 finish
2014-06-12 14:46:51.225583 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.225585 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.225913 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533cd80 (1 active)
2014-06-12 14:46:51.225922 7f9d82171700  5 _do_op 0x6e6e720 seq 2411 osr(3.60e 0x276dcf0)/0x276dcf0 start
2014-06-12 14:46:51.225927 7f9d82171700 10 _do_transaction on 0x1e78b430
2014-06-12 14:46:51.225934 7f9d82171700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:51.226099 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:51.226132 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:51.226194 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:51.226224 7f9d82171700 15 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:51.226229 7f9d82171700 10 _omap_rmkeys meta/1fe13dcb/pglog_3.60e/0//-1 r = 0
2014-06-12 14:46:51.226234 7f9d82171700 15 _omap_setkeys meta/1fe13dcb/pglog_3.60e/0//-1
2014-06-12 14:46:51.226246 7f9d82171700 15 _write 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 3428352~765952
2014-06-12 14:46:51.226384 7f9d82171700 10 lookup_strip_header done  cid 3.60e_head oid 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:51.226584 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:51.229218 7f9d82171700 10 _generic_write 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 3428352~765952 = 0
2014-06-12 14:46:51.229305 7f9d82171700 15 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:51.229324 7f9d82171700 10 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 = 0
2014-06-12 14:46:51.229329 7f9d82171700 15 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3
2014-06-12 14:46:51.229337 7f9d82171700 10 _setattrs 3.60e_head/8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 = 0
2014-06-12 14:46:51.229595 7f9d82171700 20 set_header setting 17483 cid 3.60e_head oid 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 parent seq 0
2014-06-12 14:46:51.229706 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:51.229721 7f9d82171700 20 set_header setting 2240 cid meta oid 1fe13dcb/pglog_3.60e/0//-1 parent seq 0
2014-06-12 14:46:51.229730 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:51.234128 7f9d82171700 10 _do_op 0x6e6e720 seq 2411 r = 0, finisher 0x98f5920 0
2014-06-12 14:46:51.234244 7f9d7715b700  5 -- op tracker -- , seq: 748087, time: 2014-06-12 14:46:51.234226, event: commit_sent, op: osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.234255 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2569 (0'0,173'2569] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2567 active] sub_op_modify_commit on op osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:51.234285 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:51.234341 7f9d82171700 10 _finish_op 0x6e6e720 seq 2411 osr(3.60e 0x276dcf0)/0x276dcf0
2014-06-12 14:46:51.234375 7f9d7795c700  5 -- op tracker -- , seq: 748087, time: 2014-06-12 14:46:51.234370, event: sub_op_applied, op: osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.234383 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2569 (0'0,173'2569] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2568 active] sub_op_modify_applied on 0x1e78b340 op osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:51.234397 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2569 (0'0,173'2569] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2568 active] op_applied on primary on version 173'2569
2014-06-12 14:46:51.234407 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.60e( v 173'2569 (0'0,173'2569] local-les=153 n=286 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2568 active] op_applied on replica on version 173'2569
2014-06-12 14:46:51.234489 7f9d82171700  5 -- op tracker -- , seq: 748087, time: 2014-06-12 14:46:51.234480, event: done, op: osd_sub_op(client.4298.0:145527 3.60e 8225360e/rbd_data.10a16b8b4567.0000000000001043/head//3 [] v 173'2569 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.234598 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533cd80 (0 active)
2014-06-12 14:46:51.234621 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:51.247024 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:51.247197 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:51.479392 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:51.479424 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:51.479450 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 10.81 >= max 0.5 = no, load too high
2014-06-12 14:46:51.479462 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:51.479464 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92659 < max (604800 seconds)
2014-06-12 14:46:51.479475 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:51.479477 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:51.479478 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:51.479483 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 19; oldest is 18.318138 seconds old
2014-06-12 14:46:51.609702 7f9d5bec4700  5 -- op tracker -- , seq: 748088, time: 2014-06-12 14:46:51.609465, event: header_read, op: osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.609715 7f9d5bec4700  5 -- op tracker -- , seq: 748088, time: 2014-06-12 14:46:51.609468, event: throttled, op: osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.609728 7f9d5bec4700  5 -- op tracker -- , seq: 748088, time: 2014-06-12 14:46:51.609665, event: all_read, op: osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.609733 7f9d5bec4700  5 -- op tracker -- , seq: 748088, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.609737 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:51.609742 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:51.609749 7f9d5bec4700 15 osd.2 173 enqueue_op 0x1e3eac00 prio 127 cost 131879 latency 0.000284 osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:51.609772 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.609790 7f9d62bd6700 10 osd.2 173 dequeue_op 0x1e3eac00 prio 127 cost 131879 latency 0.000325 osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.640( v 173'3013 (79'12,173'3013] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3012 active]
2014-06-12 14:46:51.609815 7f9d62bd6700  5 -- op tracker -- , seq: 748088, time: 2014-06-12 14:46:51.609811, event: reached_pg, op: osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.609825 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3013 (79'12,173'3013] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3012 active] handle_message: 0x1e3eac00
2014-06-12 14:46:51.609837 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3013 (79'12,173'3013] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3012 active] sub_op_modify trans a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 v 173'3014 (transaction) 174
2014-06-12 14:46:51.609852 7f9d62bd6700  5 -- op tracker -- , seq: 748088, time: 2014-06-12 14:46:51.609849, event: started, op: osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.609867 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3013 (79'12,173'3013] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3012 active] append_log log((79'12,173'3013], crt=0'0) [173'3014 (173'861) modify   a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 by client.4316.0:147286 2014-06-12 08:45:53.953485]
2014-06-12 14:46:51.609891 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3014 (79'12,173'3014] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3012 active] add_log_entry 173'3014 (173'861) modify   a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 by client.4316.0:147286 2014-06-12 08:45:53.953485
2014-06-12 14:46:51.609912 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3014 (79'12,173'3014] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3012 active] append_log  adding 1 keys
2014-06-12 14:46:51.609928 7f9d62bd6700 10 trim log((79'12,173'3014], crt=0'0) to 79'13
2014-06-12 14:46:51.609954 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3014, trimmed: 79'13
2014-06-12 14:46:51.609985 7f9d62bd6700  5 -- op tracker -- , seq: 748088, time: 2014-06-12 14:46:51.609980, event: started, op: osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.609996 7f9d62bd6700  5 queue_transactions existing osr(3.640 0x276dc10)/0x276dc10
2014-06-12 14:46:51.610000 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:51.610002 7f9d62bd6700  5 queue_op 0x11004960 seq 2771 osr(3.640 0x276dc10) 133400 bytes   (queue has 1 ops and 133400 bytes)
2014-06-12 14:46:51.610008 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.640( v 173'3014 (79'13,173'3014] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3012 active] do_sub_op osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:51.610031 7f9d62bd6700 10 osd.2 173 dequeue_op 0x1e3eac00 finish
2014-06-12 14:46:51.610034 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.610036 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.610042 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6900 (1 active)
2014-06-12 14:46:51.610049 7f9d7e96a700  5 _do_op 0x11004960 seq 2771 osr(3.640 0x276dc10)/0x276dc10 start
2014-06-12 14:46:51.610054 7f9d7e96a700 10 _do_transaction on 0x95482b0
2014-06-12 14:46:51.610059 7f9d7e96a700 15 _omap_setkeys meta/1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:51.610324 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:51.610351 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:51.610382 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:51.610401 7f9d7e96a700 15 _omap_rmkeys meta/1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:51.610408 7f9d7e96a700 10 _omap_rmkeys meta/1fe13b9a/pglog_3.640/0//-1 r = 0
2014-06-12 14:46:51.610412 7f9d7e96a700 15 _omap_setkeys meta/1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:51.610424 7f9d7e96a700 15 _write 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 0~131072
2014-06-12 14:46:51.610695 7f9d7e96a700 10 lookup_strip_header done  cid 3.640_head oid a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3
2014-06-12 14:46:51.610719 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:51.611126 7f9d7e96a700 10 _generic_write 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 0~131072 = 0
2014-06-12 14:46:51.611138 7f9d7e96a700 15 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3
2014-06-12 14:46:51.611146 7f9d7e96a700 10 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 = 0
2014-06-12 14:46:51.611151 7f9d7e96a700 15 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3
2014-06-12 14:46:51.611156 7f9d7e96a700 10 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 = 0
2014-06-12 14:46:51.611239 7f9d7e96a700 20 set_header setting 17474 cid 3.640_head oid a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 parent seq 0
2014-06-12 14:46:51.611257 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:51.611266 7f9d7e96a700 20 set_header setting 2200 cid meta oid 1fe13b9a/pglog_3.640/0//-1 parent seq 0
2014-06-12 14:46:51.611273 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:51.611708 7f9d7e96a700 10 _do_op 0x11004960 seq 2771 r = 0, finisher 0x85e10e0 0
2014-06-12 14:46:51.611717 7f9d7e96a700 10 _finish_op 0x11004960 seq 2771 osr(3.640 0x276dc10)/0x276dc10
2014-06-12 14:46:51.611724 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6900 (0 active)
2014-06-12 14:46:51.611727 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:51.611735 7f9d7715b700  5 -- op tracker -- , seq: 748088, time: 2014-06-12 14:46:51.611727, event: commit_sent, op: osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.611867 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3014 (79'13,173'3014] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3012 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:51.611888 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:51.611919 7f9d7795c700  5 -- op tracker -- , seq: 748088, time: 2014-06-12 14:46:51.611913, event: sub_op_applied, op: osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.611929 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3014 (79'13,173'3014] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3013 active] sub_op_modify_applied on 0x95481c0 op osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:51.611940 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3014 (79'13,173'3014] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3013 active] op_applied on primary on version 173'3014
2014-06-12 14:46:51.611946 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3014 (79'13,173'3014] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3013 active] op_applied on replica on version 173'3014
2014-06-12 14:46:51.611966 7f9d7795c700  5 -- op tracker -- , seq: 748088, time: 2014-06-12 14:46:51.611962, event: done, op: osd_sub_op(client.4316.0:147286 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3014 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:51.640781 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:51.641041 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:51.645105 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:51.645158 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:51.787260 7f9d44424700  5 -- op tracker -- , seq: 748089, time: 2014-06-12 14:46:51.786180, event: header_read, op: osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.787274 7f9d44424700  5 -- op tracker -- , seq: 748089, time: 2014-06-12 14:46:51.786190, event: throttled, op: osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.787284 7f9d44424700  5 -- op tracker -- , seq: 748089, time: 2014-06-12 14:46:51.787221, event: all_read, op: osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.787294 7f9d44424700  5 -- op tracker -- , seq: 748089, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.787302 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:51.787327 7f9d44424700 15 osd.2 173 enqueue_op 0x8dbce00 prio 63 cost 483328 latency 0.001147 osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.787358 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.787365 7f9d683e1700 10 osd.2 173 dequeue_op 0x8dbce00 prio 63 cost 483328 latency 0.001185 osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean]
2014-06-12 14:46:51.787391 7f9d683e1700  5 -- op tracker -- , seq: 748089, time: 2014-06-12 14:46:51.787386, event: reached_pg, op: osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.787401 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:51.787417 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] handle_message: 0x8dbce00
2014-06-12 14:46:51.787429 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] do_op osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:51.787453 7f9d683e1700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_'
2014-06-12 14:46:51.821683 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19423 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,1,0,0,0,0,0,0,13,2,4])
2014-06-12 14:46:51.821704 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19423 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,1,0,0,0,0,0,0,13,2,4])
2014-06-12 14:46:51.822085 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:51.822112 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:51.822137 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:51.822158 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:51.822188 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:51.822212 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:51.822245 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:51.822267 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:51.822289 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:51.822324 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:51.822354 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:51.822373 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:51.822392 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:51.822410 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:51.822427 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:51.822444 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:51.822490 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:51.822508 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:51.822524 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:51.822539 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:51.822837 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:51.822868 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:51.823385 7f9d683e1700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_' = 0
2014-06-12 14:46:51.823406 7f9d683e1700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset'
2014-06-12 14:46:51.823517 7f9d683e1700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset' = 0
2014-06-12 14:46:51.823527 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] populate_obc_watchers 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.823544 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.823556 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] get_object_context: creating obc from disk: 0x6727340
2014-06-12 14:46:51.823566 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] get_object_context: 0x6727340 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 rwstate(none n=0 w=0) oi: 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'688 client.4317.0:38412 wrlock_by=unknown.0.0:0 dirty s 4194304 uv688) ssc: 0x1e15f340 snapset: 0=[]:[]+head
2014-06-12 14:46:51.823584 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] find_object_context 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 @head oi=24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'688 client.4317.0:38412 wrlock_by=unknown.0.0:0 dirty s 4194304 uv688)
2014-06-12 14:46:51.823616 7f9d683e1700  5 -- op tracker -- , seq: 748089, time: 2014-06-12 14:46:51.823609, event: started, op: osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.823623 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] execute_ctx 0x1fc55600
2014-06-12 14:46:51.823642 7f9d683e1700  5 -- op tracker -- , seq: 748089, time: 2014-06-12 14:46:51.823636, event: started, op: osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.823651 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] do_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] ov 173'688 av 173'2888 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:51.823667 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] do_osd_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328]
2014-06-12 14:46:51.823681 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:51.823696 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] do_osd_op  write 0~483328
2014-06-12 14:46:51.823714 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:51.823727 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 snapset=0x1e15f378  snapc=0=[]
2014-06-12 14:46:51.823741 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:51.823753 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] finish_ctx 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 0x1fc55600 op modify  
2014-06-12 14:46:51.823767 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean]  set mtime to 2014-06-12 08:45:54.165686
2014-06-12 14:46:51.823790 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean]  final snapset 0=[]:[]+head in 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.823809 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean]  zeroing write result code 0
2014-06-12 14:46:51.823822 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] new_repop rep_tid 236337 on osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.823840 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] issue_repop rep_tid 236337 o 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.823868 7f9d683e1700  5 -- op tracker -- , seq: 748089, time: 2014-06-12 14:46:51.823862, event: waiting for subops from 28, op: osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.823886 7f9d683e1700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:51.823909 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2887 (0'0,173'2887] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] append_log log((0'0,173'2887], crt=0'0) [173'2888 (173'688) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143977 2014-06-12 08:45:54.165686]
2014-06-12 14:46:51.823928 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2887 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] add_log_entry 173'2888 (173'688) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143977 2014-06-12 08:45:54.165686
2014-06-12 14:46:51.823954 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2887 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] append_log  adding 1 keys
2014-06-12 14:46:51.823987 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2888, trimmed: 
2014-06-12 14:46:51.824010 7f9d683e1700  5 queue_transactions existing osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.824015 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:51.824017 7f9d683e1700  5 queue_op 0x13909bc0 seq 2711 osr(3.721 0x276de60) 485621 bytes   (queue has 1 ops and 485621 bytes)
2014-06-12 14:46:51.824028 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8360 (1 active)
2014-06-12 14:46:51.824024 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2887 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] eval_repop repgather(0x12ae2240 173'2888 rep_tid=236337 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.824039 7f9d8116f700  5 _do_op 0x13909bc0 seq 2711 osr(3.721 0x276de60)/0x276de60 start
2014-06-12 14:46:51.824044 7f9d8116f700 10 _do_transaction on 0x5112e40
2014-06-12 14:46:51.824043 7f9d683e1700 10 osd.2 173 dequeue_op 0x8dbce00 finish
2014-06-12 14:46:51.824048 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.824050 7f9d8116f700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.824052 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.824119 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.824145 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:51.824189 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:51.824207 7f9d8116f700 15 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.824215 7f9d8116f700 10 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1 r = 0
2014-06-12 14:46:51.824221 7f9d8116f700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.824235 7f9d8116f700 15 _write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 0~483328
2014-06-12 14:46:51.824365 7f9d8116f700 10 lookup_strip_header done  cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.824387 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:51.826122 7f9d8116f700 10 _generic_write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 0~483328 = 0
2014-06-12 14:46:51.826168 7f9d8116f700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.826183 7f9d8116f700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.826192 7f9d8116f700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.826201 7f9d8116f700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.826398 7f9d8116f700 20 set_header setting 16857 cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 parent seq 0
2014-06-12 14:46:51.826466 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:51.826481 7f9d8116f700 20 set_header setting 2130 cid meta oid 1fe0ff5a/pglog_3.721/0//-1 parent seq 0
2014-06-12 14:46:51.826493 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:51.828756 7f9d8116f700 10 _do_op 0x13909bc0 seq 2711 r = 0, finisher 0x786cbc0 0x198e6200
2014-06-12 14:46:51.828777 7f9d8116f700 10 _finish_op 0x13909bc0 seq 2711 osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.828785 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8360 (0 active)
2014-06-12 14:46:51.828789 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:51.828783 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2887 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] op_commit: 236337
2014-06-12 14:46:51.828814 7f9d7715b700  5 -- op tracker -- , seq: 748089, time: 2014-06-12 14:46:51.828806, event: op_commit, op: osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.828833 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2887 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] op_applied: 236337
2014-06-12 14:46:51.828867 7f9d7795c700  5 -- op tracker -- , seq: 748089, time: 2014-06-12 14:46:51.828857, event: op_applied, op: osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.828879 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2887 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] op_applied on primary on version 173'2888
2014-06-12 14:46:51.829431 7f9d5b6bc700  5 -- op tracker -- , seq: 748090, time: 2014-06-12 14:46:51.829384, event: header_read, op: osd_sub_op_reply(client.4317.0:143977 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.829444 7f9d5b6bc700  5 -- op tracker -- , seq: 748090, time: 2014-06-12 14:46:51.829387, event: throttled, op: osd_sub_op_reply(client.4317.0:143977 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.829454 7f9d5b6bc700  5 -- op tracker -- , seq: 748090, time: 2014-06-12 14:46:51.829399, event: all_read, op: osd_sub_op_reply(client.4317.0:143977 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.829464 7f9d5b6bc700  5 -- op tracker -- , seq: 748090, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143977 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.829470 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143977 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:51.829477 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:51.829485 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x2b014b00 prio 196 cost 0 latency 0.000101 osd_sub_op_reply(client.4317.0:143977 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:51.829520 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.829528 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2b014b00 prio 196 cost 0 latency 0.000144 osd_sub_op_reply(client.4317.0:143977 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 pg pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2887 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean]
2014-06-12 14:46:51.829549 7f9d6b3e7700  5 -- op tracker -- , seq: 748090, time: 2014-06-12 14:46:51.829545, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143977 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.829557 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2887 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] handle_message: 0x2b014b00
2014-06-12 14:46:51.829575 7f9d6b3e7700  5 -- op tracker -- , seq: 748090, time: 2014-06-12 14:46:51.829571, event: started, op: osd_sub_op_reply(client.4317.0:143977 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.829581 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2887 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] sub_op_modify_reply: tid 236337 op  ack_type 4 from 28
2014-06-12 14:46:51.829602 7f9d6b3e7700  5 -- op tracker -- , seq: 748089, time: 2014-06-12 14:46:51.829596, event: sub_op_commit_rec, op: osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.829611 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2887 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] repop_all_applied: repop tid 236337 all applied 
2014-06-12 14:46:51.829630 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2887 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] eval_repop repgather(0x12ae2240 173'2888 rep_tid=236337 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.829647 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2887 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean]  sending ack on repgather(0x12ae2240 173'2888 rep_tid=236337 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0xbc09180
2014-06-12 14:46:51.829680 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2887 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] repop_all_committed: repop tid 236337 all committed 
2014-06-12 14:46:51.829692 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2886 active+clean] eval_repop repgather(0x12ae2240 173'2888 rep_tid=236337 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.829713 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2886 active+clean] log_op_stats osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 inb 483607 outb 0 rlat 0.043498 lat 0.043532
2014-06-12 14:46:51.829734 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2886 active+clean] publish_stats_to_osd 173:3702
2014-06-12 14:46:51.829750 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2886 active+clean]  sending commit on repgather(0x12ae2240 173'2888 rep_tid=236337 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x15cd6c80
2014-06-12 14:46:51.829784 7f9d6b3e7700  5 -- op tracker -- , seq: 748089, time: 2014-06-12 14:46:51.829778, event: commit_sent, op: osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.829794 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2886 active+clean]  requeue_ops 
2014-06-12 14:46:51.829805 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean]  removing repgather(0x12ae2240 173'2888 rep_tid=236337 committed?=1 applied?=1 op=osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.829821 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean]    q front is repgather(0x12ae2240 173'2888 rep_tid=236337 committed?=1 applied?=1 op=osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.829836 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] remove_repop repgather(0x12ae2240 173'2888 rep_tid=236337 committed?=1 applied?=1 op=osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.829859 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean]  requeue_ops 
2014-06-12 14:46:51.829894 7f9d6b3e7700  5 -- op tracker -- , seq: 748089, time: 2014-06-12 14:46:51.829889, event: done, op: osd_op(client.4317.0:143977 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~483328] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.829913 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2b014b00 finish
2014-06-12 14:46:51.829923 7f9d6b3e7700  5 -- op tracker -- , seq: 748090, time: 2014-06-12 14:46:51.829919, event: done, op: osd_sub_op_reply(client.4317.0:143977 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.829933 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.829936 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.833707 7f9d44424700  5 -- op tracker -- , seq: 748091, time: 2014-06-12 14:46:51.832609, event: header_read, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.833727 7f9d44424700  5 -- op tracker -- , seq: 748091, time: 2014-06-12 14:46:51.832611, event: throttled, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.833738 7f9d44424700  5 -- op tracker -- , seq: 748091, time: 2014-06-12 14:46:51.833662, event: all_read, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.833763 7f9d44424700  5 -- op tracker -- , seq: 748091, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.833773 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:51.833804 7f9d44424700 15 osd.2 173 enqueue_op 0x68a6300 prio 63 cost 516096 latency 0.001194 osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.833839 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.833850 7f9d693e3700 10 osd.2 173 dequeue_op 0x68a6300 prio 63 cost 516096 latency 0.001241 osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean]
2014-06-12 14:46:51.833877 7f9d693e3700  5 -- op tracker -- , seq: 748091, time: 2014-06-12 14:46:51.833872, event: reached_pg, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.833886 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:51.833902 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] handle_message: 0x68a6300
2014-06-12 14:46:51.833912 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] do_op osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:51.833939 7f9d693e3700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_'
2014-06-12 14:46:51.834104 7f9d693e3700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_' = 0
2014-06-12 14:46:51.834124 7f9d693e3700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset'
2014-06-12 14:46:51.834213 7f9d693e3700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset' = 0
2014-06-12 14:46:51.834225 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] populate_obc_watchers 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.834239 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.834251 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] get_object_context: creating obc from disk: 0x17da5b80
2014-06-12 14:46:51.834263 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] get_object_context: 0x17da5b80 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 rwstate(none n=0 w=0) oi: 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2888 client.4317.0:143977 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2888) ssc: 0x1e15efc0 snapset: 0=[]:[]+head
2014-06-12 14:46:51.834281 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] find_object_context 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 @head oi=24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2888 client.4317.0:143977 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2888)
2014-06-12 14:46:51.834309 7f9d693e3700  5 -- op tracker -- , seq: 748091, time: 2014-06-12 14:46:51.834303, event: started, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.834319 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] execute_ctx 0x6245600
2014-06-12 14:46:51.834339 7f9d693e3700  5 -- op tracker -- , seq: 748091, time: 2014-06-12 14:46:51.834334, event: started, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.834348 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] do_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] ov 173'2888 av 173'2889 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:51.834364 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] do_osd_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096]
2014-06-12 14:46:51.834377 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:51.834392 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] do_osd_op  write 483328~516096
2014-06-12 14:46:51.834410 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:51.834423 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 snapset=0x1e15eff8  snapc=0=[]
2014-06-12 14:46:51.834437 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:51.834450 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] finish_ctx 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 0x6245600 op modify  
2014-06-12 14:46:51.834465 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean]  set mtime to 2014-06-12 08:45:54.211971
2014-06-12 14:46:51.834488 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean]  final snapset 0=[]:[]+head in 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.834509 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean]  zeroing write result code 0
2014-06-12 14:46:51.834522 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] new_repop rep_tid 236338 on osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.834539 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] issue_repop rep_tid 236338 o 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.834563 7f9d693e3700  5 -- op tracker -- , seq: 748091, time: 2014-06-12 14:46:51.834558, event: waiting for subops from 28, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.834584 7f9d693e3700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:51.834606 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2888 (0'0,173'2888] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] append_log log((0'0,173'2888], crt=0'0) [173'2889 (173'2888) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143978 2014-06-12 08:45:54.211971]
2014-06-12 14:46:51.834681 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] add_log_entry 173'2889 (173'2888) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143978 2014-06-12 08:45:54.211971
2014-06-12 14:46:51.834716 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] append_log  adding 1 keys
2014-06-12 14:46:51.834919 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2889, trimmed: 
2014-06-12 14:46:51.834943 7f9d693e3700  5 queue_transactions existing osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.834948 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:51.834950 7f9d693e3700  5 queue_op 0x5948000 seq 2712 osr(3.721 0x276de60) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:51.834957 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] eval_repop repgather(0x3403b740 173'2889 rep_tid=236338 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.834974 7f9d693e3700 10 osd.2 173 dequeue_op 0x68a6300 finish
2014-06-12 14:46:51.834978 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.834980 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.835002 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8360 (1 active)
2014-06-12 14:46:51.835010 7f9d7d167700  5 _do_op 0x5948000 seq 2712 osr(3.721 0x276de60)/0x276de60 start
2014-06-12 14:46:51.835014 7f9d7d167700 10 _do_transaction on 0x2dc1e900
2014-06-12 14:46:51.835021 7f9d7d167700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.835092 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.835119 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:51.835158 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:51.835178 7f9d7d167700 15 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.835184 7f9d7d167700 10 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1 r = 0
2014-06-12 14:46:51.835190 7f9d7d167700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.835204 7f9d7d167700 15 _write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 483328~516096
2014-06-12 14:46:51.835333 7f9d7d167700 10 lookup_strip_header done  cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.835504 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:51.837248 7f9d7d167700 10 _generic_write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 483328~516096 = 0
2014-06-12 14:46:51.837302 7f9d7d167700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.837318 7f9d7d167700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.837326 7f9d7d167700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.837335 7f9d7d167700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.837541 7f9d7d167700 20 set_header setting 16857 cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 parent seq 0
2014-06-12 14:46:51.837625 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:51.837640 7f9d7d167700 20 set_header setting 2130 cid meta oid 1fe0ff5a/pglog_3.721/0//-1 parent seq 0
2014-06-12 14:46:51.837654 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:51.840468 7f9d7d167700 10 _do_op 0x5948000 seq 2712 r = 0, finisher 0x654bec0 0x198e6040
2014-06-12 14:46:51.840482 7f9d7d167700 10 _finish_op 0x5948000 seq 2712 osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.840490 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8360 (0 active)
2014-06-12 14:46:51.840500 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:51.840488 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] op_commit: 236338
2014-06-12 14:46:51.840516 7f9d7715b700  5 -- op tracker -- , seq: 748091, time: 2014-06-12 14:46:51.840508, event: op_commit, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.840530 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] op_applied: 236338
2014-06-12 14:46:51.840554 7f9d7795c700  5 -- op tracker -- , seq: 748091, time: 2014-06-12 14:46:51.840547, event: op_applied, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.840575 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] op_applied on primary on version 173'2889
2014-06-12 14:46:51.841817 7f9d5b6bc700  5 -- op tracker -- , seq: 748092, time: 2014-06-12 14:46:51.841778, event: header_read, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.841827 7f9d5b6bc700  5 -- op tracker -- , seq: 748092, time: 2014-06-12 14:46:51.841780, event: throttled, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.841842 7f9d5b6bc700  5 -- op tracker -- , seq: 748092, time: 2014-06-12 14:46:51.841788, event: all_read, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.841847 7f9d5b6bc700  5 -- op tracker -- , seq: 748092, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.841854 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:51.841858 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:51.841864 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x17c2af00 prio 196 cost 0 latency 0.000086 osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0) v2
2014-06-12 14:46:51.841884 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.841891 7f9d66bde700 10 osd.2 173 dequeue_op 0x17c2af00 prio 196 cost 0 latency 0.000113 osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0) v2 pg pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean]
2014-06-12 14:46:51.841910 7f9d66bde700  5 -- op tracker -- , seq: 748092, time: 2014-06-12 14:46:51.841906, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.841925 7f9d5b6bc700  5 -- op tracker -- , seq: 748093, time: 2014-06-12 14:46:51.841879, event: header_read, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.841921 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] handle_message: 0x17c2af00
2014-06-12 14:46:51.841935 7f9d5b6bc700  5 -- op tracker -- , seq: 748093, time: 2014-06-12 14:46:51.841880, event: throttled, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.841955 7f9d5b6bc700  5 -- op tracker -- , seq: 748093, time: 2014-06-12 14:46:51.841891, event: all_read, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.841965 7f9d5b6bc700  5 -- op tracker -- , seq: 748093, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.841970 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:51.841975 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:51.841974 7f9d66bde700  5 -- op tracker -- , seq: 748092, time: 2014-06-12 14:46:51.841933, event: started, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.841980 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x2b015c00 prio 196 cost 0 latency 0.000101 osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:51.841985 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] sub_op_modify_reply: tid 236338 op  ack_type 1 from 28
2014-06-12 14:46:51.842004 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:51.842004 7f9d66bde700  5 -- op tracker -- , seq: 748091, time: 2014-06-12 14:46:51.841997, event: sub_op_applied_rec, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.842012 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] repop_all_applied: repop tid 236338 all applied 
2014-06-12 14:46:51.842022 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] eval_repop repgather(0x3403b740 173'2889 rep_tid=236338 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.842037 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean]  sending ack on repgather(0x3403b740 173'2889 rep_tid=236338 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x16cf2500
2014-06-12 14:46:51.842058 7f9d66bde700 10 osd.2 173 dequeue_op 0x17c2af00 finish
2014-06-12 14:46:51.842066 7f9d66bde700  5 -- op tracker -- , seq: 748092, time: 2014-06-12 14:46:51.842062, event: done, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.842079 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:51.842083 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.842083 7f9d64bda700 10 osd.2 173 dequeue_op 0x2b015c00 prio 196 cost 0 latency 0.000204 osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 pg pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean]
2014-06-12 14:46:51.842100 7f9d64bda700  5 -- op tracker -- , seq: 748093, time: 2014-06-12 14:46:51.842097, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.842106 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] handle_message: 0x2b015c00
2014-06-12 14:46:51.842115 7f9d64bda700  5 -- op tracker -- , seq: 748093, time: 2014-06-12 14:46:51.842113, event: started, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.842118 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] sub_op_modify_reply: tid 236338 op  ack_type 4 from 28
2014-06-12 14:46:51.842128 7f9d64bda700  5 -- op tracker -- , seq: 748091, time: 2014-06-12 14:46:51.842125, event: sub_op_commit_rec, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.842133 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2888 crt=0'0 lcod 173'2887 mlcod 173'2887 active+clean] repop_all_committed: repop tid 236338 all committed 
2014-06-12 14:46:51.842138 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2887 active+clean] eval_repop repgather(0x3403b740 173'2889 rep_tid=236338 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.842148 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2887 active+clean] log_op_stats osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.009447 lat 0.009538
2014-06-12 14:46:51.842161 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2887 active+clean] publish_stats_to_osd 173:3703
2014-06-12 14:46:51.842169 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2887 active+clean]  sending commit on repgather(0x3403b740 173'2889 rep_tid=236338 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x353e8f00
2014-06-12 14:46:51.842191 7f9d64bda700  5 -- op tracker -- , seq: 748091, time: 2014-06-12 14:46:51.842186, event: commit_sent, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.842198 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2887 active+clean]  requeue_ops 
2014-06-12 14:46:51.842209 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean]  removing repgather(0x3403b740 173'2889 rep_tid=236338 committed?=1 applied?=1 op=osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.842219 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean]    q front is repgather(0x3403b740 173'2889 rep_tid=236338 committed?=1 applied?=1 op=osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.842227 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] remove_repop repgather(0x3403b740 173'2889 rep_tid=236338 committed?=1 applied?=1 op=osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.842235 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean]  requeue_ops 
2014-06-12 14:46:51.842250 7f9d64bda700  5 -- op tracker -- , seq: 748091, time: 2014-06-12 14:46:51.842247, event: done, op: osd_op(client.4317.0:143978 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 483328~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.842261 7f9d64bda700 10 osd.2 173 dequeue_op 0x2b015c00 finish
2014-06-12 14:46:51.842266 7f9d64bda700  5 -- op tracker -- , seq: 748093, time: 2014-06-12 14:46:51.842264, event: done, op: osd_sub_op_reply(client.4317.0:143978 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.842272 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.842274 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.845728 7f9d44424700  5 -- op tracker -- , seq: 748094, time: 2014-06-12 14:46:51.844593, event: header_read, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.845742 7f9d44424700  5 -- op tracker -- , seq: 748094, time: 2014-06-12 14:46:51.844595, event: throttled, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.845753 7f9d44424700  5 -- op tracker -- , seq: 748094, time: 2014-06-12 14:46:51.845685, event: all_read, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.845763 7f9d44424700  5 -- op tracker -- , seq: 748094, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.845770 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:51.845797 7f9d44424700 15 osd.2 173 enqueue_op 0x68a7a00 prio 63 cost 516096 latency 0.001204 osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.845845 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.845851 7f9d6abe6700 10 osd.2 173 dequeue_op 0x68a7a00 prio 63 cost 516096 latency 0.001258 osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean]
2014-06-12 14:46:51.845874 7f9d6abe6700  5 -- op tracker -- , seq: 748094, time: 2014-06-12 14:46:51.845869, event: reached_pg, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.845883 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:51.845900 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] handle_message: 0x68a7a00
2014-06-12 14:46:51.845911 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] do_op osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:51.845952 7f9d6abe6700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_'
2014-06-12 14:46:51.846112 7f9d6abe6700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_' = 0
2014-06-12 14:46:51.846129 7f9d6abe6700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset'
2014-06-12 14:46:51.846213 7f9d6abe6700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset' = 0
2014-06-12 14:46:51.846224 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] populate_obc_watchers 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.846244 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.846253 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] get_object_context: creating obc from disk: 0x6ac1b80
2014-06-12 14:46:51.846263 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] get_object_context: 0x6ac1b80 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 rwstate(none n=0 w=0) oi: 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2889 client.4317.0:143978 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2889) ssc: 0x72b96c0 snapset: 0=[]:[]+head
2014-06-12 14:46:51.846278 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] find_object_context 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 @head oi=24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2889 client.4317.0:143978 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2889)
2014-06-12 14:46:51.846303 7f9d6abe6700  5 -- op tracker -- , seq: 748094, time: 2014-06-12 14:46:51.846298, event: started, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.846313 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] execute_ctx 0x7ad0c00
2014-06-12 14:46:51.846345 7f9d6abe6700  5 -- op tracker -- , seq: 748094, time: 2014-06-12 14:46:51.846340, event: started, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.846362 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] do_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] ov 173'2889 av 173'2890 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:51.846378 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] do_osd_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096]
2014-06-12 14:46:51.846389 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:51.846405 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] do_osd_op  write 999424~516096
2014-06-12 14:46:51.846422 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:51.846435 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 snapset=0x72b96f8  snapc=0=[]
2014-06-12 14:46:51.846449 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:51.846460 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] finish_ctx 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 0x7ad0c00 op modify  
2014-06-12 14:46:51.846475 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean]  set mtime to 2014-06-12 08:45:54.223956
2014-06-12 14:46:51.846512 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean]  final snapset 0=[]:[]+head in 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.846531 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean]  zeroing write result code 0
2014-06-12 14:46:51.846542 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] new_repop rep_tid 236339 on osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.846557 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] issue_repop rep_tid 236339 o 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.846581 7f9d6abe6700  5 -- op tracker -- , seq: 748094, time: 2014-06-12 14:46:51.846576, event: waiting for subops from 28, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.846618 7f9d6abe6700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:51.846639 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2889 (0'0,173'2889] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] append_log log((0'0,173'2889], crt=0'0) [173'2890 (173'2889) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143979 2014-06-12 08:45:54.223956]
2014-06-12 14:46:51.846664 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] add_log_entry 173'2890 (173'2889) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143979 2014-06-12 08:45:54.223956
2014-06-12 14:46:51.846688 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] append_log  adding 1 keys
2014-06-12 14:46:51.846722 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2890, trimmed: 
2014-06-12 14:46:51.846745 7f9d6abe6700  5 queue_transactions existing osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.846751 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:51.846753 7f9d6abe6700  5 queue_op 0x6b5bb60 seq 2713 osr(3.721 0x276de60) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:51.846760 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] eval_repop repgather(0x52f5bc0 173'2890 rep_tid=236339 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.846781 7f9d6abe6700 10 osd.2 173 dequeue_op 0x68a7a00 finish
2014-06-12 14:46:51.846786 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.846788 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.846776 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8360 (1 active)
2014-06-12 14:46:51.846828 7f9d79960700  5 _do_op 0x6b5bb60 seq 2713 osr(3.721 0x276de60)/0x276de60 start
2014-06-12 14:46:51.846847 7f9d79960700 10 _do_transaction on 0x51f6840
2014-06-12 14:46:51.846866 7f9d79960700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.846950 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.847088 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:51.847141 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:51.847163 7f9d79960700 15 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.847172 7f9d79960700 10 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1 r = 0
2014-06-12 14:46:51.847178 7f9d79960700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.847197 7f9d79960700 15 _write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 999424~516096
2014-06-12 14:46:51.847437 7f9d79960700 10 lookup_strip_header done  cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.847562 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:51.848808 7f9d79960700 10 _generic_write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 999424~516096 = 0
2014-06-12 14:46:51.848853 7f9d79960700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.848867 7f9d79960700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.848878 7f9d79960700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.848890 7f9d79960700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.849033 7f9d79960700 20 set_header setting 16857 cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 parent seq 0
2014-06-12 14:46:51.849087 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:51.849102 7f9d79960700 20 set_header setting 2130 cid meta oid 1fe0ff5a/pglog_3.721/0//-1 parent seq 0
2014-06-12 14:46:51.849117 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:51.850710 7f9d79960700 10 _do_op 0x6b5bb60 seq 2713 r = 0, finisher 0x172feee0 0x6be5d00
2014-06-12 14:46:51.850724 7f9d79960700 10 _finish_op 0x6b5bb60 seq 2713 osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.850732 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8360 (0 active)
2014-06-12 14:46:51.850738 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:51.850731 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] op_commit: 236339
2014-06-12 14:46:51.850762 7f9d7715b700  5 -- op tracker -- , seq: 748094, time: 2014-06-12 14:46:51.850752, event: op_commit, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.850806 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] op_applied: 236339
2014-06-12 14:46:51.850829 7f9d7795c700  5 -- op tracker -- , seq: 748094, time: 2014-06-12 14:46:51.850824, event: op_applied, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.850839 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] op_applied on primary on version 173'2890
2014-06-12 14:46:51.852181 7f9d5b6bc700  5 -- op tracker -- , seq: 748095, time: 2014-06-12 14:46:51.852133, event: header_read, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.852192 7f9d5b6bc700  5 -- op tracker -- , seq: 748095, time: 2014-06-12 14:46:51.852135, event: throttled, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.852200 7f9d5b6bc700  5 -- op tracker -- , seq: 748095, time: 2014-06-12 14:46:51.852146, event: all_read, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.852216 7f9d5b6bc700  5 -- op tracker -- , seq: 748095, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.852221 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:51.852227 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:51.852235 7f9d5b6bc700 15 osd.2 173 enqueue_op 0xe20bd00 prio 196 cost 0 latency 0.000102 osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0) v2
2014-06-12 14:46:51.852266 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.852278 7f9d6a3e5700 10 osd.2 173 dequeue_op 0xe20bd00 prio 196 cost 0 latency 0.000144 osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0) v2 pg pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean]
2014-06-12 14:46:51.852309 7f9d6a3e5700  5 -- op tracker -- , seq: 748095, time: 2014-06-12 14:46:51.852305, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.852327 7f9d5b6bc700  5 -- op tracker -- , seq: 748096, time: 2014-06-12 14:46:51.852261, event: header_read, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.852322 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] handle_message: 0xe20bd00
2014-06-12 14:46:51.852351 7f9d5b6bc700  5 -- op tracker -- , seq: 748096, time: 2014-06-12 14:46:51.852262, event: throttled, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.852361 7f9d5b6bc700  5 -- op tracker -- , seq: 748096, time: 2014-06-12 14:46:51.852276, event: all_read, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.852371 7f9d5b6bc700  5 -- op tracker -- , seq: 748096, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.852376 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:51.852381 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:51.852387 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x34017700 prio 196 cost 0 latency 0.000126 osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:51.852381 7f9d6a3e5700  5 -- op tracker -- , seq: 748095, time: 2014-06-12 14:46:51.852350, event: started, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.852397 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] sub_op_modify_reply: tid 236339 op  ack_type 1 from 28
2014-06-12 14:46:51.852424 7f9d6a3e5700  5 -- op tracker -- , seq: 748094, time: 2014-06-12 14:46:51.852410, event: sub_op_applied_rec, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.852437 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:51.852431 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] repop_all_applied: repop tid 236339 all applied 
2014-06-12 14:46:51.852442 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] eval_repop repgather(0x52f5bc0 173'2890 rep_tid=236339 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.852472 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean]  sending ack on repgather(0x52f5bc0 173'2890 rep_tid=236339 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x6700280
2014-06-12 14:46:51.852506 7f9d6a3e5700 10 osd.2 173 dequeue_op 0xe20bd00 finish
2014-06-12 14:46:51.852517 7f9d6a3e5700  5 -- op tracker -- , seq: 748095, time: 2014-06-12 14:46:51.852514, event: done, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.852539 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:51.852542 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.852546 7f9d69be4700 10 osd.2 173 dequeue_op 0x34017700 prio 196 cost 0 latency 0.000285 osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 pg pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean]
2014-06-12 14:46:51.852571 7f9d69be4700  5 -- op tracker -- , seq: 748096, time: 2014-06-12 14:46:51.852567, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.852587 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] handle_message: 0x34017700
2014-06-12 14:46:51.852605 7f9d69be4700  5 -- op tracker -- , seq: 748096, time: 2014-06-12 14:46:51.852601, event: started, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.852615 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] sub_op_modify_reply: tid 236339 op  ack_type 4 from 28
2014-06-12 14:46:51.852636 7f9d69be4700  5 -- op tracker -- , seq: 748094, time: 2014-06-12 14:46:51.852629, event: sub_op_commit_rec, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.852643 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2889 crt=0'0 lcod 173'2888 mlcod 173'2888 active+clean] repop_all_committed: repop tid 236339 all committed 
2014-06-12 14:46:51.852658 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2888 active+clean] eval_repop repgather(0x52f5bc0 173'2890 rep_tid=236339 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.852675 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2888 active+clean] log_op_stats osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.007911 lat 0.008081
2014-06-12 14:46:51.852694 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2888 active+clean] publish_stats_to_osd 173:3704
2014-06-12 14:46:51.852709 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2888 active+clean]  sending commit on repgather(0x52f5bc0 173'2890 rep_tid=236339 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x55a7400
2014-06-12 14:46:51.852746 7f9d69be4700  5 -- op tracker -- , seq: 748094, time: 2014-06-12 14:46:51.852740, event: commit_sent, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.852753 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2888 active+clean]  requeue_ops 
2014-06-12 14:46:51.852764 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean]  removing repgather(0x52f5bc0 173'2890 rep_tid=236339 committed?=1 applied?=1 op=osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.852781 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean]    q front is repgather(0x52f5bc0 173'2890 rep_tid=236339 committed?=1 applied?=1 op=osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.852797 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] remove_repop repgather(0x52f5bc0 173'2890 rep_tid=236339 committed?=1 applied?=1 op=osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.852813 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean]  requeue_ops 
2014-06-12 14:46:51.852840 7f9d69be4700  5 -- op tracker -- , seq: 748094, time: 2014-06-12 14:46:51.852834, event: done, op: osd_op(client.4317.0:143979 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 999424~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.852858 7f9d69be4700 10 osd.2 173 dequeue_op 0x34017700 finish
2014-06-12 14:46:51.852868 7f9d69be4700  5 -- op tracker -- , seq: 748096, time: 2014-06-12 14:46:51.852864, event: done, op: osd_sub_op_reply(client.4317.0:143979 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.852878 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.852882 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.856367 7f9d44424700  5 -- op tracker -- , seq: 748097, time: 2014-06-12 14:46:51.855129, event: header_read, op: osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.856381 7f9d44424700  5 -- op tracker -- , seq: 748097, time: 2014-06-12 14:46:51.855131, event: throttled, op: osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.856393 7f9d44424700  5 -- op tracker -- , seq: 748097, time: 2014-06-12 14:46:51.856332, event: all_read, op: osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.856405 7f9d44424700  5 -- op tracker -- , seq: 748097, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.856411 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:51.856438 7f9d44424700 15 osd.2 173 enqueue_op 0x6d11a00 prio 63 cost 516096 latency 0.001308 osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.856492 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.856502 7f9d653db700 10 osd.2 173 dequeue_op 0x6d11a00 prio 63 cost 516096 latency 0.001372 osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean]
2014-06-12 14:46:51.856523 7f9d653db700  5 -- op tracker -- , seq: 748097, time: 2014-06-12 14:46:51.856519, event: reached_pg, op: osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.856533 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:51.856545 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] handle_message: 0x6d11a00
2014-06-12 14:46:51.856554 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] do_op osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:51.856574 7f9d653db700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_'
2014-06-12 14:46:51.856684 7f9d653db700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_' = 0
2014-06-12 14:46:51.856697 7f9d653db700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset'
2014-06-12 14:46:51.856742 7f9d653db700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset' = 0
2014-06-12 14:46:51.856750 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] populate_obc_watchers 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.856759 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.856766 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] get_object_context: creating obc from disk: 0x1551c000
2014-06-12 14:46:51.856771 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] get_object_context: 0x1551c000 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 rwstate(none n=0 w=0) oi: 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2890 client.4317.0:143979 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2890) ssc: 0x5bb80e0 snapset: 0=[]:[]+head
2014-06-12 14:46:51.856782 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] find_object_context 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 @head oi=24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2890 client.4317.0:143979 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2890)
2014-06-12 14:46:51.856800 7f9d653db700  5 -- op tracker -- , seq: 748097, time: 2014-06-12 14:46:51.856797, event: started, op: osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.856807 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] execute_ctx 0x17cfb800
2014-06-12 14:46:51.856817 7f9d653db700  5 -- op tracker -- , seq: 748097, time: 2014-06-12 14:46:51.856814, event: started, op: osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.856822 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] do_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] ov 173'2890 av 173'2891 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:51.856831 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] do_osd_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096]
2014-06-12 14:46:51.856839 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:51.856848 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] do_osd_op  write 1515520~516096
2014-06-12 14:46:51.856860 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:51.856868 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 snapset=0x5bb8118  snapc=0=[]
2014-06-12 14:46:51.856875 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:51.856882 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] finish_ctx 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 0x17cfb800 op modify  
2014-06-12 14:46:51.856890 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean]  set mtime to 2014-06-12 08:45:54.234545
2014-06-12 14:46:51.856904 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean]  final snapset 0=[]:[]+head in 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.856920 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean]  zeroing write result code 0
2014-06-12 14:46:51.856929 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] new_repop rep_tid 236340 on osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.856944 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] issue_repop rep_tid 236340 o 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.856960 7f9d653db700  5 -- op tracker -- , seq: 748097, time: 2014-06-12 14:46:51.856957, event: waiting for subops from 28, op: osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.856975 7f9d653db700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:51.856990 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2890 (0'0,173'2890] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] append_log log((0'0,173'2890], crt=0'0) [173'2891 (173'2890) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143980 2014-06-12 08:45:54.234545]
2014-06-12 14:46:51.857007 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2890 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] add_log_entry 173'2891 (173'2890) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143980 2014-06-12 08:45:54.234545
2014-06-12 14:46:51.857020 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2890 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] append_log  adding 1 keys
2014-06-12 14:46:51.857040 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2891, trimmed: 
2014-06-12 14:46:51.857060 7f9d653db700  5 queue_transactions existing osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.857063 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:51.857064 7f9d653db700  5 queue_op 0xfa430e0 seq 2714 osr(3.721 0x276de60) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:51.857074 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8360 (1 active)
2014-06-12 14:46:51.857069 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2890 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] eval_repop repgather(0x519d380 173'2891 rep_tid=236340 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.857081 7f9d653db700 10 osd.2 173 dequeue_op 0x6d11a00 finish
2014-06-12 14:46:51.857081 7f9d8016d700  5 _do_op 0xfa430e0 seq 2714 osr(3.721 0x276de60)/0x276de60 start
2014-06-12 14:46:51.857083 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.857085 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.857085 7f9d8016d700 10 _do_transaction on 0x519dec0
2014-06-12 14:46:51.857091 7f9d8016d700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.857163 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.857189 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:51.857234 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:51.857258 7f9d8016d700 15 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.857266 7f9d8016d700 10 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1 r = 0
2014-06-12 14:46:51.857272 7f9d8016d700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.857290 7f9d8016d700 15 _write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 1515520~516096
2014-06-12 14:46:51.857628 7f9d8016d700 10 lookup_strip_header done  cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.857793 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:51.859840 7f9d8016d700 10 _generic_write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 1515520~516096 = 0
2014-06-12 14:46:51.859903 7f9d8016d700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.859924 7f9d8016d700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.859934 7f9d8016d700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.859943 7f9d8016d700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.860154 7f9d8016d700 20 set_header setting 16857 cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 parent seq 0
2014-06-12 14:46:51.860235 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:51.860252 7f9d8016d700 20 set_header setting 2130 cid meta oid 1fe0ff5a/pglog_3.721/0//-1 parent seq 0
2014-06-12 14:46:51.860262 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:51.862168 7f9d5b6bc700  5 -- op tracker -- , seq: 748098, time: 2014-06-12 14:46:51.862117, event: header_read, op: osd_sub_op_reply(client.4317.0:143980 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.862179 7f9d5b6bc700  5 -- op tracker -- , seq: 748098, time: 2014-06-12 14:46:51.862119, event: throttled, op: osd_sub_op_reply(client.4317.0:143980 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.862204 7f9d5b6bc700  5 -- op tracker -- , seq: 748098, time: 2014-06-12 14:46:51.862132, event: all_read, op: osd_sub_op_reply(client.4317.0:143980 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.862210 7f9d5b6bc700  5 -- op tracker -- , seq: 748098, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143980 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.862214 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143980 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:51.862219 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:51.862226 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x34017100 prio 196 cost 0 latency 0.000109 osd_sub_op_reply(client.4317.0:143980 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:51.862254 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.862262 7f9d63bd8700 10 osd.2 173 dequeue_op 0x34017100 prio 196 cost 0 latency 0.000145 osd_sub_op_reply(client.4317.0:143980 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 pg pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2890 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean]
2014-06-12 14:46:51.862282 7f9d63bd8700  5 -- op tracker -- , seq: 748098, time: 2014-06-12 14:46:51.862280, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143980 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.862289 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2890 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] handle_message: 0x34017100
2014-06-12 14:46:51.862299 7f9d63bd8700  5 -- op tracker -- , seq: 748098, time: 2014-06-12 14:46:51.862297, event: started, op: osd_sub_op_reply(client.4317.0:143980 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.862303 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2890 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] sub_op_modify_reply: tid 236340 op  ack_type 4 from 28
2014-06-12 14:46:51.862317 7f9d63bd8700  5 -- op tracker -- , seq: 748097, time: 2014-06-12 14:46:51.862311, event: sub_op_commit_rec, op: osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.862322 7f9d63bd8700 10 osd.2 173 dequeue_op 0x34017100 finish
2014-06-12 14:46:51.862326 7f9d63bd8700  5 -- op tracker -- , seq: 748098, time: 2014-06-12 14:46:51.862324, event: done, op: osd_sub_op_reply(client.4317.0:143980 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.862344 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.862346 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.863086 7f9d8016d700 10 _do_op 0xfa430e0 seq 2714 r = 0, finisher 0x172fe0e0 0x2f966240
2014-06-12 14:46:51.863101 7f9d8016d700 10 _finish_op 0xfa430e0 seq 2714 osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.863110 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8360 (0 active)
2014-06-12 14:46:51.863150 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:51.863124 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2890 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] op_applied: 236340
2014-06-12 14:46:51.863194 7f9d7795c700  5 -- op tracker -- , seq: 748097, time: 2014-06-12 14:46:51.863186, event: op_applied, op: osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.863207 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2890 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] op_applied on primary on version 173'2891
2014-06-12 14:46:51.863217 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2890 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] repop_all_applied: repop tid 236340 all applied 
2014-06-12 14:46:51.863223 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2890 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] eval_repop repgather(0x519d380 173'2891 rep_tid=236340 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.863234 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2890 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean]  sending ack on repgather(0x519d380 173'2891 rep_tid=236340 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x22ef7180
2014-06-12 14:46:51.863264 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2890 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] op_commit: 236340
2014-06-12 14:46:51.863287 7f9d7715b700  5 -- op tracker -- , seq: 748097, time: 2014-06-12 14:46:51.863282, event: op_commit, op: osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.863293 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2890 crt=0'0 lcod 173'2889 mlcod 173'2889 active+clean] repop_all_committed: repop tid 236340 all committed 
2014-06-12 14:46:51.863300 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2889 active+clean] eval_repop repgather(0x519d380 173'2891 rep_tid=236340 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.863311 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2889 active+clean] log_op_stats osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008127 lat 0.008181
2014-06-12 14:46:51.863322 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2889 active+clean] publish_stats_to_osd 173:3705
2014-06-12 14:46:51.863330 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2889 active+clean]  sending commit on repgather(0x519d380 173'2891 rep_tid=236340 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x28acf680
2014-06-12 14:46:51.863350 7f9d7715b700  5 -- op tracker -- , seq: 748097, time: 2014-06-12 14:46:51.863347, event: commit_sent, op: osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.863355 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2889 active+clean]  requeue_ops 
2014-06-12 14:46:51.863361 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean]  removing repgather(0x519d380 173'2891 rep_tid=236340 committed?=1 applied?=1 op=osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.863370 7f9d7715b700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean]    q front is repgather(0x519d380 173'2891 rep_tid=236340 committed?=1 applied?=1 op=osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.863378 7f9d7715b700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] remove_repop repgather(0x519d380 173'2891 rep_tid=236340 committed?=1 applied?=1 op=osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.863386 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean]  requeue_ops 
2014-06-12 14:46:51.863404 7f9d7715b700  5 -- op tracker -- , seq: 748097, time: 2014-06-12 14:46:51.863401, event: done, op: osd_op(client.4317.0:143980 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 1515520~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.863721 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:51.863724 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:51.863725 7f9d74bfa700 20 osd.2 173 _dispatch 0x177041a0 ping v1
2014-06-12 14:46:51.863727 7f9d74bfa700 10 osd.2 173 ping from client.4314
2014-06-12 14:46:51.863729 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:51.863730 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:51.866842 7f9d44424700  5 -- op tracker -- , seq: 748099, time: 2014-06-12 14:46:51.865667, event: header_read, op: osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.866859 7f9d44424700  5 -- op tracker -- , seq: 748099, time: 2014-06-12 14:46:51.865669, event: throttled, op: osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.866868 7f9d44424700  5 -- op tracker -- , seq: 748099, time: 2014-06-12 14:46:51.866798, event: all_read, op: osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.866876 7f9d44424700  5 -- op tracker -- , seq: 748099, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.866884 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:51.866909 7f9d44424700 15 osd.2 173 enqueue_op 0x6d11700 prio 63 cost 516096 latency 0.001242 osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.866945 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.866954 7f9d68be2700 10 osd.2 173 dequeue_op 0x6d11700 prio 63 cost 516096 latency 0.001287 osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean]
2014-06-12 14:46:51.866978 7f9d68be2700  5 -- op tracker -- , seq: 748099, time: 2014-06-12 14:46:51.866974, event: reached_pg, op: osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.866988 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:51.867004 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] handle_message: 0x6d11700
2014-06-12 14:46:51.867014 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] do_op osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:51.867038 7f9d68be2700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_'
2014-06-12 14:46:51.867195 7f9d68be2700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_' = 0
2014-06-12 14:46:51.867213 7f9d68be2700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset'
2014-06-12 14:46:51.867273 7f9d68be2700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset' = 0
2014-06-12 14:46:51.867282 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] populate_obc_watchers 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.867291 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.867297 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] get_object_context: creating obc from disk: 0x2060eb00
2014-06-12 14:46:51.867302 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] get_object_context: 0x2060eb00 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 rwstate(none n=0 w=0) oi: 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2891 client.4317.0:143980 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2891) ssc: 0x407e000 snapset: 0=[]:[]+head
2014-06-12 14:46:51.867314 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] find_object_context 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 @head oi=24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2891 client.4317.0:143980 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2891)
2014-06-12 14:46:51.867333 7f9d68be2700  5 -- op tracker -- , seq: 748099, time: 2014-06-12 14:46:51.867329, event: started, op: osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.867339 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] execute_ctx 0x33b2d800
2014-06-12 14:46:51.867350 7f9d68be2700  5 -- op tracker -- , seq: 748099, time: 2014-06-12 14:46:51.867347, event: started, op: osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.867355 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] do_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] ov 173'2891 av 173'2892 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:51.867365 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] do_osd_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096]
2014-06-12 14:46:51.867372 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:51.867382 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] do_osd_op  write 2031616~516096
2014-06-12 14:46:51.867393 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:51.867401 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 snapset=0x407e038  snapc=0=[]
2014-06-12 14:46:51.867409 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:51.867415 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] finish_ctx 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 0x33b2d800 op modify  
2014-06-12 14:46:51.867426 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean]  set mtime to 2014-06-12 08:45:54.245130
2014-06-12 14:46:51.867440 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean]  final snapset 0=[]:[]+head in 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.867456 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean]  zeroing write result code 0
2014-06-12 14:46:51.867464 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] new_repop rep_tid 236341 on osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.867476 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] issue_repop rep_tid 236341 o 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.867493 7f9d68be2700  5 -- op tracker -- , seq: 748099, time: 2014-06-12 14:46:51.867490, event: waiting for subops from 28, op: osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.867509 7f9d68be2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:51.867526 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2891 (0'0,173'2891] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] append_log log((0'0,173'2891], crt=0'0) [173'2892 (173'2891) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143981 2014-06-12 08:45:54.245130]
2014-06-12 14:46:51.867542 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2891 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] add_log_entry 173'2892 (173'2891) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143981 2014-06-12 08:45:54.245130
2014-06-12 14:46:51.867556 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2891 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] append_log  adding 1 keys
2014-06-12 14:46:51.867577 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2892, trimmed: 
2014-06-12 14:46:51.867595 7f9d68be2700  5 queue_transactions existing osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.867599 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:51.867600 7f9d68be2700  5 queue_op 0x6478c00 seq 2715 osr(3.721 0x276de60) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:51.867609 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8360 (1 active)
2014-06-12 14:46:51.867615 7f9d7915f700  5 _do_op 0x6478c00 seq 2715 osr(3.721 0x276de60)/0x276de60 start
2014-06-12 14:46:51.867619 7f9d7915f700 10 _do_transaction on 0x1b72b40
2014-06-12 14:46:51.867605 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2891 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] eval_repop repgather(0x2681980 173'2892 rep_tid=236341 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.867623 7f9d68be2700 10 osd.2 173 dequeue_op 0x6d11700 finish
2014-06-12 14:46:51.867625 7f9d7915f700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.867626 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.867628 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.867680 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.867705 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:51.867734 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:51.867753 7f9d7915f700 15 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.867758 7f9d7915f700 10 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1 r = 0
2014-06-12 14:46:51.867762 7f9d7915f700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.867910 7f9d7915f700 15 _write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 2031616~516096
2014-06-12 14:46:51.867992 7f9d7915f700 10 lookup_strip_header done  cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.868106 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:51.869296 7f9d7915f700 10 _generic_write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 2031616~516096 = 0
2014-06-12 14:46:51.869331 7f9d7915f700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.869340 7f9d7915f700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.869347 7f9d7915f700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.869352 7f9d7915f700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.869488 7f9d7915f700 20 set_header setting 16857 cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 parent seq 0
2014-06-12 14:46:51.869537 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:51.869547 7f9d7915f700 20 set_header setting 2130 cid meta oid 1fe0ff5a/pglog_3.721/0//-1 parent seq 0
2014-06-12 14:46:51.869555 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:51.871297 7f9d7915f700 10 _do_op 0x6478c00 seq 2715 r = 0, finisher 0x428a3e0 0x28eb15c0
2014-06-12 14:46:51.871310 7f9d7915f700 10 _finish_op 0x6478c00 seq 2715 osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.871317 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8360 (0 active)
2014-06-12 14:46:51.871324 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:51.871315 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2891 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] op_commit: 236341
2014-06-12 14:46:51.871344 7f9d7715b700  5 -- op tracker -- , seq: 748099, time: 2014-06-12 14:46:51.871335, event: op_commit, op: osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.871361 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2891 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] op_applied: 236341
2014-06-12 14:46:51.871378 7f9d7795c700  5 -- op tracker -- , seq: 748099, time: 2014-06-12 14:46:51.871373, event: op_applied, op: osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.871386 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2891 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] op_applied on primary on version 173'2892
2014-06-12 14:46:51.873347 7f9d5b6bc700  5 -- op tracker -- , seq: 748100, time: 2014-06-12 14:46:51.873300, event: header_read, op: osd_sub_op_reply(client.4317.0:143981 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.873368 7f9d5b6bc700  5 -- op tracker -- , seq: 748100, time: 2014-06-12 14:46:51.873302, event: throttled, op: osd_sub_op_reply(client.4317.0:143981 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.873373 7f9d5b6bc700  5 -- op tracker -- , seq: 748100, time: 2014-06-12 14:46:51.873314, event: all_read, op: osd_sub_op_reply(client.4317.0:143981 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.873377 7f9d5b6bc700  5 -- op tracker -- , seq: 748100, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143981 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.873381 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143981 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:51.873385 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:51.873392 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x34016300 prio 196 cost 0 latency 0.000092 osd_sub_op_reply(client.4317.0:143981 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:51.873413 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.873421 7f9d623d5700 10 osd.2 173 dequeue_op 0x34016300 prio 196 cost 0 latency 0.000120 osd_sub_op_reply(client.4317.0:143981 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 pg pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2891 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean]
2014-06-12 14:46:51.873439 7f9d623d5700  5 -- op tracker -- , seq: 748100, time: 2014-06-12 14:46:51.873435, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143981 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.873452 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2891 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] handle_message: 0x34016300
2014-06-12 14:46:51.873465 7f9d623d5700  5 -- op tracker -- , seq: 748100, time: 2014-06-12 14:46:51.873463, event: started, op: osd_sub_op_reply(client.4317.0:143981 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.873473 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2891 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] sub_op_modify_reply: tid 236341 op  ack_type 4 from 28
2014-06-12 14:46:51.873487 7f9d623d5700  5 -- op tracker -- , seq: 748099, time: 2014-06-12 14:46:51.873481, event: sub_op_commit_rec, op: osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.873492 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2891 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] repop_all_applied: repop tid 236341 all applied 
2014-06-12 14:46:51.873499 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2891 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] eval_repop repgather(0x2681980 173'2892 rep_tid=236341 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.873508 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2891 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean]  sending ack on repgather(0x2681980 173'2892 rep_tid=236341 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x38137900
2014-06-12 14:46:51.873530 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2891 crt=0'0 lcod 173'2890 mlcod 173'2890 active+clean] repop_all_committed: repop tid 236341 all committed 
2014-06-12 14:46:51.873536 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2890 active+clean] eval_repop repgather(0x2681980 173'2892 rep_tid=236341 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.873546 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2890 active+clean] log_op_stats osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.007862 lat 0.007878
2014-06-12 14:46:51.873557 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2890 active+clean] publish_stats_to_osd 173:3706
2014-06-12 14:46:51.873565 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2890 active+clean]  sending commit on repgather(0x2681980 173'2892 rep_tid=236341 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x133ff900
2014-06-12 14:46:51.873583 7f9d623d5700  5 -- op tracker -- , seq: 748099, time: 2014-06-12 14:46:51.873580, event: commit_sent, op: osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.873588 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2890 active+clean]  requeue_ops 
2014-06-12 14:46:51.873595 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean]  removing repgather(0x2681980 173'2892 rep_tid=236341 committed?=1 applied?=1 op=osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.873604 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean]    q front is repgather(0x2681980 173'2892 rep_tid=236341 committed?=1 applied?=1 op=osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.873619 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] remove_repop repgather(0x2681980 173'2892 rep_tid=236341 committed?=1 applied?=1 op=osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.873627 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean]  requeue_ops 
2014-06-12 14:46:51.873645 7f9d623d5700  5 -- op tracker -- , seq: 748099, time: 2014-06-12 14:46:51.873642, event: done, op: osd_op(client.4317.0:143981 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.873659 7f9d623d5700 10 osd.2 173 dequeue_op 0x34016300 finish
2014-06-12 14:46:51.873663 7f9d623d5700  5 -- op tracker -- , seq: 748100, time: 2014-06-12 14:46:51.873661, event: done, op: osd_sub_op_reply(client.4317.0:143981 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.873675 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.873677 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.876882 7f9d44424700  5 -- op tracker -- , seq: 748101, time: 2014-06-12 14:46:51.875823, event: header_read, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.876891 7f9d44424700  5 -- op tracker -- , seq: 748101, time: 2014-06-12 14:46:51.875825, event: throttled, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.876913 7f9d44424700  5 -- op tracker -- , seq: 748101, time: 2014-06-12 14:46:51.876851, event: all_read, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.876922 7f9d44424700  5 -- op tracker -- , seq: 748101, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.876927 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:51.876947 7f9d44424700 15 osd.2 173 enqueue_op 0x6d11500 prio 63 cost 516096 latency 0.001124 osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.876971 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.876981 7f9d643d9700 10 osd.2 173 dequeue_op 0x6d11500 prio 63 cost 516096 latency 0.001158 osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean]
2014-06-12 14:46:51.877009 7f9d643d9700  5 -- op tracker -- , seq: 748101, time: 2014-06-12 14:46:51.877003, event: reached_pg, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.877018 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:51.877035 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] handle_message: 0x6d11500
2014-06-12 14:46:51.877046 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] do_op osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:51.877073 7f9d643d9700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_'
2014-06-12 14:46:51.877195 7f9d643d9700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_' = 0
2014-06-12 14:46:51.877212 7f9d643d9700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset'
2014-06-12 14:46:51.877267 7f9d643d9700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset' = 0
2014-06-12 14:46:51.877277 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] populate_obc_watchers 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.877291 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.877302 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] get_object_context: creating obc from disk: 0x66bc840
2014-06-12 14:46:51.877312 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] get_object_context: 0x66bc840 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 rwstate(none n=0 w=0) oi: 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2892 client.4317.0:143981 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2892) ssc: 0x5d5bb20 snapset: 0=[]:[]+head
2014-06-12 14:46:51.877328 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] find_object_context 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 @head oi=24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2892 client.4317.0:143981 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2892)
2014-06-12 14:46:51.877356 7f9d643d9700  5 -- op tracker -- , seq: 748101, time: 2014-06-12 14:46:51.877350, event: started, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.877364 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] execute_ctx 0x30f7e400
2014-06-12 14:46:51.877382 7f9d643d9700  5 -- op tracker -- , seq: 748101, time: 2014-06-12 14:46:51.877376, event: started, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.877391 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] do_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] ov 173'2892 av 173'2893 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:51.877407 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] do_osd_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096]
2014-06-12 14:46:51.877419 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:51.877432 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] do_osd_op  write 2547712~516096
2014-06-12 14:46:51.877448 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:51.877460 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 snapset=0x5d5bb58  snapc=0=[]
2014-06-12 14:46:51.877473 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:51.877486 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] finish_ctx 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 0x30f7e400 op modify  
2014-06-12 14:46:51.877499 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean]  set mtime to 2014-06-12 08:45:54.255273
2014-06-12 14:46:51.877521 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean]  final snapset 0=[]:[]+head in 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.877545 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean]  zeroing write result code 0
2014-06-12 14:46:51.877557 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] new_repop rep_tid 236342 on osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.877575 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] issue_repop rep_tid 236342 o 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.877602 7f9d643d9700  5 -- op tracker -- , seq: 748101, time: 2014-06-12 14:46:51.877597, event: waiting for subops from 28, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.877635 7f9d643d9700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:51.877657 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2892 (0'0,173'2892] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] append_log log((0'0,173'2892], crt=0'0) [173'2893 (173'2892) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143982 2014-06-12 08:45:54.255273]
2014-06-12 14:46:51.877677 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] add_log_entry 173'2893 (173'2892) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143982 2014-06-12 08:45:54.255273
2014-06-12 14:46:51.877698 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] append_log  adding 1 keys
2014-06-12 14:46:51.877731 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2893, trimmed: 
2014-06-12 14:46:51.877758 7f9d643d9700  5 queue_transactions existing osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.877764 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:51.877765 7f9d643d9700  5 queue_op 0x6d04ea0 seq 2716 osr(3.721 0x276de60) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:51.877778 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8360 (1 active)
2014-06-12 14:46:51.877786 7f9d7d968700  5 _do_op 0x6d04ea0 seq 2716 osr(3.721 0x276de60)/0x276de60 start
2014-06-12 14:46:51.877772 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] eval_repop repgather(0x1687c9c0 173'2893 rep_tid=236342 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.877790 7f9d7d968700 10 _do_transaction on 0x11c7e180
2014-06-12 14:46:51.877792 7f9d643d9700 10 osd.2 173 dequeue_op 0x6d11500 finish
2014-06-12 14:46:51.877796 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.877797 7f9d7d968700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.877799 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.877958 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.877986 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:51.878039 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:51.878065 7f9d7d968700 15 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.878073 7f9d7d968700 10 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1 r = 0
2014-06-12 14:46:51.878079 7f9d7d968700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.878102 7f9d7d968700 15 _write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 2547712~516096
2014-06-12 14:46:51.878385 7f9d7d968700 10 lookup_strip_header done  cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.878529 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:51.880264 7f9d7d968700 10 _generic_write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 2547712~516096 = 0
2014-06-12 14:46:51.880308 7f9d7d968700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.880322 7f9d7d968700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.880331 7f9d7d968700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.880339 7f9d7d968700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.880545 7f9d7d968700 20 set_header setting 16857 cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 parent seq 0
2014-06-12 14:46:51.880619 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:51.880635 7f9d7d968700 20 set_header setting 2130 cid meta oid 1fe0ff5a/pglog_3.721/0//-1 parent seq 0
2014-06-12 14:46:51.880647 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:51.883803 7f9d7d968700 10 _do_op 0x6d04ea0 seq 2716 r = 0, finisher 0x31d0d800 0xd945a80
2014-06-12 14:46:51.883828 7f9d7d968700 10 _finish_op 0x6d04ea0 seq 2716 osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.883832 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] op_commit: 236342
2014-06-12 14:46:51.883850 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8360 (0 active)
2014-06-12 14:46:51.883862 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:51.883876 7f9d7715b700  5 -- op tracker -- , seq: 748101, time: 2014-06-12 14:46:51.883861, event: op_commit, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.883911 7f9d5b6bc700  5 -- op tracker -- , seq: 748102, time: 2014-06-12 14:46:51.883865, event: header_read, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.883922 7f9d5b6bc700  5 -- op tracker -- , seq: 748102, time: 2014-06-12 14:46:51.883867, event: throttled, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.883912 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] op_applied: 236342
2014-06-12 14:46:51.883927 7f9d5b6bc700  5 -- op tracker -- , seq: 748102, time: 2014-06-12 14:46:51.883882, event: all_read, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.883935 7f9d5b6bc700  5 -- op tracker -- , seq: 748102, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.883941 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:51.883946 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:51.883947 7f9d7795c700  5 -- op tracker -- , seq: 748101, time: 2014-06-12 14:46:51.883928, event: op_applied, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.883952 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x20e86900 prio 196 cost 0 latency 0.000087 osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0) v2
2014-06-12 14:46:51.883957 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] op_applied on primary on version 173'2893
2014-06-12 14:46:51.883977 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.883985 7f9d633d7700 10 osd.2 173 dequeue_op 0x20e86900 prio 196 cost 0 latency 0.000120 osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0) v2 pg pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean]
2014-06-12 14:46:51.884004 7f9d633d7700  5 -- op tracker -- , seq: 748102, time: 2014-06-12 14:46:51.884000, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.884012 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] handle_message: 0x20e86900
2014-06-12 14:46:51.884014 7f9d5b6bc700  5 -- op tracker -- , seq: 748103, time: 2014-06-12 14:46:51.883971, event: header_read, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.884036 7f9d5b6bc700  5 -- op tracker -- , seq: 748103, time: 2014-06-12 14:46:51.883971, event: throttled, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.884046 7f9d5b6bc700  5 -- op tracker -- , seq: 748103, time: 2014-06-12 14:46:51.883995, event: all_read, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.884064 7f9d5b6bc700  5 -- op tracker -- , seq: 748103, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.884070 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:51.884075 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:51.884075 7f9d633d7700  5 -- op tracker -- , seq: 748102, time: 2014-06-12 14:46:51.884023, event: started, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.884081 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x20e87e00 prio 196 cost 0 latency 0.000110 osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:51.884086 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] sub_op_modify_reply: tid 236342 op  ack_type 1 from 28
2014-06-12 14:46:51.884102 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:51.884107 7f9d633d7700  5 -- op tracker -- , seq: 748101, time: 2014-06-12 14:46:51.884102, event: sub_op_applied_rec, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.884113 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] repop_all_applied: repop tid 236342 all applied 
2014-06-12 14:46:51.884120 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] eval_repop repgather(0x1687c9c0 173'2893 rep_tid=236342 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.884130 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean]  sending ack on repgather(0x1687c9c0 173'2893 rep_tid=236342 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x30ccaa00
2014-06-12 14:46:51.884150 7f9d633d7700 10 osd.2 173 dequeue_op 0x20e86900 finish
2014-06-12 14:46:51.884159 7f9d633d7700  5 -- op tracker -- , seq: 748102, time: 2014-06-12 14:46:51.884156, event: done, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.884174 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:51.884161 7f9d61bd4700 10 osd.2 173 dequeue_op 0x20e87e00 prio 196 cost 0 latency 0.000190 osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 pg pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean]
2014-06-12 14:46:51.884177 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.884182 7f9d61bd4700  5 -- op tracker -- , seq: 748103, time: 2014-06-12 14:46:51.884178, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.884190 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] handle_message: 0x20e87e00
2014-06-12 14:46:51.884212 7f9d61bd4700  5 -- op tracker -- , seq: 748103, time: 2014-06-12 14:46:51.884208, event: started, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.884217 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] sub_op_modify_reply: tid 236342 op  ack_type 4 from 28
2014-06-12 14:46:51.884235 7f9d61bd4700  5 -- op tracker -- , seq: 748101, time: 2014-06-12 14:46:51.884228, event: sub_op_commit_rec, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.884241 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2892 crt=0'0 lcod 173'2891 mlcod 173'2891 active+clean] repop_all_committed: repop tid 236342 all committed 
2014-06-12 14:46:51.884252 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2891 active+clean] eval_repop repgather(0x1687c9c0 173'2893 rep_tid=236342 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.884268 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2891 active+clean] log_op_stats osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008326 lat 0.008444
2014-06-12 14:46:51.884286 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2891 active+clean] publish_stats_to_osd 173:3707
2014-06-12 14:46:51.884302 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2891 active+clean]  sending commit on repgather(0x1687c9c0 173'2893 rep_tid=236342 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x5958000
2014-06-12 14:46:51.884335 7f9d61bd4700  5 -- op tracker -- , seq: 748101, time: 2014-06-12 14:46:51.884329, event: commit_sent, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.884353 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2891 active+clean]  requeue_ops 
2014-06-12 14:46:51.884364 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean]  removing repgather(0x1687c9c0 173'2893 rep_tid=236342 committed?=1 applied?=1 op=osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.884379 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean]    q front is repgather(0x1687c9c0 173'2893 rep_tid=236342 committed?=1 applied?=1 op=osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.884393 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] remove_repop repgather(0x1687c9c0 173'2893 rep_tid=236342 committed?=1 applied?=1 op=osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.884414 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean]  requeue_ops 
2014-06-12 14:46:51.884439 7f9d61bd4700  5 -- op tracker -- , seq: 748101, time: 2014-06-12 14:46:51.884433, event: done, op: osd_op(client.4317.0:143982 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 2547712~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.884465 7f9d61bd4700 10 osd.2 173 dequeue_op 0x20e87e00 finish
2014-06-12 14:46:51.884473 7f9d61bd4700  5 -- op tracker -- , seq: 748103, time: 2014-06-12 14:46:51.884469, event: done, op: osd_sub_op_reply(client.4317.0:143982 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.884494 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.884497 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.887742 7f9d44424700  5 -- op tracker -- , seq: 748104, time: 2014-06-12 14:46:51.886632, event: header_read, op: osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.887753 7f9d44424700  5 -- op tracker -- , seq: 748104, time: 2014-06-12 14:46:51.886635, event: throttled, op: osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.887759 7f9d44424700  5 -- op tracker -- , seq: 748104, time: 2014-06-12 14:46:51.887706, event: all_read, op: osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.887764 7f9d44424700  5 -- op tracker -- , seq: 748104, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.887770 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:51.887792 7f9d44424700 15 osd.2 173 enqueue_op 0x3b42b100 prio 63 cost 516096 latency 0.001159 osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.887815 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.887823 7f9d67be0700 10 osd.2 173 dequeue_op 0x3b42b100 prio 63 cost 516096 latency 0.001191 osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean]
2014-06-12 14:46:51.887841 7f9d67be0700  5 -- op tracker -- , seq: 748104, time: 2014-06-12 14:46:51.887838, event: reached_pg, op: osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.887851 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:51.887862 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] handle_message: 0x3b42b100
2014-06-12 14:46:51.887868 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] do_op osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:51.887885 7f9d67be0700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_'
2014-06-12 14:46:51.888004 7f9d67be0700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_' = 0
2014-06-12 14:46:51.888018 7f9d67be0700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset'
2014-06-12 14:46:51.888064 7f9d67be0700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset' = 0
2014-06-12 14:46:51.888072 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] populate_obc_watchers 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.888081 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.888087 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] get_object_context: creating obc from disk: 0x12d11b80
2014-06-12 14:46:51.888092 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] get_object_context: 0x12d11b80 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 rwstate(none n=0 w=0) oi: 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2893 client.4317.0:143982 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2893) ssc: 0x16ee41c0 snapset: 0=[]:[]+head
2014-06-12 14:46:51.888103 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] find_object_context 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 @head oi=24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2893 client.4317.0:143982 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2893)
2014-06-12 14:46:51.888120 7f9d67be0700  5 -- op tracker -- , seq: 748104, time: 2014-06-12 14:46:51.888116, event: started, op: osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.888126 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] execute_ctx 0xa04f200
2014-06-12 14:46:51.888137 7f9d67be0700  5 -- op tracker -- , seq: 748104, time: 2014-06-12 14:46:51.888134, event: started, op: osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.888142 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] do_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] ov 173'2893 av 173'2894 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:51.888152 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] do_osd_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096]
2014-06-12 14:46:51.888159 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:51.888169 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] do_osd_op  write 3063808~516096
2014-06-12 14:46:51.888179 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:51.888187 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 snapset=0x16ee41f8  snapc=0=[]
2014-06-12 14:46:51.888194 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:51.888201 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] finish_ctx 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 0xa04f200 op modify  
2014-06-12 14:46:51.888209 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean]  set mtime to 2014-06-12 08:45:54.266008
2014-06-12 14:46:51.888223 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean]  final snapset 0=[]:[]+head in 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.888236 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean]  zeroing write result code 0
2014-06-12 14:46:51.888243 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] new_repop rep_tid 236343 on osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.888254 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] issue_repop rep_tid 236343 o 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.888270 7f9d67be0700  5 -- op tracker -- , seq: 748104, time: 2014-06-12 14:46:51.888267, event: waiting for subops from 28, op: osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.888285 7f9d67be0700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:51.888300 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2893 (0'0,173'2893] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] append_log log((0'0,173'2893], crt=0'0) [173'2894 (173'2893) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143983 2014-06-12 08:45:54.266008]
2014-06-12 14:46:51.888317 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2893 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] add_log_entry 173'2894 (173'2893) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143983 2014-06-12 08:45:54.266008
2014-06-12 14:46:51.888331 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2893 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] append_log  adding 1 keys
2014-06-12 14:46:51.888353 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2894, trimmed: 
2014-06-12 14:46:51.888370 7f9d67be0700  5 queue_transactions existing osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.888373 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:51.888375 7f9d67be0700  5 queue_op 0x1c4a8360 seq 2717 osr(3.721 0x276de60) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:51.888384 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8360 (1 active)
2014-06-12 14:46:51.888390 7f9d7c165700  5 _do_op 0x1c4a8360 seq 2717 osr(3.721 0x276de60)/0x276de60 start
2014-06-12 14:46:51.888393 7f9d7c165700 10 _do_transaction on 0x2563380
2014-06-12 14:46:51.888379 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2893 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] eval_repop repgather(0x13d23bc0 173'2894 rep_tid=236343 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.888396 7f9d67be0700 10 osd.2 173 dequeue_op 0x3b42b100 finish
2014-06-12 14:46:51.888399 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.888399 7f9d7c165700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.888402 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.888456 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.888473 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:51.888502 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:51.888520 7f9d7c165700 15 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.888524 7f9d7c165700 10 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1 r = 0
2014-06-12 14:46:51.888528 7f9d7c165700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.888658 7f9d7c165700 15 _write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 3063808~516096
2014-06-12 14:46:51.888741 7f9d7c165700 10 lookup_strip_header done  cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.888864 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:51.890030 7f9d7c165700 10 _generic_write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 3063808~516096 = 0
2014-06-12 14:46:51.890062 7f9d7c165700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.890070 7f9d7c165700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.890074 7f9d7c165700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.890079 7f9d7c165700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.890213 7f9d7c165700 20 set_header setting 16857 cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 parent seq 0
2014-06-12 14:46:51.890259 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:51.890268 7f9d7c165700 20 set_header setting 2130 cid meta oid 1fe0ff5a/pglog_3.721/0//-1 parent seq 0
2014-06-12 14:46:51.890274 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:51.892413 7f9d7c165700 10 _do_op 0x1c4a8360 seq 2717 r = 0, finisher 0x692f000 0x3786fa80
2014-06-12 14:46:51.892430 7f9d7c165700 10 _finish_op 0x1c4a8360 seq 2717 osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.892439 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8360 (0 active)
2014-06-12 14:46:51.892443 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:51.892434 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2893 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] op_commit: 236343
2014-06-12 14:46:51.892463 7f9d7715b700  5 -- op tracker -- , seq: 748104, time: 2014-06-12 14:46:51.892455, event: op_commit, op: osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.892482 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2893 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] op_applied: 236343
2014-06-12 14:46:51.892508 7f9d7795c700  5 -- op tracker -- , seq: 748104, time: 2014-06-12 14:46:51.892501, event: op_applied, op: osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.892519 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2893 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] op_applied on primary on version 173'2894
2014-06-12 14:46:51.896695 7f9d5b6bc700  5 -- op tracker -- , seq: 748105, time: 2014-06-12 14:46:51.896646, event: header_read, op: osd_sub_op_reply(client.4317.0:143983 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.896708 7f9d5b6bc700  5 -- op tracker -- , seq: 748105, time: 2014-06-12 14:46:51.896648, event: throttled, op: osd_sub_op_reply(client.4317.0:143983 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.896723 7f9d5b6bc700  5 -- op tracker -- , seq: 748105, time: 2014-06-12 14:46:51.896661, event: all_read, op: osd_sub_op_reply(client.4317.0:143983 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.896731 7f9d5b6bc700  5 -- op tracker -- , seq: 748105, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143983 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.896737 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143983 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:51.896743 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:51.896752 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x31d60f00 prio 196 cost 0 latency 0.000105 osd_sub_op_reply(client.4317.0:143983 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:51.896776 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.896791 7f9d673df700 10 osd.2 173 dequeue_op 0x31d60f00 prio 196 cost 0 latency 0.000144 osd_sub_op_reply(client.4317.0:143983 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 pg pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2893 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean]
2014-06-12 14:46:51.896806 7f9d673df700  5 -- op tracker -- , seq: 748105, time: 2014-06-12 14:46:51.896804, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143983 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.896813 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2893 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] handle_message: 0x31d60f00
2014-06-12 14:46:51.896823 7f9d673df700  5 -- op tracker -- , seq: 748105, time: 2014-06-12 14:46:51.896821, event: started, op: osd_sub_op_reply(client.4317.0:143983 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.896826 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2893 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] sub_op_modify_reply: tid 236343 op  ack_type 4 from 28
2014-06-12 14:46:51.896839 7f9d673df700  5 -- op tracker -- , seq: 748104, time: 2014-06-12 14:46:51.896834, event: sub_op_commit_rec, op: osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.896843 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2893 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] repop_all_applied: repop tid 236343 all applied 
2014-06-12 14:46:51.896858 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2893 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] eval_repop repgather(0x13d23bc0 173'2894 rep_tid=236343 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.896869 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2893 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean]  sending ack on repgather(0x13d23bc0 173'2894 rep_tid=236343 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x1710b180
2014-06-12 14:46:51.896889 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2893 crt=0'0 lcod 173'2892 mlcod 173'2892 active+clean] repop_all_committed: repop tid 236343 all committed 
2014-06-12 14:46:51.896896 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2892 active+clean] eval_repop repgather(0x13d23bc0 173'2894 rep_tid=236343 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.896906 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2892 active+clean] log_op_stats osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.010256 lat 0.010273
2014-06-12 14:46:51.896918 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2892 active+clean] publish_stats_to_osd 173:3708
2014-06-12 14:46:51.896926 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2892 active+clean]  sending commit on repgather(0x13d23bc0 173'2894 rep_tid=236343 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x38137400
2014-06-12 14:46:51.896946 7f9d673df700  5 -- op tracker -- , seq: 748104, time: 2014-06-12 14:46:51.896943, event: commit_sent, op: osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.896953 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2892 active+clean]  requeue_ops 
2014-06-12 14:46:51.896960 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean]  removing repgather(0x13d23bc0 173'2894 rep_tid=236343 committed?=1 applied?=1 op=osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.896970 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean]    q front is repgather(0x13d23bc0 173'2894 rep_tid=236343 committed?=1 applied?=1 op=osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.896980 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] remove_repop repgather(0x13d23bc0 173'2894 rep_tid=236343 committed?=1 applied?=1 op=osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.896989 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean]  requeue_ops 
2014-06-12 14:46:51.897007 7f9d673df700  5 -- op tracker -- , seq: 748104, time: 2014-06-12 14:46:51.897004, event: done, op: osd_op(client.4317.0:143983 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3063808~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.897021 7f9d673df700 10 osd.2 173 dequeue_op 0x31d60f00 finish
2014-06-12 14:46:51.897027 7f9d673df700  5 -- op tracker -- , seq: 748105, time: 2014-06-12 14:46:51.897024, event: done, op: osd_sub_op_reply(client.4317.0:143983 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.897033 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.897035 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.900483 7f9d44424700  5 -- op tracker -- , seq: 748106, time: 2014-06-12 14:46:51.899329, event: header_read, op: osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.900495 7f9d44424700  5 -- op tracker -- , seq: 748106, time: 2014-06-12 14:46:51.899336, event: throttled, op: osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.900509 7f9d44424700  5 -- op tracker -- , seq: 748106, time: 2014-06-12 14:46:51.900444, event: all_read, op: osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.900515 7f9d44424700  5 -- op tracker -- , seq: 748106, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.900521 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:51.900542 7f9d44424700 15 osd.2 173 enqueue_op 0x329ebd00 prio 63 cost 516096 latency 0.001213 osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.900566 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.900576 7f9d663dd700 10 osd.2 173 dequeue_op 0x329ebd00 prio 63 cost 516096 latency 0.001247 osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean]
2014-06-12 14:46:51.900596 7f9d663dd700  5 -- op tracker -- , seq: 748106, time: 2014-06-12 14:46:51.900593, event: reached_pg, op: osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.900604 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:51.900614 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] handle_message: 0x329ebd00
2014-06-12 14:46:51.900620 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] do_op osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:51.900636 7f9d663dd700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_'
2014-06-12 14:46:51.900763 7f9d663dd700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_' = 0
2014-06-12 14:46:51.900778 7f9d663dd700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset'
2014-06-12 14:46:51.900822 7f9d663dd700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset' = 0
2014-06-12 14:46:51.900829 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] populate_obc_watchers 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.900837 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.900843 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] get_object_context: creating obc from disk: 0xa546840
2014-06-12 14:46:51.900849 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] get_object_context: 0xa546840 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 rwstate(none n=0 w=0) oi: 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2894 client.4317.0:143983 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2894) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:46:51.900859 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] find_object_context 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 @head oi=24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2894 client.4317.0:143983 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2894)
2014-06-12 14:46:51.900875 7f9d663dd700  5 -- op tracker -- , seq: 748106, time: 2014-06-12 14:46:51.900872, event: started, op: osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.900881 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] execute_ctx 0x2e411800
2014-06-12 14:46:51.900891 7f9d663dd700  5 -- op tracker -- , seq: 748106, time: 2014-06-12 14:46:51.900888, event: started, op: osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.900895 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] do_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] ov 173'2894 av 173'2895 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:51.900903 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] do_osd_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096]
2014-06-12 14:46:51.900910 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:51.900918 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] do_osd_op  write 3579904~516096
2014-06-12 14:46:51.900928 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:51.900934 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:46:51.900941 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:51.900946 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] finish_ctx 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 0x2e411800 op modify  
2014-06-12 14:46:51.900954 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean]  set mtime to 2014-06-12 08:45:54.278751
2014-06-12 14:46:51.900966 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean]  final snapset 0=[]:[]+head in 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.900977 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean]  zeroing write result code 0
2014-06-12 14:46:51.900983 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] new_repop rep_tid 236344 on osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.900997 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] issue_repop rep_tid 236344 o 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.901013 7f9d663dd700  5 -- op tracker -- , seq: 748106, time: 2014-06-12 14:46:51.901010, event: waiting for subops from 28, op: osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.901028 7f9d663dd700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:51.901043 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2894 (0'0,173'2894] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] append_log log((0'0,173'2894], crt=0'0) [173'2895 (173'2894) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143984 2014-06-12 08:45:54.278751]
2014-06-12 14:46:51.901067 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2894 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] add_log_entry 173'2895 (173'2894) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143984 2014-06-12 08:45:54.278751
2014-06-12 14:46:51.901087 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2894 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] append_log  adding 1 keys
2014-06-12 14:46:51.901114 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2895, trimmed: 
2014-06-12 14:46:51.901135 7f9d663dd700  5 queue_transactions existing osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.901139 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:51.901141 7f9d663dd700  5 queue_op 0x6de0f00 seq 2718 osr(3.721 0x276de60) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:51.901150 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8360 (1 active)
2014-06-12 14:46:51.901156 7f9d8096e700  5 _do_op 0x6de0f00 seq 2718 osr(3.721 0x276de60)/0x276de60 start
2014-06-12 14:46:51.901159 7f9d8096e700 10 _do_transaction on 0x3403afc0
2014-06-12 14:46:51.901147 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2894 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] eval_repop repgather(0x234dbc0 173'2895 rep_tid=236344 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.901163 7f9d663dd700 10 osd.2 173 dequeue_op 0x329ebd00 finish
2014-06-12 14:46:51.901164 7f9d8096e700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.901167 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.901170 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.901222 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.901239 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:51.901270 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:51.901288 7f9d8096e700 15 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.901293 7f9d8096e700 10 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1 r = 0
2014-06-12 14:46:51.901297 7f9d8096e700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.901409 7f9d8096e700 15 _write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 3579904~516096
2014-06-12 14:46:51.901494 7f9d8096e700 10 lookup_strip_header done  cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.901606 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:51.902937 7f9d8096e700 10 _generic_write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 3579904~516096 = 0
2014-06-12 14:46:51.902974 7f9d8096e700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.902985 7f9d8096e700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.902990 7f9d8096e700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.902995 7f9d8096e700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.903132 7f9d8096e700 20 set_header setting 16857 cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 parent seq 0
2014-06-12 14:46:51.903182 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:51.903192 7f9d8096e700 20 set_header setting 2130 cid meta oid 1fe0ff5a/pglog_3.721/0//-1 parent seq 0
2014-06-12 14:46:51.903198 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:51.905071 7f9d8096e700 10 _do_op 0x6de0f00 seq 2718 r = 0, finisher 0x1ecc72e0 0x19b2d540
2014-06-12 14:46:51.905087 7f9d8096e700 10 _finish_op 0x6de0f00 seq 2718 osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.905096 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8360 (0 active)
2014-06-12 14:46:51.905102 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:51.905094 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2894 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] op_commit: 236344
2014-06-12 14:46:51.905129 7f9d7715b700  5 -- op tracker -- , seq: 748106, time: 2014-06-12 14:46:51.905122, event: op_commit, op: osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.905141 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2894 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] op_applied: 236344
2014-06-12 14:46:51.905168 7f9d7795c700  5 -- op tracker -- , seq: 748106, time: 2014-06-12 14:46:51.905161, event: op_applied, op: osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.905179 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2894 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] op_applied on primary on version 173'2895
2014-06-12 14:46:51.909404 7f9d5b6bc700  5 -- op tracker -- , seq: 748107, time: 2014-06-12 14:46:51.909365, event: header_read, op: osd_sub_op_reply(client.4317.0:143984 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.909414 7f9d5b6bc700  5 -- op tracker -- , seq: 748107, time: 2014-06-12 14:46:51.909367, event: throttled, op: osd_sub_op_reply(client.4317.0:143984 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.909426 7f9d5b6bc700  5 -- op tracker -- , seq: 748107, time: 2014-06-12 14:46:51.909377, event: all_read, op: osd_sub_op_reply(client.4317.0:143984 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.909431 7f9d5b6bc700  5 -- op tracker -- , seq: 748107, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143984 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.909435 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143984 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:51.909439 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:51.909446 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x31d61f00 prio 196 cost 0 latency 0.000080 osd_sub_op_reply(client.4317.0:143984 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:51.909469 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.909486 7f9d65bdc700 10 osd.2 173 dequeue_op 0x31d61f00 prio 196 cost 0 latency 0.000120 osd_sub_op_reply(client.4317.0:143984 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 pg pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2894 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean]
2014-06-12 14:46:51.909502 7f9d65bdc700  5 -- op tracker -- , seq: 748107, time: 2014-06-12 14:46:51.909500, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143984 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.909509 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2894 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] handle_message: 0x31d61f00
2014-06-12 14:46:51.909520 7f9d65bdc700  5 -- op tracker -- , seq: 748107, time: 2014-06-12 14:46:51.909518, event: started, op: osd_sub_op_reply(client.4317.0:143984 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.909523 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2894 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] sub_op_modify_reply: tid 236344 op  ack_type 4 from 28
2014-06-12 14:46:51.909537 7f9d65bdc700  5 -- op tracker -- , seq: 748106, time: 2014-06-12 14:46:51.909531, event: sub_op_commit_rec, op: osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.909544 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2894 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] repop_all_applied: repop tid 236344 all applied 
2014-06-12 14:46:51.909552 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2894 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] eval_repop repgather(0x234dbc0 173'2895 rep_tid=236344 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.909563 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2894 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean]  sending ack on repgather(0x234dbc0 173'2895 rep_tid=236344 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x10097b80
2014-06-12 14:46:51.909585 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2894 crt=0'0 lcod 173'2893 mlcod 173'2893 active+clean] repop_all_committed: repop tid 236344 all committed 
2014-06-12 14:46:51.909592 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2893 active+clean] eval_repop repgather(0x234dbc0 173'2895 rep_tid=236344 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.909630 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2893 active+clean] log_op_stats osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.010255 lat 0.010282
2014-06-12 14:46:51.909649 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2893 active+clean] publish_stats_to_osd 173:3709
2014-06-12 14:46:51.909667 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2893 active+clean]  sending commit on repgather(0x234dbc0 173'2895 rep_tid=236344 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x2183b680
2014-06-12 14:46:51.909689 7f9d65bdc700  5 -- op tracker -- , seq: 748106, time: 2014-06-12 14:46:51.909685, event: commit_sent, op: osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.909703 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2893 active+clean]  requeue_ops 
2014-06-12 14:46:51.909712 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean]  removing repgather(0x234dbc0 173'2895 rep_tid=236344 committed?=1 applied?=1 op=osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.909720 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean]    q front is repgather(0x234dbc0 173'2895 rep_tid=236344 committed?=1 applied?=1 op=osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.909729 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] remove_repop repgather(0x234dbc0 173'2895 rep_tid=236344 committed?=1 applied?=1 op=osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.909745 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean]  requeue_ops 
2014-06-12 14:46:51.909762 7f9d65bdc700  5 -- op tracker -- , seq: 748106, time: 2014-06-12 14:46:51.909759, event: done, op: osd_op(client.4317.0:143984 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 3579904~516096] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.909777 7f9d65bdc700 10 osd.2 173 dequeue_op 0x31d61f00 finish
2014-06-12 14:46:51.909783 7f9d65bdc700  5 -- op tracker -- , seq: 748107, time: 2014-06-12 14:46:51.909780, event: done, op: osd_sub_op_reply(client.4317.0:143984 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.909791 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.909795 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.911323 7f9d44424700  5 -- op tracker -- , seq: 748108, time: 2014-06-12 14:46:51.911113, event: header_read, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.911336 7f9d44424700  5 -- op tracker -- , seq: 748108, time: 2014-06-12 14:46:51.911115, event: throttled, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.911345 7f9d44424700  5 -- op tracker -- , seq: 748108, time: 2014-06-12 14:46:51.911296, event: all_read, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.911353 7f9d44424700  5 -- op tracker -- , seq: 748108, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.911360 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:51.911384 7f9d44424700 15 osd.2 173 enqueue_op 0x329ebc00 prio 63 cost 98304 latency 0.000271 osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.911408 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.911415 7f9d62bd6700 10 osd.2 173 dequeue_op 0x329ebc00 prio 63 cost 98304 latency 0.000302 osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean]
2014-06-12 14:46:51.911433 7f9d62bd6700  5 -- op tracker -- , seq: 748108, time: 2014-06-12 14:46:51.911429, event: reached_pg, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.911441 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:51.911451 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] handle_message: 0x329ebc00
2014-06-12 14:46:51.911458 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] do_op osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:51.911475 7f9d62bd6700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_'
2014-06-12 14:46:51.911607 7f9d62bd6700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 '_' = 0
2014-06-12 14:46:51.911621 7f9d62bd6700 15 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset'
2014-06-12 14:46:51.911667 7f9d62bd6700 10 getattr 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 'snapset' = 0
2014-06-12 14:46:51.911674 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] populate_obc_watchers 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.911683 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.911689 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] get_object_context: creating obc from disk: 0x135b6580
2014-06-12 14:46:51.911695 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] get_object_context: 0x135b6580 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 rwstate(none n=0 w=0) oi: 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2895 client.4317.0:143984 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2895) ssc: 0x1e15e460 snapset: 0=[]:[]+head
2014-06-12 14:46:51.911705 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] find_object_context 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 @head oi=24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3(173'2895 client.4317.0:143984 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2895)
2014-06-12 14:46:51.911722 7f9d62bd6700  5 -- op tracker -- , seq: 748108, time: 2014-06-12 14:46:51.911718, event: started, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.911728 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] execute_ctx 0x2e411e00
2014-06-12 14:46:51.911739 7f9d62bd6700  5 -- op tracker -- , seq: 748108, time: 2014-06-12 14:46:51.911736, event: started, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.911743 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] do_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] ov 173'2895 av 173'2896 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:51.911754 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] do_osd_op 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304]
2014-06-12 14:46:51.911761 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:51.911769 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] do_osd_op  write 4096000~98304
2014-06-12 14:46:51.911780 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:51.911788 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 snapset=0x1e15e498  snapc=0=[]
2014-06-12 14:46:51.911796 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] make_writeable 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:51.911802 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] finish_ctx 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 0x2e411e00 op modify  
2014-06-12 14:46:51.911809 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean]  set mtime to 2014-06-12 08:45:54.291263
2014-06-12 14:46:51.911823 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean]  final snapset 0=[]:[]+head in 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.911835 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean]  zeroing write result code 0
2014-06-12 14:46:51.911842 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] new_repop rep_tid 236345 on osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:51.911854 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] issue_repop rep_tid 236345 o 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.911870 7f9d62bd6700  5 -- op tracker -- , seq: 748108, time: 2014-06-12 14:46:51.911867, event: waiting for subops from 28, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.911884 7f9d62bd6700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:51.911898 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2895 (0'0,173'2895] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] append_log log((0'0,173'2895], crt=0'0) [173'2896 (173'2895) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143985 2014-06-12 08:45:54.291263]
2014-06-12 14:46:51.911915 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] add_log_entry 173'2896 (173'2895) modify   24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 by client.4317.0:143985 2014-06-12 08:45:54.291263
2014-06-12 14:46:51.911929 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] append_log  adding 1 keys
2014-06-12 14:46:51.911950 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2896, trimmed: 
2014-06-12 14:46:51.911967 7f9d62bd6700  5 queue_transactions existing osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.911970 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:51.911972 7f9d62bd6700  5 queue_op 0x1efba180 seq 2719 osr(3.721 0x276de60) 100597 bytes   (queue has 1 ops and 100597 bytes)
2014-06-12 14:46:51.911976 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] eval_repop repgather(0x96dbe00 173'2896 rep_tid=236345 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.911988 7f9d62bd6700 10 osd.2 173 dequeue_op 0x329ebc00 finish
2014-06-12 14:46:51.911981 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8360 (1 active)
2014-06-12 14:46:51.911990 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.911992 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.911992 7f9d7c966700  5 _do_op 0x1efba180 seq 2719 osr(3.721 0x276de60)/0x276de60 start
2014-06-12 14:46:51.911997 7f9d7c966700 10 _do_transaction on 0xe833200
2014-06-12 14:46:51.912003 7f9d7c966700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.912073 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.912101 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:51.912146 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:51.912246 7f9d7c966700 15 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.912256 7f9d7c966700 10 _omap_rmkeys meta/1fe0ff5a/pglog_3.721/0//-1 r = 0
2014-06-12 14:46:51.912277 7f9d7c966700 15 _omap_setkeys meta/1fe0ff5a/pglog_3.721/0//-1
2014-06-12 14:46:51.912299 7f9d7c966700 15 _write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 4096000~98304
2014-06-12 14:46:51.912434 7f9d7c966700 10 lookup_strip_header done  cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.912569 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:51.912879 7f9d7c966700 10 _generic_write 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 4096000~98304 = 0
2014-06-12 14:46:51.912898 7f9d7c966700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.912930 7f9d7c966700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.912938 7f9d7c966700 15 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3
2014-06-12 14:46:51.912947 7f9d7c966700 10 _setattrs 3.721_head/24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 = 0
2014-06-12 14:46:51.913082 7f9d7c966700 20 set_header setting 16857 cid 3.721_head oid 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 parent seq 0
2014-06-12 14:46:51.913111 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:51.913126 7f9d7c966700 20 set_header setting 2130 cid meta oid 1fe0ff5a/pglog_3.721/0//-1 parent seq 0
2014-06-12 14:46:51.913138 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:51.913640 7f9d7c966700 10 _do_op 0x1efba180 seq 2719 r = 0, finisher 0x6cb8f80 0x46d1fc0
2014-06-12 14:46:51.913651 7f9d7c966700 10 _finish_op 0x1efba180 seq 2719 osr(3.721 0x276de60)/0x276de60
2014-06-12 14:46:51.913657 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8360 (0 active)
2014-06-12 14:46:51.913660 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:51.913665 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] op_applied: 236345
2014-06-12 14:46:51.913694 7f9d7795c700  5 -- op tracker -- , seq: 748108, time: 2014-06-12 14:46:51.913686, event: op_applied, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.913705 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] op_applied on primary on version 173'2896
2014-06-12 14:46:51.913723 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] op_commit: 236345
2014-06-12 14:46:51.913748 7f9d7715b700  5 -- op tracker -- , seq: 748108, time: 2014-06-12 14:46:51.913741, event: op_commit, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.916290 7f9d5b6bc700  5 -- op tracker -- , seq: 748109, time: 2014-06-12 14:46:51.916184, event: header_read, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.916301 7f9d5b6bc700  5 -- op tracker -- , seq: 748109, time: 2014-06-12 14:46:51.916186, event: throttled, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.916309 7f9d5b6bc700  5 -- op tracker -- , seq: 748109, time: 2014-06-12 14:46:51.916217, event: all_read, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.916316 7f9d5b6bc700  5 -- op tracker -- , seq: 748109, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.916321 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:51.916327 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:51.916335 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x31d60b00 prio 196 cost 0 latency 0.000151 osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0) v2
2014-06-12 14:46:51.916365 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.916372 7f9d683e1700 10 osd.2 173 dequeue_op 0x31d60b00 prio 196 cost 0 latency 0.000187 osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0) v2 pg pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean]
2014-06-12 14:46:51.916393 7f9d683e1700  5 -- op tracker -- , seq: 748109, time: 2014-06-12 14:46:51.916389, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.916402 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] handle_message: 0x31d60b00
2014-06-12 14:46:51.916418 7f9d683e1700  5 -- op tracker -- , seq: 748109, time: 2014-06-12 14:46:51.916415, event: started, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.916424 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] sub_op_modify_reply: tid 236345 op  ack_type 1 from 28
2014-06-12 14:46:51.916445 7f9d683e1700  5 -- op tracker -- , seq: 748108, time: 2014-06-12 14:46:51.916437, event: sub_op_applied_rec, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.916452 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] repop_all_applied: repop tid 236345 all applied 
2014-06-12 14:46:51.916464 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] eval_repop repgather(0x96dbe00 173'2896 rep_tid=236345 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.916481 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean]  sending ack on repgather(0x96dbe00 173'2896 rep_tid=236345 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x1687a280
2014-06-12 14:46:51.916513 7f9d683e1700 10 osd.2 173 dequeue_op 0x31d60b00 finish
2014-06-12 14:46:51.916520 7f9d683e1700  5 -- op tracker -- , seq: 748109, time: 2014-06-12 14:46:51.916517, event: done, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ack, result = 0)
2014-06-12 14:46:51.916536 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.916539 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.916585 7f9d5b6bc700  5 -- op tracker -- , seq: 748110, time: 2014-06-12 14:46:51.916549, event: header_read, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.916594 7f9d5b6bc700  5 -- op tracker -- , seq: 748110, time: 2014-06-12 14:46:51.916549, event: throttled, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.916602 7f9d5b6bc700  5 -- op tracker -- , seq: 748110, time: 2014-06-12 14:46:51.916565, event: all_read, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.916615 7f9d5b6bc700  5 -- op tracker -- , seq: 748110, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.916618 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:51.916623 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:46:51.916628 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x31d61d00 prio 196 cost 0 latency 0.000079 osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:51.916651 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:51.916657 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x31d61d00 prio 196 cost 0 latency 0.000107 osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0) v2 pg pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean]
2014-06-12 14:46:51.916674 7f9d6b3e7700  5 -- op tracker -- , seq: 748110, time: 2014-06-12 14:46:51.916671, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.916682 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] handle_message: 0x31d61d00
2014-06-12 14:46:51.916696 7f9d6b3e7700  5 -- op tracker -- , seq: 748110, time: 2014-06-12 14:46:51.916693, event: started, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.916707 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] sub_op_modify_reply: tid 236345 op  ack_type 4 from 28
2014-06-12 14:46:51.916724 7f9d6b3e7700  5 -- op tracker -- , seq: 748108, time: 2014-06-12 14:46:51.916718, event: sub_op_commit_rec, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.916730 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 luod=173'2895 crt=0'0 lcod 173'2894 mlcod 173'2894 active+clean] repop_all_committed: repop tid 236345 all committed 
2014-06-12 14:46:51.916745 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2894 active+clean] eval_repop repgather(0x96dbe00 173'2896 rep_tid=236345 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:51.916761 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2894 active+clean] log_op_stats osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4 inb 98583 outb 0 rlat 0.005398 lat 0.005647
2014-06-12 14:46:51.916778 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2894 active+clean] publish_stats_to_osd 173:3710
2014-06-12 14:46:51.916792 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2894 active+clean]  sending commit on repgather(0x96dbe00 173'2896 rep_tid=236345 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4) 0x30cca780
2014-06-12 14:46:51.916821 7f9d6b3e7700  5 -- op tracker -- , seq: 748108, time: 2014-06-12 14:46:51.916816, event: commit_sent, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.916828 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2894 active+clean]  requeue_ops 
2014-06-12 14:46:51.916839 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2895 active+clean]  removing repgather(0x96dbe00 173'2896 rep_tid=236345 committed?=1 applied?=1 op=osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.916853 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2895 active+clean]    q front is repgather(0x96dbe00 173'2896 rep_tid=236345 committed?=1 applied?=1 op=osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.916866 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2895 active+clean] remove_repop repgather(0x96dbe00 173'2896 rep_tid=236345 committed?=1 applied?=1 op=osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:51.916880 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2895 active+clean]  requeue_ops 
2014-06-12 14:46:51.916905 7f9d6b3e7700  5 -- op tracker -- , seq: 748108, time: 2014-06-12 14:46:51.916900, event: done, op: osd_op(client.4317.0:143985 rbd_data.10b46b8b4567.0000000000001054 [set-alloc-hint object_size 4194304 write_size 4194304,write 4096000~98304] 3.24a2bf21 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:51.916920 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x31d61d00 finish
2014-06-12 14:46:51.916927 7f9d6b3e7700  5 -- op tracker -- , seq: 748110, time: 2014-06-12 14:46:51.916923, event: done, op: osd_sub_op_reply(client.4317.0:143985 3.721 24a2bf21/rbd_data.10b46b8b4567.0000000000001054/head//3 [] ondisk, result = 0)
2014-06-12 14:46:51.916936 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:51.916939 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:51.970075 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:51.970487 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:52.018219 7f9d44424700  5 -- op tracker -- , seq: 748111, time: 2014-06-12 14:46:52.017474, event: header_read, op: osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.018231 7f9d44424700  5 -- op tracker -- , seq: 748111, time: 2014-06-12 14:46:52.017478, event: throttled, op: osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.018237 7f9d44424700  5 -- op tracker -- , seq: 748111, time: 2014-06-12 14:46:52.018151, event: all_read, op: osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.018247 7f9d44424700  5 -- op tracker -- , seq: 748111, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.018255 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:52.018281 7f9d44424700 15 osd.2 173 enqueue_op 0x24c9c100 prio 63 cost 352256 latency 0.000807 osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.018311 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.018320 7f9d693e3700 10 osd.2 173 dequeue_op 0x24c9c100 prio 63 cost 352256 latency 0.000846 osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean]
2014-06-12 14:46:52.018347 7f9d693e3700  5 -- op tracker -- , seq: 748111, time: 2014-06-12 14:46:52.018342, event: reached_pg, op: osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.018355 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:52.018369 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] handle_message: 0x24c9c100
2014-06-12 14:46:52.018379 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] do_op osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:52.018410 7f9d693e3700 15 getattr 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 '_'
2014-06-12 14:46:52.028481 7f9d44424700  5 -- op tracker -- , seq: 748112, time: 2014-06-12 14:46:52.027336, event: header_read, op: osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.028491 7f9d44424700  5 -- op tracker -- , seq: 748112, time: 2014-06-12 14:46:52.027340, event: throttled, op: osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.028497 7f9d44424700  5 -- op tracker -- , seq: 748112, time: 2014-06-12 14:46:52.028447, event: all_read, op: osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.028502 7f9d44424700  5 -- op tracker -- , seq: 748112, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.028509 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:52.028531 7f9d44424700 15 osd.2 173 enqueue_op 0x24c9d500 prio 63 cost 516096 latency 0.001195 osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.028556 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:52.036293 7f9d693e3700 10 getattr 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 '_' = 0
2014-06-12 14:46:52.036317 7f9d693e3700 15 getattr 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 'snapset'
2014-06-12 14:46:52.036384 7f9d693e3700 10 getattr 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 'snapset' = 0
2014-06-12 14:46:52.036394 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] populate_obc_watchers e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.036406 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.036413 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] get_object_context: creating obc from disk: 0x2e7ad8c0
2014-06-12 14:46:52.036419 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] get_object_context: 0x2e7ad8c0 e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 rwstate(none n=0 w=0) oi: e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'889 client.4317.0:38422 wrlock_by=unknown.0.0:0 dirty s 4194304 uv889) ssc: 0x9345880 snapset: 0=[]:[]+head
2014-06-12 14:46:52.036432 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] find_object_context e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 @head oi=e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'889 client.4317.0:38422 wrlock_by=unknown.0.0:0 dirty s 4194304 uv889)
2014-06-12 14:46:52.036463 7f9d693e3700  5 -- op tracker -- , seq: 748111, time: 2014-06-12 14:46:52.036457, event: started, op: osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.036471 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] execute_ctx 0x60aa600
2014-06-12 14:46:52.036482 7f9d693e3700  5 -- op tracker -- , seq: 748111, time: 2014-06-12 14:46:52.036479, event: started, op: osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.036487 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] do_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] ov 173'889 av 173'2955 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:52.036497 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] do_osd_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256]
2014-06-12 14:46:52.036504 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:52.036514 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] do_osd_op  write 0~352256
2014-06-12 14:46:52.036527 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:52.036535 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 snapset=0x93458b8  snapc=0=[]
2014-06-12 14:46:52.036543 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:52.036550 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] finish_ctx e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 0x60aa600 op modify  
2014-06-12 14:46:52.036558 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean]  set mtime to 2014-06-12 08:45:54.397202
2014-06-12 14:46:52.036572 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean]  final snapset 0=[]:[]+head in e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.036588 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean]  zeroing write result code 0
2014-06-12 14:46:52.036595 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] new_repop rep_tid 236346 on osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.036607 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] issue_repop rep_tid 236346 o e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.036625 7f9d693e3700  5 -- op tracker -- , seq: 748111, time: 2014-06-12 14:46:52.036622, event: waiting for subops from 21, op: osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.036643 7f9d693e3700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:52.036660 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2954 (0'0,173'2954] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] append_log log((0'0,173'2954], crt=0'0) [173'2955 (173'889) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:143995 2014-06-12 08:45:54.397202]
2014-06-12 14:46:52.036676 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] add_log_entry 173'2955 (173'889) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:143995 2014-06-12 08:45:54.397202
2014-06-12 14:46:52.036689 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] append_log  adding 1 keys
2014-06-12 14:46:52.036711 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2955, trimmed: 
2014-06-12 14:46:52.036727 7f9d693e3700  5 queue_transactions existing osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.036731 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.036732 7f9d693e3700  5 queue_op 0x5949ce0 seq 2618 osr(3.3a 0x276cce0) 354540 bytes   (queue has 1 ops and 354540 bytes)
2014-06-12 14:46:52.036737 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] eval_repop repgather(0x24fa7200 173'2955 rep_tid=236346 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.036751 7f9d693e3700 10 osd.2 173 dequeue_op 0x24c9c100 finish
2014-06-12 14:46:52.036758 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:52.036763 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.036775 7f9d66bde700 10 osd.2 173 dequeue_op 0x24c9d500 prio 63 cost 516096 latency 0.009438 osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean]
2014-06-12 14:46:52.036807 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b70e0 (1 active)
2014-06-12 14:46:52.036813 7f9d81970700  5 _do_op 0x5949ce0 seq 2618 osr(3.3a 0x276cce0)/0x276cce0 start
2014-06-12 14:46:52.036816 7f9d81970700 10 _do_transaction on 0x1771d5c0
2014-06-12 14:46:52.036821 7f9d81970700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.036817 7f9d66bde700  5 -- op tracker -- , seq: 748112, time: 2014-06-12 14:46:52.036811, event: reached_pg, op: osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.036827 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:52.036841 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] handle_message: 0x24c9d500
2014-06-12 14:46:52.036852 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] do_op osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:52.036878 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] get_object_context: found obc in cache: 0x2e7ad8c0
2014-06-12 14:46:52.036890 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] get_object_context: 0x2e7ad8c0 e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 rwstate(write n=1 w=0) oi: e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2955 client.4317.0:143995 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2955) ssc: 0x9345880 snapset: 0=[]:[]+head
2014-06-12 14:46:52.036905 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] find_object_context e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 @head oi=e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2955 client.4317.0:143995 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2955)
2014-06-12 14:46:52.036921 7f9d81970700 10 lookup_strip_header done  cid meta oid 5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.036929 7f9d66bde700  5 -- op tracker -- , seq: 748112, time: 2014-06-12 14:46:52.036923, event: started, op: osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.036937 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] execute_ctx 0x5e4c400
2014-06-12 14:46:52.036946 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.036955 7f9d66bde700  5 -- op tracker -- , seq: 748112, time: 2014-06-12 14:46:52.036950, event: started, op: osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.036963 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] do_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] ov 173'2955 av 173'2956 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:52.036977 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] do_osd_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096]
2014-06-12 14:46:52.036993 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.036989 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:52.037001 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] do_osd_op  write 352256~516096
2014-06-12 14:46:52.037015 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:52.037027 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 snapset=0x93458b8  snapc=0=[]
2014-06-12 14:46:52.037039 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:52.037050 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] finish_ctx e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 0x5e4c400 op modify  
2014-06-12 14:46:52.037063 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean]  set mtime to 2014-06-12 08:45:54.406506
2014-06-12 14:46:52.037087 7f9d81970700 15 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.037094 7f9d81970700 10 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1 r = 0
2014-06-12 14:46:52.037098 7f9d81970700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.037111 7f9d81970700 15 _write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 0~352256
2014-06-12 14:46:52.037108 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean]  final snapset 0=[]:[]+head in e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.037126 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean]  zeroing write result code 0
2014-06-12 14:46:52.037137 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] new_repop rep_tid 236347 on osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.037155 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] issue_repop rep_tid 236347 o e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.037178 7f9d66bde700  5 -- op tracker -- , seq: 748112, time: 2014-06-12 14:46:52.037172, event: waiting for subops from 21, op: osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.037195 7f9d81970700 10 lookup_strip_header done  cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.037197 7f9d66bde700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:52.037214 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2955 (0'0,173'2955] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] append_log log((0'0,173'2955], crt=0'0) [173'2956 (173'2955) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:143996 2014-06-12 08:45:54.406506]
2014-06-12 14:46:52.037238 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] add_log_entry 173'2956 (173'2955) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:143996 2014-06-12 08:45:54.406506
2014-06-12 14:46:52.037258 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] append_log  adding 1 keys
2014-06-12 14:46:52.037284 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2956, trimmed: 
2014-06-12 14:46:52.037305 7f9d66bde700  5 queue_transactions existing osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.037308 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.037309 7f9d66bde700  5 queue_op 0x69aed80 seq 2619 osr(3.3a 0x276cce0) 518380 bytes   (queue has 2 ops and 872920 bytes)
2014-06-12 14:46:52.037320 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b70e0 (2 active)
2014-06-12 14:46:52.037317 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] eval_repop repgather(0xeaacfc0 173'2956 rep_tid=236347 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.037348 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:52.037348 7f9d66bde700 10 osd.2 173 dequeue_op 0x24c9d500 finish
2014-06-12 14:46:52.037354 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.037357 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.038380 7f9d81970700 10 _generic_write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 0~352256 = 0
2014-06-12 14:46:52.038416 7f9d81970700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.038427 7f9d81970700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.038431 7f9d81970700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.038436 7f9d81970700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.038554 7f9d81970700 20 set_header setting 16858 cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 parent seq 0
2014-06-12 14:46:52.038596 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.038606 7f9d81970700 20 set_header setting 1980 cid meta oid 5168262c/pglog_3.3a/0//-1 parent seq 0
2014-06-12 14:46:52.038614 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:52.041033 7f9d81970700 10 _do_op 0x5949ce0 seq 2618 r = 0, finisher 0x6cb9ca0 0x42cdac0
2014-06-12 14:46:52.041050 7f9d81970700 10 _finish_op 0x5949ce0 seq 2618 osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.041058 7f9d7a962700  5 _do_op 0x69aed80 seq 2619 osr(3.3a 0x276cce0)/0x276cce0 start
2014-06-12 14:46:52.041061 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b70e0 (1 active)
2014-06-12 14:46:52.041064 7f9d7a962700 10 _do_transaction on 0xeaad500
2014-06-12 14:46:52.041065 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.041071 7f9d7a962700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.041056 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] op_commit: 236346
2014-06-12 14:46:52.041089 7f9d7715b700  5 -- op tracker -- , seq: 748111, time: 2014-06-12 14:46:52.041078, event: op_commit, op: osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.041109 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] op_applied: 236346
2014-06-12 14:46:52.041137 7f9d7795c700  5 -- op tracker -- , seq: 748111, time: 2014-06-12 14:46:52.041131, event: op_applied, op: osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.041153 7f9d7a962700 10 lookup_strip_header done  cid meta oid 5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.041148 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] op_applied on primary on version 173'2955
2014-06-12 14:46:52.041170 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.041200 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.041276 7f9d7a962700 15 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.041282 7f9d7a962700 10 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1 r = 0
2014-06-12 14:46:52.041286 7f9d7a962700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.041299 7f9d7a962700 15 _write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 352256~516096
2014-06-12 14:46:52.041389 7f9d7a962700 10 lookup_strip_header done  cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.041499 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:52.042805 7f9d7a962700 10 _generic_write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 352256~516096 = 0
2014-06-12 14:46:52.042836 7f9d7a962700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.042850 7f9d7a962700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.042855 7f9d7a962700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.042860 7f9d7a962700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.042996 7f9d7a962700 20 set_header setting 16858 cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 parent seq 0
2014-06-12 14:46:52.043046 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.043060 7f9d7a962700 20 set_header setting 1980 cid meta oid 5168262c/pglog_3.3a/0//-1 parent seq 0
2014-06-12 14:46:52.043066 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:52.045002 7f9d7a962700 10 _do_op 0x69aed80 seq 2619 r = 0, finisher 0x5fefb00 0x11fb4180
2014-06-12 14:46:52.045015 7f9d7a962700 10 _finish_op 0x69aed80 seq 2619 osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.045022 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b70e0 (0 active)
2014-06-12 14:46:52.045027 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.045020 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] op_commit: 236347
2014-06-12 14:46:52.045028 7f9d44101700  5 -- op tracker -- , seq: 748113, time: 2014-06-12 14:46:52.044987, event: header_read, op: osd_sub_op_reply(client.4317.0:143995 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.045058 7f9d44101700  5 -- op tracker -- , seq: 748113, time: 2014-06-12 14:46:52.044989, event: throttled, op: osd_sub_op_reply(client.4317.0:143995 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.045065 7f9d44101700  5 -- op tracker -- , seq: 748113, time: 2014-06-12 14:46:52.045000, event: all_read, op: osd_sub_op_reply(client.4317.0:143995 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.045076 7f9d44101700  5 -- op tracker -- , seq: 748113, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143995 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.045082 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143995 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:52.045086 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:52.045090 7f9d7715b700  5 -- op tracker -- , seq: 748112, time: 2014-06-12 14:46:52.045040, event: op_commit, op: osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.045093 7f9d44101700 15 osd.2 173 enqueue_op 0x6f4ba00 prio 196 cost 0 latency 0.000106 osd_sub_op_reply(client.4317.0:143995 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:52.045104 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] op_applied: 236347
2014-06-12 14:46:52.045134 7f9d7795c700  5 -- op tracker -- , seq: 748112, time: 2014-06-12 14:46:52.045129, event: op_applied, op: osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.045137 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.045145 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] op_applied on primary on version 173'2956
2014-06-12 14:46:52.045157 7f9d64bda700 10 osd.2 173 dequeue_op 0x6f4ba00 prio 196 cost 0 latency 0.000170 osd_sub_op_reply(client.4317.0:143995 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 pg pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean]
2014-06-12 14:46:52.045175 7f9d64bda700  5 -- op tracker -- , seq: 748113, time: 2014-06-12 14:46:52.045172, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143995 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.045191 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] handle_message: 0x6f4ba00
2014-06-12 14:46:52.045204 7f9d64bda700  5 -- op tracker -- , seq: 748113, time: 2014-06-12 14:46:52.045201, event: started, op: osd_sub_op_reply(client.4317.0:143995 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.045212 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] sub_op_modify_reply: tid 236346 op  ack_type 4 from 21
2014-06-12 14:46:52.045225 7f9d64bda700  5 -- op tracker -- , seq: 748111, time: 2014-06-12 14:46:52.045220, event: sub_op_commit_rec, op: osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.045230 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] repop_all_applied: repop tid 236346 all applied 
2014-06-12 14:46:52.045237 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] eval_repop repgather(0x24fa7200 173'2955 rep_tid=236346 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.045248 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean]  sending ack on repgather(0x24fa7200 173'2955 rep_tid=236346 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x18419400
2014-06-12 14:46:52.045269 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2954 crt=0'0 lcod 173'2953 mlcod 173'2953 active+clean] repop_all_committed: repop tid 236346 all committed 
2014-06-12 14:46:52.045276 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2953 active+clean] eval_repop repgather(0x24fa7200 173'2955 rep_tid=236346 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.045286 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2953 active+clean] log_op_stats osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 inb 352533 outb 0 rlat 0.027794 lat 0.027811
2014-06-12 14:46:52.045297 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2953 active+clean] publish_stats_to_osd 173:3837
2014-06-12 14:46:52.045305 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2953 active+clean]  sending commit on repgather(0x24fa7200 173'2955 rep_tid=236346 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x28aac500
2014-06-12 14:46:52.045324 7f9d64bda700  5 -- op tracker -- , seq: 748111, time: 2014-06-12 14:46:52.045321, event: commit_sent, op: osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.045330 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2953 active+clean]  requeue_ops 
2014-06-12 14:46:52.045337 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean]  removing repgather(0x24fa7200 173'2955 rep_tid=236346 committed?=1 applied?=1 op=osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.045349 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean]    q front is repgather(0x24fa7200 173'2955 rep_tid=236346 committed?=1 applied?=1 op=osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.045359 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] remove_repop repgather(0x24fa7200 173'2955 rep_tid=236346 committed?=1 applied?=1 op=osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.045368 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean]  requeue_ops 
2014-06-12 14:46:52.045380 7f9d64bda700  5 -- op tracker -- , seq: 748111, time: 2014-06-12 14:46:52.045377, event: done, op: osd_op(client.4317.0:143995 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.045393 7f9d64bda700 10 osd.2 173 dequeue_op 0x6f4ba00 finish
2014-06-12 14:46:52.045399 7f9d64bda700  5 -- op tracker -- , seq: 748113, time: 2014-06-12 14:46:52.045397, event: done, op: osd_sub_op_reply(client.4317.0:143995 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.045412 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.045415 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.048784 7f9d44424700  5 -- op tracker -- , seq: 748114, time: 2014-06-12 14:46:52.047743, event: header_read, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.048794 7f9d44424700  5 -- op tracker -- , seq: 748114, time: 2014-06-12 14:46:52.047746, event: throttled, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.048800 7f9d44424700  5 -- op tracker -- , seq: 748114, time: 2014-06-12 14:46:52.048754, event: all_read, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.048805 7f9d44424700  5 -- op tracker -- , seq: 748114, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.048812 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:52.048833 7f9d44424700 15 osd.2 173 enqueue_op 0x598e000 prio 63 cost 516096 latency 0.001090 osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.048888 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.048899 7f9d6abe6700 10 osd.2 173 dequeue_op 0x598e000 prio 63 cost 516096 latency 0.001155 osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean]
2014-06-12 14:46:52.048922 7f9d6abe6700  5 -- op tracker -- , seq: 748114, time: 2014-06-12 14:46:52.048918, event: reached_pg, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.048932 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:52.048945 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] handle_message: 0x598e000
2014-06-12 14:46:52.048953 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] do_op osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:52.048973 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] get_object_context: found obc in cache: 0x2e7ad8c0
2014-06-12 14:46:52.048980 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] get_object_context: 0x2e7ad8c0 e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 rwstate(write n=1 w=0) oi: e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2956 client.4317.0:143996 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2956) ssc: 0x9345880 snapset: 0=[]:[]+head
2014-06-12 14:46:52.048991 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] find_object_context e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 @head oi=e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2956 client.4317.0:143996 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2956)
2014-06-12 14:46:52.049011 7f9d6abe6700  5 -- op tracker -- , seq: 748114, time: 2014-06-12 14:46:52.049007, event: started, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.049017 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] execute_ctx 0x7ad1200
2014-06-12 14:46:52.049028 7f9d6abe6700  5 -- op tracker -- , seq: 748114, time: 2014-06-12 14:46:52.049025, event: started, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.049033 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] do_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] ov 173'2956 av 173'2957 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:52.049043 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] do_osd_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096]
2014-06-12 14:46:52.049050 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:52.049060 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] do_osd_op  write 868352~516096
2014-06-12 14:46:52.049071 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:52.049079 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 snapset=0x93458b8  snapc=0=[]
2014-06-12 14:46:52.049092 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:52.049099 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] finish_ctx e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 0x7ad1200 op modify  
2014-06-12 14:46:52.049107 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean]  set mtime to 2014-06-12 08:45:54.427117
2014-06-12 14:46:52.049123 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean]  final snapset 0=[]:[]+head in e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.049137 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean]  zeroing write result code 0
2014-06-12 14:46:52.049145 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] new_repop rep_tid 236348 on osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.049160 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] issue_repop rep_tid 236348 o e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.049177 7f9d6abe6700  5 -- op tracker -- , seq: 748114, time: 2014-06-12 14:46:52.049174, event: waiting for subops from 21, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.049195 7f9d6abe6700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:52.049211 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2956 (0'0,173'2956] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] append_log log((0'0,173'2956], crt=0'0) [173'2957 (173'2956) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:143997 2014-06-12 08:45:54.427117]
2014-06-12 14:46:52.049228 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] add_log_entry 173'2957 (173'2956) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:143997 2014-06-12 08:45:54.427117
2014-06-12 14:46:52.049242 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] append_log  adding 1 keys
2014-06-12 14:46:52.049265 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2957, trimmed: 
2014-06-12 14:46:52.049282 7f9d6abe6700  5 queue_transactions existing osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.049286 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.049287 7f9d6abe6700  5 queue_op 0x6d9ec60 seq 2620 osr(3.3a 0x276cce0) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:46:52.049294 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b70e0 (1 active)
2014-06-12 14:46:52.049299 7f9d82972700  5 _do_op 0x6d9ec60 seq 2620 osr(3.3a 0x276cce0)/0x276cce0 start
2014-06-12 14:46:52.049292 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] eval_repop repgather(0x34d8e300 173'2957 rep_tid=236348 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.049302 7f9d82972700 10 _do_transaction on 0x224e180
2014-06-12 14:46:52.049303 7f9d6abe6700 10 osd.2 173 dequeue_op 0x598e000 finish
2014-06-12 14:46:52.049306 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.049308 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.049307 7f9d82972700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.049401 7f9d82972700 10 lookup_strip_header done  cid meta oid 5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.049422 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.049454 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.049541 7f9d82972700 15 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.049546 7f9d82972700 10 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1 r = 0
2014-06-12 14:46:52.049550 7f9d82972700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.049563 7f9d82972700 15 _write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 868352~516096
2014-06-12 14:46:52.049686 7f9d82972700 10 lookup_strip_header done  cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.049811 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:52.051447 7f9d82972700 10 _generic_write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 868352~516096 = 0
2014-06-12 14:46:52.051489 7f9d82972700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.051507 7f9d82972700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.051517 7f9d82972700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.051525 7f9d82972700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.051733 7f9d82972700 20 set_header setting 16858 cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 parent seq 0
2014-06-12 14:46:52.051811 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.051830 7f9d82972700 20 set_header setting 1980 cid meta oid 5168262c/pglog_3.3a/0//-1 parent seq 0
2014-06-12 14:46:52.051838 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:52.052147 7f9d44101700  5 -- op tracker -- , seq: 748115, time: 2014-06-12 14:46:52.052112, event: header_read, op: osd_sub_op_reply(client.4317.0:143996 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.052158 7f9d44101700  5 -- op tracker -- , seq: 748115, time: 2014-06-12 14:46:52.052114, event: throttled, op: osd_sub_op_reply(client.4317.0:143996 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.052165 7f9d44101700  5 -- op tracker -- , seq: 748115, time: 2014-06-12 14:46:52.052123, event: all_read, op: osd_sub_op_reply(client.4317.0:143996 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.052175 7f9d44101700  5 -- op tracker -- , seq: 748115, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143996 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.052179 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143996 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:52.052187 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:52.052194 7f9d44101700 15 osd.2 173 enqueue_op 0x34017b00 prio 196 cost 0 latency 0.000081 osd_sub_op_reply(client.4317.0:143996 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:52.052245 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.052255 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34017b00 prio 196 cost 0 latency 0.000143 osd_sub_op_reply(client.4317.0:143996 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 pg pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean]
2014-06-12 14:46:52.052274 7f9d6a3e5700  5 -- op tracker -- , seq: 748115, time: 2014-06-12 14:46:52.052272, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143996 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.052283 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] handle_message: 0x34017b00
2014-06-12 14:46:52.052295 7f9d6a3e5700  5 -- op tracker -- , seq: 748115, time: 2014-06-12 14:46:52.052293, event: started, op: osd_sub_op_reply(client.4317.0:143996 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.052300 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] sub_op_modify_reply: tid 236347 op  ack_type 4 from 21
2014-06-12 14:46:52.052315 7f9d6a3e5700  5 -- op tracker -- , seq: 748112, time: 2014-06-12 14:46:52.052310, event: sub_op_commit_rec, op: osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.052322 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] repop_all_applied: repop tid 236347 all applied 
2014-06-12 14:46:52.052330 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] eval_repop repgather(0xeaacfc0 173'2956 rep_tid=236347 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.052341 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean]  sending ack on repgather(0xeaacfc0 173'2956 rep_tid=236347 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x5f94500
2014-06-12 14:46:52.052363 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2955 crt=0'0 lcod 173'2954 mlcod 173'2954 active+clean] repop_all_committed: repop tid 236347 all committed 
2014-06-12 14:46:52.052370 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2954 active+clean] eval_repop repgather(0xeaacfc0 173'2956 rep_tid=236347 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.052380 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2954 active+clean] log_op_stats osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 inb 516373 outb 0 rlat 0.025025 lat 0.025043
2014-06-12 14:46:52.052390 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2954 active+clean] publish_stats_to_osd 173:3838
2014-06-12 14:46:52.052399 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2954 active+clean]  sending commit on repgather(0xeaacfc0 173'2956 rep_tid=236347 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x16cd2f00
2014-06-12 14:46:52.052419 7f9d6a3e5700  5 -- op tracker -- , seq: 748112, time: 2014-06-12 14:46:52.052415, event: commit_sent, op: osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.052425 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2954 active+clean]  requeue_ops 
2014-06-12 14:46:52.052431 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean]  removing repgather(0xeaacfc0 173'2956 rep_tid=236347 committed?=1 applied?=1 op=osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.052440 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean]    q front is repgather(0xeaacfc0 173'2956 rep_tid=236347 committed?=1 applied?=1 op=osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.052449 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] remove_repop repgather(0xeaacfc0 173'2956 rep_tid=236347 committed?=1 applied?=1 op=osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.052458 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean]  requeue_ops 
2014-06-12 14:46:52.052470 7f9d6a3e5700  5 -- op tracker -- , seq: 748112, time: 2014-06-12 14:46:52.052467, event: done, op: osd_op(client.4317.0:143996 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.052486 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34017b00 finish
2014-06-12 14:46:52.052491 7f9d6a3e5700  5 -- op tracker -- , seq: 748115, time: 2014-06-12 14:46:52.052489, event: done, op: osd_sub_op_reply(client.4317.0:143996 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.052498 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.052500 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.053499 7f9d44424700  5 -- op tracker -- , seq: 748116, time: 2014-06-12 14:46:52.053433, event: header_read, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.053508 7f9d44424700  5 -- op tracker -- , seq: 748116, time: 2014-06-12 14:46:52.053436, event: throttled, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.053514 7f9d44424700  5 -- op tracker -- , seq: 748116, time: 2014-06-12 14:46:52.053456, event: all_read, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.053519 7f9d44424700  5 -- op tracker -- , seq: 748116, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.053524 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:52.053543 7f9d44424700 15 osd.2 173 enqueue_op 0x20e86500 prio 63 cost 4096 latency 0.000110 osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.053564 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.053572 7f9d69be4700 10 osd.2 173 dequeue_op 0x20e86500 prio 63 cost 4096 latency 0.000138 osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean]
2014-06-12 14:46:52.053588 7f9d69be4700  5 -- op tracker -- , seq: 748116, time: 2014-06-12 14:46:52.053585, event: reached_pg, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.053595 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:52.053625 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] handle_message: 0x20e86500
2014-06-12 14:46:52.053643 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] do_op osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:52.053674 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] get_object_context: found obc in cache: 0x2e7ad8c0
2014-06-12 14:46:52.053682 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] get_object_context: 0x2e7ad8c0 e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 rwstate(write n=1 w=0) oi: e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2957 client.4317.0:143997 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2957) ssc: 0x9345880 snapset: 0=[]:[]+head
2014-06-12 14:46:52.053693 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] find_object_context e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 @head oi=e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2957 client.4317.0:143997 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2957)
2014-06-12 14:46:52.053712 7f9d69be4700  5 -- op tracker -- , seq: 748116, time: 2014-06-12 14:46:52.053708, event: started, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.053717 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] execute_ctx 0x7ad0c00
2014-06-12 14:46:52.053728 7f9d69be4700  5 -- op tracker -- , seq: 748116, time: 2014-06-12 14:46:52.053725, event: started, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.053733 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] do_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] ov 173'2957 av 173'2958 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:52.053743 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] do_osd_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096]
2014-06-12 14:46:52.053750 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:52.053759 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] do_osd_op  write 1384448~4096
2014-06-12 14:46:52.053769 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:52.053777 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 snapset=0x93458b8  snapc=0=[]
2014-06-12 14:46:52.053786 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:52.053792 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] finish_ctx e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 0x7ad0c00 op modify  
2014-06-12 14:46:52.053799 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean]  set mtime to 2014-06-12 08:45:54.433802
2014-06-12 14:46:52.053813 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean]  final snapset 0=[]:[]+head in e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.053825 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean]  zeroing write result code 0
2014-06-12 14:46:52.053833 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] new_repop rep_tid 236349 on osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.053844 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] issue_repop rep_tid 236349 o e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.053860 7f9d69be4700  5 -- op tracker -- , seq: 748116, time: 2014-06-12 14:46:52.053857, event: waiting for subops from 21, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.053878 7f9d69be4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:52.053891 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2957 (0'0,173'2957] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] append_log log((0'0,173'2957], crt=0'0) [173'2958 (173'2957) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:143998 2014-06-12 08:45:54.433802]
2014-06-12 14:46:52.053907 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] add_log_entry 173'2958 (173'2957) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:143998 2014-06-12 08:45:54.433802
2014-06-12 14:46:52.053921 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] append_log  adding 1 keys
2014-06-12 14:46:52.053941 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2958, trimmed: 
2014-06-12 14:46:52.053959 7f9d69be4700  5 queue_transactions existing osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.053963 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.053964 7f9d69be4700  5 queue_op 0x802fc80 seq 2621 osr(3.3a 0x276cce0) 6380 bytes   (queue has 2 ops and 524760 bytes)
2014-06-12 14:46:52.053969 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] eval_repop repgather(0x32ff5740 173'2958 rep_tid=236349 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.053988 7f9d69be4700 10 osd.2 173 dequeue_op 0x20e86500 finish
2014-06-12 14:46:52.053991 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.053993 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.054001 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b70e0 (2 active)
2014-06-12 14:46:52.054226 7f9d82972700 10 _do_op 0x6d9ec60 seq 2620 r = 0, finisher 0x62cd060 0x352bd340
2014-06-12 14:46:52.054248 7f9d82972700 10 _finish_op 0x6d9ec60 seq 2620 osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.054260 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b70e0 (1 active)
2014-06-12 14:46:52.054275 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.054256 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] op_commit: 236348
2014-06-12 14:46:52.054290 7f9d7895e700  5 _do_op 0x802fc80 seq 2621 osr(3.3a 0x276cce0)/0x276cce0 start
2014-06-12 14:46:52.054291 7f9d7715b700  5 -- op tracker -- , seq: 748114, time: 2014-06-12 14:46:52.054283, event: op_commit, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.054303 7f9d7895e700 10 _do_transaction on 0x32ff5140
2014-06-12 14:46:52.054313 7f9d7895e700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.054312 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] op_applied: 236348
2014-06-12 14:46:52.054348 7f9d7795c700  5 -- op tracker -- , seq: 748114, time: 2014-06-12 14:46:52.054338, event: op_applied, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.054369 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] op_applied on primary on version 173'2957
2014-06-12 14:46:52.054433 7f9d7895e700 10 lookup_strip_header done  cid meta oid 5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.054474 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.054534 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.054566 7f9d7895e700 15 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.054577 7f9d7895e700 10 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1 r = 0
2014-06-12 14:46:52.054585 7f9d7895e700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.054701 7f9d7895e700 15 _write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 1384448~4096
2014-06-12 14:46:52.054888 7f9d7895e700 10 lookup_strip_header done  cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.054904 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:52.054928 7f9d7895e700 10 _generic_write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 1384448~4096 = 0
2014-06-12 14:46:52.054941 7f9d7895e700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.054956 7f9d7895e700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.054963 7f9d7895e700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.054973 7f9d7895e700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.055115 7f9d7895e700 20 set_header setting 16858 cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 parent seq 0
2014-06-12 14:46:52.055143 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.055162 7f9d7895e700 20 set_header setting 1980 cid meta oid 5168262c/pglog_3.3a/0//-1 parent seq 0
2014-06-12 14:46:52.055178 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:52.055302 7f9d7895e700 10 _do_op 0x802fc80 seq 2621 r = 0, finisher 0x28ae920 0x26c4f640
2014-06-12 14:46:52.055316 7f9d7895e700 10 _finish_op 0x802fc80 seq 2621 osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.055325 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b70e0 (0 active)
2014-06-12 14:46:52.055329 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.055321 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] op_commit: 236349
2014-06-12 14:46:52.055347 7f9d7715b700  5 -- op tracker -- , seq: 748116, time: 2014-06-12 14:46:52.055340, event: op_commit, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.055369 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] op_applied: 236349
2014-06-12 14:46:52.055395 7f9d7795c700  5 -- op tracker -- , seq: 748116, time: 2014-06-12 14:46:52.055387, event: op_applied, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.055422 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] op_applied on primary on version 173'2958
2014-06-12 14:46:52.062341 7f9d44101700  5 -- op tracker -- , seq: 748117, time: 2014-06-12 14:46:52.062290, event: header_read, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.062365 7f9d44101700  5 -- op tracker -- , seq: 748117, time: 2014-06-12 14:46:52.062293, event: throttled, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.062375 7f9d44101700  5 -- op tracker -- , seq: 748117, time: 2014-06-12 14:46:52.062306, event: all_read, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.062385 7f9d44101700  5 -- op tracker -- , seq: 748117, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.062391 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:52.062397 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:52.062405 7f9d44101700 15 osd.2 173 enqueue_op 0x2a903c00 prio 196 cost 0 latency 0.000115 osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0) v2
2014-06-12 14:46:52.062433 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.062439 7f9d653db700 10 osd.2 173 dequeue_op 0x2a903c00 prio 196 cost 0 latency 0.000149 osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0) v2 pg pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean]
2014-06-12 14:46:52.062457 7f9d653db700  5 -- op tracker -- , seq: 748117, time: 2014-06-12 14:46:52.062454, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.062468 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] handle_message: 0x2a903c00
2014-06-12 14:46:52.062473 7f9d44101700  5 -- op tracker -- , seq: 748118, time: 2014-06-12 14:46:52.062429, event: header_read, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.062492 7f9d44101700  5 -- op tracker -- , seq: 748118, time: 2014-06-12 14:46:52.062429, event: throttled, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.062504 7f9d44101700  5 -- op tracker -- , seq: 748118, time: 2014-06-12 14:46:52.062439, event: all_read, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.062514 7f9d44101700  5 -- op tracker -- , seq: 748118, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.062520 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:52.062525 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:52.062525 7f9d653db700  5 -- op tracker -- , seq: 748117, time: 2014-06-12 14:46:52.062483, event: started, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.062530 7f9d44101700 15 osd.2 173 enqueue_op 0x23e08a00 prio 196 cost 0 latency 0.000101 osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:52.062531 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] sub_op_modify_reply: tid 236348 op  ack_type 1 from 21
2014-06-12 14:46:52.062549 7f9d653db700  5 -- op tracker -- , seq: 748114, time: 2014-06-12 14:46:52.062543, event: sub_op_applied_rec, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.062569 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] repop_all_applied: repop tid 236348 all applied 
2014-06-12 14:46:52.062579 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:52.062577 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] eval_repop repgather(0x34d8e300 173'2957 rep_tid=236348 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.062587 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean]  sending ack on repgather(0x34d8e300 173'2957 rep_tid=236348 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x355c0280
2014-06-12 14:46:52.062613 7f9d653db700 10 osd.2 173 dequeue_op 0x2a903c00 finish
2014-06-12 14:46:52.062620 7f9d653db700  5 -- op tracker -- , seq: 748117, time: 2014-06-12 14:46:52.062618, event: done, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.062637 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:52.062639 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.062651 7f9d63bd8700 10 osd.2 173 dequeue_op 0x23e08a00 prio 196 cost 0 latency 0.000222 osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 pg pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean]
2014-06-12 14:46:52.062695 7f9d63bd8700  5 -- op tracker -- , seq: 748118, time: 2014-06-12 14:46:52.062684, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.062705 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] handle_message: 0x23e08a00
2014-06-12 14:46:52.062724 7f9d63bd8700  5 -- op tracker -- , seq: 748118, time: 2014-06-12 14:46:52.062721, event: started, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.062728 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] sub_op_modify_reply: tid 236348 op  ack_type 4 from 21
2014-06-12 14:46:52.062740 7f9d63bd8700  5 -- op tracker -- , seq: 748114, time: 2014-06-12 14:46:52.062736, event: sub_op_commit_rec, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.062744 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2956 crt=0'0 lcod 173'2955 mlcod 173'2955 active+clean] repop_all_committed: repop tid 236348 all committed 
2014-06-12 14:46:52.062750 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2955 active+clean] eval_repop repgather(0x34d8e300 173'2957 rep_tid=236348 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.062760 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2955 active+clean] log_op_stats osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 inb 516373 outb 0 rlat 0.014869 lat 0.015017
2014-06-12 14:46:52.062767 7f9d44101700  5 -- op tracker -- , seq: 748119, time: 2014-06-12 14:46:52.062740, event: header_read, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.062772 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2955 active+clean] publish_stats_to_osd 173:3839
2014-06-12 14:46:52.062779 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2955 active+clean]  sending commit on repgather(0x34d8e300 173'2957 rep_tid=236348 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x151e6f00
2014-06-12 14:46:52.062778 7f9d44101700  5 -- op tracker -- , seq: 748119, time: 2014-06-12 14:46:52.062741, event: throttled, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.062806 7f9d44101700  5 -- op tracker -- , seq: 748119, time: 2014-06-12 14:46:52.062748, event: all_read, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.062812 7f9d44101700  5 -- op tracker -- , seq: 748119, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.062815 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:52.062818 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:52.062822 7f9d44101700 15 osd.2 173 enqueue_op 0x5ee8800 prio 196 cost 0 latency 0.000082 osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0) v2
2014-06-12 14:46:52.062822 7f9d63bd8700  5 -- op tracker -- , seq: 748114, time: 2014-06-12 14:46:52.062797, event: commit_sent, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.062829 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2955 active+clean]  requeue_ops 
2014-06-12 14:46:52.062840 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:52.062837 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean]  removing repgather(0x34d8e300 173'2957 rep_tid=236348 committed?=1 applied?=1 op=osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.062847 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean]    q front is repgather(0x34d8e300 173'2957 rep_tid=236348 committed?=1 applied?=1 op=osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.062856 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean] remove_repop repgather(0x34d8e300 173'2957 rep_tid=236348 committed?=1 applied?=1 op=osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.062860 7f9d44101700  5 -- op tracker -- , seq: 748120, time: 2014-06-12 14:46:52.062835, event: header_read, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.062865 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean]  requeue_ops 
2014-06-12 14:46:52.062868 7f9d44101700  5 -- op tracker -- , seq: 748120, time: 2014-06-12 14:46:52.062835, event: throttled, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.062873 7f9d44101700  5 -- op tracker -- , seq: 748120, time: 2014-06-12 14:46:52.062846, event: all_read, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.062884 7f9d63bd8700  5 -- op tracker -- , seq: 748114, time: 2014-06-12 14:46:52.062874, event: done, op: osd_op(client.4317.0:143997 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.062899 7f9d63bd8700 10 osd.2 173 dequeue_op 0x23e08a00 finish
2014-06-12 14:46:52.062904 7f9d44101700  5 -- op tracker -- , seq: 748120, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.062911 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:52.062916 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:52.062920 7f9d63bd8700  5 -- op tracker -- , seq: 748118, time: 2014-06-12 14:46:52.062902, event: done, op: osd_sub_op_reply(client.4317.0:143997 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.062921 7f9d44101700 15 osd.2 173 enqueue_op 0x5ee9200 prio 196 cost 0 latency 0.000086 osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:52.062907 7f9d68be2700 10 osd.2 173 dequeue_op 0x5ee8800 prio 196 cost 0 latency 0.000167 osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0) v2 pg pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean]
2014-06-12 14:46:52.062940 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:52.062937 7f9d68be2700  5 -- op tracker -- , seq: 748119, time: 2014-06-12 14:46:52.062933, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.062948 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:52.062949 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean] handle_message: 0x5ee8800
2014-06-12 14:46:52.062956 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.062961 7f9d68be2700  5 -- op tracker -- , seq: 748119, time: 2014-06-12 14:46:52.062958, event: started, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.062970 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean] sub_op_modify_reply: tid 236349 op  ack_type 1 from 21
2014-06-12 14:46:52.062990 7f9d68be2700  5 -- op tracker -- , seq: 748116, time: 2014-06-12 14:46:52.062986, event: sub_op_applied_rec, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.062994 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean] repop_all_applied: repop tid 236349 all applied 
2014-06-12 14:46:52.063000 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean] eval_repop repgather(0x32ff5740 173'2958 rep_tid=236349 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.063009 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean]  sending ack on repgather(0x32ff5740 173'2958 rep_tid=236349 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0xc0c8500
2014-06-12 14:46:52.063026 7f9d68be2700 10 osd.2 173 dequeue_op 0x5ee8800 finish
2014-06-12 14:46:52.063031 7f9d68be2700  5 -- op tracker -- , seq: 748119, time: 2014-06-12 14:46:52.063029, event: done, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.063053 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:52.063055 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.063062 7f9d63bd8700 10 osd.2 173 dequeue_op 0x5ee9200 prio 196 cost 0 latency 0.000226 osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 pg pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean]
2014-06-12 14:46:52.063086 7f9d63bd8700  5 -- op tracker -- , seq: 748120, time: 2014-06-12 14:46:52.063084, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.063093 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean] handle_message: 0x5ee9200
2014-06-12 14:46:52.063102 7f9d63bd8700  5 -- op tracker -- , seq: 748120, time: 2014-06-12 14:46:52.063100, event: started, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.063107 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean] sub_op_modify_reply: tid 236349 op  ack_type 4 from 21
2014-06-12 14:46:52.063118 7f9d63bd8700  5 -- op tracker -- , seq: 748116, time: 2014-06-12 14:46:52.063114, event: sub_op_commit_rec, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.063123 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2957 crt=0'0 lcod 173'2956 mlcod 173'2956 active+clean] repop_all_committed: repop tid 236349 all committed 
2014-06-12 14:46:52.063132 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2956 active+clean] eval_repop repgather(0x32ff5740 173'2958 rep_tid=236349 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.063143 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2956 active+clean] log_op_stats osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 inb 4373 outb 0 rlat 0.009592 lat 0.009710
2014-06-12 14:46:52.063154 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2956 active+clean] publish_stats_to_osd 173:3840
2014-06-12 14:46:52.063160 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2956 active+clean]  sending commit on repgather(0x32ff5740 173'2958 rep_tid=236349 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x26e57180
2014-06-12 14:46:52.063181 7f9d63bd8700  5 -- op tracker -- , seq: 748116, time: 2014-06-12 14:46:52.063178, event: commit_sent, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.063187 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2956 active+clean]  requeue_ops 
2014-06-12 14:46:52.063193 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean]  removing repgather(0x32ff5740 173'2958 rep_tid=236349 committed?=1 applied?=1 op=osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.063201 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean]    q front is repgather(0x32ff5740 173'2958 rep_tid=236349 committed?=1 applied?=1 op=osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.063208 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] remove_repop repgather(0x32ff5740 173'2958 rep_tid=236349 committed?=1 applied?=1 op=osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.063216 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean]  requeue_ops 
2014-06-12 14:46:52.063230 7f9d63bd8700  5 -- op tracker -- , seq: 748116, time: 2014-06-12 14:46:52.063227, event: done, op: osd_op(client.4317.0:143998 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~4096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.063240 7f9d63bd8700 10 osd.2 173 dequeue_op 0x5ee9200 finish
2014-06-12 14:46:52.063245 7f9d63bd8700  5 -- op tracker -- , seq: 748120, time: 2014-06-12 14:46:52.063242, event: done, op: osd_sub_op_reply(client.4317.0:143998 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.063252 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.063254 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.066304 7f9d44424700  5 -- op tracker -- , seq: 748121, time: 2014-06-12 14:46:52.065187, event: header_read, op: osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.066326 7f9d44424700  5 -- op tracker -- , seq: 748121, time: 2014-06-12 14:46:52.065190, event: throttled, op: osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.066336 7f9d44424700  5 -- op tracker -- , seq: 748121, time: 2014-06-12 14:46:52.066269, event: all_read, op: osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.066353 7f9d44424700  5 -- op tracker -- , seq: 748121, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.066361 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:52.066384 7f9d44424700 15 osd.2 173 enqueue_op 0x20e86c00 prio 63 cost 516096 latency 0.001197 osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.066446 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.066469 7f9d643d9700 10 osd.2 173 dequeue_op 0x20e86c00 prio 63 cost 516096 latency 0.001281 osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean]
2014-06-12 14:46:52.066514 7f9d643d9700  5 -- op tracker -- , seq: 748121, time: 2014-06-12 14:46:52.066505, event: reached_pg, op: osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.066531 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:52.066557 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] handle_message: 0x20e86c00
2014-06-12 14:46:52.066564 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] do_op osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:52.066581 7f9d643d9700 15 getattr 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 '_'
2014-06-12 14:46:52.066767 7f9d643d9700 10 getattr 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 '_' = 0
2014-06-12 14:46:52.066793 7f9d643d9700 15 getattr 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 'snapset'
2014-06-12 14:46:52.066865 7f9d643d9700 10 getattr 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 'snapset' = 0
2014-06-12 14:46:52.066874 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] populate_obc_watchers e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.066882 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.066888 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] get_object_context: creating obc from disk: 0x1703d340
2014-06-12 14:46:52.066895 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] get_object_context: 0x1703d340 e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 rwstate(none n=0 w=0) oi: e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2958 client.4317.0:143998 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2958) ssc: 0x5d5a700 snapset: 0=[]:[]+head
2014-06-12 14:46:52.066905 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] find_object_context e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 @head oi=e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2958 client.4317.0:143998 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2958)
2014-06-12 14:46:52.066923 7f9d643d9700  5 -- op tracker -- , seq: 748121, time: 2014-06-12 14:46:52.066919, event: started, op: osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.066928 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] execute_ctx 0x2cec2400
2014-06-12 14:46:52.066938 7f9d643d9700  5 -- op tracker -- , seq: 748121, time: 2014-06-12 14:46:52.066936, event: started, op: osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.066942 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] do_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] ov 173'2958 av 173'2959 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:52.066951 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] do_osd_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096]
2014-06-12 14:46:52.066958 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:52.066966 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] do_osd_op  write 1388544~516096
2014-06-12 14:46:52.066976 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:52.066984 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 snapset=0x5d5a738  snapc=0=[]
2014-06-12 14:46:52.066992 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:52.066999 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] finish_ctx e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 0x2cec2400 op modify  
2014-06-12 14:46:52.067006 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean]  set mtime to 2014-06-12 08:45:54.444553
2014-06-12 14:46:52.067031 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean]  final snapset 0=[]:[]+head in e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.067045 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean]  zeroing write result code 0
2014-06-12 14:46:52.067053 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] new_repop rep_tid 236350 on osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.067063 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] issue_repop rep_tid 236350 o e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.067080 7f9d643d9700  5 -- op tracker -- , seq: 748121, time: 2014-06-12 14:46:52.067077, event: waiting for subops from 21, op: osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.067097 7f9d643d9700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:52.067112 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2958 (0'0,173'2958] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] append_log log((0'0,173'2958], crt=0'0) [173'2959 (173'2958) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:143999 2014-06-12 08:45:54.444553]
2014-06-12 14:46:52.067128 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] add_log_entry 173'2959 (173'2958) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:143999 2014-06-12 08:45:54.444553
2014-06-12 14:46:52.067142 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] append_log  adding 1 keys
2014-06-12 14:46:52.067165 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2959, trimmed: 
2014-06-12 14:46:52.067181 7f9d643d9700  5 queue_transactions existing osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.067185 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.067187 7f9d643d9700  5 queue_op 0x28e6ea0 seq 2622 osr(3.3a 0x276cce0) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:46:52.067191 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] eval_repop repgather(0x15e57080 173'2959 rep_tid=236350 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.067197 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b70e0 (1 active)
2014-06-12 14:46:52.067203 7f9d643d9700 10 osd.2 173 dequeue_op 0x20e86c00 finish
2014-06-12 14:46:52.067205 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.067207 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.067206 7f9d83173700  5 _do_op 0x28e6ea0 seq 2622 osr(3.3a 0x276cce0)/0x276cce0 start
2014-06-12 14:46:52.067214 7f9d83173700 10 _do_transaction on 0x6591b00
2014-06-12 14:46:52.067220 7f9d83173700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.067294 7f9d83173700 10 lookup_strip_header done  cid meta oid 5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.067314 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.067355 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.067375 7f9d83173700 15 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.067383 7f9d83173700 10 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1 r = 0
2014-06-12 14:46:52.067388 7f9d83173700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.067474 7f9d83173700 15 _write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 1388544~516096
2014-06-12 14:46:52.067610 7f9d83173700 10 lookup_strip_header done  cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.067790 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:52.068385 7f9d44424700  5 -- op tracker -- , seq: 748122, time: 2014-06-12 14:46:52.066755, event: header_read, op: osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.068397 7f9d44424700  5 -- op tracker -- , seq: 748122, time: 2014-06-12 14:46:52.066758, event: throttled, op: osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.068407 7f9d44424700  5 -- op tracker -- , seq: 748122, time: 2014-06-12 14:46:52.068361, event: all_read, op: osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.068418 7f9d44424700  5 -- op tracker -- , seq: 748122, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.068425 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:52.068446 7f9d44424700 15 osd.2 173 enqueue_op 0x20e86700 prio 63 cost 516096 latency 0.001691 osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.068477 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.068485 7f9d633d7700 10 osd.2 173 dequeue_op 0x20e86700 prio 63 cost 516096 latency 0.001730 osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean]
2014-06-12 14:46:52.068516 7f9d633d7700  5 -- op tracker -- , seq: 748122, time: 2014-06-12 14:46:52.068509, event: reached_pg, op: osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.068526 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:52.068542 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] handle_message: 0x20e86700
2014-06-12 14:46:52.068555 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] do_op osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:52.068581 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] get_object_context: found obc in cache: 0x1703d340
2014-06-12 14:46:52.068592 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] get_object_context: 0x1703d340 e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 rwstate(write n=1 w=0) oi: e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2959 client.4317.0:143999 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2959) ssc: 0x5d5a700 snapset: 0=[]:[]+head
2014-06-12 14:46:52.068610 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] find_object_context e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 @head oi=e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2959 client.4317.0:143999 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2959)
2014-06-12 14:46:52.068638 7f9d633d7700  5 -- op tracker -- , seq: 748122, time: 2014-06-12 14:46:52.068632, event: started, op: osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.068645 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] execute_ctx 0xfd18000
2014-06-12 14:46:52.068664 7f9d633d7700  5 -- op tracker -- , seq: 748122, time: 2014-06-12 14:46:52.068658, event: started, op: osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.068672 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] do_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] ov 173'2959 av 173'2960 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:52.068688 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] do_osd_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096]
2014-06-12 14:46:52.068703 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:52.068717 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] do_osd_op  write 1904640~516096
2014-06-12 14:46:52.068734 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:52.068748 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 snapset=0x5d5a738  snapc=0=[]
2014-06-12 14:46:52.068763 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:52.068776 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] finish_ctx e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 0xfd18000 op modify  
2014-06-12 14:46:52.068790 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean]  set mtime to 2014-06-12 08:45:54.445251
2014-06-12 14:46:52.068813 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean]  final snapset 0=[]:[]+head in e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.068834 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean]  zeroing write result code 0
2014-06-12 14:46:52.068846 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] new_repop rep_tid 236351 on osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.068867 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] issue_repop rep_tid 236351 o e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.068899 7f9d633d7700  5 -- op tracker -- , seq: 748122, time: 2014-06-12 14:46:52.068895, event: waiting for subops from 21, op: osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.068919 7f9d633d7700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:52.068940 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2959 (0'0,173'2959] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] append_log log((0'0,173'2959], crt=0'0) [173'2960 (173'2959) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:144000 2014-06-12 08:45:54.445251]
2014-06-12 14:46:52.068964 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] add_log_entry 173'2960 (173'2959) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:144000 2014-06-12 08:45:54.445251
2014-06-12 14:46:52.068982 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] append_log  adding 1 keys
2014-06-12 14:46:52.069009 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2960, trimmed: 
2014-06-12 14:46:52.069027 7f9d633d7700  5 queue_transactions existing osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.069030 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.069031 7f9d633d7700  5 queue_op 0x79fa0c0 seq 2623 osr(3.3a 0x276cce0) 518380 bytes   (queue has 2 ops and 1036760 bytes)
2014-06-12 14:46:52.069037 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] eval_repop repgather(0x2a37aa80 173'2960 rep_tid=236351 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.069053 7f9d633d7700 10 osd.2 173 dequeue_op 0x20e86700 finish
2014-06-12 14:46:52.069057 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.069059 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.069066 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b70e0 (2 active)
2014-06-12 14:46:52.069186 7f9d83173700 10 _generic_write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 1388544~516096 = 0
2014-06-12 14:46:52.069214 7f9d83173700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.069227 7f9d83173700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.069231 7f9d83173700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.069236 7f9d83173700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.069379 7f9d83173700 20 set_header setting 16858 cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 parent seq 0
2014-06-12 14:46:52.069425 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.069436 7f9d83173700 20 set_header setting 1980 cid meta oid 5168262c/pglog_3.3a/0//-1 parent seq 0
2014-06-12 14:46:52.069444 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:52.072321 7f9d44101700  5 -- op tracker -- , seq: 748123, time: 2014-06-12 14:46:52.072271, event: header_read, op: osd_sub_op_reply(client.4317.0:143999 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.072331 7f9d44101700  5 -- op tracker -- , seq: 748123, time: 2014-06-12 14:46:52.072273, event: throttled, op: osd_sub_op_reply(client.4317.0:143999 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.072338 7f9d44101700  5 -- op tracker -- , seq: 748123, time: 2014-06-12 14:46:52.072287, event: all_read, op: osd_sub_op_reply(client.4317.0:143999 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.072345 7f9d44101700  5 -- op tracker -- , seq: 748123, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:143999 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.072350 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:143999 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:52.072355 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:52.072363 7f9d44101700 15 osd.2 173 enqueue_op 0x329ea400 prio 196 cost 0 latency 0.000092 osd_sub_op_reply(client.4317.0:143999 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:52.072424 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.072449 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329ea400 prio 196 cost 0 latency 0.000178 osd_sub_op_reply(client.4317.0:143999 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 pg pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean]
2014-06-12 14:46:52.072486 7f9d61bd4700  5 -- op tracker -- , seq: 748123, time: 2014-06-12 14:46:52.072466, event: reached_pg, op: osd_sub_op_reply(client.4317.0:143999 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.072500 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] handle_message: 0x329ea400
2014-06-12 14:46:52.072515 7f9d61bd4700  5 -- op tracker -- , seq: 748123, time: 2014-06-12 14:46:52.072511, event: started, op: osd_sub_op_reply(client.4317.0:143999 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.072521 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] sub_op_modify_reply: tid 236350 op  ack_type 4 from 21
2014-06-12 14:46:52.072544 7f9d61bd4700  5 -- op tracker -- , seq: 748121, time: 2014-06-12 14:46:52.072538, event: sub_op_commit_rec, op: osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.072552 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329ea400 finish
2014-06-12 14:46:52.072558 7f9d61bd4700  5 -- op tracker -- , seq: 748123, time: 2014-06-12 14:46:52.072555, event: done, op: osd_sub_op_reply(client.4317.0:143999 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.072571 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.072574 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.073166 7f9d83173700 10 _do_op 0x28e6ea0 seq 2622 r = 0, finisher 0x13d13180 0x8821600
2014-06-12 14:46:52.073192 7f9d83173700 10 _finish_op 0x28e6ea0 seq 2622 osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.073207 7f9d7a161700  5 _do_op 0x79fa0c0 seq 2623 osr(3.3a 0x276cce0)/0x276cce0 start
2014-06-12 14:46:52.073216 7f9d7a161700 10 _do_transaction on 0x2a12fa40
2014-06-12 14:46:52.073211 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b70e0 (1 active)
2014-06-12 14:46:52.073222 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.073198 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] op_commit: 236350
2014-06-12 14:46:52.073228 7f9d7a161700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.073242 7f9d7715b700  5 -- op tracker -- , seq: 748121, time: 2014-06-12 14:46:52.073230, event: op_commit, op: osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.073279 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2958 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] repop_all_committed: repop tid 236350 all committed 
2014-06-12 14:46:52.073302 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2957 active+clean] eval_repop repgather(0x15e57080 173'2959 rep_tid=236350 committed?=1 applied?=0 lock=0 op=osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.073317 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2957 active+clean] log_op_stats osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 inb 516373 outb 0 rlat 0.000000 lat 0.008128
2014-06-12 14:46:52.073332 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2957 active+clean] publish_stats_to_osd 173:3841
2014-06-12 14:46:52.073342 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2957 active+clean]  sending commit on repgather(0x15e57080 173'2959 rep_tid=236350 committed?=1 applied?=0 lock=0 op=osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x10097900
2014-06-12 14:46:52.073354 7f9d7a161700 10 lookup_strip_header done  cid meta oid 5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.073372 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.073377 7f9d7715b700  5 -- op tracker -- , seq: 748121, time: 2014-06-12 14:46:52.073372, event: commit_sent, op: osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.073396 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2957 active+clean] op_applied: 236350
2014-06-12 14:46:52.073418 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.073420 7f9d7795c700  5 -- op tracker -- , seq: 748121, time: 2014-06-12 14:46:52.073413, event: op_applied, op: osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.073427 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2957 active+clean] op_applied on primary on version 173'2959
2014-06-12 14:46:52.073438 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2957 active+clean] repop_all_applied: repop tid 236350 all applied 
2014-06-12 14:46:52.073446 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2957 active+clean] eval_repop repgather(0x15e57080 173'2959 rep_tid=236350 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.073461 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2957 active+clean] log_op_stats osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 inb 516373 outb 0 rlat 0.000000 lat 0.008273
2014-06-12 14:46:52.073475 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2957 active+clean] publish_stats_to_osd 173:3842
2014-06-12 14:46:52.073489 7f9d7a161700 15 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.073485 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2957 active+clean]  requeue_ops 
2014-06-12 14:46:52.073493 7f9d7a161700 10 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1 r = 0
2014-06-12 14:46:52.073498 7f9d7a161700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.073494 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean]  removing repgather(0x15e57080 173'2959 rep_tid=236350 committed?=1 applied?=1 op=osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.073510 7f9d7a161700 15 _write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 1904640~516096
2014-06-12 14:46:52.073508 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean]    q front is repgather(0x15e57080 173'2959 rep_tid=236350 committed?=1 applied?=1 op=osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.073521 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] remove_repop repgather(0x15e57080 173'2959 rep_tid=236350 committed?=1 applied?=1 op=osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.073533 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean]  requeue_ops 
2014-06-12 14:46:52.073551 7f9d7795c700  5 -- op tracker -- , seq: 748121, time: 2014-06-12 14:46:52.073546, event: done, op: osd_op(client.4317.0:143999 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1388544~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.073602 7f9d7a161700 10 lookup_strip_header done  cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.073742 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:52.075073 7f9d7a161700 10 _generic_write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 1904640~516096 = 0
2014-06-12 14:46:52.075118 7f9d7a161700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.075129 7f9d7a161700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.075133 7f9d7a161700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.075138 7f9d7a161700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.075274 7f9d7a161700 20 set_header setting 16858 cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 parent seq 0
2014-06-12 14:46:52.075329 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.075340 7f9d7a161700 20 set_header setting 1980 cid meta oid 5168262c/pglog_3.3a/0//-1 parent seq 0
2014-06-12 14:46:52.075347 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:52.076998 7f9d44424700  5 -- op tracker -- , seq: 748124, time: 2014-06-12 14:46:52.075790, event: header_read, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.077016 7f9d44424700  5 -- op tracker -- , seq: 748124, time: 2014-06-12 14:46:52.075795, event: throttled, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.077025 7f9d44424700  5 -- op tracker -- , seq: 748124, time: 2014-06-12 14:46:52.076931, event: all_read, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.077033 7f9d44424700  5 -- op tracker -- , seq: 748124, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.077042 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:52.077071 7f9d44424700 15 osd.2 173 enqueue_op 0x20e86a00 prio 63 cost 516096 latency 0.001281 osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.077318 7f9d44101700  5 -- op tracker -- , seq: 748125, time: 2014-06-12 14:46:52.077279, event: header_read, op: osd_sub_op_reply(client.4317.0:144000 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.077328 7f9d44101700  5 -- op tracker -- , seq: 748125, time: 2014-06-12 14:46:52.077280, event: throttled, op: osd_sub_op_reply(client.4317.0:144000 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.077335 7f9d44101700  5 -- op tracker -- , seq: 748125, time: 2014-06-12 14:46:52.077294, event: all_read, op: osd_sub_op_reply(client.4317.0:144000 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.077341 7f9d44101700  5 -- op tracker -- , seq: 748125, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:144000 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.077345 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:144000 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:52.077351 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:52.077358 7f9d44101700 15 osd.2 173 enqueue_op 0x329eac00 prio 196 cost 0 latency 0.000079 osd_sub_op_reply(client.4317.0:144000 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:52.077387 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.077394 7f9d673df700 10 osd.2 173 dequeue_op 0x329eac00 prio 196 cost 0 latency 0.000115 osd_sub_op_reply(client.4317.0:144000 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 pg pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean]
2014-06-12 14:46:52.077418 7f9d673df700  5 -- op tracker -- , seq: 748125, time: 2014-06-12 14:46:52.077415, event: reached_pg, op: osd_sub_op_reply(client.4317.0:144000 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.077426 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] handle_message: 0x329eac00
2014-06-12 14:46:52.077443 7f9d673df700  5 -- op tracker -- , seq: 748125, time: 2014-06-12 14:46:52.077440, event: started, op: osd_sub_op_reply(client.4317.0:144000 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.077448 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] sub_op_modify_reply: tid 236351 op  ack_type 4 from 21
2014-06-12 14:46:52.077469 7f9d673df700  5 -- op tracker -- , seq: 748122, time: 2014-06-12 14:46:52.077462, event: sub_op_commit_rec, op: osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.077478 7f9d673df700 10 osd.2 173 dequeue_op 0x329eac00 finish
2014-06-12 14:46:52.077484 7f9d673df700  5 -- op tracker -- , seq: 748125, time: 2014-06-12 14:46:52.077481, event: done, op: osd_sub_op_reply(client.4317.0:144000 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.077497 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.077503 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.077506 7f9d673df700 10 osd.2 173 dequeue_op 0x20e86a00 prio 63 cost 516096 latency 0.001716 osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean]
2014-06-12 14:46:52.077523 7f9d673df700  5 -- op tracker -- , seq: 748124, time: 2014-06-12 14:46:52.077519, event: reached_pg, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.077529 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:52.077542 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] handle_message: 0x20e86a00
2014-06-12 14:46:52.077551 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] do_op osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:52.077575 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] get_object_context: found obc in cache: 0x1703d340
2014-06-12 14:46:52.077586 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] get_object_context: 0x1703d340 e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 rwstate(write n=1 w=0) oi: e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2960 client.4317.0:144000 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2960) ssc: 0x5d5a700 snapset: 0=[]:[]+head
2014-06-12 14:46:52.077602 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] find_object_context e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 @head oi=e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2960 client.4317.0:144000 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2960)
2014-06-12 14:46:52.077644 7f9d673df700  5 -- op tracker -- , seq: 748124, time: 2014-06-12 14:46:52.077639, event: started, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.077652 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] execute_ctx 0xa04f200
2014-06-12 14:46:52.077667 7f9d673df700  5 -- op tracker -- , seq: 748124, time: 2014-06-12 14:46:52.077663, event: started, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.077677 7f9d7a161700 10 _do_op 0x79fa0c0 seq 2623 r = 0, finisher 0x5af7bc0 0x37e4d00
2014-06-12 14:46:52.077675 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] do_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] ov 173'2960 av 173'2961 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:52.077689 7f9d7a161700 10 _finish_op 0x79fa0c0 seq 2623 osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.077688 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] do_osd_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096]
2014-06-12 14:46:52.077698 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:52.077698 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b70e0 (0 active)
2014-06-12 14:46:52.077709 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.077711 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] do_osd_op  write 2420736~516096
2014-06-12 14:46:52.077718 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.077726 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:52.077734 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 snapset=0x5d5a738  snapc=0=[]
2014-06-12 14:46:52.077742 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:52.077748 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] finish_ctx e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 0xa04f200 op modify  
2014-06-12 14:46:52.077756 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean]  set mtime to 2014-06-12 08:45:54.455170
2014-06-12 14:46:52.077770 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean]  final snapset 0=[]:[]+head in e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.077783 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean]  zeroing write result code 0
2014-06-12 14:46:52.077793 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] new_repop rep_tid 236352 on osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.077843 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] issue_repop rep_tid 236352 o e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.077950 7f9d673df700  5 -- op tracker -- , seq: 748124, time: 2014-06-12 14:46:52.077945, event: waiting for subops from 21, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.077968 7f9d673df700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:52.077981 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2960 (0'0,173'2960] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] append_log log((0'0,173'2960], crt=0'0) [173'2961 (173'2960) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:144001 2014-06-12 08:45:54.455170]
2014-06-12 14:46:52.077999 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] add_log_entry 173'2961 (173'2960) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:144001 2014-06-12 08:45:54.455170
2014-06-12 14:46:52.078013 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] append_log  adding 1 keys
2014-06-12 14:46:52.078035 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2961, trimmed: 
2014-06-12 14:46:52.078052 7f9d673df700  5 queue_transactions existing osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.078056 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.078057 7f9d673df700  5 queue_op 0x87dc600 seq 2624 osr(3.3a 0x276cce0) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:46:52.078066 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b70e0 (1 active)
2014-06-12 14:46:52.078071 7f9d7f16b700  5 _do_op 0x87dc600 seq 2624 osr(3.3a 0x276cce0)/0x276cce0 start
2014-06-12 14:46:52.078075 7f9d7f16b700 10 _do_transaction on 0x7415800
2014-06-12 14:46:52.078062 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] eval_repop repgather(0x2fe220c0 173'2961 rep_tid=236352 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.078079 7f9d673df700 10 osd.2 173 dequeue_op 0x20e86a00 finish
2014-06-12 14:46:52.078081 7f9d7f16b700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.078085 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.078088 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.078116 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] op_applied: 236351
2014-06-12 14:46:52.078140 7f9d7795c700  5 -- op tracker -- , seq: 748122, time: 2014-06-12 14:46:52.078133, event: op_applied, op: osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.078147 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] op_applied on primary on version 173'2960
2014-06-12 14:46:52.078158 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] repop_all_applied: repop tid 236351 all applied 
2014-06-12 14:46:52.078167 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] eval_repop repgather(0x2a37aa80 173'2960 rep_tid=236351 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.078181 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.078181 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean]  sending ack on repgather(0x2a37aa80 173'2960 rep_tid=236351 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0xbc09400
2014-06-12 14:46:52.078203 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.078242 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.078259 7f9d7f16b700 15 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.078263 7f9d7f16b700 10 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1 r = 0
2014-06-12 14:46:52.078268 7f9d7f16b700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.078373 7f9d7f16b700 15 _write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 2420736~516096
2014-06-12 14:46:52.078510 7f9d7f16b700 10 lookup_strip_header done  cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.078540 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] op_commit: 236351
2014-06-12 14:46:52.078561 7f9d7715b700  5 -- op tracker -- , seq: 748122, time: 2014-06-12 14:46:52.078556, event: op_commit, op: osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.078574 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2959 crt=0'0 lcod 173'2958 mlcod 173'2958 active+clean] repop_all_committed: repop tid 236351 all committed 
2014-06-12 14:46:52.078583 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2958 active+clean] eval_repop repgather(0x2a37aa80 173'2960 rep_tid=236351 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.078598 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2958 active+clean] log_op_stats osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 inb 516373 outb 0 rlat 0.011453 lat 0.011842
2014-06-12 14:46:52.078614 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2958 active+clean] publish_stats_to_osd 173:3843
2014-06-12 14:46:52.078625 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2958 active+clean]  sending commit on repgather(0x2a37aa80 173'2960 rep_tid=236351 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x28acfb80
2014-06-12 14:46:52.078642 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:52.078661 7f9d7715b700  5 -- op tracker -- , seq: 748122, time: 2014-06-12 14:46:52.078655, event: commit_sent, op: osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.078670 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2958 active+clean]  requeue_ops 
2014-06-12 14:46:52.078682 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean]  removing repgather(0x2a37aa80 173'2960 rep_tid=236351 committed?=1 applied?=1 op=osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.078698 7f9d7715b700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean]    q front is repgather(0x2a37aa80 173'2960 rep_tid=236351 committed?=1 applied?=1 op=osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.078714 7f9d7715b700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] remove_repop repgather(0x2a37aa80 173'2960 rep_tid=236351 committed?=1 applied?=1 op=osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.078728 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean]  requeue_ops 
2014-06-12 14:46:52.078748 7f9d7715b700  5 -- op tracker -- , seq: 748122, time: 2014-06-12 14:46:52.078743, event: done, op: osd_op(client.4317.0:144000 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.080120 7f9d7f16b700 10 _generic_write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 2420736~516096 = 0
2014-06-12 14:46:52.080165 7f9d7f16b700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.080176 7f9d7f16b700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.080180 7f9d7f16b700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.080185 7f9d7f16b700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.080321 7f9d7f16b700 20 set_header setting 16858 cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 parent seq 0
2014-06-12 14:46:52.080377 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.080385 7f9d7f16b700 20 set_header setting 1980 cid meta oid 5168262c/pglog_3.3a/0//-1 parent seq 0
2014-06-12 14:46:52.080391 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:52.082293 7f9d44424700  5 -- op tracker -- , seq: 748126, time: 2014-06-12 14:46:52.081148, event: header_read, op: osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.082312 7f9d44424700  5 -- op tracker -- , seq: 748126, time: 2014-06-12 14:46:52.081152, event: throttled, op: osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.082322 7f9d44424700  5 -- op tracker -- , seq: 748126, time: 2014-06-12 14:46:52.082226, event: all_read, op: osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.082331 7f9d44424700  5 -- op tracker -- , seq: 748126, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.082340 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:52.082368 7f9d44424700 15 osd.2 173 enqueue_op 0x20e86600 prio 63 cost 516096 latency 0.001220 osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.082401 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.082410 7f9d663dd700 10 osd.2 173 dequeue_op 0x20e86600 prio 63 cost 516096 latency 0.001261 osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean]
2014-06-12 14:46:52.082430 7f9d663dd700  5 -- op tracker -- , seq: 748126, time: 2014-06-12 14:46:52.082427, event: reached_pg, op: osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.082437 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:52.082457 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] handle_message: 0x20e86600
2014-06-12 14:46:52.082464 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] do_op osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:52.082485 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] get_object_context: found obc in cache: 0x1703d340
2014-06-12 14:46:52.082492 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] get_object_context: 0x1703d340 e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 rwstate(write n=1 w=0) oi: e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2961 client.4317.0:144001 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2961) ssc: 0x5d5a700 snapset: 0=[]:[]+head
2014-06-12 14:46:52.082505 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] find_object_context e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 @head oi=e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2961 client.4317.0:144001 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2961)
2014-06-12 14:46:52.082526 7f9d663dd700  5 -- op tracker -- , seq: 748126, time: 2014-06-12 14:46:52.082523, event: started, op: osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.082536 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] execute_ctx 0x2d2a6c00
2014-06-12 14:46:52.082547 7f9d663dd700  5 -- op tracker -- , seq: 748126, time: 2014-06-12 14:46:52.082544, event: started, op: osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.082551 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] do_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] ov 173'2961 av 173'2962 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:52.082560 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] do_osd_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096]
2014-06-12 14:46:52.082567 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:52.082576 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] do_osd_op  write 2936832~516096
2014-06-12 14:46:52.082586 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:52.082592 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 snapset=0x5d5a738  snapc=0=[]
2014-06-12 14:46:52.082599 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:52.082605 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] finish_ctx e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 0x2d2a6c00 op modify  
2014-06-12 14:46:52.082612 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean]  set mtime to 2014-06-12 08:45:54.460519
2014-06-12 14:46:52.082626 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean]  final snapset 0=[]:[]+head in e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.082640 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean]  zeroing write result code 0
2014-06-12 14:46:52.082648 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] new_repop rep_tid 236353 on osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.082659 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] issue_repop rep_tid 236353 o e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.082675 7f9d663dd700  5 -- op tracker -- , seq: 748126, time: 2014-06-12 14:46:52.082672, event: waiting for subops from 21, op: osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.082692 7f9d663dd700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:52.082709 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2961 (0'0,173'2961] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] append_log log((0'0,173'2961], crt=0'0) [173'2962 (173'2961) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:144002 2014-06-12 08:45:54.460519]
2014-06-12 14:46:52.082733 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] add_log_entry 173'2962 (173'2961) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:144002 2014-06-12 08:45:54.460519
2014-06-12 14:46:52.082756 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] append_log  adding 1 keys
2014-06-12 14:46:52.082786 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2962, trimmed: 
2014-06-12 14:46:52.082814 7f9d663dd700  5 queue_transactions existing osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.082819 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.082820 7f9d663dd700  5 queue_op 0x568eb40 seq 2625 osr(3.3a 0x276cce0) 518380 bytes   (queue has 2 ops and 1036760 bytes)
2014-06-12 14:46:52.082827 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] eval_repop repgather(0x5390540 173'2962 rep_tid=236353 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.082845 7f9d663dd700 10 osd.2 173 dequeue_op 0x20e86600 finish
2014-06-12 14:46:52.082848 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.082850 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.082885 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b70e0 (2 active)
2014-06-12 14:46:52.083568 7f9d7f16b700 10 _do_op 0x87dc600 seq 2624 r = 0, finisher 0x65e4660 0x465ac80
2014-06-12 14:46:52.083584 7f9d7f16b700 10 _finish_op 0x87dc600 seq 2624 osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.083594 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b70e0 (1 active)
2014-06-12 14:46:52.083601 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.083611 7f9d7e169700  5 _do_op 0x568eb40 seq 2625 osr(3.3a 0x276cce0)/0x276cce0 start
2014-06-12 14:46:52.083615 7f9d7e169700 10 _do_transaction on 0x28df3740
2014-06-12 14:46:52.083602 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] op_commit: 236352
2014-06-12 14:46:52.083620 7f9d7e169700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.083627 7f9d7715b700  5 -- op tracker -- , seq: 748124, time: 2014-06-12 14:46:52.083619, event: op_commit, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.083640 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] op_applied: 236352
2014-06-12 14:46:52.083655 7f9d7795c700  5 -- op tracker -- , seq: 748124, time: 2014-06-12 14:46:52.083651, event: op_applied, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.083661 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] op_applied on primary on version 173'2961
2014-06-12 14:46:52.083749 7f9d7e169700 10 lookup_strip_header done  cid meta oid 5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.083765 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.083795 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.083900 7f9d7e169700 15 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.083903 7f9d7e169700 10 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1 r = 0
2014-06-12 14:46:52.083907 7f9d7e169700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.083920 7f9d7e169700 15 _write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 2936832~516096
2014-06-12 14:46:52.084010 7f9d7e169700 10 lookup_strip_header done  cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.084163 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:52.085477 7f9d7e169700 10 _generic_write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 2936832~516096 = 0
2014-06-12 14:46:52.085509 7f9d7e169700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.085522 7f9d7e169700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.085529 7f9d7e169700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.085534 7f9d7e169700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.085695 7f9d7e169700 20 set_header setting 16858 cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 parent seq 0
2014-06-12 14:46:52.085742 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.085751 7f9d7e169700 20 set_header setting 1980 cid meta oid 5168262c/pglog_3.3a/0//-1 parent seq 0
2014-06-12 14:46:52.085757 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:52.086495 7f9d44101700  5 -- op tracker -- , seq: 748127, time: 2014-06-12 14:46:52.086454, event: header_read, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.086507 7f9d44101700  5 -- op tracker -- , seq: 748127, time: 2014-06-12 14:46:52.086456, event: throttled, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.086515 7f9d44101700  5 -- op tracker -- , seq: 748127, time: 2014-06-12 14:46:52.086467, event: all_read, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.086533 7f9d44101700  5 -- op tracker -- , seq: 748127, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.086538 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:52.086542 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:52.086549 7f9d44101700 15 osd.2 173 enqueue_op 0xe20a100 prio 196 cost 0 latency 0.000095 osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0) v2
2014-06-12 14:46:52.086577 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.086588 7f9d65bdc700 10 osd.2 173 dequeue_op 0xe20a100 prio 196 cost 0 latency 0.000133 osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0) v2 pg pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean]
2014-06-12 14:46:52.086612 7f9d65bdc700  5 -- op tracker -- , seq: 748127, time: 2014-06-12 14:46:52.086608, event: reached_pg, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.086623 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] handle_message: 0xe20a100
2014-06-12 14:46:52.086641 7f9d65bdc700  5 -- op tracker -- , seq: 748127, time: 2014-06-12 14:46:52.086637, event: started, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.086649 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] sub_op_modify_reply: tid 236352 op  ack_type 1 from 21
2014-06-12 14:46:52.086671 7f9d65bdc700  5 -- op tracker -- , seq: 748124, time: 2014-06-12 14:46:52.086663, event: sub_op_applied_rec, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.086679 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] repop_all_applied: repop tid 236352 all applied 
2014-06-12 14:46:52.086692 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] eval_repop repgather(0x2fe220c0 173'2961 rep_tid=236352 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.086709 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean]  sending ack on repgather(0x2fe220c0 173'2961 rep_tid=236352 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x21ed8a00
2014-06-12 14:46:52.086744 7f9d65bdc700 10 osd.2 173 dequeue_op 0xe20a100 finish
2014-06-12 14:46:52.086753 7f9d65bdc700  5 -- op tracker -- , seq: 748127, time: 2014-06-12 14:46:52.086750, event: done, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ack, result = 0)
2014-06-12 14:46:52.086768 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.086773 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.087901 7f9d7e169700 10 _do_op 0x568eb40 seq 2625 r = 0, finisher 0x6571060 0x152d8740
2014-06-12 14:46:52.087916 7f9d7e169700 10 _finish_op 0x568eb40 seq 2625 osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.087934 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b70e0 (0 active)
2014-06-12 14:46:52.087922 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] op_commit: 236353
2014-06-12 14:46:52.087938 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.087948 7f9d7715b700  5 -- op tracker -- , seq: 748126, time: 2014-06-12 14:46:52.087939, event: op_commit, op: osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.087970 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] op_applied: 236353
2014-06-12 14:46:52.087986 7f9d7795c700  5 -- op tracker -- , seq: 748126, time: 2014-06-12 14:46:52.087982, event: op_applied, op: osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.087996 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] op_applied on primary on version 173'2962
2014-06-12 14:46:52.089451 7f9d44101700  5 -- op tracker -- , seq: 748128, time: 2014-06-12 14:46:52.089416, event: header_read, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.089469 7f9d44101700  5 -- op tracker -- , seq: 748128, time: 2014-06-12 14:46:52.089417, event: throttled, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.089480 7f9d44101700  5 -- op tracker -- , seq: 748128, time: 2014-06-12 14:46:52.089426, event: all_read, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.089489 7f9d44101700  5 -- op tracker -- , seq: 748128, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.089494 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:52.089498 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:52.089505 7f9d44101700 15 osd.2 173 enqueue_op 0xe20b800 prio 196 cost 0 latency 0.000088 osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:52.089527 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.089533 7f9d62bd6700 10 osd.2 173 dequeue_op 0xe20b800 prio 196 cost 0 latency 0.000117 osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 pg pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean]
2014-06-12 14:46:52.089554 7f9d62bd6700  5 -- op tracker -- , seq: 748128, time: 2014-06-12 14:46:52.089550, event: reached_pg, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.089563 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] handle_message: 0xe20b800
2014-06-12 14:46:52.089582 7f9d62bd6700  5 -- op tracker -- , seq: 748128, time: 2014-06-12 14:46:52.089578, event: started, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.089589 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] sub_op_modify_reply: tid 236352 op  ack_type 4 from 21
2014-06-12 14:46:52.089609 7f9d62bd6700  5 -- op tracker -- , seq: 748124, time: 2014-06-12 14:46:52.089603, event: sub_op_commit_rec, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.089623 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2960 crt=0'0 lcod 173'2959 mlcod 173'2959 active+clean] repop_all_committed: repop tid 236352 all committed 
2014-06-12 14:46:52.089634 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2959 active+clean] eval_repop repgather(0x2fe220c0 173'2961 rep_tid=236352 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.089651 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2959 active+clean] log_op_stats osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 inb 516373 outb 0 rlat 0.010952 lat 0.013860
2014-06-12 14:46:52.089667 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2959 active+clean] publish_stats_to_osd 173:3844
2014-06-12 14:46:52.089679 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2959 active+clean]  sending commit on repgather(0x2fe220c0 173'2961 rep_tid=236352 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x1b774500
2014-06-12 14:46:52.089715 7f9d62bd6700  5 -- op tracker -- , seq: 748124, time: 2014-06-12 14:46:52.089710, event: commit_sent, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.089722 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2959 active+clean]  requeue_ops 
2014-06-12 14:46:52.089730 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2960 active+clean]  removing repgather(0x2fe220c0 173'2961 rep_tid=236352 committed?=1 applied?=1 op=osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.089743 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2960 active+clean]    q front is repgather(0x2fe220c0 173'2961 rep_tid=236352 committed?=1 applied?=1 op=osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.089755 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2960 active+clean] remove_repop repgather(0x2fe220c0 173'2961 rep_tid=236352 committed?=1 applied?=1 op=osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.089767 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2960 active+clean]  requeue_ops 
2014-06-12 14:46:52.089783 7f9d62bd6700  5 -- op tracker -- , seq: 748124, time: 2014-06-12 14:46:52.089779, event: done, op: osd_op(client.4317.0:144001 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2420736~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.089801 7f9d62bd6700 10 osd.2 173 dequeue_op 0xe20b800 finish
2014-06-12 14:46:52.089807 7f9d62bd6700  5 -- op tracker -- , seq: 748128, time: 2014-06-12 14:46:52.089804, event: done, op: osd_sub_op_reply(client.4317.0:144001 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.089815 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.089817 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.092071 7f9d44101700  5 -- op tracker -- , seq: 748129, time: 2014-06-12 14:46:52.092034, event: header_read, op: osd_sub_op_reply(client.4317.0:144002 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.092082 7f9d44101700  5 -- op tracker -- , seq: 748129, time: 2014-06-12 14:46:52.092036, event: throttled, op: osd_sub_op_reply(client.4317.0:144002 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.092089 7f9d44101700  5 -- op tracker -- , seq: 748129, time: 2014-06-12 14:46:52.092044, event: all_read, op: osd_sub_op_reply(client.4317.0:144002 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.092098 7f9d44101700  5 -- op tracker -- , seq: 748129, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:144002 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.092102 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:144002 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:52.092107 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:52.092114 7f9d44101700 15 osd.2 173 enqueue_op 0xe20aa00 prio 196 cost 0 latency 0.000080 osd_sub_op_reply(client.4317.0:144002 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:52.092242 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.092252 7f9d683e1700 10 osd.2 173 dequeue_op 0xe20aa00 prio 196 cost 0 latency 0.000217 osd_sub_op_reply(client.4317.0:144002 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 pg pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2960 active+clean]
2014-06-12 14:46:52.092273 7f9d683e1700  5 -- op tracker -- , seq: 748129, time: 2014-06-12 14:46:52.092269, event: reached_pg, op: osd_sub_op_reply(client.4317.0:144002 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.092290 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2960 active+clean] handle_message: 0xe20aa00
2014-06-12 14:46:52.092309 7f9d683e1700  5 -- op tracker -- , seq: 748129, time: 2014-06-12 14:46:52.092305, event: started, op: osd_sub_op_reply(client.4317.0:144002 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.092315 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2960 active+clean] sub_op_modify_reply: tid 236353 op  ack_type 4 from 21
2014-06-12 14:46:52.092335 7f9d683e1700  5 -- op tracker -- , seq: 748126, time: 2014-06-12 14:46:52.092330, event: sub_op_commit_rec, op: osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.092342 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2960 active+clean] repop_all_applied: repop tid 236353 all applied 
2014-06-12 14:46:52.092353 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2960 active+clean] eval_repop repgather(0x5390540 173'2962 rep_tid=236353 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.092368 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2960 active+clean]  sending ack on repgather(0x5390540 173'2962 rep_tid=236353 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x75cc000
2014-06-12 14:46:52.092404 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2961 crt=0'0 lcod 173'2960 mlcod 173'2960 active+clean] repop_all_committed: repop tid 236353 all committed 
2014-06-12 14:46:52.092415 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2960 active+clean] eval_repop repgather(0x5390540 173'2962 rep_tid=236353 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.092433 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2960 active+clean] log_op_stats osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 inb 516373 outb 0 rlat 0.011254 lat 0.011284
2014-06-12 14:46:52.092446 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2960 active+clean] publish_stats_to_osd 173:3845
2014-06-12 14:46:52.092458 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2960 active+clean]  sending commit on repgather(0x5390540 173'2962 rep_tid=236353 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x5ffec80
2014-06-12 14:46:52.092495 7f9d683e1700  5 -- op tracker -- , seq: 748126, time: 2014-06-12 14:46:52.092489, event: commit_sent, op: osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.092504 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2960 active+clean]  requeue_ops 
2014-06-12 14:46:52.092515 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean]  removing repgather(0x5390540 173'2962 rep_tid=236353 committed?=1 applied?=1 op=osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.092533 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean]    q front is repgather(0x5390540 173'2962 rep_tid=236353 committed?=1 applied?=1 op=osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.092551 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] remove_repop repgather(0x5390540 173'2962 rep_tid=236353 committed?=1 applied?=1 op=osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.092566 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean]  requeue_ops 
2014-06-12 14:46:52.092583 7f9d683e1700  5 -- op tracker -- , seq: 748126, time: 2014-06-12 14:46:52.092580, event: done, op: osd_op(client.4317.0:144002 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.092594 7f9d683e1700 10 osd.2 173 dequeue_op 0xe20aa00 finish
2014-06-12 14:46:52.092598 7f9d683e1700  5 -- op tracker -- , seq: 748129, time: 2014-06-12 14:46:52.092596, event: done, op: osd_sub_op_reply(client.4317.0:144002 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.092604 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.092606 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.093294 7f9d44424700  5 -- op tracker -- , seq: 748130, time: 2014-06-12 14:46:52.092139, event: header_read, op: osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.093307 7f9d44424700  5 -- op tracker -- , seq: 748130, time: 2014-06-12 14:46:52.092141, event: throttled, op: osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.093318 7f9d44424700  5 -- op tracker -- , seq: 748130, time: 2014-06-12 14:46:52.093265, event: all_read, op: osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.093329 7f9d44424700  5 -- op tracker -- , seq: 748130, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.093338 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:52.093368 7f9d44424700 15 osd.2 173 enqueue_op 0x2afce00 prio 63 cost 516096 latency 0.001228 osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.093393 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.093400 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2afce00 prio 63 cost 516096 latency 0.001261 osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean]
2014-06-12 14:46:52.093433 7f9d6b3e7700  5 -- op tracker -- , seq: 748130, time: 2014-06-12 14:46:52.093419, event: reached_pg, op: osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.093444 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:52.093461 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] handle_message: 0x2afce00
2014-06-12 14:46:52.093472 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] do_op osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:52.093496 7f9d6b3e7700 15 getattr 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 '_'
2014-06-12 14:46:52.093653 7f9d6b3e7700 10 getattr 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 '_' = 0
2014-06-12 14:46:52.093670 7f9d6b3e7700 15 getattr 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 'snapset'
2014-06-12 14:46:52.093732 7f9d6b3e7700 10 getattr 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 'snapset' = 0
2014-06-12 14:46:52.093742 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] populate_obc_watchers e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.093756 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.093768 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] get_object_context: creating obc from disk: 0x142642c0
2014-06-12 14:46:52.093785 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] get_object_context: 0x142642c0 e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 rwstate(none n=0 w=0) oi: e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2962 client.4317.0:144002 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2962) ssc: 0x1e15e460 snapset: 0=[]:[]+head
2014-06-12 14:46:52.093805 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] find_object_context e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 @head oi=e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2962 client.4317.0:144002 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2962)
2014-06-12 14:46:52.093838 7f9d6b3e7700  5 -- op tracker -- , seq: 748130, time: 2014-06-12 14:46:52.093832, event: started, op: osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.093847 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] execute_ctx 0x2e411e00
2014-06-12 14:46:52.093866 7f9d6b3e7700  5 -- op tracker -- , seq: 748130, time: 2014-06-12 14:46:52.093861, event: started, op: osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.093872 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] do_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] ov 173'2962 av 173'2963 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:52.093888 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] do_osd_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096]
2014-06-12 14:46:52.093903 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:52.093920 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] do_osd_op  write 3452928~516096
2014-06-12 14:46:52.093943 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:52.093955 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 snapset=0x1e15e498  snapc=0=[]
2014-06-12 14:46:52.093970 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:52.093983 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] finish_ctx e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 0x2e411e00 op modify  
2014-06-12 14:46:52.093997 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean]  set mtime to 2014-06-12 08:45:54.471510
2014-06-12 14:46:52.094019 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean]  final snapset 0=[]:[]+head in e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.094039 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean]  zeroing write result code 0
2014-06-12 14:46:52.094051 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] new_repop rep_tid 236354 on osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.094070 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] issue_repop rep_tid 236354 o e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.094096 7f9d6b3e7700  5 -- op tracker -- , seq: 748130, time: 2014-06-12 14:46:52.094091, event: waiting for subops from 21, op: osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.094120 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:52.094143 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2962 (0'0,173'2962] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] append_log log((0'0,173'2962], crt=0'0) [173'2963 (173'2962) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:144003 2014-06-12 08:45:54.471510]
2014-06-12 14:46:52.094161 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] add_log_entry 173'2963 (173'2962) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:144003 2014-06-12 08:45:54.471510
2014-06-12 14:46:52.094183 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] append_log  adding 1 keys
2014-06-12 14:46:52.094217 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2963, trimmed: 
2014-06-12 14:46:52.094240 7f9d6b3e7700  5 queue_transactions existing osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.094246 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.094248 7f9d6b3e7700  5 queue_op 0x171029c0 seq 2626 osr(3.3a 0x276cce0) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:46:52.094255 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] eval_repop repgather(0x5323b00 173'2963 rep_tid=236354 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.094276 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2afce00 finish
2014-06-12 14:46:52.094281 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.094285 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.094569 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b70e0 (1 active)
2014-06-12 14:46:52.094578 7f9d7815d700  5 _do_op 0x171029c0 seq 2626 osr(3.3a 0x276cce0)/0x276cce0 start
2014-06-12 14:46:52.094582 7f9d7815d700 10 _do_transaction on 0x6590780
2014-06-12 14:46:52.094593 7f9d7815d700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.094683 7f9d7815d700 10 lookup_strip_header done  cid meta oid 5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.094706 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.094824 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.094846 7f9d7815d700 15 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.094854 7f9d7815d700 10 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1 r = 0
2014-06-12 14:46:52.094860 7f9d7815d700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.094879 7f9d7815d700 15 _write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 3452928~516096
2014-06-12 14:46:52.095014 7f9d7815d700 10 lookup_strip_header done  cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.095164 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:52.095625 7f9d44424700  5 -- op tracker -- , seq: 748131, time: 2014-06-12 14:46:52.095243, event: header_read, op: osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.095638 7f9d44424700  5 -- op tracker -- , seq: 748131, time: 2014-06-12 14:46:52.095246, event: throttled, op: osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.095649 7f9d44424700  5 -- op tracker -- , seq: 748131, time: 2014-06-12 14:46:52.095594, event: all_read, op: osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.095665 7f9d44424700  5 -- op tracker -- , seq: 748131, time: 0.000000, event: dispatched, op: osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.095674 7f9d44424700 20 osd.2 173 should_share_map client.4317 172.16.96.220:0/1244727 173
2014-06-12 14:46:52.095707 7f9d44424700 15 osd.2 173 enqueue_op 0x2afd100 prio 63 cost 225280 latency 0.000463 osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.095743 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.095753 7f9d693e3700 10 osd.2 173 dequeue_op 0x2afd100 prio 63 cost 225280 latency 0.000510 osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean]
2014-06-12 14:46:52.095785 7f9d693e3700  5 -- op tracker -- , seq: 748131, time: 2014-06-12 14:46:52.095780, event: reached_pg, op: osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.095796 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:52.095816 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] handle_message: 0x2afd100
2014-06-12 14:46:52.095830 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] do_op osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:52.095879 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] get_object_context: found obc in cache: 0x142642c0
2014-06-12 14:46:52.095892 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] get_object_context: 0x142642c0 e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 rwstate(write n=1 w=0) oi: e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2963 client.4317.0:144003 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2963) ssc: 0x1e15e460 snapset: 0=[]:[]+head
2014-06-12 14:46:52.095910 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] find_object_context e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 @head oi=e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2963 client.4317.0:144003 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2963)
2014-06-12 14:46:52.095951 7f9d693e3700  5 -- op tracker -- , seq: 748131, time: 2014-06-12 14:46:52.095932, event: started, op: osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.095962 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] execute_ctx 0x25b21000
2014-06-12 14:46:52.095982 7f9d693e3700  5 -- op tracker -- , seq: 748131, time: 2014-06-12 14:46:52.095976, event: started, op: osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.095990 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] do_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] ov 173'2963 av 173'2964 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:52.096006 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] do_osd_op e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280]
2014-06-12 14:46:52.096019 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:52.096037 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] do_osd_op  write 3969024~225280
2014-06-12 14:46:52.096053 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] do_osd_op_effects on session 0x53ad400
2014-06-12 14:46:52.096065 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 snapset=0x1e15e498  snapc=0=[]
2014-06-12 14:46:52.096077 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] make_writeable e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:52.096091 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] finish_ctx e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 0x25b21000 op modify  
2014-06-12 14:46:52.096111 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean]  set mtime to 2014-06-12 08:45:54.475174
2014-06-12 14:46:52.096137 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean]  final snapset 0=[]:[]+head in e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.096157 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean]  zeroing write result code 0
2014-06-12 14:46:52.096170 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] new_repop rep_tid 236355 on osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:52.096186 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] issue_repop rep_tid 236355 o e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.096213 7f9d693e3700  5 -- op tracker -- , seq: 748131, time: 2014-06-12 14:46:52.096208, event: waiting for subops from 21, op: osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.096238 7f9d693e3700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:52.096266 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2963 (0'0,173'2963] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] append_log log((0'0,173'2963], crt=0'0) [173'2964 (173'2963) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:144004 2014-06-12 08:45:54.475174]
2014-06-12 14:46:52.096294 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] add_log_entry 173'2964 (173'2963) modify   e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 by client.4317.0:144004 2014-06-12 08:45:54.475174
2014-06-12 14:46:52.096316 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] append_log  adding 1 keys
2014-06-12 14:46:52.096351 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2964, trimmed: 
2014-06-12 14:46:52.096375 7f9d693e3700  5 queue_transactions existing osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.096382 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.096385 7f9d693e3700  5 queue_op 0x69567e0 seq 2627 osr(3.3a 0x276cce0) 227564 bytes   (queue has 2 ops and 745944 bytes)
2014-06-12 14:46:52.096394 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] eval_repop repgather(0x3403b980 173'2964 rep_tid=236355 committed?=0 applied?=0 lock=0 op=osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.096411 7f9d693e3700 10 osd.2 173 dequeue_op 0x2afd100 finish
2014-06-12 14:46:52.096415 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.096402 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b70e0 (2 active)
2014-06-12 14:46:52.096418 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.097206 7f9d7815d700 10 _generic_write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 3452928~516096 = 0
2014-06-12 14:46:52.097260 7f9d7815d700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.097276 7f9d7815d700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.097285 7f9d7815d700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.097298 7f9d7815d700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.097483 7f9d7815d700 20 set_header setting 16858 cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 parent seq 0
2014-06-12 14:46:52.097566 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.097582 7f9d7815d700 20 set_header setting 1980 cid meta oid 5168262c/pglog_3.3a/0//-1 parent seq 0
2014-06-12 14:46:52.097593 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:52.100385 7f9d7815d700 10 _do_op 0x171029c0 seq 2626 r = 0, finisher 0x82c6300 0x1a3f0140
2014-06-12 14:46:52.100401 7f9d7815d700 10 _finish_op 0x171029c0 seq 2626 osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.100409 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b70e0 (1 active)
2014-06-12 14:46:52.100416 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.100415 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] op_applied: 236354
2014-06-12 14:46:52.100442 7f9d82171700  5 _do_op 0x69567e0 seq 2627 osr(3.3a 0x276cce0)/0x276cce0 start
2014-06-12 14:46:52.100442 7f9d7795c700  5 -- op tracker -- , seq: 748130, time: 2014-06-12 14:46:52.100434, event: op_applied, op: osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.100449 7f9d82171700 10 _do_transaction on 0x5361680
2014-06-12 14:46:52.100449 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] op_applied on primary on version 173'2963
2014-06-12 14:46:52.100456 7f9d82171700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.100550 7f9d82171700 10 lookup_strip_header done  cid meta oid 5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.100556 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] op_commit: 236354
2014-06-12 14:46:52.100572 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.100579 7f9d7715b700  5 -- op tracker -- , seq: 748130, time: 2014-06-12 14:46:52.100573, event: op_commit, op: osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.100618 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.100638 7f9d82171700 15 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.100646 7f9d82171700 10 _omap_rmkeys meta/5168262c/pglog_3.3a/0//-1 r = 0
2014-06-12 14:46:52.100653 7f9d82171700 15 _omap_setkeys meta/5168262c/pglog_3.3a/0//-1
2014-06-12 14:46:52.100668 7f9d82171700 15 _write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 3969024~225280
2014-06-12 14:46:52.100811 7f9d82171700 10 lookup_strip_header done  cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.100983 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:52.101814 7f9d82171700 10 _generic_write 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 3969024~225280 = 0
2014-06-12 14:46:52.101825 7f9d44101700  5 -- op tracker -- , seq: 748132, time: 2014-06-12 14:46:52.101788, event: header_read, op: osd_sub_op_reply(client.4317.0:144003 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.101836 7f9d44101700  5 -- op tracker -- , seq: 748132, time: 2014-06-12 14:46:52.101789, event: throttled, op: osd_sub_op_reply(client.4317.0:144003 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.101838 7f9d82171700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.101844 7f9d44101700  5 -- op tracker -- , seq: 748132, time: 2014-06-12 14:46:52.101797, event: all_read, op: osd_sub_op_reply(client.4317.0:144003 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.101849 7f9d82171700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.101852 7f9d44101700  5 -- op tracker -- , seq: 748132, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:144003 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.101855 7f9d82171700 15 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:46:52.101856 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:144003 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:52.101861 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:52.101863 7f9d82171700 10 _setattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:46:52.101869 7f9d44101700 15 osd.2 173 enqueue_op 0xe20af00 prio 196 cost 0 latency 0.000081 osd_sub_op_reply(client.4317.0:144003 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:52.101893 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.101900 7f9d66bde700 10 osd.2 173 dequeue_op 0xe20af00 prio 196 cost 0 latency 0.000112 osd_sub_op_reply(client.4317.0:144003 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean]
2014-06-12 14:46:52.101919 7f9d66bde700  5 -- op tracker -- , seq: 748132, time: 2014-06-12 14:46:52.101916, event: reached_pg, op: osd_sub_op_reply(client.4317.0:144003 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.101928 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] handle_message: 0xe20af00
2014-06-12 14:46:52.101945 7f9d66bde700  5 -- op tracker -- , seq: 748132, time: 2014-06-12 14:46:52.101942, event: started, op: osd_sub_op_reply(client.4317.0:144003 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.101950 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] sub_op_modify_reply: tid 236354 op  ack_type 4 from 21
2014-06-12 14:46:52.101968 7f9d66bde700  5 -- op tracker -- , seq: 748130, time: 2014-06-12 14:46:52.101962, event: sub_op_commit_rec, op: osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.101979 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] repop_all_applied: repop tid 236354 all applied 
2014-06-12 14:46:52.101989 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] eval_repop repgather(0x5323b00 173'2963 rep_tid=236354 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.102010 7f9d82171700 20 set_header setting 16858 cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 parent seq 0
2014-06-12 14:46:52.102007 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean]  sending ack on repgather(0x5323b00 173'2963 rep_tid=236354 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x5ffe500
2014-06-12 14:46:52.102035 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2962 crt=0'0 lcod 173'2961 mlcod 173'2961 active+clean] repop_all_committed: repop tid 236354 all committed 
2014-06-12 14:46:52.102049 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2961 active+clean] eval_repop repgather(0x5323b00 173'2963 rep_tid=236354 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.102065 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.102064 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2961 active+clean] log_op_stats osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 inb 516373 outb 0 rlat 0.009895 lat 0.009924
2014-06-12 14:46:52.102078 7f9d82171700 20 set_header setting 1980 cid meta oid 5168262c/pglog_3.3a/0//-1 parent seq 0
2014-06-12 14:46:52.102079 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2961 active+clean] publish_stats_to_osd 173:3846
2014-06-12 14:46:52.102088 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:52.102091 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2961 active+clean]  sending commit on repgather(0x5323b00 173'2963 rep_tid=236354 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x6487b80
2014-06-12 14:46:52.102118 7f9d66bde700  5 -- op tracker -- , seq: 748130, time: 2014-06-12 14:46:52.102114, event: commit_sent, op: osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.102125 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2961 active+clean]  requeue_ops 
2014-06-12 14:46:52.102134 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean]  removing repgather(0x5323b00 173'2963 rep_tid=236354 committed?=1 applied?=1 op=osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.102148 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean]    q front is repgather(0x5323b00 173'2963 rep_tid=236354 committed?=1 applied?=1 op=osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.102161 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean] remove_repop repgather(0x5323b00 173'2963 rep_tid=236354 committed?=1 applied?=1 op=osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.102174 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean]  requeue_ops 
2014-06-12 14:46:52.102191 7f9d66bde700  5 -- op tracker -- , seq: 748130, time: 2014-06-12 14:46:52.102186, event: done, op: osd_op(client.4317.0:144003 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3452928~516096] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.102205 7f9d66bde700 10 osd.2 173 dequeue_op 0xe20af00 finish
2014-06-12 14:46:52.102212 7f9d66bde700  5 -- op tracker -- , seq: 748132, time: 2014-06-12 14:46:52.102209, event: done, op: osd_sub_op_reply(client.4317.0:144003 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.102222 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.102225 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.103160 7f9d82171700 10 _do_op 0x69567e0 seq 2627 r = 0, finisher 0x6dbc060 0x3eabb900
2014-06-12 14:46:52.103175 7f9d82171700 10 _finish_op 0x69567e0 seq 2627 osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:46:52.103184 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b70e0 (0 active)
2014-06-12 14:46:52.103188 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.103181 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean] op_commit: 236355
2014-06-12 14:46:52.103207 7f9d7715b700  5 -- op tracker -- , seq: 748131, time: 2014-06-12 14:46:52.103200, event: op_commit, op: osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.103230 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean] op_applied: 236355
2014-06-12 14:46:52.103253 7f9d7795c700  5 -- op tracker -- , seq: 748131, time: 2014-06-12 14:46:52.103247, event: op_applied, op: osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.103260 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean] op_applied on primary on version 173'2964
2014-06-12 14:46:52.104136 7f9d44101700  5 -- op tracker -- , seq: 748133, time: 2014-06-12 14:46:52.104080, event: header_read, op: osd_sub_op_reply(client.4317.0:144004 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.104149 7f9d44101700  5 -- op tracker -- , seq: 748133, time: 2014-06-12 14:46:52.104082, event: throttled, op: osd_sub_op_reply(client.4317.0:144004 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.104157 7f9d44101700  5 -- op tracker -- , seq: 748133, time: 2014-06-12 14:46:52.104106, event: all_read, op: osd_sub_op_reply(client.4317.0:144004 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.104168 7f9d44101700  5 -- op tracker -- , seq: 748133, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4317.0:144004 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.104174 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4317.0:144004 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:52.104180 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:46:52.104187 7f9d44101700 15 osd.2 173 enqueue_op 0x17c2a400 prio 196 cost 0 latency 0.000107 osd_sub_op_reply(client.4317.0:144004 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:52.104216 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.104222 7f9d64bda700 10 osd.2 173 dequeue_op 0x17c2a400 prio 196 cost 0 latency 0.000142 osd_sub_op_reply(client.4317.0:144004 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0) v2 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean]
2014-06-12 14:46:52.104244 7f9d64bda700  5 -- op tracker -- , seq: 748133, time: 2014-06-12 14:46:52.104240, event: reached_pg, op: osd_sub_op_reply(client.4317.0:144004 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.104252 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean] handle_message: 0x17c2a400
2014-06-12 14:46:52.104268 7f9d64bda700  5 -- op tracker -- , seq: 748133, time: 2014-06-12 14:46:52.104264, event: started, op: osd_sub_op_reply(client.4317.0:144004 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.104273 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean] sub_op_modify_reply: tid 236355 op  ack_type 4 from 21
2014-06-12 14:46:52.104292 7f9d64bda700  5 -- op tracker -- , seq: 748131, time: 2014-06-12 14:46:52.104285, event: sub_op_commit_rec, op: osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.104301 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean] repop_all_applied: repop tid 236355 all applied 
2014-06-12 14:46:52.104312 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean] eval_repop repgather(0x3403b980 173'2964 rep_tid=236355 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.104329 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean]  sending ack on repgather(0x3403b980 173'2964 rep_tid=236355 committed?=0 applied?=1 lock=0 op=osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x15571400
2014-06-12 14:46:52.104361 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2963 crt=0'0 lcod 173'2962 mlcod 173'2962 active+clean] repop_all_committed: repop tid 236355 all committed 
2014-06-12 14:46:52.104374 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2962 active+clean] eval_repop repgather(0x3403b980 173'2964 rep_tid=236355 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:52.104392 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2962 active+clean] log_op_stats osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4 inb 225557 outb 0 rlat 0.009117 lat 0.009148
2014-06-12 14:46:52.104408 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2962 active+clean] publish_stats_to_osd 173:3847
2014-06-12 14:46:52.104420 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2962 active+clean]  sending commit on repgather(0x3403b980 173'2964 rep_tid=236355 committed?=1 applied?=1 lock=0 op=osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4) 0x1a58d680
2014-06-12 14:46:52.104448 7f9d64bda700  5 -- op tracker -- , seq: 748131, time: 2014-06-12 14:46:52.104444, event: commit_sent, op: osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.104460 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2962 active+clean]  requeue_ops 
2014-06-12 14:46:52.104471 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean]  removing repgather(0x3403b980 173'2964 rep_tid=236355 committed?=1 applied?=1 op=osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.104485 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean]    q front is repgather(0x3403b980 173'2964 rep_tid=236355 committed?=1 applied?=1 op=osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.104499 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean] remove_repop repgather(0x3403b980 173'2964 rep_tid=236355 committed?=1 applied?=1 op=osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:52.104513 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean]  requeue_ops 
2014-06-12 14:46:52.104539 7f9d64bda700  5 -- op tracker -- , seq: 748131, time: 2014-06-12 14:46:52.104533, event: done, op: osd_op(client.4317.0:144004 rbd_data.10b46b8b4567.0000000000001056 [set-alloc-hint object_size 4194304 write_size 4194304,write 3969024~225280] 3.e30483a ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:52.104554 7f9d64bda700 10 osd.2 173 dequeue_op 0x17c2a400 finish
2014-06-12 14:46:52.104561 7f9d64bda700  5 -- op tracker -- , seq: 748133, time: 2014-06-12 14:46:52.104558, event: done, op: osd_sub_op_reply(client.4317.0:144004 3.3a e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 [] ondisk, result = 0)
2014-06-12 14:46:52.104571 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.104574 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.155049 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:52.195171 7f9d56432700  5 -- op tracker -- , seq: 748134, time: 2014-06-12 14:46:52.194705, event: header_read, op: osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.195186 7f9d56432700  5 -- op tracker -- , seq: 748134, time: 2014-06-12 14:46:52.194708, event: throttled, op: osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.195198 7f9d56432700  5 -- op tracker -- , seq: 748134, time: 2014-06-12 14:46:52.195118, event: all_read, op: osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.195207 7f9d56432700  5 -- op tracker -- , seq: 748134, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.195214 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:52.195222 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:52.195229 7f9d56432700 15 osd.2 173 enqueue_op 0x2b015d00 prio 127 cost 156455 latency 0.000524 osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.195265 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.195278 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2b015d00 prio 127 cost 156455 latency 0.000573 osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'2999 (0'0,173'2999] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2998 active]
2014-06-12 14:46:52.195299 7f9d6abe6700  5 -- op tracker -- , seq: 748134, time: 2014-06-12 14:46:52.195296, event: reached_pg, op: osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.195306 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2999 (0'0,173'2999] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2998 active] handle_message: 0x2b015d00
2014-06-12 14:46:52.195328 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2999 (0'0,173'2999] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2998 active] sub_op_modify trans d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 v 173'3000 (transaction) 174
2014-06-12 14:46:52.195346 7f9d6abe6700  5 -- op tracker -- , seq: 748134, time: 2014-06-12 14:46:52.195342, event: started, op: osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.195364 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'2999 (0'0,173'2999] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2998 active] append_log log((0'0,173'2999], crt=0'0) [173'3000 (173'777) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145532 2014-06-12 08:45:54.573257]
2014-06-12 14:46:52.195389 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3000 (0'0,173'3000] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2998 active] add_log_entry 173'3000 (173'777) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145532 2014-06-12 08:45:54.573257
2014-06-12 14:46:52.195413 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3000 (0'0,173'3000] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2998 active] append_log  adding 1 keys
2014-06-12 14:46:52.195448 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3000, trimmed: 
2014-06-12 14:46:52.195473 7f9d6abe6700  5 -- op tracker -- , seq: 748134, time: 2014-06-12 14:46:52.195469, event: started, op: osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.195485 7f9d6abe6700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:52.195489 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.195491 7f9d6abe6700  5 queue_op 0xd5ba8a0 seq 2802 osr(3.458 0x276d960) 157941 bytes   (queue has 1 ops and 157941 bytes)
2014-06-12 14:46:52.195497 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'3000 (0'0,173'3000] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2998 active] do_sub_op osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.195510 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2b015d00 finish
2014-06-12 14:46:52.195513 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.195515 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.195523 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:52.195528 7f9d7e96a700  5 _do_op 0xd5ba8a0 seq 2802 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:52.195532 7f9d7e96a700 10 _do_transaction on 0x2c4e1270
2014-06-12 14:46:52.195538 7f9d7e96a700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:52.200378 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:52.200420 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.200468 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.200490 7f9d7e96a700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:52.200499 7f9d7e96a700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:52.200507 7f9d7e96a700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:52.200524 7f9d7e96a700 15 _write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 0~155648
2014-06-12 14:46:52.200708 7f9d7e96a700 10 lookup_strip_header done  cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:52.200728 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:52.201299 7f9d7e96a700 10 _generic_write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 0~155648 = 0
2014-06-12 14:46:52.201319 7f9d7e96a700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:52.201334 7f9d7e96a700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:46:52.201343 7f9d7e96a700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:52.201351 7f9d7e96a700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:46:52.201494 7f9d7e96a700 20 set_header setting 17709 cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 parent seq 0
2014-06-12 14:46:52.201527 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.201545 7f9d7e96a700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:52.201559 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:52.202253 7f9d56432700  5 -- op tracker -- , seq: 748135, time: 2014-06-12 14:46:52.201250, event: header_read, op: osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.202260 7f9d7e96a700 10 _do_op 0xd5ba8a0 seq 2802 r = 0, finisher 0x33064200 0
2014-06-12 14:46:52.202265 7f9d56432700  5 -- op tracker -- , seq: 748135, time: 2014-06-12 14:46:52.201252, event: throttled, op: osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.202270 7f9d7e96a700 10 _finish_op 0xd5ba8a0 seq 2802 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:52.202276 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:52.202279 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.202274 7f9d56432700  5 -- op tracker -- , seq: 748135, time: 2014-06-12 14:46:52.202225, event: all_read, op: osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.202298 7f9d56432700  5 -- op tracker -- , seq: 748135, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.202305 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:52.202312 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:52.202320 7f9d56432700 15 osd.2 173 enqueue_op 0x2b014f00 prio 127 cost 516903 latency 0.001070 osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.202350 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.202350 7f9d7715b700  5 -- op tracker -- , seq: 748134, time: 2014-06-12 14:46:52.202283, event: commit_sent, op: osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.202358 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3000 (0'0,173'3000] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2998 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:52.202381 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:52.202405 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2b014f00 prio 127 cost 516903 latency 0.001155 osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'3000 (0'0,173'3000] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2999 active]
2014-06-12 14:46:52.202430 7f9d6a3e5700  5 -- op tracker -- , seq: 748135, time: 2014-06-12 14:46:52.202426, event: reached_pg, op: osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.202442 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3000 (0'0,173'3000] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2999 active] handle_message: 0x2b014f00
2014-06-12 14:46:52.202455 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3000 (0'0,173'3000] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2999 active] sub_op_modify trans d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 v 173'3001 (transaction) 174
2014-06-12 14:46:52.202475 7f9d6a3e5700  5 -- op tracker -- , seq: 748135, time: 2014-06-12 14:46:52.202470, event: started, op: osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.202492 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3000 (0'0,173'3000] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2999 active] append_log log((0'0,173'3000], crt=0'0) [173'3001 (173'3000) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145533 2014-06-12 08:45:54.578433]
2014-06-12 14:46:52.202522 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2999 active] add_log_entry 173'3001 (173'3000) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145533 2014-06-12 08:45:54.578433
2014-06-12 14:46:52.202546 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2999 active] append_log  adding 1 keys
2014-06-12 14:46:52.202581 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3001, trimmed: 
2014-06-12 14:46:52.202608 7f9d6a3e5700  5 -- op tracker -- , seq: 748135, time: 2014-06-12 14:46:52.202603, event: started, op: osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.202620 7f9d6a3e5700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:52.202623 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.202625 7f9d6a3e5700  5 queue_op 0x241a6e40 seq 2803 osr(3.458 0x276d960) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:52.202634 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:52.202631 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2999 active] do_sub_op osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.202645 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2b014f00 finish
2014-06-12 14:46:52.202647 7f9d8116f700  5 _do_op 0x241a6e40 seq 2803 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:52.202651 7f9d8116f700 10 _do_transaction on 0x6670630
2014-06-12 14:46:52.202651 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.202654 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.202657 7f9d8116f700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:52.202661 7f9d7795c700  5 -- op tracker -- , seq: 748134, time: 2014-06-12 14:46:52.202654, event: sub_op_applied, op: osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.202670 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2999 active] sub_op_modify_applied on 0x2c4e1180 op osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.202687 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2999 active] op_applied on primary on version 173'3000
2014-06-12 14:46:52.202697 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2999 active] op_applied on replica on version 173'3000
2014-06-12 14:46:52.202719 7f9d7795c700  5 -- op tracker -- , seq: 748134, time: 2014-06-12 14:46:52.202714, event: done, op: osd_sub_op(client.4299.0:145532 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3000 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.202740 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:52.202757 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.202797 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.202878 7f9d8116f700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:52.202888 7f9d8116f700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:52.202895 7f9d8116f700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:52.202915 7f9d8116f700 15 _write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 155648~516096
2014-06-12 14:46:52.203054 7f9d8116f700 10 lookup_strip_header done  cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:52.203198 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:52.204940 7f9d8116f700 10 _generic_write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 155648~516096 = 0
2014-06-12 14:46:52.204988 7f9d8116f700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:52.205003 7f9d8116f700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:46:52.205010 7f9d8116f700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:52.205018 7f9d8116f700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:46:52.205220 7f9d8116f700 20 set_header setting 17709 cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 parent seq 0
2014-06-12 14:46:52.205291 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.205304 7f9d8116f700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:52.205318 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:52.211040 7f9d8116f700 10 _do_op 0x241a6e40 seq 2803 r = 0, finisher 0x17729200 0
2014-06-12 14:46:52.211109 7f9d7715b700  5 -- op tracker -- , seq: 748135, time: 2014-06-12 14:46:52.211100, event: commit_sent, op: osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.211120 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2999 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:52.211152 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:52.211260 7f9d8116f700 10 _finish_op 0x241a6e40 seq 2803 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:52.211286 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:52.211307 7f9d7795c700  5 -- op tracker -- , seq: 748135, time: 2014-06-12 14:46:52.211302, event: sub_op_applied, op: osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.211333 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3000 active] sub_op_modify_applied on 0x6670540 op osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.211349 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3000 active] op_applied on primary on version 173'3001
2014-06-12 14:46:52.211360 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3000 active] op_applied on replica on version 173'3001
2014-06-12 14:46:52.211920 7f9d7795c700  5 -- op tracker -- , seq: 748135, time: 2014-06-12 14:46:52.211914, event: done, op: osd_sub_op(client.4299.0:145533 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3001 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.211969 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.244628 7f9d5bec4700  5 -- op tracker -- , seq: 748136, time: 2014-06-12 14:46:52.243366, event: header_read, op: osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.244638 7f9d5bec4700  5 -- op tracker -- , seq: 748136, time: 2014-06-12 14:46:52.243368, event: throttled, op: osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.244644 7f9d5bec4700  5 -- op tracker -- , seq: 748136, time: 2014-06-12 14:46:52.244594, event: all_read, op: osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.244649 7f9d5bec4700  5 -- op tracker -- , seq: 748136, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.244653 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:52.244657 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:52.244662 7f9d5bec4700 15 osd.2 173 enqueue_op 0x25e42d00 prio 127 cost 721703 latency 0.001296 osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.244687 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.244698 7f9d69be4700 10 osd.2 173 dequeue_op 0x25e42d00 prio 127 cost 721703 latency 0.001332 osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4ba( v 173'2928 (0'0,173'2928] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2927 active]
2014-06-12 14:46:52.244724 7f9d69be4700  5 -- op tracker -- , seq: 748136, time: 2014-06-12 14:46:52.244719, event: reached_pg, op: osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.244733 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2928 (0'0,173'2928] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2927 active] handle_message: 0x25e42d00
2014-06-12 14:46:52.244748 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2928 (0'0,173'2928] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2927 active] sub_op_modify trans 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 v 173'2929 (transaction) 174
2014-06-12 14:46:52.244768 7f9d69be4700  5 -- op tracker -- , seq: 748136, time: 2014-06-12 14:46:52.244764, event: started, op: osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.244785 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2928 (0'0,173'2928] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2927 active] append_log log((0'0,173'2928], crt=0'0) [173'2929 (173'826) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146612 2014-06-12 08:45:54.585297]
2014-06-12 14:46:52.244808 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2929 (0'0,173'2929] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2927 active] add_log_entry 173'2929 (173'826) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146612 2014-06-12 08:45:54.585297
2014-06-12 14:46:52.244831 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2929 (0'0,173'2929] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2927 active] append_log  adding 1 keys
2014-06-12 14:46:52.244882 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2929, trimmed: 
2014-06-12 14:46:52.244907 7f9d69be4700  5 -- op tracker -- , seq: 748136, time: 2014-06-12 14:46:52.244903, event: started, op: osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.244917 7f9d69be4700  5 queue_transactions existing osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:52.244921 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.244923 7f9d69be4700  5 queue_op 0x79fb3e0 seq 2653 osr(3.4ba 0x276d830) 723189 bytes   (queue has 1 ops and 723189 bytes)
2014-06-12 14:46:52.244933 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d560 (1 active)
2014-06-12 14:46:52.244939 7f9d7d167700  5 _do_op 0x79fb3e0 seq 2653 osr(3.4ba 0x276d830)/0x276d830 start
2014-06-12 14:46:52.244942 7f9d7d167700 10 _do_transaction on 0x27756470
2014-06-12 14:46:52.244930 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2929 (0'0,173'2929] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2927 active] do_sub_op osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.244946 7f9d69be4700 10 osd.2 173 dequeue_op 0x25e42d00 finish
2014-06-12 14:46:52.244947 7f9d7d167700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:52.244950 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.244952 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.245029 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:52.245046 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.245072 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.245155 7f9d7d167700 15 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:52.245161 7f9d7d167700 10 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1 r = 0
2014-06-12 14:46:52.245165 7f9d7d167700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:52.245178 7f9d7d167700 15 _write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 0~720896
2014-06-12 14:46:52.245410 7f9d7d167700 10 lookup_strip_header done  cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:52.245438 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:52.247243 7f9d7d167700 10 _generic_write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 0~720896 = 0
2014-06-12 14:46:52.247299 7f9d7d167700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:52.247310 7f9d7d167700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:52.247316 7f9d7d167700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:52.247320 7f9d7d167700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:52.247478 7f9d7d167700 20 set_header setting 17403 cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 parent seq 0
2014-06-12 14:46:52.247536 7f9d7d167700 20 set_header setting 2300 cid meta oid 1fe00356/pglog_3.4ba/0//-1 parent seq 0
2014-06-12 14:46:52.247547 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.247553 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:52.249970 7f9d7d167700 10 _do_op 0x79fb3e0 seq 2653 r = 0, finisher 0x17728280 0
2014-06-12 14:46:52.249994 7f9d7d167700 10 _finish_op 0x79fb3e0 seq 2653 osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:52.250011 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d560 (0 active)
2014-06-12 14:46:52.250014 7f9d7715b700  5 -- op tracker -- , seq: 748136, time: 2014-06-12 14:46:52.250002, event: commit_sent, op: osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.250022 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.250024 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2929 (0'0,173'2929] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2927 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:52.250050 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:52.250082 7f9d7795c700  5 -- op tracker -- , seq: 748136, time: 2014-06-12 14:46:52.250076, event: sub_op_applied, op: osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.250090 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2929 (0'0,173'2929] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2928 active] sub_op_modify_applied on 0x27756380 op osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.250101 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2929 (0'0,173'2929] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2928 active] op_applied on primary on version 173'2929
2014-06-12 14:46:52.250107 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2929 (0'0,173'2929] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2928 active] op_applied on replica on version 173'2929
2014-06-12 14:46:52.250130 7f9d7795c700  5 -- op tracker -- , seq: 748136, time: 2014-06-12 14:46:52.250128, event: done, op: osd_sub_op(client.4313.0:146612 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.310080 7f9d5bec4700  5 -- op tracker -- , seq: 748137, time: 2014-06-12 14:46:52.308352, event: header_read, op: osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.310096 7f9d5bec4700  5 -- op tracker -- , seq: 748137, time: 2014-06-12 14:46:52.308354, event: throttled, op: osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.310112 7f9d5bec4700  5 -- op tracker -- , seq: 748137, time: 2014-06-12 14:46:52.310031, event: all_read, op: osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.310760 7f9d5bec4700  5 -- op tracker -- , seq: 748137, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.310805 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:52.310857 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:52.310926 7f9d5bec4700 15 osd.2 173 enqueue_op 0x25e42c00 prio 127 cost 1032999 latency 0.002574 osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.310955 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.310964 7f9d653db700 10 osd.2 173 dequeue_op 0x25e42c00 prio 127 cost 1032999 latency 0.002612 osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.640( v 173'3014 (79'13,173'3014] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3013 active]
2014-06-12 14:46:52.310987 7f9d653db700  5 -- op tracker -- , seq: 748137, time: 2014-06-12 14:46:52.310983, event: reached_pg, op: osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.311001 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3014 (79'13,173'3014] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3013 active] handle_message: 0x25e42c00
2014-06-12 14:46:52.311018 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3014 (79'13,173'3014] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3013 active] sub_op_modify trans a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 v 173'3015 (transaction) 174
2014-06-12 14:46:52.311035 7f9d653db700  5 -- op tracker -- , seq: 748137, time: 2014-06-12 14:46:52.311031, event: started, op: osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.311052 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3014 (79'13,173'3014] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3013 active] append_log log((79'13,173'3014], crt=0'0) [173'3015 (173'3014) modify   a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 by client.4316.0:147287 2014-06-12 08:45:54.683183]
2014-06-12 14:46:52.311081 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3015 (79'13,173'3015] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3013 active] add_log_entry 173'3015 (173'3014) modify   a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 by client.4316.0:147287 2014-06-12 08:45:54.683183
2014-06-12 14:46:52.311106 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3015 (79'13,173'3015] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3013 active] append_log  adding 1 keys
2014-06-12 14:46:52.311123 7f9d653db700 10 trim log((79'13,173'3015], crt=0'0) to 79'14
2014-06-12 14:46:52.311152 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3015, trimmed: 79'14
2014-06-12 14:46:52.311184 7f9d653db700  5 -- op tracker -- , seq: 748137, time: 2014-06-12 14:46:52.311177, event: started, op: osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.311206 7f9d653db700  5 queue_transactions existing osr(3.640 0x276dc10)/0x276dc10
2014-06-12 14:46:52.311214 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.311218 7f9d653db700  5 queue_op 0xda7c300 seq 2772 osr(3.640 0x276dc10) 1034520 bytes   (queue has 1 ops and 1034520 bytes)
2014-06-12 14:46:52.311229 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.640( v 173'3015 (79'14,173'3015] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3013 active] do_sub_op osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.311244 7f9d653db700 10 osd.2 173 dequeue_op 0x25e42c00 finish
2014-06-12 14:46:52.311248 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.311250 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.311268 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6900 (1 active)
2014-06-12 14:46:52.311273 7f9d79960700  5 _do_op 0xda7c300 seq 2772 osr(3.640 0x276dc10)/0x276dc10 start
2014-06-12 14:46:52.311277 7f9d79960700 10 _do_transaction on 0x2721270
2014-06-12 14:46:52.311284 7f9d79960700 15 _omap_setkeys meta/1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:52.311395 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:52.311430 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.311478 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.311501 7f9d79960700 15 _omap_rmkeys meta/1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:52.311511 7f9d79960700 10 _omap_rmkeys meta/1fe13b9a/pglog_3.640/0//-1 r = 0
2014-06-12 14:46:52.311518 7f9d79960700 15 _omap_setkeys meta/1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:52.311617 7f9d79960700 15 _write 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 131072~1032192
2014-06-12 14:46:52.311787 7f9d79960700 10 lookup_strip_header done  cid 3.640_head oid a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3
2014-06-12 14:46:52.311938 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:52.315986 7f9d79960700 10 _generic_write 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 131072~1032192 = 0
2014-06-12 14:46:52.316074 7f9d79960700 15 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3
2014-06-12 14:46:52.316110 7f9d79960700 10 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 = 0
2014-06-12 14:46:52.316120 7f9d79960700 15 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3
2014-06-12 14:46:52.316128 7f9d79960700 10 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 = 0
2014-06-12 14:46:52.316425 7f9d79960700 20 set_header setting 17474 cid 3.640_head oid a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 parent seq 0
2014-06-12 14:46:52.316540 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.316558 7f9d79960700 20 set_header setting 2200 cid meta oid 1fe13b9a/pglog_3.640/0//-1 parent seq 0
2014-06-12 14:46:52.316571 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:52.322256 7f9d79960700 10 _do_op 0xda7c300 seq 2772 r = 0, finisher 0x7eb01a0 0
2014-06-12 14:46:52.322278 7f9d79960700 10 _finish_op 0xda7c300 seq 2772 osr(3.640 0x276dc10)/0x276dc10
2014-06-12 14:46:52.322285 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6900 (0 active)
2014-06-12 14:46:52.322294 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.322295 7f9d7715b700  5 -- op tracker -- , seq: 748137, time: 2014-06-12 14:46:52.322285, event: commit_sent, op: osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.322304 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3015 (79'14,173'3015] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3013 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:52.322331 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:52.322366 7f9d7795c700  5 -- op tracker -- , seq: 748137, time: 2014-06-12 14:46:52.322360, event: sub_op_applied, op: osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.322379 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3015 (79'14,173'3015] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active] sub_op_modify_applied on 0x2721180 op osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.322394 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3015 (79'14,173'3015] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active] op_applied on primary on version 173'3015
2014-06-12 14:46:52.322404 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3015 (79'14,173'3015] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active] op_applied on replica on version 173'3015
2014-06-12 14:46:52.322430 7f9d7795c700  5 -- op tracker -- , seq: 748137, time: 2014-06-12 14:46:52.322426, event: done, op: osd_sub_op(client.4316.0:147287 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3015 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.332462 7f9d5bec4700  5 -- op tracker -- , seq: 748138, time: 2014-06-12 14:46:52.330659, event: header_read, op: osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.332476 7f9d5bec4700  5 -- op tracker -- , seq: 748138, time: 2014-06-12 14:46:52.330662, event: throttled, op: osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.332485 7f9d5bec4700  5 -- op tracker -- , seq: 748138, time: 2014-06-12 14:46:52.332417, event: all_read, op: osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.332494 7f9d5bec4700  5 -- op tracker -- , seq: 748138, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.332500 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:52.332508 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:52.332517 7f9d5bec4700 15 osd.2 173 enqueue_op 0x25e42700 prio 127 cost 1032999 latency 0.001858 osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.332570 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.332582 7f9d623d5700 10 osd.2 173 dequeue_op 0x25e42700 prio 127 cost 1032999 latency 0.001922 osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.640( v 173'3015 (79'14,173'3015] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active]
2014-06-12 14:46:52.332612 7f9d623d5700  5 -- op tracker -- , seq: 748138, time: 2014-06-12 14:46:52.332607, event: reached_pg, op: osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.332622 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3015 (79'14,173'3015] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active] handle_message: 0x25e42700
2014-06-12 14:46:52.332635 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3015 (79'14,173'3015] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active] sub_op_modify trans a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 v 173'3016 (transaction) 174
2014-06-12 14:46:52.332652 7f9d623d5700  5 -- op tracker -- , seq: 748138, time: 2014-06-12 14:46:52.332648, event: started, op: osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.332670 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3015 (79'14,173'3015] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active] append_log log((79'14,173'3015], crt=0'0) [173'3016 (173'3015) modify   a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 by client.4316.0:147288 2014-06-12 08:45:54.705386]
2014-06-12 14:46:52.332696 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3016 (79'14,173'3016] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active] add_log_entry 173'3016 (173'3015) modify   a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 by client.4316.0:147288 2014-06-12 08:45:54.705386
2014-06-12 14:46:52.332721 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3016 (79'14,173'3016] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active] append_log  adding 1 keys
2014-06-12 14:46:52.332735 7f9d623d5700 10 trim log((79'14,173'3016], crt=0'0) to 79'15
2014-06-12 14:46:52.332765 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3016, trimmed: 79'15
2014-06-12 14:46:52.332793 7f9d623d5700  5 -- op tracker -- , seq: 748138, time: 2014-06-12 14:46:52.332788, event: started, op: osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.332804 7f9d623d5700  5 queue_transactions existing osr(3.640 0x276dc10)/0x276dc10
2014-06-12 14:46:52.332808 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.332810 7f9d623d5700  5 queue_op 0x6cdf440 seq 2773 osr(3.640 0x276dc10) 1034520 bytes   (queue has 1 ops and 1034520 bytes)
2014-06-12 14:46:52.332818 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.640( v 173'3016 (79'15,173'3016] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active] do_sub_op osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.332833 7f9d623d5700 10 osd.2 173 dequeue_op 0x25e42700 finish
2014-06-12 14:46:52.332837 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.332840 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.332849 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6900 (1 active)
2014-06-12 14:46:52.332855 7f9d8016d700  5 _do_op 0x6cdf440 seq 2773 osr(3.640 0x276dc10)/0x276dc10 start
2014-06-12 14:46:52.332859 7f9d8016d700 10 _do_transaction on 0x61ceb70
2014-06-12 14:46:52.332865 7f9d8016d700 15 _omap_setkeys meta/1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:52.332980 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:52.333005 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.333061 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.333172 7f9d8016d700 15 _omap_rmkeys meta/1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:52.333182 7f9d8016d700 10 _omap_rmkeys meta/1fe13b9a/pglog_3.640/0//-1 r = 0
2014-06-12 14:46:52.333189 7f9d8016d700 15 _omap_setkeys meta/1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:52.333209 7f9d8016d700 15 _write 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 1163264~1032192
2014-06-12 14:46:52.333367 7f9d8016d700 10 lookup_strip_header done  cid 3.640_head oid a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3
2014-06-12 14:46:52.333524 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:52.338064 7f9d8016d700 10 _generic_write 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 1163264~1032192 = 0
2014-06-12 14:46:52.338179 7f9d8016d700 15 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3
2014-06-12 14:46:52.338197 7f9d8016d700 10 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 = 0
2014-06-12 14:46:52.338205 7f9d8016d700 15 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3
2014-06-12 14:46:52.338216 7f9d8016d700 10 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 = 0
2014-06-12 14:46:52.338539 7f9d8016d700 20 set_header setting 17474 cid 3.640_head oid a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 parent seq 0
2014-06-12 14:46:52.338647 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.338662 7f9d8016d700 20 set_header setting 2200 cid meta oid 1fe13b9a/pglog_3.640/0//-1 parent seq 0
2014-06-12 14:46:52.338677 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:52.344283 7f9d8016d700 10 _do_op 0x6cdf440 seq 2773 r = 0, finisher 0x1ae4ec60 0
2014-06-12 14:46:52.344307 7f9d8016d700 10 _finish_op 0x6cdf440 seq 2773 osr(3.640 0x276dc10)/0x276dc10
2014-06-12 14:46:52.344337 7f9d7795c700  5 -- op tracker -- , seq: 748138, time: 2014-06-12 14:46:52.344327, event: sub_op_applied, op: osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.344348 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3016 (79'15,173'3016] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active] sub_op_modify_applied on 0x61cea80 op osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.344377 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:52.344399 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3016 (79'15,173'3016] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active] op_applied on primary on version 173'3016
2014-06-12 14:46:52.344415 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3016 (79'15,173'3016] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active] op_applied on replica on version 173'3016
2014-06-12 14:46:52.344431 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6900 (0 active)
2014-06-12 14:46:52.344438 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.344444 7f9d7715b700  5 -- op tracker -- , seq: 748138, time: 2014-06-12 14:46:52.344436, event: commit_sent, op: osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.344452 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3016 (79'15,173'3016] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3014 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:52.344471 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:52.344502 7f9d7715b700  5 -- op tracker -- , seq: 748138, time: 2014-06-12 14:46:52.344496, event: done, op: osd_sub_op(client.4316.0:147288 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3016 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.357740 7f9d5bec4700  5 -- op tracker -- , seq: 748139, time: 2014-06-12 14:46:52.354302, event: header_read, op: osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.357754 7f9d5bec4700  5 -- op tracker -- , seq: 748139, time: 2014-06-12 14:46:52.354304, event: throttled, op: osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.357762 7f9d5bec4700  5 -- op tracker -- , seq: 748139, time: 2014-06-12 14:46:52.357694, event: all_read, op: osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.357770 7f9d5bec4700  5 -- op tracker -- , seq: 748139, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.357776 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:52.357784 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:52.357792 7f9d5bec4700 15 osd.2 173 enqueue_op 0x25e42a00 prio 127 cost 1999655 latency 0.003490 osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.357825 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.357837 7f9d68be2700 10 osd.2 173 dequeue_op 0x25e42a00 prio 127 cost 1999655 latency 0.003534 osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.640( v 173'3016 (79'15,173'3016] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3015 active]
2014-06-12 14:46:52.357864 7f9d68be2700  5 -- op tracker -- , seq: 748139, time: 2014-06-12 14:46:52.357859, event: reached_pg, op: osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.357875 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3016 (79'15,173'3016] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3015 active] handle_message: 0x25e42a00
2014-06-12 14:46:52.357889 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3016 (79'15,173'3016] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3015 active] sub_op_modify trans a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 v 173'3017 (transaction) 174
2014-06-12 14:46:52.357908 7f9d68be2700  5 -- op tracker -- , seq: 748139, time: 2014-06-12 14:46:52.357904, event: started, op: osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.357925 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3016 (79'15,173'3016] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3015 active] append_log log((79'15,173'3016], crt=0'0) [173'3017 (173'3016) modify   a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 by client.4316.0:147289 2014-06-12 08:45:54.722937]
2014-06-12 14:46:52.357953 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3017 (79'15,173'3017] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3015 active] add_log_entry 173'3017 (173'3016) modify   a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 by client.4316.0:147289 2014-06-12 08:45:54.722937
2014-06-12 14:46:52.357976 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3017 (79'15,173'3017] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3015 active] append_log  adding 1 keys
2014-06-12 14:46:52.357991 7f9d68be2700 10 trim log((79'15,173'3017], crt=0'0) to 79'16
2014-06-12 14:46:52.358019 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3017, trimmed: 79'16
2014-06-12 14:46:52.358052 7f9d68be2700  5 -- op tracker -- , seq: 748139, time: 2014-06-12 14:46:52.358047, event: started, op: osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.358064 7f9d68be2700  5 queue_transactions existing osr(3.640 0x276dc10)/0x276dc10
2014-06-12 14:46:52.358069 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.358070 7f9d68be2700  5 queue_op 0x565bb60 seq 2774 osr(3.640 0x276dc10) 2001176 bytes   (queue has 1 ops and 2001176 bytes)
2014-06-12 14:46:52.358077 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.640( v 173'3017 (79'16,173'3017] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3015 active] do_sub_op osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.358093 7f9d68be2700 10 osd.2 173 dequeue_op 0x25e42a00 finish
2014-06-12 14:46:52.358097 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.358099 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.358107 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b6900 (1 active)
2014-06-12 14:46:52.358112 7f9d7915f700  5 _do_op 0x565bb60 seq 2774 osr(3.640 0x276dc10)/0x276dc10 start
2014-06-12 14:46:52.358117 7f9d7915f700 10 _do_transaction on 0x23fbfeb0
2014-06-12 14:46:52.358123 7f9d7915f700 15 _omap_setkeys meta/1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:52.358232 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:52.358261 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.358306 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.358423 7f9d7915f700 15 _omap_rmkeys meta/1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:52.358434 7f9d7915f700 10 _omap_rmkeys meta/1fe13b9a/pglog_3.640/0//-1 r = 0
2014-06-12 14:46:52.358442 7f9d7915f700 15 _omap_setkeys meta/1fe13b9a/pglog_3.640/0//-1
2014-06-12 14:46:52.358460 7f9d7915f700 15 _write 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 2195456~1998848
2014-06-12 14:46:52.358594 7f9d7915f700 10 lookup_strip_header done  cid 3.640_head oid a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3
2014-06-12 14:46:52.358750 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:52.365561 7f9d7915f700 10 _generic_write 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 2195456~1998848 = 0
2014-06-12 14:46:52.365768 7f9d7915f700 15 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3
2014-06-12 14:46:52.365784 7f9d7915f700 10 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 = 0
2014-06-12 14:46:52.365790 7f9d7915f700 15 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3
2014-06-12 14:46:52.365795 7f9d7915f700 10 _setattrs 3.640_head/a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 = 0
2014-06-12 14:46:52.366245 7f9d7915f700 20 set_header setting 17474 cid 3.640_head oid a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 parent seq 0
2014-06-12 14:46:52.366367 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.366379 7f9d7915f700 20 set_header setting 2200 cid meta oid 1fe13b9a/pglog_3.640/0//-1 parent seq 0
2014-06-12 14:46:52.366388 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:52.369564 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:52.375773 7f9d7915f700 10 _do_op 0x565bb60 seq 2774 r = 0, finisher 0x686e260 0
2014-06-12 14:46:52.375800 7f9d7915f700 10 _finish_op 0x565bb60 seq 2774 osr(3.640 0x276dc10)/0x276dc10
2014-06-12 14:46:52.375807 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b6900 (0 active)
2014-06-12 14:46:52.375818 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.375823 7f9d7715b700  5 -- op tracker -- , seq: 748139, time: 2014-06-12 14:46:52.375810, event: commit_sent, op: osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.375842 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3017 (79'16,173'3017] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3015 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:52.375871 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:52.375930 7f9d7795c700  5 -- op tracker -- , seq: 748139, time: 2014-06-12 14:46:52.375922, event: sub_op_applied, op: osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.375954 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3017 (79'16,173'3017] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3016 active] sub_op_modify_applied on 0x23fbfdc0 op osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.375972 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3017 (79'16,173'3017] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3016 active] op_applied on primary on version 173'3017
2014-06-12 14:46:52.375979 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.640( v 173'3017 (79'16,173'3017] local-les=149 n=318 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'3016 active] op_applied on replica on version 173'3017
2014-06-12 14:46:52.376000 7f9d7795c700  5 -- op tracker -- , seq: 748139, time: 2014-06-12 14:46:52.375997, event: done, op: osd_sub_op(client.4316.0:147289 3.640 a99e9e40/rbd_data.10b36b8b4567.0000000000001138/head//3 [] v 173'3017 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.408069 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:52.408118 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:52.479547 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:52.479566 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:46:52.479569 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:46:52.479572 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:46:52.479577 7f9d84175700 10 osd.2 173 send_pg_stats - 2 pgs updated
2014-06-12 14:46:52.479611 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:52.479614 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:52.479616 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:52.479618 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 19; oldest is 19.318273 seconds old
2014-06-12 14:46:52.479629 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:52.520830 7f9d56432700  5 -- op tracker -- , seq: 748140, time: 2014-06-12 14:46:52.520016, event: header_read, op: osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.520842 7f9d56432700  5 -- op tracker -- , seq: 748140, time: 2014-06-12 14:46:52.520019, event: throttled, op: osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.520851 7f9d56432700  5 -- op tracker -- , seq: 748140, time: 2014-06-12 14:46:52.520797, event: all_read, op: osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.520860 7f9d56432700  5 -- op tracker -- , seq: 748140, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.520866 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:52.520873 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:52.520881 7f9d56432700 15 osd.2 173 enqueue_op 0x2b014000 prio 127 cost 516903 latency 0.000865 osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.520908 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:52.520917 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2b014000 prio 127 cost 516903 latency 0.000901 osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3000 active]
2014-06-12 14:46:52.520941 7f9d63bd8700  5 -- op tracker -- , seq: 748140, time: 2014-06-12 14:46:52.520937, event: reached_pg, op: osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.520950 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3000 active] handle_message: 0x2b014000
2014-06-12 14:46:52.520962 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3000 active] sub_op_modify trans d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 v 173'3002 (transaction) 174
2014-06-12 14:46:52.520979 7f9d63bd8700  5 -- op tracker -- , seq: 748140, time: 2014-06-12 14:46:52.520975, event: started, op: osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.520996 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3001 (0'0,173'3001] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3000 active] append_log log((0'0,173'3001], crt=0'0) [173'3002 (173'3001) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145534 2014-06-12 08:45:54.897352]
2014-06-12 14:46:52.521017 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3002 (0'0,173'3002] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3000 active] add_log_entry 173'3002 (173'3001) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145534 2014-06-12 08:45:54.897352
2014-06-12 14:46:52.521038 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3002 (0'0,173'3002] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3000 active] append_log  adding 1 keys
2014-06-12 14:46:52.521051 7f9d63bd8700 10 trim log((0'0,173'3002], crt=0'0) to 79'1
2014-06-12 14:46:52.521075 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3002, trimmed: 79'1
2014-06-12 14:46:52.521098 7f9d63bd8700  5 -- op tracker -- , seq: 748140, time: 2014-06-12 14:46:52.521094, event: started, op: osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.521109 7f9d63bd8700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:52.521112 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:52.521114 7f9d63bd8700  5 queue_op 0x139d9c20 seq 2804 osr(3.458 0x276d960) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:52.521123 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:52.521130 7f9d7d968700  5 _do_op 0x139d9c20 seq 2804 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:52.521119 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'3002 (79'1,173'3002] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3000 active] do_sub_op osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.521133 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2b014000 finish
2014-06-12 14:46:52.521135 7f9d7d968700 10 _do_transaction on 0x1da3e2b0
2014-06-12 14:46:52.521136 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:52.521138 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:52.521141 7f9d7d968700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:52.521255 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:52.521277 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:52.521329 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:52.521458 7f9d7d968700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:52.521469 7f9d7d968700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:52.521477 7f9d7d968700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:52.521497 7f9d7d968700 15 _write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 671744~516096
2014-06-12 14:46:52.521664 7f9d7d968700 10 lookup_strip_header done  cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:52.521867 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:52.522916 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:52.522986 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:52.524343 7f9d7d968700 10 _generic_write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 671744~516096 = 0
2014-06-12 14:46:52.524411 7f9d7d968700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:52.524428 7f9d7d968700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:46:52.524433 7f9d7d968700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:52.524441 7f9d7d968700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:46:52.524633 7f9d7d968700 20 set_header setting 17709 cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 parent seq 0
2014-06-12 14:46:52.524741 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:52.524762 7f9d7d968700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:52.524774 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:52.527202 7f9d7d968700 10 _do_op 0x139d9c20 seq 2804 r = 0, finisher 0x7b420c0 0
2014-06-12 14:46:52.527223 7f9d7d968700 10 _finish_op 0x139d9c20 seq 2804 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:52.527239 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:52.527250 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:52.527244 7f9d7715b700  5 -- op tracker -- , seq: 748140, time: 2014-06-12 14:46:52.527233, event: commit_sent, op: osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.527254 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3002 (79'1,173'3002] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3000 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:52.527284 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:52.527326 7f9d7795c700  5 -- op tracker -- , seq: 748140, time: 2014-06-12 14:46:52.527320, event: sub_op_applied, op: osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.527337 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3002 (79'1,173'3002] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3001 active] sub_op_modify_applied on 0x1da3e1c0 op osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:52.527354 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3002 (79'1,173'3002] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3001 active] op_applied on primary on version 173'3002
2014-06-12 14:46:52.527365 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3002 (79'1,173'3002] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3001 active] op_applied on replica on version 173'3002
2014-06-12 14:46:52.527393 7f9d7795c700  5 -- op tracker -- , seq: 748140, time: 2014-06-12 14:46:52.527388, event: done, op: osd_sub_op(client.4299.0:145534 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3002 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:52.697283 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:52.697286 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:52.697288 7f9d74bfa700 20 osd.2 173 _dispatch 0x2705e380 pg_stats_ack(2 pgs tid 4856) v1
2014-06-12 14:46:52.697290 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:46:52.697296 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:52.697297 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:52.755737 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:52.757525 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:53.345536 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:53.345584 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:53.411208 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:53.411259 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:53.479686 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:53.479713 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:53.479716 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:53.479718 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:53.479721 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 19; oldest is 20.318376 seconds old
2014-06-12 14:46:53.479734 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:53.844420 7f9d5a16f700  5 -- op tracker -- , seq: 748141, time: 2014-06-12 14:46:53.841870, event: header_read, op: osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.844433 7f9d5a16f700  5 -- op tracker -- , seq: 748141, time: 2014-06-12 14:46:53.841874, event: throttled, op: osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.844455 7f9d5a16f700  5 -- op tracker -- , seq: 748141, time: 2014-06-12 14:46:53.844383, event: all_read, op: osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.844464 7f9d5a16f700  5 -- op tracker -- , seq: 748141, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.844469 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:53.844475 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:53.844483 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2910200 prio 127 cost 1032999 latency 0.002613 osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:53.844511 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:53.844517 7f9d643d9700 10 osd.2 173 dequeue_op 0x2910200 prio 127 cost 1032999 latency 0.002647 osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2504 (0'0,173'2504] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2503 active]
2014-06-12 14:46:53.844542 7f9d643d9700  5 -- op tracker -- , seq: 748141, time: 2014-06-12 14:46:53.844538, event: reached_pg, op: osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.844550 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2504 (0'0,173'2504] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2503 active] handle_message: 0x2910200
2014-06-12 14:46:53.844563 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2504 (0'0,173'2504] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2503 active] sub_op_modify trans a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 v 173'2505 (transaction) 174
2014-06-12 14:46:53.844580 7f9d643d9700  5 -- op tracker -- , seq: 748141, time: 2014-06-12 14:46:53.844576, event: started, op: osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.844593 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2504 (0'0,173'2504] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2503 active] append_log log((0'0,173'2504], crt=0'0) [173'2505 (173'2504) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146440 2014-06-12 08:41:36.679419]
2014-06-12 14:46:53.844617 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2505 (0'0,173'2505] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2503 active] add_log_entry 173'2505 (173'2504) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146440 2014-06-12 08:41:36.679419
2014-06-12 14:46:53.844639 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2505 (0'0,173'2505] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2503 active] append_log  adding 1 keys
2014-06-12 14:46:53.844671 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2505, trimmed: 
2014-06-12 14:46:53.844694 7f9d643d9700  5 -- op tracker -- , seq: 748141, time: 2014-06-12 14:46:53.844689, event: started, op: osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.844704 7f9d643d9700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:46:53.844708 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:53.844709 7f9d643d9700  5 queue_op 0x1b39e240 seq 2351 osr(3.168 0x276ce40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:53.844716 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2505 (0'0,173'2505] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2503 active] do_sub_op osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:53.844730 7f9d643d9700 10 osd.2 173 dequeue_op 0x2910200 finish
2014-06-12 14:46:53.844734 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:53.844736 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:53.844744 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:46:53.844749 7f9d7c165700  5 _do_op 0x1b39e240 seq 2351 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:46:53.844754 7f9d7c165700 10 _do_transaction on 0x13f847f0
2014-06-12 14:46:53.844759 7f9d7c165700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:53.844879 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:53.844905 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:53.844955 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:53.845053 7f9d7c165700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:53.845057 7f9d7c165700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:46:53.845061 7f9d7c165700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:53.845073 7f9d7c165700 15 _write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 32768~1032192
2014-06-12 14:46:53.845300 7f9d7c165700 10 lookup_strip_header done  cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:46:53.845329 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:53.848441 7f9d7c165700 10 _generic_write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 32768~1032192 = 0
2014-06-12 14:46:53.848520 7f9d7c165700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:46:53.848532 7f9d7c165700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:46:53.848536 7f9d7c165700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:46:53.848543 7f9d7c165700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:46:53.848743 7f9d7c165700 20 set_header setting 16934 cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 parent seq 0
2014-06-12 14:46:53.848824 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:53.848837 7f9d7c165700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:46:53.848843 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:53.851555 7f9d5a16f700  5 -- op tracker -- , seq: 748142, time: 2014-06-12 14:46:53.850663, event: header_read, op: osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.851569 7f9d5a16f700  5 -- op tracker -- , seq: 748142, time: 2014-06-12 14:46:53.850666, event: throttled, op: osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.851578 7f9d5a16f700  5 -- op tracker -- , seq: 748142, time: 2014-06-12 14:46:53.851512, event: all_read, op: osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.851587 7f9d5a16f700  5 -- op tracker -- , seq: 748142, time: 0.000000, event: dispatched, op: osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.851592 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:53.851599 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:53.851607 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2911f00 prio 127 cost 516903 latency 0.000943 osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:53.851636 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:53.851644 7f9d633d7700 10 osd.2 173 dequeue_op 0x2911f00 prio 127 cost 516903 latency 0.000980 osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2723 (0'0,173'2723] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active]
2014-06-12 14:46:53.851666 7f9d633d7700  5 -- op tracker -- , seq: 748142, time: 2014-06-12 14:46:53.851661, event: reached_pg, op: osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.851675 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2723 (0'0,173'2723] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] handle_message: 0x2911f00
2014-06-12 14:46:53.851687 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2723 (0'0,173'2723] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] sub_op_modify trans cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 v 173'2724 (transaction) 174
2014-06-12 14:46:53.851703 7f9d633d7700  5 -- op tracker -- , seq: 748142, time: 2014-06-12 14:46:53.851699, event: started, op: osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.851721 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2723 (0'0,173'2723] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] append_log log((0'0,173'2723], crt=0'0) [173'2724 (173'2723) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143146 2014-06-12 08:43:36.359166]
2014-06-12 14:46:53.851747 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2724 (0'0,173'2724] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] add_log_entry 173'2724 (173'2723) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143146 2014-06-12 08:43:36.359166
2014-06-12 14:46:53.851770 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2724 (0'0,173'2724] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] append_log  adding 1 keys
2014-06-12 14:46:53.851807 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2724, trimmed: 
2014-06-12 14:46:53.851835 7f9d633d7700  5 -- op tracker -- , seq: 748142, time: 2014-06-12 14:46:53.851830, event: started, op: osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.851845 7f9d633d7700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:53.851848 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:53.851849 7f9d633d7700  5 queue_op 0x76d8f60 seq 2471 osr(3.21c 0x276d020) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:46:53.851857 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2724 (0'0,173'2724] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] do_sub_op osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:53.851871 7f9d633d7700 10 osd.2 173 dequeue_op 0x2911f00 finish
2014-06-12 14:46:53.851875 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:53.851878 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:53.851891 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (2 active)
2014-06-12 14:46:53.851895 7f9d8096e700  5 _do_op 0x76d8f60 seq 2471 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:46:53.851899 7f9d8096e700 10 _do_transaction on 0x122f7430
2014-06-12 14:46:53.851905 7f9d8096e700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:53.852023 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:53.852043 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:53.852658 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:53.852756 7f9d8096e700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:53.852763 7f9d8096e700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:46:53.852769 7f9d8096e700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:53.852788 7f9d8096e700 15 _write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 2187264~516096
2014-06-12 14:46:53.852868 7f9d7c165700 10 _do_op 0x1b39e240 seq 2351 r = 0, finisher 0x66b6660 0
2014-06-12 14:46:53.852880 7f9d7c165700 10 _finish_op 0x1b39e240 seq 2351 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:46:53.852896 7f9d7715b700  5 -- op tracker -- , seq: 748141, time: 2014-06-12 14:46:53.852889, event: commit_sent, op: osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.852896 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (1 active)
2014-06-12 14:46:53.852904 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:53.852905 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2505 (0'0,173'2505] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2503 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:53.852925 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:53.852972 7f9d7795c700  5 -- op tracker -- , seq: 748141, time: 2014-06-12 14:46:53.852962, event: sub_op_applied, op: osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.852978 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2505 (0'0,173'2505] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2504 active] sub_op_modify_applied on 0x13f84700 op osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:53.852988 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2505 (0'0,173'2505] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2504 active] op_applied on primary on version 173'2505
2014-06-12 14:46:53.852994 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2505 (0'0,173'2505] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2504 active] op_applied on replica on version 173'2505
2014-06-12 14:46:53.853010 7f9d7795c700  5 -- op tracker -- , seq: 748141, time: 2014-06-12 14:46:53.853008, event: done, op: osd_sub_op(client.4316.0:146440 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2505 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.853082 7f9d8096e700 10 lookup_strip_header done  cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:53.853109 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:53.855100 7f9d8096e700 10 _generic_write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 2187264~516096 = 0
2014-06-12 14:46:53.855156 7f9d8096e700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:53.855172 7f9d8096e700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:53.855178 7f9d8096e700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:53.855186 7f9d8096e700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:53.855389 7f9d8096e700 20 set_header setting 16774 cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 parent seq 0
2014-06-12 14:46:53.855463 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:53.855478 7f9d8096e700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:46:53.855488 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:53.858331 7f9d8096e700 10 _do_op 0x76d8f60 seq 2471 r = 0, finisher 0x1f25460 0
2014-06-12 14:46:53.858350 7f9d8096e700 10 _finish_op 0x76d8f60 seq 2471 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:53.858364 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (0 active)
2014-06-12 14:46:53.858372 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:53.858383 7f9d7795c700  5 -- op tracker -- , seq: 748142, time: 2014-06-12 14:46:53.858372, event: sub_op_applied, op: osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.858392 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2724 (0'0,173'2724] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] sub_op_modify_applied on 0x122f7340 op osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:53.858419 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:53.858440 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2724 (0'0,173'2724] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] op_applied on primary on version 173'2724
2014-06-12 14:46:53.858460 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2724 (0'0,173'2724] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] op_applied on replica on version 173'2724
2014-06-12 14:46:53.858482 7f9d7715b700  5 -- op tracker -- , seq: 748142, time: 2014-06-12 14:46:53.858477, event: commit_sent, op: osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.858489 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2724 (0'0,173'2724] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] sub_op_modify_commit on op osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:53.858507 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:53.858538 7f9d7715b700  5 -- op tracker -- , seq: 748142, time: 2014-06-12 14:46:53.858533, event: done, op: osd_sub_op(client.4279.0:143146 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:53.913719 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.041128 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.045096 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.054339 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.055412 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.072639 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.073285 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.077853 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.077869 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.079575 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.082913 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.083657 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.086840 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.088000 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.089878 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.093399 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.094349 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.096480 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.100477 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.102295 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.103248 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.104635 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.107660 7f9d5a16f700  5 -- op tracker -- , seq: 748143, time: 2014-06-12 14:46:54.106652, event: header_read, op: osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.107677 7f9d5a16f700  5 -- op tracker -- , seq: 748143, time: 2014-06-12 14:46:54.106654, event: throttled, op: osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.107685 7f9d5a16f700  5 -- op tracker -- , seq: 748143, time: 2014-06-12 14:46:54.107627, event: all_read, op: osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.107694 7f9d5a16f700  5 -- op tracker -- , seq: 748143, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.107700 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.107708 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:54.107718 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2910100 prio 127 cost 516903 latency 0.001066 osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.107770 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.107778 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2910100 prio 127 cost 516903 latency 0.001125 osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2470 (0'0,173'2470] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2469 active]
2014-06-12 14:46:54.107800 7f9d6abe6700  5 -- op tracker -- , seq: 748143, time: 2014-06-12 14:46:54.107795, event: reached_pg, op: osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.107808 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2470 (0'0,173'2470] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2469 active] handle_message: 0x2910100
2014-06-12 14:46:54.107821 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2470 (0'0,173'2470] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2469 active] sub_op_modify trans 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 v 173'2471 (transaction) 174
2014-06-12 14:46:54.107837 7f9d6abe6700  5 -- op tracker -- , seq: 748143, time: 2014-06-12 14:46:54.107833, event: started, op: osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.107858 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2470 (0'0,173'2470] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2469 active] append_log log((0'0,173'2470], crt=0'0) [173'2471 (173'2470) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143630 2014-06-12 08:43:21.148762]
2014-06-12 14:46:54.107890 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2471 (0'0,173'2471] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2469 active] add_log_entry 173'2471 (173'2470) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143630 2014-06-12 08:43:21.148762
2014-06-12 14:46:54.107911 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2471 (0'0,173'2471] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2469 active] append_log  adding 1 keys
2014-06-12 14:46:54.107944 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2471, trimmed: 
2014-06-12 14:46:54.107967 7f9d6abe6700  5 -- op tracker -- , seq: 748143, time: 2014-06-12 14:46:54.107962, event: started, op: osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.107977 7f9d6abe6700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:54.107981 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.107983 7f9d6abe6700  5 queue_op 0x13c50d20 seq 2304 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:54.107990 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2471 (0'0,173'2471] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2469 active] do_sub_op osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.108004 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2910100 finish
2014-06-12 14:46:54.108007 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.108010 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.108019 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:46:54.108028 7f9d7e96a700  5 _do_op 0x13c50d20 seq 2304 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:46:54.108032 7f9d7e96a700 10 _do_transaction on 0x12c25b30
2014-06-12 14:46:54.108038 7f9d7e96a700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:54.108137 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:54.108154 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.108198 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.108213 7f9d7e96a700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:54.108217 7f9d7e96a700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:46:54.108221 7f9d7e96a700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:54.108233 7f9d7e96a700 15 _write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 3137536~516096
2014-06-12 14:46:54.108404 7f9d7e96a700 10 lookup_strip_header done  cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:54.108436 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:54.110644 7f9d7e96a700 10 _generic_write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 3137536~516096 = 0
2014-06-12 14:46:54.110762 7f9d7e96a700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:54.110787 7f9d7e96a700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:54.110792 7f9d7e96a700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:54.110800 7f9d7e96a700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:54.111040 7f9d7e96a700 20 set_header setting 17388 cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 parent seq 0
2014-06-12 14:46:54.111142 7f9d7e96a700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:46:54.111159 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.111168 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:54.113404 7f9d7e96a700 10 _do_op 0x13c50d20 seq 2304 r = 0, finisher 0x69e7b20 0
2014-06-12 14:46:54.113424 7f9d7e96a700 10 _finish_op 0x13c50d20 seq 2304 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:54.113434 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:46:54.113441 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.113447 7f9d7715b700  5 -- op tracker -- , seq: 748143, time: 2014-06-12 14:46:54.113433, event: commit_sent, op: osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.113457 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2471 (0'0,173'2471] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2469 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:54.113488 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:54.113536 7f9d7795c700  5 -- op tracker -- , seq: 748143, time: 2014-06-12 14:46:54.113529, event: sub_op_applied, op: osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.113548 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2471 (0'0,173'2471] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active] sub_op_modify_applied on 0x12c25a40 op osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.113563 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2471 (0'0,173'2471] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active] op_applied on primary on version 173'2471
2014-06-12 14:46:54.113570 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2471 (0'0,173'2471] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active] op_applied on replica on version 173'2471
2014-06-12 14:46:54.113596 7f9d7795c700  5 -- op tracker -- , seq: 748143, time: 2014-06-12 14:46:54.113593, event: done, op: osd_sub_op(client.4294.0:143630 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2471 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.155126 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:54.201565 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:54.201645 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:54.202715 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.212071 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.223446 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:54.223475 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:54.245014 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.250082 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.270448 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:54.271033 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:54.311316 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.322352 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.332907 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.344503 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.358164 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.369631 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:54.375900 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.434088 7f9d494c9700  5 -- op tracker -- , seq: 748144, time: 2014-06-12 14:46:54.429739, event: header_read, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.434101 7f9d494c9700  5 -- op tracker -- , seq: 748144, time: 2014-06-12 14:46:54.429744, event: throttled, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.434109 7f9d494c9700  5 -- op tracker -- , seq: 748144, time: 2014-06-12 14:46:54.434040, event: all_read, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.434117 7f9d494c9700  5 -- op tracker -- , seq: 748144, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.434126 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:46:54.434157 7f9d494c9700 15 osd.2 173 enqueue_op 0x17c2ac00 prio 63 cost 1548288 latency 0.004417 osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:54.434194 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.434203 7f9d63bd8700 10 osd.2 173 dequeue_op 0x17c2ac00 prio 63 cost 1548288 latency 0.004464 osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]
2014-06-12 14:46:54.434230 7f9d63bd8700  5 -- op tracker -- , seq: 748144, time: 2014-06-12 14:46:54.434225, event: reached_pg, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.434241 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:54.434256 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] handle_message: 0x17c2ac00
2014-06-12 14:46:54.434267 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_op osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:54.434292 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] get_object_context: found obc in cache: 0x244de000
2014-06-12 14:46:54.434303 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] get_object_context: 0x244de000 d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 rwstate(write n=1 w=0) oi: d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3(173'2513 client.4290.0:143736 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2513) ssc: 0x679b420 snapset: 0=[]:[]+head
2014-06-12 14:46:54.434319 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] find_object_context d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 @head oi=d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3(173'2513 client.4290.0:143736 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2513)
2014-06-12 14:46:54.434352 7f9d63bd8700  5 -- op tracker -- , seq: 748144, time: 2014-06-12 14:46:54.434347, event: started, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.434360 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] execute_ctx 0x7ad0c00
2014-06-12 14:46:54.434378 7f9d63bd8700  5 -- op tracker -- , seq: 748144, time: 2014-06-12 14:46:54.434373, event: started, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.434386 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_op d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] ov 173'2513 av 173'2514 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:54.434400 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288]
2014-06-12 14:46:54.434411 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:54.434424 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op  write 98304~1548288
2014-06-12 14:46:54.434440 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:46:54.434451 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] make_writeable d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 snapset=0x679b458  snapc=0=[]
2014-06-12 14:46:54.434464 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] make_writeable d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:54.434474 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] finish_ctx d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 0x7ad0c00 op modify  
2014-06-12 14:46:54.434487 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  set mtime to 2014-06-12 08:45:56.807165
2014-06-12 14:46:54.434511 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  final snapset 0=[]:[]+head in d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:54.434530 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  zeroing write result code 0
2014-06-12 14:46:54.434541 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] new_repop rep_tid 236356 on osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:54.434557 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] issue_repop rep_tid 236356 o d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:54.434583 7f9d63bd8700  5 -- op tracker -- , seq: 748144, time: 2014-06-12 14:46:54.434577, event: waiting for subops from 26, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.434603 7f9d63bd8700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:54.434623 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2513 (0'0,173'2513] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] append_log log((0'0,173'2513], crt=0'0) [173'2514 (173'2513) modify   d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 by client.4290.0:143737 2014-06-12 08:45:56.807165]
2014-06-12 14:46:54.434646 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] add_log_entry 173'2514 (173'2513) modify   d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 by client.4290.0:143737 2014-06-12 08:45:56.807165
2014-06-12 14:46:54.434664 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] append_log  adding 1 keys
2014-06-12 14:46:54.434684 7f9d5501e700  5 -- op tracker -- , seq: 748145, time: 2014-06-12 14:46:54.430456, event: header_read, op: osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.434694 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2514, trimmed: 
2014-06-12 14:46:54.434701 7f9d5501e700  5 -- op tracker -- , seq: 748145, time: 2014-06-12 14:46:54.430458, event: throttled, op: osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.434710 7f9d5501e700  5 -- op tracker -- , seq: 748145, time: 2014-06-12 14:46:54.434659, event: all_read, op: osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.434717 7f9d63bd8700  5 queue_transactions existing osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:54.434721 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.434723 7f9d63bd8700  5 queue_op 0x67e44e0 seq 2330 osr(3.7ae 0x49b4010) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:46:54.434718 7f9d5501e700  5 -- op tracker -- , seq: 748145, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.434728 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.434735 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:54.434741 7f9d5501e700 15 osd.2 173 enqueue_op 0x124e2500 prio 127 cost 1032999 latency 0.004285 osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.434731 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] eval_repop repgather(0x13b429c0 173'2514 rep_tid=236356 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:54.434751 7f9d63bd8700 10 osd.2 173 dequeue_op 0x17c2ac00 finish
2014-06-12 14:46:54.434756 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.434761 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.434768 7f9d63bd8700 10 osd.2 173 dequeue_op 0x124e2500 prio 127 cost 1032999 latency 0.004312 osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2782 (0'0,173'2782] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2781 active]
2014-06-12 14:46:54.434787 7f9d63bd8700  5 -- op tracker -- , seq: 748145, time: 2014-06-12 14:46:54.434783, event: reached_pg, op: osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.434795 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2782 (0'0,173'2782] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2781 active] handle_message: 0x124e2500
2014-06-12 14:46:54.434806 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2782 (0'0,173'2782] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2781 active] sub_op_modify trans 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 v 173'2783 (transaction) 174
2014-06-12 14:46:54.434827 7f9d63bd8700  5 -- op tracker -- , seq: 748145, time: 2014-06-12 14:46:54.434823, event: started, op: osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.434843 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2782 (0'0,173'2782] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2781 active] append_log log((0'0,173'2782], crt=0'0) [173'2783 (173'2782) modify   1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 by client.4305.0:142478 2014-06-12 08:45:56.804299]
2014-06-12 14:46:54.434861 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.434867 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2783 (0'0,173'2783] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2781 active] add_log_entry 173'2783 (173'2782) modify   1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 by client.4305.0:142478 2014-06-12 08:45:56.804299
2014-06-12 14:46:54.434888 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2783 (0'0,173'2783] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2781 active] append_log  adding 1 keys
2014-06-12 14:46:54.434918 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2783, trimmed: 
2014-06-12 14:46:54.434939 7f9d63bd8700  5 -- op tracker -- , seq: 748145, time: 2014-06-12 14:46:54.434934, event: started, op: osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.434948 7f9d63bd8700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:54.434951 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.434952 7f9d63bd8700  5 queue_op 0x67e5d40 seq 2520 osr(3.6d1 0x276df10) 1034485 bytes   (queue has 2 ops and 2585066 bytes)
2014-06-12 14:46:54.434958 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2783 (0'0,173'2783] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2781 active] do_sub_op osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.434972 7f9d63bd8700 10 osd.2 173 dequeue_op 0x124e2500 finish
2014-06-12 14:46:54.434976 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.434978 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.434997 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1c0b440 (1 active)
2014-06-12 14:46:54.435003 7f9d7d968700  5 _do_op 0x67e44e0 seq 2330 osr(3.7ae 0x49b4010)/0x49b4010 start
2014-06-12 14:46:54.435007 7f9d7d968700 10 _do_transaction on 0x384e3ec0
2014-06-12 14:46:54.435014 7f9d7d968700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:54.435138 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:54.435169 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.435215 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.435308 7f9d7d968700 15 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:54.435315 7f9d7d968700 10 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1 r = 0
2014-06-12 14:46:54.435321 7f9d7d968700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:54.435340 7f9d7d968700 15 _write 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 98304~1548288
2014-06-12 14:46:54.435484 7f9d7d968700 10 lookup_strip_header done  cid 3.7ae_head oid d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:54.435682 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:54.436829 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (2 active)
2014-06-12 14:46:54.436837 7f9d7c165700  5 _do_op 0x67e5d40 seq 2520 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:54.436842 7f9d7c165700 10 _do_transaction on 0x591c7f0
2014-06-12 14:46:54.436847 7f9d7c165700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.436914 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.437021 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.441935 7f9d7d968700 10 _generic_write 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 98304~1548288 = 0
2014-06-12 14:46:54.442073 7f9d7d968700 15 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:54.442095 7f9d7d968700 10 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:54.442102 7f9d7d968700 15 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:54.442110 7f9d7d968700 10 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:54.442486 7f9d7d968700 20 set_header setting 15929 cid 3.7ae_head oid d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 parent seq 0
2014-06-12 14:46:54.442634 7f9d7d968700 20 set_header setting 2150 cid meta oid 1fe77416/pglog_3.7ae/0//-1 parent seq 0
2014-06-12 14:46:54.442651 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.442661 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:54.450055 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.450088 7f9d7c165700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.450094 7f9d7c165700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:54.450100 7f9d7c165700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.450113 7f9d7c165700 15 _write 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 299008~1032192
2014-06-12 14:46:54.450362 7f9d7c165700 10 lookup_strip_header done  cid 3.6d1_head oid 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.450462 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:54.450561 7f9d7d968700 10 _do_op 0x67e44e0 seq 2330 r = 0, finisher 0x772ab00 0x465a980
2014-06-12 14:46:54.450575 7f9d7d968700 10 _finish_op 0x67e44e0 seq 2330 osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:54.450592 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_applied: 236356
2014-06-12 14:46:54.450621 7f9d7795c700  5 -- op tracker -- , seq: 748144, time: 2014-06-12 14:46:54.450612, event: op_applied, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.450633 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_applied on primary on version 173'2514
2014-06-12 14:46:54.450657 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1c0b440 (1 active)
2014-06-12 14:46:54.450664 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.450679 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_commit: 236356
2014-06-12 14:46:54.450700 7f9d7715b700  5 -- op tracker -- , seq: 748144, time: 2014-06-12 14:46:54.450694, event: op_commit, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.452963 7f9d7c165700 10 _generic_write 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 299008~1032192 = 0
2014-06-12 14:46:54.453032 7f9d7c165700 15 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.453049 7f9d7c165700 10 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 = 0
2014-06-12 14:46:54.453056 7f9d7c165700 15 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.453061 7f9d7c165700 10 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 = 0
2014-06-12 14:46:54.453282 7f9d7c165700 20 set_header setting 16429 cid 3.6d1_head oid 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 parent seq 0
2014-06-12 14:46:54.453357 7f9d7c165700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:54.453370 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.453375 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:54.458072 7f9d7c165700 10 _do_op 0x67e5d40 seq 2520 r = 0, finisher 0x17727f20 0
2014-06-12 14:46:54.458098 7f9d7c165700 10 _finish_op 0x67e5d40 seq 2520 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:54.458105 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:54.458111 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.458120 7f9d7715b700  5 -- op tracker -- , seq: 748145, time: 2014-06-12 14:46:54.458108, event: commit_sent, op: osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.458130 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2783 (0'0,173'2783] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2781 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:54.458159 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:54.458196 7f9d7795c700  5 -- op tracker -- , seq: 748145, time: 2014-06-12 14:46:54.458190, event: sub_op_applied, op: osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.458205 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2783 (0'0,173'2783] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2782 active] sub_op_modify_applied on 0x591c700 op osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.458223 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2783 (0'0,173'2783] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2782 active] op_applied on primary on version 173'2783
2014-06-12 14:46:54.458233 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2783 (0'0,173'2783] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2782 active] op_applied on replica on version 173'2783
2014-06-12 14:46:54.458257 7f9d7795c700  5 -- op tracker -- , seq: 748145, time: 2014-06-12 14:46:54.458253, event: done, op: osd_sub_op(client.4305.0:142478 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2783 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.465258 7f9d5501e700  5 -- op tracker -- , seq: 748146, time: 2014-06-12 14:46:54.464174, event: header_read, op: osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.465268 7f9d5501e700  5 -- op tracker -- , seq: 748146, time: 2014-06-12 14:46:54.464176, event: throttled, op: osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.465284 7f9d5501e700  5 -- op tracker -- , seq: 748146, time: 2014-06-12 14:46:54.465229, event: all_read, op: osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.465289 7f9d5501e700  5 -- op tracker -- , seq: 748146, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.465293 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.465298 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:54.465303 7f9d5501e700 15 osd.2 173 enqueue_op 0x124e2a00 prio 127 cost 520999 latency 0.001129 osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.465328 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.465337 7f9d633d7700 10 osd.2 173 dequeue_op 0x124e2a00 prio 127 cost 520999 latency 0.001162 osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2783 (0'0,173'2783] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2782 active]
2014-06-12 14:46:54.465361 7f9d633d7700  5 -- op tracker -- , seq: 748146, time: 2014-06-12 14:46:54.465356, event: reached_pg, op: osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.465371 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2783 (0'0,173'2783] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2782 active] handle_message: 0x124e2a00
2014-06-12 14:46:54.465386 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2783 (0'0,173'2783] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2782 active] sub_op_modify trans 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 v 173'2784 (transaction) 174
2014-06-12 14:46:54.465408 7f9d633d7700  5 -- op tracker -- , seq: 748146, time: 2014-06-12 14:46:54.465404, event: started, op: osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.465424 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2783 (0'0,173'2783] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2782 active] append_log log((0'0,173'2783], crt=0'0) [173'2784 (173'2783) modify   1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 by client.4305.0:142479 2014-06-12 08:45:56.840352]
2014-06-12 14:46:54.465449 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2784 (0'0,173'2784] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2782 active] add_log_entry 173'2784 (173'2783) modify   1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 by client.4305.0:142479 2014-06-12 08:45:56.840352
2014-06-12 14:46:54.465472 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2784 (0'0,173'2784] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2782 active] append_log  adding 1 keys
2014-06-12 14:46:54.465507 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2784, trimmed: 
2014-06-12 14:46:54.465535 7f9d633d7700  5 -- op tracker -- , seq: 748146, time: 2014-06-12 14:46:54.465530, event: started, op: osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.465547 7f9d633d7700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:54.465551 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.465553 7f9d633d7700  5 queue_op 0x2847b8c0 seq 2521 osr(3.6d1 0x276df10) 522485 bytes   (queue has 1 ops and 522485 bytes)
2014-06-12 14:46:54.465561 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (1 active)
2014-06-12 14:46:54.465566 7f9d8096e700  5 _do_op 0x2847b8c0 seq 2521 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:54.465569 7f9d8096e700 10 _do_transaction on 0x7d46630
2014-06-12 14:46:54.465559 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2784 (0'0,173'2784] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2782 active] do_sub_op osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.465573 7f9d8096e700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.465574 7f9d633d7700 10 osd.2 173 dequeue_op 0x124e2a00 finish
2014-06-12 14:46:54.465578 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.465580 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.465685 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.465708 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.465740 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.465757 7f9d8096e700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.465762 7f9d8096e700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:54.465765 7f9d8096e700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.465857 7f9d8096e700 15 _write 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 1331200~520192
2014-06-12 14:46:54.465952 7f9d8096e700 10 lookup_strip_header done  cid 3.6d1_head oid 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.466075 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:54.467310 7f9d8096e700 10 _generic_write 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 1331200~520192 = 0
2014-06-12 14:46:54.467343 7f9d8096e700 15 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.467351 7f9d8096e700 10 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 = 0
2014-06-12 14:46:54.467355 7f9d8096e700 15 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.467360 7f9d8096e700 10 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 = 0
2014-06-12 14:46:54.467508 7f9d8096e700 20 set_header setting 16429 cid 3.6d1_head oid 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 parent seq 0
2014-06-12 14:46:54.467568 7f9d8096e700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:54.467580 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.467585 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:54.469370 7f9d8096e700 10 _do_op 0x2847b8c0 seq 2521 r = 0, finisher 0x63d1120 0
2014-06-12 14:46:54.469385 7f9d8096e700 10 _finish_op 0x2847b8c0 seq 2521 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:54.469391 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:54.469398 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.469406 7f9d7715b700  5 -- op tracker -- , seq: 748146, time: 2014-06-12 14:46:54.469396, event: commit_sent, op: osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.469418 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2784 (0'0,173'2784] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2782 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:54.469446 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:54.469482 7f9d7795c700  5 -- op tracker -- , seq: 748146, time: 2014-06-12 14:46:54.469475, event: sub_op_applied, op: osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.469492 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2784 (0'0,173'2784] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2783 active] sub_op_modify_applied on 0x7d46540 op osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.469513 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2784 (0'0,173'2784] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2783 active] op_applied on primary on version 173'2784
2014-06-12 14:46:54.469520 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2784 (0'0,173'2784] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2783 active] op_applied on replica on version 173'2784
2014-06-12 14:46:54.469536 7f9d7795c700  5 -- op tracker -- , seq: 748146, time: 2014-06-12 14:46:54.469533, event: done, op: osd_sub_op(client.4305.0:142479 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2784 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.479801 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:54.479823 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:54.479834 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:54.479836 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:54.479837 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:54.479839 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 20; oldest is 21.318494 seconds old
2014-06-12 14:46:54.481819 7f9d5501e700  5 -- op tracker -- , seq: 748147, time: 2014-06-12 14:46:54.479857, event: header_read, op: osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.481840 7f9d5501e700  5 -- op tracker -- , seq: 748147, time: 2014-06-12 14:46:54.479860, event: throttled, op: osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.481856 7f9d5501e700  5 -- op tracker -- , seq: 748147, time: 2014-06-12 14:46:54.481782, event: all_read, op: osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.481865 7f9d5501e700  5 -- op tracker -- , seq: 748147, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.481870 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.481882 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:54.481891 7f9d5501e700 15 osd.2 173 enqueue_op 0x329eba00 prio 127 cost 1032999 latency 0.002033 osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.481924 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.481935 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329eba00 prio 127 cost 1032999 latency 0.002078 osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2784 (0'0,173'2784] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2783 active]
2014-06-12 14:46:54.481962 7f9d61bd4700  5 -- op tracker -- , seq: 748147, time: 2014-06-12 14:46:54.481957, event: reached_pg, op: osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.481971 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2784 (0'0,173'2784] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2783 active] handle_message: 0x329eba00
2014-06-12 14:46:54.481984 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2784 (0'0,173'2784] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2783 active] sub_op_modify trans 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 v 173'2785 (transaction) 174
2014-06-12 14:46:54.482001 7f9d61bd4700  5 -- op tracker -- , seq: 748147, time: 2014-06-12 14:46:54.481997, event: started, op: osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.482022 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2784 (0'0,173'2784] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2783 active] append_log log((0'0,173'2784], crt=0'0) [173'2785 (173'2784) modify   1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 by client.4305.0:142480 2014-06-12 08:45:56.853776]
2014-06-12 14:46:54.482045 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2785 (0'0,173'2785] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2783 active] add_log_entry 173'2785 (173'2784) modify   1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 by client.4305.0:142480 2014-06-12 08:45:56.853776
2014-06-12 14:46:54.482068 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2785 (0'0,173'2785] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2783 active] append_log  adding 1 keys
2014-06-12 14:46:54.482102 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2785, trimmed: 
2014-06-12 14:46:54.482127 7f9d61bd4700  5 -- op tracker -- , seq: 748147, time: 2014-06-12 14:46:54.482123, event: started, op: osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.482141 7f9d61bd4700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:54.482145 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.482147 7f9d61bd4700  5 queue_op 0x58f8840 seq 2522 osr(3.6d1 0x276df10) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:54.482154 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2785 (0'0,173'2785] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2783 active] do_sub_op osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.482176 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329eba00 finish
2014-06-12 14:46:54.482180 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.482183 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.482187 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (1 active)
2014-06-12 14:46:54.482195 7f9d7c966700  5 _do_op 0x58f8840 seq 2522 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:54.482199 7f9d7c966700 10 _do_transaction on 0x609feb0
2014-06-12 14:46:54.482204 7f9d7c966700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.482305 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.482324 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.482356 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.482368 7f9d7c966700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.482373 7f9d7c966700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:54.482377 7f9d7c966700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.482385 7f9d7c966700 15 _write 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 1851392~1032192
2014-06-12 14:46:54.482471 7f9d7c966700 10 lookup_strip_header done  cid 3.6d1_head oid 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.482544 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:54.485314 7f9d7c966700 10 _generic_write 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 1851392~1032192 = 0
2014-06-12 14:46:54.485397 7f9d7c966700 15 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.485419 7f9d7c966700 10 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 = 0
2014-06-12 14:46:54.485426 7f9d7c966700 15 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.485433 7f9d7c966700 10 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 = 0
2014-06-12 14:46:54.485719 7f9d7c966700 20 set_header setting 16429 cid 3.6d1_head oid 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 parent seq 0
2014-06-12 14:46:54.485820 7f9d7c966700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:54.485835 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.485841 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:54.491212 7f9d7c966700 10 _do_op 0x58f8840 seq 2522 r = 0, finisher 0x6ce6820 0
2014-06-12 14:46:54.491227 7f9d7c966700 10 _finish_op 0x58f8840 seq 2522 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:54.491232 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:54.491238 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.491245 7f9d7715b700  5 -- op tracker -- , seq: 748147, time: 2014-06-12 14:46:54.491235, event: commit_sent, op: osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.491254 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2785 (0'0,173'2785] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2783 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:54.491282 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:54.491316 7f9d7795c700  5 -- op tracker -- , seq: 748147, time: 2014-06-12 14:46:54.491309, event: sub_op_applied, op: osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.491326 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2785 (0'0,173'2785] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2784 active] sub_op_modify_applied on 0x609fdc0 op osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.491344 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2785 (0'0,173'2785] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2784 active] op_applied on primary on version 173'2785
2014-06-12 14:46:54.491356 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2785 (0'0,173'2785] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2784 active] op_applied on replica on version 173'2785
2014-06-12 14:46:54.491382 7f9d7795c700  5 -- op tracker -- , seq: 748147, time: 2014-06-12 14:46:54.491378, event: done, op: osd_sub_op(client.4305.0:142480 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2785 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.501901 7f9d5501e700  5 -- op tracker -- , seq: 748148, time: 2014-06-12 14:46:54.500011, event: header_read, op: osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.501913 7f9d5501e700  5 -- op tracker -- , seq: 748148, time: 2014-06-12 14:46:54.500013, event: throttled, op: osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.501921 7f9d5501e700  5 -- op tracker -- , seq: 748148, time: 2014-06-12 14:46:54.501856, event: all_read, op: osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.501928 7f9d5501e700  5 -- op tracker -- , seq: 748148, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.501933 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.501939 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:54.501946 7f9d5501e700 15 osd.2 173 enqueue_op 0x329ebf00 prio 127 cost 1032999 latency 0.001935 osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.501975 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.501985 7f9d67be0700 10 osd.2 173 dequeue_op 0x329ebf00 prio 127 cost 1032999 latency 0.001974 osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2785 (0'0,173'2785] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2784 active]
2014-06-12 14:46:54.502022 7f9d67be0700  5 -- op tracker -- , seq: 748148, time: 2014-06-12 14:46:54.502010, event: reached_pg, op: osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.502034 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2785 (0'0,173'2785] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2784 active] handle_message: 0x329ebf00
2014-06-12 14:46:54.502048 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2785 (0'0,173'2785] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2784 active] sub_op_modify trans 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 v 173'2786 (transaction) 174
2014-06-12 14:46:54.502069 7f9d67be0700  5 -- op tracker -- , seq: 748148, time: 2014-06-12 14:46:54.502064, event: started, op: osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.502085 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2785 (0'0,173'2785] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2784 active] append_log log((0'0,173'2785], crt=0'0) [173'2786 (173'2785) modify   1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 by client.4305.0:142481 2014-06-12 08:45:56.873810]
2014-06-12 14:46:54.502107 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2786 (0'0,173'2786] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2784 active] add_log_entry 173'2786 (173'2785) modify   1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 by client.4305.0:142481 2014-06-12 08:45:56.873810
2014-06-12 14:46:54.502131 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2786 (0'0,173'2786] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2784 active] append_log  adding 1 keys
2014-06-12 14:46:54.502168 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2786, trimmed: 
2014-06-12 14:46:54.502194 7f9d67be0700  5 -- op tracker -- , seq: 748148, time: 2014-06-12 14:46:54.502189, event: started, op: osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.502204 7f9d67be0700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:54.502209 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.502211 7f9d67be0700  5 queue_op 0x2273560 seq 2523 osr(3.6d1 0x276df10) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:46:54.502219 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2786 (0'0,173'2786] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2784 active] do_sub_op osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.502234 7f9d67be0700 10 osd.2 173 dequeue_op 0x329ebf00 finish
2014-06-12 14:46:54.502238 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.502241 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.502250 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (1 active)
2014-06-12 14:46:54.502255 7f9d81970700  5 _do_op 0x2273560 seq 2523 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:54.502260 7f9d81970700 10 _do_transaction on 0x52c8d30
2014-06-12 14:46:54.502266 7f9d81970700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.502383 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.502413 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.502455 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.502581 7f9d81970700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.502590 7f9d81970700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:54.502597 7f9d81970700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.502618 7f9d81970700 15 _write 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 2883584~1032192
2014-06-12 14:46:54.502723 7f9d81970700 10 lookup_strip_header done  cid 3.6d1_head oid 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.502871 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:54.506570 7f9d81970700 10 _generic_write 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 2883584~1032192 = 0
2014-06-12 14:46:54.506662 7f9d81970700 15 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.506677 7f9d81970700 10 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 = 0
2014-06-12 14:46:54.506688 7f9d81970700 15 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.506695 7f9d81970700 10 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 = 0
2014-06-12 14:46:54.506981 7f9d81970700 20 set_header setting 16429 cid 3.6d1_head oid 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 parent seq 0
2014-06-12 14:46:54.507101 7f9d81970700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:54.507122 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.507130 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:54.512083 7f9d81970700 10 _do_op 0x2273560 seq 2523 r = 0, finisher 0x6d50260 0
2014-06-12 14:46:54.512846 7f9d7715b700  5 -- op tracker -- , seq: 748148, time: 2014-06-12 14:46:54.512819, event: commit_sent, op: osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.512889 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2786 (0'0,173'2786] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2784 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:54.513031 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:54.513089 7f9d81970700 10 _finish_op 0x2273560 seq 2523 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:54.513114 7f9d7795c700  5 -- op tracker -- , seq: 748148, time: 2014-06-12 14:46:54.513108, event: sub_op_applied, op: osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.513122 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2786 (0'0,173'2786] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active] sub_op_modify_applied on 0x52c8c40 op osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.513170 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2786 (0'0,173'2786] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active] op_applied on primary on version 173'2786
2014-06-12 14:46:54.513184 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2786 (0'0,173'2786] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active] op_applied on replica on version 173'2786
2014-06-12 14:46:54.513269 7f9d81970700  5 -- op tracker -- , seq: 748148, time: 2014-06-12 14:46:54.513216, event: done, op: osd_sub_op(client.4305.0:142481 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2786 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.513317 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:54.513323 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.519120 7f9d5a16f700  5 -- op tracker -- , seq: 748149, time: 2014-06-12 14:46:54.518590, event: header_read, op: osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.519135 7f9d5a16f700  5 -- op tracker -- , seq: 748149, time: 2014-06-12 14:46:54.518594, event: throttled, op: osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.519143 7f9d5a16f700  5 -- op tracker -- , seq: 748149, time: 2014-06-12 14:46:54.519062, event: all_read, op: osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.519152 7f9d5a16f700  5 -- op tracker -- , seq: 748149, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.519160 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.519169 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:54.519177 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2911d00 prio 127 cost 516903 latency 0.000587 osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.519209 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.519218 7f9d673df700 10 osd.2 173 dequeue_op 0x2911d00 prio 127 cost 516903 latency 0.000628 osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2505 (0'0,173'2505] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2504 active]
2014-06-12 14:46:54.519261 7f9d673df700  5 -- op tracker -- , seq: 748149, time: 2014-06-12 14:46:54.519257, event: reached_pg, op: osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.519271 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2505 (0'0,173'2505] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2504 active] handle_message: 0x2911d00
2014-06-12 14:46:54.519285 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2505 (0'0,173'2505] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2504 active] sub_op_modify trans a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 v 173'2506 (transaction) 174
2014-06-12 14:46:54.519303 7f9d673df700  5 -- op tracker -- , seq: 748149, time: 2014-06-12 14:46:54.519298, event: started, op: osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.519321 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2505 (0'0,173'2505] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2504 active] append_log log((0'0,173'2505], crt=0'0) [173'2506 (173'2505) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146441 2014-06-12 08:41:37.025645]
2014-06-12 14:46:54.519346 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2506 (0'0,173'2506] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2504 active] add_log_entry 173'2506 (173'2505) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146441 2014-06-12 08:41:37.025645
2014-06-12 14:46:54.519389 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2506 (0'0,173'2506] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2504 active] append_log  adding 1 keys
2014-06-12 14:46:54.519430 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2506, trimmed: 
2014-06-12 14:46:54.519459 7f9d673df700  5 -- op tracker -- , seq: 748149, time: 2014-06-12 14:46:54.519454, event: started, op: osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.519473 7f9d673df700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:46:54.519502 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.519505 7f9d673df700  5 queue_op 0x6a7bc20 seq 2352 osr(3.168 0x276ce40) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:54.519517 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2506 (0'0,173'2506] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2504 active] do_sub_op osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.519533 7f9d673df700 10 osd.2 173 dequeue_op 0x2911d00 finish
2014-06-12 14:46:54.519536 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.519539 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.519549 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:46:54.519555 7f9d7a962700  5 _do_op 0x6a7bc20 seq 2352 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:46:54.519560 7f9d7a962700 10 _do_transaction on 0x1cef67f0
2014-06-12 14:46:54.519568 7f9d7a962700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:54.519747 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:54.519778 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.519830 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.519968 7f9d7a962700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:54.519975 7f9d7a962700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:46:54.519981 7f9d7a962700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:54.519987 7f9d5501e700  5 -- op tracker -- , seq: 748150, time: 2014-06-12 14:46:54.519482, event: header_read, op: osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.519998 7f9d5501e700  5 -- op tracker -- , seq: 748150, time: 2014-06-12 14:46:54.519484, event: throttled, op: osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.520000 7f9d7a962700 15 _write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 1064960~516096
2014-06-12 14:46:54.520007 7f9d5501e700  5 -- op tracker -- , seq: 748150, time: 2014-06-12 14:46:54.519953, event: all_read, op: osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.520015 7f9d5501e700  5 -- op tracker -- , seq: 748150, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.520020 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.520026 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:54.520032 7f9d5501e700 15 osd.2 173 enqueue_op 0x329ea200 prio 127 cost 279335 latency 0.000550 osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.520076 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.520082 7f9d663dd700 10 osd.2 173 dequeue_op 0x329ea200 prio 127 cost 279335 latency 0.000600 osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6d1( v 173'2786 (0'0,173'2786] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active]
2014-06-12 14:46:54.520104 7f9d663dd700  5 -- op tracker -- , seq: 748150, time: 2014-06-12 14:46:54.520099, event: reached_pg, op: osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.520111 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2786 (0'0,173'2786] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active] handle_message: 0x329ea200
2014-06-12 14:46:54.520122 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2786 (0'0,173'2786] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active] sub_op_modify trans 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 v 173'2787 (transaction) 174
2014-06-12 14:46:54.520137 7f9d663dd700  5 -- op tracker -- , seq: 748150, time: 2014-06-12 14:46:54.520133, event: started, op: osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.520164 7f9d7a962700 10 lookup_strip_header done  cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:46:54.520153 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2786 (0'0,173'2786] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active] append_log log((0'0,173'2786], crt=0'0) [173'2787 (173'2786) modify   1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 by client.4305.0:142482 2014-06-12 08:45:56.895418]
2014-06-12 14:46:54.520179 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2787 (0'0,173'2787] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active] add_log_entry 173'2787 (173'2786) modify   1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 by client.4305.0:142482 2014-06-12 08:45:56.895418
2014-06-12 14:46:54.520203 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2787 (0'0,173'2787] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active] append_log  adding 1 keys
2014-06-12 14:46:54.520231 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2787, trimmed: 
2014-06-12 14:46:54.520251 7f9d663dd700  5 -- op tracker -- , seq: 748150, time: 2014-06-12 14:46:54.520247, event: started, op: osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.520260 7f9d663dd700  5 queue_transactions existing osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:54.520263 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.520264 7f9d663dd700  5 queue_op 0x683f020 seq 2524 osr(3.6d1 0x276df10) 280821 bytes   (queue has 2 ops and 799210 bytes)
2014-06-12 14:46:54.520272 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2787 (0'0,173'2787] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active] do_sub_op osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.520284 7f9d663dd700 10 osd.2 173 dequeue_op 0x329ea200 finish
2014-06-12 14:46:54.520287 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.520289 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.520323 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:54.520433 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533c480 (2 active)
2014-06-12 14:46:54.520439 7f9d82972700  5 _do_op 0x683f020 seq 2524 osr(3.6d1 0x276df10)/0x276df10 start
2014-06-12 14:46:54.520443 7f9d82972700 10 _do_transaction on 0x66ca0f0
2014-06-12 14:46:54.520448 7f9d82972700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.520524 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.520544 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.522195 7f9d7a962700 10 _generic_write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 1064960~516096 = 0
2014-06-12 14:46:54.522249 7f9d7a962700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:46:54.522264 7f9d7a962700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:46:54.522272 7f9d7a962700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:46:54.522283 7f9d7a962700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:46:54.522426 7f9d7a962700 20 set_header setting 16934 cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 parent seq 0
2014-06-12 14:46:54.522479 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.522496 7f9d7a962700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:46:54.522508 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:54.523261 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:54.523291 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:54.524522 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.524560 7f9d82972700 15 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.524568 7f9d82972700 10 _omap_rmkeys meta/1fe794b5/pglog_3.6d1/0//-1 r = 0
2014-06-12 14:46:54.524572 7f9d7a962700 10 _do_op 0x6a7bc20 seq 2352 r = 0, finisher 0x9658ae0 0
2014-06-12 14:46:54.524575 7f9d82972700 15 _omap_setkeys meta/1fe794b5/pglog_3.6d1/0//-1
2014-06-12 14:46:54.524584 7f9d7a962700 10 _finish_op 0x6a7bc20 seq 2352 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:46:54.524595 7f9d82972700 15 _write 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 3915776~278528
2014-06-12 14:46:54.524592 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (1 active)
2014-06-12 14:46:54.524603 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.524603 7f9d7715b700  5 -- op tracker -- , seq: 748149, time: 2014-06-12 14:46:54.524592, event: commit_sent, op: osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.524613 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2506 (0'0,173'2506] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2504 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:54.524644 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:54.524678 7f9d7795c700  5 -- op tracker -- , seq: 748149, time: 2014-06-12 14:46:54.524671, event: sub_op_applied, op: osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.524686 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2506 (0'0,173'2506] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2505 active] sub_op_modify_applied on 0x1cef6700 op osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.524697 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2506 (0'0,173'2506] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2505 active] op_applied on primary on version 173'2506
2014-06-12 14:46:54.524704 7f9d82972700 10 lookup_strip_header done  cid 3.6d1_head oid 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.524703 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2506 (0'0,173'2506] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2505 active] op_applied on replica on version 173'2506
2014-06-12 14:46:54.524722 7f9d7795c700  5 -- op tracker -- , seq: 748149, time: 2014-06-12 14:46:54.524719, event: done, op: osd_sub_op(client.4316.0:146441 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2506 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.524920 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:54.525554 7f9d82972700 10 _generic_write 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 3915776~278528 = 0
2014-06-12 14:46:54.525578 7f9d82972700 15 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.525587 7f9d82972700 10 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 = 0
2014-06-12 14:46:54.525595 7f9d82972700 15 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3
2014-06-12 14:46:54.525600 7f9d82972700 10 _setattrs 3.6d1_head/1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 = 0
2014-06-12 14:46:54.525752 7f9d82972700 20 set_header setting 16429 cid 3.6d1_head oid 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 parent seq 0
2014-06-12 14:46:54.525807 7f9d82972700 20 set_header setting 2276 cid meta oid 1fe794b5/pglog_3.6d1/0//-1 parent seq 0
2014-06-12 14:46:54.525825 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.525836 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:54.527137 7f9d82972700 10 _do_op 0x683f020 seq 2524 r = 0, finisher 0x58d6ce0 0
2014-06-12 14:46:54.527155 7f9d82972700 10 _finish_op 0x683f020 seq 2524 osr(3.6d1 0x276df10)/0x276df10
2014-06-12 14:46:54.527163 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533c480 (0 active)
2014-06-12 14:46:54.527169 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.527185 7f9d7795c700  5 -- op tracker -- , seq: 748150, time: 2014-06-12 14:46:54.527173, event: sub_op_applied, op: osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.527197 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2787 (0'0,173'2787] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active] sub_op_modify_applied on 0x66ca000 op osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.527232 7f9d7795c700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:54.527258 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2787 (0'0,173'2787] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active] op_applied on primary on version 173'2787
2014-06-12 14:46:54.527269 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2787 (0'0,173'2787] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active] op_applied on replica on version 173'2787
2014-06-12 14:46:54.527295 7f9d7715b700  5 -- op tracker -- , seq: 748150, time: 2014-06-12 14:46:54.527288, event: commit_sent, op: osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.527306 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.6d1( v 173'2787 (0'0,173'2787] local-les=161 n=294 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2785 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:54.527338 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:54.527377 7f9d7715b700  5 -- op tracker -- , seq: 748150, time: 2014-06-12 14:46:54.527362, event: done, op: osd_sub_op(client.4305.0:142482 3.6d1 1fec36d1/rbd_data.10a86b8b4567.0000000000000ee1/head//3 [] v 173'2787 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.545058 7f9d5a16f700  5 -- op tracker -- , seq: 748151, time: 2014-06-12 14:46:54.544261, event: header_read, op: osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.545074 7f9d5a16f700  5 -- op tracker -- , seq: 748151, time: 2014-06-12 14:46:54.544264, event: throttled, op: osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.545084 7f9d5a16f700  5 -- op tracker -- , seq: 748151, time: 2014-06-12 14:46:54.545011, event: all_read, op: osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.545093 7f9d5a16f700  5 -- op tracker -- , seq: 748151, time: 0.000000, event: dispatched, op: osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.545099 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.545106 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:54.545115 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2910b00 prio 127 cost 516903 latency 0.000854 osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.545152 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.545162 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2910b00 prio 127 cost 516903 latency 0.000900 osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2724 (0'0,173'2724] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active]
2014-06-12 14:46:54.545188 7f9d65bdc700  5 -- op tracker -- , seq: 748151, time: 2014-06-12 14:46:54.545184, event: reached_pg, op: osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.545207 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2724 (0'0,173'2724] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] handle_message: 0x2910b00
2014-06-12 14:46:54.545222 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2724 (0'0,173'2724] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] sub_op_modify trans cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 v 173'2725 (transaction) 174
2014-06-12 14:46:54.545242 7f9d65bdc700  5 -- op tracker -- , seq: 748151, time: 2014-06-12 14:46:54.545238, event: started, op: osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.545261 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2724 (0'0,173'2724] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] append_log log((0'0,173'2724], crt=0'0) [173'2725 (173'2724) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143147 2014-06-12 08:43:41.611051]
2014-06-12 14:46:54.545289 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2725 (0'0,173'2725] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] add_log_entry 173'2725 (173'2724) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143147 2014-06-12 08:43:41.611051
2014-06-12 14:46:54.545314 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2725 (0'0,173'2725] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] append_log  adding 1 keys
2014-06-12 14:46:54.545351 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2725, trimmed: 
2014-06-12 14:46:54.545379 7f9d65bdc700  5 -- op tracker -- , seq: 748151, time: 2014-06-12 14:46:54.545374, event: started, op: osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.545392 7f9d65bdc700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:54.545398 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.545400 7f9d65bdc700  5 queue_op 0x1efba3c0 seq 2472 osr(3.21c 0x276d020) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:54.545411 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (1 active)
2014-06-12 14:46:54.545419 7f9d7895e700  5 _do_op 0x1efba3c0 seq 2472 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:46:54.545407 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2725 (0'0,173'2725] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] do_sub_op osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.545421 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2910b00 finish
2014-06-12 14:46:54.545423 7f9d7895e700 10 _do_transaction on 0x668d430
2014-06-12 14:46:54.545425 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.545427 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.545429 7f9d7895e700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:54.545643 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:54.545660 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.545695 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.545707 7f9d7895e700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:54.545711 7f9d7895e700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:46:54.545714 7f9d7895e700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:54.545723 7f9d7895e700 15 _write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 2703360~516096
2014-06-12 14:46:54.545814 7f9d7895e700 10 lookup_strip_header done  cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:54.545840 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:54.547153 7f9d7895e700 10 _generic_write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 2703360~516096 = 0
2014-06-12 14:46:54.547191 7f9d7895e700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:54.547204 7f9d7895e700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:54.547208 7f9d7895e700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:54.547213 7f9d7895e700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:54.547352 7f9d7895e700 20 set_header setting 16774 cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 parent seq 0
2014-06-12 14:46:54.547408 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.547418 7f9d7895e700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:46:54.547424 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:54.549027 7f9d7895e700 10 _do_op 0x1efba3c0 seq 2472 r = 0, finisher 0x4f601a0 0
2014-06-12 14:46:54.549039 7f9d7895e700 10 _finish_op 0x1efba3c0 seq 2472 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:54.549046 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (0 active)
2014-06-12 14:46:54.549049 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.549055 7f9d7715b700  5 -- op tracker -- , seq: 748151, time: 2014-06-12 14:46:54.549047, event: commit_sent, op: osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.549065 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2725 (0'0,173'2725] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] sub_op_modify_commit on op osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:54.549095 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:54.549132 7f9d7795c700  5 -- op tracker -- , seq: 748151, time: 2014-06-12 14:46:54.549124, event: sub_op_applied, op: osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.549142 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2725 (0'0,173'2725] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] sub_op_modify_applied on 0x668d340 op osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.549159 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2725 (0'0,173'2725] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] op_applied on primary on version 173'2725
2014-06-12 14:46:54.549167 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2725 (0'0,173'2725] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] op_applied on replica on version 173'2725
2014-06-12 14:46:54.549193 7f9d7795c700  5 -- op tracker -- , seq: 748151, time: 2014-06-12 14:46:54.549190, event: done, op: osd_sub_op(client.4279.0:143147 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.716494 7f9d5bec4700  5 -- op tracker -- , seq: 748152, time: 2014-06-12 14:46:54.715198, event: header_read, op: osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.716514 7f9d5bec4700  5 -- op tracker -- , seq: 748152, time: 2014-06-12 14:46:54.715201, event: throttled, op: osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.716522 7f9d5bec4700  5 -- op tracker -- , seq: 748152, time: 2014-06-12 14:46:54.716441, event: all_read, op: osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.716530 7f9d5bec4700  5 -- op tracker -- , seq: 748152, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.716538 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.716545 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:54.716554 7f9d5bec4700 15 osd.2 173 enqueue_op 0x876ec00 prio 127 cost 516903 latency 0.001356 osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.716594 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.716665 7f9d62bd6700 10 osd.2 173 dequeue_op 0x876ec00 prio 127 cost 516903 latency 0.001466 osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4ba( v 173'2929 (0'0,173'2929] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2928 active]
2014-06-12 14:46:54.716697 7f9d62bd6700  5 -- op tracker -- , seq: 748152, time: 2014-06-12 14:46:54.716692, event: reached_pg, op: osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.716711 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2929 (0'0,173'2929] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2928 active] handle_message: 0x876ec00
2014-06-12 14:46:54.716753 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2929 (0'0,173'2929] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2928 active] sub_op_modify trans 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 v 173'2930 (transaction) 174
2014-06-12 14:46:54.716776 7f9d62bd6700  5 -- op tracker -- , seq: 748152, time: 2014-06-12 14:46:54.716771, event: started, op: osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.716796 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2929 (0'0,173'2929] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2928 active] append_log log((0'0,173'2929], crt=0'0) [173'2930 (173'2929) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146613 2014-06-12 08:45:57.091599]
2014-06-12 14:46:54.716850 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2930 (0'0,173'2930] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2928 active] add_log_entry 173'2930 (173'2929) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146613 2014-06-12 08:45:57.091599
2014-06-12 14:46:54.716879 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2930 (0'0,173'2930] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2928 active] append_log  adding 1 keys
2014-06-12 14:46:54.716917 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2930, trimmed: 
2014-06-12 14:46:54.716951 7f9d62bd6700  5 -- op tracker -- , seq: 748152, time: 2014-06-12 14:46:54.716945, event: started, op: osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.716965 7f9d62bd6700  5 queue_transactions existing osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:54.716970 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.716971 7f9d62bd6700  5 queue_op 0x1c4a9080 seq 2654 osr(3.4ba 0x276d830) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:54.716979 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2930 (0'0,173'2930] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2928 active] do_sub_op osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.716994 7f9d62bd6700 10 osd.2 173 dequeue_op 0x876ec00 finish
2014-06-12 14:46:54.716998 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.717001 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.717009 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d560 (1 active)
2014-06-12 14:46:54.717015 7f9d83173700  5 _do_op 0x1c4a9080 seq 2654 osr(3.4ba 0x276d830)/0x276d830 start
2014-06-12 14:46:54.717019 7f9d83173700 10 _do_transaction on 0x286f3eb0
2014-06-12 14:46:54.717026 7f9d83173700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.717178 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.717207 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.717253 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.717358 7f9d83173700 15 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.717365 7f9d83173700 10 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1 r = 0
2014-06-12 14:46:54.717372 7f9d83173700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.717397 7f9d83173700 15 _write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 720896~516096
2014-06-12 14:46:54.717577 7f9d83173700 10 lookup_strip_header done  cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.717712 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:54.719711 7f9d83173700 10 _generic_write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 720896~516096 = 0
2014-06-12 14:46:54.719764 7f9d83173700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.719781 7f9d83173700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:54.719788 7f9d83173700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.719797 7f9d83173700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:54.720012 7f9d83173700 20 set_header setting 17403 cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 parent seq 0
2014-06-12 14:46:54.720093 7f9d83173700 20 set_header setting 2300 cid meta oid 1fe00356/pglog_3.4ba/0//-1 parent seq 0
2014-06-12 14:46:54.720109 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.720118 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:54.723241 7f9d83173700 10 _do_op 0x1c4a9080 seq 2654 r = 0, finisher 0x4efc280 0
2014-06-12 14:46:54.723264 7f9d83173700 10 _finish_op 0x1c4a9080 seq 2654 osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:54.723271 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d560 (0 active)
2014-06-12 14:46:54.723283 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.723280 7f9d7715b700  5 -- op tracker -- , seq: 748152, time: 2014-06-12 14:46:54.723271, event: commit_sent, op: osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.723293 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2930 (0'0,173'2930] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2928 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:54.723317 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:54.723350 7f9d7795c700  5 -- op tracker -- , seq: 748152, time: 2014-06-12 14:46:54.723342, event: sub_op_applied, op: osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.723357 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2930 (0'0,173'2930] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2929 active] sub_op_modify_applied on 0x286f3dc0 op osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.723374 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2930 (0'0,173'2930] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2929 active] op_applied on primary on version 173'2930
2014-06-12 14:46:54.723383 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2930 (0'0,173'2930] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2929 active] op_applied on replica on version 173'2930
2014-06-12 14:46:54.723407 7f9d7795c700  5 -- op tracker -- , seq: 748152, time: 2014-06-12 14:46:54.723402, event: done, op: osd_sub_op(client.4313.0:146613 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.723829 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19421 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,1,0,0,0,1,14,4])
2014-06-12 14:46:54.723844 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19421 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,1,0,0,0,1,14,4])
2014-06-12 14:46:54.724298 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:54.724353 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:54.724380 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:54.724509 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:54.724521 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:54.724535 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:54.725666 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:54.725688 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:54.725716 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:54.725735 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:54.725753 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:54.725776 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:54.725792 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:54.725816 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:54.725848 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:54.725864 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:54.725883 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:54.725903 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:54.725921 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:54.725939 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:54.725954 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:54.725970 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:54.729854 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:54.729877 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:54.734374 7f9d5bec4700  5 -- op tracker -- , seq: 748153, time: 2014-06-12 14:46:54.729737, event: header_read, op: osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.734387 7f9d5bec4700  5 -- op tracker -- , seq: 748153, time: 2014-06-12 14:46:54.729739, event: throttled, op: osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.734396 7f9d5bec4700  5 -- op tracker -- , seq: 748153, time: 2014-06-12 14:46:54.734327, event: all_read, op: osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.734405 7f9d5bec4700  5 -- op tracker -- , seq: 748153, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.734411 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.734417 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:54.734426 7f9d5bec4700 15 osd.2 173 enqueue_op 0x876e800 prio 127 cost 516903 latency 0.004688 osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.737647 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.737667 7f9d683e1700 10 osd.2 173 dequeue_op 0x876e800 prio 127 cost 516903 latency 0.007929 osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4ba( v 173'2930 (0'0,173'2930] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2929 active]
2014-06-12 14:46:54.737698 7f9d683e1700  5 -- op tracker -- , seq: 748153, time: 2014-06-12 14:46:54.737692, event: reached_pg, op: osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.737718 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2930 (0'0,173'2930] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2929 active] handle_message: 0x876e800
2014-06-12 14:46:54.737734 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2930 (0'0,173'2930] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2929 active] sub_op_modify trans 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 v 173'2931 (transaction) 174
2014-06-12 14:46:54.737753 7f9d683e1700  5 -- op tracker -- , seq: 748153, time: 2014-06-12 14:46:54.737748, event: started, op: osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.737772 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2930 (0'0,173'2930] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2929 active] append_log log((0'0,173'2930], crt=0'0) [173'2931 (173'2930) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146614 2014-06-12 08:45:57.105311]
2014-06-12 14:46:54.737800 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2931 (0'0,173'2931] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2929 active] add_log_entry 173'2931 (173'2930) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146614 2014-06-12 08:45:57.105311
2014-06-12 14:46:54.737824 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2931 (0'0,173'2931] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2929 active] append_log  adding 1 keys
2014-06-12 14:46:54.737863 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2931, trimmed: 
2014-06-12 14:46:54.737893 7f9d683e1700  5 -- op tracker -- , seq: 748153, time: 2014-06-12 14:46:54.737888, event: started, op: osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.737906 7f9d683e1700  5 queue_transactions existing osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:54.737913 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.737915 7f9d683e1700  5 queue_op 0x33612c0 seq 2655 osr(3.4ba 0x276d830) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:54.737927 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2931 (0'0,173'2931] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2929 active] do_sub_op osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.737944 7f9d683e1700 10 osd.2 173 dequeue_op 0x876e800 finish
2014-06-12 14:46:54.737947 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.737950 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.738193 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d560 (1 active)
2014-06-12 14:46:54.738202 7f9d7a161700  5 _do_op 0x33612c0 seq 2655 osr(3.4ba 0x276d830)/0x276d830 start
2014-06-12 14:46:54.738206 7f9d7a161700 10 _do_transaction on 0x62c09b0
2014-06-12 14:46:54.738212 7f9d7a161700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.738370 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.738387 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.738430 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.738454 7f9d7a161700 15 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.738459 7f9d7a161700 10 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1 r = 0
2014-06-12 14:46:54.738464 7f9d7a161700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.738580 7f9d7a161700 15 _write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 1236992~516096
2014-06-12 14:46:54.738733 7f9d7a161700 10 lookup_strip_header done  cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.738871 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:54.740847 7f9d7a161700 10 _generic_write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 1236992~516096 = 0
2014-06-12 14:46:54.741164 7f9d7a161700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.741178 7f9d7a161700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:54.741182 7f9d7a161700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.741187 7f9d7a161700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:54.741335 7f9d7a161700 20 set_header setting 17403 cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 parent seq 0
2014-06-12 14:46:54.741393 7f9d7a161700 20 set_header setting 2300 cid meta oid 1fe00356/pglog_3.4ba/0//-1 parent seq 0
2014-06-12 14:46:54.741403 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.741408 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:54.743302 7f9d7a161700 10 _do_op 0x33612c0 seq 2655 r = 0, finisher 0x7f57120 0
2014-06-12 14:46:54.743317 7f9d7a161700 10 _finish_op 0x33612c0 seq 2655 osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:54.743324 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d560 (0 active)
2014-06-12 14:46:54.743333 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.743332 7f9d7715b700  5 -- op tracker -- , seq: 748153, time: 2014-06-12 14:46:54.743324, event: commit_sent, op: osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.743352 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2931 (0'0,173'2931] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2929 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:54.743371 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:54.743401 7f9d7795c700  5 -- op tracker -- , seq: 748153, time: 2014-06-12 14:46:54.743394, event: sub_op_applied, op: osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.743412 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2931 (0'0,173'2931] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2930 active] sub_op_modify_applied on 0x62c08c0 op osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.743426 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2931 (0'0,173'2931] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2930 active] op_applied on primary on version 173'2931
2014-06-12 14:46:54.743433 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2931 (0'0,173'2931] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2930 active] op_applied on replica on version 173'2931
2014-06-12 14:46:54.743448 7f9d7795c700  5 -- op tracker -- , seq: 748153, time: 2014-06-12 14:46:54.743445, event: done, op: osd_sub_op(client.4313.0:146614 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.749551 7f9d5bec4700  5 -- op tracker -- , seq: 748154, time: 2014-06-12 14:46:54.748626, event: header_read, op: osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.749565 7f9d5bec4700  5 -- op tracker -- , seq: 748154, time: 2014-06-12 14:46:54.748628, event: throttled, op: osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.749573 7f9d5bec4700  5 -- op tracker -- , seq: 748154, time: 2014-06-12 14:46:54.749511, event: all_read, op: osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.749589 7f9d5bec4700  5 -- op tracker -- , seq: 748154, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.749595 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.749602 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:54.749610 7f9d5bec4700 15 osd.2 173 enqueue_op 0x876f900 prio 127 cost 516903 latency 0.000984 osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.749663 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.749670 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x876f900 prio 127 cost 516903 latency 0.001043 osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4ba( v 173'2931 (0'0,173'2931] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2930 active]
2014-06-12 14:46:54.749693 7f9d6b3e7700  5 -- op tracker -- , seq: 748154, time: 2014-06-12 14:46:54.749688, event: reached_pg, op: osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.749702 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2931 (0'0,173'2931] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2930 active] handle_message: 0x876f900
2014-06-12 14:46:54.749715 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2931 (0'0,173'2931] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2930 active] sub_op_modify trans 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 v 173'2932 (transaction) 174
2014-06-12 14:46:54.749731 7f9d6b3e7700  5 -- op tracker -- , seq: 748154, time: 2014-06-12 14:46:54.749727, event: started, op: osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.749746 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2931 (0'0,173'2931] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2930 active] append_log log((0'0,173'2931], crt=0'0) [173'2932 (173'2931) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146615 2014-06-12 08:45:57.125278]
2014-06-12 14:46:54.749771 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2932 (0'0,173'2932] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2930 active] add_log_entry 173'2932 (173'2931) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146615 2014-06-12 08:45:57.125278
2014-06-12 14:46:54.749793 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2932 (0'0,173'2932] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2930 active] append_log  adding 1 keys
2014-06-12 14:46:54.749828 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2932, trimmed: 
2014-06-12 14:46:54.749850 7f9d6b3e7700  5 -- op tracker -- , seq: 748154, time: 2014-06-12 14:46:54.749845, event: started, op: osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.749861 7f9d6b3e7700  5 queue_transactions existing osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:54.749865 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.749867 7f9d6b3e7700  5 queue_op 0x663e5a0 seq 2656 osr(3.4ba 0x276d830) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:54.749874 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2932 (0'0,173'2932] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2930 active] do_sub_op osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.749888 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x876f900 finish
2014-06-12 14:46:54.749892 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.749894 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.751362 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d560 (1 active)
2014-06-12 14:46:54.751377 7f9d7f16b700  5 _do_op 0x663e5a0 seq 2656 osr(3.4ba 0x276d830)/0x276d830 start
2014-06-12 14:46:54.751384 7f9d7f16b700 10 _do_transaction on 0x66be9b0
2014-06-12 14:46:54.751392 7f9d7f16b700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.751521 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.751546 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.751592 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.751619 7f9d7f16b700 15 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.751625 7f9d7f16b700 10 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1 r = 0
2014-06-12 14:46:54.751631 7f9d7f16b700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.751737 7f9d7f16b700 15 _write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 1753088~516096
2014-06-12 14:46:54.751896 7f9d7f16b700 10 lookup_strip_header done  cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.752019 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:54.753928 7f9d7f16b700 10 _generic_write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 1753088~516096 = 0
2014-06-12 14:46:54.753984 7f9d7f16b700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.754002 7f9d7f16b700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:54.754011 7f9d7f16b700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.754020 7f9d7f16b700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:54.754232 7f9d7f16b700 20 set_header setting 17403 cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 parent seq 0
2014-06-12 14:46:54.754313 7f9d7f16b700 20 set_header setting 2300 cid meta oid 1fe00356/pglog_3.4ba/0//-1 parent seq 0
2014-06-12 14:46:54.754331 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.754342 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:54.757171 7f9d7f16b700 10 _do_op 0x663e5a0 seq 2656 r = 0, finisher 0x80797e0 0
2014-06-12 14:46:54.757222 7f9d7715b700  5 -- op tracker -- , seq: 748154, time: 2014-06-12 14:46:54.757210, event: commit_sent, op: osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.757235 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2932 (0'0,173'2932] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2930 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:54.757265 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:54.757296 7f9d7f16b700 10 _finish_op 0x663e5a0 seq 2656 osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:54.757322 7f9d7795c700  5 -- op tracker -- , seq: 748154, time: 2014-06-12 14:46:54.757316, event: sub_op_applied, op: osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.757338 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2932 (0'0,173'2932] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2931 active] sub_op_modify_applied on 0x66be8c0 op osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.757353 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2932 (0'0,173'2932] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2931 active] op_applied on primary on version 173'2932
2014-06-12 14:46:54.757363 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2932 (0'0,173'2932] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2931 active] op_applied on replica on version 173'2932
2014-06-12 14:46:54.757403 7f9d7f16b700  5 -- op tracker -- , seq: 748154, time: 2014-06-12 14:46:54.757394, event: done, op: osd_sub_op(client.4313.0:146615 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.757434 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d560 (0 active)
2014-06-12 14:46:54.757438 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.766097 7f9d5bec4700  5 -- op tracker -- , seq: 748155, time: 2014-06-12 14:46:54.763097, event: header_read, op: osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.766113 7f9d5bec4700  5 -- op tracker -- , seq: 748155, time: 2014-06-12 14:46:54.763099, event: throttled, op: osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.766121 7f9d5bec4700  5 -- op tracker -- , seq: 748155, time: 2014-06-12 14:46:54.766050, event: all_read, op: osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.766130 7f9d5bec4700  5 -- op tracker -- , seq: 748155, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.766134 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.766140 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:54.766148 7f9d5bec4700 15 osd.2 173 enqueue_op 0x876fe00 prio 127 cost 516903 latency 0.003051 osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.766180 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.766186 7f9d693e3700 10 osd.2 173 dequeue_op 0x876fe00 prio 127 cost 516903 latency 0.003089 osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4ba( v 173'2932 (0'0,173'2932] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2931 active]
2014-06-12 14:46:54.766207 7f9d693e3700  5 -- op tracker -- , seq: 748155, time: 2014-06-12 14:46:54.766203, event: reached_pg, op: osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.766215 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2932 (0'0,173'2932] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2931 active] handle_message: 0x876fe00
2014-06-12 14:46:54.766227 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2932 (0'0,173'2932] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2931 active] sub_op_modify trans 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 v 173'2933 (transaction) 174
2014-06-12 14:46:54.766243 7f9d693e3700  5 -- op tracker -- , seq: 748155, time: 2014-06-12 14:46:54.766239, event: started, op: osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.766258 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2932 (0'0,173'2932] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2931 active] append_log log((0'0,173'2932], crt=0'0) [173'2933 (173'2932) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146616 2014-06-12 08:45:57.139829]
2014-06-12 14:46:54.766282 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2933 (0'0,173'2933] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2931 active] add_log_entry 173'2933 (173'2932) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146616 2014-06-12 08:45:57.139829
2014-06-12 14:46:54.766303 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2933 (0'0,173'2933] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2931 active] append_log  adding 1 keys
2014-06-12 14:46:54.766336 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2933, trimmed: 
2014-06-12 14:46:54.766361 7f9d693e3700  5 -- op tracker -- , seq: 748155, time: 2014-06-12 14:46:54.766356, event: started, op: osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.766370 7f9d693e3700  5 queue_transactions existing osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:54.766374 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.766376 7f9d693e3700  5 queue_op 0xbc5f0e0 seq 2657 osr(3.4ba 0x276d830) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:54.766389 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d560 (1 active)
2014-06-12 14:46:54.766398 7f9d7e169700  5 _do_op 0xbc5f0e0 seq 2657 osr(3.4ba 0x276d830)/0x276d830 start
2014-06-12 14:46:54.766384 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2933 (0'0,173'2933] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2931 active] do_sub_op osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.766403 7f9d7e169700 10 _do_transaction on 0x63a75f0
2014-06-12 14:46:54.766403 7f9d693e3700 10 osd.2 173 dequeue_op 0x876fe00 finish
2014-06-12 14:46:54.766407 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.766409 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.766409 7f9d7e169700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.766534 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.766555 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.766600 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.766626 7f9d7e169700 15 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.766632 7f9d7e169700 10 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1 r = 0
2014-06-12 14:46:54.766638 7f9d7e169700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.766766 7f9d7e169700 15 _write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 2269184~516096
2014-06-12 14:46:54.766925 7f9d7e169700 10 lookup_strip_header done  cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.767054 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:54.767064 7f9d53c11700  5 -- op tracker -- , seq: 748156, time: 2014-06-12 14:46:54.766739, event: header_read, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.767079 7f9d53c11700  5 -- op tracker -- , seq: 748156, time: 2014-06-12 14:46:54.766742, event: throttled, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.767090 7f9d53c11700  5 -- op tracker -- , seq: 748156, time: 2014-06-12 14:46:54.767031, event: all_read, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.767099 7f9d53c11700  5 -- op tracker -- , seq: 748156, time: 0.000000, event: dispatched, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.767106 7f9d53c11700 20 osd.2 173 should_share_map client.4300 172.16.96.220:0/1243739 173
2014-06-12 14:46:54.767133 7f9d53c11700 15 osd.2 173 enqueue_op 0x124e3b00 prio 63 cost 143360 latency 0.000394 osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:54.769035 7f9d7e169700 10 _generic_write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 2269184~516096 = 0
2014-06-12 14:46:54.769090 7f9d7e169700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.769106 7f9d7e169700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:54.769112 7f9d7e169700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.769120 7f9d7e169700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:54.769336 7f9d7e169700 20 set_header setting 17403 cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 parent seq 0
2014-06-12 14:46:54.769416 7f9d7e169700 20 set_header setting 2300 cid meta oid 1fe00356/pglog_3.4ba/0//-1 parent seq 0
2014-06-12 14:46:54.769432 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.769441 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:54.769563 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.769571 7f9d66bde700 10 osd.2 173 dequeue_op 0x124e3b00 prio 63 cost 143360 latency 0.002831 osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]
2014-06-12 14:46:54.769602 7f9d66bde700  5 -- op tracker -- , seq: 748156, time: 2014-06-12 14:46:54.769596, event: reached_pg, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.769622 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:54.769637 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] handle_message: 0x124e3b00
2014-06-12 14:46:54.769648 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_op osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:54.769672 7f9d66bde700 15 getattr 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 '_'
2014-06-12 14:46:54.770841 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:54.770938 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:54.773791 7f9d7e169700 10 _do_op 0xbc5f0e0 seq 2657 r = 0, finisher 0x583fc20 0
2014-06-12 14:46:54.773813 7f9d7e169700 10 _finish_op 0xbc5f0e0 seq 2657 osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:54.773821 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d560 (0 active)
2014-06-12 14:46:54.773830 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.773833 7f9d7715b700  5 -- op tracker -- , seq: 748155, time: 2014-06-12 14:46:54.773821, event: commit_sent, op: osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.773852 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2933 (0'0,173'2933] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2931 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:54.773880 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:54.773913 7f9d7795c700  5 -- op tracker -- , seq: 748155, time: 2014-06-12 14:46:54.773905, event: sub_op_applied, op: osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.773920 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2933 (0'0,173'2933] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active] sub_op_modify_applied on 0x63a7500 op osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.773937 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2933 (0'0,173'2933] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active] op_applied on primary on version 173'2933
2014-06-12 14:46:54.773947 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2933 (0'0,173'2933] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active] op_applied on replica on version 173'2933
2014-06-12 14:46:54.773970 7f9d7795c700  5 -- op tracker -- , seq: 748155, time: 2014-06-12 14:46:54.773966, event: done, op: osd_sub_op(client.4313.0:146616 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.781294 7f9d53c11700  5 -- op tracker -- , seq: 748157, time: 2014-06-12 14:46:54.780061, event: header_read, op: osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.781307 7f9d53c11700  5 -- op tracker -- , seq: 748157, time: 2014-06-12 14:46:54.780065, event: throttled, op: osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.781313 7f9d53c11700  5 -- op tracker -- , seq: 748157, time: 2014-06-12 14:46:54.781256, event: all_read, op: osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.781318 7f9d53c11700  5 -- op tracker -- , seq: 748157, time: 0.000000, event: dispatched, op: osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.781326 7f9d53c11700 20 osd.2 173 should_share_map client.4300 172.16.96.220:0/1243739 173
2014-06-12 14:46:54.781349 7f9d53c11700 15 osd.2 173 enqueue_op 0x124e2800 prio 63 cost 516096 latency 0.001288 osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:54.781377 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:54.781982 7f9d5bec4700  5 -- op tracker -- , seq: 748158, time: 2014-06-12 14:46:54.781031, event: header_read, op: osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.781996 7f9d5bec4700  5 -- op tracker -- , seq: 748158, time: 2014-06-12 14:46:54.781033, event: throttled, op: osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.782006 7f9d5bec4700  5 -- op tracker -- , seq: 748158, time: 2014-06-12 14:46:54.781952, event: all_read, op: osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.782018 7f9d5bec4700  5 -- op tracker -- , seq: 748158, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.782022 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.782030 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:54.782038 7f9d5bec4700 15 osd.2 173 enqueue_op 0x876e200 prio 127 cost 516903 latency 0.001006 osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.782063 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active)
2014-06-12 14:46:54.782070 7f9d6abe6700 10 osd.2 173 dequeue_op 0x876e200 prio 127 cost 516903 latency 0.001039 osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4ba( v 173'2933 (0'0,173'2933] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active]
2014-06-12 14:46:54.782089 7f9d6abe6700  5 -- op tracker -- , seq: 748158, time: 2014-06-12 14:46:54.782087, event: reached_pg, op: osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.782097 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2933 (0'0,173'2933] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active] handle_message: 0x876e200
2014-06-12 14:46:54.782107 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2933 (0'0,173'2933] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active] sub_op_modify trans 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 v 173'2934 (transaction) 174
2014-06-12 14:46:54.782124 7f9d6abe6700  5 -- op tracker -- , seq: 748158, time: 2014-06-12 14:46:54.782121, event: started, op: osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.782136 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2933 (0'0,173'2933] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active] append_log log((0'0,173'2933], crt=0'0) [173'2934 (173'2933) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146617 2014-06-12 08:45:57.157232]
2014-06-12 14:46:54.782155 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2934 (0'0,173'2934] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active] add_log_entry 173'2934 (173'2933) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146617 2014-06-12 08:45:57.157232
2014-06-12 14:46:54.782171 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2934 (0'0,173'2934] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active] append_log  adding 1 keys
2014-06-12 14:46:54.782201 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2934, trimmed: 
2014-06-12 14:46:54.782219 7f9d6abe6700  5 -- op tracker -- , seq: 748158, time: 2014-06-12 14:46:54.782216, event: started, op: osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.782228 7f9d6abe6700  5 queue_transactions existing osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:54.782231 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.782232 7f9d6abe6700  5 queue_op 0x6d5a360 seq 2658 osr(3.4ba 0x276d830) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:54.782241 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d560 (1 active)
2014-06-12 14:46:54.782240 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2934 (0'0,173'2934] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active] do_sub_op osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.782250 7f9d6abe6700 10 osd.2 173 dequeue_op 0x876e200 finish
2014-06-12 14:46:54.782249 7f9d7815d700  5 _do_op 0x6d5a360 seq 2658 osr(3.4ba 0x276d830)/0x276d830 start
2014-06-12 14:46:54.782252 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active)
2014-06-12 14:46:54.782254 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.782253 7f9d7815d700 10 _do_transaction on 0x12c25430
2014-06-12 14:46:54.782261 7f9d7815d700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.782453 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.782477 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.782524 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.782547 7f9d7815d700 15 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.782556 7f9d7815d700 10 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1 r = 0
2014-06-12 14:46:54.782562 7f9d7815d700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.782583 7f9d7815d700 15 _write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 2785280~516096
2014-06-12 14:46:54.782731 7f9d7815d700 10 lookup_strip_header done  cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.782984 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:54.784744 7f9d7815d700 10 _generic_write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 2785280~516096 = 0
2014-06-12 14:46:54.784799 7f9d7815d700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.784813 7f9d7815d700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:54.784821 7f9d7815d700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.784830 7f9d7815d700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:54.785035 7f9d7815d700 20 set_header setting 17403 cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 parent seq 0
2014-06-12 14:46:54.785111 7f9d7815d700 20 set_header setting 2300 cid meta oid 1fe00356/pglog_3.4ba/0//-1 parent seq 0
2014-06-12 14:46:54.785128 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.785139 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:54.787893 7f9d7815d700 10 _do_op 0x6d5a360 seq 2658 r = 0, finisher 0x80f6ac0 0
2014-06-12 14:46:54.787909 7f9d7815d700 10 _finish_op 0x6d5a360 seq 2658 osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:54.787923 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d560 (0 active)
2014-06-12 14:46:54.787926 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.787943 7f9d7795c700  5 -- op tracker -- , seq: 748158, time: 2014-06-12 14:46:54.787934, event: sub_op_applied, op: osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.787951 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2934 (0'0,173'2934] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active] sub_op_modify_applied on 0x12c25340 op osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.787976 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:54.787993 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2934 (0'0,173'2934] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active] op_applied on primary on version 173'2934
2014-06-12 14:46:54.788013 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2934 (0'0,173'2934] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active] op_applied on replica on version 173'2934
2014-06-12 14:46:54.788053 7f9d7715b700  5 -- op tracker -- , seq: 748158, time: 2014-06-12 14:46:54.788044, event: commit_sent, op: osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.788062 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2934 (0'0,173'2934] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2932 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:54.788096 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:54.788138 7f9d7715b700  5 -- op tracker -- , seq: 748158, time: 2014-06-12 14:46:54.788134, event: done, op: osd_sub_op(client.4313.0:146617 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.790340 7f9d5bec4700  5 -- op tracker -- , seq: 748159, time: 2014-06-12 14:46:54.790271, event: header_read, op: osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.790350 7f9d5bec4700  5 -- op tracker -- , seq: 748159, time: 2014-06-12 14:46:54.790273, event: throttled, op: osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.790356 7f9d5bec4700  5 -- op tracker -- , seq: 748159, time: 2014-06-12 14:46:54.790317, event: all_read, op: osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.790361 7f9d5bec4700  5 -- op tracker -- , seq: 748159, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.790365 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.790369 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:54.790374 7f9d5bec4700 15 osd.2 173 enqueue_op 0x876ea00 prio 127 cost 17191 latency 0.000103 osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.790400 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active)
2014-06-12 14:46:54.790417 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x876ea00 prio 127 cost 17191 latency 0.000146 osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4ba( v 173'2934 (0'0,173'2934] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2933 active]
2014-06-12 14:46:54.790441 7f9d6a3e5700  5 -- op tracker -- , seq: 748159, time: 2014-06-12 14:46:54.790437, event: reached_pg, op: osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.790452 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2934 (0'0,173'2934] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2933 active] handle_message: 0x876ea00
2014-06-12 14:46:54.790466 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2934 (0'0,173'2934] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2933 active] sub_op_modify trans 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 v 173'2935 (transaction) 174
2014-06-12 14:46:54.790484 7f9d6a3e5700  5 -- op tracker -- , seq: 748159, time: 2014-06-12 14:46:54.790480, event: started, op: osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.790501 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2934 (0'0,173'2934] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2933 active] append_log log((0'0,173'2934], crt=0'0) [173'2935 (173'2934) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146618 2014-06-12 08:45:57.169740]
2014-06-12 14:46:54.790528 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2935 (0'0,173'2935] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2933 active] add_log_entry 173'2935 (173'2934) modify   721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 by client.4313.0:146618 2014-06-12 08:45:57.169740
2014-06-12 14:46:54.790551 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2935 (0'0,173'2935] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2933 active] append_log  adding 1 keys
2014-06-12 14:46:54.790589 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2935, trimmed: 
2014-06-12 14:46:54.790614 7f9d6a3e5700  5 -- op tracker -- , seq: 748159, time: 2014-06-12 14:46:54.790609, event: started, op: osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.790625 7f9d6a3e5700  5 queue_transactions existing osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:54.790629 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.790630 7f9d6a3e5700  5 queue_op 0x53f4d20 seq 2659 osr(3.4ba 0x276d830) 18677 bytes   (queue has 1 ops and 18677 bytes)
2014-06-12 14:46:54.790641 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d560 (1 active)
2014-06-12 14:46:54.790648 7f9d82171700  5 _do_op 0x53f4d20 seq 2659 osr(3.4ba 0x276d830)/0x276d830 start
2014-06-12 14:46:54.790637 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2935 (0'0,173'2935] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2933 active] do_sub_op osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.790651 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x876ea00 finish
2014-06-12 14:46:54.790654 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active)
2014-06-12 14:46:54.790652 7f9d82171700 10 _do_transaction on 0x2daa75f0
2014-06-12 14:46:54.790657 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.790663 7f9d82171700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.790758 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.790779 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.790821 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.790843 7f9d82171700 15 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.790850 7f9d82171700 10 _omap_rmkeys meta/1fe00356/pglog_3.4ba/0//-1 r = 0
2014-06-12 14:46:54.790856 7f9d82171700 15 _omap_setkeys meta/1fe00356/pglog_3.4ba/0//-1
2014-06-12 14:46:54.790874 7f9d82171700 15 _write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 3301376~16384
2014-06-12 14:46:54.790986 7f9d82171700 10 lookup_strip_header done  cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.790996 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:54.791048 7f9d82171700 10 _generic_write 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 3301376~16384 = 0
2014-06-12 14:46:54.791059 7f9d82171700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.791069 7f9d82171700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:54.791075 7f9d82171700 15 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3
2014-06-12 14:46:54.791084 7f9d82171700 10 _setattrs 3.4ba_head/721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 = 0
2014-06-12 14:46:54.791163 7f9d82171700 20 set_header setting 17403 cid 3.4ba_head oid 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 parent seq 0
2014-06-12 14:46:54.791182 7f9d82171700 20 set_header setting 2300 cid meta oid 1fe00356/pglog_3.4ba/0//-1 parent seq 0
2014-06-12 14:46:54.791196 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.791207 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:54.791331 7f9d82171700 10 _do_op 0x53f4d20 seq 2659 r = 0, finisher 0x534d040 0
2014-06-12 14:46:54.791340 7f9d82171700 10 _finish_op 0x53f4d20 seq 2659 osr(3.4ba 0x276d830)/0x276d830
2014-06-12 14:46:54.791345 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d560 (0 active)
2014-06-12 14:46:54.791347 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.791352 7f9d7715b700  5 -- op tracker -- , seq: 748159, time: 2014-06-12 14:46:54.791346, event: commit_sent, op: osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.791367 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2935 (0'0,173'2935] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2933 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:54.791385 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:54.791418 7f9d7795c700  5 -- op tracker -- , seq: 748159, time: 2014-06-12 14:46:54.791413, event: sub_op_applied, op: osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.791433 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2935 (0'0,173'2935] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2934 active] sub_op_modify_applied on 0x2daa7500 op osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.791449 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2935 (0'0,173'2935] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2934 active] op_applied on primary on version 173'2935
2014-06-12 14:46:54.791465 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4ba( v 173'2935 (0'0,173'2935] local-les=149 n=297 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2934 active] op_applied on replica on version 173'2935
2014-06-12 14:46:54.791492 7f9d7795c700  5 -- op tracker -- , seq: 748159, time: 2014-06-12 14:46:54.791488, event: done, op: osd_sub_op(client.4313.0:146618 3.4ba 721394ba/rbd_data.10b06b8b4567.0000000000000fd1/head//3 [] v 173'2935 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.819007 7f9d66bde700 10 getattr 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 '_' = 0
2014-06-12 14:46:54.819036 7f9d66bde700 15 getattr 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 'snapset'
2014-06-12 14:46:54.819183 7f9d66bde700 10 getattr 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 'snapset' = 0
2014-06-12 14:46:54.819196 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] populate_obc_watchers c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.819212 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.819220 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] get_object_context: creating obc from disk: 0xe9f02c0
2014-06-12 14:46:54.819234 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] get_object_context: 0xe9f02c0 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 rwstate(none n=0 w=0) oi: c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3(173'610 client.4300.0:35097 wrlock_by=unknown.0.0:0 dirty s 4194304 uv610) ssc: 0x81ba460 snapset: 0=[]:[]+head
2014-06-12 14:46:54.819245 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] find_object_context c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 @head oi=c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3(173'610 client.4300.0:35097 wrlock_by=unknown.0.0:0 dirty s 4194304 uv610)
2014-06-12 14:46:54.819272 7f9d66bde700  5 -- op tracker -- , seq: 748156, time: 2014-06-12 14:46:54.819265, event: started, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.819279 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] execute_ctx 0x110c5000
2014-06-12 14:46:54.819289 7f9d66bde700  5 -- op tracker -- , seq: 748156, time: 2014-06-12 14:46:54.819287, event: started, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.819293 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_op c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] ov 173'610 av 173'2536 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:54.819302 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360]
2014-06-12 14:46:54.819308 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:54.819316 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op  write 0~143360
2014-06-12 14:46:54.819327 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op_effects on session 0x1b9b7200
2014-06-12 14:46:54.819336 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] make_writeable c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 snapset=0x81ba498  snapc=0=[]
2014-06-12 14:46:54.819343 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] make_writeable c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:54.819348 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] finish_ctx c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 0x110c5000 op modify  
2014-06-12 14:46:54.819359 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  set mtime to 2014-06-12 08:45:57.146773
2014-06-12 14:46:54.819371 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  final snapset 0=[]:[]+head in c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.819387 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  zeroing write result code 0
2014-06-12 14:46:54.819409 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] new_repop rep_tid 236357 on osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:54.819420 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] issue_repop rep_tid 236357 o c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.819436 7f9d66bde700  5 -- op tracker -- , seq: 748156, time: 2014-06-12 14:46:54.819433, event: waiting for subops from 26, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.819451 7f9d66bde700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:54.819468 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2535 (0'0,173'2535] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] append_log log((0'0,173'2535], crt=0'0) [173'2536 (173'610) modify   c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 by client.4300.0:140290 2014-06-12 08:45:57.146773]
2014-06-12 14:46:54.819499 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] add_log_entry 173'2536 (173'610) modify   c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 by client.4300.0:140290 2014-06-12 08:45:57.146773
2014-06-12 14:46:54.819527 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] append_log  adding 1 keys
2014-06-12 14:46:54.819556 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2536, trimmed: 
2014-06-12 14:46:54.819581 7f9d66bde700  5 queue_transactions existing osr(3.1e1 0x276d140)/0x276d140
2014-06-12 14:46:54.819587 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.819589 7f9d66bde700  5 queue_op 0x17dd8fc0 seq 2373 osr(3.1e1 0x276d140) 145653 bytes   (queue has 1 ops and 145653 bytes)
2014-06-12 14:46:54.819600 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x553f320 (1 active)
2014-06-12 14:46:54.819608 7f9d7e96a700  5 _do_op 0x17dd8fc0 seq 2373 osr(3.1e1 0x276d140)/0x276d140 start
2014-06-12 14:46:54.819612 7f9d7e96a700 10 _do_transaction on 0x15acf980
2014-06-12 14:46:54.819597 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] eval_repop repgather(0x15acfbc0 173'2536 rep_tid=236357 committed?=0 applied?=0 lock=0 op=osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:54.819615 7f9d66bde700 10 osd.2 173 dequeue_op 0x124e3b00 finish
2014-06-12 14:46:54.819618 7f9d7e96a700 15 _omap_setkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.819624 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:54.819630 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.819659 7f9d64bda700 10 osd.2 173 dequeue_op 0x124e2800 prio 63 cost 516096 latency 0.039597 osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]
2014-06-12 14:46:54.819706 7f9d64bda700  5 -- op tracker -- , seq: 748157, time: 2014-06-12 14:46:54.819694, event: reached_pg, op: osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.819734 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:54.819751 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.819753 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] handle_message: 0x124e2800
2014-06-12 14:46:54.819759 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_op osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:54.819777 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.819774 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] get_object_context: found obc in cache: 0xe9f02c0
2014-06-12 14:46:54.819780 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] get_object_context: 0xe9f02c0 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 rwstate(write n=1 w=0) oi: c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3(173'2536 client.4300.0:140290 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2536) ssc: 0x81ba460 snapset: 0=[]:[]+head
2014-06-12 14:46:54.819789 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] find_object_context c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 @head oi=c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3(173'2536 client.4300.0:140290 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2536)
2014-06-12 14:46:54.819803 7f9d64bda700  5 -- op tracker -- , seq: 748157, time: 2014-06-12 14:46:54.819800, event: started, op: osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.819807 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] execute_ctx 0x25b21000
2014-06-12 14:46:54.819815 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.819817 7f9d64bda700  5 -- op tracker -- , seq: 748157, time: 2014-06-12 14:46:54.819814, event: started, op: osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.819821 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_op c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] ov 173'2536 av 173'2537 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:54.819833 7f9d7e96a700 15 _omap_rmkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.819829 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096]
2014-06-12 14:46:54.819836 7f9d7e96a700 10 _omap_rmkeys meta/1fe10f55/pglog_3.1e1/0//-1 r = 0
2014-06-12 14:46:54.819835 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:54.819840 7f9d7e96a700 15 _omap_setkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.819843 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op  write 143360~516096
2014-06-12 14:46:54.819851 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op_effects on session 0x1b9b7200
2014-06-12 14:46:54.819854 7f9d7e96a700 15 _write 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 0~143360
2014-06-12 14:46:54.819857 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] make_writeable c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 snapset=0x81ba498  snapc=0=[]
2014-06-12 14:46:54.819864 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] make_writeable c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:54.819870 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] finish_ctx c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 0x25b21000 op modify  
2014-06-12 14:46:54.819877 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  set mtime to 2014-06-12 08:45:57.159425
2014-06-12 14:46:54.819889 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  final snapset 0=[]:[]+head in c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.819899 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  zeroing write result code 0
2014-06-12 14:46:54.819905 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] new_repop rep_tid 236358 on osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:54.819913 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] issue_repop rep_tid 236358 o c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.819927 7f9d64bda700  5 -- op tracker -- , seq: 748157, time: 2014-06-12 14:46:54.819924, event: waiting for subops from 26, op: osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.819939 7f9d7e96a700 10 lookup_strip_header done  cid 3.1e1_head oid c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.819942 7f9d64bda700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:54.819954 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2536 (0'0,173'2536] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] append_log log((0'0,173'2536], crt=0'0) [173'2537 (173'2536) modify   c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 by client.4300.0:140291 2014-06-12 08:45:57.159425]
2014-06-12 14:46:54.819968 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] add_log_entry 173'2537 (173'2536) modify   c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 by client.4300.0:140291 2014-06-12 08:45:57.159425
2014-06-12 14:46:54.819994 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] append_log  adding 1 keys
2014-06-12 14:46:54.820013 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2537, trimmed: 
2014-06-12 14:46:54.820025 7f9d64bda700  5 queue_transactions existing osr(3.1e1 0x276d140)/0x276d140
2014-06-12 14:46:54.820027 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.820028 7f9d64bda700  5 queue_op 0x2029bc0 seq 2374 osr(3.1e1 0x276d140) 518389 bytes   (queue has 2 ops and 664042 bytes)
2014-06-12 14:46:54.820037 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x553f320 (2 active)
2014-06-12 14:46:54.820033 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] eval_repop repgather(0x2335c840 173'2537 rep_tid=236358 committed?=0 applied?=0 lock=0 op=osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:54.820044 7f9d64bda700 10 osd.2 173 dequeue_op 0x124e2800 finish
2014-06-12 14:46:54.820047 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.820049 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.820144 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:54.820505 7f9d7e96a700 10 _generic_write 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 0~143360 = 0
2014-06-12 14:46:54.820530 7f9d7e96a700 15 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.820539 7f9d7e96a700 10 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:54.820544 7f9d7e96a700 15 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.820557 7f9d7e96a700 10 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:54.820651 7f9d7e96a700 20 set_header setting 16580 cid 3.1e1_head oid c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 parent seq 0
2014-06-12 14:46:54.820681 7f9d7e96a700 20 set_header setting 2008 cid meta oid 1fe10f55/pglog_3.1e1/0//-1 parent seq 0
2014-06-12 14:46:54.820700 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.820705 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:54.821232 7f9d7e96a700 10 _do_op 0x17dd8fc0 seq 2373 r = 0, finisher 0x4f55b80 0x1e6268c0
2014-06-12 14:46:54.821242 7f9d7e96a700 10 _finish_op 0x17dd8fc0 seq 2373 osr(3.1e1 0x276d140)/0x276d140
2014-06-12 14:46:54.821249 7f9d8116f700  5 _do_op 0x2029bc0 seq 2374 osr(3.1e1 0x276d140)/0x276d140 start
2014-06-12 14:46:54.821250 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x553f320 (1 active)
2014-06-12 14:46:54.821253 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.821254 7f9d8116f700 10 _do_transaction on 0x2335ca80
2014-06-12 14:46:54.821260 7f9d8116f700 15 _omap_setkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.821257 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_applied: 236357
2014-06-12 14:46:54.821276 7f9d7795c700  5 -- op tracker -- , seq: 748156, time: 2014-06-12 14:46:54.821271, event: op_applied, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.821283 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_applied on primary on version 173'2536
2014-06-12 14:46:54.821319 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.821332 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.821322 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_commit: 236357
2014-06-12 14:46:54.821342 7f9d7715b700  5 -- op tracker -- , seq: 748156, time: 2014-06-12 14:46:54.821337, event: op_commit, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.821359 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.821378 7f9d8116f700 15 _omap_rmkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.821382 7f9d8116f700 10 _omap_rmkeys meta/1fe10f55/pglog_3.1e1/0//-1 r = 0
2014-06-12 14:46:54.821385 7f9d8116f700 15 _omap_setkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.821474 7f9d8116f700 15 _write 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 143360~516096
2014-06-12 14:46:54.821555 7f9d8116f700 10 lookup_strip_header done  cid 3.1e1_head oid c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.821661 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:54.822738 7f9d8116f700 10 _generic_write 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 143360~516096 = 0
2014-06-12 14:46:54.822765 7f9d8116f700 15 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.822774 7f9d8116f700 10 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:54.822778 7f9d8116f700 15 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.822782 7f9d8116f700 10 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:54.822911 7f9d8116f700 20 set_header setting 16580 cid 3.1e1_head oid c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 parent seq 0
2014-06-12 14:46:54.822957 7f9d8116f700 20 set_header setting 2008 cid meta oid 1fe10f55/pglog_3.1e1/0//-1 parent seq 0
2014-06-12 14:46:54.822967 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.822972 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:54.824514 7f9d8116f700 10 _do_op 0x2029bc0 seq 2374 r = 0, finisher 0x5b76b00 0x31aec4c0
2014-06-12 14:46:54.824523 7f9d8116f700 10 _finish_op 0x2029bc0 seq 2374 osr(3.1e1 0x276d140)/0x276d140
2014-06-12 14:46:54.824528 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x553f320 (0 active)
2014-06-12 14:46:54.824532 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.824533 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_applied: 236358
2014-06-12 14:46:54.824558 7f9d7795c700  5 -- op tracker -- , seq: 748157, time: 2014-06-12 14:46:54.824553, event: op_applied, op: osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.824567 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_applied on primary on version 173'2537
2014-06-12 14:46:54.824608 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_commit: 236358
2014-06-12 14:46:54.824629 7f9d7715b700  5 -- op tracker -- , seq: 748157, time: 2014-06-12 14:46:54.824624, event: op_commit, op: osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.864220 7f9d5a16f700  5 -- op tracker -- , seq: 748160, time: 2014-06-12 14:46:54.862999, event: header_read, op: osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.864237 7f9d5a16f700  5 -- op tracker -- , seq: 748160, time: 2014-06-12 14:46:54.863002, event: throttled, op: osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.864246 7f9d5a16f700  5 -- op tracker -- , seq: 748160, time: 2014-06-12 14:46:54.864176, event: all_read, op: osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.864255 7f9d5a16f700  5 -- op tracker -- , seq: 748160, time: 0.000000, event: dispatched, op: osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.864260 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.864267 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:54.864276 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2910a00 prio 127 cost 541479 latency 0.001277 osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.864305 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.864314 7f9d69be4700 10 osd.2 173 dequeue_op 0x2910a00 prio 127 cost 541479 latency 0.001315 osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2471 (0'0,173'2471] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active]
2014-06-12 14:46:54.864364 7f9d69be4700  5 -- op tracker -- , seq: 748160, time: 2014-06-12 14:46:54.864359, event: reached_pg, op: osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.864372 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2471 (0'0,173'2471] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active] handle_message: 0x2910a00
2014-06-12 14:46:54.864389 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2471 (0'0,173'2471] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active] sub_op_modify trans 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 v 173'2472 (transaction) 174
2014-06-12 14:46:54.864409 7f9d69be4700  5 -- op tracker -- , seq: 748160, time: 2014-06-12 14:46:54.864405, event: started, op: osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.864425 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2471 (0'0,173'2471] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active] append_log log((0'0,173'2471], crt=0'0) [173'2472 (173'2471) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143631 2014-06-12 08:43:21.435474]
2014-06-12 14:46:54.864459 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2472 (0'0,173'2472] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active] add_log_entry 173'2472 (173'2471) modify   3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 by client.4294.0:143631 2014-06-12 08:43:21.435474
2014-06-12 14:46:54.864489 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2472 (0'0,173'2472] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active] append_log  adding 1 keys
2014-06-12 14:46:54.864531 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2472, trimmed: 
2014-06-12 14:46:54.864557 7f9d69be4700  5 -- op tracker -- , seq: 748160, time: 2014-06-12 14:46:54.864552, event: started, op: osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.864570 7f9d69be4700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:54.864574 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.864575 7f9d69be4700  5 queue_op 0x62c67e0 seq 2305 osr(3.4f4 0x276db70) 542965 bytes   (queue has 1 ops and 542965 bytes)
2014-06-12 14:46:54.864586 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:46:54.864593 7f9d7d167700  5 _do_op 0x62c67e0 seq 2305 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:46:54.864582 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2472 (0'0,173'2472] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active] do_sub_op osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.864598 7f9d7d167700 10 _do_transaction on 0x27757270
2014-06-12 14:46:54.864598 7f9d69be4700 10 osd.2 173 dequeue_op 0x2910a00 finish
2014-06-12 14:46:54.864602 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.864605 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.864604 7f9d7d167700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:54.864735 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:54.864769 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.864822 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.864941 7f9d7d167700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:54.864951 7f9d7d167700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:46:54.864958 7f9d7d167700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:46:54.864978 7f9d7d167700 15 _write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 3653632~540672
2014-06-12 14:46:54.865136 7f9d7d167700 10 lookup_strip_header done  cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:54.865303 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:54.867313 7f9d7d167700 10 _generic_write 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 3653632~540672 = 0
2014-06-12 14:46:54.867371 7f9d7d167700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:54.867387 7f9d7d167700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:54.867393 7f9d7d167700 15 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3
2014-06-12 14:46:54.867405 7f9d7d167700 10 _setattrs 3.4f4_head/3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 = 0
2014-06-12 14:46:54.867620 7f9d7d167700 20 set_header setting 17388 cid 3.4f4_head oid 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 parent seq 0
2014-06-12 14:46:54.867707 7f9d7d167700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:46:54.867726 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.867738 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:54.870934 7f9d7d167700 10 _do_op 0x62c67e0 seq 2305 r = 0, finisher 0x31eda40 0
2014-06-12 14:46:54.870955 7f9d7d167700 10 _finish_op 0x62c67e0 seq 2305 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:46:54.870962 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:46:54.870981 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.871003 7f9d7795c700  5 -- op tracker -- , seq: 748160, time: 2014-06-12 14:46:54.870993, event: sub_op_applied, op: osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.871026 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2472 (0'0,173'2472] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active] sub_op_modify_applied on 0x27757180 op osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.871049 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:54.871063 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2472 (0'0,173'2472] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active] op_applied on primary on version 173'2472
2014-06-12 14:46:54.871074 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2472 (0'0,173'2472] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active] op_applied on replica on version 173'2472
2014-06-12 14:46:54.871101 7f9d7715b700  5 -- op tracker -- , seq: 748160, time: 2014-06-12 14:46:54.871091, event: commit_sent, op: osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.871110 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2472 (0'0,173'2472] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2470 active] sub_op_modify_commit on op osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:54.871122 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:54.871149 7f9d7715b700  5 -- op tracker -- , seq: 748160, time: 2014-06-12 14:46:54.871144, event: done, op: osd_sub_op(client.4294.0:143631 3.4f4 3e9fd4f4/rbd_data.109d6b8b4567.00000000000010d9/head//3 [] v 173'2472 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.911925 7f9d494c9700  5 -- op tracker -- , seq: 748161, time: 2014-06-12 14:46:54.910342, event: header_read, op: osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.911942 7f9d494c9700  5 -- op tracker -- , seq: 748161, time: 2014-06-12 14:46:54.910345, event: throttled, op: osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.911953 7f9d494c9700  5 -- op tracker -- , seq: 748161, time: 2014-06-12 14:46:54.911879, event: all_read, op: osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.911963 7f9d494c9700  5 -- op tracker -- , seq: 748161, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.911970 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:46:54.911997 7f9d494c9700 15 osd.2 173 enqueue_op 0x24c9d400 prio 63 cost 516096 latency 0.001655 osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:54.912028 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.912039 7f9d653db700 10 osd.2 173 dequeue_op 0x24c9d400 prio 63 cost 516096 latency 0.001696 osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]
2014-06-12 14:46:54.912067 7f9d653db700  5 -- op tracker -- , seq: 748161, time: 2014-06-12 14:46:54.912062, event: reached_pg, op: osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.912078 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:54.912091 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] handle_message: 0x24c9d400
2014-06-12 14:46:54.912100 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_op osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:54.912123 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] get_object_context: found obc in cache: 0x244de000
2014-06-12 14:46:54.912135 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] get_object_context: 0x244de000 d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 rwstate(write n=2 w=0) oi: d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3(173'2514 client.4290.0:143737 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2514) ssc: 0x679b420 snapset: 0=[]:[]+head
2014-06-12 14:46:54.912149 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] find_object_context d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 @head oi=d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3(173'2514 client.4290.0:143737 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2514)
2014-06-12 14:46:54.912170 7f9d653db700  5 -- op tracker -- , seq: 748161, time: 2014-06-12 14:46:54.912165, event: started, op: osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.912178 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] execute_ctx 0x2a84a00
2014-06-12 14:46:54.912194 7f9d653db700  5 -- op tracker -- , seq: 748161, time: 2014-06-12 14:46:54.912189, event: started, op: osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.912199 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_op d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] ov 173'2514 av 173'2515 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:54.912213 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096]
2014-06-12 14:46:54.912222 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:54.912233 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op  write 1646592~516096
2014-06-12 14:46:54.912244 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:46:54.912253 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] make_writeable d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 snapset=0x679b458  snapc=0=[]
2014-06-12 14:46:54.912263 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] make_writeable d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:54.912271 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] finish_ctx d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 0x2a84a00 op modify  
2014-06-12 14:46:54.912281 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  set mtime to 2014-06-12 08:45:57.289685
2014-06-12 14:46:54.912303 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  final snapset 0=[]:[]+head in d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:54.912318 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  zeroing write result code 0
2014-06-12 14:46:54.912328 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] new_repop rep_tid 236359 on osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:54.912343 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] issue_repop rep_tid 236359 o d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:54.912363 7f9d653db700  5 -- op tracker -- , seq: 748161, time: 2014-06-12 14:46:54.912358, event: waiting for subops from 26, op: osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.912382 7f9d653db700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:54.912401 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2514 (0'0,173'2514] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] append_log log((0'0,173'2514], crt=0'0) [173'2515 (173'2514) modify   d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 by client.4290.0:143738 2014-06-12 08:45:57.289685]
2014-06-12 14:46:54.912421 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] add_log_entry 173'2515 (173'2514) modify   d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 by client.4290.0:143738 2014-06-12 08:45:57.289685
2014-06-12 14:46:54.912440 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] append_log  adding 1 keys
2014-06-12 14:46:54.912471 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2515, trimmed: 
2014-06-12 14:46:54.912493 7f9d653db700  5 queue_transactions existing osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:54.912498 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.912500 7f9d653db700  5 queue_op 0x142a9e00 seq 2331 osr(3.7ae 0x49b4010) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:54.912508 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] eval_repop repgather(0x6590cc0 173'2515 rep_tid=236359 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:54.912524 7f9d653db700 10 osd.2 173 dequeue_op 0x24c9d400 finish
2014-06-12 14:46:54.912527 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.912530 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.913457 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1c0b440 (1 active)
2014-06-12 14:46:54.913469 7f9d79960700  5 _do_op 0x142a9e00 seq 2331 osr(3.7ae 0x49b4010)/0x49b4010 start
2014-06-12 14:46:54.913475 7f9d79960700 10 _do_transaction on 0xe833140
2014-06-12 14:46:54.913491 7f9d79960700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:54.913663 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:54.913848 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.913916 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.913936 7f9d79960700 15 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:54.913940 7f9d79960700 10 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1 r = 0
2014-06-12 14:46:54.913945 7f9d79960700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:46:54.913957 7f9d79960700 15 _write 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 1646592~516096
2014-06-12 14:46:54.914283 7f9d79960700 10 lookup_strip_header done  cid 3.7ae_head oid d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:54.914493 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:54.916546 7f9d79960700 10 _generic_write 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 1646592~516096 = 0
2014-06-12 14:46:54.916597 7f9d79960700 15 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:54.916616 7f9d79960700 10 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:54.916623 7f9d79960700 15 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:46:54.916631 7f9d79960700 10 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:46:54.916845 7f9d79960700 20 set_header setting 15929 cid 3.7ae_head oid d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 parent seq 0
2014-06-12 14:46:54.916919 7f9d79960700 20 set_header setting 2150 cid meta oid 1fe77416/pglog_3.7ae/0//-1 parent seq 0
2014-06-12 14:46:54.916934 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.916943 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:54.917831 7f9d53c11700  5 -- op tracker -- , seq: 748162, time: 2014-06-12 14:46:54.912854, event: header_read, op: osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.917850 7f9d53c11700  5 -- op tracker -- , seq: 748162, time: 2014-06-12 14:46:54.912857, event: throttled, op: osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.917860 7f9d53c11700  5 -- op tracker -- , seq: 748162, time: 2014-06-12 14:46:54.917766, event: all_read, op: osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.917869 7f9d53c11700  5 -- op tracker -- , seq: 748162, time: 0.000000, event: dispatched, op: osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.917882 7f9d53c11700 20 osd.2 173 should_share_map client.4300 172.16.96.220:0/1243739 173
2014-06-12 14:46:54.917923 7f9d53c11700 15 osd.2 173 enqueue_op 0x124e2d00 prio 63 cost 1548288 latency 0.005068 osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:54.918181 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.918191 7f9d623d5700 10 osd.2 173 dequeue_op 0x124e2d00 prio 63 cost 1548288 latency 0.005337 osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]
2014-06-12 14:46:54.918237 7f9d623d5700  5 -- op tracker -- , seq: 748162, time: 2014-06-12 14:46:54.918231, event: reached_pg, op: osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.918247 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:54.918264 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] handle_message: 0x124e2d00
2014-06-12 14:46:54.918275 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_op osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:54.918301 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] get_object_context: found obc in cache: 0xe9f02c0
2014-06-12 14:46:54.918324 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] get_object_context: 0xe9f02c0 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 rwstate(write n=2 w=0) oi: c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3(173'2537 client.4300.0:140291 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2537) ssc: 0x81ba460 snapset: 0=[]:[]+head
2014-06-12 14:46:54.918340 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] find_object_context c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 @head oi=c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3(173'2537 client.4300.0:140291 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2537)
2014-06-12 14:46:54.918380 7f9d623d5700  5 -- op tracker -- , seq: 748162, time: 2014-06-12 14:46:54.918371, event: started, op: osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.918389 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] execute_ctx 0x33b2d800
2014-06-12 14:46:54.918406 7f9d623d5700  5 -- op tracker -- , seq: 748162, time: 2014-06-12 14:46:54.918401, event: started, op: osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.918423 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_op c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] ov 173'2537 av 173'2538 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:54.918438 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288]
2014-06-12 14:46:54.918449 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:54.918462 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op  write 659456~1548288
2014-06-12 14:46:54.918479 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op_effects on session 0x1b9b7200
2014-06-12 14:46:54.918489 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] make_writeable c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 snapset=0x81ba498  snapc=0=[]
2014-06-12 14:46:54.918502 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] make_writeable c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:54.918511 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] finish_ctx c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 0x33b2d800 op modify  
2014-06-12 14:46:54.918527 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  set mtime to 2014-06-12 08:45:57.290254
2014-06-12 14:46:54.918549 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  final snapset 0=[]:[]+head in c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.918571 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  zeroing write result code 0
2014-06-12 14:46:54.918582 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] new_repop rep_tid 236360 on osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:54.918605 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] issue_repop rep_tid 236360 o c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.918633 7f9d623d5700  5 -- op tracker -- , seq: 748162, time: 2014-06-12 14:46:54.918628, event: waiting for subops from 26, op: osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.918660 7f9d623d5700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:54.918689 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2537 (0'0,173'2537] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] append_log log((0'0,173'2537], crt=0'0) [173'2538 (173'2537) modify   c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 by client.4300.0:140292 2014-06-12 08:45:57.290254]
2014-06-12 14:46:54.918716 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] add_log_entry 173'2538 (173'2537) modify   c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 by client.4300.0:140292 2014-06-12 08:45:57.290254
2014-06-12 14:46:54.918735 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] append_log  adding 1 keys
2014-06-12 14:46:54.918780 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2538, trimmed: 
2014-06-12 14:46:54.918821 7f9d623d5700  5 queue_transactions existing osr(3.1e1 0x276d140)/0x276d140
2014-06-12 14:46:54.918827 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.918829 7f9d623d5700  5 queue_op 0x6a7b260 seq 2375 osr(3.1e1 0x276d140) 1550581 bytes   (queue has 2 ops and 2068970 bytes)
2014-06-12 14:46:54.918839 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] eval_repop repgather(0x11c7f440 173'2538 rep_tid=236360 committed?=0 applied?=0 lock=0 op=osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:54.918859 7f9d623d5700 10 osd.2 173 dequeue_op 0x124e2d00 finish
2014-06-12 14:46:54.918864 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.918868 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.919427 7f9d53c11700  5 -- op tracker -- , seq: 748163, time: 2014-06-12 14:46:54.917971, event: header_read, op: osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.919439 7f9d53c11700  5 -- op tracker -- , seq: 748163, time: 2014-06-12 14:46:54.917976, event: throttled, op: osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.919448 7f9d53c11700  5 -- op tracker -- , seq: 748163, time: 2014-06-12 14:46:54.919396, event: all_read, op: osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.919456 7f9d53c11700  5 -- op tracker -- , seq: 748163, time: 0.000000, event: dispatched, op: osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.919463 7f9d53c11700 20 osd.2 173 should_share_map client.4300 172.16.96.220:0/1243739 173
2014-06-12 14:46:54.919489 7f9d53c11700 15 osd.2 173 enqueue_op 0xd38e900 prio 63 cost 516096 latency 0.001517 osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:54.919524 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.919536 7f9d68be2700 10 osd.2 173 dequeue_op 0xd38e900 prio 63 cost 516096 latency 0.001564 osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]
2014-06-12 14:46:54.919564 7f9d68be2700  5 -- op tracker -- , seq: 748163, time: 2014-06-12 14:46:54.919559, event: reached_pg, op: osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.919579 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:54.919593 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] handle_message: 0xd38e900
2014-06-12 14:46:54.919602 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_op osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:54.919639 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] get_object_context: found obc in cache: 0xe9f02c0
2014-06-12 14:46:54.919652 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] get_object_context: 0xe9f02c0 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 rwstate(write n=3 w=0) oi: c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3(173'2538 client.4300.0:140292 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2538) ssc: 0x81ba460 snapset: 0=[]:[]+head
2014-06-12 14:46:54.919669 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] find_object_context c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 @head oi=c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3(173'2538 client.4300.0:140292 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2538)
2014-06-12 14:46:54.919703 7f9d68be2700  5 -- op tracker -- , seq: 748163, time: 2014-06-12 14:46:54.919697, event: started, op: osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.919710 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] execute_ctx 0x2a84400
2014-06-12 14:46:54.919727 7f9d68be2700  5 -- op tracker -- , seq: 748163, time: 2014-06-12 14:46:54.919722, event: started, op: osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.919733 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_op c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] ov 173'2538 av 173'2539 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:54.919751 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096]
2014-06-12 14:46:54.919762 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:54.919777 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op  write 2207744~516096
2014-06-12 14:46:54.919795 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op_effects on session 0x1b9b7200
2014-06-12 14:46:54.919806 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] make_writeable c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 snapset=0x81ba498  snapc=0=[]
2014-06-12 14:46:54.919818 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] make_writeable c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:54.919828 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] finish_ctx c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 0x2a84400 op modify  
2014-06-12 14:46:54.919841 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  set mtime to 2014-06-12 08:45:57.291277
2014-06-12 14:46:54.919865 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  final snapset 0=[]:[]+head in c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.919888 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  zeroing write result code 0
2014-06-12 14:46:54.919900 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] new_repop rep_tid 236361 on osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:54.919919 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] issue_repop rep_tid 236361 o c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.919956 7f9d68be2700  5 -- op tracker -- , seq: 748163, time: 2014-06-12 14:46:54.919950, event: waiting for subops from 26, op: osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.919995 7f9d68be2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:54.920025 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2538 (0'0,173'2538] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] append_log log((0'0,173'2538], crt=0'0) [173'2539 (173'2538) modify   c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 by client.4300.0:140293 2014-06-12 08:45:57.291277]
2014-06-12 14:46:54.920062 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] add_log_entry 173'2539 (173'2538) modify   c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 by client.4300.0:140293 2014-06-12 08:45:57.291277
2014-06-12 14:46:54.920080 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] append_log  adding 1 keys
2014-06-12 14:46:54.920105 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2539, trimmed: 
2014-06-12 14:46:54.920123 7f9d68be2700  5 queue_transactions existing osr(3.1e1 0x276d140)/0x276d140
2014-06-12 14:46:54.920127 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.920128 7f9d68be2700  5 queue_op 0x5d515c0 seq 2376 osr(3.1e1 0x276d140) 518389 bytes   (queue has 3 ops and 2587359 bytes)
2014-06-12 14:46:54.920142 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] eval_repop repgather(0x24fa7680 173'2539 rep_tid=236361 committed?=0 applied?=0 lock=0 op=osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:54.920161 7f9d68be2700 10 osd.2 173 dequeue_op 0xd38e900 finish
2014-06-12 14:46:54.920164 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.920167 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.920231 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x553f320 (2 active)
2014-06-12 14:46:54.920238 7f9d8016d700  5 _do_op 0x6a7b260 seq 2375 osr(3.1e1 0x276d140)/0x276d140 start
2014-06-12 14:46:54.920243 7f9d8016d700 10 _do_transaction on 0x11c7eb40
2014-06-12 14:46:54.920249 7f9d8016d700 15 _omap_setkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.920494 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.920500 7f9d79960700 10 _do_op 0x142a9e00 seq 2331 r = 0, finisher 0x7b43840 0x2a1656c0
2014-06-12 14:46:54.920511 7f9d79960700 10 _finish_op 0x142a9e00 seq 2331 osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:46:54.920522 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.920518 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1c0b440 (1 active)
2014-06-12 14:46:54.920526 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x553f320 (2 active)
2014-06-12 14:46:54.920525 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_applied: 236359
2014-06-12 14:46:54.920545 7f9d7795c700  5 -- op tracker -- , seq: 748161, time: 2014-06-12 14:46:54.920540, event: op_applied, op: osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.920553 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_applied on primary on version 173'2515
2014-06-12 14:46:54.920577 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.920618 7f9d8016d700 15 _omap_rmkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.920625 7f9d8016d700 10 _omap_rmkeys meta/1fe10f55/pglog_3.1e1/0//-1 r = 0
2014-06-12 14:46:54.920631 7f9d8016d700 15 _omap_setkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.920650 7f9d8016d700 15 _write 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 659456~1548288
2014-06-12 14:46:54.920659 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.920890 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_commit: 236359
2014-06-12 14:46:54.920912 7f9d7715b700  5 -- op tracker -- , seq: 748161, time: 2014-06-12 14:46:54.920906, event: op_commit, op: osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.921000 7f9d8016d700 10 lookup_strip_header done  cid 3.1e1_head oid c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.921178 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:54.927674 7f9d56432700  5 -- op tracker -- , seq: 748164, time: 2014-06-12 14:46:54.923419, event: header_read, op: osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.927687 7f9d56432700  5 -- op tracker -- , seq: 748164, time: 2014-06-12 14:46:54.923421, event: throttled, op: osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.927696 7f9d56432700  5 -- op tracker -- , seq: 748164, time: 2014-06-12 14:46:54.927636, event: all_read, op: osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.927705 7f9d56432700  5 -- op tracker -- , seq: 748164, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.927710 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:54.927717 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:54.927725 7f9d56432700 15 osd.2 173 enqueue_op 0x2b014500 prio 127 cost 1549095 latency 0.004306 osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.927872 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:54.927880 7f9d643d9700 10 osd.2 173 dequeue_op 0x2b014500 prio 127 cost 1549095 latency 0.004461 osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'3002 (79'1,173'3002] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3001 active]
2014-06-12 14:46:54.927905 7f9d643d9700  5 -- op tracker -- , seq: 748164, time: 2014-06-12 14:46:54.927901, event: reached_pg, op: osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.927915 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3002 (79'1,173'3002] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3001 active] handle_message: 0x2b014500
2014-06-12 14:46:54.927927 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3002 (79'1,173'3002] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3001 active] sub_op_modify trans d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 v 173'3003 (transaction) 174
2014-06-12 14:46:54.927943 7f9d643d9700  5 -- op tracker -- , seq: 748164, time: 2014-06-12 14:46:54.927939, event: started, op: osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.927959 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3002 (79'1,173'3002] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3001 active] append_log log((79'1,173'3002], crt=0'0) [173'3003 (173'3002) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145535 2014-06-12 08:45:57.290294]
2014-06-12 14:46:54.927981 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3003 (79'1,173'3003] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3001 active] add_log_entry 173'3003 (173'3002) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145535 2014-06-12 08:45:57.290294
2014-06-12 14:46:54.928006 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3003 (79'1,173'3003] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3001 active] append_log  adding 1 keys
2014-06-12 14:46:54.928019 7f9d643d9700 10 trim log((79'1,173'3003], crt=0'0) to 79'2
2014-06-12 14:46:54.928042 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3003, trimmed: 79'2
2014-06-12 14:46:54.928066 7f9d643d9700  5 -- op tracker -- , seq: 748164, time: 2014-06-12 14:46:54.928061, event: started, op: osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.928078 7f9d643d9700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:54.928082 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:54.928083 7f9d643d9700  5 queue_op 0xb045f20 seq 2805 osr(3.458 0x276d960) 1550616 bytes   (queue has 3 ops and 3619586 bytes)
2014-06-12 14:46:54.928090 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'3003 (79'2,173'3003] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3001 active] do_sub_op osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.928103 7f9d643d9700 10 osd.2 173 dequeue_op 0x2b014500 finish
2014-06-12 14:46:54.928106 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:54.928109 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:54.929351 7f9d8016d700 10 _generic_write 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 659456~1548288 = 0
2014-06-12 14:46:54.929546 7f9d8016d700 15 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.929565 7f9d8016d700 10 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:54.929577 7f9d8016d700 15 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.929586 7f9d8016d700 10 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:54.929637 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (3 active)
2014-06-12 14:46:54.929646 7f9d7d968700  5 _do_op 0xb045f20 seq 2805 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:54.929651 7f9d7d968700 10 _do_transaction on 0x13f842b0
2014-06-12 14:46:54.929656 7f9d7d968700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:54.929782 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:54.929803 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.930093 7f9d8016d700 20 set_header setting 16580 cid 3.1e1_head oid c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 parent seq 0
2014-06-12 14:46:54.930250 7f9d8016d700 20 set_header setting 2008 cid meta oid 1fe10f55/pglog_3.1e1/0//-1 parent seq 0
2014-06-12 14:46:54.930266 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.930275 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:54.943795 7f9d8016d700 10 _do_op 0x6a7b260 seq 2375 r = 0, finisher 0x32e66fe0 0xcb619c0
2014-06-12 14:46:54.943818 7f9d8016d700 10 _finish_op 0x6a7b260 seq 2375 osr(3.1e1 0x276d140)/0x276d140
2014-06-12 14:46:54.943832 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x553f320 (2 active)
2014-06-12 14:46:54.943842 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.945434 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_applied: 236360
2014-06-12 14:46:54.945482 7f9d7795c700  5 -- op tracker -- , seq: 748162, time: 2014-06-12 14:46:54.945471, event: op_applied, op: osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.945499 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_applied on primary on version 173'2538
2014-06-12 14:46:54.945580 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_commit: 236360
2014-06-12 14:46:54.945606 7f9d7715b700  5 -- op tracker -- , seq: 748162, time: 2014-06-12 14:46:54.945598, event: op_commit, op: osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.946297 7f9d79960700  5 _do_op 0x5d515c0 seq 2376 osr(3.1e1 0x276d140)/0x276d140 start
2014-06-12 14:46:54.946309 7f9d79960700 10 _do_transaction on 0x7414240
2014-06-12 14:46:54.946317 7f9d79960700 15 _omap_setkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.946448 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.946472 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:54.946528 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.946547 7f9d79960700 15 _omap_rmkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.946553 7f9d79960700 10 _omap_rmkeys meta/1fe10f55/pglog_3.1e1/0//-1 r = 0
2014-06-12 14:46:54.946559 7f9d79960700 15 _omap_setkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:46:54.946743 7f9d79960700 15 _write 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 2207744~516096
2014-06-12 14:46:54.946905 7f9d79960700 10 lookup_strip_header done  cid 3.1e1_head oid c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.947062 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:54.949270 7f9d79960700 10 _generic_write 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 2207744~516096 = 0
2014-06-12 14:46:54.949325 7f9d79960700 15 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.949341 7f9d79960700 10 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:54.949347 7f9d79960700 15 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:46:54.949356 7f9d79960700 10 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:46:54.949579 7f9d79960700 20 set_header setting 16580 cid 3.1e1_head oid c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 parent seq 0
2014-06-12 14:46:54.949756 7f9d79960700 20 set_header setting 2008 cid meta oid 1fe10f55/pglog_3.1e1/0//-1 parent seq 0
2014-06-12 14:46:54.949773 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.949782 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:54.953036 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:54.953071 7f9d7d968700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:54.953080 7f9d7d968700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:54.953087 7f9d7d968700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:54.953261 7f9d7d968700 15 _write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 1187840~1548288
2014-06-12 14:46:54.953429 7f9d7d968700 10 lookup_strip_header done  cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:54.954137 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:54.956222 7f9d79960700 10 _do_op 0x5d515c0 seq 2376 r = 0, finisher 0x65f62c0 0x465b740
2014-06-12 14:46:54.956240 7f9d79960700 10 _finish_op 0x5d515c0 seq 2376 osr(3.1e1 0x276d140)/0x276d140
2014-06-12 14:46:54.956268 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_commit: 236361
2014-06-12 14:46:54.956286 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x553f320 (1 active)
2014-06-12 14:46:54.956294 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.956302 7f9d7715b700  5 -- op tracker -- , seq: 748163, time: 2014-06-12 14:46:54.956293, event: op_commit, op: osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.956317 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_applied: 236361
2014-06-12 14:46:54.956342 7f9d7795c700  5 -- op tracker -- , seq: 748163, time: 2014-06-12 14:46:54.956332, event: op_applied, op: osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:54.956349 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_applied on primary on version 173'2539
2014-06-12 14:46:54.959025 7f9d7d968700 10 _generic_write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 1187840~1548288 = 0
2014-06-12 14:46:54.959150 7f9d7d968700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:54.959169 7f9d7d968700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:46:54.959176 7f9d7d968700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:54.959184 7f9d7d968700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:46:54.959546 7f9d7d968700 20 set_header setting 17709 cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 parent seq 0
2014-06-12 14:46:54.959658 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:54.959676 7f9d7d968700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:54.959689 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:54.965670 7f9d7d968700 10 _do_op 0xb045f20 seq 2805 r = 0, finisher 0x13d128a0 0
2014-06-12 14:46:54.965734 7f9d7715b700  5 -- op tracker -- , seq: 748164, time: 2014-06-12 14:46:54.965724, event: commit_sent, op: osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.965753 7f9d7d968700 10 _finish_op 0xb045f20 seq 2805 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:54.965745 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3003 (79'2,173'3003] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3001 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:54.965771 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:54.965779 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:54.965788 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:54.965823 7f9d7795c700  5 -- op tracker -- , seq: 748164, time: 2014-06-12 14:46:54.965816, event: sub_op_applied, op: osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:54.965838 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3003 (79'2,173'3003] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3002 active] sub_op_modify_applied on 0x13f841c0 op osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:54.965854 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3003 (79'2,173'3003] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3002 active] op_applied on primary on version 173'3003
2014-06-12 14:46:54.965861 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3003 (79'2,173'3003] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3002 active] op_applied on replica on version 173'3003
2014-06-12 14:46:54.965876 7f9d7795c700  5 -- op tracker -- , seq: 748164, time: 2014-06-12 14:46:54.965873, event: done, op: osd_sub_op(client.4299.0:145535 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3003 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.111937 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:55.112328 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:55.159161 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:55.159167 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:55.159169 7f9d74bfa700 20 osd.2 173 _dispatch 0xa24eb60 ping v1
2014-06-12 14:46:55.159171 7f9d74bfa700 10 osd.2 173 ping from client.4293
2014-06-12 14:46:55.159172 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:55.159173 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:55.173353 7f9d5bec4700  5 -- op tracker -- , seq: 748165, time: 2014-06-12 14:46:55.172658, event: header_read, op: osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.173368 7f9d5bec4700  5 -- op tracker -- , seq: 748165, time: 2014-06-12 14:46:55.172662, event: throttled, op: osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.173376 7f9d5bec4700  5 -- op tracker -- , seq: 748165, time: 2014-06-12 14:46:55.173311, event: all_read, op: osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.173381 7f9d5bec4700  5 -- op tracker -- , seq: 748165, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.173385 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:55.173390 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:55.173396 7f9d5bec4700 15 osd.2 173 enqueue_op 0x876f700 prio 127 cost 402215 latency 0.000738 osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.173428 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:55.173440 7f9d63bd8700 10 osd.2 173 dequeue_op 0x876f700 prio 127 cost 402215 latency 0.000782 osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.56d( v 173'2895 (0'0,173'2895] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2894 active]
2014-06-12 14:46:55.173468 7f9d63bd8700  5 -- op tracker -- , seq: 748165, time: 2014-06-12 14:46:55.173463, event: reached_pg, op: osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.173480 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2895 (0'0,173'2895] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2894 active] handle_message: 0x876f700
2014-06-12 14:46:55.173495 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2895 (0'0,173'2895] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2894 active] sub_op_modify trans b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 v 173'2896 (transaction) 174
2014-06-12 14:46:55.173514 7f9d63bd8700  5 -- op tracker -- , seq: 748165, time: 2014-06-12 14:46:55.173510, event: started, op: osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.173533 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2895 (0'0,173'2895] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2894 active] append_log log((0'0,173'2895], crt=0'0) [173'2896 (173'716) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144131 2014-06-12 08:45:57.514498]
2014-06-12 14:46:55.173559 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2896 (0'0,173'2896] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2894 active] add_log_entry 173'2896 (173'716) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144131 2014-06-12 08:45:57.514498
2014-06-12 14:46:55.173584 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2896 (0'0,173'2896] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2894 active] append_log  adding 1 keys
2014-06-12 14:46:55.173631 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2896, trimmed: 
2014-06-12 14:46:55.173665 7f9d63bd8700  5 -- op tracker -- , seq: 748165, time: 2014-06-12 14:46:55.173653, event: started, op: osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.173677 7f9d63bd8700  5 queue_transactions existing osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.173682 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:55.173683 7f9d63bd8700  5 queue_op 0x696bda0 seq 2669 osr(3.56d 0x276daa0) 403701 bytes   (queue has 1 ops and 403701 bytes)
2014-06-12 14:46:55.173694 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541d440 (1 active)
2014-06-12 14:46:55.173690 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.56d( v 173'2896 (0'0,173'2896] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2894 active] do_sub_op osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.173704 7f9d63bd8700 10 osd.2 173 dequeue_op 0x876f700 finish
2014-06-12 14:46:55.173704 7f9d7c165700  5 _do_op 0x696bda0 seq 2669 osr(3.56d 0x276daa0)/0x276daa0 start
2014-06-12 14:46:55.173708 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:55.173709 7f9d7c165700 10 _do_transaction on 0x6318d30
2014-06-12 14:46:55.173711 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:55.173715 7f9d7c165700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.173949 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.173967 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:55.174011 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:55.174027 7f9d7c165700 15 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.174032 7f9d7c165700 10 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1 r = 0
2014-06-12 14:46:55.174036 7f9d7c165700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.174047 7f9d7c165700 15 _write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 0~401408
2014-06-12 14:46:55.174310 7f9d7c165700 10 lookup_strip_header done  cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.174416 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:55.175592 7f9d7c165700 10 _generic_write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 0~401408 = 0
2014-06-12 14:46:55.175648 7f9d7c165700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.175663 7f9d7c165700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.175676 7f9d7c165700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.175683 7f9d7c165700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.175878 7f9d7c165700 20 set_header setting 15606 cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 parent seq 0
2014-06-12 14:46:55.175954 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:55.175971 7f9d7c165700 20 set_header setting 2238 cid meta oid 1fe11a4b/pglog_3.56d/0//-1 parent seq 0
2014-06-12 14:46:55.175988 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:55.178370 7f9d7c165700 10 _do_op 0x696bda0 seq 2669 r = 0, finisher 0x20aad300 0
2014-06-12 14:46:55.178446 7f9d7c165700 10 _finish_op 0x696bda0 seq 2669 osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.178470 7f9d7715b700  5 -- op tracker -- , seq: 748165, time: 2014-06-12 14:46:55.178458, event: commit_sent, op: osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.178471 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541d440 (0 active)
2014-06-12 14:46:55.178488 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:55.178487 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2896 (0'0,173'2896] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2894 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:55.178519 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:55.178556 7f9d7795c700  5 -- op tracker -- , seq: 748165, time: 2014-06-12 14:46:55.178549, event: sub_op_applied, op: osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.178564 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2896 (0'0,173'2896] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2895 active] sub_op_modify_applied on 0x6318c40 op osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.178581 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2896 (0'0,173'2896] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2895 active] op_applied on primary on version 173'2896
2014-06-12 14:46:55.178591 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2896 (0'0,173'2896] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2895 active] op_applied on replica on version 173'2896
2014-06-12 14:46:55.178613 7f9d7795c700  5 -- op tracker -- , seq: 748165, time: 2014-06-12 14:46:55.178608, event: done, op: osd_sub_op(client.4297.0:144131 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2896 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.184570 7f9d5bec4700  5 -- op tracker -- , seq: 748166, time: 2014-06-12 14:46:55.183795, event: header_read, op: osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.184581 7f9d5bec4700  5 -- op tracker -- , seq: 748166, time: 2014-06-12 14:46:55.183798, event: throttled, op: osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.184599 7f9d5bec4700  5 -- op tracker -- , seq: 748166, time: 2014-06-12 14:46:55.184534, event: all_read, op: osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.184604 7f9d5bec4700  5 -- op tracker -- , seq: 748166, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.184609 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:55.184619 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:55.184625 7f9d5bec4700 15 osd.2 173 enqueue_op 0x876f800 prio 127 cost 516903 latency 0.000830 osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.184689 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:55.184701 7f9d633d7700 10 osd.2 173 dequeue_op 0x876f800 prio 127 cost 516903 latency 0.000905 osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.56d( v 173'2896 (0'0,173'2896] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2895 active]
2014-06-12 14:46:55.184728 7f9d633d7700  5 -- op tracker -- , seq: 748166, time: 2014-06-12 14:46:55.184723, event: reached_pg, op: osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.184739 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2896 (0'0,173'2896] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2895 active] handle_message: 0x876f800
2014-06-12 14:46:55.184754 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2896 (0'0,173'2896] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2895 active] sub_op_modify trans b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 v 173'2897 (transaction) 174
2014-06-12 14:46:55.184774 7f9d633d7700  5 -- op tracker -- , seq: 748166, time: 2014-06-12 14:46:55.184769, event: started, op: osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.184793 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2896 (0'0,173'2896] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2895 active] append_log log((0'0,173'2896], crt=0'0) [173'2897 (173'2896) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144132 2014-06-12 08:45:57.560799]
2014-06-12 14:46:55.184821 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2897 (0'0,173'2897] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2895 active] add_log_entry 173'2897 (173'2896) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144132 2014-06-12 08:45:57.560799
2014-06-12 14:46:55.184846 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2897 (0'0,173'2897] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2895 active] append_log  adding 1 keys
2014-06-12 14:46:55.184883 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2897, trimmed: 
2014-06-12 14:46:55.184908 7f9d633d7700  5 -- op tracker -- , seq: 748166, time: 2014-06-12 14:46:55.184903, event: started, op: osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.184919 7f9d633d7700  5 queue_transactions existing osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.184925 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:55.184927 7f9d633d7700  5 queue_op 0x165af920 seq 2670 osr(3.56d 0x276daa0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:55.184935 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.56d( v 173'2897 (0'0,173'2897] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2895 active] do_sub_op osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.184950 7f9d633d7700 10 osd.2 173 dequeue_op 0x876f800 finish
2014-06-12 14:46:55.184954 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:55.184956 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:55.184964 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541d440 (1 active)
2014-06-12 14:46:55.184971 7f9d8096e700  5 _do_op 0x165af920 seq 2670 osr(3.56d 0x276daa0)/0x276daa0 start
2014-06-12 14:46:55.184976 7f9d8096e700 10 _do_transaction on 0x7d47cf0
2014-06-12 14:46:55.184981 7f9d8096e700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.185117 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.185136 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:55.185195 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:55.185291 7f9d8096e700 15 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.185299 7f9d8096e700 10 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1 r = 0
2014-06-12 14:46:55.185305 7f9d8096e700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.185321 7f9d8096e700 15 _write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 401408~516096
2014-06-12 14:46:55.185448 7f9d8096e700 10 lookup_strip_header done  cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.185588 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:55.187739 7f9d8096e700 10 _generic_write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 401408~516096 = 0
2014-06-12 14:46:55.187813 7f9d8096e700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.187833 7f9d8096e700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.187841 7f9d8096e700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.187849 7f9d8096e700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.188069 7f9d8096e700 20 set_header setting 15606 cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 parent seq 0
2014-06-12 14:46:55.188155 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:55.188173 7f9d8096e700 20 set_header setting 2238 cid meta oid 1fe11a4b/pglog_3.56d/0//-1 parent seq 0
2014-06-12 14:46:55.188200 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:55.190868 7f9d8096e700 10 _do_op 0x165af920 seq 2670 r = 0, finisher 0x13d12000 0
2014-06-12 14:46:55.190937 7f9d7715b700  5 -- op tracker -- , seq: 748166, time: 2014-06-12 14:46:55.190928, event: commit_sent, op: osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.190947 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2897 (0'0,173'2897] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2895 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:55.190971 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:55.191064 7f9d8096e700 10 _finish_op 0x165af920 seq 2670 osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.191102 7f9d7795c700  5 -- op tracker -- , seq: 748166, time: 2014-06-12 14:46:55.191097, event: sub_op_applied, op: osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.191113 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2897 (0'0,173'2897] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2896 active] sub_op_modify_applied on 0x7d47c00 op osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.191130 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2897 (0'0,173'2897] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2896 active] op_applied on primary on version 173'2897
2014-06-12 14:46:55.191139 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2897 (0'0,173'2897] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2896 active] op_applied on replica on version 173'2897
2014-06-12 14:46:55.191216 7f9d8096e700  5 -- op tracker -- , seq: 748166, time: 2014-06-12 14:46:55.191206, event: done, op: osd_sub_op(client.4297.0:144132 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2897 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.191263 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541d440 (0 active)
2014-06-12 14:46:55.191285 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:55.198084 7f9d5bec4700  5 -- op tracker -- , seq: 748167, time: 2014-06-12 14:46:55.197182, event: header_read, op: osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.198097 7f9d5bec4700  5 -- op tracker -- , seq: 748167, time: 2014-06-12 14:46:55.197184, event: throttled, op: osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.198106 7f9d5bec4700  5 -- op tracker -- , seq: 748167, time: 2014-06-12 14:46:55.198049, event: all_read, op: osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.198115 7f9d5bec4700  5 -- op tracker -- , seq: 748167, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.198120 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:55.198127 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:55.198136 7f9d5bec4700 15 osd.2 173 enqueue_op 0x876f000 prio 127 cost 516903 latency 0.000954 osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.198163 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:55.198171 7f9d61bd4700 10 osd.2 173 dequeue_op 0x876f000 prio 127 cost 516903 latency 0.000989 osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.56d( v 173'2897 (0'0,173'2897] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2896 active]
2014-06-12 14:46:55.198199 7f9d61bd4700  5 -- op tracker -- , seq: 748167, time: 2014-06-12 14:46:55.198194, event: reached_pg, op: osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.198211 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2897 (0'0,173'2897] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2896 active] handle_message: 0x876f000
2014-06-12 14:46:55.198225 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2897 (0'0,173'2897] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2896 active] sub_op_modify trans b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 v 173'2898 (transaction) 174
2014-06-12 14:46:55.198243 7f9d61bd4700  5 -- op tracker -- , seq: 748167, time: 2014-06-12 14:46:55.198238, event: started, op: osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.198258 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2897 (0'0,173'2897] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2896 active] append_log log((0'0,173'2897], crt=0'0) [173'2898 (173'2897) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144133 2014-06-12 08:45:57.573552]
2014-06-12 14:46:55.198284 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2898 (0'0,173'2898] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2896 active] add_log_entry 173'2898 (173'2897) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144133 2014-06-12 08:45:57.573552
2014-06-12 14:46:55.198313 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2898 (0'0,173'2898] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2896 active] append_log  adding 1 keys
2014-06-12 14:46:55.198348 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2898, trimmed: 
2014-06-12 14:46:55.198372 7f9d61bd4700  5 -- op tracker -- , seq: 748167, time: 2014-06-12 14:46:55.198367, event: started, op: osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.198383 7f9d61bd4700  5 queue_transactions existing osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.198387 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:55.198389 7f9d61bd4700  5 queue_op 0x7713c20 seq 2671 osr(3.56d 0x276daa0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:55.198398 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541d440 (1 active)
2014-06-12 14:46:55.198406 7f9d7c966700  5 _do_op 0x7713c20 seq 2671 osr(3.56d 0x276daa0)/0x276daa0 start
2014-06-12 14:46:55.198395 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.56d( v 173'2898 (0'0,173'2898] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2896 active] do_sub_op osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.198409 7f9d61bd4700 10 osd.2 173 dequeue_op 0x876f000 finish
2014-06-12 14:46:55.198411 7f9d7c966700 10 _do_transaction on 0x2e9b20f0
2014-06-12 14:46:55.198413 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:55.198415 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:55.198418 7f9d7c966700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.198541 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.198565 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:55.198616 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:55.198750 7f9d7c966700 15 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.198759 7f9d7c966700 10 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1 r = 0
2014-06-12 14:46:55.198767 7f9d7c966700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.198786 7f9d7c966700 15 _write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 917504~516096
2014-06-12 14:46:55.198899 7f9d7c966700 10 lookup_strip_header done  cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.199077 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:55.200566 7f9d7c966700 10 _generic_write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 917504~516096 = 0
2014-06-12 14:46:55.200611 7f9d7c966700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.200626 7f9d7c966700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.200633 7f9d7c966700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.200642 7f9d7c966700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.200789 7f9d7c966700 20 set_header setting 15606 cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 parent seq 0
2014-06-12 14:46:55.200854 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:55.200872 7f9d7c966700 20 set_header setting 2238 cid meta oid 1fe11a4b/pglog_3.56d/0//-1 parent seq 0
2014-06-12 14:46:55.200886 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:55.203553 7f9d7c966700 10 _do_op 0x7713c20 seq 2671 r = 0, finisher 0x6dbcac0 0
2014-06-12 14:46:55.203658 7f9d7715b700  5 -- op tracker -- , seq: 748167, time: 2014-06-12 14:46:55.203636, event: commit_sent, op: osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.203672 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2898 (0'0,173'2898] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2896 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:55.203709 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:55.203813 7f9d7c966700 10 _finish_op 0x7713c20 seq 2671 osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.203825 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541d440 (0 active)
2014-06-12 14:46:55.203829 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:55.203844 7f9d7795c700  5 -- op tracker -- , seq: 748167, time: 2014-06-12 14:46:55.203839, event: sub_op_applied, op: osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.203851 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2898 (0'0,173'2898] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2897 active] sub_op_modify_applied on 0x2e9b2000 op osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.203865 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2898 (0'0,173'2898] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2897 active] op_applied on primary on version 173'2898
2014-06-12 14:46:55.203873 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2898 (0'0,173'2898] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2897 active] op_applied on replica on version 173'2898
2014-06-12 14:46:55.203900 7f9d7795c700  5 -- op tracker -- , seq: 748167, time: 2014-06-12 14:46:55.203896, event: done, op: osd_sub_op(client.4297.0:144133 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2898 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.210312 7f9d5bec4700  5 -- op tracker -- , seq: 748168, time: 2014-06-12 14:46:55.209382, event: header_read, op: osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.210324 7f9d5bec4700  5 -- op tracker -- , seq: 748168, time: 2014-06-12 14:46:55.209385, event: throttled, op: osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.210340 7f9d5bec4700  5 -- op tracker -- , seq: 748168, time: 2014-06-12 14:46:55.210278, event: all_read, op: osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.210346 7f9d5bec4700  5 -- op tracker -- , seq: 748168, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.210350 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:55.210355 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:55.210361 7f9d5bec4700 15 osd.2 173 enqueue_op 0x22b3a400 prio 127 cost 516903 latency 0.000979 osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.210391 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:55.210400 7f9d67be0700 10 osd.2 173 dequeue_op 0x22b3a400 prio 127 cost 516903 latency 0.001018 osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.56d( v 173'2898 (0'0,173'2898] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2897 active]
2014-06-12 14:46:55.210431 7f9d67be0700  5 -- op tracker -- , seq: 748168, time: 2014-06-12 14:46:55.210426, event: reached_pg, op: osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.210445 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2898 (0'0,173'2898] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2897 active] handle_message: 0x22b3a400
2014-06-12 14:46:55.210462 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2898 (0'0,173'2898] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2897 active] sub_op_modify trans b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 v 173'2899 (transaction) 174
2014-06-12 14:46:55.210484 7f9d67be0700  5 -- op tracker -- , seq: 748168, time: 2014-06-12 14:46:55.210480, event: started, op: osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.210506 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2898 (0'0,173'2898] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2897 active] append_log log((0'0,173'2898], crt=0'0) [173'2899 (173'2898) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144134 2014-06-12 08:45:57.585869]
2014-06-12 14:46:55.210534 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2899 (0'0,173'2899] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2897 active] add_log_entry 173'2899 (173'2898) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144134 2014-06-12 08:45:57.585869
2014-06-12 14:46:55.210559 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2899 (0'0,173'2899] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2897 active] append_log  adding 1 keys
2014-06-12 14:46:55.210596 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2899, trimmed: 
2014-06-12 14:46:55.210624 7f9d67be0700  5 -- op tracker -- , seq: 748168, time: 2014-06-12 14:46:55.210619, event: started, op: osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.210637 7f9d67be0700  5 queue_transactions existing osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.210642 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:55.210644 7f9d67be0700  5 queue_op 0x93609c0 seq 2672 osr(3.56d 0x276daa0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:55.210653 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541d440 (1 active)
2014-06-12 14:46:55.210659 7f9d81970700  5 _do_op 0x93609c0 seq 2672 osr(3.56d 0x276daa0)/0x276daa0 start
2014-06-12 14:46:55.210662 7f9d81970700 10 _do_transaction on 0x52c82b0
2014-06-12 14:46:55.210650 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.56d( v 173'2899 (0'0,173'2899] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2897 active] do_sub_op osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.210665 7f9d67be0700 10 osd.2 173 dequeue_op 0x22b3a400 finish
2014-06-12 14:46:55.210667 7f9d81970700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.210669 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:55.210672 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:55.210777 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.210800 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:55.210849 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:55.210867 7f9d81970700 15 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.210873 7f9d81970700 10 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1 r = 0
2014-06-12 14:46:55.210876 7f9d81970700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.210888 7f9d81970700 15 _write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 1433600~516096
2014-06-12 14:46:55.211018 7f9d81970700 10 lookup_strip_header done  cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.211195 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:55.212930 7f9d81970700 10 _generic_write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 1433600~516096 = 0
2014-06-12 14:46:55.212981 7f9d81970700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.212998 7f9d81970700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.213005 7f9d81970700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.213014 7f9d81970700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.213218 7f9d81970700 20 set_header setting 15606 cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 parent seq 0
2014-06-12 14:46:55.213302 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:55.213320 7f9d81970700 20 set_header setting 2238 cid meta oid 1fe11a4b/pglog_3.56d/0//-1 parent seq 0
2014-06-12 14:46:55.213331 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:55.216291 7f9d81970700 10 _do_op 0x93609c0 seq 2672 r = 0, finisher 0x7bd8be0 0
2014-06-12 14:46:55.216316 7f9d81970700 10 _finish_op 0x93609c0 seq 2672 osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.216333 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541d440 (0 active)
2014-06-12 14:46:55.216343 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:55.216336 7f9d7715b700  5 -- op tracker -- , seq: 748168, time: 2014-06-12 14:46:55.216325, event: commit_sent, op: osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.216372 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2899 (0'0,173'2899] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2897 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:55.216416 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:55.216461 7f9d7795c700  5 -- op tracker -- , seq: 748168, time: 2014-06-12 14:46:55.216455, event: sub_op_applied, op: osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.216472 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2899 (0'0,173'2899] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2898 active] sub_op_modify_applied on 0x52c81c0 op osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.216487 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2899 (0'0,173'2899] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2898 active] op_applied on primary on version 173'2899
2014-06-12 14:46:55.216493 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2899 (0'0,173'2899] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2898 active] op_applied on replica on version 173'2899
2014-06-12 14:46:55.216510 7f9d7795c700  5 -- op tracker -- , seq: 748168, time: 2014-06-12 14:46:55.216507, event: done, op: osd_sub_op(client.4297.0:144134 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2899 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.221953 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:55.222156 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:55.224392 7f9d5bec4700  5 -- op tracker -- , seq: 748169, time: 2014-06-12 14:46:55.223167, event: header_read, op: osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.224407 7f9d5bec4700  5 -- op tracker -- , seq: 748169, time: 2014-06-12 14:46:55.223169, event: throttled, op: osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.224414 7f9d5bec4700  5 -- op tracker -- , seq: 748169, time: 2014-06-12 14:46:55.224345, event: all_read, op: osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.224419 7f9d5bec4700  5 -- op tracker -- , seq: 748169, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.224424 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:55.224431 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:55.224440 7f9d5bec4700 15 osd.2 173 enqueue_op 0x22b3b200 prio 127 cost 516903 latency 0.001273 osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.224481 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:55.224487 7f9d673df700 10 osd.2 173 dequeue_op 0x22b3b200 prio 127 cost 516903 latency 0.001320 osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.56d( v 173'2899 (0'0,173'2899] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2898 active]
2014-06-12 14:46:55.224506 7f9d673df700  5 -- op tracker -- , seq: 748169, time: 2014-06-12 14:46:55.224502, event: reached_pg, op: osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.224515 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2899 (0'0,173'2899] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2898 active] handle_message: 0x22b3b200
2014-06-12 14:46:55.224528 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2899 (0'0,173'2899] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2898 active] sub_op_modify trans b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 v 173'2900 (transaction) 174
2014-06-12 14:46:55.224545 7f9d673df700  5 -- op tracker -- , seq: 748169, time: 2014-06-12 14:46:55.224542, event: started, op: osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.224558 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2899 (0'0,173'2899] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2898 active] append_log log((0'0,173'2899], crt=0'0) [173'2900 (173'2899) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144135 2014-06-12 08:45:57.598732]
2014-06-12 14:46:55.224583 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2900 (0'0,173'2900] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2898 active] add_log_entry 173'2900 (173'2899) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144135 2014-06-12 08:45:57.598732
2014-06-12 14:46:55.224608 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2900 (0'0,173'2900] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2898 active] append_log  adding 1 keys
2014-06-12 14:46:55.224644 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2900, trimmed: 
2014-06-12 14:46:55.224665 7f9d673df700  5 -- op tracker -- , seq: 748169, time: 2014-06-12 14:46:55.224661, event: started, op: osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.224675 7f9d673df700  5 queue_transactions existing osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.224679 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:55.224687 7f9d673df700  5 queue_op 0x92517a0 seq 2673 osr(3.56d 0x276daa0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:55.224694 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.56d( v 173'2900 (0'0,173'2900] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2898 active] do_sub_op osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.224706 7f9d673df700 10 osd.2 173 dequeue_op 0x22b3b200 finish
2014-06-12 14:46:55.224710 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:55.224712 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:55.224717 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541d440 (1 active)
2014-06-12 14:46:55.224722 7f9d7a962700  5 _do_op 0x92517a0 seq 2673 osr(3.56d 0x276daa0)/0x276daa0 start
2014-06-12 14:46:55.224725 7f9d7a962700 10 _do_transaction on 0x1da3e630
2014-06-12 14:46:55.224730 7f9d7a962700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.224855 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.224876 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:55.224928 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:55.225029 7f9d7a962700 15 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.225037 7f9d7a962700 10 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1 r = 0
2014-06-12 14:46:55.225042 7f9d7a962700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.225059 7f9d7a962700 15 _write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 1949696~516096
2014-06-12 14:46:55.225214 7f9d7a962700 10 lookup_strip_header done  cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.225419 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:55.227351 7f9d7a962700 10 _generic_write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 1949696~516096 = 0
2014-06-12 14:46:55.227403 7f9d7a962700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.227418 7f9d7a962700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.227424 7f9d7a962700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.227434 7f9d7a962700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.227627 7f9d7a962700 20 set_header setting 15606 cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 parent seq 0
2014-06-12 14:46:55.227702 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:55.227718 7f9d7a962700 20 set_header setting 2238 cid meta oid 1fe11a4b/pglog_3.56d/0//-1 parent seq 0
2014-06-12 14:46:55.227728 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:55.230597 7f9d7a962700 10 _do_op 0x92517a0 seq 2673 r = 0, finisher 0x6dbc560 0
2014-06-12 14:46:55.230618 7f9d7a962700 10 _finish_op 0x92517a0 seq 2673 osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.230637 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541d440 (0 active)
2014-06-12 14:46:55.230639 7f9d7715b700  5 -- op tracker -- , seq: 748169, time: 2014-06-12 14:46:55.230627, event: commit_sent, op: osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.230647 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:55.230649 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2900 (0'0,173'2900] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2898 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:55.230679 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:55.230717 7f9d7795c700  5 -- op tracker -- , seq: 748169, time: 2014-06-12 14:46:55.230710, event: sub_op_applied, op: osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.230726 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2900 (0'0,173'2900] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2899 active] sub_op_modify_applied on 0x1da3e540 op osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.230743 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2900 (0'0,173'2900] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2899 active] op_applied on primary on version 173'2900
2014-06-12 14:46:55.230752 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2900 (0'0,173'2900] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2899 active] op_applied on replica on version 173'2900
2014-06-12 14:46:55.230769 7f9d7795c700  5 -- op tracker -- , seq: 748169, time: 2014-06-12 14:46:55.230767, event: done, op: osd_sub_op(client.4297.0:144135 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2900 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.237054 7f9d5bec4700  5 -- op tracker -- , seq: 748170, time: 2014-06-12 14:46:55.235997, event: header_read, op: osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.237072 7f9d5bec4700  5 -- op tracker -- , seq: 748170, time: 2014-06-12 14:46:55.236000, event: throttled, op: osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.237080 7f9d5bec4700  5 -- op tracker -- , seq: 748170, time: 2014-06-12 14:46:55.236999, event: all_read, op: osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.237088 7f9d5bec4700  5 -- op tracker -- , seq: 748170, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.237102 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:55.237109 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:55.237118 7f9d5bec4700 15 osd.2 173 enqueue_op 0x22b3bc00 prio 127 cost 516903 latency 0.001121 osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.237153 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:55.237182 7f9d663dd700 10 osd.2 173 dequeue_op 0x22b3bc00 prio 127 cost 516903 latency 0.001166 osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.56d( v 173'2900 (0'0,173'2900] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2899 active]
2014-06-12 14:46:55.237209 7f9d663dd700  5 -- op tracker -- , seq: 748170, time: 2014-06-12 14:46:55.237205, event: reached_pg, op: osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.237230 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2900 (0'0,173'2900] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2899 active] handle_message: 0x22b3bc00
2014-06-12 14:46:55.237244 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2900 (0'0,173'2900] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2899 active] sub_op_modify trans b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 v 173'2901 (transaction) 174
2014-06-12 14:46:55.237262 7f9d663dd700  5 -- op tracker -- , seq: 748170, time: 2014-06-12 14:46:55.237257, event: started, op: osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.237308 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2900 (0'0,173'2900] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2899 active] append_log log((0'0,173'2900], crt=0'0) [173'2901 (173'2900) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144136 2014-06-12 08:45:57.612925]
2014-06-12 14:46:55.237339 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2901 (0'0,173'2901] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2899 active] add_log_entry 173'2901 (173'2900) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144136 2014-06-12 08:45:57.612925
2014-06-12 14:46:55.237363 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2901 (0'0,173'2901] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2899 active] append_log  adding 1 keys
2014-06-12 14:46:55.237424 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2901, trimmed: 
2014-06-12 14:46:55.237453 7f9d663dd700  5 -- op tracker -- , seq: 748170, time: 2014-06-12 14:46:55.237448, event: started, op: osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.237467 7f9d663dd700  5 queue_transactions existing osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.237471 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:55.237473 7f9d663dd700  5 queue_op 0x5f67a40 seq 2674 osr(3.56d 0x276daa0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:55.237480 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.56d( v 173'2901 (0'0,173'2901] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2899 active] do_sub_op osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.237495 7f9d663dd700 10 osd.2 173 dequeue_op 0x22b3bc00 finish
2014-06-12 14:46:55.237499 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:55.237502 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:55.237511 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541d440 (1 active)
2014-06-12 14:46:55.237516 7f9d82972700  5 _do_op 0x5f67a40 seq 2674 osr(3.56d 0x276daa0)/0x276daa0 start
2014-06-12 14:46:55.237520 7f9d82972700 10 _do_transaction on 0x60b5970
2014-06-12 14:46:55.237526 7f9d82972700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.237686 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.237706 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:55.237772 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:55.237800 7f9d82972700 15 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.237808 7f9d82972700 10 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1 r = 0
2014-06-12 14:46:55.237814 7f9d82972700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.237831 7f9d82972700 15 _write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 2465792~516096
2014-06-12 14:46:55.238001 7f9d82972700 10 lookup_strip_header done  cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.238136 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:55.240752 7f9d82972700 10 _generic_write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 2465792~516096 = 0
2014-06-12 14:46:55.240815 7f9d82972700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.240837 7f9d82972700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.240843 7f9d82972700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.240857 7f9d82972700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.240980 7f9d5a16f700  5 -- op tracker -- , seq: 748171, time: 2014-06-12 14:46:55.239305, event: header_read, op: osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.241058 7f9d5a16f700  5 -- op tracker -- , seq: 748171, time: 2014-06-12 14:46:55.239309, event: throttled, op: osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.241067 7f9d5a16f700  5 -- op tracker -- , seq: 748171, time: 2014-06-12 14:46:55.240954, event: all_read, op: osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.241075 7f9d82972700 20 set_header setting 15606 cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 parent seq 0
2014-06-12 14:46:55.241080 7f9d5a16f700  5 -- op tracker -- , seq: 748171, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.241085 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:55.241092 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:55.241100 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2910500 prio 127 cost 1032999 latency 0.001795 osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.241139 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:55.241142 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:55.241158 7f9d82972700 20 set_header setting 2238 cid meta oid 1fe11a4b/pglog_3.56d/0//-1 parent seq 0
2014-06-12 14:46:55.241146 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2910500 prio 127 cost 1032999 latency 0.001841 osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2506 (0'0,173'2506] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2505 active]
2014-06-12 14:46:55.241169 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:55.241168 7f9d65bdc700  5 -- op tracker -- , seq: 748171, time: 2014-06-12 14:46:55.241164, event: reached_pg, op: osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.241176 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2506 (0'0,173'2506] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2505 active] handle_message: 0x2910500
2014-06-12 14:46:55.241196 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2506 (0'0,173'2506] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2505 active] sub_op_modify trans a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 v 173'2507 (transaction) 174
2014-06-12 14:46:55.241210 7f9d65bdc700  5 -- op tracker -- , seq: 748171, time: 2014-06-12 14:46:55.241207, event: started, op: osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.241228 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2506 (0'0,173'2506] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2505 active] append_log log((0'0,173'2506], crt=0'0) [173'2507 (173'2506) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146442 2014-06-12 08:41:37.906217]
2014-06-12 14:46:55.241258 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2507 (0'0,173'2507] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2505 active] add_log_entry 173'2507 (173'2506) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146442 2014-06-12 08:41:37.906217
2014-06-12 14:46:55.241285 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2507 (0'0,173'2507] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2505 active] append_log  adding 1 keys
2014-06-12 14:46:55.241322 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2507, trimmed: 
2014-06-12 14:46:55.241362 7f9d65bdc700  5 -- op tracker -- , seq: 748171, time: 2014-06-12 14:46:55.241356, event: started, op: osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.241377 7f9d65bdc700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:46:55.241381 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:55.241396 7f9d65bdc700  5 queue_op 0x1697d8c0 seq 2353 osr(3.168 0x276ce40) 1034485 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:46:55.241406 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2507 (0'0,173'2507] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2505 active] do_sub_op osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.241421 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2910500 finish
2014-06-12 14:46:55.241424 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:55.241436 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:55.241435 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (2 active)
2014-06-12 14:46:55.241449 7f9d7895e700  5 _do_op 0x1697d8c0 seq 2353 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:46:55.241549 7f9d7895e700 10 _do_transaction on 0x6220630
2014-06-12 14:46:55.241700 7f9d7895e700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:55.241921 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:55.241945 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:55.244041 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:55.244072 7f9d7895e700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:55.244077 7f9d7895e700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:46:55.244083 7f9d7895e700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:46:55.244096 7f9d7895e700 15 _write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 1581056~1032192
2014-06-12 14:46:55.244162 7f9d82972700 10 _do_op 0x5f67a40 seq 2674 r = 0, finisher 0x7bd8f00 0
2014-06-12 14:46:55.244174 7f9d82972700 10 _finish_op 0x5f67a40 seq 2674 osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.244181 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541d440 (1 active)
2014-06-12 14:46:55.244188 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:55.244191 7f9d7715b700  5 -- op tracker -- , seq: 748170, time: 2014-06-12 14:46:55.244182, event: commit_sent, op: osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.244201 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2901 (0'0,173'2901] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2899 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:55.244227 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:55.244256 7f9d7795c700  5 -- op tracker -- , seq: 748170, time: 2014-06-12 14:46:55.244251, event: sub_op_applied, op: osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.244263 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2901 (0'0,173'2901] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2900 active] sub_op_modify_applied on 0x60b5880 op osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.244281 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2901 (0'0,173'2901] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2900 active] op_applied on primary on version 173'2901
2014-06-12 14:46:55.244293 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2901 (0'0,173'2901] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2900 active] op_applied on replica on version 173'2901
2014-06-12 14:46:55.244316 7f9d7895e700 10 lookup_strip_header done  cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:46:55.244321 7f9d7795c700  5 -- op tracker -- , seq: 748170, time: 2014-06-12 14:46:55.244316, event: done, op: osd_sub_op(client.4297.0:144136 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2901 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.244343 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:55.247004 7f9d7895e700 10 _generic_write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 1581056~1032192 = 0
2014-06-12 14:46:55.247071 7f9d7895e700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:46:55.247098 7f9d7895e700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:46:55.247104 7f9d7895e700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:46:55.247109 7f9d7895e700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:46:55.247312 7f9d7895e700 20 set_header setting 16934 cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 parent seq 0
2014-06-12 14:46:55.247386 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:55.247397 7f9d7895e700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:46:55.247403 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:55.250260 7f9d5bec4700  5 -- op tracker -- , seq: 748172, time: 2014-06-12 14:46:55.249497, event: header_read, op: osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.250273 7f9d5bec4700  5 -- op tracker -- , seq: 748172, time: 2014-06-12 14:46:55.249498, event: throttled, op: osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.250288 7f9d5bec4700  5 -- op tracker -- , seq: 748172, time: 2014-06-12 14:46:55.250222, event: all_read, op: osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.250293 7f9d5bec4700  5 -- op tracker -- , seq: 748172, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.250298 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:55.250303 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:55.250308 7f9d5bec4700 15 osd.2 173 enqueue_op 0x22b3a000 prio 127 cost 516903 latency 0.000811 osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.250332 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:55.250341 7f9d62bd6700 10 osd.2 173 dequeue_op 0x22b3a000 prio 127 cost 516903 latency 0.000843 osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.56d( v 173'2901 (0'0,173'2901] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2900 active]
2014-06-12 14:46:55.250365 7f9d62bd6700  5 -- op tracker -- , seq: 748172, time: 2014-06-12 14:46:55.250361, event: reached_pg, op: osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.250382 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2901 (0'0,173'2901] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2900 active] handle_message: 0x22b3a000
2014-06-12 14:46:55.250398 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2901 (0'0,173'2901] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2900 active] sub_op_modify trans b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 v 173'2902 (transaction) 174
2014-06-12 14:46:55.250415 7f9d62bd6700  5 -- op tracker -- , seq: 748172, time: 2014-06-12 14:46:55.250411, event: started, op: osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.250431 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2901 (0'0,173'2901] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2900 active] append_log log((0'0,173'2901], crt=0'0) [173'2902 (173'2901) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144137 2014-06-12 08:45:57.626414]
2014-06-12 14:46:55.250455 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2902 (0'0,173'2902] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2900 active] add_log_entry 173'2902 (173'2901) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144137 2014-06-12 08:45:57.626414
2014-06-12 14:46:55.250478 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2902 (0'0,173'2902] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2900 active] append_log  adding 1 keys
2014-06-12 14:46:55.250516 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2902, trimmed: 
2014-06-12 14:46:55.250551 7f9d62bd6700  5 -- op tracker -- , seq: 748172, time: 2014-06-12 14:46:55.250546, event: started, op: osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.250566 7f9d62bd6700  5 queue_transactions existing osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.250572 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:55.250574 7f9d62bd6700  5 queue_op 0x5d507e0 seq 2675 osr(3.56d 0x276daa0) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:46:55.250587 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541d440 (2 active)
2014-06-12 14:46:55.250592 7f9d83173700  5 _do_op 0x5d507e0 seq 2675 osr(3.56d 0x276daa0)/0x276daa0 start
2014-06-12 14:46:55.250596 7f9d83173700 10 _do_transaction on 0x570c0f0
2014-06-12 14:46:55.250584 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.56d( v 173'2902 (0'0,173'2902] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2900 active] do_sub_op osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.250598 7f9d62bd6700 10 osd.2 173 dequeue_op 0x22b3a000 finish
2014-06-12 14:46:55.250601 7f9d83173700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.250601 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:55.250604 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:55.250714 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.250731 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:55.251499 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:55.251579 7f9d83173700 15 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.251585 7f9d83173700 10 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1 r = 0
2014-06-12 14:46:55.251589 7f9d83173700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.251602 7f9d83173700 15 _write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 2981888~516096
2014-06-12 14:46:55.251698 7f9d83173700 10 lookup_strip_header done  cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.251714 7f9d7895e700 10 _do_op 0x1697d8c0 seq 2353 r = 0, finisher 0x73c3b00 0
2014-06-12 14:46:55.251725 7f9d7895e700 10 _finish_op 0x1697d8c0 seq 2353 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:46:55.251731 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (1 active)
2014-06-12 14:46:55.251737 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:55.251743 7f9d7715b700  5 -- op tracker -- , seq: 748171, time: 2014-06-12 14:46:55.251735, event: commit_sent, op: osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.251755 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2507 (0'0,173'2507] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2505 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:55.251778 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:55.251810 7f9d7795c700  5 -- op tracker -- , seq: 748171, time: 2014-06-12 14:46:55.251803, event: sub_op_applied, op: osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.251819 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2507 (0'0,173'2507] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2506 active] sub_op_modify_applied on 0x6220540 op osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.251834 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2507 (0'0,173'2507] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2506 active] op_applied on primary on version 173'2507
2014-06-12 14:46:55.251845 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:55.251843 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2507 (0'0,173'2507] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2506 active] op_applied on replica on version 173'2507
2014-06-12 14:46:55.251863 7f9d7795c700  5 -- op tracker -- , seq: 748171, time: 2014-06-12 14:46:55.251859, event: done, op: osd_sub_op(client.4316.0:146442 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2507 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.253114 7f9d83173700 10 _generic_write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 2981888~516096 = 0
2014-06-12 14:46:55.253163 7f9d83173700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.253176 7f9d83173700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.253186 7f9d83173700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.253191 7f9d83173700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.253322 7f9d83173700 20 set_header setting 15606 cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 parent seq 0
2014-06-12 14:46:55.253377 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:55.253388 7f9d83173700 20 set_header setting 2238 cid meta oid 1fe11a4b/pglog_3.56d/0//-1 parent seq 0
2014-06-12 14:46:55.253394 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:55.256664 7f9d83173700 10 _do_op 0x5d507e0 seq 2675 r = 0, finisher 0x6dbd7a0 0
2014-06-12 14:46:55.256682 7f9d83173700 10 _finish_op 0x5d507e0 seq 2675 osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.256692 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541d440 (0 active)
2014-06-12 14:46:55.256699 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:55.256700 7f9d7715b700  5 -- op tracker -- , seq: 748172, time: 2014-06-12 14:46:55.256690, event: commit_sent, op: osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.256717 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2902 (0'0,173'2902] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2900 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:55.256752 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:55.256779 7f9d7795c700  5 -- op tracker -- , seq: 748172, time: 2014-06-12 14:46:55.256773, event: sub_op_applied, op: osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.256794 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2902 (0'0,173'2902] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2901 active] sub_op_modify_applied on 0x570c000 op osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.256805 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2902 (0'0,173'2902] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2901 active] op_applied on primary on version 173'2902
2014-06-12 14:46:55.256811 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2902 (0'0,173'2902] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2901 active] op_applied on replica on version 173'2902
2014-06-12 14:46:55.256825 7f9d7795c700  5 -- op tracker -- , seq: 748172, time: 2014-06-12 14:46:55.256822, event: done, op: osd_sub_op(client.4297.0:144137 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2902 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.262570 7f9d5a16f700  5 -- op tracker -- , seq: 748173, time: 2014-06-12 14:46:55.260682, event: header_read, op: osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.262583 7f9d5a16f700  5 -- op tracker -- , seq: 748173, time: 2014-06-12 14:46:55.260685, event: throttled, op: osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.262591 7f9d5a16f700  5 -- op tracker -- , seq: 748173, time: 2014-06-12 14:46:55.262529, event: all_read, op: osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.262598 7f9d5a16f700  5 -- op tracker -- , seq: 748173, time: 0.000000, event: dispatched, op: osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.262603 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:55.262609 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:46:55.262616 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2910900 prio 127 cost 975655 latency 0.001934 osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.262641 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:55.262649 7f9d683e1700 10 osd.2 173 dequeue_op 0x2910900 prio 127 cost 975655 latency 0.001966 osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2725 (0'0,173'2725] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active]
2014-06-12 14:46:55.262665 7f9d683e1700  5 -- op tracker -- , seq: 748173, time: 2014-06-12 14:46:55.262662, event: reached_pg, op: osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.262672 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2725 (0'0,173'2725] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] handle_message: 0x2910900
2014-06-12 14:46:55.262679 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2725 (0'0,173'2725] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] sub_op_modify trans cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 v 173'2726 (transaction) 174
2014-06-12 14:46:55.262689 7f9d683e1700  5 -- op tracker -- , seq: 748173, time: 2014-06-12 14:46:55.262687, event: started, op: osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.262698 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2725 (0'0,173'2725] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] append_log log((0'0,173'2725], crt=0'0) [173'2726 (173'2725) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143148 2014-06-12 08:43:41.801422]
2014-06-12 14:46:55.262712 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2726 (0'0,173'2726] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] add_log_entry 173'2726 (173'2725) modify   cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 by client.4279.0:143148 2014-06-12 08:43:41.801422
2014-06-12 14:46:55.262726 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2726 (0'0,173'2726] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] append_log  adding 1 keys
2014-06-12 14:46:55.262747 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2726, trimmed: 
2014-06-12 14:46:55.262761 7f9d683e1700  5 -- op tracker -- , seq: 748173, time: 2014-06-12 14:46:55.262758, event: started, op: osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.262767 7f9d683e1700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:55.262770 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:55.262771 7f9d683e1700  5 queue_op 0x8103200 seq 2473 osr(3.21c 0x276d020) 977141 bytes   (queue has 1 ops and 977141 bytes)
2014-06-12 14:46:55.262776 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2726 (0'0,173'2726] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] do_sub_op osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.262790 7f9d683e1700 10 osd.2 173 dequeue_op 0x2910900 finish
2014-06-12 14:46:55.262794 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:55.262796 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:55.262802 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (1 active)
2014-06-12 14:46:55.262808 7f9d7a161700  5 _do_op 0x8103200 seq 2473 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:46:55.262812 7f9d7a161700 10 _do_transaction on 0x29c57b30
2014-06-12 14:46:55.262816 7f9d7a161700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:55.262931 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:55.262954 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:55.262998 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:55.263087 7f9d7a161700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:55.263093 7f9d7a161700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:46:55.263097 7f9d7a161700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:46:55.263110 7f9d7a161700 15 _write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 3219456~974848
2014-06-12 14:46:55.263337 7f9d7a161700 10 lookup_strip_header done  cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:55.263362 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:55.264411 7f9d5bec4700  5 -- op tracker -- , seq: 748174, time: 2014-06-12 14:46:55.263316, event: header_read, op: osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.264430 7f9d5bec4700  5 -- op tracker -- , seq: 748174, time: 2014-06-12 14:46:55.263318, event: throttled, op: osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.264438 7f9d5bec4700  5 -- op tracker -- , seq: 748174, time: 2014-06-12 14:46:55.264341, event: all_read, op: osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.264445 7f9d5bec4700  5 -- op tracker -- , seq: 748174, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.264453 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:55.264463 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:55.264473 7f9d5bec4700 15 osd.2 173 enqueue_op 0x22b3b000 prio 127 cost 516903 latency 0.001156 osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.264511 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:55.264521 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x22b3b000 prio 127 cost 516903 latency 0.001205 osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.56d( v 173'2902 (0'0,173'2902] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2901 active]
2014-06-12 14:46:55.264548 7f9d6b3e7700  5 -- op tracker -- , seq: 748174, time: 2014-06-12 14:46:55.264543, event: reached_pg, op: osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.264557 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2902 (0'0,173'2902] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2901 active] handle_message: 0x22b3b000
2014-06-12 14:46:55.264571 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2902 (0'0,173'2902] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2901 active] sub_op_modify trans b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 v 173'2903 (transaction) 174
2014-06-12 14:46:55.264591 7f9d6b3e7700  5 -- op tracker -- , seq: 748174, time: 2014-06-12 14:46:55.264587, event: started, op: osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.264612 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2902 (0'0,173'2902] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2901 active] append_log log((0'0,173'2902], crt=0'0) [173'2903 (173'2902) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144138 2014-06-12 08:45:57.640297]
2014-06-12 14:46:55.264637 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2903 (0'0,173'2903] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2901 active] add_log_entry 173'2903 (173'2902) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144138 2014-06-12 08:45:57.640297
2014-06-12 14:46:55.264661 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2903 (0'0,173'2903] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2901 active] append_log  adding 1 keys
2014-06-12 14:46:55.264699 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2903, trimmed: 
2014-06-12 14:46:55.264725 7f9d6b3e7700  5 -- op tracker -- , seq: 748174, time: 2014-06-12 14:46:55.264720, event: started, op: osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.264739 7f9d6b3e7700  5 queue_transactions existing osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.264746 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:55.264748 7f9d6b3e7700  5 queue_op 0x603ac60 seq 2676 osr(3.56d 0x276daa0) 518389 bytes   (queue has 2 ops and 1495530 bytes)
2014-06-12 14:46:55.264757 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.56d( v 173'2903 (0'0,173'2903] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2901 active] do_sub_op osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.264772 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x22b3b000 finish
2014-06-12 14:46:55.264775 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:55.264779 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:55.264831 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541d440 (2 active)
2014-06-12 14:46:55.264867 7f9d7f16b700  5 _do_op 0x603ac60 seq 2676 osr(3.56d 0x276daa0)/0x276daa0 start
2014-06-12 14:46:55.264891 7f9d7f16b700 10 _do_transaction on 0x20c43cf0
2014-06-12 14:46:55.264917 7f9d7f16b700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.265172 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.265228 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:55.266278 7f9d7a161700 10 _generic_write 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 3219456~974848 = 0
2014-06-12 14:46:55.266372 7f9d7a161700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:55.266386 7f9d7a161700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:55.266394 7f9d7a161700 15 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3
2014-06-12 14:46:55.266399 7f9d7a161700 10 _setattrs 3.21c_head/cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 = 0
2014-06-12 14:46:55.266647 7f9d7a161700 20 set_header setting 16774 cid 3.21c_head oid cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 parent seq 0
2014-06-12 14:46:55.267302 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:55.267321 7f9d7a161700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:46:55.267327 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:55.271930 7f9d7a161700 10 _do_op 0x8103200 seq 2473 r = 0, finisher 0x4efdce0 0
2014-06-12 14:46:55.272006 7f9d7715b700  5 -- op tracker -- , seq: 748173, time: 2014-06-12 14:46:55.271995, event: commit_sent, op: osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.272016 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2726 (0'0,173'2726] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] sub_op_modify_commit on op osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:46:55.272046 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:55.272112 7f9d7a161700 10 _finish_op 0x8103200 seq 2473 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:46:55.272147 7f9d7795c700  5 -- op tracker -- , seq: 748173, time: 2014-06-12 14:46:55.272141, event: sub_op_applied, op: osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.272155 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2726 (0'0,173'2726] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] sub_op_modify_applied on 0x29c57a40 op osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.272170 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2726 (0'0,173'2726] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] op_applied on primary on version 173'2726
2014-06-12 14:46:55.272179 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2726 (0'0,173'2726] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] op_applied on replica on version 173'2726
2014-06-12 14:46:55.272265 7f9d7a161700  5 -- op tracker -- , seq: 748173, time: 2014-06-12 14:46:55.272256, event: done, op: osd_sub_op(client.4279.0:143148 3.21c cf2b0a1c/rbd_data.108e6b8b4567.0000000000000ff5/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.272296 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (1 active)
2014-06-12 14:46:55.272299 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:55.272457 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:55.272502 7f9d7f16b700 15 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.272528 7f9d7f16b700 10 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1 r = 0
2014-06-12 14:46:55.272553 7f9d7f16b700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.272587 7f9d7f16b700 15 _write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 3497984~516096
2014-06-12 14:46:55.272767 7f9d7f16b700 10 lookup_strip_header done  cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.272962 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:55.275009 7f9d7f16b700 10 _generic_write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 3497984~516096 = 0
2014-06-12 14:46:55.275068 7f9d7f16b700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.275085 7f9d7f16b700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.275098 7f9d7f16b700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.275108 7f9d7f16b700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.275758 7f9d7f16b700 20 set_header setting 15606 cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 parent seq 0
2014-06-12 14:46:55.275826 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:55.275845 7f9d7f16b700 20 set_header setting 2238 cid meta oid 1fe11a4b/pglog_3.56d/0//-1 parent seq 0
2014-06-12 14:46:55.275855 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:55.278105 7f9d7f16b700 10 _do_op 0x603ac60 seq 2676 r = 0, finisher 0x1ecc8720 0
2014-06-12 14:46:55.278123 7f9d7f16b700 10 _finish_op 0x603ac60 seq 2676 osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.278137 7f9d7715b700  5 -- op tracker -- , seq: 748174, time: 2014-06-12 14:46:55.278130, event: commit_sent, op: osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.278140 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541d440 (0 active)
2014-06-12 14:46:55.278150 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:55.278148 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2903 (0'0,173'2903] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2901 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:55.278166 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:55.278194 7f9d7795c700  5 -- op tracker -- , seq: 748174, time: 2014-06-12 14:46:55.278188, event: sub_op_applied, op: osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.278203 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2903 (0'0,173'2903] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2902 active] sub_op_modify_applied on 0x20c43c00 op osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.278220 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2903 (0'0,173'2903] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2902 active] op_applied on primary on version 173'2903
2014-06-12 14:46:55.278231 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2903 (0'0,173'2903] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2902 active] op_applied on replica on version 173'2903
2014-06-12 14:46:55.278259 7f9d7795c700  5 -- op tracker -- , seq: 748174, time: 2014-06-12 14:46:55.278254, event: done, op: osd_sub_op(client.4297.0:144138 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2903 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.284267 7f9d5bec4700  5 -- op tracker -- , seq: 748175, time: 2014-06-12 14:46:55.283911, event: header_read, op: osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.284288 7f9d5bec4700  5 -- op tracker -- , seq: 748175, time: 2014-06-12 14:46:55.283913, event: throttled, op: osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.284297 7f9d5bec4700  5 -- op tracker -- , seq: 748175, time: 2014-06-12 14:46:55.284234, event: all_read, op: osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.284306 7f9d5bec4700  5 -- op tracker -- , seq: 748175, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.284311 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:55.284318 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:46:55.284326 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e87400 prio 127 cost 181031 latency 0.000415 osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.284350 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:55.284356 7f9d693e3700 10 osd.2 173 dequeue_op 0x20e87400 prio 127 cost 181031 latency 0.000445 osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.56d( v 173'2903 (0'0,173'2903] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2902 active]
2014-06-12 14:46:55.284372 7f9d693e3700  5 -- op tracker -- , seq: 748175, time: 2014-06-12 14:46:55.284369, event: reached_pg, op: osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.284381 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2903 (0'0,173'2903] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2902 active] handle_message: 0x20e87400
2014-06-12 14:46:55.284389 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2903 (0'0,173'2903] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2902 active] sub_op_modify trans b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 v 173'2904 (transaction) 174
2014-06-12 14:46:55.284399 7f9d693e3700  5 -- op tracker -- , seq: 748175, time: 2014-06-12 14:46:55.284397, event: started, op: osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.284411 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2903 (0'0,173'2903] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2902 active] append_log log((0'0,173'2903], crt=0'0) [173'2904 (173'2903) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144139 2014-06-12 08:45:57.660092]
2014-06-12 14:46:55.284429 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2904 (0'0,173'2904] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2902 active] add_log_entry 173'2904 (173'2903) modify   b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 by client.4297.0:144139 2014-06-12 08:45:57.660092
2014-06-12 14:46:55.284445 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2904 (0'0,173'2904] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2902 active] append_log  adding 1 keys
2014-06-12 14:46:55.284469 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2904, trimmed: 
2014-06-12 14:46:55.284503 7f9d693e3700  5 -- op tracker -- , seq: 748175, time: 2014-06-12 14:46:55.284499, event: started, op: osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.284520 7f9d693e3700  5 queue_transactions existing osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.284524 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:55.284525 7f9d693e3700  5 queue_op 0x2464b40 seq 2677 osr(3.56d 0x276daa0) 182517 bytes   (queue has 1 ops and 182517 bytes)
2014-06-12 14:46:55.284533 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x541d440 (1 active)
2014-06-12 14:46:55.284539 7f9d7e169700  5 _do_op 0x2464b40 seq 2677 osr(3.56d 0x276daa0)/0x276daa0 start
2014-06-12 14:46:55.284530 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.56d( v 173'2904 (0'0,173'2904] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2902 active] do_sub_op osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.284543 7f9d7e169700 10 _do_transaction on 0x2705f7b0
2014-06-12 14:46:55.284544 7f9d693e3700 10 osd.2 173 dequeue_op 0x20e87400 finish
2014-06-12 14:46:55.284547 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:55.284550 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:55.284550 7f9d7e169700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.284673 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.284777 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:55.284831 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:55.284851 7f9d7e169700 15 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.284856 7f9d7e169700 10 _omap_rmkeys meta/1fe11a4b/pglog_3.56d/0//-1 r = 0
2014-06-12 14:46:55.284867 7f9d7e169700 15 _omap_setkeys meta/1fe11a4b/pglog_3.56d/0//-1
2014-06-12 14:46:55.284888 7f9d7e169700 15 _write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 4014080~180224
2014-06-12 14:46:55.284990 7f9d7e169700 10 lookup_strip_header done  cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.285124 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:55.285559 7f9d7e169700 10 _generic_write 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 4014080~180224 = 0
2014-06-12 14:46:55.285577 7f9d7e169700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.285585 7f9d7e169700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.285588 7f9d7e169700 15 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3
2014-06-12 14:46:55.285593 7f9d7e169700 10 _setattrs 3.56d_head/b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 = 0
2014-06-12 14:46:55.285684 7f9d7e169700 20 set_header setting 15606 cid 3.56d_head oid b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 parent seq 0
2014-06-12 14:46:55.285705 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:55.285714 7f9d7e169700 20 set_header setting 2238 cid meta oid 1fe11a4b/pglog_3.56d/0//-1 parent seq 0
2014-06-12 14:46:55.285720 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:55.286342 7f9d7e169700 10 _do_op 0x2464b40 seq 2677 r = 0, finisher 0x1ecc8d80 0
2014-06-12 14:46:55.286353 7f9d7e169700 10 _finish_op 0x2464b40 seq 2677 osr(3.56d 0x276daa0)/0x276daa0
2014-06-12 14:46:55.286360 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x541d440 (0 active)
2014-06-12 14:46:55.286363 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:55.286368 7f9d7715b700  5 -- op tracker -- , seq: 748175, time: 2014-06-12 14:46:55.286361, event: commit_sent, op: osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.286380 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2904 (0'0,173'2904] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2902 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:46:55.286399 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:55.286451 7f9d7795c700  5 -- op tracker -- , seq: 748175, time: 2014-06-12 14:46:55.286441, event: sub_op_applied, op: osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.286458 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2904 (0'0,173'2904] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2903 active] sub_op_modify_applied on 0x2705f6c0 op osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:55.286469 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2904 (0'0,173'2904] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2903 active] op_applied on primary on version 173'2904
2014-06-12 14:46:55.286475 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.56d( v 173'2904 (0'0,173'2904] local-les=149 n=302 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2903 active] op_applied on replica on version 173'2904
2014-06-12 14:46:55.286490 7f9d7795c700  5 -- op tracker -- , seq: 748175, time: 2014-06-12 14:46:55.286488, event: done, op: osd_sub_op(client.4297.0:144139 3.56d b0abb56d/rbd_data.10a06b8b4567.000000000000107a/head//3 [] v 173'2904 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:55.308497 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:55.308578 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:46:55.479903 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:55.479921 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:55.479925 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:55.479934 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:55.479934 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:55.479936 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 25; oldest is 22.318592 seconds old
2014-06-12 14:46:55.947138 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:55.947188 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:56.155204 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:56.288718 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:56.288723 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:56.288732 7f9d74bfa700 20 osd.2 173 _dispatch 0x1e700340 ping v1
2014-06-12 14:46:56.288734 7f9d74bfa700 10 osd.2 173 ping from client.4306
2014-06-12 14:46:56.288736 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:56.288737 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:56.341427 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:56.341500 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:46:56.369700 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:56.480029 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:56.480060 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:56.480087 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 10.74 >= max 0.5 = no, load too high
2014-06-12 14:46:56.480100 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:56.480102 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92664 < max (604800 seconds)
2014-06-12 14:46:56.480125 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:56.480132 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:56.480133 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:56.480134 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 25; oldest is 23.318790 seconds old
2014-06-12 14:46:56.659205 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:56.659414 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:46:56.782347 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:56.787985 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:56.790718 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:56.791406 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:56.819697 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:56.820109 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:56.821313 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:56.824589 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:56.864203 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:56.864207 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:56.864208 7f9d74bfa700 20 osd.2 173 _dispatch 0x17705d40 ping v1
2014-06-12 14:46:56.864210 7f9d74bfa700 10 osd.2 173 ping from client.4314
2014-06-12 14:46:56.864220 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:56.864221 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:56.864694 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:56.871042 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:56.912592 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:56.918929 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:56.920229 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:56.920746 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:56.922076 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:56.922219 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:46:56.928166 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:56.943904 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:56.956366 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:56.965840 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.086030 7f9d56432700  5 -- op tracker -- , seq: 748176, time: 2014-06-12 14:46:57.085227, event: header_read, op: osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.086044 7f9d56432700  5 -- op tracker -- , seq: 748176, time: 2014-06-12 14:46:57.085231, event: throttled, op: osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.086053 7f9d56432700  5 -- op tracker -- , seq: 748176, time: 2014-06-12 14:46:57.085993, event: all_read, op: osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.086063 7f9d56432700  5 -- op tracker -- , seq: 748176, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.086069 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:57.086076 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:57.086084 7f9d56432700 15 osd.2 173 enqueue_op 0x6d10100 prio 127 cost 385831 latency 0.000857 osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.086116 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.086134 7f9d63bd8700 10 osd.2 173 dequeue_op 0x6d10100 prio 127 cost 385831 latency 0.000907 osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2643 (0'0,173'2643] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active]
2014-06-12 14:46:57.086166 7f9d63bd8700  5 -- op tracker -- , seq: 748176, time: 2014-06-12 14:46:57.086162, event: reached_pg, op: osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.086179 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2643 (0'0,173'2643] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] handle_message: 0x6d10100
2014-06-12 14:46:57.086192 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2643 (0'0,173'2643] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] sub_op_modify trans af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 v 173'2644 (transaction) 174
2014-06-12 14:46:57.086218 7f9d63bd8700  5 -- op tracker -- , seq: 748176, time: 2014-06-12 14:46:57.086206, event: started, op: osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.086242 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2643 (0'0,173'2643] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] append_log log((0'0,173'2643], crt=0'0) [173'2644 (173'681) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147303 2014-06-12 08:45:59.462345]
2014-06-12 14:46:57.086276 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2644 (0'0,173'2644] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] add_log_entry 173'2644 (173'681) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147303 2014-06-12 08:45:59.462345
2014-06-12 14:46:57.086305 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2644 (0'0,173'2644] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] append_log  adding 1 keys
2014-06-12 14:46:57.086351 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2644, trimmed: 
2014-06-12 14:46:57.086384 7f9d63bd8700  5 -- op tracker -- , seq: 748176, time: 2014-06-12 14:46:57.086379, event: started, op: osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.086396 7f9d63bd8700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:57.086400 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.086402 7f9d63bd8700  5 queue_op 0xdcb8c60 seq 2464 osr(3.1c7 0x276d170) 387317 bytes   (queue has 1 ops and 387317 bytes)
2014-06-12 14:46:57.086410 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:46:57.086417 7f9d7c165700  5 _do_op 0xdcb8c60 seq 2464 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:46:57.086420 7f9d7c165700 10 _do_transaction on 0x286f22b0
2014-06-12 14:46:57.086408 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2644 (0'0,173'2644] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] do_sub_op osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.086423 7f9d63bd8700 10 osd.2 173 dequeue_op 0x6d10100 finish
2014-06-12 14:46:57.086425 7f9d7c165700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:57.086426 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.086429 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.086545 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:57.086575 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.086609 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.086627 7f9d7c165700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:57.086631 7f9d7c165700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:46:57.086635 7f9d7c165700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:57.086749 7f9d7c165700 15 _write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 0~385024
2014-06-12 14:46:57.115056 7f9d7c165700 10 lookup_strip_header done  cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:46:57.115229 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:57.117108 7f9d7c165700 10 _generic_write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 0~385024 = 0
2014-06-12 14:46:57.117155 7f9d7c165700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:46:57.117170 7f9d7c165700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:46:57.117185 7f9d7c165700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:46:57.117196 7f9d7c165700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:46:57.117476 7f9d7c165700 20 set_header setting 17479 cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 parent seq 0
2014-06-12 14:46:57.117564 7f9d7c165700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:46:57.117598 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.117632 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:57.119235 7f9d7c165700 10 _do_op 0xdcb8c60 seq 2464 r = 0, finisher 0x88a4d40 0
2014-06-12 14:46:57.119250 7f9d7c165700 10 _finish_op 0xdcb8c60 seq 2464 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:57.119257 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:46:57.119260 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.119300 7f9d7715b700  5 -- op tracker -- , seq: 748176, time: 2014-06-12 14:46:57.119287, event: commit_sent, op: osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.119312 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2644 (0'0,173'2644] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2642 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:57.119335 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:57.119365 7f9d7795c700  5 -- op tracker -- , seq: 748176, time: 2014-06-12 14:46:57.119359, event: sub_op_applied, op: osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.119373 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2644 (0'0,173'2644] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] sub_op_modify_applied on 0x286f21c0 op osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.119385 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2644 (0'0,173'2644] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] op_applied on primary on version 173'2644
2014-06-12 14:46:57.119390 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2644 (0'0,173'2644] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] op_applied on replica on version 173'2644
2014-06-12 14:46:57.119438 7f9d7795c700  5 -- op tracker -- , seq: 748176, time: 2014-06-12 14:46:57.119435, event: done, op: osd_sub_op(client.4316.0:147303 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2644 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.185022 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.191360 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.198474 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.203889 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.210740 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.216411 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.224775 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.230707 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.237567 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.241495 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.244247 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.250668 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.251792 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.256758 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.262877 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.264839 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.272365 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.278212 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.284610 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.286445 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.334879 7f9d4397a700  5 -- op tracker -- , seq: 748177, time: 2014-06-12 14:46:57.333757, event: header_read, op: osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.334903 7f9d4397a700  5 -- op tracker -- , seq: 748177, time: 2014-06-12 14:46:57.333762, event: throttled, op: osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.334938 7f9d4397a700  5 -- op tracker -- , seq: 748177, time: 2014-06-12 14:46:57.334821, event: all_read, op: osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.334951 7f9d4397a700  5 -- op tracker -- , seq: 748177, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.334965 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:57.334984 7f9d4397a700 15 osd.2 173 enqueue_op 0x876fc00 prio 63 cost 360448 latency 0.001226 osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.335033 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.335043 7f9d6abe6700 10 osd.2 173 dequeue_op 0x876fc00 prio 63 cost 360448 latency 0.001286 osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean]
2014-06-12 14:46:57.335072 7f9d6abe6700  5 -- op tracker -- , seq: 748177, time: 2014-06-12 14:46:57.335066, event: reached_pg, op: osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.335081 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:57.335092 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] handle_message: 0x876fc00
2014-06-12 14:46:57.335098 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] do_op osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:57.335129 7f9d6abe6700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_'
2014-06-12 14:46:57.381838 7f9d6abe6700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_' = 0
2014-06-12 14:46:57.381875 7f9d6abe6700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset'
2014-06-12 14:46:57.381999 7f9d6abe6700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset' = 0
2014-06-12 14:46:57.382008 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] populate_obc_watchers 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.382029 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.382036 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] get_object_context: creating obc from disk: 0x37e4a840
2014-06-12 14:46:57.382042 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] get_object_context: 0x37e4a840 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 rwstate(none n=0 w=0) oi: 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'782 client.4309.0:37299 wrlock_by=unknown.0.0:0 dirty s 4194304 uv782) ssc: 0x5bb8620 snapset: 0=[]:[]+head
2014-06-12 14:46:57.382052 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] find_object_context 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 @head oi=103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'782 client.4309.0:37299 wrlock_by=unknown.0.0:0 dirty s 4194304 uv782)
2014-06-12 14:46:57.382074 7f9d6abe6700  5 -- op tracker -- , seq: 748177, time: 2014-06-12 14:46:57.382069, event: started, op: osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.382086 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] execute_ctx 0x55a4a00
2014-06-12 14:46:57.382097 7f9d6abe6700  5 -- op tracker -- , seq: 748177, time: 2014-06-12 14:46:57.382094, event: started, op: osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.382101 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] do_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] ov 173'782 av 173'2774 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:57.382110 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] do_osd_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448]
2014-06-12 14:46:57.382116 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:57.382125 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] do_osd_op  write 0~360448
2014-06-12 14:46:57.382136 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:57.382145 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 snapset=0x5bb8658  snapc=0=[]
2014-06-12 14:46:57.382155 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:57.382163 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] finish_ctx 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 0x55a4a00 op modify  
2014-06-12 14:46:57.382173 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean]  set mtime to 2014-06-12 08:45:59.713334
2014-06-12 14:46:57.382188 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean]  final snapset 0=[]:[]+head in 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.382201 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean]  zeroing write result code 0
2014-06-12 14:46:57.382209 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] new_repop rep_tid 236362 on osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.382220 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] issue_repop rep_tid 236362 o 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.382238 7f9d6abe6700  5 -- op tracker -- , seq: 748177, time: 2014-06-12 14:46:57.382235, event: waiting for subops from 27, op: osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.382271 7f9d6abe6700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:57.382291 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2773 (0'0,173'2773] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] append_log log((0'0,173'2773], crt=0'0) [173'2774 (173'782) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141791 2014-06-12 08:45:59.713334]
2014-06-12 14:46:57.382325 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2773 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] add_log_entry 173'2774 (173'782) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141791 2014-06-12 08:45:59.713334
2014-06-12 14:46:57.382356 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2773 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] append_log  adding 1 keys
2014-06-12 14:46:57.382385 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2774, trimmed: 
2014-06-12 14:46:57.382423 7f9d6abe6700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.382426 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.382427 7f9d6abe6700  5 queue_op 0x6cea2a0 seq 2489 osr(3.312 0x276d540) 362741 bytes   (queue has 1 ops and 362741 bytes)
2014-06-12 14:46:57.382434 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2773 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] eval_repop repgather(0x1771cc00 173'2774 rep_tid=236362 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.382455 7f9d6abe6700 10 osd.2 173 dequeue_op 0x876fc00 finish
2014-06-12 14:46:57.382460 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.382463 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.382471 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (1 active)
2014-06-12 14:46:57.382475 7f9d7815d700  5 _do_op 0x6cea2a0 seq 2489 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:57.382479 7f9d7815d700 10 _do_transaction on 0x1771da40
2014-06-12 14:46:57.382484 7f9d7815d700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.382581 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.382626 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.382682 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.382824 7f9d7815d700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.382831 7f9d7815d700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:57.382835 7f9d7815d700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.382849 7f9d7815d700 15 _write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 0~360448
2014-06-12 14:46:57.382948 7f9d7815d700 10 lookup_strip_header done  cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.383115 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:57.384017 7f9d7815d700 10 _generic_write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 0~360448 = 0
2014-06-12 14:46:57.384049 7f9d7815d700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.384059 7f9d7815d700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.384063 7f9d7815d700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.384067 7f9d7815d700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.384185 7f9d7815d700 20 set_header setting 17155 cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 parent seq 0
2014-06-12 14:46:57.384228 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.384238 7f9d7815d700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:57.384244 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:57.386460 7f9d7815d700 10 _do_op 0x6cea2a0 seq 2489 r = 0, finisher 0x6cb8b00 0xcb61840
2014-06-12 14:46:57.386484 7f9d7815d700 10 _finish_op 0x6cea2a0 seq 2489 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.386493 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:57.386503 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.386503 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2773 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] op_applied: 236362
2014-06-12 14:46:57.386541 7f9d7795c700  5 -- op tracker -- , seq: 748177, time: 2014-06-12 14:46:57.386532, event: op_applied, op: osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.386552 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2773 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] op_applied on primary on version 173'2774
2014-06-12 14:46:57.386568 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2773 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] op_commit: 236362
2014-06-12 14:46:57.386592 7f9d7715b700  5 -- op tracker -- , seq: 748177, time: 2014-06-12 14:46:57.386585, event: op_commit, op: osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.387232 7f9d58c5a700  5 -- op tracker -- , seq: 748178, time: 2014-06-12 14:46:57.387187, event: header_read, op: osd_sub_op_reply(client.4309.0:141791 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.387243 7f9d58c5a700  5 -- op tracker -- , seq: 748178, time: 2014-06-12 14:46:57.387189, event: throttled, op: osd_sub_op_reply(client.4309.0:141791 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.387251 7f9d58c5a700  5 -- op tracker -- , seq: 748178, time: 2014-06-12 14:46:57.387200, event: all_read, op: osd_sub_op_reply(client.4309.0:141791 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.387260 7f9d58c5a700  5 -- op tracker -- , seq: 748178, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141791 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.387266 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141791 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:57.387274 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:57.387284 7f9d58c5a700 15 osd.2 173 enqueue_op 0x6f4af00 prio 196 cost 0 latency 0.000097 osd_sub_op_reply(client.4309.0:141791 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:57.387316 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.387327 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x6f4af00 prio 196 cost 0 latency 0.000140 osd_sub_op_reply(client.4309.0:141791 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2773 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean]
2014-06-12 14:46:57.387349 7f9d6a3e5700  5 -- op tracker -- , seq: 748178, time: 2014-06-12 14:46:57.387345, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141791 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.387359 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2773 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] handle_message: 0x6f4af00
2014-06-12 14:46:57.387377 7f9d6a3e5700  5 -- op tracker -- , seq: 748178, time: 2014-06-12 14:46:57.387373, event: started, op: osd_sub_op_reply(client.4309.0:141791 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.387384 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2773 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] sub_op_modify_reply: tid 236362 op  ack_type 4 from 27
2014-06-12 14:46:57.387404 7f9d6a3e5700  5 -- op tracker -- , seq: 748177, time: 2014-06-12 14:46:57.387398, event: sub_op_commit_rec, op: osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.387418 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2773 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] repop_all_applied: repop tid 236362 all applied 
2014-06-12 14:46:57.387431 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2773 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] eval_repop repgather(0x1771cc00 173'2774 rep_tid=236362 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.387448 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2773 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean]  sending ack on repgather(0x1771cc00 173'2774 rep_tid=236362 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x54b1900
2014-06-12 14:46:57.387483 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2773 crt=0'0 lcod 173'2772 mlcod 173'2772 active+clean] repop_all_committed: repop tid 236362 all committed 
2014-06-12 14:46:57.387495 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2772 active+clean] eval_repop repgather(0x1771cc00 173'2774 rep_tid=236362 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.389556 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2772 active+clean] log_op_stats osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 inb 360727 outb 0 rlat 0.053725 lat 0.055797
2014-06-12 14:46:57.389574 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2772 active+clean] publish_stats_to_osd 173:3462
2014-06-12 14:46:57.389588 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2772 active+clean]  sending commit on repgather(0x1771cc00 173'2774 rep_tid=236362 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x21eba280
2014-06-12 14:46:57.389649 7f9d6a3e5700  5 -- op tracker -- , seq: 748177, time: 2014-06-12 14:46:57.389642, event: commit_sent, op: osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.389660 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2772 active+clean]  requeue_ops 
2014-06-12 14:46:57.389670 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean]  removing repgather(0x1771cc00 173'2774 rep_tid=236362 committed?=1 applied?=1 op=osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.389690 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean]    q front is repgather(0x1771cc00 173'2774 rep_tid=236362 committed?=1 applied?=1 op=osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.389704 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] remove_repop repgather(0x1771cc00 173'2774 rep_tid=236362 committed?=1 applied?=1 op=osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.389720 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean]  requeue_ops 
2014-06-12 14:46:57.389747 7f9d6a3e5700  5 -- op tracker -- , seq: 748177, time: 2014-06-12 14:46:57.389742, event: done, op: osd_op(client.4309.0:141791 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~360448] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.389772 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x6f4af00 finish
2014-06-12 14:46:57.389783 7f9d6a3e5700  5 -- op tracker -- , seq: 748178, time: 2014-06-12 14:46:57.389778, event: done, op: osd_sub_op_reply(client.4309.0:141791 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.389798 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.389804 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.393145 7f9d4397a700  5 -- op tracker -- , seq: 748179, time: 2014-06-12 14:46:57.391981, event: header_read, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.393176 7f9d4397a700  5 -- op tracker -- , seq: 748179, time: 2014-06-12 14:46:57.391983, event: throttled, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.393186 7f9d4397a700  5 -- op tracker -- , seq: 748179, time: 2014-06-12 14:46:57.393086, event: all_read, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.393194 7f9d4397a700  5 -- op tracker -- , seq: 748179, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.393200 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:57.393224 7f9d4397a700 15 osd.2 173 enqueue_op 0x329eba00 prio 63 cost 516096 latency 0.001243 osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.393272 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.393279 7f9d66bde700 10 osd.2 173 dequeue_op 0x329eba00 prio 63 cost 516096 latency 0.001298 osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean]
2014-06-12 14:46:57.393302 7f9d66bde700  5 -- op tracker -- , seq: 748179, time: 2014-06-12 14:46:57.393297, event: reached_pg, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.393313 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:57.393329 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] handle_message: 0x329eba00
2014-06-12 14:46:57.393340 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] do_op osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:57.393365 7f9d66bde700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_'
2014-06-12 14:46:57.393542 7f9d66bde700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_' = 0
2014-06-12 14:46:57.393558 7f9d66bde700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset'
2014-06-12 14:46:57.393639 7f9d66bde700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset' = 0
2014-06-12 14:46:57.393650 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] populate_obc_watchers 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.393663 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.393674 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] get_object_context: creating obc from disk: 0x2b9fa580
2014-06-12 14:46:57.393684 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] get_object_context: 0x2b9fa580 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 rwstate(none n=0 w=0) oi: 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2774 client.4309.0:141791 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2774) ssc: 0x81ba540 snapset: 0=[]:[]+head
2014-06-12 14:46:57.393701 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] find_object_context 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 @head oi=103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2774 client.4309.0:141791 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2774)
2014-06-12 14:46:57.393727 7f9d66bde700  5 -- op tracker -- , seq: 748179, time: 2014-06-12 14:46:57.393722, event: started, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.393737 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] execute_ctx 0x110c5600
2014-06-12 14:46:57.393755 7f9d66bde700  5 -- op tracker -- , seq: 748179, time: 2014-06-12 14:46:57.393750, event: started, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.393763 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] do_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] ov 173'2774 av 173'2775 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:57.393779 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] do_osd_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096]
2014-06-12 14:46:57.393791 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:57.393806 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] do_osd_op  write 360448~516096
2014-06-12 14:46:57.393822 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:57.393835 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 snapset=0x81ba578  snapc=0=[]
2014-06-12 14:46:57.393849 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:57.393861 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] finish_ctx 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 0x110c5600 op modify  
2014-06-12 14:46:57.393874 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean]  set mtime to 2014-06-12 08:45:59.771313
2014-06-12 14:46:57.393895 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean]  final snapset 0=[]:[]+head in 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.393915 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean]  zeroing write result code 0
2014-06-12 14:46:57.393928 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] new_repop rep_tid 236363 on osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.393945 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] issue_repop rep_tid 236363 o 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.393972 7f9d66bde700  5 -- op tracker -- , seq: 748179, time: 2014-06-12 14:46:57.393967, event: waiting for subops from 27, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.394002 7f9d66bde700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:57.394026 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2774 (0'0,173'2774] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] append_log log((0'0,173'2774], crt=0'0) [173'2775 (173'2774) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141792 2014-06-12 08:45:59.771313]
2014-06-12 14:46:57.394050 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] add_log_entry 173'2775 (173'2774) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141792 2014-06-12 08:45:59.771313
2014-06-12 14:46:57.394072 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] append_log  adding 1 keys
2014-06-12 14:46:57.394130 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2775, trimmed: 
2014-06-12 14:46:57.394155 7f9d66bde700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.394161 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.394162 7f9d66bde700  5 queue_op 0x5411680 seq 2490 osr(3.312 0x276d540) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:57.394173 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (1 active)
2014-06-12 14:46:57.394181 7f9d82171700  5 _do_op 0x5411680 seq 2490 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:57.394169 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] eval_repop repgather(0x15acfb00 173'2775 rep_tid=236363 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.394186 7f9d82171700 10 _do_transaction on 0x15acf440
2014-06-12 14:46:57.394187 7f9d66bde700 10 osd.2 173 dequeue_op 0x329eba00 finish
2014-06-12 14:46:57.394191 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.394192 7f9d82171700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.394194 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.394274 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.394297 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.394348 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.394366 7f9d82171700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.394379 7f9d82171700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:57.394385 7f9d82171700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.394398 7f9d82171700 15 _write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 360448~516096
2014-06-12 14:46:57.394531 7f9d82171700 10 lookup_strip_header done  cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.394665 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:57.396696 7f9d82171700 10 _generic_write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 360448~516096 = 0
2014-06-12 14:46:57.396754 7f9d82171700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.396777 7f9d82171700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.396783 7f9d82171700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.396791 7f9d82171700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.397012 7f9d82171700 20 set_header setting 17155 cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 parent seq 0
2014-06-12 14:46:57.397080 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.397113 7f9d82171700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:57.397128 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:57.399488 7f9d82171700 10 _do_op 0x5411680 seq 2490 r = 0, finisher 0x87fbcc0 0x1e627580
2014-06-12 14:46:57.399504 7f9d82171700 10 _finish_op 0x5411680 seq 2490 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.399512 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:57.399528 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.399529 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] op_applied: 236363
2014-06-12 14:46:57.399569 7f9d7795c700  5 -- op tracker -- , seq: 748179, time: 2014-06-12 14:46:57.399559, event: op_applied, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.399579 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] op_applied on primary on version 173'2775
2014-06-12 14:46:57.399632 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] op_commit: 236363
2014-06-12 14:46:57.399663 7f9d7715b700  5 -- op tracker -- , seq: 748179, time: 2014-06-12 14:46:57.399658, event: op_commit, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.400661 7f9d58c5a700  5 -- op tracker -- , seq: 748180, time: 2014-06-12 14:46:57.400623, event: header_read, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.400671 7f9d58c5a700  5 -- op tracker -- , seq: 748180, time: 2014-06-12 14:46:57.400624, event: throttled, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.400679 7f9d58c5a700  5 -- op tracker -- , seq: 748180, time: 2014-06-12 14:46:57.400634, event: all_read, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.400686 7f9d58c5a700  5 -- op tracker -- , seq: 748180, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.400691 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:57.400696 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:57.400704 7f9d58c5a700 15 osd.2 173 enqueue_op 0x12bf7800 prio 196 cost 0 latency 0.000080 osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0) v2
2014-06-12 14:46:57.400729 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.400737 7f9d64bda700 10 osd.2 173 dequeue_op 0x12bf7800 prio 196 cost 0 latency 0.000114 osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0) v2 pg pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean]
2014-06-12 14:46:57.400756 7f9d64bda700  5 -- op tracker -- , seq: 748180, time: 2014-06-12 14:46:57.400753, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.400788 7f9d58c5a700  5 -- op tracker -- , seq: 748181, time: 2014-06-12 14:46:57.400730, event: header_read, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.400782 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] handle_message: 0x12bf7800
2014-06-12 14:46:57.400797 7f9d58c5a700  5 -- op tracker -- , seq: 748181, time: 2014-06-12 14:46:57.400731, event: throttled, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.400807 7f9d58c5a700  5 -- op tracker -- , seq: 748181, time: 2014-06-12 14:46:57.400742, event: all_read, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.400818 7f9d58c5a700  5 -- op tracker -- , seq: 748181, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.400824 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:57.400829 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:57.400828 7f9d64bda700  5 -- op tracker -- , seq: 748180, time: 2014-06-12 14:46:57.400796, event: started, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.400834 7f9d58c5a700 15 osd.2 173 enqueue_op 0x12bf6500 prio 196 cost 0 latency 0.000104 osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:57.400835 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] sub_op_modify_reply: tid 236363 op  ack_type 1 from 27
2014-06-12 14:46:57.400853 7f9d64bda700  5 -- op tracker -- , seq: 748179, time: 2014-06-12 14:46:57.400847, event: sub_op_applied_rec, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.400856 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:57.400859 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] repop_all_applied: repop tid 236363 all applied 
2014-06-12 14:46:57.400868 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] eval_repop repgather(0x15acfb00 173'2775 rep_tid=236363 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.400883 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean]  sending ack on repgather(0x15acfb00 173'2775 rep_tid=236363 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x2b2fcf00
2014-06-12 14:46:57.400915 7f9d64bda700 10 osd.2 173 dequeue_op 0x12bf7800 finish
2014-06-12 14:46:57.400925 7f9d64bda700  5 -- op tracker -- , seq: 748180, time: 2014-06-12 14:46:57.400921, event: done, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.400925 7f9d69be4700 10 osd.2 173 dequeue_op 0x12bf6500 prio 196 cost 0 latency 0.000195 osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean]
2014-06-12 14:46:57.400951 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:57.400954 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.400955 7f9d69be4700  5 -- op tracker -- , seq: 748181, time: 2014-06-12 14:46:57.400943, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.400968 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] handle_message: 0x12bf6500
2014-06-12 14:46:57.400980 7f9d69be4700  5 -- op tracker -- , seq: 748181, time: 2014-06-12 14:46:57.400977, event: started, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.400984 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] sub_op_modify_reply: tid 236363 op  ack_type 4 from 27
2014-06-12 14:46:57.400996 7f9d69be4700  5 -- op tracker -- , seq: 748179, time: 2014-06-12 14:46:57.400992, event: sub_op_commit_rec, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.401000 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2774 crt=0'0 lcod 173'2773 mlcod 173'2773 active+clean] repop_all_committed: repop tid 236363 all committed 
2014-06-12 14:46:57.401006 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2773 active+clean] eval_repop repgather(0x15acfb00 173'2775 rep_tid=236363 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.401015 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2773 active+clean] log_op_stats osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008933 lat 0.009034
2014-06-12 14:46:57.401025 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2773 active+clean] publish_stats_to_osd 173:3463
2014-06-12 14:46:57.401032 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2773 active+clean]  sending commit on repgather(0x15acfb00 173'2775 rep_tid=236363 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x54b1b80
2014-06-12 14:46:57.401055 7f9d69be4700  5 -- op tracker -- , seq: 748179, time: 2014-06-12 14:46:57.401049, event: commit_sent, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.401062 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2773 active+clean]  requeue_ops 
2014-06-12 14:46:57.401072 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean]  removing repgather(0x15acfb00 173'2775 rep_tid=236363 committed?=1 applied?=1 op=osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.401084 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean]    q front is repgather(0x15acfb00 173'2775 rep_tid=236363 committed?=1 applied?=1 op=osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.401092 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] remove_repop repgather(0x15acfb00 173'2775 rep_tid=236363 committed?=1 applied?=1 op=osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.401100 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean]  requeue_ops 
2014-06-12 14:46:57.401115 7f9d69be4700  5 -- op tracker -- , seq: 748179, time: 2014-06-12 14:46:57.401112, event: done, op: osd_op(client.4309.0:141792 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 360448~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.401126 7f9d69be4700 10 osd.2 173 dequeue_op 0x12bf6500 finish
2014-06-12 14:46:57.401131 7f9d69be4700  5 -- op tracker -- , seq: 748181, time: 2014-06-12 14:46:57.401129, event: done, op: osd_sub_op_reply(client.4309.0:141792 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.401137 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.401138 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.404848 7f9d56432700  5 -- op tracker -- , seq: 748182, time: 2014-06-12 14:46:57.403699, event: header_read, op: osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.404859 7f9d56432700  5 -- op tracker -- , seq: 748182, time: 2014-06-12 14:46:57.403701, event: throttled, op: osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.404876 7f9d56432700  5 -- op tracker -- , seq: 748182, time: 2014-06-12 14:46:57.404807, event: all_read, op: osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.404881 7f9d56432700  5 -- op tracker -- , seq: 748182, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.404888 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:57.404893 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:57.404904 7f9d56432700 15 osd.2 173 enqueue_op 0x34016600 prio 127 cost 516903 latency 0.001205 osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.404947 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.404956 7f9d653db700 10 osd.2 173 dequeue_op 0x34016600 prio 127 cost 516903 latency 0.001257 osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'3003 (79'2,173'3003] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3002 active]
2014-06-12 14:46:57.404982 7f9d653db700  5 -- op tracker -- , seq: 748182, time: 2014-06-12 14:46:57.404978, event: reached_pg, op: osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.404997 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3003 (79'2,173'3003] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3002 active] handle_message: 0x34016600
2014-06-12 14:46:57.405009 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3003 (79'2,173'3003] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3002 active] sub_op_modify trans d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 v 173'3004 (transaction) 174
2014-06-12 14:46:57.405030 7f9d653db700  5 -- op tracker -- , seq: 748182, time: 2014-06-12 14:46:57.405025, event: started, op: osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.405050 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3003 (79'2,173'3003] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3002 active] append_log log((79'2,173'3003], crt=0'0) [173'3004 (173'3003) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145536 2014-06-12 08:45:59.780207]
2014-06-12 14:46:57.405078 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3004 (79'2,173'3004] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3002 active] add_log_entry 173'3004 (173'3003) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145536 2014-06-12 08:45:59.780207
2014-06-12 14:46:57.405104 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3004 (79'2,173'3004] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3002 active] append_log  adding 1 keys
2014-06-12 14:46:57.405121 7f9d653db700 10 trim log((79'2,173'3004], crt=0'0) to 79'3
2014-06-12 14:46:57.405150 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3004, trimmed: 79'3
2014-06-12 14:46:57.405180 7f9d653db700  5 -- op tracker -- , seq: 748182, time: 2014-06-12 14:46:57.405175, event: started, op: osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.405193 7f9d653db700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:57.405197 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.405199 7f9d653db700  5 queue_op 0x63de180 seq 2806 osr(3.458 0x276d960) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:57.405209 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:57.405214 7f9d7e96a700  5 _do_op 0x63de180 seq 2806 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:57.405218 7f9d7e96a700 10 _do_transaction on 0x13ec15f0
2014-06-12 14:46:57.405206 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'3004 (79'3,173'3004] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3002 active] do_sub_op osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.405221 7f9d653db700 10 osd.2 173 dequeue_op 0x34016600 finish
2014-06-12 14:46:57.405223 7f9d7e96a700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:57.405225 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.405227 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.405313 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:57.405329 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.405364 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.405377 7f9d7e96a700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:57.405384 7f9d7e96a700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:57.405388 7f9d7e96a700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:57.405397 7f9d7e96a700 15 _write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 2736128~516096
2014-06-12 14:46:57.405545 7f9d7e96a700 10 lookup_strip_header done  cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:57.405564 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:57.405889 7f9d4397a700  5 -- op tracker -- , seq: 748183, time: 2014-06-12 14:46:57.403406, event: header_read, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.405905 7f9d4397a700  5 -- op tracker -- , seq: 748183, time: 2014-06-12 14:46:57.403408, event: throttled, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.405915 7f9d4397a700  5 -- op tracker -- , seq: 748183, time: 2014-06-12 14:46:57.405855, event: all_read, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.405925 7f9d4397a700  5 -- op tracker -- , seq: 748183, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.405933 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:57.405960 7f9d4397a700 15 osd.2 173 enqueue_op 0x2afce00 prio 63 cost 516096 latency 0.002554 osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.405994 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.406001 7f9d623d5700 10 osd.2 173 dequeue_op 0x2afce00 prio 63 cost 516096 latency 0.002595 osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean]
2014-06-12 14:46:57.406026 7f9d623d5700  5 -- op tracker -- , seq: 748183, time: 2014-06-12 14:46:57.406021, event: reached_pg, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.406036 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:57.406050 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] handle_message: 0x2afce00
2014-06-12 14:46:57.406060 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] do_op osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:57.406089 7f9d623d5700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_'
2014-06-12 14:46:57.406245 7f9d623d5700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_' = 0
2014-06-12 14:46:57.406271 7f9d623d5700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset'
2014-06-12 14:46:57.406355 7f9d623d5700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset' = 0
2014-06-12 14:46:57.406368 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] populate_obc_watchers 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.406382 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.406391 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] get_object_context: creating obc from disk: 0x135b6000
2014-06-12 14:46:57.406401 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] get_object_context: 0x135b6000 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 rwstate(none n=0 w=0) oi: 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2775 client.4309.0:141792 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2775) ssc: 0x407e000 snapset: 0=[]:[]+head
2014-06-12 14:46:57.406427 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] find_object_context 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 @head oi=103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2775 client.4309.0:141792 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2775)
2014-06-12 14:46:57.406454 7f9d623d5700  5 -- op tracker -- , seq: 748183, time: 2014-06-12 14:46:57.406448, event: started, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.406464 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] execute_ctx 0x1b264a00
2014-06-12 14:46:57.406482 7f9d623d5700  5 -- op tracker -- , seq: 748183, time: 2014-06-12 14:46:57.406477, event: started, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.406490 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] do_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] ov 173'2775 av 173'2776 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:57.406506 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] do_osd_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096]
2014-06-12 14:46:57.406519 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:57.406535 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] do_osd_op  write 876544~516096
2014-06-12 14:46:57.406553 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:57.406566 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 snapset=0x407e038  snapc=0=[]
2014-06-12 14:46:57.406580 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:57.406592 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] finish_ctx 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 0x1b264a00 op modify  
2014-06-12 14:46:57.406605 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean]  set mtime to 2014-06-12 08:45:59.782705
2014-06-12 14:46:57.406628 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean]  final snapset 0=[]:[]+head in 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.406649 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean]  zeroing write result code 0
2014-06-12 14:46:57.406662 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] new_repop rep_tid 236364 on osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.406679 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] issue_repop rep_tid 236364 o 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.406711 7f9d623d5700  5 -- op tracker -- , seq: 748183, time: 2014-06-12 14:46:57.406706, event: waiting for subops from 27, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.406754 7f9d623d5700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:57.406781 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2775 (0'0,173'2775] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] append_log log((0'0,173'2775], crt=0'0) [173'2776 (173'2775) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141793 2014-06-12 08:45:59.782705]
2014-06-12 14:46:57.406807 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2775 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] add_log_entry 173'2776 (173'2775) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141793 2014-06-12 08:45:59.782705
2014-06-12 14:46:57.406832 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2775 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] append_log  adding 1 keys
2014-06-12 14:46:57.406865 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2776, trimmed: 
2014-06-12 14:46:57.406888 7f9d623d5700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.406894 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.406896 7f9d623d5700  5 queue_op 0x8cd22a0 seq 2491 osr(3.312 0x276d540) 518389 bytes   (queue has 2 ops and 1036813 bytes)
2014-06-12 14:46:57.406905 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (2 active)
2014-06-12 14:46:57.406912 7f9d8116f700  5 _do_op 0x8cd22a0 seq 2491 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:57.406916 7f9d8116f700 10 _do_transaction on 0x228e0240
2014-06-12 14:46:57.406903 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2775 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] eval_repop repgather(0x228e0f00 173'2776 rep_tid=236364 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.406922 7f9d623d5700 10 osd.2 173 dequeue_op 0x2afce00 finish
2014-06-12 14:46:57.406922 7f9d8116f700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.406927 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.406931 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.407012 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.407033 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.407098 7f9d7e96a700 10 _generic_write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 2736128~516096 = 0
2014-06-12 14:46:57.407138 7f9d7e96a700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:57.407147 7f9d7e96a700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:46:57.407151 7f9d7e96a700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:57.407155 7f9d7e96a700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:46:57.407315 7f9d7e96a700 20 set_header setting 17709 cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 parent seq 0
2014-06-12 14:46:57.407360 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.407371 7f9d7e96a700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:57.407392 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:57.409701 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.409767 7f9d7e96a700 10 _do_op 0x63de180 seq 2806 r = 0, finisher 0x693b780 0
2014-06-12 14:46:57.409779 7f9d7e96a700 10 _finish_op 0x63de180 seq 2806 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:57.409786 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (1 active)
2014-06-12 14:46:57.409798 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.409796 7f9d7715b700  5 -- op tracker -- , seq: 748182, time: 2014-06-12 14:46:57.409787, event: commit_sent, op: osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.409806 7f9d8116f700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.409815 7f9d8116f700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:57.409809 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3004 (79'3,173'3004] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3002 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:57.409822 7f9d8116f700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.409830 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:57.409844 7f9d8116f700 15 _write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 876544~516096
2014-06-12 14:46:57.409859 7f9d7795c700  5 -- op tracker -- , seq: 748182, time: 2014-06-12 14:46:57.409852, event: sub_op_applied, op: osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.409868 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3004 (79'3,173'3004] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active] sub_op_modify_applied on 0x13ec1500 op osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.409889 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3004 (79'3,173'3004] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active] op_applied on primary on version 173'3004
2014-06-12 14:46:57.409899 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3004 (79'3,173'3004] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active] op_applied on replica on version 173'3004
2014-06-12 14:46:57.409925 7f9d7795c700  5 -- op tracker -- , seq: 748182, time: 2014-06-12 14:46:57.409920, event: done, op: osd_sub_op(client.4299.0:145536 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3004 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.409975 7f9d8116f700 10 lookup_strip_header done  cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.410108 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:57.411309 7f9d8116f700 10 _generic_write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 876544~516096 = 0
2014-06-12 14:46:57.411354 7f9d8116f700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.411368 7f9d8116f700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.411374 7f9d8116f700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.411382 7f9d8116f700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.411518 7f9d8116f700 20 set_header setting 17155 cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 parent seq 0
2014-06-12 14:46:57.411570 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.411586 7f9d8116f700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:57.411599 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:57.413130 7f9d58c5a700  5 -- op tracker -- , seq: 748184, time: 2014-06-12 14:46:57.413083, event: header_read, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.413140 7f9d58c5a700  5 -- op tracker -- , seq: 748184, time: 2014-06-12 14:46:57.413085, event: throttled, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.413147 7f9d58c5a700  5 -- op tracker -- , seq: 748184, time: 2014-06-12 14:46:57.413097, event: all_read, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.413155 7f9d58c5a700  5 -- op tracker -- , seq: 748184, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.413160 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:57.413165 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:57.413174 7f9d58c5a700 15 osd.2 173 enqueue_op 0x12bf6800 prio 196 cost 0 latency 0.000091 osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0) v2
2014-06-12 14:46:57.413203 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.413213 7f9d68be2700 10 osd.2 173 dequeue_op 0x12bf6800 prio 196 cost 0 latency 0.000129 osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0) v2 pg pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2775 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean]
2014-06-12 14:46:57.413231 7f9d58c5a700  5 -- op tracker -- , seq: 748185, time: 2014-06-12 14:46:57.413196, event: header_read, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.413248 7f9d58c5a700  5 -- op tracker -- , seq: 748185, time: 2014-06-12 14:46:57.413196, event: throttled, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.413258 7f9d58c5a700  5 -- op tracker -- , seq: 748185, time: 2014-06-12 14:46:57.413209, event: all_read, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.413277 7f9d58c5a700  5 -- op tracker -- , seq: 748185, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.413282 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:57.413287 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:57.413288 7f9d68be2700  5 -- op tracker -- , seq: 748184, time: 2014-06-12 14:46:57.413231, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.413292 7f9d58c5a700 15 osd.2 173 enqueue_op 0x12bf6400 prio 196 cost 0 latency 0.000096 osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:57.413299 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2775 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] handle_message: 0x12bf6800
2014-06-12 14:46:57.413314 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:57.413315 7f9d68be2700  5 -- op tracker -- , seq: 748184, time: 2014-06-12 14:46:57.413312, event: started, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.413322 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2775 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] sub_op_modify_reply: tid 236364 op  ack_type 1 from 27
2014-06-12 14:46:57.413348 7f9d68be2700  5 -- op tracker -- , seq: 748183, time: 2014-06-12 14:46:57.413339, event: sub_op_applied_rec, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.413356 7f9d68be2700 10 osd.2 173 dequeue_op 0x12bf6800 finish
2014-06-12 14:46:57.413367 7f9d68be2700  5 -- op tracker -- , seq: 748184, time: 2014-06-12 14:46:57.413363, event: done, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ack, result = 0)
2014-06-12 14:46:57.413366 7f9d643d9700 10 osd.2 173 dequeue_op 0x12bf6400 prio 196 cost 0 latency 0.000170 osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2775 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean]
2014-06-12 14:46:57.413379 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:57.413382 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.413384 7f9d643d9700  5 -- op tracker -- , seq: 748185, time: 2014-06-12 14:46:57.413381, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.413391 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2775 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] handle_message: 0x12bf6400
2014-06-12 14:46:57.413407 7f9d643d9700  5 -- op tracker -- , seq: 748185, time: 2014-06-12 14:46:57.413404, event: started, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.413417 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2775 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] sub_op_modify_reply: tid 236364 op  ack_type 4 from 27
2014-06-12 14:46:57.413435 7f9d643d9700  5 -- op tracker -- , seq: 748183, time: 2014-06-12 14:46:57.413429, event: sub_op_commit_rec, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.413441 7f9d643d9700 10 osd.2 173 dequeue_op 0x12bf6400 finish
2014-06-12 14:46:57.413449 7f9d643d9700  5 -- op tracker -- , seq: 748185, time: 2014-06-12 14:46:57.413445, event: done, op: osd_sub_op_reply(client.4309.0:141793 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.413460 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.413463 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.413648 7f9d8116f700 10 _do_op 0x8cd22a0 seq 2491 r = 0, finisher 0x5db2b20 0x1b743680
2014-06-12 14:46:57.413659 7f9d8116f700 10 _finish_op 0x8cd22a0 seq 2491 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.413665 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:57.413676 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.413665 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2775 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] op_commit: 236364
2014-06-12 14:46:57.413691 7f9d7715b700  5 -- op tracker -- , seq: 748183, time: 2014-06-12 14:46:57.413684, event: op_commit, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.413719 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2775 crt=0'0 lcod 173'2774 mlcod 173'2774 active+clean] repop_all_committed: repop tid 236364 all committed 
2014-06-12 14:46:57.413731 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2774 active+clean] eval_repop repgather(0x228e0f00 173'2776 rep_tid=236364 committed?=1 applied?=0 lock=0 op=osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.413749 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2774 active+clean] log_op_stats osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.010341
2014-06-12 14:46:57.413768 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2774 active+clean] publish_stats_to_osd 173:3464
2014-06-12 14:46:57.413778 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2774 active+clean]  sending commit on repgather(0x228e0f00 173'2776 rep_tid=236364 committed?=1 applied?=0 lock=0 op=osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x6989400
2014-06-12 14:46:57.413812 7f9d7715b700  5 -- op tracker -- , seq: 748183, time: 2014-06-12 14:46:57.413806, event: commit_sent, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.413834 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2774 active+clean] op_applied: 236364
2014-06-12 14:46:57.413856 7f9d7795c700  5 -- op tracker -- , seq: 748183, time: 2014-06-12 14:46:57.413850, event: op_applied, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.413864 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2774 active+clean] op_applied on primary on version 173'2776
2014-06-12 14:46:57.413885 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2774 active+clean] repop_all_applied: repop tid 236364 all applied 
2014-06-12 14:46:57.413894 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2774 active+clean] eval_repop repgather(0x228e0f00 173'2776 rep_tid=236364 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.413918 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2774 active+clean] log_op_stats osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.010511
2014-06-12 14:46:57.413933 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2774 active+clean] publish_stats_to_osd 173:3465
2014-06-12 14:46:57.413951 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2774 active+clean]  requeue_ops 
2014-06-12 14:46:57.413961 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean]  removing repgather(0x228e0f00 173'2776 rep_tid=236364 committed?=1 applied?=1 op=osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.413979 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean]    q front is repgather(0x228e0f00 173'2776 rep_tid=236364 committed?=1 applied?=1 op=osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.413992 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] remove_repop repgather(0x228e0f00 173'2776 rep_tid=236364 committed?=1 applied?=1 op=osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.414005 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean]  requeue_ops 
2014-06-12 14:46:57.414030 7f9d7795c700  5 -- op tracker -- , seq: 748183, time: 2014-06-12 14:46:57.414024, event: done, op: osd_op(client.4309.0:141793 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 876544~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.417176 7f9d4397a700  5 -- op tracker -- , seq: 748186, time: 2014-06-12 14:46:57.416101, event: header_read, op: osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.417194 7f9d4397a700  5 -- op tracker -- , seq: 748186, time: 2014-06-12 14:46:57.416103, event: throttled, op: osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.417200 7f9d4397a700  5 -- op tracker -- , seq: 748186, time: 2014-06-12 14:46:57.417144, event: all_read, op: osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.417205 7f9d4397a700  5 -- op tracker -- , seq: 748186, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.417210 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:57.417230 7f9d4397a700 15 osd.2 173 enqueue_op 0x2afd400 prio 63 cost 516096 latency 0.001129 osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.417270 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.417278 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2afd400 prio 63 cost 516096 latency 0.001176 osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean]
2014-06-12 14:46:57.417299 7f9d63bd8700  5 -- op tracker -- , seq: 748186, time: 2014-06-12 14:46:57.417295, event: reached_pg, op: osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.417307 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:57.417318 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] handle_message: 0x2afd400
2014-06-12 14:46:57.417325 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] do_op osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:57.417343 7f9d63bd8700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_'
2014-06-12 14:46:57.417473 7f9d63bd8700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_' = 0
2014-06-12 14:46:57.417486 7f9d63bd8700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset'
2014-06-12 14:46:57.417533 7f9d63bd8700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset' = 0
2014-06-12 14:46:57.417541 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] populate_obc_watchers 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.417548 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.417554 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] get_object_context: creating obc from disk: 0x299e4580
2014-06-12 14:46:57.417560 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] get_object_context: 0x299e4580 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 rwstate(none n=0 w=0) oi: 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2776 client.4309.0:141793 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2776) ssc: 0x9345880 snapset: 0=[]:[]+head
2014-06-12 14:46:57.417569 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] find_object_context 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 @head oi=103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2776 client.4309.0:141793 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2776)
2014-06-12 14:46:57.417586 7f9d63bd8700  5 -- op tracker -- , seq: 748186, time: 2014-06-12 14:46:57.417582, event: started, op: osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.417595 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] execute_ctx 0x7ad1200
2014-06-12 14:46:57.417606 7f9d63bd8700  5 -- op tracker -- , seq: 748186, time: 2014-06-12 14:46:57.417603, event: started, op: osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.417621 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] do_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] ov 173'2776 av 173'2777 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:57.417630 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] do_osd_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096]
2014-06-12 14:46:57.417637 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:57.417645 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] do_osd_op  write 1392640~516096
2014-06-12 14:46:57.417656 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:57.417663 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 snapset=0x93458b8  snapc=0=[]
2014-06-12 14:46:57.417670 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:57.417676 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] finish_ctx 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 0x7ad1200 op modify  
2014-06-12 14:46:57.417684 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean]  set mtime to 2014-06-12 08:45:59.795430
2014-06-12 14:46:57.417696 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean]  final snapset 0=[]:[]+head in 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.417707 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean]  zeroing write result code 0
2014-06-12 14:46:57.417714 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] new_repop rep_tid 236365 on osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.417724 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] issue_repop rep_tid 236365 o 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.417738 7f9d63bd8700  5 -- op tracker -- , seq: 748186, time: 2014-06-12 14:46:57.417735, event: waiting for subops from 27, op: osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.417762 7f9d63bd8700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:57.417777 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2776 (0'0,173'2776] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] append_log log((0'0,173'2776], crt=0'0) [173'2777 (173'2776) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141794 2014-06-12 08:45:59.795430]
2014-06-12 14:46:57.417793 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2776 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] add_log_entry 173'2777 (173'2776) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141794 2014-06-12 08:45:59.795430
2014-06-12 14:46:57.417806 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2776 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] append_log  adding 1 keys
2014-06-12 14:46:57.417826 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2777, trimmed: 
2014-06-12 14:46:57.417843 7f9d63bd8700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.417846 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.417847 7f9d63bd8700  5 queue_op 0x7b47e60 seq 2492 osr(3.312 0x276d540) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:57.417852 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2776 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] eval_repop repgather(0x2335ccc0 173'2777 rep_tid=236365 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.417864 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2afd400 finish
2014-06-12 14:46:57.417866 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.417868 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.417854 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (1 active)
2014-06-12 14:46:57.417872 7f9d7d167700  5 _do_op 0x7b47e60 seq 2492 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:57.417877 7f9d7d167700 10 _do_transaction on 0xd5e2fc0
2014-06-12 14:46:57.417884 7f9d7d167700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.418034 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.418060 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.418102 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.418191 7f9d7d167700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.418199 7f9d7d167700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:57.418204 7f9d7d167700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.418217 7f9d7d167700 15 _write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 1392640~516096
2014-06-12 14:46:57.418334 7f9d7d167700 10 lookup_strip_header done  cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.418470 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:57.419754 7f9d7d167700 10 _generic_write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 1392640~516096 = 0
2014-06-12 14:46:57.419796 7f9d7d167700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.419807 7f9d7d167700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.419816 7f9d7d167700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.419821 7f9d7d167700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.420222 7f9d7d167700 20 set_header setting 17155 cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 parent seq 0
2014-06-12 14:46:57.420282 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.420293 7f9d7d167700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:57.420300 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:57.422414 7f9d7d167700 10 _do_op 0x7b47e60 seq 2492 r = 0, finisher 0x6647880 0x3786fe80
2014-06-12 14:46:57.422435 7f9d7d167700 10 _finish_op 0x7b47e60 seq 2492 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.422442 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:57.422446 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.422441 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2776 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] op_commit: 236365
2014-06-12 14:46:57.422478 7f9d7715b700  5 -- op tracker -- , seq: 748186, time: 2014-06-12 14:46:57.422466, event: op_commit, op: osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.422499 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2776 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] op_applied: 236365
2014-06-12 14:46:57.422526 7f9d7795c700  5 -- op tracker -- , seq: 748186, time: 2014-06-12 14:46:57.422519, event: op_applied, op: osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.422538 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2776 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] op_applied on primary on version 173'2777
2014-06-12 14:46:57.424926 7f9d58c5a700  5 -- op tracker -- , seq: 748187, time: 2014-06-12 14:46:57.424889, event: header_read, op: osd_sub_op_reply(client.4309.0:141794 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.424941 7f9d58c5a700  5 -- op tracker -- , seq: 748187, time: 2014-06-12 14:46:57.424890, event: throttled, op: osd_sub_op_reply(client.4309.0:141794 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.424948 7f9d58c5a700  5 -- op tracker -- , seq: 748187, time: 2014-06-12 14:46:57.424899, event: all_read, op: osd_sub_op_reply(client.4309.0:141794 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.424956 7f9d58c5a700  5 -- op tracker -- , seq: 748187, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141794 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.424967 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141794 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:57.424971 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:57.424977 7f9d58c5a700 15 osd.2 173 enqueue_op 0x12bf7c00 prio 196 cost 0 latency 0.000087 osd_sub_op_reply(client.4309.0:141794 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:57.425029 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.425036 7f9d633d7700 10 osd.2 173 dequeue_op 0x12bf7c00 prio 196 cost 0 latency 0.000146 osd_sub_op_reply(client.4309.0:141794 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2776 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean]
2014-06-12 14:46:57.425052 7f9d633d7700  5 -- op tracker -- , seq: 748187, time: 2014-06-12 14:46:57.425049, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141794 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.425062 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2776 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] handle_message: 0x12bf7c00
2014-06-12 14:46:57.425073 7f9d633d7700  5 -- op tracker -- , seq: 748187, time: 2014-06-12 14:46:57.425071, event: started, op: osd_sub_op_reply(client.4309.0:141794 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.425077 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2776 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] sub_op_modify_reply: tid 236365 op  ack_type 4 from 27
2014-06-12 14:46:57.425090 7f9d633d7700  5 -- op tracker -- , seq: 748186, time: 2014-06-12 14:46:57.425086, event: sub_op_commit_rec, op: osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.425098 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2776 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] repop_all_applied: repop tid 236365 all applied 
2014-06-12 14:46:57.425105 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2776 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] eval_repop repgather(0x2335ccc0 173'2777 rep_tid=236365 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.425114 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2776 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean]  sending ack on repgather(0x2335ccc0 173'2777 rep_tid=236365 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x8df2500
2014-06-12 14:46:57.425133 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2776 crt=0'0 lcod 173'2775 mlcod 173'2775 active+clean] repop_all_committed: repop tid 236365 all committed 
2014-06-12 14:46:57.425145 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2775 active+clean] eval_repop repgather(0x2335ccc0 173'2777 rep_tid=236365 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.425174 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2775 active+clean] log_op_stats osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.009031 lat 0.009072
2014-06-12 14:46:57.425188 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2775 active+clean] publish_stats_to_osd 173:3466
2014-06-12 14:46:57.425196 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2775 active+clean]  sending commit on repgather(0x2335ccc0 173'2777 rep_tid=236365 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x133fe000
2014-06-12 14:46:57.425220 7f9d633d7700  5 -- op tracker -- , seq: 748186, time: 2014-06-12 14:46:57.425215, event: commit_sent, op: osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.425230 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2775 active+clean]  requeue_ops 
2014-06-12 14:46:57.425240 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean]  removing repgather(0x2335ccc0 173'2777 rep_tid=236365 committed?=1 applied?=1 op=osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.425251 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean]    q front is repgather(0x2335ccc0 173'2777 rep_tid=236365 committed?=1 applied?=1 op=osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.425259 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] remove_repop repgather(0x2335ccc0 173'2777 rep_tid=236365 committed?=1 applied?=1 op=osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.425267 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean]  requeue_ops 
2014-06-12 14:46:57.425283 7f9d633d7700  5 -- op tracker -- , seq: 748186, time: 2014-06-12 14:46:57.425280, event: done, op: osd_op(client.4309.0:141794 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.425297 7f9d633d7700 10 osd.2 173 dequeue_op 0x12bf7c00 finish
2014-06-12 14:46:57.425302 7f9d633d7700  5 -- op tracker -- , seq: 748187, time: 2014-06-12 14:46:57.425300, event: done, op: osd_sub_op_reply(client.4309.0:141794 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.425308 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.425318 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.428618 7f9d4397a700  5 -- op tracker -- , seq: 748188, time: 2014-06-12 14:46:57.427521, event: header_read, op: osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.428640 7f9d4397a700  5 -- op tracker -- , seq: 748188, time: 2014-06-12 14:46:57.427523, event: throttled, op: osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.428650 7f9d4397a700  5 -- op tracker -- , seq: 748188, time: 2014-06-12 14:46:57.428585, event: all_read, op: osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.428663 7f9d4397a700  5 -- op tracker -- , seq: 748188, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.428670 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:57.428695 7f9d4397a700 15 osd.2 173 enqueue_op 0x17c2af00 prio 63 cost 516096 latency 0.001174 osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.428739 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.428746 7f9d61bd4700 10 osd.2 173 dequeue_op 0x17c2af00 prio 63 cost 516096 latency 0.001225 osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean]
2014-06-12 14:46:57.428765 7f9d61bd4700  5 -- op tracker -- , seq: 748188, time: 2014-06-12 14:46:57.428761, event: reached_pg, op: osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.428774 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:57.428785 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] handle_message: 0x17c2af00
2014-06-12 14:46:57.428793 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] do_op osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:57.428811 7f9d61bd4700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_'
2014-06-12 14:46:57.428935 7f9d61bd4700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_' = 0
2014-06-12 14:46:57.428948 7f9d61bd4700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset'
2014-06-12 14:46:57.429027 7f9d61bd4700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset' = 0
2014-06-12 14:46:57.429047 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] populate_obc_watchers 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.429067 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.429077 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] get_object_context: creating obc from disk: 0x212fd340
2014-06-12 14:46:57.429087 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] get_object_context: 0x212fd340 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 rwstate(none n=0 w=0) oi: 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2777 client.4309.0:141794 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2777) ssc: 0x5d5bb20 snapset: 0=[]:[]+head
2014-06-12 14:46:57.429097 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] find_object_context 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 @head oi=103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2777 client.4309.0:141794 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2777)
2014-06-12 14:46:57.429122 7f9d61bd4700  5 -- op tracker -- , seq: 748188, time: 2014-06-12 14:46:57.429116, event: started, op: osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.429129 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] execute_ctx 0x30f7e400
2014-06-12 14:46:57.429149 7f9d61bd4700  5 -- op tracker -- , seq: 748188, time: 2014-06-12 14:46:57.429144, event: started, op: osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.429158 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] do_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] ov 173'2777 av 173'2778 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:57.429177 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] do_osd_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096]
2014-06-12 14:46:57.429185 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:57.429198 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] do_osd_op  write 1908736~516096
2014-06-12 14:46:57.429214 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:57.429226 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 snapset=0x5d5bb58  snapc=0=[]
2014-06-12 14:46:57.429240 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:57.429248 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] finish_ctx 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 0x30f7e400 op modify  
2014-06-12 14:46:57.429256 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean]  set mtime to 2014-06-12 08:45:59.806846
2014-06-12 14:46:57.429284 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean]  final snapset 0=[]:[]+head in 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.429298 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean]  zeroing write result code 0
2014-06-12 14:46:57.429311 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] new_repop rep_tid 236366 on osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.429345 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] issue_repop rep_tid 236366 o 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.429371 7f9d61bd4700  5 -- op tracker -- , seq: 748188, time: 2014-06-12 14:46:57.429366, event: waiting for subops from 27, op: osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.429391 7f9d61bd4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:57.429428 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2777 (0'0,173'2777] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] append_log log((0'0,173'2777], crt=0'0) [173'2778 (173'2777) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141795 2014-06-12 08:45:59.806846]
2014-06-12 14:46:57.429452 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2777 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] add_log_entry 173'2778 (173'2777) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141795 2014-06-12 08:45:59.806846
2014-06-12 14:46:57.429472 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2777 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] append_log  adding 1 keys
2014-06-12 14:46:57.429499 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2778, trimmed: 
2014-06-12 14:46:57.429516 7f9d61bd4700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.429520 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.429521 7f9d61bd4700  5 queue_op 0x9bac960 seq 2493 osr(3.312 0x276d540) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:57.429527 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2777 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] eval_repop repgather(0x5323080 173'2778 rep_tid=236366 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.429544 7f9d61bd4700 10 osd.2 173 dequeue_op 0x17c2af00 finish
2014-06-12 14:46:57.429548 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.429550 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.429556 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (1 active)
2014-06-12 14:46:57.429560 7f9d7915f700  5 _do_op 0x9bac960 seq 2493 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:57.429563 7f9d7915f700 10 _do_transaction on 0x5323380
2014-06-12 14:46:57.429567 7f9d7915f700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.429805 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.429825 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.429878 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.429906 7f9d7915f700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.429914 7f9d7915f700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:57.429920 7f9d7915f700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.429938 7f9d7915f700 15 _write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 1908736~516096
2014-06-12 14:46:57.430177 7f9d7915f700 10 lookup_strip_header done  cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.430290 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:57.431650 7f9d7915f700 10 _generic_write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 1908736~516096 = 0
2014-06-12 14:46:57.431676 7f9d7915f700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.431686 7f9d7915f700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.431691 7f9d7915f700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.431696 7f9d7915f700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.431831 7f9d7915f700 20 set_header setting 17155 cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 parent seq 0
2014-06-12 14:46:57.431882 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.431893 7f9d7915f700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:57.431900 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:57.433582 7f9d7915f700 10 _do_op 0x9bac960 seq 2493 r = 0, finisher 0x5d40160 0x1b743540
2014-06-12 14:46:57.433598 7f9d7915f700 10 _finish_op 0x9bac960 seq 2493 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.433604 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:57.433626 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.433605 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2777 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] op_commit: 236366
2014-06-12 14:46:57.433650 7f9d7715b700  5 -- op tracker -- , seq: 748188, time: 2014-06-12 14:46:57.433638, event: op_commit, op: osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.433667 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2777 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] op_applied: 236366
2014-06-12 14:46:57.433695 7f9d7795c700  5 -- op tracker -- , seq: 748188, time: 2014-06-12 14:46:57.433688, event: op_applied, op: osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.433707 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2777 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] op_applied on primary on version 173'2778
2014-06-12 14:46:57.436684 7f9d58c5a700  5 -- op tracker -- , seq: 748189, time: 2014-06-12 14:46:57.436646, event: header_read, op: osd_sub_op_reply(client.4309.0:141795 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.436691 7f9d58c5a700  5 -- op tracker -- , seq: 748189, time: 2014-06-12 14:46:57.436647, event: throttled, op: osd_sub_op_reply(client.4309.0:141795 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.436696 7f9d58c5a700  5 -- op tracker -- , seq: 748189, time: 2014-06-12 14:46:57.436656, event: all_read, op: osd_sub_op_reply(client.4309.0:141795 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.436708 7f9d58c5a700  5 -- op tracker -- , seq: 748189, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141795 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.436712 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141795 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:57.436716 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:57.436721 7f9d58c5a700 15 osd.2 173 enqueue_op 0x28fd8b00 prio 196 cost 0 latency 0.000074 osd_sub_op_reply(client.4309.0:141795 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:57.436762 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.436768 7f9d67be0700 10 osd.2 173 dequeue_op 0x28fd8b00 prio 196 cost 0 latency 0.000122 osd_sub_op_reply(client.4309.0:141795 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2777 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean]
2014-06-12 14:46:57.436783 7f9d67be0700  5 -- op tracker -- , seq: 748189, time: 2014-06-12 14:46:57.436780, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141795 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.436790 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2777 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] handle_message: 0x28fd8b00
2014-06-12 14:46:57.436800 7f9d67be0700  5 -- op tracker -- , seq: 748189, time: 2014-06-12 14:46:57.436798, event: started, op: osd_sub_op_reply(client.4309.0:141795 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.436805 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2777 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] sub_op_modify_reply: tid 236366 op  ack_type 4 from 27
2014-06-12 14:46:57.436817 7f9d67be0700  5 -- op tracker -- , seq: 748188, time: 2014-06-12 14:46:57.436813, event: sub_op_commit_rec, op: osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.436823 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2777 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] repop_all_applied: repop tid 236366 all applied 
2014-06-12 14:46:57.436830 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2777 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] eval_repop repgather(0x5323080 173'2778 rep_tid=236366 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.436840 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2777 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean]  sending ack on repgather(0x5323080 173'2778 rep_tid=236366 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x30ccc280
2014-06-12 14:46:57.436860 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2777 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] repop_all_committed: repop tid 236366 all committed 
2014-06-12 14:46:57.436870 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2776 active+clean] eval_repop repgather(0x5323080 173'2778 rep_tid=236366 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.436885 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2776 active+clean] log_op_stats osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.009338 lat 0.009363
2014-06-12 14:46:57.436903 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2776 active+clean] publish_stats_to_osd 173:3467
2014-06-12 14:46:57.436912 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2776 active+clean]  sending commit on repgather(0x5323080 173'2778 rep_tid=236366 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0xc0c8000
2014-06-12 14:46:57.436933 7f9d67be0700  5 -- op tracker -- , seq: 748188, time: 2014-06-12 14:46:57.436928, event: commit_sent, op: osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.436943 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2776 active+clean]  requeue_ops 
2014-06-12 14:46:57.436953 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean]  removing repgather(0x5323080 173'2778 rep_tid=236366 committed?=1 applied?=1 op=osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.436963 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean]    q front is repgather(0x5323080 173'2778 rep_tid=236366 committed?=1 applied?=1 op=osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.436971 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] remove_repop repgather(0x5323080 173'2778 rep_tid=236366 committed?=1 applied?=1 op=osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.436979 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean]  requeue_ops 
2014-06-12 14:46:57.436995 7f9d67be0700  5 -- op tracker -- , seq: 748188, time: 2014-06-12 14:46:57.436992, event: done, op: osd_op(client.4309.0:141795 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 1908736~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.437012 7f9d67be0700 10 osd.2 173 dequeue_op 0x28fd8b00 finish
2014-06-12 14:46:57.437016 7f9d67be0700  5 -- op tracker -- , seq: 748189, time: 2014-06-12 14:46:57.437014, event: done, op: osd_sub_op_reply(client.4309.0:141795 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.437026 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.437028 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.440187 7f9d4397a700  5 -- op tracker -- , seq: 748190, time: 2014-06-12 14:46:57.439213, event: header_read, op: osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.440196 7f9d4397a700  5 -- op tracker -- , seq: 748190, time: 2014-06-12 14:46:57.439215, event: throttled, op: osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.440202 7f9d4397a700  5 -- op tracker -- , seq: 748190, time: 2014-06-12 14:46:57.440161, event: all_read, op: osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.440207 7f9d4397a700  5 -- op tracker -- , seq: 748190, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.440212 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:57.440241 7f9d4397a700 15 osd.2 173 enqueue_op 0x6f4a900 prio 63 cost 516096 latency 0.001028 osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.440273 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.440281 7f9d673df700 10 osd.2 173 dequeue_op 0x6f4a900 prio 63 cost 516096 latency 0.001068 osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean]
2014-06-12 14:46:57.440306 7f9d673df700  5 -- op tracker -- , seq: 748190, time: 2014-06-12 14:46:57.440300, event: reached_pg, op: osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.440315 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:57.440331 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] handle_message: 0x6f4a900
2014-06-12 14:46:57.440342 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] do_op osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:57.440366 7f9d673df700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_'
2014-06-12 14:46:57.440512 7f9d673df700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_' = 0
2014-06-12 14:46:57.440528 7f9d673df700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset'
2014-06-12 14:46:57.440585 7f9d673df700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset' = 0
2014-06-12 14:46:57.440594 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] populate_obc_watchers 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.440608 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.440619 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] get_object_context: creating obc from disk: 0x28127340
2014-06-12 14:46:57.440629 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] get_object_context: 0x28127340 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 rwstate(none n=0 w=0) oi: 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2778 client.4309.0:141795 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2778) ssc: 0x16ee41c0 snapset: 0=[]:[]+head
2014-06-12 14:46:57.440644 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] find_object_context 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 @head oi=103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2778 client.4309.0:141795 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2778)
2014-06-12 14:46:57.440672 7f9d673df700  5 -- op tracker -- , seq: 748190, time: 2014-06-12 14:46:57.440666, event: started, op: osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.440679 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] execute_ctx 0x1b262600
2014-06-12 14:46:57.440697 7f9d673df700  5 -- op tracker -- , seq: 748190, time: 2014-06-12 14:46:57.440692, event: started, op: osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.440706 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] do_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] ov 173'2778 av 173'2779 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:57.440725 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] do_osd_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096]
2014-06-12 14:46:57.440747 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:57.440761 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] do_osd_op  write 2424832~516096
2014-06-12 14:46:57.440777 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:57.440797 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 snapset=0x16ee41f8  snapc=0=[]
2014-06-12 14:46:57.440810 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:57.440822 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] finish_ctx 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 0x1b262600 op modify  
2014-06-12 14:46:57.440839 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean]  set mtime to 2014-06-12 08:45:59.818542
2014-06-12 14:46:57.440861 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean]  final snapset 0=[]:[]+head in 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.440879 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean]  zeroing write result code 0
2014-06-12 14:46:57.440892 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] new_repop rep_tid 236367 on osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.440910 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] issue_repop rep_tid 236367 o 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.440937 7f9d673df700  5 -- op tracker -- , seq: 748190, time: 2014-06-12 14:46:57.440932, event: waiting for subops from 27, op: osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.440957 7f9d673df700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:57.440979 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2778 (0'0,173'2778] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] append_log log((0'0,173'2778], crt=0'0) [173'2779 (173'2778) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141796 2014-06-12 08:45:59.818542]
2014-06-12 14:46:57.441002 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2778 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] add_log_entry 173'2779 (173'2778) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141796 2014-06-12 08:45:59.818542
2014-06-12 14:46:57.441020 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2778 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] append_log  adding 1 keys
2014-06-12 14:46:57.441046 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2779, trimmed: 
2014-06-12 14:46:57.441062 7f9d673df700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.441065 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.441067 7f9d673df700  5 queue_op 0x5e431a0 seq 2494 osr(3.312 0x276d540) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:57.441073 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2778 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] eval_repop repgather(0x515e780 173'2779 rep_tid=236367 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.441095 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (1 active)
2014-06-12 14:46:57.441098 7f9d673df700 10 osd.2 173 dequeue_op 0x6f4a900 finish
2014-06-12 14:46:57.441102 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.441102 7f9d8016d700  5 _do_op 0x5e431a0 seq 2494 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:57.441104 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.441105 7f9d8016d700 10 _do_transaction on 0x384e2a80
2014-06-12 14:46:57.441109 7f9d8016d700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.441191 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.441227 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.441259 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.441476 7f9d8016d700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.441483 7f9d8016d700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:57.441487 7f9d8016d700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.441507 7f9d8016d700 15 _write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 2424832~516096
2014-06-12 14:46:57.441600 7f9d8016d700 10 lookup_strip_header done  cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.441846 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:57.443107 7f9d8016d700 10 _generic_write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 2424832~516096 = 0
2014-06-12 14:46:57.443133 7f9d8016d700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.443143 7f9d8016d700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.443147 7f9d8016d700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.443151 7f9d8016d700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.443291 7f9d8016d700 20 set_header setting 17155 cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 parent seq 0
2014-06-12 14:46:57.443332 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.443342 7f9d8016d700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:57.443348 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:57.445395 7f9d8016d700 10 _do_op 0x5e431a0 seq 2494 r = 0, finisher 0x10026f40 0x1a3f0280
2014-06-12 14:46:57.445410 7f9d8016d700 10 _finish_op 0x5e431a0 seq 2494 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.445417 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:57.445430 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.445424 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2778 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] op_commit: 236367
2014-06-12 14:46:57.445453 7f9d7715b700  5 -- op tracker -- , seq: 748190, time: 2014-06-12 14:46:57.445443, event: op_commit, op: osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.445470 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2778 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] op_applied: 236367
2014-06-12 14:46:57.445497 7f9d7795c700  5 -- op tracker -- , seq: 748190, time: 2014-06-12 14:46:57.445490, event: op_applied, op: osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.445508 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2778 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] op_applied on primary on version 173'2779
2014-06-12 14:46:57.447453 7f9d58c5a700  5 -- op tracker -- , seq: 748191, time: 2014-06-12 14:46:57.447421, event: header_read, op: osd_sub_op_reply(client.4309.0:141796 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.447462 7f9d58c5a700  5 -- op tracker -- , seq: 748191, time: 2014-06-12 14:46:57.447422, event: throttled, op: osd_sub_op_reply(client.4309.0:141796 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.447467 7f9d58c5a700  5 -- op tracker -- , seq: 748191, time: 2014-06-12 14:46:57.447432, event: all_read, op: osd_sub_op_reply(client.4309.0:141796 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.447471 7f9d58c5a700  5 -- op tracker -- , seq: 748191, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141796 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.447475 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141796 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:57.447478 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:57.447484 7f9d58c5a700 15 osd.2 173 enqueue_op 0x28fd8c00 prio 196 cost 0 latency 0.000063 osd_sub_op_reply(client.4309.0:141796 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:57.447509 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.447518 7f9d663dd700 10 osd.2 173 dequeue_op 0x28fd8c00 prio 196 cost 0 latency 0.000097 osd_sub_op_reply(client.4309.0:141796 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2778 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean]
2014-06-12 14:46:57.447537 7f9d663dd700  5 -- op tracker -- , seq: 748191, time: 2014-06-12 14:46:57.447534, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141796 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.447545 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2778 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] handle_message: 0x28fd8c00
2014-06-12 14:46:57.447560 7f9d663dd700  5 -- op tracker -- , seq: 748191, time: 2014-06-12 14:46:57.447557, event: started, op: osd_sub_op_reply(client.4309.0:141796 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.447567 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2778 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] sub_op_modify_reply: tid 236367 op  ack_type 4 from 27
2014-06-12 14:46:57.447585 7f9d663dd700  5 -- op tracker -- , seq: 748190, time: 2014-06-12 14:46:57.447579, event: sub_op_commit_rec, op: osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.447592 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2778 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] repop_all_applied: repop tid 236367 all applied 
2014-06-12 14:46:57.447604 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2778 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] eval_repop repgather(0x515e780 173'2779 rep_tid=236367 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.447620 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2778 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean]  sending ack on repgather(0x515e780 173'2779 rep_tid=236367 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x5ffe000
2014-06-12 14:46:57.447651 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2778 crt=0'0 lcod 173'2777 mlcod 173'2777 active+clean] repop_all_committed: repop tid 236367 all committed 
2014-06-12 14:46:57.447662 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2777 active+clean] eval_repop repgather(0x515e780 173'2779 rep_tid=236367 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.447679 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2777 active+clean] log_op_stats osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008436 lat 0.008465
2014-06-12 14:46:57.447697 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2777 active+clean] publish_stats_to_osd 173:3468
2014-06-12 14:46:57.447710 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2777 active+clean]  sending commit on repgather(0x515e780 173'2779 rep_tid=236367 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x6d1d180
2014-06-12 14:46:57.447742 7f9d663dd700  5 -- op tracker -- , seq: 748190, time: 2014-06-12 14:46:57.447737, event: commit_sent, op: osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.447750 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2777 active+clean]  requeue_ops 
2014-06-12 14:46:57.447761 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean]  removing repgather(0x515e780 173'2779 rep_tid=236367 committed?=1 applied?=1 op=osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.447776 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean]    q front is repgather(0x515e780 173'2779 rep_tid=236367 committed?=1 applied?=1 op=osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.447791 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] remove_repop repgather(0x515e780 173'2779 rep_tid=236367 committed?=1 applied?=1 op=osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.447806 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean]  requeue_ops 
2014-06-12 14:46:57.447840 7f9d663dd700  5 -- op tracker -- , seq: 748190, time: 2014-06-12 14:46:57.447835, event: done, op: osd_op(client.4309.0:141796 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.447864 7f9d663dd700 10 osd.2 173 dequeue_op 0x28fd8c00 finish
2014-06-12 14:46:57.447873 7f9d663dd700  5 -- op tracker -- , seq: 748191, time: 2014-06-12 14:46:57.447869, event: done, op: osd_sub_op_reply(client.4309.0:141796 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.447884 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.447886 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.448811 7f9d4397a700  5 -- op tracker -- , seq: 748192, time: 2014-06-12 14:46:57.448725, event: header_read, op: osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.448834 7f9d4397a700  5 -- op tracker -- , seq: 748192, time: 2014-06-12 14:46:57.448730, event: throttled, op: osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.448847 7f9d4397a700  5 -- op tracker -- , seq: 748192, time: 2014-06-12 14:46:57.448761, event: all_read, op: osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.448873 7f9d4397a700  5 -- op tracker -- , seq: 748192, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.448881 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:57.448909 7f9d4397a700 15 osd.2 173 enqueue_op 0x9face00 prio 63 cost 4096 latency 0.000183 osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.448952 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.448962 7f9d65bdc700 10 osd.2 173 dequeue_op 0x9face00 prio 63 cost 4096 latency 0.000237 osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean]
2014-06-12 14:46:57.449001 7f9d65bdc700  5 -- op tracker -- , seq: 748192, time: 2014-06-12 14:46:57.448996, event: reached_pg, op: osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.449015 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:57.449032 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] handle_message: 0x9face00
2014-06-12 14:46:57.449043 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] do_op osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:57.449069 7f9d65bdc700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_'
2014-06-12 14:46:57.449269 7f9d65bdc700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_' = 0
2014-06-12 14:46:57.449294 7f9d65bdc700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset'
2014-06-12 14:46:57.449469 7f9d65bdc700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset' = 0
2014-06-12 14:46:57.449487 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] populate_obc_watchers 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.449505 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.449518 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] get_object_context: creating obc from disk: 0x15697340
2014-06-12 14:46:57.449538 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] get_object_context: 0x15697340 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 rwstate(none n=0 w=0) oi: 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2779 client.4309.0:141796 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2779) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:46:57.449554 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] find_object_context 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 @head oi=103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2779 client.4309.0:141796 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2779)
2014-06-12 14:46:57.449580 7f9d65bdc700  5 -- op tracker -- , seq: 748192, time: 2014-06-12 14:46:57.449574, event: started, op: osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.449589 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] execute_ctx 0x2e411800
2014-06-12 14:46:57.449605 7f9d65bdc700  5 -- op tracker -- , seq: 748192, time: 2014-06-12 14:46:57.449601, event: started, op: osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.449628 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] do_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] ov 173'2779 av 173'2780 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:57.449656 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] do_osd_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096]
2014-06-12 14:46:57.449675 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:57.449690 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] do_osd_op  write 2940928~4096
2014-06-12 14:46:57.449707 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:57.449720 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:46:57.449732 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:57.449742 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] finish_ctx 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 0x2e411800 op modify  
2014-06-12 14:46:57.449753 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean]  set mtime to 2014-06-12 08:45:59.828956
2014-06-12 14:46:57.449776 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean]  final snapset 0=[]:[]+head in 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.449797 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean]  zeroing write result code 0
2014-06-12 14:46:57.449809 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] new_repop rep_tid 236368 on osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.449827 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] issue_repop rep_tid 236368 o 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.449854 7f9d65bdc700  5 -- op tracker -- , seq: 748192, time: 2014-06-12 14:46:57.449849, event: waiting for subops from 27, op: osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.449875 7f9d65bdc700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:57.449897 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2779 (0'0,173'2779] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] append_log log((0'0,173'2779], crt=0'0) [173'2780 (173'2779) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141797 2014-06-12 08:45:59.828956]
2014-06-12 14:46:57.449924 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2779 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] add_log_entry 173'2780 (173'2779) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141797 2014-06-12 08:45:59.828956
2014-06-12 14:46:57.449947 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2779 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] append_log  adding 1 keys
2014-06-12 14:46:57.449981 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2780, trimmed: 
2014-06-12 14:46:57.450012 7f9d65bdc700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.450015 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.450016 7f9d65bdc700  5 queue_op 0x7782d20 seq 2495 osr(3.312 0x276d540) 6389 bytes   (queue has 1 ops and 6389 bytes)
2014-06-12 14:46:57.450026 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (1 active)
2014-06-12 14:46:57.450021 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2779 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] eval_repop repgather(0x2b618fc0 173'2780 rep_tid=236368 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.450033 7f9d65bdc700 10 osd.2 173 dequeue_op 0x9face00 finish
2014-06-12 14:46:57.450033 7f9d79960700  5 _do_op 0x7782d20 seq 2495 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:57.450045 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.450047 7f9d79960700 10 _do_transaction on 0x2b618480
2014-06-12 14:46:57.450048 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.450055 7f9d79960700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.450168 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.450223 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.450316 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.450351 7f9d79960700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.450359 7f9d79960700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:57.450374 7f9d79960700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.450499 7f9d79960700 15 _write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 2940928~4096
2014-06-12 14:46:57.450646 7f9d79960700 10 lookup_strip_header done  cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.450668 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:57.450695 7f9d79960700 10 _generic_write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 2940928~4096 = 0
2014-06-12 14:46:57.450708 7f9d79960700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.450719 7f9d79960700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.450727 7f9d79960700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.450736 7f9d79960700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.450847 7f9d79960700 20 set_header setting 17155 cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 parent seq 0
2014-06-12 14:46:57.450884 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.451051 7f9d79960700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:57.451067 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:57.451166 7f9d79960700 10 _do_op 0x7782d20 seq 2495 r = 0, finisher 0x654de00 0x2a39b140
2014-06-12 14:46:57.451177 7f9d79960700 10 _finish_op 0x7782d20 seq 2495 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.451185 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:57.451188 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.451185 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2779 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] op_commit: 236368
2014-06-12 14:46:57.451226 7f9d7715b700  5 -- op tracker -- , seq: 748192, time: 2014-06-12 14:46:57.451216, event: op_commit, op: osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.451263 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2779 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] op_applied: 236368
2014-06-12 14:46:57.451281 7f9d58c5a700  5 -- op tracker -- , seq: 748193, time: 2014-06-12 14:46:57.451245, event: header_read, op: osd_sub_op_reply(client.4309.0:141797 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.451300 7f9d7795c700  5 -- op tracker -- , seq: 748192, time: 2014-06-12 14:46:57.451295, event: op_applied, op: osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.451310 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2779 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] op_applied on primary on version 173'2780
2014-06-12 14:46:57.451314 7f9d58c5a700  5 -- op tracker -- , seq: 748193, time: 2014-06-12 14:46:57.451246, event: throttled, op: osd_sub_op_reply(client.4309.0:141797 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.451320 7f9d58c5a700  5 -- op tracker -- , seq: 748193, time: 2014-06-12 14:46:57.451256, event: all_read, op: osd_sub_op_reply(client.4309.0:141797 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.451324 7f9d58c5a700  5 -- op tracker -- , seq: 748193, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141797 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.451327 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141797 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:57.451330 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:57.451333 7f9d58c5a700 15 osd.2 173 enqueue_op 0x28fd9800 prio 196 cost 0 latency 0.000088 osd_sub_op_reply(client.4309.0:141797 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:57.451351 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.451356 7f9d62bd6700 10 osd.2 173 dequeue_op 0x28fd9800 prio 196 cost 0 latency 0.000111 osd_sub_op_reply(client.4309.0:141797 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2779 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean]
2014-06-12 14:46:57.451368 7f9d62bd6700  5 -- op tracker -- , seq: 748193, time: 2014-06-12 14:46:57.451366, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141797 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.451373 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2779 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] handle_message: 0x28fd9800
2014-06-12 14:46:57.451382 7f9d62bd6700  5 -- op tracker -- , seq: 748193, time: 2014-06-12 14:46:57.451380, event: started, op: osd_sub_op_reply(client.4309.0:141797 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.451385 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2779 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] sub_op_modify_reply: tid 236368 op  ack_type 4 from 27
2014-06-12 14:46:57.451395 7f9d62bd6700  5 -- op tracker -- , seq: 748192, time: 2014-06-12 14:46:57.451392, event: sub_op_commit_rec, op: osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.451402 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2779 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] repop_all_applied: repop tid 236368 all applied 
2014-06-12 14:46:57.451408 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2779 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] eval_repop repgather(0x2b618fc0 173'2780 rep_tid=236368 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.451417 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2779 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean]  sending ack on repgather(0x2b618fc0 173'2780 rep_tid=236368 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x16cf3400
2014-06-12 14:46:57.451436 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2779 crt=0'0 lcod 173'2778 mlcod 173'2778 active+clean] repop_all_committed: repop tid 236368 all committed 
2014-06-12 14:46:57.451447 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2778 active+clean] eval_repop repgather(0x2b618fc0 173'2780 rep_tid=236368 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.451465 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2778 active+clean] log_op_stats osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 inb 4375 outb 0 rlat 0.002710 lat 0.002740
2014-06-12 14:46:57.451482 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2778 active+clean] publish_stats_to_osd 173:3469
2014-06-12 14:46:57.451494 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2778 active+clean]  sending commit on repgather(0x2b618fc0 173'2780 rep_tid=236368 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x28efb400
2014-06-12 14:46:57.451530 7f9d62bd6700  5 -- op tracker -- , seq: 748192, time: 2014-06-12 14:46:57.451524, event: commit_sent, op: osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.451546 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2778 active+clean]  requeue_ops 
2014-06-12 14:46:57.451557 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean]  removing repgather(0x2b618fc0 173'2780 rep_tid=236368 committed?=1 applied?=1 op=osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.451577 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean]    q front is repgather(0x2b618fc0 173'2780 rep_tid=236368 committed?=1 applied?=1 op=osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.451592 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] remove_repop repgather(0x2b618fc0 173'2780 rep_tid=236368 committed?=1 applied?=1 op=osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.451605 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean]  requeue_ops 
2014-06-12 14:46:57.451627 7f9d62bd6700  5 -- op tracker -- , seq: 748192, time: 2014-06-12 14:46:57.451621, event: done, op: osd_op(client.4309.0:141797 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.451642 7f9d62bd6700 10 osd.2 173 dequeue_op 0x28fd9800 finish
2014-06-12 14:46:57.451650 7f9d62bd6700  5 -- op tracker -- , seq: 748193, time: 2014-06-12 14:46:57.451646, event: done, op: osd_sub_op_reply(client.4309.0:141797 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.451664 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.451669 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.455091 7f9d4397a700  5 -- op tracker -- , seq: 748194, time: 2014-06-12 14:46:57.454124, event: header_read, op: osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.455101 7f9d4397a700  5 -- op tracker -- , seq: 748194, time: 2014-06-12 14:46:57.454126, event: throttled, op: osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.455107 7f9d4397a700  5 -- op tracker -- , seq: 748194, time: 2014-06-12 14:46:57.455058, event: all_read, op: osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.455112 7f9d4397a700  5 -- op tracker -- , seq: 748194, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.455118 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:57.455139 7f9d4397a700 15 osd.2 173 enqueue_op 0x22b3b200 prio 63 cost 516096 latency 0.001014 osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.455160 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.455166 7f9d683e1700 10 osd.2 173 dequeue_op 0x22b3b200 prio 63 cost 516096 latency 0.001042 osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean]
2014-06-12 14:46:57.455184 7f9d683e1700  5 -- op tracker -- , seq: 748194, time: 2014-06-12 14:46:57.455180, event: reached_pg, op: osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.455190 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:57.455202 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] handle_message: 0x22b3b200
2014-06-12 14:46:57.455209 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] do_op osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:57.455226 7f9d683e1700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_'
2014-06-12 14:46:57.455399 7f9d683e1700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_' = 0
2014-06-12 14:46:57.455425 7f9d683e1700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset'
2014-06-12 14:46:57.455530 7f9d683e1700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset' = 0
2014-06-12 14:46:57.455545 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] populate_obc_watchers 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.455567 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.455581 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] get_object_context: creating obc from disk: 0x66bc840
2014-06-12 14:46:57.455593 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] get_object_context: 0x66bc840 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 rwstate(none n=0 w=0) oi: 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2780 client.4309.0:141797 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2780) ssc: 0x5d5a700 snapset: 0=[]:[]+head
2014-06-12 14:46:57.455616 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] find_object_context 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 @head oi=103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2780 client.4309.0:141797 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2780)
2014-06-12 14:46:57.455645 7f9d683e1700  5 -- op tracker -- , seq: 748194, time: 2014-06-12 14:46:57.455642, event: started, op: osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.455652 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] execute_ctx 0x2d2a6c00
2014-06-12 14:46:57.455662 7f9d683e1700  5 -- op tracker -- , seq: 748194, time: 2014-06-12 14:46:57.455660, event: started, op: osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.455668 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] do_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] ov 173'2780 av 173'2781 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:57.455678 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] do_osd_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096]
2014-06-12 14:46:57.455685 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:57.455693 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] do_osd_op  write 2945024~516096
2014-06-12 14:46:57.455704 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:57.455712 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 snapset=0x5d5a738  snapc=0=[]
2014-06-12 14:46:57.455719 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:57.455725 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] finish_ctx 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 0x2d2a6c00 op modify  
2014-06-12 14:46:57.455733 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean]  set mtime to 2014-06-12 08:45:59.833345
2014-06-12 14:46:57.455745 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean]  final snapset 0=[]:[]+head in 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.455757 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean]  zeroing write result code 0
2014-06-12 14:46:57.455765 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] new_repop rep_tid 236369 on osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.455776 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] issue_repop rep_tid 236369 o 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.455790 7f9d683e1700  5 -- op tracker -- , seq: 748194, time: 2014-06-12 14:46:57.455788, event: waiting for subops from 27, op: osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.455806 7f9d683e1700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:57.455822 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2780 (0'0,173'2780] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] append_log log((0'0,173'2780], crt=0'0) [173'2781 (173'2780) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141798 2014-06-12 08:45:59.833345]
2014-06-12 14:46:57.455838 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2780 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] add_log_entry 173'2781 (173'2780) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141798 2014-06-12 08:45:59.833345
2014-06-12 14:46:57.455852 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2780 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] append_log  adding 1 keys
2014-06-12 14:46:57.455873 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2781, trimmed: 
2014-06-12 14:46:57.455888 7f9d683e1700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.455892 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.455893 7f9d683e1700  5 queue_op 0x68539e0 seq 2496 osr(3.312 0x276d540) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:57.455898 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2780 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] eval_repop repgather(0x1f066e40 173'2781 rep_tid=236369 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.455909 7f9d683e1700 10 osd.2 173 dequeue_op 0x22b3b200 finish
2014-06-12 14:46:57.455911 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.455913 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.455995 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (1 active)
2014-06-12 14:46:57.456001 7f9d7d968700  5 _do_op 0x68539e0 seq 2496 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:57.456001 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:57.456005 7f9d7d968700 10 _do_transaction on 0x51aca80
2014-06-12 14:46:57.456011 7f9d7d968700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.456044 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:46:57.456098 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.456124 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.456170 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.456188 7f9d7d968700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.456196 7f9d7d968700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:57.456202 7f9d7d968700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.456220 7f9d7d968700 15 _write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 2945024~516096
2014-06-12 14:46:57.456581 7f9d7d968700 10 lookup_strip_header done  cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.456732 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:57.458014 7f9d7d968700 10 _generic_write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 2945024~516096 = 0
2014-06-12 14:46:57.458042 7f9d7d968700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.458053 7f9d7d968700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.458058 7f9d7d968700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.458062 7f9d7d968700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.458196 7f9d7d968700 20 set_header setting 17155 cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 parent seq 0
2014-06-12 14:46:57.458249 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.458259 7f9d7d968700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:57.458266 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:57.459923 7f9d7d968700 10 _do_op 0x68539e0 seq 2496 r = 0, finisher 0x85884e0 0x1a3f0a00
2014-06-12 14:46:57.459941 7f9d7d968700 10 _finish_op 0x68539e0 seq 2496 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.459947 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:57.459954 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.459949 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2780 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] op_commit: 236369
2014-06-12 14:46:57.459999 7f9d7715b700  5 -- op tracker -- , seq: 748194, time: 2014-06-12 14:46:57.459981, event: op_commit, op: osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.460029 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2780 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] op_applied: 236369
2014-06-12 14:46:57.460069 7f9d7795c700  5 -- op tracker -- , seq: 748194, time: 2014-06-12 14:46:57.460058, event: op_applied, op: osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.460088 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2780 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] op_applied on primary on version 173'2781
2014-06-12 14:46:57.461897 7f9d58c5a700  5 -- op tracker -- , seq: 748195, time: 2014-06-12 14:46:57.461828, event: header_read, op: osd_sub_op_reply(client.4309.0:141798 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.461920 7f9d58c5a700  5 -- op tracker -- , seq: 748195, time: 2014-06-12 14:46:57.461830, event: throttled, op: osd_sub_op_reply(client.4309.0:141798 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.461928 7f9d58c5a700  5 -- op tracker -- , seq: 748195, time: 2014-06-12 14:46:57.461848, event: all_read, op: osd_sub_op_reply(client.4309.0:141798 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.461935 7f9d58c5a700  5 -- op tracker -- , seq: 748195, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141798 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.461941 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141798 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:57.461959 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:57.461969 7f9d58c5a700 15 osd.2 173 enqueue_op 0x28fd8500 prio 196 cost 0 latency 0.000141 osd_sub_op_reply(client.4309.0:141798 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:57.462012 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.462025 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x28fd8500 prio 196 cost 0 latency 0.000197 osd_sub_op_reply(client.4309.0:141798 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2780 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean]
2014-06-12 14:46:57.462052 7f9d6b3e7700  5 -- op tracker -- , seq: 748195, time: 2014-06-12 14:46:57.462047, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141798 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.462065 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2780 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] handle_message: 0x28fd8500
2014-06-12 14:46:57.462084 7f9d6b3e7700  5 -- op tracker -- , seq: 748195, time: 2014-06-12 14:46:57.462079, event: started, op: osd_sub_op_reply(client.4309.0:141798 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.462091 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2780 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] sub_op_modify_reply: tid 236369 op  ack_type 4 from 27
2014-06-12 14:46:57.462114 7f9d6b3e7700  5 -- op tracker -- , seq: 748194, time: 2014-06-12 14:46:57.462106, event: sub_op_commit_rec, op: osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.462123 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2780 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] repop_all_applied: repop tid 236369 all applied 
2014-06-12 14:46:57.462137 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2780 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] eval_repop repgather(0x1f066e40 173'2781 rep_tid=236369 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.462157 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2780 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean]  sending ack on repgather(0x1f066e40 173'2781 rep_tid=236369 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x37813b80
2014-06-12 14:46:57.462198 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2780 crt=0'0 lcod 173'2779 mlcod 173'2779 active+clean] repop_all_committed: repop tid 236369 all committed 
2014-06-12 14:46:57.462212 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2779 active+clean] eval_repop repgather(0x1f066e40 173'2781 rep_tid=236369 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.462232 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2779 active+clean] log_op_stats osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.008072 lat 0.008107
2014-06-12 14:46:57.462254 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2779 active+clean] publish_stats_to_osd 173:3470
2014-06-12 14:46:57.462270 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2779 active+clean]  sending commit on repgather(0x1f066e40 173'2781 rep_tid=236369 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x20236780
2014-06-12 14:46:57.462331 7f9d6b3e7700  5 -- op tracker -- , seq: 748194, time: 2014-06-12 14:46:57.462323, event: commit_sent, op: osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.462341 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2779 active+clean]  requeue_ops 
2014-06-12 14:46:57.462355 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean]  removing repgather(0x1f066e40 173'2781 rep_tid=236369 committed?=1 applied?=1 op=osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.462382 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean]    q front is repgather(0x1f066e40 173'2781 rep_tid=236369 committed?=1 applied?=1 op=osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.462391 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] remove_repop repgather(0x1f066e40 173'2781 rep_tid=236369 committed?=1 applied?=1 op=osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.462403 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean]  requeue_ops 
2014-06-12 14:46:57.462442 7f9d6b3e7700  5 -- op tracker -- , seq: 748194, time: 2014-06-12 14:46:57.462435, event: done, op: osd_op(client.4309.0:141798 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.462466 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x28fd8500 finish
2014-06-12 14:46:57.462475 7f9d6b3e7700  5 -- op tracker -- , seq: 748195, time: 2014-06-12 14:46:57.462470, event: done, op: osd_sub_op_reply(client.4309.0:141798 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.462487 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.462491 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.465694 7f9d4397a700  5 -- op tracker -- , seq: 748196, time: 2014-06-12 14:46:57.464787, event: header_read, op: osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.465706 7f9d4397a700  5 -- op tracker -- , seq: 748196, time: 2014-06-12 14:46:57.464796, event: throttled, op: osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.465759 7f9d4397a700  5 -- op tracker -- , seq: 748196, time: 2014-06-12 14:46:57.465658, event: all_read, op: osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.465767 7f9d4397a700  5 -- op tracker -- , seq: 748196, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.465775 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:57.465800 7f9d4397a700 15 osd.2 173 enqueue_op 0x22b3a400 prio 63 cost 516096 latency 0.001012 osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.465851 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.465862 7f9d693e3700 10 osd.2 173 dequeue_op 0x22b3a400 prio 63 cost 516096 latency 0.001075 osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean]
2014-06-12 14:46:57.465887 7f9d693e3700  5 -- op tracker -- , seq: 748196, time: 2014-06-12 14:46:57.465883, event: reached_pg, op: osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.465896 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:57.465907 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] handle_message: 0x22b3a400
2014-06-12 14:46:57.465915 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] do_op osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:57.465942 7f9d693e3700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_'
2014-06-12 14:46:57.466078 7f9d693e3700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_' = 0
2014-06-12 14:46:57.466092 7f9d693e3700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset'
2014-06-12 14:46:57.466140 7f9d693e3700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset' = 0
2014-06-12 14:46:57.466148 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] populate_obc_watchers 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.466155 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.466161 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] get_object_context: creating obc from disk: 0x3a08d600
2014-06-12 14:46:57.466167 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] get_object_context: 0x3a08d600 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 rwstate(none n=0 w=0) oi: 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2781 client.4309.0:141798 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2781) ssc: 0x679b260 snapset: 0=[]:[]+head
2014-06-12 14:46:57.466176 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] find_object_context 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 @head oi=103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2781 client.4309.0:141798 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2781)
2014-06-12 14:46:57.466192 7f9d693e3700  5 -- op tracker -- , seq: 748196, time: 2014-06-12 14:46:57.466189, event: started, op: osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.466199 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] execute_ctx 0x25b1e000
2014-06-12 14:46:57.466209 7f9d693e3700  5 -- op tracker -- , seq: 748196, time: 2014-06-12 14:46:57.466207, event: started, op: osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.466214 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] do_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] ov 173'2781 av 173'2782 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:57.466223 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] do_osd_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096]
2014-06-12 14:46:57.466229 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:57.466237 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] do_osd_op  write 3461120~516096
2014-06-12 14:46:57.466247 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:57.466253 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 snapset=0x679b298  snapc=0=[]
2014-06-12 14:46:57.466260 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:57.466266 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] finish_ctx 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 0x25b1e000 op modify  
2014-06-12 14:46:57.466273 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean]  set mtime to 2014-06-12 08:45:59.844023
2014-06-12 14:46:57.466285 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean]  final snapset 0=[]:[]+head in 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.466297 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean]  zeroing write result code 0
2014-06-12 14:46:57.466303 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] new_repop rep_tid 236370 on osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.466313 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] issue_repop rep_tid 236370 o 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.466328 7f9d693e3700  5 -- op tracker -- , seq: 748196, time: 2014-06-12 14:46:57.466326, event: waiting for subops from 27, op: osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.466344 7f9d693e3700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:57.466360 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2781 (0'0,173'2781] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] append_log log((0'0,173'2781], crt=0'0) [173'2782 (173'2781) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141799 2014-06-12 08:45:59.844023]
2014-06-12 14:46:57.466383 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2781 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] add_log_entry 173'2782 (173'2781) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141799 2014-06-12 08:45:59.844023
2014-06-12 14:46:57.466403 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2781 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] append_log  adding 1 keys
2014-06-12 14:46:57.466440 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2782, trimmed: 
2014-06-12 14:46:57.466464 7f9d693e3700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.466469 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.466471 7f9d693e3700  5 queue_op 0x68b0720 seq 2497 osr(3.312 0x276d540) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:57.466477 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2781 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] eval_repop repgather(0x51dd740 173'2782 rep_tid=236370 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.466493 7f9d693e3700 10 osd.2 173 dequeue_op 0x22b3a400 finish
2014-06-12 14:46:57.466497 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.466499 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.466504 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (1 active)
2014-06-12 14:46:57.466512 7f9d7c165700  5 _do_op 0x68b0720 seq 2497 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:57.466516 7f9d7c165700 10 _do_transaction on 0x51dccc0
2014-06-12 14:46:57.466521 7f9d7c165700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.466595 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.466616 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.466649 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.466668 7f9d7c165700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.466673 7f9d7c165700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:57.466677 7f9d7c165700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.466824 7f9d7c165700 15 _write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 3461120~516096
2014-06-12 14:46:57.466918 7f9d7c165700 10 lookup_strip_header done  cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.467039 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:57.468327 7f9d7c165700 10 _generic_write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 3461120~516096 = 0
2014-06-12 14:46:57.468364 7f9d7c165700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.468372 7f9d7c165700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.468376 7f9d7c165700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.468381 7f9d7c165700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.468517 7f9d7c165700 20 set_header setting 17155 cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 parent seq 0
2014-06-12 14:46:57.468560 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.468567 7f9d7c165700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:57.468573 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:57.470560 7f9d7c165700 10 _do_op 0x68b0720 seq 2497 r = 0, finisher 0x80bf860 0x9af2480
2014-06-12 14:46:57.470571 7f9d7c165700 10 _finish_op 0x68b0720 seq 2497 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.470576 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:57.470581 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.470577 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2781 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] op_commit: 236370
2014-06-12 14:46:57.470621 7f9d7715b700  5 -- op tracker -- , seq: 748196, time: 2014-06-12 14:46:57.470605, event: op_commit, op: osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.470664 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2781 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] op_applied: 236370
2014-06-12 14:46:57.470691 7f9d7795c700  5 -- op tracker -- , seq: 748196, time: 2014-06-12 14:46:57.470685, event: op_applied, op: osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.470699 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2781 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] op_applied on primary on version 173'2782
2014-06-12 14:46:57.472230 7f9d58c5a700  5 -- op tracker -- , seq: 748197, time: 2014-06-12 14:46:57.472181, event: header_read, op: osd_sub_op_reply(client.4309.0:141799 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.472243 7f9d58c5a700  5 -- op tracker -- , seq: 748197, time: 2014-06-12 14:46:57.472182, event: throttled, op: osd_sub_op_reply(client.4309.0:141799 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.472260 7f9d58c5a700  5 -- op tracker -- , seq: 748197, time: 2014-06-12 14:46:57.472193, event: all_read, op: osd_sub_op_reply(client.4309.0:141799 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.472267 7f9d58c5a700  5 -- op tracker -- , seq: 748197, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141799 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.472272 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141799 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:57.472278 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:57.472285 7f9d58c5a700 15 osd.2 173 enqueue_op 0x2d17b700 prio 196 cost 0 latency 0.000104 osd_sub_op_reply(client.4309.0:141799 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:57.472349 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.472359 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2d17b700 prio 196 cost 0 latency 0.000178 osd_sub_op_reply(client.4309.0:141799 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2781 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean]
2014-06-12 14:46:57.472377 7f9d6abe6700  5 -- op tracker -- , seq: 748197, time: 2014-06-12 14:46:57.472374, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141799 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.472384 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2781 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] handle_message: 0x2d17b700
2014-06-12 14:46:57.472395 7f9d6abe6700  5 -- op tracker -- , seq: 748197, time: 2014-06-12 14:46:57.472392, event: started, op: osd_sub_op_reply(client.4309.0:141799 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.472399 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2781 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] sub_op_modify_reply: tid 236370 op  ack_type 4 from 27
2014-06-12 14:46:57.472411 7f9d6abe6700  5 -- op tracker -- , seq: 748196, time: 2014-06-12 14:46:57.472407, event: sub_op_commit_rec, op: osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.472416 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2781 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] repop_all_applied: repop tid 236370 all applied 
2014-06-12 14:46:57.472423 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2781 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] eval_repop repgather(0x51dd740 173'2782 rep_tid=236370 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.472433 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2781 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean]  sending ack on repgather(0x51dd740 173'2782 rep_tid=236370 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x28677b80
2014-06-12 14:46:57.472454 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2781 crt=0'0 lcod 173'2780 mlcod 173'2780 active+clean] repop_all_committed: repop tid 236370 all committed 
2014-06-12 14:46:57.472461 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2780 active+clean] eval_repop repgather(0x51dd740 173'2782 rep_tid=236370 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.472479 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2780 active+clean] log_op_stats osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.007665 lat 0.007692
2014-06-12 14:46:57.472491 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2780 active+clean] publish_stats_to_osd 173:3471
2014-06-12 14:46:57.472501 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2780 active+clean]  sending commit on repgather(0x51dd740 173'2782 rep_tid=236370 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x30cccc80
2014-06-12 14:46:57.472529 7f9d6abe6700  5 -- op tracker -- , seq: 748196, time: 2014-06-12 14:46:57.472526, event: commit_sent, op: osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.472535 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2780 active+clean]  requeue_ops 
2014-06-12 14:46:57.472551 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean]  removing repgather(0x51dd740 173'2782 rep_tid=236370 committed?=1 applied?=1 op=osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.472559 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean]    q front is repgather(0x51dd740 173'2782 rep_tid=236370 committed?=1 applied?=1 op=osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.472568 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] remove_repop repgather(0x51dd740 173'2782 rep_tid=236370 committed?=1 applied?=1 op=osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.472592 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean]  requeue_ops 
2014-06-12 14:46:57.472617 7f9d6abe6700  5 -- op tracker -- , seq: 748196, time: 2014-06-12 14:46:57.472614, event: done, op: osd_op(client.4309.0:141799 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3461120~516096] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.472631 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2d17b700 finish
2014-06-12 14:46:57.472635 7f9d6abe6700  5 -- op tracker -- , seq: 748197, time: 2014-06-12 14:46:57.472633, event: done, op: osd_sub_op_reply(client.4309.0:141799 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.472641 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.472643 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.474661 7f9d4397a700  5 -- op tracker -- , seq: 748198, time: 2014-06-12 14:46:57.474333, event: header_read, op: osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.474671 7f9d4397a700  5 -- op tracker -- , seq: 748198, time: 2014-06-12 14:46:57.474336, event: throttled, op: osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.474693 7f9d4397a700  5 -- op tracker -- , seq: 748198, time: 2014-06-12 14:46:57.474636, event: all_read, op: osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.474699 7f9d4397a700  5 -- op tracker -- , seq: 748198, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.474704 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:46:57.474725 7f9d4397a700 15 osd.2 173 enqueue_op 0x22b3b000 prio 63 cost 217088 latency 0.000392 osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.474744 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.474751 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x22b3b000 prio 63 cost 217088 latency 0.000418 osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean]
2014-06-12 14:46:57.474770 7f9d6a3e5700  5 -- op tracker -- , seq: 748198, time: 2014-06-12 14:46:57.474767, event: reached_pg, op: osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.474776 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:46:57.474786 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] handle_message: 0x22b3b000
2014-06-12 14:46:57.474792 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] do_op osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:46:57.474807 7f9d6a3e5700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_'
2014-06-12 14:46:57.474929 7f9d6a3e5700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 '_' = 0
2014-06-12 14:46:57.474941 7f9d6a3e5700 15 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset'
2014-06-12 14:46:57.474990 7f9d6a3e5700 10 getattr 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 'snapset' = 0
2014-06-12 14:46:57.474997 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] populate_obc_watchers 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.475005 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.475011 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] get_object_context: creating obc from disk: 0x563f600
2014-06-12 14:46:57.475016 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] get_object_context: 0x563f600 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 rwstate(none n=0 w=0) oi: 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2782 client.4309.0:141799 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2782) ssc: 0x5bb8620 snapset: 0=[]:[]+head
2014-06-12 14:46:57.475026 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] find_object_context 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 @head oi=103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3(173'2782 client.4309.0:141799 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2782)
2014-06-12 14:46:57.475042 7f9d6a3e5700  5 -- op tracker -- , seq: 748198, time: 2014-06-12 14:46:57.475039, event: started, op: osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.475048 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] execute_ctx 0x55a4a00
2014-06-12 14:46:57.475058 7f9d6a3e5700  5 -- op tracker -- , seq: 748198, time: 2014-06-12 14:46:57.475055, event: started, op: osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.475062 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] do_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] ov 173'2782 av 173'2783 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:46:57.475070 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] do_osd_op 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088]
2014-06-12 14:46:57.475076 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:46:57.475084 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] do_osd_op  write 3977216~217088
2014-06-12 14:46:57.475094 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:46:57.475100 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 snapset=0x5bb8658  snapc=0=[]
2014-06-12 14:46:57.475107 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] make_writeable 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:46:57.475113 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] finish_ctx 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 0x55a4a00 op modify  
2014-06-12 14:46:57.475120 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean]  set mtime to 2014-06-12 08:45:59.854154
2014-06-12 14:46:57.475132 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean]  final snapset 0=[]:[]+head in 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.475144 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean]  zeroing write result code 0
2014-06-12 14:46:57.475150 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] new_repop rep_tid 236371 on osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:46:57.475161 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] issue_repop rep_tid 236371 o 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.475175 7f9d6a3e5700  5 -- op tracker -- , seq: 748198, time: 2014-06-12 14:46:57.475172, event: waiting for subops from 27, op: osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.475187 7f9d6a3e5700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:57.475201 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2782 (0'0,173'2782] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] append_log log((0'0,173'2782], crt=0'0) [173'2783 (173'2782) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141800 2014-06-12 08:45:59.854154]
2014-06-12 14:46:57.475215 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2782 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] add_log_entry 173'2783 (173'2782) modify   103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 by client.4309.0:141800 2014-06-12 08:45:59.854154
2014-06-12 14:46:57.475227 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2782 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] append_log  adding 1 keys
2014-06-12 14:46:57.475246 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2783, trimmed: 
2014-06-12 14:46:57.475258 7f9d6a3e5700  5 queue_transactions existing osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.475260 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.475261 7f9d6a3e5700  5 queue_op 0x830dbc0 seq 2498 osr(3.312 0x276d540) 219381 bytes   (queue has 1 ops and 219381 bytes)
2014-06-12 14:46:57.475266 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2782 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] eval_repop repgather(0x5312cc0 173'2783 rep_tid=236371 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.475272 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5389d40 (1 active)
2014-06-12 14:46:57.475278 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x22b3b000 finish
2014-06-12 14:46:57.475280 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.475280 7f9d8096e700  5 _do_op 0x830dbc0 seq 2498 osr(3.312 0x276d540)/0x276d540 start
2014-06-12 14:46:57.475282 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.475284 7f9d8096e700 10 _do_transaction on 0x5360e40
2014-06-12 14:46:57.475290 7f9d8096e700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.475390 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.475411 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.475459 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.475477 7f9d8096e700 15 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.475485 7f9d8096e700 10 _omap_rmkeys meta/1fe228ba/pglog_3.312/0//-1 r = 0
2014-06-12 14:46:57.475491 7f9d8096e700 15 _omap_setkeys meta/1fe228ba/pglog_3.312/0//-1
2014-06-12 14:46:57.475504 7f9d8096e700 15 _write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 3977216~217088
2014-06-12 14:46:57.475633 7f9d8096e700 10 lookup_strip_header done  cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.475847 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:57.476527 7f9d8096e700 10 _generic_write 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 3977216~217088 = 0
2014-06-12 14:46:57.476557 7f9d8096e700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.476570 7f9d8096e700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.476578 7f9d8096e700 15 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3
2014-06-12 14:46:57.476587 7f9d8096e700 10 _setattrs 3.312_head/103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 = 0
2014-06-12 14:46:57.476725 7f9d8096e700 20 set_header setting 17155 cid 3.312_head oid 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 parent seq 0
2014-06-12 14:46:57.476776 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.476792 7f9d8096e700 20 set_header setting 2044 cid meta oid 1fe228ba/pglog_3.312/0//-1 parent seq 0
2014-06-12 14:46:57.476805 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:57.477900 7f9d8096e700 10 _do_op 0x830dbc0 seq 2498 r = 0, finisher 0x6f599a0 0x31aece80
2014-06-12 14:46:57.477914 7f9d8096e700 10 _finish_op 0x830dbc0 seq 2498 osr(3.312 0x276d540)/0x276d540
2014-06-12 14:46:57.477921 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5389d40 (0 active)
2014-06-12 14:46:57.477924 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.477943 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2782 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] op_commit: 236371
2014-06-12 14:46:57.477974 7f9d7715b700  5 -- op tracker -- , seq: 748198, time: 2014-06-12 14:46:57.477966, event: op_commit, op: osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.477989 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2782 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] op_applied: 236371
2014-06-12 14:46:57.478011 7f9d7795c700  5 -- op tracker -- , seq: 748198, time: 2014-06-12 14:46:57.478007, event: op_applied, op: osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.478018 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2782 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] op_applied on primary on version 173'2783
2014-06-12 14:46:57.478242 7f9d58c5a700  5 -- op tracker -- , seq: 748199, time: 2014-06-12 14:46:57.478197, event: header_read, op: osd_sub_op_reply(client.4309.0:141800 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.478252 7f9d58c5a700  5 -- op tracker -- , seq: 748199, time: 2014-06-12 14:46:57.478198, event: throttled, op: osd_sub_op_reply(client.4309.0:141800 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.478259 7f9d58c5a700  5 -- op tracker -- , seq: 748199, time: 2014-06-12 14:46:57.478210, event: all_read, op: osd_sub_op_reply(client.4309.0:141800 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.478264 7f9d58c5a700  5 -- op tracker -- , seq: 748199, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141800 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.478270 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141800 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:57.478276 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:57.478285 7f9d58c5a700 15 osd.2 173 enqueue_op 0x2d17b400 prio 196 cost 0 latency 0.000088 osd_sub_op_reply(client.4309.0:141800 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:57.478310 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.478319 7f9d66bde700 10 osd.2 173 dequeue_op 0x2d17b400 prio 196 cost 0 latency 0.000122 osd_sub_op_reply(client.4309.0:141800 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0) v2 pg pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2782 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean]
2014-06-12 14:46:57.478347 7f9d66bde700  5 -- op tracker -- , seq: 748199, time: 2014-06-12 14:46:57.478343, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141800 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.478355 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2782 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] handle_message: 0x2d17b400
2014-06-12 14:46:57.478373 7f9d66bde700  5 -- op tracker -- , seq: 748199, time: 2014-06-12 14:46:57.478369, event: started, op: osd_sub_op_reply(client.4309.0:141800 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.478379 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2782 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] sub_op_modify_reply: tid 236371 op  ack_type 4 from 27
2014-06-12 14:46:57.478400 7f9d66bde700  5 -- op tracker -- , seq: 748198, time: 2014-06-12 14:46:57.478393, event: sub_op_commit_rec, op: osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.478409 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2782 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] repop_all_applied: repop tid 236371 all applied 
2014-06-12 14:46:57.478422 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2782 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] eval_repop repgather(0x5312cc0 173'2783 rep_tid=236371 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.478442 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2782 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean]  sending ack on repgather(0x5312cc0 173'2783 rep_tid=236371 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x564c280
2014-06-12 14:46:57.478476 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 luod=173'2782 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] repop_all_committed: repop tid 236371 all committed 
2014-06-12 14:46:57.478485 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2781 active+clean] eval_repop repgather(0x5312cc0 173'2783 rep_tid=236371 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:57.478511 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2781 active+clean] log_op_stats osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4 inb 217367 outb 0 rlat 0.004142 lat 0.004177
2014-06-12 14:46:57.478538 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2781 active+clean] publish_stats_to_osd 173:3472
2014-06-12 14:46:57.478554 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2781 active+clean]  sending commit on repgather(0x5312cc0 173'2783 rep_tid=236371 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4) 0x1978d900
2014-06-12 14:46:57.478587 7f9d66bde700  5 -- op tracker -- , seq: 748198, time: 2014-06-12 14:46:57.478581, event: commit_sent, op: osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.478602 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2781 active+clean]  requeue_ops 
2014-06-12 14:46:57.478616 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2782 active+clean]  removing repgather(0x5312cc0 173'2783 rep_tid=236371 committed?=1 applied?=1 op=osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.478639 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2782 active+clean]    q front is repgather(0x5312cc0 173'2783 rep_tid=236371 committed?=1 applied?=1 op=osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.478654 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2782 active+clean] remove_repop repgather(0x5312cc0 173'2783 rep_tid=236371 committed?=1 applied?=1 op=osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:57.478670 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2782 active+clean]  requeue_ops 
2014-06-12 14:46:57.478695 7f9d66bde700  5 -- op tracker -- , seq: 748198, time: 2014-06-12 14:46:57.478690, event: done, op: osd_op(client.4309.0:141800 rbd_data.10ac6b8b4567.0000000000000f82 [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.103c8312 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:57.478777 7f9d66bde700 10 osd.2 173 dequeue_op 0x2d17b400 finish
2014-06-12 14:46:57.478787 7f9d66bde700  5 -- op tracker -- , seq: 748199, time: 2014-06-12 14:46:57.478783, event: done, op: osd_sub_op_reply(client.4309.0:141800 3.312 103c8312/rbd_data.10ac6b8b4567.0000000000000f82/head//3 [] ondisk, result = 0)
2014-06-12 14:46:57.478798 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.478803 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.480193 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:57.480203 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:46:57.480204 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:46:57.480207 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:46:57.480209 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:46:57.480260 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 10.74 >= max 0.5 = no, load too high
2014-06-12 14:46:57.480268 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:57.480271 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:57.480273 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92665 < max (604800 seconds)
2014-06-12 14:46:57.480283 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:57.480285 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:57.480286 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:57.480287 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 25; oldest is 24.318943 seconds old
2014-06-12 14:46:57.522564 7f9d56432700  5 -- op tracker -- , seq: 748200, time: 2014-06-12 14:46:57.521793, event: header_read, op: osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.522590 7f9d56432700  5 -- op tracker -- , seq: 748200, time: 2014-06-12 14:46:57.521796, event: throttled, op: osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.522595 7f9d56432700  5 -- op tracker -- , seq: 748200, time: 2014-06-12 14:46:57.522534, event: all_read, op: osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.522600 7f9d56432700  5 -- op tracker -- , seq: 748200, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.522604 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:57.522609 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:57.522614 7f9d56432700 15 osd.2 173 enqueue_op 0x59dd000 prio 127 cost 516903 latency 0.000821 osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.522643 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.522653 7f9d64bda700 10 osd.2 173 dequeue_op 0x59dd000 prio 127 cost 516903 latency 0.000859 osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2644 (0'0,173'2644] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active]
2014-06-12 14:46:57.522679 7f9d64bda700  5 -- op tracker -- , seq: 748200, time: 2014-06-12 14:46:57.522675, event: reached_pg, op: osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.522689 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2644 (0'0,173'2644] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] handle_message: 0x59dd000
2014-06-12 14:46:57.522701 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2644 (0'0,173'2644] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] sub_op_modify trans af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 v 173'2645 (transaction) 174
2014-06-12 14:46:57.522718 7f9d64bda700  5 -- op tracker -- , seq: 748200, time: 2014-06-12 14:46:57.522714, event: started, op: osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.522751 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2644 (0'0,173'2644] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] append_log log((0'0,173'2644], crt=0'0) [173'2645 (173'2644) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147304 2014-06-12 08:45:59.898666]
2014-06-12 14:46:57.522774 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2645 (0'0,173'2645] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] add_log_entry 173'2645 (173'2644) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147304 2014-06-12 08:45:59.898666
2014-06-12 14:46:57.522803 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2645 (0'0,173'2645] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] append_log  adding 1 keys
2014-06-12 14:46:57.522838 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2645, trimmed: 
2014-06-12 14:46:57.522863 7f9d64bda700  5 -- op tracker -- , seq: 748200, time: 2014-06-12 14:46:57.522858, event: started, op: osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.522873 7f9d64bda700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:57.522877 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.522879 7f9d64bda700  5 queue_op 0xd504240 seq 2465 osr(3.1c7 0x276d170) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:57.522887 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2645 (0'0,173'2645] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] do_sub_op osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.522903 7f9d64bda700 10 osd.2 173 dequeue_op 0x59dd000 finish
2014-06-12 14:46:57.522907 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.522910 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.522908 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:46:57.522916 7f9d7c966700  5 _do_op 0xd504240 seq 2465 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:46:57.522919 7f9d7c966700 10 _do_transaction on 0x31d09430
2014-06-12 14:46:57.522924 7f9d7c966700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:57.523012 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:57.523031 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.523061 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.523079 7f9d7c966700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:57.523084 7f9d7c966700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:46:57.523087 7f9d7c966700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:57.523190 7f9d7c966700 15 _write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 385024~516096
2014-06-12 14:46:57.523281 7f9d7c966700 10 lookup_strip_header done  cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:46:57.523401 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:57.524604 7f9d7c966700 10 _generic_write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 385024~516096 = 0
2014-06-12 14:46:57.524635 7f9d7c966700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:46:57.524642 7f9d7c966700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:46:57.524646 7f9d7c966700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:46:57.524650 7f9d7c966700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:46:57.524784 7f9d7c966700 20 set_header setting 17479 cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 parent seq 0
2014-06-12 14:46:57.524829 7f9d7c966700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:46:57.524842 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.524847 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:57.527752 7f9d7c966700 10 _do_op 0xd504240 seq 2465 r = 0, finisher 0x98f5d00 0
2014-06-12 14:46:57.527775 7f9d7c966700 10 _finish_op 0xd504240 seq 2465 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:57.527781 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:46:57.527788 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.527795 7f9d7715b700  5 -- op tracker -- , seq: 748200, time: 2014-06-12 14:46:57.527783, event: commit_sent, op: osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.527804 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2645 (0'0,173'2645] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2643 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:57.527835 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:57.527873 7f9d7795c700  5 -- op tracker -- , seq: 748200, time: 2014-06-12 14:46:57.527866, event: sub_op_applied, op: osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.527880 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2645 (0'0,173'2645] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2644 active] sub_op_modify_applied on 0x31d09340 op osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.527897 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2645 (0'0,173'2645] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2644 active] op_applied on primary on version 173'2645
2014-06-12 14:46:57.527907 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2645 (0'0,173'2645] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2644 active] op_applied on replica on version 173'2645
2014-06-12 14:46:57.527931 7f9d7795c700  5 -- op tracker -- , seq: 748200, time: 2014-06-12 14:46:57.527927, event: done, op: osd_sub_op(client.4316.0:147304 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2645 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.647844 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:57.647882 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:57.721543 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:57.721546 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:57.721547 7f9d74bfa700 20 osd.2 173 _dispatch 0x2705e8c0 pg_stats_ack(1 pgs tid 4857) v1
2014-06-12 14:46:57.721549 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:46:57.721554 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:57.721555 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:57.876669 7f9d56432700  5 -- op tracker -- , seq: 748201, time: 2014-06-12 14:46:57.875943, event: header_read, op: osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.876687 7f9d56432700  5 -- op tracker -- , seq: 748201, time: 2014-06-12 14:46:57.875946, event: throttled, op: osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.876692 7f9d56432700  5 -- op tracker -- , seq: 748201, time: 2014-06-12 14:46:57.876632, event: all_read, op: osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.876698 7f9d56432700  5 -- op tracker -- , seq: 748201, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.876701 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:57.876706 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:57.876712 7f9d56432700 15 osd.2 173 enqueue_op 0x59dd700 prio 127 cost 516903 latency 0.000768 osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.876775 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.876789 7f9d69be4700 10 osd.2 173 dequeue_op 0x59dd700 prio 127 cost 516903 latency 0.000846 osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2645 (0'0,173'2645] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2644 active]
2014-06-12 14:46:57.876826 7f9d69be4700  5 -- op tracker -- , seq: 748201, time: 2014-06-12 14:46:57.876814, event: reached_pg, op: osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.876839 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2645 (0'0,173'2645] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2644 active] handle_message: 0x59dd700
2014-06-12 14:46:57.876848 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2645 (0'0,173'2645] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2644 active] sub_op_modify trans af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 v 173'2646 (transaction) 174
2014-06-12 14:46:57.876858 7f9d69be4700  5 -- op tracker -- , seq: 748201, time: 2014-06-12 14:46:57.876855, event: started, op: osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.876873 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2645 (0'0,173'2645] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2644 active] append_log log((0'0,173'2645], crt=0'0) [173'2646 (173'2645) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147305 2014-06-12 08:46:00.251835]
2014-06-12 14:46:57.876891 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2646 (0'0,173'2646] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2644 active] add_log_entry 173'2646 (173'2645) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147305 2014-06-12 08:46:00.251835
2014-06-12 14:46:57.876907 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2646 (0'0,173'2646] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2644 active] append_log  adding 1 keys
2014-06-12 14:46:57.876932 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2646, trimmed: 
2014-06-12 14:46:57.876950 7f9d69be4700  5 -- op tracker -- , seq: 748201, time: 2014-06-12 14:46:57.876947, event: started, op: osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.876959 7f9d69be4700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:57.876962 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.876963 7f9d69be4700  5 queue_op 0x70278c0 seq 2466 osr(3.1c7 0x276d170) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:57.876968 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2646 (0'0,173'2646] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2644 active] do_sub_op osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.876977 7f9d69be4700 10 osd.2 173 dequeue_op 0x59dd700 finish
2014-06-12 14:46:57.876987 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.876989 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.877001 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:46:57.877009 7f9d81970700  5 _do_op 0x70278c0 seq 2466 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:46:57.877021 7f9d81970700 10 _do_transaction on 0x5b7bb30
2014-06-12 14:46:57.877027 7f9d81970700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:57.877193 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:57.877240 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.877306 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.877339 7f9d81970700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:57.877350 7f9d81970700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:46:57.877375 7f9d81970700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:57.877410 7f9d81970700 15 _write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 901120~516096
2014-06-12 14:46:57.877632 7f9d81970700 10 lookup_strip_header done  cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:46:57.877922 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:57.880214 7f9d81970700 10 _generic_write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 901120~516096 = 0
2014-06-12 14:46:57.880295 7f9d81970700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:46:57.880344 7f9d81970700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:46:57.880389 7f9d81970700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:46:57.880411 7f9d81970700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:46:57.880707 7f9d81970700 20 set_header setting 17479 cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 parent seq 0
2014-06-12 14:46:57.880831 7f9d81970700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:46:57.880865 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.880887 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:57.884426 7f9d81970700 10 _do_op 0x70278c0 seq 2466 r = 0, finisher 0x29c890e0 0
2014-06-12 14:46:57.884464 7f9d81970700 10 _finish_op 0x70278c0 seq 2466 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:57.884478 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:46:57.884489 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.884502 7f9d7715b700  5 -- op tracker -- , seq: 748201, time: 2014-06-12 14:46:57.884481, event: commit_sent, op: osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.884523 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2646 (0'0,173'2646] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2644 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:57.884570 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:57.884675 7f9d7795c700  5 -- op tracker -- , seq: 748201, time: 2014-06-12 14:46:57.884661, event: sub_op_applied, op: osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.884699 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2646 (0'0,173'2646] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active] sub_op_modify_applied on 0x5b7ba40 op osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.884727 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2646 (0'0,173'2646] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active] op_applied on primary on version 173'2646
2014-06-12 14:46:57.884745 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2646 (0'0,173'2646] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active] op_applied on replica on version 173'2646
2014-06-12 14:46:57.884764 7f9d7795c700  5 -- op tracker -- , seq: 748201, time: 2014-06-12 14:46:57.884761, event: done, op: osd_sub_op(client.4316.0:147305 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2646 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.902410 7f9d56432700  5 -- op tracker -- , seq: 748202, time: 2014-06-12 14:46:57.901557, event: header_read, op: osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.902422 7f9d56432700  5 -- op tracker -- , seq: 748202, time: 2014-06-12 14:46:57.901559, event: throttled, op: osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.902428 7f9d56432700  5 -- op tracker -- , seq: 748202, time: 2014-06-12 14:46:57.902364, event: all_read, op: osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.902442 7f9d56432700  5 -- op tracker -- , seq: 748202, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.902445 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:57.902450 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:57.902456 7f9d56432700 15 osd.2 173 enqueue_op 0x59ddd00 prio 127 cost 516903 latency 0.000899 osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.902480 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.902489 7f9d653db700 10 osd.2 173 dequeue_op 0x59ddd00 prio 127 cost 516903 latency 0.000932 osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'3004 (79'3,173'3004] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active]
2014-06-12 14:46:57.902512 7f9d653db700  5 -- op tracker -- , seq: 748202, time: 2014-06-12 14:46:57.902508, event: reached_pg, op: osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.902534 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3004 (79'3,173'3004] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active] handle_message: 0x59ddd00
2014-06-12 14:46:57.902549 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3004 (79'3,173'3004] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active] sub_op_modify trans d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 v 173'3005 (transaction) 174
2014-06-12 14:46:57.902566 7f9d653db700  5 -- op tracker -- , seq: 748202, time: 2014-06-12 14:46:57.902562, event: started, op: osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.902579 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3004 (79'3,173'3004] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active] append_log log((79'3,173'3004], crt=0'0) [173'3005 (173'3004) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145537 2014-06-12 08:46:00.278228]
2014-06-12 14:46:57.902603 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3005 (79'3,173'3005] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active] add_log_entry 173'3005 (173'3004) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145537 2014-06-12 08:46:00.278228
2014-06-12 14:46:57.902626 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3005 (79'3,173'3005] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active] append_log  adding 1 keys
2014-06-12 14:46:57.902642 7f9d653db700 10 trim log((79'3,173'3005], crt=0'0) to 79'4
2014-06-12 14:46:57.902668 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3005, trimmed: 79'4
2014-06-12 14:46:57.902699 7f9d653db700  5 -- op tracker -- , seq: 748202, time: 2014-06-12 14:46:57.902694, event: started, op: osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.902710 7f9d653db700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:57.902714 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.902715 7f9d653db700  5 queue_op 0x132b1f80 seq 2807 osr(3.458 0x276d960) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:46:57.902722 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'3005 (79'4,173'3005] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active] do_sub_op osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.902737 7f9d653db700 10 osd.2 173 dequeue_op 0x59ddd00 finish
2014-06-12 14:46:57.902741 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.902743 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.902748 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:46:57.902756 7f9d7a962700  5 _do_op 0x132b1f80 seq 2807 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:46:57.902760 7f9d7a962700 10 _do_transaction on 0x7f87430
2014-06-12 14:46:57.902765 7f9d7a962700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:57.902855 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:57.902883 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.902944 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.902963 7f9d7a962700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:57.902970 7f9d7a962700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:46:57.902974 7f9d7a962700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:46:57.903056 7f9d7a962700 15 _write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 3252224~516096
2014-06-12 14:46:57.903177 7f9d7a962700 10 lookup_strip_header done  cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:57.903305 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:57.904636 7f9d7a962700 10 _generic_write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 3252224~516096 = 0
2014-06-12 14:46:57.904672 7f9d7a962700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:57.904683 7f9d7a962700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:46:57.904688 7f9d7a962700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:46:57.904693 7f9d7a962700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:46:57.904866 7f9d7a962700 20 set_header setting 17709 cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 parent seq 0
2014-06-12 14:46:57.904924 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.904935 7f9d7a962700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:46:57.904946 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:57.906450 7f9d7a962700 10 _do_op 0x132b1f80 seq 2807 r = 0, finisher 0x2896700 0
2014-06-12 14:46:57.906463 7f9d7a962700 10 _finish_op 0x132b1f80 seq 2807 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:46:57.906468 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (0 active)
2014-06-12 14:46:57.906483 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.906495 7f9d7795c700  5 -- op tracker -- , seq: 748202, time: 2014-06-12 14:46:57.906487, event: sub_op_applied, op: osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.906506 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3005 (79'4,173'3005] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active] sub_op_modify_applied on 0x7f87340 op osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.906533 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:57.906548 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3005 (79'4,173'3005] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active] op_applied on primary on version 173'3005
2014-06-12 14:46:57.906560 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3005 (79'4,173'3005] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active] op_applied on replica on version 173'3005
2014-06-12 14:46:57.906620 7f9d7715b700  5 -- op tracker -- , seq: 748202, time: 2014-06-12 14:46:57.906614, event: commit_sent, op: osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.906632 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3005 (79'4,173'3005] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3003 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:57.906648 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:57.906670 7f9d7715b700  5 -- op tracker -- , seq: 748202, time: 2014-06-12 14:46:57.906667, event: done, op: osd_sub_op(client.4299.0:145537 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3005 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.966288 7f9d58c5a700  5 -- op tracker -- , seq: 748203, time: 2014-06-12 14:46:57.963324, event: header_read, op: osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.966315 7f9d58c5a700  5 -- op tracker -- , seq: 748203, time: 2014-06-12 14:46:57.963327, event: throttled, op: osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.966406 7f9d58c5a700  5 -- op tracker -- , seq: 748203, time: 2014-06-12 14:46:57.966201, event: all_read, op: osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.966417 7f9d58c5a700  5 -- op tracker -- , seq: 748203, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.966424 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:57.966431 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:57.966441 7f9d58c5a700 15 osd.2 173 enqueue_op 0x31d61200 prio 127 cost 1114919 latency 0.003117 osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.966477 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:57.966487 7f9d623d5700 10 osd.2 173 dequeue_op 0x31d61200 prio 127 cost 1114919 latency 0.003163 osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.34f( v 173'2926 (0'0,173'2926] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2925 active]
2014-06-12 14:46:57.966535 7f9d623d5700  5 -- op tracker -- , seq: 748203, time: 2014-06-12 14:46:57.966529, event: reached_pg, op: osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.966549 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2926 (0'0,173'2926] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2925 active] handle_message: 0x31d61200
2014-06-12 14:46:57.966565 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2926 (0'0,173'2926] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2925 active] sub_op_modify trans 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 v 173'2927 (transaction) 174
2014-06-12 14:46:57.966587 7f9d623d5700  5 -- op tracker -- , seq: 748203, time: 2014-06-12 14:46:57.966581, event: started, op: osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.966609 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2926 (0'0,173'2926] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2925 active] append_log log((0'0,173'2926], crt=0'0) [173'2927 (173'753) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144291 2014-06-12 08:46:00.293368]
2014-06-12 14:46:57.966645 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2927 (0'0,173'2927] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2925 active] add_log_entry 173'2927 (173'753) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144291 2014-06-12 08:46:00.293368
2014-06-12 14:46:57.966678 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2927 (0'0,173'2927] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2925 active] append_log  adding 1 keys
2014-06-12 14:46:57.966738 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2927, trimmed: 
2014-06-12 14:46:57.966790 7f9d623d5700  5 -- op tracker -- , seq: 748203, time: 2014-06-12 14:46:57.966775, event: started, op: osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.966805 7f9d623d5700  5 queue_transactions existing osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:57.966811 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:57.966813 7f9d623d5700  5 queue_op 0x2b4a9ce0 seq 2641 osr(3.34f 0x276d480) 1116405 bytes   (queue has 1 ops and 1116405 bytes)
2014-06-12 14:46:57.966822 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.34f( v 173'2927 (0'0,173'2927] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2925 active] do_sub_op osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.966843 7f9d623d5700 10 osd.2 173 dequeue_op 0x31d61200 finish
2014-06-12 14:46:57.966848 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:57.966852 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:57.966859 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540e240 (1 active)
2014-06-12 14:46:57.966875 7f9d82972700  5 _do_op 0x2b4a9ce0 seq 2641 osr(3.34f 0x276d480)/0x276d480 start
2014-06-12 14:46:57.966882 7f9d82972700 10 _do_transaction on 0x5e9ab70
2014-06-12 14:46:57.966892 7f9d82972700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:57.967048 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:57.967176 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:57.967222 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:57.967244 7f9d82972700 15 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:57.967250 7f9d82972700 10 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1 r = 0
2014-06-12 14:46:57.967253 7f9d82972700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:57.967266 7f9d82972700 15 _write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 0~1114112
2014-06-12 14:46:57.967376 7f9d82972700 10 lookup_strip_header done  cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:57.967570 7f9d82972700 10 file_to_extents done 
2014-06-12 14:46:57.970304 7f9d82972700 10 _generic_write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 0~1114112 = 0
2014-06-12 14:46:57.970369 7f9d82972700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:57.970383 7f9d82972700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:57.970389 7f9d82972700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:57.970398 7f9d82972700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:57.970605 7f9d82972700 20 set_header setting 16877 cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 parent seq 0
2014-06-12 14:46:57.970676 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:57.970687 7f9d82972700 20 set_header setting 2162 cid meta oid 1fe1d62b/pglog_3.34f/0//-1 parent seq 0
2014-06-12 14:46:57.970693 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:46:57.974473 7f9d82972700 10 _do_op 0x2b4a9ce0 seq 2641 r = 0, finisher 0x534c8e0 0
2014-06-12 14:46:57.974490 7f9d82972700 10 _finish_op 0x2b4a9ce0 seq 2641 osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:57.974496 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540e240 (0 active)
2014-06-12 14:46:57.974508 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:57.974527 7f9d7795c700  5 -- op tracker -- , seq: 748203, time: 2014-06-12 14:46:57.974516, event: sub_op_applied, op: osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.974535 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2927 (0'0,173'2927] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2925 active] sub_op_modify_applied on 0x5e9aa80 op osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:57.974588 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:57.974617 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2927 (0'0,173'2927] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2925 active] op_applied on primary on version 173'2927
2014-06-12 14:46:57.974633 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2927 (0'0,173'2927] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2925 active] op_applied on replica on version 173'2927
2014-06-12 14:46:57.974703 7f9d7715b700  5 -- op tracker -- , seq: 748203, time: 2014-06-12 14:46:57.974689, event: commit_sent, op: osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:57.974737 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2927 (0'0,173'2927] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2925 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:57.974758 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:57.974796 7f9d7715b700  5 -- op tracker -- , seq: 748203, time: 2014-06-12 14:46:57.974792, event: done, op: osd_sub_op(client.4306.0:144291 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2927 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.012615 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:58.012715 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:46:58.148450 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:58.148517 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:58.155273 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:46:58.157338 7f9d56432700  5 -- op tracker -- , seq: 748204, time: 2014-06-12 14:46:58.157278, event: header_read, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ack, result = 0)
2014-06-12 14:46:58.157352 7f9d56432700  5 -- op tracker -- , seq: 748204, time: 2014-06-12 14:46:58.157280, event: throttled, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ack, result = 0)
2014-06-12 14:46:58.157370 7f9d56432700  5 -- op tracker -- , seq: 748204, time: 2014-06-12 14:46:58.157298, event: all_read, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ack, result = 0)
2014-06-12 14:46:58.157384 7f9d56432700  5 -- op tracker -- , seq: 748204, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ack, result = 0)
2014-06-12 14:46:58.157388 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:46:58.157393 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:58.157400 7f9d56432700 15 osd.2 173 enqueue_op 0x59ddb00 prio 196 cost 0 latency 0.000122 osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ack, result = 0) v2
2014-06-12 14:46:58.157425 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.157431 7f9d68be2700 10 osd.2 173 dequeue_op 0x59ddb00 prio 196 cost 0 latency 0.000153 osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ack, result = 0) v2 pg pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]
2014-06-12 14:46:58.157453 7f9d56432700  5 -- op tracker -- , seq: 748205, time: 2014-06-12 14:46:58.157420, event: header_read, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.157476 7f9d56432700  5 -- op tracker -- , seq: 748205, time: 2014-06-12 14:46:58.157420, event: throttled, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.157486 7f9d56432700  5 -- op tracker -- , seq: 748205, time: 2014-06-12 14:46:58.157428, event: all_read, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.157496 7f9d56432700  5 -- op tracker -- , seq: 748205, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.157501 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:58.157506 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:58.157504 7f9d68be2700  5 -- op tracker -- , seq: 748204, time: 2014-06-12 14:46:58.157452, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ack, result = 0)
2014-06-12 14:46:58.157511 7f9d56432700 15 osd.2 173 enqueue_op 0x59dcf00 prio 196 cost 0 latency 0.000091 osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:58.157515 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] handle_message: 0x59ddb00
2014-06-12 14:46:58.157527 7f9d68be2700  5 -- op tracker -- , seq: 748204, time: 2014-06-12 14:46:58.157524, event: started, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ack, result = 0)
2014-06-12 14:46:58.157531 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:46:58.157533 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] sub_op_modify_reply: tid 236277 op  ack_type 1 from 26
2014-06-12 14:46:58.157551 7f9d68be2700  5 -- op tracker -- , seq: 747883, time: 2014-06-12 14:46:58.157544, event: sub_op_applied_rec, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:58.157557 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] repop_all_applied: repop tid 236277 all applied 
2014-06-12 14:46:58.157564 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] eval_repop repgather(0x2237200 173'2815 rep_tid=236277 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:58.157574 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]  sending ack on repgather(0x2237200 173'2815 rep_tid=236277 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) 0x29de6a00
2014-06-12 14:46:58.157594 7f9d68be2700 10 osd.2 173 dequeue_op 0x59ddb00 finish
2014-06-12 14:46:58.157601 7f9d68be2700  5 -- op tracker -- , seq: 748204, time: 2014-06-12 14:46:58.157598, event: done, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ack, result = 0)
2014-06-12 14:46:58.157600 7f9d643d9700 10 osd.2 173 dequeue_op 0x59dcf00 prio 196 cost 0 latency 0.000180 osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0) v2 pg pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean]
2014-06-12 14:46:58.157628 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:46:58.157631 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.157632 7f9d643d9700  5 -- op tracker -- , seq: 748205, time: 2014-06-12 14:46:58.157628, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.157639 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] handle_message: 0x59dcf00
2014-06-12 14:46:58.157653 7f9d643d9700  5 -- op tracker -- , seq: 748205, time: 2014-06-12 14:46:58.157650, event: started, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.157671 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] sub_op_modify_reply: tid 236277 op  ack_type 4 from 26
2014-06-12 14:46:58.157696 7f9d643d9700  5 -- op tracker -- , seq: 747883, time: 2014-06-12 14:46:58.157691, event: sub_op_commit_rec, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:58.157701 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2814 crt=0'0 lcod 173'2813 mlcod 173'2813 active+clean] repop_all_committed: repop tid 236277 all committed 
2014-06-12 14:46:58.157708 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2813 active+clean] eval_repop repgather(0x2237200 173'2815 rep_tid=236277 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:58.157718 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2813 active+clean] log_op_stats osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4 inb 49431 outb 0 rlat 24.996248 lat 24.996373
2014-06-12 14:46:58.157730 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2813 active+clean] publish_stats_to_osd 173:3140
2014-06-12 14:46:58.157739 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2813 active+clean]  sending commit on repgather(0x2237200 173'2815 rep_tid=236277 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) 0x33771400
2014-06-12 14:46:58.157761 7f9d643d9700  5 -- op tracker -- , seq: 747883, time: 2014-06-12 14:46:58.157757, event: commit_sent, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:58.157767 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2813 active+clean]  requeue_ops 
2014-06-12 14:46:58.157782 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2814 active+clean]  removing repgather(0x2237200 173'2815 rep_tid=236277 committed?=1 applied?=1 op=osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:58.157791 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2814 active+clean]    q front is repgather(0x2237200 173'2815 rep_tid=236277 committed?=1 applied?=1 op=osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:58.157799 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2814 active+clean] remove_repop repgather(0x2237200 173'2815 rep_tid=236277 committed?=1 applied?=1 op=osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:58.157807 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2814 active+clean]  requeue_ops 
2014-06-12 14:46:58.157820 7f9d643d9700  5 -- op tracker -- , seq: 747883, time: 2014-06-12 14:46:58.157817, event: done, op: osd_op(client.4293.0:144174 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~49152] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:58.157839 7f9d643d9700 10 osd.2 173 dequeue_op 0x59dcf00 finish
2014-06-12 14:46:58.157844 7f9d643d9700  5 -- op tracker -- , seq: 748205, time: 2014-06-12 14:46:58.157842, event: done, op: osd_sub_op_reply(client.4293.0:144174 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.157850 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.157852 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.166427 7f9d5703e700  5 -- op tracker -- , seq: 748206, time: 2014-06-12 14:46:58.165775, event: header_read, op: osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.166442 7f9d5703e700  5 -- op tracker -- , seq: 748206, time: 2014-06-12 14:46:58.165778, event: throttled, op: osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.166452 7f9d5703e700  5 -- op tracker -- , seq: 748206, time: 2014-06-12 14:46:58.166394, event: all_read, op: osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.166461 7f9d5703e700  5 -- op tracker -- , seq: 748206, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.166466 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.166473 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:58.166481 7f9d5703e700 15 osd.2 173 enqueue_op 0x17c2b200 prio 127 cost 414503 latency 0.000706 osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.166522 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.166532 7f9d63bd8700 10 osd.2 173 dequeue_op 0x17c2b200 prio 127 cost 414503 latency 0.000757 osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active]
2014-06-12 14:46:58.166550 7f9d63bd8700  5 -- op tracker -- , seq: 748206, time: 2014-06-12 14:46:58.166547, event: reached_pg, op: osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.166559 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active] handle_message: 0x17c2b200
2014-06-12 14:46:58.166567 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active] sub_op_modify trans b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 v 173'2844 (transaction) 174
2014-06-12 14:46:58.166578 7f9d63bd8700  5 -- op tracker -- , seq: 748206, time: 2014-06-12 14:46:58.166575, event: started, op: osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.166594 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2843 (0'0,173'2843] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active] append_log log((0'0,173'2843], crt=0'0) [173'2844 (173'896) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144053 2014-06-12 08:46:00.542964]
2014-06-12 14:46:58.166611 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2844 (0'0,173'2844] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active] add_log_entry 173'2844 (173'896) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144053 2014-06-12 08:46:00.542964
2014-06-12 14:46:58.166629 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2844 (0'0,173'2844] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active] append_log  adding 1 keys
2014-06-12 14:46:58.166649 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2844, trimmed: 
2014-06-12 14:46:58.166663 7f9d63bd8700  5 -- op tracker -- , seq: 748206, time: 2014-06-12 14:46:58.166660, event: started, op: osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.166671 7f9d63bd8700  5 queue_transactions existing osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.166674 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.166675 7f9d63bd8700  5 queue_op 0x80c4a20 seq 2601 osr(3.237 0x276d3f0) 415989 bytes   (queue has 1 ops and 415989 bytes)
2014-06-12 14:46:58.166678 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.237( v 173'2844 (0'0,173'2844] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active] do_sub_op osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.166688 7f9d63bd8700 10 osd.2 173 dequeue_op 0x17c2b200 finish
2014-06-12 14:46:58.166698 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.166700 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.166710 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540eb40 (1 active)
2014-06-12 14:46:58.166718 7f9d7895e700  5 _do_op 0x80c4a20 seq 2601 osr(3.237 0x276d3f0)/0x276d3f0 start
2014-06-12 14:46:58.166722 7f9d7895e700 10 _do_transaction on 0x27756b70
2014-06-12 14:46:58.166727 7f9d7895e700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.166826 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.166968 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.167026 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.167050 7f9d7895e700 15 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.167054 7f9d7895e700 10 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1 r = 0
2014-06-12 14:46:58.167058 7f9d7895e700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.167069 7f9d7895e700 15 _write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 0~413696
2014-06-12 14:46:58.167344 7f9d7895e700 10 lookup_strip_header done  cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.167376 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:46:58.168340 7f9d7895e700 10 _generic_write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 0~413696 = 0
2014-06-12 14:46:58.168376 7f9d7895e700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.168384 7f9d7895e700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.168388 7f9d7895e700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.168393 7f9d7895e700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.168517 7f9d7895e700 20 set_header setting 16862 cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 parent seq 0
2014-06-12 14:46:58.168559 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.168585 7f9d7895e700 20 set_header setting 2190 cid meta oid 1fe26b1a/pglog_3.237/0//-1 parent seq 0
2014-06-12 14:46:58.168591 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:46:58.170169 7f9d7895e700 10 _do_op 0x80c4a20 seq 2601 r = 0, finisher 0x4ffdac0 0
2014-06-12 14:46:58.170181 7f9d7895e700 10 _finish_op 0x80c4a20 seq 2601 osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.170186 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540eb40 (0 active)
2014-06-12 14:46:58.170188 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.170207 7f9d7795c700  5 -- op tracker -- , seq: 748206, time: 2014-06-12 14:46:58.170198, event: sub_op_applied, op: osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.170220 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2844 (0'0,173'2844] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active] sub_op_modify_applied on 0x27756a80 op osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.170246 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:58.170264 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2844 (0'0,173'2844] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active] op_applied on primary on version 173'2844
2014-06-12 14:46:58.170278 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2844 (0'0,173'2844] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active] op_applied on replica on version 173'2844
2014-06-12 14:46:58.170307 7f9d7715b700  5 -- op tracker -- , seq: 748206, time: 2014-06-12 14:46:58.170300, event: commit_sent, op: osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.170321 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2844 (0'0,173'2844] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2842 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:58.170343 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:58.170377 7f9d7715b700  5 -- op tracker -- , seq: 748206, time: 2014-06-12 14:46:58.170373, event: done, op: osd_sub_op(client.4317.0:144053 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2844 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.176690 7f9d5703e700  5 -- op tracker -- , seq: 748207, time: 2014-06-12 14:46:58.175867, event: header_read, op: osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.176710 7f9d5703e700  5 -- op tracker -- , seq: 748207, time: 2014-06-12 14:46:58.175869, event: throttled, op: osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.176718 7f9d5703e700  5 -- op tracker -- , seq: 748207, time: 2014-06-12 14:46:58.176660, event: all_read, op: osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.176731 7f9d5703e700  5 -- op tracker -- , seq: 748207, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.176736 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.176744 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:58.176751 7f9d5703e700 15 osd.2 173 enqueue_op 0x598f700 prio 127 cost 516903 latency 0.000883 osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.176795 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.176810 7f9d633d7700 10 osd.2 173 dequeue_op 0x598f700 prio 127 cost 516903 latency 0.000943 osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.237( v 173'2844 (0'0,173'2844] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2843 active]
2014-06-12 14:46:58.176833 7f9d633d7700  5 -- op tracker -- , seq: 748207, time: 2014-06-12 14:46:58.176831, event: reached_pg, op: osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.176847 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2844 (0'0,173'2844] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2843 active] handle_message: 0x598f700
2014-06-12 14:46:58.176855 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2844 (0'0,173'2844] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2843 active] sub_op_modify trans b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 v 173'2845 (transaction) 174
2014-06-12 14:46:58.176865 7f9d633d7700  5 -- op tracker -- , seq: 748207, time: 2014-06-12 14:46:58.176862, event: started, op: osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.176876 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2844 (0'0,173'2844] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2843 active] append_log log((0'0,173'2844], crt=0'0) [173'2845 (173'2844) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144054 2014-06-12 08:46:00.552607]
2014-06-12 14:46:58.176901 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2845 (0'0,173'2845] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2843 active] add_log_entry 173'2845 (173'2844) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144054 2014-06-12 08:46:00.552607
2014-06-12 14:46:58.176922 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2845 (0'0,173'2845] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2843 active] append_log  adding 1 keys
2014-06-12 14:46:58.176944 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2845, trimmed: 
2014-06-12 14:46:58.176976 7f9d633d7700  5 -- op tracker -- , seq: 748207, time: 2014-06-12 14:46:58.176973, event: started, op: osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.176983 7f9d633d7700  5 queue_transactions existing osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.176985 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.176986 7f9d633d7700  5 queue_op 0x6fbff20 seq 2602 osr(3.237 0x276d3f0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:58.176990 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.237( v 173'2845 (0'0,173'2845] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2843 active] do_sub_op osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.176999 7f9d633d7700 10 osd.2 173 dequeue_op 0x598f700 finish
2014-06-12 14:46:58.177001 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.177011 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.177021 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540eb40 (1 active)
2014-06-12 14:46:58.177036 7f9d83173700  5 _do_op 0x6fbff20 seq 2602 osr(3.237 0x276d3f0)/0x276d3f0 start
2014-06-12 14:46:58.177040 7f9d83173700 10 _do_transaction on 0x2a29b270
2014-06-12 14:46:58.177045 7f9d83173700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.177136 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.177163 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.177200 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.177222 7f9d83173700 15 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.177227 7f9d83173700 10 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1 r = 0
2014-06-12 14:46:58.177230 7f9d83173700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.177241 7f9d83173700 15 _write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 413696~516096
2014-06-12 14:46:58.177334 7f9d83173700 10 lookup_strip_header done  cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.177533 7f9d83173700 10 file_to_extents done 
2014-06-12 14:46:58.178644 7f9d83173700 10 _generic_write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 413696~516096 = 0
2014-06-12 14:46:58.178676 7f9d83173700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.178685 7f9d83173700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.178689 7f9d83173700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.178693 7f9d83173700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.178820 7f9d83173700 20 set_header setting 16862 cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 parent seq 0
2014-06-12 14:46:58.178860 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.178869 7f9d83173700 20 set_header setting 2190 cid meta oid 1fe26b1a/pglog_3.237/0//-1 parent seq 0
2014-06-12 14:46:58.178875 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:46:58.180351 7f9d83173700 10 _do_op 0x6fbff20 seq 2602 r = 0, finisher 0x2e1cfd00 0
2014-06-12 14:46:58.180361 7f9d83173700 10 _finish_op 0x6fbff20 seq 2602 osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.180366 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540eb40 (0 active)
2014-06-12 14:46:58.180371 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.180383 7f9d7795c700  5 -- op tracker -- , seq: 748207, time: 2014-06-12 14:46:58.180376, event: sub_op_applied, op: osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.180391 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2845 (0'0,173'2845] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2843 active] sub_op_modify_applied on 0x2a29b180 op osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.180413 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:58.180431 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2845 (0'0,173'2845] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2843 active] op_applied on primary on version 173'2845
2014-06-12 14:46:58.180444 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2845 (0'0,173'2845] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2843 active] op_applied on replica on version 173'2845
2014-06-12 14:46:58.180468 7f9d7715b700  5 -- op tracker -- , seq: 748207, time: 2014-06-12 14:46:58.180461, event: commit_sent, op: osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.180479 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2845 (0'0,173'2845] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2843 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:58.180502 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:58.180535 7f9d7715b700  5 -- op tracker -- , seq: 748207, time: 2014-06-12 14:46:58.180531, event: done, op: osd_sub_op(client.4317.0:144054 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2845 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.186710 7f9d5703e700  5 -- op tracker -- , seq: 748208, time: 2014-06-12 14:46:58.185942, event: header_read, op: osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.186722 7f9d5703e700  5 -- op tracker -- , seq: 748208, time: 2014-06-12 14:46:58.185943, event: throttled, op: osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.186731 7f9d5703e700  5 -- op tracker -- , seq: 748208, time: 2014-06-12 14:46:58.186683, event: all_read, op: osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.186740 7f9d5703e700  5 -- op tracker -- , seq: 748208, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.186744 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.186750 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:58.186757 7f9d5703e700 15 osd.2 173 enqueue_op 0x598f500 prio 127 cost 516903 latency 0.000815 osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.186806 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.186816 7f9d61bd4700 10 osd.2 173 dequeue_op 0x598f500 prio 127 cost 516903 latency 0.000873 osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.237( v 173'2845 (0'0,173'2845] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active]
2014-06-12 14:46:58.186835 7f9d61bd4700  5 -- op tracker -- , seq: 748208, time: 2014-06-12 14:46:58.186831, event: reached_pg, op: osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.186844 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2845 (0'0,173'2845] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active] handle_message: 0x598f500
2014-06-12 14:46:58.186853 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2845 (0'0,173'2845] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active] sub_op_modify trans b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 v 173'2846 (transaction) 174
2014-06-12 14:46:58.186866 7f9d61bd4700  5 -- op tracker -- , seq: 748208, time: 2014-06-12 14:46:58.186863, event: started, op: osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.186878 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2845 (0'0,173'2845] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active] append_log log((0'0,173'2845], crt=0'0) [173'2846 (173'2845) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144055 2014-06-12 08:46:00.563070]
2014-06-12 14:46:58.186896 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2846 (0'0,173'2846] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active] add_log_entry 173'2846 (173'2845) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144055 2014-06-12 08:46:00.563070
2014-06-12 14:46:58.186911 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2846 (0'0,173'2846] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active] append_log  adding 1 keys
2014-06-12 14:46:58.186934 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2846, trimmed: 
2014-06-12 14:46:58.186951 7f9d61bd4700  5 -- op tracker -- , seq: 748208, time: 2014-06-12 14:46:58.186948, event: started, op: osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.186958 7f9d61bd4700  5 queue_transactions existing osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.186965 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.186966 7f9d61bd4700  5 queue_op 0x9eaef00 seq 2603 osr(3.237 0x276d3f0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:58.186970 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.237( v 173'2846 (0'0,173'2846] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active] do_sub_op osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.186974 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540eb40 (1 active)
2014-06-12 14:46:58.186978 7f9d61bd4700 10 osd.2 173 dequeue_op 0x598f500 finish
2014-06-12 14:46:58.186982 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.186983 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.186982 7f9d7a161700  5 _do_op 0x9eaef00 seq 2603 osr(3.237 0x276d3f0)/0x276d3f0 start
2014-06-12 14:46:58.186986 7f9d7a161700 10 _do_transaction on 0x6233cf0
2014-06-12 14:46:58.186993 7f9d7a161700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.187096 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.187122 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.187167 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.187186 7f9d7a161700 15 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.187194 7f9d7a161700 10 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1 r = 0
2014-06-12 14:46:58.187200 7f9d7a161700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.187213 7f9d7a161700 15 _write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 929792~516096
2014-06-12 14:46:58.187354 7f9d7a161700 10 lookup_strip_header done  cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.187525 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:46:58.189341 7f9d7a161700 10 _generic_write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 929792~516096 = 0
2014-06-12 14:46:58.189396 7f9d7a161700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.189412 7f9d7a161700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.189420 7f9d7a161700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.189428 7f9d7a161700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.189715 7f9d7a161700 20 set_header setting 16862 cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 parent seq 0
2014-06-12 14:46:58.189832 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.189855 7f9d7a161700 20 set_header setting 2190 cid meta oid 1fe26b1a/pglog_3.237/0//-1 parent seq 0
2014-06-12 14:46:58.189869 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:46:58.191105 7f9d5703e700  5 -- op tracker -- , seq: 748209, time: 2014-06-12 14:46:58.189875, event: header_read, op: osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.191133 7f9d5703e700  5 -- op tracker -- , seq: 748209, time: 2014-06-12 14:46:58.189878, event: throttled, op: osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.191153 7f9d5703e700  5 -- op tracker -- , seq: 748209, time: 2014-06-12 14:46:58.191052, event: all_read, op: osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.191194 7f9d5703e700  5 -- op tracker -- , seq: 748209, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.191205 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.191219 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:58.191235 7f9d5703e700 15 osd.2 173 enqueue_op 0x12bf7900 prio 127 cost 516903 latency 0.001360 osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.191287 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.191308 7f9d67be0700 10 osd.2 173 dequeue_op 0x12bf7900 prio 127 cost 516903 latency 0.001432 osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.237( v 173'2846 (0'0,173'2846] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active]
2014-06-12 14:46:58.191348 7f9d67be0700  5 -- op tracker -- , seq: 748209, time: 2014-06-12 14:46:58.191341, event: reached_pg, op: osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.191367 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2846 (0'0,173'2846] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active] handle_message: 0x12bf7900
2014-06-12 14:46:58.191392 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2846 (0'0,173'2846] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active] sub_op_modify trans b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 v 173'2847 (transaction) 174
2014-06-12 14:46:58.191415 7f9d67be0700  5 -- op tracker -- , seq: 748209, time: 2014-06-12 14:46:58.191409, event: started, op: osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.191442 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2846 (0'0,173'2846] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active] append_log log((0'0,173'2846], crt=0'0) [173'2847 (173'2846) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144056 2014-06-12 08:46:00.567234]
2014-06-12 14:46:58.191479 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2847 (0'0,173'2847] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active] add_log_entry 173'2847 (173'2846) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144056 2014-06-12 08:46:00.567234
2014-06-12 14:46:58.191515 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2847 (0'0,173'2847] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active] append_log  adding 1 keys
2014-06-12 14:46:58.191563 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2847, trimmed: 
2014-06-12 14:46:58.191602 7f9d67be0700  5 -- op tracker -- , seq: 748209, time: 2014-06-12 14:46:58.191595, event: started, op: osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.191618 7f9d67be0700  5 queue_transactions existing osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.191629 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.191631 7f9d67be0700  5 queue_op 0x62c6240 seq 2604 osr(3.237 0x276d3f0) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:58.191639 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.237( v 173'2847 (0'0,173'2847] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active] do_sub_op osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.191659 7f9d67be0700 10 osd.2 173 dequeue_op 0x12bf7900 finish
2014-06-12 14:46:58.191665 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.191668 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.191675 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540eb40 (2 active)
2014-06-12 14:46:58.193818 7f9d7a161700 10 _do_op 0x9eaef00 seq 2603 r = 0, finisher 0x29c88120 0
2014-06-12 14:46:58.193829 7f9d7a161700 10 _finish_op 0x9eaef00 seq 2603 osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.193834 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540eb40 (1 active)
2014-06-12 14:46:58.193841 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.193843 7f9d7715b700  5 -- op tracker -- , seq: 748208, time: 2014-06-12 14:46:58.193836, event: commit_sent, op: osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.193868 7f9d7f16b700  5 _do_op 0x62c6240 seq 2604 osr(3.237 0x276d3f0)/0x276d3f0 start
2014-06-12 14:46:58.193875 7f9d7f16b700 10 _do_transaction on 0x5afc9b0
2014-06-12 14:46:58.193859 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2847 (0'0,173'2847] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2844 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:58.193891 7f9d7f16b700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.193891 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:58.194005 7f9d7795c700  5 -- op tracker -- , seq: 748208, time: 2014-06-12 14:46:58.193988, event: sub_op_applied, op: osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.194035 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.194034 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2847 (0'0,173'2847] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2845 active] sub_op_modify_applied on 0x6233c00 op osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.194069 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.194069 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2847 (0'0,173'2847] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2845 active] op_applied on primary on version 173'2846
2014-06-12 14:46:58.194083 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2847 (0'0,173'2847] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2845 active] op_applied on replica on version 173'2846
2014-06-12 14:46:58.194111 7f9d7795c700  5 -- op tracker -- , seq: 748208, time: 2014-06-12 14:46:58.194105, event: done, op: osd_sub_op(client.4317.0:144055 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2846 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.194127 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.194154 7f9d7f16b700 15 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.194162 7f9d7f16b700 10 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1 r = 0
2014-06-12 14:46:58.194169 7f9d7f16b700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.194297 7f9d7f16b700 15 _write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 1445888~516096
2014-06-12 14:46:58.194492 7f9d7f16b700 10 lookup_strip_header done  cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.194662 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:46:58.196798 7f9d7f16b700 10 _generic_write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 1445888~516096 = 0
2014-06-12 14:46:58.196857 7f9d7f16b700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.196876 7f9d7f16b700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.196887 7f9d7f16b700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.196898 7f9d7f16b700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.197159 7f9d7f16b700 20 set_header setting 16862 cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 parent seq 0
2014-06-12 14:46:58.197258 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.197279 7f9d7f16b700 20 set_header setting 2190 cid meta oid 1fe26b1a/pglog_3.237/0//-1 parent seq 0
2014-06-12 14:46:58.197293 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:46:58.200067 7f9d5703e700  5 -- op tracker -- , seq: 748210, time: 2014-06-12 14:46:58.199018, event: header_read, op: osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.200088 7f9d5703e700  5 -- op tracker -- , seq: 748210, time: 2014-06-12 14:46:58.199020, event: throttled, op: osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.200100 7f9d5703e700  5 -- op tracker -- , seq: 748210, time: 2014-06-12 14:46:58.200015, event: all_read, op: osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.200129 7f9d5703e700  5 -- op tracker -- , seq: 748210, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.200137 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.200146 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:58.200156 7f9d5703e700 15 osd.2 173 enqueue_op 0x876fb00 prio 127 cost 516903 latency 0.001137 osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.200194 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.200206 7f9d673df700 10 osd.2 173 dequeue_op 0x876fb00 prio 127 cost 516903 latency 0.001188 osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.237( v 173'2847 (0'0,173'2847] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2845 active]
2014-06-12 14:46:58.200243 7f9d673df700  5 -- op tracker -- , seq: 748210, time: 2014-06-12 14:46:58.200237, event: reached_pg, op: osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.200270 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2847 (0'0,173'2847] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2845 active] handle_message: 0x876fb00
2014-06-12 14:46:58.200278 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2847 (0'0,173'2847] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2845 active] sub_op_modify trans b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 v 173'2848 (transaction) 174
2014-06-12 14:46:58.200288 7f9d673df700  5 -- op tracker -- , seq: 748210, time: 2014-06-12 14:46:58.200285, event: started, op: osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.200299 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2847 (0'0,173'2847] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2845 active] append_log log((0'0,173'2847], crt=0'0) [173'2848 (173'2847) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144057 2014-06-12 08:46:00.576165]
2014-06-12 14:46:58.200316 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2845 active] add_log_entry 173'2848 (173'2847) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144057 2014-06-12 08:46:00.576165
2014-06-12 14:46:58.200330 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2845 active] append_log  adding 1 keys
2014-06-12 14:46:58.200353 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2848, trimmed: 
2014-06-12 14:46:58.200369 7f9d673df700  5 -- op tracker -- , seq: 748210, time: 2014-06-12 14:46:58.200366, event: started, op: osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.200376 7f9d673df700  5 queue_transactions existing osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.200379 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.200380 7f9d673df700  5 queue_op 0x6b404e0 seq 2605 osr(3.237 0x276d3f0) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:58.200385 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2845 active] do_sub_op osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.200394 7f9d673df700 10 osd.2 173 dequeue_op 0x876fb00 finish
2014-06-12 14:46:58.200397 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.200398 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.200395 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540eb40 (2 active)
2014-06-12 14:46:58.200619 7f9d7f16b700 10 _do_op 0x62c6240 seq 2604 r = 0, finisher 0x889b700 0
2014-06-12 14:46:58.200638 7f9d7f16b700 10 _finish_op 0x62c6240 seq 2604 osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.200649 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540eb40 (1 active)
2014-06-12 14:46:58.200657 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.200668 7f9d7e169700  5 _do_op 0x6b404e0 seq 2605 osr(3.237 0x276d3f0)/0x276d3f0 start
2014-06-12 14:46:58.200678 7f9d7e169700 10 _do_transaction on 0x1da3ed30
2014-06-12 14:46:58.200686 7f9d7e169700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.200673 7f9d7715b700  5 -- op tracker -- , seq: 748209, time: 2014-06-12 14:46:58.200657, event: commit_sent, op: osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.200700 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2845 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:58.200749 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:58.200782 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.200802 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.200813 7f9d7795c700  5 -- op tracker -- , seq: 748209, time: 2014-06-12 14:46:58.200807, event: sub_op_applied, op: osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.200830 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.200824 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2846 active] sub_op_modify_applied on 0x5afc8c0 op osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.200844 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2846 active] op_applied on primary on version 173'2847
2014-06-12 14:46:58.200851 7f9d7e169700 15 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.200855 7f9d7e169700 10 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1 r = 0
2014-06-12 14:46:58.200851 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2846 active] op_applied on replica on version 173'2847
2014-06-12 14:46:58.200859 7f9d7e169700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.200866 7f9d7e169700 15 _write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 1961984~516096
2014-06-12 14:46:58.200864 7f9d7795c700  5 -- op tracker -- , seq: 748209, time: 2014-06-12 14:46:58.200862, event: done, op: osd_sub_op(client.4317.0:144056 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2847 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.200946 7f9d7e169700 10 lookup_strip_header done  cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.200958 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:46:58.201994 7f9d7e169700 10 _generic_write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 1961984~516096 = 0
2014-06-12 14:46:58.202021 7f9d7e169700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.202029 7f9d7e169700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.202035 7f9d7e169700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.202040 7f9d7e169700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.202173 7f9d7e169700 20 set_header setting 16862 cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 parent seq 0
2014-06-12 14:46:58.202210 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.202220 7f9d7e169700 20 set_header setting 2190 cid meta oid 1fe26b1a/pglog_3.237/0//-1 parent seq 0
2014-06-12 14:46:58.202227 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:46:58.203960 7f9d7e169700 10 _do_op 0x6b404e0 seq 2605 r = 0, finisher 0x5956180 0
2014-06-12 14:46:58.203982 7f9d7e169700 10 _finish_op 0x6b404e0 seq 2605 osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.203988 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540eb40 (0 active)
2014-06-12 14:46:58.203991 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.203984 7f9d7715b700  5 -- op tracker -- , seq: 748210, time: 2014-06-12 14:46:58.203976, event: commit_sent, op: osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.203995 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2846 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:58.204037 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:58.204165 7f9d7795c700  5 -- op tracker -- , seq: 748210, time: 2014-06-12 14:46:58.204151, event: sub_op_applied, op: osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.204189 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] sub_op_modify_applied on 0x1da3ec40 op osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.204215 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] op_applied on primary on version 173'2848
2014-06-12 14:46:58.204229 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] op_applied on replica on version 173'2848
2014-06-12 14:46:58.204260 7f9d7795c700  5 -- op tracker -- , seq: 748210, time: 2014-06-12 14:46:58.204254, event: done, op: osd_sub_op(client.4317.0:144057 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2848 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.207238 7f9d5703e700  5 -- op tracker -- , seq: 748211, time: 2014-06-12 14:46:58.206134, event: header_read, op: osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.207257 7f9d5703e700  5 -- op tracker -- , seq: 748211, time: 2014-06-12 14:46:58.206136, event: throttled, op: osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.207269 7f9d5703e700  5 -- op tracker -- , seq: 748211, time: 2014-06-12 14:46:58.207180, event: all_read, op: osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.207280 7f9d5703e700  5 -- op tracker -- , seq: 748211, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.207287 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.207296 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:58.207305 7f9d5703e700 15 osd.2 173 enqueue_op 0x876e900 prio 127 cost 516903 latency 0.001171 osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.207377 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.207397 7f9d663dd700 10 osd.2 173 dequeue_op 0x876e900 prio 127 cost 516903 latency 0.001263 osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active]
2014-06-12 14:46:58.207434 7f9d663dd700  5 -- op tracker -- , seq: 748211, time: 2014-06-12 14:46:58.207428, event: reached_pg, op: osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.207457 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] handle_message: 0x876e900
2014-06-12 14:46:58.207473 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] sub_op_modify trans b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 v 173'2849 (transaction) 174
2014-06-12 14:46:58.207495 7f9d663dd700  5 -- op tracker -- , seq: 748211, time: 2014-06-12 14:46:58.207489, event: started, op: osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.207515 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2848 (0'0,173'2848] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] append_log log((0'0,173'2848], crt=0'0) [173'2849 (173'2848) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144058 2014-06-12 08:46:00.582916]
2014-06-12 14:46:58.207549 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2849 (0'0,173'2849] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] add_log_entry 173'2849 (173'2848) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144058 2014-06-12 08:46:00.582916
2014-06-12 14:46:58.207599 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2849 (0'0,173'2849] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] append_log  adding 1 keys
2014-06-12 14:46:58.207649 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2849, trimmed: 
2014-06-12 14:46:58.207683 7f9d663dd700  5 -- op tracker -- , seq: 748211, time: 2014-06-12 14:46:58.207676, event: started, op: osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.207699 7f9d663dd700  5 queue_transactions existing osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.207704 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.207706 7f9d663dd700  5 queue_op 0x1dfbb080 seq 2606 osr(3.237 0x276d3f0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:58.207714 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.237( v 173'2849 (0'0,173'2849] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] do_sub_op osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.207734 7f9d663dd700 10 osd.2 173 dequeue_op 0x876e900 finish
2014-06-12 14:46:58.207739 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.207742 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.207750 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540eb40 (1 active)
2014-06-12 14:46:58.207766 7f9d7815d700  5 _do_op 0x1dfbb080 seq 2606 osr(3.237 0x276d3f0)/0x276d3f0 start
2014-06-12 14:46:58.207773 7f9d7815d700 10 _do_transaction on 0x9b93430
2014-06-12 14:46:58.207782 7f9d7815d700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.207871 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.207891 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.207919 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.207936 7f9d7815d700 15 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.207940 7f9d7815d700 10 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1 r = 0
2014-06-12 14:46:58.207943 7f9d7815d700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.207954 7f9d7815d700 15 _write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 2478080~516096
2014-06-12 14:46:58.208039 7f9d7815d700 10 lookup_strip_header done  cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.208203 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:46:58.209245 7f9d7815d700 10 _generic_write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 2478080~516096 = 0
2014-06-12 14:46:58.209281 7f9d7815d700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.209289 7f9d7815d700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.209293 7f9d7815d700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.209298 7f9d7815d700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.209431 7f9d7815d700 20 set_header setting 16862 cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 parent seq 0
2014-06-12 14:46:58.209481 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.209491 7f9d7815d700 20 set_header setting 2190 cid meta oid 1fe26b1a/pglog_3.237/0//-1 parent seq 0
2014-06-12 14:46:58.209497 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:46:58.209589 7f9d5703e700  5 -- op tracker -- , seq: 748212, time: 2014-06-12 14:46:58.208774, event: header_read, op: osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.209598 7f9d5703e700  5 -- op tracker -- , seq: 748212, time: 2014-06-12 14:46:58.208776, event: throttled, op: osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.209604 7f9d5703e700  5 -- op tracker -- , seq: 748212, time: 2014-06-12 14:46:58.209560, event: all_read, op: osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.209629 7f9d5703e700  5 -- op tracker -- , seq: 748212, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.209633 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.209637 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:58.209655 7f9d5703e700 15 osd.2 173 enqueue_op 0x68a7900 prio 127 cost 516903 latency 0.000880 osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.209695 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.209709 7f9d65bdc700 10 osd.2 173 dequeue_op 0x68a7900 prio 127 cost 516903 latency 0.000934 osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.237( v 173'2849 (0'0,173'2849] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active]
2014-06-12 14:46:58.209747 7f9d65bdc700  5 -- op tracker -- , seq: 748212, time: 2014-06-12 14:46:58.209741, event: reached_pg, op: osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.209760 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2849 (0'0,173'2849] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] handle_message: 0x68a7900
2014-06-12 14:46:58.209778 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2849 (0'0,173'2849] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] sub_op_modify trans b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 v 173'2850 (transaction) 174
2014-06-12 14:46:58.209801 7f9d65bdc700  5 -- op tracker -- , seq: 748212, time: 2014-06-12 14:46:58.209796, event: started, op: osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.209820 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2849 (0'0,173'2849] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] append_log log((0'0,173'2849], crt=0'0) [173'2850 (173'2849) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144059 2014-06-12 08:46:00.586179]
2014-06-12 14:46:58.209850 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2850 (0'0,173'2850] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] add_log_entry 173'2850 (173'2849) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144059 2014-06-12 08:46:00.586179
2014-06-12 14:46:58.209883 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2850 (0'0,173'2850] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] append_log  adding 1 keys
2014-06-12 14:46:58.209932 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2850, trimmed: 
2014-06-12 14:46:58.209964 7f9d65bdc700  5 -- op tracker -- , seq: 748212, time: 2014-06-12 14:46:58.209957, event: started, op: osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.209978 7f9d65bdc700  5 queue_transactions existing osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.209983 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.209985 7f9d65bdc700  5 queue_op 0x5428000 seq 2607 osr(3.237 0x276d3f0) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:46:58.209994 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.237( v 173'2850 (0'0,173'2850] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] do_sub_op osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.210013 7f9d65bdc700 10 osd.2 173 dequeue_op 0x68a7900 finish
2014-06-12 14:46:58.210018 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.210022 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.210030 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540eb40 (2 active)
2014-06-12 14:46:58.212664 7f9d7815d700 10 _do_op 0x1dfbb080 seq 2606 r = 0, finisher 0x6ab84e0 0
2014-06-12 14:46:58.212684 7f9d7815d700 10 _finish_op 0x1dfbb080 seq 2606 osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.212694 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540eb40 (1 active)
2014-06-12 14:46:58.212705 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.212714 7f9d7715b700  5 -- op tracker -- , seq: 748211, time: 2014-06-12 14:46:58.212697, event: commit_sent, op: osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.212730 7f9d82171700  5 _do_op 0x5428000 seq 2607 osr(3.237 0x276d3f0)/0x276d3f0 start
2014-06-12 14:46:58.212743 7f9d82171700 10 _do_transaction on 0x2b2ae630
2014-06-12 14:46:58.212753 7f9d82171700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.212732 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2850 (0'0,173'2850] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2847 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:58.212774 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:58.212867 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.212870 7f9d7795c700  5 -- op tracker -- , seq: 748211, time: 2014-06-12 14:46:58.212856, event: sub_op_applied, op: osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.212908 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.212889 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2850 (0'0,173'2850] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2848 active] sub_op_modify_applied on 0x9b93340 op osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.212915 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2850 (0'0,173'2850] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2848 active] op_applied on primary on version 173'2849
2014-06-12 14:46:58.212928 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2850 (0'0,173'2850] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2848 active] op_applied on replica on version 173'2849
2014-06-12 14:46:58.212955 7f9d7795c700  5 -- op tracker -- , seq: 748211, time: 2014-06-12 14:46:58.212949, event: done, op: osd_sub_op(client.4317.0:144058 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2849 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.212967 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.213065 7f9d82171700 15 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.213078 7f9d82171700 10 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1 r = 0
2014-06-12 14:46:58.213086 7f9d82171700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.213111 7f9d82171700 15 _write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 2994176~516096
2014-06-12 14:46:58.213288 7f9d82171700 10 lookup_strip_header done  cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.213452 7f9d82171700 10 file_to_extents done 
2014-06-12 14:46:58.215651 7f9d82171700 10 _generic_write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 2994176~516096 = 0
2014-06-12 14:46:58.215724 7f9d82171700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.215744 7f9d82171700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.215752 7f9d82171700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.215763 7f9d82171700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.216019 7f9d82171700 20 set_header setting 16862 cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 parent seq 0
2014-06-12 14:46:58.216129 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.216154 7f9d82171700 20 set_header setting 2190 cid meta oid 1fe26b1a/pglog_3.237/0//-1 parent seq 0
2014-06-12 14:46:58.216168 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:46:58.219959 7f9d5703e700  5 -- op tracker -- , seq: 748213, time: 2014-06-12 14:46:58.218556, event: header_read, op: osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.219983 7f9d5703e700  5 -- op tracker -- , seq: 748213, time: 2014-06-12 14:46:58.218559, event: throttled, op: osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.219995 7f9d5703e700  5 -- op tracker -- , seq: 748213, time: 2014-06-12 14:46:58.219889, event: all_read, op: osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.220038 7f9d5703e700  5 -- op tracker -- , seq: 748213, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.220046 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.220056 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:58.220067 7f9d5703e700 15 osd.2 173 enqueue_op 0x68a6700 prio 127 cost 516903 latency 0.001511 osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.220144 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.220167 7f9d62bd6700 10 osd.2 173 dequeue_op 0x68a6700 prio 127 cost 516903 latency 0.001611 osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.237( v 173'2850 (0'0,173'2850] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2848 active]
2014-06-12 14:46:58.220215 7f9d62bd6700  5 -- op tracker -- , seq: 748213, time: 2014-06-12 14:46:58.220208, event: reached_pg, op: osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.220232 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2850 (0'0,173'2850] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2848 active] handle_message: 0x68a6700
2014-06-12 14:46:58.220249 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2850 (0'0,173'2850] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2848 active] sub_op_modify trans b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 v 173'2851 (transaction) 174
2014-06-12 14:46:58.220270 7f9d62bd6700  5 -- op tracker -- , seq: 748213, time: 2014-06-12 14:46:58.220265, event: started, op: osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.220298 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2850 (0'0,173'2850] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2848 active] append_log log((0'0,173'2850], crt=0'0) [173'2851 (173'2850) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144060 2014-06-12 08:46:00.595386]
2014-06-12 14:46:58.220335 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2851 (0'0,173'2851] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2848 active] add_log_entry 173'2851 (173'2850) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144060 2014-06-12 08:46:00.595386
2014-06-12 14:46:58.220370 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2851 (0'0,173'2851] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2848 active] append_log  adding 1 keys
2014-06-12 14:46:58.220415 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2851, trimmed: 
2014-06-12 14:46:58.220436 7f9d82171700 10 _do_op 0x5428000 seq 2607 r = 0, finisher 0x28f4280 0
2014-06-12 14:46:58.220453 7f9d82171700 10 _finish_op 0x5428000 seq 2607 osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.220464 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540eb40 (0 active)
2014-06-12 14:46:58.220477 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.220453 7f9d62bd6700  5 -- op tracker -- , seq: 748213, time: 2014-06-12 14:46:58.220446, event: started, op: osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.220491 7f9d62bd6700  5 queue_transactions existing osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.220499 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.220501 7f9d62bd6700  5 queue_op 0x4c77da0 seq 2608 osr(3.237 0x276d3f0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:58.220511 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.237( v 173'2851 (0'0,173'2851] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2848 active] do_sub_op osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.220532 7f9d62bd6700 10 osd.2 173 dequeue_op 0x68a6700 finish
2014-06-12 14:46:58.220540 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.220544 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.220544 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540eb40 (1 active)
2014-06-12 14:46:58.220560 7f9d7e96a700  5 _do_op 0x4c77da0 seq 2608 osr(3.237 0x276d3f0)/0x276d3f0 start
2014-06-12 14:46:58.220567 7f9d7e96a700 10 _do_transaction on 0x601b270
2014-06-12 14:46:58.220576 7f9d7e96a700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.220564 7f9d7715b700  5 -- op tracker -- , seq: 748212, time: 2014-06-12 14:46:58.220548, event: commit_sent, op: osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.220587 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2851 (0'0,173'2851] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2848 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:58.220631 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:58.220708 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.220739 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.220724 7f9d7795c700  5 -- op tracker -- , seq: 748212, time: 2014-06-12 14:46:58.220710, event: sub_op_applied, op: osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.220749 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2851 (0'0,173'2851] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2849 active] sub_op_modify_applied on 0x2b2ae540 op osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.220776 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2851 (0'0,173'2851] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2849 active] op_applied on primary on version 173'2850
2014-06-12 14:46:58.220797 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.220789 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2851 (0'0,173'2851] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2849 active] op_applied on replica on version 173'2850
2014-06-12 14:46:58.220823 7f9d7e96a700 15 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.220824 7f9d7795c700  5 -- op tracker -- , seq: 748212, time: 2014-06-12 14:46:58.220818, event: done, op: osd_sub_op(client.4317.0:144059 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2850 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.220831 7f9d7e96a700 10 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1 r = 0
2014-06-12 14:46:58.220838 7f9d7e96a700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.220958 7f9d7e96a700 15 _write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 3510272~516096
2014-06-12 14:46:58.221149 7f9d7e96a700 10 lookup_strip_header done  cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.221275 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:46:58.223295 7f9d7e96a700 10 _generic_write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 3510272~516096 = 0
2014-06-12 14:46:58.223351 7f9d7e96a700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.223371 7f9d7e96a700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.223379 7f9d7e96a700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.223390 7f9d7e96a700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.223636 7f9d7e96a700 20 set_header setting 16862 cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 parent seq 0
2014-06-12 14:46:58.223737 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.223758 7f9d7e96a700 20 set_header setting 2190 cid meta oid 1fe26b1a/pglog_3.237/0//-1 parent seq 0
2014-06-12 14:46:58.223772 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:46:58.224537 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:58.224905 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:58.225308 7f9d5703e700  5 -- op tracker -- , seq: 748214, time: 2014-06-12 14:46:58.224234, event: header_read, op: osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.225336 7f9d5703e700  5 -- op tracker -- , seq: 748214, time: 2014-06-12 14:46:58.224237, event: throttled, op: osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.225354 7f9d5703e700  5 -- op tracker -- , seq: 748214, time: 2014-06-12 14:46:58.225243, event: all_read, op: osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.225378 7f9d5703e700  5 -- op tracker -- , seq: 748214, time: 0.000000, event: dispatched, op: osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.225387 7f9d5703e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.225401 7f9d5703e700 20 osd.2 173 should_share_map osd.20 172.16.96.202:6802/1090 173
2014-06-12 14:46:58.225413 7f9d5703e700 15 osd.2 173 enqueue_op 0x68a7400 prio 127 cost 168743 latency 0.001179 osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.225491 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.225513 7f9d683e1700 10 osd.2 173 dequeue_op 0x68a7400 prio 127 cost 168743 latency 0.001279 osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.237( v 173'2851 (0'0,173'2851] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2849 active]
2014-06-12 14:46:58.225553 7f9d683e1700  5 -- op tracker -- , seq: 748214, time: 2014-06-12 14:46:58.225546, event: reached_pg, op: osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.225569 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2851 (0'0,173'2851] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2849 active] handle_message: 0x68a7400
2014-06-12 14:46:58.225585 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2851 (0'0,173'2851] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2849 active] sub_op_modify trans b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 v 173'2852 (transaction) 174
2014-06-12 14:46:58.225606 7f9d683e1700  5 -- op tracker -- , seq: 748214, time: 2014-06-12 14:46:58.225601, event: started, op: osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.225640 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2851 (0'0,173'2851] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2849 active] append_log log((0'0,173'2851], crt=0'0) [173'2852 (173'2851) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144061 2014-06-12 08:46:00.602900]
2014-06-12 14:46:58.225676 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2852 (0'0,173'2852] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2849 active] add_log_entry 173'2852 (173'2851) modify   b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 by client.4317.0:144061 2014-06-12 08:46:00.602900
2014-06-12 14:46:58.225708 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2852 (0'0,173'2852] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2849 active] append_log  adding 1 keys
2014-06-12 14:46:58.225751 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2852, trimmed: 
2014-06-12 14:46:58.225785 7f9d683e1700  5 -- op tracker -- , seq: 748214, time: 2014-06-12 14:46:58.225778, event: started, op: osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.225798 7f9d683e1700  5 queue_transactions existing osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.225803 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.225805 7f9d683e1700  5 queue_op 0xed29f80 seq 2609 osr(3.237 0x276d3f0) 170229 bytes   (queue has 2 ops and 688618 bytes)
2014-06-12 14:46:58.225822 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540eb40 (2 active)
2014-06-12 14:46:58.225813 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.237( v 173'2852 (0'0,173'2852] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2849 active] do_sub_op osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.225833 7f9d683e1700 10 osd.2 173 dequeue_op 0x68a7400 finish
2014-06-12 14:46:58.225838 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.225842 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.228229 7f9d7e96a700 10 _do_op 0x4c77da0 seq 2608 r = 0, finisher 0x692f4c0 0
2014-06-12 14:46:58.228250 7f9d7e96a700 10 _finish_op 0x4c77da0 seq 2608 osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.228261 7f9d8116f700  5 _do_op 0xed29f80 seq 2609 osr(3.237 0x276d3f0)/0x276d3f0 start
2014-06-12 14:46:58.228262 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540eb40 (1 active)
2014-06-12 14:46:58.228271 7f9d8116f700 10 _do_transaction on 0x21255f0
2014-06-12 14:46:58.228272 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.228282 7f9d8116f700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.228314 7f9d7715b700  5 -- op tracker -- , seq: 748213, time: 2014-06-12 14:46:58.228295, event: commit_sent, op: osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.228345 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2852 (0'0,173'2852] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2849 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:58.228378 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:58.228402 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.228432 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.228431 7f9d7795c700  5 -- op tracker -- , seq: 748213, time: 2014-06-12 14:46:58.228415, event: sub_op_applied, op: osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.228451 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2852 (0'0,173'2852] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2850 active] sub_op_modify_applied on 0x601b180 op osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.228473 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2852 (0'0,173'2852] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2850 active] op_applied on primary on version 173'2851
2014-06-12 14:46:58.228489 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.228485 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2852 (0'0,173'2852] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2850 active] op_applied on replica on version 173'2851
2014-06-12 14:46:58.228514 7f9d8116f700 15 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.228515 7f9d7795c700  5 -- op tracker -- , seq: 748213, time: 2014-06-12 14:46:58.228510, event: done, op: osd_sub_op(client.4317.0:144060 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2851 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.228521 7f9d8116f700 10 _omap_rmkeys meta/1fe26b1a/pglog_3.237/0//-1 r = 0
2014-06-12 14:46:58.228529 7f9d8116f700 15 _omap_setkeys meta/1fe26b1a/pglog_3.237/0//-1
2014-06-12 14:46:58.228645 7f9d8116f700 15 _write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 4026368~167936
2014-06-12 14:46:58.228832 7f9d8116f700 10 lookup_strip_header done  cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.228960 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:46:58.229758 7f9d8116f700 10 _generic_write 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 4026368~167936 = 0
2014-06-12 14:46:58.229795 7f9d8116f700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.229811 7f9d8116f700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.229819 7f9d8116f700 15 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3
2014-06-12 14:46:58.229828 7f9d8116f700 10 _setattrs 3.237_head/b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 = 0
2014-06-12 14:46:58.230013 7f9d8116f700 20 set_header setting 16862 cid 3.237_head oid b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 parent seq 0
2014-06-12 14:46:58.230059 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.230077 7f9d8116f700 20 set_header setting 2190 cid meta oid 1fe26b1a/pglog_3.237/0//-1 parent seq 0
2014-06-12 14:46:58.230089 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:46:58.231834 7f9d8116f700 10 _do_op 0xed29f80 seq 2609 r = 0, finisher 0x17445640 0
2014-06-12 14:46:58.231849 7f9d8116f700 10 _finish_op 0xed29f80 seq 2609 osr(3.237 0x276d3f0)/0x276d3f0
2014-06-12 14:46:58.231857 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540eb40 (0 active)
2014-06-12 14:46:58.231861 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.231915 7f9d7795c700  5 -- op tracker -- , seq: 748214, time: 2014-06-12 14:46:58.231897, event: sub_op_applied, op: osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.231956 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2852 (0'0,173'2852] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2850 active] sub_op_modify_applied on 0x2125500 op osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.231989 7f9d7795c700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:58.232013 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2852 (0'0,173'2852] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2850 active] op_applied on primary on version 173'2852
2014-06-12 14:46:58.232031 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2852 (0'0,173'2852] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2850 active] op_applied on replica on version 173'2852
2014-06-12 14:46:58.232069 7f9d7715b700  5 -- op tracker -- , seq: 748214, time: 2014-06-12 14:46:58.232056, event: commit_sent, op: osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.232086 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.237( v 173'2852 (0'0,173'2852] local-les=137 n=285 ec=23 les/c 137/137 136/136/136) [20,2] r=1 lpr=136 pi=23-135/12 luod=0'0 crt=0'0 lcod 173'2850 active] sub_op_modify_commit on op osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.20
2014-06-12 14:46:58.232116 7f9d7715b700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:58.232155 7f9d7715b700  5 -- op tracker -- , seq: 748214, time: 2014-06-12 14:46:58.232148, event: done, op: osd_sub_op(client.4317.0:144061 3.237 b1dd0a37/rbd_data.10b46b8b4567.000000000000105d/head//3 [] v 173'2852 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.302063 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:58.302095 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:58.324237 7f9d58c5a700  5 -- op tracker -- , seq: 748215, time: 2014-06-12 14:46:58.323315, event: header_read, op: osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.324261 7f9d58c5a700  5 -- op tracker -- , seq: 748215, time: 2014-06-12 14:46:58.323318, event: throttled, op: osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.324278 7f9d58c5a700  5 -- op tracker -- , seq: 748215, time: 2014-06-12 14:46:58.324184, event: all_read, op: osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.324288 7f9d58c5a700  5 -- op tracker -- , seq: 748215, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.324294 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.324302 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:58.324312 7f9d58c5a700 15 osd.2 173 enqueue_op 0x31d61a00 prio 127 cost 516903 latency 0.000997 osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.324349 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.324374 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x31d61a00 prio 127 cost 516903 latency 0.001058 osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.34f( v 173'2927 (0'0,173'2927] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2926 active]
2014-06-12 14:46:58.324427 7f9d6b3e7700  5 -- op tracker -- , seq: 748215, time: 2014-06-12 14:46:58.324417, event: reached_pg, op: osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.324458 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2927 (0'0,173'2927] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2926 active] handle_message: 0x31d61a00
2014-06-12 14:46:58.324489 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2927 (0'0,173'2927] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2926 active] sub_op_modify trans 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 v 173'2928 (transaction) 174
2014-06-12 14:46:58.324537 7f9d6b3e7700  5 -- op tracker -- , seq: 748215, time: 2014-06-12 14:46:58.324526, event: started, op: osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.324578 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2927 (0'0,173'2927] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2926 active] append_log log((0'0,173'2927], crt=0'0) [173'2928 (173'2927) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144292 2014-06-12 08:46:00.700065]
2014-06-12 14:46:58.324620 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2928 (0'0,173'2928] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2926 active] add_log_entry 173'2928 (173'2927) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144292 2014-06-12 08:46:00.700065
2014-06-12 14:46:58.324643 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2928 (0'0,173'2928] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2926 active] append_log  adding 1 keys
2014-06-12 14:46:58.324702 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2928, trimmed: 
2014-06-12 14:46:58.324756 7f9d6b3e7700  5 -- op tracker -- , seq: 748215, time: 2014-06-12 14:46:58.324746, event: started, op: osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.324775 7f9d6b3e7700  5 queue_transactions existing osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.324783 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.324787 7f9d6b3e7700  5 queue_op 0x7694240 seq 2642 osr(3.34f 0x276d480) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:58.324801 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.34f( v 173'2928 (0'0,173'2928] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2926 active] do_sub_op osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.324837 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x31d61a00 finish
2014-06-12 14:46:58.324846 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.324852 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.324872 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540e240 (1 active)
2014-06-12 14:46:58.324878 7f9d7d167700  5 _do_op 0x7694240 seq 2642 osr(3.34f 0x276d480)/0x276d480 start
2014-06-12 14:46:58.324883 7f9d7d167700 10 _do_transaction on 0x14672ef0
2014-06-12 14:46:58.324889 7f9d7d167700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.325083 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.325117 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.325162 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.325187 7f9d7d167700 15 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.325194 7f9d7d167700 10 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1 r = 0
2014-06-12 14:46:58.325201 7f9d7d167700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.325341 7f9d7d167700 15 _write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 1114112~516096
2014-06-12 14:46:58.325520 7f9d7d167700 10 lookup_strip_header done  cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.325686 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:46:58.328315 7f9d7d167700 10 _generic_write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 1114112~516096 = 0
2014-06-12 14:46:58.328405 7f9d7d167700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.328425 7f9d7d167700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.328433 7f9d7d167700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.328444 7f9d7d167700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.328769 7f9d7d167700 20 set_header setting 16877 cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 parent seq 0
2014-06-12 14:46:58.328874 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.328896 7f9d7d167700 20 set_header setting 2162 cid meta oid 1fe1d62b/pglog_3.34f/0//-1 parent seq 0
2014-06-12 14:46:58.328909 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:46:58.333081 7f9d7d167700 10 _do_op 0x7694240 seq 2642 r = 0, finisher 0x82c1400 0
2014-06-12 14:46:58.333105 7f9d7d167700 10 _finish_op 0x7694240 seq 2642 osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.333113 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540e240 (0 active)
2014-06-12 14:46:58.333127 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.333151 7f9d7795c700  5 -- op tracker -- , seq: 748215, time: 2014-06-12 14:46:58.333132, event: sub_op_applied, op: osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.333170 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2928 (0'0,173'2928] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2926 active] sub_op_modify_applied on 0x14672e00 op osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.333212 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:58.333246 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2928 (0'0,173'2928] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2926 active] op_applied on primary on version 173'2928
2014-06-12 14:46:58.333283 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2928 (0'0,173'2928] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2926 active] op_applied on replica on version 173'2928
2014-06-12 14:46:58.333347 7f9d7715b700  5 -- op tracker -- , seq: 748215, time: 2014-06-12 14:46:58.333332, event: commit_sent, op: osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.333372 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2928 (0'0,173'2928] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2926 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:58.333405 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:58.333451 7f9d7715b700  5 -- op tracker -- , seq: 748215, time: 2014-06-12 14:46:58.333444, event: done, op: osd_sub_op(client.4306.0:144292 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2928 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.338912 7f9d58c5a700  5 -- op tracker -- , seq: 748216, time: 2014-06-12 14:46:58.338143, event: header_read, op: osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.338924 7f9d58c5a700  5 -- op tracker -- , seq: 748216, time: 2014-06-12 14:46:58.338145, event: throttled, op: osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.338941 7f9d58c5a700  5 -- op tracker -- , seq: 748216, time: 2014-06-12 14:46:58.338870, event: all_read, op: osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.338946 7f9d58c5a700  5 -- op tracker -- , seq: 748216, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.338951 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.338956 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:58.338962 7f9d58c5a700 15 osd.2 173 enqueue_op 0x31d60100 prio 127 cost 516903 latency 0.000819 osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.338992 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.339001 7f9d693e3700 10 osd.2 173 dequeue_op 0x31d60100 prio 127 cost 516903 latency 0.000858 osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.34f( v 173'2928 (0'0,173'2928] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2927 active]
2014-06-12 14:46:58.339019 7f9d693e3700  5 -- op tracker -- , seq: 748216, time: 2014-06-12 14:46:58.339016, event: reached_pg, op: osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.339026 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2928 (0'0,173'2928] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2927 active] handle_message: 0x31d60100
2014-06-12 14:46:58.339034 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2928 (0'0,173'2928] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2927 active] sub_op_modify trans 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 v 173'2929 (transaction) 174
2014-06-12 14:46:58.339049 7f9d693e3700  5 -- op tracker -- , seq: 748216, time: 2014-06-12 14:46:58.339046, event: started, op: osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.339061 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2928 (0'0,173'2928] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2927 active] append_log log((0'0,173'2928], crt=0'0) [173'2929 (173'2928) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144293 2014-06-12 08:46:00.715216]
2014-06-12 14:46:58.339088 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2929 (0'0,173'2929] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2927 active] add_log_entry 173'2929 (173'2928) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144293 2014-06-12 08:46:00.715216
2014-06-12 14:46:58.339115 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2929 (0'0,173'2929] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2927 active] append_log  adding 1 keys
2014-06-12 14:46:58.339139 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2929, trimmed: 
2014-06-12 14:46:58.339174 7f9d693e3700  5 -- op tracker -- , seq: 748216, time: 2014-06-12 14:46:58.339171, event: started, op: osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.339183 7f9d693e3700  5 queue_transactions existing osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.339187 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.339188 7f9d693e3700  5 queue_op 0x58f9da0 seq 2643 osr(3.34f 0x276d480) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:58.339193 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.34f( v 173'2929 (0'0,173'2929] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2927 active] do_sub_op osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.339203 7f9d693e3700 10 osd.2 173 dequeue_op 0x31d60100 finish
2014-06-12 14:46:58.339199 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540e240 (1 active)
2014-06-12 14:46:58.339205 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.339207 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.339207 7f9d7915f700  5 _do_op 0x58f9da0 seq 2643 osr(3.34f 0x276d480)/0x276d480 start
2014-06-12 14:46:58.339211 7f9d7915f700 10 _do_transaction on 0x29dab70
2014-06-12 14:46:58.339217 7f9d7915f700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.339337 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.339360 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.339413 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.339505 7f9d7915f700 15 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.339519 7f9d7915f700 10 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1 r = 0
2014-06-12 14:46:58.339527 7f9d7915f700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.339548 7f9d7915f700 15 _write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 1630208~516096
2014-06-12 14:46:58.339687 7f9d7915f700 10 lookup_strip_header done  cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.339813 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:46:58.341038 7f9d7915f700 10 _generic_write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 1630208~516096 = 0
2014-06-12 14:46:58.341070 7f9d7915f700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.341078 7f9d7915f700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.341082 7f9d7915f700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.341086 7f9d7915f700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.341220 7f9d7915f700 20 set_header setting 16877 cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 parent seq 0
2014-06-12 14:46:58.341268 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.341280 7f9d7915f700 20 set_header setting 2162 cid meta oid 1fe1d62b/pglog_3.34f/0//-1 parent seq 0
2014-06-12 14:46:58.341286 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:46:58.344506 7f9d7915f700 10 _do_op 0x58f9da0 seq 2643 r = 0, finisher 0x10026d00 0
2014-06-12 14:46:58.344526 7f9d7915f700 10 _finish_op 0x58f9da0 seq 2643 osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.344536 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540e240 (0 active)
2014-06-12 14:46:58.344550 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.344567 7f9d7795c700  5 -- op tracker -- , seq: 748216, time: 2014-06-12 14:46:58.344556, event: sub_op_applied, op: osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.344577 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2929 (0'0,173'2929] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2927 active] sub_op_modify_applied on 0x29daa80 op osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.344612 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:58.344634 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2929 (0'0,173'2929] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2927 active] op_applied on primary on version 173'2929
2014-06-12 14:46:58.344649 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2929 (0'0,173'2929] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2927 active] op_applied on replica on version 173'2929
2014-06-12 14:46:58.344715 7f9d7715b700  5 -- op tracker -- , seq: 748216, time: 2014-06-12 14:46:58.344701, event: commit_sent, op: osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.344727 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2929 (0'0,173'2929] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2927 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:58.344750 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:58.344776 7f9d7715b700  5 -- op tracker -- , seq: 748216, time: 2014-06-12 14:46:58.344772, event: done, op: osd_sub_op(client.4306.0:144293 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2929 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.350302 7f9d58c5a700  5 -- op tracker -- , seq: 748217, time: 2014-06-12 14:46:58.349593, event: header_read, op: osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.350314 7f9d58c5a700  5 -- op tracker -- , seq: 748217, time: 2014-06-12 14:46:58.349596, event: throttled, op: osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.350330 7f9d58c5a700  5 -- op tracker -- , seq: 748217, time: 2014-06-12 14:46:58.350267, event: all_read, op: osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.350335 7f9d58c5a700  5 -- op tracker -- , seq: 748217, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.350340 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.350345 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:58.350352 7f9d58c5a700 15 osd.2 173 enqueue_op 0x68a7600 prio 127 cost 516903 latency 0.000759 osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.350382 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.350389 7f9d6abe6700 10 osd.2 173 dequeue_op 0x68a7600 prio 127 cost 516903 latency 0.000796 osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.34f( v 173'2929 (0'0,173'2929] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2928 active]
2014-06-12 14:46:58.350409 7f9d6abe6700  5 -- op tracker -- , seq: 748217, time: 2014-06-12 14:46:58.350406, event: reached_pg, op: osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.350416 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2929 (0'0,173'2929] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2928 active] handle_message: 0x68a7600
2014-06-12 14:46:58.350424 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2929 (0'0,173'2929] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2928 active] sub_op_modify trans 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 v 173'2930 (transaction) 174
2014-06-12 14:46:58.350435 7f9d6abe6700  5 -- op tracker -- , seq: 748217, time: 2014-06-12 14:46:58.350433, event: started, op: osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.350447 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2929 (0'0,173'2929] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2928 active] append_log log((0'0,173'2929], crt=0'0) [173'2930 (173'2929) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144294 2014-06-12 08:46:00.726643]
2014-06-12 14:46:58.350464 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2930 (0'0,173'2930] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2928 active] add_log_entry 173'2930 (173'2929) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144294 2014-06-12 08:46:00.726643
2014-06-12 14:46:58.350483 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2930 (0'0,173'2930] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2928 active] append_log  adding 1 keys
2014-06-12 14:46:58.350511 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2930, trimmed: 
2014-06-12 14:46:58.350533 7f9d6abe6700  5 -- op tracker -- , seq: 748217, time: 2014-06-12 14:46:58.350529, event: started, op: osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.350542 7f9d6abe6700  5 queue_transactions existing osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.350545 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.350546 7f9d6abe6700  5 queue_op 0x83ef680 seq 2644 osr(3.34f 0x276d480) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:58.350555 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540e240 (1 active)
2014-06-12 14:46:58.350562 7f9d8016d700  5 _do_op 0x83ef680 seq 2644 osr(3.34f 0x276d480)/0x276d480 start
2014-06-12 14:46:58.350551 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.34f( v 173'2930 (0'0,173'2930] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2928 active] do_sub_op osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.350565 7f9d6abe6700 10 osd.2 173 dequeue_op 0x68a7600 finish
2014-06-12 14:46:58.350566 7f9d8016d700 10 _do_transaction on 0x206ed5f0
2014-06-12 14:46:58.350568 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.350570 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.350573 7f9d8016d700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.350685 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.350710 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.350747 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.350765 7f9d8016d700 15 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.350770 7f9d8016d700 10 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1 r = 0
2014-06-12 14:46:58.350774 7f9d8016d700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.350879 7f9d8016d700 15 _write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 2146304~516096
2014-06-12 14:46:58.350974 7f9d8016d700 10 lookup_strip_header done  cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.351073 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:46:58.352297 7f9d8016d700 10 _generic_write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 2146304~516096 = 0
2014-06-12 14:46:58.352333 7f9d8016d700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.352343 7f9d8016d700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.352347 7f9d8016d700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.352351 7f9d8016d700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.352489 7f9d8016d700 20 set_header setting 16877 cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 parent seq 0
2014-06-12 14:46:58.352534 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.352544 7f9d8016d700 20 set_header setting 2162 cid meta oid 1fe1d62b/pglog_3.34f/0//-1 parent seq 0
2014-06-12 14:46:58.352550 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:46:58.354417 7f9d8016d700 10 _do_op 0x83ef680 seq 2644 r = 0, finisher 0x583fda0 0
2014-06-12 14:46:58.354433 7f9d8016d700 10 _finish_op 0x83ef680 seq 2644 osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.354438 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540e240 (0 active)
2014-06-12 14:46:58.354442 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.354450 7f9d7715b700  5 -- op tracker -- , seq: 748217, time: 2014-06-12 14:46:58.354441, event: commit_sent, op: osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.354466 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2930 (0'0,173'2930] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2928 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:58.354489 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:58.354520 7f9d7795c700  5 -- op tracker -- , seq: 748217, time: 2014-06-12 14:46:58.354513, event: sub_op_applied, op: osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.354531 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2930 (0'0,173'2930] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2929 active] sub_op_modify_applied on 0x206ed500 op osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.354550 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2930 (0'0,173'2930] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2929 active] op_applied on primary on version 173'2930
2014-06-12 14:46:58.354561 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2930 (0'0,173'2930] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2929 active] op_applied on replica on version 173'2930
2014-06-12 14:46:58.354585 7f9d7795c700  5 -- op tracker -- , seq: 748217, time: 2014-06-12 14:46:58.354580, event: done, op: osd_sub_op(client.4306.0:144294 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2930 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.360352 7f9d58c5a700  5 -- op tracker -- , seq: 748218, time: 2014-06-12 14:46:58.359540, event: header_read, op: osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.360371 7f9d58c5a700  5 -- op tracker -- , seq: 748218, time: 2014-06-12 14:46:58.359542, event: throttled, op: osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.360385 7f9d58c5a700  5 -- op tracker -- , seq: 748218, time: 2014-06-12 14:46:58.360316, event: all_read, op: osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.360390 7f9d58c5a700  5 -- op tracker -- , seq: 748218, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.360394 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.360399 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:58.360404 7f9d58c5a700 15 osd.2 173 enqueue_op 0x68a7000 prio 127 cost 516903 latency 0.000864 osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.360469 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.360480 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x68a7000 prio 127 cost 516903 latency 0.000939 osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.34f( v 173'2930 (0'0,173'2930] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2929 active]
2014-06-12 14:46:58.360500 7f9d6a3e5700  5 -- op tracker -- , seq: 748218, time: 2014-06-12 14:46:58.360496, event: reached_pg, op: osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.360510 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2930 (0'0,173'2930] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2929 active] handle_message: 0x68a7000
2014-06-12 14:46:58.360520 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2930 (0'0,173'2930] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2929 active] sub_op_modify trans 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 v 173'2931 (transaction) 174
2014-06-12 14:46:58.360532 7f9d6a3e5700  5 -- op tracker -- , seq: 748218, time: 2014-06-12 14:46:58.360529, event: started, op: osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.360546 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2930 (0'0,173'2930] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2929 active] append_log log((0'0,173'2930], crt=0'0) [173'2931 (173'2930) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144295 2014-06-12 08:46:00.736535]
2014-06-12 14:46:58.360564 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2931 (0'0,173'2931] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2929 active] add_log_entry 173'2931 (173'2930) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144295 2014-06-12 08:46:00.736535
2014-06-12 14:46:58.360583 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2931 (0'0,173'2931] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2929 active] append_log  adding 1 keys
2014-06-12 14:46:58.360607 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2931, trimmed: 
2014-06-12 14:46:58.360624 7f9d6a3e5700  5 -- op tracker -- , seq: 748218, time: 2014-06-12 14:46:58.360620, event: started, op: osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.360632 7f9d6a3e5700  5 queue_transactions existing osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.360635 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.360636 7f9d6a3e5700  5 queue_op 0x8cd3f20 seq 2645 osr(3.34f 0x276d480) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:58.360641 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.34f( v 173'2931 (0'0,173'2931] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2929 active] do_sub_op osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.360650 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x68a7000 finish
2014-06-12 14:46:58.360660 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.360662 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.360674 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540e240 (1 active)
2014-06-12 14:46:58.360690 7f9d79960700  5 _do_op 0x8cd3f20 seq 2645 osr(3.34f 0x276d480)/0x276d480 start
2014-06-12 14:46:58.360694 7f9d79960700 10 _do_transaction on 0x62ad430
2014-06-12 14:46:58.360699 7f9d79960700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.360786 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.360804 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.360839 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.360851 7f9d79960700 15 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.360856 7f9d79960700 10 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1 r = 0
2014-06-12 14:46:58.360860 7f9d79960700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.360868 7f9d79960700 15 _write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 2662400~516096
2014-06-12 14:46:58.360957 7f9d79960700 10 lookup_strip_header done  cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.361034 7f9d79960700 10 file_to_extents done 
2014-06-12 14:46:58.362253 7f9d79960700 10 _generic_write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 2662400~516096 = 0
2014-06-12 14:46:58.362287 7f9d79960700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.362295 7f9d79960700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.362299 7f9d79960700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.362303 7f9d79960700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.362437 7f9d79960700 20 set_header setting 16877 cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 parent seq 0
2014-06-12 14:46:58.362476 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.362490 7f9d79960700 20 set_header setting 2162 cid meta oid 1fe1d62b/pglog_3.34f/0//-1 parent seq 0
2014-06-12 14:46:58.362498 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:46:58.363963 7f9d79960700 10 _do_op 0x8cd3f20 seq 2645 r = 0, finisher 0x7ffdbe0 0
2014-06-12 14:46:58.363975 7f9d79960700 10 _finish_op 0x8cd3f20 seq 2645 osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.363980 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540e240 (0 active)
2014-06-12 14:46:58.363986 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.363998 7f9d7715b700  5 -- op tracker -- , seq: 748218, time: 2014-06-12 14:46:58.363988, event: commit_sent, op: osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.364011 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2931 (0'0,173'2931] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2929 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:58.364044 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:58.364085 7f9d7795c700  5 -- op tracker -- , seq: 748218, time: 2014-06-12 14:46:58.364071, event: sub_op_applied, op: osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.364096 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2931 (0'0,173'2931] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2930 active] sub_op_modify_applied on 0x62ad340 op osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.364127 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2931 (0'0,173'2931] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2930 active] op_applied on primary on version 173'2931
2014-06-12 14:46:58.364142 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2931 (0'0,173'2931] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2930 active] op_applied on replica on version 173'2931
2014-06-12 14:46:58.364174 7f9d7795c700  5 -- op tracker -- , seq: 748218, time: 2014-06-12 14:46:58.364170, event: done, op: osd_sub_op(client.4306.0:144295 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2931 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.369804 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:46:58.370375 7f9d58c5a700  5 -- op tracker -- , seq: 748219, time: 2014-06-12 14:46:58.369395, event: header_read, op: osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.370397 7f9d58c5a700  5 -- op tracker -- , seq: 748219, time: 2014-06-12 14:46:58.369397, event: throttled, op: osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.370414 7f9d58c5a700  5 -- op tracker -- , seq: 748219, time: 2014-06-12 14:46:58.370343, event: all_read, op: osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.370423 7f9d58c5a700  5 -- op tracker -- , seq: 748219, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.370428 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.370435 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:58.370443 7f9d58c5a700 15 osd.2 173 enqueue_op 0x68a7e00 prio 127 cost 516903 latency 0.001048 osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.370471 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.370481 7f9d66bde700 10 osd.2 173 dequeue_op 0x68a7e00 prio 127 cost 516903 latency 0.001085 osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.34f( v 173'2931 (0'0,173'2931] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2930 active]
2014-06-12 14:46:58.370524 7f9d66bde700  5 -- op tracker -- , seq: 748219, time: 2014-06-12 14:46:58.370512, event: reached_pg, op: osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.370533 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2931 (0'0,173'2931] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2930 active] handle_message: 0x68a7e00
2014-06-12 14:46:58.370544 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2931 (0'0,173'2931] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2930 active] sub_op_modify trans 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 v 173'2932 (transaction) 174
2014-06-12 14:46:58.370559 7f9d66bde700  5 -- op tracker -- , seq: 748219, time: 2014-06-12 14:46:58.370556, event: started, op: osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.370574 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2931 (0'0,173'2931] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2930 active] append_log log((0'0,173'2931], crt=0'0) [173'2932 (173'2931) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144296 2014-06-12 08:46:00.746482]
2014-06-12 14:46:58.370599 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2932 (0'0,173'2932] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2930 active] add_log_entry 173'2932 (173'2931) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144296 2014-06-12 08:46:00.746482
2014-06-12 14:46:58.370626 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2932 (0'0,173'2932] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2930 active] append_log  adding 1 keys
2014-06-12 14:46:58.370658 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2932, trimmed: 
2014-06-12 14:46:58.370682 7f9d66bde700  5 -- op tracker -- , seq: 748219, time: 2014-06-12 14:46:58.370677, event: started, op: osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.370693 7f9d66bde700  5 queue_transactions existing osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.370697 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.370699 7f9d66bde700  5 queue_op 0x68ea420 seq 2646 osr(3.34f 0x276d480) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:58.370705 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.34f( v 173'2932 (0'0,173'2932] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2930 active] do_sub_op osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.370719 7f9d66bde700 10 osd.2 173 dequeue_op 0x68a7e00 finish
2014-06-12 14:46:58.370722 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.370725 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.370737 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540e240 (1 active)
2014-06-12 14:46:58.370743 7f9d7d968700  5 _do_op 0x68ea420 seq 2646 osr(3.34f 0x276d480)/0x276d480 start
2014-06-12 14:46:58.370746 7f9d7d968700 10 _do_transaction on 0x2d2c1eb0
2014-06-12 14:46:58.370751 7f9d7d968700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.370850 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.370870 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.370906 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.370987 7f9d7d968700 15 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.370992 7f9d7d968700 10 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1 r = 0
2014-06-12 14:46:58.370996 7f9d7d968700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.371009 7f9d7d968700 15 _write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 3178496~516096
2014-06-12 14:46:58.371100 7f9d7d968700 10 lookup_strip_header done  cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.371217 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:46:58.372472 7f9d7d968700 10 _generic_write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 3178496~516096 = 0
2014-06-12 14:46:58.372501 7f9d7d968700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.372510 7f9d7d968700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.372514 7f9d7d968700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.372518 7f9d7d968700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.372653 7f9d7d968700 20 set_header setting 16877 cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 parent seq 0
2014-06-12 14:46:58.372692 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.372702 7f9d7d968700 20 set_header setting 2162 cid meta oid 1fe1d62b/pglog_3.34f/0//-1 parent seq 0
2014-06-12 14:46:58.372707 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:46:58.374927 7f9d7d968700 10 _do_op 0x68ea420 seq 2646 r = 0, finisher 0x291b620 0
2014-06-12 14:46:58.374951 7f9d7d968700 10 _finish_op 0x68ea420 seq 2646 osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.374965 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540e240 (0 active)
2014-06-12 14:46:58.374980 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.374981 7f9d7715b700  5 -- op tracker -- , seq: 748219, time: 2014-06-12 14:46:58.374965, event: commit_sent, op: osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.375010 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2932 (0'0,173'2932] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2930 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:58.375052 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:58.375128 7f9d7795c700  5 -- op tracker -- , seq: 748219, time: 2014-06-12 14:46:58.375111, event: sub_op_applied, op: osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.375139 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2932 (0'0,173'2932] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2931 active] sub_op_modify_applied on 0x2d2c1dc0 op osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.375151 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2932 (0'0,173'2932] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2931 active] op_applied on primary on version 173'2932
2014-06-12 14:46:58.375158 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2932 (0'0,173'2932] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2931 active] op_applied on replica on version 173'2932
2014-06-12 14:46:58.375173 7f9d7795c700  5 -- op tracker -- , seq: 748219, time: 2014-06-12 14:46:58.375171, event: done, op: osd_sub_op(client.4306.0:144296 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2932 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.378283 7f9d58c5a700  5 -- op tracker -- , seq: 748220, time: 2014-06-12 14:46:58.378167, event: header_read, op: osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.378311 7f9d58c5a700  5 -- op tracker -- , seq: 748220, time: 2014-06-12 14:46:58.378171, event: throttled, op: osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.378321 7f9d58c5a700  5 -- op tracker -- , seq: 748220, time: 2014-06-12 14:46:58.378224, event: all_read, op: osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.378336 7f9d58c5a700  5 -- op tracker -- , seq: 748220, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.378341 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.378348 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:58.378354 7f9d58c5a700 15 osd.2 173 enqueue_op 0x2a902300 prio 127 cost 4903 latency 0.000187 osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.378385 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.378395 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a902300 prio 127 cost 4903 latency 0.000228 osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.34f( v 173'2932 (0'0,173'2932] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2931 active]
2014-06-12 14:46:58.378420 7f9d64bda700  5 -- op tracker -- , seq: 748220, time: 2014-06-12 14:46:58.378415, event: reached_pg, op: osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.378433 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2932 (0'0,173'2932] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2931 active] handle_message: 0x2a902300
2014-06-12 14:46:58.378445 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2932 (0'0,173'2932] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2931 active] sub_op_modify trans 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 v 173'2933 (transaction) 174
2014-06-12 14:46:58.378463 7f9d64bda700  5 -- op tracker -- , seq: 748220, time: 2014-06-12 14:46:58.378458, event: started, op: osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.378479 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2932 (0'0,173'2932] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2931 active] append_log log((0'0,173'2932], crt=0'0) [173'2933 (173'2932) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144297 2014-06-12 08:46:00.757740]
2014-06-12 14:46:58.378505 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2933 (0'0,173'2933] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2931 active] add_log_entry 173'2933 (173'2932) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144297 2014-06-12 08:46:00.757740
2014-06-12 14:46:58.378533 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2933 (0'0,173'2933] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2931 active] append_log  adding 1 keys
2014-06-12 14:46:58.378568 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2933, trimmed: 
2014-06-12 14:46:58.378620 7f9d64bda700  5 -- op tracker -- , seq: 748220, time: 2014-06-12 14:46:58.378614, event: started, op: osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.378643 7f9d64bda700  5 queue_transactions existing osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.378648 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.378650 7f9d64bda700  5 queue_op 0x663f0e0 seq 2647 osr(3.34f 0x276d480) 6389 bytes   (queue has 1 ops and 6389 bytes)
2014-06-12 14:46:58.378666 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.34f( v 173'2933 (0'0,173'2933] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2931 active] do_sub_op osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.378683 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a902300 finish
2014-06-12 14:46:58.378687 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.378690 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.378698 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540e240 (1 active)
2014-06-12 14:46:58.378711 7f9d7c165700  5 _do_op 0x663f0e0 seq 2647 osr(3.34f 0x276d480)/0x276d480 start
2014-06-12 14:46:58.378717 7f9d7c165700 10 _do_transaction on 0x128c97b0
2014-06-12 14:46:58.378724 7f9d7c165700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.378836 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.378860 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.378894 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.378976 7f9d7c165700 15 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.378982 7f9d7c165700 10 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1 r = 0
2014-06-12 14:46:58.378986 7f9d7c165700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.378998 7f9d7c165700 15 _write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 3694592~4096
2014-06-12 14:46:58.379089 7f9d7c165700 10 lookup_strip_header done  cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.379096 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:46:58.379191 7f9d7c165700 10 _generic_write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 3694592~4096 = 0
2014-06-12 14:46:58.379198 7f9d7c165700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.379207 7f9d7c165700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.379211 7f9d7c165700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.379216 7f9d7c165700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.379287 7f9d7c165700 20 set_header setting 16877 cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 parent seq 0
2014-06-12 14:46:58.379301 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.379309 7f9d7c165700 20 set_header setting 2162 cid meta oid 1fe1d62b/pglog_3.34f/0//-1 parent seq 0
2014-06-12 14:46:58.379315 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:46:58.379369 7f9d7c165700 10 _do_op 0x663f0e0 seq 2647 r = 0, finisher 0x80c2d80 0
2014-06-12 14:46:58.379375 7f9d7c165700 10 _finish_op 0x663f0e0 seq 2647 osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.379378 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540e240 (0 active)
2014-06-12 14:46:58.379380 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.379408 7f9d7715b700  5 -- op tracker -- , seq: 748220, time: 2014-06-12 14:46:58.379403, event: commit_sent, op: osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.379417 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2933 (0'0,173'2933] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2931 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:58.379431 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:58.379522 7f9d7795c700  5 -- op tracker -- , seq: 748220, time: 2014-06-12 14:46:58.379512, event: sub_op_applied, op: osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.379542 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2933 (0'0,173'2933] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active] sub_op_modify_applied on 0x128c96c0 op osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.379566 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2933 (0'0,173'2933] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active] op_applied on primary on version 173'2933
2014-06-12 14:46:58.379581 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2933 (0'0,173'2933] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active] op_applied on replica on version 173'2933
2014-06-12 14:46:58.379596 7f9d7795c700  5 -- op tracker -- , seq: 748220, time: 2014-06-12 14:46:58.379593, event: done, op: osd_sub_op(client.4306.0:144297 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2933 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.384726 7f9d58c5a700  5 -- op tracker -- , seq: 748221, time: 2014-06-12 14:46:58.383705, event: header_read, op: osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.384766 7f9d58c5a700  5 -- op tracker -- , seq: 748221, time: 2014-06-12 14:46:58.383708, event: throttled, op: osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.384779 7f9d58c5a700  5 -- op tracker -- , seq: 748221, time: 2014-06-12 14:46:58.384668, event: all_read, op: osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.384790 7f9d58c5a700  5 -- op tracker -- , seq: 748221, time: 0.000000, event: dispatched, op: osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.384804 7f9d58c5a700 10 osd.2 173 handle_replica_op osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.384812 7f9d58c5a700 20 osd.2 173 should_share_map osd.27 172.16.96.202:6855/4073 173
2014-06-12 14:46:58.384819 7f9d58c5a700 15 osd.2 173 enqueue_op 0x34017a00 prio 127 cost 496423 latency 0.001114 osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.384876 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.384887 7f9d69be4700 10 osd.2 173 dequeue_op 0x34017a00 prio 127 cost 496423 latency 0.001181 osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.34f( v 173'2933 (0'0,173'2933] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active]
2014-06-12 14:46:58.384923 7f9d69be4700  5 -- op tracker -- , seq: 748221, time: 2014-06-12 14:46:58.384916, event: reached_pg, op: osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.384941 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2933 (0'0,173'2933] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active] handle_message: 0x34017a00
2014-06-12 14:46:58.384948 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2933 (0'0,173'2933] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active] sub_op_modify trans 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 v 173'2934 (transaction) 174
2014-06-12 14:46:58.384957 7f9d69be4700  5 -- op tracker -- , seq: 748221, time: 2014-06-12 14:46:58.384955, event: started, op: osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.384966 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2933 (0'0,173'2933] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active] append_log log((0'0,173'2933], crt=0'0) [173'2934 (173'2933) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144298 2014-06-12 08:46:00.761278]
2014-06-12 14:46:58.384982 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2934 (0'0,173'2934] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active] add_log_entry 173'2934 (173'2933) modify   4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 by client.4306.0:144298 2014-06-12 08:46:00.761278
2014-06-12 14:46:58.384997 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2934 (0'0,173'2934] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active] append_log  adding 1 keys
2014-06-12 14:46:58.385017 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2934, trimmed: 
2014-06-12 14:46:58.385051 7f9d69be4700  5 -- op tracker -- , seq: 748221, time: 2014-06-12 14:46:58.385048, event: started, op: osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.385057 7f9d69be4700  5 queue_transactions existing osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.385059 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.385060 7f9d69be4700  5 queue_op 0x2e7c300 seq 2648 osr(3.34f 0x276d480) 497909 bytes   (queue has 1 ops and 497909 bytes)
2014-06-12 14:46:58.385064 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.34f( v 173'2934 (0'0,173'2934] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active] do_sub_op osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.385073 7f9d69be4700 10 osd.2 173 dequeue_op 0x34017a00 finish
2014-06-12 14:46:58.385075 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.385084 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.385097 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540e240 (1 active)
2014-06-12 14:46:58.385104 7f9d8096e700  5 _do_op 0x2e7c300 seq 2648 osr(3.34f 0x276d480)/0x276d480 start
2014-06-12 14:46:58.385115 7f9d8096e700 10 _do_transaction on 0x277562b0
2014-06-12 14:46:58.385119 7f9d8096e700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.385179 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.385193 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.385239 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.385251 7f9d8096e700 15 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.385261 7f9d8096e700 10 _omap_rmkeys meta/1fe1d62b/pglog_3.34f/0//-1 r = 0
2014-06-12 14:46:58.385265 7f9d8096e700 15 _omap_setkeys meta/1fe1d62b/pglog_3.34f/0//-1
2014-06-12 14:46:58.385275 7f9d8096e700 15 _write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 3698688~495616
2014-06-12 14:46:58.385358 7f9d8096e700 10 lookup_strip_header done  cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.385373 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:46:58.386477 7f9d8096e700 10 _generic_write 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 3698688~495616 = 0
2014-06-12 14:46:58.386509 7f9d8096e700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.386517 7f9d8096e700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.386522 7f9d8096e700 15 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3
2014-06-12 14:46:58.386526 7f9d8096e700 10 _setattrs 3.34f_head/4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 = 0
2014-06-12 14:46:58.386641 7f9d8096e700 20 set_header setting 16877 cid 3.34f_head oid 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 parent seq 0
2014-06-12 14:46:58.386679 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.386688 7f9d8096e700 20 set_header setting 2162 cid meta oid 1fe1d62b/pglog_3.34f/0//-1 parent seq 0
2014-06-12 14:46:58.386694 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:46:58.388062 7f9d8096e700 10 _do_op 0x2e7c300 seq 2648 r = 0, finisher 0x64f9940 0
2014-06-12 14:46:58.388070 7f9d8096e700 10 _finish_op 0x2e7c300 seq 2648 osr(3.34f 0x276d480)/0x276d480
2014-06-12 14:46:58.388074 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540e240 (0 active)
2014-06-12 14:46:58.388077 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.388134 7f9d7795c700  5 -- op tracker -- , seq: 748221, time: 2014-06-12 14:46:58.388127, event: sub_op_applied, op: osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.388153 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2934 (0'0,173'2934] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active] sub_op_modify_applied on 0x277561c0 op osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.388172 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:58.388186 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2934 (0'0,173'2934] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active] op_applied on primary on version 173'2934
2014-06-12 14:46:58.388195 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2934 (0'0,173'2934] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active] op_applied on replica on version 173'2934
2014-06-12 14:46:58.388256 7f9d7715b700  5 -- op tracker -- , seq: 748221, time: 2014-06-12 14:46:58.388236, event: commit_sent, op: osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.388282 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.34f( v 173'2934 (0'0,173'2934] local-les=156 n=308 ec=23 les/c 156/156 155/155/155) [27,2] r=1 lpr=155 pi=23-154/12 luod=0'0 crt=0'0 lcod 173'2932 active] sub_op_modify_commit on op osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.27
2014-06-12 14:46:58.388314 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:46:58.388362 7f9d7715b700  5 -- op tracker -- , seq: 748221, time: 2014-06-12 14:46:58.388352, event: done, op: osd_sub_op(client.4306.0:144298 3.34f 4354034f/rbd_data.10a96b8b4567.0000000000000fb8/head//3 [] v 173'2934 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.480355 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:58.480378 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:46:58.480380 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:58.480381 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:58.480383 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 24; oldest is 25.307058 seconds old
2014-06-12 14:46:58.480382 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:58.701651 7f9d56432700  5 -- op tracker -- , seq: 748222, time: 2014-06-12 14:46:58.701557, event: header_read, op: osd_sub_op_reply(client.4293.0:144175 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.701670 7f9d56432700  5 -- op tracker -- , seq: 748222, time: 2014-06-12 14:46:58.701561, event: throttled, op: osd_sub_op_reply(client.4293.0:144175 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.701687 7f9d56432700  5 -- op tracker -- , seq: 748222, time: 2014-06-12 14:46:58.701580, event: all_read, op: osd_sub_op_reply(client.4293.0:144175 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.701692 7f9d56432700  5 -- op tracker -- , seq: 748222, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144175 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.701698 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144175 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:46:58.701703 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:58.701711 7f9d56432700 15 osd.2 173 enqueue_op 0x59ddf00 prio 196 cost 0 latency 0.000154 osd_sub_op_reply(client.4293.0:144175 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0) v2
2014-06-12 14:46:58.701751 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.701761 7f9d653db700 10 osd.2 173 dequeue_op 0x59ddf00 prio 196 cost 0 latency 0.000203 osd_sub_op_reply(client.4293.0:144175 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0) v2 pg pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2814 active+clean]
2014-06-12 14:46:58.701786 7f9d653db700  5 -- op tracker -- , seq: 748222, time: 2014-06-12 14:46:58.701782, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144175 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.701798 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2814 active+clean] handle_message: 0x59ddf00
2014-06-12 14:46:58.701814 7f9d653db700  5 -- op tracker -- , seq: 748222, time: 2014-06-12 14:46:58.701810, event: started, op: osd_sub_op_reply(client.4293.0:144175 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.701820 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2814 active+clean] sub_op_modify_reply: tid 236278 op  ack_type 4 from 26
2014-06-12 14:46:58.701845 7f9d653db700  5 -- op tracker -- , seq: 747887, time: 2014-06-12 14:46:58.701834, event: sub_op_commit_rec, op: osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:58.701861 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2814 active+clean] repop_all_applied: repop tid 236278 all applied 
2014-06-12 14:46:58.701873 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2814 active+clean] eval_repop repgather(0x1a445200 173'2816 rep_tid=236278 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:58.701888 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2814 active+clean]  sending ack on repgather(0x1a445200 173'2816 rep_tid=236278 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) 0xc0c8a00
2014-06-12 14:46:58.701926 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2815 crt=0'0 lcod 173'2814 mlcod 173'2814 active+clean] repop_all_committed: repop tid 236278 all committed 
2014-06-12 14:46:58.701938 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2814 active+clean] eval_repop repgather(0x1a445200 173'2816 rep_tid=236278 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:46:58.701960 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2814 active+clean] log_op_stats osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4 inb 2064663 outb 0 rlat 25.528599 lat 25.528633
2014-06-12 14:46:58.701985 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2814 active+clean] publish_stats_to_osd 173:3141
2014-06-12 14:46:58.702005 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2814 active+clean]  sending commit on repgather(0x1a445200 173'2816 rep_tid=236278 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) 0x1d54f180
2014-06-12 14:46:58.702041 7f9d653db700  5 -- op tracker -- , seq: 747887, time: 2014-06-12 14:46:58.702035, event: commit_sent, op: osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:58.702051 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2814 active+clean]  requeue_ops 
2014-06-12 14:46:58.702061 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2815 active+clean]  removing repgather(0x1a445200 173'2816 rep_tid=236278 committed?=1 applied?=1 op=osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:58.702078 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2815 active+clean]    q front is repgather(0x1a445200 173'2816 rep_tid=236278 committed?=1 applied?=1 op=osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:58.702094 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2815 active+clean] remove_repop repgather(0x1a445200 173'2816 rep_tid=236278 committed?=1 applied?=1 op=osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:46:58.702110 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2815 active+clean]  requeue_ops 
2014-06-12 14:46:58.702131 7f9d653db700  5 -- op tracker -- , seq: 747887, time: 2014-06-12 14:46:58.702126, event: done, op: osd_op(client.4293.0:144175 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 49152~2064384] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:46:58.702153 7f9d653db700 10 osd.2 173 dequeue_op 0x59ddf00 finish
2014-06-12 14:46:58.702162 7f9d653db700  5 -- op tracker -- , seq: 748222, time: 2014-06-12 14:46:58.702159, event: done, op: osd_sub_op_reply(client.4293.0:144175 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:46:58.702173 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.702175 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.802661 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:58.802718 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:58.809910 7f9d56432700  5 -- op tracker -- , seq: 748223, time: 2014-06-12 14:46:58.808896, event: header_read, op: osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.809925 7f9d56432700  5 -- op tracker -- , seq: 748223, time: 2014-06-12 14:46:58.808898, event: throttled, op: osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.809948 7f9d56432700  5 -- op tracker -- , seq: 748223, time: 2014-06-12 14:46:58.809872, event: all_read, op: osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.809958 7f9d56432700  5 -- op tracker -- , seq: 748223, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.809963 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.809970 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:46:58.809979 7f9d56432700 15 osd.2 173 enqueue_op 0x59dc600 prio 127 cost 516903 latency 0.001082 osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.810012 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.810019 7f9d623d5700 10 osd.2 173 dequeue_op 0x59dc600 prio 127 cost 516903 latency 0.001122 osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2646 (0'0,173'2646] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active]
2014-06-12 14:46:58.810040 7f9d623d5700  5 -- op tracker -- , seq: 748223, time: 2014-06-12 14:46:58.810036, event: reached_pg, op: osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.810049 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2646 (0'0,173'2646] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active] handle_message: 0x59dc600
2014-06-12 14:46:58.810061 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2646 (0'0,173'2646] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active] sub_op_modify trans af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 v 173'2647 (transaction) 174
2014-06-12 14:46:58.810078 7f9d623d5700  5 -- op tracker -- , seq: 748223, time: 2014-06-12 14:46:58.810074, event: started, op: osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.810093 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2646 (0'0,173'2646] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active] append_log log((0'0,173'2646], crt=0'0) [173'2647 (173'2646) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147306 2014-06-12 08:46:01.185972]
2014-06-12 14:46:58.810116 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2647 (0'0,173'2647] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active] add_log_entry 173'2647 (173'2646) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147306 2014-06-12 08:46:01.185972
2014-06-12 14:46:58.810138 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2647 (0'0,173'2647] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active] append_log  adding 1 keys
2014-06-12 14:46:58.810170 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2647, trimmed: 
2014-06-12 14:46:58.810192 7f9d623d5700  5 -- op tracker -- , seq: 748223, time: 2014-06-12 14:46:58.810187, event: started, op: osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.810201 7f9d623d5700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:58.810205 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.810207 7f9d623d5700  5 queue_op 0x575f7a0 seq 2467 osr(3.1c7 0x276d170) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:46:58.810216 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:46:58.810222 7f9d7c966700  5 _do_op 0x575f7a0 seq 2467 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:46:58.810225 7f9d7c966700 10 _do_transaction on 0x1451c0f0
2014-06-12 14:46:58.810214 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2647 (0'0,173'2647] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active] do_sub_op osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.810229 7f9d623d5700 10 osd.2 173 dequeue_op 0x59dc600 finish
2014-06-12 14:46:58.810230 7f9d7c966700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:58.810232 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.810235 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.810447 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:58.810473 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.810513 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.810531 7f9d7c966700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:58.810534 7f9d7c966700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:46:58.810538 7f9d7c966700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:46:58.810549 7f9d7c966700 15 _write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 1417216~516096
2014-06-12 14:46:58.810654 7f9d7c966700 10 lookup_strip_header done  cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:46:58.810877 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:46:58.812174 7f9d7c966700 10 _generic_write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 1417216~516096 = 0
2014-06-12 14:46:58.812221 7f9d7c966700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:46:58.812231 7f9d7c966700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:46:58.812235 7f9d7c966700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:46:58.812240 7f9d7c966700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:46:58.812382 7f9d7c966700 20 set_header setting 17479 cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 parent seq 0
2014-06-12 14:46:58.812436 7f9d7c966700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:46:58.812448 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.812453 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:46:58.814614 7f9d7c966700 10 _do_op 0x575f7a0 seq 2467 r = 0, finisher 0x960c540 0
2014-06-12 14:46:58.814636 7f9d7c966700 10 _finish_op 0x575f7a0 seq 2467 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:46:58.814641 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:46:58.814648 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.814660 7f9d7795c700  5 -- op tracker -- , seq: 748223, time: 2014-06-12 14:46:58.814649, event: sub_op_applied, op: osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.814667 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2647 (0'0,173'2647] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active] sub_op_modify_applied on 0x1451c000 op osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.814690 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:58.814719 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2647 (0'0,173'2647] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active] op_applied on primary on version 173'2647
2014-06-12 14:46:58.814742 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2647 (0'0,173'2647] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active] op_applied on replica on version 173'2647
2014-06-12 14:46:58.814779 7f9d7715b700  5 -- op tracker -- , seq: 748223, time: 2014-06-12 14:46:58.814774, event: commit_sent, op: osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.814788 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2647 (0'0,173'2647] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2645 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:46:58.814801 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:58.814833 7f9d7715b700  5 -- op tracker -- , seq: 748223, time: 2014-06-12 14:46:58.814827, event: done, op: osd_sub_op(client.4316.0:147306 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2647 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.828286 7f9d5501e700  5 -- op tracker -- , seq: 748224, time: 2014-06-12 14:46:58.827997, event: header_read, op: osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.828302 7f9d5501e700  5 -- op tracker -- , seq: 748224, time: 2014-06-12 14:46:58.828001, event: throttled, op: osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.828318 7f9d5501e700  5 -- op tracker -- , seq: 748224, time: 2014-06-12 14:46:58.828236, event: all_read, op: osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.828324 7f9d5501e700  5 -- op tracker -- , seq: 748224, time: 0.000000, event: dispatched, op: osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.828328 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.828334 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:58.828340 7f9d5501e700 15 osd.2 173 enqueue_op 0x329ebe00 prio 127 cost 90919 latency 0.000343 osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.828369 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.828377 7f9d68be2700 10 osd.2 173 dequeue_op 0x329ebe00 prio 127 cost 90919 latency 0.000379 osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.169( v 173'2866 (0'0,173'2866] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2865 active]
2014-06-12 14:46:58.828394 7f9d68be2700  5 -- op tracker -- , seq: 748224, time: 2014-06-12 14:46:58.828391, event: reached_pg, op: osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.828404 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2866 (0'0,173'2866] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2865 active] handle_message: 0x329ebe00
2014-06-12 14:46:58.828412 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2866 (0'0,173'2866] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2865 active] sub_op_modify trans 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 v 173'2867 (transaction) 174
2014-06-12 14:46:58.828422 7f9d68be2700  5 -- op tracker -- , seq: 748224, time: 2014-06-12 14:46:58.828420, event: started, op: osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.828443 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2866 (0'0,173'2866] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2865 active] append_log log((0'0,173'2866], crt=0'0) [173'2867 (173'845) modify   99028969/rbd_data.10966b8b4567.0000000000001096/head//3 by client.4287.0:141320 2014-06-12 08:46:01.175119]
2014-06-12 14:46:58.828468 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2867 (0'0,173'2867] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2865 active] add_log_entry 173'2867 (173'845) modify   99028969/rbd_data.10966b8b4567.0000000000001096/head//3 by client.4287.0:141320 2014-06-12 08:46:01.175119
2014-06-12 14:46:58.828496 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2867 (0'0,173'2867] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2865 active] append_log  adding 1 keys
2014-06-12 14:46:58.828537 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2867, trimmed: 
2014-06-12 14:46:58.828563 7f9d68be2700  5 -- op tracker -- , seq: 748224, time: 2014-06-12 14:46:58.828559, event: started, op: osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.828573 7f9d68be2700  5 queue_transactions existing osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:46:58.828576 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.828577 7f9d68be2700  5 queue_op 0x6fcc480 seq 2605 osr(3.169 0x276ce50) 92405 bytes   (queue has 1 ops and 92405 bytes)
2014-06-12 14:46:58.828585 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54f1200 (1 active)
2014-06-12 14:46:58.828591 7f9d81970700  5 _do_op 0x6fcc480 seq 2605 osr(3.169 0x276ce50)/0x276ce50 start
2014-06-12 14:46:58.828582 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.169( v 173'2867 (0'0,173'2867] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2865 active] do_sub_op osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.828596 7f9d81970700 10 _do_transaction on 0x338c6b70
2014-06-12 14:46:58.828596 7f9d68be2700 10 osd.2 173 dequeue_op 0x329ebe00 finish
2014-06-12 14:46:58.828599 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.828601 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.828602 7f9d81970700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:46:58.828719 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:46:58.828745 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.828786 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.828875 7f9d81970700 15 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:46:58.828881 7f9d81970700 10 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1 r = 0
2014-06-12 14:46:58.828885 7f9d81970700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:46:58.828897 7f9d81970700 15 _write 3.169_head/99028969/rbd_data.10966b8b4567.0000000000001096/head//3 0~90112
2014-06-12 14:46:58.841708 7f9d81970700 10 lookup_strip_header done  cid 3.169_head oid 99028969/rbd_data.10966b8b4567.0000000000001096/head//3
2014-06-12 14:46:58.841868 7f9d81970700 10 file_to_extents done 
2014-06-12 14:46:58.842115 7f9d81970700 10 _generic_write 3.169_head/99028969/rbd_data.10966b8b4567.0000000000001096/head//3 0~90112 = 0
2014-06-12 14:46:58.842134 7f9d81970700 15 _setattrs 3.169_head/99028969/rbd_data.10966b8b4567.0000000000001096/head//3
2014-06-12 14:46:58.842146 7f9d81970700 10 _setattrs 3.169_head/99028969/rbd_data.10966b8b4567.0000000000001096/head//3 = 0
2014-06-12 14:46:58.842154 7f9d81970700 15 _setattrs 3.169_head/99028969/rbd_data.10966b8b4567.0000000000001096/head//3
2014-06-12 14:46:58.842164 7f9d81970700 10 _setattrs 3.169_head/99028969/rbd_data.10966b8b4567.0000000000001096/head//3 = 0
2014-06-12 14:46:58.842256 7f9d81970700 20 set_header setting 17990 cid 3.169_head oid 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 parent seq 0
2014-06-12 14:46:58.842286 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.842330 7f9d81970700 20 set_header setting 2306 cid meta oid 1fe2acfa/pglog_3.169/0//-1 parent seq 0
2014-06-12 14:46:58.842344 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:46:58.842693 7f9d81970700 10 _do_op 0x6fcc480 seq 2605 r = 0, finisher 0x29f7c440 0
2014-06-12 14:46:58.842705 7f9d81970700 10 _finish_op 0x6fcc480 seq 2605 osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:46:58.842722 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54f1200 (0 active)
2014-06-12 14:46:58.842728 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.842722 7f9d7715b700  5 -- op tracker -- , seq: 748224, time: 2014-06-12 14:46:58.842712, event: commit_sent, op: osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.842739 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2867 (0'0,173'2867] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2865 active] sub_op_modify_commit on op osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:58.842765 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:58.842801 7f9d7795c700  5 -- op tracker -- , seq: 748224, time: 2014-06-12 14:46:58.842794, event: sub_op_applied, op: osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.842810 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2867 (0'0,173'2867] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2866 active] sub_op_modify_applied on 0x338c6a80 op osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.842827 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2867 (0'0,173'2867] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2866 active] op_applied on primary on version 173'2867
2014-06-12 14:46:58.842835 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2867 (0'0,173'2867] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2866 active] op_applied on replica on version 173'2867
2014-06-12 14:46:58.842851 7f9d7795c700  5 -- op tracker -- , seq: 748224, time: 2014-06-12 14:46:58.842848, event: done, op: osd_sub_op(client.4287.0:141320 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.891704 7f9d5501e700  5 -- op tracker -- , seq: 748225, time: 2014-06-12 14:46:58.884357, event: header_read, op: osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.891720 7f9d5501e700  5 -- op tracker -- , seq: 748225, time: 2014-06-12 14:46:58.884360, event: throttled, op: osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.891729 7f9d5501e700  5 -- op tracker -- , seq: 748225, time: 2014-06-12 14:46:58.891656, event: all_read, op: osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.891737 7f9d5501e700  5 -- op tracker -- , seq: 748225, time: 0.000000, event: dispatched, op: osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.891741 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:46:58.891748 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:46:58.891756 7f9d5501e700 15 osd.2 173 enqueue_op 0x329ea100 prio 127 cost 4104999 latency 0.007399 osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.891785 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:46:58.891793 7f9d643d9700 10 osd.2 173 dequeue_op 0x329ea100 prio 127 cost 4104999 latency 0.007436 osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.169( v 173'2867 (0'0,173'2867] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2866 active]
2014-06-12 14:46:58.891821 7f9d643d9700  5 -- op tracker -- , seq: 748225, time: 2014-06-12 14:46:58.891816, event: reached_pg, op: osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.891830 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2867 (0'0,173'2867] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2866 active] handle_message: 0x329ea100
2014-06-12 14:46:58.891845 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2867 (0'0,173'2867] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2866 active] sub_op_modify trans 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 v 173'2868 (transaction) 174
2014-06-12 14:46:58.891861 7f9d643d9700  5 -- op tracker -- , seq: 748225, time: 2014-06-12 14:46:58.891857, event: started, op: osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.891874 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2867 (0'0,173'2867] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2866 active] append_log log((0'0,173'2867], crt=0'0) [173'2868 (173'2867) modify   99028969/rbd_data.10966b8b4567.0000000000001096/head//3 by client.4287.0:141321 2014-06-12 08:46:01.245078]
2014-06-12 14:46:58.891898 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2868 (0'0,173'2868] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2866 active] add_log_entry 173'2868 (173'2867) modify   99028969/rbd_data.10966b8b4567.0000000000001096/head//3 by client.4287.0:141321 2014-06-12 08:46:01.245078
2014-06-12 14:46:58.891921 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2868 (0'0,173'2868] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2866 active] append_log  adding 1 keys
2014-06-12 14:46:58.891955 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2868, trimmed: 
2014-06-12 14:46:58.891982 7f9d643d9700  5 -- op tracker -- , seq: 748225, time: 2014-06-12 14:46:58.891978, event: started, op: osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.891992 7f9d643d9700  5 queue_transactions existing osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:46:58.891996 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:46:58.891997 7f9d643d9700  5 queue_op 0x2b00120 seq 2606 osr(3.169 0x276ce50) 4106485 bytes   (queue has 1 ops and 4106485 bytes)
2014-06-12 14:46:58.892008 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54f1200 (1 active)
2014-06-12 14:46:58.892003 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.169( v 173'2868 (0'0,173'2868] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2866 active] do_sub_op osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.892017 7f9d643d9700 10 osd.2 173 dequeue_op 0x329ea100 finish
2014-06-12 14:46:58.892016 7f9d7a962700  5 _do_op 0x2b00120 seq 2606 osr(3.169 0x276ce50)/0x276ce50 start
2014-06-12 14:46:58.892021 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:46:58.892021 7f9d7a962700 10 _do_transaction on 0x13f855f0
2014-06-12 14:46:58.892023 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:46:58.892026 7f9d7a962700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:46:58.892123 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:46:58.892146 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:46:58.892186 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:46:58.892279 7f9d7a962700 15 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:46:58.892284 7f9d7a962700 10 _omap_rmkeys meta/1fe2acfa/pglog_3.169/0//-1 r = 0
2014-06-12 14:46:58.892288 7f9d7a962700 15 _omap_setkeys meta/1fe2acfa/pglog_3.169/0//-1
2014-06-12 14:46:58.892301 7f9d7a962700 15 _write 3.169_head/99028969/rbd_data.10966b8b4567.0000000000001096/head//3 90112~4104192
2014-06-12 14:46:58.892402 7f9d7a962700 10 lookup_strip_header done  cid 3.169_head oid 99028969/rbd_data.10966b8b4567.0000000000001096/head//3
2014-06-12 14:46:58.892630 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:46:58.905263 7f9d7a962700 10 _generic_write 3.169_head/99028969/rbd_data.10966b8b4567.0000000000001096/head//3 90112~4104192 = 0
2014-06-12 14:46:58.905628 7f9d7a962700 15 _setattrs 3.169_head/99028969/rbd_data.10966b8b4567.0000000000001096/head//3
2014-06-12 14:46:58.905650 7f9d7a962700 10 _setattrs 3.169_head/99028969/rbd_data.10966b8b4567.0000000000001096/head//3 = 0
2014-06-12 14:46:58.905657 7f9d7a962700 15 _setattrs 3.169_head/99028969/rbd_data.10966b8b4567.0000000000001096/head//3
2014-06-12 14:46:58.905662 7f9d7a962700 10 _setattrs 3.169_head/99028969/rbd_data.10966b8b4567.0000000000001096/head//3 = 0
2014-06-12 14:46:58.906303 7f9d7a962700 20 set_header setting 17990 cid 3.169_head oid 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 parent seq 0
2014-06-12 14:46:58.906552 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:46:58.906564 7f9d7a962700 20 set_header setting 2306 cid meta oid 1fe2acfa/pglog_3.169/0//-1 parent seq 0
2014-06-12 14:46:58.906572 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:46:58.921986 7f9d7a962700 10 _do_op 0x2b00120 seq 2606 r = 0, finisher 0x88a54e0 0
2014-06-12 14:46:58.922041 7f9d7715b700  5 -- op tracker -- , seq: 748225, time: 2014-06-12 14:46:58.922028, event: commit_sent, op: osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.922061 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2868 (0'0,173'2868] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2866 active] sub_op_modify_commit on op osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:46:58.922092 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:46:58.922121 7f9d7a962700 10 _finish_op 0x2b00120 seq 2606 osr(3.169 0x276ce50)/0x276ce50
2014-06-12 14:46:58.922129 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54f1200 (0 active)
2014-06-12 14:46:58.922139 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:46:58.922168 7f9d7795c700  5 -- op tracker -- , seq: 748225, time: 2014-06-12 14:46:58.922160, event: sub_op_applied, op: osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:58.922186 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2868 (0'0,173'2868] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2867 active] sub_op_modify_applied on 0x13f85500 op osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:46:58.922210 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2868 (0'0,173'2868] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2867 active] op_applied on primary on version 173'2868
2014-06-12 14:46:58.922225 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.169( v 173'2868 (0'0,173'2868] local-les=161 n=293 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'2867 active] op_applied on replica on version 173'2868
2014-06-12 14:46:58.922250 7f9d7795c700  5 -- op tracker -- , seq: 748225, time: 2014-06-12 14:46:58.922245, event: done, op: osd_sub_op(client.4287.0:141321 3.169 99028969/rbd_data.10966b8b4567.0000000000001096/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:46:59.245910 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:59.245968 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:46:59.303342 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:59.303407 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:46:59.471622 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:59.471666 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:46:59.480448 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:46:59.480502 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:46:59.480502 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 10.74 >= max 0.5 = no, load too high
2014-06-12 14:46:59.480514 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:46:59.480516 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92667 < max (604800 seconds)
2014-06-12 14:46:59.480535 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:46:59.480538 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:59.480540 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:59.480542 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 23; oldest is 25.814674 seconds old
2014-06-12 14:46:59.823928 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:59.823987 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:46:59.847152 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:59.847156 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:59.847157 7f9d74bfa700 20 osd.2 173 _dispatch 0x18387ba0 ping v1
2014-06-12 14:46:59.847167 7f9d74bfa700 10 osd.2 173 ping from client.4290
2014-06-12 14:46:59.847170 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:46:59.847171 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:46:59.974565 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.024285 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19426 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,6,13,4])
2014-06-12 14:47:00.024303 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19426 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,6,13,4])
2014-06-12 14:47:00.024784 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:00.024813 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:00.024831 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:00.024848 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:00.024865 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:00.024904 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:00.024923 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:00.024939 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:00.024956 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:00.024983 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:00.025006 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:00.025035 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:00.025060 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:00.025081 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:00.025099 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:00.025116 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:00.025132 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:00.025149 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:00.025165 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:00.025210 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:00.025236 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:00.025304 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:00.155362 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:00.159642 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:00.159645 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:00.159655 7f9d74bfa700 20 osd.2 173 _dispatch 0x32118820 ping v1
2014-06-12 14:47:00.159658 7f9d74bfa700 10 osd.2 173 ping from client.4293
2014-06-12 14:47:00.159659 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:00.159660 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:00.166775 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.170267 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.177099 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.180454 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.187044 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.191741 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.193909 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.196136 7f9d5a16f700  5 -- op tracker -- , seq: 748226, time: 2014-06-12 14:47:00.194928, event: header_read, op: osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.196181 7f9d5a16f700  5 -- op tracker -- , seq: 748226, time: 2014-06-12 14:47:00.194932, event: throttled, op: osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.196200 7f9d5a16f700  5 -- op tracker -- , seq: 748226, time: 2014-06-12 14:47:00.196076, event: all_read, op: osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.196220 7f9d5a16f700  5 -- op tracker -- , seq: 748226, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.196230 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:00.196245 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:00.196262 7f9d5a16f700 15 osd.2 173 enqueue_op 0x28fd8e00 prio 127 cost 516903 latency 0.001334 osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.196341 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.196370 7f9d673df700 10 osd.2 173 dequeue_op 0x28fd8e00 prio 127 cost 516903 latency 0.001441 osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2507 (0'0,173'2507] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2506 active]
2014-06-12 14:47:00.196400 7f9d673df700  5 -- op tracker -- , seq: 748226, time: 2014-06-12 14:47:00.196395, event: reached_pg, op: osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.196412 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2507 (0'0,173'2507] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2506 active] handle_message: 0x28fd8e00
2014-06-12 14:47:00.196424 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2507 (0'0,173'2507] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2506 active] sub_op_modify trans a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 v 173'2508 (transaction) 174
2014-06-12 14:47:00.196440 7f9d673df700  5 -- op tracker -- , seq: 748226, time: 2014-06-12 14:47:00.196436, event: started, op: osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.196458 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2507 (0'0,173'2507] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2506 active] append_log log((0'0,173'2507], crt=0'0) [173'2508 (173'2507) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146443 2014-06-12 08:41:37.959212]
2014-06-12 14:47:00.196483 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2508 (0'0,173'2508] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2506 active] add_log_entry 173'2508 (173'2507) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146443 2014-06-12 08:41:37.959212
2014-06-12 14:47:00.196508 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2508 (0'0,173'2508] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2506 active] append_log  adding 1 keys
2014-06-12 14:47:00.196542 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2508, trimmed: 
2014-06-12 14:47:00.196568 7f9d673df700  5 -- op tracker -- , seq: 748226, time: 2014-06-12 14:47:00.196563, event: started, op: osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.196578 7f9d673df700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:00.196584 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.196585 7f9d673df700  5 queue_op 0x2b3e000 seq 2354 osr(3.168 0x276ce40) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:00.196599 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:47:00.196608 7f9d7f16b700  5 _do_op 0x2b3e000 seq 2354 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:47:00.196592 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2508 (0'0,173'2508] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2506 active] do_sub_op osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.196614 7f9d7f16b700 10 _do_transaction on 0x6176b70
2014-06-12 14:47:00.196613 7f9d673df700 10 osd.2 173 dequeue_op 0x28fd8e00 finish
2014-06-12 14:47:00.196619 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.196622 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.196622 7f9d7f16b700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:00.196784 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:00.196819 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.196886 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.197023 7f9d7f16b700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:00.197035 7f9d7f16b700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:47:00.197043 7f9d7f16b700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:00.197062 7f9d7f16b700 15 _write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 2613248~516096
2014-06-12 14:47:00.197256 7f9d7f16b700 10 lookup_strip_header done  cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:47:00.197464 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:47:00.198796 7f9d7f16b700 10 _generic_write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 2613248~516096 = 0
2014-06-12 14:47:00.198834 7f9d7f16b700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:47:00.198843 7f9d7f16b700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:47:00.198846 7f9d7f16b700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:47:00.198851 7f9d7f16b700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:47:00.198973 7f9d7f16b700 20 set_header setting 16934 cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 parent seq 0
2014-06-12 14:47:00.199020 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.199029 7f9d7f16b700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:47:00.199035 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:47:00.201203 7f9d7f16b700 10 _do_op 0x2b3e000 seq 2354 r = 0, finisher 0xa6470c0 0
2014-06-12 14:47:00.201222 7f9d7f16b700 10 _finish_op 0x2b3e000 seq 2354 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:00.201229 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:47:00.201238 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.201241 7f9d7715b700  5 -- op tracker -- , seq: 748226, time: 2014-06-12 14:47:00.201231, event: commit_sent, op: osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.201251 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2508 (0'0,173'2508] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2506 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:00.201276 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:00.201321 7f9d7795c700  5 -- op tracker -- , seq: 748226, time: 2014-06-12 14:47:00.201315, event: sub_op_applied, op: osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.201331 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2508 (0'0,173'2508] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2507 active] sub_op_modify_applied on 0x6176a80 op osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.201354 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2508 (0'0,173'2508] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2507 active] op_applied on primary on version 173'2508
2014-06-12 14:47:00.201364 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2508 (0'0,173'2508] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2507 active] op_applied on replica on version 173'2508
2014-06-12 14:47:00.201390 7f9d7795c700  5 -- op tracker -- , seq: 748226, time: 2014-06-12 14:47:00.201386, event: done, op: osd_sub_op(client.4316.0:146443 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2508 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.204054 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.207809 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.210086 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.212767 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.220544 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.220609 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.225921 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.228330 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.231921 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.324919 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.333188 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.339271 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.344611 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.350632 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.354498 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.356585 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:00.356635 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:00.360726 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.364051 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.369877 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:00.370794 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.375042 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.378753 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.379436 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.385147 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.388166 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.422552 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:00.422809 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:00.480620 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:00.480695 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:00.480691 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 10.28 >= max 0.5 = no, load too high
2014-06-12 14:47:00.480711 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:47:00.480715 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92668 < max (604800 seconds)
2014-06-12 14:47:00.480742 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:47:00.480745 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:00.480747 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:00.480750 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 23; oldest is 26.814881 seconds old
2014-06-12 14:47:00.519113 7f9d494c9700  5 -- op tracker -- , seq: 748227, time: 2014-06-12 14:47:00.517779, event: header_read, op: osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.519137 7f9d494c9700  5 -- op tracker -- , seq: 748227, time: 2014-06-12 14:47:00.517783, event: throttled, op: osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.519155 7f9d494c9700  5 -- op tracker -- , seq: 748227, time: 2014-06-12 14:47:00.519057, event: all_read, op: osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.519165 7f9d494c9700  5 -- op tracker -- , seq: 748227, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.519173 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:47:00.519206 7f9d494c9700 15 osd.2 173 enqueue_op 0x6f4b200 prio 63 cost 516096 latency 0.001427 osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:00.519268 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.519289 7f9d653db700 10 osd.2 173 dequeue_op 0x6f4b200 prio 63 cost 516096 latency 0.001509 osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]
2014-06-12 14:47:00.519330 7f9d653db700  5 -- op tracker -- , seq: 748227, time: 2014-06-12 14:47:00.519325, event: reached_pg, op: osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.519349 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:00.519368 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] handle_message: 0x6f4b200
2014-06-12 14:47:00.519378 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_op osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:00.519403 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] get_object_context: found obc in cache: 0x244de000
2014-06-12 14:47:00.519414 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] get_object_context: 0x244de000 d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 rwstate(write n=3 w=0) oi: d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3(173'2515 client.4290.0:143738 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2515) ssc: 0x679b420 snapset: 0=[]:[]+head
2014-06-12 14:47:00.519430 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] find_object_context d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 @head oi=d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3(173'2515 client.4290.0:143738 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2515)
2014-06-12 14:47:00.519456 7f9d653db700  5 -- op tracker -- , seq: 748227, time: 2014-06-12 14:47:00.519451, event: started, op: osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.519465 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] execute_ctx 0x5e4ca00
2014-06-12 14:47:00.519504 7f9d653db700  5 -- op tracker -- , seq: 748227, time: 2014-06-12 14:47:00.519499, event: started, op: osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.519512 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_op d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] ov 173'2515 av 173'2516 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:00.519529 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096]
2014-06-12 14:47:00.519552 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:00.519566 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op  write 2162688~516096
2014-06-12 14:47:00.519585 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:47:00.519600 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] make_writeable d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 snapset=0x679b458  snapc=0=[]
2014-06-12 14:47:00.519614 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] make_writeable d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:00.519627 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] finish_ctx d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 0x5e4ca00 op modify  
2014-06-12 14:47:00.519642 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  set mtime to 2014-06-12 08:46:02.897005
2014-06-12 14:47:00.519666 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  final snapset 0=[]:[]+head in d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.519687 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  zeroing write result code 0
2014-06-12 14:47:00.519698 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] new_repop rep_tid 236372 on osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:00.519710 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] issue_repop rep_tid 236372 o d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.519727 7f9d653db700  5 -- op tracker -- , seq: 748227, time: 2014-06-12 14:47:00.519724, event: waiting for subops from 26, op: osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.519746 7f9d653db700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:00.519763 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2515 (0'0,173'2515] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] append_log log((0'0,173'2515], crt=0'0) [173'2516 (173'2515) modify   d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 by client.4290.0:143739 2014-06-12 08:46:02.897005]
2014-06-12 14:47:00.519780 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] add_log_entry 173'2516 (173'2515) modify   d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 by client.4290.0:143739 2014-06-12 08:46:02.897005
2014-06-12 14:47:00.519794 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] append_log  adding 1 keys
2014-06-12 14:47:00.519817 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2516, trimmed: 
2014-06-12 14:47:00.519834 7f9d653db700  5 queue_transactions existing osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:47:00.519838 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.519839 7f9d653db700  5 queue_op 0x142a8660 seq 2332 osr(3.7ae 0x49b4010) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:00.519849 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1c0b440 (1 active)
2014-06-12 14:47:00.519856 7f9d7c966700  5 _do_op 0x142a8660 seq 2332 osr(3.7ae 0x49b4010)/0x49b4010 start
2014-06-12 14:47:00.519845 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] eval_repop repgather(0x1f067c80 173'2516 rep_tid=236372 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:00.519861 7f9d7c966700 10 _do_transaction on 0x34d8e6c0
2014-06-12 14:47:00.519862 7f9d653db700 10 osd.2 173 dequeue_op 0x6f4b200 finish
2014-06-12 14:47:00.519865 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.519868 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.519867 7f9d7c966700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:47:00.519975 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:47:00.519994 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.520032 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.520121 7f9d7c966700 15 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:47:00.520128 7f9d7c966700 10 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1 r = 0
2014-06-12 14:47:00.520132 7f9d7c966700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:47:00.520145 7f9d7c966700 15 _write 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 2162688~516096
2014-06-12 14:47:00.520245 7f9d7c966700 10 lookup_strip_header done  cid 3.7ae_head oid d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.520394 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:47:00.521655 7f9d7c966700 10 _generic_write 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 2162688~516096 = 0
2014-06-12 14:47:00.521698 7f9d7c966700 15 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.521708 7f9d7c966700 10 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:47:00.521714 7f9d7c966700 15 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.521719 7f9d7c966700 10 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:47:00.521880 7f9d7c966700 20 set_header setting 15929 cid 3.7ae_head oid d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 parent seq 0
2014-06-12 14:47:00.521939 7f9d7c966700 20 set_header setting 2150 cid meta oid 1fe77416/pglog_3.7ae/0//-1 parent seq 0
2014-06-12 14:47:00.521951 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.521956 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:47:00.523221 7f9d494c9700  5 -- op tracker -- , seq: 748228, time: 2014-06-12 14:47:00.520321, event: header_read, op: osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.523248 7f9d494c9700  5 -- op tracker -- , seq: 748228, time: 2014-06-12 14:47:00.520324, event: throttled, op: osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.523260 7f9d494c9700  5 -- op tracker -- , seq: 748228, time: 2014-06-12 14:47:00.523178, event: all_read, op: osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.523270 7f9d494c9700  5 -- op tracker -- , seq: 748228, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.523279 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:47:00.523310 7f9d494c9700 15 osd.2 173 enqueue_op 0x5abf600 prio 63 cost 1032192 latency 0.002989 osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:00.523344 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.523353 7f9d623d5700 10 osd.2 173 dequeue_op 0x5abf600 prio 63 cost 1032192 latency 0.003032 osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]
2014-06-12 14:47:00.523381 7f9d623d5700  5 -- op tracker -- , seq: 748228, time: 2014-06-12 14:47:00.523376, event: reached_pg, op: osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.523390 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:00.523406 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] handle_message: 0x5abf600
2014-06-12 14:47:00.523416 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_op osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:00.523440 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] get_object_context: found obc in cache: 0x244de000
2014-06-12 14:47:00.523452 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] get_object_context: 0x244de000 d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 rwstate(write n=4 w=0) oi: d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3(173'2516 client.4290.0:143739 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2516) ssc: 0x679b420 snapset: 0=[]:[]+head
2014-06-12 14:47:00.523467 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] find_object_context d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 @head oi=d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3(173'2516 client.4290.0:143739 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2516)
2014-06-12 14:47:00.523494 7f9d623d5700  5 -- op tracker -- , seq: 748228, time: 2014-06-12 14:47:00.523486, event: started, op: osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.523502 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] execute_ctx 0x2963a600
2014-06-12 14:47:00.523520 7f9d623d5700  5 -- op tracker -- , seq: 748228, time: 2014-06-12 14:47:00.523515, event: started, op: osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.523527 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_op d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] ov 173'2516 av 173'2517 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:00.523540 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192]
2014-06-12 14:47:00.523551 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:00.523563 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op  write 2678784~1032192
2014-06-12 14:47:00.523580 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:47:00.523590 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] make_writeable d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 snapset=0x679b458  snapc=0=[]
2014-06-12 14:47:00.523602 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] make_writeable d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:00.523611 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] finish_ctx d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 0x2963a600 op modify  
2014-06-12 14:47:00.523625 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  set mtime to 2014-06-12 08:46:02.898487
2014-06-12 14:47:00.523645 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  final snapset 0=[]:[]+head in d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.523662 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  zeroing write result code 0
2014-06-12 14:47:00.523673 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] new_repop rep_tid 236373 on osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:00.523689 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] issue_repop rep_tid 236373 o d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.523714 7f9d623d5700  5 -- op tracker -- , seq: 748228, time: 2014-06-12 14:47:00.523709, event: waiting for subops from 26, op: osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.523736 7f9d623d5700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:00.523758 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2516 (0'0,173'2516] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] append_log log((0'0,173'2516], crt=0'0) [173'2517 (173'2516) modify   d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 by client.4290.0:143740 2014-06-12 08:46:02.898487]
2014-06-12 14:47:00.523786 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] add_log_entry 173'2517 (173'2516) modify   d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 by client.4290.0:143740 2014-06-12 08:46:02.898487
2014-06-12 14:47:00.523809 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] append_log  adding 1 keys
2014-06-12 14:47:00.523842 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2517, trimmed: 
2014-06-12 14:47:00.523867 7f9d623d5700  5 queue_transactions existing osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:47:00.523871 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.523873 7f9d623d5700  5 queue_op 0x16a789c0 seq 2333 osr(3.7ae 0x49b4010) 1034485 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:47:00.523881 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] eval_repop repgather(0x51dcb40 173'2517 rep_tid=236373 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:00.523899 7f9d623d5700 10 osd.2 173 dequeue_op 0x5abf600 finish
2014-06-12 14:47:00.523903 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.523905 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.524158 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1c0b440 (2 active)
2014-06-12 14:47:00.525514 7f9d7c966700 10 _do_op 0x142a8660 seq 2332 r = 0, finisher 0x6cfc860 0x31aed5c0
2014-06-12 14:47:00.525531 7f9d7c966700 10 _finish_op 0x142a8660 seq 2332 osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:47:00.525548 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1c0b440 (1 active)
2014-06-12 14:47:00.525555 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.525555 7f9d81970700  5 _do_op 0x16a789c0 seq 2333 osr(3.7ae 0x49b4010)/0x49b4010 start
2014-06-12 14:47:00.525564 7f9d81970700 10 _do_transaction on 0x51f6d80
2014-06-12 14:47:00.525545 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_commit: 236372
2014-06-12 14:47:00.525574 7f9d81970700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:47:00.525584 7f9d7715b700  5 -- op tracker -- , seq: 748227, time: 2014-06-12 14:47:00.525576, event: op_commit, op: osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.525600 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_applied: 236372
2014-06-12 14:47:00.525642 7f9d7795c700  5 -- op tracker -- , seq: 748227, time: 2014-06-12 14:47:00.525635, event: op_applied, op: osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.525650 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_applied on primary on version 173'2516
2014-06-12 14:47:00.525713 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:47:00.525736 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.525788 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.525808 7f9d81970700 15 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:47:00.525817 7f9d81970700 10 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1 r = 0
2014-06-12 14:47:00.525823 7f9d81970700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:47:00.525838 7f9d81970700 15 _write 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 2678784~1032192
2014-06-12 14:47:00.525936 7f9d81970700 10 lookup_strip_header done  cid 3.7ae_head oid d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.525957 7f9d81970700 10 file_to_extents done 
2014-06-12 14:47:00.529043 7f9d81970700 10 _generic_write 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 2678784~1032192 = 0
2014-06-12 14:47:00.529201 7f9d81970700 15 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.529247 7f9d81970700 10 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:47:00.529254 7f9d81970700 15 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.529263 7f9d81970700 10 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:47:00.529791 7f9d81970700 20 set_header setting 15929 cid 3.7ae_head oid d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 parent seq 0
2014-06-12 14:47:00.530020 7f9d81970700 20 set_header setting 2150 cid meta oid 1fe77416/pglog_3.7ae/0//-1 parent seq 0
2014-06-12 14:47:00.530058 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.530078 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:47:00.536317 7f9d81970700 10 _do_op 0x16a789c0 seq 2333 r = 0, finisher 0x682ab40 0x12d18140
2014-06-12 14:47:00.536342 7f9d81970700 10 _finish_op 0x16a789c0 seq 2333 osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:47:00.536353 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1c0b440 (0 active)
2014-06-12 14:47:00.536357 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.536348 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_commit: 236373
2014-06-12 14:47:00.536383 7f9d7715b700  5 -- op tracker -- , seq: 748228, time: 2014-06-12 14:47:00.536372, event: op_commit, op: osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.536403 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_applied: 236373
2014-06-12 14:47:00.536426 7f9d7795c700  5 -- op tracker -- , seq: 748228, time: 2014-06-12 14:47:00.536420, event: op_applied, op: osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.536434 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_applied on primary on version 173'2517
2014-06-12 14:47:00.547745 7f9d494c9700  5 -- op tracker -- , seq: 748229, time: 2014-06-12 14:47:00.546301, event: header_read, op: osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.547761 7f9d494c9700  5 -- op tracker -- , seq: 748229, time: 2014-06-12 14:47:00.546305, event: throttled, op: osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.547770 7f9d494c9700  5 -- op tracker -- , seq: 748229, time: 2014-06-12 14:47:00.547692, event: all_read, op: osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.547779 7f9d494c9700  5 -- op tracker -- , seq: 748229, time: 0.000000, event: dispatched, op: osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.547789 7f9d494c9700 20 osd.2 173 should_share_map client.4290 172.16.96.220:0/1241710 173
2014-06-12 14:47:00.547822 7f9d494c9700 15 osd.2 173 enqueue_op 0x9fad500 prio 63 cost 483328 latency 0.001521 osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:00.547861 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.547868 7f9d68be2700 10 osd.2 173 dequeue_op 0x9fad500 prio 63 cost 483328 latency 0.001567 osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]
2014-06-12 14:47:00.547894 7f9d68be2700  5 -- op tracker -- , seq: 748229, time: 2014-06-12 14:47:00.547889, event: reached_pg, op: osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.547904 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:00.547919 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] handle_message: 0x9fad500
2014-06-12 14:47:00.547930 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_op osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:00.547956 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] get_object_context: found obc in cache: 0x244de000
2014-06-12 14:47:00.547966 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] get_object_context: 0x244de000 d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 rwstate(write n=5 w=0) oi: d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3(173'2517 client.4290.0:143740 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2517) ssc: 0x679b420 snapset: 0=[]:[]+head
2014-06-12 14:47:00.547982 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] find_object_context d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 @head oi=d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3(173'2517 client.4290.0:143740 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2517)
2014-06-12 14:47:00.548009 7f9d68be2700  5 -- op tracker -- , seq: 748229, time: 2014-06-12 14:47:00.548003, event: started, op: osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.548016 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] execute_ctx 0x2a83e00
2014-06-12 14:47:00.548033 7f9d68be2700  5 -- op tracker -- , seq: 748229, time: 2014-06-12 14:47:00.548028, event: started, op: osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.548040 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_op d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] ov 173'2517 av 173'2518 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:00.548053 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328]
2014-06-12 14:47:00.548063 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:00.548076 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op  write 3710976~483328
2014-06-12 14:47:00.548091 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] do_osd_op_effects on session 0x5c8dc00
2014-06-12 14:47:00.548104 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] make_writeable d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 snapset=0x679b458  snapc=0=[]
2014-06-12 14:47:00.548120 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] make_writeable d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:00.548132 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] finish_ctx d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 0x2a83e00 op modify  
2014-06-12 14:47:00.548149 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  set mtime to 2014-06-12 08:46:02.925577
2014-06-12 14:47:00.548177 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  final snapset 0=[]:[]+head in d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.548196 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  zeroing write result code 0
2014-06-12 14:47:00.548206 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] new_repop rep_tid 236374 on osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:00.548221 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] issue_repop rep_tid 236374 o d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.548248 7f9d68be2700  5 -- op tracker -- , seq: 748229, time: 2014-06-12 14:47:00.548243, event: waiting for subops from 26, op: osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.548272 7f9d68be2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:00.548294 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2517 (0'0,173'2517] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] append_log log((0'0,173'2517], crt=0'0) [173'2518 (173'2517) modify   d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 by client.4290.0:143741 2014-06-12 08:46:02.925577]
2014-06-12 14:47:00.548318 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] add_log_entry 173'2518 (173'2517) modify   d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 by client.4290.0:143741 2014-06-12 08:46:02.925577
2014-06-12 14:47:00.548339 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] append_log  adding 1 keys
2014-06-12 14:47:00.548371 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2518, trimmed: 
2014-06-12 14:47:00.548393 7f9d68be2700  5 queue_transactions existing osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:47:00.548398 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.548400 7f9d68be2700  5 queue_op 0x4f779e0 seq 2334 osr(3.7ae 0x49b4010) 485621 bytes   (queue has 1 ops and 485621 bytes)
2014-06-12 14:47:00.548407 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] eval_repop repgather(0xbe0d740 173'2518 rep_tid=236374 committed?=0 applied?=0 lock=0 op=osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:00.548425 7f9d68be2700 10 osd.2 173 dequeue_op 0x9fad500 finish
2014-06-12 14:47:00.548429 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.548431 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.548836 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1c0b440 (1 active)
2014-06-12 14:47:00.548842 7f9d7a962700  5 _do_op 0x4f779e0 seq 2334 osr(3.7ae 0x49b4010)/0x49b4010 start
2014-06-12 14:47:00.548846 7f9d7a962700 10 _do_transaction on 0x2680fc0
2014-06-12 14:47:00.548853 7f9d7a962700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:47:00.549022 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:47:00.549044 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.549091 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.549422 7f9d7a962700 15 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:47:00.549429 7f9d7a962700 10 _omap_rmkeys meta/1fe77416/pglog_3.7ae/0//-1 r = 0
2014-06-12 14:47:00.549435 7f9d7a962700 15 _omap_setkeys meta/1fe77416/pglog_3.7ae/0//-1
2014-06-12 14:47:00.549474 7f9d7a962700 15 _write 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 3710976~483328
2014-06-12 14:47:00.550165 7f9d7a962700 10 lookup_strip_header done  cid 3.7ae_head oid d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.550398 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:47:00.552198 7f9d7a962700 10 _generic_write 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 3710976~483328 = 0
2014-06-12 14:47:00.552236 7f9d7a962700 15 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.552250 7f9d7a962700 10 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:47:00.552255 7f9d7a962700 15 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3
2014-06-12 14:47:00.552262 7f9d7a962700 10 _setattrs 3.7ae_head/d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 = 0
2014-06-12 14:47:00.552445 7f9d7a962700 20 set_header setting 15929 cid 3.7ae_head oid d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 parent seq 0
2014-06-12 14:47:00.552508 7f9d7a962700 20 set_header setting 2150 cid meta oid 1fe77416/pglog_3.7ae/0//-1 parent seq 0
2014-06-12 14:47:00.552521 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.552632 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:47:00.555036 7f9d7a962700 10 _do_op 0x4f779e0 seq 2334 r = 0, finisher 0x6961fa0 0x381e6e40
2014-06-12 14:47:00.555068 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_commit: 236374
2014-06-12 14:47:00.555102 7f9d7715b700  5 -- op tracker -- , seq: 748229, time: 2014-06-12 14:47:00.555091, event: op_commit, op: osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.555119 7f9d7a962700 10 _finish_op 0x4f779e0 seq 2334 osr(3.7ae 0x49b4010)/0x49b4010
2014-06-12 14:47:00.555130 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1c0b440 (0 active)
2014-06-12 14:47:00.555139 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.555138 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_applied: 236374
2014-06-12 14:47:00.555164 7f9d7795c700  5 -- op tracker -- , seq: 748229, time: 2014-06-12 14:47:00.555157, event: op_applied, op: osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.555178 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] op_applied on primary on version 173'2518
2014-06-12 14:47:00.609056 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:00.609092 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:00.759642 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:00.759684 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:00.764980 7f9d43bdc700  5 -- op tracker -- , seq: 748230, time: 2014-06-12 14:47:00.764698, event: header_read, op: osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.764990 7f9d43bdc700  5 -- op tracker -- , seq: 748230, time: 2014-06-12 14:47:00.764701, event: throttled, op: osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.764997 7f9d43bdc700  5 -- op tracker -- , seq: 748230, time: 2014-06-12 14:47:00.764949, event: all_read, op: osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.765002 7f9d43bdc700  5 -- op tracker -- , seq: 748230, time: 0.000000, event: dispatched, op: osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.765008 7f9d43bdc700 20 osd.2 173 should_share_map client.4310 172.16.96.220:0/1244085 173
2014-06-12 14:47:00.765038 7f9d43bdc700 15 osd.2 173 enqueue_op 0x5ee9100 prio 63 cost 77824 latency 0.000340 osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:00.765065 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.765083 7f9d643d9700 10 osd.2 173 dequeue_op 0x5ee9100 prio 63 cost 77824 latency 0.000385 osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]
2014-06-12 14:47:00.765126 7f9d643d9700  5 -- op tracker -- , seq: 748230, time: 2014-06-12 14:47:00.765120, event: reached_pg, op: osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.765137 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:00.765162 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] handle_message: 0x5ee9100
2014-06-12 14:47:00.765174 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] do_op osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:00.765199 7f9d643d9700 15 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 '_'
2014-06-12 14:47:00.765591 7f9d643d9700 10 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 '_' = 0
2014-06-12 14:47:00.765610 7f9d643d9700 15 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 'snapset'
2014-06-12 14:47:00.765729 7f9d643d9700 10 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 'snapset' = 0
2014-06-12 14:47:00.765741 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] populate_obc_watchers 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:00.765756 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:00.765768 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] get_object_context: creating obc from disk: 0x2b9fadc0
2014-06-12 14:47:00.765780 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] get_object_context: 0x2b9fadc0 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 rwstate(none n=0 w=0) oi: 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3(173'664 client.4310.0:36399 wrlock_by=unknown.0.0:0 dirty s 4194304 uv664) ssc: 0x5d5a620 snapset: 0=[]:[]+head
2014-06-12 14:47:00.765797 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] find_object_context 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 @head oi=39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3(173'664 client.4310.0:36399 wrlock_by=unknown.0.0:0 dirty s 4194304 uv664)
2014-06-12 14:47:00.765825 7f9d643d9700  5 -- op tracker -- , seq: 748230, time: 2014-06-12 14:47:00.765819, event: started, op: osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.765836 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] execute_ctx 0x1b264400
2014-06-12 14:47:00.765856 7f9d643d9700  5 -- op tracker -- , seq: 748230, time: 2014-06-12 14:47:00.765850, event: started, op: osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.765865 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] do_op 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] ov 173'664 av 173'2707 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:00.765882 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] do_osd_op 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824]
2014-06-12 14:47:00.765895 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:00.765910 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] do_osd_op  write 0~77824
2014-06-12 14:47:00.765927 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] do_osd_op_effects on session 0xa703a00
2014-06-12 14:47:00.765941 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] make_writeable 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 snapset=0x5d5a658  snapc=0=[]
2014-06-12 14:47:00.765955 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] make_writeable 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:00.765967 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] finish_ctx 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 0x1b264400 op modify  
2014-06-12 14:47:00.765981 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]  set mtime to 2014-06-12 08:46:03.144733
2014-06-12 14:47:00.766005 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]  final snapset 0=[]:[]+head in 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:00.766025 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]  zeroing write result code 0
2014-06-12 14:47:00.766038 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] new_repop rep_tid 236375 on osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:00.766055 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] issue_repop rep_tid 236375 o 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:00.766082 7f9d643d9700  5 -- op tracker -- , seq: 748230, time: 2014-06-12 14:47:00.766076, event: waiting for subops from 23, op: osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.766104 7f9d643d9700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:00.766128 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2706 (0'0,173'2706] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] append_log log((0'0,173'2706], crt=0'0) [173'2707 (173'664) modify   39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 by client.4310.0:142687 2014-06-12 08:46:03.144733]
2014-06-12 14:47:00.766156 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] add_log_entry 173'2707 (173'664) modify   39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 by client.4310.0:142687 2014-06-12 08:46:03.144733
2014-06-12 14:47:00.766179 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] append_log  adding 1 keys
2014-06-12 14:47:00.766213 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2707, trimmed: 
2014-06-12 14:47:00.766240 7f9d643d9700  5 queue_transactions existing osr(3.63e 0x276dc60)/0x276dc60
2014-06-12 14:47:00.766247 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.766249 7f9d643d9700  5 queue_op 0x6feb260 seq 2512 osr(3.63e 0x276dc60) 80117 bytes   (queue has 1 ops and 80117 bytes)
2014-06-12 14:47:00.766258 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a26c0 (1 active)
2014-06-12 14:47:00.766264 7f9d82972700  5 _do_op 0x6feb260 seq 2512 osr(3.63e 0x276dc60)/0x276dc60 start
2014-06-12 14:47:00.766267 7f9d82972700 10 _do_transaction on 0x1c378900
2014-06-12 14:47:00.766256 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] eval_repop repgather(0x32be2e40 173'2707 rep_tid=236375 committed?=0 applied?=0 lock=0 op=osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:00.766272 7f9d82972700 15 _omap_setkeys meta/1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:00.766274 7f9d643d9700 10 osd.2 173 dequeue_op 0x5ee9100 finish
2014-06-12 14:47:00.766278 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.766281 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.766348 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:00.766367 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.766405 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.766544 7f9d82972700 15 _omap_rmkeys meta/1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:00.766550 7f9d82972700 10 _omap_rmkeys meta/1fe0c49b/pglog_3.63e/0//-1 r = 0
2014-06-12 14:47:00.766554 7f9d82972700 15 _omap_setkeys meta/1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:00.766566 7f9d82972700 15 _write 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 0~77824
2014-06-12 14:47:00.766880 7f9d82972700 10 lookup_strip_header done  cid 3.63e_head oid 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:00.767046 7f9d82972700 10 file_to_extents done 
2014-06-12 14:47:00.767235 7f9d82972700 10 _generic_write 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 0~77824 = 0
2014-06-12 14:47:00.767254 7f9d82972700 15 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:00.767270 7f9d82972700 10 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 = 0
2014-06-12 14:47:00.767280 7f9d82972700 15 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:00.767289 7f9d82972700 10 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 = 0
2014-06-12 14:47:00.767377 7f9d82972700 20 set_header setting 16668 cid 3.63e_head oid 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 parent seq 0
2014-06-12 14:47:00.767482 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.767498 7f9d82972700 20 set_header setting 2112 cid meta oid 1fe0c49b/pglog_3.63e/0//-1 parent seq 0
2014-06-12 14:47:00.767510 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:47:00.767783 7f9d82972700 10 _do_op 0x6feb260 seq 2512 r = 0, finisher 0x6ab80c0 0x31aed900
2014-06-12 14:47:00.767792 7f9d82972700 10 _finish_op 0x6feb260 seq 2512 osr(3.63e 0x276dc60)/0x276dc60
2014-06-12 14:47:00.767800 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a26c0 (0 active)
2014-06-12 14:47:00.767802 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.767832 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] op_commit: 236375
2014-06-12 14:47:00.767856 7f9d7715b700  5 -- op tracker -- , seq: 748230, time: 2014-06-12 14:47:00.767848, event: op_commit, op: osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.767881 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] op_applied: 236375
2014-06-12 14:47:00.767902 7f9d7795c700  5 -- op tracker -- , seq: 748230, time: 2014-06-12 14:47:00.767895, event: op_applied, op: osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.767909 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] op_applied on primary on version 173'2707
2014-06-12 14:47:00.770439 7f9d53c11700  5 -- op tracker -- , seq: 748231, time: 2014-06-12 14:47:00.766295, event: header_read, op: osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.770451 7f9d53c11700  5 -- op tracker -- , seq: 748231, time: 2014-06-12 14:47:00.766297, event: throttled, op: osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.770459 7f9d53c11700  5 -- op tracker -- , seq: 748231, time: 2014-06-12 14:47:00.770396, event: all_read, op: osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.770467 7f9d53c11700  5 -- op tracker -- , seq: 748231, time: 0.000000, event: dispatched, op: osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.770474 7f9d53c11700 20 osd.2 173 should_share_map client.4300 172.16.96.220:0/1243739 173
2014-06-12 14:47:00.770497 7f9d53c11700 15 osd.2 173 enqueue_op 0xd38f800 prio 63 cost 1032192 latency 0.004202 osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:00.770573 7f9d57543700  5 -- op tracker -- , seq: 748232, time: 2014-06-12 14:47:00.770544, event: header_read, op: osd_sub_op_reply(client.4310.0:142687 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:00.770581 7f9d57543700  5 -- op tracker -- , seq: 748232, time: 2014-06-12 14:47:00.770545, event: throttled, op: osd_sub_op_reply(client.4310.0:142687 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:00.770587 7f9d57543700  5 -- op tracker -- , seq: 748232, time: 2014-06-12 14:47:00.770553, event: all_read, op: osd_sub_op_reply(client.4310.0:142687 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:00.770594 7f9d57543700  5 -- op tracker -- , seq: 748232, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142687 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:00.770597 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142687 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:00.770602 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:00.770609 7f9d57543700 15 osd.2 173 enqueue_op 0x2519cb00 prio 196 cost 0 latency 0.000065 osd_sub_op_reply(client.4310.0:142687 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:00.770648 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.770658 7f9d56432700  5 -- op tracker -- , seq: 748233, time: 2014-06-12 14:47:00.769658, event: header_read, op: osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.770668 7f9d56432700  5 -- op tracker -- , seq: 748233, time: 2014-06-12 14:47:00.769663, event: throttled, op: osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.770658 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2519cb00 prio 196 cost 0 latency 0.000114 osd_sub_op_reply(client.4310.0:142687 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0) v2 pg pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]
2014-06-12 14:47:00.770677 7f9d56432700  5 -- op tracker -- , seq: 748233, time: 2014-06-12 14:47:00.770631, event: all_read, op: osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.770688 7f9d56432700  5 -- op tracker -- , seq: 748233, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.770695 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:00.770701 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:00.770707 7f9d56432700 15 osd.2 173 enqueue_op 0x59dcc00 prio 127 cost 426791 latency 0.001048 osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.770744 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:00.770750 7f9d61bd4700 10 osd.2 173 dequeue_op 0x59dcc00 prio 127 cost 426791 latency 0.001092 osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.458( v 173'3005 (79'4,173'3005] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3004 active]
2014-06-12 14:47:00.770773 7f9d61bd4700  5 -- op tracker -- , seq: 748233, time: 2014-06-12 14:47:00.770768, event: reached_pg, op: osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.770782 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3005 (79'4,173'3005] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3004 active] handle_message: 0x59dcc00
2014-06-12 14:47:00.770796 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3005 (79'4,173'3005] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3004 active] sub_op_modify trans d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 v 173'3006 (transaction) 174
2014-06-12 14:47:00.770813 7f9d61bd4700  5 -- op tracker -- , seq: 748233, time: 2014-06-12 14:47:00.770809, event: started, op: osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.770829 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3005 (79'4,173'3005] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3004 active] append_log log((79'4,173'3005], crt=0'0) [173'3006 (173'3005) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145538 2014-06-12 08:46:03.145634]
2014-06-12 14:47:00.770941 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3006 (79'4,173'3006] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3004 active] add_log_entry 173'3006 (173'3005) modify   d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 by client.4299.0:145538 2014-06-12 08:46:03.145634
2014-06-12 14:47:00.770966 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3006 (79'4,173'3006] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3004 active] append_log  adding 1 keys
2014-06-12 14:47:00.770980 7f9d61bd4700 10 trim log((79'4,173'3006], crt=0'0) to 79'5
2014-06-12 14:47:00.771009 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3006, trimmed: 79'5
2014-06-12 14:47:00.771017 7f9d63bd8700  5 -- op tracker -- , seq: 748232, time: 2014-06-12 14:47:00.770676, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142687 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:00.771025 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] handle_message: 0x2519cb00
2014-06-12 14:47:00.771041 7f9d63bd8700  5 -- op tracker -- , seq: 748232, time: 2014-06-12 14:47:00.771037, event: started, op: osd_sub_op_reply(client.4310.0:142687 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:00.771049 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] sub_op_modify_reply: tid 236375 op  ack_type 4 from 23
2014-06-12 14:47:00.771069 7f9d63bd8700  5 -- op tracker -- , seq: 748230, time: 2014-06-12 14:47:00.771062, event: sub_op_commit_rec, op: osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.771077 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] repop_all_applied: repop tid 236375 all applied 
2014-06-12 14:47:00.771088 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] eval_repop repgather(0x32be2e40 173'2707 rep_tid=236375 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:00.771102 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean]  sending ack on repgather(0x32be2e40 173'2707 rep_tid=236375 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) 0x24674780
2014-06-12 14:47:00.771131 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2706 crt=0'0 lcod 173'2705 mlcod 173'2705 active+clean] repop_all_committed: repop tid 236375 all committed 
2014-06-12 14:47:00.771141 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean] eval_repop repgather(0x32be2e40 173'2707 rep_tid=236375 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:00.771155 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean] log_op_stats osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4 inb 78103 outb 0 rlat 0.006431 lat 0.006455
2014-06-12 14:47:00.771169 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean] publish_stats_to_osd 173:3558
2014-06-12 14:47:00.771180 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean]  sending commit on repgather(0x32be2e40 173'2707 rep_tid=236375 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) 0x15560280
2014-06-12 14:47:00.771205 7f9d63bd8700  5 -- op tracker -- , seq: 748230, time: 2014-06-12 14:47:00.771201, event: commit_sent, op: osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.771213 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2705 active+clean]  requeue_ops 
2014-06-12 14:47:00.771223 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]  removing repgather(0x32be2e40 173'2707 rep_tid=236375 committed?=1 applied?=1 op=osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:00.771243 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]    q front is repgather(0x32be2e40 173'2707 rep_tid=236375 committed?=1 applied?=1 op=osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:00.771258 7f9d61bd4700  5 -- op tracker -- , seq: 748233, time: 2014-06-12 14:47:00.771038, event: started, op: osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.771257 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] remove_repop repgather(0x32be2e40 173'2707 rep_tid=236375 committed?=1 applied?=1 op=osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:00.771269 7f9d61bd4700  5 queue_transactions existing osr(3.458 0x276d960)/0x276d960
2014-06-12 14:47:00.771272 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.771274 7f9d61bd4700  5 queue_op 0xcc96de0 seq 2808 osr(3.458 0x276d960) 428312 bytes   (queue has 1 ops and 428312 bytes)
2014-06-12 14:47:00.771270 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]  requeue_ops 
2014-06-12 14:47:00.771285 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8000 (1 active)
2014-06-12 14:47:00.771281 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.458( v 173'3006 (79'5,173'3006] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3004 active] do_sub_op osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.771295 7f9d61bd4700 10 osd.2 173 dequeue_op 0x59dcc00 finish
2014-06-12 14:47:00.771293 7f9d63bd8700  5 -- op tracker -- , seq: 748230, time: 2014-06-12 14:47:00.771289, event: done, op: osd_op(client.4310.0:142687 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~77824] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.771299 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:00.771300 7f9d7895e700  5 _do_op 0xcc96de0 seq 2808 osr(3.458 0x276d960)/0x276d960 start
2014-06-12 14:47:00.771305 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:00.771305 7f9d7895e700 10 _do_transaction on 0x1ebedeb0
2014-06-12 14:47:00.771312 7f9d7895e700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:47:00.771308 7f9d61bd4700 10 osd.2 173 dequeue_op 0xd38f800 prio 63 cost 1032192 latency 0.005013 osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]
2014-06-12 14:47:00.771330 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2519cb00 finish
2014-06-12 14:47:00.771335 7f9d61bd4700  5 -- op tracker -- , seq: 748231, time: 2014-06-12 14:47:00.771325, event: reached_pg, op: osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.771353 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:00.771368 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] handle_message: 0xd38f800
2014-06-12 14:47:00.771379 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_op osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:00.771404 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] get_object_context: found obc in cache: 0xe9f02c0
2014-06-12 14:47:00.771415 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] get_object_context: 0xe9f02c0 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 rwstate(write n=4 w=0) oi: c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3(173'2539 client.4300.0:140293 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2539) ssc: 0x81ba460 snapset: 0=[]:[]+head
2014-06-12 14:47:00.771432 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] find_object_context c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 @head oi=c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3(173'2539 client.4300.0:140293 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2539)
2014-06-12 14:47:00.771449 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:47:00.771473 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.771465 7f9d61bd4700  5 -- op tracker -- , seq: 748231, time: 2014-06-12 14:47:00.771459, event: started, op: osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.771480 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] execute_ctx 0x283c6000
2014-06-12 14:47:00.771499 7f9d61bd4700  5 -- op tracker -- , seq: 748231, time: 2014-06-12 14:47:00.771494, event: started, op: osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.771507 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_op c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] ov 173'2539 av 173'2540 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:00.771530 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.771521 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192]
2014-06-12 14:47:00.771533 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:00.771554 7f9d7895e700 15 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:47:00.771548 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op  write 2723840~1032192
2014-06-12 14:47:00.771562 7f9d7895e700 10 _omap_rmkeys meta/1fe19d2a/pglog_3.458/0//-1 r = 0
2014-06-12 14:47:00.771568 7f9d7895e700 15 _omap_setkeys meta/1fe19d2a/pglog_3.458/0//-1
2014-06-12 14:47:00.771565 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op_effects on session 0x1b9b7200
2014-06-12 14:47:00.771581 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] make_writeable c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 snapset=0x81ba498  snapc=0=[]
2014-06-12 14:47:00.771598 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] make_writeable c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:00.771608 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] finish_ctx c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 0x283c6000 op modify  
2014-06-12 14:47:00.771622 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  set mtime to 2014-06-12 08:46:03.144531
2014-06-12 14:47:00.771645 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  final snapset 0=[]:[]+head in c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:47:00.771663 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  zeroing write result code 0
2014-06-12 14:47:00.771673 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] new_repop rep_tid 236376 on osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:00.771690 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] issue_repop rep_tid 236376 o c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:47:00.771716 7f9d61bd4700  5 -- op tracker -- , seq: 748231, time: 2014-06-12 14:47:00.771710, event: waiting for subops from 26, op: osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.771724 7f9d7895e700 15 _write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 3768320~425984
2014-06-12 14:47:00.771735 7f9d61bd4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:00.771757 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2539 (0'0,173'2539] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] append_log log((0'0,173'2539], crt=0'0) [173'2540 (173'2539) modify   c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 by client.4300.0:140294 2014-06-12 08:46:03.144531]
2014-06-12 14:47:00.771782 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] add_log_entry 173'2540 (173'2539) modify   c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 by client.4300.0:140294 2014-06-12 08:46:03.144531
2014-06-12 14:47:00.771803 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] append_log  adding 1 keys
2014-06-12 14:47:00.771835 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2540, trimmed: 
2014-06-12 14:47:00.771853 7f9d61bd4700  5 queue_transactions existing osr(3.1e1 0x276d140)/0x276d140
2014-06-12 14:47:00.771857 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.771858 7f9d61bd4700  5 queue_op 0x2c873e0 seq 2377 osr(3.1e1 0x276d140) 1034485 bytes   (queue has 2 ops and 1462797 bytes)
2014-06-12 14:47:00.771869 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x553f320 (2 active)
2014-06-12 14:47:00.771875 7f9d83173700  5 _do_op 0x2c873e0 seq 2377 osr(3.1e1 0x276d140)/0x276d140 start
2014-06-12 14:47:00.771878 7f9d83173700 10 _do_transaction on 0x1687de00
2014-06-12 14:47:00.771866 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] eval_repop repgather(0x53918c0 173'2540 rep_tid=236376 committed?=0 applied?=0 lock=0 op=osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:00.771883 7f9d83173700 15 _omap_setkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:47:00.771885 7f9d61bd4700 10 osd.2 173 dequeue_op 0xd38f800 finish
2014-06-12 14:47:00.771887 7f9d7895e700 10 lookup_strip_header done  cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:47:00.771889 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:00.771892 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.771953 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:47:00.771967 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.772001 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:47:00.772828 7f9d63bd8700  5 -- op tracker -- , seq: 748232, time: 2014-06-12 14:47:00.771333, event: done, op: osd_sub_op_reply(client.4310.0:142687 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:00.772857 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.772862 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.773594 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.773781 7f9d7895e700 10 _generic_write 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 3768320~425984 = 0
2014-06-12 14:47:00.773828 7f9d7895e700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:47:00.773846 7f9d7895e700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:47:00.773854 7f9d7895e700 15 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3
2014-06-12 14:47:00.773863 7f9d7895e700 10 _setattrs 3.458_head/d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 = 0
2014-06-12 14:47:00.774061 7f9d7895e700 20 set_header setting 17709 cid 3.458_head oid d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 parent seq 0
2014-06-12 14:47:00.774134 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.774152 7f9d7895e700 20 set_header setting 2332 cid meta oid 1fe19d2a/pglog_3.458/0//-1 parent seq 0
2014-06-12 14:47:00.774166 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:47:00.776059 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.776097 7f9d83173700 15 _omap_rmkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:47:00.776103 7f9d83173700 10 _omap_rmkeys meta/1fe10f55/pglog_3.1e1/0//-1 r = 0
2014-06-12 14:47:00.776110 7f9d83173700 15 _omap_setkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:47:00.776109 7f9d7895e700 10 _do_op 0xcc96de0 seq 2808 r = 0, finisher 0x90f6720 0
2014-06-12 14:47:00.776183 7f9d7715b700  5 -- op tracker -- , seq: 748233, time: 2014-06-12 14:47:00.776175, event: commit_sent, op: osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.776192 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3006 (79'5,173'3006] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3004 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:47:00.776213 7f9d83173700 15 _write 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 2723840~1032192
2014-06-12 14:47:00.776217 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:00.776292 7f9d7895e700 10 _finish_op 0xcc96de0 seq 2808 osr(3.458 0x276d960)/0x276d960
2014-06-12 14:47:00.776301 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8000 (1 active)
2014-06-12 14:47:00.776309 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.776354 7f9d7795c700  5 -- op tracker -- , seq: 748233, time: 2014-06-12 14:47:00.776348, event: sub_op_applied, op: osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.776361 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3006 (79'5,173'3006] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3005 active] sub_op_modify_applied on 0x1ebeddc0 op osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.776376 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3006 (79'5,173'3006] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3005 active] op_applied on primary on version 173'3006
2014-06-12 14:47:00.776385 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.458( v 173'3006 (79'5,173'3006] local-les=153 n=329 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'3005 active] op_applied on replica on version 173'3006
2014-06-12 14:47:00.776396 7f9d83173700 10 lookup_strip_header done  cid 3.1e1_head oid c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:47:00.776411 7f9d7795c700  5 -- op tracker -- , seq: 748233, time: 2014-06-12 14:47:00.776407, event: done, op: osd_sub_op(client.4299.0:145538 3.458 d4ec2458/rbd_data.10a26b8b4567.0000000000000f70/head//3 [] v 173'3006 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.776517 7f9d83173700 10 file_to_extents done 
2014-06-12 14:47:00.779439 7f9d83173700 10 _generic_write 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 2723840~1032192 = 0
2014-06-12 14:47:00.779528 7f9d83173700 15 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:47:00.779545 7f9d83173700 10 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:47:00.779560 7f9d83173700 15 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:47:00.779571 7f9d83173700 10 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:47:00.779778 7f9d83173700 20 set_header setting 16580 cid 3.1e1_head oid c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 parent seq 0
2014-06-12 14:47:00.779863 7f9d83173700 20 set_header setting 2008 cid meta oid 1fe10f55/pglog_3.1e1/0//-1 parent seq 0
2014-06-12 14:47:00.779880 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.779895 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:47:00.783176 7f9d56432700  5 -- op tracker -- , seq: 748234, time: 2014-06-12 14:47:00.777045, event: header_read, op: osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.783195 7f9d56432700  5 -- op tracker -- , seq: 748234, time: 2014-06-12 14:47:00.777047, event: throttled, op: osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.783204 7f9d56432700  5 -- op tracker -- , seq: 748234, time: 2014-06-12 14:47:00.783127, event: all_read, op: osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.783212 7f9d56432700  5 -- op tracker -- , seq: 748234, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.783219 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:00.783227 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:00.783236 7f9d56432700 15 osd.2 173 enqueue_op 0x59dd100 prio 127 cost 1639207 latency 0.006191 osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.783278 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.783290 7f9d67be0700 10 osd.2 173 dequeue_op 0x59dd100 prio 127 cost 1639207 latency 0.006245 osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2647 (0'0,173'2647] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2646 active]
2014-06-12 14:47:00.783317 7f9d67be0700  5 -- op tracker -- , seq: 748234, time: 2014-06-12 14:47:00.783313, event: reached_pg, op: osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.783330 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2647 (0'0,173'2647] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2646 active] handle_message: 0x59dd100
2014-06-12 14:47:00.783345 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2647 (0'0,173'2647] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2646 active] sub_op_modify trans 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 v 173'2648 (transaction) 174
2014-06-12 14:47:00.783365 7f9d67be0700  5 -- op tracker -- , seq: 748234, time: 2014-06-12 14:47:00.783361, event: started, op: osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.783382 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2647 (0'0,173'2647] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2646 active] append_log log((0'0,173'2647], crt=0'0) [173'2648 (173'695) modify   6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 by client.4299.0:145539 2014-06-12 08:46:03.146619]
2014-06-12 14:47:00.783409 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2648 (0'0,173'2648] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2646 active] add_log_entry 173'2648 (173'695) modify   6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 by client.4299.0:145539 2014-06-12 08:46:03.146619
2014-06-12 14:47:00.783434 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2648 (0'0,173'2648] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2646 active] append_log  adding 1 keys
2014-06-12 14:47:00.783472 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2648, trimmed: 
2014-06-12 14:47:00.783500 7f9d67be0700  5 -- op tracker -- , seq: 748234, time: 2014-06-12 14:47:00.783495, event: started, op: osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.783515 7f9d67be0700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:00.783520 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.783522 7f9d67be0700  5 queue_op 0xff14a80 seq 2468 osr(3.1c7 0x276d170) 1640693 bytes   (queue has 2 ops and 2675178 bytes)
2014-06-12 14:47:00.783534 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (2 active)
2014-06-12 14:47:00.783542 7f9d7a161700  5 _do_op 0xff14a80 seq 2468 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:47:00.783530 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2648 (0'0,173'2648] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2646 active] do_sub_op osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.783545 7f9d67be0700 10 osd.2 173 dequeue_op 0x59dd100 finish
2014-06-12 14:47:00.783546 7f9d7a161700 10 _do_transaction on 0x2a670470
2014-06-12 14:47:00.783549 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.783551 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.783553 7f9d7a161700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:00.783665 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:00.783690 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.784474 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.784553 7f9d7a161700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:00.784560 7f9d7a161700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:47:00.784565 7f9d7a161700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:00.784581 7f9d7a161700 15 _write 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 0~1638400
2014-06-12 14:47:00.784731 7f9d7a161700 10 lookup_strip_header done  cid 3.1c7_head oid 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3
2014-06-12 14:47:00.784785 7f9d83173700 10 _do_op 0x2c873e0 seq 2377 r = 0, finisher 0x6641280 0x1b742b40
2014-06-12 14:47:00.784802 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_commit: 236376
2014-06-12 14:47:00.784825 7f9d7715b700  5 -- op tracker -- , seq: 748231, time: 2014-06-12 14:47:00.784817, event: op_commit, op: osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.784838 7f9d83173700 10 _finish_op 0x2c873e0 seq 2377 osr(3.1e1 0x276d140)/0x276d140
2014-06-12 14:47:00.784851 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_applied: 236376
2014-06-12 14:47:00.784870 7f9d7795c700  5 -- op tracker -- , seq: 748231, time: 2014-06-12 14:47:00.784864, event: op_applied, op: osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:00.784880 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:47:00.784877 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_applied on primary on version 173'2540
2014-06-12 14:47:00.784905 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x553f320 (1 active)
2014-06-12 14:47:00.784908 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.788827 7f9d7a161700 10 _generic_write 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 0~1638400 = 0
2014-06-12 14:47:00.788927 7f9d7a161700 15 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3
2014-06-12 14:47:00.788942 7f9d7a161700 10 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 = 0
2014-06-12 14:47:00.788946 7f9d7a161700 15 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3
2014-06-12 14:47:00.788951 7f9d7a161700 10 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 = 0
2014-06-12 14:47:00.789211 7f9d7a161700 20 set_header setting 17711 cid 3.1c7_head oid 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 parent seq 0
2014-06-12 14:47:00.789316 7f9d7a161700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:47:00.789327 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.789333 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:47:00.795494 7f9d7a161700 10 _do_op 0xff14a80 seq 2468 r = 0, finisher 0x7eb0f80 0
2014-06-12 14:47:00.795515 7f9d7a161700 10 _finish_op 0xff14a80 seq 2468 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:00.795523 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:47:00.795528 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.795533 7f9d7715b700  5 -- op tracker -- , seq: 748234, time: 2014-06-12 14:47:00.795522, event: commit_sent, op: osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.795546 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2648 (0'0,173'2648] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2646 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:47:00.795568 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:00.795598 7f9d7795c700  5 -- op tracker -- , seq: 748234, time: 2014-06-12 14:47:00.795593, event: sub_op_applied, op: osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.795610 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2648 (0'0,173'2648] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2647 active] sub_op_modify_applied on 0x2a670380 op osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.795628 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2648 (0'0,173'2648] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2647 active] op_applied on primary on version 173'2648
2014-06-12 14:47:00.795635 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2648 (0'0,173'2648] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2647 active] op_applied on replica on version 173'2648
2014-06-12 14:47:00.795650 7f9d7795c700  5 -- op tracker -- , seq: 748234, time: 2014-06-12 14:47:00.795647, event: done, op: osd_sub_op(client.4299.0:145539 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2648 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.891176 7f9d5a16f700  5 -- op tracker -- , seq: 748235, time: 2014-06-12 14:47:00.890017, event: header_read, op: osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.891194 7f9d5a16f700  5 -- op tracker -- , seq: 748235, time: 2014-06-12 14:47:00.890020, event: throttled, op: osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.891204 7f9d5a16f700  5 -- op tracker -- , seq: 748235, time: 2014-06-12 14:47:00.891135, event: all_read, op: osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.891210 7f9d5a16f700  5 -- op tracker -- , seq: 748235, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.891215 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:00.891220 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:00.891227 7f9d5a16f700 15 osd.2 173 enqueue_op 0x68a6c00 prio 127 cost 516903 latency 0.001209 osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.891258 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.891267 7f9d673df700 10 osd.2 173 dequeue_op 0x68a6c00 prio 127 cost 516903 latency 0.001249 osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2508 (0'0,173'2508] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2507 active]
2014-06-12 14:47:00.891292 7f9d673df700  5 -- op tracker -- , seq: 748235, time: 2014-06-12 14:47:00.891288, event: reached_pg, op: osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.891301 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2508 (0'0,173'2508] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2507 active] handle_message: 0x68a6c00
2014-06-12 14:47:00.891313 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2508 (0'0,173'2508] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2507 active] sub_op_modify trans a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 v 173'2509 (transaction) 174
2014-06-12 14:47:00.891330 7f9d673df700  5 -- op tracker -- , seq: 748235, time: 2014-06-12 14:47:00.891326, event: started, op: osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.891349 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2508 (0'0,173'2508] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2507 active] append_log log((0'0,173'2508], crt=0'0) [173'2509 (173'2508) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146444 2014-06-12 08:41:37.981487]
2014-06-12 14:47:00.891372 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2509 (0'0,173'2509] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2507 active] add_log_entry 173'2509 (173'2508) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146444 2014-06-12 08:41:37.981487
2014-06-12 14:47:00.891392 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2509 (0'0,173'2509] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2507 active] append_log  adding 1 keys
2014-06-12 14:47:00.891416 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2509, trimmed: 
2014-06-12 14:47:00.891433 7f9d673df700  5 -- op tracker -- , seq: 748235, time: 2014-06-12 14:47:00.891430, event: started, op: osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.891447 7f9d673df700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:00.891452 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.891454 7f9d673df700  5 queue_op 0x6d05f20 seq 2355 osr(3.168 0x276ce40) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:00.891463 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2509 (0'0,173'2509] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2507 active] do_sub_op osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.891478 7f9d673df700 10 osd.2 173 dequeue_op 0x68a6c00 finish
2014-06-12 14:47:00.891482 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.891485 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.891531 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:47:00.891540 7f9d7f16b700  5 _do_op 0x6d05f20 seq 2355 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:47:00.891544 7f9d7f16b700 10 _do_transaction on 0x3317d0b0
2014-06-12 14:47:00.891552 7f9d7f16b700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:00.891700 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:00.891721 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.891782 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.891809 7f9d7f16b700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:00.891815 7f9d7f16b700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:47:00.891822 7f9d7f16b700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:00.891839 7f9d7f16b700 15 _write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 3129344~516096
2014-06-12 14:47:00.891995 7f9d7f16b700 10 lookup_strip_header done  cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:47:00.892230 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:47:00.894248 7f9d7f16b700 10 _generic_write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 3129344~516096 = 0
2014-06-12 14:47:00.894298 7f9d7f16b700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:47:00.894312 7f9d7f16b700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:47:00.894323 7f9d7f16b700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:47:00.894332 7f9d7f16b700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:47:00.894540 7f9d7f16b700 20 set_header setting 16934 cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 parent seq 0
2014-06-12 14:47:00.894619 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.894634 7f9d7f16b700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:47:00.894645 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:47:00.897160 7f9d7f16b700 10 _do_op 0x6d05f20 seq 2355 r = 0, finisher 0x98ef840 0
2014-06-12 14:47:00.897418 7f9d7715b700  5 -- op tracker -- , seq: 748235, time: 2014-06-12 14:47:00.897407, event: commit_sent, op: osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.897429 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2509 (0'0,173'2509] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2507 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:00.897455 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:00.897560 7f9d7f16b700 10 _finish_op 0x6d05f20 seq 2355 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:00.897571 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:47:00.897579 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.897597 7f9d7795c700  5 -- op tracker -- , seq: 748235, time: 2014-06-12 14:47:00.897591, event: sub_op_applied, op: osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.897606 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2509 (0'0,173'2509] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2508 active] sub_op_modify_applied on 0x3317cfc0 op osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.897637 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2509 (0'0,173'2509] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2508 active] op_applied on primary on version 173'2509
2014-06-12 14:47:00.897647 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2509 (0'0,173'2509] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2508 active] op_applied on replica on version 173'2509
2014-06-12 14:47:00.897683 7f9d7795c700  5 -- op tracker -- , seq: 748235, time: 2014-06-12 14:47:00.897679, event: done, op: osd_sub_op(client.4316.0:146444 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2509 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.909922 7f9d5bec4700  5 -- op tracker -- , seq: 748236, time: 2014-06-12 14:47:00.909021, event: header_read, op: osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.909942 7f9d5bec4700  5 -- op tracker -- , seq: 748236, time: 2014-06-12 14:47:00.909024, event: throttled, op: osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.909964 7f9d5bec4700  5 -- op tracker -- , seq: 748236, time: 2014-06-12 14:47:00.909873, event: all_read, op: osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.909969 7f9d5bec4700  5 -- op tracker -- , seq: 748236, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.909973 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:00.909978 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:47:00.909984 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e87c00 prio 127 cost 488231 latency 0.000963 osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.910048 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.910059 7f9d663dd700 10 osd.2 173 dequeue_op 0x20e87c00 prio 127 cost 488231 latency 0.001038 osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2863 (0'0,173'2863] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2862 active]
2014-06-12 14:47:00.910081 7f9d663dd700  5 -- op tracker -- , seq: 748236, time: 2014-06-12 14:47:00.910078, event: reached_pg, op: osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.910091 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2863 (0'0,173'2863] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2862 active] handle_message: 0x20e87c00
2014-06-12 14:47:00.910102 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2863 (0'0,173'2863] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2862 active] sub_op_modify trans eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 v 173'2864 (transaction) 174
2014-06-12 14:47:00.910113 7f9d663dd700  5 -- op tracker -- , seq: 748236, time: 2014-06-12 14:47:00.910110, event: started, op: osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.910128 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2863 (0'0,173'2863] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2862 active] append_log log((0'0,173'2863], crt=0'0) [173'2864 (173'797) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143753 2014-06-12 08:46:03.270617]
2014-06-12 14:47:00.910151 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2864 (0'0,173'2864] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2862 active] add_log_entry 173'2864 (173'797) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143753 2014-06-12 08:46:03.270617
2014-06-12 14:47:00.910167 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2864 (0'0,173'2864] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2862 active] append_log  adding 1 keys
2014-06-12 14:47:00.910192 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2864, trimmed: 
2014-06-12 14:47:00.910210 7f9d663dd700  5 -- op tracker -- , seq: 748236, time: 2014-06-12 14:47:00.910207, event: started, op: osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.910218 7f9d663dd700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:00.910222 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.910223 7f9d663dd700  5 queue_op 0x2e7d0e0 seq 2630 osr(3.402 0x276d620) 489717 bytes   (queue has 1 ops and 489717 bytes)
2014-06-12 14:47:00.910228 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2864 (0'0,173'2864] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2862 active] do_sub_op osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.910236 7f9d663dd700 10 osd.2 173 dequeue_op 0x20e87c00 finish
2014-06-12 14:47:00.910239 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.910248 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.910262 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:47:00.910280 7f9d7e169700  5 _do_op 0x2e7d0e0 seq 2630 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:47:00.910285 7f9d7e169700 10 _do_transaction on 0x303237b0
2014-06-12 14:47:00.910290 7f9d7e169700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.910397 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.910417 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.910454 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.910472 7f9d7e169700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.910477 7f9d7e169700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:47:00.910481 7f9d7e169700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.910493 7f9d7e169700 15 _write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 0~487424
2014-06-12 14:47:00.928130 7f9d5a16f700  5 -- op tracker -- , seq: 748237, time: 2014-06-12 14:47:00.927969, event: header_read, op: osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.928147 7f9d5a16f700  5 -- op tracker -- , seq: 748237, time: 2014-06-12 14:47:00.927971, event: throttled, op: osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.928164 7f9d5a16f700  5 -- op tracker -- , seq: 748237, time: 2014-06-12 14:47:00.928092, event: all_read, op: osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.928169 7f9d5a16f700  5 -- op tracker -- , seq: 748237, time: 0.000000, event: dispatched, op: osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.928172 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:00.928176 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:00.928182 7f9d5a16f700 15 osd.2 173 enqueue_op 0x68a7700 prio 127 cost 54055 latency 0.000213 osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.928204 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.928212 7f9d65bdc700 10 osd.2 173 dequeue_op 0x68a7700 prio 127 cost 54055 latency 0.000243 osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2670 (0'0,173'2670] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2669 active]
2014-06-12 14:47:00.928238 7f9d65bdc700  5 -- op tracker -- , seq: 748237, time: 2014-06-12 14:47:00.928234, event: reached_pg, op: osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.928249 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2670 (0'0,173'2670] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2669 active] handle_message: 0x68a7700
2014-06-12 14:47:00.928269 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2670 (0'0,173'2670] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2669 active] sub_op_modify trans 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 v 173'2671 (transaction) 174
2014-06-12 14:47:00.928295 7f9d65bdc700  5 -- op tracker -- , seq: 748237, time: 2014-06-12 14:47:00.928290, event: started, op: osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.928311 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2670 (0'0,173'2670] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2669 active] append_log log((0'0,173'2670], crt=0'0) [173'2671 (173'816) modify   57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 by client.4282.0:145225 2014-06-12 08:44:32.059474]
2014-06-12 14:47:00.928335 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2671 (0'0,173'2671] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2669 active] add_log_entry 173'2671 (173'816) modify   57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 by client.4282.0:145225 2014-06-12 08:44:32.059474
2014-06-12 14:47:00.928358 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2671 (0'0,173'2671] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2669 active] append_log  adding 1 keys
2014-06-12 14:47:00.928391 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2671, trimmed: 
2014-06-12 14:47:00.928420 7f9d65bdc700  5 -- op tracker -- , seq: 748237, time: 2014-06-12 14:47:00.928415, event: started, op: osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.928432 7f9d65bdc700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:00.928436 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.928438 7f9d65bdc700  5 queue_op 0x77488a0 seq 2481 osr(3.1b7 0x276d1a0) 55541 bytes   (queue has 2 ops and 545258 bytes)
2014-06-12 14:47:00.928444 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2671 (0'0,173'2671] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2669 active] do_sub_op osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.928457 7f9d65bdc700 10 osd.2 173 dequeue_op 0x68a7700 finish
2014-06-12 14:47:00.928460 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.928463 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.928464 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (2 active)
2014-06-12 14:47:00.928472 7f9d7815d700  5 _do_op 0x77488a0 seq 2481 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:00.928476 7f9d7815d700 10 _do_transaction on 0x2b2aeef0
2014-06-12 14:47:00.928481 7f9d7815d700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:00.937898 7f9d7e169700 10 lookup_strip_header done  cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.937930 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:00.937957 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.938086 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:47:00.939954 7f9d7e169700 10 _generic_write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 0~487424 = 0
2014-06-12 14:47:00.940020 7f9d7e169700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.940039 7f9d7e169700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:00.940060 7f9d7e169700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.940075 7f9d7e169700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:00.940307 7f9d7e169700 20 set_header setting 15936 cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 parent seq 0
2014-06-12 14:47:00.940538 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.940557 7f9d7e169700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:47:00.940569 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:47:00.943709 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.943800 7f9d7e169700 10 _do_op 0x2e7d0e0 seq 2630 r = 0, finisher 0x7fbfdc0 0
2014-06-12 14:47:00.943817 7f9d7e169700 10 _finish_op 0x2e7d0e0 seq 2630 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:00.943826 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (1 active)
2014-06-12 14:47:00.943835 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.943834 7f9d7815d700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:00.943837 7f9d7715b700  5 -- op tracker -- , seq: 748236, time: 2014-06-12 14:47:00.943827, event: commit_sent, op: osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.943842 7f9d7815d700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:00.943850 7f9d7815d700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:00.943845 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2864 (0'0,173'2864] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2862 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:47:00.943870 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:00.943869 7f9d7815d700 15 _write 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 0~53248
2014-06-12 14:47:00.943904 7f9d7795c700  5 -- op tracker -- , seq: 748236, time: 2014-06-12 14:47:00.943898, event: sub_op_applied, op: osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.943917 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2864 (0'0,173'2864] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2863 active] sub_op_modify_applied on 0x303236c0 op osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.943932 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2864 (0'0,173'2864] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2863 active] op_applied on primary on version 173'2864
2014-06-12 14:47:00.943941 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2864 (0'0,173'2864] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2863 active] op_applied on replica on version 173'2864
2014-06-12 14:47:00.943963 7f9d7795c700  5 -- op tracker -- , seq: 748236, time: 2014-06-12 14:47:00.943960, event: done, op: osd_sub_op(client.4290.0:143753 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2864 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.944050 7f9d7815d700 10 lookup_strip_header done  cid 3.1b7_head oid 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:00.944191 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:47:00.944386 7f9d7815d700 10 _generic_write 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 0~53248 = 0
2014-06-12 14:47:00.944402 7f9d7815d700 15 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:00.944413 7f9d7815d700 10 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:47:00.944420 7f9d7815d700 15 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:00.944428 7f9d7815d700 10 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:47:00.944552 7f9d7815d700 20 set_header setting 18080 cid 3.1b7_head oid 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 parent seq 0
2014-06-12 14:47:00.944576 7f9d7815d700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:00.944591 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.944600 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:47:00.944857 7f9d7815d700 10 _do_op 0x77488a0 seq 2481 r = 0, finisher 0x7bdcf20 0
2014-06-12 14:47:00.944866 7f9d7815d700 10 _finish_op 0x77488a0 seq 2481 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:00.944871 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:00.944873 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.944878 7f9d7715b700  5 -- op tracker -- , seq: 748237, time: 2014-06-12 14:47:00.944870, event: commit_sent, op: osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.944888 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2671 (0'0,173'2671] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2669 active] sub_op_modify_commit on op osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:00.944902 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:00.944922 7f9d7795c700  5 -- op tracker -- , seq: 748237, time: 2014-06-12 14:47:00.944917, event: sub_op_applied, op: osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.944929 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2671 (0'0,173'2671] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2670 active] sub_op_modify_applied on 0x2b2aee00 op osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.944940 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2671 (0'0,173'2671] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2670 active] op_applied on primary on version 173'2671
2014-06-12 14:47:00.944946 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2671 (0'0,173'2671] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2670 active] op_applied on replica on version 173'2671
2014-06-12 14:47:00.944958 7f9d7795c700  5 -- op tracker -- , seq: 748237, time: 2014-06-12 14:47:00.944955, event: done, op: osd_sub_op(client.4282.0:145225 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2671 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.950364 7f9d5bec4700  5 -- op tracker -- , seq: 748238, time: 2014-06-12 14:47:00.949576, event: header_read, op: osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.950378 7f9d5bec4700  5 -- op tracker -- , seq: 748238, time: 2014-06-12 14:47:00.949578, event: throttled, op: osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.950384 7f9d5bec4700  5 -- op tracker -- , seq: 748238, time: 2014-06-12 14:47:00.950326, event: all_read, op: osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.950390 7f9d5bec4700  5 -- op tracker -- , seq: 748238, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.950394 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:00.950399 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:47:00.950405 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e87a00 prio 127 cost 516903 latency 0.000829 osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.950432 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.950443 7f9d62bd6700 10 osd.2 173 dequeue_op 0x20e87a00 prio 127 cost 516903 latency 0.000867 osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2864 (0'0,173'2864] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2863 active]
2014-06-12 14:47:00.950468 7f9d62bd6700  5 -- op tracker -- , seq: 748238, time: 2014-06-12 14:47:00.950464, event: reached_pg, op: osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.950478 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2864 (0'0,173'2864] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2863 active] handle_message: 0x20e87a00
2014-06-12 14:47:00.950499 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2864 (0'0,173'2864] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2863 active] sub_op_modify trans eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 v 173'2865 (transaction) 174
2014-06-12 14:47:00.950519 7f9d62bd6700  5 -- op tracker -- , seq: 748238, time: 2014-06-12 14:47:00.950514, event: started, op: osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.950538 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2864 (0'0,173'2864] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2863 active] append_log log((0'0,173'2864], crt=0'0) [173'2865 (173'2864) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143754 2014-06-12 08:46:03.326064]
2014-06-12 14:47:00.950564 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2865 (0'0,173'2865] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2863 active] add_log_entry 173'2865 (173'2864) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143754 2014-06-12 08:46:03.326064
2014-06-12 14:47:00.950587 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2865 (0'0,173'2865] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2863 active] append_log  adding 1 keys
2014-06-12 14:47:00.950623 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2865, trimmed: 
2014-06-12 14:47:00.950647 7f9d62bd6700  5 -- op tracker -- , seq: 748238, time: 2014-06-12 14:47:00.950643, event: started, op: osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.950657 7f9d62bd6700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:00.950661 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.950663 7f9d62bd6700  5 queue_op 0x1fb3e420 seq 2631 osr(3.402 0x276d620) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:00.950670 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2865 (0'0,173'2865] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2863 active] do_sub_op osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.950685 7f9d62bd6700 10 osd.2 173 dequeue_op 0x20e87a00 finish
2014-06-12 14:47:00.950689 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.950693 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.950690 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:47:00.950698 7f9d82171700  5 _do_op 0x1fb3e420 seq 2631 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:47:00.950703 7f9d82171700 10 _do_transaction on 0x2720b70
2014-06-12 14:47:00.950707 7f9d82171700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.950825 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.950841 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.950880 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.950896 7f9d82171700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.950901 7f9d82171700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:47:00.950905 7f9d82171700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.950917 7f9d82171700 15 _write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 487424~516096
2014-06-12 14:47:00.951015 7f9d82171700 10 lookup_strip_header done  cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.951260 7f9d82171700 10 file_to_extents done 
2014-06-12 14:47:00.952518 7f9d82171700 10 _generic_write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 487424~516096 = 0
2014-06-12 14:47:00.952555 7f9d82171700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.952564 7f9d82171700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:00.952568 7f9d82171700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.952573 7f9d82171700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:00.952708 7f9d82171700 20 set_header setting 15936 cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 parent seq 0
2014-06-12 14:47:00.952759 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.952769 7f9d82171700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:47:00.952775 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:47:00.954958 7f9d82171700 10 _do_op 0x1fb3e420 seq 2631 r = 0, finisher 0x7160040 0
2014-06-12 14:47:00.954972 7f9d82171700 10 _finish_op 0x1fb3e420 seq 2631 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:00.954977 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:47:00.954980 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.954987 7f9d7715b700  5 -- op tracker -- , seq: 748238, time: 2014-06-12 14:47:00.954979, event: commit_sent, op: osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.954995 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2865 (0'0,173'2865] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2863 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:47:00.955028 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:00.955060 7f9d7795c700  5 -- op tracker -- , seq: 748238, time: 2014-06-12 14:47:00.955051, event: sub_op_applied, op: osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.955070 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2865 (0'0,173'2865] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2864 active] sub_op_modify_applied on 0x2720a80 op osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.955088 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2865 (0'0,173'2865] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2864 active] op_applied on primary on version 173'2865
2014-06-12 14:47:00.955100 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2865 (0'0,173'2865] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2864 active] op_applied on replica on version 173'2865
2014-06-12 14:47:00.955139 7f9d7795c700  5 -- op tracker -- , seq: 748238, time: 2014-06-12 14:47:00.955133, event: done, op: osd_sub_op(client.4290.0:143754 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2865 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.964029 7f9d5bec4700  5 -- op tracker -- , seq: 748239, time: 2014-06-12 14:47:00.962887, event: header_read, op: osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.964051 7f9d5bec4700  5 -- op tracker -- , seq: 748239, time: 2014-06-12 14:47:00.962890, event: throttled, op: osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.964062 7f9d5bec4700  5 -- op tracker -- , seq: 748239, time: 2014-06-12 14:47:00.963971, event: all_read, op: osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.964073 7f9d5bec4700  5 -- op tracker -- , seq: 748239, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.964080 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:00.964089 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:47:00.964108 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e86f00 prio 127 cost 516903 latency 0.001220 osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.964163 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.964180 7f9d683e1700 10 osd.2 173 dequeue_op 0x20e86f00 prio 127 cost 516903 latency 0.001292 osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2865 (0'0,173'2865] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2864 active]
2014-06-12 14:47:00.964206 7f9d683e1700  5 -- op tracker -- , seq: 748239, time: 2014-06-12 14:47:00.964202, event: reached_pg, op: osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.964218 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2865 (0'0,173'2865] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2864 active] handle_message: 0x20e86f00
2014-06-12 14:47:00.964228 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2865 (0'0,173'2865] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2864 active] sub_op_modify trans eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 v 173'2866 (transaction) 174
2014-06-12 14:47:00.964242 7f9d683e1700  5 -- op tracker -- , seq: 748239, time: 2014-06-12 14:47:00.964239, event: started, op: osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.964258 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2865 (0'0,173'2865] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2864 active] append_log log((0'0,173'2865], crt=0'0) [173'2866 (173'2865) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143755 2014-06-12 08:46:03.338888]
2014-06-12 14:47:00.964280 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2866 (0'0,173'2866] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2864 active] add_log_entry 173'2866 (173'2865) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143755 2014-06-12 08:46:03.338888
2014-06-12 14:47:00.964299 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2866 (0'0,173'2866] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2864 active] append_log  adding 1 keys
2014-06-12 14:47:00.964328 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2866, trimmed: 
2014-06-12 14:47:00.964349 7f9d683e1700  5 -- op tracker -- , seq: 748239, time: 2014-06-12 14:47:00.964345, event: started, op: osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.964358 7f9d683e1700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:00.964361 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.964363 7f9d683e1700  5 queue_op 0xed4c300 seq 2632 osr(3.402 0x276d620) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:00.964368 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2866 (0'0,173'2866] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2864 active] do_sub_op osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.964380 7f9d683e1700 10 osd.2 173 dequeue_op 0x20e86f00 finish
2014-06-12 14:47:00.964384 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.964386 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.964387 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:47:00.964394 7f9d7e96a700  5 _do_op 0xed4c300 seq 2632 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:47:00.964398 7f9d7e96a700 10 _do_transaction on 0x2124d30
2014-06-12 14:47:00.964404 7f9d7e96a700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.964553 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.964593 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.964666 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.964688 7f9d7e96a700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.964698 7f9d7e96a700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:47:00.964703 7f9d7e96a700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.964818 7f9d7e96a700 15 _write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 1003520~516096
2014-06-12 14:47:00.965031 7f9d7e96a700 10 lookup_strip_header done  cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.965199 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:47:00.966811 7f9d7e96a700 10 _generic_write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 1003520~516096 = 0
2014-06-12 14:47:00.966855 7f9d7e96a700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.966866 7f9d7e96a700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:00.966871 7f9d7e96a700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.966876 7f9d7e96a700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:00.967012 7f9d7e96a700 20 set_header setting 15936 cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 parent seq 0
2014-06-12 14:47:00.967064 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.967074 7f9d7e96a700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:47:00.967081 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:47:00.968774 7f9d7e96a700 10 _do_op 0xed4c300 seq 2632 r = 0, finisher 0x7bdc580 0
2014-06-12 14:47:00.968787 7f9d7e96a700 10 _finish_op 0xed4c300 seq 2632 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:00.968794 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:47:00.968800 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.968804 7f9d7715b700  5 -- op tracker -- , seq: 748239, time: 2014-06-12 14:47:00.968794, event: commit_sent, op: osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.968818 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2866 (0'0,173'2866] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2864 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:47:00.968839 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:00.968868 7f9d7795c700  5 -- op tracker -- , seq: 748239, time: 2014-06-12 14:47:00.968860, event: sub_op_applied, op: osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.968878 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2866 (0'0,173'2866] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2865 active] sub_op_modify_applied on 0x2124c40 op osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.968893 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2866 (0'0,173'2866] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2865 active] op_applied on primary on version 173'2866
2014-06-12 14:47:00.968902 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2866 (0'0,173'2866] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2865 active] op_applied on replica on version 173'2866
2014-06-12 14:47:00.968921 7f9d7795c700  5 -- op tracker -- , seq: 748239, time: 2014-06-12 14:47:00.968918, event: done, op: osd_sub_op(client.4290.0:143755 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2866 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.977165 7f9d5bec4700  5 -- op tracker -- , seq: 748240, time: 2014-06-12 14:47:00.976253, event: header_read, op: osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.977175 7f9d5bec4700  5 -- op tracker -- , seq: 748240, time: 2014-06-12 14:47:00.976256, event: throttled, op: osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.977198 7f9d5bec4700  5 -- op tracker -- , seq: 748240, time: 2014-06-12 14:47:00.977127, event: all_read, op: osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.977206 7f9d5bec4700  5 -- op tracker -- , seq: 748240, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.977211 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:00.977220 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:47:00.977226 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e87f00 prio 127 cost 516903 latency 0.000973 osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.977291 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.977302 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x20e87f00 prio 127 cost 516903 latency 0.001049 osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2866 (0'0,173'2866] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2865 active]
2014-06-12 14:47:00.977378 7f9d6b3e7700  5 -- op tracker -- , seq: 748240, time: 2014-06-12 14:47:00.977373, event: reached_pg, op: osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.977390 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2866 (0'0,173'2866] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2865 active] handle_message: 0x20e87f00
2014-06-12 14:47:00.977403 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2866 (0'0,173'2866] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2865 active] sub_op_modify trans eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 v 173'2867 (transaction) 174
2014-06-12 14:47:00.977417 7f9d6b3e7700  5 -- op tracker -- , seq: 748240, time: 2014-06-12 14:47:00.977414, event: started, op: osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.977438 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2866 (0'0,173'2866] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2865 active] append_log log((0'0,173'2866], crt=0'0) [173'2867 (173'2866) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143756 2014-06-12 08:46:03.353421]
2014-06-12 14:47:00.977457 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2867 (0'0,173'2867] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2865 active] add_log_entry 173'2867 (173'2866) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143756 2014-06-12 08:46:03.353421
2014-06-12 14:47:00.977474 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2867 (0'0,173'2867] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2865 active] append_log  adding 1 keys
2014-06-12 14:47:00.977497 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2867, trimmed: 
2014-06-12 14:47:00.977515 7f9d6b3e7700  5 -- op tracker -- , seq: 748240, time: 2014-06-12 14:47:00.977512, event: started, op: osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.977523 7f9d6b3e7700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:00.977526 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.977527 7f9d6b3e7700  5 queue_op 0x8335d40 seq 2633 osr(3.402 0x276d620) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:00.977531 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2867 (0'0,173'2867] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2865 active] do_sub_op osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.977541 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x20e87f00 finish
2014-06-12 14:47:00.977551 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.977557 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.977563 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:47:00.977570 7f9d8116f700  5 _do_op 0x8335d40 seq 2633 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:47:00.977574 7f9d8116f700 10 _do_transaction on 0x328a7270
2014-06-12 14:47:00.977588 7f9d8116f700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.977710 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.977728 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.977768 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.977781 7f9d8116f700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.977785 7f9d8116f700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:47:00.977788 7f9d8116f700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.977796 7f9d8116f700 15 _write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 1519616~516096
2014-06-12 14:47:00.977893 7f9d8116f700 10 lookup_strip_header done  cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.977907 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:47:00.979152 7f9d8116f700 10 _generic_write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 1519616~516096 = 0
2014-06-12 14:47:00.979185 7f9d8116f700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.979194 7f9d8116f700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:00.979197 7f9d8116f700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.979201 7f9d8116f700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:00.979338 7f9d8116f700 20 set_header setting 15936 cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 parent seq 0
2014-06-12 14:47:00.979380 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.979389 7f9d8116f700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:47:00.979395 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:47:00.982315 7f9d8116f700 10 _do_op 0x8335d40 seq 2633 r = 0, finisher 0x13d12880 0
2014-06-12 14:47:00.982333 7f9d8116f700 10 _finish_op 0x8335d40 seq 2633 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:00.982347 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:47:00.982352 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.982362 7f9d7715b700  5 -- op tracker -- , seq: 748240, time: 2014-06-12 14:47:00.982350, event: commit_sent, op: osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.982390 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2867 (0'0,173'2867] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2865 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:47:00.982420 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:00.982464 7f9d7795c700  5 -- op tracker -- , seq: 748240, time: 2014-06-12 14:47:00.982449, event: sub_op_applied, op: osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.982483 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2867 (0'0,173'2867] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2866 active] sub_op_modify_applied on 0x328a7180 op osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.982507 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2867 (0'0,173'2867] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2866 active] op_applied on primary on version 173'2867
2014-06-12 14:47:00.982517 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2867 (0'0,173'2867] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2866 active] op_applied on replica on version 173'2867
2014-06-12 14:47:00.982548 7f9d7795c700  5 -- op tracker -- , seq: 748240, time: 2014-06-12 14:47:00.982543, event: done, op: osd_sub_op(client.4290.0:143756 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.988582 7f9d5bec4700  5 -- op tracker -- , seq: 748241, time: 2014-06-12 14:47:00.987619, event: header_read, op: osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.988610 7f9d5bec4700  5 -- op tracker -- , seq: 748241, time: 2014-06-12 14:47:00.987622, event: throttled, op: osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.988619 7f9d5bec4700  5 -- op tracker -- , seq: 748241, time: 2014-06-12 14:47:00.988531, event: all_read, op: osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.988627 7f9d5bec4700  5 -- op tracker -- , seq: 748241, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.988633 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:00.988640 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:47:00.988649 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e87600 prio 127 cost 516903 latency 0.001030 osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.988684 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.988695 7f9d693e3700 10 osd.2 173 dequeue_op 0x20e87600 prio 127 cost 516903 latency 0.001075 osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2867 (0'0,173'2867] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2866 active]
2014-06-12 14:47:00.988728 7f9d693e3700  5 -- op tracker -- , seq: 748241, time: 2014-06-12 14:47:00.988723, event: reached_pg, op: osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.988743 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2867 (0'0,173'2867] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2866 active] handle_message: 0x20e87600
2014-06-12 14:47:00.988758 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2867 (0'0,173'2867] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2866 active] sub_op_modify trans eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 v 173'2868 (transaction) 174
2014-06-12 14:47:00.988776 7f9d693e3700  5 -- op tracker -- , seq: 748241, time: 2014-06-12 14:47:00.988772, event: started, op: osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.988797 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2867 (0'0,173'2867] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2866 active] append_log log((0'0,173'2867], crt=0'0) [173'2868 (173'2867) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143757 2014-06-12 08:46:03.364531]
2014-06-12 14:47:00.988825 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2868 (0'0,173'2868] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2866 active] add_log_entry 173'2868 (173'2867) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143757 2014-06-12 08:46:03.364531
2014-06-12 14:47:00.988852 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2868 (0'0,173'2868] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2866 active] append_log  adding 1 keys
2014-06-12 14:47:00.988889 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2868, trimmed: 
2014-06-12 14:47:00.988917 7f9d693e3700  5 -- op tracker -- , seq: 748241, time: 2014-06-12 14:47:00.988912, event: started, op: osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.988930 7f9d693e3700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:00.988934 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.988936 7f9d693e3700  5 queue_op 0x1efbbce0 seq 2634 osr(3.402 0x276d620) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:00.988947 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:47:00.988954 7f9d7d167700  5 _do_op 0x1efbbce0 seq 2634 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:47:00.988943 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2868 (0'0,173'2868] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2866 active] do_sub_op osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.988958 7f9d7d167700 10 _do_transaction on 0x62c1430
2014-06-12 14:47:00.988958 7f9d693e3700 10 osd.2 173 dequeue_op 0x20e87600 finish
2014-06-12 14:47:00.988962 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.988965 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.988965 7f9d7d167700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.989104 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.989134 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.989191 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.989215 7f9d7d167700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.989224 7f9d7d167700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:47:00.989230 7f9d7d167700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.989348 7f9d7d167700 15 _write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 2035712~516096
2014-06-12 14:47:00.989504 7f9d7d167700 10 lookup_strip_header done  cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.989630 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:47:00.991161 7f9d7d167700 10 _generic_write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 2035712~516096 = 0
2014-06-12 14:47:00.991214 7f9d7d167700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.991225 7f9d7d167700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:00.991229 7f9d7d167700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.991234 7f9d7d167700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:00.991375 7f9d7d167700 20 set_header setting 15936 cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 parent seq 0
2014-06-12 14:47:00.991427 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:00.991438 7f9d7d167700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:47:00.991444 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:47:00.993279 7f9d7d167700 10 _do_op 0x1efbbce0 seq 2634 r = 0, finisher 0x28f51e0 0
2014-06-12 14:47:00.993320 7f9d7715b700  5 -- op tracker -- , seq: 748241, time: 2014-06-12 14:47:00.993310, event: commit_sent, op: osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.993343 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2868 (0'0,173'2868] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2866 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:47:00.993374 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:00.993404 7f9d7d167700 10 _finish_op 0x1efbbce0 seq 2634 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:00.993411 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:47:00.993420 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:00.993429 7f9d7795c700  5 -- op tracker -- , seq: 748241, time: 2014-06-12 14:47:00.993422, event: sub_op_applied, op: osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.993437 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2868 (0'0,173'2868] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2867 active] sub_op_modify_applied on 0x62c1340 op osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.993467 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2868 (0'0,173'2868] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2867 active] op_applied on primary on version 173'2868
2014-06-12 14:47:00.993478 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2868 (0'0,173'2868] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2867 active] op_applied on replica on version 173'2868
2014-06-12 14:47:00.993500 7f9d7795c700  5 -- op tracker -- , seq: 748241, time: 2014-06-12 14:47:00.993496, event: done, op: osd_sub_op(client.4290.0:143757 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.998636 7f9d5bec4700  5 -- op tracker -- , seq: 748242, time: 2014-06-12 14:47:00.997871, event: header_read, op: osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.998649 7f9d5bec4700  5 -- op tracker -- , seq: 748242, time: 2014-06-12 14:47:00.997874, event: throttled, op: osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.998658 7f9d5bec4700  5 -- op tracker -- , seq: 748242, time: 2014-06-12 14:47:00.998598, event: all_read, op: osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.998668 7f9d5bec4700  5 -- op tracker -- , seq: 748242, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.998674 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:00.998681 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:47:00.998690 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e86800 prio 127 cost 516903 latency 0.000819 osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.998724 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:00.998735 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20e86800 prio 127 cost 516903 latency 0.000864 osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2868 (0'0,173'2868] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2867 active]
2014-06-12 14:47:00.998763 7f9d6abe6700  5 -- op tracker -- , seq: 748242, time: 2014-06-12 14:47:00.998759, event: reached_pg, op: osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.998775 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2868 (0'0,173'2868] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2867 active] handle_message: 0x20e86800
2014-06-12 14:47:00.998790 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2868 (0'0,173'2868] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2867 active] sub_op_modify trans eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 v 173'2869 (transaction) 174
2014-06-12 14:47:00.998809 7f9d6abe6700  5 -- op tracker -- , seq: 748242, time: 2014-06-12 14:47:00.998804, event: started, op: osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.998827 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2868 (0'0,173'2868] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2867 active] append_log log((0'0,173'2868], crt=0'0) [173'2869 (173'2868) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143758 2014-06-12 08:46:03.375202]
2014-06-12 14:47:00.998855 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2869 (0'0,173'2869] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2867 active] add_log_entry 173'2869 (173'2868) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143758 2014-06-12 08:46:03.375202
2014-06-12 14:47:00.998880 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2869 (0'0,173'2869] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2867 active] append_log  adding 1 keys
2014-06-12 14:47:00.998916 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2869, trimmed: 
2014-06-12 14:47:00.998943 7f9d6abe6700  5 -- op tracker -- , seq: 748242, time: 2014-06-12 14:47:00.998937, event: started, op: osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:00.998956 7f9d6abe6700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:00.998963 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:00.998965 7f9d6abe6700  5 queue_op 0xcc960c0 seq 2635 osr(3.402 0x276d620) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:00.998976 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:47:00.998982 7f9d7915f700  5 _do_op 0xcc960c0 seq 2635 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:47:00.998971 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2869 (0'0,173'2869] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2867 active] do_sub_op osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:00.998986 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20e86800 finish
2014-06-12 14:47:00.998987 7f9d7915f700 10 _do_transaction on 0x66680f0
2014-06-12 14:47:00.998990 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:00.998993 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:00.998994 7f9d7915f700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.999112 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.999136 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:00.999183 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:00.999208 7f9d7915f700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.999215 7f9d7915f700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:47:00.999222 7f9d7915f700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:00.999327 7f9d7915f700 15 _write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 2551808~516096
2014-06-12 14:47:00.999439 7f9d7915f700 10 lookup_strip_header done  cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:00.999549 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:47:01.000791 7f9d7915f700 10 _generic_write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 2551808~516096 = 0
2014-06-12 14:47:01.000848 7f9d7915f700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:01.000867 7f9d7915f700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:01.000880 7f9d7915f700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:01.000890 7f9d7915f700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:01.001041 7f9d7915f700 20 set_header setting 15936 cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 parent seq 0
2014-06-12 14:47:01.001103 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.001118 7f9d7915f700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:47:01.001133 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:47:01.002667 7f9d7915f700 10 _do_op 0xcc960c0 seq 2635 r = 0, finisher 0x82c0220 0
2014-06-12 14:47:01.002682 7f9d7915f700 10 _finish_op 0xcc960c0 seq 2635 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:01.002689 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:47:01.002699 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.002696 7f9d7715b700  5 -- op tracker -- , seq: 748242, time: 2014-06-12 14:47:01.002688, event: commit_sent, op: osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.002703 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2869 (0'0,173'2869] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2867 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:47:01.002723 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:01.002753 7f9d7795c700  5 -- op tracker -- , seq: 748242, time: 2014-06-12 14:47:01.002746, event: sub_op_applied, op: osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.002764 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2869 (0'0,173'2869] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2868 active] sub_op_modify_applied on 0x6668000 op osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.002782 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2869 (0'0,173'2869] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2868 active] op_applied on primary on version 173'2869
2014-06-12 14:47:01.002794 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2869 (0'0,173'2869] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2868 active] op_applied on replica on version 173'2869
2014-06-12 14:47:01.002820 7f9d7795c700  5 -- op tracker -- , seq: 748242, time: 2014-06-12 14:47:01.002815, event: done, op: osd_sub_op(client.4290.0:143758 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.008336 7f9d5bec4700  5 -- op tracker -- , seq: 748243, time: 2014-06-12 14:47:01.007426, event: header_read, op: osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.008351 7f9d5bec4700  5 -- op tracker -- , seq: 748243, time: 2014-06-12 14:47:01.007428, event: throttled, op: osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.008359 7f9d5bec4700  5 -- op tracker -- , seq: 748243, time: 2014-06-12 14:47:01.008308, event: all_read, op: osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.008368 7f9d5bec4700  5 -- op tracker -- , seq: 748243, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.008374 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:01.008381 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:47:01.008388 7f9d5bec4700 15 osd.2 173 enqueue_op 0x20e86400 prio 127 cost 516903 latency 0.000962 osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.008416 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.008423 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x20e86400 prio 127 cost 516903 latency 0.000996 osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2869 (0'0,173'2869] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2868 active]
2014-06-12 14:47:01.008465 7f9d6a3e5700  5 -- op tracker -- , seq: 748243, time: 2014-06-12 14:47:01.008462, event: reached_pg, op: osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.008480 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2869 (0'0,173'2869] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2868 active] handle_message: 0x20e86400
2014-06-12 14:47:01.008489 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2869 (0'0,173'2869] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2868 active] sub_op_modify trans eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 v 173'2870 (transaction) 174
2014-06-12 14:47:01.008499 7f9d6a3e5700  5 -- op tracker -- , seq: 748243, time: 2014-06-12 14:47:01.008496, event: started, op: osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.008515 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2869 (0'0,173'2869] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2868 active] append_log log((0'0,173'2869], crt=0'0) [173'2870 (173'2869) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143759 2014-06-12 08:46:03.384556]
2014-06-12 14:47:01.008532 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2870 (0'0,173'2870] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2868 active] add_log_entry 173'2870 (173'2869) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143759 2014-06-12 08:46:03.384556
2014-06-12 14:47:01.008547 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2870 (0'0,173'2870] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2868 active] append_log  adding 1 keys
2014-06-12 14:47:01.008572 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2870, trimmed: 
2014-06-12 14:47:01.008589 7f9d6a3e5700  5 -- op tracker -- , seq: 748243, time: 2014-06-12 14:47:01.008586, event: started, op: osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.008598 7f9d6a3e5700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:01.008600 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.008601 7f9d6a3e5700  5 queue_op 0x58db1a0 seq 2636 osr(3.402 0x276d620) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:01.008612 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:47:01.008606 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2870 (0'0,173'2870] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2868 active] do_sub_op osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.008617 7f9d8016d700  5 _do_op 0x58db1a0 seq 2636 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:47:01.008619 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x20e86400 finish
2014-06-12 14:47:01.008621 7f9d8016d700 10 _do_transaction on 0x66657b0
2014-06-12 14:47:01.008623 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.008625 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.008627 7f9d8016d700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:01.008731 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:01.008752 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.008790 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.008808 7f9d8016d700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:01.008813 7f9d8016d700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:47:01.008817 7f9d8016d700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:01.008914 7f9d8016d700 15 _write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 3067904~516096
2014-06-12 14:47:01.009016 7f9d8016d700 10 lookup_strip_header done  cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:01.009104 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:47:01.010292 7f9d8016d700 10 _generic_write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 3067904~516096 = 0
2014-06-12 14:47:01.010323 7f9d8016d700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:01.010333 7f9d8016d700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:01.010339 7f9d8016d700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:01.010344 7f9d8016d700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:01.010482 7f9d8016d700 20 set_header setting 15936 cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 parent seq 0
2014-06-12 14:47:01.010531 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.010541 7f9d8016d700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:47:01.010549 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:47:01.012312 7f9d8016d700 10 _do_op 0x58db1a0 seq 2636 r = 0, finisher 0x7691fa0 0
2014-06-12 14:47:01.012325 7f9d8016d700 10 _finish_op 0x58db1a0 seq 2636 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:01.012332 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:47:01.012344 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.012342 7f9d7715b700  5 -- op tracker -- , seq: 748243, time: 2014-06-12 14:47:01.012333, event: commit_sent, op: osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.012356 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2870 (0'0,173'2870] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2868 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:47:01.012377 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:01.012408 7f9d7795c700  5 -- op tracker -- , seq: 748243, time: 2014-06-12 14:47:01.012401, event: sub_op_applied, op: osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.012418 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2870 (0'0,173'2870] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active] sub_op_modify_applied on 0x66656c0 op osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.012437 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2870 (0'0,173'2870] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active] op_applied on primary on version 173'2870
2014-06-12 14:47:01.012450 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2870 (0'0,173'2870] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active] op_applied on replica on version 173'2870
2014-06-12 14:47:01.012479 7f9d7795c700  5 -- op tracker -- , seq: 748243, time: 2014-06-12 14:47:01.012474, event: done, op: osd_sub_op(client.4290.0:143759 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.020797 7f9d5bec4700  5 -- op tracker -- , seq: 748244, time: 2014-06-12 14:47:01.019895, event: header_read, op: osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.020818 7f9d5bec4700  5 -- op tracker -- , seq: 748244, time: 2014-06-12 14:47:01.019898, event: throttled, op: osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.020826 7f9d5bec4700  5 -- op tracker -- , seq: 748244, time: 2014-06-12 14:47:01.020763, event: all_read, op: osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.020834 7f9d5bec4700  5 -- op tracker -- , seq: 748244, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.020840 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:01.020848 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:47:01.020855 7f9d5bec4700 15 osd.2 173 enqueue_op 0x2b015f00 prio 127 cost 516903 latency 0.000960 osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.020885 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.020893 7f9d66bde700 10 osd.2 173 dequeue_op 0x2b015f00 prio 127 cost 516903 latency 0.000998 osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2870 (0'0,173'2870] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active]
2014-06-12 14:47:01.020913 7f9d66bde700  5 -- op tracker -- , seq: 748244, time: 2014-06-12 14:47:01.020910, event: reached_pg, op: osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.020923 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2870 (0'0,173'2870] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active] handle_message: 0x2b015f00
2014-06-12 14:47:01.020931 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2870 (0'0,173'2870] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active] sub_op_modify trans eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 v 173'2871 (transaction) 174
2014-06-12 14:47:01.020942 7f9d66bde700  5 -- op tracker -- , seq: 748244, time: 2014-06-12 14:47:01.020940, event: started, op: osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.020957 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2870 (0'0,173'2870] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active] append_log log((0'0,173'2870], crt=0'0) [173'2871 (173'2870) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143760 2014-06-12 08:46:03.394371]
2014-06-12 14:47:01.020975 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2871 (0'0,173'2871] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active] add_log_entry 173'2871 (173'2870) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143760 2014-06-12 08:46:03.394371
2014-06-12 14:47:01.020991 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2871 (0'0,173'2871] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active] append_log  adding 1 keys
2014-06-12 14:47:01.021016 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2871, trimmed: 
2014-06-12 14:47:01.021034 7f9d66bde700  5 -- op tracker -- , seq: 748244, time: 2014-06-12 14:47:01.021030, event: started, op: osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.021042 7f9d66bde700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:01.021046 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.021047 7f9d66bde700  5 queue_op 0x2465b00 seq 2637 osr(3.402 0x276d620) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:01.021052 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2871 (0'0,173'2871] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active] do_sub_op osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.021061 7f9d66bde700 10 osd.2 173 dequeue_op 0x2b015f00 finish
2014-06-12 14:47:01.021057 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:47:01.021063 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.021065 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.021065 7f9d79960700  5 _do_op 0x2465b00 seq 2637 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:47:01.021070 7f9d79960700 10 _do_transaction on 0x5fe97b0
2014-06-12 14:47:01.021076 7f9d79960700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:01.021224 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:01.021257 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.021314 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.021339 7f9d79960700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:01.021347 7f9d79960700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:47:01.021353 7f9d79960700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:01.021476 7f9d79960700 15 _write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 3584000~516096
2014-06-12 14:47:01.021645 7f9d79960700 10 lookup_strip_header done  cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:01.021759 7f9d79960700 10 file_to_extents done 
2014-06-12 14:47:01.023442 7f9d79960700 10 _generic_write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 3584000~516096 = 0
2014-06-12 14:47:01.023489 7f9d79960700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:01.023504 7f9d79960700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:01.023511 7f9d79960700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:01.023520 7f9d79960700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:01.023721 7f9d79960700 20 set_header setting 15936 cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 parent seq 0
2014-06-12 14:47:01.023787 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.023802 7f9d79960700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:47:01.023814 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:47:01.025926 7f9d79960700 10 _do_op 0x2465b00 seq 2637 r = 0, finisher 0x1ecc9420 0
2014-06-12 14:47:01.025941 7f9d79960700 10 _finish_op 0x2465b00 seq 2637 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:01.025948 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:47:01.025959 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.025987 7f9d7795c700  5 -- op tracker -- , seq: 748244, time: 2014-06-12 14:47:01.025978, event: sub_op_applied, op: osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.025999 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2871 (0'0,173'2871] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active] sub_op_modify_applied on 0x5fe96c0 op osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.026020 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:01.026040 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2871 (0'0,173'2871] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active] op_applied on primary on version 173'2871
2014-06-12 14:47:01.026049 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2871 (0'0,173'2871] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active] op_applied on replica on version 173'2871
2014-06-12 14:47:01.026068 7f9d7715b700  5 -- op tracker -- , seq: 748244, time: 2014-06-12 14:47:01.026062, event: commit_sent, op: osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.026076 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2871 (0'0,173'2871] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2869 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:47:01.026089 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:01.026109 7f9d7715b700  5 -- op tracker -- , seq: 748244, time: 2014-06-12 14:47:01.026106, event: done, op: osd_sub_op(client.4290.0:143760 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.030001 7f9d5bec4700  5 -- op tracker -- , seq: 748245, time: 2014-06-12 14:47:01.029754, event: header_read, op: osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.030020 7f9d5bec4700  5 -- op tracker -- , seq: 748245, time: 2014-06-12 14:47:01.029756, event: throttled, op: osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.030031 7f9d5bec4700  5 -- op tracker -- , seq: 748245, time: 2014-06-12 14:47:01.029973, event: all_read, op: osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.030038 7f9d5bec4700  5 -- op tracker -- , seq: 748245, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.030043 7f9d5bec4700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:01.030049 7f9d5bec4700 20 osd.2 173 should_share_map osd.25 172.16.96.202:6834/3153 173
2014-06-12 14:47:01.030056 7f9d5bec4700 15 osd.2 173 enqueue_op 0x124e3700 prio 127 cost 95015 latency 0.000301 osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.030084 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.030095 7f9d64bda700 10 osd.2 173 dequeue_op 0x124e3700 prio 127 cost 95015 latency 0.000341 osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.402( v 173'2871 (0'0,173'2871] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2870 active]
2014-06-12 14:47:01.030121 7f9d64bda700  5 -- op tracker -- , seq: 748245, time: 2014-06-12 14:47:01.030116, event: reached_pg, op: osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.030132 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2871 (0'0,173'2871] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2870 active] handle_message: 0x124e3700
2014-06-12 14:47:01.030146 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2871 (0'0,173'2871] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2870 active] sub_op_modify trans eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 v 173'2872 (transaction) 174
2014-06-12 14:47:01.030166 7f9d64bda700  5 -- op tracker -- , seq: 748245, time: 2014-06-12 14:47:01.030161, event: started, op: osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.030184 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2871 (0'0,173'2871] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2870 active] append_log log((0'0,173'2871], crt=0'0) [173'2872 (173'2871) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143761 2014-06-12 08:46:03.408025]
2014-06-12 14:47:01.030212 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2872 (0'0,173'2872] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2870 active] add_log_entry 173'2872 (173'2871) modify   eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 by client.4290.0:143761 2014-06-12 08:46:03.408025
2014-06-12 14:47:01.030237 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2872 (0'0,173'2872] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2870 active] append_log  adding 1 keys
2014-06-12 14:47:01.030273 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2872, trimmed: 
2014-06-12 14:47:01.030301 7f9d64bda700  5 -- op tracker -- , seq: 748245, time: 2014-06-12 14:47:01.030295, event: started, op: osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.030314 7f9d64bda700  5 queue_transactions existing osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:01.030317 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.030319 7f9d64bda700  5 queue_op 0x6d2eae0 seq 2638 osr(3.402 0x276d620) 96501 bytes   (queue has 1 ops and 96501 bytes)
2014-06-12 14:47:01.030329 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5438d80 (1 active)
2014-06-12 14:47:01.030337 7f9d7d968700  5 _do_op 0x6d2eae0 seq 2638 osr(3.402 0x276d620)/0x276d620 start
2014-06-12 14:47:01.030325 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.402( v 173'2872 (0'0,173'2872] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2870 active] do_sub_op osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.030340 7f9d7d968700 10 _do_transaction on 0x128c80f0
2014-06-12 14:47:01.030340 7f9d64bda700 10 osd.2 173 dequeue_op 0x124e3700 finish
2014-06-12 14:47:01.030343 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.030346 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.030345 7f9d7d968700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:01.030444 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:01.030466 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.030504 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.030521 7f9d7d968700 15 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:01.030526 7f9d7d968700 10 _omap_rmkeys meta/1fe1e75a/pglog_3.402/0//-1 r = 0
2014-06-12 14:47:01.030530 7f9d7d968700 15 _omap_setkeys meta/1fe1e75a/pglog_3.402/0//-1
2014-06-12 14:47:01.030629 7f9d7d968700 15 _write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 4100096~94208
2014-06-12 14:47:01.030730 7f9d7d968700 10 lookup_strip_header done  cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:01.030744 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:47:01.030999 7f9d7d968700 10 _generic_write 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 4100096~94208 = 0
2014-06-12 14:47:01.031010 7f9d7d968700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:01.031021 7f9d7d968700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:01.031025 7f9d7d968700 15 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3
2014-06-12 14:47:01.031030 7f9d7d968700 10 _setattrs 3.402_head/eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 = 0
2014-06-12 14:47:01.031112 7f9d7d968700 20 set_header setting 15936 cid 3.402_head oid eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 parent seq 0
2014-06-12 14:47:01.031128 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.031137 7f9d7d968700 20 set_header setting 2254 cid meta oid 1fe1e75a/pglog_3.402/0//-1 parent seq 0
2014-06-12 14:47:01.031143 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:47:01.031438 7f9d7d968700 10 _do_op 0x6d2eae0 seq 2638 r = 0, finisher 0x82c0980 0
2014-06-12 14:47:01.031446 7f9d7d968700 10 _finish_op 0x6d2eae0 seq 2638 osr(3.402 0x276d620)/0x276d620
2014-06-12 14:47:01.031451 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5438d80 (0 active)
2014-06-12 14:47:01.031452 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.031462 7f9d7715b700  5 -- op tracker -- , seq: 748245, time: 2014-06-12 14:47:01.031455, event: commit_sent, op: osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.031470 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2872 (0'0,173'2872] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2870 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.25
2014-06-12 14:47:01.031494 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:01.031524 7f9d7795c700  5 -- op tracker -- , seq: 748245, time: 2014-06-12 14:47:01.031519, event: sub_op_applied, op: osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.031531 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2872 (0'0,173'2872] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2871 active] sub_op_modify_applied on 0x128c8000 op osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.031542 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2872 (0'0,173'2872] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2871 active] op_applied on primary on version 173'2872
2014-06-12 14:47:01.031549 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.402( v 173'2872 (0'0,173'2872] local-les=149 n=287 ec=23 les/c 149/149 148/148/148) [25,2] r=1 lpr=148 pi=23-147/12 luod=0'0 crt=0'0 lcod 173'2871 active] op_applied on replica on version 173'2872
2014-06-12 14:47:01.031562 7f9d7795c700  5 -- op tracker -- , seq: 748245, time: 2014-06-12 14:47:01.031560, event: done, op: osd_sub_op(client.4290.0:143761 3.402 eae67c02/rbd_data.10996b8b4567.0000000000001036/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.041928 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:01.041985 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:01.109477 7f9d56432700  5 -- op tracker -- , seq: 748246, time: 2014-06-12 14:47:01.108546, event: header_read, op: osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.109496 7f9d56432700  5 -- op tracker -- , seq: 748246, time: 2014-06-12 14:47:01.108550, event: throttled, op: osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.109505 7f9d56432700  5 -- op tracker -- , seq: 748246, time: 2014-06-12 14:47:01.109423, event: all_read, op: osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.109512 7f9d56432700  5 -- op tracker -- , seq: 748246, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.109518 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:01.109525 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:01.109533 7f9d56432700 15 osd.2 173 enqueue_op 0x25e43700 prio 127 cost 516903 latency 0.000987 osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.109565 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.109575 7f9d69be4700 10 osd.2 173 dequeue_op 0x25e43700 prio 127 cost 516903 latency 0.001028 osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2648 (0'0,173'2648] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2647 active]
2014-06-12 14:47:01.109601 7f9d69be4700  5 -- op tracker -- , seq: 748246, time: 2014-06-12 14:47:01.109596, event: reached_pg, op: osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.109610 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2648 (0'0,173'2648] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2647 active] handle_message: 0x25e43700
2014-06-12 14:47:01.109634 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2648 (0'0,173'2648] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2647 active] sub_op_modify trans af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 v 173'2649 (transaction) 174
2014-06-12 14:47:01.109652 7f9d69be4700  5 -- op tracker -- , seq: 748246, time: 2014-06-12 14:47:01.109648, event: started, op: osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.109672 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2648 (0'0,173'2648] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2647 active] append_log log((0'0,173'2648], crt=0'0) [173'2649 (173'2647) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147307 2014-06-12 08:46:03.485836]
2014-06-12 14:47:01.109697 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2649 (0'0,173'2649] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2647 active] add_log_entry 173'2649 (173'2647) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147307 2014-06-12 08:46:03.485836
2014-06-12 14:47:01.109721 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2649 (0'0,173'2649] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2647 active] append_log  adding 1 keys
2014-06-12 14:47:01.109755 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2649, trimmed: 
2014-06-12 14:47:01.109780 7f9d69be4700  5 -- op tracker -- , seq: 748246, time: 2014-06-12 14:47:01.109776, event: started, op: osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.109790 7f9d69be4700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:01.109795 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.109797 7f9d69be4700  5 queue_op 0x7418a80 seq 2469 osr(3.1c7 0x276d170) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:01.109813 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:47:01.109822 7f9d7c165700  5 _do_op 0x7418a80 seq 2469 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:47:01.109809 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2649 (0'0,173'2649] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2647 active] do_sub_op osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.109824 7f9d69be4700 10 osd.2 173 dequeue_op 0x25e43700 finish
2014-06-12 14:47:01.109826 7f9d7c165700 10 _do_transaction on 0x27756630
2014-06-12 14:47:01.109828 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.109830 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.109832 7f9d7c165700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.109976 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.109996 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.110051 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.110068 7f9d7c165700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.110075 7f9d7c165700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:47:01.110081 7f9d7c165700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.110093 7f9d7c165700 15 _write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 1933312~516096
2014-06-12 14:47:01.110247 7f9d7c165700 10 lookup_strip_header done  cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:47:01.110371 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:47:01.112359 7f9d7c165700 10 _generic_write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 1933312~516096 = 0
2014-06-12 14:47:01.112408 7f9d7c165700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:47:01.112421 7f9d7c165700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:47:01.112427 7f9d7c165700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:47:01.112432 7f9d7c165700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:47:01.112585 7f9d7c165700 20 set_header setting 17479 cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 parent seq 0
2014-06-12 14:47:01.112655 7f9d7c165700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:47:01.112672 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.112680 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:47:01.115371 7f9d7c165700 10 _do_op 0x7418a80 seq 2469 r = 0, finisher 0x6600580 0
2014-06-12 14:47:01.115391 7f9d7c165700 10 _finish_op 0x7418a80 seq 2469 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:01.115398 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:47:01.115408 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.115408 7f9d7715b700  5 -- op tracker -- , seq: 748246, time: 2014-06-12 14:47:01.115398, event: commit_sent, op: osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.115417 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2649 (0'0,173'2649] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2647 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:47:01.115450 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:01.115482 7f9d43bdc700  5 -- op tracker -- , seq: 748247, time: 2014-06-12 14:47:01.111611, event: header_read, op: osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.115491 7f9d43bdc700  5 -- op tracker -- , seq: 748247, time: 2014-06-12 14:47:01.111627, event: throttled, op: osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.115496 7f9d43bdc700  5 -- op tracker -- , seq: 748247, time: 2014-06-12 14:47:01.115457, event: all_read, op: osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.115505 7f9d43bdc700  5 -- op tracker -- , seq: 748247, time: 0.000000, event: dispatched, op: osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.115511 7f9d43bdc700 20 osd.2 173 should_share_map client.4310 172.16.96.220:0/1244085 173
2014-06-12 14:47:01.115534 7f9d43bdc700 15 osd.2 173 enqueue_op 0x22b3b100 prio 63 cost 1548288 latency 0.003922 osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.115561 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.115569 7f9d653db700 10 osd.2 173 dequeue_op 0x22b3b100 prio 63 cost 1548288 latency 0.003958 osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]
2014-06-12 14:47:01.115589 7f9d653db700  5 -- op tracker -- , seq: 748247, time: 2014-06-12 14:47:01.115585, event: reached_pg, op: osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.115596 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:01.115606 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] handle_message: 0x22b3b100
2014-06-12 14:47:01.115613 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] do_op osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:01.115632 7f9d653db700 15 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 '_'
2014-06-12 14:47:01.115797 7f9d653db700 10 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 '_' = 0
2014-06-12 14:47:01.115812 7f9d653db700 15 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 'snapset'
2014-06-12 14:47:01.115874 7f9d653db700 10 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 'snapset' = 0
2014-06-12 14:47:01.115883 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] populate_obc_watchers 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.115892 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.115898 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] get_object_context: creating obc from disk: 0x152e4580
2014-06-12 14:47:01.115904 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] get_object_context: 0x152e4580 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 rwstate(none n=0 w=0) oi: 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3(173'2707 client.4310.0:142687 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2707) ssc: 0x5bb9880 snapset: 0=[]:[]+head
2014-06-12 14:47:01.115914 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] find_object_context 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 @head oi=39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3(173'2707 client.4310.0:142687 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2707)
2014-06-12 14:47:01.115931 7f9d653db700  5 -- op tracker -- , seq: 748247, time: 2014-06-12 14:47:01.115927, event: started, op: osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.115938 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] execute_ctx 0xb572400
2014-06-12 14:47:01.115948 7f9d653db700  5 -- op tracker -- , seq: 748247, time: 2014-06-12 14:47:01.115945, event: started, op: osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.115952 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] do_op 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] ov 173'2707 av 173'2708 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:01.115961 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] do_osd_op 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288]
2014-06-12 14:47:01.115967 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:01.115976 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] do_osd_op  write 77824~1548288
2014-06-12 14:47:01.115986 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] do_osd_op_effects on session 0xa703a00
2014-06-12 14:47:01.115993 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] make_writeable 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 snapset=0x5bb98b8  snapc=0=[]
2014-06-12 14:47:01.116001 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] make_writeable 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:01.116007 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] finish_ctx 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 0xb572400 op modify  
2014-06-12 14:47:01.116015 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]  set mtime to 2014-06-12 08:46:03.489003
2014-06-12 14:47:01.116029 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]  final snapset 0=[]:[]+head in 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.116041 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]  zeroing write result code 0
2014-06-12 14:47:01.116049 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] new_repop rep_tid 236377 on osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.116062 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] issue_repop rep_tid 236377 o 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.116080 7f9d653db700  5 -- op tracker -- , seq: 748247, time: 2014-06-12 14:47:01.116077, event: waiting for subops from 23, op: osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.116096 7f9d653db700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:01.116111 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2707 (0'0,173'2707] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] append_log log((0'0,173'2707], crt=0'0) [173'2708 (173'2707) modify   39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 by client.4310.0:142688 2014-06-12 08:46:03.489003]
2014-06-12 14:47:01.116129 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] add_log_entry 173'2708 (173'2707) modify   39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 by client.4310.0:142688 2014-06-12 08:46:03.489003
2014-06-12 14:47:01.116145 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] append_log  adding 1 keys
2014-06-12 14:47:01.116166 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2708, trimmed: 
2014-06-12 14:47:01.116183 7f9d653db700  5 queue_transactions existing osr(3.63e 0x276dc60)/0x276dc60
2014-06-12 14:47:01.116185 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.116186 7f9d653db700  5 queue_op 0x28ab320 seq 2513 osr(3.63e 0x276dc60) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:47:01.116192 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] eval_repop repgather(0x22b78600 173'2708 rep_tid=236377 committed?=0 applied?=0 lock=0 op=osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.116204 7f9d653db700 10 osd.2 173 dequeue_op 0x22b3b100 finish
2014-06-12 14:47:01.116206 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.116208 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.116359 7f9d7795c700  5 -- op tracker -- , seq: 748246, time: 2014-06-12 14:47:01.116351, event: sub_op_applied, op: osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.116369 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2649 (0'0,173'2649] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2648 active] sub_op_modify_applied on 0x27756540 op osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.116387 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2649 (0'0,173'2649] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2648 active] op_applied on primary on version 173'2649
2014-06-12 14:47:01.116398 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2649 (0'0,173'2649] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2648 active] op_applied on replica on version 173'2649
2014-06-12 14:47:01.116422 7f9d7795c700  5 -- op tracker -- , seq: 748246, time: 2014-06-12 14:47:01.116418, event: done, op: osd_sub_op(client.4316.0:147307 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2649 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.117549 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a26c0 (1 active)
2014-06-12 14:47:01.117561 7f9d8096e700  5 _do_op 0x28ab320 seq 2513 osr(3.63e 0x276dc60)/0x276dc60 start
2014-06-12 14:47:01.117566 7f9d8096e700 10 _do_transaction on 0x384e23c0
2014-06-12 14:47:01.117573 7f9d8096e700 15 _omap_setkeys meta/1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:01.117717 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:01.117740 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.117801 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.117927 7f9d8096e700 15 _omap_rmkeys meta/1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:01.117934 7f9d8096e700 10 _omap_rmkeys meta/1fe0c49b/pglog_3.63e/0//-1 r = 0
2014-06-12 14:47:01.117940 7f9d8096e700 15 _omap_setkeys meta/1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:01.117960 7f9d8096e700 15 _write 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 77824~1548288
2014-06-12 14:47:01.118112 7f9d8096e700 10 lookup_strip_header done  cid 3.63e_head oid 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.118256 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:47:01.127342 7f9d8096e700 10 _generic_write 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 77824~1548288 = 0
2014-06-12 14:47:01.127558 7f9d8096e700 15 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.127592 7f9d8096e700 10 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 = 0
2014-06-12 14:47:01.127602 7f9d8096e700 15 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.127611 7f9d8096e700 10 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 = 0
2014-06-12 14:47:01.128109 7f9d8096e700 20 set_header setting 16668 cid 3.63e_head oid 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 parent seq 0
2014-06-12 14:47:01.128271 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.128290 7f9d8096e700 20 set_header setting 2112 cid meta oid 1fe0c49b/pglog_3.63e/0//-1 parent seq 0
2014-06-12 14:47:01.128304 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:47:01.134321 7f9d57543700  5 -- op tracker -- , seq: 748248, time: 2014-06-12 14:47:01.134269, event: header_read, op: osd_sub_op_reply(client.4310.0:142688 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.134332 7f9d57543700  5 -- op tracker -- , seq: 748248, time: 2014-06-12 14:47:01.134271, event: throttled, op: osd_sub_op_reply(client.4310.0:142688 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.134338 7f9d57543700  5 -- op tracker -- , seq: 748248, time: 2014-06-12 14:47:01.134281, event: all_read, op: osd_sub_op_reply(client.4310.0:142688 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.134345 7f9d57543700  5 -- op tracker -- , seq: 748248, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142688 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.134349 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142688 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:01.134355 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:01.134362 7f9d57543700 15 osd.2 173 enqueue_op 0x2519d400 prio 196 cost 0 latency 0.000093 osd_sub_op_reply(client.4310.0:142688 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:01.134673 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.134685 7f9d623d5700 10 osd.2 173 dequeue_op 0x2519d400 prio 196 cost 0 latency 0.000415 osd_sub_op_reply(client.4310.0:142688 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0) v2 pg pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean]
2014-06-12 14:47:01.134708 7f9d623d5700  5 -- op tracker -- , seq: 748248, time: 2014-06-12 14:47:01.134705, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142688 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.134717 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] handle_message: 0x2519d400
2014-06-12 14:47:01.134735 7f9d623d5700  5 -- op tracker -- , seq: 748248, time: 2014-06-12 14:47:01.134732, event: started, op: osd_sub_op_reply(client.4310.0:142688 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.134741 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sub_op_modify_reply: tid 236377 op  ack_type 4 from 23
2014-06-12 14:47:01.134764 7f9d623d5700  5 -- op tracker -- , seq: 748247, time: 2014-06-12 14:47:01.134754, event: sub_op_commit_rec, op: osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.134773 7f9d623d5700 10 osd.2 173 dequeue_op 0x2519d400 finish
2014-06-12 14:47:01.134779 7f9d623d5700  5 -- op tracker -- , seq: 748248, time: 2014-06-12 14:47:01.134776, event: done, op: osd_sub_op_reply(client.4310.0:142688 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.134799 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.134801 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.144091 7f9d8096e700 10 _do_op 0x28ab320 seq 2513 r = 0, finisher 0x2501a00 0x3393fa40
2014-06-12 14:47:01.144105 7f9d8096e700 10 _finish_op 0x28ab320 seq 2513 osr(3.63e 0x276dc60)/0x276dc60
2014-06-12 14:47:01.144111 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a26c0 (0 active)
2014-06-12 14:47:01.144115 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.144113 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] op_commit: 236377
2014-06-12 14:47:01.144146 7f9d7715b700  5 -- op tracker -- , seq: 748247, time: 2014-06-12 14:47:01.144132, event: op_commit, op: osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.144153 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2707 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] repop_all_committed: repop tid 236377 all committed 
2014-06-12 14:47:01.144161 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean] eval_repop repgather(0x22b78600 173'2708 rep_tid=236377 committed?=1 applied?=0 lock=0 op=osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.144173 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean] log_op_stats osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 0.000000 lat 0.032561
2014-06-12 14:47:01.144185 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean] publish_stats_to_osd 173:3559
2014-06-12 14:47:01.144192 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean]  sending commit on repgather(0x22b78600 173'2708 rep_tid=236377 committed?=1 applied?=0 lock=0 op=osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) 0x1d54ef00
2014-06-12 14:47:01.144217 7f9d7715b700  5 -- op tracker -- , seq: 748247, time: 2014-06-12 14:47:01.144214, event: commit_sent, op: osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.144233 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean] op_applied: 236377
2014-06-12 14:47:01.144253 7f9d7795c700  5 -- op tracker -- , seq: 748247, time: 2014-06-12 14:47:01.144249, event: op_applied, op: osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.144259 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean] op_applied on primary on version 173'2708
2014-06-12 14:47:01.144265 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean] repop_all_applied: repop tid 236377 all applied 
2014-06-12 14:47:01.144270 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean] eval_repop repgather(0x22b78600 173'2708 rep_tid=236377 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.144279 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean] log_op_stats osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 0.000000 lat 0.032668
2014-06-12 14:47:01.144288 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean] publish_stats_to_osd 173:3560
2014-06-12 14:47:01.144295 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2706 active+clean]  requeue_ops 
2014-06-12 14:47:01.144301 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  removing repgather(0x22b78600 173'2708 rep_tid=236377 committed?=1 applied?=1 op=osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.144321 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]    q front is repgather(0x22b78600 173'2708 rep_tid=236377 committed?=1 applied?=1 op=osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.144329 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] remove_repop repgather(0x22b78600 173'2708 rep_tid=236377 committed?=1 applied?=1 op=osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.144337 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  requeue_ops 
2014-06-12 14:47:01.144354 7f9d7795c700  5 -- op tracker -- , seq: 748247, time: 2014-06-12 14:47:01.144352, event: done, op: osd_op(client.4310.0:142688 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 77824~1548288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.150451 7f9d43bdc700  5 -- op tracker -- , seq: 748249, time: 2014-06-12 14:47:01.148141, event: header_read, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.150466 7f9d43bdc700  5 -- op tracker -- , seq: 748249, time: 2014-06-12 14:47:01.148150, event: throttled, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.150476 7f9d43bdc700  5 -- op tracker -- , seq: 748249, time: 2014-06-12 14:47:01.150417, event: all_read, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.150485 7f9d43bdc700  5 -- op tracker -- , seq: 748249, time: 0.000000, event: dispatched, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.150491 7f9d43bdc700 20 osd.2 173 should_share_map client.4310 172.16.96.220:0/1244085 173
2014-06-12 14:47:01.150514 7f9d43bdc700 15 osd.2 173 enqueue_op 0x2b015000 prio 63 cost 1036288 latency 0.002373 osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.150539 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.150547 7f9d68be2700 10 osd.2 173 dequeue_op 0x2b015000 prio 63 cost 1036288 latency 0.002406 osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]
2014-06-12 14:47:01.150566 7f9d68be2700  5 -- op tracker -- , seq: 748249, time: 2014-06-12 14:47:01.150562, event: reached_pg, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.150572 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:01.150581 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] handle_message: 0x2b015000
2014-06-12 14:47:01.150587 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_op osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:01.150601 7f9d68be2700 15 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 '_'
2014-06-12 14:47:01.150780 7f9d68be2700 10 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 '_' = 0
2014-06-12 14:47:01.150791 7f9d68be2700 15 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 'snapset'
2014-06-12 14:47:01.150853 7f9d68be2700 10 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 'snapset' = 0
2014-06-12 14:47:01.150860 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] populate_obc_watchers 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.150867 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.150873 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] get_object_context: creating obc from disk: 0x299e58c0
2014-06-12 14:47:01.150878 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] get_object_context: 0x299e58c0 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 rwstate(none n=0 w=0) oi: 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3(173'2708 client.4310.0:142688 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2708) ssc: 0x407fea0 snapset: 0=[]:[]+head
2014-06-12 14:47:01.150887 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] find_object_context 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 @head oi=39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3(173'2708 client.4310.0:142688 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2708)
2014-06-12 14:47:01.150901 7f9d68be2700  5 -- op tracker -- , seq: 748249, time: 2014-06-12 14:47:01.150898, event: started, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.150906 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] execute_ctx 0x2a83800
2014-06-12 14:47:01.150916 7f9d68be2700  5 -- op tracker -- , seq: 748249, time: 2014-06-12 14:47:01.150913, event: started, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.150921 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_op 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] ov 173'2708 av 173'2709 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:01.150930 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288]
2014-06-12 14:47:01.150936 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:01.150943 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op  write 1626112~1036288
2014-06-12 14:47:01.150951 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] do_osd_op_effects on session 0xa703a00
2014-06-12 14:47:01.150957 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] make_writeable 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 snapset=0x407fed8  snapc=0=[]
2014-06-12 14:47:01.150963 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] make_writeable 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:01.150969 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] finish_ctx 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 0x2a83800 op modify  
2014-06-12 14:47:01.150976 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  set mtime to 2014-06-12 08:46:03.526476
2014-06-12 14:47:01.150987 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  final snapset 0=[]:[]+head in 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.150997 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  zeroing write result code 0
2014-06-12 14:47:01.151003 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] new_repop rep_tid 236378 on osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.151012 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] issue_repop rep_tid 236378 o 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.151025 7f9d68be2700  5 -- op tracker -- , seq: 748249, time: 2014-06-12 14:47:01.151022, event: waiting for subops from 23, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.151038 7f9d68be2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:01.151052 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2708 (0'0,173'2708] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] append_log log((0'0,173'2708], crt=0'0) [173'2709 (173'2708) modify   39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 by client.4310.0:142689 2014-06-12 08:46:03.526476]
2014-06-12 14:47:01.151066 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] add_log_entry 173'2709 (173'2708) modify   39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 by client.4310.0:142689 2014-06-12 08:46:03.526476
2014-06-12 14:47:01.151079 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] append_log  adding 1 keys
2014-06-12 14:47:01.151096 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2709, trimmed: 
2014-06-12 14:47:01.151109 7f9d68be2700  5 queue_transactions existing osr(3.63e 0x276dc60)/0x276dc60
2014-06-12 14:47:01.151112 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.151113 7f9d68be2700  5 queue_op 0x782dd40 seq 2514 osr(3.63e 0x276dc60) 1038581 bytes   (queue has 1 ops and 1038581 bytes)
2014-06-12 14:47:01.151121 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a26c0 (1 active)
2014-06-12 14:47:01.151117 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] eval_repop repgather(0x24e8db00 173'2709 rep_tid=236378 committed?=0 applied?=0 lock=0 op=osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.151129 7f9d68be2700 10 osd.2 173 dequeue_op 0x2b015000 finish
2014-06-12 14:47:01.151128 7f9d7c966700  5 _do_op 0x782dd40 seq 2514 osr(3.63e 0x276dc60)/0x276dc60 start
2014-06-12 14:47:01.151131 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.151131 7f9d7c966700 10 _do_transaction on 0x24e8c0c0
2014-06-12 14:47:01.151133 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.151136 7f9d7c966700 15 _omap_setkeys meta/1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:01.151247 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:01.151275 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.151331 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.151439 7f9d7c966700 15 _omap_rmkeys meta/1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:01.151448 7f9d7c966700 10 _omap_rmkeys meta/1fe0c49b/pglog_3.63e/0//-1 r = 0
2014-06-12 14:47:01.151454 7f9d7c966700 15 _omap_setkeys meta/1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:01.151473 7f9d7c966700 15 _write 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 1626112~1036288
2014-06-12 14:47:01.151610 7f9d7c966700 10 lookup_strip_header done  cid 3.63e_head oid 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.151815 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:47:01.155989 7f9d7c966700 10 _generic_write 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 1626112~1036288 = 0
2014-06-12 14:47:01.156070 7f9d7c966700 15 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.156082 7f9d7c966700 10 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 = 0
2014-06-12 14:47:01.156086 7f9d7c966700 15 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.156092 7f9d7c966700 10 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 = 0
2014-06-12 14:47:01.156287 7f9d7c966700 20 set_header setting 16668 cid 3.63e_head oid 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 parent seq 0
2014-06-12 14:47:01.156357 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.156369 7f9d7c966700 20 set_header setting 2112 cid meta oid 1fe0c49b/pglog_3.63e/0//-1 parent seq 0
2014-06-12 14:47:01.156375 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:47:01.161129 7f9d7c966700 10 _do_op 0x782dd40 seq 2514 r = 0, finisher 0x630d2a0 0x8820080
2014-06-12 14:47:01.161158 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] op_commit: 236378
2014-06-12 14:47:01.161186 7f9d7715b700  5 -- op tracker -- , seq: 748249, time: 2014-06-12 14:47:01.161178, event: op_commit, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.161199 7f9d7c966700 10 _finish_op 0x782dd40 seq 2514 osr(3.63e 0x276dc60)/0x276dc60
2014-06-12 14:47:01.161206 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a26c0 (0 active)
2014-06-12 14:47:01.161209 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.161212 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] op_applied: 236378
2014-06-12 14:47:01.161237 7f9d7795c700  5 -- op tracker -- , seq: 748249, time: 2014-06-12 14:47:01.161230, event: op_applied, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.161249 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] op_applied on primary on version 173'2709
2014-06-12 14:47:01.161711 7f9d57543700  5 -- op tracker -- , seq: 748250, time: 2014-06-12 14:47:01.161673, event: header_read, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ack, result = 0)
2014-06-12 14:47:01.161730 7f9d57543700  5 -- op tracker -- , seq: 748250, time: 2014-06-12 14:47:01.161674, event: throttled, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ack, result = 0)
2014-06-12 14:47:01.161738 7f9d57543700  5 -- op tracker -- , seq: 748250, time: 2014-06-12 14:47:01.161684, event: all_read, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ack, result = 0)
2014-06-12 14:47:01.161746 7f9d57543700  5 -- op tracker -- , seq: 748250, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ack, result = 0)
2014-06-12 14:47:01.161751 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:01.161757 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:01.161764 7f9d57543700 15 osd.2 173 enqueue_op 0x2519c000 prio 196 cost 0 latency 0.000091 osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ack, result = 0) v2
2014-06-12 14:47:01.161788 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.161796 7f9d643d9700 10 osd.2 173 dequeue_op 0x2519c000 prio 196 cost 0 latency 0.000122 osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ack, result = 0) v2 pg pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]
2014-06-12 14:47:01.161816 7f9d643d9700  5 -- op tracker -- , seq: 748250, time: 2014-06-12 14:47:01.161812, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ack, result = 0)
2014-06-12 14:47:01.161827 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] handle_message: 0x2519c000
2014-06-12 14:47:01.161835 7f9d57543700  5 -- op tracker -- , seq: 748251, time: 2014-06-12 14:47:01.161783, event: header_read, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.161845 7f9d57543700  5 -- op tracker -- , seq: 748251, time: 2014-06-12 14:47:01.161784, event: throttled, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.161865 7f9d57543700  5 -- op tracker -- , seq: 748251, time: 2014-06-12 14:47:01.161795, event: all_read, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.161875 7f9d57543700  5 -- op tracker -- , seq: 748251, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.161880 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:01.161885 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:01.161886 7f9d643d9700  5 -- op tracker -- , seq: 748250, time: 2014-06-12 14:47:01.161842, event: started, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ack, result = 0)
2014-06-12 14:47:01.161890 7f9d57543700 15 osd.2 173 enqueue_op 0x2519dc00 prio 196 cost 0 latency 0.000107 osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:01.161896 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] sub_op_modify_reply: tid 236378 op  ack_type 1 from 23
2014-06-12 14:47:01.161911 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:01.161915 7f9d643d9700  5 -- op tracker -- , seq: 748249, time: 2014-06-12 14:47:01.161908, event: sub_op_applied_rec, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.161923 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] repop_all_applied: repop tid 236378 all applied 
2014-06-12 14:47:01.161934 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] eval_repop repgather(0x24e8db00 173'2709 rep_tid=236378 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.161950 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]  sending ack on repgather(0x24e8db00 173'2709 rep_tid=236378 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) 0x33770f00
2014-06-12 14:47:01.161983 7f9d643d9700 10 osd.2 173 dequeue_op 0x2519c000 finish
2014-06-12 14:47:01.161993 7f9d643d9700  5 -- op tracker -- , seq: 748250, time: 2014-06-12 14:47:01.161989, event: done, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ack, result = 0)
2014-06-12 14:47:01.162007 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:01.161994 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2519dc00 prio 196 cost 0 latency 0.000211 osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0) v2 pg pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean]
2014-06-12 14:47:01.162011 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.162014 7f9d61bd4700  5 -- op tracker -- , seq: 748251, time: 2014-06-12 14:47:01.162011, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.162024 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] handle_message: 0x2519dc00
2014-06-12 14:47:01.162039 7f9d61bd4700  5 -- op tracker -- , seq: 748251, time: 2014-06-12 14:47:01.162036, event: started, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.162054 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] sub_op_modify_reply: tid 236378 op  ack_type 4 from 23
2014-06-12 14:47:01.162075 7f9d61bd4700  5 -- op tracker -- , seq: 748249, time: 2014-06-12 14:47:01.162068, event: sub_op_commit_rec, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.162083 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2708 crt=0'0 lcod 173'2707 mlcod 173'2707 active+clean] repop_all_committed: repop tid 236378 all committed 
2014-06-12 14:47:01.162096 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2707 active+clean] eval_repop repgather(0x24e8db00 173'2709 rep_tid=236378 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.162115 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2707 active+clean] log_op_stats osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4 inb 1036567 outb 0 rlat 0.013840 lat 0.013973
2014-06-12 14:47:01.162134 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2707 active+clean] publish_stats_to_osd 173:3561
2014-06-12 14:47:01.162148 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2707 active+clean]  sending commit on repgather(0x24e8db00 173'2709 rep_tid=236378 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) 0x18419b80
2014-06-12 14:47:01.162184 7f9d61bd4700  5 -- op tracker -- , seq: 748249, time: 2014-06-12 14:47:01.162179, event: commit_sent, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.162194 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2707 active+clean]  requeue_ops 
2014-06-12 14:47:01.162204 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]  removing repgather(0x24e8db00 173'2709 rep_tid=236378 committed?=1 applied?=1 op=osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.162220 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]    q front is repgather(0x24e8db00 173'2709 rep_tid=236378 committed?=1 applied?=1 op=osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.162236 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] remove_repop repgather(0x24e8db00 173'2709 rep_tid=236378 committed?=1 applied?=1 op=osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.162253 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]  requeue_ops 
2014-06-12 14:47:01.162279 7f9d61bd4700  5 -- op tracker -- , seq: 748249, time: 2014-06-12 14:47:01.162274, event: done, op: osd_op(client.4310.0:142689 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 1626112~1036288] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.162297 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2519dc00 finish
2014-06-12 14:47:01.162307 7f9d61bd4700  5 -- op tracker -- , seq: 748251, time: 2014-06-12 14:47:01.162303, event: done, op: osd_sub_op_reply(client.4310.0:142689 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.162319 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.162322 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.170193 7f9d43bdc700  5 -- op tracker -- , seq: 748252, time: 2014-06-12 14:47:01.167228, event: header_read, op: osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.170206 7f9d43bdc700  5 -- op tracker -- , seq: 748252, time: 2014-06-12 14:47:01.167230, event: throttled, op: osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.170212 7f9d43bdc700  5 -- op tracker -- , seq: 748252, time: 2014-06-12 14:47:01.170158, event: all_read, op: osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.170218 7f9d43bdc700  5 -- op tracker -- , seq: 748252, time: 0.000000, event: dispatched, op: osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.170228 7f9d43bdc700 20 osd.2 173 should_share_map client.4310 172.16.96.220:0/1244085 173
2014-06-12 14:47:01.170258 7f9d43bdc700 15 osd.2 173 enqueue_op 0x2a903f00 prio 63 cost 1531904 latency 0.003030 osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.170287 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.170297 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2a903f00 prio 63 cost 1531904 latency 0.003068 osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]
2014-06-12 14:47:01.170324 7f9d63bd8700  5 -- op tracker -- , seq: 748252, time: 2014-06-12 14:47:01.170318, event: reached_pg, op: osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.170335 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:01.170351 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] handle_message: 0x2a903f00
2014-06-12 14:47:01.170363 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] do_op osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:01.170389 7f9d63bd8700 15 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 '_'
2014-06-12 14:47:01.170604 7f9d63bd8700 10 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 '_' = 0
2014-06-12 14:47:01.170622 7f9d63bd8700 15 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 'snapset'
2014-06-12 14:47:01.170724 7f9d63bd8700 10 getattr 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 'snapset' = 0
2014-06-12 14:47:01.170736 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] populate_obc_watchers 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.170750 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.170762 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] get_object_context: creating obc from disk: 0x135b78c0
2014-06-12 14:47:01.170774 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] get_object_context: 0x135b78c0 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 rwstate(none n=0 w=0) oi: 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3(173'2709 client.4310.0:142689 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2709) ssc: 0x5d5a620 snapset: 0=[]:[]+head
2014-06-12 14:47:01.170792 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] find_object_context 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 @head oi=39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3(173'2709 client.4310.0:142689 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2709)
2014-06-12 14:47:01.170820 7f9d63bd8700  5 -- op tracker -- , seq: 748252, time: 2014-06-12 14:47:01.170814, event: started, op: osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.170830 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] execute_ctx 0x121fe000
2014-06-12 14:47:01.170850 7f9d63bd8700  5 -- op tracker -- , seq: 748252, time: 2014-06-12 14:47:01.170844, event: started, op: osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.170858 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] do_op 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] ov 173'2709 av 173'2710 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:01.170874 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] do_osd_op 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904]
2014-06-12 14:47:01.170887 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:01.170903 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] do_osd_op  write 2662400~1531904
2014-06-12 14:47:01.170920 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] do_osd_op_effects on session 0xa703a00
2014-06-12 14:47:01.170932 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] make_writeable 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 snapset=0x5d5a658  snapc=0=[]
2014-06-12 14:47:01.170947 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] make_writeable 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:01.170963 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] finish_ctx 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 0x121fe000 op modify  
2014-06-12 14:47:01.170977 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]  set mtime to 2014-06-12 08:46:03.544647
2014-06-12 14:47:01.171000 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]  final snapset 0=[]:[]+head in 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.171020 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]  zeroing write result code 0
2014-06-12 14:47:01.171033 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] new_repop rep_tid 236379 on osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.171051 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] issue_repop rep_tid 236379 o 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.171077 7f9d63bd8700  5 -- op tracker -- , seq: 748252, time: 2014-06-12 14:47:01.171072, event: waiting for subops from 23, op: osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.171100 7f9d63bd8700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:01.171125 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2709 (0'0,173'2709] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] append_log log((0'0,173'2709], crt=0'0) [173'2710 (173'2709) modify   39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 by client.4310.0:142690 2014-06-12 08:46:03.544647]
2014-06-12 14:47:01.171151 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] add_log_entry 173'2710 (173'2709) modify   39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 by client.4310.0:142690 2014-06-12 08:46:03.544647
2014-06-12 14:47:01.171174 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] append_log  adding 1 keys
2014-06-12 14:47:01.171208 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2710, trimmed: 
2014-06-12 14:47:01.171231 7f9d63bd8700  5 queue_transactions existing osr(3.63e 0x276dc60)/0x276dc60
2014-06-12 14:47:01.171237 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.171239 7f9d63bd8700  5 queue_op 0x3f10ea0 seq 2515 osr(3.63e 0x276dc60) 1534197 bytes   (queue has 1 ops and 1534197 bytes)
2014-06-12 14:47:01.171249 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54a26c0 (1 active)
2014-06-12 14:47:01.171256 7f9d81970700  5 _do_op 0x3f10ea0 seq 2515 osr(3.63e 0x276dc60)/0x276dc60 start
2014-06-12 14:47:01.171259 7f9d81970700 10 _do_transaction on 0x13b42a80
2014-06-12 14:47:01.171246 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] eval_repop repgather(0x2dc1fb00 173'2710 rep_tid=236379 committed?=0 applied?=0 lock=0 op=osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.171265 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2a903f00 finish
2014-06-12 14:47:01.171265 7f9d81970700 15 _omap_setkeys meta/1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:01.171269 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.171271 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.171361 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:01.171493 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.171549 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.171573 7f9d81970700 15 _omap_rmkeys meta/1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:01.171579 7f9d81970700 10 _omap_rmkeys meta/1fe0c49b/pglog_3.63e/0//-1 r = 0
2014-06-12 14:47:01.171584 7f9d81970700 15 _omap_setkeys meta/1fe0c49b/pglog_3.63e/0//-1
2014-06-12 14:47:01.171600 7f9d81970700 15 _write 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 2662400~1531904
2014-06-12 14:47:01.171875 7f9d81970700 10 lookup_strip_header done  cid 3.63e_head oid 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.172008 7f9d81970700 10 file_to_extents done 
2014-06-12 14:47:01.175904 7f9d81970700 10 _generic_write 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 2662400~1531904 = 0
2014-06-12 14:47:01.176005 7f9d81970700 15 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.176033 7f9d81970700 10 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 = 0
2014-06-12 14:47:01.176041 7f9d81970700 15 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3
2014-06-12 14:47:01.176050 7f9d81970700 10 _setattrs 3.63e_head/39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 = 0
2014-06-12 14:47:01.176313 7f9d81970700 20 set_header setting 16668 cid 3.63e_head oid 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 parent seq 0
2014-06-12 14:47:01.176413 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.176430 7f9d81970700 20 set_header setting 2112 cid meta oid 1fe0c49b/pglog_3.63e/0//-1 parent seq 0
2014-06-12 14:47:01.176442 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:47:01.182329 7f9d81970700 10 _do_op 0x3f10ea0 seq 2515 r = 0, finisher 0x1ae4e660 0x27b4bfc0
2014-06-12 14:47:01.182355 7f9d81970700 10 _finish_op 0x3f10ea0 seq 2515 osr(3.63e 0x276dc60)/0x276dc60
2014-06-12 14:47:01.182360 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] op_commit: 236379
2014-06-12 14:47:01.182373 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54a26c0 (0 active)
2014-06-12 14:47:01.182384 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.182390 7f9d7715b700  5 -- op tracker -- , seq: 748252, time: 2014-06-12 14:47:01.182382, event: op_commit, op: osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.182403 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] op_applied: 236379
2014-06-12 14:47:01.182427 7f9d7795c700  5 -- op tracker -- , seq: 748252, time: 2014-06-12 14:47:01.182420, event: op_applied, op: osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.182441 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] op_applied on primary on version 173'2710
2014-06-12 14:47:01.186460 7f9d57543700  5 -- op tracker -- , seq: 748253, time: 2014-06-12 14:47:01.186427, event: header_read, op: osd_sub_op_reply(client.4310.0:142690 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.186469 7f9d57543700  5 -- op tracker -- , seq: 748253, time: 2014-06-12 14:47:01.186427, event: throttled, op: osd_sub_op_reply(client.4310.0:142690 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.186481 7f9d57543700  5 -- op tracker -- , seq: 748253, time: 2014-06-12 14:47:01.186436, event: all_read, op: osd_sub_op_reply(client.4310.0:142690 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.186486 7f9d57543700  5 -- op tracker -- , seq: 748253, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142690 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.186489 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142690 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:01.186493 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:01.186498 7f9d57543700 15 osd.2 173 enqueue_op 0x31d61e00 prio 196 cost 0 latency 0.000071 osd_sub_op_reply(client.4310.0:142690 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:01.186519 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.186526 7f9d633d7700 10 osd.2 173 dequeue_op 0x31d61e00 prio 196 cost 0 latency 0.000099 osd_sub_op_reply(client.4310.0:142690 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0) v2 pg pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]
2014-06-12 14:47:01.186541 7f9d633d7700  5 -- op tracker -- , seq: 748253, time: 2014-06-12 14:47:01.186539, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142690 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.186547 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] handle_message: 0x31d61e00
2014-06-12 14:47:01.186557 7f9d633d7700  5 -- op tracker -- , seq: 748253, time: 2014-06-12 14:47:01.186555, event: started, op: osd_sub_op_reply(client.4310.0:142690 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.186561 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] sub_op_modify_reply: tid 236379 op  ack_type 4 from 23
2014-06-12 14:47:01.186574 7f9d633d7700  5 -- op tracker -- , seq: 748252, time: 2014-06-12 14:47:01.186569, event: sub_op_commit_rec, op: osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.186580 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] repop_all_applied: repop tid 236379 all applied 
2014-06-12 14:47:01.186586 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] eval_repop repgather(0x2dc1fb00 173'2710 rep_tid=236379 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.186597 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean]  sending ack on repgather(0x2dc1fb00 173'2710 rep_tid=236379 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) 0x30d71400
2014-06-12 14:47:01.186618 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2709 crt=0'0 lcod 173'2708 mlcod 173'2708 active+clean] repop_all_committed: repop tid 236379 all committed 
2014-06-12 14:47:01.186625 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2708 active+clean] eval_repop repgather(0x2dc1fb00 173'2710 rep_tid=236379 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.186636 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2708 active+clean] log_op_stats osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4 inb 1532183 outb 0 rlat 0.019389 lat 0.019407
2014-06-12 14:47:01.186647 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2708 active+clean] publish_stats_to_osd 173:3562
2014-06-12 14:47:01.186654 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2708 active+clean]  sending commit on repgather(0x2dc1fb00 173'2710 rep_tid=236379 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4) 0x33771900
2014-06-12 14:47:01.186674 7f9d633d7700  5 -- op tracker -- , seq: 748252, time: 2014-06-12 14:47:01.186671, event: commit_sent, op: osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.186680 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2708 active+clean]  requeue_ops 
2014-06-12 14:47:01.186687 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean]  removing repgather(0x2dc1fb00 173'2710 rep_tid=236379 committed?=1 applied?=1 op=osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.186697 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean]    q front is repgather(0x2dc1fb00 173'2710 rep_tid=236379 committed?=1 applied?=1 op=osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.186706 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] remove_repop repgather(0x2dc1fb00 173'2710 rep_tid=236379 committed?=1 applied?=1 op=osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.186716 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean]  requeue_ops 
2014-06-12 14:47:01.186732 7f9d633d7700  5 -- op tracker -- , seq: 748252, time: 2014-06-12 14:47:01.186729, event: done, op: osd_op(client.4310.0:142690 rbd_data.10ad6b8b4567.0000000000000f1f [set-alloc-hint object_size 4194304 write_size 4194304,write 2662400~1531904] 3.39d74e3e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.186746 7f9d633d7700 10 osd.2 173 dequeue_op 0x31d61e00 finish
2014-06-12 14:47:01.186751 7f9d633d7700  5 -- op tracker -- , seq: 748253, time: 2014-06-12 14:47:01.186749, event: done, op: osd_sub_op_reply(client.4310.0:142690 3.63e 39d74e3e/rbd_data.10ad6b8b4567.0000000000000f1f/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.186757 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.186759 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.289210 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:01.289215 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:01.289217 7f9d74bfa700 20 osd.2 173 _dispatch 0x1e700000 ping v1
2014-06-12 14:47:01.289219 7f9d74bfa700 10 osd.2 173 ping from client.4306
2014-06-12 14:47:01.289221 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:01.289222 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:01.318672 7f9d53c11700  5 -- op tracker -- , seq: 748254, time: 2014-06-12 14:47:01.316579, event: header_read, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.318687 7f9d53c11700  5 -- op tracker -- , seq: 748254, time: 2014-06-12 14:47:01.316588, event: throttled, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.318697 7f9d53c11700  5 -- op tracker -- , seq: 748254, time: 2014-06-12 14:47:01.318613, event: all_read, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.318705 7f9d53c11700  5 -- op tracker -- , seq: 748254, time: 0.000000, event: dispatched, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.318720 7f9d53c11700 20 osd.2 173 should_share_map client.4300 172.16.96.220:0/1243739 173
2014-06-12 14:47:01.318774 7f9d53c11700 15 osd.2 173 enqueue_op 0xd38fa00 prio 63 cost 438272 latency 0.002194 osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.318827 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.318836 7f9d67be0700 10 osd.2 173 dequeue_op 0xd38fa00 prio 63 cost 438272 latency 0.002257 osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]
2014-06-12 14:47:01.318866 7f9d67be0700  5 -- op tracker -- , seq: 748254, time: 2014-06-12 14:47:01.318860, event: reached_pg, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.318883 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:01.318906 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] handle_message: 0xd38fa00
2014-06-12 14:47:01.318917 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_op osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:01.318943 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] get_object_context: found obc in cache: 0xe9f02c0
2014-06-12 14:47:01.318953 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] get_object_context: 0xe9f02c0 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 rwstate(write n=5 w=0) oi: c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3(173'2540 client.4300.0:140294 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2540) ssc: 0x81ba460 snapset: 0=[]:[]+head
2014-06-12 14:47:01.318969 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] find_object_context c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 @head oi=c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3(173'2540 client.4300.0:140294 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2540)
2014-06-12 14:47:01.318994 7f9d67be0700  5 -- op tracker -- , seq: 748254, time: 2014-06-12 14:47:01.318989, event: started, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.319002 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] execute_ctx 0x30f7e400
2014-06-12 14:47:01.319018 7f9d67be0700  5 -- op tracker -- , seq: 748254, time: 2014-06-12 14:47:01.319013, event: started, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.319028 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_op c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] ov 173'2540 av 173'2541 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:01.319045 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272]
2014-06-12 14:47:01.319055 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:01.319067 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op  write 3756032~438272
2014-06-12 14:47:01.319084 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] do_osd_op_effects on session 0x1b9b7200
2014-06-12 14:47:01.319095 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] make_writeable c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 snapset=0x81ba498  snapc=0=[]
2014-06-12 14:47:01.319104 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] make_writeable c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:01.319114 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] finish_ctx c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 0x30f7e400 op modify  
2014-06-12 14:47:01.319125 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  set mtime to 2014-06-12 08:46:03.695916
2014-06-12 14:47:01.319146 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  final snapset 0=[]:[]+head in c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:47:01.319165 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  zeroing write result code 0
2014-06-12 14:47:01.319177 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] new_repop rep_tid 236380 on osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.319193 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] issue_repop rep_tid 236380 o c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:47:01.319221 7f9d67be0700  5 -- op tracker -- , seq: 748254, time: 2014-06-12 14:47:01.319216, event: waiting for subops from 26, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.319244 7f9d67be0700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:01.319268 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2540 (0'0,173'2540] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] append_log log((0'0,173'2540], crt=0'0) [173'2541 (173'2540) modify   c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 by client.4300.0:140295 2014-06-12 08:46:03.695916]
2014-06-12 14:47:01.319291 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] add_log_entry 173'2541 (173'2540) modify   c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 by client.4300.0:140295 2014-06-12 08:46:03.695916
2014-06-12 14:47:01.319310 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] append_log  adding 1 keys
2014-06-12 14:47:01.319340 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2541, trimmed: 
2014-06-12 14:47:01.319359 7f9d67be0700  5 queue_transactions existing osr(3.1e1 0x276d140)/0x276d140
2014-06-12 14:47:01.319363 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.319364 7f9d67be0700  5 queue_op 0x91abbc0 seq 2378 osr(3.1e1 0x276d140) 440565 bytes   (queue has 1 ops and 440565 bytes)
2014-06-12 14:47:01.319371 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] eval_repop repgather(0x3403a9c0 173'2541 rep_tid=236380 committed?=0 applied?=0 lock=0 op=osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.319389 7f9d67be0700 10 osd.2 173 dequeue_op 0xd38fa00 finish
2014-06-12 14:47:01.319393 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.319396 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.319403 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x553f320 (1 active)
2014-06-12 14:47:01.319408 7f9d7a962700  5 _do_op 0x91abbc0 seq 2378 osr(3.1e1 0x276d140)/0x276d140 start
2014-06-12 14:47:01.319412 7f9d7a962700 10 _do_transaction on 0x53238c0
2014-06-12 14:47:01.319417 7f9d7a962700 15 _omap_setkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:47:01.319547 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:47:01.319680 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.319742 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.319769 7f9d7a962700 15 _omap_rmkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:47:01.319774 7f9d7a962700 10 _omap_rmkeys meta/1fe10f55/pglog_3.1e1/0//-1 r = 0
2014-06-12 14:47:01.319780 7f9d7a962700 15 _omap_setkeys meta/1fe10f55/pglog_3.1e1/0//-1
2014-06-12 14:47:01.319797 7f9d7a962700 15 _write 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 3756032~438272
2014-06-12 14:47:01.319956 7f9d7a962700 10 lookup_strip_header done  cid 3.1e1_head oid c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:47:01.320139 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:47:01.321973 7f9d56432700  5 -- op tracker -- , seq: 748255, time: 2014-06-12 14:47:01.321068, event: header_read, op: osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.321991 7f9d56432700  5 -- op tracker -- , seq: 748255, time: 2014-06-12 14:47:01.321071, event: throttled, op: osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.322001 7f9d56432700  5 -- op tracker -- , seq: 748255, time: 2014-06-12 14:47:01.321928, event: all_read, op: osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.322025 7f9d56432700  5 -- op tracker -- , seq: 748255, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.322032 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:01.322040 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:01.322049 7f9d56432700 15 osd.2 173 enqueue_op 0x25e42200 prio 127 cost 516903 latency 0.000981 osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.322085 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.322092 7f9d673df700 10 osd.2 173 dequeue_op 0x25e42200 prio 127 cost 516903 latency 0.001024 osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2649 (0'0,173'2649] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2648 active]
2014-06-12 14:47:01.322117 7f9d673df700  5 -- op tracker -- , seq: 748255, time: 2014-06-12 14:47:01.322113, event: reached_pg, op: osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.322139 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2649 (0'0,173'2649] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2648 active] handle_message: 0x25e42200
2014-06-12 14:47:01.322153 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2649 (0'0,173'2649] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2648 active] sub_op_modify trans 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 v 173'2650 (transaction) 174
2014-06-12 14:47:01.322173 7f9d673df700  5 -- op tracker -- , seq: 748255, time: 2014-06-12 14:47:01.322169, event: started, op: osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.322191 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2649 (0'0,173'2649] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2648 active] append_log log((0'0,173'2649], crt=0'0) [173'2650 (173'2648) modify   6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 by client.4299.0:145540 2014-06-12 08:46:03.695224]
2014-06-12 14:47:01.322217 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2650 (0'0,173'2650] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2648 active] add_log_entry 173'2650 (173'2648) modify   6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 by client.4299.0:145540 2014-06-12 08:46:03.695224
2014-06-12 14:47:01.322243 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2650 (0'0,173'2650] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2648 active] append_log  adding 1 keys
2014-06-12 14:47:01.322278 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2650, trimmed: 
2014-06-12 14:47:01.322301 7f9d7a962700 10 _generic_write 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 3756032~438272 = 0
2014-06-12 14:47:01.322306 7f9d673df700  5 -- op tracker -- , seq: 748255, time: 2014-06-12 14:47:01.322301, event: started, op: osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.322335 7f9d7a962700 15 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:47:01.322338 7f9d673df700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:01.322343 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.322345 7f9d673df700  5 queue_op 0x15a01440 seq 2470 osr(3.1c7 0x276d170) 518389 bytes   (queue has 2 ops and 958954 bytes)
2014-06-12 14:47:01.322347 7f9d7a962700 10 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:47:01.322352 7f9d7a962700 15 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3
2014-06-12 14:47:01.322357 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (2 active)
2014-06-12 14:47:01.322360 7f9d7a962700 10 _setattrs 3.1e1_head/c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 = 0
2014-06-12 14:47:01.322363 7f9d82972700  5 _do_op 0x15a01440 seq 2470 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:47:01.322353 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2650 (0'0,173'2650] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2648 active] do_sub_op osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.322368 7f9d82972700 10 _do_transaction on 0x23fbeef0
2014-06-12 14:47:01.322369 7f9d673df700 10 osd.2 173 dequeue_op 0x25e42200 finish
2014-06-12 14:47:01.322373 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.322375 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.322374 7f9d82972700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.322510 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.322529 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.322558 7f9d7a962700 20 set_header setting 16580 cid 3.1e1_head oid c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 parent seq 0
2014-06-12 14:47:01.322631 7f9d7a962700 20 set_header setting 2008 cid meta oid 1fe10f55/pglog_3.1e1/0//-1 parent seq 0
2014-06-12 14:47:01.322649 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.322658 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:47:01.326871 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.326912 7f9d82972700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.326919 7f9d82972700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:47:01.326927 7f9d82972700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.326943 7f9d82972700 15 _write 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 1638400~516096
2014-06-12 14:47:01.327129 7f9d82972700 10 lookup_strip_header done  cid 3.1c7_head oid 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3
2014-06-12 14:47:01.327148 7f9d82972700 10 file_to_extents done 
2014-06-12 14:47:01.327148 7f9d7a962700 10 _do_op 0x91abbc0 seq 2378 r = 0, finisher 0x31bdff20 0x7a76b00
2014-06-12 14:47:01.327185 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_commit: 236380
2014-06-12 14:47:01.327233 7f9d7715b700  5 -- op tracker -- , seq: 748254, time: 2014-06-12 14:47:01.327222, event: op_commit, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.327255 7f9d7a962700 10 _finish_op 0x91abbc0 seq 2378 osr(3.1e1 0x276d140)/0x276d140
2014-06-12 14:47:01.327280 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_applied: 236380
2014-06-12 14:47:01.327301 7f9d7795c700  5 -- op tracker -- , seq: 748254, time: 2014-06-12 14:47:01.327295, event: op_applied, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.327308 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] op_applied on primary on version 173'2541
2014-06-12 14:47:01.327334 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x553f320 (1 active)
2014-06-12 14:47:01.327342 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.329629 7f9d82972700 10 _generic_write 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 1638400~516096 = 0
2014-06-12 14:47:01.329693 7f9d82972700 15 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3
2014-06-12 14:47:01.329709 7f9d82972700 10 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 = 0
2014-06-12 14:47:01.329714 7f9d82972700 15 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3
2014-06-12 14:47:01.329721 7f9d82972700 10 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 = 0
2014-06-12 14:47:01.329942 7f9d82972700 20 set_header setting 17711 cid 3.1c7_head oid 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 parent seq 0
2014-06-12 14:47:01.330007 7f9d82972700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:47:01.330023 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.330031 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:47:01.332701 7f9d82972700 10 _do_op 0x15a01440 seq 2470 r = 0, finisher 0x4f091e0 0
2014-06-12 14:47:01.332725 7f9d82972700 10 _finish_op 0x15a01440 seq 2470 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:01.332733 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:47:01.332742 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.332742 7f9d7715b700  5 -- op tracker -- , seq: 748255, time: 2014-06-12 14:47:01.332733, event: commit_sent, op: osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.332753 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2650 (0'0,173'2650] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2648 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:47:01.332789 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:01.332841 7f9d7795c700  5 -- op tracker -- , seq: 748255, time: 2014-06-12 14:47:01.332835, event: sub_op_applied, op: osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.332851 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2650 (0'0,173'2650] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2649 active] sub_op_modify_applied on 0x23fbee00 op osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.332866 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2650 (0'0,173'2650] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2649 active] op_applied on primary on version 173'2650
2014-06-12 14:47:01.332876 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2650 (0'0,173'2650] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2649 active] op_applied on replica on version 173'2650
2014-06-12 14:47:01.332902 7f9d7795c700  5 -- op tracker -- , seq: 748255, time: 2014-06-12 14:47:01.332898, event: done, op: osd_sub_op(client.4299.0:145540 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2650 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.360143 7f9d5a16f700  5 -- op tracker -- , seq: 748256, time: 2014-06-12 14:47:01.355242, event: header_read, op: osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.360159 7f9d5a16f700  5 -- op tracker -- , seq: 748256, time: 2014-06-12 14:47:01.355246, event: throttled, op: osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.360169 7f9d5a16f700  5 -- op tracker -- , seq: 748256, time: 2014-06-12 14:47:01.360098, event: all_read, op: osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.360183 7f9d5a16f700  5 -- op tracker -- , seq: 748256, time: 0.000000, event: dispatched, op: osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.360188 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:01.360196 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:01.360205 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2b014d00 prio 127 cost 877351 latency 0.004963 osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.361645 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.361660 7f9d663dd700 10 osd.2 173 dequeue_op 0x2b014d00 prio 127 cost 877351 latency 0.006418 osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2509 (0'0,173'2509] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2508 active]
2014-06-12 14:47:01.361690 7f9d663dd700  5 -- op tracker -- , seq: 748256, time: 2014-06-12 14:47:01.361685, event: reached_pg, op: osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.361702 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2509 (0'0,173'2509] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2508 active] handle_message: 0x2b014d00
2014-06-12 14:47:01.361715 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2509 (0'0,173'2509] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2508 active] sub_op_modify trans f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 v 173'2510 (transaction) 174
2014-06-12 14:47:01.361734 7f9d663dd700  5 -- op tracker -- , seq: 748256, time: 2014-06-12 14:47:01.361730, event: started, op: osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.361751 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2509 (0'0,173'2509] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2508 active] append_log log((0'0,173'2509], crt=0'0) [173'2510 (173'546) modify   f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 by client.4310.0:142326 2014-06-12 08:44:14.938829]
2014-06-12 14:47:01.361776 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2510 (0'0,173'2510] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2508 active] add_log_entry 173'2510 (173'546) modify   f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 by client.4310.0:142326 2014-06-12 08:44:14.938829
2014-06-12 14:47:01.361800 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2510 (0'0,173'2510] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2508 active] append_log  adding 1 keys
2014-06-12 14:47:01.361833 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2510, trimmed: 
2014-06-12 14:47:01.361863 7f9d663dd700  5 -- op tracker -- , seq: 748256, time: 2014-06-12 14:47:01.361858, event: started, op: osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.361876 7f9d663dd700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:01.361882 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.361884 7f9d663dd700  5 queue_op 0x565b740 seq 2356 osr(3.168 0x276ce40) 878837 bytes   (queue has 1 ops and 878837 bytes)
2014-06-12 14:47:01.361892 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2510 (0'0,173'2510] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2508 active] do_sub_op osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.361907 7f9d663dd700 10 osd.2 173 dequeue_op 0x2b014d00 finish
2014-06-12 14:47:01.361911 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.361914 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.361925 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:47:01.361936 7f9d7895e700  5 _do_op 0x565b740 seq 2356 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:47:01.361943 7f9d7895e700 10 _do_transaction on 0x8dced30
2014-06-12 14:47:01.361948 7f9d7895e700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:01.362143 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:01.362167 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.362227 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.362330 7f9d7895e700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:01.362338 7f9d7895e700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:47:01.362344 7f9d7895e700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:01.362361 7f9d7895e700 15 _write 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 0~876544
2014-06-12 14:47:01.362707 7f9d7895e700 10 lookup_strip_header done  cid 3.168_head oid f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:01.362742 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:47:01.367134 7f9d7895e700 10 _generic_write 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 0~876544 = 0
2014-06-12 14:47:01.367225 7f9d7895e700 15 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:01.367245 7f9d7895e700 10 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 = 0
2014-06-12 14:47:01.367252 7f9d7895e700 15 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:01.367260 7f9d7895e700 10 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 = 0
2014-06-12 14:47:01.367558 7f9d7895e700 20 set_header setting 16446 cid 3.168_head oid f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 parent seq 0
2014-06-12 14:47:01.367679 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.367695 7f9d7895e700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:47:01.367707 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:47:01.372059 7f9d7895e700 10 _do_op 0x565b740 seq 2356 r = 0, finisher 0x31bdfd40 0
2014-06-12 14:47:01.372129 7f9d7715b700  5 -- op tracker -- , seq: 748256, time: 2014-06-12 14:47:01.372116, event: commit_sent, op: osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.372143 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2510 (0'0,173'2510] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2508 active] sub_op_modify_commit on op osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:01.372178 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:01.372217 7f9d7895e700 10 _finish_op 0x565b740 seq 2356 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:01.372240 7f9d7795c700  5 -- op tracker -- , seq: 748256, time: 2014-06-12 14:47:01.372235, event: sub_op_applied, op: osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.372247 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2510 (0'0,173'2510] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2509 active] sub_op_modify_applied on 0x8dcec40 op osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.372262 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2510 (0'0,173'2510] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2509 active] op_applied on primary on version 173'2510
2014-06-12 14:47:01.372271 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2510 (0'0,173'2510] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2509 active] op_applied on replica on version 173'2510
2014-06-12 14:47:01.372309 7f9d7895e700  5 -- op tracker -- , seq: 748256, time: 2014-06-12 14:47:01.372299, event: done, op: osd_sub_op(client.4310.0:142326 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2510 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.372355 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:47:01.372359 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.395166 7f9d56432700  5 -- op tracker -- , seq: 748257, time: 2014-06-12 14:47:01.393435, event: header_read, op: osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.395178 7f9d56432700  5 -- op tracker -- , seq: 748257, time: 2014-06-12 14:47:01.393438, event: throttled, op: osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.395185 7f9d56432700  5 -- op tracker -- , seq: 748257, time: 2014-06-12 14:47:01.395117, event: all_read, op: osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.395193 7f9d56432700  5 -- op tracker -- , seq: 748257, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.395198 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:01.395205 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:01.395212 7f9d56432700 15 osd.2 173 enqueue_op 0x25e43500 prio 127 cost 516903 latency 0.001777 osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.395451 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.395459 7f9d65bdc700 10 osd.2 173 dequeue_op 0x25e43500 prio 127 cost 516903 latency 0.002024 osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2650 (0'0,173'2650] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2649 active]
2014-06-12 14:47:01.395480 7f9d65bdc700  5 -- op tracker -- , seq: 748257, time: 2014-06-12 14:47:01.395476, event: reached_pg, op: osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.395488 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2650 (0'0,173'2650] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2649 active] handle_message: 0x25e43500
2014-06-12 14:47:01.395499 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2650 (0'0,173'2650] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2649 active] sub_op_modify trans af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 v 173'2651 (transaction) 174
2014-06-12 14:47:01.395515 7f9d65bdc700  5 -- op tracker -- , seq: 748257, time: 2014-06-12 14:47:01.395511, event: started, op: osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.395529 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2650 (0'0,173'2650] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2649 active] append_log log((0'0,173'2650], crt=0'0) [173'2651 (173'2649) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147308 2014-06-12 08:46:03.769373]
2014-06-12 14:47:01.395551 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2651 (0'0,173'2651] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2649 active] add_log_entry 173'2651 (173'2649) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147308 2014-06-12 08:46:03.769373
2014-06-12 14:47:01.395577 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2651 (0'0,173'2651] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2649 active] append_log  adding 1 keys
2014-06-12 14:47:01.395607 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2651, trimmed: 
2014-06-12 14:47:01.395627 7f9d65bdc700  5 -- op tracker -- , seq: 748257, time: 2014-06-12 14:47:01.395623, event: started, op: osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.395636 7f9d65bdc700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:01.395639 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.395641 7f9d65bdc700  5 queue_op 0x6fdb0e0 seq 2471 osr(3.1c7 0x276d170) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:01.395653 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:47:01.395648 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2651 (0'0,173'2651] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2649 active] do_sub_op osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.395662 7f9d65bdc700 10 osd.2 173 dequeue_op 0x25e43500 finish
2014-06-12 14:47:01.395662 7f9d83173700  5 _do_op 0x6fdb0e0 seq 2471 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:47:01.395665 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.395667 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.395667 7f9d83173700 10 _do_transaction on 0x2b2af7b0
2014-06-12 14:47:01.395675 7f9d83173700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.395841 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.395862 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.395918 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.396069 7f9d83173700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.396079 7f9d83173700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:47:01.396084 7f9d83173700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.396102 7f9d83173700 15 _write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 2449408~516096
2014-06-12 14:47:01.396253 7f9d83173700 10 lookup_strip_header done  cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:47:01.396444 7f9d83173700 10 file_to_extents done 
2014-06-12 14:47:01.398340 7f9d83173700 10 _generic_write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 2449408~516096 = 0
2014-06-12 14:47:01.398411 7f9d83173700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:47:01.398438 7f9d83173700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:47:01.398445 7f9d83173700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:47:01.398452 7f9d83173700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:47:01.398684 7f9d83173700 20 set_header setting 17479 cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 parent seq 0
2014-06-12 14:47:01.398775 7f9d83173700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:47:01.398790 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.398798 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:47:01.401892 7f9d83173700 10 _do_op 0x6fdb0e0 seq 2471 r = 0, finisher 0x20aadde0 0
2014-06-12 14:47:01.401912 7f9d83173700 10 _finish_op 0x6fdb0e0 seq 2471 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:01.401918 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:47:01.401929 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.401934 7f9d7715b700  5 -- op tracker -- , seq: 748257, time: 2014-06-12 14:47:01.401923, event: commit_sent, op: osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.401949 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2651 (0'0,173'2651] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2649 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:47:01.401976 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:01.402013 7f9d7795c700  5 -- op tracker -- , seq: 748257, time: 2014-06-12 14:47:01.402007, event: sub_op_applied, op: osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.402022 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2651 (0'0,173'2651] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active] sub_op_modify_applied on 0x2b2af6c0 op osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.402038 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2651 (0'0,173'2651] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active] op_applied on primary on version 173'2651
2014-06-12 14:47:01.402048 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2651 (0'0,173'2651] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active] op_applied on replica on version 173'2651
2014-06-12 14:47:01.402075 7f9d7795c700  5 -- op tracker -- , seq: 748257, time: 2014-06-12 14:47:01.402070, event: done, op: osd_sub_op(client.4316.0:147308 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2651 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.481059 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:01.481084 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:01.481087 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:01.481089 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:01.481091 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 28; oldest is 27.815222 seconds old
2014-06-12 14:47:01.481090 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:01.492213 7f9d5a16f700  5 -- op tracker -- , seq: 748258, time: 2014-06-12 14:47:01.491322, event: header_read, op: osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.492225 7f9d5a16f700  5 -- op tracker -- , seq: 748258, time: 2014-06-12 14:47:01.491324, event: throttled, op: osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.492232 7f9d5a16f700  5 -- op tracker -- , seq: 748258, time: 2014-06-12 14:47:01.492175, event: all_read, op: osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.492239 7f9d5a16f700  5 -- op tracker -- , seq: 748258, time: 0.000000, event: dispatched, op: osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.492243 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:01.492250 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:01.492257 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2b015100 prio 127 cost 516903 latency 0.000935 osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.492287 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.492294 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2b015100 prio 127 cost 516903 latency 0.000971 osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2671 (0'0,173'2671] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2670 active]
2014-06-12 14:47:01.492316 7f9d62bd6700  5 -- op tracker -- , seq: 748258, time: 2014-06-12 14:47:01.492312, event: reached_pg, op: osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.492325 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2671 (0'0,173'2671] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2670 active] handle_message: 0x2b015100
2014-06-12 14:47:01.492337 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2671 (0'0,173'2671] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2670 active] sub_op_modify trans 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 v 173'2672 (transaction) 174
2014-06-12 14:47:01.492354 7f9d62bd6700  5 -- op tracker -- , seq: 748258, time: 2014-06-12 14:47:01.492351, event: started, op: osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.492371 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2671 (0'0,173'2671] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2670 active] append_log log((0'0,173'2671], crt=0'0) [173'2672 (173'2671) modify   57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 by client.4282.0:145226 2014-06-12 08:44:32.064150]
2014-06-12 14:47:01.492397 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2672 (0'0,173'2672] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2670 active] add_log_entry 173'2672 (173'2671) modify   57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 by client.4282.0:145226 2014-06-12 08:44:32.064150
2014-06-12 14:47:01.492418 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2672 (0'0,173'2672] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2670 active] append_log  adding 1 keys
2014-06-12 14:47:01.492450 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2672, trimmed: 
2014-06-12 14:47:01.492471 7f9d62bd6700  5 -- op tracker -- , seq: 748258, time: 2014-06-12 14:47:01.492467, event: started, op: osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.492480 7f9d62bd6700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:01.492485 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.492486 7f9d62bd6700  5 queue_op 0x68b1080 seq 2482 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:01.492494 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2672 (0'0,173'2672] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2670 active] do_sub_op osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.492510 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2b015100 finish
2014-06-12 14:47:01.492513 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.492516 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.492525 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:01.492530 7f9d7a161700  5 _do_op 0x68b1080 seq 2482 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:01.492534 7f9d7a161700 10 _do_transaction on 0x30b075f0
2014-06-12 14:47:01.492541 7f9d7a161700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:01.492699 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:01.492728 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.492784 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.492809 7f9d7a161700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:01.492815 7f9d7a161700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:01.492821 7f9d7a161700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:01.493019 7f9d7a161700 15 _write 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 53248~516096
2014-06-12 14:47:01.493178 7f9d7a161700 10 lookup_strip_header done  cid 3.1b7_head oid 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:01.493347 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:47:01.495380 7f9d7a161700 10 _generic_write 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 53248~516096 = 0
2014-06-12 14:47:01.495432 7f9d7a161700 15 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:01.495451 7f9d7a161700 10 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:47:01.495458 7f9d7a161700 15 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:01.495467 7f9d7a161700 10 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:47:01.495697 7f9d7a161700 20 set_header setting 18080 cid 3.1b7_head oid 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 parent seq 0
2014-06-12 14:47:01.495772 7f9d7a161700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:01.495791 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.495801 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:47:01.498722 7f9d7a161700 10 _do_op 0x68b1080 seq 2482 r = 0, finisher 0x4efd3e0 0
2014-06-12 14:47:01.498749 7f9d7a161700 10 _finish_op 0x68b1080 seq 2482 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:01.498758 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:01.498774 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.498772 7f9d7715b700  5 -- op tracker -- , seq: 748258, time: 2014-06-12 14:47:01.498759, event: commit_sent, op: osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.498788 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2672 (0'0,173'2672] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2670 active] sub_op_modify_commit on op osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:01.498824 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:01.501651 7f9d7795c700  5 -- op tracker -- , seq: 748258, time: 2014-06-12 14:47:01.501636, event: sub_op_applied, op: osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.501669 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2672 (0'0,173'2672] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2671 active] sub_op_modify_applied on 0x30b07500 op osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.501692 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2672 (0'0,173'2672] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2671 active] op_applied on primary on version 173'2672
2014-06-12 14:47:01.501702 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2672 (0'0,173'2672] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2671 active] op_applied on replica on version 173'2672
2014-06-12 14:47:01.501729 7f9d7795c700  5 -- op tracker -- , seq: 748258, time: 2014-06-12 14:47:01.501725, event: done, op: osd_sub_op(client.4282.0:145226 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2672 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.525086 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:01.525251 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:01.560702 7f9d56432700  5 -- op tracker -- , seq: 748259, time: 2014-06-12 14:47:01.558206, event: header_read, op: osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.560713 7f9d56432700  5 -- op tracker -- , seq: 748259, time: 2014-06-12 14:47:01.558208, event: throttled, op: osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.560725 7f9d56432700  5 -- op tracker -- , seq: 748259, time: 2014-06-12 14:47:01.560659, event: all_read, op: osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.560730 7f9d56432700  5 -- op tracker -- , seq: 748259, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.560735 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:01.560740 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:01.560746 7f9d56432700 15 osd.2 173 enqueue_op 0x25e42100 prio 127 cost 1549095 latency 0.002540 osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.560775 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.560788 7f9d683e1700 10 osd.2 173 dequeue_op 0x25e42100 prio 127 cost 1549095 latency 0.002581 osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2651 (0'0,173'2651] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active]
2014-06-12 14:47:01.560815 7f9d683e1700  5 -- op tracker -- , seq: 748259, time: 2014-06-12 14:47:01.560810, event: reached_pg, op: osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.560827 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2651 (0'0,173'2651] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active] handle_message: 0x25e42100
2014-06-12 14:47:01.560843 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2651 (0'0,173'2651] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active] sub_op_modify trans 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 v 173'2652 (transaction) 174
2014-06-12 14:47:01.560864 7f9d683e1700  5 -- op tracker -- , seq: 748259, time: 2014-06-12 14:47:01.560859, event: started, op: osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.560884 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2651 (0'0,173'2651] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active] append_log log((0'0,173'2651], crt=0'0) [173'2652 (173'2650) modify   6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 by client.4299.0:145541 2014-06-12 08:46:03.916387]
2014-06-12 14:47:01.560911 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2652 (0'0,173'2652] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active] add_log_entry 173'2652 (173'2650) modify   6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 by client.4299.0:145541 2014-06-12 08:46:03.916387
2014-06-12 14:47:01.560942 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2652 (0'0,173'2652] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active] append_log  adding 1 keys
2014-06-12 14:47:01.560980 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2652, trimmed: 
2014-06-12 14:47:01.561008 7f9d683e1700  5 -- op tracker -- , seq: 748259, time: 2014-06-12 14:47:01.561003, event: started, op: osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.561024 7f9d683e1700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:01.561028 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.561029 7f9d683e1700  5 queue_op 0x5ce47e0 seq 2472 osr(3.1c7 0x276d170) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:47:01.561035 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2652 (0'0,173'2652] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active] do_sub_op osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.561050 7f9d683e1700 10 osd.2 173 dequeue_op 0x25e42100 finish
2014-06-12 14:47:01.561053 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.561056 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.561074 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:47:01.561083 7f9d7f16b700  5 _do_op 0x5ce47e0 seq 2472 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:47:01.561089 7f9d7f16b700 10 _do_transaction on 0x2125270
2014-06-12 14:47:01.561096 7f9d7f16b700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.561269 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.561361 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.561417 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.561437 7f9d7f16b700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.561443 7f9d7f16b700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:47:01.561448 7f9d7f16b700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:01.561461 7f9d7f16b700 15 _write 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 2154496~1548288
2014-06-12 14:47:01.561894 7f9d7f16b700 10 lookup_strip_header done  cid 3.1c7_head oid 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3
2014-06-12 14:47:01.562012 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:47:01.569152 7f9d7f16b700 10 _generic_write 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 2154496~1548288 = 0
2014-06-12 14:47:01.569316 7f9d7f16b700 15 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3
2014-06-12 14:47:01.569340 7f9d7f16b700 10 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 = 0
2014-06-12 14:47:01.569349 7f9d7f16b700 15 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3
2014-06-12 14:47:01.569358 7f9d7f16b700 10 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 = 0
2014-06-12 14:47:01.569746 7f9d7f16b700 20 set_header setting 17711 cid 3.1c7_head oid 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 parent seq 0
2014-06-12 14:47:01.569855 7f9d7f16b700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:47:01.569873 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.569885 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:47:01.574186 7f9d5a16f700  5 -- op tracker -- , seq: 748260, time: 2014-06-12 14:47:01.573233, event: header_read, op: osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.574200 7f9d5a16f700  5 -- op tracker -- , seq: 748260, time: 2014-06-12 14:47:01.573236, event: throttled, op: osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.574209 7f9d5a16f700  5 -- op tracker -- , seq: 748260, time: 2014-06-12 14:47:01.574141, event: all_read, op: osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.574217 7f9d5a16f700  5 -- op tracker -- , seq: 748260, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.574222 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:01.574229 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:01.574243 7f9d5a16f700 15 osd.2 173 enqueue_op 0x9fad300 prio 127 cost 549671 latency 0.001010 osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.574275 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.574285 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x9fad300 prio 127 cost 549671 latency 0.001051 osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2510 (0'0,173'2510] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2509 active]
2014-06-12 14:47:01.574307 7f9d6b3e7700  5 -- op tracker -- , seq: 748260, time: 2014-06-12 14:47:01.574303, event: reached_pg, op: osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.574318 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2510 (0'0,173'2510] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2509 active] handle_message: 0x9fad300
2014-06-12 14:47:01.574331 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2510 (0'0,173'2510] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2509 active] sub_op_modify trans a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 v 173'2511 (transaction) 174
2014-06-12 14:47:01.574348 7f9d6b3e7700  5 -- op tracker -- , seq: 748260, time: 2014-06-12 14:47:01.574344, event: started, op: osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.574368 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2510 (0'0,173'2510] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2509 active] append_log log((0'0,173'2510], crt=0'0) [173'2511 (173'2509) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146445 2014-06-12 08:41:38.092603]
2014-06-12 14:47:01.574393 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2511 (0'0,173'2511] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2509 active] add_log_entry 173'2511 (173'2509) modify   a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 by client.4316.0:146445 2014-06-12 08:41:38.092603
2014-06-12 14:47:01.574415 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2511 (0'0,173'2511] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2509 active] append_log  adding 1 keys
2014-06-12 14:47:01.574447 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2511, trimmed: 
2014-06-12 14:47:01.574476 7f9d6b3e7700  5 -- op tracker -- , seq: 748260, time: 2014-06-12 14:47:01.574471, event: started, op: osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.574487 7f9d6b3e7700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:01.574491 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.574493 7f9d6b3e7700  5 queue_op 0x7fba960 seq 2357 osr(3.168 0x276ce40) 551157 bytes   (queue has 2 ops and 2101738 bytes)
2014-06-12 14:47:01.574502 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (2 active)
2014-06-12 14:47:01.574510 7f9d7e169700  5 _do_op 0x7fba960 seq 2357 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:47:01.574499 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2511 (0'0,173'2511] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2509 active] do_sub_op osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.574513 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x9fad300 finish
2014-06-12 14:47:01.574517 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.574519 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.574535 7f9d7e169700 10 _do_transaction on 0x328a6ef0
2014-06-12 14:47:01.574542 7f9d7e169700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:01.574698 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:01.574721 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.577608 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.577667 7f9d7e169700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:01.577675 7f9d7e169700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:47:01.577683 7f9d7e169700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:01.577821 7f9d7e169700 15 _write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 3645440~548864
2014-06-12 14:47:01.578007 7f9d7e169700 10 lookup_strip_header done  cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:47:01.578170 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:47:01.578923 7f9d7f16b700 10 _do_op 0x5ce47e0 seq 2472 r = 0, finisher 0x276549c0 0
2014-06-12 14:47:01.578941 7f9d7f16b700 10 _finish_op 0x5ce47e0 seq 2472 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:01.578948 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (1 active)
2014-06-12 14:47:01.578956 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.578969 7f9d7795c700  5 -- op tracker -- , seq: 748259, time: 2014-06-12 14:47:01.578960, event: sub_op_applied, op: osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.578977 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2652 (0'0,173'2652] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active] sub_op_modify_applied on 0x2125180 op osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.579003 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:01.579024 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2652 (0'0,173'2652] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active] op_applied on primary on version 173'2652
2014-06-12 14:47:01.579035 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2652 (0'0,173'2652] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active] op_applied on replica on version 173'2652
2014-06-12 14:47:01.579433 7f9d7715b700  5 -- op tracker -- , seq: 748259, time: 2014-06-12 14:47:01.579424, event: commit_sent, op: osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.579450 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2652 (0'0,173'2652] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2650 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:47:01.579475 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:01.579516 7f9d7715b700  5 -- op tracker -- , seq: 748259, time: 2014-06-12 14:47:01.579511, event: done, op: osd_sub_op(client.4299.0:145541 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2652 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.579875 7f9d7e169700 10 _generic_write 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 3645440~548864 = 0
2014-06-12 14:47:01.579940 7f9d7e169700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:47:01.579959 7f9d7e169700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:47:01.579967 7f9d7e169700 15 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3
2014-06-12 14:47:01.579976 7f9d7e169700 10 _setattrs 3.168_head/a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 = 0
2014-06-12 14:47:01.580177 7f9d7e169700 20 set_header setting 16934 cid 3.168_head oid a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 parent seq 0
2014-06-12 14:47:01.580262 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.580281 7f9d7e169700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:47:01.580295 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:47:01.584055 7f9d7e169700 10 _do_op 0x7fba960 seq 2357 r = 0, finisher 0x31d0f240 0
2014-06-12 14:47:01.584102 7f9d7715b700  5 -- op tracker -- , seq: 748260, time: 2014-06-12 14:47:01.584091, event: commit_sent, op: osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.584115 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2511 (0'0,173'2511] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2509 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:01.584142 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:01.584172 7f9d7e169700 10 _finish_op 0x7fba960 seq 2357 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:01.584181 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:47:01.584192 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.584198 7f9d7795c700  5 -- op tracker -- , seq: 748260, time: 2014-06-12 14:47:01.584190, event: sub_op_applied, op: osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.584214 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2511 (0'0,173'2511] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2510 active] sub_op_modify_applied on 0x328a6e00 op osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.584231 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2511 (0'0,173'2511] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2510 active] op_applied on primary on version 173'2511
2014-06-12 14:47:01.584240 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2511 (0'0,173'2511] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2510 active] op_applied on replica on version 173'2511
2014-06-12 14:47:01.584262 7f9d7795c700  5 -- op tracker -- , seq: 748260, time: 2014-06-12 14:47:01.584258, event: done, op: osd_sub_op(client.4316.0:146445 3.168 a9645968/rbd_data.10b36b8b4567.00000000000010bf/head//3 [] v 173'2511 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.648780 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:01.648892 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:01.668081 7f9d43bdc700  5 -- op tracker -- , seq: 748261, time: 2014-06-12 14:47:01.667842, event: header_read, op: osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.668097 7f9d43bdc700  5 -- op tracker -- , seq: 748261, time: 2014-06-12 14:47:01.667846, event: throttled, op: osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.668106 7f9d43bdc700  5 -- op tracker -- , seq: 748261, time: 2014-06-12 14:47:01.668041, event: all_read, op: osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.668122 7f9d43bdc700  5 -- op tracker -- , seq: 748261, time: 0.000000, event: dispatched, op: osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.668130 7f9d43bdc700 20 osd.2 173 should_share_map client.4310 172.16.96.220:0/1244085 173
2014-06-12 14:47:01.668157 7f9d43bdc700 15 osd.2 173 enqueue_op 0x20e89300 prio 63 cost 81920 latency 0.000315 osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.668200 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.668210 7f9d693e3700 10 osd.2 173 dequeue_op 0x20e89300 prio 63 cost 81920 latency 0.000368 osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]
2014-06-12 14:47:01.668238 7f9d693e3700  5 -- op tracker -- , seq: 748261, time: 2014-06-12 14:47:01.668233, event: reached_pg, op: osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.668248 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:01.668261 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] handle_message: 0x20e89300
2014-06-12 14:47:01.668270 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_op osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:01.668295 7f9d693e3700 15 getattr 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 '_'
2014-06-12 14:47:01.668628 7f9d693e3700 10 getattr 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 '_' = 0
2014-06-12 14:47:01.668652 7f9d693e3700 15 getattr 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 'snapset'
2014-06-12 14:47:01.668753 7f9d693e3700 10 getattr 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 'snapset' = 0
2014-06-12 14:47:01.668771 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] populate_obc_watchers 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.668788 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.668797 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] get_object_context: creating obc from disk: 0x37e4b080
2014-06-12 14:47:01.668807 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] get_object_context: 0x37e4b080 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 rwstate(none n=0 w=0) oi: 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3(173'654 client.4310.0:36463 wrlock_by=unknown.0.0:0 dirty s 4194304 uv654) ssc: 0x679a700 snapset: 0=[]:[]+head
2014-06-12 14:47:01.668826 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] find_object_context 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 @head oi=912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3(173'654 client.4310.0:36463 wrlock_by=unknown.0.0:0 dirty s 4194304 uv654)
2014-06-12 14:47:01.668863 7f9d693e3700  5 -- op tracker -- , seq: 748261, time: 2014-06-12 14:47:01.668854, event: started, op: osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.668877 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] execute_ctx 0x2747e400
2014-06-12 14:47:01.668893 7f9d693e3700  5 -- op tracker -- , seq: 748261, time: 2014-06-12 14:47:01.668889, event: started, op: osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.668904 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_op 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] ov 173'654 av 173'2645 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:01.668916 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_osd_op 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920]
2014-06-12 14:47:01.668926 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:01.668938 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_osd_op  write 0~81920
2014-06-12 14:47:01.668956 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_osd_op_effects on session 0xa703a00
2014-06-12 14:47:01.668972 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] make_writeable 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 snapset=0x679a738  snapc=0=[]
2014-06-12 14:47:01.668983 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] make_writeable 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:01.668993 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] finish_ctx 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 0x2747e400 op modify  
2014-06-12 14:47:01.669004 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]  set mtime to 2014-06-12 08:46:04.047843
2014-06-12 14:47:01.669024 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]  final snapset 0=[]:[]+head in 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.669040 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]  zeroing write result code 0
2014-06-12 14:47:01.669050 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] new_repop rep_tid 236381 on osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.669065 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] issue_repop rep_tid 236381 o 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.669088 7f9d693e3700  5 -- op tracker -- , seq: 748261, time: 2014-06-12 14:47:01.669084, event: waiting for subops from 23, op: osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.669111 7f9d693e3700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:01.669132 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2644 (0'0,173'2644] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] append_log log((0'0,173'2644], crt=0'0) [173'2645 (173'654) modify   912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 by client.4310.0:142725 2014-06-12 08:46:04.047843]
2014-06-12 14:47:01.669160 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] add_log_entry 173'2645 (173'654) modify   912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 by client.4310.0:142725 2014-06-12 08:46:04.047843
2014-06-12 14:47:01.669179 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] append_log  adding 1 keys
2014-06-12 14:47:01.669210 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2645, trimmed: 
2014-06-12 14:47:01.669234 7f9d693e3700  5 queue_transactions existing osr(3.76e 0x49b4160)/0x49b4160
2014-06-12 14:47:01.669239 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.669241 7f9d693e3700  5 queue_op 0x83ee780 seq 2411 osr(3.76e 0x49b4160) 84213 bytes   (queue has 1 ops and 84213 bytes)
2014-06-12 14:47:01.669251 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] eval_repop repgather(0x519da40 173'2645 rep_tid=236381 committed?=0 applied?=0 lock=0 op=osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.669270 7f9d693e3700 10 osd.2 173 dequeue_op 0x20e89300 finish
2014-06-12 14:47:01.669275 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.669280 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.669651 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5417c20 (1 active)
2014-06-12 14:47:01.669664 7f9d7815d700  5 _do_op 0x83ee780 seq 2411 osr(3.76e 0x49b4160)/0x49b4160 start
2014-06-12 14:47:01.669671 7f9d7815d700 10 _do_transaction on 0x519d680
2014-06-12 14:47:01.669681 7f9d7815d700 15 _omap_setkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.670063 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.670093 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.670189 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.670204 7f9d7815d700 15 _omap_rmkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.670209 7f9d7815d700 10 _omap_rmkeys meta/1fe0bd7b/pglog_3.76e/0//-1 r = 0
2014-06-12 14:47:01.670216 7f9d7815d700 15 _omap_setkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.670229 7f9d7815d700 15 _write 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 0~81920
2014-06-12 14:47:01.670390 7f9d7815d700 10 lookup_strip_header done  cid 3.76e_head oid 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.670402 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:47:01.670700 7f9d7815d700 10 _generic_write 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 0~81920 = 0
2014-06-12 14:47:01.670716 7f9d7815d700 15 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.670731 7f9d7815d700 10 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 = 0
2014-06-12 14:47:01.670737 7f9d7815d700 15 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.670744 7f9d7815d700 10 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 = 0
2014-06-12 14:47:01.670881 7f9d7815d700 20 set_header setting 16689 cid 3.76e_head oid 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 parent seq 0
2014-06-12 14:47:01.670908 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.670921 7f9d7815d700 20 set_header setting 2136 cid meta oid 1fe0bd7b/pglog_3.76e/0//-1 parent seq 0
2014-06-12 14:47:01.670931 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:47:01.671401 7f9d7815d700 10 _do_op 0x83ee780 seq 2411 r = 0, finisher 0x8257da0 0x31ddcec0
2014-06-12 14:47:01.671413 7f9d7815d700 10 _finish_op 0x83ee780 seq 2411 osr(3.76e 0x49b4160)/0x49b4160
2014-06-12 14:47:01.671421 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5417c20 (0 active)
2014-06-12 14:47:01.671424 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.671418 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] op_commit: 236381
2014-06-12 14:47:01.671452 7f9d7715b700  5 -- op tracker -- , seq: 748261, time: 2014-06-12 14:47:01.671442, event: op_commit, op: osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.672120 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] op_applied: 236381
2014-06-12 14:47:01.672175 7f9d7795c700  5 -- op tracker -- , seq: 748261, time: 2014-06-12 14:47:01.672162, event: op_applied, op: osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.672187 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] op_applied on primary on version 173'2645
2014-06-12 14:47:01.688337 7f9d43bdc700  5 -- op tracker -- , seq: 748262, time: 2014-06-12 14:47:01.687008, event: header_read, op: osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.688355 7f9d43bdc700  5 -- op tracker -- , seq: 748262, time: 2014-06-12 14:47:01.687012, event: throttled, op: osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.688364 7f9d43bdc700  5 -- op tracker -- , seq: 748262, time: 2014-06-12 14:47:01.688274, event: all_read, op: osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.688373 7f9d43bdc700  5 -- op tracker -- , seq: 748262, time: 0.000000, event: dispatched, op: osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.688384 7f9d43bdc700 20 osd.2 173 should_share_map client.4310 172.16.96.220:0/1244085 173
2014-06-12 14:47:01.688416 7f9d43bdc700 15 osd.2 173 enqueue_op 0x20354b00 prio 63 cost 516096 latency 0.001407 osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.688454 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.688464 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20354b00 prio 63 cost 516096 latency 0.001456 osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]
2014-06-12 14:47:01.688485 7f9d6abe6700  5 -- op tracker -- , seq: 748262, time: 2014-06-12 14:47:01.688481, event: reached_pg, op: osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.688492 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:01.688502 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] handle_message: 0x20354b00
2014-06-12 14:47:01.688509 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_op osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:01.688526 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] get_object_context: found obc in cache: 0x37e4b080
2014-06-12 14:47:01.688534 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] get_object_context: 0x37e4b080 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 rwstate(write n=1 w=0) oi: 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3(173'2645 client.4310.0:142725 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2645) ssc: 0x679a700 snapset: 0=[]:[]+head
2014-06-12 14:47:01.688545 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] find_object_context 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 @head oi=912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3(173'2645 client.4310.0:142725 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2645)
2014-06-12 14:47:01.688562 7f9d6abe6700  5 -- op tracker -- , seq: 748262, time: 2014-06-12 14:47:01.688559, event: started, op: osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.688567 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] execute_ctx 0x25b1e000
2014-06-12 14:47:01.688578 7f9d6abe6700  5 -- op tracker -- , seq: 748262, time: 2014-06-12 14:47:01.688575, event: started, op: osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.688582 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_op 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] ov 173'2645 av 173'2646 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:01.688590 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_osd_op 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096]
2014-06-12 14:47:01.688597 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:01.688606 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_osd_op  write 81920~516096
2014-06-12 14:47:01.688616 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_osd_op_effects on session 0xa703a00
2014-06-12 14:47:01.688623 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] make_writeable 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 snapset=0x679a738  snapc=0=[]
2014-06-12 14:47:01.688630 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] make_writeable 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:01.688635 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] finish_ctx 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 0x25b1e000 op modify  
2014-06-12 14:47:01.688643 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]  set mtime to 2014-06-12 08:46:04.066263
2014-06-12 14:47:01.688657 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]  final snapset 0=[]:[]+head in 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.688674 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]  zeroing write result code 0
2014-06-12 14:47:01.688681 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] new_repop rep_tid 236382 on osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.688690 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] issue_repop rep_tid 236382 o 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.688707 7f9d6abe6700  5 -- op tracker -- , seq: 748262, time: 2014-06-12 14:47:01.688704, event: waiting for subops from 23, op: osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.688724 7f9d6abe6700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:01.688741 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2645 (0'0,173'2645] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] append_log log((0'0,173'2645], crt=0'0) [173'2646 (173'2645) modify   912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 by client.4310.0:142726 2014-06-12 08:46:04.066263]
2014-06-12 14:47:01.688763 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] add_log_entry 173'2646 (173'2645) modify   912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 by client.4310.0:142726 2014-06-12 08:46:04.066263
2014-06-12 14:47:01.688785 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] append_log  adding 1 keys
2014-06-12 14:47:01.688817 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2646, trimmed: 
2014-06-12 14:47:01.688839 7f9d6abe6700  5 queue_transactions existing osr(3.76e 0x49b4160)/0x49b4160
2014-06-12 14:47:01.688845 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.688846 7f9d6abe6700  5 queue_op 0x230fce0 seq 2412 osr(3.76e 0x49b4160) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:01.688853 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] eval_repop repgather(0x5353a40 173'2646 rep_tid=236382 committed?=0 applied?=0 lock=0 op=osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.688869 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20354b00 finish
2014-06-12 14:47:01.688872 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.688875 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.688882 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5417c20 (1 active)
2014-06-12 14:47:01.688886 7f9d82171700  5 _do_op 0x230fce0 seq 2412 osr(3.76e 0x49b4160)/0x49b4160 start
2014-06-12 14:47:01.688891 7f9d82171700 10 _do_transaction on 0x2f89fc80
2014-06-12 14:47:01.688896 7f9d82171700 15 _omap_setkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.689053 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.689079 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.689148 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.689246 7f9d82171700 15 _omap_rmkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.689254 7f9d82171700 10 _omap_rmkeys meta/1fe0bd7b/pglog_3.76e/0//-1 r = 0
2014-06-12 14:47:01.689258 7f9d82171700 15 _omap_setkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.689273 7f9d82171700 15 _write 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 81920~516096
2014-06-12 14:47:01.689375 7f9d82171700 10 lookup_strip_header done  cid 3.76e_head oid 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.689516 7f9d82171700 10 file_to_extents done 
2014-06-12 14:47:01.691609 7f9d82171700 10 _generic_write 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 81920~516096 = 0
2014-06-12 14:47:01.691681 7f9d82171700 15 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.691699 7f9d82171700 10 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 = 0
2014-06-12 14:47:01.691708 7f9d82171700 15 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.691724 7f9d82171700 10 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 = 0
2014-06-12 14:47:01.692018 7f9d82171700 20 set_header setting 16689 cid 3.76e_head oid 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 parent seq 0
2014-06-12 14:47:01.692119 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.692137 7f9d82171700 20 set_header setting 2136 cid meta oid 1fe0bd7b/pglog_3.76e/0//-1 parent seq 0
2014-06-12 14:47:01.692147 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:47:01.693967 7f9d43bdc700  5 -- op tracker -- , seq: 748263, time: 2014-06-12 14:47:01.690830, event: header_read, op: osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.693981 7f9d43bdc700  5 -- op tracker -- , seq: 748263, time: 2014-06-12 14:47:01.690834, event: throttled, op: osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.693990 7f9d43bdc700  5 -- op tracker -- , seq: 748263, time: 2014-06-12 14:47:01.693917, event: all_read, op: osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.694000 7f9d43bdc700  5 -- op tracker -- , seq: 748263, time: 0.000000, event: dispatched, op: osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.694009 7f9d43bdc700 20 osd.2 173 should_share_map client.4310 172.16.96.220:0/1244085 173
2014-06-12 14:47:01.694040 7f9d43bdc700 15 osd.2 173 enqueue_op 0x20355f00 prio 63 cost 1548288 latency 0.003209 osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.694079 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.694089 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x20355f00 prio 63 cost 1548288 latency 0.003258 osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]
2014-06-12 14:47:01.694116 7f9d6a3e5700  5 -- op tracker -- , seq: 748263, time: 2014-06-12 14:47:01.694110, event: reached_pg, op: osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.694125 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:01.694141 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] handle_message: 0x20355f00
2014-06-12 14:47:01.694153 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_op osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:01.694181 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] get_object_context: found obc in cache: 0x37e4b080
2014-06-12 14:47:01.694192 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] get_object_context: 0x37e4b080 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 rwstate(write n=2 w=0) oi: 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3(173'2646 client.4310.0:142726 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2646) ssc: 0x679a700 snapset: 0=[]:[]+head
2014-06-12 14:47:01.694210 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] find_object_context 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 @head oi=912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3(173'2646 client.4310.0:142726 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2646)
2014-06-12 14:47:01.694240 7f9d6a3e5700  5 -- op tracker -- , seq: 748263, time: 2014-06-12 14:47:01.694234, event: started, op: osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.694247 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] execute_ctx 0x5e4c400
2014-06-12 14:47:01.694267 7f9d6a3e5700  5 -- op tracker -- , seq: 748263, time: 2014-06-12 14:47:01.694261, event: started, op: osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.694275 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_op 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] ov 173'2646 av 173'2647 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:01.694293 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_osd_op 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288]
2014-06-12 14:47:01.694308 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:01.694323 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_osd_op  write 598016~1548288
2014-06-12 14:47:01.694340 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] do_osd_op_effects on session 0xa703a00
2014-06-12 14:47:01.694353 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] make_writeable 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 snapset=0x679a738  snapc=0=[]
2014-06-12 14:47:01.694368 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] make_writeable 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:01.694381 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] finish_ctx 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 0x5e4c400 op modify  
2014-06-12 14:47:01.694398 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]  set mtime to 2014-06-12 08:46:04.068248
2014-06-12 14:47:01.694422 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]  final snapset 0=[]:[]+head in 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.694442 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]  zeroing write result code 0
2014-06-12 14:47:01.694454 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] new_repop rep_tid 236383 on osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.694473 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] issue_repop rep_tid 236383 o 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.694500 7f9d6a3e5700  5 -- op tracker -- , seq: 748263, time: 2014-06-12 14:47:01.694496, event: waiting for subops from 23, op: osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.694523 7f9d6a3e5700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:01.694549 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2646 (0'0,173'2646] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] append_log log((0'0,173'2646], crt=0'0) [173'2647 (173'2646) modify   912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 by client.4310.0:142727 2014-06-12 08:46:04.068248]
2014-06-12 14:47:01.694569 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] add_log_entry 173'2647 (173'2646) modify   912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 by client.4310.0:142727 2014-06-12 08:46:04.068248
2014-06-12 14:47:01.694595 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] append_log  adding 1 keys
2014-06-12 14:47:01.694602 7f9d82171700 10 _do_op 0x230fce0 seq 2412 r = 0, finisher 0x8257ce0 0x38e0df80
2014-06-12 14:47:01.694625 7f9d82171700 10 _finish_op 0x230fce0 seq 2412 osr(3.76e 0x49b4160)/0x49b4160
2014-06-12 14:47:01.694629 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2647, trimmed: 
2014-06-12 14:47:01.694642 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5417c20 (0 active)
2014-06-12 14:47:01.694651 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.694652 7f9d6a3e5700  5 queue_transactions existing osr(3.76e 0x49b4160)/0x49b4160
2014-06-12 14:47:01.694656 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.694658 7f9d6a3e5700  5 queue_op 0x289aae0 seq 2413 osr(3.76e 0x49b4160) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:47:01.694668 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5417c20 (1 active)
2014-06-12 14:47:01.694675 7f9d7e96a700  5 _do_op 0x289aae0 seq 2413 osr(3.76e 0x49b4160)/0x49b4160 start
2014-06-12 14:47:01.694679 7f9d7e96a700 10 _do_transaction on 0x6e53080
2014-06-12 14:47:01.694665 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] eval_repop repgather(0x6e52540 173'2647 rep_tid=236383 committed?=0 applied?=0 lock=0 op=osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.694686 7f9d7e96a700 15 _omap_setkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.694686 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x20355f00 finish
2014-06-12 14:47:01.694694 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.694697 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.694829 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.694855 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.694918 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.695004 7f9d7e96a700 15 _omap_rmkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.695013 7f9d7e96a700 10 _omap_rmkeys meta/1fe0bd7b/pglog_3.76e/0//-1 r = 0
2014-06-12 14:47:01.695021 7f9d7e96a700 15 _omap_setkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.695040 7f9d7e96a700 15 _write 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 598016~1548288
2014-06-12 14:47:01.695192 7f9d7e96a700 10 lookup_strip_header done  cid 3.76e_head oid 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.695434 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:47:01.695858 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] op_commit: 236382
2014-06-12 14:47:01.695891 7f9d7715b700  5 -- op tracker -- , seq: 748262, time: 2014-06-12 14:47:01.695882, event: op_commit, op: osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.695915 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] op_applied: 236382
2014-06-12 14:47:01.695936 7f9d7795c700  5 -- op tracker -- , seq: 748262, time: 2014-06-12 14:47:01.695930, event: op_applied, op: osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.695946 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] op_applied on primary on version 173'2646
2014-06-12 14:47:01.699973 7f9d57543700  5 -- op tracker -- , seq: 748264, time: 2014-06-12 14:47:01.699893, event: header_read, op: osd_sub_op_reply(client.4310.0:142725 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.699993 7f9d57543700  5 -- op tracker -- , seq: 748264, time: 2014-06-12 14:47:01.699895, event: throttled, op: osd_sub_op_reply(client.4310.0:142725 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.700001 7f9d57543700  5 -- op tracker -- , seq: 748264, time: 2014-06-12 14:47:01.699913, event: all_read, op: osd_sub_op_reply(client.4310.0:142725 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.700009 7f9d57543700  5 -- op tracker -- , seq: 748264, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142725 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.700015 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142725 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:01.700023 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:01.700032 7f9d57543700 15 osd.2 173 enqueue_op 0x31d60500 prio 196 cost 0 latency 0.000138 osd_sub_op_reply(client.4310.0:142725 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:01.700167 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.700175 7f9d66bde700 10 osd.2 173 dequeue_op 0x31d60500 prio 196 cost 0 latency 0.000282 osd_sub_op_reply(client.4310.0:142725 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2 pg pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]
2014-06-12 14:47:01.700195 7f9d66bde700  5 -- op tracker -- , seq: 748264, time: 2014-06-12 14:47:01.700192, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142725 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.700203 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] handle_message: 0x31d60500
2014-06-12 14:47:01.700218 7f9d66bde700  5 -- op tracker -- , seq: 748264, time: 2014-06-12 14:47:01.700215, event: started, op: osd_sub_op_reply(client.4310.0:142725 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.700224 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] sub_op_modify_reply: tid 236381 op  ack_type 4 from 23
2014-06-12 14:47:01.700243 7f9d66bde700  5 -- op tracker -- , seq: 748261, time: 2014-06-12 14:47:01.700236, event: sub_op_commit_rec, op: osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.700250 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] repop_all_applied: repop tid 236381 all applied 
2014-06-12 14:47:01.700261 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] eval_repop repgather(0x519da40 173'2645 rep_tid=236381 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.700276 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean]  sending ack on repgather(0x519da40 173'2645 rep_tid=236381 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) 0x28676000
2014-06-12 14:47:01.700305 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2644 crt=0'0 lcod 173'2643 mlcod 173'2643 active+clean] repop_all_committed: repop tid 236381 all committed 
2014-06-12 14:47:01.700315 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2643 active+clean] eval_repop repgather(0x519da40 173'2645 rep_tid=236381 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.700331 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2643 active+clean] log_op_stats osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 inb 82199 outb 0 rlat 0.032462 lat 0.032488
2014-06-12 14:47:01.700346 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2643 active+clean] publish_stats_to_osd 173:3468
2014-06-12 14:47:01.700359 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2643 active+clean]  sending commit on repgather(0x519da40 173'2645 rep_tid=236381 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) 0x348b7b80
2014-06-12 14:47:01.700386 7f9d66bde700  5 -- op tracker -- , seq: 748261, time: 2014-06-12 14:47:01.700382, event: commit_sent, op: osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.700393 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2643 active+clean]  requeue_ops 
2014-06-12 14:47:01.700403 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2644 active+clean]  removing repgather(0x519da40 173'2645 rep_tid=236381 committed?=1 applied?=1 op=osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.700416 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2644 active+clean]    q front is repgather(0x519da40 173'2645 rep_tid=236381 committed?=1 applied?=1 op=osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.700446 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2644 active+clean] remove_repop repgather(0x519da40 173'2645 rep_tid=236381 committed?=1 applied?=1 op=osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.700459 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2644 active+clean]  requeue_ops 
2014-06-12 14:47:01.700477 7f9d66bde700  5 -- op tracker -- , seq: 748261, time: 2014-06-12 14:47:01.700472, event: done, op: osd_op(client.4310.0:142725 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~81920] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.700493 7f9d66bde700 10 osd.2 173 dequeue_op 0x31d60500 finish
2014-06-12 14:47:01.700500 7f9d66bde700  5 -- op tracker -- , seq: 748264, time: 2014-06-12 14:47:01.700497, event: done, op: osd_sub_op_reply(client.4310.0:142725 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.700510 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.700513 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.703326 7f9d7e96a700 10 _generic_write 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 598016~1548288 = 0
2014-06-12 14:47:01.703599 7f9d7e96a700 15 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.703618 7f9d7e96a700 10 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 = 0
2014-06-12 14:47:01.703628 7f9d7e96a700 15 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.703636 7f9d7e96a700 10 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 = 0
2014-06-12 14:47:01.704246 7f9d7e96a700 20 set_header setting 16689 cid 3.76e_head oid 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 parent seq 0
2014-06-12 14:47:01.704396 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.704414 7f9d7e96a700 20 set_header setting 2136 cid meta oid 1fe0bd7b/pglog_3.76e/0//-1 parent seq 0
2014-06-12 14:47:01.704424 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:47:01.706681 7f9d43bdc700  5 -- op tracker -- , seq: 748265, time: 2014-06-12 14:47:01.704398, event: header_read, op: osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.706695 7f9d43bdc700  5 -- op tracker -- , seq: 748265, time: 2014-06-12 14:47:01.704401, event: throttled, op: osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.706705 7f9d43bdc700  5 -- op tracker -- , seq: 748265, time: 2014-06-12 14:47:01.706611, event: all_read, op: osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.706714 7f9d43bdc700  5 -- op tracker -- , seq: 748265, time: 0.000000, event: dispatched, op: osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.706724 7f9d43bdc700 20 osd.2 173 should_share_map client.4310 172.16.96.220:0/1244085 173
2014-06-12 14:47:01.706753 7f9d43bdc700 15 osd.2 173 enqueue_op 0x20355b00 prio 63 cost 1032192 latency 0.002354 osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.711612 7f9d57543700  5 -- op tracker -- , seq: 748266, time: 2014-06-12 14:47:01.711542, event: header_read, op: osd_sub_op_reply(client.4310.0:142726 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.711634 7f9d57543700  5 -- op tracker -- , seq: 748266, time: 2014-06-12 14:47:01.711545, event: throttled, op: osd_sub_op_reply(client.4310.0:142726 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.711642 7f9d57543700  5 -- op tracker -- , seq: 748266, time: 2014-06-12 14:47:01.711566, event: all_read, op: osd_sub_op_reply(client.4310.0:142726 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.711650 7f9d57543700  5 -- op tracker -- , seq: 748266, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142726 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.711661 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142726 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:01.711667 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:01.711676 7f9d57543700 15 osd.2 173 enqueue_op 0x1702e700 prio 196 cost 0 latency 0.000134 osd_sub_op_reply(client.4310.0:142726 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:01.711714 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.711715 7f9d43bdc700  5 -- op tracker -- , seq: 748267, time: 2014-06-12 14:47:01.707379, event: header_read, op: osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.711727 7f9d43bdc700  5 -- op tracker -- , seq: 748267, time: 2014-06-12 14:47:01.707380, event: throttled, op: osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.711722 7f9d69be4700 10 osd.2 173 dequeue_op 0x1702e700 prio 196 cost 0 latency 0.000179 osd_sub_op_reply(client.4310.0:142726 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2 pg pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2644 active+clean]
2014-06-12 14:47:01.711736 7f9d43bdc700  5 -- op tracker -- , seq: 748267, time: 2014-06-12 14:47:01.711668, event: all_read, op: osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.711743 7f9d69be4700  5 -- op tracker -- , seq: 748266, time: 2014-06-12 14:47:01.711740, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142726 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.711755 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2644 active+clean] handle_message: 0x1702e700
2014-06-12 14:47:01.711771 7f9d69be4700  5 -- op tracker -- , seq: 748266, time: 2014-06-12 14:47:01.711767, event: started, op: osd_sub_op_reply(client.4310.0:142726 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.711776 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2644 active+clean] sub_op_modify_reply: tid 236382 op  ack_type 4 from 23
2014-06-12 14:47:01.711795 7f9d69be4700  5 -- op tracker -- , seq: 748262, time: 2014-06-12 14:47:01.711788, event: sub_op_commit_rec, op: osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.711802 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2644 active+clean] repop_all_applied: repop tid 236382 all applied 
2014-06-12 14:47:01.711812 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2644 active+clean] eval_repop repgather(0x5353a40 173'2646 rep_tid=236382 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.711832 7f9d43bdc700  5 -- op tracker -- , seq: 748267, time: 0.000000, event: dispatched, op: osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.711828 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2644 active+clean]  sending ack on repgather(0x5353a40 173'2646 rep_tid=236382 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) 0x348b6780
2014-06-12 14:47:01.711840 7f9d43bdc700 20 osd.2 173 should_share_map client.4310 172.16.96.220:0/1244085 173
2014-06-12 14:47:01.711845 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:01.711857 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2645 crt=0'0 lcod 173'2644 mlcod 173'2644 active+clean] repop_all_committed: repop tid 236382 all committed 
2014-06-12 14:47:01.711867 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2644 active+clean] eval_repop repgather(0x5353a40 173'2646 rep_tid=236382 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.711883 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2644 active+clean] log_op_stats osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.024847 lat 0.024874
2014-06-12 14:47:01.711892 7f9d43bdc700 15 osd.2 173 enqueue_op 0x20355300 prio 63 cost 1015808 latency 0.004513 osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.711899 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2644 active+clean] publish_stats_to_osd 173:3469
2014-06-12 14:47:01.711918 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active)
2014-06-12 14:47:01.711912 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2644 active+clean]  sending commit on repgather(0x5353a40 173'2646 rep_tid=236382 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) 0x355c0f00
2014-06-12 14:47:01.711945 7f9d69be4700  5 -- op tracker -- , seq: 748262, time: 2014-06-12 14:47:01.711940, event: commit_sent, op: osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.711963 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2644 active+clean]  requeue_ops 
2014-06-12 14:47:01.711971 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean]  removing repgather(0x5353a40 173'2646 rep_tid=236382 committed?=1 applied?=1 op=osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.711983 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean]    q front is repgather(0x5353a40 173'2646 rep_tid=236382 committed?=1 applied?=1 op=osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.711992 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] remove_repop repgather(0x5353a40 173'2646 rep_tid=236382 committed?=1 applied?=1 op=osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.712000 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean]  requeue_ops 
2014-06-12 14:47:01.712013 7f9d69be4700  5 -- op tracker -- , seq: 748262, time: 2014-06-12 14:47:01.712010, event: done, op: osd_op(client.4310.0:142726 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 81920~516096] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.712026 7f9d69be4700 10 osd.2 173 dequeue_op 0x1702e700 finish
2014-06-12 14:47:01.712033 7f9d69be4700  5 -- op tracker -- , seq: 748266, time: 2014-06-12 14:47:01.712031, event: done, op: osd_sub_op_reply(client.4310.0:142726 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.712040 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active)
2014-06-12 14:47:01.712046 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.712034 7f9d64bda700 10 osd.2 173 dequeue_op 0x20355b00 prio 63 cost 1032192 latency 0.007636 osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean]
2014-06-12 14:47:01.712059 7f9d64bda700  5 -- op tracker -- , seq: 748265, time: 2014-06-12 14:47:01.712055, event: reached_pg, op: osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.712067 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:01.712079 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] handle_message: 0x20355b00
2014-06-12 14:47:01.712088 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] do_op osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:01.712111 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] get_object_context: found obc in cache: 0x37e4b080
2014-06-12 14:47:01.712120 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] get_object_context: 0x37e4b080 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 rwstate(write n=1 w=0) oi: 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3(173'2647 client.4310.0:142727 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2647) ssc: 0x679a700 snapset: 0=[]:[]+head
2014-06-12 14:47:01.712134 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] find_object_context 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 @head oi=912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3(173'2647 client.4310.0:142727 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2647)
2014-06-12 14:47:01.712157 7f9d64bda700  5 -- op tracker -- , seq: 748265, time: 2014-06-12 14:47:01.712152, event: started, op: osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.712163 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] execute_ctx 0x60aa600
2014-06-12 14:47:01.712179 7f9d64bda700  5 -- op tracker -- , seq: 748265, time: 2014-06-12 14:47:01.712174, event: started, op: osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.712185 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] do_op 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] ov 173'2647 av 173'2648 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:01.712198 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] do_osd_op 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192]
2014-06-12 14:47:01.712208 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:01.712220 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] do_osd_op  write 2146304~1032192
2014-06-12 14:47:01.712234 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] do_osd_op_effects on session 0xa703a00
2014-06-12 14:47:01.712244 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] make_writeable 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 snapset=0x679a738  snapc=0=[]
2014-06-12 14:47:01.712256 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] make_writeable 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:01.712265 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] finish_ctx 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 0x60aa600 op modify  
2014-06-12 14:47:01.712284 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean]  set mtime to 2014-06-12 08:46:04.082708
2014-06-12 14:47:01.712304 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean]  final snapset 0=[]:[]+head in 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.712320 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean]  zeroing write result code 0
2014-06-12 14:47:01.712330 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] new_repop rep_tid 236384 on osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.712344 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] issue_repop rep_tid 236384 o 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.712366 7f9d64bda700  5 -- op tracker -- , seq: 748265, time: 2014-06-12 14:47:01.712361, event: waiting for subops from 23, op: osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.712384 7f9d64bda700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:01.712402 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2647 (0'0,173'2647] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] append_log log((0'0,173'2647], crt=0'0) [173'2648 (173'2647) modify   912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 by client.4310.0:142728 2014-06-12 08:46:04.082708]
2014-06-12 14:47:01.712429 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] add_log_entry 173'2648 (173'2647) modify   912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 by client.4310.0:142728 2014-06-12 08:46:04.082708
2014-06-12 14:47:01.712449 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] append_log  adding 1 keys
2014-06-12 14:47:01.712478 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2648, trimmed: 
2014-06-12 14:47:01.712497 7f9d64bda700  5 queue_transactions existing osr(3.76e 0x49b4160)/0x49b4160
2014-06-12 14:47:01.712501 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.712503 7f9d64bda700  5 queue_op 0x213fa8a0 seq 2414 osr(3.76e 0x49b4160) 1034485 bytes   (queue has 2 ops and 2585066 bytes)
2014-06-12 14:47:01.712513 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5417c20 (2 active)
2014-06-12 14:47:01.712510 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] eval_repop repgather(0x1a444780 173'2648 rep_tid=236384 committed?=0 applied?=0 lock=0 op=osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.712528 7f9d64bda700 10 osd.2 173 dequeue_op 0x20355b00 finish
2014-06-12 14:47:01.712534 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:01.712536 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.712535 7f9d653db700 10 osd.2 173 dequeue_op 0x20355300 prio 63 cost 1015808 latency 0.005156 osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean]
2014-06-12 14:47:01.712558 7f9d653db700  5 -- op tracker -- , seq: 748267, time: 2014-06-12 14:47:01.712552, event: reached_pg, op: osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.712566 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:01.712579 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] handle_message: 0x20355300
2014-06-12 14:47:01.712589 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] do_op osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:01.712610 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] get_object_context: found obc in cache: 0x37e4b080
2014-06-12 14:47:01.712621 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] get_object_context: 0x37e4b080 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 rwstate(write n=2 w=0) oi: 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3(173'2648 client.4310.0:142728 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2648) ssc: 0x679a700 snapset: 0=[]:[]+head
2014-06-12 14:47:01.712636 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] find_object_context 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 @head oi=912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3(173'2648 client.4310.0:142728 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2648)
2014-06-12 14:47:01.712658 7f9d653db700  5 -- op tracker -- , seq: 748267, time: 2014-06-12 14:47:01.712652, event: started, op: osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.712664 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] execute_ctx 0x1a916000
2014-06-12 14:47:01.712681 7f9d653db700  5 -- op tracker -- , seq: 748267, time: 2014-06-12 14:47:01.712676, event: started, op: osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.712688 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] do_op 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] ov 173'2648 av 173'2649 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:01.712702 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] do_osd_op 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808]
2014-06-12 14:47:01.712713 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:01.712729 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] do_osd_op  write 3178496~1015808
2014-06-12 14:47:01.712743 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] do_osd_op_effects on session 0xa703a00
2014-06-12 14:47:01.712754 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] make_writeable 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 snapset=0x679a738  snapc=0=[]
2014-06-12 14:47:01.712771 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] make_writeable 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:01.712782 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] finish_ctx 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 0x1a916000 op modify  
2014-06-12 14:47:01.712794 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean]  set mtime to 2014-06-12 08:46:04.084518
2014-06-12 14:47:01.712814 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean]  final snapset 0=[]:[]+head in 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.712830 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean]  zeroing write result code 0
2014-06-12 14:47:01.712841 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] new_repop rep_tid 236385 on osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.712862 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] issue_repop rep_tid 236385 o 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.712885 7f9d653db700  5 -- op tracker -- , seq: 748267, time: 2014-06-12 14:47:01.712880, event: waiting for subops from 23, op: osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.713298 7f9d653db700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:01.713322 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2648 (0'0,173'2648] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] append_log log((0'0,173'2648], crt=0'0) [173'2649 (173'2648) modify   912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 by client.4310.0:142729 2014-06-12 08:46:04.084518]
2014-06-12 14:47:01.713359 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] add_log_entry 173'2649 (173'2648) modify   912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 by client.4310.0:142729 2014-06-12 08:46:04.084518
2014-06-12 14:47:01.713386 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] append_log  adding 1 keys
2014-06-12 14:47:01.713415 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2649, trimmed: 
2014-06-12 14:47:01.713434 7f9d653db700  5 queue_transactions existing osr(3.76e 0x49b4160)/0x49b4160
2014-06-12 14:47:01.713438 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.713439 7f9d653db700  5 queue_op 0x6872120 seq 2415 osr(3.76e 0x49b4160) 1018101 bytes   (queue has 3 ops and 3603167 bytes)
2014-06-12 14:47:01.713454 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] eval_repop repgather(0x10451740 173'2649 rep_tid=236385 committed?=0 applied?=0 lock=0 op=osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.713473 7f9d653db700 10 osd.2 173 dequeue_op 0x20355300 finish
2014-06-12 14:47:01.713478 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.713481 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.713488 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5417c20 (3 active)
2014-06-12 14:47:01.719972 7f9d7e96a700 10 _do_op 0x289aae0 seq 2413 r = 0, finisher 0x692f8a0 0x2fef1c80
2014-06-12 14:47:01.719989 7f9d7e96a700 10 _finish_op 0x289aae0 seq 2413 osr(3.76e 0x49b4160)/0x49b4160
2014-06-12 14:47:01.719998 7f9d8116f700  5 _do_op 0x213fa8a0 seq 2414 osr(3.76e 0x49b4160)/0x49b4160 start
2014-06-12 14:47:01.719999 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5417c20 (2 active)
2014-06-12 14:47:01.720004 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.720005 7f9d8116f700 10 _do_transaction on 0x28df2840
2014-06-12 14:47:01.719993 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] op_commit: 236383
2014-06-12 14:47:01.720013 7f9d8116f700 15 _omap_setkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.720016 7f9d7715b700  5 -- op tracker -- , seq: 748263, time: 2014-06-12 14:47:01.720009, event: op_commit, op: osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.720034 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] op_applied: 236383
2014-06-12 14:47:01.720051 7f9d7795c700  5 -- op tracker -- , seq: 748263, time: 2014-06-12 14:47:01.720046, event: op_applied, op: osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.720063 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] op_applied on primary on version 173'2647
2014-06-12 14:47:01.720193 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.720217 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.720265 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.720287 7f9d8116f700 15 _omap_rmkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.720295 7f9d8116f700 10 _omap_rmkeys meta/1fe0bd7b/pglog_3.76e/0//-1 r = 0
2014-06-12 14:47:01.720302 7f9d8116f700 15 _omap_setkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.720320 7f9d8116f700 15 _write 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 2146304~1032192
2014-06-12 14:47:01.720428 7f9d8116f700 10 lookup_strip_header done  cid 3.76e_head oid 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.720547 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:47:01.722952 7f9d8116f700 10 _generic_write 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 2146304~1032192 = 0
2014-06-12 14:47:01.723035 7f9d8116f700 15 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.723053 7f9d8116f700 10 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 = 0
2014-06-12 14:47:01.723061 7f9d8116f700 15 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.723070 7f9d8116f700 10 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 = 0
2014-06-12 14:47:01.723358 7f9d8116f700 20 set_header setting 16689 cid 3.76e_head oid 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 parent seq 0
2014-06-12 14:47:01.723470 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.723487 7f9d8116f700 20 set_header setting 2136 cid meta oid 1fe0bd7b/pglog_3.76e/0//-1 parent seq 0
2014-06-12 14:47:01.723508 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:47:01.724771 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:01.724914 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:01.729755 7f9d8116f700 10 _do_op 0x213fa8a0 seq 2414 r = 0, finisher 0x76881a0 0x12aed000
2014-06-12 14:47:01.729774 7f9d8116f700 10 _finish_op 0x213fa8a0 seq 2414 osr(3.76e 0x49b4160)/0x49b4160
2014-06-12 14:47:01.729784 7f9d7d167700  5 _do_op 0x6872120 seq 2415 osr(3.76e 0x49b4160)/0x49b4160 start
2014-06-12 14:47:01.729790 7f9d7d167700 10 _do_transaction on 0x2010ef00
2014-06-12 14:47:01.729786 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5417c20 (1 active)
2014-06-12 14:47:01.729794 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.729780 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] op_commit: 236384
2014-06-12 14:47:01.729799 7f9d7d167700 15 _omap_setkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.729811 7f9d7715b700  5 -- op tracker -- , seq: 748265, time: 2014-06-12 14:47:01.729802, event: op_commit, op: osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.729831 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] op_applied: 236384
2014-06-12 14:47:01.729852 7f9d7795c700  5 -- op tracker -- , seq: 748265, time: 2014-06-12 14:47:01.729847, event: op_applied, op: osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.729862 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] op_applied on primary on version 173'2648
2014-06-12 14:47:01.729908 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.729934 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.729969 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.730051 7f9d7d167700 15 _omap_rmkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.730057 7f9d7d167700 10 _omap_rmkeys meta/1fe0bd7b/pglog_3.76e/0//-1 r = 0
2014-06-12 14:47:01.730062 7f9d7d167700 15 _omap_setkeys meta/1fe0bd7b/pglog_3.76e/0//-1
2014-06-12 14:47:01.730076 7f9d7d167700 15 _write 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 3178496~1015808
2014-06-12 14:47:01.730166 7f9d7d167700 10 lookup_strip_header done  cid 3.76e_head oid 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.730291 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:47:01.732601 7f9d7d167700 10 _generic_write 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 3178496~1015808 = 0
2014-06-12 14:47:01.732678 7f9d7d167700 15 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.732691 7f9d7d167700 10 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 = 0
2014-06-12 14:47:01.732694 7f9d7d167700 15 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3
2014-06-12 14:47:01.732707 7f9d7d167700 10 _setattrs 3.76e_head/912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 = 0
2014-06-12 14:47:01.732910 7f9d7d167700 20 set_header setting 16689 cid 3.76e_head oid 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 parent seq 0
2014-06-12 14:47:01.732995 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.733006 7f9d7d167700 20 set_header setting 2136 cid meta oid 1fe0bd7b/pglog_3.76e/0//-1 parent seq 0
2014-06-12 14:47:01.733020 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:47:01.733888 7f9d57543700  5 -- op tracker -- , seq: 748268, time: 2014-06-12 14:47:01.733841, event: header_read, op: osd_sub_op_reply(client.4310.0:142727 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.733896 7f9d57543700  5 -- op tracker -- , seq: 748268, time: 2014-06-12 14:47:01.733843, event: throttled, op: osd_sub_op_reply(client.4310.0:142727 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.733901 7f9d57543700  5 -- op tracker -- , seq: 748268, time: 2014-06-12 14:47:01.733854, event: all_read, op: osd_sub_op_reply(client.4310.0:142727 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.733905 7f9d57543700  5 -- op tracker -- , seq: 748268, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142727 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.733910 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142727 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:01.733914 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:01.733921 7f9d57543700 15 osd.2 173 enqueue_op 0x1702eb00 prio 196 cost 0 latency 0.000079 osd_sub_op_reply(client.4310.0:142727 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:01.733947 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.733956 7f9d623d5700 10 osd.2 173 dequeue_op 0x1702eb00 prio 196 cost 0 latency 0.000115 osd_sub_op_reply(client.4310.0:142727 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2 pg pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean]
2014-06-12 14:47:01.733979 7f9d623d5700  5 -- op tracker -- , seq: 748268, time: 2014-06-12 14:47:01.733975, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142727 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.733988 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] handle_message: 0x1702eb00
2014-06-12 14:47:01.734004 7f9d623d5700  5 -- op tracker -- , seq: 748268, time: 2014-06-12 14:47:01.734001, event: started, op: osd_sub_op_reply(client.4310.0:142727 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.734017 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] sub_op_modify_reply: tid 236383 op  ack_type 4 from 23
2014-06-12 14:47:01.734039 7f9d623d5700  5 -- op tracker -- , seq: 748263, time: 2014-06-12 14:47:01.734031, event: sub_op_commit_rec, op: osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.734049 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] repop_all_applied: repop tid 236383 all applied 
2014-06-12 14:47:01.734062 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] eval_repop repgather(0x6e52540 173'2647 rep_tid=236383 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.734081 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean]  sending ack on repgather(0x6e52540 173'2647 rep_tid=236383 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) 0x348b6f00
2014-06-12 14:47:01.734116 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2646 crt=0'0 lcod 173'2645 mlcod 173'2645 active+clean] repop_all_committed: repop tid 236383 all committed 
2014-06-12 14:47:01.734129 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2645 active+clean] eval_repop repgather(0x6e52540 173'2647 rep_tid=236383 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.734152 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2645 active+clean] log_op_stats osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 0.043284 lat 0.043321
2014-06-12 14:47:01.734172 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2645 active+clean] publish_stats_to_osd 173:3470
2014-06-12 14:47:01.734187 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2645 active+clean]  sending commit on repgather(0x6e52540 173'2647 rep_tid=236383 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) 0x16970c80
2014-06-12 14:47:01.734221 7f9d623d5700  5 -- op tracker -- , seq: 748263, time: 2014-06-12 14:47:01.734216, event: commit_sent, op: osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.734231 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2645 active+clean]  requeue_ops 
2014-06-12 14:47:01.734241 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean]  removing repgather(0x6e52540 173'2647 rep_tid=236383 committed?=1 applied?=1 op=osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.734259 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean]    q front is repgather(0x6e52540 173'2647 rep_tid=236383 committed?=1 applied?=1 op=osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.734275 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean] remove_repop repgather(0x6e52540 173'2647 rep_tid=236383 committed?=1 applied?=1 op=osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.734291 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean]  requeue_ops 
2014-06-12 14:47:01.734309 7f9d623d5700  5 -- op tracker -- , seq: 748263, time: 2014-06-12 14:47:01.734305, event: done, op: osd_op(client.4310.0:142727 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 598016~1548288] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.734329 7f9d623d5700 10 osd.2 173 dequeue_op 0x1702eb00 finish
2014-06-12 14:47:01.734338 7f9d623d5700  5 -- op tracker -- , seq: 748268, time: 2014-06-12 14:47:01.734334, event: done, op: osd_sub_op_reply(client.4310.0:142727 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.734349 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.734351 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.737894 7f9d7d167700 10 _do_op 0x6872120 seq 2415 r = 0, finisher 0x895de40 0x4e2a580
2014-06-12 14:47:01.737917 7f9d7d167700 10 _finish_op 0x6872120 seq 2415 osr(3.76e 0x49b4160)/0x49b4160
2014-06-12 14:47:01.737927 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5417c20 (0 active)
2014-06-12 14:47:01.737936 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.737923 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean] op_commit: 236385
2014-06-12 14:47:01.737955 7f9d7715b700  5 -- op tracker -- , seq: 748267, time: 2014-06-12 14:47:01.737945, event: op_commit, op: osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.737977 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean] op_applied: 236385
2014-06-12 14:47:01.737992 7f9d7795c700  5 -- op tracker -- , seq: 748267, time: 2014-06-12 14:47:01.737988, event: op_applied, op: osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.737997 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean] op_applied on primary on version 173'2649
2014-06-12 14:47:01.744651 7f9d57543700  5 -- op tracker -- , seq: 748269, time: 2014-06-12 14:47:01.744594, event: header_read, op: osd_sub_op_reply(client.4310.0:142728 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.744667 7f9d57543700  5 -- op tracker -- , seq: 748269, time: 2014-06-12 14:47:01.744597, event: throttled, op: osd_sub_op_reply(client.4310.0:142728 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.744671 7f9d57543700  5 -- op tracker -- , seq: 748269, time: 2014-06-12 14:47:01.744621, event: all_read, op: osd_sub_op_reply(client.4310.0:142728 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.744676 7f9d57543700  5 -- op tracker -- , seq: 748269, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142728 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.744680 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142728 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:01.744684 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:01.744690 7f9d57543700 15 osd.2 173 enqueue_op 0x23e09400 prio 196 cost 0 latency 0.000096 osd_sub_op_reply(client.4310.0:142728 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:01.744714 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.744723 7f9d68be2700 10 osd.2 173 dequeue_op 0x23e09400 prio 196 cost 0 latency 0.000129 osd_sub_op_reply(client.4310.0:142728 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2 pg pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean]
2014-06-12 14:47:01.744747 7f9d68be2700  5 -- op tracker -- , seq: 748269, time: 2014-06-12 14:47:01.744743, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142728 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.744760 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean] handle_message: 0x23e09400
2014-06-12 14:47:01.744779 7f9d68be2700  5 -- op tracker -- , seq: 748269, time: 2014-06-12 14:47:01.744775, event: started, op: osd_sub_op_reply(client.4310.0:142728 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.744788 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean] sub_op_modify_reply: tid 236384 op  ack_type 4 from 23
2014-06-12 14:47:01.744813 7f9d68be2700  5 -- op tracker -- , seq: 748265, time: 2014-06-12 14:47:01.744805, event: sub_op_commit_rec, op: osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.744820 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean] repop_all_applied: repop tid 236384 all applied 
2014-06-12 14:47:01.744834 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean] eval_repop repgather(0x1a444780 173'2648 rep_tid=236384 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.744852 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean]  sending ack on repgather(0x1a444780 173'2648 rep_tid=236384 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) 0x564ca00
2014-06-12 14:47:01.744906 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2647 crt=0'0 lcod 173'2646 mlcod 173'2646 active+clean] repop_all_committed: repop tid 236384 all committed 
2014-06-12 14:47:01.744917 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2646 active+clean] eval_repop repgather(0x1a444780 173'2648 rep_tid=236384 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.744935 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2646 active+clean] log_op_stats osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.040506 lat 0.040536
2014-06-12 14:47:01.744961 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2646 active+clean] publish_stats_to_osd 173:3471
2014-06-12 14:47:01.744982 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2646 active+clean]  sending commit on repgather(0x1a444780 173'2648 rep_tid=236384 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) 0x30321400
2014-06-12 14:47:01.745015 7f9d68be2700  5 -- op tracker -- , seq: 748265, time: 2014-06-12 14:47:01.745009, event: commit_sent, op: osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.745024 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2646 active+clean]  requeue_ops 
2014-06-12 14:47:01.745050 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean]  removing repgather(0x1a444780 173'2648 rep_tid=236384 committed?=1 applied?=1 op=osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.745065 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean]    q front is repgather(0x1a444780 173'2648 rep_tid=236384 committed?=1 applied?=1 op=osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.745089 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean] remove_repop repgather(0x1a444780 173'2648 rep_tid=236384 committed?=1 applied?=1 op=osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.745106 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean]  requeue_ops 
2014-06-12 14:47:01.745126 7f9d68be2700  5 -- op tracker -- , seq: 748265, time: 2014-06-12 14:47:01.745121, event: done, op: osd_op(client.4310.0:142728 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 2146304~1032192] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.745163 7f9d68be2700 10 osd.2 173 dequeue_op 0x23e09400 finish
2014-06-12 14:47:01.745171 7f9d68be2700  5 -- op tracker -- , seq: 748269, time: 2014-06-12 14:47:01.745167, event: done, op: osd_sub_op_reply(client.4310.0:142728 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.745182 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.745185 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.772241 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:01.772271 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:01.854878 7f9d57543700  5 -- op tracker -- , seq: 748270, time: 2014-06-12 14:47:01.854803, event: header_read, op: osd_sub_op_reply(client.4310.0:142729 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.854894 7f9d57543700  5 -- op tracker -- , seq: 748270, time: 2014-06-12 14:47:01.854806, event: throttled, op: osd_sub_op_reply(client.4310.0:142729 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.854903 7f9d57543700  5 -- op tracker -- , seq: 748270, time: 2014-06-12 14:47:01.854823, event: all_read, op: osd_sub_op_reply(client.4310.0:142729 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.854911 7f9d57543700  5 -- op tracker -- , seq: 748270, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4310.0:142729 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.854918 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4310.0:142729 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:01.854925 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:01.854934 7f9d57543700 15 osd.2 173 enqueue_op 0x23e08d00 prio 196 cost 0 latency 0.000131 osd_sub_op_reply(client.4310.0:142729 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:01.854969 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.854981 7f9d643d9700 10 osd.2 173 dequeue_op 0x23e08d00 prio 196 cost 0 latency 0.000178 osd_sub_op_reply(client.4310.0:142729 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0) v2 pg pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean]
2014-06-12 14:47:01.855009 7f9d643d9700  5 -- op tracker -- , seq: 748270, time: 2014-06-12 14:47:01.855005, event: reached_pg, op: osd_sub_op_reply(client.4310.0:142729 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.855018 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean] handle_message: 0x23e08d00
2014-06-12 14:47:01.855038 7f9d643d9700  5 -- op tracker -- , seq: 748270, time: 2014-06-12 14:47:01.855034, event: started, op: osd_sub_op_reply(client.4310.0:142729 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.855044 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean] sub_op_modify_reply: tid 236385 op  ack_type 4 from 23
2014-06-12 14:47:01.855071 7f9d643d9700  5 -- op tracker -- , seq: 748267, time: 2014-06-12 14:47:01.855061, event: sub_op_commit_rec, op: osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.855082 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean] repop_all_applied: repop tid 236385 all applied 
2014-06-12 14:47:01.855095 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean] eval_repop repgather(0x10451740 173'2649 rep_tid=236385 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.855116 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean]  sending ack on repgather(0x10451740 173'2649 rep_tid=236385 committed?=0 applied?=1 lock=0 op=osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) 0x1d54ec80
2014-06-12 14:47:01.855152 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2648 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean] repop_all_committed: repop tid 236385 all committed 
2014-06-12 14:47:01.855165 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2647 active+clean] eval_repop repgather(0x10451740 173'2649 rep_tid=236385 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:01.855185 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2647 active+clean] log_op_stats osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4 inb 1016087 outb 0 rlat 0.147771 lat 0.147805
2014-06-12 14:47:01.855205 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2647 active+clean] publish_stats_to_osd 173:3472
2014-06-12 14:47:01.855223 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2647 active+clean]  sending commit on repgather(0x10451740 173'2649 rep_tid=236385 committed?=1 applied?=1 lock=0 op=osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4) 0x912af00
2014-06-12 14:47:01.855260 7f9d643d9700  5 -- op tracker -- , seq: 748267, time: 2014-06-12 14:47:01.855254, event: commit_sent, op: osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.855271 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2647 active+clean]  requeue_ops 
2014-06-12 14:47:01.855284 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2648 active+clean]  removing repgather(0x10451740 173'2649 rep_tid=236385 committed?=1 applied?=1 op=osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.855302 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2648 active+clean]    q front is repgather(0x10451740 173'2649 rep_tid=236385 committed?=1 applied?=1 op=osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.855325 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2648 active+clean] remove_repop repgather(0x10451740 173'2649 rep_tid=236385 committed?=1 applied?=1 op=osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:01.855343 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2648 active+clean]  requeue_ops 
2014-06-12 14:47:01.855374 7f9d643d9700  5 -- op tracker -- , seq: 748267, time: 2014-06-12 14:47:01.855368, event: done, op: osd_op(client.4310.0:142729 rbd_data.10ad6b8b4567.0000000000000f27 [set-alloc-hint object_size 4194304 write_size 4194304,write 3178496~1015808] 3.912e676e ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.855396 7f9d643d9700 10 osd.2 173 dequeue_op 0x23e08d00 finish
2014-06-12 14:47:01.855406 7f9d643d9700  5 -- op tracker -- , seq: 748270, time: 2014-06-12 14:47:01.855402, event: done, op: osd_sub_op_reply(client.4310.0:142729 3.76e 912e676e/rbd_data.10ad6b8b4567.0000000000000f27/head//3 [] ondisk, result = 0)
2014-06-12 14:47:01.855417 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.855420 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.865057 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:01.865061 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:01.865063 7f9d74bfa700 20 osd.2 173 _dispatch 0x17705520 ping v1
2014-06-12 14:47:01.865065 7f9d74bfa700 10 osd.2 173 ping from client.4314
2014-06-12 14:47:01.865068 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:01.865069 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:01.889434 7f9d5a16f700  5 -- op tracker -- , seq: 748271, time: 2014-06-12 14:47:01.888332, event: header_read, op: osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.889445 7f9d5a16f700  5 -- op tracker -- , seq: 748271, time: 2014-06-12 14:47:01.888336, event: throttled, op: osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.889468 7f9d5a16f700  5 -- op tracker -- , seq: 748271, time: 2014-06-12 14:47:01.889395, event: all_read, op: osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.889486 7f9d5a16f700  5 -- op tracker -- , seq: 748271, time: 0.000000, event: dispatched, op: osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.889491 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:01.889495 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:01.889502 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9c600 prio 127 cost 516903 latency 0.001170 osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.889526 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.889534 7f9d61bd4700 10 osd.2 173 dequeue_op 0x24c9c600 prio 127 cost 516903 latency 0.001201 osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2511 (0'0,173'2511] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2510 active]
2014-06-12 14:47:01.889551 7f9d61bd4700  5 -- op tracker -- , seq: 748271, time: 2014-06-12 14:47:01.889549, event: reached_pg, op: osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.889558 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2511 (0'0,173'2511] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2510 active] handle_message: 0x24c9c600
2014-06-12 14:47:01.889567 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2511 (0'0,173'2511] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2510 active] sub_op_modify trans f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 v 173'2512 (transaction) 174
2014-06-12 14:47:01.889577 7f9d61bd4700  5 -- op tracker -- , seq: 748271, time: 2014-06-12 14:47:01.889574, event: started, op: osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.889589 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2511 (0'0,173'2511] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2510 active] append_log log((0'0,173'2511], crt=0'0) [173'2512 (173'2510) modify   f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 by client.4310.0:142327 2014-06-12 08:44:15.577282]
2014-06-12 14:47:01.889608 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2512 (0'0,173'2512] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2510 active] add_log_entry 173'2512 (173'2510) modify   f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 by client.4310.0:142327 2014-06-12 08:44:15.577282
2014-06-12 14:47:01.889632 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2512 (0'0,173'2512] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2510 active] append_log  adding 1 keys
2014-06-12 14:47:01.889657 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2512, trimmed: 
2014-06-12 14:47:01.889675 7f9d61bd4700  5 -- op tracker -- , seq: 748271, time: 2014-06-12 14:47:01.889672, event: started, op: osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.889683 7f9d61bd4700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:01.889686 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:01.889687 7f9d61bd4700  5 queue_op 0x8372ba0 seq 2358 osr(3.168 0x276ce40) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:01.889693 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2512 (0'0,173'2512] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2510 active] do_sub_op osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.889698 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:47:01.889702 7f9d61bd4700 10 osd.2 173 dequeue_op 0x24c9c600 finish
2014-06-12 14:47:01.889704 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:01.889706 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:01.889705 7f9d7915f700  5 _do_op 0x8372ba0 seq 2358 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:47:01.889711 7f9d7915f700 10 _do_transaction on 0x5f4ad30
2014-06-12 14:47:01.889718 7f9d7915f700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:01.889867 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:01.889888 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:01.889941 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:01.890052 7f9d7915f700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:01.890059 7f9d7915f700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:47:01.890065 7f9d7915f700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:01.890083 7f9d7915f700 15 _write 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 876544~516096
2014-06-12 14:47:01.890269 7f9d7915f700 10 lookup_strip_header done  cid 3.168_head oid f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:01.890446 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:47:01.892546 7f9d7915f700 10 _generic_write 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 876544~516096 = 0
2014-06-12 14:47:01.892598 7f9d7915f700 15 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:01.892615 7f9d7915f700 10 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 = 0
2014-06-12 14:47:01.892620 7f9d7915f700 15 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:01.892628 7f9d7915f700 10 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 = 0
2014-06-12 14:47:01.892832 7f9d7915f700 20 set_header setting 16446 cid 3.168_head oid f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 parent seq 0
2014-06-12 14:47:01.892902 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:01.892916 7f9d7915f700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:47:01.892926 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:47:01.895381 7f9d7915f700 10 _do_op 0x8372ba0 seq 2358 r = 0, finisher 0x5f76c00 0
2014-06-12 14:47:01.895399 7f9d7915f700 10 _finish_op 0x8372ba0 seq 2358 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:01.895408 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:47:01.895419 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:01.895413 7f9d7715b700  5 -- op tracker -- , seq: 748271, time: 2014-06-12 14:47:01.895405, event: commit_sent, op: osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.895422 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2512 (0'0,173'2512] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2510 active] sub_op_modify_commit on op osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:01.895439 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:01.895466 7f9d7795c700  5 -- op tracker -- , seq: 748271, time: 2014-06-12 14:47:01.895459, event: sub_op_applied, op: osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.895488 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2512 (0'0,173'2512] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2511 active] sub_op_modify_applied on 0x5f4ac40 op osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:01.895499 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2512 (0'0,173'2512] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2511 active] op_applied on primary on version 173'2512
2014-06-12 14:47:01.895504 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2512 (0'0,173'2512] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2511 active] op_applied on replica on version 173'2512
2014-06-12 14:47:01.895518 7f9d7795c700  5 -- op tracker -- , seq: 748271, time: 2014-06-12 14:47:01.895516, event: done, op: osd_sub_op(client.4310.0:142327 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2512 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:01.995540 7f9d4a5d9700  5 -- op tracker -- , seq: 748272, time: 2014-06-12 14:47:01.995000, event: header_read, op: osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.995552 7f9d4a5d9700  5 -- op tracker -- , seq: 748272, time: 2014-06-12 14:47:01.995007, event: throttled, op: osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.995576 7f9d4a5d9700  5 -- op tracker -- , seq: 748272, time: 2014-06-12 14:47:01.995497, event: all_read, op: osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.995584 7f9d4a5d9700  5 -- op tracker -- , seq: 748272, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.995591 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:01.995613 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x68a7200 prio 63 cost 233472 latency 0.000613 osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:01.995642 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:01.995652 7f9d63bd8700 10 osd.2 173 dequeue_op 0x68a7200 prio 63 cost 233472 latency 0.000652 osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]
2014-06-12 14:47:01.995681 7f9d63bd8700  5 -- op tracker -- , seq: 748272, time: 2014-06-12 14:47:01.995675, event: reached_pg, op: osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:01.995693 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:01.995709 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] handle_message: 0x68a7200
2014-06-12 14:47:01.995720 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_op osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:01.995746 7f9d63bd8700 15 getattr 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 '_'
2014-06-12 14:47:02.011247 7f9d4a5d9700  5 -- op tracker -- , seq: 748273, time: 2014-06-12 14:47:02.010098, event: header_read, op: osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.011259 7f9d4a5d9700  5 -- op tracker -- , seq: 748273, time: 2014-06-12 14:47:02.010102, event: throttled, op: osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.011265 7f9d4a5d9700  5 -- op tracker -- , seq: 748273, time: 2014-06-12 14:47:02.011206, event: all_read, op: osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.011270 7f9d4a5d9700  5 -- op tracker -- , seq: 748273, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.011277 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:02.011300 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x17c2a200 prio 63 cost 516096 latency 0.001202 osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:02.011327 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:02.018106 7f9d63bd8700 10 getattr 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 '_' = 0
2014-06-12 14:47:02.018129 7f9d63bd8700 15 getattr 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 'snapset'
2014-06-12 14:47:02.018262 7f9d63bd8700 10 getattr 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 'snapset' = 0
2014-06-12 14:47:02.018271 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] populate_obc_watchers cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.018283 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.018290 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] get_object_context: creating obc from disk: 0x15574840
2014-06-12 14:47:02.018296 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] get_object_context: 0x15574840 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 rwstate(none n=0 w=0) oi: cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'822 client.4305.0:36511 wrlock_by=unknown.0.0:0 dirty s 4194304 uv822) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:47:02.018310 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] find_object_context cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 @head oi=cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'822 client.4305.0:36511 wrlock_by=unknown.0.0:0 dirty s 4194304 uv822)
2014-06-12 14:47:02.018332 7f9d63bd8700  5 -- op tracker -- , seq: 748272, time: 2014-06-12 14:47:02.018326, event: started, op: osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.018339 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] execute_ctx 0x121ffe00
2014-06-12 14:47:02.018350 7f9d63bd8700  5 -- op tracker -- , seq: 748272, time: 2014-06-12 14:47:02.018347, event: started, op: osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.018355 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] ov 173'822 av 173'2858 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:02.018364 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472]
2014-06-12 14:47:02.018372 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:02.018381 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op  write 0~233472
2014-06-12 14:47:02.018393 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:02.018401 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:47:02.018409 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:02.018416 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] finish_ctx cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 0x121ffe00 op modify  
2014-06-12 14:47:02.018424 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  set mtime to 2014-06-12 08:46:04.374733
2014-06-12 14:47:02.018437 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  final snapset 0=[]:[]+head in cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.018451 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  zeroing write result code 0
2014-06-12 14:47:02.018459 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] new_repop rep_tid 236386 on osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:02.018471 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] issue_repop rep_tid 236386 o cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.018488 7f9d63bd8700  5 -- op tracker -- , seq: 748272, time: 2014-06-12 14:47:02.018485, event: waiting for subops from 23, op: osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.018506 7f9d63bd8700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:02.018524 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2857 (0'0,173'2857] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] append_log log((0'0,173'2857], crt=0'0) [173'2858 (173'822) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142531 2014-06-12 08:46:04.374733]
2014-06-12 14:47:02.018542 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] add_log_entry 173'2858 (173'822) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142531 2014-06-12 08:46:04.374733
2014-06-12 14:47:02.018556 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] append_log  adding 1 keys
2014-06-12 14:47:02.018582 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2858, trimmed: 
2014-06-12 14:47:02.018600 7f9d63bd8700  5 queue_transactions existing osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.018603 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:02.018605 7f9d63bd8700  5 queue_op 0x208be900 seq 2603 osr(3.650 0x276dff0) 235765 bytes   (queue has 1 ops and 235765 bytes)
2014-06-12 14:47:02.018615 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5440 (1 active)
2014-06-12 14:47:02.018621 7f9d8016d700  5 _do_op 0x208be900 seq 2603 osr(3.650 0x276dff0)/0x276dff0 start
2014-06-12 14:47:02.018624 7f9d8016d700 10 _do_transaction on 0x53232c0
2014-06-12 14:47:02.018610 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] eval_repop repgather(0x3a097440 173'2858 rep_tid=236386 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.018628 7f9d63bd8700 10 osd.2 173 dequeue_op 0x68a7200 finish
2014-06-12 14:47:02.018630 7f9d8016d700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.018637 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:02.018642 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.018643 7f9d633d7700 10 osd.2 173 dequeue_op 0x17c2a200 prio 63 cost 516096 latency 0.008544 osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]
2014-06-12 14:47:02.018672 7f9d633d7700  5 -- op tracker -- , seq: 748273, time: 2014-06-12 14:47:02.018666, event: reached_pg, op: osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.018684 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:02.018703 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] handle_message: 0x17c2a200
2014-06-12 14:47:02.018716 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_op osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:02.018736 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.018740 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] get_object_context: found obc in cache: 0x15574840
2014-06-12 14:47:02.018755 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:02.018753 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] get_object_context: 0x15574840 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 rwstate(write n=1 w=0) oi: cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'2858 client.4305.0:142531 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2858) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:47:02.018771 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] find_object_context cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 @head oi=cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'2858 client.4305.0:142531 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2858)
2014-06-12 14:47:02.018794 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:02.018796 7f9d633d7700  5 -- op tracker -- , seq: 748273, time: 2014-06-12 14:47:02.018790, event: started, op: osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.018801 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] execute_ctx 0x121fe000
2014-06-12 14:47:02.018814 7f9d633d7700  5 -- op tracker -- , seq: 748273, time: 2014-06-12 14:47:02.018811, event: started, op: osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.018820 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] ov 173'2858 av 173'2859 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:02.018832 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096]
2014-06-12 14:47:02.018843 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:02.018853 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op  write 233472~516096
2014-06-12 14:47:02.018878 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:02.018885 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:47:02.018893 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:02.018906 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] finish_ctx cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 0x121fe000 op modify  
2014-06-12 14:47:02.018922 7f9d8016d700 15 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.018914 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  set mtime to 2014-06-12 08:46:04.389344
2014-06-12 14:47:02.018928 7f9d8016d700 10 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1 r = 0
2014-06-12 14:47:02.018932 7f9d8016d700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.018937 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  final snapset 0=[]:[]+head in cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.018945 7f9d8016d700 15 _write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 0~233472
2014-06-12 14:47:02.018948 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  zeroing write result code 0
2014-06-12 14:47:02.018954 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] new_repop rep_tid 236387 on osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:02.018963 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] issue_repop rep_tid 236387 o cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.018976 7f9d633d7700  5 -- op tracker -- , seq: 748273, time: 2014-06-12 14:47:02.018973, event: waiting for subops from 23, op: osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.018991 7f9d633d7700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:02.019012 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2858 (0'0,173'2858] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] append_log log((0'0,173'2858], crt=0'0) [173'2859 (173'2858) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142532 2014-06-12 08:46:04.389344]
2014-06-12 14:47:02.019032 7f9d8016d700 10 lookup_strip_header done  cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.019035 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] add_log_entry 173'2859 (173'2858) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142532 2014-06-12 08:46:04.389344
2014-06-12 14:47:02.019054 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] append_log  adding 1 keys
2014-06-12 14:47:02.019083 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2859, trimmed: 
2014-06-12 14:47:02.019108 7f9d633d7700  5 queue_transactions existing osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.019112 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:02.019113 7f9d633d7700  5 queue_op 0x8103e00 seq 2604 osr(3.650 0x276dff0) 518389 bytes   (queue has 2 ops and 754154 bytes)
2014-06-12 14:47:02.019123 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5440 (2 active)
2014-06-12 14:47:02.019119 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] eval_repop repgather(0xd5e3bc0 173'2859 rep_tid=236387 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.019135 7f9d633d7700 10 osd.2 173 dequeue_op 0x17c2a200 finish
2014-06-12 14:47:02.019138 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.019141 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.019164 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:47:02.019813 7f9d8016d700 10 _generic_write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 0~233472 = 0
2014-06-12 14:47:02.019843 7f9d8016d700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.019852 7f9d8016d700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.019857 7f9d8016d700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.019862 7f9d8016d700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.019957 7f9d8016d700 20 set_header setting 16458 cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 parent seq 0
2014-06-12 14:47:02.019995 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:02.020005 7f9d8016d700 20 set_header setting 2114 cid meta oid 1fe1320a/pglog_3.650/0//-1 parent seq 0
2014-06-12 14:47:02.020013 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:47:02.020926 7f9d8016d700 10 _do_op 0x208be900 seq 2603 r = 0, finisher 0x89410e0 0x38e0cbc0
2014-06-12 14:47:02.020945 7f9d8016d700 10 _finish_op 0x208be900 seq 2603 osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.020959 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5440 (1 active)
2014-06-12 14:47:02.020962 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:02.020953 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_commit: 236386
2014-06-12 14:47:02.020988 7f9d79960700  5 _do_op 0x8103e00 seq 2604 osr(3.650 0x276dff0)/0x276dff0 start
2014-06-12 14:47:02.020991 7f9d79960700 10 _do_transaction on 0xd5e2240
2014-06-12 14:47:02.020997 7f9d79960700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.020991 7f9d7715b700  5 -- op tracker -- , seq: 748272, time: 2014-06-12 14:47:02.020979, event: op_commit, op: osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.021008 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_applied: 236386
2014-06-12 14:47:02.021030 7f9d7795c700  5 -- op tracker -- , seq: 748272, time: 2014-06-12 14:47:02.021025, event: op_applied, op: osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.021036 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_applied on primary on version 173'2858
2014-06-12 14:47:02.021094 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.021117 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:02.021153 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:02.021175 7f9d79960700 15 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.021182 7f9d79960700 10 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1 r = 0
2014-06-12 14:47:02.021187 7f9d79960700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.021344 7f9d79960700 15 _write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 233472~516096
2014-06-12 14:47:02.021479 7f9d79960700 10 lookup_strip_header done  cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.021604 7f9d79960700 10 file_to_extents done 
2014-06-12 14:47:02.023010 7f9d57543700  5 -- op tracker -- , seq: 748274, time: 2014-06-12 14:47:02.022967, event: header_read, op: osd_sub_op_reply(client.4305.0:142531 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.023020 7f9d57543700  5 -- op tracker -- , seq: 748274, time: 2014-06-12 14:47:02.022969, event: throttled, op: osd_sub_op_reply(client.4305.0:142531 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.023036 7f9d57543700  5 -- op tracker -- , seq: 748274, time: 2014-06-12 14:47:02.022984, event: all_read, op: osd_sub_op_reply(client.4305.0:142531 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.023042 7f9d57543700  5 -- op tracker -- , seq: 748274, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142531 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.023042 7f9d79960700 10 _generic_write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 233472~516096 = 0
2014-06-12 14:47:02.023046 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142531 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:02.023051 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:02.023058 7f9d57543700 15 osd.2 173 enqueue_op 0x23e09a00 prio 196 cost 0 latency 0.000090 osd_sub_op_reply(client.4305.0:142531 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:02.023078 7f9d79960700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.023085 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.023088 7f9d79960700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.023093 7f9d79960700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.023097 7f9d79960700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.023092 7f9d67be0700 10 osd.2 173 dequeue_op 0x23e09a00 prio 196 cost 0 latency 0.000125 osd_sub_op_reply(client.4305.0:142531 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 pg pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]
2014-06-12 14:47:02.023107 7f9d67be0700  5 -- op tracker -- , seq: 748274, time: 2014-06-12 14:47:02.023105, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142531 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.023114 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] handle_message: 0x23e09a00
2014-06-12 14:47:02.023124 7f9d67be0700  5 -- op tracker -- , seq: 748274, time: 2014-06-12 14:47:02.023122, event: started, op: osd_sub_op_reply(client.4305.0:142531 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.023128 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] sub_op_modify_reply: tid 236386 op  ack_type 4 from 23
2014-06-12 14:47:02.023145 7f9d67be0700  5 -- op tracker -- , seq: 748272, time: 2014-06-12 14:47:02.023137, event: sub_op_commit_rec, op: osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.023152 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] repop_all_applied: repop tid 236386 all applied 
2014-06-12 14:47:02.023160 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] eval_repop repgather(0x3a097440 173'2858 rep_tid=236386 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.023170 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  sending ack on repgather(0x3a097440 173'2858 rep_tid=236386 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0x1f889900
2014-06-12 14:47:02.023191 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] repop_all_committed: repop tid 236386 all committed 
2014-06-12 14:47:02.023201 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean] eval_repop repgather(0x3a097440 173'2858 rep_tid=236386 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.023217 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean] log_op_stats osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 inb 233751 outb 0 rlat 0.028189 lat 0.028216
2014-06-12 14:47:02.023230 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean] publish_stats_to_osd 173:3717
2014-06-12 14:47:02.023243 7f9d79960700 20 set_header setting 16458 cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 parent seq 0
2014-06-12 14:47:02.023240 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean]  sending commit on repgather(0x3a097440 173'2858 rep_tid=236386 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0xc0c9b80
2014-06-12 14:47:02.023279 7f9d67be0700  5 -- op tracker -- , seq: 748272, time: 2014-06-12 14:47:02.023273, event: commit_sent, op: osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.023290 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean]  requeue_ops 
2014-06-12 14:47:02.023299 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  removing repgather(0x3a097440 173'2858 rep_tid=236386 committed?=1 applied?=1 op=osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.023311 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]    q front is repgather(0x3a097440 173'2858 rep_tid=236386 committed?=1 applied?=1 op=osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.023321 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:02.023321 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] remove_repop repgather(0x3a097440 173'2858 rep_tid=236386 committed?=1 applied?=1 op=osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.023331 7f9d79960700 20 set_header setting 2114 cid meta oid 1fe1320a/pglog_3.650/0//-1 parent seq 0
2014-06-12 14:47:02.023330 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  requeue_ops 
2014-06-12 14:47:02.023337 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:47:02.023342 7f9d67be0700  5 -- op tracker -- , seq: 748272, time: 2014-06-12 14:47:02.023339, event: done, op: osd_op(client.4305.0:142531 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.023441 7f9d67be0700 10 osd.2 173 dequeue_op 0x23e09a00 finish
2014-06-12 14:47:02.023448 7f9d67be0700  5 -- op tracker -- , seq: 748274, time: 2014-06-12 14:47:02.023446, event: done, op: osd_sub_op_reply(client.4305.0:142531 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.023455 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.023457 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.024120 7f9d4a5d9700  5 -- op tracker -- , seq: 748275, time: 2014-06-12 14:47:02.021796, event: header_read, op: osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.024133 7f9d4a5d9700  5 -- op tracker -- , seq: 748275, time: 2014-06-12 14:47:02.021799, event: throttled, op: osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.024143 7f9d4a5d9700  5 -- op tracker -- , seq: 748275, time: 2014-06-12 14:47:02.024090, event: all_read, op: osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.024157 7f9d4a5d9700  5 -- op tracker -- , seq: 748275, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.024165 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:02.024191 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x17c2ba00 prio 63 cost 1032192 latency 0.002395 osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:02.024217 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.024224 7f9d673df700 10 osd.2 173 dequeue_op 0x17c2ba00 prio 63 cost 1032192 latency 0.002428 osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]
2014-06-12 14:47:02.024242 7f9d673df700  5 -- op tracker -- , seq: 748275, time: 2014-06-12 14:47:02.024239, event: reached_pg, op: osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.024250 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:02.024261 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] handle_message: 0x17c2ba00
2014-06-12 14:47:02.024268 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_op osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:02.024286 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] get_object_context: found obc in cache: 0x15574840
2014-06-12 14:47:02.024293 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] get_object_context: 0x15574840 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 rwstate(write n=1 w=0) oi: cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'2859 client.4305.0:142532 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2859) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:47:02.024305 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] find_object_context cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 @head oi=cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'2859 client.4305.0:142532 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2859)
2014-06-12 14:47:02.024323 7f9d673df700  5 -- op tracker -- , seq: 748275, time: 2014-06-12 14:47:02.024320, event: started, op: osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.024329 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] execute_ctx 0x580be00
2014-06-12 14:47:02.024339 7f9d673df700  5 -- op tracker -- , seq: 748275, time: 2014-06-12 14:47:02.024336, event: started, op: osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.024344 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] ov 173'2859 av 173'2860 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:02.024353 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_osd_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192]
2014-06-12 14:47:02.024361 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:02.024370 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_osd_op  write 749568~1032192
2014-06-12 14:47:02.024381 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:02.024389 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:47:02.024398 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:02.024404 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] finish_ctx cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 0x580be00 op modify  
2014-06-12 14:47:02.024413 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  set mtime to 2014-06-12 08:46:04.400122
2014-06-12 14:47:02.024429 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  final snapset 0=[]:[]+head in cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.024441 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  zeroing write result code 0
2014-06-12 14:47:02.024449 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] new_repop rep_tid 236388 on osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:02.024461 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] issue_repop rep_tid 236388 o cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.024477 7f9d673df700  5 -- op tracker -- , seq: 748275, time: 2014-06-12 14:47:02.024474, event: waiting for subops from 23, op: osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.024493 7f9d673df700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:02.024509 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2859 (0'0,173'2859] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] append_log log((0'0,173'2859], crt=0'0) [173'2860 (173'2859) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142533 2014-06-12 08:46:04.400122]
2014-06-12 14:47:02.024526 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] add_log_entry 173'2860 (173'2859) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142533 2014-06-12 08:46:04.400122
2014-06-12 14:47:02.024540 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] append_log  adding 1 keys
2014-06-12 14:47:02.024561 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2860, trimmed: 
2014-06-12 14:47:02.024583 7f9d673df700  5 queue_transactions existing osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.024587 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:02.024588 7f9d673df700  5 queue_op 0x1d206540 seq 2605 osr(3.650 0x276dff0) 1034485 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:47:02.024597 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5440 (2 active)
2014-06-12 14:47:02.024594 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] eval_repop repgather(0x5312240 173'2860 rep_tid=236388 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.024610 7f9d673df700 10 osd.2 173 dequeue_op 0x17c2ba00 finish
2014-06-12 14:47:02.024614 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.024616 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.025636 7f9d79960700 10 _do_op 0x8103e00 seq 2604 r = 0, finisher 0x89f8c60 0x7fa5bc0
2014-06-12 14:47:02.025655 7f9d79960700 10 _finish_op 0x8103e00 seq 2604 osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.025663 7f9d7d968700  5 _do_op 0x1d206540 seq 2605 osr(3.650 0x276dff0)/0x276dff0 start
2014-06-12 14:47:02.025664 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5440 (1 active)
2014-06-12 14:47:02.025670 7f9d7d968700 10 _do_transaction on 0x5312000
2014-06-12 14:47:02.025671 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:02.025661 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] op_commit: 236387
2014-06-12 14:47:02.025679 7f9d7d968700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.025689 7f9d7715b700  5 -- op tracker -- , seq: 748273, time: 2014-06-12 14:47:02.025680, event: op_commit, op: osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.025703 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] op_applied: 236387
2014-06-12 14:47:02.025740 7f9d7795c700  5 -- op tracker -- , seq: 748273, time: 2014-06-12 14:47:02.025734, event: op_applied, op: osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.025751 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] op_applied on primary on version 173'2859
2014-06-12 14:47:02.025848 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.025867 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:02.025900 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:02.025912 7f9d7d968700 15 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.025917 7f9d7d968700 10 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1 r = 0
2014-06-12 14:47:02.025921 7f9d7d968700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.025930 7f9d7d968700 15 _write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 749568~1032192
2014-06-12 14:47:02.026021 7f9d7d968700 10 lookup_strip_header done  cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.026047 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:47:02.028358 7f9d57543700  5 -- op tracker -- , seq: 748276, time: 2014-06-12 14:47:02.028244, event: header_read, op: osd_sub_op_reply(client.4305.0:142532 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.028389 7f9d57543700  5 -- op tracker -- , seq: 748276, time: 2014-06-12 14:47:02.028247, event: throttled, op: osd_sub_op_reply(client.4305.0:142532 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.028445 7f9d57543700  5 -- op tracker -- , seq: 748276, time: 2014-06-12 14:47:02.028268, event: all_read, op: osd_sub_op_reply(client.4305.0:142532 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.028453 7f9d57543700  5 -- op tracker -- , seq: 748276, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142532 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.028463 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142532 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:02.028472 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:02.028486 7f9d57543700 15 osd.2 173 enqueue_op 0x23e09800 prio 196 cost 0 latency 0.000241 osd_sub_op_reply(client.4305.0:142532 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:02.028524 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.028535 7f9d663dd700 10 osd.2 173 dequeue_op 0x23e09800 prio 196 cost 0 latency 0.000291 osd_sub_op_reply(client.4305.0:142532 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 pg pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]
2014-06-12 14:47:02.028567 7f9d663dd700  5 -- op tracker -- , seq: 748276, time: 2014-06-12 14:47:02.028563, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142532 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.028577 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] handle_message: 0x23e09800
2014-06-12 14:47:02.028592 7f9d663dd700  5 -- op tracker -- , seq: 748276, time: 2014-06-12 14:47:02.028589, event: started, op: osd_sub_op_reply(client.4305.0:142532 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.028598 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] sub_op_modify_reply: tid 236387 op  ack_type 4 from 23
2014-06-12 14:47:02.028618 7f9d663dd700  5 -- op tracker -- , seq: 748273, time: 2014-06-12 14:47:02.028610, event: sub_op_commit_rec, op: osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.028625 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] repop_all_applied: repop tid 236387 all applied 
2014-06-12 14:47:02.028641 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] eval_repop repgather(0xd5e3bc0 173'2859 rep_tid=236387 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.028656 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  sending ack on repgather(0xd5e3bc0 173'2859 rep_tid=236387 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0xfdb2500
2014-06-12 14:47:02.028693 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] repop_all_committed: repop tid 236387 all committed 
2014-06-12 14:47:02.028702 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] eval_repop repgather(0xd5e3bc0 173'2859 rep_tid=236387 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.028724 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] log_op_stats osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.018592 lat 0.018624
2014-06-12 14:47:02.028741 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] publish_stats_to_osd 173:3718
2014-06-12 14:47:02.028754 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean]  sending commit on repgather(0xd5e3bc0 173'2859 rep_tid=236387 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0x33770000
2014-06-12 14:47:02.028784 7f9d663dd700  5 -- op tracker -- , seq: 748273, time: 2014-06-12 14:47:02.028780, event: commit_sent, op: osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.028792 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean]  requeue_ops 
2014-06-12 14:47:02.028803 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  removing repgather(0xd5e3bc0 173'2859 rep_tid=236387 committed?=1 applied?=1 op=osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.028816 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]    q front is repgather(0xd5e3bc0 173'2859 rep_tid=236387 committed?=1 applied?=1 op=osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.028828 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] remove_repop repgather(0xd5e3bc0 173'2859 rep_tid=236387 committed?=1 applied?=1 op=osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.028840 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  requeue_ops 
2014-06-12 14:47:02.028860 7f9d663dd700  5 -- op tracker -- , seq: 748273, time: 2014-06-12 14:47:02.028855, event: done, op: osd_op(client.4305.0:142532 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.028882 7f9d663dd700 10 osd.2 173 dequeue_op 0x23e09800 finish
2014-06-12 14:47:02.028888 7f9d663dd700  5 -- op tracker -- , seq: 748276, time: 2014-06-12 14:47:02.028885, event: done, op: osd_sub_op_reply(client.4305.0:142532 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.028899 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.028902 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.029655 7f9d7d968700 10 _generic_write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 749568~1032192 = 0
2014-06-12 14:47:02.029776 7f9d7d968700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.029800 7f9d7d968700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.029812 7f9d7d968700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.029823 7f9d7d968700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.030187 7f9d7d968700 20 set_header setting 16458 cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 parent seq 0
2014-06-12 14:47:02.030296 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:02.030312 7f9d7d968700 20 set_header setting 2114 cid meta oid 1fe1320a/pglog_3.650/0//-1 parent seq 0
2014-06-12 14:47:02.030326 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:47:02.032143 7f9d4a5d9700  5 -- op tracker -- , seq: 748277, time: 2014-06-12 14:47:02.031040, event: header_read, op: osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.032156 7f9d4a5d9700  5 -- op tracker -- , seq: 748277, time: 2014-06-12 14:47:02.031044, event: throttled, op: osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.032165 7f9d4a5d9700  5 -- op tracker -- , seq: 748277, time: 2014-06-12 14:47:02.032095, event: all_read, op: osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.032173 7f9d4a5d9700  5 -- op tracker -- , seq: 748277, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.032183 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:02.032212 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x17c2ab00 prio 63 cost 516096 latency 0.001172 osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:02.032249 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.032260 7f9d65bdc700 10 osd.2 173 dequeue_op 0x17c2ab00 prio 63 cost 516096 latency 0.001220 osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]
2014-06-12 14:47:02.032292 7f9d65bdc700  5 -- op tracker -- , seq: 748277, time: 2014-06-12 14:47:02.032285, event: reached_pg, op: osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.032310 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:02.032327 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] handle_message: 0x17c2ab00
2014-06-12 14:47:02.032338 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_op osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:02.032363 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] get_object_context: found obc in cache: 0x15574840
2014-06-12 14:47:02.032376 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] get_object_context: 0x15574840 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 rwstate(write n=1 w=0) oi: cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'2860 client.4305.0:142533 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2860) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:47:02.032394 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] find_object_context cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 @head oi=cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'2860 client.4305.0:142533 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2860)
2014-06-12 14:47:02.032427 7f9d65bdc700  5 -- op tracker -- , seq: 748277, time: 2014-06-12 14:47:02.032422, event: started, op: osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.032439 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] execute_ctx 0x55a3200
2014-06-12 14:47:02.032464 7f9d65bdc700  5 -- op tracker -- , seq: 748277, time: 2014-06-12 14:47:02.032459, event: started, op: osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.032471 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] ov 173'2860 av 173'2861 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:02.032485 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096]
2014-06-12 14:47:02.032499 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:02.032521 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op  write 1781760~516096
2014-06-12 14:47:02.032547 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:02.032561 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:47:02.032583 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:02.032600 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] finish_ctx cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 0x55a3200 op modify  
2014-06-12 14:47:02.032614 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  set mtime to 2014-06-12 08:46:04.410292
2014-06-12 14:47:02.032639 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  final snapset 0=[]:[]+head in cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.032658 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  zeroing write result code 0
2014-06-12 14:47:02.032672 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] new_repop rep_tid 236389 on osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:02.032692 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] issue_repop rep_tid 236389 o cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.032721 7f9d65bdc700  5 -- op tracker -- , seq: 748277, time: 2014-06-12 14:47:02.032715, event: waiting for subops from 23, op: osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.032751 7f9d65bdc700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:02.032780 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2860 (0'0,173'2860] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] append_log log((0'0,173'2860], crt=0'0) [173'2861 (173'2860) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142534 2014-06-12 08:46:04.410292]
2014-06-12 14:47:02.032808 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] add_log_entry 173'2861 (173'2860) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142534 2014-06-12 08:46:04.410292
2014-06-12 14:47:02.032833 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] append_log  adding 1 keys
2014-06-12 14:47:02.032869 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2861, trimmed: 
2014-06-12 14:47:02.032897 7f9d65bdc700  5 queue_transactions existing osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.032903 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:02.032905 7f9d65bdc700  5 queue_op 0x17dd84e0 seq 2606 osr(3.650 0x276dff0) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:47:02.032913 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] eval_repop repgather(0x2dc1f500 173'2861 rep_tid=236389 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.032934 7f9d65bdc700 10 osd.2 173 dequeue_op 0x17c2ab00 finish
2014-06-12 14:47:02.032940 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.032944 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.032954 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5440 (2 active)
2014-06-12 14:47:02.035662 7f9d4a5d9700  5 -- op tracker -- , seq: 748278, time: 2014-06-12 14:47:02.033451, event: header_read, op: osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.035678 7f9d4a5d9700  5 -- op tracker -- , seq: 748278, time: 2014-06-12 14:47:02.033454, event: throttled, op: osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.035686 7f9d4a5d9700  5 -- op tracker -- , seq: 748278, time: 2014-06-12 14:47:02.035612, event: all_read, op: osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.035700 7f9d4a5d9700  5 -- op tracker -- , seq: 748278, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.035708 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:02.035735 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x17c2b400 prio 63 cost 1032192 latency 0.002283 osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:02.035771 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.035780 7f9d62bd6700 10 osd.2 173 dequeue_op 0x17c2b400 prio 63 cost 1032192 latency 0.002328 osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]
2014-06-12 14:47:02.035805 7f9d62bd6700  5 -- op tracker -- , seq: 748278, time: 2014-06-12 14:47:02.035800, event: reached_pg, op: osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.035819 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:02.035834 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] handle_message: 0x17c2b400
2014-06-12 14:47:02.035843 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_op osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:02.035866 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] get_object_context: found obc in cache: 0x15574840
2014-06-12 14:47:02.035875 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] get_object_context: 0x15574840 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 rwstate(write n=2 w=0) oi: cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'2861 client.4305.0:142534 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2861) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:47:02.035891 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] find_object_context cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 @head oi=cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'2861 client.4305.0:142534 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2861)
2014-06-12 14:47:02.035915 7f9d62bd6700  5 -- op tracker -- , seq: 748278, time: 2014-06-12 14:47:02.035910, event: started, op: osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.035921 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] execute_ctx 0x2e411800
2014-06-12 14:47:02.035938 7f9d62bd6700  5 -- op tracker -- , seq: 748278, time: 2014-06-12 14:47:02.035933, event: started, op: osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.035944 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] ov 173'2861 av 173'2862 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:02.035959 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192]
2014-06-12 14:47:02.035970 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:02.035984 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op  write 2297856~1032192
2014-06-12 14:47:02.036000 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:02.036012 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:47:02.036024 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:02.036035 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] finish_ctx cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 0x2e411800 op modify  
2014-06-12 14:47:02.036047 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  set mtime to 2014-06-12 08:46:04.411530
2014-06-12 14:47:02.036069 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  final snapset 0=[]:[]+head in cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.036089 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  zeroing write result code 0
2014-06-12 14:47:02.036100 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] new_repop rep_tid 236390 on osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:02.036117 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] issue_repop rep_tid 236390 o cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.036141 7f9d62bd6700  5 -- op tracker -- , seq: 748278, time: 2014-06-12 14:47:02.036136, event: waiting for subops from 23, op: osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.036162 7f9d62bd6700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:02.036186 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2861 (0'0,173'2861] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] append_log log((0'0,173'2861], crt=0'0) [173'2862 (173'2861) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142535 2014-06-12 08:46:04.411530]
2014-06-12 14:47:02.036212 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] add_log_entry 173'2862 (173'2861) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142535 2014-06-12 08:46:04.411530
2014-06-12 14:47:02.036233 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] append_log  adding 1 keys
2014-06-12 14:47:02.036263 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2862, trimmed: 
2014-06-12 14:47:02.036284 7f9d62bd6700  5 queue_transactions existing osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.036289 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:02.036291 7f9d62bd6700  5 queue_op 0x61c63c0 seq 2607 osr(3.650 0x276dff0) 1034485 bytes   (queue has 3 ops and 2587359 bytes)
2014-06-12 14:47:02.036298 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] eval_repop repgather(0x1c378b40 173'2862 rep_tid=236390 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.036315 7f9d62bd6700 10 osd.2 173 dequeue_op 0x17c2b400 finish
2014-06-12 14:47:02.036318 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.036320 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.036328 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5440 (3 active)
2014-06-12 14:47:02.037384 7f9d7d968700 10 _do_op 0x1d206540 seq 2605 r = 0, finisher 0x693bb20 0x6be5600
2014-06-12 14:47:02.037404 7f9d7d968700 10 _finish_op 0x1d206540 seq 2605 osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.037411 7f9d7c165700  5 _do_op 0x17dd84e0 seq 2606 osr(3.650 0x276dff0)/0x276dff0 start
2014-06-12 14:47:02.037416 7f9d7c165700 10 _do_transaction on 0xbe0ccc0
2014-06-12 14:47:02.037409 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_commit: 236388
2014-06-12 14:47:02.037425 7f9d7c165700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.037436 7f9d7715b700  5 -- op tracker -- , seq: 748275, time: 2014-06-12 14:47:02.037428, event: op_commit, op: osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.037535 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.037555 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:02.037593 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:02.037608 7f9d7c165700 15 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.037625 7f9d7c165700 10 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1 r = 0
2014-06-12 14:47:02.037636 7f9d7c165700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.037786 7f9d7c165700 15 _write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 1781760~516096
2014-06-12 14:47:02.037885 7f9d7c165700 10 lookup_strip_header done  cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.037992 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:47:02.038113 7f9d4a5d9700  5 -- op tracker -- , seq: 748279, time: 2014-06-12 14:47:02.037008, event: header_read, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.038123 7f9d4a5d9700  5 -- op tracker -- , seq: 748279, time: 2014-06-12 14:47:02.037012, event: throttled, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.038132 7f9d4a5d9700  5 -- op tracker -- , seq: 748279, time: 2014-06-12 14:47:02.038081, event: all_read, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.038140 7f9d4a5d9700  5 -- op tracker -- , seq: 748279, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.038147 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:02.038173 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x17c2b800 prio 63 cost 516096 latency 0.001164 osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:02.038214 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.038221 7f9d683e1700 10 osd.2 173 dequeue_op 0x17c2b800 prio 63 cost 516096 latency 0.001213 osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]
2014-06-12 14:47:02.038243 7f9d683e1700  5 -- op tracker -- , seq: 748279, time: 2014-06-12 14:47:02.038238, event: reached_pg, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.038251 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:02.038265 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] handle_message: 0x17c2b800
2014-06-12 14:47:02.038274 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_op osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:02.038298 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] get_object_context: found obc in cache: 0x15574840
2014-06-12 14:47:02.038308 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] get_object_context: 0x15574840 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 rwstate(write n=3 w=0) oi: cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'2862 client.4305.0:142535 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2862) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:47:02.038322 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] find_object_context cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 @head oi=cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'2862 client.4305.0:142535 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2862)
2014-06-12 14:47:02.038353 7f9d683e1700  5 -- op tracker -- , seq: 748279, time: 2014-06-12 14:47:02.038348, event: started, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.038360 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] execute_ctx 0x292b800
2014-06-12 14:47:02.038375 7f9d683e1700  5 -- op tracker -- , seq: 748279, time: 2014-06-12 14:47:02.038371, event: started, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.038381 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] ov 173'2862 av 173'2863 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:02.038393 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096]
2014-06-12 14:47:02.038403 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:02.038419 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op  write 3330048~516096
2014-06-12 14:47:02.038434 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:02.038445 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:47:02.038456 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:02.038465 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] finish_ctx cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 0x292b800 op modify  
2014-06-12 14:47:02.038478 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  set mtime to 2014-06-12 08:46:04.416268
2014-06-12 14:47:02.038498 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  final snapset 0=[]:[]+head in cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.038515 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  zeroing write result code 0
2014-06-12 14:47:02.038525 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] new_repop rep_tid 236391 on osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:02.038544 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] issue_repop rep_tid 236391 o cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.038568 7f9d683e1700  5 -- op tracker -- , seq: 748279, time: 2014-06-12 14:47:02.038563, event: waiting for subops from 23, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.038590 7f9d683e1700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:02.038619 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2862 (0'0,173'2862] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] append_log log((0'0,173'2862], crt=0'0) [173'2863 (173'2862) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142536 2014-06-12 08:46:04.416268]
2014-06-12 14:47:02.038642 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] add_log_entry 173'2863 (173'2862) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142536 2014-06-12 08:46:04.416268
2014-06-12 14:47:02.038661 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] append_log  adding 1 keys
2014-06-12 14:47:02.038692 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2863, trimmed: 
2014-06-12 14:47:02.038713 7f9d683e1700  5 queue_transactions existing osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.038717 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:02.038718 7f9d683e1700  5 queue_op 0xe2e6a80 seq 2608 osr(3.650 0x276dff0) 518389 bytes   (queue has 4 ops and 3105748 bytes)
2014-06-12 14:47:02.039465 7f9d7c165700 10 _generic_write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 1781760~516096 = 0
2014-06-12 14:47:02.039512 7f9d7c165700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.039524 7f9d7c165700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.039528 7f9d7c165700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.039533 7f9d7c165700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.039679 7f9d7c165700 20 set_header setting 16458 cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 parent seq 0
2014-06-12 14:47:02.039738 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:02.039748 7f9d7c165700 20 set_header setting 2114 cid meta oid 1fe1320a/pglog_3.650/0//-1 parent seq 0
2014-06-12 14:47:02.039755 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:47:02.040647 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5440 (2 active)
2014-06-12 14:47:02.040679 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:02.040696 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5440 (3 active)
2014-06-12 14:47:02.040696 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] eval_repop repgather(0x6591740 173'2863 rep_tid=236391 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.040737 7f9d683e1700 10 osd.2 173 dequeue_op 0x17c2b800 finish
2014-06-12 14:47:02.040746 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.040751 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.040775 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_applied: 236388
2014-06-12 14:47:02.040798 7f9d7795c700  5 -- op tracker -- , seq: 748275, time: 2014-06-12 14:47:02.040792, event: op_applied, op: osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.040812 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_applied on primary on version 173'2860
2014-06-12 14:47:02.041732 7f9d57543700  5 -- op tracker -- , seq: 748280, time: 2014-06-12 14:47:02.041683, event: header_read, op: osd_sub_op_reply(client.4305.0:142533 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.041744 7f9d57543700  5 -- op tracker -- , seq: 748280, time: 2014-06-12 14:47:02.041685, event: throttled, op: osd_sub_op_reply(client.4305.0:142533 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.041752 7f9d57543700  5 -- op tracker -- , seq: 748280, time: 2014-06-12 14:47:02.041697, event: all_read, op: osd_sub_op_reply(client.4305.0:142533 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.041766 7f9d57543700  5 -- op tracker -- , seq: 748280, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142533 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.041771 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142533 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:02.041777 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:02.041784 7f9d57543700 15 osd.2 173 enqueue_op 0x23e08000 prio 196 cost 0 latency 0.000101 osd_sub_op_reply(client.4305.0:142533 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:02.041840 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.041851 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x23e08000 prio 196 cost 0 latency 0.000168 osd_sub_op_reply(client.4305.0:142533 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 pg pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]
2014-06-12 14:47:02.041869 7f9d6b3e7700  5 -- op tracker -- , seq: 748280, time: 2014-06-12 14:47:02.041867, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142533 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.041878 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] handle_message: 0x23e08000
2014-06-12 14:47:02.041890 7f9d6b3e7700  5 -- op tracker -- , seq: 748280, time: 2014-06-12 14:47:02.041887, event: started, op: osd_sub_op_reply(client.4305.0:142533 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.041895 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] sub_op_modify_reply: tid 236388 op  ack_type 4 from 23
2014-06-12 14:47:02.041908 7f9d6b3e7700  5 -- op tracker -- , seq: 748275, time: 2014-06-12 14:47:02.041903, event: sub_op_commit_rec, op: osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.041915 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] repop_all_applied: repop tid 236388 all applied 
2014-06-12 14:47:02.041921 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] eval_repop repgather(0x5312240 173'2860 rep_tid=236388 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.041933 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  sending ack on repgather(0x5312240 173'2860 rep_tid=236388 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0x11670c80
2014-06-12 14:47:02.041955 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] repop_all_committed: repop tid 236388 all committed 
2014-06-12 14:47:02.041961 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean] eval_repop repgather(0x5312240 173'2860 rep_tid=236388 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.041970 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean] log_op_stats osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.020157 lat 0.020173
2014-06-12 14:47:02.041982 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean] publish_stats_to_osd 173:3719
2014-06-12 14:47:02.041990 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean]  sending commit on repgather(0x5312240 173'2860 rep_tid=236388 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0x348b7900
2014-06-12 14:47:02.042009 7f9d6b3e7700  5 -- op tracker -- , seq: 748275, time: 2014-06-12 14:47:02.042006, event: commit_sent, op: osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.042014 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean]  requeue_ops 
2014-06-12 14:47:02.042020 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  removing repgather(0x5312240 173'2860 rep_tid=236388 committed?=1 applied?=1 op=osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.042029 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]    q front is repgather(0x5312240 173'2860 rep_tid=236388 committed?=1 applied?=1 op=osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.042038 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] remove_repop repgather(0x5312240 173'2860 rep_tid=236388 committed?=1 applied?=1 op=osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.042046 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  requeue_ops 
2014-06-12 14:47:02.042058 7f9d6b3e7700  5 -- op tracker -- , seq: 748275, time: 2014-06-12 14:47:02.042056, event: done, op: osd_op(client.4305.0:142533 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.042072 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x23e08000 finish
2014-06-12 14:47:02.042076 7f9d6b3e7700  5 -- op tracker -- , seq: 748280, time: 2014-06-12 14:47:02.042074, event: done, op: osd_sub_op_reply(client.4305.0:142533 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.042083 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.042085 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.042192 7f9d7c165700 10 _do_op 0x17dd84e0 seq 2606 r = 0, finisher 0x31d36c40 0xcb612c0
2014-06-12 14:47:02.042204 7f9d7c165700 10 _finish_op 0x17dd84e0 seq 2606 osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.042214 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5440 (2 active)
2014-06-12 14:47:02.042216 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:02.042208 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] op_commit: 236389
2014-06-12 14:47:02.042238 7f9d7715b700  5 -- op tracker -- , seq: 748277, time: 2014-06-12 14:47:02.042231, event: op_commit, op: osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.042243 7f9d8096e700  5 _do_op 0x61c63c0 seq 2607 osr(3.650 0x276dff0)/0x276dff0 start
2014-06-12 14:47:02.042250 7f9d8096e700 10 _do_transaction on 0x52d1500
2014-06-12 14:47:02.042260 7f9d8096e700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.042254 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] op_applied: 236389
2014-06-12 14:47:02.042283 7f9d7795c700  5 -- op tracker -- , seq: 748277, time: 2014-06-12 14:47:02.042277, event: op_applied, op: osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.042297 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] op_applied on primary on version 173'2861
2014-06-12 14:47:02.042372 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.042399 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:02.042434 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:02.042453 7f9d8096e700 15 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.042458 7f9d8096e700 10 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1 r = 0
2014-06-12 14:47:02.042462 7f9d8096e700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.042623 7f9d8096e700 15 _write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 2297856~1032192
2014-06-12 14:47:02.042728 7f9d8096e700 10 lookup_strip_header done  cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.042739 7f9d57543700  5 -- op tracker -- , seq: 748281, time: 2014-06-12 14:47:02.042707, event: header_read, op: osd_sub_op_reply(client.4305.0:142534 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.042748 7f9d57543700  5 -- op tracker -- , seq: 748281, time: 2014-06-12 14:47:02.042708, event: throttled, op: osd_sub_op_reply(client.4305.0:142534 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.042754 7f9d57543700  5 -- op tracker -- , seq: 748281, time: 2014-06-12 14:47:02.042716, event: all_read, op: osd_sub_op_reply(client.4305.0:142534 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.042759 7f9d57543700  5 -- op tracker -- , seq: 748281, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142534 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.042761 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142534 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:02.042765 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:02.042769 7f9d57543700 15 osd.2 173 enqueue_op 0x23e09f00 prio 196 cost 0 latency 0.000061 osd_sub_op_reply(client.4305.0:142534 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:02.042787 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.042794 7f9d693e3700 10 osd.2 173 dequeue_op 0x23e09f00 prio 196 cost 0 latency 0.000087 osd_sub_op_reply(client.4305.0:142534 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 pg pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]
2014-06-12 14:47:02.042820 7f9d693e3700  5 -- op tracker -- , seq: 748281, time: 2014-06-12 14:47:02.042817, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142534 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.042870 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] handle_message: 0x23e09f00
2014-06-12 14:47:02.042881 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:47:02.042884 7f9d693e3700  5 -- op tracker -- , seq: 748281, time: 2014-06-12 14:47:02.042881, event: started, op: osd_sub_op_reply(client.4305.0:142534 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.042889 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sub_op_modify_reply: tid 236389 op  ack_type 4 from 23
2014-06-12 14:47:02.042902 7f9d693e3700  5 -- op tracker -- , seq: 748277, time: 2014-06-12 14:47:02.042898, event: sub_op_commit_rec, op: osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.042907 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] repop_all_applied: repop tid 236389 all applied 
2014-06-12 14:47:02.042914 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] eval_repop repgather(0x2dc1f500 173'2861 rep_tid=236389 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.042925 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  sending ack on repgather(0x2dc1f500 173'2861 rep_tid=236389 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0x5ffe280
2014-06-12 14:47:02.042945 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] repop_all_committed: repop tid 236389 all committed 
2014-06-12 14:47:02.042952 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2859 active+clean] eval_repop repgather(0x2dc1f500 173'2861 rep_tid=236389 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.042962 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2859 active+clean] log_op_stats osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.011904 lat 0.011921
2014-06-12 14:47:02.042971 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2859 active+clean] publish_stats_to_osd 173:3720
2014-06-12 14:47:02.042978 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2859 active+clean]  sending commit on repgather(0x2dc1f500 173'2861 rep_tid=236389 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0x28677400
2014-06-12 14:47:02.042997 7f9d693e3700  5 -- op tracker -- , seq: 748277, time: 2014-06-12 14:47:02.042994, event: commit_sent, op: osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.043002 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2859 active+clean]  requeue_ops 
2014-06-12 14:47:02.043008 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]  removing repgather(0x2dc1f500 173'2861 rep_tid=236389 committed?=1 applied?=1 op=osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.043016 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]    q front is repgather(0x2dc1f500 173'2861 rep_tid=236389 committed?=1 applied?=1 op=osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.043026 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] remove_repop repgather(0x2dc1f500 173'2861 rep_tid=236389 committed?=1 applied?=1 op=osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.043034 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]  requeue_ops 
2014-06-12 14:47:02.043047 7f9d693e3700  5 -- op tracker -- , seq: 748277, time: 2014-06-12 14:47:02.043044, event: done, op: osd_op(client.4305.0:142534 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.043057 7f9d693e3700 10 osd.2 173 dequeue_op 0x23e09f00 finish
2014-06-12 14:47:02.043062 7f9d693e3700  5 -- op tracker -- , seq: 748281, time: 2014-06-12 14:47:02.043060, event: done, op: osd_sub_op_reply(client.4305.0:142534 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.043079 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.043081 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.044950 7f9d4a5d9700  5 -- op tracker -- , seq: 748282, time: 2014-06-12 14:47:02.044232, event: header_read, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.044963 7f9d4a5d9700  5 -- op tracker -- , seq: 748282, time: 2014-06-12 14:47:02.044234, event: throttled, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.044973 7f9d4a5d9700  5 -- op tracker -- , seq: 748282, time: 2014-06-12 14:47:02.044914, event: all_read, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.044983 7f9d4a5d9700  5 -- op tracker -- , seq: 748282, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.044989 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:02.045012 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x2911800 prio 63 cost 348160 latency 0.000780 osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:02.045052 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.045059 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2911800 prio 63 cost 348160 latency 0.000827 osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]
2014-06-12 14:47:02.045083 7f9d6abe6700  5 -- op tracker -- , seq: 748282, time: 2014-06-12 14:47:02.045077, event: reached_pg, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.045092 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:02.045106 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] handle_message: 0x2911800
2014-06-12 14:47:02.045116 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] do_op osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:02.045138 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] get_object_context: found obc in cache: 0x15574840
2014-06-12 14:47:02.045149 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] get_object_context: 0x15574840 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 rwstate(write n=2 w=0) oi: cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'2863 client.4305.0:142536 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2863) ssc: 0x2ef8ec40 snapset: 0=[]:[]+head
2014-06-12 14:47:02.045165 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] find_object_context cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 @head oi=cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3(173'2863 client.4305.0:142536 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2863)
2014-06-12 14:47:02.045206 7f9d6abe6700  5 -- op tracker -- , seq: 748282, time: 2014-06-12 14:47:02.045200, event: started, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.045214 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] execute_ctx 0x55a5600
2014-06-12 14:47:02.045231 7f9d6abe6700  5 -- op tracker -- , seq: 748282, time: 2014-06-12 14:47:02.045226, event: started, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.045238 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] do_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] ov 173'2863 av 173'2864 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:02.045251 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] do_osd_op cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160]
2014-06-12 14:47:02.045278 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:02.045293 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] do_osd_op  write 3846144~348160
2014-06-12 14:47:02.045306 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:02.045316 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 snapset=0x2ef8ec78  snapc=0=[]
2014-06-12 14:47:02.045329 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] make_writeable cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:02.045349 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] finish_ctx cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 0x55a5600 op modify  
2014-06-12 14:47:02.045361 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]  set mtime to 2014-06-12 08:46:04.423769
2014-06-12 14:47:02.045389 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]  final snapset 0=[]:[]+head in cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.045405 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]  zeroing write result code 0
2014-06-12 14:47:02.045418 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] new_repop rep_tid 236392 on osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:02.045438 7f9d8096e700 10 _generic_write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 2297856~1032192 = 0
2014-06-12 14:47:02.045434 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] issue_repop rep_tid 236392 o cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.045456 7f9d6abe6700  5 -- op tracker -- , seq: 748282, time: 2014-06-12 14:47:02.045451, event: waiting for subops from 23, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.045478 7f9d6abe6700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:02.045489 7f9d8096e700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.045509 7f9d8096e700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.045514 7f9d8096e700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.045522 7f9d8096e700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.045547 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2863 (0'0,173'2863] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] append_log log((0'0,173'2863], crt=0'0) [173'2864 (173'2863) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142537 2014-06-12 08:46:04.423769]
2014-06-12 14:47:02.045573 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] add_log_entry 173'2864 (173'2863) modify   cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 by client.4305.0:142537 2014-06-12 08:46:04.423769
2014-06-12 14:47:02.045596 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] append_log  adding 1 keys
2014-06-12 14:47:02.045645 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2864, trimmed: 
2014-06-12 14:47:02.045665 7f9d6abe6700  5 queue_transactions existing osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.045669 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:02.045670 7f9d6abe6700  5 queue_op 0x139d8600 seq 2609 osr(3.650 0x276dff0) 350453 bytes   (queue has 3 ops and 1903327 bytes)
2014-06-12 14:47:02.045681 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5440 (3 active)
2014-06-12 14:47:02.045677 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] eval_repop repgather(0x12ae35c0 173'2864 rep_tid=236392 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.045695 7f9d6abe6700 10 osd.2 173 dequeue_op 0x2911800 finish
2014-06-12 14:47:02.045722 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.045726 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.045849 7f9d8096e700 20 set_header setting 16458 cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 parent seq 0
2014-06-12 14:47:02.045935 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:02.045948 7f9d8096e700 20 set_header setting 2114 cid meta oid 1fe1320a/pglog_3.650/0//-1 parent seq 0
2014-06-12 14:47:02.045955 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:47:02.050323 7f9d8096e700 10 _do_op 0x61c63c0 seq 2607 r = 0, finisher 0x6ce6a00 0x2a369a80
2014-06-12 14:47:02.050339 7f9d8096e700 10 _finish_op 0x61c63c0 seq 2607 osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.050347 7f9d7c966700  5 _do_op 0xe2e6a80 seq 2608 osr(3.650 0x276dff0)/0x276dff0 start
2014-06-12 14:47:02.050353 7f9d7c966700 10 _do_transaction on 0x1b73440
2014-06-12 14:47:02.050350 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5440 (2 active)
2014-06-12 14:47:02.050358 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:02.050343 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] op_commit: 236390
2014-06-12 14:47:02.050362 7f9d7c966700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.050375 7f9d7715b700  5 -- op tracker -- , seq: 748278, time: 2014-06-12 14:47:02.050365, event: op_commit, op: osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.050451 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.050474 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:02.050508 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:02.050524 7f9d7c966700 15 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.050529 7f9d7c966700 10 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1 r = 0
2014-06-12 14:47:02.050533 7f9d7c966700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.050666 7f9d7c966700 15 _write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 3330048~516096
2014-06-12 14:47:02.050759 7f9d7c966700 10 lookup_strip_header done  cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.050858 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:47:02.051907 7f9d7c966700 10 _generic_write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 3330048~516096 = 0
2014-06-12 14:47:02.051934 7f9d7c966700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.051943 7f9d7c966700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.051947 7f9d7c966700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.051951 7f9d7c966700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.052067 7f9d7c966700 20 set_header setting 16458 cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 parent seq 0
2014-06-12 14:47:02.052114 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:02.052124 7f9d7c966700 20 set_header setting 2114 cid meta oid 1fe1320a/pglog_3.650/0//-1 parent seq 0
2014-06-12 14:47:02.052129 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:47:02.053223 7f9d57543700  5 -- op tracker -- , seq: 748283, time: 2014-06-12 14:47:02.053183, event: header_read, op: osd_sub_op_reply(client.4305.0:142535 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.053235 7f9d57543700  5 -- op tracker -- , seq: 748283, time: 2014-06-12 14:47:02.053185, event: throttled, op: osd_sub_op_reply(client.4305.0:142535 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.053243 7f9d57543700  5 -- op tracker -- , seq: 748283, time: 2014-06-12 14:47:02.053197, event: all_read, op: osd_sub_op_reply(client.4305.0:142535 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.053235 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] op_applied: 236390
2014-06-12 14:47:02.053251 7f9d57543700  5 -- op tracker -- , seq: 748283, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142535 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.053256 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142535 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:02.053263 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:02.053262 7f9d7795c700  5 -- op tracker -- , seq: 748278, time: 2014-06-12 14:47:02.053254, event: op_applied, op: osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.053272 7f9d57543700 15 osd.2 173 enqueue_op 0x23e09e00 prio 196 cost 0 latency 0.000089 osd_sub_op_reply(client.4305.0:142535 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:02.053276 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] op_applied on primary on version 173'2862
2014-06-12 14:47:02.053313 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.053320 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x23e09e00 prio 196 cost 0 latency 0.000137 osd_sub_op_reply(client.4305.0:142535 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 pg pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]
2014-06-12 14:47:02.053339 7f9d6a3e5700  5 -- op tracker -- , seq: 748283, time: 2014-06-12 14:47:02.053336, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142535 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.053350 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] handle_message: 0x23e09e00
2014-06-12 14:47:02.053366 7f9d6a3e5700  5 -- op tracker -- , seq: 748283, time: 2014-06-12 14:47:02.053362, event: started, op: osd_sub_op_reply(client.4305.0:142535 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.053374 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] sub_op_modify_reply: tid 236390 op  ack_type 4 from 23
2014-06-12 14:47:02.053393 7f9d6a3e5700  5 -- op tracker -- , seq: 748278, time: 2014-06-12 14:47:02.053387, event: sub_op_commit_rec, op: osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.053401 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] repop_all_applied: repop tid 236390 all applied 
2014-06-12 14:47:02.053414 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] eval_repop repgather(0x1c378b40 173'2862 rep_tid=236390 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.053431 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]  sending ack on repgather(0x1c378b40 173'2862 rep_tid=236390 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0x33771180
2014-06-12 14:47:02.053464 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] repop_all_committed: repop tid 236390 all committed 
2014-06-12 14:47:02.053475 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2860 active+clean] eval_repop repgather(0x1c378b40 173'2862 rep_tid=236390 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.053493 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2860 active+clean] log_op_stats osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.020011 lat 0.020041
2014-06-12 14:47:02.053511 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2860 active+clean] publish_stats_to_osd 173:3721
2014-06-12 14:47:02.053525 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2860 active+clean]  sending commit on repgather(0x1c378b40 173'2862 rep_tid=236390 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0x3228c500
2014-06-12 14:47:02.053557 7f9d6a3e5700  5 -- op tracker -- , seq: 748278, time: 2014-06-12 14:47:02.053552, event: commit_sent, op: osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.053566 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2860 active+clean]  requeue_ops 
2014-06-12 14:47:02.053576 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]  removing repgather(0x1c378b40 173'2862 rep_tid=236390 committed?=1 applied?=1 op=osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.053594 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]    q front is repgather(0x1c378b40 173'2862 rep_tid=236390 committed?=1 applied?=1 op=osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.053609 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] remove_repop repgather(0x1c378b40 173'2862 rep_tid=236390 committed?=1 applied?=1 op=osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.053639 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]  requeue_ops 
2014-06-12 14:47:02.053657 7f9d6a3e5700  5 -- op tracker -- , seq: 748278, time: 2014-06-12 14:47:02.053653, event: done, op: osd_op(client.4305.0:142535 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 2297856~1032192] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.053677 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x23e09e00 finish
2014-06-12 14:47:02.053686 7f9d6a3e5700  5 -- op tracker -- , seq: 748283, time: 2014-06-12 14:47:02.053682, event: done, op: osd_sub_op_reply(client.4305.0:142535 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.053696 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.053699 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.053929 7f9d7c966700 10 _do_op 0xe2e6a80 seq 2608 r = 0, finisher 0x65e0d60 0x3393e3c0
2014-06-12 14:47:02.053938 7f9d7c966700 10 _finish_op 0xe2e6a80 seq 2608 osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.053944 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5440 (1 active)
2014-06-12 14:47:02.053951 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:02.053949 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] op_applied: 236391
2014-06-12 14:47:02.053968 7f9d7795c700  5 -- op tracker -- , seq: 748279, time: 2014-06-12 14:47:02.053963, event: op_applied, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.053974 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] op_applied on primary on version 173'2863
2014-06-12 14:47:02.053981 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] op_commit: 236391
2014-06-12 14:47:02.054018 7f9d7715b700  5 -- op tracker -- , seq: 748279, time: 2014-06-12 14:47:02.054011, event: op_commit, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.054032 7f9d81970700  5 _do_op 0x139d8600 seq 2609 osr(3.650 0x276dff0)/0x276dff0 start
2014-06-12 14:47:02.054038 7f9d81970700 10 _do_transaction on 0x2681bc0
2014-06-12 14:47:02.054045 7f9d81970700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.054193 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.054229 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:02.054303 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:02.054353 7f9d81970700 15 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.054362 7f9d81970700 10 _omap_rmkeys meta/1fe1320a/pglog_3.650/0//-1 r = 0
2014-06-12 14:47:02.054369 7f9d81970700 15 _omap_setkeys meta/1fe1320a/pglog_3.650/0//-1
2014-06-12 14:47:02.054388 7f9d81970700 15 _write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 3846144~348160
2014-06-12 14:47:02.054723 7f9d81970700 10 lookup_strip_header done  cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.054893 7f9d81970700 10 file_to_extents done 
2014-06-12 14:47:02.055752 7f9d81970700 10 _generic_write 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 3846144~348160 = 0
2014-06-12 14:47:02.055783 7f9d81970700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.055799 7f9d81970700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.055811 7f9d81970700 15 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3
2014-06-12 14:47:02.055821 7f9d81970700 10 _setattrs 3.650_head/cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 = 0
2014-06-12 14:47:02.055944 7f9d81970700 20 set_header setting 16458 cid 3.650_head oid cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 parent seq 0
2014-06-12 14:47:02.055988 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:02.056004 7f9d81970700 20 set_header setting 2114 cid meta oid 1fe1320a/pglog_3.650/0//-1 parent seq 0
2014-06-12 14:47:02.056014 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:47:02.058029 7f9d81970700 10 _do_op 0x139d8600 seq 2609 r = 0, finisher 0x90ee9a0 0x31aeda40
2014-06-12 14:47:02.058070 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] op_commit: 236392
2014-06-12 14:47:02.058119 7f9d7715b700  5 -- op tracker -- , seq: 748282, time: 2014-06-12 14:47:02.058110, event: op_commit, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.058153 7f9d81970700 10 _finish_op 0x139d8600 seq 2609 osr(3.650 0x276dff0)/0x276dff0
2014-06-12 14:47:02.058184 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] op_applied: 236392
2014-06-12 14:47:02.058223 7f9d7795c700  5 -- op tracker -- , seq: 748282, time: 2014-06-12 14:47:02.058217, event: op_applied, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.058231 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] op_applied on primary on version 173'2864
2014-06-12 14:47:02.058290 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5440 (0 active)
2014-06-12 14:47:02.058300 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:02.059566 7f9d57543700  5 -- op tracker -- , seq: 748284, time: 2014-06-12 14:47:02.059518, event: header_read, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.059576 7f9d57543700  5 -- op tracker -- , seq: 748284, time: 2014-06-12 14:47:02.059520, event: throttled, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.059583 7f9d57543700  5 -- op tracker -- , seq: 748284, time: 2014-06-12 14:47:02.059540, event: all_read, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.059590 7f9d57543700  5 -- op tracker -- , seq: 748284, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.059594 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:02.059599 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:02.059608 7f9d57543700 15 osd.2 173 enqueue_op 0x55c3b00 prio 196 cost 0 latency 0.000090 osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0) v2
2014-06-12 14:47:02.059636 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.059646 7f9d66bde700 10 osd.2 173 dequeue_op 0x55c3b00 prio 196 cost 0 latency 0.000128 osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0) v2 pg pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]
2014-06-12 14:47:02.059668 7f9d66bde700  5 -- op tracker -- , seq: 748284, time: 2014-06-12 14:47:02.059665, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.059683 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] handle_message: 0x55c3b00
2014-06-12 14:47:02.059699 7f9d66bde700  5 -- op tracker -- , seq: 748284, time: 2014-06-12 14:47:02.059696, event: started, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.059705 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] sub_op_modify_reply: tid 236391 op  ack_type 1 from 23
2014-06-12 14:47:02.059724 7f9d66bde700  5 -- op tracker -- , seq: 748279, time: 2014-06-12 14:47:02.059717, event: sub_op_applied_rec, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.059755 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] repop_all_applied: repop tid 236391 all applied 
2014-06-12 14:47:02.059769 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] eval_repop repgather(0x6591740 173'2863 rep_tid=236391 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.059787 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]  sending ack on repgather(0x6591740 173'2863 rep_tid=236391 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0x33770000
2014-06-12 14:47:02.059822 7f9d66bde700 10 osd.2 173 dequeue_op 0x55c3b00 finish
2014-06-12 14:47:02.059830 7f9d66bde700  5 -- op tracker -- , seq: 748284, time: 2014-06-12 14:47:02.059827, event: done, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.059845 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.059850 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.063182 7f9d57543700  5 -- op tracker -- , seq: 748285, time: 2014-06-12 14:47:02.063115, event: header_read, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.063198 7f9d57543700  5 -- op tracker -- , seq: 748285, time: 2014-06-12 14:47:02.063119, event: throttled, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.063205 7f9d57543700  5 -- op tracker -- , seq: 748285, time: 2014-06-12 14:47:02.063136, event: all_read, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.063212 7f9d57543700  5 -- op tracker -- , seq: 748285, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.063217 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:02.063223 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:02.063231 7f9d57543700 15 osd.2 173 enqueue_op 0x55c3f00 prio 196 cost 0 latency 0.000116 osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0) v2
2014-06-12 14:47:02.063265 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.063272 7f9d69be4700 10 osd.2 173 dequeue_op 0x55c3f00 prio 196 cost 0 latency 0.000157 osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0) v2 pg pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]
2014-06-12 14:47:02.063293 7f9d69be4700  5 -- op tracker -- , seq: 748285, time: 2014-06-12 14:47:02.063290, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.063300 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] handle_message: 0x55c3f00
2014-06-12 14:47:02.063315 7f9d69be4700  5 -- op tracker -- , seq: 748285, time: 2014-06-12 14:47:02.063312, event: started, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.063321 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] sub_op_modify_reply: tid 236392 op  ack_type 1 from 23
2014-06-12 14:47:02.063340 7f9d69be4700  5 -- op tracker -- , seq: 748282, time: 2014-06-12 14:47:02.063332, event: sub_op_applied_rec, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.063348 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] repop_all_applied: repop tid 236392 all applied 
2014-06-12 14:47:02.063358 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] eval_repop repgather(0x12ae35c0 173'2864 rep_tid=236392 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.063373 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]  sending ack on repgather(0x12ae35c0 173'2864 rep_tid=236392 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0x33271900
2014-06-12 14:47:02.063402 7f9d69be4700 10 osd.2 173 dequeue_op 0x55c3f00 finish
2014-06-12 14:47:02.063409 7f9d69be4700  5 -- op tracker -- , seq: 748285, time: 2014-06-12 14:47:02.063406, event: done, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ack, result = 0)
2014-06-12 14:47:02.063423 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.063426 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.065524 7f9d57543700  5 -- op tracker -- , seq: 748286, time: 2014-06-12 14:47:02.065467, event: header_read, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.065534 7f9d57543700  5 -- op tracker -- , seq: 748286, time: 2014-06-12 14:47:02.065469, event: throttled, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.065540 7f9d57543700  5 -- op tracker -- , seq: 748286, time: 2014-06-12 14:47:02.065491, event: all_read, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.065547 7f9d57543700  5 -- op tracker -- , seq: 748286, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.065552 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:02.065557 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:02.065564 7f9d57543700 15 osd.2 173 enqueue_op 0x55c3d00 prio 196 cost 0 latency 0.000097 osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:02.065592 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.065606 7f9d57543700  5 -- op tracker -- , seq: 748287, time: 2014-06-12 14:47:02.065584, event: header_read, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.065621 7f9d57543700  5 -- op tracker -- , seq: 748287, time: 2014-06-12 14:47:02.065584, event: throttled, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.065628 7f9d57543700  5 -- op tracker -- , seq: 748287, time: 2014-06-12 14:47:02.065591, event: all_read, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.065635 7f9d57543700  5 -- op tracker -- , seq: 748287, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.065638 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:02.065602 7f9d64bda700 10 osd.2 173 dequeue_op 0x55c3d00 prio 196 cost 0 latency 0.000134 osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 pg pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]
2014-06-12 14:47:02.065642 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:02.065647 7f9d57543700 15 osd.2 173 enqueue_op 0x17c2b900 prio 196 cost 0 latency 0.000063 osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:02.065650 7f9d64bda700  5 -- op tracker -- , seq: 748286, time: 2014-06-12 14:47:02.065646, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.065669 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:02.065660 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] handle_message: 0x55c3d00
2014-06-12 14:47:02.065678 7f9d64bda700  5 -- op tracker -- , seq: 748286, time: 2014-06-12 14:47:02.065675, event: started, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.065684 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] sub_op_modify_reply: tid 236391 op  ack_type 4 from 23
2014-06-12 14:47:02.065705 7f9d64bda700  5 -- op tracker -- , seq: 748279, time: 2014-06-12 14:47:02.065698, event: sub_op_commit_rec, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.065713 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] repop_all_committed: repop tid 236391 all committed 
2014-06-12 14:47:02.065724 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2861 active+clean] eval_repop repgather(0x6591740 173'2863 rep_tid=236391 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.065741 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2861 active+clean] log_op_stats osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.022812 lat 0.028732
2014-06-12 14:47:02.065759 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2861 active+clean] publish_stats_to_osd 173:3722
2014-06-12 14:47:02.065772 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2861 active+clean]  sending commit on repgather(0x6591740 173'2863 rep_tid=236391 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0x3228db80
2014-06-12 14:47:02.065808 7f9d64bda700  5 -- op tracker -- , seq: 748279, time: 2014-06-12 14:47:02.065802, event: commit_sent, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.065822 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2861 active+clean]  requeue_ops 
2014-06-12 14:47:02.065833 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean]  removing repgather(0x6591740 173'2863 rep_tid=236391 committed?=1 applied?=1 op=osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.065848 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean]    q front is repgather(0x6591740 173'2863 rep_tid=236391 committed?=1 applied?=1 op=osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.065862 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean] remove_repop repgather(0x6591740 173'2863 rep_tid=236391 committed?=1 applied?=1 op=osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.065876 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean]  requeue_ops 
2014-06-12 14:47:02.065896 7f9d64bda700  5 -- op tracker -- , seq: 748279, time: 2014-06-12 14:47:02.065891, event: done, op: osd_op(client.4305.0:142536 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.065929 7f9d64bda700 10 osd.2 173 dequeue_op 0x55c3d00 finish
2014-06-12 14:47:02.065938 7f9d653db700 10 osd.2 173 dequeue_op 0x17c2b900 prio 196 cost 0 latency 0.000354 osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0) v2 pg pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean]
2014-06-12 14:47:02.065939 7f9d64bda700  5 -- op tracker -- , seq: 748286, time: 2014-06-12 14:47:02.065935, event: done, op: osd_sub_op_reply(client.4305.0:142536 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.065990 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:02.065994 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.065993 7f9d653db700  5 -- op tracker -- , seq: 748287, time: 2014-06-12 14:47:02.065953, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.066000 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean] handle_message: 0x17c2b900
2014-06-12 14:47:02.066013 7f9d653db700  5 -- op tracker -- , seq: 748287, time: 2014-06-12 14:47:02.066011, event: started, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.066018 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean] sub_op_modify_reply: tid 236392 op  ack_type 4 from 23
2014-06-12 14:47:02.066034 7f9d653db700  5 -- op tracker -- , seq: 748282, time: 2014-06-12 14:47:02.066028, event: sub_op_commit_rec, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.066039 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean] repop_all_committed: repop tid 236392 all committed 
2014-06-12 14:47:02.066048 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2862 active+clean] eval_repop repgather(0x12ae35c0 173'2864 rep_tid=236392 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:02.066061 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2862 active+clean] log_op_stats osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4 inb 348439 outb 0 rlat 0.019168 lat 0.021829
2014-06-12 14:47:02.066074 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2862 active+clean] publish_stats_to_osd 173:3723
2014-06-12 14:47:02.066083 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2862 active+clean]  sending commit on repgather(0x12ae35c0 173'2864 rep_tid=236392 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4) 0x1d54e500
2014-06-12 14:47:02.066111 7f9d653db700  5 -- op tracker -- , seq: 748282, time: 2014-06-12 14:47:02.066106, event: commit_sent, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.066118 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2862 active+clean]  requeue_ops 
2014-06-12 14:47:02.066126 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean]  removing repgather(0x12ae35c0 173'2864 rep_tid=236392 committed?=1 applied?=1 op=osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.066138 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean]    q front is repgather(0x12ae35c0 173'2864 rep_tid=236392 committed?=1 applied?=1 op=osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.066149 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean] remove_repop repgather(0x12ae35c0 173'2864 rep_tid=236392 committed?=1 applied?=1 op=osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:02.066161 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean]  requeue_ops 
2014-06-12 14:47:02.066183 7f9d653db700  5 -- op tracker -- , seq: 748282, time: 2014-06-12 14:47:02.066178, event: done, op: osd_op(client.4305.0:142537 rbd_data.10a86b8b4567.0000000000000eea [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.cbbace50 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:02.066197 7f9d653db700 10 osd.2 173 dequeue_op 0x17c2b900 finish
2014-06-12 14:47:02.066203 7f9d653db700  5 -- op tracker -- , seq: 748287, time: 2014-06-12 14:47:02.066200, event: done, op: osd_sub_op_reply(client.4305.0:142537 3.650 cbbace50/rbd_data.10a86b8b4567.0000000000000eea/head//3 [] ondisk, result = 0)
2014-06-12 14:47:02.066212 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.066214 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.155439 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:02.203798 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:02.204007 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:02.272889 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:02.272943 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:02.369950 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:02.460170 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:02.460373 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:02.481158 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:02.481178 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:47:02.481181 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:47:02.481184 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:47:02.481188 7f9d84175700 10 osd.2 173 send_pg_stats - 4 pgs updated
2014-06-12 14:47:02.481228 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:02.481232 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:02.481234 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:02.481237 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 28; oldest is 28.815368 seconds old
2014-06-12 14:47:02.481236 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:02.513552 7f9d58d5b700  5 -- op tracker -- , seq: 748288, time: 2014-06-12 14:47:02.510505, event: header_read, op: osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.513571 7f9d58d5b700  5 -- op tracker -- , seq: 748288, time: 2014-06-12 14:47:02.510508, event: throttled, op: osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.513579 7f9d58d5b700  5 -- op tracker -- , seq: 748288, time: 2014-06-12 14:47:02.513498, event: all_read, op: osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.513587 7f9d58d5b700  5 -- op tracker -- , seq: 748288, time: 0.000000, event: dispatched, op: osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.513592 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:02.513599 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:02.513607 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34017e00 prio 127 cost 1327911 latency 0.003102 osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:02.513759 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.513772 7f9d623d5700 10 osd.2 173 dequeue_op 0x34017e00 prio 127 cost 1327911 latency 0.003266 osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.5c5( v 173'2856 (0'0,173'2856] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2855 active]
2014-06-12 14:47:02.513798 7f9d623d5700  5 -- op tracker -- , seq: 748288, time: 2014-06-12 14:47:02.513794, event: reached_pg, op: osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.513808 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2856 (0'0,173'2856] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2855 active] handle_message: 0x34017e00
2014-06-12 14:47:02.513821 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2856 (0'0,173'2856] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2855 active] sub_op_modify trans f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 v 173'2857 (transaction) 174
2014-06-12 14:47:02.513838 7f9d623d5700  5 -- op tracker -- , seq: 748288, time: 2014-06-12 14:47:02.513834, event: started, op: osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.513856 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2856 (0'0,173'2856] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2855 active] append_log log((0'0,173'2856], crt=0'0) [173'2857 (173'757) modify   f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 by client.4309.0:141827 2014-06-12 08:46:04.883281]
2014-06-12 14:47:02.513883 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2857 (0'0,173'2857] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2855 active] add_log_entry 173'2857 (173'757) modify   f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 by client.4309.0:141827 2014-06-12 08:46:04.883281
2014-06-12 14:47:02.513905 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2857 (0'0,173'2857] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2855 active] append_log  adding 1 keys
2014-06-12 14:47:02.513940 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2857, trimmed: 
2014-06-12 14:47:02.513965 7f9d623d5700  5 -- op tracker -- , seq: 748288, time: 2014-06-12 14:47:02.513960, event: started, op: osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.513975 7f9d623d5700  5 queue_transactions existing osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:47:02.513980 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:02.513982 7f9d623d5700  5 queue_op 0x1b90ede0 seq 2611 osr(3.5c5 0x276dde0) 1329397 bytes   (queue has 1 ops and 1329397 bytes)
2014-06-12 14:47:02.513994 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439d40 (1 active)
2014-06-12 14:47:02.513990 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2857 (0'0,173'2857] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2855 active] do_sub_op osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:02.514005 7f9d623d5700 10 osd.2 173 dequeue_op 0x34017e00 finish
2014-06-12 14:47:02.514006 7f9d7a962700  5 _do_op 0x1b90ede0 seq 2611 osr(3.5c5 0x276dde0)/0x276dde0 start
2014-06-12 14:47:02.514009 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.514018 7f9d7a962700 10 _do_transaction on 0x601b5f0
2014-06-12 14:47:02.514019 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.514024 7f9d7a962700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:47:02.514228 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:47:02.514255 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:02.514305 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:02.514327 7f9d7a962700 15 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:47:02.514335 7f9d7a962700 10 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1 r = 0
2014-06-12 14:47:02.514341 7f9d7a962700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:47:02.514360 7f9d7a962700 15 _write 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 0~1327104
2014-06-12 14:47:02.514644 7f9d7a962700 10 lookup_strip_header done  cid 3.5c5_head oid f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3
2014-06-12 14:47:02.514754 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:47:02.519819 7f9d7a962700 10 _generic_write 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 0~1327104 = 0
2014-06-12 14:47:02.519943 7f9d7a962700 15 _setattrs 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3
2014-06-12 14:47:02.519962 7f9d7a962700 10 _setattrs 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 = 0
2014-06-12 14:47:02.519971 7f9d7a962700 15 _setattrs 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3
2014-06-12 14:47:02.519980 7f9d7a962700 10 _setattrs 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 = 0
2014-06-12 14:47:02.520322 7f9d7a962700 20 set_header setting 17176 cid 3.5c5_head oid f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 parent seq 0
2014-06-12 14:47:02.520459 7f9d7a962700 20 set_header setting 2264 cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1 parent seq 0
2014-06-12 14:47:02.520476 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:02.520485 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:47:02.527050 7f9d7a962700 10 _do_op 0x1b90ede0 seq 2611 r = 0, finisher 0x682af20 0
2014-06-12 14:47:02.527080 7f9d7a962700 10 _finish_op 0x1b90ede0 seq 2611 osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:47:02.527088 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439d40 (0 active)
2014-06-12 14:47:02.527093 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:02.527109 7f9d7715b700  5 -- op tracker -- , seq: 748288, time: 2014-06-12 14:47:02.527089, event: commit_sent, op: osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.527118 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2857 (0'0,173'2857] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2855 active] sub_op_modify_commit on op osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:47:02.527149 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:02.527215 7f9d7795c700  5 -- op tracker -- , seq: 748288, time: 2014-06-12 14:47:02.527209, event: sub_op_applied, op: osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.527227 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2857 (0'0,173'2857] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2856 active] sub_op_modify_applied on 0x601b500 op osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:02.527241 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2857 (0'0,173'2857] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2856 active] op_applied on primary on version 173'2857
2014-06-12 14:47:02.527249 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2857 (0'0,173'2857] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2856 active] op_applied on replica on version 173'2857
2014-06-12 14:47:02.527267 7f9d7795c700  5 -- op tracker -- , seq: 748288, time: 2014-06-12 14:47:02.527263, event: done, op: osd_sub_op(client.4309.0:141827 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2857 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.544706 7f9d58d5b700  5 -- op tracker -- , seq: 748289, time: 2014-06-12 14:47:02.540444, event: header_read, op: osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.544723 7f9d58d5b700  5 -- op tracker -- , seq: 748289, time: 2014-06-12 14:47:02.540446, event: throttled, op: osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.544743 7f9d58d5b700  5 -- op tracker -- , seq: 748289, time: 2014-06-12 14:47:02.544656, event: all_read, op: osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.544749 7f9d58d5b700  5 -- op tracker -- , seq: 748289, time: 0.000000, event: dispatched, op: osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.544759 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:02.544765 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:02.544773 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34017500 prio 127 cost 2065191 latency 0.004329 osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:02.544805 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.544814 7f9d68be2700 10 osd.2 173 dequeue_op 0x34017500 prio 127 cost 2065191 latency 0.004370 osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.5c5( v 173'2857 (0'0,173'2857] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2856 active]
2014-06-12 14:47:02.544844 7f9d68be2700  5 -- op tracker -- , seq: 748289, time: 2014-06-12 14:47:02.544833, event: reached_pg, op: osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.544852 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2857 (0'0,173'2857] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2856 active] handle_message: 0x34017500
2014-06-12 14:47:02.544861 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2857 (0'0,173'2857] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2856 active] sub_op_modify trans f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 v 173'2858 (transaction) 174
2014-06-12 14:47:02.544879 7f9d68be2700  5 -- op tracker -- , seq: 748289, time: 2014-06-12 14:47:02.544869, event: started, op: osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.544892 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2857 (0'0,173'2857] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2856 active] append_log log((0'0,173'2857], crt=0'0) [173'2858 (173'2857) modify   f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 by client.4309.0:141828 2014-06-12 08:46:04.910864]
2014-06-12 14:47:02.544912 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2858 (0'0,173'2858] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2856 active] add_log_entry 173'2858 (173'2857) modify   f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 by client.4309.0:141828 2014-06-12 08:46:04.910864
2014-06-12 14:47:02.544931 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2858 (0'0,173'2858] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2856 active] append_log  adding 1 keys
2014-06-12 14:47:02.544955 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2858, trimmed: 
2014-06-12 14:47:02.544972 7f9d68be2700  5 -- op tracker -- , seq: 748289, time: 2014-06-12 14:47:02.544969, event: started, op: osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.544981 7f9d68be2700  5 queue_transactions existing osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:47:02.544984 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:02.544985 7f9d68be2700  5 queue_op 0x6d15e60 seq 2612 osr(3.5c5 0x276dde0) 2066677 bytes   (queue has 1 ops and 2066677 bytes)
2014-06-12 14:47:02.544995 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439d40 (1 active)
2014-06-12 14:47:02.544990 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2858 (0'0,173'2858] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2856 active] do_sub_op osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:02.545000 7f9d68be2700 10 osd.2 173 dequeue_op 0x34017500 finish
2014-06-12 14:47:02.545002 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.545001 7f9d82972700  5 _do_op 0x6d15e60 seq 2612 osr(3.5c5 0x276dde0)/0x276dde0 start
2014-06-12 14:47:02.545004 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.545006 7f9d82972700 10 _do_transaction on 0x2bc9b0b0
2014-06-12 14:47:02.545011 7f9d82972700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:47:02.545127 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:47:02.545148 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:02.545194 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:02.545215 7f9d82972700 15 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:47:02.545221 7f9d82972700 10 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1 r = 0
2014-06-12 14:47:02.545225 7f9d82972700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:47:02.545374 7f9d82972700 15 _write 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 1327104~2064384
2014-06-12 14:47:02.545484 7f9d82972700 10 lookup_strip_header done  cid 3.5c5_head oid f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3
2014-06-12 14:47:02.545607 7f9d82972700 10 file_to_extents done 
2014-06-12 14:47:02.551291 7f9d82972700 10 _generic_write 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 1327104~2064384 = 0
2014-06-12 14:47:02.551399 7f9d82972700 15 _setattrs 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3
2014-06-12 14:47:02.551421 7f9d82972700 10 _setattrs 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 = 0
2014-06-12 14:47:02.551427 7f9d82972700 15 _setattrs 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3
2014-06-12 14:47:02.551432 7f9d82972700 10 _setattrs 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 = 0
2014-06-12 14:47:02.551735 7f9d82972700 20 set_header setting 17176 cid 3.5c5_head oid f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 parent seq 0
2014-06-12 14:47:02.551857 7f9d82972700 20 set_header setting 2264 cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1 parent seq 0
2014-06-12 14:47:02.551868 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:02.551873 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:47:02.559362 7f9d82972700 10 _do_op 0x6d15e60 seq 2612 r = 0, finisher 0x6ab81c0 0
2014-06-12 14:47:02.559385 7f9d82972700 10 _finish_op 0x6d15e60 seq 2612 osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:47:02.559392 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439d40 (0 active)
2014-06-12 14:47:02.559399 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:02.559405 7f9d7715b700  5 -- op tracker -- , seq: 748289, time: 2014-06-12 14:47:02.559393, event: commit_sent, op: osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.559416 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2858 (0'0,173'2858] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2856 active] sub_op_modify_commit on op osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:47:02.559441 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:02.559476 7f9d7795c700  5 -- op tracker -- , seq: 748289, time: 2014-06-12 14:47:02.559468, event: sub_op_applied, op: osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.559485 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2858 (0'0,173'2858] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2857 active] sub_op_modify_applied on 0x2bc9afc0 op osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:02.559512 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2858 (0'0,173'2858] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2857 active] op_applied on primary on version 173'2858
2014-06-12 14:47:02.559522 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2858 (0'0,173'2858] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2857 active] op_applied on replica on version 173'2858
2014-06-12 14:47:02.559542 7f9d7795c700  5 -- op tracker -- , seq: 748289, time: 2014-06-12 14:47:02.559537, event: done, op: osd_sub_op(client.4309.0:141828 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2858 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.568310 7f9d58d5b700  5 -- op tracker -- , seq: 748290, time: 2014-06-12 14:47:02.566615, event: header_read, op: osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.568322 7f9d58d5b700  5 -- op tracker -- , seq: 748290, time: 2014-06-12 14:47:02.566617, event: throttled, op: osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.568338 7f9d58d5b700  5 -- op tracker -- , seq: 748290, time: 2014-06-12 14:47:02.568271, event: all_read, op: osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.568343 7f9d58d5b700  5 -- op tracker -- , seq: 748290, time: 0.000000, event: dispatched, op: osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.568348 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:02.568352 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:02.568359 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016a00 prio 127 cost 803623 latency 0.001744 osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:02.568388 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:02.568398 7f9d643d9700 10 osd.2 173 dequeue_op 0x34016a00 prio 127 cost 803623 latency 0.001783 osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.5c5( v 173'2858 (0'0,173'2858] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2857 active]
2014-06-12 14:47:02.568424 7f9d643d9700  5 -- op tracker -- , seq: 748290, time: 2014-06-12 14:47:02.568420, event: reached_pg, op: osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.568437 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2858 (0'0,173'2858] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2857 active] handle_message: 0x34016a00
2014-06-12 14:47:02.568454 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2858 (0'0,173'2858] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2857 active] sub_op_modify trans f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 v 173'2859 (transaction) 174
2014-06-12 14:47:02.568475 7f9d643d9700  5 -- op tracker -- , seq: 748290, time: 2014-06-12 14:47:02.568471, event: started, op: osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.568493 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2858 (0'0,173'2858] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2857 active] append_log log((0'0,173'2858], crt=0'0) [173'2859 (173'2858) modify   f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 by client.4309.0:141829 2014-06-12 08:46:04.942367]
2014-06-12 14:47:02.568520 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2859 (0'0,173'2859] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2857 active] add_log_entry 173'2859 (173'2858) modify   f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 by client.4309.0:141829 2014-06-12 08:46:04.942367
2014-06-12 14:47:02.568545 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2859 (0'0,173'2859] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2857 active] append_log  adding 1 keys
2014-06-12 14:47:02.568583 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2859, trimmed: 
2014-06-12 14:47:02.568611 7f9d643d9700  5 -- op tracker -- , seq: 748290, time: 2014-06-12 14:47:02.568606, event: started, op: osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.568624 7f9d643d9700  5 queue_transactions existing osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:47:02.568629 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:02.568630 7f9d643d9700  5 queue_op 0x83eff20 seq 2613 osr(3.5c5 0x276dde0) 805109 bytes   (queue has 1 ops and 805109 bytes)
2014-06-12 14:47:02.568637 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2859 (0'0,173'2859] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2857 active] do_sub_op osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:02.568653 7f9d643d9700 10 osd.2 173 dequeue_op 0x34016a00 finish
2014-06-12 14:47:02.568657 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:02.568659 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:02.568665 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439d40 (1 active)
2014-06-12 14:47:02.568673 7f9d7895e700  5 _do_op 0x83eff20 seq 2613 osr(3.5c5 0x276dde0)/0x276dde0 start
2014-06-12 14:47:02.568677 7f9d7895e700 10 _do_transaction on 0x2bc9ab70
2014-06-12 14:47:02.568682 7f9d7895e700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:47:02.568778 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:47:02.568802 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:02.568836 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:02.568852 7f9d7895e700 15 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:47:02.568857 7f9d7895e700 10 _omap_rmkeys meta/1fe7ccb5/pglog_3.5c5/0//-1 r = 0
2014-06-12 14:47:02.568861 7f9d7895e700 15 _omap_setkeys meta/1fe7ccb5/pglog_3.5c5/0//-1
2014-06-12 14:47:02.568869 7f9d7895e700 15 _write 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 3391488~802816
2014-06-12 14:47:02.568962 7f9d7895e700 10 lookup_strip_header done  cid 3.5c5_head oid f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3
2014-06-12 14:47:02.569164 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:47:02.571051 7f9d7895e700 10 _generic_write 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 3391488~802816 = 0
2014-06-12 14:47:02.571113 7f9d7895e700 15 _setattrs 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3
2014-06-12 14:47:02.571124 7f9d7895e700 10 _setattrs 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 = 0
2014-06-12 14:47:02.571128 7f9d7895e700 15 _setattrs 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3
2014-06-12 14:47:02.571133 7f9d7895e700 10 _setattrs 3.5c5_head/f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 = 0
2014-06-12 14:47:02.571300 7f9d7895e700 20 set_header setting 17176 cid 3.5c5_head oid f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 parent seq 0
2014-06-12 14:47:02.571369 7f9d7895e700 20 set_header setting 2264 cid meta oid 1fe7ccb5/pglog_3.5c5/0//-1 parent seq 0
2014-06-12 14:47:02.571380 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:02.571386 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:47:02.575962 7f9d7895e700 10 _do_op 0x83eff20 seq 2613 r = 0, finisher 0x692f020 0
2014-06-12 14:47:02.576016 7f9d7715b700  5 -- op tracker -- , seq: 748290, time: 2014-06-12 14:47:02.576004, event: commit_sent, op: osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.576031 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2859 (0'0,173'2859] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2857 active] sub_op_modify_commit on op osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.22
2014-06-12 14:47:02.576061 7f9d7715b700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:02.576090 7f9d7895e700 10 _finish_op 0x83eff20 seq 2613 osr(3.5c5 0x276dde0)/0x276dde0
2014-06-12 14:47:02.576099 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439d40 (0 active)
2014-06-12 14:47:02.576107 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:02.576118 7f9d7795c700  5 -- op tracker -- , seq: 748290, time: 2014-06-12 14:47:02.576111, event: sub_op_applied, op: osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.576135 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2859 (0'0,173'2859] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2858 active] sub_op_modify_applied on 0x2bc9aa80 op osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:02.576151 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2859 (0'0,173'2859] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2858 active] op_applied on primary on version 173'2859
2014-06-12 14:47:02.576161 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.5c5( v 173'2859 (0'0,173'2859] local-les=142 n=295 ec=23 les/c 142/142 141/141/141) [22,2] r=1 lpr=141 pi=23-140/12 luod=0'0 crt=0'0 lcod 173'2858 active] op_applied on replica on version 173'2859
2014-06-12 14:47:02.576186 7f9d7795c700  5 -- op tracker -- , seq: 748290, time: 2014-06-12 14:47:02.576182, event: done, op: osd_sub_op(client.4309.0:141829 3.5c5 f8f67dc5/rbd_data.10ac6b8b4567.0000000000000f8b/head//3 [] v 173'2859 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:02.625214 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:02.625246 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:02.743522 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:02.743525 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:02.743526 7f9d74bfa700 20 osd.2 173 _dispatch 0x2a29a1c0 pg_stats_ack(4 pgs tid 4858) v1
2014-06-12 14:47:02.743529 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:47:02.743536 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:02.743537 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:03.016403 7f9d4397a700  5 -- op tracker -- , seq: 748291, time: 2014-06-12 14:47:03.015473, event: header_read, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.016424 7f9d4397a700  5 -- op tracker -- , seq: 748291, time: 2014-06-12 14:47:03.015479, event: throttled, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.016432 7f9d4397a700  5 -- op tracker -- , seq: 748291, time: 2014-06-12 14:47:03.016356, event: all_read, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.016440 7f9d4397a700  5 -- op tracker -- , seq: 748291, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.016448 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:47:03.016484 7f9d4397a700 15 osd.2 173 enqueue_op 0x22b3a000 prio 63 cost 356352 latency 0.001011 osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:03.016562 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:03.016586 7f9d61bd4700 10 osd.2 173 dequeue_op 0x22b3a000 prio 63 cost 356352 latency 0.001112 osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean]
2014-06-12 14:47:03.016628 7f9d61bd4700  5 -- op tracker -- , seq: 748291, time: 2014-06-12 14:47:03.016620, event: reached_pg, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.016645 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:03.016665 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] handle_message: 0x22b3a000
2014-06-12 14:47:03.016679 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] do_op osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:03.016710 7f9d61bd4700 15 getattr 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 '_'
2014-06-12 14:47:03.036553 7f9d61bd4700 10 getattr 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 '_' = 0
2014-06-12 14:47:03.036589 7f9d61bd4700 15 getattr 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 'snapset'
2014-06-12 14:47:03.036725 7f9d61bd4700 10 getattr 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 'snapset' = 0
2014-06-12 14:47:03.036735 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] populate_obc_watchers 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.036750 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.036757 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] get_object_context: creating obc from disk: 0x244df340
2014-06-12 14:47:03.036763 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] get_object_context: 0x244df340 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 rwstate(none n=0 w=0) oi: 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3(173'986 client.4309.0:37417 wrlock_by=unknown.0.0:0 dirty s 4194304 uv986) ssc: 0x9344a80 snapset: 0=[]:[]+head
2014-06-12 14:47:03.036774 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] find_object_context 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 @head oi=4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3(173'986 client.4309.0:37417 wrlock_by=unknown.0.0:0 dirty s 4194304 uv986)
2014-06-12 14:47:03.036798 7f9d61bd4700  5 -- op tracker -- , seq: 748291, time: 2014-06-12 14:47:03.036791, event: started, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.036807 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] execute_ctx 0x283c8a00
2014-06-12 14:47:03.036818 7f9d61bd4700  5 -- op tracker -- , seq: 748291, time: 2014-06-12 14:47:03.036815, event: started, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.036823 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] do_op 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] ov 173'986 av 173'3274 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:03.036832 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] do_osd_op 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352]
2014-06-12 14:47:03.036838 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:03.036848 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] do_osd_op  write 0~356352
2014-06-12 14:47:03.036859 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:47:03.036865 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] make_writeable 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 snapset=0x9344ab8  snapc=0=[]
2014-06-12 14:47:03.036873 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] make_writeable 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:03.036879 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] finish_ctx 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 0x283c8a00 op modify  
2014-06-12 14:47:03.036887 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean]  set mtime to 2014-06-12 08:46:05.394902
2014-06-12 14:47:03.036900 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean]  final snapset 0=[]:[]+head in 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.036913 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean]  zeroing write result code 0
2014-06-12 14:47:03.036919 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] calc_trim_to 125'272 -> 125'273
2014-06-12 14:47:03.036925 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] new_repop rep_tid 236393 on osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:03.036936 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] issue_repop rep_tid 236393 o 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.036952 7f9d61bd4700  5 -- op tracker -- , seq: 748291, time: 2014-06-12 14:47:03.036949, event: waiting for subops from 23, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.036969 7f9d61bd4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:03.036988 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3273 (125'272,173'3273] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] append_log log((125'272,173'3273], crt=0'0) [173'3274 (173'986) modify   4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 by client.4309.0:141848 2014-06-12 08:46:05.394902]
2014-06-12 14:47:03.037017 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'272,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] add_log_entry 173'3274 (173'986) modify   4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 by client.4309.0:141848 2014-06-12 08:46:05.394902
2014-06-12 14:47:03.037032 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'272,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] append_log  adding 1 keys
2014-06-12 14:47:03.037043 7f9d61bd4700 10 trim log((125'272,173'3274], crt=0'0) to 125'273
2014-06-12 14:47:03.037070 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3274, trimmed: 125'273
2014-06-12 14:47:03.037089 7f9d61bd4700  5 queue_transactions existing osr(3.2f4 0x276d590)/0x276d590
2014-06-12 14:47:03.037093 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:03.037094 7f9d61bd4700  5 queue_op 0x2273500 seq 2955 osr(3.2f4 0x276d590) 358680 bytes   (queue has 1 ops and 358680 bytes)
2014-06-12 14:47:03.037104 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54f0240 (1 active)
2014-06-12 14:47:03.037111 7f9d83173700  5 _do_op 0x2273500 seq 2955 osr(3.2f4 0x276d590)/0x276d590 start
2014-06-12 14:47:03.037115 7f9d83173700 10 _do_transaction on 0x52d1740
2014-06-12 14:47:03.037100 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] eval_repop repgather(0x96da9c0 173'3274 rep_tid=236393 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:03.037118 7f9d61bd4700 10 osd.2 173 dequeue_op 0x22b3a000 finish
2014-06-12 14:47:03.037121 7f9d83173700 15 _omap_setkeys meta/1fe0e805/pglog_3.2f4/0//-1
2014-06-12 14:47:03.037123 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:03.037128 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:03.037277 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe0e805/pglog_3.2f4/0//-1
2014-06-12 14:47:03.037297 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:03.037388 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:03.037422 7f9d83173700 15 _omap_rmkeys meta/1fe0e805/pglog_3.2f4/0//-1
2014-06-12 14:47:03.037433 7f9d83173700 10 _omap_rmkeys meta/1fe0e805/pglog_3.2f4/0//-1 r = 0
2014-06-12 14:47:03.037441 7f9d83173700 15 _omap_setkeys meta/1fe0e805/pglog_3.2f4/0//-1
2014-06-12 14:47:03.037590 7f9d83173700 15 _write 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 0~356352
2014-06-12 14:47:03.037808 7f9d83173700 10 lookup_strip_header done  cid 3.2f4_head oid 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.037982 7f9d83173700 10 file_to_extents done 
2014-06-12 14:47:03.039581 7f9d83173700 10 _generic_write 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 0~356352 = 0
2014-06-12 14:47:03.039640 7f9d83173700 15 _setattrs 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.039657 7f9d83173700 10 _setattrs 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 = 0
2014-06-12 14:47:03.039666 7f9d83173700 15 _setattrs 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.039675 7f9d83173700 10 _setattrs 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 = 0
2014-06-12 14:47:03.039904 7f9d83173700 20 set_header setting 17217 cid 3.2f4_head oid 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 parent seq 0
2014-06-12 14:47:03.039988 7f9d83173700 20 set_header setting 2042 cid meta oid 1fe0e805/pglog_3.2f4/0//-1 parent seq 0
2014-06-12 14:47:03.040011 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:03.040022 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:47:03.042279 7f9d83173700 10 _do_op 0x2273500 seq 2955 r = 0, finisher 0x428b280 0x216a18c0
2014-06-12 14:47:03.042293 7f9d83173700 10 _finish_op 0x2273500 seq 2955 osr(3.2f4 0x276d590)/0x276d590
2014-06-12 14:47:03.042304 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54f0240 (0 active)
2014-06-12 14:47:03.042308 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:03.042299 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] op_commit: 236393
2014-06-12 14:47:03.042330 7f9d7715b700  5 -- op tracker -- , seq: 748291, time: 2014-06-12 14:47:03.042321, event: op_commit, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.042343 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] op_applied: 236393
2014-06-12 14:47:03.042372 7f9d7795c700  5 -- op tracker -- , seq: 748291, time: 2014-06-12 14:47:03.042365, event: op_applied, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.042386 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] op_applied on primary on version 173'3274
2014-06-12 14:47:03.044375 7f9d57543700  5 -- op tracker -- , seq: 748292, time: 2014-06-12 14:47:03.044326, event: header_read, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ack, result = 0)
2014-06-12 14:47:03.044386 7f9d57543700  5 -- op tracker -- , seq: 748292, time: 2014-06-12 14:47:03.044328, event: throttled, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ack, result = 0)
2014-06-12 14:47:03.044412 7f9d57543700  5 -- op tracker -- , seq: 748292, time: 2014-06-12 14:47:03.044342, event: all_read, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ack, result = 0)
2014-06-12 14:47:03.044418 7f9d57543700  5 -- op tracker -- , seq: 748292, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ack, result = 0)
2014-06-12 14:47:03.044423 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:03.044428 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:03.044435 7f9d57543700 15 osd.2 173 enqueue_op 0x17c2a900 prio 196 cost 0 latency 0.000109 osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ack, result = 0) v2
2014-06-12 14:47:03.044465 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:03.044472 7f9d63bd8700 10 osd.2 173 dequeue_op 0x17c2a900 prio 196 cost 0 latency 0.000146 osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ack, result = 0) v2 pg pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean]
2014-06-12 14:47:03.044490 7f9d63bd8700  5 -- op tracker -- , seq: 748292, time: 2014-06-12 14:47:03.044488, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ack, result = 0)
2014-06-12 14:47:03.044501 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] handle_message: 0x17c2a900
2014-06-12 14:47:03.044511 7f9d63bd8700  5 -- op tracker -- , seq: 748292, time: 2014-06-12 14:47:03.044509, event: started, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ack, result = 0)
2014-06-12 14:47:03.044515 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] sub_op_modify_reply: tid 236393 op  ack_type 1 from 23
2014-06-12 14:47:03.044527 7f9d63bd8700  5 -- op tracker -- , seq: 748291, time: 2014-06-12 14:47:03.044522, event: sub_op_applied_rec, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.044532 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] repop_all_applied: repop tid 236393 all applied 
2014-06-12 14:47:03.044538 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] eval_repop repgather(0x96da9c0 173'3274 rep_tid=236393 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:03.044548 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean]  sending ack on repgather(0x96da9c0 173'3274 rep_tid=236393 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4) 0x2fa4ca00
2014-06-12 14:47:03.044570 7f9d63bd8700 10 osd.2 173 dequeue_op 0x17c2a900 finish
2014-06-12 14:47:03.044576 7f9d63bd8700  5 -- op tracker -- , seq: 748292, time: 2014-06-12 14:47:03.044573, event: done, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ack, result = 0)
2014-06-12 14:47:03.044590 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:03.044593 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:03.045456 7f9d57543700  5 -- op tracker -- , seq: 748293, time: 2014-06-12 14:47:03.045426, event: header_read, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.045467 7f9d57543700  5 -- op tracker -- , seq: 748293, time: 2014-06-12 14:47:03.045427, event: throttled, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.045482 7f9d57543700  5 -- op tracker -- , seq: 748293, time: 2014-06-12 14:47:03.045434, event: all_read, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.045487 7f9d57543700  5 -- op tracker -- , seq: 748293, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.045490 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:03.045494 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:03.045498 7f9d57543700 15 osd.2 173 enqueue_op 0x17c2a300 prio 196 cost 0 latency 0.000071 osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:03.045517 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:03.045522 7f9d633d7700 10 osd.2 173 dequeue_op 0x17c2a300 prio 196 cost 0 latency 0.000096 osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0) v2 pg pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean]
2014-06-12 14:47:03.045535 7f9d633d7700  5 -- op tracker -- , seq: 748293, time: 2014-06-12 14:47:03.045533, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.045545 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] handle_message: 0x17c2a300
2014-06-12 14:47:03.045555 7f9d633d7700  5 -- op tracker -- , seq: 748293, time: 2014-06-12 14:47:03.045553, event: started, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.045565 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] sub_op_modify_reply: tid 236393 op  ack_type 4 from 23
2014-06-12 14:47:03.045578 7f9d633d7700  5 -- op tracker -- , seq: 748291, time: 2014-06-12 14:47:03.045573, event: sub_op_commit_rec, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.045582 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3273 crt=0'0 lcod 173'3272 mlcod 173'3272 active+clean] repop_all_committed: repop tid 236393 all committed 
2014-06-12 14:47:03.045588 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3272 active+clean] eval_repop repgather(0x96da9c0 173'3274 rep_tid=236393 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:03.045598 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3272 active+clean] log_op_stats osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4 inb 356631 outb 0 rlat 0.029095 lat 0.030124
2014-06-12 14:47:03.045609 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3272 active+clean] publish_stats_to_osd 173:4201
2014-06-12 14:47:03.045627 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3272 active+clean]  sending commit on repgather(0x96da9c0 173'3274 rep_tid=236393 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4) 0x75cd680
2014-06-12 14:47:03.045652 7f9d633d7700  5 -- op tracker -- , seq: 748291, time: 2014-06-12 14:47:03.045646, event: commit_sent, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.045659 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3272 active+clean]  requeue_ops 
2014-06-12 14:47:03.045670 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean]  removing repgather(0x96da9c0 173'3274 rep_tid=236393 committed?=1 applied?=1 op=osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:03.045682 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean]    q front is repgather(0x96da9c0 173'3274 rep_tid=236393 committed?=1 applied?=1 op=osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:03.045690 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] remove_repop repgather(0x96da9c0 173'3274 rep_tid=236393 committed?=1 applied?=1 op=osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:03.045698 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean]  requeue_ops 
2014-06-12 14:47:03.045716 7f9d633d7700  5 -- op tracker -- , seq: 748291, time: 2014-06-12 14:47:03.045713, event: done, op: osd_op(client.4309.0:141848 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~356352] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.045729 7f9d633d7700 10 osd.2 173 dequeue_op 0x17c2a300 finish
2014-06-12 14:47:03.045733 7f9d633d7700  5 -- op tracker -- , seq: 748293, time: 2014-06-12 14:47:03.045731, event: done, op: osd_sub_op_reply(client.4309.0:141848 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.045739 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:03.045741 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:03.059065 7f9d4397a700  5 -- op tracker -- , seq: 748294, time: 2014-06-12 14:47:03.056393, event: header_read, op: osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.059084 7f9d4397a700  5 -- op tracker -- , seq: 748294, time: 2014-06-12 14:47:03.056403, event: throttled, op: osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.059095 7f9d4397a700  5 -- op tracker -- , seq: 748294, time: 2014-06-12 14:47:03.059006, event: all_read, op: osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.059105 7f9d4397a700  5 -- op tracker -- , seq: 748294, time: 0.000000, event: dispatched, op: osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.059115 7f9d4397a700 20 osd.2 173 should_share_map client.4309 172.16.96.220:0/1244055 173
2014-06-12 14:47:03.059143 7f9d4397a700 15 osd.2 173 enqueue_op 0x876f000 prio 63 cost 1146880 latency 0.002750 osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:03.059177 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:03.059189 7f9d67be0700 10 osd.2 173 dequeue_op 0x876f000 prio 63 cost 1146880 latency 0.002796 osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean]
2014-06-12 14:47:03.059219 7f9d67be0700  5 -- op tracker -- , seq: 748294, time: 2014-06-12 14:47:03.059214, event: reached_pg, op: osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.059229 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:03.059248 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] handle_message: 0x876f000
2014-06-12 14:47:03.059262 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] do_op osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:03.059291 7f9d67be0700 15 getattr 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 '_'
2014-06-12 14:47:03.059466 7f9d67be0700 10 getattr 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 '_' = 0
2014-06-12 14:47:03.059485 7f9d67be0700 15 getattr 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 'snapset'
2014-06-12 14:47:03.059556 7f9d67be0700 10 getattr 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 'snapset' = 0
2014-06-12 14:47:03.059566 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] populate_obc_watchers 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.059581 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.059592 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] get_object_context: creating obc from disk: 0x37e4a000
2014-06-12 14:47:03.059602 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] get_object_context: 0x37e4a000 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 rwstate(none n=0 w=0) oi: 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3(173'3274 client.4309.0:141848 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3274) ssc: 0x1bc1b20 snapset: 0=[]:[]+head
2014-06-12 14:47:03.059624 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] find_object_context 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 @head oi=4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3(173'3274 client.4309.0:141848 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3274)
2014-06-12 14:47:03.059654 7f9d67be0700  5 -- op tracker -- , seq: 748294, time: 2014-06-12 14:47:03.059648, event: started, op: osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.059662 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] execute_ctx 0x2b883800
2014-06-12 14:47:03.059680 7f9d67be0700  5 -- op tracker -- , seq: 748294, time: 2014-06-12 14:47:03.059676, event: started, op: osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.059687 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] do_op 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] ov 173'3274 av 173'3275 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:03.059704 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] do_osd_op 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880]
2014-06-12 14:47:03.059718 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:03.059733 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] do_osd_op  write 356352~1146880
2014-06-12 14:47:03.059751 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] do_osd_op_effects on session 0x5c8d000
2014-06-12 14:47:03.059764 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] make_writeable 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 snapset=0x1bc1b58  snapc=0=[]
2014-06-12 14:47:03.059779 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] make_writeable 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:03.059792 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] finish_ctx 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 0x2b883800 op modify  
2014-06-12 14:47:03.059807 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean]  set mtime to 2014-06-12 08:46:05.434498
2014-06-12 14:47:03.059830 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean]  final snapset 0=[]:[]+head in 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.059850 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean]  zeroing write result code 0
2014-06-12 14:47:03.059861 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] calc_trim_to 125'273 -> 125'274
2014-06-12 14:47:03.059872 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] new_repop rep_tid 236394 on osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:03.059890 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] issue_repop rep_tid 236394 o 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.059918 7f9d67be0700  5 -- op tracker -- , seq: 748294, time: 2014-06-12 14:47:03.059913, event: waiting for subops from 23, op: osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.059941 7f9d67be0700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:03.059966 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3274 (125'273,173'3274] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] append_log log((125'273,173'3274], crt=0'0) [173'3275 (173'3274) modify   4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 by client.4309.0:141849 2014-06-12 08:46:05.434498]
2014-06-12 14:47:03.059992 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'273,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3274 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] add_log_entry 173'3275 (173'3274) modify   4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 by client.4309.0:141849 2014-06-12 08:46:05.434498
2014-06-12 14:47:03.060015 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'273,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3274 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] append_log  adding 1 keys
2014-06-12 14:47:03.060027 7f9d67be0700 10 trim log((125'273,173'3275], crt=0'0) to 125'274
2014-06-12 14:47:03.060051 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3275, trimmed: 125'274
2014-06-12 14:47:03.060073 7f9d67be0700  5 queue_transactions existing osr(3.2f4 0x276d590)/0x276d590
2014-06-12 14:47:03.060077 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:03.060078 7f9d67be0700  5 queue_op 0x7673aa0 seq 2956 osr(3.2f4 0x276d590) 1149208 bytes   (queue has 1 ops and 1149208 bytes)
2014-06-12 14:47:03.060085 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3274 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] eval_repop repgather(0x24fa72c0 173'3275 rep_tid=236394 committed?=0 applied?=0 lock=0 op=osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:03.060101 7f9d67be0700 10 osd.2 173 dequeue_op 0x876f000 finish
2014-06-12 14:47:03.060104 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:03.060107 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:03.060129 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54f0240 (1 active)
2014-06-12 14:47:03.060133 7f9d7a161700  5 _do_op 0x7673aa0 seq 2956 osr(3.2f4 0x276d590)/0x276d590 start
2014-06-12 14:47:03.060137 7f9d7a161700 10 _do_transaction on 0xe832780
2014-06-12 14:47:03.060142 7f9d7a161700 15 _omap_setkeys meta/1fe0e805/pglog_3.2f4/0//-1
2014-06-12 14:47:03.060223 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe0e805/pglog_3.2f4/0//-1
2014-06-12 14:47:03.060240 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:03.060290 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:03.060311 7f9d7a161700 15 _omap_rmkeys meta/1fe0e805/pglog_3.2f4/0//-1
2014-06-12 14:47:03.060318 7f9d7a161700 10 _omap_rmkeys meta/1fe0e805/pglog_3.2f4/0//-1 r = 0
2014-06-12 14:47:03.060323 7f9d7a161700 15 _omap_setkeys meta/1fe0e805/pglog_3.2f4/0//-1
2014-06-12 14:47:03.060432 7f9d7a161700 15 _write 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 356352~1146880
2014-06-12 14:47:03.060558 7f9d7a161700 10 lookup_strip_header done  cid 3.2f4_head oid 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.060705 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:47:03.063979 7f9d7a161700 10 _generic_write 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 356352~1146880 = 0
2014-06-12 14:47:03.064065 7f9d7a161700 15 _setattrs 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.064088 7f9d7a161700 10 _setattrs 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 = 0
2014-06-12 14:47:03.064097 7f9d7a161700 15 _setattrs 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3
2014-06-12 14:47:03.064106 7f9d7a161700 10 _setattrs 3.2f4_head/4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 = 0
2014-06-12 14:47:03.064325 7f9d7a161700 20 set_header setting 17217 cid 3.2f4_head oid 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 parent seq 0
2014-06-12 14:47:03.064409 7f9d7a161700 20 set_header setting 2042 cid meta oid 1fe0e805/pglog_3.2f4/0//-1 parent seq 0
2014-06-12 14:47:03.064427 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:03.064440 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:47:03.068703 7f9d7a161700 10 _do_op 0x7673aa0 seq 2956 r = 0, finisher 0x7f43b20 0x3786edc0
2014-06-12 14:47:03.068732 7f9d7a161700 10 _finish_op 0x7673aa0 seq 2956 osr(3.2f4 0x276d590)/0x276d590
2014-06-12 14:47:03.068741 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54f0240 (0 active)
2014-06-12 14:47:03.068747 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:03.068741 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3274 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] op_commit: 236394
2014-06-12 14:47:03.068779 7f9d7715b700  5 -- op tracker -- , seq: 748294, time: 2014-06-12 14:47:03.068768, event: op_commit, op: osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.068800 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3274 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] op_applied: 236394
2014-06-12 14:47:03.068819 7f9d7795c700  5 -- op tracker -- , seq: 748294, time: 2014-06-12 14:47:03.068815, event: op_applied, op: osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.068828 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3274 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] op_applied on primary on version 173'3275
2014-06-12 14:47:03.074324 7f9d57543700  5 -- op tracker -- , seq: 748295, time: 2014-06-12 14:47:03.074275, event: header_read, op: osd_sub_op_reply(client.4309.0:141849 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.074334 7f9d57543700  5 -- op tracker -- , seq: 748295, time: 2014-06-12 14:47:03.074277, event: throttled, op: osd_sub_op_reply(client.4309.0:141849 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.074340 7f9d57543700  5 -- op tracker -- , seq: 748295, time: 2014-06-12 14:47:03.074290, event: all_read, op: osd_sub_op_reply(client.4309.0:141849 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.074344 7f9d57543700  5 -- op tracker -- , seq: 748295, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4309.0:141849 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.074349 7f9d57543700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4309.0:141849 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:03.074353 7f9d57543700 20 osd.2 173 should_share_map osd.23 172.16.96.202:6820/2365 173
2014-06-12 14:47:03.074359 7f9d57543700 15 osd.2 173 enqueue_op 0x68a7e00 prio 196 cost 0 latency 0.000084 osd_sub_op_reply(client.4309.0:141849 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:03.074385 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:03.074394 7f9d673df700 10 osd.2 173 dequeue_op 0x68a7e00 prio 196 cost 0 latency 0.000118 osd_sub_op_reply(client.4309.0:141849 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0) v2 pg pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3274 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean]
2014-06-12 14:47:03.074408 7f9d673df700  5 -- op tracker -- , seq: 748295, time: 2014-06-12 14:47:03.074406, event: reached_pg, op: osd_sub_op_reply(client.4309.0:141849 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.074415 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3274 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] handle_message: 0x68a7e00
2014-06-12 14:47:03.074426 7f9d673df700  5 -- op tracker -- , seq: 748295, time: 2014-06-12 14:47:03.074424, event: started, op: osd_sub_op_reply(client.4309.0:141849 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.074430 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3274 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] sub_op_modify_reply: tid 236394 op  ack_type 4 from 23
2014-06-12 14:47:03.074442 7f9d673df700  5 -- op tracker -- , seq: 748294, time: 2014-06-12 14:47:03.074437, event: sub_op_commit_rec, op: osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.074448 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3274 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] repop_all_applied: repop tid 236394 all applied 
2014-06-12 14:47:03.074456 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3274 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] eval_repop repgather(0x24fa72c0 173'3275 rep_tid=236394 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:03.074467 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3274 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean]  sending ack on repgather(0x24fa72c0 173'3275 rep_tid=236394 committed?=0 applied?=1 lock=0 op=osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4) 0x288b9180
2014-06-12 14:47:03.074488 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 luod=173'3274 crt=0'0 lcod 173'3273 mlcod 173'3273 active+clean] repop_all_committed: repop tid 236394 all committed 
2014-06-12 14:47:03.074495 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3273 active+clean] eval_repop repgather(0x24fa72c0 173'3275 rep_tid=236394 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:03.074505 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3273 active+clean] log_op_stats osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4 inb 1147159 outb 0 rlat 0.018094 lat 0.018111
2014-06-12 14:47:03.074516 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3273 active+clean] publish_stats_to_osd 173:4202
2014-06-12 14:47:03.074524 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3273 active+clean]  sending commit on repgather(0x24fa72c0 173'3275 rep_tid=236394 committed?=1 applied?=1 lock=0 op=osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4) 0x28efa000
2014-06-12 14:47:03.074541 7f9d673df700  5 -- op tracker -- , seq: 748294, time: 2014-06-12 14:47:03.074538, event: commit_sent, op: osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.074556 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3273 active+clean]  requeue_ops 
2014-06-12 14:47:03.074567 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3274 active+clean]  removing repgather(0x24fa72c0 173'3275 rep_tid=236394 committed?=1 applied?=1 op=osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:03.074582 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3274 active+clean]    q front is repgather(0x24fa72c0 173'3275 rep_tid=236394 committed?=1 applied?=1 op=osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:03.074598 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3274 active+clean] remove_repop repgather(0x24fa72c0 173'3275 rep_tid=236394 committed?=1 applied?=1 op=osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:03.074612 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3274 active+clean]  requeue_ops 
2014-06-12 14:47:03.074638 7f9d673df700  5 -- op tracker -- , seq: 748294, time: 2014-06-12 14:47:03.074633, event: done, op: osd_op(client.4309.0:141849 rbd_data.10ac6b8b4567.0000000000000f92 [set-alloc-hint object_size 4194304 write_size 4194304,write 356352~1146880] 3.4d6e0af4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.074655 7f9d673df700 10 osd.2 173 dequeue_op 0x68a7e00 finish
2014-06-12 14:47:03.074666 7f9d673df700  5 -- op tracker -- , seq: 748295, time: 2014-06-12 14:47:03.074662, event: done, op: osd_sub_op_reply(client.4309.0:141849 3.2f4 4d6e0af4/rbd_data.10ac6b8b4567.0000000000000f92/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.074676 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:03.074680 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:03.313034 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:03.313206 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:03.481327 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:03.481357 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:03.481380 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 10.28 >= max 0.5 = no, load too high
2014-06-12 14:47:03.481392 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:47:03.481394 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92671 < max (604800 seconds)
2014-06-12 14:47:03.481404 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:47:03.481406 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:03.481406 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:03.481408 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 28; oldest is 29.815540 seconds old
2014-06-12 14:47:03.509664 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:03.509718 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:03.579016 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:03.584250 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:03.671493 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:03.694709 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:03.720064 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:03.729881 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:03.737999 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:03.744272 7f9d56432700  5 -- op tracker -- , seq: 748296, time: 2014-06-12 14:47:03.744227, event: header_read, op: osd_sub_op_reply(client.4293.0:144176 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.744284 7f9d56432700  5 -- op tracker -- , seq: 748296, time: 2014-06-12 14:47:03.744229, event: throttled, op: osd_sub_op_reply(client.4293.0:144176 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.744289 7f9d56432700  5 -- op tracker -- , seq: 748296, time: 2014-06-12 14:47:03.744244, event: all_read, op: osd_sub_op_reply(client.4293.0:144176 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.744294 7f9d56432700  5 -- op tracker -- , seq: 748296, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144176 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.744298 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144176 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:03.744302 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:03.744308 7f9d56432700 15 osd.2 173 enqueue_op 0x25e42000 prio 196 cost 0 latency 0.000081 osd_sub_op_reply(client.4293.0:144176 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:03.744336 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:03.744355 7f9d663dd700 10 osd.2 173 dequeue_op 0x25e42000 prio 196 cost 0 latency 0.000127 osd_sub_op_reply(client.4293.0:144176 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0) v2 pg pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2815 active+clean]
2014-06-12 14:47:03.744380 7f9d663dd700  5 -- op tracker -- , seq: 748296, time: 2014-06-12 14:47:03.744377, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144176 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.744392 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2815 active+clean] handle_message: 0x25e42000
2014-06-12 14:47:03.744411 7f9d663dd700  5 -- op tracker -- , seq: 748296, time: 2014-06-12 14:47:03.744408, event: started, op: osd_sub_op_reply(client.4293.0:144176 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.744420 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2815 active+clean] sub_op_modify_reply: tid 236279 op  ack_type 4 from 26
2014-06-12 14:47:03.744444 7f9d663dd700  5 -- op tracker -- , seq: 747889, time: 2014-06-12 14:47:03.744435, event: sub_op_commit_rec, op: osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.744454 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2815 active+clean] repop_all_applied: repop tid 236279 all applied 
2014-06-12 14:47:03.744474 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2815 active+clean] eval_repop repgather(0x10192240 173'2817 rep_tid=236279 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:03.744493 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2815 active+clean]  sending ack on repgather(0x10192240 173'2817 rep_tid=236279 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) 0x33271b80
2014-06-12 14:47:03.744535 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2816 crt=0'0 lcod 173'2815 mlcod 173'2815 active+clean] repop_all_committed: repop tid 236279 all committed 
2014-06-12 14:47:03.744555 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2815 active+clean] eval_repop repgather(0x10192240 173'2817 rep_tid=236279 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:03.744580 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2815 active+clean] log_op_stats osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 30.078665 lat 30.078710
2014-06-12 14:47:03.744607 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2815 active+clean] publish_stats_to_osd 173:3142
2014-06-12 14:47:03.744623 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2815 active+clean]  sending commit on repgather(0x10192240 173'2817 rep_tid=236279 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) 0x33b17180
2014-06-12 14:47:03.744661 7f9d663dd700  5 -- op tracker -- , seq: 747889, time: 2014-06-12 14:47:03.744656, event: commit_sent, op: osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.744678 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2815 active+clean]  requeue_ops 
2014-06-12 14:47:03.744698 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2816 active+clean]  removing repgather(0x10192240 173'2817 rep_tid=236279 committed?=1 applied?=1 op=osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:03.744716 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2816 active+clean]    q front is repgather(0x10192240 173'2817 rep_tid=236279 committed?=1 applied?=1 op=osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:03.744732 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2816 active+clean] remove_repop repgather(0x10192240 173'2817 rep_tid=236279 committed?=1 applied?=1 op=osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:03.744746 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2816 active+clean]  requeue_ops 
2014-06-12 14:47:03.744773 7f9d663dd700  5 -- op tracker -- , seq: 747889, time: 2014-06-12 14:47:03.744767, event: done, op: osd_op(client.4293.0:144176 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 2113536~1548288] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.744796 7f9d663dd700 10 osd.2 173 dequeue_op 0x25e42000 finish
2014-06-12 14:47:03.744805 7f9d663dd700  5 -- op tracker -- , seq: 748296, time: 2014-06-12 14:47:03.744801, event: done, op: osd_sub_op_reply(client.4293.0:144176 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.744819 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:03.744823 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:03.770601 7f9d56432700  5 -- op tracker -- , seq: 748297, time: 2014-06-12 14:47:03.768913, event: header_read, op: osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.770624 7f9d56432700  5 -- op tracker -- , seq: 748297, time: 2014-06-12 14:47:03.768918, event: throttled, op: osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.770644 7f9d56432700  5 -- op tracker -- , seq: 748297, time: 2014-06-12 14:47:03.770553, event: all_read, op: osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.770649 7f9d56432700  5 -- op tracker -- , seq: 748297, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.770653 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:03.770657 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:03.770664 7f9d56432700 15 osd.2 173 enqueue_op 0x25e42400 prio 127 cost 1032999 latency 0.001751 osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:03.770690 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:03.770700 7f9d65bdc700 10 osd.2 173 dequeue_op 0x25e42400 prio 127 cost 1032999 latency 0.001786 osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2652 (0'0,173'2652] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2651 active]
2014-06-12 14:47:03.770729 7f9d65bdc700  5 -- op tracker -- , seq: 748297, time: 2014-06-12 14:47:03.770724, event: reached_pg, op: osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.770739 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2652 (0'0,173'2652] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2651 active] handle_message: 0x25e42400
2014-06-12 14:47:03.770754 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2652 (0'0,173'2652] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2651 active] sub_op_modify trans af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 v 173'2653 (transaction) 174
2014-06-12 14:47:03.770775 7f9d65bdc700  5 -- op tracker -- , seq: 748297, time: 2014-06-12 14:47:03.770771, event: started, op: osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.770792 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2652 (0'0,173'2652] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2651 active] append_log log((0'0,173'2652], crt=0'0) [173'2653 (173'2651) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147309 2014-06-12 08:46:06.143733]
2014-06-12 14:47:03.770819 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2653 (0'0,173'2653] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2651 active] add_log_entry 173'2653 (173'2651) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147309 2014-06-12 08:46:06.143733
2014-06-12 14:47:03.770843 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2653 (0'0,173'2653] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2651 active] append_log  adding 1 keys
2014-06-12 14:47:03.770879 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2653, trimmed: 
2014-06-12 14:47:03.770905 7f9d65bdc700  5 -- op tracker -- , seq: 748297, time: 2014-06-12 14:47:03.770900, event: started, op: osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.770917 7f9d65bdc700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:03.770921 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:03.770922 7f9d65bdc700  5 queue_op 0x54116e0 seq 2473 osr(3.1c7 0x276d170) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:03.770928 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2653 (0'0,173'2653] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2651 active] do_sub_op osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:03.770950 7f9d65bdc700 10 osd.2 173 dequeue_op 0x25e42400 finish
2014-06-12 14:47:03.770953 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:03.770956 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:03.770960 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:47:03.770968 7f9d7915f700  5 _do_op 0x54116e0 seq 2473 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:47:03.770972 7f9d7915f700 10 _do_transaction on 0x1ea81430
2014-06-12 14:47:03.770977 7f9d7915f700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:03.771063 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:03.771087 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:03.771118 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:03.771137 7f9d7915f700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:03.771142 7f9d7915f700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:47:03.771146 7f9d7915f700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:03.771274 7f9d7915f700 15 _write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 2965504~1032192
2014-06-12 14:47:03.771372 7f9d7915f700 10 lookup_strip_header done  cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:47:03.771483 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:47:03.773988 7f9d7915f700 10 _generic_write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 2965504~1032192 = 0
2014-06-12 14:47:03.774059 7f9d7915f700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:47:03.774073 7f9d7915f700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:47:03.774076 7f9d7915f700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:47:03.774081 7f9d7915f700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:47:03.774275 7f9d7915f700 20 set_header setting 17479 cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 parent seq 0
2014-06-12 14:47:03.774343 7f9d7915f700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:47:03.774354 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:03.774359 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:47:03.778082 7f9d7915f700 10 _do_op 0x54116e0 seq 2473 r = 0, finisher 0x31d36500 0
2014-06-12 14:47:03.778116 7f9d7915f700 10 _finish_op 0x54116e0 seq 2473 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:03.778124 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:47:03.778139 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:03.778154 7f9d7795c700  5 -- op tracker -- , seq: 748297, time: 2014-06-12 14:47:03.778141, event: sub_op_applied, op: osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.778170 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2653 (0'0,173'2653] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2651 active] sub_op_modify_applied on 0x1ea81340 op osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:03.778208 7f9d7795c700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:03.778229 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2653 (0'0,173'2653] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2651 active] op_applied on primary on version 173'2653
2014-06-12 14:47:03.778248 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2653 (0'0,173'2653] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2651 active] op_applied on replica on version 173'2653
2014-06-12 14:47:03.778303 7f9d7715b700  5 -- op tracker -- , seq: 748297, time: 2014-06-12 14:47:03.778297, event: commit_sent, op: osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.778312 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2653 (0'0,173'2653] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2651 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:47:03.778334 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:03.778356 7f9d7715b700  5 -- op tracker -- , seq: 748297, time: 2014-06-12 14:47:03.778353, event: done, op: osd_sub_op(client.4316.0:147309 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2653 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.900918 7f9d56432700  5 -- op tracker -- , seq: 748298, time: 2014-06-12 14:47:03.900622, event: header_read, op: osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.900930 7f9d56432700  5 -- op tracker -- , seq: 748298, time: 2014-06-12 14:47:03.900625, event: throttled, op: osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.900945 7f9d56432700  5 -- op tracker -- , seq: 748298, time: 2014-06-12 14:47:03.900884, event: all_read, op: osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.900950 7f9d56432700  5 -- op tracker -- , seq: 748298, time: 0.000000, event: dispatched, op: osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.900954 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:03.900959 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:03.900965 7f9d56432700 15 osd.2 173 enqueue_op 0x25e43a00 prio 127 cost 197415 latency 0.000343 osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:03.900988 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:03.900994 7f9d62bd6700 10 osd.2 173 dequeue_op 0x25e43a00 prio 127 cost 197415 latency 0.000372 osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2653 (0'0,173'2653] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2652 active]
2014-06-12 14:47:03.901011 7f9d62bd6700  5 -- op tracker -- , seq: 748298, time: 2014-06-12 14:47:03.901008, event: reached_pg, op: osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.901017 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2653 (0'0,173'2653] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2652 active] handle_message: 0x25e43a00
2014-06-12 14:47:03.901030 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2653 (0'0,173'2653] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2652 active] sub_op_modify trans af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 v 173'2654 (transaction) 174
2014-06-12 14:47:03.901041 7f9d62bd6700  5 -- op tracker -- , seq: 748298, time: 2014-06-12 14:47:03.901038, event: started, op: osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.901052 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2653 (0'0,173'2653] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2652 active] append_log log((0'0,173'2653], crt=0'0) [173'2654 (173'2653) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147310 2014-06-12 08:46:06.279038]
2014-06-12 14:47:03.901069 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2654 (0'0,173'2654] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2652 active] add_log_entry 173'2654 (173'2653) modify   af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 by client.4316.0:147310 2014-06-12 08:46:06.279038
2014-06-12 14:47:03.901089 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2654 (0'0,173'2654] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2652 active] append_log  adding 1 keys
2014-06-12 14:47:03.901112 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2654, trimmed: 
2014-06-12 14:47:03.901132 7f9d62bd6700  5 -- op tracker -- , seq: 748298, time: 2014-06-12 14:47:03.901129, event: started, op: osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.901140 7f9d62bd6700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:03.901144 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:03.901144 7f9d62bd6700  5 queue_op 0x7ac9320 seq 2474 osr(3.1c7 0x276d170) 198901 bytes   (queue has 1 ops and 198901 bytes)
2014-06-12 14:47:03.901153 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:47:03.901150 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2654 (0'0,173'2654] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2652 active] do_sub_op osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:03.901159 7f9d62bd6700 10 osd.2 173 dequeue_op 0x25e43a00 finish
2014-06-12 14:47:03.901161 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:03.901161 7f9d8016d700  5 _do_op 0x7ac9320 seq 2474 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:47:03.901163 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:03.901165 7f9d8016d700 10 _do_transaction on 0x2a6707f0
2014-06-12 14:47:03.901172 7f9d8016d700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:03.901277 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:03.901305 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:03.901343 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:03.901362 7f9d8016d700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:03.901367 7f9d8016d700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:47:03.901372 7f9d8016d700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:03.901384 7f9d8016d700 15 _write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 3997696~196608
2014-06-12 14:47:03.901511 7f9d8016d700 10 lookup_strip_header done  cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:47:03.901526 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:47:03.902286 7f9d8016d700 10 _generic_write 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 3997696~196608 = 0
2014-06-12 14:47:03.902339 7f9d8016d700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:47:03.902349 7f9d8016d700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:47:03.902353 7f9d8016d700 15 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3
2014-06-12 14:47:03.902357 7f9d8016d700 10 _setattrs 3.1c7_head/af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 = 0
2014-06-12 14:47:03.902454 7f9d8016d700 20 set_header setting 17479 cid 3.1c7_head oid af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 parent seq 0
2014-06-12 14:47:03.902476 7f9d8016d700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:47:03.902487 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:03.902492 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:47:03.903300 7f9d8016d700 10 _do_op 0x7ac9320 seq 2474 r = 0, finisher 0x6d9d880 0
2014-06-12 14:47:03.903313 7f9d8016d700 10 _finish_op 0x7ac9320 seq 2474 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:03.903323 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:47:03.903326 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:03.903338 7f9d7715b700  5 -- op tracker -- , seq: 748298, time: 2014-06-12 14:47:03.903325, event: commit_sent, op: osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.903349 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2654 (0'0,173'2654] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2652 active] sub_op_modify_commit on op osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:47:03.903379 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:03.903418 7f9d7795c700  5 -- op tracker -- , seq: 748298, time: 2014-06-12 14:47:03.903413, event: sub_op_applied, op: osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.903425 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2654 (0'0,173'2654] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2653 active] sub_op_modify_applied on 0x2a670700 op osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:03.903441 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2654 (0'0,173'2654] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2653 active] op_applied on primary on version 173'2654
2014-06-12 14:47:03.903451 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2654 (0'0,173'2654] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2653 active] op_applied on replica on version 173'2654
2014-06-12 14:47:03.903477 7f9d7795c700  5 -- op tracker -- , seq: 748298, time: 2014-06-12 14:47:03.903473, event: done, op: osd_sub_op(client.4316.0:147310 3.1c7 af6f11c7/rbd_data.10b36b8b4567.000000000000113c/head//3 [] v 173'2654 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:03.928048 7f9d56432700  5 -- op tracker -- , seq: 748299, time: 2014-06-12 14:47:03.928007, event: header_read, op: osd_sub_op_reply(client.4293.0:144177 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.928065 7f9d56432700  5 -- op tracker -- , seq: 748299, time: 2014-06-12 14:47:03.928009, event: throttled, op: osd_sub_op_reply(client.4293.0:144177 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.928070 7f9d56432700  5 -- op tracker -- , seq: 748299, time: 2014-06-12 14:47:03.928018, event: all_read, op: osd_sub_op_reply(client.4293.0:144177 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.928074 7f9d56432700  5 -- op tracker -- , seq: 748299, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144177 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.928078 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144177 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:03.928081 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:03.928087 7f9d56432700 15 osd.2 173 enqueue_op 0x25e42300 prio 196 cost 0 latency 0.000080 osd_sub_op_reply(client.4293.0:144177 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:03.928108 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:03.928117 7f9d683e1700 10 osd.2 173 dequeue_op 0x25e42300 prio 196 cost 0 latency 0.000109 osd_sub_op_reply(client.4293.0:144177 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0) v2 pg pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2816 active+clean]
2014-06-12 14:47:03.928138 7f9d683e1700  5 -- op tracker -- , seq: 748299, time: 2014-06-12 14:47:03.928135, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144177 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.928148 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2816 active+clean] handle_message: 0x25e42300
2014-06-12 14:47:03.928164 7f9d683e1700  5 -- op tracker -- , seq: 748299, time: 2014-06-12 14:47:03.928160, event: started, op: osd_sub_op_reply(client.4293.0:144177 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.928175 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2816 active+clean] sub_op_modify_reply: tid 236280 op  ack_type 4 from 26
2014-06-12 14:47:03.928197 7f9d683e1700  5 -- op tracker -- , seq: 747890, time: 2014-06-12 14:47:03.928189, event: sub_op_commit_rec, op: osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.928205 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2816 active+clean] repop_all_applied: repop tid 236280 all applied 
2014-06-12 14:47:03.928218 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2816 active+clean] eval_repop repgather(0x2dc1ee40 173'2818 rep_tid=236280 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:03.928236 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2816 active+clean]  sending ack on repgather(0x2dc1ee40 173'2818 rep_tid=236280 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) 0x2b43180
2014-06-12 14:47:03.928286 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2817 crt=0'0 lcod 173'2816 mlcod 173'2816 active+clean] repop_all_committed: repop tid 236280 all committed 
2014-06-12 14:47:03.928306 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2816 active+clean] eval_repop repgather(0x2dc1ee40 173'2818 rep_tid=236280 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:03.928324 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2816 active+clean] log_op_stats osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4 inb 532759 outb 0 rlat 30.215700 lat 30.215738
2014-06-12 14:47:03.928341 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2816 active+clean] publish_stats_to_osd 173:3143
2014-06-12 14:47:03.928364 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2816 active+clean]  sending commit on repgather(0x2dc1ee40 173'2818 rep_tid=236280 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4) 0xfdb2500
2014-06-12 14:47:03.928427 7f9d683e1700  5 -- op tracker -- , seq: 747890, time: 2014-06-12 14:47:03.928408, event: commit_sent, op: osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.928447 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2816 active+clean]  requeue_ops 
2014-06-12 14:47:03.928457 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2817 active+clean]  removing repgather(0x2dc1ee40 173'2818 rep_tid=236280 committed?=1 applied?=1 op=osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:03.928474 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2817 active+clean]    q front is repgather(0x2dc1ee40 173'2818 rep_tid=236280 committed?=1 applied?=1 op=osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:03.928501 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2817 active+clean] remove_repop repgather(0x2dc1ee40 173'2818 rep_tid=236280 committed?=1 applied?=1 op=osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:03.928514 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2817 active+clean]  requeue_ops 
2014-06-12 14:47:03.928538 7f9d683e1700  5 -- op tracker -- , seq: 747890, time: 2014-06-12 14:47:03.928532, event: done, op: osd_op(client.4293.0:144177 rbd_data.109c6b8b4567.0000000000001058 [set-alloc-hint object_size 4194304 write_size 4194304,write 3661824~532480] 3.67092105 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:03.928565 7f9d683e1700 10 osd.2 173 dequeue_op 0x25e42300 finish
2014-06-12 14:47:03.928580 7f9d683e1700  5 -- op tracker -- , seq: 748299, time: 2014-06-12 14:47:03.928576, event: done, op: osd_sub_op_reply(client.4293.0:144177 3.105 67092105/rbd_data.109c6b8b4567.0000000000001058/head//3 [] ondisk, result = 0)
2014-06-12 14:47:03.928590 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:03.928593 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:03.973527 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:03.973836 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:04.025733 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:04.040751 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:04.042153 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:04.042282 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:04.043141 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:04.045787 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:04.050419 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:04.053757 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:04.054010 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:04.058359 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:04.059912 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:04.063488 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:04.066056 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:04.066274 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:04.155539 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:04.370049 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:04.481486 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:04.481521 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:04.481532 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:04.481534 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:04.481536 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 26; oldest is 24.173093 seconds old
2014-06-12 14:47:04.481574 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:04.504427 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:04.504614 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:04.514092 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:04.527153 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:04.545076 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:04.546408 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:04.546627 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:04.559459 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:04.568723 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:04.576179 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:04.847501 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:04.847504 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:04.847505 7f9d74bfa700 20 osd.2 173 _dispatch 0x183876c0 ping v1
2014-06-12 14:47:04.847508 7f9d74bfa700 10 osd.2 173 ping from client.4290
2014-06-12 14:47:04.847510 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:04.847511 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:05.037202 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:05.042380 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:05.044666 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:05.045806 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:05.057280 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:05.057349 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:05.060178 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:05.068819 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:05.074750 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:05.149382 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:05.149488 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:05.160079 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:05.160084 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:05.160086 7f9d74bfa700 20 osd.2 173 _dispatch 0x32119a00 ping v1
2014-06-12 14:47:05.160090 7f9d74bfa700 10 osd.2 173 ping from client.4293
2014-06-12 14:47:05.160093 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:05.160094 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:05.324853 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19425 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,1,4,6,15])
2014-06-12 14:47:05.324871 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19425 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,1,4,6,15])
2014-06-12 14:47:05.325285 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:05.325313 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:05.325335 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:05.325366 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:05.325385 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:05.325397 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:05.325409 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:05.325435 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:05.325463 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:05.325479 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:05.325489 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:05.325503 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:05.325515 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:05.325526 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:05.325538 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:05.325551 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:05.325571 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:05.325589 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:05.325602 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:05.325627 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:05.325641 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:05.325654 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:05.481630 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:05.481660 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:05.481665 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:05.481667 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:05.481670 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 26; oldest is 25.173228 seconds old
2014-06-12 14:47:05.481707 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:05.579099 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:05.584318 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:05.671582 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:05.694793 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:05.711812 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:05.711815 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:05.711816 7f9d74bfa700 20 osd.2 173 _dispatch 0x6b78ea0 ping v1
2014-06-12 14:47:05.711818 7f9d74bfa700 10 osd.2 173 ping from client.4300
2014-06-12 14:47:05.711819 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:05.711820 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:05.720136 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:05.729952 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:05.738065 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:05.742485 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:05.742576 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:05.744884 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:05.771032 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:05.778226 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:05.901227 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:05.903397 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:05.928670 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:05.960789 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:05.960833 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:06.025809 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:06.040839 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:06.042239 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:06.042362 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:06.043218 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:06.045865 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:06.050482 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:06.053820 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:06.054073 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:06.058421 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:06.059977 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:06.063556 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:06.066121 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:06.066339 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:06.125733 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:06.125824 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:06.155619 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:06.213862 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:06.214286 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:06.246965 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:06.247089 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:06.289752 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:06.289758 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:06.289761 7f9d74bfa700 20 osd.2 173 _dispatch 0x2c2dc9c0 ping v1
2014-06-12 14:47:06.289766 7f9d74bfa700 10 osd.2 173 ping from client.4306
2014-06-12 14:47:06.289771 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:06.289774 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:06.323204 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:06.323311 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:06.370137 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:06.425485 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:06.425969 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:06.481772 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:06.481803 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:06.481810 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:06.481817 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:06.481815 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:06.481821 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 26; oldest is 26.173379 seconds old
2014-06-12 14:47:06.514182 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:06.527230 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:06.545163 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:06.559535 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:06.568812 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:06.576266 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:06.804915 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:06.804992 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:06.865280 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:06.865285 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:06.865287 7f9d74bfa700 20 osd.2 173 _dispatch 0x17705a00 ping v1
2014-06-12 14:47:06.865294 7f9d74bfa700 10 osd.2 173 ping from client.4314
2014-06-12 14:47:06.865298 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:06.865300 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:07.010291 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:07.010432 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:07.037287 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:07.042452 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:07.044739 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:07.045875 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:07.060250 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:07.068901 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:07.074844 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:07.347707 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:07.347752 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:07.423575 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:07.423704 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:07.481897 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:07.481919 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:47:07.481923 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:47:07.481927 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:47:07.481931 7f9d84175700 10 osd.2 173 send_pg_stats - 2 pgs updated
2014-06-12 14:47:07.481988 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:07.482025 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 9.62 >= max 0.5 = no, load too high
2014-06-12 14:47:07.482042 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:47:07.482046 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92675 < max (604800 seconds)
2014-06-12 14:47:07.482066 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:47:07.482068 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:07.482070 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:07.482073 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 26; oldest is 27.173630 seconds old
2014-06-12 14:47:07.526030 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:07.526110 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:07.579187 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:07.584390 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:07.671670 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:07.694865 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:07.720210 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:07.730021 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:07.738127 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:07.744951 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:07.770444 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:07.770449 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:07.770451 7f9d74bfa700 20 osd.2 173 _dispatch 0x641e1c0 pg_stats_ack(2 pgs tid 4859) v1
2014-06-12 14:47:07.770454 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:47:07.770460 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:07.770461 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:07.771114 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:07.778308 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:07.901327 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:07.903480 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:07.905674 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:07.905725 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:07.928758 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.025893 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.040919 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.042341 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.042441 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.043138 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:08.043318 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:08.043542 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.045943 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.050559 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.053913 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.054145 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.058495 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.060070 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.063632 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.066213 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.066416 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.155705 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:08.370211 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:08.373328 7f9d5a16f700  5 -- op tracker -- , seq: 748300, time: 2014-06-12 14:47:08.370718, event: header_read, op: osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.373343 7f9d5a16f700  5 -- op tracker -- , seq: 748300, time: 2014-06-12 14:47:08.370720, event: throttled, op: osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.373351 7f9d5a16f700  5 -- op tracker -- , seq: 748300, time: 2014-06-12 14:47:08.373285, event: all_read, op: osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.373360 7f9d5a16f700  5 -- op tracker -- , seq: 748300, time: 0.000000, event: dispatched, op: osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.373364 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:08.373370 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:08.373380 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9d700 prio 127 cost 1032999 latency 0.002661 osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:08.373425 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.373436 7f9d623d5700 10 osd.2 173 dequeue_op 0x24c9d700 prio 127 cost 1032999 latency 0.002717 osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2672 (0'0,173'2672] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2671 active]
2014-06-12 14:47:08.373464 7f9d623d5700  5 -- op tracker -- , seq: 748300, time: 2014-06-12 14:47:08.373460, event: reached_pg, op: osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.373473 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2672 (0'0,173'2672] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2671 active] handle_message: 0x24c9d700
2014-06-12 14:47:08.373485 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2672 (0'0,173'2672] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2671 active] sub_op_modify trans 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 v 173'2673 (transaction) 174
2014-06-12 14:47:08.373503 7f9d623d5700  5 -- op tracker -- , seq: 748300, time: 2014-06-12 14:47:08.373499, event: started, op: osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.373519 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2672 (0'0,173'2672] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2671 active] append_log log((0'0,173'2672], crt=0'0) [173'2673 (173'2672) modify   57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 by client.4282.0:145227 2014-06-12 08:44:32.573566]
2014-06-12 14:47:08.373545 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2673 (0'0,173'2673] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2671 active] add_log_entry 173'2673 (173'2672) modify   57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 by client.4282.0:145227 2014-06-12 08:44:32.573566
2014-06-12 14:47:08.373569 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2673 (0'0,173'2673] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2671 active] append_log  adding 1 keys
2014-06-12 14:47:08.373605 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2673, trimmed: 
2014-06-12 14:47:08.373640 7f9d623d5700  5 -- op tracker -- , seq: 748300, time: 2014-06-12 14:47:08.373634, event: started, op: osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.373652 7f9d623d5700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:08.373657 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.373658 7f9d623d5700  5 queue_op 0x160653e0 seq 2483 osr(3.1b7 0x276d1a0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:08.373665 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2673 (0'0,173'2673] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2671 active] do_sub_op osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:08.373680 7f9d623d5700 10 osd.2 173 dequeue_op 0x24c9d700 finish
2014-06-12 14:47:08.373683 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.373686 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.373694 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:08.373699 7f9d7a962700  5 _do_op 0x160653e0 seq 2483 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:08.373703 7f9d7a962700 10 _do_transaction on 0x61e60f0
2014-06-12 14:47:08.373708 7f9d7a962700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:08.373821 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:08.373841 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.373882 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.373909 7f9d7a962700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:08.373918 7f9d7a962700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:08.373924 7f9d7a962700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:08.373941 7f9d7a962700 15 _write 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 569344~1032192
2014-06-12 14:47:08.374281 7f9d7a962700 10 lookup_strip_header done  cid 3.1b7_head oid 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:08.374471 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:47:08.378235 7f9d7a962700 10 _generic_write 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 569344~1032192 = 0
2014-06-12 14:47:08.378303 7f9d7a962700 15 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:08.378321 7f9d7a962700 10 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:47:08.378330 7f9d7a962700 15 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:08.378339 7f9d7a962700 10 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:47:08.378585 7f9d7a962700 20 set_header setting 18080 cid 3.1b7_head oid 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 parent seq 0
2014-06-12 14:47:08.378679 7f9d7a962700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:08.378697 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.378709 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:47:08.383066 7f9d7a962700 10 _do_op 0x160653e0 seq 2483 r = 0, finisher 0x2f2cbc0 0
2014-06-12 14:47:08.383093 7f9d7a962700 10 _finish_op 0x160653e0 seq 2483 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:08.383101 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:08.383106 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.383111 7f9d7715b700  5 -- op tracker -- , seq: 748300, time: 2014-06-12 14:47:08.383100, event: commit_sent, op: osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.383122 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2673 (0'0,173'2673] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2671 active] sub_op_modify_commit on op osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:08.383155 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:08.383236 7f9d7795c700  5 -- op tracker -- , seq: 748300, time: 2014-06-12 14:47:08.383229, event: sub_op_applied, op: osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.383247 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2673 (0'0,173'2673] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2672 active] sub_op_modify_applied on 0x61e6000 op osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:08.383263 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2673 (0'0,173'2673] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2672 active] op_applied on primary on version 173'2673
2014-06-12 14:47:08.383272 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2673 (0'0,173'2673] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2672 active] op_applied on replica on version 173'2673
2014-06-12 14:47:08.383316 7f9d7795c700  5 -- op tracker -- , seq: 748300, time: 2014-06-12 14:47:08.383311, event: done, op: osd_sub_op(client.4282.0:145227 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2673 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.482146 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:08.482178 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:08.482185 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:08.482189 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:08.482189 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:08.482193 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 26; oldest is 28.173750 seconds old
2014-06-12 14:47:08.545238 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.559617 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.568920 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.572206 7f9d3e9a5700  5 -- op tracker -- , seq: 748301, time: 2014-06-12 14:47:08.572134, event: header_read, op: osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.572221 7f9d3e9a5700  5 -- op tracker -- , seq: 748301, time: 2014-06-12 14:47:08.572137, event: throttled, op: osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.572231 7f9d3e9a5700  5 -- op tracker -- , seq: 748301, time: 2014-06-12 14:47:08.572174, event: all_read, op: osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.572241 7f9d3e9a5700  5 -- op tracker -- , seq: 748301, time: 0.000000, event: dispatched, op: osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.572249 7f9d3e9a5700 20 osd.2 173 should_share_map client.4299 172.16.96.220:0/1243711 173
2014-06-12 14:47:08.572278 7f9d3e9a5700 15 osd.2 173 enqueue_op 0xd38fe00 prio 63 cost 24576 latency 0.000144 osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.572304 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.572311 7f9d61bd4700 10 osd.2 173 dequeue_op 0xd38fe00 prio 63 cost 24576 latency 0.000177 osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean]
2014-06-12 14:47:08.572347 7f9d61bd4700  5 -- op tracker -- , seq: 748301, time: 2014-06-12 14:47:08.572340, event: reached_pg, op: osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.572360 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:08.572379 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] handle_message: 0xd38fe00
2014-06-12 14:47:08.572392 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] do_op osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:08.572422 7f9d61bd4700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_'
2014-06-12 14:47:08.576338 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.576505 7f9d56432700  5 -- op tracker -- , seq: 748302, time: 2014-06-12 14:47:08.575575, event: header_read, op: osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.576518 7f9d56432700  5 -- op tracker -- , seq: 748302, time: 2014-06-12 14:47:08.575578, event: throttled, op: osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.576528 7f9d56432700  5 -- op tracker -- , seq: 748302, time: 2014-06-12 14:47:08.576474, event: all_read, op: osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.576547 7f9d56432700  5 -- op tracker -- , seq: 748302, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.576553 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:08.576563 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:08.576572 7f9d56432700 15 osd.2 173 enqueue_op 0x25e43000 prio 127 cost 492327 latency 0.000997 osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:08.576626 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:08.576655 7f9d63bd8700 10 osd.2 173 dequeue_op 0x25e43000 prio 127 cost 492327 latency 0.001080 osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1c7( v 173'2654 (0'0,173'2654] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2653 active]
2014-06-12 14:47:08.576694 7f9d63bd8700  5 -- op tracker -- , seq: 748302, time: 2014-06-12 14:47:08.576687, event: reached_pg, op: osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.576711 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2654 (0'0,173'2654] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2653 active] handle_message: 0x25e43000
2014-06-12 14:47:08.576727 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2654 (0'0,173'2654] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2653 active] sub_op_modify trans 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 v 173'2655 (transaction) 174
2014-06-12 14:47:08.576749 7f9d63bd8700  5 -- op tracker -- , seq: 748302, time: 2014-06-12 14:47:08.576744, event: started, op: osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.576771 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2654 (0'0,173'2654] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2653 active] append_log log((0'0,173'2654], crt=0'0) [173'2655 (173'2652) modify   6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 by client.4299.0:145542 2014-06-12 08:46:10.951988]
2014-06-12 14:47:08.576804 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2655 (0'0,173'2655] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2653 active] add_log_entry 173'2655 (173'2652) modify   6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 by client.4299.0:145542 2014-06-12 08:46:10.951988
2014-06-12 14:47:08.576839 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2655 (0'0,173'2655] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2653 active] append_log  adding 1 keys
2014-06-12 14:47:08.576886 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2655, trimmed: 
2014-06-12 14:47:08.576921 7f9d63bd8700  5 -- op tracker -- , seq: 748302, time: 2014-06-12 14:47:08.576914, event: started, op: osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.576935 7f9d63bd8700  5 queue_transactions existing osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:08.576941 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.576943 7f9d63bd8700  5 queue_op 0x9af5380 seq 2475 osr(3.1c7 0x276d170) 493813 bytes   (queue has 1 ops and 493813 bytes)
2014-06-12 14:47:08.576951 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2655 (0'0,173'2655] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2653 active] do_sub_op osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:08.576972 7f9d63bd8700 10 osd.2 173 dequeue_op 0x25e43000 finish
2014-06-12 14:47:08.576977 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:08.576960 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5518ea0 (1 active)
2014-06-12 14:47:08.576980 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.576984 7f9d83173700  5 _do_op 0x9af5380 seq 2475 osr(3.1c7 0x276d170)/0x276d170 start
2014-06-12 14:47:08.576995 7f9d83173700 10 _do_transaction on 0x247fc9b0
2014-06-12 14:47:08.577008 7f9d83173700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:08.577208 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:08.577234 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.577273 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.577297 7f9d83173700 15 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:08.577305 7f9d83173700 10 _omap_rmkeys meta/1fe17055/pglog_3.1c7/0//-1 r = 0
2014-06-12 14:47:08.577311 7f9d83173700 15 _omap_setkeys meta/1fe17055/pglog_3.1c7/0//-1
2014-06-12 14:47:08.577453 7f9d83173700 15 _write 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 3702784~491520
2014-06-12 14:47:08.577569 7f9d83173700 10 lookup_strip_header done  cid 3.1c7_head oid 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3
2014-06-12 14:47:08.577681 7f9d83173700 10 file_to_extents done 
2014-06-12 14:47:08.579943 7f9d83173700 10 _generic_write 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 3702784~491520 = 0
2014-06-12 14:47:08.579998 7f9d83173700 15 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3
2014-06-12 14:47:08.580016 7f9d83173700 10 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 = 0
2014-06-12 14:47:08.580024 7f9d83173700 15 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3
2014-06-12 14:47:08.580034 7f9d83173700 10 _setattrs 3.1c7_head/6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 = 0
2014-06-12 14:47:08.580203 7f9d83173700 20 set_header setting 17711 cid 3.1c7_head oid 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 parent seq 0
2014-06-12 14:47:08.580274 7f9d83173700 20 set_header setting 2336 cid meta oid 1fe17055/pglog_3.1c7/0//-1 parent seq 0
2014-06-12 14:47:08.580294 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.580307 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:47:08.582674 7f9d83173700 10 _do_op 0x9af5380 seq 2475 r = 0, finisher 0x7759800 0
2014-06-12 14:47:08.582695 7f9d83173700 10 _finish_op 0x9af5380 seq 2475 osr(3.1c7 0x276d170)/0x276d170
2014-06-12 14:47:08.582716 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5518ea0 (0 active)
2014-06-12 14:47:08.582713 7f9d7715b700  5 -- op tracker -- , seq: 748302, time: 2014-06-12 14:47:08.582703, event: commit_sent, op: osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.582720 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.582722 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2655 (0'0,173'2655] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2653 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.26
2014-06-12 14:47:08.582745 7f9d7715b700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:08.582776 7f9d7795c700  5 -- op tracker -- , seq: 748302, time: 2014-06-12 14:47:08.582770, event: sub_op_applied, op: osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.582785 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2655 (0'0,173'2655] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2654 active] sub_op_modify_applied on 0x247fc8c0 op osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:08.582807 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2655 (0'0,173'2655] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2654 active] op_applied on primary on version 173'2655
2014-06-12 14:47:08.582834 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1c7( v 173'2655 (0'0,173'2655] local-les=153 n=298 ec=23 les/c 153/153 152/152/152) [26,2] r=1 lpr=152 pi=23-151/12 luod=0'0 crt=0'0 lcod 173'2654 active] op_applied on replica on version 173'2655
2014-06-12 14:47:08.582863 7f9d7795c700  5 -- op tracker -- , seq: 748302, time: 2014-06-12 14:47:08.582857, event: done, op: osd_sub_op(client.4299.0:145542 3.1c7 6bf409c7/rbd_data.10a26b8b4567.0000000000000f71/head//3 [] v 173'2655 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.616370 7f9d61bd4700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_' = 0
2014-06-12 14:47:08.616407 7f9d61bd4700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset'
2014-06-12 14:47:08.616549 7f9d61bd4700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset' = 0
2014-06-12 14:47:08.616564 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] populate_obc_watchers 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.616584 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.616596 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] get_object_context: creating obc from disk: 0x16962840
2014-06-12 14:47:08.616606 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] get_object_context: 0x16962840 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 rwstate(none n=0 w=0) oi: 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'880 client.4299.0:37592 wrlock_by=unknown.0.0:0 dirty s 4194304 uv880) ssc: 0x24d4bc00 snapset: 0=[]:[]+head
2014-06-12 14:47:08.616624 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] find_object_context 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 @head oi=6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'880 client.4299.0:37592 wrlock_by=unknown.0.0:0 dirty s 4194304 uv880)
2014-06-12 14:47:08.616659 7f9d61bd4700  5 -- op tracker -- , seq: 748301, time: 2014-06-12 14:47:08.616649, event: started, op: osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.616726 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] execute_ctx 0x12201600
2014-06-12 14:47:08.616770 7f9d61bd4700  5 -- op tracker -- , seq: 748301, time: 2014-06-12 14:47:08.616760, event: started, op: osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.616786 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] do_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] ov 173'880 av 173'3037 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:08.616803 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] do_osd_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576]
2014-06-12 14:47:08.616816 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:08.616831 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] do_osd_op  write 0~24576
2014-06-12 14:47:08.616850 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] do_osd_op_effects on session 0x6231800
2014-06-12 14:47:08.616861 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 snapset=0x24d4bc38  snapc=0=[]
2014-06-12 14:47:08.616874 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:08.616885 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] finish_ctx 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 0x12201600 op modify  
2014-06-12 14:47:08.616898 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean]  set mtime to 2014-06-12 08:46:10.952126
2014-06-12 14:47:08.616929 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean]  final snapset 0=[]:[]+head in 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.616949 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean]  zeroing write result code 0
2014-06-12 14:47:08.616960 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] calc_trim_to 79'35 -> 79'36
2014-06-12 14:47:08.616971 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] new_repop rep_tid 236395 on osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.616990 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] issue_repop rep_tid 236395 o 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.617016 7f9d61bd4700  5 -- op tracker -- , seq: 748301, time: 2014-06-12 14:47:08.617011, event: waiting for subops from 22, op: osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.617047 7f9d61bd4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:08.617075 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3036 (79'35,173'3036] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] append_log log((79'35,173'3036], crt=0'0) [173'3037 (173'880) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145543 2014-06-12 08:46:10.952126]
2014-06-12 14:47:08.617106 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'35,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3036 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] add_log_entry 173'3037 (173'880) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145543 2014-06-12 08:46:10.952126
2014-06-12 14:47:08.617134 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'35,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3036 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] append_log  adding 1 keys
2014-06-12 14:47:08.617148 7f9d61bd4700 10 trim log((79'35,173'3037], crt=0'0) to 79'36
2014-06-12 14:47:08.617176 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3037, trimmed: 79'36
2014-06-12 14:47:08.617201 7f9d61bd4700  5 queue_transactions existing osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.617208 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.617210 7f9d61bd4700  5 queue_op 0x893ba40 seq 2786 osr(3.783 0x49b40f0) 26904 bytes   (queue has 1 ops and 26904 bytes)
2014-06-12 14:47:08.617228 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d9e0 (1 active)
2014-06-12 14:47:08.617219 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3036 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] eval_repop repgather(0x1696c780 173'3037 rep_tid=236395 committed?=0 applied?=0 lock=0 op=osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.617242 7f9d61bd4700 10 osd.2 173 dequeue_op 0xd38fe00 finish
2014-06-12 14:47:08.617241 7f9d7a161700  5 _do_op 0x893ba40 seq 2786 osr(3.783 0x49b40f0)/0x49b40f0 start
2014-06-12 14:47:08.617247 7f9d7a161700 10 _do_transaction on 0x514d740
2014-06-12 14:47:08.617247 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.617251 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.617256 7f9d7a161700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.617370 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.617403 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.617469 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.617508 7f9d7a161700 15 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.617523 7f9d7a161700 10 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1 r = 0
2014-06-12 14:47:08.617532 7f9d7a161700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.617776 7f9d7a161700 15 _write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 0~24576
2014-06-12 14:47:08.618009 7f9d7a161700 10 lookup_strip_header done  cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.618041 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:47:08.618298 7f9d7a161700 10 _generic_write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 0~24576 = 0
2014-06-12 14:47:08.618319 7f9d7a161700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.618332 7f9d7a161700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.618339 7f9d7a161700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.618347 7f9d7a161700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.618487 7f9d7a161700 20 set_header setting 17712 cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 parent seq 0
2014-06-12 14:47:08.618528 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.618560 7f9d7a161700 20 set_header setting 2140 cid meta oid 1fe0ae1a/pglog_3.783/0//-1 parent seq 0
2014-06-12 14:47:08.618584 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:47:08.618867 7f9d7a161700 10 _do_op 0x893ba40 seq 2786 r = 0, finisher 0x682a7c0 0x397b5c0
2014-06-12 14:47:08.618886 7f9d7a161700 10 _finish_op 0x893ba40 seq 2786 osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.618896 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d9e0 (0 active)
2014-06-12 14:47:08.618900 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.618891 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3036 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] op_commit: 236395
2014-06-12 14:47:08.618931 7f9d7715b700  5 -- op tracker -- , seq: 748301, time: 2014-06-12 14:47:08.618920, event: op_commit, op: osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.618977 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3036 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] op_applied: 236395
2014-06-12 14:47:08.619022 7f9d7795c700  5 -- op tracker -- , seq: 748301, time: 2014-06-12 14:47:08.619010, event: op_applied, op: osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.619041 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3036 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] op_applied on primary on version 173'3037
2014-06-12 14:47:08.620403 7f9d58d5b700  5 -- op tracker -- , seq: 748303, time: 2014-06-12 14:47:08.620353, event: header_read, op: osd_sub_op_reply(client.4299.0:145543 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.620416 7f9d58d5b700  5 -- op tracker -- , seq: 748303, time: 2014-06-12 14:47:08.620356, event: throttled, op: osd_sub_op_reply(client.4299.0:145543 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.620424 7f9d58d5b700  5 -- op tracker -- , seq: 748303, time: 2014-06-12 14:47:08.620369, event: all_read, op: osd_sub_op_reply(client.4299.0:145543 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.620432 7f9d58d5b700  5 -- op tracker -- , seq: 748303, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4299.0:145543 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.620437 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4299.0:145543 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:08.620443 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:08.620451 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34017200 prio 196 cost 0 latency 0.000097 osd_sub_op_reply(client.4299.0:145543 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:08.620482 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.620494 7f9d633d7700 10 osd.2 173 dequeue_op 0x34017200 prio 196 cost 0 latency 0.000141 osd_sub_op_reply(client.4299.0:145543 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 pg pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3036 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean]
2014-06-12 14:47:08.620527 7f9d633d7700  5 -- op tracker -- , seq: 748303, time: 2014-06-12 14:47:08.620521, event: reached_pg, op: osd_sub_op_reply(client.4299.0:145543 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.620540 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3036 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] handle_message: 0x34017200
2014-06-12 14:47:08.620568 7f9d633d7700  5 -- op tracker -- , seq: 748303, time: 2014-06-12 14:47:08.620561, event: started, op: osd_sub_op_reply(client.4299.0:145543 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.620590 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3036 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] sub_op_modify_reply: tid 236395 op  ack_type 4 from 22
2014-06-12 14:47:08.620627 7f9d633d7700  5 -- op tracker -- , seq: 748301, time: 2014-06-12 14:47:08.620616, event: sub_op_commit_rec, op: osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.620656 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3036 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] repop_all_applied: repop tid 236395 all applied 
2014-06-12 14:47:08.620679 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3036 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] eval_repop repgather(0x1696c780 173'3037 rep_tid=236395 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.620711 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3036 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean]  sending ack on repgather(0x1696c780 173'3037 rep_tid=236395 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x28045180
2014-06-12 14:47:08.620768 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3036 crt=0'0 lcod 173'3035 mlcod 173'3035 active+clean] repop_all_committed: repop tid 236395 all committed 
2014-06-12 14:47:08.620790 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3035 active+clean] eval_repop repgather(0x1696c780 173'3037 rep_tid=236395 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.620823 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3035 active+clean] log_op_stats osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 24855 outb 0 rlat 0.048632 lat 0.048687
2014-06-12 14:47:08.620851 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3035 active+clean] publish_stats_to_osd 173:4039
2014-06-12 14:47:08.620872 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3035 active+clean]  sending commit on repgather(0x1696c780 173'3037 rep_tid=236395 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x25edca00
2014-06-12 14:47:08.620928 7f9d633d7700  5 -- op tracker -- , seq: 748301, time: 2014-06-12 14:47:08.620919, event: commit_sent, op: osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.620943 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3035 active+clean]  requeue_ops 
2014-06-12 14:47:08.620962 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean]  removing repgather(0x1696c780 173'3037 rep_tid=236395 committed?=1 applied?=1 op=osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.621011 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean]    q front is repgather(0x1696c780 173'3037 rep_tid=236395 committed?=1 applied?=1 op=osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.621041 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] remove_repop repgather(0x1696c780 173'3037 rep_tid=236395 committed?=1 applied?=1 op=osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.621069 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean]  requeue_ops 
2014-06-12 14:47:08.621113 7f9d633d7700  5 -- op tracker -- , seq: 748301, time: 2014-06-12 14:47:08.621103, event: done, op: osd_op(client.4299.0:145543 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.621154 7f9d633d7700 10 osd.2 173 dequeue_op 0x34017200 finish
2014-06-12 14:47:08.621171 7f9d633d7700  5 -- op tracker -- , seq: 748303, time: 2014-06-12 14:47:08.621164, event: done, op: osd_sub_op_reply(client.4299.0:145543 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.621192 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.621197 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.674168 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:08.674251 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:08.708786 7f9d3e9a5700  5 -- op tracker -- , seq: 748304, time: 2014-06-12 14:47:08.707271, event: header_read, op: osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.708806 7f9d3e9a5700  5 -- op tracker -- , seq: 748304, time: 2014-06-12 14:47:08.707274, event: throttled, op: osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.708818 7f9d3e9a5700  5 -- op tracker -- , seq: 748304, time: 2014-06-12 14:47:08.708732, event: all_read, op: osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.708830 7f9d3e9a5700  5 -- op tracker -- , seq: 748304, time: 0.000000, event: dispatched, op: osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.708839 7f9d3e9a5700 20 osd.2 173 should_share_map client.4299 172.16.96.220:0/1243711 173
2014-06-12 14:47:08.708877 7f9d3e9a5700 15 osd.2 173 enqueue_op 0xd38ea00 prio 63 cost 516096 latency 0.001605 osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.708950 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.708970 7f9d67be0700 10 osd.2 173 dequeue_op 0xd38ea00 prio 63 cost 516096 latency 0.001699 osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean]
2014-06-12 14:47:08.709012 7f9d67be0700  5 -- op tracker -- , seq: 748304, time: 2014-06-12 14:47:08.709004, event: reached_pg, op: osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.709029 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:08.709048 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] handle_message: 0xd38ea00
2014-06-12 14:47:08.709062 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] do_op osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:08.709095 7f9d67be0700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_'
2014-06-12 14:47:08.709285 7f9d67be0700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_' = 0
2014-06-12 14:47:08.709312 7f9d67be0700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset'
2014-06-12 14:47:08.709408 7f9d67be0700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset' = 0
2014-06-12 14:47:08.709423 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] populate_obc_watchers 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.709440 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.709453 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] get_object_context: creating obc from disk: 0x281278c0
2014-06-12 14:47:08.709464 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] get_object_context: 0x281278c0 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 rwstate(none n=0 w=0) oi: 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3037 client.4299.0:145543 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3037) ssc: 0x5d5a380 snapset: 0=[]:[]+head
2014-06-12 14:47:08.709485 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] find_object_context 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 @head oi=6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3037 client.4299.0:145543 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3037)
2014-06-12 14:47:08.709518 7f9d67be0700  5 -- op tracker -- , seq: 748304, time: 2014-06-12 14:47:08.709510, event: started, op: osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.709533 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] execute_ctx 0x2b883e00
2014-06-12 14:47:08.709556 7f9d67be0700  5 -- op tracker -- , seq: 748304, time: 2014-06-12 14:47:08.709550, event: started, op: osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.709564 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] do_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] ov 173'3037 av 173'3038 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:08.709582 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] do_osd_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096]
2014-06-12 14:47:08.709595 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:08.709638 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] do_osd_op  write 24576~516096
2014-06-12 14:47:08.709661 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] do_osd_op_effects on session 0x6231800
2014-06-12 14:47:08.709675 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 snapset=0x5d5a3b8  snapc=0=[]
2014-06-12 14:47:08.709690 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:08.709703 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] finish_ctx 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 0x2b883e00 op modify  
2014-06-12 14:47:08.709718 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean]  set mtime to 2014-06-12 08:46:11.086311
2014-06-12 14:47:08.709751 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean]  final snapset 0=[]:[]+head in 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.709781 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean]  zeroing write result code 0
2014-06-12 14:47:08.709794 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] calc_trim_to 79'36 -> 79'37
2014-06-12 14:47:08.709806 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] new_repop rep_tid 236396 on osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.709826 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] issue_repop rep_tid 236396 o 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.709855 7f9d67be0700  5 -- op tracker -- , seq: 748304, time: 2014-06-12 14:47:08.709849, event: waiting for subops from 22, op: osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.709885 7f9d67be0700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:08.709913 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3037 (79'36,173'3037] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] append_log log((79'36,173'3037], crt=0'0) [173'3038 (173'3037) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145544 2014-06-12 08:46:11.086311]
2014-06-12 14:47:08.709953 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'36,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3037 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] add_log_entry 173'3038 (173'3037) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145544 2014-06-12 08:46:11.086311
2014-06-12 14:47:08.709983 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'36,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3037 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] append_log  adding 1 keys
2014-06-12 14:47:08.710007 7f9d67be0700 10 trim log((79'36,173'3038], crt=0'0) to 79'37
2014-06-12 14:47:08.710042 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3038, trimmed: 79'37
2014-06-12 14:47:08.710074 7f9d67be0700  5 queue_transactions existing osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.710080 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.710082 7f9d67be0700  5 queue_op 0x1415e7e0 seq 2787 osr(3.783 0x49b40f0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:08.710092 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3037 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] eval_repop repgather(0x234c3c0 173'3038 rep_tid=236396 committed?=0 applied?=0 lock=0 op=osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.710117 7f9d67be0700 10 osd.2 173 dequeue_op 0xd38ea00 finish
2014-06-12 14:47:08.710123 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.710126 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.710115 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d9e0 (1 active)
2014-06-12 14:47:08.710134 7f9d7f16b700  5 _do_op 0x1415e7e0 seq 2787 osr(3.783 0x49b40f0)/0x49b40f0 start
2014-06-12 14:47:08.710144 7f9d7f16b700 10 _do_transaction on 0x13b42240
2014-06-12 14:47:08.710159 7f9d7f16b700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.710301 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.710364 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.710453 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.710750 7f9d7f16b700 15 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.710775 7f9d7f16b700 10 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1 r = 0
2014-06-12 14:47:08.710809 7f9d7f16b700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.710865 7f9d7f16b700 15 _write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 24576~516096
2014-06-12 14:47:08.711039 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:08.711140 7f9d7f16b700 10 lookup_strip_header done  cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.711341 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:08.711850 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:47:08.714793 7f9d7f16b700 10 _generic_write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 24576~516096 = 0
2014-06-12 14:47:08.714896 7f9d7f16b700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.714941 7f9d7f16b700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.714962 7f9d7f16b700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.714984 7f9d7f16b700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.715301 7f9d7f16b700 20 set_header setting 17712 cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 parent seq 0
2014-06-12 14:47:08.715434 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.715474 7f9d7f16b700 20 set_header setting 2140 cid meta oid 1fe0ae1a/pglog_3.783/0//-1 parent seq 0
2014-06-12 14:47:08.715511 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:47:08.717549 7f9d58d5b700  5 -- op tracker -- , seq: 748305, time: 2014-06-12 14:47:08.717451, event: header_read, op: osd_sub_op_reply(client.4299.0:145544 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.717576 7f9d58d5b700  5 -- op tracker -- , seq: 748305, time: 2014-06-12 14:47:08.717455, event: throttled, op: osd_sub_op_reply(client.4299.0:145544 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.717604 7f9d58d5b700  5 -- op tracker -- , seq: 748305, time: 2014-06-12 14:47:08.717488, event: all_read, op: osd_sub_op_reply(client.4299.0:145544 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.717640 7f9d58d5b700  5 -- op tracker -- , seq: 748305, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4299.0:145544 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.717650 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4299.0:145544 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:08.717661 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:08.717674 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016e00 prio 196 cost 0 latency 0.000223 osd_sub_op_reply(client.4299.0:145544 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:08.717999 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.718009 7f9d673df700 10 osd.2 173 dequeue_op 0x34016e00 prio 196 cost 0 latency 0.000558 osd_sub_op_reply(client.4299.0:145544 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 pg pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3037 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean]
2014-06-12 14:47:08.718033 7f9d673df700  5 -- op tracker -- , seq: 748305, time: 2014-06-12 14:47:08.718030, event: reached_pg, op: osd_sub_op_reply(client.4299.0:145544 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.718041 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3037 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] handle_message: 0x34016e00
2014-06-12 14:47:08.718058 7f9d673df700  5 -- op tracker -- , seq: 748305, time: 2014-06-12 14:47:08.718055, event: started, op: osd_sub_op_reply(client.4299.0:145544 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.718063 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3037 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] sub_op_modify_reply: tid 236396 op  ack_type 4 from 22
2014-06-12 14:47:08.718083 7f9d673df700  5 -- op tracker -- , seq: 748304, time: 2014-06-12 14:47:08.718075, event: sub_op_commit_rec, op: osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.718090 7f9d673df700 10 osd.2 173 dequeue_op 0x34016e00 finish
2014-06-12 14:47:08.718096 7f9d673df700  5 -- op tracker -- , seq: 748305, time: 2014-06-12 14:47:08.718093, event: done, op: osd_sub_op_reply(client.4299.0:145544 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.718112 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.718115 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.719347 7f9d7f16b700 10 _do_op 0x1415e7e0 seq 2787 r = 0, finisher 0x662dba0 0x397b580
2014-06-12 14:47:08.719379 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3037 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] op_commit: 236396
2014-06-12 14:47:08.719410 7f9d7715b700  5 -- op tracker -- , seq: 748304, time: 2014-06-12 14:47:08.719398, event: op_commit, op: osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.719421 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3037 crt=0'0 lcod 173'3036 mlcod 173'3036 active+clean] repop_all_committed: repop tid 236396 all committed 
2014-06-12 14:47:08.719435 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3036 active+clean] eval_repop repgather(0x234c3c0 173'3038 rep_tid=236396 committed?=1 applied?=0 lock=0 op=osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.719456 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3036 active+clean] log_op_stats osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.012184
2014-06-12 14:47:08.719476 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3036 active+clean] publish_stats_to_osd 173:4040
2014-06-12 14:47:08.719489 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3036 active+clean]  sending commit on repgather(0x234c3c0 173'3038 rep_tid=236396 committed?=1 applied?=0 lock=0 op=osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x1ae19900
2014-06-12 14:47:08.719531 7f9d7715b700  5 -- op tracker -- , seq: 748304, time: 2014-06-12 14:47:08.719525, event: commit_sent, op: osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.719545 7f9d7f16b700 10 _finish_op 0x1415e7e0 seq 2787 osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.719555 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d9e0 (0 active)
2014-06-12 14:47:08.719566 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.719562 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3036 active+clean] op_applied: 236396
2014-06-12 14:47:08.719588 7f9d7795c700  5 -- op tracker -- , seq: 748304, time: 2014-06-12 14:47:08.719581, event: op_applied, op: osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.719597 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3036 active+clean] op_applied on primary on version 173'3038
2014-06-12 14:47:08.719607 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3036 active+clean] repop_all_applied: repop tid 236396 all applied 
2014-06-12 14:47:08.719616 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3036 active+clean] eval_repop repgather(0x234c3c0 173'3038 rep_tid=236396 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.719630 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3036 active+clean] log_op_stats osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.012358
2014-06-12 14:47:08.719647 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3036 active+clean] publish_stats_to_osd 173:4041
2014-06-12 14:47:08.719658 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3036 active+clean]  requeue_ops 
2014-06-12 14:47:08.719666 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean]  removing repgather(0x234c3c0 173'3038 rep_tid=236396 committed?=1 applied?=1 op=osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.719679 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean]    q front is repgather(0x234c3c0 173'3038 rep_tid=236396 committed?=1 applied?=1 op=osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.719692 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] remove_repop repgather(0x234c3c0 173'3038 rep_tid=236396 committed?=1 applied?=1 op=osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.719705 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean]  requeue_ops 
2014-06-12 14:47:08.719732 7f9d7795c700  5 -- op tracker -- , seq: 748304, time: 2014-06-12 14:47:08.719727, event: done, op: osd_op(client.4299.0:145544 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.720708 7f9d3e9a5700  5 -- op tracker -- , seq: 748306, time: 2014-06-12 14:47:08.720647, event: header_read, op: osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.720723 7f9d3e9a5700  5 -- op tracker -- , seq: 748306, time: 2014-06-12 14:47:08.720651, event: throttled, op: osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.720731 7f9d3e9a5700  5 -- op tracker -- , seq: 748306, time: 2014-06-12 14:47:08.720672, event: all_read, op: osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.720752 7f9d3e9a5700  5 -- op tracker -- , seq: 748306, time: 0.000000, event: dispatched, op: osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.720767 7f9d3e9a5700 20 osd.2 173 should_share_map client.4299 172.16.96.220:0/1243711 173
2014-06-12 14:47:08.720794 7f9d3e9a5700 15 osd.2 173 enqueue_op 0xd38e300 prio 63 cost 4096 latency 0.000147 osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.720842 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.720848 7f9d663dd700 10 osd.2 173 dequeue_op 0xd38e300 prio 63 cost 4096 latency 0.000201 osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean]
2014-06-12 14:47:08.720867 7f9d663dd700  5 -- op tracker -- , seq: 748306, time: 2014-06-12 14:47:08.720863, event: reached_pg, op: osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.720874 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:08.720886 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] handle_message: 0xd38e300
2014-06-12 14:47:08.720894 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] do_op osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:08.720922 7f9d663dd700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_'
2014-06-12 14:47:08.721069 7f9d663dd700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_' = 0
2014-06-12 14:47:08.721086 7f9d663dd700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset'
2014-06-12 14:47:08.721137 7f9d663dd700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset' = 0
2014-06-12 14:47:08.721147 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] populate_obc_watchers 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.721156 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.721162 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] get_object_context: creating obc from disk: 0x1703cb00
2014-06-12 14:47:08.721167 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] get_object_context: 0x1703cb00 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 rwstate(none n=0 w=0) oi: 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3038 client.4299.0:145544 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3038) ssc: 0x16ee41c0 snapset: 0=[]:[]+head
2014-06-12 14:47:08.721179 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] find_object_context 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 @head oi=6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3038 client.4299.0:145544 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3038)
2014-06-12 14:47:08.721201 7f9d663dd700  5 -- op tracker -- , seq: 748306, time: 2014-06-12 14:47:08.721197, event: started, op: osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.721207 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] execute_ctx 0x30f7de00
2014-06-12 14:47:08.721218 7f9d663dd700  5 -- op tracker -- , seq: 748306, time: 2014-06-12 14:47:08.721215, event: started, op: osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.721222 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] do_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] ov 173'3038 av 173'3039 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:08.721231 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] do_osd_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096]
2014-06-12 14:47:08.721237 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:08.721246 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] do_osd_op  write 540672~4096
2014-06-12 14:47:08.721258 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] do_osd_op_effects on session 0x6231800
2014-06-12 14:47:08.721265 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 snapset=0x16ee41f8  snapc=0=[]
2014-06-12 14:47:08.721272 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:08.721278 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] finish_ctx 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 0x30f7de00 op modify  
2014-06-12 14:47:08.721285 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean]  set mtime to 2014-06-12 08:46:11.100687
2014-06-12 14:47:08.721308 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean]  final snapset 0=[]:[]+head in 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.721323 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean]  zeroing write result code 0
2014-06-12 14:47:08.721329 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] calc_trim_to 79'37 -> 79'38
2014-06-12 14:47:08.721335 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] new_repop rep_tid 236397 on osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.721346 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] issue_repop rep_tid 236397 o 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.721362 7f9d663dd700  5 -- op tracker -- , seq: 748306, time: 2014-06-12 14:47:08.721359, event: waiting for subops from 22, op: osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.721378 7f9d663dd700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:08.721395 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3038 (79'37,173'3038] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] append_log log((79'37,173'3038], crt=0'0) [173'3039 (173'3038) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145545 2014-06-12 08:46:11.100687]
2014-06-12 14:47:08.721421 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'37,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3038 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] add_log_entry 173'3039 (173'3038) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145545 2014-06-12 08:46:11.100687
2014-06-12 14:47:08.721451 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'37,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3038 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] append_log  adding 1 keys
2014-06-12 14:47:08.721469 7f9d663dd700 10 trim log((79'37,173'3039], crt=0'0) to 79'38
2014-06-12 14:47:08.721493 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3039, trimmed: 79'38
2014-06-12 14:47:08.721514 7f9d663dd700  5 queue_transactions existing osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.721519 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.721520 7f9d663dd700  5 queue_op 0x68998c0 seq 2788 osr(3.783 0x49b40f0) 6424 bytes   (queue has 1 ops and 6424 bytes)
2014-06-12 14:47:08.721530 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d9e0 (1 active)
2014-06-12 14:47:08.721537 7f9d7e169700  5 _do_op 0x68998c0 seq 2788 osr(3.783 0x49b40f0)/0x49b40f0 start
2014-06-12 14:47:08.721525 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3038 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] eval_repop repgather(0x523db00 173'3039 rep_tid=236397 committed?=0 applied?=0 lock=0 op=osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.721541 7f9d7e169700 10 _do_transaction on 0x2a32db00
2014-06-12 14:47:08.721543 7f9d663dd700 10 osd.2 173 dequeue_op 0xd38e300 finish
2014-06-12 14:47:08.721547 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.721547 7f9d7e169700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.721550 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.721717 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.721761 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.721809 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.721845 7f9d7e169700 15 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.721854 7f9d7e169700 10 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1 r = 0
2014-06-12 14:47:08.721861 7f9d7e169700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.721881 7f9d7e169700 15 _write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 540672~4096
2014-06-12 14:47:08.722030 7f9d7e169700 10 lookup_strip_header done  cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.722190 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:47:08.722215 7f9d7e169700 10 _generic_write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 540672~4096 = 0
2014-06-12 14:47:08.722224 7f9d7e169700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.722232 7f9d7e169700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.722236 7f9d7e169700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.722241 7f9d7e169700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.722341 7f9d7e169700 20 set_header setting 17712 cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 parent seq 0
2014-06-12 14:47:08.722363 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.722375 7f9d7e169700 20 set_header setting 2140 cid meta oid 1fe0ae1a/pglog_3.783/0//-1 parent seq 0
2014-06-12 14:47:08.722386 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:47:08.722461 7f9d7e169700 10 _do_op 0x68998c0 seq 2788 r = 0, finisher 0x9fdff00 0x218d1680
2014-06-12 14:47:08.722471 7f9d7e169700 10 _finish_op 0x68998c0 seq 2788 osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.722479 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d9e0 (0 active)
2014-06-12 14:47:08.722482 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.722484 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3038 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] op_applied: 236397
2014-06-12 14:47:08.722510 7f9d7795c700  5 -- op tracker -- , seq: 748306, time: 2014-06-12 14:47:08.722502, event: op_applied, op: osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.722518 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3038 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] op_applied on primary on version 173'3039
2014-06-12 14:47:08.723057 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3038 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] op_commit: 236397
2014-06-12 14:47:08.723085 7f9d7715b700  5 -- op tracker -- , seq: 748306, time: 2014-06-12 14:47:08.723077, event: op_commit, op: osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.723419 7f9d58d5b700  5 -- op tracker -- , seq: 748307, time: 2014-06-12 14:47:08.723380, event: header_read, op: osd_sub_op_reply(client.4299.0:145545 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.723429 7f9d58d5b700  5 -- op tracker -- , seq: 748307, time: 2014-06-12 14:47:08.723382, event: throttled, op: osd_sub_op_reply(client.4299.0:145545 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.723437 7f9d58d5b700  5 -- op tracker -- , seq: 748307, time: 2014-06-12 14:47:08.723392, event: all_read, op: osd_sub_op_reply(client.4299.0:145545 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.723444 7f9d58d5b700  5 -- op tracker -- , seq: 748307, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4299.0:145545 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.723449 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4299.0:145545 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:08.723455 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:08.723463 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016f00 prio 196 cost 0 latency 0.000083 osd_sub_op_reply(client.4299.0:145545 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:08.723725 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.723735 7f9d65bdc700 10 osd.2 173 dequeue_op 0x34016f00 prio 196 cost 0 latency 0.000355 osd_sub_op_reply(client.4299.0:145545 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 pg pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3038 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean]
2014-06-12 14:47:08.723754 7f9d65bdc700  5 -- op tracker -- , seq: 748307, time: 2014-06-12 14:47:08.723751, event: reached_pg, op: osd_sub_op_reply(client.4299.0:145545 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.723763 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3038 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] handle_message: 0x34016f00
2014-06-12 14:47:08.723784 7f9d65bdc700  5 -- op tracker -- , seq: 748307, time: 2014-06-12 14:47:08.723781, event: started, op: osd_sub_op_reply(client.4299.0:145545 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.723790 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3038 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] sub_op_modify_reply: tid 236397 op  ack_type 4 from 22
2014-06-12 14:47:08.723805 7f9d65bdc700  5 -- op tracker -- , seq: 748306, time: 2014-06-12 14:47:08.723799, event: sub_op_commit_rec, op: osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.723810 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3038 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] repop_all_applied: repop tid 236397 all applied 
2014-06-12 14:47:08.723817 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3038 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] eval_repop repgather(0x523db00 173'3039 rep_tid=236397 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.723828 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3038 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean]  sending ack on repgather(0x523db00 173'3039 rep_tid=236397 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x2d88e280
2014-06-12 14:47:08.723853 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3038 crt=0'0 lcod 173'3037 mlcod 173'3037 active+clean] repop_all_committed: repop tid 236397 all committed 
2014-06-12 14:47:08.723859 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3037 active+clean] eval_repop repgather(0x523db00 173'3039 rep_tid=236397 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.723869 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3037 active+clean] log_op_stats osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 4375 outb 0 rlat 0.003205 lat 0.003221
2014-06-12 14:47:08.723881 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3037 active+clean] publish_stats_to_osd 173:4042
2014-06-12 14:47:08.723889 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3037 active+clean]  sending commit on repgather(0x523db00 173'3039 rep_tid=236397 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x30af7680
2014-06-12 14:47:08.723917 7f9d65bdc700  5 -- op tracker -- , seq: 748306, time: 2014-06-12 14:47:08.723911, event: commit_sent, op: osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.723925 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3037 active+clean]  requeue_ops 
2014-06-12 14:47:08.723935 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean]  removing repgather(0x523db00 173'3039 rep_tid=236397 committed?=1 applied?=1 op=osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.723948 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean]    q front is repgather(0x523db00 173'3039 rep_tid=236397 committed?=1 applied?=1 op=osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.723961 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] remove_repop repgather(0x523db00 173'3039 rep_tid=236397 committed?=1 applied?=1 op=osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.723974 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean]  requeue_ops 
2014-06-12 14:47:08.723999 7f9d65bdc700  5 -- op tracker -- , seq: 748306, time: 2014-06-12 14:47:08.723996, event: done, op: osd_op(client.4299.0:145545 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~4096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.724014 7f9d65bdc700 10 osd.2 173 dequeue_op 0x34016f00 finish
2014-06-12 14:47:08.724018 7f9d65bdc700  5 -- op tracker -- , seq: 748307, time: 2014-06-12 14:47:08.724016, event: done, op: osd_sub_op_reply(client.4299.0:145545 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.724024 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.724026 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.727763 7f9d3e9a5700  5 -- op tracker -- , seq: 748308, time: 2014-06-12 14:47:08.726112, event: header_read, op: osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.727795 7f9d3e9a5700  5 -- op tracker -- , seq: 748308, time: 2014-06-12 14:47:08.726115, event: throttled, op: osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.727832 7f9d3e9a5700  5 -- op tracker -- , seq: 748308, time: 2014-06-12 14:47:08.727694, event: all_read, op: osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.727859 7f9d3e9a5700  5 -- op tracker -- , seq: 748308, time: 0.000000, event: dispatched, op: osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.727878 7f9d3e9a5700 20 osd.2 173 should_share_map client.4299 172.16.96.220:0/1243711 173
2014-06-12 14:47:08.727907 7f9d3e9a5700 15 osd.2 173 enqueue_op 0xd38e100 prio 63 cost 516096 latency 0.001795 osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.727965 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.727981 7f9d62bd6700 10 osd.2 173 dequeue_op 0xd38e100 prio 63 cost 516096 latency 0.001869 osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean]
2014-06-12 14:47:08.728021 7f9d62bd6700  5 -- op tracker -- , seq: 748308, time: 2014-06-12 14:47:08.728012, event: reached_pg, op: osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.728037 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:08.728058 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] handle_message: 0xd38e100
2014-06-12 14:47:08.728075 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] do_op osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:08.728093 7f9d62bd6700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_'
2014-06-12 14:47:08.728281 7f9d62bd6700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_' = 0
2014-06-12 14:47:08.728307 7f9d62bd6700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset'
2014-06-12 14:47:08.728407 7f9d62bd6700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset' = 0
2014-06-12 14:47:08.728423 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] populate_obc_watchers 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.728442 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.728456 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] get_object_context: creating obc from disk: 0x6969b80
2014-06-12 14:47:08.728468 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] get_object_context: 0x6969b80 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 rwstate(none n=0 w=0) oi: 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3039 client.4299.0:145545 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3039) ssc: 0x1e15fa40 snapset: 0=[]:[]+head
2014-06-12 14:47:08.728490 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] find_object_context 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 @head oi=6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3039 client.4299.0:145545 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3039)
2014-06-12 14:47:08.728523 7f9d62bd6700  5 -- op tracker -- , seq: 748308, time: 2014-06-12 14:47:08.728519, event: started, op: osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.728530 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] execute_ctx 0xa04f200
2014-06-12 14:47:08.728541 7f9d62bd6700  5 -- op tracker -- , seq: 748308, time: 2014-06-12 14:47:08.728538, event: started, op: osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.728546 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] do_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] ov 173'3039 av 173'3040 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:08.728555 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] do_osd_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096]
2014-06-12 14:47:08.728563 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:08.728572 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] do_osd_op  write 544768~516096
2014-06-12 14:47:08.728582 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] do_osd_op_effects on session 0x6231800
2014-06-12 14:47:08.728589 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 snapset=0x1e15fa78  snapc=0=[]
2014-06-12 14:47:08.728596 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:08.728602 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] finish_ctx 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 0xa04f200 op modify  
2014-06-12 14:47:08.728611 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean]  set mtime to 2014-06-12 08:46:11.105193
2014-06-12 14:47:08.728625 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean]  final snapset 0=[]:[]+head in 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.728638 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean]  zeroing write result code 0
2014-06-12 14:47:08.728645 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] calc_trim_to 79'38 -> 79'39
2014-06-12 14:47:08.728651 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] new_repop rep_tid 236398 on osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.728662 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] issue_repop rep_tid 236398 o 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.728678 7f9d62bd6700  5 -- op tracker -- , seq: 748308, time: 2014-06-12 14:47:08.728675, event: waiting for subops from 22, op: osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.728693 7f9d62bd6700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:08.728708 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3039 (79'38,173'3039] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] append_log log((79'38,173'3039], crt=0'0) [173'3040 (173'3039) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145546 2014-06-12 08:46:11.105193]
2014-06-12 14:47:08.728725 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'38,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3039 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] add_log_entry 173'3040 (173'3039) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145546 2014-06-12 08:46:11.105193
2014-06-12 14:47:08.728740 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'38,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3039 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] append_log  adding 1 keys
2014-06-12 14:47:08.728751 7f9d62bd6700 10 trim log((79'38,173'3040], crt=0'0) to 79'39
2014-06-12 14:47:08.728769 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3040, trimmed: 79'39
2014-06-12 14:47:08.728787 7f9d62bd6700  5 queue_transactions existing osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.728791 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.728792 7f9d62bd6700  5 queue_op 0x4b2faa0 seq 2789 osr(3.783 0x49b40f0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:08.728800 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d9e0 (1 active)
2014-06-12 14:47:08.728796 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3039 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] eval_repop repgather(0x1c3789c0 173'3040 rep_tid=236398 committed?=0 applied?=0 lock=0 op=osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.728808 7f9d62bd6700 10 osd.2 173 dequeue_op 0xd38e100 finish
2014-06-12 14:47:08.728807 7f9d7815d700  5 _do_op 0x4b2faa0 seq 2789 osr(3.783 0x49b40f0)/0x49b40f0 start
2014-06-12 14:47:08.728810 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.728812 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.728812 7f9d7815d700 10 _do_transaction on 0x1c379980
2014-06-12 14:47:08.728818 7f9d7815d700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.728884 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.728906 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.728953 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.728974 7f9d7815d700 15 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.728984 7f9d7815d700 10 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1 r = 0
2014-06-12 14:47:08.728992 7f9d7815d700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.729006 7f9d7815d700 15 _write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 544768~516096
2014-06-12 14:47:08.729293 7f9d7815d700 10 lookup_strip_header done  cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.729352 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:47:08.731390 7f9d7815d700 10 _generic_write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 544768~516096 = 0
2014-06-12 14:47:08.731435 7f9d7815d700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.731449 7f9d7815d700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.731457 7f9d7815d700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.731465 7f9d7815d700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.731686 7f9d7815d700 20 set_header setting 17712 cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 parent seq 0
2014-06-12 14:47:08.731751 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.731766 7f9d7815d700 20 set_header setting 2140 cid meta oid 1fe0ae1a/pglog_3.783/0//-1 parent seq 0
2014-06-12 14:47:08.731780 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:47:08.734346 7f9d58d5b700  5 -- op tracker -- , seq: 748309, time: 2014-06-12 14:47:08.734305, event: header_read, op: osd_sub_op_reply(client.4299.0:145546 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.734356 7f9d58d5b700  5 -- op tracker -- , seq: 748309, time: 2014-06-12 14:47:08.734307, event: throttled, op: osd_sub_op_reply(client.4299.0:145546 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.734360 7f9d58d5b700  5 -- op tracker -- , seq: 748309, time: 2014-06-12 14:47:08.734319, event: all_read, op: osd_sub_op_reply(client.4299.0:145546 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.734372 7f9d58d5b700  5 -- op tracker -- , seq: 748309, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4299.0:145546 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.734377 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4299.0:145546 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:08.734381 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:08.734387 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016400 prio 196 cost 0 latency 0.000082 osd_sub_op_reply(client.4299.0:145546 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:08.734410 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.734417 7f9d683e1700 10 osd.2 173 dequeue_op 0x34016400 prio 196 cost 0 latency 0.000112 osd_sub_op_reply(client.4299.0:145546 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 pg pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3039 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean]
2014-06-12 14:47:08.734433 7f9d683e1700  5 -- op tracker -- , seq: 748309, time: 2014-06-12 14:47:08.734431, event: reached_pg, op: osd_sub_op_reply(client.4299.0:145546 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.734442 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3039 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] handle_message: 0x34016400
2014-06-12 14:47:08.734452 7f9d683e1700  5 -- op tracker -- , seq: 748309, time: 2014-06-12 14:47:08.734450, event: started, op: osd_sub_op_reply(client.4299.0:145546 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.734456 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3039 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] sub_op_modify_reply: tid 236398 op  ack_type 4 from 22
2014-06-12 14:47:08.734470 7f9d683e1700  5 -- op tracker -- , seq: 748308, time: 2014-06-12 14:47:08.734464, event: sub_op_commit_rec, op: osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.734474 7f9d683e1700 10 osd.2 173 dequeue_op 0x34016400 finish
2014-06-12 14:47:08.734478 7f9d683e1700  5 -- op tracker -- , seq: 748309, time: 2014-06-12 14:47:08.734476, event: done, op: osd_sub_op_reply(client.4299.0:145546 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.734487 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.734489 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.734581 7f9d7815d700 10 _do_op 0x4b2faa0 seq 2789 r = 0, finisher 0x6d373e0 0x3eabbd00
2014-06-12 14:47:08.734592 7f9d7815d700 10 _finish_op 0x4b2faa0 seq 2789 osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.734609 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d9e0 (0 active)
2014-06-12 14:47:08.734612 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.734597 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3039 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] op_commit: 236398
2014-06-12 14:47:08.734621 7f9d7715b700  5 -- op tracker -- , seq: 748308, time: 2014-06-12 14:47:08.734616, event: op_commit, op: osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.734630 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3039 crt=0'0 lcod 173'3038 mlcod 173'3038 active+clean] repop_all_committed: repop tid 236398 all committed 
2014-06-12 14:47:08.734645 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3038 active+clean] eval_repop repgather(0x1c3789c0 173'3040 rep_tid=236398 committed?=1 applied?=0 lock=0 op=osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.734655 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3038 active+clean] log_op_stats osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.008543
2014-06-12 14:47:08.734666 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3038 active+clean] publish_stats_to_osd 173:4043
2014-06-12 14:47:08.734673 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3038 active+clean]  sending commit on repgather(0x1c3789c0 173'3040 rep_tid=236398 committed?=1 applied?=0 lock=0 op=osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x45df400
2014-06-12 14:47:08.734694 7f9d7715b700  5 -- op tracker -- , seq: 748308, time: 2014-06-12 14:47:08.734691, event: commit_sent, op: osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.734722 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3038 active+clean] op_applied: 236398
2014-06-12 14:47:08.734758 7f9d7795c700  5 -- op tracker -- , seq: 748308, time: 2014-06-12 14:47:08.734751, event: op_applied, op: osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.734776 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3038 active+clean] op_applied on primary on version 173'3040
2014-06-12 14:47:08.734785 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3038 active+clean] repop_all_applied: repop tid 236398 all applied 
2014-06-12 14:47:08.734790 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3038 active+clean] eval_repop repgather(0x1c3789c0 173'3040 rep_tid=236398 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.734805 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3038 active+clean] log_op_stats osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.008693
2014-06-12 14:47:08.734820 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3038 active+clean] publish_stats_to_osd 173:4044
2014-06-12 14:47:08.734827 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3038 active+clean]  requeue_ops 
2014-06-12 14:47:08.734833 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]  removing repgather(0x1c3789c0 173'3040 rep_tid=236398 committed?=1 applied?=1 op=osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.734841 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]    q front is repgather(0x1c3789c0 173'3040 rep_tid=236398 committed?=1 applied?=1 op=osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.734856 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] remove_repop repgather(0x1c3789c0 173'3040 rep_tid=236398 committed?=1 applied?=1 op=osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.734864 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]  requeue_ops 
2014-06-12 14:47:08.734881 7f9d7795c700  5 -- op tracker -- , seq: 748308, time: 2014-06-12 14:47:08.734878, event: done, op: osd_op(client.4299.0:145546 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 544768~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.738319 7f9d3e9a5700  5 -- op tracker -- , seq: 748310, time: 2014-06-12 14:47:08.736878, event: header_read, op: osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.738339 7f9d3e9a5700  5 -- op tracker -- , seq: 748310, time: 2014-06-12 14:47:08.736880, event: throttled, op: osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.738351 7f9d3e9a5700  5 -- op tracker -- , seq: 748310, time: 2014-06-12 14:47:08.738267, event: all_read, op: osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.738369 7f9d3e9a5700  5 -- op tracker -- , seq: 748310, time: 0.000000, event: dispatched, op: osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.738379 7f9d3e9a5700 20 osd.2 173 should_share_map client.4299 172.16.96.220:0/1243711 173
2014-06-12 14:47:08.738414 7f9d3e9a5700 15 osd.2 173 enqueue_op 0xd38e500 prio 63 cost 516096 latency 0.001536 osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.738488 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.738515 7f9d6b3e7700 10 osd.2 173 dequeue_op 0xd38e500 prio 63 cost 516096 latency 0.001636 osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]
2014-06-12 14:47:08.738546 7f9d6b3e7700  5 -- op tracker -- , seq: 748310, time: 2014-06-12 14:47:08.738540, event: reached_pg, op: osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.738557 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:08.738571 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] handle_message: 0xd38e500
2014-06-12 14:47:08.738581 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] do_op osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:08.738604 7f9d6b3e7700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_'
2014-06-12 14:47:08.738767 7f9d6b3e7700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_' = 0
2014-06-12 14:47:08.738787 7f9d6b3e7700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset'
2014-06-12 14:47:08.738858 7f9d6b3e7700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset' = 0
2014-06-12 14:47:08.738870 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] populate_obc_watchers 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.738882 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.738891 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] get_object_context: creating obc from disk: 0x2fdfd080
2014-06-12 14:47:08.738899 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] get_object_context: 0x2fdfd080 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 rwstate(none n=0 w=0) oi: 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3040 client.4299.0:145546 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3040) ssc: 0x5d5a700 snapset: 0=[]:[]+head
2014-06-12 14:47:08.738914 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] find_object_context 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 @head oi=6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3040 client.4299.0:145546 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3040)
2014-06-12 14:47:08.738937 7f9d6b3e7700  5 -- op tracker -- , seq: 748310, time: 2014-06-12 14:47:08.738931, event: started, op: osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.738952 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] execute_ctx 0x580be00
2014-06-12 14:47:08.738968 7f9d6b3e7700  5 -- op tracker -- , seq: 748310, time: 2014-06-12 14:47:08.738964, event: started, op: osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.738974 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] do_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] ov 173'3040 av 173'3041 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:08.738987 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] do_osd_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096]
2014-06-12 14:47:08.738997 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:08.739009 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] do_osd_op  write 1060864~516096
2014-06-12 14:47:08.739023 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] do_osd_op_effects on session 0x6231800
2014-06-12 14:47:08.739033 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 snapset=0x5d5a738  snapc=0=[]
2014-06-12 14:47:08.739043 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:08.739052 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] finish_ctx 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 0x580be00 op modify  
2014-06-12 14:47:08.739063 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]  set mtime to 2014-06-12 08:46:11.116000
2014-06-12 14:47:08.739082 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]  final snapset 0=[]:[]+head in 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.739101 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]  zeroing write result code 0
2014-06-12 14:47:08.739111 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] calc_trim_to 79'39 -> 79'40
2014-06-12 14:47:08.739120 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] new_repop rep_tid 236399 on osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.739134 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] issue_repop rep_tid 236399 o 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.739155 7f9d6b3e7700  5 -- op tracker -- , seq: 748310, time: 2014-06-12 14:47:08.739151, event: waiting for subops from 22, op: osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.739175 7f9d6b3e7700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:08.739200 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3040 (79'39,173'3040] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] append_log log((79'39,173'3040], crt=0'0) [173'3041 (173'3040) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145547 2014-06-12 08:46:11.116000]
2014-06-12 14:47:08.739224 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'39,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] add_log_entry 173'3041 (173'3040) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145547 2014-06-12 08:46:11.116000
2014-06-12 14:47:08.739244 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'39,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] append_log  adding 1 keys
2014-06-12 14:47:08.739256 7f9d6b3e7700 10 trim log((79'39,173'3041], crt=0'0) to 79'40
2014-06-12 14:47:08.739285 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3041, trimmed: 79'40
2014-06-12 14:47:08.739300 7f9d6b3e7700  5 queue_transactions existing osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.739303 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.739304 7f9d6b3e7700  5 queue_op 0x7c7a0c0 seq 2790 osr(3.783 0x49b40f0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:08.739313 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d9e0 (1 active)
2014-06-12 14:47:08.739309 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] eval_repop repgather(0x10192f00 173'3041 rep_tid=236399 committed?=0 applied?=0 lock=0 op=osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.739319 7f9d82171700  5 _do_op 0x7c7a0c0 seq 2790 osr(3.783 0x49b40f0)/0x49b40f0 start
2014-06-12 14:47:08.739321 7f9d6b3e7700 10 osd.2 173 dequeue_op 0xd38e500 finish
2014-06-12 14:47:08.739322 7f9d82171700 10 _do_transaction on 0x15e56180
2014-06-12 14:47:08.739324 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.739325 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.739327 7f9d82171700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.739397 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.739417 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.739452 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.739540 7f9d82171700 15 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.739552 7f9d82171700 10 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1 r = 0
2014-06-12 14:47:08.739558 7f9d82171700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.739576 7f9d82171700 15 _write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 1060864~516096
2014-06-12 14:47:08.739752 7f9d82171700 10 lookup_strip_header done  cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.739983 7f9d82171700 10 file_to_extents done 
2014-06-12 14:47:08.741944 7f9d82171700 10 _generic_write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 1060864~516096 = 0
2014-06-12 14:47:08.741980 7f9d82171700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.741988 7f9d82171700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.741992 7f9d82171700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.741996 7f9d82171700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.742125 7f9d82171700 20 set_header setting 17712 cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 parent seq 0
2014-06-12 14:47:08.742168 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.742175 7f9d82171700 20 set_header setting 2140 cid meta oid 1fe0ae1a/pglog_3.783/0//-1 parent seq 0
2014-06-12 14:47:08.742181 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:47:08.743789 7f9d82171700 10 _do_op 0x7c7a0c0 seq 2790 r = 0, finisher 0x98ee360 0x3393f980
2014-06-12 14:47:08.743798 7f9d82171700 10 _finish_op 0x7c7a0c0 seq 2790 osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.743803 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d9e0 (0 active)
2014-06-12 14:47:08.743805 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.743802 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] op_commit: 236399
2014-06-12 14:47:08.743828 7f9d7715b700  5 -- op tracker -- , seq: 748310, time: 2014-06-12 14:47:08.743823, event: op_commit, op: osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.743838 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] op_applied: 236399
2014-06-12 14:47:08.743891 7f9d7795c700  5 -- op tracker -- , seq: 748310, time: 2014-06-12 14:47:08.743880, event: op_applied, op: osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.743908 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] op_applied on primary on version 173'3041
2014-06-12 14:47:08.745754 7f9d58d5b700  5 -- op tracker -- , seq: 748311, time: 2014-06-12 14:47:08.745729, event: header_read, op: osd_sub_op_reply(client.4299.0:145547 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.745760 7f9d58d5b700  5 -- op tracker -- , seq: 748311, time: 2014-06-12 14:47:08.745731, event: throttled, op: osd_sub_op_reply(client.4299.0:145547 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.745773 7f9d58d5b700  5 -- op tracker -- , seq: 748311, time: 2014-06-12 14:47:08.745737, event: all_read, op: osd_sub_op_reply(client.4299.0:145547 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.745778 7f9d58d5b700  5 -- op tracker -- , seq: 748311, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4299.0:145547 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.745781 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4299.0:145547 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:08.745784 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:08.745788 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34016000 prio 196 cost 0 latency 0.000059 osd_sub_op_reply(client.4299.0:145547 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:08.745828 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.745840 7f9d693e3700 10 osd.2 173 dequeue_op 0x34016000 prio 196 cost 0 latency 0.000110 osd_sub_op_reply(client.4299.0:145547 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 pg pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]
2014-06-12 14:47:08.745873 7f9d693e3700  5 -- op tracker -- , seq: 748311, time: 2014-06-12 14:47:08.745871, event: reached_pg, op: osd_sub_op_reply(client.4299.0:145547 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.745885 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] handle_message: 0x34016000
2014-06-12 14:47:08.745901 7f9d693e3700  5 -- op tracker -- , seq: 748311, time: 2014-06-12 14:47:08.745898, event: started, op: osd_sub_op_reply(client.4299.0:145547 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.745908 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] sub_op_modify_reply: tid 236399 op  ack_type 4 from 22
2014-06-12 14:47:08.745926 7f9d693e3700  5 -- op tracker -- , seq: 748310, time: 2014-06-12 14:47:08.745921, event: sub_op_commit_rec, op: osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.745931 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] repop_all_applied: repop tid 236399 all applied 
2014-06-12 14:47:08.745938 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] eval_repop repgather(0x10192f00 173'3041 rep_tid=236399 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.745950 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean]  sending ack on repgather(0x10192f00 173'3041 rep_tid=236399 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x1687b180
2014-06-12 14:47:08.745980 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3040 crt=0'0 lcod 173'3039 mlcod 173'3039 active+clean] repop_all_committed: repop tid 236399 all committed 
2014-06-12 14:47:08.746000 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3039 active+clean] eval_repop repgather(0x10192f00 173'3041 rep_tid=236399 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.746026 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3039 active+clean] log_op_stats osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.009100 lat 0.009146
2014-06-12 14:47:08.746040 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3039 active+clean] publish_stats_to_osd 173:4045
2014-06-12 14:47:08.746048 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3039 active+clean]  sending commit on repgather(0x10192f00 173'3041 rep_tid=236399 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x16971900
2014-06-12 14:47:08.746069 7f9d693e3700  5 -- op tracker -- , seq: 748310, time: 2014-06-12 14:47:08.746065, event: commit_sent, op: osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.746074 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3039 active+clean]  requeue_ops 
2014-06-12 14:47:08.746081 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]  removing repgather(0x10192f00 173'3041 rep_tid=236399 committed?=1 applied?=1 op=osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.746092 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]    q front is repgather(0x10192f00 173'3041 rep_tid=236399 committed?=1 applied?=1 op=osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.746101 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] remove_repop repgather(0x10192f00 173'3041 rep_tid=236399 committed?=1 applied?=1 op=osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.746110 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]  requeue_ops 
2014-06-12 14:47:08.746127 7f9d693e3700  5 -- op tracker -- , seq: 748310, time: 2014-06-12 14:47:08.746124, event: done, op: osd_op(client.4299.0:145547 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1060864~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.746141 7f9d693e3700 10 osd.2 173 dequeue_op 0x34016000 finish
2014-06-12 14:47:08.746147 7f9d693e3700  5 -- op tracker -- , seq: 748311, time: 2014-06-12 14:47:08.746145, event: done, op: osd_sub_op_reply(client.4299.0:145547 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.746154 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.746156 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.749511 7f9d3e9a5700  5 -- op tracker -- , seq: 748312, time: 2014-06-12 14:47:08.748159, event: header_read, op: osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.749528 7f9d3e9a5700  5 -- op tracker -- , seq: 748312, time: 2014-06-12 14:47:08.748162, event: throttled, op: osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.749537 7f9d3e9a5700  5 -- op tracker -- , seq: 748312, time: 2014-06-12 14:47:08.749450, event: all_read, op: osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.749554 7f9d3e9a5700  5 -- op tracker -- , seq: 748312, time: 0.000000, event: dispatched, op: osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.749563 7f9d3e9a5700 20 osd.2 173 should_share_map client.4299 172.16.96.220:0/1243711 173
2014-06-12 14:47:08.749591 7f9d3e9a5700 15 osd.2 173 enqueue_op 0xd38ff00 prio 63 cost 516096 latency 0.001431 osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.749626 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.749634 7f9d6abe6700 10 osd.2 173 dequeue_op 0xd38ff00 prio 63 cost 516096 latency 0.001475 osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]
2014-06-12 14:47:08.749656 7f9d6abe6700  5 -- op tracker -- , seq: 748312, time: 2014-06-12 14:47:08.749652, event: reached_pg, op: osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.749663 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:08.749673 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] handle_message: 0xd38ff00
2014-06-12 14:47:08.749680 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] do_op osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:08.749698 7f9d6abe6700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_'
2014-06-12 14:47:08.749823 7f9d6abe6700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_' = 0
2014-06-12 14:47:08.749836 7f9d6abe6700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset'
2014-06-12 14:47:08.749882 7f9d6abe6700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset' = 0
2014-06-12 14:47:08.749890 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] populate_obc_watchers 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.749898 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.749904 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] get_object_context: creating obc from disk: 0x30ac2580
2014-06-12 14:47:08.749909 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] get_object_context: 0x30ac2580 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 rwstate(none n=0 w=0) oi: 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3041 client.4299.0:145547 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3041) ssc: 0x679b260 snapset: 0=[]:[]+head
2014-06-12 14:47:08.749920 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] find_object_context 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 @head oi=6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3041 client.4299.0:145547 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3041)
2014-06-12 14:47:08.749937 7f9d6abe6700  5 -- op tracker -- , seq: 748312, time: 2014-06-12 14:47:08.749934, event: started, op: osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.749944 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] execute_ctx 0x55a2c00
2014-06-12 14:47:08.749954 7f9d6abe6700  5 -- op tracker -- , seq: 748312, time: 2014-06-12 14:47:08.749952, event: started, op: osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.749959 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] do_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] ov 173'3041 av 173'3042 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:08.749974 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] do_osd_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096]
2014-06-12 14:47:08.749981 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:08.749989 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] do_osd_op  write 1576960~516096
2014-06-12 14:47:08.750000 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] do_osd_op_effects on session 0x6231800
2014-06-12 14:47:08.750008 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 snapset=0x679b298  snapc=0=[]
2014-06-12 14:47:08.750016 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:08.750022 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] finish_ctx 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 0x55a2c00 op modify  
2014-06-12 14:47:08.750031 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]  set mtime to 2014-06-12 08:46:11.127306
2014-06-12 14:47:08.750044 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]  final snapset 0=[]:[]+head in 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.750058 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]  zeroing write result code 0
2014-06-12 14:47:08.750065 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] calc_trim_to 79'40 -> 79'41
2014-06-12 14:47:08.750071 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] new_repop rep_tid 236400 on osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.750082 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] issue_repop rep_tid 236400 o 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.750098 7f9d6abe6700  5 -- op tracker -- , seq: 748312, time: 2014-06-12 14:47:08.750095, event: waiting for subops from 22, op: osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.750114 7f9d6abe6700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:08.750130 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3041 (79'40,173'3041] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] append_log log((79'40,173'3041], crt=0'0) [173'3042 (173'3041) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145548 2014-06-12 08:46:11.127306]
2014-06-12 14:47:08.750153 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'40,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] add_log_entry 173'3042 (173'3041) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145548 2014-06-12 08:46:11.127306
2014-06-12 14:47:08.750172 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'40,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] append_log  adding 1 keys
2014-06-12 14:47:08.750185 7f9d6abe6700 10 trim log((79'40,173'3042], crt=0'0) to 79'41
2014-06-12 14:47:08.750209 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3042, trimmed: 79'41
2014-06-12 14:47:08.750230 7f9d6abe6700  5 queue_transactions existing osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.750235 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.750236 7f9d6abe6700  5 queue_op 0x619f920 seq 2791 osr(3.783 0x49b40f0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:08.750244 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d9e0 (1 active)
2014-06-12 14:47:08.750251 7f9d7e96a700  5 _do_op 0x619f920 seq 2791 osr(3.783 0x49b40f0)/0x49b40f0 start
2014-06-12 14:47:08.750254 7f9d7e96a700 10 _do_transaction on 0x519c540
2014-06-12 14:47:08.750242 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] eval_repop repgather(0x12ae2b40 173'3042 rep_tid=236400 committed?=0 applied?=0 lock=0 op=osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.750258 7f9d7e96a700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.750259 7f9d6abe6700 10 osd.2 173 dequeue_op 0xd38ff00 finish
2014-06-12 14:47:08.750262 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.750264 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.750332 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.750348 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.750376 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.750393 7f9d7e96a700 15 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.750397 7f9d7e96a700 10 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1 r = 0
2014-06-12 14:47:08.750401 7f9d7e96a700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.750605 7f9d7e96a700 15 _write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 1576960~516096
2014-06-12 14:47:08.750690 7f9d7e96a700 10 lookup_strip_header done  cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.750895 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:47:08.753220 7f9d7e96a700 10 _generic_write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 1576960~516096 = 0
2014-06-12 14:47:08.753280 7f9d7e96a700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.753308 7f9d7e96a700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.753325 7f9d7e96a700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.753337 7f9d7e96a700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.753604 7f9d7e96a700 20 set_header setting 17712 cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 parent seq 0
2014-06-12 14:47:08.753725 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.753738 7f9d7e96a700 20 set_header setting 2140 cid meta oid 1fe0ae1a/pglog_3.783/0//-1 parent seq 0
2014-06-12 14:47:08.753752 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:47:08.756156 7f9d7e96a700 10 _do_op 0x619f920 seq 2791 r = 0, finisher 0x54a4840 0x29eae5c0
2014-06-12 14:47:08.756175 7f9d7e96a700 10 _finish_op 0x619f920 seq 2791 osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.756183 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d9e0 (0 active)
2014-06-12 14:47:08.756193 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.756184 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] op_commit: 236400
2014-06-12 14:47:08.756219 7f9d7715b700  5 -- op tracker -- , seq: 748312, time: 2014-06-12 14:47:08.756207, event: op_commit, op: osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.756235 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] op_applied: 236400
2014-06-12 14:47:08.756260 7f9d7795c700  5 -- op tracker -- , seq: 748312, time: 2014-06-12 14:47:08.756253, event: op_applied, op: osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.756271 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] op_applied on primary on version 173'3042
2014-06-12 14:47:08.757404 7f9d58d5b700  5 -- op tracker -- , seq: 748313, time: 2014-06-12 14:47:08.757359, event: header_read, op: osd_sub_op_reply(client.4299.0:145548 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.757415 7f9d58d5b700  5 -- op tracker -- , seq: 748313, time: 2014-06-12 14:47:08.757362, event: throttled, op: osd_sub_op_reply(client.4299.0:145548 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.757423 7f9d58d5b700  5 -- op tracker -- , seq: 748313, time: 2014-06-12 14:47:08.757372, event: all_read, op: osd_sub_op_reply(client.4299.0:145548 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.757432 7f9d58d5b700  5 -- op tracker -- , seq: 748313, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4299.0:145548 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.757437 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4299.0:145548 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:08.757442 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:08.757450 7f9d58d5b700 15 osd.2 173 enqueue_op 0x34017800 prio 196 cost 0 latency 0.000091 osd_sub_op_reply(client.4299.0:145548 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:08.757493 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.757502 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34017800 prio 196 cost 0 latency 0.000143 osd_sub_op_reply(client.4299.0:145548 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 pg pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]
2014-06-12 14:47:08.757519 7f9d6a3e5700  5 -- op tracker -- , seq: 748313, time: 2014-06-12 14:47:08.757516, event: reached_pg, op: osd_sub_op_reply(client.4299.0:145548 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.757529 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] handle_message: 0x34017800
2014-06-12 14:47:08.757541 7f9d6a3e5700  5 -- op tracker -- , seq: 748313, time: 2014-06-12 14:47:08.757539, event: started, op: osd_sub_op_reply(client.4299.0:145548 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.757548 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] sub_op_modify_reply: tid 236400 op  ack_type 4 from 22
2014-06-12 14:47:08.757570 7f9d6a3e5700  5 -- op tracker -- , seq: 748312, time: 2014-06-12 14:47:08.757563, event: sub_op_commit_rec, op: osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.757578 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] repop_all_applied: repop tid 236400 all applied 
2014-06-12 14:47:08.757587 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] eval_repop repgather(0x12ae2b40 173'3042 rep_tid=236400 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.757598 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean]  sending ack on repgather(0x12ae2b40 173'3042 rep_tid=236400 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x239e3400
2014-06-12 14:47:08.757626 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3041 crt=0'0 lcod 173'3040 mlcod 173'3040 active+clean] repop_all_committed: repop tid 236400 all committed 
2014-06-12 14:47:08.757638 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3040 active+clean] eval_repop repgather(0x12ae2b40 173'3042 rep_tid=236400 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.757662 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3040 active+clean] log_op_stats osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.009465 lat 0.009502
2014-06-12 14:47:08.757683 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3040 active+clean] publish_stats_to_osd 173:4046
2014-06-12 14:47:08.757698 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3040 active+clean]  sending commit on repgather(0x12ae2b40 173'3042 rep_tid=236400 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x16971680
2014-06-12 14:47:08.757724 7f9d6a3e5700  5 -- op tracker -- , seq: 748312, time: 2014-06-12 14:47:08.757720, event: commit_sent, op: osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.757735 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3040 active+clean]  requeue_ops 
2014-06-12 14:47:08.757747 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]  removing repgather(0x12ae2b40 173'3042 rep_tid=236400 committed?=1 applied?=1 op=osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.757757 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]    q front is repgather(0x12ae2b40 173'3042 rep_tid=236400 committed?=1 applied?=1 op=osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.757767 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] remove_repop repgather(0x12ae2b40 173'3042 rep_tid=236400 committed?=1 applied?=1 op=osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.757776 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]  requeue_ops 
2014-06-12 14:47:08.757794 7f9d6a3e5700  5 -- op tracker -- , seq: 748312, time: 2014-06-12 14:47:08.757790, event: done, op: osd_op(client.4299.0:145548 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 1576960~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.757808 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x34017800 finish
2014-06-12 14:47:08.757813 7f9d6a3e5700  5 -- op tracker -- , seq: 748313, time: 2014-06-12 14:47:08.757811, event: done, op: osd_sub_op_reply(client.4299.0:145548 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.757820 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.757822 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.760866 7f9d3e9a5700  5 -- op tracker -- , seq: 748314, time: 2014-06-12 14:47:08.759928, event: header_read, op: osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.760877 7f9d3e9a5700  5 -- op tracker -- , seq: 748314, time: 2014-06-12 14:47:08.759930, event: throttled, op: osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.760883 7f9d3e9a5700  5 -- op tracker -- , seq: 748314, time: 2014-06-12 14:47:08.760831, event: all_read, op: osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.760888 7f9d3e9a5700  5 -- op tracker -- , seq: 748314, time: 0.000000, event: dispatched, op: osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.760894 7f9d3e9a5700 20 osd.2 173 should_share_map client.4299 172.16.96.220:0/1243711 173
2014-06-12 14:47:08.760917 7f9d3e9a5700 15 osd.2 173 enqueue_op 0xd38f400 prio 63 cost 516096 latency 0.000989 osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.760945 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.760954 7f9d66bde700 10 osd.2 173 dequeue_op 0xd38f400 prio 63 cost 516096 latency 0.001026 osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]
2014-06-12 14:47:08.760983 7f9d66bde700  5 -- op tracker -- , seq: 748314, time: 2014-06-12 14:47:08.760977, event: reached_pg, op: osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.760993 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:08.761010 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] handle_message: 0xd38f400
2014-06-12 14:47:08.761021 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] do_op osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:08.761047 7f9d66bde700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_'
2014-06-12 14:47:08.761199 7f9d66bde700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_' = 0
2014-06-12 14:47:08.761217 7f9d66bde700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset'
2014-06-12 14:47:08.761275 7f9d66bde700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset' = 0
2014-06-12 14:47:08.761285 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] populate_obc_watchers 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.761299 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.761310 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] get_object_context: creating obc from disk: 0x12d10840
2014-06-12 14:47:08.761323 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] get_object_context: 0x12d10840 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 rwstate(none n=0 w=0) oi: 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3042 client.4299.0:145548 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3042) ssc: 0x24d4afc0 snapset: 0=[]:[]+head
2014-06-12 14:47:08.761340 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] find_object_context 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 @head oi=6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3042 client.4299.0:145548 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3042)
2014-06-12 14:47:08.761370 7f9d66bde700  5 -- op tracker -- , seq: 748314, time: 2014-06-12 14:47:08.761363, event: started, op: osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.761377 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] execute_ctx 0x2747e400
2014-06-12 14:47:08.761396 7f9d66bde700  5 -- op tracker -- , seq: 748314, time: 2014-06-12 14:47:08.761390, event: started, op: osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.761404 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] do_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] ov 173'3042 av 173'3043 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:08.761419 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] do_osd_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096]
2014-06-12 14:47:08.761432 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:08.761446 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] do_osd_op  write 2093056~516096
2014-06-12 14:47:08.761464 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] do_osd_op_effects on session 0x6231800
2014-06-12 14:47:08.761477 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 snapset=0x24d4aff8  snapc=0=[]
2014-06-12 14:47:08.761491 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:08.761504 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] finish_ctx 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 0x2747e400 op modify  
2014-06-12 14:47:08.761524 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]  set mtime to 2014-06-12 08:46:11.139045
2014-06-12 14:47:08.761546 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]  final snapset 0=[]:[]+head in 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.761566 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]  zeroing write result code 0
2014-06-12 14:47:08.761577 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] calc_trim_to 79'41 -> 79'42
2014-06-12 14:47:08.761588 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] new_repop rep_tid 236401 on osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.761605 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] issue_repop rep_tid 236401 o 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.761639 7f9d66bde700  5 -- op tracker -- , seq: 748314, time: 2014-06-12 14:47:08.761633, event: waiting for subops from 22, op: osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.761669 7f9d66bde700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:08.761692 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3042 (79'41,173'3042] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] append_log log((79'41,173'3042], crt=0'0) [173'3043 (173'3042) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145549 2014-06-12 08:46:11.139045]
2014-06-12 14:47:08.761719 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'41,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] add_log_entry 173'3043 (173'3042) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145549 2014-06-12 08:46:11.139045
2014-06-12 14:47:08.761741 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'41,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] append_log  adding 1 keys
2014-06-12 14:47:08.761757 7f9d66bde700 10 trim log((79'41,173'3043], crt=0'0) to 79'42
2014-06-12 14:47:08.761782 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3043, trimmed: 79'42
2014-06-12 14:47:08.761808 7f9d66bde700  5 queue_transactions existing osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.761813 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.761815 7f9d66bde700  5 queue_op 0x216f560 seq 2792 osr(3.783 0x49b40f0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:08.761825 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d9e0 (1 active)
2014-06-12 14:47:08.761831 7f9d8116f700  5 _do_op 0x216f560 seq 2792 osr(3.783 0x49b40f0)/0x49b40f0 start
2014-06-12 14:47:08.761834 7f9d8116f700 10 _do_transaction on 0x32b67500
2014-06-12 14:47:08.761839 7f9d8116f700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.761823 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] eval_repop repgather(0x13d23500 173'3043 rep_tid=236401 committed?=0 applied?=0 lock=0 op=osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.761842 7f9d66bde700 10 osd.2 173 dequeue_op 0xd38f400 finish
2014-06-12 14:47:08.761847 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.761850 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.761987 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.762011 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.762041 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.762060 7f9d8116f700 15 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.762067 7f9d8116f700 10 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1 r = 0
2014-06-12 14:47:08.762071 7f9d8116f700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.762084 7f9d8116f700 15 _write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 2093056~516096
2014-06-12 14:47:08.762331 7f9d8116f700 10 lookup_strip_header done  cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.762468 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:47:08.764688 7f9d8116f700 10 _generic_write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 2093056~516096 = 0
2014-06-12 14:47:08.764754 7f9d8116f700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.764774 7f9d8116f700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.764783 7f9d8116f700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.764797 7f9d8116f700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.765067 7f9d8116f700 20 set_header setting 17712 cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 parent seq 0
2014-06-12 14:47:08.765161 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.765182 7f9d8116f700 20 set_header setting 2140 cid meta oid 1fe0ae1a/pglog_3.783/0//-1 parent seq 0
2014-06-12 14:47:08.765195 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:47:08.767661 7f9d58d5b700  5 -- op tracker -- , seq: 748315, time: 2014-06-12 14:47:08.767593, event: header_read, op: osd_sub_op_reply(client.4299.0:145549 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.767679 7f9d58d5b700  5 -- op tracker -- , seq: 748315, time: 2014-06-12 14:47:08.767596, event: throttled, op: osd_sub_op_reply(client.4299.0:145549 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.767689 7f9d58d5b700  5 -- op tracker -- , seq: 748315, time: 2014-06-12 14:47:08.767615, event: all_read, op: osd_sub_op_reply(client.4299.0:145549 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.767723 7f9d58d5b700  5 -- op tracker -- , seq: 748315, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4299.0:145549 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.767730 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4299.0:145549 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:08.767738 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:08.767749 7f9d58d5b700 15 osd.2 173 enqueue_op 0x12bf6100 prio 196 cost 0 latency 0.000156 osd_sub_op_reply(client.4299.0:145549 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:08.767826 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.767849 7f9d69be4700 10 osd.2 173 dequeue_op 0x12bf6100 prio 196 cost 0 latency 0.000256 osd_sub_op_reply(client.4299.0:145549 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 pg pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean]
2014-06-12 14:47:08.767892 7f9d69be4700  5 -- op tracker -- , seq: 748315, time: 2014-06-12 14:47:08.767886, event: reached_pg, op: osd_sub_op_reply(client.4299.0:145549 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.767928 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] handle_message: 0x12bf6100
2014-06-12 14:47:08.767957 7f9d69be4700  5 -- op tracker -- , seq: 748315, time: 2014-06-12 14:47:08.767952, event: started, op: osd_sub_op_reply(client.4299.0:145549 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.767966 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] sub_op_modify_reply: tid 236401 op  ack_type 4 from 22
2014-06-12 14:47:08.767994 7f9d69be4700  5 -- op tracker -- , seq: 748314, time: 2014-06-12 14:47:08.767983, event: sub_op_commit_rec, op: osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.768003 7f9d69be4700 10 osd.2 173 dequeue_op 0x12bf6100 finish
2014-06-12 14:47:08.768012 7f9d69be4700  5 -- op tracker -- , seq: 748315, time: 2014-06-12 14:47:08.768008, event: done, op: osd_sub_op_reply(client.4299.0:145549 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.768030 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.768034 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.768693 7f9d8116f700 10 _do_op 0x216f560 seq 2792 r = 0, finisher 0x80e9ee0 0x39f510c0
2014-06-12 14:47:08.768711 7f9d8116f700 10 _finish_op 0x216f560 seq 2792 osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.768720 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d9e0 (0 active)
2014-06-12 14:47:08.768725 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.768726 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] op_commit: 236401
2014-06-12 14:47:08.768771 7f9d7715b700  5 -- op tracker -- , seq: 748314, time: 2014-06-12 14:47:08.768758, event: op_commit, op: osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.768789 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3042 crt=0'0 lcod 173'3041 mlcod 173'3041 active+clean] repop_all_committed: repop tid 236401 all committed 
2014-06-12 14:47:08.768815 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean] eval_repop repgather(0x13d23500 173'3043 rep_tid=236401 committed?=1 applied?=0 lock=0 op=osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.768851 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean] log_op_stats osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.008921
2014-06-12 14:47:08.768892 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean] publish_stats_to_osd 173:4047
2014-06-12 14:47:08.768918 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean]  sending commit on repgather(0x13d23500 173'3043 rep_tid=236401 committed?=1 applied?=0 lock=0 op=osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x33b16f00
2014-06-12 14:47:08.769001 7f9d7715b700  5 -- op tracker -- , seq: 748314, time: 2014-06-12 14:47:08.768989, event: commit_sent, op: osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.769041 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean] op_applied: 236401
2014-06-12 14:47:08.769108 7f9d7795c700  5 -- op tracker -- , seq: 748314, time: 2014-06-12 14:47:08.769090, event: op_applied, op: osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.769137 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean] op_applied on primary on version 173'3043
2014-06-12 14:47:08.769149 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean] repop_all_applied: repop tid 236401 all applied 
2014-06-12 14:47:08.769160 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean] eval_repop repgather(0x13d23500 173'3043 rep_tid=236401 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.769178 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean] log_op_stats osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.000000 lat 0.009249
2014-06-12 14:47:08.769196 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean] publish_stats_to_osd 173:4048
2014-06-12 14:47:08.769208 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3041 active+clean]  requeue_ops 
2014-06-12 14:47:08.769220 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]  removing repgather(0x13d23500 173'3043 rep_tid=236401 committed?=1 applied?=1 op=osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.769237 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]    q front is repgather(0x13d23500 173'3043 rep_tid=236401 committed?=1 applied?=1 op=osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.769254 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] remove_repop repgather(0x13d23500 173'3043 rep_tid=236401 committed?=1 applied?=1 op=osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.769269 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]  requeue_ops 
2014-06-12 14:47:08.769294 7f9d7795c700  5 -- op tracker -- , seq: 748314, time: 2014-06-12 14:47:08.769290, event: done, op: osd_op(client.4299.0:145549 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2093056~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.772261 7f9d3e9a5700  5 -- op tracker -- , seq: 748316, time: 2014-06-12 14:47:08.771033, event: header_read, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.772278 7f9d3e9a5700  5 -- op tracker -- , seq: 748316, time: 2014-06-12 14:47:08.771036, event: throttled, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.772284 7f9d3e9a5700  5 -- op tracker -- , seq: 748316, time: 2014-06-12 14:47:08.772231, event: all_read, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.772289 7f9d3e9a5700  5 -- op tracker -- , seq: 748316, time: 0.000000, event: dispatched, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.772294 7f9d3e9a5700 20 osd.2 173 should_share_map client.4299 172.16.96.220:0/1243711 173
2014-06-12 14:47:08.772315 7f9d3e9a5700 15 osd.2 173 enqueue_op 0xd38f600 prio 63 cost 516096 latency 0.001281 osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.772340 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.772348 7f9d64bda700 10 osd.2 173 dequeue_op 0xd38f600 prio 63 cost 516096 latency 0.001315 osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]
2014-06-12 14:47:08.772384 7f9d64bda700  5 -- op tracker -- , seq: 748316, time: 2014-06-12 14:47:08.772379, event: reached_pg, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.772394 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:08.772415 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] handle_message: 0xd38f600
2014-06-12 14:47:08.772425 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] do_op osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:08.772456 7f9d64bda700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_'
2014-06-12 14:47:08.772638 7f9d64bda700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_' = 0
2014-06-12 14:47:08.772672 7f9d64bda700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset'
2014-06-12 14:47:08.772792 7f9d64bda700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset' = 0
2014-06-12 14:47:08.772819 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] populate_obc_watchers 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.772843 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.772856 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] get_object_context: creating obc from disk: 0x2fc018c0
2014-06-12 14:47:08.772869 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] get_object_context: 0x2fc018c0 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 rwstate(none n=0 w=0) oi: 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3043 client.4299.0:145549 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3043) ssc: 0x1e15e460 snapset: 0=[]:[]+head
2014-06-12 14:47:08.772890 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] find_object_context 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 @head oi=6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3043 client.4299.0:145549 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3043)
2014-06-12 14:47:08.772928 7f9d64bda700  5 -- op tracker -- , seq: 748316, time: 2014-06-12 14:47:08.772919, event: started, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.772943 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] execute_ctx 0x292b800
2014-06-12 14:47:08.772967 7f9d64bda700  5 -- op tracker -- , seq: 748316, time: 2014-06-12 14:47:08.772959, event: started, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.772993 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] do_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] ov 173'3043 av 173'3044 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:08.773018 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] do_osd_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096]
2014-06-12 14:47:08.773032 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:08.773049 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] do_osd_op  write 2609152~516096
2014-06-12 14:47:08.773069 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] do_osd_op_effects on session 0x6231800
2014-06-12 14:47:08.773082 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 snapset=0x1e15e498  snapc=0=[]
2014-06-12 14:47:08.773097 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:08.773110 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] finish_ctx 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 0x292b800 op modify  
2014-06-12 14:47:08.773126 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]  set mtime to 2014-06-12 08:46:11.150166
2014-06-12 14:47:08.773167 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]  final snapset 0=[]:[]+head in 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.773192 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]  zeroing write result code 0
2014-06-12 14:47:08.773209 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] calc_trim_to 79'42 -> 79'43
2014-06-12 14:47:08.773222 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] new_repop rep_tid 236402 on osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.773243 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] issue_repop rep_tid 236402 o 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.773273 7f9d64bda700  5 -- op tracker -- , seq: 748316, time: 2014-06-12 14:47:08.773266, event: waiting for subops from 22, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.773300 7f9d64bda700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:08.773329 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3043 (79'42,173'3043] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] append_log log((79'42,173'3043], crt=0'0) [173'3044 (173'3043) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145550 2014-06-12 08:46:11.150166]
2014-06-12 14:47:08.773363 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'42,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] add_log_entry 173'3044 (173'3043) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145550 2014-06-12 08:46:11.150166
2014-06-12 14:47:08.773392 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'42,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] append_log  adding 1 keys
2014-06-12 14:47:08.773408 7f9d64bda700 10 trim log((79'42,173'3044], crt=0'0) to 79'43
2014-06-12 14:47:08.773442 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3044, trimmed: 79'43
2014-06-12 14:47:08.773473 7f9d64bda700  5 queue_transactions existing osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.773481 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.773484 7f9d64bda700  5 queue_op 0x216cde0 seq 2793 osr(3.783 0x49b40f0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:08.773501 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d9e0 (1 active)
2014-06-12 14:47:08.773514 7f9d7d167700  5 _do_op 0x216cde0 seq 2793 osr(3.783 0x49b40f0)/0x49b40f0 start
2014-06-12 14:47:08.773521 7f9d7d167700 10 _do_transaction on 0x22b79a40
2014-06-12 14:47:08.773497 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] eval_repop repgather(0x2a12f500 173'3044 rep_tid=236402 committed?=0 applied?=0 lock=0 op=osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.773532 7f9d7d167700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.773536 7f9d64bda700 10 osd.2 173 dequeue_op 0xd38f600 finish
2014-06-12 14:47:08.773545 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.773550 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.773631 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.773678 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.773754 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.773793 7f9d7d167700 15 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.773811 7f9d7d167700 10 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1 r = 0
2014-06-12 14:47:08.773823 7f9d7d167700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.773852 7f9d7d167700 15 _write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 2609152~516096
2014-06-12 14:47:08.774325 7f9d7d167700 10 lookup_strip_header done  cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.774507 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:47:08.776672 7f9d7d167700 10 _generic_write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 2609152~516096 = 0
2014-06-12 14:47:08.776741 7f9d7d167700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.776762 7f9d7d167700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.776771 7f9d7d167700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.776781 7f9d7d167700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.777047 7f9d7d167700 20 set_header setting 17712 cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 parent seq 0
2014-06-12 14:47:08.777160 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.777182 7f9d7d167700 20 set_header setting 2140 cid meta oid 1fe0ae1a/pglog_3.783/0//-1 parent seq 0
2014-06-12 14:47:08.777196 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:47:08.780337 7f9d7d167700 10 _do_op 0x216cde0 seq 2793 r = 0, finisher 0x6ce6ac0 0x6be2600
2014-06-12 14:47:08.780358 7f9d7d167700 10 _finish_op 0x216cde0 seq 2793 osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.780373 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d9e0 (0 active)
2014-06-12 14:47:08.780382 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.780381 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] op_commit: 236402
2014-06-12 14:47:08.780431 7f9d7715b700  5 -- op tracker -- , seq: 748316, time: 2014-06-12 14:47:08.780409, event: op_commit, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.780482 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] op_applied: 236402
2014-06-12 14:47:08.780547 7f9d7795c700  5 -- op tracker -- , seq: 748316, time: 2014-06-12 14:47:08.780532, event: op_applied, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.780571 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] op_applied on primary on version 173'3044
2014-06-12 14:47:08.783619 7f9d58d5b700  5 -- op tracker -- , seq: 748317, time: 2014-06-12 14:47:08.783518, event: header_read, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.783648 7f9d58d5b700  5 -- op tracker -- , seq: 748317, time: 2014-06-12 14:47:08.783521, event: throttled, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.783694 7f9d58d5b700  5 -- op tracker -- , seq: 748317, time: 2014-06-12 14:47:08.783549, event: all_read, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.783712 7f9d58d5b700  5 -- op tracker -- , seq: 748317, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.783727 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:08.783741 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:08.783758 7f9d58d5b700 15 osd.2 173 enqueue_op 0x12bf6000 prio 196 cost 0 latency 0.000240 osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0) v2
2014-06-12 14:47:08.783819 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.783844 7f9d653db700 10 osd.2 173 dequeue_op 0x12bf6000 prio 196 cost 0 latency 0.000326 osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0) v2 pg pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]
2014-06-12 14:47:08.783891 7f9d653db700  5 -- op tracker -- , seq: 748317, time: 2014-06-12 14:47:08.783883, event: reached_pg, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.783908 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] handle_message: 0x12bf6000
2014-06-12 14:47:08.783946 7f9d653db700  5 -- op tracker -- , seq: 748317, time: 2014-06-12 14:47:08.783938, event: started, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.783975 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] sub_op_modify_reply: tid 236402 op  ack_type 1 from 22
2014-06-12 14:47:08.784021 7f9d653db700  5 -- op tracker -- , seq: 748316, time: 2014-06-12 14:47:08.784005, event: sub_op_applied_rec, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.784044 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] repop_all_applied: repop tid 236402 all applied 
2014-06-12 14:47:08.784072 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] eval_repop repgather(0x2a12f500 173'3044 rep_tid=236402 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.784109 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]  sending ack on repgather(0x2a12f500 173'3044 rep_tid=236402 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0xe185b80
2014-06-12 14:47:08.784180 7f9d653db700 10 osd.2 173 dequeue_op 0x12bf6000 finish
2014-06-12 14:47:08.784192 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.784198 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.784279 7f9d58d5b700  5 -- op tracker -- , seq: 748317, time: 2014-06-12 14:47:08.784270, event: done, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.784394 7f9d58d5b700  5 -- op tracker -- , seq: 748318, time: 2014-06-12 14:47:08.784319, event: header_read, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.784419 7f9d58d5b700  5 -- op tracker -- , seq: 748318, time: 2014-06-12 14:47:08.784320, event: throttled, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.784437 7f9d58d5b700  5 -- op tracker -- , seq: 748318, time: 2014-06-12 14:47:08.784341, event: all_read, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.784459 7f9d58d5b700  5 -- op tracker -- , seq: 748318, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.784467 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:08.784478 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:08.784491 7f9d58d5b700 15 osd.2 173 enqueue_op 0x12bf6000 prio 196 cost 0 latency 0.000171 osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:08.784548 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.784571 7f9d623d5700 10 osd.2 173 dequeue_op 0x12bf6000 prio 196 cost 0 latency 0.000251 osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 pg pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean]
2014-06-12 14:47:08.784612 7f9d623d5700  5 -- op tracker -- , seq: 748318, time: 2014-06-12 14:47:08.784607, event: reached_pg, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.784624 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] handle_message: 0x12bf6000
2014-06-12 14:47:08.784644 7f9d623d5700  5 -- op tracker -- , seq: 748318, time: 2014-06-12 14:47:08.784640, event: started, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.784652 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] sub_op_modify_reply: tid 236402 op  ack_type 4 from 22
2014-06-12 14:47:08.784676 7f9d623d5700  5 -- op tracker -- , seq: 748316, time: 2014-06-12 14:47:08.784667, event: sub_op_commit_rec, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.784685 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3043 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] repop_all_committed: repop tid 236402 all committed 
2014-06-12 14:47:08.784698 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3042 active+clean] eval_repop repgather(0x2a12f500 173'3044 rep_tid=236402 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.784725 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3042 active+clean] log_op_stats osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.013143 lat 0.013691
2014-06-12 14:47:08.784748 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3042 active+clean] publish_stats_to_osd 173:4049
2014-06-12 14:47:08.784764 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3042 active+clean]  sending commit on repgather(0x2a12f500 173'3044 rep_tid=236402 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x38137b80
2014-06-12 14:47:08.784812 7f9d623d5700  5 -- op tracker -- , seq: 748316, time: 2014-06-12 14:47:08.784804, event: commit_sent, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.784822 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3042 active+clean]  requeue_ops 
2014-06-12 14:47:08.784835 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]  removing repgather(0x2a12f500 173'3044 rep_tid=236402 committed?=1 applied?=1 op=osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.784854 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]    q front is repgather(0x2a12f500 173'3044 rep_tid=236402 committed?=1 applied?=1 op=osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.784872 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] remove_repop repgather(0x2a12f500 173'3044 rep_tid=236402 committed?=1 applied?=1 op=osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.784889 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]  requeue_ops 
2014-06-12 14:47:08.784922 7f9d623d5700  5 -- op tracker -- , seq: 748316, time: 2014-06-12 14:47:08.784915, event: done, op: osd_op(client.4299.0:145550 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 2609152~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.784945 7f9d623d5700 10 osd.2 173 dequeue_op 0x12bf6000 finish
2014-06-12 14:47:08.784954 7f9d623d5700  5 -- op tracker -- , seq: 748318, time: 2014-06-12 14:47:08.784949, event: done, op: osd_sub_op_reply(client.4299.0:145550 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.784967 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.784971 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.787978 7f9d3e9a5700  5 -- op tracker -- , seq: 748319, time: 2014-06-12 14:47:08.786899, event: header_read, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.787999 7f9d3e9a5700  5 -- op tracker -- , seq: 748319, time: 2014-06-12 14:47:08.786902, event: throttled, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.788008 7f9d3e9a5700  5 -- op tracker -- , seq: 748319, time: 2014-06-12 14:47:08.787937, event: all_read, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.788017 7f9d3e9a5700  5 -- op tracker -- , seq: 748319, time: 0.000000, event: dispatched, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.788025 7f9d3e9a5700 20 osd.2 173 should_share_map client.4299 172.16.96.220:0/1243711 173
2014-06-12 14:47:08.788050 7f9d3e9a5700 15 osd.2 173 enqueue_op 0x55c2100 prio 63 cost 516096 latency 0.001151 osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.788085 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.788116 7f9d68be2700 10 osd.2 173 dequeue_op 0x55c2100 prio 63 cost 516096 latency 0.001216 osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]
2014-06-12 14:47:08.788168 7f9d68be2700  5 -- op tracker -- , seq: 748319, time: 2014-06-12 14:47:08.788157, event: reached_pg, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.788187 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:08.788216 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] handle_message: 0x55c2100
2014-06-12 14:47:08.788238 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] do_op osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:08.788285 7f9d68be2700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_'
2014-06-12 14:47:08.788578 7f9d68be2700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_' = 0
2014-06-12 14:47:08.788614 7f9d68be2700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset'
2014-06-12 14:47:08.788774 7f9d68be2700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset' = 0
2014-06-12 14:47:08.788797 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] populate_obc_watchers 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.788825 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.788847 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] get_object_context: creating obc from disk: 0x22ead080
2014-06-12 14:47:08.788867 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] get_object_context: 0x22ead080 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 rwstate(none n=0 w=0) oi: 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3044 client.4299.0:145550 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3044) ssc: 0x407f960 snapset: 0=[]:[]+head
2014-06-12 14:47:08.788899 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] find_object_context 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 @head oi=6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3044 client.4299.0:145550 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3044)
2014-06-12 14:47:08.788956 7f9d68be2700  5 -- op tracker -- , seq: 748319, time: 2014-06-12 14:47:08.788943, event: started, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.788976 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] execute_ctx 0x60aa600
2014-06-12 14:47:08.789014 7f9d68be2700  5 -- op tracker -- , seq: 748319, time: 2014-06-12 14:47:08.789003, event: started, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.789053 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] do_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] ov 173'3044 av 173'3045 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:08.789083 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] do_osd_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096]
2014-06-12 14:47:08.789106 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:08.789139 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] do_osd_op  write 3125248~516096
2014-06-12 14:47:08.789175 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] do_osd_op_effects on session 0x6231800
2014-06-12 14:47:08.789203 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 snapset=0x407f998  snapc=0=[]
2014-06-12 14:47:08.789240 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:08.789263 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] finish_ctx 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 0x60aa600 op modify  
2014-06-12 14:47:08.789288 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]  set mtime to 2014-06-12 08:46:11.166040
2014-06-12 14:47:08.789338 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]  final snapset 0=[]:[]+head in 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.789376 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]  zeroing write result code 0
2014-06-12 14:47:08.789401 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] calc_trim_to 79'43 -> 79'44
2014-06-12 14:47:08.789423 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] new_repop rep_tid 236403 on osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.789458 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] issue_repop rep_tid 236403 o 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.789506 7f9d68be2700  5 -- op tracker -- , seq: 748319, time: 2014-06-12 14:47:08.789495, event: waiting for subops from 22, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.789551 7f9d68be2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:08.789600 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3044 (79'43,173'3044] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] append_log log((79'43,173'3044], crt=0'0) [173'3045 (173'3044) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145551 2014-06-12 08:46:11.166040]
2014-06-12 14:47:08.789682 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'43,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] add_log_entry 173'3045 (173'3044) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145551 2014-06-12 08:46:11.166040
2014-06-12 14:47:08.789720 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'43,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] append_log  adding 1 keys
2014-06-12 14:47:08.789735 7f9d68be2700 10 trim log((79'43,173'3045], crt=0'0) to 79'44
2014-06-12 14:47:08.789762 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3045, trimmed: 79'44
2014-06-12 14:47:08.789789 7f9d68be2700  5 queue_transactions existing osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.789795 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.789797 7f9d68be2700  5 queue_op 0x11fb1860 seq 2794 osr(3.783 0x49b40f0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:08.789808 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d9e0 (1 active)
2014-06-12 14:47:08.789814 7f9d7915f700  5 _do_op 0x11fb1860 seq 2794 osr(3.783 0x49b40f0)/0x49b40f0 start
2014-06-12 14:47:08.789817 7f9d7915f700 10 _do_transaction on 0x24e8dd40
2014-06-12 14:47:08.789805 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] eval_repop repgather(0x6e535c0 173'3045 rep_tid=236403 committed?=0 applied?=0 lock=0 op=osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.789822 7f9d68be2700 10 osd.2 173 dequeue_op 0x55c2100 finish
2014-06-12 14:47:08.789823 7f9d7915f700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.789826 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.789829 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.789893 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.789916 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.789945 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.789965 7f9d7915f700 15 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.789972 7f9d7915f700 10 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1 r = 0
2014-06-12 14:47:08.789976 7f9d7915f700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.789989 7f9d7915f700 15 _write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 3125248~516096
2014-06-12 14:47:08.790230 7f9d7915f700 10 lookup_strip_header done  cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.790376 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:47:08.791660 7f9d7915f700 10 _generic_write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 3125248~516096 = 0
2014-06-12 14:47:08.791701 7f9d7915f700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.791713 7f9d7915f700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.791718 7f9d7915f700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.791723 7f9d7915f700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.791864 7f9d7915f700 20 set_header setting 17712 cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 parent seq 0
2014-06-12 14:47:08.791914 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.791925 7f9d7915f700 20 set_header setting 2140 cid meta oid 1fe0ae1a/pglog_3.783/0//-1 parent seq 0
2014-06-12 14:47:08.791933 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:47:08.793921 7f9d7915f700 10 _do_op 0x11fb1860 seq 2794 r = 0, finisher 0x29c34f00 0x3373cf00
2014-06-12 14:47:08.793946 7f9d7915f700 10 _finish_op 0x11fb1860 seq 2794 osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.793953 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d9e0 (0 active)
2014-06-12 14:47:08.793962 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.793953 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] op_commit: 236403
2014-06-12 14:47:08.794016 7f9d7715b700  5 -- op tracker -- , seq: 748319, time: 2014-06-12 14:47:08.794005, event: op_commit, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.794034 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] op_applied: 236403
2014-06-12 14:47:08.794077 7f9d7795c700  5 -- op tracker -- , seq: 748319, time: 2014-06-12 14:47:08.794055, event: op_applied, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.794086 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] op_applied on primary on version 173'3045
2014-06-12 14:47:08.799015 7f9d58d5b700  5 -- op tracker -- , seq: 748320, time: 2014-06-12 14:47:08.798918, event: header_read, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.799049 7f9d58d5b700  5 -- op tracker -- , seq: 748320, time: 2014-06-12 14:47:08.798921, event: throttled, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.799067 7f9d58d5b700  5 -- op tracker -- , seq: 748320, time: 2014-06-12 14:47:08.798942, event: all_read, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.799083 7f9d58d5b700  5 -- op tracker -- , seq: 748320, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.799094 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:08.799106 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:08.799124 7f9d58d5b700 15 osd.2 173 enqueue_op 0x12bf7f00 prio 196 cost 0 latency 0.000206 osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0) v2
2014-06-12 14:47:08.799206 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.799224 7f9d643d9700 10 osd.2 173 dequeue_op 0x12bf7f00 prio 196 cost 0 latency 0.000305 osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0) v2 pg pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]
2014-06-12 14:47:08.799273 7f9d58d5b700  5 -- op tracker -- , seq: 748321, time: 2014-06-12 14:47:08.799168, event: header_read, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.799287 7f9d58d5b700  5 -- op tracker -- , seq: 748321, time: 2014-06-12 14:47:08.799169, event: throttled, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.799317 7f9d58d5b700  5 -- op tracker -- , seq: 748321, time: 2014-06-12 14:47:08.799205, event: all_read, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.799340 7f9d58d5b700  5 -- op tracker -- , seq: 748321, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.799349 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:08.799362 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:08.799365 7f9d643d9700  5 -- op tracker -- , seq: 748320, time: 2014-06-12 14:47:08.799277, event: reached_pg, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.799367 7f9d58d5b700 15 osd.2 173 enqueue_op 0x2519de00 prio 196 cost 0 latency 0.000199 osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:08.799375 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] handle_message: 0x12bf7f00
2014-06-12 14:47:08.799387 7f9d643d9700  5 -- op tracker -- , seq: 748320, time: 2014-06-12 14:47:08.799385, event: started, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.799404 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] sub_op_modify_reply: tid 236403 op  ack_type 1 from 22
2014-06-12 14:47:08.799416 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:08.799419 7f9d643d9700  5 -- op tracker -- , seq: 748319, time: 2014-06-12 14:47:08.799412, event: sub_op_applied_rec, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.799425 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] repop_all_applied: repop tid 236403 all applied 
2014-06-12 14:47:08.799433 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] eval_repop repgather(0x6e535c0 173'3045 rep_tid=236403 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.799451 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]  sending ack on repgather(0x6e535c0 173'3045 rep_tid=236403 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x3b82aa00
2014-06-12 14:47:08.799473 7f9d643d9700 10 osd.2 173 dequeue_op 0x12bf7f00 finish
2014-06-12 14:47:08.799481 7f9d643d9700  5 -- op tracker -- , seq: 748320, time: 2014-06-12 14:47:08.799478, event: done, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ack, result = 0)
2014-06-12 14:47:08.799505 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:08.799509 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.799510 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2519de00 prio 196 cost 0 latency 0.000342 osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 pg pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean]
2014-06-12 14:47:08.799530 7f9d63bd8700  5 -- op tracker -- , seq: 748321, time: 2014-06-12 14:47:08.799528, event: reached_pg, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.799551 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] handle_message: 0x2519de00
2014-06-12 14:47:08.799575 7f9d63bd8700  5 -- op tracker -- , seq: 748321, time: 2014-06-12 14:47:08.799570, event: started, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.799585 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] sub_op_modify_reply: tid 236403 op  ack_type 4 from 22
2014-06-12 14:47:08.799611 7f9d63bd8700  5 -- op tracker -- , seq: 748319, time: 2014-06-12 14:47:08.799601, event: sub_op_commit_rec, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.799637 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3044 crt=0'0 lcod 173'3043 mlcod 173'3043 active+clean] repop_all_committed: repop tid 236403 all committed 
2014-06-12 14:47:08.799652 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3043 active+clean] eval_repop repgather(0x6e535c0 173'3045 rep_tid=236403 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.799682 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3043 active+clean] log_op_stats osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.012572 lat 0.012782
2014-06-12 14:47:08.799708 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3043 active+clean] publish_stats_to_osd 173:4050
2014-06-12 14:47:08.799727 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3043 active+clean]  sending commit on repgather(0x6e535c0 173'3045 rep_tid=236403 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x19e0ac80
2014-06-12 14:47:08.799777 7f9d63bd8700  5 -- op tracker -- , seq: 748319, time: 2014-06-12 14:47:08.799770, event: commit_sent, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.799790 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3043 active+clean]  requeue_ops 
2014-06-12 14:47:08.799804 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]  removing repgather(0x6e535c0 173'3045 rep_tid=236403 committed?=1 applied?=1 op=osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.799825 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]    q front is repgather(0x6e535c0 173'3045 rep_tid=236403 committed?=1 applied?=1 op=osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.799846 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] remove_repop repgather(0x6e535c0 173'3045 rep_tid=236403 committed?=1 applied?=1 op=osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.799869 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]  requeue_ops 
2014-06-12 14:47:08.799915 7f9d63bd8700  5 -- op tracker -- , seq: 748319, time: 2014-06-12 14:47:08.799907, event: done, op: osd_op(client.4299.0:145551 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3125248~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.799943 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2519de00 finish
2014-06-12 14:47:08.799957 7f9d63bd8700  5 -- op tracker -- , seq: 748321, time: 2014-06-12 14:47:08.799952, event: done, op: osd_sub_op_reply(client.4299.0:145551 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.799971 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.799976 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.803165 7f9d3e9a5700  5 -- op tracker -- , seq: 748322, time: 2014-06-12 14:47:08.801891, event: header_read, op: osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.803184 7f9d3e9a5700  5 -- op tracker -- , seq: 748322, time: 2014-06-12 14:47:08.801894, event: throttled, op: osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.803197 7f9d3e9a5700  5 -- op tracker -- , seq: 748322, time: 2014-06-12 14:47:08.803116, event: all_read, op: osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.803209 7f9d3e9a5700  5 -- op tracker -- , seq: 748322, time: 0.000000, event: dispatched, op: osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.803219 7f9d3e9a5700 20 osd.2 173 should_share_map client.4299 172.16.96.220:0/1243711 173
2014-06-12 14:47:08.803253 7f9d3e9a5700 15 osd.2 173 enqueue_op 0x55c2f00 prio 63 cost 516096 latency 0.001362 osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.803289 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.803298 7f9d61bd4700 10 osd.2 173 dequeue_op 0x55c2f00 prio 63 cost 516096 latency 0.001407 osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]
2014-06-12 14:47:08.803335 7f9d61bd4700  5 -- op tracker -- , seq: 748322, time: 2014-06-12 14:47:08.803328, event: reached_pg, op: osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.803348 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:08.803368 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] handle_message: 0x55c2f00
2014-06-12 14:47:08.803374 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] do_op osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:08.803389 7f9d61bd4700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_'
2014-06-12 14:47:08.803511 7f9d61bd4700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_' = 0
2014-06-12 14:47:08.803526 7f9d61bd4700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset'
2014-06-12 14:47:08.803571 7f9d61bd4700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset' = 0
2014-06-12 14:47:08.803578 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] populate_obc_watchers 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.803586 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.803591 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] get_object_context: creating obc from disk: 0x2cab9340
2014-06-12 14:47:08.803596 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] get_object_context: 0x2cab9340 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 rwstate(none n=0 w=0) oi: 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3045 client.4299.0:145551 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3045) ssc: 0x24d4b500 snapset: 0=[]:[]+head
2014-06-12 14:47:08.803609 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] find_object_context 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 @head oi=6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3045 client.4299.0:145551 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3045)
2014-06-12 14:47:08.803642 7f9d61bd4700  5 -- op tracker -- , seq: 748322, time: 2014-06-12 14:47:08.803624, event: started, op: osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.803650 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] execute_ctx 0x55a4400
2014-06-12 14:47:08.803661 7f9d61bd4700  5 -- op tracker -- , seq: 748322, time: 2014-06-12 14:47:08.803658, event: started, op: osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.803664 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] do_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] ov 173'3045 av 173'3046 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:08.803676 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] do_osd_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096]
2014-06-12 14:47:08.803683 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:08.803691 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] do_osd_op  write 3641344~516096
2014-06-12 14:47:08.803700 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] do_osd_op_effects on session 0x6231800
2014-06-12 14:47:08.803707 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 snapset=0x24d4b538  snapc=0=[]
2014-06-12 14:47:08.803713 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:08.803722 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] finish_ctx 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 0x55a4400 op modify  
2014-06-12 14:47:08.803729 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]  set mtime to 2014-06-12 08:46:11.180987
2014-06-12 14:47:08.803742 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]  final snapset 0=[]:[]+head in 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.803754 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]  zeroing write result code 0
2014-06-12 14:47:08.803760 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] calc_trim_to 79'44 -> 79'45
2014-06-12 14:47:08.803765 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] new_repop rep_tid 236404 on osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.803774 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] issue_repop rep_tid 236404 o 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.803789 7f9d61bd4700  5 -- op tracker -- , seq: 748322, time: 2014-06-12 14:47:08.803786, event: waiting for subops from 22, op: osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.803803 7f9d61bd4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:08.803816 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3045 (79'44,173'3045] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] append_log log((79'44,173'3045], crt=0'0) [173'3046 (173'3045) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145552 2014-06-12 08:46:11.180987]
2014-06-12 14:47:08.803839 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'44,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] add_log_entry 173'3046 (173'3045) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145552 2014-06-12 08:46:11.180987
2014-06-12 14:47:08.803851 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'44,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] append_log  adding 1 keys
2014-06-12 14:47:08.803859 7f9d61bd4700 10 trim log((79'44,173'3046], crt=0'0) to 79'45
2014-06-12 14:47:08.803877 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3046, trimmed: 79'45
2014-06-12 14:47:08.803894 7f9d61bd4700  5 queue_transactions existing osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.803900 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.803901 7f9d61bd4700  5 queue_op 0x6d6a180 seq 2795 osr(3.783 0x49b40f0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:08.803906 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] eval_repop repgather(0x26f0900 173'3046 rep_tid=236404 committed?=0 applied?=0 lock=0 op=osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.803925 7f9d61bd4700 10 osd.2 173 dequeue_op 0x55c2f00 finish
2014-06-12 14:47:08.803928 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.803929 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.803938 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d9e0 (1 active)
2014-06-12 14:47:08.803946 7f9d8016d700  5 _do_op 0x6d6a180 seq 2795 osr(3.783 0x49b40f0)/0x49b40f0 start
2014-06-12 14:47:08.803949 7f9d8016d700 10 _do_transaction on 0x31b65bc0
2014-06-12 14:47:08.803955 7f9d8016d700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.804024 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.804050 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.804079 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.804092 7f9d8016d700 15 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.804107 7f9d8016d700 10 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1 r = 0
2014-06-12 14:47:08.804119 7f9d8016d700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.804132 7f9d8016d700 15 _write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 3641344~516096
2014-06-12 14:47:08.804216 7f9d8016d700 10 lookup_strip_header done  cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.804247 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:47:08.805584 7f9d8016d700 10 _generic_write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 3641344~516096 = 0
2014-06-12 14:47:08.805631 7f9d8016d700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.805649 7f9d8016d700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.805657 7f9d8016d700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.805662 7f9d8016d700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.805796 7f9d8016d700 20 set_header setting 17712 cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 parent seq 0
2014-06-12 14:47:08.805842 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.805860 7f9d8016d700 20 set_header setting 2140 cid meta oid 1fe0ae1a/pglog_3.783/0//-1 parent seq 0
2014-06-12 14:47:08.805872 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:47:08.807277 7f9d8016d700 10 _do_op 0x6d6a180 seq 2795 r = 0, finisher 0x89408c0 0x3373d1c0
2014-06-12 14:47:08.807288 7f9d8016d700 10 _finish_op 0x6d6a180 seq 2795 osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.807293 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d9e0 (0 active)
2014-06-12 14:47:08.807296 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.807296 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] op_commit: 236404
2014-06-12 14:47:08.807326 7f9d7715b700  5 -- op tracker -- , seq: 748322, time: 2014-06-12 14:47:08.807317, event: op_commit, op: osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.807426 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] op_applied: 236404
2014-06-12 14:47:08.807481 7f9d7795c700  5 -- op tracker -- , seq: 748322, time: 2014-06-12 14:47:08.807468, event: op_applied, op: osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.807501 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] op_applied on primary on version 173'3046
2014-06-12 14:47:08.813369 7f9d58d5b700  5 -- op tracker -- , seq: 748323, time: 2014-06-12 14:47:08.813307, event: header_read, op: osd_sub_op_reply(client.4299.0:145552 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.813397 7f9d58d5b700  5 -- op tracker -- , seq: 748323, time: 2014-06-12 14:47:08.813309, event: throttled, op: osd_sub_op_reply(client.4299.0:145552 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.813429 7f9d58d5b700  5 -- op tracker -- , seq: 748323, time: 2014-06-12 14:47:08.813326, event: all_read, op: osd_sub_op_reply(client.4299.0:145552 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.813447 7f9d58d5b700  5 -- op tracker -- , seq: 748323, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4299.0:145552 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.813457 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4299.0:145552 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:08.813470 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:08.813485 7f9d58d5b700 15 osd.2 173 enqueue_op 0x2519d100 prio 196 cost 0 latency 0.000177 osd_sub_op_reply(client.4299.0:145552 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:08.813562 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.813592 7f9d633d7700 10 osd.2 173 dequeue_op 0x2519d100 prio 196 cost 0 latency 0.000284 osd_sub_op_reply(client.4299.0:145552 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 pg pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]
2014-06-12 14:47:08.813623 7f9d633d7700  5 -- op tracker -- , seq: 748323, time: 2014-06-12 14:47:08.813619, event: reached_pg, op: osd_sub_op_reply(client.4299.0:145552 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.813634 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] handle_message: 0x2519d100
2014-06-12 14:47:08.813647 7f9d633d7700  5 -- op tracker -- , seq: 748323, time: 2014-06-12 14:47:08.813645, event: started, op: osd_sub_op_reply(client.4299.0:145552 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.813655 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] sub_op_modify_reply: tid 236404 op  ack_type 4 from 22
2014-06-12 14:47:08.813671 7f9d633d7700  5 -- op tracker -- , seq: 748322, time: 2014-06-12 14:47:08.813665, event: sub_op_commit_rec, op: osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.813682 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] repop_all_applied: repop tid 236404 all applied 
2014-06-12 14:47:08.813691 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] eval_repop repgather(0x26f0900 173'3046 rep_tid=236404 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.813706 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean]  sending ack on repgather(0x26f0900 173'3046 rep_tid=236404 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x10096280
2014-06-12 14:47:08.813733 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3045 crt=0'0 lcod 173'3044 mlcod 173'3044 active+clean] repop_all_committed: repop tid 236404 all committed 
2014-06-12 14:47:08.813742 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3044 active+clean] eval_repop repgather(0x26f0900 173'3046 rep_tid=236404 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.813756 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3044 active+clean] log_op_stats osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.011841 lat 0.011864
2014-06-12 14:47:08.813774 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3044 active+clean] publish_stats_to_osd 173:4051
2014-06-12 14:47:08.813785 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3044 active+clean]  sending commit on repgather(0x26f0900 173'3046 rep_tid=236404 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x28676500
2014-06-12 14:47:08.813812 7f9d633d7700  5 -- op tracker -- , seq: 748322, time: 2014-06-12 14:47:08.813808, event: commit_sent, op: osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.813819 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3044 active+clean]  requeue_ops 
2014-06-12 14:47:08.813828 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]  removing repgather(0x26f0900 173'3046 rep_tid=236404 committed?=1 applied?=1 op=osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.813841 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]    q front is repgather(0x26f0900 173'3046 rep_tid=236404 committed?=1 applied?=1 op=osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.813855 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] remove_repop repgather(0x26f0900 173'3046 rep_tid=236404 committed?=1 applied?=1 op=osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.813867 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]  requeue_ops 
2014-06-12 14:47:08.813887 7f9d633d7700  5 -- op tracker -- , seq: 748322, time: 2014-06-12 14:47:08.813882, event: done, op: osd_op(client.4299.0:145552 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 3641344~516096] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.813903 7f9d633d7700 10 osd.2 173 dequeue_op 0x2519d100 finish
2014-06-12 14:47:08.813910 7f9d633d7700  5 -- op tracker -- , seq: 748323, time: 2014-06-12 14:47:08.813907, event: done, op: osd_sub_op_reply(client.4299.0:145552 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.813918 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.813921 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.815294 7f9d3e9a5700  5 -- op tracker -- , seq: 748324, time: 2014-06-12 14:47:08.815131, event: header_read, op: osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.815314 7f9d3e9a5700  5 -- op tracker -- , seq: 748324, time: 2014-06-12 14:47:08.815145, event: throttled, op: osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.815326 7f9d3e9a5700  5 -- op tracker -- , seq: 748324, time: 2014-06-12 14:47:08.815238, event: all_read, op: osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.815338 7f9d3e9a5700  5 -- op tracker -- , seq: 748324, time: 0.000000, event: dispatched, op: osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.815347 7f9d3e9a5700 20 osd.2 173 should_share_map client.4299 172.16.96.220:0/1243711 173
2014-06-12 14:47:08.815376 7f9d3e9a5700 15 osd.2 173 enqueue_op 0x55c3c00 prio 63 cost 36864 latency 0.000245 osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.815420 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.815439 7f9d67be0700 10 osd.2 173 dequeue_op 0x55c3c00 prio 63 cost 36864 latency 0.000308 osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]
2014-06-12 14:47:08.815476 7f9d67be0700  5 -- op tracker -- , seq: 748324, time: 2014-06-12 14:47:08.815469, event: reached_pg, op: osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.815493 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:08.815502 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] handle_message: 0x55c3c00
2014-06-12 14:47:08.815507 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] do_op osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:08.815520 7f9d67be0700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_'
2014-06-12 14:47:08.815600 7f9d67be0700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 '_' = 0
2014-06-12 14:47:08.815610 7f9d67be0700 15 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset'
2014-06-12 14:47:08.815653 7f9d67be0700 10 getattr 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 'snapset' = 0
2014-06-12 14:47:08.815661 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] populate_obc_watchers 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.815671 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.815676 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] get_object_context: creating obc from disk: 0x17da58c0
2014-06-12 14:47:08.815681 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] get_object_context: 0x17da58c0 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 rwstate(none n=0 w=0) oi: 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3046 client.4299.0:145552 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3046) ssc: 0x5d5b420 snapset: 0=[]:[]+head
2014-06-12 14:47:08.815690 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] find_object_context 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 @head oi=6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3(173'3046 client.4299.0:145552 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3046)
2014-06-12 14:47:08.815704 7f9d67be0700  5 -- op tracker -- , seq: 748324, time: 2014-06-12 14:47:08.815701, event: started, op: osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.815709 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] execute_ctx 0x2b885000
2014-06-12 14:47:08.815718 7f9d67be0700  5 -- op tracker -- , seq: 748324, time: 2014-06-12 14:47:08.815715, event: started, op: osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.815723 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] do_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] ov 173'3046 av 173'3047 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:08.815732 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] do_osd_op 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864]
2014-06-12 14:47:08.815738 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:08.815746 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] do_osd_op  write 4157440~36864
2014-06-12 14:47:08.815754 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] do_osd_op_effects on session 0x6231800
2014-06-12 14:47:08.815761 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 snapset=0x5d5b458  snapc=0=[]
2014-06-12 14:47:08.815768 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] make_writeable 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:08.815773 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] finish_ctx 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 0x2b885000 op modify  
2014-06-12 14:47:08.815779 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]  set mtime to 2014-06-12 08:46:11.195085
2014-06-12 14:47:08.815792 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]  final snapset 0=[]:[]+head in 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.815803 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]  zeroing write result code 0
2014-06-12 14:47:08.815809 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] calc_trim_to 79'45 -> 79'46
2014-06-12 14:47:08.815814 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] new_repop rep_tid 236405 on osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:08.815822 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] issue_repop rep_tid 236405 o 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.815835 7f9d67be0700  5 -- op tracker -- , seq: 748324, time: 2014-06-12 14:47:08.815832, event: waiting for subops from 22, op: osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.815846 7f9d67be0700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:08.815857 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3046 (79'45,173'3046] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] append_log log((79'45,173'3046], crt=0'0) [173'3047 (173'3046) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145553 2014-06-12 08:46:11.195085]
2014-06-12 14:47:08.815871 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'45,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] add_log_entry 173'3047 (173'3046) modify   6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 by client.4299.0:145553 2014-06-12 08:46:11.195085
2014-06-12 14:47:08.815893 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'45,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] append_log  adding 1 keys
2014-06-12 14:47:08.815900 7f9d67be0700 10 trim log((79'45,173'3047], crt=0'0) to 79'46
2014-06-12 14:47:08.815914 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3047, trimmed: 79'46
2014-06-12 14:47:08.815929 7f9d67be0700  5 queue_transactions existing osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.815931 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.815932 7f9d67be0700  5 queue_op 0xd5e7860 seq 2796 osr(3.783 0x49b40f0) 39192 bytes   (queue has 1 ops and 39192 bytes)
2014-06-12 14:47:08.815936 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] eval_repop repgather(0x3a096780 173'3047 rep_tid=236405 committed?=0 applied?=0 lock=0 op=osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.815955 7f9d67be0700 10 osd.2 173 dequeue_op 0x55c3c00 finish
2014-06-12 14:47:08.815958 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.815959 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.815976 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x550d9e0 (1 active)
2014-06-12 14:47:08.815983 7f9d79960700  5 _do_op 0xd5e7860 seq 2796 osr(3.783 0x49b40f0)/0x49b40f0 start
2014-06-12 14:47:08.815994 7f9d79960700 10 _do_transaction on 0x384e20c0
2014-06-12 14:47:08.815999 7f9d79960700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.816065 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.816080 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.816106 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.816118 7f9d79960700 15 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.816124 7f9d79960700 10 _omap_rmkeys meta/1fe0ae1a/pglog_3.783/0//-1 r = 0
2014-06-12 14:47:08.816127 7f9d79960700 15 _omap_setkeys meta/1fe0ae1a/pglog_3.783/0//-1
2014-06-12 14:47:08.816222 7f9d79960700 15 _write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 4157440~36864
2014-06-12 14:47:08.816299 7f9d79960700 10 lookup_strip_header done  cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.816310 7f9d79960700 10 file_to_extents done 
2014-06-12 14:47:08.816459 7f9d79960700 10 _generic_write 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 4157440~36864 = 0
2014-06-12 14:47:08.816469 7f9d79960700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.816475 7f9d79960700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.816480 7f9d79960700 15 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3
2014-06-12 14:47:08.816484 7f9d79960700 10 _setattrs 3.783_head/6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 = 0
2014-06-12 14:47:08.816557 7f9d79960700 20 set_header setting 17712 cid 3.783_head oid 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 parent seq 0
2014-06-12 14:47:08.816570 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.816578 7f9d79960700 20 set_header setting 2140 cid meta oid 1fe0ae1a/pglog_3.783/0//-1 parent seq 0
2014-06-12 14:47:08.816585 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:47:08.816729 7f9d79960700 10 _do_op 0xd5e7860 seq 2796 r = 0, finisher 0x288dfe0 0x35283b40
2014-06-12 14:47:08.816735 7f9d79960700 10 _finish_op 0xd5e7860 seq 2796 osr(3.783 0x49b40f0)/0x49b40f0
2014-06-12 14:47:08.816738 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x550d9e0 (0 active)
2014-06-12 14:47:08.816740 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.816739 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] op_commit: 236405
2014-06-12 14:47:08.816763 7f9d7715b700  5 -- op tracker -- , seq: 748324, time: 2014-06-12 14:47:08.816756, event: op_commit, op: osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.816797 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] op_applied: 236405
2014-06-12 14:47:08.816849 7f9d7795c700  5 -- op tracker -- , seq: 748324, time: 2014-06-12 14:47:08.816837, event: op_applied, op: osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.816868 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] op_applied on primary on version 173'3047
2014-06-12 14:47:08.818929 7f9d58d5b700  5 -- op tracker -- , seq: 748325, time: 2014-06-12 14:47:08.818862, event: header_read, op: osd_sub_op_reply(client.4299.0:145553 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.818949 7f9d58d5b700  5 -- op tracker -- , seq: 748325, time: 2014-06-12 14:47:08.818864, event: throttled, op: osd_sub_op_reply(client.4299.0:145553 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.818959 7f9d58d5b700  5 -- op tracker -- , seq: 748325, time: 2014-06-12 14:47:08.818880, event: all_read, op: osd_sub_op_reply(client.4299.0:145553 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.818969 7f9d58d5b700  5 -- op tracker -- , seq: 748325, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4299.0:145553 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.818975 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4299.0:145553 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:08.818982 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:47:08.818990 7f9d58d5b700 15 osd.2 173 enqueue_op 0x6f4b100 prio 196 cost 0 latency 0.000128 osd_sub_op_reply(client.4299.0:145553 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:08.819054 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.819075 7f9d673df700 10 osd.2 173 dequeue_op 0x6f4b100 prio 196 cost 0 latency 0.000213 osd_sub_op_reply(client.4299.0:145553 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0) v2 pg pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]
2014-06-12 14:47:08.819121 7f9d673df700  5 -- op tracker -- , seq: 748325, time: 2014-06-12 14:47:08.819107, event: reached_pg, op: osd_sub_op_reply(client.4299.0:145553 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.819129 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] handle_message: 0x6f4b100
2014-06-12 14:47:08.819140 7f9d673df700  5 -- op tracker -- , seq: 748325, time: 2014-06-12 14:47:08.819138, event: started, op: osd_sub_op_reply(client.4299.0:145553 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.819144 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] sub_op_modify_reply: tid 236405 op  ack_type 4 from 22
2014-06-12 14:47:08.819157 7f9d673df700  5 -- op tracker -- , seq: 748324, time: 2014-06-12 14:47:08.819153, event: sub_op_commit_rec, op: osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.819162 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] repop_all_applied: repop tid 236405 all applied 
2014-06-12 14:47:08.819169 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] eval_repop repgather(0x3a096780 173'3047 rep_tid=236405 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.819179 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean]  sending ack on repgather(0x3a096780 173'3047 rep_tid=236405 committed?=0 applied?=1 lock=0 op=osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x2d3c0280
2014-06-12 14:47:08.819197 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 luod=173'3046 crt=0'0 lcod 173'3045 mlcod 173'3045 active+clean] repop_all_committed: repop tid 236405 all committed 
2014-06-12 14:47:08.819204 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3045 active+clean] eval_repop repgather(0x3a096780 173'3047 rep_tid=236405 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:08.819213 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3045 active+clean] log_op_stats osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4 inb 37143 outb 0 rlat 0.004066 lat 0.004082
2014-06-12 14:47:08.819223 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3045 active+clean] publish_stats_to_osd 173:4052
2014-06-12 14:47:08.819230 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3045 active+clean]  sending commit on repgather(0x3a096780 173'3047 rep_tid=236405 committed?=1 applied?=1 lock=0 op=osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4) 0x15560c80
2014-06-12 14:47:08.819250 7f9d673df700  5 -- op tracker -- , seq: 748324, time: 2014-06-12 14:47:08.819246, event: commit_sent, op: osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.819255 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3045 active+clean]  requeue_ops 
2014-06-12 14:47:08.819261 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean]  removing repgather(0x3a096780 173'3047 rep_tid=236405 committed?=1 applied?=1 op=osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.819270 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean]    q front is repgather(0x3a096780 173'3047 rep_tid=236405 committed?=1 applied?=1 op=osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.819279 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] remove_repop repgather(0x3a096780 173'3047 rep_tid=236405 committed?=1 applied?=1 op=osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:08.819288 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean]  requeue_ops 
2014-06-12 14:47:08.819301 7f9d673df700  5 -- op tracker -- , seq: 748324, time: 2014-06-12 14:47:08.819298, event: done, op: osd_op(client.4299.0:145553 rbd_data.10a26b8b4567.0000000000000f72 [set-alloc-hint object_size 4194304 write_size 4194304,write 4157440~36864] 3.6a631f83 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:08.819310 7f9d673df700 10 osd.2 173 dequeue_op 0x6f4b100 finish
2014-06-12 14:47:08.819315 7f9d673df700  5 -- op tracker -- , seq: 748325, time: 2014-06-12 14:47:08.819313, event: done, op: osd_sub_op_reply(client.4299.0:145553 3.783 6a631f83/rbd_data.10a26b8b4567.0000000000000f72/head//3 [] ondisk, result = 0)
2014-06-12 14:47:08.819322 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.819324 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.949189 7f9d5a16f700  5 -- op tracker -- , seq: 748326, time: 2014-06-12 14:47:08.946615, event: header_read, op: osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.949223 7f9d5a16f700  5 -- op tracker -- , seq: 748326, time: 2014-06-12 14:47:08.946620, event: throttled, op: osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.949247 7f9d5a16f700  5 -- op tracker -- , seq: 748326, time: 2014-06-12 14:47:08.949116, event: all_read, op: osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.949267 7f9d5a16f700  5 -- op tracker -- , seq: 748326, time: 0.000000, event: dispatched, op: osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.949278 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:08.949294 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:08.949311 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9de00 prio 127 cost 1032999 latency 0.002695 osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:08.949397 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:08.949413 7f9d663dd700 10 osd.2 173 dequeue_op 0x24c9de00 prio 127 cost 1032999 latency 0.002798 osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2512 (0'0,173'2512] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2511 active]
2014-06-12 14:47:08.949453 7f9d663dd700  5 -- op tracker -- , seq: 748326, time: 2014-06-12 14:47:08.949446, event: reached_pg, op: osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.949478 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2512 (0'0,173'2512] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2511 active] handle_message: 0x24c9de00
2014-06-12 14:47:08.949495 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2512 (0'0,173'2512] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2511 active] sub_op_modify trans f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 v 173'2513 (transaction) 174
2014-06-12 14:47:08.949529 7f9d663dd700  5 -- op tracker -- , seq: 748326, time: 2014-06-12 14:47:08.949522, event: started, op: osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.949551 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2512 (0'0,173'2512] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2511 active] append_log log((0'0,173'2512], crt=0'0) [173'2513 (173'2512) modify   f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 by client.4310.0:142328 2014-06-12 08:44:18.004123]
2014-06-12 14:47:08.949586 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2513 (0'0,173'2513] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2511 active] add_log_entry 173'2513 (173'2512) modify   f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 by client.4310.0:142328 2014-06-12 08:44:18.004123
2014-06-12 14:47:08.949644 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2513 (0'0,173'2513] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2511 active] append_log  adding 1 keys
2014-06-12 14:47:08.949693 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2513, trimmed: 
2014-06-12 14:47:08.949728 7f9d663dd700  5 -- op tracker -- , seq: 748326, time: 2014-06-12 14:47:08.949721, event: started, op: osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.949744 7f9d663dd700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:08.949750 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:08.949751 7f9d663dd700  5 queue_op 0x83ee720 seq 2359 osr(3.168 0x276ce40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:08.949765 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:47:08.949760 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2513 (0'0,173'2513] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2511 active] do_sub_op osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:08.949777 7f9d7d968700  5 _do_op 0x83ee720 seq 2359 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:47:08.949780 7f9d663dd700 10 osd.2 173 dequeue_op 0x24c9de00 finish
2014-06-12 14:47:08.949783 7f9d7d968700 10 _do_transaction on 0x62c9970
2014-06-12 14:47:08.949785 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:08.949789 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:08.949793 7f9d7d968700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:08.949945 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:08.949988 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:08.950064 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:08.950097 7f9d7d968700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:08.950108 7f9d7d968700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:47:08.950116 7f9d7d968700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:08.950348 7f9d7d968700 15 _write 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 1392640~1032192
2014-06-12 14:47:08.950568 7f9d7d968700 10 lookup_strip_header done  cid 3.168_head oid f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:08.950760 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:47:08.956279 7f9d7d968700 10 _generic_write 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 1392640~1032192 = 0
2014-06-12 14:47:08.956416 7f9d7d968700 15 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:08.956442 7f9d7d968700 10 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 = 0
2014-06-12 14:47:08.956451 7f9d7d968700 15 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:08.956460 7f9d7d968700 10 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 = 0
2014-06-12 14:47:08.956857 7f9d7d968700 20 set_header setting 16446 cid 3.168_head oid f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 parent seq 0
2014-06-12 14:47:08.957023 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:08.957047 7f9d7d968700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:47:08.957065 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:47:08.967307 7f9d7d968700 10 _do_op 0x83ee720 seq 2359 r = 0, finisher 0x1ecc6ba0 0
2014-06-12 14:47:08.967346 7f9d7d968700 10 _finish_op 0x83ee720 seq 2359 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:08.967358 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:47:08.967366 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:08.967422 7f9d7715b700  5 -- op tracker -- , seq: 748326, time: 2014-06-12 14:47:08.967407, event: commit_sent, op: osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.967443 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2513 (0'0,173'2513] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2511 active] sub_op_modify_commit on op osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:08.967493 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:08.967585 7f9d7795c700  5 -- op tracker -- , seq: 748326, time: 2014-06-12 14:47:08.967573, event: sub_op_applied, op: osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:08.967605 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2513 (0'0,173'2513] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2512 active] sub_op_modify_applied on 0x62c9880 op osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:08.967644 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2513 (0'0,173'2513] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2512 active] op_applied on primary on version 173'2513
2014-06-12 14:47:08.967666 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2513 (0'0,173'2513] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2512 active] op_applied on replica on version 173'2513
2014-06-12 14:47:08.967711 7f9d7795c700  5 -- op tracker -- , seq: 748326, time: 2014-06-12 14:47:08.967702, event: done, op: osd_sub_op(client.4310.0:142328 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2513 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.105740 7f9d5a16f700  5 -- op tracker -- , seq: 748327, time: 2014-06-12 14:47:09.103430, event: header_read, op: osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.105760 7f9d5a16f700  5 -- op tracker -- , seq: 748327, time: 2014-06-12 14:47:09.103434, event: throttled, op: osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.105781 7f9d5a16f700  5 -- op tracker -- , seq: 748327, time: 2014-06-12 14:47:09.105676, event: all_read, op: osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.105795 7f9d5a16f700  5 -- op tracker -- , seq: 748327, time: 0.000000, event: dispatched, op: osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.105802 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:09.105810 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:09.105820 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9dc00 prio 127 cost 1032999 latency 0.002390 osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:09.105891 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:09.105906 7f9d65bdc700 10 osd.2 173 dequeue_op 0x24c9dc00 prio 127 cost 1032999 latency 0.002476 osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2673 (0'0,173'2673] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2672 active]
2014-06-12 14:47:09.105959 7f9d65bdc700  5 -- op tracker -- , seq: 748327, time: 2014-06-12 14:47:09.105946, event: reached_pg, op: osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.105968 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2673 (0'0,173'2673] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2672 active] handle_message: 0x24c9dc00
2014-06-12 14:47:09.105979 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2673 (0'0,173'2673] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2672 active] sub_op_modify trans 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 v 173'2674 (transaction) 174
2014-06-12 14:47:09.105990 7f9d65bdc700  5 -- op tracker -- , seq: 748327, time: 2014-06-12 14:47:09.105987, event: started, op: osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.106002 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2673 (0'0,173'2673] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2672 active] append_log log((0'0,173'2673], crt=0'0) [173'2674 (173'2673) modify   57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 by client.4282.0:145228 2014-06-12 08:44:32.953117]
2014-06-12 14:47:09.106020 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2674 (0'0,173'2674] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2672 active] add_log_entry 173'2674 (173'2673) modify   57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 by client.4282.0:145228 2014-06-12 08:44:32.953117
2014-06-12 14:47:09.106035 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2674 (0'0,173'2674] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2672 active] append_log  adding 1 keys
2014-06-12 14:47:09.106059 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2674, trimmed: 
2014-06-12 14:47:09.106075 7f9d65bdc700  5 -- op tracker -- , seq: 748327, time: 2014-06-12 14:47:09.106072, event: started, op: osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.106082 7f9d65bdc700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:09.106085 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:09.106086 7f9d65bdc700  5 queue_op 0x241a6420 seq 2484 osr(3.1b7 0x276d1a0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:09.106092 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2674 (0'0,173'2674] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2672 active] do_sub_op osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:09.106101 7f9d65bdc700 10 osd.2 173 dequeue_op 0x24c9dc00 finish
2014-06-12 14:47:09.106103 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:09.106099 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:09.106105 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:09.106109 7f9d7c165700  5 _do_op 0x241a6420 seq 2484 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:09.106114 7f9d7c165700 10 _do_transaction on 0xb9960f0
2014-06-12 14:47:09.106131 7f9d7c165700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:09.106272 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:09.106297 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:09.106364 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:09.106507 7f9d7c165700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:09.106520 7f9d7c165700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:09.106529 7f9d7c165700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:09.106552 7f9d7c165700 15 _write 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 1601536~1032192
2014-06-12 14:47:09.106739 7f9d7c165700 10 lookup_strip_header done  cid 3.1b7_head oid 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:09.106916 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:47:09.111596 7f9d7c165700 10 _generic_write 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 1601536~1032192 = 0
2014-06-12 14:47:09.111719 7f9d7c165700 15 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:09.111743 7f9d7c165700 10 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:47:09.111754 7f9d7c165700 15 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:09.111768 7f9d7c165700 10 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:47:09.112160 7f9d7c165700 20 set_header setting 18080 cid 3.1b7_head oid 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 parent seq 0
2014-06-12 14:47:09.112323 7f9d7c165700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:09.112348 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:09.112366 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:47:09.116474 7f9d7c165700 10 _do_op 0x241a6420 seq 2484 r = 0, finisher 0x6d377c0 0
2014-06-12 14:47:09.116495 7f9d7c165700 10 _finish_op 0x241a6420 seq 2484 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:09.116500 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:09.116507 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:09.116518 7f9d7715b700  5 -- op tracker -- , seq: 748327, time: 2014-06-12 14:47:09.116507, event: commit_sent, op: osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.116532 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2674 (0'0,173'2674] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2672 active] sub_op_modify_commit on op osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:09.116567 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:09.116633 7f9d7795c700  5 -- op tracker -- , seq: 748327, time: 2014-06-12 14:47:09.116626, event: sub_op_applied, op: osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.116644 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2674 (0'0,173'2674] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active] sub_op_modify_applied on 0xb996000 op osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:09.116658 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2674 (0'0,173'2674] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active] op_applied on primary on version 173'2674
2014-06-12 14:47:09.116665 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2674 (0'0,173'2674] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active] op_applied on replica on version 173'2674
2014-06-12 14:47:09.116688 7f9d7795c700  5 -- op tracker -- , seq: 748327, time: 2014-06-12 14:47:09.116685, event: done, op: osd_sub_op(client.4282.0:145228 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2674 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.171372 7f9d47c58700  5 -- op tracker -- , seq: 748328, time: 2014-06-12 14:47:09.170775, event: header_read, op: osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:09.171386 7f9d47c58700  5 -- op tracker -- , seq: 748328, time: 2014-06-12 14:47:09.170779, event: throttled, op: osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:09.171396 7f9d47c58700  5 -- op tracker -- , seq: 748328, time: 2014-06-12 14:47:09.171334, event: all_read, op: osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:09.171409 7f9d47c58700  5 -- op tracker -- , seq: 748328, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:09.171416 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:47:09.171441 7f9d47c58700 15 osd.2 173 enqueue_op 0x3b42a500 prio 63 cost 258048 latency 0.000666 osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:09.171473 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:09.171480 7f9d62bd6700 10 osd.2 173 dequeue_op 0x3b42a500 prio 63 cost 258048 latency 0.000705 osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]
2014-06-12 14:47:09.171512 7f9d62bd6700  5 -- op tracker -- , seq: 748328, time: 2014-06-12 14:47:09.171506, event: reached_pg, op: osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:09.171520 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:09.171537 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] handle_message: 0x3b42a500
2014-06-12 14:47:09.171550 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:09.171574 7f9d62bd6700 15 getattr 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 '_'
2014-06-12 14:47:09.184847 7f9d62bd6700 10 getattr 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 '_' = 0
2014-06-12 14:47:09.184872 7f9d62bd6700 15 getattr 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 'snapset'
2014-06-12 14:47:09.184986 7f9d62bd6700 10 getattr 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 'snapset' = 0
2014-06-12 14:47:09.184997 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] populate_obc_watchers 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:09.185015 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:09.185033 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: creating obc from disk: 0xedc6b00
2014-06-12 14:47:09.185043 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: 0xedc6b00 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 rwstate(none n=0 w=0) oi: 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'857 client.4316.0:41777 wrlock_by=unknown.0.0:0 dirty s 4194304 uv857) ssc: 0x1550f180 snapset: 0=[]:[]+head
2014-06-12 14:47:09.185068 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] find_object_context 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 @head oi=58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'857 client.4316.0:41777 wrlock_by=unknown.0.0:0 dirty s 4194304 uv857)
2014-06-12 14:47:09.185106 7f9d62bd6700  5 -- op tracker -- , seq: 748328, time: 2014-06-12 14:47:09.185096, event: started, op: osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:09.185116 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] execute_ctx 0x33b2d200
2014-06-12 14:47:09.185136 7f9d62bd6700  5 -- op tracker -- , seq: 748328, time: 2014-06-12 14:47:09.185131, event: started, op: osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:09.185145 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] ov 173'857 av 173'2976 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:09.185165 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048]
2014-06-12 14:47:09.185179 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:09.185194 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  write 0~258048
2014-06-12 14:47:09.185211 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:47:09.185224 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 snapset=0x1550f1b8  snapc=0=[]
2014-06-12 14:47:09.185239 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:09.185251 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] finish_ctx 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 0x33b2d200 op modify  
2014-06-12 14:47:09.185267 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  set mtime to 2014-06-12 08:46:11.550319
2014-06-12 14:47:09.185289 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  final snapset 0=[]:[]+head in 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:09.185308 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  zeroing write result code 0
2014-06-12 14:47:09.185321 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] new_repop rep_tid 236406 on osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:09.185343 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] issue_repop rep_tid 236406 o 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:09.185372 7f9d62bd6700  5 -- op tracker -- , seq: 748328, time: 2014-06-12 14:47:09.185366, event: waiting for subops from 24, op: osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:09.185392 7f9d62bd6700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:09.185415 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2975 (0'0,173'2975] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log log((0'0,173'2975], crt=0'0) [173'2976 (173'857) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147318 2014-06-12 08:46:11.550319]
2014-06-12 14:47:09.185435 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] add_log_entry 173'2976 (173'857) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147318 2014-06-12 08:46:11.550319
2014-06-12 14:47:09.185457 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log  adding 1 keys
2014-06-12 14:47:09.185488 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2976, trimmed: 
2014-06-12 14:47:09.185509 7f9d62bd6700  5 queue_transactions existing osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:09.185514 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:09.185516 7f9d62bd6700  5 queue_op 0x769fd40 seq 2705 osr(3.785 0x49b40d0) 260341 bytes   (queue has 1 ops and 260341 bytes)
2014-06-12 14:47:09.185527 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54bc360 (1 active)
2014-06-12 14:47:09.185536 7f9d8096e700  5 _do_op 0x769fd40 seq 2705 osr(3.785 0x49b40d0)/0x49b40d0 start
2014-06-12 14:47:09.185540 7f9d8096e700 10 _do_transaction on 0x32b68480
2014-06-12 14:47:09.185524 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] eval_repop repgather(0x24e8d680 173'2976 rep_tid=236406 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:09.185544 7f9d62bd6700 10 osd.2 173 dequeue_op 0x3b42a500 finish
2014-06-12 14:47:09.185545 7f9d8096e700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:09.185550 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:09.185554 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:09.185623 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:09.185772 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:09.185816 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:09.185837 7f9d8096e700 15 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:09.185843 7f9d8096e700 10 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1 r = 0
2014-06-12 14:47:09.185847 7f9d8096e700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:09.185858 7f9d8096e700 15 _write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 0~258048
2014-06-12 14:47:09.185938 7f9d8096e700 10 lookup_strip_header done  cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:09.186236 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:47:09.186969 7f9d8096e700 10 _generic_write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 0~258048 = 0
2014-06-12 14:47:09.187001 7f9d8096e700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:09.187012 7f9d8096e700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:09.187018 7f9d8096e700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:09.187023 7f9d8096e700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:09.187131 7f9d8096e700 20 set_header setting 17484 cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 parent seq 0
2014-06-12 14:47:09.187161 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:09.187172 7f9d8096e700 20 set_header setting 2142 cid meta oid 1fe0af7a/pglog_3.785/0//-1 parent seq 0
2014-06-12 14:47:09.187179 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:47:09.188823 7f9d8096e700 10 _do_op 0x769fd40 seq 2705 r = 0, finisher 0x7f203c0 0x4b31d40
2014-06-12 14:47:09.188839 7f9d8096e700 10 _finish_op 0x769fd40 seq 2705 osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:09.188851 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54bc360 (0 active)
2014-06-12 14:47:09.188854 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:09.188844 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_commit: 236406
2014-06-12 14:47:09.188873 7f9d7715b700  5 -- op tracker -- , seq: 748328, time: 2014-06-12 14:47:09.188865, event: op_commit, op: osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:09.188887 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied: 236406
2014-06-12 14:47:09.188907 7f9d7795c700  5 -- op tracker -- , seq: 748328, time: 2014-06-12 14:47:09.188903, event: op_applied, op: osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:09.188915 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied on primary on version 173'2976
2014-06-12 14:47:09.403625 7f9d5a16f700  5 -- op tracker -- , seq: 748329, time: 2014-06-12 14:47:09.402903, event: header_read, op: osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.403640 7f9d5a16f700  5 -- op tracker -- , seq: 748329, time: 2014-06-12 14:47:09.402907, event: throttled, op: osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.403656 7f9d5a16f700  5 -- op tracker -- , seq: 748329, time: 2014-06-12 14:47:09.403584, event: all_read, op: osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.403664 7f9d5a16f700  5 -- op tracker -- , seq: 748329, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.403669 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:09.403676 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:09.403685 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9c300 prio 127 cost 443175 latency 0.000782 osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:09.403739 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:09.403750 7f9d683e1700 10 osd.2 173 dequeue_op 0x24c9c300 prio 127 cost 443175 latency 0.000846 osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2866 (0'0,173'2866] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2865 active]
2014-06-12 14:47:09.403772 7f9d683e1700  5 -- op tracker -- , seq: 748329, time: 2014-06-12 14:47:09.403768, event: reached_pg, op: osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.403781 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2866 (0'0,173'2866] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2865 active] handle_message: 0x24c9c300
2014-06-12 14:47:09.403800 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2866 (0'0,173'2866] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2865 active] sub_op_modify trans f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 v 173'2867 (transaction) 174
2014-06-12 14:47:09.403812 7f9d683e1700  5 -- op tracker -- , seq: 748329, time: 2014-06-12 14:47:09.403809, event: started, op: osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.403827 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2866 (0'0,173'2866] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2865 active] append_log log((0'0,173'2866], crt=0'0) [173'2867 (173'859) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145699 2014-06-12 08:43:42.975911]
2014-06-12 14:47:09.403843 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2867 (0'0,173'2867] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2865 active] add_log_entry 173'2867 (173'859) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145699 2014-06-12 08:43:42.975911
2014-06-12 14:47:09.403860 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2867 (0'0,173'2867] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2865 active] append_log  adding 1 keys
2014-06-12 14:47:09.403882 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2867, trimmed: 
2014-06-12 14:47:09.403897 7f9d683e1700  5 -- op tracker -- , seq: 748329, time: 2014-06-12 14:47:09.403894, event: started, op: osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.403912 7f9d683e1700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:09.403915 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:09.403916 7f9d683e1700  5 queue_op 0x54101e0 seq 2580 osr(3.129 0x276cf20) 444661 bytes   (queue has 1 ops and 444661 bytes)
2014-06-12 14:47:09.403924 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:09.403921 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2867 (0'0,173'2867] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2865 active] do_sub_op osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:09.403930 7f9d683e1700 10 osd.2 173 dequeue_op 0x24c9c300 finish
2014-06-12 14:47:09.403931 7f9d7c966700  5 _do_op 0x54101e0 seq 2580 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:09.403933 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:09.403934 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:09.403935 7f9d7c966700 10 _do_transaction on 0x8223430
2014-06-12 14:47:09.403940 7f9d7c966700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:09.404051 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:09.404071 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:09.404115 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:09.404203 7f9d7c966700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:09.404209 7f9d7c966700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:09.404214 7f9d7c966700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:09.404231 7f9d7c966700 15 _write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 0~442368
2014-06-12 14:47:09.404389 7f9d7c966700 10 lookup_strip_header done  cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:09.404525 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:47:09.406190 7f9d7c966700 10 _generic_write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 0~442368 = 0
2014-06-12 14:47:09.406245 7f9d7c966700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:09.406275 7f9d7c966700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:09.406281 7f9d7c966700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:09.406289 7f9d7c966700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:09.406496 7f9d7c966700 20 set_header setting 16959 cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 parent seq 0
2014-06-12 14:47:09.406560 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:09.406576 7f9d7c966700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:09.406593 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:47:09.408752 7f9d7c966700 10 _do_op 0x54101e0 seq 2580 r = 0, finisher 0x4b04640 0
2014-06-12 14:47:09.408774 7f9d7c966700 10 _finish_op 0x54101e0 seq 2580 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:09.408782 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:09.408792 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:09.408798 7f9d7715b700  5 -- op tracker -- , seq: 748329, time: 2014-06-12 14:47:09.408787, event: commit_sent, op: osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.408808 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2867 (0'0,173'2867] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2865 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:09.408849 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:09.408905 7f9d7795c700  5 -- op tracker -- , seq: 748329, time: 2014-06-12 14:47:09.408897, event: sub_op_applied, op: osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.408915 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2867 (0'0,173'2867] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2866 active] sub_op_modify_applied on 0x8223340 op osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:09.408932 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2867 (0'0,173'2867] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2866 active] op_applied on primary on version 173'2867
2014-06-12 14:47:09.408941 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2867 (0'0,173'2867] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2866 active] op_applied on replica on version 173'2867
2014-06-12 14:47:09.408958 7f9d7795c700  5 -- op tracker -- , seq: 748329, time: 2014-06-12 14:47:09.408955, event: done, op: osd_sub_op(client.4302.0:145699 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2867 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.430464 7f9d5a16f700  5 -- op tracker -- , seq: 748330, time: 2014-06-12 14:47:09.427684, event: header_read, op: osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.430484 7f9d5a16f700  5 -- op tracker -- , seq: 748330, time: 2014-06-12 14:47:09.427688, event: throttled, op: osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.430509 7f9d5a16f700  5 -- op tracker -- , seq: 748330, time: 2014-06-12 14:47:09.430409, event: all_read, op: osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.430521 7f9d5a16f700  5 -- op tracker -- , seq: 748330, time: 0.000000, event: dispatched, op: osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.430527 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:09.430536 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:09.430546 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9d100 prio 127 cost 1032999 latency 0.002862 osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:09.430587 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:09.430600 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x24c9d100 prio 127 cost 1032999 latency 0.002915 osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2513 (0'0,173'2513] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2512 active]
2014-06-12 14:47:09.430642 7f9d6b3e7700  5 -- op tracker -- , seq: 748330, time: 2014-06-12 14:47:09.430639, event: reached_pg, op: osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.430650 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2513 (0'0,173'2513] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2512 active] handle_message: 0x24c9d100
2014-06-12 14:47:09.430657 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2513 (0'0,173'2513] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2512 active] sub_op_modify trans f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 v 173'2514 (transaction) 174
2014-06-12 14:47:09.430667 7f9d6b3e7700  5 -- op tracker -- , seq: 748330, time: 2014-06-12 14:47:09.430664, event: started, op: osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.430678 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2513 (0'0,173'2513] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2512 active] append_log log((0'0,173'2513], crt=0'0) [173'2514 (173'2513) modify   f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 by client.4310.0:142329 2014-06-12 08:44:18.374210]
2014-06-12 14:47:09.430699 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2514 (0'0,173'2514] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2512 active] add_log_entry 173'2514 (173'2513) modify   f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 by client.4310.0:142329 2014-06-12 08:44:18.374210
2014-06-12 14:47:09.430714 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2514 (0'0,173'2514] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2512 active] append_log  adding 1 keys
2014-06-12 14:47:09.430739 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2514, trimmed: 
2014-06-12 14:47:09.430757 7f9d6b3e7700  5 -- op tracker -- , seq: 748330, time: 2014-06-12 14:47:09.430754, event: started, op: osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.430764 7f9d6b3e7700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:09.430767 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:09.430768 7f9d6b3e7700  5 queue_op 0x22fe8de0 seq 2360 osr(3.168 0x276ce40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:09.430773 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2514 (0'0,173'2514] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2512 active] do_sub_op osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:09.430776 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:47:09.430783 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x24c9d100 finish
2014-06-12 14:47:09.430785 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:09.430785 7f9d81970700  5 _do_op 0x22fe8de0 seq 2360 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:47:09.430786 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:09.430789 7f9d81970700 10 _do_transaction on 0x6175eb0
2014-06-12 14:47:09.430794 7f9d81970700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:09.430902 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:09.430947 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:09.431009 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:09.431042 7f9d81970700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:09.431050 7f9d81970700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:47:09.431058 7f9d81970700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:09.431163 7f9d81970700 15 _write 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 2424832~1032192
2014-06-12 14:47:09.431360 7f9d81970700 10 lookup_strip_header done  cid 3.168_head oid f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:09.431516 7f9d81970700 10 file_to_extents done 
2014-06-12 14:47:09.436266 7f9d81970700 10 _generic_write 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 2424832~1032192 = 0
2014-06-12 14:47:09.436429 7f9d81970700 15 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:09.436456 7f9d81970700 10 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 = 0
2014-06-12 14:47:09.436465 7f9d81970700 15 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:09.436477 7f9d81970700 10 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 = 0
2014-06-12 14:47:09.436869 7f9d81970700 20 set_header setting 16446 cid 3.168_head oid f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 parent seq 0
2014-06-12 14:47:09.437018 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:09.437041 7f9d81970700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:47:09.437054 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:47:09.442080 7f9d81970700 10 _do_op 0x22fe8de0 seq 2360 r = 0, finisher 0x8257ea0 0
2014-06-12 14:47:09.442160 7f9d81970700 10 _finish_op 0x22fe8de0 seq 2360 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:09.442163 7f9d7715b700  5 -- op tracker -- , seq: 748330, time: 2014-06-12 14:47:09.442142, event: commit_sent, op: osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.442181 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:47:09.442195 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:09.442188 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2514 (0'0,173'2514] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2512 active] sub_op_modify_commit on op osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:09.442227 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:09.442348 7f9d7795c700  5 -- op tracker -- , seq: 748330, time: 2014-06-12 14:47:09.442336, event: sub_op_applied, op: osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.442368 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2514 (0'0,173'2514] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2513 active] sub_op_modify_applied on 0x6175dc0 op osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:09.442389 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2514 (0'0,173'2514] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2513 active] op_applied on primary on version 173'2514
2014-06-12 14:47:09.442400 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2514 (0'0,173'2514] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2513 active] op_applied on replica on version 173'2514
2014-06-12 14:47:09.442428 7f9d7795c700  5 -- op tracker -- , seq: 748330, time: 2014-06-12 14:47:09.442420, event: done, op: osd_sub_op(client.4310.0:142329 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2514 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.482261 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:09.482294 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:09.482320 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 9.62 >= max 0.5 = no, load too high
2014-06-12 14:47:09.482331 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:47:09.482333 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92677 < max (604800 seconds)
2014-06-12 14:47:09.482357 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:47:09.482359 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:09.482360 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:09.482362 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 27; oldest is 29.173919 seconds old
2014-06-12 14:47:09.714733 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:09.714786 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:09.811466 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:09.811513 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:09.825247 7f9d5a16f700  5 -- op tracker -- , seq: 748331, time: 2014-06-12 14:47:09.821560, event: header_read, op: osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.825268 7f9d5a16f700  5 -- op tracker -- , seq: 748331, time: 2014-06-12 14:47:09.821564, event: throttled, op: osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.825279 7f9d5a16f700  5 -- op tracker -- , seq: 748331, time: 2014-06-12 14:47:09.825184, event: all_read, op: osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.825290 7f9d5a16f700  5 -- op tracker -- , seq: 748331, time: 0.000000, event: dispatched, op: osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.825297 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:09.825307 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:09.825318 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9ce00 prio 127 cost 1561383 latency 0.003758 osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:09.825376 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:09.825392 7f9d693e3700 10 osd.2 173 dequeue_op 0x24c9ce00 prio 127 cost 1561383 latency 0.003832 osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2674 (0'0,173'2674] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active]
2014-06-12 14:47:09.825450 7f9d693e3700  5 -- op tracker -- , seq: 748331, time: 2014-06-12 14:47:09.825439, event: reached_pg, op: osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.825472 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2674 (0'0,173'2674] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active] handle_message: 0x24c9ce00
2014-06-12 14:47:09.825497 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2674 (0'0,173'2674] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active] sub_op_modify trans 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 v 173'2675 (transaction) 174
2014-06-12 14:47:09.825536 7f9d693e3700  5 -- op tracker -- , seq: 748331, time: 2014-06-12 14:47:09.825528, event: started, op: osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.825580 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2674 (0'0,173'2674] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active] append_log log((0'0,173'2674], crt=0'0) [173'2675 (173'2674) modify   57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 by client.4282.0:145229 2014-06-12 08:44:33.145474]
2014-06-12 14:47:09.825649 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2675 (0'0,173'2675] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active] add_log_entry 173'2675 (173'2674) modify   57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 by client.4282.0:145229 2014-06-12 08:44:33.145474
2014-06-12 14:47:09.825696 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2675 (0'0,173'2675] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active] append_log  adding 1 keys
2014-06-12 14:47:09.825756 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2675, trimmed: 
2014-06-12 14:47:09.825804 7f9d693e3700  5 -- op tracker -- , seq: 748331, time: 2014-06-12 14:47:09.825793, event: started, op: osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.825838 7f9d693e3700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:09.825846 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:09.825848 7f9d693e3700  5 queue_op 0x829a4e0 seq 2485 osr(3.1b7 0x276d1a0) 1562869 bytes   (queue has 1 ops and 1562869 bytes)
2014-06-12 14:47:09.825859 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2675 (0'0,173'2675] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active] do_sub_op osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:09.825892 7f9d693e3700 10 osd.2 173 dequeue_op 0x24c9ce00 finish
2014-06-12 14:47:09.825901 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:09.825895 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:09.825907 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:09.825911 7f9d7a962700  5 _do_op 0x829a4e0 seq 2485 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:09.825919 7f9d7a962700 10 _do_transaction on 0x62957b0
2014-06-12 14:47:09.825929 7f9d7a962700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:09.826065 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:09.826101 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:09.826159 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:09.826190 7f9d7a962700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:09.826198 7f9d7a962700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:09.826206 7f9d7a962700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:09.826434 7f9d7a962700 15 _write 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 2633728~1560576
2014-06-12 14:47:09.826662 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:09.826701 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:09.826728 7f9d7a962700 10 lookup_strip_header done  cid 3.1b7_head oid 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:09.826989 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:47:09.834394 7f9d7a962700 10 _generic_write 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 2633728~1560576 = 0
2014-06-12 14:47:09.834519 7f9d7a962700 15 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:09.834537 7f9d7a962700 10 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:47:09.834545 7f9d7a962700 15 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3
2014-06-12 14:47:09.834552 7f9d7a962700 10 _setattrs 3.1b7_head/57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 = 0
2014-06-12 14:47:09.834839 7f9d7a962700 20 set_header setting 18080 cid 3.1b7_head oid 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 parent seq 0
2014-06-12 14:47:09.834943 7f9d7a962700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:09.834958 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:09.834965 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:47:09.841453 7f9d7a962700 10 _do_op 0x829a4e0 seq 2485 r = 0, finisher 0x5243e80 0
2014-06-12 14:47:09.841471 7f9d7a962700 10 _finish_op 0x829a4e0 seq 2485 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:09.841476 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:09.841479 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:09.841496 7f9d7795c700  5 -- op tracker -- , seq: 748331, time: 2014-06-12 14:47:09.841486, event: sub_op_applied, op: osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.841506 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2675 (0'0,173'2675] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active] sub_op_modify_applied on 0x62956c0 op osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:09.841532 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:09.841554 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2675 (0'0,173'2675] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active] op_applied on primary on version 173'2675
2014-06-12 14:47:09.841568 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2675 (0'0,173'2675] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active] op_applied on replica on version 173'2675
2014-06-12 14:47:09.841597 7f9d7715b700  5 -- op tracker -- , seq: 748331, time: 2014-06-12 14:47:09.841589, event: commit_sent, op: osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.841607 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2675 (0'0,173'2675] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2673 active] sub_op_modify_commit on op osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:09.841636 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:09.841671 7f9d7715b700  5 -- op tracker -- , seq: 748331, time: 2014-06-12 14:47:09.841666, event: done, op: osd_sub_op(client.4282.0:145229 3.1b7 57f3c9b7/rbd_data.10916b8b4567.0000000000001085/head//3 [] v 173'2675 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:09.847881 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:09.847886 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:09.847888 7f9d74bfa700 20 osd.2 173 _dispatch 0x104f6b60 ping v1
2014-06-12 14:47:09.847897 7f9d74bfa700 10 osd.2 173 ping from client.4290
2014-06-12 14:47:09.847899 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:09.847900 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:09.849862 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:09.849891 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.050177 7f9d5a16f700  5 -- op tracker -- , seq: 748332, time: 2014-06-12 14:47:10.048486, event: header_read, op: osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.050207 7f9d5a16f700  5 -- op tracker -- , seq: 748332, time: 2014-06-12 14:47:10.048494, event: throttled, op: osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.050225 7f9d5a16f700  5 -- op tracker -- , seq: 748332, time: 2014-06-12 14:47:10.050101, event: all_read, op: osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.050240 7f9d5a16f700  5 -- op tracker -- , seq: 748332, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.050250 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:10.050264 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:10.050279 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9d200 prio 127 cost 516903 latency 0.001792 osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.050330 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:10.050347 7f9d6abe6700 10 osd.2 173 dequeue_op 0x24c9d200 prio 127 cost 516903 latency 0.001861 osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2867 (0'0,173'2867] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2866 active]
2014-06-12 14:47:10.050388 7f9d6abe6700  5 -- op tracker -- , seq: 748332, time: 2014-06-12 14:47:10.050382, event: reached_pg, op: osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.050407 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2867 (0'0,173'2867] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2866 active] handle_message: 0x24c9d200
2014-06-12 14:47:10.050424 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2867 (0'0,173'2867] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2866 active] sub_op_modify trans f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 v 173'2868 (transaction) 174
2014-06-12 14:47:10.050446 7f9d6abe6700  5 -- op tracker -- , seq: 748332, time: 2014-06-12 14:47:10.050440, event: started, op: osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.050468 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2867 (0'0,173'2867] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2866 active] append_log log((0'0,173'2867], crt=0'0) [173'2868 (173'2867) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145700 2014-06-12 08:43:42.977256]
2014-06-12 14:47:10.050525 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2868 (0'0,173'2868] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2866 active] add_log_entry 173'2868 (173'2867) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145700 2014-06-12 08:43:42.977256
2014-06-12 14:47:10.050590 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2868 (0'0,173'2868] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2866 active] append_log  adding 1 keys
2014-06-12 14:47:10.050659 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2868, trimmed: 
2014-06-12 14:47:10.050715 7f9d6abe6700  5 -- op tracker -- , seq: 748332, time: 2014-06-12 14:47:10.050703, event: started, op: osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.050734 7f9d6abe6700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:10.050742 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:10.050744 7f9d6abe6700  5 queue_op 0x832c900 seq 2581 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:10.050765 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:10.050777 7f9d82972700  5 _do_op 0x832c900 seq 2581 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:10.050759 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2868 (0'0,173'2868] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2866 active] do_sub_op osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.050786 7f9d82972700 10 _do_transaction on 0x66690b0
2014-06-12 14:47:10.050790 7f9d6abe6700 10 osd.2 173 dequeue_op 0x24c9d200 finish
2014-06-12 14:47:10.050798 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:10.050803 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.050802 7f9d82972700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:10.050968 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:10.051000 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:10.051062 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:10.051088 7f9d82972700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:10.051096 7f9d82972700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:10.051104 7f9d82972700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:10.051126 7f9d82972700 15 _write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 442368~516096
2014-06-12 14:47:10.051593 7f9d82972700 10 lookup_strip_header done  cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:10.051833 7f9d82972700 10 file_to_extents done 
2014-06-12 14:47:10.054193 7f9d82972700 10 _generic_write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 442368~516096 = 0
2014-06-12 14:47:10.054275 7f9d82972700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:10.054319 7f9d82972700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:10.054334 7f9d82972700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:10.054354 7f9d82972700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:10.054814 7f9d82972700 20 set_header setting 16959 cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 parent seq 0
2014-06-12 14:47:10.054989 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:10.055022 7f9d82972700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:10.055046 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:47:10.060105 7f9d82972700 10 _do_op 0x832c900 seq 2581 r = 0, finisher 0x172ff5c0 0
2014-06-12 14:47:10.060136 7f9d82972700 10 _finish_op 0x832c900 seq 2581 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:10.060172 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:10.060187 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.060172 7f9d7715b700  5 -- op tracker -- , seq: 748332, time: 2014-06-12 14:47:10.060151, event: commit_sent, op: osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.060195 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2868 (0'0,173'2868] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2866 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:10.060244 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:10.060315 7f9d7795c700  5 -- op tracker -- , seq: 748332, time: 2014-06-12 14:47:10.060301, event: sub_op_applied, op: osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.060337 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2868 (0'0,173'2868] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2867 active] sub_op_modify_applied on 0x6668fc0 op osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.060373 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2868 (0'0,173'2868] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2867 active] op_applied on primary on version 173'2868
2014-06-12 14:47:10.060394 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2868 (0'0,173'2868] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2867 active] op_applied on replica on version 173'2868
2014-06-12 14:47:10.060441 7f9d7795c700  5 -- op tracker -- , seq: 748332, time: 2014-06-12 14:47:10.060432, event: done, op: osd_sub_op(client.4302.0:145700 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2868 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.064530 7f9d5a16f700  5 -- op tracker -- , seq: 748333, time: 2014-06-12 14:47:10.063659, event: header_read, op: osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.064544 7f9d5a16f700  5 -- op tracker -- , seq: 748333, time: 2014-06-12 14:47:10.063663, event: throttled, op: osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.064552 7f9d5a16f700  5 -- op tracker -- , seq: 748333, time: 2014-06-12 14:47:10.064491, event: all_read, op: osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.064565 7f9d5a16f700  5 -- op tracker -- , seq: 748333, time: 0.000000, event: dispatched, op: osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.064569 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:10.064576 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:10.064584 7f9d5a16f700 15 osd.2 173 enqueue_op 0x20355600 prio 127 cost 516903 latency 0.000925 osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.064874 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:10.064884 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x20355600 prio 127 cost 516903 latency 0.001225 osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2514 (0'0,173'2514] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2513 active]
2014-06-12 14:47:10.064907 7f9d6a3e5700  5 -- op tracker -- , seq: 748333, time: 2014-06-12 14:47:10.064903, event: reached_pg, op: osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.064916 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2514 (0'0,173'2514] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2513 active] handle_message: 0x20355600
2014-06-12 14:47:10.064927 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2514 (0'0,173'2514] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2513 active] sub_op_modify trans f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 v 173'2515 (transaction) 174
2014-06-12 14:47:10.064941 7f9d6a3e5700  5 -- op tracker -- , seq: 748333, time: 2014-06-12 14:47:10.064938, event: started, op: osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.064953 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2514 (0'0,173'2514] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2513 active] append_log log((0'0,173'2514], crt=0'0) [173'2515 (173'2514) modify   f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 by client.4310.0:142330 2014-06-12 08:44:23.114492]
2014-06-12 14:47:10.064974 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2515 (0'0,173'2515] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2513 active] add_log_entry 173'2515 (173'2514) modify   f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 by client.4310.0:142330 2014-06-12 08:44:23.114492
2014-06-12 14:47:10.064993 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2515 (0'0,173'2515] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2513 active] append_log  adding 1 keys
2014-06-12 14:47:10.065021 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2515, trimmed: 
2014-06-12 14:47:10.065046 7f9d6a3e5700  5 -- op tracker -- , seq: 748333, time: 2014-06-12 14:47:10.065041, event: started, op: osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.065055 7f9d6a3e5700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:10.065058 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:10.065060 7f9d6a3e5700  5 queue_op 0x5e6c840 seq 2361 osr(3.168 0x276ce40) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:10.065067 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2515 (0'0,173'2515] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2513 active] do_sub_op osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.065085 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x20355600 finish
2014-06-12 14:47:10.065089 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:10.065091 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.065101 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:47:10.065106 7f9d7895e700  5 _do_op 0x5e6c840 seq 2361 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:47:10.065111 7f9d7895e700 10 _do_transaction on 0x8dcf0b0
2014-06-12 14:47:10.065116 7f9d7895e700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:10.065286 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:10.065359 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:10.065423 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:10.065450 7f9d7895e700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:10.065456 7f9d7895e700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:47:10.065462 7f9d7895e700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:10.065673 7f9d7895e700 15 _write 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 3457024~516096
2014-06-12 14:47:10.065844 7f9d7895e700 10 lookup_strip_header done  cid 3.168_head oid f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:10.066094 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:47:10.068495 7f9d7895e700 10 _generic_write 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 3457024~516096 = 0
2014-06-12 14:47:10.068558 7f9d7895e700 15 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:10.068573 7f9d7895e700 10 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 = 0
2014-06-12 14:47:10.068579 7f9d7895e700 15 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:10.068587 7f9d7895e700 10 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 = 0
2014-06-12 14:47:10.068806 7f9d7895e700 20 set_header setting 16446 cid 3.168_head oid f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 parent seq 0
2014-06-12 14:47:10.068938 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:10.068960 7f9d7895e700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:47:10.068973 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:47:10.072497 7f9d7895e700 10 _do_op 0x5e6c840 seq 2361 r = 0, finisher 0x31d36700 0
2014-06-12 14:47:10.072519 7f9d7895e700 10 _finish_op 0x5e6c840 seq 2361 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:10.072536 7f9d7715b700  5 -- op tracker -- , seq: 748333, time: 2014-06-12 14:47:10.072527, event: commit_sent, op: osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.072537 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:47:10.072546 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.072545 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2515 (0'0,173'2515] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2513 active] sub_op_modify_commit on op osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:10.072571 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:10.072601 7f9d7795c700  5 -- op tracker -- , seq: 748333, time: 2014-06-12 14:47:10.072595, event: sub_op_applied, op: osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.072609 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2515 (0'0,173'2515] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2514 active] sub_op_modify_applied on 0x8dcefc0 op osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.072620 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2515 (0'0,173'2515] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2514 active] op_applied on primary on version 173'2515
2014-06-12 14:47:10.072626 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2515 (0'0,173'2515] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2514 active] op_applied on replica on version 173'2515
2014-06-12 14:47:10.072641 7f9d7795c700  5 -- op tracker -- , seq: 748333, time: 2014-06-12 14:47:10.072639, event: done, op: osd_sub_op(client.4310.0:142330 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2515 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.087791 7f9d47c58700  5 -- op tracker -- , seq: 748334, time: 2014-06-12 14:47:10.086574, event: header_read, op: osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:10.087814 7f9d47c58700  5 -- op tracker -- , seq: 748334, time: 2014-06-12 14:47:10.086584, event: throttled, op: osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:10.087823 7f9d47c58700  5 -- op tracker -- , seq: 748334, time: 2014-06-12 14:47:10.087747, event: all_read, op: osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:10.087832 7f9d47c58700  5 -- op tracker -- , seq: 748334, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:10.087841 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:47:10.087868 7f9d47c58700 15 osd.2 173 enqueue_op 0x3b42be00 prio 63 cost 516096 latency 0.001294 osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:10.087947 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:10.087962 7f9d66bde700 10 osd.2 173 dequeue_op 0x3b42be00 prio 63 cost 516096 latency 0.001388 osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]
2014-06-12 14:47:10.087992 7f9d66bde700  5 -- op tracker -- , seq: 748334, time: 2014-06-12 14:47:10.087986, event: reached_pg, op: osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:10.088005 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:10.088020 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] handle_message: 0x3b42be00
2014-06-12 14:47:10.088029 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:10.088053 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: found obc in cache: 0xedc6b00
2014-06-12 14:47:10.088063 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: 0xedc6b00 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 rwstate(write n=1 w=0) oi: 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'2976 client.4316.0:147318 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2976) ssc: 0x1550f180 snapset: 0=[]:[]+head
2014-06-12 14:47:10.088079 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] find_object_context 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 @head oi=58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'2976 client.4316.0:147318 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2976)
2014-06-12 14:47:10.088103 7f9d66bde700  5 -- op tracker -- , seq: 748334, time: 2014-06-12 14:47:10.088098, event: started, op: osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:10.088110 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] execute_ctx 0x580d000
2014-06-12 14:47:10.088125 7f9d66bde700  5 -- op tracker -- , seq: 748334, time: 2014-06-12 14:47:10.088121, event: started, op: osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:10.088131 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] ov 173'2976 av 173'2977 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:10.088143 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096]
2014-06-12 14:47:10.088152 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:10.088163 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  write 258048~516096
2014-06-12 14:47:10.088178 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:47:10.088188 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 snapset=0x1550f1b8  snapc=0=[]
2014-06-12 14:47:10.088199 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:10.088208 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] finish_ctx 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 0x580d000 op modify  
2014-06-12 14:47:10.088219 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  set mtime to 2014-06-12 08:46:12.465574
2014-06-12 14:47:10.088239 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  final snapset 0=[]:[]+head in 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:10.088256 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  zeroing write result code 0
2014-06-12 14:47:10.088266 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] new_repop rep_tid 236407 on osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:10.088281 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] issue_repop rep_tid 236407 o 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:10.088302 7f9d66bde700  5 -- op tracker -- , seq: 748334, time: 2014-06-12 14:47:10.088298, event: waiting for subops from 24, op: osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:10.088323 7f9d66bde700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:10.088362 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2976 (0'0,173'2976] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log log((0'0,173'2976], crt=0'0) [173'2977 (173'2976) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147319 2014-06-12 08:46:12.465574]
2014-06-12 14:47:10.088384 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] add_log_entry 173'2977 (173'2976) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147319 2014-06-12 08:46:12.465574
2014-06-12 14:47:10.088403 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log  adding 1 keys
2014-06-12 14:47:10.088437 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2977, trimmed: 
2014-06-12 14:47:10.088454 7f9d66bde700  5 queue_transactions existing osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:10.088458 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:10.088459 7f9d66bde700  5 queue_op 0x6ca6f00 seq 2706 osr(3.785 0x49b40d0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:10.088465 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] eval_repop repgather(0x31b64000 173'2977 rep_tid=236407 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:10.088485 7f9d66bde700 10 osd.2 173 dequeue_op 0x3b42be00 finish
2014-06-12 14:47:10.088488 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:10.088489 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.088497 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54bc360 (1 active)
2014-06-12 14:47:10.088505 7f9d83173700  5 _do_op 0x6ca6f00 seq 2706 osr(3.785 0x49b40d0)/0x49b40d0 start
2014-06-12 14:47:10.088510 7f9d83173700 10 _do_transaction on 0x96dbbc0
2014-06-12 14:47:10.088523 7f9d83173700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:10.088622 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:10.088644 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:10.088679 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:10.088694 7f9d83173700 15 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:10.088697 7f9d83173700 10 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1 r = 0
2014-06-12 14:47:10.088700 7f9d83173700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:10.088712 7f9d83173700 15 _write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 258048~516096
2014-06-12 14:47:10.088844 7f9d83173700 10 lookup_strip_header done  cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:10.089189 7f9d83173700 10 file_to_extents done 
2014-06-12 14:47:10.091514 7f9d83173700 10 _generic_write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 258048~516096 = 0
2014-06-12 14:47:10.091579 7f9d83173700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:10.091600 7f9d83173700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:10.091614 7f9d83173700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:10.091625 7f9d83173700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:10.091895 7f9d83173700 20 set_header setting 17484 cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 parent seq 0
2014-06-12 14:47:10.091982 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:10.092002 7f9d83173700 20 set_header setting 2142 cid meta oid 1fe0af7a/pglog_3.785/0//-1 parent seq 0
2014-06-12 14:47:10.092022 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:47:10.095540 7f9d83173700 10 _do_op 0x6ca6f00 seq 2706 r = 0, finisher 0x6606c80 0x38e0d500
2014-06-12 14:47:10.095564 7f9d83173700 10 _finish_op 0x6ca6f00 seq 2706 osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:10.095578 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54bc360 (0 active)
2014-06-12 14:47:10.095588 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.095575 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_commit: 236407
2014-06-12 14:47:10.095627 7f9d7715b700  5 -- op tracker -- , seq: 748334, time: 2014-06-12 14:47:10.095613, event: op_commit, op: osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:10.095681 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied: 236407
2014-06-12 14:47:10.095734 7f9d7795c700  5 -- op tracker -- , seq: 748334, time: 2014-06-12 14:47:10.095722, event: op_applied, op: osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:10.095755 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied on primary on version 173'2977
2014-06-12 14:47:10.125046 7f9d5501e700  5 -- op tracker -- , seq: 748335, time: 2014-06-12 14:47:10.124185, event: header_read, op: osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.125069 7f9d5501e700  5 -- op tracker -- , seq: 748335, time: 2014-06-12 14:47:10.124190, event: throttled, op: osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.125098 7f9d5501e700  5 -- op tracker -- , seq: 748335, time: 2014-06-12 14:47:10.124985, event: all_read, op: osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.125109 7f9d5501e700  5 -- op tracker -- , seq: 748335, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.125117 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:10.125128 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:10.125140 7f9d5501e700 15 osd.2 173 enqueue_op 0x329eb000 prio 127 cost 299815 latency 0.000955 osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.125214 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:10.125236 7f9d69be4700 10 osd.2 173 dequeue_op 0x329eb000 prio 127 cost 299815 latency 0.001051 osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3c6( v 173'3219 (79'218,173'3219] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3218 active]
2014-06-12 14:47:10.125280 7f9d69be4700  5 -- op tracker -- , seq: 748335, time: 2014-06-12 14:47:10.125273, event: reached_pg, op: osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.125300 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3219 (79'218,173'3219] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3218 active] handle_message: 0x329eb000
2014-06-12 14:47:10.125326 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3219 (79'218,173'3219] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3218 active] sub_op_modify trans 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 v 173'3220 (transaction) 174
2014-06-12 14:47:10.125349 7f9d69be4700  5 -- op tracker -- , seq: 748335, time: 2014-06-12 14:47:10.125343, event: started, op: osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.125376 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3219 (79'218,173'3219] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3218 active] append_log log((79'218,173'3219], crt=0'0) [173'3220 (173'902) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142573 2014-06-12 08:46:12.460454]
2014-06-12 14:47:10.125413 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3220 (79'218,173'3220] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3218 active] add_log_entry 173'3220 (173'902) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142573 2014-06-12 08:46:12.460454
2014-06-12 14:47:10.125447 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3220 (79'218,173'3220] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3218 active] append_log  adding 1 keys
2014-06-12 14:47:10.125468 7f9d69be4700 10 trim log((79'218,173'3220], crt=0'0) to 79'219
2014-06-12 14:47:10.125505 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3220, trimmed: 79'219
2014-06-12 14:47:10.125544 7f9d69be4700  5 -- op tracker -- , seq: 748335, time: 2014-06-12 14:47:10.125536, event: started, op: osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.125565 7f9d69be4700  5 queue_transactions existing osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.125571 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:10.125573 7f9d69be4700  5 queue_op 0x58da840 seq 2921 osr(3.3c6 0x276d6d0) 301336 bytes   (queue has 1 ops and 301336 bytes)
2014-06-12 14:47:10.125591 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540efc0 (1 active)
2014-06-12 14:47:10.125603 7f9d7a161700  5 _do_op 0x58da840 seq 2921 osr(3.3c6 0x276d6d0)/0x276d6d0 start
2014-06-12 14:47:10.125583 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3220 (79'219,173'3220] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3218 active] do_sub_op osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.125611 7f9d7a161700 10 _do_transaction on 0x601ab70
2014-06-12 14:47:10.125612 7f9d69be4700 10 osd.2 173 dequeue_op 0x329eb000 finish
2014-06-12 14:47:10.125643 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:10.125648 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.125645 7f9d7a161700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.125788 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.125823 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:10.125885 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:10.125911 7f9d7a161700 15 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.125924 7f9d7a161700 10 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1 r = 0
2014-06-12 14:47:10.125933 7f9d7a161700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.125951 7f9d7a161700 15 _write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 0~299008
2014-06-12 14:47:10.126191 7f9d7a161700 10 lookup_strip_header done  cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.126287 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:47:10.127624 7f9d7a161700 10 _generic_write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 0~299008 = 0
2014-06-12 14:47:10.127665 7f9d7a161700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.127682 7f9d7a161700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.127691 7f9d7a161700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.127701 7f9d7a161700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.127916 7f9d7a161700 20 set_header setting 16491 cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 parent seq 0
2014-06-12 14:47:10.127978 7f9d7a161700 20 set_header setting 2260 cid meta oid 1fe0aae5/pglog_3.3c6/0//-1 parent seq 0
2014-06-12 14:47:10.127999 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:10.128012 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:47:10.129873 7f9d7a161700 10 _do_op 0x58da840 seq 2921 r = 0, finisher 0x20aaddc0 0
2014-06-12 14:47:10.129893 7f9d7a161700 10 _finish_op 0x58da840 seq 2921 osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.129901 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540efc0 (0 active)
2014-06-12 14:47:10.129905 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.129939 7f9d7795c700  5 -- op tracker -- , seq: 748335, time: 2014-06-12 14:47:10.129920, event: sub_op_applied, op: osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.129957 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3220 (79'219,173'3220] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3218 active] sub_op_modify_applied on 0x601aa80 op osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.130004 7f9d7795c700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.130042 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3220 (79'219,173'3220] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3218 active] op_applied on primary on version 173'3220
2014-06-12 14:47:10.130073 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3220 (79'219,173'3220] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3218 active] op_applied on replica on version 173'3220
2014-06-12 14:47:10.130155 7f9d7715b700  5 -- op tracker -- , seq: 748335, time: 2014-06-12 14:47:10.130140, event: commit_sent, op: osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.130180 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3220 (79'219,173'3220] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3218 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:47:10.130213 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.130274 7f9d7715b700  5 -- op tracker -- , seq: 748335, time: 2014-06-12 14:47:10.130266, event: done, op: osd_sub_op(client.4305.0:142573 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3220 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.155789 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:10.160502 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:10.160507 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:10.160509 7f9d74bfa700 20 osd.2 173 _dispatch 0x32118d00 ping v1
2014-06-12 14:47:10.160513 7f9d74bfa700 10 osd.2 173 ping from client.4293
2014-06-12 14:47:10.160516 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:10.160518 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:10.226599 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:10.226670 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:10.327474 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:10.327508 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:10.370286 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:10.430238 7f9d5501e700  5 -- op tracker -- , seq: 748336, time: 2014-06-12 14:47:10.429336, event: header_read, op: osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.430259 7f9d5501e700  5 -- op tracker -- , seq: 748336, time: 2014-06-12 14:47:10.429339, event: throttled, op: osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.430267 7f9d5501e700  5 -- op tracker -- , seq: 748336, time: 2014-06-12 14:47:10.430197, event: all_read, op: osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.430276 7f9d5501e700  5 -- op tracker -- , seq: 748336, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.430282 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:10.430289 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:10.430298 7f9d5501e700 15 osd.2 173 enqueue_op 0x329eb200 prio 127 cost 516903 latency 0.000961 osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.430364 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:10.430375 7f9d64bda700 10 osd.2 173 dequeue_op 0x329eb200 prio 127 cost 516903 latency 0.001038 osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3c6( v 173'3220 (79'219,173'3220] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3219 active]
2014-06-12 14:47:10.430402 7f9d64bda700  5 -- op tracker -- , seq: 748336, time: 2014-06-12 14:47:10.430399, event: reached_pg, op: osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.430421 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3220 (79'219,173'3220] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3219 active] handle_message: 0x329eb200
2014-06-12 14:47:10.430430 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3220 (79'219,173'3220] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3219 active] sub_op_modify trans 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 v 173'3221 (transaction) 174
2014-06-12 14:47:10.430441 7f9d64bda700  5 -- op tracker -- , seq: 748336, time: 2014-06-12 14:47:10.430438, event: started, op: osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.430452 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3220 (79'219,173'3220] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3219 active] append_log log((79'219,173'3220], crt=0'0) [173'3221 (173'3220) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142574 2014-06-12 08:46:12.805886]
2014-06-12 14:47:10.430470 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3221 (79'219,173'3221] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3219 active] add_log_entry 173'3221 (173'3220) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142574 2014-06-12 08:46:12.805886
2014-06-12 14:47:10.430486 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3221 (79'219,173'3221] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3219 active] append_log  adding 1 keys
2014-06-12 14:47:10.430495 7f9d64bda700 10 trim log((79'219,173'3221], crt=0'0) to 79'220
2014-06-12 14:47:10.430514 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3221, trimmed: 79'220
2014-06-12 14:47:10.430553 7f9d64bda700  5 -- op tracker -- , seq: 748336, time: 2014-06-12 14:47:10.430549, event: started, op: osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.430563 7f9d64bda700  5 queue_transactions existing osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.430567 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:10.430569 7f9d64bda700  5 queue_op 0x177b5920 seq 2922 osr(3.3c6 0x276d6d0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:10.430579 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540efc0 (1 active)
2014-06-12 14:47:10.430586 7f9d7f16b700  5 _do_op 0x177b5920 seq 2922 osr(3.3c6 0x276d6d0)/0x276d6d0 start
2014-06-12 14:47:10.430575 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3221 (79'220,173'3221] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3219 active] do_sub_op osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.430588 7f9d64bda700 10 osd.2 173 dequeue_op 0x329eb200 finish
2014-06-12 14:47:10.430590 7f9d7f16b700 10 _do_transaction on 0x52c8ef0
2014-06-12 14:47:10.430592 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:10.430594 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.430596 7f9d7f16b700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.430710 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.430742 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:10.430774 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:10.430805 7f9d7f16b700 15 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.430811 7f9d7f16b700 10 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1 r = 0
2014-06-12 14:47:10.430814 7f9d7f16b700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.430911 7f9d7f16b700 15 _write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 299008~516096
2014-06-12 14:47:10.431027 7f9d7f16b700 10 lookup_strip_header done  cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.431135 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:47:10.432350 7f9d7f16b700 10 _generic_write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 299008~516096 = 0
2014-06-12 14:47:10.432384 7f9d7f16b700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.432393 7f9d7f16b700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.432396 7f9d7f16b700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.432401 7f9d7f16b700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.432541 7f9d7f16b700 20 set_header setting 16491 cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 parent seq 0
2014-06-12 14:47:10.432587 7f9d7f16b700 20 set_header setting 2260 cid meta oid 1fe0aae5/pglog_3.3c6/0//-1 parent seq 0
2014-06-12 14:47:10.432598 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:10.432605 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:47:10.434167 7f9d7f16b700 10 _do_op 0x177b5920 seq 2922 r = 0, finisher 0x65e51c0 0
2014-06-12 14:47:10.434179 7f9d7f16b700 10 _finish_op 0x177b5920 seq 2922 osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.434183 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540efc0 (0 active)
2014-06-12 14:47:10.434189 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.434217 7f9d7715b700  5 -- op tracker -- , seq: 748336, time: 2014-06-12 14:47:10.434207, event: commit_sent, op: osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.434227 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3221 (79'220,173'3221] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3219 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:47:10.434254 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.434315 7f9d7795c700  5 -- op tracker -- , seq: 748336, time: 2014-06-12 14:47:10.434309, event: sub_op_applied, op: osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.434327 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3221 (79'220,173'3221] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3220 active] sub_op_modify_applied on 0x52c8e00 op osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.434347 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3221 (79'220,173'3221] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3220 active] op_applied on primary on version 173'3221
2014-06-12 14:47:10.434359 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3221 (79'220,173'3221] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3220 active] op_applied on replica on version 173'3221
2014-06-12 14:47:10.434379 7f9d7795c700  5 -- op tracker -- , seq: 748336, time: 2014-06-12 14:47:10.434376, event: done, op: osd_sub_op(client.4305.0:142574 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3221 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.440925 7f9d5501e700  5 -- op tracker -- , seq: 748337, time: 2014-06-12 14:47:10.439526, event: header_read, op: osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.440936 7f9d5501e700  5 -- op tracker -- , seq: 748337, time: 2014-06-12 14:47:10.439528, event: throttled, op: osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.440948 7f9d5501e700  5 -- op tracker -- , seq: 748337, time: 2014-06-12 14:47:10.440890, event: all_read, op: osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.440953 7f9d5501e700  5 -- op tracker -- , seq: 748337, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.440957 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:10.440961 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:10.440967 7f9d5501e700 15 osd.2 173 enqueue_op 0x329ea600 prio 127 cost 516903 latency 0.001441 osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.440990 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:10.440996 7f9d653db700 10 osd.2 173 dequeue_op 0x329ea600 prio 127 cost 516903 latency 0.001470 osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3c6( v 173'3221 (79'220,173'3221] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3220 active]
2014-06-12 14:47:10.441014 7f9d653db700  5 -- op tracker -- , seq: 748337, time: 2014-06-12 14:47:10.441011, event: reached_pg, op: osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.441023 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3221 (79'220,173'3221] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3220 active] handle_message: 0x329ea600
2014-06-12 14:47:10.441032 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3221 (79'220,173'3221] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3220 active] sub_op_modify trans 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 v 173'3222 (transaction) 174
2014-06-12 14:47:10.441043 7f9d653db700  5 -- op tracker -- , seq: 748337, time: 2014-06-12 14:47:10.441041, event: started, op: osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.441055 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3221 (79'220,173'3221] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3220 active] append_log log((79'220,173'3221], crt=0'0) [173'3222 (173'3221) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142575 2014-06-12 08:46:12.816206]
2014-06-12 14:47:10.441074 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3222 (79'220,173'3222] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3220 active] add_log_entry 173'3222 (173'3221) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142575 2014-06-12 08:46:12.816206
2014-06-12 14:47:10.441090 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3222 (79'220,173'3222] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3220 active] append_log  adding 1 keys
2014-06-12 14:47:10.441101 7f9d653db700 10 trim log((79'220,173'3222], crt=0'0) to 79'221
2014-06-12 14:47:10.441122 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3222, trimmed: 79'221
2014-06-12 14:47:10.441141 7f9d653db700  5 -- op tracker -- , seq: 748337, time: 2014-06-12 14:47:10.441138, event: started, op: osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.441150 7f9d653db700  5 queue_transactions existing osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.441153 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:10.441154 7f9d653db700  5 queue_op 0x7695080 seq 2923 osr(3.3c6 0x276d6d0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:10.441162 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540efc0 (1 active)
2014-06-12 14:47:10.441168 7f9d7e169700  5 _do_op 0x7695080 seq 2923 osr(3.3c6 0x276d6d0)/0x276d6d0 start
2014-06-12 14:47:10.441159 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3222 (79'221,173'3222] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3220 active] do_sub_op osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.441172 7f9d653db700 10 osd.2 173 dequeue_op 0x329ea600 finish
2014-06-12 14:47:10.441173 7f9d7e169700 10 _do_transaction on 0x5afc630
2014-06-12 14:47:10.441175 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:10.441178 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.441179 7f9d7e169700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.441264 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.441289 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:10.441334 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:10.441354 7f9d7e169700 15 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.441364 7f9d7e169700 10 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1 r = 0
2014-06-12 14:47:10.441370 7f9d7e169700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.441385 7f9d7e169700 15 _write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 815104~516096
2014-06-12 14:47:10.441490 7f9d7e169700 10 lookup_strip_header done  cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.441593 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:47:10.443183 7f9d7e169700 10 _generic_write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 815104~516096 = 0
2014-06-12 14:47:10.443248 7f9d7e169700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.443266 7f9d7e169700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.443274 7f9d7e169700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.443283 7f9d7e169700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.443432 7f9d7e169700 20 set_header setting 16491 cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 parent seq 0
2014-06-12 14:47:10.443486 7f9d7e169700 20 set_header setting 2260 cid meta oid 1fe0aae5/pglog_3.3c6/0//-1 parent seq 0
2014-06-12 14:47:10.443504 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:10.443516 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:47:10.446412 7f9d7e169700 10 _do_op 0x7695080 seq 2923 r = 0, finisher 0x895c460 0
2014-06-12 14:47:10.446433 7f9d7e169700 10 _finish_op 0x7695080 seq 2923 osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.446441 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540efc0 (0 active)
2014-06-12 14:47:10.446453 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.446451 7f9d7715b700  5 -- op tracker -- , seq: 748337, time: 2014-06-12 14:47:10.446440, event: commit_sent, op: osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.446470 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3222 (79'221,173'3222] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3220 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:47:10.446501 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.446543 7f9d7795c700  5 -- op tracker -- , seq: 748337, time: 2014-06-12 14:47:10.446535, event: sub_op_applied, op: osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.446555 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3222 (79'221,173'3222] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active] sub_op_modify_applied on 0x5afc540 op osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.446573 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3222 (79'221,173'3222] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active] op_applied on primary on version 173'3222
2014-06-12 14:47:10.446583 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3222 (79'221,173'3222] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active] op_applied on replica on version 173'3222
2014-06-12 14:47:10.446604 7f9d7795c700  5 -- op tracker -- , seq: 748337, time: 2014-06-12 14:47:10.446599, event: done, op: osd_sub_op(client.4305.0:142575 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3222 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.452550 7f9d5501e700  5 -- op tracker -- , seq: 748338, time: 2014-06-12 14:47:10.451190, event: header_read, op: osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.452564 7f9d5501e700  5 -- op tracker -- , seq: 748338, time: 2014-06-12 14:47:10.451193, event: throttled, op: osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.452573 7f9d5501e700  5 -- op tracker -- , seq: 748338, time: 2014-06-12 14:47:10.452506, event: all_read, op: osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.452582 7f9d5501e700  5 -- op tracker -- , seq: 748338, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.452589 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:10.452597 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:10.452607 7f9d5501e700 15 osd.2 173 enqueue_op 0x598fe00 prio 127 cost 516903 latency 0.001417 osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.452645 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:10.452654 7f9d623d5700 10 osd.2 173 dequeue_op 0x598fe00 prio 127 cost 516903 latency 0.001464 osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3c6( v 173'3222 (79'221,173'3222] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active]
2014-06-12 14:47:10.452681 7f9d623d5700  5 -- op tracker -- , seq: 748338, time: 2014-06-12 14:47:10.452676, event: reached_pg, op: osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.452690 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3222 (79'221,173'3222] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active] handle_message: 0x598fe00
2014-06-12 14:47:10.452705 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3222 (79'221,173'3222] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active] sub_op_modify trans 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 v 173'3223 (transaction) 174
2014-06-12 14:47:10.452726 7f9d623d5700  5 -- op tracker -- , seq: 748338, time: 2014-06-12 14:47:10.452721, event: started, op: osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.452744 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3222 (79'221,173'3222] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active] append_log log((79'221,173'3222], crt=0'0) [173'3223 (173'3222) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142576 2014-06-12 08:46:12.828185]
2014-06-12 14:47:10.452766 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3223 (79'221,173'3223] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active] add_log_entry 173'3223 (173'3222) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142576 2014-06-12 08:46:12.828185
2014-06-12 14:47:10.452790 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3223 (79'221,173'3223] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active] append_log  adding 1 keys
2014-06-12 14:47:10.452804 7f9d623d5700 10 trim log((79'221,173'3223], crt=0'0) to 79'222
2014-06-12 14:47:10.452834 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3223, trimmed: 79'222
2014-06-12 14:47:10.452862 7f9d623d5700  5 -- op tracker -- , seq: 748338, time: 2014-06-12 14:47:10.452858, event: started, op: osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.452873 7f9d623d5700  5 queue_transactions existing osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.452878 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:10.452880 7f9d623d5700  5 queue_op 0x56b63c0 seq 2924 osr(3.3c6 0x276d6d0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:10.452892 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540efc0 (1 active)
2014-06-12 14:47:10.452888 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3223 (79'222,173'3223] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active] do_sub_op osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.452902 7f9d7815d700  5 _do_op 0x56b63c0 seq 2924 osr(3.3c6 0x276d6d0)/0x276d6d0 start
2014-06-12 14:47:10.452903 7f9d623d5700 10 osd.2 173 dequeue_op 0x598fe00 finish
2014-06-12 14:47:10.452906 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:10.452907 7f9d7815d700 10 _do_transaction on 0x8aac2b0
2014-06-12 14:47:10.452908 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.452913 7f9d7815d700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.453007 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.453022 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:10.453053 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:10.453069 7f9d7815d700 15 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.453074 7f9d7815d700 10 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1 r = 0
2014-06-12 14:47:10.453078 7f9d7815d700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.453207 7f9d7815d700 15 _write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 1331200~516096
2014-06-12 14:47:10.453308 7f9d7815d700 10 lookup_strip_header done  cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.453428 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:47:10.454947 7f9d7815d700 10 _generic_write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 1331200~516096 = 0
2014-06-12 14:47:10.455009 7f9d7815d700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.455030 7f9d7815d700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.455036 7f9d7815d700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.455044 7f9d7815d700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.455268 7f9d7815d700 20 set_header setting 16491 cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 parent seq 0
2014-06-12 14:47:10.455348 7f9d7815d700 20 set_header setting 2260 cid meta oid 1fe0aae5/pglog_3.3c6/0//-1 parent seq 0
2014-06-12 14:47:10.455362 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:10.455370 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:47:10.458476 7f9d7815d700 10 _do_op 0x56b63c0 seq 2924 r = 0, finisher 0x2f2c180 0
2014-06-12 14:47:10.458499 7f9d7815d700 10 _finish_op 0x56b63c0 seq 2924 osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.458507 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540efc0 (0 active)
2014-06-12 14:47:10.458517 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.458528 7f9d7795c700  5 -- op tracker -- , seq: 748338, time: 2014-06-12 14:47:10.458517, event: sub_op_applied, op: osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.458545 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3223 (79'222,173'3223] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active] sub_op_modify_applied on 0x8aac1c0 op osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.458575 7f9d7795c700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.458596 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3223 (79'222,173'3223] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active] op_applied on primary on version 173'3223
2014-06-12 14:47:10.458605 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3223 (79'222,173'3223] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active] op_applied on replica on version 173'3223
2014-06-12 14:47:10.458640 7f9d7715b700  5 -- op tracker -- , seq: 748338, time: 2014-06-12 14:47:10.458636, event: commit_sent, op: osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.458648 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3223 (79'222,173'3223] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3221 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:47:10.458667 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.458699 7f9d7715b700  5 -- op tracker -- , seq: 748338, time: 2014-06-12 14:47:10.458696, event: done, op: osd_sub_op(client.4305.0:142576 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3223 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.464373 7f9d5501e700  5 -- op tracker -- , seq: 748339, time: 2014-06-12 14:47:10.463161, event: header_read, op: osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.464397 7f9d5501e700  5 -- op tracker -- , seq: 748339, time: 2014-06-12 14:47:10.463163, event: throttled, op: osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.464405 7f9d5501e700  5 -- op tracker -- , seq: 748339, time: 2014-06-12 14:47:10.464317, event: all_read, op: osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.464413 7f9d5501e700  5 -- op tracker -- , seq: 748339, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.464419 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:10.464426 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:10.464435 7f9d5501e700 15 osd.2 173 enqueue_op 0x598f900 prio 127 cost 516903 latency 0.001274 osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.464686 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:10.464697 7f9d68be2700 10 osd.2 173 dequeue_op 0x598f900 prio 127 cost 516903 latency 0.001535 osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3c6( v 173'3223 (79'222,173'3223] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3222 active]
2014-06-12 14:47:10.464717 7f9d68be2700  5 -- op tracker -- , seq: 748339, time: 2014-06-12 14:47:10.464714, event: reached_pg, op: osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.464726 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3223 (79'222,173'3223] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3222 active] handle_message: 0x598f900
2014-06-12 14:47:10.464734 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3223 (79'222,173'3223] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3222 active] sub_op_modify trans 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 v 173'3224 (transaction) 174
2014-06-12 14:47:10.464745 7f9d68be2700  5 -- op tracker -- , seq: 748339, time: 2014-06-12 14:47:10.464742, event: started, op: osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.464758 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3223 (79'222,173'3223] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3222 active] append_log log((79'222,173'3223], crt=0'0) [173'3224 (173'3223) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142577 2014-06-12 08:46:12.840357]
2014-06-12 14:47:10.464776 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3224 (79'222,173'3224] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3222 active] add_log_entry 173'3224 (173'3223) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142577 2014-06-12 08:46:12.840357
2014-06-12 14:47:10.464791 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3224 (79'222,173'3224] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3222 active] append_log  adding 1 keys
2014-06-12 14:47:10.464800 7f9d68be2700 10 trim log((79'222,173'3224], crt=0'0) to 79'223
2014-06-12 14:47:10.464822 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3224, trimmed: 79'223
2014-06-12 14:47:10.464843 7f9d68be2700  5 -- op tracker -- , seq: 748339, time: 2014-06-12 14:47:10.464839, event: started, op: osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.464851 7f9d68be2700  5 queue_transactions existing osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.464855 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:10.464856 7f9d68be2700  5 queue_op 0x8373440 seq 2925 osr(3.3c6 0x276d6d0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:10.464866 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540efc0 (1 active)
2014-06-12 14:47:10.464861 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3224 (79'223,173'3224] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3222 active] do_sub_op osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.464872 7f9d82171700  5 _do_op 0x8373440 seq 2925 osr(3.3c6 0x276d6d0)/0x276d6d0 start
2014-06-12 14:47:10.464874 7f9d68be2700 10 osd.2 173 dequeue_op 0x598f900 finish
2014-06-12 14:47:10.464877 7f9d82171700 10 _do_transaction on 0x338c69b0
2014-06-12 14:47:10.464878 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:10.464880 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.464882 7f9d82171700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.464983 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.465004 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:10.465038 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:10.465060 7f9d82171700 15 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.465067 7f9d82171700 10 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1 r = 0
2014-06-12 14:47:10.465071 7f9d82171700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.465189 7f9d82171700 15 _write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 1847296~516096
2014-06-12 14:47:10.465294 7f9d82171700 10 lookup_strip_header done  cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.465414 7f9d82171700 10 file_to_extents done 
2014-06-12 14:47:10.466822 7f9d82171700 10 _generic_write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 1847296~516096 = 0
2014-06-12 14:47:10.466876 7f9d82171700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.466890 7f9d82171700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.466893 7f9d82171700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.466898 7f9d82171700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.467044 7f9d82171700 20 set_header setting 16491 cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 parent seq 0
2014-06-12 14:47:10.467100 7f9d82171700 20 set_header setting 2260 cid meta oid 1fe0aae5/pglog_3.3c6/0//-1 parent seq 0
2014-06-12 14:47:10.467112 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:10.467117 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:47:10.469022 7f9d82171700 10 _do_op 0x8373440 seq 2925 r = 0, finisher 0x31bdd2e0 0
2014-06-12 14:47:10.469037 7f9d82171700 10 _finish_op 0x8373440 seq 2925 osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.469042 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540efc0 (0 active)
2014-06-12 14:47:10.469052 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.469055 7f9d7715b700  5 -- op tracker -- , seq: 748339, time: 2014-06-12 14:47:10.469045, event: commit_sent, op: osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.469064 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3224 (79'223,173'3224] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3222 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:47:10.469090 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.469165 7f9d7795c700  5 -- op tracker -- , seq: 748339, time: 2014-06-12 14:47:10.469159, event: sub_op_applied, op: osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.469176 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3224 (79'223,173'3224] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3223 active] sub_op_modify_applied on 0x338c68c0 op osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.469196 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3224 (79'223,173'3224] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3223 active] op_applied on primary on version 173'3224
2014-06-12 14:47:10.469203 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3224 (79'223,173'3224] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3223 active] op_applied on replica on version 173'3224
2014-06-12 14:47:10.469218 7f9d7795c700  5 -- op tracker -- , seq: 748339, time: 2014-06-12 14:47:10.469215, event: done, op: osd_sub_op(client.4305.0:142577 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3224 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.475300 7f9d5501e700  5 -- op tracker -- , seq: 748340, time: 2014-06-12 14:47:10.473960, event: header_read, op: osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.475312 7f9d5501e700  5 -- op tracker -- , seq: 748340, time: 2014-06-12 14:47:10.473962, event: throttled, op: osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.475320 7f9d5501e700  5 -- op tracker -- , seq: 748340, time: 2014-06-12 14:47:10.475266, event: all_read, op: osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.475325 7f9d5501e700  5 -- op tracker -- , seq: 748340, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.475329 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:10.475334 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:10.475340 7f9d5501e700 15 osd.2 173 enqueue_op 0x598f300 prio 127 cost 516903 latency 0.001380 osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.475367 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:10.475378 7f9d643d9700 10 osd.2 173 dequeue_op 0x598f300 prio 127 cost 516903 latency 0.001417 osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3c6( v 173'3224 (79'223,173'3224] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3223 active]
2014-06-12 14:47:10.475405 7f9d643d9700  5 -- op tracker -- , seq: 748340, time: 2014-06-12 14:47:10.475400, event: reached_pg, op: osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.475415 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3224 (79'223,173'3224] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3223 active] handle_message: 0x598f300
2014-06-12 14:47:10.475429 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3224 (79'223,173'3224] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3223 active] sub_op_modify trans 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 v 173'3225 (transaction) 174
2014-06-12 14:47:10.475448 7f9d643d9700  5 -- op tracker -- , seq: 748340, time: 2014-06-12 14:47:10.475443, event: started, op: osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.475465 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3224 (79'223,173'3224] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3223 active] append_log log((79'223,173'3224], crt=0'0) [173'3225 (173'3224) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142578 2014-06-12 08:46:12.850913]
2014-06-12 14:47:10.475490 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3225 (79'223,173'3225] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3223 active] add_log_entry 173'3225 (173'3224) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142578 2014-06-12 08:46:12.850913
2014-06-12 14:47:10.475514 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3225 (79'223,173'3225] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3223 active] append_log  adding 1 keys
2014-06-12 14:47:10.475530 7f9d643d9700 10 trim log((79'223,173'3225], crt=0'0) to 79'224
2014-06-12 14:47:10.475557 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3225, trimmed: 79'224
2014-06-12 14:47:10.475586 7f9d643d9700  5 -- op tracker -- , seq: 748340, time: 2014-06-12 14:47:10.475582, event: started, op: osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.475596 7f9d643d9700  5 queue_transactions existing osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.475601 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:10.475603 7f9d643d9700  5 queue_op 0x9182cc0 seq 2926 osr(3.3c6 0x276d6d0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:10.475612 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540efc0 (1 active)
2014-06-12 14:47:10.475617 7f9d7e96a700  5 _do_op 0x9182cc0 seq 2926 osr(3.3c6 0x276d6d0)/0x276d6d0 start
2014-06-12 14:47:10.475620 7f9d7e96a700 10 _do_transaction on 0x6671270
2014-06-12 14:47:10.475610 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3225 (79'224,173'3225] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3223 active] do_sub_op osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.475625 7f9d7e96a700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.475626 7f9d643d9700 10 osd.2 173 dequeue_op 0x598f300 finish
2014-06-12 14:47:10.475630 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:10.475633 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.475716 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.475742 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:10.475774 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:10.475857 7f9d7e96a700 15 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.475864 7f9d7e96a700 10 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1 r = 0
2014-06-12 14:47:10.475869 7f9d7e96a700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.475882 7f9d7e96a700 15 _write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 2363392~516096
2014-06-12 14:47:10.475992 7f9d7e96a700 10 lookup_strip_header done  cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.476112 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:47:10.477325 7f9d7e96a700 10 _generic_write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 2363392~516096 = 0
2014-06-12 14:47:10.477364 7f9d7e96a700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.477372 7f9d7e96a700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.477376 7f9d7e96a700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.477380 7f9d7e96a700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.477517 7f9d7e96a700 20 set_header setting 16491 cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 parent seq 0
2014-06-12 14:47:10.477564 7f9d7e96a700 20 set_header setting 2260 cid meta oid 1fe0aae5/pglog_3.3c6/0//-1 parent seq 0
2014-06-12 14:47:10.477574 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:10.477579 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:47:10.479532 7f9d7e96a700 10 _do_op 0x9182cc0 seq 2926 r = 0, finisher 0xa0144c0 0
2014-06-12 14:47:10.479548 7f9d7e96a700 10 _finish_op 0x9182cc0 seq 2926 osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.479553 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540efc0 (0 active)
2014-06-12 14:47:10.479558 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.479568 7f9d7715b700  5 -- op tracker -- , seq: 748340, time: 2014-06-12 14:47:10.479558, event: commit_sent, op: osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.479582 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3225 (79'224,173'3225] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3223 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:47:10.479618 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.479683 7f9d7795c700  5 -- op tracker -- , seq: 748340, time: 2014-06-12 14:47:10.479677, event: sub_op_applied, op: osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.479694 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3225 (79'224,173'3225] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3224 active] sub_op_modify_applied on 0x6671180 op osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.479708 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3225 (79'224,173'3225] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3224 active] op_applied on primary on version 173'3225
2014-06-12 14:47:10.479716 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3225 (79'224,173'3225] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3224 active] op_applied on replica on version 173'3225
2014-06-12 14:47:10.479731 7f9d7795c700  5 -- op tracker -- , seq: 748340, time: 2014-06-12 14:47:10.479728, event: done, op: osd_sub_op(client.4305.0:142578 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3225 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.482423 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:10.482439 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:10.482443 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:10.482445 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:10.482447 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 28; oldest is 30.174004 seconds old
2014-06-12 14:47:10.482471 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:10.482485 7f9d84175700  0 log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 30.174004 secs
2014-06-12 14:47:10.482500 7f9d84175700  0 log [WRN] : slow request 30.174004 seconds old, received at 2014-06-12 14:46:40.308442: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 26
2014-06-12 14:47:10.485574 7f9d5501e700  5 -- op tracker -- , seq: 748341, time: 2014-06-12 14:47:10.484265, event: header_read, op: osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.485584 7f9d5501e700  5 -- op tracker -- , seq: 748341, time: 2014-06-12 14:47:10.484267, event: throttled, op: osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.485589 7f9d5501e700  5 -- op tracker -- , seq: 748341, time: 2014-06-12 14:47:10.485547, event: all_read, op: osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.485594 7f9d5501e700  5 -- op tracker -- , seq: 748341, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.485598 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:10.485602 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:10.485607 7f9d5501e700 15 osd.2 173 enqueue_op 0x598ed00 prio 127 cost 516903 latency 0.001342 osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.485650 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:10.485675 7f9d63bd8700 10 osd.2 173 dequeue_op 0x598ed00 prio 127 cost 516903 latency 0.001409 osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3c6( v 173'3225 (79'224,173'3225] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3224 active]
2014-06-12 14:47:10.485698 7f9d63bd8700  5 -- op tracker -- , seq: 748341, time: 2014-06-12 14:47:10.485693, event: reached_pg, op: osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.485708 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3225 (79'224,173'3225] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3224 active] handle_message: 0x598ed00
2014-06-12 14:47:10.485724 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3225 (79'224,173'3225] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3224 active] sub_op_modify trans 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 v 173'3226 (transaction) 174
2014-06-12 14:47:10.485744 7f9d63bd8700  5 -- op tracker -- , seq: 748341, time: 2014-06-12 14:47:10.485739, event: started, op: osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.485760 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3225 (79'224,173'3225] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3224 active] append_log log((79'224,173'3225], crt=0'0) [173'3226 (173'3225) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142579 2014-06-12 08:46:12.861496]
2014-06-12 14:47:10.485795 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3226 (79'224,173'3226] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3224 active] add_log_entry 173'3226 (173'3225) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142579 2014-06-12 08:46:12.861496
2014-06-12 14:47:10.485824 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3226 (79'224,173'3226] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3224 active] append_log  adding 1 keys
2014-06-12 14:47:10.485838 7f9d63bd8700 10 trim log((79'224,173'3226], crt=0'0) to 79'225
2014-06-12 14:47:10.485874 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3226, trimmed: 79'225
2014-06-12 14:47:10.485915 7f9d63bd8700  5 -- op tracker -- , seq: 748341, time: 2014-06-12 14:47:10.485909, event: started, op: osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.485934 7f9d63bd8700  5 queue_transactions existing osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.485938 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:10.485940 7f9d63bd8700  5 queue_op 0x76f5ce0 seq 2927 osr(3.3c6 0x276d6d0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:10.485955 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540efc0 (1 active)
2014-06-12 14:47:10.485961 7f9d8116f700  5 _do_op 0x76f5ce0 seq 2927 osr(3.3c6 0x276d6d0)/0x276d6d0 start
2014-06-12 14:47:10.485965 7f9d8116f700 10 _do_transaction on 0x5ea70b0
2014-06-12 14:47:10.485953 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3226 (79'225,173'3226] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3224 active] do_sub_op osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.485968 7f9d63bd8700 10 osd.2 173 dequeue_op 0x598ed00 finish
2014-06-12 14:47:10.485970 7f9d8116f700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.485971 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:10.485974 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.486057 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.486077 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:10.486107 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:10.486126 7f9d8116f700 15 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.486132 7f9d8116f700 10 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1 r = 0
2014-06-12 14:47:10.486136 7f9d8116f700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.486221 7f9d8116f700 15 _write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 2879488~516096
2014-06-12 14:47:10.486319 7f9d8116f700 10 lookup_strip_header done  cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.486415 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:47:10.487562 7f9d8116f700 10 _generic_write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 2879488~516096 = 0
2014-06-12 14:47:10.487593 7f9d8116f700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.487601 7f9d8116f700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.487605 7f9d8116f700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.487609 7f9d8116f700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.487743 7f9d8116f700 20 set_header setting 16491 cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 parent seq 0
2014-06-12 14:47:10.487788 7f9d8116f700 20 set_header setting 2260 cid meta oid 1fe0aae5/pglog_3.3c6/0//-1 parent seq 0
2014-06-12 14:47:10.487797 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:10.487802 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:47:10.489387 7f9d8116f700 10 _do_op 0x76f5ce0 seq 2927 r = 0, finisher 0x29793ee0 0
2014-06-12 14:47:10.489397 7f9d8116f700 10 _finish_op 0x76f5ce0 seq 2927 osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.489401 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540efc0 (0 active)
2014-06-12 14:47:10.489408 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.489417 7f9d7715b700  5 -- op tracker -- , seq: 748341, time: 2014-06-12 14:47:10.489408, event: commit_sent, op: osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.489431 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3226 (79'225,173'3226] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3224 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:47:10.489459 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.489494 7f9d7795c700  5 -- op tracker -- , seq: 748341, time: 2014-06-12 14:47:10.489488, event: sub_op_applied, op: osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.489504 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3226 (79'225,173'3226] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active] sub_op_modify_applied on 0x5ea6fc0 op osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.489523 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3226 (79'225,173'3226] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active] op_applied on primary on version 173'3226
2014-06-12 14:47:10.489535 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3226 (79'225,173'3226] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active] op_applied on replica on version 173'3226
2014-06-12 14:47:10.489560 7f9d7795c700  5 -- op tracker -- , seq: 748341, time: 2014-06-12 14:47:10.489555, event: done, op: osd_sub_op(client.4305.0:142579 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3226 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.496494 7f9d5501e700  5 -- op tracker -- , seq: 748342, time: 2014-06-12 14:47:10.495361, event: header_read, op: osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.496507 7f9d5501e700  5 -- op tracker -- , seq: 748342, time: 2014-06-12 14:47:10.495364, event: throttled, op: osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.496522 7f9d5501e700  5 -- op tracker -- , seq: 748342, time: 2014-06-12 14:47:10.496460, event: all_read, op: osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.496528 7f9d5501e700  5 -- op tracker -- , seq: 748342, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.496532 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:10.496537 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:10.496548 7f9d5501e700 15 osd.2 173 enqueue_op 0x598e800 prio 127 cost 516903 latency 0.001187 osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.496576 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:10.496585 7f9d61bd4700 10 osd.2 173 dequeue_op 0x598e800 prio 127 cost 516903 latency 0.001224 osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3c6( v 173'3226 (79'225,173'3226] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active]
2014-06-12 14:47:10.496603 7f9d61bd4700  5 -- op tracker -- , seq: 748342, time: 2014-06-12 14:47:10.496600, event: reached_pg, op: osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.496613 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3226 (79'225,173'3226] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active] handle_message: 0x598e800
2014-06-12 14:47:10.496622 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3226 (79'225,173'3226] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active] sub_op_modify trans 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 v 173'3227 (transaction) 174
2014-06-12 14:47:10.496634 7f9d61bd4700  5 -- op tracker -- , seq: 748342, time: 2014-06-12 14:47:10.496631, event: started, op: osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.496646 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3226 (79'225,173'3226] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active] append_log log((79'225,173'3226], crt=0'0) [173'3227 (173'3226) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142580 2014-06-12 08:46:12.871410]
2014-06-12 14:47:10.496665 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3227 (79'225,173'3227] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active] add_log_entry 173'3227 (173'3226) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142580 2014-06-12 08:46:12.871410
2014-06-12 14:47:10.496681 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3227 (79'225,173'3227] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active] append_log  adding 1 keys
2014-06-12 14:47:10.496691 7f9d61bd4700 10 trim log((79'225,173'3227], crt=0'0) to 79'226
2014-06-12 14:47:10.496710 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3227, trimmed: 79'226
2014-06-12 14:47:10.496734 7f9d61bd4700  5 -- op tracker -- , seq: 748342, time: 2014-06-12 14:47:10.496730, event: started, op: osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.496742 7f9d61bd4700  5 queue_transactions existing osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.496745 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:10.496746 7f9d61bd4700  5 queue_op 0x1283b140 seq 2928 osr(3.3c6 0x276d6d0) 518424 bytes   (queue has 1 ops and 518424 bytes)
2014-06-12 14:47:10.496751 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3227 (79'226,173'3227] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active] do_sub_op osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.496760 7f9d61bd4700 10 osd.2 173 dequeue_op 0x598e800 finish
2014-06-12 14:47:10.496756 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540efc0 (1 active)
2014-06-12 14:47:10.496762 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:10.496764 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.496764 7f9d7d167700  5 _do_op 0x1283b140 seq 2928 osr(3.3c6 0x276d6d0)/0x276d6d0 start
2014-06-12 14:47:10.496768 7f9d7d167700 10 _do_transaction on 0x65cc630
2014-06-12 14:47:10.496775 7f9d7d167700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.496958 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.496986 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:10.497033 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:10.497055 7f9d7d167700 15 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.497065 7f9d7d167700 10 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1 r = 0
2014-06-12 14:47:10.497073 7f9d7d167700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.497091 7f9d7d167700 15 _write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 3395584~516096
2014-06-12 14:47:10.497232 7f9d7d167700 10 lookup_strip_header done  cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.497444 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:47:10.499123 7f9d7d167700 10 _generic_write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 3395584~516096 = 0
2014-06-12 14:47:10.499200 7f9d7d167700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.499212 7f9d7d167700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.499216 7f9d7d167700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.499220 7f9d7d167700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.499478 7f9d7d167700 20 set_header setting 16491 cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 parent seq 0
2014-06-12 14:47:10.499591 7f9d7d167700 20 set_header setting 2260 cid meta oid 1fe0aae5/pglog_3.3c6/0//-1 parent seq 0
2014-06-12 14:47:10.499615 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:10.499626 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:47:10.502483 7f9d7d167700 10 _do_op 0x1283b140 seq 2928 r = 0, finisher 0x6d62080 0
2014-06-12 14:47:10.502497 7f9d7d167700 10 _finish_op 0x1283b140 seq 2928 osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.502502 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540efc0 (0 active)
2014-06-12 14:47:10.502512 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.502528 7f9d7795c700  5 -- op tracker -- , seq: 748342, time: 2014-06-12 14:47:10.502516, event: sub_op_applied, op: osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.502539 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3227 (79'226,173'3227] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active] sub_op_modify_applied on 0x65cc540 op osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.502578 7f9d7795c700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.502600 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3227 (79'226,173'3227] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active] op_applied on primary on version 173'3227
2014-06-12 14:47:10.502613 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3227 (79'226,173'3227] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active] op_applied on replica on version 173'3227
2014-06-12 14:47:10.502673 7f9d7715b700  5 -- op tracker -- , seq: 748342, time: 2014-06-12 14:47:10.502658, event: commit_sent, op: osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.502685 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3227 (79'226,173'3227] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3225 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:47:10.502704 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.502737 7f9d7715b700  5 -- op tracker -- , seq: 748342, time: 2014-06-12 14:47:10.502734, event: done, op: osd_sub_op(client.4305.0:142580 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3227 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.508036 7f9d5501e700  5 -- op tracker -- , seq: 748343, time: 2014-06-12 14:47:10.507539, event: header_read, op: osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.508051 7f9d5501e700  5 -- op tracker -- , seq: 748343, time: 2014-06-12 14:47:10.507542, event: throttled, op: osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.508060 7f9d5501e700  5 -- op tracker -- , seq: 748343, time: 2014-06-12 14:47:10.508001, event: all_read, op: osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.508070 7f9d5501e700  5 -- op tracker -- , seq: 748343, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.508076 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:10.508084 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:10.508093 7f9d5501e700 15 osd.2 173 enqueue_op 0xe20ad00 prio 127 cost 283431 latency 0.000554 osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.508129 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:10.508138 7f9d633d7700 10 osd.2 173 dequeue_op 0xe20ad00 prio 127 cost 283431 latency 0.000599 osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3c6( v 173'3227 (79'226,173'3227] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3226 active]
2014-06-12 14:47:10.508167 7f9d633d7700  5 -- op tracker -- , seq: 748343, time: 2014-06-12 14:47:10.508162, event: reached_pg, op: osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.508179 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3227 (79'226,173'3227] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3226 active] handle_message: 0xe20ad00
2014-06-12 14:47:10.508193 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3227 (79'226,173'3227] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3226 active] sub_op_modify trans 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 v 173'3228 (transaction) 174
2014-06-12 14:47:10.508213 7f9d633d7700  5 -- op tracker -- , seq: 748343, time: 2014-06-12 14:47:10.508209, event: started, op: osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.508232 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3227 (79'226,173'3227] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3226 active] append_log log((79'226,173'3227], crt=0'0) [173'3228 (173'3227) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142581 2014-06-12 08:46:12.884448]
2014-06-12 14:47:10.508261 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3228 (79'226,173'3228] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3226 active] add_log_entry 173'3228 (173'3227) modify   37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 by client.4305.0:142581 2014-06-12 08:46:12.884448
2014-06-12 14:47:10.508286 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3228 (79'226,173'3228] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3226 active] append_log  adding 1 keys
2014-06-12 14:47:10.508302 7f9d633d7700 10 trim log((79'226,173'3228], crt=0'0) to 79'227
2014-06-12 14:47:10.508331 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3228, trimmed: 79'227
2014-06-12 14:47:10.508362 7f9d633d7700  5 -- op tracker -- , seq: 748343, time: 2014-06-12 14:47:10.508357, event: started, op: osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.508376 7f9d633d7700  5 queue_transactions existing osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.508381 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:10.508383 7f9d633d7700  5 queue_op 0x7c239e0 seq 2929 osr(3.3c6 0x276d6d0) 284952 bytes   (queue has 1 ops and 284952 bytes)
2014-06-12 14:47:10.508397 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x540efc0 (1 active)
2014-06-12 14:47:10.508391 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3228 (79'227,173'3228] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3226 active] do_sub_op osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.508406 7f9d633d7700 10 osd.2 173 dequeue_op 0xe20ad00 finish
2014-06-12 14:47:10.508406 7f9d7915f700  5 _do_op 0x7c239e0 seq 2929 osr(3.3c6 0x276d6d0)/0x276d6d0 start
2014-06-12 14:47:10.508410 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:10.508411 7f9d7915f700 10 _do_transaction on 0x1451deb0
2014-06-12 14:47:10.508413 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.508418 7f9d7915f700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.508555 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.508583 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:10.508634 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:10.508662 7f9d7915f700 15 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.508672 7f9d7915f700 10 _omap_rmkeys meta/1fe0aae5/pglog_3.3c6/0//-1 r = 0
2014-06-12 14:47:10.508678 7f9d7915f700 15 _omap_setkeys meta/1fe0aae5/pglog_3.3c6/0//-1
2014-06-12 14:47:10.508695 7f9d7915f700 15 _write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 3911680~282624
2014-06-12 14:47:10.508842 7f9d7915f700 10 lookup_strip_header done  cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.509076 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:47:10.509981 7f9d7915f700 10 _generic_write 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 3911680~282624 = 0
2014-06-12 14:47:10.510011 7f9d7915f700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.510021 7f9d7915f700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.510026 7f9d7915f700 15 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3
2014-06-12 14:47:10.510031 7f9d7915f700 10 _setattrs 3.3c6_head/37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 = 0
2014-06-12 14:47:10.510141 7f9d7915f700 20 set_header setting 16491 cid 3.3c6_head oid 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 parent seq 0
2014-06-12 14:47:10.510185 7f9d7915f700 20 set_header setting 2260 cid meta oid 1fe0aae5/pglog_3.3c6/0//-1 parent seq 0
2014-06-12 14:47:10.510197 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:10.510203 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:47:10.511135 7f9d7915f700 10 _do_op 0x7c239e0 seq 2929 r = 0, finisher 0x8022ea0 0
2014-06-12 14:47:10.511145 7f9d7915f700 10 _finish_op 0x7c239e0 seq 2929 osr(3.3c6 0x276d6d0)/0x276d6d0
2014-06-12 14:47:10.511155 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x540efc0 (0 active)
2014-06-12 14:47:10.511163 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.511162 7f9d7715b700  5 -- op tracker -- , seq: 748343, time: 2014-06-12 14:47:10.511154, event: commit_sent, op: osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.511174 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3228 (79'227,173'3228] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3226 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.29
2014-06-12 14:47:10.511204 7f9d7715b700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.511234 7f9d7795c700  5 -- op tracker -- , seq: 748343, time: 2014-06-12 14:47:10.511228, event: sub_op_applied, op: osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.511242 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3228 (79'227,173'3228] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3227 active] sub_op_modify_applied on 0x1451ddc0 op osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:10.511253 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3228 (79'227,173'3228] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3227 active] op_applied on primary on version 173'3228
2014-06-12 14:47:10.511260 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.3c6( v 173'3228 (79'227,173'3228] local-les=161 n=332 ec=23 les/c 161/161 160/160/160) [29,2] r=1 lpr=160 pi=23-159/12 luod=0'0 crt=0'0 lcod 173'3227 active] op_applied on replica on version 173'3228
2014-06-12 14:47:10.511273 7f9d7795c700  5 -- op tracker -- , seq: 748343, time: 2014-06-12 14:47:10.511270, event: done, op: osd_sub_op(client.4305.0:142581 3.3c6 37aa2bc6/rbd_data.10a86b8b4567.0000000000000ef1/head//3 [] v 173'3228 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:10.712126 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:10.712129 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:10.712131 7f9d74bfa700 20 osd.2 173 _dispatch 0x1e700d00 ping v1
2014-06-12 14:47:10.712134 7f9d74bfa700 10 osd.2 173 ping from client.4300
2014-06-12 14:47:10.712136 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:10.712137 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:10.807361 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.816024 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.816798 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:10.819385 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.848192 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:10.848561 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:10.949857 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:10.950234 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.950284 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:10.957990 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:10.958034 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:10.967424 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:11.106184 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:11.116572 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:11.185632 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:11.188915 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:11.225402 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19428 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,1,1,0,8,18])
2014-06-12 14:47:11.225429 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19428 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,1,1,0,8,18])
2014-06-12 14:47:11.225960 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:11.225981 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:11.226008 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:11.226039 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:11.226061 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:11.226073 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:11.226084 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:11.226097 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:11.226119 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:11.226138 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:11.226151 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:11.226167 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:11.226178 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:11.226189 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:11.226202 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:11.226214 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:11.226226 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:11.226237 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:11.226252 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:11.226264 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:11.226276 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:11.226357 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:11.290162 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:11.290165 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:11.290166 7f9d74bfa700 20 osd.2 173 _dispatch 0x2c2dd6c0 ping v1
2014-06-12 14:47:11.290169 7f9d74bfa700 10 osd.2 173 ping from client.4306
2014-06-12 14:47:11.290171 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:11.290172 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:11.327058 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:11.327087 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:11.404012 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:11.408854 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:11.415433 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:11.415456 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:11.430875 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:11.434297 7f9d47c58700  5 -- op tracker -- , seq: 748344, time: 2014-06-12 14:47:11.432037, event: header_read, op: osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.434310 7f9d47c58700  5 -- op tracker -- , seq: 748344, time: 2014-06-12 14:47:11.432041, event: throttled, op: osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.434320 7f9d47c58700  5 -- op tracker -- , seq: 748344, time: 2014-06-12 14:47:11.434256, event: all_read, op: osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.434335 7f9d47c58700  5 -- op tracker -- , seq: 748344, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.434343 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:47:11.434372 7f9d47c58700 15 osd.2 173 enqueue_op 0x3b42a900 prio 63 cost 1032192 latency 0.002335 osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:11.434408 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:11.434416 7f9d693e3700 10 osd.2 173 dequeue_op 0x3b42a900 prio 63 cost 1032192 latency 0.002379 osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]
2014-06-12 14:47:11.434445 7f9d693e3700  5 -- op tracker -- , seq: 748344, time: 2014-06-12 14:47:11.434439, event: reached_pg, op: osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.434454 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:11.434471 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] handle_message: 0x3b42a900
2014-06-12 14:47:11.434483 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:11.434513 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: found obc in cache: 0xedc6b00
2014-06-12 14:47:11.434525 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: 0xedc6b00 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 rwstate(write n=2 w=0) oi: 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'2977 client.4316.0:147319 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2977) ssc: 0x1550f180 snapset: 0=[]:[]+head
2014-06-12 14:47:11.434544 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] find_object_context 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 @head oi=58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'2977 client.4316.0:147319 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2977)
2014-06-12 14:47:11.434573 7f9d693e3700  5 -- op tracker -- , seq: 748344, time: 2014-06-12 14:47:11.434567, event: started, op: osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.434581 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] execute_ctx 0x580be00
2014-06-12 14:47:11.434600 7f9d693e3700  5 -- op tracker -- , seq: 748344, time: 2014-06-12 14:47:11.434594, event: started, op: osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.434609 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] ov 173'2977 av 173'2978 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:11.434624 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192]
2014-06-12 14:47:11.434637 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:11.434653 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  write 774144~1032192
2014-06-12 14:47:11.434671 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:47:11.434684 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 snapset=0x1550f1b8  snapc=0=[]
2014-06-12 14:47:11.434699 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:11.434712 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] finish_ctx 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 0x580be00 op modify  
2014-06-12 14:47:11.434727 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  set mtime to 2014-06-12 08:46:13.810160
2014-06-12 14:47:11.434751 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  final snapset 0=[]:[]+head in 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:11.434771 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  zeroing write result code 0
2014-06-12 14:47:11.434784 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] new_repop rep_tid 236408 on osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:11.434802 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] issue_repop rep_tid 236408 o 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:11.434831 7f9d693e3700  5 -- op tracker -- , seq: 748344, time: 2014-06-12 14:47:11.434825, event: waiting for subops from 24, op: osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.434853 7f9d693e3700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:11.434877 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2977 (0'0,173'2977] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log log((0'0,173'2977], crt=0'0) [173'2978 (173'2977) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147320 2014-06-12 08:46:13.810160]
2014-06-12 14:47:11.434906 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] add_log_entry 173'2978 (173'2977) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147320 2014-06-12 08:46:13.810160
2014-06-12 14:47:11.434928 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log  adding 1 keys
2014-06-12 14:47:11.434957 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2978, trimmed: 
2014-06-12 14:47:11.434977 7f9d693e3700  5 queue_transactions existing osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:11.434980 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:11.434982 7f9d693e3700  5 queue_op 0x83ee480 seq 2707 osr(3.785 0x49b40d0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:11.434988 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] eval_repop repgather(0x2d813380 173'2978 rep_tid=236408 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:11.435005 7f9d693e3700 10 osd.2 173 dequeue_op 0x3b42a900 finish
2014-06-12 14:47:11.435008 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:11.435010 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:11.435016 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54bc360 (1 active)
2014-06-12 14:47:11.435020 7f9d81970700  5 _do_op 0x83ee480 seq 2707 osr(3.785 0x49b40d0)/0x49b40d0 start
2014-06-12 14:47:11.435024 7f9d81970700 10 _do_transaction on 0x31b65c80
2014-06-12 14:47:11.435029 7f9d81970700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:11.435140 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:11.435160 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:11.435203 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:11.435225 7f9d81970700 15 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:11.435230 7f9d81970700 10 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1 r = 0
2014-06-12 14:47:11.435234 7f9d81970700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:11.435250 7f9d81970700 15 _write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 774144~1032192
2014-06-12 14:47:11.435557 7f9d81970700 10 lookup_strip_header done  cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:11.435717 7f9d81970700 10 file_to_extents done 
2014-06-12 14:47:11.438535 7f9d81970700 10 _generic_write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 774144~1032192 = 0
2014-06-12 14:47:11.438608 7f9d81970700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:11.438625 7f9d81970700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:11.438631 7f9d81970700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:11.438640 7f9d81970700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:11.438854 7f9d81970700 20 set_header setting 17484 cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 parent seq 0
2014-06-12 14:47:11.438941 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:11.438958 7f9d81970700 20 set_header setting 2142 cid meta oid 1fe0af7a/pglog_3.785/0//-1 parent seq 0
2014-06-12 14:47:11.438974 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:47:11.442758 7f9d81970700 10 _do_op 0x83ee480 seq 2707 r = 0, finisher 0x13bde0e0 0x25eee740
2014-06-12 14:47:11.442785 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_commit: 236408
2014-06-12 14:47:11.442822 7f9d7715b700  5 -- op tracker -- , seq: 748344, time: 2014-06-12 14:47:11.442811, event: op_commit, op: osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.442838 7f9d81970700 10 _finish_op 0x83ee480 seq 2707 osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:11.442847 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54bc360 (0 active)
2014-06-12 14:47:11.442852 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:11.442851 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied: 236408
2014-06-12 14:47:11.442870 7f9d7795c700  5 -- op tracker -- , seq: 748344, time: 2014-06-12 14:47:11.442866, event: op_applied, op: osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.442876 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied on primary on version 173'2978
2014-06-12 14:47:11.482573 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:11.482601 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:11.482605 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:11.482607 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:11.482610 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 29; oldest is 31.174167 seconds old
2014-06-12 14:47:11.482670 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:11.574801 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:11.574869 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:11.726049 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19429 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,1,0,1,1,0,5,21])
2014-06-12 14:47:11.726071 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19429 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,1,0,1,1,0,5,21])
2014-06-12 14:47:11.726533 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:11.726569 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:11.726633 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:11.726660 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:11.726686 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:11.726705 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:11.726716 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:11.726753 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:11.726778 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:11.726793 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:11.726927 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:11.726943 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:11.726955 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:11.726970 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:11.727014 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:11.727033 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:11.727111 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:11.727144 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:11.727224 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:11.727333 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:11.729700 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:11.729718 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:11.741802 7f9d56432700  5 -- op tracker -- , seq: 748345, time: 2014-06-12 14:47:11.741748, event: header_read, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ack, result = 0)
2014-06-12 14:47:11.741813 7f9d56432700  5 -- op tracker -- , seq: 748345, time: 2014-06-12 14:47:11.741750, event: throttled, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ack, result = 0)
2014-06-12 14:47:11.741835 7f9d56432700  5 -- op tracker -- , seq: 748345, time: 2014-06-12 14:47:11.741763, event: all_read, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ack, result = 0)
2014-06-12 14:47:11.741840 7f9d56432700  5 -- op tracker -- , seq: 748345, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ack, result = 0)
2014-06-12 14:47:11.741844 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:11.741848 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:11.741853 7f9d56432700 15 osd.2 173 enqueue_op 0x25e43c00 prio 196 cost 0 latency 0.000105 osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ack, result = 0) v2
2014-06-12 14:47:11.741896 7f9d56432700  5 -- op tracker -- , seq: 748346, time: 2014-06-12 14:47:11.741870, event: header_read, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.741908 7f9d56432700  5 -- op tracker -- , seq: 748346, time: 2014-06-12 14:47:11.741870, event: throttled, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.741907 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:11.741913 7f9d56432700  5 -- op tracker -- , seq: 748346, time: 2014-06-12 14:47:11.741875, event: all_read, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.741917 7f9d56432700  5 -- op tracker -- , seq: 748346, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.741920 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:11.741923 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:11.741925 7f9d56432700 15 osd.2 173 enqueue_op 0x25e42900 prio 196 cost 0 latency 0.000055 osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:11.741917 7f9d6abe6700 10 osd.2 173 dequeue_op 0x25e43c00 prio 196 cost 0 latency 0.000169 osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ack, result = 0) v2 pg pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]
2014-06-12 14:47:11.741943 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:11.741939 7f9d6abe6700  5 -- op tracker -- , seq: 748345, time: 2014-06-12 14:47:11.741935, event: reached_pg, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ack, result = 0)
2014-06-12 14:47:11.741950 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] handle_message: 0x25e43c00
2014-06-12 14:47:11.741972 7f9d6abe6700  5 -- op tracker -- , seq: 748345, time: 2014-06-12 14:47:11.741970, event: started, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ack, result = 0)
2014-06-12 14:47:11.741977 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] sub_op_modify_reply: tid 236292 op  ack_type 1 from 26
2014-06-12 14:47:11.741991 7f9d6abe6700  5 -- op tracker -- , seq: 747951, time: 2014-06-12 14:47:11.741985, event: sub_op_applied_rec, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.741996 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] repop_all_applied: repop tid 236292 all applied 
2014-06-12 14:47:11.742003 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] eval_repop repgather(0x2fe22480 173'2444 rep_tid=236292 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:11.742013 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]  sending ack on repgather(0x2fe22480 173'2444 rep_tid=236292 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) 0x6701b80
2014-06-12 14:47:11.742036 7f9d6abe6700 10 osd.2 173 dequeue_op 0x25e43c00 finish
2014-06-12 14:47:11.742045 7f9d6abe6700  5 -- op tracker -- , seq: 748345, time: 2014-06-12 14:47:11.742042, event: done, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ack, result = 0)
2014-06-12 14:47:11.742069 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:11.742072 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:11.742076 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x25e42900 prio 196 cost 0 latency 0.000206 osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2 pg pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean]
2014-06-12 14:47:11.742102 7f9d6a3e5700  5 -- op tracker -- , seq: 748346, time: 2014-06-12 14:47:11.742091, event: reached_pg, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.742109 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] handle_message: 0x25e42900
2014-06-12 14:47:11.742119 7f9d6a3e5700  5 -- op tracker -- , seq: 748346, time: 2014-06-12 14:47:11.742117, event: started, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.742123 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] sub_op_modify_reply: tid 236292 op  ack_type 4 from 26
2014-06-12 14:47:11.742140 7f9d6a3e5700  5 -- op tracker -- , seq: 747951, time: 2014-06-12 14:47:11.742134, event: sub_op_commit_rec, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.742147 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2443 crt=0'0 lcod 173'2442 mlcod 173'2442 active+clean] repop_all_committed: repop tid 236292 all committed 
2014-06-12 14:47:11.742153 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2442 active+clean] eval_repop repgather(0x2fe22480 173'2444 rep_tid=236292 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:11.742164 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2442 active+clean] log_op_stats osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 inb 200983 outb 0 rlat 31.433592 lat 31.433721
2014-06-12 14:47:11.742180 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2442 active+clean] publish_stats_to_osd 173:2848
2014-06-12 14:47:11.742189 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2442 active+clean]  sending commit on repgather(0x2fe22480 173'2444 rep_tid=236292 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) 0x56cac80
2014-06-12 14:47:11.742210 7f9d6a3e5700  5 -- op tracker -- , seq: 747951, time: 2014-06-12 14:47:11.742207, event: commit_sent, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.742217 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2442 active+clean]  requeue_ops 
2014-06-12 14:47:11.742232 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2443 active+clean]  removing repgather(0x2fe22480 173'2444 rep_tid=236292 committed?=1 applied?=1 op=osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:11.742242 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2443 active+clean]    q front is repgather(0x2fe22480 173'2444 rep_tid=236292 committed?=1 applied?=1 op=osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:11.742252 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2443 active+clean] remove_repop repgather(0x2fe22480 173'2444 rep_tid=236292 committed?=1 applied?=1 op=osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:11.742261 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2443 active+clean]  requeue_ops 
2014-06-12 14:47:11.742282 7f9d6a3e5700  5 -- op tracker -- , seq: 747951, time: 2014-06-12 14:47:11.742279, event: done, op: osd_op(client.4314.0:145115 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~200704] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.742361 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x25e42900 finish
2014-06-12 14:47:11.742368 7f9d6a3e5700  5 -- op tracker -- , seq: 748346, time: 2014-06-12 14:47:11.742366, event: done, op: osd_sub_op_reply(client.4314.0:145115 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.742375 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:11.742377 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:11.775571 7f9d56432700  5 -- op tracker -- , seq: 748347, time: 2014-06-12 14:47:11.775517, event: header_read, op: osd_sub_op_reply(client.4314.0:145116 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.775584 7f9d56432700  5 -- op tracker -- , seq: 748347, time: 2014-06-12 14:47:11.775519, event: throttled, op: osd_sub_op_reply(client.4314.0:145116 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.775615 7f9d56432700  5 -- op tracker -- , seq: 748347, time: 2014-06-12 14:47:11.775532, event: all_read, op: osd_sub_op_reply(client.4314.0:145116 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.775624 7f9d56432700  5 -- op tracker -- , seq: 748347, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:145116 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.775629 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:145116 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:11.775633 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:11.775639 7f9d56432700 15 osd.2 173 enqueue_op 0x25e42a00 prio 196 cost 0 latency 0.000122 osd_sub_op_reply(client.4314.0:145116 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:11.775699 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:11.775710 7f9d66bde700 10 osd.2 173 dequeue_op 0x25e42a00 prio 196 cost 0 latency 0.000192 osd_sub_op_reply(client.4314.0:145116 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2 pg pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2443 active+clean]
2014-06-12 14:47:11.775728 7f9d66bde700  5 -- op tracker -- , seq: 748347, time: 2014-06-12 14:47:11.775725, event: reached_pg, op: osd_sub_op_reply(client.4314.0:145116 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.775736 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2443 active+clean] handle_message: 0x25e42a00
2014-06-12 14:47:11.775747 7f9d66bde700  5 -- op tracker -- , seq: 748347, time: 2014-06-12 14:47:11.775745, event: started, op: osd_sub_op_reply(client.4314.0:145116 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.775760 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2443 active+clean] sub_op_modify_reply: tid 236293 op  ack_type 4 from 26
2014-06-12 14:47:11.775774 7f9d66bde700  5 -- op tracker -- , seq: 747954, time: 2014-06-12 14:47:11.775768, event: sub_op_commit_rec, op: osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.775779 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2443 active+clean] repop_all_applied: repop tid 236293 all applied 
2014-06-12 14:47:11.775785 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2443 active+clean] eval_repop repgather(0x2236240 173'2445 rep_tid=236293 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:11.775795 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2443 active+clean]  sending ack on repgather(0x2236240 173'2445 rep_tid=236293 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) 0x10096780
2014-06-12 14:47:11.775815 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2444 crt=0'0 lcod 173'2443 mlcod 173'2443 active+clean] repop_all_committed: repop tid 236293 all committed 
2014-06-12 14:47:11.775821 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2443 active+clean] eval_repop repgather(0x2236240 173'2445 rep_tid=236293 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:11.775839 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2443 active+clean] log_op_stats osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 28.147774 lat 28.147798
2014-06-12 14:47:11.775850 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2443 active+clean] publish_stats_to_osd 173:2849
2014-06-12 14:47:11.775858 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2443 active+clean]  sending commit on repgather(0x2236240 173'2445 rep_tid=236293 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) 0x5959180
2014-06-12 14:47:11.775876 7f9d66bde700  5 -- op tracker -- , seq: 747954, time: 2014-06-12 14:47:11.775873, event: commit_sent, op: osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.775883 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2443 active+clean]  requeue_ops 
2014-06-12 14:47:11.775889 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2444 active+clean]  removing repgather(0x2236240 173'2445 rep_tid=236293 committed?=1 applied?=1 op=osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:11.775914 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2444 active+clean]    q front is repgather(0x2236240 173'2445 rep_tid=236293 committed?=1 applied?=1 op=osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:11.775922 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2444 active+clean] remove_repop repgather(0x2236240 173'2445 rep_tid=236293 committed?=1 applied?=1 op=osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:11.775930 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2444 active+clean]  requeue_ops 
2014-06-12 14:47:11.775950 7f9d66bde700  5 -- op tracker -- , seq: 747954, time: 2014-06-12 14:47:11.775947, event: done, op: osd_op(client.4314.0:145116 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704~1032192] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:11.775963 7f9d66bde700 10 osd.2 173 dequeue_op 0x25e42a00 finish
2014-06-12 14:47:11.775967 7f9d66bde700  5 -- op tracker -- , seq: 748347, time: 2014-06-12 14:47:11.775965, event: done, op: osd_sub_op_reply(client.4314.0:145116 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:11.775973 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:11.775975 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:11.841566 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:11.861549 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:11.861601 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:11.865623 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:11.865626 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:11.865628 7f9d74bfa700 20 osd.2 173 _dispatch 0x17705040 ping v1
2014-06-12 14:47:11.865631 7f9d74bfa700 10 osd.2 173 ping from client.4314
2014-06-12 14:47:11.865639 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:11.865640 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:12.060250 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.072618 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.095655 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.125710 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.129966 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.155896 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:12.219616 7f9d56432700  5 -- op tracker -- , seq: 748348, time: 2014-06-12 14:47:12.219546, event: header_read, op: osd_sub_op_reply(client.4314.0:145117 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.219629 7f9d56432700  5 -- op tracker -- , seq: 748348, time: 2014-06-12 14:47:12.219550, event: throttled, op: osd_sub_op_reply(client.4314.0:145117 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.219637 7f9d56432700  5 -- op tracker -- , seq: 748348, time: 2014-06-12 14:47:12.219570, event: all_read, op: osd_sub_op_reply(client.4314.0:145117 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.219652 7f9d56432700  5 -- op tracker -- , seq: 748348, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:145117 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.219657 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:145117 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:12.219663 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:12.219670 7f9d56432700 15 osd.2 173 enqueue_op 0x25e42c00 prio 196 cost 0 latency 0.000124 osd_sub_op_reply(client.4314.0:145117 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:12.219729 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.219756 7f9d64bda700 10 osd.2 173 dequeue_op 0x25e42c00 prio 196 cost 0 latency 0.000210 osd_sub_op_reply(client.4314.0:145117 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2 pg pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2444 active+clean]
2014-06-12 14:47:12.219793 7f9d64bda700  5 -- op tracker -- , seq: 748348, time: 2014-06-12 14:47:12.219787, event: reached_pg, op: osd_sub_op_reply(client.4314.0:145117 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.219809 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2444 active+clean] handle_message: 0x25e42c00
2014-06-12 14:47:12.219830 7f9d64bda700  5 -- op tracker -- , seq: 748348, time: 2014-06-12 14:47:12.219826, event: started, op: osd_sub_op_reply(client.4314.0:145117 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.219844 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2444 active+clean] sub_op_modify_reply: tid 236294 op  ack_type 4 from 26
2014-06-12 14:47:12.219870 7f9d64bda700  5 -- op tracker -- , seq: 747969, time: 2014-06-12 14:47:12.219860, event: sub_op_commit_rec, op: osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.219879 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2444 active+clean] repop_all_applied: repop tid 236294 all applied 
2014-06-12 14:47:12.219893 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2444 active+clean] eval_repop repgather(0x5352240 173'2446 rep_tid=236294 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.219913 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2444 active+clean]  sending ack on repgather(0x5352240 173'2446 rep_tid=236294 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) 0x28ea2a00
2014-06-12 14:47:12.219960 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2445 crt=0'0 lcod 173'2444 mlcod 173'2444 active+clean] repop_all_committed: repop tid 236294 all committed 
2014-06-12 14:47:12.219976 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2444 active+clean] eval_repop repgather(0x5352240 173'2446 rep_tid=236294 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.219997 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2444 active+clean] log_op_stats osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 27.451003 lat 27.451041
2014-06-12 14:47:12.220018 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2444 active+clean] publish_stats_to_osd 173:2850
2014-06-12 14:47:12.220034 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2444 active+clean]  sending commit on repgather(0x5352240 173'2446 rep_tid=236294 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) 0x6824f00
2014-06-12 14:47:12.220075 7f9d64bda700  5 -- op tracker -- , seq: 747969, time: 2014-06-12 14:47:12.220068, event: commit_sent, op: osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.220087 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2444 active+clean]  requeue_ops 
2014-06-12 14:47:12.220101 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2445 active+clean]  removing repgather(0x5352240 173'2446 rep_tid=236294 committed?=1 applied?=1 op=osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.220121 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2445 active+clean]    q front is repgather(0x5352240 173'2446 rep_tid=236294 committed?=1 applied?=1 op=osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.220139 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2445 active+clean] remove_repop repgather(0x5352240 173'2446 rep_tid=236294 committed?=1 applied?=1 op=osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.220157 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2445 active+clean]  requeue_ops 
2014-06-12 14:47:12.220181 7f9d64bda700  5 -- op tracker -- , seq: 747969, time: 2014-06-12 14:47:12.220175, event: done, op: osd_op(client.4314.0:145117 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1232896~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.220205 7f9d64bda700 10 osd.2 173 dequeue_op 0x25e42c00 finish
2014-06-12 14:47:12.220214 7f9d64bda700  5 -- op tracker -- , seq: 748348, time: 2014-06-12 14:47:12.220209, event: done, op: osd_sub_op_reply(client.4314.0:145117 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.220227 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.220232 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.226617 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19429 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,1,0,2,0,5,18])
2014-06-12 14:47:12.226640 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19429 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,1,0,2,0,5,18])
2014-06-12 14:47:12.227207 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:12.227266 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:12.227312 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:12.227344 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:12.227380 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:12.227393 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:12.227418 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:12.227436 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:12.227454 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:12.227466 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:12.227525 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:12.227538 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:12.227549 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:12.227578 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:12.227604 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:12.227629 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:12.227652 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:12.227677 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:12.227706 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:12.227731 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:12.227754 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:12.227778 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:12.231683 7f9d47c58700  5 -- op tracker -- , seq: 748349, time: 2014-06-12 14:47:12.228474, event: header_read, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.231708 7f9d47c58700  5 -- op tracker -- , seq: 748349, time: 2014-06-12 14:47:12.228476, event: throttled, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.231721 7f9d47c58700  5 -- op tracker -- , seq: 748349, time: 2014-06-12 14:47:12.231621, event: all_read, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.231733 7f9d47c58700  5 -- op tracker -- , seq: 748349, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.231743 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:47:12.231779 7f9d47c58700 15 osd.2 173 enqueue_op 0x8dbde00 prio 63 cost 1032192 latency 0.003305 osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:12.231822 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.231837 7f9d653db700 10 osd.2 173 dequeue_op 0x8dbde00 prio 63 cost 1032192 latency 0.003363 osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]
2014-06-12 14:47:12.231873 7f9d653db700  5 -- op tracker -- , seq: 748349, time: 2014-06-12 14:47:12.231866, event: reached_pg, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.231909 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:12.231930 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] handle_message: 0x8dbde00
2014-06-12 14:47:12.231944 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:12.231977 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: found obc in cache: 0xedc6b00
2014-06-12 14:47:12.231994 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: 0xedc6b00 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 rwstate(write n=3 w=0) oi: 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'2978 client.4316.0:147320 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2978) ssc: 0x1550f180 snapset: 0=[]:[]+head
2014-06-12 14:47:12.232018 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] find_object_context 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 @head oi=58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'2978 client.4316.0:147320 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2978)
2014-06-12 14:47:12.232052 7f9d653db700  5 -- op tracker -- , seq: 748349, time: 2014-06-12 14:47:12.232045, event: started, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.232064 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] execute_ctx 0x55a5600
2014-06-12 14:47:12.232088 7f9d653db700  5 -- op tracker -- , seq: 748349, time: 2014-06-12 14:47:12.232081, event: started, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.232102 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] ov 173'2978 av 173'2979 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:12.232123 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192]
2014-06-12 14:47:12.232139 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:12.232157 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  write 1806336~1032192
2014-06-12 14:47:12.232177 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:47:12.232191 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 snapset=0x1550f1b8  snapc=0=[]
2014-06-12 14:47:12.232206 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:12.232219 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] finish_ctx 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 0x55a5600 op modify  
2014-06-12 14:47:12.232235 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  set mtime to 2014-06-12 08:46:14.606644
2014-06-12 14:47:12.232278 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  final snapset 0=[]:[]+head in 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.232306 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  zeroing write result code 0
2014-06-12 14:47:12.232321 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] new_repop rep_tid 236409 on osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:12.232342 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] issue_repop rep_tid 236409 o 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.232373 7f9d653db700  5 -- op tracker -- , seq: 748349, time: 2014-06-12 14:47:12.232366, event: waiting for subops from 24, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.232400 7f9d653db700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:12.232432 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2978 (0'0,173'2978] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log log((0'0,173'2978], crt=0'0) [173'2979 (173'2978) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147321 2014-06-12 08:46:14.606644]
2014-06-12 14:47:12.232466 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] add_log_entry 173'2979 (173'2978) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147321 2014-06-12 08:46:14.606644
2014-06-12 14:47:12.232497 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log  adding 1 keys
2014-06-12 14:47:12.232542 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2979, trimmed: 
2014-06-12 14:47:12.232573 7f9d653db700  5 queue_transactions existing osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:12.232581 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:12.232583 7f9d653db700  5 queue_op 0x142a8720 seq 2708 osr(3.785 0x49b40d0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:12.232591 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] eval_repop repgather(0x2680240 173'2979 rep_tid=236409 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.232602 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54bc360 (1 active)
2014-06-12 14:47:12.232617 7f9d653db700 10 osd.2 173 dequeue_op 0x8dbde00 finish
2014-06-12 14:47:12.232623 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.232621 7f9d7f16b700  5 _do_op 0x142a8720 seq 2708 osr(3.785 0x49b40d0)/0x49b40d0 start
2014-06-12 14:47:12.232627 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.232632 7f9d7f16b700 10 _do_transaction on 0x2680840
2014-06-12 14:47:12.232646 7f9d7f16b700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.232827 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.232875 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:12.232966 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:12.233100 7f9d7f16b700 15 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.233118 7f9d7f16b700 10 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1 r = 0
2014-06-12 14:47:12.233133 7f9d7f16b700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.233172 7f9d7f16b700 15 _write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 1806336~1032192
2014-06-12 14:47:12.233382 7f9d7f16b700 10 lookup_strip_header done  cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.233598 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:47:12.238286 7f9d7f16b700 10 _generic_write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 1806336~1032192 = 0
2014-06-12 14:47:12.238423 7f9d7f16b700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.238458 7f9d7f16b700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:12.238472 7f9d7f16b700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.238490 7f9d7f16b700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:12.238901 7f9d7f16b700 20 set_header setting 17484 cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 parent seq 0
2014-06-12 14:47:12.239070 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:12.239105 7f9d7f16b700 20 set_header setting 2142 cid meta oid 1fe0af7a/pglog_3.785/0//-1 parent seq 0
2014-06-12 14:47:12.239130 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:47:12.246350 7f9d7f16b700 10 _do_op 0x142a8720 seq 2708 r = 0, finisher 0x5749220 0x2519b880
2014-06-12 14:47:12.246369 7f9d7f16b700 10 _finish_op 0x142a8720 seq 2708 osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:12.246378 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54bc360 (0 active)
2014-06-12 14:47:12.246391 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.246404 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_commit: 236409
2014-06-12 14:47:12.246454 7f9d7715b700  5 -- op tracker -- , seq: 748349, time: 2014-06-12 14:47:12.246430, event: op_commit, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.246468 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied: 236409
2014-06-12 14:47:12.246488 7f9d7795c700  5 -- op tracker -- , seq: 748349, time: 2014-06-12 14:47:12.246484, event: op_applied, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.246499 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied on primary on version 173'2979
2014-06-12 14:47:12.322788 7f9d56432700  5 -- op tracker -- , seq: 748350, time: 2014-06-12 14:47:12.322701, event: header_read, op: osd_sub_op_reply(client.4314.0:145118 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.322806 7f9d56432700  5 -- op tracker -- , seq: 748350, time: 2014-06-12 14:47:12.322704, event: throttled, op: osd_sub_op_reply(client.4314.0:145118 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.322831 7f9d56432700  5 -- op tracker -- , seq: 748350, time: 2014-06-12 14:47:12.322737, event: all_read, op: osd_sub_op_reply(client.4314.0:145118 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.322867 7f9d56432700  5 -- op tracker -- , seq: 748350, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:145118 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.322874 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:145118 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:12.322881 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:12.322893 7f9d56432700 15 osd.2 173 enqueue_op 0x25e42d00 prio 196 cost 0 latency 0.000191 osd_sub_op_reply(client.4314.0:145118 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:12.322960 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.322982 7f9d623d5700 10 osd.2 173 dequeue_op 0x25e42d00 prio 196 cost 0 latency 0.000280 osd_sub_op_reply(client.4314.0:145118 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2 pg pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2445 active+clean]
2014-06-12 14:47:12.323025 7f9d623d5700  5 -- op tracker -- , seq: 748350, time: 2014-06-12 14:47:12.323019, event: reached_pg, op: osd_sub_op_reply(client.4314.0:145118 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.323041 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2445 active+clean] handle_message: 0x25e42d00
2014-06-12 14:47:12.323067 7f9d623d5700  5 -- op tracker -- , seq: 748350, time: 2014-06-12 14:47:12.323063, event: started, op: osd_sub_op_reply(client.4314.0:145118 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.323076 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2445 active+clean] sub_op_modify_reply: tid 236295 op  ack_type 4 from 26
2014-06-12 14:47:12.323104 7f9d623d5700  5 -- op tracker -- , seq: 747970, time: 2014-06-12 14:47:12.323093, event: sub_op_commit_rec, op: osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.323116 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2445 active+clean] repop_all_applied: repop tid 236295 all applied 
2014-06-12 14:47:12.323130 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2445 active+clean] eval_repop repgather(0x5237c80 173'2447 rep_tid=236295 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.323151 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2445 active+clean]  sending ack on repgather(0x5237c80 173'2447 rep_tid=236295 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) 0x564cf00
2014-06-12 14:47:12.323198 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2446 crt=0'0 lcod 173'2445 mlcod 173'2445 active+clean] repop_all_committed: repop tid 236295 all committed 
2014-06-12 14:47:12.323214 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2445 active+clean] eval_repop repgather(0x5237c80 173'2447 rep_tid=236295 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.323235 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2445 active+clean] log_op_stats osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 27.549197 lat 27.549235
2014-06-12 14:47:12.323258 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2445 active+clean] publish_stats_to_osd 173:2851
2014-06-12 14:47:12.323273 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2445 active+clean]  sending commit on repgather(0x5237c80 173'2447 rep_tid=236295 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) 0x56cb400
2014-06-12 14:47:12.323320 7f9d623d5700  5 -- op tracker -- , seq: 747970, time: 2014-06-12 14:47:12.323313, event: commit_sent, op: osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.323351 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2445 active+clean]  requeue_ops 
2014-06-12 14:47:12.323366 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean]  removing repgather(0x5237c80 173'2447 rep_tid=236295 committed?=1 applied?=1 op=osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.323385 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean]    q front is repgather(0x5237c80 173'2447 rep_tid=236295 committed?=1 applied?=1 op=osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.323405 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean] remove_repop repgather(0x5237c80 173'2447 rep_tid=236295 committed?=1 applied?=1 op=osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.323426 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean]  requeue_ops 
2014-06-12 14:47:12.323452 7f9d623d5700  5 -- op tracker -- , seq: 747970, time: 2014-06-12 14:47:12.323445, event: done, op: osd_op(client.4314.0:145118 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 1748992~516096] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.323479 7f9d623d5700 10 osd.2 173 dequeue_op 0x25e42d00 finish
2014-06-12 14:47:12.323489 7f9d623d5700  5 -- op tracker -- , seq: 748350, time: 2014-06-12 14:47:12.323484, event: done, op: osd_sub_op_reply(client.4314.0:145118 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.323503 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.323507 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.370359 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:12.446516 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.458589 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.464954 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.467156 7f9d56432700  5 -- op tracker -- , seq: 748351, time: 2014-06-12 14:47:12.467078, event: header_read, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:12.467174 7f9d56432700  5 -- op tracker -- , seq: 748351, time: 2014-06-12 14:47:12.467082, event: throttled, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:12.467184 7f9d56432700  5 -- op tracker -- , seq: 748351, time: 2014-06-12 14:47:12.467105, event: all_read, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:12.467220 7f9d56432700  5 -- op tracker -- , seq: 748351, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:12.467228 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:12.467236 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:12.467247 7f9d56432700 15 osd.2 173 enqueue_op 0x28dd0400 prio 196 cost 0 latency 0.000168 osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0) v2
2014-06-12 14:47:12.467292 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.467332 7f9d56432700  5 -- op tracker -- , seq: 748352, time: 2014-06-12 14:47:12.467280, event: header_read, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.467311 7f9d643d9700 10 osd.2 173 dequeue_op 0x28dd0400 prio 196 cost 0 latency 0.000233 osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0) v2 pg pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean]
2014-06-12 14:47:12.467347 7f9d56432700  5 -- op tracker -- , seq: 748352, time: 2014-06-12 14:47:12.467280, event: throttled, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.467377 7f9d56432700  5 -- op tracker -- , seq: 748352, time: 2014-06-12 14:47:12.467304, event: all_read, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.467400 7f9d56432700  5 -- op tracker -- , seq: 748352, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.467407 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:12.467415 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:12.467423 7f9d56432700 15 osd.2 173 enqueue_op 0x28dd0800 prio 196 cost 0 latency 0.000142 osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:12.467415 7f9d643d9700  5 -- op tracker -- , seq: 748351, time: 2014-06-12 14:47:12.467350, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:12.467438 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean] handle_message: 0x28dd0400
2014-06-12 14:47:12.467461 7f9d643d9700  5 -- op tracker -- , seq: 748351, time: 2014-06-12 14:47:12.467456, event: started, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:12.467461 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:12.467470 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean] sub_op_modify_reply: tid 236318 op  ack_type 1 from 26
2014-06-12 14:47:12.467497 7f9d643d9700  5 -- op tracker -- , seq: 748017, time: 2014-06-12 14:47:12.467486, event: sub_op_applied_rec, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.467520 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean] repop_all_applied: repop tid 236318 all applied 
2014-06-12 14:47:12.467540 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean] eval_repop repgather(0x52d1c80 173'2448 rep_tid=236318 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.467563 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean]  sending ack on repgather(0x52d1c80 173'2448 rep_tid=236318 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) 0xb954500
2014-06-12 14:47:12.467614 7f9d643d9700 10 osd.2 173 dequeue_op 0x28dd0400 finish
2014-06-12 14:47:12.467633 7f9d643d9700  5 -- op tracker -- , seq: 748351, time: 2014-06-12 14:47:12.467627, event: done, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:12.467656 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:12.467660 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.467636 7f9d63bd8700 10 osd.2 173 dequeue_op 0x28dd0800 prio 196 cost 0 latency 0.000355 osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0) v2 pg pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean]
2014-06-12 14:47:12.467685 7f9d63bd8700  5 -- op tracker -- , seq: 748352, time: 2014-06-12 14:47:12.467677, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.467702 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean] handle_message: 0x28dd0800
2014-06-12 14:47:12.467737 7f9d63bd8700  5 -- op tracker -- , seq: 748352, time: 2014-06-12 14:47:12.467728, event: started, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.467761 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean] sub_op_modify_reply: tid 236318 op  ack_type 4 from 26
2014-06-12 14:47:12.467804 7f9d63bd8700  5 -- op tracker -- , seq: 748017, time: 2014-06-12 14:47:12.467789, event: sub_op_commit_rec, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.467819 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2447 crt=0'0 lcod 173'2446 mlcod 173'2446 active+clean] repop_all_committed: repop tid 236318 all committed 
2014-06-12 14:47:12.467848 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2446 active+clean] eval_repop repgather(0x52d1c80 173'2448 rep_tid=236318 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.467891 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2446 active+clean] log_op_stats osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4 inb 479511 outb 0 rlat 27.353620 lat 27.353898
2014-06-12 14:47:12.467931 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2446 active+clean] publish_stats_to_osd 173:2852
2014-06-12 14:47:12.467959 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2446 active+clean]  sending commit on repgather(0x52d1c80 173'2448 rep_tid=236318 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) 0x353e9b80
2014-06-12 14:47:12.468041 7f9d63bd8700  5 -- op tracker -- , seq: 748017, time: 2014-06-12 14:47:12.468028, event: commit_sent, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.468057 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2446 active+clean]  requeue_ops 
2014-06-12 14:47:12.468085 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2447 active+clean]  removing repgather(0x52d1c80 173'2448 rep_tid=236318 committed?=1 applied?=1 op=osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.468122 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2447 active+clean]    q front is repgather(0x52d1c80 173'2448 rep_tid=236318 committed?=1 applied?=1 op=osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.468159 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2447 active+clean] remove_repop repgather(0x52d1c80 173'2448 rep_tid=236318 committed?=1 applied?=1 op=osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.468195 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2447 active+clean]  requeue_ops 
2014-06-12 14:47:12.468238 7f9d63bd8700  5 -- op tracker -- , seq: 748017, time: 2014-06-12 14:47:12.468227, event: done, op: osd_op(client.4306.0:144193 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.468277 7f9d63bd8700 10 osd.2 173 dequeue_op 0x28dd0800 finish
2014-06-12 14:47:12.468297 7f9d63bd8700  5 -- op tracker -- , seq: 748352, time: 2014-06-12 14:47:12.468289, event: done, op: osd_sub_op_reply(client.4306.0:144193 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.468321 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.468328 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.469122 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.475086 7f9d47c58700  5 -- op tracker -- , seq: 748353, time: 2014-06-12 14:47:12.473966, event: header_read, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.475104 7f9d47c58700  5 -- op tracker -- , seq: 748353, time: 2014-06-12 14:47:12.473968, event: throttled, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.475111 7f9d47c58700  5 -- op tracker -- , seq: 748353, time: 2014-06-12 14:47:12.475054, event: all_read, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.475116 7f9d47c58700  5 -- op tracker -- , seq: 748353, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.475122 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:47:12.475143 7f9d47c58700 15 osd.2 173 enqueue_op 0x8dbc900 prio 63 cost 516096 latency 0.001176 osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:12.475164 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.475173 7f9d61bd4700 10 osd.2 173 dequeue_op 0x8dbc900 prio 63 cost 516096 latency 0.001207 osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]
2014-06-12 14:47:12.475191 7f9d61bd4700  5 -- op tracker -- , seq: 748353, time: 2014-06-12 14:47:12.475188, event: reached_pg, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.475208 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:12.475219 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] handle_message: 0x8dbc900
2014-06-12 14:47:12.475226 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:12.475243 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: found obc in cache: 0xedc6b00
2014-06-12 14:47:12.475252 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: 0xedc6b00 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 rwstate(write n=4 w=0) oi: 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'2979 client.4316.0:147321 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2979) ssc: 0x1550f180 snapset: 0=[]:[]+head
2014-06-12 14:47:12.475263 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] find_object_context 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 @head oi=58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'2979 client.4316.0:147321 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2979)
2014-06-12 14:47:12.475282 7f9d61bd4700  5 -- op tracker -- , seq: 748353, time: 2014-06-12 14:47:12.475279, event: started, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.475287 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] execute_ctx 0x55a3800
2014-06-12 14:47:12.475299 7f9d61bd4700  5 -- op tracker -- , seq: 748353, time: 2014-06-12 14:47:12.475296, event: started, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.475306 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] ov 173'2979 av 173'2980 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:12.475315 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096]
2014-06-12 14:47:12.475321 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:12.475329 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  write 2838528~516096
2014-06-12 14:47:12.475341 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:47:12.475348 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 snapset=0x1550f1b8  snapc=0=[]
2014-06-12 14:47:12.475355 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:12.475361 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] finish_ctx 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 0x55a3800 op modify  
2014-06-12 14:47:12.475369 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  set mtime to 2014-06-12 08:46:14.853006
2014-06-12 14:47:12.475383 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  final snapset 0=[]:[]+head in 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.475401 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  zeroing write result code 0
2014-06-12 14:47:12.475417 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] new_repop rep_tid 236410 on osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:12.475427 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] issue_repop rep_tid 236410 o 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.475451 7f9d61bd4700  5 -- op tracker -- , seq: 748353, time: 2014-06-12 14:47:12.475448, event: waiting for subops from 24, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.475466 7f9d61bd4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:12.475482 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2979 (0'0,173'2979] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log log((0'0,173'2979], crt=0'0) [173'2980 (173'2979) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147322 2014-06-12 08:46:14.853006]
2014-06-12 14:47:12.475506 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] add_log_entry 173'2980 (173'2979) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147322 2014-06-12 08:46:14.853006
2014-06-12 14:47:12.475526 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log  adding 1 keys
2014-06-12 14:47:12.475558 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2980, trimmed: 
2014-06-12 14:47:12.475579 7f9d61bd4700  5 queue_transactions existing osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:12.475584 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:12.475586 7f9d61bd4700  5 queue_op 0x2b003c0 seq 2709 osr(3.785 0x49b40d0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:12.475592 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] eval_repop repgather(0x523c600 173'2980 rep_tid=236410 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.475609 7f9d61bd4700 10 osd.2 173 dequeue_op 0x8dbc900 finish
2014-06-12 14:47:12.475613 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.475615 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.475622 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54bc360 (1 active)
2014-06-12 14:47:12.475626 7f9d7e96a700  5 _do_op 0x2b003c0 seq 2709 osr(3.785 0x49b40d0)/0x49b40d0 start
2014-06-12 14:47:12.475630 7f9d7e96a700 10 _do_transaction on 0x523ce40
2014-06-12 14:47:12.475635 7f9d7e96a700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.475747 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.475770 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:12.475826 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:12.475957 7f9d7e96a700 15 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.475966 7f9d7e96a700 10 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1 r = 0
2014-06-12 14:47:12.475970 7f9d7e96a700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.475984 7f9d7e96a700 15 _write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 2838528~516096
2014-06-12 14:47:12.476100 7f9d7e96a700 10 lookup_strip_header done  cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.476241 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:47:12.477468 7f9d7e96a700 10 _generic_write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 2838528~516096 = 0
2014-06-12 14:47:12.477507 7f9d7e96a700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.477517 7f9d7e96a700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:12.477522 7f9d7e96a700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.477527 7f9d7e96a700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:12.477683 7f9d7e96a700 20 set_header setting 17484 cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 parent seq 0
2014-06-12 14:47:12.477740 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:12.477750 7f9d7e96a700 20 set_header setting 2142 cid meta oid 1fe0af7a/pglog_3.785/0//-1 parent seq 0
2014-06-12 14:47:12.477757 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:47:12.480615 7f9d7e96a700 10 _do_op 0x2b003c0 seq 2709 r = 0, finisher 0x70f5240 0x4b31bc0
2014-06-12 14:47:12.480632 7f9d7e96a700 10 _finish_op 0x2b003c0 seq 2709 osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:12.480651 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54bc360 (0 active)
2014-06-12 14:47:12.480654 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.480638 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_commit: 236410
2014-06-12 14:47:12.480667 7f9d7715b700  5 -- op tracker -- , seq: 748353, time: 2014-06-12 14:47:12.480658, event: op_commit, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.480688 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied: 236410
2014-06-12 14:47:12.480707 7f9d7795c700  5 -- op tracker -- , seq: 748353, time: 2014-06-12 14:47:12.480703, event: op_applied, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.480715 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied on primary on version 173'2980
2014-06-12 14:47:12.482672 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:12.482685 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:47:12.482687 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:47:12.482689 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:47:12.482700 7f9d84175700 10 osd.2 173 send_pg_stats - 2 pgs updated
2014-06-12 14:47:12.482731 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:12.482734 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:12.482735 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:12.482737 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 26; oldest is 27.524627 seconds old
2014-06-12 14:47:12.482748 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:12.489469 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.502571 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.508475 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.511222 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.516229 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:12.516283 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:12.546028 7f9d56432700  5 -- op tracker -- , seq: 748354, time: 2014-06-12 14:47:12.545965, event: header_read, op: osd_sub_op_reply(client.4306.0:144194 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.546045 7f9d56432700  5 -- op tracker -- , seq: 748354, time: 2014-06-12 14:47:12.545969, event: throttled, op: osd_sub_op_reply(client.4306.0:144194 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.546057 7f9d56432700  5 -- op tracker -- , seq: 748354, time: 2014-06-12 14:47:12.545995, event: all_read, op: osd_sub_op_reply(client.4306.0:144194 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.546068 7f9d56432700  5 -- op tracker -- , seq: 748354, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144194 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.546074 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144194 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:12.546081 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:12.546089 7f9d56432700 15 osd.2 173 enqueue_op 0x28dd1f00 prio 196 cost 0 latency 0.000124 osd_sub_op_reply(client.4306.0:144194 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:12.546143 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.546166 7f9d67be0700 10 osd.2 173 dequeue_op 0x28dd1f00 prio 196 cost 0 latency 0.000201 osd_sub_op_reply(client.4306.0:144194 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0) v2 pg pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2447 active+clean]
2014-06-12 14:47:12.546203 7f9d67be0700  5 -- op tracker -- , seq: 748354, time: 2014-06-12 14:47:12.546197, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144194 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.546219 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2447 active+clean] handle_message: 0x28dd1f00
2014-06-12 14:47:12.546240 7f9d67be0700  5 -- op tracker -- , seq: 748354, time: 2014-06-12 14:47:12.546235, event: started, op: osd_sub_op_reply(client.4306.0:144194 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.546248 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2447 active+clean] sub_op_modify_reply: tid 236319 op  ack_type 4 from 26
2014-06-12 14:47:12.546274 7f9d67be0700  5 -- op tracker -- , seq: 748022, time: 2014-06-12 14:47:12.546264, event: sub_op_commit_rec, op: osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.546284 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2447 active+clean] repop_all_applied: repop tid 236319 all applied 
2014-06-12 14:47:12.546297 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2447 active+clean] eval_repop repgather(0x2b62b680 173'2449 rep_tid=236319 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.546328 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2447 active+clean]  sending ack on repgather(0x2b62b680 173'2449 rep_tid=236319 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) 0x17433400
2014-06-12 14:47:12.546376 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2448 crt=0'0 lcod 173'2447 mlcod 173'2447 active+clean] repop_all_committed: repop tid 236319 all committed 
2014-06-12 14:47:12.546392 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2447 active+clean] eval_repop repgather(0x2b62b680 173'2449 rep_tid=236319 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.546414 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2447 active+clean] log_op_stats osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 27.386595 lat 27.386632
2014-06-12 14:47:12.546438 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2447 active+clean] publish_stats_to_osd 173:2853
2014-06-12 14:47:12.546454 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2447 active+clean]  sending commit on repgather(0x2b62b680 173'2449 rep_tid=236319 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) 0x6700280
2014-06-12 14:47:12.546496 7f9d67be0700  5 -- op tracker -- , seq: 748022, time: 2014-06-12 14:47:12.546489, event: commit_sent, op: osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.546506 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2447 active+clean]  requeue_ops 
2014-06-12 14:47:12.546520 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2448 active+clean]  removing repgather(0x2b62b680 173'2449 rep_tid=236319 committed?=1 applied?=1 op=osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.546539 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2448 active+clean]    q front is repgather(0x2b62b680 173'2449 rep_tid=236319 committed?=1 applied?=1 op=osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.546557 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2448 active+clean] remove_repop repgather(0x2b62b680 173'2449 rep_tid=236319 committed?=1 applied?=1 op=osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.546576 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2448 active+clean]  requeue_ops 
2014-06-12 14:47:12.546600 7f9d67be0700  5 -- op tracker -- , seq: 748022, time: 2014-06-12 14:47:12.546594, event: done, op: osd_op(client.4306.0:144194 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~1032192] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.546635 7f9d67be0700 10 osd.2 173 dequeue_op 0x28dd1f00 finish
2014-06-12 14:47:12.546645 7f9d67be0700  5 -- op tracker -- , seq: 748354, time: 2014-06-12 14:47:12.546640, event: done, op: osd_sub_op_reply(client.4306.0:144194 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.546688 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.546693 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.606577 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:12.606762 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:12.653515 7f9d47c58700  5 -- op tracker -- , seq: 748355, time: 2014-06-12 14:47:12.652065, event: header_read, op: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.653537 7f9d47c58700  5 -- op tracker -- , seq: 748355, time: 2014-06-12 14:47:12.652080, event: throttled, op: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.653561 7f9d47c58700  5 -- op tracker -- , seq: 748355, time: 2014-06-12 14:47:12.653444, event: all_read, op: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.653573 7f9d47c58700  5 -- op tracker -- , seq: 748355, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.653584 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:47:12.653647 7f9d47c58700 15 osd.2 173 enqueue_op 0x8dbc400 prio 63 cost 516096 latency 0.001581 osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:12.653726 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.653748 7f9d673df700 10 osd.2 173 dequeue_op 0x8dbc400 prio 63 cost 516096 latency 0.001682 osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]
2014-06-12 14:47:12.653791 7f9d673df700  5 -- op tracker -- , seq: 748355, time: 2014-06-12 14:47:12.653783, event: reached_pg, op: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.653808 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:12.653834 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] handle_message: 0x8dbc400
2014-06-12 14:47:12.653849 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:12.653882 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: found obc in cache: 0xedc6b00
2014-06-12 14:47:12.653902 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: 0xedc6b00 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 rwstate(write n=5 w=0) oi: 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'2980 client.4316.0:147322 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2980) ssc: 0x1550f180 snapset: 0=[]:[]+head
2014-06-12 14:47:12.653926 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] find_object_context 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 @head oi=58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'2980 client.4316.0:147322 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2980)
2014-06-12 14:47:12.653961 7f9d673df700  5 -- op tracker -- , seq: 748355, time: 2014-06-12 14:47:12.653954, event: started, op: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.653973 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] execute_ctx 0x2b885000
2014-06-12 14:47:12.653996 7f9d673df700  5 -- op tracker -- , seq: 748355, time: 2014-06-12 14:47:12.653990, event: started, op: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.654004 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] ov 173'2980 av 173'2981 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:12.654023 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096]
2014-06-12 14:47:12.654036 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:12.654054 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  write 3354624~516096
2014-06-12 14:47:12.654073 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:47:12.654088 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 snapset=0x1550f1b8  snapc=0=[]
2014-06-12 14:47:12.654103 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:12.654122 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] finish_ctx 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 0x2b885000 op modify  
2014-06-12 14:47:12.654139 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  set mtime to 2014-06-12 08:46:15.031005
2014-06-12 14:47:12.654169 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  final snapset 0=[]:[]+head in 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.654193 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  zeroing write result code 0
2014-06-12 14:47:12.654207 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] new_repop rep_tid 236411 on osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:12.654229 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] issue_repop rep_tid 236411 o 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.654261 7f9d673df700  5 -- op tracker -- , seq: 748355, time: 2014-06-12 14:47:12.654254, event: waiting for subops from 24, op: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.654292 7f9d673df700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:12.654322 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2980 (0'0,173'2980] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log log((0'0,173'2980], crt=0'0) [173'2981 (173'2980) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147323 2014-06-12 08:46:15.031005]
2014-06-12 14:47:12.654357 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] add_log_entry 173'2981 (173'2980) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147323 2014-06-12 08:46:15.031005
2014-06-12 14:47:12.654386 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log  adding 1 keys
2014-06-12 14:47:12.654428 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2981, trimmed: 
2014-06-12 14:47:12.654457 7f9d673df700  5 queue_transactions existing osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:12.654463 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:12.654465 7f9d673df700  5 queue_op 0x139e3140 seq 2710 osr(3.785 0x49b40d0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:12.654475 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] eval_repop repgather(0x2335d080 173'2981 rep_tid=236411 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.654501 7f9d673df700 10 osd.2 173 dequeue_op 0x8dbc400 finish
2014-06-12 14:47:12.654506 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.654510 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.654510 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54bc360 (1 active)
2014-06-12 14:47:12.654526 7f9d8016d700  5 _do_op 0x139e3140 seq 2710 osr(3.785 0x49b40d0)/0x49b40d0 start
2014-06-12 14:47:12.654533 7f9d8016d700 10 _do_transaction on 0x2335cf00
2014-06-12 14:47:12.654543 7f9d8016d700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.654716 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.654761 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:12.654830 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:12.654866 7f9d8016d700 15 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.654874 7f9d8016d700 10 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1 r = 0
2014-06-12 14:47:12.654881 7f9d8016d700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.655025 7f9d8016d700 15 _write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 3354624~516096
2014-06-12 14:47:12.655234 7f9d8016d700 10 lookup_strip_header done  cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.655387 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:47:12.657589 7f9d8016d700 10 _generic_write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 3354624~516096 = 0
2014-06-12 14:47:12.657683 7f9d8016d700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.657704 7f9d8016d700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:12.657712 7f9d8016d700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.657722 7f9d8016d700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:12.657995 7f9d8016d700 20 set_header setting 17484 cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 parent seq 0
2014-06-12 14:47:12.658097 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:12.658118 7f9d8016d700 20 set_header setting 2142 cid meta oid 1fe0af7a/pglog_3.785/0//-1 parent seq 0
2014-06-12 14:47:12.658131 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:47:12.658685 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:12.658799 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:12.662023 7f9d8016d700 10 _do_op 0x139e3140 seq 2710 r = 0, finisher 0x8e0c280 0x38e0ce40
2014-06-12 14:47:12.662047 7f9d8016d700 10 _finish_op 0x139e3140 seq 2710 osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:12.662056 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54bc360 (0 active)
2014-06-12 14:47:12.662068 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.662096 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_commit: 236411
2014-06-12 14:47:12.662150 7f9d7715b700  5 -- op tracker -- , seq: 748355, time: 2014-06-12 14:47:12.662137, event: op_commit, op: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.662203 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied: 236411
2014-06-12 14:47:12.662243 7f9d7795c700  5 -- op tracker -- , seq: 748355, time: 2014-06-12 14:47:12.662234, event: op_applied, op: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.662273 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied on primary on version 173'2981
2014-06-12 14:47:12.668277 7f9d56432700  5 -- op tracker -- , seq: 748356, time: 2014-06-12 14:47:12.668206, event: header_read, op: osd_sub_op_reply(client.4314.0:145119 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.668295 7f9d56432700  5 -- op tracker -- , seq: 748356, time: 2014-06-12 14:47:12.668208, event: throttled, op: osd_sub_op_reply(client.4314.0:145119 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.668311 7f9d56432700  5 -- op tracker -- , seq: 748356, time: 2014-06-12 14:47:12.668228, event: all_read, op: osd_sub_op_reply(client.4314.0:145119 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.668321 7f9d56432700  5 -- op tracker -- , seq: 748356, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4314.0:145119 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.668327 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4314.0:145119 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:12.668335 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:12.668344 7f9d56432700 15 osd.2 173 enqueue_op 0x28fd8200 prio 196 cost 0 latency 0.000138 osd_sub_op_reply(client.4314.0:145119 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:12.668415 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.668442 7f9d663dd700 10 osd.2 173 dequeue_op 0x28fd8200 prio 196 cost 0 latency 0.000235 osd_sub_op_reply(client.4314.0:145119 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0) v2 pg pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2448 active+clean]
2014-06-12 14:47:12.668480 7f9d663dd700  5 -- op tracker -- , seq: 748356, time: 2014-06-12 14:47:12.668474, event: reached_pg, op: osd_sub_op_reply(client.4314.0:145119 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.668515 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2448 active+clean] handle_message: 0x28fd8200
2014-06-12 14:47:12.668537 7f9d663dd700  5 -- op tracker -- , seq: 748356, time: 2014-06-12 14:47:12.668532, event: started, op: osd_sub_op_reply(client.4314.0:145119 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.668545 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2448 active+clean] sub_op_modify_reply: tid 236321 op  ack_type 4 from 26
2014-06-12 14:47:12.668571 7f9d663dd700  5 -- op tracker -- , seq: 748025, time: 2014-06-12 14:47:12.668561, event: sub_op_commit_rec, op: osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.668580 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2448 active+clean] repop_all_applied: repop tid 236321 all applied 
2014-06-12 14:47:12.668600 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2448 active+clean] eval_repop repgather(0x3a096480 173'2450 rep_tid=236321 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.668625 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2448 active+clean]  sending ack on repgather(0x3a096480 173'2450 rep_tid=236321 committed?=0 applied?=1 lock=0 op=osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) 0x10148000
2014-06-12 14:47:12.668671 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2449 crt=0'0 lcod 173'2448 mlcod 173'2448 active+clean] repop_all_committed: repop tid 236321 all committed 
2014-06-12 14:47:12.668687 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2448 active+clean] eval_repop repgather(0x3a096480 173'2450 rep_tid=236321 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.668708 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2448 active+clean] log_op_stats osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4 inb 1929495 outb 0 rlat 27.508534 lat 27.508572
2014-06-12 14:47:12.668730 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2448 active+clean] publish_stats_to_osd 173:2854
2014-06-12 14:47:12.668747 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2448 active+clean]  sending commit on repgather(0x3a096480 173'2450 rep_tid=236321 committed?=1 applied?=1 lock=0 op=osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4) 0x15570780
2014-06-12 14:47:12.668787 7f9d663dd700  5 -- op tracker -- , seq: 748025, time: 2014-06-12 14:47:12.668780, event: commit_sent, op: osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.668797 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2448 active+clean]  requeue_ops 
2014-06-12 14:47:12.668811 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean]  removing repgather(0x3a096480 173'2450 rep_tid=236321 committed?=1 applied?=1 op=osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.668830 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean]    q front is repgather(0x3a096480 173'2450 rep_tid=236321 committed?=1 applied?=1 op=osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.668849 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] remove_repop repgather(0x3a096480 173'2450 rep_tid=236321 committed?=1 applied?=1 op=osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.668866 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean]  requeue_ops 
2014-06-12 14:47:12.668900 7f9d663dd700  5 -- op tracker -- , seq: 748025, time: 2014-06-12 14:47:12.668893, event: done, op: osd_op(client.4314.0:145119 rbd_data.10b16b8b4567.0000000000001065 [set-alloc-hint object_size 4194304 write_size 4194304,write 2265088~1929216] 3.fd62251 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.668928 7f9d663dd700 10 osd.2 173 dequeue_op 0x28fd8200 finish
2014-06-12 14:47:12.668938 7f9d663dd700  5 -- op tracker -- , seq: 748356, time: 2014-06-12 14:47:12.668933, event: done, op: osd_sub_op_reply(client.4314.0:145119 3.251 fd62251/rbd_data.10b16b8b4567.0000000000001065/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.668951 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.668956 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.682896 7f9d47c58700  5 -- op tracker -- , seq: 748357, time: 2014-06-12 14:47:12.681846, event: header_read, op: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.682926 7f9d47c58700  5 -- op tracker -- , seq: 748357, time: 2014-06-12 14:47:12.681851, event: throttled, op: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.682949 7f9d47c58700  5 -- op tracker -- , seq: 748357, time: 2014-06-12 14:47:12.682832, event: all_read, op: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.682981 7f9d47c58700  5 -- op tracker -- , seq: 748357, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.682996 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:47:12.683040 7f9d47c58700 15 osd.2 173 enqueue_op 0x8dbcb00 prio 63 cost 323584 latency 0.001193 osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:12.683125 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.683147 7f9d65bdc700 10 osd.2 173 dequeue_op 0x8dbcb00 prio 63 cost 323584 latency 0.001300 osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]
2014-06-12 14:47:12.683190 7f9d65bdc700  5 -- op tracker -- , seq: 748357, time: 2014-06-12 14:47:12.683182, event: reached_pg, op: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.683206 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:12.683226 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] handle_message: 0x8dbcb00
2014-06-12 14:47:12.683239 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:12.683286 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: found obc in cache: 0xedc6b00
2014-06-12 14:47:12.683301 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] get_object_context: 0xedc6b00 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 rwstate(write n=6 w=0) oi: 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'2981 client.4316.0:147323 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2981) ssc: 0x1550f180 snapset: 0=[]:[]+head
2014-06-12 14:47:12.683324 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] find_object_context 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 @head oi=58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3(173'2981 client.4316.0:147323 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2981)
2014-06-12 14:47:12.683358 7f9d65bdc700  5 -- op tracker -- , seq: 748357, time: 2014-06-12 14:47:12.683351, event: started, op: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.683370 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] execute_ctx 0x30f7de00
2014-06-12 14:47:12.683393 7f9d65bdc700  5 -- op tracker -- , seq: 748357, time: 2014-06-12 14:47:12.683387, event: started, op: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.683401 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] ov 173'2981 av 173'2982 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:12.683420 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584]
2014-06-12 14:47:12.683434 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:12.683452 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op  write 3870720~323584
2014-06-12 14:47:12.683472 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:47:12.683486 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 snapset=0x1550f1b8  snapc=0=[]
2014-06-12 14:47:12.683504 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] make_writeable 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:12.683518 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] finish_ctx 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 0x30f7de00 op modify  
2014-06-12 14:47:12.683534 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  set mtime to 2014-06-12 08:46:15.061173
2014-06-12 14:47:12.683564 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  final snapset 0=[]:[]+head in 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.683591 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  zeroing write result code 0
2014-06-12 14:47:12.683605 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] new_repop rep_tid 236412 on osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:12.683627 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] issue_repop rep_tid 236412 o 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.683659 7f9d65bdc700  5 -- op tracker -- , seq: 748357, time: 2014-06-12 14:47:12.683652, event: waiting for subops from 24, op: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.683689 7f9d65bdc700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:12.683721 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2981 (0'0,173'2981] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log log((0'0,173'2981], crt=0'0) [173'2982 (173'2981) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147324 2014-06-12 08:46:15.061173]
2014-06-12 14:47:12.683756 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] add_log_entry 173'2982 (173'2981) modify   58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 by client.4316.0:147324 2014-06-12 08:46:15.061173
2014-06-12 14:47:12.683792 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] append_log  adding 1 keys
2014-06-12 14:47:12.683833 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2982, trimmed: 
2014-06-12 14:47:12.683860 7f9d65bdc700  5 queue_transactions existing osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:12.683867 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:12.683869 7f9d65bdc700  5 queue_op 0x8056240 seq 2711 osr(3.785 0x49b40d0) 325877 bytes   (queue has 1 ops and 325877 bytes)
2014-06-12 14:47:12.683886 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54bc360 (1 active)
2014-06-12 14:47:12.683879 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] eval_repop repgather(0x51dc000 173'2982 rep_tid=236412 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.683902 7f9d79960700  5 _do_op 0x8056240 seq 2711 osr(3.785 0x49b40d0)/0x49b40d0 start
2014-06-12 14:47:12.683906 7f9d65bdc700 10 osd.2 173 dequeue_op 0x8dbcb00 finish
2014-06-12 14:47:12.683911 7f9d79960700 10 _do_transaction on 0x51dc840
2014-06-12 14:47:12.683911 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.683915 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.683921 7f9d79960700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.684103 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.684149 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:12.684259 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:12.684301 7f9d79960700 15 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.684317 7f9d79960700 10 _omap_rmkeys meta/1fe0af7a/pglog_3.785/0//-1 r = 0
2014-06-12 14:47:12.684330 7f9d79960700 15 _omap_setkeys meta/1fe0af7a/pglog_3.785/0//-1
2014-06-12 14:47:12.684359 7f9d79960700 15 _write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 3870720~323584
2014-06-12 14:47:12.684583 7f9d79960700 10 lookup_strip_header done  cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.684615 7f9d79960700 10 file_to_extents done 
2014-06-12 14:47:12.686121 7f9d79960700 10 _generic_write 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 3870720~323584 = 0
2014-06-12 14:47:12.686179 7f9d79960700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.686197 7f9d79960700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:12.686206 7f9d79960700 15 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3
2014-06-12 14:47:12.686221 7f9d79960700 10 _setattrs 3.785_head/58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 = 0
2014-06-12 14:47:12.686461 7f9d79960700 20 set_header setting 17484 cid 3.785_head oid 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 parent seq 0
2014-06-12 14:47:12.686535 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:12.686552 7f9d79960700 20 set_header setting 2142 cid meta oid 1fe0af7a/pglog_3.785/0//-1 parent seq 0
2014-06-12 14:47:12.686565 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:47:12.688491 7f9d79960700 10 _do_op 0x8056240 seq 2711 r = 0, finisher 0x7445fc0 0x12d188c0
2014-06-12 14:47:12.688513 7f9d79960700 10 _finish_op 0x8056240 seq 2711 osr(3.785 0x49b40d0)/0x49b40d0
2014-06-12 14:47:12.688527 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54bc360 (0 active)
2014-06-12 14:47:12.688533 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.688525 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_commit: 236412
2014-06-12 14:47:12.688576 7f9d7715b700  5 -- op tracker -- , seq: 748357, time: 2014-06-12 14:47:12.688563, event: op_commit, op: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.688637 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied: 236412
2014-06-12 14:47:12.688690 7f9d7795c700  5 -- op tracker -- , seq: 748357, time: 2014-06-12 14:47:12.688678, event: op_applied, op: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.688711 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] op_applied on primary on version 173'2982
2014-06-12 14:47:12.723979 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:12.724342 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:12.794443 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:12.794450 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:12.794453 7f9d74bfa700 20 osd.2 173 _dispatch 0x20a16700 pg_stats_ack(2 pgs tid 4860) v1
2014-06-12 14:47:12.794458 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:47:12.794467 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:12.794471 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:12.871177 7f9d56432700  5 -- op tracker -- , seq: 748358, time: 2014-06-12 14:47:12.871100, event: header_read, op: osd_sub_op_reply(client.4306.0:144195 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.871196 7f9d56432700  5 -- op tracker -- , seq: 748358, time: 2014-06-12 14:47:12.871104, event: throttled, op: osd_sub_op_reply(client.4306.0:144195 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.871208 7f9d56432700  5 -- op tracker -- , seq: 748358, time: 2014-06-12 14:47:12.871124, event: all_read, op: osd_sub_op_reply(client.4306.0:144195 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.871239 7f9d56432700  5 -- op tracker -- , seq: 748358, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144195 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.871246 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144195 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:12.871253 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:12.871263 7f9d56432700 15 osd.2 173 enqueue_op 0x28fd9600 prio 196 cost 0 latency 0.000163 osd_sub_op_reply(client.4306.0:144195 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:12.871303 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.871316 7f9d62bd6700 10 osd.2 173 dequeue_op 0x28fd9600 prio 196 cost 0 latency 0.000216 osd_sub_op_reply(client.4306.0:144195 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0) v2 pg pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean]
2014-06-12 14:47:12.871348 7f9d62bd6700  5 -- op tracker -- , seq: 748358, time: 2014-06-12 14:47:12.871343, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144195 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.871360 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] handle_message: 0x28fd9600
2014-06-12 14:47:12.871380 7f9d62bd6700  5 -- op tracker -- , seq: 748358, time: 2014-06-12 14:47:12.871376, event: started, op: osd_sub_op_reply(client.4306.0:144195 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.871388 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] sub_op_modify_reply: tid 236322 op  ack_type 4 from 26
2014-06-12 14:47:12.871414 7f9d62bd6700  5 -- op tracker -- , seq: 748028, time: 2014-06-12 14:47:12.871404, event: sub_op_commit_rec, op: osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.871423 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] repop_all_applied: repop tid 236322 all applied 
2014-06-12 14:47:12.871446 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] eval_repop repgather(0x34d8eb40 173'2451 rep_tid=236322 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.871470 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean]  sending ack on repgather(0x34d8eb40 173'2451 rep_tid=236322 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) 0x1285af00
2014-06-12 14:47:12.871513 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2450 crt=0'0 lcod 173'2449 mlcod 173'2449 active+clean] repop_all_committed: repop tid 236322 all committed 
2014-06-12 14:47:12.871529 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2449 active+clean] eval_repop repgather(0x34d8eb40 173'2451 rep_tid=236322 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.871550 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2449 active+clean] log_op_stats osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4 inb 2580759 outb 0 rlat 27.706486 lat 27.706524
2014-06-12 14:47:12.871571 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2449 active+clean] publish_stats_to_osd 173:2855
2014-06-12 14:47:12.871586 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2449 active+clean]  sending commit on repgather(0x34d8eb40 173'2451 rep_tid=236322 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) 0x25edc500
2014-06-12 14:47:12.871627 7f9d62bd6700  5 -- op tracker -- , seq: 748028, time: 2014-06-12 14:47:12.871621, event: commit_sent, op: osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.871640 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2449 active+clean]  requeue_ops 
2014-06-12 14:47:12.871654 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]  removing repgather(0x34d8eb40 173'2451 rep_tid=236322 committed?=1 applied?=1 op=osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.871673 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]    q front is repgather(0x34d8eb40 173'2451 rep_tid=236322 committed?=1 applied?=1 op=osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.871691 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] remove_repop repgather(0x34d8eb40 173'2451 rep_tid=236322 committed?=1 applied?=1 op=osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.871709 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]  requeue_ops 
2014-06-12 14:47:12.871732 7f9d62bd6700  5 -- op tracker -- , seq: 748028, time: 2014-06-12 14:47:12.871726, event: done, op: osd_op(client.4306.0:144195 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 1511424~2580480] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.871790 7f9d62bd6700 10 osd.2 173 dequeue_op 0x28fd9600 finish
2014-06-12 14:47:12.871806 7f9d62bd6700  5 -- op tracker -- , seq: 748358, time: 2014-06-12 14:47:12.871800, event: done, op: osd_sub_op_reply(client.4306.0:144195 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.871821 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.871825 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.944210 7f9d47c58700  5 -- op tracker -- , seq: 748359, time: 2014-06-12 14:47:12.943841, event: header_read, op: osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.944235 7f9d47c58700  5 -- op tracker -- , seq: 748359, time: 2014-06-12 14:47:12.943847, event: throttled, op: osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.944266 7f9d47c58700  5 -- op tracker -- , seq: 748359, time: 2014-06-12 14:47:12.944135, event: all_read, op: osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.944278 7f9d47c58700  5 -- op tracker -- , seq: 748359, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.944289 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:47:12.944332 7f9d47c58700 15 osd.2 173 enqueue_op 0x8dbd100 prio 63 cost 126976 latency 0.000490 osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:12.944398 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.944422 7f9d683e1700 10 osd.2 173 dequeue_op 0x8dbd100 prio 63 cost 126976 latency 0.000581 osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean]
2014-06-12 14:47:12.944486 7f9d683e1700  5 -- op tracker -- , seq: 748359, time: 2014-06-12 14:47:12.944477, event: reached_pg, op: osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.944507 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:12.944530 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] handle_message: 0x8dbd100
2014-06-12 14:47:12.944538 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] do_op osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:12.944557 7f9d683e1700 15 getattr 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 '_'
2014-06-12 14:47:12.944865 7f9d683e1700 10 getattr 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 '_' = 0
2014-06-12 14:47:12.944893 7f9d683e1700 15 getattr 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 'snapset'
2014-06-12 14:47:12.945014 7f9d683e1700 10 getattr 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 'snapset' = 0
2014-06-12 14:47:12.945030 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] populate_obc_watchers ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.945051 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.945064 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] get_object_context: creating obc from disk: 0x6968000
2014-06-12 14:47:12.945077 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] get_object_context: 0x6968000 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 rwstate(none n=0 w=0) oi: ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3(173'928 client.4316.0:41791 wrlock_by=unknown.0.0:0 dirty s 4194304 uv928) ssc: 0x1e15f260 snapset: 0=[]:[]+head
2014-06-12 14:47:12.945097 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] find_object_context ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 @head oi=ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3(173'928 client.4316.0:41791 wrlock_by=unknown.0.0:0 dirty s 4194304 uv928)
2014-06-12 14:47:12.945130 7f9d683e1700  5 -- op tracker -- , seq: 748359, time: 2014-06-12 14:47:12.945126, event: started, op: osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.945145 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] execute_ctx 0x2e413600
2014-06-12 14:47:12.945176 7f9d683e1700  5 -- op tracker -- , seq: 748359, time: 2014-06-12 14:47:12.945173, event: started, op: osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.945182 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] do_op ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] ov 173'928 av 173'3154 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:12.945192 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] do_osd_op ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976]
2014-06-12 14:47:12.945227 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:12.945236 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] do_osd_op  write 0~126976
2014-06-12 14:47:12.945247 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:47:12.945256 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] make_writeable ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 snapset=0x1e15f298  snapc=0=[]
2014-06-12 14:47:12.945263 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] make_writeable ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:12.945269 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] finish_ctx ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 0x2e413600 op modify  
2014-06-12 14:47:12.945277 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean]  set mtime to 2014-06-12 08:46:15.323514
2014-06-12 14:47:12.945289 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean]  final snapset 0=[]:[]+head in ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.945301 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean]  zeroing write result code 0
2014-06-12 14:47:12.945323 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] calc_trim_to 79'152 -> 79'153
2014-06-12 14:47:12.945329 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] new_repop rep_tid 236413 on osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:12.945339 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] issue_repop rep_tid 236413 o ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.945355 7f9d683e1700  5 -- op tracker -- , seq: 748359, time: 2014-06-12 14:47:12.945352, event: waiting for subops from 21, op: osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.945373 7f9d683e1700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:12.945391 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3153 (79'152,173'3153] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] append_log log((79'152,173'3153], crt=0'0) [173'3154 (173'928) modify   ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 by client.4316.0:147334 2014-06-12 08:46:15.323514]
2014-06-12 14:47:12.945432 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'152,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3153 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] add_log_entry 173'3154 (173'928) modify   ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 by client.4316.0:147334 2014-06-12 08:46:15.323514
2014-06-12 14:47:12.945462 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'152,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3153 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] append_log  adding 1 keys
2014-06-12 14:47:12.945473 7f9d683e1700 10 trim log((79'152,173'3154], crt=0'0) to 79'153
2014-06-12 14:47:12.945497 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3154, trimmed: 79'153
2014-06-12 14:47:12.945553 7f9d683e1700  5 queue_transactions existing osr(3.401 0x276d640)/0x276d640
2014-06-12 14:47:12.945559 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:12.945561 7f9d683e1700  5 queue_op 0x6d14240 seq 2883 osr(3.401 0x276d640) 129304 bytes   (queue has 1 ops and 129304 bytes)
2014-06-12 14:47:12.945568 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3153 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] eval_repop repgather(0x13b42780 173'3154 rep_tid=236413 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.945586 7f9d683e1700 10 osd.2 173 dequeue_op 0x8dbd100 finish
2014-06-12 14:47:12.945590 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.945593 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.945594 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54747e0 (1 active)
2014-06-12 14:47:12.945603 7f9d7d968700  5 _do_op 0x6d14240 seq 2883 osr(3.401 0x276d640)/0x276d640 start
2014-06-12 14:47:12.945608 7f9d7d968700 10 _do_transaction on 0x5352900
2014-06-12 14:47:12.945625 7f9d7d968700 15 _omap_setkeys meta/1fe1e78a/pglog_3.401/0//-1
2014-06-12 14:47:12.945918 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe1e78a/pglog_3.401/0//-1
2014-06-12 14:47:12.945977 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:12.946094 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:12.946141 7f9d7d968700 15 _omap_rmkeys meta/1fe1e78a/pglog_3.401/0//-1
2014-06-12 14:47:12.946159 7f9d7d968700 10 _omap_rmkeys meta/1fe1e78a/pglog_3.401/0//-1 r = 0
2014-06-12 14:47:12.946173 7f9d7d968700 15 _omap_setkeys meta/1fe1e78a/pglog_3.401/0//-1
2014-06-12 14:47:12.946390 7f9d7d968700 15 _write 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 0~126976
2014-06-12 14:47:12.946688 7f9d7d968700 10 lookup_strip_header done  cid 3.401_head oid ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.946877 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:47:12.947478 7f9d7d968700 10 _generic_write 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 0~126976 = 0
2014-06-12 14:47:12.947511 7f9d7d968700 15 _setattrs 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.947526 7f9d7d968700 10 _setattrs 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 = 0
2014-06-12 14:47:12.947534 7f9d7d968700 15 _setattrs 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.947544 7f9d7d968700 10 _setattrs 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 = 0
2014-06-12 14:47:12.947721 7f9d7d968700 20 set_header setting 17487 cid 3.401_head oid ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 parent seq 0
2014-06-12 14:47:12.947780 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:12.947799 7f9d7d968700 20 set_header setting 2068 cid meta oid 1fe1e78a/pglog_3.401/0//-1 parent seq 0
2014-06-12 14:47:12.947813 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:47:12.948644 7f9d7d968700 10 _do_op 0x6d14240 seq 2883 r = 0, finisher 0x7229d40 0x33a4a80
2014-06-12 14:47:12.948661 7f9d7d968700 10 _finish_op 0x6d14240 seq 2883 osr(3.401 0x276d640)/0x276d640
2014-06-12 14:47:12.948673 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54747e0 (0 active)
2014-06-12 14:47:12.948677 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.948687 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3153 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] op_applied: 236413
2014-06-12 14:47:12.948732 7f9d7795c700  5 -- op tracker -- , seq: 748359, time: 2014-06-12 14:47:12.948722, event: op_applied, op: osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.948759 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3153 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] op_applied on primary on version 173'3154
2014-06-12 14:47:12.948815 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3153 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] op_commit: 236413
2014-06-12 14:47:12.948867 7f9d7715b700  5 -- op tracker -- , seq: 748359, time: 2014-06-12 14:47:12.948854, event: op_commit, op: osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.949357 7f9d44101700  5 -- op tracker -- , seq: 748360, time: 2014-06-12 14:47:12.949283, event: header_read, op: osd_sub_op_reply(client.4316.0:147334 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.949378 7f9d44101700  5 -- op tracker -- , seq: 748360, time: 2014-06-12 14:47:12.949286, event: throttled, op: osd_sub_op_reply(client.4316.0:147334 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.949407 7f9d44101700  5 -- op tracker -- , seq: 748360, time: 2014-06-12 14:47:12.949302, event: all_read, op: osd_sub_op_reply(client.4316.0:147334 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.949417 7f9d44101700  5 -- op tracker -- , seq: 748360, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147334 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.949424 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147334 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:12.949432 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:47:12.949442 7f9d44101700 15 osd.2 173 enqueue_op 0x3b42a800 prio 196 cost 0 latency 0.000158 osd_sub_op_reply(client.4316.0:147334 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:12.949487 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.949506 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x3b42a800 prio 196 cost 0 latency 0.000223 osd_sub_op_reply(client.4316.0:147334 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0) v2 pg pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3153 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean]
2014-06-12 14:47:12.949556 7f9d6b3e7700  5 -- op tracker -- , seq: 748360, time: 2014-06-12 14:47:12.949547, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147334 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.949584 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3153 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] handle_message: 0x3b42a800
2014-06-12 14:47:12.949608 7f9d6b3e7700  5 -- op tracker -- , seq: 748360, time: 2014-06-12 14:47:12.949603, event: started, op: osd_sub_op_reply(client.4316.0:147334 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.949626 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3153 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] sub_op_modify_reply: tid 236413 op  ack_type 4 from 21
2014-06-12 14:47:12.949653 7f9d6b3e7700  5 -- op tracker -- , seq: 748359, time: 2014-06-12 14:47:12.949643, event: sub_op_commit_rec, op: osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.949662 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3153 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] repop_all_applied: repop tid 236413 all applied 
2014-06-12 14:47:12.949676 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3153 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] eval_repop repgather(0x13b42780 173'3154 rep_tid=236413 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.949696 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3153 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean]  sending ack on repgather(0x13b42780 173'3154 rep_tid=236413 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4) 0x239e2f00
2014-06-12 14:47:12.949738 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3153 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] repop_all_committed: repop tid 236413 all committed 
2014-06-12 14:47:12.949760 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3152 active+clean] eval_repop repgather(0x13b42780 173'3154 rep_tid=236413 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.949794 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3152 active+clean] log_op_stats osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4 inb 127255 outb 0 rlat 0.005894 lat 0.005951
2014-06-12 14:47:12.949830 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3152 active+clean] publish_stats_to_osd 173:4018
2014-06-12 14:47:12.949848 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3152 active+clean]  sending commit on repgather(0x13b42780 173'3154 rep_tid=236413 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4) 0x24674c80
2014-06-12 14:47:12.949895 7f9d6b3e7700  5 -- op tracker -- , seq: 748359, time: 2014-06-12 14:47:12.949885, event: commit_sent, op: osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.949916 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3152 active+clean]  requeue_ops 
2014-06-12 14:47:12.949941 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean]  removing repgather(0x13b42780 173'3154 rep_tid=236413 committed?=1 applied?=1 op=osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.949980 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean]    q front is repgather(0x13b42780 173'3154 rep_tid=236413 committed?=1 applied?=1 op=osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.950001 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] remove_repop repgather(0x13b42780 173'3154 rep_tid=236413 committed?=1 applied?=1 op=osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.950019 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean]  requeue_ops 
2014-06-12 14:47:12.950051 7f9d6b3e7700  5 -- op tracker -- , seq: 748359, time: 2014-06-12 14:47:12.950044, event: done, op: osd_op(client.4316.0:147334 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~126976] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.950158 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x3b42a800 finish
2014-06-12 14:47:12.950173 7f9d6b3e7700  5 -- op tracker -- , seq: 748360, time: 2014-06-12 14:47:12.950167, event: done, op: osd_sub_op_reply(client.4316.0:147334 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.950189 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.950193 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.956234 7f9d47c58700  5 -- op tracker -- , seq: 748361, time: 2014-06-12 14:47:12.956000, event: header_read, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.956254 7f9d47c58700  5 -- op tracker -- , seq: 748361, time: 2014-06-12 14:47:12.956008, event: throttled, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.956267 7f9d47c58700  5 -- op tracker -- , seq: 748361, time: 2014-06-12 14:47:12.956173, event: all_read, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.956279 7f9d47c58700  5 -- op tracker -- , seq: 748361, time: 0.000000, event: dispatched, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.956288 7f9d47c58700 20 osd.2 173 should_share_map client.4316 172.16.96.220:0/1244699 173
2014-06-12 14:47:12.956324 7f9d47c58700 15 osd.2 173 enqueue_op 0x8dbd500 prio 63 cost 90112 latency 0.000323 osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:12.956394 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.956414 7f9d693e3700 10 osd.2 173 dequeue_op 0x8dbd500 prio 63 cost 90112 latency 0.000414 osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean]
2014-06-12 14:47:12.956462 7f9d693e3700  5 -- op tracker -- , seq: 748361, time: 2014-06-12 14:47:12.956447, event: reached_pg, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.956470 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:12.956479 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] handle_message: 0x8dbd500
2014-06-12 14:47:12.956485 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] do_op osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:12.956501 7f9d693e3700 15 getattr 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 '_'
2014-06-12 14:47:12.956622 7f9d693e3700 10 getattr 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 '_' = 0
2014-06-12 14:47:12.956634 7f9d693e3700 15 getattr 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 'snapset'
2014-06-12 14:47:12.956685 7f9d693e3700 10 getattr 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 'snapset' = 0
2014-06-12 14:47:12.956692 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] populate_obc_watchers ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.956699 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.956705 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] get_object_context: creating obc from disk: 0x2e7ad340
2014-06-12 14:47:12.956710 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] get_object_context: 0x2e7ad340 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 rwstate(none n=0 w=0) oi: ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3(173'3154 client.4316.0:147334 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3154) ssc: 0x5d5a700 snapset: 0=[]:[]+head
2014-06-12 14:47:12.956719 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] find_object_context ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 @head oi=ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3(173'3154 client.4316.0:147334 wrlock_by=unknown.0.0:0 dirty s 4194304 uv3154)
2014-06-12 14:47:12.956735 7f9d693e3700  5 -- op tracker -- , seq: 748361, time: 2014-06-12 14:47:12.956732, event: started, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.956740 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] execute_ctx 0x55a3200
2014-06-12 14:47:12.956749 7f9d693e3700  5 -- op tracker -- , seq: 748361, time: 2014-06-12 14:47:12.956747, event: started, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.956753 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] do_op ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] ov 173'3154 av 173'3155 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:12.956761 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] do_osd_op ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112]
2014-06-12 14:47:12.956767 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:12.956778 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] do_osd_op  write 126976~90112
2014-06-12 14:47:12.956787 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] do_osd_op_effects on session 0x5c8c200
2014-06-12 14:47:12.956794 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] make_writeable ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 snapset=0x5d5a738  snapc=0=[]
2014-06-12 14:47:12.956800 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] make_writeable ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:12.956806 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] finish_ctx ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 0x55a3200 op modify  
2014-06-12 14:47:12.956812 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean]  set mtime to 2014-06-12 08:46:15.335687
2014-06-12 14:47:12.956833 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean]  final snapset 0=[]:[]+head in ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.956847 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean]  zeroing write result code 0
2014-06-12 14:47:12.956854 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] calc_trim_to 79'153 -> 79'154
2014-06-12 14:47:12.956860 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] new_repop rep_tid 236414 on osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:12.956872 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] issue_repop rep_tid 236414 o ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.956887 7f9d693e3700  5 -- op tracker -- , seq: 748361, time: 2014-06-12 14:47:12.956884, event: waiting for subops from 21, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.956901 7f9d693e3700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:12.956913 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3154 (79'153,173'3154] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] append_log log((79'153,173'3154], crt=0'0) [173'3155 (173'3154) modify   ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 by client.4316.0:147335 2014-06-12 08:46:15.335687]
2014-06-12 14:47:12.956928 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'153,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] add_log_entry 173'3155 (173'3154) modify   ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 by client.4316.0:147335 2014-06-12 08:46:15.335687
2014-06-12 14:47:12.956950 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'153,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] append_log  adding 1 keys
2014-06-12 14:47:12.956958 7f9d693e3700 10 trim log((79'153,173'3155], crt=0'0) to 79'154
2014-06-12 14:47:12.956975 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'3155, trimmed: 79'154
2014-06-12 14:47:12.956990 7f9d693e3700  5 queue_transactions existing osr(3.401 0x276d640)/0x276d640
2014-06-12 14:47:12.956992 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:12.956993 7f9d693e3700  5 queue_op 0x614d9e0 seq 2884 osr(3.401 0x276d640) 92440 bytes   (queue has 1 ops and 92440 bytes)
2014-06-12 14:47:12.956998 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] eval_repop repgather(0x10192600 173'3155 rep_tid=236414 committed?=0 applied?=0 lock=0 op=osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.957017 7f9d693e3700 10 osd.2 173 dequeue_op 0x8dbd500 finish
2014-06-12 14:47:12.957020 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.957021 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.957030 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54747e0 (1 active)
2014-06-12 14:47:12.957037 7f9d7c165700  5 _do_op 0x614d9e0 seq 2884 osr(3.401 0x276d640)/0x276d640 start
2014-06-12 14:47:12.957048 7f9d7c165700 10 _do_transaction on 0x12ae20c0
2014-06-12 14:47:12.957065 7f9d7c165700 15 _omap_setkeys meta/1fe1e78a/pglog_3.401/0//-1
2014-06-12 14:47:12.957181 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe1e78a/pglog_3.401/0//-1
2014-06-12 14:47:12.957346 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:12.957429 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:12.957463 7f9d7c165700 15 _omap_rmkeys meta/1fe1e78a/pglog_3.401/0//-1
2014-06-12 14:47:12.957476 7f9d7c165700 10 _omap_rmkeys meta/1fe1e78a/pglog_3.401/0//-1 r = 0
2014-06-12 14:47:12.957485 7f9d7c165700 15 _omap_setkeys meta/1fe1e78a/pglog_3.401/0//-1
2014-06-12 14:47:12.957506 7f9d7c165700 15 _write 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 126976~90112
2014-06-12 14:47:12.957867 7f9d7c165700 10 lookup_strip_header done  cid 3.401_head oid ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.957899 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:47:12.958335 7f9d7c165700 10 _generic_write 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 126976~90112 = 0
2014-06-12 14:47:12.958358 7f9d7c165700 15 _setattrs 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.958373 7f9d7c165700 10 _setattrs 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 = 0
2014-06-12 14:47:12.958380 7f9d7c165700 15 _setattrs 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3
2014-06-12 14:47:12.958390 7f9d7c165700 10 _setattrs 3.401_head/ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 = 0
2014-06-12 14:47:12.958565 7f9d7c165700 20 set_header setting 17487 cid 3.401_head oid ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 parent seq 0
2014-06-12 14:47:12.958598 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:12.958614 7f9d7c165700 20 set_header setting 2068 cid meta oid 1fe1e78a/pglog_3.401/0//-1 parent seq 0
2014-06-12 14:47:12.958626 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:47:12.959242 7f9d7c165700 10 _do_op 0x614d9e0 seq 2884 r = 0, finisher 0x2ef7d880 0x2ea2440
2014-06-12 14:47:12.959257 7f9d7c165700 10 _finish_op 0x614d9e0 seq 2884 osr(3.401 0x276d640)/0x276d640
2014-06-12 14:47:12.959264 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54747e0 (0 active)
2014-06-12 14:47:12.959269 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:12.959289 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] op_applied: 236414
2014-06-12 14:47:12.959334 7f9d7795c700  5 -- op tracker -- , seq: 748361, time: 2014-06-12 14:47:12.959323, event: op_applied, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.959351 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] op_applied on primary on version 173'3155
2014-06-12 14:47:12.959408 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] op_commit: 236414
2014-06-12 14:47:12.959462 7f9d7715b700  5 -- op tracker -- , seq: 748361, time: 2014-06-12 14:47:12.959449, event: op_commit, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.959496 7f9d44101700  5 -- op tracker -- , seq: 748362, time: 2014-06-12 14:47:12.959434, event: header_read, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ack, result = 0)
2014-06-12 14:47:12.959502 7f9d44101700  5 -- op tracker -- , seq: 748362, time: 2014-06-12 14:47:12.959436, event: throttled, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ack, result = 0)
2014-06-12 14:47:12.959509 7f9d44101700  5 -- op tracker -- , seq: 748362, time: 2014-06-12 14:47:12.959450, event: all_read, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ack, result = 0)
2014-06-12 14:47:12.959535 7f9d44101700  5 -- op tracker -- , seq: 748362, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ack, result = 0)
2014-06-12 14:47:12.959542 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:12.959548 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:47:12.959557 7f9d44101700 15 osd.2 173 enqueue_op 0x3b42bd00 prio 196 cost 0 latency 0.000123 osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ack, result = 0) v2
2014-06-12 14:47:12.959606 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:12.959617 7f9d6abe6700 10 osd.2 173 dequeue_op 0x3b42bd00 prio 196 cost 0 latency 0.000183 osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ack, result = 0) v2 pg pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean]
2014-06-12 14:47:12.959644 7f9d44101700  5 -- op tracker -- , seq: 748363, time: 2014-06-12 14:47:12.959583, event: header_read, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.959663 7f9d44101700  5 -- op tracker -- , seq: 748363, time: 2014-06-12 14:47:12.959584, event: throttled, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.959676 7f9d44101700  5 -- op tracker -- , seq: 748363, time: 2014-06-12 14:47:12.959600, event: all_read, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.959694 7f9d44101700  5 -- op tracker -- , seq: 748363, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.959701 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:12.959708 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:47:12.959715 7f9d44101700 15 osd.2 173 enqueue_op 0x2911500 prio 196 cost 0 latency 0.000132 osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:12.959710 7f9d6abe6700  5 -- op tracker -- , seq: 748362, time: 2014-06-12 14:47:12.959649, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ack, result = 0)
2014-06-12 14:47:12.959731 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] handle_message: 0x3b42bd00
2014-06-12 14:47:12.959753 7f9d6abe6700  5 -- op tracker -- , seq: 748362, time: 2014-06-12 14:47:12.959748, event: started, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ack, result = 0)
2014-06-12 14:47:12.959776 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] sub_op_modify_reply: tid 236414 op  ack_type 1 from 21
2014-06-12 14:47:12.959803 7f9d6abe6700  5 -- op tracker -- , seq: 748361, time: 2014-06-12 14:47:12.959793, event: sub_op_applied_rec, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.959813 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:12.959815 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] repop_all_applied: repop tid 236414 all applied 
2014-06-12 14:47:12.959830 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] eval_repop repgather(0x10192600 173'3155 rep_tid=236414 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.959852 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean]  sending ack on repgather(0x10192600 173'3155 rep_tid=236414 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4) 0x28676c80
2014-06-12 14:47:12.959890 7f9d6abe6700 10 osd.2 173 dequeue_op 0x3b42bd00 finish
2014-06-12 14:47:12.959907 7f9d6abe6700  5 -- op tracker -- , seq: 748362, time: 2014-06-12 14:47:12.959900, event: done, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ack, result = 0)
2014-06-12 14:47:12.959938 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:12.959945 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.959961 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2911500 prio 196 cost 0 latency 0.000378 osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0) v2 pg pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean]
2014-06-12 14:47:12.960009 7f9d6a3e5700  5 -- op tracker -- , seq: 748363, time: 2014-06-12 14:47:12.960003, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.960045 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] handle_message: 0x2911500
2014-06-12 14:47:12.960059 7f9d6a3e5700  5 -- op tracker -- , seq: 748363, time: 2014-06-12 14:47:12.960057, event: started, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.960063 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] sub_op_modify_reply: tid 236414 op  ack_type 4 from 21
2014-06-12 14:47:12.960074 7f9d6a3e5700  5 -- op tracker -- , seq: 748361, time: 2014-06-12 14:47:12.960070, event: sub_op_commit_rec, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.960078 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'3154 crt=0'0 lcod 173'3153 mlcod 173'3153 active+clean] repop_all_committed: repop tid 236414 all committed 
2014-06-12 14:47:12.960084 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3153 active+clean] eval_repop repgather(0x10192600 173'3155 rep_tid=236414 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:12.960094 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3153 active+clean] log_op_stats osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4 inb 90391 outb 0 rlat 0.003887 lat 0.004094
2014-06-12 14:47:12.960104 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3153 active+clean] publish_stats_to_osd 173:4019
2014-06-12 14:47:12.960111 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3153 active+clean]  sending commit on repgather(0x10192600 173'3155 rep_tid=236414 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4) 0x17be8280
2014-06-12 14:47:12.960132 7f9d6a3e5700  5 -- op tracker -- , seq: 748361, time: 2014-06-12 14:47:12.960129, event: commit_sent, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.960138 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3153 active+clean]  requeue_ops 
2014-06-12 14:47:12.960143 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3154 active+clean]  removing repgather(0x10192600 173'3155 rep_tid=236414 committed?=1 applied?=1 op=osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.960153 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3154 active+clean]    q front is repgather(0x10192600 173'3155 rep_tid=236414 committed?=1 applied?=1 op=osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.960162 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3154 active+clean] remove_repop repgather(0x10192600 173'3155 rep_tid=236414 committed?=1 applied?=1 op=osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:12.960173 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3154 active+clean]  requeue_ops 
2014-06-12 14:47:12.960186 7f9d6a3e5700  5 -- op tracker -- , seq: 748361, time: 2014-06-12 14:47:12.960183, event: done, op: osd_op(client.4316.0:147335 rbd_data.10b36b8b4567.0000000000001140 [set-alloc-hint object_size 4194304 write_size 4194304,write 126976~90112] 3.ffc70401 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:12.960196 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x2911500 finish
2014-06-12 14:47:12.960202 7f9d6a3e5700  5 -- op tracker -- , seq: 748363, time: 2014-06-12 14:47:12.960200, event: done, op: osd_sub_op_reply(client.4316.0:147335 3.401 ffc70401/rbd_data.10b36b8b4567.0000000000001140/head//3 [] ondisk, result = 0)
2014-06-12 14:47:12.960210 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:12.960213 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:12.962398 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:12.962480 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:13.027675 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:13.027718 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:13.142691 7f9d56432700  5 -- op tracker -- , seq: 748364, time: 2014-06-12 14:47:13.142615, event: header_read, op: osd_sub_op_reply(client.4290.0:143736 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.142712 7f9d56432700  5 -- op tracker -- , seq: 748364, time: 2014-06-12 14:47:13.142619, event: throttled, op: osd_sub_op_reply(client.4290.0:143736 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.142744 7f9d56432700  5 -- op tracker -- , seq: 748364, time: 2014-06-12 14:47:13.142639, event: all_read, op: osd_sub_op_reply(client.4290.0:143736 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.142754 7f9d56432700  5 -- op tracker -- , seq: 748364, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143736 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.142760 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143736 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:13.142768 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:13.142778 7f9d56432700 15 osd.2 173 enqueue_op 0xe20af00 prio 196 cost 0 latency 0.000162 osd_sub_op_reply(client.4290.0:143736 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:13.142845 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:13.142860 7f9d66bde700 10 osd.2 173 dequeue_op 0xe20af00 prio 196 cost 0 latency 0.000245 osd_sub_op_reply(client.4290.0:143736 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2 pg pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]
2014-06-12 14:47:13.142894 7f9d66bde700  5 -- op tracker -- , seq: 748364, time: 2014-06-12 14:47:13.142889, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143736 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.142919 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] handle_message: 0xe20af00
2014-06-12 14:47:13.142941 7f9d66bde700  5 -- op tracker -- , seq: 748364, time: 2014-06-12 14:47:13.142936, event: started, op: osd_sub_op_reply(client.4290.0:143736 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.142959 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] sub_op_modify_reply: tid 236336 op  ack_type 4 from 26
2014-06-12 14:47:13.142986 7f9d66bde700  5 -- op tracker -- , seq: 748082, time: 2014-06-12 14:47:13.142975, event: sub_op_commit_rec, op: osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:13.142995 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] repop_all_applied: repop tid 236336 all applied 
2014-06-12 14:47:13.143008 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] eval_repop repgather(0x2f89f980 173'2513 rep_tid=236336 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:13.143030 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean]  sending ack on repgather(0x2f89f980 173'2513 rep_tid=236336 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) 0x17432a00
2014-06-12 14:47:13.143071 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2512 crt=0'0 lcod 173'2511 mlcod 173'2511 active+clean] repop_all_committed: repop tid 236336 all committed 
2014-06-12 14:47:13.143084 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2511 active+clean] eval_repop repgather(0x2f89f980 173'2513 rep_tid=236336 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:13.143105 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2511 active+clean] log_op_stats osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 inb 98583 outb 0 rlat 26.171472 lat 26.171507
2014-06-12 14:47:13.143126 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2511 active+clean] publish_stats_to_osd 173:2883
2014-06-12 14:47:13.143142 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2511 active+clean]  sending commit on repgather(0x2f89f980 173'2513 rep_tid=236336 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) 0x3b82a780
2014-06-12 14:47:13.143183 7f9d66bde700  5 -- op tracker -- , seq: 748082, time: 2014-06-12 14:47:13.143176, event: commit_sent, op: osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:13.143193 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2511 active+clean]  requeue_ops 
2014-06-12 14:47:13.143206 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean]  removing repgather(0x2f89f980 173'2513 rep_tid=236336 committed?=1 applied?=1 op=osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:13.143225 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean]    q front is repgather(0x2f89f980 173'2513 rep_tid=236336 committed?=1 applied?=1 op=osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:13.143243 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean] remove_repop repgather(0x2f89f980 173'2513 rep_tid=236336 committed?=1 applied?=1 op=osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:13.143261 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean]  requeue_ops 
2014-06-12 14:47:13.143286 7f9d66bde700  5 -- op tracker -- , seq: 748082, time: 2014-06-12 14:47:13.143280, event: done, op: osd_op(client.4290.0:143736 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~98304] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:13.143381 7f9d66bde700 10 osd.2 173 dequeue_op 0xe20af00 finish
2014-06-12 14:47:13.143391 7f9d66bde700  5 -- op tracker -- , seq: 748364, time: 2014-06-12 14:47:13.143386, event: done, op: osd_sub_op_reply(client.4290.0:143736 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.143405 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:13.143410 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:13.188989 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:13.210625 7f9d56432700  5 -- op tracker -- , seq: 748365, time: 2014-06-12 14:47:13.210548, event: header_read, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:13.210642 7f9d56432700  5 -- op tracker -- , seq: 748365, time: 2014-06-12 14:47:13.210551, event: throttled, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:13.210652 7f9d56432700  5 -- op tracker -- , seq: 748365, time: 2014-06-12 14:47:13.210580, event: all_read, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:13.210693 7f9d56432700  5 -- op tracker -- , seq: 748365, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:13.210699 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:13.210706 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:13.210717 7f9d56432700 15 osd.2 173 enqueue_op 0x329ebf00 prio 196 cost 0 latency 0.000168 osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0) v2
2014-06-12 14:47:13.210788 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:13.210794 7f9d56432700  5 -- op tracker -- , seq: 748366, time: 2014-06-12 14:47:13.210747, event: header_read, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.210821 7f9d56432700  5 -- op tracker -- , seq: 748366, time: 2014-06-12 14:47:13.210748, event: throttled, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.210817 7f9d69be4700 10 osd.2 173 dequeue_op 0x329ebf00 prio 196 cost 0 latency 0.000269 osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0) v2 pg pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]
2014-06-12 14:47:13.210848 7f9d56432700  5 -- op tracker -- , seq: 748366, time: 2014-06-12 14:47:13.210760, event: all_read, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.210864 7f9d56432700  5 -- op tracker -- , seq: 748366, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.210872 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:13.210879 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:13.210886 7f9d56432700 15 osd.2 173 enqueue_op 0x124e2700 prio 196 cost 0 latency 0.000139 osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:13.210881 7f9d69be4700  5 -- op tracker -- , seq: 748365, time: 2014-06-12 14:47:13.210847, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:13.210904 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] handle_message: 0x329ebf00
2014-06-12 14:47:13.210928 7f9d69be4700  5 -- op tracker -- , seq: 748365, time: 2014-06-12 14:47:13.210923, event: started, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:13.210936 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] sub_op_modify_reply: tid 236323 op  ack_type 1 from 26
2014-06-12 14:47:13.210950 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:13.210962 7f9d69be4700  5 -- op tracker -- , seq: 748029, time: 2014-06-12 14:47:13.210952, event: sub_op_applied_rec, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:13.210970 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] repop_all_applied: repop tid 236323 all applied 
2014-06-12 14:47:13.210984 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] eval_repop repgather(0x52f5ec0 173'2452 rep_tid=236323 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:13.211005 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]  sending ack on repgather(0x52f5ec0 173'2452 rep_tid=236323 committed?=0 applied?=1 lock=0 op=osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) 0x17f01b80
2014-06-12 14:47:13.211045 7f9d69be4700 10 osd.2 173 dequeue_op 0x329ebf00 finish
2014-06-12 14:47:13.211058 7f9d69be4700  5 -- op tracker -- , seq: 748365, time: 2014-06-12 14:47:13.211053, event: done, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ack, result = 0)
2014-06-12 14:47:13.211082 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:13.211087 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:13.211105 7f9d64bda700 10 osd.2 173 dequeue_op 0x124e2700 prio 196 cost 0 latency 0.000357 osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0) v2 pg pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean]
2014-06-12 14:47:13.211143 7f9d64bda700  5 -- op tracker -- , seq: 748366, time: 2014-06-12 14:47:13.211137, event: reached_pg, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.211165 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] handle_message: 0x124e2700
2014-06-12 14:47:13.211188 7f9d64bda700  5 -- op tracker -- , seq: 748366, time: 2014-06-12 14:47:13.211183, event: started, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.211196 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] sub_op_modify_reply: tid 236323 op  ack_type 4 from 26
2014-06-12 14:47:13.211221 7f9d64bda700  5 -- op tracker -- , seq: 748029, time: 2014-06-12 14:47:13.211212, event: sub_op_commit_rec, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:13.211232 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2451 crt=0'0 lcod 173'2450 mlcod 173'2450 active+clean] repop_all_committed: repop tid 236323 all committed 
2014-06-12 14:47:13.211245 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2450 active+clean] eval_repop repgather(0x52f5ec0 173'2452 rep_tid=236323 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:13.211267 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2450 active+clean] log_op_stats osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4 inb 102679 outb 0 rlat 27.993283 lat 27.993506
2014-06-12 14:47:13.211291 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2450 active+clean] publish_stats_to_osd 173:2856
2014-06-12 14:47:13.211311 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2450 active+clean]  sending commit on repgather(0x52f5ec0 173'2452 rep_tid=236323 committed?=1 applied?=1 lock=0 op=osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4) 0x1a58cf00
2014-06-12 14:47:13.211361 7f9d64bda700  5 -- op tracker -- , seq: 748029, time: 2014-06-12 14:47:13.211354, event: commit_sent, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:13.211371 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2450 active+clean]  requeue_ops 
2014-06-12 14:47:13.211385 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean]  removing repgather(0x52f5ec0 173'2452 rep_tid=236323 committed?=1 applied?=1 op=osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:13.211404 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean]    q front is repgather(0x52f5ec0 173'2452 rep_tid=236323 committed?=1 applied?=1 op=osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:13.211422 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] remove_repop repgather(0x52f5ec0 173'2452 rep_tid=236323 committed?=1 applied?=1 op=osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:13.211442 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean]  requeue_ops 
2014-06-12 14:47:13.211474 7f9d64bda700  5 -- op tracker -- , seq: 748029, time: 2014-06-12 14:47:13.211467, event: done, op: osd_op(client.4306.0:144196 rbd_data.10a96b8b4567.0000000000000faa [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.3eb12a51 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:13.211544 7f9d64bda700 10 osd.2 173 dequeue_op 0x124e2700 finish
2014-06-12 14:47:13.211557 7f9d64bda700  5 -- op tracker -- , seq: 748366, time: 2014-06-12 14:47:13.211552, event: done, op: osd_sub_op_reply(client.4306.0:144196 3.251 3eb12a51/rbd_data.10a96b8b4567.0000000000000faa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:13.211572 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:13.211577 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:13.250746 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:13.250782 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:13.343904 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:13.344238 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:13.408946 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:13.442922 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:13.482815 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:13.482845 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:13.482850 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:13.482853 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:13.482856 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 23; oldest is 28.524745 seconds old
2014-06-12 14:47:13.482887 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:13.751458 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:13.751501 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:13.841654 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:13.844328 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:13.844370 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:14.060345 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.072698 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.095724 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.130041 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.155993 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:14.232691 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.246449 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.323584 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.370443 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:14.446603 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.458672 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.465041 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.467735 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.468404 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.469212 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.475706 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.480724 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.482947 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:14.482974 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:14.482978 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:14.482981 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:14.482983 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 23; oldest is 29.524873 seconds old
2014-06-12 14:47:14.483020 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:14.489568 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.502663 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.508552 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.511295 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.512187 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:14.512338 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:14.527287 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19424 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,2,3,2,5,11])
2014-06-12 14:47:14.527301 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19424 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,2,3,2,5,11])
2014-06-12 14:47:14.527804 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:14.527829 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:14.527843 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:14.527857 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:14.527869 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:14.527881 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:14.527893 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:14.527904 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:14.527914 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:14.527948 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:14.527992 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:14.528015 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:14.528027 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:14.528038 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:14.528053 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:14.528076 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:14.528092 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:14.528103 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:14.528115 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:14.528128 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:14.528138 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:14.528149 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:14.546769 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.654582 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.662131 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.669047 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.684004 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.688603 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.848236 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:14.848240 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:14.848242 7f9d74bfa700 20 osd.2 173 _dispatch 0x104f7860 ping v1
2014-06-12 14:47:14.848245 7f9d74bfa700 10 osd.2 173 ping from client.4290
2014-06-12 14:47:14.848248 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:14.848249 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:14.871898 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.945667 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.948656 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:14.948736 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:14.949122 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.950261 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.957093 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.959351 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:14.960013 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:14.960290 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.028108 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:15.028526 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:15.143486 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.160791 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:15.160794 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:15.160795 7f9d74bfa700 20 osd.2 173 _dispatch 0x32119ba0 ping v1
2014-06-12 14:47:15.160797 7f9d74bfa700 10 osd.2 173 ping from client.4293
2014-06-12 14:47:15.160799 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:15.160800 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:15.189092 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:15.211154 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.211648 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.267213 7f9d5a16f700  5 -- op tracker -- , seq: 748367, time: 2014-06-12 14:47:15.265936, event: header_read, op: osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.267230 7f9d5a16f700  5 -- op tracker -- , seq: 748367, time: 2014-06-12 14:47:15.265939, event: throttled, op: osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.267240 7f9d5a16f700  5 -- op tracker -- , seq: 748367, time: 2014-06-12 14:47:15.267178, event: all_read, op: osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.267249 7f9d5a16f700  5 -- op tracker -- , seq: 748367, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.267254 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:15.267260 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:15.267267 7f9d5a16f700 15 osd.2 173 enqueue_op 0x20354d00 prio 127 cost 516903 latency 0.001330 osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:15.267288 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.267294 7f9d653db700 10 osd.2 173 dequeue_op 0x20354d00 prio 127 cost 516903 latency 0.001358 osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2868 (0'0,173'2868] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2867 active]
2014-06-12 14:47:15.267311 7f9d653db700  5 -- op tracker -- , seq: 748367, time: 2014-06-12 14:47:15.267308, event: reached_pg, op: osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.267322 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2868 (0'0,173'2868] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2867 active] handle_message: 0x20354d00
2014-06-12 14:47:15.267330 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2868 (0'0,173'2868] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2867 active] sub_op_modify trans f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 v 173'2869 (transaction) 174
2014-06-12 14:47:15.267343 7f9d653db700  5 -- op tracker -- , seq: 748367, time: 2014-06-12 14:47:15.267339, event: started, op: osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.267356 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2868 (0'0,173'2868] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2867 active] append_log log((0'0,173'2868], crt=0'0) [173'2869 (173'2868) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145701 2014-06-12 08:43:44.727270]
2014-06-12 14:47:15.267376 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2869 (0'0,173'2869] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2867 active] add_log_entry 173'2869 (173'2868) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145701 2014-06-12 08:43:44.727270
2014-06-12 14:47:15.267393 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2869 (0'0,173'2869] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2867 active] append_log  adding 1 keys
2014-06-12 14:47:15.267414 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2869, trimmed: 
2014-06-12 14:47:15.267430 7f9d653db700  5 -- op tracker -- , seq: 748367, time: 2014-06-12 14:47:15.267426, event: started, op: osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.267436 7f9d653db700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:15.267439 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:15.267440 7f9d653db700  5 queue_op 0x142a8a80 seq 2582 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:15.267444 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2869 (0'0,173'2869] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2867 active] do_sub_op osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:15.267453 7f9d653db700 10 osd.2 173 dequeue_op 0x20354d00 finish
2014-06-12 14:47:15.267456 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.267451 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:15.267457 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.267460 7f9d7c966700  5 _do_op 0x142a8a80 seq 2582 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:15.267466 7f9d7c966700 10 _do_transaction on 0xc9b4d30
2014-06-12 14:47:15.267472 7f9d7c966700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:15.267597 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:15.267622 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:15.267672 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:15.267701 7f9d7c966700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:15.267708 7f9d7c966700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:15.267714 7f9d7c966700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:15.267860 7f9d7c966700 15 _write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 958464~516096
2014-06-12 14:47:15.267989 7f9d7c966700 10 lookup_strip_header done  cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:15.268144 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:47:15.269914 7f9d7c966700 10 _generic_write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 958464~516096 = 0
2014-06-12 14:47:15.269973 7f9d7c966700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:15.269992 7f9d7c966700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:15.269999 7f9d7c966700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:15.270008 7f9d7c966700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:15.270201 7f9d7c966700 20 set_header setting 16959 cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 parent seq 0
2014-06-12 14:47:15.270280 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:15.270297 7f9d7c966700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:15.270308 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:47:15.272624 7f9d7c966700 10 _do_op 0x142a8a80 seq 2582 r = 0, finisher 0xc52b3c0 0
2014-06-12 14:47:15.272690 7f9d7715b700  5 -- op tracker -- , seq: 748367, time: 2014-06-12 14:47:15.272677, event: commit_sent, op: osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.272706 7f9d7c966700 10 _finish_op 0x142a8a80 seq 2582 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:15.272707 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2869 (0'0,173'2869] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2867 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:15.272725 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:15.272735 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:15.272743 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:15.272816 7f9d7795c700  5 -- op tracker -- , seq: 748367, time: 2014-06-12 14:47:15.272809, event: sub_op_applied, op: osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.272823 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2869 (0'0,173'2869] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2868 active] sub_op_modify_applied on 0xc9b4c40 op osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:15.272847 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2869 (0'0,173'2869] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2868 active] op_applied on primary on version 173'2869
2014-06-12 14:47:15.272859 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2869 (0'0,173'2869] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2868 active] op_applied on replica on version 173'2869
2014-06-12 14:47:15.272885 7f9d7795c700  5 -- op tracker -- , seq: 748367, time: 2014-06-12 14:47:15.272882, event: done, op: osd_sub_op(client.4302.0:145701 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2869 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.281509 7f9d5a16f700  5 -- op tracker -- , seq: 748368, time: 2014-06-12 14:47:15.281093, event: header_read, op: osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.281524 7f9d5a16f700  5 -- op tracker -- , seq: 748368, time: 2014-06-12 14:47:15.281095, event: throttled, op: osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.281532 7f9d5a16f700  5 -- op tracker -- , seq: 748368, time: 2014-06-12 14:47:15.281470, event: all_read, op: osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.281540 7f9d5a16f700  5 -- op tracker -- , seq: 748368, time: 0.000000, event: dispatched, op: osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.281545 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:15.281552 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:15.281560 7f9d5a16f700 15 osd.2 173 enqueue_op 0x20354900 prio 127 cost 221991 latency 0.000467 osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:15.281590 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.281599 7f9d623d5700 10 osd.2 173 dequeue_op 0x20354900 prio 127 cost 221991 latency 0.000505 osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2515 (0'0,173'2515] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2514 active]
2014-06-12 14:47:15.281652 7f9d623d5700  5 -- op tracker -- , seq: 748368, time: 2014-06-12 14:47:15.281647, event: reached_pg, op: osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.281683 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2515 (0'0,173'2515] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2514 active] handle_message: 0x20354900
2014-06-12 14:47:15.281698 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2515 (0'0,173'2515] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2514 active] sub_op_modify trans f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 v 173'2516 (transaction) 174
2014-06-12 14:47:15.281712 7f9d623d5700  5 -- op tracker -- , seq: 748368, time: 2014-06-12 14:47:15.281709, event: started, op: osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.281726 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2515 (0'0,173'2515] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2514 active] append_log log((0'0,173'2515], crt=0'0) [173'2516 (173'2515) modify   f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 by client.4310.0:142331 2014-06-12 08:44:24.347933]
2014-06-12 14:47:15.281743 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2516 (0'0,173'2516] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2514 active] add_log_entry 173'2516 (173'2515) modify   f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 by client.4310.0:142331 2014-06-12 08:44:24.347933
2014-06-12 14:47:15.281757 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2516 (0'0,173'2516] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2514 active] append_log  adding 1 keys
2014-06-12 14:47:15.281787 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2516, trimmed: 
2014-06-12 14:47:15.281808 7f9d623d5700  5 -- op tracker -- , seq: 748368, time: 2014-06-12 14:47:15.281804, event: started, op: osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.281816 7f9d623d5700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:15.281820 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:15.281822 7f9d623d5700  5 queue_op 0x72c5380 seq 2362 osr(3.168 0x276ce40) 223477 bytes   (queue has 1 ops and 223477 bytes)
2014-06-12 14:47:15.281832 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:47:15.281828 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2516 (0'0,173'2516] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2514 active] do_sub_op osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:15.281839 7f9d623d5700 10 osd.2 173 dequeue_op 0x20354900 finish
2014-06-12 14:47:15.281841 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.281843 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.281842 7f9d81970700  5 _do_op 0x72c5380 seq 2362 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:47:15.281848 7f9d81970700 10 _do_transaction on 0x24eb10b0
2014-06-12 14:47:15.281854 7f9d81970700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:15.281994 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:15.282026 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:15.282085 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:15.282277 7f9d81970700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:15.282288 7f9d81970700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:47:15.282294 7f9d81970700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:47:15.282318 7f9d81970700 15 _write 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 3973120~221184
2014-06-12 14:47:15.282502 7f9d81970700 10 lookup_strip_header done  cid 3.168_head oid f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:15.282734 7f9d81970700 10 file_to_extents done 
2014-06-12 14:47:15.283603 7f9d81970700 10 _generic_write 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 3973120~221184 = 0
2014-06-12 14:47:15.283647 7f9d81970700 15 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:15.283661 7f9d81970700 10 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 = 0
2014-06-12 14:47:15.283668 7f9d81970700 15 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3
2014-06-12 14:47:15.283676 7f9d81970700 10 _setattrs 3.168_head/f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 = 0
2014-06-12 14:47:15.283829 7f9d81970700 20 set_header setting 16446 cid 3.168_head oid f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 parent seq 0
2014-06-12 14:47:15.283874 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:15.283897 7f9d81970700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:47:15.283910 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:47:15.285120 7f9d81970700 10 _do_op 0x72c5380 seq 2362 r = 0, finisher 0x7bd58a0 0
2014-06-12 14:47:15.285157 7f9d7715b700  5 -- op tracker -- , seq: 748368, time: 2014-06-12 14:47:15.285149, event: commit_sent, op: osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.285167 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2516 (0'0,173'2516] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2514 active] sub_op_modify_commit on op osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:15.285192 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:15.285239 7f9d81970700 10 _finish_op 0x72c5380 seq 2362 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:47:15.285267 7f9d7795c700  5 -- op tracker -- , seq: 748368, time: 2014-06-12 14:47:15.285262, event: sub_op_applied, op: osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.285274 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2516 (0'0,173'2516] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2515 active] sub_op_modify_applied on 0x24eb0fc0 op osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:15.285289 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2516 (0'0,173'2516] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2515 active] op_applied on primary on version 173'2516
2014-06-12 14:47:15.285299 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2516 (0'0,173'2516] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2515 active] op_applied on replica on version 173'2516
2014-06-12 14:47:15.285333 7f9d81970700  5 -- op tracker -- , seq: 748368, time: 2014-06-12 14:47:15.285323, event: done, op: osd_sub_op(client.4310.0:142331 3.168 f4646968/rbd_data.10ad6b8b4567.0000000000000eeb/head//3 [] v 173'2516 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.285351 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:47:15.285354 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:15.449130 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:15.449525 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:15.483049 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:15.483080 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:15.483084 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:15.483086 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:15.483084 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:15.483089 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 23; oldest is 30.524978 seconds old
2014-06-12 14:47:15.483192 7f9d84175700  0 log [WRN] : 5 slow requests, 5 included below; oldest blocked for > 30.524978 secs
2014-06-12 14:47:15.483200 7f9d84175700  0 log [WRN] : slow request 30.524978 seconds old, received at 2014-06-12 14:46:44.958110: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:47:15.483204 7f9d84175700  0 log [WRN] : slow request 30.517307 seconds old, received at 2014-06-12 14:46:44.965781: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:47:15.483206 7f9d84175700  0 log [WRN] : slow request 30.369179 seconds old, received at 2014-06-12 14:46:45.113909: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:47:15.483208 7f9d84175700  0 log [WRN] : slow request 30.318022 seconds old, received at 2014-06-12 14:46:45.165066: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:47:15.483210 7f9d84175700  0 log [WRN] : slow request 30.122845 seconds old, received at 2014-06-12 14:46:45.360243: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:47:15.559421 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:15.559449 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:15.624538 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:15.624765 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:15.655880 7f9d4a5d9700  5 -- op tracker -- , seq: 748369, time: 2014-06-12 14:47:15.655543, event: header_read, op: osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.655904 7f9d4a5d9700  5 -- op tracker -- , seq: 748369, time: 2014-06-12 14:47:15.655549, event: throttled, op: osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.655912 7f9d4a5d9700  5 -- op tracker -- , seq: 748369, time: 2014-06-12 14:47:15.655823, event: all_read, op: osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.655929 7f9d4a5d9700  5 -- op tracker -- , seq: 748369, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.655939 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:15.655967 7f9d4a5d9700 15 osd.2 173 enqueue_op 0xe20be00 prio 63 cost 102400 latency 0.000424 osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.656016 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.656028 7f9d68be2700 10 osd.2 173 dequeue_op 0xe20be00 prio 63 cost 102400 latency 0.000485 osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]
2014-06-12 14:47:15.656098 7f9d68be2700  5 -- op tracker -- , seq: 748369, time: 2014-06-12 14:47:15.656077, event: reached_pg, op: osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.656113 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:15.656132 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] handle_message: 0xe20be00
2014-06-12 14:47:15.656146 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_op osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:15.656190 7f9d68be2700 15 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 '_'
2014-06-12 14:47:15.663518 7f9d4a5d9700  5 -- op tracker -- , seq: 748370, time: 2014-06-12 14:47:15.662208, event: header_read, op: osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.663528 7f9d4a5d9700  5 -- op tracker -- , seq: 748370, time: 2014-06-12 14:47:15.662213, event: throttled, op: osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.663534 7f9d4a5d9700  5 -- op tracker -- , seq: 748370, time: 2014-06-12 14:47:15.663472, event: all_read, op: osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.663539 7f9d4a5d9700  5 -- op tracker -- , seq: 748370, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.663547 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:15.663573 7f9d4a5d9700 15 osd.2 173 enqueue_op 0xe20a000 prio 63 cost 516096 latency 0.001364 osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.663602 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:15.693388 7f9d68be2700 10 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 '_' = 0
2014-06-12 14:47:15.693420 7f9d68be2700 15 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 'snapset'
2014-06-12 14:47:15.693552 7f9d68be2700 10 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 'snapset' = 0
2014-06-12 14:47:15.693569 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] populate_obc_watchers 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.693582 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.693589 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] get_object_context: creating obc from disk: 0x33d75340
2014-06-12 14:47:15.693595 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] get_object_context: 0x33d75340 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 rwstate(none n=0 w=0) oi: 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'706 client.4305.0:36555 wrlock_by=unknown.0.0:0 dirty s 4194304 uv706) ssc: 0x407f0a0 snapset: 0=[]:[]+head
2014-06-12 14:47:15.693628 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] find_object_context 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 @head oi=35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'706 client.4305.0:36555 wrlock_by=unknown.0.0:0 dirty s 4194304 uv706)
2014-06-12 14:47:15.693662 7f9d68be2700  5 -- op tracker -- , seq: 748369, time: 2014-06-12 14:47:15.693655, event: started, op: osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.693672 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] execute_ctx 0x2a82600
2014-06-12 14:47:15.693692 7f9d68be2700  5 -- op tracker -- , seq: 748369, time: 2014-06-12 14:47:15.693688, event: started, op: osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.693696 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] ov 173'706 av 173'2740 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:15.693705 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_osd_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400]
2014-06-12 14:47:15.693711 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:15.693720 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_osd_op  write 0~102400
2014-06-12 14:47:15.693731 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:15.693737 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 snapset=0x407f0d8  snapc=0=[]
2014-06-12 14:47:15.693744 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:15.693750 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] finish_ctx 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 0x2a82600 op modify  
2014-06-12 14:47:15.693758 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]  set mtime to 2014-06-12 08:46:18.035294
2014-06-12 14:47:15.693771 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]  final snapset 0=[]:[]+head in 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.693783 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]  zeroing write result code 0
2014-06-12 14:47:15.693789 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] new_repop rep_tid 236415 on osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.693799 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] issue_repop rep_tid 236415 o 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.693815 7f9d68be2700  5 -- op tracker -- , seq: 748369, time: 2014-06-12 14:47:15.693812, event: waiting for subops from 21, op: osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.693831 7f9d68be2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:15.693848 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2739 (0'0,173'2739] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] append_log log((0'0,173'2739], crt=0'0) [173'2740 (173'706) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142593 2014-06-12 08:46:18.035294]
2014-06-12 14:47:15.693872 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] add_log_entry 173'2740 (173'706) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142593 2014-06-12 08:46:18.035294
2014-06-12 14:47:15.693891 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] append_log  adding 1 keys
2014-06-12 14:47:15.693912 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2740, trimmed: 
2014-06-12 14:47:15.693929 7f9d68be2700  5 queue_transactions existing osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.693933 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:15.693934 7f9d68be2700  5 queue_op 0x67a1080 seq 2541 osr(3.4d4 0x276d810) 104693 bytes   (queue has 1 ops and 104693 bytes)
2014-06-12 14:47:15.693940 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] eval_repop repgather(0x5323e00 173'2740 rep_tid=236415 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.693946 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478fc0 (1 active)
2014-06-12 14:47:15.693953 7f9d68be2700 10 osd.2 173 dequeue_op 0xe20be00 finish
2014-06-12 14:47:15.693956 7f9d7a962700  5 _do_op 0x67a1080 seq 2541 osr(3.4d4 0x276d810)/0x276d810 start
2014-06-12 14:47:15.693959 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:15.693961 7f9d7a962700 10 _do_transaction on 0xba84240
2014-06-12 14:47:15.693963 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.693967 7f9d7a962700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.693968 7f9d643d9700 10 osd.2 173 dequeue_op 0xe20a000 prio 63 cost 516096 latency 0.031760 osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]
2014-06-12 14:47:15.693998 7f9d643d9700  5 -- op tracker -- , seq: 748370, time: 2014-06-12 14:47:15.693992, event: reached_pg, op: osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.694008 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:15.694022 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] handle_message: 0xe20a000
2014-06-12 14:47:15.694032 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_op osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:15.694060 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.694054 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] get_object_context: found obc in cache: 0x33d75340
2014-06-12 14:47:15.694065 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] get_object_context: 0x33d75340 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 rwstate(write n=1 w=0) oi: 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'2740 client.4305.0:142593 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2740) ssc: 0x407f0a0 snapset: 0=[]:[]+head
2014-06-12 14:47:15.694081 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:15.694079 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] find_object_context 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 @head oi=35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'2740 client.4305.0:142593 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2740)
2014-06-12 14:47:15.694100 7f9d643d9700  5 -- op tracker -- , seq: 748370, time: 2014-06-12 14:47:15.694095, event: started, op: osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.694113 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:15.694107 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] execute_ctx 0x1a916000
2014-06-12 14:47:15.694124 7f9d643d9700  5 -- op tracker -- , seq: 748370, time: 2014-06-12 14:47:15.694119, event: started, op: osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.694130 7f9d7a962700 15 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.694134 7f9d7a962700 10 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1 r = 0
2014-06-12 14:47:15.694138 7f9d7a962700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.694131 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] ov 173'2740 av 173'2741 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:15.694145 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_osd_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096]
2014-06-12 14:47:15.694156 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:15.694168 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_osd_op  write 102400~516096
2014-06-12 14:47:15.694180 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:15.694191 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 snapset=0x407f0d8  snapc=0=[]
2014-06-12 14:47:15.694206 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:15.694216 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] finish_ctx 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 0x1a916000 op modify  
2014-06-12 14:47:15.694228 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]  set mtime to 2014-06-12 08:46:18.041241
2014-06-12 14:47:15.694248 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]  final snapset 0=[]:[]+head in 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.694264 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]  zeroing write result code 0
2014-06-12 14:47:15.694277 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] new_repop rep_tid 236416 on osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.694291 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] issue_repop rep_tid 236416 o 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.694312 7f9d643d9700  5 -- op tracker -- , seq: 748370, time: 2014-06-12 14:47:15.694307, event: waiting for subops from 21, op: osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.694322 7f9d7a962700 15 _write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 0~102400
2014-06-12 14:47:15.694331 7f9d643d9700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:15.694350 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2740 (0'0,173'2740] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] append_log log((0'0,173'2740], crt=0'0) [173'2741 (173'2740) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142594 2014-06-12 08:46:18.041241]
2014-06-12 14:47:15.694372 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] add_log_entry 173'2741 (173'2740) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142594 2014-06-12 08:46:18.041241
2014-06-12 14:47:15.694398 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] append_log  adding 1 keys
2014-06-12 14:47:15.694428 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2741, trimmed: 
2014-06-12 14:47:15.694448 7f9d643d9700  5 queue_transactions existing osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.694451 7f9d7a962700 10 lookup_strip_header done  cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.694453 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:15.694454 7f9d643d9700  5 queue_op 0x68ea660 seq 2542 osr(3.4d4 0x276d810) 518389 bytes   (queue has 2 ops and 623082 bytes)
2014-06-12 14:47:15.694462 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478fc0 (2 active)
2014-06-12 14:47:15.694461 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] eval_repop repgather(0x9908c00 173'2741 rep_tid=236416 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.694478 7f9d643d9700 10 osd.2 173 dequeue_op 0xe20a000 finish
2014-06-12 14:47:15.694482 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.694485 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.694581 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:47:15.694890 7f9d7a962700 10 _generic_write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 0~102400 = 0
2014-06-12 14:47:15.694907 7f9d7a962700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.694918 7f9d7a962700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.694925 7f9d7a962700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.694933 7f9d7a962700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.695030 7f9d7a962700 20 set_header setting 16502 cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 parent seq 0
2014-06-12 14:47:15.695074 7f9d7a962700 20 set_header setting 2082 cid meta oid 1fe07345/pglog_3.4d4/0//-1 parent seq 0
2014-06-12 14:47:15.695087 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:15.695097 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:47:15.695458 7f9d7a962700 10 _do_op 0x67a1080 seq 2541 r = 0, finisher 0x5b76100 0x3dbb580
2014-06-12 14:47:15.695467 7f9d7a962700 10 _finish_op 0x67a1080 seq 2541 osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.695473 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478fc0 (1 active)
2014-06-12 14:47:15.695475 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:15.695476 7f9d82972700  5 _do_op 0x68ea660 seq 2542 osr(3.4d4 0x276d810)/0x276d810 start
2014-06-12 14:47:15.695481 7f9d82972700 10 _do_transaction on 0xc9b9680
2014-06-12 14:47:15.695471 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] op_commit: 236415
2014-06-12 14:47:15.695486 7f9d82972700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.695493 7f9d7715b700  5 -- op tracker -- , seq: 748369, time: 2014-06-12 14:47:15.695487, event: op_commit, op: osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.695516 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] op_applied: 236415
2014-06-12 14:47:15.695538 7f9d7795c700  5 -- op tracker -- , seq: 748369, time: 2014-06-12 14:47:15.695532, event: op_applied, op: osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.695551 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.695546 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] op_applied on primary on version 173'2740
2014-06-12 14:47:15.695567 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:15.695598 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:15.695673 7f9d82972700 15 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.695678 7f9d82972700 10 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1 r = 0
2014-06-12 14:47:15.695682 7f9d82972700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.695693 7f9d82972700 15 _write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 102400~516096
2014-06-12 14:47:15.695778 7f9d82972700 10 lookup_strip_header done  cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.695877 7f9d82972700 10 file_to_extents done 
2014-06-12 14:47:15.696436 7f9d44101700  5 -- op tracker -- , seq: 748371, time: 2014-06-12 14:47:15.696398, event: header_read, op: osd_sub_op_reply(client.4305.0:142593 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.696445 7f9d44101700  5 -- op tracker -- , seq: 748371, time: 2014-06-12 14:47:15.696399, event: throttled, op: osd_sub_op_reply(client.4305.0:142593 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.696459 7f9d44101700  5 -- op tracker -- , seq: 748371, time: 2014-06-12 14:47:15.696410, event: all_read, op: osd_sub_op_reply(client.4305.0:142593 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.696463 7f9d44101700  5 -- op tracker -- , seq: 748371, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142593 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.696468 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142593 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:15.696472 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:47:15.696479 7f9d44101700 15 osd.2 173 enqueue_op 0x2911300 prio 196 cost 0 latency 0.000081 osd_sub_op_reply(client.4305.0:142593 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:15.696516 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.696524 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2911300 prio 196 cost 0 latency 0.000126 osd_sub_op_reply(client.4305.0:142593 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 pg pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]
2014-06-12 14:47:15.696552 7f9d63bd8700  5 -- op tracker -- , seq: 748371, time: 2014-06-12 14:47:15.696548, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142593 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.696559 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] handle_message: 0x2911300
2014-06-12 14:47:15.696574 7f9d63bd8700  5 -- op tracker -- , seq: 748371, time: 2014-06-12 14:47:15.696571, event: started, op: osd_sub_op_reply(client.4305.0:142593 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.696580 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] sub_op_modify_reply: tid 236415 op  ack_type 4 from 21
2014-06-12 14:47:15.696599 7f9d63bd8700  5 -- op tracker -- , seq: 748369, time: 2014-06-12 14:47:15.696592, event: sub_op_commit_rec, op: osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.696606 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] repop_all_applied: repop tid 236415 all applied 
2014-06-12 14:47:15.696616 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] eval_repop repgather(0x5323e00 173'2740 rep_tid=236415 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.696632 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean]  sending ack on repgather(0x5323e00 173'2740 rep_tid=236415 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0x22b52780
2014-06-12 14:47:15.696662 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2739 crt=0'0 lcod 173'2738 mlcod 173'2738 active+clean] repop_all_committed: repop tid 236415 all committed 
2014-06-12 14:47:15.696673 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2738 active+clean] eval_repop repgather(0x5323e00 173'2740 rep_tid=236415 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.696688 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2738 active+clean] log_op_stats osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 inb 102679 outb 0 rlat 0.041118 lat 0.041144
2014-06-12 14:47:15.696705 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2738 active+clean] publish_stats_to_osd 173:3538
2014-06-12 14:47:15.696718 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2738 active+clean]  sending commit on repgather(0x5323e00 173'2740 rep_tid=236415 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0x1a668c80
2014-06-12 14:47:15.696750 7f9d63bd8700  5 -- op tracker -- , seq: 748369, time: 2014-06-12 14:47:15.696744, event: commit_sent, op: osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.696757 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2738 active+clean]  requeue_ops 
2014-06-12 14:47:15.696768 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean]  removing repgather(0x5323e00 173'2740 rep_tid=236415 committed?=1 applied?=1 op=osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.696781 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean]    q front is repgather(0x5323e00 173'2740 rep_tid=236415 committed?=1 applied?=1 op=osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.696795 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] remove_repop repgather(0x5323e00 173'2740 rep_tid=236415 committed?=1 applied?=1 op=osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.696808 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean]  requeue_ops 
2014-06-12 14:47:15.696827 7f9d63bd8700  5 -- op tracker -- , seq: 748369, time: 2014-06-12 14:47:15.696822, event: done, op: osd_op(client.4305.0:142593 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~102400] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.696900 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2911300 finish
2014-06-12 14:47:15.696908 7f9d63bd8700  5 -- op tracker -- , seq: 748371, time: 2014-06-12 14:47:15.696905, event: done, op: osd_sub_op_reply(client.4305.0:142593 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.696919 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.696922 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.697190 7f9d82972700 10 _generic_write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 102400~516096 = 0
2014-06-12 14:47:15.697225 7f9d82972700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.697236 7f9d82972700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.697239 7f9d82972700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.697244 7f9d82972700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.697378 7f9d82972700 20 set_header setting 16502 cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 parent seq 0
2014-06-12 14:47:15.697434 7f9d82972700 20 set_header setting 2082 cid meta oid 1fe07345/pglog_3.4d4/0//-1 parent seq 0
2014-06-12 14:47:15.697445 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:15.697450 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:47:15.699330 7f9d82972700 10 _do_op 0x68ea660 seq 2542 r = 0, finisher 0x698c9e0 0xcb07840
2014-06-12 14:47:15.699345 7f9d82972700 10 _finish_op 0x68ea660 seq 2542 osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.699353 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478fc0 (0 active)
2014-06-12 14:47:15.699357 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:15.699350 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] op_commit: 236416
2014-06-12 14:47:15.699385 7f9d7715b700  5 -- op tracker -- , seq: 748370, time: 2014-06-12 14:47:15.699376, event: op_commit, op: osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.699432 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] op_applied: 236416
2014-06-12 14:47:15.699457 7f9d7795c700  5 -- op tracker -- , seq: 748370, time: 2014-06-12 14:47:15.699451, event: op_applied, op: osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.699464 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] op_applied on primary on version 173'2741
2014-06-12 14:47:15.700317 7f9d44101700  5 -- op tracker -- , seq: 748372, time: 2014-06-12 14:47:15.700266, event: header_read, op: osd_sub_op_reply(client.4305.0:142594 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.700326 7f9d44101700  5 -- op tracker -- , seq: 748372, time: 2014-06-12 14:47:15.700267, event: throttled, op: osd_sub_op_reply(client.4305.0:142594 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.700343 7f9d44101700  5 -- op tracker -- , seq: 748372, time: 2014-06-12 14:47:15.700289, event: all_read, op: osd_sub_op_reply(client.4305.0:142594 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.700356 7f9d44101700  5 -- op tracker -- , seq: 748372, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142594 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.700360 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142594 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:15.700364 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:47:15.700370 7f9d44101700 15 osd.2 173 enqueue_op 0x20e89700 prio 196 cost 0 latency 0.000104 osd_sub_op_reply(client.4305.0:142594 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:15.700430 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.700440 7f9d61bd4700 10 osd.2 173 dequeue_op 0x20e89700 prio 196 cost 0 latency 0.000174 osd_sub_op_reply(client.4305.0:142594 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 pg pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean]
2014-06-12 14:47:15.700459 7f9d61bd4700  5 -- op tracker -- , seq: 748372, time: 2014-06-12 14:47:15.700456, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142594 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.700467 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] handle_message: 0x20e89700
2014-06-12 14:47:15.700477 7f9d61bd4700  5 -- op tracker -- , seq: 748372, time: 2014-06-12 14:47:15.700475, event: started, op: osd_sub_op_reply(client.4305.0:142594 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.700482 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] sub_op_modify_reply: tid 236416 op  ack_type 4 from 21
2014-06-12 14:47:15.700505 7f9d61bd4700  5 -- op tracker -- , seq: 748370, time: 2014-06-12 14:47:15.700489, event: sub_op_commit_rec, op: osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.700512 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] repop_all_applied: repop tid 236416 all applied 
2014-06-12 14:47:15.700519 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] eval_repop repgather(0x9908c00 173'2741 rep_tid=236416 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.700528 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean]  sending ack on repgather(0x9908c00 173'2741 rep_tid=236416 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0x1a668f00
2014-06-12 14:47:15.700629 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2740 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] repop_all_committed: repop tid 236416 all committed 
2014-06-12 14:47:15.700652 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2739 active+clean] eval_repop repgather(0x9908c00 173'2741 rep_tid=236416 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.700675 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2739 active+clean] log_op_stats osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.038419 lat 0.038466
2014-06-12 14:47:15.700687 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2739 active+clean] publish_stats_to_osd 173:3539
2014-06-12 14:47:15.700710 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2739 active+clean]  sending commit on repgather(0x9908c00 173'2741 rep_tid=236416 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0x10096a00
2014-06-12 14:47:15.700739 7f9d61bd4700  5 -- op tracker -- , seq: 748370, time: 2014-06-12 14:47:15.700728, event: commit_sent, op: osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.700747 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2739 active+clean]  requeue_ops 
2014-06-12 14:47:15.700759 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean]  removing repgather(0x9908c00 173'2741 rep_tid=236416 committed?=1 applied?=1 op=osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.700776 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean]    q front is repgather(0x9908c00 173'2741 rep_tid=236416 committed?=1 applied?=1 op=osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.700785 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] remove_repop repgather(0x9908c00 173'2741 rep_tid=236416 committed?=1 applied?=1 op=osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.700793 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean]  requeue_ops 
2014-06-12 14:47:15.700810 7f9d61bd4700  5 -- op tracker -- , seq: 748370, time: 2014-06-12 14:47:15.700807, event: done, op: osd_op(client.4305.0:142594 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 102400~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.700839 7f9d61bd4700 10 osd.2 173 dequeue_op 0x20e89700 finish
2014-06-12 14:47:15.700845 7f9d61bd4700  5 -- op tracker -- , seq: 748372, time: 2014-06-12 14:47:15.700843, event: done, op: osd_sub_op_reply(client.4305.0:142594 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.700859 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.700862 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.703041 7f9d4a5d9700  5 -- op tracker -- , seq: 748373, time: 2014-06-12 14:47:15.700509, event: header_read, op: osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.703052 7f9d4a5d9700  5 -- op tracker -- , seq: 748373, time: 2014-06-12 14:47:15.700511, event: throttled, op: osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.703064 7f9d4a5d9700  5 -- op tracker -- , seq: 748373, time: 2014-06-12 14:47:15.703017, event: all_read, op: osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.703070 7f9d4a5d9700  5 -- op tracker -- , seq: 748373, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.703075 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:15.703095 7f9d4a5d9700 15 osd.2 173 enqueue_op 0xe20b500 prio 63 cost 1032192 latency 0.002586 osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.703144 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.703153 7f9d633d7700 10 osd.2 173 dequeue_op 0xe20b500 prio 63 cost 1032192 latency 0.002644 osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean]
2014-06-12 14:47:15.703175 7f9d633d7700  5 -- op tracker -- , seq: 748373, time: 2014-06-12 14:47:15.703171, event: reached_pg, op: osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.703184 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:15.703194 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] handle_message: 0xe20b500
2014-06-12 14:47:15.703201 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] do_op osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:15.703218 7f9d633d7700 15 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 '_'
2014-06-12 14:47:15.703362 7f9d633d7700 10 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 '_' = 0
2014-06-12 14:47:15.703375 7f9d633d7700 15 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 'snapset'
2014-06-12 14:47:15.703432 7f9d633d7700 10 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 'snapset' = 0
2014-06-12 14:47:15.703440 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] populate_obc_watchers 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.703448 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.703454 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] get_object_context: creating obc from disk: 0x37e4a580
2014-06-12 14:47:15.703459 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] get_object_context: 0x37e4a580 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 rwstate(none n=0 w=0) oi: 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'2741 client.4305.0:142594 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2741) ssc: 0x24d4b500 snapset: 0=[]:[]+head
2014-06-12 14:47:15.703469 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] find_object_context 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 @head oi=35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'2741 client.4305.0:142594 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2741)
2014-06-12 14:47:15.703485 7f9d633d7700  5 -- op tracker -- , seq: 748373, time: 2014-06-12 14:47:15.703482, event: started, op: osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.703491 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] execute_ctx 0x55a4400
2014-06-12 14:47:15.703500 7f9d633d7700  5 -- op tracker -- , seq: 748373, time: 2014-06-12 14:47:15.703498, event: started, op: osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.703504 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] do_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] ov 173'2741 av 173'2742 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:15.703512 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] do_osd_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192]
2014-06-12 14:47:15.703519 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:15.703527 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] do_osd_op  write 618496~1032192
2014-06-12 14:47:15.703537 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:15.703543 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 snapset=0x24d4b538  snapc=0=[]
2014-06-12 14:47:15.703550 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:15.703556 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] finish_ctx 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 0x55a4400 op modify  
2014-06-12 14:47:15.703563 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean]  set mtime to 2014-06-12 08:46:18.078661
2014-06-12 14:47:15.703580 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean]  final snapset 0=[]:[]+head in 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.703592 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean]  zeroing write result code 0
2014-06-12 14:47:15.703598 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] new_repop rep_tid 236417 on osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.703608 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] issue_repop rep_tid 236417 o 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.703622 7f9d633d7700  5 -- op tracker -- , seq: 748373, time: 2014-06-12 14:47:15.703620, event: waiting for subops from 21, op: osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.703636 7f9d633d7700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:15.703650 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2741 (0'0,173'2741] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] append_log log((0'0,173'2741], crt=0'0) [173'2742 (173'2741) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142595 2014-06-12 08:46:18.078661]
2014-06-12 14:47:15.703670 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] add_log_entry 173'2742 (173'2741) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142595 2014-06-12 08:46:18.078661
2014-06-12 14:47:15.703682 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] append_log  adding 1 keys
2014-06-12 14:47:15.703705 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2742, trimmed: 
2014-06-12 14:47:15.703722 7f9d633d7700  5 queue_transactions existing osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.703725 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:15.703727 7f9d633d7700  5 queue_op 0x7ac8660 seq 2543 osr(3.4d4 0x276d810) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:15.703734 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478fc0 (1 active)
2014-06-12 14:47:15.703741 7f9d7895e700  5 _do_op 0x7ac8660 seq 2543 osr(3.4d4 0x276d810)/0x276d810 start
2014-06-12 14:47:15.703745 7f9d7895e700 10 _do_transaction on 0x13b42480
2014-06-12 14:47:15.703752 7f9d7895e700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.703731 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] eval_repop repgather(0x13b43d40 173'2742 rep_tid=236417 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.703775 7f9d633d7700 10 osd.2 173 dequeue_op 0xe20b500 finish
2014-06-12 14:47:15.703778 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.703779 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.703840 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.703867 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:15.703915 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:15.703936 7f9d7895e700 15 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.703941 7f9d7895e700 10 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1 r = 0
2014-06-12 14:47:15.703946 7f9d7895e700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.704057 7f9d7895e700 15 _write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 618496~1032192
2014-06-12 14:47:15.704148 7f9d7895e700 10 lookup_strip_header done  cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.704280 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:47:15.704722 7f9d4a5d9700  5 -- op tracker -- , seq: 748374, time: 2014-06-12 14:47:15.703639, event: header_read, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.704735 7f9d4a5d9700  5 -- op tracker -- , seq: 748374, time: 2014-06-12 14:47:15.703641, event: throttled, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.704745 7f9d4a5d9700  5 -- op tracker -- , seq: 748374, time: 2014-06-12 14:47:15.704695, event: all_read, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.704756 7f9d4a5d9700  5 -- op tracker -- , seq: 748374, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.704761 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:15.704777 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x6d10e00 prio 63 cost 516096 latency 0.001138 osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.704814 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.704821 7f9d67be0700 10 osd.2 173 dequeue_op 0x6d10e00 prio 63 cost 516096 latency 0.001182 osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean]
2014-06-12 14:47:15.704839 7f9d67be0700  5 -- op tracker -- , seq: 748374, time: 2014-06-12 14:47:15.704835, event: reached_pg, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.704845 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:15.704853 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] handle_message: 0x6d10e00
2014-06-12 14:47:15.704859 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] do_op osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:15.704875 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] get_object_context: found obc in cache: 0x37e4a580
2014-06-12 14:47:15.704881 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] get_object_context: 0x37e4a580 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 rwstate(write n=1 w=0) oi: 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'2742 client.4305.0:142595 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2742) ssc: 0x24d4b500 snapset: 0=[]:[]+head
2014-06-12 14:47:15.704892 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] find_object_context 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 @head oi=35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'2742 client.4305.0:142595 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2742)
2014-06-12 14:47:15.704906 7f9d67be0700  5 -- op tracker -- , seq: 748374, time: 2014-06-12 14:47:15.704903, event: started, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.704913 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] execute_ctx 0x26f9f000
2014-06-12 14:47:15.704923 7f9d67be0700  5 -- op tracker -- , seq: 748374, time: 2014-06-12 14:47:15.704920, event: started, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.704967 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] do_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] ov 173'2742 av 173'2743 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:15.704976 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] do_osd_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096]
2014-06-12 14:47:15.704983 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:15.704991 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] do_osd_op  write 1650688~516096
2014-06-12 14:47:15.705002 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:15.705008 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 snapset=0x24d4b538  snapc=0=[]
2014-06-12 14:47:15.705014 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:15.705020 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] finish_ctx 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 0x26f9f000 op modify  
2014-06-12 14:47:15.705027 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean]  set mtime to 2014-06-12 08:46:18.082748
2014-06-12 14:47:15.705039 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean]  final snapset 0=[]:[]+head in 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.705049 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean]  zeroing write result code 0
2014-06-12 14:47:15.705055 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] new_repop rep_tid 236418 on osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.705078 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] issue_repop rep_tid 236418 o 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.705093 7f9d67be0700  5 -- op tracker -- , seq: 748374, time: 2014-06-12 14:47:15.705090, event: waiting for subops from 21, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.705109 7f9d67be0700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:15.705121 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2742 (0'0,173'2742] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] append_log log((0'0,173'2742], crt=0'0) [173'2743 (173'2742) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142596 2014-06-12 08:46:18.082748]
2014-06-12 14:47:15.705148 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] add_log_entry 173'2743 (173'2742) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142596 2014-06-12 08:46:18.082748
2014-06-12 14:47:15.705162 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] append_log  adding 1 keys
2014-06-12 14:47:15.705182 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2743, trimmed: 
2014-06-12 14:47:15.705196 7f9d67be0700  5 queue_transactions existing osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.705198 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:15.705199 7f9d67be0700  5 queue_op 0x1f60960 seq 2544 osr(3.4d4 0x276d810) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:47:15.705209 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478fc0 (2 active)
2014-06-12 14:47:15.705204 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] eval_repop repgather(0x5214d80 173'2743 rep_tid=236418 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.705221 7f9d67be0700 10 osd.2 173 dequeue_op 0x6d10e00 finish
2014-06-12 14:47:15.705224 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.705226 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.707181 7f9d7895e700 10 _generic_write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 618496~1032192 = 0
2014-06-12 14:47:15.707250 7f9d7895e700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.707275 7f9d7895e700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.707280 7f9d7895e700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.707285 7f9d7895e700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.707485 7f9d7895e700 20 set_header setting 16502 cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 parent seq 0
2014-06-12 14:47:15.707553 7f9d7895e700 20 set_header setting 2082 cid meta oid 1fe07345/pglog_3.4d4/0//-1 parent seq 0
2014-06-12 14:47:15.707564 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:15.707569 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:47:15.711123 7f9d7895e700 10 _do_op 0x7ac8660 seq 2543 r = 0, finisher 0x698d180 0x3dbb480
2014-06-12 14:47:15.711143 7f9d7895e700 10 _finish_op 0x7ac8660 seq 2543 osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.711152 7f9d83173700  5 _do_op 0x1f60960 seq 2544 osr(3.4d4 0x276d810)/0x276d810 start
2014-06-12 14:47:15.711153 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478fc0 (1 active)
2014-06-12 14:47:15.711164 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:15.711165 7f9d83173700 10 _do_transaction on 0x52152c0
2014-06-12 14:47:15.711180 7f9d83173700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.711193 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] op_commit: 236417
2014-06-12 14:47:15.711232 7f9d7715b700  5 -- op tracker -- , seq: 748373, time: 2014-06-12 14:47:15.711223, event: op_commit, op: osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.711296 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.711292 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] op_applied: 236417
2014-06-12 14:47:15.711313 7f9d7795c700  5 -- op tracker -- , seq: 748373, time: 2014-06-12 14:47:15.711308, event: op_applied, op: osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.711320 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:15.711323 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] op_applied on primary on version 173'2742
2014-06-12 14:47:15.711351 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:15.711435 7f9d83173700 15 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.711441 7f9d83173700 10 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1 r = 0
2014-06-12 14:47:15.711445 7f9d83173700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.711456 7f9d83173700 15 _write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 1650688~516096
2014-06-12 14:47:15.711545 7f9d83173700 10 lookup_strip_header done  cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.711648 7f9d83173700 10 file_to_extents done 
2014-06-12 14:47:15.712510 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:15.712512 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:15.712513 7f9d74bfa700 20 osd.2 173 _dispatch 0x1e701520 ping v1
2014-06-12 14:47:15.712515 7f9d74bfa700 10 osd.2 173 ping from client.4300
2014-06-12 14:47:15.712516 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:15.712517 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:15.712780 7f9d83173700 10 _generic_write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 1650688~516096 = 0
2014-06-12 14:47:15.712818 7f9d83173700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.712827 7f9d83173700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.712831 7f9d83173700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.712835 7f9d83173700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.712959 7f9d83173700 20 set_header setting 16502 cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 parent seq 0
2014-06-12 14:47:15.713015 7f9d83173700 20 set_header setting 2082 cid meta oid 1fe07345/pglog_3.4d4/0//-1 parent seq 0
2014-06-12 14:47:15.713026 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:15.713031 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:47:15.713498 7f9d44101700  5 -- op tracker -- , seq: 748375, time: 2014-06-12 14:47:15.713456, event: header_read, op: osd_sub_op_reply(client.4305.0:142595 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.713508 7f9d44101700  5 -- op tracker -- , seq: 748375, time: 2014-06-12 14:47:15.713457, event: throttled, op: osd_sub_op_reply(client.4305.0:142595 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.713521 7f9d44101700  5 -- op tracker -- , seq: 748375, time: 2014-06-12 14:47:15.713471, event: all_read, op: osd_sub_op_reply(client.4305.0:142595 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.713526 7f9d44101700  5 -- op tracker -- , seq: 748375, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142595 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.713530 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142595 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:15.713534 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:47:15.713541 7f9d44101700 15 osd.2 173 enqueue_op 0x2d17bd00 prio 196 cost 0 latency 0.000085 osd_sub_op_reply(client.4305.0:142595 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:15.713568 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.713576 7f9d673df700 10 osd.2 173 dequeue_op 0x2d17bd00 prio 196 cost 0 latency 0.000120 osd_sub_op_reply(client.4305.0:142595 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 pg pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean]
2014-06-12 14:47:15.713592 7f9d673df700  5 -- op tracker -- , seq: 748375, time: 2014-06-12 14:47:15.713590, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142595 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.713598 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] handle_message: 0x2d17bd00
2014-06-12 14:47:15.713609 7f9d673df700  5 -- op tracker -- , seq: 748375, time: 2014-06-12 14:47:15.713607, event: started, op: osd_sub_op_reply(client.4305.0:142595 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.713628 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] sub_op_modify_reply: tid 236417 op  ack_type 4 from 21
2014-06-12 14:47:15.713646 7f9d673df700  5 -- op tracker -- , seq: 748373, time: 2014-06-12 14:47:15.713640, event: sub_op_commit_rec, op: osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.713651 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] repop_all_applied: repop tid 236417 all applied 
2014-06-12 14:47:15.713659 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] eval_repop repgather(0x13b43d40 173'2742 rep_tid=236417 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.713671 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean]  sending ack on repgather(0x13b43d40 173'2742 rep_tid=236417 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0x1ba79680
2014-06-12 14:47:15.713693 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2741 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] repop_all_committed: repop tid 236417 all committed 
2014-06-12 14:47:15.713704 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2740 active+clean] eval_repop repgather(0x13b43d40 173'2742 rep_tid=236417 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.713722 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2740 active+clean] log_op_stats osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.013182 lat 0.013212
2014-06-12 14:47:15.713739 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2740 active+clean] publish_stats_to_osd 173:3540
2014-06-12 14:47:15.713748 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2740 active+clean]  sending commit on repgather(0x13b43d40 173'2742 rep_tid=236417 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0xee34c80
2014-06-12 14:47:15.713771 7f9d673df700  5 -- op tracker -- , seq: 748373, time: 2014-06-12 14:47:15.713766, event: commit_sent, op: osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.713780 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2740 active+clean]  requeue_ops 
2014-06-12 14:47:15.713791 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean]  removing repgather(0x13b43d40 173'2742 rep_tid=236417 committed?=1 applied?=1 op=osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.713801 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean]    q front is repgather(0x13b43d40 173'2742 rep_tid=236417 committed?=1 applied?=1 op=osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.713814 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean] remove_repop repgather(0x13b43d40 173'2742 rep_tid=236417 committed?=1 applied?=1 op=osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.713826 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean]  requeue_ops 
2014-06-12 14:47:15.713839 7f9d673df700  5 -- op tracker -- , seq: 748373, time: 2014-06-12 14:47:15.713836, event: done, op: osd_op(client.4305.0:142595 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 618496~1032192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.713854 7f9d673df700 10 osd.2 173 dequeue_op 0x2d17bd00 finish
2014-06-12 14:47:15.713860 7f9d673df700  5 -- op tracker -- , seq: 748375, time: 2014-06-12 14:47:15.713858, event: done, op: osd_sub_op_reply(client.4305.0:142595 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.713866 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.713869 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.714639 7f9d83173700 10 _do_op 0x1f60960 seq 2544 r = 0, finisher 0x5748660 0x19b2c080
2014-06-12 14:47:15.714648 7f9d83173700 10 _finish_op 0x1f60960 seq 2544 osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.714656 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478fc0 (0 active)
2014-06-12 14:47:15.714663 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:15.714663 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean] op_applied: 236418
2014-06-12 14:47:15.714688 7f9d7795c700  5 -- op tracker -- , seq: 748374, time: 2014-06-12 14:47:15.714681, event: op_applied, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.714699 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean] op_applied on primary on version 173'2743
2014-06-12 14:47:15.714731 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean] op_commit: 236418
2014-06-12 14:47:15.714751 7f9d7715b700  5 -- op tracker -- , seq: 748374, time: 2014-06-12 14:47:15.714746, event: op_commit, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.716872 7f9d44101700  5 -- op tracker -- , seq: 748376, time: 2014-06-12 14:47:15.716836, event: header_read, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ack, result = 0)
2014-06-12 14:47:15.716883 7f9d44101700  5 -- op tracker -- , seq: 748376, time: 2014-06-12 14:47:15.716837, event: throttled, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ack, result = 0)
2014-06-12 14:47:15.716889 7f9d44101700  5 -- op tracker -- , seq: 748376, time: 2014-06-12 14:47:15.716846, event: all_read, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ack, result = 0)
2014-06-12 14:47:15.716894 7f9d44101700  5 -- op tracker -- , seq: 748376, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ack, result = 0)
2014-06-12 14:47:15.716900 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:15.716904 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:47:15.716908 7f9d44101700 15 osd.2 173 enqueue_op 0x22b3b200 prio 196 cost 0 latency 0.000072 osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ack, result = 0) v2
2014-06-12 14:47:15.716944 7f9d44101700  5 -- op tracker -- , seq: 748377, time: 2014-06-12 14:47:15.716920, event: header_read, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.716950 7f9d44101700  5 -- op tracker -- , seq: 748377, time: 2014-06-12 14:47:15.716921, event: throttled, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.716955 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.716955 7f9d44101700  5 -- op tracker -- , seq: 748377, time: 2014-06-12 14:47:15.716929, event: all_read, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.716961 7f9d44101700  5 -- op tracker -- , seq: 748377, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.716963 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:15.716966 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:47:15.716969 7f9d44101700 15 osd.2 173 enqueue_op 0x28fd9800 prio 196 cost 0 latency 0.000049 osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:15.716962 7f9d663dd700 10 osd.2 173 dequeue_op 0x22b3b200 prio 196 cost 0 latency 0.000125 osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ack, result = 0) v2 pg pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean]
2014-06-12 14:47:15.716977 7f9d663dd700  5 -- op tracker -- , seq: 748376, time: 2014-06-12 14:47:15.716975, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ack, result = 0)
2014-06-12 14:47:15.716986 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:15.716988 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean] handle_message: 0x22b3b200
2014-06-12 14:47:15.717016 7f9d663dd700  5 -- op tracker -- , seq: 748376, time: 2014-06-12 14:47:15.717014, event: started, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ack, result = 0)
2014-06-12 14:47:15.717030 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean] sub_op_modify_reply: tid 236418 op  ack_type 1 from 21
2014-06-12 14:47:15.717043 7f9d663dd700  5 -- op tracker -- , seq: 748374, time: 2014-06-12 14:47:15.717038, event: sub_op_applied_rec, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.717049 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean] repop_all_applied: repop tid 236418 all applied 
2014-06-12 14:47:15.717057 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean] eval_repop repgather(0x5214d80 173'2743 rep_tid=236418 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.717068 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean]  sending ack on repgather(0x5214d80 173'2743 rep_tid=236418 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0x1a669180
2014-06-12 14:47:15.717088 7f9d663dd700 10 osd.2 173 dequeue_op 0x22b3b200 finish
2014-06-12 14:47:15.717093 7f9d663dd700  5 -- op tracker -- , seq: 748376, time: 2014-06-12 14:47:15.717091, event: done, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ack, result = 0)
2014-06-12 14:47:15.717102 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:15.717105 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.717107 7f9d65bdc700 10 osd.2 173 dequeue_op 0x28fd9800 prio 196 cost 0 latency 0.000187 osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 pg pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean]
2014-06-12 14:47:15.717132 7f9d65bdc700  5 -- op tracker -- , seq: 748377, time: 2014-06-12 14:47:15.717128, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.717144 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean] handle_message: 0x28fd9800
2014-06-12 14:47:15.717161 7f9d65bdc700  5 -- op tracker -- , seq: 748377, time: 2014-06-12 14:47:15.717157, event: started, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.717168 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean] sub_op_modify_reply: tid 236418 op  ack_type 4 from 21
2014-06-12 14:47:15.717182 7f9d65bdc700  5 -- op tracker -- , seq: 748374, time: 2014-06-12 14:47:15.717177, event: sub_op_commit_rec, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.717187 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2742 crt=0'0 lcod 173'2741 mlcod 173'2741 active+clean] repop_all_committed: repop tid 236418 all committed 
2014-06-12 14:47:15.717194 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2741 active+clean] eval_repop repgather(0x5214d80 173'2743 rep_tid=236418 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.717205 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2741 active+clean] log_op_stats osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.013448 lat 0.013565
2014-06-12 14:47:15.717216 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2741 active+clean] publish_stats_to_osd 173:3541
2014-06-12 14:47:15.717224 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2741 active+clean]  sending commit on repgather(0x5214d80 173'2743 rep_tid=236418 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0x30af6c80
2014-06-12 14:47:15.717248 7f9d65bdc700  5 -- op tracker -- , seq: 748374, time: 2014-06-12 14:47:15.717242, event: commit_sent, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.717257 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2741 active+clean]  requeue_ops 
2014-06-12 14:47:15.717269 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean]  removing repgather(0x5214d80 173'2743 rep_tid=236418 committed?=1 applied?=1 op=osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.717281 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean]    q front is repgather(0x5214d80 173'2743 rep_tid=236418 committed?=1 applied?=1 op=osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.717290 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] remove_repop repgather(0x5214d80 173'2743 rep_tid=236418 committed?=1 applied?=1 op=osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.717300 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean]  requeue_ops 
2014-06-12 14:47:15.717317 7f9d65bdc700  5 -- op tracker -- , seq: 748374, time: 2014-06-12 14:47:15.717314, event: done, op: osd_op(client.4305.0:142596 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1650688~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.717329 7f9d65bdc700 10 osd.2 173 dequeue_op 0x28fd9800 finish
2014-06-12 14:47:15.717335 7f9d65bdc700  5 -- op tracker -- , seq: 748377, time: 2014-06-12 14:47:15.717333, event: done, op: osd_sub_op_reply(client.4305.0:142596 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.717341 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.717343 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.717490 7f9d4a5d9700  5 -- op tracker -- , seq: 748378, time: 2014-06-12 14:47:15.716407, event: header_read, op: osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.717499 7f9d4a5d9700  5 -- op tracker -- , seq: 748378, time: 2014-06-12 14:47:15.716409, event: throttled, op: osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.717505 7f9d4a5d9700  5 -- op tracker -- , seq: 748378, time: 2014-06-12 14:47:15.717470, event: all_read, op: osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.717510 7f9d4a5d9700  5 -- op tracker -- , seq: 748378, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.717515 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:15.717534 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x3850e500 prio 63 cost 520192 latency 0.001127 osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.717555 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.717563 7f9d62bd6700 10 osd.2 173 dequeue_op 0x3850e500 prio 63 cost 520192 latency 0.001156 osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean]
2014-06-12 14:47:15.717597 7f9d62bd6700  5 -- op tracker -- , seq: 748378, time: 2014-06-12 14:47:15.717592, event: reached_pg, op: osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.717608 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:15.717628 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] handle_message: 0x3850e500
2014-06-12 14:47:15.717640 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] do_op osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:15.717666 7f9d62bd6700 15 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 '_'
2014-06-12 14:47:15.717795 7f9d62bd6700 10 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 '_' = 0
2014-06-12 14:47:15.717819 7f9d62bd6700 15 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 'snapset'
2014-06-12 14:47:15.717885 7f9d62bd6700 10 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 'snapset' = 0
2014-06-12 14:47:15.717895 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] populate_obc_watchers 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.717909 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.717920 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] get_object_context: creating obc from disk: 0xc2de840
2014-06-12 14:47:15.717931 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] get_object_context: 0xc2de840 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 rwstate(none n=0 w=0) oi: 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'2743 client.4305.0:142596 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2743) ssc: 0x1550f0a0 snapset: 0=[]:[]+head
2014-06-12 14:47:15.717948 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] find_object_context 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 @head oi=35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'2743 client.4305.0:142596 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2743)
2014-06-12 14:47:15.717978 7f9d62bd6700  5 -- op tracker -- , seq: 748378, time: 2014-06-12 14:47:15.717971, event: started, op: osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.717988 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] execute_ctx 0x33b2ea00
2014-06-12 14:47:15.718006 7f9d62bd6700  5 -- op tracker -- , seq: 748378, time: 2014-06-12 14:47:15.718001, event: started, op: osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.718015 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] do_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] ov 173'2743 av 173'2744 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:15.718031 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] do_osd_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192]
2014-06-12 14:47:15.718044 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:15.718058 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] do_osd_op  write 2166784~520192
2014-06-12 14:47:15.718076 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:15.718089 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 snapset=0x1550f0d8  snapc=0=[]
2014-06-12 14:47:15.718104 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:15.718121 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] finish_ctx 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 0x33b2ea00 op modify  
2014-06-12 14:47:15.718135 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean]  set mtime to 2014-06-12 08:46:18.095414
2014-06-12 14:47:15.718157 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean]  final snapset 0=[]:[]+head in 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.718175 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean]  zeroing write result code 0
2014-06-12 14:47:15.718189 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] new_repop rep_tid 236419 on osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.718207 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] issue_repop rep_tid 236419 o 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.718234 7f9d62bd6700  5 -- op tracker -- , seq: 748378, time: 2014-06-12 14:47:15.718229, event: waiting for subops from 21, op: osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.718256 7f9d62bd6700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:15.718277 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2743 (0'0,173'2743] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] append_log log((0'0,173'2743], crt=0'0) [173'2744 (173'2743) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142597 2014-06-12 08:46:18.095414]
2014-06-12 14:47:15.718302 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] add_log_entry 173'2744 (173'2743) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142597 2014-06-12 08:46:18.095414
2014-06-12 14:47:15.718321 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] append_log  adding 1 keys
2014-06-12 14:47:15.718349 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2744, trimmed: 
2014-06-12 14:47:15.718368 7f9d62bd6700  5 queue_transactions existing osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.718372 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:15.718374 7f9d62bd6700  5 queue_op 0x4b2ed20 seq 2545 osr(3.4d4 0x276d810) 522485 bytes   (queue has 1 ops and 522485 bytes)
2014-06-12 14:47:15.718384 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478fc0 (1 active)
2014-06-12 14:47:15.718391 7f9d7a161700  5 _do_op 0x4b2ed20 seq 2545 osr(3.4d4 0x276d810)/0x276d810 start
2014-06-12 14:47:15.718380 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] eval_repop repgather(0x1ced2b40 173'2744 rep_tid=236419 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.718395 7f9d7a161700 10 _do_transaction on 0x5329440
2014-06-12 14:47:15.718396 7f9d62bd6700 10 osd.2 173 dequeue_op 0x3850e500 finish
2014-06-12 14:47:15.718399 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.718402 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.718401 7f9d7a161700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.718564 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.718590 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:15.718639 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:15.718662 7f9d7a161700 15 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.718668 7f9d7a161700 10 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1 r = 0
2014-06-12 14:47:15.718674 7f9d7a161700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.718691 7f9d7a161700 15 _write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 2166784~520192
2014-06-12 14:47:15.718824 7f9d7a161700 10 lookup_strip_header done  cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.718990 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:47:15.720573 7f9d4a5d9700  5 -- op tracker -- , seq: 748379, time: 2014-06-12 14:47:15.719429, event: header_read, op: osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.720588 7f9d4a5d9700  5 -- op tracker -- , seq: 748379, time: 2014-06-12 14:47:15.719431, event: throttled, op: osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.720598 7f9d4a5d9700  5 -- op tracker -- , seq: 748379, time: 2014-06-12 14:47:15.720544, event: all_read, op: osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.720617 7f9d4a5d9700  5 -- op tracker -- , seq: 748379, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.720624 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:15.720643 7f9d7a161700 10 _generic_write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 2166784~520192 = 0
2014-06-12 14:47:15.720644 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x3850e200 prio 63 cost 516096 latency 0.001215 osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.720669 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.720676 7f9d683e1700 10 osd.2 173 dequeue_op 0x3850e200 prio 63 cost 516096 latency 0.001247 osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean]
2014-06-12 14:47:15.720688 7f9d7a161700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.720695 7f9d683e1700  5 -- op tracker -- , seq: 748379, time: 2014-06-12 14:47:15.720691, event: reached_pg, op: osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.720701 7f9d7a161700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.720701 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:15.720708 7f9d7a161700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.720710 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] handle_message: 0x3850e200
2014-06-12 14:47:15.720716 7f9d7a161700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.720716 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] do_op osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:15.720731 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] get_object_context: found obc in cache: 0xc2de840
2014-06-12 14:47:15.720737 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] get_object_context: 0xc2de840 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 rwstate(write n=1 w=0) oi: 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'2744 client.4305.0:142597 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2744) ssc: 0x1550f0a0 snapset: 0=[]:[]+head
2014-06-12 14:47:15.720746 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] find_object_context 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 @head oi=35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'2744 client.4305.0:142597 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2744)
2014-06-12 14:47:15.720760 7f9d683e1700  5 -- op tracker -- , seq: 748379, time: 2014-06-12 14:47:15.720757, event: started, op: osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.720764 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] execute_ctx 0x292d600
2014-06-12 14:47:15.720774 7f9d683e1700  5 -- op tracker -- , seq: 748379, time: 2014-06-12 14:47:15.720771, event: started, op: osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.720778 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] do_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] ov 173'2744 av 173'2745 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:15.720786 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] do_osd_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096]
2014-06-12 14:47:15.720793 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:15.720801 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] do_osd_op  write 2686976~516096
2014-06-12 14:47:15.720809 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:15.720816 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 snapset=0x1550f0d8  snapc=0=[]
2014-06-12 14:47:15.720823 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:15.720830 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] finish_ctx 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 0x292d600 op modify  
2014-06-12 14:47:15.720837 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean]  set mtime to 2014-06-12 08:46:18.098448
2014-06-12 14:47:15.720850 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean]  final snapset 0=[]:[]+head in 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.720860 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean]  zeroing write result code 0
2014-06-12 14:47:15.720868 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] new_repop rep_tid 236420 on osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.720880 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] issue_repop rep_tid 236420 o 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.720895 7f9d683e1700  5 -- op tracker -- , seq: 748379, time: 2014-06-12 14:47:15.720892, event: waiting for subops from 21, op: osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.720909 7f9d683e1700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:15.720921 7f9d7a161700 20 set_header setting 16502 cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 parent seq 0
2014-06-12 14:47:15.720923 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2744 (0'0,173'2744] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] append_log log((0'0,173'2744], crt=0'0) [173'2745 (173'2744) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142598 2014-06-12 08:46:18.098448]
2014-06-12 14:47:15.720939 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] add_log_entry 173'2745 (173'2744) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142598 2014-06-12 08:46:18.098448
2014-06-12 14:47:15.720955 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] append_log  adding 1 keys
2014-06-12 14:47:15.720974 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2745, trimmed: 
2014-06-12 14:47:15.720987 7f9d683e1700  5 queue_transactions existing osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.720991 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:15.720991 7f9d683e1700  5 queue_op 0x774ab40 seq 2546 osr(3.4d4 0x276d810) 518389 bytes   (queue has 2 ops and 1040874 bytes)
2014-06-12 14:47:15.720991 7f9d7a161700 20 set_header setting 2082 cid meta oid 1fe07345/pglog_3.4d4/0//-1 parent seq 0
2014-06-12 14:47:15.720997 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] eval_repop repgather(0x2010fec0 173'2745 rep_tid=236420 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.721005 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:15.721008 7f9d683e1700 10 osd.2 173 dequeue_op 0x3850e200 finish
2014-06-12 14:47:15.721011 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.721012 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.721014 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:47:15.721029 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478fc0 (2 active)
2014-06-12 14:47:15.723499 7f9d7a161700 10 _do_op 0x4b2ed20 seq 2545 r = 0, finisher 0x89f5400 0x1e626040
2014-06-12 14:47:15.723513 7f9d7a161700 10 _finish_op 0x4b2ed20 seq 2545 osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.723522 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478fc0 (1 active)
2014-06-12 14:47:15.723518 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] op_commit: 236419
2014-06-12 14:47:15.723535 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:15.723542 7f9d7715b700  5 -- op tracker -- , seq: 748378, time: 2014-06-12 14:47:15.723537, event: op_commit, op: osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.723545 7f9d7f16b700  5 _do_op 0x774ab40 seq 2546 osr(3.4d4 0x276d810)/0x276d810 start
2014-06-12 14:47:15.723552 7f9d7f16b700 10 _do_transaction on 0x2010efc0
2014-06-12 14:47:15.723559 7f9d7f16b700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.723549 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] op_applied: 236419
2014-06-12 14:47:15.723573 7f9d7795c700  5 -- op tracker -- , seq: 748378, time: 2014-06-12 14:47:15.723566, event: op_applied, op: osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.723589 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] op_applied on primary on version 173'2744
2014-06-12 14:47:15.723721 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.723743 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:15.723780 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:15.723795 7f9d7f16b700 15 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.723800 7f9d7f16b700 10 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1 r = 0
2014-06-12 14:47:15.723804 7f9d7f16b700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.723815 7f9d7f16b700 15 _write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 2686976~516096
2014-06-12 14:47:15.724016 7f9d7f16b700 10 lookup_strip_header done  cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.724109 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:47:15.724921 7f9d44101700  5 -- op tracker -- , seq: 748380, time: 2014-06-12 14:47:15.724878, event: header_read, op: osd_sub_op_reply(client.4305.0:142597 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.724935 7f9d44101700  5 -- op tracker -- , seq: 748380, time: 2014-06-12 14:47:15.724880, event: throttled, op: osd_sub_op_reply(client.4305.0:142597 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.724943 7f9d44101700  5 -- op tracker -- , seq: 748380, time: 2014-06-12 14:47:15.724889, event: all_read, op: osd_sub_op_reply(client.4305.0:142597 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.724951 7f9d44101700  5 -- op tracker -- , seq: 748380, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142597 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.724956 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142597 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:15.724963 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:47:15.724971 7f9d44101700 15 osd.2 173 enqueue_op 0x28fd8500 prio 196 cost 0 latency 0.000093 osd_sub_op_reply(client.4305.0:142597 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:15.724996 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.725001 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x28fd8500 prio 196 cost 0 latency 0.000123 osd_sub_op_reply(client.4305.0:142597 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 pg pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean]
2014-06-12 14:47:15.725017 7f9d6b3e7700  5 -- op tracker -- , seq: 748380, time: 2014-06-12 14:47:15.725014, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142597 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.725023 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] handle_message: 0x28fd8500
2014-06-12 14:47:15.725033 7f9d6b3e7700  5 -- op tracker -- , seq: 748380, time: 2014-06-12 14:47:15.725031, event: started, op: osd_sub_op_reply(client.4305.0:142597 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.725036 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] sub_op_modify_reply: tid 236419 op  ack_type 4 from 21
2014-06-12 14:47:15.725048 7f9d6b3e7700  5 -- op tracker -- , seq: 748378, time: 2014-06-12 14:47:15.725044, event: sub_op_commit_rec, op: osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.725053 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] repop_all_applied: repop tid 236419 all applied 
2014-06-12 14:47:15.725065 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] eval_repop repgather(0x1ced2b40 173'2744 rep_tid=236419 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.725086 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean]  sending ack on repgather(0x1ced2b40 173'2744 rep_tid=236419 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0x22b25180
2014-06-12 14:47:15.725115 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2743 crt=0'0 lcod 173'2742 mlcod 173'2742 active+clean] repop_all_committed: repop tid 236419 all committed 
2014-06-12 14:47:15.725125 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2742 active+clean] eval_repop repgather(0x1ced2b40 173'2744 rep_tid=236419 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.725144 7f9d7f16b700 10 _generic_write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 2686976~516096 = 0
2014-06-12 14:47:15.725140 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2742 active+clean] log_op_stats osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 inb 520471 outb 0 rlat 0.008707 lat 0.008732
2014-06-12 14:47:15.725156 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2742 active+clean] publish_stats_to_osd 173:3542
2014-06-12 14:47:15.725165 7f9d7f16b700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.725177 7f9d7f16b700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.725180 7f9d7f16b700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.725168 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2742 active+clean]  sending commit on repgather(0x1ced2b40 173'2744 rep_tid=236419 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0xae7e280
2014-06-12 14:47:15.725185 7f9d7f16b700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.725200 7f9d6b3e7700  5 -- op tracker -- , seq: 748378, time: 2014-06-12 14:47:15.725195, event: commit_sent, op: osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.725208 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2742 active+clean]  requeue_ops 
2014-06-12 14:47:15.725217 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean]  removing repgather(0x1ced2b40 173'2744 rep_tid=236419 committed?=1 applied?=1 op=osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.725230 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean]    q front is repgather(0x1ced2b40 173'2744 rep_tid=236419 committed?=1 applied?=1 op=osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.725244 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean] remove_repop repgather(0x1ced2b40 173'2744 rep_tid=236419 committed?=1 applied?=1 op=osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.725259 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean]  requeue_ops 
2014-06-12 14:47:15.725278 7f9d6b3e7700  5 -- op tracker -- , seq: 748378, time: 2014-06-12 14:47:15.725273, event: done, op: osd_op(client.4305.0:142597 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2166784~520192] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.725297 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x28fd8500 finish
2014-06-12 14:47:15.725299 7f9d7f16b700 20 set_header setting 16502 cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 parent seq 0
2014-06-12 14:47:15.725307 7f9d6b3e7700  5 -- op tracker -- , seq: 748380, time: 2014-06-12 14:47:15.725303, event: done, op: osd_sub_op_reply(client.4305.0:142597 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.725318 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.725320 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.725337 7f9d7f16b700 20 set_header setting 2082 cid meta oid 1fe07345/pglog_3.4d4/0//-1 parent seq 0
2014-06-12 14:47:15.725345 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:15.725350 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:47:15.726974 7f9d7f16b700 10 _do_op 0x774ab40 seq 2546 r = 0, finisher 0x4f8c200 0x198e4b80
2014-06-12 14:47:15.726987 7f9d7f16b700 10 _finish_op 0x774ab40 seq 2546 osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.726994 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478fc0 (0 active)
2014-06-12 14:47:15.726997 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:15.726991 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean] op_commit: 236420
2014-06-12 14:47:15.727010 7f9d7715b700  5 -- op tracker -- , seq: 748379, time: 2014-06-12 14:47:15.727005, event: op_commit, op: osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.727021 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean] op_applied: 236420
2014-06-12 14:47:15.727038 7f9d7795c700  5 -- op tracker -- , seq: 748379, time: 2014-06-12 14:47:15.727033, event: op_applied, op: osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.727045 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean] op_applied on primary on version 173'2745
2014-06-12 14:47:15.728899 7f9d44101700  5 -- op tracker -- , seq: 748381, time: 2014-06-12 14:47:15.728865, event: header_read, op: osd_sub_op_reply(client.4305.0:142598 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.728915 7f9d44101700  5 -- op tracker -- , seq: 748381, time: 2014-06-12 14:47:15.728867, event: throttled, op: osd_sub_op_reply(client.4305.0:142598 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.728923 7f9d44101700  5 -- op tracker -- , seq: 748381, time: 2014-06-12 14:47:15.728878, event: all_read, op: osd_sub_op_reply(client.4305.0:142598 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.728931 7f9d44101700  5 -- op tracker -- , seq: 748381, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142598 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.728936 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142598 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:15.728942 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:47:15.728949 7f9d44101700 15 osd.2 173 enqueue_op 0x20e86a00 prio 196 cost 0 latency 0.000084 osd_sub_op_reply(client.4305.0:142598 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:15.728974 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.728980 7f9d693e3700 10 osd.2 173 dequeue_op 0x20e86a00 prio 196 cost 0 latency 0.000115 osd_sub_op_reply(client.4305.0:142598 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 pg pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean]
2014-06-12 14:47:15.728995 7f9d693e3700  5 -- op tracker -- , seq: 748381, time: 2014-06-12 14:47:15.728992, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142598 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.729001 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean] handle_message: 0x20e86a00
2014-06-12 14:47:15.729012 7f9d693e3700  5 -- op tracker -- , seq: 748381, time: 2014-06-12 14:47:15.729010, event: started, op: osd_sub_op_reply(client.4305.0:142598 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.729015 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean] sub_op_modify_reply: tid 236420 op  ack_type 4 from 21
2014-06-12 14:47:15.729027 7f9d693e3700  5 -- op tracker -- , seq: 748379, time: 2014-06-12 14:47:15.729023, event: sub_op_commit_rec, op: osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.729032 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean] repop_all_applied: repop tid 236420 all applied 
2014-06-12 14:47:15.729039 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean] eval_repop repgather(0x2010fec0 173'2745 rep_tid=236420 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.729050 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean]  sending ack on repgather(0x2010fec0 173'2745 rep_tid=236420 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0x17be8500
2014-06-12 14:47:15.729071 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2744 crt=0'0 lcod 173'2743 mlcod 173'2743 active+clean] repop_all_committed: repop tid 236420 all committed 
2014-06-12 14:47:15.729078 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2743 active+clean] eval_repop repgather(0x2010fec0 173'2745 rep_tid=236420 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.729088 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2743 active+clean] log_op_stats osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.009641 lat 0.009658
2014-06-12 14:47:15.729099 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2743 active+clean] publish_stats_to_osd 173:3543
2014-06-12 14:47:15.729108 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2743 active+clean]  sending commit on repgather(0x2010fec0 173'2745 rep_tid=236420 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0x28677900
2014-06-12 14:47:15.729135 7f9d693e3700  5 -- op tracker -- , seq: 748379, time: 2014-06-12 14:47:15.729131, event: commit_sent, op: osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.729142 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2743 active+clean]  requeue_ops 
2014-06-12 14:47:15.729149 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean]  removing repgather(0x2010fec0 173'2745 rep_tid=236420 committed?=1 applied?=1 op=osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.729157 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean]    q front is repgather(0x2010fec0 173'2745 rep_tid=236420 committed?=1 applied?=1 op=osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.729166 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] remove_repop repgather(0x2010fec0 173'2745 rep_tid=236420 committed?=1 applied?=1 op=osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.729174 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean]  requeue_ops 
2014-06-12 14:47:15.729191 7f9d693e3700  5 -- op tracker -- , seq: 748379, time: 2014-06-12 14:47:15.729188, event: done, op: osd_op(client.4305.0:142598 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2686976~516096] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.729205 7f9d693e3700 10 osd.2 173 dequeue_op 0x20e86a00 finish
2014-06-12 14:47:15.729211 7f9d693e3700  5 -- op tracker -- , seq: 748381, time: 2014-06-12 14:47:15.729208, event: done, op: osd_sub_op_reply(client.4305.0:142598 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.729217 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.729218 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.730889 7f9d4a5d9700  5 -- op tracker -- , seq: 748382, time: 2014-06-12 14:47:15.728814, event: header_read, op: osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.730897 7f9d4a5d9700  5 -- op tracker -- , seq: 748382, time: 2014-06-12 14:47:15.728816, event: throttled, op: osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.730903 7f9d4a5d9700  5 -- op tracker -- , seq: 748382, time: 2014-06-12 14:47:15.730868, event: all_read, op: osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.730908 7f9d4a5d9700  5 -- op tracker -- , seq: 748382, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.730913 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:15.730932 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x3850f100 prio 63 cost 991232 latency 0.002118 osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.730953 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.730960 7f9d6abe6700 10 osd.2 173 dequeue_op 0x3850f100 prio 63 cost 991232 latency 0.002146 osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean]
2014-06-12 14:47:15.730977 7f9d6abe6700  5 -- op tracker -- , seq: 748382, time: 2014-06-12 14:47:15.730973, event: reached_pg, op: osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.730984 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:15.730993 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] handle_message: 0x3850f100
2014-06-12 14:47:15.731000 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] do_op osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:15.731017 7f9d6abe6700 15 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 '_'
2014-06-12 14:47:15.731156 7f9d6abe6700 10 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 '_' = 0
2014-06-12 14:47:15.731169 7f9d6abe6700 15 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 'snapset'
2014-06-12 14:47:15.731226 7f9d6abe6700 10 getattr 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 'snapset' = 0
2014-06-12 14:47:15.731234 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] populate_obc_watchers 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.731243 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.731249 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] get_object_context: creating obc from disk: 0x2060edc0
2014-06-12 14:47:15.731254 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] get_object_context: 0x2060edc0 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 rwstate(none n=0 w=0) oi: 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'2745 client.4305.0:142598 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2745) ssc: 0x5bb8700 snapset: 0=[]:[]+head
2014-06-12 14:47:15.731263 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] find_object_context 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 @head oi=35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3(173'2745 client.4305.0:142598 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2745)
2014-06-12 14:47:15.731281 7f9d6abe6700  5 -- op tracker -- , seq: 748382, time: 2014-06-12 14:47:15.731277, event: started, op: osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.731287 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] execute_ctx 0x55a3e00
2014-06-12 14:47:15.731297 7f9d6abe6700  5 -- op tracker -- , seq: 748382, time: 2014-06-12 14:47:15.731294, event: started, op: osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.731302 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] do_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] ov 173'2745 av 173'2746 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:15.731312 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] do_osd_op 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232]
2014-06-12 14:47:15.731319 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:15.731327 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] do_osd_op  write 3203072~991232
2014-06-12 14:47:15.731338 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:15.731346 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 snapset=0x5bb8738  snapc=0=[]
2014-06-12 14:47:15.731353 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] make_writeable 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:15.731359 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] finish_ctx 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 0x55a3e00 op modify  
2014-06-12 14:47:15.731367 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean]  set mtime to 2014-06-12 08:46:18.106994
2014-06-12 14:47:15.731383 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean]  final snapset 0=[]:[]+head in 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.731396 7f9d6abe6700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean]  zeroing write result code 0
2014-06-12 14:47:15.731403 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] new_repop rep_tid 236421 on osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:15.731414 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] issue_repop rep_tid 236421 o 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.731430 7f9d6abe6700  5 -- op tracker -- , seq: 748382, time: 2014-06-12 14:47:15.731427, event: waiting for subops from 21, op: osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.731447 7f9d6abe6700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:15.731478 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2745 (0'0,173'2745] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] append_log log((0'0,173'2745], crt=0'0) [173'2746 (173'2745) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142599 2014-06-12 08:46:18.106994]
2014-06-12 14:47:15.731504 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2745 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] add_log_entry 173'2746 (173'2745) modify   35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 by client.4305.0:142599 2014-06-12 08:46:18.106994
2014-06-12 14:47:15.731518 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2745 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] append_log  adding 1 keys
2014-06-12 14:47:15.731539 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2746, trimmed: 
2014-06-12 14:47:15.731560 7f9d6abe6700  5 queue_transactions existing osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.731563 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:15.731564 7f9d6abe6700  5 queue_op 0x568e8a0 seq 2547 osr(3.4d4 0x276d810) 993525 bytes   (queue has 1 ops and 993525 bytes)
2014-06-12 14:47:15.731574 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5478fc0 (1 active)
2014-06-12 14:47:15.731580 7f9d7e169700  5 _do_op 0x568e8a0 seq 2547 osr(3.4d4 0x276d810)/0x276d810 start
2014-06-12 14:47:15.731583 7f9d7e169700 10 _do_transaction on 0x26809c0
2014-06-12 14:47:15.731569 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2745 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] eval_repop repgather(0x2680540 173'2746 rep_tid=236421 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.731587 7f9d6abe6700 10 osd.2 173 dequeue_op 0x3850f100 finish
2014-06-12 14:47:15.731589 7f9d7e169700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.731591 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.731593 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.731655 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.731674 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:15.731707 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:15.731723 7f9d7e169700 15 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.731728 7f9d7e169700 10 _omap_rmkeys meta/1fe07345/pglog_3.4d4/0//-1 r = 0
2014-06-12 14:47:15.731732 7f9d7e169700 15 _omap_setkeys meta/1fe07345/pglog_3.4d4/0//-1
2014-06-12 14:47:15.731806 7f9d7e169700 15 _write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 3203072~991232
2014-06-12 14:47:15.731893 7f9d7e169700 10 lookup_strip_header done  cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.732003 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:47:15.734401 7f9d7e169700 10 _generic_write 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 3203072~991232 = 0
2014-06-12 14:47:15.734475 7f9d7e169700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.734488 7f9d7e169700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.734499 7f9d7e169700 15 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3
2014-06-12 14:47:15.734504 7f9d7e169700 10 _setattrs 3.4d4_head/35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 = 0
2014-06-12 14:47:15.734710 7f9d7e169700 20 set_header setting 16502 cid 3.4d4_head oid 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 parent seq 0
2014-06-12 14:47:15.734809 7f9d7e169700 20 set_header setting 2082 cid meta oid 1fe07345/pglog_3.4d4/0//-1 parent seq 0
2014-06-12 14:47:15.734821 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:15.734826 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:47:15.738293 7f9d7e169700 10 _do_op 0x568e8a0 seq 2547 r = 0, finisher 0x73f82c0 0xf187e80
2014-06-12 14:47:15.738310 7f9d7e169700 10 _finish_op 0x568e8a0 seq 2547 osr(3.4d4 0x276d810)/0x276d810
2014-06-12 14:47:15.738318 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5478fc0 (0 active)
2014-06-12 14:47:15.738326 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:15.738317 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2745 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] op_commit: 236421
2014-06-12 14:47:15.738356 7f9d7715b700  5 -- op tracker -- , seq: 748382, time: 2014-06-12 14:47:15.738347, event: op_commit, op: osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.738397 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2745 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] op_applied: 236421
2014-06-12 14:47:15.738421 7f9d7795c700  5 -- op tracker -- , seq: 748382, time: 2014-06-12 14:47:15.738415, event: op_applied, op: osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.738442 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2745 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] op_applied on primary on version 173'2746
2014-06-12 14:47:15.741316 7f9d44101700  5 -- op tracker -- , seq: 748383, time: 2014-06-12 14:47:15.741266, event: header_read, op: osd_sub_op_reply(client.4305.0:142599 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.741337 7f9d44101700  5 -- op tracker -- , seq: 748383, time: 2014-06-12 14:47:15.741268, event: throttled, op: osd_sub_op_reply(client.4305.0:142599 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.741345 7f9d44101700  5 -- op tracker -- , seq: 748383, time: 2014-06-12 14:47:15.741281, event: all_read, op: osd_sub_op_reply(client.4305.0:142599 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.741354 7f9d44101700  5 -- op tracker -- , seq: 748383, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142599 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.741362 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142599 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:15.741370 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:47:15.741383 7f9d44101700 15 osd.2 173 enqueue_op 0x20e86900 prio 196 cost 0 latency 0.000116 osd_sub_op_reply(client.4305.0:142599 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:15.741436 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.741445 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x20e86900 prio 196 cost 0 latency 0.000179 osd_sub_op_reply(client.4305.0:142599 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0) v2 pg pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2745 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean]
2014-06-12 14:47:15.741461 7f9d6a3e5700  5 -- op tracker -- , seq: 748383, time: 2014-06-12 14:47:15.741458, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142599 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.741468 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2745 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] handle_message: 0x20e86900
2014-06-12 14:47:15.741481 7f9d6a3e5700  5 -- op tracker -- , seq: 748383, time: 2014-06-12 14:47:15.741478, event: started, op: osd_sub_op_reply(client.4305.0:142599 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.741485 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2745 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] sub_op_modify_reply: tid 236421 op  ack_type 4 from 21
2014-06-12 14:47:15.741498 7f9d6a3e5700  5 -- op tracker -- , seq: 748382, time: 2014-06-12 14:47:15.741494, event: sub_op_commit_rec, op: osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.741505 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2745 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] repop_all_applied: repop tid 236421 all applied 
2014-06-12 14:47:15.741515 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2745 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] eval_repop repgather(0x2680540 173'2746 rep_tid=236421 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.741526 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2745 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean]  sending ack on repgather(0x2680540 173'2746 rep_tid=236421 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0x282cca00
2014-06-12 14:47:15.741548 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 luod=173'2745 crt=0'0 lcod 173'2744 mlcod 173'2744 active+clean] repop_all_committed: repop tid 236421 all committed 
2014-06-12 14:47:15.741555 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2744 active+clean] eval_repop repgather(0x2680540 173'2746 rep_tid=236421 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:15.741565 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2744 active+clean] log_op_stats osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4 inb 991511 outb 0 rlat 0.012733 lat 0.012750
2014-06-12 14:47:15.741577 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2744 active+clean] publish_stats_to_osd 173:3544
2014-06-12 14:47:15.741585 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2744 active+clean]  sending commit on repgather(0x2680540 173'2746 rep_tid=236421 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4) 0xc0c8280
2014-06-12 14:47:15.741605 7f9d6a3e5700  5 -- op tracker -- , seq: 748382, time: 2014-06-12 14:47:15.741602, event: commit_sent, op: osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.741610 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2744 active+clean]  requeue_ops 
2014-06-12 14:47:15.741629 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2745 active+clean]  removing repgather(0x2680540 173'2746 rep_tid=236421 committed?=1 applied?=1 op=osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.741638 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2745 active+clean]    q front is repgather(0x2680540 173'2746 rep_tid=236421 committed?=1 applied?=1 op=osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.741646 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2745 active+clean] remove_repop repgather(0x2680540 173'2746 rep_tid=236421 committed?=1 applied?=1 op=osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:15.741655 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2745 active+clean]  requeue_ops 
2014-06-12 14:47:15.741673 7f9d6a3e5700  5 -- op tracker -- , seq: 748382, time: 2014-06-12 14:47:15.741670, event: done, op: osd_op(client.4305.0:142599 rbd_data.10a86b8b4567.0000000000000ef4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3203072~991232] 3.35736cd4 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:15.741688 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x20e86900 finish
2014-06-12 14:47:15.741694 7f9d6a3e5700  5 -- op tracker -- , seq: 748383, time: 2014-06-12 14:47:15.741691, event: done, op: osd_sub_op_reply(client.4305.0:142599 3.4d4 35736cd4/rbd_data.10a86b8b4567.0000000000000ef4/head//3 [] ondisk, result = 0)
2014-06-12 14:47:15.741700 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.741702 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.800528 7f9d5a16f700  5 -- op tracker -- , seq: 748384, time: 2014-06-12 14:47:15.799398, event: header_read, op: osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.800542 7f9d5a16f700  5 -- op tracker -- , seq: 748384, time: 2014-06-12 14:47:15.799400, event: throttled, op: osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.800559 7f9d5a16f700  5 -- op tracker -- , seq: 748384, time: 2014-06-12 14:47:15.800493, event: all_read, op: osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.800564 7f9d5a16f700  5 -- op tracker -- , seq: 748384, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.800569 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:15.800574 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:15.800581 7f9d5a16f700 15 osd.2 173 enqueue_op 0x20355d00 prio 127 cost 516903 latency 0.001183 osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:15.800637 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:15.800664 7f9d66bde700 10 osd.2 173 dequeue_op 0x20355d00 prio 127 cost 516903 latency 0.001265 osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2869 (0'0,173'2869] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2868 active]
2014-06-12 14:47:15.800695 7f9d66bde700  5 -- op tracker -- , seq: 748384, time: 2014-06-12 14:47:15.800691, event: reached_pg, op: osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.800704 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2869 (0'0,173'2869] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2868 active] handle_message: 0x20355d00
2014-06-12 14:47:15.800713 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2869 (0'0,173'2869] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2868 active] sub_op_modify trans f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 v 173'2870 (transaction) 174
2014-06-12 14:47:15.800724 7f9d66bde700  5 -- op tracker -- , seq: 748384, time: 2014-06-12 14:47:15.800721, event: started, op: osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.800742 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2869 (0'0,173'2869] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2868 active] append_log log((0'0,173'2869], crt=0'0) [173'2870 (173'2869) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145702 2014-06-12 08:43:49.229440]
2014-06-12 14:47:15.800763 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2870 (0'0,173'2870] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2868 active] add_log_entry 173'2870 (173'2869) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145702 2014-06-12 08:43:49.229440
2014-06-12 14:47:15.800810 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2870 (0'0,173'2870] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2868 active] append_log  adding 1 keys
2014-06-12 14:47:15.800848 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2870, trimmed: 
2014-06-12 14:47:15.800868 7f9d66bde700  5 -- op tracker -- , seq: 748384, time: 2014-06-12 14:47:15.800864, event: started, op: osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.800877 7f9d66bde700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:15.800880 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:15.800882 7f9d66bde700  5 queue_op 0x5e42c00 seq 2583 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:15.800895 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:15.800903 7f9d7815d700  5 _do_op 0x5e42c00 seq 2583 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:15.800892 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2870 (0'0,173'2870] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2868 active] do_sub_op osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:15.800907 7f9d66bde700 10 osd.2 173 dequeue_op 0x20355d00 finish
2014-06-12 14:47:15.800908 7f9d7815d700 10 _do_transaction on 0x1da3e0f0
2014-06-12 14:47:15.800911 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:15.800913 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:15.800915 7f9d7815d700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:15.801041 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:15.801070 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:15.801122 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:15.801232 7f9d7815d700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:15.801240 7f9d7815d700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:15.801244 7f9d7815d700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:15.801258 7f9d7815d700 15 _write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 1474560~516096
2014-06-12 14:47:15.801443 7f9d7815d700 10 lookup_strip_header done  cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:15.801667 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:47:15.803330 7f9d7815d700 10 _generic_write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 1474560~516096 = 0
2014-06-12 14:47:15.803384 7f9d7815d700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:15.803399 7f9d7815d700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:15.803408 7f9d7815d700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:15.803416 7f9d7815d700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:15.803584 7f9d7815d700 20 set_header setting 16959 cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 parent seq 0
2014-06-12 14:47:15.803640 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:15.803655 7f9d7815d700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:15.803667 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:47:15.806403 7f9d7815d700 10 _do_op 0x5e42c00 seq 2583 r = 0, finisher 0x6606060 0
2014-06-12 14:47:15.806426 7f9d7815d700 10 _finish_op 0x5e42c00 seq 2583 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:15.806447 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:15.806455 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:15.806445 7f9d7715b700  5 -- op tracker -- , seq: 748384, time: 2014-06-12 14:47:15.806435, event: commit_sent, op: osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.806463 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2870 (0'0,173'2870] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2868 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:15.806494 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:15.806555 7f9d7795c700  5 -- op tracker -- , seq: 748384, time: 2014-06-12 14:47:15.806547, event: sub_op_applied, op: osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.806569 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2870 (0'0,173'2870] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2869 active] sub_op_modify_applied on 0x1da3e000 op osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:15.806586 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2870 (0'0,173'2870] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2869 active] op_applied on primary on version 173'2870
2014-06-12 14:47:15.806596 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2870 (0'0,173'2870] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2869 active] op_applied on replica on version 173'2870
2014-06-12 14:47:15.806628 7f9d7795c700  5 -- op tracker -- , seq: 748384, time: 2014-06-12 14:47:15.806623, event: done, op: osd_sub_op(client.4302.0:145702 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2870 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:15.928294 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:15.928329 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:16.009224 7f9d5a16f700  5 -- op tracker -- , seq: 748385, time: 2014-06-12 14:47:16.009140, event: header_read, op: osd_sub_op_reply(client.4293.0:144208 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.009248 7f9d5a16f700  5 -- op tracker -- , seq: 748385, time: 2014-06-12 14:47:16.009143, event: throttled, op: osd_sub_op_reply(client.4293.0:144208 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.009261 7f9d5a16f700  5 -- op tracker -- , seq: 748385, time: 2014-06-12 14:47:16.009161, event: all_read, op: osd_sub_op_reply(client.4293.0:144208 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.009269 7f9d5a16f700  5 -- op tracker -- , seq: 748385, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144208 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.009277 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144208 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:16.009284 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:16.009297 7f9d5a16f700 15 osd.2 173 enqueue_op 0x59dcf00 prio 196 cost 0 latency 0.000156 osd_sub_op_reply(client.4293.0:144208 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:16.009341 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.009353 7f9d69be4700 10 osd.2 173 dequeue_op 0x59dcf00 prio 196 cost 0 latency 0.000212 osd_sub_op_reply(client.4293.0:144208 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2 pg pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]
2014-06-12 14:47:16.009376 7f9d69be4700  5 -- op tracker -- , seq: 748385, time: 2014-06-12 14:47:16.009373, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144208 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.009394 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] handle_message: 0x59dcf00
2014-06-12 14:47:16.009411 7f9d69be4700  5 -- op tracker -- , seq: 748385, time: 2014-06-12 14:47:16.009408, event: started, op: osd_sub_op_reply(client.4293.0:144208 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.009418 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] sub_op_modify_reply: tid 236306 op  ack_type 4 from 24
2014-06-12 14:47:16.009441 7f9d69be4700  5 -- op tracker -- , seq: 747993, time: 2014-06-12 14:47:16.009431, event: sub_op_commit_rec, op: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.009449 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] repop_all_applied: repop tid 236306 all applied 
2014-06-12 14:47:16.009461 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] eval_repop repgather(0xd5e2540 173'2591 rep_tid=236306 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.009477 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean]  sending ack on repgather(0xd5e2540 173'2591 rep_tid=236306 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) 0x25edd400
2014-06-12 14:47:16.009511 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2590 crt=0'0 lcod 173'2589 mlcod 173'2589 active+clean] repop_all_committed: repop tid 236306 all committed 
2014-06-12 14:47:16.009524 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2589 active+clean] eval_repop repgather(0xd5e2540 173'2591 rep_tid=236306 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.009543 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2589 active+clean] log_op_stats osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 inb 172311 outb 0 rlat 31.051399 lat 31.051432
2014-06-12 14:47:16.009561 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2589 active+clean] publish_stats_to_osd 173:2762
2014-06-12 14:47:16.009577 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2589 active+clean]  sending commit on repgather(0xd5e2540 173'2591 rep_tid=236306 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) 0x151e6780
2014-06-12 14:47:16.009608 7f9d69be4700  5 -- op tracker -- , seq: 747993, time: 2014-06-12 14:47:16.009603, event: commit_sent, op: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.009626 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2589 active+clean]  requeue_ops 
2014-06-12 14:47:16.009638 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean]  removing repgather(0xd5e2540 173'2591 rep_tid=236306 committed?=1 applied?=1 op=osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.009652 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean]    q front is repgather(0xd5e2540 173'2591 rep_tid=236306 committed?=1 applied?=1 op=osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.009664 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] remove_repop repgather(0xd5e2540 173'2591 rep_tid=236306 committed?=1 applied?=1 op=osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.009677 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean]  requeue_ops 
2014-06-12 14:47:16.009702 7f9d69be4700  5 -- op tracker -- , seq: 747993, time: 2014-06-12 14:47:16.009697, event: done, op: osd_op(client.4293.0:144208 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.009806 7f9d69be4700 10 osd.2 173 dequeue_op 0x59dcf00 finish
2014-06-12 14:47:16.009817 7f9d69be4700  5 -- op tracker -- , seq: 748385, time: 2014-06-12 14:47:16.009813, event: done, op: osd_sub_op_reply(client.4293.0:144208 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.009828 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.009831 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.144487 7f9d4a5d9700  5 -- op tracker -- , seq: 748386, time: 2014-06-12 14:47:16.144018, event: header_read, op: osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.144505 7f9d4a5d9700  5 -- op tracker -- , seq: 748386, time: 2014-06-12 14:47:16.144022, event: throttled, op: osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.144513 7f9d4a5d9700  5 -- op tracker -- , seq: 748386, time: 2014-06-12 14:47:16.144434, event: all_read, op: osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.144522 7f9d4a5d9700  5 -- op tracker -- , seq: 748386, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.144530 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:16.144558 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x3850e800 prio 63 cost 233472 latency 0.000540 osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.144678 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.144698 7f9d64bda700 10 osd.2 173 dequeue_op 0x3850e800 prio 63 cost 233472 latency 0.000679 osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]
2014-06-12 14:47:16.144751 7f9d64bda700  5 -- op tracker -- , seq: 748386, time: 2014-06-12 14:47:16.144740, event: reached_pg, op: osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.144774 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:16.144805 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] handle_message: 0x3850e800
2014-06-12 14:47:16.144830 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_op osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:16.144876 7f9d64bda700 15 getattr 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 '_'
2014-06-12 14:47:16.145200 7f9d64bda700 10 getattr 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 '_' = 0
2014-06-12 14:47:16.145229 7f9d64bda700 15 getattr 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 'snapset'
2014-06-12 14:47:16.145358 7f9d64bda700 10 getattr 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 'snapset' = 0
2014-06-12 14:47:16.145377 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] populate_obc_watchers f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.145401 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.145420 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] get_object_context: creating obc from disk: 0x3376e000
2014-06-12 14:47:16.145440 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] get_object_context: 0x3376e000 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 rwstate(none n=0 w=0) oi: f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'742 client.4305.0:36590 wrlock_by=unknown.0.0:0 dirty s 4194304 uv742) ssc: 0x2ef8f880 snapset: 0=[]:[]+head
2014-06-12 14:47:16.145472 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] find_object_context f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 @head oi=f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'742 client.4305.0:36590 wrlock_by=unknown.0.0:0 dirty s 4194304 uv742)
2014-06-12 14:47:16.145526 7f9d64bda700  5 -- op tracker -- , seq: 748386, time: 2014-06-12 14:47:16.145516, event: started, op: osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.145536 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] execute_ctx 0x5ca0600
2014-06-12 14:47:16.145561 7f9d64bda700  5 -- op tracker -- , seq: 748386, time: 2014-06-12 14:47:16.145553, event: started, op: osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.145572 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] ov 173'742 av 173'2858 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:16.145596 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472]
2014-06-12 14:47:16.145610 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:16.145666 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op  write 0~233472
2014-06-12 14:47:16.145685 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:16.145698 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 snapset=0x2ef8f8b8  snapc=0=[]
2014-06-12 14:47:16.145712 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:16.145726 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] finish_ctx f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 0x5ca0600 op modify  
2014-06-12 14:47:16.145740 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  set mtime to 2014-06-12 08:46:18.523490
2014-06-12 14:47:16.145762 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  final snapset 0=[]:[]+head in f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.145782 7f9d64bda700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  zeroing write result code 0
2014-06-12 14:47:16.145794 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] new_repop rep_tid 236422 on osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.145812 7f9d64bda700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] issue_repop rep_tid 236422 o f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.145840 7f9d64bda700  5 -- op tracker -- , seq: 748386, time: 2014-06-12 14:47:16.145835, event: waiting for subops from 29, op: osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.145862 7f9d64bda700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:16.145884 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2857 (0'0,173'2857] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] append_log log((0'0,173'2857], crt=0'0) [173'2858 (173'742) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142635 2014-06-12 08:46:18.523490]
2014-06-12 14:47:16.145904 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] add_log_entry 173'2858 (173'742) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142635 2014-06-12 08:46:18.523490
2014-06-12 14:47:16.145925 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] append_log  adding 1 keys
2014-06-12 14:47:16.145961 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2858, trimmed: 
2014-06-12 14:47:16.145984 7f9d64bda700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.145989 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:16.145991 7f9d64bda700  5 queue_op 0x1b39ef60 seq 2625 osr(3.4f0 0x276db90) 235765 bytes   (queue has 1 ops and 235765 bytes)
2014-06-12 14:47:16.145999 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] eval_repop repgather(0x34d8e6c0 173'2858 rep_tid=236422 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.146021 7f9d64bda700 10 osd.2 173 dequeue_op 0x3850e800 finish
2014-06-12 14:47:16.146026 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.146029 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.146044 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (1 active)
2014-06-12 14:47:16.146050 7f9d82171700  5 _do_op 0x1b39ef60 seq 2625 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:47:16.146054 7f9d82171700 10 _do_transaction on 0x51b3d40
2014-06-12 14:47:16.146060 7f9d82171700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.146149 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.146175 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:16.146219 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:16.146312 7f9d82171700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.146320 7f9d82171700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:47:16.146327 7f9d82171700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.146359 7f9d82171700 15 _write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 0~233472
2014-06-12 14:47:16.146464 7f9d82171700 10 lookup_strip_header done  cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.146616 7f9d82171700 10 file_to_extents done 
2014-06-12 14:47:16.147250 7f9d82171700 10 _generic_write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 0~233472 = 0
2014-06-12 14:47:16.147289 7f9d82171700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.147302 7f9d82171700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.147311 7f9d82171700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.147321 7f9d82171700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.147422 7f9d82171700 20 set_header setting 16512 cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 parent seq 0
2014-06-12 14:47:16.147459 7f9d82171700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:47:16.147475 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:16.147485 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:47:16.148396 7f9d82171700 10 _do_op 0x1b39ef60 seq 2625 r = 0, finisher 0x541f220 0x1726e600
2014-06-12 14:47:16.148428 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_commit: 236422
2014-06-12 14:47:16.148466 7f9d7715b700  5 -- op tracker -- , seq: 748386, time: 2014-06-12 14:47:16.148455, event: op_commit, op: osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.148483 7f9d82171700 10 _finish_op 0x1b39ef60 seq 2625 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.148501 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_applied: 236422
2014-06-12 14:47:16.148521 7f9d7795c700  5 -- op tracker -- , seq: 748386, time: 2014-06-12 14:47:16.148515, event: op_applied, op: osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.148531 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_applied on primary on version 173'2858
2014-06-12 14:47:16.148557 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (0 active)
2014-06-12 14:47:16.148566 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:16.151407 7f9d4a5d9700  5 -- op tracker -- , seq: 748387, time: 2014-06-12 14:47:16.150207, event: header_read, op: osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.151429 7f9d4a5d9700  5 -- op tracker -- , seq: 748387, time: 2014-06-12 14:47:16.150211, event: throttled, op: osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.151437 7f9d4a5d9700  5 -- op tracker -- , seq: 748387, time: 2014-06-12 14:47:16.151360, event: all_read, op: osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.151445 7f9d4a5d9700  5 -- op tracker -- , seq: 748387, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.151453 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:16.151482 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x3850ef00 prio 63 cost 516096 latency 0.001274 osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.151524 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.151541 7f9d653db700 10 osd.2 173 dequeue_op 0x3850ef00 prio 63 cost 516096 latency 0.001334 osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]
2014-06-12 14:47:16.151575 7f9d653db700  5 -- op tracker -- , seq: 748387, time: 2014-06-12 14:47:16.151569, event: reached_pg, op: osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.151585 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:16.151606 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] handle_message: 0x3850ef00
2014-06-12 14:47:16.151618 7f9d5501e700  5 -- op tracker -- , seq: 748388, time: 2014-06-12 14:47:16.151578, event: header_read, op: osd_sub_op_reply(client.4305.0:142635 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.151635 7f9d5501e700  5 -- op tracker -- , seq: 748388, time: 2014-06-12 14:47:16.151580, event: throttled, op: osd_sub_op_reply(client.4305.0:142635 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.151621 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_op osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:16.151643 7f9d5501e700  5 -- op tracker -- , seq: 748388, time: 2014-06-12 14:47:16.151591, event: all_read, op: osd_sub_op_reply(client.4305.0:142635 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.151651 7f9d5501e700  5 -- op tracker -- , seq: 748388, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142635 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.151651 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] get_object_context: found obc in cache: 0x3376e000
2014-06-12 14:47:16.151656 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142635 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:16.151667 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:16.151675 7f9d5501e700 15 osd.2 173 enqueue_op 0xe20b600 prio 196 cost 0 latency 0.000096 osd_sub_op_reply(client.4305.0:142635 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:16.151663 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] get_object_context: 0x3376e000 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 rwstate(write n=1 w=0) oi: f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2858 client.4305.0:142635 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2858) ssc: 0x2ef8f880 snapset: 0=[]:[]+head
2014-06-12 14:47:16.151681 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] find_object_context f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 @head oi=f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2858 client.4305.0:142635 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2858)
2014-06-12 14:47:16.151698 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:16.151708 7f9d653db700  5 -- op tracker -- , seq: 748387, time: 2014-06-12 14:47:16.151703, event: started, op: osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.151717 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] execute_ctx 0x1a916600
2014-06-12 14:47:16.151736 7f9d653db700  5 -- op tracker -- , seq: 748387, time: 2014-06-12 14:47:16.151730, event: started, op: osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.151743 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] ov 173'2858 av 173'2859 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:16.151758 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096]
2014-06-12 14:47:16.151770 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:16.151785 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op  write 233472~516096
2014-06-12 14:47:16.151802 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:16.151814 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 snapset=0x2ef8f8b8  snapc=0=[]
2014-06-12 14:47:16.151827 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:16.151838 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] finish_ctx f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 0x1a916600 op modify  
2014-06-12 14:47:16.151852 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  set mtime to 2014-06-12 08:46:18.529240
2014-06-12 14:47:16.151874 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  final snapset 0=[]:[]+head in f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.151896 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  zeroing write result code 0
2014-06-12 14:47:16.151911 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] new_repop rep_tid 236423 on osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.151930 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] issue_repop rep_tid 236423 o f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.151961 7f9d653db700  5 -- op tracker -- , seq: 748387, time: 2014-06-12 14:47:16.151955, event: waiting for subops from 29, op: osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.151982 7f9d653db700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:16.152004 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2858 (0'0,173'2858] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] append_log log((0'0,173'2858], crt=0'0) [173'2859 (173'2858) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142636 2014-06-12 08:46:18.529240]
2014-06-12 14:47:16.152024 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] add_log_entry 173'2859 (173'2858) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142636 2014-06-12 08:46:18.529240
2014-06-12 14:47:16.152056 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] append_log  adding 1 keys
2014-06-12 14:47:16.152086 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2859, trimmed: 
2014-06-12 14:47:16.152110 7f9d653db700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.152116 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:16.152118 7f9d653db700  5 queue_op 0x26f6900 seq 2626 osr(3.4f0 0x276db90) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:16.152128 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (1 active)
2014-06-12 14:47:16.152136 7f9d7e96a700  5 _do_op 0x26f6900 seq 2626 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:47:16.152140 7f9d7e96a700 10 _do_transaction on 0x26812c0
2014-06-12 14:47:16.152125 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] eval_repop repgather(0x2562180 173'2859 rep_tid=236423 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.152145 7f9d653db700 10 osd.2 173 dequeue_op 0x3850ef00 finish
2014-06-12 14:47:16.152147 7f9d7e96a700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.152152 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:16.152155 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.152154 7f9d623d5700 10 osd.2 173 dequeue_op 0xe20b600 prio 196 cost 0 latency 0.000576 osd_sub_op_reply(client.4305.0:142635 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]
2014-06-12 14:47:16.152171 7f9d623d5700  5 -- op tracker -- , seq: 748388, time: 2014-06-12 14:47:16.152168, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142635 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.152176 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] handle_message: 0xe20b600
2014-06-12 14:47:16.152185 7f9d623d5700  5 -- op tracker -- , seq: 748388, time: 2014-06-12 14:47:16.152183, event: started, op: osd_sub_op_reply(client.4305.0:142635 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.152188 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] sub_op_modify_reply: tid 236422 op  ack_type 4 from 29
2014-06-12 14:47:16.152199 7f9d623d5700  5 -- op tracker -- , seq: 748386, time: 2014-06-12 14:47:16.152195, event: sub_op_commit_rec, op: osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.152204 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] repop_all_applied: repop tid 236422 all applied 
2014-06-12 14:47:16.152210 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] eval_repop repgather(0x34d8e6c0 173'2858 rep_tid=236422 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.152219 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean]  sending ack on repgather(0x34d8e6c0 173'2858 rep_tid=236422 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0x17432f00
2014-06-12 14:47:16.152238 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2857 crt=0'0 lcod 173'2856 mlcod 173'2856 active+clean] repop_all_committed: repop tid 236422 all committed 
2014-06-12 14:47:16.152248 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean] eval_repop repgather(0x34d8e6c0 173'2858 rep_tid=236422 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.152263 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean] log_op_stats osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 inb 233751 outb 0 rlat 0.008218 lat 0.008244
2014-06-12 14:47:16.152274 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean] publish_stats_to_osd 173:3716
2014-06-12 14:47:16.152282 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean]  sending commit on repgather(0x34d8e6c0 173'2858 rep_tid=236422 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0x17432280
2014-06-12 14:47:16.152304 7f9d623d5700  5 -- op tracker -- , seq: 748386, time: 2014-06-12 14:47:16.152299, event: commit_sent, op: osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.152312 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2856 active+clean]  requeue_ops 
2014-06-12 14:47:16.152321 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  removing repgather(0x34d8e6c0 173'2858 rep_tid=236422 committed?=1 applied?=1 op=osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.152338 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]    q front is repgather(0x34d8e6c0 173'2858 rep_tid=236422 committed?=1 applied?=1 op=osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.152346 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] remove_repop repgather(0x34d8e6c0 173'2858 rep_tid=236422 committed?=1 applied?=1 op=osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.152355 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  requeue_ops 
2014-06-12 14:47:16.152365 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.152367 7f9d623d5700  5 -- op tracker -- , seq: 748386, time: 2014-06-12 14:47:16.152364, event: done, op: osd_op(client.4305.0:142635 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~233472] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.152393 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:16.152440 7f9d623d5700 10 osd.2 173 dequeue_op 0xe20b600 finish
2014-06-12 14:47:16.152446 7f9d623d5700  5 -- op tracker -- , seq: 748388, time: 2014-06-12 14:47:16.152443, event: done, op: osd_sub_op_reply(client.4305.0:142635 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.152452 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.152451 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:16.152454 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.152471 7f9d7e96a700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.152476 7f9d7e96a700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:47:16.152481 7f9d7e96a700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.152497 7f9d7e96a700 15 _write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 233472~516096
2014-06-12 14:47:16.152641 7f9d7e96a700 10 lookup_strip_header done  cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.152886 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:47:16.154513 7f9d7e96a700 10 _generic_write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 233472~516096 = 0
2014-06-12 14:47:16.154568 7f9d7e96a700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.154581 7f9d7e96a700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.154585 7f9d7e96a700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.154590 7f9d7e96a700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.154734 7f9d7e96a700 20 set_header setting 16512 cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 parent seq 0
2014-06-12 14:47:16.154787 7f9d7e96a700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:47:16.154799 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:16.154804 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:47:16.156073 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:16.157039 7f9d7e96a700 10 _do_op 0x26f6900 seq 2626 r = 0, finisher 0x6607180 0xcb60d40
2014-06-12 14:47:16.157052 7f9d7e96a700 10 _finish_op 0x26f6900 seq 2626 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.157058 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (0 active)
2014-06-12 14:47:16.157064 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:16.157056 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] op_commit: 236423
2014-06-12 14:47:16.157080 7f9d7715b700  5 -- op tracker -- , seq: 748387, time: 2014-06-12 14:47:16.157073, event: op_commit, op: osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.157093 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] op_applied: 236423
2014-06-12 14:47:16.157119 7f9d7795c700  5 -- op tracker -- , seq: 748387, time: 2014-06-12 14:47:16.157114, event: op_applied, op: osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.157130 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] op_applied on primary on version 173'2859
2014-06-12 14:47:16.158731 7f9d4a5d9700  5 -- op tracker -- , seq: 748389, time: 2014-06-12 14:47:16.157781, event: header_read, op: osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.158741 7f9d4a5d9700  5 -- op tracker -- , seq: 748389, time: 2014-06-12 14:47:16.157784, event: throttled, op: osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.158754 7f9d4a5d9700  5 -- op tracker -- , seq: 748389, time: 2014-06-12 14:47:16.158697, event: all_read, op: osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.158759 7f9d4a5d9700  5 -- op tracker -- , seq: 748389, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.158766 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:16.158787 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x2d17ba00 prio 63 cost 516096 latency 0.001005 osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.158841 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.158852 7f9d68be2700 10 osd.2 173 dequeue_op 0x2d17ba00 prio 63 cost 516096 latency 0.001071 osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]
2014-06-12 14:47:16.158874 7f9d68be2700  5 -- op tracker -- , seq: 748389, time: 2014-06-12 14:47:16.158869, event: reached_pg, op: osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.158884 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:16.158896 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] handle_message: 0x2d17ba00
2014-06-12 14:47:16.158904 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_op osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:16.158924 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] get_object_context: found obc in cache: 0x3376e000
2014-06-12 14:47:16.158932 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] get_object_context: 0x3376e000 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 rwstate(write n=1 w=0) oi: f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2859 client.4305.0:142636 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2859) ssc: 0x2ef8f880 snapset: 0=[]:[]+head
2014-06-12 14:47:16.158943 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] find_object_context f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 @head oi=f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2859 client.4305.0:142636 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2859)
2014-06-12 14:47:16.158961 7f9d68be2700  5 -- op tracker -- , seq: 748389, time: 2014-06-12 14:47:16.158958, event: started, op: osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.158967 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] execute_ctx 0x22136400
2014-06-12 14:47:16.158978 7f9d68be2700  5 -- op tracker -- , seq: 748389, time: 2014-06-12 14:47:16.158975, event: started, op: osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.158983 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] ov 173'2859 av 173'2860 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:16.158993 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_osd_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096]
2014-06-12 14:47:16.159001 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:16.159011 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_osd_op  write 749568~516096
2014-06-12 14:47:16.159023 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:16.159031 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 snapset=0x2ef8f8b8  snapc=0=[]
2014-06-12 14:47:16.159039 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:16.159045 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] finish_ctx f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 0x22136400 op modify  
2014-06-12 14:47:16.159053 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  set mtime to 2014-06-12 08:46:18.536767
2014-06-12 14:47:16.159068 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  final snapset 0=[]:[]+head in f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.159081 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  zeroing write result code 0
2014-06-12 14:47:16.159089 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] new_repop rep_tid 236424 on osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.159100 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] issue_repop rep_tid 236424 o f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.159117 7f9d68be2700  5 -- op tracker -- , seq: 748389, time: 2014-06-12 14:47:16.159114, event: waiting for subops from 29, op: osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.159133 7f9d68be2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:16.159150 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2859 (0'0,173'2859] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] append_log log((0'0,173'2859], crt=0'0) [173'2860 (173'2859) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142637 2014-06-12 08:46:18.536767]
2014-06-12 14:47:16.159179 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] add_log_entry 173'2860 (173'2859) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142637 2014-06-12 08:46:18.536767
2014-06-12 14:47:16.159198 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] append_log  adding 1 keys
2014-06-12 14:47:16.159228 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2860, trimmed: 
2014-06-12 14:47:16.159250 7f9d68be2700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.159254 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:16.159256 7f9d68be2700  5 queue_op 0x1beb0e40 seq 2627 osr(3.4f0 0x276db90) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:16.159264 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (1 active)
2014-06-12 14:47:16.159270 7f9d8116f700  5 _do_op 0x1beb0e40 seq 2627 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:47:16.159273 7f9d8116f700 10 _do_transaction on 0x2dc1e240
2014-06-12 14:47:16.159262 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] eval_repop repgather(0x2681b00 173'2860 rep_tid=236424 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.159278 7f9d68be2700 10 osd.2 173 dequeue_op 0x2d17ba00 finish
2014-06-12 14:47:16.159278 7f9d8116f700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.159281 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.159283 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.159371 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.159392 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:16.159426 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:16.159443 7f9d8116f700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.159448 7f9d8116f700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:47:16.159452 7f9d8116f700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.159554 7f9d8116f700 15 _write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 749568~516096
2014-06-12 14:47:16.159669 7f9d8116f700 10 lookup_strip_header done  cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.159769 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:47:16.160832 7f9d8116f700 10 _generic_write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 749568~516096 = 0
2014-06-12 14:47:16.160863 7f9d8116f700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.160871 7f9d8116f700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.160877 7f9d8116f700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.160882 7f9d8116f700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.161005 7f9d8116f700 20 set_header setting 16512 cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 parent seq 0
2014-06-12 14:47:16.161053 7f9d8116f700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:47:16.161063 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:16.161070 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:47:16.161831 7f9d5501e700  5 -- op tracker -- , seq: 748390, time: 2014-06-12 14:47:16.161783, event: header_read, op: osd_sub_op_reply(client.4305.0:142636 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.161844 7f9d5501e700  5 -- op tracker -- , seq: 748390, time: 2014-06-12 14:47:16.161785, event: throttled, op: osd_sub_op_reply(client.4305.0:142636 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.161860 7f9d5501e700  5 -- op tracker -- , seq: 748390, time: 2014-06-12 14:47:16.161795, event: all_read, op: osd_sub_op_reply(client.4305.0:142636 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.161868 7f9d5501e700  5 -- op tracker -- , seq: 748390, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142636 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.161872 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142636 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:16.161878 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:16.161885 7f9d5501e700 15 osd.2 173 enqueue_op 0xe20b000 prio 196 cost 0 latency 0.000101 osd_sub_op_reply(client.4305.0:142636 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:16.161910 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.161918 7f9d643d9700 10 osd.2 173 dequeue_op 0xe20b000 prio 196 cost 0 latency 0.000135 osd_sub_op_reply(client.4305.0:142636 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]
2014-06-12 14:47:16.161940 7f9d643d9700  5 -- op tracker -- , seq: 748390, time: 2014-06-12 14:47:16.161936, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142636 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.161950 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] handle_message: 0xe20b000
2014-06-12 14:47:16.161966 7f9d643d9700  5 -- op tracker -- , seq: 748390, time: 2014-06-12 14:47:16.161963, event: started, op: osd_sub_op_reply(client.4305.0:142636 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.161973 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] sub_op_modify_reply: tid 236423 op  ack_type 4 from 29
2014-06-12 14:47:16.161992 7f9d643d9700  5 -- op tracker -- , seq: 748387, time: 2014-06-12 14:47:16.161985, event: sub_op_commit_rec, op: osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.162000 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] repop_all_applied: repop tid 236423 all applied 
2014-06-12 14:47:16.162012 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] eval_repop repgather(0x2562180 173'2859 rep_tid=236423 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.162029 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean]  sending ack on repgather(0x2562180 173'2859 rep_tid=236423 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0x30af6a00
2014-06-12 14:47:16.162062 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2858 crt=0'0 lcod 173'2857 mlcod 173'2857 active+clean] repop_all_committed: repop tid 236423 all committed 
2014-06-12 14:47:16.162075 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] eval_repop repgather(0x2562180 173'2859 rep_tid=236423 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.162093 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] log_op_stats osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.011854 lat 0.011885
2014-06-12 14:47:16.162110 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean] publish_stats_to_osd 173:3717
2014-06-12 14:47:16.162124 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean]  sending commit on repgather(0x2562180 173'2859 rep_tid=236423 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0xbc08500
2014-06-12 14:47:16.162157 7f9d643d9700  5 -- op tracker -- , seq: 748387, time: 2014-06-12 14:47:16.162152, event: commit_sent, op: osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.162166 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2857 active+clean]  requeue_ops 
2014-06-12 14:47:16.162176 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  removing repgather(0x2562180 173'2859 rep_tid=236423 committed?=1 applied?=1 op=osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.162191 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]    q front is repgather(0x2562180 173'2859 rep_tid=236423 committed?=1 applied?=1 op=osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.162207 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] remove_repop repgather(0x2562180 173'2859 rep_tid=236423 committed?=1 applied?=1 op=osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.162229 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  requeue_ops 
2014-06-12 14:47:16.162250 7f9d643d9700  5 -- op tracker -- , seq: 748387, time: 2014-06-12 14:47:16.162242, event: done, op: osd_op(client.4305.0:142636 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 233472~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.162271 7f9d643d9700 10 osd.2 173 dequeue_op 0xe20b000 finish
2014-06-12 14:47:16.162280 7f9d643d9700  5 -- op tracker -- , seq: 748390, time: 2014-06-12 14:47:16.162276, event: done, op: osd_sub_op_reply(client.4305.0:142636 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.162293 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.162296 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.164025 7f9d8116f700 10 _do_op 0x1beb0e40 seq 2627 r = 0, finisher 0xa646020 0x465b600
2014-06-12 14:47:16.164041 7f9d8116f700 10 _finish_op 0x1beb0e40 seq 2627 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.164047 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (0 active)
2014-06-12 14:47:16.164051 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:16.164049 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_commit: 236424
2014-06-12 14:47:16.164082 7f9d7715b700  5 -- op tracker -- , seq: 748389, time: 2014-06-12 14:47:16.164073, event: op_commit, op: osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.164126 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_applied: 236424
2014-06-12 14:47:16.164146 7f9d7795c700  5 -- op tracker -- , seq: 748389, time: 2014-06-12 14:47:16.164142, event: op_applied, op: osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.164155 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_applied on primary on version 173'2860
2014-06-12 14:47:16.165302 7f9d4a5d9700  5 -- op tracker -- , seq: 748391, time: 2014-06-12 14:47:16.164329, event: header_read, op: osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.165315 7f9d4a5d9700  5 -- op tracker -- , seq: 748391, time: 2014-06-12 14:47:16.164331, event: throttled, op: osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.165326 7f9d4a5d9700  5 -- op tracker -- , seq: 748391, time: 2014-06-12 14:47:16.165265, event: all_read, op: osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.165345 7f9d4a5d9700  5 -- op tracker -- , seq: 748391, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.165353 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:16.165380 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x2d17b300 prio 63 cost 516096 latency 0.001051 osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.165413 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.165423 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2d17b300 prio 63 cost 516096 latency 0.001093 osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]
2014-06-12 14:47:16.165448 7f9d63bd8700  5 -- op tracker -- , seq: 748391, time: 2014-06-12 14:47:16.165443, event: reached_pg, op: osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.165459 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:16.165474 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] handle_message: 0x2d17b300
2014-06-12 14:47:16.165490 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_op osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:16.165516 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] get_object_context: found obc in cache: 0x3376e000
2014-06-12 14:47:16.165529 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] get_object_context: 0x3376e000 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 rwstate(write n=1 w=0) oi: f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2860 client.4305.0:142637 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2860) ssc: 0x2ef8f880 snapset: 0=[]:[]+head
2014-06-12 14:47:16.165546 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] find_object_context f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 @head oi=f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2860 client.4305.0:142637 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2860)
2014-06-12 14:47:16.165573 7f9d63bd8700  5 -- op tracker -- , seq: 748391, time: 2014-06-12 14:47:16.165568, event: started, op: osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.165582 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] execute_ctx 0x12200a00
2014-06-12 14:47:16.165601 7f9d63bd8700  5 -- op tracker -- , seq: 748391, time: 2014-06-12 14:47:16.165596, event: started, op: osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.165609 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] ov 173'2860 av 173'2861 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:16.165634 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096]
2014-06-12 14:47:16.165645 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:16.165668 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op  write 1265664~516096
2014-06-12 14:47:16.165684 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:16.165696 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 snapset=0x2ef8f8b8  snapc=0=[]
2014-06-12 14:47:16.165710 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:16.165722 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] finish_ctx f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 0x12200a00 op modify  
2014-06-12 14:47:16.165736 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  set mtime to 2014-06-12 08:46:18.543366
2014-06-12 14:47:16.165758 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  final snapset 0=[]:[]+head in f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.165782 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  zeroing write result code 0
2014-06-12 14:47:16.165794 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] new_repop rep_tid 236425 on osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.165810 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] issue_repop rep_tid 236425 o f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.165836 7f9d63bd8700  5 -- op tracker -- , seq: 748391, time: 2014-06-12 14:47:16.165831, event: waiting for subops from 29, op: osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.165859 7f9d63bd8700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:16.165883 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2860 (0'0,173'2860] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] append_log log((0'0,173'2860], crt=0'0) [173'2861 (173'2860) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142638 2014-06-12 08:46:18.543366]
2014-06-12 14:47:16.165915 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] add_log_entry 173'2861 (173'2860) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142638 2014-06-12 08:46:18.543366
2014-06-12 14:47:16.165937 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] append_log  adding 1 keys
2014-06-12 14:47:16.165971 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2861, trimmed: 
2014-06-12 14:47:16.165995 7f9d63bd8700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.166000 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:16.166002 7f9d63bd8700  5 queue_op 0x4878840 seq 2628 osr(3.4f0 0x276db90) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:16.166011 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (1 active)
2014-06-12 14:47:16.166017 7f9d7d167700  5 _do_op 0x4878840 seq 2628 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:47:16.166020 7f9d7d167700 10 _do_transaction on 0x2010fb00
2014-06-12 14:47:16.166009 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] eval_repop repgather(0x297026c0 173'2861 rep_tid=236425 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.166026 7f9d7d167700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.166027 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2d17b300 finish
2014-06-12 14:47:16.166031 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.166033 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.166120 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.166138 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:16.166172 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:16.166251 7f9d7d167700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.166260 7f9d7d167700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:47:16.166266 7f9d7d167700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.166286 7f9d7d167700 15 _write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 1265664~516096
2014-06-12 14:47:16.166406 7f9d7d167700 10 lookup_strip_header done  cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.166518 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:47:16.167236 7f9d5501e700  5 -- op tracker -- , seq: 748392, time: 2014-06-12 14:47:16.167191, event: header_read, op: osd_sub_op_reply(client.4305.0:142637 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.167248 7f9d5501e700  5 -- op tracker -- , seq: 748392, time: 2014-06-12 14:47:16.167193, event: throttled, op: osd_sub_op_reply(client.4305.0:142637 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.167266 7f9d5501e700  5 -- op tracker -- , seq: 748392, time: 2014-06-12 14:47:16.167203, event: all_read, op: osd_sub_op_reply(client.4305.0:142637 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.167276 7f9d5501e700  5 -- op tracker -- , seq: 748392, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142637 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.167280 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142637 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:16.167288 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:16.167294 7f9d5501e700 15 osd.2 173 enqueue_op 0xe20bc00 prio 196 cost 0 latency 0.000103 osd_sub_op_reply(client.4305.0:142637 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:16.167347 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.167358 7f9d61bd4700 10 osd.2 173 dequeue_op 0xe20bc00 prio 196 cost 0 latency 0.000167 osd_sub_op_reply(client.4305.0:142637 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]
2014-06-12 14:47:16.167379 7f9d61bd4700  5 -- op tracker -- , seq: 748392, time: 2014-06-12 14:47:16.167376, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142637 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.167389 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] handle_message: 0xe20bc00
2014-06-12 14:47:16.167402 7f9d61bd4700  5 -- op tracker -- , seq: 748392, time: 2014-06-12 14:47:16.167399, event: started, op: osd_sub_op_reply(client.4305.0:142637 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.167406 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] sub_op_modify_reply: tid 236424 op  ack_type 4 from 29
2014-06-12 14:47:16.167420 7f9d61bd4700  5 -- op tracker -- , seq: 748389, time: 2014-06-12 14:47:16.167415, event: sub_op_commit_rec, op: osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.167426 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] repop_all_applied: repop tid 236424 all applied 
2014-06-12 14:47:16.167433 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] eval_repop repgather(0x2681b00 173'2860 rep_tid=236424 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.167445 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean]  sending ack on repgather(0x2681b00 173'2860 rep_tid=236424 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0x22b53b80
2014-06-12 14:47:16.167465 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2859 crt=0'0 lcod 173'2858 mlcod 173'2858 active+clean] repop_all_committed: repop tid 236424 all committed 
2014-06-12 14:47:16.167476 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean] eval_repop repgather(0x2681b00 173'2860 rep_tid=236424 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.167494 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean] log_op_stats osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.009683 lat 0.009712
2014-06-12 14:47:16.167508 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean] publish_stats_to_osd 173:3718
2014-06-12 14:47:16.167517 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean]  sending commit on repgather(0x2681b00 173'2860 rep_tid=236424 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0x28045180
2014-06-12 14:47:16.167540 7f9d61bd4700  5 -- op tracker -- , seq: 748389, time: 2014-06-12 14:47:16.167535, event: commit_sent, op: osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.167549 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2858 active+clean]  requeue_ops 
2014-06-12 14:47:16.167561 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  removing repgather(0x2681b00 173'2860 rep_tid=236424 committed?=1 applied?=1 op=osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.167571 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]    q front is repgather(0x2681b00 173'2860 rep_tid=236424 committed?=1 applied?=1 op=osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.167582 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] remove_repop repgather(0x2681b00 173'2860 rep_tid=236424 committed?=1 applied?=1 op=osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.167595 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  requeue_ops 
2014-06-12 14:47:16.167607 7f9d61bd4700  5 -- op tracker -- , seq: 748389, time: 2014-06-12 14:47:16.167604, event: done, op: osd_op(client.4305.0:142637 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 749568~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.167623 7f9d61bd4700 10 osd.2 173 dequeue_op 0xe20bc00 finish
2014-06-12 14:47:16.167628 7f9d61bd4700  5 -- op tracker -- , seq: 748392, time: 2014-06-12 14:47:16.167626, event: done, op: osd_sub_op_reply(client.4305.0:142637 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.167635 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.167637 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.167670 7f9d7d167700 10 _generic_write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 1265664~516096 = 0
2014-06-12 14:47:16.167702 7f9d7d167700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.167712 7f9d7d167700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.167719 7f9d7d167700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.167724 7f9d7d167700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.167857 7f9d7d167700 20 set_header setting 16512 cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 parent seq 0
2014-06-12 14:47:16.167899 7f9d7d167700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:47:16.167909 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:16.167915 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:47:16.170282 7f9d4a5d9700  5 -- op tracker -- , seq: 748393, time: 2014-06-12 14:47:16.167962, event: header_read, op: osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.170303 7f9d4a5d9700  5 -- op tracker -- , seq: 748393, time: 2014-06-12 14:47:16.167964, event: throttled, op: osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.170312 7f9d4a5d9700  5 -- op tracker -- , seq: 748393, time: 2014-06-12 14:47:16.170234, event: all_read, op: osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.170315 7f9d7d167700 10 _do_op 0x4878840 seq 2628 r = 0, finisher 0x65fb1c0 0x34d0e740
2014-06-12 14:47:16.170321 7f9d4a5d9700  5 -- op tracker -- , seq: 748393, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.170329 7f9d7d167700 10 _finish_op 0x4878840 seq 2628 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.170331 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:16.170335 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (0 active)
2014-06-12 14:47:16.170348 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:16.170336 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] op_commit: 236425
2014-06-12 14:47:16.170361 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x2d17ad00 prio 63 cost 1032192 latency 0.002398 osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.170367 7f9d7715b700  5 -- op tracker -- , seq: 748391, time: 2014-06-12 14:47:16.170359, event: op_commit, op: osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.170423 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] op_applied: 236425
2014-06-12 14:47:16.170438 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.170449 7f9d7795c700  5 -- op tracker -- , seq: 748391, time: 2014-06-12 14:47:16.170441, event: op_applied, op: osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.170462 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] op_applied on primary on version 173'2861
2014-06-12 14:47:16.170480 7f9d633d7700 10 osd.2 173 dequeue_op 0x2d17ad00 prio 63 cost 1032192 latency 0.002518 osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]
2014-06-12 14:47:16.170506 7f9d633d7700  5 -- op tracker -- , seq: 748393, time: 2014-06-12 14:47:16.170503, event: reached_pg, op: osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.170514 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:16.170537 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] handle_message: 0x2d17ad00
2014-06-12 14:47:16.170549 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] do_op osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:16.170575 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] get_object_context: found obc in cache: 0x3376e000
2014-06-12 14:47:16.170585 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] get_object_context: 0x3376e000 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 rwstate(write n=1 w=0) oi: f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2861 client.4305.0:142638 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2861) ssc: 0x2ef8f880 snapset: 0=[]:[]+head
2014-06-12 14:47:16.170601 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] find_object_context f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 @head oi=f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2861 client.4305.0:142638 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2861)
2014-06-12 14:47:16.170626 7f9d633d7700  5 -- op tracker -- , seq: 748393, time: 2014-06-12 14:47:16.170622, event: started, op: osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.170634 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] execute_ctx 0x1a919000
2014-06-12 14:47:16.170650 7f9d633d7700  5 -- op tracker -- , seq: 748393, time: 2014-06-12 14:47:16.170646, event: started, op: osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.170657 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] do_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] ov 173'2861 av 173'2862 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:16.170671 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] do_osd_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192]
2014-06-12 14:47:16.170681 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:16.170694 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] do_osd_op  write 1781760~1032192
2014-06-12 14:47:16.170709 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:16.170721 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 snapset=0x2ef8f8b8  snapc=0=[]
2014-06-12 14:47:16.170737 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:16.170748 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] finish_ctx f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 0x1a919000 op modify  
2014-06-12 14:47:16.170759 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  set mtime to 2014-06-12 08:46:18.546040
2014-06-12 14:47:16.170780 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  final snapset 0=[]:[]+head in f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.170803 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  zeroing write result code 0
2014-06-12 14:47:16.170813 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] new_repop rep_tid 236426 on osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.170829 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] issue_repop rep_tid 236426 o f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.170853 7f9d633d7700  5 -- op tracker -- , seq: 748393, time: 2014-06-12 14:47:16.170849, event: waiting for subops from 29, op: osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.170875 7f9d633d7700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:16.170900 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2861 (0'0,173'2861] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] append_log log((0'0,173'2861], crt=0'0) [173'2862 (173'2861) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142639 2014-06-12 08:46:18.546040]
2014-06-12 14:47:16.170924 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] add_log_entry 173'2862 (173'2861) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142639 2014-06-12 08:46:18.546040
2014-06-12 14:47:16.170943 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] append_log  adding 1 keys
2014-06-12 14:47:16.170971 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2862, trimmed: 
2014-06-12 14:47:16.170998 7f9d633d7700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.171002 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:16.171004 7f9d633d7700  5 queue_op 0x694a1e0 seq 2629 osr(3.4f0 0x276db90) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:16.171012 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (1 active)
2014-06-12 14:47:16.171018 7f9d7915f700  5 _do_op 0x694a1e0 seq 2629 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:47:16.171022 7f9d7915f700 10 _do_transaction on 0x96da000
2014-06-12 14:47:16.171008 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] eval_repop repgather(0x22b79680 173'2862 rep_tid=236426 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.171025 7f9d633d7700 10 osd.2 173 dequeue_op 0x2d17ad00 finish
2014-06-12 14:47:16.171028 7f9d7915f700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.171029 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.171032 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.171120 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.171139 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:16.171172 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:16.171189 7f9d7915f700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.171194 7f9d7915f700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:47:16.171198 7f9d7915f700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.171208 7f9d7915f700 15 _write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 1781760~1032192
2014-06-12 14:47:16.171304 7f9d7915f700 10 lookup_strip_header done  cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.171459 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:47:16.172366 7f9d4a5d9700  5 -- op tracker -- , seq: 748394, time: 2014-06-12 14:47:16.171747, event: header_read, op: osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.172382 7f9d4a5d9700  5 -- op tracker -- , seq: 748394, time: 2014-06-12 14:47:16.171750, event: throttled, op: osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.172391 7f9d4a5d9700  5 -- op tracker -- , seq: 748394, time: 2014-06-12 14:47:16.172317, event: all_read, op: osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.172400 7f9d4a5d9700  5 -- op tracker -- , seq: 748394, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.172408 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:16.172436 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x2d17a400 prio 63 cost 516096 latency 0.000689 osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.172472 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.172482 7f9d67be0700 10 osd.2 173 dequeue_op 0x2d17a400 prio 63 cost 516096 latency 0.000734 osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]
2014-06-12 14:47:16.172511 7f9d67be0700  5 -- op tracker -- , seq: 748394, time: 2014-06-12 14:47:16.172505, event: reached_pg, op: osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.172519 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:16.172536 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] handle_message: 0x2d17a400
2014-06-12 14:47:16.172548 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] do_op osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:16.172576 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] get_object_context: found obc in cache: 0x3376e000
2014-06-12 14:47:16.172590 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] get_object_context: 0x3376e000 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 rwstate(write n=2 w=0) oi: f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2862 client.4305.0:142639 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2862) ssc: 0x2ef8f880 snapset: 0=[]:[]+head
2014-06-12 14:47:16.172606 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] find_object_context f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 @head oi=f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2862 client.4305.0:142639 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2862)
2014-06-12 14:47:16.172634 7f9d67be0700  5 -- op tracker -- , seq: 748394, time: 2014-06-12 14:47:16.172628, event: started, op: osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.172641 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] execute_ctx 0x2b884a00
2014-06-12 14:47:16.172660 7f9d67be0700  5 -- op tracker -- , seq: 748394, time: 2014-06-12 14:47:16.172654, event: started, op: osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.172669 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] do_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] ov 173'2862 av 173'2863 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:16.172687 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] do_osd_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096]
2014-06-12 14:47:16.172703 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:16.172723 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] do_osd_op  write 2813952~516096
2014-06-12 14:47:16.172746 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:16.172759 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 snapset=0x2ef8f8b8  snapc=0=[]
2014-06-12 14:47:16.172774 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:16.172787 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] finish_ctx f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 0x2b884a00 op modify  
2014-06-12 14:47:16.172802 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  set mtime to 2014-06-12 08:46:18.550050
2014-06-12 14:47:16.172827 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  final snapset 0=[]:[]+head in f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.172847 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  zeroing write result code 0
2014-06-12 14:47:16.172859 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] new_repop rep_tid 236427 on osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.172878 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] issue_repop rep_tid 236427 o f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.172909 7f9d67be0700  5 -- op tracker -- , seq: 748394, time: 2014-06-12 14:47:16.172903, event: waiting for subops from 29, op: osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.172930 7f9d67be0700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:16.172954 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2862 (0'0,173'2862] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] append_log log((0'0,173'2862], crt=0'0) [173'2863 (173'2862) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142640 2014-06-12 08:46:18.550050]
2014-06-12 14:47:16.172979 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] add_log_entry 173'2863 (173'2862) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142640 2014-06-12 08:46:18.550050
2014-06-12 14:47:16.172998 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] append_log  adding 1 keys
2014-06-12 14:47:16.173029 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2863, trimmed: 
2014-06-12 14:47:16.173048 7f9d67be0700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.173052 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:16.173054 7f9d67be0700  5 queue_op 0x62c7500 seq 2630 osr(3.4f0 0x276db90) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:47:16.173061 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] eval_repop repgather(0x1c379740 173'2863 rep_tid=236427 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.173081 7f9d67be0700 10 osd.2 173 dequeue_op 0x2d17a400 finish
2014-06-12 14:47:16.173084 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.173087 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.173365 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (2 active)
2014-06-12 14:47:16.174336 7f9d7915f700 10 _generic_write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 1781760~1032192 = 0
2014-06-12 14:47:16.174401 7f9d7915f700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.174423 7f9d7915f700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.174429 7f9d7915f700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.174434 7f9d7915f700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.174634 7f9d7915f700 20 set_header setting 16512 cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 parent seq 0
2014-06-12 14:47:16.174705 7f9d7915f700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:47:16.174716 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:16.174722 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:47:16.174776 7f9d5501e700  5 -- op tracker -- , seq: 748395, time: 2014-06-12 14:47:16.174732, event: header_read, op: osd_sub_op_reply(client.4305.0:142638 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.174786 7f9d5501e700  5 -- op tracker -- , seq: 748395, time: 2014-06-12 14:47:16.174735, event: throttled, op: osd_sub_op_reply(client.4305.0:142638 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.174795 7f9d5501e700  5 -- op tracker -- , seq: 748395, time: 2014-06-12 14:47:16.174748, event: all_read, op: osd_sub_op_reply(client.4305.0:142638 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.174828 7f9d5501e700  5 -- op tracker -- , seq: 748395, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142638 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.174834 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142638 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:16.174841 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:16.174850 7f9d5501e700 15 osd.2 173 enqueue_op 0x1e3eb000 prio 196 cost 0 latency 0.000118 osd_sub_op_reply(client.4305.0:142638 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:16.174882 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.174889 7f9d673df700 10 osd.2 173 dequeue_op 0x1e3eb000 prio 196 cost 0 latency 0.000157 osd_sub_op_reply(client.4305.0:142638 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]
2014-06-12 14:47:16.174912 7f9d673df700  5 -- op tracker -- , seq: 748395, time: 2014-06-12 14:47:16.174908, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142638 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.174920 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] handle_message: 0x1e3eb000
2014-06-12 14:47:16.174936 7f9d673df700  5 -- op tracker -- , seq: 748395, time: 2014-06-12 14:47:16.174933, event: started, op: osd_sub_op_reply(client.4305.0:142638 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.174942 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sub_op_modify_reply: tid 236425 op  ack_type 4 from 29
2014-06-12 14:47:16.174966 7f9d673df700  5 -- op tracker -- , seq: 748391, time: 2014-06-12 14:47:16.174958, event: sub_op_commit_rec, op: osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.174976 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] repop_all_applied: repop tid 236425 all applied 
2014-06-12 14:47:16.174990 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] eval_repop repgather(0x297026c0 173'2861 rep_tid=236425 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.175009 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean]  sending ack on repgather(0x297026c0 173'2861 rep_tid=236425 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0x18418a00
2014-06-12 14:47:16.175043 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2860 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] repop_all_committed: repop tid 236425 all committed 
2014-06-12 14:47:16.175052 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2859 active+clean] eval_repop repgather(0x297026c0 173'2861 rep_tid=236425 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.175071 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2859 active+clean] log_op_stats osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.010713 lat 0.010741
2014-06-12 14:47:16.175092 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2859 active+clean] publish_stats_to_osd 173:3719
2014-06-12 14:47:16.175106 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2859 active+clean]  sending commit on repgather(0x297026c0 173'2861 rep_tid=236425 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0x2f89d400
2014-06-12 14:47:16.175140 7f9d673df700  5 -- op tracker -- , seq: 748391, time: 2014-06-12 14:47:16.175134, event: commit_sent, op: osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.175148 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2859 active+clean]  requeue_ops 
2014-06-12 14:47:16.175159 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]  removing repgather(0x297026c0 173'2861 rep_tid=236425 committed?=1 applied?=1 op=osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.175176 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]    q front is repgather(0x297026c0 173'2861 rep_tid=236425 committed?=1 applied?=1 op=osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.175193 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] remove_repop repgather(0x297026c0 173'2861 rep_tid=236425 committed?=1 applied?=1 op=osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.175210 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]  requeue_ops 
2014-06-12 14:47:16.175232 7f9d673df700  5 -- op tracker -- , seq: 748391, time: 2014-06-12 14:47:16.175226, event: done, op: osd_op(client.4305.0:142638 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1265664~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.175253 7f9d673df700 10 osd.2 173 dequeue_op 0x1e3eb000 finish
2014-06-12 14:47:16.175262 7f9d673df700  5 -- op tracker -- , seq: 748395, time: 2014-06-12 14:47:16.175258, event: done, op: osd_sub_op_reply(client.4305.0:142638 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.175285 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.175289 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.178237 7f9d4a5d9700  5 -- op tracker -- , seq: 748396, time: 2014-06-12 14:47:16.177266, event: header_read, op: osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.178250 7f9d4a5d9700  5 -- op tracker -- , seq: 748396, time: 2014-06-12 14:47:16.177269, event: throttled, op: osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.178264 7f9d4a5d9700  5 -- op tracker -- , seq: 748396, time: 2014-06-12 14:47:16.178194, event: all_read, op: osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.178270 7f9d4a5d9700  5 -- op tracker -- , seq: 748396, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.178277 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:16.178301 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x2d17a000 prio 63 cost 516096 latency 0.001035 osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.178331 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.178345 7f9d663dd700 10 osd.2 173 dequeue_op 0x2d17a000 prio 63 cost 516096 latency 0.001078 osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]
2014-06-12 14:47:16.178375 7f9d663dd700  5 -- op tracker -- , seq: 748396, time: 2014-06-12 14:47:16.178369, event: reached_pg, op: osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.178387 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:16.178404 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] handle_message: 0x2d17a000
2014-06-12 14:47:16.178416 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] do_op osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:16.178442 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] get_object_context: found obc in cache: 0x3376e000
2014-06-12 14:47:16.178453 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] get_object_context: 0x3376e000 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 rwstate(write n=2 w=0) oi: f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2863 client.4305.0:142640 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2863) ssc: 0x2ef8f880 snapset: 0=[]:[]+head
2014-06-12 14:47:16.178469 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] find_object_context f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 @head oi=f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2863 client.4305.0:142640 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2863)
2014-06-12 14:47:16.178495 7f9d663dd700  5 -- op tracker -- , seq: 748396, time: 2014-06-12 14:47:16.178490, event: started, op: osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.178503 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] execute_ctx 0x2e413000
2014-06-12 14:47:16.178520 7f9d663dd700  5 -- op tracker -- , seq: 748396, time: 2014-06-12 14:47:16.178516, event: started, op: osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.178527 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] do_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] ov 173'2863 av 173'2864 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:16.178541 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] do_osd_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096]
2014-06-12 14:47:16.178551 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:16.178566 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] do_osd_op  write 3330048~516096
2014-06-12 14:47:16.178582 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:16.178594 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 snapset=0x2ef8f8b8  snapc=0=[]
2014-06-12 14:47:16.178606 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:16.178616 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] finish_ctx f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 0x2e413000 op modify  
2014-06-12 14:47:16.178628 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]  set mtime to 2014-06-12 08:46:18.556263
2014-06-12 14:47:16.178648 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]  final snapset 0=[]:[]+head in f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.178666 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]  zeroing write result code 0
2014-06-12 14:47:16.178677 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] new_repop rep_tid 236428 on osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.178694 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] issue_repop rep_tid 236428 o f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.178719 7f9d663dd700  5 -- op tracker -- , seq: 748396, time: 2014-06-12 14:47:16.178714, event: waiting for subops from 29, op: osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.178738 7f9d7915f700 10 _do_op 0x694a1e0 seq 2629 r = 0, finisher 0x8e0dcc0 0x1a3f1740
2014-06-12 14:47:16.178741 7f9d663dd700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:16.178765 7f9d7915f700 10 _finish_op 0x694a1e0 seq 2629 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.178773 7f9d8016d700  5 _do_op 0x62c7500 seq 2630 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:47:16.178779 7f9d8016d700 10 _do_transaction on 0x18225200
2014-06-12 14:47:16.178787 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (1 active)
2014-06-12 14:47:16.178788 7f9d8016d700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.178790 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:16.178778 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2863 (0'0,173'2863] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] append_log log((0'0,173'2863], crt=0'0) [173'2864 (173'2863) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142641 2014-06-12 08:46:18.556263]
2014-06-12 14:47:16.178803 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] add_log_entry 173'2864 (173'2863) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142641 2014-06-12 08:46:18.556263
2014-06-12 14:47:16.178821 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] append_log  adding 1 keys
2014-06-12 14:47:16.178844 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2864, trimmed: 
2014-06-12 14:47:16.178861 7f9d663dd700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.178865 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:16.178866 7f9d663dd700  5 queue_op 0x2b4a8a20 seq 2631 osr(3.4f0 0x276db90) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:47:16.178874 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (2 active)
2014-06-12 14:47:16.178871 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] eval_repop repgather(0x1ced2f00 173'2864 rep_tid=236428 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.178890 7f9d663dd700 10 osd.2 173 dequeue_op 0x2d17a000 finish
2014-06-12 14:47:16.178895 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.178897 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.178904 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.178898 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] op_commit: 236426
2014-06-12 14:47:16.178929 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:16.178933 7f9d7715b700  5 -- op tracker -- , seq: 748393, time: 2014-06-12 14:47:16.178924, event: op_commit, op: osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.178980 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:16.178975 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] op_applied: 236426
2014-06-12 14:47:16.178996 7f9d7795c700  5 -- op tracker -- , seq: 748393, time: 2014-06-12 14:47:16.178991, event: op_applied, op: osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.179007 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] op_applied on primary on version 173'2862
2014-06-12 14:47:16.179062 7f9d8016d700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.179069 7f9d8016d700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:47:16.179075 7f9d8016d700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.179100 7f9d8016d700 15 _write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 2813952~516096
2014-06-12 14:47:16.179252 7f9d8016d700 10 lookup_strip_header done  cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.179401 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:47:16.180603 7f9d8016d700 10 _generic_write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 2813952~516096 = 0
2014-06-12 14:47:16.180652 7f9d8016d700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.180667 7f9d8016d700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.180674 7f9d8016d700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.180683 7f9d8016d700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.180821 7f9d8016d700 20 set_header setting 16512 cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 parent seq 0
2014-06-12 14:47:16.180882 7f9d8016d700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:47:16.180898 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:16.180913 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:47:16.182508 7f9d8016d700 10 _do_op 0x62c7500 seq 2630 r = 0, finisher 0xc529a40 0x1e5d0ec0
2014-06-12 14:47:16.182521 7f9d8016d700 10 _finish_op 0x62c7500 seq 2630 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.182533 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (1 active)
2014-06-12 14:47:16.182547 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:16.182528 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] op_commit: 236427
2014-06-12 14:47:16.182552 7f9d79960700  5 _do_op 0x2b4a8a20 seq 2631 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:47:16.182559 7f9d79960700 10 _do_transaction on 0x1ced2d80
2014-06-12 14:47:16.182568 7f9d79960700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.182563 7f9d7715b700  5 -- op tracker -- , seq: 748394, time: 2014-06-12 14:47:16.182555, event: op_commit, op: osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.182584 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] op_applied: 236427
2014-06-12 14:47:16.182608 7f9d7795c700  5 -- op tracker -- , seq: 748394, time: 2014-06-12 14:47:16.182601, event: op_applied, op: osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.182625 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] op_applied on primary on version 173'2863
2014-06-12 14:47:16.182680 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.182696 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:16.182746 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:16.182769 7f9d79960700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.182774 7f9d79960700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:47:16.182779 7f9d79960700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.182850 7f9d79960700 15 _write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 3330048~516096
2014-06-12 14:47:16.182955 7f9d79960700 10 lookup_strip_header done  cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.183048 7f9d79960700 10 file_to_extents done 
2014-06-12 14:47:16.184112 7f9d79960700 10 _generic_write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 3330048~516096 = 0
2014-06-12 14:47:16.184145 7f9d79960700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.184154 7f9d79960700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.184160 7f9d79960700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.184167 7f9d79960700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.184291 7f9d79960700 20 set_header setting 16512 cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 parent seq 0
2014-06-12 14:47:16.184339 7f9d79960700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:47:16.184349 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:16.184356 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:47:16.184454 7f9d5501e700  5 -- op tracker -- , seq: 748397, time: 2014-06-12 14:47:16.184414, event: header_read, op: osd_sub_op_reply(client.4305.0:142639 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.184465 7f9d5501e700  5 -- op tracker -- , seq: 748397, time: 2014-06-12 14:47:16.184416, event: throttled, op: osd_sub_op_reply(client.4305.0:142639 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.184473 7f9d5501e700  5 -- op tracker -- , seq: 748397, time: 2014-06-12 14:47:16.184426, event: all_read, op: osd_sub_op_reply(client.4305.0:142639 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.184481 7f9d5501e700  5 -- op tracker -- , seq: 748397, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142639 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.184487 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142639 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:16.184493 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:16.184501 7f9d5501e700 15 osd.2 173 enqueue_op 0x1e3eaf00 prio 196 cost 0 latency 0.000086 osd_sub_op_reply(client.4305.0:142639 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:16.184530 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.184541 7f9d65bdc700 10 osd.2 173 dequeue_op 0x1e3eaf00 prio 196 cost 0 latency 0.000127 osd_sub_op_reply(client.4305.0:142639 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]
2014-06-12 14:47:16.184564 7f9d65bdc700  5 -- op tracker -- , seq: 748397, time: 2014-06-12 14:47:16.184560, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142639 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.184575 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] handle_message: 0x1e3eaf00
2014-06-12 14:47:16.184592 7f9d65bdc700  5 -- op tracker -- , seq: 748397, time: 2014-06-12 14:47:16.184588, event: started, op: osd_sub_op_reply(client.4305.0:142639 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.184600 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] sub_op_modify_reply: tid 236426 op  ack_type 4 from 29
2014-06-12 14:47:16.184623 7f9d65bdc700  5 -- op tracker -- , seq: 748393, time: 2014-06-12 14:47:16.184615, event: sub_op_commit_rec, op: osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.184632 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] repop_all_applied: repop tid 236426 all applied 
2014-06-12 14:47:16.184645 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] eval_repop repgather(0x22b79680 173'2862 rep_tid=236426 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.184664 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean]  sending ack on repgather(0x22b79680 173'2862 rep_tid=236426 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0x22f29180
2014-06-12 14:47:16.184698 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2861 crt=0'0 lcod 173'2860 mlcod 173'2860 active+clean] repop_all_committed: repop tid 236426 all committed 
2014-06-12 14:47:16.184710 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2860 active+clean] eval_repop repgather(0x22b79680 173'2862 rep_tid=236426 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.184729 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2860 active+clean] log_op_stats osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 0.016734 lat 0.016766
2014-06-12 14:47:16.184748 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2860 active+clean] publish_stats_to_osd 173:3720
2014-06-12 14:47:16.184764 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2860 active+clean]  sending commit on repgather(0x22b79680 173'2862 rep_tid=236426 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0x8abf900
2014-06-12 14:47:16.184799 7f9d65bdc700  5 -- op tracker -- , seq: 748393, time: 2014-06-12 14:47:16.184794, event: commit_sent, op: osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.184809 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2860 active+clean]  requeue_ops 
2014-06-12 14:47:16.184821 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]  removing repgather(0x22b79680 173'2862 rep_tid=236426 committed?=1 applied?=1 op=osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.184840 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]    q front is repgather(0x22b79680 173'2862 rep_tid=236426 committed?=1 applied?=1 op=osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.184858 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] remove_repop repgather(0x22b79680 173'2862 rep_tid=236426 committed?=1 applied?=1 op=osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.184873 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]  requeue_ops 
2014-06-12 14:47:16.184893 7f9d65bdc700  5 -- op tracker -- , seq: 748393, time: 2014-06-12 14:47:16.184888, event: done, op: osd_op(client.4305.0:142639 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 1781760~1032192] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.184911 7f9d65bdc700 10 osd.2 173 dequeue_op 0x1e3eaf00 finish
2014-06-12 14:47:16.184919 7f9d65bdc700  5 -- op tracker -- , seq: 748397, time: 2014-06-12 14:47:16.184916, event: done, op: osd_sub_op_reply(client.4305.0:142639 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.184940 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.184943 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.186447 7f9d79960700 10 _do_op 0x2b4a8a20 seq 2631 r = 0, finisher 0x505e880 0x38997800
2014-06-12 14:47:16.186465 7f9d79960700 10 _finish_op 0x2b4a8a20 seq 2631 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.186470 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] op_commit: 236428
2014-06-12 14:47:16.186485 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (0 active)
2014-06-12 14:47:16.186492 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:16.186500 7f9d7715b700  5 -- op tracker -- , seq: 748396, time: 2014-06-12 14:47:16.186492, event: op_commit, op: osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.186518 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] op_applied: 236428
2014-06-12 14:47:16.186544 7f9d7795c700  5 -- op tracker -- , seq: 748396, time: 2014-06-12 14:47:16.186537, event: op_applied, op: osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.186555 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] op_applied on primary on version 173'2864
2014-06-12 14:47:16.188122 7f9d4a5d9700  5 -- op tracker -- , seq: 748398, time: 2014-06-12 14:47:16.186987, event: header_read, op: osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.188136 7f9d4a5d9700  5 -- op tracker -- , seq: 748398, time: 2014-06-12 14:47:16.186989, event: throttled, op: osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.188145 7f9d4a5d9700  5 -- op tracker -- , seq: 748398, time: 2014-06-12 14:47:16.188068, event: all_read, op: osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.188154 7f9d4a5d9700  5 -- op tracker -- , seq: 748398, time: 0.000000, event: dispatched, op: osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.188162 7f9d4a5d9700 20 osd.2 173 should_share_map client.4305 172.16.96.220:0/1243879 173
2014-06-12 14:47:16.188191 7f9d4a5d9700 15 osd.2 173 enqueue_op 0x2d17b800 prio 63 cost 348160 latency 0.001203 osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.188224 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.188231 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2d17b800 prio 63 cost 348160 latency 0.001243 osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 pg pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]
2014-06-12 14:47:16.193204 7f9d62bd6700  5 -- op tracker -- , seq: 748398, time: 2014-06-12 14:47:16.193197, event: reached_pg, op: osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.193217 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
2014-06-12 14:47:16.193232 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] handle_message: 0x2d17b800
2014-06-12 14:47:16.193241 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] do_op osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected
2014-06-12 14:47:16.193266 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] get_object_context: found obc in cache: 0x3376e000
2014-06-12 14:47:16.193276 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] get_object_context: 0x3376e000 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 rwstate(write n=2 w=0) oi: f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2864 client.4305.0:142641 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2864) ssc: 0x2ef8f880 snapset: 0=[]:[]+head
2014-06-12 14:47:16.193291 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] find_object_context f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 @head oi=f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3(173'2864 client.4305.0:142641 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2864)
2014-06-12 14:47:16.193315 7f9d62bd6700  5 -- op tracker -- , seq: 748398, time: 2014-06-12 14:47:16.193311, event: started, op: osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.193322 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] execute_ctx 0xa04fe00
2014-06-12 14:47:16.193340 7f9d62bd6700  5 -- op tracker -- , seq: 748398, time: 2014-06-12 14:47:16.193335, event: started, op: osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.193348 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] do_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] ov 173'2864 av 173'2865 snapc 0=[] snapset 0=[]:[]+head
2014-06-12 14:47:16.193362 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] do_osd_op f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160]
2014-06-12 14:47:16.193373 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
2014-06-12 14:47:16.193389 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] do_osd_op  write 3846144~348160
2014-06-12 14:47:16.193403 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] do_osd_op_effects on session 0x6231600
2014-06-12 14:47:16.193413 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 snapset=0x2ef8f8b8  snapc=0=[]
2014-06-12 14:47:16.193545 7f9d5501e700  5 -- op tracker -- , seq: 748399, time: 2014-06-12 14:47:16.193495, event: header_read, op: osd_sub_op_reply(client.4305.0:142640 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.193552 7f9d5501e700  5 -- op tracker -- , seq: 748399, time: 2014-06-12 14:47:16.193497, event: throttled, op: osd_sub_op_reply(client.4305.0:142640 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.193556 7f9d5501e700  5 -- op tracker -- , seq: 748399, time: 2014-06-12 14:47:16.193522, event: all_read, op: osd_sub_op_reply(client.4305.0:142640 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.193560 7f9d5501e700  5 -- op tracker -- , seq: 748399, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142640 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.193564 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142640 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:16.193568 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:16.193574 7f9d5501e700 15 osd.2 173 enqueue_op 0x1e3ea900 prio 196 cost 0 latency 0.000078 osd_sub_op_reply(client.4305.0:142640 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:16.193424 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] make_writeable f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 done, snapset=0=[]:[]+head
2014-06-12 14:47:16.193602 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:16.193599 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] finish_ctx f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 0xa04fe00 op modify  
2014-06-12 14:47:16.193635 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]  set mtime to 2014-06-12 08:46:18.566277
2014-06-12 14:47:16.193657 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]  final snapset 0=[]:[]+head in f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.193671 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]  zeroing write result code 0
2014-06-12 14:47:16.193682 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] new_repop rep_tid 236429 on osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4
2014-06-12 14:47:16.193699 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] issue_repop rep_tid 236429 o f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.193728 7f9d62bd6700  5 -- op tracker -- , seq: 748398, time: 2014-06-12 14:47:16.193723, event: waiting for subops from 29, op: osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.193749 7f9d62bd6700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:16.193769 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2864 (0'0,173'2864] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] append_log log((0'0,173'2864], crt=0'0) [173'2865 (173'2864) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142642 2014-06-12 08:46:18.566277]
2014-06-12 14:47:16.193798 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] add_log_entry 173'2865 (173'2864) modify   f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 by client.4305.0:142642 2014-06-12 08:46:18.566277
2014-06-12 14:47:16.193820 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] append_log  adding 1 keys
2014-06-12 14:47:16.193849 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2865, trimmed: 
2014-06-12 14:47:16.193870 7f9d62bd6700  5 queue_transactions existing osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.193874 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:16.193876 7f9d62bd6700  5 queue_op 0x769e3c0 seq 2632 osr(3.4f0 0x276db90) 350453 bytes   (queue has 1 ops and 350453 bytes)
2014-06-12 14:47:16.193884 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] eval_repop repgather(0x53126c0 173'2865 rep_tid=236429 committed?=0 applied?=0 lock=0 op=osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.193906 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2d17b800 finish
2014-06-12 14:47:16.193911 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:16.193914 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.193920 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f9e0 (1 active)
2014-06-12 14:47:16.193925 7f9d7d968700  5 _do_op 0x769e3c0 seq 2632 osr(3.4f0 0x276db90)/0x276db90 start
2014-06-12 14:47:16.193929 7f9d7d968700 10 _do_transaction on 0x5312c00
2014-06-12 14:47:16.193940 7f9d7d968700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.194077 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.194095 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:16.194150 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:16.194167 7f9d7d968700 15 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.194174 7f9d7d968700 10 _omap_rmkeys meta/1fe00ca5/pglog_3.4f0/0//-1 r = 0
2014-06-12 14:47:16.194179 7f9d7d968700 15 _omap_setkeys meta/1fe00ca5/pglog_3.4f0/0//-1
2014-06-12 14:47:16.194193 7f9d7d968700 15 _write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 3846144~348160
2014-06-12 14:47:16.194366 7f9d7d968700 10 lookup_strip_header done  cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.194383 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:47:16.195809 7f9d7d968700 10 _generic_write 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 3846144~348160 = 0
2014-06-12 14:47:16.195875 7f9d7d968700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.195897 7f9d7d968700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.195906 7f9d7d968700 15 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3
2014-06-12 14:47:16.195915 7f9d7d968700 10 _setattrs 3.4f0_head/f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 = 0
2014-06-12 14:47:16.196137 7f9d7d968700 20 set_header setting 16512 cid 3.4f0_head oid f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 parent seq 0
2014-06-12 14:47:16.196201 7f9d7d968700 20 set_header setting 2086 cid meta oid 1fe00ca5/pglog_3.4f0/0//-1 parent seq 0
2014-06-12 14:47:16.196215 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:16.196223 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:47:16.199505 7f9d5501e700  5 -- op tracker -- , seq: 748400, time: 2014-06-12 14:47:16.199456, event: header_read, op: osd_sub_op_reply(client.4305.0:142641 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.199516 7f9d5501e700  5 -- op tracker -- , seq: 748400, time: 2014-06-12 14:47:16.199458, event: throttled, op: osd_sub_op_reply(client.4305.0:142641 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.199523 7f9d5501e700  5 -- op tracker -- , seq: 748400, time: 2014-06-12 14:47:16.199469, event: all_read, op: osd_sub_op_reply(client.4305.0:142641 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.199530 7f9d5501e700  5 -- op tracker -- , seq: 748400, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142641 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.199534 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142641 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:16.199540 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:16.199547 7f9d5501e700 15 osd.2 173 enqueue_op 0x1e3eb700 prio 196 cost 0 latency 0.000091 osd_sub_op_reply(client.4305.0:142641 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:16.199574 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:16.199580 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x1e3ea900 prio 196 cost 0 latency 0.006085 osd_sub_op_reply(client.4305.0:142640 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]
2014-06-12 14:47:16.199599 7f9d6b3e7700  5 -- op tracker -- , seq: 748399, time: 2014-06-12 14:47:16.199596, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142640 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.199606 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] handle_message: 0x1e3ea900
2014-06-12 14:47:16.199621 7f9d6b3e7700  5 -- op tracker -- , seq: 748399, time: 2014-06-12 14:47:16.199617, event: started, op: osd_sub_op_reply(client.4305.0:142640 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.199626 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] sub_op_modify_reply: tid 236427 op  ack_type 4 from 29
2014-06-12 14:47:16.199648 7f9d6b3e7700  5 -- op tracker -- , seq: 748394, time: 2014-06-12 14:47:16.199639, event: sub_op_commit_rec, op: osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.199655 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] repop_all_applied: repop tid 236427 all applied 
2014-06-12 14:47:16.199665 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] eval_repop repgather(0x1c379740 173'2863 rep_tid=236427 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.199680 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean]  sending ack on repgather(0x1c379740 173'2863 rep_tid=236427 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0x6700f00
2014-06-12 14:47:16.199709 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2862 crt=0'0 lcod 173'2861 mlcod 173'2861 active+clean] repop_all_committed: repop tid 236427 all committed 
2014-06-12 14:47:16.199722 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2861 active+clean] eval_repop repgather(0x1c379740 173'2863 rep_tid=236427 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.199737 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2861 active+clean] log_op_stats osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.027960 lat 0.027989
2014-06-12 14:47:16.199759 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2861 active+clean] publish_stats_to_osd 173:3721
2014-06-12 14:47:16.199771 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2861 active+clean]  sending commit on repgather(0x1c379740 173'2863 rep_tid=236427 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0x24fec80
2014-06-12 14:47:16.199797 7f9d6b3e7700  5 -- op tracker -- , seq: 748394, time: 2014-06-12 14:47:16.199793, event: commit_sent, op: osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.199804 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2861 active+clean]  requeue_ops 
2014-06-12 14:47:16.199814 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean]  removing repgather(0x1c379740 173'2863 rep_tid=236427 committed?=1 applied?=1 op=osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.199828 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean]    q front is repgather(0x1c379740 173'2863 rep_tid=236427 committed?=1 applied?=1 op=osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.199842 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean] remove_repop repgather(0x1c379740 173'2863 rep_tid=236427 committed?=1 applied?=1 op=osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.199854 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean]  requeue_ops 
2014-06-12 14:47:16.199872 7f9d6b3e7700  5 -- op tracker -- , seq: 748394, time: 2014-06-12 14:47:16.199867, event: done, op: osd_op(client.4305.0:142640 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 2813952~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.199893 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x1e3ea900 finish
2014-06-12 14:47:16.199900 7f9d6b3e7700  5 -- op tracker -- , seq: 748399, time: 2014-06-12 14:47:16.199896, event: done, op: osd_sub_op_reply(client.4305.0:142640 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.199911 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:16.199914 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.201023 7f9d5501e700  5 -- op tracker -- , seq: 748401, time: 2014-06-12 14:47:16.200971, event: header_read, op: osd_sub_op_reply(client.4305.0:142642 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.201038 7f9d5501e700  5 -- op tracker -- , seq: 748401, time: 2014-06-12 14:47:16.200973, event: throttled, op: osd_sub_op_reply(client.4305.0:142642 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.201046 7f9d5501e700  5 -- op tracker -- , seq: 748401, time: 2014-06-12 14:47:16.200993, event: all_read, op: osd_sub_op_reply(client.4305.0:142642 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.201053 7f9d5501e700  5 -- op tracker -- , seq: 748401, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4305.0:142642 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.201057 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4305.0:142642 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:16.201063 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:47:16.201070 7f9d5501e700 15 osd.2 173 enqueue_op 0x1e3ea400 prio 196 cost 0 latency 0.000098 osd_sub_op_reply(client.4305.0:142642 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:16.201095 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:16.201102 7f9d693e3700 10 osd.2 173 dequeue_op 0x1e3eb700 prio 196 cost 0 latency 0.001646 osd_sub_op_reply(client.4305.0:142641 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean]
2014-06-12 14:47:16.201124 7f9d693e3700  5 -- op tracker -- , seq: 748400, time: 2014-06-12 14:47:16.201120, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142641 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.201132 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean] handle_message: 0x1e3eb700
2014-06-12 14:47:16.201147 7f9d693e3700  5 -- op tracker -- , seq: 748400, time: 2014-06-12 14:47:16.201144, event: started, op: osd_sub_op_reply(client.4305.0:142641 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.201153 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean] sub_op_modify_reply: tid 236428 op  ack_type 4 from 29
2014-06-12 14:47:16.201175 7f9d693e3700  5 -- op tracker -- , seq: 748396, time: 2014-06-12 14:47:16.201166, event: sub_op_commit_rec, op: osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.201183 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean] repop_all_applied: repop tid 236428 all applied 
2014-06-12 14:47:16.201194 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean] eval_repop repgather(0x1ced2f00 173'2864 rep_tid=236428 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.201210 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean]  sending ack on repgather(0x1ced2f00 173'2864 rep_tid=236428 committed?=0 applied?=1 lock=0 op=osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0x19e0a500
2014-06-12 14:47:16.201241 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2863 crt=0'0 lcod 173'2862 mlcod 173'2862 active+clean] repop_all_committed: repop tid 236428 all committed 
2014-06-12 14:47:16.201251 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2862 active+clean] eval_repop repgather(0x1ced2f00 173'2864 rep_tid=236428 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.201267 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2862 active+clean] log_op_stats osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 0.023973 lat 0.024000
2014-06-12 14:47:16.201284 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2862 active+clean] publish_stats_to_osd 173:3722
2014-06-12 14:47:16.201297 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2862 active+clean]  sending commit on repgather(0x1ced2f00 173'2864 rep_tid=236428 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0x8abec80
2014-06-12 14:47:16.201324 7f9d693e3700  5 -- op tracker -- , seq: 748396, time: 2014-06-12 14:47:16.201320, event: commit_sent, op: osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.201332 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2862 active+clean]  requeue_ops 
2014-06-12 14:47:16.201342 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean]  removing repgather(0x1ced2f00 173'2864 rep_tid=236428 committed?=1 applied?=1 op=osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.201356 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean]    q front is repgather(0x1ced2f00 173'2864 rep_tid=236428 committed?=1 applied?=1 op=osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.201369 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean] remove_repop repgather(0x1ced2f00 173'2864 rep_tid=236428 committed?=1 applied?=1 op=osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.201384 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean]  requeue_ops 
2014-06-12 14:47:16.201403 7f9d693e3700  5 -- op tracker -- , seq: 748396, time: 2014-06-12 14:47:16.201398, event: done, op: osd_op(client.4305.0:142641 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3330048~516096] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.201420 7f9d693e3700 10 osd.2 173 dequeue_op 0x1e3eb700 finish
2014-06-12 14:47:16.201427 7f9d693e3700  5 -- op tracker -- , seq: 748400, time: 2014-06-12 14:47:16.201424, event: done, op: osd_sub_op_reply(client.4305.0:142641 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.201438 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:16.201441 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.201658 7f9d683e1700 10 osd.2 173 dequeue_op 0x1e3ea400 prio 196 cost 0 latency 0.000686 osd_sub_op_reply(client.4305.0:142642 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0) v2 pg pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean]
2014-06-12 14:47:16.201676 7f9d683e1700  5 -- op tracker -- , seq: 748401, time: 2014-06-12 14:47:16.201673, event: reached_pg, op: osd_sub_op_reply(client.4305.0:142642 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.201684 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean] handle_message: 0x1e3ea400
2014-06-12 14:47:16.201703 7f9d683e1700  5 -- op tracker -- , seq: 748401, time: 2014-06-12 14:47:16.201699, event: started, op: osd_sub_op_reply(client.4305.0:142642 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.201708 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean] sub_op_modify_reply: tid 236429 op  ack_type 4 from 29
2014-06-12 14:47:16.201726 7f9d683e1700  5 -- op tracker -- , seq: 748398, time: 2014-06-12 14:47:16.201720, event: sub_op_commit_rec, op: osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.201733 7f9d683e1700 10 osd.2 173 dequeue_op 0x1e3ea400 finish
2014-06-12 14:47:16.201740 7f9d683e1700  5 -- op tracker -- , seq: 748401, time: 2014-06-12 14:47:16.201737, event: done, op: osd_sub_op_reply(client.4305.0:142642 3.4f0 f6cf2cf0/rbd_data.10a86b8b4567.0000000000000efa/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.201749 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.201753 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.204575 7f9d7d968700 10 _do_op 0x769e3c0 seq 2632 r = 0, finisher 0x738bd40 0x32b6c080
2014-06-12 14:47:16.204602 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean] op_commit: 236429
2014-06-12 14:47:16.204719 7f9d7715b700  5 -- op tracker -- , seq: 748398, time: 2014-06-12 14:47:16.204623, event: op_commit, op: osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.204730 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 luod=173'2864 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean] repop_all_committed: repop tid 236429 all committed 
2014-06-12 14:47:16.204743 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2863 active+clean] eval_repop repgather(0x53126c0 173'2865 rep_tid=236429 committed?=1 applied?=0 lock=0 op=osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.204791 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2863 active+clean] log_op_stats osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 inb 348439 outb 0 rlat 0.000000 lat 0.017802
2014-06-12 14:47:16.204811 7f9d7715b700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2863 active+clean] publish_stats_to_osd 173:3723
2014-06-12 14:47:16.204823 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2863 active+clean]  sending commit on repgather(0x53126c0 173'2865 rep_tid=236429 committed?=1 applied?=0 lock=0 op=osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) 0xfa09900
2014-06-12 14:47:16.204861 7f9d7715b700  5 -- op tracker -- , seq: 748398, time: 2014-06-12 14:47:16.204855, event: commit_sent, op: osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.205638 7f9d7d968700 10 _finish_op 0x769e3c0 seq 2632 osr(3.4f0 0x276db90)/0x276db90
2014-06-12 14:47:16.205671 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2863 active+clean] op_applied: 236429
2014-06-12 14:47:16.205705 7f9d7795c700  5 -- op tracker -- , seq: 748398, time: 2014-06-12 14:47:16.205697, event: op_applied, op: osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.205848 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2863 active+clean] op_applied on primary on version 173'2865
2014-06-12 14:47:16.206402 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2863 active+clean] repop_all_applied: repop tid 236429 all applied 
2014-06-12 14:47:16.206419 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2863 active+clean] eval_repop repgather(0x53126c0 173'2865 rep_tid=236429 committed?=1 applied?=1 lock=0 op=osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.206440 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2863 active+clean] log_op_stats osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4 inb 348439 outb 0 rlat 0.000000 lat 0.019451
2014-06-12 14:47:16.206458 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2863 active+clean] publish_stats_to_osd 173:3724
2014-06-12 14:47:16.206470 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2863 active+clean]  requeue_ops 
2014-06-12 14:47:16.206481 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2864 active+clean]  removing repgather(0x53126c0 173'2865 rep_tid=236429 committed?=1 applied?=1 op=osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.206497 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2864 active+clean]    q front is repgather(0x53126c0 173'2865 rep_tid=236429 committed?=1 applied?=1 op=osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.206512 7f9d7795c700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2864 active+clean] remove_repop repgather(0x53126c0 173'2865 rep_tid=236429 committed?=1 applied?=1 op=osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.206528 7f9d7795c700 15 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2864 active+clean]  requeue_ops 
2014-06-12 14:47:16.206598 7f9d7d968700  5 -- op tracker -- , seq: 748398, time: 2014-06-12 14:47:16.206583, event: done, op: osd_op(client.4305.0:142642 rbd_data.10a86b8b4567.0000000000000efa [set-alloc-hint object_size 4194304 write_size 4194304,write 3846144~348160] 3.f6cf2cf0 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.206625 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f9e0 (0 active)
2014-06-12 14:47:16.206629 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:16.275382 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:16.275434 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:16.370518 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:16.387394 7f9d5a16f700  5 -- op tracker -- , seq: 748402, time: 2014-06-12 14:47:16.385972, event: header_read, op: osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.387405 7f9d5a16f700  5 -- op tracker -- , seq: 748402, time: 2014-06-12 14:47:16.385976, event: throttled, op: osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.387422 7f9d5a16f700  5 -- op tracker -- , seq: 748402, time: 2014-06-12 14:47:16.387355, event: all_read, op: osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.387427 7f9d5a16f700  5 -- op tracker -- , seq: 748402, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.387431 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:16.387437 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:16.387444 7f9d5a16f700 15 osd.2 173 enqueue_op 0x59dd100 prio 127 cost 516903 latency 0.001472 osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:16.387470 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.387479 7f9d6abe6700 10 osd.2 173 dequeue_op 0x59dd100 prio 127 cost 516903 latency 0.001506 osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2870 (0'0,173'2870] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2869 active]
2014-06-12 14:47:16.387500 7f9d6abe6700  5 -- op tracker -- , seq: 748402, time: 2014-06-12 14:47:16.387497, event: reached_pg, op: osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.387514 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2870 (0'0,173'2870] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2869 active] handle_message: 0x59dd100
2014-06-12 14:47:16.387524 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2870 (0'0,173'2870] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2869 active] sub_op_modify trans f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 v 173'2871 (transaction) 174
2014-06-12 14:47:16.387536 7f9d6abe6700  5 -- op tracker -- , seq: 748402, time: 2014-06-12 14:47:16.387533, event: started, op: osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.387552 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2870 (0'0,173'2870] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2869 active] append_log log((0'0,173'2870], crt=0'0) [173'2871 (173'2870) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145703 2014-06-12 08:43:54.582275]
2014-06-12 14:47:16.387573 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2871 (0'0,173'2871] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2869 active] add_log_entry 173'2871 (173'2870) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145703 2014-06-12 08:43:54.582275
2014-06-12 14:47:16.387589 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2871 (0'0,173'2871] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2869 active] append_log  adding 1 keys
2014-06-12 14:47:16.387614 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2871, trimmed: 
2014-06-12 14:47:16.387633 7f9d6abe6700  5 -- op tracker -- , seq: 748402, time: 2014-06-12 14:47:16.387630, event: started, op: osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.387643 7f9d6abe6700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:16.387648 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:16.387650 7f9d6abe6700  5 queue_op 0xdf8a840 seq 2584 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:16.387655 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2871 (0'0,173'2871] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2869 active] do_sub_op osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:16.387665 7f9d6abe6700 10 osd.2 173 dequeue_op 0x59dd100 finish
2014-06-12 14:47:16.387668 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.387669 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.387842 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:16.387852 7f9d7c165700  5 _do_op 0xdf8a840 seq 2584 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:16.387857 7f9d7c165700 10 _do_transaction on 0x1b39c470
2014-06-12 14:47:16.387863 7f9d7c165700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:16.387997 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:16.388018 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:16.388081 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:16.388095 7f9d7c165700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:16.388100 7f9d7c165700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:16.388104 7f9d7c165700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:16.388112 7f9d7c165700 15 _write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 1990656~516096
2014-06-12 14:47:16.388219 7f9d7c165700 10 lookup_strip_header done  cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:16.388234 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:47:16.389929 7f9d7c165700 10 _generic_write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 1990656~516096 = 0
2014-06-12 14:47:16.389972 7f9d7c165700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:16.389984 7f9d7c165700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:16.389987 7f9d7c165700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:16.389992 7f9d7c165700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:16.390153 7f9d7c165700 20 set_header setting 16959 cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 parent seq 0
2014-06-12 14:47:16.390199 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:16.390215 7f9d7c165700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:16.390228 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:47:16.392724 7f9d7c165700 10 _do_op 0xdf8a840 seq 2584 r = 0, finisher 0x6906360 0
2014-06-12 14:47:16.392793 7f9d7715b700  5 -- op tracker -- , seq: 748402, time: 2014-06-12 14:47:16.392778, event: commit_sent, op: osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.392816 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2871 (0'0,173'2871] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2869 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:16.392869 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:16.392893 7f9d7c165700 10 _finish_op 0xdf8a840 seq 2584 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:16.392919 7f9d7795c700  5 -- op tracker -- , seq: 748402, time: 2014-06-12 14:47:16.392913, event: sub_op_applied, op: osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.392929 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2871 (0'0,173'2871] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2870 active] sub_op_modify_applied on 0x1b39c380 op osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:16.392955 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2871 (0'0,173'2871] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2870 active] op_applied on primary on version 173'2871
2014-06-12 14:47:16.392984 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2871 (0'0,173'2871] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2870 active] op_applied on replica on version 173'2871
2014-06-12 14:47:16.393033 7f9d7c165700  5 -- op tracker -- , seq: 748402, time: 2014-06-12 14:47:16.393023, event: done, op: osd_sub_op(client.4302.0:145703 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2871 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.393057 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:16.393062 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:16.483281 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:16.483312 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:16.483316 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:16.483318 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:16.483321 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 22; oldest is 31.517539 seconds old
2014-06-12 14:47:16.483319 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:16.570311 7f9d5a16f700  5 -- op tracker -- , seq: 748403, time: 2014-06-12 14:47:16.570253, event: header_read, op: osd_sub_op_reply(client.4293.0:144209 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.570327 7f9d5a16f700  5 -- op tracker -- , seq: 748403, time: 2014-06-12 14:47:16.570256, event: throttled, op: osd_sub_op_reply(client.4293.0:144209 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.570349 7f9d5a16f700  5 -- op tracker -- , seq: 748403, time: 2014-06-12 14:47:16.570270, event: all_read, op: osd_sub_op_reply(client.4293.0:144209 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.570355 7f9d5a16f700  5 -- op tracker -- , seq: 748403, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144209 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.570360 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144209 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:16.570365 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:16.570373 7f9d5a16f700 15 osd.2 173 enqueue_op 0x59dcc00 prio 196 cost 0 latency 0.000119 osd_sub_op_reply(client.4293.0:144209 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:16.570407 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.570414 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x59dcc00 prio 196 cost 0 latency 0.000161 osd_sub_op_reply(client.4293.0:144209 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2 pg pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean]
2014-06-12 14:47:16.570438 7f9d6a3e5700  5 -- op tracker -- , seq: 748403, time: 2014-06-12 14:47:16.570436, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144209 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.570445 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] handle_message: 0x59dcc00
2014-06-12 14:47:16.570455 7f9d6a3e5700  5 -- op tracker -- , seq: 748403, time: 2014-06-12 14:47:16.570453, event: started, op: osd_sub_op_reply(client.4293.0:144209 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.570459 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] sub_op_modify_reply: tid 236307 op  ack_type 4 from 24
2014-06-12 14:47:16.570474 7f9d6a3e5700  5 -- op tracker -- , seq: 747994, time: 2014-06-12 14:47:16.570467, event: sub_op_commit_rec, op: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.570486 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] repop_all_applied: repop tid 236307 all applied 
2014-06-12 14:47:16.570493 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] eval_repop repgather(0x2f89f140 173'2592 rep_tid=236307 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.570504 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean]  sending ack on repgather(0x2f89f140 173'2592 rep_tid=236307 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) 0x30f6ef00
2014-06-12 14:47:16.570523 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2591 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] repop_all_committed: repop tid 236307 all committed 
2014-06-12 14:47:16.570533 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2590 active+clean] eval_repop repgather(0x2f89f140 173'2592 rep_tid=236307 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.570544 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2590 active+clean] log_op_stats osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 inb 2580759 outb 0 rlat 31.604741 lat 31.604762
2014-06-12 14:47:16.570555 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2590 active+clean] publish_stats_to_osd 173:2763
2014-06-12 14:47:16.570564 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2590 active+clean]  sending commit on repgather(0x2f89f140 173'2592 rep_tid=236307 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) 0x8abe000
2014-06-12 14:47:16.570587 7f9d6a3e5700  5 -- op tracker -- , seq: 747994, time: 2014-06-12 14:47:16.570583, event: commit_sent, op: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.570593 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2590 active+clean]  requeue_ops 
2014-06-12 14:47:16.570600 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2591 active+clean]  removing repgather(0x2f89f140 173'2592 rep_tid=236307 committed?=1 applied?=1 op=osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.570609 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2591 active+clean]    q front is repgather(0x2f89f140 173'2592 rep_tid=236307 committed?=1 applied?=1 op=osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.570617 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2591 active+clean] remove_repop repgather(0x2f89f140 173'2592 rep_tid=236307 committed?=1 applied?=1 op=osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.570625 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2591 active+clean]  requeue_ops 
2014-06-12 14:47:16.570638 7f9d6a3e5700  5 -- op tracker -- , seq: 747994, time: 2014-06-12 14:47:16.570635, event: done, op: osd_op(client.4293.0:144209 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~2580480] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.570653 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x59dcc00 finish
2014-06-12 14:47:16.570657 7f9d6a3e5700  5 -- op tracker -- , seq: 748403, time: 2014-06-12 14:47:16.570655, event: done, op: osd_sub_op_reply(client.4293.0:144209 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.570664 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.570665 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.729707 7f9d5a16f700  5 -- op tracker -- , seq: 748404, time: 2014-06-12 14:47:16.729635, event: header_read, op: osd_sub_op_reply(client.4293.0:144210 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.729723 7f9d5a16f700  5 -- op tracker -- , seq: 748404, time: 2014-06-12 14:47:16.729639, event: throttled, op: osd_sub_op_reply(client.4293.0:144210 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.729727 7f9d5a16f700  5 -- op tracker -- , seq: 748404, time: 2014-06-12 14:47:16.729653, event: all_read, op: osd_sub_op_reply(client.4293.0:144210 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.729732 7f9d5a16f700  5 -- op tracker -- , seq: 748404, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144210 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.729737 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144210 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:16.729742 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:16.729749 7f9d5a16f700 15 osd.2 173 enqueue_op 0x59dd700 prio 196 cost 0 latency 0.000113 osd_sub_op_reply(client.4293.0:144210 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:16.729777 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.729785 7f9d66bde700 10 osd.2 173 dequeue_op 0x59dd700 prio 196 cost 0 latency 0.000150 osd_sub_op_reply(client.4293.0:144210 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2 pg pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2591 active+clean]
2014-06-12 14:47:16.729809 7f9d66bde700  5 -- op tracker -- , seq: 748404, time: 2014-06-12 14:47:16.729806, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144210 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.729818 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2591 active+clean] handle_message: 0x59dd700
2014-06-12 14:47:16.729835 7f9d66bde700  5 -- op tracker -- , seq: 748404, time: 2014-06-12 14:47:16.729831, event: started, op: osd_sub_op_reply(client.4293.0:144210 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.729841 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2591 active+clean] sub_op_modify_reply: tid 236317 op  ack_type 4 from 24
2014-06-12 14:47:16.729864 7f9d66bde700  5 -- op tracker -- , seq: 748016, time: 2014-06-12 14:47:16.729854, event: sub_op_commit_rec, op: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.729871 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2591 active+clean] repop_all_applied: repop tid 236317 all applied 
2014-06-12 14:47:16.729882 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2591 active+clean] eval_repop repgather(0x224fe00 173'2593 rep_tid=236317 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.729899 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2591 active+clean]  sending ack on repgather(0x224fe00 173'2593 rep_tid=236317 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) 0x959a280
2014-06-12 14:47:16.729932 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2592 crt=0'0 lcod 173'2591 mlcod 173'2591 active+clean] repop_all_committed: repop tid 236317 all committed 
2014-06-12 14:47:16.729943 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2591 active+clean] eval_repop repgather(0x224fe00 173'2593 rep_tid=236317 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.729960 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2591 active+clean] log_op_stats osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 31.616021 lat 31.616050
2014-06-12 14:47:16.729983 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2591 active+clean] publish_stats_to_osd 173:2764
2014-06-12 14:47:16.730000 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2591 active+clean]  sending commit on repgather(0x224fe00 173'2593 rep_tid=236317 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) 0x22b25400
2014-06-12 14:47:16.730028 7f9d66bde700  5 -- op tracker -- , seq: 748016, time: 2014-06-12 14:47:16.730023, event: commit_sent, op: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.730036 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2591 active+clean]  requeue_ops 
2014-06-12 14:47:16.730046 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2592 active+clean]  removing repgather(0x224fe00 173'2593 rep_tid=236317 committed?=1 applied?=1 op=osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.730062 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2592 active+clean]    q front is repgather(0x224fe00 173'2593 rep_tid=236317 committed?=1 applied?=1 op=osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.730080 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2592 active+clean] remove_repop repgather(0x224fe00 173'2593 rep_tid=236317 committed?=1 applied?=1 op=osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.730093 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2592 active+clean]  requeue_ops 
2014-06-12 14:47:16.730114 7f9d66bde700  5 -- op tracker -- , seq: 748016, time: 2014-06-12 14:47:16.730109, event: done, op: osd_op(client.4293.0:144210 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.730135 7f9d66bde700 10 osd.2 173 dequeue_op 0x59dd700 finish
2014-06-12 14:47:16.730142 7f9d66bde700  5 -- op tracker -- , seq: 748404, time: 2014-06-12 14:47:16.730139, event: done, op: osd_sub_op_reply(client.4293.0:144210 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.730153 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.730157 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.762496 7f9d5a16f700  5 -- op tracker -- , seq: 748405, time: 2014-06-12 14:47:16.762435, event: header_read, op: osd_sub_op_reply(client.4293.0:144211 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.762512 7f9d5a16f700  5 -- op tracker -- , seq: 748405, time: 2014-06-12 14:47:16.762438, event: throttled, op: osd_sub_op_reply(client.4293.0:144211 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.762520 7f9d5a16f700  5 -- op tracker -- , seq: 748405, time: 2014-06-12 14:47:16.762454, event: all_read, op: osd_sub_op_reply(client.4293.0:144211 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.762528 7f9d5a16f700  5 -- op tracker -- , seq: 748405, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144211 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.762538 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144211 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:16.762546 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:16.762559 7f9d5a16f700 15 osd.2 173 enqueue_op 0x59dd000 prio 196 cost 0 latency 0.000124 osd_sub_op_reply(client.4293.0:144211 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:16.762597 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.762607 7f9d69be4700 10 osd.2 173 dequeue_op 0x59dd000 prio 196 cost 0 latency 0.000172 osd_sub_op_reply(client.4293.0:144211 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2 pg pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2592 active+clean]
2014-06-12 14:47:16.762628 7f9d69be4700  5 -- op tracker -- , seq: 748405, time: 2014-06-12 14:47:16.762625, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144211 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.762637 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2592 active+clean] handle_message: 0x59dd000
2014-06-12 14:47:16.762653 7f9d69be4700  5 -- op tracker -- , seq: 748405, time: 2014-06-12 14:47:16.762649, event: started, op: osd_sub_op_reply(client.4293.0:144211 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.762659 7f9d69be4700  7 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2592 active+clean] sub_op_modify_reply: tid 236320 op  ack_type 4 from 24
2014-06-12 14:47:16.762680 7f9d69be4700  5 -- op tracker -- , seq: 748023, time: 2014-06-12 14:47:16.762672, event: sub_op_commit_rec, op: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.762688 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2592 active+clean] repop_all_applied: repop tid 236320 all applied 
2014-06-12 14:47:16.762699 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2592 active+clean] eval_repop repgather(0x32b68b40 173'2594 rep_tid=236320 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.762715 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2592 active+clean]  sending ack on repgather(0x32b68b40 173'2594 rep_tid=236320 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) 0x31657900
2014-06-12 14:47:16.762747 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2593 crt=0'0 lcod 173'2592 mlcod 173'2592 active+clean] repop_all_committed: repop tid 236320 all committed 
2014-06-12 14:47:16.762761 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2592 active+clean] eval_repop repgather(0x32b68b40 173'2594 rep_tid=236320 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:16.762777 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2592 active+clean] log_op_stats osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 31.597679 lat 31.597710
2014-06-12 14:47:16.762795 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2592 active+clean] publish_stats_to_osd 173:2765
2014-06-12 14:47:16.762808 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2592 active+clean]  sending commit on repgather(0x32b68b40 173'2594 rep_tid=236320 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) 0x6ce8500
2014-06-12 14:47:16.762841 7f9d69be4700  5 -- op tracker -- , seq: 748023, time: 2014-06-12 14:47:16.762835, event: commit_sent, op: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.762848 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2592 active+clean]  requeue_ops 
2014-06-12 14:47:16.762859 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2593 active+clean]  removing repgather(0x32b68b40 173'2594 rep_tid=236320 committed?=1 applied?=1 op=osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.762873 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2593 active+clean]    q front is repgather(0x32b68b40 173'2594 rep_tid=236320 committed?=1 applied?=1 op=osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.762888 7f9d69be4700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2593 active+clean] remove_repop repgather(0x32b68b40 173'2594 rep_tid=236320 committed?=1 applied?=1 op=osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:16.762901 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2593 active+clean]  requeue_ops 
2014-06-12 14:47:16.762920 7f9d69be4700  5 -- op tracker -- , seq: 748023, time: 2014-06-12 14:47:16.762915, event: done, op: osd_op(client.4293.0:144211 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3268608~516096] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:16.762939 7f9d69be4700 10 osd.2 173 dequeue_op 0x59dd000 finish
2014-06-12 14:47:16.762946 7f9d69be4700  5 -- op tracker -- , seq: 748405, time: 2014-06-12 14:47:16.762943, event: done, op: osd_sub_op_reply(client.4293.0:144211 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:16.762957 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.762960 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.921834 7f9d5a16f700  5 -- op tracker -- , seq: 748406, time: 2014-06-12 14:47:16.921055, event: header_read, op: osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.921852 7f9d5a16f700  5 -- op tracker -- , seq: 748406, time: 2014-06-12 14:47:16.921057, event: throttled, op: osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.921857 7f9d5a16f700  5 -- op tracker -- , seq: 748406, time: 2014-06-12 14:47:16.921800, event: all_read, op: osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.921862 7f9d5a16f700  5 -- op tracker -- , seq: 748406, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.921866 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:16.921871 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:16.921876 7f9d5a16f700 15 osd.2 173 enqueue_op 0x12bf6400 prio 127 cost 516903 latency 0.000821 osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:16.921898 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.921904 7f9d64bda700 10 osd.2 173 dequeue_op 0x12bf6400 prio 127 cost 516903 latency 0.000849 osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2871 (0'0,173'2871] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2870 active]
2014-06-12 14:47:16.921920 7f9d64bda700  5 -- op tracker -- , seq: 748406, time: 2014-06-12 14:47:16.921917, event: reached_pg, op: osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.921929 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2871 (0'0,173'2871] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2870 active] handle_message: 0x12bf6400
2014-06-12 14:47:16.921936 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2871 (0'0,173'2871] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2870 active] sub_op_modify trans f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 v 173'2872 (transaction) 174
2014-06-12 14:47:16.921947 7f9d64bda700  5 -- op tracker -- , seq: 748406, time: 2014-06-12 14:47:16.921944, event: started, op: osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.921959 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2871 (0'0,173'2871] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2870 active] append_log log((0'0,173'2871], crt=0'0) [173'2872 (173'2871) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145704 2014-06-12 08:43:55.135886]
2014-06-12 14:47:16.921993 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2872 (0'0,173'2872] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2870 active] add_log_entry 173'2872 (173'2871) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145704 2014-06-12 08:43:55.135886
2014-06-12 14:47:16.922009 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2872 (0'0,173'2872] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2870 active] append_log  adding 1 keys
2014-06-12 14:47:16.922033 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2872, trimmed: 
2014-06-12 14:47:16.922049 7f9d64bda700  5 -- op tracker -- , seq: 748406, time: 2014-06-12 14:47:16.922046, event: started, op: osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.922066 7f9d64bda700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:16.922076 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:16.922078 7f9d64bda700  5 queue_op 0x6df0480 seq 2585 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:16.922088 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:16.922093 7f9d8096e700  5 _do_op 0x6df0480 seq 2585 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:16.922098 7f9d8096e700 10 _do_transaction on 0x6293270
2014-06-12 14:47:16.922085 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2872 (0'0,173'2872] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2870 active] do_sub_op osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:16.922101 7f9d64bda700 10 osd.2 173 dequeue_op 0x12bf6400 finish
2014-06-12 14:47:16.922104 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.922103 7f9d8096e700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:16.922106 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.926362 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:16.926389 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:16.926425 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:16.926533 7f9d8096e700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:16.926540 7f9d8096e700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:16.926545 7f9d8096e700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:16.926559 7f9d8096e700 15 _write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 2506752~516096
2014-06-12 14:47:16.926673 7f9d8096e700 10 lookup_strip_header done  cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:16.926777 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:47:16.928326 7f9d8096e700 10 _generic_write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 2506752~516096 = 0
2014-06-12 14:47:16.928416 7f9d8096e700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:16.928448 7f9d8096e700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:16.928453 7f9d8096e700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:16.928459 7f9d8096e700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:16.928617 7f9d8096e700 20 set_header setting 16959 cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 parent seq 0
2014-06-12 14:47:16.928681 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:16.928693 7f9d8096e700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:16.928705 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:47:16.930895 7f9d8096e700 10 _do_op 0x6df0480 seq 2585 r = 0, finisher 0x71603a0 0
2014-06-12 14:47:16.930916 7f9d8096e700 10 _finish_op 0x6df0480 seq 2585 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:16.930934 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:16.930941 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:16.930959 7f9d7715b700  5 -- op tracker -- , seq: 748406, time: 2014-06-12 14:47:16.930946, event: commit_sent, op: osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.930970 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2872 (0'0,173'2872] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2870 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:16.931009 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:16.931048 7f9d7795c700  5 -- op tracker -- , seq: 748406, time: 2014-06-12 14:47:16.931042, event: sub_op_applied, op: osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.931058 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2872 (0'0,173'2872] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2871 active] sub_op_modify_applied on 0x6293180 op osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:16.931072 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2872 (0'0,173'2872] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2871 active] op_applied on primary on version 173'2872
2014-06-12 14:47:16.931087 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2872 (0'0,173'2872] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2871 active] op_applied on replica on version 173'2872
2014-06-12 14:47:16.931110 7f9d7795c700  5 -- op tracker -- , seq: 748406, time: 2014-06-12 14:47:16.931107, event: done, op: osd_sub_op(client.4302.0:145704 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2872 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.943149 7f9d5a16f700  5 -- op tracker -- , seq: 748407, time: 2014-06-12 14:47:16.941370, event: header_read, op: osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.943166 7f9d5a16f700  5 -- op tracker -- , seq: 748407, time: 2014-06-12 14:47:16.941372, event: throttled, op: osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.943181 7f9d5a16f700  5 -- op tracker -- , seq: 748407, time: 2014-06-12 14:47:16.943104, event: all_read, op: osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.943186 7f9d5a16f700  5 -- op tracker -- , seq: 748407, time: 0.000000, event: dispatched, op: osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.943192 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:16.943197 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:16.943204 7f9d5a16f700 15 osd.2 173 enqueue_op 0x12bf7900 prio 127 cost 967463 latency 0.001833 osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:16.943237 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:16.943247 7f9d653db700 10 osd.2 173 dequeue_op 0x12bf7900 prio 127 cost 967463 latency 0.001877 osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.760( v 173'2429 (0'0,173'2429] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2428 active]
2014-06-12 14:47:16.943273 7f9d653db700  5 -- op tracker -- , seq: 748407, time: 2014-06-12 14:47:16.943269, event: reached_pg, op: osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.943283 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2429 (0'0,173'2429] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2428 active] handle_message: 0x12bf7900
2014-06-12 14:47:16.943296 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2429 (0'0,173'2429] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2428 active] sub_op_modify trans 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 v 173'2430 (transaction) 174
2014-06-12 14:47:16.943314 7f9d653db700  5 -- op tracker -- , seq: 748407, time: 2014-06-12 14:47:16.943310, event: started, op: osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.943330 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2429 (0'0,173'2429] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2428 active] append_log log((0'0,173'2429], crt=0'0) [173'2430 (173'667) modify   816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 by client.4301.0:145108 2014-06-12 08:44:05.411147]
2014-06-12 14:47:16.943360 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2430 (0'0,173'2430] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2428 active] add_log_entry 173'2430 (173'667) modify   816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 by client.4301.0:145108 2014-06-12 08:44:05.411147
2014-06-12 14:47:16.943386 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2430 (0'0,173'2430] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2428 active] append_log  adding 1 keys
2014-06-12 14:47:16.943421 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2430, trimmed: 
2014-06-12 14:47:16.943447 7f9d653db700  5 -- op tracker -- , seq: 748407, time: 2014-06-12 14:47:16.943442, event: started, op: osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.943460 7f9d653db700  5 queue_transactions existing osr(3.760 0x49b41b0)/0x49b41b0
2014-06-12 14:47:16.943465 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:16.943467 7f9d653db700  5 queue_op 0x142a81e0 seq 2243 osr(3.760 0x49b41b0) 968949 bytes   (queue has 1 ops and 968949 bytes)
2014-06-12 14:47:16.943475 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.760( v 173'2430 (0'0,173'2430] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2428 active] do_sub_op osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:16.943490 7f9d653db700 10 osd.2 173 dequeue_op 0x12bf7900 finish
2014-06-12 14:47:16.943494 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:16.943496 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:16.943505 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x55217a0 (1 active)
2014-06-12 14:47:16.943510 7f9d7c966700  5 _do_op 0x142a81e0 seq 2243 osr(3.760 0x49b41b0)/0x49b41b0 start
2014-06-12 14:47:16.943514 7f9d7c966700 10 _do_transaction on 0x206ec2b0
2014-06-12 14:47:16.943521 7f9d7c966700 15 _omap_setkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:16.943625 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:16.943648 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:16.943690 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:16.943784 7f9d7c966700 15 _omap_rmkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:16.943793 7f9d7c966700 10 _omap_rmkeys meta/1fe099ca/pglog_3.760/0//-1 r = 0
2014-06-12 14:47:16.943800 7f9d7c966700 15 _omap_setkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:16.943818 7f9d7c966700 15 _write 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 0~966656
2014-06-12 14:47:16.944006 7f9d7c966700 10 lookup_strip_header done  cid 3.760_head oid 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:16.944157 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:47:16.947820 7f9d7c966700 10 _generic_write 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 0~966656 = 0
2014-06-12 14:47:16.947940 7f9d7c966700 15 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:16.947961 7f9d7c966700 10 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 = 0
2014-06-12 14:47:16.947967 7f9d7c966700 15 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:16.947975 7f9d7c966700 10 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 = 0
2014-06-12 14:47:16.948338 7f9d7c966700 20 set_header setting 17060 cid 3.760_head oid 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 parent seq 0
2014-06-12 14:47:16.948463 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:16.948478 7f9d7c966700 20 set_header setting 2198 cid meta oid 1fe099ca/pglog_3.760/0//-1 parent seq 0
2014-06-12 14:47:16.948489 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:47:16.953115 7f9d7c966700 10 _do_op 0x142a81e0 seq 2243 r = 0, finisher 0x65d6ec0 0
2014-06-12 14:47:16.953136 7f9d7c966700 10 _finish_op 0x142a81e0 seq 2243 osr(3.760 0x49b41b0)/0x49b41b0
2014-06-12 14:47:16.953155 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x55217a0 (0 active)
2014-06-12 14:47:16.953156 7f9d7715b700  5 -- op tracker -- , seq: 748407, time: 2014-06-12 14:47:16.953146, event: commit_sent, op: osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.953165 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:16.953166 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2430 (0'0,173'2430] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2428 active] sub_op_modify_commit on op osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:16.953192 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:16.953230 7f9d7795c700  5 -- op tracker -- , seq: 748407, time: 2014-06-12 14:47:16.953223, event: sub_op_applied, op: osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:16.953240 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2430 (0'0,173'2430] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2429 active] sub_op_modify_applied on 0x206ec1c0 op osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:16.953259 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2430 (0'0,173'2430] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2429 active] op_applied on primary on version 173'2430
2014-06-12 14:47:16.953271 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2430 (0'0,173'2430] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2429 active] op_applied on replica on version 173'2430
2014-06-12 14:47:16.953294 7f9d7795c700  5 -- op tracker -- , seq: 748407, time: 2014-06-12 14:47:16.953289, event: done, op: osd_sub_op(client.4301.0:145108 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2430 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:17.210251 7f9d5a16f700  5 -- op tracker -- , seq: 748408, time: 2014-06-12 14:47:17.210188, event: header_read, op: osd_sub_op_reply(client.4293.0:144212 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:17.210274 7f9d5a16f700  5 -- op tracker -- , seq: 748408, time: 2014-06-12 14:47:17.210191, event: throttled, op: osd_sub_op_reply(client.4293.0:144212 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:17.210282 7f9d5a16f700  5 -- op tracker -- , seq: 748408, time: 2014-06-12 14:47:17.210204, event: all_read, op: osd_sub_op_reply(client.4293.0:144212 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:17.210289 7f9d5a16f700  5 -- op tracker -- , seq: 748408, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4293.0:144212 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:17.210295 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4293.0:144212 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:17.210301 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:17.210310 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a902200 prio 196 cost 0 latency 0.000122 osd_sub_op_reply(client.4293.0:144212 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:17.210373 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:17.210384 7f9d623d5700 10 osd.2 173 dequeue_op 0x2a902200 prio 196 cost 0 latency 0.000195 osd_sub_op_reply(client.4293.0:144212 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0) v2 pg pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2593 active+clean]
2014-06-12 14:47:17.210409 7f9d623d5700  5 -- op tracker -- , seq: 748408, time: 2014-06-12 14:47:17.210407, event: reached_pg, op: osd_sub_op_reply(client.4293.0:144212 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:17.210422 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2593 active+clean] handle_message: 0x2a902200
2014-06-12 14:47:17.210433 7f9d623d5700  5 -- op tracker -- , seq: 748408, time: 2014-06-12 14:47:17.210431, event: started, op: osd_sub_op_reply(client.4293.0:144212 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:17.210438 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2593 active+clean] sub_op_modify_reply: tid 236327 op  ack_type 4 from 24
2014-06-12 14:47:17.210453 7f9d623d5700  5 -- op tracker -- , seq: 748037, time: 2014-06-12 14:47:17.210446, event: sub_op_commit_rec, op: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:17.210458 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2593 active+clean] repop_all_applied: repop tid 236327 all applied 
2014-06-12 14:47:17.210464 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2593 active+clean] eval_repop repgather(0x31ee7bc0 173'2595 rep_tid=236327 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:17.210474 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2593 active+clean]  sending ack on repgather(0x31ee7bc0 173'2595 rep_tid=236327 committed?=0 applied?=1 lock=0 op=osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) 0xc210c80
2014-06-12 14:47:17.210496 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2594 crt=0'0 lcod 173'2593 mlcod 173'2593 active+clean] repop_all_committed: repop tid 236327 all committed 
2014-06-12 14:47:17.210503 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2593 active+clean] eval_repop repgather(0x31ee7bc0 173'2595 rep_tid=236327 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:17.210515 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2593 active+clean] log_op_stats osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4 inb 409879 outb 0 rlat 31.850252 lat 31.850271
2014-06-12 14:47:17.210527 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2593 active+clean] publish_stats_to_osd 173:2766
2014-06-12 14:47:17.210537 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2593 active+clean]  sending commit on repgather(0x31ee7bc0 173'2595 rep_tid=236327 committed?=1 applied?=1 lock=0 op=osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4) 0x25edca00
2014-06-12 14:47:17.210558 7f9d623d5700  5 -- op tracker -- , seq: 748037, time: 2014-06-12 14:47:17.210554, event: commit_sent, op: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:17.210563 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2593 active+clean]  requeue_ops 
2014-06-12 14:47:17.210569 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2594 active+clean]  removing repgather(0x31ee7bc0 173'2595 rep_tid=236327 committed?=1 applied?=1 op=osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:17.210578 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2594 active+clean]    q front is repgather(0x31ee7bc0 173'2595 rep_tid=236327 committed?=1 applied?=1 op=osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:17.210586 7f9d623d5700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2594 active+clean] remove_repop repgather(0x31ee7bc0 173'2595 rep_tid=236327 committed?=1 applied?=1 op=osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:17.210594 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2594 active+clean]  requeue_ops 
2014-06-12 14:47:17.210611 7f9d623d5700  5 -- op tracker -- , seq: 748037, time: 2014-06-12 14:47:17.210608, event: done, op: osd_op(client.4293.0:144212 rbd_data.109c6b8b4567.000000000000105e [set-alloc-hint object_size 4194304 write_size 4194304,write 3784704~409600] 3.e07d4a7 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:17.210642 7f9d623d5700 10 osd.2 173 dequeue_op 0x2a902200 finish
2014-06-12 14:47:17.210647 7f9d623d5700  5 -- op tracker -- , seq: 748408, time: 2014-06-12 14:47:17.210644, event: done, op: osd_sub_op_reply(client.4293.0:144212 3.4a7 e07d4a7/rbd_data.109c6b8b4567.000000000000105e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:17.210653 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:17.210655 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:17.216812 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:17.216839 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:17.285415 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:17.328744 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:17.328772 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:17.412623 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:17.412736 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:17.423876 7f9d5a16f700  5 -- op tracker -- , seq: 748409, time: 2014-06-12 14:47:17.421269, event: header_read, op: osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:17.423889 7f9d5a16f700  5 -- op tracker -- , seq: 748409, time: 2014-06-12 14:47:17.421272, event: throttled, op: osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:17.423898 7f9d5a16f700  5 -- op tracker -- , seq: 748409, time: 2014-06-12 14:47:17.423835, event: all_read, op: osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:17.423906 7f9d5a16f700  5 -- op tracker -- , seq: 748409, time: 0.000000, event: dispatched, op: osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:17.423911 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:17.423917 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:17.423925 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903200 prio 127 cost 1032999 latency 0.002656 osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:17.427135 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:17.427167 7f9d68be2700 10 osd.2 173 dequeue_op 0x2a903200 prio 127 cost 1032999 latency 0.005897 osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.760( v 173'2430 (0'0,173'2430] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2429 active]
2014-06-12 14:47:17.427224 7f9d68be2700  5 -- op tracker -- , seq: 748409, time: 2014-06-12 14:47:17.427217, event: reached_pg, op: osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:17.427259 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2430 (0'0,173'2430] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2429 active] handle_message: 0x2a903200
2014-06-12 14:47:17.427279 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2430 (0'0,173'2430] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2429 active] sub_op_modify trans 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 v 173'2431 (transaction) 174
2014-06-12 14:47:17.427299 7f9d68be2700  5 -- op tracker -- , seq: 748409, time: 2014-06-12 14:47:17.427294, event: started, op: osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:17.427320 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2430 (0'0,173'2430] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2429 active] append_log log((0'0,173'2430], crt=0'0) [173'2431 (173'2430) modify   816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 by client.4301.0:145109 2014-06-12 08:44:05.459954]
2014-06-12 14:47:17.427360 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2431 (0'0,173'2431] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2429 active] add_log_entry 173'2431 (173'2430) modify   816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 by client.4301.0:145109 2014-06-12 08:44:05.459954
2014-06-12 14:47:17.427386 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2431 (0'0,173'2431] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2429 active] append_log  adding 1 keys
2014-06-12 14:47:17.427424 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2431, trimmed: 
2014-06-12 14:47:17.427451 7f9d68be2700  5 -- op tracker -- , seq: 748409, time: 2014-06-12 14:47:17.427446, event: started, op: osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:17.427462 7f9d68be2700  5 queue_transactions existing osr(3.760 0x49b41b0)/0x49b41b0
2014-06-12 14:47:17.427467 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:17.427469 7f9d68be2700  5 queue_op 0x69f0c60 seq 2244 osr(3.760 0x49b41b0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:17.427484 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x55217a0 (1 active)
2014-06-12 14:47:17.427477 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.760( v 173'2431 (0'0,173'2431] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2429 active] do_sub_op osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:17.427493 7f9d68be2700 10 osd.2 173 dequeue_op 0x2a903200 finish
2014-06-12 14:47:17.427493 7f9d7a962700  5 _do_op 0x69f0c60 seq 2244 osr(3.760 0x49b41b0)/0x49b41b0 start
2014-06-12 14:47:17.427497 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:17.427498 7f9d7a962700 10 _do_transaction on 0x7f875f0
2014-06-12 14:47:17.427499 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:17.427506 7f9d7a962700 15 _omap_setkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:17.427614 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:17.427632 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:17.427672 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:17.427688 7f9d7a962700 15 _omap_rmkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:17.427694 7f9d7a962700 10 _omap_rmkeys meta/1fe099ca/pglog_3.760/0//-1 r = 0
2014-06-12 14:47:17.427699 7f9d7a962700 15 _omap_setkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:17.427713 7f9d7a962700 15 _write 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 966656~1032192
2014-06-12 14:47:17.427848 7f9d7a962700 10 lookup_strip_header done  cid 3.760_head oid 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:17.427937 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:47:17.432953 7f9d7a962700 10 _generic_write 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 966656~1032192 = 0
2014-06-12 14:47:17.433180 7f9d7a962700 15 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:17.433214 7f9d7a962700 10 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 = 0
2014-06-12 14:47:17.433246 7f9d7a962700 15 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:17.433292 7f9d7a962700 10 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 = 0
2014-06-12 14:47:17.433678 7f9d7a962700 20 set_header setting 17060 cid 3.760_head oid 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 parent seq 0
2014-06-12 14:47:17.435274 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:17.435297 7f9d7a962700 20 set_header setting 2198 cid meta oid 1fe099ca/pglog_3.760/0//-1 parent seq 0
2014-06-12 14:47:17.435317 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:47:17.439573 7f9d7a962700 10 _do_op 0x69f0c60 seq 2244 r = 0, finisher 0x6d38ca0 0
2014-06-12 14:47:17.439605 7f9d7a962700 10 _finish_op 0x69f0c60 seq 2244 osr(3.760 0x49b41b0)/0x49b41b0
2014-06-12 14:47:17.439614 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x55217a0 (0 active)
2014-06-12 14:47:17.439622 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:17.439632 7f9d7715b700  5 -- op tracker -- , seq: 748409, time: 2014-06-12 14:47:17.439618, event: commit_sent, op: osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:17.439643 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2431 (0'0,173'2431] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2429 active] sub_op_modify_commit on op osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:17.439677 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:17.439715 7f9d7795c700  5 -- op tracker -- , seq: 748409, time: 2014-06-12 14:47:17.439708, event: sub_op_applied, op: osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:17.439724 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2431 (0'0,173'2431] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active] sub_op_modify_applied on 0x7f87500 op osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:17.439740 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2431 (0'0,173'2431] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active] op_applied on primary on version 173'2431
2014-06-12 14:47:17.439749 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2431 (0'0,173'2431] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active] op_applied on replica on version 173'2431
2014-06-12 14:47:17.439769 7f9d7795c700  5 -- op tracker -- , seq: 748409, time: 2014-06-12 14:47:17.439766, event: done, op: osd_sub_op(client.4301.0:145109 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2431 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:17.483389 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:17.483407 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:47:17.483409 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:47:17.483411 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:47:17.483423 7f9d84175700 10 osd.2 173 send_pg_stats - 6 pgs updated
2014-06-12 14:47:17.483482 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:17.483501 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 8.44 >= max 0.5 = no, load too high
2014-06-12 14:47:17.483512 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:47:17.483522 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92685 < max (604800 seconds)
2014-06-12 14:47:17.483540 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:47:17.483541 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:17.483542 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:17.483544 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 18; oldest is 23.053804 seconds old
2014-06-12 14:47:17.699424 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:17.711226 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:17.714723 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:17.723596 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:17.727081 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:17.738387 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:17.806516 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:17.816051 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:17.816056 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:17.816057 7f9d74bfa700 20 osd.2 173 _dispatch 0x191d0e00 pg_stats_ack(6 pgs tid 4861) v1
2014-06-12 14:47:17.816059 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:47:17.816067 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:17.816068 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:17.829450 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:17.829482 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:17.907223 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:17.907248 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:17.913266 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:17.913320 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:18.148653 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:18.156156 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:18.157132 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:18.162358 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.164110 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:18.166093 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.167698 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.170407 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:18.171092 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.173147 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.175362 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.178849 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:18.178957 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.182606 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:18.185002 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.186552 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:18.193970 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.199971 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.201502 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.201809 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.206689 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:18.262959 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:18.263139 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:18.349423 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:18.349548 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:18.370606 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:18.387744 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.393142 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:18.414194 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:18.414333 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:18.451815 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:18.451858 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:18.483635 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:18.483663 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:18.483668 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:18.483670 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:18.483672 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 18; oldest is 24.053933 seconds old
2014-06-12 14:47:18.483708 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:18.570730 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.730221 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.763050 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.922185 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.931030 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:18.943584 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:18.953243 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:19.125085 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:19.125146 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:19.210719 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:19.227872 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,5,2,11])
2014-06-12 14:47:19.227888 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,5,2,11])
2014-06-12 14:47:19.228467 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:19.228510 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:19.228523 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:19.228533 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:19.228583 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:19.228687 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:19.228720 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:19.228754 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:19.228774 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:19.228796 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:19.228816 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:19.228837 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:19.228856 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:19.228875 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:19.228896 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:19.228916 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:19.228936 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:19.228955 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:19.228974 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:19.228995 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:19.229019 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:19.229040 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:19.285497 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:19.427599 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:19.439683 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:19.483766 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:19.483797 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:19.483802 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:19.483804 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:19.483807 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 18; oldest is 25.054067 seconds old
2014-06-12 14:47:19.483825 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:19.530136 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:19.530353 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:19.699525 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:19.711326 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:19.714812 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:19.723693 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:19.727169 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:19.738470 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:19.744796 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:19.744859 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:19.806587 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:19.848632 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:19.848638 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:19.848640 7f9d74bfa700 20 osd.2 173 _dispatch 0x11953520 ping v1
2014-06-12 14:47:19.848643 7f9d74bfa700 10 osd.2 173 ping from client.4290
2014-06-12 14:47:19.848646 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:19.848648 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:20.028905 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:20.028997 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:20.115064 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:20.115252 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:20.117798 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:20.117866 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:20.148751 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:20.156253 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:20.157220 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:20.162438 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.164199 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:20.166171 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.167775 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.170485 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:20.171172 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.173239 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.175461 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.178918 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:20.179029 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.182680 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:20.185073 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.186628 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:20.194031 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.200036 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.201569 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.201865 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.206753 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:20.208001 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:20.208114 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:20.370688 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:20.387829 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.393223 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:20.483901 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:20.483931 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:20.483936 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:20.483938 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:20.483941 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 18; oldest is 26.054202 seconds old
2014-06-12 14:47:20.483977 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:20.570810 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.618888 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:20.618966 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:20.712896 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:20.712908 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:20.712920 7f9d74bfa700 20 osd.2 173 _dispatch 0x32f07a00 ping v1
2014-06-12 14:47:20.712923 7f9d74bfa700 10 osd.2 173 ping from client.4300
2014-06-12 14:47:20.712926 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:20.712929 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:20.730303 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.752383 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:20.752426 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:20.763138 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.860047 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:20.860146 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:20.922276 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.928560 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,11])
2014-06-12 14:47:20.928584 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,11])
2014-06-12 14:47:20.929265 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:20.929308 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:20.929325 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:20.929340 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:20.929415 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:20.929426 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:20.929442 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:20.929489 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:20.929530 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:20.929561 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:20.929588 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:20.929612 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:20.929676 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:20.929697 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:20.929733 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:20.929762 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:20.929789 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:20.929813 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:20.929833 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:20.929856 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:20.929884 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:20.929911 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:20.931102 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:20.943654 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:20.953323 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:21.119470 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:21.119544 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:21.163854 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:21.163938 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:21.210790 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:21.252995 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:21.253184 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:21.285578 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:21.427680 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:21.429348 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,11])
2014-06-12 14:47:21.429362 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,11])
2014-06-12 14:47:21.429820 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:21.429856 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:21.429870 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:21.429883 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:21.429938 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:21.429958 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:21.429981 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:21.430029 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:21.430041 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:21.430054 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:21.430066 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:21.430098 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:21.430127 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:21.430150 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:21.430174 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:21.430201 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:21.430226 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:21.430246 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:21.430261 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:21.430275 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:21.430289 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:21.430301 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:21.439748 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:21.467433 7f9d56432700  5 -- op tracker -- , seq: 748410, time: 2014-06-12 14:47:21.467384, event: header_read, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:21.467443 7f9d56432700  5 -- op tracker -- , seq: 748410, time: 2014-06-12 14:47:21.467385, event: throttled, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:21.467451 7f9d56432700  5 -- op tracker -- , seq: 748410, time: 2014-06-12 14:47:21.467395, event: all_read, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:21.467462 7f9d56432700  5 -- op tracker -- , seq: 748410, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:21.467465 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:21.467468 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:21.467475 7f9d56432700 15 osd.2 173 enqueue_op 0x28dd0900 prio 196 cost 0 latency 0.000091 osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0) v2
2014-06-12 14:47:21.467519 7f9d56432700  5 -- op tracker -- , seq: 748411, time: 2014-06-12 14:47:21.467489, event: header_read, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.467523 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:21.467526 7f9d56432700  5 -- op tracker -- , seq: 748411, time: 2014-06-12 14:47:21.467490, event: throttled, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.467541 7f9d56432700  5 -- op tracker -- , seq: 748411, time: 2014-06-12 14:47:21.467507, event: all_read, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.467530 7f9d643d9700 10 osd.2 173 dequeue_op 0x28dd0900 prio 196 cost 0 latency 0.000146 osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0) v2 pg pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]
2014-06-12 14:47:21.467545 7f9d56432700  5 -- op tracker -- , seq: 748411, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.467552 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:21.467555 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:21.467559 7f9d56432700 15 osd.2 173 enqueue_op 0x28dd1c00 prio 196 cost 0 latency 0.000070 osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:21.467556 7f9d643d9700  5 -- op tracker -- , seq: 748410, time: 2014-06-12 14:47:21.467545, event: reached_pg, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:21.467577 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:21.467568 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] handle_message: 0x28dd0900
2014-06-12 14:47:21.467592 7f9d643d9700  5 -- op tracker -- , seq: 748410, time: 2014-06-12 14:47:21.467581, event: started, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:21.467599 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] sub_op_modify_reply: tid 236357 op  ack_type 1 from 26
2014-06-12 14:47:21.467613 7f9d643d9700  5 -- op tracker -- , seq: 748156, time: 2014-06-12 14:47:21.467608, event: sub_op_applied_rec, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.467618 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] repop_all_applied: repop tid 236357 all applied 
2014-06-12 14:47:21.467624 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] eval_repop repgather(0x15acfbc0 173'2536 rep_tid=236357 committed?=0 applied?=1 lock=0 op=osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.467633 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]  sending ack on repgather(0x15acfbc0 173'2536 rep_tid=236357 committed?=0 applied?=1 lock=0 op=osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) 0x2b2fca00
2014-06-12 14:47:21.467653 7f9d643d9700 10 osd.2 173 dequeue_op 0x28dd0900 finish
2014-06-12 14:47:21.467661 7f9d643d9700  5 -- op tracker -- , seq: 748410, time: 2014-06-12 14:47:21.467658, event: done, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:21.467662 7f9d63bd8700 10 osd.2 173 dequeue_op 0x28dd1c00 prio 196 cost 0 latency 0.000173 osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 pg pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean]
2014-06-12 14:47:21.467674 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:21.467684 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:21.467685 7f9d63bd8700  5 -- op tracker -- , seq: 748411, time: 2014-06-12 14:47:21.467682, event: reached_pg, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.467693 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] handle_message: 0x28dd1c00
2014-06-12 14:47:21.467703 7f9d63bd8700  5 -- op tracker -- , seq: 748411, time: 2014-06-12 14:47:21.467701, event: started, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.467706 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] sub_op_modify_reply: tid 236357 op  ack_type 4 from 26
2014-06-12 14:47:21.467717 7f9d63bd8700  5 -- op tracker -- , seq: 748156, time: 2014-06-12 14:47:21.467713, event: sub_op_commit_rec, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.467721 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2535 crt=0'0 lcod 173'2534 mlcod 173'2534 active+clean] repop_all_committed: repop tid 236357 all committed 
2014-06-12 14:47:21.467727 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2534 active+clean] eval_repop repgather(0x15acfbc0 173'2536 rep_tid=236357 committed?=1 applied?=1 lock=0 op=osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.467736 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2534 active+clean] log_op_stats osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 inb 143639 outb 0 rlat 26.700913 lat 26.700997
2014-06-12 14:47:21.467748 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2534 active+clean] publish_stats_to_osd 173:2895
2014-06-12 14:47:21.467756 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2534 active+clean]  sending commit on repgather(0x15acfbc0 173'2536 rep_tid=236357 committed?=1 applied?=1 lock=0 op=osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) 0x25910780
2014-06-12 14:47:21.467777 7f9d63bd8700  5 -- op tracker -- , seq: 748156, time: 2014-06-12 14:47:21.467774, event: commit_sent, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.467782 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2534 active+clean]  requeue_ops 
2014-06-12 14:47:21.467788 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2535 active+clean]  removing repgather(0x15acfbc0 173'2536 rep_tid=236357 committed?=1 applied?=1 op=osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.467796 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2535 active+clean]    q front is repgather(0x15acfbc0 173'2536 rep_tid=236357 committed?=1 applied?=1 op=osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.467804 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2535 active+clean] remove_repop repgather(0x15acfbc0 173'2536 rep_tid=236357 committed?=1 applied?=1 op=osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.467812 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2535 active+clean]  requeue_ops 
2014-06-12 14:47:21.467832 7f9d63bd8700  5 -- op tracker -- , seq: 748156, time: 2014-06-12 14:47:21.467828, event: done, op: osd_op(client.4300.0:140290 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~143360] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.467900 7f9d63bd8700 10 osd.2 173 dequeue_op 0x28dd1c00 finish
2014-06-12 14:47:21.467906 7f9d63bd8700  5 -- op tracker -- , seq: 748411, time: 2014-06-12 14:47:21.467904, event: done, op: osd_sub_op_reply(client.4300.0:140290 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.467913 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:21.467915 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:21.480640 7f9d56432700  5 -- op tracker -- , seq: 748412, time: 2014-06-12 14:47:21.480574, event: header_read, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ack, result = 0)
2014-06-12 14:47:21.480664 7f9d56432700  5 -- op tracker -- , seq: 748412, time: 2014-06-12 14:47:21.480577, event: throttled, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ack, result = 0)
2014-06-12 14:47:21.480682 7f9d56432700  5 -- op tracker -- , seq: 748412, time: 2014-06-12 14:47:21.480596, event: all_read, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ack, result = 0)
2014-06-12 14:47:21.480701 7f9d56432700  5 -- op tracker -- , seq: 748412, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ack, result = 0)
2014-06-12 14:47:21.480707 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:21.480714 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:21.480723 7f9d56432700 15 osd.2 173 enqueue_op 0x2fa9d700 prio 196 cost 0 latency 0.000149 osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ack, result = 0) v2
2014-06-12 14:47:21.480763 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:21.480777 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2fa9d700 prio 196 cost 0 latency 0.000202 osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ack, result = 0) v2 pg pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean]
2014-06-12 14:47:21.480800 7f9d56432700  5 -- op tracker -- , seq: 748413, time: 2014-06-12 14:47:21.480751, event: header_read, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.480823 7f9d56432700  5 -- op tracker -- , seq: 748413, time: 2014-06-12 14:47:21.480752, event: throttled, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.480840 7f9d56432700  5 -- op tracker -- , seq: 748413, time: 2014-06-12 14:47:21.480774, event: all_read, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.480846 7f9d56432700  5 -- op tracker -- , seq: 748413, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.480849 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:21.480852 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:21.480855 7f9d56432700 15 osd.2 173 enqueue_op 0x2fa9c700 prio 196 cost 0 latency 0.000104 osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:21.480852 7f9d61bd4700  5 -- op tracker -- , seq: 748412, time: 2014-06-12 14:47:21.480804, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ack, result = 0)
2014-06-12 14:47:21.480867 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean] handle_message: 0x2fa9d700
2014-06-12 14:47:21.480880 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:21.480885 7f9d61bd4700  5 -- op tracker -- , seq: 748412, time: 2014-06-12 14:47:21.480883, event: started, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ack, result = 0)
2014-06-12 14:47:21.480889 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean] sub_op_modify_reply: tid 236356 op  ack_type 1 from 26
2014-06-12 14:47:21.480909 7f9d61bd4700  5 -- op tracker -- , seq: 748144, time: 2014-06-12 14:47:21.480904, event: sub_op_applied_rec, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.480913 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean] repop_all_applied: repop tid 236356 all applied 
2014-06-12 14:47:21.480920 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean] eval_repop repgather(0x13b429c0 173'2514 rep_tid=236356 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.480929 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean]  sending ack on repgather(0x13b429c0 173'2514 rep_tid=236356 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) 0x959af00
2014-06-12 14:47:21.480947 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2fa9d700 finish
2014-06-12 14:47:21.480953 7f9d61bd4700  5 -- op tracker -- , seq: 748412, time: 2014-06-12 14:47:21.480951, event: done, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ack, result = 0)
2014-06-12 14:47:21.480974 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:21.480976 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:21.480985 7f9d633d7700 10 osd.2 173 dequeue_op 0x2fa9c700 prio 196 cost 0 latency 0.000233 osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2 pg pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean]
2014-06-12 14:47:21.481017 7f9d633d7700  5 -- op tracker -- , seq: 748413, time: 2014-06-12 14:47:21.481011, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.481046 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean] handle_message: 0x2fa9c700
2014-06-12 14:47:21.481070 7f9d633d7700  5 -- op tracker -- , seq: 748413, time: 2014-06-12 14:47:21.481064, event: started, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.481083 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean] sub_op_modify_reply: tid 236356 op  ack_type 4 from 26
2014-06-12 14:47:21.481123 7f9d633d7700  5 -- op tracker -- , seq: 748144, time: 2014-06-12 14:47:21.481112, event: sub_op_commit_rec, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.481135 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2513 crt=0'0 lcod 173'2512 mlcod 173'2512 active+clean] repop_all_committed: repop tid 236356 all committed 
2014-06-12 14:47:21.481152 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2512 active+clean] eval_repop repgather(0x13b429c0 173'2514 rep_tid=236356 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.481177 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2512 active+clean] log_op_stats osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 27.051208 lat 27.051437
2014-06-12 14:47:21.481200 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2512 active+clean] publish_stats_to_osd 173:2884
2014-06-12 14:47:21.481216 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2512 active+clean]  sending commit on repgather(0x13b429c0 173'2514 rep_tid=236356 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) 0x17be9180
2014-06-12 14:47:21.481262 7f9d633d7700  5 -- op tracker -- , seq: 748144, time: 2014-06-12 14:47:21.481255, event: commit_sent, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.481275 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2512 active+clean]  requeue_ops 
2014-06-12 14:47:21.481289 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean]  removing repgather(0x13b429c0 173'2514 rep_tid=236356 committed?=1 applied?=1 op=osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.481309 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean]    q front is repgather(0x13b429c0 173'2514 rep_tid=236356 committed?=1 applied?=1 op=osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.481326 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean] remove_repop repgather(0x13b429c0 173'2514 rep_tid=236356 committed?=1 applied?=1 op=osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.481350 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean]  requeue_ops 
2014-06-12 14:47:21.481396 7f9d633d7700  5 -- op tracker -- , seq: 748144, time: 2014-06-12 14:47:21.481384, event: done, op: osd_op(client.4290.0:143737 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 98304~1548288] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.481431 7f9d633d7700 10 osd.2 173 dequeue_op 0x2fa9c700 finish
2014-06-12 14:47:21.481446 7f9d633d7700  5 -- op tracker -- , seq: 748413, time: 2014-06-12 14:47:21.481441, event: done, op: osd_sub_op_reply(client.4290.0:143737 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.481464 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:21.481469 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:21.484006 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:21.484030 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:21.484036 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:21.484040 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:21.484044 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 16; oldest is 26.703982 seconds old
2014-06-12 14:47:21.484069 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:21.621487 7f9d56432700  5 -- op tracker -- , seq: 748414, time: 2014-06-12 14:47:21.621409, event: header_read, op: osd_sub_op_reply(client.4300.0:140291 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.621508 7f9d56432700  5 -- op tracker -- , seq: 748414, time: 2014-06-12 14:47:21.621412, event: throttled, op: osd_sub_op_reply(client.4300.0:140291 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.621519 7f9d56432700  5 -- op tracker -- , seq: 748414, time: 2014-06-12 14:47:21.621433, event: all_read, op: osd_sub_op_reply(client.4300.0:140291 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.621529 7f9d56432700  5 -- op tracker -- , seq: 748414, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4300.0:140291 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.621536 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4300.0:140291 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:21.621544 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:21.621554 7f9d56432700 15 osd.2 173 enqueue_op 0x2a903a00 prio 196 cost 0 latency 0.000145 osd_sub_op_reply(client.4300.0:140291 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:21.621599 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:21.621612 7f9d67be0700 10 osd.2 173 dequeue_op 0x2a903a00 prio 196 cost 0 latency 0.000202 osd_sub_op_reply(client.4300.0:140291 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 pg pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2535 active+clean]
2014-06-12 14:47:21.621675 7f9d67be0700  5 -- op tracker -- , seq: 748414, time: 2014-06-12 14:47:21.621670, event: reached_pg, op: osd_sub_op_reply(client.4300.0:140291 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.621694 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2535 active+clean] handle_message: 0x2a903a00
2014-06-12 14:47:21.621718 7f9d67be0700  5 -- op tracker -- , seq: 748414, time: 2014-06-12 14:47:21.621714, event: started, op: osd_sub_op_reply(client.4300.0:140291 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.621727 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2535 active+clean] sub_op_modify_reply: tid 236358 op  ack_type 4 from 26
2014-06-12 14:47:21.621755 7f9d67be0700  5 -- op tracker -- , seq: 748157, time: 2014-06-12 14:47:21.621743, event: sub_op_commit_rec, op: osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.621767 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2535 active+clean] repop_all_applied: repop tid 236358 all applied 
2014-06-12 14:47:21.621782 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2535 active+clean] eval_repop repgather(0x2335c840 173'2537 rep_tid=236358 committed?=0 applied?=1 lock=0 op=osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.621808 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2535 active+clean]  sending ack on repgather(0x2335c840 173'2537 rep_tid=236358 committed?=0 applied?=1 lock=0 op=osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) 0x1990c780
2014-06-12 14:47:21.621849 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2536 crt=0'0 lcod 173'2535 mlcod 173'2535 active+clean] repop_all_committed: repop tid 236358 all committed 
2014-06-12 14:47:21.621865 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2535 active+clean] eval_repop repgather(0x2335c840 173'2537 rep_tid=236358 committed?=1 applied?=1 lock=0 op=osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.621887 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2535 active+clean] log_op_stats osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 26.841786 lat 26.841825
2014-06-12 14:47:21.621909 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2535 active+clean] publish_stats_to_osd 173:2896
2014-06-12 14:47:21.621925 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2535 active+clean]  sending commit on repgather(0x2335c840 173'2537 rep_tid=236358 committed?=1 applied?=1 lock=0 op=osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) 0x30cca780
2014-06-12 14:47:21.621968 7f9d67be0700  5 -- op tracker -- , seq: 748157, time: 2014-06-12 14:47:21.621961, event: commit_sent, op: osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.621981 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2535 active+clean]  requeue_ops 
2014-06-12 14:47:21.621995 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2536 active+clean]  removing repgather(0x2335c840 173'2537 rep_tid=236358 committed?=1 applied?=1 op=osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.622036 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2536 active+clean]    q front is repgather(0x2335c840 173'2537 rep_tid=236358 committed?=1 applied?=1 op=osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.622057 7f9d67be0700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2536 active+clean] remove_repop repgather(0x2335c840 173'2537 rep_tid=236358 committed?=1 applied?=1 op=osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.622077 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2536 active+clean]  requeue_ops 
2014-06-12 14:47:21.622105 7f9d67be0700  5 -- op tracker -- , seq: 748157, time: 2014-06-12 14:47:21.622098, event: done, op: osd_op(client.4300.0:140291 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 143360~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.622133 7f9d67be0700 10 osd.2 173 dequeue_op 0x2a903a00 finish
2014-06-12 14:47:21.622146 7f9d67be0700  5 -- op tracker -- , seq: 748414, time: 2014-06-12 14:47:21.622141, event: done, op: osd_sub_op_reply(client.4300.0:140291 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.622160 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:21.622164 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:21.644673 7f9d56432700  5 -- op tracker -- , seq: 748415, time: 2014-06-12 14:47:21.644597, event: header_read, op: osd_sub_op_reply(client.4300.0:140292 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.644694 7f9d56432700  5 -- op tracker -- , seq: 748415, time: 2014-06-12 14:47:21.644601, event: throttled, op: osd_sub_op_reply(client.4300.0:140292 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.644724 7f9d56432700  5 -- op tracker -- , seq: 748415, time: 2014-06-12 14:47:21.644620, event: all_read, op: osd_sub_op_reply(client.4300.0:140292 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.644735 7f9d56432700  5 -- op tracker -- , seq: 748415, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4300.0:140292 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.644742 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4300.0:140292 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:21.644749 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:21.644759 7f9d56432700 15 osd.2 173 enqueue_op 0x2a903f00 prio 196 cost 0 latency 0.000162 osd_sub_op_reply(client.4300.0:140292 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:21.644806 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:21.644831 7f9d673df700 10 osd.2 173 dequeue_op 0x2a903f00 prio 196 cost 0 latency 0.000233 osd_sub_op_reply(client.4300.0:140292 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 pg pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2536 active+clean]
2014-06-12 14:47:21.644882 7f9d673df700  5 -- op tracker -- , seq: 748415, time: 2014-06-12 14:47:21.644873, event: reached_pg, op: osd_sub_op_reply(client.4300.0:140292 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.644903 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2536 active+clean] handle_message: 0x2a903f00
2014-06-12 14:47:21.644943 7f9d673df700  5 -- op tracker -- , seq: 748415, time: 2014-06-12 14:47:21.644935, event: started, op: osd_sub_op_reply(client.4300.0:140292 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.644960 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2536 active+clean] sub_op_modify_reply: tid 236360 op  ack_type 4 from 26
2014-06-12 14:47:21.645010 7f9d673df700  5 -- op tracker -- , seq: 748162, time: 2014-06-12 14:47:21.644993, event: sub_op_commit_rec, op: osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.645028 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2536 active+clean] repop_all_applied: repop tid 236360 all applied 
2014-06-12 14:47:21.645056 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2536 active+clean] eval_repop repgather(0x11c7f440 173'2538 rep_tid=236360 committed?=0 applied?=1 lock=0 op=osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.645098 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2536 active+clean]  sending ack on repgather(0x11c7f440 173'2538 rep_tid=236360 committed?=0 applied?=1 lock=0 op=osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) 0x151e6280
2014-06-12 14:47:21.645171 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2537 crt=0'0 lcod 173'2536 mlcod 173'2536 active+clean] repop_all_committed: repop tid 236360 all committed 
2014-06-12 14:47:21.645199 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2536 active+clean] eval_repop repgather(0x11c7f440 173'2538 rep_tid=236360 committed?=1 applied?=1 lock=0 op=osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.645239 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2536 active+clean] log_op_stats osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 inb 1548567 outb 0 rlat 26.732314 lat 26.732383
2014-06-12 14:47:21.645279 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2536 active+clean] publish_stats_to_osd 173:2897
2014-06-12 14:47:21.645315 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2536 active+clean]  sending commit on repgather(0x11c7f440 173'2538 rep_tid=236360 committed?=1 applied?=1 lock=0 op=osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) 0x2d88e280
2014-06-12 14:47:21.645415 7f9d673df700  5 -- op tracker -- , seq: 748162, time: 2014-06-12 14:47:21.645402, event: commit_sent, op: osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.645436 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2536 active+clean]  requeue_ops 
2014-06-12 14:47:21.645464 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2537 active+clean]  removing repgather(0x11c7f440 173'2538 rep_tid=236360 committed?=1 applied?=1 op=osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.645515 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2537 active+clean]    q front is repgather(0x11c7f440 173'2538 rep_tid=236360 committed?=1 applied?=1 op=osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.645551 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2537 active+clean] remove_repop repgather(0x11c7f440 173'2538 rep_tid=236360 committed?=1 applied?=1 op=osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.645587 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2537 active+clean]  requeue_ops 
2014-06-12 14:47:21.645660 7f9d673df700  5 -- op tracker -- , seq: 748162, time: 2014-06-12 14:47:21.645646, event: done, op: osd_op(client.4300.0:140292 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 659456~1548288] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.645702 7f9d673df700 10 osd.2 173 dequeue_op 0x2a903f00 finish
2014-06-12 14:47:21.645725 7f9d673df700  5 -- op tracker -- , seq: 748415, time: 2014-06-12 14:47:21.645716, event: done, op: osd_sub_op_reply(client.4300.0:140292 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.645754 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:21.645765 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:21.661490 7f9d56432700  5 -- op tracker -- , seq: 748416, time: 2014-06-12 14:47:21.661457, event: header_read, op: osd_sub_op_reply(client.4290.0:143738 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.661507 7f9d56432700  5 -- op tracker -- , seq: 748416, time: 2014-06-12 14:47:21.661458, event: throttled, op: osd_sub_op_reply(client.4290.0:143738 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.661513 7f9d56432700  5 -- op tracker -- , seq: 748416, time: 2014-06-12 14:47:21.661466, event: all_read, op: osd_sub_op_reply(client.4290.0:143738 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.661518 7f9d56432700  5 -- op tracker -- , seq: 748416, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143738 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.661521 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143738 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:21.661524 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:21.661529 7f9d56432700 15 osd.2 173 enqueue_op 0x3b42bf00 prio 196 cost 0 latency 0.000072 osd_sub_op_reply(client.4290.0:143738 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:21.661548 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:21.661556 7f9d663dd700 10 osd.2 173 dequeue_op 0x3b42bf00 prio 196 cost 0 latency 0.000098 osd_sub_op_reply(client.4290.0:143738 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2 pg pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean]
2014-06-12 14:47:21.661576 7f9d663dd700  5 -- op tracker -- , seq: 748416, time: 2014-06-12 14:47:21.661572, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143738 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.661589 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean] handle_message: 0x3b42bf00
2014-06-12 14:47:21.661602 7f9d663dd700  5 -- op tracker -- , seq: 748416, time: 2014-06-12 14:47:21.661599, event: started, op: osd_sub_op_reply(client.4290.0:143738 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.661611 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean] sub_op_modify_reply: tid 236359 op  ack_type 4 from 26
2014-06-12 14:47:21.661639 7f9d663dd700  5 -- op tracker -- , seq: 748161, time: 2014-06-12 14:47:21.661632, event: sub_op_commit_rec, op: osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.661645 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean] repop_all_applied: repop tid 236359 all applied 
2014-06-12 14:47:21.661665 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean] eval_repop repgather(0x6590cc0 173'2515 rep_tid=236359 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.661683 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean]  sending ack on repgather(0x6590cc0 173'2515 rep_tid=236359 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) 0x959b180
2014-06-12 14:47:21.661723 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2514 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean] repop_all_committed: repop tid 236359 all committed 
2014-06-12 14:47:21.661734 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2513 active+clean] eval_repop repgather(0x6590cc0 173'2515 rep_tid=236359 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.661750 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2513 active+clean] log_op_stats osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 26.751380 lat 26.751407
2014-06-12 14:47:21.661766 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2513 active+clean] publish_stats_to_osd 173:2885
2014-06-12 14:47:21.661778 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2513 active+clean]  sending commit on repgather(0x6590cc0 173'2515 rep_tid=236359 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) 0x22b24a00
2014-06-12 14:47:21.661800 7f9d663dd700  5 -- op tracker -- , seq: 748161, time: 2014-06-12 14:47:21.661797, event: commit_sent, op: osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.661805 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2513 active+clean]  requeue_ops 
2014-06-12 14:47:21.661812 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2514 active+clean]  removing repgather(0x6590cc0 173'2515 rep_tid=236359 committed?=1 applied?=1 op=osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.661828 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2514 active+clean]    q front is repgather(0x6590cc0 173'2515 rep_tid=236359 committed?=1 applied?=1 op=osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.661844 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2514 active+clean] remove_repop repgather(0x6590cc0 173'2515 rep_tid=236359 committed?=1 applied?=1 op=osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.661853 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2514 active+clean]  requeue_ops 
2014-06-12 14:47:21.661865 7f9d663dd700  5 -- op tracker -- , seq: 748161, time: 2014-06-12 14:47:21.661862, event: done, op: osd_op(client.4290.0:143738 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 1646592~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.661875 7f9d663dd700 10 osd.2 173 dequeue_op 0x3b42bf00 finish
2014-06-12 14:47:21.661879 7f9d663dd700  5 -- op tracker -- , seq: 748416, time: 2014-06-12 14:47:21.661877, event: done, op: osd_sub_op_reply(client.4290.0:143738 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.661885 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:21.661887 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:21.699611 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:21.711397 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:21.714892 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:21.723774 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:21.727262 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:21.738557 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:21.806653 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:21.810108 7f9d56432700  5 -- op tracker -- , seq: 748417, time: 2014-06-12 14:47:21.810061, event: header_read, op: osd_sub_op_reply(client.4300.0:140293 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.810125 7f9d56432700  5 -- op tracker -- , seq: 748417, time: 2014-06-12 14:47:21.810064, event: throttled, op: osd_sub_op_reply(client.4300.0:140293 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.810130 7f9d56432700  5 -- op tracker -- , seq: 748417, time: 2014-06-12 14:47:21.810074, event: all_read, op: osd_sub_op_reply(client.4300.0:140293 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.810135 7f9d56432700  5 -- op tracker -- , seq: 748417, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4300.0:140293 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.810147 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4300.0:140293 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:21.810151 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:21.810157 7f9d56432700 15 osd.2 173 enqueue_op 0x3b42a300 prio 196 cost 0 latency 0.000096 osd_sub_op_reply(client.4300.0:140293 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:21.810181 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:21.810197 7f9d65bdc700 10 osd.2 173 dequeue_op 0x3b42a300 prio 196 cost 0 latency 0.000136 osd_sub_op_reply(client.4300.0:140293 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 pg pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2537 active+clean]
2014-06-12 14:47:21.810218 7f9d65bdc700  5 -- op tracker -- , seq: 748417, time: 2014-06-12 14:47:21.810215, event: reached_pg, op: osd_sub_op_reply(client.4300.0:140293 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.810231 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2537 active+clean] handle_message: 0x3b42a300
2014-06-12 14:47:21.810246 7f9d65bdc700  5 -- op tracker -- , seq: 748417, time: 2014-06-12 14:47:21.810243, event: started, op: osd_sub_op_reply(client.4300.0:140293 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.810257 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2537 active+clean] sub_op_modify_reply: tid 236361 op  ack_type 4 from 26
2014-06-12 14:47:21.810282 7f9d65bdc700  5 -- op tracker -- , seq: 748163, time: 2014-06-12 14:47:21.810273, event: sub_op_commit_rec, op: osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.810290 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2537 active+clean] repop_all_applied: repop tid 236361 all applied 
2014-06-12 14:47:21.810302 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2537 active+clean] eval_repop repgather(0x24fa7680 173'2539 rep_tid=236361 committed?=0 applied?=1 lock=0 op=osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.810318 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2537 active+clean]  sending ack on repgather(0x24fa7680 173'2539 rep_tid=236361 committed?=0 applied?=1 lock=0 op=osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) 0x1ba79900
2014-06-12 14:47:21.810347 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2538 crt=0'0 lcod 173'2537 mlcod 173'2537 active+clean] repop_all_committed: repop tid 236361 all committed 
2014-06-12 14:47:21.810367 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2537 active+clean] eval_repop repgather(0x24fa7680 173'2539 rep_tid=236361 committed?=1 applied?=1 lock=0 op=osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.810385 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2537 active+clean] log_op_stats osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 26.892375 lat 26.892413
2014-06-12 14:47:21.810402 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2537 active+clean] publish_stats_to_osd 173:2898
2014-06-12 14:47:21.810424 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2537 active+clean]  sending commit on repgather(0x24fa7680 173'2539 rep_tid=236361 committed?=1 applied?=1 lock=0 op=osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) 0x30af7680
2014-06-12 14:47:21.810456 7f9d65bdc700  5 -- op tracker -- , seq: 748163, time: 2014-06-12 14:47:21.810451, event: commit_sent, op: osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.810471 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2537 active+clean]  requeue_ops 
2014-06-12 14:47:21.810482 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2538 active+clean]  removing repgather(0x24fa7680 173'2539 rep_tid=236361 committed?=1 applied?=1 op=osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.810496 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2538 active+clean]    q front is repgather(0x24fa7680 173'2539 rep_tid=236361 committed?=1 applied?=1 op=osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.810510 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2538 active+clean] remove_repop repgather(0x24fa7680 173'2539 rep_tid=236361 committed?=1 applied?=1 op=osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.810524 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2538 active+clean]  requeue_ops 
2014-06-12 14:47:21.810543 7f9d65bdc700  5 -- op tracker -- , seq: 748163, time: 2014-06-12 14:47:21.810538, event: done, op: osd_op(client.4300.0:140293 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.810565 7f9d65bdc700 10 osd.2 173 dequeue_op 0x3b42a300 finish
2014-06-12 14:47:21.810574 7f9d65bdc700  5 -- op tracker -- , seq: 748417, time: 2014-06-12 14:47:21.810571, event: done, op: osd_sub_op_reply(client.4300.0:140293 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.810594 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:21.810599 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:21.842492 7f9d56432700  5 -- op tracker -- , seq: 748418, time: 2014-06-12 14:47:21.842381, event: header_read, op: osd_sub_op_reply(client.4290.0:143739 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.842541 7f9d56432700  5 -- op tracker -- , seq: 748418, time: 2014-06-12 14:47:21.842385, event: throttled, op: osd_sub_op_reply(client.4290.0:143739 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.842559 7f9d56432700  5 -- op tracker -- , seq: 748418, time: 2014-06-12 14:47:21.842412, event: all_read, op: osd_sub_op_reply(client.4290.0:143739 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.842577 7f9d56432700  5 -- op tracker -- , seq: 748418, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143739 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.842587 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143739 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:21.842600 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:21.842615 7f9d56432700 15 osd.2 173 enqueue_op 0x3b42bd00 prio 196 cost 0 latency 0.000234 osd_sub_op_reply(client.4290.0:143739 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:21.842664 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:21.842681 7f9d62bd6700 10 osd.2 173 dequeue_op 0x3b42bd00 prio 196 cost 0 latency 0.000299 osd_sub_op_reply(client.4290.0:143739 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2 pg pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2514 active+clean]
2014-06-12 14:47:21.842713 7f9d62bd6700  5 -- op tracker -- , seq: 748418, time: 2014-06-12 14:47:21.842708, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143739 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.842727 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2514 active+clean] handle_message: 0x3b42bd00
2014-06-12 14:47:21.842748 7f9d62bd6700  5 -- op tracker -- , seq: 748418, time: 2014-06-12 14:47:21.842743, event: started, op: osd_sub_op_reply(client.4290.0:143739 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.842756 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2514 active+clean] sub_op_modify_reply: tid 236372 op  ack_type 4 from 26
2014-06-12 14:47:21.842793 7f9d62bd6700  5 -- op tracker -- , seq: 748227, time: 2014-06-12 14:47:21.842781, event: sub_op_commit_rec, op: osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.842806 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2514 active+clean] repop_all_applied: repop tid 236372 all applied 
2014-06-12 14:47:21.842820 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2514 active+clean] eval_repop repgather(0x1f067c80 173'2516 rep_tid=236372 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.842842 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2514 active+clean]  sending ack on repgather(0x1f067c80 173'2516 rep_tid=236372 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) 0x16cf2500
2014-06-12 14:47:21.842889 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2515 crt=0'0 lcod 173'2514 mlcod 173'2514 active+clean] repop_all_committed: repop tid 236372 all committed 
2014-06-12 14:47:21.842905 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2514 active+clean] eval_repop repgather(0x1f067c80 173'2516 rep_tid=236372 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:21.842927 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2514 active+clean] log_op_stats osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 21.325108 lat 21.325147
2014-06-12 14:47:21.842950 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2514 active+clean] publish_stats_to_osd 173:2886
2014-06-12 14:47:21.842973 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2514 active+clean]  sending commit on repgather(0x1f067c80 173'2516 rep_tid=236372 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) 0x45dec80
2014-06-12 14:47:21.843016 7f9d62bd6700  5 -- op tracker -- , seq: 748227, time: 2014-06-12 14:47:21.843008, event: commit_sent, op: osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.843026 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2514 active+clean]  requeue_ops 
2014-06-12 14:47:21.843039 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2515 active+clean]  removing repgather(0x1f067c80 173'2516 rep_tid=236372 committed?=1 applied?=1 op=osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.843058 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2515 active+clean]    q front is repgather(0x1f067c80 173'2516 rep_tid=236372 committed?=1 applied?=1 op=osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.843077 7f9d62bd6700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2515 active+clean] remove_repop repgather(0x1f067c80 173'2516 rep_tid=236372 committed?=1 applied?=1 op=osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:21.843101 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2515 active+clean]  requeue_ops 
2014-06-12 14:47:21.843127 7f9d62bd6700  5 -- op tracker -- , seq: 748227, time: 2014-06-12 14:47:21.843121, event: done, op: osd_op(client.4290.0:143739 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:21.843154 7f9d62bd6700 10 osd.2 173 dequeue_op 0x3b42bd00 finish
2014-06-12 14:47:21.843167 7f9d62bd6700  5 -- op tracker -- , seq: 748418, time: 2014-06-12 14:47:21.843162, event: done, op: osd_sub_op_reply(client.4290.0:143739 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:21.843197 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:21.843202 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:21.965821 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:21.965823 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:21.965825 7f9d74bfa700 20 osd.2 173 _dispatch 0x1621e340 ping v1
2014-06-12 14:47:21.965827 7f9d74bfa700 10 osd.2 173 ping from client.4316
2014-06-12 14:47:21.965829 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:21.965830 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:22.026403 7f9d56432700  5 -- op tracker -- , seq: 748419, time: 2014-06-12 14:47:22.026342, event: header_read, op: osd_sub_op_reply(client.4290.0:143740 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.026415 7f9d56432700  5 -- op tracker -- , seq: 748419, time: 2014-06-12 14:47:22.026345, event: throttled, op: osd_sub_op_reply(client.4290.0:143740 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.026424 7f9d56432700  5 -- op tracker -- , seq: 748419, time: 2014-06-12 14:47:22.026359, event: all_read, op: osd_sub_op_reply(client.4290.0:143740 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.026445 7f9d56432700  5 -- op tracker -- , seq: 748419, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143740 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.026450 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143740 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:22.026455 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:22.026461 7f9d56432700 15 osd.2 173 enqueue_op 0x17c2a300 prio 196 cost 0 latency 0.000119 osd_sub_op_reply(client.4290.0:143740 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:22.026489 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:22.026496 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x17c2a300 prio 196 cost 0 latency 0.000154 osd_sub_op_reply(client.4290.0:143740 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2 pg pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2515 active+clean]
2014-06-12 14:47:22.026522 7f9d6b3e7700  5 -- op tracker -- , seq: 748419, time: 2014-06-12 14:47:22.026512, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143740 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.026538 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2515 active+clean] handle_message: 0x17c2a300
2014-06-12 14:47:22.026548 7f9d6b3e7700  5 -- op tracker -- , seq: 748419, time: 2014-06-12 14:47:22.026546, event: started, op: osd_sub_op_reply(client.4290.0:143740 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.026552 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2515 active+clean] sub_op_modify_reply: tid 236373 op  ack_type 4 from 26
2014-06-12 14:47:22.026567 7f9d6b3e7700  5 -- op tracker -- , seq: 748228, time: 2014-06-12 14:47:22.026560, event: sub_op_commit_rec, op: osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:22.026582 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2515 active+clean] repop_all_applied: repop tid 236373 all applied 
2014-06-12 14:47:22.026590 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2515 active+clean] eval_repop repgather(0x51dcb40 173'2517 rep_tid=236373 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:22.026601 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2515 active+clean]  sending ack on repgather(0x51dcb40 173'2517 rep_tid=236373 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) 0x6989680
2014-06-12 14:47:22.026622 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2516 crt=0'0 lcod 173'2515 mlcod 173'2515 active+clean] repop_all_committed: repop tid 236373 all committed 
2014-06-12 14:47:22.026633 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2515 active+clean] eval_repop repgather(0x51dcb40 173'2517 rep_tid=236373 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:22.026652 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2515 active+clean] log_op_stats osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 21.506299 lat 21.506330
2014-06-12 14:47:22.026666 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2515 active+clean] publish_stats_to_osd 173:2887
2014-06-12 14:47:22.026674 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2515 active+clean]  sending commit on repgather(0x51dcb40 173'2517 rep_tid=236373 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) 0x45df400
2014-06-12 14:47:22.026698 7f9d6b3e7700  5 -- op tracker -- , seq: 748228, time: 2014-06-12 14:47:22.026693, event: commit_sent, op: osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:22.026709 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2515 active+clean]  requeue_ops 
2014-06-12 14:47:22.026719 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2516 active+clean]  removing repgather(0x51dcb40 173'2517 rep_tid=236373 committed?=1 applied?=1 op=osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:22.026729 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2516 active+clean]    q front is repgather(0x51dcb40 173'2517 rep_tid=236373 committed?=1 applied?=1 op=osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:22.026739 7f9d6b3e7700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2516 active+clean] remove_repop repgather(0x51dcb40 173'2517 rep_tid=236373 committed?=1 applied?=1 op=osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:22.026747 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2516 active+clean]  requeue_ops 
2014-06-12 14:47:22.026760 7f9d6b3e7700  5 -- op tracker -- , seq: 748228, time: 2014-06-12 14:47:22.026757, event: done, op: osd_op(client.4290.0:143740 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~1032192] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:22.026791 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x17c2a300 finish
2014-06-12 14:47:22.026797 7f9d6b3e7700  5 -- op tracker -- , seq: 748419, time: 2014-06-12 14:47:22.026795, event: done, op: osd_sub_op_reply(client.4290.0:143740 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.026804 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:22.026806 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:22.146200 7f9d56432700  5 -- op tracker -- , seq: 748420, time: 2014-06-12 14:47:22.146148, event: header_read, op: osd_sub_op_reply(client.4290.0:143741 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.146215 7f9d56432700  5 -- op tracker -- , seq: 748420, time: 2014-06-12 14:47:22.146150, event: throttled, op: osd_sub_op_reply(client.4290.0:143741 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.146225 7f9d56432700  5 -- op tracker -- , seq: 748420, time: 2014-06-12 14:47:22.146164, event: all_read, op: osd_sub_op_reply(client.4290.0:143741 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.146242 7f9d56432700  5 -- op tracker -- , seq: 748420, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4290.0:143741 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.146247 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4290.0:143741 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:22.146258 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:22.146274 7f9d56432700 15 osd.2 173 enqueue_op 0x17c2b400 prio 196 cost 0 latency 0.000126 osd_sub_op_reply(client.4290.0:143741 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:22.146337 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:22.146348 7f9d693e3700 10 osd.2 173 dequeue_op 0x17c2b400 prio 196 cost 0 latency 0.000200 osd_sub_op_reply(client.4290.0:143741 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0) v2 pg pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2516 active+clean]
2014-06-12 14:47:22.146368 7f9d693e3700  5 -- op tracker -- , seq: 748420, time: 2014-06-12 14:47:22.146365, event: reached_pg, op: osd_sub_op_reply(client.4290.0:143741 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.146376 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2516 active+clean] handle_message: 0x17c2b400
2014-06-12 14:47:22.146397 7f9d693e3700  5 -- op tracker -- , seq: 748420, time: 2014-06-12 14:47:22.146394, event: started, op: osd_sub_op_reply(client.4290.0:143741 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.146402 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2516 active+clean] sub_op_modify_reply: tid 236374 op  ack_type 4 from 26
2014-06-12 14:47:22.146423 7f9d693e3700  5 -- op tracker -- , seq: 748229, time: 2014-06-12 14:47:22.146409, event: sub_op_commit_rec, op: osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:22.146429 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2516 active+clean] repop_all_applied: repop tid 236374 all applied 
2014-06-12 14:47:22.146436 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2516 active+clean] eval_repop repgather(0xbe0d740 173'2518 rep_tid=236374 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:22.146447 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2516 active+clean]  sending ack on repgather(0xbe0d740 173'2518 rep_tid=236374 committed?=0 applied?=1 lock=0 op=osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) 0x15cd6c80
2014-06-12 14:47:22.146477 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2517 crt=0'0 lcod 173'2516 mlcod 173'2516 active+clean] repop_all_committed: repop tid 236374 all committed 
2014-06-12 14:47:22.146484 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2516 active+clean] eval_repop repgather(0xbe0d740 173'2518 rep_tid=236374 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:22.146495 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2516 active+clean] log_op_stats osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4 inb 483607 outb 0 rlat 21.600175 lat 21.600193
2014-06-12 14:47:22.146505 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2516 active+clean] publish_stats_to_osd 173:2888
2014-06-12 14:47:22.146513 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2516 active+clean]  sending commit on repgather(0xbe0d740 173'2518 rep_tid=236374 committed?=1 applied?=1 lock=0 op=osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4) 0x1687b180
2014-06-12 14:47:22.146532 7f9d693e3700  5 -- op tracker -- , seq: 748229, time: 2014-06-12 14:47:22.146529, event: commit_sent, op: osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:22.146538 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2516 active+clean]  requeue_ops 
2014-06-12 14:47:22.146545 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2517 active+clean]  removing repgather(0xbe0d740 173'2518 rep_tid=236374 committed?=1 applied?=1 op=osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:22.146553 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2517 active+clean]    q front is repgather(0xbe0d740 173'2518 rep_tid=236374 committed?=1 applied?=1 op=osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:22.146561 7f9d693e3700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2517 active+clean] remove_repop repgather(0xbe0d740 173'2518 rep_tid=236374 committed?=1 applied?=1 op=osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:22.146570 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2517 active+clean]  requeue_ops 
2014-06-12 14:47:22.146588 7f9d693e3700  5 -- op tracker -- , seq: 748229, time: 2014-06-12 14:47:22.146585, event: done, op: osd_op(client.4290.0:143741 rbd_data.10996b8b4567.0000000000001034 [set-alloc-hint object_size 4194304 write_size 4194304,write 3710976~483328] 3.d70cb7ae ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:22.146603 7f9d693e3700 10 osd.2 173 dequeue_op 0x17c2b400 finish
2014-06-12 14:47:22.146609 7f9d693e3700  5 -- op tracker -- , seq: 748420, time: 2014-06-12 14:47:22.146607, event: done, op: osd_sub_op_reply(client.4290.0:143741 3.7ae d70cb7ae/rbd_data.10996b8b4567.0000000000001034/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.146615 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:22.146617 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:22.148823 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:22.156336 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:22.157301 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:22.164280 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:22.170568 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:22.176068 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:22.176097 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:22.178998 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:22.182760 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:22.186711 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:22.199396 7f9d56432700  5 -- op tracker -- , seq: 748421, time: 2014-06-12 14:47:22.199330, event: header_read, op: osd_sub_op_reply(client.4300.0:140294 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.199414 7f9d56432700  5 -- op tracker -- , seq: 748421, time: 2014-06-12 14:47:22.199333, event: throttled, op: osd_sub_op_reply(client.4300.0:140294 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.199424 7f9d56432700  5 -- op tracker -- , seq: 748421, time: 2014-06-12 14:47:22.199351, event: all_read, op: osd_sub_op_reply(client.4300.0:140294 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.199445 7f9d56432700  5 -- op tracker -- , seq: 748421, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4300.0:140294 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.199451 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4300.0:140294 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:22.199458 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:22.199467 7f9d56432700 15 osd.2 173 enqueue_op 0x17c2ba00 prio 196 cost 0 latency 0.000137 osd_sub_op_reply(client.4300.0:140294 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:22.199535 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:22.199549 7f9d683e1700 10 osd.2 173 dequeue_op 0x17c2ba00 prio 196 cost 0 latency 0.000219 osd_sub_op_reply(client.4300.0:140294 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 pg pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2538 active+clean]
2014-06-12 14:47:22.199592 7f9d683e1700  5 -- op tracker -- , seq: 748421, time: 2014-06-12 14:47:22.199586, event: reached_pg, op: osd_sub_op_reply(client.4300.0:140294 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.199614 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2538 active+clean] handle_message: 0x17c2ba00
2014-06-12 14:47:22.199639 7f9d683e1700  5 -- op tracker -- , seq: 748421, time: 2014-06-12 14:47:22.199634, event: started, op: osd_sub_op_reply(client.4300.0:140294 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.199647 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2538 active+clean] sub_op_modify_reply: tid 236376 op  ack_type 4 from 26
2014-06-12 14:47:22.199678 7f9d683e1700  5 -- op tracker -- , seq: 748231, time: 2014-06-12 14:47:22.199668, event: sub_op_commit_rec, op: osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:22.199690 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2538 active+clean] repop_all_applied: repop tid 236376 all applied 
2014-06-12 14:47:22.199704 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2538 active+clean] eval_repop repgather(0x53918c0 173'2540 rep_tid=236376 committed?=0 applied?=1 lock=0 op=osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:22.199727 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2538 active+clean]  sending ack on repgather(0x53918c0 173'2540 rep_tid=236376 committed?=0 applied?=1 lock=0 op=osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) 0x5ffea00
2014-06-12 14:47:22.199768 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2539 crt=0'0 lcod 173'2538 mlcod 173'2538 active+clean] repop_all_committed: repop tid 236376 all committed 
2014-06-12 14:47:22.199782 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2538 active+clean] eval_repop repgather(0x53918c0 173'2540 rep_tid=236376 committed?=1 applied?=1 lock=0 op=osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:22.199804 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2538 active+clean] log_op_stats osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 21.433472 lat 21.433507
2014-06-12 14:47:22.199827 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2538 active+clean] publish_stats_to_osd 173:2899
2014-06-12 14:47:22.199843 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2538 active+clean]  sending commit on repgather(0x53918c0 173'2540 rep_tid=236376 committed?=1 applied?=1 lock=0 op=osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) 0x3371c500
2014-06-12 14:47:22.199886 7f9d683e1700  5 -- op tracker -- , seq: 748231, time: 2014-06-12 14:47:22.199880, event: commit_sent, op: osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:22.199896 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2538 active+clean]  requeue_ops 
2014-06-12 14:47:22.199910 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean]  removing repgather(0x53918c0 173'2540 rep_tid=236376 committed?=1 applied?=1 op=osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:22.199930 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean]    q front is repgather(0x53918c0 173'2540 rep_tid=236376 committed?=1 applied?=1 op=osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:22.199948 7f9d683e1700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean] remove_repop repgather(0x53918c0 173'2540 rep_tid=236376 committed?=1 applied?=1 op=osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:22.199969 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean]  requeue_ops 
2014-06-12 14:47:22.199993 7f9d683e1700  5 -- op tracker -- , seq: 748231, time: 2014-06-12 14:47:22.199987, event: done, op: osd_op(client.4300.0:140294 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1032192] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:22.200021 7f9d683e1700 10 osd.2 173 dequeue_op 0x17c2ba00 finish
2014-06-12 14:47:22.200034 7f9d683e1700  5 -- op tracker -- , seq: 748421, time: 2014-06-12 14:47:22.200029, event: done, op: osd_sub_op_reply(client.4300.0:140294 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.200047 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:22.200052 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:22.206846 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:22.337130 7f9d56432700  5 -- op tracker -- , seq: 748422, time: 2014-06-12 14:47:22.337043, event: header_read, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:22.337149 7f9d56432700  5 -- op tracker -- , seq: 748422, time: 2014-06-12 14:47:22.337047, event: throttled, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:22.337159 7f9d56432700  5 -- op tracker -- , seq: 748422, time: 2014-06-12 14:47:22.337070, event: all_read, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:22.337190 7f9d56432700  5 -- op tracker -- , seq: 748422, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:22.337198 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:22.337206 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:22.337217 7f9d56432700 15 osd.2 173 enqueue_op 0x17c2a900 prio 196 cost 0 latency 0.000173 osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0) v2
2014-06-12 14:47:22.337287 7f9d56432700  5 -- op tracker -- , seq: 748423, time: 2014-06-12 14:47:22.337249, event: header_read, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.337293 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:22.337306 7f9d56432700  5 -- op tracker -- , seq: 748423, time: 2014-06-12 14:47:22.337249, event: throttled, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.337316 7f9d56432700  5 -- op tracker -- , seq: 748423, time: 2014-06-12 14:47:22.337261, event: all_read, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.337325 7f9d56432700  5 -- op tracker -- , seq: 748423, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.337330 7f9d56432700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:22.337337 7f9d56432700 20 osd.2 173 should_share_map osd.26 172.16.96.202:6845/3704 173
2014-06-12 14:47:22.337343 7f9d56432700 15 osd.2 173 enqueue_op 0x17c2b800 prio 196 cost 0 latency 0.000094 osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:22.337321 7f9d6abe6700 10 osd.2 173 dequeue_op 0x17c2a900 prio 196 cost 0 latency 0.000278 osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0) v2 pg pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean]
2014-06-12 14:47:22.337360 7f9d6abe6700  5 -- op tracker -- , seq: 748422, time: 2014-06-12 14:47:22.337354, event: reached_pg, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:22.337378 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:22.337388 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean] handle_message: 0x17c2a900
2014-06-12 14:47:22.337418 7f9d6abe6700  5 -- op tracker -- , seq: 748422, time: 2014-06-12 14:47:22.337413, event: started, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:22.337442 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean] sub_op_modify_reply: tid 236380 op  ack_type 1 from 26
2014-06-12 14:47:22.337471 7f9d6abe6700  5 -- op tracker -- , seq: 748254, time: 2014-06-12 14:47:22.337459, event: sub_op_applied_rec, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:22.337481 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean] repop_all_applied: repop tid 236380 all applied 
2014-06-12 14:47:22.337495 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean] eval_repop repgather(0x3403a9c0 173'2541 rep_tid=236380 committed?=0 applied?=1 lock=0 op=osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:22.337517 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean]  sending ack on repgather(0x3403a9c0 173'2541 rep_tid=236380 committed?=0 applied?=1 lock=0 op=osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) 0x33771b80
2014-06-12 14:47:22.337560 7f9d6abe6700 10 osd.2 173 dequeue_op 0x17c2a900 finish
2014-06-12 14:47:22.337575 7f9d6abe6700  5 -- op tracker -- , seq: 748422, time: 2014-06-12 14:47:22.337567, event: done, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ack, result = 0)
2014-06-12 14:47:22.337576 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x17c2b800 prio 196 cost 0 latency 0.000327 osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0) v2 pg pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean]
2014-06-12 14:47:22.337609 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:22.337629 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:22.337629 7f9d6a3e5700  5 -- op tracker -- , seq: 748423, time: 2014-06-12 14:47:22.337609, event: reached_pg, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.337646 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean] handle_message: 0x17c2b800
2014-06-12 14:47:22.337676 7f9d6a3e5700  5 -- op tracker -- , seq: 748423, time: 2014-06-12 14:47:22.337671, event: started, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.337691 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean] sub_op_modify_reply: tid 236380 op  ack_type 4 from 26
2014-06-12 14:47:22.337715 7f9d6a3e5700  5 -- op tracker -- , seq: 748254, time: 2014-06-12 14:47:22.337707, event: sub_op_commit_rec, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:22.337731 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 luod=173'2540 crt=0'0 lcod 173'2539 mlcod 173'2539 active+clean] repop_all_committed: repop tid 236380 all committed 
2014-06-12 14:47:22.337737 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2539 active+clean] eval_repop repgather(0x3403a9c0 173'2541 rep_tid=236380 committed?=1 applied?=1 lock=0 op=osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:22.337747 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2539 active+clean] log_op_stats osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4 inb 438551 outb 0 rlat 21.020978 lat 21.021167
2014-06-12 14:47:22.337758 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2539 active+clean] publish_stats_to_osd 173:2900
2014-06-12 14:47:22.337765 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2539 active+clean]  sending commit on repgather(0x3403a9c0 173'2541 rep_tid=236380 committed?=1 applied?=1 lock=0 op=osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4) 0x1a58d400
2014-06-12 14:47:22.337785 7f9d6a3e5700  5 -- op tracker -- , seq: 748254, time: 2014-06-12 14:47:22.337782, event: commit_sent, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:22.337790 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2539 active+clean]  requeue_ops 
2014-06-12 14:47:22.337805 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2540 active+clean]  removing repgather(0x3403a9c0 173'2541 rep_tid=236380 committed?=1 applied?=1 op=osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:22.337814 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2540 active+clean]    q front is repgather(0x3403a9c0 173'2541 rep_tid=236380 committed?=1 applied?=1 op=osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:22.337822 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2540 active+clean] remove_repop repgather(0x3403a9c0 173'2541 rep_tid=236380 committed?=1 applied?=1 op=osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:22.337830 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2540 active+clean]  requeue_ops 
2014-06-12 14:47:22.337857 7f9d6a3e5700  5 -- op tracker -- , seq: 748254, time: 2014-06-12 14:47:22.337854, event: done, op: osd_op(client.4300.0:140295 rbd_data.10a36b8b4567.0000000000000ee2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3756032~438272] 3.c4bba9e1 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:22.337868 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x17c2b800 finish
2014-06-12 14:47:22.337873 7f9d6a3e5700  5 -- op tracker -- , seq: 748423, time: 2014-06-12 14:47:22.337871, event: done, op: osd_sub_op_reply(client.4300.0:140295 3.1e1 c4bba9e1/rbd_data.10a36b8b4567.0000000000000ee2/head//3 [] ondisk, result = 0)
2014-06-12 14:47:22.337879 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:22.337881 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:22.370777 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:22.393292 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:22.484125 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:22.484152 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:47:22.484156 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:47:22.484160 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:47:22.484165 7f9d84175700 10 osd.2 173 send_pg_stats - 2 pgs updated
2014-06-12 14:47:22.484210 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:22.484214 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:22.484215 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:22.484218 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 7; oldest is 13.313442 seconds old
2014-06-12 14:47:22.484218 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:22.508619 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:22.508748 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:22.631868 7f9d5a16f700  5 -- op tracker -- , seq: 748424, time: 2014-06-12 14:47:22.631053, event: header_read, op: osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.631897 7f9d5a16f700  5 -- op tracker -- , seq: 748424, time: 2014-06-12 14:47:22.631056, event: throttled, op: osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.631920 7f9d5a16f700  5 -- op tracker -- , seq: 748424, time: 2014-06-12 14:47:22.631816, event: all_read, op: osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.631926 7f9d5a16f700  5 -- op tracker -- , seq: 748424, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.631931 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:22.631936 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:22.631943 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903300 prio 127 cost 516903 latency 0.000890 osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:22.631972 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:22.631991 7f9d66bde700 10 osd.2 173 dequeue_op 0x2a903300 prio 127 cost 516903 latency 0.000938 osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2872 (0'0,173'2872] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2871 active]
2014-06-12 14:47:22.632024 7f9d66bde700  5 -- op tracker -- , seq: 748424, time: 2014-06-12 14:47:22.632013, event: reached_pg, op: osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.632034 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2872 (0'0,173'2872] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2871 active] handle_message: 0x2a903300
2014-06-12 14:47:22.632047 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2872 (0'0,173'2872] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2871 active] sub_op_modify trans f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 v 173'2873 (transaction) 174
2014-06-12 14:47:22.632064 7f9d66bde700  5 -- op tracker -- , seq: 748424, time: 2014-06-12 14:47:22.632060, event: started, op: osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.632084 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2872 (0'0,173'2872] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2871 active] append_log log((0'0,173'2872], crt=0'0) [173'2873 (173'2872) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145705 2014-06-12 08:43:55.289940]
2014-06-12 14:47:22.632120 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2873 (0'0,173'2873] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2871 active] add_log_entry 173'2873 (173'2872) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145705 2014-06-12 08:43:55.289940
2014-06-12 14:47:22.632151 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2873 (0'0,173'2873] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2871 active] append_log  adding 1 keys
2014-06-12 14:47:22.632216 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2873, trimmed: 
2014-06-12 14:47:22.632252 7f9d66bde700  5 -- op tracker -- , seq: 748424, time: 2014-06-12 14:47:22.632247, event: started, op: osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.632265 7f9d66bde700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:22.632270 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:22.632272 7f9d66bde700  5 queue_op 0x7ac9e00 seq 2586 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:22.632278 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2873 (0'0,173'2873] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2871 active] do_sub_op osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:22.632293 7f9d66bde700 10 osd.2 173 dequeue_op 0x2a903300 finish
2014-06-12 14:47:22.632296 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:22.632299 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:22.632304 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:22.632312 7f9d8096e700  5 _do_op 0x7ac9e00 seq 2586 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:22.632316 7f9d8096e700 10 _do_transaction on 0x1451d0b0
2014-06-12 14:47:22.632329 7f9d8096e700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:22.632435 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:22.632454 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:22.632486 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:22.632510 7f9d8096e700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:22.632516 7f9d8096e700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:22.632520 7f9d8096e700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:22.632667 7f9d8096e700 15 _write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 3022848~516096
2014-06-12 14:47:22.632787 7f9d8096e700 10 lookup_strip_header done  cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:22.632935 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:47:22.634306 7f9d8096e700 10 _generic_write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 3022848~516096 = 0
2014-06-12 14:47:22.634342 7f9d8096e700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:22.634351 7f9d8096e700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:22.634355 7f9d8096e700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:22.634360 7f9d8096e700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:22.634495 7f9d8096e700 20 set_header setting 16959 cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 parent seq 0
2014-06-12 14:47:22.634543 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:22.634554 7f9d8096e700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:22.634560 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:47:22.636997 7f9d8096e700 10 _do_op 0x7ac9e00 seq 2586 r = 0, finisher 0x738a260 0
2014-06-12 14:47:22.637019 7f9d8096e700 10 _finish_op 0x7ac9e00 seq 2586 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:22.637025 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:22.637029 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:22.637038 7f9d7715b700  5 -- op tracker -- , seq: 748424, time: 2014-06-12 14:47:22.637028, event: commit_sent, op: osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.637049 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2873 (0'0,173'2873] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2871 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:22.637075 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:22.637110 7f9d7795c700  5 -- op tracker -- , seq: 748424, time: 2014-06-12 14:47:22.637104, event: sub_op_applied, op: osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.637117 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2873 (0'0,173'2873] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2872 active] sub_op_modify_applied on 0x1451cfc0 op osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:22.637137 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2873 (0'0,173'2873] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2872 active] op_applied on primary on version 173'2873
2014-06-12 14:47:22.637148 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2873 (0'0,173'2873] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2872 active] op_applied on replica on version 173'2873
2014-06-12 14:47:22.637173 7f9d7795c700  5 -- op tracker -- , seq: 748424, time: 2014-06-12 14:47:22.637169, event: done, op: osd_sub_op(client.4302.0:145705 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2873 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.763344 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:22.809925 7f9d5a16f700  5 -- op tracker -- , seq: 748425, time: 2014-06-12 14:47:22.806390, event: header_read, op: osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.809947 7f9d5a16f700  5 -- op tracker -- , seq: 748425, time: 2014-06-12 14:47:22.806393, event: throttled, op: osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.809958 7f9d5a16f700  5 -- op tracker -- , seq: 748425, time: 2014-06-12 14:47:22.809857, event: all_read, op: osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.809968 7f9d5a16f700  5 -- op tracker -- , seq: 748425, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.809976 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:22.809985 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:22.809996 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a902100 prio 127 cost 1536807 latency 0.003606 osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:22.810068 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:22.810084 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a902100 prio 127 cost 1536807 latency 0.003694 osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.31d( v 173'2480 (0'0,173'2480] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active]
2014-06-12 14:47:22.810135 7f9d64bda700  5 -- op tracker -- , seq: 748425, time: 2014-06-12 14:47:22.810128, event: reached_pg, op: osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.810159 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2480 (0'0,173'2480] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] handle_message: 0x2a902100
2014-06-12 14:47:22.810183 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2480 (0'0,173'2480] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] sub_op_modify trans 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 v 173'2481 (transaction) 174
2014-06-12 14:47:22.810206 7f9d64bda700  5 -- op tracker -- , seq: 748425, time: 2014-06-12 14:47:22.810200, event: started, op: osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.810229 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2480 (0'0,173'2480] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] append_log log((0'0,173'2480], crt=0'0) [173'2481 (173'638) modify   5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 by client.4315.0:144549 2014-06-12 08:44:41.639912]
2014-06-12 14:47:22.810265 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2481 (0'0,173'2481] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] add_log_entry 173'2481 (173'638) modify   5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 by client.4315.0:144549 2014-06-12 08:44:41.639912
2014-06-12 14:47:22.810296 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2481 (0'0,173'2481] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] append_log  adding 1 keys
2014-06-12 14:47:22.810340 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2481, trimmed: 
2014-06-12 14:47:22.810373 7f9d64bda700  5 -- op tracker -- , seq: 748425, time: 2014-06-12 14:47:22.810366, event: started, op: osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.810389 7f9d64bda700  5 queue_transactions existing osr(3.31d 0x276d520)/0x276d520
2014-06-12 14:47:22.810395 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:22.810397 7f9d64bda700  5 queue_op 0x3b18d80 seq 2299 osr(3.31d 0x276d520) 1538293 bytes   (queue has 1 ops and 1538293 bytes)
2014-06-12 14:47:22.810407 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.31d( v 173'2481 (0'0,173'2481] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] do_sub_op osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:22.810427 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a902100 finish
2014-06-12 14:47:22.810432 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:22.810436 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:22.810443 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439440 (1 active)
2014-06-12 14:47:22.810459 7f9d7c966700  5 _do_op 0x3b18d80 seq 2299 osr(3.31d 0x276d520)/0x276d520 start
2014-06-12 14:47:22.810466 7f9d7c966700 10 _do_transaction on 0x2a29a0f0
2014-06-12 14:47:22.810476 7f9d7c966700 15 _omap_setkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:22.810616 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:22.810656 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:22.810711 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:22.810752 7f9d7c966700 15 _omap_rmkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:22.810763 7f9d7c966700 10 _omap_rmkeys meta/1fe1ce3b/pglog_3.31d/0//-1 r = 0
2014-06-12 14:47:22.810771 7f9d7c966700 15 _omap_setkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:22.810956 7f9d7c966700 15 _write 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 0~1536000
2014-06-12 14:47:22.811189 7f9d7c966700 10 lookup_strip_header done  cid 3.31d_head oid 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:22.811384 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:47:22.818181 7f9d7c966700 10 _generic_write 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 0~1536000 = 0
2014-06-12 14:47:22.818276 7f9d7c966700 15 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:22.818298 7f9d7c966700 10 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 = 0
2014-06-12 14:47:22.818309 7f9d7c966700 15 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:22.818314 7f9d7c966700 10 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 = 0
2014-06-12 14:47:22.818556 7f9d7c966700 20 set_header setting 17151 cid 3.31d_head oid 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 parent seq 0
2014-06-12 14:47:22.818658 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:22.818669 7f9d7c966700 20 set_header setting 2268 cid meta oid 1fe1ce3b/pglog_3.31d/0//-1 parent seq 0
2014-06-12 14:47:22.818683 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:47:22.820720 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:22.820842 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:22.824430 7f9d7c966700 10 _do_op 0x3b18d80 seq 2299 r = 0, finisher 0x7330540 0
2014-06-12 14:47:22.824455 7f9d7c966700 10 _finish_op 0x3b18d80 seq 2299 osr(3.31d 0x276d520)/0x276d520
2014-06-12 14:47:22.824460 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439440 (0 active)
2014-06-12 14:47:22.824474 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:22.824515 7f9d7795c700  5 -- op tracker -- , seq: 748425, time: 2014-06-12 14:47:22.824496, event: sub_op_applied, op: osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.824525 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2481 (0'0,173'2481] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] sub_op_modify_applied on 0x2a29a000 op osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:22.824551 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:22.824569 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2481 (0'0,173'2481] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] op_applied on primary on version 173'2481
2014-06-12 14:47:22.824582 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2481 (0'0,173'2481] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] op_applied on replica on version 173'2481
2014-06-12 14:47:22.824632 7f9d7715b700  5 -- op tracker -- , seq: 748425, time: 2014-06-12 14:47:22.824626, event: commit_sent, op: osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.824644 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2481 (0'0,173'2481] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:22.824660 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:22.824692 7f9d7715b700  5 -- op tracker -- , seq: 748425, time: 2014-06-12 14:47:22.824688, event: done, op: osd_sub_op(client.4315.0:144549 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:22.840793 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:22.840801 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:22.840803 7f9d74bfa700 20 osd.2 173 _dispatch 0x2bc85a40 pg_stats_ack(2 pgs tid 4862) v1
2014-06-12 14:47:22.840805 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:47:22.840810 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:22.840812 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:22.943723 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:22.953453 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:22.953636 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:23.001692 7f9d5a16f700  5 -- op tracker -- , seq: 748426, time: 2014-06-12 14:47:22.999989, event: header_read, op: osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.001714 7f9d5a16f700  5 -- op tracker -- , seq: 748426, time: 2014-06-12 14:47:22.999991, event: throttled, op: osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.001737 7f9d5a16f700  5 -- op tracker -- , seq: 748426, time: 2014-06-12 14:47:23.001635, event: all_read, op: osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.001750 7f9d5a16f700  5 -- op tracker -- , seq: 748426, time: 0.000000, event: dispatched, op: osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.001755 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:23.001760 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:23.001767 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903d00 prio 127 cost 1147687 latency 0.001777 osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.001793 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:23.001810 7f9d623d5700 10 osd.2 173 dequeue_op 0x2a903d00 prio 127 cost 1147687 latency 0.001821 osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2675 (0'0,173'2675] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2674 active]
2014-06-12 14:47:23.001835 7f9d623d5700  5 -- op tracker -- , seq: 748426, time: 2014-06-12 14:47:23.001832, event: reached_pg, op: osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.001842 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2675 (0'0,173'2675] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2674 active] handle_message: 0x2a903d00
2014-06-12 14:47:23.001851 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2675 (0'0,173'2675] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2674 active] sub_op_modify trans c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 v 173'2676 (transaction) 174
2014-06-12 14:47:23.001863 7f9d623d5700  5 -- op tracker -- , seq: 748426, time: 2014-06-12 14:47:23.001860, event: started, op: osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.001876 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2675 (0'0,173'2675] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2674 active] append_log log((0'0,173'2675], crt=0'0) [173'2676 (173'788) modify   c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 by client.4285.0:143413 2014-06-12 08:43:26.691932]
2014-06-12 14:47:23.001893 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2676 (0'0,173'2676] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2674 active] add_log_entry 173'2676 (173'788) modify   c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 by client.4285.0:143413 2014-06-12 08:43:26.691932
2014-06-12 14:47:23.001909 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2676 (0'0,173'2676] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2674 active] append_log  adding 1 keys
2014-06-12 14:47:23.001935 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2676, trimmed: 
2014-06-12 14:47:23.001953 7f9d623d5700  5 -- op tracker -- , seq: 748426, time: 2014-06-12 14:47:23.001950, event: started, op: osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.001962 7f9d623d5700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:23.001965 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:23.001966 7f9d623d5700  5 queue_op 0x773e6c0 seq 2486 osr(3.1b7 0x276d1a0) 1149173 bytes   (queue has 1 ops and 1149173 bytes)
2014-06-12 14:47:23.001975 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:23.001971 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2676 (0'0,173'2676] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2674 active] do_sub_op osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.001981 7f9d623d5700 10 osd.2 173 dequeue_op 0x2a903d00 finish
2014-06-12 14:47:23.001981 7f9d81970700  5 _do_op 0x773e6c0 seq 2486 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:23.001983 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:23.001985 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:23.001985 7f9d81970700 10 _do_transaction on 0x5bf50b0
2014-06-12 14:47:23.001990 7f9d81970700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:23.002069 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:23.002085 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:23.002114 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:23.002184 7f9d81970700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:23.002190 7f9d81970700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:23.002194 7f9d81970700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:23.002203 7f9d81970700 15 _write 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 0~1146880
2014-06-12 14:47:23.002323 7f9d81970700 10 lookup_strip_header done  cid 3.1b7_head oid c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:23.002412 7f9d81970700 10 file_to_extents done 
2014-06-12 14:47:23.005362 7f9d81970700 10 _generic_write 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 0~1146880 = 0
2014-06-12 14:47:23.005440 7f9d81970700 15 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:23.005453 7f9d81970700 10 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 = 0
2014-06-12 14:47:23.005458 7f9d81970700 15 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:23.005463 7f9d81970700 10 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 = 0
2014-06-12 14:47:23.005676 7f9d81970700 20 set_header setting 17534 cid 3.1b7_head oid c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 parent seq 0
2014-06-12 14:47:23.005749 7f9d81970700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:23.005760 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:23.005767 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:47:23.010989 7f9d81970700 10 _do_op 0x773e6c0 seq 2486 r = 0, finisher 0x5723180 0
2014-06-12 14:47:23.011008 7f9d81970700 10 _finish_op 0x773e6c0 seq 2486 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:23.011014 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:23.011020 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:23.011026 7f9d7715b700  5 -- op tracker -- , seq: 748426, time: 2014-06-12 14:47:23.011017, event: commit_sent, op: osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.011035 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2676 (0'0,173'2676] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2674 active] sub_op_modify_commit on op osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:23.011057 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:23.011091 7f9d7795c700  5 -- op tracker -- , seq: 748426, time: 2014-06-12 14:47:23.011083, event: sub_op_applied, op: osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.011101 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2676 (0'0,173'2676] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2675 active] sub_op_modify_applied on 0x5bf4fc0 op osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.011119 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2676 (0'0,173'2676] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2675 active] op_applied on primary on version 173'2676
2014-06-12 14:47:23.011130 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2676 (0'0,173'2676] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2675 active] op_applied on replica on version 173'2676
2014-06-12 14:47:23.011152 7f9d7795c700  5 -- op tracker -- , seq: 748426, time: 2014-06-12 14:47:23.011147, event: done, op: osd_sub_op(client.4285.0:143413 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2676 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.161386 7f9d5a16f700  5 -- op tracker -- , seq: 748427, time: 2014-06-12 14:47:23.160291, event: header_read, op: osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.161411 7f9d5a16f700  5 -- op tracker -- , seq: 748427, time: 2014-06-12 14:47:23.160294, event: throttled, op: osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.161448 7f9d5a16f700  5 -- op tracker -- , seq: 748427, time: 2014-06-12 14:47:23.161318, event: all_read, op: osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.161464 7f9d5a16f700  5 -- op tracker -- , seq: 748427, time: 0.000000, event: dispatched, op: osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.161472 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:23.161482 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:23.161494 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a902d00 prio 127 cost 516903 latency 0.001203 osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.161569 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:23.161591 7f9d68be2700 10 osd.2 173 dequeue_op 0x2a902d00 prio 127 cost 516903 latency 0.001300 osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.760( v 173'2431 (0'0,173'2431] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active]
2014-06-12 14:47:23.161643 7f9d68be2700  5 -- op tracker -- , seq: 748427, time: 2014-06-12 14:47:23.161640, event: reached_pg, op: osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.161652 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2431 (0'0,173'2431] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active] handle_message: 0x2a902d00
2014-06-12 14:47:23.161661 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2431 (0'0,173'2431] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active] sub_op_modify trans 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 v 173'2432 (transaction) 174
2014-06-12 14:47:23.161672 7f9d68be2700  5 -- op tracker -- , seq: 748427, time: 2014-06-12 14:47:23.161670, event: started, op: osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.161685 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2431 (0'0,173'2431] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active] append_log log((0'0,173'2431], crt=0'0) [173'2432 (173'2431) modify   816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 by client.4301.0:145110 2014-06-12 08:44:10.690720]
2014-06-12 14:47:23.161702 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2432 (0'0,173'2432] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active] add_log_entry 173'2432 (173'2431) modify   816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 by client.4301.0:145110 2014-06-12 08:44:10.690720
2014-06-12 14:47:23.161718 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2432 (0'0,173'2432] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active] append_log  adding 1 keys
2014-06-12 14:47:23.161742 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2432, trimmed: 
2014-06-12 14:47:23.161774 7f9d68be2700  5 -- op tracker -- , seq: 748427, time: 2014-06-12 14:47:23.161771, event: started, op: osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.161782 7f9d68be2700  5 queue_transactions existing osr(3.760 0x49b41b0)/0x49b41b0
2014-06-12 14:47:23.161785 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:23.161786 7f9d68be2700  5 queue_op 0x614d320 seq 2245 osr(3.760 0x49b41b0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:23.161796 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x55217a0 (1 active)
2014-06-12 14:47:23.161802 7f9d7a962700  5 _do_op 0x614d320 seq 2245 osr(3.760 0x49b41b0)/0x49b41b0 start
2014-06-12 14:47:23.161792 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.760( v 173'2432 (0'0,173'2432] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active] do_sub_op osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.161805 7f9d68be2700 10 osd.2 173 dequeue_op 0x2a902d00 finish
2014-06-12 14:47:23.161806 7f9d7a962700 10 _do_transaction on 0x7f86b70
2014-06-12 14:47:23.161808 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:23.161810 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:23.161812 7f9d7a962700 15 _omap_setkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:23.161939 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:23.161975 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:23.162034 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:23.162065 7f9d7a962700 15 _omap_rmkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:23.162077 7f9d7a962700 10 _omap_rmkeys meta/1fe099ca/pglog_3.760/0//-1 r = 0
2014-06-12 14:47:23.162085 7f9d7a962700 15 _omap_setkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:23.162283 7f9d7a962700 15 _write 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 1998848~516096
2014-06-12 14:47:23.162478 7f9d7a962700 10 lookup_strip_header done  cid 3.760_head oid 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:23.162652 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:47:23.163224 7f9d5a16f700  5 -- op tracker -- , seq: 748428, time: 2014-06-12 14:47:23.162096, event: header_read, op: osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.163232 7f9d5a16f700  5 -- op tracker -- , seq: 748428, time: 2014-06-12 14:47:23.162099, event: throttled, op: osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.163237 7f9d5a16f700  5 -- op tracker -- , seq: 748428, time: 2014-06-12 14:47:23.163203, event: all_read, op: osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.163255 7f9d5a16f700  5 -- op tracker -- , seq: 748428, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.163259 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:23.163262 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:23.163267 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a902800 prio 127 cost 516903 latency 0.001171 osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.163288 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:23.163293 7f9d643d9700 10 osd.2 173 dequeue_op 0x2a902800 prio 127 cost 516903 latency 0.001197 osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2873 (0'0,173'2873] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2872 active]
2014-06-12 14:47:23.163309 7f9d643d9700  5 -- op tracker -- , seq: 748428, time: 2014-06-12 14:47:23.163306, event: reached_pg, op: osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.163316 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2873 (0'0,173'2873] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2872 active] handle_message: 0x2a902800
2014-06-12 14:47:23.163324 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2873 (0'0,173'2873] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2872 active] sub_op_modify trans f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 v 173'2874 (transaction) 174
2014-06-12 14:47:23.163347 7f9d643d9700  5 -- op tracker -- , seq: 748428, time: 2014-06-12 14:47:23.163331, event: started, op: osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.163375 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2873 (0'0,173'2873] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2872 active] append_log log((0'0,173'2873], crt=0'0) [173'2874 (173'2873) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145706 2014-06-12 08:43:55.889477]
2014-06-12 14:47:23.163434 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2874 (0'0,173'2874] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2872 active] add_log_entry 173'2874 (173'2873) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145706 2014-06-12 08:43:55.889477
2014-06-12 14:47:23.163489 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2874 (0'0,173'2874] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2872 active] append_log  adding 1 keys
2014-06-12 14:47:23.163549 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2874, trimmed: 
2014-06-12 14:47:23.163597 7f9d643d9700  5 -- op tracker -- , seq: 748428, time: 2014-06-12 14:47:23.163587, event: started, op: osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.163634 7f9d643d9700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:23.163641 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:23.163643 7f9d643d9700  5 queue_op 0x101cd140 seq 2587 osr(3.129 0x276cf20) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:47:23.163666 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (2 active)
2014-06-12 14:47:23.163655 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2874 (0'0,173'2874] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2872 active] do_sub_op osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.163684 7f9d643d9700 10 osd.2 173 dequeue_op 0x2a902800 finish
2014-06-12 14:47:23.163683 7f9d82972700  5 _do_op 0x101cd140 seq 2587 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:23.163691 7f9d82972700 10 _do_transaction on 0x31d080f0
2014-06-12 14:47:23.163691 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:23.163696 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:23.163703 7f9d82972700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:23.163839 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:23.163974 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:23.164575 7f9d7a962700 10 _generic_write 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 1998848~516096 = 0
2014-06-12 14:47:23.164610 7f9d7a962700 15 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:23.164623 7f9d7a962700 10 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 = 0
2014-06-12 14:47:23.164629 7f9d7a962700 15 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:23.164636 7f9d7a962700 10 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 = 0
2014-06-12 14:47:23.164824 7f9d7a962700 20 set_header setting 17060 cid 3.760_head oid 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 parent seq 0
2014-06-12 14:47:23.164883 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:23.164897 7f9d7a962700 20 set_header setting 2198 cid meta oid 1fe099ca/pglog_3.760/0//-1 parent seq 0
2014-06-12 14:47:23.164906 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:47:23.166992 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:23.167024 7f9d82972700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:23.167030 7f9d82972700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:23.167037 7f9d82972700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:23.167040 7f9d7a962700 10 _do_op 0x614d320 seq 2245 r = 0, finisher 0x267402e0 0
2014-06-12 14:47:23.167052 7f9d7a962700 10 _finish_op 0x614d320 seq 2245 osr(3.760 0x49b41b0)/0x49b41b0
2014-06-12 14:47:23.167056 7f9d82972700 15 _write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 3538944~516096
2014-06-12 14:47:23.167064 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x55217a0 (1 active)
2014-06-12 14:47:23.167073 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:23.167086 7f9d7795c700  5 -- op tracker -- , seq: 748427, time: 2014-06-12 14:47:23.167076, event: sub_op_applied, op: osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.167094 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2432 (0'0,173'2432] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active] sub_op_modify_applied on 0x7f86a80 op osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.167122 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:23.167143 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2432 (0'0,173'2432] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active] op_applied on primary on version 173'2432
2014-06-12 14:47:23.167156 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2432 (0'0,173'2432] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active] op_applied on replica on version 173'2432
2014-06-12 14:47:23.167177 7f9d7715b700  5 -- op tracker -- , seq: 748427, time: 2014-06-12 14:47:23.167172, event: commit_sent, op: osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.167196 7f9d82972700 10 lookup_strip_header done  cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:23.167184 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2432 (0'0,173'2432] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2430 active] sub_op_modify_commit on op osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:23.167204 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:23.167234 7f9d7715b700  5 -- op tracker -- , seq: 748427, time: 2014-06-12 14:47:23.167228, event: done, op: osd_sub_op(client.4301.0:145110 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2432 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.167412 7f9d82972700 10 file_to_extents done 
2014-06-12 14:47:23.168598 7f9d82972700 10 _generic_write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 3538944~516096 = 0
2014-06-12 14:47:23.168650 7f9d82972700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:23.168661 7f9d82972700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:23.168666 7f9d82972700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:23.168671 7f9d82972700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:23.168810 7f9d82972700 20 set_header setting 16959 cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 parent seq 0
2014-06-12 14:47:23.168874 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:23.168884 7f9d82972700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:23.168900 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:47:23.172641 7f9d82972700 10 _do_op 0x101cd140 seq 2587 r = 0, finisher 0x85e0ee0 0
2014-06-12 14:47:23.172665 7f9d82972700 10 _finish_op 0x101cd140 seq 2587 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:23.172675 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:23.172679 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:23.172703 7f9d7715b700  5 -- op tracker -- , seq: 748428, time: 2014-06-12 14:47:23.172684, event: commit_sent, op: osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.172723 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2874 (0'0,173'2874] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2872 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:23.172776 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:23.172840 7f9d7795c700  5 -- op tracker -- , seq: 748428, time: 2014-06-12 14:47:23.172834, event: sub_op_applied, op: osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.172851 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2874 (0'0,173'2874] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2873 active] sub_op_modify_applied on 0x31d08000 op osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.172866 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2874 (0'0,173'2874] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2873 active] op_applied on primary on version 173'2874
2014-06-12 14:47:23.172872 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2874 (0'0,173'2874] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2873 active] op_applied on replica on version 173'2874
2014-06-12 14:47:23.172890 7f9d7795c700  5 -- op tracker -- , seq: 748428, time: 2014-06-12 14:47:23.172887, event: done, op: osd_sub_op(client.4302.0:145706 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2874 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.361250 7f9d5a16f700  5 -- op tracker -- , seq: 748429, time: 2014-06-12 14:47:23.360386, event: header_read, op: osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.361269 7f9d5a16f700  5 -- op tracker -- , seq: 748429, time: 2014-06-12 14:47:23.360388, event: throttled, op: osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.361280 7f9d5a16f700  5 -- op tracker -- , seq: 748429, time: 2014-06-12 14:47:23.361198, event: all_read, op: osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.361303 7f9d5a16f700  5 -- op tracker -- , seq: 748429, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.361309 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:23.361317 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:23.361327 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a902300 prio 127 cost 516903 latency 0.000941 osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.361381 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:23.361393 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2a902300 prio 127 cost 516903 latency 0.001007 osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.31d( v 173'2481 (0'0,173'2481] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active]
2014-06-12 14:47:23.361438 7f9d63bd8700  5 -- op tracker -- , seq: 748429, time: 2014-06-12 14:47:23.361435, event: reached_pg, op: osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.361449 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2481 (0'0,173'2481] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] handle_message: 0x2a902300
2014-06-12 14:47:23.361456 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2481 (0'0,173'2481] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] sub_op_modify trans 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 v 173'2482 (transaction) 174
2014-06-12 14:47:23.361474 7f9d63bd8700  5 -- op tracker -- , seq: 748429, time: 2014-06-12 14:47:23.361464, event: started, op: osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.361486 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2481 (0'0,173'2481] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] append_log log((0'0,173'2481], crt=0'0) [173'2482 (173'2481) modify   5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 by client.4315.0:144550 2014-06-12 08:44:41.673135]
2014-06-12 14:47:23.361503 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2482 (0'0,173'2482] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] add_log_entry 173'2482 (173'2481) modify   5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 by client.4315.0:144550 2014-06-12 08:44:41.673135
2014-06-12 14:47:23.361525 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2482 (0'0,173'2482] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] append_log  adding 1 keys
2014-06-12 14:47:23.361546 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2482, trimmed: 
2014-06-12 14:47:23.361566 7f9d63bd8700  5 -- op tracker -- , seq: 748429, time: 2014-06-12 14:47:23.361563, event: started, op: osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.361572 7f9d63bd8700  5 queue_transactions existing osr(3.31d 0x276d520)/0x276d520
2014-06-12 14:47:23.361575 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:23.361576 7f9d63bd8700  5 queue_op 0x6df1a40 seq 2300 osr(3.31d 0x276d520) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:23.361580 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.31d( v 173'2482 (0'0,173'2482] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] do_sub_op osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.361589 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2a902300 finish
2014-06-12 14:47:23.361599 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:23.361601 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:23.361612 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439440 (1 active)
2014-06-12 14:47:23.361642 7f9d7895e700  5 _do_op 0x6df1a40 seq 2300 osr(3.31d 0x276d520)/0x276d520 start
2014-06-12 14:47:23.361647 7f9d7895e700 10 _do_transaction on 0xc9a49b0
2014-06-12 14:47:23.361652 7f9d7895e700 15 _omap_setkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:23.361881 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:23.361902 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:23.361942 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:23.361957 7f9d7895e700 15 _omap_rmkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:23.361963 7f9d7895e700 10 _omap_rmkeys meta/1fe1ce3b/pglog_3.31d/0//-1 r = 0
2014-06-12 14:47:23.361967 7f9d7895e700 15 _omap_setkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:23.361988 7f9d7895e700 15 _write 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 1536000~516096
2014-06-12 14:47:23.362223 7f9d7895e700 10 lookup_strip_header done  cid 3.31d_head oid 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:23.362362 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:47:23.363972 7f9d7895e700 10 _generic_write 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 1536000~516096 = 0
2014-06-12 14:47:23.364030 7f9d7895e700 15 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:23.364046 7f9d7895e700 10 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 = 0
2014-06-12 14:47:23.364056 7f9d7895e700 15 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:23.364065 7f9d7895e700 10 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 = 0
2014-06-12 14:47:23.364274 7f9d7895e700 20 set_header setting 17151 cid 3.31d_head oid 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 parent seq 0
2014-06-12 14:47:23.364349 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:23.364366 7f9d7895e700 20 set_header setting 2268 cid meta oid 1fe1ce3b/pglog_3.31d/0//-1 parent seq 0
2014-06-12 14:47:23.364377 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:47:23.366699 7f9d7895e700 10 _do_op 0x6df1a40 seq 2300 r = 0, finisher 0x8274640 0
2014-06-12 14:47:23.366719 7f9d7895e700 10 _finish_op 0x6df1a40 seq 2300 osr(3.31d 0x276d520)/0x276d520
2014-06-12 14:47:23.366734 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439440 (0 active)
2014-06-12 14:47:23.366740 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:23.366764 7f9d7795c700  5 -- op tracker -- , seq: 748429, time: 2014-06-12 14:47:23.366745, event: sub_op_applied, op: osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.366883 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2482 (0'0,173'2482] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] sub_op_modify_applied on 0xc9a48c0 op osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.366917 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:23.366933 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2482 (0'0,173'2482] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] op_applied on primary on version 173'2482
2014-06-12 14:47:23.366943 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2482 (0'0,173'2482] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] op_applied on replica on version 173'2482
2014-06-12 14:47:23.366967 7f9d7715b700  5 -- op tracker -- , seq: 748429, time: 2014-06-12 14:47:23.366960, event: commit_sent, op: osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.366986 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2482 (0'0,173'2482] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:23.367006 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:23.367033 7f9d7715b700  5 -- op tracker -- , seq: 748429, time: 2014-06-12 14:47:23.367029, event: done, op: osd_sub_op(client.4315.0:144550 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.481040 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:23.481557 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:23.484283 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:23.484301 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:23.484305 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:23.484307 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:23.484309 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 7; oldest is 14.313534 seconds old
2014-06-12 14:47:23.484308 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:23.519615 7f9d5a16f700  5 -- op tracker -- , seq: 748430, time: 2014-06-12 14:47:23.518491, event: header_read, op: osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.519636 7f9d5a16f700  5 -- op tracker -- , seq: 748430, time: 2014-06-12 14:47:23.518494, event: throttled, op: osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.519648 7f9d5a16f700  5 -- op tracker -- , seq: 748430, time: 2014-06-12 14:47:23.519556, event: all_read, op: osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.519664 7f9d5a16f700  5 -- op tracker -- , seq: 748430, time: 0.000000, event: dispatched, op: osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.519671 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:23.519680 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:23.519691 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329ebb00 prio 127 cost 516903 latency 0.001199 osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.519765 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:23.519787 7f9d67be0700 10 osd.2 173 dequeue_op 0x329ebb00 prio 127 cost 516903 latency 0.001295 osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2676 (0'0,173'2676] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2675 active]
2014-06-12 14:47:23.519829 7f9d67be0700  5 -- op tracker -- , seq: 748430, time: 2014-06-12 14:47:23.519822, event: reached_pg, op: osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.519867 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2676 (0'0,173'2676] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2675 active] handle_message: 0x329ebb00
2014-06-12 14:47:23.519888 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2676 (0'0,173'2676] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2675 active] sub_op_modify trans c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 v 173'2677 (transaction) 174
2014-06-12 14:47:23.519912 7f9d67be0700  5 -- op tracker -- , seq: 748430, time: 2014-06-12 14:47:23.519906, event: started, op: osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.519935 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2676 (0'0,173'2676] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2675 active] append_log log((0'0,173'2676], crt=0'0) [173'2677 (173'2676) modify   c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 by client.4285.0:143414 2014-06-12 08:43:26.693571]
2014-06-12 14:47:23.519971 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2677 (0'0,173'2677] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2675 active] add_log_entry 173'2677 (173'2676) modify   c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 by client.4285.0:143414 2014-06-12 08:43:26.693571
2014-06-12 14:47:23.520016 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2677 (0'0,173'2677] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2675 active] append_log  adding 1 keys
2014-06-12 14:47:23.520070 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2677, trimmed: 
2014-06-12 14:47:23.520105 7f9d67be0700  5 -- op tracker -- , seq: 748430, time: 2014-06-12 14:47:23.520098, event: started, op: osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.520119 7f9d67be0700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:23.520124 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:23.520126 7f9d67be0700  5 queue_op 0x69b29c0 seq 2487 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:23.520140 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:23.520136 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2677 (0'0,173'2677] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2675 active] do_sub_op osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.520154 7f9d83173700  5 _do_op 0x69b29c0 seq 2487 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:23.520156 7f9d67be0700 10 osd.2 173 dequeue_op 0x329ebb00 finish
2014-06-12 14:47:23.520161 7f9d83173700 10 _do_transaction on 0xc9a4630
2014-06-12 14:47:23.520162 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:23.520173 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:23.520171 7f9d83173700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:23.520596 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:23.520633 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:23.520697 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:23.520729 7f9d83173700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:23.520740 7f9d83173700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:23.520749 7f9d83173700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:23.520892 7f9d83173700 15 _write 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 1146880~516096
2014-06-12 14:47:23.521086 7f9d83173700 10 lookup_strip_header done  cid 3.1b7_head oid c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:23.521229 7f9d83173700 10 file_to_extents done 
2014-06-12 14:47:23.523746 7f9d83173700 10 _generic_write 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 1146880~516096 = 0
2014-06-12 14:47:23.523819 7f9d83173700 15 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:23.523838 7f9d83173700 10 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 = 0
2014-06-12 14:47:23.523845 7f9d83173700 15 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:23.523855 7f9d83173700 10 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 = 0
2014-06-12 14:47:23.524126 7f9d83173700 20 set_header setting 17534 cid 3.1b7_head oid c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 parent seq 0
2014-06-12 14:47:23.524226 7f9d83173700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:23.524249 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:23.524260 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:47:23.526855 7f9d83173700 10 _do_op 0x69b29c0 seq 2487 r = 0, finisher 0xc5283c0 0
2014-06-12 14:47:23.526868 7f9d83173700 10 _finish_op 0x69b29c0 seq 2487 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:23.526872 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:23.526878 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:23.526922 7f9d7715b700  5 -- op tracker -- , seq: 748430, time: 2014-06-12 14:47:23.526912, event: commit_sent, op: osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.526939 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2677 (0'0,173'2677] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2675 active] sub_op_modify_commit on op osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:23.526964 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:23.526994 7f9d7795c700  5 -- op tracker -- , seq: 748430, time: 2014-06-12 14:47:23.526988, event: sub_op_applied, op: osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.527010 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2677 (0'0,173'2677] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active] sub_op_modify_applied on 0xc9a4540 op osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.527021 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2677 (0'0,173'2677] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active] op_applied on primary on version 173'2677
2014-06-12 14:47:23.527027 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2677 (0'0,173'2677] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active] op_applied on replica on version 173'2677
2014-06-12 14:47:23.527057 7f9d7795c700  5 -- op tracker -- , seq: 748430, time: 2014-06-12 14:47:23.527050, event: done, op: osd_sub_op(client.4285.0:143414 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2677 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.645833 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:23.661981 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:23.716804 7f9d5a16f700  5 -- op tracker -- , seq: 748431, time: 2014-06-12 14:47:23.715834, event: header_read, op: osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.716822 7f9d5a16f700  5 -- op tracker -- , seq: 748431, time: 2014-06-12 14:47:23.715836, event: throttled, op: osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.716833 7f9d5a16f700  5 -- op tracker -- , seq: 748431, time: 2014-06-12 14:47:23.716757, event: all_read, op: osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.716840 7f9d5a16f700  5 -- op tracker -- , seq: 748431, time: 0.000000, event: dispatched, op: osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.716845 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:23.716852 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:23.716860 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329ea100 prio 127 cost 516903 latency 0.001026 osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.716889 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:23.716898 7f9d65bdc700 10 osd.2 173 dequeue_op 0x329ea100 prio 127 cost 516903 latency 0.001064 osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.760( v 173'2432 (0'0,173'2432] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2431 active]
2014-06-12 14:47:23.716927 7f9d65bdc700  5 -- op tracker -- , seq: 748431, time: 2014-06-12 14:47:23.716922, event: reached_pg, op: osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.716941 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2432 (0'0,173'2432] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2431 active] handle_message: 0x329ea100
2014-06-12 14:47:23.716955 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2432 (0'0,173'2432] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2431 active] sub_op_modify trans 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 v 173'2433 (transaction) 174
2014-06-12 14:47:23.716980 7f9d65bdc700  5 -- op tracker -- , seq: 748431, time: 2014-06-12 14:47:23.716976, event: started, op: osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.716997 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2432 (0'0,173'2432] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2431 active] append_log log((0'0,173'2432], crt=0'0) [173'2433 (173'2432) modify   816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 by client.4301.0:145111 2014-06-12 08:44:11.890696]
2014-06-12 14:47:23.717023 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2433 (0'0,173'2433] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2431 active] add_log_entry 173'2433 (173'2432) modify   816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 by client.4301.0:145111 2014-06-12 08:44:11.890696
2014-06-12 14:47:23.717049 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2433 (0'0,173'2433] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2431 active] append_log  adding 1 keys
2014-06-12 14:47:23.717086 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2433, trimmed: 
2014-06-12 14:47:23.717114 7f9d65bdc700  5 -- op tracker -- , seq: 748431, time: 2014-06-12 14:47:23.717109, event: started, op: osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.717127 7f9d65bdc700  5 queue_transactions existing osr(3.760 0x49b41b0)/0x49b41b0
2014-06-12 14:47:23.717132 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:23.717133 7f9d65bdc700  5 queue_op 0xb9b46c0 seq 2246 osr(3.760 0x49b41b0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:23.717143 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x55217a0 (1 active)
2014-06-12 14:47:23.717149 7f9d7a161700  5 _do_op 0xb9b46c0 seq 2246 osr(3.760 0x49b41b0)/0x49b41b0 start
2014-06-12 14:47:23.717154 7f9d7a161700 10 _do_transaction on 0xc9a57b0
2014-06-12 14:47:23.717140 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.760( v 173'2433 (0'0,173'2433] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2431 active] do_sub_op osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.717158 7f9d65bdc700 10 osd.2 173 dequeue_op 0x329ea100 finish
2014-06-12 14:47:23.717159 7f9d7a161700 15 _omap_setkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:23.717162 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:23.717165 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:23.717275 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:23.717306 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:23.717354 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:23.717456 7f9d7a161700 15 _omap_rmkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:23.717465 7f9d7a161700 10 _omap_rmkeys meta/1fe099ca/pglog_3.760/0//-1 r = 0
2014-06-12 14:47:23.717471 7f9d7a161700 15 _omap_setkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:23.717491 7f9d7a161700 15 _write 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 2514944~516096
2014-06-12 14:47:23.717652 7f9d7a161700 10 lookup_strip_header done  cid 3.760_head oid 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:23.717871 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:47:23.719678 7f9d7a161700 10 _generic_write 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 2514944~516096 = 0
2014-06-12 14:47:23.719757 7f9d7a161700 15 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:23.719774 7f9d7a161700 10 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 = 0
2014-06-12 14:47:23.719782 7f9d7a161700 15 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:23.719791 7f9d7a161700 10 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 = 0
2014-06-12 14:47:23.720008 7f9d7a161700 20 set_header setting 17060 cid 3.760_head oid 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 parent seq 0
2014-06-12 14:47:23.720094 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:23.720120 7f9d7a161700 20 set_header setting 2198 cid meta oid 1fe099ca/pglog_3.760/0//-1 parent seq 0
2014-06-12 14:47:23.720131 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:47:23.722711 7f9d7a161700 10 _do_op 0xb9b46c0 seq 2246 r = 0, finisher 0x7ffafa0 0
2014-06-12 14:47:23.722740 7f9d7a161700 10 _finish_op 0xb9b46c0 seq 2246 osr(3.760 0x49b41b0)/0x49b41b0
2014-06-12 14:47:23.722748 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x55217a0 (0 active)
2014-06-12 14:47:23.722755 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:23.722763 7f9d7715b700  5 -- op tracker -- , seq: 748431, time: 2014-06-12 14:47:23.722750, event: commit_sent, op: osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.722774 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2433 (0'0,173'2433] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2431 active] sub_op_modify_commit on op osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:23.722809 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:23.722872 7f9d7795c700  5 -- op tracker -- , seq: 748431, time: 2014-06-12 14:47:23.722866, event: sub_op_applied, op: osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.722882 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2433 (0'0,173'2433] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2432 active] sub_op_modify_applied on 0xc9a56c0 op osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.722903 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2433 (0'0,173'2433] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2432 active] op_applied on primary on version 173'2433
2014-06-12 14:47:23.722914 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2433 (0'0,173'2433] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2432 active] op_applied on replica on version 173'2433
2014-06-12 14:47:23.722939 7f9d7795c700  5 -- op tracker -- , seq: 748431, time: 2014-06-12 14:47:23.722935, event: done, op: osd_sub_op(client.4301.0:145111 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2433 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.723000 7f9d5a16f700  5 -- op tracker -- , seq: 748432, time: 2014-06-12 14:47:23.722793, event: header_read, op: osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.723012 7f9d5a16f700  5 -- op tracker -- , seq: 748432, time: 2014-06-12 14:47:23.722795, event: throttled, op: osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.723021 7f9d5a16f700  5 -- op tracker -- , seq: 748432, time: 2014-06-12 14:47:23.722966, event: all_read, op: osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.723029 7f9d5a16f700  5 -- op tracker -- , seq: 748432, time: 0.000000, event: dispatched, op: osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.723034 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:23.723044 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:23.723050 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329ebe00 prio 127 cost 140071 latency 0.000257 osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.723081 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:23.723094 7f9d62bd6700 10 osd.2 173 dequeue_op 0x329ebe00 prio 127 cost 140071 latency 0.000301 osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2874 (0'0,173'2874] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2873 active]
2014-06-12 14:47:23.723117 7f9d62bd6700  5 -- op tracker -- , seq: 748432, time: 2014-06-12 14:47:23.723113, event: reached_pg, op: osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.723126 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2874 (0'0,173'2874] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2873 active] handle_message: 0x329ebe00
2014-06-12 14:47:23.723138 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2874 (0'0,173'2874] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2873 active] sub_op_modify trans f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 v 173'2875 (transaction) 174
2014-06-12 14:47:23.723155 7f9d62bd6700  5 -- op tracker -- , seq: 748432, time: 2014-06-12 14:47:23.723151, event: started, op: osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.723170 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2874 (0'0,173'2874] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2873 active] append_log log((0'0,173'2874], crt=0'0) [173'2875 (173'2874) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145707 2014-06-12 08:43:56.238710]
2014-06-12 14:47:23.723195 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2875 (0'0,173'2875] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2873 active] add_log_entry 173'2875 (173'2874) modify   f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 by client.4302.0:145707 2014-06-12 08:43:56.238710
2014-06-12 14:47:23.723217 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2875 (0'0,173'2875] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2873 active] append_log  adding 1 keys
2014-06-12 14:47:23.723250 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2875, trimmed: 
2014-06-12 14:47:23.723274 7f9d62bd6700  5 -- op tracker -- , seq: 748432, time: 2014-06-12 14:47:23.723269, event: started, op: osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.723285 7f9d62bd6700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:23.723288 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:23.723290 7f9d62bd6700  5 queue_op 0x9346120 seq 2588 osr(3.129 0x276cf20) 141557 bytes   (queue has 1 ops and 141557 bytes)
2014-06-12 14:47:23.723297 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2875 (0'0,173'2875] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2873 active] do_sub_op osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.723310 7f9d62bd6700 10 osd.2 173 dequeue_op 0x329ebe00 finish
2014-06-12 14:47:23.723314 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:23.723316 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:23.725623 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:23.725635 7f9d7f16b700  5 _do_op 0x9346120 seq 2588 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:23.725640 7f9d7f16b700 10 _do_transaction on 0x2bc9a7f0
2014-06-12 14:47:23.725646 7f9d7f16b700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:23.725735 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:23.725750 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:23.725779 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:23.725795 7f9d7f16b700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:23.725802 7f9d7f16b700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:23.725806 7f9d7f16b700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:23.725884 7f9d7f16b700 15 _write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 4055040~139264
2014-06-12 14:47:23.725990 7f9d7f16b700 10 lookup_strip_header done  cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:23.726098 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:47:23.726447 7f9d7f16b700 10 _generic_write 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 4055040~139264 = 0
2014-06-12 14:47:23.726464 7f9d7f16b700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:23.726471 7f9d7f16b700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:23.726475 7f9d7f16b700 15 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3
2014-06-12 14:47:23.726479 7f9d7f16b700 10 _setattrs 3.129_head/f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 = 0
2014-06-12 14:47:23.726559 7f9d7f16b700 20 set_header setting 16959 cid 3.129_head oid f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 parent seq 0
2014-06-12 14:47:23.726579 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:23.726587 7f9d7f16b700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:23.726593 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:47:23.727365 7f9d7f16b700 10 _do_op 0x9346120 seq 2588 r = 0, finisher 0xabf2980 0
2014-06-12 14:47:23.727374 7f9d7f16b700 10 _finish_op 0x9346120 seq 2588 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:23.727380 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:23.727382 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:23.727394 7f9d7715b700  5 -- op tracker -- , seq: 748432, time: 2014-06-12 14:47:23.727384, event: commit_sent, op: osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.727411 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2875 (0'0,173'2875] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2873 active] sub_op_modify_commit on op osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:23.727440 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:23.727476 7f9d7795c700  5 -- op tracker -- , seq: 748432, time: 2014-06-12 14:47:23.727469, event: sub_op_applied, op: osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.727494 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2875 (0'0,173'2875] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active] sub_op_modify_applied on 0x2bc9a700 op osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.727512 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2875 (0'0,173'2875] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active] op_applied on primary on version 173'2875
2014-06-12 14:47:23.727521 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2875 (0'0,173'2875] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active] op_applied on replica on version 173'2875
2014-06-12 14:47:23.727540 7f9d7795c700  5 -- op tracker -- , seq: 748432, time: 2014-06-12 14:47:23.727536, event: done, op: osd_sub_op(client.4302.0:145707 3.129 f9093929/rbd_data.10a56b8b4567.0000000000000ff9/head//3 [] v 173'2875 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.729891 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19428 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,7])
2014-06-12 14:47:23.729912 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19428 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,7])
2014-06-12 14:47:23.730367 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:23.730392 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:23.730427 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:23.730452 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:23.730485 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:23.730521 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:23.730540 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:23.730558 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:23.730578 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:23.730595 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:23.730612 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:23.730629 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:23.730648 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:23.730665 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:23.730683 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:23.730699 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:23.730715 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:23.730731 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:23.730752 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:23.730768 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:23.730785 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:23.730801 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:23.738629 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:23.760686 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:23.760722 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:23.806725 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:23.825796 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:23.825840 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:23.845305 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:23.845476 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:23.876700 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:23.876771 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:23.918393 7f9d5a16f700  5 -- op tracker -- , seq: 748433, time: 2014-06-12 14:47:23.917604, event: header_read, op: osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.918405 7f9d5a16f700  5 -- op tracker -- , seq: 748433, time: 2014-06-12 14:47:23.917606, event: throttled, op: osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.918411 7f9d5a16f700  5 -- op tracker -- , seq: 748433, time: 2014-06-12 14:47:23.918353, event: all_read, op: osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.918416 7f9d5a16f700  5 -- op tracker -- , seq: 748433, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.918419 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:23.918424 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:23.918430 7f9d5a16f700 15 osd.2 173 enqueue_op 0x124e3700 prio 127 cost 516903 latency 0.000825 osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.918525 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:23.918534 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x124e3700 prio 127 cost 516903 latency 0.000930 osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.31d( v 173'2482 (0'0,173'2482] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active]
2014-06-12 14:47:23.918558 7f9d6b3e7700  5 -- op tracker -- , seq: 748433, time: 2014-06-12 14:47:23.918554, event: reached_pg, op: osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.918569 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2482 (0'0,173'2482] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] handle_message: 0x124e3700
2014-06-12 14:47:23.918588 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2482 (0'0,173'2482] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] sub_op_modify trans 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 v 173'2483 (transaction) 174
2014-06-12 14:47:23.918605 7f9d6b3e7700  5 -- op tracker -- , seq: 748433, time: 2014-06-12 14:47:23.918601, event: started, op: osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.918629 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2482 (0'0,173'2482] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] append_log log((0'0,173'2482], crt=0'0) [173'2483 (173'2482) modify   5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 by client.4315.0:144551 2014-06-12 08:44:41.692084]
2014-06-12 14:47:23.918660 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2483 (0'0,173'2483] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] add_log_entry 173'2483 (173'2482) modify   5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 by client.4315.0:144551 2014-06-12 08:44:41.692084
2014-06-12 14:47:23.918692 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2483 (0'0,173'2483] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] append_log  adding 1 keys
2014-06-12 14:47:23.918726 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2483, trimmed: 
2014-06-12 14:47:23.918752 7f9d6b3e7700  5 -- op tracker -- , seq: 748433, time: 2014-06-12 14:47:23.918747, event: started, op: osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.918765 7f9d6b3e7700  5 queue_transactions existing osr(3.31d 0x276d520)/0x276d520
2014-06-12 14:47:23.918769 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:23.918770 7f9d6b3e7700  5 queue_op 0x5668240 seq 2301 osr(3.31d 0x276d520) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:23.918777 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.31d( v 173'2483 (0'0,173'2483] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] do_sub_op osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.918799 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x124e3700 finish
2014-06-12 14:47:23.918802 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:23.918805 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:23.918813 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439440 (1 active)
2014-06-12 14:47:23.918818 7f9d82171700  5 _do_op 0x5668240 seq 2301 osr(3.31d 0x276d520)/0x276d520 start
2014-06-12 14:47:23.918830 7f9d82171700 10 _do_transaction on 0x5ab80f0
2014-06-12 14:47:23.918835 7f9d82171700 15 _omap_setkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:23.918945 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:23.918966 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:23.919016 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:23.919039 7f9d82171700 15 _omap_rmkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:23.919046 7f9d82171700 10 _omap_rmkeys meta/1fe1ce3b/pglog_3.31d/0//-1 r = 0
2014-06-12 14:47:23.919052 7f9d82171700 15 _omap_setkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:23.919070 7f9d82171700 15 _write 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 2052096~516096
2014-06-12 14:47:23.919214 7f9d82171700 10 lookup_strip_header done  cid 3.31d_head oid 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:23.919242 7f9d82171700 10 file_to_extents done 
2014-06-12 14:47:23.921148 7f9d82171700 10 _generic_write 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 2052096~516096 = 0
2014-06-12 14:47:23.921197 7f9d82171700 15 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:23.921211 7f9d82171700 10 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 = 0
2014-06-12 14:47:23.921219 7f9d82171700 15 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:23.921227 7f9d82171700 10 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 = 0
2014-06-12 14:47:23.921438 7f9d82171700 20 set_header setting 17151 cid 3.31d_head oid 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 parent seq 0
2014-06-12 14:47:23.921504 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:23.921519 7f9d82171700 20 set_header setting 2268 cid meta oid 1fe1ce3b/pglog_3.31d/0//-1 parent seq 0
2014-06-12 14:47:23.921532 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:47:23.923793 7f9d82171700 10 _do_op 0x5668240 seq 2301 r = 0, finisher 0xa3b7ce0 0
2014-06-12 14:47:23.923807 7f9d82171700 10 _finish_op 0x5668240 seq 2301 osr(3.31d 0x276d520)/0x276d520
2014-06-12 14:47:23.923839 7f9d7795c700  5 -- op tracker -- , seq: 748433, time: 2014-06-12 14:47:23.923832, event: sub_op_applied, op: osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.923857 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2483 (0'0,173'2483] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] sub_op_modify_applied on 0x5ab8000 op osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:23.923889 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:23.923914 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2483 (0'0,173'2483] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] op_applied on primary on version 173'2483
2014-06-12 14:47:23.923927 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2483 (0'0,173'2483] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] op_applied on replica on version 173'2483
2014-06-12 14:47:23.923843 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439440 (0 active)
2014-06-12 14:47:23.923952 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:23.923963 7f9d7715b700  5 -- op tracker -- , seq: 748433, time: 2014-06-12 14:47:23.923955, event: commit_sent, op: osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:23.923973 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2483 (0'0,173'2483] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:23.923995 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:23.924025 7f9d7715b700  5 -- op tracker -- , seq: 748433, time: 2014-06-12 14:47:23.924020, event: done, op: osd_sub_op(client.4315.0:144551 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.088419 7f9d5a16f700  5 -- op tracker -- , seq: 748434, time: 2014-06-12 14:47:24.087584, event: header_read, op: osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.088432 7f9d5a16f700  5 -- op tracker -- , seq: 748434, time: 2014-06-12 14:47:24.087586, event: throttled, op: osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.088437 7f9d5a16f700  5 -- op tracker -- , seq: 748434, time: 2014-06-12 14:47:24.088376, event: all_read, op: osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.088442 7f9d5a16f700  5 -- op tracker -- , seq: 748434, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.088446 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:24.088451 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:24.088458 7f9d5a16f700 15 osd.2 173 enqueue_op 0x68a6c00 prio 127 cost 533287 latency 0.000874 osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.088485 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:24.088503 7f9d693e3700 10 osd.2 173 dequeue_op 0x68a6c00 prio 127 cost 533287 latency 0.000918 osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2875 (0'0,173'2875] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active]
2014-06-12 14:47:24.088565 7f9d693e3700  5 -- op tracker -- , seq: 748434, time: 2014-06-12 14:47:24.088555, event: reached_pg, op: osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.088589 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2875 (0'0,173'2875] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active] handle_message: 0x68a6c00
2014-06-12 14:47:24.088618 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2875 (0'0,173'2875] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active] sub_op_modify trans 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 v 173'2876 (transaction) 174
2014-06-12 14:47:24.088656 7f9d693e3700  5 -- op tracker -- , seq: 748434, time: 2014-06-12 14:47:24.088647, event: started, op: osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.088689 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2875 (0'0,173'2875] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active] append_log log((0'0,173'2875], crt=0'0) [173'2876 (173'886) modify   69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 by client.4314.0:145055 2014-06-12 08:45:20.313416]
2014-06-12 14:47:24.088747 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2876 (0'0,173'2876] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active] add_log_entry 173'2876 (173'886) modify   69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 by client.4314.0:145055 2014-06-12 08:45:20.313416
2014-06-12 14:47:24.088796 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2876 (0'0,173'2876] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active] append_log  adding 1 keys
2014-06-12 14:47:24.088866 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2876, trimmed: 
2014-06-12 14:47:24.088929 7f9d693e3700  5 -- op tracker -- , seq: 748434, time: 2014-06-12 14:47:24.088918, event: started, op: osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.088952 7f9d693e3700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:24.088959 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:24.088969 7f9d693e3700  5 queue_op 0x76951a0 seq 2589 osr(3.129 0x276cf20) 534773 bytes   (queue has 1 ops and 534773 bytes)
2014-06-12 14:47:24.088979 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:24.088985 7f9d7e96a700  5 _do_op 0x76951a0 seq 2589 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:24.088989 7f9d7e96a700 10 _do_transaction on 0x127977b0
2014-06-12 14:47:24.088977 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2876 (0'0,173'2876] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active] do_sub_op osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.088992 7f9d693e3700 10 osd.2 173 dequeue_op 0x68a6c00 finish
2014-06-12 14:47:24.088994 7f9d7e96a700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:24.088996 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:24.088999 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:24.089098 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:24.089122 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:24.089154 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:24.089175 7f9d7e96a700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:24.089180 7f9d7e96a700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:24.089183 7f9d7e96a700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:24.089288 7f9d7e96a700 15 _write 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 0~532480
2014-06-12 14:47:24.089415 7f9d7e96a700 10 lookup_strip_header done  cid 3.129_head oid 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:24.089547 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:47:24.091303 7f9d7e96a700 10 _generic_write 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 0~532480 = 0
2014-06-12 14:47:24.091383 7f9d7e96a700 15 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:24.091399 7f9d7e96a700 10 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:47:24.091404 7f9d7e96a700 15 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:24.091409 7f9d7e96a700 10 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:47:24.091556 7f9d7e96a700 20 set_header setting 17871 cid 3.129_head oid 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 parent seq 0
2014-06-12 14:47:24.091613 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:24.091624 7f9d7e96a700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:24.091633 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:47:24.093565 7f9d7e96a700 10 _do_op 0x76951a0 seq 2589 r = 0, finisher 0xc529000 0
2014-06-12 14:47:24.093576 7f9d7e96a700 10 _finish_op 0x76951a0 seq 2589 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:24.093580 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:24.093586 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:24.093645 7f9d7795c700  5 -- op tracker -- , seq: 748434, time: 2014-06-12 14:47:24.093629, event: sub_op_applied, op: osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.093654 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2876 (0'0,173'2876] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active] sub_op_modify_applied on 0x127976c0 op osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.093691 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:24.093722 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2876 (0'0,173'2876] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active] op_applied on primary on version 173'2876
2014-06-12 14:47:24.093738 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2876 (0'0,173'2876] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active] op_applied on replica on version 173'2876
2014-06-12 14:47:24.093810 7f9d7715b700  5 -- op tracker -- , seq: 748434, time: 2014-06-12 14:47:24.093796, event: commit_sent, op: osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.093834 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2876 (0'0,173'2876] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2874 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:24.093866 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:24.093913 7f9d7715b700  5 -- op tracker -- , seq: 748434, time: 2014-06-12 14:47:24.093906, event: done, op: osd_sub_op(client.4314.0:145055 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2876 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.113846 7f9d5a16f700  5 -- op tracker -- , seq: 748435, time: 2014-06-12 14:47:24.111093, event: header_read, op: osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.113871 7f9d5a16f700  5 -- op tracker -- , seq: 748435, time: 2014-06-12 14:47:24.111098, event: throttled, op: osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.113906 7f9d5a16f700  5 -- op tracker -- , seq: 748435, time: 2014-06-12 14:47:24.113778, event: all_read, op: osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.113927 7f9d5a16f700  5 -- op tracker -- , seq: 748435, time: 0.000000, event: dispatched, op: osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.113936 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:24.113945 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:24.113956 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329ea100 prio 127 cost 1032999 latency 0.002863 osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.114057 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:24.114082 7f9d683e1700 10 osd.2 173 dequeue_op 0x329ea100 prio 127 cost 1032999 latency 0.002989 osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2677 (0'0,173'2677] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active]
2014-06-12 14:47:24.114135 7f9d683e1700  5 -- op tracker -- , seq: 748435, time: 2014-06-12 14:47:24.114125, event: reached_pg, op: osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.114154 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2677 (0'0,173'2677] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active] handle_message: 0x329ea100
2014-06-12 14:47:24.114182 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2677 (0'0,173'2677] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active] sub_op_modify trans c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 v 173'2678 (transaction) 174
2014-06-12 14:47:24.114217 7f9d683e1700  5 -- op tracker -- , seq: 748435, time: 2014-06-12 14:47:24.114208, event: started, op: osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.114249 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2677 (0'0,173'2677] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active] append_log log((0'0,173'2677], crt=0'0) [173'2678 (173'2677) modify   c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 by client.4285.0:143415 2014-06-12 08:43:27.878098]
2014-06-12 14:47:24.114301 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2678 (0'0,173'2678] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active] add_log_entry 173'2678 (173'2677) modify   c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 by client.4285.0:143415 2014-06-12 08:43:27.878098
2014-06-12 14:47:24.114350 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2678 (0'0,173'2678] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active] append_log  adding 1 keys
2014-06-12 14:47:24.114433 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2678, trimmed: 
2014-06-12 14:47:24.114483 7f9d683e1700  5 -- op tracker -- , seq: 748435, time: 2014-06-12 14:47:24.114472, event: started, op: osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.114506 7f9d683e1700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:24.114513 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:24.114516 7f9d683e1700  5 queue_op 0x1ab063c0 seq 2488 osr(3.1b7 0x276d1a0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:24.114533 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:24.114545 7f9d8116f700  5 _do_op 0x1ab063c0 seq 2488 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:24.114551 7f9d8116f700 10 _do_transaction on 0xc9a5b30
2014-06-12 14:47:24.114528 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2678 (0'0,173'2678] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active] do_sub_op osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.114560 7f9d8116f700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:24.114560 7f9d683e1700 10 osd.2 173 dequeue_op 0x329ea100 finish
2014-06-12 14:47:24.114568 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:24.114573 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:24.114798 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:24.114835 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:24.114901 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:24.114933 7f9d8116f700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:24.114943 7f9d8116f700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:24.114952 7f9d8116f700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:24.114975 7f9d8116f700 15 _write 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 1662976~1032192
2014-06-12 14:47:24.115156 7f9d8116f700 10 lookup_strip_header done  cid 3.1b7_head oid c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:24.115477 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:47:24.121599 7f9d8116f700 10 _generic_write 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 1662976~1032192 = 0
2014-06-12 14:47:24.121903 7f9d8116f700 15 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:24.121960 7f9d8116f700 10 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 = 0
2014-06-12 14:47:24.121973 7f9d8116f700 15 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:24.121991 7f9d8116f700 10 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 = 0
2014-06-12 14:47:24.122774 7f9d8116f700 20 set_header setting 17534 cid 3.1b7_head oid c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 parent seq 0
2014-06-12 14:47:24.123060 7f9d8116f700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:24.123090 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:24.123108 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:47:24.132500 7f9d8116f700 10 _do_op 0x1ab063c0 seq 2488 r = 0, finisher 0x13bb7ca0 0
2014-06-12 14:47:24.132523 7f9d8116f700 10 _finish_op 0x1ab063c0 seq 2488 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:24.132533 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:24.132539 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:24.132559 7f9d7795c700  5 -- op tracker -- , seq: 748435, time: 2014-06-12 14:47:24.132546, event: sub_op_applied, op: osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.132570 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2678 (0'0,173'2678] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active] sub_op_modify_applied on 0xc9a5a40 op osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.132606 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:24.132630 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2678 (0'0,173'2678] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active] op_applied on primary on version 173'2678
2014-06-12 14:47:24.132642 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2678 (0'0,173'2678] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active] op_applied on replica on version 173'2678
2014-06-12 14:47:24.132719 7f9d7715b700  5 -- op tracker -- , seq: 748435, time: 2014-06-12 14:47:24.132713, event: commit_sent, op: osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.132737 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2678 (0'0,173'2678] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2676 active] sub_op_modify_commit on op osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:24.132757 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:24.132785 7f9d7715b700  5 -- op tracker -- , seq: 748435, time: 2014-06-12 14:47:24.132780, event: done, op: osd_sub_op(client.4285.0:143415 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2678 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.156411 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:24.170637 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:24.179071 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:24.182832 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:24.186796 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:24.206929 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:24.230548 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,7])
2014-06-12 14:47:24.230563 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,7])
2014-06-12 14:47:24.230932 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:24.230964 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:24.230984 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:24.231031 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:24.231049 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:24.231070 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:24.231100 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:24.231134 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:24.231151 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:24.231182 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:24.231206 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:24.231226 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:24.231249 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:24.231362 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:24.231387 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:24.231416 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:24.231448 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:24.231483 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:24.231504 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:24.231515 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:24.231526 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:24.231535 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:24.231546 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:24.231562 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:24.249850 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:24.249994 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:24.317360 7f9d5a16f700  5 -- op tracker -- , seq: 748436, time: 2014-06-12 14:47:24.315552, event: header_read, op: osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.317387 7f9d5a16f700  5 -- op tracker -- , seq: 748436, time: 2014-06-12 14:47:24.315554, event: throttled, op: osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.317396 7f9d5a16f700  5 -- op tracker -- , seq: 748436, time: 2014-06-12 14:47:24.317302, event: all_read, op: osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.317403 7f9d5a16f700  5 -- op tracker -- , seq: 748436, time: 0.000000, event: dispatched, op: osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.317409 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:24.317417 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:24.317427 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329ebb00 prio 127 cost 1032999 latency 0.001874 osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.317495 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:24.317507 7f9d6abe6700 10 osd.2 173 dequeue_op 0x329ebb00 prio 127 cost 1032999 latency 0.001955 osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.760( v 173'2433 (0'0,173'2433] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2432 active]
2014-06-12 14:47:24.317530 7f9d6abe6700  5 -- op tracker -- , seq: 748436, time: 2014-06-12 14:47:24.317527, event: reached_pg, op: osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.317542 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2433 (0'0,173'2433] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2432 active] handle_message: 0x329ebb00
2014-06-12 14:47:24.317552 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2433 (0'0,173'2433] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2432 active] sub_op_modify trans 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 v 173'2434 (transaction) 174
2014-06-12 14:47:24.317567 7f9d6abe6700  5 -- op tracker -- , seq: 748436, time: 2014-06-12 14:47:24.317564, event: started, op: osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.317583 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2433 (0'0,173'2433] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2432 active] append_log log((0'0,173'2433], crt=0'0) [173'2434 (173'2433) modify   816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 by client.4301.0:145112 2014-06-12 08:44:14.992317]
2014-06-12 14:47:24.317603 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2434 (0'0,173'2434] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2432 active] add_log_entry 173'2434 (173'2433) modify   816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 by client.4301.0:145112 2014-06-12 08:44:14.992317
2014-06-12 14:47:24.317635 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2434 (0'0,173'2434] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2432 active] append_log  adding 1 keys
2014-06-12 14:47:24.317664 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2434, trimmed: 
2014-06-12 14:47:24.317683 7f9d6abe6700  5 -- op tracker -- , seq: 748436, time: 2014-06-12 14:47:24.317680, event: started, op: osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.317695 7f9d6abe6700  5 queue_transactions existing osr(3.760 0x49b41b0)/0x49b41b0
2014-06-12 14:47:24.317700 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:24.317701 7f9d6abe6700  5 queue_op 0x9286660 seq 2247 osr(3.760 0x49b41b0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:24.317706 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.760( v 173'2434 (0'0,173'2434] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2432 active] do_sub_op osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.317716 7f9d6abe6700 10 osd.2 173 dequeue_op 0x329ebb00 finish
2014-06-12 14:47:24.317712 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x55217a0 (1 active)
2014-06-12 14:47:24.317718 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:24.317720 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:24.317720 7f9d7c165700  5 _do_op 0x9286660 seq 2247 osr(3.760 0x49b41b0)/0x49b41b0 start
2014-06-12 14:47:24.317725 7f9d7c165700 10 _do_transaction on 0x1b39d0b0
2014-06-12 14:47:24.317732 7f9d7c165700 15 _omap_setkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:24.317865 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:24.317895 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:24.317940 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:24.317957 7f9d7c165700 15 _omap_rmkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:24.317965 7f9d7c165700 10 _omap_rmkeys meta/1fe099ca/pglog_3.760/0//-1 r = 0
2014-06-12 14:47:24.317971 7f9d7c165700 15 _omap_setkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:24.317984 7f9d7c165700 15 _write 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 3031040~1032192
2014-06-12 14:47:24.318107 7f9d7c165700 10 lookup_strip_header done  cid 3.760_head oid 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:24.318284 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:47:24.322797 7f9d7c165700 10 _generic_write 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 3031040~1032192 = 0
2014-06-12 14:47:24.322907 7f9d7c165700 15 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:24.322945 7f9d7c165700 10 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 = 0
2014-06-12 14:47:24.322956 7f9d7c165700 15 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:24.322968 7f9d7c165700 10 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 = 0
2014-06-12 14:47:24.323376 7f9d7c165700 20 set_header setting 17060 cid 3.760_head oid 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 parent seq 0
2014-06-12 14:47:24.323524 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:24.323547 7f9d7c165700 20 set_header setting 2198 cid meta oid 1fe099ca/pglog_3.760/0//-1 parent seq 0
2014-06-12 14:47:24.323569 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:47:24.330662 7f9d7c165700 10 _do_op 0x9286660 seq 2247 r = 0, finisher 0x6de9d80 0
2014-06-12 14:47:24.330683 7f9d7c165700 10 _finish_op 0x9286660 seq 2247 osr(3.760 0x49b41b0)/0x49b41b0
2014-06-12 14:47:24.330700 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x55217a0 (0 active)
2014-06-12 14:47:24.330705 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:24.330704 7f9d7715b700  5 -- op tracker -- , seq: 748436, time: 2014-06-12 14:47:24.330692, event: commit_sent, op: osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.330718 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2434 (0'0,173'2434] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2432 active] sub_op_modify_commit on op osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:24.330744 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:24.330812 7f9d7795c700  5 -- op tracker -- , seq: 748436, time: 2014-06-12 14:47:24.330807, event: sub_op_applied, op: osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.330819 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2434 (0'0,173'2434] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2433 active] sub_op_modify_applied on 0x1b39cfc0 op osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.330830 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2434 (0'0,173'2434] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2433 active] op_applied on primary on version 173'2434
2014-06-12 14:47:24.330836 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2434 (0'0,173'2434] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2433 active] op_applied on replica on version 173'2434
2014-06-12 14:47:24.330862 7f9d7795c700  5 -- op tracker -- , seq: 748436, time: 2014-06-12 14:47:24.330860, event: done, op: osd_sub_op(client.4301.0:145112 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2434 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.337956 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:24.370861 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:24.484384 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:24.484415 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:24.484420 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:24.484422 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:24.484425 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 7; oldest is 15.313650 seconds old
2014-06-12 14:47:24.484458 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:24.546408 7f9d5a16f700  5 -- op tracker -- , seq: 748437, time: 2014-06-12 14:47:24.544072, event: header_read, op: osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.546425 7f9d5a16f700  5 -- op tracker -- , seq: 748437, time: 2014-06-12 14:47:24.544074, event: throttled, op: osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.546436 7f9d5a16f700  5 -- op tracker -- , seq: 748437, time: 2014-06-12 14:47:24.546364, event: all_read, op: osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.546446 7f9d5a16f700  5 -- op tracker -- , seq: 748437, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.546454 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:24.546463 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:24.546473 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a902800 prio 127 cost 1032999 latency 0.002400 osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.546501 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:24.546515 7f9d66bde700 10 osd.2 173 dequeue_op 0x2a902800 prio 127 cost 1032999 latency 0.002442 osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2876 (0'0,173'2876] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2875 active]
2014-06-12 14:47:24.546546 7f9d66bde700  5 -- op tracker -- , seq: 748437, time: 2014-06-12 14:47:24.546541, event: reached_pg, op: osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.546559 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2876 (0'0,173'2876] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2875 active] handle_message: 0x2a902800
2014-06-12 14:47:24.546575 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2876 (0'0,173'2876] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2875 active] sub_op_modify trans 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 v 173'2877 (transaction) 174
2014-06-12 14:47:24.546596 7f9d66bde700  5 -- op tracker -- , seq: 748437, time: 2014-06-12 14:47:24.546591, event: started, op: osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.546615 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2876 (0'0,173'2876] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2875 active] append_log log((0'0,173'2876], crt=0'0) [173'2877 (173'2876) modify   69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 by client.4314.0:145056 2014-06-12 08:45:20.949671]
2014-06-12 14:47:24.546642 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2877 (0'0,173'2877] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2875 active] add_log_entry 173'2877 (173'2876) modify   69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 by client.4314.0:145056 2014-06-12 08:45:20.949671
2014-06-12 14:47:24.546674 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2877 (0'0,173'2877] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2875 active] append_log  adding 1 keys
2014-06-12 14:47:24.546714 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2877, trimmed: 
2014-06-12 14:47:24.546742 7f9d66bde700  5 -- op tracker -- , seq: 748437, time: 2014-06-12 14:47:24.546737, event: started, op: osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.546756 7f9d66bde700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:24.546761 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:24.546763 7f9d66bde700  5 queue_op 0x6853920 seq 2590 osr(3.129 0x276cf20) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:24.546777 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:24.546772 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2877 (0'0,173'2877] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2875 active] do_sub_op osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.546787 7f9d8096e700  5 _do_op 0x6853920 seq 2590 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:24.546788 7f9d66bde700 10 osd.2 173 dequeue_op 0x2a902800 finish
2014-06-12 14:47:24.546792 7f9d8096e700 10 _do_transaction on 0x1cc280f0
2014-06-12 14:47:24.546793 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:24.546796 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:24.546800 7f9d8096e700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:24.546937 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:24.546966 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:24.547035 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:24.547147 7f9d8096e700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:24.547157 7f9d8096e700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:24.547164 7f9d8096e700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:24.547185 7f9d8096e700 15 _write 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 532480~1032192
2014-06-12 14:47:24.547344 7f9d8096e700 10 lookup_strip_header done  cid 3.129_head oid 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:24.547662 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:47:24.550428 7f9d5a16f700  5 -- op tracker -- , seq: 748438, time: 2014-06-12 14:47:24.546502, event: header_read, op: osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.550457 7f9d5a16f700  5 -- op tracker -- , seq: 748438, time: 2014-06-12 14:47:24.546503, event: throttled, op: osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.550485 7f9d5a16f700  5 -- op tracker -- , seq: 748438, time: 2014-06-12 14:47:24.550369, event: all_read, op: osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.550504 7f9d5a16f700  5 -- op tracker -- , seq: 748438, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.550519 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:24.550533 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:24.550549 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903d00 prio 127 cost 516903 latency 0.004047 osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.550633 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:24.550656 7f9d69be4700 10 osd.2 173 dequeue_op 0x2a903d00 prio 127 cost 516903 latency 0.004153 osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.31d( v 173'2483 (0'0,173'2483] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active]
2014-06-12 14:47:24.550695 7f9d69be4700  5 -- op tracker -- , seq: 748438, time: 2014-06-12 14:47:24.550688, event: reached_pg, op: osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.550712 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2483 (0'0,173'2483] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] handle_message: 0x2a903d00
2014-06-12 14:47:24.550728 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2483 (0'0,173'2483] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] sub_op_modify trans 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 v 173'2484 (transaction) 174
2014-06-12 14:47:24.550749 7f9d69be4700  5 -- op tracker -- , seq: 748438, time: 2014-06-12 14:47:24.550744, event: started, op: osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.550771 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2483 (0'0,173'2483] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] append_log log((0'0,173'2483], crt=0'0) [173'2484 (173'2483) modify   5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 by client.4315.0:144552 2014-06-12 08:44:41.727660]
2014-06-12 14:47:24.550819 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2484 (0'0,173'2484] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] add_log_entry 173'2484 (173'2483) modify   5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 by client.4315.0:144552 2014-06-12 08:44:41.727660
2014-06-12 14:47:24.550853 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2484 (0'0,173'2484] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] append_log  adding 1 keys
2014-06-12 14:47:24.550904 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2484, trimmed: 
2014-06-12 14:47:24.550939 7f9d69be4700  5 -- op tracker -- , seq: 748438, time: 2014-06-12 14:47:24.550932, event: started, op: osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.550954 7f9d69be4700  5 queue_transactions existing osr(3.31d 0x276d520)/0x276d520
2014-06-12 14:47:24.550959 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:24.550961 7f9d69be4700  5 queue_op 0x88cb3e0 seq 2302 osr(3.31d 0x276d520) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:47:24.550969 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.31d( v 173'2484 (0'0,173'2484] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] do_sub_op osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.550978 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439440 (2 active)
2014-06-12 14:47:24.550989 7f9d69be4700 10 osd.2 173 dequeue_op 0x2a903d00 finish
2014-06-12 14:47:24.550995 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:24.550998 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:24.550995 7f9d7c966700  5 _do_op 0x88cb3e0 seq 2302 osr(3.31d 0x276d520)/0x276d520 start
2014-06-12 14:47:24.551005 7f9d7c966700 10 _do_transaction on 0x4fe7eb0
2014-06-12 14:47:24.551020 7f9d7c966700 15 _omap_setkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:24.551203 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:24.551257 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:24.554492 7f9d8096e700 10 _generic_write 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 532480~1032192 = 0
2014-06-12 14:47:24.554573 7f9d8096e700 15 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:24.554600 7f9d8096e700 10 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:47:24.554610 7f9d8096e700 15 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:24.554620 7f9d8096e700 10 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:47:24.555003 7f9d8096e700 20 set_header setting 17871 cid 3.129_head oid 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 parent seq 0
2014-06-12 14:47:24.555160 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:24.555182 7f9d8096e700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:24.555196 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:47:24.560198 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:24.560238 7f9d7c966700 15 _omap_rmkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:24.560248 7f9d7c966700 10 _omap_rmkeys meta/1fe1ce3b/pglog_3.31d/0//-1 r = 0
2014-06-12 14:47:24.560256 7f9d7c966700 15 _omap_setkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:24.560279 7f9d8096e700 10 _do_op 0x6853920 seq 2590 r = 0, finisher 0xabf2940 0
2014-06-12 14:47:24.560326 7f9d8096e700 10 _finish_op 0x6853920 seq 2590 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:24.560333 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (1 active)
2014-06-12 14:47:24.560336 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:24.560341 7f9d7715b700  5 -- op tracker -- , seq: 748437, time: 2014-06-12 14:47:24.560329, event: commit_sent, op: osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.560352 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2877 (0'0,173'2877] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2875 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:24.560387 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:24.560446 7f9d7795c700  5 -- op tracker -- , seq: 748437, time: 2014-06-12 14:47:24.560440, event: sub_op_applied, op: osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.560457 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2877 (0'0,173'2877] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2876 active] sub_op_modify_applied on 0x1cc28000 op osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.560480 7f9d7c966700 15 _write 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 2568192~516096
2014-06-12 14:47:24.560481 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2877 (0'0,173'2877] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2876 active] op_applied on primary on version 173'2877
2014-06-12 14:47:24.560488 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2877 (0'0,173'2877] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2876 active] op_applied on replica on version 173'2877
2014-06-12 14:47:24.560507 7f9d7795c700  5 -- op tracker -- , seq: 748437, time: 2014-06-12 14:47:24.560505, event: done, op: osd_sub_op(client.4314.0:145056 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2877 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.560612 7f9d7c966700 10 lookup_strip_header done  cid 3.31d_head oid 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:24.560731 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:47:24.561783 7f9d7c966700 10 _generic_write 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 2568192~516096 = 0
2014-06-12 14:47:24.561812 7f9d7c966700 15 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:24.561827 7f9d7c966700 10 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 = 0
2014-06-12 14:47:24.561834 7f9d7c966700 15 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:24.561841 7f9d7c966700 10 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 = 0
2014-06-12 14:47:24.561965 7f9d7c966700 20 set_header setting 17151 cid 3.31d_head oid 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 parent seq 0
2014-06-12 14:47:24.562014 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:24.562029 7f9d7c966700 20 set_header setting 2268 cid meta oid 1fe1ce3b/pglog_3.31d/0//-1 parent seq 0
2014-06-12 14:47:24.562039 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:47:24.564735 7f9d7c966700 10 _do_op 0x88cb3e0 seq 2302 r = 0, finisher 0x5d407c0 0
2014-06-12 14:47:24.564752 7f9d7c966700 10 _finish_op 0x88cb3e0 seq 2302 osr(3.31d 0x276d520)/0x276d520
2014-06-12 14:47:24.564759 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439440 (0 active)
2014-06-12 14:47:24.564775 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:24.564791 7f9d7795c700  5 -- op tracker -- , seq: 748438, time: 2014-06-12 14:47:24.564779, event: sub_op_applied, op: osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.564814 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2484 (0'0,173'2484] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] sub_op_modify_applied on 0x4fe7dc0 op osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.564844 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:24.564869 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2484 (0'0,173'2484] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] op_applied on primary on version 173'2484
2014-06-12 14:47:24.564881 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2484 (0'0,173'2484] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] op_applied on replica on version 173'2484
2014-06-12 14:47:24.564915 7f9d7715b700  5 -- op tracker -- , seq: 748438, time: 2014-06-12 14:47:24.564908, event: commit_sent, op: osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.564936 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2484 (0'0,173'2484] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:24.564965 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:24.564993 7f9d7715b700  5 -- op tracker -- , seq: 748438, time: 2014-06-12 14:47:24.564989, event: done, op: osd_sub_op(client.4315.0:144552 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.664295 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:24.664362 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:24.706337 7f9d5a16f700  5 -- op tracker -- , seq: 748439, time: 2014-06-12 14:47:24.705661, event: header_read, op: osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.706348 7f9d5a16f700  5 -- op tracker -- , seq: 748439, time: 2014-06-12 14:47:24.705663, event: throttled, op: osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.706369 7f9d5a16f700  5 -- op tracker -- , seq: 748439, time: 2014-06-12 14:47:24.706284, event: all_read, op: osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.706375 7f9d5a16f700  5 -- op tracker -- , seq: 748439, time: 0.000000, event: dispatched, op: osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.706379 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:24.706384 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:24.706391 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903300 prio 127 cost 516903 latency 0.000729 osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.706454 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:24.706470 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a903300 prio 127 cost 516903 latency 0.000808 osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2678 (0'0,173'2678] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2677 active]
2014-06-12 14:47:24.706496 7f9d64bda700  5 -- op tracker -- , seq: 748439, time: 2014-06-12 14:47:24.706492, event: reached_pg, op: osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.706507 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2678 (0'0,173'2678] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2677 active] handle_message: 0x2a903300
2014-06-12 14:47:24.706521 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2678 (0'0,173'2678] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2677 active] sub_op_modify trans c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 v 173'2679 (transaction) 174
2014-06-12 14:47:24.706534 7f9d64bda700  5 -- op tracker -- , seq: 748439, time: 2014-06-12 14:47:24.706531, event: started, op: osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.706548 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2678 (0'0,173'2678] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2677 active] append_log log((0'0,173'2678], crt=0'0) [173'2679 (173'2678) modify   c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 by client.4285.0:143416 2014-06-12 08:43:28.617822]
2014-06-12 14:47:24.706570 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2679 (0'0,173'2679] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2677 active] add_log_entry 173'2679 (173'2678) modify   c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 by client.4285.0:143416 2014-06-12 08:43:28.617822
2014-06-12 14:47:24.706590 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2679 (0'0,173'2679] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2677 active] append_log  adding 1 keys
2014-06-12 14:47:24.706617 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2679, trimmed: 
2014-06-12 14:47:24.706638 7f9d64bda700  5 -- op tracker -- , seq: 748439, time: 2014-06-12 14:47:24.706634, event: started, op: osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.706648 7f9d64bda700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:24.706652 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:24.706653 7f9d64bda700  5 queue_op 0x7be4000 seq 2489 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:24.706661 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:24.706667 7f9d81970700  5 _do_op 0x7be4000 seq 2489 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:24.706659 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2679 (0'0,173'2679] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2677 active] do_sub_op osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.706670 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a903300 finish
2014-06-12 14:47:24.706671 7f9d81970700 10 _do_transaction on 0x2a29beb0
2014-06-12 14:47:24.706672 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:24.706674 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:24.706677 7f9d81970700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:24.707442 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:24.707464 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:24.707503 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:24.707516 7f9d81970700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:24.707521 7f9d81970700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:24.707525 7f9d81970700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:24.707538 7f9d81970700 15 _write 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 2695168~516096
2014-06-12 14:47:24.707796 7f9d81970700 10 lookup_strip_header done  cid 3.1b7_head oid c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:24.707934 7f9d81970700 10 file_to_extents done 
2014-06-12 14:47:24.709156 7f9d81970700 10 _generic_write 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 2695168~516096 = 0
2014-06-12 14:47:24.709186 7f9d81970700 15 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:24.709195 7f9d81970700 10 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 = 0
2014-06-12 14:47:24.709199 7f9d81970700 15 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:24.709204 7f9d81970700 10 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 = 0
2014-06-12 14:47:24.709337 7f9d81970700 20 set_header setting 17534 cid 3.1b7_head oid c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 parent seq 0
2014-06-12 14:47:24.709377 7f9d81970700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:24.709387 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:24.709394 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:47:24.711912 7f9d81970700 10 _do_op 0x7be4000 seq 2489 r = 0, finisher 0x5242640 0
2014-06-12 14:47:24.711947 7f9d81970700 10 _finish_op 0x7be4000 seq 2489 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:24.711958 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:24.711969 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:24.711985 7f9d7795c700  5 -- op tracker -- , seq: 748439, time: 2014-06-12 14:47:24.711972, event: sub_op_applied, op: osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.712018 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2679 (0'0,173'2679] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2677 active] sub_op_modify_applied on 0x2a29bdc0 op osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:24.712063 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:24.712088 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2679 (0'0,173'2679] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2677 active] op_applied on primary on version 173'2679
2014-06-12 14:47:24.712102 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2679 (0'0,173'2679] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2677 active] op_applied on replica on version 173'2679
2014-06-12 14:47:24.712129 7f9d7715b700  5 -- op tracker -- , seq: 748439, time: 2014-06-12 14:47:24.712123, event: commit_sent, op: osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.712145 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2679 (0'0,173'2679] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2677 active] sub_op_modify_commit on op osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:24.712166 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:24.712204 7f9d7715b700  5 -- op tracker -- , seq: 748439, time: 2014-06-12 14:47:24.712198, event: done, op: osd_sub_op(client.4285.0:143416 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2679 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:24.943819 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:25.002060 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:25.161901 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:25.163757 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:25.167132 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:25.172754 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:25.361672 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:25.366811 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:25.409125 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:25.409188 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:25.481149 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:25.481636 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:25.484488 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:25.484504 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:25.484507 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:25.484509 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:25.484512 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 7; oldest is 16.313736 seconds old
2014-06-12 14:47:25.484538 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:25.520255 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:25.526940 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:25.645921 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:25.662057 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:25.717243 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:25.722826 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:25.723391 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:25.727468 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:25.738702 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:25.806798 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:25.918869 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:25.924023 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:25.929635 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:25.929679 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:25.950529 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:25.950607 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:26.015557 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:26.015613 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:26.089060 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:26.093637 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:26.114643 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:26.132626 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:26.145983 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:26.146084 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:26.156511 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:26.170720 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:26.179153 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:26.182940 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:26.186878 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:26.207014 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:26.317795 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:26.330776 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:26.338053 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:26.370967 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:26.484588 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:26.484620 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:26.484625 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:26.484627 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:26.484629 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 7; oldest is 17.313854 seconds old
2014-06-12 14:47:26.484665 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:26.546873 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:26.551087 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:26.560406 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:26.564835 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:26.646677 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:26.646737 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:26.706762 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:26.712032 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:26.777220 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:26.777468 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:26.943906 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:26.966230 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:26.966237 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:26.966241 7f9d74bfa700 20 osd.2 173 _dispatch 0x56bd6c0 ping v1
2014-06-12 14:47:26.966246 7f9d74bfa700 10 osd.2 173 ping from client.4316
2014-06-12 14:47:26.966250 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:26.966253 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:27.002143 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:27.131168 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:27.131187 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:27.161985 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:27.163838 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:27.167195 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:27.172823 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:27.261255 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:27.261333 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:27.361755 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:27.366903 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:27.481238 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:27.481713 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:27.484703 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:27.484725 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:47:27.484729 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:47:27.484734 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:47:27.484756 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:27.484783 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 7.38 >= max 0.5 = no, load too high
2014-06-12 14:47:27.484799 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:47:27.484803 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92695 < max (604800 seconds)
2014-06-12 14:47:27.484834 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:47:27.484837 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:27.484838 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:27.484841 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 7; oldest is 18.314066 seconds old
2014-06-12 14:47:27.520336 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:27.527036 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:27.646003 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:27.653901 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:27.654137 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:27.662134 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:27.717317 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:27.722896 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:27.723462 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:27.727548 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:27.738794 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:27.806880 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:27.918945 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:27.924098 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:28.089137 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:28.093733 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:28.108094 7f9d5b6bc700  5 -- op tracker -- , seq: 748440, time: 2014-06-12 14:47:28.107463, event: header_read, op: osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.108114 7f9d5b6bc700  5 -- op tracker -- , seq: 748440, time: 2014-06-12 14:47:28.107470, event: throttled, op: osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.108126 7f9d5b6bc700  5 -- op tracker -- , seq: 748440, time: 2014-06-12 14:47:28.108030, event: all_read, op: osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.108156 7f9d5b6bc700  5 -- op tracker -- , seq: 748440, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.108164 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:28.108173 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:47:28.108184 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x20e87300 prio 127 cost 283431 latency 0.000720 osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.108232 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:28.108252 7f9d683e1700 10 osd.2 173 dequeue_op 0x20e87300 prio 127 cost 283431 latency 0.000789 osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.228( v 173'2987 (0'0,173'2987] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active]
2014-06-12 14:47:28.108308 7f9d683e1700  5 -- op tracker -- , seq: 748440, time: 2014-06-12 14:47:28.108297, event: reached_pg, op: osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.108327 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2987 (0'0,173'2987] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] handle_message: 0x20e87300
2014-06-12 14:47:28.108366 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2987 (0'0,173'2987] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] sub_op_modify trans f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 v 173'2988 (transaction) 174
2014-06-12 14:47:28.108416 7f9d683e1700  5 -- op tracker -- , seq: 748440, time: 2014-06-12 14:47:28.108412, event: started, op: osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.108431 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2987 (0'0,173'2987] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] append_log log((0'0,173'2987], crt=0'0) [173'2988 (173'821) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145670 2014-06-12 08:46:30.484746]
2014-06-12 14:47:28.108462 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2988 (0'0,173'2988] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] add_log_entry 173'2988 (173'821) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145670 2014-06-12 08:46:30.484746
2014-06-12 14:47:28.108518 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2988 (0'0,173'2988] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] append_log  adding 1 keys
2014-06-12 14:47:28.108595 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2988, trimmed: 
2014-06-12 14:47:28.108647 7f9d683e1700  5 -- op tracker -- , seq: 748440, time: 2014-06-12 14:47:28.108634, event: started, op: osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.108671 7f9d683e1700  5 queue_transactions existing osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.108679 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:28.108682 7f9d683e1700  5 queue_op 0x6d1f1a0 seq 2775 osr(3.228 0x276d3e0) 284917 bytes   (queue has 1 ops and 284917 bytes)
2014-06-12 14:47:28.108711 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533d560 (1 active)
2014-06-12 14:47:28.108725 7f9d8116f700  5 _do_op 0x6d1f1a0 seq 2775 osr(3.228 0x276d3e0)/0x276d3e0 start
2014-06-12 14:47:28.108695 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.228( v 173'2988 (0'0,173'2988] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] do_sub_op osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.108732 7f9d8116f700 10 _do_transaction on 0x678f430
2014-06-12 14:47:28.108730 7f9d683e1700 10 osd.2 173 dequeue_op 0x20e87300 finish
2014-06-12 14:47:28.108739 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:28.108741 7f9d8116f700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.108745 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:28.108885 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.108916 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:28.108988 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:28.109023 7f9d8116f700 15 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.109031 7f9d8116f700 10 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1 r = 0
2014-06-12 14:47:28.109039 7f9d8116f700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.109237 7f9d8116f700 15 _write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 0~282624
2014-06-12 14:47:28.109493 7f9d8116f700 10 lookup_strip_header done  cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.109720 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:47:28.111019 7f9d8116f700 10 _generic_write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 0~282624 = 0
2014-06-12 14:47:28.111072 7f9d8116f700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.111088 7f9d8116f700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.111096 7f9d8116f700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.111106 7f9d8116f700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.111315 7f9d8116f700 20 set_header setting 17767 cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 parent seq 0
2014-06-12 14:47:28.111378 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:28.111394 7f9d8116f700 20 set_header setting 2274 cid meta oid 1fe2503a/pglog_3.228/0//-1 parent seq 0
2014-06-12 14:47:28.111407 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:47:28.113098 7f9d8116f700 10 _do_op 0x6d1f1a0 seq 2775 r = 0, finisher 0x786c200 0
2014-06-12 14:47:28.113118 7f9d8116f700 10 _finish_op 0x6d1f1a0 seq 2775 osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.113127 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533d560 (0 active)
2014-06-12 14:47:28.113132 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:28.113187 7f9d7795c700  5 -- op tracker -- , seq: 748440, time: 2014-06-12 14:47:28.113166, event: sub_op_applied, op: osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.113206 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2988 (0'0,173'2988] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] sub_op_modify_applied on 0x678f340 op osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.113259 7f9d7795c700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.113275 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2988 (0'0,173'2988] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] op_applied on primary on version 173'2988
2014-06-12 14:47:28.113283 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2988 (0'0,173'2988] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] op_applied on replica on version 173'2988
2014-06-12 14:47:28.113301 7f9d7715b700  5 -- op tracker -- , seq: 748440, time: 2014-06-12 14:47:28.113296, event: commit_sent, op: osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.113324 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2988 (0'0,173'2988] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2986 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:47:28.113350 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.113411 7f9d7715b700  5 -- op tracker -- , seq: 748440, time: 2014-06-12 14:47:28.113403, event: done, op: osd_sub_op(client.4299.0:145670 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2988 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.114169 7f9d5b6bc700  5 -- op tracker -- , seq: 748441, time: 2014-06-12 14:47:28.112795, event: header_read, op: osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.114197 7f9d5b6bc700  5 -- op tracker -- , seq: 748441, time: 2014-06-12 14:47:28.112798, event: throttled, op: osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.114218 7f9d5b6bc700  5 -- op tracker -- , seq: 748441, time: 2014-06-12 14:47:28.114116, event: all_read, op: osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.114244 7f9d5b6bc700  5 -- op tracker -- , seq: 748441, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.114255 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:28.114270 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:47:28.114284 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x20e87200 prio 127 cost 516903 latency 0.001488 osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.114391 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:28.114412 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20e87200 prio 127 cost 516903 latency 0.001616 osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.228( v 173'2988 (0'0,173'2988] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active]
2014-06-12 14:47:28.114447 7f9d6abe6700  5 -- op tracker -- , seq: 748441, time: 2014-06-12 14:47:28.114441, event: reached_pg, op: osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.114497 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2988 (0'0,173'2988] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] handle_message: 0x20e87200
2014-06-12 14:47:28.114516 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2988 (0'0,173'2988] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] sub_op_modify trans f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 v 173'2989 (transaction) 174
2014-06-12 14:47:28.114539 7f9d6abe6700  5 -- op tracker -- , seq: 748441, time: 2014-06-12 14:47:28.114533, event: started, op: osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.114568 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2988 (0'0,173'2988] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] append_log log((0'0,173'2988], crt=0'0) [173'2989 (173'2988) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145671 2014-06-12 08:46:30.489753]
2014-06-12 14:47:28.114619 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2989 (0'0,173'2989] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] add_log_entry 173'2989 (173'2988) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145671 2014-06-12 08:46:30.489753
2014-06-12 14:47:28.114653 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2989 (0'0,173'2989] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] append_log  adding 1 keys
2014-06-12 14:47:28.114711 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2989, trimmed: 
2014-06-12 14:47:28.114746 7f9d6abe6700  5 -- op tracker -- , seq: 748441, time: 2014-06-12 14:47:28.114738, event: started, op: osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.114763 7f9d6abe6700  5 queue_transactions existing osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.114768 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:28.114770 7f9d6abe6700  5 queue_op 0x8097560 seq 2776 osr(3.228 0x276d3e0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:28.114778 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.228( v 173'2989 (0'0,173'2989] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] do_sub_op osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.114798 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20e87200 finish
2014-06-12 14:47:28.114804 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:28.114807 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:28.114797 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533d560 (1 active)
2014-06-12 14:47:28.114814 7f9d7d167700  5 _do_op 0x8097560 seq 2776 osr(3.228 0x276d3e0)/0x276d3e0 start
2014-06-12 14:47:28.114824 7f9d7d167700 10 _do_transaction on 0x1b39c2b0
2014-06-12 14:47:28.114839 7f9d7d167700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.115030 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.115075 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:28.115171 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:28.115223 7f9d7d167700 15 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.115240 7f9d7d167700 10 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1 r = 0
2014-06-12 14:47:28.115254 7f9d7d167700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.115432 7f9d7d167700 15 _write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 282624~516096
2014-06-12 14:47:28.115648 7f9d7d167700 10 lookup_strip_header done  cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.115813 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:47:28.118073 7f9d7d167700 10 _generic_write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 282624~516096 = 0
2014-06-12 14:47:28.118171 7f9d7d167700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.118201 7f9d7d167700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.118215 7f9d7d167700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.118234 7f9d7d167700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.118527 7f9d7d167700 20 set_header setting 17767 cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 parent seq 0
2014-06-12 14:47:28.118658 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:28.118692 7f9d7d167700 20 set_header setting 2274 cid meta oid 1fe2503a/pglog_3.228/0//-1 parent seq 0
2014-06-12 14:47:28.118715 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:47:28.121798 7f9d5b6bc700  5 -- op tracker -- , seq: 748442, time: 2014-06-12 14:47:28.121015, event: header_read, op: osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.121810 7f9d5b6bc700  5 -- op tracker -- , seq: 748442, time: 2014-06-12 14:47:28.121018, event: throttled, op: osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.121871 7f9d5b6bc700  5 -- op tracker -- , seq: 748442, time: 2014-06-12 14:47:28.121754, event: all_read, op: osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.121878 7f9d5b6bc700  5 -- op tracker -- , seq: 748442, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.121884 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:28.121889 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:47:28.121897 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x1e3eb300 prio 127 cost 516903 latency 0.000881 osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.121929 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:28.121949 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x1e3eb300 prio 127 cost 516903 latency 0.000933 osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.228( v 173'2989 (0'0,173'2989] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active]
2014-06-12 14:47:28.121977 7f9d6a3e5700  5 -- op tracker -- , seq: 748442, time: 2014-06-12 14:47:28.121972, event: reached_pg, op: osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.121987 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2989 (0'0,173'2989] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] handle_message: 0x1e3eb300
2014-06-12 14:47:28.121997 7f9d7d167700 10 _do_op 0x8097560 seq 2776 r = 0, finisher 0x8356ee0 0
2014-06-12 14:47:28.121999 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2989 (0'0,173'2989] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] sub_op_modify trans f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 v 173'2990 (transaction) 174
2014-06-12 14:47:28.122010 7f9d7d167700 10 _finish_op 0x8097560 seq 2776 osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.122016 7f9d6a3e5700  5 -- op tracker -- , seq: 748442, time: 2014-06-12 14:47:28.122011, event: started, op: osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.122017 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533d560 (0 active)
2014-06-12 14:47:28.122026 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:28.122031 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2989 (0'0,173'2989] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] append_log log((0'0,173'2989], crt=0'0) [173'2990 (173'2989) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145672 2014-06-12 08:46:30.497957]
2014-06-12 14:47:28.122065 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] add_log_entry 173'2990 (173'2989) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145672 2014-06-12 08:46:30.497957
2014-06-12 14:47:28.122114 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] append_log  adding 1 keys
2014-06-12 14:47:28.122182 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2990, trimmed: 
2014-06-12 14:47:28.122234 7f9d6a3e5700  5 -- op tracker -- , seq: 748442, time: 2014-06-12 14:47:28.122223, event: started, op: osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.122274 7f9d6a3e5700  5 queue_transactions existing osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.122281 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:28.122284 7f9d6a3e5700  5 queue_op 0x641c000 seq 2777 osr(3.228 0x276d3e0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:28.122302 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533d560 (1 active)
2014-06-12 14:47:28.122312 7f9d7915f700  5 _do_op 0x641c000 seq 2777 osr(3.228 0x276d3e0)/0x276d3e0 start
2014-06-12 14:47:28.122318 7f9d7915f700 10 _do_transaction on 0xc9a5430
2014-06-12 14:47:28.122299 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] do_sub_op osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.122327 7f9d7915f700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.122329 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x1e3eb300 finish
2014-06-12 14:47:28.122342 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:28.122347 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:28.122363 7f9d7715b700  5 -- op tracker -- , seq: 748441, time: 2014-06-12 14:47:28.122350, event: commit_sent, op: osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.122384 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2987 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:47:28.122404 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.122461 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.122482 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:28.122514 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:28.122616 7f9d7915f700 15 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.122622 7f9d7915f700 10 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1 r = 0
2014-06-12 14:47:28.122627 7f9d7915f700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.122639 7f9d7915f700 15 _write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 798720~516096
2014-06-12 14:47:28.122739 7f9d7915f700 10 lookup_strip_header done  cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.123022 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:47:28.124265 7f9d7915f700 10 _generic_write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 798720~516096 = 0
2014-06-12 14:47:28.124309 7f9d7915f700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.124324 7f9d7915f700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.124335 7f9d7915f700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.124345 7f9d7915f700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.124486 7f9d7915f700 20 set_header setting 17767 cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 parent seq 0
2014-06-12 14:47:28.124541 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:28.124557 7f9d7915f700 20 set_header setting 2274 cid meta oid 1fe2503a/pglog_3.228/0//-1 parent seq 0
2014-06-12 14:47:28.124568 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:47:28.125673 7f9d7795c700  5 -- op tracker -- , seq: 748441, time: 2014-06-12 14:47:28.125663, event: sub_op_applied, op: osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.125685 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] sub_op_modify_applied on 0x1b39c1c0 op osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.125704 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] op_applied on primary on version 173'2989
2014-06-12 14:47:28.125711 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] op_applied on replica on version 173'2989
2014-06-12 14:47:28.125729 7f9d7795c700  5 -- op tracker -- , seq: 748441, time: 2014-06-12 14:47:28.125726, event: done, op: osd_sub_op(client.4299.0:145671 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2989 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.126731 7f9d7915f700 10 _do_op 0x641c000 seq 2777 r = 0, finisher 0x2095980 0
2014-06-12 14:47:28.126746 7f9d7915f700 10 _finish_op 0x641c000 seq 2777 osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.126754 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533d560 (0 active)
2014-06-12 14:47:28.126758 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:28.126785 7f9d7715b700  5 -- op tracker -- , seq: 748442, time: 2014-06-12 14:47:28.126779, event: commit_sent, op: osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.126806 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2988 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:47:28.126855 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.126915 7f9d7795c700  5 -- op tracker -- , seq: 748442, time: 2014-06-12 14:47:28.126908, event: sub_op_applied, op: osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.126925 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] sub_op_modify_applied on 0xc9a5340 op osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.126939 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] op_applied on primary on version 173'2990
2014-06-12 14:47:28.126947 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] op_applied on replica on version 173'2990
2014-06-12 14:47:28.126967 7f9d7795c700  5 -- op tracker -- , seq: 748442, time: 2014-06-12 14:47:28.126964, event: done, op: osd_sub_op(client.4299.0:145672 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2990 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.127670 7f9d5b6bc700  5 -- op tracker -- , seq: 748443, time: 2014-06-12 14:47:28.126911, event: header_read, op: osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.127679 7f9d5b6bc700  5 -- op tracker -- , seq: 748443, time: 2014-06-12 14:47:28.126913, event: throttled, op: osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.127687 7f9d5b6bc700  5 -- op tracker -- , seq: 748443, time: 2014-06-12 14:47:28.127632, event: all_read, op: osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.127692 7f9d5b6bc700  5 -- op tracker -- , seq: 748443, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.127699 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:28.127703 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:47:28.127708 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x329eb800 prio 127 cost 516903 latency 0.000797 osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.127732 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:28.127739 7f9d66bde700 10 osd.2 173 dequeue_op 0x329eb800 prio 127 cost 516903 latency 0.000828 osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active]
2014-06-12 14:47:28.127762 7f9d66bde700  5 -- op tracker -- , seq: 748443, time: 2014-06-12 14:47:28.127759, event: reached_pg, op: osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.127771 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] handle_message: 0x329eb800
2014-06-12 14:47:28.127779 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] sub_op_modify trans f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 v 173'2991 (transaction) 174
2014-06-12 14:47:28.127790 7f9d66bde700  5 -- op tracker -- , seq: 748443, time: 2014-06-12 14:47:28.127787, event: started, op: osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.127801 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2990 (0'0,173'2990] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] append_log log((0'0,173'2990], crt=0'0) [173'2991 (173'2990) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145673 2014-06-12 08:46:30.503844]
2014-06-12 14:47:28.127820 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2991 (0'0,173'2991] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] add_log_entry 173'2991 (173'2990) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145673 2014-06-12 08:46:30.503844
2014-06-12 14:47:28.127835 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2991 (0'0,173'2991] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] append_log  adding 1 keys
2014-06-12 14:47:28.127859 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2991, trimmed: 
2014-06-12 14:47:28.127877 7f9d66bde700  5 -- op tracker -- , seq: 748443, time: 2014-06-12 14:47:28.127874, event: started, op: osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.127886 7f9d66bde700  5 queue_transactions existing osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.127888 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:28.127889 7f9d66bde700  5 queue_op 0x5e6d800 seq 2778 osr(3.228 0x276d3e0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:28.127894 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.228( v 173'2991 (0'0,173'2991] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] do_sub_op osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.127903 7f9d66bde700 10 osd.2 173 dequeue_op 0x329eb800 finish
2014-06-12 14:47:28.127906 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:28.127908 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:28.127914 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533d560 (1 active)
2014-06-12 14:47:28.127921 7f9d8016d700  5 _do_op 0x5e6d800 seq 2778 osr(3.228 0x276d3e0)/0x276d3e0 start
2014-06-12 14:47:28.127924 7f9d8016d700 10 _do_transaction on 0x62c80f0
2014-06-12 14:47:28.127929 7f9d8016d700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.128021 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.128037 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:28.128070 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:28.128087 7f9d8016d700 15 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.128091 7f9d8016d700 10 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1 r = 0
2014-06-12 14:47:28.128094 7f9d8016d700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.128106 7f9d8016d700 15 _write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 1314816~516096
2014-06-12 14:47:28.128202 7f9d8016d700 10 lookup_strip_header done  cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.128432 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:47:28.129123 7f9d5b6bc700  5 -- op tracker -- , seq: 748444, time: 2014-06-12 14:47:28.128287, event: header_read, op: osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.129132 7f9d5b6bc700  5 -- op tracker -- , seq: 748444, time: 2014-06-12 14:47:28.128288, event: throttled, op: osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.129145 7f9d5b6bc700  5 -- op tracker -- , seq: 748444, time: 2014-06-12 14:47:28.129097, event: all_read, op: osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.129150 7f9d5b6bc700  5 -- op tracker -- , seq: 748444, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.129154 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:28.129157 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:47:28.129162 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x329ea900 prio 127 cost 516903 latency 0.000875 osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.129183 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:28.129188 7f9d69be4700 10 osd.2 173 dequeue_op 0x329ea900 prio 127 cost 516903 latency 0.000901 osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.228( v 173'2991 (0'0,173'2991] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active]
2014-06-12 14:47:28.129211 7f9d69be4700  5 -- op tracker -- , seq: 748444, time: 2014-06-12 14:47:28.129208, event: reached_pg, op: osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.129225 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2991 (0'0,173'2991] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] handle_message: 0x329ea900
2014-06-12 14:47:28.129233 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2991 (0'0,173'2991] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] sub_op_modify trans f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 v 173'2992 (transaction) 174
2014-06-12 14:47:28.129243 7f9d69be4700  5 -- op tracker -- , seq: 748444, time: 2014-06-12 14:47:28.129240, event: started, op: osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.129253 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2991 (0'0,173'2991] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] append_log log((0'0,173'2991], crt=0'0) [173'2992 (173'2991) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145674 2014-06-12 08:46:30.504476]
2014-06-12 14:47:28.129268 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2992 (0'0,173'2992] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] add_log_entry 173'2992 (173'2991) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145674 2014-06-12 08:46:30.504476
2014-06-12 14:47:28.129284 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2992 (0'0,173'2992] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] append_log  adding 1 keys
2014-06-12 14:47:28.129305 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2992, trimmed: 
2014-06-12 14:47:28.129324 7f9d69be4700  5 -- op tracker -- , seq: 748444, time: 2014-06-12 14:47:28.129321, event: started, op: osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.129332 7f9d69be4700  5 queue_transactions existing osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.129335 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:28.129335 7f9d69be4700  5 queue_op 0x6fbfce0 seq 2779 osr(3.228 0x276d3e0) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:47:28.129340 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.228( v 173'2992 (0'0,173'2992] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] do_sub_op osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.129349 7f9d69be4700 10 osd.2 173 dequeue_op 0x329ea900 finish
2014-06-12 14:47:28.129351 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:28.129353 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:28.129367 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533d560 (2 active)
2014-06-12 14:47:28.129929 7f9d8016d700 10 _generic_write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 1314816~516096 = 0
2014-06-12 14:47:28.129994 7f9d8016d700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.130017 7f9d8016d700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.130025 7f9d8016d700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.130035 7f9d8016d700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.130300 7f9d8016d700 20 set_header setting 17767 cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 parent seq 0
2014-06-12 14:47:28.130391 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:28.130412 7f9d8016d700 20 set_header setting 2274 cid meta oid 1fe2503a/pglog_3.228/0//-1 parent seq 0
2014-06-12 14:47:28.130425 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:47:28.133487 7f9d8016d700 10 _do_op 0x5e6d800 seq 2778 r = 0, finisher 0x7be78a0 0
2014-06-12 14:47:28.133509 7f9d8016d700 10 _finish_op 0x5e6d800 seq 2778 osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.133517 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533d560 (1 active)
2014-06-12 14:47:28.133521 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:28.133544 7f9d79960700  5 _do_op 0x6fbfce0 seq 2779 osr(3.228 0x276d3e0)/0x276d3e0 start
2014-06-12 14:47:28.133553 7f9d79960700 10 _do_transaction on 0x19d87430
2014-06-12 14:47:28.133562 7f9d79960700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.133560 7f9d7715b700  5 -- op tracker -- , seq: 748443, time: 2014-06-12 14:47:28.133550, event: commit_sent, op: osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.133577 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2992 (0'0,173'2992] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2989 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:47:28.133603 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.133670 7f9d7795c700  5 -- op tracker -- , seq: 748443, time: 2014-06-12 14:47:28.133663, event: sub_op_applied, op: osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.133683 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2992 (0'0,173'2992] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify_applied on 0x62c8000 op osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.133706 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.133704 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2992 (0'0,173'2992] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] op_applied on primary on version 173'2991
2014-06-12 14:47:28.133714 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2992 (0'0,173'2992] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] op_applied on replica on version 173'2991
2014-06-12 14:47:28.133735 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:28.133738 7f9d7795c700  5 -- op tracker -- , seq: 748443, time: 2014-06-12 14:47:28.133733, event: done, op: osd_sub_op(client.4299.0:145673 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2991 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.133782 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:28.133800 7f9d79960700 15 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.133804 7f9d79960700 10 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1 r = 0
2014-06-12 14:47:28.133808 7f9d79960700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.133903 7f9d79960700 15 _write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 1830912~516096
2014-06-12 14:47:28.134111 7f9d79960700 10 lookup_strip_header done  cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.134276 7f9d79960700 10 file_to_extents done 
2014-06-12 14:47:28.136627 7f9d79960700 10 _generic_write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 1830912~516096 = 0
2014-06-12 14:47:28.136709 7f9d79960700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.136731 7f9d79960700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.136739 7f9d79960700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.136750 7f9d79960700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.137017 7f9d79960700 20 set_header setting 17767 cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 parent seq 0
2014-06-12 14:47:28.137126 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:28.137148 7f9d79960700 20 set_header setting 2274 cid meta oid 1fe2503a/pglog_3.228/0//-1 parent seq 0
2014-06-12 14:47:28.137161 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:47:28.137308 7f9d5b6bc700  5 -- op tracker -- , seq: 748445, time: 2014-06-12 14:47:28.134994, event: header_read, op: osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.137341 7f9d5b6bc700  5 -- op tracker -- , seq: 748445, time: 2014-06-12 14:47:28.134996, event: throttled, op: osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.137362 7f9d5b6bc700  5 -- op tracker -- , seq: 748445, time: 2014-06-12 14:47:28.137241, event: all_read, op: osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.137383 7f9d5b6bc700  5 -- op tracker -- , seq: 748445, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.137400 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:28.137408 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:47:28.137416 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x329eae00 prio 127 cost 1032999 latency 0.002422 osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.137447 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:28.137456 7f9d64bda700 10 osd.2 173 dequeue_op 0x329eae00 prio 127 cost 1032999 latency 0.002462 osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.228( v 173'2992 (0'0,173'2992] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active]
2014-06-12 14:47:28.137482 7f9d64bda700  5 -- op tracker -- , seq: 748445, time: 2014-06-12 14:47:28.137477, event: reached_pg, op: osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.137491 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2992 (0'0,173'2992] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] handle_message: 0x329eae00
2014-06-12 14:47:28.137502 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2992 (0'0,173'2992] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify trans f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 v 173'2993 (transaction) 174
2014-06-12 14:47:28.137523 7f9d64bda700  5 -- op tracker -- , seq: 748445, time: 2014-06-12 14:47:28.137518, event: started, op: osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.137538 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2992 (0'0,173'2992] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] append_log log((0'0,173'2992], crt=0'0) [173'2993 (173'2992) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145675 2014-06-12 08:46:30.509321]
2014-06-12 14:47:28.137561 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2993 (0'0,173'2993] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] add_log_entry 173'2993 (173'2992) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145675 2014-06-12 08:46:30.509321
2014-06-12 14:47:28.137578 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2993 (0'0,173'2993] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] append_log  adding 1 keys
2014-06-12 14:47:28.137602 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2993, trimmed: 
2014-06-12 14:47:28.137675 7f9d64bda700  5 -- op tracker -- , seq: 748445, time: 2014-06-12 14:47:28.137671, event: started, op: osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.137686 7f9d64bda700  5 queue_transactions existing osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.137691 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:28.137693 7f9d64bda700  5 queue_op 0x80d26c0 seq 2780 osr(3.228 0x276d3e0) 1034485 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:47:28.137700 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.228( v 173'2993 (0'0,173'2993] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] do_sub_op osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.137716 7f9d64bda700 10 osd.2 173 dequeue_op 0x329eae00 finish
2014-06-12 14:47:28.137720 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:28.137722 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:28.137730 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533d560 (2 active)
2014-06-12 14:47:28.138412 7f9d5b6bc700  5 -- op tracker -- , seq: 748446, time: 2014-06-12 14:47:28.137748, event: header_read, op: osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.138428 7f9d5b6bc700  5 -- op tracker -- , seq: 748446, time: 2014-06-12 14:47:28.137749, event: throttled, op: osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.138437 7f9d5b6bc700  5 -- op tracker -- , seq: 748446, time: 2014-06-12 14:47:28.138383, event: all_read, op: osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.138445 7f9d5b6bc700  5 -- op tracker -- , seq: 748446, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.138451 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:28.138458 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:47:28.138465 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x876e500 prio 127 cost 516903 latency 0.000717 osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.138490 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:28.138496 7f9d653db700 10 osd.2 173 dequeue_op 0x876e500 prio 127 cost 516903 latency 0.000747 osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.228( v 173'2993 (0'0,173'2993] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active]
2014-06-12 14:47:28.138513 7f9d653db700  5 -- op tracker -- , seq: 748446, time: 2014-06-12 14:47:28.138510, event: reached_pg, op: osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.138528 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2993 (0'0,173'2993] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] handle_message: 0x876e500
2014-06-12 14:47:28.138538 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2993 (0'0,173'2993] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify trans f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 v 173'2994 (transaction) 174
2014-06-12 14:47:28.138550 7f9d653db700  5 -- op tracker -- , seq: 748446, time: 2014-06-12 14:47:28.138547, event: started, op: osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.138567 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2993 (0'0,173'2993] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] append_log log((0'0,173'2993], crt=0'0) [173'2994 (173'2993) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145676 2014-06-12 08:46:30.510970]
2014-06-12 14:47:28.138585 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2994 (0'0,173'2994] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] add_log_entry 173'2994 (173'2993) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145676 2014-06-12 08:46:30.510970
2014-06-12 14:47:28.138604 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2994 (0'0,173'2994] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] append_log  adding 1 keys
2014-06-12 14:47:28.138625 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2994, trimmed: 
2014-06-12 14:47:28.138644 7f9d653db700  5 -- op tracker -- , seq: 748446, time: 2014-06-12 14:47:28.138641, event: started, op: osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.138652 7f9d653db700  5 queue_transactions existing osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.138655 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:28.138655 7f9d653db700  5 queue_op 0x142a8c00 seq 2781 osr(3.228 0x276d3e0) 518389 bytes   (queue has 3 ops and 2071263 bytes)
2014-06-12 14:47:28.138663 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533d560 (3 active)
2014-06-12 14:47:28.138660 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.228( v 173'2994 (0'0,173'2994] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] do_sub_op osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.138669 7f9d653db700 10 osd.2 173 dequeue_op 0x876e500 finish
2014-06-12 14:47:28.138671 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:28.138673 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:28.138808 7f9d5b6bc700  5 -- op tracker -- , seq: 748447, time: 2014-06-12 14:47:28.138492, event: header_read, op: osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.138819 7f9d5b6bc700  5 -- op tracker -- , seq: 748447, time: 2014-06-12 14:47:28.138493, event: throttled, op: osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.138828 7f9d5b6bc700  5 -- op tracker -- , seq: 748447, time: 2014-06-12 14:47:28.138787, event: all_read, op: osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.138843 7f9d5b6bc700  5 -- op tracker -- , seq: 748447, time: 0.000000, event: dispatched, op: osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.138848 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:28.138854 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:47:28.138860 7f9d5b6bc700 15 osd.2 173 enqueue_op 0x9facb00 prio 127 cost 299815 latency 0.000368 osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.138883 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:28.138892 7f9d623d5700 10 osd.2 173 dequeue_op 0x9facb00 prio 127 cost 299815 latency 0.000400 osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.228( v 173'2994 (0'0,173'2994] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active]
2014-06-12 14:47:28.138914 7f9d623d5700  5 -- op tracker -- , seq: 748447, time: 2014-06-12 14:47:28.138910, event: reached_pg, op: osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.138922 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2994 (0'0,173'2994] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] handle_message: 0x9facb00
2014-06-12 14:47:28.138934 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2994 (0'0,173'2994] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify trans f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 v 173'2995 (transaction) 174
2014-06-12 14:47:28.138948 7f9d623d5700  5 -- op tracker -- , seq: 748447, time: 2014-06-12 14:47:28.138944, event: started, op: osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.138960 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2994 (0'0,173'2994] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] append_log log((0'0,173'2994], crt=0'0) [173'2995 (173'2994) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145677 2014-06-12 08:46:30.515668]
2014-06-12 14:47:28.138981 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] add_log_entry 173'2995 (173'2994) modify   f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 by client.4299.0:145677 2014-06-12 08:46:30.515668
2014-06-12 14:47:28.139001 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] append_log  adding 1 keys
2014-06-12 14:47:28.139029 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2995, trimmed: 
2014-06-12 14:47:28.139048 7f9d623d5700  5 -- op tracker -- , seq: 748447, time: 2014-06-12 14:47:28.139044, event: started, op: osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.139058 7f9d623d5700  5 queue_transactions existing osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.139061 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:28.139062 7f9d623d5700  5 queue_op 0x6d678c0 seq 2782 osr(3.228 0x276d3e0) 301301 bytes   (queue has 4 ops and 2372564 bytes)
2014-06-12 14:47:28.139070 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x533d560 (4 active)
2014-06-12 14:47:28.139068 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] do_sub_op osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.139081 7f9d623d5700 10 osd.2 173 dequeue_op 0x9facb00 finish
2014-06-12 14:47:28.139084 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:28.139086 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:28.139227 7f9d79960700 10 _do_op 0x6fbfce0 seq 2779 r = 0, finisher 0x541ff20 0
2014-06-12 14:47:28.139238 7f9d79960700 10 _finish_op 0x6fbfce0 seq 2779 osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.139243 7f9d7d968700  5 _do_op 0x80d26c0 seq 2780 osr(3.228 0x276d3e0)/0x276d3e0 start
2014-06-12 14:47:28.139247 7f9d7d968700 10 _do_transaction on 0x2a29a470
2014-06-12 14:47:28.139244 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533d560 (3 active)
2014-06-12 14:47:28.139252 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:28.139252 7f9d7d968700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.139262 7f9d7795c700  5 -- op tracker -- , seq: 748444, time: 2014-06-12 14:47:28.139255, event: sub_op_applied, op: osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.139270 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify_applied on 0x19d87340 op osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.139285 7f9d7795c700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.139299 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] op_applied on primary on version 173'2992
2014-06-12 14:47:28.139306 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] op_applied on replica on version 173'2992
2014-06-12 14:47:28.139328 7f9d7715b700  5 -- op tracker -- , seq: 748444, time: 2014-06-12 14:47:28.139319, event: commit_sent, op: osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.139336 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2990 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:47:28.139352 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.139357 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.139367 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:28.139385 7f9d7715b700  5 -- op tracker -- , seq: 748444, time: 2014-06-12 14:47:28.139380, event: done, op: osd_sub_op(client.4299.0:145674 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2992 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.139413 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:28.139429 7f9d7d968700 15 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.139435 7f9d7d968700 10 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1 r = 0
2014-06-12 14:47:28.139439 7f9d7d968700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.139544 7f9d7d968700 15 _write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 2347008~1032192
2014-06-12 14:47:28.139645 7f9d7d968700 10 lookup_strip_header done  cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.139726 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:47:28.142295 7f9d7d968700 10 _generic_write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 2347008~1032192 = 0
2014-06-12 14:47:28.142369 7f9d7d968700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.142383 7f9d7d968700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.142390 7f9d7d968700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.142404 7f9d7d968700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.142640 7f9d7d968700 20 set_header setting 17767 cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 parent seq 0
2014-06-12 14:47:28.142730 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:28.142745 7f9d7d968700 20 set_header setting 2274 cid meta oid 1fe2503a/pglog_3.228/0//-1 parent seq 0
2014-06-12 14:47:28.142755 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:47:28.146778 7f9d7d968700 10 _do_op 0x80d26c0 seq 2780 r = 0, finisher 0x7854ea0 0
2014-06-12 14:47:28.146800 7f9d7d968700 10 _finish_op 0x80d26c0 seq 2780 osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.146807 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533d560 (2 active)
2014-06-12 14:47:28.146815 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:28.146833 7f9d7c165700  5 _do_op 0x142a8c00 seq 2781 osr(3.228 0x276d3e0)/0x276d3e0 start
2014-06-12 14:47:28.146850 7f9d7c165700 10 _do_transaction on 0x6c562b0
2014-06-12 14:47:28.146867 7f9d7c165700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.146854 7f9d7715b700  5 -- op tracker -- , seq: 748445, time: 2014-06-12 14:47:28.146831, event: commit_sent, op: osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.146877 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2991 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:47:28.146932 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.147012 7f9d7795c700  5 -- op tracker -- , seq: 748445, time: 2014-06-12 14:47:28.146997, event: sub_op_applied, op: osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.147042 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.147042 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2992 active] sub_op_modify_applied on 0x2a29a380 op osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.147088 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:28.147084 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2992 active] op_applied on primary on version 173'2993
2014-06-12 14:47:28.147106 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2992 active] op_applied on replica on version 173'2993
2014-06-12 14:47:28.147153 7f9d7795c700  5 -- op tracker -- , seq: 748445, time: 2014-06-12 14:47:28.147143, event: done, op: osd_sub_op(client.4299.0:145675 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2993 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.147191 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:28.147233 7f9d7c165700 15 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.147243 7f9d7c165700 10 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1 r = 0
2014-06-12 14:47:28.147251 7f9d7c165700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.147437 7f9d7c165700 15 _write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 3379200~516096
2014-06-12 14:47:28.147672 7f9d7c165700 10 lookup_strip_header done  cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.147803 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:47:28.149875 7f9d7c165700 10 _generic_write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 3379200~516096 = 0
2014-06-12 14:47:28.149933 7f9d7c165700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.149956 7f9d7c165700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.149964 7f9d7c165700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.149974 7f9d7c165700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.150220 7f9d7c165700 20 set_header setting 17767 cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 parent seq 0
2014-06-12 14:47:28.150313 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:28.150335 7f9d7c165700 20 set_header setting 2274 cid meta oid 1fe2503a/pglog_3.228/0//-1 parent seq 0
2014-06-12 14:47:28.150349 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:47:28.154175 7f9d7c165700 10 _do_op 0x142a8c00 seq 2781 r = 0, finisher 0x89f8f80 0
2014-06-12 14:47:28.154191 7f9d7c165700 10 _finish_op 0x142a8c00 seq 2781 osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.154202 7f9d8096e700  5 _do_op 0x6d678c0 seq 2782 osr(3.228 0x276d3e0)/0x276d3e0 start
2014-06-12 14:47:28.154199 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533d560 (1 active)
2014-06-12 14:47:28.154210 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:28.154212 7f9d8096e700 10 _do_transaction on 0x66cfeb0
2014-06-12 14:47:28.154222 7f9d8096e700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.154222 7f9d7795c700  5 -- op tracker -- , seq: 748446, time: 2014-06-12 14:47:28.154212, event: sub_op_applied, op: osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.154231 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2992 active] sub_op_modify_applied on 0x6c561c0 op osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.154261 7f9d7795c700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.154283 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2992 active] op_applied on primary on version 173'2994
2014-06-12 14:47:28.154291 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2992 active] op_applied on replica on version 173'2994
2014-06-12 14:47:28.154312 7f9d7715b700  5 -- op tracker -- , seq: 748446, time: 2014-06-12 14:47:28.154304, event: commit_sent, op: osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.154320 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2992 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:47:28.154339 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.154343 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.154369 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:28.154370 7f9d7715b700  5 -- op tracker -- , seq: 748446, time: 2014-06-12 14:47:28.154365, event: done, op: osd_sub_op(client.4299.0:145676 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2994 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.154406 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:28.154424 7f9d8096e700 15 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.154428 7f9d8096e700 10 _omap_rmkeys meta/1fe2503a/pglog_3.228/0//-1 r = 0
2014-06-12 14:47:28.154432 7f9d8096e700 15 _omap_setkeys meta/1fe2503a/pglog_3.228/0//-1
2014-06-12 14:47:28.154446 7f9d8096e700 15 _write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 3895296~299008
2014-06-12 14:47:28.154655 7f9d8096e700 10 lookup_strip_header done  cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.154763 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:47:28.155352 7f9d8096e700 10 _generic_write 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 3895296~299008 = 0
2014-06-12 14:47:28.155369 7f9d8096e700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.155375 7f9d8096e700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.155379 7f9d8096e700 15 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3
2014-06-12 14:47:28.155383 7f9d8096e700 10 _setattrs 3.228_head/f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 = 0
2014-06-12 14:47:28.155471 7f9d8096e700 20 set_header setting 17767 cid 3.228_head oid f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 parent seq 0
2014-06-12 14:47:28.155508 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:28.155524 7f9d8096e700 20 set_header setting 2274 cid meta oid 1fe2503a/pglog_3.228/0//-1 parent seq 0
2014-06-12 14:47:28.155529 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:47:28.156362 7f9d8096e700 10 _do_op 0x6d678c0 seq 2782 r = 0, finisher 0x1e172ae0 0
2014-06-12 14:47:28.156370 7f9d8096e700 10 _finish_op 0x6d678c0 seq 2782 osr(3.228 0x276d3e0)/0x276d3e0
2014-06-12 14:47:28.156374 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x533d560 (0 active)
2014-06-12 14:47:28.156376 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:28.156386 7f9d7795c700  5 -- op tracker -- , seq: 748447, time: 2014-06-12 14:47:28.156380, event: sub_op_applied, op: osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.156392 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2993 active] sub_op_modify_applied on 0x66cfdc0 op osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:28.156415 7f9d7795c700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.156439 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2993 active] op_applied on primary on version 173'2995
2014-06-12 14:47:28.156453 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2993 active] op_applied on replica on version 173'2995
2014-06-12 14:47:28.156499 7f9d7715b700  5 -- op tracker -- , seq: 748447, time: 2014-06-12 14:47:28.156481, event: commit_sent, op: osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.156537 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.228( v 173'2995 (0'0,173'2995] local-les=157 n=309 ec=23 les/c 157/157 156/156/156) [28,2] r=1 lpr=156 pi=23-155/12 luod=0'0 crt=0'0 lcod 173'2993 active] sub_op_modify_commit on op osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.28
2014-06-12 14:47:28.156584 7f9d7715b700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.156585 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:28.156645 7f9d7715b700  5 -- op tracker -- , seq: 748447, time: 2014-06-12 14:47:28.156634, event: done, op: osd_sub_op(client.4299.0:145677 3.228 f952f228/rbd_data.10a26b8b4567.0000000000000f86/head//3 [] v 173'2995 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:28.250842 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:28.250866 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:28.371047 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:28.484934 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:28.484965 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:28.484970 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:28.484972 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:28.484975 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 7; oldest is 19.314199 seconds old
2014-06-12 14:47:28.485011 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:28.526165 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.526559 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:28.564900 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:28.712111 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:28.721161 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:28.721229 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:28.830435 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:28.830507 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:28.961734 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:28.961808 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:29.162061 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:29.163910 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:29.167260 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:29.172886 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:29.361835 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:29.364713 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:29.364912 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:29.366980 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:29.481347 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:29.481789 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:29.485056 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:29.485070 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:29.485072 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:29.485073 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:29.485082 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 7; oldest is 20.314299 seconds old
2014-06-12 14:47:29.485096 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:29.520431 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:29.527114 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:29.531017 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7])
2014-06-12 14:47:29.531043 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7])
2014-06-12 14:47:29.531673 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:29.531709 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:29.531730 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:29.531749 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:29.531769 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:29.531783 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:29.531802 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:29.531820 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:29.531835 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:29.531850 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:29.531868 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:29.531883 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:29.531907 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:29.531949 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:29.531968 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:29.531983 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:29.531999 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:29.532014 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:29.532030 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:29.532047 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:29.532064 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:29.533132 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:29.547147 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:29.547364 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:29.646078 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:29.662242 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:29.717452 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:29.722971 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:29.723538 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:29.727629 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:29.738891 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:29.774167 7f9d5a16f700  5 -- op tracker -- , seq: 748448, time: 2014-06-12 14:47:29.773713, event: header_read, op: osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.774202 7f9d5a16f700  5 -- op tracker -- , seq: 748448, time: 2014-06-12 14:47:29.773717, event: throttled, op: osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.774225 7f9d5a16f700  5 -- op tracker -- , seq: 748448, time: 2014-06-12 14:47:29.774086, event: all_read, op: osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.774247 7f9d5a16f700  5 -- op tracker -- , seq: 748448, time: 0.000000, event: dispatched, op: osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.774260 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:29.774274 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:29.774291 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903200 prio 127 cost 131879 latency 0.000578 osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:29.774350 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:29.774371 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2a903200 prio 127 cost 131879 latency 0.000657 osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.760( v 173'2434 (0'0,173'2434] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2433 active]
2014-06-12 14:47:29.774423 7f9d6b3e7700  5 -- op tracker -- , seq: 748448, time: 2014-06-12 14:47:29.774413, event: reached_pg, op: osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.774444 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2434 (0'0,173'2434] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2433 active] handle_message: 0x2a903200
2014-06-12 14:47:29.774470 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2434 (0'0,173'2434] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2433 active] sub_op_modify trans 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 v 173'2435 (transaction) 174
2014-06-12 14:47:29.774505 7f9d6b3e7700  5 -- op tracker -- , seq: 748448, time: 2014-06-12 14:47:29.774496, event: started, op: osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.774560 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2434 (0'0,173'2434] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2433 active] append_log log((0'0,173'2434], crt=0'0) [173'2435 (173'2434) modify   816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 by client.4301.0:145113 2014-06-12 08:44:15.576052]
2014-06-12 14:47:29.774619 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2435 (0'0,173'2435] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2433 active] add_log_entry 173'2435 (173'2434) modify   816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 by client.4301.0:145113 2014-06-12 08:44:15.576052
2014-06-12 14:47:29.774671 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2435 (0'0,173'2435] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2433 active] append_log  adding 1 keys
2014-06-12 14:47:29.774750 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2435, trimmed: 
2014-06-12 14:47:29.774805 7f9d6b3e7700  5 -- op tracker -- , seq: 748448, time: 2014-06-12 14:47:29.774793, event: started, op: osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.774830 7f9d6b3e7700  5 queue_transactions existing osr(3.760 0x49b41b0)/0x49b41b0
2014-06-12 14:47:29.774839 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:29.774841 7f9d6b3e7700  5 queue_op 0x6827b60 seq 2248 osr(3.760 0x49b41b0) 133365 bytes   (queue has 1 ops and 133365 bytes)
2014-06-12 14:47:29.774855 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.760( v 173'2435 (0'0,173'2435] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2433 active] do_sub_op osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:29.774887 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2a903200 finish
2014-06-12 14:47:29.774895 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:29.774901 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:29.774938 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x55217a0 (1 active)
2014-06-12 14:47:29.774948 7f9d7815d700  5 _do_op 0x6827b60 seq 2248 osr(3.760 0x49b41b0)/0x49b41b0 start
2014-06-12 14:47:29.774956 7f9d7815d700 10 _do_transaction on 0x151f20f0
2014-06-12 14:47:29.774967 7f9d7815d700 15 _omap_setkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:29.775161 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:29.775205 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:29.775317 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:29.775370 7f9d7815d700 15 _omap_rmkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:29.775387 7f9d7815d700 10 _omap_rmkeys meta/1fe099ca/pglog_3.760/0//-1 r = 0
2014-06-12 14:47:29.775401 7f9d7815d700 15 _omap_setkeys meta/1fe099ca/pglog_3.760/0//-1
2014-06-12 14:47:29.775593 7f9d7815d700 15 _write 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 4063232~131072
2014-06-12 14:47:29.775930 7f9d7815d700 10 lookup_strip_header done  cid 3.760_head oid 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:29.776108 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:47:29.777017 7f9d7815d700 10 _generic_write 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 4063232~131072 = 0
2014-06-12 14:47:29.777063 7f9d7815d700 15 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:29.777116 7f9d7815d700 10 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 = 0
2014-06-12 14:47:29.777135 7f9d7815d700 15 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3
2014-06-12 14:47:29.777154 7f9d7815d700 10 _setattrs 3.760_head/816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 = 0
2014-06-12 14:47:29.777443 7f9d7815d700 20 set_header setting 17060 cid 3.760_head oid 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 parent seq 0
2014-06-12 14:47:29.777511 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:29.777538 7f9d7815d700 20 set_header setting 2198 cid meta oid 1fe099ca/pglog_3.760/0//-1 parent seq 0
2014-06-12 14:47:29.777568 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:47:29.778278 7f9d7815d700 10 _do_op 0x6827b60 seq 2248 r = 0, finisher 0x233303a0 0
2014-06-12 14:47:29.778293 7f9d7815d700 10 _finish_op 0x6827b60 seq 2248 osr(3.760 0x49b41b0)/0x49b41b0
2014-06-12 14:47:29.778303 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x55217a0 (0 active)
2014-06-12 14:47:29.778309 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:29.778341 7f9d7715b700  5 -- op tracker -- , seq: 748448, time: 2014-06-12 14:47:29.778325, event: commit_sent, op: osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.778351 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2435 (0'0,173'2435] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2433 active] sub_op_modify_commit on op osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:29.778388 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:29.778436 7f9d7795c700  5 -- op tracker -- , seq: 748448, time: 2014-06-12 14:47:29.778431, event: sub_op_applied, op: osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.778445 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2435 (0'0,173'2435] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2434 active] sub_op_modify_applied on 0x151f2000 op osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:29.778469 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2435 (0'0,173'2435] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2434 active] op_applied on primary on version 173'2435
2014-06-12 14:47:29.778480 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.760( v 173'2435 (0'0,173'2435] local-les=147 n=273 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2434 active] op_applied on replica on version 173'2435
2014-06-12 14:47:29.778506 7f9d7795c700  5 -- op tracker -- , seq: 748448, time: 2014-06-12 14:47:29.778501, event: done, op: osd_sub_op(client.4301.0:145113 3.760 816bff60/rbd_data.10a46b8b4567.0000000000001070/head//3 [] v 173'2435 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.924182 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:29.944902 7f9d5a16f700  5 -- op tracker -- , seq: 748449, time: 2014-06-12 14:47:29.942160, event: header_read, op: osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.944920 7f9d5a16f700  5 -- op tracker -- , seq: 748449, time: 2014-06-12 14:47:29.942162, event: throttled, op: osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.944925 7f9d5a16f700  5 -- op tracker -- , seq: 748449, time: 2014-06-12 14:47:29.944860, event: all_read, op: osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.944932 7f9d5a16f700  5 -- op tracker -- , seq: 748449, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.944936 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:29.944941 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:29.944947 7f9d5a16f700 15 osd.2 173 enqueue_op 0x12bf7900 prio 127 cost 1549095 latency 0.002787 osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:29.944974 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:29.944980 7f9d693e3700 10 osd.2 173 dequeue_op 0x12bf7900 prio 127 cost 1549095 latency 0.002820 osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2877 (0'0,173'2877] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2876 active]
2014-06-12 14:47:29.945019 7f9d693e3700  5 -- op tracker -- , seq: 748449, time: 2014-06-12 14:47:29.945014, event: reached_pg, op: osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.945033 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2877 (0'0,173'2877] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2876 active] handle_message: 0x12bf7900
2014-06-12 14:47:29.945057 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2877 (0'0,173'2877] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2876 active] sub_op_modify trans 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 v 173'2878 (transaction) 174
2014-06-12 14:47:29.945080 7f9d693e3700  5 -- op tracker -- , seq: 748449, time: 2014-06-12 14:47:29.945075, event: started, op: osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.945108 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2877 (0'0,173'2877] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2876 active] append_log log((0'0,173'2877], crt=0'0) [173'2878 (173'2877) modify   69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 by client.4314.0:145057 2014-06-12 08:45:20.989621]
2014-06-12 14:47:29.945143 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2878 (0'0,173'2878] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2876 active] add_log_entry 173'2878 (173'2877) modify   69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 by client.4314.0:145057 2014-06-12 08:45:20.989621
2014-06-12 14:47:29.945199 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2878 (0'0,173'2878] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2876 active] append_log  adding 1 keys
2014-06-12 14:47:29.945249 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2878, trimmed: 
2014-06-12 14:47:29.945285 7f9d693e3700  5 -- op tracker -- , seq: 748449, time: 2014-06-12 14:47:29.945278, event: started, op: osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.945301 7f9d693e3700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:29.945306 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:29.945308 7f9d693e3700  5 queue_op 0x5a721e0 seq 2591 osr(3.129 0x276cf20) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:47:29.945324 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:29.945334 7f9d7e96a700  5 _do_op 0x5a721e0 seq 2591 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:29.945341 7f9d7e96a700 10 _do_transaction on 0x191d0470
2014-06-12 14:47:29.945317 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2878 (0'0,173'2878] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2876 active] do_sub_op osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:29.945346 7f9d693e3700 10 osd.2 173 dequeue_op 0x12bf7900 finish
2014-06-12 14:47:29.945352 7f9d7e96a700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:29.945355 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:29.945360 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:29.945471 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:29.945495 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:29.945533 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:29.945661 7f9d7e96a700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:29.945669 7f9d7e96a700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:29.945675 7f9d7e96a700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:29.945694 7f9d7e96a700 15 _write 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 1564672~1548288
2014-06-12 14:47:29.945857 7f9d7e96a700 10 lookup_strip_header done  cid 3.129_head oid 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:29.946182 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:47:29.951375 7f9d7e96a700 10 _generic_write 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 1564672~1548288 = 0
2014-06-12 14:47:29.951475 7f9d7e96a700 15 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:29.951490 7f9d7e96a700 10 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:47:29.951495 7f9d7e96a700 15 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:29.951500 7f9d7e96a700 10 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:47:29.951764 7f9d7e96a700 20 set_header setting 17871 cid 3.129_head oid 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 parent seq 0
2014-06-12 14:47:29.951881 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:29.951893 7f9d7e96a700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:29.951906 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:47:29.957893 7f9d7e96a700 10 _do_op 0x5a721e0 seq 2591 r = 0, finisher 0x31389e80 0
2014-06-12 14:47:29.957912 7f9d7e96a700 10 _finish_op 0x5a721e0 seq 2591 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:29.957918 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:29.957922 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:29.957938 7f9d7715b700  5 -- op tracker -- , seq: 748449, time: 2014-06-12 14:47:29.957925, event: commit_sent, op: osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.957963 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2878 (0'0,173'2878] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2876 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:29.957993 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:29.958065 7f9d7795c700  5 -- op tracker -- , seq: 748449, time: 2014-06-12 14:47:29.958059, event: sub_op_applied, op: osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.958076 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2878 (0'0,173'2878] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active] sub_op_modify_applied on 0x191d0380 op osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:29.958089 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2878 (0'0,173'2878] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active] op_applied on primary on version 173'2878
2014-06-12 14:47:29.958096 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2878 (0'0,173'2878] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active] op_applied on replica on version 173'2878
2014-06-12 14:47:29.958124 7f9d7795c700  5 -- op tracker -- , seq: 748449, time: 2014-06-12 14:47:29.958121, event: done, op: osd_sub_op(client.4314.0:145057 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2878 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.964417 7f9d5a16f700  5 -- op tracker -- , seq: 748450, time: 2014-06-12 14:47:29.962956, event: header_read, op: osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.964431 7f9d5a16f700  5 -- op tracker -- , seq: 748450, time: 2014-06-12 14:47:29.962958, event: throttled, op: osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.964446 7f9d5a16f700  5 -- op tracker -- , seq: 748450, time: 2014-06-12 14:47:29.964386, event: all_read, op: osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.964451 7f9d5a16f700  5 -- op tracker -- , seq: 748450, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.964455 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:29.964459 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:29.964465 7f9d5a16f700 15 osd.2 173 enqueue_op 0x12bf6400 prio 127 cost 1032999 latency 0.001509 osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:29.964488 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:29.964496 7f9d683e1700 10 osd.2 173 dequeue_op 0x12bf6400 prio 127 cost 1032999 latency 0.001540 osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.31d( v 173'2484 (0'0,173'2484] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active]
2014-06-12 14:47:29.964528 7f9d683e1700  5 -- op tracker -- , seq: 748450, time: 2014-06-12 14:47:29.964523, event: reached_pg, op: osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.964537 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2484 (0'0,173'2484] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] handle_message: 0x12bf6400
2014-06-12 14:47:29.964549 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2484 (0'0,173'2484] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] sub_op_modify trans 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 v 173'2485 (transaction) 174
2014-06-12 14:47:29.964566 7f9d683e1700  5 -- op tracker -- , seq: 748450, time: 2014-06-12 14:47:29.964562, event: started, op: osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.964578 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2484 (0'0,173'2484] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] append_log log((0'0,173'2484], crt=0'0) [173'2485 (173'2484) modify   5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 by client.4315.0:144553 2014-06-12 08:44:41.790164]
2014-06-12 14:47:29.964604 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2485 (0'0,173'2485] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] add_log_entry 173'2485 (173'2484) modify   5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 by client.4315.0:144553 2014-06-12 08:44:41.790164
2014-06-12 14:47:29.964627 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2485 (0'0,173'2485] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] append_log  adding 1 keys
2014-06-12 14:47:29.964658 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2485, trimmed: 
2014-06-12 14:47:29.964680 7f9d683e1700  5 -- op tracker -- , seq: 748450, time: 2014-06-12 14:47:29.964674, event: started, op: osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.964691 7f9d683e1700  5 queue_transactions existing osr(3.31d 0x276d520)/0x276d520
2014-06-12 14:47:29.964695 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:29.964696 7f9d683e1700  5 queue_op 0x54295c0 seq 2303 osr(3.31d 0x276d520) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:29.964708 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439440 (1 active)
2014-06-12 14:47:29.964702 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.31d( v 173'2485 (0'0,173'2485] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] do_sub_op osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:29.964716 7f9d683e1700 10 osd.2 173 dequeue_op 0x12bf6400 finish
2014-06-12 14:47:29.964717 7f9d8116f700  5 _do_op 0x54295c0 seq 2303 osr(3.31d 0x276d520)/0x276d520 start
2014-06-12 14:47:29.964719 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:29.964722 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:29.964722 7f9d8116f700 10 _do_transaction on 0x678feb0
2014-06-12 14:47:29.964729 7f9d8116f700 15 _omap_setkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:29.964853 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:29.964882 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:29.964930 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:29.964957 7f9d8116f700 15 _omap_rmkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:29.964964 7f9d8116f700 10 _omap_rmkeys meta/1fe1ce3b/pglog_3.31d/0//-1 r = 0
2014-06-12 14:47:29.964970 7f9d8116f700 15 _omap_setkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:29.965117 7f9d8116f700 15 _write 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 3084288~1032192
2014-06-12 14:47:29.965227 7f9d8116f700 10 lookup_strip_header done  cid 3.31d_head oid 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:29.965364 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:47:29.968887 7f9d8116f700 10 _generic_write 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 3084288~1032192 = 0
2014-06-12 14:47:29.968999 7f9d8116f700 15 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:29.969028 7f9d8116f700 10 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 = 0
2014-06-12 14:47:29.969036 7f9d8116f700 15 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:29.969043 7f9d8116f700 10 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 = 0
2014-06-12 14:47:29.969591 7f9d8116f700 20 set_header setting 17151 cid 3.31d_head oid 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 parent seq 0
2014-06-12 14:47:29.969725 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:29.969749 7f9d8116f700 20 set_header setting 2268 cid meta oid 1fe1ce3b/pglog_3.31d/0//-1 parent seq 0
2014-06-12 14:47:29.969760 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:47:29.974270 7f9d8116f700 10 _do_op 0x54295c0 seq 2303 r = 0, finisher 0x73f8920 0
2014-06-12 14:47:29.974286 7f9d8116f700 10 _finish_op 0x54295c0 seq 2303 osr(3.31d 0x276d520)/0x276d520
2014-06-12 14:47:29.974290 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439440 (0 active)
2014-06-12 14:47:29.974295 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:29.974313 7f9d7715b700  5 -- op tracker -- , seq: 748450, time: 2014-06-12 14:47:29.974295, event: commit_sent, op: osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.974324 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2485 (0'0,173'2485] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:29.974359 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:29.974418 7f9d7795c700  5 -- op tracker -- , seq: 748450, time: 2014-06-12 14:47:29.974412, event: sub_op_applied, op: osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:29.974437 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2485 (0'0,173'2485] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] sub_op_modify_applied on 0x678fdc0 op osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:29.974457 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2485 (0'0,173'2485] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] op_applied on primary on version 173'2485
2014-06-12 14:47:29.974464 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2485 (0'0,173'2485] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] op_applied on replica on version 173'2485
2014-06-12 14:47:29.974484 7f9d7795c700  5 -- op tracker -- , seq: 748450, time: 2014-06-12 14:47:29.974481, event: done, op: osd_sub_op(client.4315.0:144553 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.062397 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:30.062452 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:30.084306 7f9d5a16f700  5 -- op tracker -- , seq: 748451, time: 2014-06-12 14:47:30.081861, event: header_read, op: osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.084337 7f9d5a16f700  5 -- op tracker -- , seq: 748451, time: 2014-06-12 14:47:30.081877, event: throttled, op: osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.084356 7f9d5a16f700  5 -- op tracker -- , seq: 748451, time: 2014-06-12 14:47:30.084233, event: all_read, op: osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.084380 7f9d5a16f700  5 -- op tracker -- , seq: 748451, time: 0.000000, event: dispatched, op: osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.084390 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:30.084403 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:30.084417 7f9d5a16f700 15 osd.2 173 enqueue_op 0x20355d00 prio 127 cost 983847 latency 0.002556 osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:30.084479 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:30.084505 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20355d00 prio 127 cost 983847 latency 0.002643 osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2679 (0'0,173'2679] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2678 active]
2014-06-12 14:47:30.084558 7f9d6abe6700  5 -- op tracker -- , seq: 748451, time: 2014-06-12 14:47:30.084548, event: reached_pg, op: osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.084579 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2679 (0'0,173'2679] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2678 active] handle_message: 0x20355d00
2014-06-12 14:47:30.084605 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2679 (0'0,173'2679] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2678 active] sub_op_modify trans c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 v 173'2680 (transaction) 174
2014-06-12 14:47:30.084652 7f9d6abe6700  5 -- op tracker -- , seq: 748451, time: 2014-06-12 14:47:30.084642, event: started, op: osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.084689 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2679 (0'0,173'2679] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2678 active] append_log log((0'0,173'2679], crt=0'0) [173'2680 (173'2679) modify   c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 by client.4285.0:143417 2014-06-12 08:43:34.045332]
2014-06-12 14:47:30.084744 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2680 (0'0,173'2680] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2678 active] add_log_entry 173'2680 (173'2679) modify   c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 by client.4285.0:143417 2014-06-12 08:43:34.045332
2014-06-12 14:47:30.084798 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2680 (0'0,173'2680] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2678 active] append_log  adding 1 keys
2014-06-12 14:47:30.084868 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2680, trimmed: 
2014-06-12 14:47:30.084921 7f9d6abe6700  5 -- op tracker -- , seq: 748451, time: 2014-06-12 14:47:30.084910, event: started, op: osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.084945 7f9d6abe6700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:30.084953 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:30.084956 7f9d6abe6700  5 queue_op 0x83ef800 seq 2490 osr(3.1b7 0x276d1a0) 985333 bytes   (queue has 1 ops and 985333 bytes)
2014-06-12 14:47:30.084979 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:30.084969 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2680 (0'0,173'2680] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2678 active] do_sub_op osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:30.084996 7f9d7d167700  5 _do_op 0x83ef800 seq 2490 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:30.085003 7f9d6abe6700 10 osd.2 173 dequeue_op 0x20355d00 finish
2014-06-12 14:47:30.085005 7f9d7d167700 10 _do_transaction on 0x1b39cb70
2014-06-12 14:47:30.085011 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:30.085017 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:30.085020 7f9d7d167700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:30.085210 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:30.085265 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:30.085359 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:30.085530 7f9d7d167700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:30.085553 7f9d7d167700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:30.085568 7f9d7d167700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:30.085608 7f9d7d167700 15 _write 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 3211264~983040
2014-06-12 14:47:30.085897 7f9d7d167700 10 lookup_strip_header done  cid 3.1b7_head oid c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:30.086244 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:47:30.091287 7f9d7d167700 10 _generic_write 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 3211264~983040 = 0
2014-06-12 14:47:30.091414 7f9d7d167700 15 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:30.091429 7f9d7d167700 10 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 = 0
2014-06-12 14:47:30.091442 7f9d7d167700 15 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3
2014-06-12 14:47:30.091452 7f9d7d167700 10 _setattrs 3.1b7_head/c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 = 0
2014-06-12 14:47:30.091750 7f9d7d167700 20 set_header setting 17534 cid 3.1b7_head oid c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 parent seq 0
2014-06-12 14:47:30.091868 7f9d7d167700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:30.091885 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:30.091900 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:47:30.098446 7f9d7d167700 10 _do_op 0x83ef800 seq 2490 r = 0, finisher 0x65ed7a0 0
2014-06-12 14:47:30.098470 7f9d7d167700 10 _finish_op 0x83ef800 seq 2490 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:30.098478 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:30.098485 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:30.098490 7f9d7715b700  5 -- op tracker -- , seq: 748451, time: 2014-06-12 14:47:30.098478, event: commit_sent, op: osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.098503 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2680 (0'0,173'2680] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2678 active] sub_op_modify_commit on op osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:30.098551 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:30.098590 7f9d7795c700  5 -- op tracker -- , seq: 748451, time: 2014-06-12 14:47:30.098583, event: sub_op_applied, op: osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.098601 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2680 (0'0,173'2680] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2679 active] sub_op_modify_applied on 0x1b39ca80 op osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:30.098622 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2680 (0'0,173'2680] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2679 active] op_applied on primary on version 173'2680
2014-06-12 14:47:30.098633 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2680 (0'0,173'2680] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2679 active] op_applied on replica on version 173'2680
2014-06-12 14:47:30.098662 7f9d7795c700  5 -- op tracker -- , seq: 748451, time: 2014-06-12 14:47:30.098657, event: done, op: osd_sub_op(client.4285.0:143417 3.1b7 c6a751b7/rbd_data.10946b8b4567.0000000000001118/head//3 [] v 173'2680 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.122415 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:30.126821 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:30.127969 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:30.129420 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:30.133625 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:30.137786 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:30.138730 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:30.139160 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:30.139311 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:30.146880 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:30.154275 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:30.156446 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:30.156655 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:30.371132 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:30.485149 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:30.485181 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:30.485187 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:30.485189 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:30.485192 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 7; oldest is 21.314416 seconds old
2014-06-12 14:47:30.485190 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:30.491366 7f9d5a16f700  5 -- op tracker -- , seq: 748452, time: 2014-06-12 14:47:30.489051, event: header_read, op: osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.491408 7f9d5a16f700  5 -- op tracker -- , seq: 748452, time: 2014-06-12 14:47:30.489054, event: throttled, op: osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.491427 7f9d5a16f700  5 -- op tracker -- , seq: 748452, time: 2014-06-12 14:47:30.491297, event: all_read, op: osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.491444 7f9d5a16f700  5 -- op tracker -- , seq: 748452, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.491454 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:30.491467 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:30.491505 7f9d5a16f700 15 osd.2 173 enqueue_op 0x20354d00 prio 127 cost 1032999 latency 0.002453 osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:30.491564 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:30.491577 7f9d68be2700 10 osd.2 173 dequeue_op 0x20354d00 prio 127 cost 1032999 latency 0.002526 osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2878 (0'0,173'2878] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active]
2014-06-12 14:47:30.491638 7f9d68be2700  5 -- op tracker -- , seq: 748452, time: 2014-06-12 14:47:30.491628, event: reached_pg, op: osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.491656 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2878 (0'0,173'2878] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active] handle_message: 0x20354d00
2014-06-12 14:47:30.491684 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2878 (0'0,173'2878] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active] sub_op_modify trans 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 v 173'2879 (transaction) 174
2014-06-12 14:47:30.491719 7f9d68be2700  5 -- op tracker -- , seq: 748452, time: 2014-06-12 14:47:30.491710, event: started, op: osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.491751 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2878 (0'0,173'2878] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active] append_log log((0'0,173'2878], crt=0'0) [173'2879 (173'2878) modify   69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 by client.4314.0:145058 2014-06-12 08:45:21.330863]
2014-06-12 14:47:30.491811 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2879 (0'0,173'2879] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active] add_log_entry 173'2879 (173'2878) modify   69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 by client.4314.0:145058 2014-06-12 08:45:21.330863
2014-06-12 14:47:30.491850 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2879 (0'0,173'2879] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active] append_log  adding 1 keys
2014-06-12 14:47:30.491942 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2879, trimmed: 
2014-06-12 14:47:30.492010 7f9d68be2700  5 -- op tracker -- , seq: 748452, time: 2014-06-12 14:47:30.492002, event: started, op: osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.492029 7f9d68be2700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:30.492036 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:30.492039 7f9d68be2700  5 queue_op 0x28d8180 seq 2592 osr(3.129 0x276cf20) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:30.492060 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:30.492075 7f9d7c966700  5 _do_op 0x28d8180 seq 2592 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:30.492083 7f9d7c966700 10 _do_transaction on 0x90f0630
2014-06-12 14:47:30.492052 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2879 (0'0,173'2879] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active] do_sub_op osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:30.492090 7f9d68be2700 10 osd.2 173 dequeue_op 0x20354d00 finish
2014-06-12 14:47:30.492094 7f9d7c966700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:30.492098 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:30.492103 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:30.492390 7f9d5a16f700  5 -- op tracker -- , seq: 748453, time: 2014-06-12 14:47:30.492142, event: header_read, op: osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.492410 7f9d5a16f700  5 -- op tracker -- , seq: 748453, time: 2014-06-12 14:47:30.492144, event: throttled, op: osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.492435 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:30.492482 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:30.492484 7f9d5a16f700  5 -- op tracker -- , seq: 748453, time: 2014-06-12 14:47:30.492332, event: all_read, op: osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.492501 7f9d5a16f700  5 -- op tracker -- , seq: 748453, time: 0.000000, event: dispatched, op: osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.492510 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:30.492521 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:30.492544 7f9d5a16f700 15 osd.2 173 enqueue_op 0xe20ad00 prio 127 cost 78631 latency 0.000402 osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:30.492575 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:30.492612 7f9d7c966700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:30.492635 7f9d7c966700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:30.492646 7f9d7c966700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:30.492643 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:30.492728 7f9d643d9700 10 osd.2 173 dequeue_op 0xe20ad00 prio 127 cost 78631 latency 0.000517 osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.31d( v 173'2485 (0'0,173'2485] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active]
2014-06-12 14:47:30.492794 7f9d643d9700  5 -- op tracker -- , seq: 748453, time: 2014-06-12 14:47:30.492773, event: reached_pg, op: osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.492818 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2485 (0'0,173'2485] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] handle_message: 0xe20ad00
2014-06-12 14:47:30.492841 7f9d7c966700 15 _write 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 3112960~1032192
2014-06-12 14:47:30.492842 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2485 (0'0,173'2485] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] sub_op_modify trans 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 v 173'2486 (transaction) 174
2014-06-12 14:47:30.492875 7f9d643d9700  5 -- op tracker -- , seq: 748453, time: 2014-06-12 14:47:30.492867, event: started, op: osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.492905 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2485 (0'0,173'2485] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] append_log log((0'0,173'2485], crt=0'0) [173'2486 (173'2485) modify   5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 by client.4315.0:144554 2014-06-12 08:44:42.036487]
2014-06-12 14:47:30.492968 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2486 (0'0,173'2486] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] add_log_entry 173'2486 (173'2485) modify   5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 by client.4315.0:144554 2014-06-12 08:44:42.036487
2014-06-12 14:47:30.493017 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2486 (0'0,173'2486] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] append_log  adding 1 keys
2014-06-12 14:47:30.493071 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2486, trimmed: 
2014-06-12 14:47:30.493085 7f9d7c966700 10 lookup_strip_header done  cid 3.129_head oid 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:30.493108 7f9d643d9700  5 -- op tracker -- , seq: 748453, time: 2014-06-12 14:47:30.493100, event: started, op: osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.493126 7f9d643d9700  5 queue_transactions existing osr(3.31d 0x276d520)/0x276d520
2014-06-12 14:47:30.493131 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:30.493133 7f9d643d9700  5 queue_op 0x8349ce0 seq 2304 osr(3.31d 0x276d520) 80117 bytes   (queue has 2 ops and 1114602 bytes)
2014-06-12 14:47:30.493142 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.31d( v 173'2486 (0'0,173'2486] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] do_sub_op osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:30.493165 7f9d643d9700 10 osd.2 173 dequeue_op 0xe20ad00 finish
2014-06-12 14:47:30.493171 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:30.493178 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:30.493198 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439440 (2 active)
2014-06-12 14:47:30.493205 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:47:30.493209 7f9d81970700  5 _do_op 0x8349ce0 seq 2304 osr(3.31d 0x276d520)/0x276d520 start
2014-06-12 14:47:30.493216 7f9d81970700 10 _do_transaction on 0x5a43270
2014-06-12 14:47:30.493225 7f9d81970700 15 _omap_setkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:30.493363 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:30.493404 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:30.497481 7f9d7c966700 10 _generic_write 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 3112960~1032192 = 0
2014-06-12 14:47:30.497592 7f9d7c966700 15 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:30.497640 7f9d7c966700 10 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:47:30.497651 7f9d7c966700 15 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:30.497661 7f9d7c966700 10 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:47:30.497885 7f9d7c966700 20 set_header setting 17871 cid 3.129_head oid 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 parent seq 0
2014-06-12 14:47:30.497968 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:30.497979 7f9d7c966700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:30.497992 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:47:30.501389 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:30.501505 7f9d81970700 15 _omap_rmkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:30.501514 7f9d81970700 10 _omap_rmkeys meta/1fe1ce3b/pglog_3.31d/0//-1 r = 0
2014-06-12 14:47:30.501521 7f9d81970700 15 _omap_setkeys meta/1fe1ce3b/pglog_3.31d/0//-1
2014-06-12 14:47:30.501543 7f9d81970700 15 _write 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 4116480~77824
2014-06-12 14:47:30.501665 7f9d7c966700 10 _do_op 0x28d8180 seq 2592 r = 0, finisher 0x73c0300 0
2014-06-12 14:47:30.501681 7f9d7c966700 10 _finish_op 0x28d8180 seq 2592 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:30.501700 7f9d81970700 10 lookup_strip_header done  cid 3.31d_head oid 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:30.501712 7f9d7795c700  5 -- op tracker -- , seq: 748452, time: 2014-06-12 14:47:30.501705, event: sub_op_applied, op: osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.501729 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2879 (0'0,173'2879] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active] sub_op_modify_applied on 0x90f0540 op osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:30.501752 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:30.501767 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2879 (0'0,173'2879] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active] op_applied on primary on version 173'2879
2014-06-12 14:47:30.501785 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2879 (0'0,173'2879] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active] op_applied on replica on version 173'2879
2014-06-12 14:47:30.501799 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (1 active)
2014-06-12 14:47:30.501806 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:30.501809 7f9d7715b700  5 -- op tracker -- , seq: 748452, time: 2014-06-12 14:47:30.501801, event: commit_sent, op: osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.501819 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2879 (0'0,173'2879] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2877 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:30.501836 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:30.501845 7f9d81970700 10 file_to_extents done 
2014-06-12 14:47:30.501860 7f9d7715b700  5 -- op tracker -- , seq: 748452, time: 2014-06-12 14:47:30.501855, event: done, op: osd_sub_op(client.4314.0:145058 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2879 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.502072 7f9d81970700 10 _generic_write 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 4116480~77824 = 0
2014-06-12 14:47:30.502088 7f9d81970700 15 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:30.502100 7f9d81970700 10 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 = 0
2014-06-12 14:47:30.502105 7f9d81970700 15 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3
2014-06-12 14:47:30.502114 7f9d81970700 10 _setattrs 3.31d_head/5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 = 0
2014-06-12 14:47:30.502203 7f9d81970700 20 set_header setting 17151 cid 3.31d_head oid 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 parent seq 0
2014-06-12 14:47:30.502228 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:30.502242 7f9d81970700 20 set_header setting 2268 cid meta oid 1fe1ce3b/pglog_3.31d/0//-1 parent seq 0
2014-06-12 14:47:30.502253 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:47:30.502542 7f9d81970700 10 _do_op 0x8349ce0 seq 2304 r = 0, finisher 0xb755ec0 0
2014-06-12 14:47:30.502551 7f9d81970700 10 _finish_op 0x8349ce0 seq 2304 osr(3.31d 0x276d520)/0x276d520
2014-06-12 14:47:30.502557 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439440 (0 active)
2014-06-12 14:47:30.502560 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:30.502561 7f9d7715b700  5 -- op tracker -- , seq: 748453, time: 2014-06-12 14:47:30.502555, event: commit_sent, op: osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.502567 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2486 (0'0,173'2486] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] sub_op_modify_commit on op osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:30.502582 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:30.502607 7f9d7795c700  5 -- op tracker -- , seq: 748453, time: 2014-06-12 14:47:30.502600, event: sub_op_applied, op: osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.502616 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2486 (0'0,173'2486] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2485 active] sub_op_modify_applied on 0x5a43180 op osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:30.502646 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2486 (0'0,173'2486] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2485 active] op_applied on primary on version 173'2486
2014-06-12 14:47:30.502655 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.31d( v 173'2486 (0'0,173'2486] local-les=147 n=287 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2485 active] op_applied on replica on version 173'2486
2014-06-12 14:47:30.502686 7f9d7795c700  5 -- op tracker -- , seq: 748453, time: 2014-06-12 14:47:30.502681, event: done, op: osd_sub_op(client.4315.0:144554 3.31d 5780d31d/rbd_data.10b26b8b4567.0000000000001053/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:30.554479 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:30.554509 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:31.167335 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:31.172951 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:31.223102 7f9d5a16f700  5 -- op tracker -- , seq: 748454, time: 2014-06-12 14:47:31.222900, event: header_read, op: osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:31.223122 7f9d5a16f700  5 -- op tracker -- , seq: 748454, time: 2014-06-12 14:47:31.222904, event: throttled, op: osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:31.223133 7f9d5a16f700  5 -- op tracker -- , seq: 748454, time: 2014-06-12 14:47:31.223043, event: all_read, op: osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:31.223166 7f9d5a16f700  5 -- op tracker -- , seq: 748454, time: 0.000000, event: dispatched, op: osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:31.223173 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:31.223183 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:31.223194 7f9d5a16f700 15 osd.2 173 enqueue_op 0x598ed00 prio 127 cost 49959 latency 0.000294 osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:31.223244 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:31.223265 7f9d63bd8700 10 osd.2 173 dequeue_op 0x598ed00 prio 127 cost 49959 latency 0.000364 osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2879 (0'0,173'2879] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2878 active]
2014-06-12 14:47:31.223319 7f9d63bd8700  5 -- op tracker -- , seq: 748454, time: 2014-06-12 14:47:31.223309, event: reached_pg, op: osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:31.223343 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2879 (0'0,173'2879] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2878 active] handle_message: 0x598ed00
2014-06-12 14:47:31.223381 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2879 (0'0,173'2879] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2878 active] sub_op_modify trans 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 v 173'2880 (transaction) 174
2014-06-12 14:47:31.223423 7f9d63bd8700  5 -- op tracker -- , seq: 748454, time: 2014-06-12 14:47:31.223413, event: started, op: osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:31.223461 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2879 (0'0,173'2879] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2878 active] append_log log((0'0,173'2879], crt=0'0) [173'2880 (173'2879) modify   69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 by client.4314.0:145059 2014-06-12 08:45:21.332251]
2014-06-12 14:47:31.223513 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2880 (0'0,173'2880] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2878 active] add_log_entry 173'2880 (173'2879) modify   69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 by client.4314.0:145059 2014-06-12 08:45:21.332251
2014-06-12 14:47:31.223566 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2880 (0'0,173'2880] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2878 active] append_log  adding 1 keys
2014-06-12 14:47:31.223642 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2880, trimmed: 
2014-06-12 14:47:31.223694 7f9d63bd8700  5 -- op tracker -- , seq: 748454, time: 2014-06-12 14:47:31.223682, event: started, op: osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:31.223718 7f9d63bd8700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:31.223726 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:31.223730 7f9d63bd8700  5 queue_op 0x76965a0 seq 2593 osr(3.129 0x276cf20) 51445 bytes   (queue has 1 ops and 51445 bytes)
2014-06-12 14:47:31.223744 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2880 (0'0,173'2880] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2878 active] do_sub_op osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:31.223779 7f9d63bd8700 10 osd.2 173 dequeue_op 0x598ed00 finish
2014-06-12 14:47:31.223788 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:31.223781 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:31.223794 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:31.223798 7f9d7895e700  5 _do_op 0x76965a0 seq 2593 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:31.223806 7f9d7895e700 10 _do_transaction on 0x165e6630
2014-06-12 14:47:31.223816 7f9d7895e700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:31.223997 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:31.224037 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:31.224114 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:31.224260 7f9d7895e700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:31.224273 7f9d7895e700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:31.224282 7f9d7895e700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:31.224306 7f9d7895e700 15 _write 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 4145152~49152
2014-06-12 14:47:31.224504 7f9d7895e700 10 lookup_strip_header done  cid 3.129_head oid 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:31.224685 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:47:31.224898 7f9d7895e700 10 _generic_write 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 4145152~49152 = 0
2014-06-12 14:47:31.224925 7f9d7895e700 15 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:31.224939 7f9d7895e700 10 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:47:31.224946 7f9d7895e700 15 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3
2014-06-12 14:47:31.224956 7f9d7895e700 10 _setattrs 3.129_head/69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 = 0
2014-06-12 14:47:31.225125 7f9d7895e700 20 set_header setting 17871 cid 3.129_head oid 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 parent seq 0
2014-06-12 14:47:31.225152 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:31.225167 7f9d7895e700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:31.225180 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:47:31.225596 7f9d7895e700 10 _do_op 0x76965a0 seq 2593 r = 0, finisher 0xa015f80 0
2014-06-12 14:47:31.225611 7f9d7895e700 10 _finish_op 0x76965a0 seq 2593 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:31.225642 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:31.225647 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:31.225670 7f9d7715b700  5 -- op tracker -- , seq: 748454, time: 2014-06-12 14:47:31.225655, event: commit_sent, op: osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:31.225707 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2880 (0'0,173'2880] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2878 active] sub_op_modify_commit on op osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:31.225739 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:31.225826 7f9d7795c700  5 -- op tracker -- , seq: 748454, time: 2014-06-12 14:47:31.225813, event: sub_op_applied, op: osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:31.225843 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2880 (0'0,173'2880] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active] sub_op_modify_applied on 0x165e6540 op osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:31.225870 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2880 (0'0,173'2880] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active] op_applied on primary on version 173'2880
2014-06-12 14:47:31.225883 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2880 (0'0,173'2880] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active] op_applied on replica on version 173'2880
2014-06-12 14:47:31.225912 7f9d7795c700  5 -- op tracker -- , seq: 748454, time: 2014-06-12 14:47:31.225906, event: done, op: osd_sub_op(client.4314.0:145059 3.129 69c3b929/rbd_data.10b16b8b4567.0000000000001058/head//3 [] v 173'2880 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:31.247771 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:31.247841 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:31.309703 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:31.309747 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:31.316126 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:31.316149 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:31.481426 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:31.481858 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:31.485265 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:31.485292 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:31.485297 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:31.485299 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:31.485302 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 7; oldest is 22.314526 seconds old
2014-06-12 14:47:31.485337 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:31.520506 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:31.527194 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:31.599994 7f9d5a16f700  5 -- op tracker -- , seq: 748455, time: 2014-06-12 14:47:31.599928, event: header_read, op: osd_sub_op_reply(client.4316.0:147318 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:31.600006 7f9d5a16f700  5 -- op tracker -- , seq: 748455, time: 2014-06-12 14:47:31.599932, event: throttled, op: osd_sub_op_reply(client.4316.0:147318 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:31.600014 7f9d5a16f700  5 -- op tracker -- , seq: 748455, time: 2014-06-12 14:47:31.599949, event: all_read, op: osd_sub_op_reply(client.4316.0:147318 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:31.600022 7f9d5a16f700  5 -- op tracker -- , seq: 748455, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147318 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:31.600028 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147318 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:31.600034 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:31.600043 7f9d5a16f700 15 osd.2 173 enqueue_op 0x598f300 prio 196 cost 0 latency 0.000114 osd_sub_op_reply(client.4316.0:147318 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:31.600075 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:31.600084 7f9d673df700 10 osd.2 173 dequeue_op 0x598f300 prio 196 cost 0 latency 0.000155 osd_sub_op_reply(client.4316.0:147318 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 pg pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]
2014-06-12 14:47:31.600100 7f9d673df700  5 -- op tracker -- , seq: 748455, time: 2014-06-12 14:47:31.600097, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147318 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:31.600107 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] handle_message: 0x598f300
2014-06-12 14:47:31.600117 7f9d673df700  5 -- op tracker -- , seq: 748455, time: 2014-06-12 14:47:31.600115, event: started, op: osd_sub_op_reply(client.4316.0:147318 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:31.600122 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] sub_op_modify_reply: tid 236406 op  ack_type 4 from 24
2014-06-12 14:47:31.600139 7f9d673df700  5 -- op tracker -- , seq: 748328, time: 2014-06-12 14:47:31.600131, event: sub_op_commit_rec, op: osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:31.600144 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] repop_all_applied: repop tid 236406 all applied 
2014-06-12 14:47:31.600151 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] eval_repop repgather(0x24e8d680 173'2976 rep_tid=236406 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:31.600163 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean]  sending ack on repgather(0x24e8d680 173'2976 rep_tid=236406 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x25edc780
2014-06-12 14:47:31.600192 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2975 crt=0'0 lcod 173'2974 mlcod 173'2974 active+clean] repop_all_committed: repop tid 236406 all committed 
2014-06-12 14:47:31.600198 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2974 active+clean] eval_repop repgather(0x24e8d680 173'2976 rep_tid=236406 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:31.600208 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2974 active+clean] log_op_stats osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 inb 258327 outb 0 rlat 22.429415 lat 22.429432
2014-06-12 14:47:31.600220 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2974 active+clean] publish_stats_to_osd 173:3172
2014-06-12 14:47:31.600229 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2974 active+clean]  sending commit on repgather(0x24e8d680 173'2976 rep_tid=236406 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x15561400
2014-06-12 14:47:31.600249 7f9d673df700  5 -- op tracker -- , seq: 748328, time: 2014-06-12 14:47:31.600245, event: commit_sent, op: osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:31.600255 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2974 active+clean]  requeue_ops 
2014-06-12 14:47:31.600262 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2975 active+clean]  removing repgather(0x24e8d680 173'2976 rep_tid=236406 committed?=1 applied?=1 op=osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:31.600270 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2975 active+clean]    q front is repgather(0x24e8d680 173'2976 rep_tid=236406 committed?=1 applied?=1 op=osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:31.600279 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2975 active+clean] remove_repop repgather(0x24e8d680 173'2976 rep_tid=236406 committed?=1 applied?=1 op=osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:31.600287 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2975 active+clean]  requeue_ops 
2014-06-12 14:47:31.600301 7f9d673df700  5 -- op tracker -- , seq: 748328, time: 2014-06-12 14:47:31.600298, event: done, op: osd_op(client.4316.0:147318 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~258048] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:31.600412 7f9d673df700 10 osd.2 173 dequeue_op 0x598f300 finish
2014-06-12 14:47:31.600420 7f9d673df700  5 -- op tracker -- , seq: 748455, time: 2014-06-12 14:47:31.600417, event: done, op: osd_sub_op_reply(client.4316.0:147318 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:31.600451 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:31.600458 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:31.662334 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:31.717526 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:31.723043 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:31.723609 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:31.727700 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:31.738963 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:31.774977 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:31.778379 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:31.831778 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:31.832031 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:31.924271 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:31.945426 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:31.957997 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:31.964782 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:31.966600 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:31.966614 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:31.966616 7f9d74bfa700 20 osd.2 173 _dispatch 0x79e1860 ping v1
2014-06-12 14:47:31.966620 7f9d74bfa700 10 osd.2 173 ping from client.4316
2014-06-12 14:47:31.966623 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:31.966624 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:31.974365 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:32.077668 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:32.077826 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:32.085110 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:32.098556 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:32.122500 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:32.126912 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:32.128039 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:32.129520 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:32.133703 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:32.137872 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:32.138807 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:32.139246 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:32.139411 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:32.146961 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:32.154370 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:32.156515 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:32.156724 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:32.331149 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:32.331197 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:32.371211 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:32.485368 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:32.485386 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:47:32.485388 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:47:32.485390 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:47:32.485401 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:47:32.485437 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:32.485440 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:32.485441 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:32.485443 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 6; oldest is 22.398869 seconds old
2014-06-12 14:47:32.485443 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:32.492183 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:32.493239 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:32.501864 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:32.502620 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:32.876132 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:32.876148 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:32.876153 7f9d74bfa700 20 osd.2 173 _dispatch 0x16bcf340 pg_stats_ack(1 pgs tid 4863) v1
2014-06-12 14:47:32.876158 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:47:32.876168 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:32.876171 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:32.948523 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:32.948601 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:32.951186 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:32.951353 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:33.017113 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:33.017191 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:33.167425 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:33.173024 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:33.223892 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:33.225710 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:33.481506 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:33.481925 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:33.485519 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:33.485550 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:33.485555 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:33.485557 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:33.485560 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 6; oldest is 23.398985 seconds old
2014-06-12 14:47:33.485556 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:33.520602 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:33.527293 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:33.600522 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:33.662443 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:33.717591 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:33.723136 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:33.723677 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:33.727799 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:33.739039 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:33.775080 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:33.778460 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:33.924361 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:33.945524 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:33.958077 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:33.964862 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:33.974448 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:34.021705 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:34.021836 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:34.065084 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:34.065243 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:34.085195 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:34.098629 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:34.117464 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:34.117493 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:34.122574 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:34.126985 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:34.128122 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:34.129605 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:34.133786 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:34.137957 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:34.138892 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:34.139329 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:34.139493 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:34.147061 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:34.154445 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:34.156604 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:34.156807 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:34.371308 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:34.426557 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:34.426587 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:34.485642 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:34.485681 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:34.485684 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:34.485685 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:34.485686 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 6; oldest is 24.399112 seconds old
2014-06-12 14:47:34.485723 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:34.492277 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:34.493316 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:34.501936 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:34.502697 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:34.763117 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:34.763203 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:34.831770 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19427 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6])
2014-06-12 14:47:34.831794 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19427 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6])
2014-06-12 14:47:34.832574 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:34.832630 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:34.832677 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:34.832705 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:34.832727 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:34.832765 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:34.832818 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:34.832842 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:34.832863 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:34.832884 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:34.832905 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:34.832924 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:34.832945 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:34.832965 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:34.832985 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:34.833005 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:34.833026 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:34.833049 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:34.833070 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:34.833090 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:34.833109 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:34.833129 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:34.978070 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:34.978263 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:35.167505 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:35.173103 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:35.223998 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:35.225790 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:35.255272 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:35.255507 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:35.410466 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:35.410541 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:35.481582 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:35.481985 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:35.485760 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:35.485787 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:35.485792 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:35.485794 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:35.485797 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 6; oldest is 25.399222 seconds old
2014-06-12 14:47:35.485833 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:35.520705 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:35.527390 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:35.600593 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:35.662528 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:35.717662 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:35.722169 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:35.722389 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:35.723208 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:35.723744 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:35.727869 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:35.739121 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:35.775171 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:35.778541 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:35.924436 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:35.945596 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:35.958187 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:35.964944 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:35.974529 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:36.085281 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:36.098694 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:36.122670 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:36.127083 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:36.128221 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:36.129702 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:36.133883 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:36.138033 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:36.138975 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:36.139412 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:36.139576 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:36.147159 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:36.154528 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:36.156685 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:36.156887 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:36.371413 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:36.451754 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:36.451847 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:36.485896 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:36.485915 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:36.485926 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:36.485927 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:36.485929 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 6; oldest is 26.399355 seconds old
2014-06-12 14:47:36.485929 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:36.492362 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:36.493398 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:36.502011 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:36.502777 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:36.822898 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:36.823134 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:36.966865 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:36.966871 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:36.966873 7f9d74bfa700 20 osd.2 173 _dispatch 0x79e0680 ping v1
2014-06-12 14:47:36.966877 7f9d74bfa700 10 osd.2 173 ping from client.4316
2014-06-12 14:47:36.966880 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:36.966881 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:37.111035 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:37.111084 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:37.132635 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19427 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6])
2014-06-12 14:47:37.132651 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19427 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6])
2014-06-12 14:47:37.133107 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:37.133141 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:37.133170 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:37.133218 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:37.133243 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:37.133275 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:37.133296 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:37.133317 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:37.133340 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:37.133352 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:37.133385 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:37.133398 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:37.133415 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:37.133427 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:37.133439 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:37.133449 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:37.133461 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:37.133472 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:37.133483 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:37.133494 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:37.133504 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:37.133516 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:37.167587 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:37.173199 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:37.224088 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:37.225886 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:37.481661 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:37.482054 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:37.486003 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:37.486025 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:47:37.486029 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:47:37.486034 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:47:37.486045 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:37.486051 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:37.486054 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:37.486057 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 6; oldest is 27.399483 seconds old
2014-06-12 14:47:37.486057 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:37.520789 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:37.527471 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:37.600678 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:37.618307 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:37.618681 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:37.662603 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:37.663593 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:37.663644 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:37.717758 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:37.723299 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:37.723823 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:37.727944 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:37.732408 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:37.732880 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:37.739192 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:37.775243 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:37.778610 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:37.924518 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:37.927144 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:37.927244 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:37.945675 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:37.958260 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:37.965016 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:37.974626 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:38.085392 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:38.098774 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:38.122759 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:38.127164 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:38.128298 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:38.129806 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:38.133969 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:38.138120 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:38.139061 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:38.139489 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:38.139679 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:38.147236 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:38.154599 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:38.156753 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:38.156961 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:38.164292 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:38.164345 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:38.231780 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:38.231841 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:38.233198 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19427 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6])
2014-06-12 14:47:38.233217 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19427 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6])
2014-06-12 14:47:38.233532 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:38.233556 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:38.233681 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:38.233699 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:38.233721 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:38.233755 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:38.233803 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:38.233847 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:38.233872 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:38.233885 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:38.233928 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:38.233955 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:38.233976 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:38.233998 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:38.234018 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:38.234043 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:38.234064 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:38.234087 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:38.234109 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:38.234132 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:38.234152 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:38.234174 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:38.313815 7f9d5a16f700  5 -- op tracker -- , seq: 748456, time: 2014-06-12 14:47:38.313341, event: header_read, op: osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.313825 7f9d5a16f700  5 -- op tracker -- , seq: 748456, time: 2014-06-12 14:47:38.313345, event: throttled, op: osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.313830 7f9d5a16f700  5 -- op tracker -- , seq: 748456, time: 2014-06-12 14:47:38.313776, event: all_read, op: osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.313835 7f9d5a16f700  5 -- op tracker -- , seq: 748456, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.313840 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:38.313845 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:38.313852 7f9d5a16f700 15 osd.2 173 enqueue_op 0x598f900 prio 127 cost 279331 latency 0.000511 osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:38.313909 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:38.313919 7f9d68be2700 10 osd.2 173 dequeue_op 0x598f900 prio 127 cost 279331 latency 0.000578 osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.f9( v 173'2717 (0'0,173'2717] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active]
2014-06-12 14:47:38.313946 7f9d68be2700  5 -- op tracker -- , seq: 748456, time: 2014-06-12 14:47:38.313941, event: reached_pg, op: osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.313957 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2717 (0'0,173'2717] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] handle_message: 0x598f900
2014-06-12 14:47:38.313971 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2717 (0'0,173'2717] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] sub_op_modify trans 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 v 173'2718 (transaction) 174
2014-06-12 14:47:38.313988 7f9d68be2700  5 -- op tracker -- , seq: 748456, time: 2014-06-12 14:47:38.313983, event: started, op: osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.314006 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2717 (0'0,173'2717] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] append_log log((0'0,173'2717], crt=0'0) [173'2718 (173'682) modify   3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 by client.4284.0:141490 2014-06-12 08:45:17.446008]
2014-06-12 14:47:38.314033 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2718 (0'0,173'2718] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] add_log_entry 173'2718 (173'682) modify   3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 by client.4284.0:141490 2014-06-12 08:45:17.446008
2014-06-12 14:47:38.314056 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2718 (0'0,173'2718] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] append_log  adding 1 keys
2014-06-12 14:47:38.314092 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2718, trimmed: 
2014-06-12 14:47:38.314118 7f9d68be2700  5 -- op tracker -- , seq: 748456, time: 2014-06-12 14:47:38.314113, event: started, op: osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.314128 7f9d68be2700  5 queue_transactions existing osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:38.314132 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:38.314140 7f9d68be2700  5 queue_op 0x6e432c0 seq 2527 osr(3.f9 0x276cfa0) 280812 bytes   (queue has 1 ops and 280812 bytes)
2014-06-12 14:47:38.314148 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.f9( v 173'2718 (0'0,173'2718] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] do_sub_op osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:38.314161 7f9d68be2700 10 osd.2 173 dequeue_op 0x598f900 finish
2014-06-12 14:47:38.314165 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:38.314168 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:38.314167 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b77a0 (1 active)
2014-06-12 14:47:38.314175 7f9d7c966700  5 _do_op 0x6e432c0 seq 2527 osr(3.f9 0x276cfa0)/0x276cfa0 start
2014-06-12 14:47:38.314179 7f9d7c966700 10 _do_transaction on 0x5b25cf0
2014-06-12 14:47:38.314186 7f9d7c966700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:38.314370 7f9d7c966700 10 lookup_strip_header done  cid meta oid 516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:38.314411 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:38.314502 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:38.314532 7f9d7c966700 15 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:38.314552 7f9d7c966700 10 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1 r = 0
2014-06-12 14:47:38.314559 7f9d7c966700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:38.314913 7f9d7c966700 15 _write 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 0~278528
2014-06-12 14:47:38.327227 7f9d7c966700 10 lookup_strip_header done  cid 3.f9_head oid 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:38.327390 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:47:38.328122 7f9d7c966700 10 _generic_write 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 0~278528 = 0
2014-06-12 14:47:38.328142 7f9d7c966700 15 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:38.328149 7f9d7c966700 10 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 = 0
2014-06-12 14:47:38.328153 7f9d7c966700 15 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:38.328158 7f9d7c966700 10 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 = 0
2014-06-12 14:47:38.328261 7f9d7c966700 20 set_header setting 17380 cid 3.f9_head oid 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 parent seq 0
2014-06-12 14:47:38.328289 7f9d7c966700 20 set_header setting 2218 cid meta oid 516e9ea6/pglog_3.f9/0//-1 parent seq 0
2014-06-12 14:47:38.328298 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:38.328303 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:47:38.329202 7f9d7c966700 10 _do_op 0x6e432c0 seq 2527 r = 0, finisher 0x66b1c60 0
2014-06-12 14:47:38.329212 7f9d7c966700 10 _finish_op 0x6e432c0 seq 2527 osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:38.329220 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b77a0 (0 active)
2014-06-12 14:47:38.329224 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:38.329226 7f9d7715b700  5 -- op tracker -- , seq: 748456, time: 2014-06-12 14:47:38.329219, event: commit_sent, op: osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.329250 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2718 (0'0,173'2718] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2716 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:38.329269 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:38.329342 7f9d7795c700  5 -- op tracker -- , seq: 748456, time: 2014-06-12 14:47:38.329327, event: sub_op_applied, op: osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.329357 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2718 (0'0,173'2718] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] sub_op_modify_applied on 0x5b25c00 op osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:38.329374 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2718 (0'0,173'2718] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] op_applied on primary on version 173'2718
2014-06-12 14:47:38.329383 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2718 (0'0,173'2718] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] op_applied on replica on version 173'2718
2014-06-12 14:47:38.329409 7f9d7795c700  5 -- op tracker -- , seq: 748456, time: 2014-06-12 14:47:38.329406, event: done, op: osd_sub_op(client.4284.0:141490 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2718 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.371517 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:38.478541 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:38.478570 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:38.486123 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:38.486146 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:38.486149 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:38.486151 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:38.486153 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 6; oldest is 28.399578 seconds old
2014-06-12 14:47:38.486164 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:38.493481 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:38.502846 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:38.664308 7f9d5a16f700  5 -- op tracker -- , seq: 748457, time: 2014-06-12 14:47:38.663337, event: header_read, op: osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.664323 7f9d5a16f700  5 -- op tracker -- , seq: 748457, time: 2014-06-12 14:47:38.663349, event: throttled, op: osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.664332 7f9d5a16f700  5 -- op tracker -- , seq: 748457, time: 2014-06-12 14:47:38.664267, event: all_read, op: osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.664340 7f9d5a16f700  5 -- op tracker -- , seq: 748457, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.664345 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:38.664351 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:38.664359 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329ea600 prio 127 cost 516899 latency 0.001022 osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:38.664387 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:38.664397 7f9d63bd8700 10 osd.2 173 dequeue_op 0x329ea600 prio 127 cost 516899 latency 0.001060 osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.f9( v 173'2718 (0'0,173'2718] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active]
2014-06-12 14:47:38.664422 7f9d63bd8700  5 -- op tracker -- , seq: 748457, time: 2014-06-12 14:47:38.664417, event: reached_pg, op: osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.664432 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2718 (0'0,173'2718] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] handle_message: 0x329ea600
2014-06-12 14:47:38.664444 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2718 (0'0,173'2718] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] sub_op_modify trans 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 v 173'2719 (transaction) 174
2014-06-12 14:47:38.664464 7f9d63bd8700  5 -- op tracker -- , seq: 748457, time: 2014-06-12 14:47:38.664459, event: started, op: osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.664478 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2718 (0'0,173'2718] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] append_log log((0'0,173'2718], crt=0'0) [173'2719 (173'2718) modify   3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 by client.4284.0:141491 2014-06-12 08:45:17.452782]
2014-06-12 14:47:38.664504 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2719 (0'0,173'2719] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] add_log_entry 173'2719 (173'2718) modify   3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 by client.4284.0:141491 2014-06-12 08:45:17.452782
2014-06-12 14:47:38.664525 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2719 (0'0,173'2719] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] append_log  adding 1 keys
2014-06-12 14:47:38.664553 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2719, trimmed: 
2014-06-12 14:47:38.664573 7f9d63bd8700  5 -- op tracker -- , seq: 748457, time: 2014-06-12 14:47:38.664569, event: started, op: osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.664583 7f9d63bd8700  5 queue_transactions existing osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:38.664586 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:38.664587 7f9d63bd8700  5 queue_op 0x769fe60 seq 2528 osr(3.f9 0x276cfa0) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:47:38.664594 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.f9( v 173'2719 (0'0,173'2719] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] do_sub_op osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:38.664608 7f9d63bd8700 10 osd.2 173 dequeue_op 0x329ea600 finish
2014-06-12 14:47:38.664612 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:38.664614 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:38.664622 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b77a0 (1 active)
2014-06-12 14:47:38.664627 7f9d7a962700  5 _do_op 0x769fe60 seq 2528 osr(3.f9 0x276cfa0)/0x276cfa0 start
2014-06-12 14:47:38.664631 7f9d7a962700 10 _do_transaction on 0x61cf7b0
2014-06-12 14:47:38.664636 7f9d7a962700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:38.664761 7f9d7a962700 10 lookup_strip_header done  cid meta oid 516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:38.664777 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:38.664824 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:38.664845 7f9d7a962700 15 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:38.664851 7f9d7a962700 10 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1 r = 0
2014-06-12 14:47:38.664856 7f9d7a962700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:38.664966 7f9d7a962700 15 _write 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 278528~516096
2014-06-12 14:47:38.665087 7f9d7a962700 10 lookup_strip_header done  cid 3.f9_head oid 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:38.665241 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:47:38.665428 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:38.665735 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:38.667270 7f9d7a962700 10 _generic_write 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 278528~516096 = 0
2014-06-12 14:47:38.667327 7f9d7a962700 15 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:38.667343 7f9d7a962700 10 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 = 0
2014-06-12 14:47:38.667349 7f9d7a962700 15 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:38.667357 7f9d7a962700 10 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 = 0
2014-06-12 14:47:38.667564 7f9d7a962700 20 set_header setting 17380 cid 3.f9_head oid 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 parent seq 0
2014-06-12 14:47:38.667675 7f9d7a962700 20 set_header setting 2218 cid meta oid 516e9ea6/pglog_3.f9/0//-1 parent seq 0
2014-06-12 14:47:38.667691 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:38.667701 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:47:38.670948 7f9d7a962700 10 _do_op 0x769fe60 seq 2528 r = 0, finisher 0x80e8960 0
2014-06-12 14:47:38.670992 7f9d7715b700  5 -- op tracker -- , seq: 748457, time: 2014-06-12 14:47:38.670981, event: commit_sent, op: osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.671007 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2719 (0'0,173'2719] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2717 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:38.671042 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:38.671092 7f9d7a962700 10 _finish_op 0x769fe60 seq 2528 osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:38.671104 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b77a0 (0 active)
2014-06-12 14:47:38.671107 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:38.671117 7f9d7795c700  5 -- op tracker -- , seq: 748457, time: 2014-06-12 14:47:38.671111, event: sub_op_applied, op: osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.671125 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2719 (0'0,173'2719] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] sub_op_modify_applied on 0x61cf6c0 op osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:38.671137 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2719 (0'0,173'2719] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] op_applied on primary on version 173'2719
2014-06-12 14:47:38.671143 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2719 (0'0,173'2719] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] op_applied on replica on version 173'2719
2014-06-12 14:47:38.671161 7f9d7795c700  5 -- op tracker -- , seq: 748457, time: 2014-06-12 14:47:38.671159, event: done, op: osd_sub_op(client.4284.0:141491 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2719 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.698119 7f9d5a16f700  5 -- op tracker -- , seq: 748458, time: 2014-06-12 14:47:38.696842, event: header_read, op: osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.698146 7f9d5a16f700  5 -- op tracker -- , seq: 748458, time: 2014-06-12 14:47:38.696845, event: throttled, op: osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.698154 7f9d5a16f700  5 -- op tracker -- , seq: 748458, time: 2014-06-12 14:47:38.698066, event: all_read, op: osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.698162 7f9d5a16f700  5 -- op tracker -- , seq: 748458, time: 0.000000, event: dispatched, op: osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.698168 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:38.698176 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:38.698185 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329eb200 prio 127 cost 758563 latency 0.001343 osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:38.698217 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:38.698228 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329eb200 prio 127 cost 758563 latency 0.001385 osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.f9( v 173'2719 (0'0,173'2719] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active]
2014-06-12 14:47:38.698257 7f9d61bd4700  5 -- op tracker -- , seq: 748458, time: 2014-06-12 14:47:38.698253, event: reached_pg, op: osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.698271 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2719 (0'0,173'2719] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] handle_message: 0x329eb200
2014-06-12 14:47:38.698286 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2719 (0'0,173'2719] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] sub_op_modify trans 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 v 173'2720 (transaction) 174
2014-06-12 14:47:38.698306 7f9d61bd4700  5 -- op tracker -- , seq: 748458, time: 2014-06-12 14:47:38.698301, event: started, op: osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.698325 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2719 (0'0,173'2719] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] append_log log((0'0,173'2719], crt=0'0) [173'2720 (173'666) modify   183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 by client.4288.0:139267 2014-06-12 08:44:23.628464]
2014-06-12 14:47:38.698360 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2720 (0'0,173'2720] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] add_log_entry 173'2720 (173'666) modify   183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 by client.4288.0:139267 2014-06-12 08:44:23.628464
2014-06-12 14:47:38.698385 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2720 (0'0,173'2720] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] append_log  adding 1 keys
2014-06-12 14:47:38.698424 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2720, trimmed: 
2014-06-12 14:47:38.698450 7f9d61bd4700  5 -- op tracker -- , seq: 748458, time: 2014-06-12 14:47:38.698445, event: started, op: osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.698463 7f9d61bd4700  5 queue_transactions existing osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:38.698469 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:38.698471 7f9d61bd4700  5 queue_op 0x5f67260 seq 2529 osr(3.f9 0x276cfa0) 760044 bytes   (queue has 1 ops and 760044 bytes)
2014-06-12 14:47:38.698481 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b77a0 (1 active)
2014-06-12 14:47:38.698489 7f9d82972700  5 _do_op 0x5f67260 seq 2529 osr(3.f9 0x276cfa0)/0x276cfa0 start
2014-06-12 14:47:38.698478 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.f9( v 173'2720 (0'0,173'2720] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] do_sub_op osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:38.698492 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329eb200 finish
2014-06-12 14:47:38.698494 7f9d82972700 10 _do_transaction on 0x19d860f0
2014-06-12 14:47:38.698496 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:38.698498 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:38.698500 7f9d82972700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:38.698630 7f9d82972700 10 lookup_strip_header done  cid meta oid 516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:38.698660 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:38.698709 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:38.698805 7f9d82972700 15 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:38.698815 7f9d82972700 10 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1 r = 0
2014-06-12 14:47:38.698823 7f9d82972700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:38.698842 7f9d82972700 15 _write 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 0~757760
2014-06-12 14:47:38.699007 7f9d82972700 10 lookup_strip_header done  cid 3.f9_head oid 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3
2014-06-12 14:47:38.699219 7f9d82972700 10 file_to_extents done 
2014-06-12 14:47:38.701704 7f9d82972700 10 _generic_write 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 0~757760 = 0
2014-06-12 14:47:38.701762 7f9d82972700 15 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3
2014-06-12 14:47:38.701787 7f9d82972700 10 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 = 0
2014-06-12 14:47:38.701796 7f9d82972700 15 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3
2014-06-12 14:47:38.701805 7f9d82972700 10 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 = 0
2014-06-12 14:47:38.702047 7f9d82972700 20 set_header setting 16861 cid 3.f9_head oid 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 parent seq 0
2014-06-12 14:47:38.702125 7f9d82972700 20 set_header setting 2218 cid meta oid 516e9ea6/pglog_3.f9/0//-1 parent seq 0
2014-06-12 14:47:38.702142 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:38.702153 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:47:38.705440 7f9d82972700 10 _do_op 0x5f67260 seq 2529 r = 0, finisher 0x5b76100 0
2014-06-12 14:47:38.705478 7f9d82972700 10 _finish_op 0x5f67260 seq 2529 osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:38.705487 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b77a0 (0 active)
2014-06-12 14:47:38.705481 7f9d7715b700  5 -- op tracker -- , seq: 748458, time: 2014-06-12 14:47:38.705471, event: commit_sent, op: osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.705494 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:38.705496 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2720 (0'0,173'2720] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2718 active] sub_op_modify_commit on op osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:38.705533 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:38.705575 7f9d7795c700  5 -- op tracker -- , seq: 748458, time: 2014-06-12 14:47:38.705567, event: sub_op_applied, op: osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.705588 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2720 (0'0,173'2720] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] sub_op_modify_applied on 0x19d86000 op osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:38.705607 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2720 (0'0,173'2720] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] op_applied on primary on version 173'2720
2014-06-12 14:47:38.705626 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2720 (0'0,173'2720] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] op_applied on replica on version 173'2720
2014-06-12 14:47:38.705650 7f9d7795c700  5 -- op tracker -- , seq: 748458, time: 2014-06-12 14:47:38.705646, event: done, op: osd_sub_op(client.4288.0:139267 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2720 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:38.718871 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:38.718915 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:38.810991 7f9d5a16f700  5 -- op tracker -- , seq: 748459, time: 2014-06-12 14:47:38.810921, event: header_read, op: osd_sub_op_reply(client.4316.0:147319 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:38.811014 7f9d5a16f700  5 -- op tracker -- , seq: 748459, time: 2014-06-12 14:47:38.810923, event: throttled, op: osd_sub_op_reply(client.4316.0:147319 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:38.811021 7f9d5a16f700  5 -- op tracker -- , seq: 748459, time: 2014-06-12 14:47:38.810939, event: all_read, op: osd_sub_op_reply(client.4316.0:147319 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:38.811028 7f9d5a16f700  5 -- op tracker -- , seq: 748459, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147319 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:38.811034 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147319 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:38.811042 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:38.811051 7f9d5a16f700 15 osd.2 173 enqueue_op 0x20355600 prio 196 cost 0 latency 0.000130 osd_sub_op_reply(client.4316.0:147319 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:38.811118 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:38.811129 7f9d633d7700 10 osd.2 173 dequeue_op 0x20355600 prio 196 cost 0 latency 0.000208 osd_sub_op_reply(client.4316.0:147319 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 pg pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2975 active+clean]
2014-06-12 14:47:38.811158 7f9d633d7700  5 -- op tracker -- , seq: 748459, time: 2014-06-12 14:47:38.811155, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147319 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:38.811167 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2975 active+clean] handle_message: 0x20355600
2014-06-12 14:47:38.811178 7f9d633d7700  5 -- op tracker -- , seq: 748459, time: 2014-06-12 14:47:38.811176, event: started, op: osd_sub_op_reply(client.4316.0:147319 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:38.811183 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2975 active+clean] sub_op_modify_reply: tid 236407 op  ack_type 4 from 24
2014-06-12 14:47:38.811199 7f9d633d7700  5 -- op tracker -- , seq: 748334, time: 2014-06-12 14:47:38.811191, event: sub_op_commit_rec, op: osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:38.811207 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2975 active+clean] repop_all_applied: repop tid 236407 all applied 
2014-06-12 14:47:38.811214 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2975 active+clean] eval_repop repgather(0x31b64000 173'2977 rep_tid=236407 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:38.811225 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2975 active+clean]  sending ack on repgather(0x31b64000 173'2977 rep_tid=236407 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x1710b400
2014-06-12 14:47:38.811248 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2976 crt=0'0 lcod 173'2975 mlcod 173'2975 active+clean] repop_all_committed: repop tid 236407 all committed 
2014-06-12 14:47:38.811256 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2975 active+clean] eval_repop repgather(0x31b64000 173'2977 rep_tid=236407 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:38.811267 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2975 active+clean] log_op_stats osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 28.724673 lat 28.724692
2014-06-12 14:47:38.811279 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2975 active+clean] publish_stats_to_osd 173:3173
2014-06-12 14:47:38.811288 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2975 active+clean]  sending commit on repgather(0x31b64000 173'2977 rep_tid=236407 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x1f889400
2014-06-12 14:47:38.811306 7f9d633d7700  5 -- op tracker -- , seq: 748334, time: 2014-06-12 14:47:38.811303, event: commit_sent, op: osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:38.811313 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2975 active+clean]  requeue_ops 
2014-06-12 14:47:38.811320 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2976 active+clean]  removing repgather(0x31b64000 173'2977 rep_tid=236407 committed?=1 applied?=1 op=osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:38.811329 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2976 active+clean]    q front is repgather(0x31b64000 173'2977 rep_tid=236407 committed?=1 applied?=1 op=osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:38.811337 7f9d633d7700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2976 active+clean] remove_repop repgather(0x31b64000 173'2977 rep_tid=236407 committed?=1 applied?=1 op=osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:38.811347 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2976 active+clean]  requeue_ops 
2014-06-12 14:47:38.811360 7f9d633d7700  5 -- op tracker -- , seq: 748334, time: 2014-06-12 14:47:38.811357, event: done, op: osd_op(client.4316.0:147319 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 258048~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:38.811378 7f9d633d7700 10 osd.2 173 dequeue_op 0x20355600 finish
2014-06-12 14:47:38.811387 7f9d633d7700  5 -- op tracker -- , seq: 748459, time: 2014-06-12 14:47:38.811383, event: done, op: osd_sub_op_reply(client.4316.0:147319 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:38.811396 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:38.811398 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:38.848896 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:38.849367 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:39.008016 7f9d5a16f700  5 -- op tracker -- , seq: 748460, time: 2014-06-12 14:47:39.007660, event: header_read, op: osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.008030 7f9d5a16f700  5 -- op tracker -- , seq: 748460, time: 2014-06-12 14:47:39.007665, event: throttled, op: osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.008038 7f9d5a16f700  5 -- op tracker -- , seq: 748460, time: 2014-06-12 14:47:39.007973, event: all_read, op: osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.008047 7f9d5a16f700  5 -- op tracker -- , seq: 748460, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.008056 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:39.008064 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:39.008076 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9d200 prio 127 cost 254759 latency 0.000416 osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.008121 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:39.008132 7f9d67be0700 10 osd.2 173 dequeue_op 0x24c9d200 prio 127 cost 254759 latency 0.000472 osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2680 (0'0,173'2680] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2679 active]
2014-06-12 14:47:39.008159 7f9d67be0700  5 -- op tracker -- , seq: 748460, time: 2014-06-12 14:47:39.008155, event: reached_pg, op: osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.008169 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2680 (0'0,173'2680] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2679 active] handle_message: 0x24c9d200
2014-06-12 14:47:39.008184 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2680 (0'0,173'2680] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2679 active] sub_op_modify trans 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 v 173'2681 (transaction) 174
2014-06-12 14:47:39.008212 7f9d67be0700  5 -- op tracker -- , seq: 748460, time: 2014-06-12 14:47:39.008207, event: started, op: osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.008230 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2680 (0'0,173'2680] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2679 active] append_log log((0'0,173'2680], crt=0'0) [173'2681 (173'662) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142256 2014-06-12 08:44:04.262960]
2014-06-12 14:47:39.008254 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2681 (0'0,173'2681] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2679 active] add_log_entry 173'2681 (173'662) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142256 2014-06-12 08:44:04.262960
2014-06-12 14:47:39.008279 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2681 (0'0,173'2681] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2679 active] append_log  adding 1 keys
2014-06-12 14:47:39.008320 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2681, trimmed: 
2014-06-12 14:47:39.008347 7f9d67be0700  5 -- op tracker -- , seq: 748460, time: 2014-06-12 14:47:39.008343, event: started, op: osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.008359 7f9d67be0700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:39.008363 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:39.008365 7f9d67be0700  5 queue_op 0x656fe60 seq 2491 osr(3.1b7 0x276d1a0) 256245 bytes   (queue has 1 ops and 256245 bytes)
2014-06-12 14:47:39.008375 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:39.008382 7f9d7895e700  5 _do_op 0x656fe60 seq 2491 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:39.008386 7f9d7895e700 10 _do_transaction on 0x116aa9b0
2014-06-12 14:47:39.008373 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2681 (0'0,173'2681] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2679 active] do_sub_op osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.008389 7f9d67be0700 10 osd.2 173 dequeue_op 0x24c9d200 finish
2014-06-12 14:47:39.008391 7f9d7895e700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:39.008392 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:39.008395 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:39.008494 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:39.008512 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:39.008547 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:39.008567 7f9d7895e700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:39.008573 7f9d7895e700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:39.008577 7f9d7895e700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:39.008594 7f9d7895e700 15 _write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 0~253952
2014-06-12 14:47:39.008742 7f9d7895e700 10 lookup_strip_header done  cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:39.008861 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:47:39.009701 7f9d7895e700 10 _generic_write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 0~253952 = 0
2014-06-12 14:47:39.009731 7f9d7895e700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:39.009743 7f9d7895e700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:39.009747 7f9d7895e700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:39.009752 7f9d7895e700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:39.009857 7f9d7895e700 20 set_header setting 16305 cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:47:39.009887 7f9d7895e700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:39.009897 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:39.009902 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:47:39.011081 7f9d7895e700 10 _do_op 0x656fe60 seq 2491 r = 0, finisher 0x87fbe40 0
2014-06-12 14:47:39.011095 7f9d7895e700 10 _finish_op 0x656fe60 seq 2491 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:39.011110 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:39.011113 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:39.011121 7f9d7715b700  5 -- op tracker -- , seq: 748460, time: 2014-06-12 14:47:39.011113, event: commit_sent, op: osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.011150 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2681 (0'0,173'2681] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2679 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:39.011171 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:39.011199 7f9d7795c700  5 -- op tracker -- , seq: 748460, time: 2014-06-12 14:47:39.011194, event: sub_op_applied, op: osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.011206 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2681 (0'0,173'2681] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active] sub_op_modify_applied on 0x116aa8c0 op osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.011222 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2681 (0'0,173'2681] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active] op_applied on primary on version 173'2681
2014-06-12 14:47:39.011229 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2681 (0'0,173'2681] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active] op_applied on replica on version 173'2681
2014-06-12 14:47:39.011245 7f9d7795c700  5 -- op tracker -- , seq: 748460, time: 2014-06-12 14:47:39.011242, event: done, op: osd_sub_op(client.4305.0:142256 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2681 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.177148 7f9d5a16f700  5 -- op tracker -- , seq: 748461, time: 2014-06-12 14:47:39.175487, event: header_read, op: osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.177158 7f9d5a16f700  5 -- op tracker -- , seq: 748461, time: 2014-06-12 14:47:39.175489, event: throttled, op: osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.177163 7f9d5a16f700  5 -- op tracker -- , seq: 748461, time: 2014-06-12 14:47:39.177111, event: all_read, op: osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.177177 7f9d5a16f700  5 -- op tracker -- , seq: 748461, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.177181 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:39.177185 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:39.177191 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9d100 prio 127 cost 1032995 latency 0.001704 osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.177215 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:39.177221 7f9d673df700 10 osd.2 173 dequeue_op 0x24c9d100 prio 127 cost 1032995 latency 0.001734 osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.f9( v 173'2720 (0'0,173'2720] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active]
2014-06-12 14:47:39.177238 7f9d673df700  5 -- op tracker -- , seq: 748461, time: 2014-06-12 14:47:39.177235, event: reached_pg, op: osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.177245 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2720 (0'0,173'2720] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] handle_message: 0x24c9d100
2014-06-12 14:47:39.177255 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2720 (0'0,173'2720] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] sub_op_modify trans 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 v 173'2721 (transaction) 174
2014-06-12 14:47:39.177281 7f9d673df700  5 -- op tracker -- , seq: 748461, time: 2014-06-12 14:47:39.177277, event: started, op: osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.177298 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2720 (0'0,173'2720] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] append_log log((0'0,173'2720], crt=0'0) [173'2721 (173'2719) modify   3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 by client.4284.0:141492 2014-06-12 08:45:20.248101]
2014-06-12 14:47:39.177327 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2721 (0'0,173'2721] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] add_log_entry 173'2721 (173'2719) modify   3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 by client.4284.0:141492 2014-06-12 08:45:20.248101
2014-06-12 14:47:39.177353 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2721 (0'0,173'2721] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] append_log  adding 1 keys
2014-06-12 14:47:39.177391 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2721, trimmed: 
2014-06-12 14:47:39.177416 7f9d673df700  5 -- op tracker -- , seq: 748461, time: 2014-06-12 14:47:39.177411, event: started, op: osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.177428 7f9d673df700  5 queue_transactions existing osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:39.177432 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:39.177434 7f9d673df700  5 queue_op 0x563cfc0 seq 2530 osr(3.f9 0x276cfa0) 1034476 bytes   (queue has 1 ops and 1034476 bytes)
2014-06-12 14:47:39.177441 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.f9( v 173'2721 (0'0,173'2721] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] do_sub_op osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.177456 7f9d673df700 10 osd.2 173 dequeue_op 0x24c9d100 finish
2014-06-12 14:47:39.177460 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:39.177463 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:39.177466 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b77a0 (1 active)
2014-06-12 14:47:39.177474 7f9d83173700  5 _do_op 0x563cfc0 seq 2530 osr(3.f9 0x276cfa0)/0x276cfa0 start
2014-06-12 14:47:39.177478 7f9d83173700 10 _do_transaction on 0x73d6ef0
2014-06-12 14:47:39.177484 7f9d83173700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.177610 7f9d83173700 10 lookup_strip_header done  cid meta oid 516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.177648 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:39.177695 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:39.177793 7f9d83173700 15 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.177802 7f9d83173700 10 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1 r = 0
2014-06-12 14:47:39.177807 7f9d83173700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.177825 7f9d83173700 15 _write 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 794624~1032192
2014-06-12 14:47:39.177981 7f9d83173700 10 lookup_strip_header done  cid 3.f9_head oid 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:39.178150 7f9d83173700 10 file_to_extents done 
2014-06-12 14:47:39.182252 7f9d83173700 10 _generic_write 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 794624~1032192 = 0
2014-06-12 14:47:39.182347 7f9d83173700 15 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:39.182364 7f9d83173700 10 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 = 0
2014-06-12 14:47:39.182370 7f9d83173700 15 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:39.182377 7f9d83173700 10 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 = 0
2014-06-12 14:47:39.182671 7f9d83173700 20 set_header setting 17380 cid 3.f9_head oid 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 parent seq 0
2014-06-12 14:47:39.182794 7f9d83173700 20 set_header setting 2218 cid meta oid 516e9ea6/pglog_3.f9/0//-1 parent seq 0
2014-06-12 14:47:39.182809 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:39.182825 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:47:39.187748 7f9d83173700 10 _do_op 0x563cfc0 seq 2530 r = 0, finisher 0x8357260 0
2014-06-12 14:47:39.187798 7f9d83173700 10 _finish_op 0x563cfc0 seq 2530 osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:39.187804 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b77a0 (0 active)
2014-06-12 14:47:39.187809 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:39.187815 7f9d7715b700  5 -- op tracker -- , seq: 748461, time: 2014-06-12 14:47:39.187805, event: commit_sent, op: osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.187824 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2721 (0'0,173'2721] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2719 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:39.187852 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:39.187880 7f9d7795c700  5 -- op tracker -- , seq: 748461, time: 2014-06-12 14:47:39.187874, event: sub_op_applied, op: osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.187890 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2721 (0'0,173'2721] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] sub_op_modify_applied on 0x73d6e00 op osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.187901 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2721 (0'0,173'2721] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] op_applied on primary on version 173'2721
2014-06-12 14:47:39.187907 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2721 (0'0,173'2721] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] op_applied on replica on version 173'2721
2014-06-12 14:47:39.187926 7f9d7795c700  5 -- op tracker -- , seq: 748461, time: 2014-06-12 14:47:39.187923, event: done, op: osd_sub_op(client.4284.0:141492 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2721 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.354264 7f9d5a16f700  5 -- op tracker -- , seq: 748462, time: 2014-06-12 14:47:39.353261, event: header_read, op: osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.354292 7f9d5a16f700  5 -- op tracker -- , seq: 748462, time: 2014-06-12 14:47:39.353266, event: throttled, op: osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.354311 7f9d5a16f700  5 -- op tracker -- , seq: 748462, time: 2014-06-12 14:47:39.354209, event: all_read, op: osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.354367 7f9d5a16f700  5 -- op tracker -- , seq: 748462, time: 0.000000, event: dispatched, op: osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.354376 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:39.354387 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:39.354400 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9c300 prio 127 cost 516899 latency 0.001138 osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.354451 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:39.354477 7f9d663dd700 10 osd.2 173 dequeue_op 0x24c9c300 prio 127 cost 516899 latency 0.001216 osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.f9( v 173'2721 (0'0,173'2721] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active]
2014-06-12 14:47:39.354505 7f9d663dd700  5 -- op tracker -- , seq: 748462, time: 2014-06-12 14:47:39.354503, event: reached_pg, op: osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.354518 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2721 (0'0,173'2721] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] handle_message: 0x24c9c300
2014-06-12 14:47:39.354535 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2721 (0'0,173'2721] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] sub_op_modify trans 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 v 173'2722 (transaction) 174
2014-06-12 14:47:39.354546 7f9d663dd700  5 -- op tracker -- , seq: 748462, time: 2014-06-12 14:47:39.354544, event: started, op: osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.354559 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2721 (0'0,173'2721] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] append_log log((0'0,173'2721], crt=0'0) [173'2722 (173'2720) modify   183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 by client.4288.0:139268 2014-06-12 08:44:23.639974]
2014-06-12 14:47:39.354579 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2722 (0'0,173'2722] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] add_log_entry 173'2722 (173'2720) modify   183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 by client.4288.0:139268 2014-06-12 08:44:23.639974
2014-06-12 14:47:39.354598 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2722 (0'0,173'2722] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] append_log  adding 1 keys
2014-06-12 14:47:39.354636 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2722, trimmed: 
2014-06-12 14:47:39.354663 7f9d663dd700  5 -- op tracker -- , seq: 748462, time: 2014-06-12 14:47:39.354660, event: started, op: osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.354682 7f9d663dd700  5 queue_transactions existing osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:39.354685 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:39.354686 7f9d663dd700  5 queue_op 0x5541ce0 seq 2531 osr(3.f9 0x276cfa0) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:47:39.354695 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b77a0 (1 active)
2014-06-12 14:47:39.354701 7f9d7a161700  5 _do_op 0x5541ce0 seq 2531 osr(3.f9 0x276cfa0)/0x276cfa0 start
2014-06-12 14:47:39.354691 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.f9( v 173'2722 (0'0,173'2722] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] do_sub_op osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.354704 7f9d663dd700 10 osd.2 173 dequeue_op 0x24c9c300 finish
2014-06-12 14:47:39.354706 7f9d7a161700 10 _do_transaction on 0x594a7f0
2014-06-12 14:47:39.354707 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:39.354709 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:39.354712 7f9d7a161700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.354839 7f9d7a161700 10 lookup_strip_header done  cid meta oid 516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.354875 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:39.354940 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:39.355077 7f9d7a161700 15 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.355083 7f9d7a161700 10 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1 r = 0
2014-06-12 14:47:39.355088 7f9d7a161700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.355111 7f9d7a161700 15 _write 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 757760~516096
2014-06-12 14:47:39.355255 7f9d7a161700 10 lookup_strip_header done  cid 3.f9_head oid 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3
2014-06-12 14:47:39.355458 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:47:39.357174 7f9d7a161700 10 _generic_write 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 757760~516096 = 0
2014-06-12 14:47:39.357222 7f9d7a161700 15 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3
2014-06-12 14:47:39.357232 7f9d7a161700 10 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 = 0
2014-06-12 14:47:39.357236 7f9d7a161700 15 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3
2014-06-12 14:47:39.357241 7f9d7a161700 10 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 = 0
2014-06-12 14:47:39.357376 7f9d7a161700 20 set_header setting 16861 cid 3.f9_head oid 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 parent seq 0
2014-06-12 14:47:39.357425 7f9d7a161700 20 set_header setting 2218 cid meta oid 516e9ea6/pglog_3.f9/0//-1 parent seq 0
2014-06-12 14:47:39.357436 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:39.357441 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:47:39.360566 7f9d7a161700 10 _do_op 0x5541ce0 seq 2531 r = 0, finisher 0x89f51e0 0
2014-06-12 14:47:39.360590 7f9d7a161700 10 _finish_op 0x5541ce0 seq 2531 osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:39.360603 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b77a0 (0 active)
2014-06-12 14:47:39.360614 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:39.360621 7f9d7715b700  5 -- op tracker -- , seq: 748462, time: 2014-06-12 14:47:39.360606, event: commit_sent, op: osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.360640 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2722 (0'0,173'2722] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2720 active] sub_op_modify_commit on op osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:39.360682 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:39.360713 7f9d7795c700  5 -- op tracker -- , seq: 748462, time: 2014-06-12 14:47:39.360705, event: sub_op_applied, op: osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.360743 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2722 (0'0,173'2722] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] sub_op_modify_applied on 0x594a700 op osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.360780 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2722 (0'0,173'2722] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] op_applied on primary on version 173'2722
2014-06-12 14:47:39.360801 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2722 (0'0,173'2722] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] op_applied on replica on version 173'2722
2014-06-12 14:47:39.360852 7f9d7795c700  5 -- op tracker -- , seq: 748462, time: 2014-06-12 14:47:39.360842, event: done, op: osd_sub_op(client.4288.0:139268 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2722 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.361464 7f9d5a16f700  5 -- op tracker -- , seq: 748463, time: 2014-06-12 14:47:39.361400, event: header_read, op: osd_sub_op_reply(client.4316.0:147320 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.361487 7f9d5a16f700  5 -- op tracker -- , seq: 748463, time: 2014-06-12 14:47:39.361404, event: throttled, op: osd_sub_op_reply(client.4316.0:147320 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.361519 7f9d5a16f700  5 -- op tracker -- , seq: 748463, time: 2014-06-12 14:47:39.361420, event: all_read, op: osd_sub_op_reply(client.4316.0:147320 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.361529 7f9d5a16f700  5 -- op tracker -- , seq: 748463, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147320 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.361536 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147320 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:39.361543 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:39.361553 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9dc00 prio 196 cost 0 latency 0.000152 osd_sub_op_reply(client.4316.0:147320 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:39.361600 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:39.361641 7f9d65bdc700 10 osd.2 173 dequeue_op 0x24c9dc00 prio 196 cost 0 latency 0.000240 osd_sub_op_reply(client.4316.0:147320 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 pg pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2976 active+clean]
2014-06-12 14:47:39.361692 7f9d65bdc700  5 -- op tracker -- , seq: 748463, time: 2014-06-12 14:47:39.361684, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147320 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.361718 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2976 active+clean] handle_message: 0x24c9dc00
2014-06-12 14:47:39.361749 7f9d65bdc700  5 -- op tracker -- , seq: 748463, time: 2014-06-12 14:47:39.361743, event: started, op: osd_sub_op_reply(client.4316.0:147320 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.361765 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2976 active+clean] sub_op_modify_reply: tid 236408 op  ack_type 4 from 24
2014-06-12 14:47:39.361816 7f9d65bdc700  5 -- op tracker -- , seq: 748344, time: 2014-06-12 14:47:39.361800, event: sub_op_commit_rec, op: osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:39.361836 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2976 active+clean] repop_all_applied: repop tid 236408 all applied 
2014-06-12 14:47:39.361859 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2976 active+clean] eval_repop repgather(0x2d813380 173'2978 rep_tid=236408 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:39.361895 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2976 active+clean]  sending ack on repgather(0x2d813380 173'2978 rep_tid=236408 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x21eba500
2014-06-12 14:47:39.361962 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2977 crt=0'0 lcod 173'2976 mlcod 173'2976 active+clean] repop_all_committed: repop tid 236408 all committed 
2014-06-12 14:47:39.361986 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2976 active+clean] eval_repop repgather(0x2d813380 173'2978 rep_tid=236408 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:39.362026 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2976 active+clean] log_op_stats osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 27.929921 lat 27.929987
2014-06-12 14:47:39.362063 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2976 active+clean] publish_stats_to_osd 173:3174
2014-06-12 14:47:39.362094 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2976 active+clean]  sending commit on repgather(0x2d813380 173'2978 rep_tid=236408 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x32df1180
2014-06-12 14:47:39.362177 7f9d65bdc700  5 -- op tracker -- , seq: 748344, time: 2014-06-12 14:47:39.362165, event: commit_sent, op: osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:39.362226 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2976 active+clean]  requeue_ops 
2014-06-12 14:47:39.362265 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean]  removing repgather(0x2d813380 173'2978 rep_tid=236408 committed?=1 applied?=1 op=osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:39.362320 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean]    q front is repgather(0x2d813380 173'2978 rep_tid=236408 committed?=1 applied?=1 op=osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:39.362348 7f9d65bdc700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean] remove_repop repgather(0x2d813380 173'2978 rep_tid=236408 committed?=1 applied?=1 op=osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:39.362363 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean]  requeue_ops 
2014-06-12 14:47:39.362388 7f9d65bdc700  5 -- op tracker -- , seq: 748344, time: 2014-06-12 14:47:39.362383, event: done, op: osd_op(client.4316.0:147320 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 774144~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:39.362405 7f9d65bdc700 10 osd.2 173 dequeue_op 0x24c9dc00 finish
2014-06-12 14:47:39.362412 7f9d65bdc700  5 -- op tracker -- , seq: 748463, time: 2014-06-12 14:47:39.362409, event: done, op: osd_sub_op_reply(client.4316.0:147320 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.362425 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:39.364471 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:39.433106 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:39.433230 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:39.486233 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:39.486314 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:39.486319 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 6.54 >= max 0.5 = no, load too high
2014-06-12 14:47:39.486338 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:47:39.486342 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92707 < max (604800 seconds)
2014-06-12 14:47:39.486396 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:47:39.486400 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:39.486402 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:39.486405 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 4; oldest is 27.257930 seconds old
2014-06-12 14:47:39.580574 7f9d5a16f700  5 -- op tracker -- , seq: 748464, time: 2014-06-12 14:47:39.576958, event: header_read, op: osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.580588 7f9d5a16f700  5 -- op tracker -- , seq: 748464, time: 2014-06-12 14:47:39.576964, event: throttled, op: osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.580597 7f9d5a16f700  5 -- op tracker -- , seq: 748464, time: 2014-06-12 14:47:39.580537, event: all_read, op: osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.580605 7f9d5a16f700  5 -- op tracker -- , seq: 748464, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.580609 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:39.580615 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:39.580634 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9de00 prio 127 cost 1032999 latency 0.003676 osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.580728 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:39.580755 7f9d62bd6700 10 osd.2 173 dequeue_op 0x24c9de00 prio 127 cost 1032999 latency 0.003796 osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2681 (0'0,173'2681] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active]
2014-06-12 14:47:39.580815 7f9d62bd6700  5 -- op tracker -- , seq: 748464, time: 2014-06-12 14:47:39.580811, event: reached_pg, op: osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.580825 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2681 (0'0,173'2681] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active] handle_message: 0x24c9de00
2014-06-12 14:47:39.580840 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2681 (0'0,173'2681] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active] sub_op_modify trans 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 v 173'2682 (transaction) 174
2014-06-12 14:47:39.580858 7f9d62bd6700  5 -- op tracker -- , seq: 748464, time: 2014-06-12 14:47:39.580853, event: started, op: osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.580875 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2681 (0'0,173'2681] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active] append_log log((0'0,173'2681], crt=0'0) [173'2682 (173'2681) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142257 2014-06-12 08:44:04.306502]
2014-06-12 14:47:39.580899 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2682 (0'0,173'2682] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active] add_log_entry 173'2682 (173'2681) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142257 2014-06-12 08:44:04.306502
2014-06-12 14:47:39.580929 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2682 (0'0,173'2682] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active] append_log  adding 1 keys
2014-06-12 14:47:39.580964 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2682, trimmed: 
2014-06-12 14:47:39.580991 7f9d62bd6700  5 -- op tracker -- , seq: 748464, time: 2014-06-12 14:47:39.580985, event: started, op: osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.581000 7f9d62bd6700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:39.581004 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:39.581006 7f9d62bd6700  5 queue_op 0x8349980 seq 2492 osr(3.1b7 0x276d1a0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:39.581013 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2682 (0'0,173'2682] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active] do_sub_op osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.581028 7f9d62bd6700 10 osd.2 173 dequeue_op 0x24c9de00 finish
2014-06-12 14:47:39.581032 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:39.581035 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:39.581043 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:39.581049 7f9d7f16b700  5 _do_op 0x8349980 seq 2492 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:39.581054 7f9d7f16b700 10 _do_transaction on 0x551a0f0
2014-06-12 14:47:39.581060 7f9d7f16b700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:39.581181 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:39.581202 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:39.581248 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:39.581275 7f9d7f16b700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:39.581282 7f9d7f16b700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:39.581288 7f9d7f16b700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:39.581309 7f9d7f16b700 15 _write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 253952~1032192
2014-06-12 14:47:39.581421 7f9d7f16b700 10 lookup_strip_header done  cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:39.581447 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:47:39.584955 7f9d7f16b700 10 _generic_write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 253952~1032192 = 0
2014-06-12 14:47:39.585077 7f9d7f16b700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:39.585097 7f9d7f16b700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:39.585106 7f9d7f16b700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:39.585116 7f9d7f16b700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:39.585591 7f9d7f16b700 20 set_header setting 16305 cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:47:39.585685 7f9d7f16b700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:39.585703 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:39.585716 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:47:39.589865 7f9d7f16b700 10 _do_op 0x8349980 seq 2492 r = 0, finisher 0x1ed3c9a0 0
2014-06-12 14:47:39.589890 7f9d7f16b700 10 _finish_op 0x8349980 seq 2492 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:39.589929 7f9d7795c700  5 -- op tracker -- , seq: 748464, time: 2014-06-12 14:47:39.589918, event: sub_op_applied, op: osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.589949 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2682 (0'0,173'2682] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active] sub_op_modify_applied on 0x551a000 op osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.589980 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:39.590013 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2682 (0'0,173'2682] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active] op_applied on primary on version 173'2682
2014-06-12 14:47:39.590023 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2682 (0'0,173'2682] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active] op_applied on replica on version 173'2682
2014-06-12 14:47:39.590040 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:39.590048 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:39.590051 7f9d7715b700  5 -- op tracker -- , seq: 748464, time: 2014-06-12 14:47:39.590042, event: commit_sent, op: osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.590067 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2682 (0'0,173'2682] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2680 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:39.590089 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:39.590126 7f9d7715b700  5 -- op tracker -- , seq: 748464, time: 2014-06-12 14:47:39.590121, event: done, op: osd_sub_op(client.4305.0:142257 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2682 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.739257 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:39.775324 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:39.778685 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:39.781370 7f9d5a16f700  5 -- op tracker -- , seq: 748465, time: 2014-06-12 14:47:39.780534, event: header_read, op: osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.781389 7f9d5a16f700  5 -- op tracker -- , seq: 748465, time: 2014-06-12 14:47:39.780537, event: throttled, op: osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.781415 7f9d5a16f700  5 -- op tracker -- , seq: 748465, time: 2014-06-12 14:47:39.781318, event: all_read, op: osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.781427 7f9d5a16f700  5 -- op tracker -- , seq: 748465, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.781435 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:39.781445 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:39.781455 7f9d5a16f700 15 osd.2 173 enqueue_op 0xd38f400 prio 127 cost 336679 latency 0.000920 osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.781510 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:39.781525 7f9d693e3700 10 osd.2 173 dequeue_op 0xd38f400 prio 127 cost 336679 latency 0.000991 osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2880 (0'0,173'2880] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active]
2014-06-12 14:47:39.781552 7f9d693e3700  5 -- op tracker -- , seq: 748465, time: 2014-06-12 14:47:39.781548, event: reached_pg, op: osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.781564 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2880 (0'0,173'2880] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active] handle_message: 0xd38f400
2014-06-12 14:47:39.781575 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2880 (0'0,173'2880] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active] sub_op_modify trans a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 v 173'2881 (transaction) 174
2014-06-12 14:47:39.781588 7f9d693e3700  5 -- op tracker -- , seq: 748465, time: 2014-06-12 14:47:39.781585, event: started, op: osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.781620 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2880 (0'0,173'2880] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active] append_log log((0'0,173'2880], crt=0'0) [173'2881 (173'873) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146505 2014-06-12 08:45:37.643441]
2014-06-12 14:47:39.781643 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2881 (0'0,173'2881] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active] add_log_entry 173'2881 (173'873) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146505 2014-06-12 08:45:37.643441
2014-06-12 14:47:39.781662 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2881 (0'0,173'2881] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active] append_log  adding 1 keys
2014-06-12 14:47:39.781690 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2881, trimmed: 
2014-06-12 14:47:39.781711 7f9d693e3700  5 -- op tracker -- , seq: 748465, time: 2014-06-12 14:47:39.781707, event: started, op: osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.781732 7f9d693e3700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:39.781736 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:39.781738 7f9d693e3700  5 queue_op 0x7be4060 seq 2594 osr(3.129 0x276cf20) 338165 bytes   (queue has 1 ops and 338165 bytes)
2014-06-12 14:47:39.781744 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2881 (0'0,173'2881] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active] do_sub_op osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.781750 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:39.781757 7f9d693e3700 10 osd.2 173 dequeue_op 0xd38f400 finish
2014-06-12 14:47:39.781760 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:39.781762 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:39.781760 7f9d82171700  5 _do_op 0x7be4060 seq 2594 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:39.781766 7f9d82171700 10 _do_transaction on 0x191d15f0
2014-06-12 14:47:39.781774 7f9d82171700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:39.782031 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:39.782065 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:39.782132 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:39.782164 7f9d82171700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:39.782175 7f9d82171700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:39.782182 7f9d82171700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:39.782202 7f9d82171700 15 _write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 0~335872
2014-06-12 14:47:39.782600 7f9d82171700 10 lookup_strip_header done  cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:39.782734 7f9d82171700 10 file_to_extents done 
2014-06-12 14:47:39.784271 7f9d82171700 10 _generic_write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 0~335872 = 0
2014-06-12 14:47:39.784339 7f9d82171700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:39.784356 7f9d82171700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:39.784364 7f9d82171700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:39.784374 7f9d82171700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:39.784601 7f9d82171700 20 set_header setting 17398 cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 parent seq 0
2014-06-12 14:47:39.784669 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:39.784687 7f9d82171700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:39.784700 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:47:39.787020 7f9d82171700 10 _do_op 0x7be4060 seq 2594 r = 0, finisher 0x65e5b20 0
2014-06-12 14:47:39.787043 7f9d82171700 10 _finish_op 0x7be4060 seq 2594 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:39.787049 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:39.787061 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:39.787074 7f9d7795c700  5 -- op tracker -- , seq: 748465, time: 2014-06-12 14:47:39.787061, event: sub_op_applied, op: osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.787083 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2881 (0'0,173'2881] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active] sub_op_modify_applied on 0x191d1500 op osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.787109 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:39.787127 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2881 (0'0,173'2881] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active] op_applied on primary on version 173'2881
2014-06-12 14:47:39.787135 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2881 (0'0,173'2881] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active] op_applied on replica on version 173'2881
2014-06-12 14:47:39.787187 7f9d7715b700  5 -- op tracker -- , seq: 748465, time: 2014-06-12 14:47:39.787180, event: commit_sent, op: osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.787196 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2881 (0'0,173'2881] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2879 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:39.787216 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:39.787252 7f9d7715b700  5 -- op tracker -- , seq: 748465, time: 2014-06-12 14:47:39.787248, event: done, op: osd_sub_op(client.4313.0:146505 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2881 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.839088 7f9d5a16f700  5 -- op tracker -- , seq: 748466, time: 2014-06-12 14:47:39.838163, event: header_read, op: osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.839098 7f9d5a16f700  5 -- op tracker -- , seq: 748466, time: 2014-06-12 14:47:39.838165, event: throttled, op: osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.839122 7f9d5a16f700  5 -- op tracker -- , seq: 748466, time: 2014-06-12 14:47:39.839048, event: all_read, op: osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.839128 7f9d5a16f700  5 -- op tracker -- , seq: 748466, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.839132 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:39.839137 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:39.839143 7f9d5a16f700 15 osd.2 173 enqueue_op 0xd38e100 prio 127 cost 516899 latency 0.000980 osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.839166 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:39.839182 7f9d683e1700 10 osd.2 173 dequeue_op 0xd38e100 prio 127 cost 516899 latency 0.001019 osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.f9( v 173'2722 (0'0,173'2722] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active]
2014-06-12 14:47:39.839209 7f9d683e1700  5 -- op tracker -- , seq: 748466, time: 2014-06-12 14:47:39.839204, event: reached_pg, op: osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.839218 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2722 (0'0,173'2722] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] handle_message: 0xd38e100
2014-06-12 14:47:39.839233 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2722 (0'0,173'2722] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] sub_op_modify trans 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 v 173'2723 (transaction) 174
2014-06-12 14:47:39.839259 7f9d683e1700  5 -- op tracker -- , seq: 748466, time: 2014-06-12 14:47:39.839254, event: started, op: osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.839275 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2722 (0'0,173'2722] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] append_log log((0'0,173'2722], crt=0'0) [173'2723 (173'2721) modify   3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 by client.4284.0:141493 2014-06-12 08:45:20.371430]
2014-06-12 14:47:39.839306 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2723 (0'0,173'2723] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] add_log_entry 173'2723 (173'2721) modify   3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 by client.4284.0:141493 2014-06-12 08:45:20.371430
2014-06-12 14:47:39.839329 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2723 (0'0,173'2723] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] append_log  adding 1 keys
2014-06-12 14:47:39.839359 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2723, trimmed: 
2014-06-12 14:47:39.839381 7f9d683e1700  5 -- op tracker -- , seq: 748466, time: 2014-06-12 14:47:39.839376, event: started, op: osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.839391 7f9d683e1700  5 queue_transactions existing osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:39.839395 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:39.839396 7f9d683e1700  5 queue_op 0x2e7c120 seq 2532 osr(3.f9 0x276cfa0) 518380 bytes   (queue has 1 ops and 518380 bytes)
2014-06-12 14:47:39.839415 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b77a0 (1 active)
2014-06-12 14:47:39.839420 7f9d7e96a700  5 _do_op 0x2e7c120 seq 2532 osr(3.f9 0x276cfa0)/0x276cfa0 start
2014-06-12 14:47:39.839411 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.f9( v 173'2723 (0'0,173'2723] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] do_sub_op osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.839425 7f9d7e96a700 10 _do_transaction on 0x678f970
2014-06-12 14:47:39.839427 7f9d683e1700 10 osd.2 173 dequeue_op 0xd38e100 finish
2014-06-12 14:47:39.839430 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:39.839431 7f9d7e96a700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.839433 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:39.839550 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.839563 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:39.839601 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:39.839792 7f9d7e96a700 15 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.839796 7f9d7e96a700 10 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1 r = 0
2014-06-12 14:47:39.839800 7f9d7e96a700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.839812 7f9d7e96a700 15 _write 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 1826816~516096
2014-06-12 14:47:39.839902 7f9d7e96a700 10 lookup_strip_header done  cid 3.f9_head oid 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:39.840069 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:47:39.841312 7f9d7e96a700 10 _generic_write 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 1826816~516096 = 0
2014-06-12 14:47:39.841339 7f9d7e96a700 15 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:39.841354 7f9d7e96a700 10 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 = 0
2014-06-12 14:47:39.841357 7f9d7e96a700 15 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:39.841362 7f9d7e96a700 10 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 = 0
2014-06-12 14:47:39.841494 7f9d7e96a700 20 set_header setting 17380 cid 3.f9_head oid 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 parent seq 0
2014-06-12 14:47:39.841532 7f9d7e96a700 20 set_header setting 2218 cid meta oid 516e9ea6/pglog_3.f9/0//-1 parent seq 0
2014-06-12 14:47:39.841541 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:39.841546 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:47:39.843424 7f9d7e96a700 10 _do_op 0x2e7c120 seq 2532 r = 0, finisher 0x80b8e20 0
2014-06-12 14:47:39.843439 7f9d7e96a700 10 _finish_op 0x2e7c120 seq 2532 osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:39.843447 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b77a0 (0 active)
2014-06-12 14:47:39.843455 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:39.843471 7f9d7715b700  5 -- op tracker -- , seq: 748466, time: 2014-06-12 14:47:39.843463, event: commit_sent, op: osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.843483 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2723 (0'0,173'2723] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2721 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:39.843505 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:39.843533 7f9d7795c700  5 -- op tracker -- , seq: 748466, time: 2014-06-12 14:47:39.843526, event: sub_op_applied, op: osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.843544 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2723 (0'0,173'2723] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] sub_op_modify_applied on 0x678f880 op osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.843557 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2723 (0'0,173'2723] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] op_applied on primary on version 173'2723
2014-06-12 14:47:39.843563 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2723 (0'0,173'2723] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] op_applied on replica on version 173'2723
2014-06-12 14:47:39.843581 7f9d7795c700  5 -- op tracker -- , seq: 748466, time: 2014-06-12 14:47:39.843578, event: done, op: osd_sub_op(client.4284.0:141493 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2723 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.932261 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:39.932528 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:39.933623 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4])
2014-06-12 14:47:39.933639 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4])
2014-06-12 14:47:39.934039 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:39.934078 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:39.934126 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:39.934159 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:39.934185 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:39.934206 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:39.934227 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:39.934247 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:39.934267 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:39.934288 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:39.934307 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:39.934327 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:39.934347 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:39.934368 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:39.934388 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:39.934410 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:39.934430 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:39.934475 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:39.935710 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:39.935731 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:39.935759 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:39.935778 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:39.947309 7f9d5a16f700  5 -- op tracker -- , seq: 748467, time: 2014-06-12 14:47:39.947242, event: header_read, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:39.947350 7f9d5a16f700  5 -- op tracker -- , seq: 748467, time: 2014-06-12 14:47:39.947245, event: throttled, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:39.947363 7f9d5a16f700  5 -- op tracker -- , seq: 748467, time: 2014-06-12 14:47:39.947262, event: all_read, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:39.947387 7f9d5a16f700  5 -- op tracker -- , seq: 748467, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:39.947397 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:39.947404 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:39.947412 7f9d5a16f700 15 osd.2 173 enqueue_op 0xd38ea00 prio 196 cost 0 latency 0.000170 osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0) v2
2014-06-12 14:47:39.947444 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:39.947477 7f9d5a16f700  5 -- op tracker -- , seq: 748468, time: 2014-06-12 14:47:39.947437, event: header_read, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.947467 7f9d6abe6700 10 osd.2 173 dequeue_op 0xd38ea00 prio 196 cost 0 latency 0.000224 osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0) v2 pg pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean]
2014-06-12 14:47:39.947494 7f9d5a16f700  5 -- op tracker -- , seq: 748468, time: 2014-06-12 14:47:39.947438, event: throttled, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.947522 7f9d5a16f700  5 -- op tracker -- , seq: 748468, time: 2014-06-12 14:47:39.947446, event: all_read, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.947536 7f9d5a16f700  5 -- op tracker -- , seq: 748468, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.947543 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:39.947556 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:39.947557 7f9d6abe6700  5 -- op tracker -- , seq: 748467, time: 2014-06-12 14:47:39.947494, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:39.947561 7f9d5a16f700 15 osd.2 173 enqueue_op 0x25e43000 prio 196 cost 0 latency 0.000124 osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:39.947567 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean] handle_message: 0xd38ea00
2014-06-12 14:47:39.947581 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (2 active)
2014-06-12 14:47:39.947584 7f9d6abe6700  5 -- op tracker -- , seq: 748467, time: 2014-06-12 14:47:39.947580, event: started, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:39.947596 7f9d6abe6700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean] sub_op_modify_reply: tid 236409 op  ack_type 1 from 24
2014-06-12 14:47:39.947620 7f9d6abe6700  5 -- op tracker -- , seq: 748349, time: 2014-06-12 14:47:39.947609, event: sub_op_applied_rec, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:39.947628 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean] repop_all_applied: repop tid 236409 all applied 
2014-06-12 14:47:39.947639 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean] eval_repop repgather(0x2680240 173'2979 rep_tid=236409 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:39.947656 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean]  sending ack on repgather(0x2680240 173'2979 rep_tid=236409 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x8df3180
2014-06-12 14:47:39.947688 7f9d6abe6700 10 osd.2 173 dequeue_op 0xd38ea00 finish
2014-06-12 14:47:39.947697 7f9d6abe6700  5 -- op tracker -- , seq: 748467, time: 2014-06-12 14:47:39.947694, event: done, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:39.947695 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x25e43000 prio 196 cost 0 latency 0.000258 osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 pg pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean]
2014-06-12 14:47:39.947713 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (1 active)
2014-06-12 14:47:39.947717 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:39.947715 7f9d6a3e5700  5 -- op tracker -- , seq: 748468, time: 2014-06-12 14:47:39.947707, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.947725 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean] handle_message: 0x25e43000
2014-06-12 14:47:39.947734 7f9d6a3e5700  5 -- op tracker -- , seq: 748468, time: 2014-06-12 14:47:39.947732, event: started, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.947738 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean] sub_op_modify_reply: tid 236409 op  ack_type 4 from 24
2014-06-12 14:47:39.947749 7f9d6a3e5700  5 -- op tracker -- , seq: 748349, time: 2014-06-12 14:47:39.947745, event: sub_op_commit_rec, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:39.947753 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2978 crt=0'0 lcod 173'2977 mlcod 173'2977 active+clean] repop_all_committed: repop tid 236409 all committed 
2014-06-12 14:47:39.947758 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2977 active+clean] eval_repop repgather(0x2680240 173'2979 rep_tid=236409 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:39.947773 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2977 active+clean] log_op_stats osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 inb 1032471 outb 0 rlat 27.719212 lat 27.719299
2014-06-12 14:47:39.947790 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2977 active+clean] publish_stats_to_osd 173:3175
2014-06-12 14:47:39.947798 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2977 active+clean]  sending commit on repgather(0x2680240 173'2979 rep_tid=236409 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x2b55400
2014-06-12 14:47:39.947819 7f9d6a3e5700  5 -- op tracker -- , seq: 748349, time: 2014-06-12 14:47:39.947815, event: commit_sent, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:39.947824 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2977 active+clean]  requeue_ops 
2014-06-12 14:47:39.947830 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean]  removing repgather(0x2680240 173'2979 rep_tid=236409 committed?=1 applied?=1 op=osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:39.947838 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean]    q front is repgather(0x2680240 173'2979 rep_tid=236409 committed?=1 applied?=1 op=osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:39.947847 7f9d6a3e5700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean] remove_repop repgather(0x2680240 173'2979 rep_tid=236409 committed?=1 applied?=1 op=osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:39.947855 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean]  requeue_ops 
2014-06-12 14:47:39.947866 7f9d6a3e5700  5 -- op tracker -- , seq: 748349, time: 2014-06-12 14:47:39.947863, event: done, op: osd_op(client.4316.0:147321 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 1806336~1032192] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:39.947877 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x25e43000 finish
2014-06-12 14:47:39.947881 7f9d6a3e5700  5 -- op tracker -- , seq: 748468, time: 2014-06-12 14:47:39.947879, event: done, op: osd_sub_op_reply(client.4316.0:147321 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:39.947887 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:39.947889 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:39.966357 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:39.966703 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:39.974694 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:39.975898 7f9d5a16f700  5 -- op tracker -- , seq: 748469, time: 2014-06-12 14:47:39.971872, event: header_read, op: osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.975910 7f9d5a16f700  5 -- op tracker -- , seq: 748469, time: 2014-06-12 14:47:39.971875, event: throttled, op: osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.975917 7f9d5a16f700  5 -- op tracker -- , seq: 748469, time: 2014-06-12 14:47:39.975854, event: all_read, op: osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.975925 7f9d5a16f700  5 -- op tracker -- , seq: 748469, time: 0.000000, event: dispatched, op: osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.975931 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:39.975937 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:39.975946 7f9d5a16f700 15 osd.2 173 enqueue_op 0x24c9d700 prio 127 cost 1549091 latency 0.004073 osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.975977 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:39.975983 7f9d66bde700 10 osd.2 173 dequeue_op 0x24c9d700 prio 127 cost 1549091 latency 0.004111 osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.f9( v 173'2723 (0'0,173'2723] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active]
2014-06-12 14:47:39.976012 7f9d66bde700  5 -- op tracker -- , seq: 748469, time: 2014-06-12 14:47:39.976008, event: reached_pg, op: osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.976020 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2723 (0'0,173'2723] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] handle_message: 0x24c9d700
2014-06-12 14:47:39.976036 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2723 (0'0,173'2723] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] sub_op_modify trans 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 v 173'2724 (transaction) 174
2014-06-12 14:47:39.976051 7f9d66bde700  5 -- op tracker -- , seq: 748469, time: 2014-06-12 14:47:39.976048, event: started, op: osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.976067 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2723 (0'0,173'2723] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] append_log log((0'0,173'2723], crt=0'0) [173'2724 (173'2722) modify   183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 by client.4288.0:139269 2014-06-12 08:44:23.654709]
2014-06-12 14:47:39.976091 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2724 (0'0,173'2724] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] add_log_entry 173'2724 (173'2722) modify   183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 by client.4288.0:139269 2014-06-12 08:44:23.654709
2014-06-12 14:47:39.976113 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2724 (0'0,173'2724] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] append_log  adding 1 keys
2014-06-12 14:47:39.976151 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2724, trimmed: 
2014-06-12 14:47:39.976175 7f9d66bde700  5 -- op tracker -- , seq: 748469, time: 2014-06-12 14:47:39.976171, event: started, op: osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.976186 7f9d66bde700  5 queue_transactions existing osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:39.976190 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:39.976191 7f9d66bde700  5 queue_op 0x61f2780 seq 2533 osr(3.f9 0x276cfa0) 1550572 bytes   (queue has 1 ops and 1550572 bytes)
2014-06-12 14:47:39.976197 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.f9( v 173'2724 (0'0,173'2724] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] do_sub_op osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.976211 7f9d66bde700 10 osd.2 173 dequeue_op 0x24c9d700 finish
2014-06-12 14:47:39.976214 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:39.976216 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:39.976224 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b77a0 (1 active)
2014-06-12 14:47:39.976229 7f9d7d167700  5 _do_op 0x61f2780 seq 2533 osr(3.f9 0x276cfa0)/0x276cfa0 start
2014-06-12 14:47:39.976233 7f9d7d167700 10 _do_transaction on 0x65af0b0
2014-06-12 14:47:39.976238 7f9d7d167700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.976459 7f9d7d167700 10 lookup_strip_header done  cid meta oid 516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.976479 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:39.976523 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:39.976542 7f9d7d167700 15 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.976548 7f9d7d167700 10 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1 r = 0
2014-06-12 14:47:39.976554 7f9d7d167700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:39.976570 7f9d7d167700 15 _write 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 1273856~1548288
2014-06-12 14:47:39.976894 7f9d7d167700 10 lookup_strip_header done  cid 3.f9_head oid 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3
2014-06-12 14:47:39.977058 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:47:39.981972 7f9d7d167700 10 _generic_write 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 1273856~1548288 = 0
2014-06-12 14:47:39.982118 7f9d7d167700 15 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3
2014-06-12 14:47:39.982135 7f9d7d167700 10 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 = 0
2014-06-12 14:47:39.982139 7f9d7d167700 15 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3
2014-06-12 14:47:39.982147 7f9d7d167700 10 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 = 0
2014-06-12 14:47:39.982475 7f9d7d167700 20 set_header setting 16861 cid 3.f9_head oid 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 parent seq 0
2014-06-12 14:47:39.982576 7f9d7d167700 20 set_header setting 2218 cid meta oid 516e9ea6/pglog_3.f9/0//-1 parent seq 0
2014-06-12 14:47:39.982586 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:39.982591 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:47:39.991409 7f9d7d167700 10 _do_op 0x61f2780 seq 2533 r = 0, finisher 0xabf3b40 0
2014-06-12 14:47:39.991433 7f9d7d167700 10 _finish_op 0x61f2780 seq 2533 osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:39.991457 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b77a0 (0 active)
2014-06-12 14:47:39.991466 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:39.991460 7f9d7715b700  5 -- op tracker -- , seq: 748469, time: 2014-06-12 14:47:39.991448, event: commit_sent, op: osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.991469 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2724 (0'0,173'2724] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2722 active] sub_op_modify_commit on op osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:39.991500 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:39.991530 7f9d7795c700  5 -- op tracker -- , seq: 748469, time: 2014-06-12 14:47:39.991524, event: sub_op_applied, op: osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:39.991541 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2724 (0'0,173'2724] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] sub_op_modify_applied on 0x65aefc0 op osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:39.991558 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2724 (0'0,173'2724] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] op_applied on primary on version 173'2724
2014-06-12 14:47:39.991568 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2724 (0'0,173'2724] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] op_applied on replica on version 173'2724
2014-06-12 14:47:39.991595 7f9d7795c700  5 -- op tracker -- , seq: 748469, time: 2014-06-12 14:47:39.991591, event: done, op: osd_sub_op(client.4288.0:139269 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2724 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:40.127249 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:40.129881 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:40.134051 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:40.138204 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:40.139136 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:40.139562 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:40.139775 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:40.147311 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:40.154674 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:40.156819 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:40.157029 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:40.179062 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:40.179289 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:40.314246 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:40.323545 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:40.323778 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:40.329283 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:40.365767 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:40.365812 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:40.371615 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:40.486470 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:40.486489 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:40.486492 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:40.486493 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:40.486494 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 3; oldest is 28.012528 seconds old
2014-06-12 14:47:40.486524 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:40.493589 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:40.502924 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:40.611603 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:40.611858 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:40.664692 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:40.671177 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:40.698560 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:40.705554 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:40.811463 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:41.008458 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:41.011174 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:41.034127 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19431 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3])
2014-06-12 14:47:41.034144 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19431 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3])
2014-06-12 14:47:41.034531 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:41.034556 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:41.034622 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:41.034652 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:41.034686 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:41.034708 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:41.034728 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:41.034742 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:41.034753 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:41.034763 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:41.034775 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:41.034784 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:41.034798 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:41.034815 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:41.034834 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:41.034845 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:41.034857 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:41.034871 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:41.034886 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:41.034905 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:41.034919 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:41.034942 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:41.155737 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:41.155789 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:41.177527 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:41.187870 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:41.279967 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:41.280009 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:41.354785 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:41.360676 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:41.364536 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:41.427374 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:41.427635 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:41.486559 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:41.486611 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:41.486615 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 6.42 >= max 0.5 = no, load too high
2014-06-12 14:47:41.486625 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:47:41.486628 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92709 < max (604800 seconds)
2014-06-12 14:47:41.486646 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:47:41.486648 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:41.486648 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:41.486650 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 3; oldest is 29.012683 seconds old
2014-06-12 14:47:41.534633 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19432 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3])
2014-06-12 14:47:41.534661 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19432 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3])
2014-06-12 14:47:41.535331 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:41.535379 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:41.535418 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:41.535460 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:41.535515 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:41.535548 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:41.535569 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:41.535586 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:41.535622 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:41.535643 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:41.535674 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:41.535700 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:41.535721 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:41.535741 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:41.535761 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:41.535782 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:41.535803 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:41.535879 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:41.535921 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:41.535955 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:41.535981 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:41.536004 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:41.581094 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:41.590112 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:41.739332 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:41.752198 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:41.752255 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:41.775430 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:41.778784 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:41.781853 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:41.787165 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:41.839507 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:41.843526 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:41.947778 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:41.947947 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:41.967092 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:41.967104 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:41.967105 7f9d74bfa700 20 osd.2 173 _dispatch 0x13bc3a00 ping v1
2014-06-12 14:47:41.967108 7f9d74bfa700 10 osd.2 173 ping from client.4316
2014-06-12 14:47:41.967110 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:41.967111 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:41.974784 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:41.976277 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:41.991527 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:42.127356 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:42.129970 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:42.134133 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:42.138272 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:42.139202 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:42.139628 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:42.139839 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:42.147380 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:42.154742 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:42.156883 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:42.157096 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:42.314341 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:42.329355 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:42.333852 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:42.334308 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:42.371692 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:42.486742 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:42.486769 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:47:42.486776 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:47:42.486783 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:47:42.486791 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:47:42.486860 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:42.486865 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:42.486869 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:42.486872 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 3; oldest is 30.012906 seconds old
2014-06-12 14:47:42.486908 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:42.486927 7f9d84175700  0 log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 30.012906 secs
2014-06-12 14:47:42.486938 7f9d84175700  0 log [WRN] : slow request 30.012906 seconds old, received at 2014-06-12 14:47:12.473966: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:47:42.493673 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:42.503005 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:42.540238 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:42.540242 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:42.540243 7f9d74bfa700 20 osd.2 173 _dispatch 0x22572700 pg_stats_ack(1 pgs tid 4864) v1
2014-06-12 14:47:42.540246 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:47:42.540252 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:42.540254 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:42.664769 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:42.671259 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:42.698642 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:42.705633 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:42.811551 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:42.856463 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:42.856525 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:42.949752 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:42.949919 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:43.008542 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:43.011249 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:43.177627 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:43.187964 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:43.354870 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:43.360771 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:43.364612 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:43.432873 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:43.432968 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:43.466355 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:43.466415 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:43.487017 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:43.487044 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:43.487049 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:43.487051 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:43.487054 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 3; oldest is 31.013088 seconds old
2014-06-12 14:47:43.487082 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:43.487108 7f9d84175700  0 log [WRN] : 3 slow requests, 2 included below; oldest blocked for > 31.013088 secs
2014-06-12 14:47:43.487114 7f9d84175700  0 log [WRN] : slow request 30.834989 seconds old, received at 2014-06-12 14:47:12.652065: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:47:43.487119 7f9d84175700  0 log [WRN] : slow request 30.805208 seconds old, received at 2014-06-12 14:47:12.681846: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) currently waiting for subops from 24
2014-06-12 14:47:43.581170 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:43.590215 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:43.739403 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:43.775534 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:43.778882 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:43.781940 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:43.787255 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:43.839595 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:43.843607 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:43.947859 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:43.948014 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:43.974859 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:43.976371 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:43.991610 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:44.019759 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:44.020234 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:44.127459 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:44.130048 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:44.134213 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:44.138351 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:44.139291 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:44.139705 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:44.139903 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:44.147461 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:44.154818 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:44.156955 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:44.157169 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:44.314418 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:44.329425 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:44.371801 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:44.424097 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:44.424155 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:44.487199 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:44.487225 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:44.487230 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:44.487232 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:44.487235 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 3; oldest is 32.013269 seconds old
2014-06-12 14:47:44.487265 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:44.493752 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:44.503077 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:44.653079 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:44.653147 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:44.664854 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:44.671340 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:44.698741 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:44.705708 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:44.811654 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:44.924646 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:44.924840 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:45.008631 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:45.011348 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:45.035424 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19427 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3])
2014-06-12 14:47:45.035448 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19427 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3])
2014-06-12 14:47:45.036234 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:45.036334 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:45.036396 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:45.036451 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:45.036500 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:45.036549 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:45.036592 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:45.036637 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:45.036684 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:45.036731 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:45.036777 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:45.036823 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:45.036871 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:45.036920 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:45.036968 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:45.037014 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:45.037062 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:45.037108 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:45.037162 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:45.037210 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:45.037251 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:45.037306 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:45.177734 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:45.188036 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:45.250253 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:45.250418 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:45.354945 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:45.360869 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:45.364696 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:45.487300 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:45.487317 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:45.487319 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:45.487320 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:45.487322 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 3; oldest is 33.013355 seconds old
2014-06-12 14:47:45.487360 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:45.581288 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:45.590289 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:45.720194 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:45.720236 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:45.739502 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:45.756969 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:45.757005 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:45.775618 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:45.778979 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:45.782019 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:45.787349 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:45.839682 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:45.843689 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:45.868494 7f9d5a16f700  5 -- op tracker -- , seq: 748470, time: 2014-06-12 14:47:45.867551, event: header_read, op: osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:45.868514 7f9d5a16f700  5 -- op tracker -- , seq: 748470, time: 2014-06-12 14:47:45.867555, event: throttled, op: osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:45.868523 7f9d5a16f700  5 -- op tracker -- , seq: 748470, time: 2014-06-12 14:47:45.868440, event: all_read, op: osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:45.868531 7f9d5a16f700  5 -- op tracker -- , seq: 748470, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:45.868538 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:45.868545 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:45.868554 7f9d5a16f700 15 osd.2 173 enqueue_op 0x25e43a00 prio 127 cost 516903 latency 0.001003 osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:45.868638 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:45.868650 7f9d6abe6700 10 osd.2 173 dequeue_op 0x25e43a00 prio 127 cost 516903 latency 0.001098 osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2682 (0'0,173'2682] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2681 active]
2014-06-12 14:47:45.868681 7f9d6abe6700  5 -- op tracker -- , seq: 748470, time: 2014-06-12 14:47:45.868676, event: reached_pg, op: osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:45.868691 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2682 (0'0,173'2682] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2681 active] handle_message: 0x25e43a00
2014-06-12 14:47:45.868706 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2682 (0'0,173'2682] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2681 active] sub_op_modify trans 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 v 173'2683 (transaction) 174
2014-06-12 14:47:45.868727 7f9d6abe6700  5 -- op tracker -- , seq: 748470, time: 2014-06-12 14:47:45.868722, event: started, op: osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:45.868748 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2682 (0'0,173'2682] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2681 active] append_log log((0'0,173'2682], crt=0'0) [173'2683 (173'2682) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142258 2014-06-12 08:44:04.379941]
2014-06-12 14:47:45.868772 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2683 (0'0,173'2683] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2681 active] add_log_entry 173'2683 (173'2682) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142258 2014-06-12 08:44:04.379941
2014-06-12 14:47:45.868796 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2683 (0'0,173'2683] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2681 active] append_log  adding 1 keys
2014-06-12 14:47:45.868833 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2683, trimmed: 
2014-06-12 14:47:45.868867 7f9d6abe6700  5 -- op tracker -- , seq: 748470, time: 2014-06-12 14:47:45.868863, event: started, op: osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:45.868878 7f9d6abe6700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:45.868883 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:45.868885 7f9d6abe6700  5 queue_op 0x11b13f80 seq 2493 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:45.868898 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:45.868905 7f9d8116f700  5 _do_op 0x11b13f80 seq 2493 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:45.868893 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2683 (0'0,173'2683] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2681 active] do_sub_op osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:45.868909 7f9d6abe6700 10 osd.2 173 dequeue_op 0x25e43a00 finish
2014-06-12 14:47:45.868910 7f9d8116f700 10 _do_transaction on 0x62ce470
2014-06-12 14:47:45.868913 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:45.868916 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:45.868917 7f9d8116f700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:45.869045 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:45.869176 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:45.869228 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:45.869255 7f9d8116f700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:45.869262 7f9d8116f700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:45.869269 7f9d8116f700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:45.869286 7f9d8116f700 15 _write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 1286144~516096
2014-06-12 14:47:45.869637 7f9d8116f700 10 lookup_strip_header done  cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:45.869781 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:47:45.871587 7f9d8116f700 10 _generic_write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 1286144~516096 = 0
2014-06-12 14:47:45.871629 7f9d8116f700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:45.871642 7f9d8116f700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:45.871647 7f9d8116f700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:45.871652 7f9d8116f700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:45.871803 7f9d8116f700 20 set_header setting 16305 cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:47:45.871859 7f9d8116f700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:45.871868 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:45.871873 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:47:45.873713 7f9d8116f700 10 _do_op 0x11b13f80 seq 2493 r = 0, finisher 0x1f24860 0
2014-06-12 14:47:45.873727 7f9d8116f700 10 _finish_op 0x11b13f80 seq 2493 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:45.873733 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:45.873736 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:45.873741 7f9d7715b700  5 -- op tracker -- , seq: 748470, time: 2014-06-12 14:47:45.873734, event: commit_sent, op: osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:45.873759 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2683 (0'0,173'2683] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2681 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:45.873777 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:45.873805 7f9d7795c700  5 -- op tracker -- , seq: 748470, time: 2014-06-12 14:47:45.873797, event: sub_op_applied, op: osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:45.873821 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2683 (0'0,173'2683] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2682 active] sub_op_modify_applied on 0x62ce380 op osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:45.873838 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2683 (0'0,173'2683] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2682 active] op_applied on primary on version 173'2683
2014-06-12 14:47:45.873844 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2683 (0'0,173'2683] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2682 active] op_applied on replica on version 173'2683
2014-06-12 14:47:45.873857 7f9d7795c700  5 -- op tracker -- , seq: 748470, time: 2014-06-12 14:47:45.873854, event: done, op: osd_sub_op(client.4305.0:142258 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2683 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:45.902509 7f9d5a16f700  5 -- op tracker -- , seq: 748471, time: 2014-06-12 14:47:45.902453, event: header_read, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:45.902528 7f9d5a16f700  5 -- op tracker -- , seq: 748471, time: 2014-06-12 14:47:45.902455, event: throttled, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:45.902543 7f9d5a16f700  5 -- op tracker -- , seq: 748471, time: 2014-06-12 14:47:45.902467, event: all_read, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:45.902549 7f9d5a16f700  5 -- op tracker -- , seq: 748471, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:45.902555 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0) v2 epoch 173
2014-06-12 14:47:45.902560 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:45.902568 7f9d5a16f700 15 osd.2 173 enqueue_op 0x25e43500 prio 196 cost 0 latency 0.000115 osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0) v2
2014-06-12 14:47:45.902600 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:45.902610 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x25e43500 prio 196 cost 0 latency 0.000157 osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0) v2 pg pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean]
2014-06-12 14:47:45.902634 7f9d6a3e5700  5 -- op tracker -- , seq: 748471, time: 2014-06-12 14:47:45.902630, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:45.902643 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean] handle_message: 0x25e43500
2014-06-12 14:47:45.902663 7f9d6a3e5700  5 -- op tracker -- , seq: 748471, time: 2014-06-12 14:47:45.902659, event: started, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:45.902670 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean] sub_op_modify_reply: tid 236410 op  ack_type 1 from 24
2014-06-12 14:47:45.902696 7f9d6a3e5700  5 -- op tracker -- , seq: 748353, time: 2014-06-12 14:47:45.902685, event: sub_op_applied_rec, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:45.902706 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean] repop_all_applied: repop tid 236410 all applied 
2014-06-12 14:47:45.902719 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean] eval_repop repgather(0x523c600 173'2980 rep_tid=236410 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:45.902739 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean]  sending ack on repgather(0x523c600 173'2980 rep_tid=236410 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x10097400
2014-06-12 14:47:45.902772 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x25e43500 finish
2014-06-12 14:47:45.902781 7f9d6a3e5700  5 -- op tracker -- , seq: 748471, time: 2014-06-12 14:47:45.902777, event: done, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ack, result = 0)
2014-06-12 14:47:45.902795 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:45.902799 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:45.904652 7f9d5a16f700  5 -- op tracker -- , seq: 748472, time: 2014-06-12 14:47:45.904610, event: header_read, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:45.904663 7f9d5a16f700  5 -- op tracker -- , seq: 748472, time: 2014-06-12 14:47:45.904612, event: throttled, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:45.904668 7f9d5a16f700  5 -- op tracker -- , seq: 748472, time: 2014-06-12 14:47:45.904627, event: all_read, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:45.904672 7f9d5a16f700  5 -- op tracker -- , seq: 748472, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:45.904675 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:45.904678 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:45.904682 7f9d5a16f700 15 osd.2 173 enqueue_op 0x25e42200 prio 196 cost 0 latency 0.000072 osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:45.904702 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:45.904708 7f9d66bde700 10 osd.2 173 dequeue_op 0x25e42200 prio 196 cost 0 latency 0.000097 osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 pg pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean]
2014-06-12 14:47:45.904723 7f9d66bde700  5 -- op tracker -- , seq: 748472, time: 2014-06-12 14:47:45.904720, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:45.904728 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean] handle_message: 0x25e42200
2014-06-12 14:47:45.904739 7f9d66bde700  5 -- op tracker -- , seq: 748472, time: 2014-06-12 14:47:45.904737, event: started, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:45.904743 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean] sub_op_modify_reply: tid 236410 op  ack_type 4 from 24
2014-06-12 14:47:45.904758 7f9d66bde700  5 -- op tracker -- , seq: 748353, time: 2014-06-12 14:47:45.904753, event: sub_op_commit_rec, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:45.904763 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2979 crt=0'0 lcod 173'2978 mlcod 173'2978 active+clean] repop_all_committed: repop tid 236410 all committed 
2014-06-12 14:47:45.904770 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2978 active+clean] eval_repop repgather(0x523c600 173'2980 rep_tid=236410 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:45.904781 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2978 active+clean] log_op_stats osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 33.428804 lat 33.430814
2014-06-12 14:47:45.904793 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2978 active+clean] publish_stats_to_osd 173:3176
2014-06-12 14:47:45.904802 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2978 active+clean]  sending commit on repgather(0x523c600 173'2980 rep_tid=236410 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x14c46c80
2014-06-12 14:47:45.904827 7f9d66bde700  5 -- op tracker -- , seq: 748353, time: 2014-06-12 14:47:45.904821, event: commit_sent, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:45.904836 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2978 active+clean]  requeue_ops 
2014-06-12 14:47:45.904850 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2979 active+clean]  removing repgather(0x523c600 173'2980 rep_tid=236410 committed?=1 applied?=1 op=osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:45.904862 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2979 active+clean]    q front is repgather(0x523c600 173'2980 rep_tid=236410 committed?=1 applied?=1 op=osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:45.904872 7f9d66bde700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2979 active+clean] remove_repop repgather(0x523c600 173'2980 rep_tid=236410 committed?=1 applied?=1 op=osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:45.904881 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2979 active+clean]  requeue_ops 
2014-06-12 14:47:45.904895 7f9d66bde700  5 -- op tracker -- , seq: 748353, time: 2014-06-12 14:47:45.904891, event: done, op: osd_op(client.4316.0:147322 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 2838528~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:45.904912 7f9d66bde700 10 osd.2 173 dequeue_op 0x25e42200 finish
2014-06-12 14:47:45.904922 7f9d66bde700  5 -- op tracker -- , seq: 748472, time: 2014-06-12 14:47:45.904918, event: done, op: osd_sub_op_reply(client.4316.0:147322 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:45.904934 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:45.904936 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:45.980584 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:45.980831 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:45.991680 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.009895 7f9d5a16f700  5 -- op tracker -- , seq: 748473, time: 2014-06-12 14:47:46.008833, event: header_read, op: osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.009931 7f9d5a16f700  5 -- op tracker -- , seq: 748473, time: 2014-06-12 14:47:46.008838, event: throttled, op: osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.010016 7f9d5a16f700  5 -- op tracker -- , seq: 748473, time: 2014-06-12 14:47:46.009841, event: all_read, op: osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.010024 7f9d5a16f700  5 -- op tracker -- , seq: 748473, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.010028 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:46.010033 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:46.010040 7f9d5a16f700 15 osd.2 173 enqueue_op 0x22b3b100 prio 127 cost 516903 latency 0.001207 osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.010100 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:46.010111 7f9d69be4700 10 osd.2 173 dequeue_op 0x22b3b100 prio 127 cost 516903 latency 0.001277 osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2881 (0'0,173'2881] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2880 active]
2014-06-12 14:47:46.010132 7f9d69be4700  5 -- op tracker -- , seq: 748473, time: 2014-06-12 14:47:46.010129, event: reached_pg, op: osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.010141 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2881 (0'0,173'2881] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2880 active] handle_message: 0x22b3b100
2014-06-12 14:47:46.010150 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2881 (0'0,173'2881] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2880 active] sub_op_modify trans a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 v 173'2882 (transaction) 174
2014-06-12 14:47:46.010161 7f9d69be4700  5 -- op tracker -- , seq: 748473, time: 2014-06-12 14:47:46.010158, event: started, op: osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.010173 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2881 (0'0,173'2881] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2880 active] append_log log((0'0,173'2881], crt=0'0) [173'2882 (173'2881) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146506 2014-06-12 08:45:37.647468]
2014-06-12 14:47:46.010191 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2882 (0'0,173'2882] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2880 active] add_log_entry 173'2882 (173'2881) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146506 2014-06-12 08:45:37.647468
2014-06-12 14:47:46.010206 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2882 (0'0,173'2882] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2880 active] append_log  adding 1 keys
2014-06-12 14:47:46.010229 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2882, trimmed: 
2014-06-12 14:47:46.010246 7f9d69be4700  5 -- op tracker -- , seq: 748473, time: 2014-06-12 14:47:46.010243, event: started, op: osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.010255 7f9d69be4700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:46.010258 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:46.010259 7f9d69be4700  5 queue_op 0x15cdba40 seq 2595 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:46.010264 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2882 (0'0,173'2882] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2880 active] do_sub_op osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.010274 7f9d69be4700 10 osd.2 173 dequeue_op 0x22b3b100 finish
2014-06-12 14:47:46.010283 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:46.010285 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.010296 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:46.010303 7f9d7915f700  5 _do_op 0x15cdba40 seq 2595 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:46.010314 7f9d7915f700 10 _do_transaction on 0x6295970
2014-06-12 14:47:46.010320 7f9d7915f700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:46.010411 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:46.010436 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:46.010464 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:46.010481 7f9d7915f700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:46.010486 7f9d7915f700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:46.010489 7f9d7915f700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:46.010501 7f9d7915f700 15 _write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 335872~516096
2014-06-12 14:47:46.010710 7f9d7915f700 10 lookup_strip_header done  cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:46.010802 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:47:46.011953 7f9d7915f700 10 _generic_write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 335872~516096 = 0
2014-06-12 14:47:46.011983 7f9d7915f700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:46.011992 7f9d7915f700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:46.011996 7f9d7915f700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:46.012000 7f9d7915f700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:46.012147 7f9d7915f700 20 set_header setting 17398 cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 parent seq 0
2014-06-12 14:47:46.012196 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:46.012205 7f9d7915f700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:46.012211 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:47:46.013725 7f9d7915f700 10 _do_op 0x15cdba40 seq 2595 r = 0, finisher 0x89f4400 0
2014-06-12 14:47:46.013736 7f9d7915f700 10 _finish_op 0x15cdba40 seq 2595 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:46.013740 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:46.013744 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.013778 7f9d7715b700  5 -- op tracker -- , seq: 748473, time: 2014-06-12 14:47:46.013769, event: commit_sent, op: osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.013791 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2882 (0'0,173'2882] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2880 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:46.013814 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.013843 7f9d7795c700  5 -- op tracker -- , seq: 748473, time: 2014-06-12 14:47:46.013837, event: sub_op_applied, op: osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.013851 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2882 (0'0,173'2882] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active] sub_op_modify_applied on 0x6295880 op osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.013862 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2882 (0'0,173'2882] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active] op_applied on primary on version 173'2882
2014-06-12 14:47:46.013867 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2882 (0'0,173'2882] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active] op_applied on replica on version 173'2882
2014-06-12 14:47:46.013880 7f9d7795c700  5 -- op tracker -- , seq: 748473, time: 2014-06-12 14:47:46.013878, event: done, op: osd_sub_op(client.4313.0:146506 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2882 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.040101 7f9d5a16f700  5 -- op tracker -- , seq: 748474, time: 2014-06-12 14:47:46.035909, event: header_read, op: osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.040127 7f9d5a16f700  5 -- op tracker -- , seq: 748474, time: 2014-06-12 14:47:46.035911, event: throttled, op: osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.040156 7f9d5a16f700  5 -- op tracker -- , seq: 748474, time: 2014-06-12 14:47:46.039935, event: all_read, op: osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.040168 7f9d5a16f700  5 -- op tracker -- , seq: 748474, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.040176 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:46.040210 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:46.040222 7f9d5a16f700 15 osd.2 173 enqueue_op 0x25e43700 prio 127 cost 1549091 latency 0.004313 osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.040298 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:46.040321 7f9d64bda700 10 osd.2 173 dequeue_op 0x25e43700 prio 127 cost 1549091 latency 0.004412 osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.f9( v 173'2724 (0'0,173'2724] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active]
2014-06-12 14:47:46.040367 7f9d64bda700  5 -- op tracker -- , seq: 748474, time: 2014-06-12 14:47:46.040359, event: reached_pg, op: osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.040388 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2724 (0'0,173'2724] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] handle_message: 0x25e43700
2014-06-12 14:47:46.040407 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2724 (0'0,173'2724] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] sub_op_modify trans 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 v 173'2725 (transaction) 174
2014-06-12 14:47:46.040442 7f9d64bda700  5 -- op tracker -- , seq: 748474, time: 2014-06-12 14:47:46.040436, event: started, op: osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.040467 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2724 (0'0,173'2724] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] append_log log((0'0,173'2724], crt=0'0) [173'2725 (173'2723) modify   3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 by client.4284.0:141494 2014-06-12 08:45:20.902119]
2014-06-12 14:47:46.040505 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2725 (0'0,173'2725] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] add_log_entry 173'2725 (173'2723) modify   3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 by client.4284.0:141494 2014-06-12 08:45:20.902119
2014-06-12 14:47:46.040539 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2725 (0'0,173'2725] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] append_log  adding 1 keys
2014-06-12 14:47:46.040588 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2725, trimmed: 
2014-06-12 14:47:46.040624 7f9d64bda700  5 -- op tracker -- , seq: 748474, time: 2014-06-12 14:47:46.040617, event: started, op: osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.040640 7f9d64bda700  5 queue_transactions existing osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:46.040646 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:46.040648 7f9d64bda700  5 queue_op 0x287f5c0 seq 2534 osr(3.f9 0x276cfa0) 1550572 bytes   (queue has 1 ops and 1550572 bytes)
2014-06-12 14:47:46.040667 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b77a0 (1 active)
2014-06-12 14:47:46.040680 7f9d8016d700  5 _do_op 0x287f5c0 seq 2534 osr(3.f9 0x276cfa0)/0x276cfa0 start
2014-06-12 14:47:46.040657 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.f9( v 173'2725 (0'0,173'2725] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] do_sub_op osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.040687 7f9d64bda700 10 osd.2 173 dequeue_op 0x25e43700 finish
2014-06-12 14:47:46.040688 7f9d8016d700 10 _do_transaction on 0x2a29b430
2014-06-12 14:47:46.040693 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:46.040698 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.040700 7f9d8016d700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:46.040881 7f9d8016d700 10 lookup_strip_header done  cid meta oid 516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:46.040936 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:46.040986 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:46.041014 7f9d8016d700 15 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:46.041023 7f9d8016d700 10 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1 r = 0
2014-06-12 14:47:46.041029 7f9d8016d700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:46.041170 7f9d8016d700 15 _write 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 2342912~1548288
2014-06-12 14:47:46.041223 7f9d5a16f700  5 -- op tracker -- , seq: 748475, time: 2014-06-12 14:47:46.040260, event: header_read, op: osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.041253 7f9d5a16f700  5 -- op tracker -- , seq: 748475, time: 2014-06-12 14:47:46.040261, event: throttled, op: osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.041279 7f9d5a16f700  5 -- op tracker -- , seq: 748475, time: 2014-06-12 14:47:46.041168, event: all_read, op: osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.041289 7f9d5a16f700  5 -- op tracker -- , seq: 748475, time: 0.000000, event: dispatched, op: osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.041293 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:46.041299 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:46.041304 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903200 prio 127 cost 545575 latency 0.001044 osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.041335 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:46.041338 7f9d8016d700 10 lookup_strip_header done  cid 3.f9_head oid 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:46.041343 7f9d653db700 10 osd.2 173 dequeue_op 0x2a903200 prio 127 cost 545575 latency 0.001083 osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2683 (0'0,173'2683] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2682 active]
2014-06-12 14:47:46.041383 7f9d653db700  5 -- op tracker -- , seq: 748475, time: 2014-06-12 14:47:46.041378, event: reached_pg, op: osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.041391 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2683 (0'0,173'2683] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2682 active] handle_message: 0x2a903200
2014-06-12 14:47:46.041401 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2683 (0'0,173'2683] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2682 active] sub_op_modify trans c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 v 173'2684 (transaction) 174
2014-06-12 14:47:46.041411 7f9d653db700  5 -- op tracker -- , seq: 748475, time: 2014-06-12 14:47:46.041408, event: started, op: osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.041419 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2683 (0'0,173'2683] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2682 active] append_log log((0'0,173'2683], crt=0'0) [173'2684 (173'801) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141251 2014-06-12 08:45:21.217733]
2014-06-12 14:47:46.041433 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2684 (0'0,173'2684] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2682 active] add_log_entry 173'2684 (173'801) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141251 2014-06-12 08:45:21.217733
2014-06-12 14:47:46.041447 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2684 (0'0,173'2684] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2682 active] append_log  adding 1 keys
2014-06-12 14:47:46.041459 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:47:46.041466 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2684, trimmed: 
2014-06-12 14:47:46.041480 7f9d653db700  5 -- op tracker -- , seq: 748475, time: 2014-06-12 14:47:46.041477, event: started, op: osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.041485 7f9d653db700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:46.041487 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:46.041488 7f9d653db700  5 queue_op 0x1d207b60 seq 2494 osr(3.1b7 0x276d1a0) 547061 bytes   (queue has 2 ops and 2097633 bytes)
2014-06-12 14:47:46.041497 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (2 active)
2014-06-12 14:47:46.041493 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2684 (0'0,173'2684] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2682 active] do_sub_op osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.041502 7f9d653db700 10 osd.2 173 dequeue_op 0x2a903200 finish
2014-06-12 14:47:46.041503 7f9d79960700  5 _do_op 0x1d207b60 seq 2494 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:46.041504 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:46.041506 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.041506 7f9d79960700 10 _do_transaction on 0x1f2869b0
2014-06-12 14:47:46.041510 7f9d79960700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.041567 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.041587 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:46.049007 7f9d8016d700 10 _generic_write 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 2342912~1548288 = 0
2014-06-12 14:47:46.049234 7f9d8016d700 15 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:46.049266 7f9d8016d700 10 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 = 0
2014-06-12 14:47:46.049275 7f9d8016d700 15 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:46.049285 7f9d8016d700 10 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 = 0
2014-06-12 14:47:46.049806 7f9d8016d700 20 set_header setting 17380 cid 3.f9_head oid 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 parent seq 0
2014-06-12 14:47:46.049917 7f9d8016d700 20 set_header setting 2218 cid meta oid 516e9ea6/pglog_3.f9/0//-1 parent seq 0
2014-06-12 14:47:46.049929 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:46.049935 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:47:46.056127 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:46.056291 7f9d79960700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.056301 7f9d79960700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:46.056309 7f9d79960700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.056329 7f9d79960700 15 _write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 0~544768
2014-06-12 14:47:46.056487 7f9d79960700 10 lookup_strip_header done  cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:46.056658 7f9d79960700 10 file_to_extents done 
2014-06-12 14:47:46.056701 7f9d8016d700 10 _do_op 0x287f5c0 seq 2534 r = 0, finisher 0x1f24860 0
2014-06-12 14:47:46.056716 7f9d8016d700 10 _finish_op 0x287f5c0 seq 2534 osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:46.056728 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b77a0 (1 active)
2014-06-12 14:47:46.056735 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.056740 7f9d7715b700  5 -- op tracker -- , seq: 748474, time: 2014-06-12 14:47:46.056729, event: commit_sent, op: osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.056756 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2725 (0'0,173'2725] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2723 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:46.056786 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.056865 7f9d7795c700  5 -- op tracker -- , seq: 748474, time: 2014-06-12 14:47:46.056858, event: sub_op_applied, op: osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.056875 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2725 (0'0,173'2725] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] sub_op_modify_applied on 0x2a29b340 op osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.056893 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2725 (0'0,173'2725] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] op_applied on primary on version 173'2725
2014-06-12 14:47:46.056903 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2725 (0'0,173'2725] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] op_applied on replica on version 173'2725
2014-06-12 14:47:46.056924 7f9d7795c700  5 -- op tracker -- , seq: 748474, time: 2014-06-12 14:47:46.056920, event: done, op: osd_sub_op(client.4284.0:141494 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2725 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.058245 7f9d79960700 10 _generic_write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 0~544768 = 0
2014-06-12 14:47:46.058294 7f9d79960700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:46.058304 7f9d79960700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:46.058308 7f9d79960700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:46.058312 7f9d79960700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:46.058455 7f9d79960700 20 set_header setting 17923 cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 parent seq 0
2014-06-12 14:47:46.058507 7f9d79960700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:46.058518 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:46.058524 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:47:46.060366 7f9d79960700 10 _do_op 0x1d207b60 seq 2494 r = 0, finisher 0x7a8b200 0
2014-06-12 14:47:46.060382 7f9d79960700 10 _finish_op 0x1d207b60 seq 2494 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:46.060386 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:46.060398 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.060397 7f9d7715b700  5 -- op tracker -- , seq: 748475, time: 2014-06-12 14:47:46.060388, event: commit_sent, op: osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.060404 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2684 (0'0,173'2684] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2682 active] sub_op_modify_commit on op osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:46.060432 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.060476 7f9d7795c700  5 -- op tracker -- , seq: 748475, time: 2014-06-12 14:47:46.060470, event: sub_op_applied, op: osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.060488 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2684 (0'0,173'2684] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active] sub_op_modify_applied on 0x1f2868c0 op osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.060507 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2684 (0'0,173'2684] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active] op_applied on primary on version 173'2684
2014-06-12 14:47:46.060514 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2684 (0'0,173'2684] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active] op_applied on replica on version 173'2684
2014-06-12 14:47:46.060528 7f9d7795c700  5 -- op tracker -- , seq: 748475, time: 2014-06-12 14:47:46.060525, event: done, op: osd_sub_op(client.4287.0:141251 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2684 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.139806 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.147557 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.154888 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.157025 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.157240 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:46.190207 7f9d5a16f700  5 -- op tracker -- , seq: 748476, time: 2014-06-12 14:47:46.188115, event: header_read, op: osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.190218 7f9d5a16f700  5 -- op tracker -- , seq: 748476, time: 2014-06-12 14:47:46.188117, event: throttled, op: osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.190234 7f9d5a16f700  5 -- op tracker -- , seq: 748476, time: 2014-06-12 14:47:46.190171, event: all_read, op: osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.190239 7f9d5a16f700  5 -- op tracker -- , seq: 748476, time: 0.000000, event: dispatched, op: osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.190243 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:46.190248 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:46.190254 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903f00 prio 127 cost 1372963 latency 0.002139 osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.190277 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:46.190284 7f9d68be2700 10 osd.2 173 dequeue_op 0x2a903f00 prio 127 cost 1372963 latency 0.002168 osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.f9( v 173'2725 (0'0,173'2725] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active]
2014-06-12 14:47:46.190300 7f9d68be2700  5 -- op tracker -- , seq: 748476, time: 2014-06-12 14:47:46.190298, event: reached_pg, op: osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.190314 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2725 (0'0,173'2725] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] handle_message: 0x2a903f00
2014-06-12 14:47:46.190322 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2725 (0'0,173'2725] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] sub_op_modify trans 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 v 173'2726 (transaction) 174
2014-06-12 14:47:46.190333 7f9d68be2700  5 -- op tracker -- , seq: 748476, time: 2014-06-12 14:47:46.190330, event: started, op: osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.190345 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2725 (0'0,173'2725] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] append_log log((0'0,173'2725], crt=0'0) [173'2726 (173'2724) modify   183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 by client.4288.0:139270 2014-06-12 08:44:23.712093]
2014-06-12 14:47:46.190365 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2726 (0'0,173'2726] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] add_log_entry 173'2726 (173'2724) modify   183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 by client.4288.0:139270 2014-06-12 08:44:23.712093
2014-06-12 14:47:46.190380 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2726 (0'0,173'2726] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] append_log  adding 1 keys
2014-06-12 14:47:46.190406 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2726, trimmed: 
2014-06-12 14:47:46.190423 7f9d68be2700  5 -- op tracker -- , seq: 748476, time: 2014-06-12 14:47:46.190420, event: started, op: osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.190432 7f9d68be2700  5 queue_transactions existing osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:46.190436 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:46.190437 7f9d68be2700  5 queue_op 0x6cf7500 seq 2535 osr(3.f9 0x276cfa0) 1374444 bytes   (queue has 1 ops and 1374444 bytes)
2014-06-12 14:47:46.190442 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.f9( v 173'2726 (0'0,173'2726] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] do_sub_op osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.190452 7f9d68be2700 10 osd.2 173 dequeue_op 0x2a903f00 finish
2014-06-12 14:47:46.190462 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:46.190463 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.190474 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b77a0 (1 active)
2014-06-12 14:47:46.190482 7f9d7c966700  5 _do_op 0x6cf7500 seq 2535 osr(3.f9 0x276cfa0)/0x276cfa0 start
2014-06-12 14:47:46.190487 7f9d7c966700 10 _do_transaction on 0x5b24630
2014-06-12 14:47:46.190492 7f9d7c966700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:46.190574 7f9d7c966700 10 lookup_strip_header done  cid meta oid 516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:46.190598 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:46.190623 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:46.190642 7f9d7c966700 15 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:46.190647 7f9d7c966700 10 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1 r = 0
2014-06-12 14:47:46.190651 7f9d7c966700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:46.190663 7f9d7c966700 15 _write 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 2822144~1372160
2014-06-12 14:47:46.190750 7f9d7c966700 10 lookup_strip_header done  cid 3.f9_head oid 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3
2014-06-12 14:47:46.190929 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:47:46.194635 7f9d7c966700 10 _generic_write 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 2822144~1372160 = 0
2014-06-12 14:47:46.194769 7f9d7c966700 15 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3
2014-06-12 14:47:46.194788 7f9d7c966700 10 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 = 0
2014-06-12 14:47:46.194792 7f9d7c966700 15 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3
2014-06-12 14:47:46.194797 7f9d7c966700 10 _setattrs 3.f9_head/183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 = 0
2014-06-12 14:47:46.195051 7f9d7c966700 20 set_header setting 16861 cid 3.f9_head oid 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 parent seq 0
2014-06-12 14:47:46.195188 7f9d7c966700 20 set_header setting 2218 cid meta oid 516e9ea6/pglog_3.f9/0//-1 parent seq 0
2014-06-12 14:47:46.195206 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:46.195220 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:47:46.202142 7f9d7c966700 10 _do_op 0x6cf7500 seq 2535 r = 0, finisher 0x6d63900 0
2014-06-12 14:47:46.202161 7f9d7c966700 10 _finish_op 0x6cf7500 seq 2535 osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:46.202168 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b77a0 (0 active)
2014-06-12 14:47:46.202174 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.202177 7f9d7715b700  5 -- op tracker -- , seq: 748476, time: 2014-06-12 14:47:46.202169, event: commit_sent, op: osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.202188 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2726 (0'0,173'2726] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2724 active] sub_op_modify_commit on op osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:46.202206 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.202253 7f9d7795c700  5 -- op tracker -- , seq: 748476, time: 2014-06-12 14:47:46.202245, event: sub_op_applied, op: osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.202266 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2726 (0'0,173'2726] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] sub_op_modify_applied on 0x5b24540 op osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.202285 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2726 (0'0,173'2726] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] op_applied on primary on version 173'2726
2014-06-12 14:47:46.202292 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2726 (0'0,173'2726] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] op_applied on replica on version 173'2726
2014-06-12 14:47:46.202307 7f9d7795c700  5 -- op tracker -- , seq: 748476, time: 2014-06-12 14:47:46.202304, event: done, op: osd_sub_op(client.4288.0:139270 3.f9 183828f9/rbd_data.10976b8b4567.0000000000000ee8/head//3 [] v 173'2726 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.208579 7f9d5a16f700  5 -- op tracker -- , seq: 748477, time: 2014-06-12 14:47:46.208517, event: header_read, op: osd_sub_op_reply(client.4316.0:147323 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.208596 7f9d5a16f700  5 -- op tracker -- , seq: 748477, time: 2014-06-12 14:47:46.208520, event: throttled, op: osd_sub_op_reply(client.4316.0:147323 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.208624 7f9d5a16f700  5 -- op tracker -- , seq: 748477, time: 2014-06-12 14:47:46.208538, event: all_read, op: osd_sub_op_reply(client.4316.0:147323 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.208635 7f9d5a16f700  5 -- op tracker -- , seq: 748477, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147323 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.208641 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147323 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:46.208649 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:46.208657 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329ea100 prio 196 cost 0 latency 0.000140 osd_sub_op_reply(client.4316.0:147323 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:46.208727 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:46.208750 7f9d643d9700 10 osd.2 173 dequeue_op 0x329ea100 prio 196 cost 0 latency 0.000232 osd_sub_op_reply(client.4316.0:147323 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 pg pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2979 active+clean]
2014-06-12 14:47:46.208788 7f9d643d9700  5 -- op tracker -- , seq: 748477, time: 2014-06-12 14:47:46.208782, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147323 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.208809 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2979 active+clean] handle_message: 0x329ea100
2014-06-12 14:47:46.208832 7f9d643d9700  5 -- op tracker -- , seq: 748477, time: 2014-06-12 14:47:46.208827, event: started, op: osd_sub_op_reply(client.4316.0:147323 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.208841 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2979 active+clean] sub_op_modify_reply: tid 236411 op  ack_type 4 from 24
2014-06-12 14:47:46.208875 7f9d643d9700  5 -- op tracker -- , seq: 748355, time: 2014-06-12 14:47:46.208863, event: sub_op_commit_rec, op: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:46.208887 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2979 active+clean] repop_all_applied: repop tid 236411 all applied 
2014-06-12 14:47:46.208902 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2979 active+clean] eval_repop repgather(0x2335d080 173'2981 rep_tid=236411 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:46.208925 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2979 active+clean]  sending ack on repgather(0x2335d080 173'2981 rep_tid=236411 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x890a000
2014-06-12 14:47:46.208971 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2980 crt=0'0 lcod 173'2979 mlcod 173'2979 active+clean] repop_all_committed: repop tid 236411 all committed 
2014-06-12 14:47:46.208987 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2979 active+clean] eval_repop repgather(0x2335d080 173'2981 rep_tid=236411 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:46.209017 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2979 active+clean] log_op_stats osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 inb 516375 outb 0 rlat 33.556904 lat 33.556951
2014-06-12 14:47:46.209041 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2979 active+clean] publish_stats_to_osd 173:3177
2014-06-12 14:47:46.209057 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2979 active+clean]  sending commit on repgather(0x2335d080 173'2981 rep_tid=236411 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x912a780
2014-06-12 14:47:46.209106 7f9d643d9700  5 -- op tracker -- , seq: 748355, time: 2014-06-12 14:47:46.209094, event: commit_sent, op: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:46.209119 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2979 active+clean]  requeue_ops 
2014-06-12 14:47:46.209134 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2980 active+clean]  removing repgather(0x2335d080 173'2981 rep_tid=236411 committed?=1 applied?=1 op=osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:46.209154 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2980 active+clean]    q front is repgather(0x2335d080 173'2981 rep_tid=236411 committed?=1 applied?=1 op=osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:46.209176 7f9d643d9700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2980 active+clean] remove_repop repgather(0x2335d080 173'2981 rep_tid=236411 committed?=1 applied?=1 op=osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:46.209197 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2980 active+clean]  requeue_ops 
2014-06-12 14:47:46.209222 7f9d643d9700  5 -- op tracker -- , seq: 748355, time: 2014-06-12 14:47:46.209215, event: done, op: osd_op(client.4316.0:147323 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3354624~516096] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:46.209244 7f9d643d9700 10 osd.2 173 dequeue_op 0x329ea100 finish
2014-06-12 14:47:46.209271 7f9d643d9700  5 -- op tracker -- , seq: 748477, time: 2014-06-12 14:47:46.209259, event: done, op: osd_sub_op_reply(client.4316.0:147323 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.209287 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:46.209291 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.266336 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:46.266388 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:46.339441 7f9d5a16f700  5 -- op tracker -- , seq: 748478, time: 2014-06-12 14:47:46.339383, event: header_read, op: osd_sub_op_reply(client.4316.0:147324 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.339459 7f9d5a16f700  5 -- op tracker -- , seq: 748478, time: 2014-06-12 14:47:46.339386, event: throttled, op: osd_sub_op_reply(client.4316.0:147324 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.339472 7f9d5a16f700  5 -- op tracker -- , seq: 748478, time: 2014-06-12 14:47:46.339400, event: all_read, op: osd_sub_op_reply(client.4316.0:147324 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.339481 7f9d5a16f700  5 -- op tracker -- , seq: 748478, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4316.0:147324 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.339486 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(client.4316.0:147324 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 epoch 173
2014-06-12 14:47:46.339490 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:46.339496 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329eb000 prio 196 cost 0 latency 0.000113 osd_sub_op_reply(client.4316.0:147324 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2
2014-06-12 14:47:46.339563 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:46.339589 7f9d63bd8700 10 osd.2 173 dequeue_op 0x329eb000 prio 196 cost 0 latency 0.000206 osd_sub_op_reply(client.4316.0:147324 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0) v2 pg pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2980 active+clean]
2014-06-12 14:47:46.339608 7f9d63bd8700  5 -- op tracker -- , seq: 748478, time: 2014-06-12 14:47:46.339606, event: reached_pg, op: osd_sub_op_reply(client.4316.0:147324 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.339617 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2980 active+clean] handle_message: 0x329eb000
2014-06-12 14:47:46.339628 7f9d63bd8700  5 -- op tracker -- , seq: 748478, time: 2014-06-12 14:47:46.339626, event: started, op: osd_sub_op_reply(client.4316.0:147324 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.339632 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2980 active+clean] sub_op_modify_reply: tid 236412 op  ack_type 4 from 24
2014-06-12 14:47:46.339651 7f9d63bd8700  5 -- op tracker -- , seq: 748357, time: 2014-06-12 14:47:46.339645, event: sub_op_commit_rec, op: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:46.339656 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2980 active+clean] repop_all_applied: repop tid 236412 all applied 
2014-06-12 14:47:46.339663 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2980 active+clean] eval_repop repgather(0x51dc000 173'2982 rep_tid=236412 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:46.339677 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2980 active+clean]  sending ack on repgather(0x51dc000 173'2982 rep_tid=236412 committed?=0 applied?=1 lock=0 op=osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x5ab0000
2014-06-12 14:47:46.339700 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 luod=173'2981 crt=0'0 lcod 173'2980 mlcod 173'2980 active+clean] repop_all_committed: repop tid 236412 all committed 
2014-06-12 14:47:46.339707 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2980 active+clean] eval_repop repgather(0x51dc000 173'2982 rep_tid=236412 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) wants=ad
2014-06-12 14:47:46.339725 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2980 active+clean] log_op_stats osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4 inb 323863 outb 0 rlat 33.657852 lat 33.657870
2014-06-12 14:47:46.339737 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2980 active+clean] publish_stats_to_osd 173:3178
2014-06-12 14:47:46.339746 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2980 active+clean]  sending commit on repgather(0x51dc000 173'2982 rep_tid=236412 committed?=1 applied?=1 lock=0 op=osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4) 0x912b680
2014-06-12 14:47:46.339768 7f9d63bd8700  5 -- op tracker -- , seq: 748357, time: 2014-06-12 14:47:46.339761, event: commit_sent, op: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:46.339773 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2980 active+clean]  requeue_ops 
2014-06-12 14:47:46.339779 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean]  removing repgather(0x51dc000 173'2982 rep_tid=236412 committed?=1 applied?=1 op=osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:46.339795 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean]    q front is repgather(0x51dc000 173'2982 rep_tid=236412 committed?=1 applied?=1 op=osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:46.339804 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean] remove_repop repgather(0x51dc000 173'2982 rep_tid=236412 committed?=1 applied?=1 op=osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173) v4)
2014-06-12 14:47:46.339812 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean]  requeue_ops 
2014-06-12 14:47:46.339830 7f9d63bd8700  5 -- op tracker -- , seq: 748357, time: 2014-06-12 14:47:46.339827, event: done, op: osd_op(client.4316.0:147324 rbd_data.10b36b8b4567.000000000000113e [set-alloc-hint object_size 4194304 write_size 4194304,write 3870720~323584] 3.58ccef85 ack+ondisk+write+known_if_redirected e173)
2014-06-12 14:47:46.339846 7f9d63bd8700 10 osd.2 173 dequeue_op 0x329eb000 finish
2014-06-12 14:47:46.339850 7f9d63bd8700  5 -- op tracker -- , seq: 748478, time: 2014-06-12 14:47:46.339848, event: done, op: osd_sub_op_reply(client.4316.0:147324 3.785 58ccef85/rbd_data.10b36b8b4567.000000000000113e/head//3 [] ondisk, result = 0)
2014-06-12 14:47:46.339859 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:46.339861 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.360413 7f9d5a16f700  5 -- op tracker -- , seq: 748479, time: 2014-06-12 14:47:46.359205, event: header_read, op: osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.360445 7f9d5a16f700  5 -- op tracker -- , seq: 748479, time: 2014-06-12 14:47:46.359209, event: throttled, op: osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.360467 7f9d5a16f700  5 -- op tracker -- , seq: 748479, time: 2014-06-12 14:47:46.360335, event: all_read, op: osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.360484 7f9d5a16f700  5 -- op tracker -- , seq: 748479, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.360491 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:46.360503 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:46.360518 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329eae00 prio 127 cost 516903 latency 0.001313 osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.360567 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:46.360582 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329eae00 prio 127 cost 516903 latency 0.001377 osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2684 (0'0,173'2684] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active]
2014-06-12 14:47:46.360626 7f9d61bd4700  5 -- op tracker -- , seq: 748479, time: 2014-06-12 14:47:46.360620, event: reached_pg, op: osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.360640 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2684 (0'0,173'2684] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active] handle_message: 0x329eae00
2014-06-12 14:47:46.360659 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2684 (0'0,173'2684] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active] sub_op_modify trans 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 v 173'2685 (transaction) 174
2014-06-12 14:47:46.360668 7f9d61bd4700  5 -- op tracker -- , seq: 748479, time: 2014-06-12 14:47:46.360666, event: started, op: osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.360682 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2684 (0'0,173'2684] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active] append_log log((0'0,173'2684], crt=0'0) [173'2685 (173'2683) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142259 2014-06-12 08:44:04.470817]
2014-06-12 14:47:46.360699 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2685 (0'0,173'2685] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active] add_log_entry 173'2685 (173'2683) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142259 2014-06-12 08:44:04.470817
2014-06-12 14:47:46.360713 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2685 (0'0,173'2685] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active] append_log  adding 1 keys
2014-06-12 14:47:46.360733 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2685, trimmed: 
2014-06-12 14:47:46.360755 7f9d61bd4700  5 -- op tracker -- , seq: 748479, time: 2014-06-12 14:47:46.360752, event: started, op: osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.360770 7f9d61bd4700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:46.360781 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:46.360782 7f9d61bd4700  5 queue_op 0x8e9aae0 seq 2495 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:46.360790 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:46.360787 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2685 (0'0,173'2685] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active] do_sub_op osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.360796 7f9d61bd4700 10 osd.2 173 dequeue_op 0x329eae00 finish
2014-06-12 14:47:46.360798 7f9d81970700  5 _do_op 0x8e9aae0 seq 2495 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:46.360799 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:46.360800 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.360801 7f9d81970700 10 _do_transaction on 0x6676b70
2014-06-12 14:47:46.360806 7f9d81970700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.360892 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.360924 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:46.360950 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:46.360962 7f9d81970700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.360967 7f9d81970700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:46.360971 7f9d81970700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.360979 7f9d81970700 15 _write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 1802240~516096
2014-06-12 14:47:46.361076 7f9d81970700 10 lookup_strip_header done  cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:46.361091 7f9d81970700 10 file_to_extents done 
2014-06-12 14:47:46.362476 7f9d81970700 10 _generic_write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 1802240~516096 = 0
2014-06-12 14:47:46.362520 7f9d81970700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:46.362530 7f9d81970700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:46.362534 7f9d81970700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:46.362538 7f9d81970700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:46.362691 7f9d81970700 20 set_header setting 16305 cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:47:46.362733 7f9d81970700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:46.362743 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:46.362748 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:47:46.364695 7f9d81970700 10 _do_op 0x8e9aae0 seq 2495 r = 0, finisher 0x6607780 0
2014-06-12 14:47:46.364705 7f9d81970700 10 _finish_op 0x8e9aae0 seq 2495 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:46.364712 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:46.364717 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.364744 7f9d7795c700  5 -- op tracker -- , seq: 748479, time: 2014-06-12 14:47:46.364732, event: sub_op_applied, op: osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.364762 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2685 (0'0,173'2685] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active] sub_op_modify_applied on 0x6676a80 op osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.364791 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.364813 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2685 (0'0,173'2685] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active] op_applied on primary on version 173'2685
2014-06-12 14:47:46.364825 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2685 (0'0,173'2685] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active] op_applied on replica on version 173'2685
2014-06-12 14:47:46.364858 7f9d7715b700  5 -- op tracker -- , seq: 748479, time: 2014-06-12 14:47:46.364851, event: commit_sent, op: osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.364866 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2685 (0'0,173'2685] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2683 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:46.364881 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.364908 7f9d7715b700  5 -- op tracker -- , seq: 748479, time: 2014-06-12 14:47:46.364897, event: done, op: osd_sub_op(client.4305.0:142259 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2685 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.371876 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:46.487399 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:46.487431 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:46.487436 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:46.487438 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:46.487476 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:46.510300 7f9d5a16f700  5 -- op tracker -- , seq: 748480, time: 2014-06-12 14:47:46.509450, event: header_read, op: osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.510330 7f9d5a16f700  5 -- op tracker -- , seq: 748480, time: 2014-06-12 14:47:46.509452, event: throttled, op: osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.510344 7f9d5a16f700  5 -- op tracker -- , seq: 748480, time: 2014-06-12 14:47:46.510258, event: all_read, op: osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.510353 7f9d5a16f700  5 -- op tracker -- , seq: 748480, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.510359 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:46.510366 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:46.510375 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329eb800 prio 127 cost 516903 latency 0.000925 osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.510439 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:46.510450 7f9d633d7700 10 osd.2 173 dequeue_op 0x329eb800 prio 127 cost 516903 latency 0.001000 osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2882 (0'0,173'2882] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active]
2014-06-12 14:47:46.510478 7f9d633d7700  5 -- op tracker -- , seq: 748480, time: 2014-06-12 14:47:46.510475, event: reached_pg, op: osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.510489 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2882 (0'0,173'2882] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active] handle_message: 0x329eb800
2014-06-12 14:47:46.510498 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2882 (0'0,173'2882] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active] sub_op_modify trans a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 v 173'2883 (transaction) 174
2014-06-12 14:47:46.510509 7f9d633d7700  5 -- op tracker -- , seq: 748480, time: 2014-06-12 14:47:46.510506, event: started, op: osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.510521 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2882 (0'0,173'2882] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active] append_log log((0'0,173'2882], crt=0'0) [173'2883 (173'2882) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146507 2014-06-12 08:45:37.651475]
2014-06-12 14:47:46.510544 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2883 (0'0,173'2883] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active] add_log_entry 173'2883 (173'2882) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146507 2014-06-12 08:45:37.651475
2014-06-12 14:47:46.510561 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2883 (0'0,173'2883] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active] append_log  adding 1 keys
2014-06-12 14:47:46.510594 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2883, trimmed: 
2014-06-12 14:47:46.510621 7f9d633d7700  5 -- op tracker -- , seq: 748480, time: 2014-06-12 14:47:46.510617, event: started, op: osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.510629 7f9d633d7700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:46.510632 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:46.510633 7f9d633d7700  5 queue_op 0x74184e0 seq 2596 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:46.510642 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:46.510638 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2883 (0'0,173'2883] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active] do_sub_op osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.510647 7f9d633d7700 10 osd.2 173 dequeue_op 0x329eb800 finish
2014-06-12 14:47:46.510650 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:46.510649 7f9d7a962700  5 _do_op 0x74184e0 seq 2596 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:46.510651 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.510654 7f9d7a962700 10 _do_transaction on 0x73d7cf0
2014-06-12 14:47:46.510661 7f9d7a962700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:46.510776 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:46.510807 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:46.510847 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:46.510873 7f9d7a962700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:46.510879 7f9d7a962700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:46.510885 7f9d7a962700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:46.510998 7f9d7a962700 15 _write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 851968~516096
2014-06-12 14:47:46.511139 7f9d7a962700 10 lookup_strip_header done  cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:46.511264 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:47:46.511943 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:46.512074 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:46.513178 7f9d7a962700 10 _generic_write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 851968~516096 = 0
2014-06-12 14:47:46.513234 7f9d7a962700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:46.513252 7f9d7a962700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:46.513259 7f9d7a962700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:46.513268 7f9d7a962700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:46.513474 7f9d7a962700 20 set_header setting 17398 cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 parent seq 0
2014-06-12 14:47:46.513545 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:46.513561 7f9d7a962700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:46.513573 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:47:46.516247 7f9d7a962700 10 _do_op 0x74184e0 seq 2596 r = 0, finisher 0x69e7b20 0
2014-06-12 14:47:46.516264 7f9d7a962700 10 _finish_op 0x74184e0 seq 2596 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:46.516271 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:46.516286 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.516323 7f9d7795c700  5 -- op tracker -- , seq: 748480, time: 2014-06-12 14:47:46.516303, event: sub_op_applied, op: osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.516334 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2883 (0'0,173'2883] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active] sub_op_modify_applied on 0x73d7c00 op osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.516364 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.516385 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2883 (0'0,173'2883] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active] op_applied on primary on version 173'2883
2014-06-12 14:47:46.516406 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2883 (0'0,173'2883] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active] op_applied on replica on version 173'2883
2014-06-12 14:47:46.516432 7f9d7715b700  5 -- op tracker -- , seq: 748480, time: 2014-06-12 14:47:46.516425, event: commit_sent, op: osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.516454 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2883 (0'0,173'2883] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2881 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:46.516481 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.516509 7f9d7715b700  5 -- op tracker -- , seq: 748480, time: 2014-06-12 14:47:46.516504, event: done, op: osd_sub_op(client.4313.0:146507 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2883 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.543670 7f9d5a16f700  5 -- op tracker -- , seq: 748481, time: 2014-06-12 14:47:46.543097, event: header_read, op: osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.543699 7f9d5a16f700  5 -- op tracker -- , seq: 748481, time: 2014-06-12 14:47:46.543100, event: throttled, op: osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.543733 7f9d5a16f700  5 -- op tracker -- , seq: 748481, time: 2014-06-12 14:47:46.543614, event: all_read, op: osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.543749 7f9d5a16f700  5 -- op tracker -- , seq: 748481, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.543753 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:46.543758 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:46.543765 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329eac00 prio 127 cost 303907 latency 0.000668 osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.543792 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:46.543803 7f9d67be0700 10 osd.2 173 dequeue_op 0x329eac00 prio 127 cost 303907 latency 0.000705 osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.f9( v 173'2726 (0'0,173'2726] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active]
2014-06-12 14:47:46.543847 7f9d67be0700  5 -- op tracker -- , seq: 748481, time: 2014-06-12 14:47:46.543837, event: reached_pg, op: osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.543868 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2726 (0'0,173'2726] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] handle_message: 0x329eac00
2014-06-12 14:47:46.543909 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2726 (0'0,173'2726] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] sub_op_modify trans 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 v 173'2727 (transaction) 174
2014-06-12 14:47:46.543951 7f9d67be0700  5 -- op tracker -- , seq: 748481, time: 2014-06-12 14:47:46.543941, event: started, op: osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.543990 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2726 (0'0,173'2726] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] append_log log((0'0,173'2726], crt=0'0) [173'2727 (173'2725) modify   3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 by client.4284.0:141495 2014-06-12 08:45:20.906369]
2014-06-12 14:47:46.544016 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2727 (0'0,173'2727] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] add_log_entry 173'2727 (173'2725) modify   3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 by client.4284.0:141495 2014-06-12 08:45:20.906369
2014-06-12 14:47:46.544038 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2727 (0'0,173'2727] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] append_log  adding 1 keys
2014-06-12 14:47:46.544108 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2727, trimmed: 
2014-06-12 14:47:46.544162 7f9d67be0700  5 -- op tracker -- , seq: 748481, time: 2014-06-12 14:47:46.544151, event: started, op: osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.544188 7f9d67be0700  5 queue_transactions existing osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:46.544195 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:46.544198 7f9d67be0700  5 queue_op 0x213fb0e0 seq 2536 osr(3.f9 0x276cfa0) 305388 bytes   (queue has 1 ops and 305388 bytes)
2014-06-12 14:47:46.544216 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.f9( v 173'2727 (0'0,173'2727] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] do_sub_op osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.544231 7f9d67be0700 10 osd.2 173 dequeue_op 0x329eac00 finish
2014-06-12 14:47:46.544235 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:46.544238 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.544260 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b77a0 (1 active)
2014-06-12 14:47:46.544267 7f9d82972700  5 _do_op 0x213fb0e0 seq 2536 osr(3.f9 0x276cfa0)/0x276cfa0 start
2014-06-12 14:47:46.544271 7f9d82972700 10 _do_transaction on 0x31d087f0
2014-06-12 14:47:46.544277 7f9d82972700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:46.544466 7f9d82972700 10 lookup_strip_header done  cid meta oid 516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:46.544498 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:46.544564 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:46.544596 7f9d82972700 15 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:46.544604 7f9d82972700 10 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1 r = 0
2014-06-12 14:47:46.544609 7f9d82972700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:46.544627 7f9d82972700 15 _write 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 3891200~303104
2014-06-12 14:47:46.544763 7f9d82972700 10 lookup_strip_header done  cid 3.f9_head oid 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:46.544986 7f9d5a16f700  5 -- op tracker -- , seq: 748482, time: 2014-06-12 14:47:46.544017, event: header_read, op: osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.544998 7f9d5a16f700  5 -- op tracker -- , seq: 748482, time: 2014-06-12 14:47:46.544018, event: throttled, op: osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.545017 7f9d5a16f700  5 -- op tracker -- , seq: 748482, time: 2014-06-12 14:47:46.544951, event: all_read, op: osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.545024 7f9d5a16f700  5 -- op tracker -- , seq: 748482, time: 0.000000, event: dispatched, op: osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.545028 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:46.545034 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:46.545040 7f9d5a16f700 15 osd.2 173 enqueue_op 0x68a6c00 prio 127 cost 516903 latency 0.001022 osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.545053 7f9d82972700 10 file_to_extents done 
2014-06-12 14:47:46.545065 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:46.545070 7f9d673df700 10 osd.2 173 dequeue_op 0x68a6c00 prio 127 cost 516903 latency 0.001053 osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2685 (0'0,173'2685] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2684 active]
2014-06-12 14:47:46.545092 7f9d673df700  5 -- op tracker -- , seq: 748482, time: 2014-06-12 14:47:46.545088, event: reached_pg, op: osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.545099 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2685 (0'0,173'2685] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2684 active] handle_message: 0x68a6c00
2014-06-12 14:47:46.545110 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2685 (0'0,173'2685] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2684 active] sub_op_modify trans c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 v 173'2686 (transaction) 174
2014-06-12 14:47:46.545125 7f9d673df700  5 -- op tracker -- , seq: 748482, time: 2014-06-12 14:47:46.545121, event: started, op: osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.545135 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2685 (0'0,173'2685] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2684 active] append_log log((0'0,173'2685], crt=0'0) [173'2686 (173'2684) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141252 2014-06-12 08:45:21.246335]
2014-06-12 14:47:46.545160 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2686 (0'0,173'2686] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2684 active] add_log_entry 173'2686 (173'2684) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141252 2014-06-12 08:45:21.246335
2014-06-12 14:47:46.545184 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2686 (0'0,173'2686] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2684 active] append_log  adding 1 keys
2014-06-12 14:47:46.545213 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2686, trimmed: 
2014-06-12 14:47:46.545234 7f9d673df700  5 -- op tracker -- , seq: 748482, time: 2014-06-12 14:47:46.545229, event: started, op: osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.545243 7f9d673df700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:46.545247 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:46.545248 7f9d673df700  5 queue_op 0x8102480 seq 2496 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 2 ops and 823777 bytes)
2014-06-12 14:47:46.545254 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2686 (0'0,173'2686] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2684 active] do_sub_op osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.545268 7f9d673df700 10 osd.2 173 dequeue_op 0x68a6c00 finish
2014-06-12 14:47:46.545271 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:46.545273 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.545282 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (2 active)
2014-06-12 14:47:46.545286 7f9d7895e700  5 _do_op 0x8102480 seq 2496 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:46.545290 7f9d7895e700 10 _do_transaction on 0x580eb70
2014-06-12 14:47:46.545294 7f9d7895e700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.545363 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.545387 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:46.546044 7f9d82972700 10 _generic_write 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 3891200~303104 = 0
2014-06-12 14:47:46.546086 7f9d82972700 15 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:46.546101 7f9d82972700 10 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 = 0
2014-06-12 14:47:46.546108 7f9d82972700 15 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3
2014-06-12 14:47:46.546115 7f9d82972700 10 _setattrs 3.f9_head/3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 = 0
2014-06-12 14:47:46.546275 7f9d82972700 20 set_header setting 17380 cid 3.f9_head oid 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 parent seq 0
2014-06-12 14:47:46.546325 7f9d82972700 20 set_header setting 2218 cid meta oid 516e9ea6/pglog_3.f9/0//-1 parent seq 0
2014-06-12 14:47:46.546341 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:46.546349 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:47:46.547967 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:46.548086 7f9d82972700 10 _do_op 0x213fb0e0 seq 2536 r = 0, finisher 0x1ed3c720 0
2014-06-12 14:47:46.548099 7f9d82972700 10 _finish_op 0x213fb0e0 seq 2536 osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:46.548102 7f9d7895e700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.548106 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b77a0 (1 active)
2014-06-12 14:47:46.548109 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.548109 7f9d7895e700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:46.548117 7f9d7895e700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.548117 7f9d7715b700  5 -- op tracker -- , seq: 748481, time: 2014-06-12 14:47:46.548107, event: commit_sent, op: osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.548138 7f9d7895e700 15 _write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 544768~516096
2014-06-12 14:47:46.548126 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2727 (0'0,173'2727] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:46.548149 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.548188 7f9d7795c700  5 -- op tracker -- , seq: 748481, time: 2014-06-12 14:47:46.548183, event: sub_op_applied, op: osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.548195 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2727 (0'0,173'2727] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] sub_op_modify_applied on 0x31d08700 op osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.548209 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2727 (0'0,173'2727] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] op_applied on primary on version 173'2727
2014-06-12 14:47:46.548218 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2727 (0'0,173'2727] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] op_applied on replica on version 173'2727
2014-06-12 14:47:46.548237 7f9d7795c700  5 -- op tracker -- , seq: 748481, time: 2014-06-12 14:47:46.548233, event: done, op: osd_sub_op(client.4284.0:141495 3.f9 3bfeb8f9/rbd_data.10936b8b4567.0000000000000fe1/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.548276 7f9d7895e700 10 lookup_strip_header done  cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:46.548381 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:47:46.550105 7f9d7895e700 10 _generic_write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 544768~516096 = 0
2014-06-12 14:47:46.550174 7f9d7895e700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:46.550193 7f9d7895e700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:46.550200 7f9d7895e700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:46.550208 7f9d7895e700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:46.550423 7f9d7895e700 20 set_header setting 17923 cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 parent seq 0
2014-06-12 14:47:46.550490 7f9d7895e700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:46.550506 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:46.550516 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:47:46.555996 7f9d7895e700 10 _do_op 0x8102480 seq 2496 r = 0, finisher 0x66b53a0 0
2014-06-12 14:47:46.556013 7f9d7895e700 10 _finish_op 0x8102480 seq 2496 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:46.556020 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:46.556034 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.556044 7f9d7795c700  5 -- op tracker -- , seq: 748482, time: 2014-06-12 14:47:46.556035, event: sub_op_applied, op: osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.556051 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2686 (0'0,173'2686] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2684 active] sub_op_modify_applied on 0x580ea80 op osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.556081 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.556098 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2686 (0'0,173'2686] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2684 active] op_applied on primary on version 173'2686
2014-06-12 14:47:46.556116 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2686 (0'0,173'2686] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2684 active] op_applied on replica on version 173'2686
2014-06-12 14:47:46.556142 7f9d7715b700  5 -- op tracker -- , seq: 748482, time: 2014-06-12 14:47:46.556134, event: commit_sent, op: osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.556159 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2686 (0'0,173'2686] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2684 active] sub_op_modify_commit on op osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:46.556179 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.556208 7f9d7715b700  5 -- op tracker -- , seq: 748482, time: 2014-06-12 14:47:46.556202, event: done, op: osd_sub_op(client.4287.0:141252 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2686 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.727856 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:46.727919 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:46.849948 7f9d5a16f700  5 -- op tracker -- , seq: 748483, time: 2014-06-12 14:47:46.848697, event: header_read, op: osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.849975 7f9d5a16f700  5 -- op tracker -- , seq: 748483, time: 2014-06-12 14:47:46.848699, event: throttled, op: osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.850005 7f9d5a16f700  5 -- op tracker -- , seq: 748483, time: 2014-06-12 14:47:46.849902, event: all_read, op: osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.850014 7f9d5a16f700  5 -- op tracker -- , seq: 748483, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.850022 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:46.850042 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:46.850055 7f9d5a16f700 15 osd.2 173 enqueue_op 0x68a7400 prio 127 cost 516903 latency 0.001358 osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.850092 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:46.850101 7f9d663dd700 10 osd.2 173 dequeue_op 0x68a7400 prio 127 cost 516903 latency 0.001403 osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2686 (0'0,173'2686] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2685 active]
2014-06-12 14:47:46.850124 7f9d663dd700  5 -- op tracker -- , seq: 748483, time: 2014-06-12 14:47:46.850120, event: reached_pg, op: osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.850133 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2686 (0'0,173'2686] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2685 active] handle_message: 0x68a7400
2014-06-12 14:47:46.850144 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2686 (0'0,173'2686] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2685 active] sub_op_modify trans 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 v 173'2687 (transaction) 174
2014-06-12 14:47:46.850160 7f9d663dd700  5 -- op tracker -- , seq: 748483, time: 2014-06-12 14:47:46.850157, event: started, op: osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.850175 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2686 (0'0,173'2686] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2685 active] append_log log((0'0,173'2686], crt=0'0) [173'2687 (173'2685) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142260 2014-06-12 08:44:04.570460]
2014-06-12 14:47:46.850212 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2687 (0'0,173'2687] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2685 active] add_log_entry 173'2687 (173'2685) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142260 2014-06-12 08:44:04.570460
2014-06-12 14:47:46.850239 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2687 (0'0,173'2687] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2685 active] append_log  adding 1 keys
2014-06-12 14:47:46.850281 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2687, trimmed: 
2014-06-12 14:47:46.850305 7f9d663dd700  5 -- op tracker -- , seq: 748483, time: 2014-06-12 14:47:46.850300, event: started, op: osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.850316 7f9d663dd700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:46.850321 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:46.850322 7f9d663dd700  5 queue_op 0x61f34a0 seq 2497 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:46.850329 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2687 (0'0,173'2687] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2685 active] do_sub_op osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.850343 7f9d663dd700 10 osd.2 173 dequeue_op 0x68a7400 finish
2014-06-12 14:47:46.850347 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:46.850349 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.850358 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:46.850362 7f9d83173700  5 _do_op 0x61f34a0 seq 2497 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:46.850366 7f9d83173700 10 _do_transaction on 0x5e9b7b0
2014-06-12 14:47:46.850371 7f9d83173700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.850514 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.850540 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:46.850585 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:46.850611 7f9d83173700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.850618 7f9d83173700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:46.850623 7f9d83173700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:46.850814 7f9d83173700 15 _write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 2318336~516096
2014-06-12 14:47:46.850957 7f9d83173700 10 lookup_strip_header done  cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:46.851181 7f9d83173700 10 file_to_extents done 
2014-06-12 14:47:46.852756 7f9d83173700 10 _generic_write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 2318336~516096 = 0
2014-06-12 14:47:46.852798 7f9d83173700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:46.852810 7f9d83173700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:46.852814 7f9d83173700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:46.852819 7f9d83173700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:46.852956 7f9d83173700 20 set_header setting 16305 cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:47:46.853004 7f9d83173700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:46.853015 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:46.853021 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:47:46.855050 7f9d83173700 10 _do_op 0x61f34a0 seq 2497 r = 0, finisher 0x98f53a0 0
2014-06-12 14:47:46.855066 7f9d83173700 10 _finish_op 0x61f34a0 seq 2497 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:46.855073 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:46.855083 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.855082 7f9d7715b700  5 -- op tracker -- , seq: 748483, time: 2014-06-12 14:47:46.855074, event: commit_sent, op: osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.855104 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2687 (0'0,173'2687] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2685 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:46.855130 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.855186 7f9d7795c700  5 -- op tracker -- , seq: 748483, time: 2014-06-12 14:47:46.855179, event: sub_op_applied, op: osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.855197 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2687 (0'0,173'2687] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2686 active] sub_op_modify_applied on 0x5e9b6c0 op osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.855215 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2687 (0'0,173'2687] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2686 active] op_applied on primary on version 173'2687
2014-06-12 14:47:46.855222 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2687 (0'0,173'2687] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2686 active] op_applied on replica on version 173'2687
2014-06-12 14:47:46.855235 7f9d7795c700  5 -- op tracker -- , seq: 748483, time: 2014-06-12 14:47:46.855232, event: done, op: osd_sub_op(client.4305.0:142260 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2687 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.895520 7f9d5a16f700  5 -- op tracker -- , seq: 748484, time: 2014-06-12 14:47:46.894486, event: header_read, op: osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.895541 7f9d5a16f700  5 -- op tracker -- , seq: 748484, time: 2014-06-12 14:47:46.894489, event: throttled, op: osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.895564 7f9d5a16f700  5 -- op tracker -- , seq: 748484, time: 2014-06-12 14:47:46.895460, event: all_read, op: osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.895569 7f9d5a16f700  5 -- op tracker -- , seq: 748484, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.895574 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:46.895579 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:46.895594 7f9d5a16f700 15 osd.2 173 enqueue_op 0x68a7e00 prio 127 cost 475943 latency 0.001108 osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.895698 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:46.895720 7f9d65bdc700 10 osd.2 173 dequeue_op 0x68a7e00 prio 127 cost 475943 latency 0.001233 osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2472 (0'0,173'2472] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2471 active]
2014-06-12 14:47:46.895763 7f9d65bdc700  5 -- op tracker -- , seq: 748484, time: 2014-06-12 14:47:46.895756, event: reached_pg, op: osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.895780 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2472 (0'0,173'2472] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2471 active] handle_message: 0x68a7e00
2014-06-12 14:47:46.895796 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2472 (0'0,173'2472] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2471 active] sub_op_modify trans 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 v 173'2473 (transaction) 174
2014-06-12 14:47:46.895820 7f9d65bdc700  5 -- op tracker -- , seq: 748484, time: 2014-06-12 14:47:46.895814, event: started, op: osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.895841 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2472 (0'0,173'2472] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2471 active] append_log log((0'0,173'2472], crt=0'0) [173'2473 (173'617) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144675 2014-06-12 08:44:58.901196]
2014-06-12 14:47:46.895876 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2473 (0'0,173'2473] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2471 active] add_log_entry 173'2473 (173'617) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144675 2014-06-12 08:44:58.901196
2014-06-12 14:47:46.895908 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2473 (0'0,173'2473] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2471 active] append_log  adding 1 keys
2014-06-12 14:47:46.895954 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2473, trimmed: 
2014-06-12 14:47:46.895993 7f9d65bdc700  5 -- op tracker -- , seq: 748484, time: 2014-06-12 14:47:46.895986, event: started, op: osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.896009 7f9d65bdc700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:46.896014 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:46.896016 7f9d65bdc700  5 queue_op 0x683e180 seq 2306 osr(3.4f4 0x276db70) 477429 bytes   (queue has 1 ops and 477429 bytes)
2014-06-12 14:47:46.896024 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2473 (0'0,173'2473] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2471 active] do_sub_op osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.896047 7f9d65bdc700 10 osd.2 173 dequeue_op 0x68a7e00 finish
2014-06-12 14:47:46.896052 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:46.896055 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.896064 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:47:46.896081 7f9d7a161700  5 _do_op 0x683e180 seq 2306 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:47:46.896088 7f9d7a161700 10 _do_transaction on 0x2d2c02b0
2014-06-12 14:47:46.896098 7f9d7a161700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:46.896240 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:46.896283 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:46.896345 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:46.896460 7f9d7a161700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:46.896472 7f9d7a161700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:47:46.896481 7f9d7a161700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:46.896504 7f9d7a161700 15 _write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 0~475136
2014-06-12 14:47:46.896864 7f9d7a161700 10 lookup_strip_header done  cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:46.896999 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:47:46.899009 7f9d7a161700 10 _generic_write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 0~475136 = 0
2014-06-12 14:47:46.899068 7f9d7a161700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:46.899087 7f9d7a161700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:46.899094 7f9d7a161700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:46.899104 7f9d7a161700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:46.899357 7f9d7a161700 20 set_header setting 17353 cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 parent seq 0
2014-06-12 14:47:46.899448 7f9d7a161700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:47:46.899473 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:46.899485 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:47:46.902364 7f9d7a161700 10 _do_op 0x683e180 seq 2306 r = 0, finisher 0x59fb160 0
2014-06-12 14:47:46.902381 7f9d7a161700 10 _finish_op 0x683e180 seq 2306 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:46.902391 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:47:46.902395 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.902407 7f9d7715b700  5 -- op tracker -- , seq: 748484, time: 2014-06-12 14:47:46.902397, event: commit_sent, op: osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.902434 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2473 (0'0,173'2473] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2471 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:46.902477 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.902548 7f9d7795c700  5 -- op tracker -- , seq: 748484, time: 2014-06-12 14:47:46.902527, event: sub_op_applied, op: osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.902569 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2473 (0'0,173'2473] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active] sub_op_modify_applied on 0x2d2c01c0 op osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.902596 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2473 (0'0,173'2473] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active] op_applied on primary on version 173'2473
2014-06-12 14:47:46.902610 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2473 (0'0,173'2473] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active] op_applied on replica on version 173'2473
2014-06-12 14:47:46.902634 7f9d7795c700  5 -- op tracker -- , seq: 748484, time: 2014-06-12 14:47:46.902628, event: done, op: osd_sub_op(client.4303.0:144675 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2473 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.917389 7f9d5a16f700  5 -- op tracker -- , seq: 748485, time: 2014-06-12 14:47:46.914931, event: header_read, op: osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.917402 7f9d5a16f700  5 -- op tracker -- , seq: 748485, time: 2014-06-12 14:47:46.914933, event: throttled, op: osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.917409 7f9d5a16f700  5 -- op tracker -- , seq: 748485, time: 2014-06-12 14:47:46.917348, event: all_read, op: osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.917418 7f9d5a16f700  5 -- op tracker -- , seq: 748485, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.917423 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:46.917429 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:46.917437 7f9d5a16f700 15 osd.2 173 enqueue_op 0x124e3700 prio 127 cost 1549095 latency 0.002506 osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.917474 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:46.917488 7f9d62bd6700 10 osd.2 173 dequeue_op 0x124e3700 prio 127 cost 1549095 latency 0.002557 osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2883 (0'0,173'2883] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2882 active]
2014-06-12 14:47:46.917530 7f9d62bd6700  5 -- op tracker -- , seq: 748485, time: 2014-06-12 14:47:46.917517, event: reached_pg, op: osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.917551 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2883 (0'0,173'2883] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2882 active] handle_message: 0x124e3700
2014-06-12 14:47:46.917565 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2883 (0'0,173'2883] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2882 active] sub_op_modify trans a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 v 173'2884 (transaction) 174
2014-06-12 14:47:46.917587 7f9d62bd6700  5 -- op tracker -- , seq: 748485, time: 2014-06-12 14:47:46.917582, event: started, op: osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.917604 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2883 (0'0,173'2883] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2882 active] append_log log((0'0,173'2883], crt=0'0) [173'2884 (173'2883) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146508 2014-06-12 08:45:38.132658]
2014-06-12 14:47:46.917633 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2884 (0'0,173'2884] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2882 active] add_log_entry 173'2884 (173'2883) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146508 2014-06-12 08:45:38.132658
2014-06-12 14:47:46.917658 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2884 (0'0,173'2884] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2882 active] append_log  adding 1 keys
2014-06-12 14:47:46.917695 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2884, trimmed: 
2014-06-12 14:47:46.917724 7f9d62bd6700  5 -- op tracker -- , seq: 748485, time: 2014-06-12 14:47:46.917720, event: started, op: osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.917735 7f9d62bd6700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:46.917739 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:46.917741 7f9d62bd6700  5 queue_op 0x76f40c0 seq 2597 osr(3.129 0x276cf20) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:47:46.917757 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:46.917762 7f9d7f16b700  5 _do_op 0x76f40c0 seq 2597 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:46.917765 7f9d7f16b700 10 _do_transaction on 0x7f862b0
2014-06-12 14:47:46.917754 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2884 (0'0,173'2884] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2882 active] do_sub_op osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.917769 7f9d62bd6700 10 osd.2 173 dequeue_op 0x124e3700 finish
2014-06-12 14:47:46.917771 7f9d7f16b700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:46.917774 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:46.917776 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:46.918025 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:46.918065 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:46.918125 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:46.918155 7f9d7f16b700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:46.918166 7f9d7f16b700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:46.918174 7f9d7f16b700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:46.918195 7f9d7f16b700 15 _write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 1368064~1548288
2014-06-12 14:47:46.918546 7f9d7f16b700 10 lookup_strip_header done  cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:46.918734 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:47:46.925798 7f9d7f16b700 10 _generic_write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 1368064~1548288 = 0
2014-06-12 14:47:46.925946 7f9d7f16b700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:46.925976 7f9d7f16b700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:46.925981 7f9d7f16b700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:46.925985 7f9d7f16b700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:46.926276 7f9d7f16b700 20 set_header setting 17398 cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 parent seq 0
2014-06-12 14:47:46.926409 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:46.926421 7f9d7f16b700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:46.926437 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:47:46.934057 7f9d7f16b700 10 _do_op 0x76f40c0 seq 2597 r = 0, finisher 0x65e6ac0 0
2014-06-12 14:47:46.934082 7f9d7f16b700 10 _finish_op 0x76f40c0 seq 2597 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:46.934090 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:46.934103 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:46.934133 7f9d7715b700  5 -- op tracker -- , seq: 748485, time: 2014-06-12 14:47:46.934121, event: commit_sent, op: osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.934149 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2884 (0'0,173'2884] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2882 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:46.934173 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:46.934249 7f9d7795c700  5 -- op tracker -- , seq: 748485, time: 2014-06-12 14:47:46.934244, event: sub_op_applied, op: osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.934257 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2884 (0'0,173'2884] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2883 active] sub_op_modify_applied on 0x7f861c0 op osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:46.934268 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2884 (0'0,173'2884] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2883 active] op_applied on primary on version 173'2884
2014-06-12 14:47:46.934282 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2884 (0'0,173'2884] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2883 active] op_applied on replica on version 173'2884
2014-06-12 14:47:46.934294 7f9d7795c700  5 -- op tracker -- , seq: 748485, time: 2014-06-12 14:47:46.934292, event: done, op: osd_sub_op(client.4313.0:146508 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2884 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:46.966979 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:46.967184 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:47.225492 7f9d5a16f700  5 -- op tracker -- , seq: 748486, time: 2014-06-12 14:47:47.223879, event: header_read, op: osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.225513 7f9d5a16f700  5 -- op tracker -- , seq: 748486, time: 2014-06-12 14:47:47.223881, event: throttled, op: osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.225537 7f9d5a16f700  5 -- op tracker -- , seq: 748486, time: 2014-06-12 14:47:47.225423, event: all_read, op: osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.225549 7f9d5a16f700  5 -- op tracker -- , seq: 748486, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.225557 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:47.225567 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:47.225579 7f9d5a16f700 15 osd.2 173 enqueue_op 0x124e2300 prio 127 cost 664359 latency 0.001700 osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.225664 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:47.225689 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x124e2300 prio 127 cost 664359 latency 0.001809 osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2687 (0'0,173'2687] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2686 active]
2014-06-12 14:47:47.225738 7f9d6b3e7700  5 -- op tracker -- , seq: 748486, time: 2014-06-12 14:47:47.225729, event: reached_pg, op: osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.225757 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2687 (0'0,173'2687] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2686 active] handle_message: 0x124e2300
2014-06-12 14:47:47.225780 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2687 (0'0,173'2687] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2686 active] sub_op_modify trans 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 v 173'2688 (transaction) 174
2014-06-12 14:47:47.225813 7f9d6b3e7700  5 -- op tracker -- , seq: 748486, time: 2014-06-12 14:47:47.225805, event: started, op: osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.225846 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2687 (0'0,173'2687] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2686 active] append_log log((0'0,173'2687], crt=0'0) [173'2688 (173'778) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141501 2014-06-12 08:45:21.368571]
2014-06-12 14:47:47.225897 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2688 (0'0,173'2688] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2686 active] add_log_entry 173'2688 (173'778) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141501 2014-06-12 08:45:21.368571
2014-06-12 14:47:47.225944 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2688 (0'0,173'2688] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2686 active] append_log  adding 1 keys
2014-06-12 14:47:47.226013 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2688, trimmed: 
2014-06-12 14:47:47.226064 7f9d6b3e7700  5 -- op tracker -- , seq: 748486, time: 2014-06-12 14:47:47.226054, event: started, op: osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.226088 7f9d6b3e7700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:47.226095 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:47.226098 7f9d6b3e7700  5 queue_op 0x6d3bec0 seq 2498 osr(3.1b7 0x276d1a0) 665845 bytes   (queue has 1 ops and 665845 bytes)
2014-06-12 14:47:47.226109 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2688 (0'0,173'2688] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2686 active] do_sub_op osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.226137 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x124e2300 finish
2014-06-12 14:47:47.226144 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:47.226149 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:47.226145 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:47.226161 7f9d7e169700  5 _do_op 0x6d3bec0 seq 2498 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:47.226169 7f9d7e169700 10 _do_transaction on 0x5cb5b30
2014-06-12 14:47:47.226179 7f9d7e169700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.226324 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.226369 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:47.226435 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:47.226626 7f9d7e169700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.226640 7f9d7e169700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:47.226649 7f9d7e169700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.226673 7f9d7e169700 15 _write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 0~663552
2014-06-12 14:47:47.226900 7f9d7e169700 10 lookup_strip_header done  cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:47.227213 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:47:47.230109 7f9d7e169700 10 _generic_write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 0~663552 = 0
2014-06-12 14:47:47.230182 7f9d7e169700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:47.230205 7f9d7e169700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:47.230213 7f9d7e169700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:47.230223 7f9d7e169700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:47.230508 7f9d7e169700 20 set_header setting 17393 cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 parent seq 0
2014-06-12 14:47:47.230623 7f9d7e169700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:47.230646 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:47.230661 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:47:47.234876 7f9d7e169700 10 _do_op 0x6d3bec0 seq 2498 r = 0, finisher 0x682abc0 0
2014-06-12 14:47:47.234896 7f9d7e169700 10 _finish_op 0x6d3bec0 seq 2498 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:47.234912 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:47.234918 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:47.234928 7f9d7715b700  5 -- op tracker -- , seq: 748486, time: 2014-06-12 14:47:47.234913, event: commit_sent, op: osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.234953 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2688 (0'0,173'2688] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2686 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:47.234989 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.235089 7f9d7795c700  5 -- op tracker -- , seq: 748486, time: 2014-06-12 14:47:47.235073, event: sub_op_applied, op: osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.235114 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2688 (0'0,173'2688] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2687 active] sub_op_modify_applied on 0x5cb5a40 op osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.235141 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2688 (0'0,173'2688] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2687 active] op_applied on primary on version 173'2688
2014-06-12 14:47:47.235154 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2688 (0'0,173'2688] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2687 active] op_applied on replica on version 173'2688
2014-06-12 14:47:47.235184 7f9d7795c700  5 -- op tracker -- , seq: 748486, time: 2014-06-12 14:47:47.235178, event: done, op: osd_sub_op(client.4284.0:141501 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2688 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.365937 7f9d5a16f700  5 -- op tracker -- , seq: 748487, time: 2014-06-12 14:47:47.364817, event: header_read, op: osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.365948 7f9d5a16f700  5 -- op tracker -- , seq: 748487, time: 2014-06-12 14:47:47.364820, event: throttled, op: osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.365954 7f9d5a16f700  5 -- op tracker -- , seq: 748487, time: 2014-06-12 14:47:47.365891, event: all_read, op: osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.365959 7f9d5a16f700  5 -- op tracker -- , seq: 748487, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.365963 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:47.365968 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:47.365974 7f9d5a16f700 15 osd.2 173 enqueue_op 0x124e2d00 prio 127 cost 516903 latency 0.001157 osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.366006 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:47.366016 7f9d693e3700 10 osd.2 173 dequeue_op 0x124e2d00 prio 127 cost 516903 latency 0.001198 osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2473 (0'0,173'2473] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active]
2014-06-12 14:47:47.366040 7f9d693e3700  5 -- op tracker -- , seq: 748487, time: 2014-06-12 14:47:47.366036, event: reached_pg, op: osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.366050 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2473 (0'0,173'2473] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active] handle_message: 0x124e2d00
2014-06-12 14:47:47.366065 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2473 (0'0,173'2473] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active] sub_op_modify trans 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 v 173'2474 (transaction) 174
2014-06-12 14:47:47.366086 7f9d693e3700  5 -- op tracker -- , seq: 748487, time: 2014-06-12 14:47:47.366081, event: started, op: osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.366104 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2473 (0'0,173'2473] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active] append_log log((0'0,173'2473], crt=0'0) [173'2474 (173'2473) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144676 2014-06-12 08:44:58.908046]
2014-06-12 14:47:47.366137 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2474 (0'0,173'2474] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active] add_log_entry 173'2474 (173'2473) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144676 2014-06-12 08:44:58.908046
2014-06-12 14:47:47.366167 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2474 (0'0,173'2474] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active] append_log  adding 1 keys
2014-06-12 14:47:47.366217 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2474, trimmed: 
2014-06-12 14:47:47.366250 7f9d693e3700  5 -- op tracker -- , seq: 748487, time: 2014-06-12 14:47:47.366243, event: started, op: osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.366265 7f9d693e3700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:47.366271 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:47.366273 7f9d693e3700  5 queue_op 0x7cb7ce0 seq 2307 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:47.366283 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2474 (0'0,173'2474] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active] do_sub_op osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.366304 7f9d693e3700 10 osd.2 173 dequeue_op 0x124e2d00 finish
2014-06-12 14:47:47.366309 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:47.366313 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:47.366337 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:47:47.366346 7f9d7815d700  5 _do_op 0x7cb7ce0 seq 2307 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:47:47.366356 7f9d7815d700 10 _do_transaction on 0x16bce9b0
2014-06-12 14:47:47.366363 7f9d7815d700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:47.366540 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:47.366568 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:47.366614 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:47.366631 7f9d7815d700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:47.366637 7f9d7815d700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:47:47.366640 7f9d7815d700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:47.366653 7f9d7815d700 15 _write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 475136~516096
2014-06-12 14:47:47.366746 7f9d7815d700 10 lookup_strip_header done  cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:47.366762 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:47:47.369054 7f9d7815d700 10 _generic_write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 475136~516096 = 0
2014-06-12 14:47:47.369142 7f9d7815d700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:47.369161 7f9d7815d700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:47.369168 7f9d7815d700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:47.369174 7f9d7815d700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:47.369466 7f9d7815d700 20 set_header setting 17353 cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 parent seq 0
2014-06-12 14:47:47.369562 7f9d7815d700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:47:47.369585 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:47.369596 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:47:47.371751 7f9d7815d700 10 _do_op 0x7cb7ce0 seq 2307 r = 0, finisher 0x7bd5880 0
2014-06-12 14:47:47.371770 7f9d7815d700 10 _finish_op 0x7cb7ce0 seq 2307 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:47.371776 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:47:47.371784 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:47.371798 7f9d7795c700  5 -- op tracker -- , seq: 748487, time: 2014-06-12 14:47:47.371786, event: sub_op_applied, op: osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.371812 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2474 (0'0,173'2474] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active] sub_op_modify_applied on 0x16bce8c0 op osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.371843 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.371864 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2474 (0'0,173'2474] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active] op_applied on primary on version 173'2474
2014-06-12 14:47:47.371874 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2474 (0'0,173'2474] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active] op_applied on replica on version 173'2474
2014-06-12 14:47:47.371914 7f9d7715b700  5 -- op tracker -- , seq: 748487, time: 2014-06-12 14:47:47.371908, event: commit_sent, op: osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.371923 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2474 (0'0,173'2474] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2472 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:47.371938 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.371958 7f9d7715b700  5 -- op tracker -- , seq: 748487, time: 2014-06-12 14:47:47.371955, event: done, op: osd_sub_op(client.4303.0:144676 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2474 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.375771 7f9d5a16f700  5 -- op tracker -- , seq: 748488, time: 2014-06-12 14:47:47.374128, event: header_read, op: osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.375781 7f9d5a16f700  5 -- op tracker -- , seq: 748488, time: 2014-06-12 14:47:47.374129, event: throttled, op: osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.375787 7f9d5a16f700  5 -- op tracker -- , seq: 748488, time: 2014-06-12 14:47:47.375743, event: all_read, op: osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.375792 7f9d5a16f700  5 -- op tracker -- , seq: 748488, time: 0.000000, event: dispatched, op: osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.375796 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:47.375801 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:47.375806 7f9d5a16f700 15 osd.2 173 enqueue_op 0x598e800 prio 127 cost 1032999 latency 0.001678 osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.375832 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:47.375841 7f9d683e1700 10 osd.2 173 dequeue_op 0x598e800 prio 127 cost 1032999 latency 0.001713 osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2688 (0'0,173'2688] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2687 active]
2014-06-12 14:47:47.375867 7f9d683e1700  5 -- op tracker -- , seq: 748488, time: 2014-06-12 14:47:47.375863, event: reached_pg, op: osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.375877 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2688 (0'0,173'2688] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2687 active] handle_message: 0x598e800
2014-06-12 14:47:47.375891 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2688 (0'0,173'2688] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2687 active] sub_op_modify trans c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 v 173'2689 (transaction) 174
2014-06-12 14:47:47.375909 7f9d683e1700  5 -- op tracker -- , seq: 748488, time: 2014-06-12 14:47:47.375905, event: started, op: osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.375928 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2688 (0'0,173'2688] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2687 active] append_log log((0'0,173'2688], crt=0'0) [173'2689 (173'2686) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141253 2014-06-12 08:45:27.169046]
2014-06-12 14:47:47.375964 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2689 (0'0,173'2689] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2687 active] add_log_entry 173'2689 (173'2686) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141253 2014-06-12 08:45:27.169046
2014-06-12 14:47:47.375999 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2689 (0'0,173'2689] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2687 active] append_log  adding 1 keys
2014-06-12 14:47:47.376037 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2689, trimmed: 
2014-06-12 14:47:47.376099 7f9d683e1700  5 -- op tracker -- , seq: 748488, time: 2014-06-12 14:47:47.376083, event: started, op: osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.376127 7f9d683e1700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:47.376134 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:47.376138 7f9d683e1700  5 queue_op 0x7713e60 seq 2499 osr(3.1b7 0x276d1a0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:47.376151 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2689 (0'0,173'2689] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2687 active] do_sub_op osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.376184 7f9d683e1700 10 osd.2 173 dequeue_op 0x598e800 finish
2014-06-12 14:47:47.376182 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:47.376189 7f9d82171700  5 _do_op 0x7713e60 seq 2499 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:47.376188 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:47.376193 7f9d82171700 10 _do_transaction on 0x678e2b0
2014-06-12 14:47:47.376195 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:47.376198 7f9d82171700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.376283 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.376305 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:47.376337 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:47.376422 7f9d82171700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.376428 7f9d82171700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:47.376432 7f9d82171700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.376444 7f9d82171700 15 _write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 1060864~1032192
2014-06-12 14:47:47.376536 7f9d82171700 10 lookup_strip_header done  cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:47.376705 7f9d82171700 10 file_to_extents done 
2014-06-12 14:47:47.380177 7f9d82171700 10 _generic_write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 1060864~1032192 = 0
2014-06-12 14:47:47.380262 7f9d82171700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:47.380276 7f9d82171700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:47.380288 7f9d82171700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:47.380293 7f9d82171700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:47.380499 7f9d82171700 20 set_header setting 17923 cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 parent seq 0
2014-06-12 14:47:47.380595 7f9d82171700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:47.380608 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:47.380622 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:47:47.384179 7f9d82171700 10 _do_op 0x7713e60 seq 2499 r = 0, finisher 0xc52be40 0
2014-06-12 14:47:47.384197 7f9d82171700 10 _finish_op 0x7713e60 seq 2499 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:47.384211 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:47.384215 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:47.384223 7f9d7715b700  5 -- op tracker -- , seq: 748488, time: 2014-06-12 14:47:47.384213, event: commit_sent, op: osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.384244 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2689 (0'0,173'2689] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2687 active] sub_op_modify_commit on op osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:47.384266 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.384299 7f9d7795c700  5 -- op tracker -- , seq: 748488, time: 2014-06-12 14:47:47.384290, event: sub_op_applied, op: osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.384318 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2689 (0'0,173'2689] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2688 active] sub_op_modify_applied on 0x678e1c0 op osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.384335 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2689 (0'0,173'2689] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2688 active] op_applied on primary on version 173'2689
2014-06-12 14:47:47.384345 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2689 (0'0,173'2689] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2688 active] op_applied on replica on version 173'2689
2014-06-12 14:47:47.384368 7f9d7795c700  5 -- op tracker -- , seq: 748488, time: 2014-06-12 14:47:47.384364, event: done, op: osd_sub_op(client.4287.0:141253 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2689 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.487513 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:47.487542 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:47:47.487548 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:47:47.487562 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:47:47.487568 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:47:47.487628 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:47.487634 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:47.487638 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:47.487676 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:47.533417 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:47.533483 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:47.539737 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:47.539745 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:47.539747 7f9d74bfa700 20 osd.2 173 _dispatch 0x24b06e00 pg_stats_ack(1 pgs tid 4865) v1
2014-06-12 14:47:47.539750 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:47:47.539756 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:47.539758 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:47.541777 7f9d5a16f700  5 -- op tracker -- , seq: 748489, time: 2014-06-12 14:47:47.540932, event: header_read, op: osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.541791 7f9d5a16f700  5 -- op tracker -- , seq: 748489, time: 2014-06-12 14:47:47.540934, event: throttled, op: osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.541799 7f9d5a16f700  5 -- op tracker -- , seq: 748489, time: 2014-06-12 14:47:47.541743, event: all_read, op: osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.541807 7f9d5a16f700  5 -- op tracker -- , seq: 748489, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.541811 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:47.541817 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:47.541824 7f9d5a16f700 15 osd.2 173 enqueue_op 0x598fe00 prio 127 cost 516903 latency 0.000892 osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.541849 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:47.541856 7f9d6abe6700 10 osd.2 173 dequeue_op 0x598fe00 prio 127 cost 516903 latency 0.000924 osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2689 (0'0,173'2689] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2688 active]
2014-06-12 14:47:47.541889 7f9d6abe6700  5 -- op tracker -- , seq: 748489, time: 2014-06-12 14:47:47.541884, event: reached_pg, op: osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.541898 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2689 (0'0,173'2689] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2688 active] handle_message: 0x598fe00
2014-06-12 14:47:47.541914 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2689 (0'0,173'2689] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2688 active] sub_op_modify trans 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 v 173'2690 (transaction) 174
2014-06-12 14:47:47.541931 7f9d6abe6700  5 -- op tracker -- , seq: 748489, time: 2014-06-12 14:47:47.541928, event: started, op: osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.541948 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2689 (0'0,173'2689] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2688 active] append_log log((0'0,173'2689], crt=0'0) [173'2690 (173'2687) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142261 2014-06-12 08:44:04.616259]
2014-06-12 14:47:47.541972 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2690 (0'0,173'2690] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2688 active] add_log_entry 173'2690 (173'2687) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142261 2014-06-12 08:44:04.616259
2014-06-12 14:47:47.541997 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2690 (0'0,173'2690] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2688 active] append_log  adding 1 keys
2014-06-12 14:47:47.542034 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2690, trimmed: 
2014-06-12 14:47:47.542059 7f9d6abe6700  5 -- op tracker -- , seq: 748489, time: 2014-06-12 14:47:47.542054, event: started, op: osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.542070 7f9d6abe6700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:47.542073 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:47.542074 7f9d6abe6700  5 queue_op 0x6e74120 seq 2500 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:47.542082 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:47.542088 7f9d7e96a700  5 _do_op 0x6e74120 seq 2500 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:47.542079 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2690 (0'0,173'2690] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2688 active] do_sub_op osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.542092 7f9d7e96a700 10 _do_transaction on 0x5732ef0
2014-06-12 14:47:47.542093 7f9d6abe6700 10 osd.2 173 dequeue_op 0x598fe00 finish
2014-06-12 14:47:47.542096 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:47.542098 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:47.542098 7f9d7e96a700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.542188 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.542208 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:47.542240 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:47.542256 7f9d7e96a700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.542260 7f9d7e96a700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:47.542264 7f9d7e96a700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.542377 7f9d7e96a700 15 _write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 2834432~516096
2014-06-12 14:47:47.542471 7f9d7e96a700 10 lookup_strip_header done  cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:47.542607 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:47:47.543892 7f9d7e96a700 10 _generic_write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 2834432~516096 = 0
2014-06-12 14:47:47.543930 7f9d7e96a700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:47.543950 7f9d7e96a700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:47.543954 7f9d7e96a700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:47.543958 7f9d7e96a700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:47.544113 7f9d7e96a700 20 set_header setting 16305 cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:47:47.544175 7f9d7e96a700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:47.544202 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:47.544208 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:47:47.546302 7f9d7e96a700 10 _do_op 0x6e74120 seq 2500 r = 0, finisher 0x2a3af20 0
2014-06-12 14:47:47.546314 7f9d7e96a700 10 _finish_op 0x6e74120 seq 2500 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:47.546319 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:47.546333 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:47.546341 7f9d7715b700  5 -- op tracker -- , seq: 748489, time: 2014-06-12 14:47:47.546331, event: commit_sent, op: osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.546361 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2690 (0'0,173'2690] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2688 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:47.546385 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.546422 7f9d7795c700  5 -- op tracker -- , seq: 748489, time: 2014-06-12 14:47:47.546414, event: sub_op_applied, op: osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.546440 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2690 (0'0,173'2690] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2689 active] sub_op_modify_applied on 0x5732e00 op osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.546464 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2690 (0'0,173'2690] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2689 active] op_applied on primary on version 173'2690
2014-06-12 14:47:47.546477 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2690 (0'0,173'2690] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2689 active] op_applied on replica on version 173'2690
2014-06-12 14:47:47.546502 7f9d7795c700  5 -- op tracker -- , seq: 748489, time: 2014-06-12 14:47:47.546498, event: done, op: osd_sub_op(client.4305.0:142261 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2690 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.563289 7f9d5a16f700  5 -- op tracker -- , seq: 748490, time: 2014-06-12 14:47:47.562219, event: header_read, op: osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.563314 7f9d5a16f700  5 -- op tracker -- , seq: 748490, time: 2014-06-12 14:47:47.562222, event: throttled, op: osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.563350 7f9d5a16f700  5 -- op tracker -- , seq: 748490, time: 2014-06-12 14:47:47.563229, event: all_read, op: osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.563356 7f9d5a16f700  5 -- op tracker -- , seq: 748490, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.563360 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:47.563365 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:47.563372 7f9d5a16f700 15 osd.2 173 enqueue_op 0x25e42400 prio 127 cost 516903 latency 0.001152 osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.563398 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:47.563425 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x25e42400 prio 127 cost 516903 latency 0.001206 osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2884 (0'0,173'2884] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2883 active]
2014-06-12 14:47:47.563455 7f9d6a3e5700  5 -- op tracker -- , seq: 748490, time: 2014-06-12 14:47:47.563452, event: reached_pg, op: osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.563466 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2884 (0'0,173'2884] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2883 active] handle_message: 0x25e42400
2014-06-12 14:47:47.563474 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2884 (0'0,173'2884] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2883 active] sub_op_modify trans a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 v 173'2885 (transaction) 174
2014-06-12 14:47:47.563487 7f9d6a3e5700  5 -- op tracker -- , seq: 748490, time: 2014-06-12 14:47:47.563484, event: started, op: osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.563499 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2884 (0'0,173'2884] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2883 active] append_log log((0'0,173'2884], crt=0'0) [173'2885 (173'2884) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146509 2014-06-12 08:45:38.880505]
2014-06-12 14:47:47.563516 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2885 (0'0,173'2885] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2883 active] add_log_entry 173'2885 (173'2884) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146509 2014-06-12 08:45:38.880505
2014-06-12 14:47:47.563534 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2885 (0'0,173'2885] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2883 active] append_log  adding 1 keys
2014-06-12 14:47:47.563557 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2885, trimmed: 
2014-06-12 14:47:47.563578 7f9d6a3e5700  5 -- op tracker -- , seq: 748490, time: 2014-06-12 14:47:47.563575, event: started, op: osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.563598 7f9d6a3e5700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:47.563604 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:47.563606 7f9d6a3e5700  5 queue_op 0x6d66960 seq 2598 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:47.563614 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2885 (0'0,173'2885] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2883 active] do_sub_op osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.563639 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x25e42400 finish
2014-06-12 14:47:47.563641 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:47.563643 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:47.563650 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:47.563658 7f9d8116f700  5 _do_op 0x6d66960 seq 2598 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:47.563661 7f9d8116f700 10 _do_transaction on 0x2e636d30
2014-06-12 14:47:47.563667 7f9d8116f700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:47.563782 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:47.563815 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:47.563862 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:47.563888 7f9d8116f700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:47.563896 7f9d8116f700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:47.563902 7f9d8116f700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:47.564005 7f9d8116f700 15 _write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 2916352~516096
2014-06-12 14:47:47.564139 7f9d8116f700 10 lookup_strip_header done  cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:47.564272 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:47:47.566004 7f9d8116f700 10 _generic_write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 2916352~516096 = 0
2014-06-12 14:47:47.566065 7f9d8116f700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:47.566086 7f9d8116f700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:47.566099 7f9d8116f700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:47.566110 7f9d8116f700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:47.566390 7f9d8116f700 20 set_header setting 17398 cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 parent seq 0
2014-06-12 14:47:47.566488 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:47.566509 7f9d8116f700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:47.566528 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:47:47.569943 7f9d8116f700 10 _do_op 0x6d66960 seq 2598 r = 0, finisher 0x6d79400 0
2014-06-12 14:47:47.569963 7f9d8116f700 10 _finish_op 0x6d66960 seq 2598 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:47.569971 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:47.569979 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:47.569983 7f9d7715b700  5 -- op tracker -- , seq: 748490, time: 2014-06-12 14:47:47.569973, event: commit_sent, op: osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.569995 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2885 (0'0,173'2885] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2883 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:47.570022 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.570080 7f9d7795c700  5 -- op tracker -- , seq: 748490, time: 2014-06-12 14:47:47.570064, event: sub_op_applied, op: osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.570094 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2885 (0'0,173'2885] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2884 active] sub_op_modify_applied on 0x2e636c40 op osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.570119 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2885 (0'0,173'2885] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2884 active] op_applied on primary on version 173'2885
2014-06-12 14:47:47.570130 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2885 (0'0,173'2885] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2884 active] op_applied on replica on version 173'2885
2014-06-12 14:47:47.570148 7f9d7795c700  5 -- op tracker -- , seq: 748490, time: 2014-06-12 14:47:47.570144, event: done, op: osd_sub_op(client.4313.0:146509 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2885 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.612584 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:47.612649 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:47.634230 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:47.634301 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:47.692548 7f9d5a16f700  5 -- op tracker -- , seq: 748491, time: 2014-06-12 14:47:47.691276, event: header_read, op: osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.692570 7f9d5a16f700  5 -- op tracker -- , seq: 748491, time: 2014-06-12 14:47:47.691279, event: throttled, op: osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.692606 7f9d5a16f700  5 -- op tracker -- , seq: 748491, time: 2014-06-12 14:47:47.692487, event: all_read, op: osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.692618 7f9d5a16f700  5 -- op tracker -- , seq: 748491, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.692626 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:47.692635 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:47.692648 7f9d5a16f700 15 osd.2 173 enqueue_op 0x25e42100 prio 127 cost 516903 latency 0.001371 osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.692696 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:47.692710 7f9d66bde700 10 osd.2 173 dequeue_op 0x25e42100 prio 127 cost 516903 latency 0.001434 osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2690 (0'0,173'2690] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2689 active]
2014-06-12 14:47:47.692745 7f9d66bde700  5 -- op tracker -- , seq: 748491, time: 2014-06-12 14:47:47.692739, event: reached_pg, op: osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.692759 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2690 (0'0,173'2690] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2689 active] handle_message: 0x25e42100
2014-06-12 14:47:47.692775 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2690 (0'0,173'2690] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2689 active] sub_op_modify trans 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 v 173'2691 (transaction) 174
2014-06-12 14:47:47.692797 7f9d66bde700  5 -- op tracker -- , seq: 748491, time: 2014-06-12 14:47:47.692792, event: started, op: osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.692819 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2690 (0'0,173'2690] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2689 active] append_log log((0'0,173'2690], crt=0'0) [173'2691 (173'2688) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141502 2014-06-12 08:45:21.416526]
2014-06-12 14:47:47.692859 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2691 (0'0,173'2691] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2689 active] add_log_entry 173'2691 (173'2688) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141502 2014-06-12 08:45:21.416526
2014-06-12 14:47:47.692889 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2691 (0'0,173'2691] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2689 active] append_log  adding 1 keys
2014-06-12 14:47:47.692944 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2691, trimmed: 
2014-06-12 14:47:47.692986 7f9d66bde700  5 -- op tracker -- , seq: 748491, time: 2014-06-12 14:47:47.692979, event: started, op: osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.693003 7f9d66bde700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:47.693008 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:47.693010 7f9d66bde700  5 queue_op 0x11345e00 seq 2501 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:47.693027 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:47.693038 7f9d7d167700  5 _do_op 0x11345e00 seq 2501 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:47.693046 7f9d7d167700 10 _do_transaction on 0x19d877b0
2014-06-12 14:47:47.693019 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2691 (0'0,173'2691] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2689 active] do_sub_op osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.693049 7f9d66bde700 10 osd.2 173 dequeue_op 0x25e42100 finish
2014-06-12 14:47:47.693056 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:47.693057 7f9d7d167700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.693061 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:47.693217 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.693252 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:47.693319 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:47.693357 7f9d7d167700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.693368 7f9d7d167700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:47.693376 7f9d7d167700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.693664 7f9d7d167700 15 _write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 663552~516096
2014-06-12 14:47:47.693796 7f9d7d167700 10 lookup_strip_header done  cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:47.693954 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:47:47.695700 7f9d7d167700 10 _generic_write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 663552~516096 = 0
2014-06-12 14:47:47.695746 7f9d7d167700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:47.695757 7f9d7d167700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:47.695761 7f9d7d167700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:47.695766 7f9d7d167700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:47.695913 7f9d7d167700 20 set_header setting 17393 cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 parent seq 0
2014-06-12 14:47:47.695980 7f9d7d167700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:47.695992 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:47.696005 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:47:47.698474 7f9d7d167700 10 _do_op 0x11345e00 seq 2501 r = 0, finisher 0x216fd160 0
2014-06-12 14:47:47.698501 7f9d7d167700 10 _finish_op 0x11345e00 seq 2501 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:47.698515 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:47.698524 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:47.698539 7f9d7715b700  5 -- op tracker -- , seq: 748491, time: 2014-06-12 14:47:47.698526, event: commit_sent, op: osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.698558 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2691 (0'0,173'2691] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2689 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:47.698595 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.698642 7f9d7795c700  5 -- op tracker -- , seq: 748491, time: 2014-06-12 14:47:47.698627, event: sub_op_applied, op: osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.698661 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2691 (0'0,173'2691] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2690 active] sub_op_modify_applied on 0x19d876c0 op osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.698685 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2691 (0'0,173'2691] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2690 active] op_applied on primary on version 173'2691
2014-06-12 14:47:47.698695 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2691 (0'0,173'2691] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2690 active] op_applied on replica on version 173'2691
2014-06-12 14:47:47.698720 7f9d7795c700  5 -- op tracker -- , seq: 748491, time: 2014-06-12 14:47:47.698715, event: done, op: osd_sub_op(client.4284.0:141502 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2691 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.714133 7f9d5a16f700  5 -- op tracker -- , seq: 748492, time: 2014-06-12 14:47:47.711241, event: header_read, op: osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.714143 7f9d5a16f700  5 -- op tracker -- , seq: 748492, time: 2014-06-12 14:47:47.711243, event: throttled, op: osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.714157 7f9d5a16f700  5 -- op tracker -- , seq: 748492, time: 2014-06-12 14:47:47.714102, event: all_read, op: osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.714163 7f9d5a16f700  5 -- op tracker -- , seq: 748492, time: 0.000000, event: dispatched, op: osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.714172 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:47.714177 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:47.714182 7f9d5a16f700 15 osd.2 173 enqueue_op 0x31d60100 prio 127 cost 1069859 latency 0.002941 osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.714204 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:47.714210 7f9d69be4700 10 osd.2 173 dequeue_op 0x31d60100 prio 127 cost 1069859 latency 0.002969 osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.f9( v 173'2727 (0'0,173'2727] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active]
2014-06-12 14:47:47.714230 7f9d69be4700  5 -- op tracker -- , seq: 748492, time: 2014-06-12 14:47:47.714227, event: reached_pg, op: osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.714237 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2727 (0'0,173'2727] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] handle_message: 0x31d60100
2014-06-12 14:47:47.714244 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2727 (0'0,173'2727] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] sub_op_modify trans 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 v 173'2728 (transaction) 174
2014-06-12 14:47:47.714254 7f9d69be4700  5 -- op tracker -- , seq: 748492, time: 2014-06-12 14:47:47.714252, event: started, op: osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.714263 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2727 (0'0,173'2727] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] append_log log((0'0,173'2727], crt=0'0) [173'2728 (173'657) modify   5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 by client.4295.0:143159 2014-06-12 08:44:44.231401]
2014-06-12 14:47:47.714280 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2728 (0'0,173'2728] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] add_log_entry 173'2728 (173'657) modify   5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 by client.4295.0:143159 2014-06-12 08:44:44.231401
2014-06-12 14:47:47.714296 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2728 (0'0,173'2728] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] append_log  adding 1 keys
2014-06-12 14:47:47.714317 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2728, trimmed: 
2014-06-12 14:47:47.714364 7f9d69be4700  5 -- op tracker -- , seq: 748492, time: 2014-06-12 14:47:47.714360, event: started, op: osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.714372 7f9d69be4700  5 queue_transactions existing osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:47.714375 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:47.714376 7f9d69be4700  5 queue_op 0x1ab079e0 seq 2537 osr(3.f9 0x276cfa0) 1071340 bytes   (queue has 1 ops and 1071340 bytes)
2014-06-12 14:47:47.714383 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b77a0 (1 active)
2014-06-12 14:47:47.714389 7f9d7915f700  5 _do_op 0x1ab079e0 seq 2537 osr(3.f9 0x276cfa0)/0x276cfa0 start
2014-06-12 14:47:47.714380 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.f9( v 173'2728 (0'0,173'2728] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] do_sub_op osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.714393 7f9d7915f700 10 _do_transaction on 0x2125430
2014-06-12 14:47:47.714393 7f9d69be4700 10 osd.2 173 dequeue_op 0x31d60100 finish
2014-06-12 14:47:47.714396 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:47.714398 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:47.714398 7f9d7915f700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:47.714483 7f9d7915f700 10 lookup_strip_header done  cid meta oid 516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:47.714506 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:47.714538 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:47.714631 7f9d7915f700 15 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:47.714636 7f9d7915f700 10 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1 r = 0
2014-06-12 14:47:47.714640 7f9d7915f700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:47.714652 7f9d7915f700 15 _write 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 0~1069056
2014-06-12 14:47:47.725452 7f9d7915f700 10 lookup_strip_header done  cid 3.f9_head oid 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3
2014-06-12 14:47:47.725711 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:47:47.728542 7f9d7915f700 10 _generic_write 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 0~1069056 = 0
2014-06-12 14:47:47.728627 7f9d7915f700 15 _setattrs 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3
2014-06-12 14:47:47.728655 7f9d7915f700 10 _setattrs 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 = 0
2014-06-12 14:47:47.728659 7f9d7915f700 15 _setattrs 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3
2014-06-12 14:47:47.728663 7f9d7915f700 10 _setattrs 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 = 0
2014-06-12 14:47:47.728876 7f9d7915f700 20 set_header setting 16838 cid 3.f9_head oid 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 parent seq 0
2014-06-12 14:47:47.728984 7f9d7915f700 20 set_header setting 2218 cid meta oid 516e9ea6/pglog_3.f9/0//-1 parent seq 0
2014-06-12 14:47:47.728998 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:47.729011 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:47:47.734432 7f9d7915f700 10 _do_op 0x1ab079e0 seq 2537 r = 0, finisher 0x541ef20 0
2014-06-12 14:47:47.734468 7f9d7915f700 10 _finish_op 0x1ab079e0 seq 2537 osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:47.734478 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b77a0 (0 active)
2014-06-12 14:47:47.734499 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:47.734514 7f9d7715b700  5 -- op tracker -- , seq: 748492, time: 2014-06-12 14:47:47.734482, event: commit_sent, op: osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.734533 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2728 (0'0,173'2728] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] sub_op_modify_commit on op osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:47.734571 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.734627 7f9d7795c700  5 -- op tracker -- , seq: 748492, time: 2014-06-12 14:47:47.734614, event: sub_op_applied, op: osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.734647 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2728 (0'0,173'2728] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] sub_op_modify_applied on 0x2125340 op osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.734685 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2728 (0'0,173'2728] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] op_applied on primary on version 173'2728
2014-06-12 14:47:47.734701 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2728 (0'0,173'2728] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] op_applied on replica on version 173'2728
2014-06-12 14:47:47.734727 7f9d7795c700  5 -- op tracker -- , seq: 748492, time: 2014-06-12 14:47:47.734721, event: done, op: osd_sub_op(client.4295.0:143159 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.738349 7f9d5a16f700  5 -- op tracker -- , seq: 748493, time: 2014-06-12 14:47:47.737061, event: header_read, op: osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.738379 7f9d5a16f700  5 -- op tracker -- , seq: 748493, time: 2014-06-12 14:47:47.737063, event: throttled, op: osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.738399 7f9d5a16f700  5 -- op tracker -- , seq: 748493, time: 2014-06-12 14:47:47.738296, event: all_read, op: osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.738426 7f9d5a16f700  5 -- op tracker -- , seq: 748493, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.738437 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:47.738452 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:47.738464 7f9d5a16f700 15 osd.2 173 enqueue_op 0x31d61a00 prio 127 cost 516903 latency 0.001403 osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.738508 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:47.738527 7f9d64bda700 10 osd.2 173 dequeue_op 0x31d61a00 prio 127 cost 516903 latency 0.001466 osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2474 (0'0,173'2474] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2473 active]
2014-06-12 14:47:47.738588 7f9d64bda700  5 -- op tracker -- , seq: 748493, time: 2014-06-12 14:47:47.738578, event: reached_pg, op: osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.738619 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2474 (0'0,173'2474] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2473 active] handle_message: 0x31d61a00
2014-06-12 14:47:47.738660 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2474 (0'0,173'2474] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2473 active] sub_op_modify trans 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 v 173'2475 (transaction) 174
2014-06-12 14:47:47.738707 7f9d64bda700  5 -- op tracker -- , seq: 748493, time: 2014-06-12 14:47:47.738698, event: started, op: osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.738735 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2474 (0'0,173'2474] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2473 active] append_log log((0'0,173'2474], crt=0'0) [173'2475 (173'2474) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144677 2014-06-12 08:44:59.118851]
2014-06-12 14:47:47.738793 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2475 (0'0,173'2475] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2473 active] add_log_entry 173'2475 (173'2474) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144677 2014-06-12 08:44:59.118851
2014-06-12 14:47:47.738856 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2475 (0'0,173'2475] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2473 active] append_log  adding 1 keys
2014-06-12 14:47:47.738949 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2475, trimmed: 
2014-06-12 14:47:47.739017 7f9d64bda700  5 -- op tracker -- , seq: 748493, time: 2014-06-12 14:47:47.739006, event: started, op: osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.739039 7f9d64bda700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:47.739046 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:47.739049 7f9d64bda700  5 queue_op 0x208bfb60 seq 2308 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:47.739075 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:47:47.739086 7f9d8016d700  5 _do_op 0x208bfb60 seq 2308 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:47:47.739092 7f9d8016d700 10 _do_transaction on 0x2a29b0b0
2014-06-12 14:47:47.739069 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2475 (0'0,173'2475] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2473 active] do_sub_op osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.739098 7f9d64bda700 10 osd.2 173 dequeue_op 0x31d61a00 finish
2014-06-12 14:47:47.739100 7f9d8016d700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:47.739105 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:47.739110 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:47.739232 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:47.739265 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:47.739326 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:47.739361 7f9d8016d700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:47.739371 7f9d8016d700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:47:47.739379 7f9d8016d700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:47.739652 7f9d8016d700 15 _write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 991232~516096
2014-06-12 14:47:47.739881 7f9d8016d700 10 lookup_strip_header done  cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:47.740084 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:47:47.742627 7f9d8016d700 10 _generic_write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 991232~516096 = 0
2014-06-12 14:47:47.742682 7f9d8016d700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:47.742704 7f9d8016d700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:47.742713 7f9d8016d700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:47.742723 7f9d8016d700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:47.742983 7f9d8016d700 20 set_header setting 17353 cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 parent seq 0
2014-06-12 14:47:47.743071 7f9d8016d700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:47:47.743095 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:47.743109 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:47:47.746695 7f9d5a16f700  5 -- op tracker -- , seq: 748494, time: 2014-06-12 14:47:47.745282, event: header_read, op: osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.746727 7f9d5a16f700  5 -- op tracker -- , seq: 748494, time: 2014-06-12 14:47:47.745285, event: throttled, op: osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.746746 7f9d5a16f700  5 -- op tracker -- , seq: 748494, time: 2014-06-12 14:47:47.746636, event: all_read, op: osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.746771 7f9d5a16f700  5 -- op tracker -- , seq: 748494, time: 0.000000, event: dispatched, op: osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.746783 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:47.746799 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:47.746819 7f9d5a16f700 15 osd.2 173 enqueue_op 0x31d61500 prio 127 cost 516903 latency 0.001536 osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.746904 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:47.746926 7f9d653db700 10 osd.2 173 dequeue_op 0x31d61500 prio 127 cost 516903 latency 0.001643 osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2691 (0'0,173'2691] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2690 active]
2014-06-12 14:47:47.746965 7f9d653db700  5 -- op tracker -- , seq: 748494, time: 2014-06-12 14:47:47.746959, event: reached_pg, op: osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.746982 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2691 (0'0,173'2691] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2690 active] handle_message: 0x31d61500
2014-06-12 14:47:47.746998 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2691 (0'0,173'2691] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2690 active] sub_op_modify trans c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 v 173'2692 (transaction) 174
2014-06-12 14:47:47.747020 7f9d653db700  5 -- op tracker -- , seq: 748494, time: 2014-06-12 14:47:47.747015, event: started, op: osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.747042 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2691 (0'0,173'2691] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2690 active] append_log log((0'0,173'2691], crt=0'0) [173'2692 (173'2689) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141254 2014-06-12 08:45:27.170054]
2014-06-12 14:47:47.747088 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2692 (0'0,173'2692] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2690 active] add_log_entry 173'2692 (173'2689) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141254 2014-06-12 08:45:27.170054
2014-06-12 14:47:47.747123 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2692 (0'0,173'2692] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2690 active] append_log  adding 1 keys
2014-06-12 14:47:47.747168 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2692, trimmed: 
2014-06-12 14:47:47.747203 7f9d653db700  5 -- op tracker -- , seq: 748494, time: 2014-06-12 14:47:47.747196, event: started, op: osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.747217 7f9d653db700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:47.747223 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:47.747225 7f9d653db700  5 queue_op 0x2273320 seq 2502 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:47:47.747244 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (2 active)
2014-06-12 14:47:47.747255 7f9d79960700  5 _do_op 0x2273320 seq 2502 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:47.747262 7f9d79960700 10 _do_transaction on 0x5733cf0
2014-06-12 14:47:47.747234 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2692 (0'0,173'2692] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2690 active] do_sub_op osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.747267 7f9d653db700 10 osd.2 173 dequeue_op 0x31d61500 finish
2014-06-12 14:47:47.747274 7f9d79960700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.747275 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:47.747280 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:47.747416 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.747556 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:47.748651 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:47.748693 7f9d79960700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.748702 7f9d79960700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:47.748710 7f9d79960700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.748734 7f9d79960700 15 _write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 2093056~516096
2014-06-12 14:47:47.748819 7f9d8016d700 10 _do_op 0x208bfb60 seq 2308 r = 0, finisher 0x6388560 0
2014-06-12 14:47:47.748840 7f9d8016d700 10 _finish_op 0x208bfb60 seq 2308 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:47.748849 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (1 active)
2014-06-12 14:47:47.748858 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:47.748872 7f9d7715b700  5 -- op tracker -- , seq: 748493, time: 2014-06-12 14:47:47.748855, event: commit_sent, op: osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.748919 7f9d79960700 10 lookup_strip_header done  cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:47.748891 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2475 (0'0,173'2475] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2473 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:47.748952 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.749024 7f9d7795c700  5 -- op tracker -- , seq: 748493, time: 2014-06-12 14:47:47.749010, event: sub_op_applied, op: osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.749042 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2475 (0'0,173'2475] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2474 active] sub_op_modify_applied on 0x2a29afc0 op osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.749084 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2475 (0'0,173'2475] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2474 active] op_applied on primary on version 173'2475
2014-06-12 14:47:47.749113 7f9d79960700 10 file_to_extents done 
2014-06-12 14:47:47.749108 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2475 (0'0,173'2475] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2474 active] op_applied on replica on version 173'2475
2014-06-12 14:47:47.749161 7f9d7795c700  5 -- op tracker -- , seq: 748493, time: 2014-06-12 14:47:47.749150, event: done, op: osd_sub_op(client.4303.0:144677 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2475 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.751210 7f9d79960700 10 _generic_write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 2093056~516096 = 0
2014-06-12 14:47:47.751270 7f9d79960700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:47.751291 7f9d79960700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:47.751299 7f9d79960700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:47.751309 7f9d79960700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:47.751556 7f9d79960700 20 set_header setting 17923 cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 parent seq 0
2014-06-12 14:47:47.751646 7f9d79960700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:47.751669 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:47.751682 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:47:47.755480 7f9d79960700 10 _do_op 0x2273320 seq 2502 r = 0, finisher 0x1ecc7380 0
2014-06-12 14:47:47.755510 7f9d79960700 10 _finish_op 0x2273320 seq 2502 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:47.755520 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:47.755532 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:47.755547 7f9d7715b700  5 -- op tracker -- , seq: 748494, time: 2014-06-12 14:47:47.755527, event: commit_sent, op: osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.755570 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2692 (0'0,173'2692] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2690 active] sub_op_modify_commit on op osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:47.755620 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.755731 7f9d7795c700  5 -- op tracker -- , seq: 748494, time: 2014-06-12 14:47:47.755716, event: sub_op_applied, op: osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.755754 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2692 (0'0,173'2692] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active] sub_op_modify_applied on 0x5733c00 op osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.755782 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2692 (0'0,173'2692] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active] op_applied on primary on version 173'2692
2014-06-12 14:47:47.755795 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2692 (0'0,173'2692] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active] op_applied on replica on version 173'2692
2014-06-12 14:47:47.755825 7f9d7795c700  5 -- op tracker -- , seq: 748494, time: 2014-06-12 14:47:47.755819, event: done, op: osd_sub_op(client.4287.0:141254 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2692 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.825602 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.825846 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.939889 7f9d5a16f700  5 -- op tracker -- , seq: 748495, time: 2014-06-12 14:47:47.938908, event: header_read, op: osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.939902 7f9d5a16f700  5 -- op tracker -- , seq: 748495, time: 2014-06-12 14:47:47.938911, event: throttled, op: osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.939912 7f9d5a16f700  5 -- op tracker -- , seq: 748495, time: 2014-06-12 14:47:47.939822, event: all_read, op: osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.939921 7f9d5a16f700  5 -- op tracker -- , seq: 748495, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.939929 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:47.939936 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:47.939946 7f9d5a16f700 15 osd.2 173 enqueue_op 0x31d60300 prio 127 cost 516903 latency 0.001038 osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.939991 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:47.940004 7f9d623d5700 10 osd.2 173 dequeue_op 0x31d60300 prio 127 cost 516903 latency 0.001096 osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2885 (0'0,173'2885] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2884 active]
2014-06-12 14:47:47.940037 7f9d623d5700  5 -- op tracker -- , seq: 748495, time: 2014-06-12 14:47:47.940033, event: reached_pg, op: osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.940054 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2885 (0'0,173'2885] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2884 active] handle_message: 0x31d60300
2014-06-12 14:47:47.940066 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2885 (0'0,173'2885] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2884 active] sub_op_modify trans a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 v 173'2886 (transaction) 174
2014-06-12 14:47:47.940083 7f9d623d5700  5 -- op tracker -- , seq: 748495, time: 2014-06-12 14:47:47.940079, event: started, op: osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.940123 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2885 (0'0,173'2885] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2884 active] append_log log((0'0,173'2885], crt=0'0) [173'2886 (173'2885) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146510 2014-06-12 08:45:39.119643]
2014-06-12 14:47:47.940165 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2886 (0'0,173'2886] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2884 active] add_log_entry 173'2886 (173'2885) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146510 2014-06-12 08:45:39.119643
2014-06-12 14:47:47.940206 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2886 (0'0,173'2886] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2884 active] append_log  adding 1 keys
2014-06-12 14:47:47.940271 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2886, trimmed: 
2014-06-12 14:47:47.940310 7f9d623d5700  5 -- op tracker -- , seq: 748495, time: 2014-06-12 14:47:47.940304, event: started, op: osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.940322 7f9d623d5700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:47.940328 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:47.940330 7f9d623d5700  5 queue_op 0x72c5e00 seq 2599 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:47.940339 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2886 (0'0,173'2886] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2884 active] do_sub_op osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.940354 7f9d623d5700 10 osd.2 173 dequeue_op 0x31d60300 finish
2014-06-12 14:47:47.940358 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:47.940363 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:47.945652 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:47.945685 7f9d7d968700  5 _do_op 0x72c5e00 seq 2599 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:47.945704 7f9d7d968700 10 _do_transaction on 0x65ae7f0
2014-06-12 14:47:47.945718 7f9d7d968700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:47.945913 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:47.945948 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:47.946005 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:47.946191 7f9d7d968700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:47.946200 7f9d7d968700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:47.946215 7f9d7d968700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:47.946236 7f9d7d968700 15 _write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 3432448~516096
2014-06-12 14:47:47.946401 7f9d7d968700 10 lookup_strip_header done  cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:47.946717 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:47:47.948257 7f9d5a16f700  5 -- op tracker -- , seq: 748496, time: 2014-06-12 14:47:47.941149, event: header_read, op: osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.948270 7f9d5a16f700  5 -- op tracker -- , seq: 748496, time: 2014-06-12 14:47:47.941151, event: throttled, op: osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.948280 7f9d5a16f700  5 -- op tracker -- , seq: 748496, time: 2014-06-12 14:47:47.948216, event: all_read, op: osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.948289 7f9d5a16f700  5 -- op tracker -- , seq: 748496, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.948296 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:47.948304 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:47.948313 7f9d5a16f700 15 osd.2 173 enqueue_op 0x31d60f00 prio 127 cost 516903 latency 0.007164 osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.948690 7f9d7d968700 10 _generic_write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 3432448~516096 = 0
2014-06-12 14:47:47.948740 7f9d7d968700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:47.948773 7f9d7d968700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:47.948780 7f9d7d968700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:47.948788 7f9d7d968700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:47.948998 7f9d7d968700 20 set_header setting 17398 cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 parent seq 0
2014-06-12 14:47:47.949078 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:47.949095 7f9d7d968700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:47.949105 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:47:47.951425 7f9d7d968700 10 _do_op 0x72c5e00 seq 2599 r = 0, finisher 0x2897000 0
2014-06-12 14:47:47.951449 7f9d7d968700 10 _finish_op 0x72c5e00 seq 2599 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:47.951482 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:47.951481 7f9d7715b700  5 -- op tracker -- , seq: 748495, time: 2014-06-12 14:47:47.951468, event: commit_sent, op: osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.951490 7f9d68be2700 10 osd.2 173 dequeue_op 0x31d60f00 prio 127 cost 516903 latency 0.010341 osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2692 (0'0,173'2692] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active]
2014-06-12 14:47:47.951499 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2886 (0'0,173'2886] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2884 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:47.951517 7f9d68be2700  5 -- op tracker -- , seq: 748496, time: 2014-06-12 14:47:47.951513, event: reached_pg, op: osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.951524 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.951528 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2692 (0'0,173'2692] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active] handle_message: 0x31d60f00
2014-06-12 14:47:47.951540 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2692 (0'0,173'2692] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active] sub_op_modify trans 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 v 173'2693 (transaction) 174
2014-06-12 14:47:47.951555 7f9d68be2700  5 -- op tracker -- , seq: 748496, time: 2014-06-12 14:47:47.951551, event: started, op: osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.951573 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2692 (0'0,173'2692] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active] append_log log((0'0,173'2692], crt=0'0) [173'2693 (173'2690) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142262 2014-06-12 08:44:04.652699]
2014-06-12 14:47:47.951605 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2693 (0'0,173'2693] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active] add_log_entry 173'2693 (173'2690) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142262 2014-06-12 08:44:04.652699
2014-06-12 14:47:47.951628 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2693 (0'0,173'2693] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active] append_log  adding 1 keys
2014-06-12 14:47:47.951663 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2693, trimmed: 
2014-06-12 14:47:47.951687 7f9d68be2700  5 -- op tracker -- , seq: 748496, time: 2014-06-12 14:47:47.951682, event: started, op: osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.951698 7f9d68be2700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:47.951702 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:47.951704 7f9d68be2700  5 queue_op 0x61d77a0 seq 2503 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:47.951721 7f9d7795c700  5 -- op tracker -- , seq: 748495, time: 2014-06-12 14:47:47.951716, event: sub_op_applied, op: osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.951729 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2886 (0'0,173'2886] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2885 active] sub_op_modify_applied on 0x65ae700 op osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.951746 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2886 (0'0,173'2886] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2885 active] op_applied on primary on version 173'2886
2014-06-12 14:47:47.951757 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2886 (0'0,173'2886] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2885 active] op_applied on replica on version 173'2886
2014-06-12 14:47:47.951789 7f9d7d968700  5 -- op tracker -- , seq: 748495, time: 2014-06-12 14:47:47.951780, event: done, op: osd_sub_op(client.4313.0:146510 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2886 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.951814 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:47.951819 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:47.951834 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:47.951843 7f9d7c165700  5 _do_op 0x61d77a0 seq 2503 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:47.951832 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2693 (0'0,173'2693] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active] do_sub_op osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.951847 7f9d7c165700 10 _do_transaction on 0x5b7ad30
2014-06-12 14:47:47.951848 7f9d68be2700 10 osd.2 173 dequeue_op 0x31d60f00 finish
2014-06-12 14:47:47.951852 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:47.951854 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:47.951853 7f9d7c165700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.951963 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.951992 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:47.952040 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:47.952065 7f9d7c165700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.952073 7f9d7c165700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:47.952079 7f9d7c165700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:47.952167 7f9d7c165700 15 _write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 3350528~516096
2014-06-12 14:47:47.952318 7f9d7c165700 10 lookup_strip_header done  cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:47.952438 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:47:47.954321 7f9d7c165700 10 _generic_write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 3350528~516096 = 0
2014-06-12 14:47:47.954409 7f9d7c165700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:47.954428 7f9d7c165700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:47.954436 7f9d7c165700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:47.954444 7f9d7c165700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:47.954662 7f9d7c165700 20 set_header setting 16305 cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:47:47.954749 7f9d7c165700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:47.954767 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:47.954776 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:47:47.957320 7f9d7c165700 10 _do_op 0x61d77a0 seq 2503 r = 0, finisher 0x5748320 0
2014-06-12 14:47:47.957338 7f9d7c165700 10 _finish_op 0x61d77a0 seq 2503 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:47.957342 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:47.957358 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:47.957385 7f9d7795c700  5 -- op tracker -- , seq: 748496, time: 2014-06-12 14:47:47.957375, event: sub_op_applied, op: osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.957397 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2693 (0'0,173'2693] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active] sub_op_modify_applied on 0x5b7ac40 op osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:47.957421 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.957435 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2693 (0'0,173'2693] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active] op_applied on primary on version 173'2693
2014-06-12 14:47:47.957444 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2693 (0'0,173'2693] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active] op_applied on replica on version 173'2693
2014-06-12 14:47:47.957461 7f9d7715b700  5 -- op tracker -- , seq: 748496, time: 2014-06-12 14:47:47.957456, event: commit_sent, op: osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:47.957470 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2693 (0'0,173'2693] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2691 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:47.957482 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:47.957501 7f9d7715b700  5 -- op tracker -- , seq: 748496, time: 2014-06-12 14:47:47.957498, event: done, op: osd_sub_op(client.4305.0:142262 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2693 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:48.153576 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:48.153782 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:48.157099 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:48.157309 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:48.202231 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:48.209349 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:48.339940 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:48.360872 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:48.364805 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:48.371981 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:48.428524 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:48.428613 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:48.487735 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:48.487767 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:48.487772 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:48.487774 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:48.487811 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:48.510709 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:48.516345 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:48.544298 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:48.545339 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:48.548195 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:48.556105 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:48.850432 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:48.855161 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:48.881249 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:48.881276 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:48.896123 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:48.902466 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:48.917837 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:48.934177 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:49.226214 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:49.234980 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:49.366377 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:49.371854 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:49.376270 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:49.381983 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:49.382130 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:49.384294 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:49.487866 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:49.487897 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:49.487902 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:49.487904 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:49.487944 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:49.529302 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:49.529336 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:49.542172 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:49.546403 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:49.563733 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:49.570049 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:49.693149 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:49.698609 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:49.714473 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:49.734586 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:49.739201 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:49.747345 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:49.748922 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:49.755602 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:49.882782 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:49.882826 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:49.940442 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:49.951909 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:49.951943 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:49.957427 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:50.157183 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:50.157400 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:50.202303 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:50.209426 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:50.336298 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19429 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:47:50.336314 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19429 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:47:50.336695 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:50.336729 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:50.336906 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:50.336937 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:50.336961 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:50.336973 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:50.336993 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:50.337008 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:50.337032 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:50.337053 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:50.337071 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:50.337088 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:50.337102 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:50.337173 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:50.337202 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:50.337217 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:50.337230 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:50.337270 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:50.337295 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:50.337392 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:50.337414 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:50.337494 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:50.340015 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:50.360947 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:50.364876 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:50.372055 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:50.421160 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:50.421253 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:50.433773 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:50.433868 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:50.487999 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:50.488071 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:50.488069 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 5.59 >= max 0.5 = no, load too high
2014-06-12 14:47:50.488085 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:47:50.488089 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92718 < max (604800 seconds)
2014-06-12 14:47:50.488120 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:47:50.488123 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:50.488125 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:50.510793 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:50.516422 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:50.534968 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:50.535069 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:50.544394 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:50.545416 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:50.548275 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:50.556186 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:50.850541 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:50.855241 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:50.896195 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:50.902563 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:50.917929 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:50.934250 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:51.057660 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:51.057855 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:51.150476 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:51.150539 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:51.226307 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:51.235056 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:51.366462 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:51.371953 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:51.376369 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:51.384373 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:51.488214 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:51.488241 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:51.488245 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:51.488248 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:51.488287 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:51.542246 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:51.546476 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:51.563816 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:51.566968 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:51.567017 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:51.570130 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:51.583713 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:51.583742 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:51.693252 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:51.698694 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:51.714556 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:51.734656 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:51.739272 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:51.747415 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:51.748994 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:51.755672 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:51.940533 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:51.952007 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:51.952024 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:51.957500 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:52.157260 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:52.157473 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:52.202398 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:52.209512 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:52.253970 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:52.254008 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:52.267656 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:52.267684 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:52.340082 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:52.361025 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:52.364956 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:52.372117 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:52.488313 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:52.488329 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:47:52.488332 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:47:52.488334 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:47:52.488340 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:52.488342 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:52.488343 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:52.488370 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:52.510871 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:52.513962 7f9d5a16f700  5 -- op tracker -- , seq: 748497, time: 2014-06-12 14:47:52.513836, event: header_read, op: osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.513992 7f9d5a16f700  5 -- op tracker -- , seq: 748497, time: 2014-06-12 14:47:52.513839, event: throttled, op: osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.514012 7f9d5a16f700  5 -- op tracker -- , seq: 748497, time: 2014-06-12 14:47:52.513906, event: all_read, op: osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.514039 7f9d5a16f700  5 -- op tracker -- , seq: 748497, time: 0.000000, event: dispatched, op: osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.514044 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:52.514052 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:52.514061 7f9d5a16f700 15 osd.2 173 enqueue_op 0x17c2a000 prio 127 cost 4899 latency 0.000224 osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.514088 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:52.514098 7f9d67be0700 10 osd.2 173 dequeue_op 0x17c2a000 prio 127 cost 4899 latency 0.000261 osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.f9( v 173'2728 (0'0,173'2728] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active]
2014-06-12 14:47:52.514124 7f9d67be0700  5 -- op tracker -- , seq: 748497, time: 2014-06-12 14:47:52.514119, event: reached_pg, op: osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.514134 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2728 (0'0,173'2728] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] handle_message: 0x17c2a000
2014-06-12 14:47:52.514149 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2728 (0'0,173'2728] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] sub_op_modify trans 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 v 173'2729 (transaction) 174
2014-06-12 14:47:52.514166 7f9d67be0700  5 -- op tracker -- , seq: 748497, time: 2014-06-12 14:47:52.514162, event: started, op: osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.514182 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2728 (0'0,173'2728] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] append_log log((0'0,173'2728], crt=0'0) [173'2729 (173'2728) modify   5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 by client.4295.0:143160 2014-06-12 08:44:44.236342]
2014-06-12 14:47:52.514221 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2729 (0'0,173'2729] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] add_log_entry 173'2729 (173'2728) modify   5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 by client.4295.0:143160 2014-06-12 08:44:44.236342
2014-06-12 14:47:52.514267 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2729 (0'0,173'2729] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] append_log  adding 1 keys
2014-06-12 14:47:52.514332 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2729, trimmed: 
2014-06-12 14:47:52.514379 7f9d67be0700  5 -- op tracker -- , seq: 748497, time: 2014-06-12 14:47:52.514365, event: started, op: osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.514391 7f9d67be0700  5 queue_transactions existing osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:52.514394 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:52.514396 7f9d67be0700  5 queue_op 0x69c92c0 seq 2538 osr(3.f9 0x276cfa0) 6380 bytes   (queue has 1 ops and 6380 bytes)
2014-06-12 14:47:52.514403 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.f9( v 173'2729 (0'0,173'2729] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] do_sub_op osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.514410 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b77a0 (1 active)
2014-06-12 14:47:52.514417 7f9d67be0700 10 osd.2 173 dequeue_op 0x17c2a000 finish
2014-06-12 14:47:52.514420 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:52.514420 7f9d7a962700  5 _do_op 0x69c92c0 seq 2538 osr(3.f9 0x276cfa0)/0x276cfa0 start
2014-06-12 14:47:52.514423 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:52.514425 7f9d7a962700 10 _do_transaction on 0x191d02b0
2014-06-12 14:47:52.514431 7f9d7a962700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:52.514587 7f9d7a962700 10 lookup_strip_header done  cid meta oid 516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:52.514629 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:52.514731 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:52.514879 7f9d7a962700 15 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:52.514899 7f9d7a962700 10 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1 r = 0
2014-06-12 14:47:52.514914 7f9d7a962700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:52.514949 7f9d7a962700 15 _write 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 1069056~4096
2014-06-12 14:47:52.515288 7f9d7a962700 10 lookup_strip_header done  cid 3.f9_head oid 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3
2014-06-12 14:47:52.515309 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:47:52.515542 7f9d7a962700 10 _generic_write 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 1069056~4096 = 0
2014-06-12 14:47:52.515569 7f9d7a962700 15 _setattrs 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3
2014-06-12 14:47:52.515599 7f9d7a962700 10 _setattrs 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 = 0
2014-06-12 14:47:52.515611 7f9d7a962700 15 _setattrs 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3
2014-06-12 14:47:52.515625 7f9d7a962700 10 _setattrs 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 = 0
2014-06-12 14:47:52.515872 7f9d7a962700 20 set_header setting 16838 cid 3.f9_head oid 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 parent seq 0
2014-06-12 14:47:52.515917 7f9d7a962700 20 set_header setting 2218 cid meta oid 516e9ea6/pglog_3.f9/0//-1 parent seq 0
2014-06-12 14:47:52.515947 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:52.515972 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:47:52.516289 7f9d7a962700 10 _do_op 0x69c92c0 seq 2538 r = 0, finisher 0x88906e0 0
2014-06-12 14:47:52.516314 7f9d7a962700 10 _finish_op 0x69c92c0 seq 2538 osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:52.516345 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b77a0 (0 active)
2014-06-12 14:47:52.516354 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:52.516343 7f9d7715b700  5 -- op tracker -- , seq: 748497, time: 2014-06-12 14:47:52.516327, event: commit_sent, op: osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.516364 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2729 (0'0,173'2729] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] sub_op_modify_commit on op osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:52.516411 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:52.516554 7f9d7795c700  5 -- op tracker -- , seq: 748497, time: 2014-06-12 14:47:52.516547, event: sub_op_applied, op: osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.516581 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2729 (0'0,173'2729] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] sub_op_modify_applied on 0x191d01c0 op osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.516626 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2729 (0'0,173'2729] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] op_applied on primary on version 173'2729
2014-06-12 14:47:52.516654 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2729 (0'0,173'2729] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] op_applied on replica on version 173'2729
2014-06-12 14:47:52.516857 7f9d7795c700  5 -- op tracker -- , seq: 748497, time: 2014-06-12 14:47:52.516845, event: done, op: osd_sub_op(client.4295.0:143160 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.517048 7f9d5a16f700  5 -- op tracker -- , seq: 748498, time: 2014-06-12 14:47:52.515423, event: header_read, op: osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.517074 7f9d5a16f700  5 -- op tracker -- , seq: 748498, time: 2014-06-12 14:47:52.515426, event: throttled, op: osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.517086 7f9d5a16f700  5 -- op tracker -- , seq: 748498, time: 2014-06-12 14:47:52.517005, event: all_read, op: osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.517096 7f9d5a16f700  5 -- op tracker -- , seq: 748498, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.517103 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:52.517111 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:52.517120 7f9d5a16f700 15 osd.2 173 enqueue_op 0x20e86d00 prio 127 cost 516903 latency 0.001697 osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.517161 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:52.517174 7f9d673df700 10 osd.2 173 dequeue_op 0x20e86d00 prio 127 cost 516903 latency 0.001750 osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2693 (0'0,173'2693] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2692 active]
2014-06-12 14:47:52.517205 7f9d673df700  5 -- op tracker -- , seq: 748498, time: 2014-06-12 14:47:52.517198, event: reached_pg, op: osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.517217 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2693 (0'0,173'2693] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2692 active] handle_message: 0x20e86d00
2014-06-12 14:47:52.517237 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2693 (0'0,173'2693] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2692 active] sub_op_modify trans 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 v 173'2694 (transaction) 174
2014-06-12 14:47:52.517247 7f9d673df700  5 -- op tracker -- , seq: 748498, time: 2014-06-12 14:47:52.517244, event: started, op: osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.517255 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2693 (0'0,173'2693] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2692 active] append_log log((0'0,173'2693], crt=0'0) [173'2694 (173'2691) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141503 2014-06-12 08:45:27.294312]
2014-06-12 14:47:52.517270 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2694 (0'0,173'2694] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2692 active] add_log_entry 173'2694 (173'2691) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141503 2014-06-12 08:45:27.294312
2014-06-12 14:47:52.517285 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2694 (0'0,173'2694] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2692 active] append_log  adding 1 keys
2014-06-12 14:47:52.517308 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2694, trimmed: 
2014-06-12 14:47:52.517325 7f9d673df700  5 -- op tracker -- , seq: 748498, time: 2014-06-12 14:47:52.517322, event: started, op: osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.517333 7f9d673df700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:52.517337 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:52.517339 7f9d673df700  5 queue_op 0x15e61f20 seq 2504 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:52.517346 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2694 (0'0,173'2694] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2692 active] do_sub_op osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.517356 7f9d673df700 10 osd.2 173 dequeue_op 0x20e86d00 finish
2014-06-12 14:47:52.517359 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:52.517360 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:52.517436 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:52.517452 7f9d82972700  5 _do_op 0x15e61f20 seq 2504 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:52.517460 7f9d82972700 10 _do_transaction on 0x61d87f0
2014-06-12 14:47:52.517470 7f9d82972700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.517660 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.517705 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:52.517751 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:52.517776 7f9d82972700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.517783 7f9d82972700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:52.517788 7f9d82972700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.517805 7f9d82972700 15 _write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 1179648~516096
2014-06-12 14:47:52.517960 7f9d82972700 10 lookup_strip_header done  cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:52.518194 7f9d82972700 10 file_to_extents done 
2014-06-12 14:47:52.520659 7f9d82972700 10 _generic_write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 1179648~516096 = 0
2014-06-12 14:47:52.520723 7f9d82972700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:52.520741 7f9d82972700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:52.520749 7f9d82972700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:52.520756 7f9d82972700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:52.520989 7f9d82972700 20 set_header setting 17393 cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 parent seq 0
2014-06-12 14:47:52.521060 7f9d82972700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:52.521077 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:52.521087 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:47:52.523975 7f9d82972700 10 _do_op 0x15e61f20 seq 2504 r = 0, finisher 0xa014fa0 0
2014-06-12 14:47:52.524012 7f9d7715b700  5 -- op tracker -- , seq: 748498, time: 2014-06-12 14:47:52.524003, event: commit_sent, op: osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.524025 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2694 (0'0,173'2694] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2692 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:52.524049 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:52.524075 7f9d82972700 10 _finish_op 0x15e61f20 seq 2504 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:52.524095 7f9d7795c700  5 -- op tracker -- , seq: 748498, time: 2014-06-12 14:47:52.524090, event: sub_op_applied, op: osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.524103 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2694 (0'0,173'2694] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2693 active] sub_op_modify_applied on 0x61d8700 op osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.524119 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2694 (0'0,173'2694] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2693 active] op_applied on primary on version 173'2694
2014-06-12 14:47:52.524131 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2694 (0'0,173'2694] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2693 active] op_applied on replica on version 173'2694
2014-06-12 14:47:52.524160 7f9d82972700  5 -- op tracker -- , seq: 748498, time: 2014-06-12 14:47:52.524151, event: done, op: osd_sub_op(client.4284.0:141503 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2694 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.524187 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:52.524192 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:52.556258 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:52.661739 7f9d5a16f700  5 -- op tracker -- , seq: 748499, time: 2014-06-12 14:47:52.660240, event: header_read, op: osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.661756 7f9d5a16f700  5 -- op tracker -- , seq: 748499, time: 2014-06-12 14:47:52.660243, event: throttled, op: osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.661766 7f9d5a16f700  5 -- op tracker -- , seq: 748499, time: 2014-06-12 14:47:52.661667, event: all_read, op: osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.661775 7f9d5a16f700  5 -- op tracker -- , seq: 748499, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.661781 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:52.661806 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:52.661814 7f9d5a16f700 15 osd.2 173 enqueue_op 0x12bf6400 prio 127 cost 516903 latency 0.001574 osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.661861 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:52.661872 7f9d663dd700 10 osd.2 173 dequeue_op 0x12bf6400 prio 127 cost 516903 latency 0.001631 osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2475 (0'0,173'2475] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2474 active]
2014-06-12 14:47:52.661892 7f9d663dd700  5 -- op tracker -- , seq: 748499, time: 2014-06-12 14:47:52.661889, event: reached_pg, op: osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.661903 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2475 (0'0,173'2475] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2474 active] handle_message: 0x12bf6400
2014-06-12 14:47:52.661913 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2475 (0'0,173'2475] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2474 active] sub_op_modify trans 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 v 173'2476 (transaction) 174
2014-06-12 14:47:52.661926 7f9d663dd700  5 -- op tracker -- , seq: 748499, time: 2014-06-12 14:47:52.661923, event: started, op: osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.661946 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2475 (0'0,173'2475] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2474 active] append_log log((0'0,173'2475], crt=0'0) [173'2476 (173'2475) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144678 2014-06-12 08:44:59.141899]
2014-06-12 14:47:52.661965 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2476 (0'0,173'2476] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2474 active] add_log_entry 173'2476 (173'2475) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144678 2014-06-12 08:44:59.141899
2014-06-12 14:47:52.661982 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2476 (0'0,173'2476] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2474 active] append_log  adding 1 keys
2014-06-12 14:47:52.662011 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2476, trimmed: 
2014-06-12 14:47:52.662036 7f9d663dd700  5 -- op tracker -- , seq: 748499, time: 2014-06-12 14:47:52.662025, event: started, op: osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.662044 7f9d663dd700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:52.662055 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:52.662057 7f9d663dd700  5 queue_op 0x2a08b40 seq 2309 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:52.662068 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:47:52.662062 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2476 (0'0,173'2476] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2474 active] do_sub_op osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.662075 7f9d83173700  5 _do_op 0x2a08b40 seq 2309 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:47:52.662076 7f9d663dd700 10 osd.2 173 dequeue_op 0x12bf6400 finish
2014-06-12 14:47:52.662079 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:52.662081 7f9d83173700 10 _do_transaction on 0x28411eb0
2014-06-12 14:47:52.662082 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:52.662086 7f9d83173700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:52.662196 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:52.662228 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:52.662261 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:52.662371 7f9d83173700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:52.662377 7f9d83173700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:47:52.662381 7f9d83173700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:52.662394 7f9d83173700 15 _write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 1507328~516096
2014-06-12 14:47:52.662539 7f9d83173700 10 lookup_strip_header done  cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:52.662757 7f9d83173700 10 file_to_extents done 
2014-06-12 14:47:52.664098 7f9d83173700 10 _generic_write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 1507328~516096 = 0
2014-06-12 14:47:52.664139 7f9d83173700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:52.664150 7f9d83173700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:52.664154 7f9d83173700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:52.664159 7f9d83173700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:52.664305 7f9d83173700 20 set_header setting 17353 cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 parent seq 0
2014-06-12 14:47:52.664365 7f9d83173700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:47:52.664376 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:52.664381 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:47:52.664833 7f9d5a16f700  5 -- op tracker -- , seq: 748500, time: 2014-06-12 14:47:52.663563, event: header_read, op: osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.664847 7f9d5a16f700  5 -- op tracker -- , seq: 748500, time: 2014-06-12 14:47:52.663565, event: throttled, op: osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.664860 7f9d5a16f700  5 -- op tracker -- , seq: 748500, time: 2014-06-12 14:47:52.664797, event: all_read, op: osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.664872 7f9d5a16f700  5 -- op tracker -- , seq: 748500, time: 0.000000, event: dispatched, op: osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.664878 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:52.664886 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:52.664895 7f9d5a16f700 15 osd.2 173 enqueue_op 0x12bf7900 prio 127 cost 516903 latency 0.001332 osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.664929 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:52.664940 7f9d65bdc700 10 osd.2 173 dequeue_op 0x12bf7900 prio 127 cost 516903 latency 0.001377 osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2694 (0'0,173'2694] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2693 active]
2014-06-12 14:47:52.664966 7f9d65bdc700  5 -- op tracker -- , seq: 748500, time: 2014-06-12 14:47:52.664961, event: reached_pg, op: osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.664975 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2694 (0'0,173'2694] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2693 active] handle_message: 0x12bf7900
2014-06-12 14:47:52.664986 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2694 (0'0,173'2694] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2693 active] sub_op_modify trans c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 v 173'2695 (transaction) 174
2014-06-12 14:47:52.665001 7f9d65bdc700  5 -- op tracker -- , seq: 748500, time: 2014-06-12 14:47:52.664998, event: started, op: osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.665022 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2694 (0'0,173'2694] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2693 active] append_log log((0'0,173'2694], crt=0'0) [173'2695 (173'2692) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141255 2014-06-12 08:45:27.521688]
2014-06-12 14:47:52.665046 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2695 (0'0,173'2695] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2693 active] add_log_entry 173'2695 (173'2692) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141255 2014-06-12 08:45:27.521688
2014-06-12 14:47:52.665073 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2695 (0'0,173'2695] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2693 active] append_log  adding 1 keys
2014-06-12 14:47:52.665103 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2695, trimmed: 
2014-06-12 14:47:52.665131 7f9d65bdc700  5 -- op tracker -- , seq: 748500, time: 2014-06-12 14:47:52.665126, event: started, op: osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.665153 7f9d65bdc700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:52.665160 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:52.665162 7f9d65bdc700  5 queue_op 0x1fb3f080 seq 2505 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 2 ops and 1036778 bytes)
2014-06-12 14:47:52.665172 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2695 (0'0,173'2695] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2693 active] do_sub_op osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.665191 7f9d65bdc700 10 osd.2 173 dequeue_op 0x12bf7900 finish
2014-06-12 14:47:52.665196 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:52.665199 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:52.665200 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (2 active)
2014-06-12 14:47:52.665207 7f9d7a161700  5 _do_op 0x1fb3f080 seq 2505 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:52.665211 7f9d7a161700 10 _do_transaction on 0x1e78b7b0
2014-06-12 14:47:52.665216 7f9d7a161700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.665326 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.665350 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:52.666985 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:52.667060 7f9d83173700 10 _do_op 0x2a08b40 seq 2309 r = 0, finisher 0x59567a0 0
2014-06-12 14:47:52.667078 7f9d7a161700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.667088 7f9d7a161700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:52.667096 7f9d7a161700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.667099 7f9d83173700 10 _finish_op 0x2a08b40 seq 2309 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:52.667094 7f9d7715b700  5 -- op tracker -- , seq: 748499, time: 2014-06-12 14:47:52.667084, event: commit_sent, op: osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.667116 7f9d7a161700 15 _write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 2609152~516096
2014-06-12 14:47:52.667119 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2476 (0'0,173'2476] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2474 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:52.667143 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:52.667177 7f9d7795c700  5 -- op tracker -- , seq: 748499, time: 2014-06-12 14:47:52.667172, event: sub_op_applied, op: osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.667188 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2476 (0'0,173'2476] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2475 active] sub_op_modify_applied on 0x28411dc0 op osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.667202 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2476 (0'0,173'2476] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2475 active] op_applied on primary on version 173'2476
2014-06-12 14:47:52.667210 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2476 (0'0,173'2476] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2475 active] op_applied on replica on version 173'2476
2014-06-12 14:47:52.667249 7f9d83173700  5 -- op tracker -- , seq: 748499, time: 2014-06-12 14:47:52.667240, event: done, op: osd_sub_op(client.4303.0:144678 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2476 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.667264 7f9d7a161700 10 lookup_strip_header done  cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:52.667277 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (1 active)
2014-06-12 14:47:52.667281 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:52.667445 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:47:52.668626 7f9d7a161700 10 _generic_write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 2609152~516096 = 0
2014-06-12 14:47:52.668673 7f9d7a161700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:52.668692 7f9d7a161700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:52.668700 7f9d7a161700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:52.668709 7f9d7a161700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:52.668884 7f9d7a161700 20 set_header setting 17923 cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 parent seq 0
2014-06-12 14:47:52.669009 7f9d7a161700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:52.669025 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:52.669047 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:47:52.671734 7f9d7a161700 10 _do_op 0x1fb3f080 seq 2505 r = 0, finisher 0x8894a60 0
2014-06-12 14:47:52.671752 7f9d7a161700 10 _finish_op 0x1fb3f080 seq 2505 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:52.671761 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:52.671773 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:52.671774 7f9d7715b700  5 -- op tracker -- , seq: 748500, time: 2014-06-12 14:47:52.671762, event: commit_sent, op: osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.671788 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2695 (0'0,173'2695] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2693 active] sub_op_modify_commit on op osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:52.671818 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:52.671903 7f9d7795c700  5 -- op tracker -- , seq: 748500, time: 2014-06-12 14:47:52.671897, event: sub_op_applied, op: osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.671913 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2695 (0'0,173'2695] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2694 active] sub_op_modify_applied on 0x1e78b6c0 op osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.671931 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2695 (0'0,173'2695] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2694 active] op_applied on primary on version 173'2695
2014-06-12 14:47:52.671941 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2695 (0'0,173'2695] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2694 active] op_applied on replica on version 173'2695
2014-06-12 14:47:52.671963 7f9d7795c700  5 -- op tracker -- , seq: 748500, time: 2014-06-12 14:47:52.671959, event: done, op: osd_sub_op(client.4287.0:141255 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2695 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.722113 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:52.722174 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:52.803749 7f9d5a16f700  5 -- op tracker -- , seq: 748501, time: 2014-06-12 14:47:52.802979, event: header_read, op: osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.803779 7f9d5a16f700  5 -- op tracker -- , seq: 748501, time: 2014-06-12 14:47:52.802982, event: throttled, op: osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.803822 7f9d5a16f700  5 -- op tracker -- , seq: 748501, time: 2014-06-12 14:47:52.803682, event: all_read, op: osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.803841 7f9d5a16f700  5 -- op tracker -- , seq: 748501, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.803851 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:52.803864 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:52.803880 7f9d5a16f700 15 osd.2 173 enqueue_op 0x1e3eb000 prio 127 cost 328487 latency 0.000900 osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.803965 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:52.803987 7f9d62bd6700 10 osd.2 173 dequeue_op 0x1e3eb000 prio 127 cost 328487 latency 0.001008 osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2695 (0'0,173'2695] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2694 active]
2014-06-12 14:47:52.804048 7f9d62bd6700  5 -- op tracker -- , seq: 748501, time: 2014-06-12 14:47:52.804043, event: reached_pg, op: osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.804058 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2695 (0'0,173'2695] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2694 active] handle_message: 0x1e3eb000
2014-06-12 14:47:52.804069 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2695 (0'0,173'2695] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2694 active] sub_op_modify trans 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 v 173'2696 (transaction) 174
2014-06-12 14:47:52.804080 7f9d62bd6700  5 -- op tracker -- , seq: 748501, time: 2014-06-12 14:47:52.804077, event: started, op: osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.804092 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2695 (0'0,173'2695] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2694 active] append_log log((0'0,173'2695], crt=0'0) [173'2696 (173'2693) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142263 2014-06-12 08:44:04.767624]
2014-06-12 14:47:52.804109 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2696 (0'0,173'2696] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2694 active] add_log_entry 173'2696 (173'2693) modify   3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 by client.4305.0:142263 2014-06-12 08:44:04.767624
2014-06-12 14:47:52.804136 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2696 (0'0,173'2696] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2694 active] append_log  adding 1 keys
2014-06-12 14:47:52.804160 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2696, trimmed: 
2014-06-12 14:47:52.804182 7f9d62bd6700  5 -- op tracker -- , seq: 748501, time: 2014-06-12 14:47:52.804179, event: started, op: osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.804191 7f9d62bd6700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:52.804194 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:52.804196 7f9d62bd6700  5 queue_op 0x24c66a20 seq 2506 osr(3.1b7 0x276d1a0) 329973 bytes   (queue has 1 ops and 329973 bytes)
2014-06-12 14:47:52.804207 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:52.804203 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2696 (0'0,173'2696] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2694 active] do_sub_op osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.804216 7f9d7f16b700  5 _do_op 0x24c66a20 seq 2506 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:52.804218 7f9d62bd6700 10 osd.2 173 dequeue_op 0x1e3eb000 finish
2014-06-12 14:47:52.804222 7f9d7f16b700 10 _do_transaction on 0x5a429b0
2014-06-12 14:47:52.804222 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:52.804225 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:52.804229 7f9d7f16b700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.804421 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.804479 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:52.804550 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:52.804773 7f9d7f16b700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.804786 7f9d7f16b700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:52.804795 7f9d7f16b700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.804818 7f9d7f16b700 15 _write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 3866624~327680
2014-06-12 14:47:52.805012 7f9d7f16b700 10 lookup_strip_header done  cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:52.805373 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:47:52.806713 7f9d5a16f700  5 -- op tracker -- , seq: 748502, time: 2014-06-12 14:47:52.806609, event: header_read, op: osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.806727 7f9d7f16b700 10 _generic_write 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 3866624~327680 = 0
2014-06-12 14:47:52.806732 7f9d5a16f700  5 -- op tracker -- , seq: 748502, time: 2014-06-12 14:47:52.806611, event: throttled, op: osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.806743 7f9d5a16f700  5 -- op tracker -- , seq: 748502, time: 2014-06-12 14:47:52.806664, event: all_read, op: osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.806754 7f9d5a16f700  5 -- op tracker -- , seq: 748502, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.806761 7f9d7f16b700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:52.806760 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:52.806769 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:52.806777 7f9d7f16b700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:52.806778 7f9d5a16f700 15 osd.2 173 enqueue_op 0x6f4b100 prio 127 cost 4903 latency 0.000168 osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.806784 7f9d7f16b700 15 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3
2014-06-12 14:47:52.806804 7f9d7f16b700 10 _setattrs 3.1b7_head/3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 = 0
2014-06-12 14:47:52.806827 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:52.806834 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x6f4b100 prio 127 cost 4903 latency 0.000225 osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2886 (0'0,173'2886] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2885 active]
2014-06-12 14:47:52.806859 7f9d6b3e7700  5 -- op tracker -- , seq: 748502, time: 2014-06-12 14:47:52.806854, event: reached_pg, op: osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.806868 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2886 (0'0,173'2886] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2885 active] handle_message: 0x6f4b100
2014-06-12 14:47:52.806879 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2886 (0'0,173'2886] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2885 active] sub_op_modify trans a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 v 173'2887 (transaction) 174
2014-06-12 14:47:52.806897 7f9d6b3e7700  5 -- op tracker -- , seq: 748502, time: 2014-06-12 14:47:52.806892, event: started, op: osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.806910 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2886 (0'0,173'2886] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2885 active] append_log log((0'0,173'2886], crt=0'0) [173'2887 (173'2886) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146511 2014-06-12 08:45:40.531194]
2014-06-12 14:47:52.806930 7f9d7f16b700 20 set_header setting 16305 cid 3.1b7_head oid 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 parent seq 0
2014-06-12 14:47:52.806933 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2887 (0'0,173'2887] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2885 active] add_log_entry 173'2887 (173'2886) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146511 2014-06-12 08:45:40.531194
2014-06-12 14:47:52.806963 7f9d7f16b700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:52.806955 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2887 (0'0,173'2887] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2885 active] append_log  adding 1 keys
2014-06-12 14:47:52.806971 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:52.806976 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:47:52.806988 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2887, trimmed: 
2014-06-12 14:47:52.807011 7f9d6b3e7700  5 -- op tracker -- , seq: 748502, time: 2014-06-12 14:47:52.807006, event: started, op: osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.807020 7f9d6b3e7700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:52.807023 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:52.807025 7f9d6b3e7700  5 queue_op 0x16a788a0 seq 2600 osr(3.129 0x276cf20) 6389 bytes   (queue has 2 ops and 336362 bytes)
2014-06-12 14:47:52.807032 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2887 (0'0,173'2887] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2885 active] do_sub_op osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.807038 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (2 active)
2014-06-12 14:47:52.807047 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x6f4b100 finish
2014-06-12 14:47:52.807050 7f9d7e169700  5 _do_op 0x16a788a0 seq 2600 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:52.807051 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:52.807054 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:52.807055 7f9d7e169700 10 _do_transaction on 0x5cb5970
2014-06-12 14:47:52.807061 7f9d7e169700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:52.807142 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:52.807159 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:52.809179 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:52.809216 7f9d7e169700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:52.809220 7f9d7e169700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:52.809220 7f9d7f16b700 10 _do_op 0x24c66a20 seq 2506 r = 0, finisher 0x8895720 0
2014-06-12 14:47:52.809224 7f9d7e169700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:52.809238 7f9d7e169700 15 _write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 3948544~4096
2014-06-12 14:47:52.809318 7f9d7715b700  5 -- op tracker -- , seq: 748501, time: 2014-06-12 14:47:52.809310, event: commit_sent, op: osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.809336 7f9d7e169700 10 lookup_strip_header done  cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:52.809328 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2696 (0'0,173'2696] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2694 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:52.809370 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:52.809468 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:47:52.809490 7f9d7e169700 10 _generic_write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 3948544~4096 = 0
2014-06-12 14:47:52.809500 7f9d7e169700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:52.809512 7f9d7e169700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:52.809517 7f9d7e169700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:52.809524 7f9d7e169700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:52.809514 7f9d7f16b700 10 _finish_op 0x24c66a20 seq 2506 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:52.809584 7f9d7795c700  5 -- op tracker -- , seq: 748501, time: 2014-06-12 14:47:52.809578, event: sub_op_applied, op: osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.809610 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2696 (0'0,173'2696] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active] sub_op_modify_applied on 0x5a428c0 op osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.809632 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2696 (0'0,173'2696] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active] op_applied on primary on version 173'2696
2014-06-12 14:47:52.809639 7f9d7e169700 20 set_header setting 17398 cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 parent seq 0
2014-06-12 14:47:52.809686 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:52.809642 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2696 (0'0,173'2696] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active] op_applied on replica on version 173'2696
2014-06-12 14:47:52.809711 7f9d7e169700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:52.809733 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:47:52.809769 7f9d7f16b700  5 -- op tracker -- , seq: 748501, time: 2014-06-12 14:47:52.809726, event: done, op: osd_sub_op(client.4305.0:142263 3.1b7 3dc659b7/rbd_data.10a86b8b4567.0000000000000ec2/head//3 [] v 173'2696 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.809840 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (1 active)
2014-06-12 14:47:52.809848 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:52.810049 7f9d7e169700 10 _do_op 0x16a788a0 seq 2600 r = 0, finisher 0x7eb8a40 0
2014-06-12 14:47:52.810068 7f9d7e169700 10 _finish_op 0x16a788a0 seq 2600 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:52.810081 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:52.810087 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:52.810109 7f9d7715b700  5 -- op tracker -- , seq: 748502, time: 2014-06-12 14:47:52.810094, event: commit_sent, op: osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.810124 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2887 (0'0,173'2887] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2885 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:52.810157 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:52.811816 7f9d7795c700  5 -- op tracker -- , seq: 748502, time: 2014-06-12 14:47:52.811798, event: sub_op_applied, op: osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.811839 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2887 (0'0,173'2887] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active] sub_op_modify_applied on 0x5cb5880 op osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.811888 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2887 (0'0,173'2887] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active] op_applied on primary on version 173'2887
2014-06-12 14:47:52.811910 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2887 (0'0,173'2887] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active] op_applied on replica on version 173'2887
2014-06-12 14:47:52.811960 7f9d7795c700  5 -- op tracker -- , seq: 748502, time: 2014-06-12 14:47:52.811950, event: done, op: osd_sub_op(client.4313.0:146511 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2887 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.865960 7f9d5a16f700  5 -- op tracker -- , seq: 748503, time: 2014-06-12 14:47:52.864924, event: header_read, op: osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.865989 7f9d5a16f700  5 -- op tracker -- , seq: 748503, time: 2014-06-12 14:47:52.864944, event: throttled, op: osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.865994 7f9d5a16f700  5 -- op tracker -- , seq: 748503, time: 2014-06-12 14:47:52.865903, event: all_read, op: osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.865999 7f9d5a16f700  5 -- op tracker -- , seq: 748503, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.866004 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:52.866010 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:52.866017 7f9d5a16f700 15 osd.2 173 enqueue_op 0x876f000 prio 127 cost 516903 latency 0.001092 osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.866078 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:52.866086 7f9d693e3700 10 osd.2 173 dequeue_op 0x876f000 prio 127 cost 516903 latency 0.001162 osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2696 (0'0,173'2696] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active]
2014-06-12 14:47:52.866117 7f9d693e3700  5 -- op tracker -- , seq: 748503, time: 2014-06-12 14:47:52.866106, event: reached_pg, op: osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.866134 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2696 (0'0,173'2696] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active] handle_message: 0x876f000
2014-06-12 14:47:52.866143 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2696 (0'0,173'2696] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active] sub_op_modify trans 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 v 173'2697 (transaction) 174
2014-06-12 14:47:52.866153 7f9d693e3700  5 -- op tracker -- , seq: 748503, time: 2014-06-12 14:47:52.866151, event: started, op: osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.866166 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2696 (0'0,173'2696] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active] append_log log((0'0,173'2696], crt=0'0) [173'2697 (173'2694) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141504 2014-06-12 08:45:27.353248]
2014-06-12 14:47:52.866182 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2697 (0'0,173'2697] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active] add_log_entry 173'2697 (173'2694) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141504 2014-06-12 08:45:27.353248
2014-06-12 14:47:52.866197 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2697 (0'0,173'2697] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active] append_log  adding 1 keys
2014-06-12 14:47:52.866222 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2697, trimmed: 
2014-06-12 14:47:52.866255 7f9d693e3700  5 -- op tracker -- , seq: 748503, time: 2014-06-12 14:47:52.866252, event: started, op: osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.866263 7f9d693e3700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:52.866266 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:52.866267 7f9d693e3700  5 queue_op 0x83ef1a0 seq 2507 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:52.866272 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2697 (0'0,173'2697] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active] do_sub_op osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.866281 7f9d693e3700 10 osd.2 173 dequeue_op 0x876f000 finish
2014-06-12 14:47:52.866292 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:52.866293 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:52.866304 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:52.866312 7f9d7815d700  5 _do_op 0x83ef1a0 seq 2507 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:52.866324 7f9d7815d700 10 _do_transaction on 0x26838470
2014-06-12 14:47:52.866329 7f9d7815d700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.866431 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.866447 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:52.866487 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:52.866582 7f9d7815d700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.866588 7f9d7815d700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:52.866591 7f9d7815d700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:52.866612 7f9d7815d700 15 _write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 1695744~516096
2014-06-12 14:47:52.866706 7f9d7815d700 10 lookup_strip_header done  cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:52.866945 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:47:52.868270 7f9d7815d700 10 _generic_write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 1695744~516096 = 0
2014-06-12 14:47:52.868310 7f9d7815d700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:52.868321 7f9d7815d700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:52.868324 7f9d7815d700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:52.868329 7f9d7815d700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:52.868458 7f9d7815d700 20 set_header setting 17393 cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 parent seq 0
2014-06-12 14:47:52.868507 7f9d7815d700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:52.868520 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:52.868528 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:47:52.870175 7f9d7815d700 10 _do_op 0x83ef1a0 seq 2507 r = 0, finisher 0x70c62c0 0
2014-06-12 14:47:52.870193 7f9d7815d700 10 _finish_op 0x83ef1a0 seq 2507 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:52.870198 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:52.870201 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:52.870231 7f9d7795c700  5 -- op tracker -- , seq: 748503, time: 2014-06-12 14:47:52.870219, event: sub_op_applied, op: osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.870242 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2697 (0'0,173'2697] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active] sub_op_modify_applied on 0x26838380 op osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.870276 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:52.870297 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2697 (0'0,173'2697] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active] op_applied on primary on version 173'2697
2014-06-12 14:47:52.870317 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2697 (0'0,173'2697] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active] op_applied on replica on version 173'2697
2014-06-12 14:47:52.870359 7f9d7715b700  5 -- op tracker -- , seq: 748503, time: 2014-06-12 14:47:52.870352, event: commit_sent, op: osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.870377 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2697 (0'0,173'2697] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2695 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:52.870400 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:52.870440 7f9d7715b700  5 -- op tracker -- , seq: 748503, time: 2014-06-12 14:47:52.870427, event: done, op: osd_sub_op(client.4284.0:141504 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2697 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.913277 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:52.913327 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:52.985790 7f9d5a16f700  5 -- op tracker -- , seq: 748504, time: 2014-06-12 14:47:52.978302, event: header_read, op: osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.985819 7f9d5a16f700  5 -- op tracker -- , seq: 748504, time: 2014-06-12 14:47:52.978305, event: throttled, op: osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.985839 7f9d5a16f700  5 -- op tracker -- , seq: 748504, time: 2014-06-12 14:47:52.985705, event: all_read, op: osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.985858 7f9d5a16f700  5 -- op tracker -- , seq: 748504, time: 0.000000, event: dispatched, op: osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.985867 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:52.985882 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:52.985897 7f9d5a16f700 15 osd.2 173 enqueue_op 0x55c3c00 prio 127 cost 3121955 latency 0.007595 osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.985976 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:52.986000 7f9d683e1700 10 osd.2 173 dequeue_op 0x55c3c00 prio 127 cost 3121955 latency 0.007697 osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.f9( v 173'2729 (0'0,173'2729] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active]
2014-06-12 14:47:52.986040 7f9d683e1700  5 -- op tracker -- , seq: 748504, time: 2014-06-12 14:47:52.986033, event: reached_pg, op: osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.986057 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2729 (0'0,173'2729] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] handle_message: 0x55c3c00
2014-06-12 14:47:52.986072 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2729 (0'0,173'2729] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] sub_op_modify trans 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 v 173'2730 (transaction) 174
2014-06-12 14:47:52.986094 7f9d683e1700  5 -- op tracker -- , seq: 748504, time: 2014-06-12 14:47:52.986088, event: started, op: osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.986114 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2729 (0'0,173'2729] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] append_log log((0'0,173'2729], crt=0'0) [173'2730 (173'2729) modify   5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 by client.4295.0:143161 2014-06-12 08:44:44.245992]
2014-06-12 14:47:52.986154 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2730 (0'0,173'2730] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] add_log_entry 173'2730 (173'2729) modify   5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 by client.4295.0:143161 2014-06-12 08:44:44.245992
2014-06-12 14:47:52.986189 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2730 (0'0,173'2730] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] append_log  adding 1 keys
2014-06-12 14:47:52.986230 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2730, trimmed: 
2014-06-12 14:47:52.986263 7f9d683e1700  5 -- op tracker -- , seq: 748504, time: 2014-06-12 14:47:52.986256, event: started, op: osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:52.986276 7f9d683e1700  5 queue_transactions existing osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:52.986280 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:52.986282 7f9d683e1700  5 queue_op 0x1697df20 seq 2539 osr(3.f9 0x276cfa0) 3123436 bytes   (queue has 1 ops and 3123436 bytes)
2014-06-12 14:47:52.986291 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.f9( v 173'2730 (0'0,173'2730] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] do_sub_op osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:52.986311 7f9d683e1700 10 osd.2 173 dequeue_op 0x55c3c00 finish
2014-06-12 14:47:52.986316 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:52.986320 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:52.986327 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b77a0 (1 active)
2014-06-12 14:47:52.986342 7f9d82171700  5 _do_op 0x1697df20 seq 2539 osr(3.f9 0x276cfa0)/0x276cfa0 start
2014-06-12 14:47:52.986349 7f9d82171700 10 _do_transaction on 0x678fb30
2014-06-12 14:47:52.986358 7f9d82171700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:52.986493 7f9d82171700 10 lookup_strip_header done  cid meta oid 516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:52.986539 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:52.986607 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:52.986757 7f9d82171700 15 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:52.986769 7f9d82171700 10 _omap_rmkeys meta/516e9ea6/pglog_3.f9/0//-1 r = 0
2014-06-12 14:47:52.986778 7f9d82171700 15 _omap_setkeys meta/516e9ea6/pglog_3.f9/0//-1
2014-06-12 14:47:52.986801 7f9d82171700 15 _write 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 1073152~3121152
2014-06-12 14:47:52.986984 7f9d82171700 10 lookup_strip_header done  cid 3.f9_head oid 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3
2014-06-12 14:47:52.987357 7f9d82171700 10 file_to_extents done 
2014-06-12 14:47:53.000937 7f9d82171700 10 _generic_write 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 1073152~3121152 = 0
2014-06-12 14:47:53.001306 7f9d82171700 15 _setattrs 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3
2014-06-12 14:47:53.001343 7f9d82171700 10 _setattrs 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 = 0
2014-06-12 14:47:53.001347 7f9d82171700 15 _setattrs 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3
2014-06-12 14:47:53.001353 7f9d82171700 10 _setattrs 3.f9_head/5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 = 0
2014-06-12 14:47:53.001971 7f9d82171700 20 set_header setting 16838 cid 3.f9_head oid 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 parent seq 0
2014-06-12 14:47:53.002181 7f9d82171700 20 set_header setting 2218 cid meta oid 516e9ea6/pglog_3.f9/0//-1 parent seq 0
2014-06-12 14:47:53.002196 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:53.002201 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:47:53.010134 7f9d5a16f700  5 -- op tracker -- , seq: 748505, time: 2014-06-12 14:47:53.008502, event: header_read, op: osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.010145 7f9d5a16f700  5 -- op tracker -- , seq: 748505, time: 2014-06-12 14:47:53.008505, event: throttled, op: osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.010158 7f9d5a16f700  5 -- op tracker -- , seq: 748505, time: 2014-06-12 14:47:53.010090, event: all_read, op: osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.010163 7f9d5a16f700  5 -- op tracker -- , seq: 748505, time: 0.000000, event: dispatched, op: osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.010167 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:53.010171 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:53.010176 7f9d5a16f700 15 osd.2 173 enqueue_op 0x3b42b100 prio 127 cost 1032999 latency 0.001674 osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.010239 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:53.010249 7f9d6abe6700 10 osd.2 173 dequeue_op 0x3b42b100 prio 127 cost 1032999 latency 0.001747 osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2697 (0'0,173'2697] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2696 active]
2014-06-12 14:47:53.010270 7f9d6abe6700  5 -- op tracker -- , seq: 748505, time: 2014-06-12 14:47:53.010267, event: reached_pg, op: osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.010280 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2697 (0'0,173'2697] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2696 active] handle_message: 0x3b42b100
2014-06-12 14:47:53.010290 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2697 (0'0,173'2697] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2696 active] sub_op_modify trans c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 v 173'2698 (transaction) 174
2014-06-12 14:47:53.010302 7f9d6abe6700  5 -- op tracker -- , seq: 748505, time: 2014-06-12 14:47:53.010299, event: started, op: osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.010319 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2697 (0'0,173'2697] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2696 active] append_log log((0'0,173'2697], crt=0'0) [173'2698 (173'2695) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141256 2014-06-12 08:45:27.652266]
2014-06-12 14:47:53.010338 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2698 (0'0,173'2698] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2696 active] add_log_entry 173'2698 (173'2695) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141256 2014-06-12 08:45:27.652266
2014-06-12 14:47:53.010361 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2698 (0'0,173'2698] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2696 active] append_log  adding 1 keys
2014-06-12 14:47:53.010386 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2698, trimmed: 
2014-06-12 14:47:53.010405 7f9d6abe6700  5 -- op tracker -- , seq: 748505, time: 2014-06-12 14:47:53.010402, event: started, op: osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.010414 7f9d6abe6700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:53.010417 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:53.010418 7f9d6abe6700  5 queue_op 0x5e6c300 seq 2508 osr(3.1b7 0x276d1a0) 1034485 bytes   (queue has 2 ops and 4157921 bytes)
2014-06-12 14:47:53.010423 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2698 (0'0,173'2698] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2696 active] do_sub_op osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.010433 7f9d6abe6700 10 osd.2 173 dequeue_op 0x3b42b100 finish
2014-06-12 14:47:53.010435 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:53.010436 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:53.010443 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (2 active)
2014-06-12 14:47:53.010451 7f9d7e96a700  5 _do_op 0x5e6c300 seq 2508 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:53.010454 7f9d7e96a700 10 _do_transaction on 0x5732b70
2014-06-12 14:47:53.010460 7f9d7e96a700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.010563 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.010584 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:53.012775 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:53.012800 7f9d7e96a700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.012807 7f9d7e96a700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:53.012811 7f9d7e96a700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.012906 7f9d7e96a700 15 _write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 3125248~1032192
2014-06-12 14:47:53.013003 7f9d7e96a700 10 lookup_strip_header done  cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:53.013120 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:47:53.013367 7f9d82171700 10 _do_op 0x1697df20 seq 2539 r = 0, finisher 0x27656740 0
2014-06-12 14:47:53.013378 7f9d82171700 10 _finish_op 0x1697df20 seq 2539 osr(3.f9 0x276cfa0)/0x276cfa0
2014-06-12 14:47:53.013383 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b77a0 (1 active)
2014-06-12 14:47:53.013388 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:53.013402 7f9d7795c700  5 -- op tracker -- , seq: 748504, time: 2014-06-12 14:47:53.013394, event: sub_op_applied, op: osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.013411 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2730 (0'0,173'2730] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] sub_op_modify_applied on 0x678fa40 op osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.013452 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.013478 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2730 (0'0,173'2730] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] op_applied on primary on version 173'2730
2014-06-12 14:47:53.013495 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2730 (0'0,173'2730] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] op_applied on replica on version 173'2730
2014-06-12 14:47:53.013543 7f9d7715b700  5 -- op tracker -- , seq: 748504, time: 2014-06-12 14:47:53.013526, event: commit_sent, op: osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.013575 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.f9( v 173'2730 (0'0,173'2730] local-les=147 n=306 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] sub_op_modify_commit on op osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:53.013637 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.013699 7f9d7715b700  5 -- op tracker -- , seq: 748504, time: 2014-06-12 14:47:53.013688, event: done, op: osd_sub_op(client.4295.0:143161 3.f9 5a1560f9/rbd_data.109e6b8b4567.0000000000000fab/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.017102 7f9d7e96a700 10 _generic_write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 3125248~1032192 = 0
2014-06-12 14:47:53.017204 7f9d7e96a700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:53.017227 7f9d7e96a700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:53.017240 7f9d7e96a700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:53.017253 7f9d7e96a700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:53.017626 7f9d7e96a700 20 set_header setting 17923 cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 parent seq 0
2014-06-12 14:47:53.017784 7f9d7e96a700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:53.017808 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:53.017824 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:47:53.023811 7f9d7e96a700 10 _do_op 0x5e6c300 seq 2508 r = 0, finisher 0x6de9dc0 0
2014-06-12 14:47:53.023836 7f9d7e96a700 10 _finish_op 0x5e6c300 seq 2508 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:53.023842 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:53.023856 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:53.023885 7f9d7715b700  5 -- op tracker -- , seq: 748505, time: 2014-06-12 14:47:53.023874, event: commit_sent, op: osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.023905 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2698 (0'0,173'2698] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2696 active] sub_op_modify_commit on op osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:53.023955 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.024035 7f9d7795c700  5 -- op tracker -- , seq: 748505, time: 2014-06-12 14:47:53.024023, event: sub_op_applied, op: osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.024056 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2698 (0'0,173'2698] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2697 active] sub_op_modify_applied on 0x5732a80 op osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.024079 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2698 (0'0,173'2698] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2697 active] op_applied on primary on version 173'2698
2014-06-12 14:47:53.024094 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2698 (0'0,173'2698] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2697 active] op_applied on replica on version 173'2698
2014-06-12 14:47:53.024107 7f9d7795c700  5 -- op tracker -- , seq: 748505, time: 2014-06-12 14:47:53.024105, event: done, op: osd_sub_op(client.4287.0:141256 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2698 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.029781 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:53.030133 7f9d5a16f700  5 -- op tracker -- , seq: 748506, time: 2014-06-12 14:47:53.029168, event: header_read, op: osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.030147 7f9d5a16f700  5 -- op tracker -- , seq: 748506, time: 2014-06-12 14:47:53.029171, event: throttled, op: osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.030155 7f9d5a16f700  5 -- op tracker -- , seq: 748506, time: 2014-06-12 14:47:53.030096, event: all_read, op: osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.030159 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:53.030163 7f9d5a16f700  5 -- op tracker -- , seq: 748506, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.030169 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:53.030184 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:53.030205 7f9d5a16f700 15 osd.2 173 enqueue_op 0x8dbdb00 prio 127 cost 516903 latency 0.001037 osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.030267 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:53.030279 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x8dbdb00 prio 127 cost 516903 latency 0.001110 osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2476 (0'0,173'2476] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2475 active]
2014-06-12 14:47:53.030307 7f9d6a3e5700  5 -- op tracker -- , seq: 748506, time: 2014-06-12 14:47:53.030302, event: reached_pg, op: osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.030324 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2476 (0'0,173'2476] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2475 active] handle_message: 0x8dbdb00
2014-06-12 14:47:53.030358 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2476 (0'0,173'2476] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2475 active] sub_op_modify trans 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 v 173'2477 (transaction) 174
2014-06-12 14:47:53.030391 7f9d6a3e5700  5 -- op tracker -- , seq: 748506, time: 2014-06-12 14:47:53.030383, event: started, op: osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.030424 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2476 (0'0,173'2476] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2475 active] append_log log((0'0,173'2476], crt=0'0) [173'2477 (173'2476) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144679 2014-06-12 08:44:59.630651]
2014-06-12 14:47:53.030462 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2477 (0'0,173'2477] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2475 active] add_log_entry 173'2477 (173'2476) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144679 2014-06-12 08:44:59.630651
2014-06-12 14:47:53.030486 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2477 (0'0,173'2477] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2475 active] append_log  adding 1 keys
2014-06-12 14:47:53.030526 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2477, trimmed: 
2014-06-12 14:47:53.030553 7f9d6a3e5700  5 -- op tracker -- , seq: 748506, time: 2014-06-12 14:47:53.030549, event: started, op: osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.030583 7f9d6a3e5700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:53.030590 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:53.030593 7f9d6a3e5700  5 queue_op 0x67e4180 seq 2310 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:53.030606 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2477 (0'0,173'2477] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2475 active] do_sub_op osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.030642 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x8dbdb00 finish
2014-06-12 14:47:53.030649 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:53.030654 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:53.030703 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:47:53.030730 7f9d8116f700  5 _do_op 0x67e4180 seq 2310 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:47:53.030736 7f9d8116f700 10 _do_transaction on 0x12c09430
2014-06-12 14:47:53.030745 7f9d8116f700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:53.030880 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:53.030909 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:53.030972 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:53.031080 7f9d8116f700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:53.031088 7f9d8116f700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:47:53.031101 7f9d8116f700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:53.031121 7f9d8116f700 15 _write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 2023424~516096
2014-06-12 14:47:53.031262 7f9d8116f700 10 lookup_strip_header done  cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:53.031416 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:47:53.032936 7f9d8116f700 10 _generic_write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 2023424~516096 = 0
2014-06-12 14:47:53.032977 7f9d8116f700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:53.032988 7f9d8116f700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:53.032992 7f9d8116f700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:53.032997 7f9d8116f700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:53.033129 7f9d8116f700 20 set_header setting 17353 cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 parent seq 0
2014-06-12 14:47:53.033172 7f9d8116f700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:47:53.033182 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:53.033188 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:47:53.035133 7f9d8116f700 10 _do_op 0x67e4180 seq 2310 r = 0, finisher 0x4efc220 0
2014-06-12 14:47:53.035158 7f9d8116f700 10 _finish_op 0x67e4180 seq 2310 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:53.035165 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:47:53.035176 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:53.035181 7f9d7715b700  5 -- op tracker -- , seq: 748506, time: 2014-06-12 14:47:53.035169, event: commit_sent, op: osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.035195 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2477 (0'0,173'2477] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2475 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:53.035228 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.035310 7f9d7795c700  5 -- op tracker -- , seq: 748506, time: 2014-06-12 14:47:53.035302, event: sub_op_applied, op: osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.035324 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2477 (0'0,173'2477] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active] sub_op_modify_applied on 0x12c09340 op osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.035348 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2477 (0'0,173'2477] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active] op_applied on primary on version 173'2477
2014-06-12 14:47:53.035357 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2477 (0'0,173'2477] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active] op_applied on replica on version 173'2477
2014-06-12 14:47:53.035385 7f9d7795c700  5 -- op tracker -- , seq: 748506, time: 2014-06-12 14:47:53.035378, event: done, op: osd_sub_op(client.4303.0:144679 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2477 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.205343 7f9d5a16f700  5 -- op tracker -- , seq: 748507, time: 2014-06-12 14:47:53.204846, event: header_read, op: osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.205355 7f9d5a16f700  5 -- op tracker -- , seq: 748507, time: 2014-06-12 14:47:53.204847, event: throttled, op: osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.205370 7f9d5a16f700  5 -- op tracker -- , seq: 748507, time: 2014-06-12 14:47:53.205283, event: all_read, op: osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.205375 7f9d5a16f700  5 -- op tracker -- , seq: 748507, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.205380 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:53.205385 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:53.205391 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a902100 prio 127 cost 242471 latency 0.000545 osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.205421 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:53.205438 7f9d66bde700 10 osd.2 173 dequeue_op 0x2a902100 prio 127 cost 242471 latency 0.000592 osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2887 (0'0,173'2887] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active]
2014-06-12 14:47:53.205466 7f9d66bde700  5 -- op tracker -- , seq: 748507, time: 2014-06-12 14:47:53.205461, event: reached_pg, op: osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.205480 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2887 (0'0,173'2887] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active] handle_message: 0x2a902100
2014-06-12 14:47:53.205494 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2887 (0'0,173'2887] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active] sub_op_modify trans a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 v 173'2888 (transaction) 174
2014-06-12 14:47:53.205513 7f9d66bde700  5 -- op tracker -- , seq: 748507, time: 2014-06-12 14:47:53.205509, event: started, op: osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.205532 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2887 (0'0,173'2887] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active] append_log log((0'0,173'2887], crt=0'0) [173'2888 (173'2887) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146512 2014-06-12 08:45:41.111295]
2014-06-12 14:47:53.205561 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2888 (0'0,173'2888] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active] add_log_entry 173'2888 (173'2887) modify   a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 by client.4313.0:146512 2014-06-12 08:45:41.111295
2014-06-12 14:47:53.205586 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2888 (0'0,173'2888] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active] append_log  adding 1 keys
2014-06-12 14:47:53.205642 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2888, trimmed: 
2014-06-12 14:47:53.205685 7f9d66bde700  5 -- op tracker -- , seq: 748507, time: 2014-06-12 14:47:53.205679, event: started, op: osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.205705 7f9d66bde700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:53.205709 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:53.205710 7f9d66bde700  5 queue_op 0x6d4dbc0 seq 2601 osr(3.129 0x276cf20) 243957 bytes   (queue has 1 ops and 243957 bytes)
2014-06-12 14:47:53.205717 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2888 (0'0,173'2888] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active] do_sub_op osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.205740 7f9d66bde700 10 osd.2 173 dequeue_op 0x2a902100 finish
2014-06-12 14:47:53.205744 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:53.205747 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:53.205749 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:47:53.205759 7f9d7d167700  5 _do_op 0x6d4dbc0 seq 2601 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:47:53.205765 7f9d7d167700 10 _do_transaction on 0x206ec2b0
2014-06-12 14:47:53.205773 7f9d7d167700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:53.205929 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:53.205951 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:53.206009 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:53.206139 7f9d7d167700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:53.206145 7f9d7d167700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:47:53.206149 7f9d7d167700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:47:53.206171 7f9d7d167700 15 _write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 3952640~241664
2014-06-12 14:47:53.206281 7f9d7d167700 10 lookup_strip_header done  cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:53.206472 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:47:53.207036 7f9d7d167700 10 _generic_write 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 3952640~241664 = 0
2014-06-12 14:47:53.207067 7f9d7d167700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:53.207080 7f9d7d167700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:53.207085 7f9d7d167700 15 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3
2014-06-12 14:47:53.207093 7f9d7d167700 10 _setattrs 3.129_head/a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 = 0
2014-06-12 14:47:53.207196 7f9d7d167700 20 set_header setting 17398 cid 3.129_head oid a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 parent seq 0
2014-06-12 14:47:53.207267 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:53.207277 7f9d7d167700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:47:53.207283 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:47:53.207993 7f9d7d167700 10 _do_op 0x6d4dbc0 seq 2601 r = 0, finisher 0x216fbac0 0
2014-06-12 14:47:53.208002 7f9d7d167700 10 _finish_op 0x6d4dbc0 seq 2601 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:47:53.208007 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:47:53.208010 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:53.208023 7f9d7795c700  5 -- op tracker -- , seq: 748507, time: 2014-06-12 14:47:53.208016, event: sub_op_applied, op: osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.208040 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2888 (0'0,173'2888] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active] sub_op_modify_applied on 0x206ec1c0 op osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.208058 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.208071 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2888 (0'0,173'2888] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active] op_applied on primary on version 173'2888
2014-06-12 14:47:53.208081 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2888 (0'0,173'2888] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active] op_applied on replica on version 173'2888
2014-06-12 14:47:53.208128 7f9d7715b700  5 -- op tracker -- , seq: 748507, time: 2014-06-12 14:47:53.208121, event: commit_sent, op: osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.208140 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2888 (0'0,173'2888] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2886 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:53.208155 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.208176 7f9d7715b700  5 -- op tracker -- , seq: 748507, time: 2014-06-12 14:47:53.208173, event: done, op: osd_sub_op(client.4313.0:146512 3.129 a221f929/rbd_data.10b06b8b4567.0000000000000fbf/head//3 [] v 173'2888 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.216160 7f9d5a16f700  5 -- op tracker -- , seq: 748508, time: 2014-06-12 14:47:53.214291, event: header_read, op: osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.216177 7f9d5a16f700  5 -- op tracker -- , seq: 748508, time: 2014-06-12 14:47:53.214293, event: throttled, op: osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.216190 7f9d5a16f700  5 -- op tracker -- , seq: 748508, time: 2014-06-12 14:47:53.216097, event: all_read, op: osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.216198 7f9d5a16f700  5 -- op tracker -- , seq: 748508, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.216204 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:53.216211 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:53.216219 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903600 prio 127 cost 1032999 latency 0.001928 osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.216253 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:53.216265 7f9d69be4700 10 osd.2 173 dequeue_op 0x2a903600 prio 127 cost 1032999 latency 0.001973 osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2698 (0'0,173'2698] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2697 active]
2014-06-12 14:47:53.216293 7f9d69be4700  5 -- op tracker -- , seq: 748508, time: 2014-06-12 14:47:53.216288, event: reached_pg, op: osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.216304 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2698 (0'0,173'2698] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2697 active] handle_message: 0x2a903600
2014-06-12 14:47:53.216317 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2698 (0'0,173'2698] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2697 active] sub_op_modify trans 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 v 173'2699 (transaction) 174
2014-06-12 14:47:53.216336 7f9d69be4700  5 -- op tracker -- , seq: 748508, time: 2014-06-12 14:47:53.216331, event: started, op: osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.216350 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2698 (0'0,173'2698] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2697 active] append_log log((0'0,173'2698], crt=0'0) [173'2699 (173'2697) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141505 2014-06-12 08:45:27.478303]
2014-06-12 14:47:53.216377 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2699 (0'0,173'2699] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2697 active] add_log_entry 173'2699 (173'2697) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141505 2014-06-12 08:45:27.478303
2014-06-12 14:47:53.216403 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2699 (0'0,173'2699] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2697 active] append_log  adding 1 keys
2014-06-12 14:47:53.216440 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2699, trimmed: 
2014-06-12 14:47:53.216466 7f9d69be4700  5 -- op tracker -- , seq: 748508, time: 2014-06-12 14:47:53.216460, event: started, op: osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.216478 7f9d69be4700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:53.216483 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:53.216484 7f9d69be4700  5 queue_op 0x45bec60 seq 2509 osr(3.1b7 0x276d1a0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:53.216492 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2699 (0'0,173'2699] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2697 active] do_sub_op osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.216507 7f9d69be4700 10 osd.2 173 dequeue_op 0x2a903600 finish
2014-06-12 14:47:53.216511 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:53.216514 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:53.216754 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:53.216767 7f9d7915f700  5 _do_op 0x45bec60 seq 2509 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:53.216774 7f9d7915f700 10 _do_transaction on 0x66770b0
2014-06-12 14:47:53.216789 7f9d7915f700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.216901 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.216930 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:53.216976 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:53.217144 7f9d7915f700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.217153 7f9d7915f700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:53.217160 7f9d7915f700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.217182 7f9d7915f700 15 _write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 2211840~1032192
2014-06-12 14:47:53.217306 7f9d7915f700 10 lookup_strip_header done  cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:53.217626 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:47:53.220486 7f9d7915f700 10 _generic_write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 2211840~1032192 = 0
2014-06-12 14:47:53.220562 7f9d7915f700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:53.220595 7f9d7915f700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:53.220603 7f9d7915f700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:53.220611 7f9d7915f700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:53.220812 7f9d7915f700 20 set_header setting 17393 cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 parent seq 0
2014-06-12 14:47:53.220892 7f9d7915f700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:53.220909 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:53.220930 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:47:53.226875 7f9d7915f700 10 _do_op 0x45bec60 seq 2509 r = 0, finisher 0x6d78840 0
2014-06-12 14:47:53.226898 7f9d7915f700 10 _finish_op 0x45bec60 seq 2509 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:53.226906 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:53.226912 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:53.226928 7f9d7715b700  5 -- op tracker -- , seq: 748508, time: 2014-06-12 14:47:53.226910, event: commit_sent, op: osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.226947 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2699 (0'0,173'2699] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2697 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:53.226998 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.227099 7f9d7795c700  5 -- op tracker -- , seq: 748508, time: 2014-06-12 14:47:53.227081, event: sub_op_applied, op: osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.227129 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2699 (0'0,173'2699] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2698 active] sub_op_modify_applied on 0x6676fc0 op osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.227166 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2699 (0'0,173'2699] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2698 active] op_applied on primary on version 173'2699
2014-06-12 14:47:53.227180 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2699 (0'0,173'2699] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2698 active] op_applied on replica on version 173'2699
2014-06-12 14:47:53.227218 7f9d7795c700  5 -- op tracker -- , seq: 748508, time: 2014-06-12 14:47:53.227212, event: done, op: osd_sub_op(client.4284.0:141505 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2699 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.236812 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:47:53.236836 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:47:53.237317 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:53.237404 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:53.237452 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:53.237493 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:53.237512 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:53.237537 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:53.237578 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:53.237595 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:53.237657 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:53.237681 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:53.237698 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:53.237721 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.237743 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.237760 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:53.237778 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:53.237797 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:53.237818 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:53.237837 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:53.237861 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:53.237881 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:53.237899 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:53.237920 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:53.267528 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:53.267589 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:53.334456 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:53.334539 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:53.381812 7f9d5a16f700  5 -- op tracker -- , seq: 748509, time: 2014-06-12 14:47:53.381496, event: header_read, op: osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.381831 7f9d5a16f700  5 -- op tracker -- , seq: 748509, time: 2014-06-12 14:47:53.381500, event: throttled, op: osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.381870 7f9d5a16f700  5 -- op tracker -- , seq: 748509, time: 2014-06-12 14:47:53.381693, event: all_read, op: osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.381878 7f9d5a16f700  5 -- op tracker -- , seq: 748509, time: 0.000000, event: dispatched, op: osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.381888 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:53.381897 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:53.381908 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903f00 prio 127 cost 37671 latency 0.000411 osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.381962 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:53.381974 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a903f00 prio 127 cost 37671 latency 0.000478 osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2699 (0'0,173'2699] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2698 active]
2014-06-12 14:47:53.382009 7f9d64bda700  5 -- op tracker -- , seq: 748509, time: 2014-06-12 14:47:53.381997, event: reached_pg, op: osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.382027 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2699 (0'0,173'2699] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2698 active] handle_message: 0x2a903f00
2014-06-12 14:47:53.382042 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2699 (0'0,173'2699] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2698 active] sub_op_modify trans c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 v 173'2700 (transaction) 174
2014-06-12 14:47:53.382057 7f9d64bda700  5 -- op tracker -- , seq: 748509, time: 2014-06-12 14:47:53.382053, event: started, op: osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.382073 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2699 (0'0,173'2699] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2698 active] append_log log((0'0,173'2699], crt=0'0) [173'2700 (173'2698) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141257 2014-06-12 08:45:27.691316]
2014-06-12 14:47:53.382115 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2700 (0'0,173'2700] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2698 active] add_log_entry 173'2700 (173'2698) modify   c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 by client.4287.0:141257 2014-06-12 08:45:27.691316
2014-06-12 14:47:53.382138 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2700 (0'0,173'2700] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2698 active] append_log  adding 1 keys
2014-06-12 14:47:53.382181 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2700, trimmed: 
2014-06-12 14:47:53.382216 7f9d64bda700  5 -- op tracker -- , seq: 748509, time: 2014-06-12 14:47:53.382211, event: started, op: osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.382231 7f9d64bda700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:53.382236 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:53.382238 7f9d64bda700  5 queue_op 0x69a13e0 seq 2510 osr(3.1b7 0x276d1a0) 39157 bytes   (queue has 1 ops and 39157 bytes)
2014-06-12 14:47:53.382245 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2700 (0'0,173'2700] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2698 active] do_sub_op osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.382253 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:53.382259 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a903f00 finish
2014-06-12 14:47:53.382263 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:53.382262 7f9d8016d700  5 _do_op 0x69a13e0 seq 2510 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:53.382265 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:53.382267 7f9d8016d700 10 _do_transaction on 0x2a29b970
2014-06-12 14:47:53.382273 7f9d8016d700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.382416 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.382440 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:53.382514 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:53.382630 7f9d8016d700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.382646 7f9d8016d700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:53.382652 7f9d8016d700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.382672 7f9d8016d700 15 _write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 4157440~36864
2014-06-12 14:47:53.383000 7f9d8016d700 10 lookup_strip_header done  cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:53.383021 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:47:53.383155 7f9d8016d700 10 _generic_write 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 4157440~36864 = 0
2014-06-12 14:47:53.383177 7f9d8016d700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:53.383186 7f9d8016d700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:53.383191 7f9d8016d700 15 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3
2014-06-12 14:47:53.383198 7f9d8016d700 10 _setattrs 3.1b7_head/c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 = 0
2014-06-12 14:47:53.383313 7f9d8016d700 20 set_header setting 17923 cid 3.1b7_head oid c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 parent seq 0
2014-06-12 14:47:53.383333 7f9d8016d700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:53.383344 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:53.383352 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:47:53.384094 7f9d8016d700 10 _do_op 0x69a13e0 seq 2510 r = 0, finisher 0x88ccc20 0
2014-06-12 14:47:53.384110 7f9d8016d700 10 _finish_op 0x69a13e0 seq 2510 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:53.384119 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:53.384123 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:53.384133 7f9d7715b700  5 -- op tracker -- , seq: 748509, time: 2014-06-12 14:47:53.384121, event: commit_sent, op: osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.384164 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2700 (0'0,173'2700] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2698 active] sub_op_modify_commit on op osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:53.384195 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.384234 7f9d7795c700  5 -- op tracker -- , seq: 748509, time: 2014-06-12 14:47:53.384229, event: sub_op_applied, op: osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.384243 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2700 (0'0,173'2700] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2699 active] sub_op_modify_applied on 0x2a29b880 op osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.384258 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2700 (0'0,173'2700] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2699 active] op_applied on primary on version 173'2700
2014-06-12 14:47:53.384267 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2700 (0'0,173'2700] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2699 active] op_applied on replica on version 173'2700
2014-06-12 14:47:53.384289 7f9d7795c700  5 -- op tracker -- , seq: 748509, time: 2014-06-12 14:47:53.384285, event: done, op: osd_sub_op(client.4287.0:141257 3.1b7 c545b9b7/rbd_data.10966b8b4567.0000000000001087/head//3 [] v 173'2700 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.451081 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:53.451131 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:53.488406 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:53.488423 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:53.488426 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:53.488427 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:53.488467 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:53.592603 7f9d5a16f700  5 -- op tracker -- , seq: 748510, time: 2014-06-12 14:47:53.591870, event: header_read, op: osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.592621 7f9d5a16f700  5 -- op tracker -- , seq: 748510, time: 2014-06-12 14:47:53.591873, event: throttled, op: osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.592627 7f9d5a16f700  5 -- op tracker -- , seq: 748510, time: 2014-06-12 14:47:53.592568, event: all_read, op: osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.592635 7f9d5a16f700  5 -- op tracker -- , seq: 748510, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.592639 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:53.592644 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:53.592651 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903200 prio 127 cost 516903 latency 0.000780 osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.592723 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:53.592734 7f9d653db700 10 osd.2 173 dequeue_op 0x2a903200 prio 127 cost 516903 latency 0.000864 osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2477 (0'0,173'2477] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active]
2014-06-12 14:47:53.592766 7f9d653db700  5 -- op tracker -- , seq: 748510, time: 2014-06-12 14:47:53.592763, event: reached_pg, op: osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.592783 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2477 (0'0,173'2477] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active] handle_message: 0x2a903200
2014-06-12 14:47:53.592793 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2477 (0'0,173'2477] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active] sub_op_modify trans 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 v 173'2478 (transaction) 174
2014-06-12 14:47:53.592804 7f9d653db700  5 -- op tracker -- , seq: 748510, time: 2014-06-12 14:47:53.592802, event: started, op: osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.592819 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2477 (0'0,173'2477] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active] append_log log((0'0,173'2477], crt=0'0) [173'2478 (173'2477) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144680 2014-06-12 08:45:04.203025]
2014-06-12 14:47:53.592837 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2478 (0'0,173'2478] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active] add_log_entry 173'2478 (173'2477) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144680 2014-06-12 08:45:04.203025
2014-06-12 14:47:53.592854 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2478 (0'0,173'2478] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active] append_log  adding 1 keys
2014-06-12 14:47:53.592878 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2478, trimmed: 
2014-06-12 14:47:53.592911 7f9d653db700  5 -- op tracker -- , seq: 748510, time: 2014-06-12 14:47:53.592908, event: started, op: osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.592921 7f9d653db700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:53.592924 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:53.592925 7f9d653db700  5 queue_op 0x142a87e0 seq 2311 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:53.592930 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2478 (0'0,173'2478] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active] do_sub_op osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.592938 7f9d653db700 10 osd.2 173 dequeue_op 0x2a903200 finish
2014-06-12 14:47:53.592935 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:47:53.592941 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:53.592942 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:53.592944 7f9d79960700  5 _do_op 0x142a87e0 seq 2311 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:47:53.592949 7f9d79960700 10 _do_transaction on 0x62c9eb0
2014-06-12 14:47:53.592957 7f9d79960700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:53.593082 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:53.593113 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:53.593156 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:53.593180 7f9d79960700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:53.593187 7f9d79960700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:47:53.593193 7f9d79960700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:53.593318 7f9d79960700 15 _write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 2539520~516096
2014-06-12 14:47:53.593431 7f9d79960700 10 lookup_strip_header done  cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:53.593539 7f9d79960700 10 file_to_extents done 
2014-06-12 14:47:53.594781 7f9d79960700 10 _generic_write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 2539520~516096 = 0
2014-06-12 14:47:53.594818 7f9d79960700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:53.594831 7f9d79960700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:53.594838 7f9d79960700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:53.594846 7f9d79960700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:53.594979 7f9d79960700 20 set_header setting 17353 cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 parent seq 0
2014-06-12 14:47:53.595027 7f9d79960700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:47:53.595042 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:53.595052 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:47:53.597070 7f9d79960700 10 _do_op 0x142a87e0 seq 2311 r = 0, finisher 0x4efdec0 0
2014-06-12 14:47:53.597087 7f9d79960700 10 _finish_op 0x142a87e0 seq 2311 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:53.597102 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:47:53.597106 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:53.597115 7f9d7795c700  5 -- op tracker -- , seq: 748510, time: 2014-06-12 14:47:53.597108, event: sub_op_applied, op: osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.597125 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2478 (0'0,173'2478] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active] sub_op_modify_applied on 0x62c9dc0 op osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.597147 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.597169 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2478 (0'0,173'2478] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active] op_applied on primary on version 173'2478
2014-06-12 14:47:53.597182 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2478 (0'0,173'2478] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active] op_applied on replica on version 173'2478
2014-06-12 14:47:53.597205 7f9d7715b700  5 -- op tracker -- , seq: 748510, time: 2014-06-12 14:47:53.597198, event: commit_sent, op: osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.597213 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2478 (0'0,173'2478] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2476 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:53.597239 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.597271 7f9d7715b700  5 -- op tracker -- , seq: 748510, time: 2014-06-12 14:47:53.597265, event: done, op: osd_sub_op(client.4303.0:144680 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2478 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.726008 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.726442 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.737447 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:47:53.737481 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19430 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:47:53.738332 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:53.738413 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:53.738476 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.738522 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:53.738562 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:53.738693 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:53.738730 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:53.738766 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:53.738803 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:53.738836 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:53.738874 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:53.738912 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:53.738950 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:53.738983 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:53.739016 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.739051 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:53.739101 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:53.739133 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:53.739196 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:53.739243 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:53.739270 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:53.739066 7f9d5a16f700  5 -- op tracker -- , seq: 748511, time: 2014-06-12 14:47:53.737661, event: header_read, op: osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.739301 7f9d5a16f700  5 -- op tracker -- , seq: 748511, time: 2014-06-12 14:47:53.737664, event: throttled, op: osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.739321 7f9d5a16f700  5 -- op tracker -- , seq: 748511, time: 2014-06-12 14:47:53.739003, event: all_read, op: osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.739328 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:53.739339 7f9d5a16f700  5 -- op tracker -- , seq: 748511, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.739349 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:53.739361 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:53.739371 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a902800 prio 127 cost 516903 latency 0.001709 osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.739591 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:53.739608 7f9d623d5700 10 osd.2 173 dequeue_op 0x2a902800 prio 127 cost 516903 latency 0.001947 osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2700 (0'0,173'2700] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2699 active]
2014-06-12 14:47:53.739663 7f9d623d5700  5 -- op tracker -- , seq: 748511, time: 2014-06-12 14:47:53.739653, event: reached_pg, op: osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.739681 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2700 (0'0,173'2700] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2699 active] handle_message: 0x2a902800
2014-06-12 14:47:53.739709 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2700 (0'0,173'2700] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2699 active] sub_op_modify trans 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 v 173'2701 (transaction) 174
2014-06-12 14:47:53.739754 7f9d623d5700  5 -- op tracker -- , seq: 748511, time: 2014-06-12 14:47:53.739744, event: started, op: osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.739787 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2700 (0'0,173'2700] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2699 active] append_log log((0'0,173'2700], crt=0'0) [173'2701 (173'2699) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141506 2014-06-12 08:45:27.694328]
2014-06-12 14:47:53.739837 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2701 (0'0,173'2701] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2699 active] add_log_entry 173'2701 (173'2699) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141506 2014-06-12 08:45:27.694328
2014-06-12 14:47:53.739889 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2701 (0'0,173'2701] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2699 active] append_log  adding 1 keys
2014-06-12 14:47:53.739969 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2701, trimmed: 
2014-06-12 14:47:53.740024 7f9d623d5700  5 -- op tracker -- , seq: 748511, time: 2014-06-12 14:47:53.740012, event: started, op: osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.740045 7f9d623d5700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:53.740053 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:53.740057 7f9d623d5700  5 queue_op 0x56053e0 seq 2511 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:53.740071 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2701 (0'0,173'2701] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2699 active] do_sub_op osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.740105 7f9d623d5700 10 osd.2 173 dequeue_op 0x2a902800 finish
2014-06-12 14:47:53.740114 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:53.740120 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:53.740136 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:53.740146 7f9d7d968700  5 _do_op 0x56053e0 seq 2511 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:53.740154 7f9d7d968700 10 _do_transaction on 0x1ea810b0
2014-06-12 14:47:53.740167 7f9d7d968700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.740360 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.740406 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:53.740502 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:53.740640 7f9d7d968700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.740659 7f9d7d968700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:53.740673 7f9d7d968700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:53.740705 7f9d7d968700 15 _write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 3244032~516096
2014-06-12 14:47:53.740923 7f9d7d968700 10 lookup_strip_header done  cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:53.741149 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:47:53.745013 7f9d7d968700 10 _generic_write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 3244032~516096 = 0
2014-06-12 14:47:53.745110 7f9d7d968700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:53.745142 7f9d7d968700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:53.745155 7f9d7d968700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:53.745172 7f9d7d968700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:53.745598 7f9d7d968700 20 set_header setting 17393 cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 parent seq 0
2014-06-12 14:47:53.745765 7f9d7d968700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:53.745801 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:53.745820 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:47:53.748457 7f9d7d968700 10 _do_op 0x56053e0 seq 2511 r = 0, finisher 0x3fd9840 0
2014-06-12 14:47:53.748471 7f9d7d968700 10 _finish_op 0x56053e0 seq 2511 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:53.748476 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:53.748482 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:53.748498 7f9d7715b700  5 -- op tracker -- , seq: 748511, time: 2014-06-12 14:47:53.748480, event: commit_sent, op: osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.748513 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2701 (0'0,173'2701] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2699 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:53.748546 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:53.748583 7f9d7795c700  5 -- op tracker -- , seq: 748511, time: 2014-06-12 14:47:53.748570, event: sub_op_applied, op: osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.748591 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2701 (0'0,173'2701] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active] sub_op_modify_applied on 0x1ea80fc0 op osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.748603 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2701 (0'0,173'2701] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active] op_applied on primary on version 173'2701
2014-06-12 14:47:53.748609 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2701 (0'0,173'2701] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active] op_applied on replica on version 173'2701
2014-06-12 14:47:53.748621 7f9d7795c700  5 -- op tracker -- , seq: 748511, time: 2014-06-12 14:47:53.748619, event: done, op: osd_sub_op(client.4284.0:141506 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2701 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.952112 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:53.957580 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:53.958073 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:53.958102 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:53.968269 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:53.968465 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:53.984640 7f9d5a16f700  5 -- op tracker -- , seq: 748512, time: 2014-06-12 14:47:53.982089, event: header_read, op: osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.984666 7f9d5a16f700  5 -- op tracker -- , seq: 748512, time: 2014-06-12 14:47:53.982092, event: throttled, op: osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.984728 7f9d5a16f700  5 -- op tracker -- , seq: 748512, time: 2014-06-12 14:47:53.984583, event: all_read, op: osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.984749 7f9d5a16f700  5 -- op tracker -- , seq: 748512, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.984756 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:53.984765 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:53.984776 7f9d5a16f700 15 osd.2 173 enqueue_op 0x876ef00 prio 127 cost 1032999 latency 0.002687 osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.984814 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:53.984835 7f9d643d9700 10 osd.2 173 dequeue_op 0x876ef00 prio 127 cost 1032999 latency 0.002745 osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2478 (0'0,173'2478] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2477 active]
2014-06-12 14:47:53.984870 7f9d643d9700  5 -- op tracker -- , seq: 748512, time: 2014-06-12 14:47:53.984864, event: reached_pg, op: osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.984889 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2478 (0'0,173'2478] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2477 active] handle_message: 0x876ef00
2014-06-12 14:47:53.984905 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2478 (0'0,173'2478] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2477 active] sub_op_modify trans 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 v 173'2479 (transaction) 174
2014-06-12 14:47:53.984926 7f9d643d9700  5 -- op tracker -- , seq: 748512, time: 2014-06-12 14:47:53.984921, event: started, op: osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.984944 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2478 (0'0,173'2478] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2477 active] append_log log((0'0,173'2478], crt=0'0) [173'2479 (173'2478) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144681 2014-06-12 08:45:05.376100]
2014-06-12 14:47:53.984976 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2479 (0'0,173'2479] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2477 active] add_log_entry 173'2479 (173'2478) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144681 2014-06-12 08:45:05.376100
2014-06-12 14:47:53.985012 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2479 (0'0,173'2479] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2477 active] append_log  adding 1 keys
2014-06-12 14:47:53.985058 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2479, trimmed: 
2014-06-12 14:47:53.985089 7f9d643d9700  5 -- op tracker -- , seq: 748512, time: 2014-06-12 14:47:53.985082, event: started, op: osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:53.985104 7f9d643d9700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:53.985109 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:53.985111 7f9d643d9700  5 queue_op 0x1af4cd80 seq 2312 osr(3.4f4 0x276db70) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:47:53.985119 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2479 (0'0,173'2479] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2477 active] do_sub_op osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:53.985139 7f9d643d9700 10 osd.2 173 dequeue_op 0x876ef00 finish
2014-06-12 14:47:53.985145 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:53.985148 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:53.985156 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:47:53.985172 7f9d8096e700  5 _do_op 0x1af4cd80 seq 2312 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:47:53.985179 7f9d8096e700 10 _do_transaction on 0x5b7aef0
2014-06-12 14:47:53.985189 7f9d8096e700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:53.985334 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:53.985372 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:53.985442 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:53.985477 7f9d8096e700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:53.985485 7f9d8096e700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:47:53.985493 7f9d8096e700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:53.985594 7f9d8096e700 15 _write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 3055616~1032192
2014-06-12 14:47:53.985819 7f9d8096e700 10 lookup_strip_header done  cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:53.985982 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:47:53.990988 7f9d8096e700 10 _generic_write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 3055616~1032192 = 0
2014-06-12 14:47:53.991119 7f9d8096e700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:53.991144 7f9d8096e700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:53.991153 7f9d8096e700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:53.991163 7f9d8096e700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:53.991578 7f9d8096e700 20 set_header setting 17353 cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 parent seq 0
2014-06-12 14:47:53.991749 7f9d8096e700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:47:53.991777 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:53.991789 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:47:54.000008 7f9d8096e700 10 _do_op 0x1af4cd80 seq 2312 r = 0, finisher 0x13bb7e80 0
2014-06-12 14:47:54.000048 7f9d8096e700 10 _finish_op 0x1af4cd80 seq 2312 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:54.000062 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:47:54.000076 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:54.000078 7f9d7715b700  5 -- op tracker -- , seq: 748512, time: 2014-06-12 14:47:54.000059, event: commit_sent, op: osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.000118 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2479 (0'0,173'2479] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2477 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:54.000161 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:54.000229 7f9d7795c700  5 -- op tracker -- , seq: 748512, time: 2014-06-12 14:47:54.000210, event: sub_op_applied, op: osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.000255 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2479 (0'0,173'2479] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2478 active] sub_op_modify_applied on 0x5b7ae00 op osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:54.000321 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2479 (0'0,173'2479] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2478 active] op_applied on primary on version 173'2479
2014-06-12 14:47:54.000346 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2479 (0'0,173'2479] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2478 active] op_applied on replica on version 173'2479
2014-06-12 14:47:54.000394 7f9d7795c700  5 -- op tracker -- , seq: 748512, time: 2014-06-12 14:47:54.000385, event: done, op: osd_sub_op(client.4303.0:144681 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2479 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.035481 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:54.035512 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:54.157563 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:54.178295 7f9d5a16f700  5 -- op tracker -- , seq: 748513, time: 2014-06-12 14:47:54.177293, event: header_read, op: osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.178318 7f9d5a16f700  5 -- op tracker -- , seq: 748513, time: 2014-06-12 14:47:54.177295, event: throttled, op: osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.178350 7f9d5a16f700  5 -- op tracker -- , seq: 748513, time: 2014-06-12 14:47:54.178233, event: all_read, op: osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.178361 7f9d5a16f700  5 -- op tracker -- , seq: 748513, time: 0.000000, event: dispatched, op: osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.178369 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:54.178378 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:54.178390 7f9d5a16f700 15 osd.2 173 enqueue_op 0x55c3f00 prio 127 cost 434983 latency 0.001096 osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:54.178463 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:54.178485 7f9d63bd8700 10 osd.2 173 dequeue_op 0x55c3f00 prio 127 cost 434983 latency 0.001191 osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2701 (0'0,173'2701] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active]
2014-06-12 14:47:54.178531 7f9d63bd8700  5 -- op tracker -- , seq: 748513, time: 2014-06-12 14:47:54.178528, event: reached_pg, op: osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.178540 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2701 (0'0,173'2701] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active] handle_message: 0x55c3f00
2014-06-12 14:47:54.178548 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2701 (0'0,173'2701] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active] sub_op_modify trans 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 v 173'2702 (transaction) 174
2014-06-12 14:47:54.178558 7f9d63bd8700  5 -- op tracker -- , seq: 748513, time: 2014-06-12 14:47:54.178555, event: started, op: osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.178578 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2701 (0'0,173'2701] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active] append_log log((0'0,173'2701], crt=0'0) [173'2702 (173'2701) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141507 2014-06-12 08:45:28.378568]
2014-06-12 14:47:54.178597 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2702 (0'0,173'2702] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active] add_log_entry 173'2702 (173'2701) modify   3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 by client.4284.0:141507 2014-06-12 08:45:28.378568
2014-06-12 14:47:54.178613 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2702 (0'0,173'2702] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active] append_log  adding 1 keys
2014-06-12 14:47:54.178690 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2702, trimmed: 
2014-06-12 14:47:54.178711 7f9d63bd8700  5 -- op tracker -- , seq: 748513, time: 2014-06-12 14:47:54.178707, event: started, op: osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.178721 7f9d63bd8700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:54.178725 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:54.178727 7f9d63bd8700  5 queue_op 0x69a1620 seq 2512 osr(3.1b7 0x276d1a0) 436469 bytes   (queue has 1 ops and 436469 bytes)
2014-06-12 14:47:54.178738 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:47:54.178748 7f9d7c966700  5 _do_op 0x69a1620 seq 2512 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:47:54.178754 7f9d7c966700 10 _do_transaction on 0x5b7b430
2014-06-12 14:47:54.178733 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2702 (0'0,173'2702] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active] do_sub_op osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:54.178758 7f9d63bd8700 10 osd.2 173 dequeue_op 0x55c3f00 finish
2014-06-12 14:47:54.178761 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:54.178762 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:54.178763 7f9d7c966700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:54.178952 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:54.179017 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:54.179130 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:54.179336 7f9d7c966700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:54.179344 7f9d7c966700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:47:54.179348 7f9d7c966700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:47:54.179361 7f9d7c966700 15 _write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 3760128~434176
2014-06-12 14:47:54.179650 7f9d7c966700 10 lookup_strip_header done  cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:54.179768 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:47:54.180901 7f9d7c966700 10 _generic_write 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 3760128~434176 = 0
2014-06-12 14:47:54.180942 7f9d7c966700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:54.180959 7f9d7c966700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:54.180964 7f9d7c966700 15 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3
2014-06-12 14:47:54.180969 7f9d7c966700 10 _setattrs 3.1b7_head/3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:47:54.181095 7f9d7c966700 20 set_header setting 17393 cid 3.1b7_head oid 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 parent seq 0
2014-06-12 14:47:54.181132 7f9d7c966700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:47:54.181143 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:54.181149 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:47:54.182672 7f9d7c966700 10 _do_op 0x69a1620 seq 2512 r = 0, finisher 0x654b5c0 0
2014-06-12 14:47:54.182689 7f9d7c966700 10 _finish_op 0x69a1620 seq 2512 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:47:54.182702 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:47:54.182711 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:54.182722 7f9d7795c700  5 -- op tracker -- , seq: 748513, time: 2014-06-12 14:47:54.182713, event: sub_op_applied, op: osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.182731 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2702 (0'0,173'2702] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active] sub_op_modify_applied on 0x5b7b340 op osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:54.182752 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:54.182767 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2702 (0'0,173'2702] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active] op_applied on primary on version 173'2702
2014-06-12 14:47:54.182774 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2702 (0'0,173'2702] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active] op_applied on replica on version 173'2702
2014-06-12 14:47:54.182796 7f9d7715b700  5 -- op tracker -- , seq: 748513, time: 2014-06-12 14:47:54.182787, event: commit_sent, op: osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.182814 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2702 (0'0,173'2702] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2700 active] sub_op_modify_commit on op osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:54.182836 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:54.182871 7f9d7715b700  5 -- op tracker -- , seq: 748513, time: 2014-06-12 14:47:54.182867, event: done, op: osd_sub_op(client.4284.0:141507 3.1b7 3a7509b7/rbd_data.10936b8b4567.0000000000000fe3/head//3 [] v 173'2702 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.304261 7f9d5a16f700  5 -- op tracker -- , seq: 748514, time: 2014-06-12 14:47:54.304037, event: header_read, op: osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.304272 7f9d5a16f700  5 -- op tracker -- , seq: 748514, time: 2014-06-12 14:47:54.304038, event: throttled, op: osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.304297 7f9d5a16f700  5 -- op tracker -- , seq: 748514, time: 2014-06-12 14:47:54.304220, event: all_read, op: osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.304302 7f9d5a16f700  5 -- op tracker -- , seq: 748514, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.304305 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:54.304310 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:54.304315 7f9d5a16f700 15 osd.2 173 enqueue_op 0x1702e700 prio 127 cost 107303 latency 0.000278 osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:54.304367 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:54.304378 7f9d61bd4700 10 osd.2 173 dequeue_op 0x1702e700 prio 127 cost 107303 latency 0.000340 osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2479 (0'0,173'2479] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2478 active]
2014-06-12 14:47:54.304406 7f9d61bd4700  5 -- op tracker -- , seq: 748514, time: 2014-06-12 14:47:54.304403, event: reached_pg, op: osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.304415 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2479 (0'0,173'2479] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2478 active] handle_message: 0x1702e700
2014-06-12 14:47:54.304424 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2479 (0'0,173'2479] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2478 active] sub_op_modify trans 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 v 173'2480 (transaction) 174
2014-06-12 14:47:54.304434 7f9d61bd4700  5 -- op tracker -- , seq: 748514, time: 2014-06-12 14:47:54.304431, event: started, op: osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.304444 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2479 (0'0,173'2479] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2478 active] append_log log((0'0,173'2479], crt=0'0) [173'2480 (173'2479) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144682 2014-06-12 08:45:05.821713]
2014-06-12 14:47:54.304470 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2480 (0'0,173'2480] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2478 active] add_log_entry 173'2480 (173'2479) modify   426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 by client.4303.0:144682 2014-06-12 08:45:05.821713
2014-06-12 14:47:54.304485 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2480 (0'0,173'2480] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2478 active] append_log  adding 1 keys
2014-06-12 14:47:54.304509 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2480, trimmed: 
2014-06-12 14:47:54.304527 7f9d61bd4700  5 -- op tracker -- , seq: 748514, time: 2014-06-12 14:47:54.304524, event: started, op: osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.304536 7f9d61bd4700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:54.304539 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:54.304540 7f9d61bd4700  5 queue_op 0x63de240 seq 2313 osr(3.4f4 0x276db70) 108789 bytes   (queue has 1 ops and 108789 bytes)
2014-06-12 14:47:54.304544 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2480 (0'0,173'2480] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2478 active] do_sub_op osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:54.304553 7f9d61bd4700 10 osd.2 173 dequeue_op 0x1702e700 finish
2014-06-12 14:47:54.304556 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:54.304557 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:54.304547 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:47:54.304562 7f9d81970700  5 _do_op 0x63de240 seq 2313 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:47:54.304567 7f9d81970700 10 _do_transaction on 0x29eb22b0
2014-06-12 14:47:54.304573 7f9d81970700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:54.304700 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:54.304727 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:54.304766 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:54.304804 7f9d81970700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:54.304813 7f9d81970700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:47:54.304819 7f9d81970700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:54.304970 7f9d81970700 15 _write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 4087808~106496
2014-06-12 14:47:54.305123 7f9d81970700 10 lookup_strip_header done  cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:54.305304 7f9d81970700 10 file_to_extents done 
2014-06-12 14:47:54.305696 7f9d81970700 10 _generic_write 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 4087808~106496 = 0
2014-06-12 14:47:54.305717 7f9d81970700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:54.305727 7f9d81970700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:54.305730 7f9d81970700 15 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3
2014-06-12 14:47:54.305735 7f9d81970700 10 _setattrs 3.4f4_head/426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 = 0
2014-06-12 14:47:54.305856 7f9d81970700 20 set_header setting 17353 cid 3.4f4_head oid 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 parent seq 0
2014-06-12 14:47:54.305934 7f9d81970700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:47:54.305951 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:54.305960 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:47:54.306513 7f9d81970700 10 _do_op 0x63de240 seq 2313 r = 0, finisher 0x82c81a0 0
2014-06-12 14:47:54.306524 7f9d81970700 10 _finish_op 0x63de240 seq 2313 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:54.306531 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:47:54.306533 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:54.306577 7f9d7715b700  5 -- op tracker -- , seq: 748514, time: 2014-06-12 14:47:54.306568, event: commit_sent, op: osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.306591 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2480 (0'0,173'2480] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2478 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:54.306621 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:54.306683 7f9d7795c700  5 -- op tracker -- , seq: 748514, time: 2014-06-12 14:47:54.306675, event: sub_op_applied, op: osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.306697 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2480 (0'0,173'2480] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] sub_op_modify_applied on 0x29eb21c0 op osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:54.306723 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2480 (0'0,173'2480] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] op_applied on primary on version 173'2480
2014-06-12 14:47:54.306732 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2480 (0'0,173'2480] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] op_applied on replica on version 173'2480
2014-06-12 14:47:54.306752 7f9d7795c700  5 -- op tracker -- , seq: 748514, time: 2014-06-12 14:47:54.306749, event: done, op: osd_sub_op(client.4303.0:144682 3.4f4 426c6cf4/rbd_data.10a66b8b4567.0000000000000fe4/head//3 [] v 173'2480 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:54.368012 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:54.368050 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:54.372185 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:54.488520 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:54.488556 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:54.488564 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:54.488567 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:54.488596 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:54.510977 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:54.514484 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:54.516416 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:54.517447 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:54.524251 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:54.556355 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:54.662146 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:54.665281 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:54.667340 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:54.671859 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:54.804284 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:54.807116 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:54.809909 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:54.810174 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:54.866358 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:54.870267 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:54.986397 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:55.010525 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:55.013449 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:55.022926 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:55.023130 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:55.023915 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:55.030716 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:55.035237 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:55.068910 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:55.068967 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:55.154884 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:55.155151 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:55.205825 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:55.208095 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:55.216590 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:55.226985 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:55.382327 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:55.384183 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:55.426542 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:55.426627 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:55.488660 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:55.488691 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:55.488695 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:55.488698 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:55.488736 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:55.593028 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:55.597179 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:55.736390 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:55.736480 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:55.740211 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:55.748568 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:55.814261 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:55.814304 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:55.952193 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:55.957658 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:55.985241 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:56.000147 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:56.157669 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:56.178823 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:56.182774 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:56.304647 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:56.306604 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:56.372281 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:56.488773 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:56.488803 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:56.488808 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:56.488811 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:56.488847 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:56.511073 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:56.514548 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:56.516481 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:56.517533 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:56.524315 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:56.556440 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:56.638336 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:47:56.638352 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:47:56.638761 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:56.638786 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:56.638806 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:56.638819 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:56.638832 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:56.638845 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:56.638874 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:56.638888 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:56.638900 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:56.639026 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:56.639044 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:56.639062 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:56.639075 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:56.639088 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:56.639101 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:56.639112 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:56.639124 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:56.639135 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:56.639145 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:56.639156 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:56.639171 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:56.639184 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:56.662237 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:56.665373 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:56.667415 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:56.671949 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:56.804351 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:56.807208 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:56.809995 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:56.810249 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:56.837141 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:56.837273 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:56.866448 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:56.870348 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:56.986498 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:57.010606 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:57.013538 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:57.023991 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:57.030807 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:57.035326 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:57.130420 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:57.130477 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:57.205906 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:57.208165 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:57.216662 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:57.227065 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:57.382428 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:57.384272 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:57.458791 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:57.458830 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:57.484174 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:57.484358 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:57.488890 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:57.488910 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:47:57.488914 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:47:57.488918 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:47:57.488969 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:57.488973 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 5.3 >= max 0.5 = no, load too high
2014-06-12 14:47:57.488989 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:47:57.488992 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92725 < max (604800 seconds)
2014-06-12 14:47:57.489013 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:47:57.489016 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:57.489018 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:57.593134 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:57.597258 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:57.727316 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:57.727383 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:57.740296 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:57.748648 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:57.868562 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:57.868632 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:47:57.952278 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:57.957738 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:57.985325 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:47:58.000216 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:58.055345 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:58.055574 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:58.152002 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:58.152155 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:58.157758 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:47:58.178905 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:47:58.182848 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:58.231393 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:58.231466 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:58.304723 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:58.306674 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:58.338879 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:47:58.338902 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:47:58.339573 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:58.339633 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:58.339648 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:58.339668 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:58.339691 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:58.339718 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:58.339755 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:58.339779 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:58.339791 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:58.339809 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:58.339830 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:58.339847 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:58.339879 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:58.339939 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:58.339964 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:58.339997 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:58.340049 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:58.340077 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:58.340097 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:58.340118 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:58.340130 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:58.340141 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:58.372378 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:47:58.489092 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:58.489123 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:58.489128 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:58.489130 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:58.489168 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:58.511143 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:58.514612 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:47:58.516544 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:58.517598 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:47:58.524378 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:58.556539 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:58.569648 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:58.569807 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:58.635279 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:58.635325 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:58.662322 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:47:58.665469 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:47:58.667496 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:58.672046 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:58.804418 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:58.807307 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:47:58.810077 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:58.810321 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:58.866533 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:47:58.870419 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:58.986602 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:47:59.010691 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:47:59.013638 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:59.024072 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:59.030889 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:59.035404 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:59.159031 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:59.159060 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:59.205980 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:47:59.208231 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:59.216727 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:47:59.227146 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:59.314781 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:59.314858 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:59.382508 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:47:59.384350 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:59.422660 7f9d5a16f700  5 -- op tracker -- , seq: 748515, time: 2014-06-12 14:47:59.421681, event: header_read, op: osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.422682 7f9d5a16f700  5 -- op tracker -- , seq: 748515, time: 2014-06-12 14:47:59.421686, event: throttled, op: osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.422716 7f9d5a16f700  5 -- op tracker -- , seq: 748515, time: 2014-06-12 14:47:59.422602, event: all_read, op: osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.422727 7f9d5a16f700  5 -- op tracker -- , seq: 748515, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.422733 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:59.422743 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:59.422754 7f9d5a16f700 15 osd.2 173 enqueue_op 0x17c2a000 prio 127 cost 344871 latency 0.001072 osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:59.422907 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:59.422928 7f9d653db700 10 osd.2 173 dequeue_op 0x17c2a000 prio 127 cost 344871 latency 0.001246 osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2480 (0'0,173'2480] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active]
2014-06-12 14:47:59.422982 7f9d653db700  5 -- op tracker -- , seq: 748515, time: 2014-06-12 14:47:59.422972, event: reached_pg, op: osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.423000 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2480 (0'0,173'2480] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] handle_message: 0x17c2a000
2014-06-12 14:47:59.423029 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2480 (0'0,173'2480] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] sub_op_modify trans 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 v 173'2481 (transaction) 174
2014-06-12 14:47:59.423069 7f9d653db700  5 -- op tracker -- , seq: 748515, time: 2014-06-12 14:47:59.423059, event: started, op: osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.423100 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2480 (0'0,173'2480] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] append_log log((0'0,173'2480], crt=0'0) [173'2481 (173'621) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144692 2014-06-12 08:45:06.369894]
2014-06-12 14:47:59.423154 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2481 (0'0,173'2481] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] add_log_entry 173'2481 (173'621) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144692 2014-06-12 08:45:06.369894
2014-06-12 14:47:59.423203 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2481 (0'0,173'2481] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] append_log  adding 1 keys
2014-06-12 14:47:59.423272 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2481, trimmed: 
2014-06-12 14:47:59.423321 7f9d653db700  5 -- op tracker -- , seq: 748515, time: 2014-06-12 14:47:59.423310, event: started, op: osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.423344 7f9d653db700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:59.423352 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:59.423355 7f9d653db700  5 queue_op 0x142a9260 seq 2314 osr(3.4f4 0x276db70) 346357 bytes   (queue has 1 ops and 346357 bytes)
2014-06-12 14:47:59.423374 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:47:59.423388 7f9d79960700  5 _do_op 0x142a9260 seq 2314 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:47:59.423396 7f9d79960700 10 _do_transaction on 0x59c07f0
2014-06-12 14:47:59.423368 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2481 (0'0,173'2481] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] do_sub_op osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:59.423400 7f9d653db700 10 osd.2 173 dequeue_op 0x17c2a000 finish
2014-06-12 14:47:59.423409 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:59.423409 7f9d79960700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:59.423414 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:47:59.423719 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:59.423757 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:59.423862 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:59.423901 7f9d79960700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:59.423912 7f9d79960700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:47:59.423924 7f9d79960700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:59.423956 7f9d79960700 15 _write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 0~344064
2014-06-12 14:47:59.424520 7f9d79960700 10 lookup_strip_header done  cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:47:59.424716 7f9d79960700 10 file_to_extents done 
2014-06-12 14:47:59.427222 7f9d79960700 10 _generic_write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 0~344064 = 0
2014-06-12 14:47:59.427305 7f9d79960700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:47:59.427332 7f9d79960700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:47:59.427349 7f9d79960700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:47:59.427367 7f9d79960700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:47:59.427737 7f9d79960700 20 set_header setting 17360 cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 parent seq 0
2014-06-12 14:47:59.427872 7f9d79960700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:47:59.427903 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:59.427922 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:47:59.431035 7f9d79960700 10 _do_op 0x142a9260 seq 2314 r = 0, finisher 0x668f8e0 0
2014-06-12 14:47:59.431081 7f9d7715b700  5 -- op tracker -- , seq: 748515, time: 2014-06-12 14:47:59.431069, event: commit_sent, op: osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.431091 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2481 (0'0,173'2481] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2479 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:59.431119 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:59.431177 7f9d79960700 10 _finish_op 0x142a9260 seq 2314 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:59.431186 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:47:59.431195 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:59.431203 7f9d7795c700  5 -- op tracker -- , seq: 748515, time: 2014-06-12 14:47:59.431195, event: sub_op_applied, op: osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.431211 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2481 (0'0,173'2481] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] sub_op_modify_applied on 0x59c0700 op osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:59.431237 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2481 (0'0,173'2481] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] op_applied on primary on version 173'2481
2014-06-12 14:47:59.431247 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2481 (0'0,173'2481] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] op_applied on replica on version 173'2481
2014-06-12 14:47:59.431269 7f9d7795c700  5 -- op tracker -- , seq: 748515, time: 2014-06-12 14:47:59.431265, event: done, op: osd_sub_op(client.4303.0:144692 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2481 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.439700 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:47:59.439728 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:47:59.440176 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:59.440206 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:47:59.440222 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:59.440254 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:59.440271 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:59.440290 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:59.440318 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:59.440342 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:59.440362 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:59.440384 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:47:59.440406 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:47:59.440418 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:47:59.440448 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:59.440458 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:59.440479 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:47:59.440507 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:59.440535 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:47:59.440574 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:59.440601 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:59.440628 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:59.440655 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:47:59.440681 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:47:59.489227 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:47:59.489260 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:47:59.489265 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:47:59.489267 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:47:59.489305 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:47:59.723493 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:59.723910 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:47:59.740373 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:47:59.748719 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:59.823875 7f9d5a16f700  5 -- op tracker -- , seq: 748516, time: 2014-06-12 14:47:59.822441, event: header_read, op: osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.823886 7f9d5a16f700  5 -- op tracker -- , seq: 748516, time: 2014-06-12 14:47:59.822455, event: throttled, op: osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.823912 7f9d5a16f700  5 -- op tracker -- , seq: 748516, time: 2014-06-12 14:47:59.823837, event: all_read, op: osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.823918 7f9d5a16f700  5 -- op tracker -- , seq: 748516, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.823922 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:47:59.823928 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:47:59.823935 7f9d5a16f700 15 osd.2 173 enqueue_op 0x31d61500 prio 127 cost 516903 latency 0.001494 osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:59.823957 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:47:59.823966 7f9d68be2700 10 osd.2 173 dequeue_op 0x31d61500 prio 127 cost 516903 latency 0.001525 osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2481 (0'0,173'2481] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active]
2014-06-12 14:47:59.824011 7f9d68be2700  5 -- op tracker -- , seq: 748516, time: 2014-06-12 14:47:59.824004, event: reached_pg, op: osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.824033 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2481 (0'0,173'2481] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] handle_message: 0x31d61500
2014-06-12 14:47:59.824052 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2481 (0'0,173'2481] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] sub_op_modify trans 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 v 173'2482 (transaction) 174
2014-06-12 14:47:59.824072 7f9d68be2700  5 -- op tracker -- , seq: 748516, time: 2014-06-12 14:47:59.824066, event: started, op: osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.824097 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2481 (0'0,173'2481] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] append_log log((0'0,173'2481], crt=0'0) [173'2482 (173'2481) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144693 2014-06-12 08:45:07.334821]
2014-06-12 14:47:59.824147 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2482 (0'0,173'2482] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] add_log_entry 173'2482 (173'2481) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144693 2014-06-12 08:45:07.334821
2014-06-12 14:47:59.824199 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2482 (0'0,173'2482] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] append_log  adding 1 keys
2014-06-12 14:47:59.824366 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2482, trimmed: 
2014-06-12 14:47:59.824411 7f9d68be2700  5 -- op tracker -- , seq: 748516, time: 2014-06-12 14:47:59.824400, event: started, op: osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.824450 7f9d68be2700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:59.824458 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:47:59.824461 7f9d68be2700  5 queue_op 0x4310d80 seq 2315 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:47:59.824476 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2482 (0'0,173'2482] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] do_sub_op osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:59.824512 7f9d68be2700 10 osd.2 173 dequeue_op 0x31d61500 finish
2014-06-12 14:47:59.824521 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:47:59.824527 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:47:59.824570 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:47:59.824576 7f9d7c165700  5 _do_op 0x4310d80 seq 2315 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:47:59.824581 7f9d7c165700 10 _do_transaction on 0x5b7a470
2014-06-12 14:47:59.824586 7f9d7c165700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:59.824694 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:59.824714 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:47:59.824768 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:47:59.824860 7f9d7c165700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:59.824867 7f9d7c165700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:47:59.824871 7f9d7c165700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:47:59.824883 7f9d7c165700 15 _write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 344064~516096
2014-06-12 14:47:59.824984 7f9d7c165700 10 lookup_strip_header done  cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:47:59.825187 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:47:59.826627 7f9d7c165700 10 _generic_write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 344064~516096 = 0
2014-06-12 14:47:59.826668 7f9d7c165700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:47:59.826679 7f9d7c165700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:47:59.826683 7f9d7c165700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:47:59.826688 7f9d7c165700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:47:59.826828 7f9d7c165700 20 set_header setting 17360 cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 parent seq 0
2014-06-12 14:47:59.826882 7f9d7c165700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:47:59.826900 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:47:59.826913 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:47:59.828896 7f9d7c165700 10 _do_op 0x4310d80 seq 2315 r = 0, finisher 0x6607ee0 0
2014-06-12 14:47:59.828918 7f9d7c165700 10 _finish_op 0x4310d80 seq 2315 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:47:59.828925 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:47:59.828928 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:47:59.828959 7f9d7795c700  5 -- op tracker -- , seq: 748516, time: 2014-06-12 14:47:59.828947, event: sub_op_applied, op: osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.828975 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2482 (0'0,173'2482] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] sub_op_modify_applied on 0x5b7a380 op osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:47:59.829001 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:59.829019 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2482 (0'0,173'2482] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] op_applied on primary on version 173'2482
2014-06-12 14:47:59.829026 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2482 (0'0,173'2482] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] op_applied on replica on version 173'2482
2014-06-12 14:47:59.829046 7f9d7715b700  5 -- op tracker -- , seq: 748516, time: 2014-06-12 14:47:59.829040, event: commit_sent, op: osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.829054 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2482 (0'0,173'2482] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2480 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:47:59.829067 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:47:59.829092 7f9d7715b700  5 -- op tracker -- , seq: 748516, time: 2014-06-12 14:47:59.829089, event: done, op: osd_sub_op(client.4303.0:144693 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2482 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:47:59.931930 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:59.932181 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:47:59.985397 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:00.000285 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:00.116162 7f9d5a16f700  5 -- op tracker -- , seq: 748517, time: 2014-06-12 14:48:00.114713, event: header_read, op: osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.116173 7f9d5a16f700  5 -- op tracker -- , seq: 748517, time: 2014-06-12 14:48:00.114716, event: throttled, op: osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.116196 7f9d5a16f700  5 -- op tracker -- , seq: 748517, time: 2014-06-12 14:48:00.116130, event: all_read, op: osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.116203 7f9d5a16f700  5 -- op tracker -- , seq: 748517, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.116207 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:00.116212 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:00.116219 7f9d5a16f700 15 osd.2 173 enqueue_op 0x31d61a00 prio 127 cost 787239 latency 0.001505 osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.116243 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:00.116252 7f9d63bd8700 10 osd.2 173 dequeue_op 0x31d61a00 prio 127 cost 787239 latency 0.001539 osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2702 (0'0,173'2702] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2701 active]
2014-06-12 14:48:00.116271 7f9d63bd8700  5 -- op tracker -- , seq: 748517, time: 2014-06-12 14:48:00.116268, event: reached_pg, op: osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.116278 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2702 (0'0,173'2702] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2701 active] handle_message: 0x31d61a00
2014-06-12 14:48:00.116285 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2702 (0'0,173'2702] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2701 active] sub_op_modify trans a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 v 173'2703 (transaction) 174
2014-06-12 14:48:00.116295 7f9d63bd8700  5 -- op tracker -- , seq: 748517, time: 2014-06-12 14:48:00.116293, event: started, op: osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.116305 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2702 (0'0,173'2702] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2701 active] append_log log((0'0,173'2702], crt=0'0) [173'2703 (173'783) modify   a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 by client.4313.0:146607 2014-06-12 08:45:53.775228]
2014-06-12 14:48:00.116322 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2703 (0'0,173'2703] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2701 active] add_log_entry 173'2703 (173'783) modify   a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 by client.4313.0:146607 2014-06-12 08:45:53.775228
2014-06-12 14:48:00.116338 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2703 (0'0,173'2703] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2701 active] append_log  adding 1 keys
2014-06-12 14:48:00.116359 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2703, trimmed: 
2014-06-12 14:48:00.116375 7f9d63bd8700  5 -- op tracker -- , seq: 748517, time: 2014-06-12 14:48:00.116372, event: started, op: osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.116383 7f9d63bd8700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:48:00.116385 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:00.116386 7f9d63bd8700  5 queue_op 0x5411080 seq 2513 osr(3.1b7 0x276d1a0) 788725 bytes   (queue has 1 ops and 788725 bytes)
2014-06-12 14:48:00.116395 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:48:00.116401 7f9d7c966700  5 _do_op 0x5411080 seq 2513 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:48:00.116391 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2703 (0'0,173'2703] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2701 active] do_sub_op osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.116404 7f9d63bd8700 10 osd.2 173 dequeue_op 0x31d61a00 finish
2014-06-12 14:48:00.116405 7f9d7c966700 10 _do_transaction on 0x556bb30
2014-06-12 14:48:00.116407 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:00.116410 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:00.116411 7f9d7c966700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.116512 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.116532 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:00.116568 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:00.116663 7f9d7c966700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.116668 7f9d7c966700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:48:00.116672 7f9d7c966700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.116686 7f9d7c966700 15 _write 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 0~786432
2014-06-12 14:48:00.116802 7f9d7c966700 10 lookup_strip_header done  cid 3.1b7_head oid a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:00.116969 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:48:00.120439 7f9d7c966700 10 _generic_write 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 0~786432 = 0
2014-06-12 14:48:00.120536 7f9d7c966700 15 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:00.120561 7f9d7c966700 10 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 = 0
2014-06-12 14:48:00.120569 7f9d7c966700 15 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:00.120579 7f9d7c966700 10 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 = 0
2014-06-12 14:48:00.120912 7f9d7c966700 20 set_header setting 17402 cid 3.1b7_head oid a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 parent seq 0
2014-06-12 14:48:00.121045 7f9d7c966700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:48:00.121069 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:00.121081 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:48:00.125916 7f9d7c966700 10 _do_op 0x5411080 seq 2513 r = 0, finisher 0x87fb900 0
2014-06-12 14:48:00.125932 7f9d7c966700 10 _finish_op 0x5411080 seq 2513 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:48:00.125936 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:48:00.125946 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:00.125955 7f9d7795c700  5 -- op tracker -- , seq: 748517, time: 2014-06-12 14:48:00.125946, event: sub_op_applied, op: osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.125969 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2703 (0'0,173'2703] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2701 active] sub_op_modify_applied on 0x556ba40 op osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.125988 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.126000 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2703 (0'0,173'2703] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2701 active] op_applied on primary on version 173'2703
2014-06-12 14:48:00.126009 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2703 (0'0,173'2703] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2701 active] op_applied on replica on version 173'2703
2014-06-12 14:48:00.126038 7f9d7715b700  5 -- op tracker -- , seq: 748517, time: 2014-06-12 14:48:00.126030, event: commit_sent, op: osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.126052 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2703 (0'0,173'2703] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2701 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:00.126074 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.126105 7f9d7715b700  5 -- op tracker -- , seq: 748517, time: 2014-06-12 14:48:00.126100, event: done, op: osd_sub_op(client.4313.0:146607 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2703 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.157831 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:00.259587 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:00.259636 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:00.266494 7f9d5a16f700  5 -- op tracker -- , seq: 748518, time: 2014-06-12 14:48:00.265177, event: header_read, op: osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.266520 7f9d5a16f700  5 -- op tracker -- , seq: 748518, time: 2014-06-12 14:48:00.265182, event: throttled, op: osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.266539 7f9d5a16f700  5 -- op tracker -- , seq: 748518, time: 2014-06-12 14:48:00.266401, event: all_read, op: osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.266556 7f9d5a16f700  5 -- op tracker -- , seq: 748518, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.266565 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:00.266579 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:00.266593 7f9d5a16f700 15 osd.2 173 enqueue_op 0x31d60100 prio 127 cost 516903 latency 0.001415 osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.266644 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:00.266659 7f9d61bd4700 10 osd.2 173 dequeue_op 0x31d60100 prio 127 cost 516903 latency 0.001481 osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2482 (0'0,173'2482] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active]
2014-06-12 14:48:00.266702 7f9d61bd4700  5 -- op tracker -- , seq: 748518, time: 2014-06-12 14:48:00.266692, event: reached_pg, op: osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.266722 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2482 (0'0,173'2482] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] handle_message: 0x31d60100
2014-06-12 14:48:00.266749 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2482 (0'0,173'2482] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] sub_op_modify trans 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 v 173'2483 (transaction) 174
2014-06-12 14:48:00.266787 7f9d61bd4700  5 -- op tracker -- , seq: 748518, time: 2014-06-12 14:48:00.266777, event: started, op: osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.266813 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2482 (0'0,173'2482] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] append_log log((0'0,173'2482], crt=0'0) [173'2483 (173'2482) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144694 2014-06-12 08:45:11.298761]
2014-06-12 14:48:00.266847 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2483 (0'0,173'2483] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] add_log_entry 173'2483 (173'2482) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144694 2014-06-12 08:45:11.298761
2014-06-12 14:48:00.266877 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2483 (0'0,173'2483] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] append_log  adding 1 keys
2014-06-12 14:48:00.266919 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2483, trimmed: 
2014-06-12 14:48:00.266949 7f9d61bd4700  5 -- op tracker -- , seq: 748518, time: 2014-06-12 14:48:00.266942, event: started, op: osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.266962 7f9d61bd4700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:00.266966 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:00.266976 7f9d61bd4700  5 queue_op 0x6e55da0 seq 2316 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:00.266995 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:00.266986 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2483 (0'0,173'2483] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] do_sub_op osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.267006 7f9d61bd4700 10 osd.2 173 dequeue_op 0x31d60100 finish
2014-06-12 14:48:00.267008 7f9d81970700  5 _do_op 0x6e55da0 seq 2316 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:00.267011 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:00.267015 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:00.267016 7f9d81970700 10 _do_transaction on 0x5afc9b0
2014-06-12 14:48:00.267026 7f9d81970700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.267212 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.267245 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:00.267344 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:00.267374 7f9d81970700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.267384 7f9d81970700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:00.267396 7f9d81970700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.267423 7f9d81970700 15 _write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 860160~516096
2014-06-12 14:48:00.267716 7f9d81970700 10 lookup_strip_header done  cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:00.267746 7f9d81970700 10 file_to_extents done 
2014-06-12 14:48:00.271002 7f9d81970700 10 _generic_write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 860160~516096 = 0
2014-06-12 14:48:00.271096 7f9d81970700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:00.271116 7f9d81970700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:48:00.271123 7f9d81970700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:00.271134 7f9d81970700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:48:00.271385 7f9d81970700 20 set_header setting 17360 cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 parent seq 0
2014-06-12 14:48:00.271472 7f9d81970700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:00.271503 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:00.271514 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:48:00.274645 7f9d81970700 10 _do_op 0x6e55da0 seq 2316 r = 0, finisher 0x654af80 0
2014-06-12 14:48:00.274670 7f9d81970700 10 _finish_op 0x6e55da0 seq 2316 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:00.274678 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:00.274682 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:00.274701 7f9d7795c700  5 -- op tracker -- , seq: 748518, time: 2014-06-12 14:48:00.274689, event: sub_op_applied, op: osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.274710 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2483 (0'0,173'2483] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] sub_op_modify_applied on 0x5afc8c0 op osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.274741 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.274760 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2483 (0'0,173'2483] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] op_applied on primary on version 173'2483
2014-06-12 14:48:00.274772 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2483 (0'0,173'2483] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] op_applied on replica on version 173'2483
2014-06-12 14:48:00.274925 7f9d7715b700  5 -- op tracker -- , seq: 748518, time: 2014-06-12 14:48:00.274918, event: commit_sent, op: osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.274935 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2483 (0'0,173'2483] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2481 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:00.274956 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.274996 7f9d7715b700  5 -- op tracker -- , seq: 748518, time: 2014-06-12 14:48:00.274990, event: done, op: osd_sub_op(client.4303.0:144694 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2483 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.372450 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:00.430462 7f9d5a16f700  5 -- op tracker -- , seq: 748519, time: 2014-06-12 14:48:00.428450, event: header_read, op: osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.430477 7f9d5a16f700  5 -- op tracker -- , seq: 748519, time: 2014-06-12 14:48:00.428454, event: throttled, op: osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.430486 7f9d5a16f700  5 -- op tracker -- , seq: 748519, time: 2014-06-12 14:48:00.430423, event: all_read, op: osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.430495 7f9d5a16f700  5 -- op tracker -- , seq: 748519, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.430500 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:00.430507 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:00.430516 7f9d5a16f700 15 osd.2 173 enqueue_op 0x25e42100 prio 127 cost 1032999 latency 0.002065 osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.430544 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:00.430552 7f9d633d7700 10 osd.2 173 dequeue_op 0x25e42100 prio 127 cost 1032999 latency 0.002102 osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2703 (0'0,173'2703] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2702 active]
2014-06-12 14:48:00.430577 7f9d633d7700  5 -- op tracker -- , seq: 748519, time: 2014-06-12 14:48:00.430573, event: reached_pg, op: osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.430586 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2703 (0'0,173'2703] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2702 active] handle_message: 0x25e42100
2014-06-12 14:48:00.430597 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2703 (0'0,173'2703] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2702 active] sub_op_modify trans a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 v 173'2704 (transaction) 174
2014-06-12 14:48:00.430610 7f9d633d7700  5 -- op tracker -- , seq: 748519, time: 2014-06-12 14:48:00.430607, event: started, op: osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.430622 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2703 (0'0,173'2703] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2702 active] append_log log((0'0,173'2703], crt=0'0) [173'2704 (173'2703) modify   a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 by client.4313.0:146608 2014-06-12 08:45:53.787611]
2014-06-12 14:48:00.430642 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2704 (0'0,173'2704] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2702 active] add_log_entry 173'2704 (173'2703) modify   a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 by client.4313.0:146608 2014-06-12 08:45:53.787611
2014-06-12 14:48:00.430661 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2704 (0'0,173'2704] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2702 active] append_log  adding 1 keys
2014-06-12 14:48:00.430692 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2704, trimmed: 
2014-06-12 14:48:00.430716 7f9d633d7700  5 -- op tracker -- , seq: 748519, time: 2014-06-12 14:48:00.430711, event: started, op: osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.430724 7f9d633d7700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:48:00.430727 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:00.430728 7f9d633d7700  5 queue_op 0x58d1860 seq 2514 osr(3.1b7 0x276d1a0) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:48:00.430733 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2704 (0'0,173'2704] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2702 active] do_sub_op osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.430746 7f9d633d7700 10 osd.2 173 dequeue_op 0x25e42100 finish
2014-06-12 14:48:00.430749 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:00.430751 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:00.430752 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:48:00.430759 7f9d7a962700  5 _do_op 0x58d1860 seq 2514 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:48:00.430763 7f9d7a962700 10 _do_transaction on 0x591d970
2014-06-12 14:48:00.430768 7f9d7a962700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.430862 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.430877 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:00.430910 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:00.430921 7f9d7a962700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.430924 7f9d7a962700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:48:00.430928 7f9d7a962700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.430935 7f9d7a962700 15 _write 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 786432~1032192
2014-06-12 14:48:00.431028 7f9d7a962700 10 lookup_strip_header done  cid 3.1b7_head oid a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:00.431114 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:48:00.434549 7f9d7a962700 10 _generic_write 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 786432~1032192 = 0
2014-06-12 14:48:00.434685 7f9d7a962700 15 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:00.434715 7f9d7a962700 10 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 = 0
2014-06-12 14:48:00.434724 7f9d7a962700 15 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:00.434734 7f9d7a962700 10 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 = 0
2014-06-12 14:48:00.435113 7f9d7a962700 20 set_header setting 17402 cid 3.1b7_head oid a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 parent seq 0
2014-06-12 14:48:00.435264 7f9d7a962700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:48:00.435286 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:00.435295 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:48:00.441900 7f9d7a962700 10 _do_op 0x58d1860 seq 2514 r = 0, finisher 0x654b8a0 0
2014-06-12 14:48:00.441918 7f9d7a962700 10 _finish_op 0x58d1860 seq 2514 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:48:00.441923 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:48:00.441930 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:00.441938 7f9d7715b700  5 -- op tracker -- , seq: 748519, time: 2014-06-12 14:48:00.441926, event: commit_sent, op: osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.441950 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2704 (0'0,173'2704] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2702 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:00.441981 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.442020 7f9d7795c700  5 -- op tracker -- , seq: 748519, time: 2014-06-12 14:48:00.442012, event: sub_op_applied, op: osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.442029 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2704 (0'0,173'2704] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2703 active] sub_op_modify_applied on 0x591d880 op osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.442045 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2704 (0'0,173'2704] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2703 active] op_applied on primary on version 173'2704
2014-06-12 14:48:00.442055 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2704 (0'0,173'2704] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2703 active] op_applied on replica on version 173'2704
2014-06-12 14:48:00.442081 7f9d7795c700  5 -- op tracker -- , seq: 748519, time: 2014-06-12 14:48:00.442077, event: done, op: osd_sub_op(client.4313.0:146608 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2704 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.489334 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:00.489356 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:00.489359 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:00.489360 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:00.489362 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:00.509514 7f9d5a16f700  5 -- op tracker -- , seq: 748520, time: 2014-06-12 14:48:00.507416, event: header_read, op: osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.509537 7f9d5a16f700  5 -- op tracker -- , seq: 748520, time: 2014-06-12 14:48:00.507418, event: throttled, op: osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.509546 7f9d5a16f700  5 -- op tracker -- , seq: 748520, time: 2014-06-12 14:48:00.509469, event: all_read, op: osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.509554 7f9d5a16f700  5 -- op tracker -- , seq: 748520, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.509561 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:00.509568 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:00.509578 7f9d5a16f700 15 osd.2 173 enqueue_op 0x25e42400 prio 127 cost 1032999 latency 0.002161 osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.509635 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:00.509644 7f9d67be0700 10 osd.2 173 dequeue_op 0x25e42400 prio 127 cost 1032999 latency 0.002227 osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2483 (0'0,173'2483] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active]
2014-06-12 14:48:00.509667 7f9d67be0700  5 -- op tracker -- , seq: 748520, time: 2014-06-12 14:48:00.509664, event: reached_pg, op: osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.509678 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2483 (0'0,173'2483] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] handle_message: 0x25e42400
2014-06-12 14:48:00.509688 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2483 (0'0,173'2483] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] sub_op_modify trans 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 v 173'2484 (transaction) 174
2014-06-12 14:48:00.509703 7f9d67be0700  5 -- op tracker -- , seq: 748520, time: 2014-06-12 14:48:00.509700, event: started, op: osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.509717 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2483 (0'0,173'2483] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] append_log log((0'0,173'2483], crt=0'0) [173'2484 (173'2483) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144695 2014-06-12 08:45:12.516450]
2014-06-12 14:48:00.509735 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2484 (0'0,173'2484] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] add_log_entry 173'2484 (173'2483) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144695 2014-06-12 08:45:12.516450
2014-06-12 14:48:00.509752 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2484 (0'0,173'2484] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] append_log  adding 1 keys
2014-06-12 14:48:00.509777 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2484, trimmed: 
2014-06-12 14:48:00.509795 7f9d67be0700  5 -- op tracker -- , seq: 748520, time: 2014-06-12 14:48:00.509792, event: started, op: osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.509803 7f9d67be0700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:00.509806 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:00.509808 7f9d67be0700  5 queue_op 0xa2dad80 seq 2317 osr(3.4f4 0x276db70) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:48:00.509816 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:00.509812 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2484 (0'0,173'2484] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] do_sub_op osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.509822 7f9d67be0700 10 osd.2 173 dequeue_op 0x25e42400 finish
2014-06-12 14:48:00.509823 7f9d82972700  5 _do_op 0xa2dad80 seq 2317 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:00.509824 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:00.509826 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:00.509827 7f9d82972700 10 _do_transaction on 0x61cfcf0
2014-06-12 14:48:00.509832 7f9d82972700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.509933 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.509950 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:00.509994 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:00.510007 7f9d82972700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.510012 7f9d82972700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:00.510015 7f9d82972700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.510032 7f9d82972700 15 _write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 1376256~1032192
2014-06-12 14:48:00.510132 7f9d82972700 10 lookup_strip_header done  cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:00.510363 7f9d82972700 10 file_to_extents done 
2014-06-12 14:48:00.514187 7f9d82972700 10 _generic_write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 1376256~1032192 = 0
2014-06-12 14:48:00.514276 7f9d82972700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:00.514306 7f9d82972700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:48:00.514312 7f9d82972700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:00.514319 7f9d82972700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:48:00.514604 7f9d82972700 20 set_header setting 17360 cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 parent seq 0
2014-06-12 14:48:00.514705 7f9d82972700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:00.514727 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:00.514735 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:48:00.517659 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:00.519787 7f9d82972700 10 _do_op 0xa2dad80 seq 2317 r = 0, finisher 0x693a360 0
2014-06-12 14:48:00.519818 7f9d82972700 10 _finish_op 0xa2dad80 seq 2317 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:00.519844 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:00.519854 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:00.519856 7f9d7715b700  5 -- op tracker -- , seq: 748520, time: 2014-06-12 14:48:00.519843, event: commit_sent, op: osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.519868 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2484 (0'0,173'2484] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2482 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:00.519901 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.519941 7f9d7795c700  5 -- op tracker -- , seq: 748520, time: 2014-06-12 14:48:00.519935, event: sub_op_applied, op: osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.519952 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2484 (0'0,173'2484] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] sub_op_modify_applied on 0x61cfc00 op osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.519970 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2484 (0'0,173'2484] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] op_applied on primary on version 173'2484
2014-06-12 14:48:00.519978 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2484 (0'0,173'2484] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] op_applied on replica on version 173'2484
2014-06-12 14:48:00.519999 7f9d7795c700  5 -- op tracker -- , seq: 748520, time: 2014-06-12 14:48:00.519994, event: done, op: osd_sub_op(client.4303.0:144695 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2484 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.532414 7f9d5a16f700  5 -- op tracker -- , seq: 748521, time: 2014-06-12 14:48:00.530886, event: header_read, op: osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.532436 7f9d5a16f700  5 -- op tracker -- , seq: 748521, time: 2014-06-12 14:48:00.530889, event: throttled, op: osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.532449 7f9d5a16f700  5 -- op tracker -- , seq: 748521, time: 2014-06-12 14:48:00.532365, event: all_read, op: osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.532462 7f9d5a16f700  5 -- op tracker -- , seq: 748521, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.532470 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:00.532480 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:00.532492 7f9d5a16f700 15 osd.2 173 enqueue_op 0x25e43700 prio 127 cost 795431 latency 0.001606 osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.532549 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:00.532563 7f9d663dd700 10 osd.2 173 dequeue_op 0x25e43700 prio 127 cost 795431 latency 0.001676 osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2888 (0'0,173'2888] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2887 active]
2014-06-12 14:48:00.532589 7f9d663dd700  5 -- op tracker -- , seq: 748521, time: 2014-06-12 14:48:00.532585, event: reached_pg, op: osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.532608 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2888 (0'0,173'2888] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2887 active] handle_message: 0x25e43700
2014-06-12 14:48:00.532616 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2888 (0'0,173'2888] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2887 active] sub_op_modify trans bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 v 173'2889 (transaction) 174
2014-06-12 14:48:00.532625 7f9d663dd700  5 -- op tracker -- , seq: 748521, time: 2014-06-12 14:48:00.532623, event: started, op: osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.532641 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2888 (0'0,173'2888] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2887 active] append_log log((0'0,173'2888], crt=0'0) [173'2889 (173'820) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144063 2014-06-12 08:45:21.075596]
2014-06-12 14:48:00.532657 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2889 (0'0,173'2889] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2887 active] add_log_entry 173'2889 (173'820) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144063 2014-06-12 08:45:21.075596
2014-06-12 14:48:00.532681 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2889 (0'0,173'2889] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2887 active] append_log  adding 1 keys
2014-06-12 14:48:00.532712 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2889, trimmed: 
2014-06-12 14:48:00.532727 7f9d663dd700  5 -- op tracker -- , seq: 748521, time: 2014-06-12 14:48:00.532724, event: started, op: osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.532736 7f9d663dd700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:00.532740 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:00.532740 7f9d663dd700  5 queue_op 0x6fbe300 seq 2602 osr(3.129 0x276cf20) 796917 bytes   (queue has 1 ops and 796917 bytes)
2014-06-12 14:48:00.532753 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2889 (0'0,173'2889] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2887 active] do_sub_op osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.532762 7f9d663dd700 10 osd.2 173 dequeue_op 0x25e43700 finish
2014-06-12 14:48:00.532772 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:00.532774 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:00.532792 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:48:00.532811 7f9d7895e700  5 _do_op 0x6fbe300 seq 2602 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:48:00.532819 7f9d7895e700 10 _do_transaction on 0x62755f0
2014-06-12 14:48:00.532829 7f9d7895e700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:00.532997 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:00.533025 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:00.533074 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:00.533180 7f9d7895e700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:00.533189 7f9d7895e700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:48:00.533195 7f9d7895e700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:00.533211 7f9d7895e700 15 _write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 0~794624
2014-06-12 14:48:00.533374 7f9d7895e700 10 lookup_strip_header done  cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:00.533646 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:48:00.537126 7f9d7895e700 10 _generic_write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 0~794624 = 0
2014-06-12 14:48:00.537224 7f9d7895e700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:00.537255 7f9d7895e700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:00.537264 7f9d7895e700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:00.537274 7f9d7895e700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:00.537663 7f9d7895e700 20 set_header setting 15698 cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 parent seq 0
2014-06-12 14:48:00.537801 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:00.537824 7f9d7895e700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:48:00.537839 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:48:00.542648 7f9d7895e700 10 _do_op 0x6fbe300 seq 2602 r = 0, finisher 0x73c2ca0 0
2014-06-12 14:48:00.542676 7f9d7895e700 10 _finish_op 0x6fbe300 seq 2602 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:00.542685 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:48:00.542691 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:00.542737 7f9d7715b700  5 -- op tracker -- , seq: 748521, time: 2014-06-12 14:48:00.542719, event: commit_sent, op: osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.542767 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2889 (0'0,173'2889] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2887 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:00.542805 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.542891 7f9d7795c700  5 -- op tracker -- , seq: 748521, time: 2014-06-12 14:48:00.542874, event: sub_op_applied, op: osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.542920 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2889 (0'0,173'2889] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active] sub_op_modify_applied on 0x6275500 op osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.542959 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2889 (0'0,173'2889] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active] op_applied on primary on version 173'2889
2014-06-12 14:48:00.542965 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2889 (0'0,173'2889] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active] op_applied on replica on version 173'2889
2014-06-12 14:48:00.542978 7f9d7795c700  5 -- op tracker -- , seq: 748521, time: 2014-06-12 14:48:00.542975, event: done, op: osd_sub_op(client.4297.0:144063 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2889 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.615278 7f9d5a16f700  5 -- op tracker -- , seq: 748522, time: 2014-06-12 14:48:00.614090, event: header_read, op: osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.615294 7f9d5a16f700  5 -- op tracker -- , seq: 748522, time: 2014-06-12 14:48:00.614093, event: throttled, op: osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.615303 7f9d5a16f700  5 -- op tracker -- , seq: 748522, time: 2014-06-12 14:48:00.615229, event: all_read, op: osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.615312 7f9d5a16f700  5 -- op tracker -- , seq: 748522, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.615318 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:00.615326 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:00.615335 7f9d5a16f700 15 osd.2 173 enqueue_op 0x25e43a00 prio 127 cost 516903 latency 0.001245 osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.615374 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:00.615385 7f9d65bdc700 10 osd.2 173 dequeue_op 0x25e43a00 prio 127 cost 516903 latency 0.001294 osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2704 (0'0,173'2704] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2703 active]
2014-06-12 14:48:00.615410 7f9d65bdc700  5 -- op tracker -- , seq: 748522, time: 2014-06-12 14:48:00.615405, event: reached_pg, op: osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.615420 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2704 (0'0,173'2704] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2703 active] handle_message: 0x25e43a00
2014-06-12 14:48:00.615432 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2704 (0'0,173'2704] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2703 active] sub_op_modify trans a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 v 173'2705 (transaction) 174
2014-06-12 14:48:00.615448 7f9d65bdc700  5 -- op tracker -- , seq: 748522, time: 2014-06-12 14:48:00.615444, event: started, op: osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.615464 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2704 (0'0,173'2704] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2703 active] append_log log((0'0,173'2704], crt=0'0) [173'2705 (173'2704) modify   a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 by client.4313.0:146609 2014-06-12 08:45:53.794419]
2014-06-12 14:48:00.615488 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2705 (0'0,173'2705] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2703 active] add_log_entry 173'2705 (173'2704) modify   a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 by client.4313.0:146609 2014-06-12 08:45:53.794419
2014-06-12 14:48:00.615509 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2705 (0'0,173'2705] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2703 active] append_log  adding 1 keys
2014-06-12 14:48:00.615543 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2705, trimmed: 
2014-06-12 14:48:00.615572 7f9d65bdc700  5 -- op tracker -- , seq: 748522, time: 2014-06-12 14:48:00.615567, event: started, op: osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.615585 7f9d65bdc700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:48:00.615590 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:00.615592 7f9d65bdc700  5 queue_op 0x4b2e000 seq 2515 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:00.615602 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2705 (0'0,173'2705] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2703 active] do_sub_op osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.615617 7f9d65bdc700 10 osd.2 173 dequeue_op 0x25e43a00 finish
2014-06-12 14:48:00.615621 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:00.615624 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:00.615632 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:48:00.615637 7f9d83173700  5 _do_op 0x4b2e000 seq 2515 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:48:00.615642 7f9d83173700 10 _do_transaction on 0x21250b0
2014-06-12 14:48:00.615648 7f9d83173700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.615899 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.615921 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:00.615975 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:00.615997 7f9d83173700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.616002 7f9d83173700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:48:00.616008 7f9d83173700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.616021 7f9d83173700 15 _write 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 1818624~516096
2014-06-12 14:48:00.616175 7f9d83173700 10 lookup_strip_header done  cid 3.1b7_head oid a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:00.616399 7f9d83173700 10 file_to_extents done 
2014-06-12 14:48:00.618626 7f9d83173700 10 _generic_write 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 1818624~516096 = 0
2014-06-12 14:48:00.618698 7f9d83173700 15 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:00.618715 7f9d83173700 10 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 = 0
2014-06-12 14:48:00.618727 7f9d83173700 15 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:00.618736 7f9d83173700 10 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 = 0
2014-06-12 14:48:00.618953 7f9d83173700 20 set_header setting 17402 cid 3.1b7_head oid a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 parent seq 0
2014-06-12 14:48:00.619044 7f9d83173700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:48:00.619061 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:00.619071 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:48:00.621990 7f9d83173700 10 _do_op 0x4b2e000 seq 2515 r = 0, finisher 0x7fbee40 0
2014-06-12 14:48:00.622011 7f9d83173700 10 _finish_op 0x4b2e000 seq 2515 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:48:00.622060 7f9d7715b700  5 -- op tracker -- , seq: 748522, time: 2014-06-12 14:48:00.622049, event: commit_sent, op: osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.622074 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2705 (0'0,173'2705] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2703 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:00.622101 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.622136 7f9d7795c700  5 -- op tracker -- , seq: 748522, time: 2014-06-12 14:48:00.622129, event: sub_op_applied, op: osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.622147 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2705 (0'0,173'2705] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2704 active] sub_op_modify_applied on 0x2124fc0 op osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.622159 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2705 (0'0,173'2705] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2704 active] op_applied on primary on version 173'2705
2014-06-12 14:48:00.622112 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:48:00.622169 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:00.622165 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2705 (0'0,173'2705] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2704 active] op_applied on replica on version 173'2705
2014-06-12 14:48:00.622179 7f9d7795c700  5 -- op tracker -- , seq: 748522, time: 2014-06-12 14:48:00.622176, event: done, op: osd_sub_op(client.4313.0:146609 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2705 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.672116 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:00.769157 7f9d5a16f700  5 -- op tracker -- , seq: 748523, time: 2014-06-12 14:48:00.768048, event: header_read, op: osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.769172 7f9d5a16f700  5 -- op tracker -- , seq: 748523, time: 2014-06-12 14:48:00.768050, event: throttled, op: osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.769180 7f9d5a16f700  5 -- op tracker -- , seq: 748523, time: 2014-06-12 14:48:00.769110, event: all_read, op: osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.769189 7f9d5a16f700  5 -- op tracker -- , seq: 748523, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.769194 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:00.769362 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:00.769451 7f9d5a16f700 15 osd.2 173 enqueue_op 0x598fe00 prio 127 cost 516903 latency 0.001403 osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.769805 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:00.769815 7f9d62bd6700 10 osd.2 173 dequeue_op 0x598fe00 prio 127 cost 516903 latency 0.001767 osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2889 (0'0,173'2889] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active]
2014-06-12 14:48:00.769839 7f9d62bd6700  5 -- op tracker -- , seq: 748523, time: 2014-06-12 14:48:00.769835, event: reached_pg, op: osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.769847 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2889 (0'0,173'2889] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active] handle_message: 0x598fe00
2014-06-12 14:48:00.769910 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2889 (0'0,173'2889] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active] sub_op_modify trans bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 v 173'2890 (transaction) 174
2014-06-12 14:48:00.769927 7f9d62bd6700  5 -- op tracker -- , seq: 748523, time: 2014-06-12 14:48:00.769923, event: started, op: osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.769945 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2889 (0'0,173'2889] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active] append_log log((0'0,173'2889], crt=0'0) [173'2890 (173'2889) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144064 2014-06-12 08:45:21.086924]
2014-06-12 14:48:00.770039 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2890 (0'0,173'2890] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active] add_log_entry 173'2890 (173'2889) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144064 2014-06-12 08:45:21.086924
2014-06-12 14:48:00.770062 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2890 (0'0,173'2890] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active] append_log  adding 1 keys
2014-06-12 14:48:00.770138 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2890, trimmed: 
2014-06-12 14:48:00.770164 7f9d62bd6700  5 -- op tracker -- , seq: 748523, time: 2014-06-12 14:48:00.770159, event: started, op: osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.770174 7f9d62bd6700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:00.770178 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:00.770180 7f9d62bd6700  5 queue_op 0x5a73620 seq 2603 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:00.770191 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:48:00.770200 7f9d7f16b700  5 _do_op 0x5a73620 seq 2603 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:48:00.770205 7f9d7f16b700 10 _do_transaction on 0x2e637430
2014-06-12 14:48:00.770212 7f9d7f16b700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:00.770186 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2890 (0'0,173'2890] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active] do_sub_op osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.770267 7f9d62bd6700 10 osd.2 173 dequeue_op 0x598fe00 finish
2014-06-12 14:48:00.770271 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:00.770273 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:00.770357 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:00.770375 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:00.770430 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:00.770525 7f9d7f16b700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:00.770535 7f9d7f16b700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:48:00.770542 7f9d7f16b700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:00.770556 7f9d7f16b700 15 _write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 794624~516096
2014-06-12 14:48:00.770721 7f9d7f16b700 10 lookup_strip_header done  cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:00.770859 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:48:00.772925 7f9d7f16b700 10 _generic_write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 794624~516096 = 0
2014-06-12 14:48:00.772986 7f9d7f16b700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:00.773003 7f9d7f16b700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:00.773011 7f9d7f16b700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:00.773021 7f9d7f16b700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:00.773236 7f9d7f16b700 20 set_header setting 15698 cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 parent seq 0
2014-06-12 14:48:00.773318 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:00.773335 7f9d7f16b700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:48:00.773348 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:48:00.776581 7f9d7f16b700 10 _do_op 0x5a73620 seq 2603 r = 0, finisher 0x31d0e1e0 0
2014-06-12 14:48:00.776616 7f9d7f16b700 10 _finish_op 0x5a73620 seq 2603 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:00.776626 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:48:00.776634 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:00.776648 7f9d7795c700  5 -- op tracker -- , seq: 748523, time: 2014-06-12 14:48:00.776636, event: sub_op_applied, op: osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.776657 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2890 (0'0,173'2890] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active] sub_op_modify_applied on 0x2e637340 op osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.776686 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.776707 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2890 (0'0,173'2890] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active] op_applied on primary on version 173'2890
2014-06-12 14:48:00.776722 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2890 (0'0,173'2890] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active] op_applied on replica on version 173'2890
2014-06-12 14:48:00.776764 7f9d7715b700  5 -- op tracker -- , seq: 748523, time: 2014-06-12 14:48:00.776757, event: commit_sent, op: osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.776779 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2890 (0'0,173'2890] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2888 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:00.776799 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.776831 7f9d7715b700  5 -- op tracker -- , seq: 748523, time: 2014-06-12 14:48:00.776826, event: done, op: osd_sub_op(client.4297.0:144064 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2890 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.807385 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:00.810387 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:00.864735 7f9d5a16f700  5 -- op tracker -- , seq: 748524, time: 2014-06-12 14:48:00.862261, event: header_read, op: osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.864759 7f9d5a16f700  5 -- op tracker -- , seq: 748524, time: 2014-06-12 14:48:00.862263, event: throttled, op: osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.864768 7f9d5a16f700  5 -- op tracker -- , seq: 748524, time: 2014-06-12 14:48:00.864682, event: all_read, op: osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.864777 7f9d5a16f700  5 -- op tracker -- , seq: 748524, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.864788 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:00.864795 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:00.864804 7f9d5a16f700 15 osd.2 173 enqueue_op 0x598e800 prio 127 cost 1585959 latency 0.002543 osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.864863 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:00.864875 7f9d693e3700 10 osd.2 173 dequeue_op 0x598e800 prio 127 cost 1585959 latency 0.002614 osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2484 (0'0,173'2484] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active]
2014-06-12 14:48:00.864902 7f9d693e3700  5 -- op tracker -- , seq: 748524, time: 2014-06-12 14:48:00.864898, event: reached_pg, op: osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.864911 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2484 (0'0,173'2484] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] handle_message: 0x598e800
2014-06-12 14:48:00.864919 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2484 (0'0,173'2484] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] sub_op_modify trans c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 v 173'2485 (transaction) 174
2014-06-12 14:48:00.864930 7f9d693e3700  5 -- op tracker -- , seq: 748524, time: 2014-06-12 14:48:00.864927, event: started, op: osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.864943 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2484 (0'0,173'2484] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] append_log log((0'0,173'2484], crt=0'0) [173'2485 (173'578) modify   c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 by client.4290.0:143621 2014-06-12 08:45:06.109259]
2014-06-12 14:48:00.864963 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2485 (0'0,173'2485] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] add_log_entry 173'2485 (173'578) modify   c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 by client.4290.0:143621 2014-06-12 08:45:06.109259
2014-06-12 14:48:00.864982 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2485 (0'0,173'2485] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] append_log  adding 1 keys
2014-06-12 14:48:00.865009 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2485, trimmed: 
2014-06-12 14:48:00.865027 7f9d693e3700  5 -- op tracker -- , seq: 748524, time: 2014-06-12 14:48:00.865024, event: started, op: osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.865036 7f9d693e3700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:00.865039 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:00.865040 7f9d693e3700  5 queue_op 0x775f140 seq 2318 osr(3.4f4 0x276db70) 1587445 bytes   (queue has 1 ops and 1587445 bytes)
2014-06-12 14:48:00.865045 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2485 (0'0,173'2485] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] do_sub_op osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.865054 7f9d693e3700 10 osd.2 173 dequeue_op 0x598e800 finish
2014-06-12 14:48:00.865064 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:00.865066 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:00.865077 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:00.865092 7f9d7815d700  5 _do_op 0x775f140 seq 2318 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:00.865096 7f9d7815d700 10 _do_transaction on 0x26839b30
2014-06-12 14:48:00.865102 7f9d7815d700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.865223 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.865245 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:00.865280 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:00.865363 7f9d7815d700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.865369 7f9d7815d700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:00.865373 7f9d7815d700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.865385 7f9d7815d700 15 _write 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 0~1585152
2014-06-12 14:48:00.865508 7f9d7815d700 10 lookup_strip_header done  cid 3.4f4_head oid c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:00.865722 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:48:00.869719 7f9d7815d700 10 _generic_write 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 0~1585152 = 0
2014-06-12 14:48:00.869816 7f9d7815d700 15 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:00.869832 7f9d7815d700 10 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 = 0
2014-06-12 14:48:00.869839 7f9d7815d700 15 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:00.869843 7f9d7815d700 10 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 = 0
2014-06-12 14:48:00.870088 7f9d7815d700 20 set_header setting 15882 cid 3.4f4_head oid c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 parent seq 0
2014-06-12 14:48:00.870181 7f9d7815d700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:00.870192 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:00.870198 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:48:00.875560 7f9d7815d700 10 _do_op 0x775f140 seq 2318 r = 0, finisher 0x1e3bc2e0 0
2014-06-12 14:48:00.875583 7f9d7815d700 10 _finish_op 0x775f140 seq 2318 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:00.875591 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:00.875597 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:00.875613 7f9d7715b700  5 -- op tracker -- , seq: 748524, time: 2014-06-12 14:48:00.875594, event: commit_sent, op: osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.875624 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2485 (0'0,173'2485] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2483 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:00.875656 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.875721 7f9d7795c700  5 -- op tracker -- , seq: 748524, time: 2014-06-12 14:48:00.875714, event: sub_op_applied, op: osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.875734 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2485 (0'0,173'2485] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] sub_op_modify_applied on 0x26839a40 op osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.875753 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2485 (0'0,173'2485] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] op_applied on primary on version 173'2485
2014-06-12 14:48:00.875764 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2485 (0'0,173'2485] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] op_applied on replica on version 173'2485
2014-06-12 14:48:00.875784 7f9d7795c700  5 -- op tracker -- , seq: 748524, time: 2014-06-12 14:48:00.875781, event: done, op: osd_sub_op(client.4290.0:143621 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2485 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.973035 7f9d5a16f700  5 -- op tracker -- , seq: 748525, time: 2014-06-12 14:48:00.972156, event: header_read, op: osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.973056 7f9d5a16f700  5 -- op tracker -- , seq: 748525, time: 2014-06-12 14:48:00.972158, event: throttled, op: osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.973066 7f9d5a16f700  5 -- op tracker -- , seq: 748525, time: 2014-06-12 14:48:00.972981, event: all_read, op: osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.973075 7f9d5a16f700  5 -- op tracker -- , seq: 748525, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.973081 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:00.973089 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:00.973099 7f9d5a16f700 15 osd.2 173 enqueue_op 0x598f900 prio 127 cost 516903 latency 0.000942 osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.973134 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:00.973142 7f9d683e1700 10 osd.2 173 dequeue_op 0x598f900 prio 127 cost 516903 latency 0.000986 osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2705 (0'0,173'2705] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2704 active]
2014-06-12 14:48:00.973170 7f9d683e1700  5 -- op tracker -- , seq: 748525, time: 2014-06-12 14:48:00.973166, event: reached_pg, op: osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.973180 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2705 (0'0,173'2705] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2704 active] handle_message: 0x598f900
2014-06-12 14:48:00.973195 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2705 (0'0,173'2705] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2704 active] sub_op_modify trans a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 v 173'2706 (transaction) 174
2014-06-12 14:48:00.973217 7f9d683e1700  5 -- op tracker -- , seq: 748525, time: 2014-06-12 14:48:00.973213, event: started, op: osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.973234 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2705 (0'0,173'2705] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2704 active] append_log log((0'0,173'2705], crt=0'0) [173'2706 (173'2705) modify   a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 by client.4313.0:146610 2014-06-12 08:45:53.812480]
2014-06-12 14:48:00.973262 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2706 (0'0,173'2706] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2704 active] add_log_entry 173'2706 (173'2705) modify   a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 by client.4313.0:146610 2014-06-12 08:45:53.812480
2014-06-12 14:48:00.973287 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2706 (0'0,173'2706] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2704 active] append_log  adding 1 keys
2014-06-12 14:48:00.973322 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2706, trimmed: 
2014-06-12 14:48:00.973349 7f9d683e1700  5 -- op tracker -- , seq: 748525, time: 2014-06-12 14:48:00.973344, event: started, op: osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.973362 7f9d683e1700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:48:00.973367 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:00.973368 7f9d683e1700  5 queue_op 0x61f3bc0 seq 2516 osr(3.1b7 0x276d1a0) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:00.973383 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:48:00.973376 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2706 (0'0,173'2706] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2704 active] do_sub_op osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.973391 7f9d683e1700 10 osd.2 173 dequeue_op 0x598f900 finish
2014-06-12 14:48:00.973391 7f9d82171700  5 _do_op 0x61f3bc0 seq 2516 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:48:00.973395 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:00.973397 7f9d82171700 10 _do_transaction on 0x33f3b7b0
2014-06-12 14:48:00.973397 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:00.973404 7f9d82171700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.973545 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.973570 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:00.973644 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:00.973922 7f9d82171700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.973932 7f9d82171700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:48:00.973937 7f9d82171700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:00.973955 7f9d82171700 15 _write 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 2334720~516096
2014-06-12 14:48:00.974098 7f9d82171700 10 lookup_strip_header done  cid 3.1b7_head oid a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:00.974316 7f9d82171700 10 file_to_extents done 
2014-06-12 14:48:00.975703 7f9d82171700 10 _generic_write 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 2334720~516096 = 0
2014-06-12 14:48:00.975728 7f9d82171700 15 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:00.975743 7f9d82171700 10 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 = 0
2014-06-12 14:48:00.975747 7f9d82171700 15 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:00.975751 7f9d82171700 10 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 = 0
2014-06-12 14:48:00.975869 7f9d82171700 20 set_header setting 17402 cid 3.1b7_head oid a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 parent seq 0
2014-06-12 14:48:00.975909 7f9d82171700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:48:00.975920 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:00.975925 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:48:00.977917 7f9d82171700 10 _do_op 0x61f3bc0 seq 2516 r = 0, finisher 0x90efb20 0
2014-06-12 14:48:00.977931 7f9d82171700 10 _finish_op 0x61f3bc0 seq 2516 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:48:00.977937 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:48:00.977950 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:00.977969 7f9d7715b700  5 -- op tracker -- , seq: 748525, time: 2014-06-12 14:48:00.977960, event: commit_sent, op: osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.977988 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2706 (0'0,173'2706] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2704 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:00.978011 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.978046 7f9d7795c700  5 -- op tracker -- , seq: 748525, time: 2014-06-12 14:48:00.978032, event: sub_op_applied, op: osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.978054 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2706 (0'0,173'2706] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2705 active] sub_op_modify_applied on 0x33f3b6c0 op osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.978068 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2706 (0'0,173'2706] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2705 active] op_applied on primary on version 173'2706
2014-06-12 14:48:00.978074 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2706 (0'0,173'2706] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2705 active] op_applied on replica on version 173'2706
2014-06-12 14:48:00.978086 7f9d7795c700  5 -- op tracker -- , seq: 748525, time: 2014-06-12 14:48:00.978083, event: done, op: osd_sub_op(client.4313.0:146610 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2706 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.981379 7f9d5a16f700  5 -- op tracker -- , seq: 748526, time: 2014-06-12 14:48:00.980506, event: header_read, op: osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.981392 7f9d5a16f700  5 -- op tracker -- , seq: 748526, time: 2014-06-12 14:48:00.980507, event: throttled, op: osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.981401 7f9d5a16f700  5 -- op tracker -- , seq: 748526, time: 2014-06-12 14:48:00.981350, event: all_read, op: osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.981418 7f9d5a16f700  5 -- op tracker -- , seq: 748526, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.981424 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:00.981431 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:00.981438 7f9d5a16f700 15 osd.2 173 enqueue_op 0x598ed00 prio 127 cost 516903 latency 0.000931 osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.981464 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:00.981472 7f9d6abe6700 10 osd.2 173 dequeue_op 0x598ed00 prio 127 cost 516903 latency 0.000966 osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2485 (0'0,173'2485] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active]
2014-06-12 14:48:00.981493 7f9d6abe6700  5 -- op tracker -- , seq: 748526, time: 2014-06-12 14:48:00.981490, event: reached_pg, op: osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.981505 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2485 (0'0,173'2485] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] handle_message: 0x598ed00
2014-06-12 14:48:00.981513 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2485 (0'0,173'2485] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] sub_op_modify trans 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 v 173'2486 (transaction) 174
2014-06-12 14:48:00.981522 7f9d6abe6700  5 -- op tracker -- , seq: 748526, time: 2014-06-12 14:48:00.981520, event: started, op: osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.981534 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2485 (0'0,173'2485] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] append_log log((0'0,173'2485], crt=0'0) [173'2486 (173'2484) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144696 2014-06-12 08:45:13.481041]
2014-06-12 14:48:00.981552 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2486 (0'0,173'2486] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] add_log_entry 173'2486 (173'2484) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144696 2014-06-12 08:45:13.481041
2014-06-12 14:48:00.981568 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2486 (0'0,173'2486] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] append_log  adding 1 keys
2014-06-12 14:48:00.981589 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2486, trimmed: 
2014-06-12 14:48:00.981606 7f9d6abe6700  5 -- op tracker -- , seq: 748526, time: 2014-06-12 14:48:00.981603, event: started, op: osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.981618 7f9d6abe6700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:00.981621 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:00.981622 7f9d6abe6700  5 queue_op 0xcb33020 seq 2319 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:00.981626 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2486 (0'0,173'2486] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] do_sub_op osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.981635 7f9d6abe6700 10 osd.2 173 dequeue_op 0x598ed00 finish
2014-06-12 14:48:00.981645 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:00.981647 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:00.981658 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:00.981666 7f9d7e96a700  5 _do_op 0xcb33020 seq 2319 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:00.981669 7f9d7e96a700 10 _do_transaction on 0x1909f7b0
2014-06-12 14:48:00.981674 7f9d7e96a700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.981855 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.981872 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:00.981905 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:00.981920 7f9d7e96a700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.981924 7f9d7e96a700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:00.981927 7f9d7e96a700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:00.981938 7f9d7e96a700 15 _write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 2408448~516096
2014-06-12 14:48:00.982165 7f9d7e96a700 10 lookup_strip_header done  cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:00.982268 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:48:00.983347 7f9d7e96a700 10 _generic_write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 2408448~516096 = 0
2014-06-12 14:48:00.983373 7f9d7e96a700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:00.983381 7f9d7e96a700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:48:00.983385 7f9d7e96a700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:00.983390 7f9d7e96a700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:48:00.983515 7f9d7e96a700 20 set_header setting 17360 cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 parent seq 0
2014-06-12 14:48:00.983558 7f9d7e96a700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:00.983567 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:00.983573 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:48:00.985115 7f9d7e96a700 10 _do_op 0xcb33020 seq 2319 r = 0, finisher 0x90ee1c0 0
2014-06-12 14:48:00.985126 7f9d7e96a700 10 _finish_op 0xcb33020 seq 2319 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:00.985131 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:00.985148 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:00.985159 7f9d7795c700  5 -- op tracker -- , seq: 748526, time: 2014-06-12 14:48:00.985150, event: sub_op_applied, op: osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.985168 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2486 (0'0,173'2486] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] sub_op_modify_applied on 0x1909f6c0 op osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:00.985197 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.985212 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2486 (0'0,173'2486] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] op_applied on primary on version 173'2486
2014-06-12 14:48:00.985231 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2486 (0'0,173'2486] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] op_applied on replica on version 173'2486
2014-06-12 14:48:00.985256 7f9d7715b700  5 -- op tracker -- , seq: 748526, time: 2014-06-12 14:48:00.985249, event: commit_sent, op: osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:00.985277 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2486 (0'0,173'2486] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2484 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:00.985296 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:00.985328 7f9d7715b700  5 -- op tracker -- , seq: 748526, time: 2014-06-12 14:48:00.985322, event: done, op: osd_sub_op(client.4303.0:144696 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2486 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.003237 7f9d5a16f700  5 -- op tracker -- , seq: 748527, time: 2014-06-12 14:48:01.002303, event: header_read, op: osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.003252 7f9d5a16f700  5 -- op tracker -- , seq: 748527, time: 2014-06-12 14:48:01.002304, event: throttled, op: osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.003261 7f9d5a16f700  5 -- op tracker -- , seq: 748527, time: 2014-06-12 14:48:01.003192, event: all_read, op: osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.003270 7f9d5a16f700  5 -- op tracker -- , seq: 748527, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.003275 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:01.003281 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:01.003290 7f9d5a16f700 15 osd.2 173 enqueue_op 0x124e2300 prio 127 cost 516903 latency 0.000987 osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.003317 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:01.003328 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x124e2300 prio 127 cost 516903 latency 0.001025 osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2890 (0'0,173'2890] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2889 active]
2014-06-12 14:48:01.003355 7f9d6a3e5700  5 -- op tracker -- , seq: 748527, time: 2014-06-12 14:48:01.003350, event: reached_pg, op: osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.003364 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2890 (0'0,173'2890] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2889 active] handle_message: 0x124e2300
2014-06-12 14:48:01.003380 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2890 (0'0,173'2890] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2889 active] sub_op_modify trans bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 v 173'2891 (transaction) 174
2014-06-12 14:48:01.003406 7f9d6a3e5700  5 -- op tracker -- , seq: 748527, time: 2014-06-12 14:48:01.003402, event: started, op: osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.003421 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2890 (0'0,173'2890] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2889 active] append_log log((0'0,173'2890], crt=0'0) [173'2891 (173'2890) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144065 2014-06-12 08:45:21.340995]
2014-06-12 14:48:01.003446 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2891 (0'0,173'2891] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2889 active] add_log_entry 173'2891 (173'2890) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144065 2014-06-12 08:45:21.340995
2014-06-12 14:48:01.003470 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2891 (0'0,173'2891] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2889 active] append_log  adding 1 keys
2014-06-12 14:48:01.003504 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2891, trimmed: 
2014-06-12 14:48:01.003529 7f9d6a3e5700  5 -- op tracker -- , seq: 748527, time: 2014-06-12 14:48:01.003524, event: started, op: osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.003539 7f9d6a3e5700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:01.003544 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:01.003546 7f9d6a3e5700  5 queue_op 0x1d772240 seq 2604 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:01.003553 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2891 (0'0,173'2891] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2889 active] do_sub_op osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.003561 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:48:01.003569 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x124e2300 finish
2014-06-12 14:48:01.003572 7f9d8116f700  5 _do_op 0x1d772240 seq 2604 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:48:01.003573 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:01.003576 7f9d8116f700 10 _do_transaction on 0x1ebedb30
2014-06-12 14:48:01.003576 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:01.003581 7f9d8116f700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:01.003714 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:01.003735 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:01.003790 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:01.003813 7f9d8116f700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:01.003820 7f9d8116f700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:48:01.003826 7f9d8116f700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:01.003954 7f9d8116f700 15 _write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 1310720~516096
2014-06-12 14:48:01.004102 7f9d8116f700 10 lookup_strip_header done  cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:01.004203 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:48:01.005920 7f9d8116f700 10 _generic_write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 1310720~516096 = 0
2014-06-12 14:48:01.005964 7f9d8116f700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:01.005976 7f9d8116f700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:01.005983 7f9d8116f700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:01.005988 7f9d8116f700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:01.006132 7f9d8116f700 20 set_header setting 15698 cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 parent seq 0
2014-06-12 14:48:01.006198 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:01.006209 7f9d8116f700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:48:01.006223 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:48:01.008240 7f9d8116f700 10 _do_op 0x1d772240 seq 2604 r = 0, finisher 0x4e9d6a0 0
2014-06-12 14:48:01.008255 7f9d8116f700 10 _finish_op 0x1d772240 seq 2604 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:01.008268 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:48:01.008277 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:01.008284 7f9d7715b700  5 -- op tracker -- , seq: 748527, time: 2014-06-12 14:48:01.008273, event: commit_sent, op: osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.008301 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2891 (0'0,173'2891] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2889 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:01.008330 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:01.008366 7f9d7795c700  5 -- op tracker -- , seq: 748527, time: 2014-06-12 14:48:01.008361, event: sub_op_applied, op: osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.008374 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2891 (0'0,173'2891] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2890 active] sub_op_modify_applied on 0x1ebeda40 op osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.008385 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2891 (0'0,173'2891] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2890 active] op_applied on primary on version 173'2891
2014-06-12 14:48:01.008392 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2891 (0'0,173'2891] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2890 active] op_applied on replica on version 173'2891
2014-06-12 14:48:01.008404 7f9d7795c700  5 -- op tracker -- , seq: 748527, time: 2014-06-12 14:48:01.008402, event: done, op: osd_sub_op(client.4297.0:144065 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2891 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.105113 7f9d5a16f700  5 -- op tracker -- , seq: 748528, time: 2014-06-12 14:48:01.102659, event: header_read, op: osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.105127 7f9d5a16f700  5 -- op tracker -- , seq: 748528, time: 2014-06-12 14:48:01.102661, event: throttled, op: osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.105136 7f9d5a16f700  5 -- op tracker -- , seq: 748528, time: 2014-06-12 14:48:01.105069, event: all_read, op: osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.105145 7f9d5a16f700  5 -- op tracker -- , seq: 748528, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.105151 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:01.105158 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:01.105167 7f9d5a16f700 15 osd.2 173 enqueue_op 0x124e3700 prio 127 cost 1032999 latency 0.002508 osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.105199 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:01.105205 7f9d66bde700 10 osd.2 173 dequeue_op 0x124e3700 prio 127 cost 1032999 latency 0.002546 osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2486 (0'0,173'2486] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2485 active]
2014-06-12 14:48:01.105229 7f9d66bde700  5 -- op tracker -- , seq: 748528, time: 2014-06-12 14:48:01.105225, event: reached_pg, op: osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.105240 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2486 (0'0,173'2486] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2485 active] handle_message: 0x124e3700
2014-06-12 14:48:01.105253 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2486 (0'0,173'2486] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2485 active] sub_op_modify trans c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 v 173'2487 (transaction) 174
2014-06-12 14:48:01.105270 7f9d66bde700  5 -- op tracker -- , seq: 748528, time: 2014-06-12 14:48:01.105266, event: started, op: osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.105284 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2486 (0'0,173'2486] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2485 active] append_log log((0'0,173'2486], crt=0'0) [173'2487 (173'2485) modify   c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 by client.4290.0:143622 2014-06-12 08:45:06.120056]
2014-06-12 14:48:01.105308 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2487 (0'0,173'2487] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2485 active] add_log_entry 173'2487 (173'2485) modify   c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 by client.4290.0:143622 2014-06-12 08:45:06.120056
2014-06-12 14:48:01.105331 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2487 (0'0,173'2487] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2485 active] append_log  adding 1 keys
2014-06-12 14:48:01.105363 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2487, trimmed: 
2014-06-12 14:48:01.105389 7f9d66bde700  5 -- op tracker -- , seq: 748528, time: 2014-06-12 14:48:01.105384, event: started, op: osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.105399 7f9d66bde700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:01.105404 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:01.105405 7f9d66bde700  5 queue_op 0x53c3560 seq 2320 osr(3.4f4 0x276db70) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:48:01.105412 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2487 (0'0,173'2487] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2485 active] do_sub_op osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.105433 7f9d66bde700 10 osd.2 173 dequeue_op 0x124e3700 finish
2014-06-12 14:48:01.105437 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:01.105440 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:01.105448 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:01.105461 7f9d7d167700  5 _do_op 0x53c3560 seq 2320 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:01.105465 7f9d7d167700 10 _do_transaction on 0x5cb4470
2014-06-12 14:48:01.105471 7f9d7d167700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:01.105632 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:01.105654 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:01.105710 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:01.105732 7f9d7d167700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:01.105739 7f9d7d167700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:01.105745 7f9d7d167700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:01.105761 7f9d7d167700 15 _write 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 1585152~1032192
2014-06-12 14:48:01.105946 7f9d7d167700 10 lookup_strip_header done  cid 3.4f4_head oid c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:01.106071 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:48:01.110471 7f9d7d167700 10 _generic_write 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 1585152~1032192 = 0
2014-06-12 14:48:01.110567 7f9d7d167700 15 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:01.110587 7f9d7d167700 10 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 = 0
2014-06-12 14:48:01.110598 7f9d7d167700 15 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:01.110607 7f9d7d167700 10 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 = 0
2014-06-12 14:48:01.110931 7f9d7d167700 20 set_header setting 15882 cid 3.4f4_head oid c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 parent seq 0
2014-06-12 14:48:01.111041 7f9d7d167700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:01.111059 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:01.111073 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:48:01.115410 7f9d7d167700 10 _do_op 0x53c3560 seq 2320 r = 0, finisher 0x7efe180 0
2014-06-12 14:48:01.115437 7f9d7d167700 10 _finish_op 0x53c3560 seq 2320 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:01.115443 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:01.115447 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:01.115461 7f9d7715b700  5 -- op tracker -- , seq: 748528, time: 2014-06-12 14:48:01.115448, event: commit_sent, op: osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.115471 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2487 (0'0,173'2487] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2485 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:01.115502 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:01.115541 7f9d7795c700  5 -- op tracker -- , seq: 748528, time: 2014-06-12 14:48:01.115534, event: sub_op_applied, op: osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.115552 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2487 (0'0,173'2487] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2486 active] sub_op_modify_applied on 0x5cb4380 op osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.115572 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2487 (0'0,173'2487] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2486 active] op_applied on primary on version 173'2487
2014-06-12 14:48:01.115581 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2487 (0'0,173'2487] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2486 active] op_applied on replica on version 173'2487
2014-06-12 14:48:01.115602 7f9d7795c700  5 -- op tracker -- , seq: 748528, time: 2014-06-12 14:48:01.115598, event: done, op: osd_sub_op(client.4290.0:143622 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2487 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.166158 7f9d5a16f700  5 -- op tracker -- , seq: 748529, time: 2014-06-12 14:48:01.162742, event: header_read, op: osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.166197 7f9d5a16f700  5 -- op tracker -- , seq: 748529, time: 2014-06-12 14:48:01.162745, event: throttled, op: osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.166217 7f9d5a16f700  5 -- op tracker -- , seq: 748529, time: 2014-06-12 14:48:01.166072, event: all_read, op: osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.166249 7f9d5a16f700  5 -- op tracker -- , seq: 748529, time: 0.000000, event: dispatched, op: osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.166259 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:01.166273 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:01.166288 7f9d5a16f700 15 osd.2 173 enqueue_op 0x68a7e00 prio 127 cost 1344295 latency 0.003545 osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.166368 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:01.166390 7f9d69be4700 10 osd.2 173 dequeue_op 0x68a7e00 prio 127 cost 1344295 latency 0.003648 osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.1b7( v 173'2706 (0'0,173'2706] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2705 active]
2014-06-12 14:48:01.166428 7f9d69be4700  5 -- op tracker -- , seq: 748529, time: 2014-06-12 14:48:01.166422, event: reached_pg, op: osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.166444 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2706 (0'0,173'2706] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2705 active] handle_message: 0x68a7e00
2014-06-12 14:48:01.166459 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2706 (0'0,173'2706] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2705 active] sub_op_modify trans a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 v 173'2707 (transaction) 174
2014-06-12 14:48:01.166480 7f9d69be4700  5 -- op tracker -- , seq: 748529, time: 2014-06-12 14:48:01.166475, event: started, op: osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.166500 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2706 (0'0,173'2706] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2705 active] append_log log((0'0,173'2706], crt=0'0) [173'2707 (173'2706) modify   a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 by client.4313.0:146611 2014-06-12 08:45:54.584953]
2014-06-12 14:48:01.166532 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2707 (0'0,173'2707] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2705 active] add_log_entry 173'2707 (173'2706) modify   a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 by client.4313.0:146611 2014-06-12 08:45:54.584953
2014-06-12 14:48:01.166565 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2707 (0'0,173'2707] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2705 active] append_log  adding 1 keys
2014-06-12 14:48:01.166607 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2707, trimmed: 
2014-06-12 14:48:01.166641 7f9d69be4700  5 -- op tracker -- , seq: 748529, time: 2014-06-12 14:48:01.166634, event: started, op: osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.166656 7f9d69be4700  5 queue_transactions existing osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:48:01.166661 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:01.166663 7f9d69be4700  5 queue_op 0x6064c60 seq 2517 osr(3.1b7 0x276d1a0) 1345781 bytes   (queue has 1 ops and 1345781 bytes)
2014-06-12 14:48:01.166672 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2707 (0'0,173'2707] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2705 active] do_sub_op osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.166691 7f9d69be4700 10 osd.2 173 dequeue_op 0x68a7e00 finish
2014-06-12 14:48:01.166696 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:01.166692 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5479c20 (1 active)
2014-06-12 14:48:01.166700 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:01.166704 7f9d7915f700  5 _do_op 0x6064c60 seq 2517 osr(3.1b7 0x276d1a0)/0x276d1a0 start
2014-06-12 14:48:01.166710 7f9d7915f700 10 _do_transaction on 0x66bfb30
2014-06-12 14:48:01.166719 7f9d7915f700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:01.166853 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:01.166880 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:01.166914 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:01.166951 7f9d7915f700 15 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:01.166955 7f9d7915f700 10 _omap_rmkeys meta/1fe179a5/pglog_3.1b7/0//-1 r = 0
2014-06-12 14:48:01.166959 7f9d7915f700 15 _omap_setkeys meta/1fe179a5/pglog_3.1b7/0//-1
2014-06-12 14:48:01.166970 7f9d7915f700 15 _write 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 2850816~1343488
2014-06-12 14:48:01.167235 7f9d7915f700 10 lookup_strip_header done  cid 3.1b7_head oid a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:01.167373 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:48:01.170841 7f9d7915f700 10 _generic_write 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 2850816~1343488 = 0
2014-06-12 14:48:01.170918 7f9d7915f700 15 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:01.170935 7f9d7915f700 10 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 = 0
2014-06-12 14:48:01.170939 7f9d7915f700 15 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3
2014-06-12 14:48:01.170943 7f9d7915f700 10 _setattrs 3.1b7_head/a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 = 0
2014-06-12 14:48:01.171182 7f9d7915f700 20 set_header setting 17402 cid 3.1b7_head oid a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 parent seq 0
2014-06-12 14:48:01.171269 7f9d7915f700 20 set_header setting 2178 cid meta oid 1fe179a5/pglog_3.1b7/0//-1 parent seq 0
2014-06-12 14:48:01.171280 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:01.171286 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:48:01.177110 7f9d7915f700 10 _do_op 0x6064c60 seq 2517 r = 0, finisher 0x6704d20 0
2014-06-12 14:48:01.177155 7f9d7915f700 10 _finish_op 0x6064c60 seq 2517 osr(3.1b7 0x276d1a0)/0x276d1a0
2014-06-12 14:48:01.177168 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5479c20 (0 active)
2014-06-12 14:48:01.177179 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:01.177194 7f9d7715b700  5 -- op tracker -- , seq: 748529, time: 2014-06-12 14:48:01.177176, event: commit_sent, op: osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.177206 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2707 (0'0,173'2707] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2705 active] sub_op_modify_commit on op osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:01.177233 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:01.177301 7f9d7795c700  5 -- op tracker -- , seq: 748529, time: 2014-06-12 14:48:01.177282, event: sub_op_applied, op: osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.177330 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2707 (0'0,173'2707] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2706 active] sub_op_modify_applied on 0x66bfa40 op osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.177375 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2707 (0'0,173'2707] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2706 active] op_applied on primary on version 173'2707
2014-06-12 14:48:01.177401 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.1b7( v 173'2707 (0'0,173'2707] local-les=147 n=301 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2706 active] op_applied on replica on version 173'2707
2014-06-12 14:48:01.177443 7f9d7795c700  5 -- op tracker -- , seq: 748529, time: 2014-06-12 14:48:01.177433, event: done, op: osd_sub_op(client.4313.0:146611 3.1b7 a17969b7/rbd_data.10b06b8b4567.0000000000000fd0/head//3 [] v 173'2707 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.179642 7f9d5a16f700  5 -- op tracker -- , seq: 748530, time: 2014-06-12 14:48:01.178398, event: header_read, op: osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.179658 7f9d5a16f700  5 -- op tracker -- , seq: 748530, time: 2014-06-12 14:48:01.178401, event: throttled, op: osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.179680 7f9d5a16f700  5 -- op tracker -- , seq: 748530, time: 2014-06-12 14:48:01.179579, event: all_read, op: osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.179709 7f9d5a16f700  5 -- op tracker -- , seq: 748530, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.179715 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:01.179723 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:01.179740 7f9d5a16f700 15 osd.2 173 enqueue_op 0x68a7400 prio 127 cost 516903 latency 0.001342 osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.179769 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:01.179777 7f9d64bda700 10 osd.2 173 dequeue_op 0x68a7400 prio 127 cost 516903 latency 0.001379 osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2487 (0'0,173'2487] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2486 active]
2014-06-12 14:48:01.179804 7f9d64bda700  5 -- op tracker -- , seq: 748530, time: 2014-06-12 14:48:01.179793, event: reached_pg, op: osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.179819 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2487 (0'0,173'2487] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2486 active] handle_message: 0x68a7400
2014-06-12 14:48:01.179827 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2487 (0'0,173'2487] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2486 active] sub_op_modify trans 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 v 173'2488 (transaction) 174
2014-06-12 14:48:01.179837 7f9d64bda700  5 -- op tracker -- , seq: 748530, time: 2014-06-12 14:48:01.179835, event: started, op: osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.179854 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2487 (0'0,173'2487] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2486 active] append_log log((0'0,173'2487], crt=0'0) [173'2488 (173'2486) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144697 2014-06-12 08:45:14.259709]
2014-06-12 14:48:01.179894 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2488 (0'0,173'2488] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2486 active] add_log_entry 173'2488 (173'2486) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144697 2014-06-12 08:45:14.259709
2014-06-12 14:48:01.179926 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2488 (0'0,173'2488] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2486 active] append_log  adding 1 keys
2014-06-12 14:48:01.179969 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2488, trimmed: 
2014-06-12 14:48:01.180001 7f9d64bda700  5 -- op tracker -- , seq: 748530, time: 2014-06-12 14:48:01.179994, event: started, op: osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.180033 7f9d64bda700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:01.180045 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:01.180046 7f9d64bda700  5 queue_op 0x3360300 seq 2321 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:01.180051 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2488 (0'0,173'2488] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2486 active] do_sub_op osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.180061 7f9d64bda700 10 osd.2 173 dequeue_op 0x68a7400 finish
2014-06-12 14:48:01.180057 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:01.180063 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:01.180065 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:01.180066 7f9d8016d700  5 _do_op 0x3360300 seq 2321 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:01.180072 7f9d8016d700 10 _do_transaction on 0x232ab70
2014-06-12 14:48:01.180079 7f9d8016d700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:01.180232 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:01.180277 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:01.180388 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:01.180428 7f9d8016d700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:01.180445 7f9d8016d700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:01.180460 7f9d8016d700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:01.180495 7f9d8016d700 15 _write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 2924544~516096
2014-06-12 14:48:01.180684 7f9d8016d700 10 lookup_strip_header done  cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:01.180724 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:48:01.182459 7f9d8016d700 10 _generic_write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 2924544~516096 = 0
2014-06-12 14:48:01.182507 7f9d8016d700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:01.182521 7f9d8016d700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:48:01.182529 7f9d8016d700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:01.182539 7f9d8016d700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:48:01.182685 7f9d8016d700 20 set_header setting 17360 cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 parent seq 0
2014-06-12 14:48:01.182735 7f9d8016d700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:01.182766 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:01.182776 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:48:01.184442 7f9d8016d700 10 _do_op 0x3360300 seq 2321 r = 0, finisher 0x78702a0 0
2014-06-12 14:48:01.184465 7f9d8016d700 10 _finish_op 0x3360300 seq 2321 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:01.184473 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:01.184477 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:01.184513 7f9d7715b700  5 -- op tracker -- , seq: 748530, time: 2014-06-12 14:48:01.184501, event: commit_sent, op: osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.184527 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2488 (0'0,173'2488] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2486 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:01.184553 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:01.184585 7f9d7795c700  5 -- op tracker -- , seq: 748530, time: 2014-06-12 14:48:01.184578, event: sub_op_applied, op: osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.184597 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2488 (0'0,173'2488] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2487 active] sub_op_modify_applied on 0x232aa80 op osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.184609 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2488 (0'0,173'2488] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2487 active] op_applied on primary on version 173'2488
2014-06-12 14:48:01.184615 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2488 (0'0,173'2488] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2487 active] op_applied on replica on version 173'2488
2014-06-12 14:48:01.184634 7f9d7795c700  5 -- op tracker -- , seq: 748530, time: 2014-06-12 14:48:01.184632, event: done, op: osd_sub_op(client.4303.0:144697 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2488 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.196666 7f9d5a16f700  5 -- op tracker -- , seq: 748531, time: 2014-06-12 14:48:01.195950, event: header_read, op: osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.196685 7f9d5a16f700  5 -- op tracker -- , seq: 748531, time: 2014-06-12 14:48:01.195952, event: throttled, op: osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.196697 7f9d5a16f700  5 -- op tracker -- , seq: 748531, time: 2014-06-12 14:48:01.196614, event: all_read, op: osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.196732 7f9d5a16f700  5 -- op tracker -- , seq: 748531, time: 0.000000, event: dispatched, op: osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.196739 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:01.196754 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:01.196765 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329eac00 prio 127 cost 332583 latency 0.000814 osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.196815 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:01.196835 7f9d653db700 10 osd.2 173 dequeue_op 0x329eac00 prio 127 cost 332583 latency 0.000884 osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2726 (0'0,173'2726] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active]
2014-06-12 14:48:01.196901 7f9d653db700  5 -- op tracker -- , seq: 748531, time: 2014-06-12 14:48:01.196890, event: reached_pg, op: osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.196929 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2726 (0'0,173'2726] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] handle_message: 0x329eac00
2014-06-12 14:48:01.196969 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2726 (0'0,173'2726] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] sub_op_modify trans c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 v 173'2727 (transaction) 174
2014-06-12 14:48:01.197008 7f9d653db700  5 -- op tracker -- , seq: 748531, time: 2014-06-12 14:48:01.196999, event: started, op: osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.197047 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2726 (0'0,173'2726] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] append_log log((0'0,173'2726], crt=0'0) [173'2727 (173'761) modify   c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 by client.4311.0:139807 2014-06-12 08:45:21.152957]
2014-06-12 14:48:01.197096 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2727 (0'0,173'2727] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] add_log_entry 173'2727 (173'761) modify   c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 by client.4311.0:139807 2014-06-12 08:45:21.152957
2014-06-12 14:48:01.197150 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2727 (0'0,173'2727] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] append_log  adding 1 keys
2014-06-12 14:48:01.197226 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2727, trimmed: 
2014-06-12 14:48:01.197279 7f9d653db700  5 -- op tracker -- , seq: 748531, time: 2014-06-12 14:48:01.197268, event: started, op: osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.197300 7f9d653db700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:48:01.197307 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:01.197310 7f9d653db700  5 queue_op 0x138cd680 seq 2474 osr(3.21c 0x276d020) 334069 bytes   (queue has 1 ops and 334069 bytes)
2014-06-12 14:48:01.197322 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2727 (0'0,173'2727] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] do_sub_op osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.197356 7f9d653db700 10 osd.2 173 dequeue_op 0x329eac00 finish
2014-06-12 14:48:01.197365 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:01.197359 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (1 active)
2014-06-12 14:48:01.197371 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:01.197376 7f9d79960700  5 _do_op 0x138cd680 seq 2474 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:48:01.197382 7f9d79960700 10 _do_transaction on 0x20c43270
2014-06-12 14:48:01.197397 7f9d79960700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:01.197522 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:01.197610 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:01.197681 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:01.197698 7f9d79960700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:01.197701 7f9d79960700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:48:01.197705 7f9d79960700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:01.197716 7f9d79960700 15 _write 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 0~331776
2014-06-12 14:48:01.197830 7f9d79960700 10 lookup_strip_header done  cid 3.21c_head oid c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:01.198004 7f9d79960700 10 file_to_extents done 
2014-06-12 14:48:01.198716 7f9d79960700 10 _generic_write 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 0~331776 = 0
2014-06-12 14:48:01.198750 7f9d79960700 15 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:01.198758 7f9d79960700 10 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 = 0
2014-06-12 14:48:01.198762 7f9d79960700 15 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:01.198766 7f9d79960700 10 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 = 0
2014-06-12 14:48:01.198880 7f9d79960700 20 set_header setting 16171 cid 3.21c_head oid c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 parent seq 0
2014-06-12 14:48:01.198918 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:01.198926 7f9d79960700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:48:01.198931 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:48:01.200148 7f9d79960700 10 _do_op 0x138cd680 seq 2474 r = 0, finisher 0x772ba00 0
2014-06-12 14:48:01.200158 7f9d79960700 10 _finish_op 0x138cd680 seq 2474 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:48:01.200162 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (0 active)
2014-06-12 14:48:01.200166 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:01.200214 7f9d7715b700  5 -- op tracker -- , seq: 748531, time: 2014-06-12 14:48:01.200197, event: commit_sent, op: osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.200227 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2727 (0'0,173'2727] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2725 active] sub_op_modify_commit on op osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:01.200246 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:01.200285 7f9d7795c700  5 -- op tracker -- , seq: 748531, time: 2014-06-12 14:48:01.200269, event: sub_op_applied, op: osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.200306 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2727 (0'0,173'2727] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] sub_op_modify_applied on 0x20c43180 op osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.200325 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2727 (0'0,173'2727] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] op_applied on primary on version 173'2727
2014-06-12 14:48:01.200336 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2727 (0'0,173'2727] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] op_applied on replica on version 173'2727
2014-06-12 14:48:01.200367 7f9d7795c700  5 -- op tracker -- , seq: 748531, time: 2014-06-12 14:48:01.200362, event: done, op: osd_sub_op(client.4311.0:139807 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2727 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.272580 7f9d5a16f700  5 -- op tracker -- , seq: 748532, time: 2014-06-12 14:48:01.271887, event: header_read, op: osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.272611 7f9d5a16f700  5 -- op tracker -- , seq: 748532, time: 2014-06-12 14:48:01.271890, event: throttled, op: osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.272616 7f9d5a16f700  5 -- op tracker -- , seq: 748532, time: 2014-06-12 14:48:01.272527, event: all_read, op: osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.272624 7f9d5a16f700  5 -- op tracker -- , seq: 748532, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.272629 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:01.272634 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:01.272641 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329eb200 prio 127 cost 516903 latency 0.000754 osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.272718 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:01.272741 7f9d623d5700 10 osd.2 173 dequeue_op 0x329eb200 prio 127 cost 516903 latency 0.000854 osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2891 (0'0,173'2891] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2890 active]
2014-06-12 14:48:01.272788 7f9d623d5700  5 -- op tracker -- , seq: 748532, time: 2014-06-12 14:48:01.272781, event: reached_pg, op: osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.272805 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2891 (0'0,173'2891] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2890 active] handle_message: 0x329eb200
2014-06-12 14:48:01.272821 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2891 (0'0,173'2891] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2890 active] sub_op_modify trans bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 v 173'2892 (transaction) 174
2014-06-12 14:48:01.272843 7f9d623d5700  5 -- op tracker -- , seq: 748532, time: 2014-06-12 14:48:01.272838, event: started, op: osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.272868 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2891 (0'0,173'2891] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2890 active] append_log log((0'0,173'2891], crt=0'0) [173'2892 (173'2891) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144066 2014-06-12 08:45:21.512690]
2014-06-12 14:48:01.272904 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2892 (0'0,173'2892] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2890 active] add_log_entry 173'2892 (173'2891) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144066 2014-06-12 08:45:21.512690
2014-06-12 14:48:01.272936 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2892 (0'0,173'2892] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2890 active] append_log  adding 1 keys
2014-06-12 14:48:01.272986 7f9d623d5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2892, trimmed: 
2014-06-12 14:48:01.273023 7f9d623d5700  5 -- op tracker -- , seq: 748532, time: 2014-06-12 14:48:01.273015, event: started, op: osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.273040 7f9d623d5700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:01.273046 7f9d623d5700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:01.273048 7f9d623d5700  5 queue_op 0xe2e7aa0 seq 2605 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:01.273057 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2892 (0'0,173'2892] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2890 active] do_sub_op osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.273077 7f9d623d5700 10 osd.2 173 dequeue_op 0x329eb200 finish
2014-06-12 14:48:01.273082 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:01.273085 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:01.273093 7f9d7d968700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:48:01.273109 7f9d7d968700  5 _do_op 0xe2e7aa0 seq 2605 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:48:01.273117 7f9d7d968700 10 _do_transaction on 0x587feb0
2014-06-12 14:48:01.273127 7f9d7d968700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:01.273261 7f9d7d968700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:01.273360 7f9d7d968700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:01.273404 7f9d7d968700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:01.273421 7f9d7d968700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:01.273426 7f9d7d968700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:48:01.273430 7f9d7d968700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:01.273442 7f9d7d968700 15 _write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 1826816~516096
2014-06-12 14:48:01.273534 7f9d7d968700 10 lookup_strip_header done  cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:01.273737 7f9d7d968700 10 file_to_extents done 
2014-06-12 14:48:01.274935 7f9d7d968700 10 _generic_write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 1826816~516096 = 0
2014-06-12 14:48:01.274985 7f9d7d968700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:01.274995 7f9d7d968700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:01.275000 7f9d7d968700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:01.275004 7f9d7d968700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:01.275151 7f9d7d968700 20 set_header setting 15698 cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 parent seq 0
2014-06-12 14:48:01.275205 7f9d7d968700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:01.275216 7f9d7d968700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:48:01.275222 7f9d7d968700  5 submit_transaction r = 0
2014-06-12 14:48:01.288731 7f9d7d968700 10 _do_op 0xe2e7aa0 seq 2605 r = 0, finisher 0x6570140 0
2014-06-12 14:48:01.288774 7f9d7715b700  5 -- op tracker -- , seq: 748532, time: 2014-06-12 14:48:01.288763, event: commit_sent, op: osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.288793 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2892 (0'0,173'2892] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2890 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:01.288822 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:01.288850 7f9d7d968700 10 _finish_op 0xe2e7aa0 seq 2605 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:01.288858 7f9d7d968700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:48:01.288866 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:01.288876 7f9d7795c700  5 -- op tracker -- , seq: 748532, time: 2014-06-12 14:48:01.288868, event: sub_op_applied, op: osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.288888 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2892 (0'0,173'2892] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2891 active] sub_op_modify_applied on 0x587fdc0 op osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.288906 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2892 (0'0,173'2892] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2891 active] op_applied on primary on version 173'2892
2014-06-12 14:48:01.288917 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2892 (0'0,173'2892] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2891 active] op_applied on replica on version 173'2892
2014-06-12 14:48:01.288941 7f9d7795c700  5 -- op tracker -- , seq: 748532, time: 2014-06-12 14:48:01.288937, event: done, op: osd_sub_op(client.4297.0:144066 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2892 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.327203 7f9d5a16f700  5 -- op tracker -- , seq: 748533, time: 2014-06-12 14:48:01.325977, event: header_read, op: osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.327218 7f9d5a16f700  5 -- op tracker -- , seq: 748533, time: 2014-06-12 14:48:01.325979, event: throttled, op: osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.327235 7f9d5a16f700  5 -- op tracker -- , seq: 748533, time: 2014-06-12 14:48:01.327165, event: all_read, op: osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.327243 7f9d5a16f700  5 -- op tracker -- , seq: 748533, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.327247 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:01.327254 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:01.327261 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329ea600 prio 127 cost 516903 latency 0.001284 osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.327324 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:01.327334 7f9d68be2700 10 osd.2 173 dequeue_op 0x329ea600 prio 127 cost 516903 latency 0.001357 osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2488 (0'0,173'2488] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2487 active]
2014-06-12 14:48:01.327354 7f9d68be2700  5 -- op tracker -- , seq: 748533, time: 2014-06-12 14:48:01.327350, event: reached_pg, op: osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.327362 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2488 (0'0,173'2488] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2487 active] handle_message: 0x329ea600
2014-06-12 14:48:01.327373 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2488 (0'0,173'2488] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2487 active] sub_op_modify trans c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 v 173'2489 (transaction) 174
2014-06-12 14:48:01.327384 7f9d68be2700  5 -- op tracker -- , seq: 748533, time: 2014-06-12 14:48:01.327381, event: started, op: osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.327393 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2488 (0'0,173'2488] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2487 active] append_log log((0'0,173'2488], crt=0'0) [173'2489 (173'2487) modify   c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 by client.4290.0:143623 2014-06-12 08:45:06.124330]
2014-06-12 14:48:01.327417 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2489 (0'0,173'2489] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2487 active] add_log_entry 173'2489 (173'2487) modify   c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 by client.4290.0:143623 2014-06-12 08:45:06.124330
2014-06-12 14:48:01.327435 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2489 (0'0,173'2489] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2487 active] append_log  adding 1 keys
2014-06-12 14:48:01.327455 7f9d68be2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2489, trimmed: 
2014-06-12 14:48:01.327471 7f9d68be2700  5 -- op tracker -- , seq: 748533, time: 2014-06-12 14:48:01.327468, event: started, op: osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.327479 7f9d68be2700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:01.327482 7f9d68be2700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:01.327483 7f9d68be2700  5 queue_op 0x2e7d320 seq 2322 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:01.327495 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2489 (0'0,173'2489] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2487 active] do_sub_op osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.327504 7f9d68be2700 10 osd.2 173 dequeue_op 0x329ea600 finish
2014-06-12 14:48:01.327506 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:01.327515 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:01.327528 7f9d7c165700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:01.327536 7f9d7c165700  5 _do_op 0x2e7d320 seq 2322 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:01.327547 7f9d7c165700 10 _do_transaction on 0x20c42ef0
2014-06-12 14:48:01.327552 7f9d7c165700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:01.327653 7f9d7c165700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:01.327677 7f9d7c165700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:01.327708 7f9d7c165700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:01.327837 7f9d7c165700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:01.327845 7f9d7c165700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:01.327849 7f9d7c165700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:01.327870 7f9d7c165700 15 _write 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 2617344~516096
2014-06-12 14:48:01.327981 7f9d7c165700 10 lookup_strip_header done  cid 3.4f4_head oid c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:01.328095 7f9d7c165700 10 file_to_extents done 
2014-06-12 14:48:01.329465 7f9d7c165700 10 _generic_write 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 2617344~516096 = 0
2014-06-12 14:48:01.329513 7f9d7c165700 15 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:01.329524 7f9d7c165700 10 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 = 0
2014-06-12 14:48:01.329528 7f9d7c165700 15 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:01.329533 7f9d7c165700 10 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 = 0
2014-06-12 14:48:01.329684 7f9d7c165700 20 set_header setting 15882 cid 3.4f4_head oid c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 parent seq 0
2014-06-12 14:48:01.329742 7f9d7c165700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:01.329769 7f9d7c165700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:01.329775 7f9d7c165700  5 submit_transaction r = 0
2014-06-12 14:48:01.331730 7f9d7c165700 10 _do_op 0x2e7d320 seq 2322 r = 0, finisher 0x6646240 0
2014-06-12 14:48:01.331745 7f9d7c165700 10 _finish_op 0x2e7d320 seq 2322 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:01.331759 7f9d7c165700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:01.331765 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:01.331781 7f9d7715b700  5 -- op tracker -- , seq: 748533, time: 2014-06-12 14:48:01.331769, event: commit_sent, op: osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.331798 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2489 (0'0,173'2489] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2487 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:01.331827 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:01.331900 7f9d7795c700  5 -- op tracker -- , seq: 748533, time: 2014-06-12 14:48:01.331885, event: sub_op_applied, op: osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.331925 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2489 (0'0,173'2489] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active] sub_op_modify_applied on 0x20c42e00 op osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.331957 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2489 (0'0,173'2489] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active] op_applied on primary on version 173'2489
2014-06-12 14:48:01.331972 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2489 (0'0,173'2489] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active] op_applied on replica on version 173'2489
2014-06-12 14:48:01.332003 7f9d7795c700  5 -- op tracker -- , seq: 748533, time: 2014-06-12 14:48:01.332000, event: done, op: osd_sub_op(client.4290.0:143623 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2489 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.364152 7f9d5a16f700  5 -- op tracker -- , seq: 748534, time: 2014-06-12 14:48:01.360595, event: header_read, op: osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.364165 7f9d5a16f700  5 -- op tracker -- , seq: 748534, time: 2014-06-12 14:48:01.360596, event: throttled, op: osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.364181 7f9d5a16f700  5 -- op tracker -- , seq: 748534, time: 2014-06-12 14:48:01.364098, event: all_read, op: osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.364186 7f9d5a16f700  5 -- op tracker -- , seq: 748534, time: 0.000000, event: dispatched, op: osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.364191 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:01.364196 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:01.364203 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329ebb00 prio 127 cost 1549095 latency 0.003608 osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.364254 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:01.364267 7f9d643d9700 10 osd.2 173 dequeue_op 0x329ebb00 prio 127 cost 1549095 latency 0.003671 osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2727 (0'0,173'2727] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active]
2014-06-12 14:48:01.364290 7f9d643d9700  5 -- op tracker -- , seq: 748534, time: 2014-06-12 14:48:01.364287, event: reached_pg, op: osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.364300 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2727 (0'0,173'2727] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] handle_message: 0x329ebb00
2014-06-12 14:48:01.364311 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2727 (0'0,173'2727] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] sub_op_modify trans c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 v 173'2728 (transaction) 174
2014-06-12 14:48:01.364322 7f9d643d9700  5 -- op tracker -- , seq: 748534, time: 2014-06-12 14:48:01.364320, event: started, op: osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.364336 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2727 (0'0,173'2727] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] append_log log((0'0,173'2727], crt=0'0) [173'2728 (173'2727) modify   c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 by client.4311.0:139808 2014-06-12 08:45:26.976124]
2014-06-12 14:48:01.364354 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2728 (0'0,173'2728] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] add_log_entry 173'2728 (173'2727) modify   c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 by client.4311.0:139808 2014-06-12 08:45:26.976124
2014-06-12 14:48:01.364372 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2728 (0'0,173'2728] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] append_log  adding 1 keys
2014-06-12 14:48:01.364397 7f9d643d9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2728, trimmed: 
2014-06-12 14:48:01.364416 7f9d643d9700  5 -- op tracker -- , seq: 748534, time: 2014-06-12 14:48:01.364412, event: started, op: osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.364424 7f9d643d9700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:48:01.364427 7f9d643d9700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:01.364428 7f9d643d9700  5 queue_op 0x11eb3800 seq 2475 osr(3.21c 0x276d020) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:48:01.364434 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2728 (0'0,173'2728] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] do_sub_op osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.364443 7f9d643d9700 10 osd.2 173 dequeue_op 0x329ebb00 finish
2014-06-12 14:48:01.364439 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (1 active)
2014-06-12 14:48:01.364445 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:01.364447 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:01.364447 7f9d8096e700  5 _do_op 0x11eb3800 seq 2475 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:48:01.364452 7f9d8096e700 10 _do_transaction on 0x52c8470
2014-06-12 14:48:01.364458 7f9d8096e700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:01.364613 7f9d8096e700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:01.364645 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:01.364692 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:01.364852 7f9d8096e700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:01.364864 7f9d8096e700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:48:01.364871 7f9d8096e700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:01.364892 7f9d8096e700 15 _write 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 331776~1548288
2014-06-12 14:48:01.365048 7f9d8096e700 10 lookup_strip_header done  cid 3.21c_head oid c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:01.365350 7f9d8096e700 10 file_to_extents done 
2014-06-12 14:48:01.371523 7f9d8096e700 10 _generic_write 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 331776~1548288 = 0
2014-06-12 14:48:01.371671 7f9d8096e700 15 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:01.371697 7f9d8096e700 10 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 = 0
2014-06-12 14:48:01.371702 7f9d8096e700 15 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:01.371709 7f9d8096e700 10 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 = 0
2014-06-12 14:48:01.372084 7f9d8096e700 20 set_header setting 16171 cid 3.21c_head oid c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 parent seq 0
2014-06-12 14:48:01.372236 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:01.372252 7f9d8096e700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:48:01.372261 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:48:01.380828 7f9d8096e700 10 _do_op 0x11eb3800 seq 2475 r = 0, finisher 0x4f3eb00 0
2014-06-12 14:48:01.380848 7f9d8096e700 10 _finish_op 0x11eb3800 seq 2475 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:48:01.380861 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (0 active)
2014-06-12 14:48:01.380866 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:01.380886 7f9d7715b700  5 -- op tracker -- , seq: 748534, time: 2014-06-12 14:48:01.380866, event: commit_sent, op: osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.380903 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2728 (0'0,173'2728] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2726 active] sub_op_modify_commit on op osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:01.380939 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:01.381011 7f9d7795c700  5 -- op tracker -- , seq: 748534, time: 2014-06-12 14:48:01.381004, event: sub_op_applied, op: osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.381022 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2728 (0'0,173'2728] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] sub_op_modify_applied on 0x52c8380 op osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:01.381043 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2728 (0'0,173'2728] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] op_applied on primary on version 173'2728
2014-06-12 14:48:01.381050 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2728 (0'0,173'2728] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] op_applied on replica on version 173'2728
2014-06-12 14:48:01.381075 7f9d7795c700  5 -- op tracker -- , seq: 748534, time: 2014-06-12 14:48:01.381072, event: done, op: osd_sub_op(client.4311.0:139808 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2728 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:01.489426 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:01.489446 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:01.489449 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:01.489450 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:01.489488 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:01.537672 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:01.537753 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:01.969070 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:01.969101 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:02.116474 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:02.126006 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:02.157912 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:02.267104 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:02.274744 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:02.340280 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:02.340305 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:02.340894 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:02.341016 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:02.341059 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:02.341090 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:02.341121 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:02.341160 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:02.341226 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:02.341263 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:02.341291 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:02.341341 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:02.341417 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:02.341477 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:02.341517 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:02.341540 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:02.341561 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:02.341583 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:02.341605 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:02.341679 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:02.341716 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:02.341747 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:02.341772 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:02.341798 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:02.372526 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:02.427851 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:02.427884 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:02.430815 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:02.441991 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:02.469567 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:02.469640 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:02.489514 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:02.489530 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:48:02.489532 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:48:02.489535 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:48:02.489549 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:02.489551 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:02.489552 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:02.489558 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:02.509888 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:02.517764 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:02.519925 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:02.532834 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:02.542763 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:02.615688 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:02.622243 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:02.670254 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:02.670353 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:02.672210 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:02.770343 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:02.776718 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:02.784882 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:02.785048 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:02.807466 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:02.810465 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:02.832351 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:02.832400 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:02.865155 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:02.875682 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:02.973490 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:02.978023 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:02.981710 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:02.985231 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:03.003649 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:03.008348 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:03.105515 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:03.115508 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:03.166777 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:03.177248 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:03.180136 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:03.184546 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:03.197461 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:03.200251 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:03.224557 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:03.224656 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:03.273164 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:03.288952 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:03.327588 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:03.331835 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:03.355787 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:03.355913 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:03.364535 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:03.380935 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:03.452654 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:03.452754 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:03.489638 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:03.489668 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:03.489673 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:03.489675 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:03.489704 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:03.770738 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:03.770799 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:03.935936 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:03.936009 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:04.015038 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:04.015081 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:04.041003 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:04.041027 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:04.041516 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:04.041580 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:04.041629 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:04.041716 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:04.041767 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:04.041828 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:04.041856 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:04.041883 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:04.041905 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:04.041931 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:04.041954 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:04.041975 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:04.041996 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:04.042017 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:04.042040 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:04.042064 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:04.042087 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:04.042111 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:04.042141 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:04.042176 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:04.042198 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:04.042223 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:04.116557 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:04.126072 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:04.157986 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:04.267194 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:04.274830 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:04.372628 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:04.430887 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:04.438763 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:04.439047 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:04.442070 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:04.485168 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:04.485201 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:04.489742 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:04.489766 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:04.489769 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:04.489771 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:04.489811 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:04.509983 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:04.517849 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:04.520021 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:04.532928 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:04.542846 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:04.615790 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:04.622324 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:04.672279 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:04.770424 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:04.776789 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:04.807542 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:04.810535 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:04.865256 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:04.875759 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:04.939486 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:04.939578 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:04.959947 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:04.960013 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:04.973574 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:04.978089 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:04.981775 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:04.985302 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:05.003720 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:05.008428 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:05.105604 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:05.115599 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:05.166871 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:05.177329 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:05.180236 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:05.184626 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:05.197544 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:05.200331 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:05.273272 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:05.289049 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:05.327675 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:05.331918 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:05.364634 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:05.370140 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:05.370230 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:05.381020 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:05.440037 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:05.440283 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:05.471622 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:05.471783 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:05.489838 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:05.489862 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:05.489866 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:05.489868 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:05.489908 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:05.585775 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:05.586144 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:05.928452 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:05.928511 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:06.116649 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:06.126153 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:06.158064 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:06.267290 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:06.274912 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:06.299165 7f9d5a16f700  5 -- op tracker -- , seq: 748535, time: 2014-06-12 14:48:06.298010, event: header_read, op: osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.299208 7f9d5a16f700  5 -- op tracker -- , seq: 748535, time: 2014-06-12 14:48:06.298030, event: throttled, op: osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.299229 7f9d5a16f700  5 -- op tracker -- , seq: 748535, time: 2014-06-12 14:48:06.299095, event: all_read, op: osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.299248 7f9d5a16f700  5 -- op tracker -- , seq: 748535, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.299257 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:06.299270 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:06.299285 7f9d5a16f700 15 osd.2 173 enqueue_op 0x329ebe00 prio 127 cost 516903 latency 0.001275 osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.299342 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:06.299359 7f9d633d7700 10 osd.2 173 dequeue_op 0x329ebe00 prio 127 cost 516903 latency 0.001348 osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2892 (0'0,173'2892] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2891 active]
2014-06-12 14:48:06.299409 7f9d633d7700  5 -- op tracker -- , seq: 748535, time: 2014-06-12 14:48:06.299400, event: reached_pg, op: osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.299428 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2892 (0'0,173'2892] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2891 active] handle_message: 0x329ebe00
2014-06-12 14:48:06.299452 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2892 (0'0,173'2892] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2891 active] sub_op_modify trans bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 v 173'2893 (transaction) 174
2014-06-12 14:48:06.299486 7f9d633d7700  5 -- op tracker -- , seq: 748535, time: 2014-06-12 14:48:06.299478, event: started, op: osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.299512 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2892 (0'0,173'2892] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2891 active] append_log log((0'0,173'2892], crt=0'0) [173'2893 (173'2892) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144067 2014-06-12 08:45:27.327903]
2014-06-12 14:48:06.299558 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2893 (0'0,173'2893] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2891 active] add_log_entry 173'2893 (173'2892) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144067 2014-06-12 08:45:27.327903
2014-06-12 14:48:06.299602 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2893 (0'0,173'2893] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2891 active] append_log  adding 1 keys
2014-06-12 14:48:06.299665 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2893, trimmed: 
2014-06-12 14:48:06.299710 7f9d633d7700  5 -- op tracker -- , seq: 748535, time: 2014-06-12 14:48:06.299700, event: started, op: osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.299729 7f9d633d7700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:06.299735 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:06.299738 7f9d633d7700  5 queue_op 0x18550de0 seq 2606 osr(3.129 0x276cf20) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:06.299761 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:48:06.299776 7f9d7a962700  5 _do_op 0x18550de0 seq 2606 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:48:06.299752 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2893 (0'0,173'2893] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2891 active] do_sub_op osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.299784 7f9d7a962700 10 _do_transaction on 0x591deb0
2014-06-12 14:48:06.299783 7f9d633d7700 10 osd.2 173 dequeue_op 0x329ebe00 finish
2014-06-12 14:48:06.299791 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:06.299797 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:06.299795 7f9d7a962700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:06.300201 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:06.300244 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:06.300334 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:06.300374 7f9d7a962700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:06.300389 7f9d7a962700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:48:06.300401 7f9d7a962700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:06.300436 7f9d7a962700 15 _write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 2342912~516096
2014-06-12 14:48:06.300963 7f9d7a962700 10 lookup_strip_header done  cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:06.301187 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:48:06.304792 7f9d7a962700 10 _generic_write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 2342912~516096 = 0
2014-06-12 14:48:06.304878 7f9d7a962700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:06.304902 7f9d7a962700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:06.304914 7f9d7a962700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:06.304930 7f9d7a962700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:06.305341 7f9d7a962700 20 set_header setting 15698 cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 parent seq 0
2014-06-12 14:48:06.305502 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:06.305534 7f9d7a962700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:48:06.305554 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:48:06.307309 7f9d5a16f700  5 -- op tracker -- , seq: 748536, time: 2014-06-12 14:48:06.305710, event: header_read, op: osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.307322 7f9d5a16f700  5 -- op tracker -- , seq: 748536, time: 2014-06-12 14:48:06.305714, event: throttled, op: osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.307331 7f9d5a16f700  5 -- op tracker -- , seq: 748536, time: 2014-06-12 14:48:06.307271, event: all_read, op: osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.307340 7f9d5a16f700  5 -- op tracker -- , seq: 748536, time: 0.000000, event: dispatched, op: osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.307346 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:06.307354 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:06.307364 7f9d5a16f700 15 osd.2 173 enqueue_op 0x22b3b100 prio 127 cost 754471 latency 0.001654 osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.307401 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:06.307412 7f9d67be0700 10 osd.2 173 dequeue_op 0x22b3b100 prio 127 cost 754471 latency 0.001701 osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2489 (0'0,173'2489] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active]
2014-06-12 14:48:06.307440 7f9d67be0700  5 -- op tracker -- , seq: 748536, time: 2014-06-12 14:48:06.307434, event: reached_pg, op: osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.307452 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2489 (0'0,173'2489] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active] handle_message: 0x22b3b100
2014-06-12 14:48:06.307465 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2489 (0'0,173'2489] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active] sub_op_modify trans 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 v 173'2490 (transaction) 174
2014-06-12 14:48:06.307483 7f9d67be0700  5 -- op tracker -- , seq: 748536, time: 2014-06-12 14:48:06.307480, event: started, op: osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.307500 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2489 (0'0,173'2489] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active] append_log log((0'0,173'2489], crt=0'0) [173'2490 (173'2488) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144698 2014-06-12 08:45:14.261087]
2014-06-12 14:48:06.307525 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2490 (0'0,173'2490] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active] add_log_entry 173'2490 (173'2488) modify   2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 by client.4303.0:144698 2014-06-12 08:45:14.261087
2014-06-12 14:48:06.307548 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2490 (0'0,173'2490] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active] append_log  adding 1 keys
2014-06-12 14:48:06.307587 7f9d67be0700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2490, trimmed: 
2014-06-12 14:48:06.307611 7f9d67be0700  5 -- op tracker -- , seq: 748536, time: 2014-06-12 14:48:06.307606, event: started, op: osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.307623 7f9d67be0700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:06.307627 7f9d67be0700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:06.307629 7f9d67be0700  5 queue_op 0x65038c0 seq 2323 osr(3.4f4 0x276db70) 755957 bytes   (queue has 2 ops and 1274346 bytes)
2014-06-12 14:48:06.307639 7f9d82972700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (2 active)
2014-06-12 14:48:06.307635 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2490 (0'0,173'2490] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active] do_sub_op osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.307646 7f9d82972700  5 _do_op 0x65038c0 seq 2323 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:06.307648 7f9d67be0700 10 osd.2 173 dequeue_op 0x22b3b100 finish
2014-06-12 14:48:06.307650 7f9d82972700 10 _do_transaction on 0x6177b30
2014-06-12 14:48:06.307651 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:06.307654 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:06.307656 7f9d82972700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:06.307763 7f9d82972700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:06.307784 7f9d82972700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:06.308034 7f9d82972700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:06.308168 7f9d7a962700 10 _do_op 0x18550de0 seq 2606 r = 0, finisher 0x8144980 0
2014-06-12 14:48:06.308172 7f9d82972700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:06.308181 7f9d7a962700 10 _finish_op 0x18550de0 seq 2606 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:06.308185 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (1 active)
2014-06-12 14:48:06.308188 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:06.308194 7f9d7715b700  5 -- op tracker -- , seq: 748535, time: 2014-06-12 14:48:06.308188, event: commit_sent, op: osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.308202 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2893 (0'0,173'2893] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2891 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:06.308225 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:06.308179 7f9d82972700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:06.308255 7f9d82972700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:06.308255 7f9d7795c700  5 -- op tracker -- , seq: 748535, time: 2014-06-12 14:48:06.308249, event: sub_op_applied, op: osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.308275 7f9d82972700 15 _write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 3440640~753664
2014-06-12 14:48:06.308268 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2893 (0'0,173'2893] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2892 active] sub_op_modify_applied on 0x591ddc0 op osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.308281 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2893 (0'0,173'2893] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2892 active] op_applied on primary on version 173'2893
2014-06-12 14:48:06.308287 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2893 (0'0,173'2893] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2892 active] op_applied on replica on version 173'2893
2014-06-12 14:48:06.308302 7f9d7795c700  5 -- op tracker -- , seq: 748535, time: 2014-06-12 14:48:06.308299, event: done, op: osd_sub_op(client.4297.0:144067 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2893 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.308573 7f9d82972700 10 lookup_strip_header done  cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:06.308692 7f9d82972700 10 file_to_extents done 
2014-06-12 14:48:06.311358 7f9d82972700 10 _generic_write 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 3440640~753664 = 0
2014-06-12 14:48:06.311413 7f9d82972700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:06.311426 7f9d82972700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:48:06.311430 7f9d82972700 15 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3
2014-06-12 14:48:06.311436 7f9d82972700 10 _setattrs 3.4f4_head/2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 = 0
2014-06-12 14:48:06.311610 7f9d82972700 20 set_header setting 17360 cid 3.4f4_head oid 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 parent seq 0
2014-06-12 14:48:06.311681 7f9d82972700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:06.311693 7f9d82972700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:06.311698 7f9d82972700  5 submit_transaction r = 0
2014-06-12 14:48:06.317088 7f9d82972700 10 _do_op 0x65038c0 seq 2323 r = 0, finisher 0xb837980 0
2014-06-12 14:48:06.317118 7f9d82972700 10 _finish_op 0x65038c0 seq 2323 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:06.317126 7f9d82972700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:06.317132 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:06.317195 7f9d7795c700  5 -- op tracker -- , seq: 748536, time: 2014-06-12 14:48:06.317175, event: sub_op_applied, op: osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.317219 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2490 (0'0,173'2490] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active] sub_op_modify_applied on 0x6177a40 op osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.317263 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:06.317278 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2490 (0'0,173'2490] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active] op_applied on primary on version 173'2490
2014-06-12 14:48:06.317290 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2490 (0'0,173'2490] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active] op_applied on replica on version 173'2490
2014-06-12 14:48:06.317344 7f9d7715b700  5 -- op tracker -- , seq: 748536, time: 2014-06-12 14:48:06.317337, event: commit_sent, op: osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.317362 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2490 (0'0,173'2490] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2488 active] sub_op_modify_commit on op osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:06.317376 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:06.317410 7f9d7715b700  5 -- op tracker -- , seq: 748536, time: 2014-06-12 14:48:06.317399, event: done, op: osd_sub_op(client.4303.0:144698 3.4f4 2f039cf4/rbd_data.10a66b8b4567.0000000000000fe6/head//3 [] v 173'2490 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.353259 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:06.353399 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:06.372725 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:06.489944 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:06.490017 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:06.490018 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 4.71 >= max 0.5 = no, load too high
2014-06-12 14:48:06.490041 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:06.490044 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92734 < max (604800 seconds)
2014-06-12 14:48:06.490068 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:06.490070 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:06.490072 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:06.492777 7f9d5a16f700  5 -- op tracker -- , seq: 748537, time: 2014-06-12 14:48:06.491261, event: header_read, op: osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.492807 7f9d5a16f700  5 -- op tracker -- , seq: 748537, time: 2014-06-12 14:48:06.491264, event: throttled, op: osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.492827 7f9d5a16f700  5 -- op tracker -- , seq: 748537, time: 2014-06-12 14:48:06.492705, event: all_read, op: osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.492847 7f9d5a16f700  5 -- op tracker -- , seq: 748537, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.492857 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:06.492871 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:06.492887 7f9d5a16f700 15 osd.2 173 enqueue_op 0x1e3eb300 prio 127 cost 516903 latency 0.001626 osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.492968 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:06.492991 7f9d673df700 10 osd.2 173 dequeue_op 0x1e3eb300 prio 127 cost 516903 latency 0.001729 osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2490 (0'0,173'2490] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2489 active]
2014-06-12 14:48:06.493030 7f9d673df700  5 -- op tracker -- , seq: 748537, time: 2014-06-12 14:48:06.493024, event: reached_pg, op: osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.493048 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2490 (0'0,173'2490] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2489 active] handle_message: 0x1e3eb300
2014-06-12 14:48:06.493063 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2490 (0'0,173'2490] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2489 active] sub_op_modify trans c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 v 173'2491 (transaction) 174
2014-06-12 14:48:06.493084 7f9d673df700  5 -- op tracker -- , seq: 748537, time: 2014-06-12 14:48:06.493079, event: started, op: osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.493124 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2490 (0'0,173'2490] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2489 active] append_log log((0'0,173'2490], crt=0'0) [173'2491 (173'2489) modify   c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 by client.4290.0:143624 2014-06-12 08:45:11.672104]
2014-06-12 14:48:06.493157 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2491 (0'0,173'2491] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2489 active] add_log_entry 173'2491 (173'2489) modify   c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 by client.4290.0:143624 2014-06-12 08:45:11.672104
2014-06-12 14:48:06.493187 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2491 (0'0,173'2491] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2489 active] append_log  adding 1 keys
2014-06-12 14:48:06.493229 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2491, trimmed: 
2014-06-12 14:48:06.493268 7f9d673df700  5 -- op tracker -- , seq: 748537, time: 2014-06-12 14:48:06.493261, event: started, op: osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.493285 7f9d673df700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:06.493290 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:06.493292 7f9d673df700  5 queue_op 0x8147320 seq 2324 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:06.493300 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2491 (0'0,173'2491] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2489 active] do_sub_op osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.493320 7f9d673df700 10 osd.2 173 dequeue_op 0x1e3eb300 finish
2014-06-12 14:48:06.493325 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:06.493329 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:06.493336 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:06.493351 7f9d7895e700  5 _do_op 0x8147320 seq 2324 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:06.493358 7f9d7895e700 10 _do_transaction on 0x6238b70
2014-06-12 14:48:06.493368 7f9d7895e700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:06.493498 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:06.493531 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:06.493598 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:06.493805 7f9d7895e700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:06.493819 7f9d7895e700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:06.493829 7f9d7895e700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:06.493853 7f9d7895e700 15 _write 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 3133440~516096
2014-06-12 14:48:06.494052 7f9d7895e700 10 lookup_strip_header done  cid 3.4f4_head oid c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:06.494364 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:48:06.494820 7f9d5a16f700  5 -- op tracker -- , seq: 748538, time: 2014-06-12 14:48:06.492945, event: header_read, op: osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.494839 7f9d5a16f700  5 -- op tracker -- , seq: 748538, time: 2014-06-12 14:48:06.492947, event: throttled, op: osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.494873 7f9d5a16f700  5 -- op tracker -- , seq: 748538, time: 2014-06-12 14:48:06.494775, event: all_read, op: osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.494885 7f9d5a16f700  5 -- op tracker -- , seq: 748538, time: 0.000000, event: dispatched, op: osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.494892 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:06.494901 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:06.494913 7f9d5a16f700 15 osd.2 173 enqueue_op 0x20e87200 prio 127 cost 1032999 latency 0.001968 osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.495115 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:06.495153 7f9d663dd700 10 osd.2 173 dequeue_op 0x20e87200 prio 127 cost 1032999 latency 0.002194 osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2728 (0'0,173'2728] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active]
2014-06-12 14:48:06.495238 7f9d663dd700  5 -- op tracker -- , seq: 748538, time: 2014-06-12 14:48:06.495228, event: reached_pg, op: osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.495282 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2728 (0'0,173'2728] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] handle_message: 0x20e87200
2014-06-12 14:48:06.495334 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2728 (0'0,173'2728] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] sub_op_modify trans c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 v 173'2729 (transaction) 174
2014-06-12 14:48:06.495392 7f9d663dd700  5 -- op tracker -- , seq: 748538, time: 2014-06-12 14:48:06.495382, event: started, op: osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.495442 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2728 (0'0,173'2728] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] append_log log((0'0,173'2728], crt=0'0) [173'2729 (173'2728) modify   c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 by client.4311.0:139809 2014-06-12 08:45:27.516064]
2014-06-12 14:48:06.495494 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2729 (0'0,173'2729] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] add_log_entry 173'2729 (173'2728) modify   c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 by client.4311.0:139809 2014-06-12 08:45:27.516064
2014-06-12 14:48:06.495544 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2729 (0'0,173'2729] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] append_log  adding 1 keys
2014-06-12 14:48:06.495614 7f9d663dd700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2729, trimmed: 
2014-06-12 14:48:06.495661 7f9d663dd700  5 -- op tracker -- , seq: 748538, time: 2014-06-12 14:48:06.495651, event: started, op: osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.495682 7f9d663dd700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:48:06.495689 7f9d663dd700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:06.495692 7f9d663dd700  5 queue_op 0x6d14420 seq 2476 osr(3.21c 0x276d020) 1034485 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:48:06.495712 7f9d83173700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (2 active)
2014-06-12 14:48:06.495727 7f9d83173700  5 _do_op 0x6d14420 seq 2476 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:48:06.495736 7f9d83173700 10 _do_transaction on 0x62ea0f0
2014-06-12 14:48:06.495706 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2729 (0'0,173'2729] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] do_sub_op osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.495741 7f9d663dd700 10 osd.2 173 dequeue_op 0x20e87200 finish
2014-06-12 14:48:06.495748 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:06.495748 7f9d83173700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:06.495753 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:06.495934 7f9d83173700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:06.495978 7f9d83173700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:06.497229 7f9d7895e700 10 _generic_write 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 3133440~516096 = 0
2014-06-12 14:48:06.497325 7f9d7895e700 15 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:06.497354 7f9d7895e700 10 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 = 0
2014-06-12 14:48:06.497365 7f9d7895e700 15 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:06.497379 7f9d7895e700 10 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 = 0
2014-06-12 14:48:06.497810 7f9d7895e700 20 set_header setting 15882 cid 3.4f4_head oid c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 parent seq 0
2014-06-12 14:48:06.497887 7f9d7895e700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:06.497902 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:06.497910 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:48:06.500590 7f9d83173700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:06.500654 7f9d83173700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:06.500666 7f9d83173700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:48:06.500687 7f9d83173700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:06.500798 7f9d7895e700 10 _do_op 0x8147320 seq 2324 r = 0, finisher 0x8e0d920 0
2014-06-12 14:48:06.500814 7f9d7895e700 10 _finish_op 0x8147320 seq 2324 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:06.500824 7f9d83173700 15 _write 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 1880064~1032192
2014-06-12 14:48:06.500840 7f9d7795c700  5 -- op tracker -- , seq: 748537, time: 2014-06-12 14:48:06.500832, event: sub_op_applied, op: osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.500848 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2491 (0'0,173'2491] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2489 active] sub_op_modify_applied on 0x6238a80 op osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.500878 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:06.500895 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2491 (0'0,173'2491] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2489 active] op_applied on primary on version 173'2491
2014-06-12 14:48:06.500905 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2491 (0'0,173'2491] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2489 active] op_applied on replica on version 173'2491
2014-06-12 14:48:06.500920 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (1 active)
2014-06-12 14:48:06.500923 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:06.500992 7f9d83173700 10 lookup_strip_header done  cid 3.21c_head oid c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:06.501177 7f9d83173700 10 file_to_extents done 
2014-06-12 14:48:06.503114 7f9d5a16f700  5 -- op tracker -- , seq: 748539, time: 2014-06-12 14:48:06.501863, event: header_read, op: osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.503163 7f9d5a16f700  5 -- op tracker -- , seq: 748539, time: 2014-06-12 14:48:06.501865, event: throttled, op: osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.503189 7f9d5a16f700  5 -- op tracker -- , seq: 748539, time: 2014-06-12 14:48:06.502952, event: all_read, op: osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.503210 7f9d5a16f700  5 -- op tracker -- , seq: 748539, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.503221 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:06.503236 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:06.503252 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2519d400 prio 127 cost 516903 latency 0.001389 osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.503316 7f9d7715b700  5 -- op tracker -- , seq: 748537, time: 2014-06-12 14:48:06.503159, event: commit_sent, op: osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.503335 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2491 (0'0,173'2491] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2489 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:06.503385 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:06.503458 7f9d7715b700  5 -- op tracker -- , seq: 748537, time: 2014-06-12 14:48:06.503446, event: done, op: osd_sub_op(client.4290.0:143624 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2491 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.504051 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:06.504071 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2519d400 prio 127 cost 516903 latency 0.002207 osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2893 (0'0,173'2893] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2892 active]
2014-06-12 14:48:06.504126 7f9d65bdc700  5 -- op tracker -- , seq: 748539, time: 2014-06-12 14:48:06.504115, event: reached_pg, op: osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.504144 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2893 (0'0,173'2893] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2892 active] handle_message: 0x2519d400
2014-06-12 14:48:06.504173 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2893 (0'0,173'2893] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2892 active] sub_op_modify trans bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 v 173'2894 (transaction) 174
2014-06-12 14:48:06.504225 7f9d65bdc700  5 -- op tracker -- , seq: 748539, time: 2014-06-12 14:48:06.504215, event: started, op: osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.504255 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2893 (0'0,173'2893] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2892 active] append_log log((0'0,173'2893], crt=0'0) [173'2894 (173'2893) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144068 2014-06-12 08:45:27.339549]
2014-06-12 14:48:06.504303 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2894 (0'0,173'2894] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2892 active] add_log_entry 173'2894 (173'2893) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144068 2014-06-12 08:45:27.339549
2014-06-12 14:48:06.504354 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2894 (0'0,173'2894] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2892 active] append_log  adding 1 keys
2014-06-12 14:48:06.504428 7f9d65bdc700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2894, trimmed: 
2014-06-12 14:48:06.504478 7f9d65bdc700  5 -- op tracker -- , seq: 748539, time: 2014-06-12 14:48:06.504467, event: started, op: osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.504500 7f9d65bdc700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:06.504508 7f9d65bdc700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:06.504511 7f9d65bdc700  5 queue_op 0x568e1e0 seq 2607 osr(3.129 0x276cf20) 518389 bytes   (queue has 2 ops and 1552874 bytes)
2014-06-12 14:48:06.504528 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2894 (0'0,173'2894] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2892 active] do_sub_op osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.504563 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2519d400 finish
2014-06-12 14:48:06.504573 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:06.504567 7f9d7a161700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (2 active)
2014-06-12 14:48:06.504579 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:06.504586 7f9d7a161700  5 _do_op 0x568e1e0 seq 2607 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:48:06.504593 7f9d7a161700 10 _do_transaction on 0x1451c7f0
2014-06-12 14:48:06.504603 7f9d7a161700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:06.504747 7f9d7a161700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:06.504775 7f9d7a161700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:06.508444 7f9d83173700 10 _generic_write 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 1880064~1032192 = 0
2014-06-12 14:48:06.508653 7f9d83173700 15 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:06.508686 7f9d83173700 10 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 = 0
2014-06-12 14:48:06.508698 7f9d83173700 15 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:06.508715 7f9d83173700 10 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 = 0
2014-06-12 14:48:06.509327 7f9d83173700 20 set_header setting 16171 cid 3.21c_head oid c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 parent seq 0
2014-06-12 14:48:06.509468 7f9d83173700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:06.509501 7f9d83173700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:48:06.509522 7f9d83173700  5 submit_transaction r = 0
2014-06-12 14:48:06.513673 7f9d7a161700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:06.513798 7f9d7a161700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:06.513809 7f9d7a161700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:48:06.513818 7f9d7a161700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:06.513823 7f9d83173700 10 _do_op 0x6d14420 seq 2476 r = 0, finisher 0xabf30e0 0
2014-06-12 14:48:06.513835 7f9d83173700 10 _finish_op 0x6d14420 seq 2476 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:48:06.513839 7f9d7a161700 15 _write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 2859008~516096
2014-06-12 14:48:06.513840 7f9d83173700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (1 active)
2014-06-12 14:48:06.513844 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:06.513851 7f9d7715b700  5 -- op tracker -- , seq: 748538, time: 2014-06-12 14:48:06.513842, event: commit_sent, op: osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.513870 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2729 (0'0,173'2729] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2727 active] sub_op_modify_commit on op osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:06.513896 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:06.513983 7f9d7a161700 10 lookup_strip_header done  cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:06.514128 7f9d7a161700 10 file_to_extents done 
2014-06-12 14:48:06.516977 7f9d7a161700 10 _generic_write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 2859008~516096 = 0
2014-06-12 14:48:06.517082 7f9d7a161700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:06.517113 7f9d7a161700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:06.517127 7f9d7a161700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:06.517143 7f9d7a161700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:06.517556 7f9d7a161700 20 set_header setting 15698 cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 parent seq 0
2014-06-12 14:48:06.517667 7f9d7795c700  5 -- op tracker -- , seq: 748538, time: 2014-06-12 14:48:06.517651, event: sub_op_applied, op: osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.517691 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2729 (0'0,173'2729] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] sub_op_modify_applied on 0x62ea000 op osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.517740 7f9d7a161700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:06.517728 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2729 (0'0,173'2729] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] op_applied on primary on version 173'2729
2014-06-12 14:48:06.517748 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2729 (0'0,173'2729] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] op_applied on replica on version 173'2729
2014-06-12 14:48:06.517769 7f9d7a161700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:48:06.517790 7f9d7a161700  5 submit_transaction r = 0
2014-06-12 14:48:06.517791 7f9d7795c700  5 -- op tracker -- , seq: 748538, time: 2014-06-12 14:48:06.517781, event: done, op: osd_sub_op(client.4311.0:139809 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2729 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.523002 7f9d7a161700 10 _do_op 0x568e1e0 seq 2607 r = 0, finisher 0x65b9e60 0
2014-06-12 14:48:06.523026 7f9d7a161700 10 _finish_op 0x568e1e0 seq 2607 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:06.523035 7f9d7a161700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:48:06.523043 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:06.523064 7f9d7715b700  5 -- op tracker -- , seq: 748539, time: 2014-06-12 14:48:06.523053, event: commit_sent, op: osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.523075 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2894 (0'0,173'2894] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2892 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:06.523113 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:06.523191 7f9d7795c700  5 -- op tracker -- , seq: 748539, time: 2014-06-12 14:48:06.523185, event: sub_op_applied, op: osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.523202 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2894 (0'0,173'2894] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active] sub_op_modify_applied on 0x1451c700 op osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.523222 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2894 (0'0,173'2894] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active] op_applied on primary on version 173'2894
2014-06-12 14:48:06.523229 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2894 (0'0,173'2894] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active] op_applied on replica on version 173'2894
2014-06-12 14:48:06.523245 7f9d7795c700  5 -- op tracker -- , seq: 748539, time: 2014-06-12 14:48:06.523242, event: done, op: osd_sub_op(client.4297.0:144068 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2894 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.641999 7f9d5a16f700  5 -- op tracker -- , seq: 748540, time: 2014-06-12 14:48:06.640907, event: header_read, op: osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.642013 7f9d5a16f700  5 -- op tracker -- , seq: 748540, time: 2014-06-12 14:48:06.640909, event: throttled, op: osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.642029 7f9d5a16f700  5 -- op tracker -- , seq: 748540, time: 2014-06-12 14:48:06.641956, event: all_read, op: osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.642038 7f9d5a16f700  5 -- op tracker -- , seq: 748540, time: 0.000000, event: dispatched, op: osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.642043 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:06.642057 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:06.642065 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2519db00 prio 127 cost 545575 latency 0.001158 osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.642092 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:06.642100 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2519db00 prio 127 cost 545575 latency 0.001193 osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2491 (0'0,173'2491] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2490 active]
2014-06-12 14:48:06.642116 7f9d62bd6700  5 -- op tracker -- , seq: 748540, time: 2014-06-12 14:48:06.642114, event: reached_pg, op: osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.642123 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2491 (0'0,173'2491] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2490 active] handle_message: 0x2519db00
2014-06-12 14:48:06.642130 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2491 (0'0,173'2491] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2490 active] sub_op_modify trans c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 v 173'2492 (transaction) 174
2014-06-12 14:48:06.642140 7f9d62bd6700  5 -- op tracker -- , seq: 748540, time: 2014-06-12 14:48:06.642138, event: started, op: osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.642149 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2491 (0'0,173'2491] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2490 active] append_log log((0'0,173'2491], crt=0'0) [173'2492 (173'2491) modify   c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 by client.4290.0:143625 2014-06-12 08:45:11.868523]
2014-06-12 14:48:06.642164 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2492 (0'0,173'2492] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2490 active] add_log_entry 173'2492 (173'2491) modify   c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 by client.4290.0:143625 2014-06-12 08:45:11.868523
2014-06-12 14:48:06.642178 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2492 (0'0,173'2492] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2490 active] append_log  adding 1 keys
2014-06-12 14:48:06.642198 7f9d62bd6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2492, trimmed: 
2014-06-12 14:48:06.642212 7f9d62bd6700  5 -- op tracker -- , seq: 748540, time: 2014-06-12 14:48:06.642209, event: started, op: osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.642218 7f9d62bd6700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:06.642221 7f9d62bd6700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:06.642222 7f9d62bd6700  5 queue_op 0x2b3e240 seq 2325 osr(3.4f4 0x276db70) 547061 bytes   (queue has 1 ops and 547061 bytes)
2014-06-12 14:48:06.642227 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2492 (0'0,173'2492] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2490 active] do_sub_op osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.642230 7f9d7f16b700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:06.642236 7f9d62bd6700 10 osd.2 173 dequeue_op 0x2519db00 finish
2014-06-12 14:48:06.642238 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:06.642240 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:06.642239 7f9d7f16b700  5 _do_op 0x2b3e240 seq 2325 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:06.642243 7f9d7f16b700 10 _do_transaction on 0x331f7430
2014-06-12 14:48:06.642249 7f9d7f16b700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:06.642352 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:06.642376 7f9d7f16b700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:06.642425 7f9d7f16b700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:06.642538 7f9d7f16b700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:06.642545 7f9d7f16b700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:06.642551 7f9d7f16b700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:06.642570 7f9d7f16b700 15 _write 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 3649536~544768
2014-06-12 14:48:06.642716 7f9d7f16b700 10 lookup_strip_header done  cid 3.4f4_head oid c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:06.642868 7f9d7f16b700 10 file_to_extents done 
2014-06-12 14:48:06.646402 7f9d7f16b700 10 _generic_write 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 3649536~544768 = 0
2014-06-12 14:48:06.646533 7f9d7f16b700 15 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:06.646567 7f9d7f16b700 10 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 = 0
2014-06-12 14:48:06.646580 7f9d7f16b700 15 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3
2014-06-12 14:48:06.646598 7f9d7f16b700 10 _setattrs 3.4f4_head/c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 = 0
2014-06-12 14:48:06.647042 7f9d7f16b700 20 set_header setting 15882 cid 3.4f4_head oid c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 parent seq 0
2014-06-12 14:48:06.647232 7f9d7f16b700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:06.647267 7f9d7f16b700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:06.647290 7f9d7f16b700  5 submit_transaction r = 0
2014-06-12 14:48:06.652346 7f9d7f16b700 10 _do_op 0x2b3e240 seq 2325 r = 0, finisher 0x534d080 0
2014-06-12 14:48:06.652385 7f9d7f16b700 10 _finish_op 0x2b3e240 seq 2325 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:06.652418 7f9d7f16b700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:06.652431 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:06.652429 7f9d7715b700  5 -- op tracker -- , seq: 748540, time: 2014-06-12 14:48:06.652410, event: commit_sent, op: osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.652462 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2492 (0'0,173'2492] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2490 active] sub_op_modify_commit on op osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:06.652505 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:06.652536 7f9d7795c700  5 -- op tracker -- , seq: 748540, time: 2014-06-12 14:48:06.652530, event: sub_op_applied, op: osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.652562 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2492 (0'0,173'2492] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2491 active] sub_op_modify_applied on 0x331f7340 op osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.652597 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2492 (0'0,173'2492] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2491 active] op_applied on primary on version 173'2492
2014-06-12 14:48:06.652618 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2492 (0'0,173'2492] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2491 active] op_applied on replica on version 173'2492
2014-06-12 14:48:06.652663 7f9d7795c700  5 -- op tracker -- , seq: 748540, time: 2014-06-12 14:48:06.652655, event: done, op: osd_sub_op(client.4290.0:143625 3.4f4 c1cd14f4/rbd_data.10996b8b4567.0000000000001025/head//3 [] v 173'2492 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.683711 7f9d5a16f700  5 -- op tracker -- , seq: 748541, time: 2014-06-12 14:48:06.682450, event: header_read, op: osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.683735 7f9d5a16f700  5 -- op tracker -- , seq: 748541, time: 2014-06-12 14:48:06.682453, event: throttled, op: osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.683767 7f9d5a16f700  5 -- op tracker -- , seq: 748541, time: 2014-06-12 14:48:06.683645, event: all_read, op: osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.683779 7f9d5a16f700  5 -- op tracker -- , seq: 748541, time: 0.000000, event: dispatched, op: osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.683787 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:06.683797 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:06.683810 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2911300 prio 127 cost 516903 latency 0.001359 osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.683860 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:06.683877 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2911300 prio 127 cost 516903 latency 0.001427 osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2729 (0'0,173'2729] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active]
2014-06-12 14:48:06.683923 7f9d6b3e7700  5 -- op tracker -- , seq: 748541, time: 2014-06-12 14:48:06.683915, event: reached_pg, op: osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.683942 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2729 (0'0,173'2729] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] handle_message: 0x2911300
2014-06-12 14:48:06.683967 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2729 (0'0,173'2729] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] sub_op_modify trans c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 v 173'2730 (transaction) 174
2014-06-12 14:48:06.684010 7f9d6b3e7700  5 -- op tracker -- , seq: 748541, time: 2014-06-12 14:48:06.684005, event: started, op: osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.684031 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2729 (0'0,173'2729] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] append_log log((0'0,173'2729], crt=0'0) [173'2730 (173'2729) modify   c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 by client.4311.0:139810 2014-06-12 08:45:27.839234]
2014-06-12 14:48:06.684058 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2730 (0'0,173'2730] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] add_log_entry 173'2730 (173'2729) modify   c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 by client.4311.0:139810 2014-06-12 08:45:27.839234
2014-06-12 14:48:06.684076 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2730 (0'0,173'2730] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] append_log  adding 1 keys
2014-06-12 14:48:06.684113 7f9d6b3e7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2730, trimmed: 
2014-06-12 14:48:06.684141 7f9d6b3e7700  5 -- op tracker -- , seq: 748541, time: 2014-06-12 14:48:06.684136, event: started, op: osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.684151 7f9d6b3e7700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:48:06.684156 7f9d6b3e7700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:06.684158 7f9d6b3e7700  5 queue_op 0x584bc20 seq 2477 osr(3.21c 0x276d020) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:06.684165 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2730 (0'0,173'2730] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] do_sub_op osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.684187 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2911300 finish
2014-06-12 14:48:06.684191 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:06.684194 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:06.684196 7f9d7e169700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (1 active)
2014-06-12 14:48:06.684204 7f9d7e169700  5 _do_op 0x584bc20 seq 2477 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:48:06.684209 7f9d7e169700 10 _do_transaction on 0x2c5c80f0
2014-06-12 14:48:06.684214 7f9d7e169700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:06.684364 7f9d7e169700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:06.684413 7f9d7e169700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:06.684514 7f9d7e169700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:06.684561 7f9d7e169700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:06.684577 7f9d7e169700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:48:06.684583 7f9d7e169700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:06.684599 7f9d7e169700 15 _write 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 2912256~516096
2014-06-12 14:48:06.684842 7f9d7e169700 10 lookup_strip_header done  cid 3.21c_head oid c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:06.684985 7f9d7e169700 10 file_to_extents done 
2014-06-12 14:48:06.688455 7f9d7e169700 10 _generic_write 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 2912256~516096 = 0
2014-06-12 14:48:06.688561 7f9d7e169700 15 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:06.688591 7f9d7e169700 10 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 = 0
2014-06-12 14:48:06.688603 7f9d7e169700 15 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:06.688623 7f9d7e169700 10 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 = 0
2014-06-12 14:48:06.688916 7f9d7e169700 20 set_header setting 16171 cid 3.21c_head oid c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 parent seq 0
2014-06-12 14:48:06.689006 7f9d7e169700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:06.689059 7f9d7e169700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:48:06.689083 7f9d7e169700  5 submit_transaction r = 0
2014-06-12 14:48:06.692427 7f9d7e169700 10 _do_op 0x584bc20 seq 2477 r = 0, finisher 0x13d13860 0
2014-06-12 14:48:06.692463 7f9d7e169700 10 _finish_op 0x584bc20 seq 2477 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:48:06.692477 7f9d7e169700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (0 active)
2014-06-12 14:48:06.692490 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:06.692491 7f9d7715b700  5 -- op tracker -- , seq: 748541, time: 2014-06-12 14:48:06.692475, event: commit_sent, op: osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.692507 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2730 (0'0,173'2730] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2728 active] sub_op_modify_commit on op osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:06.692543 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:06.692584 7f9d7795c700  5 -- op tracker -- , seq: 748541, time: 2014-06-12 14:48:06.692576, event: sub_op_applied, op: osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.692597 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2730 (0'0,173'2730] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2729 active] sub_op_modify_applied on 0x2c5c8000 op osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.692616 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2730 (0'0,173'2730] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2729 active] op_applied on primary on version 173'2730
2014-06-12 14:48:06.692628 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2730 (0'0,173'2730] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2729 active] op_applied on replica on version 173'2730
2014-06-12 14:48:06.692688 7f9d7795c700  5 -- op tracker -- , seq: 748541, time: 2014-06-12 14:48:06.692677, event: done, op: osd_sub_op(client.4311.0:139810 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2730 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.823411 7f9d5a16f700  5 -- op tracker -- , seq: 748542, time: 2014-06-12 14:48:06.820935, event: header_read, op: osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.823445 7f9d5a16f700  5 -- op tracker -- , seq: 748542, time: 2014-06-12 14:48:06.820939, event: throttled, op: osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.823472 7f9d5a16f700  5 -- op tracker -- , seq: 748542, time: 2014-06-12 14:48:06.823319, event: all_read, op: osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.823488 7f9d5a16f700  5 -- op tracker -- , seq: 748542, time: 0.000000, event: dispatched, op: osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.823499 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:06.823511 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:06.823527 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2911400 prio 127 cost 820007 latency 0.002591 osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.823612 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:06.823642 7f9d693e3700 10 osd.2 173 dequeue_op 0x2911400 prio 127 cost 820007 latency 0.002706 osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.129( v 173'2894 (0'0,173'2894] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active]
2014-06-12 14:48:06.823707 7f9d693e3700  5 -- op tracker -- , seq: 748542, time: 2014-06-12 14:48:06.823696, event: reached_pg, op: osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.823729 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2894 (0'0,173'2894] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active] handle_message: 0x2911400
2014-06-12 14:48:06.823747 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2894 (0'0,173'2894] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active] sub_op_modify trans bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 v 173'2895 (transaction) 174
2014-06-12 14:48:06.823781 7f9d693e3700  5 -- op tracker -- , seq: 748542, time: 2014-06-12 14:48:06.823772, event: started, op: osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.823812 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2894 (0'0,173'2894] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active] append_log log((0'0,173'2894], crt=0'0) [173'2895 (173'2894) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144069 2014-06-12 08:45:27.521968]
2014-06-12 14:48:06.823851 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2895 (0'0,173'2895] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active] add_log_entry 173'2895 (173'2894) modify   bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 by client.4297.0:144069 2014-06-12 08:45:27.521968
2014-06-12 14:48:06.823914 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2895 (0'0,173'2895] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active] append_log  adding 1 keys
2014-06-12 14:48:06.823986 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2895, trimmed: 
2014-06-12 14:48:06.824038 7f9d693e3700  5 -- op tracker -- , seq: 748542, time: 2014-06-12 14:48:06.824028, event: started, op: osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.824060 7f9d693e3700  5 queue_transactions existing osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:06.824068 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:06.824071 7f9d693e3700  5 queue_op 0x95ed740 seq 2608 osr(3.129 0x276cf20) 821493 bytes   (queue has 1 ops and 821493 bytes)
2014-06-12 14:48:06.824084 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.129( v 173'2895 (0'0,173'2895] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active] do_sub_op osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.824118 7f9d693e3700 10 osd.2 173 dequeue_op 0x2911400 finish
2014-06-12 14:48:06.824125 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:06.824130 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:06.824145 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x54b8fc0 (1 active)
2014-06-12 14:48:06.824153 7f9d7815d700  5 _do_op 0x95ed740 seq 2608 osr(3.129 0x276cf20)/0x276cf20 start
2014-06-12 14:48:06.824160 7f9d7815d700 10 _do_transaction on 0x1ea80ef0
2014-06-12 14:48:06.824170 7f9d7815d700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:06.824366 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:06.824413 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:06.824505 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:06.824765 7f9d7815d700 15 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:06.824785 7f9d7815d700 10 _omap_rmkeys meta/1fe282ba/pglog_3.129/0//-1 r = 0
2014-06-12 14:48:06.824799 7f9d7815d700 15 _omap_setkeys meta/1fe282ba/pglog_3.129/0//-1
2014-06-12 14:48:06.824837 7f9d7815d700 15 _write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 3375104~819200
2014-06-12 14:48:06.825061 7f9d7815d700 10 lookup_strip_header done  cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:06.825419 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:48:06.828889 7f9d7815d700 10 _generic_write 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 3375104~819200 = 0
2014-06-12 14:48:06.828992 7f9d7815d700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:06.829023 7f9d7815d700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:06.829031 7f9d7815d700 15 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3
2014-06-12 14:48:06.829041 7f9d7815d700 10 _setattrs 3.129_head/bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 = 0
2014-06-12 14:48:06.829414 7f9d7815d700 20 set_header setting 15698 cid 3.129_head oid bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 parent seq 0
2014-06-12 14:48:06.829550 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:06.829570 7f9d7815d700 20 set_header setting 2310 cid meta oid 1fe282ba/pglog_3.129/0//-1 parent seq 0
2014-06-12 14:48:06.829583 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:48:06.834462 7f9d5a16f700  5 -- op tracker -- , seq: 748543, time: 2014-06-12 14:48:06.832956, event: header_read, op: osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.834477 7f9d5a16f700  5 -- op tracker -- , seq: 748543, time: 2014-06-12 14:48:06.832962, event: throttled, op: osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.834539 7f9d5a16f700  5 -- op tracker -- , seq: 748543, time: 2014-06-12 14:48:06.834422, event: all_read, op: osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.834548 7f9d5a16f700  5 -- op tracker -- , seq: 748543, time: 0.000000, event: dispatched, op: osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.834553 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:06.834560 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:06.834568 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2911600 prio 127 cost 766759 latency 0.001612 osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.834597 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:06.834610 7f9d7815d700 10 _do_op 0x95ed740 seq 2608 r = 0, finisher 0x28ae380 0
2014-06-12 14:48:06.834605 7f9d683e1700 10 osd.2 173 dequeue_op 0x2911600 prio 127 cost 766759 latency 0.001649 osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.21c( v 173'2730 (0'0,173'2730] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2729 active]
2014-06-12 14:48:06.834623 7f9d7815d700 10 _finish_op 0x95ed740 seq 2608 osr(3.129 0x276cf20)/0x276cf20
2014-06-12 14:48:06.834635 7f9d683e1700  5 -- op tracker -- , seq: 748543, time: 2014-06-12 14:48:06.834631, event: reached_pg, op: osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.834637 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x54b8fc0 (0 active)
2014-06-12 14:48:06.834643 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:06.834644 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2730 (0'0,173'2730] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2729 active] handle_message: 0x2911600
2014-06-12 14:48:06.834655 7f9d7795c700  5 -- op tracker -- , seq: 748542, time: 2014-06-12 14:48:06.834647, event: sub_op_applied, op: osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.834657 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2730 (0'0,173'2730] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2729 active] sub_op_modify trans c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 v 173'2731 (transaction) 174
2014-06-12 14:48:06.834664 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2895 (0'0,173'2895] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active] sub_op_modify_applied on 0x1ea80e00 op osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.834673 7f9d683e1700  5 -- op tracker -- , seq: 748543, time: 2014-06-12 14:48:06.834669, event: started, op: osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.834690 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:06.834701 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2730 (0'0,173'2730] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2729 active] append_log log((0'0,173'2730], crt=0'0) [173'2731 (173'2730) modify   c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 by client.4311.0:139811 2014-06-12 08:45:28.178179]
2014-06-12 14:48:06.834709 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2895 (0'0,173'2895] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active] op_applied on primary on version 173'2895
2014-06-12 14:48:06.834727 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2895 (0'0,173'2895] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active] op_applied on replica on version 173'2895
2014-06-12 14:48:06.834724 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2731 (0'0,173'2731] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2729 active] add_log_entry 173'2731 (173'2730) modify   c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 by client.4311.0:139811 2014-06-12 08:45:28.178179
2014-06-12 14:48:06.834745 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2731 (0'0,173'2731] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2729 active] append_log  adding 1 keys
2014-06-12 14:48:06.834776 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2731, trimmed: 
2014-06-12 14:48:06.834798 7f9d683e1700  5 -- op tracker -- , seq: 748543, time: 2014-06-12 14:48:06.834793, event: started, op: osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.834809 7f9d683e1700  5 queue_transactions existing osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:48:06.834813 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:06.834814 7f9d683e1700  5 queue_op 0x1283a360 seq 2478 osr(3.21c 0x276d020) 768245 bytes   (queue has 1 ops and 768245 bytes)
2014-06-12 14:48:06.834814 7f9d7715b700  5 -- op tracker -- , seq: 748542, time: 2014-06-12 14:48:06.834809, event: commit_sent, op: osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.834824 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5439b00 (1 active)
2014-06-12 14:48:06.834830 7f9d82171700  5 _do_op 0x1283a360 seq 2478 osr(3.21c 0x276d020)/0x276d020 start
2014-06-12 14:48:06.834820 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.21c( v 173'2731 (0'0,173'2731] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2729 active] do_sub_op osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.834834 7f9d82171700 10 _do_transaction on 0x33f3ab70
2014-06-12 14:48:06.834833 7f9d683e1700 10 osd.2 173 dequeue_op 0x2911600 finish
2014-06-12 14:48:06.834837 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:06.834821 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.129( v 173'2895 (0'0,173'2895] local-les=147 n=312 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2893 active] sub_op_modify_commit on op osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:06.834839 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:06.834840 7f9d82171700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:06.834842 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:06.834871 7f9d7715b700  5 -- op tracker -- , seq: 748542, time: 2014-06-12 14:48:06.834867, event: done, op: osd_sub_op(client.4297.0:144069 3.129 bede1929/rbd_data.10a06b8b4567.000000000000106c/head//3 [] v 173'2895 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.834966 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:06.834989 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:06.835024 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:06.835109 7f9d82171700 15 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:06.835115 7f9d82171700 10 _omap_rmkeys meta/1fe27a5b/pglog_3.21c/0//-1 r = 0
2014-06-12 14:48:06.835120 7f9d82171700 15 _omap_setkeys meta/1fe27a5b/pglog_3.21c/0//-1
2014-06-12 14:48:06.835133 7f9d82171700 15 _write 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 3428352~765952
2014-06-12 14:48:06.835228 7f9d82171700 10 lookup_strip_header done  cid 3.21c_head oid c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:06.835347 7f9d82171700 10 file_to_extents done 
2014-06-12 14:48:06.837455 7f9d82171700 10 _generic_write 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 3428352~765952 = 0
2014-06-12 14:48:06.837537 7f9d82171700 15 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:06.837550 7f9d82171700 10 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 = 0
2014-06-12 14:48:06.837554 7f9d82171700 15 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3
2014-06-12 14:48:06.837559 7f9d82171700 10 _setattrs 3.21c_head/c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 = 0
2014-06-12 14:48:06.837797 7f9d82171700 20 set_header setting 16171 cid 3.21c_head oid c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 parent seq 0
2014-06-12 14:48:06.837863 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:06.837874 7f9d82171700 20 set_header setting 2284 cid meta oid 1fe27a5b/pglog_3.21c/0//-1 parent seq 0
2014-06-12 14:48:06.837880 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:48:06.840844 7f9d82171700 10 _do_op 0x1283a360 seq 2478 r = 0, finisher 0x33065ea0 0
2014-06-12 14:48:06.840862 7f9d82171700 10 _finish_op 0x1283a360 seq 2478 osr(3.21c 0x276d020)/0x276d020
2014-06-12 14:48:06.840869 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5439b00 (0 active)
2014-06-12 14:48:06.840879 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:06.840878 7f9d7715b700  5 -- op tracker -- , seq: 748543, time: 2014-06-12 14:48:06.840870, event: commit_sent, op: osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.840902 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2731 (0'0,173'2731] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2729 active] sub_op_modify_commit on op osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:06.840928 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:06.840983 7f9d7795c700  5 -- op tracker -- , seq: 748543, time: 2014-06-12 14:48:06.840976, event: sub_op_applied, op: osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.841002 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2731 (0'0,173'2731] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2730 active] sub_op_modify_applied on 0x33f3aa80 op osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:06.841014 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2731 (0'0,173'2731] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2730 active] op_applied on primary on version 173'2731
2014-06-12 14:48:06.841020 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.21c( v 173'2731 (0'0,173'2731] local-les=147 n=299 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2730 active] op_applied on replica on version 173'2731
2014-06-12 14:48:06.841038 7f9d7795c700  5 -- op tracker -- , seq: 748543, time: 2014-06-12 14:48:06.841035, event: done, op: osd_sub_op(client.4311.0:139811 3.21c c10b8a1c/rbd_data.10ae6b8b4567.0000000000000edd/head//3 [] v 173'2731 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:06.932970 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:06.933100 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:06.981839 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:06.985367 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:07.003796 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:07.008526 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:07.105709 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:07.115696 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:07.140486 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:07.140763 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:07.166948 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:07.177417 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:07.180311 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:07.184707 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:07.197646 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:07.200427 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:07.273388 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:07.289149 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:07.327765 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:07.331997 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:07.364708 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:07.381103 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:07.490139 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:07.490157 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:48:07.490159 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:48:07.490161 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:48:07.490169 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:07.490171 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:07.490172 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:07.490211 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:07.861443 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:07.861472 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:07.965461 7f9d5a16f700  5 -- op tracker -- , seq: 748544, time: 2014-06-12 14:48:07.965164, event: header_read, op: osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:07.965472 7f9d5a16f700  5 -- op tracker -- , seq: 748544, time: 2014-06-12 14:48:07.965166, event: throttled, op: osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:07.965490 7f9d5a16f700  5 -- op tracker -- , seq: 748544, time: 2014-06-12 14:48:07.965429, event: all_read, op: osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:07.965496 7f9d5a16f700  5 -- op tracker -- , seq: 748544, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:07.965499 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:07.965504 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:07.965510 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2afdb00 prio 127 cost 172839 latency 0.000345 osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:07.965531 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:07.965548 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2afdb00 prio 127 cost 172839 latency 0.000383 osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2492 (0'0,173'2492] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2491 active]
2014-06-12 14:48:07.965570 7f9d63bd8700  5 -- op tracker -- , seq: 748544, time: 2014-06-12 14:48:07.965567, event: reached_pg, op: osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:07.965578 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2492 (0'0,173'2492] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2491 active] handle_message: 0x2afdb00
2014-06-12 14:48:07.965593 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2492 (0'0,173'2492] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2491 active] sub_op_modify trans 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 v 173'2493 (transaction) 174
2014-06-12 14:48:07.965607 7f9d63bd8700  5 -- op tracker -- , seq: 748544, time: 2014-06-12 14:48:07.965603, event: started, op: osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:07.965627 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2492 (0'0,173'2492] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2491 active] append_log log((0'0,173'2492], crt=0'0) [173'2493 (173'600) modify   66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 by client.4305.0:142489 2014-06-12 08:46:00.900034]
2014-06-12 14:48:07.965661 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2493 (0'0,173'2493] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2491 active] add_log_entry 173'2493 (173'600) modify   66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 by client.4305.0:142489 2014-06-12 08:46:00.900034
2014-06-12 14:48:07.965678 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2493 (0'0,173'2493] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2491 active] append_log  adding 1 keys
2014-06-12 14:48:07.965699 7f9d63bd8700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2493, trimmed: 
2014-06-12 14:48:07.965715 7f9d63bd8700  5 -- op tracker -- , seq: 748544, time: 2014-06-12 14:48:07.965712, event: started, op: osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:07.965722 7f9d63bd8700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:07.965725 7f9d63bd8700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:07.965726 7f9d63bd8700  5 queue_op 0x652d260 seq 2326 osr(3.4f4 0x276db70) 174325 bytes   (queue has 1 ops and 174325 bytes)
2014-06-12 14:48:07.965734 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:07.965740 7f9d7c966700  5 _do_op 0x652d260 seq 2326 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:07.965731 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2493 (0'0,173'2493] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2491 active] do_sub_op osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:07.965744 7f9d7c966700 10 _do_transaction on 0x29eb2470
2014-06-12 14:48:07.965744 7f9d63bd8700 10 osd.2 173 dequeue_op 0x2afdb00 finish
2014-06-12 14:48:07.965748 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:07.965750 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:07.965751 7f9d7c966700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:07.965871 7f9d7c966700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:07.965888 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:07.965925 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:07.965943 7f9d7c966700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:07.965948 7f9d7c966700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:07.965951 7f9d7c966700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:07.966104 7f9d7c966700 15 _write 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 0~172032
2014-06-12 14:48:07.966235 7f9d7c966700 10 lookup_strip_header done  cid 3.4f4_head oid 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:07.966371 7f9d7c966700 10 file_to_extents done 
2014-06-12 14:48:07.966809 7f9d7c966700 10 _generic_write 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 0~172032 = 0
2014-06-12 14:48:07.966826 7f9d7c966700 15 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:07.966834 7f9d7c966700 10 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 = 0
2014-06-12 14:48:07.966838 7f9d7c966700 15 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:07.966842 7f9d7c966700 10 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 = 0
2014-06-12 14:48:07.966937 7f9d7c966700 20 set_header setting 16444 cid 3.4f4_head oid 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 parent seq 0
2014-06-12 14:48:07.966965 7f9d7c966700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:07.966975 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:07.966980 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:48:07.967539 7f9d7c966700 10 _do_op 0x652d260 seq 2326 r = 0, finisher 0x100248c0 0
2014-06-12 14:48:07.967547 7f9d7c966700 10 _finish_op 0x652d260 seq 2326 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:07.967555 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:07.967559 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:07.967563 7f9d7715b700  5 -- op tracker -- , seq: 748544, time: 2014-06-12 14:48:07.967556, event: commit_sent, op: osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:07.967575 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2493 (0'0,173'2493] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2491 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:07.967598 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:07.969434 7f9d7795c700  5 -- op tracker -- , seq: 748544, time: 2014-06-12 14:48:07.969425, event: sub_op_applied, op: osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:07.969446 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2493 (0'0,173'2493] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2492 active] sub_op_modify_applied on 0x29eb2380 op osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:07.969468 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2493 (0'0,173'2493] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2492 active] op_applied on primary on version 173'2493
2014-06-12 14:48:07.969479 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2493 (0'0,173'2493] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2492 active] op_applied on replica on version 173'2493
2014-06-12 14:48:07.969512 7f9d7795c700  5 -- op tracker -- , seq: 748544, time: 2014-06-12 14:48:07.969507, event: done, op: osd_sub_op(client.4305.0:142489 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2493 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.049829 7f9d5a16f700  5 -- op tracker -- , seq: 748545, time: 2014-06-12 14:48:08.048190, event: header_read, op: osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.049864 7f9d5a16f700  5 -- op tracker -- , seq: 748545, time: 2014-06-12 14:48:08.048192, event: throttled, op: osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.049872 7f9d5a16f700  5 -- op tracker -- , seq: 748545, time: 2014-06-12 14:48:08.049773, event: all_read, op: osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.049880 7f9d5a16f700  5 -- op tracker -- , seq: 748545, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.049886 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:08.049901 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:08.049910 7f9d5a16f700 15 osd.2 173 enqueue_op 0x28dd1800 prio 127 cost 516903 latency 0.001719 osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:08.050017 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:08.050028 7f9d61bd4700 10 osd.2 173 dequeue_op 0x28dd1800 prio 127 cost 516903 latency 0.001838 osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2493 (0'0,173'2493] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2492 active]
2014-06-12 14:48:08.050053 7f9d61bd4700  5 -- op tracker -- , seq: 748545, time: 2014-06-12 14:48:08.050049, event: reached_pg, op: osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.050061 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2493 (0'0,173'2493] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2492 active] handle_message: 0x28dd1800
2014-06-12 14:48:08.050074 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2493 (0'0,173'2493] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2492 active] sub_op_modify trans 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 v 173'2494 (transaction) 174
2014-06-12 14:48:08.050091 7f9d61bd4700  5 -- op tracker -- , seq: 748545, time: 2014-06-12 14:48:08.050087, event: started, op: osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.050105 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2493 (0'0,173'2493] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2492 active] append_log log((0'0,173'2493], crt=0'0) [173'2494 (173'2493) modify   66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 by client.4305.0:142490 2014-06-12 08:46:01.033806]
2014-06-12 14:48:08.050129 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2494 (0'0,173'2494] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2492 active] add_log_entry 173'2494 (173'2493) modify   66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 by client.4305.0:142490 2014-06-12 08:46:01.033806
2014-06-12 14:48:08.050150 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2494 (0'0,173'2494] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2492 active] append_log  adding 1 keys
2014-06-12 14:48:08.050183 7f9d61bd4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2494, trimmed: 
2014-06-12 14:48:08.050206 7f9d61bd4700  5 -- op tracker -- , seq: 748545, time: 2014-06-12 14:48:08.050201, event: started, op: osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.050216 7f9d61bd4700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:08.050220 7f9d61bd4700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:08.050221 7f9d61bd4700  5 queue_op 0xff14360 seq 2327 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:08.050233 7f9d81970700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:08.050229 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2494 (0'0,173'2494] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2492 active] do_sub_op osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:08.050244 7f9d61bd4700 10 osd.2 173 dequeue_op 0x28dd1800 finish
2014-06-12 14:48:08.050245 7f9d81970700  5 _do_op 0xff14360 seq 2327 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:08.050248 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:08.050250 7f9d81970700 10 _do_transaction on 0x13ec10b0
2014-06-12 14:48:08.050251 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:08.050256 7f9d81970700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:08.050355 7f9d81970700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:08.050378 7f9d81970700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:08.050421 7f9d81970700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:08.050514 7f9d81970700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:08.050521 7f9d81970700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:08.050525 7f9d81970700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:08.050542 7f9d81970700 15 _write 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 172032~516096
2014-06-12 14:48:08.050646 7f9d81970700 10 lookup_strip_header done  cid 3.4f4_head oid 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:08.050869 7f9d81970700 10 file_to_extents done 
2014-06-12 14:48:08.052238 7f9d81970700 10 _generic_write 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 172032~516096 = 0
2014-06-12 14:48:08.052327 7f9d81970700 15 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:08.052361 7f9d81970700 10 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 = 0
2014-06-12 14:48:08.052369 7f9d81970700 15 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:08.052378 7f9d81970700 10 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 = 0
2014-06-12 14:48:08.052533 7f9d81970700 20 set_header setting 16444 cid 3.4f4_head oid 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 parent seq 0
2014-06-12 14:48:08.052605 7f9d81970700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:08.052622 7f9d81970700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:08.052636 7f9d81970700  5 submit_transaction r = 0
2014-06-12 14:48:08.053766 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:08.054137 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:08.055508 7f9d81970700 10 _do_op 0xff14360 seq 2327 r = 0, finisher 0x69068c0 0
2014-06-12 14:48:08.055529 7f9d81970700 10 _finish_op 0xff14360 seq 2327 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:08.055536 7f9d81970700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:08.055539 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:08.055548 7f9d7715b700  5 -- op tracker -- , seq: 748545, time: 2014-06-12 14:48:08.055538, event: commit_sent, op: osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.055555 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2494 (0'0,173'2494] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2492 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:08.055575 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:08.055611 7f9d7795c700  5 -- op tracker -- , seq: 748545, time: 2014-06-12 14:48:08.055607, event: sub_op_applied, op: osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.055615 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2494 (0'0,173'2494] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active] sub_op_modify_applied on 0x13ec0fc0 op osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:08.055625 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2494 (0'0,173'2494] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active] op_applied on primary on version 173'2494
2014-06-12 14:48:08.055631 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2494 (0'0,173'2494] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active] op_applied on replica on version 173'2494
2014-06-12 14:48:08.055649 7f9d7795c700  5 -- op tracker -- , seq: 748545, time: 2014-06-12 14:48:08.055647, event: done, op: osd_sub_op(client.4305.0:142490 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2494 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.158150 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:08.218138 7f9d5a16f700  5 -- op tracker -- , seq: 748546, time: 2014-06-12 14:48:08.216245, event: header_read, op: osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.218151 7f9d5a16f700  5 -- op tracker -- , seq: 748546, time: 2014-06-12 14:48:08.216247, event: throttled, op: osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.218159 7f9d5a16f700  5 -- op tracker -- , seq: 748546, time: 2014-06-12 14:48:08.218099, event: all_read, op: osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.218166 7f9d5a16f700  5 -- op tracker -- , seq: 748546, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.218171 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:08.218177 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:08.218184 7f9d5a16f700 15 osd.2 173 enqueue_op 0x28dd0800 prio 127 cost 1032999 latency 0.001939 osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:08.218233 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:08.218249 7f9d633d7700 10 osd.2 173 dequeue_op 0x28dd0800 prio 127 cost 1032999 latency 0.002004 osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2494 (0'0,173'2494] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active]
2014-06-12 14:48:08.218270 7f9d633d7700  5 -- op tracker -- , seq: 748546, time: 2014-06-12 14:48:08.218267, event: reached_pg, op: osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.218279 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2494 (0'0,173'2494] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active] handle_message: 0x28dd0800
2014-06-12 14:48:08.218292 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2494 (0'0,173'2494] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active] sub_op_modify trans 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 v 173'2495 (transaction) 174
2014-06-12 14:48:08.218321 7f9d633d7700  5 -- op tracker -- , seq: 748546, time: 2014-06-12 14:48:08.218318, event: started, op: osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.218339 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2494 (0'0,173'2494] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active] append_log log((0'0,173'2494], crt=0'0) [173'2495 (173'2494) modify   66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 by client.4305.0:142491 2014-06-12 08:46:01.056840]
2014-06-12 14:48:08.218357 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2495 (0'0,173'2495] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active] add_log_entry 173'2495 (173'2494) modify   66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 by client.4305.0:142491 2014-06-12 08:46:01.056840
2014-06-12 14:48:08.218374 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2495 (0'0,173'2495] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active] append_log  adding 1 keys
2014-06-12 14:48:08.218402 7f9d633d7700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2495, trimmed: 
2014-06-12 14:48:08.218419 7f9d633d7700  5 -- op tracker -- , seq: 748546, time: 2014-06-12 14:48:08.218416, event: started, op: osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.218427 7f9d633d7700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:08.218430 7f9d633d7700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:08.218431 7f9d633d7700  5 queue_op 0x9bac900 seq 2328 osr(3.4f4 0x276db70) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:48:08.218436 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2495 (0'0,173'2495] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active] do_sub_op osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:08.218440 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:08.218445 7f9d633d7700 10 osd.2 173 dequeue_op 0x28dd0800 finish
2014-06-12 14:48:08.218447 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:08.218449 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:08.218447 7f9d7a962700  5 _do_op 0x9bac900 seq 2328 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:08.218452 7f9d7a962700 10 _do_transaction on 0x1cef7430
2014-06-12 14:48:08.218458 7f9d7a962700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:08.218570 7f9d7a962700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:08.218597 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:08.218650 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:08.218747 7f9d7a962700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:08.218756 7f9d7a962700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:08.218762 7f9d7a962700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:08.218782 7f9d7a962700 15 _write 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 688128~1032192
2014-06-12 14:48:08.218886 7f9d7a962700 10 lookup_strip_header done  cid 3.4f4_head oid 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:08.219013 7f9d7a962700 10 file_to_extents done 
2014-06-12 14:48:08.222060 7f9d7a962700 10 _generic_write 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 688128~1032192 = 0
2014-06-12 14:48:08.222159 7f9d7a962700 15 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:08.222185 7f9d7a962700 10 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 = 0
2014-06-12 14:48:08.222192 7f9d7a962700 15 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:08.222202 7f9d7a962700 10 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 = 0
2014-06-12 14:48:08.222424 7f9d7a962700 20 set_header setting 16444 cid 3.4f4_head oid 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 parent seq 0
2014-06-12 14:48:08.222509 7f9d7a962700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:08.222529 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:08.222540 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:48:08.226887 7f9d7a962700 10 _do_op 0x9bac900 seq 2328 r = 0, finisher 0x4f549e0 0
2014-06-12 14:48:08.226906 7f9d7a962700 10 _finish_op 0x9bac900 seq 2328 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:08.226914 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:08.226919 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:08.226962 7f9d7795c700  5 -- op tracker -- , seq: 748546, time: 2014-06-12 14:48:08.226944, event: sub_op_applied, op: osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.226983 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2495 (0'0,173'2495] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active] sub_op_modify_applied on 0x1cef7340 op osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:08.227025 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:08.227054 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2495 (0'0,173'2495] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active] op_applied on primary on version 173'2495
2014-06-12 14:48:08.227072 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2495 (0'0,173'2495] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active] op_applied on replica on version 173'2495
2014-06-12 14:48:08.227212 7f9d7715b700  5 -- op tracker -- , seq: 748546, time: 2014-06-12 14:48:08.227189, event: commit_sent, op: osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.227238 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2495 (0'0,173'2495] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2493 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:08.227276 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:08.227332 7f9d7715b700  5 -- op tracker -- , seq: 748546, time: 2014-06-12 14:48:08.227325, event: done, op: osd_sub_op(client.4305.0:142491 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2495 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.307717 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:08.317191 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:08.356600 7f9d5a16f700  5 -- op tracker -- , seq: 748547, time: 2014-06-12 14:48:08.355565, event: header_read, op: osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.356617 7f9d5a16f700  5 -- op tracker -- , seq: 748547, time: 2014-06-12 14:48:08.355567, event: throttled, op: osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.356626 7f9d5a16f700  5 -- op tracker -- , seq: 748547, time: 2014-06-12 14:48:08.356552, event: all_read, op: osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.356633 7f9d5a16f700  5 -- op tracker -- , seq: 748547, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.356638 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:08.356646 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:08.356655 7f9d5a16f700 15 osd.2 173 enqueue_op 0x59dd100 prio 127 cost 516903 latency 0.001090 osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:08.356710 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:08.356724 7f9d673df700 10 osd.2 173 dequeue_op 0x59dd100 prio 127 cost 516903 latency 0.001159 osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2495 (0'0,173'2495] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2494 active]
2014-06-12 14:48:08.356756 7f9d673df700  5 -- op tracker -- , seq: 748547, time: 2014-06-12 14:48:08.356751, event: reached_pg, op: osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.356776 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2495 (0'0,173'2495] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2494 active] handle_message: 0x59dd100
2014-06-12 14:48:08.356790 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2495 (0'0,173'2495] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2494 active] sub_op_modify trans 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 v 173'2496 (transaction) 174
2014-06-12 14:48:08.356807 7f9d673df700  5 -- op tracker -- , seq: 748547, time: 2014-06-12 14:48:08.356803, event: started, op: osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.356818 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2495 (0'0,173'2495] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2494 active] append_log log((0'0,173'2495], crt=0'0) [173'2496 (173'2495) modify   66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 by client.4305.0:142492 2014-06-12 08:46:01.187306]
2014-06-12 14:48:08.356843 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2496 (0'0,173'2496] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2494 active] add_log_entry 173'2496 (173'2495) modify   66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 by client.4305.0:142492 2014-06-12 08:46:01.187306
2014-06-12 14:48:08.356867 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2496 (0'0,173'2496] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2494 active] append_log  adding 1 keys
2014-06-12 14:48:08.356902 7f9d673df700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2496, trimmed: 
2014-06-12 14:48:08.356926 7f9d673df700  5 -- op tracker -- , seq: 748547, time: 2014-06-12 14:48:08.356920, event: started, op: osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.356937 7f9d673df700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:08.356941 7f9d673df700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:08.356943 7f9d673df700  5 queue_op 0x2847b320 seq 2329 osr(3.4f4 0x276db70) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:08.356948 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2496 (0'0,173'2496] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2494 active] do_sub_op osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:08.356962 7f9d673df700 10 osd.2 173 dequeue_op 0x59dd100 finish
2014-06-12 14:48:08.356966 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:08.356968 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:08.356965 7f9d7895e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:08.356975 7f9d7895e700  5 _do_op 0x2847b320 seq 2329 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:08.356981 7f9d7895e700 10 _do_transaction on 0x6239270
2014-06-12 14:48:08.356988 7f9d7895e700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:08.357111 7f9d7895e700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:08.357139 7f9d7895e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:08.357188 7f9d7895e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:08.357317 7f9d7895e700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:08.357325 7f9d7895e700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:08.357332 7f9d7895e700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:08.357352 7f9d7895e700 15 _write 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 1720320~516096
2014-06-12 14:48:08.357459 7f9d7895e700 10 lookup_strip_header done  cid 3.4f4_head oid 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:08.357707 7f9d7895e700 10 file_to_extents done 
2014-06-12 14:48:08.359360 7f9d7895e700 10 _generic_write 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 1720320~516096 = 0
2014-06-12 14:48:08.359414 7f9d7895e700 15 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:08.359428 7f9d7895e700 10 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 = 0
2014-06-12 14:48:08.359434 7f9d7895e700 15 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:08.359443 7f9d7895e700 10 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 = 0
2014-06-12 14:48:08.359599 7f9d7895e700 20 set_header setting 16444 cid 3.4f4_head oid 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 parent seq 0
2014-06-12 14:48:08.359665 7f9d7895e700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:08.359682 7f9d7895e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:08.359696 7f9d7895e700  5 submit_transaction r = 0
2014-06-12 14:48:08.361390 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:08.361574 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:08.361764 7f9d7895e700 10 _do_op 0x2847b320 seq 2329 r = 0, finisher 0x6de9020 0
2014-06-12 14:48:08.361855 7f9d7895e700 10 _finish_op 0x2847b320 seq 2329 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:08.361867 7f9d7895e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:08.361879 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:08.361876 7f9d7715b700  5 -- op tracker -- , seq: 748547, time: 2014-06-12 14:48:08.361860, event: commit_sent, op: osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.361889 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2496 (0'0,173'2496] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2494 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:08.361926 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:08.361969 7f9d7795c700  5 -- op tracker -- , seq: 748547, time: 2014-06-12 14:48:08.361960, event: sub_op_applied, op: osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.361982 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2496 (0'0,173'2496] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2495 active] sub_op_modify_applied on 0x6239180 op osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:08.362002 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2496 (0'0,173'2496] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2495 active] op_applied on primary on version 173'2496
2014-06-12 14:48:08.362012 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2496 (0'0,173'2496] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2495 active] op_applied on replica on version 173'2496
2014-06-12 14:48:08.362041 7f9d7795c700  5 -- op tracker -- , seq: 748547, time: 2014-06-12 14:48:08.362037, event: done, op: osd_sub_op(client.4305.0:142492 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2496 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:08.372066 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:08.372223 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:08.372793 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:08.490252 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:08.490288 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:08.490293 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:08.490295 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:08.490329 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:08.495835 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:08.504642 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:08.513899 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:08.523103 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:08.524390 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:08.524446 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:08.642304 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:08.652493 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:08.656483 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:08.656519 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:08.684259 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:08.692551 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:08.715624 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:08.715653 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:08.741595 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19434 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:08.741634 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19434 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:08.741986 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:08.742077 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:08.742099 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:08.742116 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:08.742151 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:08.742170 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:08.742189 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:08.742220 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:08.742243 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:08.742261 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:08.742278 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:08.742296 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:08.742321 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:08.742339 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:08.742355 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:08.742372 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:08.742388 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:08.742404 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:08.742426 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:08.742443 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:08.742460 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:08.742477 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:08.824224 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:08.834702 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:08.834902 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:08.840937 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:08.861934 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:08.861998 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:08.981913 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:08.985434 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:09.003868 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:09.008596 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:09.086329 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:09.086380 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:09.105786 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:09.115766 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:09.167035 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:09.177513 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:09.180379 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:09.184784 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:09.197722 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:09.200498 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:09.273472 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:09.289231 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:09.327868 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:09.332087 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:09.364814 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:09.381177 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:09.490361 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:09.490380 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:09.490383 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:09.490392 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:09.490407 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:09.836596 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:09.836652 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:09.965834 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:09.967644 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:10.050325 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:10.055610 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:10.158244 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:10.187128 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:10.187264 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:10.218521 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:10.226990 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:10.307805 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:10.317273 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:10.337137 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:10.337254 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:10.357031 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:10.361940 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:10.372857 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:10.490469 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:10.490548 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:10.490550 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 4.5 >= max 0.5 = no, load too high
2014-06-12 14:48:10.490567 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:10.490571 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92738 < max (604800 seconds)
2014-06-12 14:48:10.490596 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:10.490599 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:10.490601 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:10.495918 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:10.504720 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:10.513976 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:10.523179 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:10.562515 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:10.562543 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:10.642391 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:10.652578 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:10.684340 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:10.692626 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:10.824301 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:10.834781 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:10.834981 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:10.837786 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:10.837822 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:10.841002 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:10.981998 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:10.985528 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:11.003949 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:11.008672 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:11.042064 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19434 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:11.042079 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19434 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:11.042448 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:11.042500 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:11.042606 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:11.042639 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:11.042656 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:11.042678 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:11.042753 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:11.042772 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:11.042831 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:11.042869 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:11.042901 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:11.042919 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:11.042940 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:11.042975 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:11.042995 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:11.043030 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:11.043061 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:11.043090 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:11.043115 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:11.043142 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:11.043168 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:11.043189 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:11.105869 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:11.115847 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:11.167141 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:11.177610 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:11.180461 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:11.184866 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:11.197797 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:11.200588 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:11.229088 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:11.229148 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:11.271155 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:11.271222 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:11.273554 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:11.289331 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:11.327979 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:11.332169 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:11.364915 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:11.381274 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:11.425048 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:11.425291 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:11.490676 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:11.490705 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:11.490712 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:11.490716 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:11.490742 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:11.965928 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:11.967740 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:12.050409 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:12.055711 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:12.154556 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:12.154707 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:12.158333 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:12.218598 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:12.227075 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:12.307894 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:12.317355 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:12.357102 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:12.362015 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:12.372929 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:12.472747 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:12.472800 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:12.490794 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:12.490815 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:48:12.490819 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:48:12.490823 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:48:12.490832 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:12.490836 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:12.490838 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:12.490875 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:12.496016 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:12.504799 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:12.514059 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:12.523277 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:12.642482 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:12.652674 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:12.684440 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:12.692698 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:12.824401 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:12.833798 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:12.833943 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:12.834856 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:12.835055 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:12.841079 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:12.972092 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:12.972178 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:12.982087 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:12.985608 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:13.004029 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:13.008753 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:13.041269 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:13.041336 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:13.105957 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:13.115945 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:13.138427 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:13.138530 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:13.167243 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:13.177705 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:13.180562 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:13.184963 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:13.197881 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:13.200683 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:13.273635 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:13.289401 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:13.328077 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:13.332259 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:13.364985 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:13.381344 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:13.490931 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:13.490990 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 4.5 >= max 0.5 = no, load too high
2014-06-12 14:48:13.491004 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:13.491008 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:13.491008 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92741 < max (604800 seconds)
2014-06-12 14:48:13.491029 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:13.491031 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:13.491033 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:13.966018 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:13.967838 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:14.050490 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:14.055807 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:14.129999 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:14.130064 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:14.158433 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:14.218673 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:14.227156 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:14.307996 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:14.317436 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:14.357181 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:14.362096 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:14.373024 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:14.455067 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:14.455174 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:14.491128 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:14.491153 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:14.491158 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:14.491161 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:14.491201 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:14.496114 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:14.504894 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:14.514141 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:14.523371 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:14.557193 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:14.557458 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:14.616507 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:14.616721 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:14.642574 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:14.652756 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:14.684541 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:14.692779 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:14.824501 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:14.834950 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:14.835139 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:14.841159 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:14.925802 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:14.925925 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:14.982168 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:14.985703 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:15.004125 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:15.008833 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:15.057797 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:15.057857 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:15.106061 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:15.116018 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:15.167335 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:15.177788 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:15.180664 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:15.185058 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:15.197981 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:15.200779 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:15.272967 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:15.273060 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:15.273706 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:15.289487 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:15.328178 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:15.332358 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:15.365078 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:15.381419 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:15.488002 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:15.488092 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:15.491251 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:15.491305 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 4.22 >= max 0.5 = no, load too high
2014-06-12 14:48:15.491320 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:15.491321 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:15.491323 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92743 < max (604800 seconds)
2014-06-12 14:48:15.491342 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:15.491345 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:15.491347 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:15.558718 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:15.558825 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:15.734697 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:15.734989 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:15.966107 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:15.967936 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.050572 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:16.055893 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.158530 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:16.218756 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:16.227252 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.308084 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:16.317517 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.342561 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19434 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:16.342581 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19434 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:16.343344 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:16.343435 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:16.343504 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:16.343561 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:16.343608 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:16.343632 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:16.343664 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:16.343690 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:16.343711 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:16.343732 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:16.343756 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:16.343784 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:16.343816 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:16.343844 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:16.343868 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:16.343891 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:16.343919 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:16.343942 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:16.343966 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:16.343993 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:16.344017 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:16.344041 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:16.357274 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:16.362179 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.373119 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:16.463104 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:16.463229 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:16.491435 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:16.491459 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:16.491464 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:16.491467 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:16.491506 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:16.496210 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:16.504992 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:16.514216 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.523465 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.642658 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:16.652852 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.684642 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:16.692862 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.741189 7f9d5a16f700  5 -- op tracker -- , seq: 748548, time: 2014-06-12 14:48:16.738243, event: header_read, op: osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.741213 7f9d5a16f700  5 -- op tracker -- , seq: 748548, time: 2014-06-12 14:48:16.738247, event: throttled, op: osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.741244 7f9d5a16f700  5 -- op tracker -- , seq: 748548, time: 2014-06-12 14:48:16.741113, event: all_read, op: osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.741255 7f9d5a16f700  5 -- op tracker -- , seq: 748548, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.741263 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:16.741273 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:16.741283 7f9d5a16f700 15 osd.2 173 enqueue_op 0x59dc600 prio 127 cost 1549095 latency 0.003040 osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.741360 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:16.741384 7f9d693e3700 10 osd.2 173 dequeue_op 0x59dc600 prio 127 cost 1549095 latency 0.003140 osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2496 (0'0,173'2496] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2495 active]
2014-06-12 14:48:16.741427 7f9d693e3700  5 -- op tracker -- , seq: 748548, time: 2014-06-12 14:48:16.741419, event: reached_pg, op: osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.741483 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2496 (0'0,173'2496] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2495 active] handle_message: 0x59dc600
2014-06-12 14:48:16.741512 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2496 (0'0,173'2496] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2495 active] sub_op_modify trans 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 v 173'2497 (transaction) 174
2014-06-12 14:48:16.741558 7f9d693e3700  5 -- op tracker -- , seq: 748548, time: 2014-06-12 14:48:16.741551, event: started, op: osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.741604 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2496 (0'0,173'2496] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2495 active] append_log log((0'0,173'2496], crt=0'0) [173'2497 (173'2496) modify   66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 by client.4305.0:142493 2014-06-12 08:46:03.147038]
2014-06-12 14:48:16.741685 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2497 (0'0,173'2497] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2495 active] add_log_entry 173'2497 (173'2496) modify   66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 by client.4305.0:142493 2014-06-12 08:46:03.147038
2014-06-12 14:48:16.741740 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2497 (0'0,173'2497] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2495 active] append_log  adding 1 keys
2014-06-12 14:48:16.741805 7f9d693e3700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2497, trimmed: 
2014-06-12 14:48:16.741866 7f9d693e3700  5 -- op tracker -- , seq: 748548, time: 2014-06-12 14:48:16.741858, event: started, op: osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.741902 7f9d693e3700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:16.741908 7f9d693e3700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:16.741910 7f9d693e3700  5 queue_op 0x6873740 seq 2330 osr(3.4f4 0x276db70) 1550581 bytes   (queue has 1 ops and 1550581 bytes)
2014-06-12 14:48:16.741920 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2497 (0'0,173'2497] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2495 active] do_sub_op osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.741940 7f9d693e3700 10 osd.2 173 dequeue_op 0x59dc600 finish
2014-06-12 14:48:16.741934 7f9d7815d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:16.741945 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:16.741949 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:16.741951 7f9d7815d700  5 _do_op 0x6873740 seq 2330 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:16.741962 7f9d7815d700 10 _do_transaction on 0xe0d7eb0
2014-06-12 14:48:16.741977 7f9d7815d700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:16.742172 7f9d7815d700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:16.742213 7f9d7815d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:16.742300 7f9d7815d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:16.742452 7f9d7815d700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:16.742472 7f9d7815d700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:16.742486 7f9d7815d700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:16.742524 7f9d7815d700 15 _write 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 2236416~1548288
2014-06-12 14:48:16.742830 7f9d7815d700 10 lookup_strip_header done  cid 3.4f4_head oid 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:16.743221 7f9d7815d700 10 file_to_extents done 
2014-06-12 14:48:16.748947 7f9d7815d700 10 _generic_write 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 2236416~1548288 = 0
2014-06-12 14:48:16.749075 7f9d7815d700 15 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:16.749097 7f9d7815d700 10 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 = 0
2014-06-12 14:48:16.749115 7f9d7815d700 15 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:16.749122 7f9d7815d700 10 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 = 0
2014-06-12 14:48:16.749417 7f9d7815d700 20 set_header setting 16444 cid 3.4f4_head oid 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 parent seq 0
2014-06-12 14:48:16.749546 7f9d7815d700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:16.749568 7f9d7815d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:16.749574 7f9d7815d700  5 submit_transaction r = 0
2014-06-12 14:48:16.759794 7f9d7815d700 10 _do_op 0x6873740 seq 2330 r = 0, finisher 0x7bd52c0 0
2014-06-12 14:48:16.759810 7f9d7815d700 10 _finish_op 0x6873740 seq 2330 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:16.759817 7f9d7815d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:16.759829 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.759843 7f9d7715b700  5 -- op tracker -- , seq: 748548, time: 2014-06-12 14:48:16.759826, event: commit_sent, op: osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.759855 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2497 (0'0,173'2497] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2495 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:16.759885 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:16.759951 7f9d7795c700  5 -- op tracker -- , seq: 748548, time: 2014-06-12 14:48:16.759944, event: sub_op_applied, op: osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.759971 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2497 (0'0,173'2497] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2496 active] sub_op_modify_applied on 0xe0d7dc0 op osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.759994 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2497 (0'0,173'2497] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2496 active] op_applied on primary on version 173'2497
2014-06-12 14:48:16.760014 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2497 (0'0,173'2497] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2496 active] op_applied on replica on version 173'2497
2014-06-12 14:48:16.760033 7f9d7795c700  5 -- op tracker -- , seq: 748548, time: 2014-06-12 14:48:16.760030, event: done, op: osd_sub_op(client.4305.0:142493 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2497 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.786747 7f9d5a16f700  5 -- op tracker -- , seq: 748549, time: 2014-06-12 14:48:16.785909, event: header_read, op: osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.786767 7f9d5a16f700  5 -- op tracker -- , seq: 748549, time: 2014-06-12 14:48:16.785912, event: throttled, op: osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.786780 7f9d5a16f700  5 -- op tracker -- , seq: 748549, time: 2014-06-12 14:48:16.786688, event: all_read, op: osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.786792 7f9d5a16f700  5 -- op tracker -- , seq: 748549, time: 0.000000, event: dispatched, op: osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.786799 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:16.786808 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:16.786818 7f9d5a16f700 15 osd.2 173 enqueue_op 0x59dce00 prio 127 cost 410407 latency 0.000908 osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.786889 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:16.786913 7f9d683e1700 10 osd.2 173 dequeue_op 0x59dce00 prio 127 cost 410407 latency 0.001004 osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.4f4( v 173'2497 (0'0,173'2497] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2496 active]
2014-06-12 14:48:16.786956 7f9d683e1700  5 -- op tracker -- , seq: 748549, time: 2014-06-12 14:48:16.786948, event: reached_pg, op: osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.786972 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2497 (0'0,173'2497] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2496 active] handle_message: 0x59dce00
2014-06-12 14:48:16.786988 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2497 (0'0,173'2497] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2496 active] sub_op_modify trans 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 v 173'2498 (transaction) 174
2014-06-12 14:48:16.787010 7f9d683e1700  5 -- op tracker -- , seq: 748549, time: 2014-06-12 14:48:16.787004, event: started, op: osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.787029 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2497 (0'0,173'2497] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2496 active] append_log log((0'0,173'2497], crt=0'0) [173'2498 (173'2497) modify   66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 by client.4305.0:142494 2014-06-12 08:46:03.486904]
2014-06-12 14:48:16.787062 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2498 (0'0,173'2498] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2496 active] add_log_entry 173'2498 (173'2497) modify   66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 by client.4305.0:142494 2014-06-12 08:46:03.486904
2014-06-12 14:48:16.787103 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2498 (0'0,173'2498] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2496 active] append_log  adding 1 keys
2014-06-12 14:48:16.787146 7f9d683e1700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2498, trimmed: 
2014-06-12 14:48:16.787179 7f9d683e1700  5 -- op tracker -- , seq: 748549, time: 2014-06-12 14:48:16.787172, event: started, op: osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.787195 7f9d683e1700  5 queue_transactions existing osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:16.787200 7f9d683e1700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:16.787202 7f9d683e1700  5 queue_op 0xcb337a0 seq 2331 osr(3.4f4 0x276db70) 411893 bytes   (queue has 1 ops and 411893 bytes)
2014-06-12 14:48:16.787210 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2498 (0'0,173'2498] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2496 active] do_sub_op osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.787230 7f9d683e1700 10 osd.2 173 dequeue_op 0x59dce00 finish
2014-06-12 14:48:16.787235 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:16.787239 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:16.787253 7f9d82171700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x52e7e60 (1 active)
2014-06-12 14:48:16.787269 7f9d82171700  5 _do_op 0xcb337a0 seq 2331 osr(3.4f4 0x276db70)/0x276db70 start
2014-06-12 14:48:16.787276 7f9d82171700 10 _do_transaction on 0x33f3aef0
2014-06-12 14:48:16.787286 7f9d82171700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:16.787423 7f9d82171700 10 lookup_strip_header done  cid meta oid 1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:16.787462 7f9d82171700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:16.787532 7f9d82171700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:16.787567 7f9d82171700 15 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:16.787574 7f9d82171700 10 _omap_rmkeys meta/1fe00265/pglog_3.4f4/0//-1 r = 0
2014-06-12 14:48:16.787582 7f9d82171700 15 _omap_setkeys meta/1fe00265/pglog_3.4f4/0//-1
2014-06-12 14:48:16.787604 7f9d82171700 15 _write 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 3784704~409600
2014-06-12 14:48:16.787782 7f9d82171700 10 lookup_strip_header done  cid 3.4f4_head oid 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:16.788127 7f9d82171700 10 file_to_extents done 
2014-06-12 14:48:16.789852 7f9d82171700 10 _generic_write 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 3784704~409600 = 0
2014-06-12 14:48:16.789900 7f9d82171700 15 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:16.789916 7f9d82171700 10 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 = 0
2014-06-12 14:48:16.789924 7f9d82171700 15 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3
2014-06-12 14:48:16.789934 7f9d82171700 10 _setattrs 3.4f4_head/66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 = 0
2014-06-12 14:48:16.790170 7f9d82171700 20 set_header setting 16444 cid 3.4f4_head oid 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 parent seq 0
2014-06-12 14:48:16.790246 7f9d82171700 20 set_header setting 2252 cid meta oid 1fe00265/pglog_3.4f4/0//-1 parent seq 0
2014-06-12 14:48:16.790272 7f9d82171700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:16.790283 7f9d82171700  5 submit_transaction r = 0
2014-06-12 14:48:16.793003 7f9d82171700 10 _do_op 0xcb337a0 seq 2331 r = 0, finisher 0x7fbff40 0
2014-06-12 14:48:16.793022 7f9d82171700 10 _finish_op 0xcb337a0 seq 2331 osr(3.4f4 0x276db70)/0x276db70
2014-06-12 14:48:16.793033 7f9d82171700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x52e7e60 (0 active)
2014-06-12 14:48:16.793038 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.793053 7f9d7715b700  5 -- op tracker -- , seq: 748549, time: 2014-06-12 14:48:16.793039, event: commit_sent, op: osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.793073 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2498 (0'0,173'2498] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2496 active] sub_op_modify_commit on op osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:16.793104 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:16.793187 7f9d7795c700  5 -- op tracker -- , seq: 748549, time: 2014-06-12 14:48:16.793169, event: sub_op_applied, op: osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.793215 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2498 (0'0,173'2498] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2497 active] sub_op_modify_applied on 0x33f3ae00 op osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.793251 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2498 (0'0,173'2498] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2497 active] op_applied on primary on version 173'2498
2014-06-12 14:48:16.793264 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.4f4( v 173'2498 (0'0,173'2498] local-les=147 n=285 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2497 active] op_applied on replica on version 173'2498
2014-06-12 14:48:16.793294 7f9d7795c700  5 -- op tracker -- , seq: 748549, time: 2014-06-12 14:48:16.793288, event: done, op: osd_sub_op(client.4305.0:142494 3.4f4 66fa8cf4/rbd_data.10a86b8b4567.0000000000000ee3/head//3 [] v 173'2498 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.845524 7f9d5a16f700  5 -- op tracker -- , seq: 748550, time: 2014-06-12 14:48:16.843189, event: header_read, op: osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.845544 7f9d5a16f700  5 -- op tracker -- , seq: 748550, time: 2014-06-12 14:48:16.843192, event: throttled, op: osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.845557 7f9d5a16f700  5 -- op tracker -- , seq: 748550, time: 2014-06-12 14:48:16.845467, event: all_read, op: osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.845568 7f9d5a16f700  5 -- op tracker -- , seq: 748550, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.845575 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:16.845584 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:16.845594 7f9d5a16f700 15 osd.2 173 enqueue_op 0x59dd000 prio 127 cost 799527 latency 0.002404 osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.845674 7f9d6abe6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:16.845692 7f9d6abe6700 10 osd.2 173 dequeue_op 0x59dd000 prio 127 cost 799527 latency 0.002502 osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2516 (0'0,173'2516] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2515 active]
2014-06-12 14:48:16.845738 7f9d6abe6700  5 -- op tracker -- , seq: 748550, time: 2014-06-12 14:48:16.845731, event: reached_pg, op: osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.845762 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2516 (0'0,173'2516] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2515 active] handle_message: 0x59dd000
2014-06-12 14:48:16.845778 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2516 (0'0,173'2516] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2515 active] sub_op_modify trans 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 v 173'2517 (transaction) 174
2014-06-12 14:48:16.845805 7f9d6abe6700  5 -- op tracker -- , seq: 748550, time: 2014-06-12 14:48:16.845799, event: started, op: osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.845825 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2516 (0'0,173'2516] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2515 active] append_log log((0'0,173'2516], crt=0'0) [173'2517 (173'555) modify   2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 by client.4300.0:140083 2014-06-12 08:45:01.910922]
2014-06-12 14:48:16.845861 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2517 (0'0,173'2517] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2515 active] add_log_entry 173'2517 (173'555) modify   2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 by client.4300.0:140083 2014-06-12 08:45:01.910922
2014-06-12 14:48:16.845897 7f9d6abe6700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2517 (0'0,173'2517] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2515 active] append_log  adding 1 keys
2014-06-12 14:48:16.845943 7f9d6abe6700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2517, trimmed: 
2014-06-12 14:48:16.845978 7f9d6abe6700  5 -- op tracker -- , seq: 748550, time: 2014-06-12 14:48:16.845971, event: started, op: osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.845996 7f9d6abe6700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:48:16.846002 7f9d6abe6700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:16.846003 7f9d6abe6700  5 queue_op 0x790a840 seq 2363 osr(3.168 0x276ce40) 801013 bytes   (queue has 1 ops and 801013 bytes)
2014-06-12 14:48:16.846011 7f9d6abe6700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2517 (0'0,173'2517] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2515 active] do_sub_op osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.846031 7f9d6abe6700 10 osd.2 173 dequeue_op 0x59dd000 finish
2014-06-12 14:48:16.846036 7f9d6abe6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:16.846040 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:16.846047 7f9d7e96a700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:48:16.846064 7f9d7e96a700  5 _do_op 0x790a840 seq 2363 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:48:16.846071 7f9d7e96a700 10 _do_transaction on 0x1909eb70
2014-06-12 14:48:16.846081 7f9d7e96a700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:16.846241 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:16.846286 7f9d7e96a700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:16.846354 7f9d7e96a700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:16.846462 7f9d7e96a700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:16.846474 7f9d7e96a700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:48:16.846483 7f9d7e96a700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:16.846506 7f9d7e96a700 15 _write 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 0~798720
2014-06-12 14:48:16.846749 7f9d7e96a700 10 lookup_strip_header done  cid 3.168_head oid 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:16.846978 7f9d7e96a700 10 file_to_extents done 
2014-06-12 14:48:16.850482 7f9d7e96a700 10 _generic_write 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 0~798720 = 0
2014-06-12 14:48:16.850562 7f9d7e96a700 15 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:16.850586 7f9d7e96a700 10 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 = 0
2014-06-12 14:48:16.850606 7f9d7e96a700 15 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:16.850617 7f9d7e96a700 10 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 = 0
2014-06-12 14:48:16.850954 7f9d7e96a700 20 set_header setting 16545 cid 3.168_head oid 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 parent seq 0
2014-06-12 14:48:16.851081 7f9d7e96a700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:16.851104 7f9d7e96a700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:48:16.851117 7f9d7e96a700  5 submit_transaction r = 0
2014-06-12 14:48:16.856135 7f9d7e96a700 10 _do_op 0x790a840 seq 2363 r = 0, finisher 0x6f58320 0
2014-06-12 14:48:16.856158 7f9d7e96a700 10 _finish_op 0x790a840 seq 2363 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:48:16.856184 7f9d7e96a700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:48:16.856190 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.856192 7f9d7715b700  5 -- op tracker -- , seq: 748550, time: 2014-06-12 14:48:16.856181, event: commit_sent, op: osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.856203 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2517 (0'0,173'2517] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2515 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:16.856234 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:16.856309 7f9d7795c700  5 -- op tracker -- , seq: 748550, time: 2014-06-12 14:48:16.856304, event: sub_op_applied, op: osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.856317 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2517 (0'0,173'2517] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active] sub_op_modify_applied on 0x1909ea80 op osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.856327 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2517 (0'0,173'2517] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active] op_applied on primary on version 173'2517
2014-06-12 14:48:16.856342 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2517 (0'0,173'2517] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active] op_applied on replica on version 173'2517
2014-06-12 14:48:16.856362 7f9d7795c700  5 -- op tracker -- , seq: 748550, time: 2014-06-12 14:48:16.856360, event: done, op: osd_sub_op(client.4300.0:140083 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2517 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.918423 7f9d5a16f700  5 -- op tracker -- , seq: 748551, time: 2014-06-12 14:48:16.916862, event: header_read, op: osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.918440 7f9d5a16f700  5 -- op tracker -- , seq: 748551, time: 2014-06-12 14:48:16.916867, event: throttled, op: osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.918452 7f9d5a16f700  5 -- op tracker -- , seq: 748551, time: 2014-06-12 14:48:16.918363, event: all_read, op: osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.918496 7f9d5a16f700  5 -- op tracker -- , seq: 748551, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.918506 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:16.918519 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:16.918534 7f9d5a16f700 15 osd.2 173 enqueue_op 0x59dcf00 prio 127 cost 516903 latency 0.001672 osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.918592 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:16.918615 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x59dcf00 prio 127 cost 516903 latency 0.001752 osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2517 (0'0,173'2517] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active]
2014-06-12 14:48:16.918675 7f9d6a3e5700  5 -- op tracker -- , seq: 748551, time: 2014-06-12 14:48:16.918664, event: reached_pg, op: osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.918694 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2517 (0'0,173'2517] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active] handle_message: 0x59dcf00
2014-06-12 14:48:16.918727 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2517 (0'0,173'2517] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active] sub_op_modify trans 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 v 173'2518 (transaction) 174
2014-06-12 14:48:16.918766 7f9d6a3e5700  5 -- op tracker -- , seq: 748551, time: 2014-06-12 14:48:16.918756, event: started, op: osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.918796 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2517 (0'0,173'2517] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active] append_log log((0'0,173'2517], crt=0'0) [173'2518 (173'2517) modify   2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 by client.4300.0:140084 2014-06-12 08:45:01.919126]
2014-06-12 14:48:16.918842 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2518 (0'0,173'2518] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active] add_log_entry 173'2518 (173'2517) modify   2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 by client.4300.0:140084 2014-06-12 08:45:01.919126
2014-06-12 14:48:16.918896 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2518 (0'0,173'2518] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active] append_log  adding 1 keys
2014-06-12 14:48:16.918973 7f9d6a3e5700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2518, trimmed: 
2014-06-12 14:48:16.919025 7f9d6a3e5700  5 -- op tracker -- , seq: 748551, time: 2014-06-12 14:48:16.919013, event: started, op: osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.919045 7f9d6a3e5700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:48:16.919053 7f9d6a3e5700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:16.919056 7f9d6a3e5700  5 queue_op 0x802f800 seq 2364 osr(3.168 0x276ce40) 518389 bytes   (queue has 1 ops and 518389 bytes)
2014-06-12 14:48:16.919074 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:48:16.919086 7f9d8116f700  5 _do_op 0x802f800 seq 2364 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:48:16.919092 7f9d8116f700 10 _do_transaction on 0x31d08ef0
2014-06-12 14:48:16.919068 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2518 (0'0,173'2518] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active] do_sub_op osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.919100 7f9d8116f700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:16.919102 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x59dcf00 finish
2014-06-12 14:48:16.919111 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:16.919117 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:16.919250 7f9d8116f700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:16.919286 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:16.919357 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:16.919483 7f9d8116f700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:16.919496 7f9d8116f700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:48:16.919505 7f9d8116f700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:16.919530 7f9d8116f700 15 _write 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 798720~516096
2014-06-12 14:48:16.919729 7f9d8116f700 10 lookup_strip_header done  cid 3.168_head oid 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:16.919931 7f9d8116f700 10 file_to_extents done 
2014-06-12 14:48:16.922302 7f9d8116f700 10 _generic_write 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 798720~516096 = 0
2014-06-12 14:48:16.922369 7f9d8116f700 15 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:16.922389 7f9d8116f700 10 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 = 0
2014-06-12 14:48:16.922397 7f9d8116f700 15 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:16.922407 7f9d8116f700 10 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 = 0
2014-06-12 14:48:16.922768 7f9d8116f700 20 set_header setting 16545 cid 3.168_head oid 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 parent seq 0
2014-06-12 14:48:16.922977 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:16.923013 7f9d8116f700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:48:16.923033 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:48:16.927232 7f9d8116f700 10 _do_op 0x802f800 seq 2364 r = 0, finisher 0x2079f60 0
2014-06-12 14:48:16.927252 7f9d8116f700 10 _finish_op 0x802f800 seq 2364 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:48:16.927290 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:48:16.927305 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.927286 7f9d7795c700  5 -- op tracker -- , seq: 748551, time: 2014-06-12 14:48:16.927277, event: sub_op_applied, op: osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.927341 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2518 (0'0,173'2518] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active] sub_op_modify_applied on 0x31d08e00 op osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.927366 7f9d7795c700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:16.927381 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2518 (0'0,173'2518] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active] op_applied on primary on version 173'2518
2014-06-12 14:48:16.927391 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2518 (0'0,173'2518] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active] op_applied on replica on version 173'2518
2014-06-12 14:48:16.927412 7f9d7715b700  5 -- op tracker -- , seq: 748551, time: 2014-06-12 14:48:16.927407, event: commit_sent, op: osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.927420 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2518 (0'0,173'2518] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2516 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:16.927449 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:16.927473 7f9d7715b700  5 -- op tracker -- , seq: 748551, time: 2014-06-12 14:48:16.927470, event: done, op: osd_sub_op(client.4300.0:140084 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2518 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.952388 7f9d5a16f700  5 -- op tracker -- , seq: 748552, time: 2014-06-12 14:48:16.951391, event: header_read, op: osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.952407 7f9d5a16f700  5 -- op tracker -- , seq: 748552, time: 2014-06-12 14:48:16.951395, event: throttled, op: osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.952426 7f9d5a16f700  5 -- op tracker -- , seq: 748552, time: 2014-06-12 14:48:16.952330, event: all_read, op: osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.952432 7f9d5a16f700  5 -- op tracker -- , seq: 748552, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.952437 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:16.952443 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:16.952451 7f9d5a16f700 15 osd.2 173 enqueue_op 0x59ddb00 prio 127 cost 520999 latency 0.001059 osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.952486 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:16.952503 7f9d66bde700 10 osd.2 173 dequeue_op 0x59ddb00 prio 127 cost 520999 latency 0.001112 osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2518 (0'0,173'2518] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2517 active]
2014-06-12 14:48:16.952532 7f9d66bde700  5 -- op tracker -- , seq: 748552, time: 2014-06-12 14:48:16.952528, event: reached_pg, op: osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.952544 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2518 (0'0,173'2518] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2517 active] handle_message: 0x59ddb00
2014-06-12 14:48:16.952560 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2518 (0'0,173'2518] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2517 active] sub_op_modify trans 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 v 173'2519 (transaction) 174
2014-06-12 14:48:16.952581 7f9d66bde700  5 -- op tracker -- , seq: 748552, time: 2014-06-12 14:48:16.952577, event: started, op: osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.952601 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2518 (0'0,173'2518] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2517 active] append_log log((0'0,173'2518], crt=0'0) [173'2519 (173'2518) modify   2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 by client.4300.0:140085 2014-06-12 08:45:01.929006]
2014-06-12 14:48:16.952629 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2519 (0'0,173'2519] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2517 active] add_log_entry 173'2519 (173'2518) modify   2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 by client.4300.0:140085 2014-06-12 08:45:01.929006
2014-06-12 14:48:16.952654 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2519 (0'0,173'2519] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2517 active] append_log  adding 1 keys
2014-06-12 14:48:16.952693 7f9d66bde700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2519, trimmed: 
2014-06-12 14:48:16.952722 7f9d66bde700  5 -- op tracker -- , seq: 748552, time: 2014-06-12 14:48:16.952717, event: started, op: osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.952735 7f9d66bde700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:48:16.952740 7f9d66bde700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:16.952742 7f9d66bde700  5 queue_op 0x132b84e0 seq 2365 osr(3.168 0x276ce40) 522485 bytes   (queue has 1 ops and 522485 bytes)
2014-06-12 14:48:16.952749 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2519 (0'0,173'2519] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2517 active] do_sub_op osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.952771 7f9d66bde700 10 osd.2 173 dequeue_op 0x59ddb00 finish
2014-06-12 14:48:16.952775 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:16.952778 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:16.952781 7f9d7d167700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:48:16.952791 7f9d7d167700  5 _do_op 0x132b84e0 seq 2365 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:48:16.952795 7f9d7d167700 10 _do_transaction on 0x5ea70b0
2014-06-12 14:48:16.952802 7f9d7d167700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:16.953097 7f9d7d167700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:16.953122 7f9d7d167700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:16.953158 7f9d7d167700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:16.953177 7f9d7d167700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:16.953182 7f9d7d167700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:48:16.953186 7f9d7d167700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:16.953198 7f9d7d167700 15 _write 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 1314816~520192
2014-06-12 14:48:16.953467 7f9d7d167700 10 lookup_strip_header done  cid 3.168_head oid 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:16.953606 7f9d7d167700 10 file_to_extents done 
2014-06-12 14:48:16.955083 7f9d7d167700 10 _generic_write 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 1314816~520192 = 0
2014-06-12 14:48:16.955127 7f9d7d167700 15 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:16.955139 7f9d7d167700 10 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 = 0
2014-06-12 14:48:16.955143 7f9d7d167700 15 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:16.955148 7f9d7d167700 10 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 = 0
2014-06-12 14:48:16.955292 7f9d7d167700 20 set_header setting 16545 cid 3.168_head oid 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 parent seq 0
2014-06-12 14:48:16.955338 7f9d7d167700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:16.955348 7f9d7d167700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:48:16.955355 7f9d7d167700  5 submit_transaction r = 0
2014-06-12 14:48:16.957147 7f9d7d167700 10 _do_op 0x132b84e0 seq 2365 r = 0, finisher 0x7f561a0 0
2014-06-12 14:48:16.957165 7f9d7d167700 10 _finish_op 0x132b84e0 seq 2365 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:48:16.957172 7f9d7d167700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:48:16.957183 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:16.957187 7f9d7715b700  5 -- op tracker -- , seq: 748552, time: 2014-06-12 14:48:16.957175, event: commit_sent, op: osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.957197 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2519 (0'0,173'2519] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2517 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:16.957233 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:16.957284 7f9d7795c700  5 -- op tracker -- , seq: 748552, time: 2014-06-12 14:48:16.957270, event: sub_op_applied, op: osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:16.957294 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2519 (0'0,173'2519] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2518 active] sub_op_modify_applied on 0x5ea6fc0 op osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:16.957308 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2519 (0'0,173'2519] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2518 active] op_applied on primary on version 173'2519
2014-06-12 14:48:16.957315 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2519 (0'0,173'2519] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2518 active] op_applied on replica on version 173'2519
2014-06-12 14:48:16.957332 7f9d7795c700  5 -- op tracker -- , seq: 748552, time: 2014-06-12 14:48:16.957329, event: done, op: osd_sub_op(client.4300.0:140085 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2519 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.112489 7f9d5a16f700  5 -- op tracker -- , seq: 748553, time: 2014-06-12 14:48:17.110589, event: header_read, op: osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.112503 7f9d5a16f700  5 -- op tracker -- , seq: 748553, time: 2014-06-12 14:48:17.110598, event: throttled, op: osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.112518 7f9d5a16f700  5 -- op tracker -- , seq: 748553, time: 2014-06-12 14:48:17.112452, event: all_read, op: osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.112523 7f9d5a16f700  5 -- op tracker -- , seq: 748553, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.112526 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:17.112530 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:17.112536 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903300 prio 127 cost 1032999 latency 0.001947 osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:17.112555 7f9d69be4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:17.112561 7f9d69be4700 10 osd.2 173 dequeue_op 0x2a903300 prio 127 cost 1032999 latency 0.001972 osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2519 (0'0,173'2519] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2518 active]
2014-06-12 14:48:17.112611 7f9d69be4700  5 -- op tracker -- , seq: 748553, time: 2014-06-12 14:48:17.112603, event: reached_pg, op: osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.112628 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2519 (0'0,173'2519] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2518 active] handle_message: 0x2a903300
2014-06-12 14:48:17.112644 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2519 (0'0,173'2519] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2518 active] sub_op_modify trans 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 v 173'2520 (transaction) 174
2014-06-12 14:48:17.112666 7f9d69be4700  5 -- op tracker -- , seq: 748553, time: 2014-06-12 14:48:17.112661, event: started, op: osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.112690 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2519 (0'0,173'2519] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2518 active] append_log log((0'0,173'2519], crt=0'0) [173'2520 (173'2519) modify   2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 by client.4300.0:140086 2014-06-12 08:45:04.350741]
2014-06-12 14:48:17.112726 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2520 (0'0,173'2520] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2518 active] add_log_entry 173'2520 (173'2519) modify   2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 by client.4300.0:140086 2014-06-12 08:45:04.350741
2014-06-12 14:48:17.112764 7f9d69be4700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2520 (0'0,173'2520] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2518 active] append_log  adding 1 keys
2014-06-12 14:48:17.112805 7f9d69be4700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2520, trimmed: 
2014-06-12 14:48:17.112838 7f9d69be4700  5 -- op tracker -- , seq: 748553, time: 2014-06-12 14:48:17.112831, event: started, op: osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.112851 7f9d69be4700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:48:17.112856 7f9d69be4700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:17.112858 7f9d69be4700  5 queue_op 0xe835d40 seq 2366 osr(3.168 0x276ce40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:48:17.112874 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:48:17.112867 7f9d69be4700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2520 (0'0,173'2520] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2518 active] do_sub_op osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:17.112887 7f9d69be4700 10 osd.2 173 dequeue_op 0x2a903300 finish
2014-06-12 14:48:17.112892 7f9d69be4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:17.112891 7f9d7915f700  5 _do_op 0xe835d40 seq 2366 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:48:17.112896 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:17.112901 7f9d7915f700 10 _do_transaction on 0x165e69b0
2014-06-12 14:48:17.112914 7f9d7915f700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:17.113210 7f9d7915f700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:17.113295 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:17.113412 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:17.113707 7f9d7915f700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:17.113726 7f9d7915f700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:48:17.113741 7f9d7915f700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:17.113786 7f9d7915f700 15 _write 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 1835008~1032192
2014-06-12 14:48:17.114022 7f9d7915f700 10 lookup_strip_header done  cid 3.168_head oid 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:17.114329 7f9d7915f700 10 file_to_extents done 
2014-06-12 14:48:17.118697 7f9d7915f700 10 _generic_write 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 1835008~1032192 = 0
2014-06-12 14:48:17.118777 7f9d7915f700 15 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:17.118799 7f9d7915f700 10 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 = 0
2014-06-12 14:48:17.118807 7f9d7915f700 15 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:17.118818 7f9d7915f700 10 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 = 0
2014-06-12 14:48:17.119045 7f9d7915f700 20 set_header setting 16545 cid 3.168_head oid 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 parent seq 0
2014-06-12 14:48:17.119142 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:17.119159 7f9d7915f700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:48:17.119177 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:48:17.123825 7f9d7915f700 10 _do_op 0xe835d40 seq 2366 r = 0, finisher 0x6606880 0
2014-06-12 14:48:17.123884 7f9d7915f700 10 _finish_op 0xe835d40 seq 2366 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:48:17.123907 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:48:17.123917 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:17.123918 7f9d7715b700  5 -- op tracker -- , seq: 748553, time: 2014-06-12 14:48:17.123901, event: commit_sent, op: osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.123941 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2520 (0'0,173'2520] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2518 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:17.123971 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:17.124037 7f9d7795c700  5 -- op tracker -- , seq: 748553, time: 2014-06-12 14:48:17.124031, event: sub_op_applied, op: osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.124048 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2520 (0'0,173'2520] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2519 active] sub_op_modify_applied on 0x165e68c0 op osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:17.124062 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2520 (0'0,173'2520] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2519 active] op_applied on primary on version 173'2520
2014-06-12 14:48:17.124070 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2520 (0'0,173'2520] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2519 active] op_applied on replica on version 173'2520
2014-06-12 14:48:17.124091 7f9d7795c700  5 -- op tracker -- , seq: 748553, time: 2014-06-12 14:48:17.124088, event: done, op: osd_sub_op(client.4300.0:140086 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2520 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.132902 7f9d5a16f700  5 -- op tracker -- , seq: 748554, time: 2014-06-12 14:48:17.130927, event: header_read, op: osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.132917 7f9d5a16f700  5 -- op tracker -- , seq: 748554, time: 2014-06-12 14:48:17.130930, event: throttled, op: osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.132932 7f9d5a16f700  5 -- op tracker -- , seq: 748554, time: 2014-06-12 14:48:17.132853, event: all_read, op: osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.132939 7f9d5a16f700  5 -- op tracker -- , seq: 748554, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.132944 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:17.132950 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:17.132958 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903d00 prio 127 cost 1032999 latency 0.002030 osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:17.133026 7f9d64bda700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:17.133050 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a903d00 prio 127 cost 1032999 latency 0.002122 osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2520 (0'0,173'2520] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2519 active]
2014-06-12 14:48:17.133093 7f9d64bda700  5 -- op tracker -- , seq: 748554, time: 2014-06-12 14:48:17.133086, event: reached_pg, op: osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.133110 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2520 (0'0,173'2520] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2519 active] handle_message: 0x2a903d00
2014-06-12 14:48:17.133126 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2520 (0'0,173'2520] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2519 active] sub_op_modify trans 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 v 173'2521 (transaction) 174
2014-06-12 14:48:17.133149 7f9d64bda700  5 -- op tracker -- , seq: 748554, time: 2014-06-12 14:48:17.133144, event: started, op: osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.133175 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2520 (0'0,173'2520] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2519 active] append_log log((0'0,173'2520], crt=0'0) [173'2521 (173'2520) modify   2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 by client.4300.0:140087 2014-06-12 08:45:05.376684]
2014-06-12 14:48:17.133208 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2521 (0'0,173'2521] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2519 active] add_log_entry 173'2521 (173'2520) modify   2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 by client.4300.0:140087 2014-06-12 08:45:05.376684
2014-06-12 14:48:17.133247 7f9d64bda700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2521 (0'0,173'2521] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2519 active] append_log  adding 1 keys
2014-06-12 14:48:17.133294 7f9d64bda700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2521, trimmed: 
2014-06-12 14:48:17.133331 7f9d64bda700  5 -- op tracker -- , seq: 748554, time: 2014-06-12 14:48:17.133324, event: started, op: osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.133348 7f9d64bda700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:48:17.133354 7f9d64bda700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:17.133356 7f9d64bda700  5 queue_op 0x165afb00 seq 2367 osr(3.168 0x276ce40) 1034485 bytes   (queue has 1 ops and 1034485 bytes)
2014-06-12 14:48:17.133370 7f9d8016d700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:48:17.133364 7f9d64bda700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2521 (0'0,173'2521] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2519 active] do_sub_op osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:17.133385 7f9d64bda700 10 osd.2 173 dequeue_op 0x2a903d00 finish
2014-06-12 14:48:17.133386 7f9d8016d700  5 _do_op 0x165afb00 seq 2367 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:48:17.133390 7f9d64bda700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:17.133392 7f9d8016d700 10 _do_transaction on 0x29eb30b0
2014-06-12 14:48:17.133393 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:17.133401 7f9d8016d700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:17.133540 7f9d8016d700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:17.133566 7f9d8016d700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:17.133639 7f9d8016d700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:17.133666 7f9d8016d700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:17.133672 7f9d8016d700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:48:17.133685 7f9d8016d700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:17.133810 7f9d8016d700 15 _write 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 2867200~1032192
2014-06-12 14:48:17.133925 7f9d8016d700 10 lookup_strip_header done  cid 3.168_head oid 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:17.134066 7f9d8016d700 10 file_to_extents done 
2014-06-12 14:48:17.137783 7f9d8016d700 10 _generic_write 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 2867200~1032192 = 0
2014-06-12 14:48:17.137867 7f9d8016d700 15 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:17.137883 7f9d8016d700 10 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 = 0
2014-06-12 14:48:17.137889 7f9d8016d700 15 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:17.137896 7f9d8016d700 10 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 = 0
2014-06-12 14:48:17.138181 7f9d8016d700 20 set_header setting 16545 cid 3.168_head oid 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 parent seq 0
2014-06-12 14:48:17.138298 7f9d8016d700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:17.138316 7f9d8016d700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:48:17.138325 7f9d8016d700  5 submit_transaction r = 0
2014-06-12 14:48:17.142828 7f9d8016d700 10 _do_op 0x165afb00 seq 2367 r = 0, finisher 0x267411a0 0
2014-06-12 14:48:17.142847 7f9d8016d700 10 _finish_op 0x165afb00 seq 2367 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:48:17.142854 7f9d8016d700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:48:17.142858 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:17.142874 7f9d7715b700  5 -- op tracker -- , seq: 748554, time: 2014-06-12 14:48:17.142856, event: commit_sent, op: osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.142889 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2521 (0'0,173'2521] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2519 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:17.142920 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:17.142977 7f9d7795c700  5 -- op tracker -- , seq: 748554, time: 2014-06-12 14:48:17.142969, event: sub_op_applied, op: osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.142990 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2521 (0'0,173'2521] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2520 active] sub_op_modify_applied on 0x29eb2fc0 op osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:17.143010 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2521 (0'0,173'2521] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2520 active] op_applied on primary on version 173'2521
2014-06-12 14:48:17.143021 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2521 (0'0,173'2521] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2520 active] op_applied on replica on version 173'2521
2014-06-12 14:48:17.143041 7f9d7795c700  5 -- op tracker -- , seq: 748554, time: 2014-06-12 14:48:17.143038, event: done, op: osd_sub_op(client.4300.0:140087 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2521 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.168643 7f9d5a16f700  5 -- op tracker -- , seq: 748555, time: 2014-06-12 14:48:17.168081, event: header_read, op: osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.168654 7f9d5a16f700  5 -- op tracker -- , seq: 748555, time: 2014-06-12 14:48:17.168083, event: throttled, op: osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.168677 7f9d5a16f700  5 -- op tracker -- , seq: 748555, time: 2014-06-12 14:48:17.168603, event: all_read, op: osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.168682 7f9d5a16f700  5 -- op tracker -- , seq: 748555, time: 0.000000, event: dispatched, op: osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.168686 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:48:17.168691 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:48:17.168697 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a902300 prio 127 cost 295719 latency 0.000616 osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:17.168762 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:48:17.168773 7f9d653db700 10 osd.2 173 dequeue_op 0x2a902300 prio 127 cost 295719 latency 0.000692 osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.168( v 173'2521 (0'0,173'2521] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2520 active]
2014-06-12 14:48:17.168793 7f9d653db700  5 -- op tracker -- , seq: 748555, time: 2014-06-12 14:48:17.168790, event: reached_pg, op: osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.168802 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2521 (0'0,173'2521] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2520 active] handle_message: 0x2a902300
2014-06-12 14:48:17.168810 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2521 (0'0,173'2521] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2520 active] sub_op_modify trans 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 v 173'2522 (transaction) 174
2014-06-12 14:48:17.168821 7f9d653db700  5 -- op tracker -- , seq: 748555, time: 2014-06-12 14:48:17.168819, event: started, op: osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.168834 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2521 (0'0,173'2521] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2520 active] append_log log((0'0,173'2521], crt=0'0) [173'2522 (173'2521) modify   2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 by client.4300.0:140088 2014-06-12 08:45:11.284206]
2014-06-12 14:48:17.168852 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2522 (0'0,173'2522] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2520 active] add_log_entry 173'2522 (173'2521) modify   2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 by client.4300.0:140088 2014-06-12 08:45:11.284206
2014-06-12 14:48:17.168866 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2522 (0'0,173'2522] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2520 active] append_log  adding 1 keys
2014-06-12 14:48:17.168888 7f9d653db700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 173'2522, trimmed: 
2014-06-12 14:48:17.168906 7f9d653db700  5 -- op tracker -- , seq: 748555, time: 2014-06-12 14:48:17.168903, event: started, op: osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.168914 7f9d653db700  5 queue_transactions existing osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:48:17.168917 7f9d653db700  5 queue_transactions (trailing journal)  
2014-06-12 14:48:17.168924 7f9d653db700  5 queue_op 0x58d1da0 seq 2368 osr(3.168 0x276ce40) 297205 bytes   (queue has 1 ops and 297205 bytes)
2014-06-12 14:48:17.168928 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.168( v 173'2522 (0'0,173'2522] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2520 active] do_sub_op osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:17.168937 7f9d653db700 10 osd.2 173 dequeue_op 0x2a902300 finish
2014-06-12 14:48:17.168939 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:48:17.168948 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:17.168962 7f9d79960700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x5474fc0 (1 active)
2014-06-12 14:48:17.168977 7f9d79960700  5 _do_op 0x58d1da0 seq 2368 osr(3.168 0x276ce40)/0x276ce40 start
2014-06-12 14:48:17.168981 7f9d79960700 10 _do_transaction on 0x8dce630
2014-06-12 14:48:17.168986 7f9d79960700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:17.169090 7f9d79960700 10 lookup_strip_header done  cid meta oid 1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:17.169111 7f9d79960700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:48:17.169146 7f9d79960700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:48:17.169164 7f9d79960700 15 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:17.169169 7f9d79960700 10 _omap_rmkeys meta/1fe2af2a/pglog_3.168/0//-1 r = 0
2014-06-12 14:48:17.169172 7f9d79960700 15 _omap_setkeys meta/1fe2af2a/pglog_3.168/0//-1
2014-06-12 14:48:17.169260 7f9d79960700 15 _write 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 3899392~294912
2014-06-12 14:48:17.169362 7f9d79960700 10 lookup_strip_header done  cid 3.168_head oid 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:17.169473 7f9d79960700 10 file_to_extents done 
2014-06-12 14:48:17.170196 7f9d79960700 10 _generic_write 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 3899392~294912 = 0
2014-06-12 14:48:17.170220 7f9d79960700 15 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:17.170229 7f9d79960700 10 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 = 0
2014-06-12 14:48:17.170232 7f9d79960700 15 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3
2014-06-12 14:48:17.170237 7f9d79960700 10 _setattrs 3.168_head/2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 = 0
2014-06-12 14:48:17.170342 7f9d79960700 20 set_header setting 16545 cid 3.168_head oid 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 parent seq 0
2014-06-12 14:48:17.170371 7f9d79960700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:48:17.170380 7f9d79960700 20 set_header setting 2298 cid meta oid 1fe2af2a/pglog_3.168/0//-1 parent seq 0
2014-06-12 14:48:17.170385 7f9d79960700  5 submit_transaction r = 0
2014-06-12 14:48:17.171244 7f9d79960700 10 _do_op 0x58d1da0 seq 2368 r = 0, finisher 0x698df60 0
2014-06-12 14:48:17.171253 7f9d79960700 10 _finish_op 0x58d1da0 seq 2368 osr(3.168 0x276ce40)/0x276ce40
2014-06-12 14:48:17.171257 7f9d79960700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x5474fc0 (0 active)
2014-06-12 14:48:17.171260 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:17.171303 7f9d7715b700  5 -- op tracker -- , seq: 748555, time: 2014-06-12 14:48:17.171296, event: commit_sent, op: osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.171322 7f9d7715b700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2522 (0'0,173'2522] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2520 active] sub_op_modify_commit on op osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[]) v10, sending commit to osd.24
2014-06-12 14:48:17.171345 7f9d7715b700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:17.171413 7f9d7795c700  5 -- op tracker -- , seq: 748555, time: 2014-06-12 14:48:17.171407, event: sub_op_applied, op: osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.171424 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2522 (0'0,173'2522] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2521 active] sub_op_modify_applied on 0x8dce540 op osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:48:17.171436 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2522 (0'0,173'2522] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2521 active] op_applied on primary on version 173'2522
2014-06-12 14:48:17.171442 7f9d7795c700 10 osd.2 pg_epoch: 173 pg[3.168( v 173'2522 (0'0,173'2522] local-les=147 n=296 ec=23 les/c 147/147 146/146/146) [24,2] r=1 lpr=146 pi=23-145/12 luod=0'0 crt=0'0 lcod 173'2521 active] op_applied on replica on version 173'2522
2014-06-12 14:48:17.171457 7f9d7795c700  5 -- op tracker -- , seq: 748555, time: 2014-06-12 14:48:17.171454, event: done, op: osd_sub_op(client.4300.0:140088 3.168 2f23b968/rbd_data.10a36b8b4567.0000000000000ec8/head//3 [] v 173'2522 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:48:17.188198 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:17.188224 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:17.273780 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:17.289558 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:17.328265 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:17.332460 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:17.365163 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:17.381502 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:17.491560 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:17.491586 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:48:17.491590 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:48:17.491595 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:48:17.491607 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:17.491611 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:17.491615 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:17.491646 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:17.630699 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:17.631049 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:17.859702 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:17.860091 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:17.955741 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:17.956000 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:17.966194 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:17.968033 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:18.050655 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:18.055990 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:18.117422 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:18.117832 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:18.158620 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:18.218848 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:18.227349 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:18.308167 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:18.317626 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:18.341567 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:18.341729 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:18.357377 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:18.362261 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:18.373196 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:18.373367 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:18.373568 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:18.439303 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:18.439403 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:18.491707 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:18.491734 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:18.491739 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:18.491741 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:18.491780 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:18.496310 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:18.505073 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:18.514298 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:18.523560 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:18.642745 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:18.652947 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:18.684727 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:18.692944 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:18.742041 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:18.759900 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:18.787324 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:18.793120 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:18.846129 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:18.856275 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:18.919209 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:18.927376 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:18.952859 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:18.957268 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:19.026361 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:19.026557 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:19.112972 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:19.124004 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:19.133492 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:19.142920 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:19.169036 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:19.171345 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:19.235210 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:19.235455 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:19.273863 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:19.289639 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:19.328354 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:19.332546 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:19.365266 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:19.381588 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:19.491832 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:19.491894 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 4.22 >= max 0.5 = no, load too high
2014-06-12 14:48:19.491910 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:19.491908 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:19.491914 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92747 < max (604800 seconds)
2014-06-12 14:48:19.491949 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:19.491952 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:19.491954 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:19.843434 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:19.843456 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:19.844182 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:19.844262 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:19.844312 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:19.844348 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:19.844435 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:19.844478 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:19.844507 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:19.844532 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:19.844559 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:19.844587 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:19.844624 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:19.844651 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:19.844678 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:19.844706 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:19.844733 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:19.844760 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:19.844787 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:19.844811 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:19.844831 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:19.844857 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:19.844881 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:19.844906 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:19.966280 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:19.968127 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:19.973979 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:19.974043 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:20.042759 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:20.042854 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:20.050746 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:20.056087 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:20.158703 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:20.218945 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:20.227430 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:20.308253 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:20.317710 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:20.357480 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:20.362343 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:20.373290 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:20.492042 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:20.492110 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:20.492112 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 3.96 >= max 0.5 = no, load too high
2014-06-12 14:48:20.492131 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:20.492137 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92748 < max (604800 seconds)
2014-06-12 14:48:20.492173 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:20.492180 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:20.492183 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:20.496403 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:20.505153 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:20.514379 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:20.523655 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:20.642819 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:20.653018 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:20.684813 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:20.693024 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:20.742113 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:20.759970 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:20.787409 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:20.793217 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:20.846200 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:20.856371 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:20.919309 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:20.927458 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:20.952942 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:20.957349 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:21.113075 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:21.124101 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:21.133582 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:21.143000 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:21.143491 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:21.143642 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:21.169120 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:21.171441 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:21.273951 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:21.274245 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:21.274581 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:21.289721 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:21.328436 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:21.332635 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:21.365363 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:21.381669 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:21.456218 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:21.456272 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:21.492259 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:21.492284 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:21.492288 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:21.492291 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:21.492329 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:21.618191 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:21.618351 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:21.731366 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:21.731486 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:21.763469 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:21.763565 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:21.888977 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:21.889107 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:21.926880 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:21.927197 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:21.940100 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:21.940164 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:21.966371 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:21.968209 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:22.050823 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:22.056184 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:22.158796 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:22.219026 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:22.227511 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:22.308339 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:22.317792 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:22.357584 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:22.362424 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:22.373372 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:22.492364 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:22.492385 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:48:22.492389 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:48:22.492394 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:48:22.492403 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:22.492407 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:22.492409 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:22.492443 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:22.496502 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:22.505237 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:22.514461 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:22.523750 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:22.642899 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:22.653107 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:22.684898 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:22.693107 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:22.742194 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:22.760060 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:22.787508 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:22.793312 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:22.846278 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:22.856467 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:22.919409 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:22.927539 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:22.953040 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:22.957433 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:23.113162 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:23.124197 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:23.133685 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:23.143100 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:23.160476 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:23.160567 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:23.169219 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:23.171537 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:23.274040 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:23.289807 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:23.328520 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:23.332732 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:23.365462 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:23.381756 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:23.464138 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:23.464473 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:23.474798 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:23.474876 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:23.492485 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:23.492541 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 3.96 >= max 0.5 = no, load too high
2014-06-12 14:48:23.492555 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:23.492558 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:23.492559 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92751 < max (604800 seconds)
2014-06-12 14:48:23.492579 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:23.492582 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:23.492583 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:23.966458 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:23.968306 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:24.050928 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:24.056265 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:24.158893 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:24.189866 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:24.189932 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:24.219108 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:24.227594 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:24.308427 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:24.317874 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:24.357688 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:24.362506 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:24.373453 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:24.492647 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:24.492668 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:24.492671 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:24.492673 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:24.492700 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:24.496573 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:24.505307 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:24.514532 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:24.523821 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:24.544296 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:24.544319 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:24.544897 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:24.544927 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:24.544948 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:24.544961 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:24.544985 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:24.545043 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:24.545082 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:24.545157 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:24.545186 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:24.545219 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:24.545263 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:24.545321 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:24.545372 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:24.545429 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:24.545476 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:24.545523 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:24.545569 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:24.545635 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:24.545695 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:24.545718 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:24.545752 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:24.545763 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:24.564372 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:24.564415 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:24.632104 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:24.632158 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:24.642973 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:24.643555 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:24.643604 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:24.653203 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:24.684987 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:24.693178 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:24.742281 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:24.760142 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:24.774823 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:24.775376 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:24.787579 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:24.793395 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:24.827675 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:24.827764 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:24.846363 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:24.856550 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:24.919509 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:24.927619 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:24.953138 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:24.957513 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:25.113246 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:25.124294 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:25.133759 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:25.136010 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:25.136147 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:25.143195 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:25.169320 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:25.171631 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:25.274126 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:25.289889 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:25.328601 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:25.332807 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:25.365561 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:25.381834 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:25.492747 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:25.492776 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:25.492784 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:25.492788 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:25.492815 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:25.966545 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:25.968402 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:26.051027 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:26.056345 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:26.158993 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:26.219193 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:26.227677 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:26.308520 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:26.317957 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:26.357788 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:26.362588 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:26.373536 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:26.490225 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:26.490416 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:26.492878 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:26.492903 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:26.492908 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:26.492911 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:26.492930 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:26.496653 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:26.505383 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:26.514607 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:26.523912 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:26.640796 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:26.640850 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:26.643041 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:26.653284 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:26.661005 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:26.661116 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:26.685084 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:26.693241 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:26.742358 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:26.756808 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:26.756843 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:26.760225 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:26.787671 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:26.793476 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:26.836083 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:26.836264 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:26.846453 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:26.856625 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:26.918965 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:26.919417 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:26.919587 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:26.927690 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:26.953222 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:26.957594 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:27.113335 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:27.124389 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:27.133836 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:27.143266 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:27.169422 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:27.171725 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:27.274215 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:27.289972 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:27.328687 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:27.332903 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:27.365660 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:27.381933 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:27.492985 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:27.493007 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:48:27.493011 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:48:27.493015 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:48:27.493025 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:27.493029 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:27.493031 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:27.493064 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:27.533130 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:27.533199 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:27.966631 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:27.968497 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:28.051121 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:28.056441 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:28.065268 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:28.065471 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:28.159099 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:28.219278 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:28.227758 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:28.308606 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:28.318055 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:28.357870 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:28.362688 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:28.373625 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:28.493122 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:28.493149 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:28.493154 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:28.493156 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:28.493195 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:28.496748 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:28.505461 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:28.514681 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:28.524007 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:28.619571 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:28.619666 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:28.643120 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:28.653366 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:28.685178 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:28.693313 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:28.742441 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:28.744438 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:28.744797 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:28.760308 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:28.775666 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:28.775744 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:28.787755 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:28.793558 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:28.846542 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:28.856716 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:28.919685 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:28.927764 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:28.953306 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:28.957678 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:29.113427 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:29.124482 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:29.133936 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:29.143361 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:29.169518 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:29.171821 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:29.245056 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:29.245381 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:29.274298 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:29.290055 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:29.328777 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:29.332990 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:29.365759 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:29.382014 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:29.493235 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:29.493265 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:29.493273 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:29.493277 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:29.493304 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:29.720197 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:29.720347 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:29.845012 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:29.845033 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:29.845808 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:29.845891 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:29.845926 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:29.845983 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:29.846037 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:29.846066 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:29.846091 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:29.846122 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:29.846150 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:29.846177 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:29.846207 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:29.846234 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:29.846261 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:29.846284 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:29.846312 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:29.846336 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:29.846360 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:29.846387 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:29.846411 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:29.846435 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:29.846464 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:29.846489 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:29.966720 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:29.968591 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:30.051215 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:30.056522 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:30.128245 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:30.128389 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:30.159194 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:30.219362 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:30.227852 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:30.257322 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:30.257385 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:30.308693 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:30.318139 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:30.337172 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:30.337225 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:30.357941 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:30.362759 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:30.373705 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:30.493361 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:30.493433 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:30.493435 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 3.5 >= max 0.5 = no, load too high
2014-06-12 14:48:30.493452 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:30.493456 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92758 < max (604800 seconds)
2014-06-12 14:48:30.493477 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:30.493480 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:30.493482 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:30.496863 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:30.505546 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:30.514764 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:30.524104 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:30.643206 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:30.653448 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:30.675689 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:30.675956 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:30.685276 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:30.693387 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:30.742532 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:30.760388 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:30.787853 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:30.793639 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:30.846624 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:30.856808 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:30.919783 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:30.927844 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:30.953395 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:30.957759 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:31.113511 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:31.124577 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:31.134036 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:31.143458 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:31.169601 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:31.171901 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:31.191107 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:31.191229 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:31.274398 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:31.290139 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:31.328876 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:31.333073 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:31.341394 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:31.341478 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:31.361831 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:31.361944 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:31.365830 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:31.382093 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:31.493556 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:31.493583 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:31.493588 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:31.493590 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:31.493630 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:31.676573 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:31.676657 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:31.957928 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:31.958093 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:31.966811 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:31.968686 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:32.051292 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:32.056618 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:32.145860 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:32.146271 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:32.159295 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:32.166118 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:32.166415 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:32.219449 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:32.227949 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:32.308779 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:32.318221 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:32.358035 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:32.362849 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:32.373801 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:32.376375 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:32.376529 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:32.493681 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:32.493702 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:48:32.493706 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:48:32.493710 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:48:32.493720 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:32.493724 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:32.493725 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:32.493760 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:32.496958 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:32.505637 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:32.514846 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:32.524184 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:32.643295 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:32.653535 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:32.666569 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:32.666689 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:32.685376 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:32.693472 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:32.742607 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:32.745899 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:32.745918 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:32.746533 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:32.746574 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:32.746624 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:32.746686 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:32.746714 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:32.746736 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:32.746805 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:32.746838 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:32.746872 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:32.746894 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:32.746914 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:32.746936 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:32.746971 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:32.746993 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:32.747014 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:32.747041 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:32.747062 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:32.747084 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:32.747110 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:32.747135 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:32.747157 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:32.747193 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:32.760471 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:32.787952 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:32.793739 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:32.834069 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:32.834212 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:32.846718 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:32.856905 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:32.919881 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:32.927925 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:32.953489 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:32.957831 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:33.113627 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:33.124658 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:33.134120 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:33.143554 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:33.169707 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:33.172000 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:33.220814 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:33.221057 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:33.274488 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:33.290228 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:33.328972 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:33.333157 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:33.365910 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:33.382171 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:33.493801 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:33.493831 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:33.493839 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:33.493843 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:33.493870 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:33.966900 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:33.968783 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:34.051397 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:34.056700 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:34.159395 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:34.219521 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:34.228020 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:34.308865 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:34.318302 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:34.358109 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:34.362930 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:34.373882 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:34.437956 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:34.438262 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:34.446622 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:34.446647 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:34.447411 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:34.447469 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:34.447501 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:34.447527 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:34.447551 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:34.447578 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:34.447601 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:34.447640 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:34.447672 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:34.447698 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:34.447722 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:34.447743 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:34.447768 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:34.447793 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:34.447820 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:34.447847 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:34.447871 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:34.447894 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:34.447928 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:34.447953 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:34.447977 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:34.448088 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:34.493922 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:34.494002 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:34.493999 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 3.5 >= max 0.5 = no, load too high
2014-06-12 14:48:34.494017 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:34.494020 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92762 < max (604800 seconds)
2014-06-12 14:48:34.494051 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:34.494055 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:34.494056 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:34.497058 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:34.505733 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:34.514929 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:34.524281 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:34.643381 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:34.653628 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:34.685475 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:34.691491 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:34.691561 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:34.693545 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:34.742689 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:34.760550 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:34.788033 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:34.793818 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:34.846817 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:34.857000 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:34.919979 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:34.928004 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:34.953573 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:34.957905 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:35.113711 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:35.124754 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:35.134220 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:35.143628 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:35.169783 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:35.172093 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:35.177371 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:35.177512 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:35.274574 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:35.290328 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:35.329071 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:35.333241 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:35.365993 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:35.382252 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:35.429185 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:35.429600 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:35.494132 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:35.494185 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:35.494191 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 3.3 >= max 0.5 = no, load too high
2014-06-12 14:48:35.494207 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:35.494211 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92763 < max (604800 seconds)
2014-06-12 14:48:35.494231 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:35.494234 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:35.494236 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:35.567434 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:35.567820 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:35.877072 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:35.877231 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:35.966988 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:35.968864 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:36.042113 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:36.042168 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:36.051495 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:36.056794 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:36.062415 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:36.062513 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:36.159499 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:36.219592 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:36.228109 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:36.308964 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:36.318383 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:36.358197 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:36.363026 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:36.373976 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:36.494311 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:36.494340 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:36.494347 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:36.494351 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:36.494378 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:36.497157 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:36.505818 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:36.515013 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:36.524375 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:36.643471 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:36.653720 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:36.685576 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:36.693618 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:36.742787 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:36.760629 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:36.788130 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:36.793901 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:36.846901 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:36.857072 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:36.920078 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:36.928088 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:36.953656 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:36.958003 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:37.113797 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:37.124852 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:37.134297 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:37.143723 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:37.169865 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:37.172187 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:37.274665 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:37.290408 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:37.321505 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:37.321735 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:37.329155 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:37.333339 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:37.338338 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:37.338413 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:37.366085 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:37.382335 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:37.494429 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:37.494451 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:48:37.494457 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:48:37.494464 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:48:37.494514 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:37.494513 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 3.3 >= max 0.5 = no, load too high
2014-06-12 14:48:37.494530 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:37.494536 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92765 < max (604800 seconds)
2014-06-12 14:48:37.494562 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:37.494567 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:37.494571 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:37.858802 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:37.858860 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:37.967077 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:37.968944 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:38.046281 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:38.046363 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:38.051588 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:38.056892 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:38.134342 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:38.134506 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:38.159599 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:38.219671 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:38.228204 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:38.309050 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:38.318471 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:38.342907 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:38.343009 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:38.358296 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:38.363107 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:38.374073 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:38.422486 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:38.422591 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:38.494650 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:38.494677 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:38.494682 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:38.494684 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:38.494704 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:38.497255 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:38.505902 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:38.515093 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:38.524460 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:38.547525 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:38.547545 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:38.548328 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:38.548423 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:38.548470 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:38.548541 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:38.548637 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:38.548675 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:38.548701 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:38.548730 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:38.548768 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:38.548793 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:38.548820 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:38.548847 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:38.548875 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:38.548901 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:38.548927 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:38.548954 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:38.548980 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:38.549007 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:38.549033 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:38.549060 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:38.549085 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:38.549111 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:38.643561 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:38.653793 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:38.685663 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:38.693686 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:38.742888 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:38.760715 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:38.788213 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:38.793982 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:38.847002 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:38.857164 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:38.920159 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:38.928168 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:38.953749 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:38.958101 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:39.113883 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:39.124951 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:39.134382 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:39.143821 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:39.169940 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:39.172283 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:39.274762 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:39.278376 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:39.278477 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:39.290490 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:39.329253 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:39.333437 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:39.366169 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:39.382416 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:39.494778 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:39.494806 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:39.494811 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:39.494813 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:39.494852 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:39.746844 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:39.747287 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:39.967170 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:39.969040 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:39.977825 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:39.978187 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:39.991833 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:39.991924 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:40.051691 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:40.056970 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:40.123787 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:40.123863 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:40.159697 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:40.219770 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:40.228299 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:40.239247 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:40.239363 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:40.309141 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:40.318552 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:40.358381 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:40.363205 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:40.374171 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:40.494904 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:40.494931 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:40.494936 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:40.494939 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:40.494978 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:40.497339 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:40.505981 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:40.515176 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:40.524555 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:40.643644 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:40.653868 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:40.685768 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:40.693758 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:40.742987 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:40.760809 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:40.763129 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:40.763246 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:40.788312 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:40.794064 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:40.847101 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:40.848381 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:40.848400 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:40.849142 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:40.849207 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:40.849259 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:40.849305 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:40.849354 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:40.849404 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:40.849463 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:40.849511 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:40.849557 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:40.849602 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:40.849676 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:40.849721 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:40.849769 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:40.849814 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:40.849858 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:40.849904 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:40.849950 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:40.849996 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:40.850046 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:40.850091 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:40.850137 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:40.850178 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:40.857245 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:40.920257 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:40.928247 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:40.953833 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:40.958182 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:41.113971 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:41.125045 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:41.134472 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:41.143919 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:41.170024 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:41.172377 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:41.274845 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:41.290571 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:41.329337 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:41.329908 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:41.329967 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:41.333533 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:41.366269 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:41.382494 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:41.467855 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:41.468238 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:41.495031 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:41.495098 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:41.495094 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 3.12 >= max 0.5 = no, load too high
2014-06-12 14:48:41.495127 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:41.495133 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92769 < max (604800 seconds)
2014-06-12 14:48:41.495159 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:41.495164 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:41.495168 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:41.830759 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:41.830815 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:41.967259 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:41.969118 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:42.047514 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:42.047608 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:42.051770 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:42.057052 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:42.159798 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:42.179257 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:42.179405 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:42.219869 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:42.228383 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:42.309230 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:42.318639 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:42.331422 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:42.331521 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:42.358465 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:42.363302 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:42.374252 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:42.495243 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:42.495265 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:48:42.495269 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:48:42.495273 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:48:42.495282 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:42.495286 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:42.495288 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:42.495323 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:42.497423 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:42.506065 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:42.515257 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:42.524640 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:42.559354 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:42.559790 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:42.643724 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:42.653942 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:42.685872 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:42.693825 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:42.743085 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:42.760880 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:42.788395 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:42.794143 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:42.847183 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:42.857327 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:42.878386 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:42.878540 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:42.892381 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:42.892859 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:42.920354 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:42.928328 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:42.953920 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:42.958262 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:43.024068 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:43.024474 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:43.063254 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:43.063649 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:43.114053 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:43.125141 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:43.134554 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:43.144017 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:43.168302 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:43.168519 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:43.170114 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:43.172473 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:43.274946 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:43.290661 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:43.329418 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:43.333624 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:43.366353 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:43.382577 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:43.495381 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:43.495440 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 3.12 >= max 0.5 = no, load too high
2014-06-12 14:48:43.495455 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:43.495453 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:43.495458 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92771 < max (604800 seconds)
2014-06-12 14:48:43.495478 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:43.495481 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:43.495483 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:43.643670 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:43.643764 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:43.967350 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:43.969212 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:44.035218 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:44.035325 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:44.051867 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:44.057150 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:44.144547 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:44.144608 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:44.159885 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:44.219953 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:44.228464 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:44.309309 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:44.318720 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:44.349249 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:44.349270 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:44.350020 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:44.350093 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:44.350150 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:44.350197 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:44.350295 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:44.350321 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:44.350353 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:44.350407 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:44.350455 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:44.350490 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:44.350517 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:44.350539 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:44.350629 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:44.350707 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:44.350749 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:44.350779 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:44.350812 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:44.350836 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:44.350861 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:44.350885 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:44.350905 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:44.350925 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:44.358551 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:44.363398 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:44.374333 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:44.495573 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:44.495626 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 3.12 >= max 0.5 = no, load too high
2014-06-12 14:48:44.495640 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:44.495645 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:44.495644 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92772 < max (604800 seconds)
2014-06-12 14:48:44.495664 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:44.495667 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:44.495668 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:44.497506 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:44.506160 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:44.515337 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:44.524735 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:44.536390 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:44.536493 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:44.632072 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:44.632523 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:44.643810 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:44.654038 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:44.685955 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:44.693898 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:44.743183 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:44.760959 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:44.788477 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:44.794223 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:44.847284 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:44.857409 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:44.920451 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:44.928408 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:44.954004 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:44.958359 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:45.036599 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:45.036665 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:45.080097 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:45.080206 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:45.114154 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:45.125236 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:45.134647 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:45.144114 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:45.170210 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:45.172570 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:45.275045 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:45.290757 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:45.329502 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:45.333696 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:45.366425 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:45.382646 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:45.468726 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:45.468768 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:45.495760 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:45.495787 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:45.495792 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:45.495795 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:45.495833 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:45.540027 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:45.540168 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:45.779102 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:45.779230 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:45.967436 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:45.969292 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:46.051971 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:46.057236 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:46.159983 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:46.220036 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:46.228545 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:46.309395 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:46.318819 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:46.358642 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:46.363495 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:46.374417 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:46.392853 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:46.392907 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:46.495884 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:46.495910 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:46.495915 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:46.495917 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:46.495956 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:46.497588 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:46.506237 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:46.515418 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:46.524831 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:46.643884 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:46.654120 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:46.686039 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:46.693971 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:46.743284 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:46.748168 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:46.748291 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:46.761041 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:46.788576 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:46.794323 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:46.847369 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:46.857492 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:46.920551 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:46.928490 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:46.954088 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:46.958440 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:47.114240 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:47.124832 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:47.125039 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:47.125316 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:47.134745 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:47.144209 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:47.170295 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:47.172663 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:47.275146 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:47.290853 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:47.329589 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:47.333773 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:47.337560 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:47.337663 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:47.366519 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:47.382736 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:47.496008 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:47.496028 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:48:47.496032 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:48:47.496036 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:48:47.496046 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:47.496049 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:47.496051 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:47.496088 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:47.850083 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:47.850107 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:47.850873 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:47.850920 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:47.850957 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:47.850991 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:47.851051 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:47.851080 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:47.851122 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:47.851148 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:47.851224 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:47.851296 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:47.851340 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:47.851368 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:47.851394 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:47.851418 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:47.851445 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:47.851469 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:47.851490 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:47.851514 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:47.851538 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:47.851559 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:47.851580 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:47.851603 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:47.967523 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:47.969380 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:48.052072 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:48.057319 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:48.160080 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:48.220131 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:48.228637 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:48.309479 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:48.318900 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:48.358742 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:48.363654 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:48.363678 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:48.364182 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:48.374498 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:48.460371 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:48.460454 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:48.496142 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:48.496198 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 2.95 >= max 0.5 = no, load too high
2014-06-12 14:48:48.496213 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:48.496216 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:48.496216 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92776 < max (604800 seconds)
2014-06-12 14:48:48.496236 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:48.496239 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:48.496240 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:48.497670 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:48.506323 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:48.515498 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:48.524926 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:48.643967 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:48.654218 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:48.686128 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:48.694038 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:48.743382 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:48.761126 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:48.788672 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:48.794408 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:48.847466 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:48.857571 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:48.864289 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:48.864745 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:48.920633 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:48.928573 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:48.954189 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:48.958536 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:49.114321 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:49.125410 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:49.134844 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:49.144303 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:49.170378 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:49.172743 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:49.275235 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:49.290934 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:49.329675 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:49.332873 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:49.333095 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:49.333856 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:49.366591 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:49.382831 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:49.496331 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:49.496401 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:49.496397 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 2.95 >= max 0.5 = no, load too high
2014-06-12 14:48:49.496430 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:49.496437 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92777 < max (604800 seconds)
2014-06-12 14:48:49.496464 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:49.496469 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:49.496472 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:49.967608 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:49.969462 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:50.045429 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:50.045509 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:50.052150 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:50.057400 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:50.160179 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:50.169291 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:50.169646 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:50.220211 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:50.228734 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:50.248733 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:50.249089 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:50.309573 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:50.318998 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:50.358845 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:50.363748 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:50.374587 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:50.381080 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:50.381125 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:50.493160 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:50.493457 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:50.496537 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:50.496549 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:50.496551 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:50.496553 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:50.496593 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:50.497741 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:50.506395 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:50.515579 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:50.525009 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:50.644046 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:50.654303 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:50.686216 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:50.694102 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:50.743485 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:50.761207 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:50.788757 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:50.794489 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:50.840784 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:50.841091 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:50.847550 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:50.857659 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:50.920735 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:50.928654 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:50.954288 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:50.958625 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:51.033900 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:51.034010 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:51.079932 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:51.080008 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:51.114411 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:51.125490 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:51.134927 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:51.144399 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:51.170463 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:51.172838 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:51.275336 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:51.291016 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:51.329749 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:51.333930 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:51.366686 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:51.382926 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:51.496642 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:51.496672 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:51.496680 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:51.496685 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:51.496712 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:51.534170 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:51.534599 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:51.960644 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:51.960907 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:51.967693 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:51.969542 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:52.052224 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:52.057480 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:52.160262 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:52.220295 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:52.228831 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:52.309659 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:52.319094 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:52.358951 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:52.363844 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:52.374684 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:52.496761 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:52.496784 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:48:52.496788 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:48:52.496792 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:48:52.496802 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:52.496806 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:52.496808 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:52.496843 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:52.497820 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:52.506472 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:52.515660 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:52.525105 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:52.638003 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:52.638476 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:52.644134 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:52.654395 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:52.686299 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:52.694166 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:52.743588 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:52.761287 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:52.788856 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:52.794576 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:52.847650 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:52.857745 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:52.920834 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:52.928733 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:52.954373 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:52.958723 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:53.025743 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:53.025830 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:53.114497 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:53.125587 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:53.135026 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:53.144496 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:53.150952 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:53.150972 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:53.151673 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:53.151740 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:53.151830 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:53.151890 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:53.151958 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:53.151997 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:53.152024 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:53.152048 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:53.152068 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:53.152174 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:53.152214 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:53.152235 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:53.152256 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:53.152278 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:53.152304 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:53.152330 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:53.152356 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:53.152381 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:53.152402 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:53.152422 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:53.152446 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:53.153316 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:53.170562 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:53.172932 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:53.275437 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:53.291111 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:53.329824 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:53.334031 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:53.366785 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:53.383021 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:53.393809 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:53.393961 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:53.496883 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:53.496946 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 2.79 >= max 0.5 = no, load too high
2014-06-12 14:48:53.496956 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:53.496962 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:53.496967 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92781 < max (604800 seconds)
2014-06-12 14:48:53.496996 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:53.496998 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:53.497000 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:53.669995 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:53.670325 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:53.834833 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:53.835207 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:53.967777 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:53.969623 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:54.052301 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:54.057551 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:54.146316 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:54.146404 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:54.160335 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:54.220381 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:54.228927 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:54.309750 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:54.319191 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:54.335880 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:54.339996 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:54.359049 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:54.363940 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:54.374755 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:54.497065 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:54.497086 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:54.497089 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:54.497092 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:54.497119 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:54.497891 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:54.506557 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:54.515732 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:54.525185 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:54.644212 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:54.654467 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:54.686387 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:54.694238 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:54.743688 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:54.761371 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:54.765426 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:54.765517 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:54.788953 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:54.794672 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:54.847736 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:54.857832 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:54.920933 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:54.928832 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:54.954458 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:54.958822 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:55.114583 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:55.125655 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:55.135125 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:55.144581 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:55.170660 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:55.173026 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:55.275535 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:55.291208 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:55.329907 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:55.334110 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:55.366885 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:55.383115 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:55.497165 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:55.497191 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:55.497196 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:55.497198 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:55.497234 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:55.549330 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:55.549848 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:55.681858 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:55.681924 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:48:55.926647 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:55.926707 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:55.967862 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:55.969696 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:56.052404 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:56.057633 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:56.138872 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:56.138930 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:56.160427 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:56.220467 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:56.229021 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:56.294481 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:56.294956 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:56.309838 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:56.319286 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:56.359139 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:56.364020 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:56.374823 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:56.379985 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:56.380380 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:56.447153 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:56.447272 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:56.497276 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:56.497330 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 2.65 >= max 0.5 = no, load too high
2014-06-12 14:48:56.497344 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:48:56.497347 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:56.497347 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92784 < max (604800 seconds)
2014-06-12 14:48:56.497367 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:48:56.497370 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:56.497372 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:56.497957 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:56.506651 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:56.515811 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:56.525266 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:56.644305 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:56.651781 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:56.651800 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:56.652557 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:56.652622 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:56.652704 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:56.652759 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:56.652781 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:56.652801 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:56.652822 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:56.652843 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:56.652863 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:56.652913 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:56.652939 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:56.652962 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:56.653021 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:56.653061 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:56.653095 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:56.653115 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:56.653136 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:56.653160 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:56.653187 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:56.653210 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:56.653236 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:56.653262 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:56.654536 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:56.686477 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:56.694310 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:56.741169 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:56.741226 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:56.743787 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:56.761453 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:56.789039 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:56.794768 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:56.847837 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:56.857905 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:56.921032 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:56.928912 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:56.947980 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:56.948070 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:56.954536 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:56.958903 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:57.114666 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:57.125753 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:57.135210 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:57.144678 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:57.170760 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:57.173122 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:57.242102 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:57.242488 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:57.261205 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:57.261271 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:57.275636 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:57.291294 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:57.329990 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:57.334213 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:57.366968 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:57.383198 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:57.497447 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:57.497468 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:48:57.497473 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:48:57.497477 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:48:57.497487 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:57.497491 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:57.497493 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:57.497524 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:57.743097 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:57.743196 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:57.967946 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:57.969766 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.052481 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:58.057715 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.160529 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:48:58.220562 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:58.229116 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.309926 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:48:58.319367 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.352671 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:58.352692 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:48:58.353312 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:58.353357 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:58.353386 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:58.353436 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:58.353477 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:58.353505 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:58.353537 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:58.353579 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:48:58.353647 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:58.353709 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:58.353760 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:58.353802 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:58.353827 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:58.353848 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:58.353898 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:58.353932 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:58.353954 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:58.353978 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:48:58.354004 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:48:58.354032 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:48:58.354055 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:48:58.354083 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:58.359224 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:48:58.364100 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.374893 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:48:58.449930 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:58.450543 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:48:58.497568 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:58.497599 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:58.497604 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:58.497606 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:58.497644 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:58.498039 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:48:58.506734 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:48:58.515893 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.525364 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.644392 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:58.654618 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.686581 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:48:58.694382 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.743869 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:48:58.761535 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.789123 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:48:58.794863 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.847935 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:48:58.857987 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.865507 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:58.865750 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:48:58.921116 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:58.928992 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.954616 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:48:58.959002 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:58.970916 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:58.971165 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:48:59.114768 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:48:59.125834 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:59.135311 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:48:59.144750 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:59.170861 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:48:59.173216 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:59.275738 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:48:59.291391 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:59.330074 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:48:59.334302 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:59.367068 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:48:59.383292 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:48:59.427186 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:59.427289 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:48:59.497683 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:48:59.497708 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:48:59.497713 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:48:59.497716 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:48:59.497747 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:48:59.636199 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:59.636535 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:48:59.848992 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:59.849084 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:48:59.968030 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:48:59.969832 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:00.043850 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:00.043924 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:00.052570 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:00.057789 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:00.160618 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:00.161839 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:00.162181 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:00.220663 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:00.229195 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:00.310017 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:00.319442 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:00.359324 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:00.364194 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:00.374983 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:00.497789 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:00.497817 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:00.497822 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:00.497825 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:00.497862 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:00.498120 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:00.506817 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:00.515974 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:00.525462 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:00.644474 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:00.654697 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:00.686680 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:00.694454 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:00.743954 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:00.761627 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:00.789208 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:00.794944 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:00.839706 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:00.840134 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:00.848033 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:00.858063 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:00.921198 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:00.929076 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:00.954698 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:00.959098 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:01.114855 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:01.125931 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:01.135394 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:01.144829 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:01.166166 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:01.166296 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:01.170943 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:01.173312 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:01.275839 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:01.291487 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:01.330157 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:01.334403 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:01.367165 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:01.383388 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:01.497901 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:01.497955 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 2.52 >= max 0.5 = no, load too high
2014-06-12 14:49:01.497969 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:01.497972 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:01.497973 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92789 < max (604800 seconds)
2014-06-12 14:49:01.497993 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:01.497996 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:01.497998 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:01.549828 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:01.549904 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:01.582883 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:01.582988 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:01.594954 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:01.595283 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:01.666983 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:01.667168 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:01.871364 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:01.871484 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:01.950547 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:01.950578 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:01.968114 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:01.969914 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:02.052651 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:02.057863 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:02.083695 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:02.083826 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:02.160719 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:02.220772 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:02.229280 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:02.280334 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:02.280433 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:02.310102 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:02.319539 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:02.359424 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:02.364288 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:02.375064 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:02.498086 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:02.498109 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:49:02.498113 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:49:02.498117 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:49:02.498156 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 2.52 >= max 0.5 = no, load too high
2014-06-12 14:49:02.498169 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:02.498167 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:02.498172 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92790 < max (604800 seconds)
2014-06-12 14:49:02.498200 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:02.498201 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:02.498205 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:02.498207 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:02.506905 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:02.516054 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:02.525558 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:02.584556 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:02.584668 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:02.644555 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:02.654780 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:02.686780 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:02.694525 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:02.744053 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:02.761698 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:02.781223 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:02.781349 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:02.789296 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:02.795042 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:02.848129 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:02.858142 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:02.921281 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:02.929156 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:02.954780 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:02.959180 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:03.114940 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:03.126022 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:03.135494 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:03.144928 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:03.171028 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:03.173405 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:03.250660 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:03.250719 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:03.275939 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:03.281865 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:03.281958 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:03.291583 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:03.330240 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:03.334494 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:03.367263 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:03.383484 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:03.498286 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:03.498311 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:03.498316 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:03.498319 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:03.498357 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:03.662671 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:03.662932 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:03.968190 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:03.969984 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:04.052751 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:04.057949 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:04.160820 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:04.163440 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:04.163534 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:04.220858 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:04.229367 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:04.253411 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:04.253430 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:04.254256 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:04.254301 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:04.254340 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:04.254375 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:04.254397 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:04.254419 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:04.254440 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:04.254461 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:04.254494 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:04.254517 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:04.254538 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:04.254558 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:04.254580 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:04.254604 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:04.254631 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:04.254655 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:04.254679 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:04.254702 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:04.254729 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:04.254753 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:04.254777 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:04.254802 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:04.310192 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:04.319635 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:04.359525 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:04.364384 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:04.375160 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:04.498279 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:04.498394 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:04.498418 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:04.498423 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:04.498425 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:04.498463 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:04.506979 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:04.516159 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:04.525653 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:04.567597 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:04.567708 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:04.644642 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:04.654880 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:04.686881 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:04.694599 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:04.744153 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:04.761779 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:04.789395 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:04.795137 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:04.848229 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:04.858226 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:04.885299 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:04.885352 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:04.921354 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:04.929227 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:04.936701 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:04.936739 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:04.954865 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:04.959262 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:05.115027 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:05.126104 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:05.135564 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:05.145003 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:05.171101 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:05.173475 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:05.276041 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:05.291653 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:05.327486 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:05.327558 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:05.330325 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:05.334591 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:05.344635 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:05.344788 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:05.367336 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:05.371933 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:05.372019 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:05.383564 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:05.498519 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:05.498581 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 2.4 >= max 0.5 = no, load too high
2014-06-12 14:49:05.498595 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:05.498596 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:05.498598 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92793 < max (604800 seconds)
2014-06-12 14:49:05.498618 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:05.498621 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:05.498623 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:05.540559 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:05.540768 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:05.582593 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:05.582678 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:05.968285 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:05.970076 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:06.051311 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:06.051380 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:06.052838 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:06.058034 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:06.160917 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:06.220940 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:06.229452 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:06.267804 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:06.268171 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:06.310282 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:06.319731 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:06.359611 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:06.364480 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:06.375255 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:06.498364 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:06.498697 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:06.498723 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:06.498727 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:06.498730 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:06.498767 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:06.507052 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:06.516242 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:06.525726 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:06.644719 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:06.654979 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:06.686980 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:06.694683 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:06.744254 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:06.761871 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:06.789478 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:06.795224 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:06.848325 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:06.858313 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:06.895682 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:06.896116 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:06.921432 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:06.929300 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:06.954966 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:06.959343 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:07.028453 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:07.028599 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:07.115117 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:07.126202 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:07.135634 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:07.145075 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:07.171180 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:07.173563 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:07.237632 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:07.237784 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:07.276144 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:07.291728 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:07.330413 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:07.334676 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:07.367429 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:07.383660 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:07.498820 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:07.498842 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:49:07.498846 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:49:07.498850 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:49:07.498890 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 2.4 >= max 0.5 = no, load too high
2014-06-12 14:49:07.498903 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:07.498900 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:07.498907 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92795 < max (604800 seconds)
2014-06-12 14:49:07.498930 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:07.498934 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:07.498935 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:07.968327 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:07.968413 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:07.968513 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:07.970156 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:08.052942 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:08.058116 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:08.161001 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:08.221026 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:08.229544 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:08.310367 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:08.319811 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:08.338185 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:08.338359 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:08.359714 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:08.364576 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:08.375351 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:08.498456 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:08.499008 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:08.499036 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:08.499043 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:08.499047 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:08.499074 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:08.507134 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:08.516322 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:08.525805 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:08.644806 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:08.655077 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:08.687065 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:08.694750 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:08.744339 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:08.761952 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:08.789561 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:08.795319 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:08.845491 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:08.845729 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:08.848398 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:08.858396 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:08.921512 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:08.929382 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:08.952070 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:08.952220 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:08.954349 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:08.954369 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:08.955040 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:08.955156 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:08.955205 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:08.955238 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:08.955262 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:08.955284 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:08.955304 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:08.955348 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:08.955397 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:08.955472 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:08.955522 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:08.955575 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:08.955620 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:08.955660 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:08.955685 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:08.955708 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:08.955735 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:08.955761 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:08.955784 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:08.955808 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:08.955832 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:08.955855 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:08.955880 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:08.959416 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:09.115206 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:09.126283 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:09.135730 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:09.145147 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:09.151470 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:09.151539 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:09.171279 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:09.173699 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:09.276245 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:09.291824 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:09.330509 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:09.334776 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:09.367527 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:09.383755 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:09.464098 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:09.464232 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:09.472804 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:09.473292 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:09.499139 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:09.499165 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:09.499170 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:09.499172 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:09.499210 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:09.968595 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:09.970253 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:10.038762 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:10.038866 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:10.053031 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:10.058192 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:10.161083 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:10.186116 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:10.186286 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:10.221110 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:10.229639 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:10.241098 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:10.241535 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:10.310459 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:10.319890 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:10.359815 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:10.364671 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:10.375448 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:10.498530 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:10.499235 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:10.499252 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:10.499255 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:10.499257 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:10.499283 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:10.507206 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:10.516404 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:10.525873 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:10.529098 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:10.529166 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:10.644880 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:10.655151 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:10.687162 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:10.694818 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:10.744415 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:10.762031 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:10.789656 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:10.795414 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:10.848476 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:10.858478 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:10.883677 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:10.883744 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:10.921597 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:10.929456 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:10.955139 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:10.959505 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:11.115295 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:11.126364 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:11.135827 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:11.139673 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:11.139818 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:11.145227 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:11.171382 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:11.173781 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:11.276347 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:11.291920 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:11.330593 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:11.334876 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:11.367626 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:11.383850 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:11.499346 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:11.499409 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 2.28 >= max 0.5 = no, load too high
2014-06-12 14:49:11.499417 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:11.499424 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:11.499428 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92799 < max (604800 seconds)
2014-06-12 14:49:11.499457 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:11.499460 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:11.499462 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:11.640090 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:11.640542 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:11.852658 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:11.852921 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:11.968687 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:11.970336 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:12.053113 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:12.058274 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:12.161167 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:12.221194 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:12.229709 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:12.310547 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:12.319972 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:12.359913 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:12.364752 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:12.373587 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:12.374203 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:12.375525 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:12.498628 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:12.499527 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:12.499546 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:49:12.499549 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:49:12.499553 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:49:12.499562 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:12.499565 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:12.499567 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:12.499601 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:12.507285 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:12.516485 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:12.525957 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:12.541767 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:12.541828 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:12.644966 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:12.655242 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:12.687260 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:12.694899 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:12.744496 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:12.762115 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:12.789733 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:12.795506 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:12.796315 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:12.796502 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:12.848578 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:12.858566 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:12.921681 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:12.929536 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:12.955237 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:12.959600 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:13.115384 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:13.126452 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:13.135927 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:13.145325 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:13.171481 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:13.173863 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:13.252480 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:13.252563 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:13.269049 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:13.269196 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:13.276448 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:13.292017 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:13.297449 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:13.297576 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:13.330694 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:13.334975 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:13.367722 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:13.383930 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:13.499658 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:13.499713 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 2.28 >= max 0.5 = no, load too high
2014-06-12 14:49:13.499727 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:13.499728 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:13.499730 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92801 < max (604800 seconds)
2014-06-12 14:49:13.499768 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:13.499771 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:13.499773 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:13.546363 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:13.546481 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:13.564911 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:13.565105 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:13.686906 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:13.686999 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:13.797968 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:13.798083 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:13.968773 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:13.970426 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:14.053204 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:14.058356 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:14.161256 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:14.221278 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:14.229783 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:14.310651 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:14.320053 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:14.360001 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:14.364847 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:14.375616 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:14.384099 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:14.384228 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:14.498733 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:14.499838 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:14.499862 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:14.499868 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:14.499872 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:14.499900 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:14.507373 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:14.516566 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:14.526038 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:14.630120 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:14.630207 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:14.645053 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:14.655313 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:14.687350 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:14.694978 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:14.744568 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:14.762186 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:14.789805 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:14.795576 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:14.848661 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:14.855218 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:14.855239 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:14.855859 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:14.855938 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:14.856016 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:14.856060 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:14.856109 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:14.856156 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:14.856258 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:14.856389 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:14.856422 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:14.856454 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:14.856482 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:14.856509 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:14.856536 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:14.856562 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:14.856589 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:14.856615 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:14.856641 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:14.856668 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:14.856694 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:14.856720 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:14.856746 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:14.857754 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:14.858648 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:14.921770 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:14.929628 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:14.955334 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:14.959697 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:15.115485 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:15.126534 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:15.136010 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:15.140802 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:15.141114 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:15.145407 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:15.171581 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:15.173960 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:15.274199 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:15.274453 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:15.276530 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:15.292111 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:15.330794 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:15.335078 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:15.367817 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:15.384011 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:15.499988 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:15.500021 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:15.500028 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:15.500032 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:15.500067 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:15.553524 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:15.553598 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:15.968858 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:15.970523 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:16.053288 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:16.058447 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:16.098291 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:16.098823 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:16.161340 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:16.221364 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:16.229864 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:16.310739 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:16.320135 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:16.360088 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:16.364942 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:16.375713 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:16.498834 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:16.500112 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:16.500138 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:16.500143 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:16.500145 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:16.500182 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:16.507465 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:16.516658 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:16.526121 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:16.645139 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:16.655403 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:16.687448 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:16.695062 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:16.744645 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:16.762260 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:16.789897 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:16.795665 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:16.848759 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:16.858745 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:16.921857 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:16.929711 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:16.955432 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:16.959793 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:17.115588 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:17.126633 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:17.136108 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:17.145489 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:17.153306 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:17.153610 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:17.171679 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:17.174040 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:17.276629 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:17.292207 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:17.330892 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:17.335176 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:17.367915 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:17.369374 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:17.369447 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:17.384082 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:17.500211 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:17.500231 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:49:17.500234 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:49:17.500238 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:49:17.500247 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:17.500250 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:17.500252 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:17.500286 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:17.530941 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:17.531070 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:17.665649 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:17.665840 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:17.842458 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:17.842524 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:17.968942 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:17.970618 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:18.053375 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:18.058543 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:18.161421 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:18.221459 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:18.229944 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:18.247168 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:18.247474 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:18.310823 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:18.320216 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:18.343412 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:18.343547 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:18.355968 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:18.355987 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:18.356606 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:18.356673 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:18.356698 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:18.356726 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:18.356747 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:18.356782 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:18.356809 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:18.356829 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:18.356850 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:18.356945 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:18.357008 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:18.357103 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:18.357135 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:18.357156 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:18.357177 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:18.357207 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:18.357233 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:18.357257 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:18.357280 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:18.357307 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:18.357329 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:18.357353 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:18.360175 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:18.365023 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:18.375810 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:18.387716 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:18.387880 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:18.498908 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:18.500337 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:18.500353 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:18.500355 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:18.500356 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:18.500391 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:18.507535 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:18.516733 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:18.526201 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:18.645212 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:18.655483 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:18.687545 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:18.695162 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:18.744734 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:18.762348 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:18.789983 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:18.795761 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:18.848855 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:18.858825 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:18.921937 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:18.929794 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:18.955531 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:18.959889 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:19.084690 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:19.084768 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:19.115675 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:19.126730 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:19.136197 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:19.145570 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:19.171779 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:19.174122 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:19.276728 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:19.292302 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:19.330991 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:19.335261 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:19.366473 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:19.366603 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:19.367999 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:19.384164 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:19.500422 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:19.500447 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:19.500450 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:19.500452 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:19.500492 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:19.670101 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:19.670171 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:19.969025 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:19.970715 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:20.053475 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:20.058641 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:20.161506 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:20.221557 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:20.230024 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:20.310910 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:20.320290 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:20.360266 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:20.365120 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:20.375907 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:20.431820 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:20.431925 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:20.441482 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:20.441561 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:20.498990 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:20.500527 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:20.500554 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:20.500559 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:20.500561 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:20.500561 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:20.507612 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:20.516830 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:20.526286 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:20.574940 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:20.575002 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:20.645295 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:20.655556 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:20.687648 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:20.695240 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:20.744838 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:20.762429 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:20.790083 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:20.795858 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:20.798868 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:20.798944 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:20.848937 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:20.858921 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:20.922025 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:20.929876 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:20.955629 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:20.959968 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:21.115748 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:21.126800 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:21.136279 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:21.145648 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:21.171878 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:21.174201 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:21.276827 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:21.292398 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:21.331089 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:21.335359 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:21.368097 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:21.384244 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:21.454470 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:21.454559 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:21.500628 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:21.500651 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:21.500654 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:21.500656 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:21.500697 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:21.969112 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:21.970813 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:22.053549 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:22.058713 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:22.161579 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:22.221646 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:22.230093 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:22.310999 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:22.320368 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:22.360356 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:22.365214 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:22.375988 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:22.453821 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:22.454250 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:22.499090 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:22.499717 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:22.500004 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:22.500731 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:22.500752 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:49:22.500758 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:49:22.500765 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:49:22.500817 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:22.500818 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 2.17 >= max 0.5 = no, load too high
2014-06-12 14:49:22.500837 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:22.500842 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92810 < max (604800 seconds)
2014-06-12 14:49:22.500868 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:22.500872 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:22.500875 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:22.507695 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:22.516919 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:22.526368 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:22.645387 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:22.655650 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:22.687748 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:22.695319 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:22.744923 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:22.762511 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:22.790167 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:22.795953 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:22.849021 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:22.859021 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:22.867014 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:22.867137 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:22.922125 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:22.929955 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:22.947951 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:22.948014 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:22.955727 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:22.960050 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:23.000557 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:23.000624 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:23.088668 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:23.088764 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:23.115836 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:23.126880 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:23.136369 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:23.145730 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:23.171977 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:23.174284 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:23.276910 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:23.292493 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:23.331176 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:23.335442 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:23.342057 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:23.342324 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:23.368195 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:23.384338 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:23.500951 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:23.500976 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:23.500980 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:23.500982 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:23.501020 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:23.656719 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:23.656739 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:23.657346 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:23.657428 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:23.657470 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:23.657507 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:23.657536 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:23.657631 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:23.657676 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:23.657745 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:23.657788 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:23.657860 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:23.657919 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:23.657958 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:23.657995 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:23.658034 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:23.658070 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:23.658107 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:23.658154 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:23.658193 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:23.658243 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:23.658272 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:23.658294 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:23.658315 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:23.755315 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:23.755412 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:23.786138 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:23.786188 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:23.842627 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:23.843015 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:23.969198 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:23.970909 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.053646 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:24.058804 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.161657 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:24.221747 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:24.230169 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.243940 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:24.244329 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:24.311087 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:24.320464 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.360456 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:24.365309 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.376082 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:24.499186 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:24.501058 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:24.501084 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:24.501088 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:24.501090 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:24.501127 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:24.507779 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:24.517008 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.526464 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.645462 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:24.655747 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.687847 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:24.695399 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.745007 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:24.762585 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.790246 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:24.796034 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.849121 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:24.859118 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.922208 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:24.930035 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.955827 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:24.960132 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:24.970565 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:24.970894 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:25.115920 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:25.126977 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:25.136466 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:25.145804 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:25.172060 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:25.174374 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:25.248828 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:25.249209 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:25.276995 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:25.292573 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:25.301385 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:25.301963 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:25.331251 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:25.335541 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:25.368292 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:25.384420 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:25.501162 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:25.501188 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:25.501193 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:25.501195 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:25.501233 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:25.732015 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:25.732068 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:25.802084 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:25.802130 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:25.875468 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:25.875512 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:25.969283 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:25.970990 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.053741 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:26.058902 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.161732 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:26.221833 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:26.230250 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.311173 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:26.320560 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.360544 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:26.365400 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.376167 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:26.499285 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:26.501270 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:26.501296 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:26.501301 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:26.501303 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:26.501339 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:26.507867 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:26.517103 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.526541 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.557444 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:26.557465 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:26.558152 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:26.558240 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:26.558303 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:26.558373 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:26.558409 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:26.558478 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:26.558517 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:26.558555 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:26.558580 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:26.558620 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:26.558661 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:26.558692 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:26.558714 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:26.558735 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:26.558756 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:26.558777 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:26.558801 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:26.558831 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:26.558855 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:26.558879 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:26.558912 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:26.558939 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:26.645541 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:26.655843 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.671288 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:26.671633 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:26.687947 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:26.695476 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.743271 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:26.743593 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:26.745106 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:26.762664 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.790329 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:26.796130 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.849204 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:26.859213 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.922295 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:26.930112 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.955925 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:26.960211 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:26.967766 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:26.967884 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:27.116006 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:27.127056 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:27.136566 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:27.145900 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:27.172007 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:27.172196 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:27.172418 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:27.174465 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:27.277082 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:27.292670 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:27.331341 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:27.335640 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:27.368393 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:27.384514 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:27.501378 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:27.501398 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:49:27.501402 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:49:27.501407 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:49:27.501417 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:27.501421 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:27.501423 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:27.501458 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:27.789507 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:27.789673 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:27.969367 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:27.971087 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:28.053837 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:28.058998 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:28.161807 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:28.221909 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:28.230342 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:28.311262 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:28.320658 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:28.344552 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:28.345079 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:28.354441 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:28.354892 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:28.360628 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:28.365481 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:28.376251 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:28.444023 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:28.444350 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:28.456107 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:28.456219 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:28.486306 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:28.486562 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:28.499387 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:28.501498 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:28.501526 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:28.501532 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:28.501536 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:28.501565 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:28.507951 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:28.517187 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:28.526623 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:28.645656 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:28.655940 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:28.688045 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:28.695555 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:28.745188 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:28.762746 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:28.790409 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:28.796226 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:28.849289 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:28.855515 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:28.856018 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:28.859299 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:28.922390 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:28.930192 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:28.956023 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:28.960309 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:29.116090 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:29.127137 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:29.136664 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:29.145984 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:29.172289 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:29.174563 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:29.277169 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:29.292767 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:29.331411 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:29.335713 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:29.356095 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:29.356125 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:29.368477 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:29.384594 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:29.501629 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:29.501660 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:29.501666 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:29.501668 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:29.501699 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:29.544595 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:29.545091 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:29.969453 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:29.971181 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:30.053919 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:30.058224 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:30.058245 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:30.058847 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:30.058924 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:30.058975 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:30.059019 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:30.059068 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:30.059072 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:30.059097 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:30.059117 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:30.059138 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:30.059158 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:30.059180 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:30.059211 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:30.059236 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:30.059260 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:30.059283 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:30.059310 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:30.059336 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:30.059359 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:30.059383 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:30.059406 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:30.059431 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:30.059451 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:30.059471 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:30.090432 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:30.090511 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:30.161905 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:30.221992 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:30.230437 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:30.311356 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:30.320753 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:30.360721 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:30.365561 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:30.376348 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:30.499487 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:30.501748 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:30.501773 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:30.501778 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:30.501780 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:30.501820 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:30.508036 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:30.517276 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:30.526719 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:30.549843 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:30.549999 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:30.558952 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:30.558972 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:30.559708 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:30.559773 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:30.559801 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:30.559829 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:30.559855 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:30.559879 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:30.559904 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:30.559927 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:30.559959 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:30.559984 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:30.560009 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:30.560036 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:30.560061 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:30.560085 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:30.560108 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:30.560132 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:30.560156 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:30.560183 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:30.560207 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:30.560232 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:30.560260 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:30.563622 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:30.645296 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:30.645726 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:30.645738 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:30.656020 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:30.688144 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:30.695637 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:30.745274 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:30.762828 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:30.786991 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:30.787078 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:30.790507 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:30.796322 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:30.849371 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:30.859392 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:30.922488 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:30.930273 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:30.956123 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:30.960410 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:31.032513 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:31.032636 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:31.050640 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:31.050820 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:31.102685 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:31.102823 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:31.116177 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:31.127220 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:31.136749 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:31.145870 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:31.146094 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:31.146256 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:31.172388 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:31.174658 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:31.277260 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:31.292863 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:31.331505 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:31.335807 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:31.368575 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:31.384691 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:31.501854 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:31.501878 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:31.501883 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:31.501885 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:31.501924 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:31.657154 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:31.657425 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:31.776299 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:31.776426 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:31.872588 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:31.873005 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:31.969541 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:31.971277 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:32.053994 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:32.059148 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:32.161981 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:32.222068 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:32.230534 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:32.268874 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:32.268959 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:32.276779 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:32.276854 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:32.311441 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:32.320833 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:32.360821 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:32.365655 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:32.376430 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:32.445253 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:32.445323 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:32.499573 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:32.501976 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:32.501996 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:49:32.502000 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:49:32.502003 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:49:32.502042 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.91 >= max 0.5 = no, load too high
2014-06-12 14:49:32.502056 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:32.502055 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:32.502059 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92820 < max (604800 seconds)
2014-06-12 14:49:32.502087 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:32.502090 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:32.502092 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:32.508116 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:32.517370 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:32.526816 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:32.645815 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:32.656116 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:32.688243 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:32.695718 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:32.745358 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:32.762906 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:32.777380 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:32.777513 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:32.790605 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:32.796394 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:32.849452 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:32.859487 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:32.922585 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:32.930353 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:32.956200 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:32.960509 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:33.116251 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:33.127317 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:33.136831 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:33.146175 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:33.172483 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:33.174753 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:33.277354 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:33.292959 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:33.331601 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:33.335906 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:33.368672 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:33.384774 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:33.502159 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:33.502181 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:33.502185 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:33.502187 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:33.502229 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:33.756968 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:33.757050 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:33.969292 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:33.969523 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:33.969709 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:33.971370 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:34.046363 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:34.046678 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:34.054095 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:34.059233 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:34.146085 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:34.146221 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:34.162055 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:34.222148 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:34.230631 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:34.311539 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:34.320912 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:34.360905 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:34.365755 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:34.376505 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:34.499670 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:34.502266 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:34.502331 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:34.502328 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.91 >= max 0.5 = no, load too high
2014-06-12 14:49:34.502349 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:34.502354 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92822 < max (604800 seconds)
2014-06-12 14:49:34.502379 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:34.502385 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:34.502388 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:34.508201 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:34.517466 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:34.526907 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:34.645888 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:34.656219 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:34.688341 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:34.695792 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:34.745441 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:34.762982 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:34.790703 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:34.796487 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:34.849536 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:34.859583 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:34.922682 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:34.930433 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:34.956296 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:34.960606 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:35.077737 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:35.077969 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:35.116342 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:35.127413 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:35.136924 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:35.146259 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:35.172580 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:35.174847 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:35.277452 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:35.293055 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:35.331682 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:35.335991 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:35.368768 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:35.373118 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:35.373201 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:35.384854 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:35.391401 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:35.391499 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:35.502480 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:35.502504 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:35.502508 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:35.502511 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:35.502550 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:35.747104 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:35.747615 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:35.859793 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:35.859813 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:35.860519 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:35.860584 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:35.860633 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:35.860656 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:35.860677 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:35.860700 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:35.860776 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:35.860818 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:35.860842 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:35.860892 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:35.860936 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:35.860973 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:35.861013 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:35.861040 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:35.861065 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:35.861093 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:35.861116 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:35.861141 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:35.861172 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:35.861196 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:35.861219 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:35.861243 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:35.873778 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:35.874272 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:35.892183 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:35.892329 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:35.969786 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:35.971465 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:36.054188 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:36.059322 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:36.087626 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:36.087830 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:36.162129 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:36.222223 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:36.230727 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:36.311638 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:36.320993 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:36.360988 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:36.365833 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:36.376601 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:36.403442 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:36.403798 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:36.499769 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:36.502585 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:36.502649 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:36.502651 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.75 >= max 0.5 = no, load too high
2014-06-12 14:49:36.502671 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:36.502676 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92824 < max (604800 seconds)
2014-06-12 14:49:36.502701 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:36.502707 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:36.502711 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:36.508289 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:36.517564 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:36.526999 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:36.588433 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:36.588504 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:36.645963 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:36.656318 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:36.657767 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:36.657861 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:36.688439 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:36.695871 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:36.745531 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:36.763065 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:36.790801 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:36.796583 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:36.849625 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:36.859681 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:36.869808 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:36.869975 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:36.922782 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:36.930514 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:36.933483 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:36.933608 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:36.951394 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:36.951612 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:36.956395 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:36.957759 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:36.957871 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:36.960702 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:37.116425 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:37.127510 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:37.137006 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:37.146355 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:37.172677 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:37.174931 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:37.277552 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:37.293150 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:37.331779 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:37.336091 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:37.368864 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:37.384938 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:37.433955 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:37.434074 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:37.448114 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:37.448218 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:37.502777 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:37.502794 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:49:37.502797 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:49:37.502801 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:49:37.502809 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:37.502812 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:37.502814 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:37.502847 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:37.934842 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:37.935226 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:37.969864 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:37.971560 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:38.054275 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:38.059416 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:38.162228 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:38.222319 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:38.230808 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:38.311721 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:38.321076 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:38.358546 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:38.358635 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:38.361060 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:38.365930 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:38.376697 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:38.499865 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:38.502901 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:38.502924 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:38.502928 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:38.502930 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:38.502970 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:38.508364 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:38.517670 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:38.527096 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:38.646039 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:38.656412 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:38.688533 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:38.695952 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:38.704607 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:38.704671 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:38.745628 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:38.763146 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:38.790899 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:38.796679 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:38.846687 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:38.846839 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:38.849704 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:38.859776 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:38.922880 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:38.930594 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:38.956492 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:38.960795 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:39.116519 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:39.127606 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:39.137090 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:39.146452 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:39.172781 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:39.175025 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:39.277651 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:39.293246 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:39.331878 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:39.336189 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:39.368944 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:39.385019 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:39.503020 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:39.503072 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.75 >= max 0.5 = no, load too high
2014-06-12 14:49:39.503086 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:39.503090 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:39.503089 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92827 < max (604800 seconds)
2014-06-12 14:49:39.503108 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:39.503111 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:39.503112 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:39.805018 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:39.805292 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:39.960621 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:39.960640 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:39.961331 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:39.961398 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:39.961440 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:39.961464 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:39.961486 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:39.961507 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:39.961532 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:39.961554 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:39.961582 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:39.961607 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:39.961664 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:39.961691 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:39.961720 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:39.961746 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:39.961771 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:39.961796 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:39.961821 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:39.961848 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:39.961874 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:39.961898 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:39.961921 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:39.961945 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:39.969947 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:39.971653 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:39.974416 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:39.974574 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:40.054378 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:40.059510 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:40.162321 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:40.222417 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:40.230887 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:40.235652 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:40.236123 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:40.311803 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:40.321158 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:40.361149 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:40.366005 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:40.376779 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:40.378403 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:40.378488 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:40.458804 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:40.458946 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:40.499965 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:40.503202 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:40.503255 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.61 >= max 0.5 = no, load too high
2014-06-12 14:49:40.503268 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:40.503273 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:40.503272 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92828 < max (604800 seconds)
2014-06-12 14:49:40.503290 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:40.503293 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:40.503295 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:40.508437 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:40.517744 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:40.527193 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:40.646133 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:40.656510 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:40.688632 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:40.696026 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:40.745701 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:40.763225 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:40.790994 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:40.796766 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:40.849785 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:40.859873 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:40.922978 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:40.930674 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:40.956591 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:40.960879 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:40.970532 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:40.970693 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:41.052425 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:41.052616 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:41.116602 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:41.127702 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:41.137174 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:41.146549 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:41.147489 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:41.147676 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:41.172864 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:41.175120 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:41.277735 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:41.293342 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:41.331977 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:41.336289 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:41.369029 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:41.385100 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:41.503386 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:41.503410 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:41.503414 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:41.503416 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:41.503457 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:41.505934 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:41.506290 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:41.793092 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:41.793168 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:41.970030 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:41.971749 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:42.054455 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:42.059606 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:42.162419 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:42.222500 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:42.230968 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:42.311889 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:42.321244 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:42.361247 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:42.366085 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:42.376857 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:42.459443 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:42.459529 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:42.489463 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:42.489683 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:42.500048 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:42.503481 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:42.503498 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:49:42.503502 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:49:42.503506 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:49:42.503516 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:42.503520 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:42.503521 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:42.503522 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:42.508518 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:42.517825 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:42.527288 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:42.606772 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:42.607171 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:42.646212 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:42.656606 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:42.688731 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:42.696115 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:42.745776 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:42.748576 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:42.748759 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:42.763305 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:42.791095 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:42.796836 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:42.849866 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:42.859969 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:42.923075 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:42.930753 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:42.956689 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:42.960960 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:43.107564 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:43.108031 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:43.116684 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:43.127798 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:43.137256 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:43.146645 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:43.172948 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:43.175214 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:43.277819 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:43.293439 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:43.332058 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:43.336389 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:43.352872 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:43.353228 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:43.369113 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:43.385179 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:43.503610 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:43.503633 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:43.503638 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:43.503640 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:43.503662 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:43.853669 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:43.853749 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:43.878934 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:43.879011 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:43.970115 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:43.971842 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:44.054554 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:44.059701 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:44.061433 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:44.061452 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:44.062066 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:44.062138 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:44.062170 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:44.062241 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:44.062292 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:44.062385 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:44.062505 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:44.062614 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:44.062644 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:44.062669 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:44.062696 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:44.062718 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:44.062739 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:44.062759 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:44.062779 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:44.062804 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:44.062828 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:44.062851 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:44.062878 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:44.062902 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:44.062925 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:44.062945 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:44.162502 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:44.222598 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:44.231049 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:44.311975 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:44.321323 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:44.361337 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:44.366166 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:44.376937 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:44.500145 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:44.503729 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:44.503781 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.61 >= max 0.5 = no, load too high
2014-06-12 14:49:44.503795 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:44.503799 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:44.503798 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92832 < max (604800 seconds)
2014-06-12 14:49:44.503818 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:44.503821 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:44.503822 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:44.508606 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:44.517897 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:44.527383 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:44.646289 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:44.656695 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:44.688822 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:44.696195 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:44.745858 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:44.760208 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:44.760329 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:44.763385 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:44.791193 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:44.796926 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:44.849950 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:44.860064 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:44.923174 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:44.930828 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:44.956787 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:44.961041 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:45.116768 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:45.127881 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:45.137338 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:45.146741 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:45.159122 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:45.159183 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:45.173029 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:45.175310 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:45.274870 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:45.275178 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:45.277895 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:45.293535 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:45.332143 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:45.336480 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:45.369194 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:45.385261 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:45.503895 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:45.503948 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.48 >= max 0.5 = no, load too high
2014-06-12 14:49:45.503962 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:45.503966 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:45.503965 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92833 < max (604800 seconds)
2014-06-12 14:49:45.503984 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:45.503987 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:45.503989 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:45.536322 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:45.536439 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:45.762167 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:45.762186 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:45.762944 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:45.762997 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:45.763021 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:45.763076 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:45.763127 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:45.763291 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:45.763332 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:45.763358 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:45.763380 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:45.763405 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:45.763428 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:45.763456 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:45.763481 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:45.763506 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:45.763530 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:45.763555 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:45.763588 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:45.763612 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:45.763635 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:45.763659 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:45.763680 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:45.763704 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:45.893989 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:45.894080 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:45.970224 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:45.971937 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:46.054656 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:46.059798 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:46.162599 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:46.222680 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:46.231130 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:46.271084 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:46.271539 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:46.312058 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:46.321406 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:46.361426 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:46.366251 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:46.375589 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:46.375898 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:46.377011 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:46.448236 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:46.448478 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:46.500242 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:46.504060 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:46.504084 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:46.504088 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:46.504090 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:46.504130 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:46.508689 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:46.517984 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:46.527477 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:46.589964 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:46.590201 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:46.608275 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:46.608525 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:46.646371 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:46.656778 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:46.688908 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:46.696276 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:46.745932 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:46.754419 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:46.754513 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:46.763467 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:46.791293 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:46.797006 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:46.848905 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:46.849002 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:46.850023 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:46.860159 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:46.923271 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:46.930904 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:46.956884 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:46.961122 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:47.116852 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:47.127978 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:47.137424 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:47.146838 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:47.173112 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:47.175405 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:47.277970 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:47.293635 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:47.332236 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:47.336580 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:47.369276 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:47.385341 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:47.504179 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:47.504198 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:49:47.504202 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:49:47.504205 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:49:47.504209 7f9d84175700 10 osd.2 173 send_pg_stats - 0 pgs updated
2014-06-12 14:49:47.504245 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:47.504249 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:47.504251 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:47.504283 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:47.556673 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:47.556684 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:47.556687 7f9d74bfa700 20 osd.2 173 _dispatch 0x52c9880 pg_stats_ack(0 pgs tid 4866) v1
2014-06-12 14:49:47.556690 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:49:47.556693 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:47.556695 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:47.970309 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:47.972032 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:48.054749 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:48.059894 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:48.076354 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:48.076545 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:48.162697 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:48.194815 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:48.194883 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:48.222764 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:48.231221 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:48.260934 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:48.261032 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:48.312137 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:48.321487 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:48.361513 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:48.366344 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:48.377086 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:48.455742 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:48.455837 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:48.500341 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:48.504339 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:48.504394 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.48 >= max 0.5 = no, load too high
2014-06-12 14:49:48.504407 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:48.504411 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:48.504411 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92836 < max (604800 seconds)
2014-06-12 14:49:48.504430 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:48.504432 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:48.504434 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:48.508765 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:48.518059 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:48.527554 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:48.579744 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:48.579881 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:48.646447 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:48.656865 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:48.688995 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:48.695601 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:48.695730 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:48.696346 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:48.746006 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:48.763548 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:48.791391 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:48.797086 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:48.850103 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:48.860250 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:48.890832 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:48.891278 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:48.923369 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:48.930985 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:48.956982 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:48.961202 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:49.037019 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:49.037264 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:49.116938 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:49.128075 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:49.137505 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:49.146920 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:49.149558 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:49.149664 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:49.173203 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:49.175500 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:49.262954 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:49.262972 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:49.263781 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:49.263854 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:49.263903 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:49.263942 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:49.264049 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:49.264084 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:49.264128 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:49.264163 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:49.264189 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:49.264208 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:49.264231 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:49.264251 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:49.264272 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:49.264297 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:49.264321 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:49.264344 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:49.264369 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:49.264392 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:49.264413 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:49.264432 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:49.264453 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:49.264477 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:49.278054 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:49.293710 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:49.332332 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:49.336680 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:49.369360 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:49.385422 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:49.391750 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:49.392089 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:49.504524 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:49.504547 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:49.504552 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:49.504554 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:49.504595 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:49.860100 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:49.860271 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:49.948916 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:49.949151 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:49.970391 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:49.972127 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:50.054838 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:50.059988 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:50.156266 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:50.156422 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:50.162797 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:50.222846 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:50.231316 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:50.312207 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:50.321568 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:50.361604 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:50.366440 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:50.377166 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:50.500441 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:50.504644 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:50.504689 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.37 >= max 0.5 = no, load too high
2014-06-12 14:49:50.504696 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:50.504703 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:50.504707 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92838 < max (604800 seconds)
2014-06-12 14:49:50.504727 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:50.504729 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:50.504731 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:50.508836 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:50.518138 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:50.527649 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:50.646529 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:50.656948 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:50.689084 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:50.696416 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:50.746105 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:50.763631 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:50.791489 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:50.797174 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:50.850180 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:50.860332 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:50.923465 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:50.931069 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:50.957080 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:50.961282 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:51.117020 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:51.128165 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:51.137587 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:51.147016 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:51.173300 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:51.175595 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:51.278131 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:51.293781 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:51.332431 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:51.336772 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:51.369447 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:51.385500 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:51.504818 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:51.504843 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:51.504848 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:51.504850 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:51.504889 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:51.571977 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:51.572185 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:51.596363 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:51.596452 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:51.761879 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:51.761960 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:51.908809 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:51.908902 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:51.970481 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:51.972222 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:52.054915 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:52.060067 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:52.162881 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:52.222932 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:52.231411 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:52.292516 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:52.293035 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:52.312301 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:52.321658 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:52.361697 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:52.366537 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:52.377247 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:52.500540 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:52.504926 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:52.504947 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:49:52.504953 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:49:52.504959 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:49:52.504998 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.37 >= max 0.5 = no, load too high
2014-06-12 14:49:52.505010 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:52.505017 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:52.505022 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92840 < max (604800 seconds)
2014-06-12 14:49:52.505050 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:52.505055 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:52.505059 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:52.508909 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:52.518224 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:52.527744 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:52.646612 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:52.657047 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:52.689176 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:52.696496 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:52.746188 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:52.763712 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:52.776929 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:52.777066 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:52.791588 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:52.797270 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:52.850265 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:52.860428 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:52.923563 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:52.931152 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:52.957178 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:52.961363 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:53.117108 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:53.128262 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:53.137680 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:53.147113 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:53.173397 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:53.175689 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:53.272565 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:53.272647 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:53.278207 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:53.280431 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:53.280574 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:53.293878 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:53.297140 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:53.297244 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:53.332527 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:53.336855 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:53.363856 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:53.363875 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:53.364606 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:53.364668 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:53.364696 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:53.364766 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:53.364814 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:53.364840 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:53.364861 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:53.364975 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:53.365002 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:53.365022 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:53.365046 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:53.365067 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:53.365088 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:53.365112 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:53.365136 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:53.365164 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:53.365192 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:53.365216 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:53.365240 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:53.365263 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:53.365288 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:53.365311 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:53.369527 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:53.385581 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:53.505135 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:53.505179 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.37 >= max 0.5 = no, load too high
2014-06-12 14:49:53.505192 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:53.505205 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:53.505195 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92841 < max (604800 seconds)
2014-06-12 14:49:53.505214 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:53.505217 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:53.505219 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:53.737731 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:53.738153 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:53.773510 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:53.773599 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:53.970565 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:53.972316 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:53.993075 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:53.993585 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:54.054998 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:54.060155 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:54.162965 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:54.223015 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:54.231507 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:54.312399 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:54.321750 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:54.361776 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:54.366620 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:54.377327 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:54.450781 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:54.450851 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:54.500628 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:54.505293 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:54.505346 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.37 >= max 0.5 = no, load too high
2014-06-12 14:49:54.505359 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:54.505380 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:54.505385 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92842 < max (604800 seconds)
2014-06-12 14:49:54.505410 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:54.505415 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:54.505419 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:54.508981 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:54.518310 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:54.527835 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:54.560664 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:54.560816 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:54.646693 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:54.649829 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:54.650128 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:54.657131 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:54.689266 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:54.696579 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:54.746284 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:54.763793 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:54.791672 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:54.797365 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:54.809579 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:54.810059 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:54.850365 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:54.860524 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:54.923660 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:54.931233 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:54.957276 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:54.961444 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:54.998030 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:54.998093 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:49:55.077583 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:55.077677 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:55.117191 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:55.128356 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:55.137755 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:55.147193 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:55.173496 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:55.175784 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:55.278302 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:55.293958 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:55.310277 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:55.310329 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:55.332624 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:55.336937 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:55.369640 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:55.385659 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:55.505494 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:55.505540 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.26 >= max 0.5 = no, load too high
2014-06-12 14:49:55.505554 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:55.505565 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:55.505557 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92843 < max (604800 seconds)
2014-06-12 14:49:55.505576 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:55.505579 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:55.505580 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:55.693827 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:55.694004 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:55.970648 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:55.972412 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:56.055087 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:56.057234 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:56.057289 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:56.060252 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:56.151119 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:56.151247 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:56.163064 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:56.223102 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:56.231603 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:56.312496 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:56.321830 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:56.361866 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:56.366701 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:56.377413 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:56.500727 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:56.505649 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:56.505673 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:56.505678 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:56.505680 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:56.505719 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:56.509053 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:56.518409 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:56.527927 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:56.646777 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:56.652136 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:56.652258 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:56.657215 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:56.689354 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:56.696659 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:56.746369 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:56.763873 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:56.791767 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:56.797460 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:56.850446 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:56.860607 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:56.861283 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:56.861411 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:56.923757 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:56.931312 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:56.957374 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:56.961525 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:57.062628 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:57.062765 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:57.117272 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:57.128452 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:57.137853 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:57.147289 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:57.173594 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:57.175880 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:57.274261 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:57.274429 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:57.278391 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:57.294038 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:57.332724 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:57.337018 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:57.369724 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:57.385739 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:57.505769 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:57.505788 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:49:57.505792 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:49:57.505796 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:49:57.505805 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:57.505809 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:57.505811 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:57.505823 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:57.970732 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:57.972506 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:57.978361 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:57.978808 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:57.981043 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:57.981381 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:58.055185 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:58.060345 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:58.163157 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:49:58.210891 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:58.211077 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:58.223197 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:58.231684 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:58.312580 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:49:58.321911 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:58.361950 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:49:58.366781 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:58.377492 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:49:58.438856 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:58.438946 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:58.500827 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:49:58.505888 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:58.505932 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.26 >= max 0.5 = no, load too high
2014-06-12 14:49:58.505946 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:49:58.505942 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:58.505949 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92846 < max (604800 seconds)
2014-06-12 14:49:58.505968 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:49:58.505971 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:58.505973 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:58.509126 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:49:58.518504 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:58.528023 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:58.646861 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:58.657301 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:58.664702 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:58.664721 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:49:58.665390 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:58.665484 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:49:58.665527 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:58.665580 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:58.665679 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:58.665741 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:58.665790 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:58.665838 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:49:58.665881 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:58.665912 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:58.665936 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:49:58.665962 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:58.665986 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:49:58.666010 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:58.666045 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:49:58.666070 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:58.666094 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:58.666122 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:58.666145 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:49:58.666168 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:58.666194 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:49:58.666218 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:49:58.689442 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:49:58.696740 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:58.746446 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:49:58.763948 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:58.791865 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:49:58.797557 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:58.850528 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:49:58.860688 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:58.923843 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:58.931392 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:58.957471 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:49:58.961634 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:59.079028 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:59.079099 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:49:59.117355 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:49:59.128549 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:59.137935 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:49:59.147385 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:59.173677 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:49:59.175974 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:59.278471 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:49:59.294123 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:59.332824 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:49:59.337100 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:59.369809 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:49:59.385839 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:49:59.506047 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:49:59.506070 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:49:59.506074 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:49:59.506076 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:49:59.506119 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:49:59.911756 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:59.912205 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:49:59.950475 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:59.950556 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:49:59.970816 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:49:59.972603 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:00.055289 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:00.060443 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:00.157844 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:00.157936 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:00.163254 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:00.223272 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:00.231764 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:00.281705 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:00.281876 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:00.298719 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:00.298986 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:00.312664 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:00.322008 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:00.362029 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:00.365465 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:00.365485 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:00.366204 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:00.366285 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:00.366334 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:00.366402 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:00.366458 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:00.366529 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:00.366574 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:00.366617 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:00.366660 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:00.366697 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:00.366734 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:00.366786 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:00.366831 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:00.366852 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:00.366872 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:00.366915 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:00.366961 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:00.367006 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:00.367051 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:00.367091 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:00.367139 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:00.367181 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:00.367227 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:00.377577 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:00.500925 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:00.506150 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:00.506172 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:00.506177 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:00.506179 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:00.506219 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:00.509198 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:00.518590 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:00.528119 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:00.563408 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:00.563488 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:00.646941 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:00.657381 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:00.689532 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:00.696816 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:00.746528 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:00.764027 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:00.775035 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:00.775426 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:00.791963 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:00.797639 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:00.850615 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:00.860785 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:00.923941 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:00.931464 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:00.957568 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:00.961705 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:01.012456 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:01.012772 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:01.117439 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:01.128644 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:01.138018 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:01.147482 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:01.173768 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:01.176069 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:01.278553 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:01.294203 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:01.332920 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:01.337182 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:01.352681 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:01.352836 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:01.369884 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:01.385920 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:01.506270 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:01.506295 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:01.506300 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:01.506302 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:01.506341 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:01.594467 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:01.594788 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:01.970902 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:01.972698 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:02.055394 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:02.060540 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:02.162242 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:02.162364 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:02.163345 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:02.223374 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:02.231848 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:02.264127 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:02.264246 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:02.312739 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:02.322101 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:02.362105 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:02.366927 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:02.377666 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:02.458690 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:02.458808 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:02.501024 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:02.506392 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:02.506411 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:50:02.506414 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:50:02.506418 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:50:02.506450 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.16 >= max 0.5 = no, load too high
2014-06-12 14:50:02.506463 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:02.506471 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:02.506466 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92850 < max (604800 seconds)
2014-06-12 14:50:02.506485 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:02.506488 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:02.506490 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:02.509269 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:02.518673 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:02.528215 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:02.647023 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:02.657471 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:02.663013 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:02.663160 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:02.689633 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:02.696890 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:02.746613 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:02.764110 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:02.792061 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:02.797727 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:02.850699 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:02.851228 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:02.851478 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:02.860882 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:02.924040 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:02.931544 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:02.957652 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:02.961802 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:03.117521 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:03.128732 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:03.138101 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:03.139292 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:03.139381 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:03.147578 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:03.173852 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:03.176164 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:03.266288 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:03.266307 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:03.266992 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:03.267114 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:03.267160 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:03.267184 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:03.267214 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:03.267239 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:03.267264 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:03.267329 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:03.267380 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:03.267412 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:03.267461 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:03.267483 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:03.267503 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:03.267529 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:03.267558 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:03.267586 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:03.267610 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:03.267636 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:03.267663 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:03.267690 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:03.267713 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:03.267741 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:03.278636 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:03.294282 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:03.333005 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:03.337268 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:03.369974 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:03.386001 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:03.506564 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:03.506587 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:03.506591 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:03.506593 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:03.506635 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:03.970986 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:03.972793 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.055484 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:04.060637 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.163427 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:04.223443 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:04.231928 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.312821 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:04.322181 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.362189 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:04.367009 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.377761 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:04.399557 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:04.399663 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:04.501123 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:04.506666 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:04.506709 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.16 >= max 0.5 = no, load too high
2014-06-12 14:50:04.506723 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:04.506719 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:04.506726 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92852 < max (604800 seconds)
2014-06-12 14:50:04.506745 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:04.506748 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:04.506750 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:04.509342 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:04.518760 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.528312 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.551921 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:04.552425 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:04.647106 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:04.657558 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.689710 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:04.696964 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.746696 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:04.764207 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.792160 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:04.797810 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.850785 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:04.853711 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:04.853804 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:04.861308 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.924137 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:04.931637 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.957735 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:04.961877 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:04.979659 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:04.979906 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:05.117603 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:05.128814 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:05.138186 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:05.147658 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:05.173935 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:05.176260 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:05.278722 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:05.294362 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:05.333099 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:05.337355 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:05.354176 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:05.354488 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:05.370048 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:05.386081 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:05.439696 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:05.439792 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:05.506828 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:05.506882 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.06 >= max 0.5 = no, load too high
2014-06-12 14:50:05.506894 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:05.506915 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:05.506922 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92853 < max (604800 seconds)
2014-06-12 14:50:05.506947 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:05.506953 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:05.506957 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:05.582348 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:05.582470 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:05.695166 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:05.695278 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:05.971068 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:05.972890 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:06.055580 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:06.060730 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:06.083141 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:06.083519 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:06.163525 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:06.223526 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:06.232012 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:06.312904 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:06.322262 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:06.362265 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:06.367099 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:06.377840 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:06.501221 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:06.507037 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:06.507061 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:06.507065 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:06.507067 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:06.507108 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:06.509415 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:06.518848 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:06.528404 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:06.647199 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:06.657640 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:06.675767 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:06.676098 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:06.689796 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:06.697036 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:06.700250 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:06.700304 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:06.746781 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:06.764285 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:06.792259 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:06.797891 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:06.850866 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:06.861386 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:06.913241 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:06.913386 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:06.924234 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:06.931736 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:06.957810 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:06.961953 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:07.117719 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:07.128892 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:07.138263 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:07.147754 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:07.174024 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:07.176340 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:07.278824 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:07.294444 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:07.333198 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:07.337439 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:07.367080 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:07.367099 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:07.367704 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:07.367802 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:07.367881 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:07.367929 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:07.367983 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:07.368057 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:07.368084 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:07.368108 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:07.368135 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:07.368186 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:07.368225 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:07.368292 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:07.368333 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:07.368360 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:07.368385 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:07.368413 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:07.368437 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:07.368461 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:07.368485 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:07.368505 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:07.368529 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:07.368553 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:07.370131 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:07.386178 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:07.507156 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:07.507174 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:50:07.507177 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:50:07.507181 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:50:07.507212 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.06 >= max 0.5 = no, load too high
2014-06-12 14:50:07.507225 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:07.507236 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:07.507228 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92855 < max (604800 seconds)
2014-06-12 14:50:07.507247 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:07.507250 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:07.507252 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:07.971154 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:07.972986 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:07.996011 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:07.996138 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:08.055684 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:08.060827 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:08.163624 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:08.164940 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:08.165007 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:08.223610 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:08.232103 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:08.312980 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:08.322359 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:08.359454 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:08.359643 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:08.362347 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:08.367198 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:08.377934 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:08.501318 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:08.507341 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:08.507365 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:08.507370 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:08.507372 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:08.507411 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:08.509486 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:08.518934 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:08.528496 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:08.563863 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:08.564112 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:08.647277 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:08.652591 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:08.652692 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:08.657716 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:08.689878 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:08.697109 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:08.746881 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:08.764367 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:08.792357 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:08.797988 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:08.850966 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:08.861483 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:08.924332 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:08.931835 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:08.957891 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:08.962041 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:09.117801 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:09.128972 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:09.138339 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:09.147850 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:09.174106 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:09.176433 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:09.278906 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:09.294524 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:09.333294 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:09.337530 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:09.370230 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:09.386258 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:09.454799 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:09.454879 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:09.507461 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:09.507507 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 1.06 >= max 0.5 = no, load too high
2014-06-12 14:50:09.507521 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:09.507532 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:09.507524 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92857 < max (604800 seconds)
2014-06-12 14:50:09.507543 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:09.507546 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:09.507547 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:09.971237 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:09.973079 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:10.055780 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:10.060906 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:10.163721 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:10.176215 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:10.176363 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:10.223694 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:10.232198 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:10.313051 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:10.322439 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:10.362430 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:10.367298 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:10.378014 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:10.501417 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:10.507638 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:10.507661 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:10.507666 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:10.507668 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:10.507709 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:10.509557 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:10.519025 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:10.528586 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:10.647359 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:10.657802 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:10.660173 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:10.660358 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:10.689959 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:10.697184 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:10.740432 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:10.740519 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:10.746967 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:10.764448 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:10.792455 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:10.798068 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:10.851055 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:10.861578 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:10.880169 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:10.880292 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:10.924429 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:10.931934 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:10.953247 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:10.953380 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:10.957974 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:10.962122 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:11.117873 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:11.129054 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:11.138422 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:11.147943 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:11.174188 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:11.176529 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:11.241338 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:11.241398 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:11.278994 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:11.294612 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:11.333394 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:11.337628 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:11.370306 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:11.386355 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:11.507757 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:11.507782 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:11.507787 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:11.507789 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:11.507828 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:11.614051 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:11.614354 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:11.876861 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:11.877152 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:11.971321 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:11.973175 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:11.983802 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:11.984161 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:12.001025 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:12.001154 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:12.055868 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:12.060991 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:12.163820 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:12.223780 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:12.232295 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:12.313134 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:12.322535 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:12.362510 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:12.367393 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:12.378111 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:12.501516 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:12.507880 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:12.507898 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:50:12.507902 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:50:12.507906 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:50:12.507915 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:12.507918 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:12.507920 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:12.507959 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:12.509633 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:12.519125 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:12.528665 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:12.647435 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:12.657887 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:12.665117 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:12.665226 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:12.667783 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:12.667803 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:12.668330 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:12.668435 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:12.668549 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:12.668607 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:12.668655 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:12.668690 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:12.668717 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:12.668827 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:12.668850 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:12.668872 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:12.668893 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:12.668933 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:12.668956 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:12.668976 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:12.668996 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:12.669032 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:12.669059 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:12.669080 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:12.669100 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:12.669133 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:12.669159 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:12.669180 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:12.690040 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:12.697259 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:12.747061 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:12.764528 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:12.792553 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:12.798154 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:12.851153 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:12.861660 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:12.865752 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:12.865852 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:12.924528 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:12.932034 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:12.958074 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:12.962202 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:13.117953 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:13.129135 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:13.138505 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:13.148038 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:13.174278 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:13.176618 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:13.279090 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:13.294696 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:13.333497 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:13.337711 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:13.370398 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:13.386434 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:13.508010 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:13.508034 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:13.508039 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:13.508041 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:13.508081 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:13.701777 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:13.701852 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:13.896801 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:13.897094 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:13.914971 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:13.915379 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:13.971406 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:13.973270 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.055963 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:14.061087 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.163904 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:14.202616 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:14.202668 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:14.223861 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:14.232375 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.313224 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:14.322615 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.362613 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:14.367488 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.378190 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:14.453903 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:14.454242 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:14.501627 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:14.508131 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:14.508157 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:14.508162 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:14.508164 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:14.508202 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:14.509709 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:14.519228 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.528736 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.647511 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:14.657964 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.690122 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:14.697332 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.747159 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:14.755650 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:14.756084 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:14.764609 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.792652 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:14.798232 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.851252 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:14.861753 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.924625 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:14.932131 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.958152 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:14.962284 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:14.965647 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:14.965787 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:15.118028 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:15.129217 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:15.138589 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:15.148129 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:15.174363 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:15.176699 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:15.279189 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:15.294780 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:15.333596 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:15.337812 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:15.360925 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:15.361179 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:15.370483 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:15.377642 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:15.377884 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:15.386514 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:15.466204 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:15.466537 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:15.508253 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:15.508277 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:15.508281 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:15.508283 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:15.508308 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:15.580800 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:15.581152 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:15.856634 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:15.857047 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:15.971490 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:15.973365 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:16.056062 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:16.061181 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:16.084795 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:16.084922 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:16.164003 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:16.197684 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:16.197979 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:16.223946 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:16.232471 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:16.313309 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:16.322697 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:16.362702 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:16.367581 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:16.378284 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:16.501704 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:16.508372 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:16.508396 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:16.508400 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:16.508402 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:16.508442 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:16.509779 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:16.519323 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:16.528814 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:16.647592 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:16.658051 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:16.690219 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:16.697413 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:16.747260 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:16.764688 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:16.792731 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:16.798329 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:16.851349 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:16.861832 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:16.924723 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:16.932227 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:16.958250 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:16.962381 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:17.118097 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:17.129298 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:17.138674 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:17.142231 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:17.142565 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:17.148224 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:17.174447 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:17.176779 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:17.279288 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:17.294862 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:17.333652 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:17.337897 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:17.370582 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:17.386593 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:17.508476 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:17.508496 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:50:17.508500 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:50:17.508504 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:50:17.508536 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.9 >= max 0.5 = no, load too high
2014-06-12 14:50:17.508548 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:17.508557 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:17.508552 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92865 < max (604800 seconds)
2014-06-12 14:50:17.508571 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:17.508573 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:17.508575 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:17.557334 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:17.557409 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:17.954628 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:17.954748 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:17.968400 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:17.968419 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:17.969146 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:17.969201 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:17.969256 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:17.969315 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:17.969345 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:17.969366 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:17.969392 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:17.969432 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:17.969472 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:17.969516 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:17.969541 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:17.969563 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:17.969592 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:17.969632 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:17.969654 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:17.969719 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:17.969748 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:17.969769 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:17.969808 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:17.969831 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:17.969858 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:17.969882 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:17.971566 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:17.973443 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:18.056169 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:18.061279 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:18.164101 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:18.224039 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:18.232568 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:18.278311 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:18.278514 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:18.313394 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:18.322776 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:18.362791 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:18.367079 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:18.367199 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:18.367656 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:18.378366 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:18.385104 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:18.385508 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:18.501785 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:18.508665 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:18.508688 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:18.508692 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:18.508695 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:18.508736 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:18.509850 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:18.519420 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:18.528894 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:18.647675 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:18.658134 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:18.690307 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:18.697494 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:18.747356 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:18.764770 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:18.766554 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:18.766627 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:18.792814 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:18.798410 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:18.851446 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:18.861912 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:18.885779 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:18.886195 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:18.903391 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:18.903499 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:18.924822 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:18.932327 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:18.958334 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:18.962462 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:19.118168 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:19.129377 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:19.138774 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:19.148318 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:19.174530 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:19.176862 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:19.215720 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:19.215836 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:19.279387 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:19.294952 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:19.333749 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:19.337984 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:19.370668 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:19.386674 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:19.508786 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:19.508811 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:19.508816 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:19.508818 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:19.508846 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:19.669220 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:19.669238 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:19.670034 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:19.670137 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:19.670190 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:19.670259 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:19.670310 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:19.670354 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:19.670405 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:19.670446 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:19.670485 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:19.670530 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:19.670585 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:19.670630 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:19.670670 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:19.670720 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:19.670766 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:19.670810 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:19.670856 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:19.670899 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:19.670943 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:19.670986 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:19.671028 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:19.671075 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:19.971647 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:19.973537 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:19.979436 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:19.979499 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:20.056270 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:20.061374 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:20.067755 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:20.068124 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:20.164198 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:20.170095 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:20.170112 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:20.170740 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:20.170858 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:20.170890 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:20.170911 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:20.170934 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:20.170959 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:20.170986 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:20.171034 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:20.171080 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:20.171105 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:20.171143 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:20.171164 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:20.171195 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:20.171238 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:20.171274 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:20.171303 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:20.171327 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:20.171368 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:20.171404 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:20.171427 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:20.171457 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:20.171481 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:20.224135 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:20.232649 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:20.313480 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:20.322858 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:20.362880 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:20.367745 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:20.378448 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:20.501869 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:20.508908 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:20.508932 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:20.508937 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:20.508939 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:20.508980 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:20.509920 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:20.519516 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:20.528967 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:20.647759 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:20.658215 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:20.661699 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:20.661983 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:20.690393 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:20.697575 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:20.747454 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:20.764850 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:20.792898 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:20.798508 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:20.851545 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:20.861986 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:20.898439 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:20.898869 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:20.916539 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:20.916608 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:20.924919 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:20.932425 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:20.958432 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:20.962541 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:21.118240 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:21.129466 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:21.138856 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:21.148413 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:21.174614 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:21.176943 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:21.242812 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:21.242891 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:21.279488 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:21.295046 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:21.333834 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:21.338070 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:21.370769 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:21.386753 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:21.417366 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:21.417501 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:21.481255 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:21.481353 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:21.509014 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:21.509037 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:21.509041 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:21.509044 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:21.509085 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:21.679938 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:21.680021 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:21.768484 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:21.768718 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:21.786322 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:21.786437 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:21.971734 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:21.973636 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:22.056360 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:22.061470 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:22.164280 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:22.224234 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:22.232730 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:22.258142 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:22.258303 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:22.267375 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:22.267477 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:22.313554 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:22.322945 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:22.362977 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:22.367843 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:22.378526 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:22.404156 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:22.404218 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:22.501962 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:22.509118 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:22.509137 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:50:22.509141 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:50:22.509144 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:50:22.509176 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.91 >= max 0.5 = no, load too high
2014-06-12 14:50:22.509188 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:22.509198 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:22.509192 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92870 < max (604800 seconds)
2014-06-12 14:50:22.509211 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:22.509214 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:22.509216 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:22.509990 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:22.519614 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:22.529044 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:22.647842 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:22.658301 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:22.690478 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:22.697649 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:22.747552 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:22.764929 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:22.792981 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:22.798590 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:22.851644 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:22.862080 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:22.904921 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:22.905034 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:22.925016 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:22.932523 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:22.958514 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:22.962623 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:23.118311 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:23.129560 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:23.138956 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:23.148509 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:23.174700 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:23.177023 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:23.255618 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:23.255714 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:23.279588 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:23.295144 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:23.333919 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:23.338156 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:23.370850 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:23.386832 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:23.486894 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:23.487107 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:23.509290 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:23.509314 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:23.509318 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:23.509320 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:23.509362 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:23.543730 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:23.544032 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:23.562474 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:23.562588 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:23.670840 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:23.670860 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:23.671725 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:23.671796 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:23.671857 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:23.671904 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:23.671972 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:23.672021 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:23.672070 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:23.672113 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:23.672139 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:23.672186 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:23.672219 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:23.672245 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:23.672270 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:23.672294 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:23.672321 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:23.672348 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:23.672373 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:23.672401 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:23.672425 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:23.672449 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:23.672473 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:23.672497 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:23.971818 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:23.973733 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.044277 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:24.044612 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:24.056453 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:24.061566 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.164376 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:24.224324 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:24.232821 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.313642 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:24.323029 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.363074 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:24.367938 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.378609 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:24.502062 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:24.509404 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:24.509427 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:24.509432 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:24.509434 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:24.509475 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:24.510060 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:24.519710 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.529126 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.558839 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:24.559335 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:24.647925 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:24.658385 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.690562 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:24.697723 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.747650 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:24.765009 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.793065 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:24.798672 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.851742 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:24.862161 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.925114 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:24.932621 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.958613 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:24.962720 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:24.981799 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:24.981846 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:25.118383 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:25.129665 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:25.139053 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:25.148591 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:25.174782 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:25.177105 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:25.180790 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:25.181018 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:25.187552 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:25.187667 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:25.263370 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:25.263527 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:25.279687 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:25.295240 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:25.334018 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:25.338241 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:25.370941 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:25.386929 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:25.509501 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:25.509542 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.83 >= max 0.5 = no, load too high
2014-06-12 14:50:25.509554 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:25.509557 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92873 < max (604800 seconds)
2014-06-12 14:50:25.509575 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:25.509571 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:25.509578 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:25.509579 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:25.971903 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:25.973814 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:26.056538 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:26.061666 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:26.118473 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:26.118574 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:26.164474 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:26.199313 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:26.199459 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:26.224407 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:26.232901 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:26.313729 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:26.323126 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:26.363149 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:26.368035 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:26.378695 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:26.502139 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:26.509655 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:26.509707 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.83 >= max 0.5 = no, load too high
2014-06-12 14:50:26.509721 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:26.509742 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:26.509749 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92874 < max (604800 seconds)
2014-06-12 14:50:26.509775 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:26.509780 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:26.509783 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:26.510131 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:26.519810 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:26.529212 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:26.648009 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:26.658469 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:26.690646 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:26.697795 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:26.747749 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:26.756488 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:26.756545 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:26.765089 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:26.793150 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:26.798752 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:26.851840 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:26.862241 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:26.925212 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:26.932719 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:26.945036 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:26.945524 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:26.958700 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:26.962801 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:27.005734 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:27.005803 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:27.069140 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:27.069227 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:27.118456 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:27.129752 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:27.139136 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:27.148685 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:27.171809 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:27.171829 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:27.172678 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:27.172758 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:27.172816 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:27.172876 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:27.172917 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:27.172940 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:27.172961 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:27.172982 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:27.173014 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:27.173055 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:27.173082 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:27.173107 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:27.173131 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:27.173154 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:27.173181 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:27.173204 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:27.173228 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:27.173251 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:27.173277 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:27.173304 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:27.173332 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:27.173360 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:27.174860 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:27.177188 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:27.279786 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:27.282687 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:27.282791 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:27.295321 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:27.334099 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:27.338327 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:27.371042 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:27.387004 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:27.460002 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:27.460112 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:27.509866 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:27.509885 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:50:27.509889 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:50:27.509892 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:50:27.509902 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:27.509906 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:27.509908 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:27.509946 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:27.568294 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:27.568390 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:27.971989 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:27.973903 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:28.056627 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:28.061748 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:28.164572 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:28.224487 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:28.232982 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:28.313814 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:28.323221 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:28.363247 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:28.368131 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:28.378773 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:28.457412 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:28.457827 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:28.502221 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:28.509975 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:28.509997 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:28.510000 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:28.510002 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:28.510045 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:28.510202 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:28.519908 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:28.529292 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:28.560498 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:28.560687 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:28.645969 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:28.646434 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:28.648091 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:28.658550 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:28.690732 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:28.697869 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:28.747848 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:28.765171 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:28.793233 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:28.798837 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:28.851938 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:28.862338 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:28.925311 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:28.932821 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:28.958782 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:28.962898 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:29.100170 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:29.100431 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:29.118526 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:29.129849 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:29.139233 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:29.148781 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:29.174958 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:29.177267 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:29.279886 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:29.295417 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:29.334182 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:29.338428 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:29.364123 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:29.364177 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:29.371140 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:29.387099 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:29.510070 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:29.510115 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.83 >= max 0.5 = no, load too high
2014-06-12 14:50:29.510129 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:29.510141 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:29.510132 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92877 < max (604800 seconds)
2014-06-12 14:50:29.510152 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:29.510155 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:29.510156 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:29.972079 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:29.973976 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:30.056717 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:30.061822 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:30.164670 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:30.224569 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:30.233065 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:30.313907 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:30.323317 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:30.346484 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:30.346573 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:30.363346 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:30.368210 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:30.378871 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:30.469204 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:30.469313 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:30.488151 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:30.488533 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:30.502305 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:30.510230 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:30.510253 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:30.510258 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:30.510260 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:30.510294 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:30.510313 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:30.520003 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:30.529377 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:30.648174 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:30.658635 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:30.690824 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:30.697941 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:30.747948 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:30.765250 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:30.782944 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:30.783278 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:30.793320 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:30.798929 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:30.800971 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:30.801274 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:30.852035 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:30.862413 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:30.925408 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:30.932918 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:30.958869 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:30.962994 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:31.081421 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:31.081495 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:31.118598 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:31.129930 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:31.139329 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:31.148856 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:31.170360 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:31.170595 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:31.175056 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:31.177353 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:31.279984 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:31.295513 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:31.334278 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:31.338515 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:31.371238 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:31.387169 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:31.418602 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:31.418675 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:31.510336 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:31.510360 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:31.510364 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:31.510367 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:31.510408 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:31.664994 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:31.665395 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:31.972168 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:31.974056 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:32.056806 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:32.061896 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:32.164767 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:32.224652 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:32.233145 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:32.306554 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:32.306813 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:32.313994 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:32.323414 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:32.363443 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:32.368302 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:32.378964 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:32.483580 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:32.483701 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:32.502388 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:32.510383 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:32.510472 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:32.510494 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:50:32.510498 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:50:32.510502 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:50:32.510511 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:32.510515 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:32.510517 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:32.510550 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:32.520100 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:32.529471 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:32.648257 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:32.658721 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:32.690921 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:32.698014 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:32.748045 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:32.765331 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:32.782251 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:32.782547 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:32.793413 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:32.799024 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:32.852132 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:32.862511 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:32.925506 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:32.933015 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:32.958969 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:32.963090 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:33.072700 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:33.072720 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:33.073641 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:33.073708 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:33.073793 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:33.073878 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:33.073908 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:33.073939 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:33.073960 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:33.073984 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:33.074009 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:33.074033 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:33.074058 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:33.074081 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:33.074106 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:33.074131 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:33.074156 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:33.074179 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:33.074203 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:33.074227 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:33.074259 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:33.074282 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:33.074306 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:33.074332 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:33.118667 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:33.130010 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:33.139428 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:33.148951 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:33.175154 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:33.177449 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:33.280083 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:33.283133 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:33.283534 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:33.295596 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:33.334362 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:33.338601 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:33.371321 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:33.387263 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:33.407551 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:33.407635 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:33.471058 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:33.471163 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:33.510592 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:33.510636 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.77 >= max 0.5 = no, load too high
2014-06-12 14:50:33.510650 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:33.510660 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:33.510653 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92881 < max (604800 seconds)
2014-06-12 14:50:33.510672 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:33.510677 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:33.510679 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:33.970041 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:33.970162 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:33.972250 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:33.974137 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:34.056897 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:34.061987 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:34.164851 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:34.224725 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:34.233227 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:34.314079 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:34.319473 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:34.319817 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:34.323495 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:34.358009 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:34.358137 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:34.363541 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:34.368395 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:34.379060 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:34.461172 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:34.461277 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:34.470945 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:34.471057 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:34.502472 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:34.510452 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:34.510745 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:34.510764 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:34.510768 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:34.510770 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:34.510813 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:34.520198 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:34.529568 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:34.588995 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:34.589053 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:34.648341 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:34.658802 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:34.691012 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:34.698086 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:34.748144 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:34.765410 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:34.793512 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:34.799122 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:34.852235 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:34.862593 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:34.901729 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:34.902164 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:34.925603 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:34.933113 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:34.959068 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:34.963184 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:35.108424 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:35.108545 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:35.118770 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:35.130099 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:35.139529 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:35.149048 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:35.165868 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:35.166040 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:35.175237 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:35.177529 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:35.280181 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:35.295692 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:35.334446 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:35.338685 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:35.371417 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:35.387358 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:35.510844 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:35.510868 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:35.510873 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:35.510875 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:35.510900 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:35.972335 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:35.974216 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:35.984178 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:35.984260 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:36.020231 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:36.020556 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:36.056985 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:36.062063 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:36.164951 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:36.224809 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:36.233310 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:36.247502 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:36.247602 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:36.314170 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:36.323591 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:36.363617 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:36.368477 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:36.379153 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:36.484797 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:36.484956 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:36.502554 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:36.510522 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:36.510940 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:36.510960 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:36.510964 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:36.510966 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:36.511010 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:36.520296 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:36.529651 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:36.573706 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:36.573725 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:36.574574 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:36.574698 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:36.574806 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:36.574856 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:36.574883 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:36.574904 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:36.574926 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:36.574948 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:36.574981 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:36.575027 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:36.575054 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:36.575078 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:36.575127 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:36.575174 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:36.575203 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:36.575230 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:36.575254 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:36.575277 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:36.575302 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:36.575325 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:36.575349 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:36.575372 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:36.648430 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:36.658895 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:36.691110 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:36.698155 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:36.748242 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:36.765496 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:36.793626 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:36.799218 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:36.852334 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:36.862673 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:36.889261 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:36.889705 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:36.925718 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:36.933212 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:36.959165 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:36.963279 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:37.118846 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:37.130195 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:37.139627 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:37.149145 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:37.175336 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:37.177625 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:37.202569 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:37.202674 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:37.280281 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:37.295771 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:37.334547 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:37.338786 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:37.371514 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:37.387455 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:37.511054 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:37.511072 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:50:37.511076 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:50:37.511080 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:50:37.511110 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.71 >= max 0.5 = no, load too high
2014-06-12 14:50:37.511123 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:37.511133 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:37.511126 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92885 < max (604800 seconds)
2014-06-12 14:50:37.511146 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:37.511148 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:37.511150 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:37.972427 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:37.974297 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:38.057081 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:38.062161 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:38.165049 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:38.224890 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:38.233390 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:38.274668 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:38.274687 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:38.275518 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:38.275610 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:38.275681 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:38.275719 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:38.275752 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:38.275777 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:38.275808 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:38.275829 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:38.275955 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:38.275992 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:38.276020 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:38.276048 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:38.276072 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:38.276095 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:38.276124 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:38.276148 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:38.276171 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:38.276195 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:38.276223 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:38.276246 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:38.276270 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:38.276294 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:38.314254 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:38.323671 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:38.363716 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:38.368560 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:38.379248 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:38.502636 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:38.510592 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:38.511226 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:38.511270 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.71 >= max 0.5 = no, load too high
2014-06-12 14:50:38.511283 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:38.511297 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:38.511287 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92886 < max (604800 seconds)
2014-06-12 14:50:38.511307 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:38.511310 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:38.511311 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:38.520393 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:38.529730 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:38.648515 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:38.658990 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:38.666689 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:38.666893 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:38.691209 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:38.698226 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:38.748341 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:38.765579 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:38.771642 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:38.772022 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:38.793732 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:38.799312 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:38.852432 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:38.862756 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:38.925792 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:38.933307 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:38.959264 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:38.963375 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:39.058840 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:39.059084 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:39.118918 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:39.130278 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:39.139725 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:39.149241 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:39.175433 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:39.177703 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:39.183761 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:39.183835 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:39.280379 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:39.295864 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:39.334631 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:39.338882 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:39.371611 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:39.387552 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:39.511385 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:39.511409 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:39.511414 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:39.511416 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:39.511447 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:39.684604 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:39.684758 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:39.771781 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:39.771882 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:39.809296 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:39.809469 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:39.972517 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:39.974379 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.057171 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:40.062238 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.120675 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:40.120820 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:40.165146 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:40.224974 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:40.233473 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.314341 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:40.323767 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.347851 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:40.348430 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:40.361447 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:40.361503 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:40.363791 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:40.368643 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.379345 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:40.472182 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:40.472300 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:40.502721 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:40.510672 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:40.511493 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:40.511517 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:40.511521 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:40.511523 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:40.511530 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:40.520480 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.529812 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.648608 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:40.659067 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.691286 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:40.698304 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.703158 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:40.703545 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:40.748440 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:40.765654 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.793815 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:40.799408 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.852529 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:40.862847 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.925890 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:40.933394 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.959363 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:40.963469 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:40.972955 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:40.973469 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:41.118990 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:41.130376 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:41.139822 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:41.149327 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:41.175534 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:41.177785 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:41.280467 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:41.295960 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:41.334717 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:41.338985 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:41.371695 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:41.385781 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:41.385865 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:41.387648 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:41.511612 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:41.511635 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:41.511640 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:41.511642 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:41.511665 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:41.775607 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:41.775627 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:41.776342 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:41.776463 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:41.776503 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:41.776557 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:41.776621 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:41.776684 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:41.776723 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:41.776774 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:41.776828 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:41.776873 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:41.776902 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:41.776928 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:41.776953 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:41.776976 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:41.777001 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:41.777024 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:41.777059 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:41.777083 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:41.777107 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:41.777131 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:41.777154 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:41.777181 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:41.785326 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:41.785543 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:41.972610 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:41.974476 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:42.057259 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:42.062328 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:42.110273 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:42.110369 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:42.165243 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:42.189921 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:42.190298 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:42.225061 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:42.233556 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:42.314426 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:42.323864 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:42.363886 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:42.368724 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:42.379427 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:42.421331 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:42.421718 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:42.502820 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:42.510755 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:42.511716 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:42.511734 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:50:42.511738 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:50:42.511741 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:50:42.511752 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:42.511755 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:42.511757 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:42.511769 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:42.520571 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:42.529892 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:42.648632 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:42.649101 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:42.649115 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:42.659165 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:42.691387 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:42.698389 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:42.748538 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:42.765727 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:42.793916 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:42.799504 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:42.852626 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:42.862928 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:42.925973 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:42.933484 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:42.959459 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:42.963565 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:43.119063 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:43.130457 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:43.139906 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:43.149422 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:43.175616 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:43.177864 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:43.280565 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:43.296056 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:43.334809 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:43.339084 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:43.371793 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:43.387741 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:43.511848 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:43.511872 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:43.511876 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:43.511878 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:43.511919 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:43.872634 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:43.872736 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:43.890410 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:43.890900 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:43.967266 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:43.967750 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:43.972700 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:43.974561 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:44.057356 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:44.062414 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:44.165340 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:44.203813 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:44.204225 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:44.225144 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:44.233637 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:44.285917 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:44.286360 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:44.314513 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:44.323958 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:44.363982 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:44.368812 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:44.379521 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:44.391187 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:44.391247 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:44.502893 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:44.510844 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:44.511953 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:44.511977 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:44.511982 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:44.511984 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:44.512006 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:44.520656 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:44.529974 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:44.649191 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:44.659261 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:44.676429 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:44.676448 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:44.677109 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:44.677214 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:44.677271 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:44.677308 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:44.677346 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:44.677383 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:44.677410 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:44.677432 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:44.677474 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:44.677498 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:44.677566 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:44.677641 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:44.677678 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:44.677707 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:44.677731 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:44.677756 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:44.677780 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:44.677804 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:44.677827 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:44.677851 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:44.677879 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:44.677907 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:44.691473 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:44.698471 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:44.748638 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:44.765802 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:44.794002 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:44.799600 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:44.852708 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:44.863023 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:44.926071 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:44.933581 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:44.959423 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:44.959527 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:44.959919 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:44.963660 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:45.010889 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:45.011044 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:45.062489 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:45.062566 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:45.119132 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:45.130539 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:45.140004 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:45.149518 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:45.175714 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:45.177946 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:45.280650 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:45.296153 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:45.334909 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:45.339184 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:45.371891 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:45.387837 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:45.512057 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:45.512091 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:45.512102 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.6 >= max 0.5 = no, load too high
2014-06-12 14:50:45.512116 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:45.512119 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92893 < max (604800 seconds)
2014-06-12 14:50:45.512145 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:45.512148 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:45.512149 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:45.673666 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:45.673907 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:45.885935 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:45.886364 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:45.922166 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:45.922274 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:45.972795 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:45.974643 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:46.057455 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:46.062502 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:46.111745 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:46.111810 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:46.165437 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:46.225225 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:46.233717 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:46.314596 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:46.324053 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:46.364058 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:46.368888 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:46.377183 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:46.377203 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:46.377951 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:46.378009 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:46.378108 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:46.378156 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:46.378239 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:46.378270 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:46.378314 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:46.378411 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:46.378437 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:46.378475 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:46.378498 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:46.378519 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:46.378539 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:46.378566 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:46.378590 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:46.378617 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:46.378640 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:46.378663 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:46.378686 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:46.378710 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:46.378735 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:46.378759 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:46.379594 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:46.502989 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:46.510926 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:46.512223 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:46.512266 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.6 >= max 0.5 = no, load too high
2014-06-12 14:50:46.512280 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:46.512294 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:46.512283 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92894 < max (604800 seconds)
2014-06-12 14:50:46.512304 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:46.512307 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:46.512309 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:46.520737 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:46.530055 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:46.649289 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:46.659357 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:46.691571 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:46.698576 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:46.748736 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:46.765876 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:46.794085 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:46.799696 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:46.852806 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:46.863121 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:46.926165 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:46.933667 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:46.960032 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:46.963757 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:47.119202 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:47.130620 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:47.140089 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:47.149628 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:47.175813 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:47.178032 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:47.186619 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:47.186987 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:47.280735 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:47.296248 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:47.335006 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:47.339285 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:47.371989 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:47.387932 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:47.467999 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:47.468490 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:47.512382 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:47.512401 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:50:47.512405 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:50:47.512409 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:50:47.512440 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.6 >= max 0.5 = no, load too high
2014-06-12 14:50:47.512454 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:47.512461 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:47.512457 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92895 < max (604800 seconds)
2014-06-12 14:50:47.512476 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:47.512479 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:47.512481 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:47.972895 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:47.973467 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:47.973787 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:47.974714 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:48.057557 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:48.062574 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:48.078010 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:48.078030 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:48.078776 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:48.078899 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:48.078963 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:48.079045 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:48.079162 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:48.079226 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:48.079265 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:48.079294 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:48.079321 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:48.079350 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:48.079376 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:48.079402 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:48.079426 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:48.079450 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:48.079485 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:48.079512 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:48.079535 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:48.079562 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:48.079586 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:48.079610 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:48.079635 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:48.079662 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:48.165535 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:48.225308 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:48.233798 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:48.314682 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:48.324148 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:48.364155 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:48.368974 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:48.379686 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:48.460158 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:48.460244 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:48.503089 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:48.511009 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:48.512555 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:48.512578 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:48.512582 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:48.512584 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:48.512625 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:48.520824 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:48.530150 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:48.549363 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:48.549544 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:48.649374 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:48.659456 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:48.691674 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:48.698652 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:48.748835 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:48.765955 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:48.794168 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:48.799792 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:48.852904 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:48.863217 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:48.926248 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:48.933767 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:48.960130 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:48.963853 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:49.092413 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:49.092526 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:49.119270 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:49.130702 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:49.140172 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:49.149727 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:49.168691 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:49.169127 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:49.174437 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:49.174810 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:49.175910 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:49.178114 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:49.280820 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:49.296343 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:49.335104 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:49.339384 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:49.372087 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:49.388026 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:49.422679 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:49.422755 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:49.504497 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:49.504695 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:49.512658 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:49.512683 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:49.512687 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:49.512689 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:49.512728 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:49.674393 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:49.674494 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:49.675359 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:49.675542 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:49.972995 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:49.974811 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:49.986639 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:49.986778 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:50.057657 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:50.062658 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:50.165607 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:50.225391 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:50.233896 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:50.314777 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:50.324244 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:50.364244 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:50.369057 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:50.379781 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:50.503189 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:50.511118 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:50.512767 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:50.512812 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.55 >= max 0.5 = no, load too high
2014-06-12 14:50:50.512826 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:50.512836 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:50.512829 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92898 < max (604800 seconds)
2014-06-12 14:50:50.512848 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:50.512851 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:50.512853 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:50.520910 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:50.530227 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:50.649445 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:50.659536 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:50.691772 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:50.698724 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:50.748933 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:50.766032 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:50.794264 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:50.799888 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:50.849841 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:50.849954 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:50.853003 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:50.863314 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:50.926331 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:50.933856 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:50.960222 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:50.962927 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:50.962978 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:50.963927 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:51.119339 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:51.130794 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:51.140260 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:51.149798 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:51.176009 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:51.178195 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:51.280904 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:51.296424 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:51.335204 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:51.339483 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:51.372187 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:51.388122 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:51.512928 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:51.512951 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:51.512955 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:51.512957 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:51.512997 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:51.578842 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:51.578860 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:51.579711 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:51.579787 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:51.579898 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:51.579954 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:51.579996 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:51.580047 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:51.580078 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:51.580112 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:51.580142 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:51.580166 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:51.580190 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:51.580214 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:51.580238 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:51.580262 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:51.580287 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:51.580310 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:51.580334 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:51.580358 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:51.580381 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:51.580405 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:51.580430 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:51.580450 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:51.973087 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:51.974906 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:52.012598 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:52.012680 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:52.057742 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:52.062750 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:52.165711 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:52.225476 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:52.233977 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:52.314877 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:52.324324 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:52.364344 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:52.369146 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:52.379873 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:52.487760 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:52.487862 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:52.503271 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:52.511205 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:52.513029 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:52.513047 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:50:52.513051 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:50:52.513055 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:50:52.513065 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:52.513069 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:52.513070 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:52.513107 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:52.521001 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:52.530322 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:52.592561 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:52.592959 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:52.649523 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:52.659633 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:52.691871 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:52.698794 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:52.749033 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:52.766110 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:52.794349 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:52.799985 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:52.853102 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:52.863409 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:52.926412 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:52.933943 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:52.960320 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:52.964021 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:53.005234 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:53.005360 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:53.093182 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:53.093270 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:53.119409 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:53.130889 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:53.140358 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:53.149878 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:53.175133 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:53.175242 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:53.176084 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:53.178268 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:53.280988 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:53.296506 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:53.335300 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:53.339584 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:53.372286 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:53.388218 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:53.513145 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:53.513170 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:53.513174 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:53.513176 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:53.513215 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:53.761140 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:53.761203 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:53.776210 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:53.776272 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:53.973176 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:53.975004 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:54.057829 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:54.062837 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:54.165794 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:54.225548 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:54.234074 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:54.314960 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:54.324417 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:54.364444 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:54.369240 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:54.379967 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:54.463512 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:54.463591 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:54.503370 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:54.511291 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:54.513257 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:54.513280 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:54.513285 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:54.513287 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:54.513327 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:54.521088 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:54.530400 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:54.649609 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:54.659727 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:54.687432 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:54.687668 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:54.691971 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:54.698867 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:54.706034 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:54.706174 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:54.723461 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:54.723563 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:54.749107 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:54.766190 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:54.794434 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:54.800079 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:54.853195 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:54.863504 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:54.926526 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:54.934031 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:54.960403 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:54.964116 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:55.069329 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:55.069422 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:55.119478 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:55.130987 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:55.140456 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:55.149960 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:55.176183 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:55.178348 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:55.281075 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:55.296604 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:55.335399 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:55.339683 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:55.372382 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:55.387937 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:55.388005 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:55.388341 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:55.513373 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:55.513397 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:55.513401 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:55.513403 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:55.513444 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:55.550661 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:55.550971 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:55.888846 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:55.889071 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:55.973265 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:55.975101 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:56.057915 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:56.062933 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:56.165869 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:56.225632 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:56.234157 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:56.315059 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:56.324510 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:56.364520 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:56.369335 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:56.380062 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:56.503467 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:56.511376 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:56.513482 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:56.513506 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:56.513510 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:56.513513 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:56.513534 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:56.521175 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:56.530487 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:56.649704 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:56.659824 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:56.662677 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:56.662793 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:56.692071 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:56.698940 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:56.749200 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:56.764570 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:56.764693 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:56.766269 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:56.794534 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:56.800174 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:56.853279 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:56.863599 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:56.926613 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:56.934117 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:56.960486 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:56.964212 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:57.119548 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:57.131083 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:57.140555 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:57.150056 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:57.176280 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:57.178431 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:57.193972 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:57.194080 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:57.275991 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:57.276075 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:57.281158 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:57.296684 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:57.335498 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:57.339783 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:57.372480 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:57.388439 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:57.479806 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:57.479825 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:57.480618 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:57.480662 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:57.480732 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:57.480799 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:57.480834 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:57.480892 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:57.480953 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:57.480982 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:57.481009 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:57.481030 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:57.481061 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:57.481086 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:57.481112 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:57.481137 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:57.481164 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:57.481189 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:57.481211 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:57.481238 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:57.481265 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:57.481289 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:57.481312 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:57.481337 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:57.513602 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:57.513641 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:50:57.513645 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:50:57.513649 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:50:57.513659 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:57.513662 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:57.513664 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:57.513698 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:57.913371 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:57.913564 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:50:57.973351 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:57.975198 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:57.980707 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:57.980726 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:50:57.981579 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:57.981725 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:50:57.981773 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:57.981814 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:57.981901 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:57.981970 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:57.982017 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:57.982063 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:50:57.982109 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:57.982150 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:57.982200 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:57.982244 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:50:57.982296 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:57.982341 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:50:57.982385 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:57.982431 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:50:57.982475 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:57.982519 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:57.982562 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:57.982605 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:57.982646 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:57.982690 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:50:58.057993 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:58.063029 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:58.165962 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:50:58.189453 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:58.189932 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:50:58.224160 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:58.224305 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:58.225719 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:58.234241 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:58.315143 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:50:58.324604 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:58.364619 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:50:58.369432 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:58.380158 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:50:58.503550 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:50:58.511461 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:50:58.513742 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:58.513764 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:50:58.513769 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:58.513771 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:58.513812 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:58.521272 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:58.530568 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:58.649789 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:58.659920 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:58.692167 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:50:58.699008 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:58.749283 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:50:58.766348 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:58.794618 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:50:58.800270 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:58.853363 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:50:58.863696 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:58.926703 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:58.934205 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:58.960579 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:50:58.964308 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:59.077115 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:59.077472 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:50:59.119618 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:50:59.131180 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:59.140654 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:50:59.150128 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:59.176365 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:50:59.178527 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:59.281255 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:50:59.296767 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:59.335582 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:50:59.339882 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:59.372578 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:50:59.388126 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:59.388241 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:50:59.388512 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:50:59.513835 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:50:59.513877 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.51 >= max 0.5 = no, load too high
2014-06-12 14:50:59.513889 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:50:59.513892 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92907 < max (604800 seconds)
2014-06-12 14:50:59.513905 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:50:59.513911 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:50:59.513914 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:50:59.513916 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:50:59.770033 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:59.770181 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:50:59.924961 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:59.925096 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:50:59.973439 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:50:59.975294 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:00.058077 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:00.063125 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:00.162974 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:00.163040 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:00.166062 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:00.177908 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:00.178004 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:00.225821 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:00.234324 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:00.264956 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:00.265040 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:00.281655 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:00.281677 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:00.282385 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:00.282456 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:00.282537 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:00.282568 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:00.282589 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:00.282611 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:00.282662 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:00.282710 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:00.282759 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:00.282790 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:00.282815 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:00.282842 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:00.282893 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:00.282930 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:00.282954 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:00.282981 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:00.283008 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:00.283035 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:00.283059 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:00.283082 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:00.283107 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:00.283134 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:00.315226 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:00.324699 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:00.364714 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:00.369518 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:00.380256 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:00.503648 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:00.511547 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:00.513989 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:00.514011 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:00.514016 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:00.514018 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:00.514058 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:00.521366 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:00.530665 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:00.606753 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:00.606881 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:00.649873 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:00.660000 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:00.663767 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:00.663863 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:00.692269 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:00.699080 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:00.749382 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:00.766429 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:00.794711 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:00.800353 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:00.851392 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:00.851541 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:00.853436 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:00.863791 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:00.926800 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:00.934290 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:00.960661 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:00.964406 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:01.119688 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:01.131276 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:01.140739 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:01.150224 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:01.176447 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:01.178611 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:01.281352 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:01.296847 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:01.335674 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:01.339981 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:01.372677 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:01.388606 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:01.514109 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:01.514133 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:01.514138 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:01.514140 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:01.514179 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:01.878647 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:01.878761 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:01.965549 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:01.965695 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:01.973528 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:01.975388 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:02.058157 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:02.063222 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:02.166145 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:02.225918 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:02.234407 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:02.290140 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:02.290353 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:02.315333 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:02.324796 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:02.364813 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:02.369604 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:02.380350 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:02.494675 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:02.494922 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:02.503747 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:02.511658 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:02.514230 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:02.514252 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:51:02.514258 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:51:02.514264 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:51:02.514278 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:02.514284 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:02.514287 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:02.514317 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:02.521466 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:02.530759 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:02.576818 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:02.576886 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:02.614286 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:02.614408 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:02.649948 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:02.660099 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:02.692369 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:02.699149 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:02.749480 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:02.766508 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:02.794808 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:02.800434 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:02.853517 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:02.863886 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:02.926896 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:02.934383 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:02.960744 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:02.964502 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:03.119758 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:03.131374 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:03.140839 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:03.150306 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:03.176530 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:03.178701 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:03.281451 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:03.296933 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:03.335752 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:03.340080 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:03.372773 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:03.388698 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:03.514362 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:03.514410 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.47 < max 0.5 = yes
2014-06-12 14:51:03.514425 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=1
2014-06-12 14:51:03.514436 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:03.514431 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.16 at 2014-06-11 13:02:32.499677
2014-06-12 14:51:03.514447 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 -> 1 (max 1, active 0)
2014-06-12 14:51:03.514450 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean] sched_scrub: reserved locally, reserving replicas
2014-06-12 14:51:03.514473 7f9d84175700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean] scrub requesting reserve from osd.22
2014-06-12 14:51:03.514503 7f9d84175700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:03.514573 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean] sched_scrub: reserved 2, waiting for replicas
2014-06-12 14:51:03.514591 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:03.514594 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:03.514595 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:03.515498 7f9d58d5b700  5 -- op tracker -- , seq: 748556, time: 2014-06-12 14:51:03.515432, event: header_read, op: osd_sub_op_reply(unknown.0.0:0 3.16 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:03.515515 7f9d58d5b700  5 -- op tracker -- , seq: 748556, time: 2014-06-12 14:51:03.515435, event: throttled, op: osd_sub_op_reply(unknown.0.0:0 3.16 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:03.515525 7f9d58d5b700  5 -- op tracker -- , seq: 748556, time: 2014-06-12 14:51:03.515454, event: all_read, op: osd_sub_op_reply(unknown.0.0:0 3.16 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:03.515560 7f9d58d5b700  5 -- op tracker -- , seq: 748556, time: 0.000000, event: dispatched, op: osd_sub_op_reply(unknown.0.0:0 3.16 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:03.515566 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op_reply(unknown.0.0:0 3.16 0//0//3 [scrub-reserve] ack, result = 0) v2 epoch 173
2014-06-12 14:51:03.515573 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:51:03.515582 7f9d58d5b700 15 osd.2 173 enqueue_op 0x6f4ad00 prio 127 cost 1 latency 0.000149 osd_sub_op_reply(unknown.0.0:0 3.16 0//0//3 [scrub-reserve] ack, result = 0) v2
2014-06-12 14:51:03.515645 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:03.515668 7f9d63bd8700 10 osd.2 173 dequeue_op 0x6f4ad00 prio 127 cost 1 latency 0.000236 osd_sub_op_reply(unknown.0.0:0 3.16 0//0//3 [scrub-reserve] ack, result = 0) v2 pg pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean]
2014-06-12 14:51:03.515705 7f9d63bd8700  5 -- op tracker -- , seq: 748556, time: 2014-06-12 14:51:03.515699, event: reached_pg, op: osd_sub_op_reply(unknown.0.0:0 3.16 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:03.515738 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean] handle_message: 0x6f4ad00
2014-06-12 14:51:03.515755 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean] sub_op_scrub_reserve_reply
2014-06-12 14:51:03.515772 7f9d63bd8700  5 -- op tracker -- , seq: 748556, time: 2014-06-12 14:51:03.515767, event: started, op: osd_sub_op_reply(unknown.0.0:0 3.16 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:03.515802 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean]  osd.22 scrub reserve = success
2014-06-12 14:51:03.515822 7f9d63bd8700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean] sched_scrub: success, reserved self and replicas
2014-06-12 14:51:03.515840 7f9d63bd8700 10 osd.2 173 dequeue_op 0x6f4ad00 finish
2014-06-12 14:51:03.515849 7f9d63bd8700  5 -- op tracker -- , seq: 748556, time: 2014-06-12 14:51:03.515844, event: done, op: osd_sub_op_reply(unknown.0.0:0 3.16 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:03.515873 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:03.515877 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:03.515878 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:03.515898 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] starting a new chunky scrub
2014-06-12 14:51:03.515929 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state INACTIVE
2014-06-12 14:51:03.515944 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub start
2014-06-12 14:51:03.515958 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] publish_stats_to_osd 173:3192
2014-06-12 14:51:03.515971 7f9d60bd2700 20 osd.2 173 inc_scrubs_active 0 -> 1 (max 1, pending 1 -> 0)
2014-06-12 14:51:03.515975 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:03.515991 7f9d60bd2700 10 collection_list_partial 3.16_head start:0//0//-1 is_max:0
2014-06-12 14:51:03.516707 7f9d60bd2700 20 objects: [63a00016/rbd_data.10a96b8b4567.000000000000177f/head//3,42120016/rbd_data.109f6b8b4567.00000000000028c2/head//3,d5920016/rbd_data.108d6b8b4567.0000000000000f81/head//3,3ca40016/rbd_data.10a06b8b4567.0000000000003bea/head//3,95c40016/rbd_data.109e6b8b4567.0000000000001bb1/head//3,47e50016/rbd_data.10926b8b4567.000000000000297a/head//3,7b790016/rbd_data.10ad6b8b4567.0000000000001ade/head//3,3e890016/rbd_data.10a86b8b4567.0000000000002643/head//3,f6990016/rbd_data.109b6b8b4567.0000000000001a9f/head//3,ec321016/rbd_data.10996b8b4567.00000000000018f9/head//3,d8131016/rbd_data.109b6b8b4567.0000000000001733/head//3,7c331016/rbd_data.10ad6b8b4567.0000000000001067/head//3,4f061016/rbd_data.109b6b8b4567.0000000000002252/head//3,a6d71016/rbd_data.10b26b8b4567.0000000000002a60/head//3,c4191016/rbd_data.10956b8b4567.00000000000004f1/head//3,ff302016/rbd_data.10956b8b4567.0000000000002d8d/head//3,32f02016/rbd_data.10a56b8b4567.0000000000003938/head//3,c2112016/rbd_data.10b26b8b4567.00000000000005ad/head//3,c9e12016/rbd_data.10936b8b4567.000000000000348c/head//3,6f922016/rbd_data.10ac6b8b4567.0000000000001a54/head//3,daf52016/rbd_data.10ac6b8b4567.00000000000002f5/head//3,f4b62016/rbd_data.10a76b8b4567.0000000000003264/head//3,b4372016/rbd_data.10ab6b8b4567.0000000000001b59/head//3,207a2016/rbd_data.108d6b8b4567.0000000000002fcf/head//3,90ce2016/rbd_data.10ad6b8b4567.0000000000002a5f/head//3]
2014-06-12 14:51:03.517301 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  requesting scrubmap from osd.22
2014-06-12 14:51:03.517343 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:03.517369 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:03.517383 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:03.517393 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:03.517405 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:03.517416 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_map_chunk [0//0//-1,941f2016//0//-1)
2014-06-12 14:51:03.517429 7f9d60bd2700 10 collection_list_partial 3.16_head start:0//0//-1 is_max:0
2014-06-12 14:51:03.518494 7f9d60bd2700 20 objects: [63a00016/rbd_data.10a96b8b4567.000000000000177f/head//3,42120016/rbd_data.109f6b8b4567.00000000000028c2/head//3,d5920016/rbd_data.108d6b8b4567.0000000000000f81/head//3,3ca40016/rbd_data.10a06b8b4567.0000000000003bea/head//3,95c40016/rbd_data.109e6b8b4567.0000000000001bb1/head//3,47e50016/rbd_data.10926b8b4567.000000000000297a/head//3,7b790016/rbd_data.10ad6b8b4567.0000000000001ade/head//3,3e890016/rbd_data.10a86b8b4567.0000000000002643/head//3,f6990016/rbd_data.109b6b8b4567.0000000000001a9f/head//3,ec321016/rbd_data.10996b8b4567.00000000000018f9/head//3,d8131016/rbd_data.109b6b8b4567.0000000000001733/head//3,7c331016/rbd_data.10ad6b8b4567.0000000000001067/head//3,4f061016/rbd_data.109b6b8b4567.0000000000002252/head//3,a6d71016/rbd_data.10b26b8b4567.0000000000002a60/head//3,c4191016/rbd_data.10956b8b4567.00000000000004f1/head//3,ff302016/rbd_data.10956b8b4567.0000000000002d8d/head//3,32f02016/rbd_data.10a56b8b4567.0000000000003938/head//3,c2112016/rbd_data.10b26b8b4567.00000000000005ad/head//3,c9e12016/rbd_data.10936b8b4567.000000000000348c/head//3,6f922016/rbd_data.10ac6b8b4567.0000000000001a54/head//3,daf52016/rbd_data.10ac6b8b4567.00000000000002f5/head//3,f4b62016/rbd_data.10a76b8b4567.0000000000003264/head//3,b4372016/rbd_data.10ab6b8b4567.0000000000001b59/head//3,207a2016/rbd_data.108d6b8b4567.0000000000002fcf/head//3,90ce2016/rbd_data.10ad6b8b4567.0000000000002a5f/head//3,941f2016/rbd_data.10aa6b8b4567.0000000000000ef2/head//3,1503016/rbd_data.10a76b8b4567.0000000000000c0a/head//3,738c3016/rbd_data.10a66b8b4567.00000000000036b3/head//3,87ce3016/rbd_data.10936b8b4567.00000000000027d1/head//3,44014016/rbd_data.108f6b8b4567.0000000000002610/head//3,d4014016/rbd_data.10b26b8b4567.0000000000000fc1/head//3,42144016/rbd_data.10ab6b8b4567.0000000000002600/head//3,93684016/rbd_data.10a86b8b4567.00000000000019ed/head//3,1d7a4016/rbd_data.10976b8b4567.00000000000012cd/head//3,cb7b4016/rbd_data.108e6b8b4567.0000000000003309/head//3,65cb4016/rbd_data.10b26b8b4567.00000000000009b8/head//3,a3ad4016/rbd_data.10a86b8b4567.00000000000027bb/head//3,2bbf4016/rbd_data.10b26b8b4567.00000000000004c8/head//3,17365016/rbd_data.10b16b8b4567.0000000000001d1e/head//3,3ea85016/rbd_data.10a56b8b4567.0000000000001d85/head//3,2e195016/rbd_data.10ae6b8b4567.000000000000225c/head//3,b3e95016/rbd_data.10946b8b4567.00000000000034ba/head//3,1f2a5016/rbd_data.10a66b8b4567.000000000000039a/head//3,8f9b5016/rbd_data.10a16b8b4567.0000000000001e9a/head//3,9e9d5016/rbd_data.10ae6b8b4567.0000000000001ef3/head//3,381e5016/rbd_data.10af6b8b4567.00000000000026cd/head//3,d7e5016/rbd_data.10b46b8b4567.0000000000002155/head//3,2aaf5016/rbd_data.10ac6b8b4567.0000000000002da0/head//3,23cf5016/rbd_data.10a26b8b4567.0000000000002aa0/head//3,67b06016/rbd_data.10a36b8b4567.000000000000011d/head//3,ff926016/rbd_data.109e6b8b4567.000000000000357f/head//3,81d26016/rbd_data.109d6b8b4567.0000000000002ee8/head//3,5d26016/rbd_data.109b6b8b4567.00000000000001b6/head//3,a2336016/rbd_data.10a86b8b4567.0000000000000b93/head//3,ae636016/rbd_data.10a36b8b4567.0000000000001be7/head//3,78056016/rbd_data.10ab6b8b4567.000000000000196b/head//3,2bb56016/rbd_data.109a6b8b4567.0000000000001729/head//3,3b166016/rbd_data.108f6b8b4567.0000000000002174/head//3,f6ba6016/rbd_data.10a16b8b4567.0000000000001e87/head//3,597b6016/rbd_data.10b06b8b4567.000000000000119f/head//3,1ad6016/rbd_data.10ae6b8b4567.0000000000001333/head//3,467e6016/rbd_data.10b06b8b4567.0000000000002949/head//3,2bd27016/rbd_data.10a36b8b4567.00000000000000b0/head//3,89137016/rbd_data.10a26b8b4567.0000000000001f6f/head//3]
2014-06-12 14:51:03.518642 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:03.518668 7f9d60bd2700 10 stat 3.16_head/63a00016/rbd_data.10a96b8b4567.000000000000177f/head//3
2014-06-12 14:51:03.518907 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 63a00016/rbd_data.10a96b8b4567.000000000000177f/head//3
2014-06-12 14:51:03.572106 7f9d60bd2700 10 getattrs 3.16_head/63a00016/rbd_data.10a96b8b4567.000000000000177f/head//3 = 0
2014-06-12 14:51:03.572114 7f9d60bd2700 10 stat 3.16_head/42120016/rbd_data.109f6b8b4567.00000000000028c2/head//3
2014-06-12 14:51:03.572204 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 42120016/rbd_data.109f6b8b4567.00000000000028c2/head//3
2014-06-12 14:51:03.582670 7f9d60bd2700 10 getattrs 3.16_head/42120016/rbd_data.109f6b8b4567.00000000000028c2/head//3 = 0
2014-06-12 14:51:03.582679 7f9d60bd2700 10 stat 3.16_head/d5920016/rbd_data.108d6b8b4567.0000000000000f81/head//3
2014-06-12 14:51:03.582779 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid d5920016/rbd_data.108d6b8b4567.0000000000000f81/head//3
2014-06-12 14:51:03.595450 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:03.595489 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:03.596591 7f9d60bd2700 10 getattrs 3.16_head/d5920016/rbd_data.108d6b8b4567.0000000000000f81/head//3 = 0
2014-06-12 14:51:03.596601 7f9d60bd2700 10 stat 3.16_head/3ca40016/rbd_data.10a06b8b4567.0000000000003bea/head//3
2014-06-12 14:51:03.596725 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 3ca40016/rbd_data.10a06b8b4567.0000000000003bea/head//3
2014-06-12 14:51:03.616870 7f9d60bd2700 10 getattrs 3.16_head/3ca40016/rbd_data.10a06b8b4567.0000000000003bea/head//3 = 0
2014-06-12 14:51:03.616881 7f9d60bd2700 10 stat 3.16_head/95c40016/rbd_data.109e6b8b4567.0000000000001bb1/head//3
2014-06-12 14:51:03.616999 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 95c40016/rbd_data.109e6b8b4567.0000000000001bb1/head//3
2014-06-12 14:51:03.636693 7f9d60bd2700 10 getattrs 3.16_head/95c40016/rbd_data.109e6b8b4567.0000000000001bb1/head//3 = 0
2014-06-12 14:51:03.636702 7f9d60bd2700 10 stat 3.16_head/47e50016/rbd_data.10926b8b4567.000000000000297a/head//3
2014-06-12 14:51:03.636817 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 47e50016/rbd_data.10926b8b4567.000000000000297a/head//3
2014-06-12 14:51:03.669342 7f9d60bd2700 10 getattrs 3.16_head/47e50016/rbd_data.10926b8b4567.000000000000297a/head//3 = 0
2014-06-12 14:51:03.669353 7f9d60bd2700 10 stat 3.16_head/7b790016/rbd_data.10ad6b8b4567.0000000000001ade/head//3
2014-06-12 14:51:03.669490 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 7b790016/rbd_data.10ad6b8b4567.0000000000001ade/head//3
2014-06-12 14:51:03.689682 7f9d60bd2700 10 getattrs 3.16_head/7b790016/rbd_data.10ad6b8b4567.0000000000001ade/head//3 = 0
2014-06-12 14:51:03.689693 7f9d60bd2700 10 stat 3.16_head/3e890016/rbd_data.10a86b8b4567.0000000000002643/head//3
2014-06-12 14:51:03.689823 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 3e890016/rbd_data.10a86b8b4567.0000000000002643/head//3
2014-06-12 14:51:03.717599 7f9d60bd2700 10 getattrs 3.16_head/3e890016/rbd_data.10a86b8b4567.0000000000002643/head//3 = 0
2014-06-12 14:51:03.717626 7f9d60bd2700 10 stat 3.16_head/f6990016/rbd_data.109b6b8b4567.0000000000001a9f/head//3
2014-06-12 14:51:03.717757 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid f6990016/rbd_data.109b6b8b4567.0000000000001a9f/head//3
2014-06-12 14:51:03.750182 7f9d60bd2700 10 getattrs 3.16_head/f6990016/rbd_data.109b6b8b4567.0000000000001a9f/head//3 = 0
2014-06-12 14:51:03.750193 7f9d60bd2700 10 stat 3.16_head/ec321016/rbd_data.10996b8b4567.00000000000018f9/head//3
2014-06-12 14:51:03.750321 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid ec321016/rbd_data.10996b8b4567.00000000000018f9/head//3
2014-06-12 14:51:03.752307 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:03.752568 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:03.779682 7f9d60bd2700 10 getattrs 3.16_head/ec321016/rbd_data.10996b8b4567.00000000000018f9/head//3 = 0
2014-06-12 14:51:03.779692 7f9d60bd2700 10 stat 3.16_head/d8131016/rbd_data.109b6b8b4567.0000000000001733/head//3
2014-06-12 14:51:03.779804 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid d8131016/rbd_data.109b6b8b4567.0000000000001733/head//3
2014-06-12 14:51:03.800056 7f9d60bd2700 10 getattrs 3.16_head/d8131016/rbd_data.109b6b8b4567.0000000000001733/head//3 = 0
2014-06-12 14:51:03.800066 7f9d60bd2700 10 stat 3.16_head/7c331016/rbd_data.10ad6b8b4567.0000000000001067/head//3
2014-06-12 14:51:03.800176 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 7c331016/rbd_data.10ad6b8b4567.0000000000001067/head//3
2014-06-12 14:51:03.800315 7f9d60bd2700 10 getattrs 3.16_head/7c331016/rbd_data.10ad6b8b4567.0000000000001067/head//3 = 0
2014-06-12 14:51:03.800322 7f9d60bd2700 10 stat 3.16_head/4f061016/rbd_data.109b6b8b4567.0000000000002252/head//3
2014-06-12 14:51:03.800423 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 4f061016/rbd_data.109b6b8b4567.0000000000002252/head//3
2014-06-12 14:51:03.830905 7f9d60bd2700 10 getattrs 3.16_head/4f061016/rbd_data.109b6b8b4567.0000000000002252/head//3 = 0
2014-06-12 14:51:03.830915 7f9d60bd2700 10 stat 3.16_head/a6d71016/rbd_data.10b26b8b4567.0000000000002a60/head//3
2014-06-12 14:51:03.831038 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid a6d71016/rbd_data.10b26b8b4567.0000000000002a60/head//3
2014-06-12 14:51:03.845594 7f9d60bd2700 10 getattrs 3.16_head/a6d71016/rbd_data.10b26b8b4567.0000000000002a60/head//3 = 0
2014-06-12 14:51:03.845605 7f9d60bd2700 10 stat 3.16_head/c4191016/rbd_data.10956b8b4567.00000000000004f1/head//3
2014-06-12 14:51:03.845736 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid c4191016/rbd_data.10956b8b4567.00000000000004f1/head//3
2014-06-12 14:51:03.907158 7f9d60bd2700 10 getattrs 3.16_head/c4191016/rbd_data.10956b8b4567.00000000000004f1/head//3 = 0
2014-06-12 14:51:03.907171 7f9d60bd2700 10 stat 3.16_head/ff302016/rbd_data.10956b8b4567.0000000000002d8d/head//3
2014-06-12 14:51:03.907299 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid ff302016/rbd_data.10956b8b4567.0000000000002d8d/head//3
2014-06-12 14:51:03.941281 7f9d60bd2700 10 getattrs 3.16_head/ff302016/rbd_data.10956b8b4567.0000000000002d8d/head//3 = 0
2014-06-12 14:51:03.941292 7f9d60bd2700 10 stat 3.16_head/32f02016/rbd_data.10a56b8b4567.0000000000003938/head//3
2014-06-12 14:51:03.941416 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 32f02016/rbd_data.10a56b8b4567.0000000000003938/head//3
2014-06-12 14:51:03.965637 7f9d60bd2700 10 getattrs 3.16_head/32f02016/rbd_data.10a56b8b4567.0000000000003938/head//3 = 0
2014-06-12 14:51:03.965663 7f9d60bd2700 10 stat 3.16_head/c2112016/rbd_data.10b26b8b4567.00000000000005ad/head//3
2014-06-12 14:51:03.965811 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid c2112016/rbd_data.10b26b8b4567.00000000000005ad/head//3
2014-06-12 14:51:03.975470 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:03.991011 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:03.991218 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:04.011406 7f9d60bd2700 10 getattrs 3.16_head/c2112016/rbd_data.10b26b8b4567.00000000000005ad/head//3 = 0
2014-06-12 14:51:04.011417 7f9d60bd2700 10 stat 3.16_head/c9e12016/rbd_data.10936b8b4567.000000000000348c/head//3
2014-06-12 14:51:04.011545 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid c9e12016/rbd_data.10936b8b4567.000000000000348c/head//3
2014-06-12 14:51:04.029466 7f9d60bd2700 10 getattrs 3.16_head/c9e12016/rbd_data.10936b8b4567.000000000000348c/head//3 = 0
2014-06-12 14:51:04.029475 7f9d60bd2700 10 stat 3.16_head/6f922016/rbd_data.10ac6b8b4567.0000000000001a54/head//3
2014-06-12 14:51:04.029574 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 6f922016/rbd_data.10ac6b8b4567.0000000000001a54/head//3
2014-06-12 14:51:04.053871 7f9d58d5b700  5 -- op tracker -- , seq: 748557, time: 2014-06-12 14:51:04.053786, event: header_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.053889 7f9d58d5b700  5 -- op tracker -- , seq: 748557, time: 2014-06-12 14:51:04.053788, event: throttled, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.053902 7f9d58d5b700  5 -- op tracker -- , seq: 748557, time: 2014-06-12 14:51:04.053829, event: all_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.053907 7f9d58d5b700  5 -- op tracker -- , seq: 748557, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.053911 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:04.053915 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:51:04.053921 7f9d58d5b700 15 osd.2 173 enqueue_op 0x6f4a600 prio 127 cost 10717 latency 0.000135 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:04.053987 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:04.063300 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:04.064059 7f9d60bd2700 10 getattrs 3.16_head/6f922016/rbd_data.10ac6b8b4567.0000000000001a54/head//3 = 0
2014-06-12 14:51:04.064073 7f9d60bd2700 10 stat 3.16_head/daf52016/rbd_data.10ac6b8b4567.00000000000002f5/head//3
2014-06-12 14:51:04.064207 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid daf52016/rbd_data.10ac6b8b4567.00000000000002f5/head//3
2014-06-12 14:51:04.101947 7f9d60bd2700 10 getattrs 3.16_head/daf52016/rbd_data.10ac6b8b4567.00000000000002f5/head//3 = 0
2014-06-12 14:51:04.101960 7f9d60bd2700 10 stat 3.16_head/f4b62016/rbd_data.10a76b8b4567.0000000000003264/head//3
2014-06-12 14:51:04.102064 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid f4b62016/rbd_data.10a76b8b4567.0000000000003264/head//3
2014-06-12 14:51:04.124293 7f9d60bd2700 10 getattrs 3.16_head/f4b62016/rbd_data.10a76b8b4567.0000000000003264/head//3 = 0
2014-06-12 14:51:04.124302 7f9d60bd2700 10 stat 3.16_head/b4372016/rbd_data.10ab6b8b4567.0000000000001b59/head//3
2014-06-12 14:51:04.124410 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid b4372016/rbd_data.10ab6b8b4567.0000000000001b59/head//3
2014-06-12 14:51:04.145134 7f9d60bd2700 10 getattrs 3.16_head/b4372016/rbd_data.10ab6b8b4567.0000000000001b59/head//3 = 0
2014-06-12 14:51:04.145149 7f9d60bd2700 10 stat 3.16_head/207a2016/rbd_data.108d6b8b4567.0000000000002fcf/head//3
2014-06-12 14:51:04.145253 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 207a2016/rbd_data.108d6b8b4567.0000000000002fcf/head//3
2014-06-12 14:51:04.162754 7f9d60bd2700 10 getattrs 3.16_head/207a2016/rbd_data.108d6b8b4567.0000000000002fcf/head//3 = 0
2014-06-12 14:51:04.162762 7f9d60bd2700 10 stat 3.16_head/90ce2016/rbd_data.10ad6b8b4567.0000000000002a5f/head//3
2014-06-12 14:51:04.162865 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 90ce2016/rbd_data.10ad6b8b4567.0000000000002a5f/head//3
2014-06-12 14:51:04.187146 7f9d60bd2700 10 getattrs 3.16_head/90ce2016/rbd_data.10ad6b8b4567.0000000000002a5f/head//3 = 0
2014-06-12 14:51:04.187157 7f9d60bd2700 10 collection_getattrs 3.16_head
2014-06-12 14:51:04.210070 7f9d60bd2700 10 collection_getattrs 3.16_head = 0
2014-06-12 14:51:04.210078 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:04.210135 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:04.210142 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:04.210151 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:04.210154 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:04.210160 7f9d61bd4700 10 osd.2 173 dequeue_op 0x6f4a600 prio 127 cost 10717 latency 0.156373 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]
2014-06-12 14:51:04.210196 7f9d61bd4700  5 -- op tracker -- , seq: 748557, time: 2014-06-12 14:51:04.210192, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.210207 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] handle_message: 0x6f4a600
2014-06-12 14:51:04.210222 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:04.210239 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:04.210253 7f9d61bd4700  5 -- op tracker -- , seq: 748557, time: 2014-06-12 14:51:04.210250, event: started, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.210259 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  got 22 scrub map
2014-06-12 14:51:04.210319 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] map version is 173'2396
2014-06-12 14:51:04.210331 7f9d61bd4700 10 osd.2 173 dequeue_op 0x6f4a600 finish
2014-06-12 14:51:04.210339 7f9d61bd4700  5 -- op tracker -- , seq: 748557, time: 2014-06-12 14:51:04.210335, event: done, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.210377 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:04.210382 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:04.210411 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:04.210418 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:04.210432 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:04.210450 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:04.210471 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:04.210479 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:04.210485 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub replica 22 has 25 items
2014-06-12 14:51:04.213751 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 63a00016/rbd_data.10a96b8b4567.000000000000177f/head//3, auth == maps.end()
2014-06-12 14:51:04.213774 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 63a00016/rbd_data.10a96b8b4567.000000000000177f/head//3
2014-06-12 14:51:04.213796 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 42120016/rbd_data.109f6b8b4567.00000000000028c2/head//3, auth == maps.end()
2014-06-12 14:51:04.213805 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 42120016/rbd_data.109f6b8b4567.00000000000028c2/head//3
2014-06-12 14:51:04.213815 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d5920016/rbd_data.108d6b8b4567.0000000000000f81/head//3, auth == maps.end()
2014-06-12 14:51:04.213823 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj d5920016/rbd_data.108d6b8b4567.0000000000000f81/head//3
2014-06-12 14:51:04.213831 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3ca40016/rbd_data.10a06b8b4567.0000000000003bea/head//3, auth == maps.end()
2014-06-12 14:51:04.213838 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 3ca40016/rbd_data.10a06b8b4567.0000000000003bea/head//3
2014-06-12 14:51:04.213846 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 95c40016/rbd_data.109e6b8b4567.0000000000001bb1/head//3, auth == maps.end()
2014-06-12 14:51:04.213853 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 95c40016/rbd_data.109e6b8b4567.0000000000001bb1/head//3
2014-06-12 14:51:04.213861 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 47e50016/rbd_data.10926b8b4567.000000000000297a/head//3, auth == maps.end()
2014-06-12 14:51:04.213868 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 47e50016/rbd_data.10926b8b4567.000000000000297a/head//3
2014-06-12 14:51:04.213875 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7b790016/rbd_data.10ad6b8b4567.0000000000001ade/head//3, auth == maps.end()
2014-06-12 14:51:04.213883 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 7b790016/rbd_data.10ad6b8b4567.0000000000001ade/head//3
2014-06-12 14:51:04.213891 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3e890016/rbd_data.10a86b8b4567.0000000000002643/head//3, auth == maps.end()
2014-06-12 14:51:04.213897 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 3e890016/rbd_data.10a86b8b4567.0000000000002643/head//3
2014-06-12 14:51:04.213905 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f6990016/rbd_data.109b6b8b4567.0000000000001a9f/head//3, auth == maps.end()
2014-06-12 14:51:04.213916 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj f6990016/rbd_data.109b6b8b4567.0000000000001a9f/head//3
2014-06-12 14:51:04.213927 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ec321016/rbd_data.10996b8b4567.00000000000018f9/head//3, auth == maps.end()
2014-06-12 14:51:04.213936 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj ec321016/rbd_data.10996b8b4567.00000000000018f9/head//3
2014-06-12 14:51:04.213944 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d8131016/rbd_data.109b6b8b4567.0000000000001733/head//3, auth == maps.end()
2014-06-12 14:51:04.213954 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj d8131016/rbd_data.109b6b8b4567.0000000000001733/head//3
2014-06-12 14:51:04.213962 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7c331016/rbd_data.10ad6b8b4567.0000000000001067/head//3, auth == maps.end()
2014-06-12 14:51:04.213971 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 7c331016/rbd_data.10ad6b8b4567.0000000000001067/head//3
2014-06-12 14:51:04.213980 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4f061016/rbd_data.109b6b8b4567.0000000000002252/head//3, auth == maps.end()
2014-06-12 14:51:04.213989 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 4f061016/rbd_data.109b6b8b4567.0000000000002252/head//3
2014-06-12 14:51:04.213997 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a6d71016/rbd_data.10b26b8b4567.0000000000002a60/head//3, auth == maps.end()
2014-06-12 14:51:04.214005 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj a6d71016/rbd_data.10b26b8b4567.0000000000002a60/head//3
2014-06-12 14:51:04.214013 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c4191016/rbd_data.10956b8b4567.00000000000004f1/head//3, auth == maps.end()
2014-06-12 14:51:04.214022 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj c4191016/rbd_data.10956b8b4567.00000000000004f1/head//3
2014-06-12 14:51:04.214029 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ff302016/rbd_data.10956b8b4567.0000000000002d8d/head//3, auth == maps.end()
2014-06-12 14:51:04.214037 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj ff302016/rbd_data.10956b8b4567.0000000000002d8d/head//3
2014-06-12 14:51:04.214045 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 32f02016/rbd_data.10a56b8b4567.0000000000003938/head//3, auth == maps.end()
2014-06-12 14:51:04.214053 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 32f02016/rbd_data.10a56b8b4567.0000000000003938/head//3
2014-06-12 14:51:04.214062 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c2112016/rbd_data.10b26b8b4567.00000000000005ad/head//3, auth == maps.end()
2014-06-12 14:51:04.214069 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj c2112016/rbd_data.10b26b8b4567.00000000000005ad/head//3
2014-06-12 14:51:04.214078 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c9e12016/rbd_data.10936b8b4567.000000000000348c/head//3, auth == maps.end()
2014-06-12 14:51:04.214086 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj c9e12016/rbd_data.10936b8b4567.000000000000348c/head//3
2014-06-12 14:51:04.214095 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6f922016/rbd_data.10ac6b8b4567.0000000000001a54/head//3, auth == maps.end()
2014-06-12 14:51:04.214102 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 6f922016/rbd_data.10ac6b8b4567.0000000000001a54/head//3
2014-06-12 14:51:04.214111 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj daf52016/rbd_data.10ac6b8b4567.00000000000002f5/head//3, auth == maps.end()
2014-06-12 14:51:04.214119 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj daf52016/rbd_data.10ac6b8b4567.00000000000002f5/head//3
2014-06-12 14:51:04.214128 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f4b62016/rbd_data.10a76b8b4567.0000000000003264/head//3, auth == maps.end()
2014-06-12 14:51:04.214136 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj f4b62016/rbd_data.10a76b8b4567.0000000000003264/head//3
2014-06-12 14:51:04.214145 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b4372016/rbd_data.10ab6b8b4567.0000000000001b59/head//3, auth == maps.end()
2014-06-12 14:51:04.214152 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj b4372016/rbd_data.10ab6b8b4567.0000000000001b59/head//3
2014-06-12 14:51:04.214160 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 207a2016/rbd_data.108d6b8b4567.0000000000002fcf/head//3, auth == maps.end()
2014-06-12 14:51:04.214166 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 207a2016/rbd_data.108d6b8b4567.0000000000002fcf/head//3
2014-06-12 14:51:04.214174 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 90ce2016/rbd_data.10ad6b8b4567.0000000000002a5f/head//3, auth == maps.end()
2014-06-12 14:51:04.214181 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 90ce2016/rbd_data.10ad6b8b4567.0000000000002a5f/head//3
2014-06-12 14:51:04.214189 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] 
2014-06-12 14:51:04.214196 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub
2014-06-12 14:51:04.214207 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  90ce2016/rbd_data.10ad6b8b4567.0000000000002a5f/head//3 90ce2016/rbd_data.10ad6b8b4567.0000000000002a5f/head//3(173'1513 client.4310.0:85814 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1513)
2014-06-12 14:51:04.214219 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  207a2016/rbd_data.108d6b8b4567.0000000000002fcf/head//3 207a2016/rbd_data.108d6b8b4567.0000000000002fcf/head//3(173'1627 client.4278.0:96541 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1627)
2014-06-12 14:51:04.214230 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  b4372016/rbd_data.10ab6b8b4567.0000000000001b59/head//3 b4372016/rbd_data.10ab6b8b4567.0000000000001b59/head//3(173'1021 client.4308.0:59532 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1021)
2014-06-12 14:51:04.214242 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  f4b62016/rbd_data.10a76b8b4567.0000000000003264/head//3 f4b62016/rbd_data.10a76b8b4567.0000000000003264/head//3(173'1773 client.4304.0:100178 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1773)
2014-06-12 14:51:04.214253 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  daf52016/rbd_data.10ac6b8b4567.00000000000002f5/head//3 daf52016/rbd_data.10ac6b8b4567.00000000000002f5/head//3(173'2087 client.4309.0:120519 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2087)
2014-06-12 14:51:04.214263 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  6f922016/rbd_data.10ac6b8b4567.0000000000001a54/head//3 6f922016/rbd_data.10ac6b8b4567.0000000000001a54/head//3(173'962 client.4309.0:56330 wrlock_by=unknown.0.0:0 dirty s 4194304 uv962)
2014-06-12 14:51:04.214272 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  c9e12016/rbd_data.10936b8b4567.000000000000348c/head//3 c9e12016/rbd_data.10936b8b4567.000000000000348c/head//3(173'1828 client.4284.0:102005 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1828)
2014-06-12 14:51:04.214282 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  c2112016/rbd_data.10b26b8b4567.00000000000005ad/head//3 c2112016/rbd_data.10b26b8b4567.00000000000005ad/head//3(173'2193 client.4315.0:126345 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2193)
2014-06-12 14:51:04.214293 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  32f02016/rbd_data.10a56b8b4567.0000000000003938/head//3 32f02016/rbd_data.10a56b8b4567.0000000000003938/head//3(173'1964 client.4302.0:114532 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1964)
2014-06-12 14:51:04.214303 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  ff302016/rbd_data.10956b8b4567.0000000000002d8d/head//3 ff302016/rbd_data.10956b8b4567.0000000000002d8d/head//3(173'1636 client.4286.0:91855 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1636)
2014-06-12 14:51:04.214314 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  c4191016/rbd_data.10956b8b4567.00000000000004f1/head//3 c4191016/rbd_data.10956b8b4567.00000000000004f1/head//3(173'2126 client.4286.0:125398 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2126)
2014-06-12 14:51:04.214324 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  a6d71016/rbd_data.10b26b8b4567.0000000000002a60/head//3 a6d71016/rbd_data.10b26b8b4567.0000000000002a60/head//3(173'1421 client.4315.0:86161 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1421)
2014-06-12 14:51:04.214334 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  4f061016/rbd_data.109b6b8b4567.0000000000002252/head//3 4f061016/rbd_data.109b6b8b4567.0000000000002252/head//3(173'1291 client.4292.0:68994 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1291)
2014-06-12 14:51:04.214344 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  7c331016/rbd_data.10ad6b8b4567.0000000000001067/head//3 7c331016/rbd_data.10ad6b8b4567.0000000000001067/head//3(173'626 client.4310.0:38725 wrlock_by=unknown.0.0:0 dirty s 4194304 uv626)
2014-06-12 14:51:04.214357 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  d8131016/rbd_data.109b6b8b4567.0000000000001733/head//3 d8131016/rbd_data.109b6b8b4567.0000000000001733/head//3(173'937 client.4292.0:49957 wrlock_by=unknown.0.0:0 dirty s 4194304 uv937)
2014-06-12 14:51:04.214367 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  ec321016/rbd_data.10996b8b4567.00000000000018f9/head//3 ec321016/rbd_data.10996b8b4567.00000000000018f9/head//3(173'811 client.4290.0:53660 wrlock_by=unknown.0.0:0 dirty s 4194304 uv811)
2014-06-12 14:51:04.214377 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  f6990016/rbd_data.109b6b8b4567.0000000000001a9f/head//3 f6990016/rbd_data.109b6b8b4567.0000000000001a9f/head//3(173'1089 client.4292.0:55998 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1089)
2014-06-12 14:51:04.214387 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  3e890016/rbd_data.10a86b8b4567.0000000000002643/head//3 3e890016/rbd_data.10a86b8b4567.0000000000002643/head//3(173'1312 client.4305.0:77652 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1312)
2014-06-12 14:51:04.214397 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  7b790016/rbd_data.10ad6b8b4567.0000000000001ade/head//3 7b790016/rbd_data.10ad6b8b4567.0000000000001ade/head//3(173'1012 client.4310.0:57935 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1012)
2014-06-12 14:51:04.214407 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  47e50016/rbd_data.10926b8b4567.000000000000297a/head//3 47e50016/rbd_data.10926b8b4567.000000000000297a/head//3(173'1453 client.4283.0:83509 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1453)
2014-06-12 14:51:04.214418 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  95c40016/rbd_data.109e6b8b4567.0000000000001bb1/head//3 95c40016/rbd_data.109e6b8b4567.0000000000001bb1/head//3(173'1026 client.4295.0:60162 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1026)
2014-06-12 14:51:04.214428 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  3ca40016/rbd_data.10a06b8b4567.0000000000003bea/head//3 3ca40016/rbd_data.10a06b8b4567.0000000000003bea/head//3(173'1971 client.4297.0:116910 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1971)
2014-06-12 14:51:04.214438 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  d5920016/rbd_data.108d6b8b4567.0000000000000f81/head//3 d5920016/rbd_data.108d6b8b4567.0000000000000f81/head//3(173'2385 client.4278.0:143365 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2385)
2014-06-12 14:51:04.214448 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  42120016/rbd_data.109f6b8b4567.00000000000028c2/head//3 42120016/rbd_data.109f6b8b4567.00000000000028c2/head//3(173'1307 client.4296.0:80475 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1307)
2014-06-12 14:51:04.214458 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  63a00016/rbd_data.10a96b8b4567.000000000000177f/head//3 63a00016/rbd_data.10a96b8b4567.000000000000177f/head//3(173'942 client.4306.0:53603 wrlock_by=unknown.0.0:0 dirty s 4194304 uv942)
2014-06-12 14:51:04.214466 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:04.214473 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:04.214481 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:04.214482 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:04.214484 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:04.214517 7f9d60bd2700 10 collection_list_partial 3.16_head start:941f2016//0//-1 is_max:0
2014-06-12 14:51:04.214718 7f9d60bd2700 20 objects: [1503016/rbd_data.10a76b8b4567.0000000000000c0a/head//3,738c3016/rbd_data.10a66b8b4567.00000000000036b3/head//3,87ce3016/rbd_data.10936b8b4567.00000000000027d1/head//3,44014016/rbd_data.108f6b8b4567.0000000000002610/head//3,d4014016/rbd_data.10b26b8b4567.0000000000000fc1/head//3,42144016/rbd_data.10ab6b8b4567.0000000000002600/head//3,93684016/rbd_data.10a86b8b4567.00000000000019ed/head//3,1d7a4016/rbd_data.10976b8b4567.00000000000012cd/head//3,cb7b4016/rbd_data.108e6b8b4567.0000000000003309/head//3,65cb4016/rbd_data.10b26b8b4567.00000000000009b8/head//3,a3ad4016/rbd_data.10a86b8b4567.00000000000027bb/head//3,2bbf4016/rbd_data.10b26b8b4567.00000000000004c8/head//3,17365016/rbd_data.10b16b8b4567.0000000000001d1e/head//3,3ea85016/rbd_data.10a56b8b4567.0000000000001d85/head//3,2e195016/rbd_data.10ae6b8b4567.000000000000225c/head//3,b3e95016/rbd_data.10946b8b4567.00000000000034ba/head//3,1f2a5016/rbd_data.10a66b8b4567.000000000000039a/head//3,8f9b5016/rbd_data.10a16b8b4567.0000000000001e9a/head//3,9e9d5016/rbd_data.10ae6b8b4567.0000000000001ef3/head//3,381e5016/rbd_data.10af6b8b4567.00000000000026cd/head//3,d7e5016/rbd_data.10b46b8b4567.0000000000002155/head//3,2aaf5016/rbd_data.10ac6b8b4567.0000000000002da0/head//3,23cf5016/rbd_data.10a26b8b4567.0000000000002aa0/head//3,67b06016/rbd_data.10a36b8b4567.000000000000011d/head//3,ff926016/rbd_data.109e6b8b4567.000000000000357f/head//3]
2014-06-12 14:51:04.215134 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  requesting scrubmap from osd.22
2014-06-12 14:51:04.215155 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:04.215171 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:04.215180 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:04.215185 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:04.215190 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:04.215194 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_map_chunk [941f2016//0//-1,81d26016//0//-1)
2014-06-12 14:51:04.215200 7f9d60bd2700 10 collection_list_partial 3.16_head start:941f2016//0//-1 is_max:0
2014-06-12 14:51:04.215615 7f9d60bd2700 20 objects: [1503016/rbd_data.10a76b8b4567.0000000000000c0a/head//3,738c3016/rbd_data.10a66b8b4567.00000000000036b3/head//3,87ce3016/rbd_data.10936b8b4567.00000000000027d1/head//3,44014016/rbd_data.108f6b8b4567.0000000000002610/head//3,d4014016/rbd_data.10b26b8b4567.0000000000000fc1/head//3,42144016/rbd_data.10ab6b8b4567.0000000000002600/head//3,93684016/rbd_data.10a86b8b4567.00000000000019ed/head//3,1d7a4016/rbd_data.10976b8b4567.00000000000012cd/head//3,cb7b4016/rbd_data.108e6b8b4567.0000000000003309/head//3,65cb4016/rbd_data.10b26b8b4567.00000000000009b8/head//3,a3ad4016/rbd_data.10a86b8b4567.00000000000027bb/head//3,2bbf4016/rbd_data.10b26b8b4567.00000000000004c8/head//3,17365016/rbd_data.10b16b8b4567.0000000000001d1e/head//3,3ea85016/rbd_data.10a56b8b4567.0000000000001d85/head//3,2e195016/rbd_data.10ae6b8b4567.000000000000225c/head//3,b3e95016/rbd_data.10946b8b4567.00000000000034ba/head//3,1f2a5016/rbd_data.10a66b8b4567.000000000000039a/head//3,8f9b5016/rbd_data.10a16b8b4567.0000000000001e9a/head//3,9e9d5016/rbd_data.10ae6b8b4567.0000000000001ef3/head//3,381e5016/rbd_data.10af6b8b4567.00000000000026cd/head//3,d7e5016/rbd_data.10b46b8b4567.0000000000002155/head//3,2aaf5016/rbd_data.10ac6b8b4567.0000000000002da0/head//3,23cf5016/rbd_data.10a26b8b4567.0000000000002aa0/head//3,67b06016/rbd_data.10a36b8b4567.000000000000011d/head//3,ff926016/rbd_data.109e6b8b4567.000000000000357f/head//3,81d26016/rbd_data.109d6b8b4567.0000000000002ee8/head//3,5d26016/rbd_data.109b6b8b4567.00000000000001b6/head//3,a2336016/rbd_data.10a86b8b4567.0000000000000b93/head//3,ae636016/rbd_data.10a36b8b4567.0000000000001be7/head//3,78056016/rbd_data.10ab6b8b4567.000000000000196b/head//3,2bb56016/rbd_data.109a6b8b4567.0000000000001729/head//3,3b166016/rbd_data.108f6b8b4567.0000000000002174/head//3,f6ba6016/rbd_data.10a16b8b4567.0000000000001e87/head//3,597b6016/rbd_data.10b06b8b4567.000000000000119f/head//3,1ad6016/rbd_data.10ae6b8b4567.0000000000001333/head//3,467e6016/rbd_data.10b06b8b4567.0000000000002949/head//3,2bd27016/rbd_data.10a36b8b4567.00000000000000b0/head//3,89137016/rbd_data.10a26b8b4567.0000000000001f6f/head//3,8b437016/rbd_data.10966b8b4567.00000000000036cf/head//3,78777016/rbd_data.109e6b8b4567.0000000000001fd3/head//3,8987016/rbd_data.10a96b8b4567.0000000000000585/head//3,c00a7016/rbd_data.108d6b8b4567.000000000000067e/head//3,cada7016/rbd_data.109c6b8b4567.0000000000001b2b/head//3,1c818016/rbd_data.10936b8b4567.0000000000002e8d/head//3,f1a18016/rbd_data.10b16b8b4567.00000000000000a6/head//3,2ad38016/rbd_data.10a76b8b4567.0000000000000be2/head//3,9ae48016/rbd_data.109c6b8b4567.00000000000027a5/head//3,1a778016/rbd_data.109e6b8b4567.0000000000001c28/head//3,6488016/rbd_data.109e6b8b4567.0000000000002c14/head//3,8abc8016/rbd_data.10af6b8b4567.0000000000001b0a/head//3,e19d8016/rbd_data.10a56b8b4567.0000000000002ac3/head//3,47279016/rbd_data.10a26b8b4567.00000000000008c1/head//3,8bb79016/rbd_data.108d6b8b4567.0000000000000ef0/head//3,917b9016/rbd_data.10906b8b4567.00000000000013ea/head//3,7d1e9016/rbd_data.10ad6b8b4567.000000000000304b/head//3,a7fe9016/rbd_data.10ac6b8b4567.0000000000003685/head//3,835f9016/rbd_data.109e6b8b4567.0000000000001771/head//3,69f1a016/rbd_data.10ac6b8b4567.000000000000382f/head//3,d0d2a016/rbd_data.10a66b8b4567.0000000000002934/head//3,9f44a016/rbd_data.109b6b8b4567.0000000000000721/head//3,6b86a016/rbd_data.10916b8b4567.00000000000014aa/head//3,31a9a016/rbd_data.10a86b8b4567.0000000000002f08/head//3,215aa016/rbd_data.109a6b8b4567.0000000000001af5/head//3,b5cca016/rbd_data.10a96b8b4567.000000000000115a/head//3]
2014-06-12 14:51:04.215682 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:04.215695 7f9d60bd2700 10 stat 3.16_head/1503016/rbd_data.10a76b8b4567.0000000000000c0a/head//3
2014-06-12 14:51:04.215796 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 1503016/rbd_data.10a76b8b4567.0000000000000c0a/head//3
2014-06-12 14:51:04.225990 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:04.234478 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:04.264732 7f9d60bd2700 10 getattrs 3.16_head/1503016/rbd_data.10a76b8b4567.0000000000000c0a/head//3 = 0
2014-06-12 14:51:04.264747 7f9d60bd2700 10 stat 3.16_head/738c3016/rbd_data.10a66b8b4567.00000000000036b3/head//3
2014-06-12 14:51:04.264876 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 738c3016/rbd_data.10a66b8b4567.00000000000036b3/head//3
2014-06-12 14:51:04.277719 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:04.277746 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:04.295639 7f9d60bd2700 10 getattrs 3.16_head/738c3016/rbd_data.10a66b8b4567.00000000000036b3/head//3 = 0
2014-06-12 14:51:04.295646 7f9d60bd2700 10 stat 3.16_head/87ce3016/rbd_data.10936b8b4567.00000000000027d1/head//3
2014-06-12 14:51:04.295756 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 87ce3016/rbd_data.10936b8b4567.00000000000027d1/head//3
2014-06-12 14:51:04.315411 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:04.316236 7f9d60bd2700 10 getattrs 3.16_head/87ce3016/rbd_data.10936b8b4567.00000000000027d1/head//3 = 0
2014-06-12 14:51:04.316253 7f9d60bd2700 10 stat 3.16_head/44014016/rbd_data.108f6b8b4567.0000000000002610/head//3
2014-06-12 14:51:04.316359 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 44014016/rbd_data.108f6b8b4567.0000000000002610/head//3
2014-06-12 14:51:04.324878 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:04.343746 7f9d60bd2700 10 getattrs 3.16_head/44014016/rbd_data.108f6b8b4567.0000000000002610/head//3 = 0
2014-06-12 14:51:04.343758 7f9d60bd2700 10 stat 3.16_head/d4014016/rbd_data.10b26b8b4567.0000000000000fc1/head//3
2014-06-12 14:51:04.343877 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid d4014016/rbd_data.10b26b8b4567.0000000000000fc1/head//3
2014-06-12 14:51:04.344036 7f9d60bd2700 10 getattrs 3.16_head/d4014016/rbd_data.10b26b8b4567.0000000000000fc1/head//3 = 0
2014-06-12 14:51:04.344046 7f9d60bd2700 10 stat 3.16_head/42144016/rbd_data.10ab6b8b4567.0000000000002600/head//3
2014-06-12 14:51:04.344143 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 42144016/rbd_data.10ab6b8b4567.0000000000002600/head//3
2014-06-12 14:51:04.359947 7f9d60bd2700 10 getattrs 3.16_head/42144016/rbd_data.10ab6b8b4567.0000000000002600/head//3 = 0
2014-06-12 14:51:04.359956 7f9d60bd2700 10 stat 3.16_head/93684016/rbd_data.10a86b8b4567.00000000000019ed/head//3
2014-06-12 14:51:04.360062 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 93684016/rbd_data.10a86b8b4567.00000000000019ed/head//3
2014-06-12 14:51:04.364890 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:04.369701 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:04.380431 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:04.385066 7f9d60bd2700 10 getattrs 3.16_head/93684016/rbd_data.10a86b8b4567.00000000000019ed/head//3 = 0
2014-06-12 14:51:04.385075 7f9d60bd2700 10 stat 3.16_head/1d7a4016/rbd_data.10976b8b4567.00000000000012cd/head//3
2014-06-12 14:51:04.385185 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 1d7a4016/rbd_data.10976b8b4567.00000000000012cd/head//3
2014-06-12 14:51:04.416507 7f9d60bd2700 10 getattrs 3.16_head/1d7a4016/rbd_data.10976b8b4567.00000000000012cd/head//3 = 0
2014-06-12 14:51:04.416515 7f9d60bd2700 10 stat 3.16_head/cb7b4016/rbd_data.108e6b8b4567.0000000000003309/head//3
2014-06-12 14:51:04.416615 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid cb7b4016/rbd_data.108e6b8b4567.0000000000003309/head//3
2014-06-12 14:51:04.434688 7f9d60bd2700 10 getattrs 3.16_head/cb7b4016/rbd_data.108e6b8b4567.0000000000003309/head//3 = 0
2014-06-12 14:51:04.434698 7f9d60bd2700 10 stat 3.16_head/65cb4016/rbd_data.10b26b8b4567.00000000000009b8/head//3
2014-06-12 14:51:04.434802 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 65cb4016/rbd_data.10b26b8b4567.00000000000009b8/head//3
2014-06-12 14:51:04.469107 7f9d60bd2700 10 getattrs 3.16_head/65cb4016/rbd_data.10b26b8b4567.00000000000009b8/head//3 = 0
2014-06-12 14:51:04.469146 7f9d60bd2700 10 stat 3.16_head/a3ad4016/rbd_data.10a86b8b4567.00000000000027bb/head//3
2014-06-12 14:51:04.469271 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid a3ad4016/rbd_data.10a86b8b4567.00000000000027bb/head//3
2014-06-12 14:51:04.491154 7f9d60bd2700 10 getattrs 3.16_head/a3ad4016/rbd_data.10a86b8b4567.00000000000027bb/head//3 = 0
2014-06-12 14:51:04.491161 7f9d60bd2700 10 stat 3.16_head/2bbf4016/rbd_data.10b26b8b4567.00000000000004c8/head//3
2014-06-12 14:51:04.491263 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 2bbf4016/rbd_data.10b26b8b4567.00000000000004c8/head//3
2014-06-12 14:51:04.503832 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:04.511733 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:04.514662 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:04.514689 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:04.514693 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.47 < max 0.5 = yes
2014-06-12 14:51:04.514702 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=1
2014-06-12 14:51:04.521535 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:04.521603 7f9d60bd2700 10 getattrs 3.16_head/2bbf4016/rbd_data.10b26b8b4567.00000000000004c8/head//3 = 0
2014-06-12 14:51:04.521643 7f9d60bd2700 10 stat 3.16_head/17365016/rbd_data.10b16b8b4567.0000000000001d1e/head//3
2014-06-12 14:51:04.521776 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 17365016/rbd_data.10b16b8b4567.0000000000001d1e/head//3
2014-06-12 14:51:04.530829 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:04.551103 7f9d60bd2700 10 getattrs 3.16_head/17365016/rbd_data.10b16b8b4567.0000000000001d1e/head//3 = 0
2014-06-12 14:51:04.551123 7f9d60bd2700 10 stat 3.16_head/3ea85016/rbd_data.10a56b8b4567.0000000000001d85/head//3
2014-06-12 14:51:04.551243 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 3ea85016/rbd_data.10a56b8b4567.0000000000001d85/head//3
2014-06-12 14:51:04.583559 7f9d60bd2700 10 getattrs 3.16_head/3ea85016/rbd_data.10a56b8b4567.0000000000001d85/head//3 = 0
2014-06-12 14:51:04.583567 7f9d60bd2700 10 stat 3.16_head/2e195016/rbd_data.10ae6b8b4567.000000000000225c/head//3
2014-06-12 14:51:04.583671 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 2e195016/rbd_data.10ae6b8b4567.000000000000225c/head//3
2014-06-12 14:51:04.598809 7f9d60bd2700 10 getattrs 3.16_head/2e195016/rbd_data.10ae6b8b4567.000000000000225c/head//3 = 0
2014-06-12 14:51:04.598831 7f9d60bd2700 10 stat 3.16_head/b3e95016/rbd_data.10946b8b4567.00000000000034ba/head//3
2014-06-12 14:51:04.598934 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid b3e95016/rbd_data.10946b8b4567.00000000000034ba/head//3
2014-06-12 14:51:04.617126 7f9d60bd2700 10 getattrs 3.16_head/b3e95016/rbd_data.10946b8b4567.00000000000034ba/head//3 = 0
2014-06-12 14:51:04.617138 7f9d60bd2700 10 stat 3.16_head/1f2a5016/rbd_data.10a66b8b4567.000000000000039a/head//3
2014-06-12 14:51:04.617253 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 1f2a5016/rbd_data.10a66b8b4567.000000000000039a/head//3
2014-06-12 14:51:04.650031 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:04.657364 7f9d60bd2700 10 getattrs 3.16_head/1f2a5016/rbd_data.10a66b8b4567.000000000000039a/head//3 = 0
2014-06-12 14:51:04.657385 7f9d60bd2700 10 stat 3.16_head/8f9b5016/rbd_data.10a16b8b4567.0000000000001e9a/head//3
2014-06-12 14:51:04.657521 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 8f9b5016/rbd_data.10a16b8b4567.0000000000001e9a/head//3
2014-06-12 14:51:04.660183 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:04.672277 7f9d60bd2700 10 getattrs 3.16_head/8f9b5016/rbd_data.10a16b8b4567.0000000000001e9a/head//3 = 0
2014-06-12 14:51:04.672291 7f9d60bd2700 10 stat 3.16_head/9e9d5016/rbd_data.10ae6b8b4567.0000000000001ef3/head//3
2014-06-12 14:51:04.672416 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 9e9d5016/rbd_data.10ae6b8b4567.0000000000001ef3/head//3
2014-06-12 14:51:04.684800 7f9d60bd2700 10 getattrs 3.16_head/9e9d5016/rbd_data.10ae6b8b4567.0000000000001ef3/head//3 = 0
2014-06-12 14:51:04.684816 7f9d60bd2700 10 stat 3.16_head/381e5016/rbd_data.10af6b8b4567.00000000000026cd/head//3
2014-06-12 14:51:04.684960 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 381e5016/rbd_data.10af6b8b4567.00000000000026cd/head//3
2014-06-12 14:51:04.688638 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:04.688726 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:04.692441 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:04.699262 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:04.703200 7f9d60bd2700 10 getattrs 3.16_head/381e5016/rbd_data.10af6b8b4567.00000000000026cd/head//3 = 0
2014-06-12 14:51:04.703210 7f9d60bd2700 10 stat 3.16_head/d7e5016/rbd_data.10b46b8b4567.0000000000002155/head//3
2014-06-12 14:51:04.703333 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid d7e5016/rbd_data.10b46b8b4567.0000000000002155/head//3
2014-06-12 14:51:04.710091 7f9d58d5b700  5 -- op tracker -- , seq: 748558, time: 2014-06-12 14:51:04.710017, event: header_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.710112 7f9d58d5b700  5 -- op tracker -- , seq: 748558, time: 2014-06-12 14:51:04.710019, event: throttled, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.710127 7f9d58d5b700  5 -- op tracker -- , seq: 748558, time: 2014-06-12 14:51:04.710056, event: all_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.710133 7f9d58d5b700  5 -- op tracker -- , seq: 748558, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.710137 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:04.710144 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:51:04.710150 7f9d58d5b700 15 osd.2 173 enqueue_op 0x6f4b300 prio 127 cost 10717 latency 0.000133 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:04.710207 7f9d693e3700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:04.730659 7f9d60bd2700 10 getattrs 3.16_head/d7e5016/rbd_data.10b46b8b4567.0000000000002155/head//3 = 0
2014-06-12 14:51:04.730669 7f9d60bd2700 10 stat 3.16_head/2aaf5016/rbd_data.10ac6b8b4567.0000000000002da0/head//3
2014-06-12 14:51:04.730795 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 2aaf5016/rbd_data.10ac6b8b4567.0000000000002da0/head//3
2014-06-12 14:51:04.761244 7f9d60bd2700 10 getattrs 3.16_head/2aaf5016/rbd_data.10ac6b8b4567.0000000000002da0/head//3 = 0
2014-06-12 14:51:04.761255 7f9d60bd2700 10 stat 3.16_head/23cf5016/rbd_data.10a26b8b4567.0000000000002aa0/head//3
2014-06-12 14:51:04.761351 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 23cf5016/rbd_data.10a26b8b4567.0000000000002aa0/head//3
2014-06-12 14:51:04.766589 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:04.790405 7f9d60bd2700 10 getattrs 3.16_head/23cf5016/rbd_data.10a26b8b4567.0000000000002aa0/head//3 = 0
2014-06-12 14:51:04.790415 7f9d60bd2700 10 stat 3.16_head/67b06016/rbd_data.10a36b8b4567.000000000000011d/head//3
2014-06-12 14:51:04.790527 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 67b06016/rbd_data.10a36b8b4567.000000000000011d/head//3
2014-06-12 14:51:04.794875 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:04.800504 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:04.840477 7f9d60bd2700 10 getattrs 3.16_head/67b06016/rbd_data.10a36b8b4567.000000000000011d/head//3 = 0
2014-06-12 14:51:04.840494 7f9d60bd2700 10 stat 3.16_head/ff926016/rbd_data.109e6b8b4567.000000000000357f/head//3
2014-06-12 14:51:04.840619 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid ff926016/rbd_data.109e6b8b4567.000000000000357f/head//3
2014-06-12 14:51:04.853602 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:04.863967 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:04.866815 7f9d60bd2700 10 getattrs 3.16_head/ff926016/rbd_data.109e6b8b4567.000000000000357f/head//3 = 0
2014-06-12 14:51:04.866827 7f9d60bd2700 10 collection_getattrs 3.16_head
2014-06-12 14:51:04.866884 7f9d60bd2700 10 collection_getattrs 3.16_head = 0
2014-06-12 14:51:04.866888 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:04.866918 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:04.866925 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:04.866934 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:04.866937 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:04.866976 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.16 at 2014-06-11 13:02:32.499677
2014-06-12 14:51:04.867019 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2e at 2014-06-11 13:02:32.500084
2014-06-12 14:51:04.867029 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867033 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867056 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3a at 2014-06-11 13:02:32.500954
2014-06-12 14:51:04.867064 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867066 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867064 7f9d693e3700 10 osd.2 173 dequeue_op 0x6f4b300 prio 127 cost 10717 latency 0.157046 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]
2014-06-12 14:51:04.867085 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.43 at 2014-06-11 13:02:32.501298
2014-06-12 14:51:04.867118 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867120 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867117 7f9d693e3700  5 -- op tracker -- , seq: 748558, time: 2014-06-12 14:51:04.867102, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.867128 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6b at 2014-06-11 13:02:32.502023
2014-06-12 14:51:04.867137 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867131 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] handle_message: 0x6f4b300
2014-06-12 14:51:04.867138 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867146 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6c at 2014-06-11 13:02:32.502351
2014-06-12 14:51:04.867141 7f9d693e3700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:04.867149 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867150 7f9d693e3700  7 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:04.867150 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867158 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.a0 at 2014-06-11 13:02:32.502930
2014-06-12 14:51:04.867161 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867160 7f9d693e3700  5 -- op tracker -- , seq: 748558, time: 2014-06-12 14:51:04.867157, event: started, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.867162 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.a0( v 173'2803 (0'0,173'2803] local-les=157 n=294 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2802 mlcod 173'2802 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867164 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  got 22 scrub map
2014-06-12 14:51:04.867170 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.105 at 2014-06-11 13:02:32.503257
2014-06-12 14:51:04.867173 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867174 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2817 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867181 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.122 at 2014-06-11 13:02:32.503770
2014-06-12 14:51:04.867184 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867185 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867191 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.12b at 2014-06-11 13:02:32.504419
2014-06-12 14:51:04.867194 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867195 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.12b( v 173'3236 (79'235,173'3236] local-les=137 n=323 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3235 mlcod 173'3235 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867202 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.131 at 2014-06-11 13:02:32.504931
2014-06-12 14:51:04.867205 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867206 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867213 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.15d at 2014-06-11 13:02:32.505494
2014-06-12 14:51:04.867216 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867217 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867224 7f9d693e3700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] map version is 173'2396
2014-06-12 14:51:04.867224 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.165 at 2014-06-11 13:02:32.506051
2014-06-12 14:51:04.867231 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867234 7f9d693e3700 10 osd.2 173 dequeue_op 0x6f4b300 finish
2014-06-12 14:51:04.867232 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.165( v 173'3106 (79'105,173'3106] local-les=149 n=320 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3105 mlcod 173'3105 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867239 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.170 at 2014-06-11 13:02:32.506577
2014-06-12 14:51:04.867240 7f9d693e3700  5 -- op tracker -- , seq: 748558, time: 2014-06-12 14:51:04.867237, event: done, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:04.867242 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867246 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3049 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867253 7f9d693e3700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:04.867253 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1b5 at 2014-06-11 13:02:32.507250
2014-06-12 14:51:04.867256 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:04.867256 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867257 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1b5( v 173'2905 (0'0,173'2905] local-les=153 n=320 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2904 mlcod 173'2904 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867263 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:04.867264 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1bc at 2014-06-11 13:02:32.507758
2014-06-12 14:51:04.867287 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867269 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:04.867291 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2459 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867304 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1e1 at 2014-06-11 13:02:32.508326
2014-06-12 14:51:04.867311 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867301 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:04.867313 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2540 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867314 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:04.867326 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1e8 at 2014-06-11 13:02:32.508866
2014-06-12 14:51:04.867331 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867333 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3033 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867346 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1eb at 2014-06-11 13:02:32.509421
2014-06-12 14:51:04.867351 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867350 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:04.867354 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3095 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867367 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1fd at 2014-06-11 13:02:32.510047
2014-06-12 14:51:04.867365 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:04.867374 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867377 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1fd( v 173'3047 (79'46,173'3047] local-les=157 n=305 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867377 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub replica 22 has 25 items
2014-06-12 14:51:04.867391 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.204 at 2014-06-11 13:02:32.510573
2014-06-12 14:51:04.867396 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867398 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.204( v 173'2982 (0'0,173'2982] local-les=161 n=308 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867410 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.209 at 2014-06-11 13:02:32.511095
2014-06-12 14:51:04.867416 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867419 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.209( v 173'2531 (0'0,173'2531] local-les=156 n=264 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2530 mlcod 173'2530 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867416 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1503016/rbd_data.10a76b8b4567.0000000000000c0a/head//3, auth == maps.end()
2014-06-12 14:51:04.867431 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.214 at 2014-06-11 13:02:32.511670
2014-06-12 14:51:04.867437 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867438 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2934 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867442 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 1503016/rbd_data.10a76b8b4567.0000000000000c0a/head//3
2014-06-12 14:51:04.867451 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.21d at 2014-06-11 13:02:32.512202
2014-06-12 14:51:04.867457 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867459 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.21d( v 173'3024 (79'23,173'3024] local-les=156 n=318 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3023 mlcod 173'3023 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867464 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 738c3016/rbd_data.10a66b8b4567.00000000000036b3/head//3, auth == maps.end()
2014-06-12 14:51:04.867477 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.23f at 2014-06-11 13:02:32.512776
2014-06-12 14:51:04.867483 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867485 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867497 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.251 at 2014-06-11 13:02:32.513340
2014-06-12 14:51:04.867502 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867481 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 738c3016/rbd_data.10a66b8b4567.00000000000036b3/head//3
2014-06-12 14:51:04.867504 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867516 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.262 at 2014-06-11 13:02:32.513946
2014-06-12 14:51:04.867522 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867517 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 87ce3016/rbd_data.10936b8b4567.00000000000027d1/head//3, auth == maps.end()
2014-06-12 14:51:04.867524 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.262( v 173'2675 (0'0,173'2675] local-les=142 n=292 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2674 mlcod 173'2674 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867537 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.266 at 2014-06-11 13:02:32.514523
2014-06-12 14:51:04.867543 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867535 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 87ce3016/rbd_data.10936b8b4567.00000000000027d1/head//3
2014-06-12 14:51:04.867546 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2824 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867558 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.288 at 2014-06-11 13:02:32.515022
2014-06-12 14:51:04.867564 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867554 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 44014016/rbd_data.108f6b8b4567.0000000000002610/head//3, auth == maps.end()
2014-06-12 14:51:04.867566 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.288( v 173'2548 (0'0,173'2548] local-les=161 n=263 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2547 mlcod 173'2547 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867571 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 44014016/rbd_data.108f6b8b4567.0000000000002610/head//3
2014-06-12 14:51:04.867578 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2b4 at 2014-06-11 13:02:32.515543
2014-06-12 14:51:04.867584 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867587 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2b4( v 173'3109 (79'108,173'3109] local-les=137 n=312 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3108 mlcod 173'3108 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867589 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d4014016/rbd_data.10b26b8b4567.0000000000000fc1/head//3, auth == maps.end()
2014-06-12 14:51:04.867601 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2c4 at 2014-06-11 13:02:32.516106
2014-06-12 14:51:04.867606 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867607 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj d4014016/rbd_data.10b26b8b4567.0000000000000fc1/head//3
2014-06-12 14:51:04.867614 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2c4( v 173'2897 (0'0,173'2897] local-les=139 n=307 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2896 mlcod 173'2896 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867628 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2d3 at 2014-06-11 13:02:32.516627
2014-06-12 14:51:04.867634 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867626 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 42144016/rbd_data.10ab6b8b4567.0000000000002600/head//3, auth == maps.end()
2014-06-12 14:51:04.867637 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2d3( v 173'2556 (0'0,173'2556] local-les=153 n=276 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2555 mlcod 173'2555 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867649 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2da at 2014-06-11 13:02:32.517119
2014-06-12 14:51:04.867642 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 42144016/rbd_data.10ab6b8b4567.0000000000002600/head//3
2014-06-12 14:51:04.867654 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867657 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2da( v 173'2988 (0'0,173'2988] local-les=145 n=302 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2987 mlcod 173'2987 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867670 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2f4 at 2014-06-11 13:02:32.517764
2014-06-12 14:51:04.867676 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867661 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 93684016/rbd_data.10a86b8b4567.00000000000019ed/head//3, auth == maps.end()
2014-06-12 14:51:04.867678 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3274 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867690 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.312 at 2014-06-11 13:02:32.518300
2014-06-12 14:51:04.867696 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867685 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 93684016/rbd_data.10a86b8b4567.00000000000019ed/head//3
2014-06-12 14:51:04.867698 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2782 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867715 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.320 at 2014-06-11 13:02:32.518824
2014-06-12 14:51:04.867728 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867717 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1d7a4016/rbd_data.10976b8b4567.00000000000012cd/head//3, auth == maps.end()
2014-06-12 14:51:04.867730 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.320( v 173'2687 (0'0,173'2687] local-les=145 n=282 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2686 mlcod 173'2686 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867744 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.330 at 2014-06-11 13:02:32.519349
2014-06-12 14:51:04.867749 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867752 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2798 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867747 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 1d7a4016/rbd_data.10976b8b4567.00000000000012cd/head//3
2014-06-12 14:51:04.867765 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.342 at 2014-06-11 13:02:32.519873
2014-06-12 14:51:04.867770 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867772 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2464 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867772 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cb7b4016/rbd_data.108e6b8b4567.0000000000003309/head//3, auth == maps.end()
2014-06-12 14:51:04.867786 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.35b at 2014-06-11 13:02:32.520528
2014-06-12 14:51:04.867792 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867791 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj cb7b4016/rbd_data.108e6b8b4567.0000000000003309/head//3
2014-06-12 14:51:04.867795 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.35b( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867807 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.366 at 2014-06-11 13:02:32.521097
2014-06-12 14:51:04.867813 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867809 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 65cb4016/rbd_data.10b26b8b4567.00000000000009b8/head//3, auth == maps.end()
2014-06-12 14:51:04.867815 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2661 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867828 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.37a at 2014-06-11 13:02:32.521744
2014-06-12 14:51:04.867834 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867826 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 65cb4016/rbd_data.10b26b8b4567.00000000000009b8/head//3
2014-06-12 14:51:04.867836 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.37a( v 173'2555 (0'0,173'2555] local-les=153 n=292 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2554 mlcod 173'2554 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867849 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.394 at 2014-06-11 13:02:32.522362
2014-06-12 14:51:04.867854 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867844 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a3ad4016/rbd_data.10a86b8b4567.00000000000027bb/head//3, auth == maps.end()
2014-06-12 14:51:04.867857 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.394( v 173'2648 (0'0,173'2648] local-les=145 n=266 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867861 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj a3ad4016/rbd_data.10a86b8b4567.00000000000027bb/head//3
2014-06-12 14:51:04.867869 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.395 at 2014-06-11 13:02:32.522991
2014-06-12 14:51:04.867875 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867877 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3327 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867879 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2bbf4016/rbd_data.10b26b8b4567.00000000000004c8/head//3, auth == maps.end()
2014-06-12 14:51:04.867890 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3ac at 2014-06-11 13:02:32.523569
2014-06-12 14:51:04.867899 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867896 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 2bbf4016/rbd_data.10b26b8b4567.00000000000004c8/head//3
2014-06-12 14:51:04.867901 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3ac( v 173'2860 (0'0,173'2860] local-les=142 n=294 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867915 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3d4 at 2014-06-11 13:02:32.524119
2014-06-12 14:51:04.867921 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867914 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 17365016/rbd_data.10b16b8b4567.0000000000001d1e/head//3, auth == maps.end()
2014-06-12 14:51:04.867923 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3d4( v 173'2696 (0'0,173'2696] local-les=137 n=271 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867937 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3d8 at 2014-06-11 13:02:32.524641
2014-06-12 14:51:04.867930 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 17365016/rbd_data.10b16b8b4567.0000000000001d1e/head//3
2014-06-12 14:51:04.867942 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867948 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3ea85016/rbd_data.10a56b8b4567.0000000000001d85/head//3, auth == maps.end()
2014-06-12 14:51:04.867951 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3d8( v 173'2879 (0'0,173'2879] local-les=156 n=294 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867964 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.401 at 2014-06-11 13:02:32.525188
2014-06-12 14:51:04.867970 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867965 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 3ea85016/rbd_data.10a56b8b4567.0000000000001d85/head//3
2014-06-12 14:51:04.867972 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3154 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.867985 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.408 at 2014-06-11 13:02:32.525838
2014-06-12 14:51:04.867991 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.867983 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2e195016/rbd_data.10ae6b8b4567.000000000000225c/head//3, auth == maps.end()
2014-06-12 14:51:04.867993 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2883 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868006 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.411 at 2014-06-11 13:02:32.526419
2014-06-12 14:51:04.868000 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 2e195016/rbd_data.10ae6b8b4567.000000000000225c/head//3
2014-06-12 14:51:04.868011 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868013 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.411( v 173'2402 (0'0,173'2402] local-les=147 n=278 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2401 mlcod 173'2401 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868019 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b3e95016/rbd_data.10946b8b4567.00000000000034ba/head//3, auth == maps.end()
2014-06-12 14:51:04.868027 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.45f at 2014-06-11 13:02:32.526940
2014-06-12 14:51:04.868033 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868035 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2357 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868036 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj b3e95016/rbd_data.10946b8b4567.00000000000034ba/head//3
2014-06-12 14:51:04.868048 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.479 at 2014-06-11 13:02:32.527498
2014-06-12 14:51:04.868053 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868055 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.479( v 173'2926 (0'0,173'2926] local-les=142 n=298 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868057 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1f2a5016/rbd_data.10a66b8b4567.000000000000039a/head//3, auth == maps.end()
2014-06-12 14:51:04.868068 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4a7 at 2014-06-11 13:02:32.528023
2014-06-12 14:51:04.868074 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868074 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 1f2a5016/rbd_data.10a66b8b4567.000000000000039a/head//3
2014-06-12 14:51:04.868076 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2594 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868089 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4b4 at 2014-06-11 13:02:32.528548
2014-06-12 14:51:04.868094 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868092 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8f9b5016/rbd_data.10a16b8b4567.0000000000001e9a/head//3, auth == maps.end()
2014-06-12 14:51:04.868096 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4b4( v 173'2792 (0'0,173'2792] local-les=137 n=305 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868109 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4d4 at 2014-06-11 13:02:32.529466
2014-06-12 14:51:04.868115 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868109 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 8f9b5016/rbd_data.10a16b8b4567.0000000000001e9a/head//3
2014-06-12 14:51:04.868117 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2745 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868129 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4ef at 2014-06-11 13:02:32.530034
2014-06-12 14:51:04.868135 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868131 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9e9d5016/rbd_data.10ae6b8b4567.0000000000001ef3/head//3, auth == maps.end()
2014-06-12 14:51:04.868137 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4ef( v 173'3004 (79'3,173'3004] local-les=149 n=307 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868149 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4f0 at 2014-06-11 13:02:32.530572
2014-06-12 14:51:04.868155 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868147 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 9e9d5016/rbd_data.10ae6b8b4567.0000000000001ef3/head//3
2014-06-12 14:51:04.868157 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2864 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868165 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 381e5016/rbd_data.10af6b8b4567.00000000000026cd/head//3, auth == maps.end()
2014-06-12 14:51:04.868176 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.54b at 2014-06-11 13:02:32.531354
2014-06-12 14:51:04.868182 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868181 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 381e5016/rbd_data.10af6b8b4567.00000000000026cd/head//3
2014-06-12 14:51:04.868184 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2722 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868197 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.568 at 2014-06-11 13:02:32.531914
2014-06-12 14:51:04.868203 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868199 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d7e5016/rbd_data.10b46b8b4567.0000000000002155/head//3, auth == maps.end()
2014-06-12 14:51:04.868205 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.568( v 173'2727 (0'0,173'2727] local-les=161 n=270 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868217 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.58f at 2014-06-11 13:02:32.532612
2014-06-12 14:51:04.868223 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868216 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj d7e5016/rbd_data.10b46b8b4567.0000000000002155/head//3
2014-06-12 14:51:04.868225 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.58f( v 173'2707 (0'0,173'2707] local-les=153 n=304 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868238 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5a3 at 2014-06-11 13:02:32.533139
2014-06-12 14:51:04.868244 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868234 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2aaf5016/rbd_data.10ac6b8b4567.0000000000002da0/head//3, auth == maps.end()
2014-06-12 14:51:04.868246 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5a3( v 173'2874 (0'0,173'2874] local-les=142 n=296 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2873 mlcod 173'2873 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868251 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 2aaf5016/rbd_data.10ac6b8b4567.0000000000002da0/head//3
2014-06-12 14:51:04.868259 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5ce at 2014-06-11 13:02:32.533636
2014-06-12 14:51:04.868265 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868268 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5ce( v 173'2741 (0'0,173'2741] local-les=153 n=294 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868270 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 23cf5016/rbd_data.10a26b8b4567.0000000000002aa0/head//3, auth == maps.end()
2014-06-12 14:51:04.868281 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5dd at 2014-06-11 13:02:32.534211
2014-06-12 14:51:04.868286 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868288 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5dd( v 173'2852 (0'0,173'2852] local-les=156 n=307 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868288 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 23cf5016/rbd_data.10a26b8b4567.0000000000002aa0/head//3
2014-06-12 14:51:04.868302 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5fc at 2014-06-11 13:02:32.534738
2014-06-12 14:51:04.868308 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868307 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 67b06016/rbd_data.10a36b8b4567.000000000000011d/head//3, auth == maps.end()
2014-06-12 14:51:04.868310 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5fc( v 173'2714 (0'0,173'2714] local-les=157 n=284 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868322 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.602 at 2014-06-11 13:02:32.535274
2014-06-12 14:51:04.868328 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868324 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 67b06016/rbd_data.10a36b8b4567.000000000000011d/head//3
2014-06-12 14:51:04.868330 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3009 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868343 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.605 at 2014-06-11 13:02:32.535829
2014-06-12 14:51:04.868348 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868341 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ff926016/rbd_data.109e6b8b4567.000000000000357f/head//3, auth == maps.end()
2014-06-12 14:51:04.868351 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868363 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.613 at 2014-06-11 13:02:32.536448
2014-06-12 14:51:04.868368 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868360 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj ff926016/rbd_data.109e6b8b4567.000000000000357f/head//3
2014-06-12 14:51:04.868371 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868383 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.635 at 2014-06-11 13:02:32.537041
2014-06-12 14:51:04.868388 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868380 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] 
2014-06-12 14:51:04.868390 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.635( v 173'2966 (0'0,173'2966] local-les=137 n=308 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2965 mlcod 173'2965 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868393 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub
2014-06-12 14:51:04.868404 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.63b at 2014-06-11 13:02:32.537647
2014-06-12 14:51:04.868410 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868412 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868426 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.63e at 2014-06-11 13:02:32.538178
2014-06-12 14:51:04.868431 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868417 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  ff926016/rbd_data.109e6b8b4567.000000000000357f/head//3 ff926016/rbd_data.109e6b8b4567.000000000000357f/head//3(173'1789 client.4295.0:106054 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1789)
2014-06-12 14:51:04.868433 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868445 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.650 at 2014-06-11 13:02:32.538854
2014-06-12 14:51:04.868451 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868441 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  67b06016/rbd_data.10a36b8b4567.000000000000011d/head//3 67b06016/rbd_data.10a36b8b4567.000000000000011d/head//3(173'2042 client.4300.0:116564 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2042)
2014-06-12 14:51:04.868453 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868466 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.662 at 2014-06-11 13:02:32.539391
2014-06-12 14:51:04.868471 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868464 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  23cf5016/rbd_data.10a26b8b4567.0000000000002aa0/head//3 23cf5016/rbd_data.10a26b8b4567.0000000000002aa0/head//3(173'1651 client.4299.0:88580 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1651)
2014-06-12 14:51:04.868474 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.662( v 173'2604 (0'0,173'2604] local-les=161 n=257 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2603 mlcod 173'2603 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868486 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.667 at 2014-06-11 13:02:32.539999
2014-06-12 14:51:04.868492 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868486 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  2aaf5016/rbd_data.10ac6b8b4567.0000000000002da0/head//3 2aaf5016/rbd_data.10ac6b8b4567.0000000000002da0/head//3(173'1692 client.4309.0:90890 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1692)
2014-06-12 14:51:04.868494 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.667( v 173'2958 (0'0,173'2958] local-les=149 n=303 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868507 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6a8 at 2014-06-11 13:02:32.540566
2014-06-12 14:51:04.868513 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868507 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  d7e5016/rbd_data.10b46b8b4567.0000000000002155/head//3 d7e5016/rbd_data.10b46b8b4567.0000000000002155/head//3(173'1276 client.4317.0:69074 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1276)
2014-06-12 14:51:04.868515 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6a8( v 173'2854 (0'0,173'2854] local-les=137 n=299 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868528 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6ff at 2014-06-11 13:02:32.541074
2014-06-12 14:51:04.868534 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868529 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  381e5016/rbd_data.10af6b8b4567.00000000000026cd/head//3 381e5016/rbd_data.10af6b8b4567.00000000000026cd/head//3(173'1386 client.4312.0:81448 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1386)
2014-06-12 14:51:04.868536 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6ff( v 173'2792 (0'0,173'2792] local-les=161 n=298 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868549 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.711 at 2014-06-11 13:02:32.541770
2014-06-12 14:51:04.868554 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868551 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  9e9d5016/rbd_data.10ae6b8b4567.0000000000001ef3/head//3 9e9d5016/rbd_data.10ae6b8b4567.0000000000001ef3/head//3(173'1051 client.4311.0:64007 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1051)
2014-06-12 14:51:04.868556 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.711( v 173'2514 (0'0,173'2514] local-les=147 n=297 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868569 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.712 at 2014-06-11 13:02:32.542314
2014-06-12 14:51:04.868576 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868572 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  8f9b5016/rbd_data.10a16b8b4567.0000000000001e9a/head//3 8f9b5016/rbd_data.10a16b8b4567.0000000000001e9a/head//3(173'1133 client.4298.0:65430 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1133)
2014-06-12 14:51:04.868578 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.712( v 173'2782 (0'0,173'2782] local-les=153 n=305 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868591 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.717 at 2014-06-11 13:02:32.542947
2014-06-12 14:51:04.868597 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868593 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  1f2a5016/rbd_data.10a66b8b4567.000000000000039a/head//3 1f2a5016/rbd_data.10a66b8b4567.000000000000039a/head//3(173'2100 client.4303.0:123485 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2100)
2014-06-12 14:51:04.868599 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2583 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868612 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.721 at 2014-06-11 13:02:32.543618
2014-06-12 14:51:04.868617 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868614 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  b3e95016/rbd_data.10946b8b4567.00000000000034ba/head//3 b3e95016/rbd_data.10946b8b4567.00000000000034ba/head//3(173'1795 client.4285.0:103454 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1795)
2014-06-12 14:51:04.868620 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2895 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868632 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.735 at 2014-06-11 13:02:32.544147
2014-06-12 14:51:04.868638 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868640 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.735( v 173'2803 (0'0,173'2803] local-les=137 n=306 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2802 mlcod 173'2802 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868636 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  2e195016/rbd_data.10ae6b8b4567.000000000000225c/head//3 2e195016/rbd_data.10ae6b8b4567.000000000000225c/head//3(173'1177 client.4311.0:69551 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1177)
2014-06-12 14:51:04.868653 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.762 at 2014-06-11 13:02:32.544735
2014-06-12 14:51:04.868659 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868661 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.762( v 173'2726 (0'0,173'2726] local-les=147 n=302 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868657 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  3ea85016/rbd_data.10a56b8b4567.0000000000001d85/head//3 3ea85016/rbd_data.10a56b8b4567.0000000000001d85/head//3(173'1085 client.4302.0:63106 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1085)
2014-06-12 14:51:04.868674 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.76e at 2014-06-11 13:02:32.545259
2014-06-12 14:51:04.868679 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868682 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2648 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868679 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  17365016/rbd_data.10b16b8b4567.0000000000001d1e/head//3 17365016/rbd_data.10b16b8b4567.0000000000001d1e/head//3(173'1115 client.4314.0:62017 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1115)
2014-06-12 14:51:04.868694 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.77e at 2014-06-11 13:02:32.545813
2014-06-12 14:51:04.868700 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868702 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.77e( v 173'3052 (79'51,173'3052] local-les=161 n=319 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3051 mlcod 173'3051 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868701 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  2bbf4016/rbd_data.10b26b8b4567.00000000000004c8/head//3 2bbf4016/rbd_data.10b26b8b4567.00000000000004c8/head//3(173'2135 client.4315.0:124831 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2135)
2014-06-12 14:51:04.868715 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.783 at 2014-06-11 13:02:32.546436
2014-06-12 14:51:04.868720 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868722 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868722 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  a3ad4016/rbd_data.10a86b8b4567.00000000000027bb/head//3 a3ad4016/rbd_data.10a86b8b4567.00000000000027bb/head//3(173'1377 client.4305.0:80224 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1377)
2014-06-12 14:51:04.868737 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.785 at 2014-06-11 13:02:32.547026
2014-06-12 14:51:04.868743 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868745 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868758 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.793 at 2014-06-11 13:02:32.547568
2014-06-12 14:51:04.868763 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868745 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  65cb4016/rbd_data.10b26b8b4567.00000000000009b8/head//3 65cb4016/rbd_data.10b26b8b4567.00000000000009b8/head//3(173'2295 client.4315.0:133376 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2295)
2014-06-12 14:51:04.868765 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.793( v 173'2605 (0'0,173'2605] local-les=153 n=277 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2604 mlcod 173'2604 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868778 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ab at 2014-06-11 13:02:32.548162
2014-06-12 14:51:04.868784 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868772 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  cb7b4016/rbd_data.108e6b8b4567.0000000000003309/head//3 cb7b4016/rbd_data.108e6b8b4567.0000000000003309/head//3(173'1762 client.4279.0:101380 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1762)
2014-06-12 14:51:04.868786 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ab( v 173'2921 (0'0,173'2921] local-les=161 n=302 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2920 mlcod 173'2920 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868799 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ad at 2014-06-11 13:02:32.548789
2014-06-12 14:51:04.868804 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868796 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  1d7a4016/rbd_data.10976b8b4567.00000000000012cd/head//3 1d7a4016/rbd_data.10976b8b4567.00000000000012cd/head//3(173'736 client.4288.0:42094 wrlock_by=unknown.0.0:0 dirty s 4194304 uv736)
2014-06-12 14:51:04.868806 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ad( v 173'3043 (79'42,173'3043] local-les=149 n=315 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868819 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ae at 2014-06-11 13:02:32.549514
2014-06-12 14:51:04.868824 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868818 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  93684016/rbd_data.10a86b8b4567.00000000000019ed/head//3 93684016/rbd_data.10a86b8b4567.00000000000019ed/head//3(173'887 client.4305.0:55316 wrlock_by=unknown.0.0:0 dirty s 4194304 uv887)
2014-06-12 14:51:04.868827 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2517 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868843 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7bc at 2014-06-11 13:02:32.550208
2014-06-12 14:51:04.868849 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868839 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  42144016/rbd_data.10ab6b8b4567.0000000000002600/head//3 42144016/rbd_data.10ab6b8b4567.0000000000002600/head//3(173'1337 client.4308.0:78040 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1337)
2014-06-12 14:51:04.868851 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7bc( v 173'2562 (0'0,173'2562] local-les=153 n=287 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2561 mlcod 173'2561 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868863 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7cc at 2014-06-11 13:02:32.550758
2014-06-12 14:51:04.868869 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868860 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  d4014016/rbd_data.10b26b8b4567.0000000000000fc1/head//3 d4014016/rbd_data.10b26b8b4567.0000000000000fc1/head//3(173'2396 client.4315.0:143539 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2396)
2014-06-12 14:51:04.868871 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868883 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7db at 2014-06-11 13:02:32.551269
2014-06-12 14:51:04.868889 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868881 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  44014016/rbd_data.108f6b8b4567.0000000000002610/head//3 44014016/rbd_data.108f6b8b4567.0000000000002610/head//3(173'1350 client.4280.0:76702 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1350)
2014-06-12 14:51:04.868891 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2681 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868903 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7f8 at 2014-06-11 13:02:32.551909
2014-06-12 14:51:04.868908 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:04.868902 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  87ce3016/rbd_data.10936b8b4567.00000000000027d1/head//3 87ce3016/rbd_data.10936b8b4567.00000000000027d1/head//3(173'1407 client.4284.0:79335 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1407)
2014-06-12 14:51:04.868910 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:04.868923 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  738c3016/rbd_data.10a66b8b4567.00000000000036b3/head//3 738c3016/rbd_data.10a66b8b4567.00000000000036b3/head//3(173'1943 client.4303.0:108546 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1943)
2014-06-12 14:51:04.868923 7f9d84175700 10 osd.2 173 sched_scrub 0.3 at 2014-06-12 13:01:39.405748: 6565.11 < min (86400 seconds)
2014-06-12 14:51:04.868942 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:04.868945 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:04.868947 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:04.868944 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  1503016/rbd_data.10a76b8b4567.0000000000000c0a/head//3 1503016/rbd_data.10a76b8b4567.0000000000000c0a/head//3(173'2312 client.4304.0:136800 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2312)
2014-06-12 14:51:04.868960 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:04.868977 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:04.868991 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:04.868994 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:04.868996 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:04.869041 7f9d60bd2700 10 collection_list_partial 3.16_head start:81d26016//0//-1 is_max:0
2014-06-12 14:51:04.869387 7f9d60bd2700 20 objects: [5d26016/rbd_data.109b6b8b4567.00000000000001b6/head//3,a2336016/rbd_data.10a86b8b4567.0000000000000b93/head//3,ae636016/rbd_data.10a36b8b4567.0000000000001be7/head//3,78056016/rbd_data.10ab6b8b4567.000000000000196b/head//3,2bb56016/rbd_data.109a6b8b4567.0000000000001729/head//3,3b166016/rbd_data.108f6b8b4567.0000000000002174/head//3,f6ba6016/rbd_data.10a16b8b4567.0000000000001e87/head//3,597b6016/rbd_data.10b06b8b4567.000000000000119f/head//3,1ad6016/rbd_data.10ae6b8b4567.0000000000001333/head//3,467e6016/rbd_data.10b06b8b4567.0000000000002949/head//3,2bd27016/rbd_data.10a36b8b4567.00000000000000b0/head//3,89137016/rbd_data.10a26b8b4567.0000000000001f6f/head//3,8b437016/rbd_data.10966b8b4567.00000000000036cf/head//3,78777016/rbd_data.109e6b8b4567.0000000000001fd3/head//3,8987016/rbd_data.10a96b8b4567.0000000000000585/head//3,c00a7016/rbd_data.108d6b8b4567.000000000000067e/head//3,cada7016/rbd_data.109c6b8b4567.0000000000001b2b/head//3,1c818016/rbd_data.10936b8b4567.0000000000002e8d/head//3,f1a18016/rbd_data.10b16b8b4567.00000000000000a6/head//3,2ad38016/rbd_data.10a76b8b4567.0000000000000be2/head//3,9ae48016/rbd_data.109c6b8b4567.00000000000027a5/head//3,1a778016/rbd_data.109e6b8b4567.0000000000001c28/head//3,6488016/rbd_data.109e6b8b4567.0000000000002c14/head//3,8abc8016/rbd_data.10af6b8b4567.0000000000001b0a/head//3,e19d8016/rbd_data.10a56b8b4567.0000000000002ac3/head//3]
2014-06-12 14:51:04.869993 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  requesting scrubmap from osd.22
2014-06-12 14:51:04.870035 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:04.870065 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:04.870080 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:04.870091 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:04.870102 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:04.870112 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_map_chunk [81d26016//0//-1,47279016//0//-1)
2014-06-12 14:51:04.870126 7f9d60bd2700 10 collection_list_partial 3.16_head start:81d26016//0//-1 is_max:0
2014-06-12 14:51:04.870945 7f9d60bd2700 20 objects: [5d26016/rbd_data.109b6b8b4567.00000000000001b6/head//3,a2336016/rbd_data.10a86b8b4567.0000000000000b93/head//3,ae636016/rbd_data.10a36b8b4567.0000000000001be7/head//3,78056016/rbd_data.10ab6b8b4567.000000000000196b/head//3,2bb56016/rbd_data.109a6b8b4567.0000000000001729/head//3,3b166016/rbd_data.108f6b8b4567.0000000000002174/head//3,f6ba6016/rbd_data.10a16b8b4567.0000000000001e87/head//3,597b6016/rbd_data.10b06b8b4567.000000000000119f/head//3,1ad6016/rbd_data.10ae6b8b4567.0000000000001333/head//3,467e6016/rbd_data.10b06b8b4567.0000000000002949/head//3,2bd27016/rbd_data.10a36b8b4567.00000000000000b0/head//3,89137016/rbd_data.10a26b8b4567.0000000000001f6f/head//3,8b437016/rbd_data.10966b8b4567.00000000000036cf/head//3,78777016/rbd_data.109e6b8b4567.0000000000001fd3/head//3,8987016/rbd_data.10a96b8b4567.0000000000000585/head//3,c00a7016/rbd_data.108d6b8b4567.000000000000067e/head//3,cada7016/rbd_data.109c6b8b4567.0000000000001b2b/head//3,1c818016/rbd_data.10936b8b4567.0000000000002e8d/head//3,f1a18016/rbd_data.10b16b8b4567.00000000000000a6/head//3,2ad38016/rbd_data.10a76b8b4567.0000000000000be2/head//3,9ae48016/rbd_data.109c6b8b4567.00000000000027a5/head//3,1a778016/rbd_data.109e6b8b4567.0000000000001c28/head//3,6488016/rbd_data.109e6b8b4567.0000000000002c14/head//3,8abc8016/rbd_data.10af6b8b4567.0000000000001b0a/head//3,e19d8016/rbd_data.10a56b8b4567.0000000000002ac3/head//3,47279016/rbd_data.10a26b8b4567.00000000000008c1/head//3,8bb79016/rbd_data.108d6b8b4567.0000000000000ef0/head//3,917b9016/rbd_data.10906b8b4567.00000000000013ea/head//3,7d1e9016/rbd_data.10ad6b8b4567.000000000000304b/head//3,a7fe9016/rbd_data.10ac6b8b4567.0000000000003685/head//3,835f9016/rbd_data.109e6b8b4567.0000000000001771/head//3,69f1a016/rbd_data.10ac6b8b4567.000000000000382f/head//3,d0d2a016/rbd_data.10a66b8b4567.0000000000002934/head//3,9f44a016/rbd_data.109b6b8b4567.0000000000000721/head//3,6b86a016/rbd_data.10916b8b4567.00000000000014aa/head//3,31a9a016/rbd_data.10a86b8b4567.0000000000002f08/head//3,215aa016/rbd_data.109a6b8b4567.0000000000001af5/head//3,b5cca016/rbd_data.10a96b8b4567.000000000000115a/head//3,be4da016/rbd_data.10ab6b8b4567.0000000000002ee4/head//3,b4cea016/rbd_data.109c6b8b4567.0000000000000616/head//3,ac32b016/rbd_data.10b16b8b4567.00000000000038a7/head//3,d4e3b016/rbd_data.10b16b8b4567.0000000000001422/head//3,5505b016/rbd_data.109b6b8b4567.0000000000002768/head//3,f7c5b016/rbd_data.10aa6b8b4567.0000000000002543/head//3,5256b016/rbd_data.10ab6b8b4567.0000000000002d29/head//3,f929b016/rbd_data.10a56b8b4567.00000000000017a7/head//3,5dfab016/rbd_data.10a66b8b4567.0000000000002515/head//3,ebbbb016/rbd_data.109c6b8b4567.00000000000034f7/head//3,fc5db016/rbd_data.10a06b8b4567.000000000000284e/head//3,8600c016/rbd_data.10a76b8b4567.0000000000001cb2/head//3,a4a3c016/rbd_data.10976b8b4567.0000000000002326/head//3,b8f3c016/rbd_data.109d6b8b4567.00000000000030d1/head//3,f898c016/rbd_data.10a86b8b4567.0000000000001a5b/head//3,c2f8c016/rbd_data.10a56b8b4567.000000000000249d/head//3,cacc016/rbd_data.10a56b8b4567.000000000000088e/head//3,20cdc016/rbd_data.10a76b8b4567.0000000000003570/head//3,6591d016/rbd_data.10936b8b4567.0000000000002355/head//3,7c1d016/rbd_data.109d6b8b4567.0000000000001891/head//3,cfc2d016/rbd_data.10946b8b4567.00000000000017ce/head//3,3ed2d016/rbd_data.10926b8b4567.0000000000002610/head//3,d493d016/rbd_data.10956b8b4567.0000000000000bf2/head//3,e824d016/rbd_data.10a56b8b4567.00000000000021c1/head//3,335ad016/rbd_data.109c6b8b4567.00000000000017d4/head//3,a6abd016/rbd_data.109e6b8b4567.00000000000038a6/head//3]
2014-06-12 14:51:04.871066 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:04.871091 7f9d60bd2700 10 stat 3.16_head/5d26016/rbd_data.109b6b8b4567.00000000000001b6/head//3
2014-06-12 14:51:04.871315 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 5d26016/rbd_data.109b6b8b4567.00000000000001b6/head//3
2014-06-12 14:51:04.906509 7f9d60bd2700 10 getattrs 3.16_head/5d26016/rbd_data.109b6b8b4567.00000000000001b6/head//3 = 0
2014-06-12 14:51:04.906523 7f9d60bd2700 10 stat 3.16_head/a2336016/rbd_data.10a86b8b4567.0000000000000b93/head//3
2014-06-12 14:51:04.906651 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid a2336016/rbd_data.10a86b8b4567.0000000000000b93/head//3
2014-06-12 14:51:04.926968 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:04.934465 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:04.950179 7f9d60bd2700 10 getattrs 3.16_head/a2336016/rbd_data.10a86b8b4567.0000000000000b93/head//3 = 0
2014-06-12 14:51:04.950196 7f9d60bd2700 10 stat 3.16_head/ae636016/rbd_data.10a36b8b4567.0000000000001be7/head//3
2014-06-12 14:51:04.950337 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid ae636016/rbd_data.10a36b8b4567.0000000000001be7/head//3
2014-06-12 14:51:04.960827 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:04.964574 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:04.965058 7f9d60bd2700 10 getattrs 3.16_head/ae636016/rbd_data.10a36b8b4567.0000000000001be7/head//3 = 0
2014-06-12 14:51:04.965073 7f9d60bd2700 10 stat 3.16_head/78056016/rbd_data.10ab6b8b4567.000000000000196b/head//3
2014-06-12 14:51:04.965194 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 78056016/rbd_data.10ab6b8b4567.000000000000196b/head//3
2014-06-12 14:51:04.991161 7f9d60bd2700 10 getattrs 3.16_head/78056016/rbd_data.10ab6b8b4567.000000000000196b/head//3 = 0
2014-06-12 14:51:04.991175 7f9d60bd2700 10 stat 3.16_head/2bb56016/rbd_data.109a6b8b4567.0000000000001729/head//3
2014-06-12 14:51:04.991285 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 2bb56016/rbd_data.109a6b8b4567.0000000000001729/head//3
2014-06-12 14:51:05.021123 7f9d60bd2700 10 getattrs 3.16_head/2bb56016/rbd_data.109a6b8b4567.0000000000001729/head//3 = 0
2014-06-12 14:51:05.021132 7f9d60bd2700 10 stat 3.16_head/3b166016/rbd_data.108f6b8b4567.0000000000002174/head//3
2014-06-12 14:51:05.021230 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 3b166016/rbd_data.108f6b8b4567.0000000000002174/head//3
2014-06-12 14:51:05.055871 7f9d60bd2700 10 getattrs 3.16_head/3b166016/rbd_data.108f6b8b4567.0000000000002174/head//3 = 0
2014-06-12 14:51:05.055892 7f9d60bd2700 10 stat 3.16_head/f6ba6016/rbd_data.10a16b8b4567.0000000000001e87/head//3
2014-06-12 14:51:05.056037 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid f6ba6016/rbd_data.10a16b8b4567.0000000000001e87/head//3
2014-06-12 14:51:05.068273 7f9d60bd2700 10 getattrs 3.16_head/f6ba6016/rbd_data.10a16b8b4567.0000000000001e87/head//3 = 0
2014-06-12 14:51:05.068280 7f9d60bd2700 10 stat 3.16_head/597b6016/rbd_data.10b06b8b4567.000000000000119f/head//3
2014-06-12 14:51:05.068373 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 597b6016/rbd_data.10b06b8b4567.000000000000119f/head//3
2014-06-12 14:51:05.079566 7f9d60bd2700 10 getattrs 3.16_head/597b6016/rbd_data.10b06b8b4567.000000000000119f/head//3 = 0
2014-06-12 14:51:05.079574 7f9d60bd2700 10 stat 3.16_head/1ad6016/rbd_data.10ae6b8b4567.0000000000001333/head//3
2014-06-12 14:51:05.079668 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 1ad6016/rbd_data.10ae6b8b4567.0000000000001333/head//3
2014-06-12 14:51:05.105975 7f9d60bd2700 10 getattrs 3.16_head/1ad6016/rbd_data.10ae6b8b4567.0000000000001333/head//3 = 0
2014-06-12 14:51:05.105988 7f9d60bd2700 10 stat 3.16_head/467e6016/rbd_data.10b06b8b4567.0000000000002949/head//3
2014-06-12 14:51:05.106082 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 467e6016/rbd_data.10b06b8b4567.0000000000002949/head//3
2014-06-12 14:51:05.119858 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:05.122600 7f9d60bd2700 10 getattrs 3.16_head/467e6016/rbd_data.10b06b8b4567.0000000000002949/head//3 = 0
2014-06-12 14:51:05.122609 7f9d60bd2700 10 stat 3.16_head/2bd27016/rbd_data.10a36b8b4567.00000000000000b0/head//3
2014-06-12 14:51:05.122707 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 2bd27016/rbd_data.10a36b8b4567.00000000000000b0/head//3
2014-06-12 14:51:05.131445 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:05.140925 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:05.150380 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:05.176600 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:05.176596 7f9d60bd2700 10 getattrs 3.16_head/2bd27016/rbd_data.10a36b8b4567.00000000000000b0/head//3 = 0
2014-06-12 14:51:05.176613 7f9d60bd2700 10 stat 3.16_head/89137016/rbd_data.10a26b8b4567.0000000000001f6f/head//3
2014-06-12 14:51:05.176723 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 89137016/rbd_data.10a26b8b4567.0000000000001f6f/head//3
2014-06-12 14:51:05.178789 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:05.195677 7f9d60bd2700 10 getattrs 3.16_head/89137016/rbd_data.10a26b8b4567.0000000000001f6f/head//3 = 0
2014-06-12 14:51:05.195685 7f9d60bd2700 10 stat 3.16_head/8b437016/rbd_data.10966b8b4567.00000000000036cf/head//3
2014-06-12 14:51:05.195786 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 8b437016/rbd_data.10966b8b4567.00000000000036cf/head//3
2014-06-12 14:51:05.216973 7f9d60bd2700 10 getattrs 3.16_head/8b437016/rbd_data.10966b8b4567.00000000000036cf/head//3 = 0
2014-06-12 14:51:05.216981 7f9d60bd2700 10 stat 3.16_head/78777016/rbd_data.109e6b8b4567.0000000000001fd3/head//3
2014-06-12 14:51:05.217084 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 78777016/rbd_data.109e6b8b4567.0000000000001fd3/head//3
2014-06-12 14:51:05.217193 7f9d60bd2700 10 getattrs 3.16_head/78777016/rbd_data.109e6b8b4567.0000000000001fd3/head//3 = 0
2014-06-12 14:51:05.217198 7f9d60bd2700 10 stat 3.16_head/8987016/rbd_data.10a96b8b4567.0000000000000585/head//3
2014-06-12 14:51:05.217288 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 8987016/rbd_data.10a96b8b4567.0000000000000585/head//3
2014-06-12 14:51:05.249091 7f9d60bd2700 10 getattrs 3.16_head/8987016/rbd_data.10a96b8b4567.0000000000000585/head//3 = 0
2014-06-12 14:51:05.249106 7f9d60bd2700 10 stat 3.16_head/c00a7016/rbd_data.108d6b8b4567.000000000000067e/head//3
2014-06-12 14:51:05.249260 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid c00a7016/rbd_data.108d6b8b4567.000000000000067e/head//3
2014-06-12 14:51:05.265996 7f9d60bd2700 10 getattrs 3.16_head/c00a7016/rbd_data.108d6b8b4567.000000000000067e/head//3 = 0
2014-06-12 14:51:05.266013 7f9d60bd2700 10 stat 3.16_head/cada7016/rbd_data.109c6b8b4567.0000000000001b2b/head//3
2014-06-12 14:51:05.266118 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid cada7016/rbd_data.109c6b8b4567.0000000000001b2b/head//3
2014-06-12 14:51:05.281535 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:05.296306 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:05.296505 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:05.296808 7f9d60bd2700 10 getattrs 3.16_head/cada7016/rbd_data.109c6b8b4567.0000000000001b2b/head//3 = 0
2014-06-12 14:51:05.296816 7f9d60bd2700 10 stat 3.16_head/1c818016/rbd_data.10936b8b4567.0000000000002e8d/head//3
2014-06-12 14:51:05.296932 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 1c818016/rbd_data.10936b8b4567.0000000000002e8d/head//3
2014-06-12 14:51:05.297001 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:05.307441 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:05.307475 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:05.316456 7f9d60bd2700 10 getattrs 3.16_head/1c818016/rbd_data.10936b8b4567.0000000000002e8d/head//3 = 0
2014-06-12 14:51:05.316467 7f9d60bd2700 10 stat 3.16_head/f1a18016/rbd_data.10b16b8b4567.00000000000000a6/head//3
2014-06-12 14:51:05.316575 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid f1a18016/rbd_data.10b16b8b4567.00000000000000a6/head//3
2014-06-12 14:51:05.335831 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:05.340162 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:05.366720 7f9d60bd2700 10 getattrs 3.16_head/f1a18016/rbd_data.10b16b8b4567.00000000000000a6/head//3 = 0
2014-06-12 14:51:05.366736 7f9d60bd2700 10 stat 3.16_head/2ad38016/rbd_data.10a76b8b4567.0000000000000be2/head//3
2014-06-12 14:51:05.366928 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 2ad38016/rbd_data.10a76b8b4567.0000000000000be2/head//3
2014-06-12 14:51:05.372869 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:05.388779 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:05.406211 7f9d60bd2700 10 getattrs 3.16_head/2ad38016/rbd_data.10a76b8b4567.0000000000000be2/head//3 = 0
2014-06-12 14:51:05.406221 7f9d60bd2700 10 stat 3.16_head/9ae48016/rbd_data.109c6b8b4567.00000000000027a5/head//3
2014-06-12 14:51:05.406315 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 9ae48016/rbd_data.109c6b8b4567.00000000000027a5/head//3
2014-06-12 14:51:05.431129 7f9d60bd2700 10 getattrs 3.16_head/9ae48016/rbd_data.109c6b8b4567.00000000000027a5/head//3 = 0
2014-06-12 14:51:05.431137 7f9d60bd2700 10 stat 3.16_head/1a778016/rbd_data.109e6b8b4567.0000000000001c28/head//3
2014-06-12 14:51:05.431233 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 1a778016/rbd_data.109e6b8b4567.0000000000001c28/head//3
2014-06-12 14:51:05.453090 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:05.453137 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:05.456922 7f9d60bd2700 10 getattrs 3.16_head/1a778016/rbd_data.109e6b8b4567.0000000000001c28/head//3 = 0
2014-06-12 14:51:05.456930 7f9d60bd2700 10 stat 3.16_head/6488016/rbd_data.109e6b8b4567.0000000000002c14/head//3
2014-06-12 14:51:05.457017 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 6488016/rbd_data.109e6b8b4567.0000000000002c14/head//3
2014-06-12 14:51:05.466270 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:05.466313 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:05.488831 7f9d60bd2700 10 getattrs 3.16_head/6488016/rbd_data.109e6b8b4567.0000000000002c14/head//3 = 0
2014-06-12 14:51:05.488843 7f9d60bd2700 10 stat 3.16_head/8abc8016/rbd_data.10af6b8b4567.0000000000001b0a/head//3
2014-06-12 14:51:05.488935 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 8abc8016/rbd_data.10af6b8b4567.0000000000001b0a/head//3
2014-06-12 14:51:05.489028 7f9d60bd2700 10 getattrs 3.16_head/8abc8016/rbd_data.10af6b8b4567.0000000000001b0a/head//3 = 0
2014-06-12 14:51:05.489033 7f9d60bd2700 10 stat 3.16_head/e19d8016/rbd_data.10a56b8b4567.0000000000002ac3/head//3
2014-06-12 14:51:05.489113 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid e19d8016/rbd_data.10a56b8b4567.0000000000002ac3/head//3
2014-06-12 14:51:05.506011 7f9d60bd2700 10 getattrs 3.16_head/e19d8016/rbd_data.10a56b8b4567.0000000000002ac3/head//3 = 0
2014-06-12 14:51:05.506024 7f9d60bd2700 10 collection_getattrs 3.16_head
2014-06-12 14:51:05.506081 7f9d60bd2700 10 collection_getattrs 3.16_head = 0
2014-06-12 14:51:05.506085 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:05.506119 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:05.506126 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:05.506134 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:05.506136 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:05.511485 7f9d58d5b700  5 -- op tracker -- , seq: 748559, time: 2014-06-12 14:51:05.511392, event: header_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:05.511500 7f9d58d5b700  5 -- op tracker -- , seq: 748559, time: 2014-06-12 14:51:05.511395, event: throttled, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:05.511509 7f9d58d5b700  5 -- op tracker -- , seq: 748559, time: 2014-06-12 14:51:05.511446, event: all_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:05.511536 7f9d58d5b700  5 -- op tracker -- , seq: 748559, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:05.511541 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:05.511548 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:51:05.511556 7f9d58d5b700 15 osd.2 173 enqueue_op 0x6f4b700 prio 127 cost 10717 latency 0.000164 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:05.511626 7f9d63bd8700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:05.511648 7f9d63bd8700 10 osd.2 173 dequeue_op 0x6f4b700 prio 127 cost 10717 latency 0.000256 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]
2014-06-12 14:51:05.511691 7f9d63bd8700  5 -- op tracker -- , seq: 748559, time: 2014-06-12 14:51:05.511683, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:05.511714 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] handle_message: 0x6f4b700
2014-06-12 14:51:05.511730 7f9d63bd8700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:05.511750 7f9d63bd8700  7 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:05.511768 7f9d63bd8700  5 -- op tracker -- , seq: 748559, time: 2014-06-12 14:51:05.511763, event: started, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:05.511779 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  got 22 scrub map
2014-06-12 14:51:05.511876 7f9d63bd8700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] map version is 173'2396
2014-06-12 14:51:05.511896 7f9d63bd8700 10 osd.2 173 dequeue_op 0x6f4b700 finish
2014-06-12 14:51:05.511907 7f9d63bd8700  5 -- op tracker -- , seq: 748559, time: 2014-06-12 14:51:05.511901, event: done, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:05.511928 7f9d63bd8700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:05.511932 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:05.511935 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:05.511948 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:05.511977 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:05.511988 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:05.512022 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:05.512033 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:05.512043 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub replica 22 has 25 items
2014-06-12 14:51:05.512073 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5d26016/rbd_data.109b6b8b4567.00000000000001b6/head//3, auth == maps.end()
2014-06-12 14:51:05.512094 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 5d26016/rbd_data.109b6b8b4567.00000000000001b6/head//3
2014-06-12 14:51:05.512119 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a2336016/rbd_data.10a86b8b4567.0000000000000b93/head//3, auth == maps.end()
2014-06-12 14:51:05.512136 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj a2336016/rbd_data.10a86b8b4567.0000000000000b93/head//3
2014-06-12 14:51:05.512151 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ae636016/rbd_data.10a36b8b4567.0000000000001be7/head//3, auth == maps.end()
2014-06-12 14:51:05.512164 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj ae636016/rbd_data.10a36b8b4567.0000000000001be7/head//3
2014-06-12 14:51:05.512178 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 78056016/rbd_data.10ab6b8b4567.000000000000196b/head//3, auth == maps.end()
2014-06-12 14:51:05.512191 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 78056016/rbd_data.10ab6b8b4567.000000000000196b/head//3
2014-06-12 14:51:05.512204 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2bb56016/rbd_data.109a6b8b4567.0000000000001729/head//3, auth == maps.end()
2014-06-12 14:51:05.512217 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 2bb56016/rbd_data.109a6b8b4567.0000000000001729/head//3
2014-06-12 14:51:05.512232 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3b166016/rbd_data.108f6b8b4567.0000000000002174/head//3, auth == maps.end()
2014-06-12 14:51:05.512246 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 3b166016/rbd_data.108f6b8b4567.0000000000002174/head//3
2014-06-12 14:51:05.512260 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f6ba6016/rbd_data.10a16b8b4567.0000000000001e87/head//3, auth == maps.end()
2014-06-12 14:51:05.512274 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj f6ba6016/rbd_data.10a16b8b4567.0000000000001e87/head//3
2014-06-12 14:51:05.512289 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 597b6016/rbd_data.10b06b8b4567.000000000000119f/head//3, auth == maps.end()
2014-06-12 14:51:05.512304 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 597b6016/rbd_data.10b06b8b4567.000000000000119f/head//3
2014-06-12 14:51:05.512321 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1ad6016/rbd_data.10ae6b8b4567.0000000000001333/head//3, auth == maps.end()
2014-06-12 14:51:05.512337 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 1ad6016/rbd_data.10ae6b8b4567.0000000000001333/head//3
2014-06-12 14:51:05.512351 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 467e6016/rbd_data.10b06b8b4567.0000000000002949/head//3, auth == maps.end()
2014-06-12 14:51:05.512364 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 467e6016/rbd_data.10b06b8b4567.0000000000002949/head//3
2014-06-12 14:51:05.512380 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2bd27016/rbd_data.10a36b8b4567.00000000000000b0/head//3, auth == maps.end()
2014-06-12 14:51:05.512394 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 2bd27016/rbd_data.10a36b8b4567.00000000000000b0/head//3
2014-06-12 14:51:05.512410 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 89137016/rbd_data.10a26b8b4567.0000000000001f6f/head//3, auth == maps.end()
2014-06-12 14:51:05.512429 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 89137016/rbd_data.10a26b8b4567.0000000000001f6f/head//3
2014-06-12 14:51:05.512443 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8b437016/rbd_data.10966b8b4567.00000000000036cf/head//3, auth == maps.end()
2014-06-12 14:51:05.512456 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 8b437016/rbd_data.10966b8b4567.00000000000036cf/head//3
2014-06-12 14:51:05.512470 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 78777016/rbd_data.109e6b8b4567.0000000000001fd3/head//3, auth == maps.end()
2014-06-12 14:51:05.512483 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 78777016/rbd_data.109e6b8b4567.0000000000001fd3/head//3
2014-06-12 14:51:05.512496 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8987016/rbd_data.10a96b8b4567.0000000000000585/head//3, auth == maps.end()
2014-06-12 14:51:05.512509 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 8987016/rbd_data.10a96b8b4567.0000000000000585/head//3
2014-06-12 14:51:05.512523 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c00a7016/rbd_data.108d6b8b4567.000000000000067e/head//3, auth == maps.end()
2014-06-12 14:51:05.512536 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj c00a7016/rbd_data.108d6b8b4567.000000000000067e/head//3
2014-06-12 14:51:05.512549 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cada7016/rbd_data.109c6b8b4567.0000000000001b2b/head//3, auth == maps.end()
2014-06-12 14:51:05.512564 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj cada7016/rbd_data.109c6b8b4567.0000000000001b2b/head//3
2014-06-12 14:51:05.512579 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1c818016/rbd_data.10936b8b4567.0000000000002e8d/head//3, auth == maps.end()
2014-06-12 14:51:05.512594 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 1c818016/rbd_data.10936b8b4567.0000000000002e8d/head//3
2014-06-12 14:51:05.512610 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f1a18016/rbd_data.10b16b8b4567.00000000000000a6/head//3, auth == maps.end()
2014-06-12 14:51:05.512624 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj f1a18016/rbd_data.10b16b8b4567.00000000000000a6/head//3
2014-06-12 14:51:05.512640 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2ad38016/rbd_data.10a76b8b4567.0000000000000be2/head//3, auth == maps.end()
2014-06-12 14:51:05.512655 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 2ad38016/rbd_data.10a76b8b4567.0000000000000be2/head//3
2014-06-12 14:51:05.512672 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9ae48016/rbd_data.109c6b8b4567.00000000000027a5/head//3, auth == maps.end()
2014-06-12 14:51:05.512687 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 9ae48016/rbd_data.109c6b8b4567.00000000000027a5/head//3
2014-06-12 14:51:05.512701 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1a778016/rbd_data.109e6b8b4567.0000000000001c28/head//3, auth == maps.end()
2014-06-12 14:51:05.512714 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 1a778016/rbd_data.109e6b8b4567.0000000000001c28/head//3
2014-06-12 14:51:05.512730 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6488016/rbd_data.109e6b8b4567.0000000000002c14/head//3, auth == maps.end()
2014-06-12 14:51:05.512746 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 6488016/rbd_data.109e6b8b4567.0000000000002c14/head//3
2014-06-12 14:51:05.512766 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8abc8016/rbd_data.10af6b8b4567.0000000000001b0a/head//3, auth == maps.end()
2014-06-12 14:51:05.512782 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 8abc8016/rbd_data.10af6b8b4567.0000000000001b0a/head//3
2014-06-12 14:51:05.512799 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e19d8016/rbd_data.10a56b8b4567.0000000000002ac3/head//3, auth == maps.end()
2014-06-12 14:51:05.512814 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj e19d8016/rbd_data.10a56b8b4567.0000000000002ac3/head//3
2014-06-12 14:51:05.512833 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] 
2014-06-12 14:51:05.512846 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub
2014-06-12 14:51:05.512864 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  e19d8016/rbd_data.10a56b8b4567.0000000000002ac3/head//3 e19d8016/rbd_data.10a56b8b4567.0000000000002ac3/head//3(173'1557 client.4302.0:87727 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1557)
2014-06-12 14:51:05.512886 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  8abc8016/rbd_data.10af6b8b4567.0000000000001b0a/head//3 8abc8016/rbd_data.10af6b8b4567.0000000000001b0a/head//3(173'1058 client.4312.0:61029 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1058)
2014-06-12 14:51:05.512906 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  6488016/rbd_data.109e6b8b4567.0000000000002c14/head//3 6488016/rbd_data.109e6b8b4567.0000000000002c14/head//3(173'1537 client.4295.0:89507 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1537)
2014-06-12 14:51:05.512926 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  1a778016/rbd_data.109e6b8b4567.0000000000001c28/head//3 1a778016/rbd_data.109e6b8b4567.0000000000001c28/head//3(173'1035 client.4295.0:61027 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1035)
2014-06-12 14:51:05.512945 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  9ae48016/rbd_data.109c6b8b4567.00000000000027a5/head//3 9ae48016/rbd_data.109c6b8b4567.00000000000027a5/head//3(173'1435 client.4293.0:80603 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1435)
2014-06-12 14:51:05.512965 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  2ad38016/rbd_data.10a76b8b4567.0000000000000be2/head//3 2ad38016/rbd_data.10a76b8b4567.0000000000000be2/head//3(173'2308 client.4304.0:136510 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2308)
2014-06-12 14:51:05.512984 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  f1a18016/rbd_data.10b16b8b4567.00000000000000a6/head//3 f1a18016/rbd_data.10b16b8b4567.00000000000000a6/head//3(173'2029 client.4314.0:119018 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2029)
2014-06-12 14:51:05.513003 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  1c818016/rbd_data.10936b8b4567.0000000000002e8d/head//3 1c818016/rbd_data.10936b8b4567.0000000000002e8d/head//3(173'1657 client.4284.0:91593 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1657)
2014-06-12 14:51:05.513022 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  cada7016/rbd_data.109c6b8b4567.0000000000001b2b/head//3 cada7016/rbd_data.109c6b8b4567.0000000000001b2b/head//3(173'1069 client.4293.0:58077 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1069)
2014-06-12 14:51:05.513041 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  c00a7016/rbd_data.108d6b8b4567.000000000000067e/head//3 c00a7016/rbd_data.108d6b8b4567.000000000000067e/head//3(173'2178 client.4278.0:127957 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2178)
2014-06-12 14:51:05.513060 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  8987016/rbd_data.10a96b8b4567.0000000000000585/head//3 8987016/rbd_data.10a96b8b4567.0000000000000585/head//3(173'2187 client.4306.0:126635 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2187)
2014-06-12 14:51:05.513079 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  78777016/rbd_data.109e6b8b4567.0000000000001fd3/head//3 78777016/rbd_data.109e6b8b4567.0000000000001fd3/head//3(173'1182 client.4295.0:67668 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1182)
2014-06-12 14:51:05.513098 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  8b437016/rbd_data.10966b8b4567.00000000000036cf/head//3 8b437016/rbd_data.10966b8b4567.00000000000036cf/head//3(173'1845 client.4287.0:105609 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1845)
2014-06-12 14:51:05.513117 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  89137016/rbd_data.10a26b8b4567.0000000000001f6f/head//3 89137016/rbd_data.10a26b8b4567.0000000000001f6f/head//3(173'1252 client.4299.0:67461 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1252)
2014-06-12 14:51:05.513136 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  2bd27016/rbd_data.10a36b8b4567.00000000000000b0/head//3 2bd27016/rbd_data.10a36b8b4567.00000000000000b0/head//3(173'2024 client.4300.0:116088 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2024)
2014-06-12 14:51:05.513155 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  467e6016/rbd_data.10b06b8b4567.0000000000002949/head//3 467e6016/rbd_data.10b06b8b4567.0000000000002949/head//3(173'1416 client.4313.0:84881 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1416)
2014-06-12 14:51:05.513174 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  1ad6016/rbd_data.10ae6b8b4567.0000000000001333/head//3 1ad6016/rbd_data.10ae6b8b4567.0000000000001333/head//3(173'690 client.4311.0:43577 wrlock_by=unknown.0.0:0 dirty s 4194304 uv690)
2014-06-12 14:51:05.513193 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  597b6016/rbd_data.10b06b8b4567.000000000000119f/head//3 597b6016/rbd_data.10b06b8b4567.000000000000119f/head//3(173'658 client.4313.0:41820 wrlock_by=unknown.0.0:0 dirty s 4194304 uv658)
2014-06-12 14:51:05.513212 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  f6ba6016/rbd_data.10a16b8b4567.0000000000001e87/head//3 f6ba6016/rbd_data.10a16b8b4567.0000000000001e87/head//3(173'1124 client.4298.0:65277 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1124)
2014-06-12 14:51:05.513231 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  3b166016/rbd_data.108f6b8b4567.0000000000002174/head//3 3b166016/rbd_data.108f6b8b4567.0000000000002174/head//3(173'1233 client.4280.0:68804 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1233)
2014-06-12 14:51:05.513254 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  2bb56016/rbd_data.109a6b8b4567.0000000000001729/head//3 2bb56016/rbd_data.109a6b8b4567.0000000000001729/head//3(173'820 client.4291.0:50908 wrlock_by=unknown.0.0:0 dirty s 4194304 uv820)
2014-06-12 14:51:05.513273 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  78056016/rbd_data.10ab6b8b4567.000000000000196b/head//3 78056016/rbd_data.10ab6b8b4567.000000000000196b/head//3(173'951 client.4308.0:56019 wrlock_by=unknown.0.0:0 dirty s 4194304 uv951)
2014-06-12 14:51:05.513293 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  ae636016/rbd_data.10a36b8b4567.0000000000001be7/head//3 ae636016/rbd_data.10a36b8b4567.0000000000001be7/head//3(173'967 client.4300.0:58083 wrlock_by=unknown.0.0:0 dirty s 4194304 uv967)
2014-06-12 14:51:05.513311 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  a2336016/rbd_data.10a86b8b4567.0000000000000b93/head//3 a2336016/rbd_data.10a86b8b4567.0000000000000b93/head//3(173'2336 client.4305.0:136638 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2336)
2014-06-12 14:51:05.513330 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  5d26016/rbd_data.109b6b8b4567.00000000000001b6/head//3 5d26016/rbd_data.109b6b8b4567.00000000000001b6/head//3(173'2060 client.4292.0:117814 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2060)
2014-06-12 14:51:05.513346 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:05.513361 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:05.513375 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:05.513378 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:05.513380 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:05.513435 7f9d60bd2700 10 collection_list_partial 3.16_head start:47279016//0//-1 is_max:0
2014-06-12 14:51:05.513761 7f9d60bd2700 20 objects: [8bb79016/rbd_data.108d6b8b4567.0000000000000ef0/head//3,917b9016/rbd_data.10906b8b4567.00000000000013ea/head//3,7d1e9016/rbd_data.10ad6b8b4567.000000000000304b/head//3,a7fe9016/rbd_data.10ac6b8b4567.0000000000003685/head//3,835f9016/rbd_data.109e6b8b4567.0000000000001771/head//3,69f1a016/rbd_data.10ac6b8b4567.000000000000382f/head//3,d0d2a016/rbd_data.10a66b8b4567.0000000000002934/head//3,9f44a016/rbd_data.109b6b8b4567.0000000000000721/head//3,6b86a016/rbd_data.10916b8b4567.00000000000014aa/head//3,31a9a016/rbd_data.10a86b8b4567.0000000000002f08/head//3,215aa016/rbd_data.109a6b8b4567.0000000000001af5/head//3,b5cca016/rbd_data.10a96b8b4567.000000000000115a/head//3,be4da016/rbd_data.10ab6b8b4567.0000000000002ee4/head//3,b4cea016/rbd_data.109c6b8b4567.0000000000000616/head//3,ac32b016/rbd_data.10b16b8b4567.00000000000038a7/head//3,d4e3b016/rbd_data.10b16b8b4567.0000000000001422/head//3,5505b016/rbd_data.109b6b8b4567.0000000000002768/head//3,f7c5b016/rbd_data.10aa6b8b4567.0000000000002543/head//3,5256b016/rbd_data.10ab6b8b4567.0000000000002d29/head//3,f929b016/rbd_data.10a56b8b4567.00000000000017a7/head//3,5dfab016/rbd_data.10a66b8b4567.0000000000002515/head//3,ebbbb016/rbd_data.109c6b8b4567.00000000000034f7/head//3,fc5db016/rbd_data.10a06b8b4567.000000000000284e/head//3,8600c016/rbd_data.10a76b8b4567.0000000000001cb2/head//3,a4a3c016/rbd_data.10976b8b4567.0000000000002326/head//3]
2014-06-12 14:51:05.514438 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  requesting scrubmap from osd.22
2014-06-12 14:51:05.514493 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:05.514527 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:05.514543 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:05.514555 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:05.514566 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:05.514576 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_map_chunk [47279016//0//-1,b8f3c016//0//-1)
2014-06-12 14:51:05.514589 7f9d60bd2700 10 collection_list_partial 3.16_head start:47279016//0//-1 is_max:0
2014-06-12 14:51:05.515480 7f9d60bd2700 20 objects: [8bb79016/rbd_data.108d6b8b4567.0000000000000ef0/head//3,917b9016/rbd_data.10906b8b4567.00000000000013ea/head//3,7d1e9016/rbd_data.10ad6b8b4567.000000000000304b/head//3,a7fe9016/rbd_data.10ac6b8b4567.0000000000003685/head//3,835f9016/rbd_data.109e6b8b4567.0000000000001771/head//3,69f1a016/rbd_data.10ac6b8b4567.000000000000382f/head//3,d0d2a016/rbd_data.10a66b8b4567.0000000000002934/head//3,9f44a016/rbd_data.109b6b8b4567.0000000000000721/head//3,6b86a016/rbd_data.10916b8b4567.00000000000014aa/head//3,31a9a016/rbd_data.10a86b8b4567.0000000000002f08/head//3,215aa016/rbd_data.109a6b8b4567.0000000000001af5/head//3,b5cca016/rbd_data.10a96b8b4567.000000000000115a/head//3,be4da016/rbd_data.10ab6b8b4567.0000000000002ee4/head//3,b4cea016/rbd_data.109c6b8b4567.0000000000000616/head//3,ac32b016/rbd_data.10b16b8b4567.00000000000038a7/head//3,d4e3b016/rbd_data.10b16b8b4567.0000000000001422/head//3,5505b016/rbd_data.109b6b8b4567.0000000000002768/head//3,f7c5b016/rbd_data.10aa6b8b4567.0000000000002543/head//3,5256b016/rbd_data.10ab6b8b4567.0000000000002d29/head//3,f929b016/rbd_data.10a56b8b4567.00000000000017a7/head//3,5dfab016/rbd_data.10a66b8b4567.0000000000002515/head//3,ebbbb016/rbd_data.109c6b8b4567.00000000000034f7/head//3,fc5db016/rbd_data.10a06b8b4567.000000000000284e/head//3,8600c016/rbd_data.10a76b8b4567.0000000000001cb2/head//3,a4a3c016/rbd_data.10976b8b4567.0000000000002326/head//3,b8f3c016/rbd_data.109d6b8b4567.00000000000030d1/head//3,f898c016/rbd_data.10a86b8b4567.0000000000001a5b/head//3,c2f8c016/rbd_data.10a56b8b4567.000000000000249d/head//3,cacc016/rbd_data.10a56b8b4567.000000000000088e/head//3,20cdc016/rbd_data.10a76b8b4567.0000000000003570/head//3,6591d016/rbd_data.10936b8b4567.0000000000002355/head//3,7c1d016/rbd_data.109d6b8b4567.0000000000001891/head//3,cfc2d016/rbd_data.10946b8b4567.00000000000017ce/head//3,3ed2d016/rbd_data.10926b8b4567.0000000000002610/head//3,d493d016/rbd_data.10956b8b4567.0000000000000bf2/head//3,e824d016/rbd_data.10a56b8b4567.00000000000021c1/head//3,335ad016/rbd_data.109c6b8b4567.00000000000017d4/head//3,a6abd016/rbd_data.109e6b8b4567.00000000000038a6/head//3,202fd016/rbd_data.10986b8b4567.00000000000038d5/head//3,a36fd016/rbd_data.109f6b8b4567.000000000000218b/head//3,6052e016/rbd_data.10a36b8b4567.0000000000001952/head//3,a993e016/rbd_data.10a16b8b4567.000000000000041a/head//3,94c3e016/rbd_data.108f6b8b4567.0000000000002a08/head//3,54e016/rbd_data.10aa6b8b4567.000000000000016b/head//3,18b5e016/rbd_data.10966b8b4567.0000000000003621/head//3,6216e016/rbd_data.10936b8b4567.00000000000035e5/head//3,f5f6e016/rbd_data.109b6b8b4567.0000000000001357/head//3,4d69e016/rbd_data.10996b8b4567.0000000000001357/head//3,21e9e016/rbd_data.109f6b8b4567.00000000000017f2/head//3,a14ee016/rbd_data.10b16b8b4567.0000000000002d30/head//3,3df3f016/rbd_data.10a86b8b4567.0000000000000bbd/head//3,47f8f016/rbd_data.10b46b8b4567.000000000000153c/head//3,a70df016/rbd_data.10aa6b8b4567.0000000000000308/head//3,c93df016/rbd_data.10986b8b4567.000000000000083f/head//3,324ef016/rbd_data.109e6b8b4567.0000000000003306/head//3,16ef016/rbd_data.10a36b8b4567.0000000000003103/head//3,9abef016/rbd_data.109c6b8b4567.0000000000002b65/head//3,64d10816/rbd_data.109c6b8b4567.000000000000294e/head//3,38130816/rbd_data.10a06b8b4567.0000000000002ca1/head//3,22f30816/rbd_data.109a6b8b4567.0000000000000131/head//3,da570816/rbd_data.10ae6b8b4567.0000000000001d51/head//3,26190816/rbd_data.10b06b8b4567.0000000000001834/head//3,1dc90816/rbd_data.10926b8b4567.0000000000001ea6/head//3,7a3b0816/rbd_data.10916b8b4567.000000000000072c/head//3]
2014-06-12 14:51:05.515615 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:05.515646 7f9d60bd2700 10 stat 3.16_head/8bb79016/rbd_data.108d6b8b4567.0000000000000ef0/head//3
2014-06-12 14:51:05.515868 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 8bb79016/rbd_data.108d6b8b4567.0000000000000ef0/head//3
2014-06-12 14:51:05.546978 7f9d60bd2700 10 getattrs 3.16_head/8bb79016/rbd_data.108d6b8b4567.0000000000000ef0/head//3 = 0
2014-06-12 14:51:05.546994 7f9d60bd2700 10 stat 3.16_head/917b9016/rbd_data.10906b8b4567.00000000000013ea/head//3
2014-06-12 14:51:05.547148 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 917b9016/rbd_data.10906b8b4567.00000000000013ea/head//3
2014-06-12 14:51:05.574490 7f9d60bd2700 10 getattrs 3.16_head/917b9016/rbd_data.10906b8b4567.00000000000013ea/head//3 = 0
2014-06-12 14:51:05.574499 7f9d60bd2700 10 stat 3.16_head/7d1e9016/rbd_data.10ad6b8b4567.000000000000304b/head//3
2014-06-12 14:51:05.574601 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 7d1e9016/rbd_data.10ad6b8b4567.000000000000304b/head//3
2014-06-12 14:51:05.591979 7f9d60bd2700 10 getattrs 3.16_head/7d1e9016/rbd_data.10ad6b8b4567.000000000000304b/head//3 = 0
2014-06-12 14:51:05.591988 7f9d60bd2700 10 stat 3.16_head/a7fe9016/rbd_data.10ac6b8b4567.0000000000003685/head//3
2014-06-12 14:51:05.592092 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid a7fe9016/rbd_data.10ac6b8b4567.0000000000003685/head//3
2014-06-12 14:51:05.620098 7f9d60bd2700 10 getattrs 3.16_head/a7fe9016/rbd_data.10ac6b8b4567.0000000000003685/head//3 = 0
2014-06-12 14:51:05.620109 7f9d60bd2700 10 stat 3.16_head/835f9016/rbd_data.109e6b8b4567.0000000000001771/head//3
2014-06-12 14:51:05.620211 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 835f9016/rbd_data.109e6b8b4567.0000000000001771/head//3
2014-06-12 14:51:05.656981 7f9d60bd2700 10 getattrs 3.16_head/835f9016/rbd_data.109e6b8b4567.0000000000001771/head//3 = 0
2014-06-12 14:51:05.657002 7f9d60bd2700 10 stat 3.16_head/69f1a016/rbd_data.10ac6b8b4567.000000000000382f/head//3
2014-06-12 14:51:05.657135 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 69f1a016/rbd_data.10ac6b8b4567.000000000000382f/head//3
2014-06-12 14:51:05.670804 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:05.671080 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:05.674733 7f9d60bd2700 10 getattrs 3.16_head/69f1a016/rbd_data.10ac6b8b4567.000000000000382f/head//3 = 0
2014-06-12 14:51:05.674744 7f9d60bd2700 10 stat 3.16_head/d0d2a016/rbd_data.10a66b8b4567.0000000000002934/head//3
2014-06-12 14:51:05.674877 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid d0d2a016/rbd_data.10a66b8b4567.0000000000002934/head//3
2014-06-12 14:51:05.686776 7f9d60bd2700 10 getattrs 3.16_head/d0d2a016/rbd_data.10a66b8b4567.0000000000002934/head//3 = 0
2014-06-12 14:51:05.686788 7f9d60bd2700 10 stat 3.16_head/9f44a016/rbd_data.109b6b8b4567.0000000000000721/head//3
2014-06-12 14:51:05.686919 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 9f44a016/rbd_data.109b6b8b4567.0000000000000721/head//3
2014-06-12 14:51:05.717248 7f9d60bd2700 10 getattrs 3.16_head/9f44a016/rbd_data.109b6b8b4567.0000000000000721/head//3 = 0
2014-06-12 14:51:05.717260 7f9d60bd2700 10 stat 3.16_head/6b86a016/rbd_data.10916b8b4567.00000000000014aa/head//3
2014-06-12 14:51:05.717376 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 6b86a016/rbd_data.10916b8b4567.00000000000014aa/head//3
2014-06-12 14:51:05.727122 7f9d60bd2700 10 getattrs 3.16_head/6b86a016/rbd_data.10916b8b4567.00000000000014aa/head//3 = 0
2014-06-12 14:51:05.727133 7f9d60bd2700 10 stat 3.16_head/31a9a016/rbd_data.10a86b8b4567.0000000000002f08/head//3
2014-06-12 14:51:05.727256 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 31a9a016/rbd_data.10a86b8b4567.0000000000002f08/head//3
2014-06-12 14:51:05.741877 7f9d60bd2700 10 getattrs 3.16_head/31a9a016/rbd_data.10a86b8b4567.0000000000002f08/head//3 = 0
2014-06-12 14:51:05.741888 7f9d60bd2700 10 stat 3.16_head/215aa016/rbd_data.109a6b8b4567.0000000000001af5/head//3
2014-06-12 14:51:05.742012 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 215aa016/rbd_data.109a6b8b4567.0000000000001af5/head//3
2014-06-12 14:51:05.753889 7f9d60bd2700 10 getattrs 3.16_head/215aa016/rbd_data.109a6b8b4567.0000000000001af5/head//3 = 0
2014-06-12 14:51:05.753902 7f9d60bd2700 10 stat 3.16_head/b5cca016/rbd_data.10a96b8b4567.000000000000115a/head//3
2014-06-12 14:51:05.754028 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid b5cca016/rbd_data.10a96b8b4567.000000000000115a/head//3
2014-06-12 14:51:05.782714 7f9d60bd2700 10 getattrs 3.16_head/b5cca016/rbd_data.10a96b8b4567.000000000000115a/head//3 = 0
2014-06-12 14:51:05.782721 7f9d60bd2700 10 stat 3.16_head/be4da016/rbd_data.10ab6b8b4567.0000000000002ee4/head//3
2014-06-12 14:51:05.782814 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid be4da016/rbd_data.10ab6b8b4567.0000000000002ee4/head//3
2014-06-12 14:51:05.801928 7f9d60bd2700 10 getattrs 3.16_head/be4da016/rbd_data.10ab6b8b4567.0000000000002ee4/head//3 = 0
2014-06-12 14:51:05.801937 7f9d60bd2700 10 stat 3.16_head/b4cea016/rbd_data.109c6b8b4567.0000000000000616/head//3
2014-06-12 14:51:05.802044 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid b4cea016/rbd_data.109c6b8b4567.0000000000000616/head//3
2014-06-12 14:51:05.825493 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:05.825680 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:05.834942 7f9d60bd2700 10 getattrs 3.16_head/b4cea016/rbd_data.109c6b8b4567.0000000000000616/head//3 = 0
2014-06-12 14:51:05.834960 7f9d60bd2700 10 stat 3.16_head/ac32b016/rbd_data.10b16b8b4567.00000000000038a7/head//3
2014-06-12 14:51:05.835065 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid ac32b016/rbd_data.10b16b8b4567.00000000000038a7/head//3
2014-06-12 14:51:05.851436 7f9d60bd2700 10 getattrs 3.16_head/ac32b016/rbd_data.10b16b8b4567.00000000000038a7/head//3 = 0
2014-06-12 14:51:05.851450 7f9d60bd2700 10 stat 3.16_head/d4e3b016/rbd_data.10b16b8b4567.0000000000001422/head//3
2014-06-12 14:51:05.851610 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid d4e3b016/rbd_data.10b16b8b4567.0000000000001422/head//3
2014-06-12 14:51:05.869023 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:05.869049 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:05.869054 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:05.869056 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:05.869056 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:05.871754 7f9d60bd2700 10 getattrs 3.16_head/d4e3b016/rbd_data.10b16b8b4567.0000000000001422/head//3 = 0
2014-06-12 14:51:05.871761 7f9d60bd2700 10 stat 3.16_head/5505b016/rbd_data.109b6b8b4567.0000000000002768/head//3
2014-06-12 14:51:05.871840 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 5505b016/rbd_data.109b6b8b4567.0000000000002768/head//3
2014-06-12 14:51:05.886949 7f9d60bd2700 10 getattrs 3.16_head/5505b016/rbd_data.109b6b8b4567.0000000000002768/head//3 = 0
2014-06-12 14:51:05.886960 7f9d60bd2700 10 stat 3.16_head/f7c5b016/rbd_data.10aa6b8b4567.0000000000002543/head//3
2014-06-12 14:51:05.887058 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid f7c5b016/rbd_data.10aa6b8b4567.0000000000002543/head//3
2014-06-12 14:51:05.902896 7f9d60bd2700 10 getattrs 3.16_head/f7c5b016/rbd_data.10aa6b8b4567.0000000000002543/head//3 = 0
2014-06-12 14:51:05.902910 7f9d60bd2700 10 stat 3.16_head/5256b016/rbd_data.10ab6b8b4567.0000000000002d29/head//3
2014-06-12 14:51:05.903067 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 5256b016/rbd_data.10ab6b8b4567.0000000000002d29/head//3
2014-06-12 14:51:05.936508 7f9d60bd2700 10 getattrs 3.16_head/5256b016/rbd_data.10ab6b8b4567.0000000000002d29/head//3 = 0
2014-06-12 14:51:05.936518 7f9d60bd2700 10 stat 3.16_head/f929b016/rbd_data.10a56b8b4567.00000000000017a7/head//3
2014-06-12 14:51:05.936619 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid f929b016/rbd_data.10a56b8b4567.00000000000017a7/head//3
2014-06-12 14:51:05.964953 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:05.965013 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:05.966665 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:05.966698 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:05.969180 7f9d60bd2700 10 getattrs 3.16_head/f929b016/rbd_data.10a56b8b4567.00000000000017a7/head//3 = 0
2014-06-12 14:51:05.969190 7f9d60bd2700 10 stat 3.16_head/5dfab016/rbd_data.10a66b8b4567.0000000000002515/head//3
2014-06-12 14:51:05.969281 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 5dfab016/rbd_data.10a66b8b4567.0000000000002515/head//3
2014-06-12 14:51:05.975566 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:06.000032 7f9d60bd2700 10 getattrs 3.16_head/5dfab016/rbd_data.10a66b8b4567.0000000000002515/head//3 = 0
2014-06-12 14:51:06.000039 7f9d60bd2700 10 stat 3.16_head/ebbbb016/rbd_data.109c6b8b4567.00000000000034f7/head//3
2014-06-12 14:51:06.000135 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid ebbbb016/rbd_data.109c6b8b4567.00000000000034f7/head//3
2014-06-12 14:51:06.006954 7f9d58d5b700  5 -- op tracker -- , seq: 748560, time: 2014-06-12 14:51:06.006878, event: header_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.006971 7f9d58d5b700  5 -- op tracker -- , seq: 748560, time: 2014-06-12 14:51:06.006881, event: throttled, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.006977 7f9d58d5b700  5 -- op tracker -- , seq: 748560, time: 2014-06-12 14:51:06.006930, event: all_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.006981 7f9d58d5b700  5 -- op tracker -- , seq: 748560, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.006992 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:06.006997 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:51:06.007001 7f9d58d5b700 15 osd.2 173 enqueue_op 0x6f4bf00 prio 127 cost 10717 latency 0.000123 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:06.007053 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:06.011909 7f9d60bd2700 10 getattrs 3.16_head/ebbbb016/rbd_data.109c6b8b4567.00000000000034f7/head//3 = 0
2014-06-12 14:51:06.011917 7f9d60bd2700 10 stat 3.16_head/fc5db016/rbd_data.10a06b8b4567.000000000000284e/head//3
2014-06-12 14:51:06.012007 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid fc5db016/rbd_data.10a06b8b4567.000000000000284e/head//3
2014-06-12 14:51:06.037024 7f9d60bd2700 10 getattrs 3.16_head/fc5db016/rbd_data.10a06b8b4567.000000000000284e/head//3 = 0
2014-06-12 14:51:06.037038 7f9d60bd2700 10 stat 3.16_head/8600c016/rbd_data.10a76b8b4567.0000000000001cb2/head//3
2014-06-12 14:51:06.037195 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 8600c016/rbd_data.10a76b8b4567.0000000000001cb2/head//3
2014-06-12 14:51:06.063365 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:06.072469 7f9d60bd2700 10 getattrs 3.16_head/8600c016/rbd_data.10a76b8b4567.0000000000001cb2/head//3 = 0
2014-06-12 14:51:06.072486 7f9d60bd2700 10 stat 3.16_head/a4a3c016/rbd_data.10976b8b4567.0000000000002326/head//3
2014-06-12 14:51:06.072642 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid a4a3c016/rbd_data.10976b8b4567.0000000000002326/head//3
2014-06-12 14:51:06.096002 7f9d60bd2700 10 getattrs 3.16_head/a4a3c016/rbd_data.10976b8b4567.0000000000002326/head//3 = 0
2014-06-12 14:51:06.096011 7f9d60bd2700 10 collection_getattrs 3.16_head
2014-06-12 14:51:06.096064 7f9d60bd2700 10 collection_getattrs 3.16_head = 0
2014-06-12 14:51:06.096069 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:06.096090 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:06.096111 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:06.096118 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:06.096121 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:06.096168 7f9d61bd4700 10 osd.2 173 dequeue_op 0x6f4bf00 prio 127 cost 10717 latency 0.089290 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]
2014-06-12 14:51:06.096224 7f9d61bd4700  5 -- op tracker -- , seq: 748560, time: 2014-06-12 14:51:06.096216, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.096247 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] handle_message: 0x6f4bf00
2014-06-12 14:51:06.096268 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:06.096290 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:06.096309 7f9d61bd4700  5 -- op tracker -- , seq: 748560, time: 2014-06-12 14:51:06.096303, event: started, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.096317 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  got 22 scrub map
2014-06-12 14:51:06.096419 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] map version is 173'2396
2014-06-12 14:51:06.096440 7f9d61bd4700 10 osd.2 173 dequeue_op 0x6f4bf00 finish
2014-06-12 14:51:06.096451 7f9d61bd4700  5 -- op tracker -- , seq: 748560, time: 2014-06-12 14:51:06.096445, event: done, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.096474 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:06.096479 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:06.096477 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:06.096493 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:06.096550 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:06.096556 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:06.096576 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:06.096585 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:06.096591 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub replica 22 has 25 items
2014-06-12 14:51:06.096608 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8bb79016/rbd_data.108d6b8b4567.0000000000000ef0/head//3, auth == maps.end()
2014-06-12 14:51:06.096630 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 8bb79016/rbd_data.108d6b8b4567.0000000000000ef0/head//3
2014-06-12 14:51:06.096648 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 917b9016/rbd_data.10906b8b4567.00000000000013ea/head//3, auth == maps.end()
2014-06-12 14:51:06.096656 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 917b9016/rbd_data.10906b8b4567.00000000000013ea/head//3
2014-06-12 14:51:06.096664 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7d1e9016/rbd_data.10ad6b8b4567.000000000000304b/head//3, auth == maps.end()
2014-06-12 14:51:06.096670 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 7d1e9016/rbd_data.10ad6b8b4567.000000000000304b/head//3
2014-06-12 14:51:06.096678 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a7fe9016/rbd_data.10ac6b8b4567.0000000000003685/head//3, auth == maps.end()
2014-06-12 14:51:06.096684 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj a7fe9016/rbd_data.10ac6b8b4567.0000000000003685/head//3
2014-06-12 14:51:06.096692 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 835f9016/rbd_data.109e6b8b4567.0000000000001771/head//3, auth == maps.end()
2014-06-12 14:51:06.096698 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 835f9016/rbd_data.109e6b8b4567.0000000000001771/head//3
2014-06-12 14:51:06.096706 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 69f1a016/rbd_data.10ac6b8b4567.000000000000382f/head//3, auth == maps.end()
2014-06-12 14:51:06.096715 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 69f1a016/rbd_data.10ac6b8b4567.000000000000382f/head//3
2014-06-12 14:51:06.096722 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d0d2a016/rbd_data.10a66b8b4567.0000000000002934/head//3, auth == maps.end()
2014-06-12 14:51:06.096729 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj d0d2a016/rbd_data.10a66b8b4567.0000000000002934/head//3
2014-06-12 14:51:06.096736 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9f44a016/rbd_data.109b6b8b4567.0000000000000721/head//3, auth == maps.end()
2014-06-12 14:51:06.096743 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 9f44a016/rbd_data.109b6b8b4567.0000000000000721/head//3
2014-06-12 14:51:06.096750 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6b86a016/rbd_data.10916b8b4567.00000000000014aa/head//3, auth == maps.end()
2014-06-12 14:51:06.096757 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 6b86a016/rbd_data.10916b8b4567.00000000000014aa/head//3
2014-06-12 14:51:06.096764 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 31a9a016/rbd_data.10a86b8b4567.0000000000002f08/head//3, auth == maps.end()
2014-06-12 14:51:06.096771 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 31a9a016/rbd_data.10a86b8b4567.0000000000002f08/head//3
2014-06-12 14:51:06.096778 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 215aa016/rbd_data.109a6b8b4567.0000000000001af5/head//3, auth == maps.end()
2014-06-12 14:51:06.096785 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 215aa016/rbd_data.109a6b8b4567.0000000000001af5/head//3
2014-06-12 14:51:06.096796 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b5cca016/rbd_data.10a96b8b4567.000000000000115a/head//3, auth == maps.end()
2014-06-12 14:51:06.096805 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj b5cca016/rbd_data.10a96b8b4567.000000000000115a/head//3
2014-06-12 14:51:06.096812 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj be4da016/rbd_data.10ab6b8b4567.0000000000002ee4/head//3, auth == maps.end()
2014-06-12 14:51:06.096820 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj be4da016/rbd_data.10ab6b8b4567.0000000000002ee4/head//3
2014-06-12 14:51:06.096828 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b4cea016/rbd_data.109c6b8b4567.0000000000000616/head//3, auth == maps.end()
2014-06-12 14:51:06.096836 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj b4cea016/rbd_data.109c6b8b4567.0000000000000616/head//3
2014-06-12 14:51:06.096844 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ac32b016/rbd_data.10b16b8b4567.00000000000038a7/head//3, auth == maps.end()
2014-06-12 14:51:06.096854 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj ac32b016/rbd_data.10b16b8b4567.00000000000038a7/head//3
2014-06-12 14:51:06.096862 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d4e3b016/rbd_data.10b16b8b4567.0000000000001422/head//3, auth == maps.end()
2014-06-12 14:51:06.096869 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj d4e3b016/rbd_data.10b16b8b4567.0000000000001422/head//3
2014-06-12 14:51:06.096877 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5505b016/rbd_data.109b6b8b4567.0000000000002768/head//3, auth == maps.end()
2014-06-12 14:51:06.096885 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 5505b016/rbd_data.109b6b8b4567.0000000000002768/head//3
2014-06-12 14:51:06.096894 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f7c5b016/rbd_data.10aa6b8b4567.0000000000002543/head//3, auth == maps.end()
2014-06-12 14:51:06.096902 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj f7c5b016/rbd_data.10aa6b8b4567.0000000000002543/head//3
2014-06-12 14:51:06.096910 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5256b016/rbd_data.10ab6b8b4567.0000000000002d29/head//3, auth == maps.end()
2014-06-12 14:51:06.096919 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 5256b016/rbd_data.10ab6b8b4567.0000000000002d29/head//3
2014-06-12 14:51:06.096927 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f929b016/rbd_data.10a56b8b4567.00000000000017a7/head//3, auth == maps.end()
2014-06-12 14:51:06.096934 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj f929b016/rbd_data.10a56b8b4567.00000000000017a7/head//3
2014-06-12 14:51:06.096942 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5dfab016/rbd_data.10a66b8b4567.0000000000002515/head//3, auth == maps.end()
2014-06-12 14:51:06.096948 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 5dfab016/rbd_data.10a66b8b4567.0000000000002515/head//3
2014-06-12 14:51:06.096957 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ebbbb016/rbd_data.109c6b8b4567.00000000000034f7/head//3, auth == maps.end()
2014-06-12 14:51:06.096964 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj ebbbb016/rbd_data.109c6b8b4567.00000000000034f7/head//3
2014-06-12 14:51:06.096973 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fc5db016/rbd_data.10a06b8b4567.000000000000284e/head//3, auth == maps.end()
2014-06-12 14:51:06.096981 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj fc5db016/rbd_data.10a06b8b4567.000000000000284e/head//3
2014-06-12 14:51:06.096989 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8600c016/rbd_data.10a76b8b4567.0000000000001cb2/head//3, auth == maps.end()
2014-06-12 14:51:06.096997 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 8600c016/rbd_data.10a76b8b4567.0000000000001cb2/head//3
2014-06-12 14:51:06.097005 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a4a3c016/rbd_data.10976b8b4567.0000000000002326/head//3, auth == maps.end()
2014-06-12 14:51:06.097013 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj a4a3c016/rbd_data.10976b8b4567.0000000000002326/head//3
2014-06-12 14:51:06.097022 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] 
2014-06-12 14:51:06.097027 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub
2014-06-12 14:51:06.097037 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  a4a3c016/rbd_data.10976b8b4567.0000000000002326/head//3 a4a3c016/rbd_data.10976b8b4567.0000000000002326/head//3(173'1395 client.4288.0:70990 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1395)
2014-06-12 14:51:06.097048 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  8600c016/rbd_data.10a76b8b4567.0000000000001cb2/head//3 8600c016/rbd_data.10a76b8b4567.0000000000001cb2/head//3(173'1106 client.4304.0:61554 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1106)
2014-06-12 14:51:06.097057 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  fc5db016/rbd_data.10a06b8b4567.000000000000284e/head//3 fc5db016/rbd_data.10a06b8b4567.000000000000284e/head//3(173'1403 client.4297.0:80962 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1403)
2014-06-12 14:51:06.097067 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  ebbbb016/rbd_data.109c6b8b4567.00000000000034f7/head//3 ebbbb016/rbd_data.109c6b8b4567.00000000000034f7/head//3(173'1875 client.4293.0:105533 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1875)
2014-06-12 14:51:06.097076 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  5dfab016/rbd_data.10a66b8b4567.0000000000002515/head//3 5dfab016/rbd_data.10a66b8b4567.0000000000002515/head//3(173'1323 client.4303.0:74503 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1323)
2014-06-12 14:51:06.097086 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  f929b016/rbd_data.10a56b8b4567.00000000000017a7/head//3 f929b016/rbd_data.10a56b8b4567.00000000000017a7/head//3(173'876 client.4302.0:52843 wrlock_by=unknown.0.0:0 dirty s 4194304 uv876)
2014-06-12 14:51:06.097096 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  5256b016/rbd_data.10ab6b8b4567.0000000000002d29/head//3 5256b016/rbd_data.10ab6b8b4567.0000000000002d29/head//3(173'1568 client.4308.0:91143 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1568)
2014-06-12 14:51:06.097105 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  f7c5b016/rbd_data.10aa6b8b4567.0000000000002543/head//3 f7c5b016/rbd_data.10aa6b8b4567.0000000000002543/head//3(173'1300 client.4307.0:78492 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1300)
2014-06-12 14:51:06.097117 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  5505b016/rbd_data.109b6b8b4567.0000000000002768/head//3 5505b016/rbd_data.109b6b8b4567.0000000000002768/head//3(173'1467 client.4292.0:78587 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1467)
2014-06-12 14:51:06.097127 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  d4e3b016/rbd_data.10b16b8b4567.0000000000001422/head//3 d4e3b016/rbd_data.10b16b8b4567.0000000000001422/head//3(173'725 client.4314.0:46146 wrlock_by=unknown.0.0:0 dirty s 4194304 uv725)
2014-06-12 14:51:06.097136 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  ac32b016/rbd_data.10b16b8b4567.00000000000038a7/head//3 ac32b016/rbd_data.10b16b8b4567.00000000000038a7/head//3(173'1996 client.4314.0:113372 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1996)
2014-06-12 14:51:06.097145 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  b4cea016/rbd_data.109c6b8b4567.0000000000000616/head//3 b4cea016/rbd_data.109c6b8b4567.0000000000000616/head//3(173'2156 client.4293.0:126994 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2156)
2014-06-12 14:51:06.097155 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  be4da016/rbd_data.10ab6b8b4567.0000000000002ee4/head//3 be4da016/rbd_data.10ab6b8b4567.0000000000002ee4/head//3(173'1618 client.4308.0:94339 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1618)
2014-06-12 14:51:06.097165 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  b5cca016/rbd_data.10a96b8b4567.000000000000115a/head//3 b5cca016/rbd_data.10a96b8b4567.000000000000115a/head//3(173'650 client.4306.0:41831 wrlock_by=unknown.0.0:0 dirty s 4194304 uv650)
2014-06-12 14:51:06.097175 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  215aa016/rbd_data.109a6b8b4567.0000000000001af5/head//3 215aa016/rbd_data.109a6b8b4567.0000000000001af5/head//3(173'976 client.4291.0:57708 wrlock_by=unknown.0.0:0 dirty s 4194304 uv976)
2014-06-12 14:51:06.097185 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  31a9a016/rbd_data.10a86b8b4567.0000000000002f08/head//3 31a9a016/rbd_data.10a86b8b4567.0000000000002f08/head//3(173'1562 client.4305.0:93655 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1562)
2014-06-12 14:51:06.097195 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  6b86a016/rbd_data.10916b8b4567.00000000000014aa/head//3 6b86a016/rbd_data.10916b8b4567.00000000000014aa/head//3(173'798 client.4282.0:47802 wrlock_by=unknown.0.0:0 dirty s 4194304 uv798)
2014-06-12 14:51:06.097205 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  9f44a016/rbd_data.109b6b8b4567.0000000000000721/head//3 9f44a016/rbd_data.109b6b8b4567.0000000000000721/head//3(173'2222 client.4292.0:126531 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2222)
2014-06-12 14:51:06.097215 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  d0d2a016/rbd_data.10a66b8b4567.0000000000002934/head//3 d0d2a016/rbd_data.10a66b8b4567.0000000000002934/head//3(173'1476 client.4303.0:82657 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1476)
2014-06-12 14:51:06.097224 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  69f1a016/rbd_data.10ac6b8b4567.000000000000382f/head//3 69f1a016/rbd_data.10ac6b8b4567.000000000000382f/head//3(173'1956 client.4309.0:110374 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1956)
2014-06-12 14:51:06.097234 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  835f9016/rbd_data.109e6b8b4567.0000000000001771/head//3 835f9016/rbd_data.109e6b8b4567.0000000000001771/head//3(173'860 client.4295.0:52684 wrlock_by=unknown.0.0:0 dirty s 4194304 uv860)
2014-06-12 14:51:06.097244 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  a7fe9016/rbd_data.10ac6b8b4567.0000000000003685/head//3 a7fe9016/rbd_data.10ac6b8b4567.0000000000003685/head//3(173'1934 client.4309.0:107279 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1934)
2014-06-12 14:51:06.097254 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  7d1e9016/rbd_data.10ad6b8b4567.000000000000304b/head//3 7d1e9016/rbd_data.10ad6b8b4567.000000000000304b/head//3(173'1683 client.4310.0:96173 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1683)
2014-06-12 14:51:06.097264 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  917b9016/rbd_data.10906b8b4567.00000000000013ea/head//3 917b9016/rbd_data.10906b8b4567.00000000000013ea/head//3(173'707 client.4281.0:45231 wrlock_by=unknown.0.0:0 dirty s 4194304 uv707)
2014-06-12 14:51:06.097274 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  8bb79016/rbd_data.108d6b8b4567.0000000000000ef0/head//3 8bb79016/rbd_data.108d6b8b4567.0000000000000ef0/head//3(173'2359 client.4278.0:142334 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2359)
2014-06-12 14:51:06.097282 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:06.097291 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:06.097298 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:06.097299 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:06.097301 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:06.097329 7f9d60bd2700 10 collection_list_partial 3.16_head start:b8f3c016//0//-1 is_max:0
2014-06-12 14:51:06.097498 7f9d60bd2700 20 objects: [f898c016/rbd_data.10a86b8b4567.0000000000001a5b/head//3,c2f8c016/rbd_data.10a56b8b4567.000000000000249d/head//3,cacc016/rbd_data.10a56b8b4567.000000000000088e/head//3,20cdc016/rbd_data.10a76b8b4567.0000000000003570/head//3,6591d016/rbd_data.10936b8b4567.0000000000002355/head//3,7c1d016/rbd_data.109d6b8b4567.0000000000001891/head//3,cfc2d016/rbd_data.10946b8b4567.00000000000017ce/head//3,3ed2d016/rbd_data.10926b8b4567.0000000000002610/head//3,d493d016/rbd_data.10956b8b4567.0000000000000bf2/head//3,e824d016/rbd_data.10a56b8b4567.00000000000021c1/head//3,335ad016/rbd_data.109c6b8b4567.00000000000017d4/head//3,a6abd016/rbd_data.109e6b8b4567.00000000000038a6/head//3,202fd016/rbd_data.10986b8b4567.00000000000038d5/head//3,a36fd016/rbd_data.109f6b8b4567.000000000000218b/head//3,6052e016/rbd_data.10a36b8b4567.0000000000001952/head//3,a993e016/rbd_data.10a16b8b4567.000000000000041a/head//3,94c3e016/rbd_data.108f6b8b4567.0000000000002a08/head//3,54e016/rbd_data.10aa6b8b4567.000000000000016b/head//3,18b5e016/rbd_data.10966b8b4567.0000000000003621/head//3,6216e016/rbd_data.10936b8b4567.00000000000035e5/head//3,f5f6e016/rbd_data.109b6b8b4567.0000000000001357/head//3,4d69e016/rbd_data.10996b8b4567.0000000000001357/head//3,21e9e016/rbd_data.109f6b8b4567.00000000000017f2/head//3,a14ee016/rbd_data.10b16b8b4567.0000000000002d30/head//3,3df3f016/rbd_data.10a86b8b4567.0000000000000bbd/head//3]
2014-06-12 14:51:06.097919 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  requesting scrubmap from osd.22
2014-06-12 14:51:06.097939 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:06.097954 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:06.097965 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:06.097979 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:06.097984 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:06.097988 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_map_chunk [b8f3c016//0//-1,47f8f016//0//-1)
2014-06-12 14:51:06.097994 7f9d60bd2700 10 collection_list_partial 3.16_head start:b8f3c016//0//-1 is_max:0
2014-06-12 14:51:06.098443 7f9d60bd2700 20 objects: [f898c016/rbd_data.10a86b8b4567.0000000000001a5b/head//3,c2f8c016/rbd_data.10a56b8b4567.000000000000249d/head//3,cacc016/rbd_data.10a56b8b4567.000000000000088e/head//3,20cdc016/rbd_data.10a76b8b4567.0000000000003570/head//3,6591d016/rbd_data.10936b8b4567.0000000000002355/head//3,7c1d016/rbd_data.109d6b8b4567.0000000000001891/head//3,cfc2d016/rbd_data.10946b8b4567.00000000000017ce/head//3,3ed2d016/rbd_data.10926b8b4567.0000000000002610/head//3,d493d016/rbd_data.10956b8b4567.0000000000000bf2/head//3,e824d016/rbd_data.10a56b8b4567.00000000000021c1/head//3,335ad016/rbd_data.109c6b8b4567.00000000000017d4/head//3,a6abd016/rbd_data.109e6b8b4567.00000000000038a6/head//3,202fd016/rbd_data.10986b8b4567.00000000000038d5/head//3,a36fd016/rbd_data.109f6b8b4567.000000000000218b/head//3,6052e016/rbd_data.10a36b8b4567.0000000000001952/head//3,a993e016/rbd_data.10a16b8b4567.000000000000041a/head//3,94c3e016/rbd_data.108f6b8b4567.0000000000002a08/head//3,54e016/rbd_data.10aa6b8b4567.000000000000016b/head//3,18b5e016/rbd_data.10966b8b4567.0000000000003621/head//3,6216e016/rbd_data.10936b8b4567.00000000000035e5/head//3,f5f6e016/rbd_data.109b6b8b4567.0000000000001357/head//3,4d69e016/rbd_data.10996b8b4567.0000000000001357/head//3,21e9e016/rbd_data.109f6b8b4567.00000000000017f2/head//3,a14ee016/rbd_data.10b16b8b4567.0000000000002d30/head//3,3df3f016/rbd_data.10a86b8b4567.0000000000000bbd/head//3,47f8f016/rbd_data.10b46b8b4567.000000000000153c/head//3,a70df016/rbd_data.10aa6b8b4567.0000000000000308/head//3,c93df016/rbd_data.10986b8b4567.000000000000083f/head//3,324ef016/rbd_data.109e6b8b4567.0000000000003306/head//3,16ef016/rbd_data.10a36b8b4567.0000000000003103/head//3,9abef016/rbd_data.109c6b8b4567.0000000000002b65/head//3,64d10816/rbd_data.109c6b8b4567.000000000000294e/head//3,38130816/rbd_data.10a06b8b4567.0000000000002ca1/head//3,22f30816/rbd_data.109a6b8b4567.0000000000000131/head//3,da570816/rbd_data.10ae6b8b4567.0000000000001d51/head//3,26190816/rbd_data.10b06b8b4567.0000000000001834/head//3,1dc90816/rbd_data.10926b8b4567.0000000000001ea6/head//3,7a3b0816/rbd_data.10916b8b4567.000000000000072c/head//3,7aef0816/rbd_data.108e6b8b4567.0000000000003155/head//3,b6441816/rbd_data.10a96b8b4567.00000000000039d6/head//3,af171816/rbd_data.10b36b8b4567.0000000000000814/head//3,6aa81816/rbd_data.10b36b8b4567.0000000000000d79/head//3,aca1816/rbd_data.10976b8b4567.0000000000000cf9/head//3,b67d1816/rbd_data.109b6b8b4567.00000000000006e8/head//3,79302816/rbd_data.10a96b8b4567.000000000000285f/head//3,6ab02816/rbd_data.10956b8b4567.000000000000310b/head//3,97932816/rbd_data.10916b8b4567.0000000000002e8e/head//3,3e252816/rbd_data.10a66b8b4567.00000000000000ba/head//3,c0972816/rbd_data.10ac6b8b4567.0000000000000f94/head//3,96382816/rbd_data.10ae6b8b4567.0000000000002a1f/head//3,22982816/rbd_data.10b06b8b4567.0000000000001279/head//3,f2ce2816/rbd_data.10aa6b8b4567.00000000000037bb/head//3,2e703816/rbd_data.10a36b8b4567.0000000000002f56/head//3,bce13816/rbd_data.10b06b8b4567.00000000000018c1/head//3,8b433816/rbd_data.108d6b8b4567.0000000000001497/head//3,c1553816/rbd_data.10916b8b4567.000000000000199a/head//3,85f63816/rbd_data.10a56b8b4567.0000000000003ae9/head//3,53b73816/rbd_data.10ac6b8b4567.000000000000180a/head//3,8fb3816/rbd_data.10916b8b4567.000000000000219d/head//3,c42c3816/rbd_data.109e6b8b4567.00000000000008c5/head//3,db914816/rbd_data.10ae6b8b4567.0000000000000714/head//3,7cb24816/rbd_data.10a36b8b4567.00000000000027cc/head//3,9bd34816/rbd_data.109f6b8b4567.0000000000002f2b/head//3,c1844816/rbd_data.10936b8b4567.0000000000002a8f/head//3]
2014-06-12 14:51:06.098505 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:06.098518 7f9d60bd2700 10 stat 3.16_head/f898c016/rbd_data.10a86b8b4567.0000000000001a5b/head//3
2014-06-12 14:51:06.098612 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid f898c016/rbd_data.10a86b8b4567.0000000000001a5b/head//3
2014-06-12 14:51:06.115779 7f9d60bd2700 10 getattrs 3.16_head/f898c016/rbd_data.10a86b8b4567.0000000000001a5b/head//3 = 0
2014-06-12 14:51:06.115787 7f9d60bd2700 10 stat 3.16_head/c2f8c016/rbd_data.10a56b8b4567.000000000000249d/head//3
2014-06-12 14:51:06.115891 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid c2f8c016/rbd_data.10a56b8b4567.000000000000249d/head//3
2014-06-12 14:51:06.146271 7f9d60bd2700 10 getattrs 3.16_head/c2f8c016/rbd_data.10a56b8b4567.000000000000249d/head//3 = 0
2014-06-12 14:51:06.146284 7f9d60bd2700 10 stat 3.16_head/cacc016/rbd_data.10a56b8b4567.000000000000088e/head//3
2014-06-12 14:51:06.146414 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid cacc016/rbd_data.10a56b8b4567.000000000000088e/head//3
2014-06-12 14:51:06.182483 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:06.182500 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:06.183127 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:06.183185 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:06.183226 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:06.183249 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:06.183284 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:06.183305 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:06.183343 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:06.183375 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:06.183406 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:06.183426 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:06.183446 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:06.183475 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:06.183504 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:06.183543 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:06.183567 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:06.183590 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:06.183615 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:06.183638 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:06.183663 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:06.183689 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:06.183715 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:06.183741 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:06.205897 7f9d60bd2700 10 getattrs 3.16_head/cacc016/rbd_data.10a56b8b4567.000000000000088e/head//3 = 0
2014-06-12 14:51:06.205910 7f9d60bd2700 10 stat 3.16_head/20cdc016/rbd_data.10a76b8b4567.0000000000003570/head//3
2014-06-12 14:51:06.206013 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 20cdc016/rbd_data.10a76b8b4567.0000000000003570/head//3
2014-06-12 14:51:06.226063 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:06.228870 7f9d60bd2700 10 getattrs 3.16_head/20cdc016/rbd_data.10a76b8b4567.0000000000003570/head//3 = 0
2014-06-12 14:51:06.228884 7f9d60bd2700 10 stat 3.16_head/6591d016/rbd_data.10936b8b4567.0000000000002355/head//3
2014-06-12 14:51:06.229041 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 6591d016/rbd_data.10936b8b4567.0000000000002355/head//3
2014-06-12 14:51:06.234553 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:06.256722 7f9d60bd2700 10 getattrs 3.16_head/6591d016/rbd_data.10936b8b4567.0000000000002355/head//3 = 0
2014-06-12 14:51:06.256729 7f9d60bd2700 10 stat 3.16_head/7c1d016/rbd_data.109d6b8b4567.0000000000001891/head//3
2014-06-12 14:51:06.256823 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 7c1d016/rbd_data.109d6b8b4567.0000000000001891/head//3
2014-06-12 14:51:06.277149 7f9d60bd2700 10 getattrs 3.16_head/7c1d016/rbd_data.109d6b8b4567.0000000000001891/head//3 = 0
2014-06-12 14:51:06.277157 7f9d60bd2700 10 stat 3.16_head/cfc2d016/rbd_data.10946b8b4567.00000000000017ce/head//3
2014-06-12 14:51:06.277246 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid cfc2d016/rbd_data.10946b8b4567.00000000000017ce/head//3
2014-06-12 14:51:06.300598 7f9d60bd2700 10 getattrs 3.16_head/cfc2d016/rbd_data.10946b8b4567.00000000000017ce/head//3 = 0
2014-06-12 14:51:06.300612 7f9d60bd2700 10 stat 3.16_head/3ed2d016/rbd_data.10926b8b4567.0000000000002610/head//3
2014-06-12 14:51:06.300770 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 3ed2d016/rbd_data.10926b8b4567.0000000000002610/head//3
2014-06-12 14:51:06.315498 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:06.321518 7f9d60bd2700 10 getattrs 3.16_head/3ed2d016/rbd_data.10926b8b4567.0000000000002610/head//3 = 0
2014-06-12 14:51:06.321526 7f9d60bd2700 10 stat 3.16_head/d493d016/rbd_data.10956b8b4567.0000000000000bf2/head//3
2014-06-12 14:51:06.321656 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid d493d016/rbd_data.10956b8b4567.0000000000000bf2/head//3
2014-06-12 14:51:06.324974 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:06.356406 7f9d60bd2700 10 getattrs 3.16_head/d493d016/rbd_data.10956b8b4567.0000000000000bf2/head//3 = 0
2014-06-12 14:51:06.356417 7f9d60bd2700 10 stat 3.16_head/e824d016/rbd_data.10a56b8b4567.00000000000021c1/head//3
2014-06-12 14:51:06.356524 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid e824d016/rbd_data.10a56b8b4567.00000000000021c1/head//3
2014-06-12 14:51:06.364957 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:06.369781 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:06.370636 7f9d60bd2700 10 getattrs 3.16_head/e824d016/rbd_data.10a56b8b4567.00000000000021c1/head//3 = 0
2014-06-12 14:51:06.370645 7f9d60bd2700 10 stat 3.16_head/335ad016/rbd_data.109c6b8b4567.00000000000017d4/head//3
2014-06-12 14:51:06.370752 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 335ad016/rbd_data.109c6b8b4567.00000000000017d4/head//3
2014-06-12 14:51:06.380512 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:06.386057 7f9d60bd2700 10 getattrs 3.16_head/335ad016/rbd_data.109c6b8b4567.00000000000017d4/head//3 = 0
2014-06-12 14:51:06.386067 7f9d60bd2700 10 stat 3.16_head/a6abd016/rbd_data.109e6b8b4567.00000000000038a6/head//3
2014-06-12 14:51:06.386178 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid a6abd016/rbd_data.109e6b8b4567.00000000000038a6/head//3
2014-06-12 14:51:06.414443 7f9d60bd2700 10 getattrs 3.16_head/a6abd016/rbd_data.109e6b8b4567.00000000000038a6/head//3 = 0
2014-06-12 14:51:06.414450 7f9d60bd2700 10 stat 3.16_head/202fd016/rbd_data.10986b8b4567.00000000000038d5/head//3
2014-06-12 14:51:06.414544 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 202fd016/rbd_data.10986b8b4567.00000000000038d5/head//3
2014-06-12 14:51:06.426915 7f9d60bd2700 10 getattrs 3.16_head/202fd016/rbd_data.10986b8b4567.00000000000038d5/head//3 = 0
2014-06-12 14:51:06.426924 7f9d60bd2700 10 stat 3.16_head/a36fd016/rbd_data.109f6b8b4567.000000000000218b/head//3
2014-06-12 14:51:06.427025 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid a36fd016/rbd_data.109f6b8b4567.000000000000218b/head//3
2014-06-12 14:51:06.443175 7f9d60bd2700 10 getattrs 3.16_head/a36fd016/rbd_data.109f6b8b4567.000000000000218b/head//3 = 0
2014-06-12 14:51:06.443184 7f9d60bd2700 10 stat 3.16_head/6052e016/rbd_data.10a36b8b4567.0000000000001952/head//3
2014-06-12 14:51:06.443272 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 6052e016/rbd_data.10a36b8b4567.0000000000001952/head//3
2014-06-12 14:51:06.455821 7f9d60bd2700 10 getattrs 3.16_head/6052e016/rbd_data.10a36b8b4567.0000000000001952/head//3 = 0
2014-06-12 14:51:06.455829 7f9d60bd2700 10 stat 3.16_head/a993e016/rbd_data.10a16b8b4567.000000000000041a/head//3
2014-06-12 14:51:06.455916 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid a993e016/rbd_data.10a16b8b4567.000000000000041a/head//3
2014-06-12 14:51:06.503915 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:06.504765 7f9d60bd2700 10 getattrs 3.16_head/a993e016/rbd_data.10a16b8b4567.000000000000041a/head//3 = 0
2014-06-12 14:51:06.504777 7f9d60bd2700 10 stat 3.16_head/94c3e016/rbd_data.108f6b8b4567.0000000000002a08/head//3
2014-06-12 14:51:06.504875 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 94c3e016/rbd_data.108f6b8b4567.0000000000002a08/head//3
2014-06-12 14:51:06.511802 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:06.521606 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:06.530919 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:06.534772 7f9d60bd2700 10 getattrs 3.16_head/94c3e016/rbd_data.108f6b8b4567.0000000000002a08/head//3 = 0
2014-06-12 14:51:06.534783 7f9d60bd2700 10 stat 3.16_head/54e016/rbd_data.10aa6b8b4567.000000000000016b/head//3
2014-06-12 14:51:06.534909 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 54e016/rbd_data.10aa6b8b4567.000000000000016b/head//3
2014-06-12 14:51:06.572442 7f9d60bd2700 10 getattrs 3.16_head/54e016/rbd_data.10aa6b8b4567.000000000000016b/head//3 = 0
2014-06-12 14:51:06.572462 7f9d60bd2700 10 stat 3.16_head/18b5e016/rbd_data.10966b8b4567.0000000000003621/head//3
2014-06-12 14:51:06.572585 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 18b5e016/rbd_data.10966b8b4567.0000000000003621/head//3
2014-06-12 14:51:06.579316 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:06.579352 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:06.585260 7f9d60bd2700 10 getattrs 3.16_head/18b5e016/rbd_data.10966b8b4567.0000000000003621/head//3 = 0
2014-06-12 14:51:06.585271 7f9d60bd2700 10 stat 3.16_head/6216e016/rbd_data.10936b8b4567.00000000000035e5/head//3
2014-06-12 14:51:06.585411 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 6216e016/rbd_data.10936b8b4567.00000000000035e5/head//3
2014-06-12 14:51:06.610824 7f9d60bd2700 10 getattrs 3.16_head/6216e016/rbd_data.10936b8b4567.00000000000035e5/head//3 = 0
2014-06-12 14:51:06.610835 7f9d60bd2700 10 stat 3.16_head/f5f6e016/rbd_data.109b6b8b4567.0000000000001357/head//3
2014-06-12 14:51:06.610969 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid f5f6e016/rbd_data.109b6b8b4567.0000000000001357/head//3
2014-06-12 14:51:06.647883 7f9d60bd2700 10 getattrs 3.16_head/f5f6e016/rbd_data.109b6b8b4567.0000000000001357/head//3 = 0
2014-06-12 14:51:06.647893 7f9d60bd2700 10 stat 3.16_head/4d69e016/rbd_data.10996b8b4567.0000000000001357/head//3
2014-06-12 14:51:06.648022 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 4d69e016/rbd_data.10996b8b4567.0000000000001357/head//3
2014-06-12 14:51:06.650113 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:06.660267 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:06.672491 7f9d58d5b700  5 -- op tracker -- , seq: 748561, time: 2014-06-12 14:51:06.672411, event: header_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.672509 7f9d58d5b700  5 -- op tracker -- , seq: 748561, time: 2014-06-12 14:51:06.672412, event: throttled, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.672522 7f9d58d5b700  5 -- op tracker -- , seq: 748561, time: 2014-06-12 14:51:06.672456, event: all_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.672534 7f9d58d5b700  5 -- op tracker -- , seq: 748561, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.672541 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:06.672550 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:51:06.672560 7f9d58d5b700 15 osd.2 173 enqueue_op 0x55c2000 prio 127 cost 10717 latency 0.000149 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:06.672619 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:06.675982 7f9d60bd2700 10 getattrs 3.16_head/4d69e016/rbd_data.10996b8b4567.0000000000001357/head//3 = 0
2014-06-12 14:51:06.675999 7f9d60bd2700 10 stat 3.16_head/21e9e016/rbd_data.109f6b8b4567.00000000000017f2/head//3
2014-06-12 14:51:06.676131 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 21e9e016/rbd_data.109f6b8b4567.00000000000017f2/head//3
2014-06-12 14:51:06.683199 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,1])
2014-06-12 14:51:06.683222 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [0,0,0,0,1])
2014-06-12 14:51:06.683643 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:06.683670 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:06.683736 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:06.683887 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:06.683950 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:06.683996 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:06.684019 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:06.684067 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:06.684129 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:06.684157 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:06.684194 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:06.684220 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:06.684247 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:06.684275 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:06.684298 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:06.684322 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:06.684349 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:06.684374 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:06.684398 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:06.684423 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:06.684449 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:06.684476 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:06.690909 7f9d60bd2700 10 getattrs 3.16_head/21e9e016/rbd_data.109f6b8b4567.00000000000017f2/head//3 = 0
2014-06-12 14:51:06.690922 7f9d60bd2700 10 stat 3.16_head/a14ee016/rbd_data.10b16b8b4567.0000000000002d30/head//3
2014-06-12 14:51:06.691102 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid a14ee016/rbd_data.10b16b8b4567.0000000000002d30/head//3
2014-06-12 14:51:06.699336 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:06.709086 7f9d60bd2700 10 getattrs 3.16_head/a14ee016/rbd_data.10b16b8b4567.0000000000002d30/head//3 = 0
2014-06-12 14:51:06.709101 7f9d60bd2700 10 stat 3.16_head/3df3f016/rbd_data.10a86b8b4567.0000000000000bbd/head//3
2014-06-12 14:51:06.709277 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 3df3f016/rbd_data.10a86b8b4567.0000000000000bbd/head//3
2014-06-12 14:51:06.744358 7f9d60bd2700 10 getattrs 3.16_head/3df3f016/rbd_data.10a86b8b4567.0000000000000bbd/head//3 = 0
2014-06-12 14:51:06.744369 7f9d60bd2700 10 collection_getattrs 3.16_head
2014-06-12 14:51:06.744425 7f9d60bd2700 10 collection_getattrs 3.16_head = 0
2014-06-12 14:51:06.744429 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:06.744469 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:06.744480 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:06.744492 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:06.744495 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:06.744526 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x55c2000 prio 127 cost 10717 latency 0.072115 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]
2014-06-12 14:51:06.744580 7f9d6b3e7700  5 -- op tracker -- , seq: 748561, time: 2014-06-12 14:51:06.744573, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.744598 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] handle_message: 0x55c2000
2014-06-12 14:51:06.744614 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:06.744632 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:06.744649 7f9d6b3e7700  5 -- op tracker -- , seq: 748561, time: 2014-06-12 14:51:06.744644, event: started, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.744657 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  got 22 scrub map
2014-06-12 14:51:06.744759 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] map version is 173'2396
2014-06-12 14:51:06.744781 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x55c2000 finish
2014-06-12 14:51:06.744790 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:06.744792 7f9d6b3e7700  5 -- op tracker -- , seq: 748561, time: 2014-06-12 14:51:06.744786, event: done, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:06.744817 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:06.744822 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:06.744804 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:06.744836 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:06.744866 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:06.744903 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:06.744913 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:06.744923 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub replica 22 has 25 items
2014-06-12 14:51:06.744948 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f898c016/rbd_data.10a86b8b4567.0000000000001a5b/head//3, auth == maps.end()
2014-06-12 14:51:06.744966 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj f898c016/rbd_data.10a86b8b4567.0000000000001a5b/head//3
2014-06-12 14:51:06.744984 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c2f8c016/rbd_data.10a56b8b4567.000000000000249d/head//3, auth == maps.end()
2014-06-12 14:51:06.745000 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj c2f8c016/rbd_data.10a56b8b4567.000000000000249d/head//3
2014-06-12 14:51:06.745016 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cacc016/rbd_data.10a56b8b4567.000000000000088e/head//3, auth == maps.end()
2014-06-12 14:51:06.745034 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj cacc016/rbd_data.10a56b8b4567.000000000000088e/head//3
2014-06-12 14:51:06.745050 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 20cdc016/rbd_data.10a76b8b4567.0000000000003570/head//3, auth == maps.end()
2014-06-12 14:51:06.745064 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 20cdc016/rbd_data.10a76b8b4567.0000000000003570/head//3
2014-06-12 14:51:06.745080 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6591d016/rbd_data.10936b8b4567.0000000000002355/head//3, auth == maps.end()
2014-06-12 14:51:06.745095 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 6591d016/rbd_data.10936b8b4567.0000000000002355/head//3
2014-06-12 14:51:06.745109 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7c1d016/rbd_data.109d6b8b4567.0000000000001891/head//3, auth == maps.end()
2014-06-12 14:51:06.745124 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 7c1d016/rbd_data.109d6b8b4567.0000000000001891/head//3
2014-06-12 14:51:06.745139 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cfc2d016/rbd_data.10946b8b4567.00000000000017ce/head//3, auth == maps.end()
2014-06-12 14:51:06.745154 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj cfc2d016/rbd_data.10946b8b4567.00000000000017ce/head//3
2014-06-12 14:51:06.745170 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3ed2d016/rbd_data.10926b8b4567.0000000000002610/head//3, auth == maps.end()
2014-06-12 14:51:06.745185 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 3ed2d016/rbd_data.10926b8b4567.0000000000002610/head//3
2014-06-12 14:51:06.745201 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d493d016/rbd_data.10956b8b4567.0000000000000bf2/head//3, auth == maps.end()
2014-06-12 14:51:06.745215 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj d493d016/rbd_data.10956b8b4567.0000000000000bf2/head//3
2014-06-12 14:51:06.745231 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e824d016/rbd_data.10a56b8b4567.00000000000021c1/head//3, auth == maps.end()
2014-06-12 14:51:06.745245 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj e824d016/rbd_data.10a56b8b4567.00000000000021c1/head//3
2014-06-12 14:51:06.745261 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 335ad016/rbd_data.109c6b8b4567.00000000000017d4/head//3, auth == maps.end()
2014-06-12 14:51:06.745275 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 335ad016/rbd_data.109c6b8b4567.00000000000017d4/head//3
2014-06-12 14:51:06.745290 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a6abd016/rbd_data.109e6b8b4567.00000000000038a6/head//3, auth == maps.end()
2014-06-12 14:51:06.745304 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj a6abd016/rbd_data.109e6b8b4567.00000000000038a6/head//3
2014-06-12 14:51:06.745320 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 202fd016/rbd_data.10986b8b4567.00000000000038d5/head//3, auth == maps.end()
2014-06-12 14:51:06.745335 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 202fd016/rbd_data.10986b8b4567.00000000000038d5/head//3
2014-06-12 14:51:06.745350 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a36fd016/rbd_data.109f6b8b4567.000000000000218b/head//3, auth == maps.end()
2014-06-12 14:51:06.745365 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj a36fd016/rbd_data.109f6b8b4567.000000000000218b/head//3
2014-06-12 14:51:06.745381 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6052e016/rbd_data.10a36b8b4567.0000000000001952/head//3, auth == maps.end()
2014-06-12 14:51:06.745395 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 6052e016/rbd_data.10a36b8b4567.0000000000001952/head//3
2014-06-12 14:51:06.745411 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a993e016/rbd_data.10a16b8b4567.000000000000041a/head//3, auth == maps.end()
2014-06-12 14:51:06.745426 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj a993e016/rbd_data.10a16b8b4567.000000000000041a/head//3
2014-06-12 14:51:06.745441 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 94c3e016/rbd_data.108f6b8b4567.0000000000002a08/head//3, auth == maps.end()
2014-06-12 14:51:06.745456 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 94c3e016/rbd_data.108f6b8b4567.0000000000002a08/head//3
2014-06-12 14:51:06.745470 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 54e016/rbd_data.10aa6b8b4567.000000000000016b/head//3, auth == maps.end()
2014-06-12 14:51:06.745484 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 54e016/rbd_data.10aa6b8b4567.000000000000016b/head//3
2014-06-12 14:51:06.745499 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 18b5e016/rbd_data.10966b8b4567.0000000000003621/head//3, auth == maps.end()
2014-06-12 14:51:06.745514 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 18b5e016/rbd_data.10966b8b4567.0000000000003621/head//3
2014-06-12 14:51:06.745529 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6216e016/rbd_data.10936b8b4567.00000000000035e5/head//3, auth == maps.end()
2014-06-12 14:51:06.745544 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 6216e016/rbd_data.10936b8b4567.00000000000035e5/head//3
2014-06-12 14:51:06.745560 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f5f6e016/rbd_data.109b6b8b4567.0000000000001357/head//3, auth == maps.end()
2014-06-12 14:51:06.745575 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj f5f6e016/rbd_data.109b6b8b4567.0000000000001357/head//3
2014-06-12 14:51:06.745590 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4d69e016/rbd_data.10996b8b4567.0000000000001357/head//3, auth == maps.end()
2014-06-12 14:51:06.745605 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 4d69e016/rbd_data.10996b8b4567.0000000000001357/head//3
2014-06-12 14:51:06.745624 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 21e9e016/rbd_data.109f6b8b4567.00000000000017f2/head//3, auth == maps.end()
2014-06-12 14:51:06.745646 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 21e9e016/rbd_data.109f6b8b4567.00000000000017f2/head//3
2014-06-12 14:51:06.745667 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a14ee016/rbd_data.10b16b8b4567.0000000000002d30/head//3, auth == maps.end()
2014-06-12 14:51:06.745682 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj a14ee016/rbd_data.10b16b8b4567.0000000000002d30/head//3
2014-06-12 14:51:06.745697 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3df3f016/rbd_data.10a86b8b4567.0000000000000bbd/head//3, auth == maps.end()
2014-06-12 14:51:06.745711 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 3df3f016/rbd_data.10a86b8b4567.0000000000000bbd/head//3
2014-06-12 14:51:06.745728 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] 
2014-06-12 14:51:06.745739 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub
2014-06-12 14:51:06.745757 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  3df3f016/rbd_data.10a86b8b4567.0000000000000bbd/head//3 3df3f016/rbd_data.10a86b8b4567.0000000000000bbd/head//3(173'2340 client.4305.0:136898 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2340)
2014-06-12 14:51:06.745773 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  a14ee016/rbd_data.10b16b8b4567.0000000000002d30/head//3 a14ee016/rbd_data.10b16b8b4567.0000000000002d30/head//3(173'1674 client.4314.0:91594 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1674)
2014-06-12 14:51:06.745790 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  21e9e016/rbd_data.109f6b8b4567.00000000000017f2/head//3 21e9e016/rbd_data.109f6b8b4567.00000000000017f2/head//3(173'767 client.4296.0:51855 wrlock_by=unknown.0.0:0 dirty s 4194304 uv767)
2014-06-12 14:51:06.745808 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  4d69e016/rbd_data.10996b8b4567.0000000000001357/head//3 4d69e016/rbd_data.10996b8b4567.0000000000001357/head//3(173'641 client.4290.0:43532 wrlock_by=unknown.0.0:0 dirty s 4194304 uv641)
2014-06-12 14:51:06.745825 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  f5f6e016/rbd_data.109b6b8b4567.0000000000001357/head//3 f5f6e016/rbd_data.109b6b8b4567.0000000000001357/head//3(173'754 client.4292.0:43041 wrlock_by=unknown.0.0:0 dirty s 4194304 uv754)
2014-06-12 14:51:06.745844 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  6216e016/rbd_data.10936b8b4567.00000000000035e5/head//3 6216e016/rbd_data.10936b8b4567.00000000000035e5/head//3(173'1869 client.4284.0:104408 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1869)
2014-06-12 14:51:06.745861 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  18b5e016/rbd_data.10966b8b4567.0000000000003621/head//3 18b5e016/rbd_data.10966b8b4567.0000000000003621/head//3(173'1819 client.4287.0:104368 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1819)
2014-06-12 14:51:06.745878 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  54e016/rbd_data.10aa6b8b4567.000000000000016b/head//3 54e016/rbd_data.10aa6b8b4567.000000000000016b/head//3(173'2051 client.4307.0:121766 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2051)
2014-06-12 14:51:06.745895 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  94c3e016/rbd_data.108f6b8b4567.0000000000002a08/head//3 94c3e016/rbd_data.108f6b8b4567.0000000000002a08/head//3(173'1458 client.4280.0:83781 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1458)
2014-06-12 14:51:06.745912 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  a993e016/rbd_data.10a16b8b4567.000000000000041a/head//3 a993e016/rbd_data.10a16b8b4567.000000000000041a/head//3(173'2109 client.4298.0:124982 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2109)
2014-06-12 14:51:06.745929 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  6052e016/rbd_data.10a36b8b4567.0000000000001952/head//3 6052e016/rbd_data.10a36b8b4567.0000000000001952/head//3(173'921 client.4300.0:53819 wrlock_by=unknown.0.0:0 dirty s 4194304 uv921)
2014-06-12 14:51:06.745946 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  a36fd016/rbd_data.109f6b8b4567.000000000000218b/head//3 a36fd016/rbd_data.109f6b8b4567.000000000000218b/head//3(173'1140 client.4296.0:67866 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1140)
2014-06-12 14:51:06.745962 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  202fd016/rbd_data.10986b8b4567.00000000000038d5/head//3 202fd016/rbd_data.10986b8b4567.00000000000038d5/head//3(173'1885 client.4289.0:110434 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1885)
2014-06-12 14:51:06.745980 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  a6abd016/rbd_data.109e6b8b4567.00000000000038a6/head//3 a6abd016/rbd_data.109e6b8b4567.00000000000038a6/head//3(173'1892 client.4295.0:111683 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1892)
2014-06-12 14:51:06.745997 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  335ad016/rbd_data.109c6b8b4567.00000000000017d4/head//3 335ad016/rbd_data.109c6b8b4567.00000000000017d4/head//3(173'892 client.4293.0:52199 wrlock_by=unknown.0.0:0 dirty s 4194304 uv892)
2014-06-12 14:51:06.746016 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  e824d016/rbd_data.10a56b8b4567.00000000000021c1/head//3 e824d016/rbd_data.10a56b8b4567.00000000000021c1/head//3(173'1219 client.4302.0:70704 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1219)
2014-06-12 14:51:06.746033 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  d493d016/rbd_data.10956b8b4567.0000000000000bf2/head//3 d493d016/rbd_data.10956b8b4567.0000000000000bf2/head//3(173'2321 client.4286.0:137289 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2321)
2014-06-12 14:51:06.746050 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  3ed2d016/rbd_data.10926b8b4567.0000000000002610/head//3 3ed2d016/rbd_data.10926b8b4567.0000000000002610/head//3(173'1359 client.4283.0:77143 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1359)
2014-06-12 14:51:06.746068 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  cfc2d016/rbd_data.10946b8b4567.00000000000017ce/head//3 cfc2d016/rbd_data.10946b8b4567.00000000000017ce/head//3(173'803 client.4285.0:51778 wrlock_by=unknown.0.0:0 dirty s 4194304 uv803)
2014-06-12 14:51:06.746085 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  7c1d016/rbd_data.109d6b8b4567.0000000000001891/head//3 7c1d016/rbd_data.109d6b8b4567.0000000000001891/head//3(173'846 client.4294.0:53844 wrlock_by=unknown.0.0:0 dirty s 4194304 uv846)
2014-06-12 14:51:06.746102 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  6591d016/rbd_data.10936b8b4567.0000000000002355/head//3 6591d016/rbd_data.10936b8b4567.0000000000002355/head//3(173'1281 client.4284.0:71175 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1281)
2014-06-12 14:51:06.746119 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  20cdc016/rbd_data.10a76b8b4567.0000000000003570/head//3 20cdc016/rbd_data.10a76b8b4567.0000000000003570/head//3(173'1835 client.4304.0:105813 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1835)
2014-06-12 14:51:06.746136 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  cacc016/rbd_data.10a56b8b4567.000000000000088e/head//3 cacc016/rbd_data.10a56b8b4567.000000000000088e/head//3(173'2261 client.4302.0:132707 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2261)
2014-06-12 14:51:06.746153 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  c2f8c016/rbd_data.10a56b8b4567.000000000000249d/head//3 c2f8c016/rbd_data.10a56b8b4567.000000000000249d/head//3(173'1329 client.4302.0:76005 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1329)
2014-06-12 14:51:06.746170 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  f898c016/rbd_data.10a86b8b4567.0000000000001a5b/head//3 f898c016/rbd_data.10a86b8b4567.0000000000001a5b/head//3(173'899 client.4305.0:56038 wrlock_by=unknown.0.0:0 dirty s 4194304 uv899)
2014-06-12 14:51:06.746184 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:06.746199 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:06.746210 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:06.746212 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:06.746214 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:06.746253 7f9d60bd2700 10 collection_list_partial 3.16_head start:47f8f016//0//-1 is_max:0
2014-06-12 14:51:06.746438 7f9d60bd2700 20 objects: [a70df016/rbd_data.10aa6b8b4567.0000000000000308/head//3,c93df016/rbd_data.10986b8b4567.000000000000083f/head//3,324ef016/rbd_data.109e6b8b4567.0000000000003306/head//3,16ef016/rbd_data.10a36b8b4567.0000000000003103/head//3,9abef016/rbd_data.109c6b8b4567.0000000000002b65/head//3,64d10816/rbd_data.109c6b8b4567.000000000000294e/head//3,38130816/rbd_data.10a06b8b4567.0000000000002ca1/head//3,22f30816/rbd_data.109a6b8b4567.0000000000000131/head//3,da570816/rbd_data.10ae6b8b4567.0000000000001d51/head//3,26190816/rbd_data.10b06b8b4567.0000000000001834/head//3,1dc90816/rbd_data.10926b8b4567.0000000000001ea6/head//3,7a3b0816/rbd_data.10916b8b4567.000000000000072c/head//3,7aef0816/rbd_data.108e6b8b4567.0000000000003155/head//3,b6441816/rbd_data.10a96b8b4567.00000000000039d6/head//3,af171816/rbd_data.10b36b8b4567.0000000000000814/head//3,6aa81816/rbd_data.10b36b8b4567.0000000000000d79/head//3,aca1816/rbd_data.10976b8b4567.0000000000000cf9/head//3,b67d1816/rbd_data.109b6b8b4567.00000000000006e8/head//3,79302816/rbd_data.10a96b8b4567.000000000000285f/head//3,6ab02816/rbd_data.10956b8b4567.000000000000310b/head//3,97932816/rbd_data.10916b8b4567.0000000000002e8e/head//3,3e252816/rbd_data.10a66b8b4567.00000000000000ba/head//3,c0972816/rbd_data.10ac6b8b4567.0000000000000f94/head//3,96382816/rbd_data.10ae6b8b4567.0000000000002a1f/head//3,22982816/rbd_data.10b06b8b4567.0000000000001279/head//3]
2014-06-12 14:51:06.746835 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  requesting scrubmap from osd.22
2014-06-12 14:51:06.746856 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:06.746881 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:06.746890 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:06.746906 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:06.746915 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:06.746924 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_map_chunk [47f8f016//0//-1,f2ce2816//0//-1)
2014-06-12 14:51:06.746934 7f9d60bd2700 10 collection_list_partial 3.16_head start:47f8f016//0//-1 is_max:0
2014-06-12 14:51:06.747339 7f9d60bd2700 20 objects: [a70df016/rbd_data.10aa6b8b4567.0000000000000308/head//3,c93df016/rbd_data.10986b8b4567.000000000000083f/head//3,324ef016/rbd_data.109e6b8b4567.0000000000003306/head//3,16ef016/rbd_data.10a36b8b4567.0000000000003103/head//3,9abef016/rbd_data.109c6b8b4567.0000000000002b65/head//3,64d10816/rbd_data.109c6b8b4567.000000000000294e/head//3,38130816/rbd_data.10a06b8b4567.0000000000002ca1/head//3,22f30816/rbd_data.109a6b8b4567.0000000000000131/head//3,da570816/rbd_data.10ae6b8b4567.0000000000001d51/head//3,26190816/rbd_data.10b06b8b4567.0000000000001834/head//3,1dc90816/rbd_data.10926b8b4567.0000000000001ea6/head//3,7a3b0816/rbd_data.10916b8b4567.000000000000072c/head//3,7aef0816/rbd_data.108e6b8b4567.0000000000003155/head//3,b6441816/rbd_data.10a96b8b4567.00000000000039d6/head//3,af171816/rbd_data.10b36b8b4567.0000000000000814/head//3,6aa81816/rbd_data.10b36b8b4567.0000000000000d79/head//3,aca1816/rbd_data.10976b8b4567.0000000000000cf9/head//3,b67d1816/rbd_data.109b6b8b4567.00000000000006e8/head//3,79302816/rbd_data.10a96b8b4567.000000000000285f/head//3,6ab02816/rbd_data.10956b8b4567.000000000000310b/head//3,97932816/rbd_data.10916b8b4567.0000000000002e8e/head//3,3e252816/rbd_data.10a66b8b4567.00000000000000ba/head//3,c0972816/rbd_data.10ac6b8b4567.0000000000000f94/head//3,96382816/rbd_data.10ae6b8b4567.0000000000002a1f/head//3,22982816/rbd_data.10b06b8b4567.0000000000001279/head//3,f2ce2816/rbd_data.10aa6b8b4567.00000000000037bb/head//3,2e703816/rbd_data.10a36b8b4567.0000000000002f56/head//3,bce13816/rbd_data.10b06b8b4567.00000000000018c1/head//3,8b433816/rbd_data.108d6b8b4567.0000000000001497/head//3,c1553816/rbd_data.10916b8b4567.000000000000199a/head//3,85f63816/rbd_data.10a56b8b4567.0000000000003ae9/head//3,53b73816/rbd_data.10ac6b8b4567.000000000000180a/head//3,8fb3816/rbd_data.10916b8b4567.000000000000219d/head//3,c42c3816/rbd_data.109e6b8b4567.00000000000008c5/head//3,db914816/rbd_data.10ae6b8b4567.0000000000000714/head//3,7cb24816/rbd_data.10a36b8b4567.00000000000027cc/head//3,9bd34816/rbd_data.109f6b8b4567.0000000000002f2b/head//3,c1844816/rbd_data.10936b8b4567.0000000000002a8f/head//3,e2c54816/rbd_data.10936b8b4567.00000000000002f2/head//3,d0984816/rbd_data.10af6b8b4567.0000000000002d02/head//3,b0894816/rbd_data.10ac6b8b4567.000000000000260b/head//3,3fc94816/rbd_data.10af6b8b4567.0000000000001281/head//3,d73a4816/rbd_data.10ad6b8b4567.000000000000128c/head//3,826a4816/rbd_data.10ac6b8b4567.0000000000001464/head//3,4cea4816/rbd_data.10a36b8b4567.0000000000000548/head//3,654b4816/rbd_data.109b6b8b4567.00000000000009de/head//3,660e4816/rbd_data.10986b8b4567.0000000000001346/head//3,b3ef4816/rbd_data.10aa6b8b4567.0000000000001a49/head//3,58b45816/rbd_data.109a6b8b4567.0000000000002409/head//3,f1b55816/rbd_data.10986b8b4567.0000000000003b0c/head//3,1e575816/rbd_data.10af6b8b4567.0000000000001c0e/head//3,2fe85816/rbd_data.10a26b8b4567.0000000000002eaf/head//3,86495816/rbd_data.109b6b8b4567.0000000000001b44/head//3,90da5816/rbd_data.10aa6b8b4567.0000000000003bd3/head//3,fad16816/rbd_data.10a16b8b4567.0000000000001731/head//3,77e46816/rbd_data.10966b8b4567.000000000000210d/head//3,cef76816/rbd_data.10b16b8b4567.0000000000001724/head//3,1ad96816/rbd_data.10b46b8b4567.00000000000032a8/head//3,867d6816/rbd_data.10aa6b8b4567.0000000000002162/head//3,8fd6816/rbd_data.108e6b8b4567.00000000000008a3/head//3,1ee6816/rbd_data.10a56b8b4567.0000000000001ba0/head//3,84cf6816/rbd_data.108d6b8b4567.0000000000001e79/head//3,f9e17816/rbd_data.10a56b8b4567.0000000000003054/head//3,99027816/rbd_data.10ae6b8b4567.000000000000038d/head//3]
2014-06-12 14:51:06.747406 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:06.747422 7f9d60bd2700 10 stat 3.16_head/a70df016/rbd_data.10aa6b8b4567.0000000000000308/head//3
2014-06-12 14:51:06.747533 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid a70df016/rbd_data.10aa6b8b4567.0000000000000308/head//3
2014-06-12 14:51:06.766668 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:06.794954 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:06.800575 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:06.817118 7f9d60bd2700 10 getattrs 3.16_head/a70df016/rbd_data.10aa6b8b4567.0000000000000308/head//3 = 0
2014-06-12 14:51:06.817127 7f9d60bd2700 10 stat 3.16_head/c93df016/rbd_data.10986b8b4567.000000000000083f/head//3
2014-06-12 14:51:06.817224 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid c93df016/rbd_data.10986b8b4567.000000000000083f/head//3
2014-06-12 14:51:06.853692 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:06.854813 7f9d60bd2700 10 getattrs 3.16_head/c93df016/rbd_data.10986b8b4567.000000000000083f/head//3 = 0
2014-06-12 14:51:06.854828 7f9d60bd2700 10 stat 3.16_head/324ef016/rbd_data.109e6b8b4567.0000000000003306/head//3
2014-06-12 14:51:06.855006 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 324ef016/rbd_data.109e6b8b4567.0000000000003306/head//3
2014-06-12 14:51:06.864062 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:06.867330 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:06.869133 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:06.869184 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.59 >= max 0.5 = no, load too high
2014-06-12 14:51:06.869201 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:51:06.869204 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:06.869205 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92914.4 < max (604800 seconds)
2014-06-12 14:51:06.869225 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:06.869228 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:06.869230 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:06.887534 7f9d60bd2700 10 getattrs 3.16_head/324ef016/rbd_data.109e6b8b4567.0000000000003306/head//3 = 0
2014-06-12 14:51:06.887549 7f9d60bd2700 10 stat 3.16_head/16ef016/rbd_data.10a36b8b4567.0000000000003103/head//3
2014-06-12 14:51:06.887737 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 16ef016/rbd_data.10a36b8b4567.0000000000003103/head//3
2014-06-12 14:51:06.909210 7f9d60bd2700 10 getattrs 3.16_head/16ef016/rbd_data.10a36b8b4567.0000000000003103/head//3 = 0
2014-06-12 14:51:06.909241 7f9d60bd2700 10 stat 3.16_head/9abef016/rbd_data.109c6b8b4567.0000000000002b65/head//3
2014-06-12 14:51:06.909437 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 9abef016/rbd_data.109c6b8b4567.0000000000002b65/head//3
2014-06-12 14:51:06.918905 7f9d60bd2700 10 getattrs 3.16_head/9abef016/rbd_data.109c6b8b4567.0000000000002b65/head//3 = 0
2014-06-12 14:51:06.918916 7f9d60bd2700 10 stat 3.16_head/64d10816/rbd_data.109c6b8b4567.000000000000294e/head//3
2014-06-12 14:51:06.919005 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 64d10816/rbd_data.109c6b8b4567.000000000000294e/head//3
2014-06-12 14:51:06.927076 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:06.934550 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:06.935382 7f9d60bd2700 10 getattrs 3.16_head/64d10816/rbd_data.109c6b8b4567.000000000000294e/head//3 = 0
2014-06-12 14:51:06.935391 7f9d60bd2700 10 stat 3.16_head/38130816/rbd_data.10a06b8b4567.0000000000002ca1/head//3
2014-06-12 14:51:06.935506 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 38130816/rbd_data.10a06b8b4567.0000000000002ca1/head//3
2014-06-12 14:51:06.951040 7f9d60bd2700 10 getattrs 3.16_head/38130816/rbd_data.10a06b8b4567.0000000000002ca1/head//3 = 0
2014-06-12 14:51:06.951049 7f9d60bd2700 10 stat 3.16_head/22f30816/rbd_data.109a6b8b4567.0000000000000131/head//3
2014-06-12 14:51:06.951161 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 22f30816/rbd_data.109a6b8b4567.0000000000000131/head//3
2014-06-12 14:51:06.960910 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:06.964651 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:07.013490 7f9d60bd2700 10 getattrs 3.16_head/22f30816/rbd_data.109a6b8b4567.0000000000000131/head//3 = 0
2014-06-12 14:51:07.013502 7f9d60bd2700 10 stat 3.16_head/da570816/rbd_data.10ae6b8b4567.0000000000001d51/head//3
2014-06-12 14:51:07.013670 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid da570816/rbd_data.10ae6b8b4567.0000000000001d51/head//3
2014-06-12 14:51:07.044141 7f9d60bd2700 10 getattrs 3.16_head/da570816/rbd_data.10ae6b8b4567.0000000000001d51/head//3 = 0
2014-06-12 14:51:07.044153 7f9d60bd2700 10 stat 3.16_head/26190816/rbd_data.10b06b8b4567.0000000000001834/head//3
2014-06-12 14:51:07.044300 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 26190816/rbd_data.10b06b8b4567.0000000000001834/head//3
2014-06-12 14:51:07.076855 7f9d60bd2700 10 getattrs 3.16_head/26190816/rbd_data.10b06b8b4567.0000000000001834/head//3 = 0
2014-06-12 14:51:07.076867 7f9d60bd2700 10 stat 3.16_head/1dc90816/rbd_data.10926b8b4567.0000000000001ea6/head//3
2014-06-12 14:51:07.077013 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 1dc90816/rbd_data.10926b8b4567.0000000000001ea6/head//3
2014-06-12 14:51:07.089233 7f9d60bd2700 10 getattrs 3.16_head/1dc90816/rbd_data.10926b8b4567.0000000000001ea6/head//3 = 0
2014-06-12 14:51:07.089247 7f9d60bd2700 10 stat 3.16_head/7a3b0816/rbd_data.10916b8b4567.000000000000072c/head//3
2014-06-12 14:51:07.089391 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 7a3b0816/rbd_data.10916b8b4567.000000000000072c/head//3
2014-06-12 14:51:07.119927 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:07.131523 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:07.137371 7f9d60bd2700 10 getattrs 3.16_head/7a3b0816/rbd_data.10916b8b4567.000000000000072c/head//3 = 0
2014-06-12 14:51:07.137388 7f9d60bd2700 10 stat 3.16_head/7aef0816/rbd_data.108e6b8b4567.0000000000003155/head//3
2014-06-12 14:51:07.137590 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 7aef0816/rbd_data.108e6b8b4567.0000000000003155/head//3
2014-06-12 14:51:07.141010 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:07.150460 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:07.151894 7f9d60bd2700 10 getattrs 3.16_head/7aef0816/rbd_data.108e6b8b4567.0000000000003155/head//3 = 0
2014-06-12 14:51:07.151907 7f9d60bd2700 10 stat 3.16_head/b6441816/rbd_data.10a96b8b4567.00000000000039d6/head//3
2014-06-12 14:51:07.152067 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid b6441816/rbd_data.10a96b8b4567.00000000000039d6/head//3
2014-06-12 14:51:07.166878 7f9d60bd2700 10 getattrs 3.16_head/b6441816/rbd_data.10a96b8b4567.00000000000039d6/head//3 = 0
2014-06-12 14:51:07.166893 7f9d60bd2700 10 stat 3.16_head/af171816/rbd_data.10b36b8b4567.0000000000000814/head//3
2014-06-12 14:51:07.167082 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid af171816/rbd_data.10b36b8b4567.0000000000000814/head//3
2014-06-12 14:51:07.176676 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:07.178360 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:07.178403 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:07.178859 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:07.196866 7f9d60bd2700 10 getattrs 3.16_head/af171816/rbd_data.10b36b8b4567.0000000000000814/head//3 = 0
2014-06-12 14:51:07.196880 7f9d60bd2700 10 stat 3.16_head/6aa81816/rbd_data.10b36b8b4567.0000000000000d79/head//3
2014-06-12 14:51:07.196974 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 6aa81816/rbd_data.10b36b8b4567.0000000000000d79/head//3
2014-06-12 14:51:07.229935 7f9d60bd2700 10 getattrs 3.16_head/6aa81816/rbd_data.10b36b8b4567.0000000000000d79/head//3 = 0
2014-06-12 14:51:07.229951 7f9d60bd2700 10 stat 3.16_head/aca1816/rbd_data.10976b8b4567.0000000000000cf9/head//3
2014-06-12 14:51:07.230136 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid aca1816/rbd_data.10976b8b4567.0000000000000cf9/head//3
2014-06-12 14:51:07.275335 7f9d60bd2700 10 getattrs 3.16_head/aca1816/rbd_data.10976b8b4567.0000000000000cf9/head//3 = 0
2014-06-12 14:51:07.275360 7f9d60bd2700 10 stat 3.16_head/b67d1816/rbd_data.109b6b8b4567.00000000000006e8/head//3
2014-06-12 14:51:07.275450 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid b67d1816/rbd_data.109b6b8b4567.00000000000006e8/head//3
2014-06-12 14:51:07.281638 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:07.297076 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:07.330818 7f9d60bd2700 10 getattrs 3.16_head/b67d1816/rbd_data.109b6b8b4567.00000000000006e8/head//3 = 0
2014-06-12 14:51:07.330840 7f9d60bd2700 10 stat 3.16_head/79302816/rbd_data.10a96b8b4567.000000000000285f/head//3
2014-06-12 14:51:07.331004 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 79302816/rbd_data.10a96b8b4567.000000000000285f/head//3
2014-06-12 14:51:07.335912 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:07.340250 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:07.357946 7f9d60bd2700 10 getattrs 3.16_head/79302816/rbd_data.10a96b8b4567.000000000000285f/head//3 = 0
2014-06-12 14:51:07.357963 7f9d60bd2700 10 stat 3.16_head/6ab02816/rbd_data.10956b8b4567.000000000000310b/head//3
2014-06-12 14:51:07.358129 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 6ab02816/rbd_data.10956b8b4567.000000000000310b/head//3
2014-06-12 14:51:07.367734 7f9d58d5b700  5 -- op tracker -- , seq: 748562, time: 2014-06-12 14:51:07.367651, event: header_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:07.367752 7f9d58d5b700  5 -- op tracker -- , seq: 748562, time: 2014-06-12 14:51:07.367654, event: throttled, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:07.367767 7f9d58d5b700  5 -- op tracker -- , seq: 748562, time: 2014-06-12 14:51:07.367699, event: all_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:07.367776 7f9d58d5b700  5 -- op tracker -- , seq: 748562, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:07.367781 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:07.367788 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:51:07.367796 7f9d58d5b700 15 osd.2 173 enqueue_op 0x17c2aa00 prio 127 cost 10717 latency 0.000144 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:07.367867 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:07.375407 7f9d60bd2700 10 getattrs 3.16_head/6ab02816/rbd_data.10956b8b4567.000000000000310b/head//3 = 0
2014-06-12 14:51:07.375415 7f9d60bd2700 10 stat 3.16_head/97932816/rbd_data.10916b8b4567.0000000000002e8e/head//3
2014-06-12 14:51:07.375505 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 97932816/rbd_data.10916b8b4567.0000000000002e8e/head//3
2014-06-12 14:51:07.388856 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:07.400976 7f9d60bd2700 10 getattrs 3.16_head/97932816/rbd_data.10916b8b4567.0000000000002e8e/head//3 = 0
2014-06-12 14:51:07.400988 7f9d60bd2700 10 stat 3.16_head/3e252816/rbd_data.10a66b8b4567.00000000000000ba/head//3
2014-06-12 14:51:07.401111 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 3e252816/rbd_data.10a66b8b4567.00000000000000ba/head//3
2014-06-12 14:51:07.435787 7f9d60bd2700 10 getattrs 3.16_head/3e252816/rbd_data.10a66b8b4567.00000000000000ba/head//3 = 0
2014-06-12 14:51:07.435799 7f9d60bd2700 10 stat 3.16_head/c0972816/rbd_data.10ac6b8b4567.0000000000000f94/head//3
2014-06-12 14:51:07.435935 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid c0972816/rbd_data.10ac6b8b4567.0000000000000f94/head//3
2014-06-12 14:51:07.436085 7f9d60bd2700 10 getattrs 3.16_head/c0972816/rbd_data.10ac6b8b4567.0000000000000f94/head//3 = 0
2014-06-12 14:51:07.436093 7f9d60bd2700 10 stat 3.16_head/96382816/rbd_data.10ae6b8b4567.0000000000002a1f/head//3
2014-06-12 14:51:07.436210 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 96382816/rbd_data.10ae6b8b4567.0000000000002a1f/head//3
2014-06-12 14:51:07.448869 7f9d60bd2700 10 getattrs 3.16_head/96382816/rbd_data.10ae6b8b4567.0000000000002a1f/head//3 = 0
2014-06-12 14:51:07.448882 7f9d60bd2700 10 stat 3.16_head/22982816/rbd_data.10b06b8b4567.0000000000001279/head//3
2014-06-12 14:51:07.449013 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 22982816/rbd_data.10b06b8b4567.0000000000001279/head//3
2014-06-12 14:51:07.467941 7f9d60bd2700 10 getattrs 3.16_head/22982816/rbd_data.10b06b8b4567.0000000000001279/head//3 = 0
2014-06-12 14:51:07.467953 7f9d60bd2700 10 collection_getattrs 3.16_head
2014-06-12 14:51:07.468025 7f9d60bd2700 10 collection_getattrs 3.16_head = 0
2014-06-12 14:51:07.468031 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:07.468080 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:07.468100 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:07.468118 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:07.468123 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:07.468151 7f9d643d9700 10 osd.2 173 dequeue_op 0x17c2aa00 prio 127 cost 10717 latency 0.100500 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]
2014-06-12 14:51:07.468201 7f9d643d9700  5 -- op tracker -- , seq: 748562, time: 2014-06-12 14:51:07.468193, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:07.468220 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] handle_message: 0x17c2aa00
2014-06-12 14:51:07.468237 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:07.468254 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:07.468272 7f9d643d9700  5 -- op tracker -- , seq: 748562, time: 2014-06-12 14:51:07.468266, event: started, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:07.468279 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  got 22 scrub map
2014-06-12 14:51:07.468388 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] map version is 173'2396
2014-06-12 14:51:07.468411 7f9d643d9700 10 osd.2 173 dequeue_op 0x17c2aa00 finish
2014-06-12 14:51:07.468418 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:07.468422 7f9d643d9700  5 -- op tracker -- , seq: 748562, time: 2014-06-12 14:51:07.468416, event: done, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:07.468446 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:07.468450 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:07.468430 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:07.468455 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:07.468476 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:07.468514 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:07.468528 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:07.468540 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub replica 22 has 25 items
2014-06-12 14:51:07.468577 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a70df016/rbd_data.10aa6b8b4567.0000000000000308/head//3, auth == maps.end()
2014-06-12 14:51:07.468597 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj a70df016/rbd_data.10aa6b8b4567.0000000000000308/head//3
2014-06-12 14:51:07.468622 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c93df016/rbd_data.10986b8b4567.000000000000083f/head//3, auth == maps.end()
2014-06-12 14:51:07.468643 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj c93df016/rbd_data.10986b8b4567.000000000000083f/head//3
2014-06-12 14:51:07.468663 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 324ef016/rbd_data.109e6b8b4567.0000000000003306/head//3, auth == maps.end()
2014-06-12 14:51:07.468684 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 324ef016/rbd_data.109e6b8b4567.0000000000003306/head//3
2014-06-12 14:51:07.468706 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 16ef016/rbd_data.10a36b8b4567.0000000000003103/head//3, auth == maps.end()
2014-06-12 14:51:07.468727 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 16ef016/rbd_data.10a36b8b4567.0000000000003103/head//3
2014-06-12 14:51:07.468746 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9abef016/rbd_data.109c6b8b4567.0000000000002b65/head//3, auth == maps.end()
2014-06-12 14:51:07.468768 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 9abef016/rbd_data.109c6b8b4567.0000000000002b65/head//3
2014-06-12 14:51:07.468790 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 64d10816/rbd_data.109c6b8b4567.000000000000294e/head//3, auth == maps.end()
2014-06-12 14:51:07.468810 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 64d10816/rbd_data.109c6b8b4567.000000000000294e/head//3
2014-06-12 14:51:07.468833 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 38130816/rbd_data.10a06b8b4567.0000000000002ca1/head//3, auth == maps.end()
2014-06-12 14:51:07.468854 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 38130816/rbd_data.10a06b8b4567.0000000000002ca1/head//3
2014-06-12 14:51:07.468876 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 22f30816/rbd_data.109a6b8b4567.0000000000000131/head//3, auth == maps.end()
2014-06-12 14:51:07.468897 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 22f30816/rbd_data.109a6b8b4567.0000000000000131/head//3
2014-06-12 14:51:07.468917 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj da570816/rbd_data.10ae6b8b4567.0000000000001d51/head//3, auth == maps.end()
2014-06-12 14:51:07.468938 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj da570816/rbd_data.10ae6b8b4567.0000000000001d51/head//3
2014-06-12 14:51:07.468960 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 26190816/rbd_data.10b06b8b4567.0000000000001834/head//3, auth == maps.end()
2014-06-12 14:51:07.468981 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 26190816/rbd_data.10b06b8b4567.0000000000001834/head//3
2014-06-12 14:51:07.469003 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1dc90816/rbd_data.10926b8b4567.0000000000001ea6/head//3, auth == maps.end()
2014-06-12 14:51:07.469025 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 1dc90816/rbd_data.10926b8b4567.0000000000001ea6/head//3
2014-06-12 14:51:07.469047 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7a3b0816/rbd_data.10916b8b4567.000000000000072c/head//3, auth == maps.end()
2014-06-12 14:51:07.469068 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 7a3b0816/rbd_data.10916b8b4567.000000000000072c/head//3
2014-06-12 14:51:07.469090 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7aef0816/rbd_data.108e6b8b4567.0000000000003155/head//3, auth == maps.end()
2014-06-12 14:51:07.469112 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 7aef0816/rbd_data.108e6b8b4567.0000000000003155/head//3
2014-06-12 14:51:07.469134 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b6441816/rbd_data.10a96b8b4567.00000000000039d6/head//3, auth == maps.end()
2014-06-12 14:51:07.469155 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj b6441816/rbd_data.10a96b8b4567.00000000000039d6/head//3
2014-06-12 14:51:07.469174 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj af171816/rbd_data.10b36b8b4567.0000000000000814/head//3, auth == maps.end()
2014-06-12 14:51:07.469195 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj af171816/rbd_data.10b36b8b4567.0000000000000814/head//3
2014-06-12 14:51:07.469217 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6aa81816/rbd_data.10b36b8b4567.0000000000000d79/head//3, auth == maps.end()
2014-06-12 14:51:07.469238 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 6aa81816/rbd_data.10b36b8b4567.0000000000000d79/head//3
2014-06-12 14:51:07.469260 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aca1816/rbd_data.10976b8b4567.0000000000000cf9/head//3, auth == maps.end()
2014-06-12 14:51:07.469282 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj aca1816/rbd_data.10976b8b4567.0000000000000cf9/head//3
2014-06-12 14:51:07.469304 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b67d1816/rbd_data.109b6b8b4567.00000000000006e8/head//3, auth == maps.end()
2014-06-12 14:51:07.469325 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj b67d1816/rbd_data.109b6b8b4567.00000000000006e8/head//3
2014-06-12 14:51:07.469348 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 79302816/rbd_data.10a96b8b4567.000000000000285f/head//3, auth == maps.end()
2014-06-12 14:51:07.469369 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 79302816/rbd_data.10a96b8b4567.000000000000285f/head//3
2014-06-12 14:51:07.469391 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6ab02816/rbd_data.10956b8b4567.000000000000310b/head//3, auth == maps.end()
2014-06-12 14:51:07.469413 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 6ab02816/rbd_data.10956b8b4567.000000000000310b/head//3
2014-06-12 14:51:07.469432 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 97932816/rbd_data.10916b8b4567.0000000000002e8e/head//3, auth == maps.end()
2014-06-12 14:51:07.469453 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 97932816/rbd_data.10916b8b4567.0000000000002e8e/head//3
2014-06-12 14:51:07.469475 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3e252816/rbd_data.10a66b8b4567.00000000000000ba/head//3, auth == maps.end()
2014-06-12 14:51:07.469496 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 3e252816/rbd_data.10a66b8b4567.00000000000000ba/head//3
2014-06-12 14:51:07.469517 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c0972816/rbd_data.10ac6b8b4567.0000000000000f94/head//3, auth == maps.end()
2014-06-12 14:51:07.469539 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj c0972816/rbd_data.10ac6b8b4567.0000000000000f94/head//3
2014-06-12 14:51:07.469561 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 96382816/rbd_data.10ae6b8b4567.0000000000002a1f/head//3, auth == maps.end()
2014-06-12 14:51:07.469582 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 96382816/rbd_data.10ae6b8b4567.0000000000002a1f/head//3
2014-06-12 14:51:07.469604 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 22982816/rbd_data.10b06b8b4567.0000000000001279/head//3, auth == maps.end()
2014-06-12 14:51:07.469641 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 22982816/rbd_data.10b06b8b4567.0000000000001279/head//3
2014-06-12 14:51:07.469666 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] 
2014-06-12 14:51:07.469682 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub
2014-06-12 14:51:07.469706 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  22982816/rbd_data.10b06b8b4567.0000000000001279/head//3 22982816/rbd_data.10b06b8b4567.0000000000001279/head//3(173'698 client.4313.0:43484 wrlock_by=unknown.0.0:0 dirty s 4194304 uv698)
2014-06-12 14:51:07.469735 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  96382816/rbd_data.10ae6b8b4567.0000000000002a1f/head//3 96382816/rbd_data.10ae6b8b4567.0000000000002a1f/head//3(173'1368 client.4311.0:82909 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1368)
2014-06-12 14:51:07.469761 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  c0972816/rbd_data.10ac6b8b4567.0000000000000f94/head//3 c0972816/rbd_data.10ac6b8b4567.0000000000000f94/head//3(173'598 client.4309.0:37425 wrlock_by=unknown.0.0:0 dirty s 4194304 uv598)
2014-06-12 14:51:07.469787 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  3e252816/rbd_data.10a66b8b4567.00000000000000ba/head//3 3e252816/rbd_data.10a66b8b4567.00000000000000ba/head//3(173'2021 client.4303.0:119403 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2021)
2014-06-12 14:51:07.469812 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  97932816/rbd_data.10916b8b4567.0000000000002e8e/head//3 97932816/rbd_data.10916b8b4567.0000000000002e8e/head//3(173'1701 client.4282.0:94263 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1701)
2014-06-12 14:51:07.469838 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  6ab02816/rbd_data.10956b8b4567.000000000000310b/head//3 6ab02816/rbd_data.10956b8b4567.000000000000310b/head//3(173'1756 client.4286.0:98173 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1756)
2014-06-12 14:51:07.469864 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  79302816/rbd_data.10a96b8b4567.000000000000285f/head//3 79302816/rbd_data.10a96b8b4567.000000000000285f/head//3(173'1495 client.4306.0:84632 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1495)
2014-06-12 14:51:07.469889 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  b67d1816/rbd_data.109b6b8b4567.00000000000006e8/head//3 b67d1816/rbd_data.109b6b8b4567.00000000000006e8/head//3(173'2213 client.4292.0:126110 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2213)
2014-06-12 14:51:07.469914 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  aca1816/rbd_data.10976b8b4567.0000000000000cf9/head//3 aca1816/rbd_data.10976b8b4567.0000000000000cf9/head//3(173'2366 client.4288.0:136051 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2366)
2014-06-12 14:51:07.469940 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  6aa81816/rbd_data.10b36b8b4567.0000000000000d79/head//3 6aa81816/rbd_data.10b36b8b4567.0000000000000d79/head//3(173'2330 client.4316.0:140718 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2330)
2014-06-12 14:51:07.469968 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  af171816/rbd_data.10b36b8b4567.0000000000000814/head//3 af171816/rbd_data.10b36b8b4567.0000000000000814/head//3(173'2243 client.4316.0:132004 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2243)
2014-06-12 14:51:07.469993 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  b6441816/rbd_data.10a96b8b4567.00000000000039d6/head//3 b6441816/rbd_data.10a96b8b4567.00000000000039d6/head//3(173'2008 client.4306.0:116481 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2008)
2014-06-12 14:51:07.470018 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  7aef0816/rbd_data.108e6b8b4567.0000000000003155/head//3 7aef0816/rbd_data.108e6b8b4567.0000000000003155/head//3(173'1722 client.4279.0:98284 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1722)
2014-06-12 14:51:07.470044 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  7a3b0816/rbd_data.10916b8b4567.000000000000072c/head//3 7a3b0816/rbd_data.10916b8b4567.000000000000072c/head//3(173'2208 client.4282.0:129699 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2208)
2014-06-12 14:51:07.470069 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  1dc90816/rbd_data.10926b8b4567.0000000000001ea6/head//3 1dc90816/rbd_data.10926b8b4567.0000000000001ea6/head//3(173'1153 client.4283.0:64268 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1153)
2014-06-12 14:51:07.470095 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  26190816/rbd_data.10b06b8b4567.0000000000001834/head//3 26190816/rbd_data.10b06b8b4567.0000000000001834/head//3(173'907 client.4313.0:53832 wrlock_by=unknown.0.0:0 dirty s 4194304 uv907)
2014-06-12 14:51:07.470120 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  da570816/rbd_data.10ae6b8b4567.0000000000001d51/head//3 da570816/rbd_data.10ae6b8b4567.0000000000001d51/head//3(173'985 client.4311.0:61215 wrlock_by=unknown.0.0:0 dirty s 4194304 uv985)
2014-06-12 14:51:07.470144 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  22f30816/rbd_data.109a6b8b4567.0000000000000131/head//3 22f30816/rbd_data.109a6b8b4567.0000000000000131/head//3(173'2035 client.4291.0:117565 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2035)
2014-06-12 14:51:07.470170 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  38130816/rbd_data.10a06b8b4567.0000000000002ca1/head//3 38130816/rbd_data.10a06b8b4567.0000000000002ca1/head//3(173'1520 client.4297.0:88958 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1520)
2014-06-12 14:51:07.470195 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  64d10816/rbd_data.109c6b8b4567.000000000000294e/head//3 64d10816/rbd_data.109c6b8b4567.000000000000294e/head//3(173'1488 client.4293.0:83758 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1488)
2014-06-12 14:51:07.470222 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  9abef016/rbd_data.109c6b8b4567.0000000000002b65/head//3 9abef016/rbd_data.109c6b8b4567.0000000000002b65/head//3(173'1577 client.4293.0:87866 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1577)
2014-06-12 14:51:07.470248 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  16ef016/rbd_data.10a36b8b4567.0000000000003103/head//3 16ef016/rbd_data.10a36b8b4567.0000000000003103/head//3(173'1645 client.4300.0:95851 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1645)
2014-06-12 14:51:07.470273 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  324ef016/rbd_data.109e6b8b4567.0000000000003306/head//3 324ef016/rbd_data.109e6b8b4567.0000000000003306/head//3(173'1728 client.4295.0:101741 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1728)
2014-06-12 14:51:07.470298 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  c93df016/rbd_data.10986b8b4567.000000000000083f/head//3 c93df016/rbd_data.10986b8b4567.000000000000083f/head//3(173'2234 client.4289.0:128417 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2234)
2014-06-12 14:51:07.470323 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  a70df016/rbd_data.10aa6b8b4567.0000000000000308/head//3 a70df016/rbd_data.10aa6b8b4567.0000000000000308/head//3(173'2078 client.4307.0:124096 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2078)
2014-06-12 14:51:07.470343 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:07.470363 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:07.470380 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:07.470384 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:07.470387 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:07.470440 7f9d60bd2700 10 collection_list_partial 3.16_head start:f2ce2816//0//-1 is_max:0
2014-06-12 14:51:07.470671 7f9d60bd2700 20 objects: [2e703816/rbd_data.10a36b8b4567.0000000000002f56/head//3,bce13816/rbd_data.10b06b8b4567.00000000000018c1/head//3,8b433816/rbd_data.108d6b8b4567.0000000000001497/head//3,c1553816/rbd_data.10916b8b4567.000000000000199a/head//3,85f63816/rbd_data.10a56b8b4567.0000000000003ae9/head//3,53b73816/rbd_data.10ac6b8b4567.000000000000180a/head//3,8fb3816/rbd_data.10916b8b4567.000000000000219d/head//3,c42c3816/rbd_data.109e6b8b4567.00000000000008c5/head//3,db914816/rbd_data.10ae6b8b4567.0000000000000714/head//3,7cb24816/rbd_data.10a36b8b4567.00000000000027cc/head//3,9bd34816/rbd_data.109f6b8b4567.0000000000002f2b/head//3,c1844816/rbd_data.10936b8b4567.0000000000002a8f/head//3,e2c54816/rbd_data.10936b8b4567.00000000000002f2/head//3,d0984816/rbd_data.10af6b8b4567.0000000000002d02/head//3,b0894816/rbd_data.10ac6b8b4567.000000000000260b/head//3,3fc94816/rbd_data.10af6b8b4567.0000000000001281/head//3,d73a4816/rbd_data.10ad6b8b4567.000000000000128c/head//3,826a4816/rbd_data.10ac6b8b4567.0000000000001464/head//3,4cea4816/rbd_data.10a36b8b4567.0000000000000548/head//3,654b4816/rbd_data.109b6b8b4567.00000000000009de/head//3,660e4816/rbd_data.10986b8b4567.0000000000001346/head//3,b3ef4816/rbd_data.10aa6b8b4567.0000000000001a49/head//3,58b45816/rbd_data.109a6b8b4567.0000000000002409/head//3,f1b55816/rbd_data.10986b8b4567.0000000000003b0c/head//3,1e575816/rbd_data.10af6b8b4567.0000000000001c0e/head//3]
2014-06-12 14:51:07.471113 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  requesting scrubmap from osd.22
2014-06-12 14:51:07.471147 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:07.471179 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:07.471191 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:07.471203 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:07.471214 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:07.471226 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_map_chunk [f2ce2816//0//-1,2fe85816//0//-1)
2014-06-12 14:51:07.471240 7f9d60bd2700 10 collection_list_partial 3.16_head start:f2ce2816//0//-1 is_max:0
2014-06-12 14:51:07.471807 7f9d60bd2700 20 objects: [2e703816/rbd_data.10a36b8b4567.0000000000002f56/head//3,bce13816/rbd_data.10b06b8b4567.00000000000018c1/head//3,8b433816/rbd_data.108d6b8b4567.0000000000001497/head//3,c1553816/rbd_data.10916b8b4567.000000000000199a/head//3,85f63816/rbd_data.10a56b8b4567.0000000000003ae9/head//3,53b73816/rbd_data.10ac6b8b4567.000000000000180a/head//3,8fb3816/rbd_data.10916b8b4567.000000000000219d/head//3,c42c3816/rbd_data.109e6b8b4567.00000000000008c5/head//3,db914816/rbd_data.10ae6b8b4567.0000000000000714/head//3,7cb24816/rbd_data.10a36b8b4567.00000000000027cc/head//3,9bd34816/rbd_data.109f6b8b4567.0000000000002f2b/head//3,c1844816/rbd_data.10936b8b4567.0000000000002a8f/head//3,e2c54816/rbd_data.10936b8b4567.00000000000002f2/head//3,d0984816/rbd_data.10af6b8b4567.0000000000002d02/head//3,b0894816/rbd_data.10ac6b8b4567.000000000000260b/head//3,3fc94816/rbd_data.10af6b8b4567.0000000000001281/head//3,d73a4816/rbd_data.10ad6b8b4567.000000000000128c/head//3,826a4816/rbd_data.10ac6b8b4567.0000000000001464/head//3,4cea4816/rbd_data.10a36b8b4567.0000000000000548/head//3,654b4816/rbd_data.109b6b8b4567.00000000000009de/head//3,660e4816/rbd_data.10986b8b4567.0000000000001346/head//3,b3ef4816/rbd_data.10aa6b8b4567.0000000000001a49/head//3,58b45816/rbd_data.109a6b8b4567.0000000000002409/head//3,f1b55816/rbd_data.10986b8b4567.0000000000003b0c/head//3,1e575816/rbd_data.10af6b8b4567.0000000000001c0e/head//3,2fe85816/rbd_data.10a26b8b4567.0000000000002eaf/head//3,86495816/rbd_data.109b6b8b4567.0000000000001b44/head//3,90da5816/rbd_data.10aa6b8b4567.0000000000003bd3/head//3,fad16816/rbd_data.10a16b8b4567.0000000000001731/head//3,77e46816/rbd_data.10966b8b4567.000000000000210d/head//3,cef76816/rbd_data.10b16b8b4567.0000000000001724/head//3,1ad96816/rbd_data.10b46b8b4567.00000000000032a8/head//3,867d6816/rbd_data.10aa6b8b4567.0000000000002162/head//3,8fd6816/rbd_data.108e6b8b4567.00000000000008a3/head//3,1ee6816/rbd_data.10a56b8b4567.0000000000001ba0/head//3,84cf6816/rbd_data.108d6b8b4567.0000000000001e79/head//3,f9e17816/rbd_data.10a56b8b4567.0000000000003054/head//3,99027816/rbd_data.10ae6b8b4567.000000000000038d/head//3,fdb37816/rbd_data.10a16b8b4567.00000000000022f9/head//3,dc057816/rbd_data.108e6b8b4567.0000000000001546/head//3,71267816/rbd_data.10a66b8b4567.0000000000000088/head//3,e6b77816/rbd_data.10916b8b4567.0000000000002eda/head//3,dad77816/rbd_data.10a06b8b4567.0000000000002d05/head//3,68597816/rbd_data.10af6b8b4567.0000000000000cbd/head//3,8cea7816/rbd_data.109b6b8b4567.0000000000002f73/head//3,e3bb7816/rbd_data.109c6b8b4567.0000000000003a6e/head//3,ccbc7816/rbd_data.10976b8b4567.0000000000000144/head//3,9b8d7816/rbd_data.10976b8b4567.0000000000001c6d/head//3,9a9f7816/rbd_data.10976b8b4567.0000000000002740/head//3,33508816/rbd_data.108e6b8b4567.000000000000334a/head//3,aaf18816/rbd_data.10936b8b4567.0000000000000552/head//3,c3748816/rbd_data.10a66b8b4567.0000000000001f76/head//3,5be58816/rbd_data.10b46b8b4567.000000000000033c/head//3,1f768816/rbd_data.10b26b8b4567.00000000000005f8/head//3,496b8816/rbd_data.109b6b8b4567.000000000000196f/head//3,e0ee8816/rbd_data.10b16b8b4567.00000000000008f8/head//3,9dbf8816/rbd_data.10916b8b4567.00000000000020ef/head//3,5b519816/rbd_data.10af6b8b4567.00000000000031c9/head//3,d2349816/rbd_data.109c6b8b4567.0000000000001fc7/head//3,86a59816/rbd_data.108f6b8b4567.00000000000014dd/head//3,c969816/rbd_data.10ad6b8b4567.0000000000003858/head//3,e6b69816/rbd_data.108e6b8b4567.0000000000000445/head//3,55b9816/rbd_data.10a26b8b4567.0000000000000a17/head//3,329b9816/rbd_data.10ac6b8b4567.0000000000002423/head//3]
2014-06-12 14:51:07.471906 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:07.471927 7f9d60bd2700 10 stat 3.16_head/2e703816/rbd_data.10a36b8b4567.0000000000002f56/head//3
2014-06-12 14:51:07.472071 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 2e703816/rbd_data.10a36b8b4567.0000000000002f56/head//3
2014-06-12 14:51:07.486995 7f9d60bd2700 10 getattrs 3.16_head/2e703816/rbd_data.10a36b8b4567.0000000000002f56/head//3 = 0
2014-06-12 14:51:07.487011 7f9d60bd2700 10 stat 3.16_head/bce13816/rbd_data.10b06b8b4567.00000000000018c1/head//3
2014-06-12 14:51:07.487139 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid bce13816/rbd_data.10b06b8b4567.00000000000018c1/head//3
2014-06-12 14:51:07.511282 7f9d60bd2700 10 getattrs 3.16_head/bce13816/rbd_data.10b06b8b4567.00000000000018c1/head//3 = 0
2014-06-12 14:51:07.511295 7f9d60bd2700 10 stat 3.16_head/8b433816/rbd_data.108d6b8b4567.0000000000001497/head//3
2014-06-12 14:51:07.511431 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 8b433816/rbd_data.108d6b8b4567.0000000000001497/head//3
2014-06-12 14:51:07.512005 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:07.546051 7f9d60bd2700 10 getattrs 3.16_head/8b433816/rbd_data.108d6b8b4567.0000000000001497/head//3 = 0
2014-06-12 14:51:07.546062 7f9d60bd2700 10 stat 3.16_head/c1553816/rbd_data.10916b8b4567.000000000000199a/head//3
2014-06-12 14:51:07.546198 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid c1553816/rbd_data.10916b8b4567.000000000000199a/head//3
2014-06-12 14:51:07.567685 7f9d60bd2700 10 getattrs 3.16_head/c1553816/rbd_data.10916b8b4567.000000000000199a/head//3 = 0
2014-06-12 14:51:07.567695 7f9d60bd2700 10 stat 3.16_head/85f63816/rbd_data.10a56b8b4567.0000000000003ae9/head//3
2014-06-12 14:51:07.567837 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 85f63816/rbd_data.10a56b8b4567.0000000000003ae9/head//3
2014-06-12 14:51:07.582447 7f9d60bd2700 10 getattrs 3.16_head/85f63816/rbd_data.10a56b8b4567.0000000000003ae9/head//3 = 0
2014-06-12 14:51:07.582458 7f9d60bd2700 10 stat 3.16_head/53b73816/rbd_data.10ac6b8b4567.000000000000180a/head//3
2014-06-12 14:51:07.582586 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 53b73816/rbd_data.10ac6b8b4567.000000000000180a/head//3
2014-06-12 14:51:07.616743 7f9d60bd2700 10 getattrs 3.16_head/53b73816/rbd_data.10ac6b8b4567.000000000000180a/head//3 = 0
2014-06-12 14:51:07.616756 7f9d60bd2700 10 stat 3.16_head/8fb3816/rbd_data.10916b8b4567.000000000000219d/head//3
2014-06-12 14:51:07.616891 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 8fb3816/rbd_data.10916b8b4567.000000000000219d/head//3
2014-06-12 14:51:07.648007 7f9d60bd2700 10 getattrs 3.16_head/8fb3816/rbd_data.10916b8b4567.000000000000219d/head//3 = 0
2014-06-12 14:51:07.648019 7f9d60bd2700 10 stat 3.16_head/c42c3816/rbd_data.109e6b8b4567.00000000000008c5/head//3
2014-06-12 14:51:07.648156 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid c42c3816/rbd_data.109e6b8b4567.00000000000008c5/head//3
2014-06-12 14:51:07.695239 7f9d60bd2700 10 getattrs 3.16_head/c42c3816/rbd_data.109e6b8b4567.00000000000008c5/head//3 = 0
2014-06-12 14:51:07.695253 7f9d60bd2700 10 stat 3.16_head/db914816/rbd_data.10ae6b8b4567.0000000000000714/head//3
2014-06-12 14:51:07.695400 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid db914816/rbd_data.10ae6b8b4567.0000000000000714/head//3
2014-06-12 14:51:07.749375 7f9d60bd2700 10 getattrs 3.16_head/db914816/rbd_data.10ae6b8b4567.0000000000000714/head//3 = 0
2014-06-12 14:51:07.749407 7f9d60bd2700 10 stat 3.16_head/7cb24816/rbd_data.10a36b8b4567.00000000000027cc/head//3
2014-06-12 14:51:07.749554 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 7cb24816/rbd_data.10a36b8b4567.00000000000027cc/head//3
2014-06-12 14:51:07.784989 7f9d60bd2700 10 getattrs 3.16_head/7cb24816/rbd_data.10a36b8b4567.00000000000027cc/head//3 = 0
2014-06-12 14:51:07.785004 7f9d60bd2700 10 stat 3.16_head/9bd34816/rbd_data.109f6b8b4567.0000000000002f2b/head//3
2014-06-12 14:51:07.785191 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 9bd34816/rbd_data.109f6b8b4567.0000000000002f2b/head//3
2014-06-12 14:51:07.800676 7f9d60bd2700 10 getattrs 3.16_head/9bd34816/rbd_data.109f6b8b4567.0000000000002f2b/head//3 = 0
2014-06-12 14:51:07.800684 7f9d60bd2700 10 stat 3.16_head/c1844816/rbd_data.10936b8b4567.0000000000002a8f/head//3
2014-06-12 14:51:07.800796 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid c1844816/rbd_data.10936b8b4567.0000000000002a8f/head//3
2014-06-12 14:51:07.800895 7f9d60bd2700 10 getattrs 3.16_head/c1844816/rbd_data.10936b8b4567.0000000000002a8f/head//3 = 0
2014-06-12 14:51:07.800901 7f9d60bd2700 10 stat 3.16_head/e2c54816/rbd_data.10936b8b4567.00000000000002f2/head//3
2014-06-12 14:51:07.800985 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid e2c54816/rbd_data.10936b8b4567.00000000000002f2/head//3
2014-06-12 14:51:07.828301 7f9d60bd2700 10 getattrs 3.16_head/e2c54816/rbd_data.10936b8b4567.00000000000002f2/head//3 = 0
2014-06-12 14:51:07.828315 7f9d60bd2700 10 stat 3.16_head/d0984816/rbd_data.10af6b8b4567.0000000000002d02/head//3
2014-06-12 14:51:07.828405 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid d0984816/rbd_data.10af6b8b4567.0000000000002d02/head//3
2014-06-12 14:51:07.854814 7f9d60bd2700 10 getattrs 3.16_head/d0984816/rbd_data.10af6b8b4567.0000000000002d02/head//3 = 0
2014-06-12 14:51:07.854821 7f9d60bd2700 10 stat 3.16_head/b0894816/rbd_data.10ac6b8b4567.000000000000260b/head//3
2014-06-12 14:51:07.854916 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid b0894816/rbd_data.10ac6b8b4567.000000000000260b/head//3
2014-06-12 14:51:07.869305 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:07.869325 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:51:07.869330 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:51:07.869334 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:51:07.869338 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:51:07.869415 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.59 >= max 0.5 = no, load too high
2014-06-12 14:51:07.869430 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:51:07.869441 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:07.869434 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92915.4 < max (604800 seconds)
2014-06-12 14:51:07.869463 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:07.869467 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:07.869468 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:07.870894 7f9d60bd2700 10 getattrs 3.16_head/b0894816/rbd_data.10ac6b8b4567.000000000000260b/head//3 = 0
2014-06-12 14:51:07.870902 7f9d60bd2700 10 stat 3.16_head/3fc94816/rbd_data.10af6b8b4567.0000000000001281/head//3
2014-06-12 14:51:07.870994 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 3fc94816/rbd_data.10af6b8b4567.0000000000001281/head//3
2014-06-12 14:51:07.886599 7f9d60bd2700 10 getattrs 3.16_head/3fc94816/rbd_data.10af6b8b4567.0000000000001281/head//3 = 0
2014-06-12 14:51:07.886619 7f9d60bd2700 10 stat 3.16_head/d73a4816/rbd_data.10ad6b8b4567.000000000000128c/head//3
2014-06-12 14:51:07.886800 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid d73a4816/rbd_data.10ad6b8b4567.000000000000128c/head//3
2014-06-12 14:51:07.904345 7f9d60bd2700 10 getattrs 3.16_head/d73a4816/rbd_data.10ad6b8b4567.000000000000128c/head//3 = 0
2014-06-12 14:51:07.904353 7f9d60bd2700 10 stat 3.16_head/826a4816/rbd_data.10ac6b8b4567.0000000000001464/head//3
2014-06-12 14:51:07.904441 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 826a4816/rbd_data.10ac6b8b4567.0000000000001464/head//3
2014-06-12 14:51:07.919825 7f9d60bd2700 10 getattrs 3.16_head/826a4816/rbd_data.10ac6b8b4567.0000000000001464/head//3 = 0
2014-06-12 14:51:07.919834 7f9d60bd2700 10 stat 3.16_head/4cea4816/rbd_data.10a36b8b4567.0000000000000548/head//3
2014-06-12 14:51:07.919929 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 4cea4816/rbd_data.10a36b8b4567.0000000000000548/head//3
2014-06-12 14:51:07.973506 7f9d60bd2700 10 getattrs 3.16_head/4cea4816/rbd_data.10a36b8b4567.0000000000000548/head//3 = 0
2014-06-12 14:51:07.973523 7f9d60bd2700 10 stat 3.16_head/654b4816/rbd_data.109b6b8b4567.00000000000009de/head//3
2014-06-12 14:51:07.973726 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 654b4816/rbd_data.109b6b8b4567.00000000000009de/head//3
2014-06-12 14:51:07.975656 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.018052 7f9d60bd2700 10 getattrs 3.16_head/654b4816/rbd_data.109b6b8b4567.00000000000009de/head//3 = 0
2014-06-12 14:51:08.018072 7f9d60bd2700 10 stat 3.16_head/660e4816/rbd_data.10986b8b4567.0000000000001346/head//3
2014-06-12 14:51:08.018186 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 660e4816/rbd_data.10986b8b4567.0000000000001346/head//3
2014-06-12 14:51:08.050033 7f9d60bd2700 10 getattrs 3.16_head/660e4816/rbd_data.10986b8b4567.0000000000001346/head//3 = 0
2014-06-12 14:51:08.050048 7f9d60bd2700 10 stat 3.16_head/b3ef4816/rbd_data.10aa6b8b4567.0000000000001a49/head//3
2014-06-12 14:51:08.050180 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid b3ef4816/rbd_data.10aa6b8b4567.0000000000001a49/head//3
2014-06-12 14:51:08.063437 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.072471 7f9d60bd2700 10 getattrs 3.16_head/b3ef4816/rbd_data.10aa6b8b4567.0000000000001a49/head//3 = 0
2014-06-12 14:51:08.072489 7f9d60bd2700 10 stat 3.16_head/58b45816/rbd_data.109a6b8b4567.0000000000002409/head//3
2014-06-12 14:51:08.072614 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 58b45816/rbd_data.109a6b8b4567.0000000000002409/head//3
2014-06-12 14:51:08.077680 7f9d58d5b700  5 -- op tracker -- , seq: 748563, time: 2014-06-12 14:51:08.077592, event: header_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.077701 7f9d58d5b700  5 -- op tracker -- , seq: 748563, time: 2014-06-12 14:51:08.077594, event: throttled, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.077712 7f9d58d5b700  5 -- op tracker -- , seq: 748563, time: 2014-06-12 14:51:08.077646, event: all_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.077724 7f9d58d5b700  5 -- op tracker -- , seq: 748563, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.077731 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:08.077740 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:51:08.077751 7f9d58d5b700 15 osd.2 173 enqueue_op 0x2d17a800 prio 127 cost 10717 latency 0.000158 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:08.077810 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:08.089574 7f9d60bd2700 10 getattrs 3.16_head/58b45816/rbd_data.109a6b8b4567.0000000000002409/head//3 = 0
2014-06-12 14:51:08.089589 7f9d60bd2700 10 stat 3.16_head/f1b55816/rbd_data.10986b8b4567.0000000000003b0c/head//3
2014-06-12 14:51:08.089769 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid f1b55816/rbd_data.10986b8b4567.0000000000003b0c/head//3
2014-06-12 14:51:08.091766 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:08.091896 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:08.121839 7f9d60bd2700 10 getattrs 3.16_head/f1b55816/rbd_data.10986b8b4567.0000000000003b0c/head//3 = 0
2014-06-12 14:51:08.121853 7f9d60bd2700 10 stat 3.16_head/1e575816/rbd_data.10af6b8b4567.0000000000001c0e/head//3
2014-06-12 14:51:08.122013 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 1e575816/rbd_data.10af6b8b4567.0000000000001c0e/head//3
2014-06-12 14:51:08.139815 7f9d60bd2700 10 getattrs 3.16_head/1e575816/rbd_data.10af6b8b4567.0000000000001c0e/head//3 = 0
2014-06-12 14:51:08.139823 7f9d60bd2700 10 collection_getattrs 3.16_head
2014-06-12 14:51:08.139876 7f9d60bd2700 10 collection_getattrs 3.16_head = 0
2014-06-12 14:51:08.139881 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:08.139920 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:08.139927 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:08.139944 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:08.139947 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:08.139980 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2d17a800 prio 127 cost 10717 latency 0.062387 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]
2014-06-12 14:51:08.140047 7f9d61bd4700  5 -- op tracker -- , seq: 748563, time: 2014-06-12 14:51:08.140039, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.140065 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] handle_message: 0x2d17a800
2014-06-12 14:51:08.140081 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:08.140103 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:08.140122 7f9d61bd4700  5 -- op tracker -- , seq: 748563, time: 2014-06-12 14:51:08.140116, event: started, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.140133 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  got 22 scrub map
2014-06-12 14:51:08.140238 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] map version is 173'2396
2014-06-12 14:51:08.140258 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2d17a800 finish
2014-06-12 14:51:08.140269 7f9d61bd4700  5 -- op tracker -- , seq: 748563, time: 2014-06-12 14:51:08.140263, event: done, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.140292 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:08.140297 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:08.140298 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:08.140311 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:08.140333 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:08.140346 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:08.140385 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:08.140391 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:08.140396 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub replica 22 has 25 items
2014-06-12 14:51:08.140413 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2e703816/rbd_data.10a36b8b4567.0000000000002f56/head//3, auth == maps.end()
2014-06-12 14:51:08.140427 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 2e703816/rbd_data.10a36b8b4567.0000000000002f56/head//3
2014-06-12 14:51:08.140438 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bce13816/rbd_data.10b06b8b4567.00000000000018c1/head//3, auth == maps.end()
2014-06-12 14:51:08.140445 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj bce13816/rbd_data.10b06b8b4567.00000000000018c1/head//3
2014-06-12 14:51:08.140453 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8b433816/rbd_data.108d6b8b4567.0000000000001497/head//3, auth == maps.end()
2014-06-12 14:51:08.140461 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 8b433816/rbd_data.108d6b8b4567.0000000000001497/head//3
2014-06-12 14:51:08.140469 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c1553816/rbd_data.10916b8b4567.000000000000199a/head//3, auth == maps.end()
2014-06-12 14:51:08.140476 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj c1553816/rbd_data.10916b8b4567.000000000000199a/head//3
2014-06-12 14:51:08.140484 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 85f63816/rbd_data.10a56b8b4567.0000000000003ae9/head//3, auth == maps.end()
2014-06-12 14:51:08.140492 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 85f63816/rbd_data.10a56b8b4567.0000000000003ae9/head//3
2014-06-12 14:51:08.140500 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 53b73816/rbd_data.10ac6b8b4567.000000000000180a/head//3, auth == maps.end()
2014-06-12 14:51:08.140508 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 53b73816/rbd_data.10ac6b8b4567.000000000000180a/head//3
2014-06-12 14:51:08.140516 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8fb3816/rbd_data.10916b8b4567.000000000000219d/head//3, auth == maps.end()
2014-06-12 14:51:08.140522 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 8fb3816/rbd_data.10916b8b4567.000000000000219d/head//3
2014-06-12 14:51:08.140531 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c42c3816/rbd_data.109e6b8b4567.00000000000008c5/head//3, auth == maps.end()
2014-06-12 14:51:08.140538 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj c42c3816/rbd_data.109e6b8b4567.00000000000008c5/head//3
2014-06-12 14:51:08.140547 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj db914816/rbd_data.10ae6b8b4567.0000000000000714/head//3, auth == maps.end()
2014-06-12 14:51:08.140555 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj db914816/rbd_data.10ae6b8b4567.0000000000000714/head//3
2014-06-12 14:51:08.140563 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7cb24816/rbd_data.10a36b8b4567.00000000000027cc/head//3, auth == maps.end()
2014-06-12 14:51:08.140571 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 7cb24816/rbd_data.10a36b8b4567.00000000000027cc/head//3
2014-06-12 14:51:08.140579 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9bd34816/rbd_data.109f6b8b4567.0000000000002f2b/head//3, auth == maps.end()
2014-06-12 14:51:08.140587 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 9bd34816/rbd_data.109f6b8b4567.0000000000002f2b/head//3
2014-06-12 14:51:08.140596 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c1844816/rbd_data.10936b8b4567.0000000000002a8f/head//3, auth == maps.end()
2014-06-12 14:51:08.140605 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj c1844816/rbd_data.10936b8b4567.0000000000002a8f/head//3
2014-06-12 14:51:08.140614 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e2c54816/rbd_data.10936b8b4567.00000000000002f2/head//3, auth == maps.end()
2014-06-12 14:51:08.140621 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj e2c54816/rbd_data.10936b8b4567.00000000000002f2/head//3
2014-06-12 14:51:08.140630 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d0984816/rbd_data.10af6b8b4567.0000000000002d02/head//3, auth == maps.end()
2014-06-12 14:51:08.140638 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj d0984816/rbd_data.10af6b8b4567.0000000000002d02/head//3
2014-06-12 14:51:08.140646 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b0894816/rbd_data.10ac6b8b4567.000000000000260b/head//3, auth == maps.end()
2014-06-12 14:51:08.140654 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj b0894816/rbd_data.10ac6b8b4567.000000000000260b/head//3
2014-06-12 14:51:08.140662 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3fc94816/rbd_data.10af6b8b4567.0000000000001281/head//3, auth == maps.end()
2014-06-12 14:51:08.140670 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 3fc94816/rbd_data.10af6b8b4567.0000000000001281/head//3
2014-06-12 14:51:08.140677 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d73a4816/rbd_data.10ad6b8b4567.000000000000128c/head//3, auth == maps.end()
2014-06-12 14:51:08.140684 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj d73a4816/rbd_data.10ad6b8b4567.000000000000128c/head//3
2014-06-12 14:51:08.140692 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 826a4816/rbd_data.10ac6b8b4567.0000000000001464/head//3, auth == maps.end()
2014-06-12 14:51:08.140699 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 826a4816/rbd_data.10ac6b8b4567.0000000000001464/head//3
2014-06-12 14:51:08.140706 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4cea4816/rbd_data.10a36b8b4567.0000000000000548/head//3, auth == maps.end()
2014-06-12 14:51:08.140713 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 4cea4816/rbd_data.10a36b8b4567.0000000000000548/head//3
2014-06-12 14:51:08.140720 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 654b4816/rbd_data.109b6b8b4567.00000000000009de/head//3, auth == maps.end()
2014-06-12 14:51:08.140727 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 654b4816/rbd_data.109b6b8b4567.00000000000009de/head//3
2014-06-12 14:51:08.140734 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 660e4816/rbd_data.10986b8b4567.0000000000001346/head//3, auth == maps.end()
2014-06-12 14:51:08.140741 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 660e4816/rbd_data.10986b8b4567.0000000000001346/head//3
2014-06-12 14:51:08.140749 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b3ef4816/rbd_data.10aa6b8b4567.0000000000001a49/head//3, auth == maps.end()
2014-06-12 14:51:08.140755 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj b3ef4816/rbd_data.10aa6b8b4567.0000000000001a49/head//3
2014-06-12 14:51:08.140764 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 58b45816/rbd_data.109a6b8b4567.0000000000002409/head//3, auth == maps.end()
2014-06-12 14:51:08.140773 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 58b45816/rbd_data.109a6b8b4567.0000000000002409/head//3
2014-06-12 14:51:08.140781 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f1b55816/rbd_data.10986b8b4567.0000000000003b0c/head//3, auth == maps.end()
2014-06-12 14:51:08.140789 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj f1b55816/rbd_data.10986b8b4567.0000000000003b0c/head//3
2014-06-12 14:51:08.140798 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1e575816/rbd_data.10af6b8b4567.0000000000001c0e/head//3, auth == maps.end()
2014-06-12 14:51:08.140805 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 1e575816/rbd_data.10af6b8b4567.0000000000001c0e/head//3
2014-06-12 14:51:08.140814 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] 
2014-06-12 14:51:08.140820 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub
2014-06-12 14:51:08.140831 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  1e575816/rbd_data.10af6b8b4567.0000000000001c0e/head//3 1e575816/rbd_data.10af6b8b4567.0000000000001c0e/head//3(173'1097 client.4312.0:62705 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1097)
2014-06-12 14:51:08.140843 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  f1b55816/rbd_data.10986b8b4567.0000000000003b0c/head//3 f1b55816/rbd_data.10986b8b4567.0000000000003b0c/head//3(173'1922 client.4289.0:114381 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1922)
2014-06-12 14:51:08.140853 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  58b45816/rbd_data.109a6b8b4567.0000000000002409/head//3 58b45816/rbd_data.109a6b8b4567.0000000000002409/head//3(173'1259 client.4291.0:72901 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1259)
2014-06-12 14:51:08.140864 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  b3ef4816/rbd_data.10aa6b8b4567.0000000000001a49/head//3 b3ef4816/rbd_data.10aa6b8b4567.0000000000001a49/head//3(173'991 client.4307.0:59565 wrlock_by=unknown.0.0:0 dirty s 4194304 uv991)
2014-06-12 14:51:08.140874 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  660e4816/rbd_data.10986b8b4567.0000000000001346/head//3 660e4816/rbd_data.10986b8b4567.0000000000001346/head//3(173'682 client.4289.0:43366 wrlock_by=unknown.0.0:0 dirty s 4194304 uv682)
2014-06-12 14:51:08.140885 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  654b4816/rbd_data.109b6b8b4567.00000000000009de/head//3 654b4816/rbd_data.109b6b8b4567.00000000000009de/head//3(173'2300 client.4292.0:131586 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2300)
2014-06-12 14:51:08.140895 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  4cea4816/rbd_data.10a36b8b4567.0000000000000548/head//3 4cea4816/rbd_data.10a36b8b4567.0000000000000548/head//3(173'2151 client.4300.0:123048 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2151)
2014-06-12 14:51:08.140905 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  826a4816/rbd_data.10ac6b8b4567.0000000000001464/head//3 826a4816/rbd_data.10ac6b8b4567.0000000000001464/head//3(173'729 client.4309.0:45698 wrlock_by=unknown.0.0:0 dirty s 4194304 uv729)
2014-06-12 14:51:08.140915 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  d73a4816/rbd_data.10ad6b8b4567.000000000000128c/head//3 d73a4816/rbd_data.10ad6b8b4567.000000000000128c/head//3(173'673 client.4310.0:42553 wrlock_by=unknown.0.0:0 dirty s 4194304 uv673)
2014-06-12 14:51:08.140926 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  3fc94816/rbd_data.10af6b8b4567.0000000000001281/head//3 3fc94816/rbd_data.10af6b8b4567.0000000000001281/head//3(173'720 client.4312.0:45044 wrlock_by=unknown.0.0:0 dirty s 4194304 uv720)
2014-06-12 14:51:08.140936 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  b0894816/rbd_data.10ac6b8b4567.000000000000260b/head//3 b0894816/rbd_data.10ac6b8b4567.000000000000260b/head//3(173'1392 client.4309.0:76390 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1392)
2014-06-12 14:51:08.140946 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  d0984816/rbd_data.10af6b8b4567.0000000000002d02/head//3 d0984816/rbd_data.10af6b8b4567.0000000000002d02/head//3(173'1604 client.4312.0:92899 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1604)
2014-06-12 14:51:08.140956 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  e2c54816/rbd_data.10936b8b4567.00000000000002f2/head//3 e2c54816/rbd_data.10936b8b4567.00000000000002f2/head//3(173'2073 client.4284.0:119357 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2073)
2014-06-12 14:51:08.140967 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  c1844816/rbd_data.10936b8b4567.0000000000002a8f/head//3 c1844816/rbd_data.10936b8b4567.0000000000002a8f/head//3(173'1504 client.4284.0:84293 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1504)
2014-06-12 14:51:08.140977 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  9bd34816/rbd_data.109f6b8b4567.0000000000002f2b/head//3 9bd34816/rbd_data.109f6b8b4567.0000000000002f2b/head//3(173'1528 client.4296.0:92473 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1528)
2014-06-12 14:51:08.140987 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  7cb24816/rbd_data.10a36b8b4567.00000000000027cc/head//3 7cb24816/rbd_data.10a36b8b4567.00000000000027cc/head//3(173'1341 client.4300.0:79150 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1341)
2014-06-12 14:51:08.140997 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  db914816/rbd_data.10ae6b8b4567.0000000000000714/head//3 db914816/rbd_data.10ae6b8b4567.0000000000000714/head//3(173'2228 client.4311.0:126318 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2228)
2014-06-12 14:51:08.141008 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  c42c3816/rbd_data.109e6b8b4567.00000000000008c5/head//3 c42c3816/rbd_data.109e6b8b4567.00000000000008c5/head//3(173'2285 client.4295.0:131450 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2285)
2014-06-12 14:51:08.141018 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  8fb3816/rbd_data.10916b8b4567.000000000000219d/head//3 8fb3816/rbd_data.10916b8b4567.000000000000219d/head//3(173'1243 client.4282.0:70302 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1243)
2014-06-12 14:51:08.141028 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  53b73816/rbd_data.10ac6b8b4567.000000000000180a/head//3 53b73816/rbd_data.10ac6b8b4567.000000000000180a/head//3(173'867 client.4309.0:51992 wrlock_by=unknown.0.0:0 dirty s 4194304 uv867)
2014-06-12 14:51:08.141038 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  85f63816/rbd_data.10a56b8b4567.0000000000003ae9/head//3 85f63816/rbd_data.10a56b8b4567.0000000000003ae9/head//3(173'1988 client.4302.0:117515 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1988)
2014-06-12 14:51:08.141048 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  c1553816/rbd_data.10916b8b4567.000000000000199a/head//3 c1553816/rbd_data.10916b8b4567.000000000000199a/head//3(173'1003 client.4282.0:56342 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1003)
2014-06-12 14:51:08.141058 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  8b433816/rbd_data.108d6b8b4567.0000000000001497/head//3 8b433816/rbd_data.108d6b8b4567.0000000000001497/head//3(173'711 client.4278.0:47789 wrlock_by=unknown.0.0:0 dirty s 4194304 uv711)
2014-06-12 14:51:08.141069 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  bce13816/rbd_data.10b06b8b4567.00000000000018c1/head//3 bce13816/rbd_data.10b06b8b4567.00000000000018c1/head//3(173'927 client.4313.0:54793 wrlock_by=unknown.0.0:0 dirty s 4194304 uv927)
2014-06-12 14:51:08.141079 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  2e703816/rbd_data.10a36b8b4567.0000000000002f56/head//3 2e703816/rbd_data.10a36b8b4567.0000000000002f56/head//3(173'1586 client.4300.0:92764 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1586)
2014-06-12 14:51:08.141087 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:08.141095 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:08.141102 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:08.141104 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:08.141105 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:08.141133 7f9d60bd2700 10 collection_list_partial 3.16_head start:2fe85816//0//-1 is_max:0
2014-06-12 14:51:08.141308 7f9d60bd2700 20 objects: [86495816/rbd_data.109b6b8b4567.0000000000001b44/head//3,90da5816/rbd_data.10aa6b8b4567.0000000000003bd3/head//3,fad16816/rbd_data.10a16b8b4567.0000000000001731/head//3,77e46816/rbd_data.10966b8b4567.000000000000210d/head//3,cef76816/rbd_data.10b16b8b4567.0000000000001724/head//3,1ad96816/rbd_data.10b46b8b4567.00000000000032a8/head//3,867d6816/rbd_data.10aa6b8b4567.0000000000002162/head//3,8fd6816/rbd_data.108e6b8b4567.00000000000008a3/head//3,1ee6816/rbd_data.10a56b8b4567.0000000000001ba0/head//3,84cf6816/rbd_data.108d6b8b4567.0000000000001e79/head//3,f9e17816/rbd_data.10a56b8b4567.0000000000003054/head//3,99027816/rbd_data.10ae6b8b4567.000000000000038d/head//3,fdb37816/rbd_data.10a16b8b4567.00000000000022f9/head//3,dc057816/rbd_data.108e6b8b4567.0000000000001546/head//3,71267816/rbd_data.10a66b8b4567.0000000000000088/head//3,e6b77816/rbd_data.10916b8b4567.0000000000002eda/head//3,dad77816/rbd_data.10a06b8b4567.0000000000002d05/head//3,68597816/rbd_data.10af6b8b4567.0000000000000cbd/head//3,8cea7816/rbd_data.109b6b8b4567.0000000000002f73/head//3,e3bb7816/rbd_data.109c6b8b4567.0000000000003a6e/head//3,ccbc7816/rbd_data.10976b8b4567.0000000000000144/head//3,9b8d7816/rbd_data.10976b8b4567.0000000000001c6d/head//3,9a9f7816/rbd_data.10976b8b4567.0000000000002740/head//3,33508816/rbd_data.108e6b8b4567.000000000000334a/head//3,aaf18816/rbd_data.10936b8b4567.0000000000000552/head//3]
2014-06-12 14:51:08.141729 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  requesting scrubmap from osd.22
2014-06-12 14:51:08.141749 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:08.141767 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:08.141773 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:08.141777 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:08.141790 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:08.141794 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_map_chunk [2fe85816//0//-1,c3748816//0//-1)
2014-06-12 14:51:08.141800 7f9d60bd2700 10 collection_list_partial 3.16_head start:2fe85816//0//-1 is_max:0
2014-06-12 14:51:08.142215 7f9d60bd2700 20 objects: [86495816/rbd_data.109b6b8b4567.0000000000001b44/head//3,90da5816/rbd_data.10aa6b8b4567.0000000000003bd3/head//3,fad16816/rbd_data.10a16b8b4567.0000000000001731/head//3,77e46816/rbd_data.10966b8b4567.000000000000210d/head//3,cef76816/rbd_data.10b16b8b4567.0000000000001724/head//3,1ad96816/rbd_data.10b46b8b4567.00000000000032a8/head//3,867d6816/rbd_data.10aa6b8b4567.0000000000002162/head//3,8fd6816/rbd_data.108e6b8b4567.00000000000008a3/head//3,1ee6816/rbd_data.10a56b8b4567.0000000000001ba0/head//3,84cf6816/rbd_data.108d6b8b4567.0000000000001e79/head//3,f9e17816/rbd_data.10a56b8b4567.0000000000003054/head//3,99027816/rbd_data.10ae6b8b4567.000000000000038d/head//3,fdb37816/rbd_data.10a16b8b4567.00000000000022f9/head//3,dc057816/rbd_data.108e6b8b4567.0000000000001546/head//3,71267816/rbd_data.10a66b8b4567.0000000000000088/head//3,e6b77816/rbd_data.10916b8b4567.0000000000002eda/head//3,dad77816/rbd_data.10a06b8b4567.0000000000002d05/head//3,68597816/rbd_data.10af6b8b4567.0000000000000cbd/head//3,8cea7816/rbd_data.109b6b8b4567.0000000000002f73/head//3,e3bb7816/rbd_data.109c6b8b4567.0000000000003a6e/head//3,ccbc7816/rbd_data.10976b8b4567.0000000000000144/head//3,9b8d7816/rbd_data.10976b8b4567.0000000000001c6d/head//3,9a9f7816/rbd_data.10976b8b4567.0000000000002740/head//3,33508816/rbd_data.108e6b8b4567.000000000000334a/head//3,aaf18816/rbd_data.10936b8b4567.0000000000000552/head//3,c3748816/rbd_data.10a66b8b4567.0000000000001f76/head//3,5be58816/rbd_data.10b46b8b4567.000000000000033c/head//3,1f768816/rbd_data.10b26b8b4567.00000000000005f8/head//3,496b8816/rbd_data.109b6b8b4567.000000000000196f/head//3,e0ee8816/rbd_data.10b16b8b4567.00000000000008f8/head//3,9dbf8816/rbd_data.10916b8b4567.00000000000020ef/head//3,5b519816/rbd_data.10af6b8b4567.00000000000031c9/head//3,d2349816/rbd_data.109c6b8b4567.0000000000001fc7/head//3,86a59816/rbd_data.108f6b8b4567.00000000000014dd/head//3,c969816/rbd_data.10ad6b8b4567.0000000000003858/head//3,e6b69816/rbd_data.108e6b8b4567.0000000000000445/head//3,55b9816/rbd_data.10a26b8b4567.0000000000000a17/head//3,329b9816/rbd_data.10ac6b8b4567.0000000000002423/head//3,b300a816/rbd_data.10b26b8b4567.00000000000022d7/head//3,4c21a816/rbd_data.10ab6b8b4567.000000000000197e/head//3,7e42a816/rbd_data.10a06b8b4567.0000000000000e2d/head//3,5de2a816/rbd_data.10a96b8b4567.0000000000001516/head//3,5694a816/rbd_data.109d6b8b4567.000000000000064e/head//3,3876a816/rbd_data.10a66b8b4567.00000000000017e1/head//3,7379a816/rbd_data.10906b8b4567.00000000000036cb/head//3,11cba816/rbd_data.10986b8b4567.00000000000037af/head//3,75dda816/rbd_data.10ab6b8b4567.000000000000237c/head//3,801fa816/rbd_data.10ab6b8b4567.000000000000020f/head//3,d2a2b816/rbd_data.108f6b8b4567.0000000000000f59/head//3,b3a4b816/rbd_data.10a76b8b4567.0000000000000485/head//3,70fab816/rbd_data.10996b8b4567.000000000000181f/head//3,fabb816/rbd_data.109a6b8b4567.00000000000031b3/head//3,159eb816/rbd_data.108d6b8b4567.0000000000001393/head//3,9dceb816/rbd_data.10986b8b4567.000000000000115f/head//3,aa60c816/rbd_data.10b26b8b4567.00000000000010e2/head//3,6271c816/rbd_data.10b26b8b4567.000000000000165e/head//3,54b7c816/rbd_data.10ad6b8b4567.0000000000001020/head//3,d499c816/rbd_data.109f6b8b4567.0000000000002cd5/head//3,7e00d816/rbd_data.10936b8b4567.0000000000002946/head//3,172d816/rbd_data.10b16b8b4567.00000000000005a9/head//3,4d82d816/rbd_data.108e6b8b4567.0000000000002bfd/head//3,9fb3d816/rbd_data.10936b8b4567.0000000000001805/head//3,8fcfd816/rbd_data.109d6b8b4567.00000000000011b9/head//3,7d33e816/rbd_data.10986b8b4567.0000000000001672/head//3]
2014-06-12 14:51:08.142279 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:08.142290 7f9d60bd2700 10 stat 3.16_head/86495816/rbd_data.109b6b8b4567.0000000000001b44/head//3
2014-06-12 14:51:08.142383 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 86495816/rbd_data.109b6b8b4567.0000000000001b44/head//3
2014-06-12 14:51:08.158674 7f9d60bd2700 10 getattrs 3.16_head/86495816/rbd_data.109b6b8b4567.0000000000001b44/head//3 = 0
2014-06-12 14:51:08.158682 7f9d60bd2700 10 stat 3.16_head/90da5816/rbd_data.10aa6b8b4567.0000000000003bd3/head//3
2014-06-12 14:51:08.158775 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 90da5816/rbd_data.10aa6b8b4567.0000000000003bd3/head//3
2014-06-12 14:51:08.185467 7f9d60bd2700 10 getattrs 3.16_head/90da5816/rbd_data.10aa6b8b4567.0000000000003bd3/head//3 = 0
2014-06-12 14:51:08.185476 7f9d60bd2700 10 stat 3.16_head/fad16816/rbd_data.10a16b8b4567.0000000000001731/head//3
2014-06-12 14:51:08.185579 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid fad16816/rbd_data.10a16b8b4567.0000000000001731/head//3
2014-06-12 14:51:08.209466 7f9d60bd2700 10 getattrs 3.16_head/fad16816/rbd_data.10a16b8b4567.0000000000001731/head//3 = 0
2014-06-12 14:51:08.209477 7f9d60bd2700 10 stat 3.16_head/77e46816/rbd_data.10966b8b4567.000000000000210d/head//3
2014-06-12 14:51:08.209586 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 77e46816/rbd_data.10966b8b4567.000000000000210d/head//3
2014-06-12 14:51:08.226147 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:08.234634 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.240421 7f9d60bd2700 10 getattrs 3.16_head/77e46816/rbd_data.10966b8b4567.000000000000210d/head//3 = 0
2014-06-12 14:51:08.240431 7f9d60bd2700 10 stat 3.16_head/cef76816/rbd_data.10b16b8b4567.0000000000001724/head//3
2014-06-12 14:51:08.240551 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid cef76816/rbd_data.10b16b8b4567.0000000000001724/head//3
2014-06-12 14:51:08.263904 7f9d60bd2700 10 getattrs 3.16_head/cef76816/rbd_data.10b16b8b4567.0000000000001724/head//3 = 0
2014-06-12 14:51:08.263929 7f9d60bd2700 10 stat 3.16_head/1ad96816/rbd_data.10b46b8b4567.00000000000032a8/head//3
2014-06-12 14:51:08.264087 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 1ad96816/rbd_data.10b46b8b4567.00000000000032a8/head//3
2014-06-12 14:51:08.286218 7f9d60bd2700 10 getattrs 3.16_head/1ad96816/rbd_data.10b46b8b4567.00000000000032a8/head//3 = 0
2014-06-12 14:51:08.286225 7f9d60bd2700 10 stat 3.16_head/867d6816/rbd_data.10aa6b8b4567.0000000000002162/head//3
2014-06-12 14:51:08.286319 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 867d6816/rbd_data.10aa6b8b4567.0000000000002162/head//3
2014-06-12 14:51:08.315583 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:08.316742 7f9d60bd2700 10 getattrs 3.16_head/867d6816/rbd_data.10aa6b8b4567.0000000000002162/head//3 = 0
2014-06-12 14:51:08.316752 7f9d60bd2700 10 stat 3.16_head/8fd6816/rbd_data.108e6b8b4567.00000000000008a3/head//3
2014-06-12 14:51:08.316859 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 8fd6816/rbd_data.108e6b8b4567.00000000000008a3/head//3
2014-06-12 14:51:08.325069 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.365038 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:08.369765 7f9d60bd2700 10 getattrs 3.16_head/8fd6816/rbd_data.108e6b8b4567.00000000000008a3/head//3 = 0
2014-06-12 14:51:08.369794 7f9d60bd2700 10 stat 3.16_head/1ee6816/rbd_data.10a56b8b4567.0000000000001ba0/head//3
2014-06-12 14:51:08.369859 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.369998 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 1ee6816/rbd_data.10a56b8b4567.0000000000001ba0/head//3
2014-06-12 14:51:08.380593 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:08.383720 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:08.383739 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:08.384486 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:08.384542 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:08.384571 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:08.384609 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:08.384660 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:08.384717 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:08.384746 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:08.384777 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:08.384805 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:08.384829 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:08.384859 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:08.384886 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:08.384910 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:08.384933 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:08.384958 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:08.384985 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:08.385009 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:08.385032 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:08.385056 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:08.385080 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:08.385112 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:08.385140 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:08.390150 7f9d60bd2700 10 getattrs 3.16_head/1ee6816/rbd_data.10a56b8b4567.0000000000001ba0/head//3 = 0
2014-06-12 14:51:08.390179 7f9d60bd2700 10 stat 3.16_head/84cf6816/rbd_data.108d6b8b4567.0000000000001e79/head//3
2014-06-12 14:51:08.390374 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 84cf6816/rbd_data.108d6b8b4567.0000000000001e79/head//3
2014-06-12 14:51:08.390524 7f9d60bd2700 10 getattrs 3.16_head/84cf6816/rbd_data.108d6b8b4567.0000000000001e79/head//3 = 0
2014-06-12 14:51:08.390535 7f9d60bd2700 10 stat 3.16_head/f9e17816/rbd_data.10a56b8b4567.0000000000003054/head//3
2014-06-12 14:51:08.390699 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid f9e17816/rbd_data.10a56b8b4567.0000000000003054/head//3
2014-06-12 14:51:08.419719 7f9d60bd2700 10 getattrs 3.16_head/f9e17816/rbd_data.10a56b8b4567.0000000000003054/head//3 = 0
2014-06-12 14:51:08.419729 7f9d60bd2700 10 stat 3.16_head/99027816/rbd_data.10ae6b8b4567.000000000000038d/head//3
2014-06-12 14:51:08.419828 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 99027816/rbd_data.10ae6b8b4567.000000000000038d/head//3
2014-06-12 14:51:08.469093 7f9d60bd2700 10 getattrs 3.16_head/99027816/rbd_data.10ae6b8b4567.000000000000038d/head//3 = 0
2014-06-12 14:51:08.469103 7f9d60bd2700 10 stat 3.16_head/fdb37816/rbd_data.10a16b8b4567.00000000000022f9/head//3
2014-06-12 14:51:08.469202 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid fdb37816/rbd_data.10a16b8b4567.00000000000022f9/head//3
2014-06-12 14:51:08.504004 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:08.505581 7f9d60bd2700 10 getattrs 3.16_head/fdb37816/rbd_data.10a16b8b4567.00000000000022f9/head//3 = 0
2014-06-12 14:51:08.505590 7f9d60bd2700 10 stat 3.16_head/dc057816/rbd_data.108e6b8b4567.0000000000001546/head//3
2014-06-12 14:51:08.505735 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid dc057816/rbd_data.108e6b8b4567.0000000000001546/head//3
2014-06-12 14:51:08.511871 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:08.515127 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:08.515195 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:08.521702 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.522547 7f9d60bd2700 10 getattrs 3.16_head/dc057816/rbd_data.108e6b8b4567.0000000000001546/head//3 = 0
2014-06-12 14:51:08.522558 7f9d60bd2700 10 stat 3.16_head/71267816/rbd_data.10a66b8b4567.0000000000000088/head//3
2014-06-12 14:51:08.522689 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 71267816/rbd_data.10a66b8b4567.0000000000000088/head//3
2014-06-12 14:51:08.530991 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.561523 7f9d60bd2700 10 getattrs 3.16_head/71267816/rbd_data.10a66b8b4567.0000000000000088/head//3 = 0
2014-06-12 14:51:08.561541 7f9d60bd2700 10 stat 3.16_head/e6b77816/rbd_data.10916b8b4567.0000000000002eda/head//3
2014-06-12 14:51:08.561713 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid e6b77816/rbd_data.10916b8b4567.0000000000002eda/head//3
2014-06-12 14:51:08.576192 7f9d60bd2700 10 getattrs 3.16_head/e6b77816/rbd_data.10916b8b4567.0000000000002eda/head//3 = 0
2014-06-12 14:51:08.576206 7f9d60bd2700 10 stat 3.16_head/dad77816/rbd_data.10a06b8b4567.0000000000002d05/head//3
2014-06-12 14:51:08.576335 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid dad77816/rbd_data.10a06b8b4567.0000000000002d05/head//3
2014-06-12 14:51:08.589792 7f9d60bd2700 10 getattrs 3.16_head/dad77816/rbd_data.10a06b8b4567.0000000000002d05/head//3 = 0
2014-06-12 14:51:08.589801 7f9d60bd2700 10 stat 3.16_head/68597816/rbd_data.10af6b8b4567.0000000000000cbd/head//3
2014-06-12 14:51:08.589905 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 68597816/rbd_data.10af6b8b4567.0000000000000cbd/head//3
2014-06-12 14:51:08.623188 7f9d60bd2700 10 getattrs 3.16_head/68597816/rbd_data.10af6b8b4567.0000000000000cbd/head//3 = 0
2014-06-12 14:51:08.623201 7f9d60bd2700 10 stat 3.16_head/8cea7816/rbd_data.109b6b8b4567.0000000000002f73/head//3
2014-06-12 14:51:08.623297 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 8cea7816/rbd_data.109b6b8b4567.0000000000002f73/head//3
2014-06-12 14:51:08.641276 7f9d60bd2700 10 getattrs 3.16_head/8cea7816/rbd_data.109b6b8b4567.0000000000002f73/head//3 = 0
2014-06-12 14:51:08.641284 7f9d60bd2700 10 stat 3.16_head/e3bb7816/rbd_data.109c6b8b4567.0000000000003a6e/head//3
2014-06-12 14:51:08.641381 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid e3bb7816/rbd_data.109c6b8b4567.0000000000003a6e/head//3
2014-06-12 14:51:08.650204 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:08.656821 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:08.656827 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:08.656829 7f9d74bfa700 20 osd.2 173 _dispatch 0x90f0700 pg_stats_ack(1 pgs tid 4867) v1
2014-06-12 14:51:08.656833 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:51:08.656840 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:08.656841 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:08.660364 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.667268 7f9d60bd2700 10 getattrs 3.16_head/e3bb7816/rbd_data.109c6b8b4567.0000000000003a6e/head//3 = 0
2014-06-12 14:51:08.667277 7f9d60bd2700 10 stat 3.16_head/ccbc7816/rbd_data.10976b8b4567.0000000000000144/head//3
2014-06-12 14:51:08.667377 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid ccbc7816/rbd_data.10976b8b4567.0000000000000144/head//3
2014-06-12 14:51:08.671032 7f9d58d5b700  5 -- op tracker -- , seq: 748564, time: 2014-06-12 14:51:08.670965, event: header_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.671042 7f9d58d5b700  5 -- op tracker -- , seq: 748564, time: 2014-06-12 14:51:08.670966, event: throttled, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.671048 7f9d58d5b700  5 -- op tracker -- , seq: 748564, time: 2014-06-12 14:51:08.671004, event: all_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.671053 7f9d58d5b700  5 -- op tracker -- , seq: 748564, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.671057 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:08.671062 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:51:08.671068 7f9d58d5b700 15 osd.2 173 enqueue_op 0x31d60e00 prio 127 cost 10717 latency 0.000103 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:08.671096 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:08.699406 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.702499 7f9d60bd2700 10 getattrs 3.16_head/ccbc7816/rbd_data.10976b8b4567.0000000000000144/head//3 = 0
2014-06-12 14:51:08.702506 7f9d60bd2700 10 stat 3.16_head/9b8d7816/rbd_data.10976b8b4567.0000000000001c6d/head//3
2014-06-12 14:51:08.702607 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 9b8d7816/rbd_data.10976b8b4567.0000000000001c6d/head//3
2014-06-12 14:51:08.726836 7f9d60bd2700 10 getattrs 3.16_head/9b8d7816/rbd_data.10976b8b4567.0000000000001c6d/head//3 = 0
2014-06-12 14:51:08.726850 7f9d60bd2700 10 stat 3.16_head/9a9f7816/rbd_data.10976b8b4567.0000000000002740/head//3
2014-06-12 14:51:08.726964 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 9a9f7816/rbd_data.10976b8b4567.0000000000002740/head//3
2014-06-12 14:51:08.741026 7f9d60bd2700 10 getattrs 3.16_head/9a9f7816/rbd_data.10976b8b4567.0000000000002740/head//3 = 0
2014-06-12 14:51:08.741039 7f9d60bd2700 10 stat 3.16_head/33508816/rbd_data.108e6b8b4567.000000000000334a/head//3
2014-06-12 14:51:08.741164 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 33508816/rbd_data.108e6b8b4567.000000000000334a/head//3
2014-06-12 14:51:08.756258 7f9d60bd2700 10 getattrs 3.16_head/33508816/rbd_data.108e6b8b4567.000000000000334a/head//3 = 0
2014-06-12 14:51:08.756269 7f9d60bd2700 10 stat 3.16_head/aaf18816/rbd_data.10936b8b4567.0000000000000552/head//3
2014-06-12 14:51:08.756393 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid aaf18816/rbd_data.10936b8b4567.0000000000000552/head//3
2014-06-12 14:51:08.766749 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.795068 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:08.800655 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.806600 7f9d60bd2700 10 getattrs 3.16_head/aaf18816/rbd_data.10936b8b4567.0000000000000552/head//3 = 0
2014-06-12 14:51:08.806615 7f9d60bd2700 10 collection_getattrs 3.16_head
2014-06-12 14:51:08.806674 7f9d60bd2700 10 collection_getattrs 3.16_head = 0
2014-06-12 14:51:08.806685 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:08.806726 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:08.806732 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:08.806740 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:08.806742 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:08.806792 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x31d60e00 prio 127 cost 10717 latency 0.135826 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]
2014-06-12 14:51:08.806848 7f9d6b3e7700  5 -- op tracker -- , seq: 748564, time: 2014-06-12 14:51:08.806840, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.806866 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] handle_message: 0x31d60e00
2014-06-12 14:51:08.806882 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:08.806900 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:08.806917 7f9d6b3e7700  5 -- op tracker -- , seq: 748564, time: 2014-06-12 14:51:08.806912, event: started, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.806925 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  got 22 scrub map
2014-06-12 14:51:08.807029 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] map version is 173'2396
2014-06-12 14:51:08.807050 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x31d60e00 finish
2014-06-12 14:51:08.807066 7f9d6b3e7700  5 -- op tracker -- , seq: 748564, time: 2014-06-12 14:51:08.807060, event: done, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:08.807090 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:08.807095 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:08.807088 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:08.807101 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:08.807124 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:08.807136 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:08.807180 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:08.807196 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:08.807203 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub replica 22 has 25 items
2014-06-12 14:51:08.807227 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 86495816/rbd_data.109b6b8b4567.0000000000001b44/head//3, auth == maps.end()
2014-06-12 14:51:08.807238 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 86495816/rbd_data.109b6b8b4567.0000000000001b44/head//3
2014-06-12 14:51:08.807248 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 90da5816/rbd_data.10aa6b8b4567.0000000000003bd3/head//3, auth == maps.end()
2014-06-12 14:51:08.807255 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 90da5816/rbd_data.10aa6b8b4567.0000000000003bd3/head//3
2014-06-12 14:51:08.807263 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fad16816/rbd_data.10a16b8b4567.0000000000001731/head//3, auth == maps.end()
2014-06-12 14:51:08.807270 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj fad16816/rbd_data.10a16b8b4567.0000000000001731/head//3
2014-06-12 14:51:08.807278 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 77e46816/rbd_data.10966b8b4567.000000000000210d/head//3, auth == maps.end()
2014-06-12 14:51:08.807284 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 77e46816/rbd_data.10966b8b4567.000000000000210d/head//3
2014-06-12 14:51:08.807292 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cef76816/rbd_data.10b16b8b4567.0000000000001724/head//3, auth == maps.end()
2014-06-12 14:51:08.807299 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj cef76816/rbd_data.10b16b8b4567.0000000000001724/head//3
2014-06-12 14:51:08.807306 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1ad96816/rbd_data.10b46b8b4567.00000000000032a8/head//3, auth == maps.end()
2014-06-12 14:51:08.807313 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 1ad96816/rbd_data.10b46b8b4567.00000000000032a8/head//3
2014-06-12 14:51:08.807320 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 867d6816/rbd_data.10aa6b8b4567.0000000000002162/head//3, auth == maps.end()
2014-06-12 14:51:08.807327 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 867d6816/rbd_data.10aa6b8b4567.0000000000002162/head//3
2014-06-12 14:51:08.807335 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8fd6816/rbd_data.108e6b8b4567.00000000000008a3/head//3, auth == maps.end()
2014-06-12 14:51:08.807341 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 8fd6816/rbd_data.108e6b8b4567.00000000000008a3/head//3
2014-06-12 14:51:08.807353 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1ee6816/rbd_data.10a56b8b4567.0000000000001ba0/head//3, auth == maps.end()
2014-06-12 14:51:08.807362 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 1ee6816/rbd_data.10a56b8b4567.0000000000001ba0/head//3
2014-06-12 14:51:08.807371 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 84cf6816/rbd_data.108d6b8b4567.0000000000001e79/head//3, auth == maps.end()
2014-06-12 14:51:08.807379 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 84cf6816/rbd_data.108d6b8b4567.0000000000001e79/head//3
2014-06-12 14:51:08.807387 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f9e17816/rbd_data.10a56b8b4567.0000000000003054/head//3, auth == maps.end()
2014-06-12 14:51:08.807395 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj f9e17816/rbd_data.10a56b8b4567.0000000000003054/head//3
2014-06-12 14:51:08.807403 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 99027816/rbd_data.10ae6b8b4567.000000000000038d/head//3, auth == maps.end()
2014-06-12 14:51:08.807411 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 99027816/rbd_data.10ae6b8b4567.000000000000038d/head//3
2014-06-12 14:51:08.807419 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fdb37816/rbd_data.10a16b8b4567.00000000000022f9/head//3, auth == maps.end()
2014-06-12 14:51:08.807426 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj fdb37816/rbd_data.10a16b8b4567.00000000000022f9/head//3
2014-06-12 14:51:08.807434 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dc057816/rbd_data.108e6b8b4567.0000000000001546/head//3, auth == maps.end()
2014-06-12 14:51:08.807442 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj dc057816/rbd_data.108e6b8b4567.0000000000001546/head//3
2014-06-12 14:51:08.807450 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 71267816/rbd_data.10a66b8b4567.0000000000000088/head//3, auth == maps.end()
2014-06-12 14:51:08.807458 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 71267816/rbd_data.10a66b8b4567.0000000000000088/head//3
2014-06-12 14:51:08.807466 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e6b77816/rbd_data.10916b8b4567.0000000000002eda/head//3, auth == maps.end()
2014-06-12 14:51:08.807474 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj e6b77816/rbd_data.10916b8b4567.0000000000002eda/head//3
2014-06-12 14:51:08.807481 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dad77816/rbd_data.10a06b8b4567.0000000000002d05/head//3, auth == maps.end()
2014-06-12 14:51:08.807490 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj dad77816/rbd_data.10a06b8b4567.0000000000002d05/head//3
2014-06-12 14:51:08.807499 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 68597816/rbd_data.10af6b8b4567.0000000000000cbd/head//3, auth == maps.end()
2014-06-12 14:51:08.807507 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 68597816/rbd_data.10af6b8b4567.0000000000000cbd/head//3
2014-06-12 14:51:08.807516 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8cea7816/rbd_data.109b6b8b4567.0000000000002f73/head//3, auth == maps.end()
2014-06-12 14:51:08.807523 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 8cea7816/rbd_data.109b6b8b4567.0000000000002f73/head//3
2014-06-12 14:51:08.807530 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e3bb7816/rbd_data.109c6b8b4567.0000000000003a6e/head//3, auth == maps.end()
2014-06-12 14:51:08.807538 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj e3bb7816/rbd_data.109c6b8b4567.0000000000003a6e/head//3
2014-06-12 14:51:08.807546 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ccbc7816/rbd_data.10976b8b4567.0000000000000144/head//3, auth == maps.end()
2014-06-12 14:51:08.807554 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj ccbc7816/rbd_data.10976b8b4567.0000000000000144/head//3
2014-06-12 14:51:08.807562 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9b8d7816/rbd_data.10976b8b4567.0000000000001c6d/head//3, auth == maps.end()
2014-06-12 14:51:08.807570 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 9b8d7816/rbd_data.10976b8b4567.0000000000001c6d/head//3
2014-06-12 14:51:08.807577 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9a9f7816/rbd_data.10976b8b4567.0000000000002740/head//3, auth == maps.end()
2014-06-12 14:51:08.807584 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 9a9f7816/rbd_data.10976b8b4567.0000000000002740/head//3
2014-06-12 14:51:08.807592 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 33508816/rbd_data.108e6b8b4567.000000000000334a/head//3, auth == maps.end()
2014-06-12 14:51:08.807598 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 33508816/rbd_data.108e6b8b4567.000000000000334a/head//3
2014-06-12 14:51:08.807606 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aaf18816/rbd_data.10936b8b4567.0000000000000552/head//3, auth == maps.end()
2014-06-12 14:51:08.807613 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj aaf18816/rbd_data.10936b8b4567.0000000000000552/head//3
2014-06-12 14:51:08.807621 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] 
2014-06-12 14:51:08.807627 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub
2014-06-12 14:51:08.807637 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  aaf18816/rbd_data.10936b8b4567.0000000000000552/head//3 aaf18816/rbd_data.10936b8b4567.0000000000000552/head//3(173'2144 client.4284.0:123205 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2144)
2014-06-12 14:51:08.807649 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  33508816/rbd_data.108e6b8b4567.000000000000334a/head//3 33508816/rbd_data.108e6b8b4567.000000000000334a/head//3(173'1780 client.4279.0:101828 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1780)
2014-06-12 14:51:08.807660 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  9a9f7816/rbd_data.10976b8b4567.0000000000002740/head//3 9a9f7816/rbd_data.10976b8b4567.0000000000002740/head//3(173'1483 client.4288.0:78297 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1483)
2014-06-12 14:51:08.807671 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  9b8d7816/rbd_data.10976b8b4567.0000000000001c6d/head//3 9b8d7816/rbd_data.10976b8b4567.0000000000001c6d/head//3(173'1191 client.4288.0:59280 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1191)
2014-06-12 14:51:08.807681 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  ccbc7816/rbd_data.10976b8b4567.0000000000000144/head//3 ccbc7816/rbd_data.10976b8b4567.0000000000000144/head//3(173'2044 client.4288.0:116604 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2044)
2014-06-12 14:51:08.807691 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  e3bb7816/rbd_data.109c6b8b4567.0000000000003a6e/head//3 e3bb7816/rbd_data.109c6b8b4567.0000000000003a6e/head//3(173'1978 client.4293.0:115353 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1978)
2014-06-12 14:51:08.807701 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  8cea7816/rbd_data.109b6b8b4567.0000000000002f73/head//3 8cea7816/rbd_data.109b6b8b4567.0000000000002f73/head//3(173'1769 client.4292.0:93232 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1769)
2014-06-12 14:51:08.807712 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  68597816/rbd_data.10af6b8b4567.0000000000000cbd/head//3 68597816/rbd_data.10af6b8b4567.0000000000000cbd/head//3(173'2350 client.4312.0:141439 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2350)
2014-06-12 14:51:08.807723 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  dad77816/rbd_data.10a06b8b4567.0000000000002d05/head//3 dad77816/rbd_data.10a06b8b4567.0000000000002d05/head//3(173'1548 client.4297.0:89613 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1548)
2014-06-12 14:51:08.807734 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  e6b77816/rbd_data.10916b8b4567.0000000000002eda/head//3 e6b77816/rbd_data.10916b8b4567.0000000000002eda/head//3(173'1710 client.4282.0:94845 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1710)
2014-06-12 14:51:08.807744 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  71267816/rbd_data.10a66b8b4567.0000000000000088/head//3 71267816/rbd_data.10a66b8b4567.0000000000000088/head//3(173'2014 client.4303.0:119181 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2014)
2014-06-12 14:51:08.807754 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  dc057816/rbd_data.108e6b8b4567.0000000000001546/head//3 dc057816/rbd_data.108e6b8b4567.0000000000001546/head//3(173'749 client.4279.0:48342 wrlock_by=unknown.0.0:0 dirty s 4194304 uv749)
2014-06-12 14:51:08.807764 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  fdb37816/rbd_data.10a16b8b4567.00000000000022f9/head//3 fdb37816/rbd_data.10a16b8b4567.00000000000022f9/head//3(173'1229 client.4298.0:73135 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1229)
2014-06-12 14:51:08.807774 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  99027816/rbd_data.10ae6b8b4567.000000000000038d/head//3 99027816/rbd_data.10ae6b8b4567.000000000000038d/head//3(173'2117 client.4311.0:120085 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2117)
2014-06-12 14:51:08.807784 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  f9e17816/rbd_data.10a56b8b4567.0000000000003054/head//3 f9e17816/rbd_data.10a56b8b4567.0000000000003054/head//3(173'1739 client.4302.0:97989 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1739)
2014-06-12 14:51:08.807795 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  84cf6816/rbd_data.108d6b8b4567.0000000000001e79/head//3 84cf6816/rbd_data.108d6b8b4567.0000000000001e79/head//3(173'1076 client.4278.0:65570 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1076)
2014-06-12 14:51:08.807805 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  1ee6816/rbd_data.10a56b8b4567.0000000000001ba0/head//3 1ee6816/rbd_data.10a56b8b4567.0000000000001ba0/head//3(173'996 client.4302.0:59951 wrlock_by=unknown.0.0:0 dirty s 4194304 uv996)
2014-06-12 14:51:08.807814 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  8fd6816/rbd_data.108e6b8b4567.00000000000008a3/head//3 8fd6816/rbd_data.108e6b8b4567.00000000000008a3/head//3(173'2252 client.4279.0:130700 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2252)
2014-06-12 14:51:08.807824 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  867d6816/rbd_data.10aa6b8b4567.0000000000002162/head//3 867d6816/rbd_data.10aa6b8b4567.0000000000002162/head//3(173'1208 client.4307.0:72040 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1208)
2014-06-12 14:51:08.807835 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  1ad96816/rbd_data.10b46b8b4567.00000000000032a8/head//3 1ad96816/rbd_data.10b46b8b4567.00000000000032a8/head//3(173'1898 client.4317.0:100978 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1898)
2014-06-12 14:51:08.807845 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  cef76816/rbd_data.10b16b8b4567.0000000000001724/head//3 cef76816/rbd_data.10b16b8b4567.0000000000001724/head//3(173'837 client.4314.0:51625 wrlock_by=unknown.0.0:0 dirty s 4194304 uv837)
2014-06-12 14:51:08.807855 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  77e46816/rbd_data.10966b8b4567.000000000000210d/head//3 77e46816/rbd_data.10966b8b4567.000000000000210d/head//3(173'1174 client.4287.0:66512 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1174)
2014-06-12 14:51:08.807865 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  fad16816/rbd_data.10a16b8b4567.0000000000001731/head//3 fad16816/rbd_data.10a16b8b4567.0000000000001731/head//3(173'829 client.4298.0:52303 wrlock_by=unknown.0.0:0 dirty s 4194304 uv829)
2014-06-12 14:51:08.807875 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  90da5816/rbd_data.10aa6b8b4567.0000000000003bd3/head//3 90da5816/rbd_data.10aa6b8b4567.0000000000003bd3/head//3(173'1983 client.4307.0:119890 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1983)
2014-06-12 14:51:08.807885 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  86495816/rbd_data.109b6b8b4567.0000000000001b44/head//3 86495816/rbd_data.109b6b8b4567.0000000000001b44/head//3(173'1118 client.4292.0:57100 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1118)
2014-06-12 14:51:08.807893 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:08.807902 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:08.807908 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:08.807909 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:08.807910 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:08.807940 7f9d60bd2700 10 collection_list_partial 3.16_head start:c3748816//0//-1 is_max:0
2014-06-12 14:51:08.808143 7f9d60bd2700 20 objects: [5be58816/rbd_data.10b46b8b4567.000000000000033c/head//3,1f768816/rbd_data.10b26b8b4567.00000000000005f8/head//3,496b8816/rbd_data.109b6b8b4567.000000000000196f/head//3,e0ee8816/rbd_data.10b16b8b4567.00000000000008f8/head//3,9dbf8816/rbd_data.10916b8b4567.00000000000020ef/head//3,5b519816/rbd_data.10af6b8b4567.00000000000031c9/head//3,d2349816/rbd_data.109c6b8b4567.0000000000001fc7/head//3,86a59816/rbd_data.108f6b8b4567.00000000000014dd/head//3,c969816/rbd_data.10ad6b8b4567.0000000000003858/head//3,e6b69816/rbd_data.108e6b8b4567.0000000000000445/head//3,55b9816/rbd_data.10a26b8b4567.0000000000000a17/head//3,329b9816/rbd_data.10ac6b8b4567.0000000000002423/head//3,b300a816/rbd_data.10b26b8b4567.00000000000022d7/head//3,4c21a816/rbd_data.10ab6b8b4567.000000000000197e/head//3,7e42a816/rbd_data.10a06b8b4567.0000000000000e2d/head//3,5de2a816/rbd_data.10a96b8b4567.0000000000001516/head//3,5694a816/rbd_data.109d6b8b4567.000000000000064e/head//3,3876a816/rbd_data.10a66b8b4567.00000000000017e1/head//3,7379a816/rbd_data.10906b8b4567.00000000000036cb/head//3,11cba816/rbd_data.10986b8b4567.00000000000037af/head//3,75dda816/rbd_data.10ab6b8b4567.000000000000237c/head//3,801fa816/rbd_data.10ab6b8b4567.000000000000020f/head//3,d2a2b816/rbd_data.108f6b8b4567.0000000000000f59/head//3,b3a4b816/rbd_data.10a76b8b4567.0000000000000485/head//3,70fab816/rbd_data.10996b8b4567.000000000000181f/head//3]
2014-06-12 14:51:08.808525 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  requesting scrubmap from osd.22
2014-06-12 14:51:08.808545 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:08.808564 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:08.808572 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:08.808577 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:08.808581 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:08.808586 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_map_chunk [c3748816//0//-1,fabb816//0//-1)
2014-06-12 14:51:08.808591 7f9d60bd2700 10 collection_list_partial 3.16_head start:c3748816//0//-1 is_max:0
2014-06-12 14:51:08.808893 7f9d60bd2700 20 objects: [5be58816/rbd_data.10b46b8b4567.000000000000033c/head//3,1f768816/rbd_data.10b26b8b4567.00000000000005f8/head//3,496b8816/rbd_data.109b6b8b4567.000000000000196f/head//3,e0ee8816/rbd_data.10b16b8b4567.00000000000008f8/head//3,9dbf8816/rbd_data.10916b8b4567.00000000000020ef/head//3,5b519816/rbd_data.10af6b8b4567.00000000000031c9/head//3,d2349816/rbd_data.109c6b8b4567.0000000000001fc7/head//3,86a59816/rbd_data.108f6b8b4567.00000000000014dd/head//3,c969816/rbd_data.10ad6b8b4567.0000000000003858/head//3,e6b69816/rbd_data.108e6b8b4567.0000000000000445/head//3,55b9816/rbd_data.10a26b8b4567.0000000000000a17/head//3,329b9816/rbd_data.10ac6b8b4567.0000000000002423/head//3,b300a816/rbd_data.10b26b8b4567.00000000000022d7/head//3,4c21a816/rbd_data.10ab6b8b4567.000000000000197e/head//3,7e42a816/rbd_data.10a06b8b4567.0000000000000e2d/head//3,5de2a816/rbd_data.10a96b8b4567.0000000000001516/head//3,5694a816/rbd_data.109d6b8b4567.000000000000064e/head//3,3876a816/rbd_data.10a66b8b4567.00000000000017e1/head//3,7379a816/rbd_data.10906b8b4567.00000000000036cb/head//3,11cba816/rbd_data.10986b8b4567.00000000000037af/head//3,75dda816/rbd_data.10ab6b8b4567.000000000000237c/head//3,801fa816/rbd_data.10ab6b8b4567.000000000000020f/head//3,d2a2b816/rbd_data.108f6b8b4567.0000000000000f59/head//3,b3a4b816/rbd_data.10a76b8b4567.0000000000000485/head//3,70fab816/rbd_data.10996b8b4567.000000000000181f/head//3,fabb816/rbd_data.109a6b8b4567.00000000000031b3/head//3,159eb816/rbd_data.108d6b8b4567.0000000000001393/head//3,9dceb816/rbd_data.10986b8b4567.000000000000115f/head//3,aa60c816/rbd_data.10b26b8b4567.00000000000010e2/head//3,6271c816/rbd_data.10b26b8b4567.000000000000165e/head//3,54b7c816/rbd_data.10ad6b8b4567.0000000000001020/head//3,d499c816/rbd_data.109f6b8b4567.0000000000002cd5/head//3,7e00d816/rbd_data.10936b8b4567.0000000000002946/head//3,172d816/rbd_data.10b16b8b4567.00000000000005a9/head//3,4d82d816/rbd_data.108e6b8b4567.0000000000002bfd/head//3,9fb3d816/rbd_data.10936b8b4567.0000000000001805/head//3,8fcfd816/rbd_data.109d6b8b4567.00000000000011b9/head//3,7d33e816/rbd_data.10986b8b4567.0000000000001672/head//3,8093e816/rbd_data.10946b8b4567.0000000000001f6a/head//3,4e04e816/rbd_data.109e6b8b4567.0000000000002350/head//3,43f4e816/rbd_data.10b46b8b4567.00000000000034a3/head//3,5d8ae816/rbd_data.10ae6b8b4567.0000000000001fe5/head//3,61dce816/rbd_data.109e6b8b4567.0000000000001cf4/head//3,b48ee816/rbd_data.10a76b8b4567.00000000000039df/head//3,39afe816/rbd_data.109e6b8b4567.00000000000017b8/head//3,f40f816/rbd_data.10ac6b8b4567.0000000000002c72/head//3,7453f816/rbd_data.109e6b8b4567.000000000000364e/head//3,eb45f816/rbd_data.109f6b8b4567.0000000000000f57/head//3,1275f816/rbd_data.10b16b8b4567.000000000000399e/head//3,3ee5f816/rbd_data.109e6b8b4567.0000000000002130/head//3,390bf816/rbd_data.108d6b8b4567.0000000000001332/head//3,4aedf816/rbd_data.10986b8b4567.0000000000003243/head//3,c32ff816/rbd_data.10906b8b4567.0000000000002107/head//3,e56ff816/rbd_data.109a6b8b4567.000000000000371e/head//3]
2014-06-12 14:51:08.808948 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:08.808960 7f9d60bd2700 10 stat 3.16_head/5be58816/rbd_data.10b46b8b4567.000000000000033c/head//3
2014-06-12 14:51:08.809058 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 5be58816/rbd_data.10b46b8b4567.000000000000033c/head//3
2014-06-12 14:51:08.852382 7f9d60bd2700 10 getattrs 3.16_head/5be58816/rbd_data.10b46b8b4567.000000000000033c/head//3 = 0
2014-06-12 14:51:08.852398 7f9d60bd2700 10 stat 3.16_head/1f768816/rbd_data.10b26b8b4567.00000000000005f8/head//3
2014-06-12 14:51:08.852496 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 1f768816/rbd_data.10b26b8b4567.00000000000005f8/head//3
2014-06-12 14:51:08.853777 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:08.864142 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.867432 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:08.869541 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:08.869590 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.59 >= max 0.5 = no, load too high
2014-06-12 14:51:08.869607 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:51:08.869623 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:08.869610 7f9d84175700 10 osd.2 173 sched_scrub 3.16 high load at 2014-06-11 13:02:32.499677: 92916.4 < max (604800 seconds)
2014-06-12 14:51:08.869660 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:08.869664 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:08.869666 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:08.880149 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:08.880309 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:08.897231 7f9d60bd2700 10 getattrs 3.16_head/1f768816/rbd_data.10b26b8b4567.00000000000005f8/head//3 = 0
2014-06-12 14:51:08.897248 7f9d60bd2700 10 stat 3.16_head/496b8816/rbd_data.109b6b8b4567.000000000000196f/head//3
2014-06-12 14:51:08.897366 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 496b8816/rbd_data.109b6b8b4567.000000000000196f/head//3
2014-06-12 14:51:08.918451 7f9d60bd2700 10 getattrs 3.16_head/496b8816/rbd_data.109b6b8b4567.000000000000196f/head//3 = 0
2014-06-12 14:51:08.918461 7f9d60bd2700 10 stat 3.16_head/e0ee8816/rbd_data.10b16b8b4567.00000000000008f8/head//3
2014-06-12 14:51:08.918570 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid e0ee8816/rbd_data.10b16b8b4567.00000000000008f8/head//3
2014-06-12 14:51:08.927161 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:08.934634 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.958733 7f9d60bd2700 10 getattrs 3.16_head/e0ee8816/rbd_data.10b16b8b4567.00000000000008f8/head//3 = 0
2014-06-12 14:51:08.958741 7f9d60bd2700 10 stat 3.16_head/9dbf8816/rbd_data.10916b8b4567.00000000000020ef/head//3
2014-06-12 14:51:08.958835 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 9dbf8816/rbd_data.10916b8b4567.00000000000020ef/head//3
2014-06-12 14:51:08.960993 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:08.964748 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:08.987597 7f9d60bd2700 10 getattrs 3.16_head/9dbf8816/rbd_data.10916b8b4567.00000000000020ef/head//3 = 0
2014-06-12 14:51:08.987606 7f9d60bd2700 10 stat 3.16_head/5b519816/rbd_data.10af6b8b4567.00000000000031c9/head//3
2014-06-12 14:51:08.987700 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 5b519816/rbd_data.10af6b8b4567.00000000000031c9/head//3
2014-06-12 14:51:09.014942 7f9d60bd2700 10 getattrs 3.16_head/5b519816/rbd_data.10af6b8b4567.00000000000031c9/head//3 = 0
2014-06-12 14:51:09.014951 7f9d60bd2700 10 stat 3.16_head/d2349816/rbd_data.109c6b8b4567.0000000000001fc7/head//3
2014-06-12 14:51:09.015053 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid d2349816/rbd_data.109c6b8b4567.0000000000001fc7/head//3
2014-06-12 14:51:09.046652 7f9d60bd2700 10 getattrs 3.16_head/d2349816/rbd_data.109c6b8b4567.0000000000001fc7/head//3 = 0
2014-06-12 14:51:09.046665 7f9d60bd2700 10 stat 3.16_head/86a59816/rbd_data.108f6b8b4567.00000000000014dd/head//3
2014-06-12 14:51:09.046783 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 86a59816/rbd_data.108f6b8b4567.00000000000014dd/head//3
2014-06-12 14:51:09.076340 7f9d60bd2700 10 getattrs 3.16_head/86a59816/rbd_data.108f6b8b4567.00000000000014dd/head//3 = 0
2014-06-12 14:51:09.076350 7f9d60bd2700 10 stat 3.16_head/c969816/rbd_data.10ad6b8b4567.0000000000003858/head//3
2014-06-12 14:51:09.076468 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid c969816/rbd_data.10ad6b8b4567.0000000000003858/head//3
2014-06-12 14:51:09.098551 7f9d60bd2700 10 getattrs 3.16_head/c969816/rbd_data.10ad6b8b4567.0000000000003858/head//3 = 0
2014-06-12 14:51:09.098568 7f9d60bd2700 10 stat 3.16_head/e6b69816/rbd_data.108e6b8b4567.0000000000000445/head//3
2014-06-12 14:51:09.098720 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid e6b69816/rbd_data.108e6b8b4567.0000000000000445/head//3
2014-06-12 14:51:09.120051 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:09.131619 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:09.141094 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:09.142325 7f9d60bd2700 10 getattrs 3.16_head/e6b69816/rbd_data.108e6b8b4567.0000000000000445/head//3 = 0
2014-06-12 14:51:09.142340 7f9d60bd2700 10 stat 3.16_head/55b9816/rbd_data.10a26b8b4567.0000000000000a17/head//3
2014-06-12 14:51:09.142430 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 55b9816/rbd_data.10a26b8b4567.0000000000000a17/head//3
2014-06-12 14:51:09.150532 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:09.171509 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:09.171596 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:09.176749 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:09.178935 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:09.182156 7f9d60bd2700 10 getattrs 3.16_head/55b9816/rbd_data.10a26b8b4567.0000000000000a17/head//3 = 0
2014-06-12 14:51:09.182169 7f9d60bd2700 10 stat 3.16_head/329b9816/rbd_data.10ac6b8b4567.0000000000002423/head//3
2014-06-12 14:51:09.182279 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 329b9816/rbd_data.10ac6b8b4567.0000000000002423/head//3
2014-06-12 14:51:09.216540 7f9d60bd2700 10 getattrs 3.16_head/329b9816/rbd_data.10ac6b8b4567.0000000000002423/head//3 = 0
2014-06-12 14:51:09.216550 7f9d60bd2700 10 stat 3.16_head/b300a816/rbd_data.10b26b8b4567.00000000000022d7/head//3
2014-06-12 14:51:09.216668 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid b300a816/rbd_data.10b26b8b4567.00000000000022d7/head//3
2014-06-12 14:51:09.216771 7f9d60bd2700 10 getattrs 3.16_head/b300a816/rbd_data.10b26b8b4567.00000000000022d7/head//3 = 0
2014-06-12 14:51:09.216783 7f9d60bd2700 10 stat 3.16_head/4c21a816/rbd_data.10ab6b8b4567.000000000000197e/head//3
2014-06-12 14:51:09.216887 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 4c21a816/rbd_data.10ab6b8b4567.000000000000197e/head//3
2014-06-12 14:51:09.239396 7f9d60bd2700 10 getattrs 3.16_head/4c21a816/rbd_data.10ab6b8b4567.000000000000197e/head//3 = 0
2014-06-12 14:51:09.239406 7f9d60bd2700 10 stat 3.16_head/7e42a816/rbd_data.10a06b8b4567.0000000000000e2d/head//3
2014-06-12 14:51:09.239523 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 7e42a816/rbd_data.10a06b8b4567.0000000000000e2d/head//3
2014-06-12 14:51:09.262691 7f9d60bd2700 10 getattrs 3.16_head/7e42a816/rbd_data.10a06b8b4567.0000000000000e2d/head//3 = 0
2014-06-12 14:51:09.262707 7f9d60bd2700 10 stat 3.16_head/5de2a816/rbd_data.10a96b8b4567.0000000000001516/head//3
2014-06-12 14:51:09.262880 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 5de2a816/rbd_data.10a96b8b4567.0000000000001516/head//3
2014-06-12 14:51:09.274240 7f9d60bd2700 10 getattrs 3.16_head/5de2a816/rbd_data.10a96b8b4567.0000000000001516/head//3 = 0
2014-06-12 14:51:09.274264 7f9d60bd2700 10 stat 3.16_head/5694a816/rbd_data.109d6b8b4567.000000000000064e/head//3
2014-06-12 14:51:09.274454 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 5694a816/rbd_data.109d6b8b4567.000000000000064e/head//3
2014-06-12 14:51:09.281714 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:09.297170 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:09.312308 7f9d60bd2700 10 getattrs 3.16_head/5694a816/rbd_data.109d6b8b4567.000000000000064e/head//3 = 0
2014-06-12 14:51:09.312316 7f9d60bd2700 10 stat 3.16_head/3876a816/rbd_data.10a66b8b4567.00000000000017e1/head//3
2014-06-12 14:51:09.312420 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 3876a816/rbd_data.10a66b8b4567.00000000000017e1/head//3
2014-06-12 14:51:09.332056 7f9d60bd2700 10 getattrs 3.16_head/3876a816/rbd_data.10a66b8b4567.00000000000017e1/head//3 = 0
2014-06-12 14:51:09.332065 7f9d60bd2700 10 stat 3.16_head/7379a816/rbd_data.10906b8b4567.00000000000036cb/head//3
2014-06-12 14:51:09.332171 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 7379a816/rbd_data.10906b8b4567.00000000000036cb/head//3
2014-06-12 14:51:09.336004 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:09.340332 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:09.345014 7f9d60bd2700 10 getattrs 3.16_head/7379a816/rbd_data.10906b8b4567.00000000000036cb/head//3 = 0
2014-06-12 14:51:09.345030 7f9d60bd2700 10 stat 3.16_head/11cba816/rbd_data.10986b8b4567.00000000000037af/head//3
2014-06-12 14:51:09.345222 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 11cba816/rbd_data.10986b8b4567.00000000000037af/head//3
2014-06-12 14:51:09.375834 7f9d60bd2700 10 getattrs 3.16_head/11cba816/rbd_data.10986b8b4567.00000000000037af/head//3 = 0
2014-06-12 14:51:09.375851 7f9d60bd2700 10 stat 3.16_head/75dda816/rbd_data.10ab6b8b4567.000000000000237c/head//3
2014-06-12 14:51:09.376035 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 75dda816/rbd_data.10ab6b8b4567.000000000000237c/head//3
2014-06-12 14:51:09.388930 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:09.390558 7f9d60bd2700 10 getattrs 3.16_head/75dda816/rbd_data.10ab6b8b4567.000000000000237c/head//3 = 0
2014-06-12 14:51:09.390567 7f9d60bd2700 10 stat 3.16_head/801fa816/rbd_data.10ab6b8b4567.000000000000020f/head//3
2014-06-12 14:51:09.390662 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 801fa816/rbd_data.10ab6b8b4567.000000000000020f/head//3
2014-06-12 14:51:09.423248 7f9d58d5b700  5 -- op tracker -- , seq: 748565, time: 2014-06-12 14:51:09.423127, event: header_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:09.423267 7f9d58d5b700  5 -- op tracker -- , seq: 748565, time: 2014-06-12 14:51:09.423131, event: throttled, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:09.423279 7f9d58d5b700  5 -- op tracker -- , seq: 748565, time: 2014-06-12 14:51:09.423198, event: all_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:09.423312 7f9d58d5b700  5 -- op tracker -- , seq: 748565, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:09.423319 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:09.423328 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:51:09.423342 7f9d58d5b700 15 osd.2 173 enqueue_op 0x23e08f00 prio 127 cost 10717 latency 0.000214 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:09.423415 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:09.437079 7f9d60bd2700 10 getattrs 3.16_head/801fa816/rbd_data.10ab6b8b4567.000000000000020f/head//3 = 0
2014-06-12 14:51:09.437093 7f9d60bd2700 10 stat 3.16_head/d2a2b816/rbd_data.108f6b8b4567.0000000000000f59/head//3
2014-06-12 14:51:09.437223 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid d2a2b816/rbd_data.108f6b8b4567.0000000000000f59/head//3
2014-06-12 14:51:09.470750 7f9d60bd2700 10 getattrs 3.16_head/d2a2b816/rbd_data.108f6b8b4567.0000000000000f59/head//3 = 0
2014-06-12 14:51:09.470759 7f9d60bd2700 10 stat 3.16_head/b3a4b816/rbd_data.10a76b8b4567.0000000000000485/head//3
2014-06-12 14:51:09.470856 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid b3a4b816/rbd_data.10a76b8b4567.0000000000000485/head//3
2014-06-12 14:51:09.512092 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:09.532822 7f9d60bd2700 10 getattrs 3.16_head/b3a4b816/rbd_data.10a76b8b4567.0000000000000485/head//3 = 0
2014-06-12 14:51:09.532831 7f9d60bd2700 10 stat 3.16_head/70fab816/rbd_data.10996b8b4567.000000000000181f/head//3
2014-06-12 14:51:09.532930 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 70fab816/rbd_data.10996b8b4567.000000000000181f/head//3
2014-06-12 14:51:09.545565 7f9d60bd2700 10 getattrs 3.16_head/70fab816/rbd_data.10996b8b4567.000000000000181f/head//3 = 0
2014-06-12 14:51:09.545574 7f9d60bd2700 10 collection_getattrs 3.16_head
2014-06-12 14:51:09.545663 7f9d60bd2700 10 collection_getattrs 3.16_head = 0
2014-06-12 14:51:09.545670 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:09.545696 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:09.545702 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:09.545710 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:09.545712 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:09.545740 7f9d643d9700 10 osd.2 173 dequeue_op 0x23e08f00 prio 127 cost 10717 latency 0.122612 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]
2014-06-12 14:51:09.545785 7f9d643d9700  5 -- op tracker -- , seq: 748565, time: 2014-06-12 14:51:09.545778, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:09.545812 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] handle_message: 0x23e08f00
2014-06-12 14:51:09.545835 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:09.545854 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:09.545871 7f9d643d9700  5 -- op tracker -- , seq: 748565, time: 2014-06-12 14:51:09.545866, event: started, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:09.545885 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  got 22 scrub map
2014-06-12 14:51:09.545990 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] map version is 173'2396
2014-06-12 14:51:09.546011 7f9d643d9700 10 osd.2 173 dequeue_op 0x23e08f00 finish
2014-06-12 14:51:09.546022 7f9d643d9700  5 -- op tracker -- , seq: 748565, time: 2014-06-12 14:51:09.546016, event: done, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:09.546045 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:09.546050 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:09.546048 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:09.546061 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:09.546083 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:09.546096 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:09.546134 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:09.546140 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:09.546146 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub replica 22 has 25 items
2014-06-12 14:51:09.546164 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5be58816/rbd_data.10b46b8b4567.000000000000033c/head//3, auth == maps.end()
2014-06-12 14:51:09.546184 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 5be58816/rbd_data.10b46b8b4567.000000000000033c/head//3
2014-06-12 14:51:09.546197 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1f768816/rbd_data.10b26b8b4567.00000000000005f8/head//3, auth == maps.end()
2014-06-12 14:51:09.546206 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 1f768816/rbd_data.10b26b8b4567.00000000000005f8/head//3
2014-06-12 14:51:09.546214 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 496b8816/rbd_data.109b6b8b4567.000000000000196f/head//3, auth == maps.end()
2014-06-12 14:51:09.546222 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 496b8816/rbd_data.109b6b8b4567.000000000000196f/head//3
2014-06-12 14:51:09.546230 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e0ee8816/rbd_data.10b16b8b4567.00000000000008f8/head//3, auth == maps.end()
2014-06-12 14:51:09.546237 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj e0ee8816/rbd_data.10b16b8b4567.00000000000008f8/head//3
2014-06-12 14:51:09.546245 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9dbf8816/rbd_data.10916b8b4567.00000000000020ef/head//3, auth == maps.end()
2014-06-12 14:51:09.546252 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 9dbf8816/rbd_data.10916b8b4567.00000000000020ef/head//3
2014-06-12 14:51:09.546260 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5b519816/rbd_data.10af6b8b4567.00000000000031c9/head//3, auth == maps.end()
2014-06-12 14:51:09.546268 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 5b519816/rbd_data.10af6b8b4567.00000000000031c9/head//3
2014-06-12 14:51:09.546277 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d2349816/rbd_data.109c6b8b4567.0000000000001fc7/head//3, auth == maps.end()
2014-06-12 14:51:09.546286 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj d2349816/rbd_data.109c6b8b4567.0000000000001fc7/head//3
2014-06-12 14:51:09.546294 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 86a59816/rbd_data.108f6b8b4567.00000000000014dd/head//3, auth == maps.end()
2014-06-12 14:51:09.546302 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 86a59816/rbd_data.108f6b8b4567.00000000000014dd/head//3
2014-06-12 14:51:09.546311 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c969816/rbd_data.10ad6b8b4567.0000000000003858/head//3, auth == maps.end()
2014-06-12 14:51:09.546319 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj c969816/rbd_data.10ad6b8b4567.0000000000003858/head//3
2014-06-12 14:51:09.546328 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e6b69816/rbd_data.108e6b8b4567.0000000000000445/head//3, auth == maps.end()
2014-06-12 14:51:09.546336 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj e6b69816/rbd_data.108e6b8b4567.0000000000000445/head//3
2014-06-12 14:51:09.546344 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 55b9816/rbd_data.10a26b8b4567.0000000000000a17/head//3, auth == maps.end()
2014-06-12 14:51:09.546353 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 55b9816/rbd_data.10a26b8b4567.0000000000000a17/head//3
2014-06-12 14:51:09.546362 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 329b9816/rbd_data.10ac6b8b4567.0000000000002423/head//3, auth == maps.end()
2014-06-12 14:51:09.546370 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 329b9816/rbd_data.10ac6b8b4567.0000000000002423/head//3
2014-06-12 14:51:09.546378 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b300a816/rbd_data.10b26b8b4567.00000000000022d7/head//3, auth == maps.end()
2014-06-12 14:51:09.546385 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj b300a816/rbd_data.10b26b8b4567.00000000000022d7/head//3
2014-06-12 14:51:09.546393 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4c21a816/rbd_data.10ab6b8b4567.000000000000197e/head//3, auth == maps.end()
2014-06-12 14:51:09.546401 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 4c21a816/rbd_data.10ab6b8b4567.000000000000197e/head//3
2014-06-12 14:51:09.546409 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7e42a816/rbd_data.10a06b8b4567.0000000000000e2d/head//3, auth == maps.end()
2014-06-12 14:51:09.546416 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 7e42a816/rbd_data.10a06b8b4567.0000000000000e2d/head//3
2014-06-12 14:51:09.546424 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5de2a816/rbd_data.10a96b8b4567.0000000000001516/head//3, auth == maps.end()
2014-06-12 14:51:09.546433 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 5de2a816/rbd_data.10a96b8b4567.0000000000001516/head//3
2014-06-12 14:51:09.546441 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5694a816/rbd_data.109d6b8b4567.000000000000064e/head//3, auth == maps.end()
2014-06-12 14:51:09.546450 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 5694a816/rbd_data.109d6b8b4567.000000000000064e/head//3
2014-06-12 14:51:09.546460 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3876a816/rbd_data.10a66b8b4567.00000000000017e1/head//3, auth == maps.end()
2014-06-12 14:51:09.546469 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 3876a816/rbd_data.10a66b8b4567.00000000000017e1/head//3
2014-06-12 14:51:09.546478 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7379a816/rbd_data.10906b8b4567.00000000000036cb/head//3, auth == maps.end()
2014-06-12 14:51:09.546486 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 7379a816/rbd_data.10906b8b4567.00000000000036cb/head//3
2014-06-12 14:51:09.546495 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 11cba816/rbd_data.10986b8b4567.00000000000037af/head//3, auth == maps.end()
2014-06-12 14:51:09.546504 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 11cba816/rbd_data.10986b8b4567.00000000000037af/head//3
2014-06-12 14:51:09.546514 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 75dda816/rbd_data.10ab6b8b4567.000000000000237c/head//3, auth == maps.end()
2014-06-12 14:51:09.546523 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 75dda816/rbd_data.10ab6b8b4567.000000000000237c/head//3
2014-06-12 14:51:09.546533 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 801fa816/rbd_data.10ab6b8b4567.000000000000020f/head//3, auth == maps.end()
2014-06-12 14:51:09.546542 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 801fa816/rbd_data.10ab6b8b4567.000000000000020f/head//3
2014-06-12 14:51:09.546550 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d2a2b816/rbd_data.108f6b8b4567.0000000000000f59/head//3, auth == maps.end()
2014-06-12 14:51:09.546559 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj d2a2b816/rbd_data.108f6b8b4567.0000000000000f59/head//3
2014-06-12 14:51:09.546568 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b3a4b816/rbd_data.10a76b8b4567.0000000000000485/head//3, auth == maps.end()
2014-06-12 14:51:09.546576 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj b3a4b816/rbd_data.10a76b8b4567.0000000000000485/head//3
2014-06-12 14:51:09.546585 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 70fab816/rbd_data.10996b8b4567.000000000000181f/head//3, auth == maps.end()
2014-06-12 14:51:09.546593 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 70fab816/rbd_data.10996b8b4567.000000000000181f/head//3
2014-06-12 14:51:09.546602 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] 
2014-06-12 14:51:09.546608 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub
2014-06-12 14:51:09.546619 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  70fab816/rbd_data.10996b8b4567.000000000000181f/head//3 70fab816/rbd_data.10996b8b4567.000000000000181f/head//3(173'776 client.4290.0:52070 wrlock_by=unknown.0.0:0 dirty s 4194304 uv776)
2014-06-12 14:51:09.546632 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  b3a4b816/rbd_data.10a76b8b4567.0000000000000485/head//3 b3a4b816/rbd_data.10a76b8b4567.0000000000000485/head//3(173'2121 client.4304.0:123706 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2121)
2014-06-12 14:51:09.546643 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  d2a2b816/rbd_data.108f6b8b4567.0000000000000f59/head//3 d2a2b816/rbd_data.108f6b8b4567.0000000000000f59/head//3(173'583 client.4280.0:36300 wrlock_by=unknown.0.0:0 dirty s 4194304 uv583)
2014-06-12 14:51:09.546653 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  801fa816/rbd_data.10ab6b8b4567.000000000000020f/head//3 801fa816/rbd_data.10ab6b8b4567.000000000000020f/head//3(173'2064 client.4308.0:121900 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2064)
2014-06-12 14:51:09.546663 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  75dda816/rbd_data.10ab6b8b4567.000000000000237c/head//3 75dda816/rbd_data.10ab6b8b4567.000000000000237c/head//3(173'1238 client.4308.0:73719 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1238)
2014-06-12 14:51:09.546674 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  11cba816/rbd_data.10986b8b4567.00000000000037af/head//3 11cba816/rbd_data.10986b8b4567.00000000000037af/head//3(173'1861 client.4289.0:108291 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1861)
2014-06-12 14:51:09.546684 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  7379a816/rbd_data.10906b8b4567.00000000000036cb/head//3 7379a816/rbd_data.10906b8b4567.00000000000036cb/head//3(173'1810 client.4281.0:108157 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1810)
2014-06-12 14:51:09.546693 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  3876a816/rbd_data.10a66b8b4567.00000000000017e1/head//3 3876a816/rbd_data.10a66b8b4567.00000000000017e1/head//3(173'851 client.4303.0:51347 wrlock_by=unknown.0.0:0 dirty s 4194304 uv851)
2014-06-12 14:51:09.546703 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  5694a816/rbd_data.109d6b8b4567.000000000000064e/head//3 5694a816/rbd_data.109d6b8b4567.000000000000064e/head//3(173'2171 client.4294.0:126430 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2171)
2014-06-12 14:51:09.546714 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  5de2a816/rbd_data.10a96b8b4567.0000000000001516/head//3 5de2a816/rbd_data.10a96b8b4567.0000000000001516/head//3(173'792 client.4306.0:49023 wrlock_by=unknown.0.0:0 dirty s 4194304 uv792)
2014-06-12 14:51:09.546724 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  7e42a816/rbd_data.10a06b8b4567.0000000000000e2d/head//3 7e42a816/rbd_data.10a06b8b4567.0000000000000e2d/head//3(173'2372 client.4297.0:140132 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2372)
2014-06-12 14:51:09.546735 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  4c21a816/rbd_data.10ab6b8b4567.000000000000197e/head//3 4c21a816/rbd_data.10ab6b8b4567.000000000000197e/head//3(173'958 client.4308.0:56154 wrlock_by=unknown.0.0:0 dirty s 4194304 uv958)
2014-06-12 14:51:09.546745 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  b300a816/rbd_data.10b26b8b4567.00000000000022d7/head//3 b300a816/rbd_data.10b26b8b4567.00000000000022d7/head//3(173'1198 client.4315.0:72220 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1198)
2014-06-12 14:51:09.546754 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  329b9816/rbd_data.10ac6b8b4567.0000000000002423/head//3 329b9816/rbd_data.10ac6b8b4567.0000000000002423/head//3(173'1317 client.4309.0:72821 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1317)
2014-06-12 14:51:09.546765 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  55b9816/rbd_data.10a26b8b4567.0000000000000a17/head//3 55b9816/rbd_data.10a26b8b4567.0000000000000a17/head//3(173'2303 client.4299.0:136355 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2303)
2014-06-12 14:51:09.546776 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  e6b69816/rbd_data.108e6b8b4567.0000000000000445/head//3 e6b69816/rbd_data.108e6b8b4567.0000000000000445/head//3(173'2105 client.4279.0:123306 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2105)
2014-06-12 14:51:09.546787 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  c969816/rbd_data.10ad6b8b4567.0000000000003858/head//3 c969816/rbd_data.10ad6b8b4567.0000000000003858/head//3(173'1907 client.4310.0:111001 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1907)
2014-06-12 14:51:09.546798 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  86a59816/rbd_data.108f6b8b4567.00000000000014dd/head//3 86a59816/rbd_data.108f6b8b4567.00000000000014dd/head//3(173'760 client.4280.0:46614 wrlock_by=unknown.0.0:0 dirty s 4194304 uv760)
2014-06-12 14:51:09.546809 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  d2349816/rbd_data.109c6b8b4567.0000000000001fc7/head//3 d2349816/rbd_data.109c6b8b4567.0000000000001fc7/head//3(173'1213 client.4293.0:65921 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1213)
2014-06-12 14:51:09.546820 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  5b519816/rbd_data.10af6b8b4567.00000000000031c9/head//3 5b519816/rbd_data.10af6b8b4567.00000000000031c9/head//3(173'1746 client.4312.0:101478 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1746)
2014-06-12 14:51:09.546831 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  9dbf8816/rbd_data.10916b8b4567.00000000000020ef/head//3 9dbf8816/rbd_data.10916b8b4567.00000000000020ef/head//3(173'1224 client.4282.0:69058 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1224)
2014-06-12 14:51:09.546842 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  e0ee8816/rbd_data.10b16b8b4567.00000000000008f8/head//3 e0ee8816/rbd_data.10b16b8b4567.00000000000008f8/head//3(173'2271 client.4314.0:132454 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2271)
2014-06-12 14:51:09.546853 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  496b8816/rbd_data.109b6b8b4567.000000000000196f/head//3 496b8816/rbd_data.109b6b8b4567.000000000000196f/head//3(173'1044 client.4292.0:53945 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1044)
2014-06-12 14:51:09.546863 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  1f768816/rbd_data.10b26b8b4567.00000000000005f8/head//3 1f768816/rbd_data.10b26b8b4567.00000000000005f8/head//3(173'2201 client.4315.0:126884 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2201)
2014-06-12 14:51:09.546873 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  5be58816/rbd_data.10b46b8b4567.000000000000033c/head//3 5be58816/rbd_data.10b46b8b4567.000000000000033c/head//3(173'2096 client.4317.0:121785 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2096)
2014-06-12 14:51:09.546882 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:09.546891 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:09.546899 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:09.546901 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:09.546902 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:09.546936 7f9d60bd2700 10 collection_list_partial 3.16_head start:fabb816//0//-1 is_max:0
2014-06-12 14:51:09.547127 7f9d60bd2700 20 objects: [159eb816/rbd_data.108d6b8b4567.0000000000001393/head//3,9dceb816/rbd_data.10986b8b4567.000000000000115f/head//3,aa60c816/rbd_data.10b26b8b4567.00000000000010e2/head//3,6271c816/rbd_data.10b26b8b4567.000000000000165e/head//3,54b7c816/rbd_data.10ad6b8b4567.0000000000001020/head//3,d499c816/rbd_data.109f6b8b4567.0000000000002cd5/head//3,7e00d816/rbd_data.10936b8b4567.0000000000002946/head//3,172d816/rbd_data.10b16b8b4567.00000000000005a9/head//3,4d82d816/rbd_data.108e6b8b4567.0000000000002bfd/head//3,9fb3d816/rbd_data.10936b8b4567.0000000000001805/head//3,8fcfd816/rbd_data.109d6b8b4567.00000000000011b9/head//3,7d33e816/rbd_data.10986b8b4567.0000000000001672/head//3,8093e816/rbd_data.10946b8b4567.0000000000001f6a/head//3,4e04e816/rbd_data.109e6b8b4567.0000000000002350/head//3,43f4e816/rbd_data.10b46b8b4567.00000000000034a3/head//3,5d8ae816/rbd_data.10ae6b8b4567.0000000000001fe5/head//3,61dce816/rbd_data.109e6b8b4567.0000000000001cf4/head//3,b48ee816/rbd_data.10a76b8b4567.00000000000039df/head//3,39afe816/rbd_data.109e6b8b4567.00000000000017b8/head//3,f40f816/rbd_data.10ac6b8b4567.0000000000002c72/head//3,7453f816/rbd_data.109e6b8b4567.000000000000364e/head//3,eb45f816/rbd_data.109f6b8b4567.0000000000000f57/head//3,1275f816/rbd_data.10b16b8b4567.000000000000399e/head//3,3ee5f816/rbd_data.109e6b8b4567.0000000000002130/head//3,390bf816/rbd_data.108d6b8b4567.0000000000001332/head//3]
2014-06-12 14:51:09.547510 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  requesting scrubmap from osd.22
2014-06-12 14:51:09.547530 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:09.547546 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:09.547554 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:09.547559 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:09.547564 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:09.547569 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_map_chunk [fabb816//0//-1,4aedf816//0//-1)
2014-06-12 14:51:09.547575 7f9d60bd2700 10 collection_list_partial 3.16_head start:fabb816//0//-1 is_max:0
2014-06-12 14:51:09.547729 7f9d60bd2700 20 objects: [159eb816/rbd_data.108d6b8b4567.0000000000001393/head//3,9dceb816/rbd_data.10986b8b4567.000000000000115f/head//3,aa60c816/rbd_data.10b26b8b4567.00000000000010e2/head//3,6271c816/rbd_data.10b26b8b4567.000000000000165e/head//3,54b7c816/rbd_data.10ad6b8b4567.0000000000001020/head//3,d499c816/rbd_data.109f6b8b4567.0000000000002cd5/head//3,7e00d816/rbd_data.10936b8b4567.0000000000002946/head//3,172d816/rbd_data.10b16b8b4567.00000000000005a9/head//3,4d82d816/rbd_data.108e6b8b4567.0000000000002bfd/head//3,9fb3d816/rbd_data.10936b8b4567.0000000000001805/head//3,8fcfd816/rbd_data.109d6b8b4567.00000000000011b9/head//3,7d33e816/rbd_data.10986b8b4567.0000000000001672/head//3,8093e816/rbd_data.10946b8b4567.0000000000001f6a/head//3,4e04e816/rbd_data.109e6b8b4567.0000000000002350/head//3,43f4e816/rbd_data.10b46b8b4567.00000000000034a3/head//3,5d8ae816/rbd_data.10ae6b8b4567.0000000000001fe5/head//3,61dce816/rbd_data.109e6b8b4567.0000000000001cf4/head//3,b48ee816/rbd_data.10a76b8b4567.00000000000039df/head//3,39afe816/rbd_data.109e6b8b4567.00000000000017b8/head//3,f40f816/rbd_data.10ac6b8b4567.0000000000002c72/head//3,7453f816/rbd_data.109e6b8b4567.000000000000364e/head//3,eb45f816/rbd_data.109f6b8b4567.0000000000000f57/head//3,1275f816/rbd_data.10b16b8b4567.000000000000399e/head//3,3ee5f816/rbd_data.109e6b8b4567.0000000000002130/head//3,390bf816/rbd_data.108d6b8b4567.0000000000001332/head//3,4aedf816/rbd_data.10986b8b4567.0000000000003243/head//3,c32ff816/rbd_data.10906b8b4567.0000000000002107/head//3,e56ff816/rbd_data.109a6b8b4567.000000000000371e/head//3]
2014-06-12 14:51:09.547774 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:09.547788 7f9d60bd2700 10 stat 3.16_head/159eb816/rbd_data.108d6b8b4567.0000000000001393/head//3
2014-06-12 14:51:09.547885 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 159eb816/rbd_data.108d6b8b4567.0000000000001393/head//3
2014-06-12 14:51:09.553520 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:09.553593 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:09.580677 7f9d60bd2700 10 getattrs 3.16_head/159eb816/rbd_data.108d6b8b4567.0000000000001393/head//3 = 0
2014-06-12 14:51:09.580685 7f9d60bd2700 10 stat 3.16_head/9dceb816/rbd_data.10986b8b4567.000000000000115f/head//3
2014-06-12 14:51:09.580776 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 9dceb816/rbd_data.10986b8b4567.000000000000115f/head//3
2014-06-12 14:51:09.594910 7f9d60bd2700 10 getattrs 3.16_head/9dceb816/rbd_data.10986b8b4567.000000000000115f/head//3 = 0
2014-06-12 14:51:09.594925 7f9d60bd2700 10 stat 3.16_head/aa60c816/rbd_data.10b26b8b4567.00000000000010e2/head//3
2014-06-12 14:51:09.595011 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid aa60c816/rbd_data.10b26b8b4567.00000000000010e2/head//3
2014-06-12 14:51:09.595103 7f9d60bd2700 10 getattrs 3.16_head/aa60c816/rbd_data.10b26b8b4567.00000000000010e2/head//3 = 0
2014-06-12 14:51:09.595113 7f9d60bd2700 10 stat 3.16_head/6271c816/rbd_data.10b26b8b4567.000000000000165e/head//3
2014-06-12 14:51:09.595190 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 6271c816/rbd_data.10b26b8b4567.000000000000165e/head//3
2014-06-12 14:51:09.610600 7f9d60bd2700 10 getattrs 3.16_head/6271c816/rbd_data.10b26b8b4567.000000000000165e/head//3 = 0
2014-06-12 14:51:09.610614 7f9d60bd2700 10 stat 3.16_head/54b7c816/rbd_data.10ad6b8b4567.0000000000001020/head//3
2014-06-12 14:51:09.610780 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 54b7c816/rbd_data.10ad6b8b4567.0000000000001020/head//3
2014-06-12 14:51:09.645523 7f9d60bd2700 10 getattrs 3.16_head/54b7c816/rbd_data.10ad6b8b4567.0000000000001020/head//3 = 0
2014-06-12 14:51:09.645537 7f9d60bd2700 10 stat 3.16_head/d499c816/rbd_data.109f6b8b4567.0000000000002cd5/head//3
2014-06-12 14:51:09.645713 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid d499c816/rbd_data.109f6b8b4567.0000000000002cd5/head//3
2014-06-12 14:51:09.664045 7f9d60bd2700 10 getattrs 3.16_head/d499c816/rbd_data.109f6b8b4567.0000000000002cd5/head//3 = 0
2014-06-12 14:51:09.664053 7f9d60bd2700 10 stat 3.16_head/7e00d816/rbd_data.10936b8b4567.0000000000002946/head//3
2014-06-12 14:51:09.664154 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 7e00d816/rbd_data.10936b8b4567.0000000000002946/head//3
2014-06-12 14:51:09.680794 7f9d60bd2700 10 getattrs 3.16_head/7e00d816/rbd_data.10936b8b4567.0000000000002946/head//3 = 0
2014-06-12 14:51:09.680807 7f9d60bd2700 10 stat 3.16_head/172d816/rbd_data.10b16b8b4567.00000000000005a9/head//3
2014-06-12 14:51:09.680920 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 172d816/rbd_data.10b16b8b4567.00000000000005a9/head//3
2014-06-12 14:51:09.735236 7f9d60bd2700 10 getattrs 3.16_head/172d816/rbd_data.10b16b8b4567.00000000000005a9/head//3 = 0
2014-06-12 14:51:09.735250 7f9d60bd2700 10 stat 3.16_head/4d82d816/rbd_data.108e6b8b4567.0000000000002bfd/head//3
2014-06-12 14:51:09.735377 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 4d82d816/rbd_data.108e6b8b4567.0000000000002bfd/head//3
2014-06-12 14:51:09.748383 7f9d60bd2700 10 getattrs 3.16_head/4d82d816/rbd_data.108e6b8b4567.0000000000002bfd/head//3 = 0
2014-06-12 14:51:09.748394 7f9d60bd2700 10 stat 3.16_head/9fb3d816/rbd_data.10936b8b4567.0000000000001805/head//3
2014-06-12 14:51:09.748519 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 9fb3d816/rbd_data.10936b8b4567.0000000000001805/head//3
2014-06-12 14:51:09.766014 7f9d60bd2700 10 getattrs 3.16_head/9fb3d816/rbd_data.10936b8b4567.0000000000001805/head//3 = 0
2014-06-12 14:51:09.766028 7f9d60bd2700 10 stat 3.16_head/8fcfd816/rbd_data.109d6b8b4567.00000000000011b9/head//3
2014-06-12 14:51:09.766152 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 8fcfd816/rbd_data.109d6b8b4567.00000000000011b9/head//3
2014-06-12 14:51:09.781559 7f9d60bd2700 10 getattrs 3.16_head/8fcfd816/rbd_data.109d6b8b4567.00000000000011b9/head//3 = 0
2014-06-12 14:51:09.781570 7f9d60bd2700 10 stat 3.16_head/7d33e816/rbd_data.10986b8b4567.0000000000001672/head//3
2014-06-12 14:51:09.781711 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 7d33e816/rbd_data.10986b8b4567.0000000000001672/head//3
2014-06-12 14:51:09.792114 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:09.792154 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:09.793415 7f9d60bd2700 10 getattrs 3.16_head/7d33e816/rbd_data.10986b8b4567.0000000000001672/head//3 = 0
2014-06-12 14:51:09.793422 7f9d60bd2700 10 stat 3.16_head/8093e816/rbd_data.10946b8b4567.0000000000001f6a/head//3
2014-06-12 14:51:09.793503 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 8093e816/rbd_data.10946b8b4567.0000000000001f6a/head//3
2014-06-12 14:51:09.807283 7f9d60bd2700 10 getattrs 3.16_head/8093e816/rbd_data.10946b8b4567.0000000000001f6a/head//3 = 0
2014-06-12 14:51:09.807291 7f9d60bd2700 10 stat 3.16_head/4e04e816/rbd_data.109e6b8b4567.0000000000002350/head//3
2014-06-12 14:51:09.807380 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 4e04e816/rbd_data.109e6b8b4567.0000000000002350/head//3
2014-06-12 14:51:09.841100 7f9d60bd2700 10 getattrs 3.16_head/4e04e816/rbd_data.109e6b8b4567.0000000000002350/head//3 = 0
2014-06-12 14:51:09.841110 7f9d60bd2700 10 stat 3.16_head/43f4e816/rbd_data.10b46b8b4567.00000000000034a3/head//3
2014-06-12 14:51:09.841217 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 43f4e816/rbd_data.10b46b8b4567.00000000000034a3/head//3
2014-06-12 14:51:09.856366 7f9d60bd2700 10 getattrs 3.16_head/43f4e816/rbd_data.10b46b8b4567.00000000000034a3/head//3 = 0
2014-06-12 14:51:09.856376 7f9d60bd2700 10 stat 3.16_head/5d8ae816/rbd_data.10ae6b8b4567.0000000000001fe5/head//3
2014-06-12 14:51:09.856475 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 5d8ae816/rbd_data.10ae6b8b4567.0000000000001fe5/head//3
2014-06-12 14:51:09.869731 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:09.869758 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:09.869763 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:09.869766 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:09.869794 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:09.879999 7f9d60bd2700 10 getattrs 3.16_head/5d8ae816/rbd_data.10ae6b8b4567.0000000000001fe5/head//3 = 0
2014-06-12 14:51:09.880010 7f9d60bd2700 10 stat 3.16_head/61dce816/rbd_data.109e6b8b4567.0000000000001cf4/head//3
2014-06-12 14:51:09.880129 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 61dce816/rbd_data.109e6b8b4567.0000000000001cf4/head//3
2014-06-12 14:51:09.905579 7f9d60bd2700 10 getattrs 3.16_head/61dce816/rbd_data.109e6b8b4567.0000000000001cf4/head//3 = 0
2014-06-12 14:51:09.905592 7f9d60bd2700 10 stat 3.16_head/b48ee816/rbd_data.10a76b8b4567.00000000000039df/head//3
2014-06-12 14:51:09.905727 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid b48ee816/rbd_data.10a76b8b4567.00000000000039df/head//3
2014-06-12 14:51:09.941289 7f9d60bd2700 10 getattrs 3.16_head/b48ee816/rbd_data.10a76b8b4567.00000000000039df/head//3 = 0
2014-06-12 14:51:09.941300 7f9d60bd2700 10 stat 3.16_head/39afe816/rbd_data.109e6b8b4567.00000000000017b8/head//3
2014-06-12 14:51:09.941426 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 39afe816/rbd_data.109e6b8b4567.00000000000017b8/head//3
2014-06-12 14:51:09.970070 7f9d60bd2700 10 getattrs 3.16_head/39afe816/rbd_data.109e6b8b4567.00000000000017b8/head//3 = 0
2014-06-12 14:51:09.970086 7f9d60bd2700 10 stat 3.16_head/f40f816/rbd_data.10ac6b8b4567.0000000000002c72/head//3
2014-06-12 14:51:09.970208 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid f40f816/rbd_data.10ac6b8b4567.0000000000002c72/head//3
2014-06-12 14:51:09.975737 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:09.989398 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:09.989449 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:09.991365 7f9d60bd2700 10 getattrs 3.16_head/f40f816/rbd_data.10ac6b8b4567.0000000000002c72/head//3 = 0
2014-06-12 14:51:09.991379 7f9d60bd2700 10 stat 3.16_head/7453f816/rbd_data.109e6b8b4567.000000000000364e/head//3
2014-06-12 14:51:09.991523 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 7453f816/rbd_data.109e6b8b4567.000000000000364e/head//3
2014-06-12 14:51:09.996759 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:09.996855 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:10.017118 7f9d60bd2700 10 getattrs 3.16_head/7453f816/rbd_data.109e6b8b4567.000000000000364e/head//3 = 0
2014-06-12 14:51:10.017130 7f9d60bd2700 10 stat 3.16_head/eb45f816/rbd_data.109f6b8b4567.0000000000000f57/head//3
2014-06-12 14:51:10.017265 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid eb45f816/rbd_data.109f6b8b4567.0000000000000f57/head//3
2014-06-12 14:51:10.019917 7f9d58d5b700  5 -- op tracker -- , seq: 748566, time: 2014-06-12 14:51:10.019800, event: header_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.019940 7f9d58d5b700  5 -- op tracker -- , seq: 748566, time: 2014-06-12 14:51:10.019803, event: throttled, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.019968 7f9d58d5b700  5 -- op tracker -- , seq: 748566, time: 2014-06-12 14:51:10.019867, event: all_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.019979 7f9d58d5b700  5 -- op tracker -- , seq: 748566, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.019986 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:10.019995 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:51:10.020004 7f9d58d5b700 15 osd.2 173 enqueue_op 0x23e08e00 prio 127 cost 10717 latency 0.000204 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:10.020076 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:10.050347 7f9d60bd2700 10 getattrs 3.16_head/eb45f816/rbd_data.109f6b8b4567.0000000000000f57/head//3 = 0
2014-06-12 14:51:10.050362 7f9d60bd2700 10 stat 3.16_head/1275f816/rbd_data.10b16b8b4567.000000000000399e/head//3
2014-06-12 14:51:10.050519 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 1275f816/rbd_data.10b16b8b4567.000000000000399e/head//3
2014-06-12 14:51:10.063511 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:10.065264 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:10.065384 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:10.083089 7f9d60bd2700 10 getattrs 3.16_head/1275f816/rbd_data.10b16b8b4567.000000000000399e/head//3 = 0
2014-06-12 14:51:10.083097 7f9d60bd2700 10 stat 3.16_head/3ee5f816/rbd_data.109e6b8b4567.0000000000002130/head//3
2014-06-12 14:51:10.083194 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 3ee5f816/rbd_data.109e6b8b4567.0000000000002130/head//3
2014-06-12 14:51:10.109059 7f9d60bd2700 10 getattrs 3.16_head/3ee5f816/rbd_data.109e6b8b4567.0000000000002130/head//3 = 0
2014-06-12 14:51:10.109067 7f9d60bd2700 10 stat 3.16_head/390bf816/rbd_data.108d6b8b4567.0000000000001332/head//3
2014-06-12 14:51:10.109155 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid 390bf816/rbd_data.108d6b8b4567.0000000000001332/head//3
2014-06-12 14:51:10.124808 7f9d60bd2700 10 getattrs 3.16_head/390bf816/rbd_data.108d6b8b4567.0000000000001332/head//3 = 0
2014-06-12 14:51:10.124818 7f9d60bd2700 10 collection_getattrs 3.16_head
2014-06-12 14:51:10.124874 7f9d60bd2700 10 collection_getattrs 3.16_head = 0
2014-06-12 14:51:10.124878 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:10.124904 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:10.124912 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:10.124920 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:10.124923 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:10.124951 7f9d61bd4700 10 osd.2 173 dequeue_op 0x23e08e00 prio 127 cost 10717 latency 0.105150 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]
2014-06-12 14:51:10.125002 7f9d61bd4700  5 -- op tracker -- , seq: 748566, time: 2014-06-12 14:51:10.124994, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.125020 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] handle_message: 0x23e08e00
2014-06-12 14:51:10.125037 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:10.125055 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:10.125073 7f9d61bd4700  5 -- op tracker -- , seq: 748566, time: 2014-06-12 14:51:10.125067, event: started, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.125091 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  got 22 scrub map
2014-06-12 14:51:10.125191 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] map version is 173'2396
2014-06-12 14:51:10.125213 7f9d61bd4700 10 osd.2 173 dequeue_op 0x23e08e00 finish
2014-06-12 14:51:10.125224 7f9d61bd4700  5 -- op tracker -- , seq: 748566, time: 2014-06-12 14:51:10.125218, event: done, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.125247 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:10.125253 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:10.125252 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:10.125265 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:10.125287 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:10.125300 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:10.125339 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:10.125345 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:10.125351 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub replica 22 has 25 items
2014-06-12 14:51:10.125369 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 159eb816/rbd_data.108d6b8b4567.0000000000001393/head//3, auth == maps.end()
2014-06-12 14:51:10.125388 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 159eb816/rbd_data.108d6b8b4567.0000000000001393/head//3
2014-06-12 14:51:10.125400 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9dceb816/rbd_data.10986b8b4567.000000000000115f/head//3, auth == maps.end()
2014-06-12 14:51:10.125408 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 9dceb816/rbd_data.10986b8b4567.000000000000115f/head//3
2014-06-12 14:51:10.125423 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aa60c816/rbd_data.10b26b8b4567.00000000000010e2/head//3, auth == maps.end()
2014-06-12 14:51:10.125432 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj aa60c816/rbd_data.10b26b8b4567.00000000000010e2/head//3
2014-06-12 14:51:10.125440 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6271c816/rbd_data.10b26b8b4567.000000000000165e/head//3, auth == maps.end()
2014-06-12 14:51:10.125447 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 6271c816/rbd_data.10b26b8b4567.000000000000165e/head//3
2014-06-12 14:51:10.125455 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 54b7c816/rbd_data.10ad6b8b4567.0000000000001020/head//3, auth == maps.end()
2014-06-12 14:51:10.125463 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 54b7c816/rbd_data.10ad6b8b4567.0000000000001020/head//3
2014-06-12 14:51:10.125471 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d499c816/rbd_data.109f6b8b4567.0000000000002cd5/head//3, auth == maps.end()
2014-06-12 14:51:10.125478 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj d499c816/rbd_data.109f6b8b4567.0000000000002cd5/head//3
2014-06-12 14:51:10.125487 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7e00d816/rbd_data.10936b8b4567.0000000000002946/head//3, auth == maps.end()
2014-06-12 14:51:10.125494 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 7e00d816/rbd_data.10936b8b4567.0000000000002946/head//3
2014-06-12 14:51:10.125502 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 172d816/rbd_data.10b16b8b4567.00000000000005a9/head//3, auth == maps.end()
2014-06-12 14:51:10.125509 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 172d816/rbd_data.10b16b8b4567.00000000000005a9/head//3
2014-06-12 14:51:10.125518 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4d82d816/rbd_data.108e6b8b4567.0000000000002bfd/head//3, auth == maps.end()
2014-06-12 14:51:10.125525 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 4d82d816/rbd_data.108e6b8b4567.0000000000002bfd/head//3
2014-06-12 14:51:10.125533 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9fb3d816/rbd_data.10936b8b4567.0000000000001805/head//3, auth == maps.end()
2014-06-12 14:51:10.125541 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 9fb3d816/rbd_data.10936b8b4567.0000000000001805/head//3
2014-06-12 14:51:10.125549 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8fcfd816/rbd_data.109d6b8b4567.00000000000011b9/head//3, auth == maps.end()
2014-06-12 14:51:10.125557 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 8fcfd816/rbd_data.109d6b8b4567.00000000000011b9/head//3
2014-06-12 14:51:10.125565 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7d33e816/rbd_data.10986b8b4567.0000000000001672/head//3, auth == maps.end()
2014-06-12 14:51:10.125572 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 7d33e816/rbd_data.10986b8b4567.0000000000001672/head//3
2014-06-12 14:51:10.125580 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8093e816/rbd_data.10946b8b4567.0000000000001f6a/head//3, auth == maps.end()
2014-06-12 14:51:10.125588 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 8093e816/rbd_data.10946b8b4567.0000000000001f6a/head//3
2014-06-12 14:51:10.125596 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4e04e816/rbd_data.109e6b8b4567.0000000000002350/head//3, auth == maps.end()
2014-06-12 14:51:10.125604 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 4e04e816/rbd_data.109e6b8b4567.0000000000002350/head//3
2014-06-12 14:51:10.125627 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 43f4e816/rbd_data.10b46b8b4567.00000000000034a3/head//3, auth == maps.end()
2014-06-12 14:51:10.125637 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 43f4e816/rbd_data.10b46b8b4567.00000000000034a3/head//3
2014-06-12 14:51:10.125646 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5d8ae816/rbd_data.10ae6b8b4567.0000000000001fe5/head//3, auth == maps.end()
2014-06-12 14:51:10.125662 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 5d8ae816/rbd_data.10ae6b8b4567.0000000000001fe5/head//3
2014-06-12 14:51:10.125672 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 61dce816/rbd_data.109e6b8b4567.0000000000001cf4/head//3, auth == maps.end()
2014-06-12 14:51:10.125686 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 61dce816/rbd_data.109e6b8b4567.0000000000001cf4/head//3
2014-06-12 14:51:10.125696 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b48ee816/rbd_data.10a76b8b4567.00000000000039df/head//3, auth == maps.end()
2014-06-12 14:51:10.125704 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj b48ee816/rbd_data.10a76b8b4567.00000000000039df/head//3
2014-06-12 14:51:10.125713 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 39afe816/rbd_data.109e6b8b4567.00000000000017b8/head//3, auth == maps.end()
2014-06-12 14:51:10.125721 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 39afe816/rbd_data.109e6b8b4567.00000000000017b8/head//3
2014-06-12 14:51:10.125732 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f40f816/rbd_data.10ac6b8b4567.0000000000002c72/head//3, auth == maps.end()
2014-06-12 14:51:10.125740 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj f40f816/rbd_data.10ac6b8b4567.0000000000002c72/head//3
2014-06-12 14:51:10.125749 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7453f816/rbd_data.109e6b8b4567.000000000000364e/head//3, auth == maps.end()
2014-06-12 14:51:10.125759 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 7453f816/rbd_data.109e6b8b4567.000000000000364e/head//3
2014-06-12 14:51:10.125768 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj eb45f816/rbd_data.109f6b8b4567.0000000000000f57/head//3, auth == maps.end()
2014-06-12 14:51:10.125776 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj eb45f816/rbd_data.109f6b8b4567.0000000000000f57/head//3
2014-06-12 14:51:10.125785 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1275f816/rbd_data.10b16b8b4567.000000000000399e/head//3, auth == maps.end()
2014-06-12 14:51:10.125793 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 1275f816/rbd_data.10b16b8b4567.000000000000399e/head//3
2014-06-12 14:51:10.125802 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3ee5f816/rbd_data.109e6b8b4567.0000000000002130/head//3, auth == maps.end()
2014-06-12 14:51:10.125810 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 3ee5f816/rbd_data.109e6b8b4567.0000000000002130/head//3
2014-06-12 14:51:10.125819 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 390bf816/rbd_data.108d6b8b4567.0000000000001332/head//3, auth == maps.end()
2014-06-12 14:51:10.125827 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj 390bf816/rbd_data.108d6b8b4567.0000000000001332/head//3
2014-06-12 14:51:10.125838 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] 
2014-06-12 14:51:10.125844 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub
2014-06-12 14:51:10.125856 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  390bf816/rbd_data.108d6b8b4567.0000000000001332/head//3 390bf816/rbd_data.108d6b8b4567.0000000000001332/head//3(173'653 client.4278.0:45294 wrlock_by=unknown.0.0:0 dirty s 4194304 uv653)
2014-06-12 14:51:10.125873 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  3ee5f816/rbd_data.109e6b8b4567.0000000000002130/head//3 3ee5f816/rbd_data.109e6b8b4567.0000000000002130/head//3(173'1203 client.4295.0:70079 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1203)
2014-06-12 14:51:10.125884 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  1275f816/rbd_data.10b16b8b4567.000000000000399e/head//3 1275f816/rbd_data.10b16b8b4567.000000000000399e/head//3(173'2001 client.4314.0:114933 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2001)
2014-06-12 14:51:10.125897 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  eb45f816/rbd_data.109f6b8b4567.0000000000000f57/head//3 eb45f816/rbd_data.109f6b8b4567.0000000000000f57/head//3(173'2376 client.4296.0:139905 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2376)
2014-06-12 14:51:10.125908 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  7453f816/rbd_data.109e6b8b4567.000000000000364e/head//3 7453f816/rbd_data.109e6b8b4567.000000000000364e/head//3(173'1804 client.4295.0:107467 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1804)
2014-06-12 14:51:10.125920 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  f40f816/rbd_data.10ac6b8b4567.0000000000002c72/head//3 f40f816/rbd_data.10ac6b8b4567.0000000000002c72/head//3(173'1610 client.4309.0:88578 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1610)
2014-06-12 14:51:10.125932 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  39afe816/rbd_data.109e6b8b4567.00000000000017b8/head//3 39afe816/rbd_data.109e6b8b4567.00000000000017b8/head//3(173'882 client.4295.0:53172 wrlock_by=unknown.0.0:0 dirty s 4194304 uv882)
2014-06-12 14:51:10.125942 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  b48ee816/rbd_data.10a76b8b4567.00000000000039df/head//3 b48ee816/rbd_data.10a76b8b4567.00000000000039df/head//3(173'1947 client.4304.0:113807 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1947)
2014-06-12 14:51:10.125955 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  61dce816/rbd_data.109e6b8b4567.0000000000001cf4/head//3 61dce816/rbd_data.109e6b8b4567.0000000000001cf4/head//3(173'1064 client.4295.0:62464 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1064)
2014-06-12 14:51:10.125966 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  5d8ae816/rbd_data.10ae6b8b4567.0000000000001fe5/head//3 5d8ae816/rbd_data.10ae6b8b4567.0000000000001fe5/head//3(173'1093 client.4311.0:65593 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1093)
2014-06-12 14:51:10.125976 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  43f4e816/rbd_data.10b46b8b4567.00000000000034a3/head//3 43f4e816/rbd_data.10b46b8b4567.00000000000034a3/head//3(173'1930 client.4317.0:104512 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1930)
2014-06-12 14:51:10.125987 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  4e04e816/rbd_data.109e6b8b4567.0000000000002350/head//3 4e04e816/rbd_data.109e6b8b4567.0000000000002350/head//3(173'1268 client.4295.0:73785 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1268)
2014-06-12 14:51:10.125999 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  8093e816/rbd_data.10946b8b4567.0000000000001f6a/head//3 8093e816/rbd_data.10946b8b4567.0000000000001f6a/head//3(173'1146 client.4285.0:64862 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1146)
2014-06-12 14:51:10.126010 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  7d33e816/rbd_data.10986b8b4567.0000000000001672/head//3 7d33e816/rbd_data.10986b8b4567.0000000000001672/head//3(173'763 client.4289.0:49010 wrlock_by=unknown.0.0:0 dirty s 4194304 uv763)
2014-06-12 14:51:10.126023 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  8fcfd816/rbd_data.109d6b8b4567.00000000000011b9/head//3 8fcfd816/rbd_data.109d6b8b4567.00000000000011b9/head//3(173'635 client.4294.0:41300 wrlock_by=unknown.0.0:0 dirty s 4194304 uv635)
2014-06-12 14:51:10.126039 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  9fb3d816/rbd_data.10936b8b4567.0000000000001805/head//3 9fb3d816/rbd_data.10936b8b4567.0000000000001805/head//3(173'915 client.4284.0:50627 wrlock_by=unknown.0.0:0 dirty s 4194304 uv915)
2014-06-12 14:51:10.126052 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  4d82d816/rbd_data.108e6b8b4567.0000000000002bfd/head//3 4d82d816/rbd_data.108e6b8b4567.0000000000002bfd/head//3(173'1542 client.4279.0:88964 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1542)
2014-06-12 14:51:10.126064 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  172d816/rbd_data.10b16b8b4567.00000000000005a9/head//3 172d816/rbd_data.10b16b8b4567.00000000000005a9/head//3(173'2162 client.4314.0:126648 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2162)
2014-06-12 14:51:10.126076 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  7e00d816/rbd_data.10936b8b4567.0000000000002946/head//3 7e00d816/rbd_data.10936b8b4567.0000000000002946/head//3(173'1444 client.4284.0:81795 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1444)
2014-06-12 14:51:10.126089 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  d499c816/rbd_data.109f6b8b4567.0000000000002cd5/head//3 d499c816/rbd_data.109f6b8b4567.0000000000002cd5/head//3(173'1431 client.4296.0:88207 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1431)
2014-06-12 14:51:10.126101 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  54b7c816/rbd_data.10ad6b8b4567.0000000000001020/head//3 54b7c816/rbd_data.10ad6b8b4567.0000000000001020/head//3(173'612 client.4310.0:38239 wrlock_by=unknown.0.0:0 dirty s 4194304 uv612)
2014-06-12 14:51:10.126113 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  6271c816/rbd_data.10b26b8b4567.000000000000165e/head//3 6271c816/rbd_data.10b26b8b4567.000000000000165e/head//3(173'745 client.4315.0:49581 wrlock_by=unknown.0.0:0 dirty s 4194304 uv745)
2014-06-12 14:51:10.126125 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  aa60c816/rbd_data.10b26b8b4567.00000000000010e2/head//3 aa60c816/rbd_data.10b26b8b4567.00000000000010e2/head//3(173'605 client.4315.0:39746 wrlock_by=unknown.0.0:0 dirty s 4194304 uv605)
2014-06-12 14:51:10.126138 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  9dceb816/rbd_data.10986b8b4567.000000000000115f/head//3 9dceb816/rbd_data.10986b8b4567.000000000000115f/head//3(173'621 client.4289.0:39593 wrlock_by=unknown.0.0:0 dirty s 4194304 uv621)
2014-06-12 14:51:10.126150 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  159eb816/rbd_data.108d6b8b4567.0000000000001393/head//3 159eb816/rbd_data.108d6b8b4567.0000000000001393/head//3(173'664 client.4278.0:45952 wrlock_by=unknown.0.0:0 dirty s 4194304 uv664)
2014-06-12 14:51:10.126159 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:10.126170 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:10.126179 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:10.126180 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:10.126182 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:10.126216 7f9d60bd2700 10 collection_list_partial 3.16_head start:4aedf816//0//-1 is_max:0
2014-06-12 14:51:10.126301 7f9d60bd2700 20 objects: [c32ff816/rbd_data.10906b8b4567.0000000000002107/head//3,e56ff816/rbd_data.109a6b8b4567.000000000000371e/head//3]
2014-06-12 14:51:10.126644 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  requesting scrubmap from osd.22
2014-06-12 14:51:10.126666 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:10.126683 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:10.126691 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:10.126697 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:10.126703 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:10.126709 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_map_chunk [4aedf816//0//-1,MAX)
2014-06-12 14:51:10.126715 7f9d60bd2700 10 collection_list_partial 3.16_head start:4aedf816//0//-1 is_max:0
2014-06-12 14:51:10.126758 7f9d60bd2700 20 objects: [c32ff816/rbd_data.10906b8b4567.0000000000002107/head//3,e56ff816/rbd_data.109a6b8b4567.000000000000371e/head//3]
2014-06-12 14:51:10.126772 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scan_list scanning 2 objects
2014-06-12 14:51:10.126782 7f9d60bd2700 10 stat 3.16_head/c32ff816/rbd_data.10906b8b4567.0000000000002107/head//3
2014-06-12 14:51:10.126871 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid c32ff816/rbd_data.10906b8b4567.0000000000002107/head//3
2014-06-12 14:51:10.140376 7f9d60bd2700 10 getattrs 3.16_head/c32ff816/rbd_data.10906b8b4567.0000000000002107/head//3 = 0
2014-06-12 14:51:10.140385 7f9d60bd2700 10 stat 3.16_head/e56ff816/rbd_data.109a6b8b4567.000000000000371e/head//3
2014-06-12 14:51:10.140485 7f9d60bd2700 10 lookup_strip_header done  cid 3.16_head oid e56ff816/rbd_data.109a6b8b4567.000000000000371e/head//3
2014-06-12 14:51:10.161420 7f9d60bd2700 10 getattrs 3.16_head/e56ff816/rbd_data.109a6b8b4567.000000000000371e/head//3 = 0
2014-06-12 14:51:10.161432 7f9d60bd2700 10 collection_getattrs 3.16_head
2014-06-12 14:51:10.161472 7f9d60bd2700 10 collection_getattrs 3.16_head = 0
2014-06-12 14:51:10.161474 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:10.161493 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:10.161500 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:10.161506 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:10.161508 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:10.165007 7f9d58d5b700  5 -- op tracker -- , seq: 748567, time: 2014-06-12 14:51:10.164912, event: header_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.165025 7f9d58d5b700  5 -- op tracker -- , seq: 748567, time: 2014-06-12 14:51:10.164915, event: throttled, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.165037 7f9d58d5b700  5 -- op tracker -- , seq: 748567, time: 2014-06-12 14:51:10.164957, event: all_read, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.165047 7f9d58d5b700  5 -- op tracker -- , seq: 748567, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.165054 7f9d58d5b700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:10.165062 7f9d58d5b700 20 osd.2 173 should_share_map osd.22 172.16.96.202:6814/1927 173
2014-06-12 14:51:10.165071 7f9d58d5b700 15 osd.2 173 enqueue_op 0x2519cb00 prio 127 cost 896 latency 0.000158 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:10.165142 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:10.165166 7f9d633d7700 10 osd.2 173 dequeue_op 0x2519cb00 prio 127 cost 896 latency 0.000254 osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]
2014-06-12 14:51:10.165202 7f9d633d7700  5 -- op tracker -- , seq: 748567, time: 2014-06-12 14:51:10.165198, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.165210 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] handle_message: 0x2519cb00
2014-06-12 14:51:10.165218 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:10.165226 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:10.165235 7f9d633d7700  5 -- op tracker -- , seq: 748567, time: 2014-06-12 14:51:10.165233, event: started, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.165239 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  got 22 scrub map
2014-06-12 14:51:10.165253 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] map version is 173'2396
2014-06-12 14:51:10.165263 7f9d633d7700 10 osd.2 173 dequeue_op 0x2519cb00 finish
2014-06-12 14:51:10.165268 7f9d633d7700  5 -- op tracker -- , seq: 748567, time: 2014-06-12 14:51:10.165265, event: done, op: osd_sub_op(unknown.0.0:0 3.16 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:10.165278 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:10.165280 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:10.165291 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d0e000 (1 active)
2014-06-12 14:51:10.165298 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:10.165319 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:10.165333 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:10.165347 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:10.165359 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub   osd.2 has 2 items
2014-06-12 14:51:10.165371 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub replica 22 has 2 items
2014-06-12 14:51:10.165388 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c32ff816/rbd_data.10906b8b4567.0000000000002107/head//3, auth == maps.end()
2014-06-12 14:51:10.165432 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj c32ff816/rbd_data.10906b8b4567.0000000000002107/head//3
2014-06-12 14:51:10.165462 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e56ff816/rbd_data.109a6b8b4567.000000000000371e/head//3, auth == maps.end()
2014-06-12 14:51:10.165480 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] be_select_auth_object: selecting osd 22 for obj e56ff816/rbd_data.109a6b8b4567.000000000000371e/head//3
2014-06-12 14:51:10.165498 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] 
2014-06-12 14:51:10.165509 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub
2014-06-12 14:51:10.165530 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  e56ff816/rbd_data.109a6b8b4567.000000000000371e/head//3 e56ff816/rbd_data.109a6b8b4567.000000000000371e/head//3(173'1851 client.4291.0:107492 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1851)
2014-06-12 14:51:10.165554 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub  c32ff816/rbd_data.10906b8b4567.0000000000002107/head//3 c32ff816/rbd_data.10906b8b4567.0000000000002107/head//3(173'1160 client.4281.0:68436 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1160)
2014-06-12 14:51:10.165574 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:10.165587 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:10.165599 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub state FINISH
2014-06-12 14:51:10.165611 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] scrub got 252/262 objects, 0/0 clones, 252/262 dirty, 0/0 omap, 0/0 hit_set_archive, 1056964608/1098907648 bytes.
2014-06-12 14:51:10.165661 7f9d60bd2700  0 log [ERR] : 3.16 scrub stat mismatch, got 252/262 objects, 0/0 clones, 252/262 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts, 1056964608/1098907648 bytes.
2014-06-12 14:51:10.165665 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+scrubbing] process_inconsistent() checking authoritative
2014-06-12 14:51:10.165680 7f9d60bd2700  0 log [ERR] : 3.16 scrub 1 errors
2014-06-12 14:51:10.165748 7f9d60bd2700 10 log is not dirty
2014-06-12 14:51:10.165756 7f9d60bd2700  5 queue_transactions existing osr(3.16 0x276cd60)/0x276cd60
2014-06-12 14:51:10.165761 7f9d60bd2700  5 queue_transactions (trailing journal)  
2014-06-12 14:51:10.165762 7f9d60bd2700  5 queue_op 0x25e0540 seq 2191 osr(3.16 0x276cd60) 826 bytes   (queue has 1 ops and 826 bytes)
2014-06-12 14:51:10.165775 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+inconsistent] publish_stats_to_osd 173:3193
2014-06-12 14:51:10.165792 7f9d60bd2700 20 osd.2 173 dec_scrubs_active 1 -> 0 (max 1, pending 0)
2014-06-12 14:51:10.165795 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+inconsistent]  requeue_ops 
2014-06-12 14:51:10.165806 7f9d7a962700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x1c0a5a0 (1 active)
2014-06-12 14:51:10.165811 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+inconsistent] scrub requesting unreserve from osd.22
2014-06-12 14:51:10.165823 7f9d7a962700  5 _do_op 0x25e0540 seq 2191 osr(3.16 0x276cd60)/0x276cd60 start
2014-06-12 14:51:10.165829 7f9d7a962700 10 _do_transaction on 0x5139d40
2014-06-12 14:51:10.165835 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:10.165844 7f9d7a962700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:51:10.165878 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.16( v 173'2396 (0'0,173'2396] local-les=142 n=262 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2395 mlcod 173'2395 active+clean+inconsistent] share_pg_info
2014-06-12 14:51:10.165949 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:10.165973 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d0e000 (0 active)
2014-06-12 14:51:10.165975 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:10.166050 7f9d7a962700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:51:10.166096 7f9d7a962700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:51:10.166116 7f9d7a962700  5 submit_transaction r = 0
2014-06-12 14:51:10.166180 7f9d7a962700 10 _do_op 0x25e0540 seq 2191 r = 0, finisher 0x54ecaf0 0
2014-06-12 14:51:10.166190 7f9d7a962700 10 _finish_op 0x25e0540 seq 2191 osr(3.16 0x276cd60)/0x276cd60
2014-06-12 14:51:10.166199 7f9d7a962700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x1c0a5a0 (0 active)
2014-06-12 14:51:10.166203 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:10.292897 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:10.293116 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:10.315681 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:10.325167 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:10.365134 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:10.369932 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:10.380687 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:10.504106 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:10.511940 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:10.521775 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:10.526012 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:10.526395 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:10.531083 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:10.650293 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:10.660449 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:10.699477 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:10.766831 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:10.795172 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:10.800737 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:10.807168 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:10.853854 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:10.864221 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:10.867528 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:10.869846 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:10.869893 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.54 >= max 0.5 = no, load too high
2014-06-12 14:51:10.869907 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:51:10.869914 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:10.869910 7f9d84175700 10 osd.2 173 sched_scrub 3.2e high load at 2014-06-11 13:02:32.500084: 92918.4 < max (604800 seconds)
2014-06-12 14:51:10.869930 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:10.869933 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:10.869935 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:10.927255 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:10.934730 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:10.961081 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:10.964845 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:11.120160 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:11.131716 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:11.141177 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:11.150628 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:11.176829 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:11.179033 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:11.208194 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:11.208314 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:11.267489 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:11.267593 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:11.279322 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:11.279455 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:11.281787 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:11.284565 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:11.284585 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:11.285243 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:11.285311 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:11.285366 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:11.285436 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:11.285506 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:11.285552 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:11.285603 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:11.285670 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:11.285694 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:11.285720 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:11.285748 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:11.285772 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:11.285796 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:11.285820 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:11.285845 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:11.285869 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:11.285892 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:11.285918 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:11.285941 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:11.285965 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:11.285989 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:11.286015 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:11.297250 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:11.336091 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:11.340416 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:11.389025 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:11.512191 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:11.546123 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:11.780900 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:11.781322 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:11.854551 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:11.855053 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:11.870008 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:11.870032 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:11.870037 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:11.870039 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:11.870078 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:11.975832 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:12.015893 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:12.015980 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:12.063598 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:12.125329 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:12.165356 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:12.166046 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:12.166266 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:12.297170 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:12.297461 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:12.315779 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:12.325264 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:12.365206 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:12.370009 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:12.380785 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:12.504198 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:12.512029 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:12.521857 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:12.531179 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:12.650371 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:12.660536 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:12.699546 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:12.766911 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:12.795271 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:12.800820 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:12.807253 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:12.853937 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:12.864301 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:12.867628 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:12.870112 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:12.870132 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:51:12.870136 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:51:12.870145 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:51:12.870148 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:51:12.870187 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:12.870191 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:12.870192 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:12.870227 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:12.881670 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:12.881728 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:12.927352 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:12.934825 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:12.961178 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:12.964942 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:13.120235 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:13.131811 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:13.141262 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:13.150707 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:13.176910 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:13.179130 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:13.281883 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:13.297348 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:13.336168 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:13.340518 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:13.389124 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:13.512275 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:13.546208 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:13.671926 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:13.671930 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:13.671933 7f9d74bfa700 20 osd.2 173 _dispatch 0x602e700 pg_stats_ack(1 pgs tid 4868) v1
2014-06-12 14:51:13.671937 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:51:13.671944 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:13.671946 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:13.793414 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:13.793473 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:13.870270 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:13.870294 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:13.870299 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:13.870301 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:13.870329 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:13.975929 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:14.063693 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:14.090215 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:14.090455 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:14.125412 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:14.165440 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:14.166187 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:14.166196 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:14.166463 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:14.166497 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:14.185328 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:14.185348 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:14.186103 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:14.186174 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:14.186204 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:14.186235 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:14.186260 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:14.186283 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:14.186312 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:14.186408 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:14.186449 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:14.186475 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:14.186495 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:14.186520 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:14.186560 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:14.186590 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:14.186630 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:14.186658 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:14.186682 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:14.186706 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:14.186733 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:14.186760 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:14.186788 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:14.186812 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:14.315862 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:14.316642 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:14.316863 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:14.325360 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:14.365289 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:14.370084 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:14.380882 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:14.471945 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:14.472340 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:14.504279 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:14.512113 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:14.521935 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:14.531276 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:14.650452 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:14.660633 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:14.699655 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:14.709335 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:14.709437 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:14.766991 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:14.795369 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:14.800903 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:14.807333 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:14.854023 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:14.864381 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:14.867727 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:14.870376 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:14.870423 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.54 >= max 0.5 = no, load too high
2014-06-12 14:51:14.870438 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:51:14.870446 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:14.870441 7f9d84175700 10 osd.2 173 sched_scrub 3.2e high load at 2014-06-11 13:02:32.500084: 92922.4 < max (604800 seconds)
2014-06-12 14:51:14.870461 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:14.870464 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:14.870466 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:14.927434 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:14.934924 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:14.961276 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:14.965035 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:15.120308 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:15.131907 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:15.141350 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:15.150804 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:15.176993 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:15.179227 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:15.197963 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:15.198152 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:15.281959 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:15.297444 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:15.336246 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:15.340617 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:15.355318 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:15.355443 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:15.368194 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:15.368468 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:15.389205 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:15.512358 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:15.546293 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:15.866963 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:15.867065 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:15.870538 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:15.870562 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:15.870567 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:15.870569 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:15.870609 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:15.976024 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:16.063790 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:16.125485 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:16.165525 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:16.166272 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:16.166540 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:16.298661 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:16.298760 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:16.315960 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:16.325443 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:16.365362 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:16.370160 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:16.380975 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:16.426723 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:16.426966 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:16.504365 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:16.512197 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:16.522006 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:16.531362 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:16.580230 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:16.580393 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:16.650536 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:16.660732 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:16.699764 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:16.767071 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:16.795466 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:16.800987 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:16.807438 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:16.854122 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:16.864461 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:16.867826 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:16.870641 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:16.870663 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:16.870668 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:16.870670 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:16.870711 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:16.927530 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:16.935018 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:16.961373 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:16.965117 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:17.120384 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:17.132003 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:17.141448 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:17.150900 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:17.177084 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:17.179325 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:17.282058 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:17.297539 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:17.336322 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:17.340718 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:17.372741 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:17.372997 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:17.389287 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:17.512449 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:17.546391 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:17.817490 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:17.817565 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:17.870746 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:17.870766 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:51:17.870769 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:51:17.870773 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:51:17.870783 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:17.870787 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:17.870789 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:17.870825 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:17.976120 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:18.063887 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:18.125561 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:18.165630 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:18.166353 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:18.166621 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:18.316044 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:18.325526 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:18.365443 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:18.370240 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:18.381069 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:18.504448 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:18.512280 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:18.522077 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:18.531456 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:18.650623 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:18.660829 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:18.699840 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:18.767148 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:18.782436 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:18.782519 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:18.795566 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:18.801075 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:18.807537 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:18.854205 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:18.856111 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:18.856289 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:18.864559 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:18.867924 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:18.870858 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:18.870880 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:18.870884 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:18.870886 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:18.870907 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:18.927626 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:18.935115 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:18.961471 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:18.965213 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:19.120445 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:19.132098 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:19.141537 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:19.150978 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:19.177180 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:19.179420 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:19.282142 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:19.297636 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:19.336399 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:19.340817 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:19.367417 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:19.367698 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:19.389367 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:19.390880 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:19.390957 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:19.468675 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:19.468850 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:19.486167 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:19.486187 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:19.487036 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:19.487181 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:19.487217 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:19.487265 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:19.487313 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:19.487342 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:19.487379 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:19.487407 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:19.487430 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:19.487453 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:19.487477 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:19.487501 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:19.487525 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:19.487549 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:19.487573 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:19.487597 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:19.487620 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:19.487644 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:19.487676 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:19.487703 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:19.487728 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:19.487755 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:19.512535 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:19.518518 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:19.518603 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:19.546476 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:19.694149 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:19.694235 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:19.870961 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:19.870988 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:19.870995 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:19.870999 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:19.871026 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:19.927503 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:19.927975 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:19.976201 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:20.009651 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:20.009706 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:20.063983 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:20.125633 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:20.165712 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:20.166429 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:20.166704 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:20.316118 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:20.325629 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:20.365534 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:20.370318 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:20.381152 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:20.504531 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:20.512362 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:20.522150 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:20.531552 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:20.650704 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:20.660923 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:20.699912 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:20.767231 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:20.795663 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:20.801169 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:20.807620 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:20.854292 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:20.864643 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:20.868022 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:20.871077 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:20.871102 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:20.871107 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:20.871109 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:20.871149 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:20.873437 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:20.873680 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:20.927723 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:20.935209 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:20.961568 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:20.965296 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:21.120537 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:21.132193 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:21.141651 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:21.151070 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:21.177277 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:21.179514 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:21.282241 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:21.297713 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:21.336483 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:21.340910 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:21.374201 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:21.374292 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:21.389447 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:21.512619 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:21.546560 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:21.871178 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:21.871223 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.46 < max 0.5 = yes
2014-06-12 14:51:21.871236 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=1
2014-06-12 14:51:21.871242 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2e at 2014-06-11 13:02:32.500084
2014-06-12 14:51:21.871248 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:21.871253 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 -> 1 (max 1, active 0)
2014-06-12 14:51:21.871257 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] sched_scrub: reserved locally, reserving replicas
2014-06-12 14:51:21.871279 7f9d84175700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] scrub requesting reserve from osd.21
2014-06-12 14:51:21.871298 7f9d84175700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:21.871325 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] sched_scrub: reserved 2, waiting for replicas
2014-06-12 14:51:21.871340 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:21.871342 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:21.871344 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:21.872214 7f9d44101700  5 -- op tracker -- , seq: 748568, time: 2014-06-12 14:51:21.872145, event: header_read, op: osd_sub_op_reply(unknown.0.0:0 3.2e 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:21.872243 7f9d44101700  5 -- op tracker -- , seq: 748568, time: 2014-06-12 14:51:21.872147, event: throttled, op: osd_sub_op_reply(unknown.0.0:0 3.2e 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:21.872261 7f9d44101700  5 -- op tracker -- , seq: 748568, time: 2014-06-12 14:51:21.872167, event: all_read, op: osd_sub_op_reply(unknown.0.0:0 3.2e 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:21.872279 7f9d44101700  5 -- op tracker -- , seq: 748568, time: 0.000000, event: dispatched, op: osd_sub_op_reply(unknown.0.0:0 3.2e 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:21.872288 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(unknown.0.0:0 3.2e 0//0//3 [scrub-reserve] ack, result = 0) v2 epoch 173
2014-06-12 14:51:21.872300 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:21.872314 7f9d44101700 15 osd.2 173 enqueue_op 0x31d61e00 prio 127 cost 1 latency 0.000169 osd_sub_op_reply(unknown.0.0:0 3.2e 0//0//3 [scrub-reserve] ack, result = 0) v2
2014-06-12 14:51:21.872390 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:21.872412 7f9d61bd4700 10 osd.2 173 dequeue_op 0x31d61e00 prio 127 cost 1 latency 0.000267 osd_sub_op_reply(unknown.0.0:0 3.2e 0//0//3 [scrub-reserve] ack, result = 0) v2 pg pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]
2014-06-12 14:51:21.872451 7f9d61bd4700  5 -- op tracker -- , seq: 748568, time: 2014-06-12 14:51:21.872445, event: reached_pg, op: osd_sub_op_reply(unknown.0.0:0 3.2e 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:21.872467 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] handle_message: 0x31d61e00
2014-06-12 14:51:21.872482 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] sub_op_scrub_reserve_reply
2014-06-12 14:51:21.872499 7f9d61bd4700  5 -- op tracker -- , seq: 748568, time: 2014-06-12 14:51:21.872494, event: started, op: osd_sub_op_reply(unknown.0.0:0 3.2e 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:21.872507 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean]  osd.21 scrub reserve = success
2014-06-12 14:51:21.872520 7f9d61bd4700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean] sched_scrub: success, reserved self and replicas
2014-06-12 14:51:21.872537 7f9d61bd4700 10 osd.2 173 dequeue_op 0x31d61e00 finish
2014-06-12 14:51:21.872545 7f9d61bd4700  5 -- op tracker -- , seq: 748568, time: 2014-06-12 14:51:21.872541, event: done, op: osd_sub_op_reply(unknown.0.0:0 3.2e 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:21.872563 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:21.872568 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:21.872575 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:21.872593 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] starting a new chunky scrub
2014-06-12 14:51:21.872628 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state INACTIVE
2014-06-12 14:51:21.872642 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub start
2014-06-12 14:51:21.872656 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] publish_stats_to_osd 173:3612
2014-06-12 14:51:21.872669 7f9d60bd2700 20 osd.2 173 inc_scrubs_active 0 -> 1 (max 1, pending 1 -> 0)
2014-06-12 14:51:21.872674 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:21.872690 7f9d60bd2700 10 collection_list_partial 3.2e_head start:0//0//-1 is_max:0
2014-06-12 14:51:21.873489 7f9d60bd2700 20 objects: [4b85002e/rbd_data.10a76b8b4567.000000000000362a/head//3,d21b002e/rbd_data.10ab6b8b4567.000000000000096b/head//3,dffb002e/rbd_data.10b06b8b4567.0000000000000eda/head//3,8c1d002e/rbd_data.108d6b8b4567.00000000000016ce/head//3,1efd002e/rbd_data.10a86b8b4567.00000000000009ea/head//3,e664102e/rbd_data.10916b8b4567.0000000000002c71/head//3,f6f7102e/rbd_data.10936b8b4567.000000000000183b/head//3,7368102e/rbd_data.10956b8b4567.0000000000002192/head//3,cd5a102e/rbd_data.108d6b8b4567.00000000000038f6/head//3,f6fb102e/rbd_data.10ab6b8b4567.0000000000003b8a/head//3,6a1d102e/rbd_data.10b36b8b4567.00000000000034a7/head//3,1d0f102e/rbd_data.10a76b8b4567.0000000000002894/head//3,f19f102e/rbd_data.10906b8b4567.0000000000001269/head//3,c376202e/rbd_data.108d6b8b4567.0000000000003598/head//3,a0b6202e/rbd_data.10a26b8b4567.0000000000001db2/head//3,4dc6202e/rbd_data.10a46b8b4567.0000000000002658/head//3,aa39202e/rbd_data.109d6b8b4567.0000000000002430/head//3,14ca202e/rbd_data.108e6b8b4567.0000000000003899/head//3,475b202e/rbd_data.108d6b8b4567.0000000000000317/head//3,d40c202e/rbd_data.10b16b8b4567.0000000000001967/head//3,6f3d202e/rbd_data.10916b8b4567.0000000000003b2f/head//3,1a5d202e/rbd_data.10936b8b4567.0000000000003744/head//3,6acd202e/rbd_data.10ae6b8b4567.00000000000002da/head//3,8a3f202e/rbd_data.10996b8b4567.000000000000199d/head//3,8d72302e/rbd_data.10a26b8b4567.0000000000001fa7/head//3]
2014-06-12 14:51:21.874278 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:21.874324 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:21.874351 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:21.874365 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:21.874376 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:21.874386 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:21.874396 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_map_chunk [0//0//-1,5423302e//0//-1)
2014-06-12 14:51:21.874409 7f9d60bd2700 10 collection_list_partial 3.2e_head start:0//0//-1 is_max:0
2014-06-12 14:51:21.875514 7f9d60bd2700 20 objects: [4b85002e/rbd_data.10a76b8b4567.000000000000362a/head//3,d21b002e/rbd_data.10ab6b8b4567.000000000000096b/head//3,dffb002e/rbd_data.10b06b8b4567.0000000000000eda/head//3,8c1d002e/rbd_data.108d6b8b4567.00000000000016ce/head//3,1efd002e/rbd_data.10a86b8b4567.00000000000009ea/head//3,e664102e/rbd_data.10916b8b4567.0000000000002c71/head//3,f6f7102e/rbd_data.10936b8b4567.000000000000183b/head//3,7368102e/rbd_data.10956b8b4567.0000000000002192/head//3,cd5a102e/rbd_data.108d6b8b4567.00000000000038f6/head//3,f6fb102e/rbd_data.10ab6b8b4567.0000000000003b8a/head//3,6a1d102e/rbd_data.10b36b8b4567.00000000000034a7/head//3,1d0f102e/rbd_data.10a76b8b4567.0000000000002894/head//3,f19f102e/rbd_data.10906b8b4567.0000000000001269/head//3,c376202e/rbd_data.108d6b8b4567.0000000000003598/head//3,a0b6202e/rbd_data.10a26b8b4567.0000000000001db2/head//3,4dc6202e/rbd_data.10a46b8b4567.0000000000002658/head//3,aa39202e/rbd_data.109d6b8b4567.0000000000002430/head//3,14ca202e/rbd_data.108e6b8b4567.0000000000003899/head//3,475b202e/rbd_data.108d6b8b4567.0000000000000317/head//3,d40c202e/rbd_data.10b16b8b4567.0000000000001967/head//3,6f3d202e/rbd_data.10916b8b4567.0000000000003b2f/head//3,1a5d202e/rbd_data.10936b8b4567.0000000000003744/head//3,6acd202e/rbd_data.10ae6b8b4567.00000000000002da/head//3,8a3f202e/rbd_data.10996b8b4567.000000000000199d/head//3,8d72302e/rbd_data.10a26b8b4567.0000000000001fa7/head//3,5423302e/rbd_data.10a06b8b4567.0000000000002ec8/head//3,dd44302e/rbd_data.109a6b8b4567.000000000000144d/head//3,f1d4302e/rbd_data.109a6b8b4567.0000000000000b5f/head//3,e475302e/rbd_data.10ae6b8b4567.0000000000001d21/head//3,8489302e/rbd_data.10996b8b4567.00000000000022bf/head//3,10cd302e/rbd_data.108e6b8b4567.0000000000001ff8/head//3,408f302e/rbd_data.109d6b8b4567.0000000000000c77/head//3,69d0402e/rbd_data.10976b8b4567.0000000000000aeb/head//3,55d1402e/rbd_data.108d6b8b4567.0000000000000c98/head//3,ca03402e/rbd_data.10916b8b4567.0000000000002309/head//3,344402e/rbd_data.10a66b8b4567.0000000000001830/head//3,6d15402e/rbd_data.109f6b8b4567.000000000000080d/head//3,c95402e/rbd_data.10a16b8b4567.0000000000002751/head//3,13b8402e/rbd_data.10ae6b8b4567.0000000000000b0e/head//3,4a69402e/rbd_data.109e6b8b4567.00000000000031ff/head//3,477a402e/rbd_data.109a6b8b4567.0000000000000138/head//3,181b402e/rbd_data.10b46b8b4567.0000000000001e8e/head//3,667b402e/rbd_data.10af6b8b4567.000000000000083f/head//3,c42d402e/rbd_data.10ab6b8b4567.000000000000232e/head//3,b26d402e/rbd_data.108f6b8b4567.000000000000347c/head//3,40ce402e/rbd_data.10966b8b4567.0000000000002ee3/head//3,6b2502e/rbd_data.10986b8b4567.0000000000002e75/head//3,4a54502e/rbd_data.10ae6b8b4567.00000000000003d8/head//3,38e5502e/rbd_data.10aa6b8b4567.0000000000002ea3/head//3,d828502e/rbd_data.109d6b8b4567.0000000000000001/head//3,7e48502e/rbd_data.10b26b8b4567.00000000000018ea/head//3,2699502e/rbd_data.10a26b8b4567.0000000000000bd7/head//3,301b502e/rbd_data.10a96b8b4567.0000000000000366/head//3,f75b502e/rbd_data.10a76b8b4567.0000000000002e58/head//3,d21c502e/rbd_data.109c6b8b4567.00000000000001a8/head//3,76ec502e/rbd_data.10a66b8b4567.0000000000000944/head//3,fe8e502e/rbd_data.10916b8b4567.00000000000028b2/head//3,4581602e/rbd_data.10a36b8b4567.0000000000002483/head//3,4744602e/rbd_data.10a26b8b4567.0000000000000a7e/head//3,7ac4602e/rbd_data.109b6b8b4567.0000000000002337/head//3,ba35602e/rbd_data.10986b8b4567.000000000000064b/head//3,ee35602e/rbd_data.10986b8b4567.00000000000038de/head//3,7656602e/rbd_data.10a46b8b4567.0000000000002ed3/head//3,2a57602e/rbd_data.10a56b8b4567.0000000000003a3c/head//3]
2014-06-12 14:51:21.875647 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:21.875676 7f9d60bd2700 10 stat 3.2e_head/4b85002e/rbd_data.10a76b8b4567.000000000000362a/head//3
2014-06-12 14:51:21.875897 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4b85002e/rbd_data.10a76b8b4567.000000000000362a/head//3
2014-06-12 14:51:21.881171 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:21.881224 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:21.910161 7f9d60bd2700 10 getattrs 3.2e_head/4b85002e/rbd_data.10a76b8b4567.000000000000362a/head//3 = 0
2014-06-12 14:51:21.910173 7f9d60bd2700 10 stat 3.2e_head/d21b002e/rbd_data.10ab6b8b4567.000000000000096b/head//3
2014-06-12 14:51:21.910265 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d21b002e/rbd_data.10ab6b8b4567.000000000000096b/head//3
2014-06-12 14:51:21.953069 7f9d60bd2700 10 getattrs 3.2e_head/d21b002e/rbd_data.10ab6b8b4567.000000000000096b/head//3 = 0
2014-06-12 14:51:21.953079 7f9d60bd2700 10 stat 3.2e_head/dffb002e/rbd_data.10b06b8b4567.0000000000000eda/head//3
2014-06-12 14:51:21.953184 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid dffb002e/rbd_data.10b06b8b4567.0000000000000eda/head//3
2014-06-12 14:51:21.963354 7f9d60bd2700 10 getattrs 3.2e_head/dffb002e/rbd_data.10b06b8b4567.0000000000000eda/head//3 = 0
2014-06-12 14:51:21.963364 7f9d60bd2700 10 stat 3.2e_head/8c1d002e/rbd_data.108d6b8b4567.00000000000016ce/head//3
2014-06-12 14:51:21.963471 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8c1d002e/rbd_data.108d6b8b4567.00000000000016ce/head//3
2014-06-12 14:51:21.976291 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:22.002934 7f9d60bd2700 10 getattrs 3.2e_head/8c1d002e/rbd_data.108d6b8b4567.00000000000016ce/head//3 = 0
2014-06-12 14:51:22.002954 7f9d60bd2700 10 stat 3.2e_head/1efd002e/rbd_data.10a86b8b4567.00000000000009ea/head//3
2014-06-12 14:51:22.003122 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 1efd002e/rbd_data.10a86b8b4567.00000000000009ea/head//3
2014-06-12 14:51:22.039540 7f9d60bd2700 10 getattrs 3.2e_head/1efd002e/rbd_data.10a86b8b4567.00000000000009ea/head//3 = 0
2014-06-12 14:51:22.039571 7f9d60bd2700 10 stat 3.2e_head/e664102e/rbd_data.10916b8b4567.0000000000002c71/head//3
2014-06-12 14:51:22.039750 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid e664102e/rbd_data.10916b8b4567.0000000000002c71/head//3
2014-06-12 14:51:22.057710 7f9d60bd2700 10 getattrs 3.2e_head/e664102e/rbd_data.10916b8b4567.0000000000002c71/head//3 = 0
2014-06-12 14:51:22.057717 7f9d60bd2700 10 stat 3.2e_head/f6f7102e/rbd_data.10936b8b4567.000000000000183b/head//3
2014-06-12 14:51:22.057808 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f6f7102e/rbd_data.10936b8b4567.000000000000183b/head//3
2014-06-12 14:51:22.064056 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:22.072716 7f9d60bd2700 10 getattrs 3.2e_head/f6f7102e/rbd_data.10936b8b4567.000000000000183b/head//3 = 0
2014-06-12 14:51:22.072734 7f9d60bd2700 10 stat 3.2e_head/7368102e/rbd_data.10956b8b4567.0000000000002192/head//3
2014-06-12 14:51:22.072926 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 7368102e/rbd_data.10956b8b4567.0000000000002192/head//3
2014-06-12 14:51:22.091459 7f9d60bd2700 10 getattrs 3.2e_head/7368102e/rbd_data.10956b8b4567.0000000000002192/head//3 = 0
2014-06-12 14:51:22.091479 7f9d60bd2700 10 stat 3.2e_head/cd5a102e/rbd_data.108d6b8b4567.00000000000038f6/head//3
2014-06-12 14:51:22.091681 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid cd5a102e/rbd_data.108d6b8b4567.00000000000038f6/head//3
2014-06-12 14:51:22.126046 7f9d60bd2700 10 getattrs 3.2e_head/cd5a102e/rbd_data.108d6b8b4567.00000000000038f6/head//3 = 0
2014-06-12 14:51:22.126054 7f9d60bd2700 10 stat 3.2e_head/f6fb102e/rbd_data.10ab6b8b4567.0000000000003b8a/head//3
2014-06-12 14:51:22.126145 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f6fb102e/rbd_data.10ab6b8b4567.0000000000003b8a/head//3
2014-06-12 14:51:22.143151 7f9d60bd2700 10 getattrs 3.2e_head/f6fb102e/rbd_data.10ab6b8b4567.0000000000003b8a/head//3 = 0
2014-06-12 14:51:22.143158 7f9d60bd2700 10 stat 3.2e_head/6a1d102e/rbd_data.10b36b8b4567.00000000000034a7/head//3
2014-06-12 14:51:22.143250 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6a1d102e/rbd_data.10b36b8b4567.00000000000034a7/head//3
2014-06-12 14:51:22.156518 7f9d60bd2700 10 getattrs 3.2e_head/6a1d102e/rbd_data.10b36b8b4567.00000000000034a7/head//3 = 0
2014-06-12 14:51:22.156527 7f9d60bd2700 10 stat 3.2e_head/1d0f102e/rbd_data.10a76b8b4567.0000000000002894/head//3
2014-06-12 14:51:22.156630 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 1d0f102e/rbd_data.10a76b8b4567.0000000000002894/head//3
2014-06-12 14:51:22.165787 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:22.166785 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:22.170259 7f9d60bd2700 10 getattrs 3.2e_head/1d0f102e/rbd_data.10a76b8b4567.0000000000002894/head//3 = 0
2014-06-12 14:51:22.170269 7f9d60bd2700 10 stat 3.2e_head/f19f102e/rbd_data.10906b8b4567.0000000000001269/head//3
2014-06-12 14:51:22.170384 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f19f102e/rbd_data.10906b8b4567.0000000000001269/head//3
2014-06-12 14:51:22.199238 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:22.199421 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:22.199792 7f9d60bd2700 10 getattrs 3.2e_head/f19f102e/rbd_data.10906b8b4567.0000000000001269/head//3 = 0
2014-06-12 14:51:22.199812 7f9d60bd2700 10 stat 3.2e_head/c376202e/rbd_data.108d6b8b4567.0000000000003598/head//3
2014-06-12 14:51:22.199913 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid c376202e/rbd_data.108d6b8b4567.0000000000003598/head//3
2014-06-12 14:51:22.214973 7f9d60bd2700 10 getattrs 3.2e_head/c376202e/rbd_data.108d6b8b4567.0000000000003598/head//3 = 0
2014-06-12 14:51:22.214983 7f9d60bd2700 10 stat 3.2e_head/a0b6202e/rbd_data.10a26b8b4567.0000000000001db2/head//3
2014-06-12 14:51:22.215085 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a0b6202e/rbd_data.10a26b8b4567.0000000000001db2/head//3
2014-06-12 14:51:22.231342 7f9d60bd2700 10 getattrs 3.2e_head/a0b6202e/rbd_data.10a26b8b4567.0000000000001db2/head//3 = 0
2014-06-12 14:51:22.231352 7f9d60bd2700 10 stat 3.2e_head/4dc6202e/rbd_data.10a46b8b4567.0000000000002658/head//3
2014-06-12 14:51:22.231467 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4dc6202e/rbd_data.10a46b8b4567.0000000000002658/head//3
2014-06-12 14:51:22.255865 7f9d60bd2700 10 getattrs 3.2e_head/4dc6202e/rbd_data.10a46b8b4567.0000000000002658/head//3 = 0
2014-06-12 14:51:22.255878 7f9d60bd2700 10 stat 3.2e_head/aa39202e/rbd_data.109d6b8b4567.0000000000002430/head//3
2014-06-12 14:51:22.255980 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid aa39202e/rbd_data.109d6b8b4567.0000000000002430/head//3
2014-06-12 14:51:22.286239 7f9d60bd2700 10 getattrs 3.2e_head/aa39202e/rbd_data.109d6b8b4567.0000000000002430/head//3 = 0
2014-06-12 14:51:22.286250 7f9d60bd2700 10 stat 3.2e_head/14ca202e/rbd_data.108e6b8b4567.0000000000003899/head//3
2014-06-12 14:51:22.286367 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 14ca202e/rbd_data.108e6b8b4567.0000000000003899/head//3
2014-06-12 14:51:22.307989 7f9d60bd2700 10 getattrs 3.2e_head/14ca202e/rbd_data.108e6b8b4567.0000000000003899/head//3 = 0
2014-06-12 14:51:22.308001 7f9d60bd2700 10 stat 3.2e_head/475b202e/rbd_data.108d6b8b4567.0000000000000317/head//3
2014-06-12 14:51:22.308110 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 475b202e/rbd_data.108d6b8b4567.0000000000000317/head//3
2014-06-12 14:51:22.316199 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:22.325710 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:22.360515 7f9d60bd2700 10 getattrs 3.2e_head/475b202e/rbd_data.108d6b8b4567.0000000000000317/head//3 = 0
2014-06-12 14:51:22.360539 7f9d60bd2700 10 stat 3.2e_head/d40c202e/rbd_data.10b16b8b4567.0000000000001967/head//3
2014-06-12 14:51:22.360701 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d40c202e/rbd_data.10b16b8b4567.0000000000001967/head//3
2014-06-12 14:51:22.365633 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:22.370419 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:22.381237 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:22.387121 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:22.387140 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:22.387359 7f9d60bd2700 10 getattrs 3.2e_head/d40c202e/rbd_data.10b16b8b4567.0000000000001967/head//3 = 0
2014-06-12 14:51:22.387379 7f9d60bd2700 10 stat 3.2e_head/6f3d202e/rbd_data.10916b8b4567.0000000000003b2f/head//3
2014-06-12 14:51:22.387712 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6f3d202e/rbd_data.10916b8b4567.0000000000003b2f/head//3
2014-06-12 14:51:22.387885 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:22.388013 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:22.388046 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:22.388065 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:22.388085 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:22.388105 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:22.388126 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:22.388157 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:22.388192 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:22.388212 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:22.388233 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:22.388253 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:22.388277 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:22.388297 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:22.388317 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:22.388341 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:22.388363 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:22.388382 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:22.388402 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:22.388422 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:22.388442 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:22.388465 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:22.405356 7f9d60bd2700 10 getattrs 3.2e_head/6f3d202e/rbd_data.10916b8b4567.0000000000003b2f/head//3 = 0
2014-06-12 14:51:22.405372 7f9d60bd2700 10 stat 3.2e_head/1a5d202e/rbd_data.10936b8b4567.0000000000003744/head//3
2014-06-12 14:51:22.405547 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 1a5d202e/rbd_data.10936b8b4567.0000000000003744/head//3
2014-06-12 14:51:22.423935 7f9d60bd2700 10 getattrs 3.2e_head/1a5d202e/rbd_data.10936b8b4567.0000000000003744/head//3 = 0
2014-06-12 14:51:22.423955 7f9d60bd2700 10 stat 3.2e_head/6acd202e/rbd_data.10ae6b8b4567.00000000000002da/head//3
2014-06-12 14:51:22.424141 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6acd202e/rbd_data.10ae6b8b4567.00000000000002da/head//3
2014-06-12 14:51:22.455083 7f9d60bd2700 10 getattrs 3.2e_head/6acd202e/rbd_data.10ae6b8b4567.00000000000002da/head//3 = 0
2014-06-12 14:51:22.455092 7f9d60bd2700 10 stat 3.2e_head/8a3f202e/rbd_data.10996b8b4567.000000000000199d/head//3
2014-06-12 14:51:22.455189 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8a3f202e/rbd_data.10996b8b4567.000000000000199d/head//3
2014-06-12 14:51:22.471265 7f9d60bd2700 10 getattrs 3.2e_head/8a3f202e/rbd_data.10996b8b4567.000000000000199d/head//3 = 0
2014-06-12 14:51:22.471273 7f9d60bd2700 10 stat 3.2e_head/8d72302e/rbd_data.10a26b8b4567.0000000000001fa7/head//3
2014-06-12 14:51:22.471362 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8d72302e/rbd_data.10a26b8b4567.0000000000001fa7/head//3
2014-06-12 14:51:22.487772 7f9d60bd2700 10 getattrs 3.2e_head/8d72302e/rbd_data.10a26b8b4567.0000000000001fa7/head//3 = 0
2014-06-12 14:51:22.487781 7f9d60bd2700 10 collection_getattrs 3.2e_head
2014-06-12 14:51:22.487846 7f9d60bd2700 10 collection_getattrs 3.2e_head = 0
2014-06-12 14:51:22.487850 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:22.487876 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:22.487882 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:22.487888 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:22.487891 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:22.504617 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:22.512452 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:22.517660 7f9d44101700  5 -- op tracker -- , seq: 748569, time: 2014-06-12 14:51:22.517572, event: header_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:22.517693 7f9d44101700  5 -- op tracker -- , seq: 748569, time: 2014-06-12 14:51:22.517582, event: throttled, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:22.517713 7f9d44101700  5 -- op tracker -- , seq: 748569, time: 2014-06-12 14:51:22.517631, event: all_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:22.517732 7f9d44101700  5 -- op tracker -- , seq: 748569, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:22.517743 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:22.517757 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:22.517773 7f9d44101700 15 osd.2 173 enqueue_op 0x31d61d00 prio 127 cost 10717 latency 0.000201 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:22.517833 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:22.517860 7f9d62bd6700 10 osd.2 173 dequeue_op 0x31d61d00 prio 127 cost 10717 latency 0.000287 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]
2014-06-12 14:51:22.517917 7f9d62bd6700  5 -- op tracker -- , seq: 748569, time: 2014-06-12 14:51:22.517907, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:22.517939 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] handle_message: 0x31d61d00
2014-06-12 14:51:22.517963 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:22.517989 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:22.518017 7f9d62bd6700  5 -- op tracker -- , seq: 748569, time: 2014-06-12 14:51:22.518008, event: started, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:22.518040 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:22.518196 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] map version is 173'2847
2014-06-12 14:51:22.518228 7f9d62bd6700 10 osd.2 173 dequeue_op 0x31d61d00 finish
2014-06-12 14:51:22.518245 7f9d62bd6700  5 -- op tracker -- , seq: 748569, time: 2014-06-12 14:51:22.518235, event: done, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:22.518259 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:22.518276 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:22.518283 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:22.518275 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:22.518307 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:22.518331 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:22.518391 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:22.518415 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:22.518438 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:22.518497 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4b85002e/rbd_data.10a76b8b4567.000000000000362a/head//3, auth == maps.end()
2014-06-12 14:51:22.518565 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4b85002e/rbd_data.10a76b8b4567.000000000000362a/head//3
2014-06-12 14:51:22.518612 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d21b002e/rbd_data.10ab6b8b4567.000000000000096b/head//3, auth == maps.end()
2014-06-12 14:51:22.518650 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d21b002e/rbd_data.10ab6b8b4567.000000000000096b/head//3
2014-06-12 14:51:22.518689 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dffb002e/rbd_data.10b06b8b4567.0000000000000eda/head//3, auth == maps.end()
2014-06-12 14:51:22.518724 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj dffb002e/rbd_data.10b06b8b4567.0000000000000eda/head//3
2014-06-12 14:51:22.518763 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8c1d002e/rbd_data.108d6b8b4567.00000000000016ce/head//3, auth == maps.end()
2014-06-12 14:51:22.518799 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8c1d002e/rbd_data.108d6b8b4567.00000000000016ce/head//3
2014-06-12 14:51:22.518831 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1efd002e/rbd_data.10a86b8b4567.00000000000009ea/head//3, auth == maps.end()
2014-06-12 14:51:22.518865 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1efd002e/rbd_data.10a86b8b4567.00000000000009ea/head//3
2014-06-12 14:51:22.518898 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e664102e/rbd_data.10916b8b4567.0000000000002c71/head//3, auth == maps.end()
2014-06-12 14:51:22.518931 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e664102e/rbd_data.10916b8b4567.0000000000002c71/head//3
2014-06-12 14:51:22.518968 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f6f7102e/rbd_data.10936b8b4567.000000000000183b/head//3, auth == maps.end()
2014-06-12 14:51:22.519003 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f6f7102e/rbd_data.10936b8b4567.000000000000183b/head//3
2014-06-12 14:51:22.519040 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7368102e/rbd_data.10956b8b4567.0000000000002192/head//3, auth == maps.end()
2014-06-12 14:51:22.519075 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7368102e/rbd_data.10956b8b4567.0000000000002192/head//3
2014-06-12 14:51:22.519114 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cd5a102e/rbd_data.108d6b8b4567.00000000000038f6/head//3, auth == maps.end()
2014-06-12 14:51:22.519150 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj cd5a102e/rbd_data.108d6b8b4567.00000000000038f6/head//3
2014-06-12 14:51:22.519187 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f6fb102e/rbd_data.10ab6b8b4567.0000000000003b8a/head//3, auth == maps.end()
2014-06-12 14:51:22.519222 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f6fb102e/rbd_data.10ab6b8b4567.0000000000003b8a/head//3
2014-06-12 14:51:22.519256 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6a1d102e/rbd_data.10b36b8b4567.00000000000034a7/head//3, auth == maps.end()
2014-06-12 14:51:22.519289 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6a1d102e/rbd_data.10b36b8b4567.00000000000034a7/head//3
2014-06-12 14:51:22.519327 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1d0f102e/rbd_data.10a76b8b4567.0000000000002894/head//3, auth == maps.end()
2014-06-12 14:51:22.519363 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1d0f102e/rbd_data.10a76b8b4567.0000000000002894/head//3
2014-06-12 14:51:22.519400 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f19f102e/rbd_data.10906b8b4567.0000000000001269/head//3, auth == maps.end()
2014-06-12 14:51:22.519436 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f19f102e/rbd_data.10906b8b4567.0000000000001269/head//3
2014-06-12 14:51:22.519473 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c376202e/rbd_data.108d6b8b4567.0000000000003598/head//3, auth == maps.end()
2014-06-12 14:51:22.519509 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c376202e/rbd_data.108d6b8b4567.0000000000003598/head//3
2014-06-12 14:51:22.519545 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a0b6202e/rbd_data.10a26b8b4567.0000000000001db2/head//3, auth == maps.end()
2014-06-12 14:51:22.519581 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a0b6202e/rbd_data.10a26b8b4567.0000000000001db2/head//3
2014-06-12 14:51:22.519618 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4dc6202e/rbd_data.10a46b8b4567.0000000000002658/head//3, auth == maps.end()
2014-06-12 14:51:22.519653 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4dc6202e/rbd_data.10a46b8b4567.0000000000002658/head//3
2014-06-12 14:51:22.519686 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aa39202e/rbd_data.109d6b8b4567.0000000000002430/head//3, auth == maps.end()
2014-06-12 14:51:22.519727 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj aa39202e/rbd_data.109d6b8b4567.0000000000002430/head//3
2014-06-12 14:51:22.519765 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 14ca202e/rbd_data.108e6b8b4567.0000000000003899/head//3, auth == maps.end()
2014-06-12 14:51:22.519799 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 14ca202e/rbd_data.108e6b8b4567.0000000000003899/head//3
2014-06-12 14:51:22.519838 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 475b202e/rbd_data.108d6b8b4567.0000000000000317/head//3, auth == maps.end()
2014-06-12 14:51:22.519873 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 475b202e/rbd_data.108d6b8b4567.0000000000000317/head//3
2014-06-12 14:51:22.519910 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d40c202e/rbd_data.10b16b8b4567.0000000000001967/head//3, auth == maps.end()
2014-06-12 14:51:22.519945 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d40c202e/rbd_data.10b16b8b4567.0000000000001967/head//3
2014-06-12 14:51:22.519983 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6f3d202e/rbd_data.10916b8b4567.0000000000003b2f/head//3, auth == maps.end()
2014-06-12 14:51:22.520018 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6f3d202e/rbd_data.10916b8b4567.0000000000003b2f/head//3
2014-06-12 14:51:22.520056 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1a5d202e/rbd_data.10936b8b4567.0000000000003744/head//3, auth == maps.end()
2014-06-12 14:51:22.520093 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1a5d202e/rbd_data.10936b8b4567.0000000000003744/head//3
2014-06-12 14:51:22.520126 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6acd202e/rbd_data.10ae6b8b4567.00000000000002da/head//3, auth == maps.end()
2014-06-12 14:51:22.520160 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6acd202e/rbd_data.10ae6b8b4567.00000000000002da/head//3
2014-06-12 14:51:22.520197 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8a3f202e/rbd_data.10996b8b4567.000000000000199d/head//3, auth == maps.end()
2014-06-12 14:51:22.520233 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8a3f202e/rbd_data.10996b8b4567.000000000000199d/head//3
2014-06-12 14:51:22.520270 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8d72302e/rbd_data.10a26b8b4567.0000000000001fa7/head//3, auth == maps.end()
2014-06-12 14:51:22.520305 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8d72302e/rbd_data.10a26b8b4567.0000000000001fa7/head//3
2014-06-12 14:51:22.520346 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] 
2014-06-12 14:51:22.520374 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub
2014-06-12 14:51:22.520413 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8d72302e/rbd_data.10a26b8b4567.0000000000001fa7/head//3 8d72302e/rbd_data.10a26b8b4567.0000000000001fa7/head//3(173'1421 client.4299.0:67850 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1421)
2014-06-12 14:51:22.520454 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8a3f202e/rbd_data.10996b8b4567.000000000000199d/head//3 8a3f202e/rbd_data.10996b8b4567.000000000000199d/head//3(173'1065 client.4290.0:54894 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1065)
2014-06-12 14:51:22.520498 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6acd202e/rbd_data.10ae6b8b4567.00000000000002da/head//3 6acd202e/rbd_data.10ae6b8b4567.00000000000002da/head//3(173'2467 client.4311.0:118957 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2467)
2014-06-12 14:51:22.520541 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  1a5d202e/rbd_data.10936b8b4567.0000000000003744/head//3 1a5d202e/rbd_data.10936b8b4567.0000000000003744/head//3(173'2166 client.4284.0:106917 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2166)
2014-06-12 14:51:22.520586 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6f3d202e/rbd_data.10916b8b4567.0000000000003b2f/head//3 6f3d202e/rbd_data.10916b8b4567.0000000000003b2f/head//3(173'2310 client.4282.0:117700 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2310)
2014-06-12 14:51:22.520629 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d40c202e/rbd_data.10b16b8b4567.0000000000001967/head//3 d40c202e/rbd_data.10b16b8b4567.0000000000001967/head//3(173'1147 client.4314.0:55641 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1147)
2014-06-12 14:51:22.520672 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  475b202e/rbd_data.108d6b8b4567.0000000000000317/head//3 475b202e/rbd_data.108d6b8b4567.0000000000000317/head//3(173'2427 client.4278.0:122573 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2427)
2014-06-12 14:51:22.520715 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  14ca202e/rbd_data.108e6b8b4567.0000000000003899/head//3 14ca202e/rbd_data.108e6b8b4567.0000000000003899/head//3(173'2180 client.4279.0:111370 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2180)
2014-06-12 14:51:22.520757 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  aa39202e/rbd_data.109d6b8b4567.0000000000002430/head//3 aa39202e/rbd_data.109d6b8b4567.0000000000002430/head//3(173'1442 client.4294.0:74148 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1442)
2014-06-12 14:51:22.520800 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4dc6202e/rbd_data.10a46b8b4567.0000000000002658/head//3 4dc6202e/rbd_data.10a46b8b4567.0000000000002658/head//3(173'1579 client.4301.0:78637 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1579)
2014-06-12 14:51:22.520842 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a0b6202e/rbd_data.10a26b8b4567.0000000000001db2/head//3 a0b6202e/rbd_data.10a26b8b4567.0000000000001db2/head//3(173'1354 client.4299.0:64162 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1354)
2014-06-12 14:51:22.520884 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  c376202e/rbd_data.108d6b8b4567.0000000000003598/head//3 c376202e/rbd_data.108d6b8b4567.0000000000003598/head//3(173'1995 client.4278.0:107322 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1995)
2014-06-12 14:51:22.520927 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f19f102e/rbd_data.10906b8b4567.0000000000001269/head//3 f19f102e/rbd_data.10906b8b4567.0000000000001269/head//3(173'852 client.4281.0:42522 wrlock_by=unknown.0.0:0 dirty s 4194304 uv852)
2014-06-12 14:51:22.520972 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  1d0f102e/rbd_data.10a76b8b4567.0000000000002894/head//3 1d0f102e/rbd_data.10a76b8b4567.0000000000002894/head//3(173'1657 client.4304.0:82614 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1657)
2014-06-12 14:51:22.521014 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6a1d102e/rbd_data.10b36b8b4567.00000000000034a7/head//3 6a1d102e/rbd_data.10b36b8b4567.00000000000034a7/head//3(173'2000 client.4316.0:105817 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2000)
2014-06-12 14:51:22.521063 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f6fb102e/rbd_data.10ab6b8b4567.0000000000003b8a/head//3 f6fb102e/rbd_data.10ab6b8b4567.0000000000003b8a/head//3(173'2324 client.4308.0:118521 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2324)
2014-06-12 14:51:22.521106 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  cd5a102e/rbd_data.108d6b8b4567.00000000000038f6/head//3 cd5a102e/rbd_data.108d6b8b4567.00000000000038f6/head//3(173'2116 client.4278.0:113210 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2116)
2014-06-12 14:51:22.521148 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  7368102e/rbd_data.10956b8b4567.0000000000002192/head//3 7368102e/rbd_data.10956b8b4567.0000000000002192/head//3(173'1381 client.4286.0:69550 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1381)
2014-06-12 14:51:22.521190 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f6f7102e/rbd_data.10936b8b4567.000000000000183b/head//3 f6f7102e/rbd_data.10936b8b4567.000000000000183b/head//3(173'1124 client.4284.0:51012 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1124)
2014-06-12 14:51:22.521232 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  e664102e/rbd_data.10916b8b4567.0000000000002c71/head//3 e664102e/rbd_data.10916b8b4567.0000000000002c71/head//3(173'1768 client.4282.0:90171 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1768)
2014-06-12 14:51:22.521274 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  1efd002e/rbd_data.10a86b8b4567.00000000000009ea/head//3 1efd002e/rbd_data.10a86b8b4567.00000000000009ea/head//3(173'2694 client.4305.0:133851 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2694)
2014-06-12 14:51:22.521316 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8c1d002e/rbd_data.108d6b8b4567.00000000000016ce/head//3 8c1d002e/rbd_data.108d6b8b4567.00000000000016ce/head//3(173'1004 client.4278.0:51764 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1004)
2014-06-12 14:51:22.521358 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  dffb002e/rbd_data.10b06b8b4567.0000000000000eda/head//3 dffb002e/rbd_data.10b06b8b4567.0000000000000eda/head//3(173'2835 client.4313.0:145113 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2835)
2014-06-12 14:51:22.521403 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d21b002e/rbd_data.10ab6b8b4567.000000000000096b/head//3 d21b002e/rbd_data.10ab6b8b4567.000000000000096b/head//3(173'2650 client.4308.0:134780 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2650)
2014-06-12 14:51:22.521445 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4b85002e/rbd_data.10a76b8b4567.000000000000362a/head//3 4b85002e/rbd_data.10a76b8b4567.000000000000362a/head//3(173'2069 client.4304.0:107179 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2069)
2014-06-12 14:51:22.521478 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:22.521513 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:22.521544 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:22.521549 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:22.521553 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:22.521656 7f9d60bd2700 10 collection_list_partial 3.2e_head start:5423302e//0//-1 is_max:0
2014-06-12 14:51:22.522033 7f9d60bd2700 20 objects: [dd44302e/rbd_data.109a6b8b4567.000000000000144d/head//3,f1d4302e/rbd_data.109a6b8b4567.0000000000000b5f/head//3,e475302e/rbd_data.10ae6b8b4567.0000000000001d21/head//3,8489302e/rbd_data.10996b8b4567.00000000000022bf/head//3,10cd302e/rbd_data.108e6b8b4567.0000000000001ff8/head//3,408f302e/rbd_data.109d6b8b4567.0000000000000c77/head//3,69d0402e/rbd_data.10976b8b4567.0000000000000aeb/head//3,55d1402e/rbd_data.108d6b8b4567.0000000000000c98/head//3,ca03402e/rbd_data.10916b8b4567.0000000000002309/head//3,344402e/rbd_data.10a66b8b4567.0000000000001830/head//3,6d15402e/rbd_data.109f6b8b4567.000000000000080d/head//3,c95402e/rbd_data.10a16b8b4567.0000000000002751/head//3,13b8402e/rbd_data.10ae6b8b4567.0000000000000b0e/head//3,4a69402e/rbd_data.109e6b8b4567.00000000000031ff/head//3,477a402e/rbd_data.109a6b8b4567.0000000000000138/head//3,181b402e/rbd_data.10b46b8b4567.0000000000001e8e/head//3,667b402e/rbd_data.10af6b8b4567.000000000000083f/head//3,c42d402e/rbd_data.10ab6b8b4567.000000000000232e/head//3,b26d402e/rbd_data.108f6b8b4567.000000000000347c/head//3,40ce402e/rbd_data.10966b8b4567.0000000000002ee3/head//3,6b2502e/rbd_data.10986b8b4567.0000000000002e75/head//3,4a54502e/rbd_data.10ae6b8b4567.00000000000003d8/head//3,38e5502e/rbd_data.10aa6b8b4567.0000000000002ea3/head//3,d828502e/rbd_data.109d6b8b4567.0000000000000001/head//3,7e48502e/rbd_data.10b26b8b4567.00000000000018ea/head//3]
2014-06-12 14:51:22.522253 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:22.522788 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:22.522837 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:22.522882 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:22.522906 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:22.522927 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:22.522947 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:22.522967 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_map_chunk [5423302e//0//-1,2699502e//0//-1)
2014-06-12 14:51:22.523013 7f9d60bd2700 10 collection_list_partial 3.2e_head start:5423302e//0//-1 is_max:0
2014-06-12 14:51:22.531630 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:22.536036 7f9d60bd2700 20 objects: [dd44302e/rbd_data.109a6b8b4567.000000000000144d/head//3,f1d4302e/rbd_data.109a6b8b4567.0000000000000b5f/head//3,e475302e/rbd_data.10ae6b8b4567.0000000000001d21/head//3,8489302e/rbd_data.10996b8b4567.00000000000022bf/head//3,10cd302e/rbd_data.108e6b8b4567.0000000000001ff8/head//3,408f302e/rbd_data.109d6b8b4567.0000000000000c77/head//3,69d0402e/rbd_data.10976b8b4567.0000000000000aeb/head//3,55d1402e/rbd_data.108d6b8b4567.0000000000000c98/head//3,ca03402e/rbd_data.10916b8b4567.0000000000002309/head//3,344402e/rbd_data.10a66b8b4567.0000000000001830/head//3,6d15402e/rbd_data.109f6b8b4567.000000000000080d/head//3,c95402e/rbd_data.10a16b8b4567.0000000000002751/head//3,13b8402e/rbd_data.10ae6b8b4567.0000000000000b0e/head//3,4a69402e/rbd_data.109e6b8b4567.00000000000031ff/head//3,477a402e/rbd_data.109a6b8b4567.0000000000000138/head//3,181b402e/rbd_data.10b46b8b4567.0000000000001e8e/head//3,667b402e/rbd_data.10af6b8b4567.000000000000083f/head//3,c42d402e/rbd_data.10ab6b8b4567.000000000000232e/head//3,b26d402e/rbd_data.108f6b8b4567.000000000000347c/head//3,40ce402e/rbd_data.10966b8b4567.0000000000002ee3/head//3,6b2502e/rbd_data.10986b8b4567.0000000000002e75/head//3,4a54502e/rbd_data.10ae6b8b4567.00000000000003d8/head//3,38e5502e/rbd_data.10aa6b8b4567.0000000000002ea3/head//3,d828502e/rbd_data.109d6b8b4567.0000000000000001/head//3,7e48502e/rbd_data.10b26b8b4567.00000000000018ea/head//3,2699502e/rbd_data.10a26b8b4567.0000000000000bd7/head//3,301b502e/rbd_data.10a96b8b4567.0000000000000366/head//3,f75b502e/rbd_data.10a76b8b4567.0000000000002e58/head//3,d21c502e/rbd_data.109c6b8b4567.00000000000001a8/head//3,76ec502e/rbd_data.10a66b8b4567.0000000000000944/head//3,fe8e502e/rbd_data.10916b8b4567.00000000000028b2/head//3,4581602e/rbd_data.10a36b8b4567.0000000000002483/head//3,4744602e/rbd_data.10a26b8b4567.0000000000000a7e/head//3,7ac4602e/rbd_data.109b6b8b4567.0000000000002337/head//3,ba35602e/rbd_data.10986b8b4567.000000000000064b/head//3,ee35602e/rbd_data.10986b8b4567.00000000000038de/head//3,7656602e/rbd_data.10a46b8b4567.0000000000002ed3/head//3,2a57602e/rbd_data.10a56b8b4567.0000000000003a3c/head//3,a9c7602e/rbd_data.10976b8b4567.00000000000030d2/head//3,d5a8602e/rbd_data.10a16b8b4567.0000000000000197/head//3,82a9602e/rbd_data.10a06b8b4567.0000000000002de8/head//3,f21b602e/rbd_data.108e6b8b4567.0000000000002b45/head//3,b8af602e/rbd_data.10a66b8b4567.0000000000001cec/head//3,aec0702e/rbd_data.108d6b8b4567.0000000000001070/head//3,22d2702e/rbd_data.109a6b8b4567.0000000000002139/head//3,4e4702e/rbd_data.10976b8b4567.00000000000004d5/head//3,1465702e/rbd_data.10a36b8b4567.0000000000001297/head//3,24b5702e/rbd_data.10976b8b4567.0000000000000998/head//3,a4ba702e/rbd_data.109c6b8b4567.00000000000001e4/head//3,d75d702e/rbd_data.109a6b8b4567.00000000000026ac/head//3,426f702e/rbd_data.108f6b8b4567.0000000000001b64/head//3,3eff702e/rbd_data.10996b8b4567.0000000000002501/head//3,d350802e/rbd_data.10a06b8b4567.00000000000038d5/head//3,59d2802e/rbd_data.108e6b8b4567.0000000000003aec/head//3,b457802e/rbd_data.10a06b8b4567.00000000000005ae/head//3,998802e/rbd_data.10af6b8b4567.000000000000290e/head//3,dcf802e/rbd_data.10906b8b4567.000000000000399d/head//3,181a902e/rbd_data.10936b8b4567.0000000000001853/head//3,8f1e902e/rbd_data.10a26b8b4567.0000000000000805/head//3,8e2f902e/rbd_data.10a96b8b4567.0000000000003408/head//3,f34f902e/rbd_data.10936b8b4567.00000000000028ef/head//3,a5ef902e/rbd_data.10b06b8b4567.0000000000003767/head//3,da51a02e/rbd_data.10af6b8b4567.0000000000000267/head//3,6dc1a02e/rbd_data.109d6b8b4567.0000000000003928/head//3]
2014-06-12 14:51:22.536109 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:22.536131 7f9d60bd2700 10 stat 3.2e_head/dd44302e/rbd_data.109a6b8b4567.000000000000144d/head//3
2014-06-12 14:51:22.536234 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid dd44302e/rbd_data.109a6b8b4567.000000000000144d/head//3
2014-06-12 14:51:22.554122 7f9d60bd2700 10 getattrs 3.2e_head/dd44302e/rbd_data.109a6b8b4567.000000000000144d/head//3 = 0
2014-06-12 14:51:22.554136 7f9d60bd2700 10 stat 3.2e_head/f1d4302e/rbd_data.109a6b8b4567.0000000000000b5f/head//3
2014-06-12 14:51:22.554260 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f1d4302e/rbd_data.109a6b8b4567.0000000000000b5f/head//3
2014-06-12 14:51:22.593691 7f9d60bd2700 10 getattrs 3.2e_head/f1d4302e/rbd_data.109a6b8b4567.0000000000000b5f/head//3 = 0
2014-06-12 14:51:22.593709 7f9d60bd2700 10 stat 3.2e_head/e475302e/rbd_data.10ae6b8b4567.0000000000001d21/head//3
2014-06-12 14:51:22.593884 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid e475302e/rbd_data.10ae6b8b4567.0000000000001d21/head//3
2014-06-12 14:51:22.623065 7f9d60bd2700 10 getattrs 3.2e_head/e475302e/rbd_data.10ae6b8b4567.0000000000001d21/head//3 = 0
2014-06-12 14:51:22.623073 7f9d60bd2700 10 stat 3.2e_head/8489302e/rbd_data.10996b8b4567.00000000000022bf/head//3
2014-06-12 14:51:22.623168 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8489302e/rbd_data.10996b8b4567.00000000000022bf/head//3
2014-06-12 14:51:22.649096 7f9d60bd2700 10 getattrs 3.2e_head/8489302e/rbd_data.10996b8b4567.00000000000022bf/head//3 = 0
2014-06-12 14:51:22.649103 7f9d60bd2700 10 stat 3.2e_head/10cd302e/rbd_data.108e6b8b4567.0000000000001ff8/head//3
2014-06-12 14:51:22.649192 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 10cd302e/rbd_data.108e6b8b4567.0000000000001ff8/head//3
2014-06-12 14:51:22.649279 7f9d60bd2700 10 getattrs 3.2e_head/10cd302e/rbd_data.108e6b8b4567.0000000000001ff8/head//3 = 0
2014-06-12 14:51:22.649284 7f9d60bd2700 10 stat 3.2e_head/408f302e/rbd_data.109d6b8b4567.0000000000000c77/head//3
2014-06-12 14:51:22.649365 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 408f302e/rbd_data.109d6b8b4567.0000000000000c77/head//3
2014-06-12 14:51:22.661006 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:22.689354 7f9d60bd2700 10 getattrs 3.2e_head/408f302e/rbd_data.109d6b8b4567.0000000000000c77/head//3 = 0
2014-06-12 14:51:22.689371 7f9d60bd2700 10 stat 3.2e_head/69d0402e/rbd_data.10976b8b4567.0000000000000aeb/head//3
2014-06-12 14:51:22.689464 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 69d0402e/rbd_data.10976b8b4567.0000000000000aeb/head//3
2014-06-12 14:51:22.699985 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:22.741234 7f9d60bd2700 10 getattrs 3.2e_head/69d0402e/rbd_data.10976b8b4567.0000000000000aeb/head//3 = 0
2014-06-12 14:51:22.741253 7f9d60bd2700 10 stat 3.2e_head/55d1402e/rbd_data.108d6b8b4567.0000000000000c98/head//3
2014-06-12 14:51:22.741457 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 55d1402e/rbd_data.108d6b8b4567.0000000000000c98/head//3
2014-06-12 14:51:22.767312 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:22.778929 7f9d60bd2700 10 getattrs 3.2e_head/55d1402e/rbd_data.108d6b8b4567.0000000000000c98/head//3 = 0
2014-06-12 14:51:22.778943 7f9d60bd2700 10 stat 3.2e_head/ca03402e/rbd_data.10916b8b4567.0000000000002309/head//3
2014-06-12 14:51:22.779107 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid ca03402e/rbd_data.10916b8b4567.0000000000002309/head//3
2014-06-12 14:51:22.795737 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:22.801266 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:22.807702 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:22.810516 7f9d60bd2700 10 getattrs 3.2e_head/ca03402e/rbd_data.10916b8b4567.0000000000002309/head//3 = 0
2014-06-12 14:51:22.810531 7f9d60bd2700 10 stat 3.2e_head/344402e/rbd_data.10a66b8b4567.0000000000001830/head//3
2014-06-12 14:51:22.810689 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 344402e/rbd_data.10a66b8b4567.0000000000001830/head//3
2014-06-12 14:51:22.810828 7f9d60bd2700 10 getattrs 3.2e_head/344402e/rbd_data.10a66b8b4567.0000000000001830/head//3 = 0
2014-06-12 14:51:22.810837 7f9d60bd2700 10 stat 3.2e_head/6d15402e/rbd_data.109f6b8b4567.000000000000080d/head//3
2014-06-12 14:51:22.810985 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6d15402e/rbd_data.109f6b8b4567.000000000000080d/head//3
2014-06-12 14:51:22.854375 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:22.864722 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:22.868107 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:22.868638 7f9d60bd2700 10 getattrs 3.2e_head/6d15402e/rbd_data.109f6b8b4567.000000000000080d/head//3 = 0
2014-06-12 14:51:22.868659 7f9d60bd2700 10 stat 3.2e_head/c95402e/rbd_data.10a16b8b4567.0000000000002751/head//3
2014-06-12 14:51:22.868786 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid c95402e/rbd_data.10a16b8b4567.0000000000002751/head//3
2014-06-12 14:51:22.868885 7f9d60bd2700 10 getattrs 3.2e_head/c95402e/rbd_data.10a16b8b4567.0000000000002751/head//3 = 0
2014-06-12 14:51:22.868893 7f9d60bd2700 10 stat 3.2e_head/13b8402e/rbd_data.10ae6b8b4567.0000000000000b0e/head//3
2014-06-12 14:51:22.869004 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 13b8402e/rbd_data.10ae6b8b4567.0000000000000b0e/head//3
2014-06-12 14:51:22.871420 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:22.871440 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:51:22.871445 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:51:22.871449 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:51:22.871453 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:51:22.871502 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:22.871506 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:22.871508 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:22.871552 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:22.887966 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:22.887985 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:22.888635 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:22.888750 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:22.888781 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:22.888801 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:22.888825 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:22.888852 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:22.888887 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:22.888912 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:22.888933 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:22.888955 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:22.888975 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:22.888995 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:22.889015 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:22.889035 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:22.889055 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:22.889074 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:22.889094 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:22.889114 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:22.889136 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:22.889157 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:22.889176 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:22.889196 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:22.891358 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:22.891512 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:22.910216 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:22.910300 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:22.925542 7f9d60bd2700 10 getattrs 3.2e_head/13b8402e/rbd_data.10ae6b8b4567.0000000000000b0e/head//3 = 0
2014-06-12 14:51:22.925555 7f9d60bd2700 10 stat 3.2e_head/4a69402e/rbd_data.109e6b8b4567.00000000000031ff/head//3
2014-06-12 14:51:22.925702 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4a69402e/rbd_data.109e6b8b4567.00000000000031ff/head//3
2014-06-12 14:51:22.927808 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:22.935292 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:22.955939 7f9d60bd2700 10 getattrs 3.2e_head/4a69402e/rbd_data.109e6b8b4567.00000000000031ff/head//3 = 0
2014-06-12 14:51:22.955952 7f9d60bd2700 10 stat 3.2e_head/477a402e/rbd_data.109a6b8b4567.0000000000000138/head//3
2014-06-12 14:51:22.956074 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 477a402e/rbd_data.109a6b8b4567.0000000000000138/head//3
2014-06-12 14:51:22.961667 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:22.965377 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:23.003966 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:23.003971 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:23.003972 7f9d74bfa700 20 osd.2 173 _dispatch 0x1926b340 pg_stats_ack(1 pgs tid 4869) v1
2014-06-12 14:51:23.003975 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:51:23.003981 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:23.003982 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:23.016191 7f9d60bd2700 10 getattrs 3.2e_head/477a402e/rbd_data.109a6b8b4567.0000000000000138/head//3 = 0
2014-06-12 14:51:23.016212 7f9d60bd2700 10 stat 3.2e_head/181b402e/rbd_data.10b46b8b4567.0000000000001e8e/head//3
2014-06-12 14:51:23.016339 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 181b402e/rbd_data.10b46b8b4567.0000000000001e8e/head//3
2014-06-12 14:51:23.019290 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:23.019341 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:23.048248 7f9d60bd2700 10 getattrs 3.2e_head/181b402e/rbd_data.10b46b8b4567.0000000000001e8e/head//3 = 0
2014-06-12 14:51:23.048262 7f9d60bd2700 10 stat 3.2e_head/667b402e/rbd_data.10af6b8b4567.000000000000083f/head//3
2014-06-12 14:51:23.048398 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 667b402e/rbd_data.10af6b8b4567.000000000000083f/head//3
2014-06-12 14:51:23.085214 7f9d60bd2700 10 getattrs 3.2e_head/667b402e/rbd_data.10af6b8b4567.000000000000083f/head//3 = 0
2014-06-12 14:51:23.085226 7f9d60bd2700 10 stat 3.2e_head/c42d402e/rbd_data.10ab6b8b4567.000000000000232e/head//3
2014-06-12 14:51:23.085343 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid c42d402e/rbd_data.10ab6b8b4567.000000000000232e/head//3
2014-06-12 14:51:23.103729 7f9d60bd2700 10 getattrs 3.2e_head/c42d402e/rbd_data.10ab6b8b4567.000000000000232e/head//3 = 0
2014-06-12 14:51:23.103740 7f9d60bd2700 10 stat 3.2e_head/b26d402e/rbd_data.108f6b8b4567.000000000000347c/head//3
2014-06-12 14:51:23.103857 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid b26d402e/rbd_data.108f6b8b4567.000000000000347c/head//3
2014-06-12 14:51:23.116112 7f9d60bd2700 10 getattrs 3.2e_head/b26d402e/rbd_data.108f6b8b4567.000000000000347c/head//3 = 0
2014-06-12 14:51:23.116124 7f9d60bd2700 10 stat 3.2e_head/40ce402e/rbd_data.10966b8b4567.0000000000002ee3/head//3
2014-06-12 14:51:23.116237 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 40ce402e/rbd_data.10966b8b4567.0000000000002ee3/head//3
2014-06-12 14:51:23.120622 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:23.129024 7f9d60bd2700 10 getattrs 3.2e_head/40ce402e/rbd_data.10966b8b4567.0000000000002ee3/head//3 = 0
2014-06-12 14:51:23.129035 7f9d60bd2700 10 stat 3.2e_head/6b2502e/rbd_data.10986b8b4567.0000000000002e75/head//3
2014-06-12 14:51:23.129159 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6b2502e/rbd_data.10986b8b4567.0000000000002e75/head//3
2014-06-12 14:51:23.132271 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:23.141760 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:23.151148 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:23.152767 7f9d60bd2700 10 getattrs 3.2e_head/6b2502e/rbd_data.10986b8b4567.0000000000002e75/head//3 = 0
2014-06-12 14:51:23.152780 7f9d60bd2700 10 stat 3.2e_head/4a54502e/rbd_data.10ae6b8b4567.00000000000003d8/head//3
2014-06-12 14:51:23.152898 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4a54502e/rbd_data.10ae6b8b4567.00000000000003d8/head//3
2014-06-12 14:51:23.165556 7f9d44101700  5 -- op tracker -- , seq: 748570, time: 2014-06-12 14:51:23.165478, event: header_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.165568 7f9d44101700  5 -- op tracker -- , seq: 748570, time: 2014-06-12 14:51:23.165480, event: throttled, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.165576 7f9d44101700  5 -- op tracker -- , seq: 748570, time: 2014-06-12 14:51:23.165524, event: all_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.165583 7f9d44101700  5 -- op tracker -- , seq: 748570, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.165587 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:23.165592 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:23.165599 7f9d44101700 15 osd.2 173 enqueue_op 0x23e09e00 prio 127 cost 10717 latency 0.000121 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:23.165656 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:23.179593 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:23.185134 7f9d60bd2700 10 getattrs 3.2e_head/4a54502e/rbd_data.10ae6b8b4567.00000000000003d8/head//3 = 0
2014-06-12 14:51:23.185157 7f9d60bd2700 10 stat 3.2e_head/38e5502e/rbd_data.10aa6b8b4567.0000000000002ea3/head//3
2014-06-12 14:51:23.185383 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 38e5502e/rbd_data.10aa6b8b4567.0000000000002ea3/head//3
2014-06-12 14:51:23.210235 7f9d60bd2700 10 getattrs 3.2e_head/38e5502e/rbd_data.10aa6b8b4567.0000000000002ea3/head//3 = 0
2014-06-12 14:51:23.210251 7f9d60bd2700 10 stat 3.2e_head/d828502e/rbd_data.109d6b8b4567.0000000000000001/head//3
2014-06-12 14:51:23.210429 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d828502e/rbd_data.109d6b8b4567.0000000000000001/head//3
2014-06-12 14:51:23.262784 7f9d60bd2700 10 getattrs 3.2e_head/d828502e/rbd_data.109d6b8b4567.0000000000000001/head//3 = 0
2014-06-12 14:51:23.262795 7f9d60bd2700 10 stat 3.2e_head/7e48502e/rbd_data.10b26b8b4567.00000000000018ea/head//3
2014-06-12 14:51:23.262902 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 7e48502e/rbd_data.10b26b8b4567.00000000000018ea/head//3
2014-06-12 14:51:23.278381 7f9d60bd2700 10 getattrs 3.2e_head/7e48502e/rbd_data.10b26b8b4567.00000000000018ea/head//3 = 0
2014-06-12 14:51:23.278392 7f9d60bd2700 10 collection_getattrs 3.2e_head
2014-06-12 14:51:23.278457 7f9d60bd2700 10 collection_getattrs 3.2e_head = 0
2014-06-12 14:51:23.278461 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:23.278489 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:23.278495 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:23.278504 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:23.278507 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:23.278537 7f9d653db700 10 osd.2 173 dequeue_op 0x23e09e00 prio 127 cost 10717 latency 0.113058 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]
2014-06-12 14:51:23.278590 7f9d653db700  5 -- op tracker -- , seq: 748570, time: 2014-06-12 14:51:23.278582, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.278608 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] handle_message: 0x23e09e00
2014-06-12 14:51:23.278625 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:23.278642 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:23.278659 7f9d653db700  5 -- op tracker -- , seq: 748570, time: 2014-06-12 14:51:23.278654, event: started, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.278666 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:23.278777 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] map version is 173'2847
2014-06-12 14:51:23.278798 7f9d653db700 10 osd.2 173 dequeue_op 0x23e09e00 finish
2014-06-12 14:51:23.278811 7f9d653db700  5 -- op tracker -- , seq: 748570, time: 2014-06-12 14:51:23.278804, event: done, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.278833 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:23.278838 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:23.278837 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:23.278851 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:23.278873 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:23.278885 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:23.278926 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:23.278933 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:23.278940 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:23.278960 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dd44302e/rbd_data.109a6b8b4567.000000000000144d/head//3, auth == maps.end()
2014-06-12 14:51:23.278980 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj dd44302e/rbd_data.109a6b8b4567.000000000000144d/head//3
2014-06-12 14:51:23.278993 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f1d4302e/rbd_data.109a6b8b4567.0000000000000b5f/head//3, auth == maps.end()
2014-06-12 14:51:23.279002 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f1d4302e/rbd_data.109a6b8b4567.0000000000000b5f/head//3
2014-06-12 14:51:23.279011 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e475302e/rbd_data.10ae6b8b4567.0000000000001d21/head//3, auth == maps.end()
2014-06-12 14:51:23.279019 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e475302e/rbd_data.10ae6b8b4567.0000000000001d21/head//3
2014-06-12 14:51:23.279028 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8489302e/rbd_data.10996b8b4567.00000000000022bf/head//3, auth == maps.end()
2014-06-12 14:51:23.279037 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8489302e/rbd_data.10996b8b4567.00000000000022bf/head//3
2014-06-12 14:51:23.279045 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 10cd302e/rbd_data.108e6b8b4567.0000000000001ff8/head//3, auth == maps.end()
2014-06-12 14:51:23.279054 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 10cd302e/rbd_data.108e6b8b4567.0000000000001ff8/head//3
2014-06-12 14:51:23.279063 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 408f302e/rbd_data.109d6b8b4567.0000000000000c77/head//3, auth == maps.end()
2014-06-12 14:51:23.279071 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 408f302e/rbd_data.109d6b8b4567.0000000000000c77/head//3
2014-06-12 14:51:23.279079 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 69d0402e/rbd_data.10976b8b4567.0000000000000aeb/head//3, auth == maps.end()
2014-06-12 14:51:23.279091 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 69d0402e/rbd_data.10976b8b4567.0000000000000aeb/head//3
2014-06-12 14:51:23.279100 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 55d1402e/rbd_data.108d6b8b4567.0000000000000c98/head//3, auth == maps.end()
2014-06-12 14:51:23.279110 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 55d1402e/rbd_data.108d6b8b4567.0000000000000c98/head//3
2014-06-12 14:51:23.279119 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ca03402e/rbd_data.10916b8b4567.0000000000002309/head//3, auth == maps.end()
2014-06-12 14:51:23.279127 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ca03402e/rbd_data.10916b8b4567.0000000000002309/head//3
2014-06-12 14:51:23.279138 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 344402e/rbd_data.10a66b8b4567.0000000000001830/head//3, auth == maps.end()
2014-06-12 14:51:23.279148 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 344402e/rbd_data.10a66b8b4567.0000000000001830/head//3
2014-06-12 14:51:23.279159 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6d15402e/rbd_data.109f6b8b4567.000000000000080d/head//3, auth == maps.end()
2014-06-12 14:51:23.279169 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6d15402e/rbd_data.109f6b8b4567.000000000000080d/head//3
2014-06-12 14:51:23.279180 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c95402e/rbd_data.10a16b8b4567.0000000000002751/head//3, auth == maps.end()
2014-06-12 14:51:23.279189 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c95402e/rbd_data.10a16b8b4567.0000000000002751/head//3
2014-06-12 14:51:23.279200 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 13b8402e/rbd_data.10ae6b8b4567.0000000000000b0e/head//3, auth == maps.end()
2014-06-12 14:51:23.279210 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 13b8402e/rbd_data.10ae6b8b4567.0000000000000b0e/head//3
2014-06-12 14:51:23.279221 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4a69402e/rbd_data.109e6b8b4567.00000000000031ff/head//3, auth == maps.end()
2014-06-12 14:51:23.279231 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4a69402e/rbd_data.109e6b8b4567.00000000000031ff/head//3
2014-06-12 14:51:23.279240 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 477a402e/rbd_data.109a6b8b4567.0000000000000138/head//3, auth == maps.end()
2014-06-12 14:51:23.279250 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 477a402e/rbd_data.109a6b8b4567.0000000000000138/head//3
2014-06-12 14:51:23.279260 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 181b402e/rbd_data.10b46b8b4567.0000000000001e8e/head//3, auth == maps.end()
2014-06-12 14:51:23.279270 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 181b402e/rbd_data.10b46b8b4567.0000000000001e8e/head//3
2014-06-12 14:51:23.279280 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 667b402e/rbd_data.10af6b8b4567.000000000000083f/head//3, auth == maps.end()
2014-06-12 14:51:23.279294 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 667b402e/rbd_data.10af6b8b4567.000000000000083f/head//3
2014-06-12 14:51:23.279304 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c42d402e/rbd_data.10ab6b8b4567.000000000000232e/head//3, auth == maps.end()
2014-06-12 14:51:23.279314 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c42d402e/rbd_data.10ab6b8b4567.000000000000232e/head//3
2014-06-12 14:51:23.279325 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b26d402e/rbd_data.108f6b8b4567.000000000000347c/head//3, auth == maps.end()
2014-06-12 14:51:23.279335 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b26d402e/rbd_data.108f6b8b4567.000000000000347c/head//3
2014-06-12 14:51:23.279345 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 40ce402e/rbd_data.10966b8b4567.0000000000002ee3/head//3, auth == maps.end()
2014-06-12 14:51:23.279355 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 40ce402e/rbd_data.10966b8b4567.0000000000002ee3/head//3
2014-06-12 14:51:23.279364 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6b2502e/rbd_data.10986b8b4567.0000000000002e75/head//3, auth == maps.end()
2014-06-12 14:51:23.279374 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6b2502e/rbd_data.10986b8b4567.0000000000002e75/head//3
2014-06-12 14:51:23.279385 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4a54502e/rbd_data.10ae6b8b4567.00000000000003d8/head//3, auth == maps.end()
2014-06-12 14:51:23.279395 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4a54502e/rbd_data.10ae6b8b4567.00000000000003d8/head//3
2014-06-12 14:51:23.279405 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 38e5502e/rbd_data.10aa6b8b4567.0000000000002ea3/head//3, auth == maps.end()
2014-06-12 14:51:23.279415 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 38e5502e/rbd_data.10aa6b8b4567.0000000000002ea3/head//3
2014-06-12 14:51:23.279426 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d828502e/rbd_data.109d6b8b4567.0000000000000001/head//3, auth == maps.end()
2014-06-12 14:51:23.279436 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d828502e/rbd_data.109d6b8b4567.0000000000000001/head//3
2014-06-12 14:51:23.279446 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7e48502e/rbd_data.10b26b8b4567.00000000000018ea/head//3, auth == maps.end()
2014-06-12 14:51:23.279456 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7e48502e/rbd_data.10b26b8b4567.00000000000018ea/head//3
2014-06-12 14:51:23.279468 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] 
2014-06-12 14:51:23.279476 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub
2014-06-12 14:51:23.279489 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  7e48502e/rbd_data.10b26b8b4567.00000000000018ea/head//3 7e48502e/rbd_data.10b26b8b4567.00000000000018ea/head//3(173'1075 client.4315.0:54203 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1075)
2014-06-12 14:51:23.279503 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d828502e/rbd_data.109d6b8b4567.0000000000000001/head//3 d828502e/rbd_data.109d6b8b4567.0000000000000001/head//3(173'2336 client.4294.0:117154 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2336)
2014-06-12 14:51:23.279516 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  38e5502e/rbd_data.10aa6b8b4567.0000000000002ea3/head//3 38e5502e/rbd_data.10aa6b8b4567.0000000000002ea3/head//3(173'1832 client.4307.0:95870 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1832)
2014-06-12 14:51:23.279529 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4a54502e/rbd_data.10ae6b8b4567.00000000000003d8/head//3 4a54502e/rbd_data.10ae6b8b4567.00000000000003d8/head//3(173'2506 client.4311.0:120634 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2506)
2014-06-12 14:51:23.279541 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6b2502e/rbd_data.10986b8b4567.0000000000002e75/head//3 6b2502e/rbd_data.10986b8b4567.0000000000002e75/head//3(173'1811 client.4289.0:91760 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1811)
2014-06-12 14:51:23.279554 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  40ce402e/rbd_data.10966b8b4567.0000000000002ee3/head//3 40ce402e/rbd_data.10966b8b4567.0000000000002ee3/head//3(173'1759 client.4287.0:90997 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1759)
2014-06-12 14:51:23.279566 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  b26d402e/rbd_data.108f6b8b4567.000000000000347c/head//3 b26d402e/rbd_data.108f6b8b4567.000000000000347c/head//3(173'2022 client.4280.0:102730 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2022)
2014-06-12 14:51:23.279579 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  c42d402e/rbd_data.10ab6b8b4567.000000000000232e/head//3 c42d402e/rbd_data.10ab6b8b4567.000000000000232e/head//3(173'1390 client.4308.0:73176 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1390)
2014-06-12 14:51:23.279591 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  667b402e/rbd_data.10af6b8b4567.000000000000083f/head//3 667b402e/rbd_data.10af6b8b4567.000000000000083f/head//3(173'2618 client.4312.0:133590 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2618)
2014-06-12 14:51:23.279603 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  181b402e/rbd_data.10b46b8b4567.0000000000001e8e/head//3 181b402e/rbd_data.10b46b8b4567.0000000000001e8e/head//3(173'1347 client.4317.0:64210 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1347)
2014-06-12 14:51:23.279616 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  477a402e/rbd_data.109a6b8b4567.0000000000000138/head//3 477a402e/rbd_data.109a6b8b4567.0000000000000138/head//3(173'2374 client.4291.0:117607 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2374)
2014-06-12 14:51:23.279629 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4a69402e/rbd_data.109e6b8b4567.00000000000031ff/head//3 4a69402e/rbd_data.109e6b8b4567.00000000000031ff/head//3(173'1890 client.4295.0:100047 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1890)
2014-06-12 14:51:23.279641 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  13b8402e/rbd_data.10ae6b8b4567.0000000000000b0e/head//3 13b8402e/rbd_data.10ae6b8b4567.0000000000000b0e/head//3(173'2721 client.4311.0:133311 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2721)
2014-06-12 14:51:23.279653 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  c95402e/rbd_data.10a16b8b4567.0000000000002751/head//3 c95402e/rbd_data.10a16b8b4567.0000000000002751/head//3(173'1525 client.4298.0:81112 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1525)
2014-06-12 14:51:23.279666 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6d15402e/rbd_data.109f6b8b4567.000000000000080d/head//3 6d15402e/rbd_data.109f6b8b4567.000000000000080d/head//3(173'2593 client.4296.0:127710 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2593)
2014-06-12 14:51:23.279681 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  344402e/rbd_data.10a66b8b4567.0000000000001830/head//3 344402e/rbd_data.10a66b8b4567.0000000000001830/head//3(173'1093 client.4303.0:51888 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1093)
2014-06-12 14:51:23.279694 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  ca03402e/rbd_data.10916b8b4567.0000000000002309/head//3 ca03402e/rbd_data.10916b8b4567.0000000000002309/head//3(173'1478 client.4282.0:72846 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1478)
2014-06-12 14:51:23.279706 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  55d1402e/rbd_data.108d6b8b4567.0000000000000c98/head//3 55d1402e/rbd_data.108d6b8b4567.0000000000000c98/head//3(173'2706 client.4278.0:138305 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2706)
2014-06-12 14:51:23.279718 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  69d0402e/rbd_data.10976b8b4567.0000000000000aeb/head//3 69d0402e/rbd_data.10976b8b4567.0000000000000aeb/head//3(173'2700 client.4288.0:132488 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2700)
2014-06-12 14:51:23.279731 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  408f302e/rbd_data.109d6b8b4567.0000000000000c77/head//3 408f302e/rbd_data.109d6b8b4567.0000000000000c77/head//3(173'2718 client.4294.0:136861 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2718)
2014-06-12 14:51:23.279743 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  10cd302e/rbd_data.108e6b8b4567.0000000000001ff8/head//3 10cd302e/rbd_data.108e6b8b4567.0000000000001ff8/head//3(173'1326 client.4279.0:67366 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1326)
2014-06-12 14:51:23.279756 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8489302e/rbd_data.10996b8b4567.00000000000022bf/head//3 8489302e/rbd_data.10996b8b4567.00000000000022bf/head//3(173'1363 client.4290.0:70638 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1363)
2014-06-12 14:51:23.279768 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  e475302e/rbd_data.10ae6b8b4567.0000000000001d21/head//3 e475302e/rbd_data.10ae6b8b4567.0000000000001d21/head//3(173'1177 client.4311.0:60876 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1177)
2014-06-12 14:51:23.279781 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f1d4302e/rbd_data.109a6b8b4567.0000000000000b5f/head//3 f1d4302e/rbd_data.109a6b8b4567.0000000000000b5f/head//3(173'2712 client.4291.0:133927 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2712)
2014-06-12 14:51:23.279793 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  dd44302e/rbd_data.109a6b8b4567.000000000000144d/head//3 dd44302e/rbd_data.109a6b8b4567.000000000000144d/head//3(173'910 client.4291.0:45706 wrlock_by=unknown.0.0:0 dirty s 4194304 uv910)
2014-06-12 14:51:23.279803 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:23.279820 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:23.279829 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:23.279831 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:23.279833 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:23.279873 7f9d60bd2700 10 collection_list_partial 3.2e_head start:2699502e//0//-1 is_max:0
2014-06-12 14:51:23.280085 7f9d60bd2700 20 objects: [301b502e/rbd_data.10a96b8b4567.0000000000000366/head//3,f75b502e/rbd_data.10a76b8b4567.0000000000002e58/head//3,d21c502e/rbd_data.109c6b8b4567.00000000000001a8/head//3,76ec502e/rbd_data.10a66b8b4567.0000000000000944/head//3,fe8e502e/rbd_data.10916b8b4567.00000000000028b2/head//3,4581602e/rbd_data.10a36b8b4567.0000000000002483/head//3,4744602e/rbd_data.10a26b8b4567.0000000000000a7e/head//3,7ac4602e/rbd_data.109b6b8b4567.0000000000002337/head//3,ba35602e/rbd_data.10986b8b4567.000000000000064b/head//3,ee35602e/rbd_data.10986b8b4567.00000000000038de/head//3,7656602e/rbd_data.10a46b8b4567.0000000000002ed3/head//3,2a57602e/rbd_data.10a56b8b4567.0000000000003a3c/head//3,a9c7602e/rbd_data.10976b8b4567.00000000000030d2/head//3,d5a8602e/rbd_data.10a16b8b4567.0000000000000197/head//3,82a9602e/rbd_data.10a06b8b4567.0000000000002de8/head//3,f21b602e/rbd_data.108e6b8b4567.0000000000002b45/head//3,b8af602e/rbd_data.10a66b8b4567.0000000000001cec/head//3,aec0702e/rbd_data.108d6b8b4567.0000000000001070/head//3,22d2702e/rbd_data.109a6b8b4567.0000000000002139/head//3,4e4702e/rbd_data.10976b8b4567.00000000000004d5/head//3,1465702e/rbd_data.10a36b8b4567.0000000000001297/head//3,24b5702e/rbd_data.10976b8b4567.0000000000000998/head//3,a4ba702e/rbd_data.109c6b8b4567.00000000000001e4/head//3,d75d702e/rbd_data.109a6b8b4567.00000000000026ac/head//3,426f702e/rbd_data.108f6b8b4567.0000000000001b64/head//3]
2014-06-12 14:51:23.280596 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:23.280621 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:23.280637 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:23.280646 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:23.280652 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:23.280657 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:23.280663 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_map_chunk [2699502e//0//-1,3eff702e//0//-1)
2014-06-12 14:51:23.280670 7f9d60bd2700 10 collection_list_partial 3.2e_head start:2699502e//0//-1 is_max:0
2014-06-12 14:51:23.281167 7f9d60bd2700 20 objects: [301b502e/rbd_data.10a96b8b4567.0000000000000366/head//3,f75b502e/rbd_data.10a76b8b4567.0000000000002e58/head//3,d21c502e/rbd_data.109c6b8b4567.00000000000001a8/head//3,76ec502e/rbd_data.10a66b8b4567.0000000000000944/head//3,fe8e502e/rbd_data.10916b8b4567.00000000000028b2/head//3,4581602e/rbd_data.10a36b8b4567.0000000000002483/head//3,4744602e/rbd_data.10a26b8b4567.0000000000000a7e/head//3,7ac4602e/rbd_data.109b6b8b4567.0000000000002337/head//3,ba35602e/rbd_data.10986b8b4567.000000000000064b/head//3,ee35602e/rbd_data.10986b8b4567.00000000000038de/head//3,7656602e/rbd_data.10a46b8b4567.0000000000002ed3/head//3,2a57602e/rbd_data.10a56b8b4567.0000000000003a3c/head//3,a9c7602e/rbd_data.10976b8b4567.00000000000030d2/head//3,d5a8602e/rbd_data.10a16b8b4567.0000000000000197/head//3,82a9602e/rbd_data.10a06b8b4567.0000000000002de8/head//3,f21b602e/rbd_data.108e6b8b4567.0000000000002b45/head//3,b8af602e/rbd_data.10a66b8b4567.0000000000001cec/head//3,aec0702e/rbd_data.108d6b8b4567.0000000000001070/head//3,22d2702e/rbd_data.109a6b8b4567.0000000000002139/head//3,4e4702e/rbd_data.10976b8b4567.00000000000004d5/head//3,1465702e/rbd_data.10a36b8b4567.0000000000001297/head//3,24b5702e/rbd_data.10976b8b4567.0000000000000998/head//3,a4ba702e/rbd_data.109c6b8b4567.00000000000001e4/head//3,d75d702e/rbd_data.109a6b8b4567.00000000000026ac/head//3,426f702e/rbd_data.108f6b8b4567.0000000000001b64/head//3,3eff702e/rbd_data.10996b8b4567.0000000000002501/head//3,d350802e/rbd_data.10a06b8b4567.00000000000038d5/head//3,59d2802e/rbd_data.108e6b8b4567.0000000000003aec/head//3,b457802e/rbd_data.10a06b8b4567.00000000000005ae/head//3,998802e/rbd_data.10af6b8b4567.000000000000290e/head//3,dcf802e/rbd_data.10906b8b4567.000000000000399d/head//3,181a902e/rbd_data.10936b8b4567.0000000000001853/head//3,8f1e902e/rbd_data.10a26b8b4567.0000000000000805/head//3,8e2f902e/rbd_data.10a96b8b4567.0000000000003408/head//3,f34f902e/rbd_data.10936b8b4567.00000000000028ef/head//3,a5ef902e/rbd_data.10b06b8b4567.0000000000003767/head//3,da51a02e/rbd_data.10af6b8b4567.0000000000000267/head//3,6dc1a02e/rbd_data.109d6b8b4567.0000000000003928/head//3,43d2a02e/rbd_data.10b26b8b4567.000000000000385c/head//3,de53a02e/rbd_data.10b36b8b4567.000000000000284b/head//3,6e17a02e/rbd_data.10ac6b8b4567.00000000000007d1/head//3,c8cba02e/rbd_data.10a26b8b4567.0000000000002b8f/head//3,9b0ea02e/rbd_data.10ae6b8b4567.0000000000001c29/head//3,6f0ea02e/rbd_data.108e6b8b4567.000000000000181f/head//3,c001b02e/rbd_data.10946b8b4567.0000000000000319/head//3,28a4b02e/rbd_data.10ab6b8b4567.0000000000001530/head//3,c317b02e/rbd_data.10b46b8b4567.0000000000001275/head//3,9b48b02e/rbd_data.10af6b8b4567.0000000000002b67/head//3,be09b02e/rbd_data.10b36b8b4567.0000000000000583/head//3,87cb02e/rbd_data.10b26b8b4567.000000000000124f/head//3,b60c02e/rbd_data.10af6b8b4567.0000000000000050/head//3,ba31c02e/rbd_data.10b36b8b4567.0000000000001c1b/head//3,34a2c02e/rbd_data.10a46b8b4567.00000000000012a4/head//3,abe4c02e/rbd_data.10af6b8b4567.0000000000002043/head//3,9217c02e/rbd_data.10a16b8b4567.00000000000029f3/head//3,dd18c02e/rbd_data.10ab6b8b4567.000000000000133e/head//3,785ac02e/rbd_data.10a16b8b4567.000000000000127e/head//3,2a7ac02e/rbd_data.10b06b8b4567.00000000000024f3/head//3,cfcac02e/rbd_data.10946b8b4567.0000000000002290/head//3,f54bc02e/rbd_data.109a6b8b4567.00000000000010a3/head//3,72ec02e/rbd_data.10ad6b8b4567.00000000000006de/head//3,46fc02e/rbd_data.10976b8b4567.0000000000000632/head//3,afbfc02e/rbd_data.109c6b8b4567.00000000000031eb/head//3,8cb0d02e/rbd_data.10a46b8b4567.00000000000037a4/head//3]
2014-06-12 14:51:23.281242 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:23.281258 7f9d60bd2700 10 stat 3.2e_head/301b502e/rbd_data.10a96b8b4567.0000000000000366/head//3
2014-06-12 14:51:23.281374 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 301b502e/rbd_data.10a96b8b4567.0000000000000366/head//3
2014-06-12 14:51:23.282312 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:23.297796 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:23.333019 7f9d60bd2700 10 getattrs 3.2e_head/301b502e/rbd_data.10a96b8b4567.0000000000000366/head//3 = 0
2014-06-12 14:51:23.333035 7f9d60bd2700 10 stat 3.2e_head/f75b502e/rbd_data.10a76b8b4567.0000000000002e58/head//3
2014-06-12 14:51:23.333175 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f75b502e/rbd_data.10a76b8b4567.0000000000002e58/head//3
2014-06-12 14:51:23.336553 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:23.340991 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:23.363532 7f9d60bd2700 10 getattrs 3.2e_head/f75b502e/rbd_data.10a76b8b4567.0000000000002e58/head//3 = 0
2014-06-12 14:51:23.363547 7f9d60bd2700 10 stat 3.2e_head/d21c502e/rbd_data.109c6b8b4567.00000000000001a8/head//3
2014-06-12 14:51:23.363721 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d21c502e/rbd_data.109c6b8b4567.00000000000001a8/head//3
2014-06-12 14:51:23.389520 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:23.396580 7f9d60bd2700 10 getattrs 3.2e_head/d21c502e/rbd_data.109c6b8b4567.00000000000001a8/head//3 = 0
2014-06-12 14:51:23.396589 7f9d60bd2700 10 stat 3.2e_head/76ec502e/rbd_data.10a66b8b4567.0000000000000944/head//3
2014-06-12 14:51:23.396708 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 76ec502e/rbd_data.10a66b8b4567.0000000000000944/head//3
2014-06-12 14:51:23.433059 7f9d60bd2700 10 getattrs 3.2e_head/76ec502e/rbd_data.10a66b8b4567.0000000000000944/head//3 = 0
2014-06-12 14:51:23.433084 7f9d60bd2700 10 stat 3.2e_head/fe8e502e/rbd_data.10916b8b4567.00000000000028b2/head//3
2014-06-12 14:51:23.433241 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid fe8e502e/rbd_data.10916b8b4567.00000000000028b2/head//3
2014-06-12 14:51:23.459953 7f9d60bd2700 10 getattrs 3.2e_head/fe8e502e/rbd_data.10916b8b4567.00000000000028b2/head//3 = 0
2014-06-12 14:51:23.459960 7f9d60bd2700 10 stat 3.2e_head/4581602e/rbd_data.10a36b8b4567.0000000000002483/head//3
2014-06-12 14:51:23.460060 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4581602e/rbd_data.10a36b8b4567.0000000000002483/head//3
2014-06-12 14:51:23.468149 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:23.468278 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:23.485106 7f9d60bd2700 10 getattrs 3.2e_head/4581602e/rbd_data.10a36b8b4567.0000000000002483/head//3 = 0
2014-06-12 14:51:23.485116 7f9d60bd2700 10 stat 3.2e_head/4744602e/rbd_data.10a26b8b4567.0000000000000a7e/head//3
2014-06-12 14:51:23.485222 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4744602e/rbd_data.10a26b8b4567.0000000000000a7e/head//3
2014-06-12 14:51:23.512704 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:23.520003 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:23.520069 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:23.534280 7f9d60bd2700 10 getattrs 3.2e_head/4744602e/rbd_data.10a26b8b4567.0000000000000a7e/head//3 = 0
2014-06-12 14:51:23.534298 7f9d60bd2700 10 stat 3.2e_head/7ac4602e/rbd_data.109b6b8b4567.0000000000002337/head//3
2014-06-12 14:51:23.534396 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 7ac4602e/rbd_data.109b6b8b4567.0000000000002337/head//3
2014-06-12 14:51:23.546641 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:23.552875 7f9d60bd2700 10 getattrs 3.2e_head/7ac4602e/rbd_data.109b6b8b4567.0000000000002337/head//3 = 0
2014-06-12 14:51:23.552890 7f9d60bd2700 10 stat 3.2e_head/ba35602e/rbd_data.10986b8b4567.000000000000064b/head//3
2014-06-12 14:51:23.553084 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid ba35602e/rbd_data.10986b8b4567.000000000000064b/head//3
2014-06-12 14:51:23.582192 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:23.582221 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:23.582686 7f9d60bd2700 10 getattrs 3.2e_head/ba35602e/rbd_data.10986b8b4567.000000000000064b/head//3 = 0
2014-06-12 14:51:23.582704 7f9d60bd2700 10 stat 3.2e_head/ee35602e/rbd_data.10986b8b4567.00000000000038de/head//3
2014-06-12 14:51:23.582884 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid ee35602e/rbd_data.10986b8b4567.00000000000038de/head//3
2014-06-12 14:51:23.604651 7f9d60bd2700 10 getattrs 3.2e_head/ee35602e/rbd_data.10986b8b4567.00000000000038de/head//3 = 0
2014-06-12 14:51:23.604667 7f9d60bd2700 10 stat 3.2e_head/7656602e/rbd_data.10a46b8b4567.0000000000002ed3/head//3
2014-06-12 14:51:23.604775 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 7656602e/rbd_data.10a46b8b4567.0000000000002ed3/head//3
2014-06-12 14:51:23.621348 7f9d60bd2700 10 getattrs 3.2e_head/7656602e/rbd_data.10a46b8b4567.0000000000002ed3/head//3 = 0
2014-06-12 14:51:23.621360 7f9d60bd2700 10 stat 3.2e_head/2a57602e/rbd_data.10a56b8b4567.0000000000003a3c/head//3
2014-06-12 14:51:23.621461 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 2a57602e/rbd_data.10a56b8b4567.0000000000003a3c/head//3
2014-06-12 14:51:23.632634 7f9d60bd2700 10 getattrs 3.2e_head/2a57602e/rbd_data.10a56b8b4567.0000000000003a3c/head//3 = 0
2014-06-12 14:51:23.632648 7f9d60bd2700 10 stat 3.2e_head/a9c7602e/rbd_data.10976b8b4567.00000000000030d2/head//3
2014-06-12 14:51:23.632803 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a9c7602e/rbd_data.10976b8b4567.00000000000030d2/head//3
2014-06-12 14:51:23.667999 7f9d60bd2700 10 getattrs 3.2e_head/a9c7602e/rbd_data.10976b8b4567.00000000000030d2/head//3 = 0
2014-06-12 14:51:23.668008 7f9d60bd2700 10 stat 3.2e_head/d5a8602e/rbd_data.10a16b8b4567.0000000000000197/head//3
2014-06-12 14:51:23.668102 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d5a8602e/rbd_data.10a16b8b4567.0000000000000197/head//3
2014-06-12 14:51:23.705151 7f9d60bd2700 10 getattrs 3.2e_head/d5a8602e/rbd_data.10a16b8b4567.0000000000000197/head//3 = 0
2014-06-12 14:51:23.705159 7f9d60bd2700 10 stat 3.2e_head/82a9602e/rbd_data.10a06b8b4567.0000000000002de8/head//3
2014-06-12 14:51:23.705253 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 82a9602e/rbd_data.10a06b8b4567.0000000000002de8/head//3
2014-06-12 14:51:23.719924 7f9d60bd2700 10 getattrs 3.2e_head/82a9602e/rbd_data.10a06b8b4567.0000000000002de8/head//3 = 0
2014-06-12 14:51:23.719931 7f9d60bd2700 10 stat 3.2e_head/f21b602e/rbd_data.108e6b8b4567.0000000000002b45/head//3
2014-06-12 14:51:23.720017 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f21b602e/rbd_data.108e6b8b4567.0000000000002b45/head//3
2014-06-12 14:51:23.736045 7f9d60bd2700 10 getattrs 3.2e_head/f21b602e/rbd_data.108e6b8b4567.0000000000002b45/head//3 = 0
2014-06-12 14:51:23.736057 7f9d60bd2700 10 stat 3.2e_head/b8af602e/rbd_data.10a66b8b4567.0000000000001cec/head//3
2014-06-12 14:51:23.736157 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid b8af602e/rbd_data.10a66b8b4567.0000000000001cec/head//3
2014-06-12 14:51:23.768210 7f9d60bd2700 10 getattrs 3.2e_head/b8af602e/rbd_data.10a66b8b4567.0000000000001cec/head//3 = 0
2014-06-12 14:51:23.768220 7f9d60bd2700 10 stat 3.2e_head/aec0702e/rbd_data.108d6b8b4567.0000000000001070/head//3
2014-06-12 14:51:23.768337 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid aec0702e/rbd_data.108d6b8b4567.0000000000001070/head//3
2014-06-12 14:51:23.768454 7f9d60bd2700 10 getattrs 3.2e_head/aec0702e/rbd_data.108d6b8b4567.0000000000001070/head//3 = 0
2014-06-12 14:51:23.768460 7f9d60bd2700 10 stat 3.2e_head/22d2702e/rbd_data.109a6b8b4567.0000000000002139/head//3
2014-06-12 14:51:23.768560 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 22d2702e/rbd_data.109a6b8b4567.0000000000002139/head//3
2014-06-12 14:51:23.782058 7f9d60bd2700 10 getattrs 3.2e_head/22d2702e/rbd_data.109a6b8b4567.0000000000002139/head//3 = 0
2014-06-12 14:51:23.782071 7f9d60bd2700 10 stat 3.2e_head/4e4702e/rbd_data.10976b8b4567.00000000000004d5/head//3
2014-06-12 14:51:23.782184 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4e4702e/rbd_data.10976b8b4567.00000000000004d5/head//3
2014-06-12 14:51:23.810089 7f9d60bd2700 10 getattrs 3.2e_head/4e4702e/rbd_data.10976b8b4567.00000000000004d5/head//3 = 0
2014-06-12 14:51:23.810101 7f9d60bd2700 10 stat 3.2e_head/1465702e/rbd_data.10a36b8b4567.0000000000001297/head//3
2014-06-12 14:51:23.810218 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 1465702e/rbd_data.10a36b8b4567.0000000000001297/head//3
2014-06-12 14:51:23.836628 7f9d60bd2700 10 getattrs 3.2e_head/1465702e/rbd_data.10a36b8b4567.0000000000001297/head//3 = 0
2014-06-12 14:51:23.836638 7f9d60bd2700 10 stat 3.2e_head/24b5702e/rbd_data.10976b8b4567.0000000000000998/head//3
2014-06-12 14:51:23.836757 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 24b5702e/rbd_data.10976b8b4567.0000000000000998/head//3
2014-06-12 14:51:23.867464 7f9d60bd2700 10 getattrs 3.2e_head/24b5702e/rbd_data.10976b8b4567.0000000000000998/head//3 = 0
2014-06-12 14:51:23.867476 7f9d60bd2700 10 stat 3.2e_head/a4ba702e/rbd_data.109c6b8b4567.00000000000001e4/head//3
2014-06-12 14:51:23.867593 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a4ba702e/rbd_data.109c6b8b4567.00000000000001e4/head//3
2014-06-12 14:51:23.871597 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:23.871622 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:23.871627 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:23.871629 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:23.871668 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:23.872643 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:23.920774 7f9d60bd2700 10 getattrs 3.2e_head/a4ba702e/rbd_data.109c6b8b4567.00000000000001e4/head//3 = 0
2014-06-12 14:51:23.920788 7f9d60bd2700 10 stat 3.2e_head/d75d702e/rbd_data.109a6b8b4567.00000000000026ac/head//3
2014-06-12 14:51:23.920918 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d75d702e/rbd_data.109a6b8b4567.00000000000026ac/head//3
2014-06-12 14:51:23.927678 7f9d44101700  5 -- op tracker -- , seq: 748571, time: 2014-06-12 14:51:23.927591, event: header_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.927695 7f9d44101700  5 -- op tracker -- , seq: 748571, time: 2014-06-12 14:51:23.927593, event: throttled, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.927703 7f9d44101700  5 -- op tracker -- , seq: 748571, time: 2014-06-12 14:51:23.927642, event: all_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.927726 7f9d44101700  5 -- op tracker -- , seq: 748571, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.927731 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:23.927737 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:23.927745 7f9d44101700 15 osd.2 173 enqueue_op 0x23e08000 prio 127 cost 10717 latency 0.000154 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:23.927814 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:23.942298 7f9d60bd2700 10 getattrs 3.2e_head/d75d702e/rbd_data.109a6b8b4567.00000000000026ac/head//3 = 0
2014-06-12 14:51:23.942310 7f9d60bd2700 10 stat 3.2e_head/426f702e/rbd_data.108f6b8b4567.0000000000001b64/head//3
2014-06-12 14:51:23.942442 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 426f702e/rbd_data.108f6b8b4567.0000000000001b64/head//3
2014-06-12 14:51:23.962498 7f9d60bd2700 10 getattrs 3.2e_head/426f702e/rbd_data.108f6b8b4567.0000000000001b64/head//3 = 0
2014-06-12 14:51:23.962514 7f9d60bd2700 10 collection_getattrs 3.2e_head
2014-06-12 14:51:23.962606 7f9d60bd2700 10 collection_getattrs 3.2e_head = 0
2014-06-12 14:51:23.962612 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:23.962656 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:23.962667 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:23.962683 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:23.962687 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:23.962721 7f9d633d7700 10 osd.2 173 dequeue_op 0x23e08000 prio 127 cost 10717 latency 0.035130 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]
2014-06-12 14:51:23.962796 7f9d633d7700  5 -- op tracker -- , seq: 748571, time: 2014-06-12 14:51:23.962781, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.962823 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] handle_message: 0x23e08000
2014-06-12 14:51:23.962848 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:23.962866 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:23.962892 7f9d633d7700  5 -- op tracker -- , seq: 748571, time: 2014-06-12 14:51:23.962887, event: started, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.962899 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:23.963013 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] map version is 173'2847
2014-06-12 14:51:23.963036 7f9d633d7700 10 osd.2 173 dequeue_op 0x23e08000 finish
2014-06-12 14:51:23.963049 7f9d633d7700  5 -- op tracker -- , seq: 748571, time: 2014-06-12 14:51:23.963043, event: done, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:23.963070 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:23.963075 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:23.963072 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:23.963085 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:23.963121 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:23.963132 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:23.963169 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:23.963180 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:23.963191 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:23.963223 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 301b502e/rbd_data.10a96b8b4567.0000000000000366/head//3, auth == maps.end()
2014-06-12 14:51:23.963261 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 301b502e/rbd_data.10a96b8b4567.0000000000000366/head//3
2014-06-12 14:51:23.963282 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f75b502e/rbd_data.10a76b8b4567.0000000000002e58/head//3, auth == maps.end()
2014-06-12 14:51:23.963305 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f75b502e/rbd_data.10a76b8b4567.0000000000002e58/head//3
2014-06-12 14:51:23.963320 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d21c502e/rbd_data.109c6b8b4567.00000000000001a8/head//3, auth == maps.end()
2014-06-12 14:51:23.963334 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d21c502e/rbd_data.109c6b8b4567.00000000000001a8/head//3
2014-06-12 14:51:23.963348 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 76ec502e/rbd_data.10a66b8b4567.0000000000000944/head//3, auth == maps.end()
2014-06-12 14:51:23.963364 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 76ec502e/rbd_data.10a66b8b4567.0000000000000944/head//3
2014-06-12 14:51:23.963382 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fe8e502e/rbd_data.10916b8b4567.00000000000028b2/head//3, auth == maps.end()
2014-06-12 14:51:23.963396 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj fe8e502e/rbd_data.10916b8b4567.00000000000028b2/head//3
2014-06-12 14:51:23.963414 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4581602e/rbd_data.10a36b8b4567.0000000000002483/head//3, auth == maps.end()
2014-06-12 14:51:23.963428 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4581602e/rbd_data.10a36b8b4567.0000000000002483/head//3
2014-06-12 14:51:23.963442 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4744602e/rbd_data.10a26b8b4567.0000000000000a7e/head//3, auth == maps.end()
2014-06-12 14:51:23.963456 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4744602e/rbd_data.10a26b8b4567.0000000000000a7e/head//3
2014-06-12 14:51:23.963470 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7ac4602e/rbd_data.109b6b8b4567.0000000000002337/head//3, auth == maps.end()
2014-06-12 14:51:23.963483 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7ac4602e/rbd_data.109b6b8b4567.0000000000002337/head//3
2014-06-12 14:51:23.963498 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ba35602e/rbd_data.10986b8b4567.000000000000064b/head//3, auth == maps.end()
2014-06-12 14:51:23.963511 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ba35602e/rbd_data.10986b8b4567.000000000000064b/head//3
2014-06-12 14:51:23.963526 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ee35602e/rbd_data.10986b8b4567.00000000000038de/head//3, auth == maps.end()
2014-06-12 14:51:23.963541 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ee35602e/rbd_data.10986b8b4567.00000000000038de/head//3
2014-06-12 14:51:23.963557 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7656602e/rbd_data.10a46b8b4567.0000000000002ed3/head//3, auth == maps.end()
2014-06-12 14:51:23.963573 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7656602e/rbd_data.10a46b8b4567.0000000000002ed3/head//3
2014-06-12 14:51:23.963588 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2a57602e/rbd_data.10a56b8b4567.0000000000003a3c/head//3, auth == maps.end()
2014-06-12 14:51:23.963601 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2a57602e/rbd_data.10a56b8b4567.0000000000003a3c/head//3
2014-06-12 14:51:23.963616 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a9c7602e/rbd_data.10976b8b4567.00000000000030d2/head//3, auth == maps.end()
2014-06-12 14:51:23.963629 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a9c7602e/rbd_data.10976b8b4567.00000000000030d2/head//3
2014-06-12 14:51:23.963643 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d5a8602e/rbd_data.10a16b8b4567.0000000000000197/head//3, auth == maps.end()
2014-06-12 14:51:23.963657 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d5a8602e/rbd_data.10a16b8b4567.0000000000000197/head//3
2014-06-12 14:51:23.963671 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 82a9602e/rbd_data.10a06b8b4567.0000000000002de8/head//3, auth == maps.end()
2014-06-12 14:51:23.963684 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 82a9602e/rbd_data.10a06b8b4567.0000000000002de8/head//3
2014-06-12 14:51:23.963701 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f21b602e/rbd_data.108e6b8b4567.0000000000002b45/head//3, auth == maps.end()
2014-06-12 14:51:23.963715 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f21b602e/rbd_data.108e6b8b4567.0000000000002b45/head//3
2014-06-12 14:51:23.963731 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b8af602e/rbd_data.10a66b8b4567.0000000000001cec/head//3, auth == maps.end()
2014-06-12 14:51:23.963752 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b8af602e/rbd_data.10a66b8b4567.0000000000001cec/head//3
2014-06-12 14:51:23.963767 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aec0702e/rbd_data.108d6b8b4567.0000000000001070/head//3, auth == maps.end()
2014-06-12 14:51:23.963781 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj aec0702e/rbd_data.108d6b8b4567.0000000000001070/head//3
2014-06-12 14:51:23.963795 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 22d2702e/rbd_data.109a6b8b4567.0000000000002139/head//3, auth == maps.end()
2014-06-12 14:51:23.963809 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 22d2702e/rbd_data.109a6b8b4567.0000000000002139/head//3
2014-06-12 14:51:23.963823 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4e4702e/rbd_data.10976b8b4567.00000000000004d5/head//3, auth == maps.end()
2014-06-12 14:51:23.963837 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4e4702e/rbd_data.10976b8b4567.00000000000004d5/head//3
2014-06-12 14:51:23.963851 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1465702e/rbd_data.10a36b8b4567.0000000000001297/head//3, auth == maps.end()
2014-06-12 14:51:23.963866 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1465702e/rbd_data.10a36b8b4567.0000000000001297/head//3
2014-06-12 14:51:23.963881 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 24b5702e/rbd_data.10976b8b4567.0000000000000998/head//3, auth == maps.end()
2014-06-12 14:51:23.963897 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 24b5702e/rbd_data.10976b8b4567.0000000000000998/head//3
2014-06-12 14:51:23.963914 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a4ba702e/rbd_data.109c6b8b4567.00000000000001e4/head//3, auth == maps.end()
2014-06-12 14:51:23.963931 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a4ba702e/rbd_data.109c6b8b4567.00000000000001e4/head//3
2014-06-12 14:51:23.963948 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d75d702e/rbd_data.109a6b8b4567.00000000000026ac/head//3, auth == maps.end()
2014-06-12 14:51:23.963964 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d75d702e/rbd_data.109a6b8b4567.00000000000026ac/head//3
2014-06-12 14:51:23.963979 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 426f702e/rbd_data.108f6b8b4567.0000000000001b64/head//3, auth == maps.end()
2014-06-12 14:51:23.963995 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 426f702e/rbd_data.108f6b8b4567.0000000000001b64/head//3
2014-06-12 14:51:23.964015 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] 
2014-06-12 14:51:23.964028 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub
2014-06-12 14:51:23.964047 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  426f702e/rbd_data.108f6b8b4567.0000000000001b64/head//3 426f702e/rbd_data.108f6b8b4567.0000000000001b64/head//3(173'1252 client.4280.0:58373 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1252)
2014-06-12 14:51:23.964070 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d75d702e/rbd_data.109a6b8b4567.00000000000026ac/head//3 d75d702e/rbd_data.109a6b8b4567.00000000000026ac/head//3(173'1498 client.4291.0:77697 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1498)
2014-06-12 14:51:23.964092 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a4ba702e/rbd_data.109c6b8b4567.00000000000001e4/head//3 a4ba702e/rbd_data.109c6b8b4567.00000000000001e4/head//3(173'2395 client.4293.0:120322 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2395)
2014-06-12 14:51:23.964112 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  24b5702e/rbd_data.10976b8b4567.0000000000000998/head//3 24b5702e/rbd_data.10976b8b4567.0000000000000998/head//3(173'2657 client.4288.0:130223 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2657)
2014-06-12 14:51:23.964133 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  1465702e/rbd_data.10a36b8b4567.0000000000001297/head//3 1465702e/rbd_data.10a36b8b4567.0000000000001297/head//3(173'887 client.4300.0:41772 wrlock_by=unknown.0.0:0 dirty s 4194304 uv887)
2014-06-12 14:51:23.964153 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4e4702e/rbd_data.10976b8b4567.00000000000004d5/head//3 4e4702e/rbd_data.10976b8b4567.00000000000004d5/head//3(173'2517 client.4288.0:122315 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2517)
2014-06-12 14:51:23.964174 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  22d2702e/rbd_data.109a6b8b4567.0000000000002139/head//3 22d2702e/rbd_data.109a6b8b4567.0000000000002139/head//3(173'1338 client.4291.0:68258 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1338)
2014-06-12 14:51:23.964194 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  aec0702e/rbd_data.108d6b8b4567.0000000000001070/head//3 aec0702e/rbd_data.108d6b8b4567.0000000000001070/head//3(173'2829 client.4278.0:145029 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2829)
2014-06-12 14:51:23.964215 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  b8af602e/rbd_data.10a66b8b4567.0000000000001cec/head//3 b8af602e/rbd_data.10a66b8b4567.0000000000001cec/head//3(173'1243 client.4303.0:59738 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1243)
2014-06-12 14:51:23.964235 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f21b602e/rbd_data.108e6b8b4567.0000000000002b45/head//3 f21b602e/rbd_data.108e6b8b4567.0000000000002b45/head//3(173'1712 client.4279.0:87672 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1712)
2014-06-12 14:51:23.964255 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  82a9602e/rbd_data.10a06b8b4567.0000000000002de8/head//3 82a9602e/rbd_data.10a06b8b4567.0000000000002de8/head//3(173'1736 client.4297.0:91180 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1736)
2014-06-12 14:51:23.964275 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d5a8602e/rbd_data.10a16b8b4567.0000000000000197/head//3 d5a8602e/rbd_data.10a16b8b4567.0000000000000197/head//3(173'2388 client.4298.0:121252 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2388)
2014-06-12 14:51:23.964296 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a9c7602e/rbd_data.10976b8b4567.00000000000030d2/head//3 a9c7602e/rbd_data.10976b8b4567.00000000000030d2/head//3(173'1973 client.4288.0:95348 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1973)
2014-06-12 14:51:23.964316 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  2a57602e/rbd_data.10a56b8b4567.0000000000003a3c/head//3 2a57602e/rbd_data.10a56b8b4567.0000000000003a3c/head//3(173'2288 client.4302.0:116356 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2288)
2014-06-12 14:51:23.964336 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  7656602e/rbd_data.10a46b8b4567.0000000000002ed3/head//3 7656602e/rbd_data.10a46b8b4567.0000000000002ed3/head//3(173'1837 client.4301.0:94679 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1837)
2014-06-12 14:51:23.964360 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  ee35602e/rbd_data.10986b8b4567.00000000000038de/head//3 ee35602e/rbd_data.10986b8b4567.00000000000038de/head//3(173'2103 client.4289.0:110496 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2103)
2014-06-12 14:51:23.964380 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  ba35602e/rbd_data.10986b8b4567.000000000000064b/head//3 ba35602e/rbd_data.10986b8b4567.000000000000064b/head//3(173'2552 client.4289.0:125100 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2552)
2014-06-12 14:51:23.964398 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  7ac4602e/rbd_data.109b6b8b4567.0000000000002337/head//3 7ac4602e/rbd_data.109b6b8b4567.0000000000002337/head//3(173'1485 client.4292.0:70634 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1485)
2014-06-12 14:51:23.964418 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4744602e/rbd_data.10a26b8b4567.0000000000000a7e/head//3 4744602e/rbd_data.10a26b8b4567.0000000000000a7e/head//3(173'2680 client.4299.0:137078 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2680)
2014-06-12 14:51:23.964438 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4581602e/rbd_data.10a36b8b4567.0000000000002483/head//3 4581602e/rbd_data.10a36b8b4567.0000000000002483/head//3(173'1436 client.4300.0:73365 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1436)
2014-06-12 14:51:23.964458 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  fe8e502e/rbd_data.10916b8b4567.00000000000028b2/head//3 fe8e502e/rbd_data.10916b8b4567.00000000000028b2/head//3(173'1687 client.4282.0:83225 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1687)
2014-06-12 14:51:23.964478 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  76ec502e/rbd_data.10a66b8b4567.0000000000000944/head//3 76ec502e/rbd_data.10a66b8b4567.0000000000000944/head//3(173'2635 client.4303.0:133259 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2635)
2014-06-12 14:51:23.964498 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d21c502e/rbd_data.109c6b8b4567.00000000000001a8/head//3 d21c502e/rbd_data.109c6b8b4567.00000000000001a8/head//3(173'2382 client.4293.0:120005 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2382)
2014-06-12 14:51:23.964518 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f75b502e/rbd_data.10a76b8b4567.0000000000002e58/head//3 f75b502e/rbd_data.10a76b8b4567.0000000000002e58/head//3(173'1823 client.4304.0:93110 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1823)
2014-06-12 14:51:23.964538 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  301b502e/rbd_data.10a96b8b4567.0000000000000366/head//3 301b502e/rbd_data.10a96b8b4567.0000000000000366/head//3(173'2491 client.4306.0:123173 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2491)
2014-06-12 14:51:23.964554 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:23.964570 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:23.964582 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:23.964585 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:23.964587 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:23.964640 7f9d60bd2700 10 collection_list_partial 3.2e_head start:3eff702e//0//-1 is_max:0
2014-06-12 14:51:23.964954 7f9d60bd2700 20 objects: [d350802e/rbd_data.10a06b8b4567.00000000000038d5/head//3,59d2802e/rbd_data.108e6b8b4567.0000000000003aec/head//3,b457802e/rbd_data.10a06b8b4567.00000000000005ae/head//3,998802e/rbd_data.10af6b8b4567.000000000000290e/head//3,dcf802e/rbd_data.10906b8b4567.000000000000399d/head//3,181a902e/rbd_data.10936b8b4567.0000000000001853/head//3,8f1e902e/rbd_data.10a26b8b4567.0000000000000805/head//3,8e2f902e/rbd_data.10a96b8b4567.0000000000003408/head//3,f34f902e/rbd_data.10936b8b4567.00000000000028ef/head//3,a5ef902e/rbd_data.10b06b8b4567.0000000000003767/head//3,da51a02e/rbd_data.10af6b8b4567.0000000000000267/head//3,6dc1a02e/rbd_data.109d6b8b4567.0000000000003928/head//3,43d2a02e/rbd_data.10b26b8b4567.000000000000385c/head//3,de53a02e/rbd_data.10b36b8b4567.000000000000284b/head//3,6e17a02e/rbd_data.10ac6b8b4567.00000000000007d1/head//3,c8cba02e/rbd_data.10a26b8b4567.0000000000002b8f/head//3,9b0ea02e/rbd_data.10ae6b8b4567.0000000000001c29/head//3,6f0ea02e/rbd_data.108e6b8b4567.000000000000181f/head//3,c001b02e/rbd_data.10946b8b4567.0000000000000319/head//3,28a4b02e/rbd_data.10ab6b8b4567.0000000000001530/head//3,c317b02e/rbd_data.10b46b8b4567.0000000000001275/head//3,9b48b02e/rbd_data.10af6b8b4567.0000000000002b67/head//3,be09b02e/rbd_data.10b36b8b4567.0000000000000583/head//3,87cb02e/rbd_data.10b26b8b4567.000000000000124f/head//3,b60c02e/rbd_data.10af6b8b4567.0000000000000050/head//3]
2014-06-12 14:51:23.965547 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:23.965582 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:23.965605 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:23.965629 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:23.965638 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:23.965647 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:23.965656 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_map_chunk [3eff702e//0//-1,ba31c02e//0//-1)
2014-06-12 14:51:23.965667 7f9d60bd2700 10 collection_list_partial 3.2e_head start:3eff702e//0//-1 is_max:0
2014-06-12 14:51:23.966402 7f9d60bd2700 20 objects: [d350802e/rbd_data.10a06b8b4567.00000000000038d5/head//3,59d2802e/rbd_data.108e6b8b4567.0000000000003aec/head//3,b457802e/rbd_data.10a06b8b4567.00000000000005ae/head//3,998802e/rbd_data.10af6b8b4567.000000000000290e/head//3,dcf802e/rbd_data.10906b8b4567.000000000000399d/head//3,181a902e/rbd_data.10936b8b4567.0000000000001853/head//3,8f1e902e/rbd_data.10a26b8b4567.0000000000000805/head//3,8e2f902e/rbd_data.10a96b8b4567.0000000000003408/head//3,f34f902e/rbd_data.10936b8b4567.00000000000028ef/head//3,a5ef902e/rbd_data.10b06b8b4567.0000000000003767/head//3,da51a02e/rbd_data.10af6b8b4567.0000000000000267/head//3,6dc1a02e/rbd_data.109d6b8b4567.0000000000003928/head//3,43d2a02e/rbd_data.10b26b8b4567.000000000000385c/head//3,de53a02e/rbd_data.10b36b8b4567.000000000000284b/head//3,6e17a02e/rbd_data.10ac6b8b4567.00000000000007d1/head//3,c8cba02e/rbd_data.10a26b8b4567.0000000000002b8f/head//3,9b0ea02e/rbd_data.10ae6b8b4567.0000000000001c29/head//3,6f0ea02e/rbd_data.108e6b8b4567.000000000000181f/head//3,c001b02e/rbd_data.10946b8b4567.0000000000000319/head//3,28a4b02e/rbd_data.10ab6b8b4567.0000000000001530/head//3,c317b02e/rbd_data.10b46b8b4567.0000000000001275/head//3,9b48b02e/rbd_data.10af6b8b4567.0000000000002b67/head//3,be09b02e/rbd_data.10b36b8b4567.0000000000000583/head//3,87cb02e/rbd_data.10b26b8b4567.000000000000124f/head//3,b60c02e/rbd_data.10af6b8b4567.0000000000000050/head//3,ba31c02e/rbd_data.10b36b8b4567.0000000000001c1b/head//3,34a2c02e/rbd_data.10a46b8b4567.00000000000012a4/head//3,abe4c02e/rbd_data.10af6b8b4567.0000000000002043/head//3,9217c02e/rbd_data.10a16b8b4567.00000000000029f3/head//3,dd18c02e/rbd_data.10ab6b8b4567.000000000000133e/head//3,785ac02e/rbd_data.10a16b8b4567.000000000000127e/head//3,2a7ac02e/rbd_data.10b06b8b4567.00000000000024f3/head//3,cfcac02e/rbd_data.10946b8b4567.0000000000002290/head//3,f54bc02e/rbd_data.109a6b8b4567.00000000000010a3/head//3,72ec02e/rbd_data.10ad6b8b4567.00000000000006de/head//3,46fc02e/rbd_data.10976b8b4567.0000000000000632/head//3,afbfc02e/rbd_data.109c6b8b4567.00000000000031eb/head//3,8cb0d02e/rbd_data.10a46b8b4567.00000000000037a4/head//3,86c1d02e/rbd_data.10976b8b4567.0000000000003abc/head//3,18f3d02e/rbd_data.10ab6b8b4567.0000000000000c93/head//3,e55d02e/rbd_data.10ac6b8b4567.0000000000002dd9/head//3,b7f6d02e/rbd_data.109b6b8b4567.000000000000183f/head//3,3af7d02e/rbd_data.10946b8b4567.00000000000015f6/head//3,647ad02e/rbd_data.10ad6b8b4567.000000000000363a/head//3,d6fdd02e/rbd_data.10986b8b4567.0000000000001c7f/head//3,e570e02e/rbd_data.10966b8b4567.0000000000001aab/head//3,ed0e02e/rbd_data.10a96b8b4567.000000000000385a/head//3,3122e02e/rbd_data.10a36b8b4567.0000000000001e3d/head//3,5824e02e/rbd_data.10956b8b4567.0000000000002e63/head//3,e9e6e02e/rbd_data.10af6b8b4567.0000000000000dcd/head//3,8319e02e/rbd_data.10a96b8b4567.0000000000000398/head//3,23cbe02e/rbd_data.109d6b8b4567.00000000000023c0/head//3,fe6ce02e/rbd_data.109b6b8b4567.0000000000001594/head//3,e162f02e/rbd_data.108f6b8b4567.0000000000001694/head//3,2646f02e/rbd_data.109e6b8b4567.00000000000024fb/head//3,6fe7f02e/rbd_data.10a06b8b4567.000000000000000a/head//3,f9cbf02e/rbd_data.10986b8b4567.000000000000190b/head//3,d2ef02e/rbd_data.10ad6b8b4567.0000000000002d19/head//3,5a0ff02e/rbd_data.10ad6b8b4567.00000000000000b3/head//3,a86ff02e/rbd_data.109c6b8b4567.0000000000000325/head//3,8b71082e/rbd_data.10996b8b4567.0000000000000ded/head//3,cd91082e/rbd_data.10b16b8b4567.0000000000001e29/head//3,ed23082e/rbd_data.10a86b8b4567.00000000000015f8/head//3,5d85082e/rbd_data.10a66b8b4567.00000000000030b5/head//3]
2014-06-12 14:51:23.966511 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:23.966541 7f9d60bd2700 10 stat 3.2e_head/d350802e/rbd_data.10a06b8b4567.00000000000038d5/head//3
2014-06-12 14:51:23.966723 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d350802e/rbd_data.10a06b8b4567.00000000000038d5/head//3
2014-06-12 14:51:23.976373 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:23.998945 7f9d60bd2700 10 getattrs 3.2e_head/d350802e/rbd_data.10a06b8b4567.00000000000038d5/head//3 = 0
2014-06-12 14:51:23.998956 7f9d60bd2700 10 stat 3.2e_head/59d2802e/rbd_data.108e6b8b4567.0000000000003aec/head//3
2014-06-12 14:51:23.999056 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 59d2802e/rbd_data.108e6b8b4567.0000000000003aec/head//3
2014-06-12 14:51:24.016152 7f9d60bd2700 10 getattrs 3.2e_head/59d2802e/rbd_data.108e6b8b4567.0000000000003aec/head//3 = 0
2014-06-12 14:51:24.016159 7f9d60bd2700 10 stat 3.2e_head/b457802e/rbd_data.10a06b8b4567.00000000000005ae/head//3
2014-06-12 14:51:24.016254 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid b457802e/rbd_data.10a06b8b4567.00000000000005ae/head//3
2014-06-12 14:51:24.062342 7f9d60bd2700 10 getattrs 3.2e_head/b457802e/rbd_data.10a06b8b4567.00000000000005ae/head//3 = 0
2014-06-12 14:51:24.062359 7f9d60bd2700 10 stat 3.2e_head/998802e/rbd_data.10af6b8b4567.000000000000290e/head//3
2014-06-12 14:51:24.062462 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 998802e/rbd_data.10af6b8b4567.000000000000290e/head//3
2014-06-12 14:51:24.064127 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:24.074492 7f9d60bd2700 10 getattrs 3.2e_head/998802e/rbd_data.10af6b8b4567.000000000000290e/head//3 = 0
2014-06-12 14:51:24.074501 7f9d60bd2700 10 stat 3.2e_head/dcf802e/rbd_data.10906b8b4567.000000000000399d/head//3
2014-06-12 14:51:24.074603 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid dcf802e/rbd_data.10906b8b4567.000000000000399d/head//3
2014-06-12 14:51:24.082806 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:24.082853 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:24.099993 7f9d60bd2700 10 getattrs 3.2e_head/dcf802e/rbd_data.10906b8b4567.000000000000399d/head//3 = 0
2014-06-12 14:51:24.100001 7f9d60bd2700 10 stat 3.2e_head/181a902e/rbd_data.10936b8b4567.0000000000001853/head//3
2014-06-12 14:51:24.100104 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 181a902e/rbd_data.10936b8b4567.0000000000001853/head//3
2014-06-12 14:51:24.133440 7f9d60bd2700 10 getattrs 3.2e_head/181a902e/rbd_data.10936b8b4567.0000000000001853/head//3 = 0
2014-06-12 14:51:24.133450 7f9d60bd2700 10 stat 3.2e_head/8f1e902e/rbd_data.10a26b8b4567.0000000000000805/head//3
2014-06-12 14:51:24.133552 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8f1e902e/rbd_data.10a26b8b4567.0000000000000805/head//3
2014-06-12 14:51:24.154331 7f9d60bd2700 10 getattrs 3.2e_head/8f1e902e/rbd_data.10a26b8b4567.0000000000000805/head//3 = 0
2014-06-12 14:51:24.154338 7f9d60bd2700 10 stat 3.2e_head/8e2f902e/rbd_data.10a96b8b4567.0000000000003408/head//3
2014-06-12 14:51:24.154441 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8e2f902e/rbd_data.10a96b8b4567.0000000000003408/head//3
2014-06-12 14:51:24.156680 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:24.156710 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:24.166868 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:24.169225 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:24.169262 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:24.189478 7f9d60bd2700 10 getattrs 3.2e_head/8e2f902e/rbd_data.10a96b8b4567.0000000000003408/head//3 = 0
2014-06-12 14:51:24.189489 7f9d60bd2700 10 stat 3.2e_head/f34f902e/rbd_data.10936b8b4567.00000000000028ef/head//3
2014-06-12 14:51:24.189600 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f34f902e/rbd_data.10936b8b4567.00000000000028ef/head//3
2014-06-12 14:51:24.202450 7f9d60bd2700 10 getattrs 3.2e_head/f34f902e/rbd_data.10936b8b4567.00000000000028ef/head//3 = 0
2014-06-12 14:51:24.202459 7f9d60bd2700 10 stat 3.2e_head/a5ef902e/rbd_data.10b06b8b4567.0000000000003767/head//3
2014-06-12 14:51:24.202577 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a5ef902e/rbd_data.10b06b8b4567.0000000000003767/head//3
2014-06-12 14:51:24.233685 7f9d60bd2700 10 getattrs 3.2e_head/a5ef902e/rbd_data.10b06b8b4567.0000000000003767/head//3 = 0
2014-06-12 14:51:24.233703 7f9d60bd2700 10 stat 3.2e_head/da51a02e/rbd_data.10af6b8b4567.0000000000000267/head//3
2014-06-12 14:51:24.233817 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid da51a02e/rbd_data.10af6b8b4567.0000000000000267/head//3
2014-06-12 14:51:24.284981 7f9d60bd2700 10 getattrs 3.2e_head/da51a02e/rbd_data.10af6b8b4567.0000000000000267/head//3 = 0
2014-06-12 14:51:24.285004 7f9d60bd2700 10 stat 3.2e_head/6dc1a02e/rbd_data.109d6b8b4567.0000000000003928/head//3
2014-06-12 14:51:24.285126 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6dc1a02e/rbd_data.109d6b8b4567.0000000000003928/head//3
2014-06-12 14:51:24.299407 7f9d60bd2700 10 getattrs 3.2e_head/6dc1a02e/rbd_data.109d6b8b4567.0000000000003928/head//3 = 0
2014-06-12 14:51:24.299423 7f9d60bd2700 10 stat 3.2e_head/43d2a02e/rbd_data.10b26b8b4567.000000000000385c/head//3
2014-06-12 14:51:24.299534 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 43d2a02e/rbd_data.10b26b8b4567.000000000000385c/head//3
2014-06-12 14:51:24.313164 7f9d60bd2700 10 getattrs 3.2e_head/43d2a02e/rbd_data.10b26b8b4567.000000000000385c/head//3 = 0
2014-06-12 14:51:24.313175 7f9d60bd2700 10 stat 3.2e_head/de53a02e/rbd_data.10b36b8b4567.000000000000284b/head//3
2014-06-12 14:51:24.313299 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid de53a02e/rbd_data.10b36b8b4567.000000000000284b/head//3
2014-06-12 14:51:24.316289 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:24.325793 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:24.343673 7f9d60bd2700 10 getattrs 3.2e_head/de53a02e/rbd_data.10b36b8b4567.000000000000284b/head//3 = 0
2014-06-12 14:51:24.343684 7f9d60bd2700 10 stat 3.2e_head/6e17a02e/rbd_data.10ac6b8b4567.00000000000007d1/head//3
2014-06-12 14:51:24.343810 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6e17a02e/rbd_data.10ac6b8b4567.00000000000007d1/head//3
2014-06-12 14:51:24.365718 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:24.370510 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:24.381319 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:24.395006 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:24.395086 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:24.397678 7f9d60bd2700 10 getattrs 3.2e_head/6e17a02e/rbd_data.10ac6b8b4567.00000000000007d1/head//3 = 0
2014-06-12 14:51:24.397706 7f9d60bd2700 10 stat 3.2e_head/c8cba02e/rbd_data.10a26b8b4567.0000000000002b8f/head//3
2014-06-12 14:51:24.397796 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid c8cba02e/rbd_data.10a26b8b4567.0000000000002b8f/head//3
2014-06-12 14:51:24.417970 7f9d60bd2700 10 getattrs 3.2e_head/c8cba02e/rbd_data.10a26b8b4567.0000000000002b8f/head//3 = 0
2014-06-12 14:51:24.417979 7f9d60bd2700 10 stat 3.2e_head/9b0ea02e/rbd_data.10ae6b8b4567.0000000000001c29/head//3
2014-06-12 14:51:24.418079 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 9b0ea02e/rbd_data.10ae6b8b4567.0000000000001c29/head//3
2014-06-12 14:51:24.449469 7f9d60bd2700 10 getattrs 3.2e_head/9b0ea02e/rbd_data.10ae6b8b4567.0000000000001c29/head//3 = 0
2014-06-12 14:51:24.449481 7f9d60bd2700 10 stat 3.2e_head/6f0ea02e/rbd_data.108e6b8b4567.000000000000181f/head//3
2014-06-12 14:51:24.449592 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6f0ea02e/rbd_data.108e6b8b4567.000000000000181f/head//3
2014-06-12 14:51:24.469035 7f9d60bd2700 10 getattrs 3.2e_head/6f0ea02e/rbd_data.108e6b8b4567.000000000000181f/head//3 = 0
2014-06-12 14:51:24.469046 7f9d60bd2700 10 stat 3.2e_head/c001b02e/rbd_data.10946b8b4567.0000000000000319/head//3
2014-06-12 14:51:24.469168 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid c001b02e/rbd_data.10946b8b4567.0000000000000319/head//3
2014-06-12 14:51:24.504701 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:24.512540 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:24.518357 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:24.522332 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:24.526819 7f9d60bd2700 10 getattrs 3.2e_head/c001b02e/rbd_data.10946b8b4567.0000000000000319/head//3 = 0
2014-06-12 14:51:24.526834 7f9d60bd2700 10 stat 3.2e_head/28a4b02e/rbd_data.10ab6b8b4567.0000000000001530/head//3
2014-06-12 14:51:24.526963 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 28a4b02e/rbd_data.10ab6b8b4567.0000000000001530/head//3
2014-06-12 14:51:24.531728 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:24.556584 7f9d44101700  5 -- op tracker -- , seq: 748572, time: 2014-06-12 14:51:24.556493, event: header_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:24.556600 7f9d44101700  5 -- op tracker -- , seq: 748572, time: 2014-06-12 14:51:24.556495, event: throttled, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:24.556609 7f9d44101700  5 -- op tracker -- , seq: 748572, time: 2014-06-12 14:51:24.556546, event: all_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:24.556631 7f9d44101700  5 -- op tracker -- , seq: 748572, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:24.556637 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:24.556643 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:24.556651 7f9d44101700 15 osd.2 173 enqueue_op 0x23e09400 prio 127 cost 10717 latency 0.000157 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:24.556720 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:24.557823 7f9d60bd2700 10 getattrs 3.2e_head/28a4b02e/rbd_data.10ab6b8b4567.0000000000001530/head//3 = 0
2014-06-12 14:51:24.557835 7f9d60bd2700 10 stat 3.2e_head/c317b02e/rbd_data.10b46b8b4567.0000000000001275/head//3
2014-06-12 14:51:24.557967 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid c317b02e/rbd_data.10b46b8b4567.0000000000001275/head//3
2014-06-12 14:51:24.591891 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:24.592030 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:24.592935 7f9d60bd2700 10 getattrs 3.2e_head/c317b02e/rbd_data.10b46b8b4567.0000000000001275/head//3 = 0
2014-06-12 14:51:24.592947 7f9d60bd2700 10 stat 3.2e_head/9b48b02e/rbd_data.10af6b8b4567.0000000000002b67/head//3
2014-06-12 14:51:24.593077 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 9b48b02e/rbd_data.10af6b8b4567.0000000000002b67/head//3
2014-06-12 14:51:24.624180 7f9d60bd2700 10 getattrs 3.2e_head/9b48b02e/rbd_data.10af6b8b4567.0000000000002b67/head//3 = 0
2014-06-12 14:51:24.624189 7f9d60bd2700 10 stat 3.2e_head/be09b02e/rbd_data.10b36b8b4567.0000000000000583/head//3
2014-06-12 14:51:24.624279 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid be09b02e/rbd_data.10b36b8b4567.0000000000000583/head//3
2014-06-12 14:51:24.656676 7f9d60bd2700 10 getattrs 3.2e_head/be09b02e/rbd_data.10b36b8b4567.0000000000000583/head//3 = 0
2014-06-12 14:51:24.656688 7f9d60bd2700 10 stat 3.2e_head/87cb02e/rbd_data.10b26b8b4567.000000000000124f/head//3
2014-06-12 14:51:24.656793 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 87cb02e/rbd_data.10b26b8b4567.000000000000124f/head//3
2014-06-12 14:51:24.661086 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:24.674281 7f9d60bd2700 10 getattrs 3.2e_head/87cb02e/rbd_data.10b26b8b4567.000000000000124f/head//3 = 0
2014-06-12 14:51:24.674291 7f9d60bd2700 10 stat 3.2e_head/b60c02e/rbd_data.10af6b8b4567.0000000000000050/head//3
2014-06-12 14:51:24.674407 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid b60c02e/rbd_data.10af6b8b4567.0000000000000050/head//3
2014-06-12 14:51:24.683103 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:24.683163 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:24.700059 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:24.716733 7f9d60bd2700 10 getattrs 3.2e_head/b60c02e/rbd_data.10af6b8b4567.0000000000000050/head//3 = 0
2014-06-12 14:51:24.716755 7f9d60bd2700 10 collection_getattrs 3.2e_head
2014-06-12 14:51:24.716825 7f9d60bd2700 10 collection_getattrs 3.2e_head = 0
2014-06-12 14:51:24.716830 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:24.716877 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:24.716886 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:24.716900 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:24.716904 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:24.716914 7f9d683e1700 10 osd.2 173 dequeue_op 0x23e09400 prio 127 cost 10717 latency 0.160420 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]
2014-06-12 14:51:24.716966 7f9d683e1700  5 -- op tracker -- , seq: 748572, time: 2014-06-12 14:51:24.716950, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:24.716985 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] handle_message: 0x23e09400
2014-06-12 14:51:24.717018 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:24.717055 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:24.717090 7f9d683e1700  5 -- op tracker -- , seq: 748572, time: 2014-06-12 14:51:24.717080, event: started, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:24.717111 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:24.717240 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] map version is 173'2847
2014-06-12 14:51:24.717267 7f9d683e1700 10 osd.2 173 dequeue_op 0x23e09400 finish
2014-06-12 14:51:24.717288 7f9d683e1700  5 -- op tracker -- , seq: 748572, time: 2014-06-12 14:51:24.717277, event: done, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:24.717306 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:24.717322 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:24.717338 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:24.717319 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:24.717347 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:24.717356 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:24.717391 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:24.717400 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:24.717409 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:24.717443 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d350802e/rbd_data.10a06b8b4567.00000000000038d5/head//3, auth == maps.end()
2014-06-12 14:51:24.717466 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d350802e/rbd_data.10a06b8b4567.00000000000038d5/head//3
2014-06-12 14:51:24.717482 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 59d2802e/rbd_data.108e6b8b4567.0000000000003aec/head//3, auth == maps.end()
2014-06-12 14:51:24.717494 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 59d2802e/rbd_data.108e6b8b4567.0000000000003aec/head//3
2014-06-12 14:51:24.717506 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b457802e/rbd_data.10a06b8b4567.00000000000005ae/head//3, auth == maps.end()
2014-06-12 14:51:24.717518 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b457802e/rbd_data.10a06b8b4567.00000000000005ae/head//3
2014-06-12 14:51:24.717530 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 998802e/rbd_data.10af6b8b4567.000000000000290e/head//3, auth == maps.end()
2014-06-12 14:51:24.717541 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 998802e/rbd_data.10af6b8b4567.000000000000290e/head//3
2014-06-12 14:51:24.717553 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dcf802e/rbd_data.10906b8b4567.000000000000399d/head//3, auth == maps.end()
2014-06-12 14:51:24.717564 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj dcf802e/rbd_data.10906b8b4567.000000000000399d/head//3
2014-06-12 14:51:24.717575 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 181a902e/rbd_data.10936b8b4567.0000000000001853/head//3, auth == maps.end()
2014-06-12 14:51:24.717586 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 181a902e/rbd_data.10936b8b4567.0000000000001853/head//3
2014-06-12 14:51:24.717598 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8f1e902e/rbd_data.10a26b8b4567.0000000000000805/head//3, auth == maps.end()
2014-06-12 14:51:24.717609 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8f1e902e/rbd_data.10a26b8b4567.0000000000000805/head//3
2014-06-12 14:51:24.717640 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8e2f902e/rbd_data.10a96b8b4567.0000000000003408/head//3, auth == maps.end()
2014-06-12 14:51:24.717652 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8e2f902e/rbd_data.10a96b8b4567.0000000000003408/head//3
2014-06-12 14:51:24.717665 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f34f902e/rbd_data.10936b8b4567.00000000000028ef/head//3, auth == maps.end()
2014-06-12 14:51:24.717676 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f34f902e/rbd_data.10936b8b4567.00000000000028ef/head//3
2014-06-12 14:51:24.717688 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a5ef902e/rbd_data.10b06b8b4567.0000000000003767/head//3, auth == maps.end()
2014-06-12 14:51:24.717699 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a5ef902e/rbd_data.10b06b8b4567.0000000000003767/head//3
2014-06-12 14:51:24.717711 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj da51a02e/rbd_data.10af6b8b4567.0000000000000267/head//3, auth == maps.end()
2014-06-12 14:51:24.717725 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj da51a02e/rbd_data.10af6b8b4567.0000000000000267/head//3
2014-06-12 14:51:24.717738 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6dc1a02e/rbd_data.109d6b8b4567.0000000000003928/head//3, auth == maps.end()
2014-06-12 14:51:24.717751 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6dc1a02e/rbd_data.109d6b8b4567.0000000000003928/head//3
2014-06-12 14:51:24.717765 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 43d2a02e/rbd_data.10b26b8b4567.000000000000385c/head//3, auth == maps.end()
2014-06-12 14:51:24.717777 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 43d2a02e/rbd_data.10b26b8b4567.000000000000385c/head//3
2014-06-12 14:51:24.717789 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj de53a02e/rbd_data.10b36b8b4567.000000000000284b/head//3, auth == maps.end()
2014-06-12 14:51:24.717802 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj de53a02e/rbd_data.10b36b8b4567.000000000000284b/head//3
2014-06-12 14:51:24.717815 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6e17a02e/rbd_data.10ac6b8b4567.00000000000007d1/head//3, auth == maps.end()
2014-06-12 14:51:24.717827 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6e17a02e/rbd_data.10ac6b8b4567.00000000000007d1/head//3
2014-06-12 14:51:24.717839 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c8cba02e/rbd_data.10a26b8b4567.0000000000002b8f/head//3, auth == maps.end()
2014-06-12 14:51:24.717850 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c8cba02e/rbd_data.10a26b8b4567.0000000000002b8f/head//3
2014-06-12 14:51:24.717862 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9b0ea02e/rbd_data.10ae6b8b4567.0000000000001c29/head//3, auth == maps.end()
2014-06-12 14:51:24.717876 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9b0ea02e/rbd_data.10ae6b8b4567.0000000000001c29/head//3
2014-06-12 14:51:24.717888 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6f0ea02e/rbd_data.108e6b8b4567.000000000000181f/head//3, auth == maps.end()
2014-06-12 14:51:24.717899 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6f0ea02e/rbd_data.108e6b8b4567.000000000000181f/head//3
2014-06-12 14:51:24.717913 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c001b02e/rbd_data.10946b8b4567.0000000000000319/head//3, auth == maps.end()
2014-06-12 14:51:24.717925 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c001b02e/rbd_data.10946b8b4567.0000000000000319/head//3
2014-06-12 14:51:24.717939 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 28a4b02e/rbd_data.10ab6b8b4567.0000000000001530/head//3, auth == maps.end()
2014-06-12 14:51:24.717951 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 28a4b02e/rbd_data.10ab6b8b4567.0000000000001530/head//3
2014-06-12 14:51:24.717965 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c317b02e/rbd_data.10b46b8b4567.0000000000001275/head//3, auth == maps.end()
2014-06-12 14:51:24.717978 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c317b02e/rbd_data.10b46b8b4567.0000000000001275/head//3
2014-06-12 14:51:24.717990 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9b48b02e/rbd_data.10af6b8b4567.0000000000002b67/head//3, auth == maps.end()
2014-06-12 14:51:24.718002 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9b48b02e/rbd_data.10af6b8b4567.0000000000002b67/head//3
2014-06-12 14:51:24.718015 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj be09b02e/rbd_data.10b36b8b4567.0000000000000583/head//3, auth == maps.end()
2014-06-12 14:51:24.718029 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj be09b02e/rbd_data.10b36b8b4567.0000000000000583/head//3
2014-06-12 14:51:24.718043 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 87cb02e/rbd_data.10b26b8b4567.000000000000124f/head//3, auth == maps.end()
2014-06-12 14:51:24.718056 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 87cb02e/rbd_data.10b26b8b4567.000000000000124f/head//3
2014-06-12 14:51:24.718069 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b60c02e/rbd_data.10af6b8b4567.0000000000000050/head//3, auth == maps.end()
2014-06-12 14:51:24.718082 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b60c02e/rbd_data.10af6b8b4567.0000000000000050/head//3
2014-06-12 14:51:24.718097 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] 
2014-06-12 14:51:24.718107 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub
2014-06-12 14:51:24.718124 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  b60c02e/rbd_data.10af6b8b4567.0000000000000050/head//3 b60c02e/rbd_data.10af6b8b4567.0000000000000050/head//3(173'2343 client.4312.0:121061 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2343)
2014-06-12 14:51:24.718141 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  87cb02e/rbd_data.10b26b8b4567.000000000000124f/head//3 87cb02e/rbd_data.10b26b8b4567.000000000000124f/head//3(173'847 client.4315.0:42048 wrlock_by=unknown.0.0:0 dirty s 4194304 uv847)
2014-06-12 14:51:24.718159 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  be09b02e/rbd_data.10b36b8b4567.0000000000000583/head//3 be09b02e/rbd_data.10b36b8b4567.0000000000000583/head//3(173'2531 client.4316.0:127526 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2531)
2014-06-12 14:51:24.718175 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  9b48b02e/rbd_data.10af6b8b4567.0000000000002b67/head//3 9b48b02e/rbd_data.10af6b8b4567.0000000000002b67/head//3(173'1717 client.4312.0:89984 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1717)
2014-06-12 14:51:24.718191 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  c317b02e/rbd_data.10b46b8b4567.0000000000001275/head//3 c317b02e/rbd_data.10b46b8b4567.0000000000001275/head//3(173'871 client.4317.0:42548 wrlock_by=unknown.0.0:0 dirty s 4194304 uv871)
2014-06-12 14:51:24.718208 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  28a4b02e/rbd_data.10ab6b8b4567.0000000000001530/head//3 28a4b02e/rbd_data.10ab6b8b4567.0000000000001530/head//3(173'1029 client.4308.0:48539 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1029)
2014-06-12 14:51:24.718225 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  c001b02e/rbd_data.10946b8b4567.0000000000000319/head//3 c001b02e/rbd_data.10946b8b4567.0000000000000319/head//3(173'2461 client.4285.0:120671 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2461)
2014-06-12 14:51:24.718242 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6f0ea02e/rbd_data.108e6b8b4567.000000000000181f/head//3 6f0ea02e/rbd_data.108e6b8b4567.000000000000181f/head//3(173'1079 client.4279.0:53078 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1079)
2014-06-12 14:51:24.718259 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  9b0ea02e/rbd_data.10ae6b8b4567.0000000000001c29/head//3 9b0ea02e/rbd_data.10ae6b8b4567.0000000000001c29/head//3(173'1156 client.4311.0:59367 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1156)
2014-06-12 14:51:24.718276 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  c8cba02e/rbd_data.10a26b8b4567.0000000000002b8f/head//3 c8cba02e/rbd_data.10a26b8b4567.0000000000002b8f/head//3(173'1858 client.4299.0:90228 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1858)
2014-06-12 14:51:24.718293 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6e17a02e/rbd_data.10ac6b8b4567.00000000000007d1/head//3 6e17a02e/rbd_data.10ac6b8b4567.00000000000007d1/head//3(173'2607 client.4309.0:128497 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2607)
2014-06-12 14:51:24.718309 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  de53a02e/rbd_data.10b36b8b4567.000000000000284b/head//3 de53a02e/rbd_data.10b36b8b4567.000000000000284b/head//3(173'1631 client.4316.0:83216 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1631)
2014-06-12 14:51:24.718324 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  43d2a02e/rbd_data.10b26b8b4567.000000000000385c/head//3 43d2a02e/rbd_data.10b26b8b4567.000000000000385c/head//3(173'2064 client.4315.0:111574 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2064)
2014-06-12 14:51:24.718340 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6dc1a02e/rbd_data.109d6b8b4567.0000000000003928/head//3 6dc1a02e/rbd_data.109d6b8b4567.0000000000003928/head//3(173'2210 client.4294.0:112153 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2210)
2014-06-12 14:51:24.718357 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  da51a02e/rbd_data.10af6b8b4567.0000000000000267/head//3 da51a02e/rbd_data.10af6b8b4567.0000000000000267/head//3(173'2440 client.4312.0:123808 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2440)
2014-06-12 14:51:24.718378 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a5ef902e/rbd_data.10b06b8b4567.0000000000003767/head//3 a5ef902e/rbd_data.10b06b8b4567.0000000000003767/head//3(173'2133 client.4313.0:111189 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2133)
2014-06-12 14:51:24.718394 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f34f902e/rbd_data.10936b8b4567.00000000000028ef/head//3 f34f902e/rbd_data.10936b8b4567.00000000000028ef/head//3(173'1637 client.4284.0:81162 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1637)
2014-06-12 14:51:24.718411 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8e2f902e/rbd_data.10a96b8b4567.0000000000003408/head//3 8e2f902e/rbd_data.10a96b8b4567.0000000000003408/head//3(173'2194 client.4306.0:106066 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2194)
2014-06-12 14:51:24.718428 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8f1e902e/rbd_data.10a26b8b4567.0000000000000805/head//3 8f1e902e/rbd_data.10a26b8b4567.0000000000000805/head//3(173'2627 client.4299.0:132803 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2627)
2014-06-12 14:51:24.718444 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  181a902e/rbd_data.10936b8b4567.0000000000001853/head//3 181a902e/rbd_data.10936b8b4567.0000000000001853/head//3(173'1133 client.4284.0:51170 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1133)
2014-06-12 14:51:24.718461 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  dcf802e/rbd_data.10906b8b4567.000000000000399d/head//3 dcf802e/rbd_data.10906b8b4567.000000000000399d/head//3(173'2109 client.4281.0:113188 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2109)
2014-06-12 14:51:24.718478 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  998802e/rbd_data.10af6b8b4567.000000000000290e/head//3 998802e/rbd_data.10af6b8b4567.000000000000290e/head//3(173'1646 client.4312.0:85663 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1646)
2014-06-12 14:51:24.718493 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  b457802e/rbd_data.10a06b8b4567.00000000000005ae/head//3 b457802e/rbd_data.10a06b8b4567.00000000000005ae/head//3(173'2548 client.4297.0:125721 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2548)
2014-06-12 14:51:24.718510 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  59d2802e/rbd_data.108e6b8b4567.0000000000003aec/head//3 59d2802e/rbd_data.108e6b8b4567.0000000000003aec/head//3(173'2267 client.4279.0:115711 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2267)
2014-06-12 14:51:24.718526 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d350802e/rbd_data.10a06b8b4567.00000000000038d5/head//3 d350802e/rbd_data.10a06b8b4567.00000000000038d5/head//3(173'2152 client.4297.0:111279 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2152)
2014-06-12 14:51:24.718539 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:24.718553 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:24.718564 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:24.718567 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:24.718569 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:24.718611 7f9d60bd2700 10 collection_list_partial 3.2e_head start:ba31c02e//0//-1 is_max:0
2014-06-12 14:51:24.718872 7f9d60bd2700 20 objects: [34a2c02e/rbd_data.10a46b8b4567.00000000000012a4/head//3,abe4c02e/rbd_data.10af6b8b4567.0000000000002043/head//3,9217c02e/rbd_data.10a16b8b4567.00000000000029f3/head//3,dd18c02e/rbd_data.10ab6b8b4567.000000000000133e/head//3,785ac02e/rbd_data.10a16b8b4567.000000000000127e/head//3,2a7ac02e/rbd_data.10b06b8b4567.00000000000024f3/head//3,cfcac02e/rbd_data.10946b8b4567.0000000000002290/head//3,f54bc02e/rbd_data.109a6b8b4567.00000000000010a3/head//3,72ec02e/rbd_data.10ad6b8b4567.00000000000006de/head//3,46fc02e/rbd_data.10976b8b4567.0000000000000632/head//3,afbfc02e/rbd_data.109c6b8b4567.00000000000031eb/head//3,8cb0d02e/rbd_data.10a46b8b4567.00000000000037a4/head//3,86c1d02e/rbd_data.10976b8b4567.0000000000003abc/head//3,18f3d02e/rbd_data.10ab6b8b4567.0000000000000c93/head//3,e55d02e/rbd_data.10ac6b8b4567.0000000000002dd9/head//3,b7f6d02e/rbd_data.109b6b8b4567.000000000000183f/head//3,3af7d02e/rbd_data.10946b8b4567.00000000000015f6/head//3,647ad02e/rbd_data.10ad6b8b4567.000000000000363a/head//3,d6fdd02e/rbd_data.10986b8b4567.0000000000001c7f/head//3,e570e02e/rbd_data.10966b8b4567.0000000000001aab/head//3,ed0e02e/rbd_data.10a96b8b4567.000000000000385a/head//3,3122e02e/rbd_data.10a36b8b4567.0000000000001e3d/head//3,5824e02e/rbd_data.10956b8b4567.0000000000002e63/head//3,e9e6e02e/rbd_data.10af6b8b4567.0000000000000dcd/head//3,8319e02e/rbd_data.10a96b8b4567.0000000000000398/head//3]
2014-06-12 14:51:24.719416 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:24.719450 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:24.719470 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:24.719482 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:24.719490 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:24.719497 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:24.719505 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_map_chunk [ba31c02e//0//-1,23cbe02e//0//-1)
2014-06-12 14:51:24.719514 7f9d60bd2700 10 collection_list_partial 3.2e_head start:ba31c02e//0//-1 is_max:0
2014-06-12 14:51:24.720120 7f9d60bd2700 20 objects: [34a2c02e/rbd_data.10a46b8b4567.00000000000012a4/head//3,abe4c02e/rbd_data.10af6b8b4567.0000000000002043/head//3,9217c02e/rbd_data.10a16b8b4567.00000000000029f3/head//3,dd18c02e/rbd_data.10ab6b8b4567.000000000000133e/head//3,785ac02e/rbd_data.10a16b8b4567.000000000000127e/head//3,2a7ac02e/rbd_data.10b06b8b4567.00000000000024f3/head//3,cfcac02e/rbd_data.10946b8b4567.0000000000002290/head//3,f54bc02e/rbd_data.109a6b8b4567.00000000000010a3/head//3,72ec02e/rbd_data.10ad6b8b4567.00000000000006de/head//3,46fc02e/rbd_data.10976b8b4567.0000000000000632/head//3,afbfc02e/rbd_data.109c6b8b4567.00000000000031eb/head//3,8cb0d02e/rbd_data.10a46b8b4567.00000000000037a4/head//3,86c1d02e/rbd_data.10976b8b4567.0000000000003abc/head//3,18f3d02e/rbd_data.10ab6b8b4567.0000000000000c93/head//3,e55d02e/rbd_data.10ac6b8b4567.0000000000002dd9/head//3,b7f6d02e/rbd_data.109b6b8b4567.000000000000183f/head//3,3af7d02e/rbd_data.10946b8b4567.00000000000015f6/head//3,647ad02e/rbd_data.10ad6b8b4567.000000000000363a/head//3,d6fdd02e/rbd_data.10986b8b4567.0000000000001c7f/head//3,e570e02e/rbd_data.10966b8b4567.0000000000001aab/head//3,ed0e02e/rbd_data.10a96b8b4567.000000000000385a/head//3,3122e02e/rbd_data.10a36b8b4567.0000000000001e3d/head//3,5824e02e/rbd_data.10956b8b4567.0000000000002e63/head//3,e9e6e02e/rbd_data.10af6b8b4567.0000000000000dcd/head//3,8319e02e/rbd_data.10a96b8b4567.0000000000000398/head//3,23cbe02e/rbd_data.109d6b8b4567.00000000000023c0/head//3,fe6ce02e/rbd_data.109b6b8b4567.0000000000001594/head//3,e162f02e/rbd_data.108f6b8b4567.0000000000001694/head//3,2646f02e/rbd_data.109e6b8b4567.00000000000024fb/head//3,6fe7f02e/rbd_data.10a06b8b4567.000000000000000a/head//3,f9cbf02e/rbd_data.10986b8b4567.000000000000190b/head//3,d2ef02e/rbd_data.10ad6b8b4567.0000000000002d19/head//3,5a0ff02e/rbd_data.10ad6b8b4567.00000000000000b3/head//3,a86ff02e/rbd_data.109c6b8b4567.0000000000000325/head//3,8b71082e/rbd_data.10996b8b4567.0000000000000ded/head//3,cd91082e/rbd_data.10b16b8b4567.0000000000001e29/head//3,ed23082e/rbd_data.10a86b8b4567.00000000000015f8/head//3,5d85082e/rbd_data.10a66b8b4567.00000000000030b5/head//3,e00c082e/rbd_data.10a16b8b4567.00000000000035b4/head//3,244c082e/rbd_data.10b26b8b4567.00000000000034e7/head//3,7c8c082e/rbd_data.10996b8b4567.0000000000003ac3/head//3,4b9c082e/rbd_data.108f6b8b4567.0000000000001232/head//3,27ad082e/rbd_data.10b06b8b4567.00000000000022b9/head//3,9ffe082e/rbd_data.109e6b8b4567.00000000000005a0/head//3,fd9f082e/rbd_data.10ac6b8b4567.0000000000003102/head//3,2f0182e/rbd_data.10ac6b8b4567.0000000000000dfe/head//3,a762182e/rbd_data.10ab6b8b4567.0000000000002f00/head//3,b153182e/rbd_data.10946b8b4567.0000000000000fa5/head//3,bed3182e/rbd_data.109c6b8b4567.00000000000017a4/head//3,53b4182e/rbd_data.108e6b8b4567.0000000000002760/head//3,2028182e/rbd_data.10aa6b8b4567.0000000000003580/head//3,919182e/rbd_data.108d6b8b4567.0000000000003342/head//3,7a3e182e/rbd_data.109c6b8b4567.0000000000002ee6/head//3,a58e182e/rbd_data.108e6b8b4567.0000000000002612/head//3,177f182e/rbd_data.10ac6b8b4567.00000000000025ad/head//3,fd02282e/rbd_data.10a76b8b4567.0000000000003a8c/head//3,2f24282e/rbd_data.10b36b8b4567.00000000000016fc/head//3,7e85282e/rbd_data.10926b8b4567.00000000000015b2/head//3,5618282e/rbd_data.109b6b8b4567.0000000000001c3b/head//3,9378282e/rbd_data.10b06b8b4567.0000000000003a8a/head//3,2fbb282e/rbd_data.10a36b8b4567.00000000000001d8/head//3,777f282e/rbd_data.10956b8b4567.00000000000014f5/head//3,cc9f282e/rbd_data.109d6b8b4567.00000000000024bc/head//3,b383382e/rbd_data.10ae6b8b4567.000000000000052d/head//3]
2014-06-12 14:51:24.720218 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:24.720237 7f9d60bd2700 10 stat 3.2e_head/34a2c02e/rbd_data.10a46b8b4567.00000000000012a4/head//3
2014-06-12 14:51:24.720393 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 34a2c02e/rbd_data.10a46b8b4567.00000000000012a4/head//3
2014-06-12 14:51:24.744741 7f9d60bd2700 10 getattrs 3.2e_head/34a2c02e/rbd_data.10a46b8b4567.00000000000012a4/head//3 = 0
2014-06-12 14:51:24.744749 7f9d60bd2700 10 stat 3.2e_head/abe4c02e/rbd_data.10af6b8b4567.0000000000002043/head//3
2014-06-12 14:51:24.744847 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid abe4c02e/rbd_data.10af6b8b4567.0000000000002043/head//3
2014-06-12 14:51:24.767383 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:24.774822 7f9d60bd2700 10 getattrs 3.2e_head/abe4c02e/rbd_data.10af6b8b4567.0000000000002043/head//3 = 0
2014-06-12 14:51:24.774836 7f9d60bd2700 10 stat 3.2e_head/9217c02e/rbd_data.10a16b8b4567.00000000000029f3/head//3
2014-06-12 14:51:24.774977 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 9217c02e/rbd_data.10a16b8b4567.00000000000029f3/head//3
2014-06-12 14:51:24.793150 7f9d60bd2700 10 getattrs 3.2e_head/9217c02e/rbd_data.10a16b8b4567.00000000000029f3/head//3 = 0
2014-06-12 14:51:24.793161 7f9d60bd2700 10 stat 3.2e_head/dd18c02e/rbd_data.10ab6b8b4567.000000000000133e/head//3
2014-06-12 14:51:24.793300 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid dd18c02e/rbd_data.10ab6b8b4567.000000000000133e/head//3
2014-06-12 14:51:24.801347 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:24.807033 7f9d60bd2700 10 getattrs 3.2e_head/dd18c02e/rbd_data.10ab6b8b4567.000000000000133e/head//3 = 0
2014-06-12 14:51:24.807040 7f9d60bd2700 10 stat 3.2e_head/785ac02e/rbd_data.10a16b8b4567.000000000000127e/head//3
2014-06-12 14:51:24.807137 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 785ac02e/rbd_data.10a16b8b4567.000000000000127e/head//3
2014-06-12 14:51:24.807788 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:24.837334 7f9d60bd2700 10 getattrs 3.2e_head/785ac02e/rbd_data.10a16b8b4567.000000000000127e/head//3 = 0
2014-06-12 14:51:24.837342 7f9d60bd2700 10 stat 3.2e_head/2a7ac02e/rbd_data.10b06b8b4567.00000000000024f3/head//3
2014-06-12 14:51:24.837439 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 2a7ac02e/rbd_data.10b06b8b4567.00000000000024f3/head//3
2014-06-12 14:51:24.854454 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:24.862012 7f9d60bd2700 10 getattrs 3.2e_head/2a7ac02e/rbd_data.10b06b8b4567.00000000000024f3/head//3 = 0
2014-06-12 14:51:24.862023 7f9d60bd2700 10 stat 3.2e_head/cfcac02e/rbd_data.10946b8b4567.0000000000002290/head//3
2014-06-12 14:51:24.862119 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid cfcac02e/rbd_data.10946b8b4567.0000000000002290/head//3
2014-06-12 14:51:24.864808 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:24.868196 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:24.871705 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:24.871758 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.46 < max 0.5 = yes
2014-06-12 14:51:24.871775 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=1
2014-06-12 14:51:24.871777 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:24.878625 7f9d60bd2700 10 getattrs 3.2e_head/cfcac02e/rbd_data.10946b8b4567.0000000000002290/head//3 = 0
2014-06-12 14:51:24.878632 7f9d60bd2700 10 stat 3.2e_head/f54bc02e/rbd_data.109a6b8b4567.00000000000010a3/head//3
2014-06-12 14:51:24.878724 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f54bc02e/rbd_data.109a6b8b4567.00000000000010a3/head//3
2014-06-12 14:51:24.878824 7f9d60bd2700 10 getattrs 3.2e_head/f54bc02e/rbd_data.109a6b8b4567.00000000000010a3/head//3 = 0
2014-06-12 14:51:24.878829 7f9d60bd2700 10 stat 3.2e_head/72ec02e/rbd_data.10ad6b8b4567.00000000000006de/head//3
2014-06-12 14:51:24.878922 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 72ec02e/rbd_data.10ad6b8b4567.00000000000006de/head//3
2014-06-12 14:51:24.927890 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:24.928927 7f9d60bd2700 10 getattrs 3.2e_head/72ec02e/rbd_data.10ad6b8b4567.00000000000006de/head//3 = 0
2014-06-12 14:51:24.928942 7f9d60bd2700 10 stat 3.2e_head/46fc02e/rbd_data.10976b8b4567.0000000000000632/head//3
2014-06-12 14:51:24.929056 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 46fc02e/rbd_data.10976b8b4567.0000000000000632/head//3
2014-06-12 14:51:24.935371 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:24.961746 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:24.965457 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:24.968968 7f9d60bd2700 10 getattrs 3.2e_head/46fc02e/rbd_data.10976b8b4567.0000000000000632/head//3 = 0
2014-06-12 14:51:24.968991 7f9d60bd2700 10 stat 3.2e_head/afbfc02e/rbd_data.109c6b8b4567.00000000000031eb/head//3
2014-06-12 14:51:24.969150 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid afbfc02e/rbd_data.109c6b8b4567.00000000000031eb/head//3
2014-06-12 14:51:25.002684 7f9d60bd2700 10 getattrs 3.2e_head/afbfc02e/rbd_data.109c6b8b4567.00000000000031eb/head//3 = 0
2014-06-12 14:51:25.002700 7f9d60bd2700 10 stat 3.2e_head/8cb0d02e/rbd_data.10a46b8b4567.00000000000037a4/head//3
2014-06-12 14:51:25.002867 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8cb0d02e/rbd_data.10a46b8b4567.00000000000037a4/head//3
2014-06-12 14:51:25.014817 7f9d60bd2700 10 getattrs 3.2e_head/8cb0d02e/rbd_data.10a46b8b4567.00000000000037a4/head//3 = 0
2014-06-12 14:51:25.014831 7f9d60bd2700 10 stat 3.2e_head/86c1d02e/rbd_data.10976b8b4567.0000000000003abc/head//3
2014-06-12 14:51:25.014995 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 86c1d02e/rbd_data.10976b8b4567.0000000000003abc/head//3
2014-06-12 14:51:25.032039 7f9d60bd2700 10 getattrs 3.2e_head/86c1d02e/rbd_data.10976b8b4567.0000000000003abc/head//3 = 0
2014-06-12 14:51:25.032046 7f9d60bd2700 10 stat 3.2e_head/18f3d02e/rbd_data.10ab6b8b4567.0000000000000c93/head//3
2014-06-12 14:51:25.032141 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 18f3d02e/rbd_data.10ab6b8b4567.0000000000000c93/head//3
2014-06-12 14:51:25.086269 7f9d60bd2700 10 getattrs 3.2e_head/18f3d02e/rbd_data.10ab6b8b4567.0000000000000c93/head//3 = 0
2014-06-12 14:51:25.086278 7f9d60bd2700 10 stat 3.2e_head/e55d02e/rbd_data.10ac6b8b4567.0000000000002dd9/head//3
2014-06-12 14:51:25.086372 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid e55d02e/rbd_data.10ac6b8b4567.0000000000002dd9/head//3
2014-06-12 14:51:25.119822 7f9d60bd2700 10 getattrs 3.2e_head/e55d02e/rbd_data.10ac6b8b4567.0000000000002dd9/head//3 = 0
2014-06-12 14:51:25.119830 7f9d60bd2700 10 stat 3.2e_head/b7f6d02e/rbd_data.109b6b8b4567.000000000000183f/head//3
2014-06-12 14:51:25.119929 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid b7f6d02e/rbd_data.109b6b8b4567.000000000000183f/head//3
2014-06-12 14:51:25.120708 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:25.132358 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:25.141845 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:25.145347 7f9d60bd2700 10 getattrs 3.2e_head/b7f6d02e/rbd_data.109b6b8b4567.000000000000183f/head//3 = 0
2014-06-12 14:51:25.145354 7f9d60bd2700 10 stat 3.2e_head/3af7d02e/rbd_data.10946b8b4567.00000000000015f6/head//3
2014-06-12 14:51:25.145454 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 3af7d02e/rbd_data.10946b8b4567.00000000000015f6/head//3
2014-06-12 14:51:25.151225 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:25.163469 7f9d60bd2700 10 getattrs 3.2e_head/3af7d02e/rbd_data.10946b8b4567.00000000000015f6/head//3 = 0
2014-06-12 14:51:25.163477 7f9d60bd2700 10 stat 3.2e_head/647ad02e/rbd_data.10ad6b8b4567.000000000000363a/head//3
2014-06-12 14:51:25.163566 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 647ad02e/rbd_data.10ad6b8b4567.000000000000363a/head//3
2014-06-12 14:51:25.176733 7f9d60bd2700 10 getattrs 3.2e_head/647ad02e/rbd_data.10ad6b8b4567.000000000000363a/head//3 = 0
2014-06-12 14:51:25.176742 7f9d60bd2700 10 stat 3.2e_head/d6fdd02e/rbd_data.10986b8b4567.0000000000001c7f/head//3
2014-06-12 14:51:25.176849 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d6fdd02e/rbd_data.10986b8b4567.0000000000001c7f/head//3
2014-06-12 14:51:25.179688 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:25.193699 7f9d60bd2700 10 getattrs 3.2e_head/d6fdd02e/rbd_data.10986b8b4567.0000000000001c7f/head//3 = 0
2014-06-12 14:51:25.193708 7f9d60bd2700 10 stat 3.2e_head/e570e02e/rbd_data.10966b8b4567.0000000000001aab/head//3
2014-06-12 14:51:25.193810 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid e570e02e/rbd_data.10966b8b4567.0000000000001aab/head//3
2014-06-12 14:51:25.208837 7f9d60bd2700 10 getattrs 3.2e_head/e570e02e/rbd_data.10966b8b4567.0000000000001aab/head//3 = 0
2014-06-12 14:51:25.208846 7f9d60bd2700 10 stat 3.2e_head/ed0e02e/rbd_data.10a96b8b4567.000000000000385a/head//3
2014-06-12 14:51:25.208945 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid ed0e02e/rbd_data.10a96b8b4567.000000000000385a/head//3
2014-06-12 14:51:25.209018 7f9d60bd2700 10 getattrs 3.2e_head/ed0e02e/rbd_data.10a96b8b4567.000000000000385a/head//3 = 0
2014-06-12 14:51:25.209024 7f9d60bd2700 10 stat 3.2e_head/3122e02e/rbd_data.10a36b8b4567.0000000000001e3d/head//3
2014-06-12 14:51:25.209112 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 3122e02e/rbd_data.10a36b8b4567.0000000000001e3d/head//3
2014-06-12 14:51:25.228206 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:25.228358 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:25.240177 7f9d60bd2700 10 getattrs 3.2e_head/3122e02e/rbd_data.10a36b8b4567.0000000000001e3d/head//3 = 0
2014-06-12 14:51:25.240196 7f9d60bd2700 10 stat 3.2e_head/5824e02e/rbd_data.10956b8b4567.0000000000002e63/head//3
2014-06-12 14:51:25.240314 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 5824e02e/rbd_data.10956b8b4567.0000000000002e63/head//3
2014-06-12 14:51:25.257258 7f9d44101700  5 -- op tracker -- , seq: 748573, time: 2014-06-12 14:51:25.257185, event: header_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.257276 7f9d44101700  5 -- op tracker -- , seq: 748573, time: 2014-06-12 14:51:25.257187, event: throttled, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.257287 7f9d44101700  5 -- op tracker -- , seq: 748573, time: 2014-06-12 14:51:25.257226, event: all_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.257299 7f9d44101700  5 -- op tracker -- , seq: 748573, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.257305 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:25.257313 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:25.257323 7f9d44101700 15 osd.2 173 enqueue_op 0x876fc00 prio 127 cost 10717 latency 0.000138 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:25.257380 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:25.275141 7f9d60bd2700 10 getattrs 3.2e_head/5824e02e/rbd_data.10956b8b4567.0000000000002e63/head//3 = 0
2014-06-12 14:51:25.275151 7f9d60bd2700 10 stat 3.2e_head/e9e6e02e/rbd_data.10af6b8b4567.0000000000000dcd/head//3
2014-06-12 14:51:25.275271 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid e9e6e02e/rbd_data.10af6b8b4567.0000000000000dcd/head//3
2014-06-12 14:51:25.282410 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:25.291082 7f9d60bd2700 10 getattrs 3.2e_head/e9e6e02e/rbd_data.10af6b8b4567.0000000000000dcd/head//3 = 0
2014-06-12 14:51:25.291092 7f9d60bd2700 10 stat 3.2e_head/8319e02e/rbd_data.10a96b8b4567.0000000000000398/head//3
2014-06-12 14:51:25.291205 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8319e02e/rbd_data.10a96b8b4567.0000000000000398/head//3
2014-06-12 14:51:25.297878 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:25.336645 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:25.341073 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:25.342842 7f9d60bd2700 10 getattrs 3.2e_head/8319e02e/rbd_data.10a96b8b4567.0000000000000398/head//3 = 0
2014-06-12 14:51:25.342860 7f9d60bd2700 10 collection_getattrs 3.2e_head
2014-06-12 14:51:25.342962 7f9d60bd2700 10 collection_getattrs 3.2e_head = 0
2014-06-12 14:51:25.342970 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:25.343042 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:25.343064 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:25.343090 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:25.343096 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:25.343121 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2e at 2014-06-11 13:02:32.500084
2014-06-12 14:51:25.343166 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3a at 2014-06-11 13:02:32.500954
2014-06-12 14:51:25.343176 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343181 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343204 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.43 at 2014-06-11 13:02:32.501298
2014-06-12 14:51:25.343232 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343208 7f9d653db700 10 osd.2 173 dequeue_op 0x876fc00 prio 127 cost 10717 latency 0.086022 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]
2014-06-12 14:51:25.343235 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343251 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6b at 2014-06-11 13:02:32.502023
2014-06-12 14:51:25.343257 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343253 7f9d653db700  5 -- op tracker -- , seq: 748573, time: 2014-06-12 14:51:25.343245, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.343259 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343276 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6c at 2014-06-11 13:02:32.502351
2014-06-12 14:51:25.343282 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343272 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] handle_message: 0x876fc00
2014-06-12 14:51:25.343284 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343289 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:25.343297 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.a0 at 2014-06-11 13:02:32.502930
2014-06-12 14:51:25.343313 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343306 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:25.343315 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.a0( v 173'2803 (0'0,173'2803] local-les=157 n=294 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2802 mlcod 173'2802 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343330 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.105 at 2014-06-11 13:02:32.503257
2014-06-12 14:51:25.343336 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343324 7f9d653db700  5 -- op tracker -- , seq: 748573, time: 2014-06-12 14:51:25.343318, event: started, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.343338 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2817 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343340 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:25.343351 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.122 at 2014-06-11 13:02:32.503770
2014-06-12 14:51:25.343357 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343364 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343376 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.12b at 2014-06-11 13:02:32.504419
2014-06-12 14:51:25.343387 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343389 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.12b( v 173'3236 (79'235,173'3236] local-les=137 n=323 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3235 mlcod 173'3235 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343403 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.131 at 2014-06-11 13:02:32.504931
2014-06-12 14:51:25.343409 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343411 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343423 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.15d at 2014-06-11 13:02:32.505494
2014-06-12 14:51:25.343428 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343430 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343443 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.165 at 2014-06-11 13:02:32.506051
2014-06-12 14:51:25.343449 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343444 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] map version is 173'2847
2014-06-12 14:51:25.343451 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.165( v 173'3106 (79'105,173'3106] local-les=149 n=320 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3105 mlcod 173'3105 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343464 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.170 at 2014-06-11 13:02:32.506577
2014-06-12 14:51:25.343467 7f9d653db700 10 osd.2 173 dequeue_op 0x876fc00 finish
2014-06-12 14:51:25.343469 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343472 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3049 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343478 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:25.343485 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1b5 at 2014-06-11 13:02:32.507250
2014-06-12 14:51:25.343480 7f9d653db700  5 -- op tracker -- , seq: 748573, time: 2014-06-12 14:51:25.343474, event: done, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.343490 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343493 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1b5( v 173'2905 (0'0,173'2905] local-les=153 n=320 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2904 mlcod 173'2904 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343513 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:25.343514 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1bc at 2014-06-11 13:02:32.507758
2014-06-12 14:51:25.343518 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:25.343519 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343492 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:25.343521 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2459 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343532 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1e1 at 2014-06-11 13:02:32.508326
2014-06-12 14:51:25.343540 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343526 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:25.343542 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2540 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343554 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1e8 at 2014-06-11 13:02:32.508866
2014-06-12 14:51:25.343559 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343545 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:25.343561 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3033 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343571 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1eb at 2014-06-11 13:02:32.509421
2014-06-12 14:51:25.343591 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343594 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3095 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343608 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1fd at 2014-06-11 13:02:32.510047
2014-06-12 14:51:25.343615 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343599 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:25.343617 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1fd( v 173'3047 (79'46,173'3047] local-les=157 n=305 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343628 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.204 at 2014-06-11 13:02:32.510573
2014-06-12 14:51:25.343633 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343620 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:25.343635 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.204( v 173'2982 (0'0,173'2982] local-les=161 n=308 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343646 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.209 at 2014-06-11 13:02:32.511095
2014-06-12 14:51:25.343651 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343639 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:25.343652 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.209( v 173'2531 (0'0,173'2531] local-les=156 n=264 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2530 mlcod 173'2530 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343663 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.214 at 2014-06-11 13:02:32.511670
2014-06-12 14:51:25.343669 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343671 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2934 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343682 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.21d at 2014-06-11 13:02:32.512202
2014-06-12 14:51:25.343686 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343688 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.21d( v 173'3024 (79'23,173'3024] local-les=156 n=318 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3023 mlcod 173'3023 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343699 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.23f at 2014-06-11 13:02:32.512776
2014-06-12 14:51:25.343704 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343705 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343716 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.251 at 2014-06-11 13:02:32.513340
2014-06-12 14:51:25.343694 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 34a2c02e/rbd_data.10a46b8b4567.00000000000012a4/head//3, auth == maps.end()
2014-06-12 14:51:25.343720 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343722 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343733 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.262 at 2014-06-11 13:02:32.513946
2014-06-12 14:51:25.343742 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343731 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 34a2c02e/rbd_data.10a46b8b4567.00000000000012a4/head//3
2014-06-12 14:51:25.343744 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.262( v 173'2675 (0'0,173'2675] local-les=142 n=292 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2674 mlcod 173'2674 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343756 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.266 at 2014-06-11 13:02:32.514523
2014-06-12 14:51:25.343760 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343762 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2824 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343764 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj abe4c02e/rbd_data.10af6b8b4567.0000000000002043/head//3, auth == maps.end()
2014-06-12 14:51:25.343773 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.288 at 2014-06-11 13:02:32.515022
2014-06-12 14:51:25.343792 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343795 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.288( v 173'2548 (0'0,173'2548] local-les=161 n=263 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2547 mlcod 173'2547 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343807 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2b4 at 2014-06-11 13:02:32.515543
2014-06-12 14:51:25.343812 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343792 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj abe4c02e/rbd_data.10af6b8b4567.0000000000002043/head//3
2014-06-12 14:51:25.343814 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2b4( v 173'3109 (79'108,173'3109] local-les=137 n=312 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3108 mlcod 173'3108 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343828 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2c4 at 2014-06-11 13:02:32.516106
2014-06-12 14:51:25.343833 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343835 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2c4( v 173'2897 (0'0,173'2897] local-les=139 n=307 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2896 mlcod 173'2896 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343825 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9217c02e/rbd_data.10a16b8b4567.00000000000029f3/head//3, auth == maps.end()
2014-06-12 14:51:25.343845 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2d3 at 2014-06-11 13:02:32.516627
2014-06-12 14:51:25.343850 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343852 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2d3( v 173'2556 (0'0,173'2556] local-les=153 n=276 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2555 mlcod 173'2555 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343862 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2da at 2014-06-11 13:02:32.517119
2014-06-12 14:51:25.343867 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343852 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9217c02e/rbd_data.10a16b8b4567.00000000000029f3/head//3
2014-06-12 14:51:25.343869 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2da( v 173'2988 (0'0,173'2988] local-les=145 n=302 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2987 mlcod 173'2987 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343880 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2f4 at 2014-06-11 13:02:32.517764
2014-06-12 14:51:25.343884 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343886 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3274 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343879 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dd18c02e/rbd_data.10ab6b8b4567.000000000000133e/head//3, auth == maps.end()
2014-06-12 14:51:25.343897 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.312 at 2014-06-11 13:02:32.518300
2014-06-12 14:51:25.343902 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343903 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2782 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343913 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.320 at 2014-06-11 13:02:32.518824
2014-06-12 14:51:25.343922 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343906 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj dd18c02e/rbd_data.10ab6b8b4567.000000000000133e/head//3
2014-06-12 14:51:25.343924 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.320( v 173'2687 (0'0,173'2687] local-les=145 n=282 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2686 mlcod 173'2686 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343936 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.330 at 2014-06-11 13:02:32.519349
2014-06-12 14:51:25.343941 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343934 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 785ac02e/rbd_data.10a16b8b4567.000000000000127e/head//3, auth == maps.end()
2014-06-12 14:51:25.343942 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2798 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343956 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.342 at 2014-06-11 13:02:32.519873
2014-06-12 14:51:25.343963 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343965 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2464 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343976 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.35b at 2014-06-11 13:02:32.520528
2014-06-12 14:51:25.343961 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 785ac02e/rbd_data.10a16b8b4567.000000000000127e/head//3
2014-06-12 14:51:25.343980 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343982 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.35b( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.343994 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.366 at 2014-06-11 13:02:32.521097
2014-06-12 14:51:25.343999 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.343989 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2a7ac02e/rbd_data.10b06b8b4567.00000000000024f3/head//3, auth == maps.end()
2014-06-12 14:51:25.344001 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2661 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344012 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.37a at 2014-06-11 13:02:32.521744
2014-06-12 14:51:25.344017 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344019 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.37a( v 173'2555 (0'0,173'2555] local-les=153 n=292 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2554 mlcod 173'2554 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344014 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2a7ac02e/rbd_data.10b06b8b4567.00000000000024f3/head//3
2014-06-12 14:51:25.344030 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.394 at 2014-06-11 13:02:32.522362
2014-06-12 14:51:25.344049 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344041 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cfcac02e/rbd_data.10946b8b4567.0000000000002290/head//3, auth == maps.end()
2014-06-12 14:51:25.344052 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.394( v 173'2648 (0'0,173'2648] local-les=145 n=266 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344064 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.395 at 2014-06-11 13:02:32.522991
2014-06-12 14:51:25.344073 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344075 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3327 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344067 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj cfcac02e/rbd_data.10946b8b4567.0000000000002290/head//3
2014-06-12 14:51:25.344087 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3ac at 2014-06-11 13:02:32.523569
2014-06-12 14:51:25.344092 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344093 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3ac( v 173'2860 (0'0,173'2860] local-les=142 n=294 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344104 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3d4 at 2014-06-11 13:02:32.524119
2014-06-12 14:51:25.344109 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344095 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f54bc02e/rbd_data.109a6b8b4567.00000000000010a3/head//3, auth == maps.end()
2014-06-12 14:51:25.344111 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3d4( v 173'2696 (0'0,173'2696] local-les=137 n=271 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344121 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3d8 at 2014-06-11 13:02:32.524641
2014-06-12 14:51:25.344126 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344128 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3d8( v 173'2879 (0'0,173'2879] local-les=156 n=294 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344122 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f54bc02e/rbd_data.109a6b8b4567.00000000000010a3/head//3
2014-06-12 14:51:25.344139 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.401 at 2014-06-11 13:02:32.525188
2014-06-12 14:51:25.344144 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344146 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3154 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344150 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 72ec02e/rbd_data.10ad6b8b4567.00000000000006de/head//3, auth == maps.end()
2014-06-12 14:51:25.344157 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.408 at 2014-06-11 13:02:32.525838
2014-06-12 14:51:25.344176 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344179 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2883 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344176 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 72ec02e/rbd_data.10ad6b8b4567.00000000000006de/head//3
2014-06-12 14:51:25.344191 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.411 at 2014-06-11 13:02:32.526419
2014-06-12 14:51:25.344196 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344198 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.411( v 173'2402 (0'0,173'2402] local-les=147 n=278 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2401 mlcod 173'2401 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344204 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 46fc02e/rbd_data.10976b8b4567.0000000000000632/head//3, auth == maps.end()
2014-06-12 14:51:25.344209 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.45f at 2014-06-11 13:02:32.526940
2014-06-12 14:51:25.344227 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344230 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2357 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344230 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 46fc02e/rbd_data.10976b8b4567.0000000000000632/head//3
2014-06-12 14:51:25.344242 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.479 at 2014-06-11 13:02:32.527498
2014-06-12 14:51:25.344260 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344262 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.479( v 173'2926 (0'0,173'2926] local-les=142 n=298 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344257 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj afbfc02e/rbd_data.109c6b8b4567.00000000000031eb/head//3, auth == maps.end()
2014-06-12 14:51:25.344278 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4a7 at 2014-06-11 13:02:32.528023
2014-06-12 14:51:25.344295 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344283 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj afbfc02e/rbd_data.109c6b8b4567.00000000000031eb/head//3
2014-06-12 14:51:25.344298 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2594 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344310 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4b4 at 2014-06-11 13:02:32.528548
2014-06-12 14:51:25.344327 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344311 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8cb0d02e/rbd_data.10a46b8b4567.00000000000037a4/head//3, auth == maps.end()
2014-06-12 14:51:25.344330 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4b4( v 173'2792 (0'0,173'2792] local-les=137 n=305 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344343 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4d4 at 2014-06-11 13:02:32.529466
2014-06-12 14:51:25.344348 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344336 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8cb0d02e/rbd_data.10a46b8b4567.00000000000037a4/head//3
2014-06-12 14:51:25.344350 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2745 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344360 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4ef at 2014-06-11 13:02:32.530034
2014-06-12 14:51:25.344375 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344364 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 86c1d02e/rbd_data.10976b8b4567.0000000000003abc/head//3, auth == maps.end()
2014-06-12 14:51:25.344377 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4ef( v 173'3004 (79'3,173'3004] local-les=149 n=307 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344389 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4f0 at 2014-06-11 13:02:32.530572
2014-06-12 14:51:25.344394 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344396 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2864 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344390 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 86c1d02e/rbd_data.10976b8b4567.0000000000003abc/head//3
2014-06-12 14:51:25.344407 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.54b at 2014-06-11 13:02:32.531354
2014-06-12 14:51:25.344412 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344414 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2722 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344425 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.568 at 2014-06-11 13:02:32.531914
2014-06-12 14:51:25.344418 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 18f3d02e/rbd_data.10ab6b8b4567.0000000000000c93/head//3, auth == maps.end()
2014-06-12 14:51:25.344439 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344442 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.568( v 173'2727 (0'0,173'2727] local-les=161 n=270 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344453 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.58f at 2014-06-11 13:02:32.532612
2014-06-12 14:51:25.344458 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344446 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 18f3d02e/rbd_data.10ab6b8b4567.0000000000000c93/head//3
2014-06-12 14:51:25.344460 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.58f( v 173'2707 (0'0,173'2707] local-les=153 n=304 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344470 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5a3 at 2014-06-11 13:02:32.533139
2014-06-12 14:51:25.344475 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344477 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5a3( v 173'2874 (0'0,173'2874] local-les=142 n=296 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2873 mlcod 173'2873 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344473 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e55d02e/rbd_data.10ac6b8b4567.0000000000002dd9/head//3, auth == maps.end()
2014-06-12 14:51:25.344488 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5ce at 2014-06-11 13:02:32.533636
2014-06-12 14:51:25.344503 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344506 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5ce( v 173'2741 (0'0,173'2741] local-les=153 n=294 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344500 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e55d02e/rbd_data.10ac6b8b4567.0000000000002dd9/head//3
2014-06-12 14:51:25.344517 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5dd at 2014-06-11 13:02:32.534211
2014-06-12 14:51:25.344523 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344525 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5dd( v 173'2852 (0'0,173'2852] local-les=156 n=307 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344539 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5fc at 2014-06-11 13:02:32.534738
2014-06-12 14:51:25.344544 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344528 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b7f6d02e/rbd_data.109b6b8b4567.000000000000183f/head//3, auth == maps.end()
2014-06-12 14:51:25.344546 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5fc( v 173'2714 (0'0,173'2714] local-les=157 n=284 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344556 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.602 at 2014-06-11 13:02:32.535274
2014-06-12 14:51:25.344561 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344563 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3009 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344555 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b7f6d02e/rbd_data.109b6b8b4567.000000000000183f/head//3
2014-06-12 14:51:25.344574 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.605 at 2014-06-11 13:02:32.535829
2014-06-12 14:51:25.344579 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344580 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344591 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.613 at 2014-06-11 13:02:32.536448
2014-06-12 14:51:25.344600 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344584 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3af7d02e/rbd_data.10946b8b4567.00000000000015f6/head//3, auth == maps.end()
2014-06-12 14:51:25.344602 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344613 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.635 at 2014-06-11 13:02:32.537041
2014-06-12 14:51:25.344618 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344619 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.635( v 173'2966 (0'0,173'2966] local-les=137 n=308 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2965 mlcod 173'2965 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344615 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3af7d02e/rbd_data.10946b8b4567.00000000000015f6/head//3
2014-06-12 14:51:25.344630 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.63b at 2014-06-11 13:02:32.537647
2014-06-12 14:51:25.344636 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344638 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344650 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.63e at 2014-06-11 13:02:32.538178
2014-06-12 14:51:25.344644 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 647ad02e/rbd_data.10ad6b8b4567.000000000000363a/head//3, auth == maps.end()
2014-06-12 14:51:25.344664 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344667 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344681 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.650 at 2014-06-11 13:02:32.538854
2014-06-12 14:51:25.344690 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344671 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 647ad02e/rbd_data.10ad6b8b4567.000000000000363a/head//3
2014-06-12 14:51:25.344692 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344703 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.662 at 2014-06-11 13:02:32.539391
2014-06-12 14:51:25.344708 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344710 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.662( v 173'2604 (0'0,173'2604] local-les=161 n=257 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2603 mlcod 173'2603 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344702 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d6fdd02e/rbd_data.10986b8b4567.0000000000001c7f/head//3, auth == maps.end()
2014-06-12 14:51:25.344720 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.667 at 2014-06-11 13:02:32.539999
2014-06-12 14:51:25.344728 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344730 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.667( v 173'2958 (0'0,173'2958] local-les=149 n=303 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344742 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6a8 at 2014-06-11 13:02:32.540566
2014-06-12 14:51:25.344729 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d6fdd02e/rbd_data.10986b8b4567.0000000000001c7f/head//3
2014-06-12 14:51:25.344748 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344753 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6a8( v 173'2854 (0'0,173'2854] local-les=137 n=299 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344764 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6ff at 2014-06-11 13:02:32.541074
2014-06-12 14:51:25.344769 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344757 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e570e02e/rbd_data.10966b8b4567.0000000000001aab/head//3, auth == maps.end()
2014-06-12 14:51:25.344770 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6ff( v 173'2792 (0'0,173'2792] local-les=161 n=298 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344781 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.711 at 2014-06-11 13:02:32.541770
2014-06-12 14:51:25.344786 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344788 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.711( v 173'2514 (0'0,173'2514] local-les=147 n=297 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344798 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.712 at 2014-06-11 13:02:32.542314
2014-06-12 14:51:25.344784 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e570e02e/rbd_data.10966b8b4567.0000000000001aab/head//3
2014-06-12 14:51:25.344803 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344805 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.712( v 173'2782 (0'0,173'2782] local-les=153 n=305 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344815 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.717 at 2014-06-11 13:02:32.542947
2014-06-12 14:51:25.344820 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344822 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2583 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344812 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ed0e02e/rbd_data.10a96b8b4567.000000000000385a/head//3, auth == maps.end()
2014-06-12 14:51:25.344832 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.721 at 2014-06-11 13:02:32.543618
2014-06-12 14:51:25.344837 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344838 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2895 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344839 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ed0e02e/rbd_data.10a96b8b4567.000000000000385a/head//3
2014-06-12 14:51:25.344849 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.735 at 2014-06-11 13:02:32.544147
2014-06-12 14:51:25.344865 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344867 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.735( v 173'2803 (0'0,173'2803] local-les=137 n=306 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2802 mlcod 173'2802 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344880 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.762 at 2014-06-11 13:02:32.544735
2014-06-12 14:51:25.344866 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3122e02e/rbd_data.10a36b8b4567.0000000000001e3d/head//3, auth == maps.end()
2014-06-12 14:51:25.344885 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344887 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.762( v 173'2726 (0'0,173'2726] local-les=147 n=302 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344897 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.76e at 2014-06-11 13:02:32.545259
2014-06-12 14:51:25.344902 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344892 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3122e02e/rbd_data.10a36b8b4567.0000000000001e3d/head//3
2014-06-12 14:51:25.344903 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2648 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344914 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.77e at 2014-06-11 13:02:32.545813
2014-06-12 14:51:25.344919 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344921 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.77e( v 173'3052 (79'51,173'3052] local-les=161 n=319 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3051 mlcod 173'3051 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344931 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.783 at 2014-06-11 13:02:32.546436
2014-06-12 14:51:25.344936 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344920 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5824e02e/rbd_data.10956b8b4567.0000000000002e63/head//3, auth == maps.end()
2014-06-12 14:51:25.344938 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344951 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.785 at 2014-06-11 13:02:32.547026
2014-06-12 14:51:25.344956 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344947 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5824e02e/rbd_data.10956b8b4567.0000000000002e63/head//3
2014-06-12 14:51:25.344957 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344968 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.793 at 2014-06-11 13:02:32.547568
2014-06-12 14:51:25.344974 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344976 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.793( v 173'2605 (0'0,173'2605] local-les=153 n=277 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2604 mlcod 173'2604 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.344987 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ab at 2014-06-11 13:02:32.548162
2014-06-12 14:51:25.344974 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e9e6e02e/rbd_data.10af6b8b4567.0000000000000dcd/head//3, auth == maps.end()
2014-06-12 14:51:25.344995 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.344997 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ab( v 173'2921 (0'0,173'2921] local-les=161 n=302 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2920 mlcod 173'2920 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.345008 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ad at 2014-06-11 13:02:32.548789
2014-06-12 14:51:25.345013 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.345001 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e9e6e02e/rbd_data.10af6b8b4567.0000000000000dcd/head//3
2014-06-12 14:51:25.345014 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ad( v 173'3043 (79'42,173'3043] local-les=149 n=315 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.345025 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ae at 2014-06-11 13:02:32.549514
2014-06-12 14:51:25.345030 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.345032 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2517 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.345030 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8319e02e/rbd_data.10a96b8b4567.0000000000000398/head//3, auth == maps.end()
2014-06-12 14:51:25.345043 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7bc at 2014-06-11 13:02:32.550208
2014-06-12 14:51:25.345058 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.345061 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7bc( v 173'2562 (0'0,173'2562] local-les=153 n=287 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2561 mlcod 173'2561 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.345056 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8319e02e/rbd_data.10a96b8b4567.0000000000000398/head//3
2014-06-12 14:51:25.345072 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7cc at 2014-06-11 13:02:32.550758
2014-06-12 14:51:25.345077 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.345079 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.345090 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7db at 2014-06-11 13:02:32.551269
2014-06-12 14:51:25.345094 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.345087 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] 
2014-06-12 14:51:25.345096 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2681 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.345106 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7f8 at 2014-06-11 13:02:32.551909
2014-06-12 14:51:25.345115 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:25.345106 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub
2014-06-12 14:51:25.345117 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:25.345128 7f9d84175700 10 osd.2 173 sched_scrub 0.3 at 2014-06-12 13:01:39.405748: 6585.47 < min (86400 seconds)
2014-06-12 14:51:25.345142 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:25.345145 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:25.345147 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:25.345141 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8319e02e/rbd_data.10a96b8b4567.0000000000000398/head//3 8319e02e/rbd_data.10a96b8b4567.0000000000000398/head//3(173'2496 client.4306.0:123479 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2496)
2014-06-12 14:51:25.345178 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  e9e6e02e/rbd_data.10af6b8b4567.0000000000000dcd/head//3 e9e6e02e/rbd_data.10af6b8b4567.0000000000000dcd/head//3(173'2826 client.4312.0:143089 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2826)
2014-06-12 14:51:25.345212 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  5824e02e/rbd_data.10956b8b4567.0000000000002e63/head//3 5824e02e/rbd_data.10956b8b4567.0000000000002e63/head//3(173'1840 client.4286.0:93317 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1840)
2014-06-12 14:51:25.345245 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  3122e02e/rbd_data.10a36b8b4567.0000000000001e3d/head//3 3122e02e/rbd_data.10a36b8b4567.0000000000001e3d/head//3(173'1261 client.4300.0:62191 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1261)
2014-06-12 14:51:25.345279 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  ed0e02e/rbd_data.10a96b8b4567.000000000000385a/head//3 ed0e02e/rbd_data.10a96b8b4567.000000000000385a/head//3(173'2329 client.4306.0:114161 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2329)
2014-06-12 14:51:25.345311 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  e570e02e/rbd_data.10966b8b4567.0000000000001aab/head//3 e570e02e/rbd_data.10966b8b4567.0000000000001aab/head//3(173'1135 client.4287.0:55970 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1135)
2014-06-12 14:51:25.345344 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d6fdd02e/rbd_data.10986b8b4567.0000000000001c7f/head//3 d6fdd02e/rbd_data.10986b8b4567.0000000000001c7f/head//3(173'1223 client.4289.0:59821 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1223)
2014-06-12 14:51:25.345377 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  647ad02e/rbd_data.10ad6b8b4567.000000000000363a/head//3 647ad02e/rbd_data.10ad6b8b4567.000000000000363a/head//3(173'2057 client.4310.0:107247 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2057)
2014-06-12 14:51:25.345409 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  3af7d02e/rbd_data.10946b8b4567.00000000000015f6/head//3 3af7d02e/rbd_data.10946b8b4567.00000000000015f6/head//3(173'981 client.4285.0:48340 wrlock_by=unknown.0.0:0 dirty s 4194304 uv981)
2014-06-12 14:51:25.345442 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  b7f6d02e/rbd_data.109b6b8b4567.000000000000183f/head//3 b7f6d02e/rbd_data.109b6b8b4567.000000000000183f/head//3(173'1161 client.4292.0:51784 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1161)
2014-06-12 14:51:25.345475 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  e55d02e/rbd_data.10ac6b8b4567.0000000000002dd9/head//3 e55d02e/rbd_data.10ac6b8b4567.0000000000002dd9/head//3(173'1866 client.4309.0:91350 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1866)
2014-06-12 14:51:25.345510 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  18f3d02e/rbd_data.10ab6b8b4567.0000000000000c93/head//3 18f3d02e/rbd_data.10ab6b8b4567.0000000000000c93/head//3(173'2753 client.4308.0:140104 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2753)
2014-06-12 14:51:25.345543 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  86c1d02e/rbd_data.10976b8b4567.0000000000003abc/head//3 86c1d02e/rbd_data.10976b8b4567.0000000000003abc/head//3(173'2315 client.4288.0:113195 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2315)
2014-06-12 14:51:25.345576 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8cb0d02e/rbd_data.10a46b8b4567.00000000000037a4/head//3 8cb0d02e/rbd_data.10a46b8b4567.00000000000037a4/head//3(173'2160 client.4301.0:111170 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2160)
2014-06-12 14:51:25.345609 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  afbfc02e/rbd_data.109c6b8b4567.00000000000031eb/head//3 afbfc02e/rbd_data.109c6b8b4567.00000000000031eb/head//3(173'1980 client.4293.0:100228 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1980)
2014-06-12 14:51:25.345657 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  46fc02e/rbd_data.10976b8b4567.0000000000000632/head//3 46fc02e/rbd_data.10976b8b4567.0000000000000632/head//3(173'2572 client.4288.0:124544 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2572)
2014-06-12 14:51:25.345687 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  72ec02e/rbd_data.10ad6b8b4567.00000000000006de/head//3 72ec02e/rbd_data.10ad6b8b4567.00000000000006de/head//3(173'2588 client.4310.0:128200 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2588)
2014-06-12 14:51:25.345721 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f54bc02e/rbd_data.109a6b8b4567.00000000000010a3/head//3 f54bc02e/rbd_data.109a6b8b4567.00000000000010a3/head//3(173'827 client.4291.0:39172 wrlock_by=unknown.0.0:0 dirty s 4194304 uv827)
2014-06-12 14:51:25.345753 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  cfcac02e/rbd_data.10946b8b4567.0000000000002290/head//3 cfcac02e/rbd_data.10946b8b4567.0000000000002290/head//3(173'1367 client.4285.0:70354 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1367)
2014-06-12 14:51:25.345786 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  2a7ac02e/rbd_data.10b06b8b4567.00000000000024f3/head//3 2a7ac02e/rbd_data.10b06b8b4567.00000000000024f3/head//3(173'1469 client.4313.0:76831 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1469)
2014-06-12 14:51:25.345819 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  785ac02e/rbd_data.10a16b8b4567.000000000000127e/head//3 785ac02e/rbd_data.10a16b8b4567.000000000000127e/head//3(173'895 client.4298.0:43821 wrlock_by=unknown.0.0:0 dirty s 4194304 uv895)
2014-06-12 14:51:25.345851 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  dd18c02e/rbd_data.10ab6b8b4567.000000000000133e/head//3 dd18c02e/rbd_data.10ab6b8b4567.000000000000133e/head//3(173'905 client.4308.0:44749 wrlock_by=unknown.0.0:0 dirty s 4194304 uv905)
2014-06-12 14:51:25.345884 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  9217c02e/rbd_data.10a16b8b4567.00000000000029f3/head//3 9217c02e/rbd_data.10a16b8b4567.00000000000029f3/head//3(173'1651 client.4298.0:86067 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1651)
2014-06-12 14:51:25.345916 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  abe4c02e/rbd_data.10af6b8b4567.0000000000002043/head//3 abe4c02e/rbd_data.10af6b8b4567.0000000000002043/head//3(173'1378 client.4312.0:69776 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1378)
2014-06-12 14:51:25.345949 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  34a2c02e/rbd_data.10a46b8b4567.00000000000012a4/head//3 34a2c02e/rbd_data.10a46b8b4567.00000000000012a4/head//3(173'861 client.4301.0:43660 wrlock_by=unknown.0.0:0 dirty s 4194304 uv861)
2014-06-12 14:51:25.345975 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:25.346015 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:25.346038 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:25.346043 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:25.346072 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:25.346135 7f9d60bd2700 10 collection_list_partial 3.2e_head start:23cbe02e//0//-1 is_max:0
2014-06-12 14:51:25.346667 7f9d60bd2700 20 objects: [fe6ce02e/rbd_data.109b6b8b4567.0000000000001594/head//3,e162f02e/rbd_data.108f6b8b4567.0000000000001694/head//3,2646f02e/rbd_data.109e6b8b4567.00000000000024fb/head//3,6fe7f02e/rbd_data.10a06b8b4567.000000000000000a/head//3,f9cbf02e/rbd_data.10986b8b4567.000000000000190b/head//3,d2ef02e/rbd_data.10ad6b8b4567.0000000000002d19/head//3,5a0ff02e/rbd_data.10ad6b8b4567.00000000000000b3/head//3,a86ff02e/rbd_data.109c6b8b4567.0000000000000325/head//3,8b71082e/rbd_data.10996b8b4567.0000000000000ded/head//3,cd91082e/rbd_data.10b16b8b4567.0000000000001e29/head//3,ed23082e/rbd_data.10a86b8b4567.00000000000015f8/head//3,5d85082e/rbd_data.10a66b8b4567.00000000000030b5/head//3,e00c082e/rbd_data.10a16b8b4567.00000000000035b4/head//3,244c082e/rbd_data.10b26b8b4567.00000000000034e7/head//3,7c8c082e/rbd_data.10996b8b4567.0000000000003ac3/head//3,4b9c082e/rbd_data.108f6b8b4567.0000000000001232/head//3,27ad082e/rbd_data.10b06b8b4567.00000000000022b9/head//3,9ffe082e/rbd_data.109e6b8b4567.00000000000005a0/head//3,fd9f082e/rbd_data.10ac6b8b4567.0000000000003102/head//3,2f0182e/rbd_data.10ac6b8b4567.0000000000000dfe/head//3,a762182e/rbd_data.10ab6b8b4567.0000000000002f00/head//3,b153182e/rbd_data.10946b8b4567.0000000000000fa5/head//3,bed3182e/rbd_data.109c6b8b4567.00000000000017a4/head//3,53b4182e/rbd_data.108e6b8b4567.0000000000002760/head//3,2028182e/rbd_data.10aa6b8b4567.0000000000003580/head//3]
2014-06-12 14:51:25.347421 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:25.347473 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:25.347509 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:25.347530 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:25.347547 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:25.347564 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:25.347581 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_map_chunk [23cbe02e//0//-1,919182e//0//-1)
2014-06-12 14:51:25.347600 7f9d60bd2700 10 collection_list_partial 3.2e_head start:23cbe02e//0//-1 is_max:0
2014-06-12 14:51:25.348882 7f9d60bd2700 20 objects: [fe6ce02e/rbd_data.109b6b8b4567.0000000000001594/head//3,e162f02e/rbd_data.108f6b8b4567.0000000000001694/head//3,2646f02e/rbd_data.109e6b8b4567.00000000000024fb/head//3,6fe7f02e/rbd_data.10a06b8b4567.000000000000000a/head//3,f9cbf02e/rbd_data.10986b8b4567.000000000000190b/head//3,d2ef02e/rbd_data.10ad6b8b4567.0000000000002d19/head//3,5a0ff02e/rbd_data.10ad6b8b4567.00000000000000b3/head//3,a86ff02e/rbd_data.109c6b8b4567.0000000000000325/head//3,8b71082e/rbd_data.10996b8b4567.0000000000000ded/head//3,cd91082e/rbd_data.10b16b8b4567.0000000000001e29/head//3,ed23082e/rbd_data.10a86b8b4567.00000000000015f8/head//3,5d85082e/rbd_data.10a66b8b4567.00000000000030b5/head//3,e00c082e/rbd_data.10a16b8b4567.00000000000035b4/head//3,244c082e/rbd_data.10b26b8b4567.00000000000034e7/head//3,7c8c082e/rbd_data.10996b8b4567.0000000000003ac3/head//3,4b9c082e/rbd_data.108f6b8b4567.0000000000001232/head//3,27ad082e/rbd_data.10b06b8b4567.00000000000022b9/head//3,9ffe082e/rbd_data.109e6b8b4567.00000000000005a0/head//3,fd9f082e/rbd_data.10ac6b8b4567.0000000000003102/head//3,2f0182e/rbd_data.10ac6b8b4567.0000000000000dfe/head//3,a762182e/rbd_data.10ab6b8b4567.0000000000002f00/head//3,b153182e/rbd_data.10946b8b4567.0000000000000fa5/head//3,bed3182e/rbd_data.109c6b8b4567.00000000000017a4/head//3,53b4182e/rbd_data.108e6b8b4567.0000000000002760/head//3,2028182e/rbd_data.10aa6b8b4567.0000000000003580/head//3,919182e/rbd_data.108d6b8b4567.0000000000003342/head//3,7a3e182e/rbd_data.109c6b8b4567.0000000000002ee6/head//3,a58e182e/rbd_data.108e6b8b4567.0000000000002612/head//3,177f182e/rbd_data.10ac6b8b4567.00000000000025ad/head//3,fd02282e/rbd_data.10a76b8b4567.0000000000003a8c/head//3,2f24282e/rbd_data.10b36b8b4567.00000000000016fc/head//3,7e85282e/rbd_data.10926b8b4567.00000000000015b2/head//3,5618282e/rbd_data.109b6b8b4567.0000000000001c3b/head//3,9378282e/rbd_data.10b06b8b4567.0000000000003a8a/head//3,2fbb282e/rbd_data.10a36b8b4567.00000000000001d8/head//3,777f282e/rbd_data.10956b8b4567.00000000000014f5/head//3,cc9f282e/rbd_data.109d6b8b4567.00000000000024bc/head//3,b383382e/rbd_data.10ae6b8b4567.000000000000052d/head//3,e7f3382e/rbd_data.109f6b8b4567.0000000000003878/head//3,b525382e/rbd_data.10976b8b4567.0000000000000043/head//3,5558382e/rbd_data.10996b8b4567.0000000000000e74/head//3,4fca382e/rbd_data.10b46b8b4567.0000000000001f40/head//3,29da382e/rbd_data.10a16b8b4567.0000000000003834/head//3,ccbb382e/rbd_data.10b26b8b4567.0000000000003754/head//3,62bd382e/rbd_data.10926b8b4567.00000000000022cb/head//3,bdf382e/rbd_data.10956b8b4567.00000000000005f3/head//3,2ec0482e/rbd_data.10a26b8b4567.0000000000002398/head//3,7581482e/rbd_data.109f6b8b4567.0000000000002450/head//3,aab1482e/rbd_data.10b16b8b4567.0000000000000e5c/head//3,66f1482e/rbd_data.10a36b8b4567.0000000000000ce3/head//3,f4a3482e/rbd_data.109d6b8b4567.0000000000001d14/head//3,aef7482e/rbd_data.109a6b8b4567.00000000000039d6/head//3,8a19482e/rbd_data.10ab6b8b4567.000000000000025e/head//3,c7f9482e/rbd_data.108e6b8b4567.0000000000003aa3/head//3,19ea482e/rbd_data.10aa6b8b4567.00000000000006c0/head//3,9ad482e/rbd_data.10956b8b4567.000000000000205f/head//3,e03f482e/rbd_data.10af6b8b4567.00000000000013dd/head//3,6870582e/rbd_data.10ae6b8b4567.0000000000001dcd/head//3,1ac2582e/rbd_data.109c6b8b4567.00000000000037f4/head//3,44f3582e/rbd_data.109c6b8b4567.000000000000156d/head//3,b474582e/rbd_data.10ae6b8b4567.00000000000001c7/head//3,54a4582e/rbd_data.108e6b8b4567.00000000000003db/head//3,8e08582e/rbd_data.10976b8b4567.00000000000024bd/head//3,8278582e/rbd_data.10ae6b8b4567.0000000000000cf8/head//3]
2014-06-12 14:51:25.349058 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:25.349095 7f9d60bd2700 10 stat 3.2e_head/fe6ce02e/rbd_data.109b6b8b4567.0000000000001594/head//3
2014-06-12 14:51:25.349376 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid fe6ce02e/rbd_data.109b6b8b4567.0000000000001594/head//3
2014-06-12 14:51:25.364968 7f9d60bd2700 10 getattrs 3.2e_head/fe6ce02e/rbd_data.109b6b8b4567.0000000000001594/head//3 = 0
2014-06-12 14:51:25.364976 7f9d60bd2700 10 stat 3.2e_head/e162f02e/rbd_data.108f6b8b4567.0000000000001694/head//3
2014-06-12 14:51:25.365068 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid e162f02e/rbd_data.108f6b8b4567.0000000000001694/head//3
2014-06-12 14:51:25.382092 7f9d60bd2700 10 getattrs 3.2e_head/e162f02e/rbd_data.108f6b8b4567.0000000000001694/head//3 = 0
2014-06-12 14:51:25.382112 7f9d60bd2700 10 stat 3.2e_head/2646f02e/rbd_data.109e6b8b4567.00000000000024fb/head//3
2014-06-12 14:51:25.382303 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 2646f02e/rbd_data.109e6b8b4567.00000000000024fb/head//3
2014-06-12 14:51:25.389645 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:25.408776 7f9d60bd2700 10 getattrs 3.2e_head/2646f02e/rbd_data.109e6b8b4567.00000000000024fb/head//3 = 0
2014-06-12 14:51:25.408786 7f9d60bd2700 10 stat 3.2e_head/6fe7f02e/rbd_data.10a06b8b4567.000000000000000a/head//3
2014-06-12 14:51:25.408902 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6fe7f02e/rbd_data.10a06b8b4567.000000000000000a/head//3
2014-06-12 14:51:25.464933 7f9d60bd2700 10 getattrs 3.2e_head/6fe7f02e/rbd_data.10a06b8b4567.000000000000000a/head//3 = 0
2014-06-12 14:51:25.464945 7f9d60bd2700 10 stat 3.2e_head/f9cbf02e/rbd_data.10986b8b4567.000000000000190b/head//3
2014-06-12 14:51:25.465093 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f9cbf02e/rbd_data.10986b8b4567.000000000000190b/head//3
2014-06-12 14:51:25.482219 7f9d60bd2700 10 getattrs 3.2e_head/f9cbf02e/rbd_data.10986b8b4567.000000000000190b/head//3 = 0
2014-06-12 14:51:25.482237 7f9d60bd2700 10 stat 3.2e_head/d2ef02e/rbd_data.10ad6b8b4567.0000000000002d19/head//3
2014-06-12 14:51:25.482440 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d2ef02e/rbd_data.10ad6b8b4567.0000000000002d19/head//3
2014-06-12 14:51:25.500467 7f9d60bd2700 10 getattrs 3.2e_head/d2ef02e/rbd_data.10ad6b8b4567.0000000000002d19/head//3 = 0
2014-06-12 14:51:25.500490 7f9d60bd2700 10 stat 3.2e_head/5a0ff02e/rbd_data.10ad6b8b4567.00000000000000b3/head//3
2014-06-12 14:51:25.500707 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 5a0ff02e/rbd_data.10ad6b8b4567.00000000000000b3/head//3
2014-06-12 14:51:25.512791 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:25.540322 7f9d60bd2700 10 getattrs 3.2e_head/5a0ff02e/rbd_data.10ad6b8b4567.00000000000000b3/head//3 = 0
2014-06-12 14:51:25.540333 7f9d60bd2700 10 stat 3.2e_head/a86ff02e/rbd_data.109c6b8b4567.0000000000000325/head//3
2014-06-12 14:51:25.540443 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a86ff02e/rbd_data.109c6b8b4567.0000000000000325/head//3
2014-06-12 14:51:25.546716 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:25.575596 7f9d60bd2700 10 getattrs 3.2e_head/a86ff02e/rbd_data.109c6b8b4567.0000000000000325/head//3 = 0
2014-06-12 14:51:25.575605 7f9d60bd2700 10 stat 3.2e_head/8b71082e/rbd_data.10996b8b4567.0000000000000ded/head//3
2014-06-12 14:51:25.575693 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8b71082e/rbd_data.10996b8b4567.0000000000000ded/head//3
2014-06-12 14:51:25.618303 7f9d60bd2700 10 getattrs 3.2e_head/8b71082e/rbd_data.10996b8b4567.0000000000000ded/head//3 = 0
2014-06-12 14:51:25.618314 7f9d60bd2700 10 stat 3.2e_head/cd91082e/rbd_data.10b16b8b4567.0000000000001e29/head//3
2014-06-12 14:51:25.618427 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid cd91082e/rbd_data.10b16b8b4567.0000000000001e29/head//3
2014-06-12 14:51:25.633552 7f9d60bd2700 10 getattrs 3.2e_head/cd91082e/rbd_data.10b16b8b4567.0000000000001e29/head//3 = 0
2014-06-12 14:51:25.633566 7f9d60bd2700 10 stat 3.2e_head/ed23082e/rbd_data.10a86b8b4567.00000000000015f8/head//3
2014-06-12 14:51:25.633675 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid ed23082e/rbd_data.10a86b8b4567.00000000000015f8/head//3
2014-06-12 14:51:25.661318 7f9d60bd2700 10 getattrs 3.2e_head/ed23082e/rbd_data.10a86b8b4567.00000000000015f8/head//3 = 0
2014-06-12 14:51:25.661331 7f9d60bd2700 10 stat 3.2e_head/5d85082e/rbd_data.10a66b8b4567.00000000000030b5/head//3
2014-06-12 14:51:25.661441 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 5d85082e/rbd_data.10a66b8b4567.00000000000030b5/head//3
2014-06-12 14:51:25.680588 7f9d60bd2700 10 getattrs 3.2e_head/5d85082e/rbd_data.10a66b8b4567.00000000000030b5/head//3 = 0
2014-06-12 14:51:25.680597 7f9d60bd2700 10 stat 3.2e_head/e00c082e/rbd_data.10a16b8b4567.00000000000035b4/head//3
2014-06-12 14:51:25.680712 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid e00c082e/rbd_data.10a16b8b4567.00000000000035b4/head//3
2014-06-12 14:51:25.696166 7f9d60bd2700 10 getattrs 3.2e_head/e00c082e/rbd_data.10a16b8b4567.00000000000035b4/head//3 = 0
2014-06-12 14:51:25.696175 7f9d60bd2700 10 stat 3.2e_head/244c082e/rbd_data.10b26b8b4567.00000000000034e7/head//3
2014-06-12 14:51:25.696294 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 244c082e/rbd_data.10b26b8b4567.00000000000034e7/head//3
2014-06-12 14:51:25.707789 7f9d60bd2700 10 getattrs 3.2e_head/244c082e/rbd_data.10b26b8b4567.00000000000034e7/head//3 = 0
2014-06-12 14:51:25.707799 7f9d60bd2700 10 stat 3.2e_head/7c8c082e/rbd_data.10996b8b4567.0000000000003ac3/head//3
2014-06-12 14:51:25.707913 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 7c8c082e/rbd_data.10996b8b4567.0000000000003ac3/head//3
2014-06-12 14:51:25.721164 7f9d60bd2700 10 getattrs 3.2e_head/7c8c082e/rbd_data.10996b8b4567.0000000000003ac3/head//3 = 0
2014-06-12 14:51:25.721177 7f9d60bd2700 10 stat 3.2e_head/4b9c082e/rbd_data.108f6b8b4567.0000000000001232/head//3
2014-06-12 14:51:25.721292 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4b9c082e/rbd_data.108f6b8b4567.0000000000001232/head//3
2014-06-12 14:51:25.747610 7f9d60bd2700 10 getattrs 3.2e_head/4b9c082e/rbd_data.108f6b8b4567.0000000000001232/head//3 = 0
2014-06-12 14:51:25.747627 7f9d60bd2700 10 stat 3.2e_head/27ad082e/rbd_data.10b06b8b4567.00000000000022b9/head//3
2014-06-12 14:51:25.747810 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 27ad082e/rbd_data.10b06b8b4567.00000000000022b9/head//3
2014-06-12 14:51:25.768369 7f9d60bd2700 10 getattrs 3.2e_head/27ad082e/rbd_data.10b06b8b4567.00000000000022b9/head//3 = 0
2014-06-12 14:51:25.768391 7f9d60bd2700 10 stat 3.2e_head/9ffe082e/rbd_data.109e6b8b4567.00000000000005a0/head//3
2014-06-12 14:51:25.768580 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 9ffe082e/rbd_data.109e6b8b4567.00000000000005a0/head//3
2014-06-12 14:51:25.801454 7f9d60bd2700 10 getattrs 3.2e_head/9ffe082e/rbd_data.109e6b8b4567.00000000000005a0/head//3 = 0
2014-06-12 14:51:25.801471 7f9d60bd2700 10 stat 3.2e_head/fd9f082e/rbd_data.10ac6b8b4567.0000000000003102/head//3
2014-06-12 14:51:25.801579 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid fd9f082e/rbd_data.10ac6b8b4567.0000000000003102/head//3
2014-06-12 14:51:25.825711 7f9d60bd2700 10 getattrs 3.2e_head/fd9f082e/rbd_data.10ac6b8b4567.0000000000003102/head//3 = 0
2014-06-12 14:51:25.825721 7f9d60bd2700 10 stat 3.2e_head/2f0182e/rbd_data.10ac6b8b4567.0000000000000dfe/head//3
2014-06-12 14:51:25.825834 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 2f0182e/rbd_data.10ac6b8b4567.0000000000000dfe/head//3
2014-06-12 14:51:25.859972 7f9d44101700  5 -- op tracker -- , seq: 748574, time: 2014-06-12 14:51:25.859862, event: header_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.859992 7f9d44101700  5 -- op tracker -- , seq: 748574, time: 2014-06-12 14:51:25.859865, event: throttled, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.860024 7f9d44101700  5 -- op tracker -- , seq: 748574, time: 2014-06-12 14:51:25.859926, event: all_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.860034 7f9d44101700  5 -- op tracker -- , seq: 748574, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.860040 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:25.860048 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:25.860058 7f9d44101700 15 osd.2 173 enqueue_op 0x876fd00 prio 127 cost 10717 latency 0.000195 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:25.860131 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:25.863055 7f9d60bd2700 10 getattrs 3.2e_head/2f0182e/rbd_data.10ac6b8b4567.0000000000000dfe/head//3 = 0
2014-06-12 14:51:25.863069 7f9d60bd2700 10 stat 3.2e_head/a762182e/rbd_data.10ab6b8b4567.0000000000002f00/head//3
2014-06-12 14:51:25.863224 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a762182e/rbd_data.10ab6b8b4567.0000000000002f00/head//3
2014-06-12 14:51:25.887846 7f9d60bd2700 10 getattrs 3.2e_head/a762182e/rbd_data.10ab6b8b4567.0000000000002f00/head//3 = 0
2014-06-12 14:51:25.887855 7f9d60bd2700 10 stat 3.2e_head/b153182e/rbd_data.10946b8b4567.0000000000000fa5/head//3
2014-06-12 14:51:25.887953 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid b153182e/rbd_data.10946b8b4567.0000000000000fa5/head//3
2014-06-12 14:51:25.919084 7f9d60bd2700 10 getattrs 3.2e_head/b153182e/rbd_data.10946b8b4567.0000000000000fa5/head//3 = 0
2014-06-12 14:51:25.919092 7f9d60bd2700 10 stat 3.2e_head/bed3182e/rbd_data.109c6b8b4567.00000000000017a4/head//3
2014-06-12 14:51:25.919182 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid bed3182e/rbd_data.109c6b8b4567.00000000000017a4/head//3
2014-06-12 14:51:25.932783 7f9d60bd2700 10 getattrs 3.2e_head/bed3182e/rbd_data.109c6b8b4567.00000000000017a4/head//3 = 0
2014-06-12 14:51:25.932791 7f9d60bd2700 10 stat 3.2e_head/53b4182e/rbd_data.108e6b8b4567.0000000000002760/head//3
2014-06-12 14:51:25.932880 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 53b4182e/rbd_data.108e6b8b4567.0000000000002760/head//3
2014-06-12 14:51:25.963152 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:25.965274 7f9d60bd2700 10 getattrs 3.2e_head/53b4182e/rbd_data.108e6b8b4567.0000000000002760/head//3 = 0
2014-06-12 14:51:25.965287 7f9d60bd2700 10 stat 3.2e_head/2028182e/rbd_data.10aa6b8b4567.0000000000003580/head//3
2014-06-12 14:51:25.965395 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 2028182e/rbd_data.10aa6b8b4567.0000000000003580/head//3
2014-06-12 14:51:25.976455 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:25.993279 7f9d60bd2700 10 getattrs 3.2e_head/2028182e/rbd_data.10aa6b8b4567.0000000000003580/head//3 = 0
2014-06-12 14:51:25.993297 7f9d60bd2700 10 collection_getattrs 3.2e_head
2014-06-12 14:51:25.993382 7f9d60bd2700 10 collection_getattrs 3.2e_head = 0
2014-06-12 14:51:25.993391 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:25.993432 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:25.993444 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:25.993459 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:25.993465 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:25.993498 7f9d61bd4700 10 osd.2 173 dequeue_op 0x876fd00 prio 127 cost 10717 latency 0.133635 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]
2014-06-12 14:51:25.993550 7f9d61bd4700  5 -- op tracker -- , seq: 748574, time: 2014-06-12 14:51:25.993543, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.993579 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] handle_message: 0x876fd00
2014-06-12 14:51:25.993596 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:25.993639 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:25.993659 7f9d61bd4700  5 -- op tracker -- , seq: 748574, time: 2014-06-12 14:51:25.993654, event: started, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.993667 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:25.993787 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] map version is 173'2847
2014-06-12 14:51:25.993808 7f9d61bd4700 10 osd.2 173 dequeue_op 0x876fd00 finish
2014-06-12 14:51:25.993820 7f9d61bd4700  5 -- op tracker -- , seq: 748574, time: 2014-06-12 14:51:25.993814, event: done, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:25.993843 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:25.993848 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:25.993844 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:25.993858 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:25.993886 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:25.993897 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:25.993933 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:25.993944 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:25.993955 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:25.993987 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fe6ce02e/rbd_data.109b6b8b4567.0000000000001594/head//3, auth == maps.end()
2014-06-12 14:51:25.994026 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj fe6ce02e/rbd_data.109b6b8b4567.0000000000001594/head//3
2014-06-12 14:51:25.994050 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e162f02e/rbd_data.108f6b8b4567.0000000000001694/head//3, auth == maps.end()
2014-06-12 14:51:25.994066 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e162f02e/rbd_data.108f6b8b4567.0000000000001694/head//3
2014-06-12 14:51:25.994081 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2646f02e/rbd_data.109e6b8b4567.00000000000024fb/head//3, auth == maps.end()
2014-06-12 14:51:25.994095 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2646f02e/rbd_data.109e6b8b4567.00000000000024fb/head//3
2014-06-12 14:51:25.994110 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6fe7f02e/rbd_data.10a06b8b4567.000000000000000a/head//3, auth == maps.end()
2014-06-12 14:51:25.994124 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6fe7f02e/rbd_data.10a06b8b4567.000000000000000a/head//3
2014-06-12 14:51:25.994141 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f9cbf02e/rbd_data.10986b8b4567.000000000000190b/head//3, auth == maps.end()
2014-06-12 14:51:25.994157 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f9cbf02e/rbd_data.10986b8b4567.000000000000190b/head//3
2014-06-12 14:51:25.994173 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d2ef02e/rbd_data.10ad6b8b4567.0000000000002d19/head//3, auth == maps.end()
2014-06-12 14:51:25.994187 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d2ef02e/rbd_data.10ad6b8b4567.0000000000002d19/head//3
2014-06-12 14:51:25.994203 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5a0ff02e/rbd_data.10ad6b8b4567.00000000000000b3/head//3, auth == maps.end()
2014-06-12 14:51:25.994217 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5a0ff02e/rbd_data.10ad6b8b4567.00000000000000b3/head//3
2014-06-12 14:51:25.994234 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a86ff02e/rbd_data.109c6b8b4567.0000000000000325/head//3, auth == maps.end()
2014-06-12 14:51:25.994248 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a86ff02e/rbd_data.109c6b8b4567.0000000000000325/head//3
2014-06-12 14:51:25.994266 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8b71082e/rbd_data.10996b8b4567.0000000000000ded/head//3, auth == maps.end()
2014-06-12 14:51:25.994280 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8b71082e/rbd_data.10996b8b4567.0000000000000ded/head//3
2014-06-12 14:51:25.994297 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cd91082e/rbd_data.10b16b8b4567.0000000000001e29/head//3, auth == maps.end()
2014-06-12 14:51:25.994311 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj cd91082e/rbd_data.10b16b8b4567.0000000000001e29/head//3
2014-06-12 14:51:25.994325 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ed23082e/rbd_data.10a86b8b4567.00000000000015f8/head//3, auth == maps.end()
2014-06-12 14:51:25.994339 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ed23082e/rbd_data.10a86b8b4567.00000000000015f8/head//3
2014-06-12 14:51:25.994354 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5d85082e/rbd_data.10a66b8b4567.00000000000030b5/head//3, auth == maps.end()
2014-06-12 14:51:25.994367 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5d85082e/rbd_data.10a66b8b4567.00000000000030b5/head//3
2014-06-12 14:51:25.994382 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e00c082e/rbd_data.10a16b8b4567.00000000000035b4/head//3, auth == maps.end()
2014-06-12 14:51:25.994395 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e00c082e/rbd_data.10a16b8b4567.00000000000035b4/head//3
2014-06-12 14:51:25.994409 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 244c082e/rbd_data.10b26b8b4567.00000000000034e7/head//3, auth == maps.end()
2014-06-12 14:51:25.994423 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 244c082e/rbd_data.10b26b8b4567.00000000000034e7/head//3
2014-06-12 14:51:25.994440 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7c8c082e/rbd_data.10996b8b4567.0000000000003ac3/head//3, auth == maps.end()
2014-06-12 14:51:25.994456 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7c8c082e/rbd_data.10996b8b4567.0000000000003ac3/head//3
2014-06-12 14:51:25.994471 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4b9c082e/rbd_data.108f6b8b4567.0000000000001232/head//3, auth == maps.end()
2014-06-12 14:51:25.994487 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4b9c082e/rbd_data.108f6b8b4567.0000000000001232/head//3
2014-06-12 14:51:25.994502 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 27ad082e/rbd_data.10b06b8b4567.00000000000022b9/head//3, auth == maps.end()
2014-06-12 14:51:25.994522 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 27ad082e/rbd_data.10b06b8b4567.00000000000022b9/head//3
2014-06-12 14:51:25.994539 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9ffe082e/rbd_data.109e6b8b4567.00000000000005a0/head//3, auth == maps.end()
2014-06-12 14:51:25.994555 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9ffe082e/rbd_data.109e6b8b4567.00000000000005a0/head//3
2014-06-12 14:51:25.994573 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fd9f082e/rbd_data.10ac6b8b4567.0000000000003102/head//3, auth == maps.end()
2014-06-12 14:51:25.994589 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj fd9f082e/rbd_data.10ac6b8b4567.0000000000003102/head//3
2014-06-12 14:51:25.994607 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2f0182e/rbd_data.10ac6b8b4567.0000000000000dfe/head//3, auth == maps.end()
2014-06-12 14:51:25.994623 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2f0182e/rbd_data.10ac6b8b4567.0000000000000dfe/head//3
2014-06-12 14:51:25.994639 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a762182e/rbd_data.10ab6b8b4567.0000000000002f00/head//3, auth == maps.end()
2014-06-12 14:51:25.994655 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a762182e/rbd_data.10ab6b8b4567.0000000000002f00/head//3
2014-06-12 14:51:25.994672 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b153182e/rbd_data.10946b8b4567.0000000000000fa5/head//3, auth == maps.end()
2014-06-12 14:51:25.994688 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b153182e/rbd_data.10946b8b4567.0000000000000fa5/head//3
2014-06-12 14:51:25.994704 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bed3182e/rbd_data.109c6b8b4567.00000000000017a4/head//3, auth == maps.end()
2014-06-12 14:51:25.994720 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj bed3182e/rbd_data.109c6b8b4567.00000000000017a4/head//3
2014-06-12 14:51:25.994737 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 53b4182e/rbd_data.108e6b8b4567.0000000000002760/head//3, auth == maps.end()
2014-06-12 14:51:25.994754 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 53b4182e/rbd_data.108e6b8b4567.0000000000002760/head//3
2014-06-12 14:51:25.994770 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2028182e/rbd_data.10aa6b8b4567.0000000000003580/head//3, auth == maps.end()
2014-06-12 14:51:25.994787 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2028182e/rbd_data.10aa6b8b4567.0000000000003580/head//3
2014-06-12 14:51:25.994805 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] 
2014-06-12 14:51:25.994818 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub
2014-06-12 14:51:25.994838 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  2028182e/rbd_data.10aa6b8b4567.0000000000003580/head//3 2028182e/rbd_data.10aa6b8b4567.0000000000003580/head//3(173'2077 client.4307.0:108761 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2077)
2014-06-12 14:51:25.994861 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  53b4182e/rbd_data.108e6b8b4567.0000000000002760/head//3 53b4182e/rbd_data.108e6b8b4567.0000000000002760/head//3(173'1605 client.4279.0:80599 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1605)
2014-06-12 14:51:25.994883 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  bed3182e/rbd_data.109c6b8b4567.00000000000017a4/head//3 bed3182e/rbd_data.109c6b8b4567.00000000000017a4/head//3(173'1100 client.4293.0:51863 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1100)
2014-06-12 14:51:25.994903 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  b153182e/rbd_data.10946b8b4567.0000000000000fa5/head//3 b153182e/rbd_data.10946b8b4567.0000000000000fa5/head//3(173'2823 client.4285.0:141021 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2823)
2014-06-12 14:51:25.994923 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a762182e/rbd_data.10ab6b8b4567.0000000000002f00/head//3 a762182e/rbd_data.10ab6b8b4567.0000000000002f00/head//3(173'1820 client.4308.0:94574 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1820)
2014-06-12 14:51:25.994944 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  2f0182e/rbd_data.10ac6b8b4567.0000000000000dfe/head//3 2f0182e/rbd_data.10ac6b8b4567.0000000000000dfe/head//3(173'2804 client.4309.0:139115 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2804)
2014-06-12 14:51:25.994964 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  fd9f082e/rbd_data.10ac6b8b4567.0000000000003102/head//3 fd9f082e/rbd_data.10ac6b8b4567.0000000000003102/head//3(173'1993 client.4309.0:97417 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1993)
2014-06-12 14:51:25.994984 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  9ffe082e/rbd_data.109e6b8b4567.00000000000005a0/head//3 9ffe082e/rbd_data.109e6b8b4567.00000000000005a0/head//3(173'2526 client.4295.0:125685 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2526)
2014-06-12 14:51:25.995005 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  27ad082e/rbd_data.10b06b8b4567.00000000000022b9/head//3 27ad082e/rbd_data.10b06b8b4567.00000000000022b9/head//3(173'1404 client.4313.0:72755 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1404)
2014-06-12 14:51:25.995025 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4b9c082e/rbd_data.108f6b8b4567.0000000000001232/head//3 4b9c082e/rbd_data.108f6b8b4567.0000000000001232/head//3(173'878 client.4280.0:41606 wrlock_by=unknown.0.0:0 dirty s 4194304 uv878)
2014-06-12 14:51:25.995045 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  7c8c082e/rbd_data.10996b8b4567.0000000000003ac3/head//3 7c8c082e/rbd_data.10996b8b4567.0000000000003ac3/head//3(173'2274 client.4290.0:114522 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2274)
2014-06-12 14:51:25.995066 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  244c082e/rbd_data.10b26b8b4567.00000000000034e7/head//3 244c082e/rbd_data.10b26b8b4567.00000000000034e7/head//3(173'1945 client.4315.0:105164 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1945)
2014-06-12 14:51:25.995086 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  e00c082e/rbd_data.10a16b8b4567.00000000000035b4/head//3 e00c082e/rbd_data.10a16b8b4567.00000000000035b4/head//3(173'2091 client.4298.0:107728 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2091)
2014-06-12 14:51:25.995106 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  5d85082e/rbd_data.10a66b8b4567.00000000000030b5/head//3 5d85082e/rbd_data.10a66b8b4567.00000000000030b5/head//3(173'1930 client.4303.0:96972 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1930)
2014-06-12 14:51:25.995126 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  ed23082e/rbd_data.10a86b8b4567.00000000000015f8/head//3 ed23082e/rbd_data.10a86b8b4567.00000000000015f8/head//3(173'953 client.4305.0:48500 wrlock_by=unknown.0.0:0 dirty s 4194304 uv953)
2014-06-12 14:51:25.995149 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  cd91082e/rbd_data.10b16b8b4567.0000000000001e29/head//3 cd91082e/rbd_data.10b16b8b4567.0000000000001e29/head//3(173'1306 client.4314.0:63747 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1306)
2014-06-12 14:51:25.995170 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8b71082e/rbd_data.10996b8b4567.0000000000000ded/head//3 8b71082e/rbd_data.10996b8b4567.0000000000000ded/head//3(173'2771 client.4290.0:139815 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2771)
2014-06-12 14:51:25.995189 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a86ff02e/rbd_data.109c6b8b4567.0000000000000325/head//3 a86ff02e/rbd_data.109c6b8b4567.0000000000000325/head//3(173'2452 client.4293.0:122135 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2452)
2014-06-12 14:51:25.995210 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  5a0ff02e/rbd_data.10ad6b8b4567.00000000000000b3/head//3 5a0ff02e/rbd_data.10ad6b8b4567.00000000000000b3/head//3(173'2357 client.4310.0:118373 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2357)
2014-06-12 14:51:25.995230 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d2ef02e/rbd_data.10ad6b8b4567.0000000000002d19/head//3 d2ef02e/rbd_data.10ad6b8b4567.0000000000002d19/head//3(173'1741 client.4310.0:90691 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1741)
2014-06-12 14:51:25.995250 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f9cbf02e/rbd_data.10986b8b4567.000000000000190b/head//3 f9cbf02e/rbd_data.10986b8b4567.000000000000190b/head//3(173'1109 client.4289.0:53863 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1109)
2014-06-12 14:51:25.995270 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6fe7f02e/rbd_data.10a06b8b4567.000000000000000a/head//3 6fe7f02e/rbd_data.10a06b8b4567.000000000000000a/head//3(173'2338 client.4297.0:117064 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2338)
2014-06-12 14:51:25.995290 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  2646f02e/rbd_data.109e6b8b4567.00000000000024fb/head//3 2646f02e/rbd_data.109e6b8b4567.00000000000024fb/head//3(173'1493 client.4295.0:76935 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1493)
2014-06-12 14:51:25.995310 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  e162f02e/rbd_data.108f6b8b4567.0000000000001694/head//3 e162f02e/rbd_data.108f6b8b4567.0000000000001694/head//3(173'1056 client.4280.0:49765 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1056)
2014-06-12 14:51:25.995330 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  fe6ce02e/rbd_data.109b6b8b4567.0000000000001594/head//3 fe6ce02e/rbd_data.109b6b8b4567.0000000000001594/head//3(173'1084 client.4292.0:47039 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1084)
2014-06-12 14:51:25.995346 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:25.995363 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:25.995377 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:25.995380 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:25.995382 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:25.995431 7f9d60bd2700 10 collection_list_partial 3.2e_head start:919182e//0//-1 is_max:0
2014-06-12 14:51:25.995741 7f9d60bd2700 20 objects: [7a3e182e/rbd_data.109c6b8b4567.0000000000002ee6/head//3,a58e182e/rbd_data.108e6b8b4567.0000000000002612/head//3,177f182e/rbd_data.10ac6b8b4567.00000000000025ad/head//3,fd02282e/rbd_data.10a76b8b4567.0000000000003a8c/head//3,2f24282e/rbd_data.10b36b8b4567.00000000000016fc/head//3,7e85282e/rbd_data.10926b8b4567.00000000000015b2/head//3,5618282e/rbd_data.109b6b8b4567.0000000000001c3b/head//3,9378282e/rbd_data.10b06b8b4567.0000000000003a8a/head//3,2fbb282e/rbd_data.10a36b8b4567.00000000000001d8/head//3,777f282e/rbd_data.10956b8b4567.00000000000014f5/head//3,cc9f282e/rbd_data.109d6b8b4567.00000000000024bc/head//3,b383382e/rbd_data.10ae6b8b4567.000000000000052d/head//3,e7f3382e/rbd_data.109f6b8b4567.0000000000003878/head//3,b525382e/rbd_data.10976b8b4567.0000000000000043/head//3,5558382e/rbd_data.10996b8b4567.0000000000000e74/head//3,4fca382e/rbd_data.10b46b8b4567.0000000000001f40/head//3,29da382e/rbd_data.10a16b8b4567.0000000000003834/head//3,ccbb382e/rbd_data.10b26b8b4567.0000000000003754/head//3,62bd382e/rbd_data.10926b8b4567.00000000000022cb/head//3,bdf382e/rbd_data.10956b8b4567.00000000000005f3/head//3,2ec0482e/rbd_data.10a26b8b4567.0000000000002398/head//3,7581482e/rbd_data.109f6b8b4567.0000000000002450/head//3,aab1482e/rbd_data.10b16b8b4567.0000000000000e5c/head//3,66f1482e/rbd_data.10a36b8b4567.0000000000000ce3/head//3,f4a3482e/rbd_data.109d6b8b4567.0000000000001d14/head//3]
2014-06-12 14:51:25.996316 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:25.996350 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:25.996375 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:25.996388 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:25.996398 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:25.996407 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:25.996417 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_map_chunk [919182e//0//-1,aef7482e//0//-1)
2014-06-12 14:51:25.996429 7f9d60bd2700 10 collection_list_partial 3.2e_head start:919182e//0//-1 is_max:0
2014-06-12 14:51:25.997208 7f9d60bd2700 20 objects: [7a3e182e/rbd_data.109c6b8b4567.0000000000002ee6/head//3,a58e182e/rbd_data.108e6b8b4567.0000000000002612/head//3,177f182e/rbd_data.10ac6b8b4567.00000000000025ad/head//3,fd02282e/rbd_data.10a76b8b4567.0000000000003a8c/head//3,2f24282e/rbd_data.10b36b8b4567.00000000000016fc/head//3,7e85282e/rbd_data.10926b8b4567.00000000000015b2/head//3,5618282e/rbd_data.109b6b8b4567.0000000000001c3b/head//3,9378282e/rbd_data.10b06b8b4567.0000000000003a8a/head//3,2fbb282e/rbd_data.10a36b8b4567.00000000000001d8/head//3,777f282e/rbd_data.10956b8b4567.00000000000014f5/head//3,cc9f282e/rbd_data.109d6b8b4567.00000000000024bc/head//3,b383382e/rbd_data.10ae6b8b4567.000000000000052d/head//3,e7f3382e/rbd_data.109f6b8b4567.0000000000003878/head//3,b525382e/rbd_data.10976b8b4567.0000000000000043/head//3,5558382e/rbd_data.10996b8b4567.0000000000000e74/head//3,4fca382e/rbd_data.10b46b8b4567.0000000000001f40/head//3,29da382e/rbd_data.10a16b8b4567.0000000000003834/head//3,ccbb382e/rbd_data.10b26b8b4567.0000000000003754/head//3,62bd382e/rbd_data.10926b8b4567.00000000000022cb/head//3,bdf382e/rbd_data.10956b8b4567.00000000000005f3/head//3,2ec0482e/rbd_data.10a26b8b4567.0000000000002398/head//3,7581482e/rbd_data.109f6b8b4567.0000000000002450/head//3,aab1482e/rbd_data.10b16b8b4567.0000000000000e5c/head//3,66f1482e/rbd_data.10a36b8b4567.0000000000000ce3/head//3,f4a3482e/rbd_data.109d6b8b4567.0000000000001d14/head//3,aef7482e/rbd_data.109a6b8b4567.00000000000039d6/head//3,8a19482e/rbd_data.10ab6b8b4567.000000000000025e/head//3,c7f9482e/rbd_data.108e6b8b4567.0000000000003aa3/head//3,19ea482e/rbd_data.10aa6b8b4567.00000000000006c0/head//3,9ad482e/rbd_data.10956b8b4567.000000000000205f/head//3,e03f482e/rbd_data.10af6b8b4567.00000000000013dd/head//3,6870582e/rbd_data.10ae6b8b4567.0000000000001dcd/head//3,1ac2582e/rbd_data.109c6b8b4567.00000000000037f4/head//3,44f3582e/rbd_data.109c6b8b4567.000000000000156d/head//3,b474582e/rbd_data.10ae6b8b4567.00000000000001c7/head//3,54a4582e/rbd_data.108e6b8b4567.00000000000003db/head//3,8e08582e/rbd_data.10976b8b4567.00000000000024bd/head//3,8278582e/rbd_data.10ae6b8b4567.0000000000000cf8/head//3,41c8582e/rbd_data.10ab6b8b4567.00000000000002ce/head//3,4e30682e/rbd_data.10b26b8b4567.0000000000002fc6/head//3,b331682e/rbd_data.10966b8b4567.0000000000003768/head//3,d7f1682e/rbd_data.10b26b8b4567.0000000000002e68/head//3,a632682e/rbd_data.109f6b8b4567.0000000000002d7e/head//3,a472682e/rbd_data.108d6b8b4567.000000000000336c/head//3,5184682e/rbd_data.10aa6b8b4567.000000000000252e/head//3,efc5682e/rbd_data.10976b8b4567.0000000000000a87/head//3,9f66682e/rbd_data.109d6b8b4567.00000000000036b8/head//3,9e07682e/rbd_data.10b36b8b4567.00000000000009e0/head//3,aea7682e/rbd_data.10a16b8b4567.00000000000035b6/head//3,8b09682e/rbd_data.10b46b8b4567.00000000000005f4/head//3,bca682e/rbd_data.10936b8b4567.000000000000144b/head//3,54bb682e/rbd_data.10a76b8b4567.0000000000000b33/head//3,d1ce682e/rbd_data.10ab6b8b4567.00000000000031e2/head//3,3e10782e/rbd_data.10ac6b8b4567.0000000000003557/head//3,d30782e/rbd_data.10a96b8b4567.00000000000010ef/head//3,e960782e/rbd_data.10a46b8b4567.0000000000001091/head//3,fb61782e/rbd_data.109c6b8b4567.00000000000035b5/head//3,5943782e/rbd_data.109e6b8b4567.0000000000000e31/head//3,c1c4782e/rbd_data.10956b8b4567.00000000000016e6/head//3,2ff4782e/rbd_data.10a06b8b4567.0000000000000801/head//3,73c5782e/rbd_data.10a76b8b4567.0000000000003810/head//3,2e86782e/rbd_data.10a96b8b4567.00000000000000e8/head//3,f2a8782e/rbd_data.10946b8b4567.00000000000034f7/head//3,6159782e/rbd_data.10966b8b4567.0000000000003b0b/head//3]
2014-06-12 14:51:25.997322 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:25.997348 7f9d60bd2700 10 stat 3.2e_head/7a3e182e/rbd_data.109c6b8b4567.0000000000002ee6/head//3
2014-06-12 14:51:25.997534 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 7a3e182e/rbd_data.109c6b8b4567.0000000000002ee6/head//3
2014-06-12 14:51:26.021814 7f9d60bd2700 10 getattrs 3.2e_head/7a3e182e/rbd_data.109c6b8b4567.0000000000002ee6/head//3 = 0
2014-06-12 14:51:26.021836 7f9d60bd2700 10 stat 3.2e_head/a58e182e/rbd_data.108e6b8b4567.0000000000002612/head//3
2014-06-12 14:51:26.021993 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a58e182e/rbd_data.108e6b8b4567.0000000000002612/head//3
2014-06-12 14:51:26.040159 7f9d60bd2700 10 getattrs 3.2e_head/a58e182e/rbd_data.108e6b8b4567.0000000000002612/head//3 = 0
2014-06-12 14:51:26.040168 7f9d60bd2700 10 stat 3.2e_head/177f182e/rbd_data.10ac6b8b4567.00000000000025ad/head//3
2014-06-12 14:51:26.040269 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 177f182e/rbd_data.10ac6b8b4567.00000000000025ad/head//3
2014-06-12 14:51:26.052203 7f9d60bd2700 10 getattrs 3.2e_head/177f182e/rbd_data.10ac6b8b4567.00000000000025ad/head//3 = 0
2014-06-12 14:51:26.052211 7f9d60bd2700 10 stat 3.2e_head/fd02282e/rbd_data.10a76b8b4567.0000000000003a8c/head//3
2014-06-12 14:51:26.052304 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid fd02282e/rbd_data.10a76b8b4567.0000000000003a8c/head//3
2014-06-12 14:51:26.064196 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:26.068407 7f9d60bd2700 10 getattrs 3.2e_head/fd02282e/rbd_data.10a76b8b4567.0000000000003a8c/head//3 = 0
2014-06-12 14:51:26.068415 7f9d60bd2700 10 stat 3.2e_head/2f24282e/rbd_data.10b36b8b4567.00000000000016fc/head//3
2014-06-12 14:51:26.068507 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 2f24282e/rbd_data.10b36b8b4567.00000000000016fc/head//3
2014-06-12 14:51:26.089499 7f9d60bd2700 10 getattrs 3.2e_head/2f24282e/rbd_data.10b36b8b4567.00000000000016fc/head//3 = 0
2014-06-12 14:51:26.089506 7f9d60bd2700 10 stat 3.2e_head/7e85282e/rbd_data.10926b8b4567.00000000000015b2/head//3
2014-06-12 14:51:26.089596 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 7e85282e/rbd_data.10926b8b4567.00000000000015b2/head//3
2014-06-12 14:51:26.104400 7f9d60bd2700 10 getattrs 3.2e_head/7e85282e/rbd_data.10926b8b4567.00000000000015b2/head//3 = 0
2014-06-12 14:51:26.104410 7f9d60bd2700 10 stat 3.2e_head/5618282e/rbd_data.109b6b8b4567.0000000000001c3b/head//3
2014-06-12 14:51:26.104510 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 5618282e/rbd_data.109b6b8b4567.0000000000001c3b/head//3
2014-06-12 14:51:26.127424 7f9d60bd2700 10 getattrs 3.2e_head/5618282e/rbd_data.109b6b8b4567.0000000000001c3b/head//3 = 0
2014-06-12 14:51:26.127438 7f9d60bd2700 10 stat 3.2e_head/9378282e/rbd_data.10b06b8b4567.0000000000003a8a/head//3
2014-06-12 14:51:26.127595 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 9378282e/rbd_data.10b06b8b4567.0000000000003a8a/head//3
2014-06-12 14:51:26.140715 7f9d60bd2700 10 getattrs 3.2e_head/9378282e/rbd_data.10b06b8b4567.0000000000003a8a/head//3 = 0
2014-06-12 14:51:26.140722 7f9d60bd2700 10 stat 3.2e_head/2fbb282e/rbd_data.10a36b8b4567.00000000000001d8/head//3
2014-06-12 14:51:26.140816 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 2fbb282e/rbd_data.10a36b8b4567.00000000000001d8/head//3
2014-06-12 14:51:26.166951 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:26.201378 7f9d60bd2700 10 getattrs 3.2e_head/2fbb282e/rbd_data.10a36b8b4567.00000000000001d8/head//3 = 0
2014-06-12 14:51:26.201394 7f9d60bd2700 10 stat 3.2e_head/777f282e/rbd_data.10956b8b4567.00000000000014f5/head//3
2014-06-12 14:51:26.201525 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 777f282e/rbd_data.10956b8b4567.00000000000014f5/head//3
2014-06-12 14:51:26.212189 7f9d60bd2700 10 getattrs 3.2e_head/777f282e/rbd_data.10956b8b4567.00000000000014f5/head//3 = 0
2014-06-12 14:51:26.212201 7f9d60bd2700 10 stat 3.2e_head/cc9f282e/rbd_data.109d6b8b4567.00000000000024bc/head//3
2014-06-12 14:51:26.212311 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid cc9f282e/rbd_data.109d6b8b4567.00000000000024bc/head//3
2014-06-12 14:51:26.239014 7f9d60bd2700 10 getattrs 3.2e_head/cc9f282e/rbd_data.109d6b8b4567.00000000000024bc/head//3 = 0
2014-06-12 14:51:26.239025 7f9d60bd2700 10 stat 3.2e_head/b383382e/rbd_data.10ae6b8b4567.000000000000052d/head//3
2014-06-12 14:51:26.239147 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid b383382e/rbd_data.10ae6b8b4567.000000000000052d/head//3
2014-06-12 14:51:26.285748 7f9d60bd2700 10 getattrs 3.2e_head/b383382e/rbd_data.10ae6b8b4567.000000000000052d/head//3 = 0
2014-06-12 14:51:26.285761 7f9d60bd2700 10 stat 3.2e_head/e7f3382e/rbd_data.109f6b8b4567.0000000000003878/head//3
2014-06-12 14:51:26.285859 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid e7f3382e/rbd_data.109f6b8b4567.0000000000003878/head//3
2014-06-12 14:51:26.292634 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:26.292765 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:26.308728 7f9d60bd2700 10 getattrs 3.2e_head/e7f3382e/rbd_data.109f6b8b4567.0000000000003878/head//3 = 0
2014-06-12 14:51:26.308737 7f9d60bd2700 10 stat 3.2e_head/b525382e/rbd_data.10976b8b4567.0000000000000043/head//3
2014-06-12 14:51:26.308845 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid b525382e/rbd_data.10976b8b4567.0000000000000043/head//3
2014-06-12 14:51:26.316372 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:26.325885 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:26.345222 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:26.345248 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:26.345253 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:26.345255 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:26.345294 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:26.362714 7f9d60bd2700 10 getattrs 3.2e_head/b525382e/rbd_data.10976b8b4567.0000000000000043/head//3 = 0
2014-06-12 14:51:26.362730 7f9d60bd2700 10 stat 3.2e_head/5558382e/rbd_data.10996b8b4567.0000000000000e74/head//3
2014-06-12 14:51:26.362830 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 5558382e/rbd_data.10996b8b4567.0000000000000e74/head//3
2014-06-12 14:51:26.365791 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:26.370609 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:26.379617 7f9d60bd2700 10 getattrs 3.2e_head/5558382e/rbd_data.10996b8b4567.0000000000000e74/head//3 = 0
2014-06-12 14:51:26.379625 7f9d60bd2700 10 stat 3.2e_head/4fca382e/rbd_data.10b46b8b4567.0000000000001f40/head//3
2014-06-12 14:51:26.379723 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4fca382e/rbd_data.10b46b8b4567.0000000000001f40/head//3
2014-06-12 14:51:26.381412 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:26.408899 7f9d60bd2700 10 getattrs 3.2e_head/4fca382e/rbd_data.10b46b8b4567.0000000000001f40/head//3 = 0
2014-06-12 14:51:26.408909 7f9d60bd2700 10 stat 3.2e_head/29da382e/rbd_data.10a16b8b4567.0000000000003834/head//3
2014-06-12 14:51:26.409006 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 29da382e/rbd_data.10a16b8b4567.0000000000003834/head//3
2014-06-12 14:51:26.411051 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:26.411182 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:26.443037 7f9d60bd2700 10 getattrs 3.2e_head/29da382e/rbd_data.10a16b8b4567.0000000000003834/head//3 = 0
2014-06-12 14:51:26.443046 7f9d60bd2700 10 stat 3.2e_head/ccbb382e/rbd_data.10b26b8b4567.0000000000003754/head//3
2014-06-12 14:51:26.443147 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid ccbb382e/rbd_data.10b26b8b4567.0000000000003754/head//3
2014-06-12 14:51:26.462207 7f9d60bd2700 10 getattrs 3.2e_head/ccbb382e/rbd_data.10b26b8b4567.0000000000003754/head//3 = 0
2014-06-12 14:51:26.462224 7f9d60bd2700 10 stat 3.2e_head/62bd382e/rbd_data.10926b8b4567.00000000000022cb/head//3
2014-06-12 14:51:26.462379 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 62bd382e/rbd_data.10926b8b4567.00000000000022cb/head//3
2014-06-12 14:51:26.483961 7f9d60bd2700 10 getattrs 3.2e_head/62bd382e/rbd_data.10926b8b4567.00000000000022cb/head//3 = 0
2014-06-12 14:51:26.483968 7f9d60bd2700 10 stat 3.2e_head/bdf382e/rbd_data.10956b8b4567.00000000000005f3/head//3
2014-06-12 14:51:26.484062 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid bdf382e/rbd_data.10956b8b4567.00000000000005f3/head//3
2014-06-12 14:51:26.504787 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:26.512630 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:26.518435 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:26.522404 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:26.529504 7f9d60bd2700 10 getattrs 3.2e_head/bdf382e/rbd_data.10956b8b4567.00000000000005f3/head//3 = 0
2014-06-12 14:51:26.529516 7f9d60bd2700 10 stat 3.2e_head/2ec0482e/rbd_data.10a26b8b4567.0000000000002398/head//3
2014-06-12 14:51:26.529661 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 2ec0482e/rbd_data.10a26b8b4567.0000000000002398/head//3
2014-06-12 14:51:26.531824 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:26.556575 7f9d60bd2700 10 getattrs 3.2e_head/2ec0482e/rbd_data.10a26b8b4567.0000000000002398/head//3 = 0
2014-06-12 14:51:26.556587 7f9d60bd2700 10 stat 3.2e_head/7581482e/rbd_data.109f6b8b4567.0000000000002450/head//3
2014-06-12 14:51:26.556723 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 7581482e/rbd_data.109f6b8b4567.0000000000002450/head//3
2014-06-12 14:51:26.568790 7f9d60bd2700 10 getattrs 3.2e_head/7581482e/rbd_data.109f6b8b4567.0000000000002450/head//3 = 0
2014-06-12 14:51:26.568814 7f9d60bd2700 10 stat 3.2e_head/aab1482e/rbd_data.10b16b8b4567.0000000000000e5c/head//3
2014-06-12 14:51:26.568942 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid aab1482e/rbd_data.10b16b8b4567.0000000000000e5c/head//3
2014-06-12 14:51:26.599361 7f9d60bd2700 10 getattrs 3.2e_head/aab1482e/rbd_data.10b16b8b4567.0000000000000e5c/head//3 = 0
2014-06-12 14:51:26.599375 7f9d60bd2700 10 stat 3.2e_head/66f1482e/rbd_data.10a36b8b4567.0000000000000ce3/head//3
2014-06-12 14:51:26.599528 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 66f1482e/rbd_data.10a36b8b4567.0000000000000ce3/head//3
2014-06-12 14:51:26.648492 7f9d60bd2700 10 getattrs 3.2e_head/66f1482e/rbd_data.10a36b8b4567.0000000000000ce3/head//3 = 0
2014-06-12 14:51:26.648501 7f9d60bd2700 10 stat 3.2e_head/f4a3482e/rbd_data.109d6b8b4567.0000000000001d14/head//3
2014-06-12 14:51:26.648603 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f4a3482e/rbd_data.109d6b8b4567.0000000000001d14/head//3
2014-06-12 14:51:26.649381 7f9d44101700  5 -- op tracker -- , seq: 748575, time: 2014-06-12 14:51:26.649293, event: header_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:26.649399 7f9d44101700  5 -- op tracker -- , seq: 748575, time: 2014-06-12 14:51:26.649296, event: throttled, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:26.649404 7f9d44101700  5 -- op tracker -- , seq: 748575, time: 2014-06-12 14:51:26.649347, event: all_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:26.649437 7f9d44101700  5 -- op tracker -- , seq: 748575, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:26.649442 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:26.649447 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:26.649457 7f9d44101700 15 osd.2 173 enqueue_op 0x6d11900 prio 127 cost 10717 latency 0.000164 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:26.649520 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:26.661172 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:26.664286 7f9d60bd2700 10 getattrs 3.2e_head/f4a3482e/rbd_data.109d6b8b4567.0000000000001d14/head//3 = 0
2014-06-12 14:51:26.664296 7f9d60bd2700 10 collection_getattrs 3.2e_head
2014-06-12 14:51:26.664357 7f9d60bd2700 10 collection_getattrs 3.2e_head = 0
2014-06-12 14:51:26.664361 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:26.664395 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:26.664402 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:26.664413 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:26.664415 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:26.664441 7f9d683e1700 10 osd.2 173 dequeue_op 0x6d11900 prio 127 cost 10717 latency 0.015148 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]
2014-06-12 14:51:26.664484 7f9d683e1700  5 -- op tracker -- , seq: 748575, time: 2014-06-12 14:51:26.664477, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:26.664516 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] handle_message: 0x6d11900
2014-06-12 14:51:26.664532 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:26.664550 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:26.664567 7f9d683e1700  5 -- op tracker -- , seq: 748575, time: 2014-06-12 14:51:26.664562, event: started, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:26.664581 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:26.664683 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] map version is 173'2847
2014-06-12 14:51:26.664703 7f9d683e1700 10 osd.2 173 dequeue_op 0x6d11900 finish
2014-06-12 14:51:26.664715 7f9d683e1700  5 -- op tracker -- , seq: 748575, time: 2014-06-12 14:51:26.664709, event: done, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:26.664722 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:26.664737 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:26.664742 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:26.664732 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:26.664749 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:26.664761 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:26.664803 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:26.664813 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:26.664820 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:26.664840 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7a3e182e/rbd_data.109c6b8b4567.0000000000002ee6/head//3, auth == maps.end()
2014-06-12 14:51:26.664853 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7a3e182e/rbd_data.109c6b8b4567.0000000000002ee6/head//3
2014-06-12 14:51:26.664869 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a58e182e/rbd_data.108e6b8b4567.0000000000002612/head//3, auth == maps.end()
2014-06-12 14:51:26.664880 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a58e182e/rbd_data.108e6b8b4567.0000000000002612/head//3
2014-06-12 14:51:26.664889 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 177f182e/rbd_data.10ac6b8b4567.00000000000025ad/head//3, auth == maps.end()
2014-06-12 14:51:26.664898 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 177f182e/rbd_data.10ac6b8b4567.00000000000025ad/head//3
2014-06-12 14:51:26.664907 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fd02282e/rbd_data.10a76b8b4567.0000000000003a8c/head//3, auth == maps.end()
2014-06-12 14:51:26.664915 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj fd02282e/rbd_data.10a76b8b4567.0000000000003a8c/head//3
2014-06-12 14:51:26.664924 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2f24282e/rbd_data.10b36b8b4567.00000000000016fc/head//3, auth == maps.end()
2014-06-12 14:51:26.664932 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2f24282e/rbd_data.10b36b8b4567.00000000000016fc/head//3
2014-06-12 14:51:26.664941 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7e85282e/rbd_data.10926b8b4567.00000000000015b2/head//3, auth == maps.end()
2014-06-12 14:51:26.664949 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7e85282e/rbd_data.10926b8b4567.00000000000015b2/head//3
2014-06-12 14:51:26.664958 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5618282e/rbd_data.109b6b8b4567.0000000000001c3b/head//3, auth == maps.end()
2014-06-12 14:51:26.664967 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5618282e/rbd_data.109b6b8b4567.0000000000001c3b/head//3
2014-06-12 14:51:26.664976 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9378282e/rbd_data.10b06b8b4567.0000000000003a8a/head//3, auth == maps.end()
2014-06-12 14:51:26.664985 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9378282e/rbd_data.10b06b8b4567.0000000000003a8a/head//3
2014-06-12 14:51:26.664995 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2fbb282e/rbd_data.10a36b8b4567.00000000000001d8/head//3, auth == maps.end()
2014-06-12 14:51:26.665003 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2fbb282e/rbd_data.10a36b8b4567.00000000000001d8/head//3
2014-06-12 14:51:26.665012 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 777f282e/rbd_data.10956b8b4567.00000000000014f5/head//3, auth == maps.end()
2014-06-12 14:51:26.665022 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 777f282e/rbd_data.10956b8b4567.00000000000014f5/head//3
2014-06-12 14:51:26.665031 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cc9f282e/rbd_data.109d6b8b4567.00000000000024bc/head//3, auth == maps.end()
2014-06-12 14:51:26.665040 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj cc9f282e/rbd_data.109d6b8b4567.00000000000024bc/head//3
2014-06-12 14:51:26.665050 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b383382e/rbd_data.10ae6b8b4567.000000000000052d/head//3, auth == maps.end()
2014-06-12 14:51:26.665059 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b383382e/rbd_data.10ae6b8b4567.000000000000052d/head//3
2014-06-12 14:51:26.665069 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e7f3382e/rbd_data.109f6b8b4567.0000000000003878/head//3, auth == maps.end()
2014-06-12 14:51:26.665078 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e7f3382e/rbd_data.109f6b8b4567.0000000000003878/head//3
2014-06-12 14:51:26.665089 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b525382e/rbd_data.10976b8b4567.0000000000000043/head//3, auth == maps.end()
2014-06-12 14:51:26.665098 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b525382e/rbd_data.10976b8b4567.0000000000000043/head//3
2014-06-12 14:51:26.665107 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5558382e/rbd_data.10996b8b4567.0000000000000e74/head//3, auth == maps.end()
2014-06-12 14:51:26.665117 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5558382e/rbd_data.10996b8b4567.0000000000000e74/head//3
2014-06-12 14:51:26.665128 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4fca382e/rbd_data.10b46b8b4567.0000000000001f40/head//3, auth == maps.end()
2014-06-12 14:51:26.665137 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4fca382e/rbd_data.10b46b8b4567.0000000000001f40/head//3
2014-06-12 14:51:26.665147 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 29da382e/rbd_data.10a16b8b4567.0000000000003834/head//3, auth == maps.end()
2014-06-12 14:51:26.665161 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 29da382e/rbd_data.10a16b8b4567.0000000000003834/head//3
2014-06-12 14:51:26.665172 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ccbb382e/rbd_data.10b26b8b4567.0000000000003754/head//3, auth == maps.end()
2014-06-12 14:51:26.665182 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ccbb382e/rbd_data.10b26b8b4567.0000000000003754/head//3
2014-06-12 14:51:26.665192 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 62bd382e/rbd_data.10926b8b4567.00000000000022cb/head//3, auth == maps.end()
2014-06-12 14:51:26.665202 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 62bd382e/rbd_data.10926b8b4567.00000000000022cb/head//3
2014-06-12 14:51:26.665211 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bdf382e/rbd_data.10956b8b4567.00000000000005f3/head//3, auth == maps.end()
2014-06-12 14:51:26.665220 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj bdf382e/rbd_data.10956b8b4567.00000000000005f3/head//3
2014-06-12 14:51:26.665230 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2ec0482e/rbd_data.10a26b8b4567.0000000000002398/head//3, auth == maps.end()
2014-06-12 14:51:26.665240 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2ec0482e/rbd_data.10a26b8b4567.0000000000002398/head//3
2014-06-12 14:51:26.665251 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7581482e/rbd_data.109f6b8b4567.0000000000002450/head//3, auth == maps.end()
2014-06-12 14:51:26.665261 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7581482e/rbd_data.109f6b8b4567.0000000000002450/head//3
2014-06-12 14:51:26.665272 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aab1482e/rbd_data.10b16b8b4567.0000000000000e5c/head//3, auth == maps.end()
2014-06-12 14:51:26.665283 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj aab1482e/rbd_data.10b16b8b4567.0000000000000e5c/head//3
2014-06-12 14:51:26.665293 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 66f1482e/rbd_data.10a36b8b4567.0000000000000ce3/head//3, auth == maps.end()
2014-06-12 14:51:26.665303 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 66f1482e/rbd_data.10a36b8b4567.0000000000000ce3/head//3
2014-06-12 14:51:26.665313 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f4a3482e/rbd_data.109d6b8b4567.0000000000001d14/head//3, auth == maps.end()
2014-06-12 14:51:26.665322 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f4a3482e/rbd_data.109d6b8b4567.0000000000001d14/head//3
2014-06-12 14:51:26.665333 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] 
2014-06-12 14:51:26.665341 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub
2014-06-12 14:51:26.665353 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f4a3482e/rbd_data.109d6b8b4567.0000000000001d14/head//3 f4a3482e/rbd_data.109d6b8b4567.0000000000001d14/head//3(173'1215 client.4294.0:61714 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1215)
2014-06-12 14:51:26.665368 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  66f1482e/rbd_data.10a36b8b4567.0000000000000ce3/head//3 66f1482e/rbd_data.10a36b8b4567.0000000000000ce3/head//3(173'2783 client.4300.0:136653 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2783)
2014-06-12 14:51:26.665381 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  aab1482e/rbd_data.10b16b8b4567.0000000000000e5c/head//3 aab1482e/rbd_data.10b16b8b4567.0000000000000e5c/head//3(173'2776 client.4314.0:141686 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2776)
2014-06-12 14:51:26.665393 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  7581482e/rbd_data.109f6b8b4567.0000000000002450/head//3 7581482e/rbd_data.109f6b8b4567.0000000000002450/head//3(173'1335 client.4296.0:72633 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1335)
2014-06-12 14:51:26.665406 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  2ec0482e/rbd_data.10a26b8b4567.0000000000002398/head//3 2ec0482e/rbd_data.10a26b8b4567.0000000000002398/head//3(173'1599 client.4299.0:75360 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1599)
2014-06-12 14:51:26.665418 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  bdf382e/rbd_data.10956b8b4567.00000000000005f3/head//3 bdf382e/rbd_data.10956b8b4567.00000000000005f3/head//3(173'2565 client.4286.0:127081 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2565)
2014-06-12 14:51:26.665431 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  62bd382e/rbd_data.10926b8b4567.00000000000022cb/head//3 62bd382e/rbd_data.10926b8b4567.00000000000022cb/head//3(173'1397 client.4283.0:71257 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1397)
2014-06-12 14:51:26.665444 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  ccbb382e/rbd_data.10b26b8b4567.0000000000003754/head//3 ccbb382e/rbd_data.10b26b8b4567.0000000000003754/head//3(173'2016 client.4315.0:109787 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2016)
2014-06-12 14:51:26.665456 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  29da382e/rbd_data.10a16b8b4567.0000000000003834/head//3 29da382e/rbd_data.10a16b8b4567.0000000000003834/head//3(173'2201 client.4298.0:112358 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2201)
2014-06-12 14:51:26.665469 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4fca382e/rbd_data.10b46b8b4567.0000000000001f40/head//3 4fca382e/rbd_data.10b46b8b4567.0000000000001f40/head//3(173'1374 client.4317.0:65394 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1374)
2014-06-12 14:51:26.665481 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  5558382e/rbd_data.10996b8b4567.0000000000000e74/head//3 5558382e/rbd_data.10996b8b4567.0000000000000e74/head//3(173'2809 client.4290.0:140771 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2809)
2014-06-12 14:51:26.665494 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  b525382e/rbd_data.10976b8b4567.0000000000000043/head//3 b525382e/rbd_data.10976b8b4567.0000000000000043/head//3(173'2341 client.4288.0:115517 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2341)
2014-06-12 14:51:26.665507 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  e7f3382e/rbd_data.109f6b8b4567.0000000000003878/head//3 e7f3382e/rbd_data.109f6b8b4567.0000000000003878/head//3(173'2036 client.4296.0:109711 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2036)
2014-06-12 14:51:26.665519 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  b383382e/rbd_data.10ae6b8b4567.000000000000052d/head//3 b383382e/rbd_data.10ae6b8b4567.000000000000052d/head//3(173'2539 client.4311.0:122899 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2539)
2014-06-12 14:51:26.665531 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  cc9f282e/rbd_data.109d6b8b4567.00000000000024bc/head//3 cc9f282e/rbd_data.109d6b8b4567.00000000000024bc/head//3(173'1455 client.4294.0:75088 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1455)
2014-06-12 14:51:26.665547 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  777f282e/rbd_data.10956b8b4567.00000000000014f5/head//3 777f282e/rbd_data.10956b8b4567.00000000000014f5/head//3(173'947 client.4286.0:47075 wrlock_by=unknown.0.0:0 dirty s 4194304 uv947)
2014-06-12 14:51:26.665559 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  2fbb282e/rbd_data.10a36b8b4567.00000000000001d8/head//3 2fbb282e/rbd_data.10a36b8b4567.00000000000001d8/head//3(173'2400 client.4300.0:117502 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2400)
2014-06-12 14:51:26.665572 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  9378282e/rbd_data.10b06b8b4567.0000000000003a8a/head//3 9378282e/rbd_data.10b06b8b4567.0000000000003a8a/head//3(173'2283 client.4313.0:117409 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2283)
2014-06-12 14:51:26.665584 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  5618282e/rbd_data.109b6b8b4567.0000000000001c3b/head//3 5618282e/rbd_data.109b6b8b4567.0000000000001c3b/head//3(173'1285 client.4292.0:58649 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1285)
2014-06-12 14:51:26.665597 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  7e85282e/rbd_data.10926b8b4567.00000000000015b2/head//3 7e85282e/rbd_data.10926b8b4567.00000000000015b2/head//3(173'939 client.4283.0:47825 wrlock_by=unknown.0.0:0 dirty s 4194304 uv939)
2014-06-12 14:51:26.665609 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  2f24282e/rbd_data.10b36b8b4567.00000000000016fc/head//3 2f24282e/rbd_data.10b36b8b4567.00000000000016fc/head//3(173'990 client.4316.0:52114 wrlock_by=unknown.0.0:0 dirty s 4194304 uv990)
2014-06-12 14:51:26.665635 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  fd02282e/rbd_data.10a76b8b4567.0000000000003a8c/head//3 fd02282e/rbd_data.10a76b8b4567.0000000000003a8c/head//3(173'2230 client.4304.0:115008 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2230)
2014-06-12 14:51:26.665649 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  177f182e/rbd_data.10ac6b8b4567.00000000000025ad/head//3 177f182e/rbd_data.10ac6b8b4567.00000000000025ad/head//3(173'1552 client.4309.0:75665 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1552)
2014-06-12 14:51:26.665662 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a58e182e/rbd_data.108e6b8b4567.0000000000002612/head//3 a58e182e/rbd_data.108e6b8b4567.0000000000002612/head//3(173'1543 client.4279.0:78318 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1543)
2014-06-12 14:51:26.665674 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  7a3e182e/rbd_data.109c6b8b4567.0000000000002ee6/head//3 7a3e182e/rbd_data.109c6b8b4567.0000000000002ee6/head//3(173'1883 client.4293.0:94522 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1883)
2014-06-12 14:51:26.665684 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:26.665695 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:26.665704 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:26.665706 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:26.665708 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:26.665739 7f9d60bd2700 10 collection_list_partial 3.2e_head start:aef7482e//0//-1 is_max:0
2014-06-12 14:51:26.665956 7f9d60bd2700 20 objects: [8a19482e/rbd_data.10ab6b8b4567.000000000000025e/head//3,c7f9482e/rbd_data.108e6b8b4567.0000000000003aa3/head//3,19ea482e/rbd_data.10aa6b8b4567.00000000000006c0/head//3,9ad482e/rbd_data.10956b8b4567.000000000000205f/head//3,e03f482e/rbd_data.10af6b8b4567.00000000000013dd/head//3,6870582e/rbd_data.10ae6b8b4567.0000000000001dcd/head//3,1ac2582e/rbd_data.109c6b8b4567.00000000000037f4/head//3,44f3582e/rbd_data.109c6b8b4567.000000000000156d/head//3,b474582e/rbd_data.10ae6b8b4567.00000000000001c7/head//3,54a4582e/rbd_data.108e6b8b4567.00000000000003db/head//3,8e08582e/rbd_data.10976b8b4567.00000000000024bd/head//3,8278582e/rbd_data.10ae6b8b4567.0000000000000cf8/head//3,41c8582e/rbd_data.10ab6b8b4567.00000000000002ce/head//3,4e30682e/rbd_data.10b26b8b4567.0000000000002fc6/head//3,b331682e/rbd_data.10966b8b4567.0000000000003768/head//3,d7f1682e/rbd_data.10b26b8b4567.0000000000002e68/head//3,a632682e/rbd_data.109f6b8b4567.0000000000002d7e/head//3,a472682e/rbd_data.108d6b8b4567.000000000000336c/head//3,5184682e/rbd_data.10aa6b8b4567.000000000000252e/head//3,efc5682e/rbd_data.10976b8b4567.0000000000000a87/head//3,9f66682e/rbd_data.109d6b8b4567.00000000000036b8/head//3,9e07682e/rbd_data.10b36b8b4567.00000000000009e0/head//3,aea7682e/rbd_data.10a16b8b4567.00000000000035b6/head//3,8b09682e/rbd_data.10b46b8b4567.00000000000005f4/head//3,bca682e/rbd_data.10936b8b4567.000000000000144b/head//3]
2014-06-12 14:51:26.666427 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:26.666451 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:26.666466 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:26.666475 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:26.666481 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:26.666486 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:26.666492 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_map_chunk [aef7482e//0//-1,54bb682e//0//-1)
2014-06-12 14:51:26.666499 7f9d60bd2700 10 collection_list_partial 3.2e_head start:aef7482e//0//-1 is_max:0
2014-06-12 14:51:26.666969 7f9d60bd2700 20 objects: [8a19482e/rbd_data.10ab6b8b4567.000000000000025e/head//3,c7f9482e/rbd_data.108e6b8b4567.0000000000003aa3/head//3,19ea482e/rbd_data.10aa6b8b4567.00000000000006c0/head//3,9ad482e/rbd_data.10956b8b4567.000000000000205f/head//3,e03f482e/rbd_data.10af6b8b4567.00000000000013dd/head//3,6870582e/rbd_data.10ae6b8b4567.0000000000001dcd/head//3,1ac2582e/rbd_data.109c6b8b4567.00000000000037f4/head//3,44f3582e/rbd_data.109c6b8b4567.000000000000156d/head//3,b474582e/rbd_data.10ae6b8b4567.00000000000001c7/head//3,54a4582e/rbd_data.108e6b8b4567.00000000000003db/head//3,8e08582e/rbd_data.10976b8b4567.00000000000024bd/head//3,8278582e/rbd_data.10ae6b8b4567.0000000000000cf8/head//3,41c8582e/rbd_data.10ab6b8b4567.00000000000002ce/head//3,4e30682e/rbd_data.10b26b8b4567.0000000000002fc6/head//3,b331682e/rbd_data.10966b8b4567.0000000000003768/head//3,d7f1682e/rbd_data.10b26b8b4567.0000000000002e68/head//3,a632682e/rbd_data.109f6b8b4567.0000000000002d7e/head//3,a472682e/rbd_data.108d6b8b4567.000000000000336c/head//3,5184682e/rbd_data.10aa6b8b4567.000000000000252e/head//3,efc5682e/rbd_data.10976b8b4567.0000000000000a87/head//3,9f66682e/rbd_data.109d6b8b4567.00000000000036b8/head//3,9e07682e/rbd_data.10b36b8b4567.00000000000009e0/head//3,aea7682e/rbd_data.10a16b8b4567.00000000000035b6/head//3,8b09682e/rbd_data.10b46b8b4567.00000000000005f4/head//3,bca682e/rbd_data.10936b8b4567.000000000000144b/head//3,54bb682e/rbd_data.10a76b8b4567.0000000000000b33/head//3,d1ce682e/rbd_data.10ab6b8b4567.00000000000031e2/head//3,3e10782e/rbd_data.10ac6b8b4567.0000000000003557/head//3,d30782e/rbd_data.10a96b8b4567.00000000000010ef/head//3,e960782e/rbd_data.10a46b8b4567.0000000000001091/head//3,fb61782e/rbd_data.109c6b8b4567.00000000000035b5/head//3,5943782e/rbd_data.109e6b8b4567.0000000000000e31/head//3,c1c4782e/rbd_data.10956b8b4567.00000000000016e6/head//3,2ff4782e/rbd_data.10a06b8b4567.0000000000000801/head//3,73c5782e/rbd_data.10a76b8b4567.0000000000003810/head//3,2e86782e/rbd_data.10a96b8b4567.00000000000000e8/head//3,f2a8782e/rbd_data.10946b8b4567.00000000000034f7/head//3,6159782e/rbd_data.10966b8b4567.0000000000003b0b/head//3,4e6a782e/rbd_data.10ae6b8b4567.0000000000001cb9/head//3,448c782e/rbd_data.109a6b8b4567.00000000000027d0/head//3,6b8f782e/rbd_data.10ab6b8b4567.0000000000001dd4/head//3,49d0882e/rbd_data.10b46b8b4567.00000000000038a3/head//3,d8a3882e/rbd_data.10a36b8b4567.0000000000000eb1/head//3,3a24882e/rbd_data.10ae6b8b4567.0000000000003409/head//3,5c35882e/rbd_data.10a96b8b4567.0000000000003a66/head//3,d686882e/rbd_data.108f6b8b4567.0000000000001980/head//3,c688882e/rbd_data.10946b8b4567.0000000000001a48/head//3,d7d8882e/rbd_data.10976b8b4567.00000000000036f9/head//3,5e89882e/rbd_data.10a56b8b4567.000000000000021c/head//3,a636982e/rbd_data.10ad6b8b4567.0000000000002567/head//3,6a07982e/rbd_data.10af6b8b4567.0000000000001b0f/head//3,31a7982e/rbd_data.10a26b8b4567.0000000000001986/head//3,4628982e/rbd_data.10966b8b4567.00000000000015af/head//3,a69a982e/rbd_data.10996b8b4567.00000000000032e8/head//3,70ba982e/rbd_data.108d6b8b4567.000000000000171e/head//3,9fcc982e/rbd_data.10916b8b4567.0000000000002b8f/head//3,3c9d982e/rbd_data.10b06b8b4567.00000000000035b4/head//3,5494a82e/rbd_data.10ae6b8b4567.0000000000000458/head//3,975a82e/rbd_data.109e6b8b4567.0000000000003135/head//3,d676a82e/rbd_data.10a46b8b4567.00000000000000c0/head//3,33f6a82e/rbd_data.109b6b8b4567.0000000000001e44/head//3,a7b7a82e/rbd_data.10a66b8b4567.0000000000001a05/head//3,9ed8a82e/rbd_data.10a46b8b4567.0000000000001f73/head//3,2a09a82e/rbd_data.10a46b8b4567.0000000000002a75/head//3]
2014-06-12 14:51:26.667039 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:26.667055 7f9d60bd2700 10 stat 3.2e_head/8a19482e/rbd_data.10ab6b8b4567.000000000000025e/head//3
2014-06-12 14:51:26.667171 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8a19482e/rbd_data.10ab6b8b4567.000000000000025e/head//3
2014-06-12 14:51:26.674806 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:26.674887 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:26.700132 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:26.700791 7f9d60bd2700 10 getattrs 3.2e_head/8a19482e/rbd_data.10ab6b8b4567.000000000000025e/head//3 = 0
2014-06-12 14:51:26.700806 7f9d60bd2700 10 stat 3.2e_head/c7f9482e/rbd_data.108e6b8b4567.0000000000003aa3/head//3
2014-06-12 14:51:26.700907 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid c7f9482e/rbd_data.108e6b8b4567.0000000000003aa3/head//3
2014-06-12 14:51:26.723271 7f9d60bd2700 10 getattrs 3.2e_head/c7f9482e/rbd_data.108e6b8b4567.0000000000003aa3/head//3 = 0
2014-06-12 14:51:26.723282 7f9d60bd2700 10 stat 3.2e_head/19ea482e/rbd_data.10aa6b8b4567.00000000000006c0/head//3
2014-06-12 14:51:26.723387 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 19ea482e/rbd_data.10aa6b8b4567.00000000000006c0/head//3
2014-06-12 14:51:26.763991 7f9d60bd2700 10 getattrs 3.2e_head/19ea482e/rbd_data.10aa6b8b4567.00000000000006c0/head//3 = 0
2014-06-12 14:51:26.764001 7f9d60bd2700 10 stat 3.2e_head/9ad482e/rbd_data.10956b8b4567.000000000000205f/head//3
2014-06-12 14:51:26.764116 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 9ad482e/rbd_data.10956b8b4567.000000000000205f/head//3
2014-06-12 14:51:26.767480 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:26.783830 7f9d60bd2700 10 getattrs 3.2e_head/9ad482e/rbd_data.10956b8b4567.000000000000205f/head//3 = 0
2014-06-12 14:51:26.783852 7f9d60bd2700 10 stat 3.2e_head/e03f482e/rbd_data.10af6b8b4567.00000000000013dd/head//3
2014-06-12 14:51:26.784032 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid e03f482e/rbd_data.10af6b8b4567.00000000000013dd/head//3
2014-06-12 14:51:26.793263 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:26.793607 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:26.799869 7f9d60bd2700 10 getattrs 3.2e_head/e03f482e/rbd_data.10af6b8b4567.00000000000013dd/head//3 = 0
2014-06-12 14:51:26.799878 7f9d60bd2700 10 stat 3.2e_head/6870582e/rbd_data.10ae6b8b4567.0000000000001dcd/head//3
2014-06-12 14:51:26.799972 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6870582e/rbd_data.10ae6b8b4567.0000000000001dcd/head//3
2014-06-12 14:51:26.801432 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:26.807886 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:26.815681 7f9d60bd2700 10 getattrs 3.2e_head/6870582e/rbd_data.10ae6b8b4567.0000000000001dcd/head//3 = 0
2014-06-12 14:51:26.815691 7f9d60bd2700 10 stat 3.2e_head/1ac2582e/rbd_data.109c6b8b4567.00000000000037f4/head//3
2014-06-12 14:51:26.815784 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 1ac2582e/rbd_data.109c6b8b4567.00000000000037f4/head//3
2014-06-12 14:51:26.827170 7f9d60bd2700 10 getattrs 3.2e_head/1ac2582e/rbd_data.109c6b8b4567.00000000000037f4/head//3 = 0
2014-06-12 14:51:26.827178 7f9d60bd2700 10 stat 3.2e_head/44f3582e/rbd_data.109c6b8b4567.000000000000156d/head//3
2014-06-12 14:51:26.827265 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 44f3582e/rbd_data.109c6b8b4567.000000000000156d/head//3
2014-06-12 14:51:26.854360 7f9d60bd2700 10 getattrs 3.2e_head/44f3582e/rbd_data.109c6b8b4567.000000000000156d/head//3 = 0
2014-06-12 14:51:26.854368 7f9d60bd2700 10 stat 3.2e_head/b474582e/rbd_data.10ae6b8b4567.00000000000001c7/head//3
2014-06-12 14:51:26.854460 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid b474582e/rbd_data.10ae6b8b4567.00000000000001c7/head//3
2014-06-12 14:51:26.854526 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:26.864888 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:26.868279 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:26.911588 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:26.911889 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:26.925259 7f9d60bd2700 10 getattrs 3.2e_head/b474582e/rbd_data.10ae6b8b4567.00000000000001c7/head//3 = 0
2014-06-12 14:51:26.925270 7f9d60bd2700 10 stat 3.2e_head/54a4582e/rbd_data.108e6b8b4567.00000000000003db/head//3
2014-06-12 14:51:26.925392 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 54a4582e/rbd_data.108e6b8b4567.00000000000003db/head//3
2014-06-12 14:51:26.927987 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:26.935452 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:26.959919 7f9d60bd2700 10 getattrs 3.2e_head/54a4582e/rbd_data.108e6b8b4567.00000000000003db/head//3 = 0
2014-06-12 14:51:26.959928 7f9d60bd2700 10 stat 3.2e_head/8e08582e/rbd_data.10976b8b4567.00000000000024bd/head//3
2014-06-12 14:51:26.960022 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8e08582e/rbd_data.10976b8b4567.00000000000024bd/head//3
2014-06-12 14:51:26.961832 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:26.965551 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:26.976387 7f9d60bd2700 10 getattrs 3.2e_head/8e08582e/rbd_data.10976b8b4567.00000000000024bd/head//3 = 0
2014-06-12 14:51:26.976395 7f9d60bd2700 10 stat 3.2e_head/8278582e/rbd_data.10ae6b8b4567.0000000000000cf8/head//3
2014-06-12 14:51:26.976483 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8278582e/rbd_data.10ae6b8b4567.0000000000000cf8/head//3
2014-06-12 14:51:27.018353 7f9d60bd2700 10 getattrs 3.2e_head/8278582e/rbd_data.10ae6b8b4567.0000000000000cf8/head//3 = 0
2014-06-12 14:51:27.018384 7f9d60bd2700 10 stat 3.2e_head/41c8582e/rbd_data.10ab6b8b4567.00000000000002ce/head//3
2014-06-12 14:51:27.018575 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 41c8582e/rbd_data.10ab6b8b4567.00000000000002ce/head//3
2014-06-12 14:51:27.062027 7f9d60bd2700 10 getattrs 3.2e_head/41c8582e/rbd_data.10ab6b8b4567.00000000000002ce/head//3 = 0
2014-06-12 14:51:27.062039 7f9d60bd2700 10 stat 3.2e_head/4e30682e/rbd_data.10b26b8b4567.0000000000002fc6/head//3
2014-06-12 14:51:27.062139 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4e30682e/rbd_data.10b26b8b4567.0000000000002fc6/head//3
2014-06-12 14:51:27.079111 7f9d60bd2700 10 getattrs 3.2e_head/4e30682e/rbd_data.10b26b8b4567.0000000000002fc6/head//3 = 0
2014-06-12 14:51:27.079120 7f9d60bd2700 10 stat 3.2e_head/b331682e/rbd_data.10966b8b4567.0000000000003768/head//3
2014-06-12 14:51:27.079219 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid b331682e/rbd_data.10966b8b4567.0000000000003768/head//3
2014-06-12 14:51:27.101148 7f9d60bd2700 10 getattrs 3.2e_head/b331682e/rbd_data.10966b8b4567.0000000000003768/head//3 = 0
2014-06-12 14:51:27.101163 7f9d60bd2700 10 stat 3.2e_head/d7f1682e/rbd_data.10b26b8b4567.0000000000002e68/head//3
2014-06-12 14:51:27.101264 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d7f1682e/rbd_data.10b26b8b4567.0000000000002e68/head//3
2014-06-12 14:51:27.115428 7f9d60bd2700 10 getattrs 3.2e_head/d7f1682e/rbd_data.10b26b8b4567.0000000000002e68/head//3 = 0
2014-06-12 14:51:27.115443 7f9d60bd2700 10 stat 3.2e_head/a632682e/rbd_data.109f6b8b4567.0000000000002d7e/head//3
2014-06-12 14:51:27.115553 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a632682e/rbd_data.109f6b8b4567.0000000000002d7e/head//3
2014-06-12 14:51:27.120792 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:27.132454 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:27.141920 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:27.142130 7f9d60bd2700 10 getattrs 3.2e_head/a632682e/rbd_data.109f6b8b4567.0000000000002d7e/head//3 = 0
2014-06-12 14:51:27.142144 7f9d60bd2700 10 stat 3.2e_head/a472682e/rbd_data.108d6b8b4567.000000000000336c/head//3
2014-06-12 14:51:27.142302 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a472682e/rbd_data.108d6b8b4567.000000000000336c/head//3
2014-06-12 14:51:27.151323 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:27.168138 7f9d60bd2700 10 getattrs 3.2e_head/a472682e/rbd_data.108d6b8b4567.000000000000336c/head//3 = 0
2014-06-12 14:51:27.168146 7f9d60bd2700 10 stat 3.2e_head/5184682e/rbd_data.10aa6b8b4567.000000000000252e/head//3
2014-06-12 14:51:27.168241 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 5184682e/rbd_data.10aa6b8b4567.000000000000252e/head//3
2014-06-12 14:51:27.179784 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:27.203875 7f9d60bd2700 10 getattrs 3.2e_head/5184682e/rbd_data.10aa6b8b4567.000000000000252e/head//3 = 0
2014-06-12 14:51:27.203889 7f9d60bd2700 10 stat 3.2e_head/efc5682e/rbd_data.10976b8b4567.0000000000000a87/head//3
2014-06-12 14:51:27.204044 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid efc5682e/rbd_data.10976b8b4567.0000000000000a87/head//3
2014-06-12 14:51:27.245202 7f9d60bd2700 10 getattrs 3.2e_head/efc5682e/rbd_data.10976b8b4567.0000000000000a87/head//3 = 0
2014-06-12 14:51:27.245219 7f9d60bd2700 10 stat 3.2e_head/9f66682e/rbd_data.109d6b8b4567.00000000000036b8/head//3
2014-06-12 14:51:27.245373 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 9f66682e/rbd_data.109d6b8b4567.00000000000036b8/head//3
2014-06-12 14:51:27.270986 7f9d60bd2700 10 getattrs 3.2e_head/9f66682e/rbd_data.109d6b8b4567.00000000000036b8/head//3 = 0
2014-06-12 14:51:27.270994 7f9d60bd2700 10 stat 3.2e_head/9e07682e/rbd_data.10b36b8b4567.00000000000009e0/head//3
2014-06-12 14:51:27.271089 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 9e07682e/rbd_data.10b36b8b4567.00000000000009e0/head//3
2014-06-12 14:51:27.282494 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:27.295938 7f9d60bd2700 10 getattrs 3.2e_head/9e07682e/rbd_data.10b36b8b4567.00000000000009e0/head//3 = 0
2014-06-12 14:51:27.295947 7f9d60bd2700 10 stat 3.2e_head/aea7682e/rbd_data.10a16b8b4567.00000000000035b6/head//3
2014-06-12 14:51:27.296038 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid aea7682e/rbd_data.10a16b8b4567.00000000000035b6/head//3
2014-06-12 14:51:27.297958 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:27.316994 7f9d44101700  5 -- op tracker -- , seq: 748576, time: 2014-06-12 14:51:27.316913, event: header_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.317005 7f9d44101700  5 -- op tracker -- , seq: 748576, time: 2014-06-12 14:51:27.316915, event: throttled, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.317011 7f9d44101700  5 -- op tracker -- , seq: 748576, time: 2014-06-12 14:51:27.316960, event: all_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.317017 7f9d44101700  5 -- op tracker -- , seq: 748576, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.317021 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:27.317026 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:27.317045 7f9d44101700 15 osd.2 173 enqueue_op 0x124e3a00 prio 127 cost 10717 latency 0.000120 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:27.317096 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:27.321253 7f9d60bd2700 10 getattrs 3.2e_head/aea7682e/rbd_data.10a16b8b4567.00000000000035b6/head//3 = 0
2014-06-12 14:51:27.321262 7f9d60bd2700 10 stat 3.2e_head/8b09682e/rbd_data.10b46b8b4567.00000000000005f4/head//3
2014-06-12 14:51:27.321362 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8b09682e/rbd_data.10b46b8b4567.00000000000005f4/head//3
2014-06-12 14:51:27.341153 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:27.343592 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:27.345331 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:27.345355 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:27.345359 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:27.345361 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:27.345364 7f9d84175700 10 -- op tracker -- ops_in_flight.size: 1; oldest is 0.028450 seconds old
2014-06-12 14:51:27.345401 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:27.367052 7f9d60bd2700 10 getattrs 3.2e_head/8b09682e/rbd_data.10b46b8b4567.00000000000005f4/head//3 = 0
2014-06-12 14:51:27.367071 7f9d60bd2700 10 stat 3.2e_head/bca682e/rbd_data.10936b8b4567.000000000000144b/head//3
2014-06-12 14:51:27.367197 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid bca682e/rbd_data.10936b8b4567.000000000000144b/head//3
2014-06-12 14:51:27.389723 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:27.401707 7f9d60bd2700 10 getattrs 3.2e_head/bca682e/rbd_data.10936b8b4567.000000000000144b/head//3 = 0
2014-06-12 14:51:27.401722 7f9d60bd2700 10 collection_getattrs 3.2e_head
2014-06-12 14:51:27.401806 7f9d60bd2700 10 collection_getattrs 3.2e_head = 0
2014-06-12 14:51:27.401815 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:27.401853 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:27.401867 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:27.401882 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:27.401886 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:27.401927 7f9d68be2700 10 osd.2 173 dequeue_op 0x124e3a00 prio 127 cost 10717 latency 0.085005 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]
2014-06-12 14:51:27.401980 7f9d68be2700  5 -- op tracker -- , seq: 748576, time: 2014-06-12 14:51:27.401972, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.401997 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] handle_message: 0x124e3a00
2014-06-12 14:51:27.402013 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:27.402030 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:27.402047 7f9d68be2700  5 -- op tracker -- , seq: 748576, time: 2014-06-12 14:51:27.402042, event: started, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.402055 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:27.402164 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] map version is 173'2847
2014-06-12 14:51:27.402186 7f9d68be2700 10 osd.2 173 dequeue_op 0x124e3a00 finish
2014-06-12 14:51:27.402191 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:27.402200 7f9d68be2700  5 -- op tracker -- , seq: 748576, time: 2014-06-12 14:51:27.402194, event: done, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.402197 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:27.402211 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:27.402222 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:27.402227 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:27.402221 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:27.402270 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:27.402281 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:27.402291 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:27.402322 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8a19482e/rbd_data.10ab6b8b4567.000000000000025e/head//3, auth == maps.end()
2014-06-12 14:51:27.402355 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8a19482e/rbd_data.10ab6b8b4567.000000000000025e/head//3
2014-06-12 14:51:27.402386 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c7f9482e/rbd_data.108e6b8b4567.0000000000003aa3/head//3, auth == maps.end()
2014-06-12 14:51:27.402408 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c7f9482e/rbd_data.108e6b8b4567.0000000000003aa3/head//3
2014-06-12 14:51:27.402427 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 19ea482e/rbd_data.10aa6b8b4567.00000000000006c0/head//3, auth == maps.end()
2014-06-12 14:51:27.402444 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 19ea482e/rbd_data.10aa6b8b4567.00000000000006c0/head//3
2014-06-12 14:51:27.402459 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9ad482e/rbd_data.10956b8b4567.000000000000205f/head//3, auth == maps.end()
2014-06-12 14:51:27.402475 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9ad482e/rbd_data.10956b8b4567.000000000000205f/head//3
2014-06-12 14:51:27.402490 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e03f482e/rbd_data.10af6b8b4567.00000000000013dd/head//3, auth == maps.end()
2014-06-12 14:51:27.402506 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e03f482e/rbd_data.10af6b8b4567.00000000000013dd/head//3
2014-06-12 14:51:27.402521 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6870582e/rbd_data.10ae6b8b4567.0000000000001dcd/head//3, auth == maps.end()
2014-06-12 14:51:27.402539 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6870582e/rbd_data.10ae6b8b4567.0000000000001dcd/head//3
2014-06-12 14:51:27.402553 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1ac2582e/rbd_data.109c6b8b4567.00000000000037f4/head//3, auth == maps.end()
2014-06-12 14:51:27.402567 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1ac2582e/rbd_data.109c6b8b4567.00000000000037f4/head//3
2014-06-12 14:51:27.402581 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 44f3582e/rbd_data.109c6b8b4567.000000000000156d/head//3, auth == maps.end()
2014-06-12 14:51:27.402594 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 44f3582e/rbd_data.109c6b8b4567.000000000000156d/head//3
2014-06-12 14:51:27.402608 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b474582e/rbd_data.10ae6b8b4567.00000000000001c7/head//3, auth == maps.end()
2014-06-12 14:51:27.402624 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b474582e/rbd_data.10ae6b8b4567.00000000000001c7/head//3
2014-06-12 14:51:27.402641 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 54a4582e/rbd_data.108e6b8b4567.00000000000003db/head//3, auth == maps.end()
2014-06-12 14:51:27.402656 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 54a4582e/rbd_data.108e6b8b4567.00000000000003db/head//3
2014-06-12 14:51:27.402672 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8e08582e/rbd_data.10976b8b4567.00000000000024bd/head//3, auth == maps.end()
2014-06-12 14:51:27.402686 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8e08582e/rbd_data.10976b8b4567.00000000000024bd/head//3
2014-06-12 14:51:27.402703 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8278582e/rbd_data.10ae6b8b4567.0000000000000cf8/head//3, auth == maps.end()
2014-06-12 14:51:27.402717 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8278582e/rbd_data.10ae6b8b4567.0000000000000cf8/head//3
2014-06-12 14:51:27.402732 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 41c8582e/rbd_data.10ab6b8b4567.00000000000002ce/head//3, auth == maps.end()
2014-06-12 14:51:27.402745 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 41c8582e/rbd_data.10ab6b8b4567.00000000000002ce/head//3
2014-06-12 14:51:27.402760 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4e30682e/rbd_data.10b26b8b4567.0000000000002fc6/head//3, auth == maps.end()
2014-06-12 14:51:27.402773 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4e30682e/rbd_data.10b26b8b4567.0000000000002fc6/head//3
2014-06-12 14:51:27.402787 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b331682e/rbd_data.10966b8b4567.0000000000003768/head//3, auth == maps.end()
2014-06-12 14:51:27.402801 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b331682e/rbd_data.10966b8b4567.0000000000003768/head//3
2014-06-12 14:51:27.402815 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d7f1682e/rbd_data.10b26b8b4567.0000000000002e68/head//3, auth == maps.end()
2014-06-12 14:51:27.402831 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d7f1682e/rbd_data.10b26b8b4567.0000000000002e68/head//3
2014-06-12 14:51:27.402846 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a632682e/rbd_data.109f6b8b4567.0000000000002d7e/head//3, auth == maps.end()
2014-06-12 14:51:27.402865 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a632682e/rbd_data.109f6b8b4567.0000000000002d7e/head//3
2014-06-12 14:51:27.402881 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a472682e/rbd_data.108d6b8b4567.000000000000336c/head//3, auth == maps.end()
2014-06-12 14:51:27.402896 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a472682e/rbd_data.108d6b8b4567.000000000000336c/head//3
2014-06-12 14:51:27.402914 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5184682e/rbd_data.10aa6b8b4567.000000000000252e/head//3, auth == maps.end()
2014-06-12 14:51:27.402930 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5184682e/rbd_data.10aa6b8b4567.000000000000252e/head//3
2014-06-12 14:51:27.402948 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj efc5682e/rbd_data.10976b8b4567.0000000000000a87/head//3, auth == maps.end()
2014-06-12 14:51:27.402964 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj efc5682e/rbd_data.10976b8b4567.0000000000000a87/head//3
2014-06-12 14:51:27.402981 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9f66682e/rbd_data.109d6b8b4567.00000000000036b8/head//3, auth == maps.end()
2014-06-12 14:51:27.402997 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9f66682e/rbd_data.109d6b8b4567.00000000000036b8/head//3
2014-06-12 14:51:27.403014 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9e07682e/rbd_data.10b36b8b4567.00000000000009e0/head//3, auth == maps.end()
2014-06-12 14:51:27.403030 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9e07682e/rbd_data.10b36b8b4567.00000000000009e0/head//3
2014-06-12 14:51:27.403048 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aea7682e/rbd_data.10a16b8b4567.00000000000035b6/head//3, auth == maps.end()
2014-06-12 14:51:27.403064 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj aea7682e/rbd_data.10a16b8b4567.00000000000035b6/head//3
2014-06-12 14:51:27.403079 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8b09682e/rbd_data.10b46b8b4567.00000000000005f4/head//3, auth == maps.end()
2014-06-12 14:51:27.403095 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8b09682e/rbd_data.10b46b8b4567.00000000000005f4/head//3
2014-06-12 14:51:27.403113 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bca682e/rbd_data.10936b8b4567.000000000000144b/head//3, auth == maps.end()
2014-06-12 14:51:27.403129 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj bca682e/rbd_data.10936b8b4567.000000000000144b/head//3
2014-06-12 14:51:27.403147 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] 
2014-06-12 14:51:27.403160 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub
2014-06-12 14:51:27.403180 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  bca682e/rbd_data.10936b8b4567.000000000000144b/head//3 bca682e/rbd_data.10936b8b4567.000000000000144b/head//3(173'959 client.4284.0:44191 wrlock_by=unknown.0.0:0 dirty s 4194304 uv959)
2014-06-12 14:51:27.403202 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8b09682e/rbd_data.10b46b8b4567.00000000000005f4/head//3 8b09682e/rbd_data.10b46b8b4567.00000000000005f4/head//3(173'2556 client.4317.0:126299 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2556)
2014-06-12 14:51:27.403224 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  aea7682e/rbd_data.10a16b8b4567.00000000000035b6/head//3 aea7682e/rbd_data.10a16b8b4567.00000000000035b6/head//3(173'2100 client.4298.0:107747 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2100)
2014-06-12 14:51:27.403244 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  9e07682e/rbd_data.10b36b8b4567.00000000000009e0/head//3 9e07682e/rbd_data.10b36b8b4567.00000000000009e0/head//3(173'2641 client.4316.0:134952 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2641)
2014-06-12 14:51:27.403265 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  9f66682e/rbd_data.109d6b8b4567.00000000000036b8/head//3 9f66682e/rbd_data.109d6b8b4567.00000000000036b8/head//3(173'2082 client.4294.0:107842 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2082)
2014-06-12 14:51:27.403285 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  efc5682e/rbd_data.10976b8b4567.0000000000000a87/head//3 efc5682e/rbd_data.10976b8b4567.0000000000000a87/head//3(173'2689 client.4288.0:131808 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2689)
2014-06-12 14:51:27.403305 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  5184682e/rbd_data.10aa6b8b4567.000000000000252e/head//3 5184682e/rbd_data.10aa6b8b4567.000000000000252e/head//3(173'1463 client.4307.0:78329 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1463)
2014-06-12 14:51:27.403326 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a472682e/rbd_data.108d6b8b4567.000000000000336c/head//3 a472682e/rbd_data.108d6b8b4567.000000000000336c/head//3(173'1954 client.4278.0:103274 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1954)
2014-06-12 14:51:27.403346 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a632682e/rbd_data.109f6b8b4567.0000000000002d7e/head//3 a632682e/rbd_data.109f6b8b4567.0000000000002d7e/head//3(173'1666 client.4296.0:89364 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1666)
2014-06-12 14:51:27.403366 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d7f1682e/rbd_data.10b26b8b4567.0000000000002e68/head//3 d7f1682e/rbd_data.10b26b8b4567.0000000000002e68/head//3(173'1721 client.4315.0:93367 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1721)
2014-06-12 14:51:27.403386 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  b331682e/rbd_data.10966b8b4567.0000000000003768/head//3 b331682e/rbd_data.10966b8b4567.0000000000003768/head//3(173'2048 client.4287.0:106593 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2048)
2014-06-12 14:51:27.403406 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4e30682e/rbd_data.10b26b8b4567.0000000000002fc6/head//3 4e30682e/rbd_data.10b26b8b4567.0000000000002fc6/head//3(173'1803 client.4315.0:95906 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1803)
2014-06-12 14:51:27.403426 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  41c8582e/rbd_data.10ab6b8b4567.00000000000002ce/head//3 41c8582e/rbd_data.10ab6b8b4567.00000000000002ce/head//3(173'2443 client.4308.0:123044 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2443)
2014-06-12 14:51:27.403446 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8278582e/rbd_data.10ae6b8b4567.0000000000000cf8/head//3 8278582e/rbd_data.10ae6b8b4567.0000000000000cf8/head//3(173'2762 client.4311.0:136516 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2762)
2014-06-12 14:51:27.403466 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8e08582e/rbd_data.10976b8b4567.00000000000024bd/head//3 8e08582e/rbd_data.10976b8b4567.00000000000024bd/head//3(173'1626 client.4288.0:73896 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1626)
2014-06-12 14:51:27.403490 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  54a4582e/rbd_data.108e6b8b4567.00000000000003db/head//3 54a4582e/rbd_data.108e6b8b4567.00000000000003db/head//3(173'2482 client.4279.0:122664 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2482)
2014-06-12 14:51:27.403510 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  b474582e/rbd_data.10ae6b8b4567.00000000000001c7/head//3 b474582e/rbd_data.10ae6b8b4567.00000000000001c7/head//3(173'2421 client.4311.0:117359 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2421)
2014-06-12 14:51:27.403529 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  44f3582e/rbd_data.109c6b8b4567.000000000000156d/head//3 44f3582e/rbd_data.109c6b8b4567.000000000000156d/head//3(173'972 client.4293.0:47682 wrlock_by=unknown.0.0:0 dirty s 4194304 uv972)
2014-06-12 14:51:27.403549 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  1ac2582e/rbd_data.109c6b8b4567.00000000000037f4/head//3 1ac2582e/rbd_data.109c6b8b4567.00000000000037f4/head//3(173'2224 client.4293.0:110811 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2224)
2014-06-12 14:51:27.403569 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6870582e/rbd_data.10ae6b8b4567.0000000000001dcd/head//3 6870582e/rbd_data.10ae6b8b4567.0000000000001dcd/head//3(173'1206 client.4311.0:62092 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1206)
2014-06-12 14:51:27.403589 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  e03f482e/rbd_data.10af6b8b4567.00000000000013dd/head//3 e03f482e/rbd_data.10af6b8b4567.00000000000013dd/head//3(173'925 client.4312.0:47581 wrlock_by=unknown.0.0:0 dirty s 4194304 uv925)
2014-06-12 14:51:27.403609 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  9ad482e/rbd_data.10956b8b4567.000000000000205f/head//3 9ad482e/rbd_data.10956b8b4567.000000000000205f/head//3(173'1317 client.4286.0:67473 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1317)
2014-06-12 14:51:27.403629 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  19ea482e/rbd_data.10aa6b8b4567.00000000000006c0/head//3 19ea482e/rbd_data.10aa6b8b4567.00000000000006c0/head//3(173'2581 client.4307.0:130453 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2581)
2014-06-12 14:51:27.403649 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  c7f9482e/rbd_data.108e6b8b4567.0000000000003aa3/head//3 c7f9482e/rbd_data.108e6b8b4567.0000000000003aa3/head//3(173'2249 client.4279.0:115177 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2249)
2014-06-12 14:51:27.403669 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8a19482e/rbd_data.10ab6b8b4567.000000000000025e/head//3 8a19482e/rbd_data.10ab6b8b4567.000000000000025e/head//3(173'2432 client.4308.0:122357 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2432)
2014-06-12 14:51:27.403686 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:27.403702 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:27.403715 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:27.403719 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:27.403721 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:27.403772 7f9d60bd2700 10 collection_list_partial 3.2e_head start:54bb682e//0//-1 is_max:0
2014-06-12 14:51:27.404082 7f9d60bd2700 20 objects: [d1ce682e/rbd_data.10ab6b8b4567.00000000000031e2/head//3,3e10782e/rbd_data.10ac6b8b4567.0000000000003557/head//3,d30782e/rbd_data.10a96b8b4567.00000000000010ef/head//3,e960782e/rbd_data.10a46b8b4567.0000000000001091/head//3,fb61782e/rbd_data.109c6b8b4567.00000000000035b5/head//3,5943782e/rbd_data.109e6b8b4567.0000000000000e31/head//3,c1c4782e/rbd_data.10956b8b4567.00000000000016e6/head//3,2ff4782e/rbd_data.10a06b8b4567.0000000000000801/head//3,73c5782e/rbd_data.10a76b8b4567.0000000000003810/head//3,2e86782e/rbd_data.10a96b8b4567.00000000000000e8/head//3,f2a8782e/rbd_data.10946b8b4567.00000000000034f7/head//3,6159782e/rbd_data.10966b8b4567.0000000000003b0b/head//3,4e6a782e/rbd_data.10ae6b8b4567.0000000000001cb9/head//3,448c782e/rbd_data.109a6b8b4567.00000000000027d0/head//3,6b8f782e/rbd_data.10ab6b8b4567.0000000000001dd4/head//3,49d0882e/rbd_data.10b46b8b4567.00000000000038a3/head//3,d8a3882e/rbd_data.10a36b8b4567.0000000000000eb1/head//3,3a24882e/rbd_data.10ae6b8b4567.0000000000003409/head//3,5c35882e/rbd_data.10a96b8b4567.0000000000003a66/head//3,d686882e/rbd_data.108f6b8b4567.0000000000001980/head//3,c688882e/rbd_data.10946b8b4567.0000000000001a48/head//3,d7d8882e/rbd_data.10976b8b4567.00000000000036f9/head//3,5e89882e/rbd_data.10a56b8b4567.000000000000021c/head//3,a636982e/rbd_data.10ad6b8b4567.0000000000002567/head//3,6a07982e/rbd_data.10af6b8b4567.0000000000001b0f/head//3]
2014-06-12 14:51:27.404649 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:27.404682 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:27.404708 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:27.404721 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:27.404731 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:27.404740 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:27.404749 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_map_chunk [54bb682e//0//-1,31a7982e//0//-1)
2014-06-12 14:51:27.404760 7f9d60bd2700 10 collection_list_partial 3.2e_head start:54bb682e//0//-1 is_max:0
2014-06-12 14:51:27.405501 7f9d60bd2700 20 objects: [d1ce682e/rbd_data.10ab6b8b4567.00000000000031e2/head//3,3e10782e/rbd_data.10ac6b8b4567.0000000000003557/head//3,d30782e/rbd_data.10a96b8b4567.00000000000010ef/head//3,e960782e/rbd_data.10a46b8b4567.0000000000001091/head//3,fb61782e/rbd_data.109c6b8b4567.00000000000035b5/head//3,5943782e/rbd_data.109e6b8b4567.0000000000000e31/head//3,c1c4782e/rbd_data.10956b8b4567.00000000000016e6/head//3,2ff4782e/rbd_data.10a06b8b4567.0000000000000801/head//3,73c5782e/rbd_data.10a76b8b4567.0000000000003810/head//3,2e86782e/rbd_data.10a96b8b4567.00000000000000e8/head//3,f2a8782e/rbd_data.10946b8b4567.00000000000034f7/head//3,6159782e/rbd_data.10966b8b4567.0000000000003b0b/head//3,4e6a782e/rbd_data.10ae6b8b4567.0000000000001cb9/head//3,448c782e/rbd_data.109a6b8b4567.00000000000027d0/head//3,6b8f782e/rbd_data.10ab6b8b4567.0000000000001dd4/head//3,49d0882e/rbd_data.10b46b8b4567.00000000000038a3/head//3,d8a3882e/rbd_data.10a36b8b4567.0000000000000eb1/head//3,3a24882e/rbd_data.10ae6b8b4567.0000000000003409/head//3,5c35882e/rbd_data.10a96b8b4567.0000000000003a66/head//3,d686882e/rbd_data.108f6b8b4567.0000000000001980/head//3,c688882e/rbd_data.10946b8b4567.0000000000001a48/head//3,d7d8882e/rbd_data.10976b8b4567.00000000000036f9/head//3,5e89882e/rbd_data.10a56b8b4567.000000000000021c/head//3,a636982e/rbd_data.10ad6b8b4567.0000000000002567/head//3,6a07982e/rbd_data.10af6b8b4567.0000000000001b0f/head//3,31a7982e/rbd_data.10a26b8b4567.0000000000001986/head//3,4628982e/rbd_data.10966b8b4567.00000000000015af/head//3,a69a982e/rbd_data.10996b8b4567.00000000000032e8/head//3,70ba982e/rbd_data.108d6b8b4567.000000000000171e/head//3,9fcc982e/rbd_data.10916b8b4567.0000000000002b8f/head//3,3c9d982e/rbd_data.10b06b8b4567.00000000000035b4/head//3,5494a82e/rbd_data.10ae6b8b4567.0000000000000458/head//3,975a82e/rbd_data.109e6b8b4567.0000000000003135/head//3,d676a82e/rbd_data.10a46b8b4567.00000000000000c0/head//3,33f6a82e/rbd_data.109b6b8b4567.0000000000001e44/head//3,a7b7a82e/rbd_data.10a66b8b4567.0000000000001a05/head//3,9ed8a82e/rbd_data.10a46b8b4567.0000000000001f73/head//3,2a09a82e/rbd_data.10a46b8b4567.0000000000002a75/head//3,935ca82e/rbd_data.10ac6b8b4567.0000000000003be6/head//3,9f7ca82e/rbd_data.10966b8b4567.0000000000001163/head//3,4d3da82e/rbd_data.10a26b8b4567.00000000000031a1/head//3,8bada82e/rbd_data.10b16b8b4567.0000000000002fa6/head//3,e4eda82e/rbd_data.10a56b8b4567.0000000000002bbf/head//3,b170b82e/rbd_data.10ac6b8b4567.000000000000252a/head//3,4e63b82e/rbd_data.108e6b8b4567.0000000000000174/head//3,d624b82e/rbd_data.108e6b8b4567.0000000000003ad9/head//3,ecd5b82e/rbd_data.109e6b8b4567.0000000000001e6e/head//3,a528b82e/rbd_data.10916b8b4567.00000000000020ba/head//3,6048b82e/rbd_data.10af6b8b4567.0000000000002973/head//3,e999b82e/rbd_data.10906b8b4567.0000000000000e08/head//3,d91cb82e/rbd_data.109b6b8b4567.0000000000000a47/head//3,9600c82e/rbd_data.10b16b8b4567.0000000000002853/head//3,3fe0c82e/rbd_data.10916b8b4567.00000000000026b0/head//3,6c61c82e/rbd_data.10a16b8b4567.0000000000001491/head//3,c113c82e/rbd_data.108d6b8b4567.000000000000042f/head//3,86a4c82e/rbd_data.10b06b8b4567.0000000000000c85/head//3,4da5c82e/rbd_data.10a36b8b4567.0000000000000bf1/head//3,ea09c82e/rbd_data.108e6b8b4567.0000000000001715/head//3,bf6ac82e/rbd_data.109e6b8b4567.0000000000002de0/head//3,6e9bc82e/rbd_data.10916b8b4567.0000000000001c90/head//3,f26ec82e/rbd_data.10b36b8b4567.0000000000002676/head//3,ee5fc82e/rbd_data.10956b8b4567.0000000000000295/head//3,f221d82e/rbd_data.10a36b8b4567.00000000000034c8/head//3,bbb2d82e/rbd_data.10af6b8b4567.0000000000002776/head//3]
2014-06-12 14:51:27.405621 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:27.405645 7f9d60bd2700 10 stat 3.2e_head/d1ce682e/rbd_data.10ab6b8b4567.00000000000031e2/head//3
2014-06-12 14:51:27.405828 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d1ce682e/rbd_data.10ab6b8b4567.00000000000031e2/head//3
2014-06-12 14:51:27.428679 7f9d60bd2700 10 getattrs 3.2e_head/d1ce682e/rbd_data.10ab6b8b4567.00000000000031e2/head//3 = 0
2014-06-12 14:51:27.428700 7f9d60bd2700 10 stat 3.2e_head/3e10782e/rbd_data.10ac6b8b4567.0000000000003557/head//3
2014-06-12 14:51:27.428855 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 3e10782e/rbd_data.10ac6b8b4567.0000000000003557/head//3
2014-06-12 14:51:27.449151 7f9d60bd2700 10 getattrs 3.2e_head/3e10782e/rbd_data.10ac6b8b4567.0000000000003557/head//3 = 0
2014-06-12 14:51:27.449160 7f9d60bd2700 10 stat 3.2e_head/d30782e/rbd_data.10a96b8b4567.00000000000010ef/head//3
2014-06-12 14:51:27.449254 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d30782e/rbd_data.10a96b8b4567.00000000000010ef/head//3
2014-06-12 14:51:27.469862 7f9d60bd2700 10 getattrs 3.2e_head/d30782e/rbd_data.10a96b8b4567.00000000000010ef/head//3 = 0
2014-06-12 14:51:27.469869 7f9d60bd2700 10 stat 3.2e_head/e960782e/rbd_data.10a46b8b4567.0000000000001091/head//3
2014-06-12 14:51:27.469969 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid e960782e/rbd_data.10a46b8b4567.0000000000001091/head//3
2014-06-12 14:51:27.470086 7f9d60bd2700 10 getattrs 3.2e_head/e960782e/rbd_data.10a46b8b4567.0000000000001091/head//3 = 0
2014-06-12 14:51:27.470091 7f9d60bd2700 10 stat 3.2e_head/fb61782e/rbd_data.109c6b8b4567.00000000000035b5/head//3
2014-06-12 14:51:27.470181 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid fb61782e/rbd_data.109c6b8b4567.00000000000035b5/head//3
2014-06-12 14:51:27.503487 7f9d60bd2700 10 getattrs 3.2e_head/fb61782e/rbd_data.109c6b8b4567.00000000000035b5/head//3 = 0
2014-06-12 14:51:27.503494 7f9d60bd2700 10 stat 3.2e_head/5943782e/rbd_data.109e6b8b4567.0000000000000e31/head//3
2014-06-12 14:51:27.503582 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 5943782e/rbd_data.109e6b8b4567.0000000000000e31/head//3
2014-06-12 14:51:27.512876 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:27.533145 7f9d60bd2700 10 getattrs 3.2e_head/5943782e/rbd_data.109e6b8b4567.0000000000000e31/head//3 = 0
2014-06-12 14:51:27.533159 7f9d60bd2700 10 stat 3.2e_head/c1c4782e/rbd_data.10956b8b4567.00000000000016e6/head//3
2014-06-12 14:51:27.533244 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid c1c4782e/rbd_data.10956b8b4567.00000000000016e6/head//3
2014-06-12 14:51:27.546802 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:27.552627 7f9d60bd2700 10 getattrs 3.2e_head/c1c4782e/rbd_data.10956b8b4567.00000000000016e6/head//3 = 0
2014-06-12 14:51:27.552645 7f9d60bd2700 10 stat 3.2e_head/2ff4782e/rbd_data.10a06b8b4567.0000000000000801/head//3
2014-06-12 14:51:27.552832 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 2ff4782e/rbd_data.10a06b8b4567.0000000000000801/head//3
2014-06-12 14:51:27.583556 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:27.583622 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:27.601232 7f9d60bd2700 10 getattrs 3.2e_head/2ff4782e/rbd_data.10a06b8b4567.0000000000000801/head//3 = 0
2014-06-12 14:51:27.601242 7f9d60bd2700 10 stat 3.2e_head/73c5782e/rbd_data.10a76b8b4567.0000000000003810/head//3
2014-06-12 14:51:27.601339 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 73c5782e/rbd_data.10a76b8b4567.0000000000003810/head//3
2014-06-12 14:51:27.618034 7f9d60bd2700 10 getattrs 3.2e_head/73c5782e/rbd_data.10a76b8b4567.0000000000003810/head//3 = 0
2014-06-12 14:51:27.618041 7f9d60bd2700 10 stat 3.2e_head/2e86782e/rbd_data.10a96b8b4567.00000000000000e8/head//3
2014-06-12 14:51:27.618138 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 2e86782e/rbd_data.10a96b8b4567.00000000000000e8/head//3
2014-06-12 14:51:27.657603 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:27.657647 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:27.661555 7f9d60bd2700 10 getattrs 3.2e_head/2e86782e/rbd_data.10a96b8b4567.00000000000000e8/head//3 = 0
2014-06-12 14:51:27.661573 7f9d60bd2700 10 stat 3.2e_head/f2a8782e/rbd_data.10946b8b4567.00000000000034f7/head//3
2014-06-12 14:51:27.661698 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f2a8782e/rbd_data.10946b8b4567.00000000000034f7/head//3
2014-06-12 14:51:27.683841 7f9d60bd2700 10 getattrs 3.2e_head/f2a8782e/rbd_data.10946b8b4567.00000000000034f7/head//3 = 0
2014-06-12 14:51:27.683851 7f9d60bd2700 10 stat 3.2e_head/6159782e/rbd_data.10966b8b4567.0000000000003b0b/head//3
2014-06-12 14:51:27.683965 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6159782e/rbd_data.10966b8b4567.0000000000003b0b/head//3
2014-06-12 14:51:27.696182 7f9d60bd2700 10 getattrs 3.2e_head/6159782e/rbd_data.10966b8b4567.0000000000003b0b/head//3 = 0
2014-06-12 14:51:27.696192 7f9d60bd2700 10 stat 3.2e_head/4e6a782e/rbd_data.10ae6b8b4567.0000000000001cb9/head//3
2014-06-12 14:51:27.696302 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4e6a782e/rbd_data.10ae6b8b4567.0000000000001cb9/head//3
2014-06-12 14:51:27.714611 7f9d60bd2700 10 getattrs 3.2e_head/4e6a782e/rbd_data.10ae6b8b4567.0000000000001cb9/head//3 = 0
2014-06-12 14:51:27.714625 7f9d60bd2700 10 stat 3.2e_head/448c782e/rbd_data.109a6b8b4567.00000000000027d0/head//3
2014-06-12 14:51:27.714734 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 448c782e/rbd_data.109a6b8b4567.00000000000027d0/head//3
2014-06-12 14:51:27.728967 7f9d60bd2700 10 getattrs 3.2e_head/448c782e/rbd_data.109a6b8b4567.00000000000027d0/head//3 = 0
2014-06-12 14:51:27.728977 7f9d60bd2700 10 stat 3.2e_head/6b8f782e/rbd_data.10ab6b8b4567.0000000000001dd4/head//3
2014-06-12 14:51:27.729086 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6b8f782e/rbd_data.10ab6b8b4567.0000000000001dd4/head//3
2014-06-12 14:51:27.745933 7f9d60bd2700 10 getattrs 3.2e_head/6b8f782e/rbd_data.10ab6b8b4567.0000000000001dd4/head//3 = 0
2014-06-12 14:51:27.745944 7f9d60bd2700 10 stat 3.2e_head/49d0882e/rbd_data.10b46b8b4567.00000000000038a3/head//3
2014-06-12 14:51:27.746054 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 49d0882e/rbd_data.10b46b8b4567.00000000000038a3/head//3
2014-06-12 14:51:27.758146 7f9d60bd2700 10 getattrs 3.2e_head/49d0882e/rbd_data.10b46b8b4567.00000000000038a3/head//3 = 0
2014-06-12 14:51:27.758153 7f9d60bd2700 10 stat 3.2e_head/d8a3882e/rbd_data.10a36b8b4567.0000000000000eb1/head//3
2014-06-12 14:51:27.758247 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d8a3882e/rbd_data.10a36b8b4567.0000000000000eb1/head//3
2014-06-12 14:51:27.758355 7f9d60bd2700 10 getattrs 3.2e_head/d8a3882e/rbd_data.10a36b8b4567.0000000000000eb1/head//3 = 0
2014-06-12 14:51:27.758360 7f9d60bd2700 10 stat 3.2e_head/3a24882e/rbd_data.10ae6b8b4567.0000000000003409/head//3
2014-06-12 14:51:27.758448 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 3a24882e/rbd_data.10ae6b8b4567.0000000000003409/head//3
2014-06-12 14:51:27.789244 7f9d60bd2700 10 getattrs 3.2e_head/3a24882e/rbd_data.10ae6b8b4567.0000000000003409/head//3 = 0
2014-06-12 14:51:27.789252 7f9d60bd2700 10 stat 3.2e_head/5c35882e/rbd_data.10a96b8b4567.0000000000003a66/head//3
2014-06-12 14:51:27.789346 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 5c35882e/rbd_data.10a96b8b4567.0000000000003a66/head//3
2014-06-12 14:51:27.807055 7f9d60bd2700 10 getattrs 3.2e_head/5c35882e/rbd_data.10a96b8b4567.0000000000003a66/head//3 = 0
2014-06-12 14:51:27.807069 7f9d60bd2700 10 stat 3.2e_head/d686882e/rbd_data.108f6b8b4567.0000000000001980/head//3
2014-06-12 14:51:27.807182 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d686882e/rbd_data.108f6b8b4567.0000000000001980/head//3
2014-06-12 14:51:27.835381 7f9d60bd2700 10 getattrs 3.2e_head/d686882e/rbd_data.108f6b8b4567.0000000000001980/head//3 = 0
2014-06-12 14:51:27.835392 7f9d60bd2700 10 stat 3.2e_head/c688882e/rbd_data.10946b8b4567.0000000000001a48/head//3
2014-06-12 14:51:27.835498 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid c688882e/rbd_data.10946b8b4567.0000000000001a48/head//3
2014-06-12 14:51:27.861685 7f9d60bd2700 10 getattrs 3.2e_head/c688882e/rbd_data.10946b8b4567.0000000000001a48/head//3 = 0
2014-06-12 14:51:27.861696 7f9d60bd2700 10 stat 3.2e_head/d7d8882e/rbd_data.10976b8b4567.00000000000036f9/head//3
2014-06-12 14:51:27.861831 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d7d8882e/rbd_data.10976b8b4567.00000000000036f9/head//3
2014-06-12 14:51:27.887454 7f9d60bd2700 10 getattrs 3.2e_head/d7d8882e/rbd_data.10976b8b4567.00000000000036f9/head//3 = 0
2014-06-12 14:51:27.887464 7f9d60bd2700 10 stat 3.2e_head/5e89882e/rbd_data.10a56b8b4567.000000000000021c/head//3
2014-06-12 14:51:27.887574 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 5e89882e/rbd_data.10a56b8b4567.000000000000021c/head//3
2014-06-12 14:51:27.905502 7f9d44101700  5 -- op tracker -- , seq: 748577, time: 2014-06-12 14:51:27.905419, event: header_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.905518 7f9d44101700  5 -- op tracker -- , seq: 748577, time: 2014-06-12 14:51:27.905422, event: throttled, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.905540 7f9d44101700  5 -- op tracker -- , seq: 748577, time: 2014-06-12 14:51:27.905467, event: all_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.905546 7f9d44101700  5 -- op tracker -- , seq: 748577, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.905551 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:27.905557 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:27.905565 7f9d44101700 15 osd.2 173 enqueue_op 0x124e3500 prio 127 cost 10717 latency 0.000146 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:27.905647 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:27.916598 7f9d60bd2700 10 getattrs 3.2e_head/5e89882e/rbd_data.10a56b8b4567.000000000000021c/head//3 = 0
2014-06-12 14:51:27.916609 7f9d60bd2700 10 stat 3.2e_head/a636982e/rbd_data.10ad6b8b4567.0000000000002567/head//3
2014-06-12 14:51:27.916690 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a636982e/rbd_data.10ad6b8b4567.0000000000002567/head//3
2014-06-12 14:51:27.929368 7f9d60bd2700 10 getattrs 3.2e_head/a636982e/rbd_data.10ad6b8b4567.0000000000002567/head//3 = 0
2014-06-12 14:51:27.929375 7f9d60bd2700 10 stat 3.2e_head/6a07982e/rbd_data.10af6b8b4567.0000000000001b0f/head//3
2014-06-12 14:51:27.929461 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6a07982e/rbd_data.10af6b8b4567.0000000000001b0f/head//3
2014-06-12 14:51:27.961749 7f9d60bd2700 10 getattrs 3.2e_head/6a07982e/rbd_data.10af6b8b4567.0000000000001b0f/head//3 = 0
2014-06-12 14:51:27.961759 7f9d60bd2700 10 collection_getattrs 3.2e_head
2014-06-12 14:51:27.961812 7f9d60bd2700 10 collection_getattrs 3.2e_head = 0
2014-06-12 14:51:27.961818 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:27.961851 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:27.961858 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:27.961871 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:27.961874 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:27.961902 7f9d633d7700 10 osd.2 173 dequeue_op 0x124e3500 prio 127 cost 10717 latency 0.056483 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]
2014-06-12 14:51:27.961950 7f9d633d7700  5 -- op tracker -- , seq: 748577, time: 2014-06-12 14:51:27.961943, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.961979 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] handle_message: 0x124e3500
2014-06-12 14:51:27.961996 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:27.962021 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:27.962041 7f9d633d7700  5 -- op tracker -- , seq: 748577, time: 2014-06-12 14:51:27.962035, event: started, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.962054 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:27.962158 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] map version is 173'2847
2014-06-12 14:51:27.962179 7f9d633d7700 10 osd.2 173 dequeue_op 0x124e3500 finish
2014-06-12 14:51:27.962191 7f9d633d7700  5 -- op tracker -- , seq: 748577, time: 2014-06-12 14:51:27.962185, event: done, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:27.962213 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:27.962218 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:27.962218 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:27.962231 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:27.962254 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:27.962274 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:27.962301 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:27.962307 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:27.962314 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:27.962333 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d1ce682e/rbd_data.10ab6b8b4567.00000000000031e2/head//3, auth == maps.end()
2014-06-12 14:51:27.962351 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d1ce682e/rbd_data.10ab6b8b4567.00000000000031e2/head//3
2014-06-12 14:51:27.962363 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3e10782e/rbd_data.10ac6b8b4567.0000000000003557/head//3, auth == maps.end()
2014-06-12 14:51:27.962372 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3e10782e/rbd_data.10ac6b8b4567.0000000000003557/head//3
2014-06-12 14:51:27.962381 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d30782e/rbd_data.10a96b8b4567.00000000000010ef/head//3, auth == maps.end()
2014-06-12 14:51:27.962392 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d30782e/rbd_data.10a96b8b4567.00000000000010ef/head//3
2014-06-12 14:51:27.962401 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e960782e/rbd_data.10a46b8b4567.0000000000001091/head//3, auth == maps.end()
2014-06-12 14:51:27.962410 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e960782e/rbd_data.10a46b8b4567.0000000000001091/head//3
2014-06-12 14:51:27.962421 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fb61782e/rbd_data.109c6b8b4567.00000000000035b5/head//3, auth == maps.end()
2014-06-12 14:51:27.962431 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj fb61782e/rbd_data.109c6b8b4567.00000000000035b5/head//3
2014-06-12 14:51:27.962440 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5943782e/rbd_data.109e6b8b4567.0000000000000e31/head//3, auth == maps.end()
2014-06-12 14:51:27.962450 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5943782e/rbd_data.109e6b8b4567.0000000000000e31/head//3
2014-06-12 14:51:27.962460 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c1c4782e/rbd_data.10956b8b4567.00000000000016e6/head//3, auth == maps.end()
2014-06-12 14:51:27.962468 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c1c4782e/rbd_data.10956b8b4567.00000000000016e6/head//3
2014-06-12 14:51:27.962478 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2ff4782e/rbd_data.10a06b8b4567.0000000000000801/head//3, auth == maps.end()
2014-06-12 14:51:27.962488 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2ff4782e/rbd_data.10a06b8b4567.0000000000000801/head//3
2014-06-12 14:51:27.962507 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 73c5782e/rbd_data.10a76b8b4567.0000000000003810/head//3, auth == maps.end()
2014-06-12 14:51:27.962516 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 73c5782e/rbd_data.10a76b8b4567.0000000000003810/head//3
2014-06-12 14:51:27.962524 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2e86782e/rbd_data.10a96b8b4567.00000000000000e8/head//3, auth == maps.end()
2014-06-12 14:51:27.962533 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2e86782e/rbd_data.10a96b8b4567.00000000000000e8/head//3
2014-06-12 14:51:27.962541 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f2a8782e/rbd_data.10946b8b4567.00000000000034f7/head//3, auth == maps.end()
2014-06-12 14:51:27.962549 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f2a8782e/rbd_data.10946b8b4567.00000000000034f7/head//3
2014-06-12 14:51:27.962558 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6159782e/rbd_data.10966b8b4567.0000000000003b0b/head//3, auth == maps.end()
2014-06-12 14:51:27.962566 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6159782e/rbd_data.10966b8b4567.0000000000003b0b/head//3
2014-06-12 14:51:27.962575 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4e6a782e/rbd_data.10ae6b8b4567.0000000000001cb9/head//3, auth == maps.end()
2014-06-12 14:51:27.962583 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4e6a782e/rbd_data.10ae6b8b4567.0000000000001cb9/head//3
2014-06-12 14:51:27.962593 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 448c782e/rbd_data.109a6b8b4567.00000000000027d0/head//3, auth == maps.end()
2014-06-12 14:51:27.962601 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 448c782e/rbd_data.109a6b8b4567.00000000000027d0/head//3
2014-06-12 14:51:27.962611 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6b8f782e/rbd_data.10ab6b8b4567.0000000000001dd4/head//3, auth == maps.end()
2014-06-12 14:51:27.962619 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6b8f782e/rbd_data.10ab6b8b4567.0000000000001dd4/head//3
2014-06-12 14:51:27.962629 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 49d0882e/rbd_data.10b46b8b4567.00000000000038a3/head//3, auth == maps.end()
2014-06-12 14:51:27.962637 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 49d0882e/rbd_data.10b46b8b4567.00000000000038a3/head//3
2014-06-12 14:51:27.962647 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d8a3882e/rbd_data.10a36b8b4567.0000000000000eb1/head//3, auth == maps.end()
2014-06-12 14:51:27.962658 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d8a3882e/rbd_data.10a36b8b4567.0000000000000eb1/head//3
2014-06-12 14:51:27.962667 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3a24882e/rbd_data.10ae6b8b4567.0000000000003409/head//3, auth == maps.end()
2014-06-12 14:51:27.962675 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3a24882e/rbd_data.10ae6b8b4567.0000000000003409/head//3
2014-06-12 14:51:27.962684 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5c35882e/rbd_data.10a96b8b4567.0000000000003a66/head//3, auth == maps.end()
2014-06-12 14:51:27.962692 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5c35882e/rbd_data.10a96b8b4567.0000000000003a66/head//3
2014-06-12 14:51:27.962700 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d686882e/rbd_data.108f6b8b4567.0000000000001980/head//3, auth == maps.end()
2014-06-12 14:51:27.962708 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d686882e/rbd_data.108f6b8b4567.0000000000001980/head//3
2014-06-12 14:51:27.962717 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c688882e/rbd_data.10946b8b4567.0000000000001a48/head//3, auth == maps.end()
2014-06-12 14:51:27.962727 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c688882e/rbd_data.10946b8b4567.0000000000001a48/head//3
2014-06-12 14:51:27.962738 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d7d8882e/rbd_data.10976b8b4567.00000000000036f9/head//3, auth == maps.end()
2014-06-12 14:51:27.962747 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d7d8882e/rbd_data.10976b8b4567.00000000000036f9/head//3
2014-06-12 14:51:27.962758 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5e89882e/rbd_data.10a56b8b4567.000000000000021c/head//3, auth == maps.end()
2014-06-12 14:51:27.962767 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5e89882e/rbd_data.10a56b8b4567.000000000000021c/head//3
2014-06-12 14:51:27.962776 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a636982e/rbd_data.10ad6b8b4567.0000000000002567/head//3, auth == maps.end()
2014-06-12 14:51:27.962784 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a636982e/rbd_data.10ad6b8b4567.0000000000002567/head//3
2014-06-12 14:51:27.962794 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6a07982e/rbd_data.10af6b8b4567.0000000000001b0f/head//3, auth == maps.end()
2014-06-12 14:51:27.962804 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6a07982e/rbd_data.10af6b8b4567.0000000000001b0f/head//3
2014-06-12 14:51:27.962816 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] 
2014-06-12 14:51:27.962824 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub
2014-06-12 14:51:27.962837 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6a07982e/rbd_data.10af6b8b4567.0000000000001b0f/head//3 6a07982e/rbd_data.10af6b8b4567.0000000000001b0f/head//3(173'1235 client.4312.0:61056 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1235)
2014-06-12 14:51:27.962851 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a636982e/rbd_data.10ad6b8b4567.0000000000002567/head//3 a636982e/rbd_data.10ad6b8b4567.0000000000002567/head//3(173'1559 client.4310.0:76360 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1559)
2014-06-12 14:51:27.962863 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  5e89882e/rbd_data.10a56b8b4567.000000000000021c/head//3 5e89882e/rbd_data.10a56b8b4567.000000000000021c/head//3(173'2405 client.4302.0:122005 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2405)
2014-06-12 14:51:27.962876 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d7d8882e/rbd_data.10976b8b4567.00000000000036f9/head//3 d7d8882e/rbd_data.10976b8b4567.00000000000036f9/head//3(173'2218 client.4288.0:106666 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2218)
2014-06-12 14:51:27.962888 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  c688882e/rbd_data.10946b8b4567.0000000000001a48/head//3 c688882e/rbd_data.10946b8b4567.0000000000001a48/head//3(173'1116 client.4285.0:56000 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1116)
2014-06-12 14:51:27.962901 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d686882e/rbd_data.108f6b8b4567.0000000000001980/head//3 d686882e/rbd_data.108f6b8b4567.0000000000001980/head//3(173'1184 client.4280.0:55219 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1184)
2014-06-12 14:51:27.962913 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  5c35882e/rbd_data.10a96b8b4567.0000000000003a66/head//3 5c35882e/rbd_data.10a96b8b4567.0000000000003a66/head//3(173'2334 client.4306.0:117241 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2334)
2014-06-12 14:51:27.962925 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  3a24882e/rbd_data.10ae6b8b4567.0000000000003409/head//3 3a24882e/rbd_data.10ae6b8b4567.0000000000003409/head//3(173'1904 client.4311.0:100852 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1904)
2014-06-12 14:51:27.962937 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d8a3882e/rbd_data.10a36b8b4567.0000000000000eb1/head//3 d8a3882e/rbd_data.10a36b8b4567.0000000000000eb1/head//3(173'2844 client.4300.0:139905 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2844)
2014-06-12 14:51:27.962949 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  49d0882e/rbd_data.10b46b8b4567.00000000000038a3/head//3 49d0882e/rbd_data.10b46b8b4567.00000000000038a3/head//3(173'2294 client.4317.0:112003 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2294)
2014-06-12 14:51:27.962961 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6b8f782e/rbd_data.10ab6b8b4567.0000000000001dd4/head//3 6b8f782e/rbd_data.10ab6b8b4567.0000000000001dd4/head//3(173'1267 client.4308.0:63828 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1267)
2014-06-12 14:51:27.962973 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  448c782e/rbd_data.109a6b8b4567.00000000000027d0/head//3 448c782e/rbd_data.109a6b8b4567.00000000000027d0/head//3(173'1582 client.4291.0:79641 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1582)
2014-06-12 14:51:27.962985 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4e6a782e/rbd_data.10ae6b8b4567.0000000000001cb9/head//3 4e6a782e/rbd_data.10ae6b8b4567.0000000000001cb9/head//3(173'1170 client.4311.0:60250 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1170)
2014-06-12 14:51:27.962997 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6159782e/rbd_data.10966b8b4567.0000000000003b0b/head//3 6159782e/rbd_data.10966b8b4567.0000000000003b0b/head//3(173'2240 client.4287.0:113543 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2240)
2014-06-12 14:51:27.963009 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f2a8782e/rbd_data.10946b8b4567.00000000000034f7/head//3 f2a8782e/rbd_data.10946b8b4567.00000000000034f7/head//3(173'1988 client.4285.0:103851 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1988)
2014-06-12 14:51:27.963025 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  2e86782e/rbd_data.10a96b8b4567.00000000000000e8/head//3 2e86782e/rbd_data.10a96b8b4567.00000000000000e8/head//3(173'2361 client.4306.0:119454 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2361)
2014-06-12 14:51:27.963037 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  73c5782e/rbd_data.10a76b8b4567.0000000000003810/head//3 73c5782e/rbd_data.10a76b8b4567.0000000000003810/head//3(173'2173 client.4304.0:110710 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2173)
2014-06-12 14:51:27.963049 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  2ff4782e/rbd_data.10a06b8b4567.0000000000000801/head//3 2ff4782e/rbd_data.10a06b8b4567.0000000000000801/head//3(173'2611 client.4297.0:129681 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2611)
2014-06-12 14:51:27.963061 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  c1c4782e/rbd_data.10956b8b4567.00000000000016e6/head//3 c1c4782e/rbd_data.10956b8b4567.00000000000016e6/head//3(173'1044 client.4286.0:50466 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1044)
2014-06-12 14:51:27.963073 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  5943782e/rbd_data.109e6b8b4567.0000000000000e31/head//3 5943782e/rbd_data.109e6b8b4567.0000000000000e31/head//3(173'2814 client.4295.0:140642 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2814)
2014-06-12 14:51:27.963085 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  fb61782e/rbd_data.109c6b8b4567.00000000000035b5/head//3 fb61782e/rbd_data.109c6b8b4567.00000000000035b5/head//3(173'2124 client.4293.0:106791 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2124)
2014-06-12 14:51:27.963097 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  e960782e/rbd_data.10a46b8b4567.0000000000001091/head//3 e960782e/rbd_data.10a46b8b4567.0000000000001091/head//3(173'2847 client.4301.0:145282 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2847)
2014-06-12 14:51:27.963109 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d30782e/rbd_data.10a96b8b4567.00000000000010ef/head//3 d30782e/rbd_data.10a96b8b4567.00000000000010ef/head//3(173'843 client.4306.0:40995 wrlock_by=unknown.0.0:0 dirty s 4194304 uv843)
2014-06-12 14:51:27.963121 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  3e10782e/rbd_data.10ac6b8b4567.0000000000003557/head//3 3e10782e/rbd_data.10ac6b8b4567.0000000000003557/head//3(173'2142 client.4309.0:105104 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2142)
2014-06-12 14:51:27.963133 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d1ce682e/rbd_data.10ab6b8b4567.00000000000031e2/head//3 d1ce682e/rbd_data.10ab6b8b4567.00000000000031e2/head//3(173'1911 client.4308.0:100028 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1911)
2014-06-12 14:51:27.963141 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:27.963152 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:27.963161 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:27.963163 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:27.963164 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:27.963206 7f9d60bd2700 10 collection_list_partial 3.2e_head start:31a7982e//0//-1 is_max:0
2014-06-12 14:51:27.963404 7f9d60bd2700 20 objects: [4628982e/rbd_data.10966b8b4567.00000000000015af/head//3,a69a982e/rbd_data.10996b8b4567.00000000000032e8/head//3,70ba982e/rbd_data.108d6b8b4567.000000000000171e/head//3,9fcc982e/rbd_data.10916b8b4567.0000000000002b8f/head//3,3c9d982e/rbd_data.10b06b8b4567.00000000000035b4/head//3,5494a82e/rbd_data.10ae6b8b4567.0000000000000458/head//3,975a82e/rbd_data.109e6b8b4567.0000000000003135/head//3,d676a82e/rbd_data.10a46b8b4567.00000000000000c0/head//3,33f6a82e/rbd_data.109b6b8b4567.0000000000001e44/head//3,a7b7a82e/rbd_data.10a66b8b4567.0000000000001a05/head//3,9ed8a82e/rbd_data.10a46b8b4567.0000000000001f73/head//3,2a09a82e/rbd_data.10a46b8b4567.0000000000002a75/head//3,935ca82e/rbd_data.10ac6b8b4567.0000000000003be6/head//3,9f7ca82e/rbd_data.10966b8b4567.0000000000001163/head//3,4d3da82e/rbd_data.10a26b8b4567.00000000000031a1/head//3,8bada82e/rbd_data.10b16b8b4567.0000000000002fa6/head//3,e4eda82e/rbd_data.10a56b8b4567.0000000000002bbf/head//3,b170b82e/rbd_data.10ac6b8b4567.000000000000252a/head//3,4e63b82e/rbd_data.108e6b8b4567.0000000000000174/head//3,d624b82e/rbd_data.108e6b8b4567.0000000000003ad9/head//3,ecd5b82e/rbd_data.109e6b8b4567.0000000000001e6e/head//3,a528b82e/rbd_data.10916b8b4567.00000000000020ba/head//3,6048b82e/rbd_data.10af6b8b4567.0000000000002973/head//3,e999b82e/rbd_data.10906b8b4567.0000000000000e08/head//3,d91cb82e/rbd_data.109b6b8b4567.0000000000000a47/head//3]
2014-06-12 14:51:27.963866 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:27.963885 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:27.963899 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:27.963907 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:27.963913 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:27.963918 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:27.963924 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_map_chunk [31a7982e//0//-1,9600c82e//0//-1)
2014-06-12 14:51:27.963930 7f9d60bd2700 10 collection_list_partial 3.2e_head start:31a7982e//0//-1 is_max:0
2014-06-12 14:51:27.964348 7f9d60bd2700 20 objects: [4628982e/rbd_data.10966b8b4567.00000000000015af/head//3,a69a982e/rbd_data.10996b8b4567.00000000000032e8/head//3,70ba982e/rbd_data.108d6b8b4567.000000000000171e/head//3,9fcc982e/rbd_data.10916b8b4567.0000000000002b8f/head//3,3c9d982e/rbd_data.10b06b8b4567.00000000000035b4/head//3,5494a82e/rbd_data.10ae6b8b4567.0000000000000458/head//3,975a82e/rbd_data.109e6b8b4567.0000000000003135/head//3,d676a82e/rbd_data.10a46b8b4567.00000000000000c0/head//3,33f6a82e/rbd_data.109b6b8b4567.0000000000001e44/head//3,a7b7a82e/rbd_data.10a66b8b4567.0000000000001a05/head//3,9ed8a82e/rbd_data.10a46b8b4567.0000000000001f73/head//3,2a09a82e/rbd_data.10a46b8b4567.0000000000002a75/head//3,935ca82e/rbd_data.10ac6b8b4567.0000000000003be6/head//3,9f7ca82e/rbd_data.10966b8b4567.0000000000001163/head//3,4d3da82e/rbd_data.10a26b8b4567.00000000000031a1/head//3,8bada82e/rbd_data.10b16b8b4567.0000000000002fa6/head//3,e4eda82e/rbd_data.10a56b8b4567.0000000000002bbf/head//3,b170b82e/rbd_data.10ac6b8b4567.000000000000252a/head//3,4e63b82e/rbd_data.108e6b8b4567.0000000000000174/head//3,d624b82e/rbd_data.108e6b8b4567.0000000000003ad9/head//3,ecd5b82e/rbd_data.109e6b8b4567.0000000000001e6e/head//3,a528b82e/rbd_data.10916b8b4567.00000000000020ba/head//3,6048b82e/rbd_data.10af6b8b4567.0000000000002973/head//3,e999b82e/rbd_data.10906b8b4567.0000000000000e08/head//3,d91cb82e/rbd_data.109b6b8b4567.0000000000000a47/head//3,9600c82e/rbd_data.10b16b8b4567.0000000000002853/head//3,3fe0c82e/rbd_data.10916b8b4567.00000000000026b0/head//3,6c61c82e/rbd_data.10a16b8b4567.0000000000001491/head//3,c113c82e/rbd_data.108d6b8b4567.000000000000042f/head//3,86a4c82e/rbd_data.10b06b8b4567.0000000000000c85/head//3,4da5c82e/rbd_data.10a36b8b4567.0000000000000bf1/head//3,ea09c82e/rbd_data.108e6b8b4567.0000000000001715/head//3,bf6ac82e/rbd_data.109e6b8b4567.0000000000002de0/head//3,6e9bc82e/rbd_data.10916b8b4567.0000000000001c90/head//3,f26ec82e/rbd_data.10b36b8b4567.0000000000002676/head//3,ee5fc82e/rbd_data.10956b8b4567.0000000000000295/head//3,f221d82e/rbd_data.10a36b8b4567.00000000000034c8/head//3,bbb2d82e/rbd_data.10af6b8b4567.0000000000002776/head//3,b453d82e/rbd_data.10aa6b8b4567.00000000000022ff/head//3,cf85d82e/rbd_data.10956b8b4567.0000000000000ea0/head//3,f176d82e/rbd_data.10b36b8b4567.000000000000316b/head//3,d45cd82e/rbd_data.10906b8b4567.0000000000003269/head//3,2302e82e/rbd_data.10ad6b8b4567.000000000000077c/head//3,9ec2e82e/rbd_data.109f6b8b4567.00000000000017db/head//3,b656e82e/rbd_data.108d6b8b4567.0000000000000f21/head//3,1158e82e/rbd_data.10b26b8b4567.000000000000189d/head//3,1a1de82e/rbd_data.10aa6b8b4567.000000000000137e/head//3,a98ee82e/rbd_data.10966b8b4567.0000000000001fcb/head//3,e651f82e/rbd_data.10b46b8b4567.000000000000014d/head//3,7402f82e/rbd_data.10b46b8b4567.00000000000024ae/head//3,8d93f82e/rbd_data.109c6b8b4567.000000000000333e/head//3,8d27f82e/rbd_data.10936b8b4567.0000000000000235/head//3,8178f82e/rbd_data.10986b8b4567.0000000000001cd9/head//3,d3c9f82e/rbd_data.109e6b8b4567.0000000000002ace/head//3,6f5cf82e/rbd_data.10b36b8b4567.00000000000025f2/head//3,fa0df82e/rbd_data.10a66b8b4567.000000000000257d/head//3,536ef82e/rbd_data.10996b8b4567.0000000000002d9f/head//3,ddeef82e/rbd_data.10a86b8b4567.0000000000001509/head//3,f44ff82e/rbd_data.10aa6b8b4567.000000000000272a/head//3]
2014-06-12 14:51:27.964415 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:27.964429 7f9d60bd2700 10 stat 3.2e_head/4628982e/rbd_data.10966b8b4567.00000000000015af/head//3
2014-06-12 14:51:27.964540 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4628982e/rbd_data.10966b8b4567.00000000000015af/head//3
2014-06-12 14:51:27.975495 7f9d60bd2700 10 getattrs 3.2e_head/4628982e/rbd_data.10966b8b4567.00000000000015af/head//3 = 0
2014-06-12 14:51:27.975508 7f9d60bd2700 10 stat 3.2e_head/a69a982e/rbd_data.10996b8b4567.00000000000032e8/head//3
2014-06-12 14:51:27.975600 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a69a982e/rbd_data.10996b8b4567.00000000000032e8/head//3
2014-06-12 14:51:27.976536 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:27.993274 7f9d60bd2700 10 getattrs 3.2e_head/a69a982e/rbd_data.10996b8b4567.00000000000032e8/head//3 = 0
2014-06-12 14:51:27.993284 7f9d60bd2700 10 stat 3.2e_head/70ba982e/rbd_data.108d6b8b4567.000000000000171e/head//3
2014-06-12 14:51:27.993386 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 70ba982e/rbd_data.108d6b8b4567.000000000000171e/head//3
2014-06-12 14:51:27.993925 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:28.020980 7f9d60bd2700 10 getattrs 3.2e_head/70ba982e/rbd_data.108d6b8b4567.000000000000171e/head//3 = 0
2014-06-12 14:51:28.020988 7f9d60bd2700 10 stat 3.2e_head/9fcc982e/rbd_data.10916b8b4567.0000000000002b8f/head//3
2014-06-12 14:51:28.021088 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 9fcc982e/rbd_data.10916b8b4567.0000000000002b8f/head//3
2014-06-12 14:51:28.050354 7f9d60bd2700 10 getattrs 3.2e_head/9fcc982e/rbd_data.10916b8b4567.0000000000002b8f/head//3 = 0
2014-06-12 14:51:28.050371 7f9d60bd2700 10 stat 3.2e_head/3c9d982e/rbd_data.10b06b8b4567.00000000000035b4/head//3
2014-06-12 14:51:28.050557 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 3c9d982e/rbd_data.10b06b8b4567.00000000000035b4/head//3
2014-06-12 14:51:28.061399 7f9d60bd2700 10 getattrs 3.2e_head/3c9d982e/rbd_data.10b06b8b4567.00000000000035b4/head//3 = 0
2014-06-12 14:51:28.061408 7f9d60bd2700 10 stat 3.2e_head/5494a82e/rbd_data.10ae6b8b4567.0000000000000458/head//3
2014-06-12 14:51:28.061494 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 5494a82e/rbd_data.10ae6b8b4567.0000000000000458/head//3
2014-06-12 14:51:28.064273 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:28.100326 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:28.100421 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:28.101258 7f9d60bd2700 10 getattrs 3.2e_head/5494a82e/rbd_data.10ae6b8b4567.0000000000000458/head//3 = 0
2014-06-12 14:51:28.101280 7f9d60bd2700 10 stat 3.2e_head/975a82e/rbd_data.109e6b8b4567.0000000000003135/head//3
2014-06-12 14:51:28.101503 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 975a82e/rbd_data.109e6b8b4567.0000000000003135/head//3
2014-06-12 14:51:28.118863 7f9d60bd2700 10 getattrs 3.2e_head/975a82e/rbd_data.109e6b8b4567.0000000000003135/head//3 = 0
2014-06-12 14:51:28.118870 7f9d60bd2700 10 stat 3.2e_head/d676a82e/rbd_data.10a46b8b4567.00000000000000c0/head//3
2014-06-12 14:51:28.118961 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d676a82e/rbd_data.10a46b8b4567.00000000000000c0/head//3
2014-06-12 14:51:28.128839 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:28.128907 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:28.167034 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:28.169431 7f9d60bd2700 10 getattrs 3.2e_head/d676a82e/rbd_data.10a46b8b4567.00000000000000c0/head//3 = 0
2014-06-12 14:51:28.169445 7f9d60bd2700 10 stat 3.2e_head/33f6a82e/rbd_data.109b6b8b4567.0000000000001e44/head//3
2014-06-12 14:51:28.169597 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 33f6a82e/rbd_data.109b6b8b4567.0000000000001e44/head//3
2014-06-12 14:51:28.186370 7f9d60bd2700 10 getattrs 3.2e_head/33f6a82e/rbd_data.109b6b8b4567.0000000000001e44/head//3 = 0
2014-06-12 14:51:28.186382 7f9d60bd2700 10 stat 3.2e_head/a7b7a82e/rbd_data.10a66b8b4567.0000000000001a05/head//3
2014-06-12 14:51:28.186534 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a7b7a82e/rbd_data.10a66b8b4567.0000000000001a05/head//3
2014-06-12 14:51:28.186664 7f9d60bd2700 10 getattrs 3.2e_head/a7b7a82e/rbd_data.10a66b8b4567.0000000000001a05/head//3 = 0
2014-06-12 14:51:28.186674 7f9d60bd2700 10 stat 3.2e_head/9ed8a82e/rbd_data.10a46b8b4567.0000000000001f73/head//3
2014-06-12 14:51:28.186804 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 9ed8a82e/rbd_data.10a46b8b4567.0000000000001f73/head//3
2014-06-12 14:51:28.188726 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:28.188746 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:28.189298 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:28.189355 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:28.189439 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:28.189477 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:28.189495 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:28.189528 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:28.189566 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:28.189592 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:28.189629 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:28.189651 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:28.189678 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:28.189714 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:28.189737 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:28.189765 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:28.189788 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:28.189813 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:28.189847 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:28.189871 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:28.189895 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:28.189921 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:28.189945 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:28.189969 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:28.218307 7f9d60bd2700 10 getattrs 3.2e_head/9ed8a82e/rbd_data.10a46b8b4567.0000000000001f73/head//3 = 0
2014-06-12 14:51:28.218321 7f9d60bd2700 10 stat 3.2e_head/2a09a82e/rbd_data.10a46b8b4567.0000000000002a75/head//3
2014-06-12 14:51:28.218509 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 2a09a82e/rbd_data.10a46b8b4567.0000000000002a75/head//3
2014-06-12 14:51:28.245331 7f9d60bd2700 10 getattrs 3.2e_head/2a09a82e/rbd_data.10a46b8b4567.0000000000002a75/head//3 = 0
2014-06-12 14:51:28.245339 7f9d60bd2700 10 stat 3.2e_head/935ca82e/rbd_data.10ac6b8b4567.0000000000003be6/head//3
2014-06-12 14:51:28.245425 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 935ca82e/rbd_data.10ac6b8b4567.0000000000003be6/head//3
2014-06-12 14:51:28.275076 7f9d60bd2700 10 getattrs 3.2e_head/935ca82e/rbd_data.10ac6b8b4567.0000000000003be6/head//3 = 0
2014-06-12 14:51:28.275086 7f9d60bd2700 10 stat 3.2e_head/9f7ca82e/rbd_data.10966b8b4567.0000000000001163/head//3
2014-06-12 14:51:28.275176 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 9f7ca82e/rbd_data.10966b8b4567.0000000000001163/head//3
2014-06-12 14:51:28.275273 7f9d60bd2700 10 getattrs 3.2e_head/9f7ca82e/rbd_data.10966b8b4567.0000000000001163/head//3 = 0
2014-06-12 14:51:28.275286 7f9d60bd2700 10 stat 3.2e_head/4d3da82e/rbd_data.10a26b8b4567.00000000000031a1/head//3
2014-06-12 14:51:28.275367 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4d3da82e/rbd_data.10a26b8b4567.00000000000031a1/head//3
2014-06-12 14:51:28.295041 7f9d60bd2700 10 getattrs 3.2e_head/4d3da82e/rbd_data.10a26b8b4567.00000000000031a1/head//3 = 0
2014-06-12 14:51:28.295050 7f9d60bd2700 10 stat 3.2e_head/8bada82e/rbd_data.10b16b8b4567.0000000000002fa6/head//3
2014-06-12 14:51:28.295155 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8bada82e/rbd_data.10b16b8b4567.0000000000002fa6/head//3
2014-06-12 14:51:28.311418 7f9d60bd2700 10 getattrs 3.2e_head/8bada82e/rbd_data.10b16b8b4567.0000000000002fa6/head//3 = 0
2014-06-12 14:51:28.311427 7f9d60bd2700 10 stat 3.2e_head/e4eda82e/rbd_data.10a56b8b4567.0000000000002bbf/head//3
2014-06-12 14:51:28.311526 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid e4eda82e/rbd_data.10a56b8b4567.0000000000002bbf/head//3
2014-06-12 14:51:28.316459 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:28.325967 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:28.339335 7f9d60bd2700 10 getattrs 3.2e_head/e4eda82e/rbd_data.10a56b8b4567.0000000000002bbf/head//3 = 0
2014-06-12 14:51:28.339344 7f9d60bd2700 10 stat 3.2e_head/b170b82e/rbd_data.10ac6b8b4567.000000000000252a/head//3
2014-06-12 14:51:28.339460 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid b170b82e/rbd_data.10ac6b8b4567.000000000000252a/head//3
2014-06-12 14:51:28.345455 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:28.345475 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:51:28.345479 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:51:28.345483 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:51:28.345493 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:28.345497 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:28.345499 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:28.345507 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:28.357371 7f9d60bd2700 10 getattrs 3.2e_head/b170b82e/rbd_data.10ac6b8b4567.000000000000252a/head//3 = 0
2014-06-12 14:51:28.357381 7f9d60bd2700 10 stat 3.2e_head/4e63b82e/rbd_data.108e6b8b4567.0000000000000174/head//3
2014-06-12 14:51:28.357495 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4e63b82e/rbd_data.108e6b8b4567.0000000000000174/head//3
2014-06-12 14:51:28.365876 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:28.370684 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:28.381493 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:28.387840 7f9d60bd2700 10 getattrs 3.2e_head/4e63b82e/rbd_data.108e6b8b4567.0000000000000174/head//3 = 0
2014-06-12 14:51:28.387859 7f9d60bd2700 10 stat 3.2e_head/d624b82e/rbd_data.108e6b8b4567.0000000000003ad9/head//3
2014-06-12 14:51:28.387980 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d624b82e/rbd_data.108e6b8b4567.0000000000003ad9/head//3
2014-06-12 14:51:28.404989 7f9d60bd2700 10 getattrs 3.2e_head/d624b82e/rbd_data.108e6b8b4567.0000000000003ad9/head//3 = 0
2014-06-12 14:51:28.404999 7f9d60bd2700 10 stat 3.2e_head/ecd5b82e/rbd_data.109e6b8b4567.0000000000001e6e/head//3
2014-06-12 14:51:28.405129 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid ecd5b82e/rbd_data.109e6b8b4567.0000000000001e6e/head//3
2014-06-12 14:51:28.416171 7f9d60bd2700 10 getattrs 3.2e_head/ecd5b82e/rbd_data.109e6b8b4567.0000000000001e6e/head//3 = 0
2014-06-12 14:51:28.416181 7f9d60bd2700 10 stat 3.2e_head/a528b82e/rbd_data.10916b8b4567.00000000000020ba/head//3
2014-06-12 14:51:28.416304 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a528b82e/rbd_data.10916b8b4567.00000000000020ba/head//3
2014-06-12 14:51:28.432519 7f9d60bd2700 10 getattrs 3.2e_head/a528b82e/rbd_data.10916b8b4567.00000000000020ba/head//3 = 0
2014-06-12 14:51:28.432535 7f9d60bd2700 10 stat 3.2e_head/6048b82e/rbd_data.10af6b8b4567.0000000000002973/head//3
2014-06-12 14:51:28.432657 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6048b82e/rbd_data.10af6b8b4567.0000000000002973/head//3
2014-06-12 14:51:28.463513 7f9d60bd2700 10 getattrs 3.2e_head/6048b82e/rbd_data.10af6b8b4567.0000000000002973/head//3 = 0
2014-06-12 14:51:28.463520 7f9d60bd2700 10 stat 3.2e_head/e999b82e/rbd_data.10906b8b4567.0000000000000e08/head//3
2014-06-12 14:51:28.463609 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid e999b82e/rbd_data.10906b8b4567.0000000000000e08/head//3
2014-06-12 14:51:28.486881 7f9d44101700  5 -- op tracker -- , seq: 748578, time: 2014-06-12 14:51:28.486789, event: header_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:28.486894 7f9d44101700  5 -- op tracker -- , seq: 748578, time: 2014-06-12 14:51:28.486792, event: throttled, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:28.486910 7f9d44101700  5 -- op tracker -- , seq: 748578, time: 2014-06-12 14:51:28.486842, event: all_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:28.486918 7f9d44101700  5 -- op tracker -- , seq: 748578, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:28.486923 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:28.486930 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:28.486939 7f9d44101700 15 osd.2 173 enqueue_op 0x124e2200 prio 127 cost 10717 latency 0.000150 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:28.487005 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:28.495816 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:28.495872 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:28.509447 7f9d60bd2700 10 getattrs 3.2e_head/e999b82e/rbd_data.10906b8b4567.0000000000000e08/head//3 = 0
2014-06-12 14:51:28.509455 7f9d60bd2700 10 stat 3.2e_head/d91cb82e/rbd_data.109b6b8b4567.0000000000000a47/head//3
2014-06-12 14:51:28.509545 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d91cb82e/rbd_data.109b6b8b4567.0000000000000a47/head//3
2014-06-12 14:51:28.512719 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:28.518528 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:28.522475 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:28.531905 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:28.546825 7f9d60bd2700 10 getattrs 3.2e_head/d91cb82e/rbd_data.109b6b8b4567.0000000000000a47/head//3 = 0
2014-06-12 14:51:28.546844 7f9d60bd2700 10 collection_getattrs 3.2e_head
2014-06-12 14:51:28.546923 7f9d60bd2700 10 collection_getattrs 3.2e_head = 0
2014-06-12 14:51:28.546930 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:28.546977 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:28.546989 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:28.547004 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:28.547009 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:28.547041 7f9d663dd700 10 osd.2 173 dequeue_op 0x124e2200 prio 127 cost 10717 latency 0.060251 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]
2014-06-12 14:51:28.547089 7f9d663dd700  5 -- op tracker -- , seq: 748578, time: 2014-06-12 14:51:28.547081, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:28.547120 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] handle_message: 0x124e2200
2014-06-12 14:51:28.547138 7f9d663dd700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:28.547155 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:28.547172 7f9d663dd700  5 -- op tracker -- , seq: 748578, time: 2014-06-12 14:51:28.547167, event: started, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:28.547187 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:28.547294 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] map version is 173'2847
2014-06-12 14:51:28.547314 7f9d663dd700 10 osd.2 173 dequeue_op 0x124e2200 finish
2014-06-12 14:51:28.547327 7f9d663dd700  5 -- op tracker -- , seq: 748578, time: 2014-06-12 14:51:28.547321, event: done, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:28.547350 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:28.547356 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:28.547354 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:28.547367 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:28.547396 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:28.547407 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:28.547443 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:28.547454 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:28.547464 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:28.547496 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4628982e/rbd_data.10966b8b4567.00000000000015af/head//3, auth == maps.end()
2014-06-12 14:51:28.547536 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4628982e/rbd_data.10966b8b4567.00000000000015af/head//3
2014-06-12 14:51:28.547560 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a69a982e/rbd_data.10996b8b4567.00000000000032e8/head//3, auth == maps.end()
2014-06-12 14:51:28.547575 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a69a982e/rbd_data.10996b8b4567.00000000000032e8/head//3
2014-06-12 14:51:28.547590 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 70ba982e/rbd_data.108d6b8b4567.000000000000171e/head//3, auth == maps.end()
2014-06-12 14:51:28.547604 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 70ba982e/rbd_data.108d6b8b4567.000000000000171e/head//3
2014-06-12 14:51:28.547619 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9fcc982e/rbd_data.10916b8b4567.0000000000002b8f/head//3, auth == maps.end()
2014-06-12 14:51:28.547632 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9fcc982e/rbd_data.10916b8b4567.0000000000002b8f/head//3
2014-06-12 14:51:28.547651 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3c9d982e/rbd_data.10b06b8b4567.00000000000035b4/head//3, auth == maps.end()
2014-06-12 14:51:28.547667 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3c9d982e/rbd_data.10b06b8b4567.00000000000035b4/head//3
2014-06-12 14:51:28.547682 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5494a82e/rbd_data.10ae6b8b4567.0000000000000458/head//3, auth == maps.end()
2014-06-12 14:51:28.547696 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5494a82e/rbd_data.10ae6b8b4567.0000000000000458/head//3
2014-06-12 14:51:28.547710 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 975a82e/rbd_data.109e6b8b4567.0000000000003135/head//3, auth == maps.end()
2014-06-12 14:51:28.547726 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 975a82e/rbd_data.109e6b8b4567.0000000000003135/head//3
2014-06-12 14:51:28.547742 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d676a82e/rbd_data.10a46b8b4567.00000000000000c0/head//3, auth == maps.end()
2014-06-12 14:51:28.547756 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d676a82e/rbd_data.10a46b8b4567.00000000000000c0/head//3
2014-06-12 14:51:28.547771 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 33f6a82e/rbd_data.109b6b8b4567.0000000000001e44/head//3, auth == maps.end()
2014-06-12 14:51:28.547785 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 33f6a82e/rbd_data.109b6b8b4567.0000000000001e44/head//3
2014-06-12 14:51:28.547799 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a7b7a82e/rbd_data.10a66b8b4567.0000000000001a05/head//3, auth == maps.end()
2014-06-12 14:51:28.547813 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a7b7a82e/rbd_data.10a66b8b4567.0000000000001a05/head//3
2014-06-12 14:51:28.547827 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9ed8a82e/rbd_data.10a46b8b4567.0000000000001f73/head//3, auth == maps.end()
2014-06-12 14:51:28.547843 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9ed8a82e/rbd_data.10a46b8b4567.0000000000001f73/head//3
2014-06-12 14:51:28.547858 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2a09a82e/rbd_data.10a46b8b4567.0000000000002a75/head//3, auth == maps.end()
2014-06-12 14:51:28.547874 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2a09a82e/rbd_data.10a46b8b4567.0000000000002a75/head//3
2014-06-12 14:51:28.547892 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 935ca82e/rbd_data.10ac6b8b4567.0000000000003be6/head//3, auth == maps.end()
2014-06-12 14:51:28.547906 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 935ca82e/rbd_data.10ac6b8b4567.0000000000003be6/head//3
2014-06-12 14:51:28.547923 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9f7ca82e/rbd_data.10966b8b4567.0000000000001163/head//3, auth == maps.end()
2014-06-12 14:51:28.547937 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9f7ca82e/rbd_data.10966b8b4567.0000000000001163/head//3
2014-06-12 14:51:28.547954 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4d3da82e/rbd_data.10a26b8b4567.00000000000031a1/head//3, auth == maps.end()
2014-06-12 14:51:28.547968 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4d3da82e/rbd_data.10a26b8b4567.00000000000031a1/head//3
2014-06-12 14:51:28.547985 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8bada82e/rbd_data.10b16b8b4567.0000000000002fa6/head//3, auth == maps.end()
2014-06-12 14:51:28.547999 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8bada82e/rbd_data.10b16b8b4567.0000000000002fa6/head//3
2014-06-12 14:51:28.548016 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e4eda82e/rbd_data.10a56b8b4567.0000000000002bbf/head//3, auth == maps.end()
2014-06-12 14:51:28.548036 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e4eda82e/rbd_data.10a56b8b4567.0000000000002bbf/head//3
2014-06-12 14:51:28.548051 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b170b82e/rbd_data.10ac6b8b4567.000000000000252a/head//3, auth == maps.end()
2014-06-12 14:51:28.548067 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b170b82e/rbd_data.10ac6b8b4567.000000000000252a/head//3
2014-06-12 14:51:28.548084 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4e63b82e/rbd_data.108e6b8b4567.0000000000000174/head//3, auth == maps.end()
2014-06-12 14:51:28.548100 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4e63b82e/rbd_data.108e6b8b4567.0000000000000174/head//3
2014-06-12 14:51:28.548118 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d624b82e/rbd_data.108e6b8b4567.0000000000003ad9/head//3, auth == maps.end()
2014-06-12 14:51:28.548134 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d624b82e/rbd_data.108e6b8b4567.0000000000003ad9/head//3
2014-06-12 14:51:28.548151 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ecd5b82e/rbd_data.109e6b8b4567.0000000000001e6e/head//3, auth == maps.end()
2014-06-12 14:51:28.548167 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ecd5b82e/rbd_data.109e6b8b4567.0000000000001e6e/head//3
2014-06-12 14:51:28.548185 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a528b82e/rbd_data.10916b8b4567.00000000000020ba/head//3, auth == maps.end()
2014-06-12 14:51:28.548201 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a528b82e/rbd_data.10916b8b4567.00000000000020ba/head//3
2014-06-12 14:51:28.548219 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6048b82e/rbd_data.10af6b8b4567.0000000000002973/head//3, auth == maps.end()
2014-06-12 14:51:28.548235 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6048b82e/rbd_data.10af6b8b4567.0000000000002973/head//3
2014-06-12 14:51:28.548251 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e999b82e/rbd_data.10906b8b4567.0000000000000e08/head//3, auth == maps.end()
2014-06-12 14:51:28.548267 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e999b82e/rbd_data.10906b8b4567.0000000000000e08/head//3
2014-06-12 14:51:28.548284 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d91cb82e/rbd_data.109b6b8b4567.0000000000000a47/head//3, auth == maps.end()
2014-06-12 14:51:28.548300 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d91cb82e/rbd_data.109b6b8b4567.0000000000000a47/head//3
2014-06-12 14:51:28.548318 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] 
2014-06-12 14:51:28.548330 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub
2014-06-12 14:51:28.548350 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d91cb82e/rbd_data.109b6b8b4567.0000000000000a47/head//3 d91cb82e/rbd_data.109b6b8b4567.0000000000000a47/head//3(173'2662 client.4292.0:132324 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2662)
2014-06-12 14:51:28.548373 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  e999b82e/rbd_data.10906b8b4567.0000000000000e08/head//3 e999b82e/rbd_data.10906b8b4567.0000000000000e08/head//3(173'2750 client.4281.0:139215 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2750)
2014-06-12 14:51:28.548394 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6048b82e/rbd_data.10af6b8b4567.0000000000002973/head//3 6048b82e/rbd_data.10af6b8b4567.0000000000002973/head//3(173'1670 client.4312.0:86431 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1670)
2014-06-12 14:51:28.548415 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a528b82e/rbd_data.10916b8b4567.00000000000020ba/head//3 a528b82e/rbd_data.10916b8b4567.00000000000020ba/head//3(173'1385 client.4282.0:68699 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1385)
2014-06-12 14:51:28.548435 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  ecd5b82e/rbd_data.109e6b8b4567.0000000000001e6e/head//3 ecd5b82e/rbd_data.109e6b8b4567.0000000000001e6e/head//3(173'1294 client.4295.0:65130 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1294)
2014-06-12 14:51:28.548456 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d624b82e/rbd_data.108e6b8b4567.0000000000003ad9/head//3 d624b82e/rbd_data.108e6b8b4567.0000000000003ad9/head//3(173'2258 client.4279.0:115566 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2258)
2014-06-12 14:51:28.548476 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4e63b82e/rbd_data.108e6b8b4567.0000000000000174/head//3 4e63b82e/rbd_data.108e6b8b4567.0000000000000174/head//3(173'2368 client.4279.0:119117 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2368)
2014-06-12 14:51:28.548496 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  b170b82e/rbd_data.10ac6b8b4567.000000000000252a/head//3 b170b82e/rbd_data.10ac6b8b4567.000000000000252a/head//3(173'1507 client.4309.0:74759 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1507)
2014-06-12 14:51:28.548516 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  e4eda82e/rbd_data.10a56b8b4567.0000000000002bbf/head//3 e4eda82e/rbd_data.10a56b8b4567.0000000000002bbf/head//3(173'1777 client.4302.0:89587 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1777)
2014-06-12 14:51:28.548536 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8bada82e/rbd_data.10b16b8b4567.0000000000002fa6/head//3 8bada82e/rbd_data.10b16b8b4567.0000000000002fa6/head//3(173'1921 client.4314.0:96249 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1921)
2014-06-12 14:51:28.548556 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4d3da82e/rbd_data.10a26b8b4567.00000000000031a1/head//3 4d3da82e/rbd_data.10a26b8b4567.00000000000031a1/head//3(173'2027 client.4299.0:101287 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2027)
2014-06-12 14:51:28.548576 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  9f7ca82e/rbd_data.10966b8b4567.0000000000001163/head//3 9f7ca82e/rbd_data.10966b8b4567.0000000000001163/head//3(173'833 client.4287.0:40226 wrlock_by=unknown.0.0:0 dirty s 4194304 uv833)
2014-06-12 14:51:28.548597 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  935ca82e/rbd_data.10ac6b8b4567.0000000000003be6/head//3 935ca82e/rbd_data.10ac6b8b4567.0000000000003be6/head//3(173'2301 client.4309.0:116653 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2301)
2014-06-12 14:51:28.548617 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  2a09a82e/rbd_data.10a46b8b4567.0000000000002a75/head//3 2a09a82e/rbd_data.10a46b8b4567.0000000000002a75/head//3(173'1703 client.4301.0:86383 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1703)
2014-06-12 14:51:28.548637 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  9ed8a82e/rbd_data.10a46b8b4567.0000000000001f73/head//3 9ed8a82e/rbd_data.10a46b8b4567.0000000000001f73/head//3(173'1299 client.4301.0:66453 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1299)
2014-06-12 14:51:28.548661 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a7b7a82e/rbd_data.10a66b8b4567.0000000000001a05/head//3 a7b7a82e/rbd_data.10a66b8b4567.0000000000001a05/head//3(173'1149 client.4303.0:55099 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1149)
2014-06-12 14:51:28.548681 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  33f6a82e/rbd_data.109b6b8b4567.0000000000001e44/head//3 33f6a82e/rbd_data.109b6b8b4567.0000000000001e44/head//3(173'1312 client.4292.0:62065 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1312)
2014-06-12 14:51:28.548701 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d676a82e/rbd_data.10a46b8b4567.00000000000000c0/head//3 d676a82e/rbd_data.10a46b8b4567.00000000000000c0/head//3(173'2351 client.4301.0:119700 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2351)
2014-06-12 14:51:28.548721 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  975a82e/rbd_data.109e6b8b4567.0000000000003135/head//3 975a82e/rbd_data.109e6b8b4567.0000000000003135/head//3(173'1875 client.4295.0:98706 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1875)
2014-06-12 14:51:28.548741 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  5494a82e/rbd_data.10ae6b8b4567.0000000000000458/head//3 5494a82e/rbd_data.10ae6b8b4567.0000000000000458/head//3(173'2511 client.4311.0:121427 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2511)
2014-06-12 14:51:28.548761 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  3c9d982e/rbd_data.10b06b8b4567.00000000000035b4/head//3 3c9d982e/rbd_data.10b06b8b4567.00000000000035b4/head//3(173'2042 client.4313.0:108024 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2042)
2014-06-12 14:51:28.548781 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  9fcc982e/rbd_data.10916b8b4567.0000000000002b8f/head//3 9fcc982e/rbd_data.10916b8b4567.0000000000002b8f/head//3(173'1730 client.4282.0:88508 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1730)
2014-06-12 14:51:28.548801 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  70ba982e/rbd_data.108d6b8b4567.000000000000171e/head//3 70ba982e/rbd_data.108d6b8b4567.000000000000171e/head//3(173'1013 client.4278.0:52305 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1013)
2014-06-12 14:51:28.548821 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a69a982e/rbd_data.10996b8b4567.00000000000032e8/head//3 a69a982e/rbd_data.10996b8b4567.00000000000032e8/head//3(173'1967 client.4290.0:100245 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1967)
2014-06-12 14:51:28.548839 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4628982e/rbd_data.10966b8b4567.00000000000015af/head//3 4628982e/rbd_data.10966b8b4567.00000000000015af/head//3(173'994 client.4287.0:47773 wrlock_by=unknown.0.0:0 dirty s 4194304 uv994)
2014-06-12 14:51:28.548855 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:28.548871 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:28.548885 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:28.548888 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:28.548890 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:28.548944 7f9d60bd2700 10 collection_list_partial 3.2e_head start:9600c82e//0//-1 is_max:0
2014-06-12 14:51:28.549252 7f9d60bd2700 20 objects: [3fe0c82e/rbd_data.10916b8b4567.00000000000026b0/head//3,6c61c82e/rbd_data.10a16b8b4567.0000000000001491/head//3,c113c82e/rbd_data.108d6b8b4567.000000000000042f/head//3,86a4c82e/rbd_data.10b06b8b4567.0000000000000c85/head//3,4da5c82e/rbd_data.10a36b8b4567.0000000000000bf1/head//3,ea09c82e/rbd_data.108e6b8b4567.0000000000001715/head//3,bf6ac82e/rbd_data.109e6b8b4567.0000000000002de0/head//3,6e9bc82e/rbd_data.10916b8b4567.0000000000001c90/head//3,f26ec82e/rbd_data.10b36b8b4567.0000000000002676/head//3,ee5fc82e/rbd_data.10956b8b4567.0000000000000295/head//3,f221d82e/rbd_data.10a36b8b4567.00000000000034c8/head//3,bbb2d82e/rbd_data.10af6b8b4567.0000000000002776/head//3,b453d82e/rbd_data.10aa6b8b4567.00000000000022ff/head//3,cf85d82e/rbd_data.10956b8b4567.0000000000000ea0/head//3,f176d82e/rbd_data.10b36b8b4567.000000000000316b/head//3,d45cd82e/rbd_data.10906b8b4567.0000000000003269/head//3,2302e82e/rbd_data.10ad6b8b4567.000000000000077c/head//3,9ec2e82e/rbd_data.109f6b8b4567.00000000000017db/head//3,b656e82e/rbd_data.108d6b8b4567.0000000000000f21/head//3,1158e82e/rbd_data.10b26b8b4567.000000000000189d/head//3,1a1de82e/rbd_data.10aa6b8b4567.000000000000137e/head//3,a98ee82e/rbd_data.10966b8b4567.0000000000001fcb/head//3,e651f82e/rbd_data.10b46b8b4567.000000000000014d/head//3,7402f82e/rbd_data.10b46b8b4567.00000000000024ae/head//3,8d93f82e/rbd_data.109c6b8b4567.000000000000333e/head//3]
2014-06-12 14:51:28.549806 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:28.549825 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:28.549839 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:28.549846 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:28.549858 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:28.549864 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:28.549868 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_map_chunk [9600c82e//0//-1,8d27f82e//0//-1)
2014-06-12 14:51:28.549874 7f9d60bd2700 10 collection_list_partial 3.2e_head start:9600c82e//0//-1 is_max:0
2014-06-12 14:51:28.550063 7f9d60bd2700 20 objects: [3fe0c82e/rbd_data.10916b8b4567.00000000000026b0/head//3,6c61c82e/rbd_data.10a16b8b4567.0000000000001491/head//3,c113c82e/rbd_data.108d6b8b4567.000000000000042f/head//3,86a4c82e/rbd_data.10b06b8b4567.0000000000000c85/head//3,4da5c82e/rbd_data.10a36b8b4567.0000000000000bf1/head//3,ea09c82e/rbd_data.108e6b8b4567.0000000000001715/head//3,bf6ac82e/rbd_data.109e6b8b4567.0000000000002de0/head//3,6e9bc82e/rbd_data.10916b8b4567.0000000000001c90/head//3,f26ec82e/rbd_data.10b36b8b4567.0000000000002676/head//3,ee5fc82e/rbd_data.10956b8b4567.0000000000000295/head//3,f221d82e/rbd_data.10a36b8b4567.00000000000034c8/head//3,bbb2d82e/rbd_data.10af6b8b4567.0000000000002776/head//3,b453d82e/rbd_data.10aa6b8b4567.00000000000022ff/head//3,cf85d82e/rbd_data.10956b8b4567.0000000000000ea0/head//3,f176d82e/rbd_data.10b36b8b4567.000000000000316b/head//3,d45cd82e/rbd_data.10906b8b4567.0000000000003269/head//3,2302e82e/rbd_data.10ad6b8b4567.000000000000077c/head//3,9ec2e82e/rbd_data.109f6b8b4567.00000000000017db/head//3,b656e82e/rbd_data.108d6b8b4567.0000000000000f21/head//3,1158e82e/rbd_data.10b26b8b4567.000000000000189d/head//3,1a1de82e/rbd_data.10aa6b8b4567.000000000000137e/head//3,a98ee82e/rbd_data.10966b8b4567.0000000000001fcb/head//3,e651f82e/rbd_data.10b46b8b4567.000000000000014d/head//3,7402f82e/rbd_data.10b46b8b4567.00000000000024ae/head//3,8d93f82e/rbd_data.109c6b8b4567.000000000000333e/head//3,8d27f82e/rbd_data.10936b8b4567.0000000000000235/head//3,8178f82e/rbd_data.10986b8b4567.0000000000001cd9/head//3,d3c9f82e/rbd_data.109e6b8b4567.0000000000002ace/head//3,6f5cf82e/rbd_data.10b36b8b4567.00000000000025f2/head//3,fa0df82e/rbd_data.10a66b8b4567.000000000000257d/head//3,536ef82e/rbd_data.10996b8b4567.0000000000002d9f/head//3,ddeef82e/rbd_data.10a86b8b4567.0000000000001509/head//3,f44ff82e/rbd_data.10aa6b8b4567.000000000000272a/head//3]
2014-06-12 14:51:28.550104 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:28.550116 7f9d60bd2700 10 stat 3.2e_head/3fe0c82e/rbd_data.10916b8b4567.00000000000026b0/head//3
2014-06-12 14:51:28.550211 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 3fe0c82e/rbd_data.10916b8b4567.00000000000026b0/head//3
2014-06-12 14:51:28.581280 7f9d60bd2700 10 getattrs 3.2e_head/3fe0c82e/rbd_data.10916b8b4567.00000000000026b0/head//3 = 0
2014-06-12 14:51:28.581287 7f9d60bd2700 10 stat 3.2e_head/6c61c82e/rbd_data.10a16b8b4567.0000000000001491/head//3
2014-06-12 14:51:28.581384 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6c61c82e/rbd_data.10a16b8b4567.0000000000001491/head//3
2014-06-12 14:51:28.607554 7f9d60bd2700 10 getattrs 3.2e_head/6c61c82e/rbd_data.10a16b8b4567.0000000000001491/head//3 = 0
2014-06-12 14:51:28.607565 7f9d60bd2700 10 stat 3.2e_head/c113c82e/rbd_data.108d6b8b4567.000000000000042f/head//3
2014-06-12 14:51:28.607665 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid c113c82e/rbd_data.108d6b8b4567.000000000000042f/head//3
2014-06-12 14:51:28.656738 7f9d60bd2700 10 getattrs 3.2e_head/c113c82e/rbd_data.108d6b8b4567.000000000000042f/head//3 = 0
2014-06-12 14:51:28.656750 7f9d60bd2700 10 stat 3.2e_head/86a4c82e/rbd_data.10b06b8b4567.0000000000000c85/head//3
2014-06-12 14:51:28.656853 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 86a4c82e/rbd_data.10b06b8b4567.0000000000000c85/head//3
2014-06-12 14:51:28.661262 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:28.664815 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:28.692709 7f9d60bd2700 10 getattrs 3.2e_head/86a4c82e/rbd_data.10b06b8b4567.0000000000000c85/head//3 = 0
2014-06-12 14:51:28.692729 7f9d60bd2700 10 stat 3.2e_head/4da5c82e/rbd_data.10a36b8b4567.0000000000000bf1/head//3
2014-06-12 14:51:28.692845 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 4da5c82e/rbd_data.10a36b8b4567.0000000000000bf1/head//3
2014-06-12 14:51:28.700206 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:28.702163 7f9d60bd2700 10 getattrs 3.2e_head/4da5c82e/rbd_data.10a36b8b4567.0000000000000bf1/head//3 = 0
2014-06-12 14:51:28.702184 7f9d60bd2700 10 stat 3.2e_head/ea09c82e/rbd_data.108e6b8b4567.0000000000001715/head//3
2014-06-12 14:51:28.702301 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid ea09c82e/rbd_data.108e6b8b4567.0000000000001715/head//3
2014-06-12 14:51:28.724213 7f9d60bd2700 10 getattrs 3.2e_head/ea09c82e/rbd_data.108e6b8b4567.0000000000001715/head//3 = 0
2014-06-12 14:51:28.724225 7f9d60bd2700 10 stat 3.2e_head/bf6ac82e/rbd_data.109e6b8b4567.0000000000002de0/head//3
2014-06-12 14:51:28.724341 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid bf6ac82e/rbd_data.109e6b8b4567.0000000000002de0/head//3
2014-06-12 14:51:28.743518 7f9d60bd2700 10 getattrs 3.2e_head/bf6ac82e/rbd_data.109e6b8b4567.0000000000002de0/head//3 = 0
2014-06-12 14:51:28.743531 7f9d60bd2700 10 stat 3.2e_head/6e9bc82e/rbd_data.10916b8b4567.0000000000001c90/head//3
2014-06-12 14:51:28.743648 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6e9bc82e/rbd_data.10916b8b4567.0000000000001c90/head//3
2014-06-12 14:51:28.767566 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:28.768847 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:28.768882 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:28.772159 7f9d60bd2700 10 getattrs 3.2e_head/6e9bc82e/rbd_data.10916b8b4567.0000000000001c90/head//3 = 0
2014-06-12 14:51:28.772170 7f9d60bd2700 10 stat 3.2e_head/f26ec82e/rbd_data.10b36b8b4567.0000000000002676/head//3
2014-06-12 14:51:28.772298 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f26ec82e/rbd_data.10b36b8b4567.0000000000002676/head//3
2014-06-12 14:51:28.783213 7f9d60bd2700 10 getattrs 3.2e_head/f26ec82e/rbd_data.10b36b8b4567.0000000000002676/head//3 = 0
2014-06-12 14:51:28.783226 7f9d60bd2700 10 stat 3.2e_head/ee5fc82e/rbd_data.10956b8b4567.0000000000000295/head//3
2014-06-12 14:51:28.783352 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid ee5fc82e/rbd_data.10956b8b4567.0000000000000295/head//3
2014-06-12 14:51:28.801512 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:28.806820 7f9d60bd2700 10 getattrs 3.2e_head/ee5fc82e/rbd_data.10956b8b4567.0000000000000295/head//3 = 0
2014-06-12 14:51:28.806832 7f9d60bd2700 10 stat 3.2e_head/f221d82e/rbd_data.10a36b8b4567.00000000000034c8/head//3
2014-06-12 14:51:28.806960 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f221d82e/rbd_data.10a36b8b4567.00000000000034c8/head//3
2014-06-12 14:51:28.807968 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:28.820612 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:28.820659 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:28.838357 7f9d60bd2700 10 getattrs 3.2e_head/f221d82e/rbd_data.10a36b8b4567.00000000000034c8/head//3 = 0
2014-06-12 14:51:28.838368 7f9d60bd2700 10 stat 3.2e_head/bbb2d82e/rbd_data.10af6b8b4567.0000000000002776/head//3
2014-06-12 14:51:28.838482 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid bbb2d82e/rbd_data.10af6b8b4567.0000000000002776/head//3
2014-06-12 14:51:28.854192 7f9d60bd2700 10 getattrs 3.2e_head/bbb2d82e/rbd_data.10af6b8b4567.0000000000002776/head//3 = 0
2014-06-12 14:51:28.854205 7f9d60bd2700 10 stat 3.2e_head/b453d82e/rbd_data.10aa6b8b4567.00000000000022ff/head//3
2014-06-12 14:51:28.854325 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid b453d82e/rbd_data.10aa6b8b4567.00000000000022ff/head//3
2014-06-12 14:51:28.854605 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:28.864971 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:28.868069 7f9d60bd2700 10 getattrs 3.2e_head/b453d82e/rbd_data.10aa6b8b4567.00000000000022ff/head//3 = 0
2014-06-12 14:51:28.868079 7f9d60bd2700 10 stat 3.2e_head/cf85d82e/rbd_data.10956b8b4567.0000000000000ea0/head//3
2014-06-12 14:51:28.868192 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid cf85d82e/rbd_data.10956b8b4567.0000000000000ea0/head//3
2014-06-12 14:51:28.868349 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:28.870116 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:28.870160 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:28.906626 7f9d60bd2700 10 getattrs 3.2e_head/cf85d82e/rbd_data.10956b8b4567.0000000000000ea0/head//3 = 0
2014-06-12 14:51:28.906636 7f9d60bd2700 10 stat 3.2e_head/f176d82e/rbd_data.10b36b8b4567.000000000000316b/head//3
2014-06-12 14:51:28.906752 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f176d82e/rbd_data.10b36b8b4567.000000000000316b/head//3
2014-06-12 14:51:28.928077 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:28.935538 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:28.938013 7f9d60bd2700 10 getattrs 3.2e_head/f176d82e/rbd_data.10b36b8b4567.000000000000316b/head//3 = 0
2014-06-12 14:51:28.938029 7f9d60bd2700 10 stat 3.2e_head/d45cd82e/rbd_data.10906b8b4567.0000000000003269/head//3
2014-06-12 14:51:28.938161 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d45cd82e/rbd_data.10906b8b4567.0000000000003269/head//3
2014-06-12 14:51:28.951184 7f9d60bd2700 10 getattrs 3.2e_head/d45cd82e/rbd_data.10906b8b4567.0000000000003269/head//3 = 0
2014-06-12 14:51:28.951196 7f9d60bd2700 10 stat 3.2e_head/2302e82e/rbd_data.10ad6b8b4567.000000000000077c/head//3
2014-06-12 14:51:28.951325 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 2302e82e/rbd_data.10ad6b8b4567.000000000000077c/head//3
2014-06-12 14:51:28.961915 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:28.965642 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:28.995072 7f9d60bd2700 10 getattrs 3.2e_head/2302e82e/rbd_data.10ad6b8b4567.000000000000077c/head//3 = 0
2014-06-12 14:51:28.995088 7f9d60bd2700 10 stat 3.2e_head/9ec2e82e/rbd_data.109f6b8b4567.00000000000017db/head//3
2014-06-12 14:51:28.995241 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 9ec2e82e/rbd_data.109f6b8b4567.00000000000017db/head//3
2014-06-12 14:51:29.009731 7f9d60bd2700 10 getattrs 3.2e_head/9ec2e82e/rbd_data.109f6b8b4567.00000000000017db/head//3 = 0
2014-06-12 14:51:29.009740 7f9d60bd2700 10 stat 3.2e_head/b656e82e/rbd_data.108d6b8b4567.0000000000000f21/head//3
2014-06-12 14:51:29.009850 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid b656e82e/rbd_data.108d6b8b4567.0000000000000f21/head//3
2014-06-12 14:51:29.046209 7f9d60bd2700 10 getattrs 3.2e_head/b656e82e/rbd_data.108d6b8b4567.0000000000000f21/head//3 = 0
2014-06-12 14:51:29.046230 7f9d60bd2700 10 stat 3.2e_head/1158e82e/rbd_data.10b26b8b4567.000000000000189d/head//3
2014-06-12 14:51:29.046356 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 1158e82e/rbd_data.10b26b8b4567.000000000000189d/head//3
2014-06-12 14:51:29.068704 7f9d60bd2700 10 getattrs 3.2e_head/1158e82e/rbd_data.10b26b8b4567.000000000000189d/head//3 = 0
2014-06-12 14:51:29.068715 7f9d60bd2700 10 stat 3.2e_head/1a1de82e/rbd_data.10aa6b8b4567.000000000000137e/head//3
2014-06-12 14:51:29.068842 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 1a1de82e/rbd_data.10aa6b8b4567.000000000000137e/head//3
2014-06-12 14:51:29.099177 7f9d60bd2700 10 getattrs 3.2e_head/1a1de82e/rbd_data.10aa6b8b4567.000000000000137e/head//3 = 0
2014-06-12 14:51:29.099189 7f9d60bd2700 10 stat 3.2e_head/a98ee82e/rbd_data.10966b8b4567.0000000000001fcb/head//3
2014-06-12 14:51:29.099319 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid a98ee82e/rbd_data.10966b8b4567.0000000000001fcb/head//3
2014-06-12 14:51:29.120892 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:29.126422 7f9d60bd2700 10 getattrs 3.2e_head/a98ee82e/rbd_data.10966b8b4567.0000000000001fcb/head//3 = 0
2014-06-12 14:51:29.126429 7f9d60bd2700 10 stat 3.2e_head/e651f82e/rbd_data.10b46b8b4567.000000000000014d/head//3
2014-06-12 14:51:29.126507 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid e651f82e/rbd_data.10b46b8b4567.000000000000014d/head//3
2014-06-12 14:51:29.132552 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:29.142005 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:29.151406 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:29.164962 7f9d60bd2700 10 getattrs 3.2e_head/e651f82e/rbd_data.10b46b8b4567.000000000000014d/head//3 = 0
2014-06-12 14:51:29.164974 7f9d60bd2700 10 stat 3.2e_head/7402f82e/rbd_data.10b46b8b4567.00000000000024ae/head//3
2014-06-12 14:51:29.165092 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 7402f82e/rbd_data.10b46b8b4567.00000000000024ae/head//3
2014-06-12 14:51:29.177262 7f9d60bd2700 10 getattrs 3.2e_head/7402f82e/rbd_data.10b46b8b4567.00000000000024ae/head//3 = 0
2014-06-12 14:51:29.177272 7f9d60bd2700 10 stat 3.2e_head/8d93f82e/rbd_data.109c6b8b4567.000000000000333e/head//3
2014-06-12 14:51:29.177383 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8d93f82e/rbd_data.109c6b8b4567.000000000000333e/head//3
2014-06-12 14:51:29.179880 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:29.194099 7f9d60bd2700 10 getattrs 3.2e_head/8d93f82e/rbd_data.109c6b8b4567.000000000000333e/head//3 = 0
2014-06-12 14:51:29.194111 7f9d60bd2700 10 collection_getattrs 3.2e_head
2014-06-12 14:51:29.194169 7f9d60bd2700 10 collection_getattrs 3.2e_head = 0
2014-06-12 14:51:29.194176 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:29.194215 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:29.194224 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:29.194234 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:29.194237 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:29.218125 7f9d44101700  5 -- op tracker -- , seq: 748579, time: 2014-06-12 14:51:29.218003, event: header_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.218148 7f9d44101700  5 -- op tracker -- , seq: 748579, time: 2014-06-12 14:51:29.218006, event: throttled, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.218186 7f9d44101700  5 -- op tracker -- , seq: 748579, time: 2014-06-12 14:51:29.218076, event: all_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.218198 7f9d44101700  5 -- op tracker -- , seq: 748579, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.218204 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:29.218213 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:29.218225 7f9d44101700 15 osd.2 173 enqueue_op 0x329eb600 prio 127 cost 10717 latency 0.000221 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:29.218297 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:29.218319 7f9d623d5700 10 osd.2 173 dequeue_op 0x329eb600 prio 127 cost 10717 latency 0.000315 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]
2014-06-12 14:51:29.218355 7f9d623d5700  5 -- op tracker -- , seq: 748579, time: 2014-06-12 14:51:29.218349, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.218373 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] handle_message: 0x329eb600
2014-06-12 14:51:29.218389 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:29.218406 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:29.218423 7f9d623d5700  5 -- op tracker -- , seq: 748579, time: 2014-06-12 14:51:29.218417, event: started, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.218430 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:29.218535 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] map version is 173'2847
2014-06-12 14:51:29.218555 7f9d623d5700 10 osd.2 173 dequeue_op 0x329eb600 finish
2014-06-12 14:51:29.218566 7f9d623d5700  5 -- op tracker -- , seq: 748579, time: 2014-06-12 14:51:29.218560, event: done, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.218587 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:29.218592 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:29.218595 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:29.218609 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:29.218631 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:29.218644 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:29.218688 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:29.218701 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:29.218714 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:29.218751 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3fe0c82e/rbd_data.10916b8b4567.00000000000026b0/head//3, auth == maps.end()
2014-06-12 14:51:29.218787 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3fe0c82e/rbd_data.10916b8b4567.00000000000026b0/head//3
2014-06-12 14:51:29.218819 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6c61c82e/rbd_data.10a16b8b4567.0000000000001491/head//3, auth == maps.end()
2014-06-12 14:51:29.218840 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6c61c82e/rbd_data.10a16b8b4567.0000000000001491/head//3
2014-06-12 14:51:29.218859 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c113c82e/rbd_data.108d6b8b4567.000000000000042f/head//3, auth == maps.end()
2014-06-12 14:51:29.218879 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c113c82e/rbd_data.108d6b8b4567.000000000000042f/head//3
2014-06-12 14:51:29.218897 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 86a4c82e/rbd_data.10b06b8b4567.0000000000000c85/head//3, auth == maps.end()
2014-06-12 14:51:29.218913 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 86a4c82e/rbd_data.10b06b8b4567.0000000000000c85/head//3
2014-06-12 14:51:29.218930 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4da5c82e/rbd_data.10a36b8b4567.0000000000000bf1/head//3, auth == maps.end()
2014-06-12 14:51:29.218946 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4da5c82e/rbd_data.10a36b8b4567.0000000000000bf1/head//3
2014-06-12 14:51:29.218964 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ea09c82e/rbd_data.108e6b8b4567.0000000000001715/head//3, auth == maps.end()
2014-06-12 14:51:29.218980 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ea09c82e/rbd_data.108e6b8b4567.0000000000001715/head//3
2014-06-12 14:51:29.218997 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bf6ac82e/rbd_data.109e6b8b4567.0000000000002de0/head//3, auth == maps.end()
2014-06-12 14:51:29.219012 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj bf6ac82e/rbd_data.109e6b8b4567.0000000000002de0/head//3
2014-06-12 14:51:29.219029 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6e9bc82e/rbd_data.10916b8b4567.0000000000001c90/head//3, auth == maps.end()
2014-06-12 14:51:29.219048 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6e9bc82e/rbd_data.10916b8b4567.0000000000001c90/head//3
2014-06-12 14:51:29.219066 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f26ec82e/rbd_data.10b36b8b4567.0000000000002676/head//3, auth == maps.end()
2014-06-12 14:51:29.219082 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f26ec82e/rbd_data.10b36b8b4567.0000000000002676/head//3
2014-06-12 14:51:29.219099 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ee5fc82e/rbd_data.10956b8b4567.0000000000000295/head//3, auth == maps.end()
2014-06-12 14:51:29.219115 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ee5fc82e/rbd_data.10956b8b4567.0000000000000295/head//3
2014-06-12 14:51:29.219132 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f221d82e/rbd_data.10a36b8b4567.00000000000034c8/head//3, auth == maps.end()
2014-06-12 14:51:29.219148 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f221d82e/rbd_data.10a36b8b4567.00000000000034c8/head//3
2014-06-12 14:51:29.219165 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bbb2d82e/rbd_data.10af6b8b4567.0000000000002776/head//3, auth == maps.end()
2014-06-12 14:51:29.219184 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj bbb2d82e/rbd_data.10af6b8b4567.0000000000002776/head//3
2014-06-12 14:51:29.219204 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b453d82e/rbd_data.10aa6b8b4567.00000000000022ff/head//3, auth == maps.end()
2014-06-12 14:51:29.219221 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b453d82e/rbd_data.10aa6b8b4567.00000000000022ff/head//3
2014-06-12 14:51:29.219241 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cf85d82e/rbd_data.10956b8b4567.0000000000000ea0/head//3, auth == maps.end()
2014-06-12 14:51:29.219258 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj cf85d82e/rbd_data.10956b8b4567.0000000000000ea0/head//3
2014-06-12 14:51:29.219277 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f176d82e/rbd_data.10b36b8b4567.000000000000316b/head//3, auth == maps.end()
2014-06-12 14:51:29.219294 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f176d82e/rbd_data.10b36b8b4567.000000000000316b/head//3
2014-06-12 14:51:29.219312 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d45cd82e/rbd_data.10906b8b4567.0000000000003269/head//3, auth == maps.end()
2014-06-12 14:51:29.219329 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d45cd82e/rbd_data.10906b8b4567.0000000000003269/head//3
2014-06-12 14:51:29.219346 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2302e82e/rbd_data.10ad6b8b4567.000000000000077c/head//3, auth == maps.end()
2014-06-12 14:51:29.219366 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2302e82e/rbd_data.10ad6b8b4567.000000000000077c/head//3
2014-06-12 14:51:29.219383 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9ec2e82e/rbd_data.109f6b8b4567.00000000000017db/head//3, auth == maps.end()
2014-06-12 14:51:29.219402 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9ec2e82e/rbd_data.109f6b8b4567.00000000000017db/head//3
2014-06-12 14:51:29.219421 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b656e82e/rbd_data.108d6b8b4567.0000000000000f21/head//3, auth == maps.end()
2014-06-12 14:51:29.219440 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b656e82e/rbd_data.108d6b8b4567.0000000000000f21/head//3
2014-06-12 14:51:29.219461 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1158e82e/rbd_data.10b26b8b4567.000000000000189d/head//3, auth == maps.end()
2014-06-12 14:51:29.219480 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1158e82e/rbd_data.10b26b8b4567.000000000000189d/head//3
2014-06-12 14:51:29.219500 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1a1de82e/rbd_data.10aa6b8b4567.000000000000137e/head//3, auth == maps.end()
2014-06-12 14:51:29.219519 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1a1de82e/rbd_data.10aa6b8b4567.000000000000137e/head//3
2014-06-12 14:51:29.219537 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a98ee82e/rbd_data.10966b8b4567.0000000000001fcb/head//3, auth == maps.end()
2014-06-12 14:51:29.219556 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a98ee82e/rbd_data.10966b8b4567.0000000000001fcb/head//3
2014-06-12 14:51:29.219576 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e651f82e/rbd_data.10b46b8b4567.000000000000014d/head//3, auth == maps.end()
2014-06-12 14:51:29.219596 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e651f82e/rbd_data.10b46b8b4567.000000000000014d/head//3
2014-06-12 14:51:29.219613 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7402f82e/rbd_data.10b46b8b4567.00000000000024ae/head//3, auth == maps.end()
2014-06-12 14:51:29.219632 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7402f82e/rbd_data.10b46b8b4567.00000000000024ae/head//3
2014-06-12 14:51:29.219653 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8d93f82e/rbd_data.109c6b8b4567.000000000000333e/head//3, auth == maps.end()
2014-06-12 14:51:29.219672 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8d93f82e/rbd_data.109c6b8b4567.000000000000333e/head//3
2014-06-12 14:51:29.219694 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] 
2014-06-12 14:51:29.219710 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub
2014-06-12 14:51:29.219732 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8d93f82e/rbd_data.109c6b8b4567.000000000000333e/head//3 8d93f82e/rbd_data.109c6b8b4567.000000000000333e/head//3(173'2009 client.4293.0:102650 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2009)
2014-06-12 14:51:29.219760 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  7402f82e/rbd_data.10b46b8b4567.00000000000024ae/head//3 7402f82e/rbd_data.10b46b8b4567.00000000000024ae/head//3(173'1618 client.4317.0:75381 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1618)
2014-06-12 14:51:29.219784 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  e651f82e/rbd_data.10b46b8b4567.000000000000014d/head//3 e651f82e/rbd_data.10b46b8b4567.000000000000014d/head//3(173'2390 client.4317.0:118769 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2390)
2014-06-12 14:51:29.219809 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  a98ee82e/rbd_data.10966b8b4567.0000000000001fcb/head//3 a98ee82e/rbd_data.10966b8b4567.0000000000001fcb/head//3(173'1277 client.4287.0:64466 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1277)
2014-06-12 14:51:29.219833 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  1a1de82e/rbd_data.10aa6b8b4567.000000000000137e/head//3 1a1de82e/rbd_data.10aa6b8b4567.000000000000137e/head//3(173'964 client.4307.0:47337 wrlock_by=unknown.0.0:0 dirty s 4194304 uv964)
2014-06-12 14:51:29.219857 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  1158e82e/rbd_data.10b26b8b4567.000000000000189d/head//3 1158e82e/rbd_data.10b26b8b4567.000000000000189d/head//3(173'1049 client.4315.0:53663 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1049)
2014-06-12 14:51:29.219881 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  b656e82e/rbd_data.108d6b8b4567.0000000000000f21/head//3 b656e82e/rbd_data.108d6b8b4567.0000000000000f21/head//3(173'2793 client.4278.0:142688 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2793)
2014-06-12 14:51:29.219905 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  9ec2e82e/rbd_data.109f6b8b4567.00000000000017db/head//3 9ec2e82e/rbd_data.109f6b8b4567.00000000000017db/head//3(173'1021 client.4296.0:51701 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1021)
2014-06-12 14:51:29.219929 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  2302e82e/rbd_data.10ad6b8b4567.000000000000077c/head//3 2302e82e/rbd_data.10ad6b8b4567.000000000000077c/head//3(173'2600 client.4310.0:129181 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2600)
2014-06-12 14:51:29.219953 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d45cd82e/rbd_data.10906b8b4567.0000000000003269/head//3 d45cd82e/rbd_data.10906b8b4567.0000000000003269/head//3(173'1849 client.4281.0:100222 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1849)
2014-06-12 14:51:29.219977 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f176d82e/rbd_data.10b36b8b4567.000000000000316b/head//3 f176d82e/rbd_data.10b36b8b4567.000000000000316b/head//3(173'1895 client.4316.0:99613 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1895)
2014-06-12 14:51:29.220001 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  cf85d82e/rbd_data.10956b8b4567.0000000000000ea0/head//3 cf85d82e/rbd_data.10956b8b4567.0000000000000ea0/head//3(173'2795 client.4286.0:141743 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2795)
2014-06-12 14:51:29.220026 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  b453d82e/rbd_data.10aa6b8b4567.00000000000022ff/head//3 b453d82e/rbd_data.10aa6b8b4567.00000000000022ff/head//3(173'1411 client.4307.0:74683 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1411)
2014-06-12 14:51:29.220049 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  bbb2d82e/rbd_data.10af6b8b4567.0000000000002776/head//3 bbb2d82e/rbd_data.10af6b8b4567.0000000000002776/head//3(173'1589 client.4312.0:82704 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1589)
2014-06-12 14:51:29.220073 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f221d82e/rbd_data.10a36b8b4567.00000000000034c8/head//3 f221d82e/rbd_data.10a36b8b4567.00000000000034c8/head//3(173'1959 client.4300.0:102883 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1959)
2014-06-12 14:51:29.220101 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  ee5fc82e/rbd_data.10956b8b4567.0000000000000295/head//3 ee5fc82e/rbd_data.10956b8b4567.0000000000000295/head//3(173'2435 client.4286.0:121691 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2435)
2014-06-12 14:51:29.220125 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f26ec82e/rbd_data.10b36b8b4567.0000000000002676/head//3 f26ec82e/rbd_data.10b36b8b4567.0000000000002676/head//3(173'1576 client.4316.0:79773 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1576)
2014-06-12 14:51:29.220149 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6e9bc82e/rbd_data.10916b8b4567.0000000000001c90/head//3 6e9bc82e/rbd_data.10916b8b4567.0000000000001c90/head//3(173'1270 client.4282.0:61419 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1270)
2014-06-12 14:51:29.220173 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  bf6ac82e/rbd_data.109e6b8b4567.0000000000002de0/head//3 bf6ac82e/rbd_data.109e6b8b4567.0000000000002de0/head//3(173'1750 client.4295.0:92762 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1750)
2014-06-12 14:51:29.220197 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  ea09c82e/rbd_data.108e6b8b4567.0000000000001715/head//3 ea09c82e/rbd_data.108e6b8b4567.0000000000001715/head//3(173'1035 client.4279.0:51349 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1035)
2014-06-12 14:51:29.220221 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  4da5c82e/rbd_data.10a36b8b4567.0000000000000bf1/head//3 4da5c82e/rbd_data.10a36b8b4567.0000000000000bf1/head//3(173'2745 client.4300.0:134960 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2745)
2014-06-12 14:51:29.220244 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  86a4c82e/rbd_data.10b06b8b4567.0000000000000c85/head//3 86a4c82e/rbd_data.10b06b8b4567.0000000000000c85/head//3(173'2735 client.4313.0:141049 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2735)
2014-06-12 14:51:29.220268 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  c113c82e/rbd_data.108d6b8b4567.000000000000042f/head//3 c113c82e/rbd_data.108d6b8b4567.000000000000042f/head//3(173'2474 client.4278.0:124225 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2474)
2014-06-12 14:51:29.220291 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6c61c82e/rbd_data.10a16b8b4567.0000000000001491/head//3 6c61c82e/rbd_data.10a16b8b4567.0000000000001491/head//3(173'931 client.4298.0:47597 wrlock_by=unknown.0.0:0 dirty s 4194304 uv931)
2014-06-12 14:51:29.220315 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  3fe0c82e/rbd_data.10916b8b4567.00000000000026b0/head//3 3fe0c82e/rbd_data.10916b8b4567.00000000000026b0/head//3(173'1609 client.4282.0:79477 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1609)
2014-06-12 14:51:29.220335 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:29.220355 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:29.220371 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:29.220375 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:29.220378 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:29.220433 7f9d60bd2700 10 collection_list_partial 3.2e_head start:8d27f82e//0//-1 is_max:0
2014-06-12 14:51:29.220616 7f9d60bd2700 20 objects: [8178f82e/rbd_data.10986b8b4567.0000000000001cd9/head//3,d3c9f82e/rbd_data.109e6b8b4567.0000000000002ace/head//3,6f5cf82e/rbd_data.10b36b8b4567.00000000000025f2/head//3,fa0df82e/rbd_data.10a66b8b4567.000000000000257d/head//3,536ef82e/rbd_data.10996b8b4567.0000000000002d9f/head//3,ddeef82e/rbd_data.10a86b8b4567.0000000000001509/head//3,f44ff82e/rbd_data.10aa6b8b4567.000000000000272a/head//3]
2014-06-12 14:51:29.221129 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:29.221169 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:29.221196 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:29.221210 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:29.221221 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:29.221231 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:29.221242 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_map_chunk [8d27f82e//0//-1,MAX)
2014-06-12 14:51:29.221255 7f9d60bd2700 10 collection_list_partial 3.2e_head start:8d27f82e//0//-1 is_max:0
2014-06-12 14:51:29.221384 7f9d60bd2700 20 objects: [8178f82e/rbd_data.10986b8b4567.0000000000001cd9/head//3,d3c9f82e/rbd_data.109e6b8b4567.0000000000002ace/head//3,6f5cf82e/rbd_data.10b36b8b4567.00000000000025f2/head//3,fa0df82e/rbd_data.10a66b8b4567.000000000000257d/head//3,536ef82e/rbd_data.10996b8b4567.0000000000002d9f/head//3,ddeef82e/rbd_data.10a86b8b4567.0000000000001509/head//3,f44ff82e/rbd_data.10aa6b8b4567.000000000000272a/head//3]
2014-06-12 14:51:29.221438 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scan_list scanning 7 objects
2014-06-12 14:51:29.221458 7f9d60bd2700 10 stat 3.2e_head/8178f82e/rbd_data.10986b8b4567.0000000000001cd9/head//3
2014-06-12 14:51:29.221679 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 8178f82e/rbd_data.10986b8b4567.0000000000001cd9/head//3
2014-06-12 14:51:29.221872 7f9d60bd2700 10 getattrs 3.2e_head/8178f82e/rbd_data.10986b8b4567.0000000000001cd9/head//3 = 0
2014-06-12 14:51:29.221886 7f9d60bd2700 10 stat 3.2e_head/d3c9f82e/rbd_data.109e6b8b4567.0000000000002ace/head//3
2014-06-12 14:51:29.222058 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid d3c9f82e/rbd_data.109e6b8b4567.0000000000002ace/head//3
2014-06-12 14:51:29.243329 7f9d60bd2700 10 getattrs 3.2e_head/d3c9f82e/rbd_data.109e6b8b4567.0000000000002ace/head//3 = 0
2014-06-12 14:51:29.243338 7f9d60bd2700 10 stat 3.2e_head/6f5cf82e/rbd_data.10b36b8b4567.00000000000025f2/head//3
2014-06-12 14:51:29.243438 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 6f5cf82e/rbd_data.10b36b8b4567.00000000000025f2/head//3
2014-06-12 14:51:29.243531 7f9d60bd2700 10 getattrs 3.2e_head/6f5cf82e/rbd_data.10b36b8b4567.00000000000025f2/head//3 = 0
2014-06-12 14:51:29.243536 7f9d60bd2700 10 stat 3.2e_head/fa0df82e/rbd_data.10a66b8b4567.000000000000257d/head//3
2014-06-12 14:51:29.243623 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid fa0df82e/rbd_data.10a66b8b4567.000000000000257d/head//3
2014-06-12 14:51:29.270151 7f9d60bd2700 10 getattrs 3.2e_head/fa0df82e/rbd_data.10a66b8b4567.000000000000257d/head//3 = 0
2014-06-12 14:51:29.270159 7f9d60bd2700 10 stat 3.2e_head/536ef82e/rbd_data.10996b8b4567.0000000000002d9f/head//3
2014-06-12 14:51:29.270250 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid 536ef82e/rbd_data.10996b8b4567.0000000000002d9f/head//3
2014-06-12 14:51:29.283403 7f9d60bd2700 10 getattrs 3.2e_head/536ef82e/rbd_data.10996b8b4567.0000000000002d9f/head//3 = 0
2014-06-12 14:51:29.283411 7f9d60bd2700 10 stat 3.2e_head/ddeef82e/rbd_data.10a86b8b4567.0000000000001509/head//3
2014-06-12 14:51:29.283504 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid ddeef82e/rbd_data.10a86b8b4567.0000000000001509/head//3
2014-06-12 14:51:29.298041 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:29.304699 7f9d60bd2700 10 getattrs 3.2e_head/ddeef82e/rbd_data.10a86b8b4567.0000000000001509/head//3 = 0
2014-06-12 14:51:29.304709 7f9d60bd2700 10 stat 3.2e_head/f44ff82e/rbd_data.10aa6b8b4567.000000000000272a/head//3
2014-06-12 14:51:29.304815 7f9d60bd2700 10 lookup_strip_header done  cid 3.2e_head oid f44ff82e/rbd_data.10aa6b8b4567.000000000000272a/head//3
2014-06-12 14:51:29.328656 7f9d60bd2700 10 getattrs 3.2e_head/f44ff82e/rbd_data.10aa6b8b4567.000000000000272a/head//3 = 0
2014-06-12 14:51:29.328665 7f9d60bd2700 10 collection_getattrs 3.2e_head
2014-06-12 14:51:29.328723 7f9d60bd2700 10 collection_getattrs 3.2e_head = 0
2014-06-12 14:51:29.328727 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:29.328759 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:29.328766 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:29.328773 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:29.328775 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:29.341240 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:29.343691 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:29.345590 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:29.345638 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:29.345643 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:29.345645 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:29.345661 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:29.358589 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:29.358858 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:29.372963 7f9d44101700  5 -- op tracker -- , seq: 748580, time: 2014-06-12 14:51:29.372865, event: header_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.372982 7f9d44101700  5 -- op tracker -- , seq: 748580, time: 2014-06-12 14:51:29.372867, event: throttled, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.372993 7f9d44101700  5 -- op tracker -- , seq: 748580, time: 2014-06-12 14:51:29.372915, event: all_read, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.373027 7f9d44101700  5 -- op tracker -- , seq: 748580, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.373034 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:29.373043 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:29.373053 7f9d44101700 15 osd.2 173 enqueue_op 0x329ebc00 prio 127 cost 3031 latency 0.000187 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:29.373120 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:29.373141 7f9d68be2700 10 osd.2 173 dequeue_op 0x329ebc00 prio 127 cost 3031 latency 0.000276 osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]
2014-06-12 14:51:29.373179 7f9d68be2700  5 -- op tracker -- , seq: 748580, time: 2014-06-12 14:51:29.373172, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.373201 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] handle_message: 0x329ebc00
2014-06-12 14:51:29.373217 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:29.373239 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:29.373257 7f9d68be2700  5 -- op tracker -- , seq: 748580, time: 2014-06-12 14:51:29.373252, event: started, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.373265 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:29.373308 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] map version is 173'2847
2014-06-12 14:51:29.373328 7f9d68be2700 10 osd.2 173 dequeue_op 0x329ebc00 finish
2014-06-12 14:51:29.373338 7f9d68be2700  5 -- op tracker -- , seq: 748580, time: 2014-06-12 14:51:29.373333, event: done, op: osd_sub_op(unknown.0.0:0 3.2e 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:29.373358 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:29.373362 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:29.373368 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d72000 (1 active)
2014-06-12 14:51:29.373381 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:29.373403 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:29.373416 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:29.373435 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:29.373448 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub   osd.2 has 7 items
2014-06-12 14:51:29.373460 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub replica 21 has 7 items
2014-06-12 14:51:29.373482 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8178f82e/rbd_data.10986b8b4567.0000000000001cd9/head//3, auth == maps.end()
2014-06-12 14:51:29.373504 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8178f82e/rbd_data.10986b8b4567.0000000000001cd9/head//3
2014-06-12 14:51:29.373530 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d3c9f82e/rbd_data.109e6b8b4567.0000000000002ace/head//3, auth == maps.end()
2014-06-12 14:51:29.373547 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d3c9f82e/rbd_data.109e6b8b4567.0000000000002ace/head//3
2014-06-12 14:51:29.373564 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6f5cf82e/rbd_data.10b36b8b4567.00000000000025f2/head//3, auth == maps.end()
2014-06-12 14:51:29.373581 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6f5cf82e/rbd_data.10b36b8b4567.00000000000025f2/head//3
2014-06-12 14:51:29.373598 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fa0df82e/rbd_data.10a66b8b4567.000000000000257d/head//3, auth == maps.end()
2014-06-12 14:51:29.373639 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj fa0df82e/rbd_data.10a66b8b4567.000000000000257d/head//3
2014-06-12 14:51:29.373664 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 536ef82e/rbd_data.10996b8b4567.0000000000002d9f/head//3, auth == maps.end()
2014-06-12 14:51:29.373682 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 536ef82e/rbd_data.10996b8b4567.0000000000002d9f/head//3
2014-06-12 14:51:29.373699 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ddeef82e/rbd_data.10a86b8b4567.0000000000001509/head//3, auth == maps.end()
2014-06-12 14:51:29.373715 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ddeef82e/rbd_data.10a86b8b4567.0000000000001509/head//3
2014-06-12 14:51:29.373732 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f44ff82e/rbd_data.10aa6b8b4567.000000000000272a/head//3, auth == maps.end()
2014-06-12 14:51:29.373749 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f44ff82e/rbd_data.10aa6b8b4567.000000000000272a/head//3
2014-06-12 14:51:29.373766 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] 
2014-06-12 14:51:29.373787 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub
2014-06-12 14:51:29.373809 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  f44ff82e/rbd_data.10aa6b8b4567.000000000000272a/head//3 f44ff82e/rbd_data.10aa6b8b4567.000000000000272a/head//3(173'1567 client.4307.0:81963 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1567)
2014-06-12 14:51:29.373836 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  ddeef82e/rbd_data.10a86b8b4567.0000000000001509/head//3 ddeef82e/rbd_data.10a86b8b4567.0000000000001509/head//3(173'920 client.4305.0:46923 wrlock_by=unknown.0.0:0 dirty s 4194304 uv920)
2014-06-12 14:51:29.373859 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  536ef82e/rbd_data.10996b8b4567.0000000000002d9f/head//3 536ef82e/rbd_data.10996b8b4567.0000000000002d9f/head//3(173'1794 client.4290.0:90676 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1794)
2014-06-12 14:51:29.373881 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  fa0df82e/rbd_data.10a66b8b4567.000000000000257d/head//3 fa0df82e/rbd_data.10a66b8b4567.000000000000257d/head//3(173'1516 client.4303.0:75306 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1516)
2014-06-12 14:51:29.373901 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  6f5cf82e/rbd_data.10b36b8b4567.00000000000025f2/head//3 6f5cf82e/rbd_data.10b36b8b4567.00000000000025f2/head//3(173'1534 client.4316.0:78931 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1534)
2014-06-12 14:51:29.373925 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  d3c9f82e/rbd_data.109e6b8b4567.0000000000002ace/head//3 d3c9f82e/rbd_data.109e6b8b4567.0000000000002ace/head//3(173'1696 client.4295.0:87328 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1696)
2014-06-12 14:51:29.373946 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub  8178f82e/rbd_data.10986b8b4567.0000000000001cd9/head//3 8178f82e/rbd_data.10986b8b4567.0000000000001cd9/head//3(173'1229 client.4289.0:60374 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1229)
2014-06-12 14:51:29.373965 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:29.373982 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:29.373993 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub state FINISH
2014-06-12 14:51:29.374005 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] scrub got 282/293 objects, 0/0 clones, 282/293 dirty, 0/0 omap, 0/0 hit_set_archive, 1182793728/1228931072 bytes.
2014-06-12 14:51:29.374027 7f9d60bd2700  0 log [ERR] : 3.2e scrub stat mismatch, got 282/293 objects, 0/0 clones, 282/293 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts, 1182793728/1228931072 bytes.
2014-06-12 14:51:29.374032 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+scrubbing] process_inconsistent() checking authoritative
2014-06-12 14:51:29.374046 7f9d60bd2700  0 log [ERR] : 3.2e scrub 1 errors
2014-06-12 14:51:29.374118 7f9d60bd2700 10 log is not dirty
2014-06-12 14:51:29.374127 7f9d60bd2700  5 queue_transactions existing osr(3.2e 0x276cd30)/0x276cd30
2014-06-12 14:51:29.374132 7f9d60bd2700  5 queue_transactions (trailing journal)  
2014-06-12 14:51:29.374134 7f9d60bd2700  5 queue_op 0x6775e00 seq 2574 osr(3.2e 0x276cd30) 826 bytes   (queue has 1 ops and 826 bytes)
2014-06-12 14:51:29.374145 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+inconsistent] publish_stats_to_osd 173:3613
2014-06-12 14:51:29.374162 7f9d60bd2700 20 osd.2 173 dec_scrubs_active 1 -> 0 (max 1, pending 0)
2014-06-12 14:51:29.374165 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+inconsistent]  requeue_ops 
2014-06-12 14:51:29.374179 7f9d8096e700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x227f7a0 (1 active)
2014-06-12 14:51:29.374185 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+inconsistent] scrub requesting unreserve from osd.21
2014-06-12 14:51:29.374194 7f9d8096e700  5 _do_op 0x6775e00 seq 2574 osr(3.2e 0x276cd30)/0x276cd30 start
2014-06-12 14:51:29.374201 7f9d8096e700 10 _do_transaction on 0x515f8c0
2014-06-12 14:51:29.374209 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:29.374212 7f9d8096e700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:51:29.374254 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.2e( v 173'2847 (0'0,173'2847] local-les=139 n=293 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2846 mlcod 173'2846 active+clean+inconsistent] share_pg_info
2014-06-12 14:51:29.374305 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:29.374335 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d72000 (0 active)
2014-06-12 14:51:29.374338 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:29.374362 7f9d8096e700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:51:29.374415 7f9d8096e700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:51:29.374441 7f9d8096e700  5 submit_transaction r = 0
2014-06-12 14:51:29.374467 7f9d8096e700 10 _do_op 0x6775e00 seq 2574 r = 0, finisher 0x49b5780 0
2014-06-12 14:51:29.374473 7f9d8096e700 10 _finish_op 0x6775e00 seq 2574 osr(3.2e 0x276cd30)/0x276cd30
2014-06-12 14:51:29.374482 7f9d8096e700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x227f7a0 (0 active)
2014-06-12 14:51:29.374486 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:29.384045 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:29.384526 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:29.512966 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:29.546902 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:29.962293 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:29.976626 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:29.994001 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:30.064357 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:30.167134 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:30.293898 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:30.294014 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:30.316554 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:30.326049 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:30.345736 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:30.345760 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:30.345765 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:30.345767 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:30.345788 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:30.365954 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:30.370778 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:30.381589 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:30.469543 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:30.470067 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:30.512807 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:30.518626 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:30.522548 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:30.532000 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:30.547434 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:30.661358 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:30.664901 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:30.700280 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:30.767663 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:30.775710 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:30.775827 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:30.801606 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:30.808052 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:30.854702 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:30.865065 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:30.868443 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:30.928160 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:30.935609 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:30.962014 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:30.965726 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:31.084518 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:31.084599 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:31.120978 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:31.132648 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:31.142080 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:31.151501 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:31.179961 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:31.218681 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:31.298124 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:31.341320 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:31.343791 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:31.345840 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:31.345886 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.46 < max 0.5 = yes
2014-06-12 14:51:31.345901 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=1
2014-06-12 14:51:31.345910 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:31.345907 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3a at 2014-06-11 13:02:32.500954
2014-06-12 14:51:31.345920 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 -> 1 (max 1, active 0)
2014-06-12 14:51:31.345923 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean] sched_scrub: reserved locally, reserving replicas
2014-06-12 14:51:31.345946 7f9d84175700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean] scrub requesting reserve from osd.21
2014-06-12 14:51:31.345966 7f9d84175700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:31.345994 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean] sched_scrub: reserved 2, waiting for replicas
2014-06-12 14:51:31.346009 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:31.346012 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:31.346013 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:31.346831 7f9d44101700  5 -- op tracker -- , seq: 748581, time: 2014-06-12 14:51:31.346767, event: header_read, op: osd_sub_op_reply(unknown.0.0:0 3.3a 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:31.346848 7f9d44101700  5 -- op tracker -- , seq: 748581, time: 2014-06-12 14:51:31.346769, event: throttled, op: osd_sub_op_reply(unknown.0.0:0 3.3a 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:31.346858 7f9d44101700  5 -- op tracker -- , seq: 748581, time: 2014-06-12 14:51:31.346788, event: all_read, op: osd_sub_op_reply(unknown.0.0:0 3.3a 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:31.346889 7f9d44101700  5 -- op tracker -- , seq: 748581, time: 0.000000, event: dispatched, op: osd_sub_op_reply(unknown.0.0:0 3.3a 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:31.346896 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op_reply(unknown.0.0:0 3.3a 0//0//3 [scrub-reserve] ack, result = 0) v2 epoch 173
2014-06-12 14:51:31.346903 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:31.346912 7f9d44101700 15 osd.2 173 enqueue_op 0x8dbce00 prio 127 cost 1 latency 0.000144 osd_sub_op_reply(unknown.0.0:0 3.3a 0//0//3 [scrub-reserve] ack, result = 0) v2
2014-06-12 14:51:31.346979 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:31.347000 7f9d68be2700 10 osd.2 173 dequeue_op 0x8dbce00 prio 127 cost 1 latency 0.000233 osd_sub_op_reply(unknown.0.0:0 3.3a 0//0//3 [scrub-reserve] ack, result = 0) v2 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean]
2014-06-12 14:51:31.347037 7f9d68be2700  5 -- op tracker -- , seq: 748581, time: 2014-06-12 14:51:31.347031, event: reached_pg, op: osd_sub_op_reply(unknown.0.0:0 3.3a 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:31.347059 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean] handle_message: 0x8dbce00
2014-06-12 14:51:31.347075 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean] sub_op_scrub_reserve_reply
2014-06-12 14:51:31.347091 7f9d68be2700  5 -- op tracker -- , seq: 748581, time: 2014-06-12 14:51:31.347087, event: started, op: osd_sub_op_reply(unknown.0.0:0 3.3a 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:31.347106 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean]  osd.21 scrub reserve = success
2014-06-12 14:51:31.347120 7f9d68be2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean] sched_scrub: success, reserved self and replicas
2014-06-12 14:51:31.347135 7f9d68be2700 10 osd.2 173 dequeue_op 0x8dbce00 finish
2014-06-12 14:51:31.347144 7f9d68be2700  5 -- op tracker -- , seq: 748581, time: 2014-06-12 14:51:31.347140, event: done, op: osd_sub_op_reply(unknown.0.0:0 3.3a 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:31.347162 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:31.347166 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:31.347174 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:31.347193 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] starting a new chunky scrub
2014-06-12 14:51:31.347221 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state INACTIVE
2014-06-12 14:51:31.347234 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub start
2014-06-12 14:51:31.347249 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] publish_stats_to_osd 173:3848
2014-06-12 14:51:31.347262 7f9d60bd2700 20 osd.2 173 inc_scrubs_active 0 -> 1 (max 1, pending 1 -> 0)
2014-06-12 14:51:31.347267 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:31.347284 7f9d60bd2700 10 collection_list_partial 3.3a_head start:0//0//-1 is_max:0
2014-06-12 14:51:31.348054 7f9d60bd2700 20 objects: [c8a1003a/rbd_data.10b16b8b4567.0000000000003682/head//3,2422003a/rbd_data.109c6b8b4567.00000000000004b6/head//3,e95003a/rbd_data.10ac6b8b4567.00000000000017a6/head//3,1a27003a/rbd_data.109b6b8b4567.0000000000003010/head//3,7e38003a/rbd_data.10a66b8b4567.00000000000035d1/head//3,4578003a/rbd_data.10936b8b4567.00000000000012d3/head//3,9399003a/rbd_data.10916b8b4567.0000000000002ab0/head//3,99bd003a/rbd_data.109c6b8b4567.0000000000000403/head//3,81f003a/rbd_data.109f6b8b4567.000000000000105e/head//3,f8bf003a/rbd_data.10966b8b4567.0000000000000ef8/head//3,8a60103a/rbd_data.109a6b8b4567.0000000000001171/head//3,ee80103a/rbd_data.10a06b8b4567.0000000000002434/head//3,393103a/rbd_data.10946b8b4567.0000000000000416/head//3,7a95103a/rbd_data.10ae6b8b4567.00000000000013de/head//3,4776103a/rbd_data.10b16b8b4567.00000000000029b0/head//3,f196103a/rbd_data.10986b8b4567.0000000000002b10/head//3,9617103a/rbd_data.10ad6b8b4567.00000000000033cd/head//3,1c68103a/rbd_data.10aa6b8b4567.0000000000001e7d/head//3,4c8a103a/rbd_data.10b26b8b4567.0000000000001f10/head//3,16ad103a/rbd_data.10966b8b4567.0000000000000fe3/head//3,d57f103a/rbd_data.10b36b8b4567.0000000000003670/head//3,b121203a/rbd_data.10b16b8b4567.0000000000000cb8/head//3,eb82203a/rbd_data.109d6b8b4567.0000000000003b4c/head//3,9d15203a/rbd_data.10936b8b4567.000000000000278a/head//3,f40b203a/rbd_data.10a96b8b4567.000000000000217a/head//3]
2014-06-12 14:51:31.348777 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:31.348817 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:31.348842 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:31.348856 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:31.348867 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:31.348877 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:31.348888 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_map_chunk [0//0//-1,6c2b203a//0//-1)
2014-06-12 14:51:31.348900 7f9d60bd2700 10 collection_list_partial 3.3a_head start:0//0//-1 is_max:0
2014-06-12 14:51:31.358138 7f9d60bd2700 20 objects: [c8a1003a/rbd_data.10b16b8b4567.0000000000003682/head//3,2422003a/rbd_data.109c6b8b4567.00000000000004b6/head//3,e95003a/rbd_data.10ac6b8b4567.00000000000017a6/head//3,1a27003a/rbd_data.109b6b8b4567.0000000000003010/head//3,7e38003a/rbd_data.10a66b8b4567.00000000000035d1/head//3,4578003a/rbd_data.10936b8b4567.00000000000012d3/head//3,9399003a/rbd_data.10916b8b4567.0000000000002ab0/head//3,99bd003a/rbd_data.109c6b8b4567.0000000000000403/head//3,81f003a/rbd_data.109f6b8b4567.000000000000105e/head//3,f8bf003a/rbd_data.10966b8b4567.0000000000000ef8/head//3,8a60103a/rbd_data.109a6b8b4567.0000000000001171/head//3,ee80103a/rbd_data.10a06b8b4567.0000000000002434/head//3,393103a/rbd_data.10946b8b4567.0000000000000416/head//3,7a95103a/rbd_data.10ae6b8b4567.00000000000013de/head//3,4776103a/rbd_data.10b16b8b4567.00000000000029b0/head//3,f196103a/rbd_data.10986b8b4567.0000000000002b10/head//3,9617103a/rbd_data.10ad6b8b4567.00000000000033cd/head//3,1c68103a/rbd_data.10aa6b8b4567.0000000000001e7d/head//3,4c8a103a/rbd_data.10b26b8b4567.0000000000001f10/head//3,16ad103a/rbd_data.10966b8b4567.0000000000000fe3/head//3,d57f103a/rbd_data.10b36b8b4567.0000000000003670/head//3,b121203a/rbd_data.10b16b8b4567.0000000000000cb8/head//3,eb82203a/rbd_data.109d6b8b4567.0000000000003b4c/head//3,9d15203a/rbd_data.10936b8b4567.000000000000278a/head//3,f40b203a/rbd_data.10a96b8b4567.000000000000217a/head//3,6c2b203a/rbd_data.10b26b8b4567.0000000000000d09/head//3,367e203a/rbd_data.10ad6b8b4567.00000000000026c8/head//3,2871303a/rbd_data.10976b8b4567.0000000000003aa4/head//3,2281303a/rbd_data.10af6b8b4567.000000000000162f/head//3,6f12303a/rbd_data.10a96b8b4567.0000000000000a14/head//3,9f63303a/rbd_data.10b36b8b4567.0000000000001f6c/head//3,c175303a/rbd_data.108f6b8b4567.0000000000000919/head//3,4a18303a/rbd_data.10aa6b8b4567.0000000000000af6/head//3,ff18303a/rbd_data.109c6b8b4567.000000000000047a/head//3,4309303a/rbd_data.10a06b8b4567.0000000000002853/head//3,baa303a/rbd_data.10986b8b4567.0000000000000434/head//3,bf9b303a/rbd_data.108e6b8b4567.0000000000000734/head//3,968d303a/rbd_data.10b16b8b4567.0000000000001655/head//3,7d3e303a/rbd_data.10a26b8b4567.0000000000001a2b/head//3,c46e303a/rbd_data.109c6b8b4567.000000000000202f/head//3,10b3403a/rbd_data.10b36b8b4567.000000000000238f/head//3,834403a/rbd_data.10a16b8b4567.0000000000002a65/head//3,c844403a/rbd_data.108f6b8b4567.00000000000022ea/head//3,2705403a/rbd_data.109e6b8b4567.0000000000002190/head//3,e8b6403a/rbd_data.10996b8b4567.0000000000001126/head//3,9e07403a/rbd_data.10a36b8b4567.0000000000000c5c/head//3,a177403a/rbd_data.109a6b8b4567.00000000000004b2/head//3,d12a403a/rbd_data.109e6b8b4567.0000000000001cf9/head//3,515a403a/rbd_data.108d6b8b4567.000000000000323a/head//3,aadb403a/rbd_data.10906b8b4567.0000000000002bd7/head//3,3df403a/rbd_data.109e6b8b4567.0000000000000668/head//3,6b41503a/rbd_data.10a96b8b4567.0000000000001d10/head//3,8c22503a/rbd_data.109c6b8b4567.00000000000034b8/head//3,eff2503a/rbd_data.109b6b8b4567.0000000000003770/head//3,6bea503a/rbd_data.10946b8b4567.0000000000002b6e/head//3,adbd503a/rbd_data.10b36b8b4567.00000000000004d1/head//3,44ef503a/rbd_data.10a46b8b4567.0000000000000ab0/head//3,d0e5603a/rbd_data.108e6b8b4567.000000000000158f/head//3,abc6603a/rbd_data.10ab6b8b4567.00000000000025d5/head//3,607603a/rbd_data.10ab6b8b4567.000000000000382f/head//3,5d87603a/rbd_data.10b16b8b4567.0000000000002596/head//3,ac6b603a/rbd_data.10a66b8b4567.00000000000023cd/head//3,5dcb603a/rbd_data.10956b8b4567.0000000000000693/head//3,c74d603a/rbd_data.10956b8b4567.00000000000007af/head//3]
2014-06-12 14:51:31.358285 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:31.358318 7f9d60bd2700 10 stat 3.3a_head/c8a1003a/rbd_data.10b16b8b4567.0000000000003682/head//3
2014-06-12 14:51:31.358538 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c8a1003a/rbd_data.10b16b8b4567.0000000000003682/head//3
2014-06-12 14:51:31.374557 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:31.377549 7f9d60bd2700 10 getattrs 3.3a_head/c8a1003a/rbd_data.10b16b8b4567.0000000000003682/head//3 = 0
2014-06-12 14:51:31.377574 7f9d60bd2700 10 stat 3.3a_head/2422003a/rbd_data.109c6b8b4567.00000000000004b6/head//3
2014-06-12 14:51:31.377754 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 2422003a/rbd_data.109c6b8b4567.00000000000004b6/head//3
2014-06-12 14:51:31.439491 7f9d60bd2700 10 getattrs 3.3a_head/2422003a/rbd_data.109c6b8b4567.00000000000004b6/head//3 = 0
2014-06-12 14:51:31.439500 7f9d60bd2700 10 stat 3.3a_head/e95003a/rbd_data.10ac6b8b4567.00000000000017a6/head//3
2014-06-12 14:51:31.439609 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid e95003a/rbd_data.10ac6b8b4567.00000000000017a6/head//3
2014-06-12 14:51:31.462105 7f9d60bd2700 10 getattrs 3.3a_head/e95003a/rbd_data.10ac6b8b4567.00000000000017a6/head//3 = 0
2014-06-12 14:51:31.462116 7f9d60bd2700 10 stat 3.3a_head/1a27003a/rbd_data.109b6b8b4567.0000000000003010/head//3
2014-06-12 14:51:31.462221 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 1a27003a/rbd_data.109b6b8b4567.0000000000003010/head//3
2014-06-12 14:51:31.483362 7f9d60bd2700 10 getattrs 3.3a_head/1a27003a/rbd_data.109b6b8b4567.0000000000003010/head//3 = 0
2014-06-12 14:51:31.483374 7f9d60bd2700 10 stat 3.3a_head/7e38003a/rbd_data.10a66b8b4567.00000000000035d1/head//3
2014-06-12 14:51:31.483473 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 7e38003a/rbd_data.10a66b8b4567.00000000000035d1/head//3
2014-06-12 14:51:31.510150 7f9d60bd2700 10 getattrs 3.3a_head/7e38003a/rbd_data.10a66b8b4567.00000000000035d1/head//3 = 0
2014-06-12 14:51:31.510160 7f9d60bd2700 10 stat 3.3a_head/4578003a/rbd_data.10936b8b4567.00000000000012d3/head//3
2014-06-12 14:51:31.510267 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 4578003a/rbd_data.10936b8b4567.00000000000012d3/head//3
2014-06-12 14:51:31.513053 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:31.531682 7f9d60bd2700 10 getattrs 3.3a_head/4578003a/rbd_data.10936b8b4567.00000000000012d3/head//3 = 0
2014-06-12 14:51:31.531693 7f9d60bd2700 10 stat 3.3a_head/9399003a/rbd_data.10916b8b4567.0000000000002ab0/head//3
2014-06-12 14:51:31.531805 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 9399003a/rbd_data.10916b8b4567.0000000000002ab0/head//3
2014-06-12 14:51:31.546987 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:31.548780 7f9d60bd2700 10 getattrs 3.3a_head/9399003a/rbd_data.10916b8b4567.0000000000002ab0/head//3 = 0
2014-06-12 14:51:31.548790 7f9d60bd2700 10 stat 3.3a_head/99bd003a/rbd_data.109c6b8b4567.0000000000000403/head//3
2014-06-12 14:51:31.548901 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 99bd003a/rbd_data.109c6b8b4567.0000000000000403/head//3
2014-06-12 14:51:31.607591 7f9d60bd2700 10 getattrs 3.3a_head/99bd003a/rbd_data.109c6b8b4567.0000000000000403/head//3 = 0
2014-06-12 14:51:31.607608 7f9d60bd2700 10 stat 3.3a_head/81f003a/rbd_data.109f6b8b4567.000000000000105e/head//3
2014-06-12 14:51:31.607763 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 81f003a/rbd_data.109f6b8b4567.000000000000105e/head//3
2014-06-12 14:51:31.629268 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:31.629394 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:31.647381 7f9d60bd2700 10 getattrs 3.3a_head/81f003a/rbd_data.109f6b8b4567.000000000000105e/head//3 = 0
2014-06-12 14:51:31.647391 7f9d60bd2700 10 stat 3.3a_head/f8bf003a/rbd_data.10966b8b4567.0000000000000ef8/head//3
2014-06-12 14:51:31.647483 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid f8bf003a/rbd_data.10966b8b4567.0000000000000ef8/head//3
2014-06-12 14:51:31.692100 7f9d60bd2700 10 getattrs 3.3a_head/f8bf003a/rbd_data.10966b8b4567.0000000000000ef8/head//3 = 0
2014-06-12 14:51:31.692117 7f9d60bd2700 10 stat 3.3a_head/8a60103a/rbd_data.109a6b8b4567.0000000000001171/head//3
2014-06-12 14:51:31.692300 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 8a60103a/rbd_data.109a6b8b4567.0000000000001171/head//3
2014-06-12 14:51:31.709924 7f9d60bd2700 10 getattrs 3.3a_head/8a60103a/rbd_data.109a6b8b4567.0000000000001171/head//3 = 0
2014-06-12 14:51:31.709946 7f9d60bd2700 10 stat 3.3a_head/ee80103a/rbd_data.10a06b8b4567.0000000000002434/head//3
2014-06-12 14:51:31.710085 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ee80103a/rbd_data.10a06b8b4567.0000000000002434/head//3
2014-06-12 14:51:31.730366 7f9d60bd2700 10 getattrs 3.3a_head/ee80103a/rbd_data.10a06b8b4567.0000000000002434/head//3 = 0
2014-06-12 14:51:31.730374 7f9d60bd2700 10 stat 3.3a_head/393103a/rbd_data.10946b8b4567.0000000000000416/head//3
2014-06-12 14:51:31.730466 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 393103a/rbd_data.10946b8b4567.0000000000000416/head//3
2014-06-12 14:51:31.786537 7f9d60bd2700 10 getattrs 3.3a_head/393103a/rbd_data.10946b8b4567.0000000000000416/head//3 = 0
2014-06-12 14:51:31.786556 7f9d60bd2700 10 stat 3.3a_head/7a95103a/rbd_data.10ae6b8b4567.00000000000013de/head//3
2014-06-12 14:51:31.786664 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 7a95103a/rbd_data.10ae6b8b4567.00000000000013de/head//3
2014-06-12 14:51:31.814166 7f9d60bd2700 10 getattrs 3.3a_head/7a95103a/rbd_data.10ae6b8b4567.00000000000013de/head//3 = 0
2014-06-12 14:51:31.814182 7f9d60bd2700 10 stat 3.3a_head/4776103a/rbd_data.10b16b8b4567.00000000000029b0/head//3
2014-06-12 14:51:31.814281 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 4776103a/rbd_data.10b16b8b4567.00000000000029b0/head//3
2014-06-12 14:51:31.831186 7f9d60bd2700 10 getattrs 3.3a_head/4776103a/rbd_data.10b16b8b4567.00000000000029b0/head//3 = 0
2014-06-12 14:51:31.831203 7f9d60bd2700 10 stat 3.3a_head/f196103a/rbd_data.10986b8b4567.0000000000002b10/head//3
2014-06-12 14:51:31.831377 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid f196103a/rbd_data.10986b8b4567.0000000000002b10/head//3
2014-06-12 14:51:31.844777 7f9d60bd2700 10 getattrs 3.3a_head/f196103a/rbd_data.10986b8b4567.0000000000002b10/head//3 = 0
2014-06-12 14:51:31.844791 7f9d60bd2700 10 stat 3.3a_head/9617103a/rbd_data.10ad6b8b4567.00000000000033cd/head//3
2014-06-12 14:51:31.844950 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 9617103a/rbd_data.10ad6b8b4567.00000000000033cd/head//3
2014-06-12 14:51:31.874521 7f9d60bd2700 10 getattrs 3.3a_head/9617103a/rbd_data.10ad6b8b4567.00000000000033cd/head//3 = 0
2014-06-12 14:51:31.874528 7f9d60bd2700 10 stat 3.3a_head/1c68103a/rbd_data.10aa6b8b4567.0000000000001e7d/head//3
2014-06-12 14:51:31.874623 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 1c68103a/rbd_data.10aa6b8b4567.0000000000001e7d/head//3
2014-06-12 14:51:31.908949 7f9d60bd2700 10 getattrs 3.3a_head/1c68103a/rbd_data.10aa6b8b4567.0000000000001e7d/head//3 = 0
2014-06-12 14:51:31.908966 7f9d60bd2700 10 stat 3.3a_head/4c8a103a/rbd_data.10b26b8b4567.0000000000001f10/head//3
2014-06-12 14:51:31.909124 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 4c8a103a/rbd_data.10b26b8b4567.0000000000001f10/head//3
2014-06-12 14:51:31.925321 7f9d60bd2700 10 getattrs 3.3a_head/4c8a103a/rbd_data.10b26b8b4567.0000000000001f10/head//3 = 0
2014-06-12 14:51:31.925328 7f9d60bd2700 10 stat 3.3a_head/16ad103a/rbd_data.10966b8b4567.0000000000000fe3/head//3
2014-06-12 14:51:31.925415 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 16ad103a/rbd_data.10966b8b4567.0000000000000fe3/head//3
2014-06-12 14:51:31.927568 7f9d44101700  5 -- op tracker -- , seq: 748582, time: 2014-06-12 14:51:31.927481, event: header_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:31.927579 7f9d44101700  5 -- op tracker -- , seq: 748582, time: 2014-06-12 14:51:31.927484, event: throttled, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:31.927602 7f9d44101700  5 -- op tracker -- , seq: 748582, time: 2014-06-12 14:51:31.927534, event: all_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:31.927608 7f9d44101700  5 -- op tracker -- , seq: 748582, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:31.927612 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:31.927617 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:31.927624 7f9d44101700 15 osd.2 173 enqueue_op 0x8dbd300 prio 127 cost 10717 latency 0.000142 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:31.927683 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:31.941580 7f9d60bd2700 10 getattrs 3.3a_head/16ad103a/rbd_data.10966b8b4567.0000000000000fe3/head//3 = 0
2014-06-12 14:51:31.941587 7f9d60bd2700 10 stat 3.3a_head/d57f103a/rbd_data.10b36b8b4567.0000000000003670/head//3
2014-06-12 14:51:31.941700 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid d57f103a/rbd_data.10b36b8b4567.0000000000003670/head//3
2014-06-12 14:51:31.968314 7f9d60bd2700 10 getattrs 3.3a_head/d57f103a/rbd_data.10b36b8b4567.0000000000003670/head//3 = 0
2014-06-12 14:51:31.968322 7f9d60bd2700 10 stat 3.3a_head/b121203a/rbd_data.10b16b8b4567.0000000000000cb8/head//3
2014-06-12 14:51:31.968420 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid b121203a/rbd_data.10b16b8b4567.0000000000000cb8/head//3
2014-06-12 14:51:31.976705 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:31.994083 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:31.996509 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:31.996548 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:31.996752 7f9d60bd2700 10 getattrs 3.3a_head/b121203a/rbd_data.10b16b8b4567.0000000000000cb8/head//3 = 0
2014-06-12 14:51:31.996796 7f9d60bd2700 10 stat 3.3a_head/eb82203a/rbd_data.109d6b8b4567.0000000000003b4c/head//3
2014-06-12 14:51:31.996915 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid eb82203a/rbd_data.109d6b8b4567.0000000000003b4c/head//3
2014-06-12 14:51:32.035896 7f9d60bd2700 10 getattrs 3.3a_head/eb82203a/rbd_data.109d6b8b4567.0000000000003b4c/head//3 = 0
2014-06-12 14:51:32.035906 7f9d60bd2700 10 stat 3.3a_head/9d15203a/rbd_data.10936b8b4567.000000000000278a/head//3
2014-06-12 14:51:32.036009 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 9d15203a/rbd_data.10936b8b4567.000000000000278a/head//3
2014-06-12 14:51:32.064058 7f9d60bd2700 10 getattrs 3.3a_head/9d15203a/rbd_data.10936b8b4567.000000000000278a/head//3 = 0
2014-06-12 14:51:32.064070 7f9d60bd2700 10 stat 3.3a_head/f40b203a/rbd_data.10a96b8b4567.000000000000217a/head//3
2014-06-12 14:51:32.064181 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid f40b203a/rbd_data.10a96b8b4567.000000000000217a/head//3
2014-06-12 14:51:32.064426 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:32.075176 7f9d60bd2700 10 getattrs 3.3a_head/f40b203a/rbd_data.10a96b8b4567.000000000000217a/head//3 = 0
2014-06-12 14:51:32.075187 7f9d60bd2700 10 collection_getattrs 3.3a_head
2014-06-12 14:51:32.075243 7f9d60bd2700 10 collection_getattrs 3.3a_head = 0
2014-06-12 14:51:32.075246 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:32.075286 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:32.075294 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:32.075308 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:32.075312 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:32.075341 7f9d633d7700 10 osd.2 173 dequeue_op 0x8dbd300 prio 127 cost 10717 latency 0.147860 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]
2014-06-12 14:51:32.075390 7f9d633d7700  5 -- op tracker -- , seq: 748582, time: 2014-06-12 14:51:32.075383, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:32.075416 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] handle_message: 0x8dbd300
2014-06-12 14:51:32.075433 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:32.075453 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:32.075472 7f9d633d7700  5 -- op tracker -- , seq: 748582, time: 2014-06-12 14:51:32.075467, event: started, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:32.075480 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:32.075590 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] map version is 173'2964
2014-06-12 14:51:32.075610 7f9d633d7700 10 osd.2 173 dequeue_op 0x8dbd300 finish
2014-06-12 14:51:32.075622 7f9d633d7700  5 -- op tracker -- , seq: 748582, time: 2014-06-12 14:51:32.075616, event: done, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:32.075646 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:32.075651 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:32.075648 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:32.075661 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:32.075682 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:32.075701 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:32.075731 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:32.075739 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:32.075746 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:32.075769 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c8a1003a/rbd_data.10b16b8b4567.0000000000003682/head//3, auth == maps.end()
2014-06-12 14:51:32.075793 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c8a1003a/rbd_data.10b16b8b4567.0000000000003682/head//3
2014-06-12 14:51:32.075809 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2422003a/rbd_data.109c6b8b4567.00000000000004b6/head//3, auth == maps.end()
2014-06-12 14:51:32.075820 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2422003a/rbd_data.109c6b8b4567.00000000000004b6/head//3
2014-06-12 14:51:32.075830 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e95003a/rbd_data.10ac6b8b4567.00000000000017a6/head//3, auth == maps.end()
2014-06-12 14:51:32.075840 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e95003a/rbd_data.10ac6b8b4567.00000000000017a6/head//3
2014-06-12 14:51:32.075850 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1a27003a/rbd_data.109b6b8b4567.0000000000003010/head//3, auth == maps.end()
2014-06-12 14:51:32.075860 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1a27003a/rbd_data.109b6b8b4567.0000000000003010/head//3
2014-06-12 14:51:32.075870 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7e38003a/rbd_data.10a66b8b4567.00000000000035d1/head//3, auth == maps.end()
2014-06-12 14:51:32.075879 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7e38003a/rbd_data.10a66b8b4567.00000000000035d1/head//3
2014-06-12 14:51:32.075889 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4578003a/rbd_data.10936b8b4567.00000000000012d3/head//3, auth == maps.end()
2014-06-12 14:51:32.075900 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4578003a/rbd_data.10936b8b4567.00000000000012d3/head//3
2014-06-12 14:51:32.075912 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9399003a/rbd_data.10916b8b4567.0000000000002ab0/head//3, auth == maps.end()
2014-06-12 14:51:32.075924 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9399003a/rbd_data.10916b8b4567.0000000000002ab0/head//3
2014-06-12 14:51:32.075936 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 99bd003a/rbd_data.109c6b8b4567.0000000000000403/head//3, auth == maps.end()
2014-06-12 14:51:32.075946 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 99bd003a/rbd_data.109c6b8b4567.0000000000000403/head//3
2014-06-12 14:51:32.075958 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 81f003a/rbd_data.109f6b8b4567.000000000000105e/head//3, auth == maps.end()
2014-06-12 14:51:32.075969 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 81f003a/rbd_data.109f6b8b4567.000000000000105e/head//3
2014-06-12 14:51:32.075980 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f8bf003a/rbd_data.10966b8b4567.0000000000000ef8/head//3, auth == maps.end()
2014-06-12 14:51:32.075991 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f8bf003a/rbd_data.10966b8b4567.0000000000000ef8/head//3
2014-06-12 14:51:32.076002 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8a60103a/rbd_data.109a6b8b4567.0000000000001171/head//3, auth == maps.end()
2014-06-12 14:51:32.076011 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8a60103a/rbd_data.109a6b8b4567.0000000000001171/head//3
2014-06-12 14:51:32.076023 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ee80103a/rbd_data.10a06b8b4567.0000000000002434/head//3, auth == maps.end()
2014-06-12 14:51:32.076034 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ee80103a/rbd_data.10a06b8b4567.0000000000002434/head//3
2014-06-12 14:51:32.076047 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 393103a/rbd_data.10946b8b4567.0000000000000416/head//3, auth == maps.end()
2014-06-12 14:51:32.076056 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 393103a/rbd_data.10946b8b4567.0000000000000416/head//3
2014-06-12 14:51:32.076068 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7a95103a/rbd_data.10ae6b8b4567.00000000000013de/head//3, auth == maps.end()
2014-06-12 14:51:32.076078 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7a95103a/rbd_data.10ae6b8b4567.00000000000013de/head//3
2014-06-12 14:51:32.076090 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4776103a/rbd_data.10b16b8b4567.00000000000029b0/head//3, auth == maps.end()
2014-06-12 14:51:32.076100 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4776103a/rbd_data.10b16b8b4567.00000000000029b0/head//3
2014-06-12 14:51:32.076112 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f196103a/rbd_data.10986b8b4567.0000000000002b10/head//3, auth == maps.end()
2014-06-12 14:51:32.076123 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f196103a/rbd_data.10986b8b4567.0000000000002b10/head//3
2014-06-12 14:51:32.076134 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9617103a/rbd_data.10ad6b8b4567.00000000000033cd/head//3, auth == maps.end()
2014-06-12 14:51:32.076148 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9617103a/rbd_data.10ad6b8b4567.00000000000033cd/head//3
2014-06-12 14:51:32.076161 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1c68103a/rbd_data.10aa6b8b4567.0000000000001e7d/head//3, auth == maps.end()
2014-06-12 14:51:32.076172 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1c68103a/rbd_data.10aa6b8b4567.0000000000001e7d/head//3
2014-06-12 14:51:32.076183 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4c8a103a/rbd_data.10b26b8b4567.0000000000001f10/head//3, auth == maps.end()
2014-06-12 14:51:32.076194 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4c8a103a/rbd_data.10b26b8b4567.0000000000001f10/head//3
2014-06-12 14:51:32.076205 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 16ad103a/rbd_data.10966b8b4567.0000000000000fe3/head//3, auth == maps.end()
2014-06-12 14:51:32.076216 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 16ad103a/rbd_data.10966b8b4567.0000000000000fe3/head//3
2014-06-12 14:51:32.076229 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d57f103a/rbd_data.10b36b8b4567.0000000000003670/head//3, auth == maps.end()
2014-06-12 14:51:32.076240 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d57f103a/rbd_data.10b36b8b4567.0000000000003670/head//3
2014-06-12 14:51:32.076253 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b121203a/rbd_data.10b16b8b4567.0000000000000cb8/head//3, auth == maps.end()
2014-06-12 14:51:32.076264 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b121203a/rbd_data.10b16b8b4567.0000000000000cb8/head//3
2014-06-12 14:51:32.076275 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj eb82203a/rbd_data.109d6b8b4567.0000000000003b4c/head//3, auth == maps.end()
2014-06-12 14:51:32.076285 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj eb82203a/rbd_data.109d6b8b4567.0000000000003b4c/head//3
2014-06-12 14:51:32.076296 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9d15203a/rbd_data.10936b8b4567.000000000000278a/head//3, auth == maps.end()
2014-06-12 14:51:32.076307 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9d15203a/rbd_data.10936b8b4567.000000000000278a/head//3
2014-06-12 14:51:32.076319 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f40b203a/rbd_data.10a96b8b4567.000000000000217a/head//3, auth == maps.end()
2014-06-12 14:51:32.076331 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f40b203a/rbd_data.10a96b8b4567.000000000000217a/head//3
2014-06-12 14:51:32.076344 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] 
2014-06-12 14:51:32.076353 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub
2014-06-12 14:51:32.076367 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  f40b203a/rbd_data.10a96b8b4567.000000000000217a/head//3 f40b203a/rbd_data.10a96b8b4567.000000000000217a/head//3(173'1611 client.4306.0:71944 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1611)
2014-06-12 14:51:32.076383 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  9d15203a/rbd_data.10936b8b4567.000000000000278a/head//3 9d15203a/rbd_data.10936b8b4567.000000000000278a/head//3(173'1766 client.4284.0:78845 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1766)
2014-06-12 14:51:32.076398 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  eb82203a/rbd_data.109d6b8b4567.0000000000003b4c/head//3 eb82203a/rbd_data.109d6b8b4567.0000000000003b4c/head//3(173'2384 client.4294.0:116030 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2384)
2014-06-12 14:51:32.076413 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  b121203a/rbd_data.10b16b8b4567.0000000000000cb8/head//3 b121203a/rbd_data.10b16b8b4567.0000000000000cb8/head//3(173'2887 client.4314.0:138919 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2887)
2014-06-12 14:51:32.076428 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  d57f103a/rbd_data.10b36b8b4567.0000000000003670/head//3 d57f103a/rbd_data.10b36b8b4567.0000000000003670/head//3(173'2276 client.4316.0:109158 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2276)
2014-06-12 14:51:32.076442 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  16ad103a/rbd_data.10966b8b4567.0000000000000fe3/head//3 16ad103a/rbd_data.10966b8b4567.0000000000000fe3/head//3(173'2951 client.4287.0:140179 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2951)
2014-06-12 14:51:32.076456 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  4c8a103a/rbd_data.10b26b8b4567.0000000000001f10/head//3 4c8a103a/rbd_data.10b26b8b4567.0000000000001f10/head//3(173'1242 client.4315.0:65589 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1242)
2014-06-12 14:51:32.076470 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  1c68103a/rbd_data.10aa6b8b4567.0000000000001e7d/head//3 1c68103a/rbd_data.10aa6b8b4567.0000000000001e7d/head//3(173'1366 client.4307.0:67186 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1366)
2014-06-12 14:51:32.076484 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  9617103a/rbd_data.10ad6b8b4567.00000000000033cd/head//3 9617103a/rbd_data.10ad6b8b4567.00000000000033cd/head//3(173'2166 client.4310.0:102663 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2166)
2014-06-12 14:51:32.076498 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  f196103a/rbd_data.10986b8b4567.0000000000002b10/head//3 f196103a/rbd_data.10986b8b4567.0000000000002b10/head//3(173'1862 client.4289.0:85570 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1862)
2014-06-12 14:51:32.076512 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  4776103a/rbd_data.10b16b8b4567.00000000000029b0/head//3 4776103a/rbd_data.10b16b8b4567.00000000000029b0/head//3(173'1898 client.4314.0:85076 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1898)
2014-06-12 14:51:32.076527 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  7a95103a/rbd_data.10ae6b8b4567.00000000000013de/head//3 7a95103a/rbd_data.10ae6b8b4567.00000000000013de/head//3(173'955 client.4311.0:44838 wrlock_by=unknown.0.0:0 dirty s 4194304 uv955)
2014-06-12 14:51:32.076541 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  393103a/rbd_data.10946b8b4567.0000000000000416/head//3 393103a/rbd_data.10946b8b4567.0000000000000416/head//3(173'2570 client.4285.0:122191 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2570)
2014-06-12 14:51:32.076554 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ee80103a/rbd_data.10a06b8b4567.0000000000002434/head//3 ee80103a/rbd_data.10a06b8b4567.0000000000002434/head//3(173'1507 client.4297.0:73283 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1507)
2014-06-12 14:51:32.076567 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  8a60103a/rbd_data.109a6b8b4567.0000000000001171/head//3 8a60103a/rbd_data.109a6b8b4567.0000000000001171/head//3(173'926 client.4291.0:40581 wrlock_by=unknown.0.0:0 dirty s 4194304 uv926)
2014-06-12 14:51:32.076585 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  f8bf003a/rbd_data.10966b8b4567.0000000000000ef8/head//3 f8bf003a/rbd_data.10966b8b4567.0000000000000ef8/head//3(173'2933 client.4287.0:138586 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2933)
2014-06-12 14:51:32.076599 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  81f003a/rbd_data.109f6b8b4567.000000000000105e/head//3 81f003a/rbd_data.109f6b8b4567.000000000000105e/head//3(173'2940 client.4296.0:141457 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2940)
2014-06-12 14:51:32.076613 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  99bd003a/rbd_data.109c6b8b4567.0000000000000403/head//3 99bd003a/rbd_data.109c6b8b4567.0000000000000403/head//3(173'2586 client.4293.0:123505 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2586)
2014-06-12 14:51:32.076627 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  9399003a/rbd_data.10916b8b4567.0000000000002ab0/head//3 9399003a/rbd_data.10916b8b4567.0000000000002ab0/head//3(173'1919 client.4282.0:86964 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1919)
2014-06-12 14:51:32.076641 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  4578003a/rbd_data.10936b8b4567.00000000000012d3/head//3 4578003a/rbd_data.10936b8b4567.00000000000012d3/head//3(173'971 client.4284.0:41561 wrlock_by=unknown.0.0:0 dirty s 4194304 uv971)
2014-06-12 14:51:32.076655 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  7e38003a/rbd_data.10a66b8b4567.00000000000035d1/head//3 7e38003a/rbd_data.10a66b8b4567.00000000000035d1/head//3(173'2302 client.4303.0:106768 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2302)
2014-06-12 14:51:32.076669 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  1a27003a/rbd_data.109b6b8b4567.0000000000003010/head//3 1a27003a/rbd_data.109b6b8b4567.0000000000003010/head//3(173'2153 client.4292.0:94336 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2153)
2014-06-12 14:51:32.076683 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  e95003a/rbd_data.10ac6b8b4567.00000000000017a6/head//3 e95003a/rbd_data.10ac6b8b4567.00000000000017a6/head//3(173'1105 client.4309.0:51349 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1105)
2014-06-12 14:51:32.076698 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  2422003a/rbd_data.109c6b8b4567.00000000000004b6/head//3 2422003a/rbd_data.109c6b8b4567.00000000000004b6/head//3(173'2623 client.4293.0:124602 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2623)
2014-06-12 14:51:32.076712 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c8a1003a/rbd_data.10b16b8b4567.0000000000003682/head//3 c8a1003a/rbd_data.10b16b8b4567.0000000000003682/head//3(173'2375 client.4314.0:109491 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2375)
2014-06-12 14:51:32.076723 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:32.076735 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:32.076745 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:32.076747 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:32.076749 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:32.076788 7f9d60bd2700 10 collection_list_partial 3.3a_head start:6c2b203a//0//-1 is_max:0
2014-06-12 14:51:32.077026 7f9d60bd2700 20 objects: [367e203a/rbd_data.10ad6b8b4567.00000000000026c8/head//3,2871303a/rbd_data.10976b8b4567.0000000000003aa4/head//3,2281303a/rbd_data.10af6b8b4567.000000000000162f/head//3,6f12303a/rbd_data.10a96b8b4567.0000000000000a14/head//3,9f63303a/rbd_data.10b36b8b4567.0000000000001f6c/head//3,c175303a/rbd_data.108f6b8b4567.0000000000000919/head//3,4a18303a/rbd_data.10aa6b8b4567.0000000000000af6/head//3,ff18303a/rbd_data.109c6b8b4567.000000000000047a/head//3,4309303a/rbd_data.10a06b8b4567.0000000000002853/head//3,baa303a/rbd_data.10986b8b4567.0000000000000434/head//3,bf9b303a/rbd_data.108e6b8b4567.0000000000000734/head//3,968d303a/rbd_data.10b16b8b4567.0000000000001655/head//3,7d3e303a/rbd_data.10a26b8b4567.0000000000001a2b/head//3,c46e303a/rbd_data.109c6b8b4567.000000000000202f/head//3,10b3403a/rbd_data.10b36b8b4567.000000000000238f/head//3,834403a/rbd_data.10a16b8b4567.0000000000002a65/head//3,c844403a/rbd_data.108f6b8b4567.00000000000022ea/head//3,2705403a/rbd_data.109e6b8b4567.0000000000002190/head//3,e8b6403a/rbd_data.10996b8b4567.0000000000001126/head//3,9e07403a/rbd_data.10a36b8b4567.0000000000000c5c/head//3,a177403a/rbd_data.109a6b8b4567.00000000000004b2/head//3,d12a403a/rbd_data.109e6b8b4567.0000000000001cf9/head//3,515a403a/rbd_data.108d6b8b4567.000000000000323a/head//3,aadb403a/rbd_data.10906b8b4567.0000000000002bd7/head//3,3df403a/rbd_data.109e6b8b4567.0000000000000668/head//3]
2014-06-12 14:51:32.077578 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:32.077602 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:32.077637 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:32.077647 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:32.077654 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:32.077660 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:32.077666 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_map_chunk [6c2b203a//0//-1,6b41503a//0//-1)
2014-06-12 14:51:32.077674 7f9d60bd2700 10 collection_list_partial 3.3a_head start:6c2b203a//0//-1 is_max:0
2014-06-12 14:51:32.078207 7f9d60bd2700 20 objects: [367e203a/rbd_data.10ad6b8b4567.00000000000026c8/head//3,2871303a/rbd_data.10976b8b4567.0000000000003aa4/head//3,2281303a/rbd_data.10af6b8b4567.000000000000162f/head//3,6f12303a/rbd_data.10a96b8b4567.0000000000000a14/head//3,9f63303a/rbd_data.10b36b8b4567.0000000000001f6c/head//3,c175303a/rbd_data.108f6b8b4567.0000000000000919/head//3,4a18303a/rbd_data.10aa6b8b4567.0000000000000af6/head//3,ff18303a/rbd_data.109c6b8b4567.000000000000047a/head//3,4309303a/rbd_data.10a06b8b4567.0000000000002853/head//3,baa303a/rbd_data.10986b8b4567.0000000000000434/head//3,bf9b303a/rbd_data.108e6b8b4567.0000000000000734/head//3,968d303a/rbd_data.10b16b8b4567.0000000000001655/head//3,7d3e303a/rbd_data.10a26b8b4567.0000000000001a2b/head//3,c46e303a/rbd_data.109c6b8b4567.000000000000202f/head//3,10b3403a/rbd_data.10b36b8b4567.000000000000238f/head//3,834403a/rbd_data.10a16b8b4567.0000000000002a65/head//3,c844403a/rbd_data.108f6b8b4567.00000000000022ea/head//3,2705403a/rbd_data.109e6b8b4567.0000000000002190/head//3,e8b6403a/rbd_data.10996b8b4567.0000000000001126/head//3,9e07403a/rbd_data.10a36b8b4567.0000000000000c5c/head//3,a177403a/rbd_data.109a6b8b4567.00000000000004b2/head//3,d12a403a/rbd_data.109e6b8b4567.0000000000001cf9/head//3,515a403a/rbd_data.108d6b8b4567.000000000000323a/head//3,aadb403a/rbd_data.10906b8b4567.0000000000002bd7/head//3,3df403a/rbd_data.109e6b8b4567.0000000000000668/head//3,6b41503a/rbd_data.10a96b8b4567.0000000000001d10/head//3,8c22503a/rbd_data.109c6b8b4567.00000000000034b8/head//3,eff2503a/rbd_data.109b6b8b4567.0000000000003770/head//3,6bea503a/rbd_data.10946b8b4567.0000000000002b6e/head//3,adbd503a/rbd_data.10b36b8b4567.00000000000004d1/head//3,44ef503a/rbd_data.10a46b8b4567.0000000000000ab0/head//3,d0e5603a/rbd_data.108e6b8b4567.000000000000158f/head//3,abc6603a/rbd_data.10ab6b8b4567.00000000000025d5/head//3,607603a/rbd_data.10ab6b8b4567.000000000000382f/head//3,5d87603a/rbd_data.10b16b8b4567.0000000000002596/head//3,ac6b603a/rbd_data.10a66b8b4567.00000000000023cd/head//3,5dcb603a/rbd_data.10956b8b4567.0000000000000693/head//3,c74d603a/rbd_data.10956b8b4567.00000000000007af/head//3,b1ce603a/rbd_data.10986b8b4567.0000000000001252/head//3,e61703a/rbd_data.109b6b8b4567.0000000000002864/head//3,4373703a/rbd_data.10aa6b8b4567.0000000000002317/head//3,d6a3703a/rbd_data.10a76b8b4567.00000000000016a6/head//3,4c47703a/rbd_data.10946b8b4567.0000000000003536/head//3,7d97703a/rbd_data.109d6b8b4567.000000000000212b/head//3,4659703a/rbd_data.10946b8b4567.00000000000034d1/head//3,eb7a703a/rbd_data.10ab6b8b4567.0000000000002ee5/head//3,37ea703a/rbd_data.10a46b8b4567.0000000000003307/head//3,8380803a/rbd_data.10966b8b4567.0000000000001ac4/head//3,c3e0803a/rbd_data.10a26b8b4567.00000000000006af/head//3,7681803a/rbd_data.10b36b8b4567.0000000000000b52/head//3,802803a/rbd_data.10a06b8b4567.0000000000000a8e/head//3,ef72803a/rbd_data.10936b8b4567.0000000000001d81/head//3,a825803a/rbd_data.10956b8b4567.0000000000000185/head//3,9a06803a/rbd_data.10a26b8b4567.000000000000395b/head//3,6036803a/rbd_data.10946b8b4567.0000000000002208/head//3,dd38803a/rbd_data.10986b8b4567.000000000000160f/head//3,cc68803a/rbd_data.10b26b8b4567.000000000000313d/head//3,39f8803a/rbd_data.109b6b8b4567.0000000000001e5e/head//3,a4ca803a/rbd_data.10b06b8b4567.0000000000001477/head//3,569b803a/rbd_data.10a66b8b4567.0000000000002682/head//3,963d803a/rbd_data.10a36b8b4567.0000000000001019/head//3,94cf803a/rbd_data.10926b8b4567.00000000000029f2/head//3,faef803a/rbd_data.10906b8b4567.00000000000016bb/head//3,8731903a/rbd_data.10a66b8b4567.0000000000003929/head//3]
2014-06-12 14:51:32.078291 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:32.078309 7f9d60bd2700 10 stat 3.3a_head/367e203a/rbd_data.10ad6b8b4567.00000000000026c8/head//3
2014-06-12 14:51:32.078436 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 367e203a/rbd_data.10ad6b8b4567.00000000000026c8/head//3
2014-06-12 14:51:32.104994 7f9d60bd2700 10 getattrs 3.3a_head/367e203a/rbd_data.10ad6b8b4567.00000000000026c8/head//3 = 0
2014-06-12 14:51:32.105009 7f9d60bd2700 10 stat 3.3a_head/2871303a/rbd_data.10976b8b4567.0000000000003aa4/head//3
2014-06-12 14:51:32.105119 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 2871303a/rbd_data.10976b8b4567.0000000000003aa4/head//3
2014-06-12 14:51:32.117018 7f9d60bd2700 10 getattrs 3.3a_head/2871303a/rbd_data.10976b8b4567.0000000000003aa4/head//3 = 0
2014-06-12 14:51:32.117030 7f9d60bd2700 10 stat 3.3a_head/2281303a/rbd_data.10af6b8b4567.000000000000162f/head//3
2014-06-12 14:51:32.117150 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 2281303a/rbd_data.10af6b8b4567.000000000000162f/head//3
2014-06-12 14:51:32.129579 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:32.129839 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:32.137531 7f9d60bd2700 10 getattrs 3.3a_head/2281303a/rbd_data.10af6b8b4567.000000000000162f/head//3 = 0
2014-06-12 14:51:32.137538 7f9d60bd2700 10 stat 3.3a_head/6f12303a/rbd_data.10a96b8b4567.0000000000000a14/head//3
2014-06-12 14:51:32.137639 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 6f12303a/rbd_data.10a96b8b4567.0000000000000a14/head//3
2014-06-12 14:51:32.167218 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:32.173833 7f9d60bd2700 10 getattrs 3.3a_head/6f12303a/rbd_data.10a96b8b4567.0000000000000a14/head//3 = 0
2014-06-12 14:51:32.173841 7f9d60bd2700 10 stat 3.3a_head/9f63303a/rbd_data.10b36b8b4567.0000000000001f6c/head//3
2014-06-12 14:51:32.173929 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 9f63303a/rbd_data.10b36b8b4567.0000000000001f6c/head//3
2014-06-12 14:51:32.204615 7f9d60bd2700 10 getattrs 3.3a_head/9f63303a/rbd_data.10b36b8b4567.0000000000001f6c/head//3 = 0
2014-06-12 14:51:32.204629 7f9d60bd2700 10 stat 3.3a_head/c175303a/rbd_data.108f6b8b4567.0000000000000919/head//3
2014-06-12 14:51:32.204762 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c175303a/rbd_data.108f6b8b4567.0000000000000919/head//3
2014-06-12 14:51:32.212208 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:32.212276 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:32.244363 7f9d60bd2700 10 getattrs 3.3a_head/c175303a/rbd_data.108f6b8b4567.0000000000000919/head//3 = 0
2014-06-12 14:51:32.244378 7f9d60bd2700 10 stat 3.3a_head/4a18303a/rbd_data.10aa6b8b4567.0000000000000af6/head//3
2014-06-12 14:51:32.244480 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 4a18303a/rbd_data.10aa6b8b4567.0000000000000af6/head//3
2014-06-12 14:51:32.292591 7f9d60bd2700 10 getattrs 3.3a_head/4a18303a/rbd_data.10aa6b8b4567.0000000000000af6/head//3 = 0
2014-06-12 14:51:32.292615 7f9d60bd2700 10 stat 3.3a_head/ff18303a/rbd_data.109c6b8b4567.000000000000047a/head//3
2014-06-12 14:51:32.292742 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ff18303a/rbd_data.109c6b8b4567.000000000000047a/head//3
2014-06-12 14:51:32.316638 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:32.326118 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:32.337034 7f9d60bd2700 10 getattrs 3.3a_head/ff18303a/rbd_data.109c6b8b4567.000000000000047a/head//3 = 0
2014-06-12 14:51:32.337054 7f9d60bd2700 10 stat 3.3a_head/4309303a/rbd_data.10a06b8b4567.0000000000002853/head//3
2014-06-12 14:51:32.337219 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 4309303a/rbd_data.10a06b8b4567.0000000000002853/head//3
2014-06-12 14:51:32.337397 7f9d60bd2700 10 getattrs 3.3a_head/4309303a/rbd_data.10a06b8b4567.0000000000002853/head//3 = 0
2014-06-12 14:51:32.337406 7f9d60bd2700 10 stat 3.3a_head/baa303a/rbd_data.10986b8b4567.0000000000000434/head//3
2014-06-12 14:51:32.337557 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid baa303a/rbd_data.10986b8b4567.0000000000000434/head//3
2014-06-12 14:51:32.346111 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:32.346142 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:32.346143 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.46 < max 0.5 = yes
2014-06-12 14:51:32.346150 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=1
2014-06-12 14:51:32.366036 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:32.370863 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:32.371733 7f9d60bd2700 10 getattrs 3.3a_head/baa303a/rbd_data.10986b8b4567.0000000000000434/head//3 = 0
2014-06-12 14:51:32.371747 7f9d60bd2700 10 stat 3.3a_head/bf9b303a/rbd_data.108e6b8b4567.0000000000000734/head//3
2014-06-12 14:51:32.371877 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid bf9b303a/rbd_data.108e6b8b4567.0000000000000734/head//3
2014-06-12 14:51:32.381667 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:32.415086 7f9d60bd2700 10 getattrs 3.3a_head/bf9b303a/rbd_data.108e6b8b4567.0000000000000734/head//3 = 0
2014-06-12 14:51:32.415113 7f9d60bd2700 10 stat 3.3a_head/968d303a/rbd_data.10b16b8b4567.0000000000001655/head//3
2014-06-12 14:51:32.415334 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 968d303a/rbd_data.10b16b8b4567.0000000000001655/head//3
2014-06-12 14:51:32.427914 7f9d60bd2700 10 getattrs 3.3a_head/968d303a/rbd_data.10b16b8b4567.0000000000001655/head//3 = 0
2014-06-12 14:51:32.427928 7f9d60bd2700 10 stat 3.3a_head/7d3e303a/rbd_data.10a26b8b4567.0000000000001a2b/head//3
2014-06-12 14:51:32.428093 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 7d3e303a/rbd_data.10a26b8b4567.0000000000001a2b/head//3
2014-06-12 14:51:32.440048 7f9d60bd2700 10 getattrs 3.3a_head/7d3e303a/rbd_data.10a26b8b4567.0000000000001a2b/head//3 = 0
2014-06-12 14:51:32.440065 7f9d60bd2700 10 stat 3.3a_head/c46e303a/rbd_data.109c6b8b4567.000000000000202f/head//3
2014-06-12 14:51:32.440257 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c46e303a/rbd_data.109c6b8b4567.000000000000202f/head//3
2014-06-12 14:51:32.467535 7f9d60bd2700 10 getattrs 3.3a_head/c46e303a/rbd_data.109c6b8b4567.000000000000202f/head//3 = 0
2014-06-12 14:51:32.467542 7f9d60bd2700 10 stat 3.3a_head/10b3403a/rbd_data.10b36b8b4567.000000000000238f/head//3
2014-06-12 14:51:32.467634 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 10b3403a/rbd_data.10b36b8b4567.000000000000238f/head//3
2014-06-12 14:51:32.488225 7f9d60bd2700 10 getattrs 3.3a_head/10b3403a/rbd_data.10b36b8b4567.000000000000238f/head//3 = 0
2014-06-12 14:51:32.488245 7f9d60bd2700 10 stat 3.3a_head/834403a/rbd_data.10a16b8b4567.0000000000002a65/head//3
2014-06-12 14:51:32.488404 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 834403a/rbd_data.10a16b8b4567.0000000000002a65/head//3
2014-06-12 14:51:32.512886 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:32.516611 7f9d60bd2700 10 getattrs 3.3a_head/834403a/rbd_data.10a16b8b4567.0000000000002a65/head//3 = 0
2014-06-12 14:51:32.516619 7f9d60bd2700 10 stat 3.3a_head/c844403a/rbd_data.108f6b8b4567.00000000000022ea/head//3
2014-06-12 14:51:32.516703 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c844403a/rbd_data.108f6b8b4567.00000000000022ea/head//3
2014-06-12 14:51:32.518725 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:32.522630 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:32.532082 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:32.532660 7f9d60bd2700 10 getattrs 3.3a_head/c844403a/rbd_data.108f6b8b4567.00000000000022ea/head//3 = 0
2014-06-12 14:51:32.532672 7f9d60bd2700 10 stat 3.3a_head/2705403a/rbd_data.109e6b8b4567.0000000000002190/head//3
2014-06-12 14:51:32.532778 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 2705403a/rbd_data.109e6b8b4567.0000000000002190/head//3
2014-06-12 14:51:32.547517 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:32.550788 7f9d60bd2700 10 getattrs 3.3a_head/2705403a/rbd_data.109e6b8b4567.0000000000002190/head//3 = 0
2014-06-12 14:51:32.550798 7f9d60bd2700 10 stat 3.3a_head/e8b6403a/rbd_data.10996b8b4567.0000000000001126/head//3
2014-06-12 14:51:32.550897 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid e8b6403a/rbd_data.10996b8b4567.0000000000001126/head//3
2014-06-12 14:51:32.566937 7f9d60bd2700 10 getattrs 3.3a_head/e8b6403a/rbd_data.10996b8b4567.0000000000001126/head//3 = 0
2014-06-12 14:51:32.566946 7f9d60bd2700 10 stat 3.3a_head/9e07403a/rbd_data.10a36b8b4567.0000000000000c5c/head//3
2014-06-12 14:51:32.567043 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 9e07403a/rbd_data.10a36b8b4567.0000000000000c5c/head//3
2014-06-12 14:51:32.615493 7f9d60bd2700 10 getattrs 3.3a_head/9e07403a/rbd_data.10a36b8b4567.0000000000000c5c/head//3 = 0
2014-06-12 14:51:32.615509 7f9d60bd2700 10 stat 3.3a_head/a177403a/rbd_data.109a6b8b4567.00000000000004b2/head//3
2014-06-12 14:51:32.615675 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a177403a/rbd_data.109a6b8b4567.00000000000004b2/head//3
2014-06-12 14:51:32.661449 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:32.664985 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:32.669370 7f9d60bd2700 10 getattrs 3.3a_head/a177403a/rbd_data.109a6b8b4567.00000000000004b2/head//3 = 0
2014-06-12 14:51:32.669381 7f9d60bd2700 10 stat 3.3a_head/d12a403a/rbd_data.109e6b8b4567.0000000000001cf9/head//3
2014-06-12 14:51:32.669495 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid d12a403a/rbd_data.109e6b8b4567.0000000000001cf9/head//3
2014-06-12 14:51:32.669595 7f9d60bd2700 10 getattrs 3.3a_head/d12a403a/rbd_data.109e6b8b4567.0000000000001cf9/head//3 = 0
2014-06-12 14:51:32.669600 7f9d60bd2700 10 stat 3.3a_head/515a403a/rbd_data.108d6b8b4567.000000000000323a/head//3
2014-06-12 14:51:32.669717 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 515a403a/rbd_data.108d6b8b4567.000000000000323a/head//3
2014-06-12 14:51:32.670954 7f9d44101700  5 -- op tracker -- , seq: 748583, time: 2014-06-12 14:51:32.670888, event: header_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:32.670966 7f9d44101700  5 -- op tracker -- , seq: 748583, time: 2014-06-12 14:51:32.670889, event: throttled, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:32.670971 7f9d44101700  5 -- op tracker -- , seq: 748583, time: 2014-06-12 14:51:32.670925, event: all_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:32.670977 7f9d44101700  5 -- op tracker -- , seq: 748583, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:32.670981 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:32.670985 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:32.670991 7f9d44101700 15 osd.2 173 enqueue_op 0x8dbd000 prio 127 cost 10717 latency 0.000103 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:32.671036 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:32.698662 7f9d60bd2700 10 getattrs 3.3a_head/515a403a/rbd_data.108d6b8b4567.000000000000323a/head//3 = 0
2014-06-12 14:51:32.698673 7f9d60bd2700 10 stat 3.3a_head/aadb403a/rbd_data.10906b8b4567.0000000000002bd7/head//3
2014-06-12 14:51:32.698773 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid aadb403a/rbd_data.10906b8b4567.0000000000002bd7/head//3
2014-06-12 14:51:32.698863 7f9d60bd2700 10 getattrs 3.3a_head/aadb403a/rbd_data.10906b8b4567.0000000000002bd7/head//3 = 0
2014-06-12 14:51:32.698869 7f9d60bd2700 10 stat 3.3a_head/3df403a/rbd_data.109e6b8b4567.0000000000000668/head//3
2014-06-12 14:51:32.698959 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 3df403a/rbd_data.109e6b8b4567.0000000000000668/head//3
2014-06-12 14:51:32.700354 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:32.747514 7f9d60bd2700 10 getattrs 3.3a_head/3df403a/rbd_data.109e6b8b4567.0000000000000668/head//3 = 0
2014-06-12 14:51:32.747531 7f9d60bd2700 10 collection_getattrs 3.3a_head
2014-06-12 14:51:32.747583 7f9d60bd2700 10 collection_getattrs 3.3a_head = 0
2014-06-12 14:51:32.747587 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:32.747614 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:32.747622 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:32.747631 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:32.747634 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:32.747671 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3a at 2014-06-11 13:02:32.500954
2014-06-12 14:51:32.747705 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.43 at 2014-06-11 13:02:32.501298
2014-06-12 14:51:32.747715 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.747719 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.747741 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6b at 2014-06-11 13:02:32.502023
2014-06-12 14:51:32.747749 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.747751 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.747765 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6c at 2014-06-11 13:02:32.502351
2014-06-12 14:51:32.747771 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.747741 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x8dbd000 prio 127 cost 10717 latency 0.076852 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]
2014-06-12 14:51:32.747773 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.747784 7f9d6b3e7700  5 -- op tracker -- , seq: 748583, time: 2014-06-12 14:51:32.747777, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:32.747787 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.a0 at 2014-06-11 13:02:32.502930
2014-06-12 14:51:32.747813 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.747811 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] handle_message: 0x8dbd000
2014-06-12 14:51:32.747816 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.a0( v 173'2803 (0'0,173'2803] local-les=157 n=294 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2802 mlcod 173'2802 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.747831 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.105 at 2014-06-11 13:02:32.503257
2014-06-12 14:51:32.747837 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.747828 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:32.747840 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2817 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.747846 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:32.747864 7f9d6b3e7700  5 -- op tracker -- , seq: 748583, time: 2014-06-12 14:51:32.747858, event: started, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:32.747857 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.122 at 2014-06-11 13:02:32.503770
2014-06-12 14:51:32.747874 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.747872 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:32.747877 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.747890 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.12b at 2014-06-11 13:02:32.504419
2014-06-12 14:51:32.747896 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.747898 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.12b( v 173'3236 (79'235,173'3236] local-les=137 n=323 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3235 mlcod 173'3235 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.747911 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.131 at 2014-06-11 13:02:32.504931
2014-06-12 14:51:32.747916 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.747918 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.747934 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.15d at 2014-06-11 13:02:32.505494
2014-06-12 14:51:32.747939 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.747941 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.747954 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.165 at 2014-06-11 13:02:32.506051
2014-06-12 14:51:32.747960 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.747962 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.165( v 173'3106 (79'105,173'3106] local-les=149 n=320 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3105 mlcod 173'3105 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.747974 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.170 at 2014-06-11 13:02:32.506577
2014-06-12 14:51:32.747980 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.747976 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] map version is 173'2964
2014-06-12 14:51:32.747982 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3049 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.747996 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x8dbd000 finish
2014-06-12 14:51:32.747995 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1b5 at 2014-06-11 13:02:32.507250
2014-06-12 14:51:32.748001 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748003 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1b5( v 173'2905 (0'0,173'2905] local-les=153 n=320 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2904 mlcod 173'2904 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748015 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1bc at 2014-06-11 13:02:32.507758
2014-06-12 14:51:32.748008 7f9d6b3e7700  5 -- op tracker -- , seq: 748583, time: 2014-06-12 14:51:32.748002, event: done, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:32.748021 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748023 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2459 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748036 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:32.748034 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1e1 at 2014-06-11 13:02:32.508326
2014-06-12 14:51:32.748032 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:32.748041 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:32.748040 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748043 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2540 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748055 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1e8 at 2014-06-11 13:02:32.508866
2014-06-12 14:51:32.748063 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748045 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:32.748065 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3033 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748067 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:32.748077 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1eb at 2014-06-11 13:02:32.509421
2014-06-12 14:51:32.748083 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748080 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:32.748085 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3095 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748103 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1fd at 2014-06-11 13:02:32.510047
2014-06-12 14:51:32.748110 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748111 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1fd( v 173'3047 (79'46,173'3047] local-les=157 n=305 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748118 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.204 at 2014-06-11 13:02:32.510573
2014-06-12 14:51:32.748122 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748120 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:32.748123 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.204( v 173'2982 (0'0,173'2982] local-les=161 n=308 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748130 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.209 at 2014-06-11 13:02:32.511095
2014-06-12 14:51:32.748127 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:32.748133 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748135 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:32.748135 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.209( v 173'2531 (0'0,173'2531] local-les=156 n=264 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2530 mlcod 173'2530 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748143 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.214 at 2014-06-11 13:02:32.511670
2014-06-12 14:51:32.748149 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748150 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2934 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748157 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.21d at 2014-06-11 13:02:32.512202
2014-06-12 14:51:32.748160 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748156 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 367e203a/rbd_data.10ad6b8b4567.00000000000026c8/head//3, auth == maps.end()
2014-06-12 14:51:32.748162 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.21d( v 173'3024 (79'23,173'3024] local-les=156 n=318 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3023 mlcod 173'3023 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748169 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.23f at 2014-06-11 13:02:32.512776
2014-06-12 14:51:32.748172 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748173 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748180 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.251 at 2014-06-11 13:02:32.513340
2014-06-12 14:51:32.748184 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748173 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 367e203a/rbd_data.10ad6b8b4567.00000000000026c8/head//3
2014-06-12 14:51:32.748185 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748192 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.262 at 2014-06-11 13:02:32.513946
2014-06-12 14:51:32.748200 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748193 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2871303a/rbd_data.10976b8b4567.0000000000003aa4/head//3, auth == maps.end()
2014-06-12 14:51:32.748202 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.262( v 173'2675 (0'0,173'2675] local-les=142 n=292 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2674 mlcod 173'2674 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748215 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.266 at 2014-06-11 13:02:32.514523
2014-06-12 14:51:32.748221 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748224 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2824 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748236 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.288 at 2014-06-11 13:02:32.515022
2014-06-12 14:51:32.748242 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748244 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.288( v 173'2548 (0'0,173'2548] local-les=161 n=263 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2547 mlcod 173'2547 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748256 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2b4 at 2014-06-11 13:02:32.515543
2014-06-12 14:51:32.748262 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748209 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2871303a/rbd_data.10976b8b4567.0000000000003aa4/head//3
2014-06-12 14:51:32.748264 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2b4( v 173'3109 (79'108,173'3109] local-les=137 n=312 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3108 mlcod 173'3108 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748277 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2c4 at 2014-06-11 13:02:32.516106
2014-06-12 14:51:32.748276 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2281303a/rbd_data.10af6b8b4567.000000000000162f/head//3, auth == maps.end()
2014-06-12 14:51:32.748286 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748289 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2c4( v 173'2897 (0'0,173'2897] local-les=139 n=307 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2896 mlcod 173'2896 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748291 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2281303a/rbd_data.10af6b8b4567.000000000000162f/head//3
2014-06-12 14:51:32.748301 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2d3 at 2014-06-11 13:02:32.516627
2014-06-12 14:51:32.748307 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748307 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6f12303a/rbd_data.10a96b8b4567.0000000000000a14/head//3, auth == maps.end()
2014-06-12 14:51:32.748309 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2d3( v 173'2556 (0'0,173'2556] local-les=153 n=276 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2555 mlcod 173'2555 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748322 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2da at 2014-06-11 13:02:32.517119
2014-06-12 14:51:32.748321 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6f12303a/rbd_data.10a96b8b4567.0000000000000a14/head//3
2014-06-12 14:51:32.748328 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748334 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2da( v 173'2988 (0'0,173'2988] local-les=145 n=302 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2987 mlcod 173'2987 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748336 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9f63303a/rbd_data.10b36b8b4567.0000000000001f6c/head//3, auth == maps.end()
2014-06-12 14:51:32.748346 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2f4 at 2014-06-11 13:02:32.517764
2014-06-12 14:51:32.748356 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748351 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9f63303a/rbd_data.10b36b8b4567.0000000000001f6c/head//3
2014-06-12 14:51:32.748359 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3274 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748371 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.312 at 2014-06-11 13:02:32.518300
2014-06-12 14:51:32.748368 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c175303a/rbd_data.108f6b8b4567.0000000000000919/head//3, auth == maps.end()
2014-06-12 14:51:32.748379 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748381 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2782 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748384 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c175303a/rbd_data.108f6b8b4567.0000000000000919/head//3
2014-06-12 14:51:32.748394 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.320 at 2014-06-11 13:02:32.518824
2014-06-12 14:51:32.748400 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748402 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.320( v 173'2687 (0'0,173'2687] local-les=145 n=282 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2686 mlcod 173'2686 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748401 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4a18303a/rbd_data.10aa6b8b4567.0000000000000af6/head//3, auth == maps.end()
2014-06-12 14:51:32.748415 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.330 at 2014-06-11 13:02:32.519349
2014-06-12 14:51:32.748423 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748417 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4a18303a/rbd_data.10aa6b8b4567.0000000000000af6/head//3
2014-06-12 14:51:32.748426 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2798 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748439 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.342 at 2014-06-11 13:02:32.519873
2014-06-12 14:51:32.748444 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748445 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2464 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748434 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ff18303a/rbd_data.109c6b8b4567.000000000000047a/head//3, auth == maps.end()
2014-06-12 14:51:32.748453 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.35b at 2014-06-11 13:02:32.520528
2014-06-12 14:51:32.748456 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748457 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.35b( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748464 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.366 at 2014-06-11 13:02:32.521097
2014-06-12 14:51:32.748467 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748458 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ff18303a/rbd_data.109c6b8b4567.000000000000047a/head//3
2014-06-12 14:51:32.748468 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2661 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748482 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.37a at 2014-06-11 13:02:32.521744
2014-06-12 14:51:32.748475 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4309303a/rbd_data.10a06b8b4567.0000000000002853/head//3, auth == maps.end()
2014-06-12 14:51:32.748487 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748489 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.37a( v 173'2555 (0'0,173'2555] local-les=153 n=292 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2554 mlcod 173'2554 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748497 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.394 at 2014-06-11 13:02:32.522362
2014-06-12 14:51:32.748500 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748492 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4309303a/rbd_data.10a06b8b4567.0000000000002853/head//3
2014-06-12 14:51:32.748501 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.394( v 173'2648 (0'0,173'2648] local-les=145 n=266 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748509 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.395 at 2014-06-11 13:02:32.522991
2014-06-12 14:51:32.748512 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748509 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj baa303a/rbd_data.10986b8b4567.0000000000000434/head//3, auth == maps.end()
2014-06-12 14:51:32.748513 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3327 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748530 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3ac at 2014-06-11 13:02:32.523569
2014-06-12 14:51:32.748534 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748525 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj baa303a/rbd_data.10986b8b4567.0000000000000434/head//3
2014-06-12 14:51:32.748535 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3ac( v 173'2860 (0'0,173'2860] local-les=142 n=294 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748542 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3d4 at 2014-06-11 13:02:32.524119
2014-06-12 14:51:32.748545 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748546 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3d4( v 173'2696 (0'0,173'2696] local-les=137 n=271 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748542 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bf9b303a/rbd_data.108e6b8b4567.0000000000000734/head//3, auth == maps.end()
2014-06-12 14:51:32.748554 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3d8 at 2014-06-11 13:02:32.524641
2014-06-12 14:51:32.748557 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748558 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3d8( v 173'2879 (0'0,173'2879] local-les=156 n=294 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748566 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.401 at 2014-06-11 13:02:32.525188
2014-06-12 14:51:32.748557 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj bf9b303a/rbd_data.108e6b8b4567.0000000000000734/head//3
2014-06-12 14:51:32.748569 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748571 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3154 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748583 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.408 at 2014-06-11 13:02:32.525838
2014-06-12 14:51:32.748574 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 968d303a/rbd_data.10b16b8b4567.0000000000001655/head//3, auth == maps.end()
2014-06-12 14:51:32.748587 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748588 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2883 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748590 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 968d303a/rbd_data.10b16b8b4567.0000000000001655/head//3
2014-06-12 14:51:32.748595 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.411 at 2014-06-11 13:02:32.526419
2014-06-12 14:51:32.748598 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748599 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.411( v 173'2402 (0'0,173'2402] local-les=147 n=278 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2401 mlcod 173'2401 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748601 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7d3e303a/rbd_data.10a26b8b4567.0000000000001a2b/head//3, auth == maps.end()
2014-06-12 14:51:32.748606 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.45f at 2014-06-11 13:02:32.526940
2014-06-12 14:51:32.748610 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748610 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7d3e303a/rbd_data.10a26b8b4567.0000000000001a2b/head//3
2014-06-12 14:51:32.748611 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2357 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748618 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.479 at 2014-06-11 13:02:32.527498
2014-06-12 14:51:32.748622 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748619 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c46e303a/rbd_data.109c6b8b4567.000000000000202f/head//3, auth == maps.end()
2014-06-12 14:51:32.748623 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.479( v 173'2926 (0'0,173'2926] local-les=142 n=298 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748632 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4a7 at 2014-06-11 13:02:32.528023
2014-06-12 14:51:32.748628 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c46e303a/rbd_data.109c6b8b4567.000000000000202f/head//3
2014-06-12 14:51:32.748635 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748636 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2594 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748638 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 10b3403a/rbd_data.10b36b8b4567.000000000000238f/head//3, auth == maps.end()
2014-06-12 14:51:32.748643 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4b4 at 2014-06-11 13:02:32.528548
2014-06-12 14:51:32.748646 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748647 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 10b3403a/rbd_data.10b36b8b4567.000000000000238f/head//3
2014-06-12 14:51:32.748648 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4b4( v 173'2792 (0'0,173'2792] local-les=137 n=305 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748655 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4d4 at 2014-06-11 13:02:32.529466
2014-06-12 14:51:32.748667 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748657 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 834403a/rbd_data.10a16b8b4567.0000000000002a65/head//3, auth == maps.end()
2014-06-12 14:51:32.748669 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2745 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748677 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4ef at 2014-06-11 13:02:32.530034
2014-06-12 14:51:32.748680 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748679 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 834403a/rbd_data.10a16b8b4567.0000000000002a65/head//3
2014-06-12 14:51:32.748682 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4ef( v 173'3004 (79'3,173'3004] local-les=149 n=307 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748689 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c844403a/rbd_data.108f6b8b4567.00000000000022ea/head//3, auth == maps.end()
2014-06-12 14:51:32.748689 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4f0 at 2014-06-11 13:02:32.530572
2014-06-12 14:51:32.748700 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748702 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2864 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748702 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c844403a/rbd_data.108f6b8b4567.00000000000022ea/head//3
2014-06-12 14:51:32.748710 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.54b at 2014-06-11 13:02:32.531354
2014-06-12 14:51:32.748714 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748715 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2722 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748723 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.568 at 2014-06-11 13:02:32.531914
2014-06-12 14:51:32.748715 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2705403a/rbd_data.109e6b8b4567.0000000000002190/head//3, auth == maps.end()
2014-06-12 14:51:32.748726 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748727 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.568( v 173'2727 (0'0,173'2727] local-les=161 n=270 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748734 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.58f at 2014-06-11 13:02:32.532612
2014-06-12 14:51:32.748737 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748730 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2705403a/rbd_data.109e6b8b4567.0000000000002190/head//3
2014-06-12 14:51:32.748738 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.58f( v 173'2707 (0'0,173'2707] local-les=153 n=304 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748745 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5a3 at 2014-06-11 13:02:32.533139
2014-06-12 14:51:32.748756 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748747 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e8b6403a/rbd_data.10996b8b4567.0000000000001126/head//3, auth == maps.end()
2014-06-12 14:51:32.748758 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5a3( v 173'2874 (0'0,173'2874] local-les=142 n=296 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2873 mlcod 173'2873 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748763 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e8b6403a/rbd_data.10996b8b4567.0000000000001126/head//3
2014-06-12 14:51:32.748772 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5ce at 2014-06-11 13:02:32.533636
2014-06-12 14:51:32.748787 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748781 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9e07403a/rbd_data.10a36b8b4567.0000000000000c5c/head//3, auth == maps.end()
2014-06-12 14:51:32.748797 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9e07403a/rbd_data.10a36b8b4567.0000000000000c5c/head//3
2014-06-12 14:51:32.748790 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5ce( v 173'2741 (0'0,173'2741] local-les=153 n=294 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748814 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a177403a/rbd_data.109a6b8b4567.00000000000004b2/head//3, auth == maps.end()
2014-06-12 14:51:32.748820 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5dd at 2014-06-11 13:02:32.534211
2014-06-12 14:51:32.748830 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a177403a/rbd_data.109a6b8b4567.00000000000004b2/head//3
2014-06-12 14:51:32.748838 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748843 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5dd( v 173'2852 (0'0,173'2852] local-les=156 n=307 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748847 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d12a403a/rbd_data.109e6b8b4567.0000000000001cf9/head//3, auth == maps.end()
2014-06-12 14:51:32.748857 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5fc at 2014-06-11 13:02:32.534738
2014-06-12 14:51:32.748864 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748863 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d12a403a/rbd_data.109e6b8b4567.0000000000001cf9/head//3
2014-06-12 14:51:32.748866 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5fc( v 173'2714 (0'0,173'2714] local-les=157 n=284 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748880 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.602 at 2014-06-11 13:02:32.535274
2014-06-12 14:51:32.748880 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 515a403a/rbd_data.108d6b8b4567.000000000000323a/head//3, auth == maps.end()
2014-06-12 14:51:32.748890 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748893 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3009 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748903 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.605 at 2014-06-11 13:02:32.535829
2014-06-12 14:51:32.748896 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 515a403a/rbd_data.108d6b8b4567.000000000000323a/head//3
2014-06-12 14:51:32.748907 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748912 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748919 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.613 at 2014-06-11 13:02:32.536448
2014-06-12 14:51:32.748922 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748912 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aadb403a/rbd_data.10906b8b4567.0000000000002bd7/head//3, auth == maps.end()
2014-06-12 14:51:32.748923 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748931 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.635 at 2014-06-11 13:02:32.537041
2014-06-12 14:51:32.748935 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748927 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj aadb403a/rbd_data.10906b8b4567.0000000000002bd7/head//3
2014-06-12 14:51:32.748936 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.635( v 173'2966 (0'0,173'2966] local-les=137 n=308 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2965 mlcod 173'2965 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748944 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.63b at 2014-06-11 13:02:32.537647
2014-06-12 14:51:32.748947 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748948 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748944 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3df403a/rbd_data.109e6b8b4567.0000000000000668/head//3, auth == maps.end()
2014-06-12 14:51:32.748956 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.63e at 2014-06-11 13:02:32.538178
2014-06-12 14:51:32.748959 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748960 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748960 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3df403a/rbd_data.109e6b8b4567.0000000000000668/head//3
2014-06-12 14:51:32.748967 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.650 at 2014-06-11 13:02:32.538854
2014-06-12 14:51:32.748976 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.748978 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] 
2014-06-12 14:51:32.748978 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.748991 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.662 at 2014-06-11 13:02:32.539391
2014-06-12 14:51:32.748990 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub
2014-06-12 14:51:32.749000 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749003 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.662( v 173'2604 (0'0,173'2604] local-les=161 n=257 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2603 mlcod 173'2603 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749017 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.667 at 2014-06-11 13:02:32.539999
2014-06-12 14:51:32.749026 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749011 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  3df403a/rbd_data.109e6b8b4567.0000000000000668/head//3 3df403a/rbd_data.109e6b8b4567.0000000000000668/head//3(173'2679 client.4295.0:127089 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2679)
2014-06-12 14:51:32.749029 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.667( v 173'2958 (0'0,173'2958] local-les=149 n=303 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749045 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6a8 at 2014-06-11 13:02:32.540566
2014-06-12 14:51:32.749049 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749036 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  aadb403a/rbd_data.10906b8b4567.0000000000002bd7/head//3 aadb403a/rbd_data.10906b8b4567.0000000000002bd7/head//3(173'1812 client.4281.0:88295 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1812)
2014-06-12 14:51:32.749050 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6a8( v 173'2854 (0'0,173'2854] local-les=137 n=299 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749057 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6ff at 2014-06-11 13:02:32.541074
2014-06-12 14:51:32.749060 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749061 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6ff( v 173'2792 (0'0,173'2792] local-les=161 n=298 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749068 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.711 at 2014-06-11 13:02:32.541770
2014-06-12 14:51:32.749072 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749059 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  515a403a/rbd_data.108d6b8b4567.000000000000323a/head//3 515a403a/rbd_data.108d6b8b4567.000000000000323a/head//3(173'2063 client.4278.0:101013 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2063)
2014-06-12 14:51:32.749073 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.711( v 173'2514 (0'0,173'2514] local-les=147 n=297 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749084 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.712 at 2014-06-11 13:02:32.542314
2014-06-12 14:51:32.749081 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  d12a403a/rbd_data.109e6b8b4567.0000000000001cf9/head//3 d12a403a/rbd_data.109e6b8b4567.0000000000001cf9/head//3(173'1271 client.4295.0:62495 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1271)
2014-06-12 14:51:32.749096 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749098 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.712( v 173'2782 (0'0,173'2782] local-les=153 n=305 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749106 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.717 at 2014-06-11 13:02:32.542947
2014-06-12 14:51:32.749110 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749102 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a177403a/rbd_data.109a6b8b4567.00000000000004b2/head//3 a177403a/rbd_data.109a6b8b4567.00000000000004b2/head//3(173'2637 client.4291.0:122985 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2637)
2014-06-12 14:51:32.749111 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2583 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749119 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.721 at 2014-06-11 13:02:32.543618
2014-06-12 14:51:32.749126 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749127 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2895 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749122 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  9e07403a/rbd_data.10a36b8b4567.0000000000000c5c/head//3 9e07403a/rbd_data.10a36b8b4567.0000000000000c5c/head//3(173'2924 client.4300.0:135665 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2924)
2014-06-12 14:51:32.749135 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.735 at 2014-06-11 13:02:32.544147
2014-06-12 14:51:32.749138 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749139 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.735( v 173'2803 (0'0,173'2803] local-les=137 n=306 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2802 mlcod 173'2802 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749147 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.762 at 2014-06-11 13:02:32.544735
2014-06-12 14:51:32.749150 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749151 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.762( v 173'2726 (0'0,173'2726] local-les=147 n=302 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749142 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  e8b6403a/rbd_data.10996b8b4567.0000000000001126/head//3 e8b6403a/rbd_data.10996b8b4567.0000000000001126/head//3(173'884 client.4290.0:39613 wrlock_by=unknown.0.0:0 dirty s 4194304 uv884)
2014-06-12 14:51:32.749158 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.76e at 2014-06-11 13:02:32.545259
2014-06-12 14:51:32.749161 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749162 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2648 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749169 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.77e at 2014-06-11 13:02:32.545813
2014-06-12 14:51:32.749172 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749163 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  2705403a/rbd_data.109e6b8b4567.0000000000002190/head//3 2705403a/rbd_data.109e6b8b4567.0000000000002190/head//3(173'1451 client.4295.0:70790 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1451)
2014-06-12 14:51:32.749173 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.77e( v 173'3052 (79'51,173'3052] local-les=161 n=319 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3051 mlcod 173'3051 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749180 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.783 at 2014-06-11 13:02:32.546436
2014-06-12 14:51:32.749188 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749189 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749184 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c844403a/rbd_data.108f6b8b4567.00000000000022ea/head//3 c844403a/rbd_data.108f6b8b4567.00000000000022ea/head//3(173'1568 client.4280.0:71319 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1568)
2014-06-12 14:51:32.749196 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.785 at 2014-06-11 13:02:32.547026
2014-06-12 14:51:32.749200 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749201 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749208 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.793 at 2014-06-11 13:02:32.547568
2014-06-12 14:51:32.749211 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749212 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.793( v 173'2605 (0'0,173'2605] local-les=153 n=277 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2604 mlcod 173'2604 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749203 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  834403a/rbd_data.10a16b8b4567.0000000000002a65/head//3 834403a/rbd_data.10a16b8b4567.0000000000002a65/head//3(173'1824 client.4298.0:86907 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1824)
2014-06-12 14:51:32.749219 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ab at 2014-06-11 13:02:32.548162
2014-06-12 14:51:32.749222 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749223 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ab( v 173'2921 (0'0,173'2921] local-les=161 n=302 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2920 mlcod 173'2920 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749234 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ad at 2014-06-11 13:02:32.548789
2014-06-12 14:51:32.749238 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749225 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  10b3403a/rbd_data.10b36b8b4567.000000000000238f/head//3 10b3403a/rbd_data.10b36b8b4567.000000000000238f/head//3(173'1584 client.4316.0:74475 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1584)
2014-06-12 14:51:32.749240 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ad( v 173'3043 (79'42,173'3043] local-les=149 n=315 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749247 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ae at 2014-06-11 13:02:32.549514
2014-06-12 14:51:32.749250 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749251 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2517 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749245 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c46e303a/rbd_data.109c6b8b4567.000000000000202f/head//3 c46e303a/rbd_data.109c6b8b4567.000000000000202f/head//3(173'1474 client.4293.0:66604 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1474)
2014-06-12 14:51:32.749258 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7bc at 2014-06-11 13:02:32.550208
2014-06-12 14:51:32.749261 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749262 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7bc( v 173'2562 (0'0,173'2562] local-les=153 n=287 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2561 mlcod 173'2561 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749269 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7cc at 2014-06-11 13:02:32.550758
2014-06-12 14:51:32.749272 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749273 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749266 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  7d3e303a/rbd_data.10a26b8b4567.0000000000001a2b/head//3 7d3e303a/rbd_data.10a26b8b4567.0000000000001a2b/head//3(173'1234 client.4299.0:57344 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1234)
2014-06-12 14:51:32.749281 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7db at 2014-06-11 13:02:32.551269
2014-06-12 14:51:32.749284 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749285 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2681 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749292 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7f8 at 2014-06-11 13:02:32.551909
2014-06-12 14:51:32.749295 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:32.749287 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  968d303a/rbd_data.10b16b8b4567.0000000000001655/head//3 968d303a/rbd_data.10b16b8b4567.0000000000001655/head//3(173'1085 client.4314.0:50211 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1085)
2014-06-12 14:51:32.749297 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:32.749303 7f9d84175700 10 osd.2 173 sched_scrub 0.3 at 2014-06-12 13:01:39.405748: 6592.94 < min (86400 seconds)
2014-06-12 14:51:32.749314 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:32.749316 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:32.749317 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:32.749308 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  bf9b303a/rbd_data.108e6b8b4567.0000000000000734/head//3 bf9b303a/rbd_data.108e6b8b4567.0000000000000734/head//3(173'2708 client.4279.0:128238 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2708)
2014-06-12 14:51:32.749333 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  baa303a/rbd_data.10986b8b4567.0000000000000434/head//3 baa303a/rbd_data.10986b8b4567.0000000000000434/head//3(173'2600 client.4289.0:121562 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2600)
2014-06-12 14:51:32.749353 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  4309303a/rbd_data.10a06b8b4567.0000000000002853/head//3 4309303a/rbd_data.10a06b8b4567.0000000000002853/head//3(173'1723 client.4297.0:81008 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1723)
2014-06-12 14:51:32.749373 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ff18303a/rbd_data.109c6b8b4567.000000000000047a/head//3 ff18303a/rbd_data.109c6b8b4567.000000000000047a/head//3(173'2621 client.4293.0:124258 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2621)
2014-06-12 14:51:32.749390 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  4a18303a/rbd_data.10aa6b8b4567.0000000000000af6/head//3 4a18303a/rbd_data.10aa6b8b4567.0000000000000af6/head//3(173'2843 client.4307.0:137825 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2843)
2014-06-12 14:51:32.749402 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c175303a/rbd_data.108f6b8b4567.0000000000000919/head//3 c175303a/rbd_data.108f6b8b4567.0000000000000919/head//3(173'2786 client.4280.0:130258 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2786)
2014-06-12 14:51:32.749413 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  9f63303a/rbd_data.10b36b8b4567.0000000000001f6c/head//3 9f63303a/rbd_data.10b36b8b4567.0000000000001f6c/head//3(173'1353 client.4316.0:67487 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1353)
2014-06-12 14:51:32.749424 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  6f12303a/rbd_data.10a96b8b4567.0000000000000a14/head//3 6f12303a/rbd_data.10a96b8b4567.0000000000000a14/head//3(173'2812 client.4306.0:134699 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2812)
2014-06-12 14:51:32.749436 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  2281303a/rbd_data.10af6b8b4567.000000000000162f/head//3 2281303a/rbd_data.10af6b8b4567.000000000000162f/head//3(173'1096 client.4312.0:51922 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1096)
2014-06-12 14:51:32.749447 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  2871303a/rbd_data.10976b8b4567.0000000000003aa4/head//3 2871303a/rbd_data.10976b8b4567.0000000000003aa4/head//3(173'2417 client.4288.0:113036 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2417)
2014-06-12 14:51:32.749458 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  367e203a/rbd_data.10ad6b8b4567.00000000000026c8/head//3 367e203a/rbd_data.10ad6b8b4567.00000000000026c8/head//3(173'1760 client.4310.0:79009 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1760)
2014-06-12 14:51:32.749466 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:32.749477 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:32.749485 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:32.749487 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:32.749488 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:32.749527 7f9d60bd2700 10 collection_list_partial 3.3a_head start:6b41503a//0//-1 is_max:0
2014-06-12 14:51:32.749792 7f9d60bd2700 20 objects: [8c22503a/rbd_data.109c6b8b4567.00000000000034b8/head//3,eff2503a/rbd_data.109b6b8b4567.0000000000003770/head//3,6bea503a/rbd_data.10946b8b4567.0000000000002b6e/head//3,adbd503a/rbd_data.10b36b8b4567.00000000000004d1/head//3,44ef503a/rbd_data.10a46b8b4567.0000000000000ab0/head//3,d0e5603a/rbd_data.108e6b8b4567.000000000000158f/head//3,abc6603a/rbd_data.10ab6b8b4567.00000000000025d5/head//3,607603a/rbd_data.10ab6b8b4567.000000000000382f/head//3,5d87603a/rbd_data.10b16b8b4567.0000000000002596/head//3,ac6b603a/rbd_data.10a66b8b4567.00000000000023cd/head//3,5dcb603a/rbd_data.10956b8b4567.0000000000000693/head//3,c74d603a/rbd_data.10956b8b4567.00000000000007af/head//3,b1ce603a/rbd_data.10986b8b4567.0000000000001252/head//3,e61703a/rbd_data.109b6b8b4567.0000000000002864/head//3,4373703a/rbd_data.10aa6b8b4567.0000000000002317/head//3,d6a3703a/rbd_data.10a76b8b4567.00000000000016a6/head//3,4c47703a/rbd_data.10946b8b4567.0000000000003536/head//3,7d97703a/rbd_data.109d6b8b4567.000000000000212b/head//3,4659703a/rbd_data.10946b8b4567.00000000000034d1/head//3,eb7a703a/rbd_data.10ab6b8b4567.0000000000002ee5/head//3,37ea703a/rbd_data.10a46b8b4567.0000000000003307/head//3,8380803a/rbd_data.10966b8b4567.0000000000001ac4/head//3,c3e0803a/rbd_data.10a26b8b4567.00000000000006af/head//3,7681803a/rbd_data.10b36b8b4567.0000000000000b52/head//3,802803a/rbd_data.10a06b8b4567.0000000000000a8e/head//3]
2014-06-12 14:51:32.750333 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:32.750358 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:32.750374 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:32.750382 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:32.750388 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:32.750393 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:32.750399 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_map_chunk [6b41503a//0//-1,ef72803a//0//-1)
2014-06-12 14:51:32.750406 7f9d60bd2700 10 collection_list_partial 3.3a_head start:6b41503a//0//-1 is_max:0
2014-06-12 14:51:32.750948 7f9d60bd2700 20 objects: [8c22503a/rbd_data.109c6b8b4567.00000000000034b8/head//3,eff2503a/rbd_data.109b6b8b4567.0000000000003770/head//3,6bea503a/rbd_data.10946b8b4567.0000000000002b6e/head//3,adbd503a/rbd_data.10b36b8b4567.00000000000004d1/head//3,44ef503a/rbd_data.10a46b8b4567.0000000000000ab0/head//3,d0e5603a/rbd_data.108e6b8b4567.000000000000158f/head//3,abc6603a/rbd_data.10ab6b8b4567.00000000000025d5/head//3,607603a/rbd_data.10ab6b8b4567.000000000000382f/head//3,5d87603a/rbd_data.10b16b8b4567.0000000000002596/head//3,ac6b603a/rbd_data.10a66b8b4567.00000000000023cd/head//3,5dcb603a/rbd_data.10956b8b4567.0000000000000693/head//3,c74d603a/rbd_data.10956b8b4567.00000000000007af/head//3,b1ce603a/rbd_data.10986b8b4567.0000000000001252/head//3,e61703a/rbd_data.109b6b8b4567.0000000000002864/head//3,4373703a/rbd_data.10aa6b8b4567.0000000000002317/head//3,d6a3703a/rbd_data.10a76b8b4567.00000000000016a6/head//3,4c47703a/rbd_data.10946b8b4567.0000000000003536/head//3,7d97703a/rbd_data.109d6b8b4567.000000000000212b/head//3,4659703a/rbd_data.10946b8b4567.00000000000034d1/head//3,eb7a703a/rbd_data.10ab6b8b4567.0000000000002ee5/head//3,37ea703a/rbd_data.10a46b8b4567.0000000000003307/head//3,8380803a/rbd_data.10966b8b4567.0000000000001ac4/head//3,c3e0803a/rbd_data.10a26b8b4567.00000000000006af/head//3,7681803a/rbd_data.10b36b8b4567.0000000000000b52/head//3,802803a/rbd_data.10a06b8b4567.0000000000000a8e/head//3,ef72803a/rbd_data.10936b8b4567.0000000000001d81/head//3,a825803a/rbd_data.10956b8b4567.0000000000000185/head//3,9a06803a/rbd_data.10a26b8b4567.000000000000395b/head//3,6036803a/rbd_data.10946b8b4567.0000000000002208/head//3,dd38803a/rbd_data.10986b8b4567.000000000000160f/head//3,cc68803a/rbd_data.10b26b8b4567.000000000000313d/head//3,39f8803a/rbd_data.109b6b8b4567.0000000000001e5e/head//3,a4ca803a/rbd_data.10b06b8b4567.0000000000001477/head//3,569b803a/rbd_data.10a66b8b4567.0000000000002682/head//3,963d803a/rbd_data.10a36b8b4567.0000000000001019/head//3,94cf803a/rbd_data.10926b8b4567.00000000000029f2/head//3,faef803a/rbd_data.10906b8b4567.00000000000016bb/head//3,8731903a/rbd_data.10a66b8b4567.0000000000003929/head//3,c947903a/rbd_data.10a26b8b4567.0000000000002381/head//3,eba0a03a/rbd_data.108e6b8b4567.0000000000001462/head//3,aba5a03a/rbd_data.10aa6b8b4567.00000000000002e9/head//3,3097a03a/rbd_data.10a86b8b4567.000000000000011b/head//3,bd18a03a/rbd_data.10ab6b8b4567.00000000000039e8/head//3,bc79a03a/rbd_data.109e6b8b4567.00000000000031e8/head//3,3a5aa03a/rbd_data.10966b8b4567.0000000000001b26/head//3,559aa03a/rbd_data.10b26b8b4567.000000000000258e/head//3,8deea03a/rbd_data.10af6b8b4567.0000000000002997/head//3,efc3b03a/rbd_data.10a46b8b4567.000000000000366c/head//3,df15b03a/rbd_data.10986b8b4567.00000000000007c8/head//3,45a6b03a/rbd_data.10a36b8b4567.0000000000003208/head//3,4d99b03a/rbd_data.10a46b8b4567.0000000000000406/head//3,dc9ab03a/rbd_data.10b46b8b4567.0000000000000fcf/head//3,13ccb03a/rbd_data.10936b8b4567.0000000000000120/head//3,ecccb03a/rbd_data.10966b8b4567.0000000000000766/head//3,b70fb03a/rbd_data.10a16b8b4567.0000000000003967/head//3,342fb03a/rbd_data.10a56b8b4567.000000000000138d/head//3,5d10c03a/rbd_data.10946b8b4567.000000000000074c/head//3,dc50c03a/rbd_data.109f6b8b4567.000000000000362e/head//3,2a90c03a/rbd_data.10906b8b4567.0000000000001ffb/head//3,5e31c03a/rbd_data.10996b8b4567.0000000000003270/head//3,1e72c03a/rbd_data.10a96b8b4567.0000000000000187/head//3,9143c03a/rbd_data.108d6b8b4567.000000000000203c/head//3,a643c03a/rbd_data.10946b8b4567.000000000000004d/head//3,e614c03a/rbd_data.108d6b8b4567.00000000000023e0/head//3]
2014-06-12 14:51:32.751024 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:32.751038 7f9d60bd2700 10 stat 3.3a_head/8c22503a/rbd_data.109c6b8b4567.00000000000034b8/head//3
2014-06-12 14:51:32.751157 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 8c22503a/rbd_data.109c6b8b4567.00000000000034b8/head//3
2014-06-12 14:51:32.767761 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:32.777960 7f9d60bd2700 10 getattrs 3.3a_head/8c22503a/rbd_data.109c6b8b4567.00000000000034b8/head//3 = 0
2014-06-12 14:51:32.777970 7f9d60bd2700 10 stat 3.3a_head/eff2503a/rbd_data.109b6b8b4567.0000000000003770/head//3
2014-06-12 14:51:32.778063 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid eff2503a/rbd_data.109b6b8b4567.0000000000003770/head//3
2014-06-12 14:51:32.794310 7f9d60bd2700 10 getattrs 3.3a_head/eff2503a/rbd_data.109b6b8b4567.0000000000003770/head//3 = 0
2014-06-12 14:51:32.794321 7f9d60bd2700 10 stat 3.3a_head/6bea503a/rbd_data.10946b8b4567.0000000000002b6e/head//3
2014-06-12 14:51:32.794425 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 6bea503a/rbd_data.10946b8b4567.0000000000002b6e/head//3
2014-06-12 14:51:32.800442 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:32.800622 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:32.801693 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:32.815677 7f9d60bd2700 10 getattrs 3.3a_head/6bea503a/rbd_data.10946b8b4567.0000000000002b6e/head//3 = 0
2014-06-12 14:51:32.815686 7f9d60bd2700 10 stat 3.3a_head/adbd503a/rbd_data.10b36b8b4567.00000000000004d1/head//3
2014-06-12 14:51:32.815793 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid adbd503a/rbd_data.10b36b8b4567.00000000000004d1/head//3
2014-06-12 14:51:32.854803 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:32.865147 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:32.868526 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:32.870145 7f9d60bd2700 10 getattrs 3.3a_head/adbd503a/rbd_data.10b36b8b4567.00000000000004d1/head//3 = 0
2014-06-12 14:51:32.870166 7f9d60bd2700 10 stat 3.3a_head/44ef503a/rbd_data.10a46b8b4567.0000000000000ab0/head//3
2014-06-12 14:51:32.870304 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 44ef503a/rbd_data.10a46b8b4567.0000000000000ab0/head//3
2014-06-12 14:51:32.889365 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:32.889386 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:32.889999 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:32.890050 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:32.890144 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:32.890199 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:32.890269 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:32.890336 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:32.890362 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:32.890387 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:32.890406 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:32.890423 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:32.890444 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:32.890460 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:32.890477 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:32.890495 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:32.890512 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:32.890528 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:32.890542 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:32.890558 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:32.890575 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:32.890591 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:32.890610 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:32.890629 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:32.904490 7f9d60bd2700 10 getattrs 3.3a_head/44ef503a/rbd_data.10a46b8b4567.0000000000000ab0/head//3 = 0
2014-06-12 14:51:32.904502 7f9d60bd2700 10 stat 3.3a_head/d0e5603a/rbd_data.108e6b8b4567.000000000000158f/head//3
2014-06-12 14:51:32.904639 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid d0e5603a/rbd_data.108e6b8b4567.000000000000158f/head//3
2014-06-12 14:51:32.916588 7f9d60bd2700 10 getattrs 3.3a_head/d0e5603a/rbd_data.108e6b8b4567.000000000000158f/head//3 = 0
2014-06-12 14:51:32.916599 7f9d60bd2700 10 stat 3.3a_head/abc6603a/rbd_data.10ab6b8b4567.00000000000025d5/head//3
2014-06-12 14:51:32.916737 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid abc6603a/rbd_data.10ab6b8b4567.00000000000025d5/head//3
2014-06-12 14:51:32.921163 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:32.921215 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:32.928244 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:32.935686 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:32.949552 7f9d60bd2700 10 getattrs 3.3a_head/abc6603a/rbd_data.10ab6b8b4567.00000000000025d5/head//3 = 0
2014-06-12 14:51:32.949560 7f9d60bd2700 10 stat 3.3a_head/607603a/rbd_data.10ab6b8b4567.000000000000382f/head//3
2014-06-12 14:51:32.949702 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 607603a/rbd_data.10ab6b8b4567.000000000000382f/head//3
2014-06-12 14:51:32.961830 7f9d60bd2700 10 getattrs 3.3a_head/607603a/rbd_data.10ab6b8b4567.000000000000382f/head//3 = 0
2014-06-12 14:51:32.961840 7f9d60bd2700 10 stat 3.3a_head/5d87603a/rbd_data.10b16b8b4567.0000000000002596/head//3
2014-06-12 14:51:32.961949 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5d87603a/rbd_data.10b16b8b4567.0000000000002596/head//3
2014-06-12 14:51:32.962084 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:32.965798 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:32.990953 7f9d60bd2700 10 getattrs 3.3a_head/5d87603a/rbd_data.10b16b8b4567.0000000000002596/head//3 = 0
2014-06-12 14:51:32.990961 7f9d60bd2700 10 stat 3.3a_head/ac6b603a/rbd_data.10a66b8b4567.00000000000023cd/head//3
2014-06-12 14:51:32.991055 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ac6b603a/rbd_data.10a66b8b4567.00000000000023cd/head//3
2014-06-12 14:51:32.991157 7f9d60bd2700 10 getattrs 3.3a_head/ac6b603a/rbd_data.10a66b8b4567.00000000000023cd/head//3 = 0
2014-06-12 14:51:32.991162 7f9d60bd2700 10 stat 3.3a_head/5dcb603a/rbd_data.10956b8b4567.0000000000000693/head//3
2014-06-12 14:51:32.991248 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5dcb603a/rbd_data.10956b8b4567.0000000000000693/head//3
2014-06-12 14:51:33.035234 7f9d60bd2700 10 getattrs 3.3a_head/5dcb603a/rbd_data.10956b8b4567.0000000000000693/head//3 = 0
2014-06-12 14:51:33.035251 7f9d60bd2700 10 stat 3.3a_head/c74d603a/rbd_data.10956b8b4567.00000000000007af/head//3
2014-06-12 14:51:33.035384 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c74d603a/rbd_data.10956b8b4567.00000000000007af/head//3
2014-06-12 14:51:33.068353 7f9d60bd2700 10 getattrs 3.3a_head/c74d603a/rbd_data.10956b8b4567.00000000000007af/head//3 = 0
2014-06-12 14:51:33.068376 7f9d60bd2700 10 stat 3.3a_head/b1ce603a/rbd_data.10986b8b4567.0000000000001252/head//3
2014-06-12 14:51:33.068533 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid b1ce603a/rbd_data.10986b8b4567.0000000000001252/head//3
2014-06-12 14:51:33.080402 7f9d60bd2700 10 getattrs 3.3a_head/b1ce603a/rbd_data.10986b8b4567.0000000000001252/head//3 = 0
2014-06-12 14:51:33.080418 7f9d60bd2700 10 stat 3.3a_head/e61703a/rbd_data.109b6b8b4567.0000000000002864/head//3
2014-06-12 14:51:33.080574 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid e61703a/rbd_data.109b6b8b4567.0000000000002864/head//3
2014-06-12 14:51:33.092955 7f9d60bd2700 10 getattrs 3.3a_head/e61703a/rbd_data.109b6b8b4567.0000000000002864/head//3 = 0
2014-06-12 14:51:33.092962 7f9d60bd2700 10 stat 3.3a_head/4373703a/rbd_data.10aa6b8b4567.0000000000002317/head//3
2014-06-12 14:51:33.093055 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 4373703a/rbd_data.10aa6b8b4567.0000000000002317/head//3
2014-06-12 14:51:33.119874 7f9d60bd2700 10 getattrs 3.3a_head/4373703a/rbd_data.10aa6b8b4567.0000000000002317/head//3 = 0
2014-06-12 14:51:33.119884 7f9d60bd2700 10 stat 3.3a_head/d6a3703a/rbd_data.10a76b8b4567.00000000000016a6/head//3
2014-06-12 14:51:33.119989 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid d6a3703a/rbd_data.10a76b8b4567.00000000000016a6/head//3
2014-06-12 14:51:33.121056 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:33.132729 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:33.142164 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:33.151581 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:33.151724 7f9d60bd2700 10 getattrs 3.3a_head/d6a3703a/rbd_data.10a76b8b4567.00000000000016a6/head//3 = 0
2014-06-12 14:51:33.151745 7f9d60bd2700 10 stat 3.3a_head/4c47703a/rbd_data.10946b8b4567.0000000000003536/head//3
2014-06-12 14:51:33.151951 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 4c47703a/rbd_data.10946b8b4567.0000000000003536/head//3
2014-06-12 14:51:33.167154 7f9d60bd2700 10 getattrs 3.3a_head/4c47703a/rbd_data.10946b8b4567.0000000000003536/head//3 = 0
2014-06-12 14:51:33.167174 7f9d60bd2700 10 stat 3.3a_head/7d97703a/rbd_data.109d6b8b4567.000000000000212b/head//3
2014-06-12 14:51:33.167361 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 7d97703a/rbd_data.109d6b8b4567.000000000000212b/head//3
2014-06-12 14:51:33.180042 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:33.199622 7f9d60bd2700 10 getattrs 3.3a_head/7d97703a/rbd_data.109d6b8b4567.000000000000212b/head//3 = 0
2014-06-12 14:51:33.199629 7f9d60bd2700 10 stat 3.3a_head/4659703a/rbd_data.10946b8b4567.00000000000034d1/head//3
2014-06-12 14:51:33.199723 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 4659703a/rbd_data.10946b8b4567.00000000000034d1/head//3
2014-06-12 14:51:33.214029 7f9d60bd2700 10 getattrs 3.3a_head/4659703a/rbd_data.10946b8b4567.00000000000034d1/head//3 = 0
2014-06-12 14:51:33.214043 7f9d60bd2700 10 stat 3.3a_head/eb7a703a/rbd_data.10ab6b8b4567.0000000000002ee5/head//3
2014-06-12 14:51:33.214198 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid eb7a703a/rbd_data.10ab6b8b4567.0000000000002ee5/head//3
2014-06-12 14:51:33.214324 7f9d60bd2700 10 getattrs 3.3a_head/eb7a703a/rbd_data.10ab6b8b4567.0000000000002ee5/head//3 = 0
2014-06-12 14:51:33.214333 7f9d60bd2700 10 stat 3.3a_head/37ea703a/rbd_data.10a46b8b4567.0000000000003307/head//3
2014-06-12 14:51:33.214474 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 37ea703a/rbd_data.10a46b8b4567.0000000000003307/head//3
2014-06-12 14:51:33.218765 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:33.225199 7f9d60bd2700 10 getattrs 3.3a_head/37ea703a/rbd_data.10a46b8b4567.0000000000003307/head//3 = 0
2014-06-12 14:51:33.225213 7f9d60bd2700 10 stat 3.3a_head/8380803a/rbd_data.10966b8b4567.0000000000001ac4/head//3
2014-06-12 14:51:33.225368 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 8380803a/rbd_data.10966b8b4567.0000000000001ac4/head//3
2014-06-12 14:51:33.244561 7f9d60bd2700 10 getattrs 3.3a_head/8380803a/rbd_data.10966b8b4567.0000000000001ac4/head//3 = 0
2014-06-12 14:51:33.244572 7f9d60bd2700 10 stat 3.3a_head/c3e0803a/rbd_data.10a26b8b4567.00000000000006af/head//3
2014-06-12 14:51:33.244667 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c3e0803a/rbd_data.10a26b8b4567.00000000000006af/head//3
2014-06-12 14:51:33.287085 7f9d60bd2700 10 getattrs 3.3a_head/c3e0803a/rbd_data.10a26b8b4567.00000000000006af/head//3 = 0
2014-06-12 14:51:33.287101 7f9d60bd2700 10 stat 3.3a_head/7681803a/rbd_data.10b36b8b4567.0000000000000b52/head//3
2014-06-12 14:51:33.287265 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 7681803a/rbd_data.10b36b8b4567.0000000000000b52/head//3
2014-06-12 14:51:33.298209 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:33.306379 7f9d44101700  5 -- op tracker -- , seq: 748584, time: 2014-06-12 14:51:33.306275, event: header_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.306398 7f9d44101700  5 -- op tracker -- , seq: 748584, time: 2014-06-12 14:51:33.306277, event: throttled, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.306426 7f9d44101700  5 -- op tracker -- , seq: 748584, time: 2014-06-12 14:51:33.306335, event: all_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.306436 7f9d44101700  5 -- op tracker -- , seq: 748584, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.306442 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:33.306450 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:33.306459 7f9d44101700 15 osd.2 173 enqueue_op 0x6f4b400 prio 127 cost 10717 latency 0.000184 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:33.306522 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:33.338301 7f9d60bd2700 10 getattrs 3.3a_head/7681803a/rbd_data.10b36b8b4567.0000000000000b52/head//3 = 0
2014-06-12 14:51:33.338325 7f9d60bd2700 10 stat 3.3a_head/802803a/rbd_data.10a06b8b4567.0000000000000a8e/head//3
2014-06-12 14:51:33.338449 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 802803a/rbd_data.10a06b8b4567.0000000000000a8e/head//3
2014-06-12 14:51:33.341399 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:33.347256 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:33.370254 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:33.370313 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:33.374634 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:33.378134 7f9d60bd2700 10 getattrs 3.3a_head/802803a/rbd_data.10a06b8b4567.0000000000000a8e/head//3 = 0
2014-06-12 14:51:33.378145 7f9d60bd2700 10 collection_getattrs 3.3a_head
2014-06-12 14:51:33.378218 7f9d60bd2700 10 collection_getattrs 3.3a_head = 0
2014-06-12 14:51:33.378222 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:33.378253 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:33.378260 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:33.378271 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:33.378274 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:33.378282 7f9d653db700 10 osd.2 173 dequeue_op 0x6f4b400 prio 127 cost 10717 latency 0.072006 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]
2014-06-12 14:51:33.378319 7f9d653db700  5 -- op tracker -- , seq: 748584, time: 2014-06-12 14:51:33.378313, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.378359 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] handle_message: 0x6f4b400
2014-06-12 14:51:33.378387 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:33.378424 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:33.378456 7f9d653db700  5 -- op tracker -- , seq: 748584, time: 2014-06-12 14:51:33.378446, event: started, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.378480 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:33.378623 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] map version is 173'2964
2014-06-12 14:51:33.378650 7f9d653db700 10 osd.2 173 dequeue_op 0x6f4b400 finish
2014-06-12 14:51:33.378670 7f9d653db700  5 -- op tracker -- , seq: 748584, time: 2014-06-12 14:51:33.378659, event: done, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.378688 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:33.378712 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:33.378728 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:33.378702 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:33.378731 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:33.378739 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:33.378769 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:33.378776 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:33.378784 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:33.378807 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8c22503a/rbd_data.109c6b8b4567.00000000000034b8/head//3, auth == maps.end()
2014-06-12 14:51:33.378827 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8c22503a/rbd_data.109c6b8b4567.00000000000034b8/head//3
2014-06-12 14:51:33.378842 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj eff2503a/rbd_data.109b6b8b4567.0000000000003770/head//3, auth == maps.end()
2014-06-12 14:51:33.378852 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj eff2503a/rbd_data.109b6b8b4567.0000000000003770/head//3
2014-06-12 14:51:33.378863 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6bea503a/rbd_data.10946b8b4567.0000000000002b6e/head//3, auth == maps.end()
2014-06-12 14:51:33.378872 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6bea503a/rbd_data.10946b8b4567.0000000000002b6e/head//3
2014-06-12 14:51:33.378882 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj adbd503a/rbd_data.10b36b8b4567.00000000000004d1/head//3, auth == maps.end()
2014-06-12 14:51:33.378892 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj adbd503a/rbd_data.10b36b8b4567.00000000000004d1/head//3
2014-06-12 14:51:33.378902 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 44ef503a/rbd_data.10a46b8b4567.0000000000000ab0/head//3, auth == maps.end()
2014-06-12 14:51:33.378912 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 44ef503a/rbd_data.10a46b8b4567.0000000000000ab0/head//3
2014-06-12 14:51:33.378922 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d0e5603a/rbd_data.108e6b8b4567.000000000000158f/head//3, auth == maps.end()
2014-06-12 14:51:33.378931 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d0e5603a/rbd_data.108e6b8b4567.000000000000158f/head//3
2014-06-12 14:51:33.378941 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj abc6603a/rbd_data.10ab6b8b4567.00000000000025d5/head//3, auth == maps.end()
2014-06-12 14:51:33.378951 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj abc6603a/rbd_data.10ab6b8b4567.00000000000025d5/head//3
2014-06-12 14:51:33.378961 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 607603a/rbd_data.10ab6b8b4567.000000000000382f/head//3, auth == maps.end()
2014-06-12 14:51:33.378970 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 607603a/rbd_data.10ab6b8b4567.000000000000382f/head//3
2014-06-12 14:51:33.378981 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5d87603a/rbd_data.10b16b8b4567.0000000000002596/head//3, auth == maps.end()
2014-06-12 14:51:33.378990 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5d87603a/rbd_data.10b16b8b4567.0000000000002596/head//3
2014-06-12 14:51:33.379001 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ac6b603a/rbd_data.10a66b8b4567.00000000000023cd/head//3, auth == maps.end()
2014-06-12 14:51:33.379013 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ac6b603a/rbd_data.10a66b8b4567.00000000000023cd/head//3
2014-06-12 14:51:33.379025 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5dcb603a/rbd_data.10956b8b4567.0000000000000693/head//3, auth == maps.end()
2014-06-12 14:51:33.379037 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5dcb603a/rbd_data.10956b8b4567.0000000000000693/head//3
2014-06-12 14:51:33.379047 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c74d603a/rbd_data.10956b8b4567.00000000000007af/head//3, auth == maps.end()
2014-06-12 14:51:33.379058 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c74d603a/rbd_data.10956b8b4567.00000000000007af/head//3
2014-06-12 14:51:33.379069 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b1ce603a/rbd_data.10986b8b4567.0000000000001252/head//3, auth == maps.end()
2014-06-12 14:51:33.379079 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b1ce603a/rbd_data.10986b8b4567.0000000000001252/head//3
2014-06-12 14:51:33.379089 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e61703a/rbd_data.109b6b8b4567.0000000000002864/head//3, auth == maps.end()
2014-06-12 14:51:33.379099 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e61703a/rbd_data.109b6b8b4567.0000000000002864/head//3
2014-06-12 14:51:33.379109 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4373703a/rbd_data.10aa6b8b4567.0000000000002317/head//3, auth == maps.end()
2014-06-12 14:51:33.379118 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4373703a/rbd_data.10aa6b8b4567.0000000000002317/head//3
2014-06-12 14:51:33.379128 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d6a3703a/rbd_data.10a76b8b4567.00000000000016a6/head//3, auth == maps.end()
2014-06-12 14:51:33.379138 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d6a3703a/rbd_data.10a76b8b4567.00000000000016a6/head//3
2014-06-12 14:51:33.379148 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4c47703a/rbd_data.10946b8b4567.0000000000003536/head//3, auth == maps.end()
2014-06-12 14:51:33.379162 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4c47703a/rbd_data.10946b8b4567.0000000000003536/head//3
2014-06-12 14:51:33.379175 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7d97703a/rbd_data.109d6b8b4567.000000000000212b/head//3, auth == maps.end()
2014-06-12 14:51:33.379186 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7d97703a/rbd_data.109d6b8b4567.000000000000212b/head//3
2014-06-12 14:51:33.379197 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4659703a/rbd_data.10946b8b4567.00000000000034d1/head//3, auth == maps.end()
2014-06-12 14:51:33.379207 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4659703a/rbd_data.10946b8b4567.00000000000034d1/head//3
2014-06-12 14:51:33.379219 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj eb7a703a/rbd_data.10ab6b8b4567.0000000000002ee5/head//3, auth == maps.end()
2014-06-12 14:51:33.379230 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj eb7a703a/rbd_data.10ab6b8b4567.0000000000002ee5/head//3
2014-06-12 14:51:33.379241 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 37ea703a/rbd_data.10a46b8b4567.0000000000003307/head//3, auth == maps.end()
2014-06-12 14:51:33.379252 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 37ea703a/rbd_data.10a46b8b4567.0000000000003307/head//3
2014-06-12 14:51:33.379263 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8380803a/rbd_data.10966b8b4567.0000000000001ac4/head//3, auth == maps.end()
2014-06-12 14:51:33.379274 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8380803a/rbd_data.10966b8b4567.0000000000001ac4/head//3
2014-06-12 14:51:33.379284 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c3e0803a/rbd_data.10a26b8b4567.00000000000006af/head//3, auth == maps.end()
2014-06-12 14:51:33.379296 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c3e0803a/rbd_data.10a26b8b4567.00000000000006af/head//3
2014-06-12 14:51:33.379306 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7681803a/rbd_data.10b36b8b4567.0000000000000b52/head//3, auth == maps.end()
2014-06-12 14:51:33.379317 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7681803a/rbd_data.10b36b8b4567.0000000000000b52/head//3
2014-06-12 14:51:33.379328 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 802803a/rbd_data.10a06b8b4567.0000000000000a8e/head//3, auth == maps.end()
2014-06-12 14:51:33.379339 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 802803a/rbd_data.10a06b8b4567.0000000000000a8e/head//3
2014-06-12 14:51:33.379352 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] 
2014-06-12 14:51:33.379361 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub
2014-06-12 14:51:33.379374 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  802803a/rbd_data.10a06b8b4567.0000000000000a8e/head//3 802803a/rbd_data.10a06b8b4567.0000000000000a8e/head//3(173'2819 client.4297.0:133915 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2819)
2014-06-12 14:51:33.379391 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  7681803a/rbd_data.10b36b8b4567.0000000000000b52/head//3 7681803a/rbd_data.10b36b8b4567.0000000000000b52/head//3(173'2824 client.4316.0:137318 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2824)
2014-06-12 14:51:33.379405 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c3e0803a/rbd_data.10a26b8b4567.00000000000006af/head//3 c3e0803a/rbd_data.10a26b8b4567.00000000000006af/head//3(173'2724 client.4299.0:130294 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2724)
2014-06-12 14:51:33.379420 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  8380803a/rbd_data.10966b8b4567.0000000000001ac4/head//3 8380803a/rbd_data.10966b8b4567.0000000000001ac4/head//3(173'1184 client.4287.0:56121 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1184)
2014-06-12 14:51:33.379434 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  37ea703a/rbd_data.10a46b8b4567.0000000000003307/head//3 37ea703a/rbd_data.10a46b8b4567.0000000000003307/head//3(173'2145 client.4301.0:102652 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2145)
2014-06-12 14:51:33.379448 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  eb7a703a/rbd_data.10ab6b8b4567.0000000000002ee5/head//3 eb7a703a/rbd_data.10ab6b8b4567.0000000000002ee5/head//3(173'1983 client.4308.0:94348 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1983)
2014-06-12 14:51:33.379462 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  4659703a/rbd_data.10946b8b4567.00000000000034d1/head//3 4659703a/rbd_data.10946b8b4567.00000000000034d1/head//3(173'2177 client.4285.0:103605 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2177)
2014-06-12 14:51:33.379477 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  7d97703a/rbd_data.109d6b8b4567.000000000000212b/head//3 7d97703a/rbd_data.109d6b8b4567.000000000000212b/head//3(173'1399 client.4294.0:68810 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1399)
2014-06-12 14:51:33.379491 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  4c47703a/rbd_data.10946b8b4567.0000000000003536/head//3 4c47703a/rbd_data.10946b8b4567.0000000000003536/head//3(173'2190 client.4285.0:104302 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2190)
2014-06-12 14:51:33.379505 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  d6a3703a/rbd_data.10a76b8b4567.00000000000016a6/head//3 d6a3703a/rbd_data.10a76b8b4567.00000000000016a6/head//3(173'1114 client.4304.0:51016 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1114)
2014-06-12 14:51:33.379519 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  4373703a/rbd_data.10aa6b8b4567.0000000000002317/head//3 4373703a/rbd_data.10aa6b8b4567.0000000000002317/head//3(173'1512 client.4307.0:74845 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1512)
2014-06-12 14:51:33.379533 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  e61703a/rbd_data.109b6b8b4567.0000000000002864/head//3 e61703a/rbd_data.109b6b8b4567.0000000000002864/head//3(173'1884 client.4292.0:80224 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1884)
2014-06-12 14:51:33.379548 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  b1ce603a/rbd_data.10986b8b4567.0000000000001252/head//3 b1ce603a/rbd_data.10986b8b4567.0000000000001252/head//3(173'916 client.4289.0:41546 wrlock_by=unknown.0.0:0 dirty s 4194304 uv916)
2014-06-12 14:51:33.379562 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c74d603a/rbd_data.10956b8b4567.00000000000007af/head//3 c74d603a/rbd_data.10956b8b4567.00000000000007af/head//3(173'2740 client.4286.0:129992 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2740)
2014-06-12 14:51:33.379576 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5dcb603a/rbd_data.10956b8b4567.0000000000000693/head//3 5dcb603a/rbd_data.10956b8b4567.0000000000000693/head//3(173'2699 client.4286.0:128119 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2699)
2014-06-12 14:51:33.379593 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ac6b603a/rbd_data.10a66b8b4567.00000000000023cd/head//3 ac6b603a/rbd_data.10a66b8b4567.00000000000023cd/head//3(173'1577 client.4303.0:72089 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1577)
2014-06-12 14:51:33.379614 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5d87603a/rbd_data.10b16b8b4567.0000000000002596/head//3 5d87603a/rbd_data.10b16b8b4567.0000000000002596/head//3(173'1747 client.4314.0:77502 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1747)
2014-06-12 14:51:33.379629 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  607603a/rbd_data.10ab6b8b4567.000000000000382f/head//3 607603a/rbd_data.10ab6b8b4567.000000000000382f/head//3(173'2371 client.4308.0:112019 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2371)
2014-06-12 14:51:33.379643 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  abc6603a/rbd_data.10ab6b8b4567.00000000000025d5/head//3 abc6603a/rbd_data.10ab6b8b4567.00000000000025d5/head//3(173'1656 client.4308.0:77734 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1656)
2014-06-12 14:51:33.379658 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  d0e5603a/rbd_data.108e6b8b4567.000000000000158f/head//3 d0e5603a/rbd_data.108e6b8b4567.000000000000158f/head//3(173'1051 client.4279.0:48823 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1051)
2014-06-12 14:51:33.379672 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  44ef503a/rbd_data.10a46b8b4567.0000000000000ab0/head//3 44ef503a/rbd_data.10a46b8b4567.0000000000000ab0/head//3(173'2795 client.4301.0:135598 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2795)
2014-06-12 14:51:33.379686 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  adbd503a/rbd_data.10b36b8b4567.00000000000004d1/head//3 adbd503a/rbd_data.10b36b8b4567.00000000000004d1/head//3(173'2645 client.4316.0:126376 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2645)
2014-06-12 14:51:33.379700 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  6bea503a/rbd_data.10946b8b4567.0000000000002b6e/head//3 6bea503a/rbd_data.10946b8b4567.0000000000002b6e/head//3(173'1848 client.4285.0:86943 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1848)
2014-06-12 14:51:33.379714 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  eff2503a/rbd_data.109b6b8b4567.0000000000003770/head//3 eff2503a/rbd_data.109b6b8b4567.0000000000003770/head//3(173'2339 client.4292.0:107745 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2339)
2014-06-12 14:51:33.379728 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  8c22503a/rbd_data.109c6b8b4567.00000000000034b8/head//3 8c22503a/rbd_data.109c6b8b4567.00000000000034b8/head//3(173'2229 client.4293.0:105074 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2229)
2014-06-12 14:51:33.379739 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:33.379751 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:33.379759 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:33.379761 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:33.379762 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:33.379804 7f9d60bd2700 10 collection_list_partial 3.3a_head start:ef72803a//0//-1 is_max:0
2014-06-12 14:51:33.380042 7f9d60bd2700 20 objects: [a825803a/rbd_data.10956b8b4567.0000000000000185/head//3,9a06803a/rbd_data.10a26b8b4567.000000000000395b/head//3,6036803a/rbd_data.10946b8b4567.0000000000002208/head//3,dd38803a/rbd_data.10986b8b4567.000000000000160f/head//3,cc68803a/rbd_data.10b26b8b4567.000000000000313d/head//3,39f8803a/rbd_data.109b6b8b4567.0000000000001e5e/head//3,a4ca803a/rbd_data.10b06b8b4567.0000000000001477/head//3,569b803a/rbd_data.10a66b8b4567.0000000000002682/head//3,963d803a/rbd_data.10a36b8b4567.0000000000001019/head//3,94cf803a/rbd_data.10926b8b4567.00000000000029f2/head//3,faef803a/rbd_data.10906b8b4567.00000000000016bb/head//3,8731903a/rbd_data.10a66b8b4567.0000000000003929/head//3,c947903a/rbd_data.10a26b8b4567.0000000000002381/head//3,eba0a03a/rbd_data.108e6b8b4567.0000000000001462/head//3,aba5a03a/rbd_data.10aa6b8b4567.00000000000002e9/head//3,3097a03a/rbd_data.10a86b8b4567.000000000000011b/head//3,bd18a03a/rbd_data.10ab6b8b4567.00000000000039e8/head//3,bc79a03a/rbd_data.109e6b8b4567.00000000000031e8/head//3,3a5aa03a/rbd_data.10966b8b4567.0000000000001b26/head//3,559aa03a/rbd_data.10b26b8b4567.000000000000258e/head//3,8deea03a/rbd_data.10af6b8b4567.0000000000002997/head//3,efc3b03a/rbd_data.10a46b8b4567.000000000000366c/head//3,df15b03a/rbd_data.10986b8b4567.00000000000007c8/head//3,45a6b03a/rbd_data.10a36b8b4567.0000000000003208/head//3,4d99b03a/rbd_data.10a46b8b4567.0000000000000406/head//3]
2014-06-12 14:51:33.380576 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:33.380604 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:33.380620 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:33.380630 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:33.380637 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:33.380643 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:33.380649 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_map_chunk [ef72803a//0//-1,dc9ab03a//0//-1)
2014-06-12 14:51:33.380657 7f9d60bd2700 10 collection_list_partial 3.3a_head start:ef72803a//0//-1 is_max:0
2014-06-12 14:51:33.394777 7f9d60bd2700 20 objects: [a825803a/rbd_data.10956b8b4567.0000000000000185/head//3,9a06803a/rbd_data.10a26b8b4567.000000000000395b/head//3,6036803a/rbd_data.10946b8b4567.0000000000002208/head//3,dd38803a/rbd_data.10986b8b4567.000000000000160f/head//3,cc68803a/rbd_data.10b26b8b4567.000000000000313d/head//3,39f8803a/rbd_data.109b6b8b4567.0000000000001e5e/head//3,a4ca803a/rbd_data.10b06b8b4567.0000000000001477/head//3,569b803a/rbd_data.10a66b8b4567.0000000000002682/head//3,963d803a/rbd_data.10a36b8b4567.0000000000001019/head//3,94cf803a/rbd_data.10926b8b4567.00000000000029f2/head//3,faef803a/rbd_data.10906b8b4567.00000000000016bb/head//3,8731903a/rbd_data.10a66b8b4567.0000000000003929/head//3,c947903a/rbd_data.10a26b8b4567.0000000000002381/head//3,eba0a03a/rbd_data.108e6b8b4567.0000000000001462/head//3,aba5a03a/rbd_data.10aa6b8b4567.00000000000002e9/head//3,3097a03a/rbd_data.10a86b8b4567.000000000000011b/head//3,bd18a03a/rbd_data.10ab6b8b4567.00000000000039e8/head//3,bc79a03a/rbd_data.109e6b8b4567.00000000000031e8/head//3,3a5aa03a/rbd_data.10966b8b4567.0000000000001b26/head//3,559aa03a/rbd_data.10b26b8b4567.000000000000258e/head//3,8deea03a/rbd_data.10af6b8b4567.0000000000002997/head//3,efc3b03a/rbd_data.10a46b8b4567.000000000000366c/head//3,df15b03a/rbd_data.10986b8b4567.00000000000007c8/head//3,45a6b03a/rbd_data.10a36b8b4567.0000000000003208/head//3,4d99b03a/rbd_data.10a46b8b4567.0000000000000406/head//3,dc9ab03a/rbd_data.10b46b8b4567.0000000000000fcf/head//3,13ccb03a/rbd_data.10936b8b4567.0000000000000120/head//3,ecccb03a/rbd_data.10966b8b4567.0000000000000766/head//3,b70fb03a/rbd_data.10a16b8b4567.0000000000003967/head//3,342fb03a/rbd_data.10a56b8b4567.000000000000138d/head//3,5d10c03a/rbd_data.10946b8b4567.000000000000074c/head//3,dc50c03a/rbd_data.109f6b8b4567.000000000000362e/head//3,2a90c03a/rbd_data.10906b8b4567.0000000000001ffb/head//3,5e31c03a/rbd_data.10996b8b4567.0000000000003270/head//3,1e72c03a/rbd_data.10a96b8b4567.0000000000000187/head//3,9143c03a/rbd_data.108d6b8b4567.000000000000203c/head//3,a643c03a/rbd_data.10946b8b4567.000000000000004d/head//3,e614c03a/rbd_data.108d6b8b4567.00000000000023e0/head//3,7544c03a/rbd_data.109b6b8b4567.000000000000175f/head//3,1da5c03a/rbd_data.10906b8b4567.0000000000003525/head//3,d729c03a/rbd_data.10a16b8b4567.000000000000188f/head//3,4b29c03a/rbd_data.10936b8b4567.0000000000001bf0/head//3,6adbc03a/rbd_data.10916b8b4567.0000000000003453/head//3,f30cc03a/rbd_data.10a36b8b4567.0000000000003112/head//3,40e0d03a/rbd_data.108f6b8b4567.0000000000002ab7/head//3,bac1d03a/rbd_data.10a26b8b4567.0000000000003a9e/head//3,2145d03a/rbd_data.10956b8b4567.0000000000000f20/head//3,5b16d03a/rbd_data.10aa6b8b4567.0000000000003380/head//3,eb56d03a/rbd_data.10a06b8b4567.0000000000001721/head//3,bd09d03a/rbd_data.10ad6b8b4567.0000000000000147/head//3,237ad03a/rbd_data.109c6b8b4567.0000000000000bc2/head//3,7fbad03a/rbd_data.10af6b8b4567.0000000000002247/head//3,159bd03a/rbd_data.10a46b8b4567.0000000000002215/head//3,c79cd03a/rbd_data.10ac6b8b4567.0000000000000443/head//3,d3ecd03a/rbd_data.10976b8b4567.00000000000032c2/head//3,e63dd03a/rbd_data.10b36b8b4567.0000000000003bae/head//3,d36dd03a/rbd_data.10a86b8b4567.0000000000003a0f/head//3,4e8dd03a/rbd_data.10a06b8b4567.00000000000025c9/head//3,c6bfd03a/rbd_data.10a96b8b4567.0000000000001b39/head//3,a6efd03a/rbd_data.10976b8b4567.00000000000009a1/head//3,84f0e03a/rbd_data.10936b8b4567.00000000000023b4/head//3,a035e03a/rbd_data.10956b8b4567.0000000000001e90/head//3,ad09e03a/rbd_data.10906b8b4567.0000000000000c16/head//3,47cce03a/rbd_data.10a76b8b4567.0000000000000889/head//3]
2014-06-12 14:51:33.394914 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:33.394942 7f9d60bd2700 10 stat 3.3a_head/a825803a/rbd_data.10956b8b4567.0000000000000185/head//3
2014-06-12 14:51:33.395180 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a825803a/rbd_data.10956b8b4567.0000000000000185/head//3
2014-06-12 14:51:33.448521 7f9d60bd2700 10 getattrs 3.3a_head/a825803a/rbd_data.10956b8b4567.0000000000000185/head//3 = 0
2014-06-12 14:51:33.448546 7f9d60bd2700 10 stat 3.3a_head/9a06803a/rbd_data.10a26b8b4567.000000000000395b/head//3
2014-06-12 14:51:33.448706 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 9a06803a/rbd_data.10a26b8b4567.000000000000395b/head//3
2014-06-12 14:51:33.466095 7f9d60bd2700 10 getattrs 3.3a_head/9a06803a/rbd_data.10a26b8b4567.000000000000395b/head//3 = 0
2014-06-12 14:51:33.466109 7f9d60bd2700 10 stat 3.3a_head/6036803a/rbd_data.10946b8b4567.0000000000002208/head//3
2014-06-12 14:51:33.466280 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 6036803a/rbd_data.10946b8b4567.0000000000002208/head//3
2014-06-12 14:51:33.483907 7f9d60bd2700 10 getattrs 3.3a_head/6036803a/rbd_data.10946b8b4567.0000000000002208/head//3 = 0
2014-06-12 14:51:33.483921 7f9d60bd2700 10 stat 3.3a_head/dd38803a/rbd_data.10986b8b4567.000000000000160f/head//3
2014-06-12 14:51:33.484087 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid dd38803a/rbd_data.10986b8b4567.000000000000160f/head//3
2014-06-12 14:51:33.501182 7f9d60bd2700 10 getattrs 3.3a_head/dd38803a/rbd_data.10986b8b4567.000000000000160f/head//3 = 0
2014-06-12 14:51:33.501188 7f9d60bd2700 10 stat 3.3a_head/cc68803a/rbd_data.10b26b8b4567.000000000000313d/head//3
2014-06-12 14:51:33.501282 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid cc68803a/rbd_data.10b26b8b4567.000000000000313d/head//3
2014-06-12 14:51:33.513138 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:33.526837 7f9d60bd2700 10 getattrs 3.3a_head/cc68803a/rbd_data.10b26b8b4567.000000000000313d/head//3 = 0
2014-06-12 14:51:33.526849 7f9d60bd2700 10 stat 3.3a_head/39f8803a/rbd_data.109b6b8b4567.0000000000001e5e/head//3
2014-06-12 14:51:33.526947 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 39f8803a/rbd_data.109b6b8b4567.0000000000001e5e/head//3
2014-06-12 14:51:33.527041 7f9d60bd2700 10 getattrs 3.3a_head/39f8803a/rbd_data.109b6b8b4567.0000000000001e5e/head//3 = 0
2014-06-12 14:51:33.527047 7f9d60bd2700 10 stat 3.3a_head/a4ca803a/rbd_data.10b06b8b4567.0000000000001477/head//3
2014-06-12 14:51:33.527132 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a4ca803a/rbd_data.10b06b8b4567.0000000000001477/head//3
2014-06-12 14:51:33.540904 7f9d60bd2700 10 getattrs 3.3a_head/a4ca803a/rbd_data.10b06b8b4567.0000000000001477/head//3 = 0
2014-06-12 14:51:33.540914 7f9d60bd2700 10 stat 3.3a_head/569b803a/rbd_data.10a66b8b4567.0000000000002682/head//3
2014-06-12 14:51:33.541011 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 569b803a/rbd_data.10a66b8b4567.0000000000002682/head//3
2014-06-12 14:51:33.547085 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:33.554490 7f9d60bd2700 10 getattrs 3.3a_head/569b803a/rbd_data.10a66b8b4567.0000000000002682/head//3 = 0
2014-06-12 14:51:33.554499 7f9d60bd2700 10 stat 3.3a_head/963d803a/rbd_data.10a36b8b4567.0000000000001019/head//3
2014-06-12 14:51:33.554600 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 963d803a/rbd_data.10a36b8b4567.0000000000001019/head//3
2014-06-12 14:51:33.566678 7f9d60bd2700 10 getattrs 3.3a_head/963d803a/rbd_data.10a36b8b4567.0000000000001019/head//3 = 0
2014-06-12 14:51:33.566686 7f9d60bd2700 10 stat 3.3a_head/94cf803a/rbd_data.10926b8b4567.00000000000029f2/head//3
2014-06-12 14:51:33.566786 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 94cf803a/rbd_data.10926b8b4567.00000000000029f2/head//3
2014-06-12 14:51:33.582247 7f9d60bd2700 10 getattrs 3.3a_head/94cf803a/rbd_data.10926b8b4567.00000000000029f2/head//3 = 0
2014-06-12 14:51:33.582256 7f9d60bd2700 10 stat 3.3a_head/faef803a/rbd_data.10906b8b4567.00000000000016bb/head//3
2014-06-12 14:51:33.582358 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid faef803a/rbd_data.10906b8b4567.00000000000016bb/head//3
2014-06-12 14:51:33.594716 7f9d60bd2700 10 getattrs 3.3a_head/faef803a/rbd_data.10906b8b4567.00000000000016bb/head//3 = 0
2014-06-12 14:51:33.594727 7f9d60bd2700 10 stat 3.3a_head/8731903a/rbd_data.10a66b8b4567.0000000000003929/head//3
2014-06-12 14:51:33.594826 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 8731903a/rbd_data.10a66b8b4567.0000000000003929/head//3
2014-06-12 14:51:33.608029 7f9d60bd2700 10 getattrs 3.3a_head/8731903a/rbd_data.10a66b8b4567.0000000000003929/head//3 = 0
2014-06-12 14:51:33.608038 7f9d60bd2700 10 stat 3.3a_head/c947903a/rbd_data.10a26b8b4567.0000000000002381/head//3
2014-06-12 14:51:33.608137 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c947903a/rbd_data.10a26b8b4567.0000000000002381/head//3
2014-06-12 14:51:33.625484 7f9d60bd2700 10 getattrs 3.3a_head/c947903a/rbd_data.10a26b8b4567.0000000000002381/head//3 = 0
2014-06-12 14:51:33.625502 7f9d60bd2700 10 stat 3.3a_head/eba0a03a/rbd_data.108e6b8b4567.0000000000001462/head//3
2014-06-12 14:51:33.625675 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid eba0a03a/rbd_data.108e6b8b4567.0000000000001462/head//3
2014-06-12 14:51:33.650263 7f9d60bd2700 10 getattrs 3.3a_head/eba0a03a/rbd_data.108e6b8b4567.0000000000001462/head//3 = 0
2014-06-12 14:51:33.650271 7f9d60bd2700 10 stat 3.3a_head/aba5a03a/rbd_data.10aa6b8b4567.00000000000002e9/head//3
2014-06-12 14:51:33.650360 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid aba5a03a/rbd_data.10aa6b8b4567.00000000000002e9/head//3
2014-06-12 14:51:33.688884 7f9d60bd2700 10 getattrs 3.3a_head/aba5a03a/rbd_data.10aa6b8b4567.00000000000002e9/head//3 = 0
2014-06-12 14:51:33.688895 7f9d60bd2700 10 stat 3.3a_head/3097a03a/rbd_data.10a86b8b4567.000000000000011b/head//3
2014-06-12 14:51:33.688993 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 3097a03a/rbd_data.10a86b8b4567.000000000000011b/head//3
2014-06-12 14:51:33.735065 7f9d60bd2700 10 getattrs 3.3a_head/3097a03a/rbd_data.10a86b8b4567.000000000000011b/head//3 = 0
2014-06-12 14:51:33.735077 7f9d60bd2700 10 stat 3.3a_head/bd18a03a/rbd_data.10ab6b8b4567.00000000000039e8/head//3
2014-06-12 14:51:33.735186 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid bd18a03a/rbd_data.10ab6b8b4567.00000000000039e8/head//3
2014-06-12 14:51:33.748043 7f9d60bd2700 10 getattrs 3.3a_head/bd18a03a/rbd_data.10ab6b8b4567.00000000000039e8/head//3 = 0
2014-06-12 14:51:33.748055 7f9d60bd2700 10 stat 3.3a_head/bc79a03a/rbd_data.109e6b8b4567.00000000000031e8/head//3
2014-06-12 14:51:33.748164 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid bc79a03a/rbd_data.109e6b8b4567.00000000000031e8/head//3
2014-06-12 14:51:33.749392 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:33.749415 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:51:33.749419 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:51:33.749424 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:51:33.749427 7f9d84175700 10 osd.2 173 send_pg_stats - 2 pgs updated
2014-06-12 14:51:33.749475 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:33.749479 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:33.749481 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:33.749514 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:33.766166 7f9d60bd2700 10 getattrs 3.3a_head/bc79a03a/rbd_data.109e6b8b4567.00000000000031e8/head//3 = 0
2014-06-12 14:51:33.766176 7f9d60bd2700 10 stat 3.3a_head/3a5aa03a/rbd_data.10966b8b4567.0000000000001b26/head//3
2014-06-12 14:51:33.766292 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 3a5aa03a/rbd_data.10966b8b4567.0000000000001b26/head//3
2014-06-12 14:51:33.793332 7f9d60bd2700 10 getattrs 3.3a_head/3a5aa03a/rbd_data.10966b8b4567.0000000000001b26/head//3 = 0
2014-06-12 14:51:33.793357 7f9d60bd2700 10 stat 3.3a_head/559aa03a/rbd_data.10b26b8b4567.000000000000258e/head//3
2014-06-12 14:51:33.793514 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 559aa03a/rbd_data.10b26b8b4567.000000000000258e/head//3
2014-06-12 14:51:33.794474 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:33.794910 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:33.807334 7f9d60bd2700 10 getattrs 3.3a_head/559aa03a/rbd_data.10b26b8b4567.000000000000258e/head//3 = 0
2014-06-12 14:51:33.807341 7f9d60bd2700 10 stat 3.3a_head/8deea03a/rbd_data.10af6b8b4567.0000000000002997/head//3
2014-06-12 14:51:33.807444 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 8deea03a/rbd_data.10af6b8b4567.0000000000002997/head//3
2014-06-12 14:51:33.833704 7f9d60bd2700 10 getattrs 3.3a_head/8deea03a/rbd_data.10af6b8b4567.0000000000002997/head//3 = 0
2014-06-12 14:51:33.833711 7f9d60bd2700 10 stat 3.3a_head/efc3b03a/rbd_data.10a46b8b4567.000000000000366c/head//3
2014-06-12 14:51:33.833816 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid efc3b03a/rbd_data.10a46b8b4567.000000000000366c/head//3
2014-06-12 14:51:33.860649 7f9d60bd2700 10 getattrs 3.3a_head/efc3b03a/rbd_data.10a46b8b4567.000000000000366c/head//3 = 0
2014-06-12 14:51:33.860658 7f9d60bd2700 10 stat 3.3a_head/df15b03a/rbd_data.10986b8b4567.00000000000007c8/head//3
2014-06-12 14:51:33.860752 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid df15b03a/rbd_data.10986b8b4567.00000000000007c8/head//3
2014-06-12 14:51:33.904597 7f9d60bd2700 10 getattrs 3.3a_head/df15b03a/rbd_data.10986b8b4567.00000000000007c8/head//3 = 0
2014-06-12 14:51:33.904615 7f9d60bd2700 10 stat 3.3a_head/45a6b03a/rbd_data.10a36b8b4567.0000000000003208/head//3
2014-06-12 14:51:33.904720 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 45a6b03a/rbd_data.10a36b8b4567.0000000000003208/head//3
2014-06-12 14:51:33.940879 7f9d60bd2700 10 getattrs 3.3a_head/45a6b03a/rbd_data.10a36b8b4567.0000000000003208/head//3 = 0
2014-06-12 14:51:33.940896 7f9d60bd2700 10 stat 3.3a_head/4d99b03a/rbd_data.10a46b8b4567.0000000000000406/head//3
2014-06-12 14:51:33.941002 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 4d99b03a/rbd_data.10a46b8b4567.0000000000000406/head//3
2014-06-12 14:51:33.957214 7f9d44101700  5 -- op tracker -- , seq: 748585, time: 2014-06-12 14:51:33.957138, event: header_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.957225 7f9d44101700  5 -- op tracker -- , seq: 748585, time: 2014-06-12 14:51:33.957140, event: throttled, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.957232 7f9d44101700  5 -- op tracker -- , seq: 748585, time: 2014-06-12 14:51:33.957181, event: all_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.957246 7f9d44101700  5 -- op tracker -- , seq: 748585, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.957250 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:33.957255 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:33.957262 7f9d44101700 15 osd.2 173 enqueue_op 0x6f4a800 prio 127 cost 10717 latency 0.000123 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:33.957324 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:33.976785 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:33.993868 7f9d60bd2700 10 getattrs 3.3a_head/4d99b03a/rbd_data.10a46b8b4567.0000000000000406/head//3 = 0
2014-06-12 14:51:33.993887 7f9d60bd2700 10 collection_getattrs 3.3a_head
2014-06-12 14:51:33.993953 7f9d60bd2700 10 collection_getattrs 3.3a_head = 0
2014-06-12 14:51:33.993958 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:33.993996 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:33.994004 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:33.994015 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:33.994019 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:33.994048 7f9d61bd4700 10 osd.2 173 dequeue_op 0x6f4a800 prio 127 cost 10717 latency 0.036910 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]
2014-06-12 14:51:33.994094 7f9d61bd4700  5 -- op tracker -- , seq: 748585, time: 2014-06-12 14:51:33.994087, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.994119 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] handle_message: 0x6f4a800
2014-06-12 14:51:33.994136 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:33.994156 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:33.994175 7f9d61bd4700  5 -- op tracker -- , seq: 748585, time: 2014-06-12 14:51:33.994169, event: started, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.994183 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:33.994285 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] map version is 173'2964
2014-06-12 14:51:33.994305 7f9d61bd4700 10 osd.2 173 dequeue_op 0x6f4a800 finish
2014-06-12 14:51:33.994317 7f9d61bd4700  5 -- op tracker -- , seq: 748585, time: 2014-06-12 14:51:33.994311, event: done, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:33.994337 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:33.994342 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:33.994343 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:33.994356 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:33.994385 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:33.994394 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:33.994427 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:33.994436 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:33.994443 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:33.994469 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a825803a/rbd_data.10956b8b4567.0000000000000185/head//3, auth == maps.end()
2014-06-12 14:51:33.994487 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a825803a/rbd_data.10956b8b4567.0000000000000185/head//3
2014-06-12 14:51:33.994508 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9a06803a/rbd_data.10a26b8b4567.000000000000395b/head//3, auth == maps.end()
2014-06-12 14:51:33.994522 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9a06803a/rbd_data.10a26b8b4567.000000000000395b/head//3
2014-06-12 14:51:33.994534 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6036803a/rbd_data.10946b8b4567.0000000000002208/head//3, auth == maps.end()
2014-06-12 14:51:33.994545 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6036803a/rbd_data.10946b8b4567.0000000000002208/head//3
2014-06-12 14:51:33.994557 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dd38803a/rbd_data.10986b8b4567.000000000000160f/head//3, auth == maps.end()
2014-06-12 14:51:33.994568 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj dd38803a/rbd_data.10986b8b4567.000000000000160f/head//3
2014-06-12 14:51:33.994579 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cc68803a/rbd_data.10b26b8b4567.000000000000313d/head//3, auth == maps.end()
2014-06-12 14:51:33.994590 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj cc68803a/rbd_data.10b26b8b4567.000000000000313d/head//3
2014-06-12 14:51:33.994601 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 39f8803a/rbd_data.109b6b8b4567.0000000000001e5e/head//3, auth == maps.end()
2014-06-12 14:51:33.994611 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 39f8803a/rbd_data.109b6b8b4567.0000000000001e5e/head//3
2014-06-12 14:51:33.994623 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a4ca803a/rbd_data.10b06b8b4567.0000000000001477/head//3, auth == maps.end()
2014-06-12 14:51:33.994636 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a4ca803a/rbd_data.10b06b8b4567.0000000000001477/head//3
2014-06-12 14:51:33.994650 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 569b803a/rbd_data.10a66b8b4567.0000000000002682/head//3, auth == maps.end()
2014-06-12 14:51:33.994662 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 569b803a/rbd_data.10a66b8b4567.0000000000002682/head//3
2014-06-12 14:51:33.994675 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 963d803a/rbd_data.10a36b8b4567.0000000000001019/head//3, auth == maps.end()
2014-06-12 14:51:33.994687 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 963d803a/rbd_data.10a36b8b4567.0000000000001019/head//3
2014-06-12 14:51:33.994701 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 94cf803a/rbd_data.10926b8b4567.00000000000029f2/head//3, auth == maps.end()
2014-06-12 14:51:33.994713 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 94cf803a/rbd_data.10926b8b4567.00000000000029f2/head//3
2014-06-12 14:51:33.994725 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj faef803a/rbd_data.10906b8b4567.00000000000016bb/head//3, auth == maps.end()
2014-06-12 14:51:33.994738 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj faef803a/rbd_data.10906b8b4567.00000000000016bb/head//3
2014-06-12 14:51:33.994750 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8731903a/rbd_data.10a66b8b4567.0000000000003929/head//3, auth == maps.end()
2014-06-12 14:51:33.994760 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8731903a/rbd_data.10a66b8b4567.0000000000003929/head//3
2014-06-12 14:51:33.994772 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c947903a/rbd_data.10a26b8b4567.0000000000002381/head//3, auth == maps.end()
2014-06-12 14:51:33.994782 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c947903a/rbd_data.10a26b8b4567.0000000000002381/head//3
2014-06-12 14:51:33.994793 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj eba0a03a/rbd_data.108e6b8b4567.0000000000001462/head//3, auth == maps.end()
2014-06-12 14:51:33.994804 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj eba0a03a/rbd_data.108e6b8b4567.0000000000001462/head//3
2014-06-12 14:51:33.994816 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aba5a03a/rbd_data.10aa6b8b4567.00000000000002e9/head//3, auth == maps.end()
2014-06-12 14:51:33.994826 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj aba5a03a/rbd_data.10aa6b8b4567.00000000000002e9/head//3
2014-06-12 14:51:33.994837 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3097a03a/rbd_data.10a86b8b4567.000000000000011b/head//3, auth == maps.end()
2014-06-12 14:51:33.994848 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3097a03a/rbd_data.10a86b8b4567.000000000000011b/head//3
2014-06-12 14:51:33.994862 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bd18a03a/rbd_data.10ab6b8b4567.00000000000039e8/head//3, auth == maps.end()
2014-06-12 14:51:33.994879 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj bd18a03a/rbd_data.10ab6b8b4567.00000000000039e8/head//3
2014-06-12 14:51:33.994891 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bc79a03a/rbd_data.109e6b8b4567.00000000000031e8/head//3, auth == maps.end()
2014-06-12 14:51:33.994902 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj bc79a03a/rbd_data.109e6b8b4567.00000000000031e8/head//3
2014-06-12 14:51:33.994913 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3a5aa03a/rbd_data.10966b8b4567.0000000000001b26/head//3, auth == maps.end()
2014-06-12 14:51:33.994924 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3a5aa03a/rbd_data.10966b8b4567.0000000000001b26/head//3
2014-06-12 14:51:33.994935 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 559aa03a/rbd_data.10b26b8b4567.000000000000258e/head//3, auth == maps.end()
2014-06-12 14:51:33.994945 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 559aa03a/rbd_data.10b26b8b4567.000000000000258e/head//3
2014-06-12 14:51:33.994957 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8deea03a/rbd_data.10af6b8b4567.0000000000002997/head//3, auth == maps.end()
2014-06-12 14:51:33.994967 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8deea03a/rbd_data.10af6b8b4567.0000000000002997/head//3
2014-06-12 14:51:33.994979 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj efc3b03a/rbd_data.10a46b8b4567.000000000000366c/head//3, auth == maps.end()
2014-06-12 14:51:33.994991 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj efc3b03a/rbd_data.10a46b8b4567.000000000000366c/head//3
2014-06-12 14:51:33.995005 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj df15b03a/rbd_data.10986b8b4567.00000000000007c8/head//3, auth == maps.end()
2014-06-12 14:51:33.995018 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj df15b03a/rbd_data.10986b8b4567.00000000000007c8/head//3
2014-06-12 14:51:33.995030 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 45a6b03a/rbd_data.10a36b8b4567.0000000000003208/head//3, auth == maps.end()
2014-06-12 14:51:33.995041 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 45a6b03a/rbd_data.10a36b8b4567.0000000000003208/head//3
2014-06-12 14:51:33.995053 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4d99b03a/rbd_data.10a46b8b4567.0000000000000406/head//3, auth == maps.end()
2014-06-12 14:51:33.995066 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4d99b03a/rbd_data.10a46b8b4567.0000000000000406/head//3
2014-06-12 14:51:33.995079 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] 
2014-06-12 14:51:33.995088 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub
2014-06-12 14:51:33.995104 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  4d99b03a/rbd_data.10a46b8b4567.0000000000000406/head//3 4d99b03a/rbd_data.10a46b8b4567.0000000000000406/head//3(173'2614 client.4301.0:124567 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2614)
2014-06-12 14:51:33.995122 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  45a6b03a/rbd_data.10a36b8b4567.0000000000003208/head//3 45a6b03a/rbd_data.10a36b8b4567.0000000000003208/head//3(173'2010 client.4300.0:97801 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2010)
2014-06-12 14:51:33.995139 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  df15b03a/rbd_data.10986b8b4567.00000000000007c8/head//3 df15b03a/rbd_data.10986b8b4567.00000000000007c8/head//3(173'2733 client.4289.0:127629 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2733)
2014-06-12 14:51:33.995155 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  efc3b03a/rbd_data.10a46b8b4567.000000000000366c/head//3 efc3b03a/rbd_data.10a46b8b4567.000000000000366c/head//3(173'2238 client.4301.0:109059 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2238)
2014-06-12 14:51:33.995171 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  8deea03a/rbd_data.10af6b8b4567.0000000000002997/head//3 8deea03a/rbd_data.10af6b8b4567.0000000000002997/head//3(173'1819 client.4312.0:86669 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1819)
2014-06-12 14:51:33.995187 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  559aa03a/rbd_data.10b26b8b4567.000000000000258e/head//3 559aa03a/rbd_data.10b26b8b4567.000000000000258e/head//3(173'1524 client.4315.0:77251 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1524)
2014-06-12 14:51:33.995203 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  3a5aa03a/rbd_data.10966b8b4567.0000000000001b26/head//3 3a5aa03a/rbd_data.10966b8b4567.0000000000001b26/head//3(173'1188 client.4287.0:56743 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1188)
2014-06-12 14:51:33.995219 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  bc79a03a/rbd_data.109e6b8b4567.00000000000031e8/head//3 bc79a03a/rbd_data.109e6b8b4567.00000000000031e8/head//3(173'2036 client.4295.0:99883 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2036)
2014-06-12 14:51:33.995235 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  bd18a03a/rbd_data.10ab6b8b4567.00000000000039e8/head//3 bd18a03a/rbd_data.10ab6b8b4567.00000000000039e8/head//3(173'2396 client.4308.0:115421 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2396)
2014-06-12 14:51:33.995251 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  3097a03a/rbd_data.10a86b8b4567.000000000000011b/head//3 3097a03a/rbd_data.10a86b8b4567.000000000000011b/head//3(173'2491 client.4305.0:118649 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2491)
2014-06-12 14:51:33.995267 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  aba5a03a/rbd_data.10aa6b8b4567.00000000000002e9/head//3 aba5a03a/rbd_data.10aa6b8b4567.00000000000002e9/head//3(173'2539 client.4307.0:123904 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2539)
2014-06-12 14:51:33.995283 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  eba0a03a/rbd_data.108e6b8b4567.0000000000001462/head//3 eba0a03a/rbd_data.108e6b8b4567.0000000000001462/head//3(173'999 client.4279.0:46795 wrlock_by=unknown.0.0:0 dirty s 4194304 uv999)
2014-06-12 14:51:33.995299 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c947903a/rbd_data.10a26b8b4567.0000000000002381/head//3 c947903a/rbd_data.10a26b8b4567.0000000000002381/head//3(173'1739 client.4299.0:75160 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1739)
2014-06-12 14:51:33.995315 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  8731903a/rbd_data.10a66b8b4567.0000000000003929/head//3 8731903a/rbd_data.10a66b8b4567.0000000000003929/head//3(173'2405 client.4303.0:113262 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2405)
2014-06-12 14:51:33.995331 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  faef803a/rbd_data.10906b8b4567.00000000000016bb/head//3 faef803a/rbd_data.10906b8b4567.00000000000016bb/head//3(173'1060 client.4281.0:50078 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1060)
2014-06-12 14:51:33.995351 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  94cf803a/rbd_data.10926b8b4567.00000000000029f2/head//3 94cf803a/rbd_data.10926b8b4567.00000000000029f2/head//3(173'1837 client.4283.0:84433 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1837)
2014-06-12 14:51:33.995367 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  963d803a/rbd_data.10a36b8b4567.0000000000001019/head//3 963d803a/rbd_data.10a36b8b4567.0000000000001019/head//3(173'907 client.4300.0:37270 wrlock_by=unknown.0.0:0 dirty s 4194304 uv907)
2014-06-12 14:51:33.995382 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  569b803a/rbd_data.10a66b8b4567.0000000000002682/head//3 569b803a/rbd_data.10a66b8b4567.0000000000002682/head//3(173'1729 client.4303.0:77246 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1729)
2014-06-12 14:51:33.995398 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a4ca803a/rbd_data.10b06b8b4567.0000000000001477/head//3 a4ca803a/rbd_data.10b06b8b4567.0000000000001477/head//3(173'1003 client.4313.0:47135 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1003)
2014-06-12 14:51:33.995414 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  39f8803a/rbd_data.109b6b8b4567.0000000000001e5e/head//3 39f8803a/rbd_data.109b6b8b4567.0000000000001e5e/head//3(173'1424 client.4292.0:62243 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1424)
2014-06-12 14:51:33.995429 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  cc68803a/rbd_data.10b26b8b4567.000000000000313d/head//3 cc68803a/rbd_data.10b26b8b4567.000000000000313d/head//3(173'1996 client.4315.0:98544 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1996)
2014-06-12 14:51:33.995445 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  dd38803a/rbd_data.10986b8b4567.000000000000160f/head//3 dd38803a/rbd_data.10986b8b4567.000000000000160f/head//3(173'1012 client.4289.0:48287 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1012)
2014-06-12 14:51:33.995461 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  6036803a/rbd_data.10946b8b4567.0000000000002208/head//3 6036803a/rbd_data.10946b8b4567.0000000000002208/head//3(173'1444 client.4285.0:69464 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1444)
2014-06-12 14:51:33.995476 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  9a06803a/rbd_data.10a26b8b4567.000000000000395b/head//3 9a06803a/rbd_data.10a26b8b4567.000000000000395b/head//3(173'2411 client.4299.0:115559 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2411)
2014-06-12 14:51:33.995492 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a825803a/rbd_data.10956b8b4567.0000000000000185/head//3 a825803a/rbd_data.10956b8b4567.0000000000000185/head//3(173'2496 client.4286.0:120431 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2496)
2014-06-12 14:51:33.995503 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:33.995515 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:33.995525 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:33.995528 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:33.995530 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:33.995570 7f9d60bd2700 10 collection_list_partial 3.3a_head start:dc9ab03a//0//-1 is_max:0
2014-06-12 14:51:33.995829 7f9d60bd2700 20 objects: [13ccb03a/rbd_data.10936b8b4567.0000000000000120/head//3,ecccb03a/rbd_data.10966b8b4567.0000000000000766/head//3,b70fb03a/rbd_data.10a16b8b4567.0000000000003967/head//3,342fb03a/rbd_data.10a56b8b4567.000000000000138d/head//3,5d10c03a/rbd_data.10946b8b4567.000000000000074c/head//3,dc50c03a/rbd_data.109f6b8b4567.000000000000362e/head//3,2a90c03a/rbd_data.10906b8b4567.0000000000001ffb/head//3,5e31c03a/rbd_data.10996b8b4567.0000000000003270/head//3,1e72c03a/rbd_data.10a96b8b4567.0000000000000187/head//3,9143c03a/rbd_data.108d6b8b4567.000000000000203c/head//3,a643c03a/rbd_data.10946b8b4567.000000000000004d/head//3,e614c03a/rbd_data.108d6b8b4567.00000000000023e0/head//3,7544c03a/rbd_data.109b6b8b4567.000000000000175f/head//3,1da5c03a/rbd_data.10906b8b4567.0000000000003525/head//3,d729c03a/rbd_data.10a16b8b4567.000000000000188f/head//3,4b29c03a/rbd_data.10936b8b4567.0000000000001bf0/head//3,6adbc03a/rbd_data.10916b8b4567.0000000000003453/head//3,f30cc03a/rbd_data.10a36b8b4567.0000000000003112/head//3,40e0d03a/rbd_data.108f6b8b4567.0000000000002ab7/head//3,bac1d03a/rbd_data.10a26b8b4567.0000000000003a9e/head//3,2145d03a/rbd_data.10956b8b4567.0000000000000f20/head//3,5b16d03a/rbd_data.10aa6b8b4567.0000000000003380/head//3,eb56d03a/rbd_data.10a06b8b4567.0000000000001721/head//3,bd09d03a/rbd_data.10ad6b8b4567.0000000000000147/head//3,237ad03a/rbd_data.109c6b8b4567.0000000000000bc2/head//3]
2014-06-12 14:51:33.996382 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:33.996409 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:33.996427 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:33.996436 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:33.996443 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:33.996450 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:33.996457 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_map_chunk [dc9ab03a//0//-1,7fbad03a//0//-1)
2014-06-12 14:51:33.996465 7f9d60bd2700 10 collection_list_partial 3.3a_head start:dc9ab03a//0//-1 is_max:0
2014-06-12 14:51:33.997084 7f9d60bd2700 20 objects: [13ccb03a/rbd_data.10936b8b4567.0000000000000120/head//3,ecccb03a/rbd_data.10966b8b4567.0000000000000766/head//3,b70fb03a/rbd_data.10a16b8b4567.0000000000003967/head//3,342fb03a/rbd_data.10a56b8b4567.000000000000138d/head//3,5d10c03a/rbd_data.10946b8b4567.000000000000074c/head//3,dc50c03a/rbd_data.109f6b8b4567.000000000000362e/head//3,2a90c03a/rbd_data.10906b8b4567.0000000000001ffb/head//3,5e31c03a/rbd_data.10996b8b4567.0000000000003270/head//3,1e72c03a/rbd_data.10a96b8b4567.0000000000000187/head//3,9143c03a/rbd_data.108d6b8b4567.000000000000203c/head//3,a643c03a/rbd_data.10946b8b4567.000000000000004d/head//3,e614c03a/rbd_data.108d6b8b4567.00000000000023e0/head//3,7544c03a/rbd_data.109b6b8b4567.000000000000175f/head//3,1da5c03a/rbd_data.10906b8b4567.0000000000003525/head//3,d729c03a/rbd_data.10a16b8b4567.000000000000188f/head//3,4b29c03a/rbd_data.10936b8b4567.0000000000001bf0/head//3,6adbc03a/rbd_data.10916b8b4567.0000000000003453/head//3,f30cc03a/rbd_data.10a36b8b4567.0000000000003112/head//3,40e0d03a/rbd_data.108f6b8b4567.0000000000002ab7/head//3,bac1d03a/rbd_data.10a26b8b4567.0000000000003a9e/head//3,2145d03a/rbd_data.10956b8b4567.0000000000000f20/head//3,5b16d03a/rbd_data.10aa6b8b4567.0000000000003380/head//3,eb56d03a/rbd_data.10a06b8b4567.0000000000001721/head//3,bd09d03a/rbd_data.10ad6b8b4567.0000000000000147/head//3,237ad03a/rbd_data.109c6b8b4567.0000000000000bc2/head//3,7fbad03a/rbd_data.10af6b8b4567.0000000000002247/head//3,159bd03a/rbd_data.10a46b8b4567.0000000000002215/head//3,c79cd03a/rbd_data.10ac6b8b4567.0000000000000443/head//3,d3ecd03a/rbd_data.10976b8b4567.00000000000032c2/head//3,e63dd03a/rbd_data.10b36b8b4567.0000000000003bae/head//3,d36dd03a/rbd_data.10a86b8b4567.0000000000003a0f/head//3,4e8dd03a/rbd_data.10a06b8b4567.00000000000025c9/head//3,c6bfd03a/rbd_data.10a96b8b4567.0000000000001b39/head//3,a6efd03a/rbd_data.10976b8b4567.00000000000009a1/head//3,84f0e03a/rbd_data.10936b8b4567.00000000000023b4/head//3,a035e03a/rbd_data.10956b8b4567.0000000000001e90/head//3,ad09e03a/rbd_data.10906b8b4567.0000000000000c16/head//3,47cce03a/rbd_data.10a76b8b4567.0000000000000889/head//3,a65de03a/rbd_data.10a56b8b4567.0000000000000162/head//3,137de03a/rbd_data.10a76b8b4567.0000000000002796/head//3,b8bde03a/rbd_data.108d6b8b4567.0000000000000e47/head//3,235fe03a/rbd_data.10aa6b8b4567.00000000000002ba/head//3,9971f03a/rbd_data.109f6b8b4567.000000000000298e/head//3,a6f3f03a/rbd_data.109e6b8b4567.000000000000335c/head//3,e865f03a/rbd_data.10b46b8b4567.0000000000000f5a/head//3,acb5f03a/rbd_data.10ab6b8b4567.00000000000035ba/head//3,8b16f03a/rbd_data.10ab6b8b4567.00000000000005b0/head//3,a4b6f03a/rbd_data.10a96b8b4567.000000000000218b/head//3,5667f03a/rbd_data.10a46b8b4567.00000000000001de/head//3,e61ef03a/rbd_data.10966b8b4567.00000000000035a8/head//3,b3cef03a/rbd_data.10a86b8b4567.0000000000003318/head//3,3daff03a/rbd_data.108e6b8b4567.0000000000000b65/head//3,b14083a/rbd_data.108d6b8b4567.0000000000003892/head//3,eb54083a/rbd_data.109f6b8b4567.00000000000000cf/head//3,8ba6083a/rbd_data.10b26b8b4567.0000000000002807/head//3,12f6083a/rbd_data.108d6b8b4567.0000000000002844/head//3,4997083a/rbd_data.109b6b8b4567.0000000000001d04/head//3,8d7b083a/rbd_data.10936b8b4567.0000000000000835/head//3,e1be083a/rbd_data.10976b8b4567.0000000000001285/head//3,ef00183a/rbd_data.10b26b8b4567.0000000000001d9c/head//3,76e2183a/rbd_data.10b36b8b4567.0000000000001bcf/head//3,b803183a/rbd_data.10986b8b4567.0000000000001df7/head//3,cf86183a/rbd_data.10a26b8b4567.000000000000334a/head//3,ab96183a/rbd_data.10a56b8b4567.0000000000002daf/head//3]
2014-06-12 14:51:33.997182 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:33.997201 7f9d60bd2700 10 stat 3.3a_head/13ccb03a/rbd_data.10936b8b4567.0000000000000120/head//3
2014-06-12 14:51:33.997344 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 13ccb03a/rbd_data.10936b8b4567.0000000000000120/head//3
2014-06-12 14:51:34.040094 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:34.040105 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:34.040108 7f9d74bfa700 20 osd.2 173 _dispatch 0x5ba9c00 pg_stats_ack(2 pgs tid 4870) v1
2014-06-12 14:51:34.040113 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:51:34.040123 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:34.040127 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:34.056101 7f9d60bd2700 10 getattrs 3.3a_head/13ccb03a/rbd_data.10936b8b4567.0000000000000120/head//3 = 0
2014-06-12 14:51:34.056123 7f9d60bd2700 10 stat 3.3a_head/ecccb03a/rbd_data.10966b8b4567.0000000000000766/head//3
2014-06-12 14:51:34.056326 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ecccb03a/rbd_data.10966b8b4567.0000000000000766/head//3
2014-06-12 14:51:34.064494 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:34.075726 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:34.097637 7f9d60bd2700 10 getattrs 3.3a_head/ecccb03a/rbd_data.10966b8b4567.0000000000000766/head//3 = 0
2014-06-12 14:51:34.097651 7f9d60bd2700 10 stat 3.3a_head/b70fb03a/rbd_data.10a16b8b4567.0000000000003967/head//3
2014-06-12 14:51:34.097791 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid b70fb03a/rbd_data.10a16b8b4567.0000000000003967/head//3
2014-06-12 14:51:34.117600 7f9d60bd2700 10 getattrs 3.3a_head/b70fb03a/rbd_data.10a16b8b4567.0000000000003967/head//3 = 0
2014-06-12 14:51:34.117607 7f9d60bd2700 10 stat 3.3a_head/342fb03a/rbd_data.10a56b8b4567.000000000000138d/head//3
2014-06-12 14:51:34.117692 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 342fb03a/rbd_data.10a56b8b4567.000000000000138d/head//3
2014-06-12 14:51:34.148706 7f9d60bd2700 10 getattrs 3.3a_head/342fb03a/rbd_data.10a56b8b4567.000000000000138d/head//3 = 0
2014-06-12 14:51:34.148720 7f9d60bd2700 10 stat 3.3a_head/5d10c03a/rbd_data.10946b8b4567.000000000000074c/head//3
2014-06-12 14:51:34.148828 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5d10c03a/rbd_data.10946b8b4567.000000000000074c/head//3
2014-06-12 14:51:34.167299 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:34.170431 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:34.170496 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:34.195466 7f9d60bd2700 10 getattrs 3.3a_head/5d10c03a/rbd_data.10946b8b4567.000000000000074c/head//3 = 0
2014-06-12 14:51:34.195481 7f9d60bd2700 10 stat 3.3a_head/dc50c03a/rbd_data.109f6b8b4567.000000000000362e/head//3
2014-06-12 14:51:34.195650 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid dc50c03a/rbd_data.109f6b8b4567.000000000000362e/head//3
2014-06-12 14:51:34.228701 7f9d60bd2700 10 getattrs 3.3a_head/dc50c03a/rbd_data.109f6b8b4567.000000000000362e/head//3 = 0
2014-06-12 14:51:34.228708 7f9d60bd2700 10 stat 3.3a_head/2a90c03a/rbd_data.10906b8b4567.0000000000001ffb/head//3
2014-06-12 14:51:34.228811 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 2a90c03a/rbd_data.10906b8b4567.0000000000001ffb/head//3
2014-06-12 14:51:34.249580 7f9d60bd2700 10 getattrs 3.3a_head/2a90c03a/rbd_data.10906b8b4567.0000000000001ffb/head//3 = 0
2014-06-12 14:51:34.249589 7f9d60bd2700 10 stat 3.3a_head/5e31c03a/rbd_data.10996b8b4567.0000000000003270/head//3
2014-06-12 14:51:34.249702 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5e31c03a/rbd_data.10996b8b4567.0000000000003270/head//3
2014-06-12 14:51:34.263755 7f9d60bd2700 10 getattrs 3.3a_head/5e31c03a/rbd_data.10996b8b4567.0000000000003270/head//3 = 0
2014-06-12 14:51:34.263763 7f9d60bd2700 10 stat 3.3a_head/1e72c03a/rbd_data.10a96b8b4567.0000000000000187/head//3
2014-06-12 14:51:34.263861 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 1e72c03a/rbd_data.10a96b8b4567.0000000000000187/head//3
2014-06-12 14:51:34.292934 7f9d60bd2700 10 getattrs 3.3a_head/1e72c03a/rbd_data.10a96b8b4567.0000000000000187/head//3 = 0
2014-06-12 14:51:34.292945 7f9d60bd2700 10 stat 3.3a_head/9143c03a/rbd_data.108d6b8b4567.000000000000203c/head//3
2014-06-12 14:51:34.293043 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 9143c03a/rbd_data.108d6b8b4567.000000000000203c/head//3
2014-06-12 14:51:34.312428 7f9d60bd2700 10 getattrs 3.3a_head/9143c03a/rbd_data.108d6b8b4567.000000000000203c/head//3 = 0
2014-06-12 14:51:34.312438 7f9d60bd2700 10 stat 3.3a_head/a643c03a/rbd_data.10946b8b4567.000000000000004d/head//3
2014-06-12 14:51:34.312539 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a643c03a/rbd_data.10946b8b4567.000000000000004d/head//3
2014-06-12 14:51:34.316724 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:34.326184 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:34.362823 7f9d60bd2700 10 getattrs 3.3a_head/a643c03a/rbd_data.10946b8b4567.000000000000004d/head//3 = 0
2014-06-12 14:51:34.362843 7f9d60bd2700 10 stat 3.3a_head/e614c03a/rbd_data.108d6b8b4567.00000000000023e0/head//3
2014-06-12 14:51:34.362969 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid e614c03a/rbd_data.108d6b8b4567.00000000000023e0/head//3
2014-06-12 14:51:34.366111 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:34.370962 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:34.381763 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:34.397831 7f9d60bd2700 10 getattrs 3.3a_head/e614c03a/rbd_data.108d6b8b4567.00000000000023e0/head//3 = 0
2014-06-12 14:51:34.397849 7f9d60bd2700 10 stat 3.3a_head/7544c03a/rbd_data.109b6b8b4567.000000000000175f/head//3
2014-06-12 14:51:34.398037 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 7544c03a/rbd_data.109b6b8b4567.000000000000175f/head//3
2014-06-12 14:51:34.415268 7f9d60bd2700 10 getattrs 3.3a_head/7544c03a/rbd_data.109b6b8b4567.000000000000175f/head//3 = 0
2014-06-12 14:51:34.415287 7f9d60bd2700 10 stat 3.3a_head/1da5c03a/rbd_data.10906b8b4567.0000000000003525/head//3
2014-06-12 14:51:34.415480 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 1da5c03a/rbd_data.10906b8b4567.0000000000003525/head//3
2014-06-12 14:51:34.433913 7f9d60bd2700 10 getattrs 3.3a_head/1da5c03a/rbd_data.10906b8b4567.0000000000003525/head//3 = 0
2014-06-12 14:51:34.433920 7f9d60bd2700 10 stat 3.3a_head/d729c03a/rbd_data.10a16b8b4567.000000000000188f/head//3
2014-06-12 14:51:34.434018 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid d729c03a/rbd_data.10a16b8b4567.000000000000188f/head//3
2014-06-12 14:51:34.448897 7f9d60bd2700 10 getattrs 3.3a_head/d729c03a/rbd_data.10a16b8b4567.000000000000188f/head//3 = 0
2014-06-12 14:51:34.448904 7f9d60bd2700 10 stat 3.3a_head/4b29c03a/rbd_data.10936b8b4567.0000000000001bf0/head//3
2014-06-12 14:51:34.449001 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 4b29c03a/rbd_data.10936b8b4567.0000000000001bf0/head//3
2014-06-12 14:51:34.467614 7f9d60bd2700 10 getattrs 3.3a_head/4b29c03a/rbd_data.10936b8b4567.0000000000001bf0/head//3 = 0
2014-06-12 14:51:34.467624 7f9d60bd2700 10 stat 3.3a_head/6adbc03a/rbd_data.10916b8b4567.0000000000003453/head//3
2014-06-12 14:51:34.467709 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 6adbc03a/rbd_data.10916b8b4567.0000000000003453/head//3
2014-06-12 14:51:34.485289 7f9d60bd2700 10 getattrs 3.3a_head/6adbc03a/rbd_data.10916b8b4567.0000000000003453/head//3 = 0
2014-06-12 14:51:34.485297 7f9d60bd2700 10 stat 3.3a_head/f30cc03a/rbd_data.10a36b8b4567.0000000000003112/head//3
2014-06-12 14:51:34.485389 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid f30cc03a/rbd_data.10a36b8b4567.0000000000003112/head//3
2014-06-12 14:51:34.512970 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:34.514249 7f9d60bd2700 10 getattrs 3.3a_head/f30cc03a/rbd_data.10a36b8b4567.0000000000003112/head//3 = 0
2014-06-12 14:51:34.514266 7f9d60bd2700 10 stat 3.3a_head/40e0d03a/rbd_data.108f6b8b4567.0000000000002ab7/head//3
2014-06-12 14:51:34.514387 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 40e0d03a/rbd_data.108f6b8b4567.0000000000002ab7/head//3
2014-06-12 14:51:34.518808 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:34.522716 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:34.528173 7f9d60bd2700 10 getattrs 3.3a_head/40e0d03a/rbd_data.108f6b8b4567.0000000000002ab7/head//3 = 0
2014-06-12 14:51:34.528182 7f9d60bd2700 10 stat 3.3a_head/bac1d03a/rbd_data.10a26b8b4567.0000000000003a9e/head//3
2014-06-12 14:51:34.528285 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid bac1d03a/rbd_data.10a26b8b4567.0000000000003a9e/head//3
2014-06-12 14:51:34.528381 7f9d60bd2700 10 getattrs 3.3a_head/bac1d03a/rbd_data.10a26b8b4567.0000000000003a9e/head//3 = 0
2014-06-12 14:51:34.528387 7f9d60bd2700 10 stat 3.3a_head/2145d03a/rbd_data.10956b8b4567.0000000000000f20/head//3
2014-06-12 14:51:34.528475 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 2145d03a/rbd_data.10956b8b4567.0000000000000f20/head//3
2014-06-12 14:51:34.532177 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:34.547603 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:34.572026 7f9d60bd2700 10 getattrs 3.3a_head/2145d03a/rbd_data.10956b8b4567.0000000000000f20/head//3 = 0
2014-06-12 14:51:34.572047 7f9d60bd2700 10 stat 3.3a_head/5b16d03a/rbd_data.10aa6b8b4567.0000000000003380/head//3
2014-06-12 14:51:34.572201 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5b16d03a/rbd_data.10aa6b8b4567.0000000000003380/head//3
2014-06-12 14:51:34.597913 7f9d60bd2700 10 getattrs 3.3a_head/5b16d03a/rbd_data.10aa6b8b4567.0000000000003380/head//3 = 0
2014-06-12 14:51:34.597922 7f9d60bd2700 10 stat 3.3a_head/eb56d03a/rbd_data.10a06b8b4567.0000000000001721/head//3
2014-06-12 14:51:34.598028 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid eb56d03a/rbd_data.10a06b8b4567.0000000000001721/head//3
2014-06-12 14:51:34.612583 7f9d60bd2700 10 getattrs 3.3a_head/eb56d03a/rbd_data.10a06b8b4567.0000000000001721/head//3 = 0
2014-06-12 14:51:34.612593 7f9d60bd2700 10 stat 3.3a_head/bd09d03a/rbd_data.10ad6b8b4567.0000000000000147/head//3
2014-06-12 14:51:34.612701 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid bd09d03a/rbd_data.10ad6b8b4567.0000000000000147/head//3
2014-06-12 14:51:34.661536 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:34.662991 7f9d60bd2700 10 getattrs 3.3a_head/bd09d03a/rbd_data.10ad6b8b4567.0000000000000147/head//3 = 0
2014-06-12 14:51:34.662999 7f9d60bd2700 10 stat 3.3a_head/237ad03a/rbd_data.109c6b8b4567.0000000000000bc2/head//3
2014-06-12 14:51:34.663080 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 237ad03a/rbd_data.109c6b8b4567.0000000000000bc2/head//3
2014-06-12 14:51:34.665070 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:34.667785 7f9d44101700  5 -- op tracker -- , seq: 748586, time: 2014-06-12 14:51:34.667708, event: header_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:34.667793 7f9d44101700  5 -- op tracker -- , seq: 748586, time: 2014-06-12 14:51:34.667710, event: throttled, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:34.667806 7f9d44101700  5 -- op tracker -- , seq: 748586, time: 2014-06-12 14:51:34.667760, event: all_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:34.667812 7f9d44101700  5 -- op tracker -- , seq: 748586, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:34.667816 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:34.667828 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:34.667834 7f9d44101700 15 osd.2 173 enqueue_op 0x2afd700 prio 127 cost 10717 latency 0.000126 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:34.667896 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:34.700437 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:34.701084 7f9d60bd2700 10 getattrs 3.3a_head/237ad03a/rbd_data.109c6b8b4567.0000000000000bc2/head//3 = 0
2014-06-12 14:51:34.701109 7f9d60bd2700 10 collection_getattrs 3.3a_head
2014-06-12 14:51:34.701196 7f9d60bd2700 10 collection_getattrs 3.3a_head = 0
2014-06-12 14:51:34.701205 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:34.701264 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:34.701279 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:34.701296 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:34.701303 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:34.701308 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2afd700 prio 127 cost 10717 latency 0.033599 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]
2014-06-12 14:51:34.701357 7f9d6b3e7700  5 -- op tracker -- , seq: 748586, time: 2014-06-12 14:51:34.701347, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:34.701385 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] handle_message: 0x2afd700
2014-06-12 14:51:34.701401 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:34.701424 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:34.701441 7f9d6b3e7700  5 -- op tracker -- , seq: 748586, time: 2014-06-12 14:51:34.701436, event: started, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:34.701473 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:34.701575 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] map version is 173'2964
2014-06-12 14:51:34.701597 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x2afd700 finish
2014-06-12 14:51:34.701605 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:34.701626 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:34.701650 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:34.701627 7f9d6b3e7700  5 -- op tracker -- , seq: 748586, time: 2014-06-12 14:51:34.701605, event: done, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:34.701667 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:34.701702 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:34.701707 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:34.701712 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:34.701720 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:34.701728 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:34.701769 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 13ccb03a/rbd_data.10936b8b4567.0000000000000120/head//3, auth == maps.end()
2014-06-12 14:51:34.701796 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 13ccb03a/rbd_data.10936b8b4567.0000000000000120/head//3
2014-06-12 14:51:34.701832 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ecccb03a/rbd_data.10966b8b4567.0000000000000766/head//3, auth == maps.end()
2014-06-12 14:51:34.701854 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ecccb03a/rbd_data.10966b8b4567.0000000000000766/head//3
2014-06-12 14:51:34.701873 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b70fb03a/rbd_data.10a16b8b4567.0000000000003967/head//3, auth == maps.end()
2014-06-12 14:51:34.701890 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b70fb03a/rbd_data.10a16b8b4567.0000000000003967/head//3
2014-06-12 14:51:34.701907 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 342fb03a/rbd_data.10a56b8b4567.000000000000138d/head//3, auth == maps.end()
2014-06-12 14:51:34.701923 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 342fb03a/rbd_data.10a56b8b4567.000000000000138d/head//3
2014-06-12 14:51:34.701948 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5d10c03a/rbd_data.10946b8b4567.000000000000074c/head//3, auth == maps.end()
2014-06-12 14:51:34.701960 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5d10c03a/rbd_data.10946b8b4567.000000000000074c/head//3
2014-06-12 14:51:34.701970 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dc50c03a/rbd_data.109f6b8b4567.000000000000362e/head//3, auth == maps.end()
2014-06-12 14:51:34.701999 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj dc50c03a/rbd_data.109f6b8b4567.000000000000362e/head//3
2014-06-12 14:51:34.702017 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2a90c03a/rbd_data.10906b8b4567.0000000000001ffb/head//3, auth == maps.end()
2014-06-12 14:51:34.702033 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2a90c03a/rbd_data.10906b8b4567.0000000000001ffb/head//3
2014-06-12 14:51:34.702050 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5e31c03a/rbd_data.10996b8b4567.0000000000003270/head//3, auth == maps.end()
2014-06-12 14:51:34.702067 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5e31c03a/rbd_data.10996b8b4567.0000000000003270/head//3
2014-06-12 14:51:34.702084 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1e72c03a/rbd_data.10a96b8b4567.0000000000000187/head//3, auth == maps.end()
2014-06-12 14:51:34.702100 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1e72c03a/rbd_data.10a96b8b4567.0000000000000187/head//3
2014-06-12 14:51:34.702117 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9143c03a/rbd_data.108d6b8b4567.000000000000203c/head//3, auth == maps.end()
2014-06-12 14:51:34.702133 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9143c03a/rbd_data.108d6b8b4567.000000000000203c/head//3
2014-06-12 14:51:34.702153 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a643c03a/rbd_data.10946b8b4567.000000000000004d/head//3, auth == maps.end()
2014-06-12 14:51:34.702172 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a643c03a/rbd_data.10946b8b4567.000000000000004d/head//3
2014-06-12 14:51:34.702190 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e614c03a/rbd_data.108d6b8b4567.00000000000023e0/head//3, auth == maps.end()
2014-06-12 14:51:34.702209 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e614c03a/rbd_data.108d6b8b4567.00000000000023e0/head//3
2014-06-12 14:51:34.702230 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7544c03a/rbd_data.109b6b8b4567.000000000000175f/head//3, auth == maps.end()
2014-06-12 14:51:34.702247 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7544c03a/rbd_data.109b6b8b4567.000000000000175f/head//3
2014-06-12 14:51:34.702266 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1da5c03a/rbd_data.10906b8b4567.0000000000003525/head//3, auth == maps.end()
2014-06-12 14:51:34.702283 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1da5c03a/rbd_data.10906b8b4567.0000000000003525/head//3
2014-06-12 14:51:34.702303 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d729c03a/rbd_data.10a16b8b4567.000000000000188f/head//3, auth == maps.end()
2014-06-12 14:51:34.702320 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d729c03a/rbd_data.10a16b8b4567.000000000000188f/head//3
2014-06-12 14:51:34.702337 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4b29c03a/rbd_data.10936b8b4567.0000000000001bf0/head//3, auth == maps.end()
2014-06-12 14:51:34.702357 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4b29c03a/rbd_data.10936b8b4567.0000000000001bf0/head//3
2014-06-12 14:51:34.702377 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6adbc03a/rbd_data.10916b8b4567.0000000000003453/head//3, auth == maps.end()
2014-06-12 14:51:34.702401 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6adbc03a/rbd_data.10916b8b4567.0000000000003453/head//3
2014-06-12 14:51:34.702419 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f30cc03a/rbd_data.10a36b8b4567.0000000000003112/head//3, auth == maps.end()
2014-06-12 14:51:34.702435 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f30cc03a/rbd_data.10a36b8b4567.0000000000003112/head//3
2014-06-12 14:51:34.702452 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 40e0d03a/rbd_data.108f6b8b4567.0000000000002ab7/head//3, auth == maps.end()
2014-06-12 14:51:34.702468 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 40e0d03a/rbd_data.108f6b8b4567.0000000000002ab7/head//3
2014-06-12 14:51:34.702485 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bac1d03a/rbd_data.10a26b8b4567.0000000000003a9e/head//3, auth == maps.end()
2014-06-12 14:51:34.702501 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj bac1d03a/rbd_data.10a26b8b4567.0000000000003a9e/head//3
2014-06-12 14:51:34.702518 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2145d03a/rbd_data.10956b8b4567.0000000000000f20/head//3, auth == maps.end()
2014-06-12 14:51:34.702535 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2145d03a/rbd_data.10956b8b4567.0000000000000f20/head//3
2014-06-12 14:51:34.702554 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5b16d03a/rbd_data.10aa6b8b4567.0000000000003380/head//3, auth == maps.end()
2014-06-12 14:51:34.702574 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5b16d03a/rbd_data.10aa6b8b4567.0000000000003380/head//3
2014-06-12 14:51:34.702595 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj eb56d03a/rbd_data.10a06b8b4567.0000000000001721/head//3, auth == maps.end()
2014-06-12 14:51:34.702614 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj eb56d03a/rbd_data.10a06b8b4567.0000000000001721/head//3
2014-06-12 14:51:34.702635 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bd09d03a/rbd_data.10ad6b8b4567.0000000000000147/head//3, auth == maps.end()
2014-06-12 14:51:34.702655 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj bd09d03a/rbd_data.10ad6b8b4567.0000000000000147/head//3
2014-06-12 14:51:34.702673 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 237ad03a/rbd_data.109c6b8b4567.0000000000000bc2/head//3, auth == maps.end()
2014-06-12 14:51:34.702692 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 237ad03a/rbd_data.109c6b8b4567.0000000000000bc2/head//3
2014-06-12 14:51:34.702713 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] 
2014-06-12 14:51:34.702728 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub
2014-06-12 14:51:34.702751 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  237ad03a/rbd_data.109c6b8b4567.0000000000000bc2/head//3 237ad03a/rbd_data.109c6b8b4567.0000000000000bc2/head//3(173'2872 client.4293.0:136978 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2872)
2014-06-12 14:51:34.702778 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  bd09d03a/rbd_data.10ad6b8b4567.0000000000000147/head//3 bd09d03a/rbd_data.10ad6b8b4567.0000000000000147/head//3(173'2503 client.4310.0:119036 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2503)
2014-06-12 14:51:34.702803 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  eb56d03a/rbd_data.10a06b8b4567.0000000000001721/head//3 eb56d03a/rbd_data.10a06b8b4567.0000000000001721/head//3(173'1021 client.4297.0:50524 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1021)
2014-06-12 14:51:34.702827 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5b16d03a/rbd_data.10aa6b8b4567.0000000000003380/head//3 5b16d03a/rbd_data.10aa6b8b4567.0000000000003380/head//3(173'2199 client.4307.0:105119 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2199)
2014-06-12 14:51:34.702851 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  2145d03a/rbd_data.10956b8b4567.0000000000000f20/head//3 2145d03a/rbd_data.10956b8b4567.0000000000000f20/head//3(173'2943 client.4286.0:142639 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2943)
2014-06-12 14:51:34.702875 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  bac1d03a/rbd_data.10a26b8b4567.0000000000003a9e/head//3 bac1d03a/rbd_data.10a26b8b4567.0000000000003a9e/head//3(173'2430 client.4299.0:117704 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2430)
2014-06-12 14:51:34.702899 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  40e0d03a/rbd_data.108f6b8b4567.0000000000002ab7/head//3 40e0d03a/rbd_data.108f6b8b4567.0000000000002ab7/head//3(173'1831 client.4280.0:84969 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1831)
2014-06-12 14:51:34.702924 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  f30cc03a/rbd_data.10a36b8b4567.0000000000003112/head//3 f30cc03a/rbd_data.10a36b8b4567.0000000000003112/head//3(173'1989 client.4300.0:95956 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1989)
2014-06-12 14:51:34.702947 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  6adbc03a/rbd_data.10916b8b4567.0000000000003453/head//3 6adbc03a/rbd_data.10916b8b4567.0000000000003453/head//3(173'2214 client.4282.0:105086 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2214)
2014-06-12 14:51:34.702972 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  4b29c03a/rbd_data.10936b8b4567.0000000000001bf0/head//3 4b29c03a/rbd_data.10936b8b4567.0000000000001bf0/head//3(173'1276 client.4284.0:57998 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1276)
2014-06-12 14:51:34.702996 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  d729c03a/rbd_data.10a16b8b4567.000000000000188f/head//3 d729c03a/rbd_data.10a16b8b4567.000000000000188f/head//3(173'1130 client.4298.0:54720 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1130)
2014-06-12 14:51:34.703020 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  1da5c03a/rbd_data.10906b8b4567.0000000000003525/head//3 1da5c03a/rbd_data.10906b8b4567.0000000000003525/head//3(173'2123 client.4281.0:105084 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2123)
2014-06-12 14:51:34.703044 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  7544c03a/rbd_data.109b6b8b4567.000000000000175f/head//3 7544c03a/rbd_data.109b6b8b4567.000000000000175f/head//3(173'1167 client.4292.0:50248 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1167)
2014-06-12 14:51:34.703068 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  e614c03a/rbd_data.108d6b8b4567.00000000000023e0/head//3 e614c03a/rbd_data.108d6b8b4567.00000000000023e0/head//3(173'1501 client.4278.0:74853 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1501)
2014-06-12 14:51:34.703092 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a643c03a/rbd_data.10946b8b4567.000000000000004d/head//3 a643c03a/rbd_data.10946b8b4567.000000000000004d/head//3(173'2441 client.4285.0:116568 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2441)
2014-06-12 14:51:34.703120 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  9143c03a/rbd_data.108d6b8b4567.000000000000203c/head//3 9143c03a/rbd_data.108d6b8b4567.000000000000203c/head//3(173'1332 client.4278.0:68743 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1332)
2014-06-12 14:51:34.703144 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  1e72c03a/rbd_data.10a96b8b4567.0000000000000187/head//3 1e72c03a/rbd_data.10a96b8b4567.0000000000000187/head//3(173'2515 client.4306.0:120285 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2515)
2014-06-12 14:51:34.703168 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5e31c03a/rbd_data.10996b8b4567.0000000000003270/head//3 5e31c03a/rbd_data.10996b8b4567.0000000000003270/head//3(173'2128 client.4290.0:99446 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2128)
2014-06-12 14:51:34.703192 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  2a90c03a/rbd_data.10906b8b4567.0000000000001ffb/head//3 2a90c03a/rbd_data.10906b8b4567.0000000000001ffb/head//3(173'1315 client.4281.0:66588 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1315)
2014-06-12 14:51:34.703216 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  dc50c03a/rbd_data.109f6b8b4567.000000000000362e/head//3 dc50c03a/rbd_data.109f6b8b4567.000000000000362e/head//3(173'2171 client.4296.0:105541 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2171)
2014-06-12 14:51:34.703240 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5d10c03a/rbd_data.10946b8b4567.000000000000074c/head//3 5d10c03a/rbd_data.10946b8b4567.000000000000074c/head//3(173'2690 client.4285.0:127412 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2690)
2014-06-12 14:51:34.703264 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  342fb03a/rbd_data.10a56b8b4567.000000000000138d/head//3 342fb03a/rbd_data.10a56b8b4567.000000000000138d/head//3(173'962 client.4302.0:45271 wrlock_by=unknown.0.0:0 dirty s 4194304 uv962)
2014-06-12 14:51:34.703288 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  b70fb03a/rbd_data.10a16b8b4567.0000000000003967/head//3 b70fb03a/rbd_data.10a16b8b4567.0000000000003967/head//3(173'2364 client.4298.0:114651 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2364)
2014-06-12 14:51:34.703312 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ecccb03a/rbd_data.10966b8b4567.0000000000000766/head//3 ecccb03a/rbd_data.10966b8b4567.0000000000000766/head//3(173'2729 client.4287.0:126238 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2729)
2014-06-12 14:51:34.703336 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  13ccb03a/rbd_data.10936b8b4567.0000000000000120/head//3 13ccb03a/rbd_data.10936b8b4567.0000000000000120/head//3(173'2471 client.4284.0:116701 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2471)
2014-06-12 14:51:34.703355 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:34.703374 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:34.703390 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:34.703394 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:34.703397 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:34.703452 7f9d60bd2700 10 collection_list_partial 3.3a_head start:7fbad03a//0//-1 is_max:0
2014-06-12 14:51:34.703827 7f9d60bd2700 20 objects: [159bd03a/rbd_data.10a46b8b4567.0000000000002215/head//3,c79cd03a/rbd_data.10ac6b8b4567.0000000000000443/head//3,d3ecd03a/rbd_data.10976b8b4567.00000000000032c2/head//3,e63dd03a/rbd_data.10b36b8b4567.0000000000003bae/head//3,d36dd03a/rbd_data.10a86b8b4567.0000000000003a0f/head//3,4e8dd03a/rbd_data.10a06b8b4567.00000000000025c9/head//3,c6bfd03a/rbd_data.10a96b8b4567.0000000000001b39/head//3,a6efd03a/rbd_data.10976b8b4567.00000000000009a1/head//3,84f0e03a/rbd_data.10936b8b4567.00000000000023b4/head//3,a035e03a/rbd_data.10956b8b4567.0000000000001e90/head//3,ad09e03a/rbd_data.10906b8b4567.0000000000000c16/head//3,47cce03a/rbd_data.10a76b8b4567.0000000000000889/head//3,a65de03a/rbd_data.10a56b8b4567.0000000000000162/head//3,137de03a/rbd_data.10a76b8b4567.0000000000002796/head//3,b8bde03a/rbd_data.108d6b8b4567.0000000000000e47/head//3,235fe03a/rbd_data.10aa6b8b4567.00000000000002ba/head//3,9971f03a/rbd_data.109f6b8b4567.000000000000298e/head//3,a6f3f03a/rbd_data.109e6b8b4567.000000000000335c/head//3,e865f03a/rbd_data.10b46b8b4567.0000000000000f5a/head//3,acb5f03a/rbd_data.10ab6b8b4567.00000000000035ba/head//3,8b16f03a/rbd_data.10ab6b8b4567.00000000000005b0/head//3,a4b6f03a/rbd_data.10a96b8b4567.000000000000218b/head//3,5667f03a/rbd_data.10a46b8b4567.00000000000001de/head//3,e61ef03a/rbd_data.10966b8b4567.00000000000035a8/head//3,b3cef03a/rbd_data.10a86b8b4567.0000000000003318/head//3]
2014-06-12 14:51:34.704471 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:34.704517 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:34.704545 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:34.704571 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:34.704594 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:34.704615 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:34.704627 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_map_chunk [7fbad03a//0//-1,3daff03a//0//-1)
2014-06-12 14:51:34.704641 7f9d60bd2700 10 collection_list_partial 3.3a_head start:7fbad03a//0//-1 is_max:0
2014-06-12 14:51:34.705511 7f9d60bd2700 20 objects: [159bd03a/rbd_data.10a46b8b4567.0000000000002215/head//3,c79cd03a/rbd_data.10ac6b8b4567.0000000000000443/head//3,d3ecd03a/rbd_data.10976b8b4567.00000000000032c2/head//3,e63dd03a/rbd_data.10b36b8b4567.0000000000003bae/head//3,d36dd03a/rbd_data.10a86b8b4567.0000000000003a0f/head//3,4e8dd03a/rbd_data.10a06b8b4567.00000000000025c9/head//3,c6bfd03a/rbd_data.10a96b8b4567.0000000000001b39/head//3,a6efd03a/rbd_data.10976b8b4567.00000000000009a1/head//3,84f0e03a/rbd_data.10936b8b4567.00000000000023b4/head//3,a035e03a/rbd_data.10956b8b4567.0000000000001e90/head//3,ad09e03a/rbd_data.10906b8b4567.0000000000000c16/head//3,47cce03a/rbd_data.10a76b8b4567.0000000000000889/head//3,a65de03a/rbd_data.10a56b8b4567.0000000000000162/head//3,137de03a/rbd_data.10a76b8b4567.0000000000002796/head//3,b8bde03a/rbd_data.108d6b8b4567.0000000000000e47/head//3,235fe03a/rbd_data.10aa6b8b4567.00000000000002ba/head//3,9971f03a/rbd_data.109f6b8b4567.000000000000298e/head//3,a6f3f03a/rbd_data.109e6b8b4567.000000000000335c/head//3,e865f03a/rbd_data.10b46b8b4567.0000000000000f5a/head//3,acb5f03a/rbd_data.10ab6b8b4567.00000000000035ba/head//3,8b16f03a/rbd_data.10ab6b8b4567.00000000000005b0/head//3,a4b6f03a/rbd_data.10a96b8b4567.000000000000218b/head//3,5667f03a/rbd_data.10a46b8b4567.00000000000001de/head//3,e61ef03a/rbd_data.10966b8b4567.00000000000035a8/head//3,b3cef03a/rbd_data.10a86b8b4567.0000000000003318/head//3,3daff03a/rbd_data.108e6b8b4567.0000000000000b65/head//3,b14083a/rbd_data.108d6b8b4567.0000000000003892/head//3,eb54083a/rbd_data.109f6b8b4567.00000000000000cf/head//3,8ba6083a/rbd_data.10b26b8b4567.0000000000002807/head//3,12f6083a/rbd_data.108d6b8b4567.0000000000002844/head//3,4997083a/rbd_data.109b6b8b4567.0000000000001d04/head//3,8d7b083a/rbd_data.10936b8b4567.0000000000000835/head//3,e1be083a/rbd_data.10976b8b4567.0000000000001285/head//3,ef00183a/rbd_data.10b26b8b4567.0000000000001d9c/head//3,76e2183a/rbd_data.10b36b8b4567.0000000000001bcf/head//3,b803183a/rbd_data.10986b8b4567.0000000000001df7/head//3,cf86183a/rbd_data.10a26b8b4567.000000000000334a/head//3,ab96183a/rbd_data.10a56b8b4567.0000000000002daf/head//3,f46e183a/rbd_data.10a66b8b4567.0000000000001cdc/head//3,7fe183a/rbd_data.10a76b8b4567.0000000000003ab9/head//3,c0ff183a/rbd_data.10b16b8b4567.0000000000002ac1/head//3,b342283a/rbd_data.10986b8b4567.00000000000001dd/head//3,49b2283a/rbd_data.10946b8b4567.0000000000001f1c/head//3,f433283a/rbd_data.10b06b8b4567.0000000000003315/head//3,7544283a/rbd_data.10906b8b4567.000000000000173f/head//3,9964283a/rbd_data.109e6b8b4567.00000000000003a4/head//3,6666283a/rbd_data.10976b8b4567.00000000000028c8/head//3,fec7283a/rbd_data.10946b8b4567.000000000000262b/head//3,ccc8283a/rbd_data.10996b8b4567.00000000000001f5/head//3,721d283a/rbd_data.108f6b8b4567.0000000000001b20/head//3,d551383a/rbd_data.109a6b8b4567.00000000000000d7/head//3,da51383a/rbd_data.10ab6b8b4567.0000000000000c82/head//3,10d1383a/rbd_data.10a06b8b4567.0000000000003aa8/head//3,4262383a/rbd_data.10ae6b8b4567.000000000000306f/head//3,5e75383a/rbd_data.10a76b8b4567.0000000000002b2d/head//3,60b9383a/rbd_data.109e6b8b4567.0000000000003347/head//3,908b383a/rbd_data.10b26b8b4567.0000000000003b2f/head//3,a9bb383a/rbd_data.10926b8b4567.00000000000026ed/head//3,c24e383a/rbd_data.10aa6b8b4567.0000000000000ba0/head//3,ffbf383a/rbd_data.10a66b8b4567.0000000000003bae/head//3,e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3,dd60483a/rbd_data.10956b8b4567.000000000000188c/head//3,f171483a/rbd_data.10a76b8b4567.0000000000003901/head//3,c23483a/rbd_data.10ae6b8b4567.0000000000001a94/head//3]
2014-06-12 14:51:34.705657 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:34.705688 7f9d60bd2700 10 stat 3.3a_head/159bd03a/rbd_data.10a46b8b4567.0000000000002215/head//3
2014-06-12 14:51:34.705903 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 159bd03a/rbd_data.10a46b8b4567.0000000000002215/head//3
2014-06-12 14:51:34.725247 7f9d60bd2700 10 getattrs 3.3a_head/159bd03a/rbd_data.10a46b8b4567.0000000000002215/head//3 = 0
2014-06-12 14:51:34.725272 7f9d60bd2700 10 stat 3.3a_head/c79cd03a/rbd_data.10ac6b8b4567.0000000000000443/head//3
2014-06-12 14:51:34.725482 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c79cd03a/rbd_data.10ac6b8b4567.0000000000000443/head//3
2014-06-12 14:51:34.749579 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:34.749643 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:34.749652 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.46 < max 0.5 = yes
2014-06-12 14:51:34.749668 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=1
2014-06-12 14:51:34.767851 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:34.780919 7f9d60bd2700 10 getattrs 3.3a_head/c79cd03a/rbd_data.10ac6b8b4567.0000000000000443/head//3 = 0
2014-06-12 14:51:34.780928 7f9d60bd2700 10 stat 3.3a_head/d3ecd03a/rbd_data.10976b8b4567.00000000000032c2/head//3
2014-06-12 14:51:34.781027 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid d3ecd03a/rbd_data.10976b8b4567.00000000000032c2/head//3
2014-06-12 14:51:34.801769 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:34.804387 7f9d60bd2700 10 getattrs 3.3a_head/d3ecd03a/rbd_data.10976b8b4567.00000000000032c2/head//3 = 0
2014-06-12 14:51:34.804396 7f9d60bd2700 10 stat 3.3a_head/e63dd03a/rbd_data.10b36b8b4567.0000000000003bae/head//3
2014-06-12 14:51:34.804501 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid e63dd03a/rbd_data.10b36b8b4567.0000000000003bae/head//3
2014-06-12 14:51:34.831308 7f9d60bd2700 10 getattrs 3.3a_head/e63dd03a/rbd_data.10b36b8b4567.0000000000003bae/head//3 = 0
2014-06-12 14:51:34.831319 7f9d60bd2700 10 stat 3.3a_head/d36dd03a/rbd_data.10a86b8b4567.0000000000003a0f/head//3
2014-06-12 14:51:34.831439 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid d36dd03a/rbd_data.10a86b8b4567.0000000000003a0f/head//3
2014-06-12 14:51:34.854902 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:34.865228 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:34.867669 7f9d60bd2700 10 getattrs 3.3a_head/d36dd03a/rbd_data.10a86b8b4567.0000000000003a0f/head//3 = 0
2014-06-12 14:51:34.867681 7f9d60bd2700 10 stat 3.3a_head/4e8dd03a/rbd_data.10a06b8b4567.00000000000025c9/head//3
2014-06-12 14:51:34.867826 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 4e8dd03a/rbd_data.10a06b8b4567.00000000000025c9/head//3
2014-06-12 14:51:34.868609 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:34.876769 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:34.876831 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:34.896056 7f9d60bd2700 10 getattrs 3.3a_head/4e8dd03a/rbd_data.10a06b8b4567.00000000000025c9/head//3 = 0
2014-06-12 14:51:34.896063 7f9d60bd2700 10 stat 3.3a_head/c6bfd03a/rbd_data.10a96b8b4567.0000000000001b39/head//3
2014-06-12 14:51:34.896158 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c6bfd03a/rbd_data.10a96b8b4567.0000000000001b39/head//3
2014-06-12 14:51:34.924069 7f9d60bd2700 10 getattrs 3.3a_head/c6bfd03a/rbd_data.10a96b8b4567.0000000000001b39/head//3 = 0
2014-06-12 14:51:34.924079 7f9d60bd2700 10 stat 3.3a_head/a6efd03a/rbd_data.10976b8b4567.00000000000009a1/head//3
2014-06-12 14:51:34.924167 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a6efd03a/rbd_data.10976b8b4567.00000000000009a1/head//3
2014-06-12 14:51:34.928325 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:34.935768 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:34.962155 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:34.962396 7f9d60bd2700 10 getattrs 3.3a_head/a6efd03a/rbd_data.10976b8b4567.00000000000009a1/head//3 = 0
2014-06-12 14:51:34.962410 7f9d60bd2700 10 stat 3.3a_head/84f0e03a/rbd_data.10936b8b4567.00000000000023b4/head//3
2014-06-12 14:51:34.962542 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 84f0e03a/rbd_data.10936b8b4567.00000000000023b4/head//3
2014-06-12 14:51:34.965893 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:34.985436 7f9d60bd2700 10 getattrs 3.3a_head/84f0e03a/rbd_data.10936b8b4567.00000000000023b4/head//3 = 0
2014-06-12 14:51:34.985447 7f9d60bd2700 10 stat 3.3a_head/a035e03a/rbd_data.10956b8b4567.0000000000001e90/head//3
2014-06-12 14:51:34.985572 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a035e03a/rbd_data.10956b8b4567.0000000000001e90/head//3
2014-06-12 14:51:35.019059 7f9d60bd2700 10 getattrs 3.3a_head/a035e03a/rbd_data.10956b8b4567.0000000000001e90/head//3 = 0
2014-06-12 14:51:35.019070 7f9d60bd2700 10 stat 3.3a_head/ad09e03a/rbd_data.10906b8b4567.0000000000000c16/head//3
2014-06-12 14:51:35.019194 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ad09e03a/rbd_data.10906b8b4567.0000000000000c16/head//3
2014-06-12 14:51:35.069677 7f9d60bd2700 10 getattrs 3.3a_head/ad09e03a/rbd_data.10906b8b4567.0000000000000c16/head//3 = 0
2014-06-12 14:51:35.069700 7f9d60bd2700 10 stat 3.3a_head/47cce03a/rbd_data.10a76b8b4567.0000000000000889/head//3
2014-06-12 14:51:35.069850 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 47cce03a/rbd_data.10a76b8b4567.0000000000000889/head//3
2014-06-12 14:51:35.120096 7f9d60bd2700 10 getattrs 3.3a_head/47cce03a/rbd_data.10a76b8b4567.0000000000000889/head//3 = 0
2014-06-12 14:51:35.120117 7f9d60bd2700 10 stat 3.3a_head/a65de03a/rbd_data.10a56b8b4567.0000000000000162/head//3
2014-06-12 14:51:35.120236 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a65de03a/rbd_data.10a56b8b4567.0000000000000162/head//3
2014-06-12 14:51:35.121125 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:35.132810 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:35.142247 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:35.143949 7f9d60bd2700 10 getattrs 3.3a_head/a65de03a/rbd_data.10a56b8b4567.0000000000000162/head//3 = 0
2014-06-12 14:51:35.143962 7f9d60bd2700 10 stat 3.3a_head/137de03a/rbd_data.10a76b8b4567.0000000000002796/head//3
2014-06-12 14:51:35.144100 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 137de03a/rbd_data.10a76b8b4567.0000000000002796/head//3
2014-06-12 14:51:35.151677 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:35.159611 7f9d60bd2700 10 getattrs 3.3a_head/137de03a/rbd_data.10a76b8b4567.0000000000002796/head//3 = 0
2014-06-12 14:51:35.159624 7f9d60bd2700 10 stat 3.3a_head/b8bde03a/rbd_data.108d6b8b4567.0000000000000e47/head//3
2014-06-12 14:51:35.159748 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid b8bde03a/rbd_data.108d6b8b4567.0000000000000e47/head//3
2014-06-12 14:51:35.180122 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:35.209207 7f9d60bd2700 10 getattrs 3.3a_head/b8bde03a/rbd_data.108d6b8b4567.0000000000000e47/head//3 = 0
2014-06-12 14:51:35.209233 7f9d60bd2700 10 stat 3.3a_head/235fe03a/rbd_data.10aa6b8b4567.00000000000002ba/head//3
2014-06-12 14:51:35.209364 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 235fe03a/rbd_data.10aa6b8b4567.00000000000002ba/head//3
2014-06-12 14:51:35.218864 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:35.259408 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:35.259456 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:35.272538 7f9d60bd2700 10 getattrs 3.3a_head/235fe03a/rbd_data.10aa6b8b4567.00000000000002ba/head//3 = 0
2014-06-12 14:51:35.272552 7f9d60bd2700 10 stat 3.3a_head/9971f03a/rbd_data.109f6b8b4567.000000000000298e/head//3
2014-06-12 14:51:35.272691 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 9971f03a/rbd_data.109f6b8b4567.000000000000298e/head//3
2014-06-12 14:51:35.284931 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:35.285014 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:35.289744 7f9d60bd2700 10 getattrs 3.3a_head/9971f03a/rbd_data.109f6b8b4567.000000000000298e/head//3 = 0
2014-06-12 14:51:35.289756 7f9d60bd2700 10 stat 3.3a_head/a6f3f03a/rbd_data.109e6b8b4567.000000000000335c/head//3
2014-06-12 14:51:35.289900 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a6f3f03a/rbd_data.109e6b8b4567.000000000000335c/head//3
2014-06-12 14:51:35.298291 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:35.307121 7f9d60bd2700 10 getattrs 3.3a_head/a6f3f03a/rbd_data.109e6b8b4567.000000000000335c/head//3 = 0
2014-06-12 14:51:35.307133 7f9d60bd2700 10 stat 3.3a_head/e865f03a/rbd_data.10b46b8b4567.0000000000000f5a/head//3
2014-06-12 14:51:35.307288 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid e865f03a/rbd_data.10b46b8b4567.0000000000000f5a/head//3
2014-06-12 14:51:35.326830 7f9d44101700  5 -- op tracker -- , seq: 748587, time: 2014-06-12 14:51:35.326735, event: header_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:35.326844 7f9d44101700  5 -- op tracker -- , seq: 748587, time: 2014-06-12 14:51:35.326737, event: throttled, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:35.326852 7f9d44101700  5 -- op tracker -- , seq: 748587, time: 2014-06-12 14:51:35.326782, event: all_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:35.326859 7f9d44101700  5 -- op tracker -- , seq: 748587, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:35.326863 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:35.326868 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:35.326876 7f9d44101700 15 osd.2 173 enqueue_op 0x2afd400 prio 127 cost 10717 latency 0.000141 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:35.326928 7f9d68be2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:35.341476 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:35.350852 7f9d60bd2700 10 getattrs 3.3a_head/e865f03a/rbd_data.10b46b8b4567.0000000000000f5a/head//3 = 0
2014-06-12 14:51:35.350867 7f9d60bd2700 10 stat 3.3a_head/acb5f03a/rbd_data.10ab6b8b4567.00000000000035ba/head//3
2014-06-12 14:51:35.351000 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid acb5f03a/rbd_data.10ab6b8b4567.00000000000035ba/head//3
2014-06-12 14:51:35.368095 7f9d60bd2700 10 getattrs 3.3a_head/acb5f03a/rbd_data.10ab6b8b4567.00000000000035ba/head//3 = 0
2014-06-12 14:51:35.368103 7f9d60bd2700 10 stat 3.3a_head/8b16f03a/rbd_data.10ab6b8b4567.00000000000005b0/head//3
2014-06-12 14:51:35.368198 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 8b16f03a/rbd_data.10ab6b8b4567.00000000000005b0/head//3
2014-06-12 14:51:35.374730 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:35.378817 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:35.424752 7f9d60bd2700 10 getattrs 3.3a_head/8b16f03a/rbd_data.10ab6b8b4567.00000000000005b0/head//3 = 0
2014-06-12 14:51:35.424764 7f9d60bd2700 10 stat 3.3a_head/a4b6f03a/rbd_data.10a96b8b4567.000000000000218b/head//3
2014-06-12 14:51:35.424879 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a4b6f03a/rbd_data.10a96b8b4567.000000000000218b/head//3
2014-06-12 14:51:35.424965 7f9d60bd2700 10 getattrs 3.3a_head/a4b6f03a/rbd_data.10a96b8b4567.000000000000218b/head//3 = 0
2014-06-12 14:51:35.424970 7f9d60bd2700 10 stat 3.3a_head/5667f03a/rbd_data.10a46b8b4567.00000000000001de/head//3
2014-06-12 14:51:35.425056 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5667f03a/rbd_data.10a46b8b4567.00000000000001de/head//3
2014-06-12 14:51:35.474166 7f9d60bd2700 10 getattrs 3.3a_head/5667f03a/rbd_data.10a46b8b4567.00000000000001de/head//3 = 0
2014-06-12 14:51:35.474180 7f9d60bd2700 10 stat 3.3a_head/e61ef03a/rbd_data.10966b8b4567.00000000000035a8/head//3
2014-06-12 14:51:35.474311 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid e61ef03a/rbd_data.10966b8b4567.00000000000035a8/head//3
2014-06-12 14:51:35.500134 7f9d60bd2700 10 getattrs 3.3a_head/e61ef03a/rbd_data.10966b8b4567.00000000000035a8/head//3 = 0
2014-06-12 14:51:35.500150 7f9d60bd2700 10 stat 3.3a_head/b3cef03a/rbd_data.10a86b8b4567.0000000000003318/head//3
2014-06-12 14:51:35.500300 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid b3cef03a/rbd_data.10a86b8b4567.0000000000003318/head//3
2014-06-12 14:51:35.513223 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:35.517832 7f9d60bd2700 10 getattrs 3.3a_head/b3cef03a/rbd_data.10a86b8b4567.0000000000003318/head//3 = 0
2014-06-12 14:51:35.517843 7f9d60bd2700 10 collection_getattrs 3.3a_head
2014-06-12 14:51:35.517898 7f9d60bd2700 10 collection_getattrs 3.3a_head = 0
2014-06-12 14:51:35.517901 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:35.517932 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:35.517937 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:35.517945 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:35.517948 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:35.517978 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3a at 2014-06-11 13:02:32.500954
2014-06-12 14:51:35.518024 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.43 at 2014-06-11 13:02:32.501298
2014-06-12 14:51:35.518035 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518039 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518061 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6b at 2014-06-11 13:02:32.502023
2014-06-12 14:51:35.518069 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518072 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518085 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6c at 2014-06-11 13:02:32.502351
2014-06-12 14:51:35.518093 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518095 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518109 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.a0 at 2014-06-11 13:02:32.502930
2014-06-12 14:51:35.518115 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518117 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.a0( v 173'2803 (0'0,173'2803] local-les=157 n=294 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2802 mlcod 173'2802 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518130 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.105 at 2014-06-11 13:02:32.503257
2014-06-12 14:51:35.518102 7f9d68be2700 10 osd.2 173 dequeue_op 0x2afd400 prio 127 cost 10717 latency 0.191367 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]
2014-06-12 14:51:35.518136 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518138 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2817 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518148 7f9d68be2700  5 -- op tracker -- , seq: 748587, time: 2014-06-12 14:51:35.518140, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:35.518152 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.122 at 2014-06-11 13:02:32.503770
2014-06-12 14:51:35.518178 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518178 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] handle_message: 0x2afd400
2014-06-12 14:51:35.518180 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518187 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.12b at 2014-06-11 13:02:32.504419
2014-06-12 14:51:35.518190 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518191 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.12b( v 173'3236 (79'235,173'3236] local-les=137 n=323 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3235 mlcod 173'3235 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518186 7f9d68be2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:35.518197 7f9d68be2700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:35.518198 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.131 at 2014-06-11 13:02:32.504931
2014-06-12 14:51:35.518209 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518207 7f9d68be2700  5 -- op tracker -- , seq: 748587, time: 2014-06-12 14:51:35.518204, event: started, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:35.518211 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518213 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:35.518218 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.15d at 2014-06-11 13:02:32.505494
2014-06-12 14:51:35.518221 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518222 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518231 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.165 at 2014-06-11 13:02:32.506051
2014-06-12 14:51:35.518233 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518234 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.165( v 173'3106 (79'105,173'3106] local-les=149 n=320 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3105 mlcod 173'3105 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518240 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.170 at 2014-06-11 13:02:32.506577
2014-06-12 14:51:35.518243 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518244 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3049 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518250 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1b5 at 2014-06-11 13:02:32.507250
2014-06-12 14:51:35.518252 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518253 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1b5( v 173'2905 (0'0,173'2905] local-les=153 n=320 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2904 mlcod 173'2904 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518264 7f9d68be2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] map version is 173'2964
2014-06-12 14:51:35.518262 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1bc at 2014-06-11 13:02:32.507758
2014-06-12 14:51:35.518271 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518273 7f9d68be2700 10 osd.2 173 dequeue_op 0x2afd400 finish
2014-06-12 14:51:35.518273 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2459 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518279 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1e1 at 2014-06-11 13:02:32.508326
2014-06-12 14:51:35.518282 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518278 7f9d68be2700  5 -- op tracker -- , seq: 748587, time: 2014-06-12 14:51:35.518276, event: done, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:35.518283 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2540 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518298 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1e8 at 2014-06-11 13:02:32.508866
2014-06-12 14:51:35.518301 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518302 7f9d68be2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:35.518305 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:35.518302 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3033 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518305 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:35.518308 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1eb at 2014-06-11 13:02:32.509421
2014-06-12 14:51:35.518312 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518313 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3095 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518311 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:35.518320 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1fd at 2014-06-11 13:02:32.510047
2014-06-12 14:51:35.518332 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518320 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:35.518333 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1fd( v 173'3047 (79'46,173'3047] local-les=157 n=305 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518334 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:35.518355 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.204 at 2014-06-11 13:02:32.510573
2014-06-12 14:51:35.518362 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518364 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.204( v 173'2982 (0'0,173'2982] local-les=161 n=308 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518376 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.209 at 2014-06-11 13:02:32.511095
2014-06-12 14:51:35.518381 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518383 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.209( v 173'2531 (0'0,173'2531] local-les=156 n=264 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2530 mlcod 173'2530 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518395 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.214 at 2014-06-11 13:02:32.511670
2014-06-12 14:51:35.518388 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:35.518401 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518403 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:35.518404 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2934 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518416 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:35.518421 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.21d at 2014-06-11 13:02:32.512202
2014-06-12 14:51:35.518429 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518431 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.21d( v 173'3024 (79'23,173'3024] local-les=156 n=318 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3023 mlcod 173'3023 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518444 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.23f at 2014-06-11 13:02:32.512776
2014-06-12 14:51:35.518453 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518455 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518455 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 159bd03a/rbd_data.10a46b8b4567.0000000000002215/head//3, auth == maps.end()
2014-06-12 14:51:35.518469 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.251 at 2014-06-11 13:02:32.513340
2014-06-12 14:51:35.518476 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518478 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518491 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.262 at 2014-06-11 13:02:32.513946
2014-06-12 14:51:35.518496 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518484 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 159bd03a/rbd_data.10a46b8b4567.0000000000002215/head//3
2014-06-12 14:51:35.518498 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.262( v 173'2675 (0'0,173'2675] local-les=142 n=292 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2674 mlcod 173'2674 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518511 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.266 at 2014-06-11 13:02:32.514523
2014-06-12 14:51:35.518516 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518510 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c79cd03a/rbd_data.10ac6b8b4567.0000000000000443/head//3, auth == maps.end()
2014-06-12 14:51:35.518518 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2824 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518527 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c79cd03a/rbd_data.10ac6b8b4567.0000000000000443/head//3
2014-06-12 14:51:35.518531 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.288 at 2014-06-11 13:02:32.515022
2014-06-12 14:51:35.518550 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518545 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d3ecd03a/rbd_data.10976b8b4567.00000000000032c2/head//3, auth == maps.end()
2014-06-12 14:51:35.518553 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.288( v 173'2548 (0'0,173'2548] local-les=161 n=263 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2547 mlcod 173'2547 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518567 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2b4 at 2014-06-11 13:02:32.515543
2014-06-12 14:51:35.518562 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d3ecd03a/rbd_data.10976b8b4567.00000000000032c2/head//3
2014-06-12 14:51:35.518573 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518575 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2b4( v 173'3109 (79'108,173'3109] local-les=137 n=312 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3108 mlcod 173'3108 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518580 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e63dd03a/rbd_data.10b36b8b4567.0000000000003bae/head//3, auth == maps.end()
2014-06-12 14:51:35.518589 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2c4 at 2014-06-11 13:02:32.516106
2014-06-12 14:51:35.518598 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518597 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e63dd03a/rbd_data.10b36b8b4567.0000000000003bae/head//3
2014-06-12 14:51:35.518601 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2c4( v 173'2897 (0'0,173'2897] local-les=139 n=307 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2896 mlcod 173'2896 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518614 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2d3 at 2014-06-11 13:02:32.516627
2014-06-12 14:51:35.518620 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518616 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d36dd03a/rbd_data.10a86b8b4567.0000000000003a0f/head//3, auth == maps.end()
2014-06-12 14:51:35.518622 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2d3( v 173'2556 (0'0,173'2556] local-les=153 n=276 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2555 mlcod 173'2555 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518635 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2da at 2014-06-11 13:02:32.517119
2014-06-12 14:51:35.518640 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518632 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d36dd03a/rbd_data.10a86b8b4567.0000000000003a0f/head//3
2014-06-12 14:51:35.518642 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2da( v 173'2988 (0'0,173'2988] local-les=145 n=302 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2987 mlcod 173'2987 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518657 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2f4 at 2014-06-11 13:02:32.517764
2014-06-12 14:51:35.518651 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4e8dd03a/rbd_data.10a06b8b4567.00000000000025c9/head//3, auth == maps.end()
2014-06-12 14:51:35.518663 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518666 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3274 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518669 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4e8dd03a/rbd_data.10a06b8b4567.00000000000025c9/head//3
2014-06-12 14:51:35.518679 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.312 at 2014-06-11 13:02:32.518300
2014-06-12 14:51:35.518684 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518687 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2782 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518687 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c6bfd03a/rbd_data.10a96b8b4567.0000000000001b39/head//3, auth == maps.end()
2014-06-12 14:51:35.518701 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.320 at 2014-06-11 13:02:32.518824
2014-06-12 14:51:35.518708 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518705 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c6bfd03a/rbd_data.10a96b8b4567.0000000000001b39/head//3
2014-06-12 14:51:35.518710 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.320( v 173'2687 (0'0,173'2687] local-les=145 n=282 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2686 mlcod 173'2686 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518724 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.330 at 2014-06-11 13:02:32.519349
2014-06-12 14:51:35.518723 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a6efd03a/rbd_data.10976b8b4567.00000000000009a1/head//3, auth == maps.end()
2014-06-12 14:51:35.518733 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518736 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2798 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518739 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a6efd03a/rbd_data.10976b8b4567.00000000000009a1/head//3
2014-06-12 14:51:35.518749 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.342 at 2014-06-11 13:02:32.519873
2014-06-12 14:51:35.518755 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518757 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2464 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518757 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 84f0e03a/rbd_data.10936b8b4567.00000000000023b4/head//3, auth == maps.end()
2014-06-12 14:51:35.518771 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.35b at 2014-06-11 13:02:32.520528
2014-06-12 14:51:35.518777 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518775 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 84f0e03a/rbd_data.10936b8b4567.00000000000023b4/head//3
2014-06-12 14:51:35.518779 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.35b( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518792 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.366 at 2014-06-11 13:02:32.521097
2014-06-12 14:51:35.518798 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518793 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a035e03a/rbd_data.10956b8b4567.0000000000001e90/head//3, auth == maps.end()
2014-06-12 14:51:35.518800 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2661 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518814 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.37a at 2014-06-11 13:02:32.521744
2014-06-12 14:51:35.518820 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518809 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a035e03a/rbd_data.10956b8b4567.0000000000001e90/head//3
2014-06-12 14:51:35.518822 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.37a( v 173'2555 (0'0,173'2555] local-les=153 n=292 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2554 mlcod 173'2554 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518827 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ad09e03a/rbd_data.10906b8b4567.0000000000000c16/head//3, auth == maps.end()
2014-06-12 14:51:35.518835 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.394 at 2014-06-11 13:02:32.522362
2014-06-12 14:51:35.518841 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518843 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.394( v 173'2648 (0'0,173'2648] local-les=145 n=266 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518844 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ad09e03a/rbd_data.10906b8b4567.0000000000000c16/head//3
2014-06-12 14:51:35.518855 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.395 at 2014-06-11 13:02:32.522991
2014-06-12 14:51:35.518861 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518861 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 47cce03a/rbd_data.10a76b8b4567.0000000000000889/head//3, auth == maps.end()
2014-06-12 14:51:35.518863 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3327 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518877 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3ac at 2014-06-11 13:02:32.523569
2014-06-12 14:51:35.518882 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518878 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 47cce03a/rbd_data.10a76b8b4567.0000000000000889/head//3
2014-06-12 14:51:35.518884 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3ac( v 173'2860 (0'0,173'2860] local-les=142 n=294 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518897 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a65de03a/rbd_data.10a56b8b4567.0000000000000162/head//3, auth == maps.end()
2014-06-12 14:51:35.518902 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3d4 at 2014-06-11 13:02:32.524119
2014-06-12 14:51:35.518909 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518911 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3d4( v 173'2696 (0'0,173'2696] local-les=137 n=271 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518913 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a65de03a/rbd_data.10a56b8b4567.0000000000000162/head//3
2014-06-12 14:51:35.518924 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3d8 at 2014-06-11 13:02:32.524641
2014-06-12 14:51:35.518929 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518931 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3d8( v 173'2879 (0'0,173'2879] local-les=156 n=294 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518930 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 137de03a/rbd_data.10a76b8b4567.0000000000002796/head//3, auth == maps.end()
2014-06-12 14:51:35.518944 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.401 at 2014-06-11 13:02:32.525188
2014-06-12 14:51:35.518950 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518947 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 137de03a/rbd_data.10a76b8b4567.0000000000002796/head//3
2014-06-12 14:51:35.518952 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3154 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518968 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.408 at 2014-06-11 13:02:32.525838
2014-06-12 14:51:35.518974 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.518965 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b8bde03a/rbd_data.108d6b8b4567.0000000000000e47/head//3, auth == maps.end()
2014-06-12 14:51:35.518976 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2883 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.518986 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b8bde03a/rbd_data.108d6b8b4567.0000000000000e47/head//3
2014-06-12 14:51:35.518988 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.411 at 2014-06-11 13:02:32.526419
2014-06-12 14:51:35.519007 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519003 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 235fe03a/rbd_data.10aa6b8b4567.00000000000002ba/head//3, auth == maps.end()
2014-06-12 14:51:35.519010 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.411( v 173'2402 (0'0,173'2402] local-les=147 n=278 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2401 mlcod 173'2401 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519023 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.45f at 2014-06-11 13:02:32.526940
2014-06-12 14:51:35.519029 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519020 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 235fe03a/rbd_data.10aa6b8b4567.00000000000002ba/head//3
2014-06-12 14:51:35.519032 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2357 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519044 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.479 at 2014-06-11 13:02:32.527498
2014-06-12 14:51:35.519050 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519038 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9971f03a/rbd_data.109f6b8b4567.000000000000298e/head//3, auth == maps.end()
2014-06-12 14:51:35.519052 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.479( v 173'2926 (0'0,173'2926] local-les=142 n=298 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519057 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9971f03a/rbd_data.109f6b8b4567.000000000000298e/head//3
2014-06-12 14:51:35.519064 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4a7 at 2014-06-11 13:02:32.528023
2014-06-12 14:51:35.519070 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519073 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2594 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519075 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a6f3f03a/rbd_data.109e6b8b4567.000000000000335c/head//3, auth == maps.end()
2014-06-12 14:51:35.519085 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4b4 at 2014-06-11 13:02:32.528548
2014-06-12 14:51:35.519090 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519093 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4b4( v 173'2792 (0'0,173'2792] local-les=137 n=305 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519091 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a6f3f03a/rbd_data.109e6b8b4567.000000000000335c/head//3
2014-06-12 14:51:35.519105 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4d4 at 2014-06-11 13:02:32.529466
2014-06-12 14:51:35.519111 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519110 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e865f03a/rbd_data.10b46b8b4567.0000000000000f5a/head//3, auth == maps.end()
2014-06-12 14:51:35.519114 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2745 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519126 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4ef at 2014-06-11 13:02:32.530034
2014-06-12 14:51:35.519132 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519127 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e865f03a/rbd_data.10b46b8b4567.0000000000000f5a/head//3
2014-06-12 14:51:35.519134 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4ef( v 173'3004 (79'3,173'3004] local-les=149 n=307 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519151 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4f0 at 2014-06-11 13:02:32.530572
2014-06-12 14:51:35.519145 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj acb5f03a/rbd_data.10ab6b8b4567.00000000000035ba/head//3, auth == maps.end()
2014-06-12 14:51:35.519156 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519159 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2864 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519161 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj acb5f03a/rbd_data.10ab6b8b4567.00000000000035ba/head//3
2014-06-12 14:51:35.519171 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.54b at 2014-06-11 13:02:32.531354
2014-06-12 14:51:35.519177 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519179 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2722 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519179 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8b16f03a/rbd_data.10ab6b8b4567.00000000000005b0/head//3, auth == maps.end()
2014-06-12 14:51:35.519193 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.568 at 2014-06-11 13:02:32.531914
2014-06-12 14:51:35.519199 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519197 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8b16f03a/rbd_data.10ab6b8b4567.00000000000005b0/head//3
2014-06-12 14:51:35.519201 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.568( v 173'2727 (0'0,173'2727] local-les=161 n=270 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519214 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.58f at 2014-06-11 13:02:32.532612
2014-06-12 14:51:35.519219 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519215 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a4b6f03a/rbd_data.10a96b8b4567.000000000000218b/head//3, auth == maps.end()
2014-06-12 14:51:35.519222 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.58f( v 173'2707 (0'0,173'2707] local-les=153 n=304 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519235 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5a3 at 2014-06-11 13:02:32.533139
2014-06-12 14:51:35.519241 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519232 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a4b6f03a/rbd_data.10a96b8b4567.000000000000218b/head//3
2014-06-12 14:51:35.519243 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5a3( v 173'2874 (0'0,173'2874] local-les=142 n=296 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2873 mlcod 173'2873 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519256 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5ce at 2014-06-11 13:02:32.533636
2014-06-12 14:51:35.519250 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5667f03a/rbd_data.10a46b8b4567.00000000000001de/head//3, auth == maps.end()
2014-06-12 14:51:35.519263 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519265 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5ce( v 173'2741 (0'0,173'2741] local-les=153 n=294 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519267 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5667f03a/rbd_data.10a46b8b4567.00000000000001de/head//3
2014-06-12 14:51:35.519277 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5dd at 2014-06-11 13:02:32.534211
2014-06-12 14:51:35.519283 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519285 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5dd( v 173'2852 (0'0,173'2852] local-les=156 n=307 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519284 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e61ef03a/rbd_data.10966b8b4567.00000000000035a8/head//3, auth == maps.end()
2014-06-12 14:51:35.519298 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5fc at 2014-06-11 13:02:32.534738
2014-06-12 14:51:35.519303 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519301 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e61ef03a/rbd_data.10966b8b4567.00000000000035a8/head//3
2014-06-12 14:51:35.519305 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5fc( v 173'2714 (0'0,173'2714] local-les=157 n=284 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519318 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.602 at 2014-06-11 13:02:32.535274
2014-06-12 14:51:35.519324 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519319 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b3cef03a/rbd_data.10a86b8b4567.0000000000003318/head//3, auth == maps.end()
2014-06-12 14:51:35.519326 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3009 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519339 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.605 at 2014-06-11 13:02:32.535829
2014-06-12 14:51:35.519344 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519336 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b3cef03a/rbd_data.10a86b8b4567.0000000000003318/head//3
2014-06-12 14:51:35.519346 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519359 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.613 at 2014-06-11 13:02:32.536448
2014-06-12 14:51:35.519365 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519356 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] 
2014-06-12 14:51:35.519368 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519371 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub
2014-06-12 14:51:35.519382 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.635 at 2014-06-11 13:02:32.537041
2014-06-12 14:51:35.519388 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519390 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.635( v 173'2966 (0'0,173'2966] local-les=137 n=308 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2965 mlcod 173'2965 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519403 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.63b at 2014-06-11 13:02:32.537647
2014-06-12 14:51:35.519409 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519394 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  b3cef03a/rbd_data.10a86b8b4567.0000000000003318/head//3 b3cef03a/rbd_data.10a86b8b4567.0000000000003318/head//3(173'2045 client.4305.0:100832 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2045)
2014-06-12 14:51:35.519411 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519424 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.63e at 2014-06-11 13:02:32.538178
2014-06-12 14:51:35.519429 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519419 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  e61ef03a/rbd_data.10966b8b4567.00000000000035a8/head//3 e61ef03a/rbd_data.10966b8b4567.00000000000035a8/head//3(173'2185 client.4287.0:103560 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2185)
2014-06-12 14:51:35.519432 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519444 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.650 at 2014-06-11 13:02:32.538854
2014-06-12 14:51:35.519450 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519442 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5667f03a/rbd_data.10a46b8b4567.00000000000001de/head//3 5667f03a/rbd_data.10a46b8b4567.00000000000001de/head//3(173'2526 client.4301.0:121294 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2526)
2014-06-12 14:51:35.519452 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519464 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.662 at 2014-06-11 13:02:32.539391
2014-06-12 14:51:35.519470 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519463 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a4b6f03a/rbd_data.10a96b8b4567.000000000000218b/head//3 a4b6f03a/rbd_data.10a96b8b4567.000000000000218b/head//3(173'1621 client.4306.0:72070 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1621)
2014-06-12 14:51:35.519472 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.662( v 173'2604 (0'0,173'2604] local-les=161 n=257 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2603 mlcod 173'2603 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519485 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.667 at 2014-06-11 13:02:32.539999
2014-06-12 14:51:35.519490 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519485 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  8b16f03a/rbd_data.10ab6b8b4567.00000000000005b0/head//3 8b16f03a/rbd_data.10ab6b8b4567.00000000000005b0/head//3(173'2658 client.4308.0:127970 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2658)
2014-06-12 14:51:35.519492 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.667( v 173'2958 (0'0,173'2958] local-les=149 n=303 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519506 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6a8 at 2014-06-11 13:02:32.540566
2014-06-12 14:51:35.519516 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519507 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  acb5f03a/rbd_data.10ab6b8b4567.00000000000035ba/head//3 acb5f03a/rbd_data.10ab6b8b4567.00000000000035ba/head//3(173'2261 client.4308.0:107310 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2261)
2014-06-12 14:51:35.519518 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6a8( v 173'2854 (0'0,173'2854] local-les=137 n=299 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519531 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6ff at 2014-06-11 13:02:32.541074
2014-06-12 14:51:35.519537 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519528 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  e865f03a/rbd_data.10b46b8b4567.0000000000000f5a/head//3 e865f03a/rbd_data.10b46b8b4567.0000000000000f5a/head//3(173'2947 client.4317.0:142314 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2947)
2014-06-12 14:51:35.519539 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6ff( v 173'2792 (0'0,173'2792] local-les=161 n=298 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519551 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.711 at 2014-06-11 13:02:32.541770
2014-06-12 14:51:35.519557 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519549 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a6f3f03a/rbd_data.109e6b8b4567.000000000000335c/head//3 a6f3f03a/rbd_data.109e6b8b4567.000000000000335c/head//3(173'2079 client.4295.0:102306 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2079)
2014-06-12 14:51:35.519559 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.711( v 173'2514 (0'0,173'2514] local-les=147 n=297 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519572 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.712 at 2014-06-11 13:02:32.542314
2014-06-12 14:51:35.519578 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519570 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  9971f03a/rbd_data.109f6b8b4567.000000000000298e/head//3 9971f03a/rbd_data.109f6b8b4567.000000000000298e/head//3(173'1663 client.4296.0:81988 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1663)
2014-06-12 14:51:35.519580 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.712( v 173'2782 (0'0,173'2782] local-les=153 n=305 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519596 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.717 at 2014-06-11 13:02:32.542947
2014-06-12 14:51:35.519602 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519592 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  235fe03a/rbd_data.10aa6b8b4567.00000000000002ba/head//3 235fe03a/rbd_data.10aa6b8b4567.00000000000002ba/head//3(173'2535 client.4307.0:123628 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2535)
2014-06-12 14:51:35.519604 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2583 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519616 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.721 at 2014-06-11 13:02:32.543618
2014-06-12 14:51:35.519622 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519613 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  b8bde03a/rbd_data.108d6b8b4567.0000000000000e47/head//3 b8bde03a/rbd_data.108d6b8b4567.0000000000000e47/head//3(173'2917 client.4278.0:141237 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2917)
2014-06-12 14:51:35.519624 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2895 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519636 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.735 at 2014-06-11 13:02:32.544147
2014-06-12 14:51:35.519642 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519635 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  137de03a/rbd_data.10a76b8b4567.0000000000002796/head//3 137de03a/rbd_data.10a76b8b4567.0000000000002796/head//3(173'1791 client.4304.0:80894 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1791)
2014-06-12 14:51:35.519644 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.735( v 173'2803 (0'0,173'2803] local-les=137 n=306 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2802 mlcod 173'2802 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519657 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.762 at 2014-06-11 13:02:32.544735
2014-06-12 14:51:35.519663 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519656 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a65de03a/rbd_data.10a56b8b4567.0000000000000162/head//3 a65de03a/rbd_data.10a56b8b4567.0000000000000162/head//3(173'2485 client.4302.0:120989 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2485)
2014-06-12 14:51:35.519665 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.762( v 173'2726 (0'0,173'2726] local-les=147 n=302 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519677 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.76e at 2014-06-11 13:02:32.545259
2014-06-12 14:51:35.519683 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519677 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  47cce03a/rbd_data.10a76b8b4567.0000000000000889/head//3 47cce03a/rbd_data.10a76b8b4567.0000000000000889/head//3(173'2748 client.4304.0:130538 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2748)
2014-06-12 14:51:35.519685 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2648 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519698 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.77e at 2014-06-11 13:02:32.545813
2014-06-12 14:51:35.519704 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519706 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.77e( v 173'3052 (79'51,173'3052] local-les=161 n=319 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3051 mlcod 173'3051 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519699 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ad09e03a/rbd_data.10906b8b4567.0000000000000c16/head//3 ad09e03a/rbd_data.10906b8b4567.0000000000000c16/head//3(173'2863 client.4281.0:135972 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2863)
2014-06-12 14:51:35.519718 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.783 at 2014-06-11 13:02:32.546436
2014-06-12 14:51:35.519723 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519726 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519724 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a035e03a/rbd_data.10956b8b4567.0000000000001e90/head//3 a035e03a/rbd_data.10956b8b4567.0000000000001e90/head//3(173'1323 client.4286.0:64105 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1323)
2014-06-12 14:51:35.519738 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.785 at 2014-06-11 13:02:32.547026
2014-06-12 14:51:35.519744 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519746 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519746 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  84f0e03a/rbd_data.10936b8b4567.00000000000023b4/head//3 84f0e03a/rbd_data.10936b8b4567.00000000000023b4/head//3(173'1602 client.4284.0:71888 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1602)
2014-06-12 14:51:35.519760 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.793 at 2014-06-11 13:02:32.547568
2014-06-12 14:51:35.519765 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519767 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.793( v 173'2605 (0'0,173'2605] local-les=153 n=277 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2604 mlcod 173'2604 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519768 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a6efd03a/rbd_data.10976b8b4567.00000000000009a1/head//3 a6efd03a/rbd_data.10976b8b4567.00000000000009a1/head//3(173'2802 client.4288.0:130280 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2802)
2014-06-12 14:51:35.519781 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ab at 2014-06-11 13:02:32.548162
2014-06-12 14:51:35.519787 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519789 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ab( v 173'2921 (0'0,173'2921] local-les=161 n=302 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2920 mlcod 173'2920 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519790 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c6bfd03a/rbd_data.10a96b8b4567.0000000000001b39/head//3 c6bfd03a/rbd_data.10a96b8b4567.0000000000001b39/head//3(173'1290 client.4306.0:60282 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1290)
2014-06-12 14:51:35.519801 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ad at 2014-06-11 13:02:32.548789
2014-06-12 14:51:35.519807 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519809 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ad( v 173'3043 (79'42,173'3043] local-les=149 n=315 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519811 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  4e8dd03a/rbd_data.10a06b8b4567.00000000000025c9/head//3 4e8dd03a/rbd_data.10a06b8b4567.00000000000025c9/head//3(173'1593 client.4297.0:76136 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1593)
2014-06-12 14:51:35.519826 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ae at 2014-06-11 13:02:32.549514
2014-06-12 14:51:35.519832 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519834 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2517 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519832 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  d36dd03a/rbd_data.10a86b8b4567.0000000000003a0f/head//3 d36dd03a/rbd_data.10a86b8b4567.0000000000003a0f/head//3(173'2286 client.4305.0:114088 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2286)
2014-06-12 14:51:35.519847 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7bc at 2014-06-11 13:02:32.550208
2014-06-12 14:51:35.519852 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519854 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7bc( v 173'2562 (0'0,173'2562] local-les=153 n=287 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2561 mlcod 173'2561 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519854 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  e63dd03a/rbd_data.10b36b8b4567.0000000000003bae/head//3 e63dd03a/rbd_data.10b36b8b4567.0000000000003bae/head//3(173'2425 client.4316.0:118897 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2425)
2014-06-12 14:51:35.519868 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7cc at 2014-06-11 13:02:32.550758
2014-06-12 14:51:35.519874 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519876 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519877 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  d3ecd03a/rbd_data.10976b8b4567.00000000000032c2/head//3 d3ecd03a/rbd_data.10976b8b4567.00000000000032c2/head//3(173'2208 client.4288.0:98874 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2208)
2014-06-12 14:51:35.519889 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7db at 2014-06-11 13:02:32.551269
2014-06-12 14:51:35.519894 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519896 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2681 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519898 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c79cd03a/rbd_data.10ac6b8b4567.0000000000000443/head//3 c79cd03a/rbd_data.10ac6b8b4567.0000000000000443/head//3(173'2606 client.4309.0:122421 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2606)
2014-06-12 14:51:35.519908 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7f8 at 2014-06-11 13:02:32.551909
2014-06-12 14:51:35.519914 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:35.519916 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:35.519918 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  159bd03a/rbd_data.10a46b8b4567.0000000000002215/head//3 159bd03a/rbd_data.10a46b8b4567.0000000000002215/head//3(173'1458 client.4301.0:70973 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1458)
2014-06-12 14:51:35.519935 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:35.519928 7f9d84175700 10 osd.2 173 sched_scrub 0.3 at 2014-06-12 13:01:39.405748: 6595.34 < min (86400 seconds)
2014-06-12 14:51:35.519947 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:35.519950 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:35.519951 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:35.519953 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:35.519966 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:35.519969 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:35.519972 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:35.520030 7f9d60bd2700 10 collection_list_partial 3.3a_head start:3daff03a//0//-1 is_max:0
2014-06-12 14:51:35.520394 7f9d60bd2700 20 objects: [b14083a/rbd_data.108d6b8b4567.0000000000003892/head//3,eb54083a/rbd_data.109f6b8b4567.00000000000000cf/head//3,8ba6083a/rbd_data.10b26b8b4567.0000000000002807/head//3,12f6083a/rbd_data.108d6b8b4567.0000000000002844/head//3,4997083a/rbd_data.109b6b8b4567.0000000000001d04/head//3,8d7b083a/rbd_data.10936b8b4567.0000000000000835/head//3,e1be083a/rbd_data.10976b8b4567.0000000000001285/head//3,ef00183a/rbd_data.10b26b8b4567.0000000000001d9c/head//3,76e2183a/rbd_data.10b36b8b4567.0000000000001bcf/head//3,b803183a/rbd_data.10986b8b4567.0000000000001df7/head//3,cf86183a/rbd_data.10a26b8b4567.000000000000334a/head//3,ab96183a/rbd_data.10a56b8b4567.0000000000002daf/head//3,f46e183a/rbd_data.10a66b8b4567.0000000000001cdc/head//3,7fe183a/rbd_data.10a76b8b4567.0000000000003ab9/head//3,c0ff183a/rbd_data.10b16b8b4567.0000000000002ac1/head//3,b342283a/rbd_data.10986b8b4567.00000000000001dd/head//3,49b2283a/rbd_data.10946b8b4567.0000000000001f1c/head//3,f433283a/rbd_data.10b06b8b4567.0000000000003315/head//3,7544283a/rbd_data.10906b8b4567.000000000000173f/head//3,9964283a/rbd_data.109e6b8b4567.00000000000003a4/head//3,6666283a/rbd_data.10976b8b4567.00000000000028c8/head//3,fec7283a/rbd_data.10946b8b4567.000000000000262b/head//3,ccc8283a/rbd_data.10996b8b4567.00000000000001f5/head//3,721d283a/rbd_data.108f6b8b4567.0000000000001b20/head//3,d551383a/rbd_data.109a6b8b4567.00000000000000d7/head//3]
2014-06-12 14:51:35.521028 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:35.521059 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:35.521091 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:35.521113 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:35.521131 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:35.521150 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:35.521166 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_map_chunk [3daff03a//0//-1,da51383a//0//-1)
2014-06-12 14:51:35.521180 7f9d60bd2700 10 collection_list_partial 3.3a_head start:3daff03a//0//-1 is_max:0
2014-06-12 14:51:35.522054 7f9d60bd2700 20 objects: [b14083a/rbd_data.108d6b8b4567.0000000000003892/head//3,eb54083a/rbd_data.109f6b8b4567.00000000000000cf/head//3,8ba6083a/rbd_data.10b26b8b4567.0000000000002807/head//3,12f6083a/rbd_data.108d6b8b4567.0000000000002844/head//3,4997083a/rbd_data.109b6b8b4567.0000000000001d04/head//3,8d7b083a/rbd_data.10936b8b4567.0000000000000835/head//3,e1be083a/rbd_data.10976b8b4567.0000000000001285/head//3,ef00183a/rbd_data.10b26b8b4567.0000000000001d9c/head//3,76e2183a/rbd_data.10b36b8b4567.0000000000001bcf/head//3,b803183a/rbd_data.10986b8b4567.0000000000001df7/head//3,cf86183a/rbd_data.10a26b8b4567.000000000000334a/head//3,ab96183a/rbd_data.10a56b8b4567.0000000000002daf/head//3,f46e183a/rbd_data.10a66b8b4567.0000000000001cdc/head//3,7fe183a/rbd_data.10a76b8b4567.0000000000003ab9/head//3,c0ff183a/rbd_data.10b16b8b4567.0000000000002ac1/head//3,b342283a/rbd_data.10986b8b4567.00000000000001dd/head//3,49b2283a/rbd_data.10946b8b4567.0000000000001f1c/head//3,f433283a/rbd_data.10b06b8b4567.0000000000003315/head//3,7544283a/rbd_data.10906b8b4567.000000000000173f/head//3,9964283a/rbd_data.109e6b8b4567.00000000000003a4/head//3,6666283a/rbd_data.10976b8b4567.00000000000028c8/head//3,fec7283a/rbd_data.10946b8b4567.000000000000262b/head//3,ccc8283a/rbd_data.10996b8b4567.00000000000001f5/head//3,721d283a/rbd_data.108f6b8b4567.0000000000001b20/head//3,d551383a/rbd_data.109a6b8b4567.00000000000000d7/head//3,da51383a/rbd_data.10ab6b8b4567.0000000000000c82/head//3,10d1383a/rbd_data.10a06b8b4567.0000000000003aa8/head//3,4262383a/rbd_data.10ae6b8b4567.000000000000306f/head//3,5e75383a/rbd_data.10a76b8b4567.0000000000002b2d/head//3,60b9383a/rbd_data.109e6b8b4567.0000000000003347/head//3,908b383a/rbd_data.10b26b8b4567.0000000000003b2f/head//3,a9bb383a/rbd_data.10926b8b4567.00000000000026ed/head//3,c24e383a/rbd_data.10aa6b8b4567.0000000000000ba0/head//3,ffbf383a/rbd_data.10a66b8b4567.0000000000003bae/head//3,e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3,dd60483a/rbd_data.10956b8b4567.000000000000188c/head//3,f171483a/rbd_data.10a76b8b4567.0000000000003901/head//3,c23483a/rbd_data.10ae6b8b4567.0000000000001a94/head//3,5644483a/rbd_data.10a46b8b4567.0000000000001a71/head//3,8078483a/rbd_data.10ae6b8b4567.000000000000082b/head//3,5a59483a/rbd_data.10a46b8b4567.0000000000002122/head//3,3efb483a/rbd_data.10a46b8b4567.000000000000366f/head//3,b7c483a/rbd_data.10946b8b4567.0000000000002088/head//3,91a1583a/rbd_data.10926b8b4567.0000000000001286/head//3,95a4583a/rbd_data.108f6b8b4567.0000000000001e1b/head//3,ac2a583a/rbd_data.10a06b8b4567.0000000000002c36/head//3,9b7a583a/rbd_data.10946b8b4567.00000000000038da/head//3,5c4c583a/rbd_data.109f6b8b4567.00000000000028e9/head//3,685d583a/rbd_data.10a56b8b4567.0000000000001919/head//3,d624683a/rbd_data.108d6b8b4567.0000000000000253/head//3,1284683a/rbd_data.10b16b8b4567.0000000000002ad5/head//3,9794683a/rbd_data.10b26b8b4567.0000000000001a4b/head//3,63b4683a/rbd_data.10976b8b4567.0000000000001bdf/head//3,2155683a/rbd_data.10a16b8b4567.000000000000239d/head//3,f648683a/rbd_data.10946b8b4567.00000000000031ef/head//3,217d683a/rbd_data.10966b8b4567.00000000000037cb/head//3,6c9d683a/rbd_data.10a16b8b4567.0000000000002092/head//3,6091783a/rbd_data.109d6b8b4567.0000000000002a9c/head//3,413783a/rbd_data.109f6b8b4567.0000000000001670/head//3,ee67783a/rbd_data.10b26b8b4567.0000000000001baf/head//3,139b783a/rbd_data.10996b8b4567.000000000000007c/head//3,b17c783a/rbd_data.10936b8b4567.0000000000002a26/head//3,3c7c783a/rbd_data.108d6b8b4567.000000000000395b/head//3,dd21883a/rbd_data.108e6b8b4567.000000000000225e/head//3]
2014-06-12 14:51:35.522189 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:35.522214 7f9d60bd2700 10 stat 3.3a_head/b14083a/rbd_data.108d6b8b4567.0000000000003892/head//3
2014-06-12 14:51:35.522431 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid b14083a/rbd_data.108d6b8b4567.0000000000003892/head//3
2014-06-12 14:51:35.547183 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:35.552968 7f9d60bd2700 10 getattrs 3.3a_head/b14083a/rbd_data.108d6b8b4567.0000000000003892/head//3 = 0
2014-06-12 14:51:35.552974 7f9d60bd2700 10 stat 3.3a_head/eb54083a/rbd_data.109f6b8b4567.00000000000000cf/head//3
2014-06-12 14:51:35.553071 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid eb54083a/rbd_data.109f6b8b4567.00000000000000cf/head//3
2014-06-12 14:51:35.588397 7f9d60bd2700 10 getattrs 3.3a_head/eb54083a/rbd_data.109f6b8b4567.00000000000000cf/head//3 = 0
2014-06-12 14:51:35.588413 7f9d60bd2700 10 stat 3.3a_head/8ba6083a/rbd_data.10b26b8b4567.0000000000002807/head//3
2014-06-12 14:51:35.588517 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 8ba6083a/rbd_data.10b26b8b4567.0000000000002807/head//3
2014-06-12 14:51:35.609064 7f9d60bd2700 10 getattrs 3.3a_head/8ba6083a/rbd_data.10b26b8b4567.0000000000002807/head//3 = 0
2014-06-12 14:51:35.609077 7f9d60bd2700 10 stat 3.3a_head/12f6083a/rbd_data.108d6b8b4567.0000000000002844/head//3
2014-06-12 14:51:35.609178 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 12f6083a/rbd_data.108d6b8b4567.0000000000002844/head//3
2014-06-12 14:51:35.626272 7f9d60bd2700 10 getattrs 3.3a_head/12f6083a/rbd_data.108d6b8b4567.0000000000002844/head//3 = 0
2014-06-12 14:51:35.626289 7f9d60bd2700 10 stat 3.3a_head/4997083a/rbd_data.109b6b8b4567.0000000000001d04/head//3
2014-06-12 14:51:35.626463 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 4997083a/rbd_data.109b6b8b4567.0000000000001d04/head//3
2014-06-12 14:51:35.644854 7f9d60bd2700 10 getattrs 3.3a_head/4997083a/rbd_data.109b6b8b4567.0000000000001d04/head//3 = 0
2014-06-12 14:51:35.644868 7f9d60bd2700 10 stat 3.3a_head/8d7b083a/rbd_data.10936b8b4567.0000000000000835/head//3
2014-06-12 14:51:35.645034 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 8d7b083a/rbd_data.10936b8b4567.0000000000000835/head//3
2014-06-12 14:51:35.676888 7f9d60bd2700 10 getattrs 3.3a_head/8d7b083a/rbd_data.10936b8b4567.0000000000000835/head//3 = 0
2014-06-12 14:51:35.676911 7f9d60bd2700 10 stat 3.3a_head/e1be083a/rbd_data.10976b8b4567.0000000000001285/head//3
2014-06-12 14:51:35.677050 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid e1be083a/rbd_data.10976b8b4567.0000000000001285/head//3
2014-06-12 14:51:35.692468 7f9d60bd2700 10 getattrs 3.3a_head/e1be083a/rbd_data.10976b8b4567.0000000000001285/head//3 = 0
2014-06-12 14:51:35.692485 7f9d60bd2700 10 stat 3.3a_head/ef00183a/rbd_data.10b26b8b4567.0000000000001d9c/head//3
2014-06-12 14:51:35.692632 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ef00183a/rbd_data.10b26b8b4567.0000000000001d9c/head//3
2014-06-12 14:51:35.710466 7f9d60bd2700 10 getattrs 3.3a_head/ef00183a/rbd_data.10b26b8b4567.0000000000001d9c/head//3 = 0
2014-06-12 14:51:35.710483 7f9d60bd2700 10 stat 3.3a_head/76e2183a/rbd_data.10b36b8b4567.0000000000001bcf/head//3
2014-06-12 14:51:35.710629 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 76e2183a/rbd_data.10b36b8b4567.0000000000001bcf/head//3
2014-06-12 14:51:35.710758 7f9d60bd2700 10 getattrs 3.3a_head/76e2183a/rbd_data.10b36b8b4567.0000000000001bcf/head//3 = 0
2014-06-12 14:51:35.710773 7f9d60bd2700 10 stat 3.3a_head/b803183a/rbd_data.10986b8b4567.0000000000001df7/head//3
2014-06-12 14:51:35.710892 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid b803183a/rbd_data.10986b8b4567.0000000000001df7/head//3
2014-06-12 14:51:35.736035 7f9d60bd2700 10 getattrs 3.3a_head/b803183a/rbd_data.10986b8b4567.0000000000001df7/head//3 = 0
2014-06-12 14:51:35.736043 7f9d60bd2700 10 stat 3.3a_head/cf86183a/rbd_data.10a26b8b4567.000000000000334a/head//3
2014-06-12 14:51:35.736127 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid cf86183a/rbd_data.10a26b8b4567.000000000000334a/head//3
2014-06-12 14:51:35.752863 7f9d60bd2700 10 getattrs 3.3a_head/cf86183a/rbd_data.10a26b8b4567.000000000000334a/head//3 = 0
2014-06-12 14:51:35.752872 7f9d60bd2700 10 stat 3.3a_head/ab96183a/rbd_data.10a56b8b4567.0000000000002daf/head//3
2014-06-12 14:51:35.752966 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ab96183a/rbd_data.10a56b8b4567.0000000000002daf/head//3
2014-06-12 14:51:35.764651 7f9d60bd2700 10 getattrs 3.3a_head/ab96183a/rbd_data.10a56b8b4567.0000000000002daf/head//3 = 0
2014-06-12 14:51:35.764660 7f9d60bd2700 10 stat 3.3a_head/f46e183a/rbd_data.10a66b8b4567.0000000000001cdc/head//3
2014-06-12 14:51:35.764759 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid f46e183a/rbd_data.10a66b8b4567.0000000000001cdc/head//3
2014-06-12 14:51:35.782881 7f9d60bd2700 10 getattrs 3.3a_head/f46e183a/rbd_data.10a66b8b4567.0000000000001cdc/head//3 = 0
2014-06-12 14:51:35.782895 7f9d60bd2700 10 stat 3.3a_head/7fe183a/rbd_data.10a76b8b4567.0000000000003ab9/head//3
2014-06-12 14:51:35.783047 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 7fe183a/rbd_data.10a76b8b4567.0000000000003ab9/head//3
2014-06-12 14:51:35.798313 7f9d60bd2700 10 getattrs 3.3a_head/7fe183a/rbd_data.10a76b8b4567.0000000000003ab9/head//3 = 0
2014-06-12 14:51:35.798320 7f9d60bd2700 10 stat 3.3a_head/c0ff183a/rbd_data.10b16b8b4567.0000000000002ac1/head//3
2014-06-12 14:51:35.798417 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c0ff183a/rbd_data.10b16b8b4567.0000000000002ac1/head//3
2014-06-12 14:51:35.798498 7f9d60bd2700 10 getattrs 3.3a_head/c0ff183a/rbd_data.10b16b8b4567.0000000000002ac1/head//3 = 0
2014-06-12 14:51:35.798507 7f9d60bd2700 10 stat 3.3a_head/b342283a/rbd_data.10986b8b4567.00000000000001dd/head//3
2014-06-12 14:51:35.798593 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid b342283a/rbd_data.10986b8b4567.00000000000001dd/head//3
2014-06-12 14:51:35.849223 7f9d60bd2700 10 getattrs 3.3a_head/b342283a/rbd_data.10986b8b4567.00000000000001dd/head//3 = 0
2014-06-12 14:51:35.849241 7f9d60bd2700 10 stat 3.3a_head/49b2283a/rbd_data.10946b8b4567.0000000000001f1c/head//3
2014-06-12 14:51:35.849354 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 49b2283a/rbd_data.10946b8b4567.0000000000001f1c/head//3
2014-06-12 14:51:35.880403 7f9d60bd2700 10 getattrs 3.3a_head/49b2283a/rbd_data.10946b8b4567.0000000000001f1c/head//3 = 0
2014-06-12 14:51:35.880421 7f9d60bd2700 10 stat 3.3a_head/f433283a/rbd_data.10b06b8b4567.0000000000003315/head//3
2014-06-12 14:51:35.880613 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid f433283a/rbd_data.10b06b8b4567.0000000000003315/head//3
2014-06-12 14:51:35.895980 7f9d60bd2700 10 getattrs 3.3a_head/f433283a/rbd_data.10b06b8b4567.0000000000003315/head//3 = 0
2014-06-12 14:51:35.895991 7f9d60bd2700 10 stat 3.3a_head/7544283a/rbd_data.10906b8b4567.000000000000173f/head//3
2014-06-12 14:51:35.896112 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 7544283a/rbd_data.10906b8b4567.000000000000173f/head//3
2014-06-12 14:51:35.925345 7f9d60bd2700 10 getattrs 3.3a_head/7544283a/rbd_data.10906b8b4567.000000000000173f/head//3 = 0
2014-06-12 14:51:35.925376 7f9d60bd2700 10 stat 3.3a_head/9964283a/rbd_data.109e6b8b4567.00000000000003a4/head//3
2014-06-12 14:51:35.925566 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 9964283a/rbd_data.109e6b8b4567.00000000000003a4/head//3
2014-06-12 14:51:35.976868 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:35.990829 7f9d60bd2700 10 getattrs 3.3a_head/9964283a/rbd_data.109e6b8b4567.00000000000003a4/head//3 = 0
2014-06-12 14:51:35.990846 7f9d60bd2700 10 stat 3.3a_head/6666283a/rbd_data.10976b8b4567.00000000000028c8/head//3
2014-06-12 14:51:35.990965 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 6666283a/rbd_data.10976b8b4567.00000000000028c8/head//3
2014-06-12 14:51:35.994418 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:36.027765 7f9d60bd2700 10 getattrs 3.3a_head/6666283a/rbd_data.10976b8b4567.00000000000028c8/head//3 = 0
2014-06-12 14:51:36.027778 7f9d60bd2700 10 stat 3.3a_head/fec7283a/rbd_data.10946b8b4567.000000000000262b/head//3
2014-06-12 14:51:36.027900 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid fec7283a/rbd_data.10946b8b4567.000000000000262b/head//3
2014-06-12 14:51:36.062125 7f9d60bd2700 10 getattrs 3.3a_head/fec7283a/rbd_data.10946b8b4567.000000000000262b/head//3 = 0
2014-06-12 14:51:36.062136 7f9d60bd2700 10 stat 3.3a_head/ccc8283a/rbd_data.10996b8b4567.00000000000001f5/head//3
2014-06-12 14:51:36.062258 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ccc8283a/rbd_data.10996b8b4567.00000000000001f5/head//3
2014-06-12 14:51:36.064565 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:36.069087 7f9d44101700  5 -- op tracker -- , seq: 748588, time: 2014-06-12 14:51:36.069011, event: header_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.069099 7f9d44101700  5 -- op tracker -- , seq: 748588, time: 2014-06-12 14:51:36.069013, event: throttled, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.069107 7f9d44101700  5 -- op tracker -- , seq: 748588, time: 2014-06-12 14:51:36.069056, event: all_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.069114 7f9d44101700  5 -- op tracker -- , seq: 748588, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.069118 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:36.069124 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:36.069132 7f9d44101700 15 osd.2 173 enqueue_op 0x25e42a00 prio 127 cost 10717 latency 0.000121 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:36.069199 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:36.097354 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:36.097412 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:36.116039 7f9d60bd2700 10 getattrs 3.3a_head/ccc8283a/rbd_data.10996b8b4567.00000000000001f5/head//3 = 0
2014-06-12 14:51:36.116057 7f9d60bd2700 10 stat 3.3a_head/721d283a/rbd_data.108f6b8b4567.0000000000001b20/head//3
2014-06-12 14:51:36.116217 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 721d283a/rbd_data.108f6b8b4567.0000000000001b20/head//3
2014-06-12 14:51:36.145467 7f9d60bd2700 10 getattrs 3.3a_head/721d283a/rbd_data.108f6b8b4567.0000000000001b20/head//3 = 0
2014-06-12 14:51:36.145474 7f9d60bd2700 10 stat 3.3a_head/d551383a/rbd_data.109a6b8b4567.00000000000000d7/head//3
2014-06-12 14:51:36.145568 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid d551383a/rbd_data.109a6b8b4567.00000000000000d7/head//3
2014-06-12 14:51:36.167380 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:36.193297 7f9d60bd2700 10 getattrs 3.3a_head/d551383a/rbd_data.109a6b8b4567.00000000000000d7/head//3 = 0
2014-06-12 14:51:36.193323 7f9d60bd2700 10 collection_getattrs 3.3a_head
2014-06-12 14:51:36.193402 7f9d60bd2700 10 collection_getattrs 3.3a_head = 0
2014-06-12 14:51:36.193409 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:36.193461 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:36.193473 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:36.193487 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:36.193491 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:36.193523 7f9d633d7700 10 osd.2 173 dequeue_op 0x25e42a00 prio 127 cost 10717 latency 0.124511 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]
2014-06-12 14:51:36.193583 7f9d633d7700  5 -- op tracker -- , seq: 748588, time: 2014-06-12 14:51:36.193575, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.193604 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] handle_message: 0x25e42a00
2014-06-12 14:51:36.193663 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:36.193682 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:36.193700 7f9d633d7700  5 -- op tracker -- , seq: 748588, time: 2014-06-12 14:51:36.193695, event: started, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.193732 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:36.193839 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] map version is 173'2964
2014-06-12 14:51:36.193860 7f9d633d7700 10 osd.2 173 dequeue_op 0x25e42a00 finish
2014-06-12 14:51:36.193872 7f9d633d7700  5 -- op tracker -- , seq: 748588, time: 2014-06-12 14:51:36.193866, event: done, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.193896 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:36.193901 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:36.193899 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:36.193912 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:36.193940 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:36.193950 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:36.193989 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:36.194000 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:36.194011 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:36.194043 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b14083a/rbd_data.108d6b8b4567.0000000000003892/head//3, auth == maps.end()
2014-06-12 14:51:36.194073 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b14083a/rbd_data.108d6b8b4567.0000000000003892/head//3
2014-06-12 14:51:36.194096 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj eb54083a/rbd_data.109f6b8b4567.00000000000000cf/head//3, auth == maps.end()
2014-06-12 14:51:36.194111 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj eb54083a/rbd_data.109f6b8b4567.00000000000000cf/head//3
2014-06-12 14:51:36.194126 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8ba6083a/rbd_data.10b26b8b4567.0000000000002807/head//3, auth == maps.end()
2014-06-12 14:51:36.194140 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8ba6083a/rbd_data.10b26b8b4567.0000000000002807/head//3
2014-06-12 14:51:36.194154 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 12f6083a/rbd_data.108d6b8b4567.0000000000002844/head//3, auth == maps.end()
2014-06-12 14:51:36.194168 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 12f6083a/rbd_data.108d6b8b4567.0000000000002844/head//3
2014-06-12 14:51:36.194182 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4997083a/rbd_data.109b6b8b4567.0000000000001d04/head//3, auth == maps.end()
2014-06-12 14:51:36.194195 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4997083a/rbd_data.109b6b8b4567.0000000000001d04/head//3
2014-06-12 14:51:36.194212 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8d7b083a/rbd_data.10936b8b4567.0000000000000835/head//3, auth == maps.end()
2014-06-12 14:51:36.194229 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8d7b083a/rbd_data.10936b8b4567.0000000000000835/head//3
2014-06-12 14:51:36.194244 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e1be083a/rbd_data.10976b8b4567.0000000000001285/head//3, auth == maps.end()
2014-06-12 14:51:36.194259 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e1be083a/rbd_data.10976b8b4567.0000000000001285/head//3
2014-06-12 14:51:36.194276 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ef00183a/rbd_data.10b26b8b4567.0000000000001d9c/head//3, auth == maps.end()
2014-06-12 14:51:36.194290 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ef00183a/rbd_data.10b26b8b4567.0000000000001d9c/head//3
2014-06-12 14:51:36.194307 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 76e2183a/rbd_data.10b36b8b4567.0000000000001bcf/head//3, auth == maps.end()
2014-06-12 14:51:36.194321 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 76e2183a/rbd_data.10b36b8b4567.0000000000001bcf/head//3
2014-06-12 14:51:36.194337 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b803183a/rbd_data.10986b8b4567.0000000000001df7/head//3, auth == maps.end()
2014-06-12 14:51:36.194351 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b803183a/rbd_data.10986b8b4567.0000000000001df7/head//3
2014-06-12 14:51:36.194366 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cf86183a/rbd_data.10a26b8b4567.000000000000334a/head//3, auth == maps.end()
2014-06-12 14:51:36.194381 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj cf86183a/rbd_data.10a26b8b4567.000000000000334a/head//3
2014-06-12 14:51:36.194396 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ab96183a/rbd_data.10a56b8b4567.0000000000002daf/head//3, auth == maps.end()
2014-06-12 14:51:36.194412 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ab96183a/rbd_data.10a56b8b4567.0000000000002daf/head//3
2014-06-12 14:51:36.194427 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f46e183a/rbd_data.10a66b8b4567.0000000000001cdc/head//3, auth == maps.end()
2014-06-12 14:51:36.194442 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f46e183a/rbd_data.10a66b8b4567.0000000000001cdc/head//3
2014-06-12 14:51:36.194457 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7fe183a/rbd_data.10a76b8b4567.0000000000003ab9/head//3, auth == maps.end()
2014-06-12 14:51:36.194470 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7fe183a/rbd_data.10a76b8b4567.0000000000003ab9/head//3
2014-06-12 14:51:36.194485 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c0ff183a/rbd_data.10b16b8b4567.0000000000002ac1/head//3, auth == maps.end()
2014-06-12 14:51:36.194499 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c0ff183a/rbd_data.10b16b8b4567.0000000000002ac1/head//3
2014-06-12 14:51:36.194513 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b342283a/rbd_data.10986b8b4567.00000000000001dd/head//3, auth == maps.end()
2014-06-12 14:51:36.194526 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b342283a/rbd_data.10986b8b4567.00000000000001dd/head//3
2014-06-12 14:51:36.194540 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 49b2283a/rbd_data.10946b8b4567.0000000000001f1c/head//3, auth == maps.end()
2014-06-12 14:51:36.194561 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 49b2283a/rbd_data.10946b8b4567.0000000000001f1c/head//3
2014-06-12 14:51:36.194578 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f433283a/rbd_data.10b06b8b4567.0000000000003315/head//3, auth == maps.end()
2014-06-12 14:51:36.194595 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f433283a/rbd_data.10b06b8b4567.0000000000003315/head//3
2014-06-12 14:51:36.194613 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7544283a/rbd_data.10906b8b4567.000000000000173f/head//3, auth == maps.end()
2014-06-12 14:51:36.194628 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7544283a/rbd_data.10906b8b4567.000000000000173f/head//3
2014-06-12 14:51:36.194644 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9964283a/rbd_data.109e6b8b4567.00000000000003a4/head//3, auth == maps.end()
2014-06-12 14:51:36.194659 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9964283a/rbd_data.109e6b8b4567.00000000000003a4/head//3
2014-06-12 14:51:36.194674 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6666283a/rbd_data.10976b8b4567.00000000000028c8/head//3, auth == maps.end()
2014-06-12 14:51:36.194690 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6666283a/rbd_data.10976b8b4567.00000000000028c8/head//3
2014-06-12 14:51:36.194706 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fec7283a/rbd_data.10946b8b4567.000000000000262b/head//3, auth == maps.end()
2014-06-12 14:51:36.194722 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj fec7283a/rbd_data.10946b8b4567.000000000000262b/head//3
2014-06-12 14:51:36.194736 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ccc8283a/rbd_data.10996b8b4567.00000000000001f5/head//3, auth == maps.end()
2014-06-12 14:51:36.194752 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ccc8283a/rbd_data.10996b8b4567.00000000000001f5/head//3
2014-06-12 14:51:36.194769 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 721d283a/rbd_data.108f6b8b4567.0000000000001b20/head//3, auth == maps.end()
2014-06-12 14:51:36.194785 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 721d283a/rbd_data.108f6b8b4567.0000000000001b20/head//3
2014-06-12 14:51:36.194802 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d551383a/rbd_data.109a6b8b4567.00000000000000d7/head//3, auth == maps.end()
2014-06-12 14:51:36.194818 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d551383a/rbd_data.109a6b8b4567.00000000000000d7/head//3
2014-06-12 14:51:36.194838 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] 
2014-06-12 14:51:36.194851 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub
2014-06-12 14:51:36.194870 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  d551383a/rbd_data.109a6b8b4567.00000000000000d7/head//3 d551383a/rbd_data.109a6b8b4567.00000000000000d7/head//3(173'2467 client.4291.0:117080 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2467)
2014-06-12 14:51:36.194893 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  721d283a/rbd_data.108f6b8b4567.0000000000001b20/head//3 721d283a/rbd_data.108f6b8b4567.0000000000001b20/head//3(173'1251 client.4280.0:57941 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1251)
2014-06-12 14:51:36.194915 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ccc8283a/rbd_data.10996b8b4567.00000000000001f5/head//3 ccc8283a/rbd_data.10996b8b4567.00000000000001f5/head//3(173'2523 client.4290.0:118965 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2523)
2014-06-12 14:51:36.194935 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  fec7283a/rbd_data.10946b8b4567.000000000000262b/head//3 fec7283a/rbd_data.10946b8b4567.000000000000262b/head//3(173'1647 client.4285.0:76571 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1647)
2014-06-12 14:51:36.194956 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  6666283a/rbd_data.10976b8b4567.00000000000028c8/head//3 6666283a/rbd_data.10976b8b4567.00000000000028c8/head//3(173'1913 client.4288.0:80918 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1913)
2014-06-12 14:51:36.194976 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  9964283a/rbd_data.109e6b8b4567.00000000000003a4/head//3 9964283a/rbd_data.109e6b8b4567.00000000000003a4/head//3(173'2548 client.4295.0:122418 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2548)
2014-06-12 14:51:36.194996 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  7544283a/rbd_data.10906b8b4567.000000000000173f/head//3 7544283a/rbd_data.10906b8b4567.000000000000173f/head//3(173'1080 client.4281.0:51004 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1080)
2014-06-12 14:51:36.195016 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  f433283a/rbd_data.10b06b8b4567.0000000000003315/head//3 f433283a/rbd_data.10b06b8b4567.0000000000003315/head//3(173'2159 client.4313.0:103161 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2159)
2014-06-12 14:51:36.195036 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  49b2283a/rbd_data.10946b8b4567.0000000000001f1c/head//3 49b2283a/rbd_data.10946b8b4567.0000000000001f1c/head//3(173'1310 client.4285.0:64354 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1310)
2014-06-12 14:51:36.195057 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  b342283a/rbd_data.10986b8b4567.00000000000001dd/head//3 b342283a/rbd_data.10986b8b4567.00000000000001dd/head//3(173'2517 client.4289.0:117968 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2517)
2014-06-12 14:51:36.195077 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c0ff183a/rbd_data.10b16b8b4567.0000000000002ac1/head//3 c0ff183a/rbd_data.10b16b8b4567.0000000000002ac1/head//3(173'1952 client.4314.0:87162 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1952)
2014-06-12 14:51:36.195097 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  7fe183a/rbd_data.10a76b8b4567.0000000000003ab9/head//3 7fe183a/rbd_data.10a76b8b4567.0000000000003ab9/head//3(173'2356 client.4304.0:115314 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2356)
2014-06-12 14:51:36.195117 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  f46e183a/rbd_data.10a66b8b4567.0000000000001cdc/head//3 f46e183a/rbd_data.10a66b8b4567.0000000000001cdc/head//3(173'1255 client.4303.0:59648 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1255)
2014-06-12 14:51:36.195138 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ab96183a/rbd_data.10a56b8b4567.0000000000002daf/head//3 ab96183a/rbd_data.10a56b8b4567.0000000000002daf/head//3(173'2019 client.4302.0:93311 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2019)
2014-06-12 14:51:36.195158 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  cf86183a/rbd_data.10a26b8b4567.000000000000334a/head//3 cf86183a/rbd_data.10a26b8b4567.000000000000334a/head//3(173'2268 client.4299.0:104346 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2268)
2014-06-12 14:51:36.195179 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  b803183a/rbd_data.10986b8b4567.0000000000001df7/head//3 b803183a/rbd_data.10986b8b4567.0000000000001df7/head//3(173'1305 client.4289.0:62312 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1305)
2014-06-12 14:51:36.195199 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  76e2183a/rbd_data.10b36b8b4567.0000000000001bcf/head//3 76e2183a/rbd_data.10b36b8b4567.0000000000001bcf/head//3(173'1198 client.4316.0:60812 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1198)
2014-06-12 14:51:36.195219 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ef00183a/rbd_data.10b26b8b4567.0000000000001d9c/head//3 ef00183a/rbd_data.10b26b8b4567.0000000000001d9c/head//3(173'1202 client.4315.0:62985 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1202)
2014-06-12 14:51:36.195239 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  e1be083a/rbd_data.10976b8b4567.0000000000001285/head//3 e1be083a/rbd_data.10976b8b4567.0000000000001285/head//3(173'996 client.4288.0:41489 wrlock_by=unknown.0.0:0 dirty s 4194304 uv996)
2014-06-12 14:51:36.195259 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  8d7b083a/rbd_data.10936b8b4567.0000000000000835/head//3 8d7b083a/rbd_data.10936b8b4567.0000000000000835/head//3(173'2757 client.4284.0:128099 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2757)
2014-06-12 14:51:36.195279 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  4997083a/rbd_data.109b6b8b4567.0000000000001d04/head//3 4997083a/rbd_data.109b6b8b4567.0000000000001d04/head//3(173'1357 client.4292.0:59980 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1357)
2014-06-12 14:51:36.195299 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  12f6083a/rbd_data.108d6b8b4567.0000000000002844/head//3 12f6083a/rbd_data.108d6b8b4567.0000000000002844/head//3(173'1714 client.4278.0:82751 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1714)
2014-06-12 14:51:36.195327 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  8ba6083a/rbd_data.10b26b8b4567.0000000000002807/head//3 8ba6083a/rbd_data.10b26b8b4567.0000000000002807/head//3(173'1672 client.4315.0:82034 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1672)
2014-06-12 14:51:36.195363 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  eb54083a/rbd_data.109f6b8b4567.00000000000000cf/head//3 eb54083a/rbd_data.109f6b8b4567.00000000000000cf/head//3(173'2461 client.4296.0:116354 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2461)
2014-06-12 14:51:36.195393 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  b14083a/rbd_data.108d6b8b4567.0000000000003892/head//3 b14083a/rbd_data.108d6b8b4567.0000000000003892/head//3(173'2256 client.4278.0:112477 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2256)
2014-06-12 14:51:36.195410 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:36.195428 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:36.195442 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:36.195445 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:36.195448 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:36.195496 7f9d60bd2700 10 collection_list_partial 3.3a_head start:da51383a//0//-1 is_max:0
2014-06-12 14:51:36.195805 7f9d60bd2700 20 objects: [10d1383a/rbd_data.10a06b8b4567.0000000000003aa8/head//3,4262383a/rbd_data.10ae6b8b4567.000000000000306f/head//3,5e75383a/rbd_data.10a76b8b4567.0000000000002b2d/head//3,60b9383a/rbd_data.109e6b8b4567.0000000000003347/head//3,908b383a/rbd_data.10b26b8b4567.0000000000003b2f/head//3,a9bb383a/rbd_data.10926b8b4567.00000000000026ed/head//3,c24e383a/rbd_data.10aa6b8b4567.0000000000000ba0/head//3,ffbf383a/rbd_data.10a66b8b4567.0000000000003bae/head//3,e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3,dd60483a/rbd_data.10956b8b4567.000000000000188c/head//3,f171483a/rbd_data.10a76b8b4567.0000000000003901/head//3,c23483a/rbd_data.10ae6b8b4567.0000000000001a94/head//3,5644483a/rbd_data.10a46b8b4567.0000000000001a71/head//3,8078483a/rbd_data.10ae6b8b4567.000000000000082b/head//3,5a59483a/rbd_data.10a46b8b4567.0000000000002122/head//3,3efb483a/rbd_data.10a46b8b4567.000000000000366f/head//3,b7c483a/rbd_data.10946b8b4567.0000000000002088/head//3,91a1583a/rbd_data.10926b8b4567.0000000000001286/head//3,95a4583a/rbd_data.108f6b8b4567.0000000000001e1b/head//3,ac2a583a/rbd_data.10a06b8b4567.0000000000002c36/head//3,9b7a583a/rbd_data.10946b8b4567.00000000000038da/head//3,5c4c583a/rbd_data.109f6b8b4567.00000000000028e9/head//3,685d583a/rbd_data.10a56b8b4567.0000000000001919/head//3,d624683a/rbd_data.108d6b8b4567.0000000000000253/head//3,1284683a/rbd_data.10b16b8b4567.0000000000002ad5/head//3]
2014-06-12 14:51:36.196385 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:36.196425 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:36.196449 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:36.196463 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:36.196473 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:36.196482 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:36.196491 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_map_chunk [da51383a//0//-1,9794683a//0//-1)
2014-06-12 14:51:36.196502 7f9d60bd2700 10 collection_list_partial 3.3a_head start:da51383a//0//-1 is_max:0
2014-06-12 14:51:36.197272 7f9d60bd2700 20 objects: [10d1383a/rbd_data.10a06b8b4567.0000000000003aa8/head//3,4262383a/rbd_data.10ae6b8b4567.000000000000306f/head//3,5e75383a/rbd_data.10a76b8b4567.0000000000002b2d/head//3,60b9383a/rbd_data.109e6b8b4567.0000000000003347/head//3,908b383a/rbd_data.10b26b8b4567.0000000000003b2f/head//3,a9bb383a/rbd_data.10926b8b4567.00000000000026ed/head//3,c24e383a/rbd_data.10aa6b8b4567.0000000000000ba0/head//3,ffbf383a/rbd_data.10a66b8b4567.0000000000003bae/head//3,e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3,dd60483a/rbd_data.10956b8b4567.000000000000188c/head//3,f171483a/rbd_data.10a76b8b4567.0000000000003901/head//3,c23483a/rbd_data.10ae6b8b4567.0000000000001a94/head//3,5644483a/rbd_data.10a46b8b4567.0000000000001a71/head//3,8078483a/rbd_data.10ae6b8b4567.000000000000082b/head//3,5a59483a/rbd_data.10a46b8b4567.0000000000002122/head//3,3efb483a/rbd_data.10a46b8b4567.000000000000366f/head//3,b7c483a/rbd_data.10946b8b4567.0000000000002088/head//3,91a1583a/rbd_data.10926b8b4567.0000000000001286/head//3,95a4583a/rbd_data.108f6b8b4567.0000000000001e1b/head//3,ac2a583a/rbd_data.10a06b8b4567.0000000000002c36/head//3,9b7a583a/rbd_data.10946b8b4567.00000000000038da/head//3,5c4c583a/rbd_data.109f6b8b4567.00000000000028e9/head//3,685d583a/rbd_data.10a56b8b4567.0000000000001919/head//3,d624683a/rbd_data.108d6b8b4567.0000000000000253/head//3,1284683a/rbd_data.10b16b8b4567.0000000000002ad5/head//3,9794683a/rbd_data.10b26b8b4567.0000000000001a4b/head//3,63b4683a/rbd_data.10976b8b4567.0000000000001bdf/head//3,2155683a/rbd_data.10a16b8b4567.000000000000239d/head//3,f648683a/rbd_data.10946b8b4567.00000000000031ef/head//3,217d683a/rbd_data.10966b8b4567.00000000000037cb/head//3,6c9d683a/rbd_data.10a16b8b4567.0000000000002092/head//3,6091783a/rbd_data.109d6b8b4567.0000000000002a9c/head//3,413783a/rbd_data.109f6b8b4567.0000000000001670/head//3,ee67783a/rbd_data.10b26b8b4567.0000000000001baf/head//3,139b783a/rbd_data.10996b8b4567.000000000000007c/head//3,b17c783a/rbd_data.10936b8b4567.0000000000002a26/head//3,3c7c783a/rbd_data.108d6b8b4567.000000000000395b/head//3,dd21883a/rbd_data.108e6b8b4567.000000000000225e/head//3,ccb1883a/rbd_data.10af6b8b4567.0000000000002a8a/head//3,2a82883a/rbd_data.10a46b8b4567.000000000000138a/head//3,a164883a/rbd_data.10946b8b4567.0000000000003234/head//3,36b4883a/rbd_data.10906b8b4567.000000000000056c/head//3,495883a/rbd_data.10b26b8b4567.000000000000064e/head//3,5c06883a/rbd_data.10b26b8b4567.000000000000168c/head//3,9356883a/rbd_data.10af6b8b4567.0000000000000345/head//3,a267883a/rbd_data.10b06b8b4567.0000000000002318/head//3,a308883a/rbd_data.10a86b8b4567.000000000000161c/head//3,7858883a/rbd_data.10a36b8b4567.0000000000002a60/head//3,ed9883a/rbd_data.109f6b8b4567.000000000000243e/head//3,d01b883a/rbd_data.10a96b8b4567.0000000000001b4f/head//3,e6fc883a/rbd_data.10b36b8b4567.00000000000017cd/head//3,26ff883a/rbd_data.10b46b8b4567.00000000000013e9/head//3,5341983a/rbd_data.10956b8b4567.0000000000002fa9/head//3,51b1983a/rbd_data.10a26b8b4567.0000000000001c33/head//3,6283983a/rbd_data.10936b8b4567.0000000000001e15/head//3,5944983a/rbd_data.10996b8b4567.00000000000003c8/head//3,3aaa983a/rbd_data.10976b8b4567.0000000000001eb2/head//3,74fa983a/rbd_data.10b06b8b4567.00000000000000ed/head//3,dcb983a/rbd_data.10af6b8b4567.00000000000027cb/head//3,d80d983a/rbd_data.10986b8b4567.00000000000012d9/head//3,ddfe983a/rbd_data.10b16b8b4567.00000000000022c4/head//3,993f983a/rbd_data.10916b8b4567.0000000000002e29/head//3,1f60a83a/rbd_data.10ad6b8b4567.000000000000320d/head//3,9b80a83a/rbd_data.10996b8b4567.00000000000006e8/head//3]
2014-06-12 14:51:36.197386 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:36.197411 7f9d60bd2700 10 stat 3.3a_head/10d1383a/rbd_data.10a06b8b4567.0000000000003aa8/head//3
2014-06-12 14:51:36.197599 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 10d1383a/rbd_data.10a06b8b4567.0000000000003aa8/head//3
2014-06-12 14:51:36.210969 7f9d60bd2700 10 getattrs 3.3a_head/10d1383a/rbd_data.10a06b8b4567.0000000000003aa8/head//3 = 0
2014-06-12 14:51:36.210977 7f9d60bd2700 10 stat 3.3a_head/4262383a/rbd_data.10ae6b8b4567.000000000000306f/head//3
2014-06-12 14:51:36.211072 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 4262383a/rbd_data.10ae6b8b4567.000000000000306f/head//3
2014-06-12 14:51:36.228312 7f9d60bd2700 10 getattrs 3.3a_head/4262383a/rbd_data.10ae6b8b4567.000000000000306f/head//3 = 0
2014-06-12 14:51:36.228319 7f9d60bd2700 10 stat 3.3a_head/5e75383a/rbd_data.10a76b8b4567.0000000000002b2d/head//3
2014-06-12 14:51:36.228402 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5e75383a/rbd_data.10a76b8b4567.0000000000002b2d/head//3
2014-06-12 14:51:36.242059 7f9d60bd2700 10 getattrs 3.3a_head/5e75383a/rbd_data.10a76b8b4567.0000000000002b2d/head//3 = 0
2014-06-12 14:51:36.242066 7f9d60bd2700 10 stat 3.3a_head/60b9383a/rbd_data.109e6b8b4567.0000000000003347/head//3
2014-06-12 14:51:36.242162 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 60b9383a/rbd_data.109e6b8b4567.0000000000003347/head//3
2014-06-12 14:51:36.259652 7f9d60bd2700 10 getattrs 3.3a_head/60b9383a/rbd_data.109e6b8b4567.0000000000003347/head//3 = 0
2014-06-12 14:51:36.259659 7f9d60bd2700 10 stat 3.3a_head/908b383a/rbd_data.10b26b8b4567.0000000000003b2f/head//3
2014-06-12 14:51:36.259738 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 908b383a/rbd_data.10b26b8b4567.0000000000003b2f/head//3
2014-06-12 14:51:36.269854 7f9d60bd2700 10 getattrs 3.3a_head/908b383a/rbd_data.10b26b8b4567.0000000000003b2f/head//3 = 0
2014-06-12 14:51:36.269861 7f9d60bd2700 10 stat 3.3a_head/a9bb383a/rbd_data.10926b8b4567.00000000000026ed/head//3
2014-06-12 14:51:36.269953 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a9bb383a/rbd_data.10926b8b4567.00000000000026ed/head//3
2014-06-12 14:51:36.299636 7f9d60bd2700 10 getattrs 3.3a_head/a9bb383a/rbd_data.10926b8b4567.00000000000026ed/head//3 = 0
2014-06-12 14:51:36.299644 7f9d60bd2700 10 stat 3.3a_head/c24e383a/rbd_data.10aa6b8b4567.0000000000000ba0/head//3
2014-06-12 14:51:36.299741 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c24e383a/rbd_data.10aa6b8b4567.0000000000000ba0/head//3
2014-06-12 14:51:36.312948 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:36.313008 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:36.316809 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:36.326254 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:36.343319 7f9d60bd2700 10 getattrs 3.3a_head/c24e383a/rbd_data.10aa6b8b4567.0000000000000ba0/head//3 = 0
2014-06-12 14:51:36.343334 7f9d60bd2700 10 stat 3.3a_head/ffbf383a/rbd_data.10a66b8b4567.0000000000003bae/head//3
2014-06-12 14:51:36.343469 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ffbf383a/rbd_data.10a66b8b4567.0000000000003bae/head//3
2014-06-12 14:51:36.365184 7f9d60bd2700 10 getattrs 3.3a_head/ffbf383a/rbd_data.10a66b8b4567.0000000000003bae/head//3 = 0
2014-06-12 14:51:36.365200 7f9d60bd2700 10 stat 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:51:36.365358 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:51:36.366179 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:36.366388 7f9d60bd2700 10 getattrs 3.3a_head/e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 = 0
2014-06-12 14:51:36.366395 7f9d60bd2700 10 stat 3.3a_head/dd60483a/rbd_data.10956b8b4567.000000000000188c/head//3
2014-06-12 14:51:36.366484 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid dd60483a/rbd_data.10956b8b4567.000000000000188c/head//3
2014-06-12 14:51:36.366588 7f9d60bd2700 10 getattrs 3.3a_head/dd60483a/rbd_data.10956b8b4567.000000000000188c/head//3 = 0
2014-06-12 14:51:36.366593 7f9d60bd2700 10 stat 3.3a_head/f171483a/rbd_data.10a76b8b4567.0000000000003901/head//3
2014-06-12 14:51:36.366665 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid f171483a/rbd_data.10a76b8b4567.0000000000003901/head//3
2014-06-12 14:51:36.371066 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:36.381842 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:36.382789 7f9d60bd2700 10 getattrs 3.3a_head/f171483a/rbd_data.10a76b8b4567.0000000000003901/head//3 = 0
2014-06-12 14:51:36.382797 7f9d60bd2700 10 stat 3.3a_head/c23483a/rbd_data.10ae6b8b4567.0000000000001a94/head//3
2014-06-12 14:51:36.382893 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c23483a/rbd_data.10ae6b8b4567.0000000000001a94/head//3
2014-06-12 14:51:36.395646 7f9d60bd2700 10 getattrs 3.3a_head/c23483a/rbd_data.10ae6b8b4567.0000000000001a94/head//3 = 0
2014-06-12 14:51:36.395652 7f9d60bd2700 10 stat 3.3a_head/5644483a/rbd_data.10a46b8b4567.0000000000001a71/head//3
2014-06-12 14:51:36.395747 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5644483a/rbd_data.10a46b8b4567.0000000000001a71/head//3
2014-06-12 14:51:36.418907 7f9d60bd2700 10 getattrs 3.3a_head/5644483a/rbd_data.10a46b8b4567.0000000000001a71/head//3 = 0
2014-06-12 14:51:36.418915 7f9d60bd2700 10 stat 3.3a_head/8078483a/rbd_data.10ae6b8b4567.000000000000082b/head//3
2014-06-12 14:51:36.419010 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 8078483a/rbd_data.10ae6b8b4567.000000000000082b/head//3
2014-06-12 14:51:36.450246 7f9d60bd2700 10 getattrs 3.3a_head/8078483a/rbd_data.10ae6b8b4567.000000000000082b/head//3 = 0
2014-06-12 14:51:36.450254 7f9d60bd2700 10 stat 3.3a_head/5a59483a/rbd_data.10a46b8b4567.0000000000002122/head//3
2014-06-12 14:51:36.450347 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5a59483a/rbd_data.10a46b8b4567.0000000000002122/head//3
2014-06-12 14:51:36.478723 7f9d60bd2700 10 getattrs 3.3a_head/5a59483a/rbd_data.10a46b8b4567.0000000000002122/head//3 = 0
2014-06-12 14:51:36.478732 7f9d60bd2700 10 stat 3.3a_head/3efb483a/rbd_data.10a46b8b4567.000000000000366f/head//3
2014-06-12 14:51:36.478836 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 3efb483a/rbd_data.10a46b8b4567.000000000000366f/head//3
2014-06-12 14:51:36.507436 7f9d60bd2700 10 getattrs 3.3a_head/3efb483a/rbd_data.10a46b8b4567.000000000000366f/head//3 = 0
2014-06-12 14:51:36.507445 7f9d60bd2700 10 stat 3.3a_head/b7c483a/rbd_data.10946b8b4567.0000000000002088/head//3
2014-06-12 14:51:36.507546 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid b7c483a/rbd_data.10946b8b4567.0000000000002088/head//3
2014-06-12 14:51:36.513055 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:36.518890 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:36.520025 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:36.520072 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.5 >= max 0.5 = no, load too high
2014-06-12 14:51:36.520085 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:51:36.520096 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:36.520089 7f9d84175700 10 osd.2 173 sched_scrub 3.3a high load at 2014-06-11 13:02:32.500954: 92944 < max (604800 seconds)
2014-06-12 14:51:36.520108 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:36.520111 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:36.520113 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:36.522804 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:36.532272 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:36.540262 7f9d60bd2700 10 getattrs 3.3a_head/b7c483a/rbd_data.10946b8b4567.0000000000002088/head//3 = 0
2014-06-12 14:51:36.540272 7f9d60bd2700 10 stat 3.3a_head/91a1583a/rbd_data.10926b8b4567.0000000000001286/head//3
2014-06-12 14:51:36.540383 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 91a1583a/rbd_data.10926b8b4567.0000000000001286/head//3
2014-06-12 14:51:36.547687 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:36.549338 7f9d60bd2700 10 getattrs 3.3a_head/91a1583a/rbd_data.10926b8b4567.0000000000001286/head//3 = 0
2014-06-12 14:51:36.549357 7f9d60bd2700 10 stat 3.3a_head/95a4583a/rbd_data.108f6b8b4567.0000000000001e1b/head//3
2014-06-12 14:51:36.549469 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 95a4583a/rbd_data.108f6b8b4567.0000000000001e1b/head//3
2014-06-12 14:51:36.577054 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:36.577443 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:36.583457 7f9d60bd2700 10 getattrs 3.3a_head/95a4583a/rbd_data.108f6b8b4567.0000000000001e1b/head//3 = 0
2014-06-12 14:51:36.583471 7f9d60bd2700 10 stat 3.3a_head/ac2a583a/rbd_data.10a06b8b4567.0000000000002c36/head//3
2014-06-12 14:51:36.583634 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ac2a583a/rbd_data.10a06b8b4567.0000000000002c36/head//3
2014-06-12 14:51:36.611030 7f9d60bd2700 10 getattrs 3.3a_head/ac2a583a/rbd_data.10a06b8b4567.0000000000002c36/head//3 = 0
2014-06-12 14:51:36.611037 7f9d60bd2700 10 stat 3.3a_head/9b7a583a/rbd_data.10946b8b4567.00000000000038da/head//3
2014-06-12 14:51:36.611126 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 9b7a583a/rbd_data.10946b8b4567.00000000000038da/head//3
2014-06-12 14:51:36.646520 7f9d60bd2700 10 getattrs 3.3a_head/9b7a583a/rbd_data.10946b8b4567.00000000000038da/head//3 = 0
2014-06-12 14:51:36.646535 7f9d60bd2700 10 stat 3.3a_head/5c4c583a/rbd_data.109f6b8b4567.00000000000028e9/head//3
2014-06-12 14:51:36.646633 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5c4c583a/rbd_data.109f6b8b4567.00000000000028e9/head//3
2014-06-12 14:51:36.660350 7f9d60bd2700 10 getattrs 3.3a_head/5c4c583a/rbd_data.109f6b8b4567.00000000000028e9/head//3 = 0
2014-06-12 14:51:36.660359 7f9d60bd2700 10 stat 3.3a_head/685d583a/rbd_data.10a56b8b4567.0000000000001919/head//3
2014-06-12 14:51:36.660453 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 685d583a/rbd_data.10a56b8b4567.0000000000001919/head//3
2014-06-12 14:51:36.661631 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:36.665155 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:36.671194 7f9d60bd2700 10 getattrs 3.3a_head/685d583a/rbd_data.10a56b8b4567.0000000000001919/head//3 = 0
2014-06-12 14:51:36.671205 7f9d60bd2700 10 stat 3.3a_head/d624683a/rbd_data.108d6b8b4567.0000000000000253/head//3
2014-06-12 14:51:36.671302 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid d624683a/rbd_data.108d6b8b4567.0000000000000253/head//3
2014-06-12 14:51:36.677149 7f9d44101700  5 -- op tracker -- , seq: 748589, time: 2014-06-12 14:51:36.677076, event: header_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.677160 7f9d44101700  5 -- op tracker -- , seq: 748589, time: 2014-06-12 14:51:36.677078, event: throttled, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.677166 7f9d44101700  5 -- op tracker -- , seq: 748589, time: 2014-06-12 14:51:36.677118, event: all_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.677171 7f9d44101700  5 -- op tracker -- , seq: 748589, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.677177 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:36.677182 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:36.677188 7f9d44101700 15 osd.2 173 enqueue_op 0x25e42c00 prio 127 cost 10717 latency 0.000112 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:36.677252 7f9d6b3e7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:36.700512 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:36.712759 7f9d60bd2700 10 getattrs 3.3a_head/d624683a/rbd_data.108d6b8b4567.0000000000000253/head//3 = 0
2014-06-12 14:51:36.712776 7f9d60bd2700 10 stat 3.3a_head/1284683a/rbd_data.10b16b8b4567.0000000000002ad5/head//3
2014-06-12 14:51:36.712934 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 1284683a/rbd_data.10b16b8b4567.0000000000002ad5/head//3
2014-06-12 14:51:36.725779 7f9d60bd2700 10 getattrs 3.3a_head/1284683a/rbd_data.10b16b8b4567.0000000000002ad5/head//3 = 0
2014-06-12 14:51:36.725787 7f9d60bd2700 10 collection_getattrs 3.3a_head
2014-06-12 14:51:36.725842 7f9d60bd2700 10 collection_getattrs 3.3a_head = 0
2014-06-12 14:51:36.725846 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:36.725883 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:36.725889 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:36.725907 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:36.725909 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:36.725941 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x25e42c00 prio 127 cost 10717 latency 0.048865 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]
2014-06-12 14:51:36.725990 7f9d6b3e7700  5 -- op tracker -- , seq: 748589, time: 2014-06-12 14:51:36.725982, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.726039 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] handle_message: 0x25e42c00
2014-06-12 14:51:36.726059 7f9d6b3e7700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:36.726077 7f9d6b3e7700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:36.726095 7f9d6b3e7700  5 -- op tracker -- , seq: 748589, time: 2014-06-12 14:51:36.726089, event: started, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.726102 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:36.726206 7f9d6b3e7700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] map version is 173'2964
2014-06-12 14:51:36.726227 7f9d6b3e7700 10 osd.2 173 dequeue_op 0x25e42c00 finish
2014-06-12 14:51:36.726238 7f9d6b3e7700  5 -- op tracker -- , seq: 748589, time: 2014-06-12 14:51:36.726233, event: done, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:36.726260 7f9d6b3e7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:36.726266 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:36.726266 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:36.726279 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:36.726304 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:36.726318 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:36.726359 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:36.726365 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:36.726371 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:36.726387 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 10d1383a/rbd_data.10a06b8b4567.0000000000003aa8/head//3, auth == maps.end()
2014-06-12 14:51:36.726408 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 10d1383a/rbd_data.10a06b8b4567.0000000000003aa8/head//3
2014-06-12 14:51:36.726427 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4262383a/rbd_data.10ae6b8b4567.000000000000306f/head//3, auth == maps.end()
2014-06-12 14:51:36.726435 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 4262383a/rbd_data.10ae6b8b4567.000000000000306f/head//3
2014-06-12 14:51:36.726442 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5e75383a/rbd_data.10a76b8b4567.0000000000002b2d/head//3, auth == maps.end()
2014-06-12 14:51:36.726450 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5e75383a/rbd_data.10a76b8b4567.0000000000002b2d/head//3
2014-06-12 14:51:36.726457 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 60b9383a/rbd_data.109e6b8b4567.0000000000003347/head//3, auth == maps.end()
2014-06-12 14:51:36.726464 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 60b9383a/rbd_data.109e6b8b4567.0000000000003347/head//3
2014-06-12 14:51:36.726472 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 908b383a/rbd_data.10b26b8b4567.0000000000003b2f/head//3, auth == maps.end()
2014-06-12 14:51:36.726479 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 908b383a/rbd_data.10b26b8b4567.0000000000003b2f/head//3
2014-06-12 14:51:36.726486 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a9bb383a/rbd_data.10926b8b4567.00000000000026ed/head//3, auth == maps.end()
2014-06-12 14:51:36.726493 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a9bb383a/rbd_data.10926b8b4567.00000000000026ed/head//3
2014-06-12 14:51:36.726500 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c24e383a/rbd_data.10aa6b8b4567.0000000000000ba0/head//3, auth == maps.end()
2014-06-12 14:51:36.726507 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c24e383a/rbd_data.10aa6b8b4567.0000000000000ba0/head//3
2014-06-12 14:51:36.726514 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ffbf383a/rbd_data.10a66b8b4567.0000000000003bae/head//3, auth == maps.end()
2014-06-12 14:51:36.726521 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ffbf383a/rbd_data.10a66b8b4567.0000000000003bae/head//3
2014-06-12 14:51:36.726528 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3, auth == maps.end()
2014-06-12 14:51:36.726535 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3
2014-06-12 14:51:36.726542 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dd60483a/rbd_data.10956b8b4567.000000000000188c/head//3, auth == maps.end()
2014-06-12 14:51:36.726551 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj dd60483a/rbd_data.10956b8b4567.000000000000188c/head//3
2014-06-12 14:51:36.726561 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f171483a/rbd_data.10a76b8b4567.0000000000003901/head//3, auth == maps.end()
2014-06-12 14:51:36.726569 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f171483a/rbd_data.10a76b8b4567.0000000000003901/head//3
2014-06-12 14:51:36.726577 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c23483a/rbd_data.10ae6b8b4567.0000000000001a94/head//3, auth == maps.end()
2014-06-12 14:51:36.726585 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c23483a/rbd_data.10ae6b8b4567.0000000000001a94/head//3
2014-06-12 14:51:36.726592 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5644483a/rbd_data.10a46b8b4567.0000000000001a71/head//3, auth == maps.end()
2014-06-12 14:51:36.726601 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5644483a/rbd_data.10a46b8b4567.0000000000001a71/head//3
2014-06-12 14:51:36.726609 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8078483a/rbd_data.10ae6b8b4567.000000000000082b/head//3, auth == maps.end()
2014-06-12 14:51:36.726617 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 8078483a/rbd_data.10ae6b8b4567.000000000000082b/head//3
2014-06-12 14:51:36.726627 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5a59483a/rbd_data.10a46b8b4567.0000000000002122/head//3, auth == maps.end()
2014-06-12 14:51:36.726634 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5a59483a/rbd_data.10a46b8b4567.0000000000002122/head//3
2014-06-12 14:51:36.726642 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3efb483a/rbd_data.10a46b8b4567.000000000000366f/head//3, auth == maps.end()
2014-06-12 14:51:36.726649 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3efb483a/rbd_data.10a46b8b4567.000000000000366f/head//3
2014-06-12 14:51:36.726657 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b7c483a/rbd_data.10946b8b4567.0000000000002088/head//3, auth == maps.end()
2014-06-12 14:51:36.726665 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b7c483a/rbd_data.10946b8b4567.0000000000002088/head//3
2014-06-12 14:51:36.726672 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 91a1583a/rbd_data.10926b8b4567.0000000000001286/head//3, auth == maps.end()
2014-06-12 14:51:36.726679 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 91a1583a/rbd_data.10926b8b4567.0000000000001286/head//3
2014-06-12 14:51:36.726686 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 95a4583a/rbd_data.108f6b8b4567.0000000000001e1b/head//3, auth == maps.end()
2014-06-12 14:51:36.726693 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 95a4583a/rbd_data.108f6b8b4567.0000000000001e1b/head//3
2014-06-12 14:51:36.726700 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ac2a583a/rbd_data.10a06b8b4567.0000000000002c36/head//3, auth == maps.end()
2014-06-12 14:51:36.726707 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ac2a583a/rbd_data.10a06b8b4567.0000000000002c36/head//3
2014-06-12 14:51:36.726714 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9b7a583a/rbd_data.10946b8b4567.00000000000038da/head//3, auth == maps.end()
2014-06-12 14:51:36.726721 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9b7a583a/rbd_data.10946b8b4567.00000000000038da/head//3
2014-06-12 14:51:36.726728 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5c4c583a/rbd_data.109f6b8b4567.00000000000028e9/head//3, auth == maps.end()
2014-06-12 14:51:36.726735 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5c4c583a/rbd_data.109f6b8b4567.00000000000028e9/head//3
2014-06-12 14:51:36.726742 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 685d583a/rbd_data.10a56b8b4567.0000000000001919/head//3, auth == maps.end()
2014-06-12 14:51:36.726750 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 685d583a/rbd_data.10a56b8b4567.0000000000001919/head//3
2014-06-12 14:51:36.726758 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d624683a/rbd_data.108d6b8b4567.0000000000000253/head//3, auth == maps.end()
2014-06-12 14:51:36.726767 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d624683a/rbd_data.108d6b8b4567.0000000000000253/head//3
2014-06-12 14:51:36.726774 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1284683a/rbd_data.10b16b8b4567.0000000000002ad5/head//3, auth == maps.end()
2014-06-12 14:51:36.726781 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1284683a/rbd_data.10b16b8b4567.0000000000002ad5/head//3
2014-06-12 14:51:36.726789 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] 
2014-06-12 14:51:36.726795 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub
2014-06-12 14:51:36.726805 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  1284683a/rbd_data.10b16b8b4567.0000000000002ad5/head//3 1284683a/rbd_data.10b16b8b4567.0000000000002ad5/head//3(173'1959 client.4314.0:87304 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1959)
2014-06-12 14:51:36.726817 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  d624683a/rbd_data.108d6b8b4567.0000000000000253/head//3 d624683a/rbd_data.108d6b8b4567.0000000000000253/head//3(173'2508 client.4278.0:121398 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2508)
2014-06-12 14:51:36.726827 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  685d583a/rbd_data.10a56b8b4567.0000000000001919/head//3 685d583a/rbd_data.10a56b8b4567.0000000000001919/head//3(173'1158 client.4302.0:55434 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1158)
2014-06-12 14:51:36.726837 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5c4c583a/rbd_data.109f6b8b4567.00000000000028e9/head//3 5c4c583a/rbd_data.109f6b8b4567.00000000000028e9/head//3(173'1638 client.4296.0:80780 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1638)
2014-06-12 14:51:36.726847 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  9b7a583a/rbd_data.10946b8b4567.00000000000038da/head//3 9b7a583a/rbd_data.10946b8b4567.00000000000038da/head//3(173'2330 client.4285.0:110739 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2330)
2014-06-12 14:51:36.726857 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ac2a583a/rbd_data.10a06b8b4567.0000000000002c36/head//3 ac2a583a/rbd_data.10a06b8b4567.0000000000002c36/head//3(173'1893 client.4297.0:88189 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1893)
2014-06-12 14:51:36.726867 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  95a4583a/rbd_data.108f6b8b4567.0000000000001e1b/head//3 95a4583a/rbd_data.108f6b8b4567.0000000000001e1b/head//3(173'1378 client.4280.0:62929 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1378)
2014-06-12 14:51:36.726877 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  91a1583a/rbd_data.10926b8b4567.0000000000001286/head//3 91a1583a/rbd_data.10926b8b4567.0000000000001286/head//3(173'921 client.4283.0:41721 wrlock_by=unknown.0.0:0 dirty s 4194304 uv921)
2014-06-12 14:51:36.726887 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  b7c483a/rbd_data.10946b8b4567.0000000000002088/head//3 b7c483a/rbd_data.10946b8b4567.0000000000002088/head//3(173'1375 client.4285.0:66762 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1375)
2014-06-12 14:51:36.726896 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  3efb483a/rbd_data.10a46b8b4567.000000000000366f/head//3 3efb483a/rbd_data.10a46b8b4567.000000000000366f/head//3(173'2247 client.4301.0:109087 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2247)
2014-06-12 14:51:36.726906 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5a59483a/rbd_data.10a46b8b4567.0000000000002122/head//3 5a59483a/rbd_data.10a46b8b4567.0000000000002122/head//3(173'1436 client.4301.0:69300 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1436)
2014-06-12 14:51:36.726916 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  8078483a/rbd_data.10ae6b8b4567.000000000000082b/head//3 8078483a/rbd_data.10ae6b8b4567.000000000000082b/head//3(173'2763 client.4311.0:128258 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2763)
2014-06-12 14:51:36.726926 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5644483a/rbd_data.10a46b8b4567.0000000000001a71/head//3 5644483a/rbd_data.10a46b8b4567.0000000000001a71/head//3(173'1192 client.4301.0:57632 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1192)
2014-06-12 14:51:36.726937 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c23483a/rbd_data.10ae6b8b4567.0000000000001a94/head//3 c23483a/rbd_data.10ae6b8b4567.0000000000001a94/head//3(173'1151 client.4311.0:56711 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1151)
2014-06-12 14:51:36.726947 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  f171483a/rbd_data.10a76b8b4567.0000000000003901/head//3 f171483a/rbd_data.10a76b8b4567.0000000000003901/head//3(173'2316 client.4304.0:112272 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2316)
2014-06-12 14:51:36.726957 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  dd60483a/rbd_data.10956b8b4567.000000000000188c/head//3 dd60483a/rbd_data.10956b8b4567.000000000000188c/head//3(173'1124 client.4286.0:53339 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1124)
2014-06-12 14:51:36.726967 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3 e30483a/rbd_data.10b46b8b4567.0000000000001056/head//3(173'2964 client.4317.0:144004 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2964)
2014-06-12 14:51:36.726977 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ffbf383a/rbd_data.10a66b8b4567.0000000000003bae/head//3 ffbf383a/rbd_data.10a66b8b4567.0000000000003bae/head//3(173'2439 client.4303.0:118230 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2439)
2014-06-12 14:51:36.726987 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c24e383a/rbd_data.10aa6b8b4567.0000000000000ba0/head//3 c24e383a/rbd_data.10aa6b8b4567.0000000000000ba0/head//3(173'2882 client.4307.0:139063 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2882)
2014-06-12 14:51:36.726996 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a9bb383a/rbd_data.10926b8b4567.00000000000026ed/head//3 a9bb383a/rbd_data.10926b8b4567.00000000000026ed/head//3(173'1699 client.4283.0:78928 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1699)
2014-06-12 14:51:36.727006 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  908b383a/rbd_data.10b26b8b4567.0000000000003b2f/head//3 908b383a/rbd_data.10b26b8b4567.0000000000003b2f/head//3(173'2321 client.4315.0:116662 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2321)
2014-06-12 14:51:36.727016 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  60b9383a/rbd_data.109e6b8b4567.0000000000003347/head//3 60b9383a/rbd_data.109e6b8b4567.0000000000003347/head//3(173'2070 client.4295.0:102170 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2070)
2014-06-12 14:51:36.727026 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5e75383a/rbd_data.10a76b8b4567.0000000000002b2d/head//3 5e75383a/rbd_data.10a76b8b4567.0000000000002b2d/head//3(173'1904 client.4304.0:87250 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1904)
2014-06-12 14:51:36.727036 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  4262383a/rbd_data.10ae6b8b4567.000000000000306f/head//3 4262383a/rbd_data.10ae6b8b4567.000000000000306f/head//3(173'1965 client.4311.0:94377 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1965)
2014-06-12 14:51:36.727047 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  10d1383a/rbd_data.10a06b8b4567.0000000000003aa8/head//3 10d1383a/rbd_data.10a06b8b4567.0000000000003aa8/head//3(173'2349 client.4297.0:114655 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2349)
2014-06-12 14:51:36.727055 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:36.727063 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:36.727070 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:36.727072 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:36.727073 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:36.727109 7f9d60bd2700 10 collection_list_partial 3.3a_head start:9794683a//0//-1 is_max:0
2014-06-12 14:51:36.727279 7f9d60bd2700 20 objects: [63b4683a/rbd_data.10976b8b4567.0000000000001bdf/head//3,2155683a/rbd_data.10a16b8b4567.000000000000239d/head//3,f648683a/rbd_data.10946b8b4567.00000000000031ef/head//3,217d683a/rbd_data.10966b8b4567.00000000000037cb/head//3,6c9d683a/rbd_data.10a16b8b4567.0000000000002092/head//3,6091783a/rbd_data.109d6b8b4567.0000000000002a9c/head//3,413783a/rbd_data.109f6b8b4567.0000000000001670/head//3,ee67783a/rbd_data.10b26b8b4567.0000000000001baf/head//3,139b783a/rbd_data.10996b8b4567.000000000000007c/head//3,b17c783a/rbd_data.10936b8b4567.0000000000002a26/head//3,3c7c783a/rbd_data.108d6b8b4567.000000000000395b/head//3,dd21883a/rbd_data.108e6b8b4567.000000000000225e/head//3,ccb1883a/rbd_data.10af6b8b4567.0000000000002a8a/head//3,2a82883a/rbd_data.10a46b8b4567.000000000000138a/head//3,a164883a/rbd_data.10946b8b4567.0000000000003234/head//3,36b4883a/rbd_data.10906b8b4567.000000000000056c/head//3,495883a/rbd_data.10b26b8b4567.000000000000064e/head//3,5c06883a/rbd_data.10b26b8b4567.000000000000168c/head//3,9356883a/rbd_data.10af6b8b4567.0000000000000345/head//3,a267883a/rbd_data.10b06b8b4567.0000000000002318/head//3,a308883a/rbd_data.10a86b8b4567.000000000000161c/head//3,7858883a/rbd_data.10a36b8b4567.0000000000002a60/head//3,ed9883a/rbd_data.109f6b8b4567.000000000000243e/head//3,d01b883a/rbd_data.10a96b8b4567.0000000000001b4f/head//3,e6fc883a/rbd_data.10b36b8b4567.00000000000017cd/head//3]
2014-06-12 14:51:36.727753 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:36.727772 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:36.727785 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:36.727791 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:36.727795 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:36.727808 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:36.727812 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_map_chunk [9794683a//0//-1,26ff883a//0//-1)
2014-06-12 14:51:36.727818 7f9d60bd2700 10 collection_list_partial 3.3a_head start:9794683a//0//-1 is_max:0
2014-06-12 14:51:36.728239 7f9d60bd2700 20 objects: [63b4683a/rbd_data.10976b8b4567.0000000000001bdf/head//3,2155683a/rbd_data.10a16b8b4567.000000000000239d/head//3,f648683a/rbd_data.10946b8b4567.00000000000031ef/head//3,217d683a/rbd_data.10966b8b4567.00000000000037cb/head//3,6c9d683a/rbd_data.10a16b8b4567.0000000000002092/head//3,6091783a/rbd_data.109d6b8b4567.0000000000002a9c/head//3,413783a/rbd_data.109f6b8b4567.0000000000001670/head//3,ee67783a/rbd_data.10b26b8b4567.0000000000001baf/head//3,139b783a/rbd_data.10996b8b4567.000000000000007c/head//3,b17c783a/rbd_data.10936b8b4567.0000000000002a26/head//3,3c7c783a/rbd_data.108d6b8b4567.000000000000395b/head//3,dd21883a/rbd_data.108e6b8b4567.000000000000225e/head//3,ccb1883a/rbd_data.10af6b8b4567.0000000000002a8a/head//3,2a82883a/rbd_data.10a46b8b4567.000000000000138a/head//3,a164883a/rbd_data.10946b8b4567.0000000000003234/head//3,36b4883a/rbd_data.10906b8b4567.000000000000056c/head//3,495883a/rbd_data.10b26b8b4567.000000000000064e/head//3,5c06883a/rbd_data.10b26b8b4567.000000000000168c/head//3,9356883a/rbd_data.10af6b8b4567.0000000000000345/head//3,a267883a/rbd_data.10b06b8b4567.0000000000002318/head//3,a308883a/rbd_data.10a86b8b4567.000000000000161c/head//3,7858883a/rbd_data.10a36b8b4567.0000000000002a60/head//3,ed9883a/rbd_data.109f6b8b4567.000000000000243e/head//3,d01b883a/rbd_data.10a96b8b4567.0000000000001b4f/head//3,e6fc883a/rbd_data.10b36b8b4567.00000000000017cd/head//3,26ff883a/rbd_data.10b46b8b4567.00000000000013e9/head//3,5341983a/rbd_data.10956b8b4567.0000000000002fa9/head//3,51b1983a/rbd_data.10a26b8b4567.0000000000001c33/head//3,6283983a/rbd_data.10936b8b4567.0000000000001e15/head//3,5944983a/rbd_data.10996b8b4567.00000000000003c8/head//3,3aaa983a/rbd_data.10976b8b4567.0000000000001eb2/head//3,74fa983a/rbd_data.10b06b8b4567.00000000000000ed/head//3,dcb983a/rbd_data.10af6b8b4567.00000000000027cb/head//3,d80d983a/rbd_data.10986b8b4567.00000000000012d9/head//3,ddfe983a/rbd_data.10b16b8b4567.00000000000022c4/head//3,993f983a/rbd_data.10916b8b4567.0000000000002e29/head//3,1f60a83a/rbd_data.10ad6b8b4567.000000000000320d/head//3,9b80a83a/rbd_data.10996b8b4567.00000000000006e8/head//3,adf0a83a/rbd_data.10b16b8b4567.00000000000024af/head//3,c913a83a/rbd_data.108f6b8b4567.000000000000114c/head//3,78f4a83a/rbd_data.10996b8b4567.0000000000002986/head//3,3396a83a/rbd_data.10a26b8b4567.00000000000028c3/head//3,a0d6a83a/rbd_data.10a26b8b4567.0000000000000340/head//3,fd68a83a/rbd_data.10a76b8b4567.000000000000098c/head//3,461ca83a/rbd_data.109f6b8b4567.0000000000000bc6/head//3,50a1b83a/rbd_data.10a06b8b4567.00000000000003b2/head//3,6de3b83a/rbd_data.10a46b8b4567.0000000000001889/head//3,2e4b83a/rbd_data.10b26b8b4567.0000000000001660/head//3,65f4b83a/rbd_data.10ad6b8b4567.0000000000001c7a/head//3,90a6b83a/rbd_data.10ab6b8b4567.0000000000000545/head//3,c289b83a/rbd_data.10996b8b4567.00000000000021ae/head//3,4daab83a/rbd_data.10946b8b4567.00000000000000db/head//3,23eab83a/rbd_data.10936b8b4567.00000000000015db/head//3,7d3cb83a/rbd_data.10b46b8b4567.0000000000000cc8/head//3,b93fb83a/rbd_data.10936b8b4567.00000000000008cc/head//3,24a1c83a/rbd_data.109b6b8b4567.00000000000027d3/head//3,3c32c83a/rbd_data.10ad6b8b4567.0000000000002293/head//3,ac52c83a/rbd_data.10b06b8b4567.0000000000002bdd/head//3,b962c83a/rbd_data.10996b8b4567.0000000000002769/head//3,55b4c83a/rbd_data.10a56b8b4567.0000000000001bd2/head//3,6a35c83a/rbd_data.10ad6b8b4567.0000000000003928/head//3,43c5c83a/rbd_data.108f6b8b4567.0000000000002fe7/head//3,e439c83a/rbd_data.10b06b8b4567.0000000000003791/head//3,6e6ac83a/rbd_data.10b36b8b4567.00000000000027d6/head//3]
2014-06-12 14:51:36.728296 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:36.728308 7f9d60bd2700 10 stat 3.3a_head/63b4683a/rbd_data.10976b8b4567.0000000000001bdf/head//3
2014-06-12 14:51:36.728399 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 63b4683a/rbd_data.10976b8b4567.0000000000001bdf/head//3
2014-06-12 14:51:36.740093 7f9d60bd2700 10 getattrs 3.3a_head/63b4683a/rbd_data.10976b8b4567.0000000000001bdf/head//3 = 0
2014-06-12 14:51:36.740101 7f9d60bd2700 10 stat 3.3a_head/2155683a/rbd_data.10a16b8b4567.000000000000239d/head//3
2014-06-12 14:51:36.740198 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 2155683a/rbd_data.10a16b8b4567.000000000000239d/head//3
2014-06-12 14:51:36.767927 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:36.770260 7f9d60bd2700 10 getattrs 3.3a_head/2155683a/rbd_data.10a16b8b4567.000000000000239d/head//3 = 0
2014-06-12 14:51:36.770277 7f9d60bd2700 10 stat 3.3a_head/f648683a/rbd_data.10946b8b4567.00000000000031ef/head//3
2014-06-12 14:51:36.770448 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid f648683a/rbd_data.10946b8b4567.00000000000031ef/head//3
2014-06-12 14:51:36.784717 7f9d60bd2700 10 getattrs 3.3a_head/f648683a/rbd_data.10946b8b4567.00000000000031ef/head//3 = 0
2014-06-12 14:51:36.784724 7f9d60bd2700 10 stat 3.3a_head/217d683a/rbd_data.10966b8b4567.00000000000037cb/head//3
2014-06-12 14:51:36.784823 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 217d683a/rbd_data.10966b8b4567.00000000000037cb/head//3
2014-06-12 14:51:36.801853 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:36.802388 7f9d60bd2700 10 getattrs 3.3a_head/217d683a/rbd_data.10966b8b4567.00000000000037cb/head//3 = 0
2014-06-12 14:51:36.802395 7f9d60bd2700 10 stat 3.3a_head/6c9d683a/rbd_data.10a16b8b4567.0000000000002092/head//3
2014-06-12 14:51:36.802492 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 6c9d683a/rbd_data.10a16b8b4567.0000000000002092/head//3
2014-06-12 14:51:36.820396 7f9d60bd2700 10 getattrs 3.3a_head/6c9d683a/rbd_data.10a16b8b4567.0000000000002092/head//3 = 0
2014-06-12 14:51:36.820404 7f9d60bd2700 10 stat 3.3a_head/6091783a/rbd_data.109d6b8b4567.0000000000002a9c/head//3
2014-06-12 14:51:36.820491 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 6091783a/rbd_data.109d6b8b4567.0000000000002a9c/head//3
2014-06-12 14:51:36.833092 7f9d60bd2700 10 getattrs 3.3a_head/6091783a/rbd_data.109d6b8b4567.0000000000002a9c/head//3 = 0
2014-06-12 14:51:36.833103 7f9d60bd2700 10 stat 3.3a_head/413783a/rbd_data.109f6b8b4567.0000000000001670/head//3
2014-06-12 14:51:36.833192 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 413783a/rbd_data.109f6b8b4567.0000000000001670/head//3
2014-06-12 14:51:36.854695 7f9d60bd2700 10 getattrs 3.3a_head/413783a/rbd_data.109f6b8b4567.0000000000001670/head//3 = 0
2014-06-12 14:51:36.854703 7f9d60bd2700 10 stat 3.3a_head/ee67783a/rbd_data.10b26b8b4567.0000000000001baf/head//3
2014-06-12 14:51:36.854795 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ee67783a/rbd_data.10b26b8b4567.0000000000001baf/head//3
2014-06-12 14:51:36.854975 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:36.865308 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:36.868693 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:36.871903 7f9d60bd2700 10 getattrs 3.3a_head/ee67783a/rbd_data.10b26b8b4567.0000000000001baf/head//3 = 0
2014-06-12 14:51:36.871914 7f9d60bd2700 10 stat 3.3a_head/139b783a/rbd_data.10996b8b4567.000000000000007c/head//3
2014-06-12 14:51:36.872015 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 139b783a/rbd_data.10996b8b4567.000000000000007c/head//3
2014-06-12 14:51:36.920322 7f9d60bd2700 10 getattrs 3.3a_head/139b783a/rbd_data.10996b8b4567.000000000000007c/head//3 = 0
2014-06-12 14:51:36.920342 7f9d60bd2700 10 stat 3.3a_head/b17c783a/rbd_data.10936b8b4567.0000000000002a26/head//3
2014-06-12 14:51:36.920466 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid b17c783a/rbd_data.10936b8b4567.0000000000002a26/head//3
2014-06-12 14:51:36.928407 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:36.935849 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:36.936761 7f9d60bd2700 10 getattrs 3.3a_head/b17c783a/rbd_data.10936b8b4567.0000000000002a26/head//3 = 0
2014-06-12 14:51:36.936770 7f9d60bd2700 10 stat 3.3a_head/3c7c783a/rbd_data.108d6b8b4567.000000000000395b/head//3
2014-06-12 14:51:36.936895 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 3c7c783a/rbd_data.108d6b8b4567.000000000000395b/head//3
2014-06-12 14:51:36.948690 7f9d60bd2700 10 getattrs 3.3a_head/3c7c783a/rbd_data.108d6b8b4567.000000000000395b/head//3 = 0
2014-06-12 14:51:36.948699 7f9d60bd2700 10 stat 3.3a_head/dd21883a/rbd_data.108e6b8b4567.000000000000225e/head//3
2014-06-12 14:51:36.948816 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid dd21883a/rbd_data.108e6b8b4567.000000000000225e/head//3
2014-06-12 14:51:36.962235 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:36.965967 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:36.972968 7f9d60bd2700 10 getattrs 3.3a_head/dd21883a/rbd_data.108e6b8b4567.000000000000225e/head//3 = 0
2014-06-12 14:51:36.972976 7f9d60bd2700 10 stat 3.3a_head/ccb1883a/rbd_data.10af6b8b4567.0000000000002a8a/head//3
2014-06-12 14:51:36.973092 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ccb1883a/rbd_data.10af6b8b4567.0000000000002a8a/head//3
2014-06-12 14:51:36.985254 7f9d60bd2700 10 getattrs 3.3a_head/ccb1883a/rbd_data.10af6b8b4567.0000000000002a8a/head//3 = 0
2014-06-12 14:51:36.985270 7f9d60bd2700 10 stat 3.3a_head/2a82883a/rbd_data.10a46b8b4567.000000000000138a/head//3
2014-06-12 14:51:36.985401 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:36.985488 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:36.985570 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 2a82883a/rbd_data.10a46b8b4567.000000000000138a/head//3
2014-06-12 14:51:36.990071 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:36.990089 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:36.990715 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:36.990765 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:36.990879 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:36.990927 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:36.990968 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:36.990993 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:36.991017 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:36.991042 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:36.991067 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:36.991091 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:36.991116 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:36.991140 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:36.991165 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:36.991189 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:36.991214 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:36.991241 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:36.991270 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:36.991295 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:36.991321 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:36.991345 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:36.991371 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:36.991395 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:37.013992 7f9d60bd2700 10 getattrs 3.3a_head/2a82883a/rbd_data.10a46b8b4567.000000000000138a/head//3 = 0
2014-06-12 14:51:37.014005 7f9d60bd2700 10 stat 3.3a_head/a164883a/rbd_data.10946b8b4567.0000000000003234/head//3
2014-06-12 14:51:37.014140 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a164883a/rbd_data.10946b8b4567.0000000000003234/head//3
2014-06-12 14:51:37.021736 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:37.021774 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:37.040577 7f9d60bd2700 10 getattrs 3.3a_head/a164883a/rbd_data.10946b8b4567.0000000000003234/head//3 = 0
2014-06-12 14:51:37.040593 7f9d60bd2700 10 stat 3.3a_head/36b4883a/rbd_data.10906b8b4567.000000000000056c/head//3
2014-06-12 14:51:37.040781 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 36b4883a/rbd_data.10906b8b4567.000000000000056c/head//3
2014-06-12 14:51:37.091893 7f9d60bd2700 10 getattrs 3.3a_head/36b4883a/rbd_data.10906b8b4567.000000000000056c/head//3 = 0
2014-06-12 14:51:37.091907 7f9d60bd2700 10 stat 3.3a_head/495883a/rbd_data.10b26b8b4567.000000000000064e/head//3
2014-06-12 14:51:37.092049 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 495883a/rbd_data.10b26b8b4567.000000000000064e/head//3
2014-06-12 14:51:37.121231 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:37.128868 7f9d60bd2700 10 getattrs 3.3a_head/495883a/rbd_data.10b26b8b4567.000000000000064e/head//3 = 0
2014-06-12 14:51:37.128884 7f9d60bd2700 10 stat 3.3a_head/5c06883a/rbd_data.10b26b8b4567.000000000000168c/head//3
2014-06-12 14:51:37.129058 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5c06883a/rbd_data.10b26b8b4567.000000000000168c/head//3
2014-06-12 14:51:37.132899 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:37.142331 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:37.145703 7f9d60bd2700 10 getattrs 3.3a_head/5c06883a/rbd_data.10b26b8b4567.000000000000168c/head//3 = 0
2014-06-12 14:51:37.145715 7f9d60bd2700 10 stat 3.3a_head/9356883a/rbd_data.10af6b8b4567.0000000000000345/head//3
2014-06-12 14:51:37.145854 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 9356883a/rbd_data.10af6b8b4567.0000000000000345/head//3
2014-06-12 14:51:37.151759 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:37.180221 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:37.190385 7f9d60bd2700 10 getattrs 3.3a_head/9356883a/rbd_data.10af6b8b4567.0000000000000345/head//3 = 0
2014-06-12 14:51:37.190410 7f9d60bd2700 10 stat 3.3a_head/a267883a/rbd_data.10b06b8b4567.0000000000002318/head//3
2014-06-12 14:51:37.190550 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a267883a/rbd_data.10b06b8b4567.0000000000002318/head//3
2014-06-12 14:51:37.213822 7f9d60bd2700 10 getattrs 3.3a_head/a267883a/rbd_data.10b06b8b4567.0000000000002318/head//3 = 0
2014-06-12 14:51:37.213843 7f9d60bd2700 10 stat 3.3a_head/a308883a/rbd_data.10a86b8b4567.000000000000161c/head//3
2014-06-12 14:51:37.213981 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a308883a/rbd_data.10a86b8b4567.000000000000161c/head//3
2014-06-12 14:51:37.218951 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:37.232404 7f9d60bd2700 10 getattrs 3.3a_head/a308883a/rbd_data.10a86b8b4567.000000000000161c/head//3 = 0
2014-06-12 14:51:37.232421 7f9d60bd2700 10 stat 3.3a_head/7858883a/rbd_data.10a36b8b4567.0000000000002a60/head//3
2014-06-12 14:51:37.232617 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 7858883a/rbd_data.10a36b8b4567.0000000000002a60/head//3
2014-06-12 14:51:37.243775 7f9d44101700  5 -- op tracker -- , seq: 748590, time: 2014-06-12 14:51:37.243696, event: header_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:37.243784 7f9d44101700  5 -- op tracker -- , seq: 748590, time: 2014-06-12 14:51:37.243699, event: throttled, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:37.243789 7f9d44101700  5 -- op tracker -- , seq: 748590, time: 2014-06-12 14:51:37.243746, event: all_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:37.243794 7f9d44101700  5 -- op tracker -- , seq: 748590, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:37.243798 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:37.243802 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:37.243808 7f9d44101700 15 osd.2 173 enqueue_op 0x55c3b00 prio 127 cost 10717 latency 0.000112 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:37.243873 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:37.244332 7f9d60bd2700 10 getattrs 3.3a_head/7858883a/rbd_data.10a36b8b4567.0000000000002a60/head//3 = 0
2014-06-12 14:51:37.244340 7f9d60bd2700 10 stat 3.3a_head/ed9883a/rbd_data.109f6b8b4567.000000000000243e/head//3
2014-06-12 14:51:37.244421 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ed9883a/rbd_data.109f6b8b4567.000000000000243e/head//3
2014-06-12 14:51:37.274774 7f9d60bd2700 10 getattrs 3.3a_head/ed9883a/rbd_data.109f6b8b4567.000000000000243e/head//3 = 0
2014-06-12 14:51:37.274785 7f9d60bd2700 10 stat 3.3a_head/d01b883a/rbd_data.10a96b8b4567.0000000000001b4f/head//3
2014-06-12 14:51:37.274882 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid d01b883a/rbd_data.10a96b8b4567.0000000000001b4f/head//3
2014-06-12 14:51:37.298372 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:37.302533 7f9d60bd2700 10 getattrs 3.3a_head/d01b883a/rbd_data.10a96b8b4567.0000000000001b4f/head//3 = 0
2014-06-12 14:51:37.302541 7f9d60bd2700 10 stat 3.3a_head/e6fc883a/rbd_data.10b36b8b4567.00000000000017cd/head//3
2014-06-12 14:51:37.302628 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid e6fc883a/rbd_data.10b36b8b4567.00000000000017cd/head//3
2014-06-12 14:51:37.315925 7f9d60bd2700 10 getattrs 3.3a_head/e6fc883a/rbd_data.10b36b8b4567.00000000000017cd/head//3 = 0
2014-06-12 14:51:37.315937 7f9d60bd2700 10 collection_getattrs 3.3a_head
2014-06-12 14:51:37.315999 7f9d60bd2700 10 collection_getattrs 3.3a_head = 0
2014-06-12 14:51:37.316003 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:37.316038 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:37.316046 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:37.316055 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:37.316058 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:37.316064 7f9d653db700 10 osd.2 173 dequeue_op 0x55c3b00 prio 127 cost 10717 latency 0.072368 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]
2014-06-12 14:51:37.316107 7f9d653db700  5 -- op tracker -- , seq: 748590, time: 2014-06-12 14:51:37.316092, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:37.316128 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] handle_message: 0x55c3b00
2014-06-12 14:51:37.316154 7f9d653db700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:37.316183 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:37.316216 7f9d653db700  5 -- op tracker -- , seq: 748590, time: 2014-06-12 14:51:37.316206, event: started, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:37.316228 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:37.316367 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] map version is 173'2964
2014-06-12 14:51:37.316396 7f9d653db700 10 osd.2 173 dequeue_op 0x55c3b00 finish
2014-06-12 14:51:37.316416 7f9d653db700  5 -- op tracker -- , seq: 748590, time: 2014-06-12 14:51:37.316406, event: done, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:37.316433 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:37.316458 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:37.316466 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:37.316447 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:37.316470 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:37.316488 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:37.316516 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:37.316522 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:37.316529 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:37.316550 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 63b4683a/rbd_data.10976b8b4567.0000000000001bdf/head//3, auth == maps.end()
2014-06-12 14:51:37.316567 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 63b4683a/rbd_data.10976b8b4567.0000000000001bdf/head//3
2014-06-12 14:51:37.316579 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2155683a/rbd_data.10a16b8b4567.000000000000239d/head//3, auth == maps.end()
2014-06-12 14:51:37.316588 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2155683a/rbd_data.10a16b8b4567.000000000000239d/head//3
2014-06-12 14:51:37.316597 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f648683a/rbd_data.10946b8b4567.00000000000031ef/head//3, auth == maps.end()
2014-06-12 14:51:37.316605 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f648683a/rbd_data.10946b8b4567.00000000000031ef/head//3
2014-06-12 14:51:37.316614 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 217d683a/rbd_data.10966b8b4567.00000000000037cb/head//3, auth == maps.end()
2014-06-12 14:51:37.316623 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 217d683a/rbd_data.10966b8b4567.00000000000037cb/head//3
2014-06-12 14:51:37.316632 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6c9d683a/rbd_data.10a16b8b4567.0000000000002092/head//3, auth == maps.end()
2014-06-12 14:51:37.316640 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6c9d683a/rbd_data.10a16b8b4567.0000000000002092/head//3
2014-06-12 14:51:37.316649 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6091783a/rbd_data.109d6b8b4567.0000000000002a9c/head//3, auth == maps.end()
2014-06-12 14:51:37.316657 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6091783a/rbd_data.109d6b8b4567.0000000000002a9c/head//3
2014-06-12 14:51:37.316666 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 413783a/rbd_data.109f6b8b4567.0000000000001670/head//3, auth == maps.end()
2014-06-12 14:51:37.316674 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 413783a/rbd_data.109f6b8b4567.0000000000001670/head//3
2014-06-12 14:51:37.316683 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ee67783a/rbd_data.10b26b8b4567.0000000000001baf/head//3, auth == maps.end()
2014-06-12 14:51:37.316691 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ee67783a/rbd_data.10b26b8b4567.0000000000001baf/head//3
2014-06-12 14:51:37.316700 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 139b783a/rbd_data.10996b8b4567.000000000000007c/head//3, auth == maps.end()
2014-06-12 14:51:37.316709 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 139b783a/rbd_data.10996b8b4567.000000000000007c/head//3
2014-06-12 14:51:37.316718 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b17c783a/rbd_data.10936b8b4567.0000000000002a26/head//3, auth == maps.end()
2014-06-12 14:51:37.316726 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b17c783a/rbd_data.10936b8b4567.0000000000002a26/head//3
2014-06-12 14:51:37.316735 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3c7c783a/rbd_data.108d6b8b4567.000000000000395b/head//3, auth == maps.end()
2014-06-12 14:51:37.316743 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3c7c783a/rbd_data.108d6b8b4567.000000000000395b/head//3
2014-06-12 14:51:37.316758 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dd21883a/rbd_data.108e6b8b4567.000000000000225e/head//3, auth == maps.end()
2014-06-12 14:51:37.316769 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj dd21883a/rbd_data.108e6b8b4567.000000000000225e/head//3
2014-06-12 14:51:37.316781 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ccb1883a/rbd_data.10af6b8b4567.0000000000002a8a/head//3, auth == maps.end()
2014-06-12 14:51:37.316791 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ccb1883a/rbd_data.10af6b8b4567.0000000000002a8a/head//3
2014-06-12 14:51:37.316800 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2a82883a/rbd_data.10a46b8b4567.000000000000138a/head//3, auth == maps.end()
2014-06-12 14:51:37.316810 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2a82883a/rbd_data.10a46b8b4567.000000000000138a/head//3
2014-06-12 14:51:37.316819 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a164883a/rbd_data.10946b8b4567.0000000000003234/head//3, auth == maps.end()
2014-06-12 14:51:37.316828 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a164883a/rbd_data.10946b8b4567.0000000000003234/head//3
2014-06-12 14:51:37.316838 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 36b4883a/rbd_data.10906b8b4567.000000000000056c/head//3, auth == maps.end()
2014-06-12 14:51:37.316847 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 36b4883a/rbd_data.10906b8b4567.000000000000056c/head//3
2014-06-12 14:51:37.316857 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 495883a/rbd_data.10b26b8b4567.000000000000064e/head//3, auth == maps.end()
2014-06-12 14:51:37.316866 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 495883a/rbd_data.10b26b8b4567.000000000000064e/head//3
2014-06-12 14:51:37.316876 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5c06883a/rbd_data.10b26b8b4567.000000000000168c/head//3, auth == maps.end()
2014-06-12 14:51:37.316885 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5c06883a/rbd_data.10b26b8b4567.000000000000168c/head//3
2014-06-12 14:51:37.316895 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9356883a/rbd_data.10af6b8b4567.0000000000000345/head//3, auth == maps.end()
2014-06-12 14:51:37.316906 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9356883a/rbd_data.10af6b8b4567.0000000000000345/head//3
2014-06-12 14:51:37.316916 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a267883a/rbd_data.10b06b8b4567.0000000000002318/head//3, auth == maps.end()
2014-06-12 14:51:37.316926 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a267883a/rbd_data.10b06b8b4567.0000000000002318/head//3
2014-06-12 14:51:37.316935 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a308883a/rbd_data.10a86b8b4567.000000000000161c/head//3, auth == maps.end()
2014-06-12 14:51:37.316945 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a308883a/rbd_data.10a86b8b4567.000000000000161c/head//3
2014-06-12 14:51:37.316956 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7858883a/rbd_data.10a36b8b4567.0000000000002a60/head//3, auth == maps.end()
2014-06-12 14:51:37.316964 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7858883a/rbd_data.10a36b8b4567.0000000000002a60/head//3
2014-06-12 14:51:37.316975 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ed9883a/rbd_data.109f6b8b4567.000000000000243e/head//3, auth == maps.end()
2014-06-12 14:51:37.316986 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ed9883a/rbd_data.109f6b8b4567.000000000000243e/head//3
2014-06-12 14:51:37.316996 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d01b883a/rbd_data.10a96b8b4567.0000000000001b4f/head//3, auth == maps.end()
2014-06-12 14:51:37.317005 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d01b883a/rbd_data.10a96b8b4567.0000000000001b4f/head//3
2014-06-12 14:51:37.317015 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e6fc883a/rbd_data.10b36b8b4567.00000000000017cd/head//3, auth == maps.end()
2014-06-12 14:51:37.317025 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e6fc883a/rbd_data.10b36b8b4567.00000000000017cd/head//3
2014-06-12 14:51:37.317036 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] 
2014-06-12 14:51:37.317043 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub
2014-06-12 14:51:37.317056 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  e6fc883a/rbd_data.10b36b8b4567.00000000000017cd/head//3 e6fc883a/rbd_data.10b36b8b4567.00000000000017cd/head//3(173'1066 client.4316.0:53544 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1066)
2014-06-12 14:51:37.317069 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  d01b883a/rbd_data.10a96b8b4567.0000000000001b4f/head//3 d01b883a/rbd_data.10a96b8b4567.0000000000001b4f/head//3(173'1298 client.4306.0:60451 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1298)
2014-06-12 14:51:37.317080 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ed9883a/rbd_data.109f6b8b4567.000000000000243e/head//3 ed9883a/rbd_data.109f6b8b4567.000000000000243e/head//3(173'1430 client.4296.0:72506 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1430)
2014-06-12 14:51:37.317091 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  7858883a/rbd_data.10a36b8b4567.0000000000002a60/head//3 7858883a/rbd_data.10a36b8b4567.0000000000002a60/head//3(173'1782 client.4300.0:83727 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1782)
2014-06-12 14:51:37.317104 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a308883a/rbd_data.10a86b8b4567.000000000000161c/head//3 a308883a/rbd_data.10a86b8b4567.000000000000161c/head//3(173'1028 client.4305.0:48747 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1028)
2014-06-12 14:51:37.317115 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a267883a/rbd_data.10b06b8b4567.0000000000002318/head//3 a267883a/rbd_data.10b06b8b4567.0000000000002318/head//3(173'1519 client.4313.0:73401 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1519)
2014-06-12 14:51:37.317127 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  9356883a/rbd_data.10af6b8b4567.0000000000000345/head//3 9356883a/rbd_data.10af6b8b4567.0000000000000345/head//3(173'2557 client.4312.0:125247 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2557)
2014-06-12 14:51:37.317139 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5c06883a/rbd_data.10b26b8b4567.000000000000168c/head//3 5c06883a/rbd_data.10b26b8b4567.000000000000168c/head//3(173'1046 client.4315.0:49962 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1046)
2014-06-12 14:51:37.317151 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  495883a/rbd_data.10b26b8b4567.000000000000064e/head//3 495883a/rbd_data.10b26b8b4567.000000000000064e/head//3(173'2685 client.4315.0:127444 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2685)
2014-06-12 14:51:37.317164 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  36b4883a/rbd_data.10906b8b4567.000000000000056c/head//3 36b4883a/rbd_data.10906b8b4567.000000000000056c/head//3(173'2640 client.4281.0:124683 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2640)
2014-06-12 14:51:37.317175 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a164883a/rbd_data.10946b8b4567.0000000000003234/head//3 a164883a/rbd_data.10946b8b4567.0000000000003234/head//3(173'2093 client.4285.0:98912 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2093)
2014-06-12 14:51:37.317187 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  2a82883a/rbd_data.10a46b8b4567.000000000000138a/head//3 2a82883a/rbd_data.10a46b8b4567.000000000000138a/head//3(173'946 client.4301.0:45476 wrlock_by=unknown.0.0:0 dirty s 4194304 uv946)
2014-06-12 14:51:37.317199 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ccb1883a/rbd_data.10af6b8b4567.0000000000002a8a/head//3 ccb1883a/rbd_data.10af6b8b4567.0000000000002a8a/head//3(173'1880 client.4312.0:88352 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1880)
2014-06-12 14:51:37.317212 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  dd21883a/rbd_data.108e6b8b4567.000000000000225e/head//3 dd21883a/rbd_data.108e6b8b4567.000000000000225e/head//3(173'1549 client.4279.0:71666 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1549)
2014-06-12 14:51:37.317224 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  3c7c783a/rbd_data.108d6b8b4567.000000000000395b/head//3 3c7c783a/rbd_data.108d6b8b4567.000000000000395b/head//3(173'2281 client.4278.0:113928 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2281)
2014-06-12 14:51:37.317236 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  b17c783a/rbd_data.10936b8b4567.0000000000002a26/head//3 b17c783a/rbd_data.10936b8b4567.0000000000002a26/head//3(173'1871 client.4284.0:83573 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1871)
2014-06-12 14:51:37.317249 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  139b783a/rbd_data.10996b8b4567.000000000000007c/head//3 139b783a/rbd_data.10996b8b4567.000000000000007c/head//3(173'2443 client.4290.0:117158 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2443)
2014-06-12 14:51:37.317261 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ee67783a/rbd_data.10b26b8b4567.0000000000001baf/head//3 ee67783a/rbd_data.10b26b8b4567.0000000000001baf/head//3(173'1176 client.4315.0:59427 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1176)
2014-06-12 14:51:37.317274 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  413783a/rbd_data.109f6b8b4567.0000000000001670/head//3 413783a/rbd_data.109f6b8b4567.0000000000001670/head//3(173'986 client.4296.0:49104 wrlock_by=unknown.0.0:0 dirty s 4194304 uv986)
2014-06-12 14:51:37.317286 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  6091783a/rbd_data.109d6b8b4567.0000000000002a9c/head//3 6091783a/rbd_data.109d6b8b4567.0000000000002a9c/head//3(173'1805 client.4294.0:85821 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1805)
2014-06-12 14:51:37.317297 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  6c9d683a/rbd_data.10a16b8b4567.0000000000002092/head//3 6c9d683a/rbd_data.10a16b8b4567.0000000000002092/head//3(173'1417 client.4298.0:68877 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1417)
2014-06-12 14:51:37.317310 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  217d683a/rbd_data.10966b8b4567.00000000000037cb/head//3 217d683a/rbd_data.10966b8b4567.00000000000037cb/head//3(173'2223 client.4287.0:107179 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2223)
2014-06-12 14:51:37.317322 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  f648683a/rbd_data.10946b8b4567.00000000000031ef/head//3 f648683a/rbd_data.10946b8b4567.00000000000031ef/head//3(173'2086 client.4285.0:98448 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2086)
2014-06-12 14:51:37.317334 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  2155683a/rbd_data.10a16b8b4567.000000000000239d/head//3 2155683a/rbd_data.10a16b8b4567.000000000000239d/head//3(173'1531 client.4298.0:74259 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1531)
2014-06-12 14:51:37.317346 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  63b4683a/rbd_data.10976b8b4567.0000000000001bdf/head//3 63b4683a/rbd_data.10976b8b4567.0000000000001bdf/head//3(173'1382 client.4288.0:58315 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1382)
2014-06-12 14:51:37.317356 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:37.317367 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:37.317375 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:37.317377 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:37.317379 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:37.317407 7f9d60bd2700 10 collection_list_partial 3.3a_head start:26ff883a//0//-1 is_max:0
2014-06-12 14:51:37.317629 7f9d60bd2700 20 objects: [5341983a/rbd_data.10956b8b4567.0000000000002fa9/head//3,51b1983a/rbd_data.10a26b8b4567.0000000000001c33/head//3,6283983a/rbd_data.10936b8b4567.0000000000001e15/head//3,5944983a/rbd_data.10996b8b4567.00000000000003c8/head//3,3aaa983a/rbd_data.10976b8b4567.0000000000001eb2/head//3,74fa983a/rbd_data.10b06b8b4567.00000000000000ed/head//3,dcb983a/rbd_data.10af6b8b4567.00000000000027cb/head//3,d80d983a/rbd_data.10986b8b4567.00000000000012d9/head//3,ddfe983a/rbd_data.10b16b8b4567.00000000000022c4/head//3,993f983a/rbd_data.10916b8b4567.0000000000002e29/head//3,1f60a83a/rbd_data.10ad6b8b4567.000000000000320d/head//3,9b80a83a/rbd_data.10996b8b4567.00000000000006e8/head//3,adf0a83a/rbd_data.10b16b8b4567.00000000000024af/head//3,c913a83a/rbd_data.108f6b8b4567.000000000000114c/head//3,78f4a83a/rbd_data.10996b8b4567.0000000000002986/head//3,3396a83a/rbd_data.10a26b8b4567.00000000000028c3/head//3,a0d6a83a/rbd_data.10a26b8b4567.0000000000000340/head//3,fd68a83a/rbd_data.10a76b8b4567.000000000000098c/head//3,461ca83a/rbd_data.109f6b8b4567.0000000000000bc6/head//3,50a1b83a/rbd_data.10a06b8b4567.00000000000003b2/head//3,6de3b83a/rbd_data.10a46b8b4567.0000000000001889/head//3,2e4b83a/rbd_data.10b26b8b4567.0000000000001660/head//3,65f4b83a/rbd_data.10ad6b8b4567.0000000000001c7a/head//3,90a6b83a/rbd_data.10ab6b8b4567.0000000000000545/head//3,c289b83a/rbd_data.10996b8b4567.00000000000021ae/head//3]
2014-06-12 14:51:37.318111 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:37.318136 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:37.318153 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:37.318161 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:37.318167 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:37.318173 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:37.318178 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_map_chunk [26ff883a//0//-1,4daab83a//0//-1)
2014-06-12 14:51:37.318185 7f9d60bd2700 10 collection_list_partial 3.3a_head start:26ff883a//0//-1 is_max:0
2014-06-12 14:51:37.318641 7f9d60bd2700 20 objects: [5341983a/rbd_data.10956b8b4567.0000000000002fa9/head//3,51b1983a/rbd_data.10a26b8b4567.0000000000001c33/head//3,6283983a/rbd_data.10936b8b4567.0000000000001e15/head//3,5944983a/rbd_data.10996b8b4567.00000000000003c8/head//3,3aaa983a/rbd_data.10976b8b4567.0000000000001eb2/head//3,74fa983a/rbd_data.10b06b8b4567.00000000000000ed/head//3,dcb983a/rbd_data.10af6b8b4567.00000000000027cb/head//3,d80d983a/rbd_data.10986b8b4567.00000000000012d9/head//3,ddfe983a/rbd_data.10b16b8b4567.00000000000022c4/head//3,993f983a/rbd_data.10916b8b4567.0000000000002e29/head//3,1f60a83a/rbd_data.10ad6b8b4567.000000000000320d/head//3,9b80a83a/rbd_data.10996b8b4567.00000000000006e8/head//3,adf0a83a/rbd_data.10b16b8b4567.00000000000024af/head//3,c913a83a/rbd_data.108f6b8b4567.000000000000114c/head//3,78f4a83a/rbd_data.10996b8b4567.0000000000002986/head//3,3396a83a/rbd_data.10a26b8b4567.00000000000028c3/head//3,a0d6a83a/rbd_data.10a26b8b4567.0000000000000340/head//3,fd68a83a/rbd_data.10a76b8b4567.000000000000098c/head//3,461ca83a/rbd_data.109f6b8b4567.0000000000000bc6/head//3,50a1b83a/rbd_data.10a06b8b4567.00000000000003b2/head//3,6de3b83a/rbd_data.10a46b8b4567.0000000000001889/head//3,2e4b83a/rbd_data.10b26b8b4567.0000000000001660/head//3,65f4b83a/rbd_data.10ad6b8b4567.0000000000001c7a/head//3,90a6b83a/rbd_data.10ab6b8b4567.0000000000000545/head//3,c289b83a/rbd_data.10996b8b4567.00000000000021ae/head//3,4daab83a/rbd_data.10946b8b4567.00000000000000db/head//3,23eab83a/rbd_data.10936b8b4567.00000000000015db/head//3,7d3cb83a/rbd_data.10b46b8b4567.0000000000000cc8/head//3,b93fb83a/rbd_data.10936b8b4567.00000000000008cc/head//3,24a1c83a/rbd_data.109b6b8b4567.00000000000027d3/head//3,3c32c83a/rbd_data.10ad6b8b4567.0000000000002293/head//3,ac52c83a/rbd_data.10b06b8b4567.0000000000002bdd/head//3,b962c83a/rbd_data.10996b8b4567.0000000000002769/head//3,55b4c83a/rbd_data.10a56b8b4567.0000000000001bd2/head//3,6a35c83a/rbd_data.10ad6b8b4567.0000000000003928/head//3,43c5c83a/rbd_data.108f6b8b4567.0000000000002fe7/head//3,e439c83a/rbd_data.10b06b8b4567.0000000000003791/head//3,6e6ac83a/rbd_data.10b36b8b4567.00000000000027d6/head//3,18dac83a/rbd_data.10946b8b4567.0000000000002e4c/head//3,fb3ec83a/rbd_data.10986b8b4567.000000000000325e/head//3,3f21d83a/rbd_data.10a16b8b4567.00000000000039f0/head//3,1d74d83a/rbd_data.108d6b8b4567.0000000000001e54/head//3,dfa4d83a/rbd_data.10a96b8b4567.00000000000005c5/head//3,d6a8d83a/rbd_data.10a26b8b4567.000000000000285e/head//3,9ea8d83a/rbd_data.10996b8b4567.0000000000002723/head//3,825ad83a/rbd_data.10a96b8b4567.0000000000001f72/head//3,c67ad83a/rbd_data.10916b8b4567.0000000000002b77/head//3,a89ed83a/rbd_data.10986b8b4567.0000000000001c1c/head//3,97fd83a/rbd_data.10b36b8b4567.0000000000001bda/head//3,46afd83a/rbd_data.10ae6b8b4567.0000000000000419/head//3,2fdfd83a/rbd_data.10a06b8b4567.00000000000032ec/head//3,3f90e83a/rbd_data.10a56b8b4567.0000000000000339/head//3,f5f1e83a/rbd_data.10936b8b4567.000000000000322f/head//3,3562e83a/rbd_data.10ab6b8b4567.0000000000000b05/head//3,7c78e83a/rbd_data.10b26b8b4567.00000000000024a9/head//3,db1ce83a/rbd_data.10956b8b4567.0000000000002b6d/head//3,ecfce83a/rbd_data.10986b8b4567.0000000000001cab/head//3,6641f83a/rbd_data.10b46b8b4567.0000000000000632/head//3,5071f83a/rbd_data.10a96b8b4567.0000000000002efc/head//3,c3c4f83a/rbd_data.10a36b8b4567.0000000000001672/head//3,d27df83a/rbd_data.10996b8b4567.0000000000000387/head//3]
2014-06-12 14:51:37.318708 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:37.318723 7f9d60bd2700 10 stat 3.3a_head/5341983a/rbd_data.10956b8b4567.0000000000002fa9/head//3
2014-06-12 14:51:37.318840 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5341983a/rbd_data.10956b8b4567.0000000000002fa9/head//3
2014-06-12 14:51:37.341559 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:37.346702 7f9d60bd2700 10 getattrs 3.3a_head/5341983a/rbd_data.10956b8b4567.0000000000002fa9/head//3 = 0
2014-06-12 14:51:37.346710 7f9d60bd2700 10 stat 3.3a_head/51b1983a/rbd_data.10a26b8b4567.0000000000001c33/head//3
2014-06-12 14:51:37.346821 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 51b1983a/rbd_data.10a26b8b4567.0000000000001c33/head//3
2014-06-12 14:51:37.373185 7f9d60bd2700 10 getattrs 3.3a_head/51b1983a/rbd_data.10a26b8b4567.0000000000001c33/head//3 = 0
2014-06-12 14:51:37.373197 7f9d60bd2700 10 stat 3.3a_head/6283983a/rbd_data.10936b8b4567.0000000000001e15/head//3
2014-06-12 14:51:37.373309 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 6283983a/rbd_data.10936b8b4567.0000000000001e15/head//3
2014-06-12 14:51:37.374812 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:37.390365 7f9d60bd2700 10 getattrs 3.3a_head/6283983a/rbd_data.10936b8b4567.0000000000001e15/head//3 = 0
2014-06-12 14:51:37.390375 7f9d60bd2700 10 stat 3.3a_head/5944983a/rbd_data.10996b8b4567.00000000000003c8/head//3
2014-06-12 14:51:37.390485 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5944983a/rbd_data.10996b8b4567.00000000000003c8/head//3
2014-06-12 14:51:37.430072 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:37.430180 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:37.436767 7f9d60bd2700 10 getattrs 3.3a_head/5944983a/rbd_data.10996b8b4567.00000000000003c8/head//3 = 0
2014-06-12 14:51:37.436778 7f9d60bd2700 10 stat 3.3a_head/3aaa983a/rbd_data.10976b8b4567.0000000000001eb2/head//3
2014-06-12 14:51:37.436892 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 3aaa983a/rbd_data.10976b8b4567.0000000000001eb2/head//3
2014-06-12 14:51:37.471055 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:37.471082 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:37.473875 7f9d60bd2700 10 getattrs 3.3a_head/3aaa983a/rbd_data.10976b8b4567.0000000000001eb2/head//3 = 0
2014-06-12 14:51:37.473889 7f9d60bd2700 10 stat 3.3a_head/74fa983a/rbd_data.10b06b8b4567.00000000000000ed/head//3
2014-06-12 14:51:37.473973 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 74fa983a/rbd_data.10b06b8b4567.00000000000000ed/head//3
2014-06-12 14:51:37.513312 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:37.518369 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:37.520132 7f9d60bd2700 10 getattrs 3.3a_head/74fa983a/rbd_data.10b06b8b4567.00000000000000ed/head//3 = 0
2014-06-12 14:51:37.520150 7f9d60bd2700 10 stat 3.3a_head/dcb983a/rbd_data.10af6b8b4567.00000000000027cb/head//3
2014-06-12 14:51:37.520183 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:37.520200 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:37.520203 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:37.520204 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:37.520230 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:37.520296 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid dcb983a/rbd_data.10af6b8b4567.00000000000027cb/head//3
2014-06-12 14:51:37.533110 7f9d60bd2700 10 getattrs 3.3a_head/dcb983a/rbd_data.10af6b8b4567.00000000000027cb/head//3 = 0
2014-06-12 14:51:37.533128 7f9d60bd2700 10 stat 3.3a_head/d80d983a/rbd_data.10986b8b4567.00000000000012d9/head//3
2014-06-12 14:51:37.533254 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid d80d983a/rbd_data.10986b8b4567.00000000000012d9/head//3
2014-06-12 14:51:37.547264 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:37.563669 7f9d60bd2700 10 getattrs 3.3a_head/d80d983a/rbd_data.10986b8b4567.00000000000012d9/head//3 = 0
2014-06-12 14:51:37.563684 7f9d60bd2700 10 stat 3.3a_head/ddfe983a/rbd_data.10b16b8b4567.00000000000022c4/head//3
2014-06-12 14:51:37.563819 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ddfe983a/rbd_data.10b16b8b4567.00000000000022c4/head//3
2014-06-12 14:51:37.594018 7f9d60bd2700 10 getattrs 3.3a_head/ddfe983a/rbd_data.10b16b8b4567.00000000000022c4/head//3 = 0
2014-06-12 14:51:37.594029 7f9d60bd2700 10 stat 3.3a_head/993f983a/rbd_data.10916b8b4567.0000000000002e29/head//3
2014-06-12 14:51:37.594163 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 993f983a/rbd_data.10916b8b4567.0000000000002e29/head//3
2014-06-12 14:51:37.611068 7f9d60bd2700 10 getattrs 3.3a_head/993f983a/rbd_data.10916b8b4567.0000000000002e29/head//3 = 0
2014-06-12 14:51:37.611083 7f9d60bd2700 10 stat 3.3a_head/1f60a83a/rbd_data.10ad6b8b4567.000000000000320d/head//3
2014-06-12 14:51:37.611217 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 1f60a83a/rbd_data.10ad6b8b4567.000000000000320d/head//3
2014-06-12 14:51:37.629338 7f9d60bd2700 10 getattrs 3.3a_head/1f60a83a/rbd_data.10ad6b8b4567.000000000000320d/head//3 = 0
2014-06-12 14:51:37.629358 7f9d60bd2700 10 stat 3.3a_head/9b80a83a/rbd_data.10996b8b4567.00000000000006e8/head//3
2014-06-12 14:51:37.629551 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 9b80a83a/rbd_data.10996b8b4567.00000000000006e8/head//3
2014-06-12 14:51:37.681495 7f9d60bd2700 10 getattrs 3.3a_head/9b80a83a/rbd_data.10996b8b4567.00000000000006e8/head//3 = 0
2014-06-12 14:51:37.681508 7f9d60bd2700 10 stat 3.3a_head/adf0a83a/rbd_data.10b16b8b4567.00000000000024af/head//3
2014-06-12 14:51:37.681618 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid adf0a83a/rbd_data.10b16b8b4567.00000000000024af/head//3
2014-06-12 14:51:37.697237 7f9d60bd2700 10 getattrs 3.3a_head/adf0a83a/rbd_data.10b16b8b4567.00000000000024af/head//3 = 0
2014-06-12 14:51:37.697245 7f9d60bd2700 10 stat 3.3a_head/c913a83a/rbd_data.108f6b8b4567.000000000000114c/head//3
2014-06-12 14:51:37.697348 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c913a83a/rbd_data.108f6b8b4567.000000000000114c/head//3
2014-06-12 14:51:37.710027 7f9d60bd2700 10 getattrs 3.3a_head/c913a83a/rbd_data.108f6b8b4567.000000000000114c/head//3 = 0
2014-06-12 14:51:37.710035 7f9d60bd2700 10 stat 3.3a_head/78f4a83a/rbd_data.10996b8b4567.0000000000002986/head//3
2014-06-12 14:51:37.710133 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 78f4a83a/rbd_data.10996b8b4567.0000000000002986/head//3
2014-06-12 14:51:37.722498 7f9d60bd2700 10 getattrs 3.3a_head/78f4a83a/rbd_data.10996b8b4567.0000000000002986/head//3 = 0
2014-06-12 14:51:37.722507 7f9d60bd2700 10 stat 3.3a_head/3396a83a/rbd_data.10a26b8b4567.00000000000028c3/head//3
2014-06-12 14:51:37.722605 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 3396a83a/rbd_data.10a26b8b4567.00000000000028c3/head//3
2014-06-12 14:51:37.746356 7f9d60bd2700 10 getattrs 3.3a_head/3396a83a/rbd_data.10a26b8b4567.00000000000028c3/head//3 = 0
2014-06-12 14:51:37.746369 7f9d60bd2700 10 stat 3.3a_head/a0d6a83a/rbd_data.10a26b8b4567.0000000000000340/head//3
2014-06-12 14:51:37.746479 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a0d6a83a/rbd_data.10a26b8b4567.0000000000000340/head//3
2014-06-12 14:51:37.777229 7f9d60bd2700 10 getattrs 3.3a_head/a0d6a83a/rbd_data.10a26b8b4567.0000000000000340/head//3 = 0
2014-06-12 14:51:37.777241 7f9d60bd2700 10 stat 3.3a_head/fd68a83a/rbd_data.10a76b8b4567.000000000000098c/head//3
2014-06-12 14:51:37.777348 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid fd68a83a/rbd_data.10a76b8b4567.000000000000098c/head//3
2014-06-12 14:51:37.822226 7f9d60bd2700 10 getattrs 3.3a_head/fd68a83a/rbd_data.10a76b8b4567.000000000000098c/head//3 = 0
2014-06-12 14:51:37.822236 7f9d60bd2700 10 stat 3.3a_head/461ca83a/rbd_data.109f6b8b4567.0000000000000bc6/head//3
2014-06-12 14:51:37.822354 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 461ca83a/rbd_data.109f6b8b4567.0000000000000bc6/head//3
2014-06-12 14:51:37.881903 7f9d60bd2700 10 getattrs 3.3a_head/461ca83a/rbd_data.109f6b8b4567.0000000000000bc6/head//3 = 0
2014-06-12 14:51:37.881917 7f9d60bd2700 10 stat 3.3a_head/50a1b83a/rbd_data.10a06b8b4567.00000000000003b2/head//3
2014-06-12 14:51:37.882044 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 50a1b83a/rbd_data.10a06b8b4567.00000000000003b2/head//3
2014-06-12 14:51:37.933097 7f9d44101700  5 -- op tracker -- , seq: 748591, time: 2014-06-12 14:51:37.932998, event: header_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:37.933113 7f9d44101700  5 -- op tracker -- , seq: 748591, time: 2014-06-12 14:51:37.933001, event: throttled, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:37.933121 7f9d44101700  5 -- op tracker -- , seq: 748591, time: 2014-06-12 14:51:37.933055, event: all_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:37.933128 7f9d44101700  5 -- op tracker -- , seq: 748591, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:37.933134 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:37.933140 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:37.933149 7f9d44101700 15 osd.2 173 enqueue_op 0x55c2100 prio 127 cost 10717 latency 0.000151 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:37.933218 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:37.934606 7f9d60bd2700 10 getattrs 3.3a_head/50a1b83a/rbd_data.10a06b8b4567.00000000000003b2/head//3 = 0
2014-06-12 14:51:37.934623 7f9d60bd2700 10 stat 3.3a_head/6de3b83a/rbd_data.10a46b8b4567.0000000000001889/head//3
2014-06-12 14:51:37.934778 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 6de3b83a/rbd_data.10a46b8b4567.0000000000001889/head//3
2014-06-12 14:51:37.952556 7f9d60bd2700 10 getattrs 3.3a_head/6de3b83a/rbd_data.10a46b8b4567.0000000000001889/head//3 = 0
2014-06-12 14:51:37.952573 7f9d60bd2700 10 stat 3.3a_head/2e4b83a/rbd_data.10b26b8b4567.0000000000001660/head//3
2014-06-12 14:51:37.952733 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 2e4b83a/rbd_data.10b26b8b4567.0000000000001660/head//3
2014-06-12 14:51:37.952869 7f9d60bd2700 10 getattrs 3.3a_head/2e4b83a/rbd_data.10b26b8b4567.0000000000001660/head//3 = 0
2014-06-12 14:51:37.952878 7f9d60bd2700 10 stat 3.3a_head/65f4b83a/rbd_data.10ad6b8b4567.0000000000001c7a/head//3
2014-06-12 14:51:37.953017 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 65f4b83a/rbd_data.10ad6b8b4567.0000000000001c7a/head//3
2014-06-12 14:51:37.968205 7f9d60bd2700 10 getattrs 3.3a_head/65f4b83a/rbd_data.10ad6b8b4567.0000000000001c7a/head//3 = 0
2014-06-12 14:51:37.968212 7f9d60bd2700 10 stat 3.3a_head/90a6b83a/rbd_data.10ab6b8b4567.0000000000000545/head//3
2014-06-12 14:51:37.968316 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 90a6b83a/rbd_data.10ab6b8b4567.0000000000000545/head//3
2014-06-12 14:51:37.976958 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:38.014739 7f9d60bd2700 10 getattrs 3.3a_head/90a6b83a/rbd_data.10ab6b8b4567.0000000000000545/head//3 = 0
2014-06-12 14:51:38.014761 7f9d60bd2700 10 stat 3.3a_head/c289b83a/rbd_data.10996b8b4567.00000000000021ae/head//3
2014-06-12 14:51:38.014975 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c289b83a/rbd_data.10996b8b4567.00000000000021ae/head//3
2014-06-12 14:51:38.032110 7f9d60bd2700 10 getattrs 3.3a_head/c289b83a/rbd_data.10996b8b4567.00000000000021ae/head//3 = 0
2014-06-12 14:51:38.032127 7f9d60bd2700 10 collection_getattrs 3.3a_head
2014-06-12 14:51:38.032218 7f9d60bd2700 10 collection_getattrs 3.3a_head = 0
2014-06-12 14:51:38.032225 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:38.032276 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:38.032292 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:38.032307 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:38.032311 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:38.032346 7f9d61bd4700 10 osd.2 173 dequeue_op 0x55c2100 prio 127 cost 10717 latency 0.099347 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]
2014-06-12 14:51:38.032403 7f9d61bd4700  5 -- op tracker -- , seq: 748591, time: 2014-06-12 14:51:38.032394, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.032421 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] handle_message: 0x55c2100
2014-06-12 14:51:38.032443 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:38.032461 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:38.032478 7f9d61bd4700  5 -- op tracker -- , seq: 748591, time: 2014-06-12 14:51:38.032473, event: started, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.032490 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:38.032597 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] map version is 173'2964
2014-06-12 14:51:38.032618 7f9d61bd4700 10 osd.2 173 dequeue_op 0x55c2100 finish
2014-06-12 14:51:38.032630 7f9d61bd4700  5 -- op tracker -- , seq: 748591, time: 2014-06-12 14:51:38.032624, event: done, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.032652 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:38.032659 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:38.032658 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:38.032672 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:38.032700 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:38.032711 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:38.032746 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:38.032758 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:38.032768 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:38.032800 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5341983a/rbd_data.10956b8b4567.0000000000002fa9/head//3, auth == maps.end()
2014-06-12 14:51:38.032829 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5341983a/rbd_data.10956b8b4567.0000000000002fa9/head//3
2014-06-12 14:51:38.032849 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 51b1983a/rbd_data.10a26b8b4567.0000000000001c33/head//3, auth == maps.end()
2014-06-12 14:51:38.032866 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 51b1983a/rbd_data.10a26b8b4567.0000000000001c33/head//3
2014-06-12 14:51:38.032882 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6283983a/rbd_data.10936b8b4567.0000000000001e15/head//3, auth == maps.end()
2014-06-12 14:51:38.032896 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6283983a/rbd_data.10936b8b4567.0000000000001e15/head//3
2014-06-12 14:51:38.032910 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5944983a/rbd_data.10996b8b4567.00000000000003c8/head//3, auth == maps.end()
2014-06-12 14:51:38.032924 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5944983a/rbd_data.10996b8b4567.00000000000003c8/head//3
2014-06-12 14:51:38.032938 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3aaa983a/rbd_data.10976b8b4567.0000000000001eb2/head//3, auth == maps.end()
2014-06-12 14:51:38.032952 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3aaa983a/rbd_data.10976b8b4567.0000000000001eb2/head//3
2014-06-12 14:51:38.032966 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 74fa983a/rbd_data.10b06b8b4567.00000000000000ed/head//3, auth == maps.end()
2014-06-12 14:51:38.032982 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 74fa983a/rbd_data.10b06b8b4567.00000000000000ed/head//3
2014-06-12 14:51:38.032999 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dcb983a/rbd_data.10af6b8b4567.00000000000027cb/head//3, auth == maps.end()
2014-06-12 14:51:38.033015 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj dcb983a/rbd_data.10af6b8b4567.00000000000027cb/head//3
2014-06-12 14:51:38.033030 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d80d983a/rbd_data.10986b8b4567.00000000000012d9/head//3, auth == maps.end()
2014-06-12 14:51:38.033043 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d80d983a/rbd_data.10986b8b4567.00000000000012d9/head//3
2014-06-12 14:51:38.033062 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ddfe983a/rbd_data.10b16b8b4567.00000000000022c4/head//3, auth == maps.end()
2014-06-12 14:51:38.033078 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ddfe983a/rbd_data.10b16b8b4567.00000000000022c4/head//3
2014-06-12 14:51:38.033093 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 993f983a/rbd_data.10916b8b4567.0000000000002e29/head//3, auth == maps.end()
2014-06-12 14:51:38.033109 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 993f983a/rbd_data.10916b8b4567.0000000000002e29/head//3
2014-06-12 14:51:38.033125 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1f60a83a/rbd_data.10ad6b8b4567.000000000000320d/head//3, auth == maps.end()
2014-06-12 14:51:38.033138 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1f60a83a/rbd_data.10ad6b8b4567.000000000000320d/head//3
2014-06-12 14:51:38.033153 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9b80a83a/rbd_data.10996b8b4567.00000000000006e8/head//3, auth == maps.end()
2014-06-12 14:51:38.033166 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9b80a83a/rbd_data.10996b8b4567.00000000000006e8/head//3
2014-06-12 14:51:38.033180 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj adf0a83a/rbd_data.10b16b8b4567.00000000000024af/head//3, auth == maps.end()
2014-06-12 14:51:38.033194 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj adf0a83a/rbd_data.10b16b8b4567.00000000000024af/head//3
2014-06-12 14:51:38.033208 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c913a83a/rbd_data.108f6b8b4567.000000000000114c/head//3, auth == maps.end()
2014-06-12 14:51:38.033221 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c913a83a/rbd_data.108f6b8b4567.000000000000114c/head//3
2014-06-12 14:51:38.033240 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 78f4a83a/rbd_data.10996b8b4567.0000000000002986/head//3, auth == maps.end()
2014-06-12 14:51:38.033256 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 78f4a83a/rbd_data.10996b8b4567.0000000000002986/head//3
2014-06-12 14:51:38.033271 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3396a83a/rbd_data.10a26b8b4567.00000000000028c3/head//3, auth == maps.end()
2014-06-12 14:51:38.033287 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3396a83a/rbd_data.10a26b8b4567.00000000000028c3/head//3
2014-06-12 14:51:38.033305 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a0d6a83a/rbd_data.10a26b8b4567.0000000000000340/head//3, auth == maps.end()
2014-06-12 14:51:38.033321 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a0d6a83a/rbd_data.10a26b8b4567.0000000000000340/head//3
2014-06-12 14:51:38.033337 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fd68a83a/rbd_data.10a76b8b4567.000000000000098c/head//3, auth == maps.end()
2014-06-12 14:51:38.033353 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj fd68a83a/rbd_data.10a76b8b4567.000000000000098c/head//3
2014-06-12 14:51:38.033368 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 461ca83a/rbd_data.109f6b8b4567.0000000000000bc6/head//3, auth == maps.end()
2014-06-12 14:51:38.033381 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 461ca83a/rbd_data.109f6b8b4567.0000000000000bc6/head//3
2014-06-12 14:51:38.033395 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 50a1b83a/rbd_data.10a06b8b4567.00000000000003b2/head//3, auth == maps.end()
2014-06-12 14:51:38.033410 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 50a1b83a/rbd_data.10a06b8b4567.00000000000003b2/head//3
2014-06-12 14:51:38.033425 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6de3b83a/rbd_data.10a46b8b4567.0000000000001889/head//3, auth == maps.end()
2014-06-12 14:51:38.033438 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6de3b83a/rbd_data.10a46b8b4567.0000000000001889/head//3
2014-06-12 14:51:38.033453 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2e4b83a/rbd_data.10b26b8b4567.0000000000001660/head//3, auth == maps.end()
2014-06-12 14:51:38.033466 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2e4b83a/rbd_data.10b26b8b4567.0000000000001660/head//3
2014-06-12 14:51:38.033481 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 65f4b83a/rbd_data.10ad6b8b4567.0000000000001c7a/head//3, auth == maps.end()
2014-06-12 14:51:38.033497 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 65f4b83a/rbd_data.10ad6b8b4567.0000000000001c7a/head//3
2014-06-12 14:51:38.033512 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 90a6b83a/rbd_data.10ab6b8b4567.0000000000000545/head//3, auth == maps.end()
2014-06-12 14:51:38.033528 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 90a6b83a/rbd_data.10ab6b8b4567.0000000000000545/head//3
2014-06-12 14:51:38.033544 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c289b83a/rbd_data.10996b8b4567.00000000000021ae/head//3, auth == maps.end()
2014-06-12 14:51:38.033560 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c289b83a/rbd_data.10996b8b4567.00000000000021ae/head//3
2014-06-12 14:51:38.033577 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] 
2014-06-12 14:51:38.033589 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub
2014-06-12 14:51:38.033609 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c289b83a/rbd_data.10996b8b4567.00000000000021ae/head//3 c289b83a/rbd_data.10996b8b4567.00000000000021ae/head//3(173'1408 client.4290.0:68885 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1408)
2014-06-12 14:51:38.033669 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  90a6b83a/rbd_data.10ab6b8b4567.0000000000000545/head//3 90a6b83a/rbd_data.10ab6b8b4567.0000000000000545/head//3(173'2653 client.4308.0:127262 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2653)
2014-06-12 14:51:38.033696 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  65f4b83a/rbd_data.10ad6b8b4567.0000000000001c7a/head//3 65f4b83a/rbd_data.10ad6b8b4567.0000000000001c7a/head//3(173'1281 client.4310.0:60818 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1281)
2014-06-12 14:51:38.033720 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  2e4b83a/rbd_data.10b26b8b4567.0000000000001660/head//3 2e4b83a/rbd_data.10b26b8b4567.0000000000001660/head//3(173'1037 client.4315.0:49599 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1037)
2014-06-12 14:51:38.033745 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  6de3b83a/rbd_data.10a46b8b4567.0000000000001889/head//3 6de3b83a/rbd_data.10a46b8b4567.0000000000001889/head//3(173'1147 client.4301.0:54228 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1147)
2014-06-12 14:51:38.033769 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  50a1b83a/rbd_data.10a06b8b4567.00000000000003b2/head//3 50a1b83a/rbd_data.10a06b8b4567.00000000000003b2/head//3(173'2580 client.4297.0:122299 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2580)
2014-06-12 14:51:38.033794 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  461ca83a/rbd_data.109f6b8b4567.0000000000000bc6/head//3 461ca83a/rbd_data.109f6b8b4567.0000000000000bc6/head//3(173'2834 client.4296.0:134346 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2834)
2014-06-12 14:51:38.033818 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  fd68a83a/rbd_data.10a76b8b4567.000000000000098c/head//3 fd68a83a/rbd_data.10a76b8b4567.000000000000098c/head//3(173'2772 client.4304.0:132384 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2772)
2014-06-12 14:51:38.033840 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a0d6a83a/rbd_data.10a26b8b4567.0000000000000340/head//3 a0d6a83a/rbd_data.10a26b8b4567.0000000000000340/head//3(173'2542 client.4299.0:124209 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2542)
2014-06-12 14:51:38.033864 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  3396a83a/rbd_data.10a26b8b4567.00000000000028c3/head//3 3396a83a/rbd_data.10a26b8b4567.00000000000028c3/head//3(173'1945 client.4299.0:85236 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1945)
2014-06-12 14:51:38.033889 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  78f4a83a/rbd_data.10996b8b4567.0000000000002986/head//3 78f4a83a/rbd_data.10996b8b4567.0000000000002986/head//3(173'1798 client.4290.0:83271 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1798)
2014-06-12 14:51:38.033913 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c913a83a/rbd_data.108f6b8b4567.000000000000114c/head//3 c913a83a/rbd_data.108f6b8b4567.000000000000114c/head//3(173'935 client.4280.0:39989 wrlock_by=unknown.0.0:0 dirty s 4194304 uv935)
2014-06-12 14:51:38.033938 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  adf0a83a/rbd_data.10b16b8b4567.00000000000024af/head//3 adf0a83a/rbd_data.10b16b8b4567.00000000000024af/head//3(173'1690 client.4314.0:75801 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1690)
2014-06-12 14:51:38.033962 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  9b80a83a/rbd_data.10996b8b4567.00000000000006e8/head//3 9b80a83a/rbd_data.10996b8b4567.00000000000006e8/head//3(173'2716 client.4290.0:127292 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2716)
2014-06-12 14:51:38.033986 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  1f60a83a/rbd_data.10ad6b8b4567.000000000000320d/head//3 1f60a83a/rbd_data.10ad6b8b4567.000000000000320d/head//3(173'2066 client.4310.0:99298 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2066)
2014-06-12 14:51:38.034010 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  993f983a/rbd_data.10916b8b4567.0000000000002e29/head//3 993f983a/rbd_data.10916b8b4567.0000000000002e29/head//3(173'2003 client.4282.0:93546 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2003)
2014-06-12 14:51:38.034034 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ddfe983a/rbd_data.10b16b8b4567.00000000000022c4/head//3 ddfe983a/rbd_data.10b16b8b4567.00000000000022c4/head//3(173'1629 client.4314.0:72046 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1629)
2014-06-12 14:51:38.034058 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  d80d983a/rbd_data.10986b8b4567.00000000000012d9/head//3 d80d983a/rbd_data.10986b8b4567.00000000000012d9/head//3(173'939 client.4289.0:42556 wrlock_by=unknown.0.0:0 dirty s 4194304 uv939)
2014-06-12 14:51:38.034081 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  dcb983a/rbd_data.10af6b8b4567.00000000000027cb/head//3 dcb983a/rbd_data.10af6b8b4567.00000000000027cb/head//3(173'1756 client.4312.0:83273 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1756)
2014-06-12 14:51:38.034105 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  74fa983a/rbd_data.10b06b8b4567.00000000000000ed/head//3 74fa983a/rbd_data.10b06b8b4567.00000000000000ed/head//3(173'2477 client.4313.0:120884 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2477)
2014-06-12 14:51:38.034129 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  3aaa983a/rbd_data.10976b8b4567.0000000000001eb2/head//3 3aaa983a/rbd_data.10976b8b4567.0000000000001eb2/head//3(173'1484 client.4288.0:63157 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1484)
2014-06-12 14:51:38.034152 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5944983a/rbd_data.10996b8b4567.00000000000003c8/head//3 5944983a/rbd_data.10996b8b4567.00000000000003c8/head//3(173'2595 client.4290.0:121811 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2595)
2014-06-12 14:51:38.034177 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  6283983a/rbd_data.10936b8b4567.0000000000001e15/head//3 6283983a/rbd_data.10936b8b4567.0000000000001e15/head//3(173'1337 client.4284.0:61837 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1337)
2014-06-12 14:51:38.034200 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  51b1983a/rbd_data.10a26b8b4567.0000000000001c33/head//3 51b1983a/rbd_data.10a26b8b4567.0000000000001c33/head//3(173'1345 client.4299.0:61262 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1345)
2014-06-12 14:51:38.034224 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5341983a/rbd_data.10956b8b4567.0000000000002fa9/head//3 5341983a/rbd_data.10956b8b4567.0000000000002fa9/head//3(173'2053 client.4286.0:95751 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2053)
2014-06-12 14:51:38.034243 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:38.034262 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:38.034279 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:38.034282 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:38.034285 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:38.034343 7f9d60bd2700 10 collection_list_partial 3.3a_head start:4daab83a//0//-1 is_max:0
2014-06-12 14:51:38.034716 7f9d60bd2700 20 objects: [23eab83a/rbd_data.10936b8b4567.00000000000015db/head//3,7d3cb83a/rbd_data.10b46b8b4567.0000000000000cc8/head//3,b93fb83a/rbd_data.10936b8b4567.00000000000008cc/head//3,24a1c83a/rbd_data.109b6b8b4567.00000000000027d3/head//3,3c32c83a/rbd_data.10ad6b8b4567.0000000000002293/head//3,ac52c83a/rbd_data.10b06b8b4567.0000000000002bdd/head//3,b962c83a/rbd_data.10996b8b4567.0000000000002769/head//3,55b4c83a/rbd_data.10a56b8b4567.0000000000001bd2/head//3,6a35c83a/rbd_data.10ad6b8b4567.0000000000003928/head//3,43c5c83a/rbd_data.108f6b8b4567.0000000000002fe7/head//3,e439c83a/rbd_data.10b06b8b4567.0000000000003791/head//3,6e6ac83a/rbd_data.10b36b8b4567.00000000000027d6/head//3,18dac83a/rbd_data.10946b8b4567.0000000000002e4c/head//3,fb3ec83a/rbd_data.10986b8b4567.000000000000325e/head//3,3f21d83a/rbd_data.10a16b8b4567.00000000000039f0/head//3,1d74d83a/rbd_data.108d6b8b4567.0000000000001e54/head//3,dfa4d83a/rbd_data.10a96b8b4567.00000000000005c5/head//3,d6a8d83a/rbd_data.10a26b8b4567.000000000000285e/head//3,9ea8d83a/rbd_data.10996b8b4567.0000000000002723/head//3,825ad83a/rbd_data.10a96b8b4567.0000000000001f72/head//3,c67ad83a/rbd_data.10916b8b4567.0000000000002b77/head//3,a89ed83a/rbd_data.10986b8b4567.0000000000001c1c/head//3,97fd83a/rbd_data.10b36b8b4567.0000000000001bda/head//3,46afd83a/rbd_data.10ae6b8b4567.0000000000000419/head//3,2fdfd83a/rbd_data.10a06b8b4567.00000000000032ec/head//3]
2014-06-12 14:51:38.035392 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:38.035432 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:38.035461 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:38.035477 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:38.035488 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:38.035499 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:38.035509 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_map_chunk [4daab83a//0//-1,3f90e83a//0//-1)
2014-06-12 14:51:38.035522 7f9d60bd2700 10 collection_list_partial 3.3a_head start:4daab83a//0//-1 is_max:0
2014-06-12 14:51:38.035955 7f9d60bd2700 20 objects: [23eab83a/rbd_data.10936b8b4567.00000000000015db/head//3,7d3cb83a/rbd_data.10b46b8b4567.0000000000000cc8/head//3,b93fb83a/rbd_data.10936b8b4567.00000000000008cc/head//3,24a1c83a/rbd_data.109b6b8b4567.00000000000027d3/head//3,3c32c83a/rbd_data.10ad6b8b4567.0000000000002293/head//3,ac52c83a/rbd_data.10b06b8b4567.0000000000002bdd/head//3,b962c83a/rbd_data.10996b8b4567.0000000000002769/head//3,55b4c83a/rbd_data.10a56b8b4567.0000000000001bd2/head//3,6a35c83a/rbd_data.10ad6b8b4567.0000000000003928/head//3,43c5c83a/rbd_data.108f6b8b4567.0000000000002fe7/head//3,e439c83a/rbd_data.10b06b8b4567.0000000000003791/head//3,6e6ac83a/rbd_data.10b36b8b4567.00000000000027d6/head//3,18dac83a/rbd_data.10946b8b4567.0000000000002e4c/head//3,fb3ec83a/rbd_data.10986b8b4567.000000000000325e/head//3,3f21d83a/rbd_data.10a16b8b4567.00000000000039f0/head//3,1d74d83a/rbd_data.108d6b8b4567.0000000000001e54/head//3,dfa4d83a/rbd_data.10a96b8b4567.00000000000005c5/head//3,d6a8d83a/rbd_data.10a26b8b4567.000000000000285e/head//3,9ea8d83a/rbd_data.10996b8b4567.0000000000002723/head//3,825ad83a/rbd_data.10a96b8b4567.0000000000001f72/head//3,c67ad83a/rbd_data.10916b8b4567.0000000000002b77/head//3,a89ed83a/rbd_data.10986b8b4567.0000000000001c1c/head//3,97fd83a/rbd_data.10b36b8b4567.0000000000001bda/head//3,46afd83a/rbd_data.10ae6b8b4567.0000000000000419/head//3,2fdfd83a/rbd_data.10a06b8b4567.00000000000032ec/head//3,3f90e83a/rbd_data.10a56b8b4567.0000000000000339/head//3,f5f1e83a/rbd_data.10936b8b4567.000000000000322f/head//3,3562e83a/rbd_data.10ab6b8b4567.0000000000000b05/head//3,7c78e83a/rbd_data.10b26b8b4567.00000000000024a9/head//3,db1ce83a/rbd_data.10956b8b4567.0000000000002b6d/head//3,ecfce83a/rbd_data.10986b8b4567.0000000000001cab/head//3,6641f83a/rbd_data.10b46b8b4567.0000000000000632/head//3,5071f83a/rbd_data.10a96b8b4567.0000000000002efc/head//3,c3c4f83a/rbd_data.10a36b8b4567.0000000000001672/head//3,d27df83a/rbd_data.10996b8b4567.0000000000000387/head//3]
2014-06-12 14:51:38.036045 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:38.036072 7f9d60bd2700 10 stat 3.3a_head/23eab83a/rbd_data.10936b8b4567.00000000000015db/head//3
2014-06-12 14:51:38.036296 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 23eab83a/rbd_data.10936b8b4567.00000000000015db/head//3
2014-06-12 14:51:38.062220 7f9d60bd2700 10 getattrs 3.3a_head/23eab83a/rbd_data.10936b8b4567.00000000000015db/head//3 = 0
2014-06-12 14:51:38.062237 7f9d60bd2700 10 stat 3.3a_head/7d3cb83a/rbd_data.10b46b8b4567.0000000000000cc8/head//3
2014-06-12 14:51:38.062432 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 7d3cb83a/rbd_data.10b46b8b4567.0000000000000cc8/head//3
2014-06-12 14:51:38.064633 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:38.090724 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:38.090744 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:38.091363 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:38.091395 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:38.091473 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:38.091507 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:38.091560 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:38.091635 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:38.091680 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:38.091704 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:38.091732 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:38.091758 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:38.091782 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:38.091806 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:38.091827 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:38.091850 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:38.091886 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:38.091909 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:38.091933 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:38.091960 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:38.091983 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:38.092007 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:38.092031 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:38.092056 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:38.103903 7f9d60bd2700 10 getattrs 3.3a_head/7d3cb83a/rbd_data.10b46b8b4567.0000000000000cc8/head//3 = 0
2014-06-12 14:51:38.103912 7f9d60bd2700 10 stat 3.3a_head/b93fb83a/rbd_data.10936b8b4567.00000000000008cc/head//3
2014-06-12 14:51:38.104026 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid b93fb83a/rbd_data.10936b8b4567.00000000000008cc/head//3
2014-06-12 14:51:38.149080 7f9d60bd2700 10 getattrs 3.3a_head/b93fb83a/rbd_data.10936b8b4567.00000000000008cc/head//3 = 0
2014-06-12 14:51:38.149097 7f9d60bd2700 10 stat 3.3a_head/24a1c83a/rbd_data.109b6b8b4567.00000000000027d3/head//3
2014-06-12 14:51:38.149267 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 24a1c83a/rbd_data.109b6b8b4567.00000000000027d3/head//3
2014-06-12 14:51:38.167462 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:38.177505 7f9d60bd2700 10 getattrs 3.3a_head/24a1c83a/rbd_data.109b6b8b4567.00000000000027d3/head//3 = 0
2014-06-12 14:51:38.177519 7f9d60bd2700 10 stat 3.3a_head/3c32c83a/rbd_data.10ad6b8b4567.0000000000002293/head//3
2014-06-12 14:51:38.177716 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 3c32c83a/rbd_data.10ad6b8b4567.0000000000002293/head//3
2014-06-12 14:51:38.190355 7f9d60bd2700 10 getattrs 3.3a_head/3c32c83a/rbd_data.10ad6b8b4567.0000000000002293/head//3 = 0
2014-06-12 14:51:38.190362 7f9d60bd2700 10 stat 3.3a_head/ac52c83a/rbd_data.10b06b8b4567.0000000000002bdd/head//3
2014-06-12 14:51:38.190459 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ac52c83a/rbd_data.10b06b8b4567.0000000000002bdd/head//3
2014-06-12 14:51:38.193975 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:38.215488 7f9d60bd2700 10 getattrs 3.3a_head/ac52c83a/rbd_data.10b06b8b4567.0000000000002bdd/head//3 = 0
2014-06-12 14:51:38.215495 7f9d60bd2700 10 stat 3.3a_head/b962c83a/rbd_data.10996b8b4567.0000000000002769/head//3
2014-06-12 14:51:38.215589 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid b962c83a/rbd_data.10996b8b4567.0000000000002769/head//3
2014-06-12 14:51:38.229136 7f9d60bd2700 10 getattrs 3.3a_head/b962c83a/rbd_data.10996b8b4567.0000000000002769/head//3 = 0
2014-06-12 14:51:38.229144 7f9d60bd2700 10 stat 3.3a_head/55b4c83a/rbd_data.10a56b8b4567.0000000000001bd2/head//3
2014-06-12 14:51:38.229230 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 55b4c83a/rbd_data.10a56b8b4567.0000000000001bd2/head//3
2014-06-12 14:51:38.256158 7f9d60bd2700 10 getattrs 3.3a_head/55b4c83a/rbd_data.10a56b8b4567.0000000000001bd2/head//3 = 0
2014-06-12 14:51:38.256171 7f9d60bd2700 10 stat 3.3a_head/6a35c83a/rbd_data.10ad6b8b4567.0000000000003928/head//3
2014-06-12 14:51:38.256273 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 6a35c83a/rbd_data.10ad6b8b4567.0000000000003928/head//3
2014-06-12 14:51:38.290139 7f9d60bd2700 10 getattrs 3.3a_head/6a35c83a/rbd_data.10ad6b8b4567.0000000000003928/head//3 = 0
2014-06-12 14:51:38.290148 7f9d60bd2700 10 stat 3.3a_head/43c5c83a/rbd_data.108f6b8b4567.0000000000002fe7/head//3
2014-06-12 14:51:38.290254 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 43c5c83a/rbd_data.108f6b8b4567.0000000000002fe7/head//3
2014-06-12 14:51:38.302496 7f9d60bd2700 10 getattrs 3.3a_head/43c5c83a/rbd_data.108f6b8b4567.0000000000002fe7/head//3 = 0
2014-06-12 14:51:38.302509 7f9d60bd2700 10 stat 3.3a_head/e439c83a/rbd_data.10b06b8b4567.0000000000003791/head//3
2014-06-12 14:51:38.302619 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid e439c83a/rbd_data.10b06b8b4567.0000000000003791/head//3
2014-06-12 14:51:38.316896 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:38.326325 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:38.336555 7f9d60bd2700 10 getattrs 3.3a_head/e439c83a/rbd_data.10b06b8b4567.0000000000003791/head//3 = 0
2014-06-12 14:51:38.336562 7f9d60bd2700 10 stat 3.3a_head/6e6ac83a/rbd_data.10b36b8b4567.00000000000027d6/head//3
2014-06-12 14:51:38.336657 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 6e6ac83a/rbd_data.10b36b8b4567.00000000000027d6/head//3
2014-06-12 14:51:38.366255 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:38.371153 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:38.372011 7f9d60bd2700 10 getattrs 3.3a_head/6e6ac83a/rbd_data.10b36b8b4567.00000000000027d6/head//3 = 0
2014-06-12 14:51:38.372024 7f9d60bd2700 10 stat 3.3a_head/18dac83a/rbd_data.10946b8b4567.0000000000002e4c/head//3
2014-06-12 14:51:38.372146 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 18dac83a/rbd_data.10946b8b4567.0000000000002e4c/head//3
2014-06-12 14:51:38.381923 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:38.391883 7f9d60bd2700 10 getattrs 3.3a_head/18dac83a/rbd_data.10946b8b4567.0000000000002e4c/head//3 = 0
2014-06-12 14:51:38.391893 7f9d60bd2700 10 stat 3.3a_head/fb3ec83a/rbd_data.10986b8b4567.000000000000325e/head//3
2014-06-12 14:51:38.392007 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid fb3ec83a/rbd_data.10986b8b4567.000000000000325e/head//3
2014-06-12 14:51:38.407527 7f9d60bd2700 10 getattrs 3.3a_head/fb3ec83a/rbd_data.10986b8b4567.000000000000325e/head//3 = 0
2014-06-12 14:51:38.407537 7f9d60bd2700 10 stat 3.3a_head/3f21d83a/rbd_data.10a16b8b4567.00000000000039f0/head//3
2014-06-12 14:51:38.407647 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 3f21d83a/rbd_data.10a16b8b4567.00000000000039f0/head//3
2014-06-12 14:51:38.419502 7f9d60bd2700 10 getattrs 3.3a_head/3f21d83a/rbd_data.10a16b8b4567.00000000000039f0/head//3 = 0
2014-06-12 14:51:38.419514 7f9d60bd2700 10 stat 3.3a_head/1d74d83a/rbd_data.108d6b8b4567.0000000000001e54/head//3
2014-06-12 14:51:38.419626 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 1d74d83a/rbd_data.108d6b8b4567.0000000000001e54/head//3
2014-06-12 14:51:38.444939 7f9d60bd2700 10 getattrs 3.3a_head/1d74d83a/rbd_data.108d6b8b4567.0000000000001e54/head//3 = 0
2014-06-12 14:51:38.444950 7f9d60bd2700 10 stat 3.3a_head/dfa4d83a/rbd_data.10a96b8b4567.00000000000005c5/head//3
2014-06-12 14:51:38.445063 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid dfa4d83a/rbd_data.10a96b8b4567.00000000000005c5/head//3
2014-06-12 14:51:38.485729 7f9d60bd2700 10 getattrs 3.3a_head/dfa4d83a/rbd_data.10a96b8b4567.00000000000005c5/head//3 = 0
2014-06-12 14:51:38.485751 7f9d60bd2700 10 stat 3.3a_head/d6a8d83a/rbd_data.10a26b8b4567.000000000000285e/head//3
2014-06-12 14:51:38.485923 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid d6a8d83a/rbd_data.10a26b8b4567.000000000000285e/head//3
2014-06-12 14:51:38.513142 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:38.516327 7f9d60bd2700 10 getattrs 3.3a_head/d6a8d83a/rbd_data.10a26b8b4567.000000000000285e/head//3 = 0
2014-06-12 14:51:38.516336 7f9d60bd2700 10 stat 3.3a_head/9ea8d83a/rbd_data.10996b8b4567.0000000000002723/head//3
2014-06-12 14:51:38.516418 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 9ea8d83a/rbd_data.10996b8b4567.0000000000002723/head//3
2014-06-12 14:51:38.518973 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:38.520279 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:38.520328 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.5 >= max 0.5 = no, load too high
2014-06-12 14:51:38.520348 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:51:38.520350 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:38.520351 7f9d84175700 10 osd.2 173 sched_scrub 3.3a high load at 2014-06-11 13:02:32.500954: 92946 < max (604800 seconds)
2014-06-12 14:51:38.520372 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:38.520375 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:38.520376 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:38.522903 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:38.532346 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:38.534615 7f9d60bd2700 10 getattrs 3.3a_head/9ea8d83a/rbd_data.10996b8b4567.0000000000002723/head//3 = 0
2014-06-12 14:51:38.534627 7f9d60bd2700 10 stat 3.3a_head/825ad83a/rbd_data.10a96b8b4567.0000000000001f72/head//3
2014-06-12 14:51:38.534738 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 825ad83a/rbd_data.10a96b8b4567.0000000000001f72/head//3
2014-06-12 14:51:38.535102 7f9d60bd2700 10 getattrs 3.3a_head/825ad83a/rbd_data.10a96b8b4567.0000000000001f72/head//3 = 0
2014-06-12 14:51:38.535109 7f9d60bd2700 10 stat 3.3a_head/c67ad83a/rbd_data.10916b8b4567.0000000000002b77/head//3
2014-06-12 14:51:38.535207 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c67ad83a/rbd_data.10916b8b4567.0000000000002b77/head//3
2014-06-12 14:51:38.547785 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:38.562810 7f9d60bd2700 10 getattrs 3.3a_head/c67ad83a/rbd_data.10916b8b4567.0000000000002b77/head//3 = 0
2014-06-12 14:51:38.562819 7f9d60bd2700 10 stat 3.3a_head/a89ed83a/rbd_data.10986b8b4567.0000000000001c1c/head//3
2014-06-12 14:51:38.562920 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid a89ed83a/rbd_data.10986b8b4567.0000000000001c1c/head//3
2014-06-12 14:51:38.580950 7f9d60bd2700 10 getattrs 3.3a_head/a89ed83a/rbd_data.10986b8b4567.0000000000001c1c/head//3 = 0
2014-06-12 14:51:38.580960 7f9d60bd2700 10 stat 3.3a_head/97fd83a/rbd_data.10b36b8b4567.0000000000001bda/head//3
2014-06-12 14:51:38.581067 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 97fd83a/rbd_data.10b36b8b4567.0000000000001bda/head//3
2014-06-12 14:51:38.615636 7f9d60bd2700 10 getattrs 3.3a_head/97fd83a/rbd_data.10b36b8b4567.0000000000001bda/head//3 = 0
2014-06-12 14:51:38.615666 7f9d60bd2700 10 stat 3.3a_head/46afd83a/rbd_data.10ae6b8b4567.0000000000000419/head//3
2014-06-12 14:51:38.615773 7f9d44101700  5 -- op tracker -- , seq: 748592, time: 2014-06-12 14:51:38.615683, event: header_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.615781 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 46afd83a/rbd_data.10ae6b8b4567.0000000000000419/head//3
2014-06-12 14:51:38.615784 7f9d44101700  5 -- op tracker -- , seq: 748592, time: 2014-06-12 14:51:38.615686, event: throttled, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.615791 7f9d44101700  5 -- op tracker -- , seq: 748592, time: 2014-06-12 14:51:38.615732, event: all_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.615797 7f9d44101700  5 -- op tracker -- , seq: 748592, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.615801 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:38.615806 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:38.615813 7f9d44101700 15 osd.2 173 enqueue_op 0x55c2f00 prio 127 cost 10717 latency 0.000130 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:38.615863 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:38.653683 7f9d60bd2700 10 getattrs 3.3a_head/46afd83a/rbd_data.10ae6b8b4567.0000000000000419/head//3 = 0
2014-06-12 14:51:38.653694 7f9d60bd2700 10 stat 3.3a_head/2fdfd83a/rbd_data.10a06b8b4567.00000000000032ec/head//3
2014-06-12 14:51:38.653831 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 2fdfd83a/rbd_data.10a06b8b4567.00000000000032ec/head//3
2014-06-12 14:51:38.661716 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:38.679401 7f9d60bd2700 10 getattrs 3.3a_head/2fdfd83a/rbd_data.10a06b8b4567.00000000000032ec/head//3 = 0
2014-06-12 14:51:38.679413 7f9d60bd2700 10 collection_getattrs 3.3a_head
2014-06-12 14:51:38.679488 7f9d60bd2700 10 collection_getattrs 3.3a_head = 0
2014-06-12 14:51:38.679493 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:38.679532 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:38.679541 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:38.679552 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:38.679556 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:38.679586 7f9d683e1700 10 osd.2 173 dequeue_op 0x55c2f00 prio 127 cost 10717 latency 0.063902 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]
2014-06-12 14:51:38.679644 7f9d683e1700  5 -- op tracker -- , seq: 748592, time: 2014-06-12 14:51:38.679636, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.679669 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] handle_message: 0x55c2f00
2014-06-12 14:51:38.679685 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:38.679703 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:38.679720 7f9d683e1700  5 -- op tracker -- , seq: 748592, time: 2014-06-12 14:51:38.679715, event: started, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.679748 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:38.679851 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] map version is 173'2964
2014-06-12 14:51:38.679872 7f9d683e1700 10 osd.2 173 dequeue_op 0x55c2f00 finish
2014-06-12 14:51:38.679884 7f9d683e1700  5 -- op tracker -- , seq: 748592, time: 2014-06-12 14:51:38.679878, event: done, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.679906 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:38.679913 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:38.679911 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:38.679925 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:38.679953 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:38.679962 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:38.679995 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:38.680006 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:38.680014 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub replica 21 has 25 items
2014-06-12 14:51:38.680039 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 23eab83a/rbd_data.10936b8b4567.00000000000015db/head//3, auth == maps.end()
2014-06-12 14:51:38.680061 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 23eab83a/rbd_data.10936b8b4567.00000000000015db/head//3
2014-06-12 14:51:38.680079 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7d3cb83a/rbd_data.10b46b8b4567.0000000000000cc8/head//3, auth == maps.end()
2014-06-12 14:51:38.680093 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7d3cb83a/rbd_data.10b46b8b4567.0000000000000cc8/head//3
2014-06-12 14:51:38.680106 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b93fb83a/rbd_data.10936b8b4567.00000000000008cc/head//3, auth == maps.end()
2014-06-12 14:51:38.680119 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b93fb83a/rbd_data.10936b8b4567.00000000000008cc/head//3
2014-06-12 14:51:38.680131 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 24a1c83a/rbd_data.109b6b8b4567.00000000000027d3/head//3, auth == maps.end()
2014-06-12 14:51:38.680142 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 24a1c83a/rbd_data.109b6b8b4567.00000000000027d3/head//3
2014-06-12 14:51:38.680154 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3c32c83a/rbd_data.10ad6b8b4567.0000000000002293/head//3, auth == maps.end()
2014-06-12 14:51:38.680165 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3c32c83a/rbd_data.10ad6b8b4567.0000000000002293/head//3
2014-06-12 14:51:38.680176 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ac52c83a/rbd_data.10b06b8b4567.0000000000002bdd/head//3, auth == maps.end()
2014-06-12 14:51:38.680187 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ac52c83a/rbd_data.10b06b8b4567.0000000000002bdd/head//3
2014-06-12 14:51:38.680199 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b962c83a/rbd_data.10996b8b4567.0000000000002769/head//3, auth == maps.end()
2014-06-12 14:51:38.680209 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj b962c83a/rbd_data.10996b8b4567.0000000000002769/head//3
2014-06-12 14:51:38.680221 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 55b4c83a/rbd_data.10a56b8b4567.0000000000001bd2/head//3, auth == maps.end()
2014-06-12 14:51:38.680233 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 55b4c83a/rbd_data.10a56b8b4567.0000000000001bd2/head//3
2014-06-12 14:51:38.680246 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6a35c83a/rbd_data.10ad6b8b4567.0000000000003928/head//3, auth == maps.end()
2014-06-12 14:51:38.680258 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6a35c83a/rbd_data.10ad6b8b4567.0000000000003928/head//3
2014-06-12 14:51:38.680271 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 43c5c83a/rbd_data.108f6b8b4567.0000000000002fe7/head//3, auth == maps.end()
2014-06-12 14:51:38.680282 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 43c5c83a/rbd_data.108f6b8b4567.0000000000002fe7/head//3
2014-06-12 14:51:38.680294 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e439c83a/rbd_data.10b06b8b4567.0000000000003791/head//3, auth == maps.end()
2014-06-12 14:51:38.680307 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj e439c83a/rbd_data.10b06b8b4567.0000000000003791/head//3
2014-06-12 14:51:38.680319 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6e6ac83a/rbd_data.10b36b8b4567.00000000000027d6/head//3, auth == maps.end()
2014-06-12 14:51:38.680331 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6e6ac83a/rbd_data.10b36b8b4567.00000000000027d6/head//3
2014-06-12 14:51:38.680344 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 18dac83a/rbd_data.10946b8b4567.0000000000002e4c/head//3, auth == maps.end()
2014-06-12 14:51:38.680355 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 18dac83a/rbd_data.10946b8b4567.0000000000002e4c/head//3
2014-06-12 14:51:38.680366 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fb3ec83a/rbd_data.10986b8b4567.000000000000325e/head//3, auth == maps.end()
2014-06-12 14:51:38.680377 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj fb3ec83a/rbd_data.10986b8b4567.000000000000325e/head//3
2014-06-12 14:51:38.680388 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3f21d83a/rbd_data.10a16b8b4567.00000000000039f0/head//3, auth == maps.end()
2014-06-12 14:51:38.680399 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3f21d83a/rbd_data.10a16b8b4567.00000000000039f0/head//3
2014-06-12 14:51:38.680411 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1d74d83a/rbd_data.108d6b8b4567.0000000000001e54/head//3, auth == maps.end()
2014-06-12 14:51:38.680421 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 1d74d83a/rbd_data.108d6b8b4567.0000000000001e54/head//3
2014-06-12 14:51:38.680433 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dfa4d83a/rbd_data.10a96b8b4567.00000000000005c5/head//3, auth == maps.end()
2014-06-12 14:51:38.680446 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj dfa4d83a/rbd_data.10a96b8b4567.00000000000005c5/head//3
2014-06-12 14:51:38.680460 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d6a8d83a/rbd_data.10a26b8b4567.000000000000285e/head//3, auth == maps.end()
2014-06-12 14:51:38.680473 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d6a8d83a/rbd_data.10a26b8b4567.000000000000285e/head//3
2014-06-12 14:51:38.680487 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9ea8d83a/rbd_data.10996b8b4567.0000000000002723/head//3, auth == maps.end()
2014-06-12 14:51:38.680500 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 9ea8d83a/rbd_data.10996b8b4567.0000000000002723/head//3
2014-06-12 14:51:38.680513 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 825ad83a/rbd_data.10a96b8b4567.0000000000001f72/head//3, auth == maps.end()
2014-06-12 14:51:38.680526 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 825ad83a/rbd_data.10a96b8b4567.0000000000001f72/head//3
2014-06-12 14:51:38.680540 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c67ad83a/rbd_data.10916b8b4567.0000000000002b77/head//3, auth == maps.end()
2014-06-12 14:51:38.680553 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c67ad83a/rbd_data.10916b8b4567.0000000000002b77/head//3
2014-06-12 14:51:38.680566 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a89ed83a/rbd_data.10986b8b4567.0000000000001c1c/head//3, auth == maps.end()
2014-06-12 14:51:38.680578 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj a89ed83a/rbd_data.10986b8b4567.0000000000001c1c/head//3
2014-06-12 14:51:38.680591 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 97fd83a/rbd_data.10b36b8b4567.0000000000001bda/head//3, auth == maps.end()
2014-06-12 14:51:38.680604 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 97fd83a/rbd_data.10b36b8b4567.0000000000001bda/head//3
2014-06-12 14:51:38.680616 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 46afd83a/rbd_data.10ae6b8b4567.0000000000000419/head//3, auth == maps.end()
2014-06-12 14:51:38.680628 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 46afd83a/rbd_data.10ae6b8b4567.0000000000000419/head//3
2014-06-12 14:51:38.680640 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2fdfd83a/rbd_data.10a06b8b4567.00000000000032ec/head//3, auth == maps.end()
2014-06-12 14:51:38.680651 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 2fdfd83a/rbd_data.10a06b8b4567.00000000000032ec/head//3
2014-06-12 14:51:38.680664 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] 
2014-06-12 14:51:38.680672 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub
2014-06-12 14:51:38.680687 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  2fdfd83a/rbd_data.10a06b8b4567.00000000000032ec/head//3 2fdfd83a/rbd_data.10a06b8b4567.00000000000032ec/head//3(173'2107 client.4297.0:100413 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2107)
2014-06-12 14:51:38.680706 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  46afd83a/rbd_data.10ae6b8b4567.0000000000000419/head//3 46afd83a/rbd_data.10ae6b8b4567.0000000000000419/head//3(173'2632 client.4311.0:121038 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2632)
2014-06-12 14:51:38.680723 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  97fd83a/rbd_data.10b36b8b4567.0000000000001bda/head//3 97fd83a/rbd_data.10b36b8b4567.0000000000001bda/head//3(173'1211 client.4316.0:60894 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1211)
2014-06-12 14:51:38.680739 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  a89ed83a/rbd_data.10986b8b4567.0000000000001c1c/head//3 a89ed83a/rbd_data.10986b8b4567.0000000000001c1c/head//3(173'1225 client.4289.0:59137 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1225)
2014-06-12 14:51:38.680755 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c67ad83a/rbd_data.10916b8b4567.0000000000002b77/head//3 c67ad83a/rbd_data.10916b8b4567.0000000000002b77/head//3(173'1934 client.4282.0:88292 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1934)
2014-06-12 14:51:38.680771 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  825ad83a/rbd_data.10a96b8b4567.0000000000001f72/head//3 825ad83a/rbd_data.10a96b8b4567.0000000000001f72/head//3(173'1492 client.4306.0:68248 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1492)
2014-06-12 14:51:38.680787 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  9ea8d83a/rbd_data.10996b8b4567.0000000000002723/head//3 9ea8d83a/rbd_data.10996b8b4567.0000000000002723/head//3(173'1681 client.4290.0:78799 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1681)
2014-06-12 14:51:38.680803 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  d6a8d83a/rbd_data.10a26b8b4567.000000000000285e/head//3 d6a8d83a/rbd_data.10a26b8b4567.000000000000285e/head//3(173'1939 client.4299.0:84488 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1939)
2014-06-12 14:51:38.680819 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  dfa4d83a/rbd_data.10a96b8b4567.00000000000005c5/head//3 dfa4d83a/rbd_data.10a96b8b4567.00000000000005c5/head//3(173'2667 client.4306.0:127187 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2667)
2014-06-12 14:51:38.680835 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  1d74d83a/rbd_data.108d6b8b4567.0000000000001e54/head//3 1d74d83a/rbd_data.108d6b8b4567.0000000000001e54/head//3(173'1264 client.4278.0:65301 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1264)
2014-06-12 14:51:38.680852 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  3f21d83a/rbd_data.10a16b8b4567.00000000000039f0/head//3 3f21d83a/rbd_data.10a16b8b4567.00000000000039f0/head//3(173'2390 client.4298.0:115675 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2390)
2014-06-12 14:51:38.680867 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  fb3ec83a/rbd_data.10986b8b4567.000000000000325e/head//3 fb3ec83a/rbd_data.10986b8b4567.000000000000325e/head//3(173'2099 client.4289.0:98939 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2099)
2014-06-12 14:51:38.680883 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  18dac83a/rbd_data.10946b8b4567.0000000000002e4c/head//3 18dac83a/rbd_data.10946b8b4567.0000000000002e4c/head//3(173'1974 client.4285.0:92043 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1974)
2014-06-12 14:51:38.680899 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  6e6ac83a/rbd_data.10b36b8b4567.00000000000027d6/head//3 6e6ac83a/rbd_data.10b36b8b4567.00000000000027d6/head//3(173'1773 client.4316.0:82369 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1773)
2014-06-12 14:51:38.680915 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  e439c83a/rbd_data.10b06b8b4567.0000000000003791/head//3 e439c83a/rbd_data.10b06b8b4567.0000000000003791/head//3(173'2293 client.4313.0:111509 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2293)
2014-06-12 14:51:38.680931 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  43c5c83a/rbd_data.108f6b8b4567.0000000000002fe7/head//3 43c5c83a/rbd_data.108f6b8b4567.0000000000002fe7/head//3(173'2027 client.4280.0:94252 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2027)
2014-06-12 14:51:38.680947 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  6a35c83a/rbd_data.10ad6b8b4567.0000000000003928/head//3 6a35c83a/rbd_data.10ad6b8b4567.0000000000003928/head//3(173'2308 client.4310.0:112421 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2308)
2014-06-12 14:51:38.680963 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  55b4c83a/rbd_data.10a56b8b4567.0000000000001bd2/head//3 55b4c83a/rbd_data.10a56b8b4567.0000000000001bd2/head//3(173'1218 client.4302.0:60257 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1218)
2014-06-12 14:51:38.680979 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  b962c83a/rbd_data.10996b8b4567.0000000000002769/head//3 b962c83a/rbd_data.10996b8b4567.0000000000002769/head//3(173'1708 client.4290.0:79348 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1708)
2014-06-12 14:51:38.680995 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ac52c83a/rbd_data.10b06b8b4567.0000000000002bdd/head//3 ac52c83a/rbd_data.10b06b8b4567.0000000000002bdd/head//3(173'1855 client.4313.0:89652 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1855)
2014-06-12 14:51:38.681011 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  3c32c83a/rbd_data.10ad6b8b4567.0000000000002293/head//3 3c32c83a/rbd_data.10ad6b8b4567.0000000000002293/head//3(173'1540 client.4310.0:71183 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1540)
2014-06-12 14:51:38.681027 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  24a1c83a/rbd_data.109b6b8b4567.00000000000027d3/head//3 24a1c83a/rbd_data.109b6b8b4567.00000000000027d3/head//3(173'1841 client.4292.0:79320 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1841)
2014-06-12 14:51:38.681044 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  b93fb83a/rbd_data.10936b8b4567.00000000000008cc/head//3 b93fb83a/rbd_data.10936b8b4567.00000000000008cc/head//3(173'2781 client.4284.0:129141 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2781)
2014-06-12 14:51:38.681060 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  7d3cb83a/rbd_data.10b46b8b4567.0000000000000cc8/head//3 7d3cb83a/rbd_data.10b46b8b4567.0000000000000cc8/head//3(173'2896 client.4317.0:138152 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2896)
2014-06-12 14:51:38.681076 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  23eab83a/rbd_data.10936b8b4567.00000000000015db/head//3 23eab83a/rbd_data.10936b8b4567.00000000000015db/head//3(173'1089 client.4284.0:46779 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1089)
2014-06-12 14:51:38.681089 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:38.681103 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:38.681113 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:38.681116 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:38.681118 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:38.681159 7f9d60bd2700 10 collection_list_partial 3.3a_head start:3f90e83a//0//-1 is_max:0
2014-06-12 14:51:38.681329 7f9d60bd2700 20 objects: [f5f1e83a/rbd_data.10936b8b4567.000000000000322f/head//3,3562e83a/rbd_data.10ab6b8b4567.0000000000000b05/head//3,7c78e83a/rbd_data.10b26b8b4567.00000000000024a9/head//3,db1ce83a/rbd_data.10956b8b4567.0000000000002b6d/head//3,ecfce83a/rbd_data.10986b8b4567.0000000000001cab/head//3,6641f83a/rbd_data.10b46b8b4567.0000000000000632/head//3,5071f83a/rbd_data.10a96b8b4567.0000000000002efc/head//3,c3c4f83a/rbd_data.10a36b8b4567.0000000000001672/head//3,d27df83a/rbd_data.10996b8b4567.0000000000000387/head//3]
2014-06-12 14:51:38.681798 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  requesting scrubmap from osd.21
2014-06-12 14:51:38.681819 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:38.681838 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:38.681849 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:38.681856 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:38.681864 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:38.681871 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_map_chunk [3f90e83a//0//-1,MAX)
2014-06-12 14:51:38.681879 7f9d60bd2700 10 collection_list_partial 3.3a_head start:3f90e83a//0//-1 is_max:0
2014-06-12 14:51:38.682019 7f9d60bd2700 20 objects: [f5f1e83a/rbd_data.10936b8b4567.000000000000322f/head//3,3562e83a/rbd_data.10ab6b8b4567.0000000000000b05/head//3,7c78e83a/rbd_data.10b26b8b4567.00000000000024a9/head//3,db1ce83a/rbd_data.10956b8b4567.0000000000002b6d/head//3,ecfce83a/rbd_data.10986b8b4567.0000000000001cab/head//3,6641f83a/rbd_data.10b46b8b4567.0000000000000632/head//3,5071f83a/rbd_data.10a96b8b4567.0000000000002efc/head//3,c3c4f83a/rbd_data.10a36b8b4567.0000000000001672/head//3,d27df83a/rbd_data.10996b8b4567.0000000000000387/head//3]
2014-06-12 14:51:38.682060 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scan_list scanning 9 objects
2014-06-12 14:51:38.682074 7f9d60bd2700 10 stat 3.3a_head/f5f1e83a/rbd_data.10936b8b4567.000000000000322f/head//3
2014-06-12 14:51:38.682225 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid f5f1e83a/rbd_data.10936b8b4567.000000000000322f/head//3
2014-06-12 14:51:38.694962 7f9d60bd2700 10 getattrs 3.3a_head/f5f1e83a/rbd_data.10936b8b4567.000000000000322f/head//3 = 0
2014-06-12 14:51:38.694973 7f9d60bd2700 10 stat 3.3a_head/3562e83a/rbd_data.10ab6b8b4567.0000000000000b05/head//3
2014-06-12 14:51:38.695103 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 3562e83a/rbd_data.10ab6b8b4567.0000000000000b05/head//3
2014-06-12 14:51:38.700586 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:38.700804 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:38.701069 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:38.726340 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:38.727268 7f9d60bd2700 10 getattrs 3.3a_head/3562e83a/rbd_data.10ab6b8b4567.0000000000000b05/head//3 = 0
2014-06-12 14:51:38.727287 7f9d60bd2700 10 stat 3.3a_head/7c78e83a/rbd_data.10b26b8b4567.00000000000024a9/head//3
2014-06-12 14:51:38.727400 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 7c78e83a/rbd_data.10b26b8b4567.00000000000024a9/head//3
2014-06-12 14:51:38.745129 7f9d60bd2700 10 getattrs 3.3a_head/7c78e83a/rbd_data.10b26b8b4567.00000000000024a9/head//3 = 0
2014-06-12 14:51:38.745148 7f9d60bd2700 10 stat 3.3a_head/db1ce83a/rbd_data.10956b8b4567.0000000000002b6d/head//3
2014-06-12 14:51:38.745307 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid db1ce83a/rbd_data.10956b8b4567.0000000000002b6d/head//3
2014-06-12 14:51:38.765676 7f9d60bd2700 10 getattrs 3.3a_head/db1ce83a/rbd_data.10956b8b4567.0000000000002b6d/head//3 = 0
2014-06-12 14:51:38.765689 7f9d60bd2700 10 stat 3.3a_head/ecfce83a/rbd_data.10986b8b4567.0000000000001cab/head//3
2014-06-12 14:51:38.765860 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid ecfce83a/rbd_data.10986b8b4567.0000000000001cab/head//3
2014-06-12 14:51:38.768000 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:38.785435 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:38.785673 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:38.794467 7f9d60bd2700 10 getattrs 3.3a_head/ecfce83a/rbd_data.10986b8b4567.0000000000001cab/head//3 = 0
2014-06-12 14:51:38.794484 7f9d60bd2700 10 stat 3.3a_head/6641f83a/rbd_data.10b46b8b4567.0000000000000632/head//3
2014-06-12 14:51:38.794632 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 6641f83a/rbd_data.10b46b8b4567.0000000000000632/head//3
2014-06-12 14:51:38.801950 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:38.855055 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:38.855549 7f9d60bd2700 10 getattrs 3.3a_head/6641f83a/rbd_data.10b46b8b4567.0000000000000632/head//3 = 0
2014-06-12 14:51:38.855556 7f9d60bd2700 10 stat 3.3a_head/5071f83a/rbd_data.10a96b8b4567.0000000000002efc/head//3
2014-06-12 14:51:38.855658 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid 5071f83a/rbd_data.10a96b8b4567.0000000000002efc/head//3
2014-06-12 14:51:38.865403 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:38.868777 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:38.869433 7f9d60bd2700 10 getattrs 3.3a_head/5071f83a/rbd_data.10a96b8b4567.0000000000002efc/head//3 = 0
2014-06-12 14:51:38.869442 7f9d60bd2700 10 stat 3.3a_head/c3c4f83a/rbd_data.10a36b8b4567.0000000000001672/head//3
2014-06-12 14:51:38.869531 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid c3c4f83a/rbd_data.10a36b8b4567.0000000000001672/head//3
2014-06-12 14:51:38.904492 7f9d60bd2700 10 getattrs 3.3a_head/c3c4f83a/rbd_data.10a36b8b4567.0000000000001672/head//3 = 0
2014-06-12 14:51:38.904499 7f9d60bd2700 10 stat 3.3a_head/d27df83a/rbd_data.10996b8b4567.0000000000000387/head//3
2014-06-12 14:51:38.904594 7f9d60bd2700 10 lookup_strip_header done  cid 3.3a_head oid d27df83a/rbd_data.10996b8b4567.0000000000000387/head//3
2014-06-12 14:51:38.904695 7f9d60bd2700 10 getattrs 3.3a_head/d27df83a/rbd_data.10996b8b4567.0000000000000387/head//3 = 0
2014-06-12 14:51:38.904700 7f9d60bd2700 10 collection_getattrs 3.3a_head
2014-06-12 14:51:38.904754 7f9d60bd2700 10 collection_getattrs 3.3a_head = 0
2014-06-12 14:51:38.904757 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:38.904787 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:38.904802 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:38.904808 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:38.904811 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:38.906409 7f9d44101700  5 -- op tracker -- , seq: 748593, time: 2014-06-12 14:51:38.906335, event: header_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.906425 7f9d44101700  5 -- op tracker -- , seq: 748593, time: 2014-06-12 14:51:38.906338, event: throttled, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.906430 7f9d44101700  5 -- op tracker -- , seq: 748593, time: 2014-06-12 14:51:38.906382, event: all_read, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.906435 7f9d44101700  5 -- op tracker -- , seq: 748593, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.906449 7f9d44101700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:38.906453 7f9d44101700 20 osd.2 173 should_share_map osd.21 172.16.96.202:6808/1410 173
2014-06-12 14:51:38.906458 7f9d44101700 15 osd.2 173 enqueue_op 0x59dd400 prio 127 cost 3885 latency 0.000123 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:38.906482 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:38.906492 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x59dd400 prio 127 cost 3885 latency 0.000156 osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]
2014-06-12 14:51:38.906548 7f9d6a3e5700  5 -- op tracker -- , seq: 748593, time: 2014-06-12 14:51:38.906537, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.906567 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] handle_message: 0x59dd400
2014-06-12 14:51:38.906593 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:38.906629 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:38.906661 7f9d6a3e5700  5 -- op tracker -- , seq: 748593, time: 2014-06-12 14:51:38.906652, event: started, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.906687 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  got 21 scrub map
2014-06-12 14:51:38.906764 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] map version is 173'2964
2014-06-12 14:51:38.906790 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x59dd400 finish
2014-06-12 14:51:38.906808 7f9d6a3e5700  5 -- op tracker -- , seq: 748593, time: 2014-06-12 14:51:38.906798, event: done, op: osd_sub_op(unknown.0.0:0 3.3a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:38.906827 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2d70000 (1 active)
2014-06-12 14:51:38.906848 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:38.906856 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:38.906841 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:38.906864 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:38.906877 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:38.906907 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:38.906913 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub   osd.2 has 9 items
2014-06-12 14:51:38.906919 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub replica 21 has 9 items
2014-06-12 14:51:38.906929 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f5f1e83a/rbd_data.10936b8b4567.000000000000322f/head//3, auth == maps.end()
2014-06-12 14:51:38.906950 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj f5f1e83a/rbd_data.10936b8b4567.000000000000322f/head//3
2014-06-12 14:51:38.906962 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3562e83a/rbd_data.10ab6b8b4567.0000000000000b05/head//3, auth == maps.end()
2014-06-12 14:51:38.906970 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 3562e83a/rbd_data.10ab6b8b4567.0000000000000b05/head//3
2014-06-12 14:51:38.906979 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7c78e83a/rbd_data.10b26b8b4567.00000000000024a9/head//3, auth == maps.end()
2014-06-12 14:51:38.906988 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 7c78e83a/rbd_data.10b26b8b4567.00000000000024a9/head//3
2014-06-12 14:51:38.906997 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj db1ce83a/rbd_data.10956b8b4567.0000000000002b6d/head//3, auth == maps.end()
2014-06-12 14:51:38.907005 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj db1ce83a/rbd_data.10956b8b4567.0000000000002b6d/head//3
2014-06-12 14:51:38.907014 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ecfce83a/rbd_data.10986b8b4567.0000000000001cab/head//3, auth == maps.end()
2014-06-12 14:51:38.907022 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj ecfce83a/rbd_data.10986b8b4567.0000000000001cab/head//3
2014-06-12 14:51:38.907031 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6641f83a/rbd_data.10b46b8b4567.0000000000000632/head//3, auth == maps.end()
2014-06-12 14:51:38.907039 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 6641f83a/rbd_data.10b46b8b4567.0000000000000632/head//3
2014-06-12 14:51:38.907047 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5071f83a/rbd_data.10a96b8b4567.0000000000002efc/head//3, auth == maps.end()
2014-06-12 14:51:38.907055 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj 5071f83a/rbd_data.10a96b8b4567.0000000000002efc/head//3
2014-06-12 14:51:38.907064 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c3c4f83a/rbd_data.10a36b8b4567.0000000000001672/head//3, auth == maps.end()
2014-06-12 14:51:38.907072 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj c3c4f83a/rbd_data.10a36b8b4567.0000000000001672/head//3
2014-06-12 14:51:38.907080 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d27df83a/rbd_data.10996b8b4567.0000000000000387/head//3, auth == maps.end()
2014-06-12 14:51:38.907089 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] be_select_auth_object: selecting osd 21 for obj d27df83a/rbd_data.10996b8b4567.0000000000000387/head//3
2014-06-12 14:51:38.907098 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] 
2014-06-12 14:51:38.907104 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub
2014-06-12 14:51:38.907115 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  d27df83a/rbd_data.10996b8b4567.0000000000000387/head//3 d27df83a/rbd_data.10996b8b4567.0000000000000387/head//3(173'2561 client.4290.0:121442 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2561)
2014-06-12 14:51:38.907126 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  c3c4f83a/rbd_data.10a36b8b4567.0000000000001672/head//3 c3c4f83a/rbd_data.10a36b8b4567.0000000000001672/head//3(173'1073 client.4300.0:48799 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1073)
2014-06-12 14:51:38.907137 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  5071f83a/rbd_data.10a96b8b4567.0000000000002efc/head//3 5071f83a/rbd_data.10a96b8b4567.0000000000002efc/head//3(173'2138 client.4306.0:96616 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2138)
2014-06-12 14:51:38.907147 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  6641f83a/rbd_data.10b46b8b4567.0000000000000632/head//3 6641f83a/rbd_data.10b46b8b4567.0000000000000632/head//3(173'2673 client.4317.0:126675 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2673)
2014-06-12 14:51:38.907158 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  ecfce83a/rbd_data.10986b8b4567.0000000000001cab/head//3 ecfce83a/rbd_data.10986b8b4567.0000000000001cab/head//3(173'1246 client.4289.0:60109 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1246)
2014-06-12 14:51:38.907168 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  db1ce83a/rbd_data.10956b8b4567.0000000000002b6d/head//3 db1ce83a/rbd_data.10956b8b4567.0000000000002b6d/head//3(173'1925 client.4286.0:88132 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1925)
2014-06-12 14:51:38.907179 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  7c78e83a/rbd_data.10b26b8b4567.00000000000024a9/head//3 7c78e83a/rbd_data.10b26b8b4567.00000000000024a9/head//3(173'1467 client.4315.0:75602 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1467)
2014-06-12 14:51:38.907189 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  3562e83a/rbd_data.10ab6b8b4567.0000000000000b05/head//3 3562e83a/rbd_data.10ab6b8b4567.0000000000000b05/head//3(173'2859 client.4308.0:137548 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2859)
2014-06-12 14:51:38.907199 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub  f5f1e83a/rbd_data.10936b8b4567.000000000000322f/head//3 f5f1e83a/rbd_data.10936b8b4567.000000000000322f/head//3(173'2114 client.4284.0:98043 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2114)
2014-06-12 14:51:38.907208 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:38.907215 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:38.907221 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub state FINISH
2014-06-12 14:51:38.907226 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] scrub got 284/295 objects, 0/0 clones, 284/295 dirty, 0/0 omap, 0/0 hit_set_archive, 1191182336/1237319680 bytes.
2014-06-12 14:51:38.907238 7f9d60bd2700  0 log [ERR] : 3.3a scrub stat mismatch, got 284/295 objects, 0/0 clones, 284/295 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts, 1191182336/1237319680 bytes.
2014-06-12 14:51:38.907240 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+scrubbing] process_inconsistent() checking authoritative
2014-06-12 14:51:38.907246 7f9d60bd2700  0 log [ERR] : 3.3a scrub 1 errors
2014-06-12 14:51:38.907282 7f9d60bd2700 10 log is not dirty
2014-06-12 14:51:38.907287 7f9d60bd2700  5 queue_transactions existing osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:51:38.907290 7f9d60bd2700  5 queue_transactions (trailing journal)  
2014-06-12 14:51:38.907291 7f9d60bd2700  5 queue_op 0x1283a120 seq 2628 osr(3.3a 0x276cce0) 826 bytes   (queue has 1 ops and 826 bytes)
2014-06-12 14:51:38.907297 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+inconsistent] publish_stats_to_osd 173:3849
2014-06-12 14:51:38.907313 7f9d60bd2700 20 osd.2 173 dec_scrubs_active 1 -> 0 (max 1, pending 0)
2014-06-12 14:51:38.907314 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+inconsistent]  requeue_ops 
2014-06-12 14:51:38.907320 7f9d8116f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53b70e0 (1 active)
2014-06-12 14:51:38.907324 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+inconsistent] scrub requesting unreserve from osd.21
2014-06-12 14:51:38.907328 7f9d8116f700  5 _do_op 0x1283a120 seq 2628 osr(3.3a 0x276cce0)/0x276cce0 start
2014-06-12 14:51:38.907340 7f9d8116f700 10 _do_transaction on 0x5352d80
2014-06-12 14:51:38.907345 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:38.907349 7f9d8116f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:51:38.907367 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.3a( v 173'2964 (0'0,173'2964] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2963 mlcod 173'2963 active+clean+inconsistent] share_pg_info
2014-06-12 14:51:38.907396 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:38.907406 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2d70000 (0 active)
2014-06-12 14:51:38.907408 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:38.907436 7f9d8116f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:51:38.907498 7f9d8116f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:51:38.907525 7f9d8116f700  5 submit_transaction r = 0
2014-06-12 14:51:38.907562 7f9d8116f700 10 _do_op 0x1283a120 seq 2628 r = 0, finisher 0x54edd50 0
2014-06-12 14:51:38.907570 7f9d8116f700 10 _finish_op 0x1283a120 seq 2628 osr(3.3a 0x276cce0)/0x276cce0
2014-06-12 14:51:38.907585 7f9d8116f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53b70e0 (0 active)
2014-06-12 14:51:38.907591 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:38.962320 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:38.966050 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:39.121349 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:39.132981 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:39.142429 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:39.151855 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:39.180303 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:39.219050 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:39.298466 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:39.316556 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:39.341641 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:39.374908 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:39.471184 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:39.471570 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:39.513410 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:39.518462 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:39.520448 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:39.520467 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:51:39.520471 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:51:39.520475 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:51:39.520478 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:51:39.520518 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:39.520522 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:39.520524 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:39.520557 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:39.547361 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:39.694869 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:39.694997 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:39.959824 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:39.959962 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:39.977054 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:40.032732 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:40.059377 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:40.059381 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:40.059383 7f9d74bfa700 20 osd.2 173 _dispatch 0x12797a40 pg_stats_ack(1 pgs tid 4871) v1
2014-06-12 14:51:40.059386 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:51:40.059392 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:40.059394 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:40.064702 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:40.078121 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:40.078252 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:40.167560 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:40.194051 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:40.316980 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:40.326393 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:40.366340 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:40.371252 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:40.382017 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:40.413736 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:40.414023 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:40.486244 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:40.486328 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:40.513227 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:40.519059 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:40.520599 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:40.520623 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:40.520627 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:40.520630 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:40.520669 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:40.523001 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:40.532441 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:40.547883 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:40.661797 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:40.679995 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:40.700660 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:40.726416 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:40.768080 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:40.802029 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:40.855141 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:40.865497 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:40.868868 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:40.906945 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:40.907480 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:40.907668 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:40.962409 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:40.966136 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:41.086267 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:41.086419 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:41.121426 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:41.133077 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:41.142503 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:41.151952 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:41.180399 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:41.219148 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:41.298562 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:41.316655 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:41.341740 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:41.375005 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:41.513509 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:41.518563 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:41.520703 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:41.520728 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:41.520733 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:41.520735 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:41.520774 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:41.530626 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:41.531012 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:41.547459 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:41.722353 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:41.722444 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:41.977152 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:41.998129 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:41.998519 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:42.032822 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:42.064783 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:42.114541 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:42.114878 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:42.167656 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:42.187166 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:42.187226 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:42.194136 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:42.201512 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:42.201609 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:42.317062 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:42.326464 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:42.366433 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:42.371354 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:42.382100 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:42.513313 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:42.519158 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:42.520812 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:42.520859 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.46 < max 0.5 = yes
2014-06-12 14:51:42.520873 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=1
2014-06-12 14:51:42.520878 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.43 at 2014-06-11 13:02:32.501298
2014-06-12 14:51:42.520883 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:42.520891 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 -> 1 (max 1, active 0)
2014-06-12 14:51:42.520894 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean] sched_scrub: reserved locally, reserving replicas
2014-06-12 14:51:42.520921 7f9d84175700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean] scrub requesting reserve from osd.24
2014-06-12 14:51:42.520941 7f9d84175700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:42.520977 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean] sched_scrub: reserved 2, waiting for replicas
2014-06-12 14:51:42.520991 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:42.520993 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:42.520995 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:42.521906 7f9d5a16f700  5 -- op tracker -- , seq: 748594, time: 2014-06-12 14:51:42.521840, event: header_read, op: osd_sub_op_reply(unknown.0.0:0 3.43 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:42.521923 7f9d5a16f700  5 -- op tracker -- , seq: 748594, time: 2014-06-12 14:51:42.521843, event: throttled, op: osd_sub_op_reply(unknown.0.0:0 3.43 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:42.521933 7f9d5a16f700  5 -- op tracker -- , seq: 748594, time: 2014-06-12 14:51:42.521863, event: all_read, op: osd_sub_op_reply(unknown.0.0:0 3.43 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:42.521964 7f9d5a16f700  5 -- op tracker -- , seq: 748594, time: 0.000000, event: dispatched, op: osd_sub_op_reply(unknown.0.0:0 3.43 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:42.521971 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op_reply(unknown.0.0:0 3.43 0//0//3 [scrub-reserve] ack, result = 0) v2 epoch 173
2014-06-12 14:51:42.521978 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:51:42.521987 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a902d00 prio 127 cost 1 latency 0.000146 osd_sub_op_reply(unknown.0.0:0 3.43 0//0//3 [scrub-reserve] ack, result = 0) v2
2014-06-12 14:51:42.522053 7f9d663dd700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:42.522074 7f9d663dd700 10 osd.2 173 dequeue_op 0x2a902d00 prio 127 cost 1 latency 0.000233 osd_sub_op_reply(unknown.0.0:0 3.43 0//0//3 [scrub-reserve] ack, result = 0) v2 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean]
2014-06-12 14:51:42.522113 7f9d663dd700  5 -- op tracker -- , seq: 748594, time: 2014-06-12 14:51:42.522108, event: reached_pg, op: osd_sub_op_reply(unknown.0.0:0 3.43 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:42.522133 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean] handle_message: 0x2a902d00
2014-06-12 14:51:42.522152 7f9d663dd700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean] sub_op_scrub_reserve_reply
2014-06-12 14:51:42.522170 7f9d663dd700  5 -- op tracker -- , seq: 748594, time: 2014-06-12 14:51:42.522166, event: started, op: osd_sub_op_reply(unknown.0.0:0 3.43 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:42.522178 7f9d663dd700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean]  osd.24 scrub reserve = success
2014-06-12 14:51:42.522192 7f9d663dd700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean] sched_scrub: success, reserved self and replicas
2014-06-12 14:51:42.522214 7f9d663dd700 10 osd.2 173 dequeue_op 0x2a902d00 finish
2014-06-12 14:51:42.522223 7f9d663dd700  5 -- op tracker -- , seq: 748594, time: 2014-06-12 14:51:42.522219, event: done, op: osd_sub_op_reply(unknown.0.0:0 3.43 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:42.522242 7f9d663dd700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:42.522246 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:42.522252 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:42.522272 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] starting a new chunky scrub
2014-06-12 14:51:42.522321 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state INACTIVE
2014-06-12 14:51:42.522336 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub start
2014-06-12 14:51:42.522351 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] publish_stats_to_osd 173:2748
2014-06-12 14:51:42.522364 7f9d60bd2700 20 osd.2 173 inc_scrubs_active 0 -> 1 (max 1, pending 1 -> 0)
2014-06-12 14:51:42.522368 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:42.522384 7f9d60bd2700 10 collection_list_partial 3.43_head start:0//0//-1 is_max:0
2014-06-12 14:51:42.523081 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:42.523163 7f9d60bd2700 20 objects: [2fa30043/rbd_data.108d6b8b4567.0000000000000812/head//3,94b30043/rbd_data.10a96b8b4567.0000000000000c48/head//3,55640043/rbd_data.109e6b8b4567.0000000000001d7e/head//3,46670043/rbd_data.109f6b8b4567.0000000000001ba2/head//3,56770043/rbd_data.10ae6b8b4567.0000000000000e5f/head//3,a0a70043/rbd_data.10966b8b4567.00000000000029cd/head//3,81d70043/rbd_data.10af6b8b4567.00000000000013e4/head//3,b5d80043/rbd_data.10ae6b8b4567.000000000000146f/head//3,41b90043/rbd_data.108e6b8b4567.0000000000003449/head//3,5d2b0043/rbd_data.10af6b8b4567.00000000000008ee/head//3,d8cd0043/rbd_data.108f6b8b4567.0000000000000933/head//3,e71e0043/rbd_data.10906b8b4567.0000000000002eeb/head//3,5aee0043/rbd_data.10b46b8b4567.0000000000003367/head//3,2d611043/rbd_data.10ab6b8b4567.0000000000000914/head//3,e3711043/rbd_data.10906b8b4567.000000000000304f/head//3,cd261043/rbd_data.10a66b8b4567.000000000000063e/head//3,dbc61043/rbd_data.10906b8b4567.0000000000001224/head//3,86f71043/rbd_data.10a96b8b4567.0000000000002178/head//3,9f7b1043/rbd_data.10a66b8b4567.0000000000001779/head//3,3b922043/rbd_data.10906b8b4567.0000000000001c27/head//3,46552043/rbd_data.10906b8b4567.0000000000001090/head//3,41662043/rbd_data.10a26b8b4567.00000000000007e2/head//3,2cc72043/rbd_data.10a16b8b4567.0000000000003068/head//3,65f82043/rbd_data.10b26b8b4567.0000000000003a6b/head//3,95ab2043/rbd_data.10936b8b4567.0000000000003b90/head//3]
2014-06-12 14:51:42.523822 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  requesting scrubmap from osd.24
2014-06-12 14:51:42.523869 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:42.523896 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:42.523910 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:42.523920 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:42.523931 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:42.523942 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_map_chunk [0//0//-1,76cb2043//0//-1)
2014-06-12 14:51:42.523954 7f9d60bd2700 10 collection_list_partial 3.43_head start:0//0//-1 is_max:0
2014-06-12 14:51:42.524997 7f9d60bd2700 20 objects: [2fa30043/rbd_data.108d6b8b4567.0000000000000812/head//3,94b30043/rbd_data.10a96b8b4567.0000000000000c48/head//3,55640043/rbd_data.109e6b8b4567.0000000000001d7e/head//3,46670043/rbd_data.109f6b8b4567.0000000000001ba2/head//3,56770043/rbd_data.10ae6b8b4567.0000000000000e5f/head//3,a0a70043/rbd_data.10966b8b4567.00000000000029cd/head//3,81d70043/rbd_data.10af6b8b4567.00000000000013e4/head//3,b5d80043/rbd_data.10ae6b8b4567.000000000000146f/head//3,41b90043/rbd_data.108e6b8b4567.0000000000003449/head//3,5d2b0043/rbd_data.10af6b8b4567.00000000000008ee/head//3,d8cd0043/rbd_data.108f6b8b4567.0000000000000933/head//3,e71e0043/rbd_data.10906b8b4567.0000000000002eeb/head//3,5aee0043/rbd_data.10b46b8b4567.0000000000003367/head//3,2d611043/rbd_data.10ab6b8b4567.0000000000000914/head//3,e3711043/rbd_data.10906b8b4567.000000000000304f/head//3,cd261043/rbd_data.10a66b8b4567.000000000000063e/head//3,dbc61043/rbd_data.10906b8b4567.0000000000001224/head//3,86f71043/rbd_data.10a96b8b4567.0000000000002178/head//3,9f7b1043/rbd_data.10a66b8b4567.0000000000001779/head//3,3b922043/rbd_data.10906b8b4567.0000000000001c27/head//3,46552043/rbd_data.10906b8b4567.0000000000001090/head//3,41662043/rbd_data.10a26b8b4567.00000000000007e2/head//3,2cc72043/rbd_data.10a16b8b4567.0000000000003068/head//3,65f82043/rbd_data.10b26b8b4567.0000000000003a6b/head//3,95ab2043/rbd_data.10936b8b4567.0000000000003b90/head//3,76cb2043/rbd_data.108e6b8b4567.0000000000002864/head//3,3d3c2043/rbd_data.10986b8b4567.00000000000010d8/head//3,d89d2043/rbd_data.108e6b8b4567.0000000000001fc5/head//3,195e2043/rbd_data.10a46b8b4567.0000000000000cd0/head//3,95e23043/rbd_data.108d6b8b4567.000000000000348f/head//3,2f733043/rbd_data.109d6b8b4567.0000000000002ffe/head//3,41043043/rbd_data.10a46b8b4567.0000000000000b8d/head//3,2e553043/rbd_data.10a26b8b4567.000000000000333f/head//3,afa63043/rbd_data.10a16b8b4567.0000000000000ac6/head//3,aec73043/rbd_data.10a36b8b4567.000000000000026c/head//3,d2583043/rbd_data.10ab6b8b4567.0000000000002fab/head//3,f1493043/rbd_data.10a06b8b4567.0000000000002bb0/head//3,c3e93043/rbd_data.10976b8b4567.00000000000038a2/head//3,303b3043/rbd_data.10936b8b4567.0000000000002e76/head//3,2a9b3043/rbd_data.10986b8b4567.0000000000001c55/head//3,ffbd3043/rbd_data.109c6b8b4567.0000000000002ea0/head//3,2f1f3043/rbd_data.10906b8b4567.00000000000004fb/head//3,65014043/rbd_data.109b6b8b4567.0000000000000270/head//3,fde34043/rbd_data.10936b8b4567.00000000000015b5/head//3,15d54043/rbd_data.10af6b8b4567.000000000000328f/head//3,7cb64043/rbd_data.10b46b8b4567.000000000000382f/head//3,547a4043/rbd_data.108e6b8b4567.000000000000004f/head//3,750b4043/rbd_data.10996b8b4567.0000000000001f83/head//3,3d9d4043/rbd_data.10a76b8b4567.000000000000255d/head//3,d5815043/rbd_data.10a76b8b4567.0000000000001858/head//3,2e345043/rbd_data.10a86b8b4567.0000000000001cd7/head//3,a9a55043/rbd_data.10ae6b8b4567.0000000000000daa/head//3,63085043/rbd_data.108d6b8b4567.0000000000003728/head//3,68a95043/rbd_data.10ac6b8b4567.0000000000000aaf/head//3,efbf5043/rbd_data.10b06b8b4567.000000000000338e/head//3,4d436043/rbd_data.109e6b8b4567.00000000000039f8/head//3,b7a66043/rbd_data.10926b8b4567.00000000000021e8/head//3,82d76043/rbd_data.10926b8b4567.00000000000001be/head//3,f1a86043/rbd_data.10b16b8b4567.0000000000001c03/head//3,b1d96043/rbd_data.109c6b8b4567.0000000000000eef/head//3,578b6043/rbd_data.10ad6b8b4567.00000000000000d8/head//3,d9b6043/rbd_data.10906b8b4567.0000000000000124/head//3,4a1f6043/rbd_data.10936b8b4567.0000000000003bf0/head//3,50217043/rbd_data.109c6b8b4567.0000000000001d6c/head//3]
2014-06-12 14:51:42.525135 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:42.525163 7f9d60bd2700 10 stat 3.43_head/2fa30043/rbd_data.108d6b8b4567.0000000000000812/head//3
2014-06-12 14:51:42.525378 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 2fa30043/rbd_data.108d6b8b4567.0000000000000812/head//3
2014-06-12 14:51:42.532534 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:42.559448 7f9d60bd2700 10 getattrs 3.43_head/2fa30043/rbd_data.108d6b8b4567.0000000000000812/head//3 = 0
2014-06-12 14:51:42.559477 7f9d60bd2700 10 stat 3.43_head/94b30043/rbd_data.10a96b8b4567.0000000000000c48/head//3
2014-06-12 14:51:42.559662 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 94b30043/rbd_data.10a96b8b4567.0000000000000c48/head//3
2014-06-12 14:51:42.609344 7f9d60bd2700 10 getattrs 3.43_head/94b30043/rbd_data.10a96b8b4567.0000000000000c48/head//3 = 0
2014-06-12 14:51:42.609354 7f9d60bd2700 10 stat 3.43_head/55640043/rbd_data.109e6b8b4567.0000000000001d7e/head//3
2014-06-12 14:51:42.609467 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 55640043/rbd_data.109e6b8b4567.0000000000001d7e/head//3
2014-06-12 14:51:42.643744 7f9d60bd2700 10 getattrs 3.43_head/55640043/rbd_data.109e6b8b4567.0000000000001d7e/head//3 = 0
2014-06-12 14:51:42.643754 7f9d60bd2700 10 stat 3.43_head/46670043/rbd_data.109f6b8b4567.0000000000001ba2/head//3
2014-06-12 14:51:42.643853 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 46670043/rbd_data.109f6b8b4567.0000000000001ba2/head//3
2014-06-12 14:51:42.661882 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:42.680080 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:42.681013 7f9d60bd2700 10 getattrs 3.43_head/46670043/rbd_data.109f6b8b4567.0000000000001ba2/head//3 = 0
2014-06-12 14:51:42.681025 7f9d60bd2700 10 stat 3.43_head/56770043/rbd_data.10ae6b8b4567.0000000000000e5f/head//3
2014-06-12 14:51:42.681133 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 56770043/rbd_data.10ae6b8b4567.0000000000000e5f/head//3
2014-06-12 14:51:42.692276 7f9d60bd2700 10 getattrs 3.43_head/56770043/rbd_data.10ae6b8b4567.0000000000000e5f/head//3 = 0
2014-06-12 14:51:42.692288 7f9d60bd2700 10 stat 3.43_head/a0a70043/rbd_data.10966b8b4567.00000000000029cd/head//3
2014-06-12 14:51:42.692394 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid a0a70043/rbd_data.10966b8b4567.00000000000029cd/head//3
2014-06-12 14:51:42.700734 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:42.709154 7f9d60bd2700 10 getattrs 3.43_head/a0a70043/rbd_data.10966b8b4567.00000000000029cd/head//3 = 0
2014-06-12 14:51:42.709163 7f9d60bd2700 10 stat 3.43_head/81d70043/rbd_data.10af6b8b4567.00000000000013e4/head//3
2014-06-12 14:51:42.709273 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 81d70043/rbd_data.10af6b8b4567.00000000000013e4/head//3
2014-06-12 14:51:42.726505 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:42.729502 7f9d60bd2700 10 getattrs 3.43_head/81d70043/rbd_data.10af6b8b4567.00000000000013e4/head//3 = 0
2014-06-12 14:51:42.729521 7f9d60bd2700 10 stat 3.43_head/b5d80043/rbd_data.10ae6b8b4567.000000000000146f/head//3
2014-06-12 14:51:42.729659 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid b5d80043/rbd_data.10ae6b8b4567.000000000000146f/head//3
2014-06-12 14:51:42.753699 7f9d60bd2700 10 getattrs 3.43_head/b5d80043/rbd_data.10ae6b8b4567.000000000000146f/head//3 = 0
2014-06-12 14:51:42.753714 7f9d60bd2700 10 stat 3.43_head/41b90043/rbd_data.108e6b8b4567.0000000000003449/head//3
2014-06-12 14:51:42.753827 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 41b90043/rbd_data.108e6b8b4567.0000000000003449/head//3
2014-06-12 14:51:42.768155 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:42.771682 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:42.771950 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:42.787802 7f9d60bd2700 10 getattrs 3.43_head/41b90043/rbd_data.108e6b8b4567.0000000000003449/head//3 = 0
2014-06-12 14:51:42.787810 7f9d60bd2700 10 stat 3.43_head/5d2b0043/rbd_data.10af6b8b4567.00000000000008ee/head//3
2014-06-12 14:51:42.787917 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 5d2b0043/rbd_data.10af6b8b4567.00000000000008ee/head//3
2014-06-12 14:51:42.791457 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:42.791483 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:42.792037 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:42.792211 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:42.792342 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:42.792382 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:42.792436 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:42.792478 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:42.792524 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:42.792566 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:42.792613 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:42.792659 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:42.792705 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:42.792753 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:42.792797 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:42.792845 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:42.792890 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:42.792936 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:42.792981 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:42.793027 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:42.793072 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:42.793120 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:42.793165 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:42.793206 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:42.802106 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:42.836972 7f9d60bd2700 10 getattrs 3.43_head/5d2b0043/rbd_data.10af6b8b4567.00000000000008ee/head//3 = 0
2014-06-12 14:51:42.836983 7f9d60bd2700 10 stat 3.43_head/d8cd0043/rbd_data.108f6b8b4567.0000000000000933/head//3
2014-06-12 14:51:42.837097 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d8cd0043/rbd_data.108f6b8b4567.0000000000000933/head//3
2014-06-12 14:51:42.855224 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:42.865594 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:42.868951 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:42.890205 7f9d60bd2700 10 getattrs 3.43_head/d8cd0043/rbd_data.108f6b8b4567.0000000000000933/head//3 = 0
2014-06-12 14:51:42.890219 7f9d60bd2700 10 stat 3.43_head/e71e0043/rbd_data.10906b8b4567.0000000000002eeb/head//3
2014-06-12 14:51:42.890354 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid e71e0043/rbd_data.10906b8b4567.0000000000002eeb/head//3
2014-06-12 14:51:42.905217 7f9d60bd2700 10 getattrs 3.43_head/e71e0043/rbd_data.10906b8b4567.0000000000002eeb/head//3 = 0
2014-06-12 14:51:42.905228 7f9d60bd2700 10 stat 3.43_head/5aee0043/rbd_data.10b46b8b4567.0000000000003367/head//3
2014-06-12 14:51:42.905335 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 5aee0043/rbd_data.10b46b8b4567.0000000000003367/head//3
2014-06-12 14:51:42.907042 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:42.907747 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:42.935019 7f9d60bd2700 10 getattrs 3.43_head/5aee0043/rbd_data.10b46b8b4567.0000000000003367/head//3 = 0
2014-06-12 14:51:42.935033 7f9d60bd2700 10 stat 3.43_head/2d611043/rbd_data.10ab6b8b4567.0000000000000914/head//3
2014-06-12 14:51:42.935150 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 2d611043/rbd_data.10ab6b8b4567.0000000000000914/head//3
2014-06-12 14:51:42.962493 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:42.966209 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:42.979533 7f9d60bd2700 10 getattrs 3.43_head/2d611043/rbd_data.10ab6b8b4567.0000000000000914/head//3 = 0
2014-06-12 14:51:42.979559 7f9d60bd2700 10 stat 3.43_head/e3711043/rbd_data.10906b8b4567.000000000000304f/head//3
2014-06-12 14:51:42.979747 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid e3711043/rbd_data.10906b8b4567.000000000000304f/head//3
2014-06-12 14:51:42.991814 7f9d60bd2700 10 getattrs 3.43_head/e3711043/rbd_data.10906b8b4567.000000000000304f/head//3 = 0
2014-06-12 14:51:42.991841 7f9d60bd2700 10 stat 3.43_head/cd261043/rbd_data.10a66b8b4567.000000000000063e/head//3
2014-06-12 14:51:42.992033 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid cd261043/rbd_data.10a66b8b4567.000000000000063e/head//3
2014-06-12 14:51:43.019556 7f9d60bd2700 10 getattrs 3.43_head/cd261043/rbd_data.10a66b8b4567.000000000000063e/head//3 = 0
2014-06-12 14:51:43.019571 7f9d60bd2700 10 stat 3.43_head/dbc61043/rbd_data.10906b8b4567.0000000000001224/head//3
2014-06-12 14:51:43.019690 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid dbc61043/rbd_data.10906b8b4567.0000000000001224/head//3
2014-06-12 14:51:43.048230 7f9d60bd2700 10 getattrs 3.43_head/dbc61043/rbd_data.10906b8b4567.0000000000001224/head//3 = 0
2014-06-12 14:51:43.048243 7f9d60bd2700 10 stat 3.43_head/86f71043/rbd_data.10a96b8b4567.0000000000002178/head//3
2014-06-12 14:51:43.048364 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 86f71043/rbd_data.10a96b8b4567.0000000000002178/head//3
2014-06-12 14:51:43.060102 7f9d60bd2700 10 getattrs 3.43_head/86f71043/rbd_data.10a96b8b4567.0000000000002178/head//3 = 0
2014-06-12 14:51:43.060113 7f9d60bd2700 10 stat 3.43_head/9f7b1043/rbd_data.10a66b8b4567.0000000000001779/head//3
2014-06-12 14:51:43.060232 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9f7b1043/rbd_data.10a66b8b4567.0000000000001779/head//3
2014-06-12 14:51:43.070288 7f9d60bd2700 10 getattrs 3.43_head/9f7b1043/rbd_data.10a66b8b4567.0000000000001779/head//3 = 0
2014-06-12 14:51:43.070299 7f9d60bd2700 10 stat 3.43_head/3b922043/rbd_data.10906b8b4567.0000000000001c27/head//3
2014-06-12 14:51:43.070418 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 3b922043/rbd_data.10906b8b4567.0000000000001c27/head//3
2014-06-12 14:51:43.081355 7f9d60bd2700 10 getattrs 3.43_head/3b922043/rbd_data.10906b8b4567.0000000000001c27/head//3 = 0
2014-06-12 14:51:43.081366 7f9d60bd2700 10 stat 3.43_head/46552043/rbd_data.10906b8b4567.0000000000001090/head//3
2014-06-12 14:51:43.081484 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 46552043/rbd_data.10906b8b4567.0000000000001090/head//3
2014-06-12 14:51:43.082491 7f9d60bd2700 10 getattrs 3.43_head/46552043/rbd_data.10906b8b4567.0000000000001090/head//3 = 0
2014-06-12 14:51:43.082509 7f9d60bd2700 10 stat 3.43_head/41662043/rbd_data.10a26b8b4567.00000000000007e2/head//3
2014-06-12 14:51:43.082625 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 41662043/rbd_data.10a26b8b4567.00000000000007e2/head//3
2014-06-12 14:51:43.107086 7f9d60bd2700 10 getattrs 3.43_head/41662043/rbd_data.10a26b8b4567.00000000000007e2/head//3 = 0
2014-06-12 14:51:43.107097 7f9d60bd2700 10 stat 3.43_head/2cc72043/rbd_data.10a16b8b4567.0000000000003068/head//3
2014-06-12 14:51:43.107195 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 2cc72043/rbd_data.10a16b8b4567.0000000000003068/head//3
2014-06-12 14:51:43.121534 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:43.131923 7f9d60bd2700 10 getattrs 3.43_head/2cc72043/rbd_data.10a16b8b4567.0000000000003068/head//3 = 0
2014-06-12 14:51:43.131937 7f9d60bd2700 10 stat 3.43_head/65f82043/rbd_data.10b26b8b4567.0000000000003a6b/head//3
2014-06-12 14:51:43.132096 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 65f82043/rbd_data.10b26b8b4567.0000000000003a6b/head//3
2014-06-12 14:51:43.133164 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:43.142582 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:43.152033 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:43.159372 7f9d60bd2700 10 getattrs 3.43_head/65f82043/rbd_data.10b26b8b4567.0000000000003a6b/head//3 = 0
2014-06-12 14:51:43.159399 7f9d60bd2700 10 stat 3.43_head/95ab2043/rbd_data.10936b8b4567.0000000000003b90/head//3
2014-06-12 14:51:43.159567 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 95ab2043/rbd_data.10936b8b4567.0000000000003b90/head//3
2014-06-12 14:51:43.159712 7f9d60bd2700 10 getattrs 3.43_head/95ab2043/rbd_data.10936b8b4567.0000000000003b90/head//3 = 0
2014-06-12 14:51:43.159724 7f9d60bd2700 10 collection_getattrs 3.43_head
2014-06-12 14:51:43.159790 7f9d60bd2700 10 collection_getattrs 3.43_head = 0
2014-06-12 14:51:43.159793 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:43.159840 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:43.159855 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:43.159866 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:43.159870 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:43.168645 7f9d5a16f700  5 -- op tracker -- , seq: 748595, time: 2014-06-12 14:51:43.168537, event: header_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.168661 7f9d5a16f700  5 -- op tracker -- , seq: 748595, time: 2014-06-12 14:51:43.168541, event: throttled, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.168670 7f9d5a16f700  5 -- op tracker -- , seq: 748595, time: 2014-06-12 14:51:43.168601, event: all_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.168698 7f9d5a16f700  5 -- op tracker -- , seq: 748595, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.168704 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:43.168712 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:51:43.168721 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a902800 prio 127 cost 10717 latency 0.000184 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:43.168795 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:43.168817 7f9d623d5700 10 osd.2 173 dequeue_op 0x2a902800 prio 127 cost 10717 latency 0.000280 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]
2014-06-12 14:51:43.168857 7f9d623d5700  5 -- op tracker -- , seq: 748595, time: 2014-06-12 14:51:43.168850, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.168875 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] handle_message: 0x2a902800
2014-06-12 14:51:43.168890 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:43.168910 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:43.168929 7f9d623d5700  5 -- op tracker -- , seq: 748595, time: 2014-06-12 14:51:43.168923, event: started, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.168936 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  got 24 scrub map
2014-06-12 14:51:43.169037 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] map version is 173'2566
2014-06-12 14:51:43.169057 7f9d623d5700 10 osd.2 173 dequeue_op 0x2a902800 finish
2014-06-12 14:51:43.169068 7f9d623d5700  5 -- op tracker -- , seq: 748595, time: 2014-06-12 14:51:43.169062, event: done, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.169093 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:43.169099 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:43.169096 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:43.169110 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:43.169138 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:43.169149 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:43.169183 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:43.169194 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:43.169204 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub replica 24 has 25 items
2014-06-12 14:51:43.169235 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2fa30043/rbd_data.108d6b8b4567.0000000000000812/head//3, auth == maps.end()
2014-06-12 14:51:43.169254 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 2fa30043/rbd_data.108d6b8b4567.0000000000000812/head//3
2014-06-12 14:51:43.169279 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 94b30043/rbd_data.10a96b8b4567.0000000000000c48/head//3, auth == maps.end()
2014-06-12 14:51:43.169296 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 94b30043/rbd_data.10a96b8b4567.0000000000000c48/head//3
2014-06-12 14:51:43.169312 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 55640043/rbd_data.109e6b8b4567.0000000000001d7e/head//3, auth == maps.end()
2014-06-12 14:51:43.169326 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 55640043/rbd_data.109e6b8b4567.0000000000001d7e/head//3
2014-06-12 14:51:43.169340 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 46670043/rbd_data.109f6b8b4567.0000000000001ba2/head//3, auth == maps.end()
2014-06-12 14:51:43.169356 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 46670043/rbd_data.109f6b8b4567.0000000000001ba2/head//3
2014-06-12 14:51:43.169372 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 56770043/rbd_data.10ae6b8b4567.0000000000000e5f/head//3, auth == maps.end()
2014-06-12 14:51:43.169386 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 56770043/rbd_data.10ae6b8b4567.0000000000000e5f/head//3
2014-06-12 14:51:43.169400 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a0a70043/rbd_data.10966b8b4567.00000000000029cd/head//3, auth == maps.end()
2014-06-12 14:51:43.169414 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj a0a70043/rbd_data.10966b8b4567.00000000000029cd/head//3
2014-06-12 14:51:43.169429 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 81d70043/rbd_data.10af6b8b4567.00000000000013e4/head//3, auth == maps.end()
2014-06-12 14:51:43.169442 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 81d70043/rbd_data.10af6b8b4567.00000000000013e4/head//3
2014-06-12 14:51:43.169456 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b5d80043/rbd_data.10ae6b8b4567.000000000000146f/head//3, auth == maps.end()
2014-06-12 14:51:43.169473 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj b5d80043/rbd_data.10ae6b8b4567.000000000000146f/head//3
2014-06-12 14:51:43.169491 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 41b90043/rbd_data.108e6b8b4567.0000000000003449/head//3, auth == maps.end()
2014-06-12 14:51:43.169505 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 41b90043/rbd_data.108e6b8b4567.0000000000003449/head//3
2014-06-12 14:51:43.169522 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5d2b0043/rbd_data.10af6b8b4567.00000000000008ee/head//3, auth == maps.end()
2014-06-12 14:51:43.169536 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 5d2b0043/rbd_data.10af6b8b4567.00000000000008ee/head//3
2014-06-12 14:51:43.169553 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d8cd0043/rbd_data.108f6b8b4567.0000000000000933/head//3, auth == maps.end()
2014-06-12 14:51:43.169569 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d8cd0043/rbd_data.108f6b8b4567.0000000000000933/head//3
2014-06-12 14:51:43.169585 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e71e0043/rbd_data.10906b8b4567.0000000000002eeb/head//3, auth == maps.end()
2014-06-12 14:51:43.169600 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj e71e0043/rbd_data.10906b8b4567.0000000000002eeb/head//3
2014-06-12 14:51:43.169636 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5aee0043/rbd_data.10b46b8b4567.0000000000003367/head//3, auth == maps.end()
2014-06-12 14:51:43.169668 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 5aee0043/rbd_data.10b46b8b4567.0000000000003367/head//3
2014-06-12 14:51:43.169689 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2d611043/rbd_data.10ab6b8b4567.0000000000000914/head//3, auth == maps.end()
2014-06-12 14:51:43.169709 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 2d611043/rbd_data.10ab6b8b4567.0000000000000914/head//3
2014-06-12 14:51:43.169730 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e3711043/rbd_data.10906b8b4567.000000000000304f/head//3, auth == maps.end()
2014-06-12 14:51:43.169747 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj e3711043/rbd_data.10906b8b4567.000000000000304f/head//3
2014-06-12 14:51:43.169767 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cd261043/rbd_data.10a66b8b4567.000000000000063e/head//3, auth == maps.end()
2014-06-12 14:51:43.169786 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj cd261043/rbd_data.10a66b8b4567.000000000000063e/head//3
2014-06-12 14:51:43.169806 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dbc61043/rbd_data.10906b8b4567.0000000000001224/head//3, auth == maps.end()
2014-06-12 14:51:43.169831 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj dbc61043/rbd_data.10906b8b4567.0000000000001224/head//3
2014-06-12 14:51:43.169851 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 86f71043/rbd_data.10a96b8b4567.0000000000002178/head//3, auth == maps.end()
2014-06-12 14:51:43.169870 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 86f71043/rbd_data.10a96b8b4567.0000000000002178/head//3
2014-06-12 14:51:43.169890 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9f7b1043/rbd_data.10a66b8b4567.0000000000001779/head//3, auth == maps.end()
2014-06-12 14:51:43.169909 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9f7b1043/rbd_data.10a66b8b4567.0000000000001779/head//3
2014-06-12 14:51:43.169930 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3b922043/rbd_data.10906b8b4567.0000000000001c27/head//3, auth == maps.end()
2014-06-12 14:51:43.169949 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 3b922043/rbd_data.10906b8b4567.0000000000001c27/head//3
2014-06-12 14:51:43.169970 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 46552043/rbd_data.10906b8b4567.0000000000001090/head//3, auth == maps.end()
2014-06-12 14:51:43.169986 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 46552043/rbd_data.10906b8b4567.0000000000001090/head//3
2014-06-12 14:51:43.170006 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 41662043/rbd_data.10a26b8b4567.00000000000007e2/head//3, auth == maps.end()
2014-06-12 14:51:43.170025 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 41662043/rbd_data.10a26b8b4567.00000000000007e2/head//3
2014-06-12 14:51:43.170045 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2cc72043/rbd_data.10a16b8b4567.0000000000003068/head//3, auth == maps.end()
2014-06-12 14:51:43.170064 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 2cc72043/rbd_data.10a16b8b4567.0000000000003068/head//3
2014-06-12 14:51:43.170084 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 65f82043/rbd_data.10b26b8b4567.0000000000003a6b/head//3, auth == maps.end()
2014-06-12 14:51:43.170103 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 65f82043/rbd_data.10b26b8b4567.0000000000003a6b/head//3
2014-06-12 14:51:43.170124 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 95ab2043/rbd_data.10936b8b4567.0000000000003b90/head//3, auth == maps.end()
2014-06-12 14:51:43.170143 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 95ab2043/rbd_data.10936b8b4567.0000000000003b90/head//3
2014-06-12 14:51:43.170165 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] 
2014-06-12 14:51:43.170180 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub
2014-06-12 14:51:43.170204 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  95ab2043/rbd_data.10936b8b4567.0000000000003b90/head//3 95ab2043/rbd_data.10936b8b4567.0000000000003b90/head//3(173'2046 client.4284.0:115005 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2046)
2014-06-12 14:51:43.170230 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  65f82043/rbd_data.10b26b8b4567.0000000000003a6b/head//3 65f82043/rbd_data.10b26b8b4567.0000000000003a6b/head//3(173'1919 client.4315.0:115302 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1919)
2014-06-12 14:51:43.170256 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  2cc72043/rbd_data.10a16b8b4567.0000000000003068/head//3 2cc72043/rbd_data.10a16b8b4567.0000000000003068/head//3(173'1658 client.4298.0:97740 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1658)
2014-06-12 14:51:43.170280 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  41662043/rbd_data.10a26b8b4567.00000000000007e2/head//3 41662043/rbd_data.10a26b8b4567.00000000000007e2/head//3(173'2287 client.4299.0:132552 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2287)
2014-06-12 14:51:43.170304 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  46552043/rbd_data.10906b8b4567.0000000000001090/head//3 46552043/rbd_data.10906b8b4567.0000000000001090/head//3(173'685 client.4281.0:39143 wrlock_by=unknown.0.0:0 dirty s 4194304 uv685)
2014-06-12 14:51:43.170328 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  3b922043/rbd_data.10906b8b4567.0000000000001c27/head//3 3b922043/rbd_data.10906b8b4567.0000000000001c27/head//3(173'1015 client.4281.0:59931 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1015)
2014-06-12 14:51:43.170352 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9f7b1043/rbd_data.10a66b8b4567.0000000000001779/head//3 9f7b1043/rbd_data.10a66b8b4567.0000000000001779/head//3(173'942 client.4303.0:50645 wrlock_by=unknown.0.0:0 dirty s 4194304 uv942)
2014-06-12 14:51:43.170376 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  86f71043/rbd_data.10a96b8b4567.0000000000002178/head//3 86f71043/rbd_data.10a96b8b4567.0000000000002178/head//3(173'1290 client.4306.0:71926 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1290)
2014-06-12 14:51:43.170400 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  dbc61043/rbd_data.10906b8b4567.0000000000001224/head//3 dbc61043/rbd_data.10906b8b4567.0000000000001224/head//3(173'726 client.4281.0:42146 wrlock_by=unknown.0.0:0 dirty s 4194304 uv726)
2014-06-12 14:51:43.170423 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  cd261043/rbd_data.10a66b8b4567.000000000000063e/head//3 cd261043/rbd_data.10a66b8b4567.000000000000063e/head//3(173'2206 client.4303.0:127924 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2206)
2014-06-12 14:51:43.170447 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  e3711043/rbd_data.10906b8b4567.000000000000304f/head//3 e3711043/rbd_data.10906b8b4567.000000000000304f/head//3(173'1553 client.4281.0:96278 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1553)
2014-06-12 14:51:43.170471 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  2d611043/rbd_data.10ab6b8b4567.0000000000000914/head//3 2d611043/rbd_data.10ab6b8b4567.0000000000000914/head//3(173'2323 client.4308.0:134236 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2323)
2014-06-12 14:51:43.170495 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  5aee0043/rbd_data.10b46b8b4567.0000000000003367/head//3 5aee0043/rbd_data.10b46b8b4567.0000000000003367/head//3(173'1912 client.4317.0:102272 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1912)
2014-06-12 14:51:43.170518 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  e71e0043/rbd_data.10906b8b4567.0000000000002eeb/head//3 e71e0043/rbd_data.10906b8b4567.0000000000002eeb/head//3(173'1520 client.4281.0:93985 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1520)
2014-06-12 14:51:43.170542 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d8cd0043/rbd_data.108f6b8b4567.0000000000000933/head//3 d8cd0043/rbd_data.108f6b8b4567.0000000000000933/head//3(173'2335 client.4280.0:130453 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2335)
2014-06-12 14:51:43.170571 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  5d2b0043/rbd_data.10af6b8b4567.00000000000008ee/head//3 5d2b0043/rbd_data.10af6b8b4567.00000000000008ee/head//3(173'2313 client.4312.0:134816 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2313)
2014-06-12 14:51:43.170594 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  41b90043/rbd_data.108e6b8b4567.0000000000003449/head//3 41b90043/rbd_data.108e6b8b4567.0000000000003449/head//3(173'1769 client.4279.0:103631 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1769)
2014-06-12 14:51:43.170618 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  b5d80043/rbd_data.10ae6b8b4567.000000000000146f/head//3 b5d80043/rbd_data.10ae6b8b4567.000000000000146f/head//3(173'791 client.4311.0:45933 wrlock_by=unknown.0.0:0 dirty s 4194304 uv791)
2014-06-12 14:51:43.170642 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  81d70043/rbd_data.10af6b8b4567.00000000000013e4/head//3 81d70043/rbd_data.10af6b8b4567.00000000000013e4/head//3(173'845 client.4312.0:47623 wrlock_by=unknown.0.0:0 dirty s 4194304 uv845)
2014-06-12 14:51:43.170666 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  a0a70043/rbd_data.10966b8b4567.00000000000029cd/head//3 a0a70043/rbd_data.10966b8b4567.00000000000029cd/head//3(173'1405 client.4287.0:81675 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1405)
2014-06-12 14:51:43.170690 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  56770043/rbd_data.10ae6b8b4567.0000000000000e5f/head//3 56770043/rbd_data.10ae6b8b4567.0000000000000e5f/head//3(173'2546 client.4311.0:138978 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2546)
2014-06-12 14:51:43.170714 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  46670043/rbd_data.109f6b8b4567.0000000000001ba2/head//3 46670043/rbd_data.109f6b8b4567.0000000000001ba2/head//3(173'952 client.4296.0:58395 wrlock_by=unknown.0.0:0 dirty s 4194304 uv952)
2014-06-12 14:51:43.170738 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  55640043/rbd_data.109e6b8b4567.0000000000001d7e/head//3 55640043/rbd_data.109e6b8b4567.0000000000001d7e/head//3(173'1127 client.4295.0:63425 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1127)
2014-06-12 14:51:43.170761 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  94b30043/rbd_data.10a96b8b4567.0000000000000c48/head//3 94b30043/rbd_data.10a96b8b4567.0000000000000c48/head//3(173'2414 client.4306.0:138465 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2414)
2014-06-12 14:51:43.170785 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  2fa30043/rbd_data.108d6b8b4567.0000000000000812/head//3 2fa30043/rbd_data.108d6b8b4567.0000000000000812/head//3(173'2256 client.4278.0:130466 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2256)
2014-06-12 14:51:43.170804 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:43.170823 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:43.170839 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:43.170843 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:43.170845 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:43.170899 7f9d60bd2700 10 collection_list_partial 3.43_head start:76cb2043//0//-1 is_max:0
2014-06-12 14:51:43.171261 7f9d60bd2700 20 objects: [3d3c2043/rbd_data.10986b8b4567.00000000000010d8/head//3,d89d2043/rbd_data.108e6b8b4567.0000000000001fc5/head//3,195e2043/rbd_data.10a46b8b4567.0000000000000cd0/head//3,95e23043/rbd_data.108d6b8b4567.000000000000348f/head//3,2f733043/rbd_data.109d6b8b4567.0000000000002ffe/head//3,41043043/rbd_data.10a46b8b4567.0000000000000b8d/head//3,2e553043/rbd_data.10a26b8b4567.000000000000333f/head//3,afa63043/rbd_data.10a16b8b4567.0000000000000ac6/head//3,aec73043/rbd_data.10a36b8b4567.000000000000026c/head//3,d2583043/rbd_data.10ab6b8b4567.0000000000002fab/head//3,f1493043/rbd_data.10a06b8b4567.0000000000002bb0/head//3,c3e93043/rbd_data.10976b8b4567.00000000000038a2/head//3,303b3043/rbd_data.10936b8b4567.0000000000002e76/head//3,2a9b3043/rbd_data.10986b8b4567.0000000000001c55/head//3,ffbd3043/rbd_data.109c6b8b4567.0000000000002ea0/head//3,2f1f3043/rbd_data.10906b8b4567.00000000000004fb/head//3,65014043/rbd_data.109b6b8b4567.0000000000000270/head//3,fde34043/rbd_data.10936b8b4567.00000000000015b5/head//3,15d54043/rbd_data.10af6b8b4567.000000000000328f/head//3,7cb64043/rbd_data.10b46b8b4567.000000000000382f/head//3,547a4043/rbd_data.108e6b8b4567.000000000000004f/head//3,750b4043/rbd_data.10996b8b4567.0000000000001f83/head//3,3d9d4043/rbd_data.10a76b8b4567.000000000000255d/head//3,d5815043/rbd_data.10a76b8b4567.0000000000001858/head//3,2e345043/rbd_data.10a86b8b4567.0000000000001cd7/head//3]
2014-06-12 14:51:43.171943 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  requesting scrubmap from osd.24
2014-06-12 14:51:43.171978 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:43.172007 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:43.172023 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:43.172034 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:43.172045 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:43.172055 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_map_chunk [76cb2043//0//-1,a9a55043//0//-1)
2014-06-12 14:51:43.172069 7f9d60bd2700 10 collection_list_partial 3.43_head start:76cb2043//0//-1 is_max:0
2014-06-12 14:51:43.172967 7f9d60bd2700 20 objects: [3d3c2043/rbd_data.10986b8b4567.00000000000010d8/head//3,d89d2043/rbd_data.108e6b8b4567.0000000000001fc5/head//3,195e2043/rbd_data.10a46b8b4567.0000000000000cd0/head//3,95e23043/rbd_data.108d6b8b4567.000000000000348f/head//3,2f733043/rbd_data.109d6b8b4567.0000000000002ffe/head//3,41043043/rbd_data.10a46b8b4567.0000000000000b8d/head//3,2e553043/rbd_data.10a26b8b4567.000000000000333f/head//3,afa63043/rbd_data.10a16b8b4567.0000000000000ac6/head//3,aec73043/rbd_data.10a36b8b4567.000000000000026c/head//3,d2583043/rbd_data.10ab6b8b4567.0000000000002fab/head//3,f1493043/rbd_data.10a06b8b4567.0000000000002bb0/head//3,c3e93043/rbd_data.10976b8b4567.00000000000038a2/head//3,303b3043/rbd_data.10936b8b4567.0000000000002e76/head//3,2a9b3043/rbd_data.10986b8b4567.0000000000001c55/head//3,ffbd3043/rbd_data.109c6b8b4567.0000000000002ea0/head//3,2f1f3043/rbd_data.10906b8b4567.00000000000004fb/head//3,65014043/rbd_data.109b6b8b4567.0000000000000270/head//3,fde34043/rbd_data.10936b8b4567.00000000000015b5/head//3,15d54043/rbd_data.10af6b8b4567.000000000000328f/head//3,7cb64043/rbd_data.10b46b8b4567.000000000000382f/head//3,547a4043/rbd_data.108e6b8b4567.000000000000004f/head//3,750b4043/rbd_data.10996b8b4567.0000000000001f83/head//3,3d9d4043/rbd_data.10a76b8b4567.000000000000255d/head//3,d5815043/rbd_data.10a76b8b4567.0000000000001858/head//3,2e345043/rbd_data.10a86b8b4567.0000000000001cd7/head//3,a9a55043/rbd_data.10ae6b8b4567.0000000000000daa/head//3,63085043/rbd_data.108d6b8b4567.0000000000003728/head//3,68a95043/rbd_data.10ac6b8b4567.0000000000000aaf/head//3,efbf5043/rbd_data.10b06b8b4567.000000000000338e/head//3,4d436043/rbd_data.109e6b8b4567.00000000000039f8/head//3,b7a66043/rbd_data.10926b8b4567.00000000000021e8/head//3,82d76043/rbd_data.10926b8b4567.00000000000001be/head//3,f1a86043/rbd_data.10b16b8b4567.0000000000001c03/head//3,b1d96043/rbd_data.109c6b8b4567.0000000000000eef/head//3,578b6043/rbd_data.10ad6b8b4567.00000000000000d8/head//3,d9b6043/rbd_data.10906b8b4567.0000000000000124/head//3,4a1f6043/rbd_data.10936b8b4567.0000000000003bf0/head//3,50217043/rbd_data.109c6b8b4567.0000000000001d6c/head//3,7a027043/rbd_data.109d6b8b4567.000000000000280e/head//3,a9c57043/rbd_data.10b26b8b4567.0000000000001430/head//3,17367043/rbd_data.10986b8b4567.0000000000001dd2/head//3,49177043/rbd_data.10956b8b4567.000000000000220a/head//3,496c7043/rbd_data.10b36b8b4567.000000000000374d/head//3,4d3f7043/rbd_data.10a86b8b4567.000000000000301e/head//3,9aff7043/rbd_data.10a26b8b4567.0000000000000a20/head//3,3eb18043/rbd_data.10976b8b4567.0000000000001a41/head//3,9d268043/rbd_data.10b26b8b4567.0000000000000db5/head//3,5a6a8043/rbd_data.10b26b8b4567.00000000000033a9/head//3,375c8043/rbd_data.10936b8b4567.0000000000001fcc/head//3,596c8043/rbd_data.109a6b8b4567.0000000000003a6a/head//3,279f8043/rbd_data.108f6b8b4567.0000000000003741/head//3,6fdf8043/rbd_data.10a06b8b4567.0000000000000294/head//3,f2039043/rbd_data.10996b8b4567.000000000000391e/head//3,f2a39043/rbd_data.109d6b8b4567.0000000000000a4e/head//3,29549043/rbd_data.10aa6b8b4567.0000000000001230/head//3,30569043/rbd_data.10b46b8b4567.0000000000003921/head//3,b089043/rbd_data.10b26b8b4567.0000000000000466/head//3,9da89043/rbd_data.10a46b8b4567.000000000000064f/head//3,e1199043/rbd_data.10ae6b8b4567.00000000000020b0/head//3,44599043/rbd_data.109b6b8b4567.00000000000023c8/head//3,81da9043/rbd_data.10af6b8b4567.0000000000003ae6/head//3,c1fa9043/rbd_data.10ac6b8b4567.0000000000003695/head//3,9cac9043/rbd_data.10956b8b4567.0000000000002d9f/head//3,cb7d9043/rbd_data.10b06b8b4567.00000000000008a4/head//3]
2014-06-12 14:51:43.173101 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:43.173130 7f9d60bd2700 10 stat 3.43_head/3d3c2043/rbd_data.10986b8b4567.00000000000010d8/head//3
2014-06-12 14:51:43.173343 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 3d3c2043/rbd_data.10986b8b4567.00000000000010d8/head//3
2014-06-12 14:51:43.180481 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:43.190378 7f9d60bd2700 10 getattrs 3.43_head/3d3c2043/rbd_data.10986b8b4567.00000000000010d8/head//3 = 0
2014-06-12 14:51:43.190411 7f9d60bd2700 10 stat 3.43_head/d89d2043/rbd_data.108e6b8b4567.0000000000001fc5/head//3
2014-06-12 14:51:43.190608 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d89d2043/rbd_data.108e6b8b4567.0000000000001fc5/head//3
2014-06-12 14:51:43.195910 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:43.196151 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:43.201361 7f9d60bd2700 10 getattrs 3.43_head/d89d2043/rbd_data.108e6b8b4567.0000000000001fc5/head//3 = 0
2014-06-12 14:51:43.201378 7f9d60bd2700 10 stat 3.43_head/195e2043/rbd_data.10a46b8b4567.0000000000000cd0/head//3
2014-06-12 14:51:43.201561 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 195e2043/rbd_data.10a46b8b4567.0000000000000cd0/head//3
2014-06-12 14:51:43.248141 7f9d60bd2700 10 getattrs 3.43_head/195e2043/rbd_data.10a46b8b4567.0000000000000cd0/head//3 = 0
2014-06-12 14:51:43.248154 7f9d60bd2700 10 stat 3.43_head/95e23043/rbd_data.108d6b8b4567.000000000000348f/head//3
2014-06-12 14:51:43.248274 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 95e23043/rbd_data.108d6b8b4567.000000000000348f/head//3
2014-06-12 14:51:43.267195 7f9d60bd2700 10 getattrs 3.43_head/95e23043/rbd_data.108d6b8b4567.000000000000348f/head//3 = 0
2014-06-12 14:51:43.267209 7f9d60bd2700 10 stat 3.43_head/2f733043/rbd_data.109d6b8b4567.0000000000002ffe/head//3
2014-06-12 14:51:43.267352 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 2f733043/rbd_data.109d6b8b4567.0000000000002ffe/head//3
2014-06-12 14:51:43.292888 7f9d60bd2700 10 getattrs 3.43_head/2f733043/rbd_data.109d6b8b4567.0000000000002ffe/head//3 = 0
2014-06-12 14:51:43.292900 7f9d60bd2700 10 stat 3.43_head/41043043/rbd_data.10a46b8b4567.0000000000000b8d/head//3
2014-06-12 14:51:43.293035 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 41043043/rbd_data.10a46b8b4567.0000000000000b8d/head//3
2014-06-12 14:51:43.298643 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:43.316754 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:43.333562 7f9d60bd2700 10 getattrs 3.43_head/41043043/rbd_data.10a46b8b4567.0000000000000b8d/head//3 = 0
2014-06-12 14:51:43.333588 7f9d60bd2700 10 stat 3.43_head/2e553043/rbd_data.10a26b8b4567.000000000000333f/head//3
2014-06-12 14:51:43.333752 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 2e553043/rbd_data.10a26b8b4567.000000000000333f/head//3
2014-06-12 14:51:43.341825 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:43.353733 7f9d60bd2700 10 getattrs 3.43_head/2e553043/rbd_data.10a26b8b4567.000000000000333f/head//3 = 0
2014-06-12 14:51:43.353740 7f9d60bd2700 10 stat 3.43_head/afa63043/rbd_data.10a16b8b4567.0000000000000ac6/head//3
2014-06-12 14:51:43.353842 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid afa63043/rbd_data.10a16b8b4567.0000000000000ac6/head//3
2014-06-12 14:51:43.375102 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:43.386382 7f9d60bd2700 10 getattrs 3.43_head/afa63043/rbd_data.10a16b8b4567.0000000000000ac6/head//3 = 0
2014-06-12 14:51:43.386396 7f9d60bd2700 10 stat 3.43_head/aec73043/rbd_data.10a36b8b4567.000000000000026c/head//3
2014-06-12 14:51:43.386488 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid aec73043/rbd_data.10a36b8b4567.000000000000026c/head//3
2014-06-12 14:51:43.452086 7f9d60bd2700 10 getattrs 3.43_head/aec73043/rbd_data.10a36b8b4567.000000000000026c/head//3 = 0
2014-06-12 14:51:43.452098 7f9d60bd2700 10 stat 3.43_head/d2583043/rbd_data.10ab6b8b4567.0000000000002fab/head//3
2014-06-12 14:51:43.452207 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d2583043/rbd_data.10ab6b8b4567.0000000000002fab/head//3
2014-06-12 14:51:43.462725 7f9d60bd2700 10 getattrs 3.43_head/d2583043/rbd_data.10ab6b8b4567.0000000000002fab/head//3 = 0
2014-06-12 14:51:43.462739 7f9d60bd2700 10 stat 3.43_head/f1493043/rbd_data.10a06b8b4567.0000000000002bb0/head//3
2014-06-12 14:51:43.462891 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f1493043/rbd_data.10a06b8b4567.0000000000002bb0/head//3
2014-06-12 14:51:43.482018 7f9d60bd2700 10 getattrs 3.43_head/f1493043/rbd_data.10a06b8b4567.0000000000002bb0/head//3 = 0
2014-06-12 14:51:43.482026 7f9d60bd2700 10 stat 3.43_head/c3e93043/rbd_data.10976b8b4567.00000000000038a2/head//3
2014-06-12 14:51:43.482112 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c3e93043/rbd_data.10976b8b4567.00000000000038a2/head//3
2014-06-12 14:51:43.498116 7f9d60bd2700 10 getattrs 3.43_head/c3e93043/rbd_data.10976b8b4567.00000000000038a2/head//3 = 0
2014-06-12 14:51:43.498123 7f9d60bd2700 10 stat 3.43_head/303b3043/rbd_data.10936b8b4567.0000000000002e76/head//3
2014-06-12 14:51:43.498221 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 303b3043/rbd_data.10936b8b4567.0000000000002e76/head//3
2014-06-12 14:51:43.513594 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:43.518662 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:43.521084 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:43.521130 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.46 < max 0.5 = yes
2014-06-12 14:51:43.521144 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=1
2014-06-12 14:51:43.521151 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:43.524683 7f9d60bd2700 10 getattrs 3.43_head/303b3043/rbd_data.10936b8b4567.0000000000002e76/head//3 = 0
2014-06-12 14:51:43.524700 7f9d60bd2700 10 stat 3.43_head/2a9b3043/rbd_data.10986b8b4567.0000000000001c55/head//3
2014-06-12 14:51:43.524864 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 2a9b3043/rbd_data.10986b8b4567.0000000000001c55/head//3
2014-06-12 14:51:43.547540 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:43.547607 7f9d60bd2700 10 getattrs 3.43_head/2a9b3043/rbd_data.10986b8b4567.0000000000001c55/head//3 = 0
2014-06-12 14:51:43.547628 7f9d60bd2700 10 stat 3.43_head/ffbd3043/rbd_data.109c6b8b4567.0000000000002ea0/head//3
2014-06-12 14:51:43.547788 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid ffbd3043/rbd_data.109c6b8b4567.0000000000002ea0/head//3
2014-06-12 14:51:43.571670 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:43.571888 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:43.576557 7f9d60bd2700 10 getattrs 3.43_head/ffbd3043/rbd_data.109c6b8b4567.0000000000002ea0/head//3 = 0
2014-06-12 14:51:43.576567 7f9d60bd2700 10 stat 3.43_head/2f1f3043/rbd_data.10906b8b4567.00000000000004fb/head//3
2014-06-12 14:51:43.576662 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 2f1f3043/rbd_data.10906b8b4567.00000000000004fb/head//3
2014-06-12 14:51:43.627825 7f9d60bd2700 10 getattrs 3.43_head/2f1f3043/rbd_data.10906b8b4567.00000000000004fb/head//3 = 0
2014-06-12 14:51:43.627838 7f9d60bd2700 10 stat 3.43_head/65014043/rbd_data.109b6b8b4567.0000000000000270/head//3
2014-06-12 14:51:43.627959 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 65014043/rbd_data.109b6b8b4567.0000000000000270/head//3
2014-06-12 14:51:43.685944 7f9d60bd2700 10 getattrs 3.43_head/65014043/rbd_data.109b6b8b4567.0000000000000270/head//3 = 0
2014-06-12 14:51:43.685957 7f9d60bd2700 10 stat 3.43_head/fde34043/rbd_data.10936b8b4567.00000000000015b5/head//3
2014-06-12 14:51:43.686085 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid fde34043/rbd_data.10936b8b4567.00000000000015b5/head//3
2014-06-12 14:51:43.698790 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:43.698888 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:43.717786 7f9d60bd2700 10 getattrs 3.43_head/fde34043/rbd_data.10936b8b4567.00000000000015b5/head//3 = 0
2014-06-12 14:51:43.717804 7f9d60bd2700 10 stat 3.43_head/15d54043/rbd_data.10af6b8b4567.000000000000328f/head//3
2014-06-12 14:51:43.718004 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 15d54043/rbd_data.10af6b8b4567.000000000000328f/head//3
2014-06-12 14:51:43.749334 7f9d5a16f700  5 -- op tracker -- , seq: 748596, time: 2014-06-12 14:51:43.749216, event: header_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.749353 7f9d5a16f700  5 -- op tracker -- , seq: 748596, time: 2014-06-12 14:51:43.749219, event: throttled, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.749366 7f9d5a16f700  5 -- op tracker -- , seq: 748596, time: 2014-06-12 14:51:43.749284, event: all_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.749401 7f9d5a16f700  5 -- op tracker -- , seq: 748596, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.749408 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:43.749416 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:51:43.749426 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903f00 prio 127 cost 10717 latency 0.000210 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:43.749498 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:43.752993 7f9d60bd2700 10 getattrs 3.43_head/15d54043/rbd_data.10af6b8b4567.000000000000328f/head//3 = 0
2014-06-12 14:51:43.753011 7f9d60bd2700 10 stat 3.43_head/7cb64043/rbd_data.10b46b8b4567.000000000000382f/head//3
2014-06-12 14:51:43.753218 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 7cb64043/rbd_data.10b46b8b4567.000000000000382f/head//3
2014-06-12 14:51:43.783214 7f9d60bd2700 10 getattrs 3.43_head/7cb64043/rbd_data.10b46b8b4567.000000000000382f/head//3 = 0
2014-06-12 14:51:43.783221 7f9d60bd2700 10 stat 3.43_head/547a4043/rbd_data.108e6b8b4567.000000000000004f/head//3
2014-06-12 14:51:43.783315 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 547a4043/rbd_data.108e6b8b4567.000000000000004f/head//3
2014-06-12 14:51:43.793244 7f9d60bd2700 10 getattrs 3.43_head/547a4043/rbd_data.108e6b8b4567.000000000000004f/head//3 = 0
2014-06-12 14:51:43.793252 7f9d60bd2700 10 stat 3.43_head/750b4043/rbd_data.10996b8b4567.0000000000001f83/head//3
2014-06-12 14:51:43.793350 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 750b4043/rbd_data.10996b8b4567.0000000000001f83/head//3
2014-06-12 14:51:43.827606 7f9d60bd2700 10 getattrs 3.43_head/750b4043/rbd_data.10996b8b4567.0000000000001f83/head//3 = 0
2014-06-12 14:51:43.827619 7f9d60bd2700 10 stat 3.43_head/3d9d4043/rbd_data.10a76b8b4567.000000000000255d/head//3
2014-06-12 14:51:43.827729 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 3d9d4043/rbd_data.10a76b8b4567.000000000000255d/head//3
2014-06-12 14:51:43.843256 7f9d60bd2700 10 getattrs 3.43_head/3d9d4043/rbd_data.10a76b8b4567.000000000000255d/head//3 = 0
2014-06-12 14:51:43.843265 7f9d60bd2700 10 stat 3.43_head/d5815043/rbd_data.10a76b8b4567.0000000000001858/head//3
2014-06-12 14:51:43.843380 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d5815043/rbd_data.10a76b8b4567.0000000000001858/head//3
2014-06-12 14:51:43.874918 7f9d60bd2700 10 getattrs 3.43_head/d5815043/rbd_data.10a76b8b4567.0000000000001858/head//3 = 0
2014-06-12 14:51:43.874930 7f9d60bd2700 10 stat 3.43_head/2e345043/rbd_data.10a86b8b4567.0000000000001cd7/head//3
2014-06-12 14:51:43.875039 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 2e345043/rbd_data.10a86b8b4567.0000000000001cd7/head//3
2014-06-12 14:51:43.890112 7f9d60bd2700 10 getattrs 3.43_head/2e345043/rbd_data.10a86b8b4567.0000000000001cd7/head//3 = 0
2014-06-12 14:51:43.890123 7f9d60bd2700 10 collection_getattrs 3.43_head
2014-06-12 14:51:43.890184 7f9d60bd2700 10 collection_getattrs 3.43_head = 0
2014-06-12 14:51:43.890190 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:43.890228 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:43.890236 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:43.890247 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:43.890250 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:43.890276 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.43 at 2014-06-11 13:02:32.501298
2014-06-12 14:51:43.890316 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6b at 2014-06-11 13:02:32.502023
2014-06-12 14:51:43.890327 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890331 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890354 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6c at 2014-06-11 13:02:32.502351
2014-06-12 14:51:43.890362 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890364 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890378 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.a0 at 2014-06-11 13:02:32.502930
2014-06-12 14:51:43.890384 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890386 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.a0( v 173'2803 (0'0,173'2803] local-les=157 n=294 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2802 mlcod 173'2802 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890365 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2a903f00 prio 127 cost 10717 latency 0.141148 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]
2014-06-12 14:51:43.890407 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.105 at 2014-06-11 13:02:32.503257
2014-06-12 14:51:43.890411 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890412 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2817 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890413 7f9d61bd4700  5 -- op tracker -- , seq: 748596, time: 2014-06-12 14:51:43.890409, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.890421 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.122 at 2014-06-11 13:02:32.503770
2014-06-12 14:51:43.890424 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890426 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890425 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] handle_message: 0x2a903f00
2014-06-12 14:51:43.890436 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:43.890434 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.12b at 2014-06-11 13:02:32.504419
2014-06-12 14:51:43.890449 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890446 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:43.890451 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.12b( v 173'3236 (79'235,173'3236] local-les=137 n=323 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3235 mlcod 173'3235 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890461 7f9d61bd4700  5 -- op tracker -- , seq: 748596, time: 2014-06-12 14:51:43.890458, event: started, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.890461 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.131 at 2014-06-11 13:02:32.504931
2014-06-12 14:51:43.890471 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  got 24 scrub map
2014-06-12 14:51:43.890476 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890479 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890488 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.15d at 2014-06-11 13:02:32.505494
2014-06-12 14:51:43.890492 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890494 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890506 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.165 at 2014-06-11 13:02:32.506051
2014-06-12 14:51:43.890510 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890511 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.165( v 173'3106 (79'105,173'3106] local-les=149 n=320 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3105 mlcod 173'3105 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890520 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.170 at 2014-06-11 13:02:32.506577
2014-06-12 14:51:43.890523 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890524 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3049 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890532 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1b5 at 2014-06-11 13:02:32.507250
2014-06-12 14:51:43.890536 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890534 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] map version is 173'2566
2014-06-12 14:51:43.890537 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1b5( v 173'2905 (0'0,173'2905] local-les=153 n=320 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2904 mlcod 173'2904 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890545 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1bc at 2014-06-11 13:02:32.507758
2014-06-12 14:51:43.890547 7f9d61bd4700 10 osd.2 173 dequeue_op 0x2a903f00 finish
2014-06-12 14:51:43.890548 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890549 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2459 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890557 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1e1 at 2014-06-11 13:02:32.508326
2014-06-12 14:51:43.890560 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890554 7f9d61bd4700  5 -- op tracker -- , seq: 748596, time: 2014-06-12 14:51:43.890550, event: done, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:43.890561 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2540 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890570 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1e8 at 2014-06-11 13:02:32.508866
2014-06-12 14:51:43.890573 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890577 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:43.890576 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:43.890574 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3033 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890581 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:43.890583 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1eb at 2014-06-11 13:02:32.509421
2014-06-12 14:51:43.890587 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890584 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:43.890588 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3095 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890598 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1fd at 2014-06-11 13:02:32.510047
2014-06-12 14:51:43.890602 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890597 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:43.890603 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1fd( v 173'3047 (79'46,173'3047] local-les=157 n=305 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890613 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:43.890626 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.204 at 2014-06-11 13:02:32.510573
2014-06-12 14:51:43.890632 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890634 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.204( v 173'2982 (0'0,173'2982] local-les=161 n=308 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890646 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.209 at 2014-06-11 13:02:32.511095
2014-06-12 14:51:43.890651 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890653 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.209( v 173'2531 (0'0,173'2531] local-les=156 n=264 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2530 mlcod 173'2530 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890654 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:43.890666 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.214 at 2014-06-11 13:02:32.511670
2014-06-12 14:51:43.890672 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890668 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:43.890682 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub replica 24 has 25 items
2014-06-12 14:51:43.890675 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2934 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890696 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.21d at 2014-06-11 13:02:32.512202
2014-06-12 14:51:43.890702 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890704 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.21d( v 173'3024 (79'23,173'3024] local-les=156 n=318 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3023 mlcod 173'3023 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890717 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.23f at 2014-06-11 13:02:32.512776
2014-06-12 14:51:43.890723 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890721 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3d3c2043/rbd_data.10986b8b4567.00000000000010d8/head//3, auth == maps.end()
2014-06-12 14:51:43.890725 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890738 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.251 at 2014-06-11 13:02:32.513340
2014-06-12 14:51:43.890744 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890746 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890745 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 3d3c2043/rbd_data.10986b8b4567.00000000000010d8/head//3
2014-06-12 14:51:43.890759 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.262 at 2014-06-11 13:02:32.513946
2014-06-12 14:51:43.890766 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890768 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.262( v 173'2675 (0'0,173'2675] local-les=142 n=292 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2674 mlcod 173'2674 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890769 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d89d2043/rbd_data.108e6b8b4567.0000000000001fc5/head//3, auth == maps.end()
2014-06-12 14:51:43.890782 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.266 at 2014-06-11 13:02:32.514523
2014-06-12 14:51:43.890795 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890787 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d89d2043/rbd_data.108e6b8b4567.0000000000001fc5/head//3
2014-06-12 14:51:43.890798 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2824 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890811 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.288 at 2014-06-11 13:02:32.515022
2014-06-12 14:51:43.890817 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890806 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 195e2043/rbd_data.10a46b8b4567.0000000000000cd0/head//3, auth == maps.end()
2014-06-12 14:51:43.890820 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.288( v 173'2548 (0'0,173'2548] local-les=161 n=263 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2547 mlcod 173'2547 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890833 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2b4 at 2014-06-11 13:02:32.515543
2014-06-12 14:51:43.890826 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 195e2043/rbd_data.10a46b8b4567.0000000000000cd0/head//3
2014-06-12 14:51:43.890839 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890841 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2b4( v 173'3109 (79'108,173'3109] local-les=137 n=312 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3108 mlcod 173'3108 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890846 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 95e23043/rbd_data.108d6b8b4567.000000000000348f/head//3, auth == maps.end()
2014-06-12 14:51:43.890854 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2c4 at 2014-06-11 13:02:32.516106
2014-06-12 14:51:43.890863 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 95e23043/rbd_data.108d6b8b4567.000000000000348f/head//3
2014-06-12 14:51:43.890872 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890876 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2c4( v 173'2897 (0'0,173'2897] local-les=139 n=307 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2896 mlcod 173'2896 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890881 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2f733043/rbd_data.109d6b8b4567.0000000000002ffe/head//3, auth == maps.end()
2014-06-12 14:51:43.890891 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2d3 at 2014-06-11 13:02:32.516627
2014-06-12 14:51:43.890902 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890898 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 2f733043/rbd_data.109d6b8b4567.0000000000002ffe/head//3
2014-06-12 14:51:43.890904 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2d3( v 173'2556 (0'0,173'2556] local-les=153 n=276 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2555 mlcod 173'2555 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890918 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2da at 2014-06-11 13:02:32.517119
2014-06-12 14:51:43.890924 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890916 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 41043043/rbd_data.10a46b8b4567.0000000000000b8d/head//3, auth == maps.end()
2014-06-12 14:51:43.890926 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2da( v 173'2988 (0'0,173'2988] local-les=145 n=302 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2987 mlcod 173'2987 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890932 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 41043043/rbd_data.10a46b8b4567.0000000000000b8d/head//3
2014-06-12 14:51:43.890944 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2f4 at 2014-06-11 13:02:32.517764
2014-06-12 14:51:43.890950 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2e553043/rbd_data.10a26b8b4567.000000000000333f/head//3, auth == maps.end()
2014-06-12 14:51:43.890961 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890964 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3274 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890966 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 2e553043/rbd_data.10a26b8b4567.000000000000333f/head//3
2014-06-12 14:51:43.890977 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.312 at 2014-06-11 13:02:32.518300
2014-06-12 14:51:43.890983 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.890983 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj afa63043/rbd_data.10a16b8b4567.0000000000000ac6/head//3, auth == maps.end()
2014-06-12 14:51:43.890986 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2782 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.890998 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.320 at 2014-06-11 13:02:32.518824
2014-06-12 14:51:43.891004 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891000 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj afa63043/rbd_data.10a16b8b4567.0000000000000ac6/head//3
2014-06-12 14:51:43.891006 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.320( v 173'2687 (0'0,173'2687] local-les=145 n=282 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2686 mlcod 173'2686 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891020 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.330 at 2014-06-11 13:02:32.519349
2014-06-12 14:51:43.891019 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aec73043/rbd_data.10a36b8b4567.000000000000026c/head//3, auth == maps.end()
2014-06-12 14:51:43.891030 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891033 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2798 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891036 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj aec73043/rbd_data.10a36b8b4567.000000000000026c/head//3
2014-06-12 14:51:43.891045 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.342 at 2014-06-11 13:02:32.519873
2014-06-12 14:51:43.891053 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891053 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d2583043/rbd_data.10ab6b8b4567.0000000000002fab/head//3, auth == maps.end()
2014-06-12 14:51:43.891055 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2464 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891069 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.35b at 2014-06-11 13:02:32.520528
2014-06-12 14:51:43.891075 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891070 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d2583043/rbd_data.10ab6b8b4567.0000000000002fab/head//3
2014-06-12 14:51:43.891077 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.35b( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891090 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.366 at 2014-06-11 13:02:32.521097
2014-06-12 14:51:43.891096 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891088 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f1493043/rbd_data.10a06b8b4567.0000000000002bb0/head//3, auth == maps.end()
2014-06-12 14:51:43.891098 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2661 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891104 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f1493043/rbd_data.10a06b8b4567.0000000000002bb0/head//3
2014-06-12 14:51:43.891111 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.37a at 2014-06-11 13:02:32.521744
2014-06-12 14:51:43.891118 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891120 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.37a( v 173'2555 (0'0,173'2555] local-les=153 n=292 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2554 mlcod 173'2554 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891122 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c3e93043/rbd_data.10976b8b4567.00000000000038a2/head//3, auth == maps.end()
2014-06-12 14:51:43.891135 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.394 at 2014-06-11 13:02:32.522362
2014-06-12 14:51:43.891144 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891139 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c3e93043/rbd_data.10976b8b4567.00000000000038a2/head//3
2014-06-12 14:51:43.891146 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.394( v 173'2648 (0'0,173'2648] local-les=145 n=266 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891160 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.395 at 2014-06-11 13:02:32.522991
2014-06-12 14:51:43.891166 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891158 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 303b3043/rbd_data.10936b8b4567.0000000000002e76/head//3, auth == maps.end()
2014-06-12 14:51:43.891168 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3327 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891174 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 303b3043/rbd_data.10936b8b4567.0000000000002e76/head//3
2014-06-12 14:51:43.891181 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3ac at 2014-06-11 13:02:32.523569
2014-06-12 14:51:43.891188 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891190 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3ac( v 173'2860 (0'0,173'2860] local-les=142 n=294 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891192 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2a9b3043/rbd_data.10986b8b4567.0000000000001c55/head//3, auth == maps.end()
2014-06-12 14:51:43.891204 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3d4 at 2014-06-11 13:02:32.524119
2014-06-12 14:51:43.891209 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891208 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 2a9b3043/rbd_data.10986b8b4567.0000000000001c55/head//3
2014-06-12 14:51:43.891212 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3d4( v 173'2696 (0'0,173'2696] local-les=137 n=271 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891225 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3d8 at 2014-06-11 13:02:32.524641
2014-06-12 14:51:43.891234 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891227 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ffbd3043/rbd_data.109c6b8b4567.0000000000002ea0/head//3, auth == maps.end()
2014-06-12 14:51:43.891236 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3d8( v 173'2879 (0'0,173'2879] local-les=156 n=294 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891250 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.401 at 2014-06-11 13:02:32.525188
2014-06-12 14:51:43.891243 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj ffbd3043/rbd_data.109c6b8b4567.0000000000002ea0/head//3
2014-06-12 14:51:43.891255 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891258 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3154 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891261 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2f1f3043/rbd_data.10906b8b4567.00000000000004fb/head//3, auth == maps.end()
2014-06-12 14:51:43.891271 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.408 at 2014-06-11 13:02:32.525838
2014-06-12 14:51:43.891277 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891278 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 2f1f3043/rbd_data.10906b8b4567.00000000000004fb/head//3
2014-06-12 14:51:43.891279 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2883 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891292 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.411 at 2014-06-11 13:02:32.526419
2014-06-12 14:51:43.891298 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891296 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 65014043/rbd_data.109b6b8b4567.0000000000000270/head//3, auth == maps.end()
2014-06-12 14:51:43.891300 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.411( v 173'2402 (0'0,173'2402] local-les=147 n=278 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2401 mlcod 173'2401 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891313 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.45f at 2014-06-11 13:02:32.526940
2014-06-12 14:51:43.891319 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891313 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 65014043/rbd_data.109b6b8b4567.0000000000000270/head//3
2014-06-12 14:51:43.891321 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2357 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891334 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.479 at 2014-06-11 13:02:32.527498
2014-06-12 14:51:43.891339 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891331 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fde34043/rbd_data.10936b8b4567.00000000000015b5/head//3, auth == maps.end()
2014-06-12 14:51:43.891341 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.479( v 173'2926 (0'0,173'2926] local-les=142 n=298 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891347 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj fde34043/rbd_data.10936b8b4567.00000000000015b5/head//3
2014-06-12 14:51:43.891357 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4a7 at 2014-06-11 13:02:32.528023
2014-06-12 14:51:43.891375 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891364 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 15d54043/rbd_data.10af6b8b4567.000000000000328f/head//3, auth == maps.end()
2014-06-12 14:51:43.891378 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2594 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891384 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 15d54043/rbd_data.10af6b8b4567.000000000000328f/head//3
2014-06-12 14:51:43.891392 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4b4 at 2014-06-11 13:02:32.528548
2014-06-12 14:51:43.891402 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891402 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7cb64043/rbd_data.10b46b8b4567.000000000000382f/head//3, auth == maps.end()
2014-06-12 14:51:43.891405 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4b4( v 173'2792 (0'0,173'2792] local-les=137 n=305 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891419 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4d4 at 2014-06-11 13:02:32.529466
2014-06-12 14:51:43.891425 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891419 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 7cb64043/rbd_data.10b46b8b4567.000000000000382f/head//3
2014-06-12 14:51:43.891427 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2745 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891437 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 547a4043/rbd_data.108e6b8b4567.000000000000004f/head//3, auth == maps.end()
2014-06-12 14:51:43.891439 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4ef at 2014-06-11 13:02:32.530034
2014-06-12 14:51:43.891453 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891453 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 547a4043/rbd_data.108e6b8b4567.000000000000004f/head//3
2014-06-12 14:51:43.891460 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4ef( v 173'3004 (79'3,173'3004] local-les=149 n=307 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891472 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 750b4043/rbd_data.10996b8b4567.0000000000001f83/head//3, auth == maps.end()
2014-06-12 14:51:43.891475 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4f0 at 2014-06-11 13:02:32.530572
2014-06-12 14:51:43.891492 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891488 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 750b4043/rbd_data.10996b8b4567.0000000000001f83/head//3
2014-06-12 14:51:43.891495 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2864 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891510 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.54b at 2014-06-11 13:02:32.531354
2014-06-12 14:51:43.891506 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3d9d4043/rbd_data.10a76b8b4567.000000000000255d/head//3, auth == maps.end()
2014-06-12 14:51:43.891516 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891519 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2722 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891522 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 3d9d4043/rbd_data.10a76b8b4567.000000000000255d/head//3
2014-06-12 14:51:43.891532 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.568 at 2014-06-11 13:02:32.531914
2014-06-12 14:51:43.891537 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891539 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.568( v 173'2727 (0'0,173'2727] local-les=161 n=270 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891539 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d5815043/rbd_data.10a76b8b4567.0000000000001858/head//3, auth == maps.end()
2014-06-12 14:51:43.891554 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.58f at 2014-06-11 13:02:32.532612
2014-06-12 14:51:43.891559 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891557 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d5815043/rbd_data.10a76b8b4567.0000000000001858/head//3
2014-06-12 14:51:43.891561 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.58f( v 173'2707 (0'0,173'2707] local-les=153 n=304 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891575 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5a3 at 2014-06-11 13:02:32.533139
2014-06-12 14:51:43.891580 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891575 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2e345043/rbd_data.10a86b8b4567.0000000000001cd7/head//3, auth == maps.end()
2014-06-12 14:51:43.891582 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5a3( v 173'2874 (0'0,173'2874] local-les=142 n=296 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2873 mlcod 173'2873 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891595 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5ce at 2014-06-11 13:02:32.533636
2014-06-12 14:51:43.891601 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891593 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 2e345043/rbd_data.10a86b8b4567.0000000000001cd7/head//3
2014-06-12 14:51:43.891603 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5ce( v 173'2741 (0'0,173'2741] local-les=153 n=294 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891615 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5dd at 2014-06-11 13:02:32.534211
2014-06-12 14:51:43.891621 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891612 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] 
2014-06-12 14:51:43.891623 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5dd( v 173'2852 (0'0,173'2852] local-les=156 n=307 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891625 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub
2014-06-12 14:51:43.891635 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5fc at 2014-06-11 13:02:32.534738
2014-06-12 14:51:43.891641 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891643 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5fc( v 173'2714 (0'0,173'2714] local-les=157 n=284 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891655 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.602 at 2014-06-11 13:02:32.535274
2014-06-12 14:51:43.891661 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891648 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  2e345043/rbd_data.10a86b8b4567.0000000000001cd7/head//3 2e345043/rbd_data.10a86b8b4567.0000000000001cd7/head//3(173'1048 client.4305.0:60727 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1048)
2014-06-12 14:51:43.891663 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3009 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891676 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.605 at 2014-06-11 13:02:32.535829
2014-06-12 14:51:43.891681 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891673 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d5815043/rbd_data.10a76b8b4567.0000000000001858/head//3 d5815043/rbd_data.10a76b8b4567.0000000000001858/head//3(173'977 client.4304.0:53992 wrlock_by=unknown.0.0:0 dirty s 4194304 uv977)
2014-06-12 14:51:43.891684 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891695 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  3d9d4043/rbd_data.10a76b8b4567.000000000000255d/head//3 3d9d4043/rbd_data.10a76b8b4567.000000000000255d/head//3(173'1344 client.4304.0:76899 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1344)
2014-06-12 14:51:43.891700 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.613 at 2014-06-11 13:02:32.536448
2014-06-12 14:51:43.891714 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891717 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891717 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  750b4043/rbd_data.10996b8b4567.0000000000001f83/head//3 750b4043/rbd_data.10996b8b4567.0000000000001f83/head//3(173'1151 client.4290.0:65426 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1151)
2014-06-12 14:51:43.891731 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.635 at 2014-06-11 13:02:32.537041
2014-06-12 14:51:43.891737 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891739 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.635( v 173'2966 (0'0,173'2966] local-les=137 n=308 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2965 mlcod 173'2965 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891740 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  547a4043/rbd_data.108e6b8b4567.000000000000004f/head//3 547a4043/rbd_data.108e6b8b4567.000000000000004f/head//3(173'2075 client.4279.0:117914 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2075)
2014-06-12 14:51:43.891752 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.63b at 2014-06-11 13:02:32.537647
2014-06-12 14:51:43.891758 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891760 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891761 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  7cb64043/rbd_data.10b46b8b4567.000000000000382f/head//3 7cb64043/rbd_data.10b46b8b4567.000000000000382f/head//3(173'2041 client.4317.0:111271 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2041)
2014-06-12 14:51:43.891773 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.63e at 2014-06-11 13:02:32.538178
2014-06-12 14:51:43.891779 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891781 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891793 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.650 at 2014-06-11 13:02:32.538854
2014-06-12 14:51:43.891783 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  15d54043/rbd_data.10af6b8b4567.000000000000328f/head//3 15d54043/rbd_data.10af6b8b4567.000000000000328f/head//3(173'1756 client.4312.0:102943 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1756)
2014-06-12 14:51:43.891799 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891801 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891814 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.662 at 2014-06-11 13:02:32.539391
2014-06-12 14:51:43.891804 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  fde34043/rbd_data.10936b8b4567.00000000000015b5/head//3 fde34043/rbd_data.10936b8b4567.00000000000015b5/head//3(173'912 client.4284.0:46568 wrlock_by=unknown.0.0:0 dirty s 4194304 uv912)
2014-06-12 14:51:43.891819 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891822 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.662( v 173'2604 (0'0,173'2604] local-les=161 n=257 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2603 mlcod 173'2603 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891834 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.667 at 2014-06-11 13:02:32.539999
2014-06-12 14:51:43.891825 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  65014043/rbd_data.109b6b8b4567.0000000000000270/head//3 65014043/rbd_data.109b6b8b4567.0000000000000270/head//3(173'2114 client.4292.0:118919 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2114)
2014-06-12 14:51:43.891839 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891842 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.667( v 173'2958 (0'0,173'2958] local-les=149 n=303 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891854 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6a8 at 2014-06-11 13:02:32.540566
2014-06-12 14:51:43.891859 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891846 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  2f1f3043/rbd_data.10906b8b4567.00000000000004fb/head//3 2f1f3043/rbd_data.10906b8b4567.00000000000004fb/head//3(173'2146 client.4281.0:123965 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2146)
2014-06-12 14:51:43.891861 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6a8( v 173'2854 (0'0,173'2854] local-les=137 n=299 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891874 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6ff at 2014-06-11 13:02:32.541074
2014-06-12 14:51:43.891880 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891868 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  ffbd3043/rbd_data.109c6b8b4567.0000000000002ea0/head//3 ffbd3043/rbd_data.109c6b8b4567.0000000000002ea0/head//3(173'1647 client.4293.0:94043 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1647)
2014-06-12 14:51:43.891882 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6ff( v 173'2792 (0'0,173'2792] local-les=161 n=298 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891895 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.711 at 2014-06-11 13:02:32.541770
2014-06-12 14:51:43.891900 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891890 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  2a9b3043/rbd_data.10986b8b4567.0000000000001c55/head//3 2a9b3043/rbd_data.10986b8b4567.0000000000001c55/head//3(173'1089 client.4289.0:59546 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1089)
2014-06-12 14:51:43.891902 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.711( v 173'2514 (0'0,173'2514] local-les=147 n=297 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891915 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.712 at 2014-06-11 13:02:32.542314
2014-06-12 14:51:43.891920 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891911 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  303b3043/rbd_data.10936b8b4567.0000000000002e76/head//3 303b3043/rbd_data.10936b8b4567.0000000000002e76/head//3(173'1600 client.4284.0:91430 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1600)
2014-06-12 14:51:43.891922 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.712( v 173'2782 (0'0,173'2782] local-les=153 n=305 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891940 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.717 at 2014-06-11 13:02:32.542947
2014-06-12 14:51:43.891945 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891932 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c3e93043/rbd_data.10976b8b4567.00000000000038a2/head//3 c3e93043/rbd_data.10976b8b4567.00000000000038a2/head//3(173'1998 client.4288.0:109547 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1998)
2014-06-12 14:51:43.891947 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2583 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891960 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.721 at 2014-06-11 13:02:32.543618
2014-06-12 14:51:43.891966 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891954 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f1493043/rbd_data.10a06b8b4567.0000000000002bb0/head//3 f1493043/rbd_data.10a06b8b4567.0000000000002bb0/head//3(173'1468 client.4297.0:87262 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1468)
2014-06-12 14:51:43.891968 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2895 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.891980 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.735 at 2014-06-11 13:02:32.544147
2014-06-12 14:51:43.891985 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891975 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d2583043/rbd_data.10ab6b8b4567.0000000000002fab/head//3 d2583043/rbd_data.10ab6b8b4567.0000000000002fab/head//3(173'1608 client.4308.0:95802 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1608)
2014-06-12 14:51:43.891987 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.735( v 173'2803 (0'0,173'2803] local-les=137 n=306 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2802 mlcod 173'2802 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892001 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.762 at 2014-06-11 13:02:32.544735
2014-06-12 14:51:43.892006 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.891997 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  aec73043/rbd_data.10a36b8b4567.000000000000026c/head//3 aec73043/rbd_data.10a36b8b4567.000000000000026c/head//3(173'2126 client.4300.0:118396 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2126)
2014-06-12 14:51:43.892008 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.762( v 173'2726 (0'0,173'2726] local-les=147 n=302 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892021 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.76e at 2014-06-11 13:02:32.545259
2014-06-12 14:51:43.892026 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.892022 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  afa63043/rbd_data.10a16b8b4567.0000000000000ac6/head//3 afa63043/rbd_data.10a16b8b4567.0000000000000ac6/head//3(173'2343 client.4298.0:136302 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2343)
2014-06-12 14:51:43.892028 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2648 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892041 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.77e at 2014-06-11 13:02:32.545813
2014-06-12 14:51:43.892046 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.892043 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  2e553043/rbd_data.10a26b8b4567.000000000000333f/head//3 2e553043/rbd_data.10a26b8b4567.000000000000333f/head//3(173'1850 client.4299.0:104266 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1850)
2014-06-12 14:51:43.892048 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.77e( v 173'3052 (79'51,173'3052] local-les=161 n=319 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3051 mlcod 173'3051 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892062 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.783 at 2014-06-11 13:02:32.546436
2014-06-12 14:51:43.892068 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.892065 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  41043043/rbd_data.10a46b8b4567.0000000000000b8d/head//3 41043043/rbd_data.10a46b8b4567.0000000000000b8d/head//3(173'2390 client.4301.0:137027 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2390)
2014-06-12 14:51:43.892070 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892085 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.785 at 2014-06-11 13:02:32.547026
2014-06-12 14:51:43.892090 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.892087 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  2f733043/rbd_data.109d6b8b4567.0000000000002ffe/head//3 2f733043/rbd_data.109d6b8b4567.0000000000002ffe/head//3(173'1580 client.4294.0:95423 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1580)
2014-06-12 14:51:43.892092 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892105 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.793 at 2014-06-11 13:02:32.547568
2014-06-12 14:51:43.892111 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.892113 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.793( v 173'2605 (0'0,173'2605] local-les=153 n=277 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2604 mlcod 173'2604 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892108 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  95e23043/rbd_data.108d6b8b4567.000000000000348f/head//3 95e23043/rbd_data.108d6b8b4567.000000000000348f/head//3(173'1774 client.4278.0:105431 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1774)
2014-06-12 14:51:43.892125 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ab at 2014-06-11 13:02:32.548162
2014-06-12 14:51:43.892131 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.892133 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ab( v 173'2921 (0'0,173'2921] local-les=161 n=302 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2920 mlcod 173'2920 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892130 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  195e2043/rbd_data.10a46b8b4567.0000000000000cd0/head//3 195e2043/rbd_data.10a46b8b4567.0000000000000cd0/head//3(173'2409 client.4301.0:139160 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2409)
2014-06-12 14:51:43.892146 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ad at 2014-06-11 13:02:32.548789
2014-06-12 14:51:43.892151 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.892154 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ad( v 173'3043 (79'42,173'3043] local-les=149 n=315 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892152 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d89d2043/rbd_data.108e6b8b4567.0000000000001fc5/head//3 d89d2043/rbd_data.108e6b8b4567.0000000000001fc5/head//3(173'1206 client.4279.0:67009 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1206)
2014-06-12 14:51:43.892167 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ae at 2014-06-11 13:02:32.549514
2014-06-12 14:51:43.892173 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.892176 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2517 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892173 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  3d3c2043/rbd_data.10986b8b4567.00000000000010d8/head//3 3d3c2043/rbd_data.10986b8b4567.00000000000010d8/head//3(173'2560 client.4289.0:143066 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2560)
2014-06-12 14:51:43.892188 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7bc at 2014-06-11 13:02:32.550208
2014-06-12 14:51:43.892198 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.892190 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:43.892201 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7bc( v 173'2562 (0'0,173'2562] local-les=153 n=287 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2561 mlcod 173'2561 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892214 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7cc at 2014-06-11 13:02:32.550758
2014-06-12 14:51:43.892209 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:43.892220 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.892223 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:43.892227 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:43.892222 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892236 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7db at 2014-06-11 13:02:32.551269
2014-06-12 14:51:43.892230 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:43.892241 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.892244 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2681 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892256 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7f8 at 2014-06-11 13:02:32.551909
2014-06-12 14:51:43.892261 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:43.892263 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:43.892275 7f9d84175700 10 osd.2 173 sched_scrub 0.3 at 2014-06-12 13:01:39.405748: 6604.12 < min (86400 seconds)
2014-06-12 14:51:43.892295 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:43.892298 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:43.892300 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:43.892303 7f9d60bd2700 10 collection_list_partial 3.43_head start:a9a55043//0//-1 is_max:0
2014-06-12 14:51:43.892655 7f9d60bd2700 20 objects: [63085043/rbd_data.108d6b8b4567.0000000000003728/head//3,68a95043/rbd_data.10ac6b8b4567.0000000000000aaf/head//3,efbf5043/rbd_data.10b06b8b4567.000000000000338e/head//3,4d436043/rbd_data.109e6b8b4567.00000000000039f8/head//3,b7a66043/rbd_data.10926b8b4567.00000000000021e8/head//3,82d76043/rbd_data.10926b8b4567.00000000000001be/head//3,f1a86043/rbd_data.10b16b8b4567.0000000000001c03/head//3,b1d96043/rbd_data.109c6b8b4567.0000000000000eef/head//3,578b6043/rbd_data.10ad6b8b4567.00000000000000d8/head//3,d9b6043/rbd_data.10906b8b4567.0000000000000124/head//3,4a1f6043/rbd_data.10936b8b4567.0000000000003bf0/head//3,50217043/rbd_data.109c6b8b4567.0000000000001d6c/head//3,7a027043/rbd_data.109d6b8b4567.000000000000280e/head//3,a9c57043/rbd_data.10b26b8b4567.0000000000001430/head//3,17367043/rbd_data.10986b8b4567.0000000000001dd2/head//3,49177043/rbd_data.10956b8b4567.000000000000220a/head//3,496c7043/rbd_data.10b36b8b4567.000000000000374d/head//3,4d3f7043/rbd_data.10a86b8b4567.000000000000301e/head//3,9aff7043/rbd_data.10a26b8b4567.0000000000000a20/head//3,3eb18043/rbd_data.10976b8b4567.0000000000001a41/head//3,9d268043/rbd_data.10b26b8b4567.0000000000000db5/head//3,5a6a8043/rbd_data.10b26b8b4567.00000000000033a9/head//3,375c8043/rbd_data.10936b8b4567.0000000000001fcc/head//3,596c8043/rbd_data.109a6b8b4567.0000000000003a6a/head//3,279f8043/rbd_data.108f6b8b4567.0000000000003741/head//3]
2014-06-12 14:51:43.893323 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  requesting scrubmap from osd.24
2014-06-12 14:51:43.893356 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:43.893388 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:43.893402 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:43.893413 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:43.893423 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:43.893433 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_map_chunk [a9a55043//0//-1,6fdf8043//0//-1)
2014-06-12 14:51:43.893446 7f9d60bd2700 10 collection_list_partial 3.43_head start:a9a55043//0//-1 is_max:0
2014-06-12 14:51:43.894363 7f9d60bd2700 20 objects: [63085043/rbd_data.108d6b8b4567.0000000000003728/head//3,68a95043/rbd_data.10ac6b8b4567.0000000000000aaf/head//3,efbf5043/rbd_data.10b06b8b4567.000000000000338e/head//3,4d436043/rbd_data.109e6b8b4567.00000000000039f8/head//3,b7a66043/rbd_data.10926b8b4567.00000000000021e8/head//3,82d76043/rbd_data.10926b8b4567.00000000000001be/head//3,f1a86043/rbd_data.10b16b8b4567.0000000000001c03/head//3,b1d96043/rbd_data.109c6b8b4567.0000000000000eef/head//3,578b6043/rbd_data.10ad6b8b4567.00000000000000d8/head//3,d9b6043/rbd_data.10906b8b4567.0000000000000124/head//3,4a1f6043/rbd_data.10936b8b4567.0000000000003bf0/head//3,50217043/rbd_data.109c6b8b4567.0000000000001d6c/head//3,7a027043/rbd_data.109d6b8b4567.000000000000280e/head//3,a9c57043/rbd_data.10b26b8b4567.0000000000001430/head//3,17367043/rbd_data.10986b8b4567.0000000000001dd2/head//3,49177043/rbd_data.10956b8b4567.000000000000220a/head//3,496c7043/rbd_data.10b36b8b4567.000000000000374d/head//3,4d3f7043/rbd_data.10a86b8b4567.000000000000301e/head//3,9aff7043/rbd_data.10a26b8b4567.0000000000000a20/head//3,3eb18043/rbd_data.10976b8b4567.0000000000001a41/head//3,9d268043/rbd_data.10b26b8b4567.0000000000000db5/head//3,5a6a8043/rbd_data.10b26b8b4567.00000000000033a9/head//3,375c8043/rbd_data.10936b8b4567.0000000000001fcc/head//3,596c8043/rbd_data.109a6b8b4567.0000000000003a6a/head//3,279f8043/rbd_data.108f6b8b4567.0000000000003741/head//3,6fdf8043/rbd_data.10a06b8b4567.0000000000000294/head//3,f2039043/rbd_data.10996b8b4567.000000000000391e/head//3,f2a39043/rbd_data.109d6b8b4567.0000000000000a4e/head//3,29549043/rbd_data.10aa6b8b4567.0000000000001230/head//3,30569043/rbd_data.10b46b8b4567.0000000000003921/head//3,b089043/rbd_data.10b26b8b4567.0000000000000466/head//3,9da89043/rbd_data.10a46b8b4567.000000000000064f/head//3,e1199043/rbd_data.10ae6b8b4567.00000000000020b0/head//3,44599043/rbd_data.109b6b8b4567.00000000000023c8/head//3,81da9043/rbd_data.10af6b8b4567.0000000000003ae6/head//3,c1fa9043/rbd_data.10ac6b8b4567.0000000000003695/head//3,9cac9043/rbd_data.10956b8b4567.0000000000002d9f/head//3,cb7d9043/rbd_data.10b06b8b4567.00000000000008a4/head//3,da0a043/rbd_data.10946b8b4567.0000000000003178/head//3,1e21a043/rbd_data.109b6b8b4567.0000000000001b40/head//3,4dc3a043/rbd_data.10aa6b8b4567.000000000000243c/head//3,d374a043/rbd_data.10976b8b4567.0000000000003917/head//3,bb74a043/rbd_data.10a66b8b4567.0000000000000d90/head//3,a495a043/rbd_data.108e6b8b4567.0000000000000538/head//3,55a6a043/rbd_data.10a96b8b4567.0000000000001210/head//3,15e7a043/rbd_data.10b26b8b4567.00000000000006f4/head//3,ae02b043/rbd_data.108d6b8b4567.000000000000220e/head//3,c792b043/rbd_data.10a16b8b4567.00000000000005ea/head//3,a9b3b043/rbd_data.109a6b8b4567.00000000000011fb/head//3,53b4b043/rbd_data.10b06b8b4567.000000000000202d/head//3,71f6b043/rbd_data.109e6b8b4567.000000000000013e/head//3,78b8b043/rbd_data.109a6b8b4567.0000000000001c36/head//3,aaeab043/rbd_data.10986b8b4567.0000000000002e32/head//3,f0cbb043/rbd_data.109b6b8b4567.0000000000002af6/head//3,6fcbb043/rbd_data.10976b8b4567.0000000000001390/head//3,848cb043/rbd_data.109c6b8b4567.0000000000000ffd/head//3,f9aeb043/rbd_data.10b16b8b4567.000000000000019e/head//3,a5e0c043/rbd_data.109e6b8b4567.000000000000148a/head//3,1852c043/rbd_data.10a16b8b4567.0000000000000fce/head//3,f715c043/rbd_data.10a06b8b4567.00000000000017b1/head//3,5006c043/rbd_data.108e6b8b4567.00000000000028e1/head//3,f438c043/rbd_data.10b16b8b4567.0000000000000ab9/head//3,9f0bc043/rbd_data.108f6b8b4567.000000000000315a/head//3,e75cc043/rbd_data.10906b8b4567.00000000000024ce/head//3]
2014-06-12 14:51:43.894498 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:43.894523 7f9d60bd2700 10 stat 3.43_head/63085043/rbd_data.108d6b8b4567.0000000000003728/head//3
2014-06-12 14:51:43.894741 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 63085043/rbd_data.108d6b8b4567.0000000000003728/head//3
2014-06-12 14:51:43.925919 7f9d60bd2700 10 getattrs 3.43_head/63085043/rbd_data.108d6b8b4567.0000000000003728/head//3 = 0
2014-06-12 14:51:43.925941 7f9d60bd2700 10 stat 3.43_head/68a95043/rbd_data.10ac6b8b4567.0000000000000aaf/head//3
2014-06-12 14:51:43.926129 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 68a95043/rbd_data.10ac6b8b4567.0000000000000aaf/head//3
2014-06-12 14:51:43.977232 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:43.981237 7f9d60bd2700 10 getattrs 3.43_head/68a95043/rbd_data.10ac6b8b4567.0000000000000aaf/head//3 = 0
2014-06-12 14:51:43.981249 7f9d60bd2700 10 stat 3.43_head/efbf5043/rbd_data.10b06b8b4567.000000000000338e/head//3
2014-06-12 14:51:43.981386 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid efbf5043/rbd_data.10b06b8b4567.000000000000338e/head//3
2014-06-12 14:51:44.013896 7f9d60bd2700 10 getattrs 3.43_head/efbf5043/rbd_data.10b06b8b4567.000000000000338e/head//3 = 0
2014-06-12 14:51:44.013907 7f9d60bd2700 10 stat 3.43_head/4d436043/rbd_data.109e6b8b4567.00000000000039f8/head//3
2014-06-12 14:51:44.014041 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 4d436043/rbd_data.109e6b8b4567.00000000000039f8/head//3
2014-06-12 14:51:44.049279 7f9d60bd2700 10 getattrs 3.43_head/4d436043/rbd_data.109e6b8b4567.00000000000039f8/head//3 = 0
2014-06-12 14:51:44.049287 7f9d60bd2700 10 stat 3.43_head/b7a66043/rbd_data.10926b8b4567.00000000000021e8/head//3
2014-06-12 14:51:44.049389 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid b7a66043/rbd_data.10926b8b4567.00000000000021e8/head//3
2014-06-12 14:51:44.064860 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:44.080333 7f9d60bd2700 10 getattrs 3.43_head/b7a66043/rbd_data.10926b8b4567.00000000000021e8/head//3 = 0
2014-06-12 14:51:44.080343 7f9d60bd2700 10 stat 3.43_head/82d76043/rbd_data.10926b8b4567.00000000000001be/head//3
2014-06-12 14:51:44.080454 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 82d76043/rbd_data.10926b8b4567.00000000000001be/head//3
2014-06-12 14:51:44.121945 7f9d60bd2700 10 getattrs 3.43_head/82d76043/rbd_data.10926b8b4567.00000000000001be/head//3 = 0
2014-06-12 14:51:44.121961 7f9d60bd2700 10 stat 3.43_head/f1a86043/rbd_data.10b16b8b4567.0000000000001c03/head//3
2014-06-12 14:51:44.122072 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f1a86043/rbd_data.10b16b8b4567.0000000000001c03/head//3
2014-06-12 14:51:44.134088 7f9d60bd2700 10 getattrs 3.43_head/f1a86043/rbd_data.10b16b8b4567.0000000000001c03/head//3 = 0
2014-06-12 14:51:44.134098 7f9d60bd2700 10 stat 3.43_head/b1d96043/rbd_data.109c6b8b4567.0000000000000eef/head//3
2014-06-12 14:51:44.134205 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid b1d96043/rbd_data.109c6b8b4567.0000000000000eef/head//3
2014-06-12 14:51:44.149161 7f9d60bd2700 10 getattrs 3.43_head/b1d96043/rbd_data.109c6b8b4567.0000000000000eef/head//3 = 0
2014-06-12 14:51:44.149173 7f9d60bd2700 10 stat 3.43_head/578b6043/rbd_data.10ad6b8b4567.00000000000000d8/head//3
2014-06-12 14:51:44.149298 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 578b6043/rbd_data.10ad6b8b4567.00000000000000d8/head//3
2014-06-12 14:51:44.167736 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:44.178397 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:44.178616 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:44.194227 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:44.201363 7f9d60bd2700 10 getattrs 3.43_head/578b6043/rbd_data.10ad6b8b4567.00000000000000d8/head//3 = 0
2014-06-12 14:51:44.201377 7f9d60bd2700 10 stat 3.43_head/d9b6043/rbd_data.10906b8b4567.0000000000000124/head//3
2014-06-12 14:51:44.201511 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d9b6043/rbd_data.10906b8b4567.0000000000000124/head//3
2014-06-12 14:51:44.262095 7f9d60bd2700 10 getattrs 3.43_head/d9b6043/rbd_data.10906b8b4567.0000000000000124/head//3 = 0
2014-06-12 14:51:44.262109 7f9d60bd2700 10 stat 3.43_head/4a1f6043/rbd_data.10936b8b4567.0000000000003bf0/head//3
2014-06-12 14:51:44.262238 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 4a1f6043/rbd_data.10936b8b4567.0000000000003bf0/head//3
2014-06-12 14:51:44.262336 7f9d60bd2700 10 getattrs 3.43_head/4a1f6043/rbd_data.10936b8b4567.0000000000003bf0/head//3 = 0
2014-06-12 14:51:44.262343 7f9d60bd2700 10 stat 3.43_head/50217043/rbd_data.109c6b8b4567.0000000000001d6c/head//3
2014-06-12 14:51:44.262457 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 50217043/rbd_data.109c6b8b4567.0000000000001d6c/head//3
2014-06-12 14:51:44.274267 7f9d60bd2700 10 getattrs 3.43_head/50217043/rbd_data.109c6b8b4567.0000000000001d6c/head//3 = 0
2014-06-12 14:51:44.274274 7f9d60bd2700 10 stat 3.43_head/7a027043/rbd_data.109d6b8b4567.000000000000280e/head//3
2014-06-12 14:51:44.274371 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 7a027043/rbd_data.109d6b8b4567.000000000000280e/head//3
2014-06-12 14:51:44.297274 7f9d60bd2700 10 getattrs 3.43_head/7a027043/rbd_data.109d6b8b4567.000000000000280e/head//3 = 0
2014-06-12 14:51:44.297285 7f9d60bd2700 10 stat 3.43_head/a9c57043/rbd_data.10b26b8b4567.0000000000001430/head//3
2014-06-12 14:51:44.297387 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid a9c57043/rbd_data.10b26b8b4567.0000000000001430/head//3
2014-06-12 14:51:44.317147 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:44.317736 7f9d60bd2700 10 getattrs 3.43_head/a9c57043/rbd_data.10b26b8b4567.0000000000001430/head//3 = 0
2014-06-12 14:51:44.317753 7f9d60bd2700 10 stat 3.43_head/17367043/rbd_data.10986b8b4567.0000000000001dd2/head//3
2014-06-12 14:51:44.317934 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 17367043/rbd_data.10986b8b4567.0000000000001dd2/head//3
2014-06-12 14:51:44.326537 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:44.349324 7f9d60bd2700 10 getattrs 3.43_head/17367043/rbd_data.10986b8b4567.0000000000001dd2/head//3 = 0
2014-06-12 14:51:44.349339 7f9d60bd2700 10 stat 3.43_head/49177043/rbd_data.10956b8b4567.000000000000220a/head//3
2014-06-12 14:51:44.349434 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 49177043/rbd_data.10956b8b4567.000000000000220a/head//3
2014-06-12 14:51:44.366516 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:44.371448 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:44.376719 7f9d60bd2700 10 getattrs 3.43_head/49177043/rbd_data.10956b8b4567.000000000000220a/head//3 = 0
2014-06-12 14:51:44.376735 7f9d60bd2700 10 stat 3.43_head/496c7043/rbd_data.10b36b8b4567.000000000000374d/head//3
2014-06-12 14:51:44.376891 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 496c7043/rbd_data.10b36b8b4567.000000000000374d/head//3
2014-06-12 14:51:44.382175 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:44.393592 7f9d60bd2700 10 getattrs 3.43_head/496c7043/rbd_data.10b36b8b4567.000000000000374d/head//3 = 0
2014-06-12 14:51:44.393606 7f9d60bd2700 10 stat 3.43_head/4d3f7043/rbd_data.10a86b8b4567.000000000000301e/head//3
2014-06-12 14:51:44.393744 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 4d3f7043/rbd_data.10a86b8b4567.000000000000301e/head//3
2014-06-12 14:51:44.408687 7f9d60bd2700 10 getattrs 3.43_head/4d3f7043/rbd_data.10a86b8b4567.000000000000301e/head//3 = 0
2014-06-12 14:51:44.408697 7f9d60bd2700 10 stat 3.43_head/9aff7043/rbd_data.10a26b8b4567.0000000000000a20/head//3
2014-06-12 14:51:44.408788 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9aff7043/rbd_data.10a26b8b4567.0000000000000a20/head//3
2014-06-12 14:51:44.455266 7f9d60bd2700 10 getattrs 3.43_head/9aff7043/rbd_data.10a26b8b4567.0000000000000a20/head//3 = 0
2014-06-12 14:51:44.455281 7f9d60bd2700 10 stat 3.43_head/3eb18043/rbd_data.10976b8b4567.0000000000001a41/head//3
2014-06-12 14:51:44.455405 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 3eb18043/rbd_data.10976b8b4567.0000000000001a41/head//3
2014-06-12 14:51:44.487818 7f9d60bd2700 10 getattrs 3.43_head/3eb18043/rbd_data.10976b8b4567.0000000000001a41/head//3 = 0
2014-06-12 14:51:44.487826 7f9d60bd2700 10 stat 3.43_head/9d268043/rbd_data.10b26b8b4567.0000000000000db5/head//3
2014-06-12 14:51:44.487911 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9d268043/rbd_data.10b26b8b4567.0000000000000db5/head//3
2014-06-12 14:51:44.499713 7f9d5a16f700  5 -- op tracker -- , seq: 748597, time: 2014-06-12 14:51:44.499596, event: header_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:44.499731 7f9d5a16f700  5 -- op tracker -- , seq: 748597, time: 2014-06-12 14:51:44.499600, event: throttled, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:44.499745 7f9d5a16f700  5 -- op tracker -- , seq: 748597, time: 2014-06-12 14:51:44.499661, event: all_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:44.499756 7f9d5a16f700  5 -- op tracker -- , seq: 748597, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:44.499765 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:44.499773 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:51:44.499784 7f9d5a16f700 15 osd.2 173 enqueue_op 0x2a903600 prio 127 cost 10717 latency 0.000188 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:44.499853 7f9d65bdc700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:44.501827 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:44.502083 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:44.519243 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:44.521734 7f9d60bd2700 10 getattrs 3.43_head/9d268043/rbd_data.10b26b8b4567.0000000000000db5/head//3 = 0
2014-06-12 14:51:44.521754 7f9d60bd2700 10 stat 3.43_head/5a6a8043/rbd_data.10b26b8b4567.00000000000033a9/head//3
2014-06-12 14:51:44.521862 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 5a6a8043/rbd_data.10b26b8b4567.00000000000033a9/head//3
2014-06-12 14:51:44.522320 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:44.523164 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:44.532614 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:44.552756 7f9d60bd2700 10 getattrs 3.43_head/5a6a8043/rbd_data.10b26b8b4567.00000000000033a9/head//3 = 0
2014-06-12 14:51:44.552771 7f9d60bd2700 10 stat 3.43_head/375c8043/rbd_data.10936b8b4567.0000000000001fcc/head//3
2014-06-12 14:51:44.552867 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 375c8043/rbd_data.10936b8b4567.0000000000001fcc/head//3
2014-06-12 14:51:44.552954 7f9d60bd2700 10 getattrs 3.43_head/375c8043/rbd_data.10936b8b4567.0000000000001fcc/head//3 = 0
2014-06-12 14:51:44.552959 7f9d60bd2700 10 stat 3.43_head/596c8043/rbd_data.109a6b8b4567.0000000000003a6a/head//3
2014-06-12 14:51:44.553045 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 596c8043/rbd_data.109a6b8b4567.0000000000003a6a/head//3
2014-06-12 14:51:44.569436 7f9d60bd2700 10 getattrs 3.43_head/596c8043/rbd_data.109a6b8b4567.0000000000003a6a/head//3 = 0
2014-06-12 14:51:44.569447 7f9d60bd2700 10 stat 3.43_head/279f8043/rbd_data.108f6b8b4567.0000000000003741/head//3
2014-06-12 14:51:44.569556 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 279f8043/rbd_data.108f6b8b4567.0000000000003741/head//3
2014-06-12 14:51:44.569656 7f9d60bd2700 10 getattrs 3.43_head/279f8043/rbd_data.108f6b8b4567.0000000000003741/head//3 = 0
2014-06-12 14:51:44.569663 7f9d60bd2700 10 collection_getattrs 3.43_head
2014-06-12 14:51:44.569712 7f9d60bd2700 10 collection_getattrs 3.43_head = 0
2014-06-12 14:51:44.569714 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:44.569741 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:44.569749 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:44.569758 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:44.569761 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:44.569790 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2a903600 prio 127 cost 10717 latency 0.070193 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]
2014-06-12 14:51:44.569843 7f9d65bdc700  5 -- op tracker -- , seq: 748597, time: 2014-06-12 14:51:44.569834, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:44.569864 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] handle_message: 0x2a903600
2014-06-12 14:51:44.569880 7f9d65bdc700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:44.569897 7f9d65bdc700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:44.569914 7f9d65bdc700  5 -- op tracker -- , seq: 748597, time: 2014-06-12 14:51:44.569909, event: started, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:44.569921 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  got 24 scrub map
2014-06-12 14:51:44.570023 7f9d65bdc700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] map version is 173'2566
2014-06-12 14:51:44.570044 7f9d65bdc700 10 osd.2 173 dequeue_op 0x2a903600 finish
2014-06-12 14:51:44.570056 7f9d65bdc700  5 -- op tracker -- , seq: 748597, time: 2014-06-12 14:51:44.570050, event: done, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:44.570078 7f9d65bdc700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:44.570082 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:44.570084 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:44.570098 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:44.570120 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:44.570140 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:44.570177 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:44.570185 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:44.570192 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub replica 24 has 25 items
2014-06-12 14:51:44.570211 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 63085043/rbd_data.108d6b8b4567.0000000000003728/head//3, auth == maps.end()
2014-06-12 14:51:44.570228 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 63085043/rbd_data.108d6b8b4567.0000000000003728/head//3
2014-06-12 14:51:44.570241 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 68a95043/rbd_data.10ac6b8b4567.0000000000000aaf/head//3, auth == maps.end()
2014-06-12 14:51:44.570250 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 68a95043/rbd_data.10ac6b8b4567.0000000000000aaf/head//3
2014-06-12 14:51:44.570259 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj efbf5043/rbd_data.10b06b8b4567.000000000000338e/head//3, auth == maps.end()
2014-06-12 14:51:44.570268 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj efbf5043/rbd_data.10b06b8b4567.000000000000338e/head//3
2014-06-12 14:51:44.570277 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4d436043/rbd_data.109e6b8b4567.00000000000039f8/head//3, auth == maps.end()
2014-06-12 14:51:44.570286 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 4d436043/rbd_data.109e6b8b4567.00000000000039f8/head//3
2014-06-12 14:51:44.570295 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b7a66043/rbd_data.10926b8b4567.00000000000021e8/head//3, auth == maps.end()
2014-06-12 14:51:44.570304 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj b7a66043/rbd_data.10926b8b4567.00000000000021e8/head//3
2014-06-12 14:51:44.570313 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 82d76043/rbd_data.10926b8b4567.00000000000001be/head//3, auth == maps.end()
2014-06-12 14:51:44.570322 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 82d76043/rbd_data.10926b8b4567.00000000000001be/head//3
2014-06-12 14:51:44.570331 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f1a86043/rbd_data.10b16b8b4567.0000000000001c03/head//3, auth == maps.end()
2014-06-12 14:51:44.570340 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f1a86043/rbd_data.10b16b8b4567.0000000000001c03/head//3
2014-06-12 14:51:44.570349 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b1d96043/rbd_data.109c6b8b4567.0000000000000eef/head//3, auth == maps.end()
2014-06-12 14:51:44.570357 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj b1d96043/rbd_data.109c6b8b4567.0000000000000eef/head//3
2014-06-12 14:51:44.570367 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 578b6043/rbd_data.10ad6b8b4567.00000000000000d8/head//3, auth == maps.end()
2014-06-12 14:51:44.570375 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 578b6043/rbd_data.10ad6b8b4567.00000000000000d8/head//3
2014-06-12 14:51:44.570385 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d9b6043/rbd_data.10906b8b4567.0000000000000124/head//3, auth == maps.end()
2014-06-12 14:51:44.570393 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d9b6043/rbd_data.10906b8b4567.0000000000000124/head//3
2014-06-12 14:51:44.570405 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4a1f6043/rbd_data.10936b8b4567.0000000000003bf0/head//3, auth == maps.end()
2014-06-12 14:51:44.570414 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 4a1f6043/rbd_data.10936b8b4567.0000000000003bf0/head//3
2014-06-12 14:51:44.570424 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 50217043/rbd_data.109c6b8b4567.0000000000001d6c/head//3, auth == maps.end()
2014-06-12 14:51:44.570435 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 50217043/rbd_data.109c6b8b4567.0000000000001d6c/head//3
2014-06-12 14:51:44.570445 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7a027043/rbd_data.109d6b8b4567.000000000000280e/head//3, auth == maps.end()
2014-06-12 14:51:44.570455 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 7a027043/rbd_data.109d6b8b4567.000000000000280e/head//3
2014-06-12 14:51:44.570465 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a9c57043/rbd_data.10b26b8b4567.0000000000001430/head//3, auth == maps.end()
2014-06-12 14:51:44.570475 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj a9c57043/rbd_data.10b26b8b4567.0000000000001430/head//3
2014-06-12 14:51:44.570484 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 17367043/rbd_data.10986b8b4567.0000000000001dd2/head//3, auth == maps.end()
2014-06-12 14:51:44.570494 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 17367043/rbd_data.10986b8b4567.0000000000001dd2/head//3
2014-06-12 14:51:44.570504 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 49177043/rbd_data.10956b8b4567.000000000000220a/head//3, auth == maps.end()
2014-06-12 14:51:44.570514 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 49177043/rbd_data.10956b8b4567.000000000000220a/head//3
2014-06-12 14:51:44.570524 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 496c7043/rbd_data.10b36b8b4567.000000000000374d/head//3, auth == maps.end()
2014-06-12 14:51:44.570532 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 496c7043/rbd_data.10b36b8b4567.000000000000374d/head//3
2014-06-12 14:51:44.570543 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4d3f7043/rbd_data.10a86b8b4567.000000000000301e/head//3, auth == maps.end()
2014-06-12 14:51:44.570552 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 4d3f7043/rbd_data.10a86b8b4567.000000000000301e/head//3
2014-06-12 14:51:44.570563 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9aff7043/rbd_data.10a26b8b4567.0000000000000a20/head//3, auth == maps.end()
2014-06-12 14:51:44.570574 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9aff7043/rbd_data.10a26b8b4567.0000000000000a20/head//3
2014-06-12 14:51:44.570583 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3eb18043/rbd_data.10976b8b4567.0000000000001a41/head//3, auth == maps.end()
2014-06-12 14:51:44.570593 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 3eb18043/rbd_data.10976b8b4567.0000000000001a41/head//3
2014-06-12 14:51:44.570603 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9d268043/rbd_data.10b26b8b4567.0000000000000db5/head//3, auth == maps.end()
2014-06-12 14:51:44.570613 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9d268043/rbd_data.10b26b8b4567.0000000000000db5/head//3
2014-06-12 14:51:44.570622 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5a6a8043/rbd_data.10b26b8b4567.00000000000033a9/head//3, auth == maps.end()
2014-06-12 14:51:44.570631 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 5a6a8043/rbd_data.10b26b8b4567.00000000000033a9/head//3
2014-06-12 14:51:44.570640 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 375c8043/rbd_data.10936b8b4567.0000000000001fcc/head//3, auth == maps.end()
2014-06-12 14:51:44.570648 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 375c8043/rbd_data.10936b8b4567.0000000000001fcc/head//3
2014-06-12 14:51:44.570657 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 596c8043/rbd_data.109a6b8b4567.0000000000003a6a/head//3, auth == maps.end()
2014-06-12 14:51:44.570666 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 596c8043/rbd_data.109a6b8b4567.0000000000003a6a/head//3
2014-06-12 14:51:44.570675 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 279f8043/rbd_data.108f6b8b4567.0000000000003741/head//3, auth == maps.end()
2014-06-12 14:51:44.570683 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 279f8043/rbd_data.108f6b8b4567.0000000000003741/head//3
2014-06-12 14:51:44.570693 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] 
2014-06-12 14:51:44.570700 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub
2014-06-12 14:51:44.570712 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  279f8043/rbd_data.108f6b8b4567.0000000000003741/head//3 279f8043/rbd_data.108f6b8b4567.0000000000003741/head//3(173'1897 client.4280.0:107740 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1897)
2014-06-12 14:51:44.570727 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  596c8043/rbd_data.109a6b8b4567.0000000000003a6a/head//3 596c8043/rbd_data.109a6b8b4567.0000000000003a6a/head//3(173'1958 client.4291.0:113524 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1958)
2014-06-12 14:51:44.570740 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  375c8043/rbd_data.10936b8b4567.0000000000001fcc/head//3 375c8043/rbd_data.10936b8b4567.0000000000001fcc/head//3(173'1231 client.4284.0:64775 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1231)
2014-06-12 14:51:44.570754 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  5a6a8043/rbd_data.10b26b8b4567.00000000000033a9/head//3 5a6a8043/rbd_data.10b26b8b4567.00000000000033a9/head//3(173'1694 client.4315.0:102895 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1694)
2014-06-12 14:51:44.570767 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9d268043/rbd_data.10b26b8b4567.0000000000000db5/head//3 9d268043/rbd_data.10b26b8b4567.0000000000000db5/head//3(173'2456 client.4315.0:140165 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2456)
2014-06-12 14:51:44.570778 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  3eb18043/rbd_data.10976b8b4567.0000000000001a41/head//3 3eb18043/rbd_data.10976b8b4567.0000000000001a41/head//3(173'1140 client.4288.0:55463 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1140)
2014-06-12 14:51:44.570791 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9aff7043/rbd_data.10a26b8b4567.0000000000000a20/head//3 9aff7043/rbd_data.10a26b8b4567.0000000000000a20/head//3(173'2379 client.4299.0:136408 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2379)
2014-06-12 14:51:44.570804 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  4d3f7043/rbd_data.10a86b8b4567.000000000000301e/head//3 4d3f7043/rbd_data.10a86b8b4567.000000000000301e/head//3(173'1568 client.4305.0:95486 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1568)
2014-06-12 14:51:44.570817 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  496c7043/rbd_data.10b36b8b4567.000000000000374d/head//3 496c7043/rbd_data.10b36b8b4567.000000000000374d/head//3(173'1927 client.4316.0:110662 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1927)
2014-06-12 14:51:44.570830 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  49177043/rbd_data.10956b8b4567.000000000000220a/head//3 49177043/rbd_data.10956b8b4567.000000000000220a/head//3(173'1261 client.4286.0:70424 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1261)
2014-06-12 14:51:44.570843 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  17367043/rbd_data.10986b8b4567.0000000000001dd2/head//3 17367043/rbd_data.10986b8b4567.0000000000001dd2/head//3(173'1134 client.4289.0:62067 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1134)
2014-06-12 14:51:44.570855 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  a9c57043/rbd_data.10b26b8b4567.0000000000001430/head//3 a9c57043/rbd_data.10b26b8b4567.0000000000001430/head//3(173'764 client.4315.0:45678 wrlock_by=unknown.0.0:0 dirty s 4194304 uv764)
2014-06-12 14:51:44.570868 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  7a027043/rbd_data.109d6b8b4567.000000000000280e/head//3 7a027043/rbd_data.109d6b8b4567.000000000000280e/head//3(173'1365 client.4294.0:81178 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1365)
2014-06-12 14:51:44.570881 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  50217043/rbd_data.109c6b8b4567.0000000000001d6c/head//3 50217043/rbd_data.109c6b8b4567.0000000000001d6c/head//3(173'1179 client.4293.0:61835 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1179)
2014-06-12 14:51:44.570892 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  4a1f6043/rbd_data.10936b8b4567.0000000000003bf0/head//3 4a1f6043/rbd_data.10936b8b4567.0000000000003bf0/head//3(173'2063 client.4284.0:115641 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2063)
2014-06-12 14:51:44.570905 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d9b6043/rbd_data.10906b8b4567.0000000000000124/head//3 d9b6043/rbd_data.10906b8b4567.0000000000000124/head//3(173'2086 client.4281.0:118425 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2086)
2014-06-12 14:51:44.570918 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  578b6043/rbd_data.10ad6b8b4567.00000000000000d8/head//3 578b6043/rbd_data.10ad6b8b4567.00000000000000d8/head//3(173'2091 client.4310.0:118540 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2091)
2014-06-12 14:51:44.570930 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  b1d96043/rbd_data.109c6b8b4567.0000000000000eef/head//3 b1d96043/rbd_data.109c6b8b4567.0000000000000eef/head//3(173'2512 client.4293.0:142023 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2512)
2014-06-12 14:51:44.570943 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f1a86043/rbd_data.10b16b8b4567.0000000000001c03/head//3 f1a86043/rbd_data.10b16b8b4567.0000000000001c03/head//3(173'1106 client.4314.0:60108 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1106)
2014-06-12 14:51:44.570956 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  82d76043/rbd_data.10926b8b4567.00000000000001be/head//3 82d76043/rbd_data.10926b8b4567.00000000000001be/head//3(173'2103 client.4283.0:119300 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2103)
2014-06-12 14:51:44.570969 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  b7a66043/rbd_data.10926b8b4567.00000000000021e8/head//3 b7a66043/rbd_data.10926b8b4567.00000000000021e8/head//3(173'1254 client.4283.0:69730 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1254)
2014-06-12 14:51:44.570981 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  4d436043/rbd_data.109e6b8b4567.00000000000039f8/head//3 4d436043/rbd_data.109e6b8b4567.00000000000039f8/head//3(173'1938 client.4295.0:113967 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1938)
2014-06-12 14:51:44.570994 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  efbf5043/rbd_data.10b06b8b4567.000000000000338e/head//3 efbf5043/rbd_data.10b06b8b4567.000000000000338e/head//3(173'1790 client.4313.0:104040 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1790)
2014-06-12 14:51:44.571007 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  68a95043/rbd_data.10ac6b8b4567.0000000000000aaf/head//3 68a95043/rbd_data.10ac6b8b4567.0000000000000aaf/head//3(173'2364 client.4309.0:133343 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2364)
2014-06-12 14:51:44.571018 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  63085043/rbd_data.108d6b8b4567.0000000000003728/head//3 63085043/rbd_data.108d6b8b4567.0000000000003728/head//3(173'1829 client.4278.0:110017 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1829)
2014-06-12 14:51:44.571028 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:44.571038 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:44.571047 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:44.571049 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:44.571051 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:44.571085 7f9d60bd2700 10 collection_list_partial 3.43_head start:6fdf8043//0//-1 is_max:0
2014-06-12 14:51:44.571308 7f9d60bd2700 20 objects: [f2039043/rbd_data.10996b8b4567.000000000000391e/head//3,f2a39043/rbd_data.109d6b8b4567.0000000000000a4e/head//3,29549043/rbd_data.10aa6b8b4567.0000000000001230/head//3,30569043/rbd_data.10b46b8b4567.0000000000003921/head//3,b089043/rbd_data.10b26b8b4567.0000000000000466/head//3,9da89043/rbd_data.10a46b8b4567.000000000000064f/head//3,e1199043/rbd_data.10ae6b8b4567.00000000000020b0/head//3,44599043/rbd_data.109b6b8b4567.00000000000023c8/head//3,81da9043/rbd_data.10af6b8b4567.0000000000003ae6/head//3,c1fa9043/rbd_data.10ac6b8b4567.0000000000003695/head//3,9cac9043/rbd_data.10956b8b4567.0000000000002d9f/head//3,cb7d9043/rbd_data.10b06b8b4567.00000000000008a4/head//3,da0a043/rbd_data.10946b8b4567.0000000000003178/head//3,1e21a043/rbd_data.109b6b8b4567.0000000000001b40/head//3,4dc3a043/rbd_data.10aa6b8b4567.000000000000243c/head//3,d374a043/rbd_data.10976b8b4567.0000000000003917/head//3,bb74a043/rbd_data.10a66b8b4567.0000000000000d90/head//3,a495a043/rbd_data.108e6b8b4567.0000000000000538/head//3,55a6a043/rbd_data.10a96b8b4567.0000000000001210/head//3,15e7a043/rbd_data.10b26b8b4567.00000000000006f4/head//3,ae02b043/rbd_data.108d6b8b4567.000000000000220e/head//3,c792b043/rbd_data.10a16b8b4567.00000000000005ea/head//3,a9b3b043/rbd_data.109a6b8b4567.00000000000011fb/head//3,53b4b043/rbd_data.10b06b8b4567.000000000000202d/head//3,71f6b043/rbd_data.109e6b8b4567.000000000000013e/head//3]
2014-06-12 14:51:44.571777 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  requesting scrubmap from osd.24
2014-06-12 14:51:44.571806 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:44.571823 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:44.571831 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:44.571837 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:44.571843 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:44.571849 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_map_chunk [6fdf8043//0//-1,78b8b043//0//-1)
2014-06-12 14:51:44.571856 7f9d60bd2700 10 collection_list_partial 3.43_head start:6fdf8043//0//-1 is_max:0
2014-06-12 14:51:44.572328 7f9d60bd2700 20 objects: [f2039043/rbd_data.10996b8b4567.000000000000391e/head//3,f2a39043/rbd_data.109d6b8b4567.0000000000000a4e/head//3,29549043/rbd_data.10aa6b8b4567.0000000000001230/head//3,30569043/rbd_data.10b46b8b4567.0000000000003921/head//3,b089043/rbd_data.10b26b8b4567.0000000000000466/head//3,9da89043/rbd_data.10a46b8b4567.000000000000064f/head//3,e1199043/rbd_data.10ae6b8b4567.00000000000020b0/head//3,44599043/rbd_data.109b6b8b4567.00000000000023c8/head//3,81da9043/rbd_data.10af6b8b4567.0000000000003ae6/head//3,c1fa9043/rbd_data.10ac6b8b4567.0000000000003695/head//3,9cac9043/rbd_data.10956b8b4567.0000000000002d9f/head//3,cb7d9043/rbd_data.10b06b8b4567.00000000000008a4/head//3,da0a043/rbd_data.10946b8b4567.0000000000003178/head//3,1e21a043/rbd_data.109b6b8b4567.0000000000001b40/head//3,4dc3a043/rbd_data.10aa6b8b4567.000000000000243c/head//3,d374a043/rbd_data.10976b8b4567.0000000000003917/head//3,bb74a043/rbd_data.10a66b8b4567.0000000000000d90/head//3,a495a043/rbd_data.108e6b8b4567.0000000000000538/head//3,55a6a043/rbd_data.10a96b8b4567.0000000000001210/head//3,15e7a043/rbd_data.10b26b8b4567.00000000000006f4/head//3,ae02b043/rbd_data.108d6b8b4567.000000000000220e/head//3,c792b043/rbd_data.10a16b8b4567.00000000000005ea/head//3,a9b3b043/rbd_data.109a6b8b4567.00000000000011fb/head//3,53b4b043/rbd_data.10b06b8b4567.000000000000202d/head//3,71f6b043/rbd_data.109e6b8b4567.000000000000013e/head//3,78b8b043/rbd_data.109a6b8b4567.0000000000001c36/head//3,aaeab043/rbd_data.10986b8b4567.0000000000002e32/head//3,f0cbb043/rbd_data.109b6b8b4567.0000000000002af6/head//3,6fcbb043/rbd_data.10976b8b4567.0000000000001390/head//3,848cb043/rbd_data.109c6b8b4567.0000000000000ffd/head//3,f9aeb043/rbd_data.10b16b8b4567.000000000000019e/head//3,a5e0c043/rbd_data.109e6b8b4567.000000000000148a/head//3,1852c043/rbd_data.10a16b8b4567.0000000000000fce/head//3,f715c043/rbd_data.10a06b8b4567.00000000000017b1/head//3,5006c043/rbd_data.108e6b8b4567.00000000000028e1/head//3,f438c043/rbd_data.10b16b8b4567.0000000000000ab9/head//3,9f0bc043/rbd_data.108f6b8b4567.000000000000315a/head//3,e75cc043/rbd_data.10906b8b4567.00000000000024ce/head//3,dbbcc043/rbd_data.10a86b8b4567.00000000000039f7/head//3,afa1d043/rbd_data.10ab6b8b4567.0000000000000ef6/head//3,6e92d043/rbd_data.10a26b8b4567.0000000000000704/head//3,47b4d043/rbd_data.10b26b8b4567.00000000000022c6/head//3,7095d043/rbd_data.10a26b8b4567.00000000000018c1/head//3,58ead043/rbd_data.108f6b8b4567.00000000000030e3/head//3,53fad043/rbd_data.10a46b8b4567.0000000000003b8f/head//3,cb0ed043/rbd_data.109f6b8b4567.0000000000000d62/head//3,4754e043/rbd_data.10a66b8b4567.0000000000003a37/head//3,ee4e043/rbd_data.10a86b8b4567.00000000000018d6/head//3,da65e043/rbd_data.10b26b8b4567.00000000000039ce/head//3,373ae043/rbd_data.10b16b8b4567.00000000000026a2/head//3,893ae043/rbd_data.10ac6b8b4567.00000000000006cf/head//3,c16be043/rbd_data.10986b8b4567.0000000000001ec7/head//3,2e9be043/rbd_data.10ae6b8b4567.00000000000030f0/head//3,665fe043/rbd_data.10976b8b4567.000000000000125b/head//3,aff1f043/rbd_data.108d6b8b4567.00000000000000ae/head//3,a502f043/rbd_data.10b26b8b4567.0000000000002b98/head//3,6f24f043/rbd_data.109f6b8b4567.0000000000003727/head//3,8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3,13c5f043/rbd_data.10a76b8b4567.00000000000035af/head//3,84d8f043/rbd_data.108d6b8b4567.0000000000003506/head//3,9cd9f043/rbd_data.10b16b8b4567.0000000000002513/head//3,10baf043/rbd_data.10a86b8b4567.0000000000002ccc/head//3,902bf043/rbd_data.10b36b8b4567.0000000000000757/head//3,18cdf043/rbd_data.10ac6b8b4567.0000000000003b1c/head//3]
2014-06-12 14:51:44.572406 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:44.572422 7f9d60bd2700 10 stat 3.43_head/f2039043/rbd_data.10996b8b4567.000000000000391e/head//3
2014-06-12 14:51:44.572537 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f2039043/rbd_data.10996b8b4567.000000000000391e/head//3
2014-06-12 14:51:44.587245 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:44.587282 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:44.591842 7f9d60bd2700 10 getattrs 3.43_head/f2039043/rbd_data.10996b8b4567.000000000000391e/head//3 = 0
2014-06-12 14:51:44.591855 7f9d60bd2700 10 stat 3.43_head/f2a39043/rbd_data.109d6b8b4567.0000000000000a4e/head//3
2014-06-12 14:51:44.591957 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f2a39043/rbd_data.109d6b8b4567.0000000000000a4e/head//3
2014-06-12 14:51:44.631940 7f9d60bd2700 10 getattrs 3.43_head/f2a39043/rbd_data.109d6b8b4567.0000000000000a4e/head//3 = 0
2014-06-12 14:51:44.631968 7f9d60bd2700 10 stat 3.43_head/29549043/rbd_data.10aa6b8b4567.0000000000001230/head//3
2014-06-12 14:51:44.632129 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 29549043/rbd_data.10aa6b8b4567.0000000000001230/head//3
2014-06-12 14:51:44.655215 7f9d60bd2700 10 getattrs 3.43_head/29549043/rbd_data.10aa6b8b4567.0000000000001230/head//3 = 0
2014-06-12 14:51:44.655222 7f9d60bd2700 10 stat 3.43_head/30569043/rbd_data.10b46b8b4567.0000000000003921/head//3
2014-06-12 14:51:44.655317 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 30569043/rbd_data.10b46b8b4567.0000000000003921/head//3
2014-06-12 14:51:44.661965 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:44.668084 7f9d60bd2700 10 getattrs 3.43_head/30569043/rbd_data.10b46b8b4567.0000000000003921/head//3 = 0
2014-06-12 14:51:44.668090 7f9d60bd2700 10 stat 3.43_head/b089043/rbd_data.10b26b8b4567.0000000000000466/head//3
2014-06-12 14:51:44.668187 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid b089043/rbd_data.10b26b8b4567.0000000000000466/head//3
2014-06-12 14:51:44.680170 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:44.700806 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:44.717625 7f9d60bd2700 10 getattrs 3.43_head/b089043/rbd_data.10b26b8b4567.0000000000000466/head//3 = 0
2014-06-12 14:51:44.717638 7f9d60bd2700 10 stat 3.43_head/9da89043/rbd_data.10a46b8b4567.000000000000064f/head//3
2014-06-12 14:51:44.717747 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9da89043/rbd_data.10a46b8b4567.000000000000064f/head//3
2014-06-12 14:51:44.726591 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:44.768255 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:44.781041 7f9d60bd2700 10 getattrs 3.43_head/9da89043/rbd_data.10a46b8b4567.000000000000064f/head//3 = 0
2014-06-12 14:51:44.781056 7f9d60bd2700 10 stat 3.43_head/e1199043/rbd_data.10ae6b8b4567.00000000000020b0/head//3
2014-06-12 14:51:44.781168 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid e1199043/rbd_data.10ae6b8b4567.00000000000020b0/head//3
2014-06-12 14:51:44.795245 7f9d60bd2700 10 getattrs 3.43_head/e1199043/rbd_data.10ae6b8b4567.00000000000020b0/head//3 = 0
2014-06-12 14:51:44.795257 7f9d60bd2700 10 stat 3.43_head/44599043/rbd_data.109b6b8b4567.00000000000023c8/head//3
2014-06-12 14:51:44.795377 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 44599043/rbd_data.109b6b8b4567.00000000000023c8/head//3
2014-06-12 14:51:44.799478 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:44.799613 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:44.802186 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:44.807229 7f9d60bd2700 10 getattrs 3.43_head/44599043/rbd_data.109b6b8b4567.00000000000023c8/head//3 = 0
2014-06-12 14:51:44.807239 7f9d60bd2700 10 stat 3.43_head/81da9043/rbd_data.10af6b8b4567.0000000000003ae6/head//3
2014-06-12 14:51:44.807356 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 81da9043/rbd_data.10af6b8b4567.0000000000003ae6/head//3
2014-06-12 14:51:44.819091 7f9d60bd2700 10 getattrs 3.43_head/81da9043/rbd_data.10af6b8b4567.0000000000003ae6/head//3 = 0
2014-06-12 14:51:44.819100 7f9d60bd2700 10 stat 3.43_head/c1fa9043/rbd_data.10ac6b8b4567.0000000000003695/head//3
2014-06-12 14:51:44.819210 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c1fa9043/rbd_data.10ac6b8b4567.0000000000003695/head//3
2014-06-12 14:51:44.854244 7f9d60bd2700 10 getattrs 3.43_head/c1fa9043/rbd_data.10ac6b8b4567.0000000000003695/head//3 = 0
2014-06-12 14:51:44.854259 7f9d60bd2700 10 stat 3.43_head/9cac9043/rbd_data.10956b8b4567.0000000000002d9f/head//3
2014-06-12 14:51:44.854374 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9cac9043/rbd_data.10956b8b4567.0000000000002d9f/head//3
2014-06-12 14:51:44.855308 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:44.865695 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:44.869034 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:44.869826 7f9d60bd2700 10 getattrs 3.43_head/9cac9043/rbd_data.10956b8b4567.0000000000002d9f/head//3 = 0
2014-06-12 14:51:44.869838 7f9d60bd2700 10 stat 3.43_head/cb7d9043/rbd_data.10b06b8b4567.00000000000008a4/head//3
2014-06-12 14:51:44.869980 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid cb7d9043/rbd_data.10b06b8b4567.00000000000008a4/head//3
2014-06-12 14:51:44.892375 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:44.892398 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:51:44.892402 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:51:44.892406 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:51:44.892410 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:51:44.892461 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:44.892480 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.46 < max 0.5 = yes
2014-06-12 14:51:44.892499 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=1
2014-06-12 14:51:44.896263 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:44.896417 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:44.907131 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:44.907819 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:44.913973 7f9d60bd2700 10 getattrs 3.43_head/cb7d9043/rbd_data.10b06b8b4567.00000000000008a4/head//3 = 0
2014-06-12 14:51:44.913983 7f9d60bd2700 10 stat 3.43_head/da0a043/rbd_data.10946b8b4567.0000000000003178/head//3
2014-06-12 14:51:44.914096 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid da0a043/rbd_data.10946b8b4567.0000000000003178/head//3
2014-06-12 14:51:44.931312 7f9d60bd2700 10 getattrs 3.43_head/da0a043/rbd_data.10946b8b4567.0000000000003178/head//3 = 0
2014-06-12 14:51:44.931320 7f9d60bd2700 10 stat 3.43_head/1e21a043/rbd_data.109b6b8b4567.0000000000001b40/head//3
2014-06-12 14:51:44.931416 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 1e21a043/rbd_data.109b6b8b4567.0000000000001b40/head//3
2014-06-12 14:51:44.957781 7f9d60bd2700 10 getattrs 3.43_head/1e21a043/rbd_data.109b6b8b4567.0000000000001b40/head//3 = 0
2014-06-12 14:51:44.957791 7f9d60bd2700 10 stat 3.43_head/4dc3a043/rbd_data.10aa6b8b4567.000000000000243c/head//3
2014-06-12 14:51:44.957902 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 4dc3a043/rbd_data.10aa6b8b4567.000000000000243c/head//3
2014-06-12 14:51:44.962578 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:44.966285 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:44.974851 7f9d60bd2700 10 getattrs 3.43_head/4dc3a043/rbd_data.10aa6b8b4567.000000000000243c/head//3 = 0
2014-06-12 14:51:44.974863 7f9d60bd2700 10 stat 3.43_head/d374a043/rbd_data.10976b8b4567.0000000000003917/head//3
2014-06-12 14:51:44.974973 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d374a043/rbd_data.10976b8b4567.0000000000003917/head//3
2014-06-12 14:51:44.991481 7f9d60bd2700 10 getattrs 3.43_head/d374a043/rbd_data.10976b8b4567.0000000000003917/head//3 = 0
2014-06-12 14:51:44.991491 7f9d60bd2700 10 stat 3.43_head/bb74a043/rbd_data.10a66b8b4567.0000000000000d90/head//3
2014-06-12 14:51:44.991603 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid bb74a043/rbd_data.10a66b8b4567.0000000000000d90/head//3
2014-06-12 14:51:45.031473 7f9d60bd2700 10 getattrs 3.43_head/bb74a043/rbd_data.10a66b8b4567.0000000000000d90/head//3 = 0
2014-06-12 14:51:45.031486 7f9d60bd2700 10 stat 3.43_head/a495a043/rbd_data.108e6b8b4567.0000000000000538/head//3
2014-06-12 14:51:45.031600 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid a495a043/rbd_data.108e6b8b4567.0000000000000538/head//3
2014-06-12 14:51:45.072219 7f9d60bd2700 10 getattrs 3.43_head/a495a043/rbd_data.108e6b8b4567.0000000000000538/head//3 = 0
2014-06-12 14:51:45.072235 7f9d60bd2700 10 stat 3.43_head/55a6a043/rbd_data.10a96b8b4567.0000000000001210/head//3
2014-06-12 14:51:45.072405 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 55a6a043/rbd_data.10a96b8b4567.0000000000001210/head//3
2014-06-12 14:51:45.092130 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:45.092150 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:45.092541 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:45.092579 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:45.092804 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:45.092877 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:45.092974 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:45.093002 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:45.093030 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:45.093058 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:45.093086 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:45.093111 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:45.093136 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:45.093160 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:45.093184 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:45.093211 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:45.093241 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:45.093264 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:45.093288 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:45.093308 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:45.093333 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:45.093357 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:45.093380 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:45.093408 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:45.100763 7f9d60bd2700 10 getattrs 3.43_head/55a6a043/rbd_data.10a96b8b4567.0000000000001210/head//3 = 0
2014-06-12 14:51:45.100772 7f9d60bd2700 10 stat 3.43_head/15e7a043/rbd_data.10b26b8b4567.00000000000006f4/head//3
2014-06-12 14:51:45.100877 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 15e7a043/rbd_data.10b26b8b4567.00000000000006f4/head//3
2014-06-12 14:51:45.121628 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:45.133244 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:45.142671 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:45.143582 7f9d60bd2700 10 getattrs 3.43_head/15e7a043/rbd_data.10b26b8b4567.00000000000006f4/head//3 = 0
2014-06-12 14:51:45.143613 7f9d60bd2700 10 stat 3.43_head/ae02b043/rbd_data.108d6b8b4567.000000000000220e/head//3
2014-06-12 14:51:45.143780 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid ae02b043/rbd_data.108d6b8b4567.000000000000220e/head//3
2014-06-12 14:51:45.145475 7f9d5a16f700  5 -- op tracker -- , seq: 748598, time: 2014-06-12 14:51:45.145352, event: header_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.145494 7f9d5a16f700  5 -- op tracker -- , seq: 748598, time: 2014-06-12 14:51:45.145356, event: throttled, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.145505 7f9d5a16f700  5 -- op tracker -- , seq: 748598, time: 2014-06-12 14:51:45.145421, event: all_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.145538 7f9d5a16f700  5 -- op tracker -- , seq: 748598, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.145545 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:45.145554 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:51:45.145563 7f9d5a16f700 15 osd.2 173 enqueue_op 0x12bf6400 prio 127 cost 10717 latency 0.000211 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:45.145649 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:45.152114 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:45.158299 7f9d60bd2700 10 getattrs 3.43_head/ae02b043/rbd_data.108d6b8b4567.000000000000220e/head//3 = 0
2014-06-12 14:51:45.158310 7f9d60bd2700 10 stat 3.43_head/c792b043/rbd_data.10a16b8b4567.00000000000005ea/head//3
2014-06-12 14:51:45.158405 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c792b043/rbd_data.10a16b8b4567.00000000000005ea/head//3
2014-06-12 14:51:45.180578 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:45.193970 7f9d60bd2700 10 getattrs 3.43_head/c792b043/rbd_data.10a16b8b4567.00000000000005ea/head//3 = 0
2014-06-12 14:51:45.193986 7f9d60bd2700 10 stat 3.43_head/a9b3b043/rbd_data.109a6b8b4567.00000000000011fb/head//3
2014-06-12 14:51:45.194086 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid a9b3b043/rbd_data.109a6b8b4567.00000000000011fb/head//3
2014-06-12 14:51:45.211947 7f9d60bd2700 10 getattrs 3.43_head/a9b3b043/rbd_data.109a6b8b4567.00000000000011fb/head//3 = 0
2014-06-12 14:51:45.211958 7f9d60bd2700 10 stat 3.43_head/53b4b043/rbd_data.10b06b8b4567.000000000000202d/head//3
2014-06-12 14:51:45.212063 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 53b4b043/rbd_data.10b06b8b4567.000000000000202d/head//3
2014-06-12 14:51:45.242264 7f9d60bd2700 10 getattrs 3.43_head/53b4b043/rbd_data.10b06b8b4567.000000000000202d/head//3 = 0
2014-06-12 14:51:45.242278 7f9d60bd2700 10 stat 3.43_head/71f6b043/rbd_data.109e6b8b4567.000000000000013e/head//3
2014-06-12 14:51:45.242365 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 71f6b043/rbd_data.109e6b8b4567.000000000000013e/head//3
2014-06-12 14:51:45.277088 7f9d60bd2700 10 getattrs 3.43_head/71f6b043/rbd_data.109e6b8b4567.000000000000013e/head//3 = 0
2014-06-12 14:51:45.277106 7f9d60bd2700 10 collection_getattrs 3.43_head
2014-06-12 14:51:45.277157 7f9d60bd2700 10 collection_getattrs 3.43_head = 0
2014-06-12 14:51:45.277161 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:45.277188 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:45.277194 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:45.277203 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:45.277207 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:45.277231 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.43 at 2014-06-11 13:02:32.501298
2014-06-12 14:51:45.277269 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6b at 2014-06-11 13:02:32.502023
2014-06-12 14:51:45.277281 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277289 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277335 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6c at 2014-06-11 13:02:32.502351
2014-06-12 14:51:45.277344 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277346 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277332 7f9d623d5700 10 osd.2 173 dequeue_op 0x12bf6400 prio 127 cost 10717 latency 0.131979 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]
2014-06-12 14:51:45.277361 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.a0 at 2014-06-11 13:02:32.502930
2014-06-12 14:51:45.277369 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277372 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.a0( v 173'2803 (0'0,173'2803] local-les=157 n=294 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2802 mlcod 173'2802 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277378 7f9d623d5700  5 -- op tracker -- , seq: 748598, time: 2014-06-12 14:51:45.277371, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.277386 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.105 at 2014-06-11 13:02:32.503257
2014-06-12 14:51:45.277408 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277410 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.105( v 173'2818 (0'0,173'2818] local-les=153 n=298 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2817 mlcod 173'2817 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277408 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] handle_message: 0x12bf6400
2014-06-12 14:51:45.277418 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.122 at 2014-06-11 13:02:32.503770
2014-06-12 14:51:45.277420 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277421 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.122( v 173'2740 (0'0,173'2740] local-les=149 n=295 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2739 mlcod 173'2739 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277417 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:45.277428 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.12b at 2014-06-11 13:02:32.504419
2014-06-12 14:51:45.277431 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277431 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:45.277432 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.12b( v 173'3236 (79'235,173'3236] local-les=137 n=323 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3235 mlcod 173'3235 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277439 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.131 at 2014-06-11 13:02:32.504931
2014-06-12 14:51:45.277447 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277440 7f9d623d5700  5 -- op tracker -- , seq: 748598, time: 2014-06-12 14:51:45.277437, event: started, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.277448 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.131( v 173'2860 (0'0,173'2860] local-les=139 n=295 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277451 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  got 24 scrub map
2014-06-12 14:51:45.277455 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.15d at 2014-06-11 13:02:32.505494
2014-06-12 14:51:45.277458 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277459 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.15d( v 173'2853 (0'0,173'2853] local-les=149 n=305 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2852 mlcod 173'2852 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277465 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.165 at 2014-06-11 13:02:32.506051
2014-06-12 14:51:45.277468 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277469 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.165( v 173'3106 (79'105,173'3106] local-les=149 n=320 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3105 mlcod 173'3105 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277476 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.170 at 2014-06-11 13:02:32.506577
2014-06-12 14:51:45.277488 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277489 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.170( v 173'3050 (79'49,173'3050] local-les=139 n=301 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3049 mlcod 173'3049 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277497 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1b5 at 2014-06-11 13:02:32.507250
2014-06-12 14:51:45.277500 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277501 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1b5( v 173'2905 (0'0,173'2905] local-les=153 n=320 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2904 mlcod 173'2904 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277511 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1bc at 2014-06-11 13:02:32.507758
2014-06-12 14:51:45.277514 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277509 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] map version is 173'2566
2014-06-12 14:51:45.277515 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1bc( v 173'2460 (0'0,173'2460] local-les=139 n=273 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2459 mlcod 173'2459 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277520 7f9d623d5700 10 osd.2 173 dequeue_op 0x12bf6400 finish
2014-06-12 14:51:45.277521 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1e1 at 2014-06-11 13:02:32.508326
2014-06-12 14:51:45.277524 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277526 7f9d623d5700  5 -- op tracker -- , seq: 748598, time: 2014-06-12 14:51:45.277523, event: done, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.277525 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1e1( v 173'2541 (0'0,173'2541] local-les=153 n=283 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2540 mlcod 173'2540 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277532 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1e8 at 2014-06-11 13:02:32.508866
2014-06-12 14:51:45.277535 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277539 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:45.277536 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1e8( v 173'3034 (79'33,173'3034] local-les=156 n=323 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3033 mlcod 173'3033 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277542 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:45.277542 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1eb at 2014-06-11 13:02:32.509421
2014-06-12 14:51:45.277545 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277547 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:45.277546 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1eb( v 173'3096 (79'95,173'3096] local-les=153 n=331 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'3095 mlcod 173'3095 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277553 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.1fd at 2014-06-11 13:02:32.510047
2014-06-12 14:51:45.277553 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:45.277571 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277575 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.1fd( v 173'3047 (79'46,173'3047] local-les=157 n=305 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277574 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:45.277589 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:45.277590 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.204 at 2014-06-11 13:02:32.510573
2014-06-12 14:51:45.277605 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277608 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.204( v 173'2982 (0'0,173'2982] local-les=161 n=308 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277641 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.209 at 2014-06-11 13:02:32.511095
2014-06-12 14:51:45.277648 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277650 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.209( v 173'2531 (0'0,173'2531] local-les=156 n=264 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2530 mlcod 173'2530 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277663 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.214 at 2014-06-11 13:02:32.511670
2014-06-12 14:51:45.277669 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277657 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:45.277671 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.214( v 173'2935 (0'0,173'2935] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2934 mlcod 173'2934 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277676 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:45.277684 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.21d at 2014-06-11 13:02:32.512202
2014-06-12 14:51:45.277690 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277689 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub replica 24 has 25 items
2014-06-12 14:51:45.277692 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.21d( v 173'3024 (79'23,173'3024] local-les=156 n=318 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'3023 mlcod 173'3023 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277706 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.23f at 2014-06-11 13:02:32.512776
2014-06-12 14:51:45.277722 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277725 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.23f( v 173'2730 (0'0,173'2730] local-les=149 n=293 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2729 mlcod 173'2729 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277729 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f2039043/rbd_data.10996b8b4567.000000000000391e/head//3, auth == maps.end()
2014-06-12 14:51:45.277740 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.251 at 2014-06-11 13:02:32.513340
2014-06-12 14:51:45.277746 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277748 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.251( v 173'2452 (0'0,173'2452] local-les=153 n=272 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2451 mlcod 173'2451 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277753 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f2039043/rbd_data.10996b8b4567.000000000000391e/head//3
2014-06-12 14:51:45.277762 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.262 at 2014-06-11 13:02:32.513946
2014-06-12 14:51:45.277768 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277770 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.262( v 173'2675 (0'0,173'2675] local-les=142 n=292 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2674 mlcod 173'2674 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277776 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f2a39043/rbd_data.109d6b8b4567.0000000000000a4e/head//3, auth == maps.end()
2014-06-12 14:51:45.277788 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.266 at 2014-06-11 13:02:32.514523
2014-06-12 14:51:45.277805 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277808 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.266( v 173'2825 (0'0,173'2825] local-les=156 n=281 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2824 mlcod 173'2824 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277822 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.288 at 2014-06-11 13:02:32.515022
2014-06-12 14:51:45.277794 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f2a39043/rbd_data.109d6b8b4567.0000000000000a4e/head//3
2014-06-12 14:51:45.277828 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277830 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.288( v 173'2548 (0'0,173'2548] local-les=161 n=263 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2547 mlcod 173'2547 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277834 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 29549043/rbd_data.10aa6b8b4567.0000000000001230/head//3, auth == maps.end()
2014-06-12 14:51:45.277843 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2b4 at 2014-06-11 13:02:32.515543
2014-06-12 14:51:45.277849 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277851 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2b4( v 173'3109 (79'108,173'3109] local-les=137 n=312 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3108 mlcod 173'3108 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277852 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 29549043/rbd_data.10aa6b8b4567.0000000000001230/head//3
2014-06-12 14:51:45.277865 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2c4 at 2014-06-11 13:02:32.516106
2014-06-12 14:51:45.277877 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277871 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 30569043/rbd_data.10b46b8b4567.0000000000003921/head//3, auth == maps.end()
2014-06-12 14:51:45.277880 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2c4( v 173'2897 (0'0,173'2897] local-les=139 n=307 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2896 mlcod 173'2896 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277893 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2d3 at 2014-06-11 13:02:32.516627
2014-06-12 14:51:45.277887 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 30569043/rbd_data.10b46b8b4567.0000000000003921/head//3
2014-06-12 14:51:45.277899 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277901 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2d3( v 173'2556 (0'0,173'2556] local-les=153 n=276 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2555 mlcod 173'2555 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277906 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b089043/rbd_data.10b26b8b4567.0000000000000466/head//3, auth == maps.end()
2014-06-12 14:51:45.277915 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2da at 2014-06-11 13:02:32.517119
2014-06-12 14:51:45.277920 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277923 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2da( v 173'2988 (0'0,173'2988] local-les=145 n=302 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2987 mlcod 173'2987 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277922 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj b089043/rbd_data.10b26b8b4567.0000000000000466/head//3
2014-06-12 14:51:45.277937 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.2f4 at 2014-06-11 13:02:32.517764
2014-06-12 14:51:45.277943 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277942 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9da89043/rbd_data.10a46b8b4567.000000000000064f/head//3, auth == maps.end()
2014-06-12 14:51:45.277945 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.2f4( v 173'3275 (125'274,173'3275] local-les=145 n=320 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'3274 mlcod 173'3274 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277958 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.312 at 2014-06-11 13:02:32.518300
2014-06-12 14:51:45.277963 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277959 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9da89043/rbd_data.10a46b8b4567.000000000000064f/head//3
2014-06-12 14:51:45.277966 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.312( v 173'2783 (0'0,173'2783] local-les=156 n=296 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2782 mlcod 173'2782 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277988 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.320 at 2014-06-11 13:02:32.518824
2014-06-12 14:51:45.277978 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e1199043/rbd_data.10ae6b8b4567.00000000000020b0/head//3, auth == maps.end()
2014-06-12 14:51:45.277994 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.277995 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.320( v 173'2687 (0'0,173'2687] local-les=145 n=282 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2686 mlcod 173'2686 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.277995 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj e1199043/rbd_data.10ae6b8b4567.00000000000020b0/head//3
2014-06-12 14:51:45.278002 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.330 at 2014-06-11 13:02:32.519349
2014-06-12 14:51:45.278005 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278005 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 44599043/rbd_data.109b6b8b4567.00000000000023c8/head//3, auth == maps.end()
2014-06-12 14:51:45.278010 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.330( v 173'2799 (0'0,173'2799] local-les=161 n=301 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2798 mlcod 173'2798 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278013 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 44599043/rbd_data.109b6b8b4567.00000000000023c8/head//3
2014-06-12 14:51:45.278017 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.342 at 2014-06-11 13:02:32.519873
2014-06-12 14:51:45.278020 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278021 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.342( v 173'2465 (0'0,173'2465] local-les=153 n=281 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2464 mlcod 173'2464 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278028 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.35b at 2014-06-11 13:02:32.520528
2014-06-12 14:51:45.278031 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278022 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 81da9043/rbd_data.10af6b8b4567.0000000000003ae6/head//3, auth == maps.end()
2014-06-12 14:51:45.278032 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.35b( v 173'2777 (0'0,173'2777] local-les=161 n=294 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2776 mlcod 173'2776 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278039 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.366 at 2014-06-11 13:02:32.521097
2014-06-12 14:51:45.278037 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 81da9043/rbd_data.10af6b8b4567.0000000000003ae6/head//3
2014-06-12 14:51:45.278044 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278045 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.366( v 173'2662 (0'0,173'2662] local-les=153 n=286 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2661 mlcod 173'2661 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278047 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c1fa9043/rbd_data.10ac6b8b4567.0000000000003695/head//3, auth == maps.end()
2014-06-12 14:51:45.278053 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.37a at 2014-06-11 13:02:32.521744
2014-06-12 14:51:45.278056 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278056 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c1fa9043/rbd_data.10ac6b8b4567.0000000000003695/head//3
2014-06-12 14:51:45.278057 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.37a( v 173'2555 (0'0,173'2555] local-les=153 n=292 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2554 mlcod 173'2554 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278064 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.394 at 2014-06-11 13:02:32.522362
2014-06-12 14:51:45.278067 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278065 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9cac9043/rbd_data.10956b8b4567.0000000000002d9f/head//3, auth == maps.end()
2014-06-12 14:51:45.278068 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.394( v 173'2648 (0'0,173'2648] local-les=145 n=266 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2647 mlcod 173'2647 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278075 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.395 at 2014-06-11 13:02:32.522991
2014-06-12 14:51:45.278074 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9cac9043/rbd_data.10956b8b4567.0000000000002d9f/head//3
2014-06-12 14:51:45.278079 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278081 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.395( v 173'3328 (125'327,173'3328] local-les=137 n=324 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3327 mlcod 173'3327 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278082 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cb7d9043/rbd_data.10b06b8b4567.00000000000008a4/head//3, auth == maps.end()
2014-06-12 14:51:45.278088 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3ac at 2014-06-11 13:02:32.523569
2014-06-12 14:51:45.278091 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj cb7d9043/rbd_data.10b06b8b4567.00000000000008a4/head//3
2014-06-12 14:51:45.278096 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278098 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3ac( v 173'2860 (0'0,173'2860] local-les=142 n=294 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2859 mlcod 173'2859 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278099 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj da0a043/rbd_data.10946b8b4567.0000000000003178/head//3, auth == maps.end()
2014-06-12 14:51:45.278105 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3d4 at 2014-06-11 13:02:32.524119
2014-06-12 14:51:45.278108 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278108 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj da0a043/rbd_data.10946b8b4567.0000000000003178/head//3
2014-06-12 14:51:45.278110 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3d4( v 173'2696 (0'0,173'2696] local-les=137 n=271 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2695 mlcod 173'2695 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278116 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.3d8 at 2014-06-11 13:02:32.524641
2014-06-12 14:51:45.278119 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278117 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1e21a043/rbd_data.109b6b8b4567.0000000000001b40/head//3, auth == maps.end()
2014-06-12 14:51:45.278120 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.3d8( v 173'2879 (0'0,173'2879] local-les=156 n=294 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2878 mlcod 173'2878 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278125 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 1e21a043/rbd_data.109b6b8b4567.0000000000001b40/head//3
2014-06-12 14:51:45.278127 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.401 at 2014-06-11 13:02:32.525188
2014-06-12 14:51:45.278132 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278134 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4dc3a043/rbd_data.10aa6b8b4567.000000000000243c/head//3, auth == maps.end()
2014-06-12 14:51:45.278133 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.401( v 173'3155 (79'154,173'3155] local-les=139 n=322 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'3154 mlcod 173'3154 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278159 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.408 at 2014-06-11 13:02:32.525838
2014-06-12 14:51:45.278166 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278159 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 4dc3a043/rbd_data.10aa6b8b4567.000000000000243c/head//3
2014-06-12 14:51:45.278168 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.408( v 173'2884 (0'0,173'2884] local-les=153 n=318 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2883 mlcod 173'2883 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278182 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.411 at 2014-06-11 13:02:32.526419
2014-06-12 14:51:45.278179 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d374a043/rbd_data.10976b8b4567.0000000000003917/head//3, auth == maps.end()
2014-06-12 14:51:45.278190 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278192 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.411( v 173'2402 (0'0,173'2402] local-les=147 n=278 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2401 mlcod 173'2401 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278195 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d374a043/rbd_data.10976b8b4567.0000000000003917/head//3
2014-06-12 14:51:45.278205 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.45f at 2014-06-11 13:02:32.526940
2014-06-12 14:51:45.278211 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278213 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.45f( v 173'2358 (0'0,173'2358] local-les=145 n=260 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2357 mlcod 173'2357 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278213 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bb74a043/rbd_data.10a66b8b4567.0000000000000d90/head//3, auth == maps.end()
2014-06-12 14:51:45.278227 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.479 at 2014-06-11 13:02:32.527498
2014-06-12 14:51:45.278233 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278231 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj bb74a043/rbd_data.10a66b8b4567.0000000000000d90/head//3
2014-06-12 14:51:45.278235 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.479( v 173'2926 (0'0,173'2926] local-les=142 n=298 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2925 mlcod 173'2925 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278248 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4a7 at 2014-06-11 13:02:32.528023
2014-06-12 14:51:45.278254 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278250 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a495a043/rbd_data.108e6b8b4567.0000000000000538/head//3, auth == maps.end()
2014-06-12 14:51:45.278256 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4a7( v 173'2595 (0'0,173'2595] local-les=147 n=287 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2594 mlcod 173'2594 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278268 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4b4 at 2014-06-11 13:02:32.528548
2014-06-12 14:51:45.278274 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278266 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj a495a043/rbd_data.108e6b8b4567.0000000000000538/head//3
2014-06-12 14:51:45.278277 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4b4( v 173'2792 (0'0,173'2792] local-les=137 n=305 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278290 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4d4 at 2014-06-11 13:02:32.529466
2014-06-12 14:51:45.278296 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278286 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 55a6a043/rbd_data.10a96b8b4567.0000000000001210/head//3, auth == maps.end()
2014-06-12 14:51:45.278298 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4d4( v 173'2746 (0'0,173'2746] local-les=139 n=294 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2745 mlcod 173'2745 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278303 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 55a6a043/rbd_data.10a96b8b4567.0000000000001210/head//3
2014-06-12 14:51:45.278310 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4ef at 2014-06-11 13:02:32.530034
2014-06-12 14:51:45.278316 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278319 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4ef( v 173'3004 (79'3,173'3004] local-les=149 n=307 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3003 mlcod 173'3003 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278321 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 15e7a043/rbd_data.10b26b8b4567.00000000000006f4/head//3, auth == maps.end()
2014-06-12 14:51:45.278331 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.4f0 at 2014-06-11 13:02:32.530572
2014-06-12 14:51:45.278337 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278338 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 15e7a043/rbd_data.10b26b8b4567.00000000000006f4/head//3
2014-06-12 14:51:45.278339 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.4f0( v 173'2865 (0'0,173'2865] local-les=161 n=293 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2864 mlcod 173'2864 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278353 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.54b at 2014-06-11 13:02:32.531354
2014-06-12 14:51:45.278362 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278356 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ae02b043/rbd_data.108d6b8b4567.000000000000220e/head//3, auth == maps.end()
2014-06-12 14:51:45.278364 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.54b( v 173'2723 (0'0,173'2723] local-les=145 n=279 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2722 mlcod 173'2722 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278372 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj ae02b043/rbd_data.108d6b8b4567.000000000000220e/head//3
2014-06-12 14:51:45.278378 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.568 at 2014-06-11 13:02:32.531914
2014-06-12 14:51:45.278393 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278390 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c792b043/rbd_data.10a16b8b4567.00000000000005ea/head//3, auth == maps.end()
2014-06-12 14:51:45.278396 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.568( v 173'2727 (0'0,173'2727] local-les=161 n=270 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2726 mlcod 173'2726 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278406 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c792b043/rbd_data.10a16b8b4567.00000000000005ea/head//3
2014-06-12 14:51:45.278414 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.58f at 2014-06-11 13:02:32.532612
2014-06-12 14:51:45.278422 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278425 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.58f( v 173'2707 (0'0,173'2707] local-les=153 n=304 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278424 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a9b3b043/rbd_data.109a6b8b4567.00000000000011fb/head//3, auth == maps.end()
2014-06-12 14:51:45.278439 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5a3 at 2014-06-11 13:02:32.533139
2014-06-12 14:51:45.278445 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278443 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj a9b3b043/rbd_data.109a6b8b4567.00000000000011fb/head//3
2014-06-12 14:51:45.278447 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5a3( v 173'2874 (0'0,173'2874] local-les=142 n=296 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'2873 mlcod 173'2873 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278460 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5ce at 2014-06-11 13:02:32.533636
2014-06-12 14:51:45.278466 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278461 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 53b4b043/rbd_data.10b06b8b4567.000000000000202d/head//3, auth == maps.end()
2014-06-12 14:51:45.278468 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5ce( v 173'2741 (0'0,173'2741] local-les=153 n=294 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2740 mlcod 173'2740 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278481 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5dd at 2014-06-11 13:02:32.534211
2014-06-12 14:51:45.278487 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278478 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 53b4b043/rbd_data.10b06b8b4567.000000000000202d/head//3
2014-06-12 14:51:45.278489 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5dd( v 173'2852 (0'0,173'2852] local-les=156 n=307 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2851 mlcod 173'2851 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278496 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 71f6b043/rbd_data.109e6b8b4567.000000000000013e/head//3, auth == maps.end()
2014-06-12 14:51:45.278502 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.5fc at 2014-06-11 13:02:32.534738
2014-06-12 14:51:45.278511 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278512 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 71f6b043/rbd_data.109e6b8b4567.000000000000013e/head//3
2014-06-12 14:51:45.278514 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.5fc( v 173'2714 (0'0,173'2714] local-les=157 n=284 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2713 mlcod 173'2713 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278527 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.602 at 2014-06-11 13:02:32.535274
2014-06-12 14:51:45.278533 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278533 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] 
2014-06-12 14:51:45.278536 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.602( v 173'3010 (79'9,173'3010] local-les=161 n=316 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3009 mlcod 173'3009 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278548 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.605 at 2014-06-11 13:02:32.535829
2014-06-12 14:51:45.278554 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278546 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub
2014-06-12 14:51:45.278556 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.605( v 173'2849 (0'0,173'2849] local-les=145 n=304 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2848 mlcod 173'2848 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278569 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.613 at 2014-06-11 13:02:32.536448
2014-06-12 14:51:45.278575 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278570 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  71f6b043/rbd_data.109e6b8b4567.000000000000013e/head//3 71f6b043/rbd_data.109e6b8b4567.000000000000013e/head//3(173'2094 client.4295.0:118779 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2094)
2014-06-12 14:51:45.278577 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.613( v 173'2707 (0'0,173'2707] local-les=147 n=304 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2706 mlcod 173'2706 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278590 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.635 at 2014-06-11 13:02:32.537041
2014-06-12 14:51:45.278596 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278598 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.635( v 173'2966 (0'0,173'2966] local-les=137 n=308 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2965 mlcod 173'2965 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278595 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  53b4b043/rbd_data.10b06b8b4567.000000000000202d/head//3 53b4b043/rbd_data.10b06b8b4567.000000000000202d/head//3(173'1213 client.4313.0:68110 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1213)
2014-06-12 14:51:45.278611 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.63b at 2014-06-11 13:02:32.537647
2014-06-12 14:51:45.278617 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278619 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.63b( v 173'3153 (79'152,173'3153] local-les=137 n=310 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'3152 mlcod 173'3152 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278618 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  a9b3b043/rbd_data.109a6b8b4567.00000000000011fb/head//3 a9b3b043/rbd_data.109a6b8b4567.00000000000011fb/head//3(173'741 client.4291.0:41668 wrlock_by=unknown.0.0:0 dirty s 4194304 uv741)
2014-06-12 14:51:45.278633 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.63e at 2014-06-11 13:02:32.538178
2014-06-12 14:51:45.278638 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278640 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c792b043/rbd_data.10a16b8b4567.00000000000005ea/head//3 c792b043/rbd_data.10a16b8b4567.00000000000005ea/head//3(173'2194 client.4298.0:127991 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2194)
2014-06-12 14:51:45.278645 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.63e( v 173'2710 (0'0,173'2710] local-les=145 n=291 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2709 mlcod 173'2709 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278659 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.650 at 2014-06-11 13:02:32.538854
2014-06-12 14:51:45.278665 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278667 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.650( v 173'2864 (0'0,173'2864] local-les=145 n=286 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2863 mlcod 173'2863 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278663 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  ae02b043/rbd_data.108d6b8b4567.000000000000220e/head//3 ae02b043/rbd_data.108d6b8b4567.000000000000220e/head//3(173'1240 client.4278.0:71708 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1240)
2014-06-12 14:51:45.278680 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.662 at 2014-06-11 13:02:32.539391
2014-06-12 14:51:45.278685 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278688 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.662( v 173'2604 (0'0,173'2604] local-les=161 n=257 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2603 mlcod 173'2603 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278684 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  15e7a043/rbd_data.10b26b8b4567.00000000000006f4/head//3 15e7a043/rbd_data.10b26b8b4567.00000000000006f4/head//3(173'2245 client.4315.0:128547 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2245)
2014-06-12 14:51:45.278701 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.667 at 2014-06-11 13:02:32.539999
2014-06-12 14:51:45.278706 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278708 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.667( v 173'2958 (0'0,173'2958] local-les=149 n=303 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'2957 mlcod 173'2957 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278707 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  55a6a043/rbd_data.10a96b8b4567.0000000000001210/head//3 55a6a043/rbd_data.10a96b8b4567.0000000000001210/head//3(173'758 client.4306.0:43210 wrlock_by=unknown.0.0:0 dirty s 4194304 uv758)
2014-06-12 14:51:45.278721 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6a8 at 2014-06-11 13:02:32.540566
2014-06-12 14:51:45.278727 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278729 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6a8( v 173'2854 (0'0,173'2854] local-les=137 n=299 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2853 mlcod 173'2853 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278729 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  a495a043/rbd_data.108e6b8b4567.0000000000000538/head//3 a495a043/rbd_data.108e6b8b4567.0000000000000538/head//3(173'2182 client.4279.0:124867 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2182)
2014-06-12 14:51:45.278744 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6ff at 2014-06-11 13:02:32.541074
2014-06-12 14:51:45.278753 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278755 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6ff( v 173'2792 (0'0,173'2792] local-les=161 n=298 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2791 mlcod 173'2791 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278752 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  bb74a043/rbd_data.10a66b8b4567.0000000000000d90/head//3 bb74a043/rbd_data.10a66b8b4567.0000000000000d90/head//3(173'2460 client.4303.0:140641 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2460)
2014-06-12 14:51:45.278769 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.711 at 2014-06-11 13:02:32.541770
2014-06-12 14:51:45.278778 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278773 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d374a043/rbd_data.10976b8b4567.0000000000003917/head//3 d374a043/rbd_data.10976b8b4567.0000000000003917/head//3(173'2015 client.4288.0:110333 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2015)
2014-06-12 14:51:45.278780 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.711( v 173'2514 (0'0,173'2514] local-les=147 n=297 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2513 mlcod 173'2513 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278794 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.712 at 2014-06-11 13:02:32.542314
2014-06-12 14:51:45.278799 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278795 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  4dc3a043/rbd_data.10aa6b8b4567.000000000000243c/head//3 4dc3a043/rbd_data.10aa6b8b4567.000000000000243c/head//3(173'1285 client.4307.0:76689 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1285)
2014-06-12 14:51:45.278801 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.712( v 173'2782 (0'0,173'2782] local-les=153 n=305 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2781 mlcod 173'2781 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278814 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.717 at 2014-06-11 13:02:32.542947
2014-06-12 14:51:45.278820 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278816 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  1e21a043/rbd_data.109b6b8b4567.0000000000001b40/head//3 1e21a043/rbd_data.109b6b8b4567.0000000000001b40/head//3(173'1161 client.4292.0:57083 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1161)
2014-06-12 14:51:45.278822 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.717( v 173'2584 (0'0,173'2584] local-les=156 n=271 ec=23 les/c 156/156 155/155/133) [2,27] r=0 lpr=155 crt=0'0 lcod 173'2583 mlcod 173'2583 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278835 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.721 at 2014-06-11 13:02:32.543618
2014-06-12 14:51:45.278841 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278837 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  da0a043/rbd_data.10946b8b4567.0000000000003178/head//3 da0a043/rbd_data.10946b8b4567.0000000000003178/head//3(173'1667 client.4285.0:97591 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1667)
2014-06-12 14:51:45.278843 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.721( v 173'2896 (0'0,173'2896] local-les=157 n=311 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2895 mlcod 173'2895 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278856 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.735 at 2014-06-11 13:02:32.544147
2014-06-12 14:51:45.278861 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278859 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  cb7d9043/rbd_data.10b06b8b4567.00000000000008a4/head//3 cb7d9043/rbd_data.10b06b8b4567.00000000000008a4/head//3(173'2318 client.4313.0:134271 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2318)
2014-06-12 14:51:45.278863 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.735( v 173'2803 (0'0,173'2803] local-les=137 n=306 ec=23 les/c 137/137 136/136/133) [2,20] r=0 lpr=136 crt=0'0 lcod 173'2802 mlcod 173'2802 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278880 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.762 at 2014-06-11 13:02:32.544735
2014-06-12 14:51:45.278887 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278880 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9cac9043/rbd_data.10956b8b4567.0000000000002d9f/head//3 9cac9043/rbd_data.10956b8b4567.0000000000002d9f/head//3(173'1590 client.4286.0:91979 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1590)
2014-06-12 14:51:45.278889 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.762( v 173'2726 (0'0,173'2726] local-les=147 n=302 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2725 mlcod 173'2725 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278901 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.76e at 2014-06-11 13:02:32.545259
2014-06-12 14:51:45.278915 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278902 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c1fa9043/rbd_data.10ac6b8b4567.0000000000003695/head//3 c1fa9043/rbd_data.10ac6b8b4567.0000000000003695/head//3(173'1946 client.4309.0:107393 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1946)
2014-06-12 14:51:45.278917 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.76e( v 173'2649 (0'0,173'2649] local-les=145 n=284 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2648 mlcod 173'2648 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278932 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.77e at 2014-06-11 13:02:32.545813
2014-06-12 14:51:45.278937 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278926 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  81da9043/rbd_data.10af6b8b4567.0000000000003ae6/head//3 81da9043/rbd_data.10af6b8b4567.0000000000003ae6/head//3(173'2035 client.4312.0:118898 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2035)
2014-06-12 14:51:45.278940 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.77e( v 173'3052 (79'51,173'3052] local-les=161 n=319 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'3051 mlcod 173'3051 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278952 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.783 at 2014-06-11 13:02:32.546436
2014-06-12 14:51:45.278958 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278947 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  44599043/rbd_data.109b6b8b4567.00000000000023c8/head//3 44599043/rbd_data.109b6b8b4567.00000000000023c8/head//3(173'1303 client.4292.0:71598 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1303)
2014-06-12 14:51:45.278960 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.783( v 173'3047 (79'46,173'3047] local-les=142 n=318 ec=23 les/c 142/142 141/141/133) [2,22] r=0 lpr=141 crt=0'0 lcod 173'3046 mlcod 173'3046 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278973 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.785 at 2014-06-11 13:02:32.547026
2014-06-12 14:51:45.278979 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278968 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  e1199043/rbd_data.10ae6b8b4567.00000000000020b0/head//3 e1199043/rbd_data.10ae6b8b4567.00000000000020b0/head//3(173'1164 client.4311.0:66902 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1164)
2014-06-12 14:51:45.278981 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.785( v 173'2982 (0'0,173'2982] local-les=147 n=334 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2981 mlcod 173'2981 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.278993 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.793 at 2014-06-11 13:02:32.547568
2014-06-12 14:51:45.278999 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.278989 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9da89043/rbd_data.10a46b8b4567.000000000000064f/head//3 9da89043/rbd_data.10a46b8b4567.000000000000064f/head//3(173'2200 client.4301.0:128121 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2200)
2014-06-12 14:51:45.279001 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.793( v 173'2605 (0'0,173'2605] local-les=153 n=277 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2604 mlcod 173'2604 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.279014 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ab at 2014-06-11 13:02:32.548162
2014-06-12 14:51:45.279019 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.279011 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  b089043/rbd_data.10b26b8b4567.0000000000000466/head//3 b089043/rbd_data.10b26b8b4567.0000000000000466/head//3(173'2153 client.4315.0:124153 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2153)
2014-06-12 14:51:45.279021 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ab( v 173'2921 (0'0,173'2921] local-les=161 n=302 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2920 mlcod 173'2920 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.279033 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ad at 2014-06-11 13:02:32.548789
2014-06-12 14:51:45.279039 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.279032 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  30569043/rbd_data.10b46b8b4567.0000000000003921/head//3 30569043/rbd_data.10b46b8b4567.0000000000003921/head//3(173'2057 client.4317.0:112850 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2057)
2014-06-12 14:51:45.279041 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ad( v 173'3043 (79'42,173'3043] local-les=149 n=315 ec=23 les/c 149/149 148/148/133) [2,25] r=0 lpr=148 crt=0'0 lcod 173'3042 mlcod 173'3042 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.279054 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7ae at 2014-06-11 13:02:32.549514
2014-06-12 14:51:45.279060 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.279053 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  29549043/rbd_data.10aa6b8b4567.0000000000001230/head//3 29549043/rbd_data.10aa6b8b4567.0000000000001230/head//3(173'804 client.4307.0:44774 wrlock_by=unknown.0.0:0 dirty s 4194304 uv804)
2014-06-12 14:51:45.279062 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7ae( v 173'2518 (0'0,173'2518] local-les=153 n=282 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2517 mlcod 173'2517 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.279075 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7bc at 2014-06-11 13:02:32.550208
2014-06-12 14:51:45.279081 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.279075 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f2a39043/rbd_data.109d6b8b4567.0000000000000a4e/head//3 f2a39043/rbd_data.109d6b8b4567.0000000000000a4e/head//3(173'2329 client.4294.0:133292 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2329)
2014-06-12 14:51:45.279083 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7bc( v 173'2562 (0'0,173'2562] local-les=153 n=287 ec=23 les/c 153/153 152/152/133) [2,26] r=0 lpr=152 crt=0'0 lcod 173'2561 mlcod 173'2561 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.279096 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7cc at 2014-06-11 13:02:32.550758
2014-06-12 14:51:45.279102 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.279097 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f2039043/rbd_data.10996b8b4567.000000000000391e/head//3 f2039043/rbd_data.10996b8b4567.000000000000391e/head//3(173'1967 client.4290.0:111471 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1967)
2014-06-12 14:51:45.279104 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7cc( v 173'2704 (0'0,173'2704] local-les=145 n=277 ec=23 les/c 145/145 144/144/133) [2,23] r=0 lpr=144 crt=0'0 lcod 173'2703 mlcod 173'2703 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.279113 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:45.279121 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7db at 2014-06-11 13:02:32.551269
2014-06-12 14:51:45.279127 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.279129 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7db( v 173'2682 (0'0,173'2682] local-les=139 n=280 ec=23 les/c 139/139 138/138/133) [2,21] r=0 lpr=138 crt=0'0 lcod 173'2681 mlcod 173'2681 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.279133 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:45.279147 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:45.279151 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:45.279144 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.7f8 at 2014-06-11 13:02:32.551909
2014-06-12 14:51:45.279154 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 + 1 active >= max 1
2014-06-12 14:51:45.279157 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.7f8( v 173'2659 (0'0,173'2659] local-les=161 n=278 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2658 mlcod 173'2658 active+clean] sched_scrub: failed to reserve locally
2014-06-12 14:51:45.279153 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:45.279170 7f9d84175700 10 osd.2 173 sched_scrub 0.3 at 2014-06-12 13:01:39.405748: 6605.49 < min (86400 seconds)
2014-06-12 14:51:45.279189 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:45.279192 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:45.279194 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:45.279235 7f9d60bd2700 10 collection_list_partial 3.43_head start:78b8b043//0//-1 is_max:0
2014-06-12 14:51:45.279625 7f9d60bd2700 20 objects: [aaeab043/rbd_data.10986b8b4567.0000000000002e32/head//3,f0cbb043/rbd_data.109b6b8b4567.0000000000002af6/head//3,6fcbb043/rbd_data.10976b8b4567.0000000000001390/head//3,848cb043/rbd_data.109c6b8b4567.0000000000000ffd/head//3,f9aeb043/rbd_data.10b16b8b4567.000000000000019e/head//3,a5e0c043/rbd_data.109e6b8b4567.000000000000148a/head//3,1852c043/rbd_data.10a16b8b4567.0000000000000fce/head//3,f715c043/rbd_data.10a06b8b4567.00000000000017b1/head//3,5006c043/rbd_data.108e6b8b4567.00000000000028e1/head//3,f438c043/rbd_data.10b16b8b4567.0000000000000ab9/head//3,9f0bc043/rbd_data.108f6b8b4567.000000000000315a/head//3,e75cc043/rbd_data.10906b8b4567.00000000000024ce/head//3,dbbcc043/rbd_data.10a86b8b4567.00000000000039f7/head//3,afa1d043/rbd_data.10ab6b8b4567.0000000000000ef6/head//3,6e92d043/rbd_data.10a26b8b4567.0000000000000704/head//3,47b4d043/rbd_data.10b26b8b4567.00000000000022c6/head//3,7095d043/rbd_data.10a26b8b4567.00000000000018c1/head//3,58ead043/rbd_data.108f6b8b4567.00000000000030e3/head//3,53fad043/rbd_data.10a46b8b4567.0000000000003b8f/head//3,cb0ed043/rbd_data.109f6b8b4567.0000000000000d62/head//3,4754e043/rbd_data.10a66b8b4567.0000000000003a37/head//3,ee4e043/rbd_data.10a86b8b4567.00000000000018d6/head//3,da65e043/rbd_data.10b26b8b4567.00000000000039ce/head//3,373ae043/rbd_data.10b16b8b4567.00000000000026a2/head//3,893ae043/rbd_data.10ac6b8b4567.00000000000006cf/head//3]
2014-06-12 14:51:45.280298 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  requesting scrubmap from osd.24
2014-06-12 14:51:45.280333 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:45.280365 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:45.280379 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:45.280390 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:45.280400 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:45.280411 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_map_chunk [78b8b043//0//-1,c16be043//0//-1)
2014-06-12 14:51:45.280424 7f9d60bd2700 10 collection_list_partial 3.43_head start:78b8b043//0//-1 is_max:0
2014-06-12 14:51:45.281273 7f9d60bd2700 20 objects: [aaeab043/rbd_data.10986b8b4567.0000000000002e32/head//3,f0cbb043/rbd_data.109b6b8b4567.0000000000002af6/head//3,6fcbb043/rbd_data.10976b8b4567.0000000000001390/head//3,848cb043/rbd_data.109c6b8b4567.0000000000000ffd/head//3,f9aeb043/rbd_data.10b16b8b4567.000000000000019e/head//3,a5e0c043/rbd_data.109e6b8b4567.000000000000148a/head//3,1852c043/rbd_data.10a16b8b4567.0000000000000fce/head//3,f715c043/rbd_data.10a06b8b4567.00000000000017b1/head//3,5006c043/rbd_data.108e6b8b4567.00000000000028e1/head//3,f438c043/rbd_data.10b16b8b4567.0000000000000ab9/head//3,9f0bc043/rbd_data.108f6b8b4567.000000000000315a/head//3,e75cc043/rbd_data.10906b8b4567.00000000000024ce/head//3,dbbcc043/rbd_data.10a86b8b4567.00000000000039f7/head//3,afa1d043/rbd_data.10ab6b8b4567.0000000000000ef6/head//3,6e92d043/rbd_data.10a26b8b4567.0000000000000704/head//3,47b4d043/rbd_data.10b26b8b4567.00000000000022c6/head//3,7095d043/rbd_data.10a26b8b4567.00000000000018c1/head//3,58ead043/rbd_data.108f6b8b4567.00000000000030e3/head//3,53fad043/rbd_data.10a46b8b4567.0000000000003b8f/head//3,cb0ed043/rbd_data.109f6b8b4567.0000000000000d62/head//3,4754e043/rbd_data.10a66b8b4567.0000000000003a37/head//3,ee4e043/rbd_data.10a86b8b4567.00000000000018d6/head//3,da65e043/rbd_data.10b26b8b4567.00000000000039ce/head//3,373ae043/rbd_data.10b16b8b4567.00000000000026a2/head//3,893ae043/rbd_data.10ac6b8b4567.00000000000006cf/head//3,c16be043/rbd_data.10986b8b4567.0000000000001ec7/head//3,2e9be043/rbd_data.10ae6b8b4567.00000000000030f0/head//3,665fe043/rbd_data.10976b8b4567.000000000000125b/head//3,aff1f043/rbd_data.108d6b8b4567.00000000000000ae/head//3,a502f043/rbd_data.10b26b8b4567.0000000000002b98/head//3,6f24f043/rbd_data.109f6b8b4567.0000000000003727/head//3,8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3,13c5f043/rbd_data.10a76b8b4567.00000000000035af/head//3,84d8f043/rbd_data.108d6b8b4567.0000000000003506/head//3,9cd9f043/rbd_data.10b16b8b4567.0000000000002513/head//3,10baf043/rbd_data.10a86b8b4567.0000000000002ccc/head//3,902bf043/rbd_data.10b36b8b4567.0000000000000757/head//3,18cdf043/rbd_data.10ac6b8b4567.0000000000003b1c/head//3,e61ef043/rbd_data.10af6b8b4567.0000000000001af0/head//3,560ff043/rbd_data.108d6b8b4567.00000000000038f7/head//3,34110843/rbd_data.109e6b8b4567.0000000000001bf9/head//3,b7320843/rbd_data.109e6b8b4567.000000000000204b/head//3,d5440843/rbd_data.10926b8b4567.000000000000386a/head//3,35670843/rbd_data.10ac6b8b4567.0000000000003722/head//3,bc670843/rbd_data.108f6b8b4567.0000000000002c0d/head//3,b2fd0843/rbd_data.10a86b8b4567.0000000000003407/head//3,677f0843/rbd_data.10a56b8b4567.0000000000001c12/head//3,d0af0843/rbd_data.10a86b8b4567.0000000000000d52/head//3,1fbf0843/rbd_data.109b6b8b4567.000000000000082f/head//3,30511843/rbd_data.10976b8b4567.0000000000001218/head//3,73631843/rbd_data.10906b8b4567.0000000000000b56/head//3,6d441843/rbd_data.10a96b8b4567.0000000000001f89/head//3,86641843/rbd_data.10b46b8b4567.000000000000041c/head//3,54e51843/rbd_data.109c6b8b4567.000000000000151d/head//3,f9361843/rbd_data.10986b8b4567.0000000000001648/head//3,c4681843/rbd_data.10916b8b4567.0000000000000e2d/head//3,93ab1843/rbd_data.10906b8b4567.0000000000000c42/head//3,752d1843/rbd_data.10946b8b4567.0000000000001b04/head//3,67ee1843/rbd_data.10a56b8b4567.0000000000000f58/head//3,c50f1843/rbd_data.10ae6b8b4567.0000000000001465/head//3,8c0f1843/rbd_data.10a06b8b4567.000000000000363d/head//3,ead02843/rbd_data.10aa6b8b4567.0000000000000f41/head//3,a9232843/rbd_data.10b36b8b4567.00000000000007db/head//3,63652843/rbd_data.10a06b8b4567.000000000000143c/head//3]
2014-06-12 14:51:45.281416 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:45.281440 7f9d60bd2700 10 stat 3.43_head/aaeab043/rbd_data.10986b8b4567.0000000000002e32/head//3
2014-06-12 14:51:45.281675 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid aaeab043/rbd_data.10986b8b4567.0000000000002e32/head//3
2014-06-12 14:51:45.284344 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:45.284352 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:45.284355 7f9d74bfa700 20 osd.2 173 _dispatch 0x127961c0 pg_stats_ack(1 pgs tid 4872) v1
2014-06-12 14:51:45.284359 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:51:45.284365 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:45.284367 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:45.292855 7f9d60bd2700 10 getattrs 3.43_head/aaeab043/rbd_data.10986b8b4567.0000000000002e32/head//3 = 0
2014-06-12 14:51:45.292878 7f9d60bd2700 10 stat 3.43_head/f0cbb043/rbd_data.109b6b8b4567.0000000000002af6/head//3
2014-06-12 14:51:45.293072 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f0cbb043/rbd_data.109b6b8b4567.0000000000002af6/head//3
2014-06-12 14:51:45.298732 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:45.302818 7f9d60bd2700 10 getattrs 3.43_head/f0cbb043/rbd_data.109b6b8b4567.0000000000002af6/head//3 = 0
2014-06-12 14:51:45.302835 7f9d60bd2700 10 stat 3.43_head/6fcbb043/rbd_data.10976b8b4567.0000000000001390/head//3
2014-06-12 14:51:45.303013 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 6fcbb043/rbd_data.10976b8b4567.0000000000001390/head//3
2014-06-12 14:51:45.316839 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:45.318453 7f9d60bd2700 10 getattrs 3.43_head/6fcbb043/rbd_data.10976b8b4567.0000000000001390/head//3 = 0
2014-06-12 14:51:45.318460 7f9d60bd2700 10 stat 3.43_head/848cb043/rbd_data.109c6b8b4567.0000000000000ffd/head//3
2014-06-12 14:51:45.318556 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 848cb043/rbd_data.109c6b8b4567.0000000000000ffd/head//3
2014-06-12 14:51:45.319177 7f9d60bd2700 10 getattrs 3.43_head/848cb043/rbd_data.109c6b8b4567.0000000000000ffd/head//3 = 0
2014-06-12 14:51:45.319182 7f9d60bd2700 10 stat 3.43_head/f9aeb043/rbd_data.10b16b8b4567.000000000000019e/head//3
2014-06-12 14:51:45.319271 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f9aeb043/rbd_data.10b16b8b4567.000000000000019e/head//3
2014-06-12 14:51:45.341900 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:45.354635 7f9d60bd2700 10 getattrs 3.43_head/f9aeb043/rbd_data.10b16b8b4567.000000000000019e/head//3 = 0
2014-06-12 14:51:45.354644 7f9d60bd2700 10 stat 3.43_head/a5e0c043/rbd_data.109e6b8b4567.000000000000148a/head//3
2014-06-12 14:51:45.354747 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid a5e0c043/rbd_data.109e6b8b4567.000000000000148a/head//3
2014-06-12 14:51:45.375185 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:45.375707 7f9d60bd2700 10 getattrs 3.43_head/a5e0c043/rbd_data.109e6b8b4567.000000000000148a/head//3 = 0
2014-06-12 14:51:45.375716 7f9d60bd2700 10 stat 3.43_head/1852c043/rbd_data.10a16b8b4567.0000000000000fce/head//3
2014-06-12 14:51:45.375815 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 1852c043/rbd_data.10a16b8b4567.0000000000000fce/head//3
2014-06-12 14:51:45.375922 7f9d60bd2700 10 getattrs 3.43_head/1852c043/rbd_data.10a16b8b4567.0000000000000fce/head//3 = 0
2014-06-12 14:51:45.375935 7f9d60bd2700 10 stat 3.43_head/f715c043/rbd_data.10a06b8b4567.00000000000017b1/head//3
2014-06-12 14:51:45.376028 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f715c043/rbd_data.10a06b8b4567.00000000000017b1/head//3
2014-06-12 14:51:45.398649 7f9d60bd2700 10 getattrs 3.43_head/f715c043/rbd_data.10a06b8b4567.00000000000017b1/head//3 = 0
2014-06-12 14:51:45.398656 7f9d60bd2700 10 stat 3.43_head/5006c043/rbd_data.108e6b8b4567.00000000000028e1/head//3
2014-06-12 14:51:45.398750 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 5006c043/rbd_data.108e6b8b4567.00000000000028e1/head//3
2014-06-12 14:51:45.426246 7f9d60bd2700 10 getattrs 3.43_head/5006c043/rbd_data.108e6b8b4567.00000000000028e1/head//3 = 0
2014-06-12 14:51:45.426254 7f9d60bd2700 10 stat 3.43_head/f438c043/rbd_data.10b16b8b4567.0000000000000ab9/head//3
2014-06-12 14:51:45.426343 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f438c043/rbd_data.10b16b8b4567.0000000000000ab9/head//3
2014-06-12 14:51:45.470807 7f9d60bd2700 10 getattrs 3.43_head/f438c043/rbd_data.10b16b8b4567.0000000000000ab9/head//3 = 0
2014-06-12 14:51:45.470817 7f9d60bd2700 10 stat 3.43_head/9f0bc043/rbd_data.108f6b8b4567.000000000000315a/head//3
2014-06-12 14:51:45.470917 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9f0bc043/rbd_data.108f6b8b4567.000000000000315a/head//3
2014-06-12 14:51:45.485657 7f9d60bd2700 10 getattrs 3.43_head/9f0bc043/rbd_data.108f6b8b4567.000000000000315a/head//3 = 0
2014-06-12 14:51:45.485679 7f9d60bd2700 10 stat 3.43_head/e75cc043/rbd_data.10906b8b4567.00000000000024ce/head//3
2014-06-12 14:51:45.485801 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid e75cc043/rbd_data.10906b8b4567.00000000000024ce/head//3
2014-06-12 14:51:45.505910 7f9d60bd2700 10 getattrs 3.43_head/e75cc043/rbd_data.10906b8b4567.00000000000024ce/head//3 = 0
2014-06-12 14:51:45.505921 7f9d60bd2700 10 stat 3.43_head/dbbcc043/rbd_data.10a86b8b4567.00000000000039f7/head//3
2014-06-12 14:51:45.506040 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid dbbcc043/rbd_data.10a86b8b4567.00000000000039f7/head//3
2014-06-12 14:51:45.513696 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:45.518761 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:45.540938 7f9d60bd2700 10 getattrs 3.43_head/dbbcc043/rbd_data.10a86b8b4567.00000000000039f7/head//3 = 0
2014-06-12 14:51:45.540955 7f9d60bd2700 10 stat 3.43_head/afa1d043/rbd_data.10ab6b8b4567.0000000000000ef6/head//3
2014-06-12 14:51:45.541112 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid afa1d043/rbd_data.10ab6b8b4567.0000000000000ef6/head//3
2014-06-12 14:51:45.547637 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:45.561424 7f9d60bd2700 10 getattrs 3.43_head/afa1d043/rbd_data.10ab6b8b4567.0000000000000ef6/head//3 = 0
2014-06-12 14:51:45.561432 7f9d60bd2700 10 stat 3.43_head/6e92d043/rbd_data.10a26b8b4567.0000000000000704/head//3
2014-06-12 14:51:45.561532 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 6e92d043/rbd_data.10a26b8b4567.0000000000000704/head//3
2014-06-12 14:51:45.602499 7f9d60bd2700 10 getattrs 3.43_head/6e92d043/rbd_data.10a26b8b4567.0000000000000704/head//3 = 0
2014-06-12 14:51:45.602509 7f9d60bd2700 10 stat 3.43_head/47b4d043/rbd_data.10b26b8b4567.00000000000022c6/head//3
2014-06-12 14:51:45.602620 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 47b4d043/rbd_data.10b26b8b4567.00000000000022c6/head//3
2014-06-12 14:51:45.602708 7f9d60bd2700 10 getattrs 3.43_head/47b4d043/rbd_data.10b26b8b4567.00000000000022c6/head//3 = 0
2014-06-12 14:51:45.602714 7f9d60bd2700 10 stat 3.43_head/7095d043/rbd_data.10a26b8b4567.00000000000018c1/head//3
2014-06-12 14:51:45.602812 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 7095d043/rbd_data.10a26b8b4567.00000000000018c1/head//3
2014-06-12 14:51:45.615482 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:45.615646 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:45.631155 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:45.631252 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:45.634045 7f9d60bd2700 10 getattrs 3.43_head/7095d043/rbd_data.10a26b8b4567.00000000000018c1/head//3 = 0
2014-06-12 14:51:45.634056 7f9d60bd2700 10 stat 3.43_head/58ead043/rbd_data.108f6b8b4567.00000000000030e3/head//3
2014-06-12 14:51:45.634181 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 58ead043/rbd_data.108f6b8b4567.00000000000030e3/head//3
2014-06-12 14:51:45.646898 7f9d60bd2700 10 getattrs 3.43_head/58ead043/rbd_data.108f6b8b4567.00000000000030e3/head//3 = 0
2014-06-12 14:51:45.646912 7f9d60bd2700 10 stat 3.43_head/53fad043/rbd_data.10a46b8b4567.0000000000003b8f/head//3
2014-06-12 14:51:45.647045 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 53fad043/rbd_data.10a46b8b4567.0000000000003b8f/head//3
2014-06-12 14:51:45.662823 7f9d60bd2700 10 getattrs 3.43_head/53fad043/rbd_data.10a46b8b4567.0000000000003b8f/head//3 = 0
2014-06-12 14:51:45.662835 7f9d60bd2700 10 stat 3.43_head/cb0ed043/rbd_data.109f6b8b4567.0000000000000d62/head//3
2014-06-12 14:51:45.662947 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid cb0ed043/rbd_data.109f6b8b4567.0000000000000d62/head//3
2014-06-12 14:51:45.688080 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:45.688150 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:45.719067 7f9d60bd2700 10 getattrs 3.43_head/cb0ed043/rbd_data.109f6b8b4567.0000000000000d62/head//3 = 0
2014-06-12 14:51:45.719088 7f9d60bd2700 10 stat 3.43_head/4754e043/rbd_data.10a66b8b4567.0000000000003a37/head//3
2014-06-12 14:51:45.719211 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 4754e043/rbd_data.10a66b8b4567.0000000000003a37/head//3
2014-06-12 14:51:45.732869 7f9d60bd2700 10 getattrs 3.43_head/4754e043/rbd_data.10a66b8b4567.0000000000003a37/head//3 = 0
2014-06-12 14:51:45.732889 7f9d60bd2700 10 stat 3.43_head/ee4e043/rbd_data.10a86b8b4567.00000000000018d6/head//3
2014-06-12 14:51:45.733007 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid ee4e043/rbd_data.10a86b8b4567.00000000000018d6/head//3
2014-06-12 14:51:45.744938 7f9d60bd2700 10 getattrs 3.43_head/ee4e043/rbd_data.10a86b8b4567.00000000000018d6/head//3 = 0
2014-06-12 14:51:45.744954 7f9d60bd2700 10 stat 3.43_head/da65e043/rbd_data.10b26b8b4567.00000000000039ce/head//3
2014-06-12 14:51:45.745062 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid da65e043/rbd_data.10b26b8b4567.00000000000039ce/head//3
2014-06-12 14:51:45.776292 7f9d60bd2700 10 getattrs 3.43_head/da65e043/rbd_data.10b26b8b4567.00000000000039ce/head//3 = 0
2014-06-12 14:51:45.776302 7f9d60bd2700 10 stat 3.43_head/373ae043/rbd_data.10b16b8b4567.00000000000026a2/head//3
2014-06-12 14:51:45.776419 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 373ae043/rbd_data.10b16b8b4567.00000000000026a2/head//3
2014-06-12 14:51:45.809224 7f9d60bd2700 10 getattrs 3.43_head/373ae043/rbd_data.10b16b8b4567.00000000000026a2/head//3 = 0
2014-06-12 14:51:45.809238 7f9d60bd2700 10 stat 3.43_head/893ae043/rbd_data.10ac6b8b4567.00000000000006cf/head//3
2014-06-12 14:51:45.809392 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 893ae043/rbd_data.10ac6b8b4567.00000000000006cf/head//3
2014-06-12 14:51:45.853264 7f9d5a16f700  5 -- op tracker -- , seq: 748599, time: 2014-06-12 14:51:45.853141, event: header_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.853283 7f9d5a16f700  5 -- op tracker -- , seq: 748599, time: 2014-06-12 14:51:45.853145, event: throttled, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.853301 7f9d5a16f700  5 -- op tracker -- , seq: 748599, time: 2014-06-12 14:51:45.853211, event: all_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.853307 7f9d5a16f700  5 -- op tracker -- , seq: 748599, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.853311 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:45.853316 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:51:45.853334 7f9d5a16f700 15 osd.2 173 enqueue_op 0x12bf6800 prio 127 cost 10717 latency 0.000192 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:45.853376 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:45.860552 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:45.860855 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:45.876771 7f9d60bd2700 10 getattrs 3.43_head/893ae043/rbd_data.10ac6b8b4567.00000000000006cf/head//3 = 0
2014-06-12 14:51:45.876787 7f9d60bd2700 10 collection_getattrs 3.43_head
2014-06-12 14:51:45.876872 7f9d60bd2700 10 collection_getattrs 3.43_head = 0
2014-06-12 14:51:45.876878 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:45.876930 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:45.876941 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:45.876956 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:45.876960 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:45.876995 7f9d61bd4700 10 osd.2 173 dequeue_op 0x12bf6800 prio 127 cost 10717 latency 0.023854 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]
2014-06-12 14:51:45.877047 7f9d61bd4700  5 -- op tracker -- , seq: 748599, time: 2014-06-12 14:51:45.877040, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.877065 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] handle_message: 0x12bf6800
2014-06-12 14:51:45.877081 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:45.877098 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:45.877116 7f9d61bd4700  5 -- op tracker -- , seq: 748599, time: 2014-06-12 14:51:45.877111, event: started, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.877123 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  got 24 scrub map
2014-06-12 14:51:45.877230 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] map version is 173'2566
2014-06-12 14:51:45.877254 7f9d61bd4700 10 osd.2 173 dequeue_op 0x12bf6800 finish
2014-06-12 14:51:45.877266 7f9d61bd4700  5 -- op tracker -- , seq: 748599, time: 2014-06-12 14:51:45.877260, event: done, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:45.877287 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:45.877292 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:45.877292 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:45.877305 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:45.877334 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:45.877345 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:45.877382 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:45.877392 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:45.877403 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub replica 24 has 25 items
2014-06-12 14:51:45.877435 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aaeab043/rbd_data.10986b8b4567.0000000000002e32/head//3, auth == maps.end()
2014-06-12 14:51:45.877472 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj aaeab043/rbd_data.10986b8b4567.0000000000002e32/head//3
2014-06-12 14:51:45.877493 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f0cbb043/rbd_data.109b6b8b4567.0000000000002af6/head//3, auth == maps.end()
2014-06-12 14:51:45.877508 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f0cbb043/rbd_data.109b6b8b4567.0000000000002af6/head//3
2014-06-12 14:51:45.877523 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6fcbb043/rbd_data.10976b8b4567.0000000000001390/head//3, auth == maps.end()
2014-06-12 14:51:45.877537 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 6fcbb043/rbd_data.10976b8b4567.0000000000001390/head//3
2014-06-12 14:51:45.877552 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 848cb043/rbd_data.109c6b8b4567.0000000000000ffd/head//3, auth == maps.end()
2014-06-12 14:51:45.877565 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 848cb043/rbd_data.109c6b8b4567.0000000000000ffd/head//3
2014-06-12 14:51:45.877582 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f9aeb043/rbd_data.10b16b8b4567.000000000000019e/head//3, auth == maps.end()
2014-06-12 14:51:45.877600 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f9aeb043/rbd_data.10b16b8b4567.000000000000019e/head//3
2014-06-12 14:51:45.877633 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a5e0c043/rbd_data.109e6b8b4567.000000000000148a/head//3, auth == maps.end()
2014-06-12 14:51:45.877649 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj a5e0c043/rbd_data.109e6b8b4567.000000000000148a/head//3
2014-06-12 14:51:45.877667 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1852c043/rbd_data.10a16b8b4567.0000000000000fce/head//3, auth == maps.end()
2014-06-12 14:51:45.877681 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 1852c043/rbd_data.10a16b8b4567.0000000000000fce/head//3
2014-06-12 14:51:45.877698 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f715c043/rbd_data.10a06b8b4567.00000000000017b1/head//3, auth == maps.end()
2014-06-12 14:51:45.877714 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f715c043/rbd_data.10a06b8b4567.00000000000017b1/head//3
2014-06-12 14:51:45.877732 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5006c043/rbd_data.108e6b8b4567.00000000000028e1/head//3, auth == maps.end()
2014-06-12 14:51:45.877746 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 5006c043/rbd_data.108e6b8b4567.00000000000028e1/head//3
2014-06-12 14:51:45.877763 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f438c043/rbd_data.10b16b8b4567.0000000000000ab9/head//3, auth == maps.end()
2014-06-12 14:51:45.877777 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f438c043/rbd_data.10b16b8b4567.0000000000000ab9/head//3
2014-06-12 14:51:45.877794 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9f0bc043/rbd_data.108f6b8b4567.000000000000315a/head//3, auth == maps.end()
2014-06-12 14:51:45.877809 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9f0bc043/rbd_data.108f6b8b4567.000000000000315a/head//3
2014-06-12 14:51:45.877826 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e75cc043/rbd_data.10906b8b4567.00000000000024ce/head//3, auth == maps.end()
2014-06-12 14:51:45.877840 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj e75cc043/rbd_data.10906b8b4567.00000000000024ce/head//3
2014-06-12 14:51:45.877857 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dbbcc043/rbd_data.10a86b8b4567.00000000000039f7/head//3, auth == maps.end()
2014-06-12 14:51:45.877871 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj dbbcc043/rbd_data.10a86b8b4567.00000000000039f7/head//3
2014-06-12 14:51:45.877886 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj afa1d043/rbd_data.10ab6b8b4567.0000000000000ef6/head//3, auth == maps.end()
2014-06-12 14:51:45.877902 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj afa1d043/rbd_data.10ab6b8b4567.0000000000000ef6/head//3
2014-06-12 14:51:45.877920 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6e92d043/rbd_data.10a26b8b4567.0000000000000704/head//3, auth == maps.end()
2014-06-12 14:51:45.877936 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 6e92d043/rbd_data.10a26b8b4567.0000000000000704/head//3
2014-06-12 14:51:45.877951 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 47b4d043/rbd_data.10b26b8b4567.00000000000022c6/head//3, auth == maps.end()
2014-06-12 14:51:45.877967 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 47b4d043/rbd_data.10b26b8b4567.00000000000022c6/head//3
2014-06-12 14:51:45.877982 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7095d043/rbd_data.10a26b8b4567.00000000000018c1/head//3, auth == maps.end()
2014-06-12 14:51:45.877999 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 7095d043/rbd_data.10a26b8b4567.00000000000018c1/head//3
2014-06-12 14:51:45.878017 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 58ead043/rbd_data.108f6b8b4567.00000000000030e3/head//3, auth == maps.end()
2014-06-12 14:51:45.878033 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 58ead043/rbd_data.108f6b8b4567.00000000000030e3/head//3
2014-06-12 14:51:45.878051 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 53fad043/rbd_data.10a46b8b4567.0000000000003b8f/head//3, auth == maps.end()
2014-06-12 14:51:45.878065 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 53fad043/rbd_data.10a46b8b4567.0000000000003b8f/head//3
2014-06-12 14:51:45.878082 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cb0ed043/rbd_data.109f6b8b4567.0000000000000d62/head//3, auth == maps.end()
2014-06-12 14:51:45.878098 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj cb0ed043/rbd_data.109f6b8b4567.0000000000000d62/head//3
2014-06-12 14:51:45.878115 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4754e043/rbd_data.10a66b8b4567.0000000000003a37/head//3, auth == maps.end()
2014-06-12 14:51:45.878132 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 4754e043/rbd_data.10a66b8b4567.0000000000003a37/head//3
2014-06-12 14:51:45.878149 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ee4e043/rbd_data.10a86b8b4567.00000000000018d6/head//3, auth == maps.end()
2014-06-12 14:51:45.878166 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj ee4e043/rbd_data.10a86b8b4567.00000000000018d6/head//3
2014-06-12 14:51:45.878183 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj da65e043/rbd_data.10b26b8b4567.00000000000039ce/head//3, auth == maps.end()
2014-06-12 14:51:45.878199 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj da65e043/rbd_data.10b26b8b4567.00000000000039ce/head//3
2014-06-12 14:51:45.878215 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 373ae043/rbd_data.10b16b8b4567.00000000000026a2/head//3, auth == maps.end()
2014-06-12 14:51:45.878231 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 373ae043/rbd_data.10b16b8b4567.00000000000026a2/head//3
2014-06-12 14:51:45.878249 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 893ae043/rbd_data.10ac6b8b4567.00000000000006cf/head//3, auth == maps.end()
2014-06-12 14:51:45.878263 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 893ae043/rbd_data.10ac6b8b4567.00000000000006cf/head//3
2014-06-12 14:51:45.878282 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] 
2014-06-12 14:51:45.878295 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub
2014-06-12 14:51:45.878315 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  893ae043/rbd_data.10ac6b8b4567.00000000000006cf/head//3 893ae043/rbd_data.10ac6b8b4567.00000000000006cf/head//3(173'2220 client.4309.0:126725 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2220)
2014-06-12 14:51:45.878337 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  373ae043/rbd_data.10b16b8b4567.00000000000026a2/head//3 373ae043/rbd_data.10b16b8b4567.00000000000026a2/head//3(173'1398 client.4314.0:79328 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1398)
2014-06-12 14:51:45.878359 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  da65e043/rbd_data.10b26b8b4567.00000000000039ce/head//3 da65e043/rbd_data.10b26b8b4567.00000000000039ce/head//3(173'1879 client.4315.0:114179 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1879)
2014-06-12 14:51:45.878379 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  ee4e043/rbd_data.10a86b8b4567.00000000000018d6/head//3 ee4e043/rbd_data.10a86b8b4567.00000000000018d6/head//3(173'926 client.4305.0:53386 wrlock_by=unknown.0.0:0 dirty s 4194304 uv926)
2014-06-12 14:51:45.878400 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  4754e043/rbd_data.10a66b8b4567.0000000000003a37/head//3 4754e043/rbd_data.10a66b8b4567.0000000000003a37/head//3(173'2051 client.4303.0:115315 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2051)
2014-06-12 14:51:45.878420 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  cb0ed043/rbd_data.109f6b8b4567.0000000000000d62/head//3 cb0ed043/rbd_data.109f6b8b4567.0000000000000d62/head//3(173'2419 client.4296.0:136934 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2419)
2014-06-12 14:51:45.878440 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  53fad043/rbd_data.10a46b8b4567.0000000000003b8f/head//3 53fad043/rbd_data.10a46b8b4567.0000000000003b8f/head//3(173'2021 client.4301.0:118340 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2021)
2014-06-12 14:51:45.878460 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  58ead043/rbd_data.108f6b8b4567.00000000000030e3/head//3 58ead043/rbd_data.108f6b8b4567.00000000000030e3/head//3(173'1708 client.4280.0:96102 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1708)
2014-06-12 14:51:45.878481 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  7095d043/rbd_data.10a26b8b4567.00000000000018c1/head//3 7095d043/rbd_data.10a26b8b4567.00000000000018c1/head//3(173'1030 client.4299.0:54710 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1030)
2014-06-12 14:51:45.878501 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  47b4d043/rbd_data.10b26b8b4567.00000000000022c6/head//3 47b4d043/rbd_data.10b26b8b4567.00000000000022c6/head//3(173'1217 client.4315.0:72110 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1217)
2014-06-12 14:51:45.878521 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  6e92d043/rbd_data.10a26b8b4567.0000000000000704/head//3 6e92d043/rbd_data.10a26b8b4567.0000000000000704/head//3(173'2239 client.4299.0:130960 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2239)
2014-06-12 14:51:45.878542 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  afa1d043/rbd_data.10ab6b8b4567.0000000000000ef6/head//3 afa1d043/rbd_data.10ab6b8b4567.0000000000000ef6/head//3(173'2534 client.4308.0:144062 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2534)
2014-06-12 14:51:45.878562 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  dbbcc043/rbd_data.10a86b8b4567.00000000000039f7/head//3 dbbcc043/rbd_data.10a86b8b4567.00000000000039f7/head//3(173'1871 client.4305.0:113905 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1871)
2014-06-12 14:51:45.878582 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  e75cc043/rbd_data.10906b8b4567.00000000000024ce/head//3 e75cc043/rbd_data.10906b8b4567.00000000000024ce/head//3(173'1279 client.4281.0:75111 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1279)
2014-06-12 14:51:45.878602 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9f0bc043/rbd_data.108f6b8b4567.000000000000315a/head//3 9f0bc043/rbd_data.108f6b8b4567.000000000000315a/head//3(173'1728 client.4280.0:97059 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1728)
2014-06-12 14:51:45.878623 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f438c043/rbd_data.10b16b8b4567.0000000000000ab9/head//3 f438c043/rbd_data.10b16b8b4567.0000000000000ab9/head//3(173'2350 client.4314.0:135418 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2350)
2014-06-12 14:51:45.878643 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  5006c043/rbd_data.108e6b8b4567.00000000000028e1/head//3 5006c043/rbd_data.108e6b8b4567.00000000000028e1/head//3(173'1421 client.4279.0:83405 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1421)
2014-06-12 14:51:45.878663 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f715c043/rbd_data.10a06b8b4567.00000000000017b1/head//3 f715c043/rbd_data.10a06b8b4567.00000000000017b1/head//3(173'882 client.4297.0:51583 wrlock_by=unknown.0.0:0 dirty s 4194304 uv882)
2014-06-12 14:51:45.878684 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  1852c043/rbd_data.10a16b8b4567.0000000000000fce/head//3 1852c043/rbd_data.10a16b8b4567.0000000000000fce/head//3(173'2539 client.4298.0:144800 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2539)
2014-06-12 14:51:45.878704 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  a5e0c043/rbd_data.109e6b8b4567.000000000000148a/head//3 a5e0c043/rbd_data.109e6b8b4567.000000000000148a/head//3(173'821 client.4295.0:47273 wrlock_by=unknown.0.0:0 dirty s 4194304 uv821)
2014-06-12 14:51:45.878724 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f9aeb043/rbd_data.10b16b8b4567.000000000000019e/head//3 f9aeb043/rbd_data.10b16b8b4567.000000000000019e/head//3(173'2100 client.4314.0:120269 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2100)
2014-06-12 14:51:45.878744 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  848cb043/rbd_data.109c6b8b4567.0000000000000ffd/head//3 848cb043/rbd_data.109c6b8b4567.0000000000000ffd/head//3(173'2554 client.4293.0:143650 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2554)
2014-06-12 14:51:45.878764 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  6fcbb043/rbd_data.10976b8b4567.0000000000001390/head//3 6fcbb043/rbd_data.10976b8b4567.0000000000001390/head//3(173'892 client.4288.0:43578 wrlock_by=unknown.0.0:0 dirty s 4194304 uv892)
2014-06-12 14:51:45.878784 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f0cbb043/rbd_data.109b6b8b4567.0000000000002af6/head//3 f0cbb043/rbd_data.109b6b8b4567.0000000000002af6/head//3(173'1533 client.4292.0:84847 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1533)
2014-06-12 14:51:45.878804 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  aaeab043/rbd_data.10986b8b4567.0000000000002e32/head//3 aaeab043/rbd_data.10986b8b4567.0000000000002e32/head//3(173'1562 client.4289.0:91229 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1562)
2014-06-12 14:51:45.878820 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:45.878837 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:45.878850 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:45.878854 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:45.878856 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:45.878910 7f9d60bd2700 10 collection_list_partial 3.43_head start:c16be043//0//-1 is_max:0
2014-06-12 14:51:45.879226 7f9d60bd2700 20 objects: [2e9be043/rbd_data.10ae6b8b4567.00000000000030f0/head//3,665fe043/rbd_data.10976b8b4567.000000000000125b/head//3,aff1f043/rbd_data.108d6b8b4567.00000000000000ae/head//3,a502f043/rbd_data.10b26b8b4567.0000000000002b98/head//3,6f24f043/rbd_data.109f6b8b4567.0000000000003727/head//3,8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3,13c5f043/rbd_data.10a76b8b4567.00000000000035af/head//3,84d8f043/rbd_data.108d6b8b4567.0000000000003506/head//3,9cd9f043/rbd_data.10b16b8b4567.0000000000002513/head//3,10baf043/rbd_data.10a86b8b4567.0000000000002ccc/head//3,902bf043/rbd_data.10b36b8b4567.0000000000000757/head//3,18cdf043/rbd_data.10ac6b8b4567.0000000000003b1c/head//3,e61ef043/rbd_data.10af6b8b4567.0000000000001af0/head//3,560ff043/rbd_data.108d6b8b4567.00000000000038f7/head//3,34110843/rbd_data.109e6b8b4567.0000000000001bf9/head//3,b7320843/rbd_data.109e6b8b4567.000000000000204b/head//3,d5440843/rbd_data.10926b8b4567.000000000000386a/head//3,35670843/rbd_data.10ac6b8b4567.0000000000003722/head//3,bc670843/rbd_data.108f6b8b4567.0000000000002c0d/head//3,b2fd0843/rbd_data.10a86b8b4567.0000000000003407/head//3,677f0843/rbd_data.10a56b8b4567.0000000000001c12/head//3,d0af0843/rbd_data.10a86b8b4567.0000000000000d52/head//3,1fbf0843/rbd_data.109b6b8b4567.000000000000082f/head//3,30511843/rbd_data.10976b8b4567.0000000000001218/head//3,73631843/rbd_data.10906b8b4567.0000000000000b56/head//3]
2014-06-12 14:51:45.879777 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  requesting scrubmap from osd.24
2014-06-12 14:51:45.879809 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:45.879832 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:45.879846 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:45.879855 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:45.879864 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:45.879873 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_map_chunk [c16be043//0//-1,6d441843//0//-1)
2014-06-12 14:51:45.879884 7f9d60bd2700 10 collection_list_partial 3.43_head start:c16be043//0//-1 is_max:0
2014-06-12 14:51:45.880606 7f9d60bd2700 20 objects: [2e9be043/rbd_data.10ae6b8b4567.00000000000030f0/head//3,665fe043/rbd_data.10976b8b4567.000000000000125b/head//3,aff1f043/rbd_data.108d6b8b4567.00000000000000ae/head//3,a502f043/rbd_data.10b26b8b4567.0000000000002b98/head//3,6f24f043/rbd_data.109f6b8b4567.0000000000003727/head//3,8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3,13c5f043/rbd_data.10a76b8b4567.00000000000035af/head//3,84d8f043/rbd_data.108d6b8b4567.0000000000003506/head//3,9cd9f043/rbd_data.10b16b8b4567.0000000000002513/head//3,10baf043/rbd_data.10a86b8b4567.0000000000002ccc/head//3,902bf043/rbd_data.10b36b8b4567.0000000000000757/head//3,18cdf043/rbd_data.10ac6b8b4567.0000000000003b1c/head//3,e61ef043/rbd_data.10af6b8b4567.0000000000001af0/head//3,560ff043/rbd_data.108d6b8b4567.00000000000038f7/head//3,34110843/rbd_data.109e6b8b4567.0000000000001bf9/head//3,b7320843/rbd_data.109e6b8b4567.000000000000204b/head//3,d5440843/rbd_data.10926b8b4567.000000000000386a/head//3,35670843/rbd_data.10ac6b8b4567.0000000000003722/head//3,bc670843/rbd_data.108f6b8b4567.0000000000002c0d/head//3,b2fd0843/rbd_data.10a86b8b4567.0000000000003407/head//3,677f0843/rbd_data.10a56b8b4567.0000000000001c12/head//3,d0af0843/rbd_data.10a86b8b4567.0000000000000d52/head//3,1fbf0843/rbd_data.109b6b8b4567.000000000000082f/head//3,30511843/rbd_data.10976b8b4567.0000000000001218/head//3,73631843/rbd_data.10906b8b4567.0000000000000b56/head//3,6d441843/rbd_data.10a96b8b4567.0000000000001f89/head//3,86641843/rbd_data.10b46b8b4567.000000000000041c/head//3,54e51843/rbd_data.109c6b8b4567.000000000000151d/head//3,f9361843/rbd_data.10986b8b4567.0000000000001648/head//3,c4681843/rbd_data.10916b8b4567.0000000000000e2d/head//3,93ab1843/rbd_data.10906b8b4567.0000000000000c42/head//3,752d1843/rbd_data.10946b8b4567.0000000000001b04/head//3,67ee1843/rbd_data.10a56b8b4567.0000000000000f58/head//3,c50f1843/rbd_data.10ae6b8b4567.0000000000001465/head//3,8c0f1843/rbd_data.10a06b8b4567.000000000000363d/head//3,ead02843/rbd_data.10aa6b8b4567.0000000000000f41/head//3,a9232843/rbd_data.10b36b8b4567.00000000000007db/head//3,63652843/rbd_data.10a06b8b4567.000000000000143c/head//3,eda82843/rbd_data.10a86b8b4567.00000000000014e3/head//3,16b92843/rbd_data.10b36b8b4567.0000000000002d47/head//3,a28a2843/rbd_data.10ac6b8b4567.0000000000001516/head//3,fcfa2843/rbd_data.10ab6b8b4567.0000000000000c79/head//3,16213843/rbd_data.10986b8b4567.0000000000001fd6/head//3,c2623843/rbd_data.10b16b8b4567.0000000000002605/head//3,c7d43843/rbd_data.10ac6b8b4567.00000000000039e1/head//3,b8b63843/rbd_data.10a26b8b4567.00000000000018b9/head//3,9fb73843/rbd_data.10a36b8b4567.0000000000002c93/head//3,f1c73843/rbd_data.109c6b8b4567.0000000000001b57/head//3,66d93843/rbd_data.109b6b8b4567.00000000000017dc/head//3,87f04843/rbd_data.109b6b8b4567.000000000000285f/head//3,86634843/rbd_data.10aa6b8b4567.0000000000001f5f/head//3,89f34843/rbd_data.10906b8b4567.000000000000020b/head//3,62464843/rbd_data.10906b8b4567.00000000000031ac/head//3,15d64843/rbd_data.10ae6b8b4567.000000000000288b/head//3,11205843/rbd_data.10af6b8b4567.00000000000019fb/head//3,26c35843/rbd_data.10b16b8b4567.0000000000003558/head//3,7045843/rbd_data.10a06b8b4567.000000000000162c/head//3,fbcb5843/rbd_data.10976b8b4567.0000000000002613/head//3,a1de5843/rbd_data.10a56b8b4567.00000000000030aa/head//3,bbdf5843/rbd_data.10af6b8b4567.0000000000002625/head//3,69406843/rbd_data.10ab6b8b4567.000000000000317a/head//3,af216843/rbd_data.10966b8b4567.0000000000001e15/head//3,6cf16843/rbd_data.10b36b8b4567.0000000000000e06/head//3,d7926843/rbd_data.10ae6b8b4567.0000000000000042/head//3]
2014-06-12 14:51:45.880726 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:45.880752 7f9d60bd2700 10 stat 3.43_head/2e9be043/rbd_data.10ae6b8b4567.00000000000030f0/head//3
2014-06-12 14:51:45.880934 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 2e9be043/rbd_data.10ae6b8b4567.00000000000030f0/head//3
2014-06-12 14:51:45.897074 7f9d60bd2700 10 getattrs 3.43_head/2e9be043/rbd_data.10ae6b8b4567.00000000000030f0/head//3 = 0
2014-06-12 14:51:45.897084 7f9d60bd2700 10 stat 3.43_head/665fe043/rbd_data.10976b8b4567.000000000000125b/head//3
2014-06-12 14:51:45.897177 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 665fe043/rbd_data.10976b8b4567.000000000000125b/head//3
2014-06-12 14:51:45.900129 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:45.900185 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:45.916751 7f9d60bd2700 10 getattrs 3.43_head/665fe043/rbd_data.10976b8b4567.000000000000125b/head//3 = 0
2014-06-12 14:51:45.916761 7f9d60bd2700 10 stat 3.43_head/aff1f043/rbd_data.108d6b8b4567.00000000000000ae/head//3
2014-06-12 14:51:45.916858 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid aff1f043/rbd_data.108d6b8b4567.00000000000000ae/head//3
2014-06-12 14:51:45.952552 7f9d60bd2700 10 getattrs 3.43_head/aff1f043/rbd_data.108d6b8b4567.00000000000000ae/head//3 = 0
2014-06-12 14:51:45.952562 7f9d60bd2700 10 stat 3.43_head/a502f043/rbd_data.10b26b8b4567.0000000000002b98/head//3
2014-06-12 14:51:45.952653 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid a502f043/rbd_data.10b26b8b4567.0000000000002b98/head//3
2014-06-12 14:51:45.961190 7f9d60bd2700 10 getattrs 3.43_head/a502f043/rbd_data.10b26b8b4567.0000000000002b98/head//3 = 0
2014-06-12 14:51:45.961198 7f9d60bd2700 10 stat 3.43_head/6f24f043/rbd_data.109f6b8b4567.0000000000003727/head//3
2014-06-12 14:51:45.961287 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 6f24f043/rbd_data.109f6b8b4567.0000000000003727/head//3
2014-06-12 14:51:45.977315 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:45.988011 7f9d60bd2700 10 getattrs 3.43_head/6f24f043/rbd_data.109f6b8b4567.0000000000003727/head//3 = 0
2014-06-12 14:51:45.988020 7f9d60bd2700 10 stat 3.43_head/8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3
2014-06-12 14:51:45.988123 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3
2014-06-12 14:51:45.988492 7f9d60bd2700 10 getattrs 3.43_head/8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 = 0
2014-06-12 14:51:45.988499 7f9d60bd2700 10 stat 3.43_head/13c5f043/rbd_data.10a76b8b4567.00000000000035af/head//3
2014-06-12 14:51:45.988593 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 13c5f043/rbd_data.10a76b8b4567.00000000000035af/head//3
2014-06-12 14:51:46.008835 7f9d60bd2700 10 getattrs 3.43_head/13c5f043/rbd_data.10a76b8b4567.00000000000035af/head//3 = 0
2014-06-12 14:51:46.008845 7f9d60bd2700 10 stat 3.43_head/84d8f043/rbd_data.108d6b8b4567.0000000000003506/head//3
2014-06-12 14:51:46.008952 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 84d8f043/rbd_data.108d6b8b4567.0000000000003506/head//3
2014-06-12 14:51:46.026134 7f9d60bd2700 10 getattrs 3.43_head/84d8f043/rbd_data.108d6b8b4567.0000000000003506/head//3 = 0
2014-06-12 14:51:46.026143 7f9d60bd2700 10 stat 3.43_head/9cd9f043/rbd_data.10b16b8b4567.0000000000002513/head//3
2014-06-12 14:51:46.026246 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9cd9f043/rbd_data.10b16b8b4567.0000000000002513/head//3
2014-06-12 14:51:46.055313 7f9d60bd2700 10 getattrs 3.43_head/9cd9f043/rbd_data.10b16b8b4567.0000000000002513/head//3 = 0
2014-06-12 14:51:46.055325 7f9d60bd2700 10 stat 3.43_head/10baf043/rbd_data.10a86b8b4567.0000000000002ccc/head//3
2014-06-12 14:51:46.055435 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 10baf043/rbd_data.10a86b8b4567.0000000000002ccc/head//3
2014-06-12 14:51:46.064941 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:46.071556 7f9d60bd2700 10 getattrs 3.43_head/10baf043/rbd_data.10a86b8b4567.0000000000002ccc/head//3 = 0
2014-06-12 14:51:46.071567 7f9d60bd2700 10 stat 3.43_head/902bf043/rbd_data.10b36b8b4567.0000000000000757/head//3
2014-06-12 14:51:46.071673 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 902bf043/rbd_data.10b36b8b4567.0000000000000757/head//3
2014-06-12 14:51:46.118625 7f9d60bd2700 10 getattrs 3.43_head/902bf043/rbd_data.10b36b8b4567.0000000000000757/head//3 = 0
2014-06-12 14:51:46.118642 7f9d60bd2700 10 stat 3.43_head/18cdf043/rbd_data.10ac6b8b4567.0000000000003b1c/head//3
2014-06-12 14:51:46.118809 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 18cdf043/rbd_data.10ac6b8b4567.0000000000003b1c/head//3
2014-06-12 14:51:46.141412 7f9d60bd2700 10 getattrs 3.43_head/18cdf043/rbd_data.10ac6b8b4567.0000000000003b1c/head//3 = 0
2014-06-12 14:51:46.141422 7f9d60bd2700 10 stat 3.43_head/e61ef043/rbd_data.10af6b8b4567.0000000000001af0/head//3
2014-06-12 14:51:46.141533 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid e61ef043/rbd_data.10af6b8b4567.0000000000001af0/head//3
2014-06-12 14:51:46.164132 7f9d60bd2700 10 getattrs 3.43_head/e61ef043/rbd_data.10af6b8b4567.0000000000001af0/head//3 = 0
2014-06-12 14:51:46.164147 7f9d60bd2700 10 stat 3.43_head/560ff043/rbd_data.108d6b8b4567.00000000000038f7/head//3
2014-06-12 14:51:46.164275 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 560ff043/rbd_data.108d6b8b4567.00000000000038f7/head//3
2014-06-12 14:51:46.164387 7f9d60bd2700 10 getattrs 3.43_head/560ff043/rbd_data.108d6b8b4567.00000000000038f7/head//3 = 0
2014-06-12 14:51:46.164393 7f9d60bd2700 10 stat 3.43_head/34110843/rbd_data.109e6b8b4567.0000000000001bf9/head//3
2014-06-12 14:51:46.164506 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 34110843/rbd_data.109e6b8b4567.0000000000001bf9/head//3
2014-06-12 14:51:46.167817 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:46.174962 7f9d60bd2700 10 getattrs 3.43_head/34110843/rbd_data.109e6b8b4567.0000000000001bf9/head//3 = 0
2014-06-12 14:51:46.174971 7f9d60bd2700 10 stat 3.43_head/b7320843/rbd_data.109e6b8b4567.000000000000204b/head//3
2014-06-12 14:51:46.175083 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid b7320843/rbd_data.109e6b8b4567.000000000000204b/head//3
2014-06-12 14:51:46.194316 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:46.205678 7f9d60bd2700 10 getattrs 3.43_head/b7320843/rbd_data.109e6b8b4567.000000000000204b/head//3 = 0
2014-06-12 14:51:46.205692 7f9d60bd2700 10 stat 3.43_head/d5440843/rbd_data.10926b8b4567.000000000000386a/head//3
2014-06-12 14:51:46.205847 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d5440843/rbd_data.10926b8b4567.000000000000386a/head//3
2014-06-12 14:51:46.220754 7f9d60bd2700 10 getattrs 3.43_head/d5440843/rbd_data.10926b8b4567.000000000000386a/head//3 = 0
2014-06-12 14:51:46.220775 7f9d60bd2700 10 stat 3.43_head/35670843/rbd_data.10ac6b8b4567.0000000000003722/head//3
2014-06-12 14:51:46.220930 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 35670843/rbd_data.10ac6b8b4567.0000000000003722/head//3
2014-06-12 14:51:46.221048 7f9d60bd2700 10 getattrs 3.43_head/35670843/rbd_data.10ac6b8b4567.0000000000003722/head//3 = 0
2014-06-12 14:51:46.221058 7f9d60bd2700 10 stat 3.43_head/bc670843/rbd_data.108f6b8b4567.0000000000002c0d/head//3
2014-06-12 14:51:46.221195 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid bc670843/rbd_data.108f6b8b4567.0000000000002c0d/head//3
2014-06-12 14:51:46.249026 7f9d60bd2700 10 getattrs 3.43_head/bc670843/rbd_data.108f6b8b4567.0000000000002c0d/head//3 = 0
2014-06-12 14:51:46.249036 7f9d60bd2700 10 stat 3.43_head/b2fd0843/rbd_data.10a86b8b4567.0000000000003407/head//3
2014-06-12 14:51:46.249141 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid b2fd0843/rbd_data.10a86b8b4567.0000000000003407/head//3
2014-06-12 14:51:46.264295 7f9d60bd2700 10 getattrs 3.43_head/b2fd0843/rbd_data.10a86b8b4567.0000000000003407/head//3 = 0
2014-06-12 14:51:46.264312 7f9d60bd2700 10 stat 3.43_head/677f0843/rbd_data.10a56b8b4567.0000000000001c12/head//3
2014-06-12 14:51:46.264421 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 677f0843/rbd_data.10a56b8b4567.0000000000001c12/head//3
2014-06-12 14:51:46.274416 7f9d60bd2700 10 getattrs 3.43_head/677f0843/rbd_data.10a56b8b4567.0000000000001c12/head//3 = 0
2014-06-12 14:51:46.274426 7f9d60bd2700 10 stat 3.43_head/d0af0843/rbd_data.10a86b8b4567.0000000000000d52/head//3
2014-06-12 14:51:46.274540 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d0af0843/rbd_data.10a86b8b4567.0000000000000d52/head//3
2014-06-12 14:51:46.279282 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:46.279307 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:46.279312 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:46.279314 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:46.279353 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:46.287455 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:46.287549 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:46.317244 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:46.326593 7f9d60bd2700 10 getattrs 3.43_head/d0af0843/rbd_data.10a86b8b4567.0000000000000d52/head//3 = 0
2014-06-12 14:51:46.326613 7f9d60bd2700 10 stat 3.43_head/1fbf0843/rbd_data.109b6b8b4567.000000000000082f/head//3
2014-06-12 14:51:46.326613 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:46.326834 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 1fbf0843/rbd_data.109b6b8b4567.000000000000082f/head//3
2014-06-12 14:51:46.366626 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:46.369457 7f9d60bd2700 10 getattrs 3.43_head/1fbf0843/rbd_data.109b6b8b4567.000000000000082f/head//3 = 0
2014-06-12 14:51:46.369467 7f9d60bd2700 10 stat 3.43_head/30511843/rbd_data.10976b8b4567.0000000000001218/head//3
2014-06-12 14:51:46.369585 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 30511843/rbd_data.10976b8b4567.0000000000001218/head//3
2014-06-12 14:51:46.371549 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:46.382252 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:46.382851 7f9d5a16f700  5 -- op tracker -- , seq: 748600, time: 2014-06-12 14:51:46.382731, event: header_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.382870 7f9d5a16f700  5 -- op tracker -- , seq: 748600, time: 2014-06-12 14:51:46.382735, event: throttled, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.382882 7f9d5a16f700  5 -- op tracker -- , seq: 748600, time: 2014-06-12 14:51:46.382801, event: all_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.382920 7f9d5a16f700  5 -- op tracker -- , seq: 748600, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.382925 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:46.382931 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:51:46.382939 7f9d5a16f700 15 osd.2 173 enqueue_op 0x55c3c00 prio 127 cost 10717 latency 0.000207 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:46.383007 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:46.384779 7f9d60bd2700 10 getattrs 3.43_head/30511843/rbd_data.10976b8b4567.0000000000001218/head//3 = 0
2014-06-12 14:51:46.384791 7f9d60bd2700 10 stat 3.43_head/73631843/rbd_data.10906b8b4567.0000000000000b56/head//3
2014-06-12 14:51:46.384909 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 73631843/rbd_data.10906b8b4567.0000000000000b56/head//3
2014-06-12 14:51:46.399331 7f9d60bd2700 10 getattrs 3.43_head/73631843/rbd_data.10906b8b4567.0000000000000b56/head//3 = 0
2014-06-12 14:51:46.399343 7f9d60bd2700 10 collection_getattrs 3.43_head
2014-06-12 14:51:46.399404 7f9d60bd2700 10 collection_getattrs 3.43_head = 0
2014-06-12 14:51:46.399412 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:46.399441 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:46.399449 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:46.399460 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:46.399465 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:46.399493 7f9d62bd6700 10 osd.2 173 dequeue_op 0x55c3c00 prio 127 cost 10717 latency 0.016761 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]
2014-06-12 14:51:46.399550 7f9d62bd6700  5 -- op tracker -- , seq: 748600, time: 2014-06-12 14:51:46.399543, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.399569 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] handle_message: 0x55c3c00
2014-06-12 14:51:46.399585 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:46.399608 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:46.399626 7f9d62bd6700  5 -- op tracker -- , seq: 748600, time: 2014-06-12 14:51:46.399621, event: started, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.399634 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  got 24 scrub map
2014-06-12 14:51:46.399739 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] map version is 173'2566
2014-06-12 14:51:46.399759 7f9d62bd6700 10 osd.2 173 dequeue_op 0x55c3c00 finish
2014-06-12 14:51:46.399771 7f9d62bd6700  5 -- op tracker -- , seq: 748600, time: 2014-06-12 14:51:46.399765, event: done, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.399793 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:46.399799 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:46.399798 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:46.399811 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:46.399857 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:46.399867 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:46.399898 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:46.399906 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:46.399914 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub replica 24 has 25 items
2014-06-12 14:51:46.399939 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2e9be043/rbd_data.10ae6b8b4567.00000000000030f0/head//3, auth == maps.end()
2014-06-12 14:51:46.399957 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 2e9be043/rbd_data.10ae6b8b4567.00000000000030f0/head//3
2014-06-12 14:51:46.399973 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 665fe043/rbd_data.10976b8b4567.000000000000125b/head//3, auth == maps.end()
2014-06-12 14:51:46.399984 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 665fe043/rbd_data.10976b8b4567.000000000000125b/head//3
2014-06-12 14:51:46.399998 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aff1f043/rbd_data.108d6b8b4567.00000000000000ae/head//3, auth == maps.end()
2014-06-12 14:51:46.400009 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj aff1f043/rbd_data.108d6b8b4567.00000000000000ae/head//3
2014-06-12 14:51:46.400020 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a502f043/rbd_data.10b26b8b4567.0000000000002b98/head//3, auth == maps.end()
2014-06-12 14:51:46.400032 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj a502f043/rbd_data.10b26b8b4567.0000000000002b98/head//3
2014-06-12 14:51:46.400044 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6f24f043/rbd_data.109f6b8b4567.0000000000003727/head//3, auth == maps.end()
2014-06-12 14:51:46.400056 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 6f24f043/rbd_data.109f6b8b4567.0000000000003727/head//3
2014-06-12 14:51:46.400068 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3, auth == maps.end()
2014-06-12 14:51:46.400078 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3
2014-06-12 14:51:46.400089 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 13c5f043/rbd_data.10a76b8b4567.00000000000035af/head//3, auth == maps.end()
2014-06-12 14:51:46.400099 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 13c5f043/rbd_data.10a76b8b4567.00000000000035af/head//3
2014-06-12 14:51:46.400110 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 84d8f043/rbd_data.108d6b8b4567.0000000000003506/head//3, auth == maps.end()
2014-06-12 14:51:46.400120 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 84d8f043/rbd_data.108d6b8b4567.0000000000003506/head//3
2014-06-12 14:51:46.400132 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9cd9f043/rbd_data.10b16b8b4567.0000000000002513/head//3, auth == maps.end()
2014-06-12 14:51:46.400142 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9cd9f043/rbd_data.10b16b8b4567.0000000000002513/head//3
2014-06-12 14:51:46.400153 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 10baf043/rbd_data.10a86b8b4567.0000000000002ccc/head//3, auth == maps.end()
2014-06-12 14:51:46.400163 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 10baf043/rbd_data.10a86b8b4567.0000000000002ccc/head//3
2014-06-12 14:51:46.400176 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 902bf043/rbd_data.10b36b8b4567.0000000000000757/head//3, auth == maps.end()
2014-06-12 14:51:46.400187 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 902bf043/rbd_data.10b36b8b4567.0000000000000757/head//3
2014-06-12 14:51:46.400198 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 18cdf043/rbd_data.10ac6b8b4567.0000000000003b1c/head//3, auth == maps.end()
2014-06-12 14:51:46.400208 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 18cdf043/rbd_data.10ac6b8b4567.0000000000003b1c/head//3
2014-06-12 14:51:46.400222 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e61ef043/rbd_data.10af6b8b4567.0000000000001af0/head//3, auth == maps.end()
2014-06-12 14:51:46.400232 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj e61ef043/rbd_data.10af6b8b4567.0000000000001af0/head//3
2014-06-12 14:51:46.400245 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 560ff043/rbd_data.108d6b8b4567.00000000000038f7/head//3, auth == maps.end()
2014-06-12 14:51:46.400256 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 560ff043/rbd_data.108d6b8b4567.00000000000038f7/head//3
2014-06-12 14:51:46.400267 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 34110843/rbd_data.109e6b8b4567.0000000000001bf9/head//3, auth == maps.end()
2014-06-12 14:51:46.400279 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 34110843/rbd_data.109e6b8b4567.0000000000001bf9/head//3
2014-06-12 14:51:46.400290 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b7320843/rbd_data.109e6b8b4567.000000000000204b/head//3, auth == maps.end()
2014-06-12 14:51:46.400301 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj b7320843/rbd_data.109e6b8b4567.000000000000204b/head//3
2014-06-12 14:51:46.400312 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d5440843/rbd_data.10926b8b4567.000000000000386a/head//3, auth == maps.end()
2014-06-12 14:51:46.400322 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d5440843/rbd_data.10926b8b4567.000000000000386a/head//3
2014-06-12 14:51:46.400333 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 35670843/rbd_data.10ac6b8b4567.0000000000003722/head//3, auth == maps.end()
2014-06-12 14:51:46.400344 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 35670843/rbd_data.10ac6b8b4567.0000000000003722/head//3
2014-06-12 14:51:46.400354 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bc670843/rbd_data.108f6b8b4567.0000000000002c0d/head//3, auth == maps.end()
2014-06-12 14:51:46.400366 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj bc670843/rbd_data.108f6b8b4567.0000000000002c0d/head//3
2014-06-12 14:51:46.400376 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b2fd0843/rbd_data.10a86b8b4567.0000000000003407/head//3, auth == maps.end()
2014-06-12 14:51:46.400387 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj b2fd0843/rbd_data.10a86b8b4567.0000000000003407/head//3
2014-06-12 14:51:46.400399 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 677f0843/rbd_data.10a56b8b4567.0000000000001c12/head//3, auth == maps.end()
2014-06-12 14:51:46.400410 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 677f0843/rbd_data.10a56b8b4567.0000000000001c12/head//3
2014-06-12 14:51:46.400423 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d0af0843/rbd_data.10a86b8b4567.0000000000000d52/head//3, auth == maps.end()
2014-06-12 14:51:46.400434 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d0af0843/rbd_data.10a86b8b4567.0000000000000d52/head//3
2014-06-12 14:51:46.400447 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1fbf0843/rbd_data.109b6b8b4567.000000000000082f/head//3, auth == maps.end()
2014-06-12 14:51:46.400459 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 1fbf0843/rbd_data.109b6b8b4567.000000000000082f/head//3
2014-06-12 14:51:46.400471 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 30511843/rbd_data.10976b8b4567.0000000000001218/head//3, auth == maps.end()
2014-06-12 14:51:46.400483 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 30511843/rbd_data.10976b8b4567.0000000000001218/head//3
2014-06-12 14:51:46.400495 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 73631843/rbd_data.10906b8b4567.0000000000000b56/head//3, auth == maps.end()
2014-06-12 14:51:46.400508 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 73631843/rbd_data.10906b8b4567.0000000000000b56/head//3
2014-06-12 14:51:46.400522 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] 
2014-06-12 14:51:46.400531 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub
2014-06-12 14:51:46.400547 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  73631843/rbd_data.10906b8b4567.0000000000000b56/head//3 73631843/rbd_data.10906b8b4567.0000000000000b56/head//3(173'2385 client.4281.0:134735 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2385)
2014-06-12 14:51:46.400564 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  30511843/rbd_data.10976b8b4567.0000000000001218/head//3 30511843/rbd_data.10976b8b4567.0000000000001218/head//3(173'773 client.4288.0:40695 wrlock_by=unknown.0.0:0 dirty s 4194304 uv773)
2014-06-12 14:51:46.400580 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  1fbf0843/rbd_data.109b6b8b4567.000000000000082f/head//3 1fbf0843/rbd_data.109b6b8b4567.000000000000082f/head//3(173'2293 client.4292.0:128514 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2293)
2014-06-12 14:51:46.400596 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d0af0843/rbd_data.10a86b8b4567.0000000000000d52/head//3 d0af0843/rbd_data.10a86b8b4567.0000000000000d52/head//3(173'2487 client.4305.0:139654 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2487)
2014-06-12 14:51:46.400611 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  677f0843/rbd_data.10a56b8b4567.0000000000001c12/head//3 677f0843/rbd_data.10a56b8b4567.0000000000001c12/head//3(173'1062 client.4302.0:60665 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1062)
2014-06-12 14:51:46.400626 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  b2fd0843/rbd_data.10a86b8b4567.0000000000003407/head//3 b2fd0843/rbd_data.10a86b8b4567.0000000000003407/head//3(173'1702 client.4305.0:102588 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1702)
2014-06-12 14:51:46.400640 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  bc670843/rbd_data.108f6b8b4567.0000000000002c0d/head//3 bc670843/rbd_data.108f6b8b4567.0000000000002c0d/head//3(173'1507 client.4280.0:87365 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1507)
2014-06-12 14:51:46.400656 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  35670843/rbd_data.10ac6b8b4567.0000000000003722/head//3 35670843/rbd_data.10ac6b8b4567.0000000000003722/head//3(173'1972 client.4309.0:108461 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1972)
2014-06-12 14:51:46.400671 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d5440843/rbd_data.10926b8b4567.000000000000386a/head//3 d5440843/rbd_data.10926b8b4567.000000000000386a/head//3(173'1988 client.4283.0:111443 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1988)
2014-06-12 14:51:46.400686 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  b7320843/rbd_data.109e6b8b4567.000000000000204b/head//3 b7320843/rbd_data.109e6b8b4567.000000000000204b/head//3(173'1223 client.4295.0:68491 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1223)
2014-06-12 14:51:46.400702 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  34110843/rbd_data.109e6b8b4567.0000000000001bf9/head//3 34110843/rbd_data.109e6b8b4567.0000000000001bf9/head//3(173'1076 client.4295.0:60686 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1076)
2014-06-12 14:51:46.400717 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  560ff043/rbd_data.108d6b8b4567.00000000000038f7/head//3 560ff043/rbd_data.108d6b8b4567.00000000000038f7/head//3(173'1859 client.4278.0:113219 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1859)
2014-06-12 14:51:46.400733 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  e61ef043/rbd_data.10af6b8b4567.0000000000001af0/head//3 e61ef043/rbd_data.10af6b8b4567.0000000000001af0/head//3(173'1100 client.4312.0:60866 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1100)
2014-06-12 14:51:46.400748 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  18cdf043/rbd_data.10ac6b8b4567.0000000000003b1c/head//3 18cdf043/rbd_data.10ac6b8b4567.0000000000003b1c/head//3(173'2029 client.4309.0:115185 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2029)
2014-06-12 14:51:46.400763 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  902bf043/rbd_data.10b36b8b4567.0000000000000757/head//3 902bf043/rbd_data.10b36b8b4567.0000000000000757/head//3(173'2252 client.4316.0:130731 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2252)
2014-06-12 14:51:46.400783 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  10baf043/rbd_data.10a86b8b4567.0000000000002ccc/head//3 10baf043/rbd_data.10a86b8b4567.0000000000002ccc/head//3(173'1475 client.4305.0:89780 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1475)
2014-06-12 14:51:46.400798 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9cd9f043/rbd_data.10b16b8b4567.0000000000002513/head//3 9cd9f043/rbd_data.10b16b8b4567.0000000000002513/head//3(173'1356 client.4314.0:76526 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1356)
2014-06-12 14:51:46.400813 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  84d8f043/rbd_data.108d6b8b4567.0000000000003506/head//3 84d8f043/rbd_data.108d6b8b4567.0000000000003506/head//3(173'1796 client.4278.0:106328 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1796)
2014-06-12 14:51:46.400829 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  13c5f043/rbd_data.10a76b8b4567.00000000000035af/head//3 13c5f043/rbd_data.10a76b8b4567.00000000000035af/head//3(173'1824 client.4304.0:106268 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1824)
2014-06-12 14:51:46.400844 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3 8eb5f043/rbd_data.109b6b8b4567.0000000000001039/head//3(173'2566 client.4292.0:142326 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2566)
2014-06-12 14:51:46.400859 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  6f24f043/rbd_data.109f6b8b4567.0000000000003727/head//3 6f24f043/rbd_data.109f6b8b4567.0000000000003727/head//3(173'1804 client.4296.0:107361 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1804)
2014-06-12 14:51:46.400874 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  a502f043/rbd_data.10b26b8b4567.0000000000002b98/head//3 a502f043/rbd_data.10b26b8b4567.0000000000002b98/head//3(173'1435 client.4315.0:88275 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1435)
2014-06-12 14:51:46.400889 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  aff1f043/rbd_data.108d6b8b4567.00000000000000ae/head//3 aff1f043/rbd_data.108d6b8b4567.00000000000000ae/head//3(173'2078 client.4278.0:119222 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2078)
2014-06-12 14:51:46.400905 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  665fe043/rbd_data.10976b8b4567.000000000000125b/head//3 665fe043/rbd_data.10976b8b4567.000000000000125b/head//3(173'796 client.4288.0:41166 wrlock_by=unknown.0.0:0 dirty s 4194304 uv796)
2014-06-12 14:51:46.400920 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  2e9be043/rbd_data.10ae6b8b4567.00000000000030f0/head//3 2e9be043/rbd_data.10ae6b8b4567.00000000000030f0/head//3(173'1574 client.4311.0:95253 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1574)
2014-06-12 14:51:46.400931 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:46.400944 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:46.400954 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:46.400957 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:46.400958 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:46.401000 7f9d60bd2700 10 collection_list_partial 3.43_head start:6d441843//0//-1 is_max:0
2014-06-12 14:51:46.401255 7f9d60bd2700 20 objects: [86641843/rbd_data.10b46b8b4567.000000000000041c/head//3,54e51843/rbd_data.109c6b8b4567.000000000000151d/head//3,f9361843/rbd_data.10986b8b4567.0000000000001648/head//3,c4681843/rbd_data.10916b8b4567.0000000000000e2d/head//3,93ab1843/rbd_data.10906b8b4567.0000000000000c42/head//3,752d1843/rbd_data.10946b8b4567.0000000000001b04/head//3,67ee1843/rbd_data.10a56b8b4567.0000000000000f58/head//3,c50f1843/rbd_data.10ae6b8b4567.0000000000001465/head//3,8c0f1843/rbd_data.10a06b8b4567.000000000000363d/head//3,ead02843/rbd_data.10aa6b8b4567.0000000000000f41/head//3,a9232843/rbd_data.10b36b8b4567.00000000000007db/head//3,63652843/rbd_data.10a06b8b4567.000000000000143c/head//3,eda82843/rbd_data.10a86b8b4567.00000000000014e3/head//3,16b92843/rbd_data.10b36b8b4567.0000000000002d47/head//3,a28a2843/rbd_data.10ac6b8b4567.0000000000001516/head//3,fcfa2843/rbd_data.10ab6b8b4567.0000000000000c79/head//3,16213843/rbd_data.10986b8b4567.0000000000001fd6/head//3,c2623843/rbd_data.10b16b8b4567.0000000000002605/head//3,c7d43843/rbd_data.10ac6b8b4567.00000000000039e1/head//3,b8b63843/rbd_data.10a26b8b4567.00000000000018b9/head//3,9fb73843/rbd_data.10a36b8b4567.0000000000002c93/head//3,f1c73843/rbd_data.109c6b8b4567.0000000000001b57/head//3,66d93843/rbd_data.109b6b8b4567.00000000000017dc/head//3,87f04843/rbd_data.109b6b8b4567.000000000000285f/head//3,86634843/rbd_data.10aa6b8b4567.0000000000001f5f/head//3]
2014-06-12 14:51:46.401756 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  requesting scrubmap from osd.24
2014-06-12 14:51:46.401782 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:46.401801 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:46.401812 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:46.401819 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:46.401826 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:46.401832 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_map_chunk [6d441843//0//-1,89f34843//0//-1)
2014-06-12 14:51:46.401841 7f9d60bd2700 10 collection_list_partial 3.43_head start:6d441843//0//-1 is_max:0
2014-06-12 14:51:46.402384 7f9d60bd2700 20 objects: [86641843/rbd_data.10b46b8b4567.000000000000041c/head//3,54e51843/rbd_data.109c6b8b4567.000000000000151d/head//3,f9361843/rbd_data.10986b8b4567.0000000000001648/head//3,c4681843/rbd_data.10916b8b4567.0000000000000e2d/head//3,93ab1843/rbd_data.10906b8b4567.0000000000000c42/head//3,752d1843/rbd_data.10946b8b4567.0000000000001b04/head//3,67ee1843/rbd_data.10a56b8b4567.0000000000000f58/head//3,c50f1843/rbd_data.10ae6b8b4567.0000000000001465/head//3,8c0f1843/rbd_data.10a06b8b4567.000000000000363d/head//3,ead02843/rbd_data.10aa6b8b4567.0000000000000f41/head//3,a9232843/rbd_data.10b36b8b4567.00000000000007db/head//3,63652843/rbd_data.10a06b8b4567.000000000000143c/head//3,eda82843/rbd_data.10a86b8b4567.00000000000014e3/head//3,16b92843/rbd_data.10b36b8b4567.0000000000002d47/head//3,a28a2843/rbd_data.10ac6b8b4567.0000000000001516/head//3,fcfa2843/rbd_data.10ab6b8b4567.0000000000000c79/head//3,16213843/rbd_data.10986b8b4567.0000000000001fd6/head//3,c2623843/rbd_data.10b16b8b4567.0000000000002605/head//3,c7d43843/rbd_data.10ac6b8b4567.00000000000039e1/head//3,b8b63843/rbd_data.10a26b8b4567.00000000000018b9/head//3,9fb73843/rbd_data.10a36b8b4567.0000000000002c93/head//3,f1c73843/rbd_data.109c6b8b4567.0000000000001b57/head//3,66d93843/rbd_data.109b6b8b4567.00000000000017dc/head//3,87f04843/rbd_data.109b6b8b4567.000000000000285f/head//3,86634843/rbd_data.10aa6b8b4567.0000000000001f5f/head//3,89f34843/rbd_data.10906b8b4567.000000000000020b/head//3,62464843/rbd_data.10906b8b4567.00000000000031ac/head//3,15d64843/rbd_data.10ae6b8b4567.000000000000288b/head//3,11205843/rbd_data.10af6b8b4567.00000000000019fb/head//3,26c35843/rbd_data.10b16b8b4567.0000000000003558/head//3,7045843/rbd_data.10a06b8b4567.000000000000162c/head//3,fbcb5843/rbd_data.10976b8b4567.0000000000002613/head//3,a1de5843/rbd_data.10a56b8b4567.00000000000030aa/head//3,bbdf5843/rbd_data.10af6b8b4567.0000000000002625/head//3,69406843/rbd_data.10ab6b8b4567.000000000000317a/head//3,af216843/rbd_data.10966b8b4567.0000000000001e15/head//3,6cf16843/rbd_data.10b36b8b4567.0000000000000e06/head//3,d7926843/rbd_data.10ae6b8b4567.0000000000000042/head//3,7a136843/rbd_data.10a26b8b4567.00000000000000fe/head//3,950a6843/rbd_data.10956b8b4567.0000000000003176/head//3,c41b6843/rbd_data.10926b8b4567.00000000000026b1/head//3,2deb6843/rbd_data.10b46b8b4567.0000000000002cb1/head//3,9c2d6843/rbd_data.10926b8b4567.0000000000002aed/head//3,864f6843/rbd_data.10956b8b4567.0000000000001275/head//3,7d807843/rbd_data.10ae6b8b4567.0000000000000448/head//3,72517843/rbd_data.108e6b8b4567.00000000000033c4/head//3,60a17843/rbd_data.108f6b8b4567.0000000000001793/head//3,ce227843/rbd_data.10b16b8b4567.0000000000002ce2/head//3,40827843/rbd_data.10926b8b4567.0000000000003ba0/head//3,64b27843/rbd_data.10ac6b8b4567.0000000000003668/head//3,14147843/rbd_data.10976b8b4567.00000000000024b0/head//3,47957843/rbd_data.10a06b8b4567.000000000000301f/head//3,c6277843/rbd_data.109e6b8b4567.00000000000030b7/head//3,d0697843/rbd_data.10956b8b4567.0000000000000ac8/head//3,d57c7843/rbd_data.10a76b8b4567.0000000000001b13/head//3,29fc7843/rbd_data.10a96b8b4567.00000000000026bd/head//3,1b5e7843/rbd_data.10926b8b4567.00000000000033dc/head//3,db6f7843/rbd_data.10b06b8b4567.0000000000000de6/head//3,a98f7843/rbd_data.10a36b8b4567.0000000000001128/head//3,11438843/rbd_data.108d6b8b4567.0000000000002e57/head//3,d9558843/rbd_data.10b26b8b4567.0000000000002817/head//3,cfd58843/rbd_data.10946b8b4567.0000000000001659/head//3,1fea8843/rbd_data.109e6b8b4567.0000000000002ba7/head//3,cad09843/rbd_data.109b6b8b4567.00000000000034ac/head//3]
2014-06-12 14:51:46.402472 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:46.402492 7f9d60bd2700 10 stat 3.43_head/86641843/rbd_data.10b46b8b4567.000000000000041c/head//3
2014-06-12 14:51:46.402630 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 86641843/rbd_data.10b46b8b4567.000000000000041c/head//3
2014-06-12 14:51:46.423027 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:46.423089 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:46.469374 7f9d60bd2700 10 getattrs 3.43_head/86641843/rbd_data.10b46b8b4567.000000000000041c/head//3 = 0
2014-06-12 14:51:46.469405 7f9d60bd2700 10 stat 3.43_head/54e51843/rbd_data.109c6b8b4567.000000000000151d/head//3
2014-06-12 14:51:46.469597 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 54e51843/rbd_data.109c6b8b4567.000000000000151d/head//3
2014-06-12 14:51:46.494846 7f9d60bd2700 10 getattrs 3.43_head/54e51843/rbd_data.109c6b8b4567.000000000000151d/head//3 = 0
2014-06-12 14:51:46.494855 7f9d60bd2700 10 stat 3.43_head/f9361843/rbd_data.10986b8b4567.0000000000001648/head//3
2014-06-12 14:51:46.494955 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f9361843/rbd_data.10986b8b4567.0000000000001648/head//3
2014-06-12 14:51:46.516508 7f9d60bd2700 10 getattrs 3.43_head/f9361843/rbd_data.10986b8b4567.0000000000001648/head//3 = 0
2014-06-12 14:51:46.516519 7f9d60bd2700 10 stat 3.43_head/c4681843/rbd_data.10916b8b4567.0000000000000e2d/head//3
2014-06-12 14:51:46.516637 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c4681843/rbd_data.10916b8b4567.0000000000000e2d/head//3
2014-06-12 14:51:46.522397 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:46.523247 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:46.532696 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:46.553378 7f9d60bd2700 10 getattrs 3.43_head/c4681843/rbd_data.10916b8b4567.0000000000000e2d/head//3 = 0
2014-06-12 14:51:46.553399 7f9d60bd2700 10 stat 3.43_head/93ab1843/rbd_data.10906b8b4567.0000000000000c42/head//3
2014-06-12 14:51:46.553530 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 93ab1843/rbd_data.10906b8b4567.0000000000000c42/head//3
2014-06-12 14:51:46.570159 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:46.585925 7f9d60bd2700 10 getattrs 3.43_head/93ab1843/rbd_data.10906b8b4567.0000000000000c42/head//3 = 0
2014-06-12 14:51:46.585943 7f9d60bd2700 10 stat 3.43_head/752d1843/rbd_data.10946b8b4567.0000000000001b04/head//3
2014-06-12 14:51:46.586096 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 752d1843/rbd_data.10946b8b4567.0000000000001b04/head//3
2014-06-12 14:51:46.604134 7f9d60bd2700 10 getattrs 3.43_head/752d1843/rbd_data.10946b8b4567.0000000000001b04/head//3 = 0
2014-06-12 14:51:46.604148 7f9d60bd2700 10 stat 3.43_head/67ee1843/rbd_data.10a56b8b4567.0000000000000f58/head//3
2014-06-12 14:51:46.604250 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 67ee1843/rbd_data.10a56b8b4567.0000000000000f58/head//3
2014-06-12 14:51:46.659325 7f9d60bd2700 10 getattrs 3.43_head/67ee1843/rbd_data.10a56b8b4567.0000000000000f58/head//3 = 0
2014-06-12 14:51:46.659334 7f9d60bd2700 10 stat 3.43_head/c50f1843/rbd_data.10ae6b8b4567.0000000000001465/head//3
2014-06-12 14:51:46.659438 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c50f1843/rbd_data.10ae6b8b4567.0000000000001465/head//3
2014-06-12 14:51:46.662046 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:46.680254 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:46.684229 7f9d60bd2700 10 getattrs 3.43_head/c50f1843/rbd_data.10ae6b8b4567.0000000000001465/head//3 = 0
2014-06-12 14:51:46.684243 7f9d60bd2700 10 stat 3.43_head/8c0f1843/rbd_data.10a06b8b4567.000000000000363d/head//3
2014-06-12 14:51:46.684407 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 8c0f1843/rbd_data.10a06b8b4567.000000000000363d/head//3
2014-06-12 14:51:46.700882 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:46.710267 7f9d60bd2700 10 getattrs 3.43_head/8c0f1843/rbd_data.10a06b8b4567.000000000000363d/head//3 = 0
2014-06-12 14:51:46.710274 7f9d60bd2700 10 stat 3.43_head/ead02843/rbd_data.10aa6b8b4567.0000000000000f41/head//3
2014-06-12 14:51:46.710370 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid ead02843/rbd_data.10aa6b8b4567.0000000000000f41/head//3
2014-06-12 14:51:46.710706 7f9d60bd2700 10 getattrs 3.43_head/ead02843/rbd_data.10aa6b8b4567.0000000000000f41/head//3 = 0
2014-06-12 14:51:46.710717 7f9d60bd2700 10 stat 3.43_head/a9232843/rbd_data.10b36b8b4567.00000000000007db/head//3
2014-06-12 14:51:46.710808 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid a9232843/rbd_data.10b36b8b4567.00000000000007db/head//3
2014-06-12 14:51:46.726676 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:46.744267 7f9d60bd2700 10 getattrs 3.43_head/a9232843/rbd_data.10b36b8b4567.00000000000007db/head//3 = 0
2014-06-12 14:51:46.744284 7f9d60bd2700 10 stat 3.43_head/63652843/rbd_data.10a06b8b4567.000000000000143c/head//3
2014-06-12 14:51:46.744406 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 63652843/rbd_data.10a06b8b4567.000000000000143c/head//3
2014-06-12 14:51:46.756836 7f9d60bd2700 10 getattrs 3.43_head/63652843/rbd_data.10a06b8b4567.000000000000143c/head//3 = 0
2014-06-12 14:51:46.756854 7f9d60bd2700 10 stat 3.43_head/eda82843/rbd_data.10a86b8b4567.00000000000014e3/head//3
2014-06-12 14:51:46.756977 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid eda82843/rbd_data.10a86b8b4567.00000000000014e3/head//3
2014-06-12 14:51:46.768340 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:46.780681 7f9d60bd2700 10 getattrs 3.43_head/eda82843/rbd_data.10a86b8b4567.00000000000014e3/head//3 = 0
2014-06-12 14:51:46.780691 7f9d60bd2700 10 stat 3.43_head/16b92843/rbd_data.10b36b8b4567.0000000000002d47/head//3
2014-06-12 14:51:46.780810 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 16b92843/rbd_data.10b36b8b4567.0000000000002d47/head//3
2014-06-12 14:51:46.791805 7f9d60bd2700 10 getattrs 3.43_head/16b92843/rbd_data.10b36b8b4567.0000000000002d47/head//3 = 0
2014-06-12 14:51:46.791816 7f9d60bd2700 10 stat 3.43_head/a28a2843/rbd_data.10ac6b8b4567.0000000000001516/head//3
2014-06-12 14:51:46.791936 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid a28a2843/rbd_data.10ac6b8b4567.0000000000001516/head//3
2014-06-12 14:51:46.802269 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:46.805882 7f9d60bd2700 10 getattrs 3.43_head/a28a2843/rbd_data.10ac6b8b4567.0000000000001516/head//3 = 0
2014-06-12 14:51:46.805893 7f9d60bd2700 10 stat 3.43_head/fcfa2843/rbd_data.10ab6b8b4567.0000000000000c79/head//3
2014-06-12 14:51:46.806024 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid fcfa2843/rbd_data.10ab6b8b4567.0000000000000c79/head//3
2014-06-12 14:51:46.825440 7f9d60bd2700 10 getattrs 3.43_head/fcfa2843/rbd_data.10ab6b8b4567.0000000000000c79/head//3 = 0
2014-06-12 14:51:46.825454 7f9d60bd2700 10 stat 3.43_head/16213843/rbd_data.10986b8b4567.0000000000001fd6/head//3
2014-06-12 14:51:46.825549 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 16213843/rbd_data.10986b8b4567.0000000000001fd6/head//3
2014-06-12 14:51:46.853575 7f9d60bd2700 10 getattrs 3.43_head/16213843/rbd_data.10986b8b4567.0000000000001fd6/head//3 = 0
2014-06-12 14:51:46.853589 7f9d60bd2700 10 stat 3.43_head/c2623843/rbd_data.10b16b8b4567.0000000000002605/head//3
2014-06-12 14:51:46.853760 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c2623843/rbd_data.10b16b8b4567.0000000000002605/head//3
2014-06-12 14:51:46.855405 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:46.865786 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:46.869117 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:46.870537 7f9d60bd2700 10 getattrs 3.43_head/c2623843/rbd_data.10b16b8b4567.0000000000002605/head//3 = 0
2014-06-12 14:51:46.870553 7f9d60bd2700 10 stat 3.43_head/c7d43843/rbd_data.10ac6b8b4567.00000000000039e1/head//3
2014-06-12 14:51:46.870709 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c7d43843/rbd_data.10ac6b8b4567.00000000000039e1/head//3
2014-06-12 14:51:46.882686 7f9d60bd2700 10 getattrs 3.43_head/c7d43843/rbd_data.10ac6b8b4567.00000000000039e1/head//3 = 0
2014-06-12 14:51:46.882699 7f9d60bd2700 10 stat 3.43_head/b8b63843/rbd_data.10a26b8b4567.00000000000018b9/head//3
2014-06-12 14:51:46.882854 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid b8b63843/rbd_data.10a26b8b4567.00000000000018b9/head//3
2014-06-12 14:51:46.895857 7f9d60bd2700 10 getattrs 3.43_head/b8b63843/rbd_data.10a26b8b4567.00000000000018b9/head//3 = 0
2014-06-12 14:51:46.895865 7f9d60bd2700 10 stat 3.43_head/9fb73843/rbd_data.10a36b8b4567.0000000000002c93/head//3
2014-06-12 14:51:46.895944 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9fb73843/rbd_data.10a36b8b4567.0000000000002c93/head//3
2014-06-12 14:51:46.897359 7f9d5a16f700  5 -- op tracker -- , seq: 748601, time: 2014-06-12 14:51:46.897223, event: header_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.897381 7f9d5a16f700  5 -- op tracker -- , seq: 748601, time: 2014-06-12 14:51:46.897227, event: throttled, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.897420 7f9d5a16f700  5 -- op tracker -- , seq: 748601, time: 2014-06-12 14:51:46.897303, event: all_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.897431 7f9d5a16f700  5 -- op tracker -- , seq: 748601, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.897439 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:46.897448 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:51:46.897460 7f9d5a16f700 15 osd.2 173 enqueue_op 0x876f000 prio 127 cost 10717 latency 0.000236 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:46.897536 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:46.907898 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:46.914629 7f9d60bd2700 10 getattrs 3.43_head/9fb73843/rbd_data.10a36b8b4567.0000000000002c93/head//3 = 0
2014-06-12 14:51:46.914637 7f9d60bd2700 10 stat 3.43_head/f1c73843/rbd_data.109c6b8b4567.0000000000001b57/head//3
2014-06-12 14:51:46.914726 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f1c73843/rbd_data.109c6b8b4567.0000000000001b57/head//3
2014-06-12 14:51:46.930117 7f9d60bd2700 10 getattrs 3.43_head/f1c73843/rbd_data.109c6b8b4567.0000000000001b57/head//3 = 0
2014-06-12 14:51:46.930133 7f9d60bd2700 10 stat 3.43_head/66d93843/rbd_data.109b6b8b4567.00000000000017dc/head//3
2014-06-12 14:51:46.930290 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 66d93843/rbd_data.109b6b8b4567.00000000000017dc/head//3
2014-06-12 14:51:46.954225 7f9d60bd2700 10 getattrs 3.43_head/66d93843/rbd_data.109b6b8b4567.00000000000017dc/head//3 = 0
2014-06-12 14:51:46.954239 7f9d60bd2700 10 stat 3.43_head/87f04843/rbd_data.109b6b8b4567.000000000000285f/head//3
2014-06-12 14:51:46.954394 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 87f04843/rbd_data.109b6b8b4567.000000000000285f/head//3
2014-06-12 14:51:46.962677 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:46.966366 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:46.970534 7f9d60bd2700 10 getattrs 3.43_head/87f04843/rbd_data.109b6b8b4567.000000000000285f/head//3 = 0
2014-06-12 14:51:46.970541 7f9d60bd2700 10 stat 3.43_head/86634843/rbd_data.10aa6b8b4567.0000000000001f5f/head//3
2014-06-12 14:51:46.970633 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 86634843/rbd_data.10aa6b8b4567.0000000000001f5f/head//3
2014-06-12 14:51:46.995082 7f9d60bd2700 10 getattrs 3.43_head/86634843/rbd_data.10aa6b8b4567.0000000000001f5f/head//3 = 0
2014-06-12 14:51:46.995100 7f9d60bd2700 10 collection_getattrs 3.43_head
2014-06-12 14:51:46.995156 7f9d60bd2700 10 collection_getattrs 3.43_head = 0
2014-06-12 14:51:46.995160 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:46.995194 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:46.995201 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:46.995211 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:46.995214 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:46.995253 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x876f000 prio 127 cost 10717 latency 0.098030 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]
2014-06-12 14:51:46.995304 7f9d6a3e5700  5 -- op tracker -- , seq: 748601, time: 2014-06-12 14:51:46.995296, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.995342 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] handle_message: 0x876f000
2014-06-12 14:51:46.995366 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:46.995385 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:46.995402 7f9d6a3e5700  5 -- op tracker -- , seq: 748601, time: 2014-06-12 14:51:46.995397, event: started, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.995416 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  got 24 scrub map
2014-06-12 14:51:46.995522 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] map version is 173'2566
2014-06-12 14:51:46.995542 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x876f000 finish
2014-06-12 14:51:46.995553 7f9d6a3e5700  5 -- op tracker -- , seq: 748601, time: 2014-06-12 14:51:46.995547, event: done, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:46.995587 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:46.995581 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:46.995592 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:46.995595 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:46.995617 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:46.995629 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:46.995668 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:46.995675 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:46.995681 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub replica 24 has 25 items
2014-06-12 14:51:46.995702 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 86641843/rbd_data.10b46b8b4567.000000000000041c/head//3, auth == maps.end()
2014-06-12 14:51:46.995721 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 86641843/rbd_data.10b46b8b4567.000000000000041c/head//3
2014-06-12 14:51:46.995735 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 54e51843/rbd_data.109c6b8b4567.000000000000151d/head//3, auth == maps.end()
2014-06-12 14:51:46.995745 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 54e51843/rbd_data.109c6b8b4567.000000000000151d/head//3
2014-06-12 14:51:46.995755 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f9361843/rbd_data.10986b8b4567.0000000000001648/head//3, auth == maps.end()
2014-06-12 14:51:46.995763 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f9361843/rbd_data.10986b8b4567.0000000000001648/head//3
2014-06-12 14:51:46.995773 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c4681843/rbd_data.10916b8b4567.0000000000000e2d/head//3, auth == maps.end()
2014-06-12 14:51:46.995781 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c4681843/rbd_data.10916b8b4567.0000000000000e2d/head//3
2014-06-12 14:51:46.995791 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 93ab1843/rbd_data.10906b8b4567.0000000000000c42/head//3, auth == maps.end()
2014-06-12 14:51:46.995800 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 93ab1843/rbd_data.10906b8b4567.0000000000000c42/head//3
2014-06-12 14:51:46.995809 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 752d1843/rbd_data.10946b8b4567.0000000000001b04/head//3, auth == maps.end()
2014-06-12 14:51:46.995818 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 752d1843/rbd_data.10946b8b4567.0000000000001b04/head//3
2014-06-12 14:51:46.995828 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 67ee1843/rbd_data.10a56b8b4567.0000000000000f58/head//3, auth == maps.end()
2014-06-12 14:51:46.995837 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 67ee1843/rbd_data.10a56b8b4567.0000000000000f58/head//3
2014-06-12 14:51:46.995848 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c50f1843/rbd_data.10ae6b8b4567.0000000000001465/head//3, auth == maps.end()
2014-06-12 14:51:46.995859 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c50f1843/rbd_data.10ae6b8b4567.0000000000001465/head//3
2014-06-12 14:51:46.995868 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8c0f1843/rbd_data.10a06b8b4567.000000000000363d/head//3, auth == maps.end()
2014-06-12 14:51:46.995879 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 8c0f1843/rbd_data.10a06b8b4567.000000000000363d/head//3
2014-06-12 14:51:46.995888 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ead02843/rbd_data.10aa6b8b4567.0000000000000f41/head//3, auth == maps.end()
2014-06-12 14:51:46.995897 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj ead02843/rbd_data.10aa6b8b4567.0000000000000f41/head//3
2014-06-12 14:51:46.995906 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a9232843/rbd_data.10b36b8b4567.00000000000007db/head//3, auth == maps.end()
2014-06-12 14:51:46.995915 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj a9232843/rbd_data.10b36b8b4567.00000000000007db/head//3
2014-06-12 14:51:46.995924 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 63652843/rbd_data.10a06b8b4567.000000000000143c/head//3, auth == maps.end()
2014-06-12 14:51:46.995933 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 63652843/rbd_data.10a06b8b4567.000000000000143c/head//3
2014-06-12 14:51:46.995942 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj eda82843/rbd_data.10a86b8b4567.00000000000014e3/head//3, auth == maps.end()
2014-06-12 14:51:46.995950 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj eda82843/rbd_data.10a86b8b4567.00000000000014e3/head//3
2014-06-12 14:51:46.995959 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 16b92843/rbd_data.10b36b8b4567.0000000000002d47/head//3, auth == maps.end()
2014-06-12 14:51:46.995968 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 16b92843/rbd_data.10b36b8b4567.0000000000002d47/head//3
2014-06-12 14:51:46.995979 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a28a2843/rbd_data.10ac6b8b4567.0000000000001516/head//3, auth == maps.end()
2014-06-12 14:51:46.995989 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj a28a2843/rbd_data.10ac6b8b4567.0000000000001516/head//3
2014-06-12 14:51:46.995999 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fcfa2843/rbd_data.10ab6b8b4567.0000000000000c79/head//3, auth == maps.end()
2014-06-12 14:51:46.996007 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj fcfa2843/rbd_data.10ab6b8b4567.0000000000000c79/head//3
2014-06-12 14:51:46.996017 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 16213843/rbd_data.10986b8b4567.0000000000001fd6/head//3, auth == maps.end()
2014-06-12 14:51:46.996031 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 16213843/rbd_data.10986b8b4567.0000000000001fd6/head//3
2014-06-12 14:51:46.996040 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c2623843/rbd_data.10b16b8b4567.0000000000002605/head//3, auth == maps.end()
2014-06-12 14:51:46.996049 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c2623843/rbd_data.10b16b8b4567.0000000000002605/head//3
2014-06-12 14:51:46.996058 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c7d43843/rbd_data.10ac6b8b4567.00000000000039e1/head//3, auth == maps.end()
2014-06-12 14:51:46.996066 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c7d43843/rbd_data.10ac6b8b4567.00000000000039e1/head//3
2014-06-12 14:51:46.996075 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b8b63843/rbd_data.10a26b8b4567.00000000000018b9/head//3, auth == maps.end()
2014-06-12 14:51:46.996084 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj b8b63843/rbd_data.10a26b8b4567.00000000000018b9/head//3
2014-06-12 14:51:46.996093 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9fb73843/rbd_data.10a36b8b4567.0000000000002c93/head//3, auth == maps.end()
2014-06-12 14:51:46.996101 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9fb73843/rbd_data.10a36b8b4567.0000000000002c93/head//3
2014-06-12 14:51:46.996110 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f1c73843/rbd_data.109c6b8b4567.0000000000001b57/head//3, auth == maps.end()
2014-06-12 14:51:46.996119 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f1c73843/rbd_data.109c6b8b4567.0000000000001b57/head//3
2014-06-12 14:51:46.996129 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 66d93843/rbd_data.109b6b8b4567.00000000000017dc/head//3, auth == maps.end()
2014-06-12 14:51:46.996139 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 66d93843/rbd_data.109b6b8b4567.00000000000017dc/head//3
2014-06-12 14:51:46.996149 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 87f04843/rbd_data.109b6b8b4567.000000000000285f/head//3, auth == maps.end()
2014-06-12 14:51:46.996159 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 87f04843/rbd_data.109b6b8b4567.000000000000285f/head//3
2014-06-12 14:51:46.996168 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 86634843/rbd_data.10aa6b8b4567.0000000000001f5f/head//3, auth == maps.end()
2014-06-12 14:51:46.996178 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 86634843/rbd_data.10aa6b8b4567.0000000000001f5f/head//3
2014-06-12 14:51:46.996190 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] 
2014-06-12 14:51:46.996198 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub
2014-06-12 14:51:46.996211 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  86634843/rbd_data.10aa6b8b4567.0000000000001f5f/head//3 86634843/rbd_data.10aa6b8b4567.0000000000001f5f/head//3(173'1198 client.4307.0:68613 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1198)
2014-06-12 14:51:46.996226 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  87f04843/rbd_data.109b6b8b4567.000000000000285f/head//3 87f04843/rbd_data.109b6b8b4567.000000000000285f/head//3(173'1488 client.4292.0:80201 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1488)
2014-06-12 14:51:46.996239 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  66d93843/rbd_data.109b6b8b4567.00000000000017dc/head//3 66d93843/rbd_data.109b6b8b4567.00000000000017dc/head//3(173'995 client.4292.0:51116 wrlock_by=unknown.0.0:0 dirty s 4194304 uv995)
2014-06-12 14:51:46.996252 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f1c73843/rbd_data.109c6b8b4567.0000000000001b57/head//3 f1c73843/rbd_data.109c6b8b4567.0000000000001b57/head//3(173'1111 client.4293.0:58335 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1111)
2014-06-12 14:51:46.996265 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9fb73843/rbd_data.10a36b8b4567.0000000000002c93/head//3 9fb73843/rbd_data.10a36b8b4567.0000000000002c93/head//3(173'1467 client.4300.0:87733 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1467)
2014-06-12 14:51:46.996278 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  b8b63843/rbd_data.10a26b8b4567.00000000000018b9/head//3 b8b63843/rbd_data.10a26b8b4567.00000000000018b9/head//3(173'1023 client.4299.0:54652 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1023)
2014-06-12 14:51:46.996291 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c7d43843/rbd_data.10ac6b8b4567.00000000000039e1/head//3 c7d43843/rbd_data.10ac6b8b4567.00000000000039e1/head//3(173'2004 client.4309.0:113179 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2004)
2014-06-12 14:51:46.996304 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c2623843/rbd_data.10b16b8b4567.0000000000002605/head//3 c2623843/rbd_data.10b16b8b4567.0000000000002605/head//3(173'1371 client.4314.0:78230 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1371)
2014-06-12 14:51:46.996316 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  16213843/rbd_data.10986b8b4567.0000000000001fd6/head//3 16213843/rbd_data.10986b8b4567.0000000000001fd6/head//3(173'1192 client.4289.0:65590 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1192)
2014-06-12 14:51:46.996329 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  fcfa2843/rbd_data.10ab6b8b4567.0000000000000c79/head//3 fcfa2843/rbd_data.10ab6b8b4567.0000000000000c79/head//3(173'2443 client.4308.0:139926 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2443)
2014-06-12 14:51:46.996342 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  a28a2843/rbd_data.10ac6b8b4567.0000000000001516/head//3 a28a2843/rbd_data.10ac6b8b4567.0000000000001516/head//3(173'860 client.4309.0:46932 wrlock_by=unknown.0.0:0 dirty s 4194304 uv860)
2014-06-12 14:51:46.996355 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  16b92843/rbd_data.10b36b8b4567.0000000000002d47/head//3 16b92843/rbd_data.10b36b8b4567.0000000000002d47/head//3(173'1545 client.4316.0:92088 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1545)
2014-06-12 14:51:46.996368 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  eda82843/rbd_data.10a86b8b4567.00000000000014e3/head//3 eda82843/rbd_data.10a86b8b4567.00000000000014e3/head//3(173'830 client.4305.0:46639 wrlock_by=unknown.0.0:0 dirty s 4194304 uv830)
2014-06-12 14:51:46.996381 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  63652843/rbd_data.10a06b8b4567.000000000000143c/head//3 63652843/rbd_data.10a06b8b4567.000000000000143c/head//3(173'736 client.4297.0:45254 wrlock_by=unknown.0.0:0 dirty s 4194304 uv736)
2014-06-12 14:51:46.996393 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  a9232843/rbd_data.10b36b8b4567.00000000000007db/head//3 a9232843/rbd_data.10b36b8b4567.00000000000007db/head//3(173'2273 client.4316.0:131619 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2273)
2014-06-12 14:51:46.996410 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  ead02843/rbd_data.10aa6b8b4567.0000000000000f41/head//3 ead02843/rbd_data.10aa6b8b4567.0000000000000f41/head//3(173'2529 client.4307.0:145531 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2529)
2014-06-12 14:51:46.996423 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  8c0f1843/rbd_data.10a06b8b4567.000000000000363d/head//3 8c0f1843/rbd_data.10a06b8b4567.000000000000363d/head//3(173'1809 client.4297.0:106421 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1809)
2014-06-12 14:51:46.996436 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c50f1843/rbd_data.10ae6b8b4567.0000000000001465/head//3 c50f1843/rbd_data.10ae6b8b4567.0000000000001465/head//3(173'782 client.4311.0:45851 wrlock_by=unknown.0.0:0 dirty s 4194304 uv782)
2014-06-12 14:51:46.996449 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  67ee1843/rbd_data.10a56b8b4567.0000000000000f58/head//3 67ee1843/rbd_data.10a56b8b4567.0000000000000f58/head//3(173'2519 client.4302.0:144626 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2519)
2014-06-12 14:51:46.996462 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  752d1843/rbd_data.10946b8b4567.0000000000001b04/head//3 752d1843/rbd_data.10946b8b4567.0000000000001b04/head//3(173'969 client.4285.0:57256 wrlock_by=unknown.0.0:0 dirty s 4194304 uv969)
2014-06-12 14:51:46.996474 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  93ab1843/rbd_data.10906b8b4567.0000000000000c42/head//3 93ab1843/rbd_data.10906b8b4567.0000000000000c42/head//3(173'2398 client.4281.0:136264 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2398)
2014-06-12 14:51:46.996487 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c4681843/rbd_data.10916b8b4567.0000000000000e2d/head//3 c4681843/rbd_data.10916b8b4567.0000000000000e2d/head//3(173'2467 client.4282.0:141312 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2467)
2014-06-12 14:51:46.996499 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f9361843/rbd_data.10986b8b4567.0000000000001648/head//3 f9361843/rbd_data.10986b8b4567.0000000000001648/head//3(173'878 client.4289.0:48704 wrlock_by=unknown.0.0:0 dirty s 4194304 uv878)
2014-06-12 14:51:46.996511 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  54e51843/rbd_data.109c6b8b4567.000000000000151d/head//3 54e51843/rbd_data.109c6b8b4567.000000000000151d/head//3(173'850 client.4293.0:47111 wrlock_by=unknown.0.0:0 dirty s 4194304 uv850)
2014-06-12 14:51:46.996524 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  86641843/rbd_data.10b46b8b4567.000000000000041c/head//3 86641843/rbd_data.10b46b8b4567.000000000000041c/head//3(173'2149 client.4317.0:123218 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2149)
2014-06-12 14:51:46.996534 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:46.996544 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:46.996553 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:46.996555 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:46.996557 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:46.996590 7f9d60bd2700 10 collection_list_partial 3.43_head start:89f34843//0//-1 is_max:0
2014-06-12 14:51:46.996808 7f9d60bd2700 20 objects: [62464843/rbd_data.10906b8b4567.00000000000031ac/head//3,15d64843/rbd_data.10ae6b8b4567.000000000000288b/head//3,11205843/rbd_data.10af6b8b4567.00000000000019fb/head//3,26c35843/rbd_data.10b16b8b4567.0000000000003558/head//3,7045843/rbd_data.10a06b8b4567.000000000000162c/head//3,fbcb5843/rbd_data.10976b8b4567.0000000000002613/head//3,a1de5843/rbd_data.10a56b8b4567.00000000000030aa/head//3,bbdf5843/rbd_data.10af6b8b4567.0000000000002625/head//3,69406843/rbd_data.10ab6b8b4567.000000000000317a/head//3,af216843/rbd_data.10966b8b4567.0000000000001e15/head//3,6cf16843/rbd_data.10b36b8b4567.0000000000000e06/head//3,d7926843/rbd_data.10ae6b8b4567.0000000000000042/head//3,7a136843/rbd_data.10a26b8b4567.00000000000000fe/head//3,950a6843/rbd_data.10956b8b4567.0000000000003176/head//3,c41b6843/rbd_data.10926b8b4567.00000000000026b1/head//3,2deb6843/rbd_data.10b46b8b4567.0000000000002cb1/head//3,9c2d6843/rbd_data.10926b8b4567.0000000000002aed/head//3,864f6843/rbd_data.10956b8b4567.0000000000001275/head//3,7d807843/rbd_data.10ae6b8b4567.0000000000000448/head//3,72517843/rbd_data.108e6b8b4567.00000000000033c4/head//3,60a17843/rbd_data.108f6b8b4567.0000000000001793/head//3,ce227843/rbd_data.10b16b8b4567.0000000000002ce2/head//3,40827843/rbd_data.10926b8b4567.0000000000003ba0/head//3,64b27843/rbd_data.10ac6b8b4567.0000000000003668/head//3,14147843/rbd_data.10976b8b4567.00000000000024b0/head//3]
2014-06-12 14:51:46.997259 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  requesting scrubmap from osd.24
2014-06-12 14:51:46.997281 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:46.997297 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:46.997305 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:46.997311 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:46.997317 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:46.997323 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_map_chunk [89f34843//0//-1,47957843//0//-1)
2014-06-12 14:51:46.997330 7f9d60bd2700 10 collection_list_partial 3.43_head start:89f34843//0//-1 is_max:0
2014-06-12 14:51:46.997841 7f9d60bd2700 20 objects: [62464843/rbd_data.10906b8b4567.00000000000031ac/head//3,15d64843/rbd_data.10ae6b8b4567.000000000000288b/head//3,11205843/rbd_data.10af6b8b4567.00000000000019fb/head//3,26c35843/rbd_data.10b16b8b4567.0000000000003558/head//3,7045843/rbd_data.10a06b8b4567.000000000000162c/head//3,fbcb5843/rbd_data.10976b8b4567.0000000000002613/head//3,a1de5843/rbd_data.10a56b8b4567.00000000000030aa/head//3,bbdf5843/rbd_data.10af6b8b4567.0000000000002625/head//3,69406843/rbd_data.10ab6b8b4567.000000000000317a/head//3,af216843/rbd_data.10966b8b4567.0000000000001e15/head//3,6cf16843/rbd_data.10b36b8b4567.0000000000000e06/head//3,d7926843/rbd_data.10ae6b8b4567.0000000000000042/head//3,7a136843/rbd_data.10a26b8b4567.00000000000000fe/head//3,950a6843/rbd_data.10956b8b4567.0000000000003176/head//3,c41b6843/rbd_data.10926b8b4567.00000000000026b1/head//3,2deb6843/rbd_data.10b46b8b4567.0000000000002cb1/head//3,9c2d6843/rbd_data.10926b8b4567.0000000000002aed/head//3,864f6843/rbd_data.10956b8b4567.0000000000001275/head//3,7d807843/rbd_data.10ae6b8b4567.0000000000000448/head//3,72517843/rbd_data.108e6b8b4567.00000000000033c4/head//3,60a17843/rbd_data.108f6b8b4567.0000000000001793/head//3,ce227843/rbd_data.10b16b8b4567.0000000000002ce2/head//3,40827843/rbd_data.10926b8b4567.0000000000003ba0/head//3,64b27843/rbd_data.10ac6b8b4567.0000000000003668/head//3,14147843/rbd_data.10976b8b4567.00000000000024b0/head//3,47957843/rbd_data.10a06b8b4567.000000000000301f/head//3,c6277843/rbd_data.109e6b8b4567.00000000000030b7/head//3,d0697843/rbd_data.10956b8b4567.0000000000000ac8/head//3,d57c7843/rbd_data.10a76b8b4567.0000000000001b13/head//3,29fc7843/rbd_data.10a96b8b4567.00000000000026bd/head//3,1b5e7843/rbd_data.10926b8b4567.00000000000033dc/head//3,db6f7843/rbd_data.10b06b8b4567.0000000000000de6/head//3,a98f7843/rbd_data.10a36b8b4567.0000000000001128/head//3,11438843/rbd_data.108d6b8b4567.0000000000002e57/head//3,d9558843/rbd_data.10b26b8b4567.0000000000002817/head//3,cfd58843/rbd_data.10946b8b4567.0000000000001659/head//3,1fea8843/rbd_data.109e6b8b4567.0000000000002ba7/head//3,cad09843/rbd_data.109b6b8b4567.00000000000034ac/head//3,28b49843/rbd_data.108e6b8b4567.00000000000035bc/head//3,f5659843/rbd_data.10a76b8b4567.0000000000000a7b/head//3,d0079843/rbd_data.109f6b8b4567.00000000000034f6/head//3,3f879843/rbd_data.10b26b8b4567.0000000000003463/head//3,b7979843/rbd_data.10a26b8b4567.00000000000002ba/head//3,59289843/rbd_data.10ad6b8b4567.00000000000021f2/head//3,a8489843/rbd_data.10a86b8b4567.0000000000000afc/head//3,25099843/rbd_data.10b46b8b4567.00000000000006da/head//3,3c7a9843/rbd_data.109c6b8b4567.0000000000000c85/head//3,9c7b9843/rbd_data.108d6b8b4567.0000000000003bcd/head//3,9ceb9843/rbd_data.10a16b8b4567.00000000000014e8/head//3,af0c9843/rbd_data.10b26b8b4567.0000000000001505/head//3,222c9843/rbd_data.108d6b8b4567.00000000000005b6/head//3,645c9843/rbd_data.10986b8b4567.0000000000001dfb/head//3,4c7c9843/rbd_data.10a96b8b4567.000000000000256a/head//3,9f1d9843/rbd_data.10a26b8b4567.00000000000036d3/head//3,a4d9843/rbd_data.10a96b8b4567.000000000000273b/head//3,8080a843/rbd_data.10a86b8b4567.000000000000182e/head//3,c592a843/rbd_data.10a86b8b4567.0000000000002113/head//3,7d34a843/rbd_data.109b6b8b4567.0000000000002b0c/head//3,3fe6a843/rbd_data.10a76b8b4567.0000000000000cf8/head//3,a16ba843/rbd_data.109a6b8b4567.0000000000000eb9/head//3,956ba843/rbd_data.10b36b8b4567.0000000000000829/head//3,d3cca843/rbd_data.10966b8b4567.00000000000028c3/head//3,dfea843/rbd_data.10a06b8b4567.0000000000001ce8/head//3,e49fa843/rbd_data.10a86b8b4567.00000000000007fe/head//3]
2014-06-12 14:51:46.997917 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:46.997933 7f9d60bd2700 10 stat 3.43_head/62464843/rbd_data.10906b8b4567.00000000000031ac/head//3
2014-06-12 14:51:46.998049 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 62464843/rbd_data.10906b8b4567.00000000000031ac/head//3
2014-06-12 14:51:47.018154 7f9d60bd2700 10 getattrs 3.43_head/62464843/rbd_data.10906b8b4567.00000000000031ac/head//3 = 0
2014-06-12 14:51:47.018179 7f9d60bd2700 10 stat 3.43_head/15d64843/rbd_data.10ae6b8b4567.000000000000288b/head//3
2014-06-12 14:51:47.018367 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 15d64843/rbd_data.10ae6b8b4567.000000000000288b/head//3
2014-06-12 14:51:47.039461 7f9d60bd2700 10 getattrs 3.43_head/15d64843/rbd_data.10ae6b8b4567.000000000000288b/head//3 = 0
2014-06-12 14:51:47.039472 7f9d60bd2700 10 stat 3.43_head/11205843/rbd_data.10af6b8b4567.00000000000019fb/head//3
2014-06-12 14:51:47.039581 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 11205843/rbd_data.10af6b8b4567.00000000000019fb/head//3
2014-06-12 14:51:47.060189 7f9d60bd2700 10 getattrs 3.43_head/11205843/rbd_data.10af6b8b4567.00000000000019fb/head//3 = 0
2014-06-12 14:51:47.060196 7f9d60bd2700 10 stat 3.43_head/26c35843/rbd_data.10b16b8b4567.0000000000003558/head//3
2014-06-12 14:51:47.060299 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 26c35843/rbd_data.10b16b8b4567.0000000000003558/head//3
2014-06-12 14:51:47.060407 7f9d60bd2700 10 getattrs 3.43_head/26c35843/rbd_data.10b16b8b4567.0000000000003558/head//3 = 0
2014-06-12 14:51:47.060413 7f9d60bd2700 10 stat 3.43_head/7045843/rbd_data.10a06b8b4567.000000000000162c/head//3
2014-06-12 14:51:47.060508 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 7045843/rbd_data.10a06b8b4567.000000000000162c/head//3
2014-06-12 14:51:47.082141 7f9d60bd2700 10 getattrs 3.43_head/7045843/rbd_data.10a06b8b4567.000000000000162c/head//3 = 0
2014-06-12 14:51:47.082148 7f9d60bd2700 10 stat 3.43_head/fbcb5843/rbd_data.10976b8b4567.0000000000002613/head//3
2014-06-12 14:51:47.082245 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid fbcb5843/rbd_data.10976b8b4567.0000000000002613/head//3
2014-06-12 14:51:47.112152 7f9d60bd2700 10 getattrs 3.43_head/fbcb5843/rbd_data.10976b8b4567.0000000000002613/head//3 = 0
2014-06-12 14:51:47.112159 7f9d60bd2700 10 stat 3.43_head/a1de5843/rbd_data.10a56b8b4567.00000000000030aa/head//3
2014-06-12 14:51:47.112252 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid a1de5843/rbd_data.10a56b8b4567.00000000000030aa/head//3
2014-06-12 14:51:47.121779 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:47.130088 7f9d60bd2700 10 getattrs 3.43_head/a1de5843/rbd_data.10a56b8b4567.00000000000030aa/head//3 = 0
2014-06-12 14:51:47.130095 7f9d60bd2700 10 stat 3.43_head/bbdf5843/rbd_data.10af6b8b4567.0000000000002625/head//3
2014-06-12 14:51:47.130184 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid bbdf5843/rbd_data.10af6b8b4567.0000000000002625/head//3
2014-06-12 14:51:47.133341 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:47.142756 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:47.148026 7f9d60bd2700 10 getattrs 3.43_head/bbdf5843/rbd_data.10af6b8b4567.0000000000002625/head//3 = 0
2014-06-12 14:51:47.148036 7f9d60bd2700 10 stat 3.43_head/69406843/rbd_data.10ab6b8b4567.000000000000317a/head//3
2014-06-12 14:51:47.148130 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 69406843/rbd_data.10ab6b8b4567.000000000000317a/head//3
2014-06-12 14:51:47.152193 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:47.165687 7f9d60bd2700 10 getattrs 3.43_head/69406843/rbd_data.10ab6b8b4567.000000000000317a/head//3 = 0
2014-06-12 14:51:47.165708 7f9d60bd2700 10 stat 3.43_head/af216843/rbd_data.10966b8b4567.0000000000001e15/head//3
2014-06-12 14:51:47.165821 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid af216843/rbd_data.10966b8b4567.0000000000001e15/head//3
2014-06-12 14:51:47.179342 7f9d60bd2700 10 getattrs 3.43_head/af216843/rbd_data.10966b8b4567.0000000000001e15/head//3 = 0
2014-06-12 14:51:47.179349 7f9d60bd2700 10 stat 3.43_head/6cf16843/rbd_data.10b36b8b4567.0000000000000e06/head//3
2014-06-12 14:51:47.179442 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 6cf16843/rbd_data.10b36b8b4567.0000000000000e06/head//3
2014-06-12 14:51:47.180660 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:47.236249 7f9d60bd2700 10 getattrs 3.43_head/6cf16843/rbd_data.10b36b8b4567.0000000000000e06/head//3 = 0
2014-06-12 14:51:47.236266 7f9d60bd2700 10 stat 3.43_head/d7926843/rbd_data.10ae6b8b4567.0000000000000042/head//3
2014-06-12 14:51:47.236378 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d7926843/rbd_data.10ae6b8b4567.0000000000000042/head//3
2014-06-12 14:51:47.277630 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:47.279400 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:47.279425 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:47.279430 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:47.279432 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:47.279472 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:47.294382 7f9d60bd2700 10 getattrs 3.43_head/d7926843/rbd_data.10ae6b8b4567.0000000000000042/head//3 = 0
2014-06-12 14:51:47.294402 7f9d60bd2700 10 stat 3.43_head/7a136843/rbd_data.10a26b8b4567.00000000000000fe/head//3
2014-06-12 14:51:47.294535 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 7a136843/rbd_data.10a26b8b4567.00000000000000fe/head//3
2014-06-12 14:51:47.298829 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:47.316925 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:47.331125 7f9d60bd2700 10 getattrs 3.43_head/7a136843/rbd_data.10a26b8b4567.00000000000000fe/head//3 = 0
2014-06-12 14:51:47.331137 7f9d60bd2700 10 stat 3.43_head/950a6843/rbd_data.10956b8b4567.0000000000003176/head//3
2014-06-12 14:51:47.331234 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 950a6843/rbd_data.10956b8b4567.0000000000003176/head//3
2014-06-12 14:51:47.331916 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:47.331943 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:47.341977 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:47.375281 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:47.457184 7f9d60bd2700 10 getattrs 3.43_head/950a6843/rbd_data.10956b8b4567.0000000000003176/head//3 = 0
2014-06-12 14:51:47.457204 7f9d60bd2700 10 stat 3.43_head/c41b6843/rbd_data.10926b8b4567.00000000000026b1/head//3
2014-06-12 14:51:47.457383 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c41b6843/rbd_data.10926b8b4567.00000000000026b1/head//3
2014-06-12 14:51:47.499065 7f9d60bd2700 10 getattrs 3.43_head/c41b6843/rbd_data.10926b8b4567.00000000000026b1/head//3 = 0
2014-06-12 14:51:47.499079 7f9d60bd2700 10 stat 3.43_head/2deb6843/rbd_data.10b46b8b4567.0000000000002cb1/head//3
2014-06-12 14:51:47.499245 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 2deb6843/rbd_data.10b46b8b4567.0000000000002cb1/head//3
2014-06-12 14:51:47.513781 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:47.518835 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:47.529173 7f9d60bd2700 10 getattrs 3.43_head/2deb6843/rbd_data.10b46b8b4567.0000000000002cb1/head//3 = 0
2014-06-12 14:51:47.529185 7f9d60bd2700 10 stat 3.43_head/9c2d6843/rbd_data.10926b8b4567.0000000000002aed/head//3
2014-06-12 14:51:47.529280 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9c2d6843/rbd_data.10926b8b4567.0000000000002aed/head//3
2014-06-12 14:51:47.543029 7f9d5a16f700  5 -- op tracker -- , seq: 748602, time: 2014-06-12 14:51:47.542905, event: header_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:47.543048 7f9d5a16f700  5 -- op tracker -- , seq: 748602, time: 2014-06-12 14:51:47.542910, event: throttled, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:47.543059 7f9d5a16f700  5 -- op tracker -- , seq: 748602, time: 2014-06-12 14:51:47.542978, event: all_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:47.543092 7f9d5a16f700  5 -- op tracker -- , seq: 748602, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:47.543099 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:47.543116 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:51:47.543124 7f9d5a16f700 15 osd.2 173 enqueue_op 0x876ef00 prio 127 cost 10717 latency 0.000218 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:47.543172 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:47.547470 7f9d60bd2700 10 getattrs 3.43_head/9c2d6843/rbd_data.10926b8b4567.0000000000002aed/head//3 = 0
2014-06-12 14:51:47.547481 7f9d60bd2700 10 stat 3.43_head/864f6843/rbd_data.10956b8b4567.0000000000001275/head//3
2014-06-12 14:51:47.547585 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 864f6843/rbd_data.10956b8b4567.0000000000001275/head//3
2014-06-12 14:51:47.547932 7f9d60bd2700 10 getattrs 3.43_head/864f6843/rbd_data.10956b8b4567.0000000000001275/head//3 = 0
2014-06-12 14:51:47.547939 7f9d60bd2700 10 stat 3.43_head/7d807843/rbd_data.10ae6b8b4567.0000000000000448/head//3
2014-06-12 14:51:47.548029 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 7d807843/rbd_data.10ae6b8b4567.0000000000000448/head//3
2014-06-12 14:51:47.606825 7f9d60bd2700 10 getattrs 3.43_head/7d807843/rbd_data.10ae6b8b4567.0000000000000448/head//3 = 0
2014-06-12 14:51:47.606845 7f9d60bd2700 10 stat 3.43_head/72517843/rbd_data.108e6b8b4567.00000000000033c4/head//3
2014-06-12 14:51:47.606973 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 72517843/rbd_data.108e6b8b4567.00000000000033c4/head//3
2014-06-12 14:51:47.641581 7f9d60bd2700 10 getattrs 3.43_head/72517843/rbd_data.108e6b8b4567.00000000000033c4/head//3 = 0
2014-06-12 14:51:47.641592 7f9d60bd2700 10 stat 3.43_head/60a17843/rbd_data.108f6b8b4567.0000000000001793/head//3
2014-06-12 14:51:47.641736 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 60a17843/rbd_data.108f6b8b4567.0000000000001793/head//3
2014-06-12 14:51:47.661103 7f9d60bd2700 10 getattrs 3.43_head/60a17843/rbd_data.108f6b8b4567.0000000000001793/head//3 = 0
2014-06-12 14:51:47.661114 7f9d60bd2700 10 stat 3.43_head/ce227843/rbd_data.10b16b8b4567.0000000000002ce2/head//3
2014-06-12 14:51:47.661239 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid ce227843/rbd_data.10b16b8b4567.0000000000002ce2/head//3
2014-06-12 14:51:47.673174 7f9d60bd2700 10 getattrs 3.43_head/ce227843/rbd_data.10b16b8b4567.0000000000002ce2/head//3 = 0
2014-06-12 14:51:47.673187 7f9d60bd2700 10 stat 3.43_head/40827843/rbd_data.10926b8b4567.0000000000003ba0/head//3
2014-06-12 14:51:47.673314 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 40827843/rbd_data.10926b8b4567.0000000000003ba0/head//3
2014-06-12 14:51:47.686651 7f9d60bd2700 10 getattrs 3.43_head/40827843/rbd_data.10926b8b4567.0000000000003ba0/head//3 = 0
2014-06-12 14:51:47.686662 7f9d60bd2700 10 stat 3.43_head/64b27843/rbd_data.10ac6b8b4567.0000000000003668/head//3
2014-06-12 14:51:47.686786 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 64b27843/rbd_data.10ac6b8b4567.0000000000003668/head//3
2014-06-12 14:51:47.702608 7f9d60bd2700 10 getattrs 3.43_head/64b27843/rbd_data.10ac6b8b4567.0000000000003668/head//3 = 0
2014-06-12 14:51:47.702622 7f9d60bd2700 10 stat 3.43_head/14147843/rbd_data.10976b8b4567.00000000000024b0/head//3
2014-06-12 14:51:47.702747 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 14147843/rbd_data.10976b8b4567.00000000000024b0/head//3
2014-06-12 14:51:47.732694 7f9d60bd2700 10 getattrs 3.43_head/14147843/rbd_data.10976b8b4567.00000000000024b0/head//3 = 0
2014-06-12 14:51:47.732713 7f9d60bd2700 10 collection_getattrs 3.43_head
2014-06-12 14:51:47.732777 7f9d60bd2700 10 collection_getattrs 3.43_head = 0
2014-06-12 14:51:47.732782 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:47.732815 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:47.732823 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:47.732835 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:47.732839 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:47.732879 7f9d643d9700 10 osd.2 173 dequeue_op 0x876ef00 prio 127 cost 10717 latency 0.189973 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]
2014-06-12 14:51:47.732931 7f9d643d9700  5 -- op tracker -- , seq: 748602, time: 2014-06-12 14:51:47.732923, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:47.732954 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] handle_message: 0x876ef00
2014-06-12 14:51:47.732971 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:47.732991 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:47.733010 7f9d643d9700  5 -- op tracker -- , seq: 748602, time: 2014-06-12 14:51:47.733005, event: started, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:47.733018 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  got 24 scrub map
2014-06-12 14:51:47.733128 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] map version is 173'2566
2014-06-12 14:51:47.733148 7f9d643d9700 10 osd.2 173 dequeue_op 0x876ef00 finish
2014-06-12 14:51:47.733160 7f9d643d9700  5 -- op tracker -- , seq: 748602, time: 2014-06-12 14:51:47.733154, event: done, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:47.733187 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:47.733199 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:47.733205 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:47.733200 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:47.733229 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:47.733238 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:47.733269 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:47.733282 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:47.733291 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub replica 24 has 25 items
2014-06-12 14:51:47.733318 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 62464843/rbd_data.10906b8b4567.00000000000031ac/head//3, auth == maps.end()
2014-06-12 14:51:47.733352 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 62464843/rbd_data.10906b8b4567.00000000000031ac/head//3
2014-06-12 14:51:47.733370 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 15d64843/rbd_data.10ae6b8b4567.000000000000288b/head//3, auth == maps.end()
2014-06-12 14:51:47.733384 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 15d64843/rbd_data.10ae6b8b4567.000000000000288b/head//3
2014-06-12 14:51:47.733397 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 11205843/rbd_data.10af6b8b4567.00000000000019fb/head//3, auth == maps.end()
2014-06-12 14:51:47.733420 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 11205843/rbd_data.10af6b8b4567.00000000000019fb/head//3
2014-06-12 14:51:47.733436 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 26c35843/rbd_data.10b16b8b4567.0000000000003558/head//3, auth == maps.end()
2014-06-12 14:51:47.733449 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 26c35843/rbd_data.10b16b8b4567.0000000000003558/head//3
2014-06-12 14:51:47.733463 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7045843/rbd_data.10a06b8b4567.000000000000162c/head//3, auth == maps.end()
2014-06-12 14:51:47.733476 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 7045843/rbd_data.10a06b8b4567.000000000000162c/head//3
2014-06-12 14:51:47.733490 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fbcb5843/rbd_data.10976b8b4567.0000000000002613/head//3, auth == maps.end()
2014-06-12 14:51:47.733503 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj fbcb5843/rbd_data.10976b8b4567.0000000000002613/head//3
2014-06-12 14:51:47.733517 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a1de5843/rbd_data.10a56b8b4567.00000000000030aa/head//3, auth == maps.end()
2014-06-12 14:51:47.733530 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj a1de5843/rbd_data.10a56b8b4567.00000000000030aa/head//3
2014-06-12 14:51:47.733543 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bbdf5843/rbd_data.10af6b8b4567.0000000000002625/head//3, auth == maps.end()
2014-06-12 14:51:47.733556 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj bbdf5843/rbd_data.10af6b8b4567.0000000000002625/head//3
2014-06-12 14:51:47.733569 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 69406843/rbd_data.10ab6b8b4567.000000000000317a/head//3, auth == maps.end()
2014-06-12 14:51:47.733581 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 69406843/rbd_data.10ab6b8b4567.000000000000317a/head//3
2014-06-12 14:51:47.733595 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj af216843/rbd_data.10966b8b4567.0000000000001e15/head//3, auth == maps.end()
2014-06-12 14:51:47.733608 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj af216843/rbd_data.10966b8b4567.0000000000001e15/head//3
2014-06-12 14:51:47.733634 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6cf16843/rbd_data.10b36b8b4567.0000000000000e06/head//3, auth == maps.end()
2014-06-12 14:51:47.733648 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 6cf16843/rbd_data.10b36b8b4567.0000000000000e06/head//3
2014-06-12 14:51:47.733662 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d7926843/rbd_data.10ae6b8b4567.0000000000000042/head//3, auth == maps.end()
2014-06-12 14:51:47.733674 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d7926843/rbd_data.10ae6b8b4567.0000000000000042/head//3
2014-06-12 14:51:47.733685 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7a136843/rbd_data.10a26b8b4567.00000000000000fe/head//3, auth == maps.end()
2014-06-12 14:51:47.733696 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 7a136843/rbd_data.10a26b8b4567.00000000000000fe/head//3
2014-06-12 14:51:47.733708 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 950a6843/rbd_data.10956b8b4567.0000000000003176/head//3, auth == maps.end()
2014-06-12 14:51:47.733719 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 950a6843/rbd_data.10956b8b4567.0000000000003176/head//3
2014-06-12 14:51:47.733730 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c41b6843/rbd_data.10926b8b4567.00000000000026b1/head//3, auth == maps.end()
2014-06-12 14:51:47.733741 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c41b6843/rbd_data.10926b8b4567.00000000000026b1/head//3
2014-06-12 14:51:47.733752 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2deb6843/rbd_data.10b46b8b4567.0000000000002cb1/head//3, auth == maps.end()
2014-06-12 14:51:47.733765 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 2deb6843/rbd_data.10b46b8b4567.0000000000002cb1/head//3
2014-06-12 14:51:47.733779 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9c2d6843/rbd_data.10926b8b4567.0000000000002aed/head//3, auth == maps.end()
2014-06-12 14:51:47.733796 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9c2d6843/rbd_data.10926b8b4567.0000000000002aed/head//3
2014-06-12 14:51:47.733810 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 864f6843/rbd_data.10956b8b4567.0000000000001275/head//3, auth == maps.end()
2014-06-12 14:51:47.733823 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 864f6843/rbd_data.10956b8b4567.0000000000001275/head//3
2014-06-12 14:51:47.733837 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7d807843/rbd_data.10ae6b8b4567.0000000000000448/head//3, auth == maps.end()
2014-06-12 14:51:47.733850 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 7d807843/rbd_data.10ae6b8b4567.0000000000000448/head//3
2014-06-12 14:51:47.733863 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 72517843/rbd_data.108e6b8b4567.00000000000033c4/head//3, auth == maps.end()
2014-06-12 14:51:47.733875 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 72517843/rbd_data.108e6b8b4567.00000000000033c4/head//3
2014-06-12 14:51:47.733888 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 60a17843/rbd_data.108f6b8b4567.0000000000001793/head//3, auth == maps.end()
2014-06-12 14:51:47.733901 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 60a17843/rbd_data.108f6b8b4567.0000000000001793/head//3
2014-06-12 14:51:47.733913 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ce227843/rbd_data.10b16b8b4567.0000000000002ce2/head//3, auth == maps.end()
2014-06-12 14:51:47.733925 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj ce227843/rbd_data.10b16b8b4567.0000000000002ce2/head//3
2014-06-12 14:51:47.733939 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 40827843/rbd_data.10926b8b4567.0000000000003ba0/head//3, auth == maps.end()
2014-06-12 14:51:47.733951 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 40827843/rbd_data.10926b8b4567.0000000000003ba0/head//3
2014-06-12 14:51:47.733964 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 64b27843/rbd_data.10ac6b8b4567.0000000000003668/head//3, auth == maps.end()
2014-06-12 14:51:47.733975 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 64b27843/rbd_data.10ac6b8b4567.0000000000003668/head//3
2014-06-12 14:51:47.733987 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 14147843/rbd_data.10976b8b4567.00000000000024b0/head//3, auth == maps.end()
2014-06-12 14:51:47.733998 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 14147843/rbd_data.10976b8b4567.00000000000024b0/head//3
2014-06-12 14:51:47.734011 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] 
2014-06-12 14:51:47.734019 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub
2014-06-12 14:51:47.734035 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  14147843/rbd_data.10976b8b4567.00000000000024b0/head//3 14147843/rbd_data.10976b8b4567.00000000000024b0/head//3(173'1393 client.4288.0:73791 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1393)
2014-06-12 14:51:47.734053 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  64b27843/rbd_data.10ac6b8b4567.0000000000003668/head//3 64b27843/rbd_data.10ac6b8b4567.0000000000003668/head//3(173'1933 client.4309.0:107081 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1933)
2014-06-12 14:51:47.734070 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  40827843/rbd_data.10926b8b4567.0000000000003ba0/head//3 40827843/rbd_data.10926b8b4567.0000000000003ba0/head//3(173'2071 client.4283.0:116803 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2071)
2014-06-12 14:51:47.734087 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  ce227843/rbd_data.10b16b8b4567.0000000000002ce2/head//3 ce227843/rbd_data.10b16b8b4567.0000000000002ce2/head//3(173'1629 client.4314.0:91044 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1629)
2014-06-12 14:51:47.734103 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  60a17843/rbd_data.108f6b8b4567.0000000000001793/head//3 60a17843/rbd_data.108f6b8b4567.0000000000001793/head//3(173'947 client.4280.0:51705 wrlock_by=unknown.0.0:0 dirty s 4194304 uv947)
2014-06-12 14:51:47.734119 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  72517843/rbd_data.108e6b8b4567.00000000000033c4/head//3 72517843/rbd_data.108e6b8b4567.00000000000033c4/head//3(173'1763 client.4279.0:102677 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1763)
2014-06-12 14:51:47.734135 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  7d807843/rbd_data.10ae6b8b4567.0000000000000448/head//3 7d807843/rbd_data.10ae6b8b4567.0000000000000448/head//3(173'2173 client.4311.0:121340 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2173)
2014-06-12 14:51:47.734151 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  864f6843/rbd_data.10956b8b4567.0000000000001275/head//3 864f6843/rbd_data.10956b8b4567.0000000000001275/head//3(173'749 client.4286.0:42458 wrlock_by=unknown.0.0:0 dirty s 4194304 uv749)
2014-06-12 14:51:47.734167 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9c2d6843/rbd_data.10926b8b4567.0000000000002aed/head//3 9c2d6843/rbd_data.10926b8b4567.0000000000002aed/head//3(173'1503 client.4283.0:86263 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1503)
2014-06-12 14:51:47.734183 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  2deb6843/rbd_data.10b46b8b4567.0000000000002cb1/head//3 2deb6843/rbd_data.10b46b8b4567.0000000000002cb1/head//3(173'1676 client.4317.0:90177 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1676)
2014-06-12 14:51:47.734199 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c41b6843/rbd_data.10926b8b4567.00000000000026b1/head//3 c41b6843/rbd_data.10926b8b4567.00000000000026b1/head//3(173'1336 client.4283.0:78400 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1336)
2014-06-12 14:51:47.734215 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  950a6843/rbd_data.10956b8b4567.0000000000003176/head//3 950a6843/rbd_data.10956b8b4567.0000000000003176/head//3(173'1735 client.4286.0:98920 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1735)
2014-06-12 14:51:47.734231 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  7a136843/rbd_data.10a26b8b4567.00000000000000fe/head//3 7a136843/rbd_data.10a26b8b4567.00000000000000fe/head//3(173'2090 client.4299.0:120849 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2090)
2014-06-12 14:51:47.734247 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d7926843/rbd_data.10ae6b8b4567.0000000000000042/head//3 d7926843/rbd_data.10ae6b8b4567.0000000000000042/head//3(173'2080 client.4311.0:115386 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2080)
2014-06-12 14:51:47.734263 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  6cf16843/rbd_data.10b36b8b4567.0000000000000e06/head//3 6cf16843/rbd_data.10b36b8b4567.0000000000000e06/head//3(173'2426 client.4316.0:141628 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2426)
2014-06-12 14:51:47.734284 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  af216843/rbd_data.10966b8b4567.0000000000001e15/head//3 af216843/rbd_data.10966b8b4567.0000000000001e15/head//3(173'1094 client.4287.0:61684 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1094)
2014-06-12 14:51:47.734300 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  69406843/rbd_data.10ab6b8b4567.000000000000317a/head//3 69406843/rbd_data.10ab6b8b4567.000000000000317a/head//3(173'1687 client.4308.0:99254 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1687)
2014-06-12 14:51:47.734316 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  bbdf5843/rbd_data.10af6b8b4567.0000000000002625/head//3 bbdf5843/rbd_data.10af6b8b4567.0000000000002625/head//3(173'1331 client.4312.0:80123 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1331)
2014-06-12 14:51:47.734332 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  a1de5843/rbd_data.10a56b8b4567.00000000000030aa/head//3 a1de5843/rbd_data.10a56b8b4567.00000000000030aa/head//3(173'1724 client.4302.0:98490 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1724)
2014-06-12 14:51:47.734348 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  fbcb5843/rbd_data.10976b8b4567.0000000000002613/head//3 fbcb5843/rbd_data.10976b8b4567.0000000000002613/head//3(173'1428 client.4288.0:76138 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1428)
2014-06-12 14:51:47.734364 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  7045843/rbd_data.10a06b8b4567.000000000000162c/head//3 7045843/rbd_data.10a06b8b4567.000000000000162c/head//3(173'839 client.4297.0:48780 wrlock_by=unknown.0.0:0 dirty s 4194304 uv839)
2014-06-12 14:51:47.734380 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  26c35843/rbd_data.10b16b8b4567.0000000000003558/head//3 26c35843/rbd_data.10b16b8b4567.0000000000003558/head//3(173'1951 client.4314.0:107284 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1951)
2014-06-12 14:51:47.734396 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  11205843/rbd_data.10af6b8b4567.00000000000019fb/head//3 11205843/rbd_data.10af6b8b4567.00000000000019fb/head//3(173'1068 client.4312.0:59171 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1068)
2014-06-12 14:51:47.734412 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  15d64843/rbd_data.10ae6b8b4567.000000000000288b/head//3 15d64843/rbd_data.10ae6b8b4567.000000000000288b/head//3(173'1297 client.4311.0:80215 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1297)
2014-06-12 14:51:47.734427 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  62464843/rbd_data.10906b8b4567.00000000000031ac/head//3 62464843/rbd_data.10906b8b4567.00000000000031ac/head//3(173'1617 client.4281.0:98896 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1617)
2014-06-12 14:51:47.734440 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:47.734453 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:47.734464 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:47.734467 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:47.734469 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:47.734515 7f9d60bd2700 10 collection_list_partial 3.43_head start:47957843//0//-1 is_max:0
2014-06-12 14:51:47.734778 7f9d60bd2700 20 objects: [c6277843/rbd_data.109e6b8b4567.00000000000030b7/head//3,d0697843/rbd_data.10956b8b4567.0000000000000ac8/head//3,d57c7843/rbd_data.10a76b8b4567.0000000000001b13/head//3,29fc7843/rbd_data.10a96b8b4567.00000000000026bd/head//3,1b5e7843/rbd_data.10926b8b4567.00000000000033dc/head//3,db6f7843/rbd_data.10b06b8b4567.0000000000000de6/head//3,a98f7843/rbd_data.10a36b8b4567.0000000000001128/head//3,11438843/rbd_data.108d6b8b4567.0000000000002e57/head//3,d9558843/rbd_data.10b26b8b4567.0000000000002817/head//3,cfd58843/rbd_data.10946b8b4567.0000000000001659/head//3,1fea8843/rbd_data.109e6b8b4567.0000000000002ba7/head//3,cad09843/rbd_data.109b6b8b4567.00000000000034ac/head//3,28b49843/rbd_data.108e6b8b4567.00000000000035bc/head//3,f5659843/rbd_data.10a76b8b4567.0000000000000a7b/head//3,d0079843/rbd_data.109f6b8b4567.00000000000034f6/head//3,3f879843/rbd_data.10b26b8b4567.0000000000003463/head//3,b7979843/rbd_data.10a26b8b4567.00000000000002ba/head//3,59289843/rbd_data.10ad6b8b4567.00000000000021f2/head//3,a8489843/rbd_data.10a86b8b4567.0000000000000afc/head//3,25099843/rbd_data.10b46b8b4567.00000000000006da/head//3,3c7a9843/rbd_data.109c6b8b4567.0000000000000c85/head//3,9c7b9843/rbd_data.108d6b8b4567.0000000000003bcd/head//3,9ceb9843/rbd_data.10a16b8b4567.00000000000014e8/head//3,af0c9843/rbd_data.10b26b8b4567.0000000000001505/head//3,222c9843/rbd_data.108d6b8b4567.00000000000005b6/head//3]
2014-06-12 14:51:47.735258 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  requesting scrubmap from osd.24
2014-06-12 14:51:47.735285 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:47.735307 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:47.735318 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:47.735326 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:47.735333 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:47.735340 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_map_chunk [47957843//0//-1,645c9843//0//-1)
2014-06-12 14:51:47.735349 7f9d60bd2700 10 collection_list_partial 3.43_head start:47957843//0//-1 is_max:0
2014-06-12 14:51:47.735933 7f9d60bd2700 20 objects: [c6277843/rbd_data.109e6b8b4567.00000000000030b7/head//3,d0697843/rbd_data.10956b8b4567.0000000000000ac8/head//3,d57c7843/rbd_data.10a76b8b4567.0000000000001b13/head//3,29fc7843/rbd_data.10a96b8b4567.00000000000026bd/head//3,1b5e7843/rbd_data.10926b8b4567.00000000000033dc/head//3,db6f7843/rbd_data.10b06b8b4567.0000000000000de6/head//3,a98f7843/rbd_data.10a36b8b4567.0000000000001128/head//3,11438843/rbd_data.108d6b8b4567.0000000000002e57/head//3,d9558843/rbd_data.10b26b8b4567.0000000000002817/head//3,cfd58843/rbd_data.10946b8b4567.0000000000001659/head//3,1fea8843/rbd_data.109e6b8b4567.0000000000002ba7/head//3,cad09843/rbd_data.109b6b8b4567.00000000000034ac/head//3,28b49843/rbd_data.108e6b8b4567.00000000000035bc/head//3,f5659843/rbd_data.10a76b8b4567.0000000000000a7b/head//3,d0079843/rbd_data.109f6b8b4567.00000000000034f6/head//3,3f879843/rbd_data.10b26b8b4567.0000000000003463/head//3,b7979843/rbd_data.10a26b8b4567.00000000000002ba/head//3,59289843/rbd_data.10ad6b8b4567.00000000000021f2/head//3,a8489843/rbd_data.10a86b8b4567.0000000000000afc/head//3,25099843/rbd_data.10b46b8b4567.00000000000006da/head//3,3c7a9843/rbd_data.109c6b8b4567.0000000000000c85/head//3,9c7b9843/rbd_data.108d6b8b4567.0000000000003bcd/head//3,9ceb9843/rbd_data.10a16b8b4567.00000000000014e8/head//3,af0c9843/rbd_data.10b26b8b4567.0000000000001505/head//3,222c9843/rbd_data.108d6b8b4567.00000000000005b6/head//3,645c9843/rbd_data.10986b8b4567.0000000000001dfb/head//3,4c7c9843/rbd_data.10a96b8b4567.000000000000256a/head//3,9f1d9843/rbd_data.10a26b8b4567.00000000000036d3/head//3,a4d9843/rbd_data.10a96b8b4567.000000000000273b/head//3,8080a843/rbd_data.10a86b8b4567.000000000000182e/head//3,c592a843/rbd_data.10a86b8b4567.0000000000002113/head//3,7d34a843/rbd_data.109b6b8b4567.0000000000002b0c/head//3,3fe6a843/rbd_data.10a76b8b4567.0000000000000cf8/head//3,a16ba843/rbd_data.109a6b8b4567.0000000000000eb9/head//3,956ba843/rbd_data.10b36b8b4567.0000000000000829/head//3,d3cca843/rbd_data.10966b8b4567.00000000000028c3/head//3,dfea843/rbd_data.10a06b8b4567.0000000000001ce8/head//3,e49fa843/rbd_data.10a86b8b4567.00000000000007fe/head//3,dea1b843/rbd_data.10926b8b4567.0000000000003993/head//3,c1b1b843/rbd_data.109c6b8b4567.0000000000001e9a/head//3,27d5b843/rbd_data.10ad6b8b4567.0000000000003857/head//3,736b843/rbd_data.10986b8b4567.00000000000004a9/head//3,9966b843/rbd_data.10936b8b4567.00000000000030a4/head//3,9c57b843/rbd_data.10a66b8b4567.00000000000007b3/head//3,c549b843/rbd_data.10a56b8b4567.0000000000002925/head//3,9b49b843/rbd_data.109e6b8b4567.00000000000038e5/head//3,60cdb843/rbd_data.10b26b8b4567.0000000000001c6c/head//3,59beb843/rbd_data.10956b8b4567.00000000000005f0/head//3,1ef1c843/rbd_data.109a6b8b4567.0000000000001abf/head//3,d303c843/rbd_data.10916b8b4567.0000000000002412/head//3,c259c843/rbd_data.109c6b8b4567.0000000000000818/head//3,b989c843/rbd_data.10946b8b4567.0000000000002e46/head//3,f6b9c843/rbd_data.109e6b8b4567.0000000000000cd6/head//3,54bbc843/rbd_data.10b36b8b4567.0000000000000710/head//3,5a3cc843/rbd_data.10ac6b8b4567.0000000000001d97/head//3,b6bcc843/rbd_data.10906b8b4567.00000000000009af/head//3,ecedc843/rbd_data.10b06b8b4567.0000000000003096/head//3,d77ec843/rbd_data.10af6b8b4567.000000000000186b/head//3,b1fc843/rbd_data.10ab6b8b4567.00000000000018f4/head//3,398fc843/rbd_data.108d6b8b4567.00000000000033d0/head//3,e213d843/rbd_data.10b06b8b4567.000000000000005e/head//3,f655d843/rbd_data.10946b8b4567.00000000000037bd/head//3,5e8d843/rbd_data.10b06b8b4567.0000000000002b59/head//3,1bd843/rbd_data.10916b8b4567.00000000000026ec/head//3]
2014-06-12 14:51:47.736023 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:47.736042 7f9d60bd2700 10 stat 3.43_head/c6277843/rbd_data.109e6b8b4567.00000000000030b7/head//3
2014-06-12 14:51:47.736186 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c6277843/rbd_data.109e6b8b4567.00000000000030b7/head//3
2014-06-12 14:51:47.753275 7f9d60bd2700 10 getattrs 3.43_head/c6277843/rbd_data.109e6b8b4567.00000000000030b7/head//3 = 0
2014-06-12 14:51:47.753285 7f9d60bd2700 10 stat 3.43_head/d0697843/rbd_data.10956b8b4567.0000000000000ac8/head//3
2014-06-12 14:51:47.753379 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d0697843/rbd_data.10956b8b4567.0000000000000ac8/head//3
2014-06-12 14:51:47.792931 7f9d60bd2700 10 getattrs 3.43_head/d0697843/rbd_data.10956b8b4567.0000000000000ac8/head//3 = 0
2014-06-12 14:51:47.792941 7f9d60bd2700 10 stat 3.43_head/d57c7843/rbd_data.10a76b8b4567.0000000000001b13/head//3
2014-06-12 14:51:47.793044 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d57c7843/rbd_data.10a76b8b4567.0000000000001b13/head//3
2014-06-12 14:51:47.807845 7f9d60bd2700 10 getattrs 3.43_head/d57c7843/rbd_data.10a76b8b4567.0000000000001b13/head//3 = 0
2014-06-12 14:51:47.807858 7f9d60bd2700 10 stat 3.43_head/29fc7843/rbd_data.10a96b8b4567.00000000000026bd/head//3
2014-06-12 14:51:47.807969 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 29fc7843/rbd_data.10a96b8b4567.00000000000026bd/head//3
2014-06-12 14:51:47.821247 7f9d60bd2700 10 getattrs 3.43_head/29fc7843/rbd_data.10a96b8b4567.00000000000026bd/head//3 = 0
2014-06-12 14:51:47.821256 7f9d60bd2700 10 stat 3.43_head/1b5e7843/rbd_data.10926b8b4567.00000000000033dc/head//3
2014-06-12 14:51:47.821353 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 1b5e7843/rbd_data.10926b8b4567.00000000000033dc/head//3
2014-06-12 14:51:47.838382 7f9d60bd2700 10 getattrs 3.43_head/1b5e7843/rbd_data.10926b8b4567.00000000000033dc/head//3 = 0
2014-06-12 14:51:47.838393 7f9d60bd2700 10 stat 3.43_head/db6f7843/rbd_data.10b06b8b4567.0000000000000de6/head//3
2014-06-12 14:51:47.838515 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid db6f7843/rbd_data.10b06b8b4567.0000000000000de6/head//3
2014-06-12 14:51:47.875141 7f9d60bd2700 10 getattrs 3.43_head/db6f7843/rbd_data.10b06b8b4567.0000000000000de6/head//3 = 0
2014-06-12 14:51:47.875152 7f9d60bd2700 10 stat 3.43_head/a98f7843/rbd_data.10a36b8b4567.0000000000001128/head//3
2014-06-12 14:51:47.875251 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid a98f7843/rbd_data.10a36b8b4567.0000000000001128/head//3
2014-06-12 14:51:47.877367 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:47.898132 7f9d60bd2700 10 getattrs 3.43_head/a98f7843/rbd_data.10a36b8b4567.0000000000001128/head//3 = 0
2014-06-12 14:51:47.898148 7f9d60bd2700 10 stat 3.43_head/11438843/rbd_data.108d6b8b4567.0000000000002e57/head//3
2014-06-12 14:51:47.898253 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 11438843/rbd_data.108d6b8b4567.0000000000002e57/head//3
2014-06-12 14:51:47.913316 7f9d60bd2700 10 getattrs 3.43_head/11438843/rbd_data.108d6b8b4567.0000000000002e57/head//3 = 0
2014-06-12 14:51:47.913327 7f9d60bd2700 10 stat 3.43_head/d9558843/rbd_data.10b26b8b4567.0000000000002817/head//3
2014-06-12 14:51:47.913434 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d9558843/rbd_data.10b26b8b4567.0000000000002817/head//3
2014-06-12 14:51:47.935919 7f9d60bd2700 10 getattrs 3.43_head/d9558843/rbd_data.10b26b8b4567.0000000000002817/head//3 = 0
2014-06-12 14:51:47.935928 7f9d60bd2700 10 stat 3.43_head/cfd58843/rbd_data.10946b8b4567.0000000000001659/head//3
2014-06-12 14:51:47.936035 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid cfd58843/rbd_data.10946b8b4567.0000000000001659/head//3
2014-06-12 14:51:47.966461 7f9d60bd2700 10 getattrs 3.43_head/cfd58843/rbd_data.10946b8b4567.0000000000001659/head//3 = 0
2014-06-12 14:51:47.966474 7f9d60bd2700 10 stat 3.43_head/1fea8843/rbd_data.109e6b8b4567.0000000000002ba7/head//3
2014-06-12 14:51:47.966585 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 1fea8843/rbd_data.109e6b8b4567.0000000000002ba7/head//3
2014-06-12 14:51:47.977395 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:47.985163 7f9d60bd2700 10 getattrs 3.43_head/1fea8843/rbd_data.109e6b8b4567.0000000000002ba7/head//3 = 0
2014-06-12 14:51:47.985172 7f9d60bd2700 10 stat 3.43_head/cad09843/rbd_data.109b6b8b4567.00000000000034ac/head//3
2014-06-12 14:51:47.985283 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid cad09843/rbd_data.109b6b8b4567.00000000000034ac/head//3
2014-06-12 14:51:47.999752 7f9d60bd2700 10 getattrs 3.43_head/cad09843/rbd_data.109b6b8b4567.00000000000034ac/head//3 = 0
2014-06-12 14:51:47.999769 7f9d60bd2700 10 stat 3.43_head/28b49843/rbd_data.108e6b8b4567.00000000000035bc/head//3
2014-06-12 14:51:47.999933 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 28b49843/rbd_data.108e6b8b4567.00000000000035bc/head//3
2014-06-12 14:51:48.002296 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:48.002722 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:48.011472 7f9d60bd2700 10 getattrs 3.43_head/28b49843/rbd_data.108e6b8b4567.00000000000035bc/head//3 = 0
2014-06-12 14:51:48.011491 7f9d60bd2700 10 stat 3.43_head/f5659843/rbd_data.10a76b8b4567.0000000000000a7b/head//3
2014-06-12 14:51:48.011659 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f5659843/rbd_data.10a76b8b4567.0000000000000a7b/head//3
2014-06-12 14:51:48.044954 7f9d60bd2700 10 getattrs 3.43_head/f5659843/rbd_data.10a76b8b4567.0000000000000a7b/head//3 = 0
2014-06-12 14:51:48.044963 7f9d60bd2700 10 stat 3.43_head/d0079843/rbd_data.109f6b8b4567.00000000000034f6/head//3
2014-06-12 14:51:48.045050 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d0079843/rbd_data.109f6b8b4567.00000000000034f6/head//3
2014-06-12 14:51:48.045123 7f9d60bd2700 10 getattrs 3.43_head/d0079843/rbd_data.109f6b8b4567.00000000000034f6/head//3 = 0
2014-06-12 14:51:48.045129 7f9d60bd2700 10 stat 3.43_head/3f879843/rbd_data.10b26b8b4567.0000000000003463/head//3
2014-06-12 14:51:48.045207 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 3f879843/rbd_data.10b26b8b4567.0000000000003463/head//3
2014-06-12 14:51:48.045265 7f9d60bd2700 10 getattrs 3.43_head/3f879843/rbd_data.10b26b8b4567.0000000000003463/head//3 = 0
2014-06-12 14:51:48.045270 7f9d60bd2700 10 stat 3.43_head/b7979843/rbd_data.10a26b8b4567.00000000000002ba/head//3
2014-06-12 14:51:48.045346 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid b7979843/rbd_data.10a26b8b4567.00000000000002ba/head//3
2014-06-12 14:51:48.065012 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:48.072393 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:48.072428 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:48.080958 7f9d60bd2700 10 getattrs 3.43_head/b7979843/rbd_data.10a26b8b4567.00000000000002ba/head//3 = 0
2014-06-12 14:51:48.080969 7f9d60bd2700 10 stat 3.43_head/59289843/rbd_data.10ad6b8b4567.00000000000021f2/head//3
2014-06-12 14:51:48.081077 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 59289843/rbd_data.10ad6b8b4567.00000000000021f2/head//3
2014-06-12 14:51:48.109814 7f9d60bd2700 10 getattrs 3.43_head/59289843/rbd_data.10ad6b8b4567.00000000000021f2/head//3 = 0
2014-06-12 14:51:48.109847 7f9d60bd2700 10 stat 3.43_head/a8489843/rbd_data.10a86b8b4567.0000000000000afc/head//3
2014-06-12 14:51:48.110045 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid a8489843/rbd_data.10a86b8b4567.0000000000000afc/head//3
2014-06-12 14:51:48.164220 7f9d60bd2700 10 getattrs 3.43_head/a8489843/rbd_data.10a86b8b4567.0000000000000afc/head//3 = 0
2014-06-12 14:51:48.164237 7f9d60bd2700 10 stat 3.43_head/25099843/rbd_data.10b46b8b4567.00000000000006da/head//3
2014-06-12 14:51:48.164407 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 25099843/rbd_data.10b46b8b4567.00000000000006da/head//3
2014-06-12 14:51:48.167914 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:48.194400 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:48.196542 7f9d60bd2700 10 getattrs 3.43_head/25099843/rbd_data.10b46b8b4567.00000000000006da/head//3 = 0
2014-06-12 14:51:48.196564 7f9d60bd2700 10 stat 3.43_head/3c7a9843/rbd_data.109c6b8b4567.0000000000000c85/head//3
2014-06-12 14:51:48.196721 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 3c7a9843/rbd_data.109c6b8b4567.0000000000000c85/head//3
2014-06-12 14:51:48.248511 7f9d60bd2700 10 getattrs 3.43_head/3c7a9843/rbd_data.109c6b8b4567.0000000000000c85/head//3 = 0
2014-06-12 14:51:48.248528 7f9d60bd2700 10 stat 3.43_head/9c7b9843/rbd_data.108d6b8b4567.0000000000003bcd/head//3
2014-06-12 14:51:48.248686 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9c7b9843/rbd_data.108d6b8b4567.0000000000003bcd/head//3
2014-06-12 14:51:48.264568 7f9d60bd2700 10 getattrs 3.43_head/9c7b9843/rbd_data.108d6b8b4567.0000000000003bcd/head//3 = 0
2014-06-12 14:51:48.264575 7f9d60bd2700 10 stat 3.43_head/9ceb9843/rbd_data.10a16b8b4567.00000000000014e8/head//3
2014-06-12 14:51:48.264668 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9ceb9843/rbd_data.10a16b8b4567.00000000000014e8/head//3
2014-06-12 14:51:48.279507 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:48.279531 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:48.279536 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:48.279538 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:48.279577 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:48.281921 7f9d60bd2700 10 getattrs 3.43_head/9ceb9843/rbd_data.10a16b8b4567.00000000000014e8/head//3 = 0
2014-06-12 14:51:48.281929 7f9d60bd2700 10 stat 3.43_head/af0c9843/rbd_data.10b26b8b4567.0000000000001505/head//3
2014-06-12 14:51:48.282021 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid af0c9843/rbd_data.10b26b8b4567.0000000000001505/head//3
2014-06-12 14:51:48.297027 7f9d60bd2700 10 getattrs 3.43_head/af0c9843/rbd_data.10b26b8b4567.0000000000001505/head//3 = 0
2014-06-12 14:51:48.297035 7f9d60bd2700 10 stat 3.43_head/222c9843/rbd_data.108d6b8b4567.00000000000005b6/head//3
2014-06-12 14:51:48.297133 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 222c9843/rbd_data.108d6b8b4567.00000000000005b6/head//3
2014-06-12 14:51:48.299602 7f9d5a16f700  5 -- op tracker -- , seq: 748603, time: 2014-06-12 14:51:48.299480, event: header_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:48.299621 7f9d5a16f700  5 -- op tracker -- , seq: 748603, time: 2014-06-12 14:51:48.299484, event: throttled, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:48.299633 7f9d5a16f700  5 -- op tracker -- , seq: 748603, time: 2014-06-12 14:51:48.299549, event: all_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:48.299644 7f9d5a16f700  5 -- op tracker -- , seq: 748603, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:48.299651 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:48.299659 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:51:48.299670 7f9d5a16f700 15 osd.2 173 enqueue_op 0x31d60f00 prio 127 cost 10717 latency 0.000189 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:48.299731 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:48.312767 7f9d60bd2700 10 getattrs 3.43_head/222c9843/rbd_data.108d6b8b4567.00000000000005b6/head//3 = 0
2014-06-12 14:51:48.312777 7f9d60bd2700 10 collection_getattrs 3.43_head
2014-06-12 14:51:48.312832 7f9d60bd2700 10 collection_getattrs 3.43_head = 0
2014-06-12 14:51:48.312840 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:48.312869 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:48.312876 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:48.312885 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:48.312888 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:48.312915 7f9d67be0700 10 osd.2 173 dequeue_op 0x31d60f00 prio 127 cost 10717 latency 0.013435 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]
2014-06-12 14:51:48.312960 7f9d67be0700  5 -- op tracker -- , seq: 748603, time: 2014-06-12 14:51:48.312953, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:48.312991 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] handle_message: 0x31d60f00
2014-06-12 14:51:48.313007 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:48.313025 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:48.313042 7f9d67be0700  5 -- op tracker -- , seq: 748603, time: 2014-06-12 14:51:48.313037, event: started, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:48.313051 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  got 24 scrub map
2014-06-12 14:51:48.313156 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] map version is 173'2566
2014-06-12 14:51:48.313177 7f9d67be0700 10 osd.2 173 dequeue_op 0x31d60f00 finish
2014-06-12 14:51:48.313189 7f9d67be0700  5 -- op tracker -- , seq: 748603, time: 2014-06-12 14:51:48.313183, event: done, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:48.313217 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:48.313226 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:48.313234 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:48.313230 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:48.313252 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:48.313277 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:48.313305 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:48.313311 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:48.313318 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub replica 24 has 25 items
2014-06-12 14:51:48.313338 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c6277843/rbd_data.109e6b8b4567.00000000000030b7/head//3, auth == maps.end()
2014-06-12 14:51:48.313366 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c6277843/rbd_data.109e6b8b4567.00000000000030b7/head//3
2014-06-12 14:51:48.313379 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d0697843/rbd_data.10956b8b4567.0000000000000ac8/head//3, auth == maps.end()
2014-06-12 14:51:48.313389 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d0697843/rbd_data.10956b8b4567.0000000000000ac8/head//3
2014-06-12 14:51:48.313398 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d57c7843/rbd_data.10a76b8b4567.0000000000001b13/head//3, auth == maps.end()
2014-06-12 14:51:48.313406 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d57c7843/rbd_data.10a76b8b4567.0000000000001b13/head//3
2014-06-12 14:51:48.313415 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 29fc7843/rbd_data.10a96b8b4567.00000000000026bd/head//3, auth == maps.end()
2014-06-12 14:51:48.313423 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 29fc7843/rbd_data.10a96b8b4567.00000000000026bd/head//3
2014-06-12 14:51:48.313432 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1b5e7843/rbd_data.10926b8b4567.00000000000033dc/head//3, auth == maps.end()
2014-06-12 14:51:48.313440 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 1b5e7843/rbd_data.10926b8b4567.00000000000033dc/head//3
2014-06-12 14:51:48.313449 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj db6f7843/rbd_data.10b06b8b4567.0000000000000de6/head//3, auth == maps.end()
2014-06-12 14:51:48.313460 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj db6f7843/rbd_data.10b06b8b4567.0000000000000de6/head//3
2014-06-12 14:51:48.313468 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a98f7843/rbd_data.10a36b8b4567.0000000000001128/head//3, auth == maps.end()
2014-06-12 14:51:48.313476 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj a98f7843/rbd_data.10a36b8b4567.0000000000001128/head//3
2014-06-12 14:51:48.313485 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 11438843/rbd_data.108d6b8b4567.0000000000002e57/head//3, auth == maps.end()
2014-06-12 14:51:48.313493 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 11438843/rbd_data.108d6b8b4567.0000000000002e57/head//3
2014-06-12 14:51:48.313502 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d9558843/rbd_data.10b26b8b4567.0000000000002817/head//3, auth == maps.end()
2014-06-12 14:51:48.313510 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d9558843/rbd_data.10b26b8b4567.0000000000002817/head//3
2014-06-12 14:51:48.313519 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cfd58843/rbd_data.10946b8b4567.0000000000001659/head//3, auth == maps.end()
2014-06-12 14:51:48.313532 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj cfd58843/rbd_data.10946b8b4567.0000000000001659/head//3
2014-06-12 14:51:48.313541 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1fea8843/rbd_data.109e6b8b4567.0000000000002ba7/head//3, auth == maps.end()
2014-06-12 14:51:48.313551 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 1fea8843/rbd_data.109e6b8b4567.0000000000002ba7/head//3
2014-06-12 14:51:48.313560 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cad09843/rbd_data.109b6b8b4567.00000000000034ac/head//3, auth == maps.end()
2014-06-12 14:51:48.313570 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj cad09843/rbd_data.109b6b8b4567.00000000000034ac/head//3
2014-06-12 14:51:48.313579 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 28b49843/rbd_data.108e6b8b4567.00000000000035bc/head//3, auth == maps.end()
2014-06-12 14:51:48.313588 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 28b49843/rbd_data.108e6b8b4567.00000000000035bc/head//3
2014-06-12 14:51:48.313596 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f5659843/rbd_data.10a76b8b4567.0000000000000a7b/head//3, auth == maps.end()
2014-06-12 14:51:48.313606 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f5659843/rbd_data.10a76b8b4567.0000000000000a7b/head//3
2014-06-12 14:51:48.313631 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d0079843/rbd_data.109f6b8b4567.00000000000034f6/head//3, auth == maps.end()
2014-06-12 14:51:48.313641 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d0079843/rbd_data.109f6b8b4567.00000000000034f6/head//3
2014-06-12 14:51:48.313660 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3f879843/rbd_data.10b26b8b4567.0000000000003463/head//3, auth == maps.end()
2014-06-12 14:51:48.313671 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 3f879843/rbd_data.10b26b8b4567.0000000000003463/head//3
2014-06-12 14:51:48.313681 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b7979843/rbd_data.10a26b8b4567.00000000000002ba/head//3, auth == maps.end()
2014-06-12 14:51:48.313694 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj b7979843/rbd_data.10a26b8b4567.00000000000002ba/head//3
2014-06-12 14:51:48.313704 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 59289843/rbd_data.10ad6b8b4567.00000000000021f2/head//3, auth == maps.end()
2014-06-12 14:51:48.313712 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 59289843/rbd_data.10ad6b8b4567.00000000000021f2/head//3
2014-06-12 14:51:48.313721 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a8489843/rbd_data.10a86b8b4567.0000000000000afc/head//3, auth == maps.end()
2014-06-12 14:51:48.313730 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj a8489843/rbd_data.10a86b8b4567.0000000000000afc/head//3
2014-06-12 14:51:48.313739 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 25099843/rbd_data.10b46b8b4567.00000000000006da/head//3, auth == maps.end()
2014-06-12 14:51:48.313747 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 25099843/rbd_data.10b46b8b4567.00000000000006da/head//3
2014-06-12 14:51:48.313757 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3c7a9843/rbd_data.109c6b8b4567.0000000000000c85/head//3, auth == maps.end()
2014-06-12 14:51:48.313765 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 3c7a9843/rbd_data.109c6b8b4567.0000000000000c85/head//3
2014-06-12 14:51:48.313774 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9c7b9843/rbd_data.108d6b8b4567.0000000000003bcd/head//3, auth == maps.end()
2014-06-12 14:51:48.313784 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9c7b9843/rbd_data.108d6b8b4567.0000000000003bcd/head//3
2014-06-12 14:51:48.313795 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9ceb9843/rbd_data.10a16b8b4567.00000000000014e8/head//3, auth == maps.end()
2014-06-12 14:51:48.313806 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9ceb9843/rbd_data.10a16b8b4567.00000000000014e8/head//3
2014-06-12 14:51:48.313817 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj af0c9843/rbd_data.10b26b8b4567.0000000000001505/head//3, auth == maps.end()
2014-06-12 14:51:48.313827 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj af0c9843/rbd_data.10b26b8b4567.0000000000001505/head//3
2014-06-12 14:51:48.313837 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 222c9843/rbd_data.108d6b8b4567.00000000000005b6/head//3, auth == maps.end()
2014-06-12 14:51:48.313845 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 222c9843/rbd_data.108d6b8b4567.00000000000005b6/head//3
2014-06-12 14:51:48.313857 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] 
2014-06-12 14:51:48.313866 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub
2014-06-12 14:51:48.313878 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  222c9843/rbd_data.108d6b8b4567.00000000000005b6/head//3 222c9843/rbd_data.108d6b8b4567.00000000000005b6/head//3(173'2188 client.4278.0:126634 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2188)
2014-06-12 14:51:48.313893 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  af0c9843/rbd_data.10b26b8b4567.0000000000001505/head//3 af0c9843/rbd_data.10b26b8b4567.0000000000001505/head//3(173'812 client.4315.0:47254 wrlock_by=unknown.0.0:0 dirty s 4194304 uv812)
2014-06-12 14:51:48.313906 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9ceb9843/rbd_data.10a16b8b4567.00000000000014e8/head//3 9ceb9843/rbd_data.10a16b8b4567.00000000000014e8/head//3(173'869 client.4298.0:48243 wrlock_by=unknown.0.0:0 dirty s 4194304 uv869)
2014-06-12 14:51:48.313920 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9c7b9843/rbd_data.108d6b8b4567.0000000000003bcd/head//3 9c7b9843/rbd_data.108d6b8b4567.0000000000003bcd/head//3(173'1981 client.4278.0:118411 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1981)
2014-06-12 14:51:48.313932 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  3c7a9843/rbd_data.109c6b8b4567.0000000000000c85/head//3 3c7a9843/rbd_data.109c6b8b4567.0000000000000c85/head//3(173'2430 client.4293.0:138137 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2430)
2014-06-12 14:51:48.313946 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  25099843/rbd_data.10b46b8b4567.00000000000006da/head//3 25099843/rbd_data.10b46b8b4567.00000000000006da/head//3(173'2226 client.4317.0:127822 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2226)
2014-06-12 14:51:48.313959 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  a8489843/rbd_data.10a86b8b4567.0000000000000afc/head//3 a8489843/rbd_data.10a86b8b4567.0000000000000afc/head//3(173'2404 client.4305.0:135648 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2404)
2014-06-12 14:51:48.313971 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  59289843/rbd_data.10ad6b8b4567.00000000000021f2/head//3 59289843/rbd_data.10ad6b8b4567.00000000000021f2/head//3(173'1265 client.4310.0:70137 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1265)
2014-06-12 14:51:48.313984 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  b7979843/rbd_data.10a26b8b4567.00000000000002ba/head//3 b7979843/rbd_data.10a26b8b4567.00000000000002ba/head//3(173'2133 client.4299.0:123330 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2133)
2014-06-12 14:51:48.313997 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  3f879843/rbd_data.10b26b8b4567.0000000000003463/head//3 3f879843/rbd_data.10b26b8b4567.0000000000003463/head//3(173'1720 client.4315.0:104233 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1720)
2014-06-12 14:51:48.314010 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d0079843/rbd_data.109f6b8b4567.00000000000034f6/head//3 d0079843/rbd_data.109f6b8b4567.00000000000034f6/head//3(173'1713 client.4296.0:103198 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1713)
2014-06-12 14:51:48.314023 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f5659843/rbd_data.10a76b8b4567.0000000000000a7b/head//3 f5659843/rbd_data.10a76b8b4567.0000000000000a7b/head//3(173'2359 client.4304.0:134060 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2359)
2014-06-12 14:51:48.314036 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  28b49843/rbd_data.108e6b8b4567.00000000000035bc/head//3 28b49843/rbd_data.108e6b8b4567.00000000000035bc/head//3(173'1816 client.4279.0:106118 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1816)
2014-06-12 14:51:48.314049 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  cad09843/rbd_data.109b6b8b4567.00000000000034ac/head//3 cad09843/rbd_data.109b6b8b4567.00000000000034ac/head//3(173'1885 client.4292.0:102684 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1885)
2014-06-12 14:51:48.314061 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  1fea8843/rbd_data.109e6b8b4567.0000000000002ba7/head//3 1fea8843/rbd_data.109e6b8b4567.0000000000002ba7/head//3(173'1494 client.4295.0:88777 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1494)
2014-06-12 14:51:48.314076 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  cfd58843/rbd_data.10946b8b4567.0000000000001659/head//3 cfd58843/rbd_data.10946b8b4567.0000000000001659/head//3(173'901 client.4285.0:49147 wrlock_by=unknown.0.0:0 dirty s 4194304 uv901)
2014-06-12 14:51:48.314089 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d9558843/rbd_data.10b26b8b4567.0000000000002817/head//3 d9558843/rbd_data.10b26b8b4567.0000000000002817/head//3(173'1325 client.4315.0:82167 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1325)
2014-06-12 14:51:48.314102 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  11438843/rbd_data.108d6b8b4567.0000000000002e57/head//3 11438843/rbd_data.108d6b8b4567.0000000000002e57/head//3(173'1527 client.4278.0:93916 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1527)
2014-06-12 14:51:48.314115 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  a98f7843/rbd_data.10a36b8b4567.0000000000001128/head//3 a98f7843/rbd_data.10a36b8b4567.0000000000001128/head//3(173'708 client.4300.0:39123 wrlock_by=unknown.0.0:0 dirty s 4194304 uv708)
2014-06-12 14:51:48.314127 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  db6f7843/rbd_data.10b06b8b4567.0000000000000de6/head//3 db6f7843/rbd_data.10b06b8b4567.0000000000000de6/head//3(173'2495 client.4313.0:143465 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2495)
2014-06-12 14:51:48.314140 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  1b5e7843/rbd_data.10926b8b4567.00000000000033dc/head//3 1b5e7843/rbd_data.10926b8b4567.00000000000033dc/head//3(173'1835 client.4283.0:103128 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1835)
2014-06-12 14:51:48.314153 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  29fc7843/rbd_data.10a96b8b4567.00000000000026bd/head//3 29fc7843/rbd_data.10a96b8b4567.00000000000026bd/head//3(173'1442 client.4306.0:81586 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1442)
2014-06-12 14:51:48.314166 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d57c7843/rbd_data.10a76b8b4567.0000000000001b13/head//3 d57c7843/rbd_data.10a76b8b4567.0000000000001b13/head//3(173'1083 client.4304.0:58715 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1083)
2014-06-12 14:51:48.314179 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d0697843/rbd_data.10956b8b4567.0000000000000ac8/head//3 d0697843/rbd_data.10956b8b4567.0000000000000ac8/head//3(173'2372 client.4286.0:135333 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2372)
2014-06-12 14:51:48.314192 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c6277843/rbd_data.109e6b8b4567.00000000000030b7/head//3 c6277843/rbd_data.109e6b8b4567.00000000000030b7/head//3(173'1635 client.4295.0:97837 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1635)
2014-06-12 14:51:48.314202 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:48.314212 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:48.314221 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:48.314223 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:48.314224 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:48.314263 7f9d60bd2700 10 collection_list_partial 3.43_head start:645c9843//0//-1 is_max:0
2014-06-12 14:51:48.314486 7f9d60bd2700 20 objects: [4c7c9843/rbd_data.10a96b8b4567.000000000000256a/head//3,9f1d9843/rbd_data.10a26b8b4567.00000000000036d3/head//3,a4d9843/rbd_data.10a96b8b4567.000000000000273b/head//3,8080a843/rbd_data.10a86b8b4567.000000000000182e/head//3,c592a843/rbd_data.10a86b8b4567.0000000000002113/head//3,7d34a843/rbd_data.109b6b8b4567.0000000000002b0c/head//3,3fe6a843/rbd_data.10a76b8b4567.0000000000000cf8/head//3,a16ba843/rbd_data.109a6b8b4567.0000000000000eb9/head//3,956ba843/rbd_data.10b36b8b4567.0000000000000829/head//3,d3cca843/rbd_data.10966b8b4567.00000000000028c3/head//3,dfea843/rbd_data.10a06b8b4567.0000000000001ce8/head//3,e49fa843/rbd_data.10a86b8b4567.00000000000007fe/head//3,dea1b843/rbd_data.10926b8b4567.0000000000003993/head//3,c1b1b843/rbd_data.109c6b8b4567.0000000000001e9a/head//3,27d5b843/rbd_data.10ad6b8b4567.0000000000003857/head//3,736b843/rbd_data.10986b8b4567.00000000000004a9/head//3,9966b843/rbd_data.10936b8b4567.00000000000030a4/head//3,9c57b843/rbd_data.10a66b8b4567.00000000000007b3/head//3,c549b843/rbd_data.10a56b8b4567.0000000000002925/head//3,9b49b843/rbd_data.109e6b8b4567.00000000000038e5/head//3,60cdb843/rbd_data.10b26b8b4567.0000000000001c6c/head//3,59beb843/rbd_data.10956b8b4567.00000000000005f0/head//3,1ef1c843/rbd_data.109a6b8b4567.0000000000001abf/head//3,d303c843/rbd_data.10916b8b4567.0000000000002412/head//3,c259c843/rbd_data.109c6b8b4567.0000000000000818/head//3]
2014-06-12 14:51:48.314922 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  requesting scrubmap from osd.24
2014-06-12 14:51:48.314943 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:48.314959 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:48.314968 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:48.314974 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:48.314980 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:48.314985 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_map_chunk [645c9843//0//-1,b989c843//0//-1)
2014-06-12 14:51:48.314993 7f9d60bd2700 10 collection_list_partial 3.43_head start:645c9843//0//-1 is_max:0
2014-06-12 14:51:48.315393 7f9d60bd2700 20 objects: [4c7c9843/rbd_data.10a96b8b4567.000000000000256a/head//3,9f1d9843/rbd_data.10a26b8b4567.00000000000036d3/head//3,a4d9843/rbd_data.10a96b8b4567.000000000000273b/head//3,8080a843/rbd_data.10a86b8b4567.000000000000182e/head//3,c592a843/rbd_data.10a86b8b4567.0000000000002113/head//3,7d34a843/rbd_data.109b6b8b4567.0000000000002b0c/head//3,3fe6a843/rbd_data.10a76b8b4567.0000000000000cf8/head//3,a16ba843/rbd_data.109a6b8b4567.0000000000000eb9/head//3,956ba843/rbd_data.10b36b8b4567.0000000000000829/head//3,d3cca843/rbd_data.10966b8b4567.00000000000028c3/head//3,dfea843/rbd_data.10a06b8b4567.0000000000001ce8/head//3,e49fa843/rbd_data.10a86b8b4567.00000000000007fe/head//3,dea1b843/rbd_data.10926b8b4567.0000000000003993/head//3,c1b1b843/rbd_data.109c6b8b4567.0000000000001e9a/head//3,27d5b843/rbd_data.10ad6b8b4567.0000000000003857/head//3,736b843/rbd_data.10986b8b4567.00000000000004a9/head//3,9966b843/rbd_data.10936b8b4567.00000000000030a4/head//3,9c57b843/rbd_data.10a66b8b4567.00000000000007b3/head//3,c549b843/rbd_data.10a56b8b4567.0000000000002925/head//3,9b49b843/rbd_data.109e6b8b4567.00000000000038e5/head//3,60cdb843/rbd_data.10b26b8b4567.0000000000001c6c/head//3,59beb843/rbd_data.10956b8b4567.00000000000005f0/head//3,1ef1c843/rbd_data.109a6b8b4567.0000000000001abf/head//3,d303c843/rbd_data.10916b8b4567.0000000000002412/head//3,c259c843/rbd_data.109c6b8b4567.0000000000000818/head//3,b989c843/rbd_data.10946b8b4567.0000000000002e46/head//3,f6b9c843/rbd_data.109e6b8b4567.0000000000000cd6/head//3,54bbc843/rbd_data.10b36b8b4567.0000000000000710/head//3,5a3cc843/rbd_data.10ac6b8b4567.0000000000001d97/head//3,b6bcc843/rbd_data.10906b8b4567.00000000000009af/head//3,ecedc843/rbd_data.10b06b8b4567.0000000000003096/head//3,d77ec843/rbd_data.10af6b8b4567.000000000000186b/head//3,b1fc843/rbd_data.10ab6b8b4567.00000000000018f4/head//3,398fc843/rbd_data.108d6b8b4567.00000000000033d0/head//3,e213d843/rbd_data.10b06b8b4567.000000000000005e/head//3,f655d843/rbd_data.10946b8b4567.00000000000037bd/head//3,5e8d843/rbd_data.10b06b8b4567.0000000000002b59/head//3,1bd843/rbd_data.10916b8b4567.00000000000026ec/head//3,2c4fd843/rbd_data.108f6b8b4567.0000000000002f8c/head//3,f66fd843/rbd_data.108e6b8b4567.0000000000000dfd/head//3,8f92e843/rbd_data.108e6b8b4567.0000000000000e2e/head//3,f73e843/rbd_data.10956b8b4567.0000000000002baa/head//3,1734e843/rbd_data.10ab6b8b4567.0000000000001a5f/head//3,5d15e843/rbd_data.10b46b8b4567.0000000000002ef8/head//3,4c06e843/rbd_data.109a6b8b4567.000000000000161f/head//3,2a9e843/rbd_data.108d6b8b4567.00000000000031ba/head//3,c41fe843/rbd_data.10a56b8b4567.00000000000024ce/head//3,eba4f843/rbd_data.10926b8b4567.0000000000001a96/head//3,7a85f843/rbd_data.10a76b8b4567.0000000000000482/head//3,dcc8f843/rbd_data.109d6b8b4567.0000000000001dea/head//3,b4f8f843/rbd_data.10aa6b8b4567.0000000000002e5d/head//3,3c49f843/rbd_data.10a26b8b4567.00000000000015a0/head//3,1cebf843/rbd_data.10a46b8b4567.0000000000000518/head//3,59edf843/rbd_data.109c6b8b4567.0000000000001efa/head//3,551ef843/rbd_data.10ab6b8b4567.0000000000003573/head//3,f02ef843/rbd_data.10b26b8b4567.0000000000003567/head//3]
2014-06-12 14:51:48.315458 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:48.315478 7f9d60bd2700 10 stat 3.43_head/4c7c9843/rbd_data.10a96b8b4567.000000000000256a/head//3
2014-06-12 14:51:48.315591 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 4c7c9843/rbd_data.10a96b8b4567.000000000000256a/head//3
2014-06-12 14:51:48.326688 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:48.339414 7f9d60bd2700 10 getattrs 3.43_head/4c7c9843/rbd_data.10a96b8b4567.000000000000256a/head//3 = 0
2014-06-12 14:51:48.339430 7f9d60bd2700 10 stat 3.43_head/9f1d9843/rbd_data.10a26b8b4567.00000000000036d3/head//3
2014-06-12 14:51:48.339575 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9f1d9843/rbd_data.10a26b8b4567.00000000000036d3/head//3
2014-06-12 14:51:48.365879 7f9d60bd2700 10 getattrs 3.43_head/9f1d9843/rbd_data.10a26b8b4567.00000000000036d3/head//3 = 0
2014-06-12 14:51:48.365893 7f9d60bd2700 10 stat 3.43_head/a4d9843/rbd_data.10a96b8b4567.000000000000273b/head//3
2014-06-12 14:51:48.366059 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid a4d9843/rbd_data.10a96b8b4567.000000000000273b/head//3
2014-06-12 14:51:48.366727 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:48.371637 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:48.382340 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:48.395631 7f9d60bd2700 10 getattrs 3.43_head/a4d9843/rbd_data.10a96b8b4567.000000000000273b/head//3 = 0
2014-06-12 14:51:48.395640 7f9d60bd2700 10 stat 3.43_head/8080a843/rbd_data.10a86b8b4567.000000000000182e/head//3
2014-06-12 14:51:48.395733 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 8080a843/rbd_data.10a86b8b4567.000000000000182e/head//3
2014-06-12 14:51:48.396902 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:48.396951 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:48.399872 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:48.422000 7f9d60bd2700 10 getattrs 3.43_head/8080a843/rbd_data.10a86b8b4567.000000000000182e/head//3 = 0
2014-06-12 14:51:48.422009 7f9d60bd2700 10 stat 3.43_head/c592a843/rbd_data.10a86b8b4567.0000000000002113/head//3
2014-06-12 14:51:48.422110 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c592a843/rbd_data.10a86b8b4567.0000000000002113/head//3
2014-06-12 14:51:48.432195 7f9d60bd2700 10 getattrs 3.43_head/c592a843/rbd_data.10a86b8b4567.0000000000002113/head//3 = 0
2014-06-12 14:51:48.432204 7f9d60bd2700 10 stat 3.43_head/7d34a843/rbd_data.109b6b8b4567.0000000000002b0c/head//3
2014-06-12 14:51:48.432301 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 7d34a843/rbd_data.109b6b8b4567.0000000000002b0c/head//3
2014-06-12 14:51:48.457992 7f9d60bd2700 10 getattrs 3.43_head/7d34a843/rbd_data.109b6b8b4567.0000000000002b0c/head//3 = 0
2014-06-12 14:51:48.458007 7f9d60bd2700 10 stat 3.43_head/3fe6a843/rbd_data.10a76b8b4567.0000000000000cf8/head//3
2014-06-12 14:51:48.458149 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 3fe6a843/rbd_data.10a76b8b4567.0000000000000cf8/head//3
2014-06-12 14:51:48.494450 7f9d60bd2700 10 getattrs 3.43_head/3fe6a843/rbd_data.10a76b8b4567.0000000000000cf8/head//3 = 0
2014-06-12 14:51:48.494463 7f9d60bd2700 10 stat 3.43_head/a16ba843/rbd_data.109a6b8b4567.0000000000000eb9/head//3
2014-06-12 14:51:48.494591 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid a16ba843/rbd_data.109a6b8b4567.0000000000000eb9/head//3
2014-06-12 14:51:48.522480 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:48.523331 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:48.529934 7f9d60bd2700 10 getattrs 3.43_head/a16ba843/rbd_data.109a6b8b4567.0000000000000eb9/head//3 = 0
2014-06-12 14:51:48.529950 7f9d60bd2700 10 stat 3.43_head/956ba843/rbd_data.10b36b8b4567.0000000000000829/head//3
2014-06-12 14:51:48.530105 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 956ba843/rbd_data.10b36b8b4567.0000000000000829/head//3
2014-06-12 14:51:48.532787 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:48.568875 7f9d60bd2700 10 getattrs 3.43_head/956ba843/rbd_data.10b36b8b4567.0000000000000829/head//3 = 0
2014-06-12 14:51:48.568882 7f9d60bd2700 10 stat 3.43_head/d3cca843/rbd_data.10966b8b4567.00000000000028c3/head//3
2014-06-12 14:51:48.568983 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d3cca843/rbd_data.10966b8b4567.00000000000028c3/head//3
2014-06-12 14:51:48.570242 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:48.580369 7f9d60bd2700 10 getattrs 3.43_head/d3cca843/rbd_data.10966b8b4567.00000000000028c3/head//3 = 0
2014-06-12 14:51:48.580376 7f9d60bd2700 10 stat 3.43_head/dfea843/rbd_data.10a06b8b4567.0000000000001ce8/head//3
2014-06-12 14:51:48.580473 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid dfea843/rbd_data.10a06b8b4567.0000000000001ce8/head//3
2014-06-12 14:51:48.588674 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:48.588718 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:48.592617 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:48.592633 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:48.593058 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:48.593142 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:48.593177 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:48.593193 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:48.593206 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:48.593335 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:48.593381 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:48.593407 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:48.593433 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:48.593458 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:48.593481 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:48.593506 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:48.593543 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:48.593568 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:48.593594 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:48.593639 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:48.593664 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:48.593687 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:48.593707 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:48.593731 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:48.593742 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:48.593762 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:48.603780 7f9d60bd2700 10 getattrs 3.43_head/dfea843/rbd_data.10a06b8b4567.0000000000001ce8/head//3 = 0
2014-06-12 14:51:48.603790 7f9d60bd2700 10 stat 3.43_head/e49fa843/rbd_data.10a86b8b4567.00000000000007fe/head//3
2014-06-12 14:51:48.603905 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid e49fa843/rbd_data.10a86b8b4567.00000000000007fe/head//3
2014-06-12 14:51:48.658812 7f9d60bd2700 10 getattrs 3.43_head/e49fa843/rbd_data.10a86b8b4567.00000000000007fe/head//3 = 0
2014-06-12 14:51:48.658820 7f9d60bd2700 10 stat 3.43_head/dea1b843/rbd_data.10926b8b4567.0000000000003993/head//3
2014-06-12 14:51:48.658913 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid dea1b843/rbd_data.10926b8b4567.0000000000003993/head//3
2014-06-12 14:51:48.662128 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:48.680058 7f9d60bd2700 10 getattrs 3.43_head/dea1b843/rbd_data.10926b8b4567.0000000000003993/head//3 = 0
2014-06-12 14:51:48.680065 7f9d60bd2700 10 stat 3.43_head/c1b1b843/rbd_data.109c6b8b4567.0000000000001e9a/head//3
2014-06-12 14:51:48.680145 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c1b1b843/rbd_data.109c6b8b4567.0000000000001e9a/head//3
2014-06-12 14:51:48.680326 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:48.700989 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:48.713503 7f9d60bd2700 10 getattrs 3.43_head/c1b1b843/rbd_data.109c6b8b4567.0000000000001e9a/head//3 = 0
2014-06-12 14:51:48.713518 7f9d60bd2700 10 stat 3.43_head/27d5b843/rbd_data.10ad6b8b4567.0000000000003857/head//3
2014-06-12 14:51:48.713639 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 27d5b843/rbd_data.10ad6b8b4567.0000000000003857/head//3
2014-06-12 14:51:48.726762 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:48.732856 7f9d60bd2700 10 getattrs 3.43_head/27d5b843/rbd_data.10ad6b8b4567.0000000000003857/head//3 = 0
2014-06-12 14:51:48.732865 7f9d60bd2700 10 stat 3.43_head/736b843/rbd_data.10986b8b4567.00000000000004a9/head//3
2014-06-12 14:51:48.732985 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 736b843/rbd_data.10986b8b4567.00000000000004a9/head//3
2014-06-12 14:51:48.768460 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:48.768860 7f9d60bd2700 10 getattrs 3.43_head/736b843/rbd_data.10986b8b4567.00000000000004a9/head//3 = 0
2014-06-12 14:51:48.768870 7f9d60bd2700 10 stat 3.43_head/9966b843/rbd_data.10936b8b4567.00000000000030a4/head//3
2014-06-12 14:51:48.768979 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9966b843/rbd_data.10936b8b4567.00000000000030a4/head//3
2014-06-12 14:51:48.769066 7f9d60bd2700 10 getattrs 3.43_head/9966b843/rbd_data.10936b8b4567.00000000000030a4/head//3 = 0
2014-06-12 14:51:48.769072 7f9d60bd2700 10 stat 3.43_head/9c57b843/rbd_data.10a66b8b4567.00000000000007b3/head//3
2014-06-12 14:51:48.769161 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9c57b843/rbd_data.10a66b8b4567.00000000000007b3/head//3
2014-06-12 14:51:48.802361 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:48.819944 7f9d60bd2700 10 getattrs 3.43_head/9c57b843/rbd_data.10a66b8b4567.00000000000007b3/head//3 = 0
2014-06-12 14:51:48.819960 7f9d60bd2700 10 stat 3.43_head/c549b843/rbd_data.10a56b8b4567.0000000000002925/head//3
2014-06-12 14:51:48.820081 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c549b843/rbd_data.10a56b8b4567.0000000000002925/head//3
2014-06-12 14:51:48.847101 7f9d60bd2700 10 getattrs 3.43_head/c549b843/rbd_data.10a56b8b4567.0000000000002925/head//3 = 0
2014-06-12 14:51:48.847114 7f9d60bd2700 10 stat 3.43_head/9b49b843/rbd_data.109e6b8b4567.00000000000038e5/head//3
2014-06-12 14:51:48.847240 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 9b49b843/rbd_data.109e6b8b4567.00000000000038e5/head//3
2014-06-12 14:51:48.855491 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:48.865867 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:48.869202 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:48.869397 7f9d60bd2700 10 getattrs 3.43_head/9b49b843/rbd_data.109e6b8b4567.00000000000038e5/head//3 = 0
2014-06-12 14:51:48.869409 7f9d60bd2700 10 stat 3.43_head/60cdb843/rbd_data.10b26b8b4567.0000000000001c6c/head//3
2014-06-12 14:51:48.869534 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 60cdb843/rbd_data.10b26b8b4567.0000000000001c6c/head//3
2014-06-12 14:51:48.872607 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:48.872701 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:48.898869 7f9d60bd2700 10 getattrs 3.43_head/60cdb843/rbd_data.10b26b8b4567.0000000000001c6c/head//3 = 0
2014-06-12 14:51:48.898884 7f9d60bd2700 10 stat 3.43_head/59beb843/rbd_data.10956b8b4567.00000000000005f0/head//3
2014-06-12 14:51:48.899008 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 59beb843/rbd_data.10956b8b4567.00000000000005f0/head//3
2014-06-12 14:51:48.907979 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:48.937399 7f9d5a16f700  5 -- op tracker -- , seq: 748604, time: 2014-06-12 14:51:48.937261, event: header_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:48.937417 7f9d5a16f700  5 -- op tracker -- , seq: 748604, time: 2014-06-12 14:51:48.937265, event: throttled, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:48.937430 7f9d5a16f700  5 -- op tracker -- , seq: 748604, time: 2014-06-12 14:51:48.937349, event: all_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:48.937469 7f9d5a16f700  5 -- op tracker -- , seq: 748604, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:48.937474 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:48.937481 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:51:48.937488 7f9d5a16f700 15 osd.2 173 enqueue_op 0x31d60300 prio 127 cost 10717 latency 0.000226 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:48.937544 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:48.949586 7f9d60bd2700 10 getattrs 3.43_head/59beb843/rbd_data.10956b8b4567.00000000000005f0/head//3 = 0
2014-06-12 14:51:48.949599 7f9d60bd2700 10 stat 3.43_head/1ef1c843/rbd_data.109a6b8b4567.0000000000001abf/head//3
2014-06-12 14:51:48.949752 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 1ef1c843/rbd_data.109a6b8b4567.0000000000001abf/head//3
2014-06-12 14:51:48.966463 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:48.975066 7f9d60bd2700 10 getattrs 3.43_head/1ef1c843/rbd_data.109a6b8b4567.0000000000001abf/head//3 = 0
2014-06-12 14:51:48.975078 7f9d60bd2700 10 stat 3.43_head/d303c843/rbd_data.10916b8b4567.0000000000002412/head//3
2014-06-12 14:51:48.975209 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d303c843/rbd_data.10916b8b4567.0000000000002412/head//3
2014-06-12 14:51:48.994042 7f9d60bd2700 10 getattrs 3.43_head/d303c843/rbd_data.10916b8b4567.0000000000002412/head//3 = 0
2014-06-12 14:51:48.994054 7f9d60bd2700 10 stat 3.43_head/c259c843/rbd_data.109c6b8b4567.0000000000000818/head//3
2014-06-12 14:51:48.994182 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c259c843/rbd_data.109c6b8b4567.0000000000000818/head//3
2014-06-12 14:51:48.995664 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:49.030021 7f9d60bd2700 10 getattrs 3.43_head/c259c843/rbd_data.109c6b8b4567.0000000000000818/head//3 = 0
2014-06-12 14:51:49.030036 7f9d60bd2700 10 collection_getattrs 3.43_head
2014-06-12 14:51:49.030099 7f9d60bd2700 10 collection_getattrs 3.43_head = 0
2014-06-12 14:51:49.030109 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:49.030154 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:49.030164 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:49.030175 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:49.030179 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:49.030209 7f9d66bde700 10 osd.2 173 dequeue_op 0x31d60300 prio 127 cost 10717 latency 0.092947 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]
2014-06-12 14:51:49.030260 7f9d66bde700  5 -- op tracker -- , seq: 748604, time: 2014-06-12 14:51:49.030253, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.030286 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] handle_message: 0x31d60300
2014-06-12 14:51:49.030302 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:49.030320 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:49.030337 7f9d66bde700  5 -- op tracker -- , seq: 748604, time: 2014-06-12 14:51:49.030332, event: started, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.030362 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  got 24 scrub map
2014-06-12 14:51:49.030466 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] map version is 173'2566
2014-06-12 14:51:49.030488 7f9d66bde700 10 osd.2 173 dequeue_op 0x31d60300 finish
2014-06-12 14:51:49.030500 7f9d66bde700  5 -- op tracker -- , seq: 748604, time: 2014-06-12 14:51:49.030494, event: done, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.030524 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:49.030539 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:49.030545 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:49.030537 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:49.030567 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:49.030576 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:49.030605 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:49.030614 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:49.030623 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub replica 24 has 25 items
2014-06-12 14:51:49.030649 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4c7c9843/rbd_data.10a96b8b4567.000000000000256a/head//3, auth == maps.end()
2014-06-12 14:51:49.030677 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 4c7c9843/rbd_data.10a96b8b4567.000000000000256a/head//3
2014-06-12 14:51:49.030695 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9f1d9843/rbd_data.10a26b8b4567.00000000000036d3/head//3, auth == maps.end()
2014-06-12 14:51:49.030709 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9f1d9843/rbd_data.10a26b8b4567.00000000000036d3/head//3
2014-06-12 14:51:49.030722 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a4d9843/rbd_data.10a96b8b4567.000000000000273b/head//3, auth == maps.end()
2014-06-12 14:51:49.030733 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj a4d9843/rbd_data.10a96b8b4567.000000000000273b/head//3
2014-06-12 14:51:49.030745 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8080a843/rbd_data.10a86b8b4567.000000000000182e/head//3, auth == maps.end()
2014-06-12 14:51:49.030757 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 8080a843/rbd_data.10a86b8b4567.000000000000182e/head//3
2014-06-12 14:51:49.030769 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c592a843/rbd_data.10a86b8b4567.0000000000002113/head//3, auth == maps.end()
2014-06-12 14:51:49.030780 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c592a843/rbd_data.10a86b8b4567.0000000000002113/head//3
2014-06-12 14:51:49.030791 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7d34a843/rbd_data.109b6b8b4567.0000000000002b0c/head//3, auth == maps.end()
2014-06-12 14:51:49.030805 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 7d34a843/rbd_data.109b6b8b4567.0000000000002b0c/head//3
2014-06-12 14:51:49.030820 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3fe6a843/rbd_data.10a76b8b4567.0000000000000cf8/head//3, auth == maps.end()
2014-06-12 14:51:49.030833 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 3fe6a843/rbd_data.10a76b8b4567.0000000000000cf8/head//3
2014-06-12 14:51:49.030845 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a16ba843/rbd_data.109a6b8b4567.0000000000000eb9/head//3, auth == maps.end()
2014-06-12 14:51:49.030857 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj a16ba843/rbd_data.109a6b8b4567.0000000000000eb9/head//3
2014-06-12 14:51:49.030869 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 956ba843/rbd_data.10b36b8b4567.0000000000000829/head//3, auth == maps.end()
2014-06-12 14:51:49.030882 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 956ba843/rbd_data.10b36b8b4567.0000000000000829/head//3
2014-06-12 14:51:49.030894 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d3cca843/rbd_data.10966b8b4567.00000000000028c3/head//3, auth == maps.end()
2014-06-12 14:51:49.030906 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d3cca843/rbd_data.10966b8b4567.00000000000028c3/head//3
2014-06-12 14:51:49.030918 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dfea843/rbd_data.10a06b8b4567.0000000000001ce8/head//3, auth == maps.end()
2014-06-12 14:51:49.030929 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj dfea843/rbd_data.10a06b8b4567.0000000000001ce8/head//3
2014-06-12 14:51:49.030941 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e49fa843/rbd_data.10a86b8b4567.00000000000007fe/head//3, auth == maps.end()
2014-06-12 14:51:49.030953 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj e49fa843/rbd_data.10a86b8b4567.00000000000007fe/head//3
2014-06-12 14:51:49.030964 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dea1b843/rbd_data.10926b8b4567.0000000000003993/head//3, auth == maps.end()
2014-06-12 14:51:49.030978 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj dea1b843/rbd_data.10926b8b4567.0000000000003993/head//3
2014-06-12 14:51:49.030990 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c1b1b843/rbd_data.109c6b8b4567.0000000000001e9a/head//3, auth == maps.end()
2014-06-12 14:51:49.031004 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c1b1b843/rbd_data.109c6b8b4567.0000000000001e9a/head//3
2014-06-12 14:51:49.031018 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 27d5b843/rbd_data.10ad6b8b4567.0000000000003857/head//3, auth == maps.end()
2014-06-12 14:51:49.031030 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 27d5b843/rbd_data.10ad6b8b4567.0000000000003857/head//3
2014-06-12 14:51:49.031043 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 736b843/rbd_data.10986b8b4567.00000000000004a9/head//3, auth == maps.end()
2014-06-12 14:51:49.031057 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 736b843/rbd_data.10986b8b4567.00000000000004a9/head//3
2014-06-12 14:51:49.031069 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9966b843/rbd_data.10936b8b4567.00000000000030a4/head//3, auth == maps.end()
2014-06-12 14:51:49.031082 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9966b843/rbd_data.10936b8b4567.00000000000030a4/head//3
2014-06-12 14:51:49.031095 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9c57b843/rbd_data.10a66b8b4567.00000000000007b3/head//3, auth == maps.end()
2014-06-12 14:51:49.031108 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9c57b843/rbd_data.10a66b8b4567.00000000000007b3/head//3
2014-06-12 14:51:49.031121 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c549b843/rbd_data.10a56b8b4567.0000000000002925/head//3, auth == maps.end()
2014-06-12 14:51:49.031134 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c549b843/rbd_data.10a56b8b4567.0000000000002925/head//3
2014-06-12 14:51:49.031146 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9b49b843/rbd_data.109e6b8b4567.00000000000038e5/head//3, auth == maps.end()
2014-06-12 14:51:49.031157 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 9b49b843/rbd_data.109e6b8b4567.00000000000038e5/head//3
2014-06-12 14:51:49.031169 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 60cdb843/rbd_data.10b26b8b4567.0000000000001c6c/head//3, auth == maps.end()
2014-06-12 14:51:49.031180 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 60cdb843/rbd_data.10b26b8b4567.0000000000001c6c/head//3
2014-06-12 14:51:49.031193 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 59beb843/rbd_data.10956b8b4567.00000000000005f0/head//3, auth == maps.end()
2014-06-12 14:51:49.031204 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 59beb843/rbd_data.10956b8b4567.00000000000005f0/head//3
2014-06-12 14:51:49.031215 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1ef1c843/rbd_data.109a6b8b4567.0000000000001abf/head//3, auth == maps.end()
2014-06-12 14:51:49.031226 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 1ef1c843/rbd_data.109a6b8b4567.0000000000001abf/head//3
2014-06-12 14:51:49.031238 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d303c843/rbd_data.10916b8b4567.0000000000002412/head//3, auth == maps.end()
2014-06-12 14:51:49.031251 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d303c843/rbd_data.10916b8b4567.0000000000002412/head//3
2014-06-12 14:51:49.031263 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c259c843/rbd_data.109c6b8b4567.0000000000000818/head//3, auth == maps.end()
2014-06-12 14:51:49.031274 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c259c843/rbd_data.109c6b8b4567.0000000000000818/head//3
2014-06-12 14:51:49.031289 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] 
2014-06-12 14:51:49.031300 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub
2014-06-12 14:51:49.031316 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c259c843/rbd_data.109c6b8b4567.0000000000000818/head//3 c259c843/rbd_data.109c6b8b4567.0000000000000818/head//3(173'2267 client.4293.0:130436 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2267)
2014-06-12 14:51:49.031335 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d303c843/rbd_data.10916b8b4567.0000000000002412/head//3 d303c843/rbd_data.10916b8b4567.0000000000002412/head//3(173'1311 client.4282.0:74727 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1311)
2014-06-12 14:51:49.031353 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  1ef1c843/rbd_data.109a6b8b4567.0000000000001abf/head//3 1ef1c843/rbd_data.109a6b8b4567.0000000000001abf/head//3(173'1011 client.4291.0:57341 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1011)
2014-06-12 14:51:49.031369 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  59beb843/rbd_data.10956b8b4567.00000000000005f0/head//3 59beb843/rbd_data.10956b8b4567.00000000000005f0/head//3(173'2213 client.4286.0:127058 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2213)
2014-06-12 14:51:49.031386 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  60cdb843/rbd_data.10b26b8b4567.0000000000001c6c/head//3 60cdb843/rbd_data.10b26b8b4567.0000000000001c6c/head//3(173'1003 client.4315.0:60886 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1003)
2014-06-12 14:51:49.031403 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9b49b843/rbd_data.109e6b8b4567.00000000000038e5/head//3 9b49b843/rbd_data.109e6b8b4567.00000000000038e5/head//3(173'1864 client.4295.0:112107 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1864)
2014-06-12 14:51:49.031419 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c549b843/rbd_data.10a56b8b4567.0000000000002925/head//3 c549b843/rbd_data.10a56b8b4567.0000000000002925/head//3(173'1482 client.4302.0:84814 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1482)
2014-06-12 14:51:49.031436 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9c57b843/rbd_data.10a66b8b4567.00000000000007b3/head//3 9c57b843/rbd_data.10a66b8b4567.00000000000007b3/head//3(173'2263 client.4303.0:130463 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2263)
2014-06-12 14:51:49.031453 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9966b843/rbd_data.10936b8b4567.00000000000030a4/head//3 9966b843/rbd_data.10936b8b4567.00000000000030a4/head//3(173'1680 client.4284.0:95316 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1680)
2014-06-12 14:51:49.031470 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  736b843/rbd_data.10986b8b4567.00000000000004a9/head//3 736b843/rbd_data.10986b8b4567.00000000000004a9/head//3(173'2165 client.4289.0:122304 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2165)
2014-06-12 14:51:49.031486 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  27d5b843/rbd_data.10ad6b8b4567.0000000000003857/head//3 27d5b843/rbd_data.10ad6b8b4567.0000000000003857/head//3(173'1892 client.4310.0:110992 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1892)
2014-06-12 14:51:49.031503 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c1b1b843/rbd_data.109c6b8b4567.0000000000001e9a/head//3 c1b1b843/rbd_data.109c6b8b4567.0000000000001e9a/head//3(173'1237 client.4293.0:63915 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1237)
2014-06-12 14:51:49.031520 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  dea1b843/rbd_data.10926b8b4567.0000000000003993/head//3 dea1b843/rbd_data.10926b8b4567.0000000000003993/head//3(173'2009 client.4283.0:113411 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2009)
2014-06-12 14:51:49.031536 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  e49fa843/rbd_data.10a86b8b4567.00000000000007fe/head//3 e49fa843/rbd_data.10a86b8b4567.00000000000007fe/head//3(173'2298 client.4305.0:130395 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2298)
2014-06-12 14:51:49.031553 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  dfea843/rbd_data.10a06b8b4567.0000000000001ce8/head//3 dfea843/rbd_data.10a06b8b4567.0000000000001ce8/head//3(173'1042 client.4297.0:60710 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1042)
2014-06-12 14:51:49.031570 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d3cca843/rbd_data.10966b8b4567.00000000000028c3/head//3 d3cca843/rbd_data.10966b8b4567.00000000000028c3/head//3(173'1359 client.4287.0:79746 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1359)
2014-06-12 14:51:49.031587 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  956ba843/rbd_data.10b36b8b4567.0000000000000829/head//3 956ba843/rbd_data.10b36b8b4567.0000000000000829/head//3(173'2281 client.4316.0:132170 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2281)
2014-06-12 14:51:49.031604 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  a16ba843/rbd_data.109a6b8b4567.0000000000000eb9/head//3 a16ba843/rbd_data.109a6b8b4567.0000000000000eb9/head//3(173'2524 client.4291.0:139610 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2524)
2014-06-12 14:51:49.031619 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  3fe6a843/rbd_data.10a76b8b4567.0000000000000cf8/head//3 3fe6a843/rbd_data.10a76b8b4567.0000000000000cf8/head//3(173'2438 client.4304.0:138450 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2438)
2014-06-12 14:51:49.031636 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  7d34a843/rbd_data.109b6b8b4567.0000000000002b0c/head//3 7d34a843/rbd_data.109b6b8b4567.0000000000002b0c/head//3(173'1539 client.4292.0:85007 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1539)
2014-06-12 14:51:49.031652 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c592a843/rbd_data.10a86b8b4567.0000000000002113/head//3 c592a843/rbd_data.10a86b8b4567.0000000000002113/head//3(173'1185 client.4305.0:68335 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1185)
2014-06-12 14:51:49.031669 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  8080a843/rbd_data.10a86b8b4567.000000000000182e/head//3 8080a843/rbd_data.10a86b8b4567.000000000000182e/head//3(173'920 client.4305.0:52365 wrlock_by=unknown.0.0:0 dirty s 4194304 uv920)
2014-06-12 14:51:49.031685 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  a4d9843/rbd_data.10a96b8b4567.000000000000273b/head//3 a4d9843/rbd_data.10a96b8b4567.000000000000273b/head//3(173'1448 client.4306.0:82497 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1448)
2014-06-12 14:51:49.031702 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  9f1d9843/rbd_data.10a26b8b4567.00000000000036d3/head//3 9f1d9843/rbd_data.10a26b8b4567.00000000000036d3/head//3(173'1991 client.4299.0:111002 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1991)
2014-06-12 14:51:49.031719 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  4c7c9843/rbd_data.10a96b8b4567.000000000000256a/head//3 4c7c9843/rbd_data.10a96b8b4567.000000000000256a/head//3(173'1389 client.4306.0:79110 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1389)
2014-06-12 14:51:49.031732 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:49.031745 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:49.031756 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:49.031759 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:49.031761 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:49.031804 7f9d60bd2700 10 collection_list_partial 3.43_head start:b989c843//0//-1 is_max:0
2014-06-12 14:51:49.032068 7f9d60bd2700 20 objects: [f6b9c843/rbd_data.109e6b8b4567.0000000000000cd6/head//3,54bbc843/rbd_data.10b36b8b4567.0000000000000710/head//3,5a3cc843/rbd_data.10ac6b8b4567.0000000000001d97/head//3,b6bcc843/rbd_data.10906b8b4567.00000000000009af/head//3,ecedc843/rbd_data.10b06b8b4567.0000000000003096/head//3,d77ec843/rbd_data.10af6b8b4567.000000000000186b/head//3,b1fc843/rbd_data.10ab6b8b4567.00000000000018f4/head//3,398fc843/rbd_data.108d6b8b4567.00000000000033d0/head//3,e213d843/rbd_data.10b06b8b4567.000000000000005e/head//3,f655d843/rbd_data.10946b8b4567.00000000000037bd/head//3,5e8d843/rbd_data.10b06b8b4567.0000000000002b59/head//3,1bd843/rbd_data.10916b8b4567.00000000000026ec/head//3,2c4fd843/rbd_data.108f6b8b4567.0000000000002f8c/head//3,f66fd843/rbd_data.108e6b8b4567.0000000000000dfd/head//3,8f92e843/rbd_data.108e6b8b4567.0000000000000e2e/head//3,f73e843/rbd_data.10956b8b4567.0000000000002baa/head//3,1734e843/rbd_data.10ab6b8b4567.0000000000001a5f/head//3,5d15e843/rbd_data.10b46b8b4567.0000000000002ef8/head//3,4c06e843/rbd_data.109a6b8b4567.000000000000161f/head//3,2a9e843/rbd_data.108d6b8b4567.00000000000031ba/head//3,c41fe843/rbd_data.10a56b8b4567.00000000000024ce/head//3,eba4f843/rbd_data.10926b8b4567.0000000000001a96/head//3,7a85f843/rbd_data.10a76b8b4567.0000000000000482/head//3,dcc8f843/rbd_data.109d6b8b4567.0000000000001dea/head//3,b4f8f843/rbd_data.10aa6b8b4567.0000000000002e5d/head//3]
2014-06-12 14:51:49.032539 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  requesting scrubmap from osd.24
2014-06-12 14:51:49.032571 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:49.032602 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:49.032616 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:49.032624 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:49.032631 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:49.032638 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_map_chunk [b989c843//0//-1,3c49f843//0//-1)
2014-06-12 14:51:49.032647 7f9d60bd2700 10 collection_list_partial 3.43_head start:b989c843//0//-1 is_max:0
2014-06-12 14:51:49.032889 7f9d60bd2700 20 objects: [f6b9c843/rbd_data.109e6b8b4567.0000000000000cd6/head//3,54bbc843/rbd_data.10b36b8b4567.0000000000000710/head//3,5a3cc843/rbd_data.10ac6b8b4567.0000000000001d97/head//3,b6bcc843/rbd_data.10906b8b4567.00000000000009af/head//3,ecedc843/rbd_data.10b06b8b4567.0000000000003096/head//3,d77ec843/rbd_data.10af6b8b4567.000000000000186b/head//3,b1fc843/rbd_data.10ab6b8b4567.00000000000018f4/head//3,398fc843/rbd_data.108d6b8b4567.00000000000033d0/head//3,e213d843/rbd_data.10b06b8b4567.000000000000005e/head//3,f655d843/rbd_data.10946b8b4567.00000000000037bd/head//3,5e8d843/rbd_data.10b06b8b4567.0000000000002b59/head//3,1bd843/rbd_data.10916b8b4567.00000000000026ec/head//3,2c4fd843/rbd_data.108f6b8b4567.0000000000002f8c/head//3,f66fd843/rbd_data.108e6b8b4567.0000000000000dfd/head//3,8f92e843/rbd_data.108e6b8b4567.0000000000000e2e/head//3,f73e843/rbd_data.10956b8b4567.0000000000002baa/head//3,1734e843/rbd_data.10ab6b8b4567.0000000000001a5f/head//3,5d15e843/rbd_data.10b46b8b4567.0000000000002ef8/head//3,4c06e843/rbd_data.109a6b8b4567.000000000000161f/head//3,2a9e843/rbd_data.108d6b8b4567.00000000000031ba/head//3,c41fe843/rbd_data.10a56b8b4567.00000000000024ce/head//3,eba4f843/rbd_data.10926b8b4567.0000000000001a96/head//3,7a85f843/rbd_data.10a76b8b4567.0000000000000482/head//3,dcc8f843/rbd_data.109d6b8b4567.0000000000001dea/head//3,b4f8f843/rbd_data.10aa6b8b4567.0000000000002e5d/head//3,3c49f843/rbd_data.10a26b8b4567.00000000000015a0/head//3,1cebf843/rbd_data.10a46b8b4567.0000000000000518/head//3,59edf843/rbd_data.109c6b8b4567.0000000000001efa/head//3,551ef843/rbd_data.10ab6b8b4567.0000000000003573/head//3,f02ef843/rbd_data.10b26b8b4567.0000000000003567/head//3]
2014-06-12 14:51:49.032960 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:49.032978 7f9d60bd2700 10 stat 3.43_head/f6b9c843/rbd_data.109e6b8b4567.0000000000000cd6/head//3
2014-06-12 14:51:49.033129 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f6b9c843/rbd_data.109e6b8b4567.0000000000000cd6/head//3
2014-06-12 14:51:49.079976 7f9d60bd2700 10 getattrs 3.43_head/f6b9c843/rbd_data.109e6b8b4567.0000000000000cd6/head//3 = 0
2014-06-12 14:51:49.079986 7f9d60bd2700 10 stat 3.43_head/54bbc843/rbd_data.10b36b8b4567.0000000000000710/head//3
2014-06-12 14:51:49.080088 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 54bbc843/rbd_data.10b36b8b4567.0000000000000710/head//3
2014-06-12 14:51:49.093151 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:49.093170 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:49.093709 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:49.093754 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:49.093776 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:49.093797 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:49.093833 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:49.093861 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:49.093913 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:49.093927 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:49.094117 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:49.094233 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:49.094254 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:49.094269 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:49.094286 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:49.094302 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:49.094317 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:49.094329 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:49.094348 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:49.094384 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:49.094408 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:49.094434 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:49.094457 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:49.094484 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:49.121847 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:49.127101 7f9d60bd2700 10 getattrs 3.43_head/54bbc843/rbd_data.10b36b8b4567.0000000000000710/head//3 = 0
2014-06-12 14:51:49.127111 7f9d60bd2700 10 stat 3.43_head/5a3cc843/rbd_data.10ac6b8b4567.0000000000001d97/head//3
2014-06-12 14:51:49.127221 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 5a3cc843/rbd_data.10ac6b8b4567.0000000000001d97/head//3
2014-06-12 14:51:49.133436 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:49.142841 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:49.152273 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:49.159395 7f9d60bd2700 10 getattrs 3.43_head/5a3cc843/rbd_data.10ac6b8b4567.0000000000001d97/head//3 = 0
2014-06-12 14:51:49.159406 7f9d60bd2700 10 stat 3.43_head/b6bcc843/rbd_data.10906b8b4567.00000000000009af/head//3
2014-06-12 14:51:49.159523 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid b6bcc843/rbd_data.10906b8b4567.00000000000009af/head//3
2014-06-12 14:51:49.180755 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:49.193693 7f9d60bd2700 10 getattrs 3.43_head/b6bcc843/rbd_data.10906b8b4567.00000000000009af/head//3 = 0
2014-06-12 14:51:49.193712 7f9d60bd2700 10 stat 3.43_head/ecedc843/rbd_data.10b06b8b4567.0000000000003096/head//3
2014-06-12 14:51:49.193832 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid ecedc843/rbd_data.10b06b8b4567.0000000000003096/head//3
2014-06-12 14:51:49.193924 7f9d60bd2700 10 getattrs 3.43_head/ecedc843/rbd_data.10b06b8b4567.0000000000003096/head//3 = 0
2014-06-12 14:51:49.193931 7f9d60bd2700 10 stat 3.43_head/d77ec843/rbd_data.10af6b8b4567.000000000000186b/head//3
2014-06-12 14:51:49.194036 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid d77ec843/rbd_data.10af6b8b4567.000000000000186b/head//3
2014-06-12 14:51:49.216378 7f9d60bd2700 10 getattrs 3.43_head/d77ec843/rbd_data.10af6b8b4567.000000000000186b/head//3 = 0
2014-06-12 14:51:49.216384 7f9d60bd2700 10 stat 3.43_head/b1fc843/rbd_data.10ab6b8b4567.00000000000018f4/head//3
2014-06-12 14:51:49.216483 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid b1fc843/rbd_data.10ab6b8b4567.00000000000018f4/head//3
2014-06-12 14:51:49.230786 7f9d60bd2700 10 getattrs 3.43_head/b1fc843/rbd_data.10ab6b8b4567.00000000000018f4/head//3 = 0
2014-06-12 14:51:49.230793 7f9d60bd2700 10 stat 3.43_head/398fc843/rbd_data.108d6b8b4567.00000000000033d0/head//3
2014-06-12 14:51:49.230890 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 398fc843/rbd_data.108d6b8b4567.00000000000033d0/head//3
2014-06-12 14:51:49.249086 7f9d60bd2700 10 getattrs 3.43_head/398fc843/rbd_data.108d6b8b4567.00000000000033d0/head//3 = 0
2014-06-12 14:51:49.249093 7f9d60bd2700 10 stat 3.43_head/e213d843/rbd_data.10b06b8b4567.000000000000005e/head//3
2014-06-12 14:51:49.249187 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid e213d843/rbd_data.10b06b8b4567.000000000000005e/head//3
2014-06-12 14:51:49.277713 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:49.279227 7f9d60bd2700 10 getattrs 3.43_head/e213d843/rbd_data.10b06b8b4567.000000000000005e/head//3 = 0
2014-06-12 14:51:49.279237 7f9d60bd2700 10 stat 3.43_head/f655d843/rbd_data.10946b8b4567.00000000000037bd/head//3
2014-06-12 14:51:49.279332 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f655d843/rbd_data.10946b8b4567.00000000000037bd/head//3
2014-06-12 14:51:49.279612 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:49.279658 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.51 >= max 0.5 = no, load too high
2014-06-12 14:51:49.279672 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:51:49.279683 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:49.279676 7f9d84175700 10 osd.2 173 sched_scrub 3.43 high load at 2014-06-11 13:02:32.501298: 92956.8 < max (604800 seconds)
2014-06-12 14:51:49.279696 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:49.279698 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:49.279700 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:49.298915 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:49.309197 7f9d60bd2700 10 getattrs 3.43_head/f655d843/rbd_data.10946b8b4567.00000000000037bd/head//3 = 0
2014-06-12 14:51:49.309207 7f9d60bd2700 10 stat 3.43_head/5e8d843/rbd_data.10b06b8b4567.0000000000002b59/head//3
2014-06-12 14:51:49.309318 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 5e8d843/rbd_data.10b06b8b4567.0000000000002b59/head//3
2014-06-12 14:51:49.309418 7f9d60bd2700 10 getattrs 3.43_head/5e8d843/rbd_data.10b06b8b4567.0000000000002b59/head//3 = 0
2014-06-12 14:51:49.309425 7f9d60bd2700 10 stat 3.43_head/1bd843/rbd_data.10916b8b4567.00000000000026ec/head//3
2014-06-12 14:51:49.309521 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 1bd843/rbd_data.10916b8b4567.00000000000026ec/head//3
2014-06-12 14:51:49.317011 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:49.324957 7f9d60bd2700 10 getattrs 3.43_head/1bd843/rbd_data.10916b8b4567.00000000000026ec/head//3 = 0
2014-06-12 14:51:49.324966 7f9d60bd2700 10 stat 3.43_head/2c4fd843/rbd_data.108f6b8b4567.0000000000002f8c/head//3
2014-06-12 14:51:49.325078 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 2c4fd843/rbd_data.108f6b8b4567.0000000000002f8c/head//3
2014-06-12 14:51:49.336399 7f9d60bd2700 10 getattrs 3.43_head/2c4fd843/rbd_data.108f6b8b4567.0000000000002f8c/head//3 = 0
2014-06-12 14:51:49.336409 7f9d60bd2700 10 stat 3.43_head/f66fd843/rbd_data.108e6b8b4567.0000000000000dfd/head//3
2014-06-12 14:51:49.336524 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f66fd843/rbd_data.108e6b8b4567.0000000000000dfd/head//3
2014-06-12 14:51:49.342054 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:49.361407 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:49.361447 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:49.375378 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:49.382820 7f9d60bd2700 10 getattrs 3.43_head/f66fd843/rbd_data.108e6b8b4567.0000000000000dfd/head//3 = 0
2014-06-12 14:51:49.382841 7f9d60bd2700 10 stat 3.43_head/8f92e843/rbd_data.108e6b8b4567.0000000000000e2e/head//3
2014-06-12 14:51:49.382954 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 8f92e843/rbd_data.108e6b8b4567.0000000000000e2e/head//3
2014-06-12 14:51:49.414624 7f9d60bd2700 10 getattrs 3.43_head/8f92e843/rbd_data.108e6b8b4567.0000000000000e2e/head//3 = 0
2014-06-12 14:51:49.414635 7f9d60bd2700 10 stat 3.43_head/f73e843/rbd_data.10956b8b4567.0000000000002baa/head//3
2014-06-12 14:51:49.414743 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f73e843/rbd_data.10956b8b4567.0000000000002baa/head//3
2014-06-12 14:51:49.427756 7f9d60bd2700 10 getattrs 3.43_head/f73e843/rbd_data.10956b8b4567.0000000000002baa/head//3 = 0
2014-06-12 14:51:49.427766 7f9d60bd2700 10 stat 3.43_head/1734e843/rbd_data.10ab6b8b4567.0000000000001a5f/head//3
2014-06-12 14:51:49.427878 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 1734e843/rbd_data.10ab6b8b4567.0000000000001a5f/head//3
2014-06-12 14:51:49.455787 7f9d60bd2700 10 getattrs 3.43_head/1734e843/rbd_data.10ab6b8b4567.0000000000001a5f/head//3 = 0
2014-06-12 14:51:49.455808 7f9d60bd2700 10 stat 3.43_head/5d15e843/rbd_data.10b46b8b4567.0000000000002ef8/head//3
2014-06-12 14:51:49.455927 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 5d15e843/rbd_data.10b46b8b4567.0000000000002ef8/head//3
2014-06-12 14:51:49.471666 7f9d60bd2700 10 getattrs 3.43_head/5d15e843/rbd_data.10b46b8b4567.0000000000002ef8/head//3 = 0
2014-06-12 14:51:49.471679 7f9d60bd2700 10 stat 3.43_head/4c06e843/rbd_data.109a6b8b4567.000000000000161f/head//3
2014-06-12 14:51:49.471789 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 4c06e843/rbd_data.109a6b8b4567.000000000000161f/head//3
2014-06-12 14:51:49.478980 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:49.479017 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:49.482982 7f9d60bd2700 10 getattrs 3.43_head/4c06e843/rbd_data.109a6b8b4567.000000000000161f/head//3 = 0
2014-06-12 14:51:49.482992 7f9d60bd2700 10 stat 3.43_head/2a9e843/rbd_data.108d6b8b4567.00000000000031ba/head//3
2014-06-12 14:51:49.483110 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 2a9e843/rbd_data.108d6b8b4567.00000000000031ba/head//3
2014-06-12 14:51:49.513869 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:49.518912 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:49.522149 7f9d60bd2700 10 getattrs 3.43_head/2a9e843/rbd_data.108d6b8b4567.00000000000031ba/head//3 = 0
2014-06-12 14:51:49.522159 7f9d60bd2700 10 stat 3.43_head/c41fe843/rbd_data.10a56b8b4567.00000000000024ce/head//3
2014-06-12 14:51:49.522269 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid c41fe843/rbd_data.10a56b8b4567.00000000000024ce/head//3
2014-06-12 14:51:49.544518 7f9d60bd2700 10 getattrs 3.43_head/c41fe843/rbd_data.10a56b8b4567.00000000000024ce/head//3 = 0
2014-06-12 14:51:49.544530 7f9d60bd2700 10 stat 3.43_head/eba4f843/rbd_data.10926b8b4567.0000000000001a96/head//3
2014-06-12 14:51:49.544626 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid eba4f843/rbd_data.10926b8b4567.0000000000001a96/head//3
2014-06-12 14:51:49.565302 7f9d60bd2700 10 getattrs 3.43_head/eba4f843/rbd_data.10926b8b4567.0000000000001a96/head//3 = 0
2014-06-12 14:51:49.565310 7f9d60bd2700 10 stat 3.43_head/7a85f843/rbd_data.10a76b8b4567.0000000000000482/head//3
2014-06-12 14:51:49.565401 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 7a85f843/rbd_data.10a76b8b4567.0000000000000482/head//3
2014-06-12 14:51:49.598352 7f9d60bd2700 10 getattrs 3.43_head/7a85f843/rbd_data.10a76b8b4567.0000000000000482/head//3 = 0
2014-06-12 14:51:49.598368 7f9d60bd2700 10 stat 3.43_head/dcc8f843/rbd_data.109d6b8b4567.0000000000001dea/head//3
2014-06-12 14:51:49.598493 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid dcc8f843/rbd_data.109d6b8b4567.0000000000001dea/head//3
2014-06-12 14:51:49.608509 7f9d5a16f700  5 -- op tracker -- , seq: 748605, time: 2014-06-12 14:51:49.608380, event: header_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.608532 7f9d5a16f700  5 -- op tracker -- , seq: 748605, time: 2014-06-12 14:51:49.608385, event: throttled, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.608575 7f9d5a16f700  5 -- op tracker -- , seq: 748605, time: 2014-06-12 14:51:49.608452, event: all_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.608599 7f9d5a16f700  5 -- op tracker -- , seq: 748605, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.608605 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:49.608614 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:51:49.608625 7f9d5a16f700 15 osd.2 173 enqueue_op 0x124e2d00 prio 127 cost 10717 latency 0.000245 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:49.608697 7f9d643d9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:49.624939 7f9d60bd2700 10 getattrs 3.43_head/dcc8f843/rbd_data.109d6b8b4567.0000000000001dea/head//3 = 0
2014-06-12 14:51:49.624954 7f9d60bd2700 10 stat 3.43_head/b4f8f843/rbd_data.10aa6b8b4567.0000000000002e5d/head//3
2014-06-12 14:51:49.625063 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid b4f8f843/rbd_data.10aa6b8b4567.0000000000002e5d/head//3
2014-06-12 14:51:49.640485 7f9d60bd2700 10 getattrs 3.43_head/b4f8f843/rbd_data.10aa6b8b4567.0000000000002e5d/head//3 = 0
2014-06-12 14:51:49.640496 7f9d60bd2700 10 collection_getattrs 3.43_head
2014-06-12 14:51:49.640561 7f9d60bd2700 10 collection_getattrs 3.43_head = 0
2014-06-12 14:51:49.640566 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:49.640605 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:49.640614 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:49.640624 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:49.640627 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:49.640659 7f9d643d9700 10 osd.2 173 dequeue_op 0x124e2d00 prio 127 cost 10717 latency 0.032279 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]
2014-06-12 14:51:49.640704 7f9d643d9700  5 -- op tracker -- , seq: 748605, time: 2014-06-12 14:51:49.640697, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.640726 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] handle_message: 0x124e2d00
2014-06-12 14:51:49.640757 7f9d643d9700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:49.640776 7f9d643d9700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:49.640794 7f9d643d9700  5 -- op tracker -- , seq: 748605, time: 2014-06-12 14:51:49.640788, event: started, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.640808 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  got 24 scrub map
2014-06-12 14:51:49.640913 7f9d643d9700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] map version is 173'2566
2014-06-12 14:51:49.640933 7f9d643d9700 10 osd.2 173 dequeue_op 0x124e2d00 finish
2014-06-12 14:51:49.640944 7f9d643d9700  5 -- op tracker -- , seq: 748605, time: 2014-06-12 14:51:49.640938, event: done, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.640961 7f9d643d9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:49.640965 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:49.640973 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:49.640986 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:49.641008 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:49.641026 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:49.641053 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:49.641064 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:49.641072 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub replica 24 has 25 items
2014-06-12 14:51:49.641094 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f6b9c843/rbd_data.109e6b8b4567.0000000000000cd6/head//3, auth == maps.end()
2014-06-12 14:51:49.641116 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f6b9c843/rbd_data.109e6b8b4567.0000000000000cd6/head//3
2014-06-12 14:51:49.641132 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 54bbc843/rbd_data.10b36b8b4567.0000000000000710/head//3, auth == maps.end()
2014-06-12 14:51:49.641144 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 54bbc843/rbd_data.10b36b8b4567.0000000000000710/head//3
2014-06-12 14:51:49.641157 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5a3cc843/rbd_data.10ac6b8b4567.0000000000001d97/head//3, auth == maps.end()
2014-06-12 14:51:49.641169 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 5a3cc843/rbd_data.10ac6b8b4567.0000000000001d97/head//3
2014-06-12 14:51:49.641181 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b6bcc843/rbd_data.10906b8b4567.00000000000009af/head//3, auth == maps.end()
2014-06-12 14:51:49.641192 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj b6bcc843/rbd_data.10906b8b4567.00000000000009af/head//3
2014-06-12 14:51:49.641204 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ecedc843/rbd_data.10b06b8b4567.0000000000003096/head//3, auth == maps.end()
2014-06-12 14:51:49.641215 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj ecedc843/rbd_data.10b06b8b4567.0000000000003096/head//3
2014-06-12 14:51:49.641227 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d77ec843/rbd_data.10af6b8b4567.000000000000186b/head//3, auth == maps.end()
2014-06-12 14:51:49.641239 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj d77ec843/rbd_data.10af6b8b4567.000000000000186b/head//3
2014-06-12 14:51:49.641251 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b1fc843/rbd_data.10ab6b8b4567.00000000000018f4/head//3, auth == maps.end()
2014-06-12 14:51:49.641262 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj b1fc843/rbd_data.10ab6b8b4567.00000000000018f4/head//3
2014-06-12 14:51:49.641275 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 398fc843/rbd_data.108d6b8b4567.00000000000033d0/head//3, auth == maps.end()
2014-06-12 14:51:49.641286 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 398fc843/rbd_data.108d6b8b4567.00000000000033d0/head//3
2014-06-12 14:51:49.641299 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e213d843/rbd_data.10b06b8b4567.000000000000005e/head//3, auth == maps.end()
2014-06-12 14:51:49.641310 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj e213d843/rbd_data.10b06b8b4567.000000000000005e/head//3
2014-06-12 14:51:49.641323 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f655d843/rbd_data.10946b8b4567.00000000000037bd/head//3, auth == maps.end()
2014-06-12 14:51:49.641333 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f655d843/rbd_data.10946b8b4567.00000000000037bd/head//3
2014-06-12 14:51:49.641345 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5e8d843/rbd_data.10b06b8b4567.0000000000002b59/head//3, auth == maps.end()
2014-06-12 14:51:49.641357 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 5e8d843/rbd_data.10b06b8b4567.0000000000002b59/head//3
2014-06-12 14:51:49.641369 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1bd843/rbd_data.10916b8b4567.00000000000026ec/head//3, auth == maps.end()
2014-06-12 14:51:49.641381 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 1bd843/rbd_data.10916b8b4567.00000000000026ec/head//3
2014-06-12 14:51:49.641393 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2c4fd843/rbd_data.108f6b8b4567.0000000000002f8c/head//3, auth == maps.end()
2014-06-12 14:51:49.641405 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 2c4fd843/rbd_data.108f6b8b4567.0000000000002f8c/head//3
2014-06-12 14:51:49.641417 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f66fd843/rbd_data.108e6b8b4567.0000000000000dfd/head//3, auth == maps.end()
2014-06-12 14:51:49.641428 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f66fd843/rbd_data.108e6b8b4567.0000000000000dfd/head//3
2014-06-12 14:51:49.641441 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8f92e843/rbd_data.108e6b8b4567.0000000000000e2e/head//3, auth == maps.end()
2014-06-12 14:51:49.641452 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 8f92e843/rbd_data.108e6b8b4567.0000000000000e2e/head//3
2014-06-12 14:51:49.641465 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f73e843/rbd_data.10956b8b4567.0000000000002baa/head//3, auth == maps.end()
2014-06-12 14:51:49.641475 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f73e843/rbd_data.10956b8b4567.0000000000002baa/head//3
2014-06-12 14:51:49.641486 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1734e843/rbd_data.10ab6b8b4567.0000000000001a5f/head//3, auth == maps.end()
2014-06-12 14:51:49.641498 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 1734e843/rbd_data.10ab6b8b4567.0000000000001a5f/head//3
2014-06-12 14:51:49.641510 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5d15e843/rbd_data.10b46b8b4567.0000000000002ef8/head//3, auth == maps.end()
2014-06-12 14:51:49.641522 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 5d15e843/rbd_data.10b46b8b4567.0000000000002ef8/head//3
2014-06-12 14:51:49.641534 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4c06e843/rbd_data.109a6b8b4567.000000000000161f/head//3, auth == maps.end()
2014-06-12 14:51:49.641545 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 4c06e843/rbd_data.109a6b8b4567.000000000000161f/head//3
2014-06-12 14:51:49.641558 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2a9e843/rbd_data.108d6b8b4567.00000000000031ba/head//3, auth == maps.end()
2014-06-12 14:51:49.641569 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 2a9e843/rbd_data.108d6b8b4567.00000000000031ba/head//3
2014-06-12 14:51:49.641581 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c41fe843/rbd_data.10a56b8b4567.00000000000024ce/head//3, auth == maps.end()
2014-06-12 14:51:49.641593 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj c41fe843/rbd_data.10a56b8b4567.00000000000024ce/head//3
2014-06-12 14:51:49.641605 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj eba4f843/rbd_data.10926b8b4567.0000000000001a96/head//3, auth == maps.end()
2014-06-12 14:51:49.641629 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj eba4f843/rbd_data.10926b8b4567.0000000000001a96/head//3
2014-06-12 14:51:49.641640 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7a85f843/rbd_data.10a76b8b4567.0000000000000482/head//3, auth == maps.end()
2014-06-12 14:51:49.641652 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 7a85f843/rbd_data.10a76b8b4567.0000000000000482/head//3
2014-06-12 14:51:49.641663 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dcc8f843/rbd_data.109d6b8b4567.0000000000001dea/head//3, auth == maps.end()
2014-06-12 14:51:49.641672 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj dcc8f843/rbd_data.109d6b8b4567.0000000000001dea/head//3
2014-06-12 14:51:49.641682 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b4f8f843/rbd_data.10aa6b8b4567.0000000000002e5d/head//3, auth == maps.end()
2014-06-12 14:51:49.641692 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj b4f8f843/rbd_data.10aa6b8b4567.0000000000002e5d/head//3
2014-06-12 14:51:49.641703 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] 
2014-06-12 14:51:49.641711 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub
2014-06-12 14:51:49.641725 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  b4f8f843/rbd_data.10aa6b8b4567.0000000000002e5d/head//3 b4f8f843/rbd_data.10aa6b8b4567.0000000000002e5d/head//3(173'1592 client.4307.0:95362 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1592)
2014-06-12 14:51:49.641742 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  dcc8f843/rbd_data.109d6b8b4567.0000000000001dea/head//3 dcc8f843/rbd_data.109d6b8b4567.0000000000001dea/head//3(173'1120 client.4294.0:63263 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1120)
2014-06-12 14:51:49.641757 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  7a85f843/rbd_data.10a76b8b4567.0000000000000482/head//3 7a85f843/rbd_data.10a76b8b4567.0000000000000482/head//3(173'2157 client.4304.0:123695 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2157)
2014-06-12 14:51:49.641770 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  eba4f843/rbd_data.10926b8b4567.0000000000001a96/head//3 eba4f843/rbd_data.10926b8b4567.0000000000001a96/head//3(173'981 client.4283.0:56807 wrlock_by=unknown.0.0:0 dirty s 4194304 uv981)
2014-06-12 14:51:49.641784 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  c41fe843/rbd_data.10a56b8b4567.00000000000024ce/head//3 c41fe843/rbd_data.10a56b8b4567.00000000000024ce/head//3(173'1319 client.4302.0:76407 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1319)
2014-06-12 14:51:49.641798 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  2a9e843/rbd_data.108d6b8b4567.00000000000031ba/head//3 2a9e843/rbd_data.108d6b8b4567.00000000000031ba/head//3(173'1662 client.4278.0:100067 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1662)
2014-06-12 14:51:49.641812 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  4c06e843/rbd_data.109a6b8b4567.000000000000161f/head//3 4c06e843/rbd_data.109a6b8b4567.000000000000161f/head//3(173'906 client.4291.0:48948 wrlock_by=unknown.0.0:0 dirty s 4194304 uv906)
2014-06-12 14:51:49.641826 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  5d15e843/rbd_data.10b46b8b4567.0000000000002ef8/head//3 5d15e843/rbd_data.10b46b8b4567.0000000000002ef8/head//3(173'1782 client.4317.0:94371 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1782)
2014-06-12 14:51:49.641841 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  1734e843/rbd_data.10ab6b8b4567.0000000000001a5f/head//3 1734e843/rbd_data.10ab6b8b4567.0000000000001a5f/head//3(173'1036 client.4308.0:57807 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1036)
2014-06-12 14:51:49.641855 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f73e843/rbd_data.10956b8b4567.0000000000002baa/head//3 f73e843/rbd_data.10956b8b4567.0000000000002baa/head//3(173'1512 client.4286.0:88568 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1512)
2014-06-12 14:51:49.641869 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  8f92e843/rbd_data.108e6b8b4567.0000000000000e2e/head//3 8f92e843/rbd_data.108e6b8b4567.0000000000000e2e/head//3(173'2482 client.4279.0:140242 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2482)
2014-06-12 14:51:49.641884 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f66fd843/rbd_data.108e6b8b4567.0000000000000dfd/head//3 f66fd843/rbd_data.108e6b8b4567.0000000000000dfd/head//3(173'2473 client.4279.0:139911 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2473)
2014-06-12 14:51:49.641898 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  2c4fd843/rbd_data.108f6b8b4567.0000000000002f8c/head//3 2c4fd843/rbd_data.108f6b8b4567.0000000000002f8c/head//3(173'1640 client.4280.0:93591 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1640)
2014-06-12 14:51:49.641913 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  1bd843/rbd_data.10916b8b4567.00000000000026ec/head//3 1bd843/rbd_data.10916b8b4567.00000000000026ec/head//3(173'1378 client.4282.0:79909 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1378)
2014-06-12 14:51:49.641928 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  5e8d843/rbd_data.10b06b8b4567.0000000000002b59/head//3 5e8d843/rbd_data.10b06b8b4567.0000000000002b59/head//3(173'1455 client.4313.0:88628 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1455)
2014-06-12 14:51:49.641943 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f655d843/rbd_data.10946b8b4567.00000000000037bd/head//3 f655d843/rbd_data.10946b8b4567.00000000000037bd/head//3(173'1904 client.4285.0:108812 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1904)
2014-06-12 14:51:49.641957 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  e213d843/rbd_data.10b06b8b4567.000000000000005e/head//3 e213d843/rbd_data.10b06b8b4567.000000000000005e/head//3(173'2083 client.4313.0:120263 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2083)
2014-06-12 14:51:49.641971 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  398fc843/rbd_data.108d6b8b4567.00000000000033d0/head//3 398fc843/rbd_data.108d6b8b4567.00000000000033d0/head//3(173'1743 client.4278.0:103965 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1743)
2014-06-12 14:51:49.641985 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  b1fc843/rbd_data.10ab6b8b4567.00000000000018f4/head//3 b1fc843/rbd_data.10ab6b8b4567.00000000000018f4/head//3(173'961 client.4308.0:55218 wrlock_by=unknown.0.0:0 dirty s 4194304 uv961)
2014-06-12 14:51:49.642000 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  d77ec843/rbd_data.10af6b8b4567.000000000000186b/head//3 d77ec843/rbd_data.10af6b8b4567.000000000000186b/head//3(173'989 client.4312.0:56246 wrlock_by=unknown.0.0:0 dirty s 4194304 uv989)
2014-06-12 14:51:49.642014 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  ecedc843/rbd_data.10b06b8b4567.0000000000003096/head//3 ecedc843/rbd_data.10b06b8b4567.0000000000003096/head//3(173'1652 client.4313.0:98484 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1652)
2014-06-12 14:51:49.642028 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  b6bcc843/rbd_data.10906b8b4567.00000000000009af/head//3 b6bcc843/rbd_data.10906b8b4567.00000000000009af/head//3(173'2305 client.4281.0:131760 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2305)
2014-06-12 14:51:49.642040 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  5a3cc843/rbd_data.10ac6b8b4567.0000000000001d97/head//3 5a3cc843/rbd_data.10ac6b8b4567.0000000000001d97/head//3(173'1155 client.4309.0:61561 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1155)
2014-06-12 14:51:49.642054 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  54bbc843/rbd_data.10b36b8b4567.0000000000000710/head//3 54bbc843/rbd_data.10b36b8b4567.0000000000000710/head//3(173'2233 client.4316.0:130170 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2233)
2014-06-12 14:51:49.642068 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f6b9c843/rbd_data.109e6b8b4567.0000000000000cd6/head//3 f6b9c843/rbd_data.109e6b8b4567.0000000000000cd6/head//3(173'2449 client.4295.0:138484 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2449)
2014-06-12 14:51:49.642080 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:49.642091 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:49.642100 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:49.642103 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:49.642104 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:49.642142 7f9d60bd2700 10 collection_list_partial 3.43_head start:3c49f843//0//-1 is_max:0
2014-06-12 14:51:49.642247 7f9d60bd2700 20 objects: [1cebf843/rbd_data.10a46b8b4567.0000000000000518/head//3,59edf843/rbd_data.109c6b8b4567.0000000000001efa/head//3,551ef843/rbd_data.10ab6b8b4567.0000000000003573/head//3,f02ef843/rbd_data.10b26b8b4567.0000000000003567/head//3]
2014-06-12 14:51:49.642630 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  requesting scrubmap from osd.24
2014-06-12 14:51:49.642654 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:49.642671 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:49.642681 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:49.642687 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:49.642694 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:49.642700 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_map_chunk [3c49f843//0//-1,MAX)
2014-06-12 14:51:49.642708 7f9d60bd2700 10 collection_list_partial 3.43_head start:3c49f843//0//-1 is_max:0
2014-06-12 14:51:49.642773 7f9d60bd2700 20 objects: [1cebf843/rbd_data.10a46b8b4567.0000000000000518/head//3,59edf843/rbd_data.109c6b8b4567.0000000000001efa/head//3,551ef843/rbd_data.10ab6b8b4567.0000000000003573/head//3,f02ef843/rbd_data.10b26b8b4567.0000000000003567/head//3]
2014-06-12 14:51:49.642787 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scan_list scanning 4 objects
2014-06-12 14:51:49.642798 7f9d60bd2700 10 stat 3.43_head/1cebf843/rbd_data.10a46b8b4567.0000000000000518/head//3
2014-06-12 14:51:49.642899 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 1cebf843/rbd_data.10a46b8b4567.0000000000000518/head//3
2014-06-12 14:51:49.660953 7f9d60bd2700 10 getattrs 3.43_head/1cebf843/rbd_data.10a46b8b4567.0000000000000518/head//3 = 0
2014-06-12 14:51:49.660960 7f9d60bd2700 10 stat 3.43_head/59edf843/rbd_data.109c6b8b4567.0000000000001efa/head//3
2014-06-12 14:51:49.661051 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 59edf843/rbd_data.109c6b8b4567.0000000000001efa/head//3
2014-06-12 14:51:49.677439 7f9d60bd2700 10 getattrs 3.43_head/59edf843/rbd_data.109c6b8b4567.0000000000001efa/head//3 = 0
2014-06-12 14:51:49.677446 7f9d60bd2700 10 stat 3.43_head/551ef843/rbd_data.10ab6b8b4567.0000000000003573/head//3
2014-06-12 14:51:49.677552 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid 551ef843/rbd_data.10ab6b8b4567.0000000000003573/head//3
2014-06-12 14:51:49.698437 7f9d60bd2700 10 getattrs 3.43_head/551ef843/rbd_data.10ab6b8b4567.0000000000003573/head//3 = 0
2014-06-12 14:51:49.698456 7f9d60bd2700 10 stat 3.43_head/f02ef843/rbd_data.10b26b8b4567.0000000000003567/head//3
2014-06-12 14:51:49.698643 7f9d60bd2700 10 lookup_strip_header done  cid 3.43_head oid f02ef843/rbd_data.10b26b8b4567.0000000000003567/head//3
2014-06-12 14:51:49.702947 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:49.703267 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:49.716414 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:49.716506 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:49.726019 7f9d60bd2700 10 getattrs 3.43_head/f02ef843/rbd_data.10b26b8b4567.0000000000003567/head//3 = 0
2014-06-12 14:51:49.726037 7f9d60bd2700 10 collection_getattrs 3.43_head
2014-06-12 14:51:49.726151 7f9d60bd2700 10 collection_getattrs 3.43_head = 0
2014-06-12 14:51:49.726160 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:49.726234 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:49.726258 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:49.726294 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:49.726301 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:49.727076 7f9d5a16f700  5 -- op tracker -- , seq: 748606, time: 2014-06-12 14:51:49.726976, event: header_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.727095 7f9d5a16f700  5 -- op tracker -- , seq: 748606, time: 2014-06-12 14:51:49.726979, event: throttled, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.727106 7f9d5a16f700  5 -- op tracker -- , seq: 748606, time: 2014-06-12 14:51:49.727029, event: all_read, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.727146 7f9d5a16f700  5 -- op tracker -- , seq: 748606, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.727154 7f9d5a16f700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:49.727162 7f9d5a16f700 20 osd.2 173 should_share_map osd.24 172.16.96.202:6827/2720 173
2014-06-12 14:51:49.727172 7f9d5a16f700 15 osd.2 173 enqueue_op 0x68a6c00 prio 127 cost 1750 latency 0.000196 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:49.727250 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:49.727270 7f9d61bd4700 10 osd.2 173 dequeue_op 0x68a6c00 prio 127 cost 1750 latency 0.000294 osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]
2014-06-12 14:51:49.727310 7f9d61bd4700  5 -- op tracker -- , seq: 748606, time: 2014-06-12 14:51:49.727304, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.727327 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] handle_message: 0x68a6c00
2014-06-12 14:51:49.727343 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:49.727359 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:49.727377 7f9d61bd4700  5 -- op tracker -- , seq: 748606, time: 2014-06-12 14:51:49.727371, event: started, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.727409 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  got 24 scrub map
2014-06-12 14:51:49.727444 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] map version is 173'2566
2014-06-12 14:51:49.727466 7f9d61bd4700 10 osd.2 173 dequeue_op 0x68a6c00 finish
2014-06-12 14:51:49.727476 7f9d61bd4700  5 -- op tracker -- , seq: 748606, time: 2014-06-12 14:51:49.727470, event: done, op: osd_sub_op(unknown.0.0:0 3.43 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:49.727475 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2dfa000 (1 active)
2014-06-12 14:51:49.727492 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:49.727496 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:49.727490 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:49.727520 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:49.727541 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:49.727566 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:49.727592 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub   osd.2 has 4 items
2014-06-12 14:51:49.727614 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub replica 24 has 4 items
2014-06-12 14:51:49.727644 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1cebf843/rbd_data.10a46b8b4567.0000000000000518/head//3, auth == maps.end()
2014-06-12 14:51:49.727680 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 1cebf843/rbd_data.10a46b8b4567.0000000000000518/head//3
2014-06-12 14:51:49.727719 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 59edf843/rbd_data.109c6b8b4567.0000000000001efa/head//3, auth == maps.end()
2014-06-12 14:51:49.727750 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 59edf843/rbd_data.109c6b8b4567.0000000000001efa/head//3
2014-06-12 14:51:49.727782 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 551ef843/rbd_data.10ab6b8b4567.0000000000003573/head//3, auth == maps.end()
2014-06-12 14:51:49.727824 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj 551ef843/rbd_data.10ab6b8b4567.0000000000003573/head//3
2014-06-12 14:51:49.727861 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f02ef843/rbd_data.10b26b8b4567.0000000000003567/head//3, auth == maps.end()
2014-06-12 14:51:49.727896 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] be_select_auth_object: selecting osd 24 for obj f02ef843/rbd_data.10b26b8b4567.0000000000003567/head//3
2014-06-12 14:51:49.727930 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] 
2014-06-12 14:51:49.727956 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub
2014-06-12 14:51:49.727995 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  f02ef843/rbd_data.10b26b8b4567.0000000000003567/head//3 f02ef843/rbd_data.10b26b8b4567.0000000000003567/head//3(173'1750 client.4315.0:106122 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1750)
2014-06-12 14:51:49.728038 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  551ef843/rbd_data.10ab6b8b4567.0000000000003573/head//3 551ef843/rbd_data.10ab6b8b4567.0000000000003573/head//3(173'1843 client.4308.0:106776 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1843)
2014-06-12 14:51:49.728082 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  59edf843/rbd_data.109c6b8b4567.0000000000001efa/head//3 59edf843/rbd_data.109c6b8b4567.0000000000001efa/head//3(173'1248 client.4293.0:64529 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1248)
2014-06-12 14:51:49.728128 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub  1cebf843/rbd_data.10a46b8b4567.0000000000000518/head//3 1cebf843/rbd_data.10a46b8b4567.0000000000000518/head//3(173'2178 client.4301.0:126229 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2178)
2014-06-12 14:51:49.728163 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:49.728189 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:49.728210 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub state FINISH
2014-06-12 14:51:49.728235 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] scrub got 279/290 objects, 0/0 clones, 279/290 dirty, 0/0 omap, 0/0 hit_set_archive, 1170210816/1216348160 bytes.
2014-06-12 14:51:49.728279 7f9d60bd2700  0 log [ERR] : 3.43 scrub stat mismatch, got 279/290 objects, 0/0 clones, 279/290 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts, 1170210816/1216348160 bytes.
2014-06-12 14:51:49.728290 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+scrubbing] process_inconsistent() checking authoritative
2014-06-12 14:51:49.728321 7f9d60bd2700  0 log [ERR] : 3.43 scrub 1 errors
2014-06-12 14:51:49.728403 7f9d60bd2700 10 log is not dirty
2014-06-12 14:51:49.728414 7f9d60bd2700  5 queue_transactions existing osr(3.43 0x276ccd0)/0x276ccd0
2014-06-12 14:51:49.728421 7f9d60bd2700  5 queue_transactions (trailing journal)  
2014-06-12 14:51:49.728425 7f9d60bd2700  5 queue_op 0x565b920 seq 2386 osr(3.43 0x276ccd0) 826 bytes   (queue has 1 ops and 826 bytes)
2014-06-12 14:51:49.728441 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+inconsistent] publish_stats_to_osd 173:2749
2014-06-12 14:51:49.728469 7f9d60bd2700 20 osd.2 173 dec_scrubs_active 1 -> 0 (max 1, pending 0)
2014-06-12 14:51:49.728474 7f9d7c966700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x543fe60 (1 active)
2014-06-12 14:51:49.728475 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+inconsistent]  requeue_ops 
2014-06-12 14:51:49.728494 7f9d7c966700  5 _do_op 0x565b920 seq 2386 osr(3.43 0x276ccd0)/0x276ccd0 start
2014-06-12 14:51:49.728501 7f9d7c966700 10 _do_transaction on 0x2335d680
2014-06-12 14:51:49.728512 7f9d7c966700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:51:49.728503 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+inconsistent] scrub requesting unreserve from osd.24
2014-06-12 14:51:49.728540 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:49.728620 7f9d7c966700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:51:49.728607 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.43( v 173'2566 (0'0,173'2566] local-les=147 n=290 ec=23 les/c 147/147 146/146/133) [2,24] r=0 lpr=146 crt=0'0 lcod 173'2565 mlcod 173'2565 active+clean+inconsistent] share_pg_info
2014-06-12 14:51:49.728659 7f9d7c966700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:51:49.728676 7f9d7c966700  5 submit_transaction r = 0
2014-06-12 14:51:49.728680 7f9d60bd2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:49.728718 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2dfa000 (0 active)
2014-06-12 14:51:49.728724 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:49.728729 7f9d7c966700 10 _do_op 0x565b920 seq 2386 r = 0, finisher 0x57a4ab0 0
2014-06-12 14:51:49.728735 7f9d7c966700 10 _finish_op 0x565b920 seq 2386 osr(3.43 0x276ccd0)/0x276ccd0
2014-06-12 14:51:49.728747 7f9d7c966700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x543fe60 (0 active)
2014-06-12 14:51:49.728751 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:49.773061 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:49.773540 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:49.788091 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:49.788626 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:50.065094 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:50.167995 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:50.194499 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:50.273738 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:50.274006 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:50.279789 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:50.279809 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:51:50.279813 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:51:50.279818 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:51:50.279822 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:51:50.279861 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:50.279864 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:50.279866 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:50.279873 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:50.288813 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:50.289192 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:50.289601 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:50.289727 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:50.303838 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:50.303842 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:50.303845 7f9d74bfa700 20 osd.2 173 _dispatch 0x7434c40 pg_stats_ack(1 pgs tid 4873) v1
2014-06-12 14:51:50.303850 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:51:50.303858 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:50.303861 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:50.313313 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:50.326764 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:50.366829 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:50.371734 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:50.382434 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:50.399953 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:50.522577 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:50.523430 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:50.532884 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:50.570332 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:50.601027 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:50.601287 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:50.662210 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:50.680416 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:50.701074 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:50.726860 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:50.768548 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:50.802458 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:50.832309 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:50.832651 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:50.855588 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:50.865943 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:50.869301 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:50.908057 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:50.966558 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:50.995763 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:51.030631 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:51.121916 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:51.133534 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:51.142916 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:51.152355 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:51.179743 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:51.180091 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:51.180827 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:51.277788 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:51.279937 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:51.279982 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.47 < max 0.5 = yes
2014-06-12 14:51:51.279996 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=1
2014-06-12 14:51:51.279990 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:51.280002 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6b at 2014-06-11 13:02:32.502023
2014-06-12 14:51:51.280014 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 -> 1 (max 1, active 0)
2014-06-12 14:51:51.280017 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] sched_scrub: reserved locally, reserving replicas
2014-06-12 14:51:51.280038 7f9d84175700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] scrub requesting reserve from osd.29
2014-06-12 14:51:51.280058 7f9d84175700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:51.280085 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] sched_scrub: reserved 2, waiting for replicas
2014-06-12 14:51:51.280099 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:51:51.280101 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:51.280103 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:51.280933 7f9d5501e700  5 -- op tracker -- , seq: 748607, time: 2014-06-12 14:51:51.280863, event: header_read, op: osd_sub_op_reply(unknown.0.0:0 3.6b 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:51.280962 7f9d5501e700  5 -- op tracker -- , seq: 748607, time: 2014-06-12 14:51:51.280865, event: throttled, op: osd_sub_op_reply(unknown.0.0:0 3.6b 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:51.280980 7f9d5501e700  5 -- op tracker -- , seq: 748607, time: 2014-06-12 14:51:51.280885, event: all_read, op: osd_sub_op_reply(unknown.0.0:0 3.6b 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:51.280999 7f9d5501e700  5 -- op tracker -- , seq: 748607, time: 0.000000, event: dispatched, op: osd_sub_op_reply(unknown.0.0:0 3.6b 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:51.281008 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op_reply(unknown.0.0:0 3.6b 0//0//3 [scrub-reserve] ack, result = 0) v2 epoch 173
2014-06-12 14:51:51.281020 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:51.281034 7f9d5501e700 15 osd.2 173 enqueue_op 0x1e3ebf00 prio 127 cost 1 latency 0.000170 osd_sub_op_reply(unknown.0.0:0 3.6b 0//0//3 [scrub-reserve] ack, result = 0) v2
2014-06-12 14:51:51.281112 7f9d653db700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:51.281134 7f9d653db700 10 osd.2 173 dequeue_op 0x1e3ebf00 prio 127 cost 1 latency 0.000270 osd_sub_op_reply(unknown.0.0:0 3.6b 0//0//3 [scrub-reserve] ack, result = 0) v2 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean]
2014-06-12 14:51:51.281173 7f9d653db700  5 -- op tracker -- , seq: 748607, time: 2014-06-12 14:51:51.281167, event: reached_pg, op: osd_sub_op_reply(unknown.0.0:0 3.6b 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:51.281190 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] handle_message: 0x1e3ebf00
2014-06-12 14:51:51.281211 7f9d653db700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] sub_op_scrub_reserve_reply
2014-06-12 14:51:51.281229 7f9d653db700  5 -- op tracker -- , seq: 748607, time: 2014-06-12 14:51:51.281224, event: started, op: osd_sub_op_reply(unknown.0.0:0 3.6b 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:51.281237 7f9d653db700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean]  osd.29 scrub reserve = success
2014-06-12 14:51:51.281250 7f9d653db700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean] sched_scrub: success, reserved self and replicas
2014-06-12 14:51:51.281265 7f9d653db700 10 osd.2 173 dequeue_op 0x1e3ebf00 finish
2014-06-12 14:51:51.281274 7f9d653db700  5 -- op tracker -- , seq: 748607, time: 2014-06-12 14:51:51.281270, event: done, op: osd_sub_op_reply(unknown.0.0:0 3.6b 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:51:51.281278 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:51.281297 7f9d653db700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:51.281302 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:51.281297 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] starting a new chunky scrub
2014-06-12 14:51:51.281350 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state INACTIVE
2014-06-12 14:51:51.281372 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub start
2014-06-12 14:51:51.281396 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] publish_stats_to_osd 173:3787
2014-06-12 14:51:51.281419 7f9d60bd2700 20 osd.2 173 inc_scrubs_active 0 -> 1 (max 1, pending 1 -> 0)
2014-06-12 14:51:51.281426 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:51.281452 7f9d60bd2700 10 collection_list_partial 3.6b_head start:0//0//-1 is_max:0
2014-06-12 14:51:51.282933 7f9d60bd2700 20 objects: [9fc2006b/rbd_data.10946b8b4567.0000000000001b38/head//3,b073006b/rbd_data.10906b8b4567.0000000000003508/head//3,8208006b/rbd_data.10b16b8b4567.00000000000000dd/head//3,a9c9006b/rbd_data.10ad6b8b4567.000000000000186d/head//3,7a3a006b/rbd_data.10956b8b4567.000000000000083c/head//3,213106b/rbd_data.10ae6b8b4567.000000000000058e/head//3,4a4106b/rbd_data.10936b8b4567.0000000000001a48/head//3,1e6106b/rbd_data.10ab6b8b4567.0000000000000247/head//3,6547106b/rbd_data.10936b8b4567.0000000000000775/head//3,2e08106b/rbd_data.10916b8b4567.00000000000002b2/head//3,2d28106b/rbd_data.108e6b8b4567.0000000000003004/head//3,e939106b/rbd_data.10ac6b8b4567.000000000000030f/head//3,c669106b/rbd_data.10a26b8b4567.000000000000187d/head//3,4aba106b/rbd_data.10a06b8b4567.00000000000033c2/head//3,310c106b/rbd_data.10926b8b4567.000000000000137d/head//3,243d106b/rbd_data.10a76b8b4567.000000000000175a/head//3,4b50206b/rbd_data.10966b8b4567.00000000000025f2/head//3,8901206b/rbd_data.10986b8b4567.0000000000001d0a/head//3,da65206b/rbd_data.10986b8b4567.0000000000000960/head//3,195206b/rbd_data.10976b8b4567.00000000000033bb/head//3,54c5206b/rbd_data.10996b8b4567.0000000000001977/head//3,9718206b/rbd_data.10ab6b8b4567.0000000000002c5f/head//3,7099206b/rbd_data.109c6b8b4567.00000000000039c6/head//3,de50306b/rbd_data.10b16b8b4567.0000000000001ec7/head//3,6fe2306b/rbd_data.109e6b8b4567.00000000000014ca/head//3]
2014-06-12 14:51:51.283751 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  requesting scrubmap from osd.29
2014-06-12 14:51:51.283809 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:51.283849 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:51.283879 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:51.283899 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:51.283920 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:51.283939 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_map_chunk [0//0//-1,a405306b//0//-1)
2014-06-12 14:51:51.283963 7f9d60bd2700 10 collection_list_partial 3.6b_head start:0//0//-1 is_max:0
2014-06-12 14:51:51.285007 7f9d60bd2700 20 objects: [9fc2006b/rbd_data.10946b8b4567.0000000000001b38/head//3,b073006b/rbd_data.10906b8b4567.0000000000003508/head//3,8208006b/rbd_data.10b16b8b4567.00000000000000dd/head//3,a9c9006b/rbd_data.10ad6b8b4567.000000000000186d/head//3,7a3a006b/rbd_data.10956b8b4567.000000000000083c/head//3,213106b/rbd_data.10ae6b8b4567.000000000000058e/head//3,4a4106b/rbd_data.10936b8b4567.0000000000001a48/head//3,1e6106b/rbd_data.10ab6b8b4567.0000000000000247/head//3,6547106b/rbd_data.10936b8b4567.0000000000000775/head//3,2e08106b/rbd_data.10916b8b4567.00000000000002b2/head//3,2d28106b/rbd_data.108e6b8b4567.0000000000003004/head//3,e939106b/rbd_data.10ac6b8b4567.000000000000030f/head//3,c669106b/rbd_data.10a26b8b4567.000000000000187d/head//3,4aba106b/rbd_data.10a06b8b4567.00000000000033c2/head//3,310c106b/rbd_data.10926b8b4567.000000000000137d/head//3,243d106b/rbd_data.10a76b8b4567.000000000000175a/head//3,4b50206b/rbd_data.10966b8b4567.00000000000025f2/head//3,8901206b/rbd_data.10986b8b4567.0000000000001d0a/head//3,da65206b/rbd_data.10986b8b4567.0000000000000960/head//3,195206b/rbd_data.10976b8b4567.00000000000033bb/head//3,54c5206b/rbd_data.10996b8b4567.0000000000001977/head//3,9718206b/rbd_data.10ab6b8b4567.0000000000002c5f/head//3,7099206b/rbd_data.109c6b8b4567.00000000000039c6/head//3,de50306b/rbd_data.10b16b8b4567.0000000000001ec7/head//3,6fe2306b/rbd_data.109e6b8b4567.00000000000014ca/head//3,a405306b/rbd_data.10956b8b4567.0000000000001ac2/head//3,290a306b/rbd_data.109c6b8b4567.0000000000003adb/head//3,3cfa306b/rbd_data.10966b8b4567.0000000000002cf2/head//3,ab9e306b/rbd_data.10b06b8b4567.0000000000003868/head//3,2b10406b/rbd_data.10a66b8b4567.0000000000003ba1/head//3,3960406b/rbd_data.10a06b8b4567.000000000000384c/head//3,d0c1406b/rbd_data.10996b8b4567.0000000000000ed8/head//3,9612406b/rbd_data.108e6b8b4567.00000000000026f7/head//3,6342406b/rbd_data.10976b8b4567.0000000000002af1/head//3,5e33406b/rbd_data.109b6b8b4567.00000000000025d8/head//3,2893406b/rbd_data.109c6b8b4567.0000000000000f4f/head//3,8514406b/rbd_data.108f6b8b4567.0000000000000b44/head//3,b61a406b/rbd_data.10b16b8b4567.0000000000002d91/head//3,c6a406b/rbd_data.10956b8b4567.0000000000000b95/head//3,a7da406b/rbd_data.10b36b8b4567.0000000000000d72/head//3,f36e406b/rbd_data.108f6b8b4567.000000000000386f/head//3,a69f406b/rbd_data.10b46b8b4567.0000000000001e2c/head//3,b6ff406b/rbd_data.10996b8b4567.0000000000000d20/head//3,7b73506b/rbd_data.10936b8b4567.000000000000143f/head//3,7314506b/rbd_data.10b16b8b4567.0000000000002563/head//3,4b64506b/rbd_data.10976b8b4567.0000000000002964/head//3,a2e4506b/rbd_data.10a06b8b4567.0000000000001f10/head//3,e125506b/rbd_data.10a56b8b4567.000000000000342e/head//3,3319506b/rbd_data.109f6b8b4567.0000000000002d74/head//3,9b79506b/rbd_data.108e6b8b4567.00000000000017a5/head//3,5c0a506b/rbd_data.10b26b8b4567.0000000000002b39/head//3,4bfc506b/rbd_data.10956b8b4567.000000000000125e/head//3,8be0606b/rbd_data.109f6b8b4567.0000000000001d87/head//3,1265606b/rbd_data.10946b8b4567.0000000000000987/head//3,c346606b/rbd_data.10af6b8b4567.0000000000002264/head//3,4407606b/rbd_data.109a6b8b4567.0000000000000689/head//3,f6f7606b/rbd_data.10ad6b8b4567.0000000000002157/head//3,caa606b/rbd_data.10af6b8b4567.0000000000002a63/head//3,c74b606b/rbd_data.10ad6b8b4567.0000000000001dd3/head//3,7cbe606b/rbd_data.10ad6b8b4567.00000000000023aa/head//3,377f606b/rbd_data.10ad6b8b4567.0000000000000647/head//3,6af606b/rbd_data.10ac6b8b4567.0000000000001d93/head//3,d5c2706b/rbd_data.109a6b8b4567.0000000000001be5/head//3,5ea3706b/rbd_data.10916b8b4567.0000000000001ed0/head//3]
2014-06-12 14:51:51.285175 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:51.285212 7f9d60bd2700 10 stat 3.6b_head/9fc2006b/rbd_data.10946b8b4567.0000000000001b38/head//3
2014-06-12 14:51:51.285450 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9fc2006b/rbd_data.10946b8b4567.0000000000001b38/head//3
2014-06-12 14:51:51.297732 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:51.297938 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:51.298988 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:51.299768 7f9d60bd2700 10 getattrs 3.6b_head/9fc2006b/rbd_data.10946b8b4567.0000000000001b38/head//3 = 0
2014-06-12 14:51:51.299777 7f9d60bd2700 10 stat 3.6b_head/b073006b/rbd_data.10906b8b4567.0000000000003508/head//3
2014-06-12 14:51:51.299880 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid b073006b/rbd_data.10906b8b4567.0000000000003508/head//3
2014-06-12 14:51:51.323219 7f9d60bd2700 10 getattrs 3.6b_head/b073006b/rbd_data.10906b8b4567.0000000000003508/head//3 = 0
2014-06-12 14:51:51.323229 7f9d60bd2700 10 stat 3.6b_head/8208006b/rbd_data.10b16b8b4567.00000000000000dd/head//3
2014-06-12 14:51:51.323360 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 8208006b/rbd_data.10b16b8b4567.00000000000000dd/head//3
2014-06-12 14:51:51.342132 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:51.355935 7f9d60bd2700 10 getattrs 3.6b_head/8208006b/rbd_data.10b16b8b4567.00000000000000dd/head//3 = 0
2014-06-12 14:51:51.355945 7f9d60bd2700 10 stat 3.6b_head/a9c9006b/rbd_data.10ad6b8b4567.000000000000186d/head//3
2014-06-12 14:51:51.356071 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a9c9006b/rbd_data.10ad6b8b4567.000000000000186d/head//3
2014-06-12 14:51:51.375460 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:51.391363 7f9d60bd2700 10 getattrs 3.6b_head/a9c9006b/rbd_data.10ad6b8b4567.000000000000186d/head//3 = 0
2014-06-12 14:51:51.391374 7f9d60bd2700 10 stat 3.6b_head/7a3a006b/rbd_data.10956b8b4567.000000000000083c/head//3
2014-06-12 14:51:51.391503 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7a3a006b/rbd_data.10956b8b4567.000000000000083c/head//3
2014-06-12 14:51:51.422694 7f9d60bd2700 10 getattrs 3.6b_head/7a3a006b/rbd_data.10956b8b4567.000000000000083c/head//3 = 0
2014-06-12 14:51:51.422717 7f9d60bd2700 10 stat 3.6b_head/213106b/rbd_data.10ae6b8b4567.000000000000058e/head//3
2014-06-12 14:51:51.422866 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 213106b/rbd_data.10ae6b8b4567.000000000000058e/head//3
2014-06-12 14:51:51.461706 7f9d60bd2700 10 getattrs 3.6b_head/213106b/rbd_data.10ae6b8b4567.000000000000058e/head//3 = 0
2014-06-12 14:51:51.461729 7f9d60bd2700 10 stat 3.6b_head/4a4106b/rbd_data.10936b8b4567.0000000000001a48/head//3
2014-06-12 14:51:51.461846 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 4a4106b/rbd_data.10936b8b4567.0000000000001a48/head//3
2014-06-12 14:51:51.478143 7f9d60bd2700 10 getattrs 3.6b_head/4a4106b/rbd_data.10936b8b4567.0000000000001a48/head//3 = 0
2014-06-12 14:51:51.478159 7f9d60bd2700 10 stat 3.6b_head/1e6106b/rbd_data.10ab6b8b4567.0000000000000247/head//3
2014-06-12 14:51:51.478269 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1e6106b/rbd_data.10ab6b8b4567.0000000000000247/head//3
2014-06-12 14:51:51.513947 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:51.518983 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:51.535348 7f9d60bd2700 10 getattrs 3.6b_head/1e6106b/rbd_data.10ab6b8b4567.0000000000000247/head//3 = 0
2014-06-12 14:51:51.535366 7f9d60bd2700 10 stat 3.6b_head/6547106b/rbd_data.10936b8b4567.0000000000000775/head//3
2014-06-12 14:51:51.535465 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6547106b/rbd_data.10936b8b4567.0000000000000775/head//3
2014-06-12 14:51:51.569813 7f9d60bd2700 10 getattrs 3.6b_head/6547106b/rbd_data.10936b8b4567.0000000000000775/head//3 = 0
2014-06-12 14:51:51.569826 7f9d60bd2700 10 stat 3.6b_head/2e08106b/rbd_data.10916b8b4567.00000000000002b2/head//3
2014-06-12 14:51:51.569966 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 2e08106b/rbd_data.10916b8b4567.00000000000002b2/head//3
2014-06-12 14:51:51.616388 7f9d60bd2700 10 getattrs 3.6b_head/2e08106b/rbd_data.10916b8b4567.00000000000002b2/head//3 = 0
2014-06-12 14:51:51.616402 7f9d60bd2700 10 stat 3.6b_head/2d28106b/rbd_data.108e6b8b4567.0000000000003004/head//3
2014-06-12 14:51:51.616532 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 2d28106b/rbd_data.108e6b8b4567.0000000000003004/head//3
2014-06-12 14:51:51.616640 7f9d60bd2700 10 getattrs 3.6b_head/2d28106b/rbd_data.108e6b8b4567.0000000000003004/head//3 = 0
2014-06-12 14:51:51.616649 7f9d60bd2700 10 stat 3.6b_head/e939106b/rbd_data.10ac6b8b4567.000000000000030f/head//3
2014-06-12 14:51:51.616763 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid e939106b/rbd_data.10ac6b8b4567.000000000000030f/head//3
2014-06-12 14:51:51.641041 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:51.672844 7f9d60bd2700 10 getattrs 3.6b_head/e939106b/rbd_data.10ac6b8b4567.000000000000030f/head//3 = 0
2014-06-12 14:51:51.672877 7f9d60bd2700 10 stat 3.6b_head/c669106b/rbd_data.10a26b8b4567.000000000000187d/head//3
2014-06-12 14:51:51.673050 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c669106b/rbd_data.10a26b8b4567.000000000000187d/head//3
2014-06-12 14:51:51.697309 7f9d60bd2700 10 getattrs 3.6b_head/c669106b/rbd_data.10a26b8b4567.000000000000187d/head//3 = 0
2014-06-12 14:51:51.697321 7f9d60bd2700 10 stat 3.6b_head/4aba106b/rbd_data.10a06b8b4567.00000000000033c2/head//3
2014-06-12 14:51:51.697470 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 4aba106b/rbd_data.10a06b8b4567.00000000000033c2/head//3
2014-06-12 14:51:51.714758 7f9d60bd2700 10 getattrs 3.6b_head/4aba106b/rbd_data.10a06b8b4567.00000000000033c2/head//3 = 0
2014-06-12 14:51:51.714780 7f9d60bd2700 10 stat 3.6b_head/310c106b/rbd_data.10926b8b4567.000000000000137d/head//3
2014-06-12 14:51:51.714921 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 310c106b/rbd_data.10926b8b4567.000000000000137d/head//3
2014-06-12 14:51:51.723681 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:51.723729 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:51.727570 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:51.728822 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:51.745851 7f9d60bd2700 10 getattrs 3.6b_head/310c106b/rbd_data.10926b8b4567.000000000000137d/head//3 = 0
2014-06-12 14:51:51.745863 7f9d60bd2700 10 stat 3.6b_head/243d106b/rbd_data.10a76b8b4567.000000000000175a/head//3
2014-06-12 14:51:51.746004 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 243d106b/rbd_data.10a76b8b4567.000000000000175a/head//3
2014-06-12 14:51:51.762593 7f9d60bd2700 10 getattrs 3.6b_head/243d106b/rbd_data.10a76b8b4567.000000000000175a/head//3 = 0
2014-06-12 14:51:51.762604 7f9d60bd2700 10 stat 3.6b_head/4b50206b/rbd_data.10966b8b4567.00000000000025f2/head//3
2014-06-12 14:51:51.762722 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 4b50206b/rbd_data.10966b8b4567.00000000000025f2/head//3
2014-06-12 14:51:51.794683 7f9d60bd2700 10 getattrs 3.6b_head/4b50206b/rbd_data.10966b8b4567.00000000000025f2/head//3 = 0
2014-06-12 14:51:51.794698 7f9d60bd2700 10 stat 3.6b_head/8901206b/rbd_data.10986b8b4567.0000000000001d0a/head//3
2014-06-12 14:51:51.794820 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 8901206b/rbd_data.10986b8b4567.0000000000001d0a/head//3
2014-06-12 14:51:51.811943 7f9d60bd2700 10 getattrs 3.6b_head/8901206b/rbd_data.10986b8b4567.0000000000001d0a/head//3 = 0
2014-06-12 14:51:51.811952 7f9d60bd2700 10 stat 3.6b_head/da65206b/rbd_data.10986b8b4567.0000000000000960/head//3
2014-06-12 14:51:51.812078 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid da65206b/rbd_data.10986b8b4567.0000000000000960/head//3
2014-06-12 14:51:51.845464 7f9d60bd2700 10 getattrs 3.6b_head/da65206b/rbd_data.10986b8b4567.0000000000000960/head//3 = 0
2014-06-12 14:51:51.845478 7f9d60bd2700 10 stat 3.6b_head/195206b/rbd_data.10976b8b4567.00000000000033bb/head//3
2014-06-12 14:51:51.845602 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 195206b/rbd_data.10976b8b4567.00000000000033bb/head//3
2014-06-12 14:51:51.863015 7f9d60bd2700 10 getattrs 3.6b_head/195206b/rbd_data.10976b8b4567.00000000000033bb/head//3 = 0
2014-06-12 14:51:51.863028 7f9d60bd2700 10 stat 3.6b_head/54c5206b/rbd_data.10996b8b4567.0000000000001977/head//3
2014-06-12 14:51:51.863145 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 54c5206b/rbd_data.10996b8b4567.0000000000001977/head//3
2014-06-12 14:51:51.891725 7f9d60bd2700 10 getattrs 3.6b_head/54c5206b/rbd_data.10996b8b4567.0000000000001977/head//3 = 0
2014-06-12 14:51:51.891740 7f9d60bd2700 10 stat 3.6b_head/9718206b/rbd_data.10ab6b8b4567.0000000000002c5f/head//3
2014-06-12 14:51:51.891858 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9718206b/rbd_data.10ab6b8b4567.0000000000002c5f/head//3
2014-06-12 14:51:51.891955 7f9d60bd2700 10 getattrs 3.6b_head/9718206b/rbd_data.10ab6b8b4567.0000000000002c5f/head//3 = 0
2014-06-12 14:51:51.891961 7f9d60bd2700 10 stat 3.6b_head/7099206b/rbd_data.109c6b8b4567.00000000000039c6/head//3
2014-06-12 14:51:51.892061 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7099206b/rbd_data.109c6b8b4567.00000000000039c6/head//3
2014-06-12 14:51:51.908831 7f9d60bd2700 10 getattrs 3.6b_head/7099206b/rbd_data.109c6b8b4567.00000000000039c6/head//3 = 0
2014-06-12 14:51:51.908843 7f9d60bd2700 10 stat 3.6b_head/de50306b/rbd_data.10b16b8b4567.0000000000001ec7/head//3
2014-06-12 14:51:51.909079 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid de50306b/rbd_data.10b16b8b4567.0000000000001ec7/head//3
2014-06-12 14:51:51.938999 7f9d60bd2700 10 getattrs 3.6b_head/de50306b/rbd_data.10b16b8b4567.0000000000001ec7/head//3 = 0
2014-06-12 14:51:51.939011 7f9d60bd2700 10 stat 3.6b_head/6fe2306b/rbd_data.109e6b8b4567.00000000000014ca/head//3
2014-06-12 14:51:51.939130 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6fe2306b/rbd_data.109e6b8b4567.00000000000014ca/head//3
2014-06-12 14:51:51.958004 7f9d60bd2700 10 getattrs 3.6b_head/6fe2306b/rbd_data.109e6b8b4567.00000000000014ca/head//3 = 0
2014-06-12 14:51:51.958015 7f9d60bd2700 10 collection_getattrs 3.6b_head
2014-06-12 14:51:51.958069 7f9d60bd2700 10 collection_getattrs 3.6b_head = 0
2014-06-12 14:51:51.958071 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:51.958099 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:51.958106 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:51.958113 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:51.958116 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:51.990659 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:51.990763 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:52.018360 7f9d5501e700  5 -- op tracker -- , seq: 748608, time: 2014-06-12 14:51:52.018261, event: header_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.018379 7f9d5501e700  5 -- op tracker -- , seq: 748608, time: 2014-06-12 14:51:52.018264, event: throttled, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.018388 7f9d5501e700  5 -- op tracker -- , seq: 748608, time: 2014-06-12 14:51:52.018319, event: all_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.018417 7f9d5501e700  5 -- op tracker -- , seq: 748608, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.018422 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:52.018429 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:52.018438 7f9d5501e700 15 osd.2 173 enqueue_op 0x1e3eb900 prio 127 cost 10717 latency 0.000177 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:52.018504 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:52.018522 7f9d633d7700 10 osd.2 173 dequeue_op 0x1e3eb900 prio 127 cost 10717 latency 0.000260 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]
2014-06-12 14:51:52.018562 7f9d633d7700  5 -- op tracker -- , seq: 748608, time: 2014-06-12 14:51:52.018556, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.018592 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] handle_message: 0x1e3eb900
2014-06-12 14:51:52.018609 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:52.018629 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:52.018647 7f9d633d7700  5 -- op tracker -- , seq: 748608, time: 2014-06-12 14:51:52.018642, event: started, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.018656 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  got 29 scrub map
2014-06-12 14:51:52.018758 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] map version is 173'2887
2014-06-12 14:51:52.018782 7f9d633d7700 10 osd.2 173 dequeue_op 0x1e3eb900 finish
2014-06-12 14:51:52.018792 7f9d633d7700  5 -- op tracker -- , seq: 748608, time: 2014-06-12 14:51:52.018786, event: done, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.018808 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:52.018812 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:52.018822 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:52.018857 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:52.018889 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:52.018900 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:52.018936 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:52.018946 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:52.018956 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub replica 29 has 25 items
2014-06-12 14:51:52.018988 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9fc2006b/rbd_data.10946b8b4567.0000000000001b38/head//3, auth == maps.end()
2014-06-12 14:51:52.019028 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9fc2006b/rbd_data.10946b8b4567.0000000000001b38/head//3
2014-06-12 14:51:52.019049 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b073006b/rbd_data.10906b8b4567.0000000000003508/head//3, auth == maps.end()
2014-06-12 14:51:52.019063 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj b073006b/rbd_data.10906b8b4567.0000000000003508/head//3
2014-06-12 14:51:52.019078 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8208006b/rbd_data.10b16b8b4567.00000000000000dd/head//3, auth == maps.end()
2014-06-12 14:51:52.019092 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 8208006b/rbd_data.10b16b8b4567.00000000000000dd/head//3
2014-06-12 14:51:52.019107 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a9c9006b/rbd_data.10ad6b8b4567.000000000000186d/head//3, auth == maps.end()
2014-06-12 14:51:52.019120 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a9c9006b/rbd_data.10ad6b8b4567.000000000000186d/head//3
2014-06-12 14:51:52.019134 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7a3a006b/rbd_data.10956b8b4567.000000000000083c/head//3, auth == maps.end()
2014-06-12 14:51:52.019148 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7a3a006b/rbd_data.10956b8b4567.000000000000083c/head//3
2014-06-12 14:51:52.019162 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 213106b/rbd_data.10ae6b8b4567.000000000000058e/head//3, auth == maps.end()
2014-06-12 14:51:52.019176 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 213106b/rbd_data.10ae6b8b4567.000000000000058e/head//3
2014-06-12 14:51:52.019190 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4a4106b/rbd_data.10936b8b4567.0000000000001a48/head//3, auth == maps.end()
2014-06-12 14:51:52.019204 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 4a4106b/rbd_data.10936b8b4567.0000000000001a48/head//3
2014-06-12 14:51:52.019219 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1e6106b/rbd_data.10ab6b8b4567.0000000000000247/head//3, auth == maps.end()
2014-06-12 14:51:52.019235 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1e6106b/rbd_data.10ab6b8b4567.0000000000000247/head//3
2014-06-12 14:51:52.019250 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6547106b/rbd_data.10936b8b4567.0000000000000775/head//3, auth == maps.end()
2014-06-12 14:51:52.019266 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6547106b/rbd_data.10936b8b4567.0000000000000775/head//3
2014-06-12 14:51:52.019283 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2e08106b/rbd_data.10916b8b4567.00000000000002b2/head//3, auth == maps.end()
2014-06-12 14:51:52.019300 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 2e08106b/rbd_data.10916b8b4567.00000000000002b2/head//3
2014-06-12 14:51:52.019317 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2d28106b/rbd_data.108e6b8b4567.0000000000003004/head//3, auth == maps.end()
2014-06-12 14:51:52.019331 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 2d28106b/rbd_data.108e6b8b4567.0000000000003004/head//3
2014-06-12 14:51:52.019346 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e939106b/rbd_data.10ac6b8b4567.000000000000030f/head//3, auth == maps.end()
2014-06-12 14:51:52.019361 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj e939106b/rbd_data.10ac6b8b4567.000000000000030f/head//3
2014-06-12 14:51:52.019376 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c669106b/rbd_data.10a26b8b4567.000000000000187d/head//3, auth == maps.end()
2014-06-12 14:51:52.019390 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c669106b/rbd_data.10a26b8b4567.000000000000187d/head//3
2014-06-12 14:51:52.019404 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4aba106b/rbd_data.10a06b8b4567.00000000000033c2/head//3, auth == maps.end()
2014-06-12 14:51:52.019417 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 4aba106b/rbd_data.10a06b8b4567.00000000000033c2/head//3
2014-06-12 14:51:52.019431 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 310c106b/rbd_data.10926b8b4567.000000000000137d/head//3, auth == maps.end()
2014-06-12 14:51:52.019445 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 310c106b/rbd_data.10926b8b4567.000000000000137d/head//3
2014-06-12 14:51:52.019459 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 243d106b/rbd_data.10a76b8b4567.000000000000175a/head//3, auth == maps.end()
2014-06-12 14:51:52.019497 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 243d106b/rbd_data.10a76b8b4567.000000000000175a/head//3
2014-06-12 14:51:52.019512 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4b50206b/rbd_data.10966b8b4567.00000000000025f2/head//3, auth == maps.end()
2014-06-12 14:51:52.019526 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 4b50206b/rbd_data.10966b8b4567.00000000000025f2/head//3
2014-06-12 14:51:52.019541 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8901206b/rbd_data.10986b8b4567.0000000000001d0a/head//3, auth == maps.end()
2014-06-12 14:51:52.019554 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 8901206b/rbd_data.10986b8b4567.0000000000001d0a/head//3
2014-06-12 14:51:52.019568 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj da65206b/rbd_data.10986b8b4567.0000000000000960/head//3, auth == maps.end()
2014-06-12 14:51:52.019582 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj da65206b/rbd_data.10986b8b4567.0000000000000960/head//3
2014-06-12 14:51:52.019596 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 195206b/rbd_data.10976b8b4567.00000000000033bb/head//3, auth == maps.end()
2014-06-12 14:51:52.019609 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 195206b/rbd_data.10976b8b4567.00000000000033bb/head//3
2014-06-12 14:51:52.019623 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 54c5206b/rbd_data.10996b8b4567.0000000000001977/head//3, auth == maps.end()
2014-06-12 14:51:52.019637 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 54c5206b/rbd_data.10996b8b4567.0000000000001977/head//3
2014-06-12 14:51:52.019651 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9718206b/rbd_data.10ab6b8b4567.0000000000002c5f/head//3, auth == maps.end()
2014-06-12 14:51:52.019664 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9718206b/rbd_data.10ab6b8b4567.0000000000002c5f/head//3
2014-06-12 14:51:52.019678 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7099206b/rbd_data.109c6b8b4567.00000000000039c6/head//3, auth == maps.end()
2014-06-12 14:51:52.019692 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7099206b/rbd_data.109c6b8b4567.00000000000039c6/head//3
2014-06-12 14:51:52.019707 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj de50306b/rbd_data.10b16b8b4567.0000000000001ec7/head//3, auth == maps.end()
2014-06-12 14:51:52.019720 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj de50306b/rbd_data.10b16b8b4567.0000000000001ec7/head//3
2014-06-12 14:51:52.019737 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6fe2306b/rbd_data.109e6b8b4567.00000000000014ca/head//3, auth == maps.end()
2014-06-12 14:51:52.019753 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6fe2306b/rbd_data.109e6b8b4567.00000000000014ca/head//3
2014-06-12 14:51:52.019773 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] 
2014-06-12 14:51:52.019786 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub
2014-06-12 14:51:52.019805 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6fe2306b/rbd_data.109e6b8b4567.00000000000014ca/head//3 6fe2306b/rbd_data.109e6b8b4567.00000000000014ca/head//3(173'908 client.4295.0:47801 wrlock_by=unknown.0.0:0 dirty s 4194304 uv908)
2014-06-12 14:51:52.019826 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  de50306b/rbd_data.10b16b8b4567.0000000000001ec7/head//3 de50306b/rbd_data.10b16b8b4567.0000000000001ec7/head//3(173'1325 client.4314.0:64850 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1325)
2014-06-12 14:51:52.019845 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7099206b/rbd_data.109c6b8b4567.00000000000039c6/head//3 7099206b/rbd_data.109c6b8b4567.00000000000039c6/head//3(173'2325 client.4293.0:114188 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2325)
2014-06-12 14:51:52.019863 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9718206b/rbd_data.10ab6b8b4567.0000000000002c5f/head//3 9718206b/rbd_data.10ab6b8b4567.0000000000002c5f/head//3(173'1789 client.4308.0:89752 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1789)
2014-06-12 14:51:52.019883 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  54c5206b/rbd_data.10996b8b4567.0000000000001977/head//3 54c5206b/rbd_data.10996b8b4567.0000000000001977/head//3(173'1002 client.4290.0:54603 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1002)
2014-06-12 14:51:52.019903 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  195206b/rbd_data.10976b8b4567.00000000000033bb/head//3 195206b/rbd_data.10976b8b4567.00000000000033bb/head//3(173'2144 client.4288.0:100799 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2144)
2014-06-12 14:51:52.019922 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  da65206b/rbd_data.10986b8b4567.0000000000000960/head//3 da65206b/rbd_data.10986b8b4567.0000000000000960/head//3(173'2645 client.4289.0:130306 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2645)
2014-06-12 14:51:52.019939 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  8901206b/rbd_data.10986b8b4567.0000000000001d0a/head//3 8901206b/rbd_data.10986b8b4567.0000000000001d0a/head//3(173'1211 client.4289.0:60660 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1211)
2014-06-12 14:51:52.019957 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  4b50206b/rbd_data.10966b8b4567.00000000000025f2/head//3 4b50206b/rbd_data.10966b8b4567.00000000000025f2/head//3(173'1473 client.4287.0:74517 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1473)
2014-06-12 14:51:52.019976 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  243d106b/rbd_data.10a76b8b4567.000000000000175a/head//3 243d106b/rbd_data.10a76b8b4567.000000000000175a/head//3(173'1074 client.4304.0:52292 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1074)
2014-06-12 14:51:52.019994 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  310c106b/rbd_data.10926b8b4567.000000000000137d/head//3 310c106b/rbd_data.10926b8b4567.000000000000137d/head//3(173'822 client.4283.0:43550 wrlock_by=unknown.0.0:0 dirty s 4194304 uv822)
2014-06-12 14:51:52.020014 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  4aba106b/rbd_data.10a06b8b4567.00000000000033c2/head//3 4aba106b/rbd_data.10a06b8b4567.00000000000033c2/head//3(173'2040 client.4297.0:101900 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2040)
2014-06-12 14:51:52.020034 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c669106b/rbd_data.10a26b8b4567.000000000000187d/head//3 c669106b/rbd_data.10a26b8b4567.000000000000187d/head//3(173'1149 client.4299.0:54221 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1149)
2014-06-12 14:51:52.020054 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  e939106b/rbd_data.10ac6b8b4567.000000000000030f/head//3 e939106b/rbd_data.10ac6b8b4567.000000000000030f/head//3(173'2480 client.4309.0:120642 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2480)
2014-06-12 14:51:52.020074 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  2d28106b/rbd_data.108e6b8b4567.0000000000003004/head//3 2d28106b/rbd_data.108e6b8b4567.0000000000003004/head//3(173'1916 client.4279.0:95948 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1916)
2014-06-12 14:51:52.020095 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  2e08106b/rbd_data.10916b8b4567.00000000000002b2/head//3 2e08106b/rbd_data.10916b8b4567.00000000000002b2/head//3(173'2460 client.4282.0:122447 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2460)
2014-06-12 14:51:52.020116 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6547106b/rbd_data.10936b8b4567.0000000000000775/head//3 6547106b/rbd_data.10936b8b4567.0000000000000775/head//3(173'2601 client.4284.0:126802 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2601)
2014-06-12 14:51:52.020136 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1e6106b/rbd_data.10ab6b8b4567.0000000000000247/head//3 1e6106b/rbd_data.10ab6b8b4567.0000000000000247/head//3(173'2451 client.4308.0:122246 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2451)
2014-06-12 14:51:52.020157 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  4a4106b/rbd_data.10936b8b4567.0000000000001a48/head//3 4a4106b/rbd_data.10936b8b4567.0000000000001a48/head//3(173'1172 client.4284.0:54890 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1172)
2014-06-12 14:51:52.020177 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  213106b/rbd_data.10ae6b8b4567.000000000000058e/head//3 213106b/rbd_data.10ae6b8b4567.000000000000058e/head//3(173'2528 client.4311.0:123586 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2528)
2014-06-12 14:51:52.020197 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7a3a006b/rbd_data.10956b8b4567.000000000000083c/head//3 7a3a006b/rbd_data.10956b8b4567.000000000000083c/head//3(173'2629 client.4286.0:130956 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2629)
2014-06-12 14:51:52.020218 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a9c9006b/rbd_data.10ad6b8b4567.000000000000186d/head//3 a9c9006b/rbd_data.10ad6b8b4567.000000000000186d/head//3(173'1100 client.4310.0:53577 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1100)
2014-06-12 14:51:52.020238 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  8208006b/rbd_data.10b16b8b4567.00000000000000dd/head//3 8208006b/rbd_data.10b16b8b4567.00000000000000dd/head//3(173'2411 client.4314.0:119256 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2411)
2014-06-12 14:51:52.020258 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  b073006b/rbd_data.10906b8b4567.0000000000003508/head//3 b073006b/rbd_data.10906b8b4567.0000000000003508/head//3(173'2033 client.4281.0:104878 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2033)
2014-06-12 14:51:52.020278 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9fc2006b/rbd_data.10946b8b4567.0000000000001b38/head//3 9fc2006b/rbd_data.10946b8b4567.0000000000001b38/head//3(173'1117 client.4285.0:57680 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1117)
2014-06-12 14:51:52.020294 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:52.020310 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:52.020324 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:52.020327 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:52.020329 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:52.020385 7f9d60bd2700 10 collection_list_partial 3.6b_head start:a405306b//0//-1 is_max:0
2014-06-12 14:51:52.020716 7f9d60bd2700 20 objects: [290a306b/rbd_data.109c6b8b4567.0000000000003adb/head//3,3cfa306b/rbd_data.10966b8b4567.0000000000002cf2/head//3,ab9e306b/rbd_data.10b06b8b4567.0000000000003868/head//3,2b10406b/rbd_data.10a66b8b4567.0000000000003ba1/head//3,3960406b/rbd_data.10a06b8b4567.000000000000384c/head//3,d0c1406b/rbd_data.10996b8b4567.0000000000000ed8/head//3,9612406b/rbd_data.108e6b8b4567.00000000000026f7/head//3,6342406b/rbd_data.10976b8b4567.0000000000002af1/head//3,5e33406b/rbd_data.109b6b8b4567.00000000000025d8/head//3,2893406b/rbd_data.109c6b8b4567.0000000000000f4f/head//3,8514406b/rbd_data.108f6b8b4567.0000000000000b44/head//3,b61a406b/rbd_data.10b16b8b4567.0000000000002d91/head//3,c6a406b/rbd_data.10956b8b4567.0000000000000b95/head//3,a7da406b/rbd_data.10b36b8b4567.0000000000000d72/head//3,f36e406b/rbd_data.108f6b8b4567.000000000000386f/head//3,a69f406b/rbd_data.10b46b8b4567.0000000000001e2c/head//3,b6ff406b/rbd_data.10996b8b4567.0000000000000d20/head//3,7b73506b/rbd_data.10936b8b4567.000000000000143f/head//3,7314506b/rbd_data.10b16b8b4567.0000000000002563/head//3,4b64506b/rbd_data.10976b8b4567.0000000000002964/head//3,a2e4506b/rbd_data.10a06b8b4567.0000000000001f10/head//3,e125506b/rbd_data.10a56b8b4567.000000000000342e/head//3,3319506b/rbd_data.109f6b8b4567.0000000000002d74/head//3,9b79506b/rbd_data.108e6b8b4567.00000000000017a5/head//3,5c0a506b/rbd_data.10b26b8b4567.0000000000002b39/head//3]
2014-06-12 14:51:52.021354 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  requesting scrubmap from osd.29
2014-06-12 14:51:52.021387 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:52.021409 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:52.021423 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:52.021432 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:52.021441 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:52.021450 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_map_chunk [a405306b//0//-1,4bfc506b//0//-1)
2014-06-12 14:51:52.021461 7f9d60bd2700 10 collection_list_partial 3.6b_head start:a405306b//0//-1 is_max:0
2014-06-12 14:51:52.022368 7f9d60bd2700 20 objects: [290a306b/rbd_data.109c6b8b4567.0000000000003adb/head//3,3cfa306b/rbd_data.10966b8b4567.0000000000002cf2/head//3,ab9e306b/rbd_data.10b06b8b4567.0000000000003868/head//3,2b10406b/rbd_data.10a66b8b4567.0000000000003ba1/head//3,3960406b/rbd_data.10a06b8b4567.000000000000384c/head//3,d0c1406b/rbd_data.10996b8b4567.0000000000000ed8/head//3,9612406b/rbd_data.108e6b8b4567.00000000000026f7/head//3,6342406b/rbd_data.10976b8b4567.0000000000002af1/head//3,5e33406b/rbd_data.109b6b8b4567.00000000000025d8/head//3,2893406b/rbd_data.109c6b8b4567.0000000000000f4f/head//3,8514406b/rbd_data.108f6b8b4567.0000000000000b44/head//3,b61a406b/rbd_data.10b16b8b4567.0000000000002d91/head//3,c6a406b/rbd_data.10956b8b4567.0000000000000b95/head//3,a7da406b/rbd_data.10b36b8b4567.0000000000000d72/head//3,f36e406b/rbd_data.108f6b8b4567.000000000000386f/head//3,a69f406b/rbd_data.10b46b8b4567.0000000000001e2c/head//3,b6ff406b/rbd_data.10996b8b4567.0000000000000d20/head//3,7b73506b/rbd_data.10936b8b4567.000000000000143f/head//3,7314506b/rbd_data.10b16b8b4567.0000000000002563/head//3,4b64506b/rbd_data.10976b8b4567.0000000000002964/head//3,a2e4506b/rbd_data.10a06b8b4567.0000000000001f10/head//3,e125506b/rbd_data.10a56b8b4567.000000000000342e/head//3,3319506b/rbd_data.109f6b8b4567.0000000000002d74/head//3,9b79506b/rbd_data.108e6b8b4567.00000000000017a5/head//3,5c0a506b/rbd_data.10b26b8b4567.0000000000002b39/head//3,4bfc506b/rbd_data.10956b8b4567.000000000000125e/head//3,8be0606b/rbd_data.109f6b8b4567.0000000000001d87/head//3,1265606b/rbd_data.10946b8b4567.0000000000000987/head//3,c346606b/rbd_data.10af6b8b4567.0000000000002264/head//3,4407606b/rbd_data.109a6b8b4567.0000000000000689/head//3,f6f7606b/rbd_data.10ad6b8b4567.0000000000002157/head//3,caa606b/rbd_data.10af6b8b4567.0000000000002a63/head//3,c74b606b/rbd_data.10ad6b8b4567.0000000000001dd3/head//3,7cbe606b/rbd_data.10ad6b8b4567.00000000000023aa/head//3,377f606b/rbd_data.10ad6b8b4567.0000000000000647/head//3,6af606b/rbd_data.10ac6b8b4567.0000000000001d93/head//3,d5c2706b/rbd_data.109a6b8b4567.0000000000001be5/head//3,5ea3706b/rbd_data.10916b8b4567.0000000000001ed0/head//3,cea5706b/rbd_data.10b26b8b4567.00000000000011d1/head//3,846706b/rbd_data.10926b8b4567.0000000000002f32/head//3,20d9706b/rbd_data.10936b8b4567.0000000000000ab9/head//3,285f706b/rbd_data.109d6b8b4567.00000000000034f6/head//3,56cf706b/rbd_data.10946b8b4567.0000000000003637/head//3,9c90806b/rbd_data.10b06b8b4567.0000000000002ecc/head//3,7501806b/rbd_data.108e6b8b4567.00000000000036a7/head//3,c7d4806b/rbd_data.10a36b8b4567.000000000000308f/head//3,11f6806b/rbd_data.109b6b8b4567.00000000000023b6/head//3,ae57806b/rbd_data.10976b8b4567.00000000000018bf/head//3,67c7806b/rbd_data.109b6b8b4567.0000000000002558/head//3,a119806b/rbd_data.10a46b8b4567.0000000000003356/head//3,85eb806b/rbd_data.10a66b8b4567.0000000000003905/head//3,2e4c806b/rbd_data.10b26b8b4567.0000000000002c83/head//3,ae806b/rbd_data.10a96b8b4567.00000000000016f0/head//3,c4ce806b/rbd_data.10a56b8b4567.0000000000000a54/head//3,7ede806b/rbd_data.10a56b8b4567.0000000000003710/head//3,6ef2906b/rbd_data.10ad6b8b4567.0000000000003bb1/head//3,4055906b/rbd_data.10946b8b4567.00000000000023c3/head//3,c4c5906b/rbd_data.10956b8b4567.00000000000039c9/head//3,5fc6906b/rbd_data.10946b8b4567.00000000000019d4/head//3,5227906b/rbd_data.10ae6b8b4567.0000000000003486/head//3,dee7906b/rbd_data.10986b8b4567.0000000000000fcb/head//3,fdf7906b/rbd_data.10916b8b4567.00000000000029ec/head//3,ae18906b/rbd_data.109f6b8b4567.000000000000095e/head//3,ccb8906b/rbd_data.10956b8b4567.000000000000218a/head//3]
2014-06-12 14:51:52.022502 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:52.022537 7f9d60bd2700 10 stat 3.6b_head/290a306b/rbd_data.109c6b8b4567.0000000000003adb/head//3
2014-06-12 14:51:52.022749 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 290a306b/rbd_data.109c6b8b4567.0000000000003adb/head//3
2014-06-12 14:51:52.040026 7f9d60bd2700 10 getattrs 3.6b_head/290a306b/rbd_data.109c6b8b4567.0000000000003adb/head//3 = 0
2014-06-12 14:51:52.040033 7f9d60bd2700 10 stat 3.6b_head/3cfa306b/rbd_data.10966b8b4567.0000000000002cf2/head//3
2014-06-12 14:51:52.040135 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 3cfa306b/rbd_data.10966b8b4567.0000000000002cf2/head//3
2014-06-12 14:51:52.065165 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:52.066128 7f9d60bd2700 10 getattrs 3.6b_head/3cfa306b/rbd_data.10966b8b4567.0000000000002cf2/head//3 = 0
2014-06-12 14:51:52.066145 7f9d60bd2700 10 stat 3.6b_head/ab9e306b/rbd_data.10b06b8b4567.0000000000003868/head//3
2014-06-12 14:51:52.066320 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ab9e306b/rbd_data.10b06b8b4567.0000000000003868/head//3
2014-06-12 14:51:52.080405 7f9d60bd2700 10 getattrs 3.6b_head/ab9e306b/rbd_data.10b06b8b4567.0000000000003868/head//3 = 0
2014-06-12 14:51:52.080412 7f9d60bd2700 10 stat 3.6b_head/2b10406b/rbd_data.10a66b8b4567.0000000000003ba1/head//3
2014-06-12 14:51:52.080505 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 2b10406b/rbd_data.10a66b8b4567.0000000000003ba1/head//3
2014-06-12 14:51:52.098073 7f9d60bd2700 10 getattrs 3.6b_head/2b10406b/rbd_data.10a66b8b4567.0000000000003ba1/head//3 = 0
2014-06-12 14:51:52.098085 7f9d60bd2700 10 stat 3.6b_head/3960406b/rbd_data.10a06b8b4567.000000000000384c/head//3
2014-06-12 14:51:52.098173 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 3960406b/rbd_data.10a06b8b4567.000000000000384c/head//3
2014-06-12 14:51:52.098253 7f9d60bd2700 10 getattrs 3.6b_head/3960406b/rbd_data.10a06b8b4567.000000000000384c/head//3 = 0
2014-06-12 14:51:52.098259 7f9d60bd2700 10 stat 3.6b_head/d0c1406b/rbd_data.10996b8b4567.0000000000000ed8/head//3
2014-06-12 14:51:52.098335 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid d0c1406b/rbd_data.10996b8b4567.0000000000000ed8/head//3
2014-06-12 14:51:52.138159 7f9d60bd2700 10 getattrs 3.6b_head/d0c1406b/rbd_data.10996b8b4567.0000000000000ed8/head//3 = 0
2014-06-12 14:51:52.138173 7f9d60bd2700 10 stat 3.6b_head/9612406b/rbd_data.108e6b8b4567.00000000000026f7/head//3
2014-06-12 14:51:52.138287 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9612406b/rbd_data.108e6b8b4567.00000000000026f7/head//3
2014-06-12 14:51:52.155547 7f9d60bd2700 10 getattrs 3.6b_head/9612406b/rbd_data.108e6b8b4567.00000000000026f7/head//3 = 0
2014-06-12 14:51:52.155560 7f9d60bd2700 10 stat 3.6b_head/6342406b/rbd_data.10976b8b4567.0000000000002af1/head//3
2014-06-12 14:51:52.155654 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6342406b/rbd_data.10976b8b4567.0000000000002af1/head//3
2014-06-12 14:51:52.168077 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:52.169078 7f9d60bd2700 10 getattrs 3.6b_head/6342406b/rbd_data.10976b8b4567.0000000000002af1/head//3 = 0
2014-06-12 14:51:52.169086 7f9d60bd2700 10 stat 3.6b_head/5e33406b/rbd_data.109b6b8b4567.00000000000025d8/head//3
2014-06-12 14:51:52.169173 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5e33406b/rbd_data.109b6b8b4567.00000000000025d8/head//3
2014-06-12 14:51:52.189180 7f9d60bd2700 10 getattrs 3.6b_head/5e33406b/rbd_data.109b6b8b4567.00000000000025d8/head//3 = 0
2014-06-12 14:51:52.189189 7f9d60bd2700 10 stat 3.6b_head/2893406b/rbd_data.109c6b8b4567.0000000000000f4f/head//3
2014-06-12 14:51:52.189286 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 2893406b/rbd_data.109c6b8b4567.0000000000000f4f/head//3
2014-06-12 14:51:52.218570 7f9d60bd2700 10 getattrs 3.6b_head/2893406b/rbd_data.109c6b8b4567.0000000000000f4f/head//3 = 0
2014-06-12 14:51:52.218582 7f9d60bd2700 10 stat 3.6b_head/8514406b/rbd_data.108f6b8b4567.0000000000000b44/head//3
2014-06-12 14:51:52.218682 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 8514406b/rbd_data.108f6b8b4567.0000000000000b44/head//3
2014-06-12 14:51:52.278365 7f9d60bd2700 10 getattrs 3.6b_head/8514406b/rbd_data.108f6b8b4567.0000000000000b44/head//3 = 0
2014-06-12 14:51:52.278381 7f9d60bd2700 10 stat 3.6b_head/b61a406b/rbd_data.10b16b8b4567.0000000000002d91/head//3
2014-06-12 14:51:52.278514 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid b61a406b/rbd_data.10b16b8b4567.0000000000002d91/head//3
2014-06-12 14:51:52.280180 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:52.280205 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:52.280211 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:52.280213 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:52.280252 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:52.303059 7f9d60bd2700 10 getattrs 3.6b_head/b61a406b/rbd_data.10b16b8b4567.0000000000002d91/head//3 = 0
2014-06-12 14:51:52.303070 7f9d60bd2700 10 stat 3.6b_head/c6a406b/rbd_data.10956b8b4567.0000000000000b95/head//3
2014-06-12 14:51:52.303171 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c6a406b/rbd_data.10956b8b4567.0000000000000b95/head//3
2014-06-12 14:51:52.313383 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:52.326841 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:52.360579 7f9d60bd2700 10 getattrs 3.6b_head/c6a406b/rbd_data.10956b8b4567.0000000000000b95/head//3 = 0
2014-06-12 14:51:52.360603 7f9d60bd2700 10 stat 3.6b_head/a7da406b/rbd_data.10b36b8b4567.0000000000000d72/head//3
2014-06-12 14:51:52.360740 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a7da406b/rbd_data.10b36b8b4567.0000000000000d72/head//3
2014-06-12 14:51:52.366911 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:52.371817 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:52.372954 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:52.373280 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:52.382516 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:52.400036 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:52.401841 7f9d60bd2700 10 getattrs 3.6b_head/a7da406b/rbd_data.10b36b8b4567.0000000000000d72/head//3 = 0
2014-06-12 14:51:52.401852 7f9d60bd2700 10 stat 3.6b_head/f36e406b/rbd_data.108f6b8b4567.000000000000386f/head//3
2014-06-12 14:51:52.402001 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid f36e406b/rbd_data.108f6b8b4567.000000000000386f/head//3
2014-06-12 14:51:52.402572 7f9d60bd2700 10 getattrs 3.6b_head/f36e406b/rbd_data.108f6b8b4567.000000000000386f/head//3 = 0
2014-06-12 14:51:52.402582 7f9d60bd2700 10 stat 3.6b_head/a69f406b/rbd_data.10b46b8b4567.0000000000001e2c/head//3
2014-06-12 14:51:52.402704 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a69f406b/rbd_data.10b46b8b4567.0000000000001e2c/head//3
2014-06-12 14:51:52.437684 7f9d60bd2700 10 getattrs 3.6b_head/a69f406b/rbd_data.10b46b8b4567.0000000000001e2c/head//3 = 0
2014-06-12 14:51:52.437702 7f9d60bd2700 10 stat 3.6b_head/b6ff406b/rbd_data.10996b8b4567.0000000000000d20/head//3
2014-06-12 14:51:52.437867 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid b6ff406b/rbd_data.10996b8b4567.0000000000000d20/head//3
2014-06-12 14:51:52.476506 7f9d60bd2700 10 getattrs 3.6b_head/b6ff406b/rbd_data.10996b8b4567.0000000000000d20/head//3 = 0
2014-06-12 14:51:52.476528 7f9d60bd2700 10 stat 3.6b_head/7b73506b/rbd_data.10936b8b4567.000000000000143f/head//3
2014-06-12 14:51:52.476655 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7b73506b/rbd_data.10936b8b4567.000000000000143f/head//3
2014-06-12 14:51:52.510409 7f9d60bd2700 10 getattrs 3.6b_head/7b73506b/rbd_data.10936b8b4567.000000000000143f/head//3 = 0
2014-06-12 14:51:52.510427 7f9d60bd2700 10 stat 3.6b_head/7314506b/rbd_data.10b16b8b4567.0000000000002563/head//3
2014-06-12 14:51:52.510560 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7314506b/rbd_data.10b16b8b4567.0000000000002563/head//3
2014-06-12 14:51:52.522662 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:52.523509 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:52.525490 7f9d60bd2700 10 getattrs 3.6b_head/7314506b/rbd_data.10b16b8b4567.0000000000002563/head//3 = 0
2014-06-12 14:51:52.525506 7f9d60bd2700 10 stat 3.6b_head/4b64506b/rbd_data.10976b8b4567.0000000000002964/head//3
2014-06-12 14:51:52.525626 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 4b64506b/rbd_data.10976b8b4567.0000000000002964/head//3
2014-06-12 14:51:52.532964 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:52.537806 7f9d60bd2700 10 getattrs 3.6b_head/4b64506b/rbd_data.10976b8b4567.0000000000002964/head//3 = 0
2014-06-12 14:51:52.537817 7f9d60bd2700 10 stat 3.6b_head/a2e4506b/rbd_data.10a06b8b4567.0000000000001f10/head//3
2014-06-12 14:51:52.537950 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a2e4506b/rbd_data.10a06b8b4567.0000000000001f10/head//3
2014-06-12 14:51:52.566254 7f9d60bd2700 10 getattrs 3.6b_head/a2e4506b/rbd_data.10a06b8b4567.0000000000001f10/head//3 = 0
2014-06-12 14:51:52.566272 7f9d60bd2700 10 stat 3.6b_head/e125506b/rbd_data.10a56b8b4567.000000000000342e/head//3
2014-06-12 14:51:52.566385 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid e125506b/rbd_data.10a56b8b4567.000000000000342e/head//3
2014-06-12 14:51:52.570417 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:52.584849 7f9d60bd2700 10 getattrs 3.6b_head/e125506b/rbd_data.10a56b8b4567.000000000000342e/head//3 = 0
2014-06-12 14:51:52.584859 7f9d60bd2700 10 stat 3.6b_head/3319506b/rbd_data.109f6b8b4567.0000000000002d74/head//3
2014-06-12 14:51:52.584969 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 3319506b/rbd_data.109f6b8b4567.0000000000002d74/head//3
2014-06-12 14:51:52.615573 7f9d60bd2700 10 getattrs 3.6b_head/3319506b/rbd_data.109f6b8b4567.0000000000002d74/head//3 = 0
2014-06-12 14:51:52.615583 7f9d60bd2700 10 stat 3.6b_head/9b79506b/rbd_data.108e6b8b4567.00000000000017a5/head//3
2014-06-12 14:51:52.615696 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9b79506b/rbd_data.108e6b8b4567.00000000000017a5/head//3
2014-06-12 14:51:52.615798 7f9d60bd2700 10 getattrs 3.6b_head/9b79506b/rbd_data.108e6b8b4567.00000000000017a5/head//3 = 0
2014-06-12 14:51:52.615805 7f9d60bd2700 10 stat 3.6b_head/5c0a506b/rbd_data.10b26b8b4567.0000000000002b39/head//3
2014-06-12 14:51:52.615903 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5c0a506b/rbd_data.10b26b8b4567.0000000000002b39/head//3
2014-06-12 14:51:52.631100 7f9d60bd2700 10 getattrs 3.6b_head/5c0a506b/rbd_data.10b26b8b4567.0000000000002b39/head//3 = 0
2014-06-12 14:51:52.631114 7f9d60bd2700 10 collection_getattrs 3.6b_head
2014-06-12 14:51:52.631171 7f9d60bd2700 10 collection_getattrs 3.6b_head = 0
2014-06-12 14:51:52.631178 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:52.631216 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:52.631225 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:52.631232 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:52.631235 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:52.643799 7f9d5501e700  5 -- op tracker -- , seq: 748609, time: 2014-06-12 14:51:52.643716, event: header_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.643810 7f9d5501e700  5 -- op tracker -- , seq: 748609, time: 2014-06-12 14:51:52.643718, event: throttled, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.643817 7f9d5501e700  5 -- op tracker -- , seq: 748609, time: 2014-06-12 14:51:52.643766, event: all_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.643823 7f9d5501e700  5 -- op tracker -- , seq: 748609, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.643828 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:52.643833 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:52.643840 7f9d5501e700 15 osd.2 173 enqueue_op 0x1e3eb200 prio 127 cost 10717 latency 0.000124 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:52.643908 7f9d683e1700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:52.643930 7f9d683e1700 10 osd.2 173 dequeue_op 0x1e3eb200 prio 127 cost 10717 latency 0.000213 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]
2014-06-12 14:51:52.643970 7f9d683e1700  5 -- op tracker -- , seq: 748609, time: 2014-06-12 14:51:52.643963, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.643994 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] handle_message: 0x1e3eb200
2014-06-12 14:51:52.644010 7f9d683e1700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:52.644032 7f9d683e1700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:52.644049 7f9d683e1700  5 -- op tracker -- , seq: 748609, time: 2014-06-12 14:51:52.644044, event: started, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.644057 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  got 29 scrub map
2014-06-12 14:51:52.644161 7f9d683e1700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] map version is 173'2887
2014-06-12 14:51:52.644181 7f9d683e1700 10 osd.2 173 dequeue_op 0x1e3eb200 finish
2014-06-12 14:51:52.644192 7f9d683e1700  5 -- op tracker -- , seq: 748609, time: 2014-06-12 14:51:52.644186, event: done, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:52.644214 7f9d683e1700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:52.644218 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:52.644222 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:52.644235 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:52.644258 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:52.644277 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:52.644305 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:52.644312 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:52.644319 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub replica 29 has 25 items
2014-06-12 14:51:52.644344 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 290a306b/rbd_data.109c6b8b4567.0000000000003adb/head//3, auth == maps.end()
2014-06-12 14:51:52.644365 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 290a306b/rbd_data.109c6b8b4567.0000000000003adb/head//3
2014-06-12 14:51:52.644381 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3cfa306b/rbd_data.10966b8b4567.0000000000002cf2/head//3, auth == maps.end()
2014-06-12 14:51:52.644393 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 3cfa306b/rbd_data.10966b8b4567.0000000000002cf2/head//3
2014-06-12 14:51:52.644404 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ab9e306b/rbd_data.10b06b8b4567.0000000000003868/head//3, auth == maps.end()
2014-06-12 14:51:52.644415 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ab9e306b/rbd_data.10b06b8b4567.0000000000003868/head//3
2014-06-12 14:51:52.644426 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2b10406b/rbd_data.10a66b8b4567.0000000000003ba1/head//3, auth == maps.end()
2014-06-12 14:51:52.644438 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 2b10406b/rbd_data.10a66b8b4567.0000000000003ba1/head//3
2014-06-12 14:51:52.644449 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3960406b/rbd_data.10a06b8b4567.000000000000384c/head//3, auth == maps.end()
2014-06-12 14:51:52.644460 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 3960406b/rbd_data.10a06b8b4567.000000000000384c/head//3
2014-06-12 14:51:52.644471 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d0c1406b/rbd_data.10996b8b4567.0000000000000ed8/head//3, auth == maps.end()
2014-06-12 14:51:52.644481 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj d0c1406b/rbd_data.10996b8b4567.0000000000000ed8/head//3
2014-06-12 14:51:52.644491 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9612406b/rbd_data.108e6b8b4567.00000000000026f7/head//3, auth == maps.end()
2014-06-12 14:51:52.644501 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9612406b/rbd_data.108e6b8b4567.00000000000026f7/head//3
2014-06-12 14:51:52.644511 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6342406b/rbd_data.10976b8b4567.0000000000002af1/head//3, auth == maps.end()
2014-06-12 14:51:52.644520 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6342406b/rbd_data.10976b8b4567.0000000000002af1/head//3
2014-06-12 14:51:52.644530 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5e33406b/rbd_data.109b6b8b4567.00000000000025d8/head//3, auth == maps.end()
2014-06-12 14:51:52.644540 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5e33406b/rbd_data.109b6b8b4567.00000000000025d8/head//3
2014-06-12 14:51:52.644550 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2893406b/rbd_data.109c6b8b4567.0000000000000f4f/head//3, auth == maps.end()
2014-06-12 14:51:52.644560 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 2893406b/rbd_data.109c6b8b4567.0000000000000f4f/head//3
2014-06-12 14:51:52.644570 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8514406b/rbd_data.108f6b8b4567.0000000000000b44/head//3, auth == maps.end()
2014-06-12 14:51:52.644582 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 8514406b/rbd_data.108f6b8b4567.0000000000000b44/head//3
2014-06-12 14:51:52.644594 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b61a406b/rbd_data.10b16b8b4567.0000000000002d91/head//3, auth == maps.end()
2014-06-12 14:51:52.644606 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj b61a406b/rbd_data.10b16b8b4567.0000000000002d91/head//3
2014-06-12 14:51:52.644616 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c6a406b/rbd_data.10956b8b4567.0000000000000b95/head//3, auth == maps.end()
2014-06-12 14:51:52.644626 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c6a406b/rbd_data.10956b8b4567.0000000000000b95/head//3
2014-06-12 14:51:52.644638 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a7da406b/rbd_data.10b36b8b4567.0000000000000d72/head//3, auth == maps.end()
2014-06-12 14:51:52.644649 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a7da406b/rbd_data.10b36b8b4567.0000000000000d72/head//3
2014-06-12 14:51:52.644660 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f36e406b/rbd_data.108f6b8b4567.000000000000386f/head//3, auth == maps.end()
2014-06-12 14:51:52.644671 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj f36e406b/rbd_data.108f6b8b4567.000000000000386f/head//3
2014-06-12 14:51:52.644682 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a69f406b/rbd_data.10b46b8b4567.0000000000001e2c/head//3, auth == maps.end()
2014-06-12 14:51:52.644691 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a69f406b/rbd_data.10b46b8b4567.0000000000001e2c/head//3
2014-06-12 14:51:52.644701 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b6ff406b/rbd_data.10996b8b4567.0000000000000d20/head//3, auth == maps.end()
2014-06-12 14:51:52.644711 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj b6ff406b/rbd_data.10996b8b4567.0000000000000d20/head//3
2014-06-12 14:51:52.644721 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7b73506b/rbd_data.10936b8b4567.000000000000143f/head//3, auth == maps.end()
2014-06-12 14:51:52.644730 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7b73506b/rbd_data.10936b8b4567.000000000000143f/head//3
2014-06-12 14:51:52.644741 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7314506b/rbd_data.10b16b8b4567.0000000000002563/head//3, auth == maps.end()
2014-06-12 14:51:52.644750 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7314506b/rbd_data.10b16b8b4567.0000000000002563/head//3
2014-06-12 14:51:52.644760 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4b64506b/rbd_data.10976b8b4567.0000000000002964/head//3, auth == maps.end()
2014-06-12 14:51:52.644770 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 4b64506b/rbd_data.10976b8b4567.0000000000002964/head//3
2014-06-12 14:51:52.644781 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a2e4506b/rbd_data.10a06b8b4567.0000000000001f10/head//3, auth == maps.end()
2014-06-12 14:51:52.644793 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a2e4506b/rbd_data.10a06b8b4567.0000000000001f10/head//3
2014-06-12 14:51:52.644804 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e125506b/rbd_data.10a56b8b4567.000000000000342e/head//3, auth == maps.end()
2014-06-12 14:51:52.644815 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj e125506b/rbd_data.10a56b8b4567.000000000000342e/head//3
2014-06-12 14:51:52.644825 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3319506b/rbd_data.109f6b8b4567.0000000000002d74/head//3, auth == maps.end()
2014-06-12 14:51:52.644836 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 3319506b/rbd_data.109f6b8b4567.0000000000002d74/head//3
2014-06-12 14:51:52.644848 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9b79506b/rbd_data.108e6b8b4567.00000000000017a5/head//3, auth == maps.end()
2014-06-12 14:51:52.644859 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9b79506b/rbd_data.108e6b8b4567.00000000000017a5/head//3
2014-06-12 14:51:52.644870 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5c0a506b/rbd_data.10b26b8b4567.0000000000002b39/head//3, auth == maps.end()
2014-06-12 14:51:52.644880 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5c0a506b/rbd_data.10b26b8b4567.0000000000002b39/head//3
2014-06-12 14:51:52.644893 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] 
2014-06-12 14:51:52.644902 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub
2014-06-12 14:51:52.644916 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5c0a506b/rbd_data.10b26b8b4567.0000000000002b39/head//3 5c0a506b/rbd_data.10b26b8b4567.0000000000002b39/head//3(173'1659 client.4315.0:87625 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1659)
2014-06-12 14:51:52.644933 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9b79506b/rbd_data.108e6b8b4567.00000000000017a5/head//3 9b79506b/rbd_data.108e6b8b4567.00000000000017a5/head//3(173'1009 client.4279.0:52291 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1009)
2014-06-12 14:51:52.644948 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  3319506b/rbd_data.109f6b8b4567.0000000000002d74/head//3 3319506b/rbd_data.109f6b8b4567.0000000000002d74/head//3(173'1678 client.4296.0:89298 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1678)
2014-06-12 14:51:52.644962 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  e125506b/rbd_data.10a56b8b4567.000000000000342e/head//3 e125506b/rbd_data.10a56b8b4567.000000000000342e/head//3(173'2104 client.4302.0:104866 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2104)
2014-06-12 14:51:52.644976 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a2e4506b/rbd_data.10a06b8b4567.0000000000001f10/head//3 a2e4506b/rbd_data.10a06b8b4567.0000000000001f10/head//3(173'1263 client.4297.0:64361 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1263)
2014-06-12 14:51:52.644990 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  4b64506b/rbd_data.10976b8b4567.0000000000002964/head//3 4b64506b/rbd_data.10976b8b4567.0000000000002964/head//3(173'1816 client.4288.0:81976 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1816)
2014-06-12 14:51:52.645005 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7314506b/rbd_data.10b16b8b4567.0000000000002563/head//3 7314506b/rbd_data.10b16b8b4567.0000000000002563/head//3(173'1579 client.4314.0:77151 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1579)
2014-06-12 14:51:52.645019 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7b73506b/rbd_data.10936b8b4567.000000000000143f/head//3 7b73506b/rbd_data.10936b8b4567.000000000000143f/head//3(173'924 client.4284.0:44095 wrlock_by=unknown.0.0:0 dirty s 4194304 uv924)
2014-06-12 14:51:52.645031 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  b6ff406b/rbd_data.10996b8b4567.0000000000000d20/head//3 b6ff406b/rbd_data.10996b8b4567.0000000000000d20/head//3(173'2769 client.4290.0:138441 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2769)
2014-06-12 14:51:52.645045 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a69f406b/rbd_data.10b46b8b4567.0000000000001e2c/head//3 a69f406b/rbd_data.10b46b8b4567.0000000000001e2c/head//3(173'1353 client.4317.0:63565 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1353)
2014-06-12 14:51:52.645060 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  f36e406b/rbd_data.108f6b8b4567.000000000000386f/head//3 f36e406b/rbd_data.108f6b8b4567.000000000000386f/head//3(173'2220 client.4280.0:109813 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2220)
2014-06-12 14:51:52.645074 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a7da406b/rbd_data.10b36b8b4567.0000000000000d72/head//3 a7da406b/rbd_data.10b36b8b4567.0000000000000d72/head//3(173'2731 client.4316.0:140671 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2731)
2014-06-12 14:51:52.645088 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c6a406b/rbd_data.10956b8b4567.0000000000000b95/head//3 c6a406b/rbd_data.10956b8b4567.0000000000000b95/head//3(173'2721 client.4286.0:136702 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2721)
2014-06-12 14:51:52.645102 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  b61a406b/rbd_data.10b16b8b4567.0000000000002d91/head//3 b61a406b/rbd_data.10b16b8b4567.0000000000002d91/head//3(173'1933 client.4314.0:92357 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1933)
2014-06-12 14:51:52.645116 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  8514406b/rbd_data.108f6b8b4567.0000000000000b44/head//3 8514406b/rbd_data.108f6b8b4567.0000000000000b44/head//3(173'2746 client.4280.0:134170 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2746)
2014-06-12 14:51:52.645134 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  2893406b/rbd_data.109c6b8b4567.0000000000000f4f/head//3 2893406b/rbd_data.109c6b8b4567.0000000000000f4f/head//3(173'2852 client.4293.0:142604 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2852)
2014-06-12 14:51:52.645148 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5e33406b/rbd_data.109b6b8b4567.00000000000025d8/head//3 5e33406b/rbd_data.109b6b8b4567.00000000000025d8/head//3(173'1609 client.4292.0:75665 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1609)
2014-06-12 14:51:52.645161 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6342406b/rbd_data.10976b8b4567.0000000000002af1/head//3 6342406b/rbd_data.10976b8b4567.0000000000002af1/head//3(173'1863 client.4288.0:84691 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1863)
2014-06-12 14:51:52.645176 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9612406b/rbd_data.108e6b8b4567.00000000000026f7/head//3 9612406b/rbd_data.108e6b8b4567.00000000000026f7/head//3(173'1586 client.4279.0:79843 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1586)
2014-06-12 14:51:52.645190 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  d0c1406b/rbd_data.10996b8b4567.0000000000000ed8/head//3 d0c1406b/rbd_data.10996b8b4567.0000000000000ed8/head//3(173'2821 client.4290.0:141452 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2821)
2014-06-12 14:51:52.645204 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  3960406b/rbd_data.10a06b8b4567.000000000000384c/head//3 3960406b/rbd_data.10a06b8b4567.000000000000384c/head//3(173'2162 client.4297.0:110230 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2162)
2014-06-12 14:51:52.645216 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  2b10406b/rbd_data.10a66b8b4567.0000000000003ba1/head//3 2b10406b/rbd_data.10a66b8b4567.0000000000003ba1/head//3(173'2388 client.4303.0:118121 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2388)
2014-06-12 14:51:52.645230 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ab9e306b/rbd_data.10b06b8b4567.0000000000003868/head//3 ab9e306b/rbd_data.10b06b8b4567.0000000000003868/head//3(173'2229 client.4313.0:113200 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2229)
2014-06-12 14:51:52.645244 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  3cfa306b/rbd_data.10966b8b4567.0000000000002cf2/head//3 3cfa306b/rbd_data.10966b8b4567.0000000000002cf2/head//3(173'1780 client.4287.0:87592 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1780)
2014-06-12 14:51:52.645258 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  290a306b/rbd_data.109c6b8b4567.0000000000003adb/head//3 290a306b/rbd_data.109c6b8b4567.0000000000003adb/head//3(173'2365 client.4293.0:116169 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2365)
2014-06-12 14:51:52.645269 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:52.645280 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:52.645288 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:52.645290 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:52.645292 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:52.645329 7f9d60bd2700 10 collection_list_partial 3.6b_head start:4bfc506b//0//-1 is_max:0
2014-06-12 14:51:52.645566 7f9d60bd2700 20 objects: [8be0606b/rbd_data.109f6b8b4567.0000000000001d87/head//3,1265606b/rbd_data.10946b8b4567.0000000000000987/head//3,c346606b/rbd_data.10af6b8b4567.0000000000002264/head//3,4407606b/rbd_data.109a6b8b4567.0000000000000689/head//3,f6f7606b/rbd_data.10ad6b8b4567.0000000000002157/head//3,caa606b/rbd_data.10af6b8b4567.0000000000002a63/head//3,c74b606b/rbd_data.10ad6b8b4567.0000000000001dd3/head//3,7cbe606b/rbd_data.10ad6b8b4567.00000000000023aa/head//3,377f606b/rbd_data.10ad6b8b4567.0000000000000647/head//3,6af606b/rbd_data.10ac6b8b4567.0000000000001d93/head//3,d5c2706b/rbd_data.109a6b8b4567.0000000000001be5/head//3,5ea3706b/rbd_data.10916b8b4567.0000000000001ed0/head//3,cea5706b/rbd_data.10b26b8b4567.00000000000011d1/head//3,846706b/rbd_data.10926b8b4567.0000000000002f32/head//3,20d9706b/rbd_data.10936b8b4567.0000000000000ab9/head//3,285f706b/rbd_data.109d6b8b4567.00000000000034f6/head//3,56cf706b/rbd_data.10946b8b4567.0000000000003637/head//3,9c90806b/rbd_data.10b06b8b4567.0000000000002ecc/head//3,7501806b/rbd_data.108e6b8b4567.00000000000036a7/head//3,c7d4806b/rbd_data.10a36b8b4567.000000000000308f/head//3,11f6806b/rbd_data.109b6b8b4567.00000000000023b6/head//3,ae57806b/rbd_data.10976b8b4567.00000000000018bf/head//3,67c7806b/rbd_data.109b6b8b4567.0000000000002558/head//3,a119806b/rbd_data.10a46b8b4567.0000000000003356/head//3,85eb806b/rbd_data.10a66b8b4567.0000000000003905/head//3]
2014-06-12 14:51:52.646135 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  requesting scrubmap from osd.29
2014-06-12 14:51:52.646160 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:52.646177 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:52.646185 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:52.646191 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:52.646197 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:52.646203 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_map_chunk [4bfc506b//0//-1,2e4c806b//0//-1)
2014-06-12 14:51:52.646211 7f9d60bd2700 10 collection_list_partial 3.6b_head start:4bfc506b//0//-1 is_max:0
2014-06-12 14:51:52.646745 7f9d60bd2700 20 objects: [8be0606b/rbd_data.109f6b8b4567.0000000000001d87/head//3,1265606b/rbd_data.10946b8b4567.0000000000000987/head//3,c346606b/rbd_data.10af6b8b4567.0000000000002264/head//3,4407606b/rbd_data.109a6b8b4567.0000000000000689/head//3,f6f7606b/rbd_data.10ad6b8b4567.0000000000002157/head//3,caa606b/rbd_data.10af6b8b4567.0000000000002a63/head//3,c74b606b/rbd_data.10ad6b8b4567.0000000000001dd3/head//3,7cbe606b/rbd_data.10ad6b8b4567.00000000000023aa/head//3,377f606b/rbd_data.10ad6b8b4567.0000000000000647/head//3,6af606b/rbd_data.10ac6b8b4567.0000000000001d93/head//3,d5c2706b/rbd_data.109a6b8b4567.0000000000001be5/head//3,5ea3706b/rbd_data.10916b8b4567.0000000000001ed0/head//3,cea5706b/rbd_data.10b26b8b4567.00000000000011d1/head//3,846706b/rbd_data.10926b8b4567.0000000000002f32/head//3,20d9706b/rbd_data.10936b8b4567.0000000000000ab9/head//3,285f706b/rbd_data.109d6b8b4567.00000000000034f6/head//3,56cf706b/rbd_data.10946b8b4567.0000000000003637/head//3,9c90806b/rbd_data.10b06b8b4567.0000000000002ecc/head//3,7501806b/rbd_data.108e6b8b4567.00000000000036a7/head//3,c7d4806b/rbd_data.10a36b8b4567.000000000000308f/head//3,11f6806b/rbd_data.109b6b8b4567.00000000000023b6/head//3,ae57806b/rbd_data.10976b8b4567.00000000000018bf/head//3,67c7806b/rbd_data.109b6b8b4567.0000000000002558/head//3,a119806b/rbd_data.10a46b8b4567.0000000000003356/head//3,85eb806b/rbd_data.10a66b8b4567.0000000000003905/head//3,2e4c806b/rbd_data.10b26b8b4567.0000000000002c83/head//3,ae806b/rbd_data.10a96b8b4567.00000000000016f0/head//3,c4ce806b/rbd_data.10a56b8b4567.0000000000000a54/head//3,7ede806b/rbd_data.10a56b8b4567.0000000000003710/head//3,6ef2906b/rbd_data.10ad6b8b4567.0000000000003bb1/head//3,4055906b/rbd_data.10946b8b4567.00000000000023c3/head//3,c4c5906b/rbd_data.10956b8b4567.00000000000039c9/head//3,5fc6906b/rbd_data.10946b8b4567.00000000000019d4/head//3,5227906b/rbd_data.10ae6b8b4567.0000000000003486/head//3,dee7906b/rbd_data.10986b8b4567.0000000000000fcb/head//3,fdf7906b/rbd_data.10916b8b4567.00000000000029ec/head//3,ae18906b/rbd_data.109f6b8b4567.000000000000095e/head//3,ccb8906b/rbd_data.10956b8b4567.000000000000218a/head//3,dfb906b/rbd_data.10b36b8b4567.0000000000002a1d/head//3,671c906b/rbd_data.10aa6b8b4567.0000000000000c88/head//3,75fc906b/rbd_data.10a76b8b4567.000000000000013c/head//3,98ad906b/rbd_data.10936b8b4567.00000000000028ea/head//3,84d1a06b/rbd_data.10986b8b4567.00000000000016fb/head//3,ca82a06b/rbd_data.10906b8b4567.0000000000000e65/head//3,5d17a06b/rbd_data.109b6b8b4567.0000000000001096/head//3,4177a06b/rbd_data.10aa6b8b4567.0000000000002a4e/head//3,5f87a06b/rbd_data.109f6b8b4567.00000000000035a4/head//3,6e99a06b/rbd_data.10b26b8b4567.0000000000003841/head//3,494aa06b/rbd_data.10a96b8b4567.0000000000002608/head//3,8fcaa06b/rbd_data.10aa6b8b4567.0000000000002552/head//3,d8bca06b/rbd_data.10a76b8b4567.00000000000032f3/head//3,417ea06b/rbd_data.10b36b8b4567.00000000000023f4/head//3,92cea06b/rbd_data.10ae6b8b4567.0000000000003a91/head//3,9141b06b/rbd_data.10916b8b4567.0000000000003a38/head//3,9132b06b/rbd_data.109b6b8b4567.0000000000001090/head//3,af63b06b/rbd_data.108e6b8b4567.000000000000186f/head//3,e019b06b/rbd_data.10916b8b4567.0000000000002730/head//3,1229b06b/rbd_data.108e6b8b4567.0000000000001fa8/head//3,8069b06b/rbd_data.10936b8b4567.00000000000036fa/head//3,a24ab06b/rbd_data.109d6b8b4567.00000000000033b5/head//3,709ab06b/rbd_data.10986b8b4567.000000000000012a/head//3,290bb06b/rbd_data.10ab6b8b4567.0000000000001011/head//3,7e1cb06b/rbd_data.108f6b8b4567.0000000000002077/head//3,45feb06b/rbd_data.108d6b8b4567.00000000000008d3/head//3]
2014-06-12 14:51:52.646827 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:52.646843 7f9d60bd2700 10 stat 3.6b_head/8be0606b/rbd_data.109f6b8b4567.0000000000001d87/head//3
2014-06-12 14:51:52.646971 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 8be0606b/rbd_data.109f6b8b4567.0000000000001d87/head//3
2014-06-12 14:51:52.662282 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:52.663516 7f9d60bd2700 10 getattrs 3.6b_head/8be0606b/rbd_data.109f6b8b4567.0000000000001d87/head//3 = 0
2014-06-12 14:51:52.663529 7f9d60bd2700 10 stat 3.6b_head/1265606b/rbd_data.10946b8b4567.0000000000000987/head//3
2014-06-12 14:51:52.663640 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1265606b/rbd_data.10946b8b4567.0000000000000987/head//3
2014-06-12 14:51:52.701152 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:52.707943 7f9d60bd2700 10 getattrs 3.6b_head/1265606b/rbd_data.10946b8b4567.0000000000000987/head//3 = 0
2014-06-12 14:51:52.707956 7f9d60bd2700 10 stat 3.6b_head/c346606b/rbd_data.10af6b8b4567.0000000000002264/head//3
2014-06-12 14:51:52.708076 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c346606b/rbd_data.10af6b8b4567.0000000000002264/head//3
2014-06-12 14:51:52.726945 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:52.727252 7f9d60bd2700 10 getattrs 3.6b_head/c346606b/rbd_data.10af6b8b4567.0000000000002264/head//3 = 0
2014-06-12 14:51:52.727267 7f9d60bd2700 10 stat 3.6b_head/4407606b/rbd_data.109a6b8b4567.0000000000000689/head//3
2014-06-12 14:51:52.727391 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 4407606b/rbd_data.109a6b8b4567.0000000000000689/head//3
2014-06-12 14:51:52.764516 7f9d60bd2700 10 getattrs 3.6b_head/4407606b/rbd_data.109a6b8b4567.0000000000000689/head//3 = 0
2014-06-12 14:51:52.764529 7f9d60bd2700 10 stat 3.6b_head/f6f7606b/rbd_data.10ad6b8b4567.0000000000002157/head//3
2014-06-12 14:51:52.764673 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid f6f7606b/rbd_data.10ad6b8b4567.0000000000002157/head//3
2014-06-12 14:51:52.768633 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:52.795156 7f9d60bd2700 10 getattrs 3.6b_head/f6f7606b/rbd_data.10ad6b8b4567.0000000000002157/head//3 = 0
2014-06-12 14:51:52.795166 7f9d60bd2700 10 stat 3.6b_head/caa606b/rbd_data.10af6b8b4567.0000000000002a63/head//3
2014-06-12 14:51:52.795275 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid caa606b/rbd_data.10af6b8b4567.0000000000002a63/head//3
2014-06-12 14:51:52.802538 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:52.817888 7f9d60bd2700 10 getattrs 3.6b_head/caa606b/rbd_data.10af6b8b4567.0000000000002a63/head//3 = 0
2014-06-12 14:51:52.817899 7f9d60bd2700 10 stat 3.6b_head/c74b606b/rbd_data.10ad6b8b4567.0000000000001dd3/head//3
2014-06-12 14:51:52.817993 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c74b606b/rbd_data.10ad6b8b4567.0000000000001dd3/head//3
2014-06-12 14:51:52.832968 7f9d60bd2700 10 getattrs 3.6b_head/c74b606b/rbd_data.10ad6b8b4567.0000000000001dd3/head//3 = 0
2014-06-12 14:51:52.832979 7f9d60bd2700 10 stat 3.6b_head/7cbe606b/rbd_data.10ad6b8b4567.00000000000023aa/head//3
2014-06-12 14:51:52.833076 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7cbe606b/rbd_data.10ad6b8b4567.00000000000023aa/head//3
2014-06-12 14:51:52.849674 7f9d60bd2700 10 getattrs 3.6b_head/7cbe606b/rbd_data.10ad6b8b4567.00000000000023aa/head//3 = 0
2014-06-12 14:51:52.849689 7f9d60bd2700 10 stat 3.6b_head/377f606b/rbd_data.10ad6b8b4567.0000000000000647/head//3
2014-06-12 14:51:52.849849 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 377f606b/rbd_data.10ad6b8b4567.0000000000000647/head//3
2014-06-12 14:51:52.855673 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:52.866049 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:52.869386 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:52.888815 7f9d60bd2700 10 getattrs 3.6b_head/377f606b/rbd_data.10ad6b8b4567.0000000000000647/head//3 = 0
2014-06-12 14:51:52.888825 7f9d60bd2700 10 stat 3.6b_head/6af606b/rbd_data.10ac6b8b4567.0000000000001d93/head//3
2014-06-12 14:51:52.888922 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6af606b/rbd_data.10ac6b8b4567.0000000000001d93/head//3
2014-06-12 14:51:52.908138 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:52.915961 7f9d60bd2700 10 getattrs 3.6b_head/6af606b/rbd_data.10ac6b8b4567.0000000000001d93/head//3 = 0
2014-06-12 14:51:52.915970 7f9d60bd2700 10 stat 3.6b_head/d5c2706b/rbd_data.109a6b8b4567.0000000000001be5/head//3
2014-06-12 14:51:52.916058 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid d5c2706b/rbd_data.109a6b8b4567.0000000000001be5/head//3
2014-06-12 14:51:52.930375 7f9d60bd2700 10 getattrs 3.6b_head/d5c2706b/rbd_data.109a6b8b4567.0000000000001be5/head//3 = 0
2014-06-12 14:51:52.930383 7f9d60bd2700 10 stat 3.6b_head/5ea3706b/rbd_data.10916b8b4567.0000000000001ed0/head//3
2014-06-12 14:51:52.930471 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5ea3706b/rbd_data.10916b8b4567.0000000000001ed0/head//3
2014-06-12 14:51:52.964790 7f9d60bd2700 10 getattrs 3.6b_head/5ea3706b/rbd_data.10916b8b4567.0000000000001ed0/head//3 = 0
2014-06-12 14:51:52.964808 7f9d60bd2700 10 stat 3.6b_head/cea5706b/rbd_data.10b26b8b4567.00000000000011d1/head//3
2014-06-12 14:51:52.964972 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid cea5706b/rbd_data.10b26b8b4567.00000000000011d1/head//3
2014-06-12 14:51:52.966633 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:52.983472 7f9d60bd2700 10 getattrs 3.6b_head/cea5706b/rbd_data.10b26b8b4567.00000000000011d1/head//3 = 0
2014-06-12 14:51:52.983479 7f9d60bd2700 10 stat 3.6b_head/846706b/rbd_data.10926b8b4567.0000000000002f32/head//3
2014-06-12 14:51:52.983573 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 846706b/rbd_data.10926b8b4567.0000000000002f32/head//3
2014-06-12 14:51:52.983929 7f9d60bd2700 10 getattrs 3.6b_head/846706b/rbd_data.10926b8b4567.0000000000002f32/head//3 = 0
2014-06-12 14:51:52.983935 7f9d60bd2700 10 stat 3.6b_head/20d9706b/rbd_data.10936b8b4567.0000000000000ab9/head//3
2014-06-12 14:51:52.984027 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 20d9706b/rbd_data.10936b8b4567.0000000000000ab9/head//3
2014-06-12 14:51:52.995843 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:52.998425 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:52.998474 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:53.015742 7f9d60bd2700 10 getattrs 3.6b_head/20d9706b/rbd_data.10936b8b4567.0000000000000ab9/head//3 = 0
2014-06-12 14:51:53.015752 7f9d60bd2700 10 stat 3.6b_head/285f706b/rbd_data.109d6b8b4567.00000000000034f6/head//3
2014-06-12 14:51:53.015852 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 285f706b/rbd_data.109d6b8b4567.00000000000034f6/head//3
2014-06-12 14:51:53.030714 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:53.031705 7f9d60bd2700 10 getattrs 3.6b_head/285f706b/rbd_data.109d6b8b4567.00000000000034f6/head//3 = 0
2014-06-12 14:51:53.031713 7f9d60bd2700 10 stat 3.6b_head/56cf706b/rbd_data.10946b8b4567.0000000000003637/head//3
2014-06-12 14:51:53.031812 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 56cf706b/rbd_data.10946b8b4567.0000000000003637/head//3
2014-06-12 14:51:53.042323 7f9d60bd2700 10 getattrs 3.6b_head/56cf706b/rbd_data.10946b8b4567.0000000000003637/head//3 = 0
2014-06-12 14:51:53.042332 7f9d60bd2700 10 stat 3.6b_head/9c90806b/rbd_data.10b06b8b4567.0000000000002ecc/head//3
2014-06-12 14:51:53.042436 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9c90806b/rbd_data.10b06b8b4567.0000000000002ecc/head//3
2014-06-12 14:51:53.075565 7f9d60bd2700 10 getattrs 3.6b_head/9c90806b/rbd_data.10b06b8b4567.0000000000002ecc/head//3 = 0
2014-06-12 14:51:53.075576 7f9d60bd2700 10 stat 3.6b_head/7501806b/rbd_data.108e6b8b4567.00000000000036a7/head//3
2014-06-12 14:51:53.075683 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7501806b/rbd_data.108e6b8b4567.00000000000036a7/head//3
2014-06-12 14:51:53.089385 7f9d60bd2700 10 getattrs 3.6b_head/7501806b/rbd_data.108e6b8b4567.00000000000036a7/head//3 = 0
2014-06-12 14:51:53.089394 7f9d60bd2700 10 stat 3.6b_head/c7d4806b/rbd_data.10a36b8b4567.000000000000308f/head//3
2014-06-12 14:51:53.089503 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c7d4806b/rbd_data.10a36b8b4567.000000000000308f/head//3
2014-06-12 14:51:53.116947 7f9d60bd2700 10 getattrs 3.6b_head/c7d4806b/rbd_data.10a36b8b4567.000000000000308f/head//3 = 0
2014-06-12 14:51:53.116958 7f9d60bd2700 10 stat 3.6b_head/11f6806b/rbd_data.109b6b8b4567.00000000000023b6/head//3
2014-06-12 14:51:53.117066 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 11f6806b/rbd_data.109b6b8b4567.00000000000023b6/head//3
2014-06-12 14:51:53.121997 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:53.133607 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:53.142998 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:53.145185 7f9d60bd2700 10 getattrs 3.6b_head/11f6806b/rbd_data.109b6b8b4567.00000000000023b6/head//3 = 0
2014-06-12 14:51:53.145195 7f9d60bd2700 10 stat 3.6b_head/ae57806b/rbd_data.10976b8b4567.00000000000018bf/head//3
2014-06-12 14:51:53.145321 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ae57806b/rbd_data.10976b8b4567.00000000000018bf/head//3
2014-06-12 14:51:53.152424 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:53.157085 7f9d60bd2700 10 getattrs 3.6b_head/ae57806b/rbd_data.10976b8b4567.00000000000018bf/head//3 = 0
2014-06-12 14:51:53.157092 7f9d60bd2700 10 stat 3.6b_head/67c7806b/rbd_data.109b6b8b4567.0000000000002558/head//3
2014-06-12 14:51:53.157184 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 67c7806b/rbd_data.109b6b8b4567.0000000000002558/head//3
2014-06-12 14:51:53.172924 7f9d60bd2700 10 getattrs 3.6b_head/67c7806b/rbd_data.109b6b8b4567.0000000000002558/head//3 = 0
2014-06-12 14:51:53.172933 7f9d60bd2700 10 stat 3.6b_head/a119806b/rbd_data.10a46b8b4567.0000000000003356/head//3
2014-06-12 14:51:53.173037 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a119806b/rbd_data.10a46b8b4567.0000000000003356/head//3
2014-06-12 14:51:53.180904 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:53.190567 7f9d5501e700  5 -- op tracker -- , seq: 748610, time: 2014-06-12 14:51:53.190494, event: header_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.190577 7f9d5501e700  5 -- op tracker -- , seq: 748610, time: 2014-06-12 14:51:53.190496, event: throttled, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.190583 7f9d5501e700  5 -- op tracker -- , seq: 748610, time: 2014-06-12 14:51:53.190538, event: all_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.190589 7f9d5501e700  5 -- op tracker -- , seq: 748610, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.190594 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:53.190599 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:53.190605 7f9d5501e700 15 osd.2 173 enqueue_op 0x3b42b500 prio 127 cost 10717 latency 0.000111 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:53.190673 7f9d623d5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:53.196599 7f9d60bd2700 10 getattrs 3.6b_head/a119806b/rbd_data.10a46b8b4567.0000000000003356/head//3 = 0
2014-06-12 14:51:53.196608 7f9d60bd2700 10 stat 3.6b_head/85eb806b/rbd_data.10a66b8b4567.0000000000003905/head//3
2014-06-12 14:51:53.196707 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 85eb806b/rbd_data.10a66b8b4567.0000000000003905/head//3
2014-06-12 14:51:53.203166 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:53.203414 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:53.217397 7f9d60bd2700 10 getattrs 3.6b_head/85eb806b/rbd_data.10a66b8b4567.0000000000003905/head//3 = 0
2014-06-12 14:51:53.217411 7f9d60bd2700 10 collection_getattrs 3.6b_head
2014-06-12 14:51:53.217490 7f9d60bd2700 10 collection_getattrs 3.6b_head = 0
2014-06-12 14:51:53.217495 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:53.217536 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:53.217553 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:53.217569 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:53.217574 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:53.217601 7f9d623d5700 10 osd.2 173 dequeue_op 0x3b42b500 prio 127 cost 10717 latency 0.027106 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]
2014-06-12 14:51:53.217638 7f9d623d5700  5 -- op tracker -- , seq: 748610, time: 2014-06-12 14:51:53.217634, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.217649 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] handle_message: 0x3b42b500
2014-06-12 14:51:53.217659 7f9d623d5700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:53.217668 7f9d623d5700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:53.217678 7f9d623d5700  5 -- op tracker -- , seq: 748610, time: 2014-06-12 14:51:53.217675, event: started, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.217812 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  got 29 scrub map
2014-06-12 14:51:53.217945 7f9d623d5700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] map version is 173'2887
2014-06-12 14:51:53.217969 7f9d623d5700 10 osd.2 173 dequeue_op 0x3b42b500 finish
2014-06-12 14:51:53.217983 7f9d623d5700  5 -- op tracker -- , seq: 748610, time: 2014-06-12 14:51:53.217975, event: done, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.218003 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:53.218010 7f9d623d5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:53.218017 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:53.218017 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:53.218039 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:53.218059 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:53.218085 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:53.218092 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:53.218099 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub replica 29 has 25 items
2014-06-12 14:51:53.218120 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8be0606b/rbd_data.109f6b8b4567.0000000000001d87/head//3, auth == maps.end()
2014-06-12 14:51:53.218145 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 8be0606b/rbd_data.109f6b8b4567.0000000000001d87/head//3
2014-06-12 14:51:53.218163 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1265606b/rbd_data.10946b8b4567.0000000000000987/head//3, auth == maps.end()
2014-06-12 14:51:53.218175 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1265606b/rbd_data.10946b8b4567.0000000000000987/head//3
2014-06-12 14:51:53.218187 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c346606b/rbd_data.10af6b8b4567.0000000000002264/head//3, auth == maps.end()
2014-06-12 14:51:53.218198 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c346606b/rbd_data.10af6b8b4567.0000000000002264/head//3
2014-06-12 14:51:53.218210 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4407606b/rbd_data.109a6b8b4567.0000000000000689/head//3, auth == maps.end()
2014-06-12 14:51:53.218220 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 4407606b/rbd_data.109a6b8b4567.0000000000000689/head//3
2014-06-12 14:51:53.218231 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f6f7606b/rbd_data.10ad6b8b4567.0000000000002157/head//3, auth == maps.end()
2014-06-12 14:51:53.218240 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj f6f7606b/rbd_data.10ad6b8b4567.0000000000002157/head//3
2014-06-12 14:51:53.218249 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj caa606b/rbd_data.10af6b8b4567.0000000000002a63/head//3, auth == maps.end()
2014-06-12 14:51:53.218259 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj caa606b/rbd_data.10af6b8b4567.0000000000002a63/head//3
2014-06-12 14:51:53.218269 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c74b606b/rbd_data.10ad6b8b4567.0000000000001dd3/head//3, auth == maps.end()
2014-06-12 14:51:53.218277 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c74b606b/rbd_data.10ad6b8b4567.0000000000001dd3/head//3
2014-06-12 14:51:53.218286 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7cbe606b/rbd_data.10ad6b8b4567.00000000000023aa/head//3, auth == maps.end()
2014-06-12 14:51:53.218295 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7cbe606b/rbd_data.10ad6b8b4567.00000000000023aa/head//3
2014-06-12 14:51:53.218304 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 377f606b/rbd_data.10ad6b8b4567.0000000000000647/head//3, auth == maps.end()
2014-06-12 14:51:53.218313 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 377f606b/rbd_data.10ad6b8b4567.0000000000000647/head//3
2014-06-12 14:51:53.218322 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6af606b/rbd_data.10ac6b8b4567.0000000000001d93/head//3, auth == maps.end()
2014-06-12 14:51:53.218331 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6af606b/rbd_data.10ac6b8b4567.0000000000001d93/head//3
2014-06-12 14:51:53.218340 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d5c2706b/rbd_data.109a6b8b4567.0000000000001be5/head//3, auth == maps.end()
2014-06-12 14:51:53.218348 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj d5c2706b/rbd_data.109a6b8b4567.0000000000001be5/head//3
2014-06-12 14:51:53.218357 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5ea3706b/rbd_data.10916b8b4567.0000000000001ed0/head//3, auth == maps.end()
2014-06-12 14:51:53.218367 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5ea3706b/rbd_data.10916b8b4567.0000000000001ed0/head//3
2014-06-12 14:51:53.218377 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cea5706b/rbd_data.10b26b8b4567.00000000000011d1/head//3, auth == maps.end()
2014-06-12 14:51:53.218385 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj cea5706b/rbd_data.10b26b8b4567.00000000000011d1/head//3
2014-06-12 14:51:53.218396 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 846706b/rbd_data.10926b8b4567.0000000000002f32/head//3, auth == maps.end()
2014-06-12 14:51:53.218405 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 846706b/rbd_data.10926b8b4567.0000000000002f32/head//3
2014-06-12 14:51:53.218416 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 20d9706b/rbd_data.10936b8b4567.0000000000000ab9/head//3, auth == maps.end()
2014-06-12 14:51:53.218424 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 20d9706b/rbd_data.10936b8b4567.0000000000000ab9/head//3
2014-06-12 14:51:53.218435 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 285f706b/rbd_data.109d6b8b4567.00000000000034f6/head//3, auth == maps.end()
2014-06-12 14:51:53.218444 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 285f706b/rbd_data.109d6b8b4567.00000000000034f6/head//3
2014-06-12 14:51:53.218455 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 56cf706b/rbd_data.10946b8b4567.0000000000003637/head//3, auth == maps.end()
2014-06-12 14:51:53.218469 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 56cf706b/rbd_data.10946b8b4567.0000000000003637/head//3
2014-06-12 14:51:53.218480 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9c90806b/rbd_data.10b06b8b4567.0000000000002ecc/head//3, auth == maps.end()
2014-06-12 14:51:53.218490 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9c90806b/rbd_data.10b06b8b4567.0000000000002ecc/head//3
2014-06-12 14:51:53.218501 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7501806b/rbd_data.108e6b8b4567.00000000000036a7/head//3, auth == maps.end()
2014-06-12 14:51:53.218511 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7501806b/rbd_data.108e6b8b4567.00000000000036a7/head//3
2014-06-12 14:51:53.218520 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c7d4806b/rbd_data.10a36b8b4567.000000000000308f/head//3, auth == maps.end()
2014-06-12 14:51:53.218530 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c7d4806b/rbd_data.10a36b8b4567.000000000000308f/head//3
2014-06-12 14:51:53.218540 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 11f6806b/rbd_data.109b6b8b4567.00000000000023b6/head//3, auth == maps.end()
2014-06-12 14:51:53.218550 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 11f6806b/rbd_data.109b6b8b4567.00000000000023b6/head//3
2014-06-12 14:51:53.218560 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ae57806b/rbd_data.10976b8b4567.00000000000018bf/head//3, auth == maps.end()
2014-06-12 14:51:53.218570 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ae57806b/rbd_data.10976b8b4567.00000000000018bf/head//3
2014-06-12 14:51:53.218579 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 67c7806b/rbd_data.109b6b8b4567.0000000000002558/head//3, auth == maps.end()
2014-06-12 14:51:53.218590 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 67c7806b/rbd_data.109b6b8b4567.0000000000002558/head//3
2014-06-12 14:51:53.218601 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a119806b/rbd_data.10a46b8b4567.0000000000003356/head//3, auth == maps.end()
2014-06-12 14:51:53.218611 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a119806b/rbd_data.10a46b8b4567.0000000000003356/head//3
2014-06-12 14:51:53.218621 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 85eb806b/rbd_data.10a66b8b4567.0000000000003905/head//3, auth == maps.end()
2014-06-12 14:51:53.218632 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 85eb806b/rbd_data.10a66b8b4567.0000000000003905/head//3
2014-06-12 14:51:53.218644 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] 
2014-06-12 14:51:53.218652 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub
2014-06-12 14:51:53.218665 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  85eb806b/rbd_data.10a66b8b4567.0000000000003905/head//3 85eb806b/rbd_data.10a66b8b4567.0000000000003905/head//3(173'2342 client.4303.0:112984 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2342)
2014-06-12 14:51:53.218680 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a119806b/rbd_data.10a46b8b4567.0000000000003356/head//3 a119806b/rbd_data.10a46b8b4567.0000000000003356/head//3(173'2067 client.4301.0:103202 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2067)
2014-06-12 14:51:53.218692 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  67c7806b/rbd_data.109b6b8b4567.0000000000002558/head//3 67c7806b/rbd_data.109b6b8b4567.0000000000002558/head//3(173'1595 client.4292.0:74608 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1595)
2014-06-12 14:51:53.218703 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ae57806b/rbd_data.10976b8b4567.00000000000018bf/head//3 ae57806b/rbd_data.10976b8b4567.00000000000018bf/head//3(173'1186 client.4288.0:52832 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1186)
2014-06-12 14:51:53.218716 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  11f6806b/rbd_data.109b6b8b4567.00000000000023b6/head//3 11f6806b/rbd_data.109b6b8b4567.00000000000023b6/head//3(173'1521 client.4292.0:71495 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1521)
2014-06-12 14:51:53.218727 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c7d4806b/rbd_data.10a36b8b4567.000000000000308f/head//3 c7d4806b/rbd_data.10a36b8b4567.000000000000308f/head//3(173'1872 client.4300.0:95040 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1872)
2014-06-12 14:51:53.218738 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7501806b/rbd_data.108e6b8b4567.00000000000036a7/head//3 7501806b/rbd_data.108e6b8b4567.00000000000036a7/head//3(173'2134 client.4279.0:107830 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2134)
2014-06-12 14:51:53.218751 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9c90806b/rbd_data.10b06b8b4567.0000000000002ecc/head//3 9c90806b/rbd_data.10b06b8b4567.0000000000002ecc/head//3(173'1857 client.4313.0:95080 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1857)
2014-06-12 14:51:53.218763 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  56cf706b/rbd_data.10946b8b4567.0000000000003637/head//3 56cf706b/rbd_data.10946b8b4567.0000000000003637/head//3(173'2111 client.4285.0:106105 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2111)
2014-06-12 14:51:53.218776 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  285f706b/rbd_data.109d6b8b4567.00000000000034f6/head//3 285f706b/rbd_data.109d6b8b4567.00000000000034f6/head//3(173'2091 client.4294.0:104648 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2091)
2014-06-12 14:51:53.218789 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  20d9706b/rbd_data.10936b8b4567.0000000000000ab9/head//3 20d9706b/rbd_data.10936b8b4567.0000000000000ab9/head//3(173'2713 client.4284.0:132567 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2713)
2014-06-12 14:51:53.218802 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  846706b/rbd_data.10926b8b4567.0000000000002f32/head//3 846706b/rbd_data.10926b8b4567.0000000000002f32/head//3(173'1962 client.4283.0:94269 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1962)
2014-06-12 14:51:53.218814 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  cea5706b/rbd_data.10b26b8b4567.00000000000011d1/head//3 cea5706b/rbd_data.10b26b8b4567.00000000000011d1/head//3(173'770 client.4315.0:41256 wrlock_by=unknown.0.0:0 dirty s 4194304 uv770)
2014-06-12 14:51:53.218827 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5ea3706b/rbd_data.10916b8b4567.0000000000001ed0/head//3 5ea3706b/rbd_data.10916b8b4567.0000000000001ed0/head//3(173'1335 client.4282.0:65234 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1335)
2014-06-12 14:51:53.218839 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  d5c2706b/rbd_data.109a6b8b4567.0000000000001be5/head//3 d5c2706b/rbd_data.109a6b8b4567.0000000000001be5/head//3(173'1158 client.4291.0:59188 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1158)
2014-06-12 14:51:53.218853 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6af606b/rbd_data.10ac6b8b4567.0000000000001d93/head//3 6af606b/rbd_data.10ac6b8b4567.0000000000001d93/head//3(173'1252 client.4309.0:61544 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1252)
2014-06-12 14:51:53.218866 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  377f606b/rbd_data.10ad6b8b4567.0000000000000647/head//3 377f606b/rbd_data.10ad6b8b4567.0000000000000647/head//3(173'2560 client.4310.0:127144 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2560)
2014-06-12 14:51:53.218878 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7cbe606b/rbd_data.10ad6b8b4567.00000000000023aa/head//3 7cbe606b/rbd_data.10ad6b8b4567.00000000000023aa/head//3(173'1497 client.4310.0:73081 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1497)
2014-06-12 14:51:53.218891 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c74b606b/rbd_data.10ad6b8b4567.0000000000001dd3/head//3 c74b606b/rbd_data.10ad6b8b4567.0000000000001dd3/head//3(173'1276 client.4310.0:63018 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1276)
2014-06-12 14:51:53.218904 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  caa606b/rbd_data.10af6b8b4567.0000000000002a63/head//3 caa606b/rbd_data.10af6b8b4567.0000000000002a63/head//3(173'1750 client.4312.0:88067 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1750)
2014-06-12 14:51:53.218916 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  f6f7606b/rbd_data.10ad6b8b4567.0000000000002157/head//3 f6f7606b/rbd_data.10ad6b8b4567.0000000000002157/head//3(173'1400 client.4310.0:69130 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1400)
2014-06-12 14:51:53.218929 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  4407606b/rbd_data.109a6b8b4567.0000000000000689/head//3 4407606b/rbd_data.109a6b8b4567.0000000000000689/head//3(173'2545 client.4291.0:126081 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2545)
2014-06-12 14:51:53.218942 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c346606b/rbd_data.10af6b8b4567.0000000000002264/head//3 c346606b/rbd_data.10af6b8b4567.0000000000002264/head//3(173'1446 client.4312.0:73397 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1446)
2014-06-12 14:51:53.218954 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1265606b/rbd_data.10946b8b4567.0000000000000987/head//3 1265606b/rbd_data.10946b8b4567.0000000000000987/head//3(173'2665 client.4285.0:131516 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2665)
2014-06-12 14:51:53.218965 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  8be0606b/rbd_data.109f6b8b4567.0000000000001d87/head//3 8be0606b/rbd_data.109f6b8b4567.0000000000001d87/head//3(173'1134 client.4296.0:61600 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1134)
2014-06-12 14:51:53.218975 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:53.218986 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:53.218994 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:53.218996 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:53.218998 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:53.219036 7f9d60bd2700 10 collection_list_partial 3.6b_head start:2e4c806b//0//-1 is_max:0
2014-06-12 14:51:53.219250 7f9d60bd2700 20 objects: [ae806b/rbd_data.10a96b8b4567.00000000000016f0/head//3,c4ce806b/rbd_data.10a56b8b4567.0000000000000a54/head//3,7ede806b/rbd_data.10a56b8b4567.0000000000003710/head//3,6ef2906b/rbd_data.10ad6b8b4567.0000000000003bb1/head//3,4055906b/rbd_data.10946b8b4567.00000000000023c3/head//3,c4c5906b/rbd_data.10956b8b4567.00000000000039c9/head//3,5fc6906b/rbd_data.10946b8b4567.00000000000019d4/head//3,5227906b/rbd_data.10ae6b8b4567.0000000000003486/head//3,dee7906b/rbd_data.10986b8b4567.0000000000000fcb/head//3,fdf7906b/rbd_data.10916b8b4567.00000000000029ec/head//3,ae18906b/rbd_data.109f6b8b4567.000000000000095e/head//3,ccb8906b/rbd_data.10956b8b4567.000000000000218a/head//3,dfb906b/rbd_data.10b36b8b4567.0000000000002a1d/head//3,671c906b/rbd_data.10aa6b8b4567.0000000000000c88/head//3,75fc906b/rbd_data.10a76b8b4567.000000000000013c/head//3,98ad906b/rbd_data.10936b8b4567.00000000000028ea/head//3,84d1a06b/rbd_data.10986b8b4567.00000000000016fb/head//3,ca82a06b/rbd_data.10906b8b4567.0000000000000e65/head//3,5d17a06b/rbd_data.109b6b8b4567.0000000000001096/head//3,4177a06b/rbd_data.10aa6b8b4567.0000000000002a4e/head//3,5f87a06b/rbd_data.109f6b8b4567.00000000000035a4/head//3,6e99a06b/rbd_data.10b26b8b4567.0000000000003841/head//3,494aa06b/rbd_data.10a96b8b4567.0000000000002608/head//3,8fcaa06b/rbd_data.10aa6b8b4567.0000000000002552/head//3,d8bca06b/rbd_data.10a76b8b4567.00000000000032f3/head//3]
2014-06-12 14:51:53.219776 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  requesting scrubmap from osd.29
2014-06-12 14:51:53.219801 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:53.219816 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:53.219824 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:53.219830 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:53.219836 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:53.219841 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_map_chunk [2e4c806b//0//-1,417ea06b//0//-1)
2014-06-12 14:51:53.219849 7f9d60bd2700 10 collection_list_partial 3.6b_head start:2e4c806b//0//-1 is_max:0
2014-06-12 14:51:53.220335 7f9d60bd2700 20 objects: [ae806b/rbd_data.10a96b8b4567.00000000000016f0/head//3,c4ce806b/rbd_data.10a56b8b4567.0000000000000a54/head//3,7ede806b/rbd_data.10a56b8b4567.0000000000003710/head//3,6ef2906b/rbd_data.10ad6b8b4567.0000000000003bb1/head//3,4055906b/rbd_data.10946b8b4567.00000000000023c3/head//3,c4c5906b/rbd_data.10956b8b4567.00000000000039c9/head//3,5fc6906b/rbd_data.10946b8b4567.00000000000019d4/head//3,5227906b/rbd_data.10ae6b8b4567.0000000000003486/head//3,dee7906b/rbd_data.10986b8b4567.0000000000000fcb/head//3,fdf7906b/rbd_data.10916b8b4567.00000000000029ec/head//3,ae18906b/rbd_data.109f6b8b4567.000000000000095e/head//3,ccb8906b/rbd_data.10956b8b4567.000000000000218a/head//3,dfb906b/rbd_data.10b36b8b4567.0000000000002a1d/head//3,671c906b/rbd_data.10aa6b8b4567.0000000000000c88/head//3,75fc906b/rbd_data.10a76b8b4567.000000000000013c/head//3,98ad906b/rbd_data.10936b8b4567.00000000000028ea/head//3,84d1a06b/rbd_data.10986b8b4567.00000000000016fb/head//3,ca82a06b/rbd_data.10906b8b4567.0000000000000e65/head//3,5d17a06b/rbd_data.109b6b8b4567.0000000000001096/head//3,4177a06b/rbd_data.10aa6b8b4567.0000000000002a4e/head//3,5f87a06b/rbd_data.109f6b8b4567.00000000000035a4/head//3,6e99a06b/rbd_data.10b26b8b4567.0000000000003841/head//3,494aa06b/rbd_data.10a96b8b4567.0000000000002608/head//3,8fcaa06b/rbd_data.10aa6b8b4567.0000000000002552/head//3,d8bca06b/rbd_data.10a76b8b4567.00000000000032f3/head//3,417ea06b/rbd_data.10b36b8b4567.00000000000023f4/head//3,92cea06b/rbd_data.10ae6b8b4567.0000000000003a91/head//3,9141b06b/rbd_data.10916b8b4567.0000000000003a38/head//3,9132b06b/rbd_data.109b6b8b4567.0000000000001090/head//3,af63b06b/rbd_data.108e6b8b4567.000000000000186f/head//3,e019b06b/rbd_data.10916b8b4567.0000000000002730/head//3,1229b06b/rbd_data.108e6b8b4567.0000000000001fa8/head//3,8069b06b/rbd_data.10936b8b4567.00000000000036fa/head//3,a24ab06b/rbd_data.109d6b8b4567.00000000000033b5/head//3,709ab06b/rbd_data.10986b8b4567.000000000000012a/head//3,290bb06b/rbd_data.10ab6b8b4567.0000000000001011/head//3,7e1cb06b/rbd_data.108f6b8b4567.0000000000002077/head//3,45feb06b/rbd_data.108d6b8b4567.00000000000008d3/head//3,6f0c06b/rbd_data.10b46b8b4567.000000000000378c/head//3,85c3c06b/rbd_data.10926b8b4567.0000000000002e09/head//3,1b04c06b/rbd_data.10976b8b4567.000000000000395f/head//3,6c45c06b/rbd_data.108d6b8b4567.000000000000360d/head//3,2d76c06b/rbd_data.109a6b8b4567.0000000000001ab0/head//3,cc47c06b/rbd_data.109a6b8b4567.0000000000001429/head//3,4d68c06b/rbd_data.10a86b8b4567.0000000000002a35/head//3,6eb8c06b/rbd_data.108f6b8b4567.0000000000002155/head//3,8d09c06b/rbd_data.10956b8b4567.00000000000032f3/head//3,ed8bc06b/rbd_data.10976b8b4567.0000000000000081/head//3,535dc06b/rbd_data.109d6b8b4567.0000000000000f84/head//3,984ec06b/rbd_data.10ad6b8b4567.0000000000000a62/head//3,ce81d06b/rbd_data.109b6b8b4567.0000000000002abd/head//3,7a74d06b/rbd_data.10976b8b4567.00000000000010c0/head//3,4175d06b/rbd_data.10b16b8b4567.00000000000001b4/head//3,3bfcd06b/rbd_data.10926b8b4567.0000000000001ea1/head//3,953fd06b/rbd_data.10a16b8b4567.00000000000018cb/head//3,a76fd06b/rbd_data.108f6b8b4567.0000000000000081/head//3,8121e06b/rbd_data.10976b8b4567.000000000000075c/head//3,2cc2e06b/rbd_data.10b06b8b4567.0000000000002075/head//3,5066e06b/rbd_data.10b46b8b4567.0000000000001a06/head//3,7268e06b/rbd_data.10a86b8b4567.0000000000002d74/head//3,a8de06b/rbd_data.108e6b8b4567.0000000000000d1b/head//3,eac1f06b/rbd_data.10b46b8b4567.0000000000002b66/head//3,5562f06b/rbd_data.109a6b8b4567.0000000000003402/head//3,dea5f06b/rbd_data.10ad6b8b4567.0000000000000c8a/head//3]
2014-06-12 14:51:53.220406 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:53.220420 7f9d60bd2700 10 stat 3.6b_head/ae806b/rbd_data.10a96b8b4567.00000000000016f0/head//3
2014-06-12 14:51:53.220555 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ae806b/rbd_data.10a96b8b4567.00000000000016f0/head//3
2014-06-12 14:51:53.237689 7f9d60bd2700 10 getattrs 3.6b_head/ae806b/rbd_data.10a96b8b4567.00000000000016f0/head//3 = 0
2014-06-12 14:51:53.237700 7f9d60bd2700 10 stat 3.6b_head/c4ce806b/rbd_data.10a56b8b4567.0000000000000a54/head//3
2014-06-12 14:51:53.237790 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c4ce806b/rbd_data.10a56b8b4567.0000000000000a54/head//3
2014-06-12 14:51:53.237903 7f9d60bd2700 10 getattrs 3.6b_head/c4ce806b/rbd_data.10a56b8b4567.0000000000000a54/head//3 = 0
2014-06-12 14:51:53.237908 7f9d60bd2700 10 stat 3.6b_head/7ede806b/rbd_data.10a56b8b4567.0000000000003710/head//3
2014-06-12 14:51:53.237995 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7ede806b/rbd_data.10a56b8b4567.0000000000003710/head//3
2014-06-12 14:51:53.264861 7f9d60bd2700 10 getattrs 3.6b_head/7ede806b/rbd_data.10a56b8b4567.0000000000003710/head//3 = 0
2014-06-12 14:51:53.264869 7f9d60bd2700 10 stat 3.6b_head/6ef2906b/rbd_data.10ad6b8b4567.0000000000003bb1/head//3
2014-06-12 14:51:53.264966 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6ef2906b/rbd_data.10ad6b8b4567.0000000000003bb1/head//3
2014-06-12 14:51:53.280290 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:53.280318 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:53.280323 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:53.280325 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:53.280326 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:53.281373 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:53.281916 7f9d60bd2700 10 getattrs 3.6b_head/6ef2906b/rbd_data.10ad6b8b4567.0000000000003bb1/head//3 = 0
2014-06-12 14:51:53.281927 7f9d60bd2700 10 stat 3.6b_head/4055906b/rbd_data.10946b8b4567.00000000000023c3/head//3
2014-06-12 14:51:53.282024 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 4055906b/rbd_data.10946b8b4567.00000000000023c3/head//3
2014-06-12 14:51:53.282104 7f9d60bd2700 10 getattrs 3.6b_head/4055906b/rbd_data.10946b8b4567.00000000000023c3/head//3 = 0
2014-06-12 14:51:53.282109 7f9d60bd2700 10 stat 3.6b_head/c4c5906b/rbd_data.10956b8b4567.00000000000039c9/head//3
2014-06-12 14:51:53.282198 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c4c5906b/rbd_data.10956b8b4567.00000000000039c9/head//3
2014-06-12 14:51:53.299069 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:53.308200 7f9d60bd2700 10 getattrs 3.6b_head/c4c5906b/rbd_data.10956b8b4567.00000000000039c9/head//3 = 0
2014-06-12 14:51:53.308215 7f9d60bd2700 10 stat 3.6b_head/5fc6906b/rbd_data.10946b8b4567.00000000000019d4/head//3
2014-06-12 14:51:53.308411 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5fc6906b/rbd_data.10946b8b4567.00000000000019d4/head//3
2014-06-12 14:51:53.341532 7f9d60bd2700 10 getattrs 3.6b_head/5fc6906b/rbd_data.10946b8b4567.00000000000019d4/head//3 = 0
2014-06-12 14:51:53.341539 7f9d60bd2700 10 stat 3.6b_head/5227906b/rbd_data.10ae6b8b4567.0000000000003486/head//3
2014-06-12 14:51:53.341674 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5227906b/rbd_data.10ae6b8b4567.0000000000003486/head//3
2014-06-12 14:51:53.342209 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:53.356123 7f9d60bd2700 10 getattrs 3.6b_head/5227906b/rbd_data.10ae6b8b4567.0000000000003486/head//3 = 0
2014-06-12 14:51:53.356142 7f9d60bd2700 10 stat 3.6b_head/dee7906b/rbd_data.10986b8b4567.0000000000000fcb/head//3
2014-06-12 14:51:53.356240 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid dee7906b/rbd_data.10986b8b4567.0000000000000fcb/head//3
2014-06-12 14:51:53.375547 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:53.387154 7f9d60bd2700 10 getattrs 3.6b_head/dee7906b/rbd_data.10986b8b4567.0000000000000fcb/head//3 = 0
2014-06-12 14:51:53.387170 7f9d60bd2700 10 stat 3.6b_head/fdf7906b/rbd_data.10916b8b4567.00000000000029ec/head//3
2014-06-12 14:51:53.387336 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid fdf7906b/rbd_data.10916b8b4567.00000000000029ec/head//3
2014-06-12 14:51:53.400345 7f9d60bd2700 10 getattrs 3.6b_head/fdf7906b/rbd_data.10916b8b4567.00000000000029ec/head//3 = 0
2014-06-12 14:51:53.400358 7f9d60bd2700 10 stat 3.6b_head/ae18906b/rbd_data.109f6b8b4567.000000000000095e/head//3
2014-06-12 14:51:53.400510 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ae18906b/rbd_data.109f6b8b4567.000000000000095e/head//3
2014-06-12 14:51:53.428734 7f9d60bd2700 10 getattrs 3.6b_head/ae18906b/rbd_data.109f6b8b4567.000000000000095e/head//3 = 0
2014-06-12 14:51:53.428741 7f9d60bd2700 10 stat 3.6b_head/ccb8906b/rbd_data.10956b8b4567.000000000000218a/head//3
2014-06-12 14:51:53.428836 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ccb8906b/rbd_data.10956b8b4567.000000000000218a/head//3
2014-06-12 14:51:53.464459 7f9d60bd2700 10 getattrs 3.6b_head/ccb8906b/rbd_data.10956b8b4567.000000000000218a/head//3 = 0
2014-06-12 14:51:53.464471 7f9d60bd2700 10 stat 3.6b_head/dfb906b/rbd_data.10b36b8b4567.0000000000002a1d/head//3
2014-06-12 14:51:53.464588 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid dfb906b/rbd_data.10b36b8b4567.0000000000002a1d/head//3
2014-06-12 14:51:53.482795 7f9d60bd2700 10 getattrs 3.6b_head/dfb906b/rbd_data.10b36b8b4567.0000000000002a1d/head//3 = 0
2014-06-12 14:51:53.482808 7f9d60bd2700 10 stat 3.6b_head/671c906b/rbd_data.10aa6b8b4567.0000000000000c88/head//3
2014-06-12 14:51:53.482921 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 671c906b/rbd_data.10aa6b8b4567.0000000000000c88/head//3
2014-06-12 14:51:53.514019 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:53.519055 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:53.530307 7f9d60bd2700 10 getattrs 3.6b_head/671c906b/rbd_data.10aa6b8b4567.0000000000000c88/head//3 = 0
2014-06-12 14:51:53.530323 7f9d60bd2700 10 stat 3.6b_head/75fc906b/rbd_data.10a76b8b4567.000000000000013c/head//3
2014-06-12 14:51:53.530492 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 75fc906b/rbd_data.10a76b8b4567.000000000000013c/head//3
2014-06-12 14:51:53.578128 7f9d60bd2700 10 getattrs 3.6b_head/75fc906b/rbd_data.10a76b8b4567.000000000000013c/head//3 = 0
2014-06-12 14:51:53.578168 7f9d60bd2700 10 stat 3.6b_head/98ad906b/rbd_data.10936b8b4567.00000000000028ea/head//3
2014-06-12 14:51:53.578313 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 98ad906b/rbd_data.10936b8b4567.00000000000028ea/head//3
2014-06-12 14:51:53.609200 7f9d60bd2700 10 getattrs 3.6b_head/98ad906b/rbd_data.10936b8b4567.00000000000028ea/head//3 = 0
2014-06-12 14:51:53.609219 7f9d60bd2700 10 stat 3.6b_head/84d1a06b/rbd_data.10986b8b4567.00000000000016fb/head//3
2014-06-12 14:51:53.609360 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 84d1a06b/rbd_data.10986b8b4567.00000000000016fb/head//3
2014-06-12 14:51:53.623958 7f9d60bd2700 10 getattrs 3.6b_head/84d1a06b/rbd_data.10986b8b4567.00000000000016fb/head//3 = 0
2014-06-12 14:51:53.623974 7f9d60bd2700 10 stat 3.6b_head/ca82a06b/rbd_data.10906b8b4567.0000000000000e65/head//3
2014-06-12 14:51:53.624085 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ca82a06b/rbd_data.10906b8b4567.0000000000000e65/head//3
2014-06-12 14:51:53.641128 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:53.676781 7f9d60bd2700 10 getattrs 3.6b_head/ca82a06b/rbd_data.10906b8b4567.0000000000000e65/head//3 = 0
2014-06-12 14:51:53.676806 7f9d60bd2700 10 stat 3.6b_head/5d17a06b/rbd_data.109b6b8b4567.0000000000001096/head//3
2014-06-12 14:51:53.676951 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5d17a06b/rbd_data.109b6b8b4567.0000000000001096/head//3
2014-06-12 14:51:53.697394 7f9d60bd2700 10 getattrs 3.6b_head/5d17a06b/rbd_data.109b6b8b4567.0000000000001096/head//3 = 0
2014-06-12 14:51:53.697410 7f9d60bd2700 10 stat 3.6b_head/4177a06b/rbd_data.10aa6b8b4567.0000000000002a4e/head//3
2014-06-12 14:51:53.697530 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 4177a06b/rbd_data.10aa6b8b4567.0000000000002a4e/head//3
2014-06-12 14:51:53.703905 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:53.703977 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:53.710418 7f9d60bd2700 10 getattrs 3.6b_head/4177a06b/rbd_data.10aa6b8b4567.0000000000002a4e/head//3 = 0
2014-06-12 14:51:53.710428 7f9d60bd2700 10 stat 3.6b_head/5f87a06b/rbd_data.109f6b8b4567.00000000000035a4/head//3
2014-06-12 14:51:53.710577 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5f87a06b/rbd_data.109f6b8b4567.00000000000035a4/head//3
2014-06-12 14:51:53.726669 7f9d60bd2700 10 getattrs 3.6b_head/5f87a06b/rbd_data.109f6b8b4567.00000000000035a4/head//3 = 0
2014-06-12 14:51:53.726682 7f9d60bd2700 10 stat 3.6b_head/6e99a06b/rbd_data.10b26b8b4567.0000000000003841/head//3
2014-06-12 14:51:53.726826 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6e99a06b/rbd_data.10b26b8b4567.0000000000003841/head//3
2014-06-12 14:51:53.727655 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:53.728917 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:53.761548 7f9d60bd2700 10 getattrs 3.6b_head/6e99a06b/rbd_data.10b26b8b4567.0000000000003841/head//3 = 0
2014-06-12 14:51:53.761561 7f9d60bd2700 10 stat 3.6b_head/494aa06b/rbd_data.10a96b8b4567.0000000000002608/head//3
2014-06-12 14:51:53.761724 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 494aa06b/rbd_data.10a96b8b4567.0000000000002608/head//3
2014-06-12 14:51:53.774303 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:53.774346 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:53.790807 7f9d60bd2700 10 getattrs 3.6b_head/494aa06b/rbd_data.10a96b8b4567.0000000000002608/head//3 = 0
2014-06-12 14:51:53.790815 7f9d60bd2700 10 stat 3.6b_head/8fcaa06b/rbd_data.10aa6b8b4567.0000000000002552/head//3
2014-06-12 14:51:53.790916 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 8fcaa06b/rbd_data.10aa6b8b4567.0000000000002552/head//3
2014-06-12 14:51:53.793804 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:53.793825 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:53.794198 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:53.794225 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:53.794341 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:53.794411 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:53.794434 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:53.794448 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:53.794459 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:53.794475 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:53.794484 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:53.794513 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:53.794607 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:53.794650 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:53.794678 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:53.794749 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:53.794775 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:53.794809 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:53.794856 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:53.794888 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:53.794913 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:53.794939 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:53.794963 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:53.794988 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:53.808806 7f9d60bd2700 10 getattrs 3.6b_head/8fcaa06b/rbd_data.10aa6b8b4567.0000000000002552/head//3 = 0
2014-06-12 14:51:53.808815 7f9d60bd2700 10 stat 3.6b_head/d8bca06b/rbd_data.10a76b8b4567.00000000000032f3/head//3
2014-06-12 14:51:53.808907 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid d8bca06b/rbd_data.10a76b8b4567.00000000000032f3/head//3
2014-06-12 14:51:53.837187 7f9d60bd2700 10 getattrs 3.6b_head/d8bca06b/rbd_data.10a76b8b4567.00000000000032f3/head//3 = 0
2014-06-12 14:51:53.837196 7f9d60bd2700 10 collection_getattrs 3.6b_head
2014-06-12 14:51:53.837245 7f9d60bd2700 10 collection_getattrs 3.6b_head = 0
2014-06-12 14:51:53.837247 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:53.837276 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:53.837290 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:53.837297 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:53.837301 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:53.872690 7f9d5501e700  5 -- op tracker -- , seq: 748611, time: 2014-06-12 14:51:53.872579, event: header_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.872706 7f9d5501e700  5 -- op tracker -- , seq: 748611, time: 2014-06-12 14:51:53.872582, event: throttled, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.872716 7f9d5501e700  5 -- op tracker -- , seq: 748611, time: 2014-06-12 14:51:53.872643, event: all_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.872743 7f9d5501e700  5 -- op tracker -- , seq: 748611, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.872750 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:53.872757 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:53.872767 7f9d5501e700 15 osd.2 173 enqueue_op 0x3b42ae00 prio 127 cost 10717 latency 0.000187 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:53.872836 7f9d633d7700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:53.872856 7f9d633d7700 10 osd.2 173 dequeue_op 0x3b42ae00 prio 127 cost 10717 latency 0.000276 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]
2014-06-12 14:51:53.872891 7f9d633d7700  5 -- op tracker -- , seq: 748611, time: 2014-06-12 14:51:53.872885, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.872905 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] handle_message: 0x3b42ae00
2014-06-12 14:51:53.872923 7f9d633d7700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:53.872938 7f9d633d7700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:53.872953 7f9d633d7700  5 -- op tracker -- , seq: 748611, time: 2014-06-12 14:51:53.872949, event: started, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.872960 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  got 29 scrub map
2014-06-12 14:51:53.873062 7f9d633d7700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] map version is 173'2887
2014-06-12 14:51:53.873081 7f9d633d7700 10 osd.2 173 dequeue_op 0x3b42ae00 finish
2014-06-12 14:51:53.873088 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:53.873090 7f9d633d7700  5 -- op tracker -- , seq: 748611, time: 2014-06-12 14:51:53.873085, event: done, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:53.873114 7f9d633d7700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:53.873117 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:53.873099 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:53.873120 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:53.873130 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:53.873167 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:53.873181 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:53.873192 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub replica 29 has 25 items
2014-06-12 14:51:53.873226 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ae806b/rbd_data.10a96b8b4567.00000000000016f0/head//3, auth == maps.end()
2014-06-12 14:51:53.873247 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ae806b/rbd_data.10a96b8b4567.00000000000016f0/head//3
2014-06-12 14:51:53.873268 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c4ce806b/rbd_data.10a56b8b4567.0000000000000a54/head//3, auth == maps.end()
2014-06-12 14:51:53.873285 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c4ce806b/rbd_data.10a56b8b4567.0000000000000a54/head//3
2014-06-12 14:51:53.873303 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7ede806b/rbd_data.10a56b8b4567.0000000000003710/head//3, auth == maps.end()
2014-06-12 14:51:53.873320 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7ede806b/rbd_data.10a56b8b4567.0000000000003710/head//3
2014-06-12 14:51:53.873337 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6ef2906b/rbd_data.10ad6b8b4567.0000000000003bb1/head//3, auth == maps.end()
2014-06-12 14:51:53.873354 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6ef2906b/rbd_data.10ad6b8b4567.0000000000003bb1/head//3
2014-06-12 14:51:53.873371 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4055906b/rbd_data.10946b8b4567.00000000000023c3/head//3, auth == maps.end()
2014-06-12 14:51:53.873387 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 4055906b/rbd_data.10946b8b4567.00000000000023c3/head//3
2014-06-12 14:51:53.873405 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c4c5906b/rbd_data.10956b8b4567.00000000000039c9/head//3, auth == maps.end()
2014-06-12 14:51:53.873419 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c4c5906b/rbd_data.10956b8b4567.00000000000039c9/head//3
2014-06-12 14:51:53.873436 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5fc6906b/rbd_data.10946b8b4567.00000000000019d4/head//3, auth == maps.end()
2014-06-12 14:51:53.873452 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5fc6906b/rbd_data.10946b8b4567.00000000000019d4/head//3
2014-06-12 14:51:53.873470 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5227906b/rbd_data.10ae6b8b4567.0000000000003486/head//3, auth == maps.end()
2014-06-12 14:51:53.873486 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5227906b/rbd_data.10ae6b8b4567.0000000000003486/head//3
2014-06-12 14:51:53.873501 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dee7906b/rbd_data.10986b8b4567.0000000000000fcb/head//3, auth == maps.end()
2014-06-12 14:51:53.873517 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj dee7906b/rbd_data.10986b8b4567.0000000000000fcb/head//3
2014-06-12 14:51:53.873534 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fdf7906b/rbd_data.10916b8b4567.00000000000029ec/head//3, auth == maps.end()
2014-06-12 14:51:53.873550 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj fdf7906b/rbd_data.10916b8b4567.00000000000029ec/head//3
2014-06-12 14:51:53.873568 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ae18906b/rbd_data.109f6b8b4567.000000000000095e/head//3, auth == maps.end()
2014-06-12 14:51:53.873584 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ae18906b/rbd_data.109f6b8b4567.000000000000095e/head//3
2014-06-12 14:51:53.873602 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ccb8906b/rbd_data.10956b8b4567.000000000000218a/head//3, auth == maps.end()
2014-06-12 14:51:53.873636 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ccb8906b/rbd_data.10956b8b4567.000000000000218a/head//3
2014-06-12 14:51:53.873656 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dfb906b/rbd_data.10b36b8b4567.0000000000002a1d/head//3, auth == maps.end()
2014-06-12 14:51:53.873672 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj dfb906b/rbd_data.10b36b8b4567.0000000000002a1d/head//3
2014-06-12 14:51:53.873688 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 671c906b/rbd_data.10aa6b8b4567.0000000000000c88/head//3, auth == maps.end()
2014-06-12 14:51:53.873708 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 671c906b/rbd_data.10aa6b8b4567.0000000000000c88/head//3
2014-06-12 14:51:53.873717 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 75fc906b/rbd_data.10a76b8b4567.000000000000013c/head//3, auth == maps.end()
2014-06-12 14:51:53.873725 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 75fc906b/rbd_data.10a76b8b4567.000000000000013c/head//3
2014-06-12 14:51:53.873734 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 98ad906b/rbd_data.10936b8b4567.00000000000028ea/head//3, auth == maps.end()
2014-06-12 14:51:53.873742 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 98ad906b/rbd_data.10936b8b4567.00000000000028ea/head//3
2014-06-12 14:51:53.873751 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 84d1a06b/rbd_data.10986b8b4567.00000000000016fb/head//3, auth == maps.end()
2014-06-12 14:51:53.873759 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 84d1a06b/rbd_data.10986b8b4567.00000000000016fb/head//3
2014-06-12 14:51:53.873767 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ca82a06b/rbd_data.10906b8b4567.0000000000000e65/head//3, auth == maps.end()
2014-06-12 14:51:53.873775 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ca82a06b/rbd_data.10906b8b4567.0000000000000e65/head//3
2014-06-12 14:51:53.873784 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5d17a06b/rbd_data.109b6b8b4567.0000000000001096/head//3, auth == maps.end()
2014-06-12 14:51:53.873792 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5d17a06b/rbd_data.109b6b8b4567.0000000000001096/head//3
2014-06-12 14:51:53.873801 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4177a06b/rbd_data.10aa6b8b4567.0000000000002a4e/head//3, auth == maps.end()
2014-06-12 14:51:53.873809 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 4177a06b/rbd_data.10aa6b8b4567.0000000000002a4e/head//3
2014-06-12 14:51:53.873817 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5f87a06b/rbd_data.109f6b8b4567.00000000000035a4/head//3, auth == maps.end()
2014-06-12 14:51:53.873824 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5f87a06b/rbd_data.109f6b8b4567.00000000000035a4/head//3
2014-06-12 14:51:53.873832 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6e99a06b/rbd_data.10b26b8b4567.0000000000003841/head//3, auth == maps.end()
2014-06-12 14:51:53.873840 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6e99a06b/rbd_data.10b26b8b4567.0000000000003841/head//3
2014-06-12 14:51:53.873849 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 494aa06b/rbd_data.10a96b8b4567.0000000000002608/head//3, auth == maps.end()
2014-06-12 14:51:53.873857 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 494aa06b/rbd_data.10a96b8b4567.0000000000002608/head//3
2014-06-12 14:51:53.873865 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8fcaa06b/rbd_data.10aa6b8b4567.0000000000002552/head//3, auth == maps.end()
2014-06-12 14:51:53.873873 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 8fcaa06b/rbd_data.10aa6b8b4567.0000000000002552/head//3
2014-06-12 14:51:53.873880 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d8bca06b/rbd_data.10a76b8b4567.00000000000032f3/head//3, auth == maps.end()
2014-06-12 14:51:53.873888 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj d8bca06b/rbd_data.10a76b8b4567.00000000000032f3/head//3
2014-06-12 14:51:53.873898 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] 
2014-06-12 14:51:53.873904 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub
2014-06-12 14:51:53.873914 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  d8bca06b/rbd_data.10a76b8b4567.00000000000032f3/head//3 d8bca06b/rbd_data.10a76b8b4567.00000000000032f3/head//3(173'2056 client.4304.0:101209 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2056)
2014-06-12 14:51:53.873926 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  8fcaa06b/rbd_data.10aa6b8b4567.0000000000002552/head//3 8fcaa06b/rbd_data.10aa6b8b4567.0000000000002552/head//3(173'1509 client.4307.0:78623 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1509)
2014-06-12 14:51:53.873936 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  494aa06b/rbd_data.10a96b8b4567.0000000000002608/head//3 494aa06b/rbd_data.10a96b8b4567.0000000000002608/head//3(173'1648 client.4306.0:80289 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1648)
2014-06-12 14:51:53.873945 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6e99a06b/rbd_data.10b26b8b4567.0000000000003841/head//3 6e99a06b/rbd_data.10b26b8b4567.0000000000003841/head//3(173'2119 client.4315.0:111368 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2119)
2014-06-12 14:51:53.873954 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5f87a06b/rbd_data.109f6b8b4567.00000000000035a4/head//3 5f87a06b/rbd_data.109f6b8b4567.00000000000035a4/head//3(173'2049 client.4296.0:104494 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2049)
2014-06-12 14:51:53.873965 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  4177a06b/rbd_data.10aa6b8b4567.0000000000002a4e/head//3 4177a06b/rbd_data.10aa6b8b4567.0000000000002a4e/head//3(173'1720 client.4307.0:87740 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1720)
2014-06-12 14:51:53.873974 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5d17a06b/rbd_data.109b6b8b4567.0000000000001096/head//3 5d17a06b/rbd_data.109b6b8b4567.0000000000001096/head//3(173'801 client.4292.0:37835 wrlock_by=unknown.0.0:0 dirty s 4194304 uv801)
2014-06-12 14:51:53.873983 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ca82a06b/rbd_data.10906b8b4567.0000000000000e65/head//3 ca82a06b/rbd_data.10906b8b4567.0000000000000e65/head//3(173'2796 client.4281.0:139773 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2796)
2014-06-12 14:51:53.873993 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  84d1a06b/rbd_data.10986b8b4567.00000000000016fb/head//3 84d1a06b/rbd_data.10986b8b4567.00000000000016fb/head//3(173'971 client.4289.0:49923 wrlock_by=unknown.0.0:0 dirty s 4194304 uv971)
2014-06-12 14:51:53.874002 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  98ad906b/rbd_data.10936b8b4567.00000000000028ea/head//3 98ad906b/rbd_data.10936b8b4567.00000000000028ea/head//3(173'1654 client.4284.0:81128 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1654)
2014-06-12 14:51:53.874012 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  75fc906b/rbd_data.10a76b8b4567.000000000000013c/head//3 75fc906b/rbd_data.10a76b8b4567.000000000000013c/head//3(173'2424 client.4304.0:118722 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2424)
2014-06-12 14:51:53.874022 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  671c906b/rbd_data.10aa6b8b4567.0000000000000c88/head//3 671c906b/rbd_data.10aa6b8b4567.0000000000000c88/head//3(173'2761 client.4307.0:140668 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2761)
2014-06-12 14:51:53.874032 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  dfb906b/rbd_data.10b36b8b4567.0000000000002a1d/head//3 dfb906b/rbd_data.10b36b8b4567.0000000000002a1d/head//3(173'1734 client.4316.0:86463 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1734)
2014-06-12 14:51:53.874042 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ccb8906b/rbd_data.10956b8b4567.000000000000218a/head//3 ccb8906b/rbd_data.10956b8b4567.000000000000218a/head//3(173'1380 client.4286.0:69501 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1380)
2014-06-12 14:51:53.874052 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ae18906b/rbd_data.109f6b8b4567.000000000000095e/head//3 ae18906b/rbd_data.109f6b8b4567.000000000000095e/head//3(173'2636 client.4296.0:130066 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2636)
2014-06-12 14:51:53.874065 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  fdf7906b/rbd_data.10916b8b4567.00000000000029ec/head//3 fdf7906b/rbd_data.10916b8b4567.00000000000029ec/head//3(173'1764 client.4282.0:85515 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1764)
2014-06-12 14:51:53.874075 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  dee7906b/rbd_data.10986b8b4567.0000000000000fcb/head//3 dee7906b/rbd_data.10986b8b4567.0000000000000fcb/head//3(173'2838 client.4289.0:141363 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2838)
2014-06-12 14:51:53.874085 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5227906b/rbd_data.10ae6b8b4567.0000000000003486/head//3 5227906b/rbd_data.10ae6b8b4567.0000000000003486/head//3(173'2015 client.4311.0:101689 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2015)
2014-06-12 14:51:53.874095 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5fc6906b/rbd_data.10946b8b4567.00000000000019d4/head//3 5fc6906b/rbd_data.10946b8b4567.00000000000019d4/head//3(173'1058 client.4285.0:55194 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1058)
2014-06-12 14:51:53.874105 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c4c5906b/rbd_data.10956b8b4567.00000000000039c9/head//3 c4c5906b/rbd_data.10956b8b4567.00000000000039c9/head//3(173'2318 client.4286.0:114795 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2318)
2014-06-12 14:51:53.874115 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  4055906b/rbd_data.10946b8b4567.00000000000023c3/head//3 4055906b/rbd_data.10946b8b4567.00000000000023c3/head//3(173'1423 client.4285.0:72344 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1423)
2014-06-12 14:51:53.874125 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6ef2906b/rbd_data.10ad6b8b4567.0000000000003bb1/head//3 6ef2906b/rbd_data.10ad6b8b4567.0000000000003bb1/head//3(173'2306 client.4310.0:117335 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2306)
2014-06-12 14:51:53.874135 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7ede806b/rbd_data.10a56b8b4567.0000000000003710/head//3 7ede806b/rbd_data.10a56b8b4567.0000000000003710/head//3(173'2211 client.4302.0:110280 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2211)
2014-06-12 14:51:53.874145 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c4ce806b/rbd_data.10a56b8b4567.0000000000000a54/head//3 c4ce806b/rbd_data.10a56b8b4567.0000000000000a54/head//3(173'2678 client.4302.0:135876 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2678)
2014-06-12 14:51:53.874155 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ae806b/rbd_data.10a96b8b4567.00000000000016f0/head//3 ae806b/rbd_data.10a96b8b4567.00000000000016f0/head//3(173'1083 client.4306.0:52457 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1083)
2014-06-12 14:51:53.874162 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:53.874171 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:53.874178 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:53.874179 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:53.874181 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:53.874215 7f9d60bd2700 10 collection_list_partial 3.6b_head start:417ea06b//0//-1 is_max:0
2014-06-12 14:51:53.874430 7f9d60bd2700 20 objects: [92cea06b/rbd_data.10ae6b8b4567.0000000000003a91/head//3,9141b06b/rbd_data.10916b8b4567.0000000000003a38/head//3,9132b06b/rbd_data.109b6b8b4567.0000000000001090/head//3,af63b06b/rbd_data.108e6b8b4567.000000000000186f/head//3,e019b06b/rbd_data.10916b8b4567.0000000000002730/head//3,1229b06b/rbd_data.108e6b8b4567.0000000000001fa8/head//3,8069b06b/rbd_data.10936b8b4567.00000000000036fa/head//3,a24ab06b/rbd_data.109d6b8b4567.00000000000033b5/head//3,709ab06b/rbd_data.10986b8b4567.000000000000012a/head//3,290bb06b/rbd_data.10ab6b8b4567.0000000000001011/head//3,7e1cb06b/rbd_data.108f6b8b4567.0000000000002077/head//3,45feb06b/rbd_data.108d6b8b4567.00000000000008d3/head//3,6f0c06b/rbd_data.10b46b8b4567.000000000000378c/head//3,85c3c06b/rbd_data.10926b8b4567.0000000000002e09/head//3,1b04c06b/rbd_data.10976b8b4567.000000000000395f/head//3,6c45c06b/rbd_data.108d6b8b4567.000000000000360d/head//3,2d76c06b/rbd_data.109a6b8b4567.0000000000001ab0/head//3,cc47c06b/rbd_data.109a6b8b4567.0000000000001429/head//3,4d68c06b/rbd_data.10a86b8b4567.0000000000002a35/head//3,6eb8c06b/rbd_data.108f6b8b4567.0000000000002155/head//3,8d09c06b/rbd_data.10956b8b4567.00000000000032f3/head//3,ed8bc06b/rbd_data.10976b8b4567.0000000000000081/head//3,535dc06b/rbd_data.109d6b8b4567.0000000000000f84/head//3,984ec06b/rbd_data.10ad6b8b4567.0000000000000a62/head//3,ce81d06b/rbd_data.109b6b8b4567.0000000000002abd/head//3]
2014-06-12 14:51:53.874924 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  requesting scrubmap from osd.29
2014-06-12 14:51:53.874942 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:53.874954 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:53.874961 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:53.874966 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:53.874971 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:53.874975 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_map_chunk [417ea06b//0//-1,7a74d06b//0//-1)
2014-06-12 14:51:53.874981 7f9d60bd2700 10 collection_list_partial 3.6b_head start:417ea06b//0//-1 is_max:0
2014-06-12 14:51:53.875381 7f9d60bd2700 20 objects: [92cea06b/rbd_data.10ae6b8b4567.0000000000003a91/head//3,9141b06b/rbd_data.10916b8b4567.0000000000003a38/head//3,9132b06b/rbd_data.109b6b8b4567.0000000000001090/head//3,af63b06b/rbd_data.108e6b8b4567.000000000000186f/head//3,e019b06b/rbd_data.10916b8b4567.0000000000002730/head//3,1229b06b/rbd_data.108e6b8b4567.0000000000001fa8/head//3,8069b06b/rbd_data.10936b8b4567.00000000000036fa/head//3,a24ab06b/rbd_data.109d6b8b4567.00000000000033b5/head//3,709ab06b/rbd_data.10986b8b4567.000000000000012a/head//3,290bb06b/rbd_data.10ab6b8b4567.0000000000001011/head//3,7e1cb06b/rbd_data.108f6b8b4567.0000000000002077/head//3,45feb06b/rbd_data.108d6b8b4567.00000000000008d3/head//3,6f0c06b/rbd_data.10b46b8b4567.000000000000378c/head//3,85c3c06b/rbd_data.10926b8b4567.0000000000002e09/head//3,1b04c06b/rbd_data.10976b8b4567.000000000000395f/head//3,6c45c06b/rbd_data.108d6b8b4567.000000000000360d/head//3,2d76c06b/rbd_data.109a6b8b4567.0000000000001ab0/head//3,cc47c06b/rbd_data.109a6b8b4567.0000000000001429/head//3,4d68c06b/rbd_data.10a86b8b4567.0000000000002a35/head//3,6eb8c06b/rbd_data.108f6b8b4567.0000000000002155/head//3,8d09c06b/rbd_data.10956b8b4567.00000000000032f3/head//3,ed8bc06b/rbd_data.10976b8b4567.0000000000000081/head//3,535dc06b/rbd_data.109d6b8b4567.0000000000000f84/head//3,984ec06b/rbd_data.10ad6b8b4567.0000000000000a62/head//3,ce81d06b/rbd_data.109b6b8b4567.0000000000002abd/head//3,7a74d06b/rbd_data.10976b8b4567.00000000000010c0/head//3,4175d06b/rbd_data.10b16b8b4567.00000000000001b4/head//3,3bfcd06b/rbd_data.10926b8b4567.0000000000001ea1/head//3,953fd06b/rbd_data.10a16b8b4567.00000000000018cb/head//3,a76fd06b/rbd_data.108f6b8b4567.0000000000000081/head//3,8121e06b/rbd_data.10976b8b4567.000000000000075c/head//3,2cc2e06b/rbd_data.10b06b8b4567.0000000000002075/head//3,5066e06b/rbd_data.10b46b8b4567.0000000000001a06/head//3,7268e06b/rbd_data.10a86b8b4567.0000000000002d74/head//3,a8de06b/rbd_data.108e6b8b4567.0000000000000d1b/head//3,eac1f06b/rbd_data.10b46b8b4567.0000000000002b66/head//3,5562f06b/rbd_data.109a6b8b4567.0000000000003402/head//3,dea5f06b/rbd_data.10ad6b8b4567.0000000000000c8a/head//3,d766f06b/rbd_data.10966b8b4567.0000000000000a56/head//3,1796f06b/rbd_data.109e6b8b4567.00000000000027ab/head//3,5567f06b/rbd_data.10aa6b8b4567.0000000000001375/head//3,419cf06b/rbd_data.10ad6b8b4567.0000000000002c9d/head//3,cc6df06b/rbd_data.10996b8b4567.0000000000001866/head//3,d60ff06b/rbd_data.109c6b8b4567.0000000000001d3b/head//3,bf9ff06b/rbd_data.10aa6b8b4567.0000000000003a1c/head//3,56a0086b/rbd_data.10a56b8b4567.0000000000000ddc/head//3,ac1086b/rbd_data.108f6b8b4567.000000000000119f/head//3,4715086b/rbd_data.10a26b8b4567.0000000000000f9b/head//3,23c5086b/rbd_data.10a76b8b4567.0000000000000769/head//3,9ae5086b/rbd_data.10a26b8b4567.0000000000003b25/head//3,7108086b/rbd_data.108d6b8b4567.0000000000003b2c/head//3,e2c9086b/rbd_data.109a6b8b4567.00000000000013c0/head//3,e07b086b/rbd_data.10b26b8b4567.0000000000003b18/head//3,5710186b/rbd_data.10ad6b8b4567.000000000000247e/head//3,fae1186b/rbd_data.10946b8b4567.0000000000002a53/head//3,2f32186b/rbd_data.10a86b8b4567.00000000000016a3/head//3,8f23186b/rbd_data.10b16b8b4567.0000000000003424/head//3,ad04186b/rbd_data.10996b8b4567.00000000000022e5/head//3,35e4186b/rbd_data.109a6b8b4567.000000000000142e/head//3,6c17186b/rbd_data.10ac6b8b4567.00000000000035b8/head//3,d957186b/rbd_data.10a26b8b4567.00000000000021e7/head//3,7f09186b/rbd_data.10b06b8b4567.0000000000003923/head//3,af3c186b/rbd_data.10b26b8b4567.0000000000001f3d/head//3,c29d186b/rbd_data.10a36b8b4567.0000000000002e8c/head//3]
2014-06-12 14:51:53.875445 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:53.875458 7f9d60bd2700 10 stat 3.6b_head/92cea06b/rbd_data.10ae6b8b4567.0000000000003a91/head//3
2014-06-12 14:51:53.875551 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 92cea06b/rbd_data.10ae6b8b4567.0000000000003a91/head//3
2014-06-12 14:51:53.875631 7f9d60bd2700 10 getattrs 3.6b_head/92cea06b/rbd_data.10ae6b8b4567.0000000000003a91/head//3 = 0
2014-06-12 14:51:53.875636 7f9d60bd2700 10 stat 3.6b_head/9141b06b/rbd_data.10916b8b4567.0000000000003a38/head//3
2014-06-12 14:51:53.875712 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9141b06b/rbd_data.10916b8b4567.0000000000003a38/head//3
2014-06-12 14:51:53.887780 7f9d60bd2700 10 getattrs 3.6b_head/9141b06b/rbd_data.10916b8b4567.0000000000003a38/head//3 = 0
2014-06-12 14:51:53.887788 7f9d60bd2700 10 stat 3.6b_head/9132b06b/rbd_data.109b6b8b4567.0000000000001090/head//3
2014-06-12 14:51:53.887882 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9132b06b/rbd_data.109b6b8b4567.0000000000001090/head//3
2014-06-12 14:51:53.902257 7f9d60bd2700 10 getattrs 3.6b_head/9132b06b/rbd_data.109b6b8b4567.0000000000001090/head//3 = 0
2014-06-12 14:51:53.902263 7f9d60bd2700 10 stat 3.6b_head/af63b06b/rbd_data.108e6b8b4567.000000000000186f/head//3
2014-06-12 14:51:53.902356 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid af63b06b/rbd_data.108e6b8b4567.000000000000186f/head//3
2014-06-12 14:51:53.922376 7f9d60bd2700 10 getattrs 3.6b_head/af63b06b/rbd_data.108e6b8b4567.000000000000186f/head//3 = 0
2014-06-12 14:51:53.922385 7f9d60bd2700 10 stat 3.6b_head/e019b06b/rbd_data.10916b8b4567.0000000000002730/head//3
2014-06-12 14:51:53.922487 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid e019b06b/rbd_data.10916b8b4567.0000000000002730/head//3
2014-06-12 14:51:53.949179 7f9d60bd2700 10 getattrs 3.6b_head/e019b06b/rbd_data.10916b8b4567.0000000000002730/head//3 = 0
2014-06-12 14:51:53.949190 7f9d60bd2700 10 stat 3.6b_head/1229b06b/rbd_data.108e6b8b4567.0000000000001fa8/head//3
2014-06-12 14:51:53.949286 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1229b06b/rbd_data.108e6b8b4567.0000000000001fa8/head//3
2014-06-12 14:51:53.949372 7f9d60bd2700 10 getattrs 3.6b_head/1229b06b/rbd_data.108e6b8b4567.0000000000001fa8/head//3 = 0
2014-06-12 14:51:53.949377 7f9d60bd2700 10 stat 3.6b_head/8069b06b/rbd_data.10936b8b4567.00000000000036fa/head//3
2014-06-12 14:51:53.949448 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 8069b06b/rbd_data.10936b8b4567.00000000000036fa/head//3
2014-06-12 14:51:53.973824 7f9d60bd2700 10 getattrs 3.6b_head/8069b06b/rbd_data.10936b8b4567.00000000000036fa/head//3 = 0
2014-06-12 14:51:53.973834 7f9d60bd2700 10 stat 3.6b_head/a24ab06b/rbd_data.109d6b8b4567.00000000000033b5/head//3
2014-06-12 14:51:53.973926 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a24ab06b/rbd_data.109d6b8b4567.00000000000033b5/head//3
2014-06-12 14:51:53.986261 7f9d60bd2700 10 getattrs 3.6b_head/a24ab06b/rbd_data.109d6b8b4567.00000000000033b5/head//3 = 0
2014-06-12 14:51:53.986270 7f9d60bd2700 10 stat 3.6b_head/709ab06b/rbd_data.10986b8b4567.000000000000012a/head//3
2014-06-12 14:51:53.986367 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 709ab06b/rbd_data.10986b8b4567.000000000000012a/head//3
2014-06-12 14:51:53.986468 7f9d60bd2700 10 getattrs 3.6b_head/709ab06b/rbd_data.10986b8b4567.000000000000012a/head//3 = 0
2014-06-12 14:51:53.986474 7f9d60bd2700 10 stat 3.6b_head/290bb06b/rbd_data.10ab6b8b4567.0000000000001011/head//3
2014-06-12 14:51:53.986561 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 290bb06b/rbd_data.10ab6b8b4567.0000000000001011/head//3
2014-06-12 14:51:53.987157 7f9d60bd2700 10 getattrs 3.6b_head/290bb06b/rbd_data.10ab6b8b4567.0000000000001011/head//3 = 0
2014-06-12 14:51:53.987164 7f9d60bd2700 10 stat 3.6b_head/7e1cb06b/rbd_data.108f6b8b4567.0000000000002077/head//3
2014-06-12 14:51:53.987261 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7e1cb06b/rbd_data.108f6b8b4567.0000000000002077/head//3
2014-06-12 14:51:53.999214 7f9d60bd2700 10 getattrs 3.6b_head/7e1cb06b/rbd_data.108f6b8b4567.0000000000002077/head//3 = 0
2014-06-12 14:51:53.999223 7f9d60bd2700 10 stat 3.6b_head/45feb06b/rbd_data.108d6b8b4567.00000000000008d3/head//3
2014-06-12 14:51:53.999325 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 45feb06b/rbd_data.108d6b8b4567.00000000000008d3/head//3
2014-06-12 14:51:54.044164 7f9d60bd2700 10 getattrs 3.6b_head/45feb06b/rbd_data.108d6b8b4567.00000000000008d3/head//3 = 0
2014-06-12 14:51:54.044185 7f9d60bd2700 10 stat 3.6b_head/6f0c06b/rbd_data.10b46b8b4567.000000000000378c/head//3
2014-06-12 14:51:54.044285 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6f0c06b/rbd_data.10b46b8b4567.000000000000378c/head//3
2014-06-12 14:51:54.065233 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:54.073556 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:54.073660 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:54.076469 7f9d60bd2700 10 getattrs 3.6b_head/6f0c06b/rbd_data.10b46b8b4567.000000000000378c/head//3 = 0
2014-06-12 14:51:54.076483 7f9d60bd2700 10 stat 3.6b_head/85c3c06b/rbd_data.10926b8b4567.0000000000002e09/head//3
2014-06-12 14:51:54.076589 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 85c3c06b/rbd_data.10926b8b4567.0000000000002e09/head//3
2014-06-12 14:51:54.080426 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:54.080468 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:54.095820 7f9d60bd2700 10 getattrs 3.6b_head/85c3c06b/rbd_data.10926b8b4567.0000000000002e09/head//3 = 0
2014-06-12 14:51:54.095833 7f9d60bd2700 10 stat 3.6b_head/1b04c06b/rbd_data.10976b8b4567.000000000000395f/head//3
2014-06-12 14:51:54.095933 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1b04c06b/rbd_data.10976b8b4567.000000000000395f/head//3
2014-06-12 14:51:54.113709 7f9d60bd2700 10 getattrs 3.6b_head/1b04c06b/rbd_data.10976b8b4567.000000000000395f/head//3 = 0
2014-06-12 14:51:54.113720 7f9d60bd2700 10 stat 3.6b_head/6c45c06b/rbd_data.108d6b8b4567.000000000000360d/head//3
2014-06-12 14:51:54.113822 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6c45c06b/rbd_data.108d6b8b4567.000000000000360d/head//3
2014-06-12 14:51:54.130942 7f9d60bd2700 10 getattrs 3.6b_head/6c45c06b/rbd_data.108d6b8b4567.000000000000360d/head//3 = 0
2014-06-12 14:51:54.130951 7f9d60bd2700 10 stat 3.6b_head/2d76c06b/rbd_data.109a6b8b4567.0000000000001ab0/head//3
2014-06-12 14:51:54.131040 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 2d76c06b/rbd_data.109a6b8b4567.0000000000001ab0/head//3
2014-06-12 14:51:54.157221 7f9d60bd2700 10 getattrs 3.6b_head/2d76c06b/rbd_data.109a6b8b4567.0000000000001ab0/head//3 = 0
2014-06-12 14:51:54.157234 7f9d60bd2700 10 stat 3.6b_head/cc47c06b/rbd_data.109a6b8b4567.0000000000001429/head//3
2014-06-12 14:51:54.157357 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid cc47c06b/rbd_data.109a6b8b4567.0000000000001429/head//3
2014-06-12 14:51:54.168159 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:54.171197 7f9d60bd2700 10 getattrs 3.6b_head/cc47c06b/rbd_data.109a6b8b4567.0000000000001429/head//3 = 0
2014-06-12 14:51:54.171205 7f9d60bd2700 10 stat 3.6b_head/4d68c06b/rbd_data.10a86b8b4567.0000000000002a35/head//3
2014-06-12 14:51:54.171300 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 4d68c06b/rbd_data.10a86b8b4567.0000000000002a35/head//3
2014-06-12 14:51:54.201446 7f9d60bd2700 10 getattrs 3.6b_head/4d68c06b/rbd_data.10a86b8b4567.0000000000002a35/head//3 = 0
2014-06-12 14:51:54.201457 7f9d60bd2700 10 stat 3.6b_head/6eb8c06b/rbd_data.108f6b8b4567.0000000000002155/head//3
2014-06-12 14:51:54.201570 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6eb8c06b/rbd_data.108f6b8b4567.0000000000002155/head//3
2014-06-12 14:51:54.214682 7f9d60bd2700 10 getattrs 3.6b_head/6eb8c06b/rbd_data.108f6b8b4567.0000000000002155/head//3 = 0
2014-06-12 14:51:54.214690 7f9d60bd2700 10 stat 3.6b_head/8d09c06b/rbd_data.10956b8b4567.00000000000032f3/head//3
2014-06-12 14:51:54.214787 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 8d09c06b/rbd_data.10956b8b4567.00000000000032f3/head//3
2014-06-12 14:51:54.227948 7f9d60bd2700 10 getattrs 3.6b_head/8d09c06b/rbd_data.10956b8b4567.00000000000032f3/head//3 = 0
2014-06-12 14:51:54.227956 7f9d60bd2700 10 stat 3.6b_head/ed8bc06b/rbd_data.10976b8b4567.0000000000000081/head//3
2014-06-12 14:51:54.228068 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ed8bc06b/rbd_data.10976b8b4567.0000000000000081/head//3
2014-06-12 14:51:54.267764 7f9d60bd2700 10 getattrs 3.6b_head/ed8bc06b/rbd_data.10976b8b4567.0000000000000081/head//3 = 0
2014-06-12 14:51:54.267776 7f9d60bd2700 10 stat 3.6b_head/535dc06b/rbd_data.109d6b8b4567.0000000000000f84/head//3
2014-06-12 14:51:54.267900 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 535dc06b/rbd_data.109d6b8b4567.0000000000000f84/head//3
2014-06-12 14:51:54.280399 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:54.280427 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:54.280432 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:54.280434 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:54.280434 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:54.293249 7f9d60bd2700 10 getattrs 3.6b_head/535dc06b/rbd_data.109d6b8b4567.0000000000000f84/head//3 = 0
2014-06-12 14:51:54.293281 7f9d60bd2700 10 stat 3.6b_head/984ec06b/rbd_data.10ad6b8b4567.0000000000000a62/head//3
2014-06-12 14:51:54.293448 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 984ec06b/rbd_data.10ad6b8b4567.0000000000000a62/head//3
2014-06-12 14:51:54.313453 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:54.326921 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:54.350336 7f9d60bd2700 10 getattrs 3.6b_head/984ec06b/rbd_data.10ad6b8b4567.0000000000000a62/head//3 = 0
2014-06-12 14:51:54.350350 7f9d60bd2700 10 stat 3.6b_head/ce81d06b/rbd_data.109b6b8b4567.0000000000002abd/head//3
2014-06-12 14:51:54.350474 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ce81d06b/rbd_data.109b6b8b4567.0000000000002abd/head//3
2014-06-12 14:51:54.366995 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:54.371902 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:54.375290 7f9d60bd2700 10 getattrs 3.6b_head/ce81d06b/rbd_data.109b6b8b4567.0000000000002abd/head//3 = 0
2014-06-12 14:51:54.375302 7f9d60bd2700 10 collection_getattrs 3.6b_head
2014-06-12 14:51:54.375366 7f9d60bd2700 10 collection_getattrs 3.6b_head = 0
2014-06-12 14:51:54.375371 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:54.375406 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:54.375414 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:54.375421 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:54.375425 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:54.382596 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:54.391665 7f9d5501e700  5 -- op tracker -- , seq: 748612, time: 2014-06-12 14:51:54.391583, event: header_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:54.391677 7f9d5501e700  5 -- op tracker -- , seq: 748612, time: 2014-06-12 14:51:54.391585, event: throttled, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:54.391684 7f9d5501e700  5 -- op tracker -- , seq: 748612, time: 2014-06-12 14:51:54.391631, event: all_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:54.391690 7f9d5501e700  5 -- op tracker -- , seq: 748612, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:54.391695 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:54.391701 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:54.391708 7f9d5501e700 15 osd.2 173 enqueue_op 0x31d61700 prio 127 cost 10717 latency 0.000124 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:54.391759 7f9d62bd6700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:54.391790 7f9d62bd6700 10 osd.2 173 dequeue_op 0x31d61700 prio 127 cost 10717 latency 0.000206 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]
2014-06-12 14:51:54.391830 7f9d62bd6700  5 -- op tracker -- , seq: 748612, time: 2014-06-12 14:51:54.391823, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:54.391847 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] handle_message: 0x31d61700
2014-06-12 14:51:54.391862 7f9d62bd6700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:54.391880 7f9d62bd6700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:54.391896 7f9d62bd6700  5 -- op tracker -- , seq: 748612, time: 2014-06-12 14:51:54.391891, event: started, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:54.391924 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  got 29 scrub map
2014-06-12 14:51:54.392032 7f9d62bd6700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] map version is 173'2887
2014-06-12 14:51:54.392053 7f9d62bd6700 10 osd.2 173 dequeue_op 0x31d61700 finish
2014-06-12 14:51:54.392064 7f9d62bd6700  5 -- op tracker -- , seq: 748612, time: 2014-06-12 14:51:54.392058, event: done, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:54.392086 7f9d62bd6700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:54.392090 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:54.392093 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:54.392106 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:54.392128 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:54.392146 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:54.392176 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:54.392184 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:54.392191 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub replica 29 has 25 items
2014-06-12 14:51:54.392214 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 92cea06b/rbd_data.10ae6b8b4567.0000000000003a91/head//3, auth == maps.end()
2014-06-12 14:51:54.392240 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 92cea06b/rbd_data.10ae6b8b4567.0000000000003a91/head//3
2014-06-12 14:51:54.392255 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9141b06b/rbd_data.10916b8b4567.0000000000003a38/head//3, auth == maps.end()
2014-06-12 14:51:54.392265 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9141b06b/rbd_data.10916b8b4567.0000000000003a38/head//3
2014-06-12 14:51:54.392276 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9132b06b/rbd_data.109b6b8b4567.0000000000001090/head//3, auth == maps.end()
2014-06-12 14:51:54.392290 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9132b06b/rbd_data.109b6b8b4567.0000000000001090/head//3
2014-06-12 14:51:54.392301 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj af63b06b/rbd_data.108e6b8b4567.000000000000186f/head//3, auth == maps.end()
2014-06-12 14:51:54.392310 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj af63b06b/rbd_data.108e6b8b4567.000000000000186f/head//3
2014-06-12 14:51:54.392321 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e019b06b/rbd_data.10916b8b4567.0000000000002730/head//3, auth == maps.end()
2014-06-12 14:51:54.392330 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj e019b06b/rbd_data.10916b8b4567.0000000000002730/head//3
2014-06-12 14:51:54.392342 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1229b06b/rbd_data.108e6b8b4567.0000000000001fa8/head//3, auth == maps.end()
2014-06-12 14:51:54.392354 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1229b06b/rbd_data.108e6b8b4567.0000000000001fa8/head//3
2014-06-12 14:51:54.392364 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8069b06b/rbd_data.10936b8b4567.00000000000036fa/head//3, auth == maps.end()
2014-06-12 14:51:54.392376 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 8069b06b/rbd_data.10936b8b4567.00000000000036fa/head//3
2014-06-12 14:51:54.392388 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a24ab06b/rbd_data.109d6b8b4567.00000000000033b5/head//3, auth == maps.end()
2014-06-12 14:51:54.392398 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a24ab06b/rbd_data.109d6b8b4567.00000000000033b5/head//3
2014-06-12 14:51:54.392410 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 709ab06b/rbd_data.10986b8b4567.000000000000012a/head//3, auth == maps.end()
2014-06-12 14:51:54.392421 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 709ab06b/rbd_data.10986b8b4567.000000000000012a/head//3
2014-06-12 14:51:54.392433 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 290bb06b/rbd_data.10ab6b8b4567.0000000000001011/head//3, auth == maps.end()
2014-06-12 14:51:54.392444 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 290bb06b/rbd_data.10ab6b8b4567.0000000000001011/head//3
2014-06-12 14:51:54.392454 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7e1cb06b/rbd_data.108f6b8b4567.0000000000002077/head//3, auth == maps.end()
2014-06-12 14:51:54.392465 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7e1cb06b/rbd_data.108f6b8b4567.0000000000002077/head//3
2014-06-12 14:51:54.392475 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 45feb06b/rbd_data.108d6b8b4567.00000000000008d3/head//3, auth == maps.end()
2014-06-12 14:51:54.392487 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 45feb06b/rbd_data.108d6b8b4567.00000000000008d3/head//3
2014-06-12 14:51:54.392498 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6f0c06b/rbd_data.10b46b8b4567.000000000000378c/head//3, auth == maps.end()
2014-06-12 14:51:54.392508 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6f0c06b/rbd_data.10b46b8b4567.000000000000378c/head//3
2014-06-12 14:51:54.392518 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 85c3c06b/rbd_data.10926b8b4567.0000000000002e09/head//3, auth == maps.end()
2014-06-12 14:51:54.392528 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 85c3c06b/rbd_data.10926b8b4567.0000000000002e09/head//3
2014-06-12 14:51:54.392538 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1b04c06b/rbd_data.10976b8b4567.000000000000395f/head//3, auth == maps.end()
2014-06-12 14:51:54.392547 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1b04c06b/rbd_data.10976b8b4567.000000000000395f/head//3
2014-06-12 14:51:54.392557 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6c45c06b/rbd_data.108d6b8b4567.000000000000360d/head//3, auth == maps.end()
2014-06-12 14:51:54.392567 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6c45c06b/rbd_data.108d6b8b4567.000000000000360d/head//3
2014-06-12 14:51:54.392577 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2d76c06b/rbd_data.109a6b8b4567.0000000000001ab0/head//3, auth == maps.end()
2014-06-12 14:51:54.392586 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 2d76c06b/rbd_data.109a6b8b4567.0000000000001ab0/head//3
2014-06-12 14:51:54.392598 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cc47c06b/rbd_data.109a6b8b4567.0000000000001429/head//3, auth == maps.end()
2014-06-12 14:51:54.392610 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj cc47c06b/rbd_data.109a6b8b4567.0000000000001429/head//3
2014-06-12 14:51:54.392620 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4d68c06b/rbd_data.10a86b8b4567.0000000000002a35/head//3, auth == maps.end()
2014-06-12 14:51:54.392632 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 4d68c06b/rbd_data.10a86b8b4567.0000000000002a35/head//3
2014-06-12 14:51:54.392644 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6eb8c06b/rbd_data.108f6b8b4567.0000000000002155/head//3, auth == maps.end()
2014-06-12 14:51:54.392655 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6eb8c06b/rbd_data.108f6b8b4567.0000000000002155/head//3
2014-06-12 14:51:54.392668 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8d09c06b/rbd_data.10956b8b4567.00000000000032f3/head//3, auth == maps.end()
2014-06-12 14:51:54.392679 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 8d09c06b/rbd_data.10956b8b4567.00000000000032f3/head//3
2014-06-12 14:51:54.392690 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ed8bc06b/rbd_data.10976b8b4567.0000000000000081/head//3, auth == maps.end()
2014-06-12 14:51:54.392701 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ed8bc06b/rbd_data.10976b8b4567.0000000000000081/head//3
2014-06-12 14:51:54.392712 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 535dc06b/rbd_data.109d6b8b4567.0000000000000f84/head//3, auth == maps.end()
2014-06-12 14:51:54.392723 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 535dc06b/rbd_data.109d6b8b4567.0000000000000f84/head//3
2014-06-12 14:51:54.392736 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 984ec06b/rbd_data.10ad6b8b4567.0000000000000a62/head//3, auth == maps.end()
2014-06-12 14:51:54.392747 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 984ec06b/rbd_data.10ad6b8b4567.0000000000000a62/head//3
2014-06-12 14:51:54.392760 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ce81d06b/rbd_data.109b6b8b4567.0000000000002abd/head//3, auth == maps.end()
2014-06-12 14:51:54.392787 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ce81d06b/rbd_data.109b6b8b4567.0000000000002abd/head//3
2014-06-12 14:51:54.392801 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] 
2014-06-12 14:51:54.392810 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub
2014-06-12 14:51:54.392824 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ce81d06b/rbd_data.109b6b8b4567.0000000000002abd/head//3 ce81d06b/rbd_data.109b6b8b4567.0000000000002abd/head//3(173'1848 client.4292.0:84439 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1848)
2014-06-12 14:51:54.392841 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  984ec06b/rbd_data.10ad6b8b4567.0000000000000a62/head//3 984ec06b/rbd_data.10ad6b8b4567.0000000000000a62/head//3(173'2703 client.4310.0:134313 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2703)
2014-06-12 14:51:54.392856 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  535dc06b/rbd_data.109d6b8b4567.0000000000000f84/head//3 535dc06b/rbd_data.109d6b8b4567.0000000000000f84/head//3(173'2804 client.4294.0:141562 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2804)
2014-06-12 14:51:54.392871 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ed8bc06b/rbd_data.10976b8b4567.0000000000000081/head//3 ed8bc06b/rbd_data.10976b8b4567.0000000000000081/head//3(173'2406 client.4288.0:115705 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2406)
2014-06-12 14:51:54.392884 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  8d09c06b/rbd_data.10956b8b4567.00000000000032f3/head//3 8d09c06b/rbd_data.10956b8b4567.00000000000032f3/head//3(173'2078 client.4286.0:101652 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2078)
2014-06-12 14:51:54.392898 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6eb8c06b/rbd_data.108f6b8b4567.0000000000002155/head//3 6eb8c06b/rbd_data.108f6b8b4567.0000000000002155/head//3(173'1414 client.4280.0:68599 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1414)
2014-06-12 14:51:54.392912 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  4d68c06b/rbd_data.10a86b8b4567.0000000000002a35/head//3 4d68c06b/rbd_data.10a86b8b4567.0000000000002a35/head//3(173'1637 client.4305.0:85128 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1637)
2014-06-12 14:51:54.392926 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  cc47c06b/rbd_data.109a6b8b4567.0000000000001429/head//3 cc47c06b/rbd_data.109a6b8b4567.0000000000001429/head//3(173'883 client.4291.0:45510 wrlock_by=unknown.0.0:0 dirty s 4194304 uv883)
2014-06-12 14:51:54.392940 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  2d76c06b/rbd_data.109a6b8b4567.0000000000001ab0/head//3 2d76c06b/rbd_data.109a6b8b4567.0000000000001ab0/head//3(173'1143 client.4291.0:57252 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1143)
2014-06-12 14:51:54.392954 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6c45c06b/rbd_data.108d6b8b4567.000000000000360d/head//3 6c45c06b/rbd_data.108d6b8b4567.000000000000360d/head//3(173'2095 client.4278.0:108073 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2095)
2014-06-12 14:51:54.392969 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1b04c06b/rbd_data.10976b8b4567.000000000000395f/head//3 1b04c06b/rbd_data.10976b8b4567.000000000000395f/head//3(173'2351 client.4288.0:110866 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2351)
2014-06-12 14:51:54.392983 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  85c3c06b/rbd_data.10926b8b4567.0000000000002e09/head//3 85c3c06b/rbd_data.10926b8b4567.0000000000002e09/head//3(173'1898 client.4283.0:92131 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1898)
2014-06-12 14:51:54.392997 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6f0c06b/rbd_data.10b46b8b4567.000000000000378c/head//3 6f0c06b/rbd_data.10b46b8b4567.000000000000378c/head//3(173'2356 client.4317.0:110181 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2356)
2014-06-12 14:51:54.393010 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  45feb06b/rbd_data.108d6b8b4567.00000000000008d3/head//3 45feb06b/rbd_data.108d6b8b4567.00000000000008d3/head//3(173'2615 client.4278.0:131798 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2615)
2014-06-12 14:51:54.393024 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7e1cb06b/rbd_data.108f6b8b4567.0000000000002077/head//3 7e1cb06b/rbd_data.108f6b8b4567.0000000000002077/head//3(173'1387 client.4280.0:67055 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1387)
2014-06-12 14:51:54.393042 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  290bb06b/rbd_data.10ab6b8b4567.0000000000001011/head//3 290bb06b/rbd_data.10ab6b8b4567.0000000000001011/head//3(173'741 client.4308.0:38866 wrlock_by=unknown.0.0:0 dirty s 4194304 uv741)
2014-06-12 14:51:54.393056 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  709ab06b/rbd_data.10986b8b4567.000000000000012a/head//3 709ab06b/rbd_data.10986b8b4567.000000000000012a/head//3(173'2408 client.4289.0:117071 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2408)
2014-06-12 14:51:54.393070 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a24ab06b/rbd_data.109d6b8b4567.00000000000033b5/head//3 a24ab06b/rbd_data.109d6b8b4567.00000000000033b5/head//3(173'2061 client.4294.0:102404 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2061)
2014-06-12 14:51:54.393084 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  8069b06b/rbd_data.10936b8b4567.00000000000036fa/head//3 8069b06b/rbd_data.10936b8b4567.00000000000036fa/head//3(173'2191 client.4284.0:106415 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2191)
2014-06-12 14:51:54.393098 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1229b06b/rbd_data.108e6b8b4567.0000000000001fa8/head//3 1229b06b/rbd_data.108e6b8b4567.0000000000001fa8/head//3(173'1348 client.4279.0:66785 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1348)
2014-06-12 14:51:54.393112 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  e019b06b/rbd_data.10916b8b4567.0000000000002730/head//3 e019b06b/rbd_data.10916b8b4567.0000000000002730/head//3(173'1628 client.4282.0:80454 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1628)
2014-06-12 14:51:54.393126 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  af63b06b/rbd_data.108e6b8b4567.000000000000186f/head//3 af63b06b/rbd_data.108e6b8b4567.000000000000186f/head//3(173'1039 client.4279.0:53615 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1039)
2014-06-12 14:51:54.393140 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9132b06b/rbd_data.109b6b8b4567.0000000000001090/head//3 9132b06b/rbd_data.109b6b8b4567.0000000000001090/head//3(173'791 client.4292.0:37780 wrlock_by=unknown.0.0:0 dirty s 4194304 uv791)
2014-06-12 14:51:54.393153 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9141b06b/rbd_data.10916b8b4567.0000000000003a38/head//3 9141b06b/rbd_data.10916b8b4567.0000000000003a38/head//3(173'2372 client.4282.0:116059 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2372)
2014-06-12 14:51:54.393167 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  92cea06b/rbd_data.10ae6b8b4567.0000000000003a91/head//3 92cea06b/rbd_data.10ae6b8b4567.0000000000003a91/head//3(173'2182 client.4311.0:112610 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2182)
2014-06-12 14:51:54.393178 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:54.393189 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:54.393199 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:54.393201 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:54.393203 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:54.393246 7f9d60bd2700 10 collection_list_partial 3.6b_head start:7a74d06b//0//-1 is_max:0
2014-06-12 14:51:54.393498 7f9d60bd2700 20 objects: [4175d06b/rbd_data.10b16b8b4567.00000000000001b4/head//3,3bfcd06b/rbd_data.10926b8b4567.0000000000001ea1/head//3,953fd06b/rbd_data.10a16b8b4567.00000000000018cb/head//3,a76fd06b/rbd_data.108f6b8b4567.0000000000000081/head//3,8121e06b/rbd_data.10976b8b4567.000000000000075c/head//3,2cc2e06b/rbd_data.10b06b8b4567.0000000000002075/head//3,5066e06b/rbd_data.10b46b8b4567.0000000000001a06/head//3,7268e06b/rbd_data.10a86b8b4567.0000000000002d74/head//3,a8de06b/rbd_data.108e6b8b4567.0000000000000d1b/head//3,eac1f06b/rbd_data.10b46b8b4567.0000000000002b66/head//3,5562f06b/rbd_data.109a6b8b4567.0000000000003402/head//3,dea5f06b/rbd_data.10ad6b8b4567.0000000000000c8a/head//3,d766f06b/rbd_data.10966b8b4567.0000000000000a56/head//3,1796f06b/rbd_data.109e6b8b4567.00000000000027ab/head//3,5567f06b/rbd_data.10aa6b8b4567.0000000000001375/head//3,419cf06b/rbd_data.10ad6b8b4567.0000000000002c9d/head//3,cc6df06b/rbd_data.10996b8b4567.0000000000001866/head//3,d60ff06b/rbd_data.109c6b8b4567.0000000000001d3b/head//3,bf9ff06b/rbd_data.10aa6b8b4567.0000000000003a1c/head//3,56a0086b/rbd_data.10a56b8b4567.0000000000000ddc/head//3,ac1086b/rbd_data.108f6b8b4567.000000000000119f/head//3,4715086b/rbd_data.10a26b8b4567.0000000000000f9b/head//3,23c5086b/rbd_data.10a76b8b4567.0000000000000769/head//3,9ae5086b/rbd_data.10a26b8b4567.0000000000003b25/head//3,7108086b/rbd_data.108d6b8b4567.0000000000003b2c/head//3]
2014-06-12 14:51:54.394025 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  requesting scrubmap from osd.29
2014-06-12 14:51:54.394044 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:54.394058 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:54.394064 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:54.394069 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:54.394074 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:54.394078 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_map_chunk [7a74d06b//0//-1,e2c9086b//0//-1)
2014-06-12 14:51:54.394084 7f9d60bd2700 10 collection_list_partial 3.6b_head start:7a74d06b//0//-1 is_max:0
2014-06-12 14:51:54.416075 7f9d60bd2700 20 objects: [4175d06b/rbd_data.10b16b8b4567.00000000000001b4/head//3,3bfcd06b/rbd_data.10926b8b4567.0000000000001ea1/head//3,953fd06b/rbd_data.10a16b8b4567.00000000000018cb/head//3,a76fd06b/rbd_data.108f6b8b4567.0000000000000081/head//3,8121e06b/rbd_data.10976b8b4567.000000000000075c/head//3,2cc2e06b/rbd_data.10b06b8b4567.0000000000002075/head//3,5066e06b/rbd_data.10b46b8b4567.0000000000001a06/head//3,7268e06b/rbd_data.10a86b8b4567.0000000000002d74/head//3,a8de06b/rbd_data.108e6b8b4567.0000000000000d1b/head//3,eac1f06b/rbd_data.10b46b8b4567.0000000000002b66/head//3,5562f06b/rbd_data.109a6b8b4567.0000000000003402/head//3,dea5f06b/rbd_data.10ad6b8b4567.0000000000000c8a/head//3,d766f06b/rbd_data.10966b8b4567.0000000000000a56/head//3,1796f06b/rbd_data.109e6b8b4567.00000000000027ab/head//3,5567f06b/rbd_data.10aa6b8b4567.0000000000001375/head//3,419cf06b/rbd_data.10ad6b8b4567.0000000000002c9d/head//3,cc6df06b/rbd_data.10996b8b4567.0000000000001866/head//3,d60ff06b/rbd_data.109c6b8b4567.0000000000001d3b/head//3,bf9ff06b/rbd_data.10aa6b8b4567.0000000000003a1c/head//3,56a0086b/rbd_data.10a56b8b4567.0000000000000ddc/head//3,ac1086b/rbd_data.108f6b8b4567.000000000000119f/head//3,4715086b/rbd_data.10a26b8b4567.0000000000000f9b/head//3,23c5086b/rbd_data.10a76b8b4567.0000000000000769/head//3,9ae5086b/rbd_data.10a26b8b4567.0000000000003b25/head//3,7108086b/rbd_data.108d6b8b4567.0000000000003b2c/head//3,e2c9086b/rbd_data.109a6b8b4567.00000000000013c0/head//3,e07b086b/rbd_data.10b26b8b4567.0000000000003b18/head//3,5710186b/rbd_data.10ad6b8b4567.000000000000247e/head//3,fae1186b/rbd_data.10946b8b4567.0000000000002a53/head//3,2f32186b/rbd_data.10a86b8b4567.00000000000016a3/head//3,8f23186b/rbd_data.10b16b8b4567.0000000000003424/head//3,ad04186b/rbd_data.10996b8b4567.00000000000022e5/head//3,35e4186b/rbd_data.109a6b8b4567.000000000000142e/head//3,6c17186b/rbd_data.10ac6b8b4567.00000000000035b8/head//3,d957186b/rbd_data.10a26b8b4567.00000000000021e7/head//3,7f09186b/rbd_data.10b06b8b4567.0000000000003923/head//3,af3c186b/rbd_data.10b26b8b4567.0000000000001f3d/head//3,c29d186b/rbd_data.10a36b8b4567.0000000000002e8c/head//3,baaf186b/rbd_data.10ab6b8b4567.000000000000316b/head//3,b203286b/rbd_data.10a46b8b4567.0000000000000c91/head//3,dd87286b/rbd_data.108f6b8b4567.00000000000013dd/head//3,e7c8286b/rbd_data.10b46b8b4567.0000000000002aa3/head//3,f4ac286b/rbd_data.108f6b8b4567.00000000000016d8/head//3,226d286b/rbd_data.10966b8b4567.00000000000020b0/head//3,492e286b/rbd_data.10a16b8b4567.0000000000001476/head//3,9682386b/rbd_data.10a46b8b4567.00000000000003dc/head//3,ca34386b/rbd_data.10916b8b4567.00000000000013c7/head//3,a5c4386b/rbd_data.10a76b8b4567.00000000000001c4/head//3,2bd4386b/rbd_data.10b46b8b4567.0000000000001357/head//3,75e5386b/rbd_data.108d6b8b4567.0000000000003316/head//3,9707386b/rbd_data.10986b8b4567.0000000000001dec/head//3,5217386b/rbd_data.10936b8b4567.0000000000000369/head//3,2058386b/rbd_data.10ac6b8b4567.0000000000002160/head//3,cdb8386b/rbd_data.10b36b8b4567.0000000000001dd8/head//3,e2ad386b/rbd_data.109c6b8b4567.000000000000287d/head//3,38bd386b/rbd_data.10aa6b8b4567.0000000000002b5e/head//3,1d8f386b/rbd_data.108e6b8b4567.0000000000003a1f/head//3,5191486b/rbd_data.10b06b8b4567.00000000000029aa/head//3,bbd1486b/rbd_data.108e6b8b4567.0000000000002410/head//3,1d13486b/rbd_data.10b26b8b4567.00000000000003c3/head//3,76d5486b/rbd_data.109f6b8b4567.0000000000003445/head//3,5928486b/rbd_data.10a76b8b4567.0000000000001f53/head//3,d94b486b/rbd_data.109a6b8b4567.000000000000101c/head//3,f6db486b/rbd_data.10aa6b8b4567.0000000000002bc7/head//3]
2014-06-12 14:51:54.416143 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:54.416161 7f9d60bd2700 10 stat 3.6b_head/4175d06b/rbd_data.10b16b8b4567.00000000000001b4/head//3
2014-06-12 14:51:54.416264 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 4175d06b/rbd_data.10b16b8b4567.00000000000001b4/head//3
2014-06-12 14:51:54.452553 7f9d60bd2700 10 getattrs 3.6b_head/4175d06b/rbd_data.10b16b8b4567.00000000000001b4/head//3 = 0
2014-06-12 14:51:54.452572 7f9d60bd2700 10 stat 3.6b_head/3bfcd06b/rbd_data.10926b8b4567.0000000000001ea1/head//3
2014-06-12 14:51:54.452667 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 3bfcd06b/rbd_data.10926b8b4567.0000000000001ea1/head//3
2014-06-12 14:51:54.465353 7f9d60bd2700 10 getattrs 3.6b_head/3bfcd06b/rbd_data.10926b8b4567.0000000000001ea1/head//3 = 0
2014-06-12 14:51:54.465361 7f9d60bd2700 10 stat 3.6b_head/953fd06b/rbd_data.10a16b8b4567.00000000000018cb/head//3
2014-06-12 14:51:54.465451 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 953fd06b/rbd_data.10a16b8b4567.00000000000018cb/head//3
2014-06-12 14:51:54.497531 7f9d60bd2700 10 getattrs 3.6b_head/953fd06b/rbd_data.10a16b8b4567.00000000000018cb/head//3 = 0
2014-06-12 14:51:54.497540 7f9d60bd2700 10 stat 3.6b_head/a76fd06b/rbd_data.108f6b8b4567.0000000000000081/head//3
2014-06-12 14:51:54.497663 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a76fd06b/rbd_data.108f6b8b4567.0000000000000081/head//3
2014-06-12 14:51:54.522749 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:54.523595 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:54.533049 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:54.552779 7f9d60bd2700 10 getattrs 3.6b_head/a76fd06b/rbd_data.108f6b8b4567.0000000000000081/head//3 = 0
2014-06-12 14:51:54.552803 7f9d60bd2700 10 stat 3.6b_head/8121e06b/rbd_data.10976b8b4567.000000000000075c/head//3
2014-06-12 14:51:54.552961 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 8121e06b/rbd_data.10976b8b4567.000000000000075c/head//3
2014-06-12 14:51:54.570503 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:54.586977 7f9d60bd2700 10 getattrs 3.6b_head/8121e06b/rbd_data.10976b8b4567.000000000000075c/head//3 = 0
2014-06-12 14:51:54.586986 7f9d60bd2700 10 stat 3.6b_head/2cc2e06b/rbd_data.10b06b8b4567.0000000000002075/head//3
2014-06-12 14:51:54.587082 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 2cc2e06b/rbd_data.10b06b8b4567.0000000000002075/head//3
2014-06-12 14:51:54.619445 7f9d60bd2700 10 getattrs 3.6b_head/2cc2e06b/rbd_data.10b06b8b4567.0000000000002075/head//3 = 0
2014-06-12 14:51:54.619453 7f9d60bd2700 10 stat 3.6b_head/5066e06b/rbd_data.10b46b8b4567.0000000000001a06/head//3
2014-06-12 14:51:54.619552 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5066e06b/rbd_data.10b46b8b4567.0000000000001a06/head//3
2014-06-12 14:51:54.643049 7f9d60bd2700 10 getattrs 3.6b_head/5066e06b/rbd_data.10b46b8b4567.0000000000001a06/head//3 = 0
2014-06-12 14:51:54.643058 7f9d60bd2700 10 stat 3.6b_head/7268e06b/rbd_data.10a86b8b4567.0000000000002d74/head//3
2014-06-12 14:51:54.643168 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7268e06b/rbd_data.10a86b8b4567.0000000000002d74/head//3
2014-06-12 14:51:54.644293 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:54.662027 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:54.662271 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:54.662370 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:54.666157 7f9d60bd2700 10 getattrs 3.6b_head/7268e06b/rbd_data.10a86b8b4567.0000000000002d74/head//3 = 0
2014-06-12 14:51:54.666167 7f9d60bd2700 10 stat 3.6b_head/a8de06b/rbd_data.108e6b8b4567.0000000000000d1b/head//3
2014-06-12 14:51:54.666289 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a8de06b/rbd_data.108e6b8b4567.0000000000000d1b/head//3
2014-06-12 14:51:54.700006 7f9d60bd2700 10 getattrs 3.6b_head/a8de06b/rbd_data.108e6b8b4567.0000000000000d1b/head//3 = 0
2014-06-12 14:51:54.700019 7f9d60bd2700 10 stat 3.6b_head/eac1f06b/rbd_data.10b46b8b4567.0000000000002b66/head//3
2014-06-12 14:51:54.700152 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid eac1f06b/rbd_data.10b46b8b4567.0000000000002b66/head//3
2014-06-12 14:51:54.701232 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:54.725531 7f9d60bd2700 10 getattrs 3.6b_head/eac1f06b/rbd_data.10b46b8b4567.0000000000002b66/head//3 = 0
2014-06-12 14:51:54.725542 7f9d60bd2700 10 stat 3.6b_head/5562f06b/rbd_data.109a6b8b4567.0000000000003402/head//3
2014-06-12 14:51:54.725663 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5562f06b/rbd_data.109a6b8b4567.0000000000003402/head//3
2014-06-12 14:51:54.727030 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:54.743659 7f9d60bd2700 10 getattrs 3.6b_head/5562f06b/rbd_data.109a6b8b4567.0000000000003402/head//3 = 0
2014-06-12 14:51:54.743667 7f9d60bd2700 10 stat 3.6b_head/dea5f06b/rbd_data.10ad6b8b4567.0000000000000c8a/head//3
2014-06-12 14:51:54.743759 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid dea5f06b/rbd_data.10ad6b8b4567.0000000000000c8a/head//3
2014-06-12 14:51:54.743872 7f9d60bd2700 10 getattrs 3.6b_head/dea5f06b/rbd_data.10ad6b8b4567.0000000000000c8a/head//3 = 0
2014-06-12 14:51:54.743878 7f9d60bd2700 10 stat 3.6b_head/d766f06b/rbd_data.10966b8b4567.0000000000000a56/head//3
2014-06-12 14:51:54.743967 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid d766f06b/rbd_data.10966b8b4567.0000000000000a56/head//3
2014-06-12 14:51:54.768709 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:54.792171 7f9d60bd2700 10 getattrs 3.6b_head/d766f06b/rbd_data.10966b8b4567.0000000000000a56/head//3 = 0
2014-06-12 14:51:54.792184 7f9d60bd2700 10 stat 3.6b_head/1796f06b/rbd_data.109e6b8b4567.00000000000027ab/head//3
2014-06-12 14:51:54.792298 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1796f06b/rbd_data.109e6b8b4567.00000000000027ab/head//3
2014-06-12 14:51:54.802622 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:54.821730 7f9d60bd2700 10 getattrs 3.6b_head/1796f06b/rbd_data.109e6b8b4567.00000000000027ab/head//3 = 0
2014-06-12 14:51:54.821739 7f9d60bd2700 10 stat 3.6b_head/5567f06b/rbd_data.10aa6b8b4567.0000000000001375/head//3
2014-06-12 14:51:54.821846 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5567f06b/rbd_data.10aa6b8b4567.0000000000001375/head//3
2014-06-12 14:51:54.855758 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:54.861424 7f9d60bd2700 10 getattrs 3.6b_head/5567f06b/rbd_data.10aa6b8b4567.0000000000001375/head//3 = 0
2014-06-12 14:51:54.861435 7f9d60bd2700 10 stat 3.6b_head/419cf06b/rbd_data.10ad6b8b4567.0000000000002c9d/head//3
2014-06-12 14:51:54.861558 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 419cf06b/rbd_data.10ad6b8b4567.0000000000002c9d/head//3
2014-06-12 14:51:54.861714 7f9d60bd2700 10 getattrs 3.6b_head/419cf06b/rbd_data.10ad6b8b4567.0000000000002c9d/head//3 = 0
2014-06-12 14:51:54.861722 7f9d60bd2700 10 stat 3.6b_head/cc6df06b/rbd_data.10996b8b4567.0000000000001866/head//3
2014-06-12 14:51:54.861834 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid cc6df06b/rbd_data.10996b8b4567.0000000000001866/head//3
2014-06-12 14:51:54.866120 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:54.869472 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:54.883044 7f9d60bd2700 10 getattrs 3.6b_head/cc6df06b/rbd_data.10996b8b4567.0000000000001866/head//3 = 0
2014-06-12 14:51:54.883056 7f9d60bd2700 10 stat 3.6b_head/d60ff06b/rbd_data.109c6b8b4567.0000000000001d3b/head//3
2014-06-12 14:51:54.883168 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid d60ff06b/rbd_data.109c6b8b4567.0000000000001d3b/head//3
2014-06-12 14:51:54.908218 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:54.915224 7f9d60bd2700 10 getattrs 3.6b_head/d60ff06b/rbd_data.109c6b8b4567.0000000000001d3b/head//3 = 0
2014-06-12 14:51:54.915236 7f9d60bd2700 10 stat 3.6b_head/bf9ff06b/rbd_data.10aa6b8b4567.0000000000003a1c/head//3
2014-06-12 14:51:54.915352 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid bf9ff06b/rbd_data.10aa6b8b4567.0000000000003a1c/head//3
2014-06-12 14:51:54.928697 7f9d60bd2700 10 getattrs 3.6b_head/bf9ff06b/rbd_data.10aa6b8b4567.0000000000003a1c/head//3 = 0
2014-06-12 14:51:54.928704 7f9d60bd2700 10 stat 3.6b_head/56a0086b/rbd_data.10a56b8b4567.0000000000000ddc/head//3
2014-06-12 14:51:54.928797 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 56a0086b/rbd_data.10a56b8b4567.0000000000000ddc/head//3
2014-06-12 14:51:54.932879 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:54.933016 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:54.947236 7f9d60bd2700 10 getattrs 3.6b_head/56a0086b/rbd_data.10a56b8b4567.0000000000000ddc/head//3 = 0
2014-06-12 14:51:54.947247 7f9d60bd2700 10 stat 3.6b_head/ac1086b/rbd_data.108f6b8b4567.000000000000119f/head//3
2014-06-12 14:51:54.947354 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ac1086b/rbd_data.108f6b8b4567.000000000000119f/head//3
2014-06-12 14:51:54.965800 7f9d60bd2700 10 getattrs 3.6b_head/ac1086b/rbd_data.108f6b8b4567.000000000000119f/head//3 = 0
2014-06-12 14:51:54.965808 7f9d60bd2700 10 stat 3.6b_head/4715086b/rbd_data.10a26b8b4567.0000000000000f9b/head//3
2014-06-12 14:51:54.965912 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 4715086b/rbd_data.10a26b8b4567.0000000000000f9b/head//3
2014-06-12 14:51:54.966715 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:54.985172 7f9d60bd2700 10 getattrs 3.6b_head/4715086b/rbd_data.10a26b8b4567.0000000000000f9b/head//3 = 0
2014-06-12 14:51:54.985188 7f9d60bd2700 10 stat 3.6b_head/23c5086b/rbd_data.10a76b8b4567.0000000000000769/head//3
2014-06-12 14:51:54.985279 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 23c5086b/rbd_data.10a76b8b4567.0000000000000769/head//3
2014-06-12 14:51:54.994148 7f9d5501e700  5 -- op tracker -- , seq: 748613, time: 2014-06-12 14:51:54.994073, event: header_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:54.994158 7f9d5501e700  5 -- op tracker -- , seq: 748613, time: 2014-06-12 14:51:54.994075, event: throttled, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:54.994164 7f9d5501e700  5 -- op tracker -- , seq: 748613, time: 2014-06-12 14:51:54.994117, event: all_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:54.994169 7f9d5501e700  5 -- op tracker -- , seq: 748613, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:54.994173 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:54.994178 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:54.994185 7f9d5501e700 15 osd.2 173 enqueue_op 0x31d60d00 prio 127 cost 10717 latency 0.000112 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:54.994253 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:55.017046 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:55.017095 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:55.020148 7f9d60bd2700 10 getattrs 3.6b_head/23c5086b/rbd_data.10a76b8b4567.0000000000000769/head//3 = 0
2014-06-12 14:51:55.020171 7f9d60bd2700 10 stat 3.6b_head/9ae5086b/rbd_data.10a26b8b4567.0000000000003b25/head//3
2014-06-12 14:51:55.020292 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9ae5086b/rbd_data.10a26b8b4567.0000000000003b25/head//3
2014-06-12 14:51:55.030798 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:55.055108 7f9d60bd2700 10 getattrs 3.6b_head/9ae5086b/rbd_data.10a26b8b4567.0000000000003b25/head//3 = 0
2014-06-12 14:51:55.055127 7f9d60bd2700 10 stat 3.6b_head/7108086b/rbd_data.108d6b8b4567.0000000000003b2c/head//3
2014-06-12 14:51:55.055261 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7108086b/rbd_data.108d6b8b4567.0000000000003b2c/head//3
2014-06-12 14:51:55.086229 7f9d60bd2700 10 getattrs 3.6b_head/7108086b/rbd_data.108d6b8b4567.0000000000003b2c/head//3 = 0
2014-06-12 14:51:55.086246 7f9d60bd2700 10 collection_getattrs 3.6b_head
2014-06-12 14:51:55.086305 7f9d60bd2700 10 collection_getattrs 3.6b_head = 0
2014-06-12 14:51:55.086309 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:55.086330 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:55.086336 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:55.086347 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:55.086351 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:55.086380 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x31d60d00 prio 127 cost 10717 latency 0.092307 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]
2014-06-12 14:51:55.086444 7f9d6a3e5700  5 -- op tracker -- , seq: 748613, time: 2014-06-12 14:51:55.086436, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:55.086468 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] handle_message: 0x31d60d00
2014-06-12 14:51:55.086484 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:55.086503 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:55.086522 7f9d6a3e5700  5 -- op tracker -- , seq: 748613, time: 2014-06-12 14:51:55.086517, event: started, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:55.086529 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  got 29 scrub map
2014-06-12 14:51:55.086638 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] map version is 173'2887
2014-06-12 14:51:55.086658 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x31d60d00 finish
2014-06-12 14:51:55.086670 7f9d6a3e5700  5 -- op tracker -- , seq: 748613, time: 2014-06-12 14:51:55.086664, event: done, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:55.086698 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:55.086703 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:55.086697 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:55.086711 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:55.086733 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:55.086746 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:55.086787 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:55.086794 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:55.086801 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub replica 29 has 25 items
2014-06-12 14:51:55.086822 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4175d06b/rbd_data.10b16b8b4567.00000000000001b4/head//3, auth == maps.end()
2014-06-12 14:51:55.086849 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 4175d06b/rbd_data.10b16b8b4567.00000000000001b4/head//3
2014-06-12 14:51:55.086863 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3bfcd06b/rbd_data.10926b8b4567.0000000000001ea1/head//3, auth == maps.end()
2014-06-12 14:51:55.086872 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 3bfcd06b/rbd_data.10926b8b4567.0000000000001ea1/head//3
2014-06-12 14:51:55.086882 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 953fd06b/rbd_data.10a16b8b4567.00000000000018cb/head//3, auth == maps.end()
2014-06-12 14:51:55.086891 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 953fd06b/rbd_data.10a16b8b4567.00000000000018cb/head//3
2014-06-12 14:51:55.086900 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a76fd06b/rbd_data.108f6b8b4567.0000000000000081/head//3, auth == maps.end()
2014-06-12 14:51:55.086909 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a76fd06b/rbd_data.108f6b8b4567.0000000000000081/head//3
2014-06-12 14:51:55.086918 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8121e06b/rbd_data.10976b8b4567.000000000000075c/head//3, auth == maps.end()
2014-06-12 14:51:55.086927 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 8121e06b/rbd_data.10976b8b4567.000000000000075c/head//3
2014-06-12 14:51:55.086936 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2cc2e06b/rbd_data.10b06b8b4567.0000000000002075/head//3, auth == maps.end()
2014-06-12 14:51:55.086945 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 2cc2e06b/rbd_data.10b06b8b4567.0000000000002075/head//3
2014-06-12 14:51:55.086954 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5066e06b/rbd_data.10b46b8b4567.0000000000001a06/head//3, auth == maps.end()
2014-06-12 14:51:55.086963 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5066e06b/rbd_data.10b46b8b4567.0000000000001a06/head//3
2014-06-12 14:51:55.086972 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7268e06b/rbd_data.10a86b8b4567.0000000000002d74/head//3, auth == maps.end()
2014-06-12 14:51:55.086981 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7268e06b/rbd_data.10a86b8b4567.0000000000002d74/head//3
2014-06-12 14:51:55.086990 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a8de06b/rbd_data.108e6b8b4567.0000000000000d1b/head//3, auth == maps.end()
2014-06-12 14:51:55.086999 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a8de06b/rbd_data.108e6b8b4567.0000000000000d1b/head//3
2014-06-12 14:51:55.087008 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj eac1f06b/rbd_data.10b46b8b4567.0000000000002b66/head//3, auth == maps.end()
2014-06-12 14:51:55.087020 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj eac1f06b/rbd_data.10b46b8b4567.0000000000002b66/head//3
2014-06-12 14:51:55.087031 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5562f06b/rbd_data.109a6b8b4567.0000000000003402/head//3, auth == maps.end()
2014-06-12 14:51:55.087042 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5562f06b/rbd_data.109a6b8b4567.0000000000003402/head//3
2014-06-12 14:51:55.087053 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dea5f06b/rbd_data.10ad6b8b4567.0000000000000c8a/head//3, auth == maps.end()
2014-06-12 14:51:55.087062 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj dea5f06b/rbd_data.10ad6b8b4567.0000000000000c8a/head//3
2014-06-12 14:51:55.087073 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d766f06b/rbd_data.10966b8b4567.0000000000000a56/head//3, auth == maps.end()
2014-06-12 14:51:55.087082 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj d766f06b/rbd_data.10966b8b4567.0000000000000a56/head//3
2014-06-12 14:51:55.087092 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1796f06b/rbd_data.109e6b8b4567.00000000000027ab/head//3, auth == maps.end()
2014-06-12 14:51:55.087101 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1796f06b/rbd_data.109e6b8b4567.00000000000027ab/head//3
2014-06-12 14:51:55.087111 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5567f06b/rbd_data.10aa6b8b4567.0000000000001375/head//3, auth == maps.end()
2014-06-12 14:51:55.087121 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5567f06b/rbd_data.10aa6b8b4567.0000000000001375/head//3
2014-06-12 14:51:55.087130 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 419cf06b/rbd_data.10ad6b8b4567.0000000000002c9d/head//3, auth == maps.end()
2014-06-12 14:51:55.087139 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 419cf06b/rbd_data.10ad6b8b4567.0000000000002c9d/head//3
2014-06-12 14:51:55.087148 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cc6df06b/rbd_data.10996b8b4567.0000000000001866/head//3, auth == maps.end()
2014-06-12 14:51:55.087157 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj cc6df06b/rbd_data.10996b8b4567.0000000000001866/head//3
2014-06-12 14:51:55.087166 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d60ff06b/rbd_data.109c6b8b4567.0000000000001d3b/head//3, auth == maps.end()
2014-06-12 14:51:55.087175 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj d60ff06b/rbd_data.109c6b8b4567.0000000000001d3b/head//3
2014-06-12 14:51:55.087184 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bf9ff06b/rbd_data.10aa6b8b4567.0000000000003a1c/head//3, auth == maps.end()
2014-06-12 14:51:55.087193 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj bf9ff06b/rbd_data.10aa6b8b4567.0000000000003a1c/head//3
2014-06-12 14:51:55.087204 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 56a0086b/rbd_data.10a56b8b4567.0000000000000ddc/head//3, auth == maps.end()
2014-06-12 14:51:55.087213 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 56a0086b/rbd_data.10a56b8b4567.0000000000000ddc/head//3
2014-06-12 14:51:55.087223 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ac1086b/rbd_data.108f6b8b4567.000000000000119f/head//3, auth == maps.end()
2014-06-12 14:51:55.087233 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ac1086b/rbd_data.108f6b8b4567.000000000000119f/head//3
2014-06-12 14:51:55.087243 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4715086b/rbd_data.10a26b8b4567.0000000000000f9b/head//3, auth == maps.end()
2014-06-12 14:51:55.087253 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 4715086b/rbd_data.10a26b8b4567.0000000000000f9b/head//3
2014-06-12 14:51:55.087264 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 23c5086b/rbd_data.10a76b8b4567.0000000000000769/head//3, auth == maps.end()
2014-06-12 14:51:55.087275 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 23c5086b/rbd_data.10a76b8b4567.0000000000000769/head//3
2014-06-12 14:51:55.087286 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9ae5086b/rbd_data.10a26b8b4567.0000000000003b25/head//3, auth == maps.end()
2014-06-12 14:51:55.087296 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9ae5086b/rbd_data.10a26b8b4567.0000000000003b25/head//3
2014-06-12 14:51:55.087307 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7108086b/rbd_data.108d6b8b4567.0000000000003b2c/head//3, auth == maps.end()
2014-06-12 14:51:55.087317 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7108086b/rbd_data.108d6b8b4567.0000000000003b2c/head//3
2014-06-12 14:51:55.087329 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] 
2014-06-12 14:51:55.087336 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub
2014-06-12 14:51:55.087349 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7108086b/rbd_data.108d6b8b4567.0000000000003b2c/head//3 7108086b/rbd_data.108d6b8b4567.0000000000003b2c/head//3(173'2260 client.4278.0:117215 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2260)
2014-06-12 14:51:55.087364 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9ae5086b/rbd_data.10a26b8b4567.0000000000003b25/head//3 9ae5086b/rbd_data.10a26b8b4567.0000000000003b25/head//3(173'2397 client.4299.0:118587 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2397)
2014-06-12 14:51:55.087378 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  23c5086b/rbd_data.10a76b8b4567.0000000000000769/head//3 23c5086b/rbd_data.10a76b8b4567.0000000000000769/head//3(173'2569 client.4304.0:128521 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2569)
2014-06-12 14:51:55.087391 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  4715086b/rbd_data.10a26b8b4567.0000000000000f9b/head//3 4715086b/rbd_data.10a26b8b4567.0000000000000f9b/head//3(173'746 client.4299.0:37896 wrlock_by=unknown.0.0:0 dirty s 4194304 uv746)
2014-06-12 14:51:55.087404 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ac1086b/rbd_data.108f6b8b4567.000000000000119f/head//3 ac1086b/rbd_data.108f6b8b4567.000000000000119f/head//3(173'782 client.4280.0:40531 wrlock_by=unknown.0.0:0 dirty s 4194304 uv782)
2014-06-12 14:51:55.087417 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  56a0086b/rbd_data.10a56b8b4567.0000000000000ddc/head//3 56a0086b/rbd_data.10a56b8b4567.0000000000000ddc/head//3(173'2786 client.4302.0:142075 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2786)
2014-06-12 14:51:55.087430 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  bf9ff06b/rbd_data.10aa6b8b4567.0000000000003a1c/head//3 bf9ff06b/rbd_data.10aa6b8b4567.0000000000003a1c/head//3(173'2287 client.4307.0:116852 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2287)
2014-06-12 14:51:55.087443 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  d60ff06b/rbd_data.109c6b8b4567.0000000000001d3b/head//3 d60ff06b/rbd_data.109c6b8b4567.0000000000001d3b/head//3(173'1307 client.4293.0:61495 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1307)
2014-06-12 14:51:55.087455 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  cc6df06b/rbd_data.10996b8b4567.0000000000001866/head//3 cc6df06b/rbd_data.10996b8b4567.0000000000001866/head//3(173'982 client.4290.0:52586 wrlock_by=unknown.0.0:0 dirty s 4194304 uv982)
2014-06-12 14:51:55.087468 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  419cf06b/rbd_data.10ad6b8b4567.0000000000002c9d/head//3 419cf06b/rbd_data.10ad6b8b4567.0000000000002c9d/head//3(173'1822 client.4310.0:89864 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1822)
2014-06-12 14:51:55.087481 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5567f06b/rbd_data.10aa6b8b4567.0000000000001375/head//3 5567f06b/rbd_data.10aa6b8b4567.0000000000001375/head//3(173'919 client.4307.0:47278 wrlock_by=unknown.0.0:0 dirty s 4194304 uv919)
2014-06-12 14:51:55.087494 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1796f06b/rbd_data.109e6b8b4567.00000000000027ab/head//3 1796f06b/rbd_data.109e6b8b4567.00000000000027ab/head//3(173'1618 client.4295.0:81991 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1618)
2014-06-12 14:51:55.087507 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  d766f06b/rbd_data.10966b8b4567.0000000000000a56/head//3 d766f06b/rbd_data.10966b8b4567.0000000000000a56/head//3(173'2694 client.4287.0:131221 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2694)
2014-06-12 14:51:55.087520 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  dea5f06b/rbd_data.10ad6b8b4567.0000000000000c8a/head//3 dea5f06b/rbd_data.10ad6b8b4567.0000000000000c8a/head//3(173'2792 client.4310.0:138133 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2792)
2014-06-12 14:51:55.087533 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5562f06b/rbd_data.109a6b8b4567.0000000000003402/head//3 5562f06b/rbd_data.109a6b8b4567.0000000000003402/head//3(173'2073 client.4291.0:101870 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2073)
2014-06-12 14:51:55.087547 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  eac1f06b/rbd_data.10b46b8b4567.0000000000002b66/head//3 eac1f06b/rbd_data.10b46b8b4567.0000000000002b66/head//3(173'1925 client.4317.0:87789 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1925)
2014-06-12 14:51:55.087560 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a8de06b/rbd_data.108e6b8b4567.0000000000000d1b/head//3 a8de06b/rbd_data.108e6b8b4567.0000000000000d1b/head//3(173'2752 client.4279.0:138400 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2752)
2014-06-12 14:51:55.087573 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7268e06b/rbd_data.10a86b8b4567.0000000000002d74/head//3 7268e06b/rbd_data.10a86b8b4567.0000000000002d74/head//3(173'1755 client.4305.0:90863 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1755)
2014-06-12 14:51:55.087586 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5066e06b/rbd_data.10b46b8b4567.0000000000001a06/head//3 5066e06b/rbd_data.10b46b8b4567.0000000000001a06/head//3(173'1179 client.4317.0:56309 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1179)
2014-06-12 14:51:55.087599 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  2cc2e06b/rbd_data.10b06b8b4567.0000000000002075/head//3 2cc2e06b/rbd_data.10b06b8b4567.0000000000002075/head//3(173'1356 client.4313.0:68585 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1356)
2014-06-12 14:51:55.087612 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  8121e06b/rbd_data.10976b8b4567.000000000000075c/head//3 8121e06b/rbd_data.10976b8b4567.000000000000075c/head//3(173'2606 client.4288.0:126562 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2606)
2014-06-12 14:51:55.087624 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a76fd06b/rbd_data.108f6b8b4567.0000000000000081/head//3 a76fd06b/rbd_data.108f6b8b4567.0000000000000081/head//3(173'2402 client.4280.0:116358 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2402)
2014-06-12 14:51:55.087637 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  953fd06b/rbd_data.10a16b8b4567.00000000000018cb/head//3 953fd06b/rbd_data.10a16b8b4567.00000000000018cb/head//3(173'1052 client.4298.0:55106 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1052)
2014-06-12 14:51:55.087649 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  3bfcd06b/rbd_data.10926b8b4567.0000000000001ea1/head//3 3bfcd06b/rbd_data.10926b8b4567.0000000000001ea1/head//3(173'1288 client.4283.0:64236 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1288)
2014-06-12 14:51:55.087662 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  4175d06b/rbd_data.10b16b8b4567.00000000000001b4/head//3 4175d06b/rbd_data.10b16b8b4567.00000000000001b4/head//3(173'2441 client.4314.0:120426 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2441)
2014-06-12 14:51:55.087672 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:55.087684 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:55.087693 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:55.087695 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:55.087696 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:55.087736 7f9d60bd2700 10 collection_list_partial 3.6b_head start:e2c9086b//0//-1 is_max:0
2014-06-12 14:51:55.087952 7f9d60bd2700 20 objects: [e07b086b/rbd_data.10b26b8b4567.0000000000003b18/head//3,5710186b/rbd_data.10ad6b8b4567.000000000000247e/head//3,fae1186b/rbd_data.10946b8b4567.0000000000002a53/head//3,2f32186b/rbd_data.10a86b8b4567.00000000000016a3/head//3,8f23186b/rbd_data.10b16b8b4567.0000000000003424/head//3,ad04186b/rbd_data.10996b8b4567.00000000000022e5/head//3,35e4186b/rbd_data.109a6b8b4567.000000000000142e/head//3,6c17186b/rbd_data.10ac6b8b4567.00000000000035b8/head//3,d957186b/rbd_data.10a26b8b4567.00000000000021e7/head//3,7f09186b/rbd_data.10b06b8b4567.0000000000003923/head//3,af3c186b/rbd_data.10b26b8b4567.0000000000001f3d/head//3,c29d186b/rbd_data.10a36b8b4567.0000000000002e8c/head//3,baaf186b/rbd_data.10ab6b8b4567.000000000000316b/head//3,b203286b/rbd_data.10a46b8b4567.0000000000000c91/head//3,dd87286b/rbd_data.108f6b8b4567.00000000000013dd/head//3,e7c8286b/rbd_data.10b46b8b4567.0000000000002aa3/head//3,f4ac286b/rbd_data.108f6b8b4567.00000000000016d8/head//3,226d286b/rbd_data.10966b8b4567.00000000000020b0/head//3,492e286b/rbd_data.10a16b8b4567.0000000000001476/head//3,9682386b/rbd_data.10a46b8b4567.00000000000003dc/head//3,ca34386b/rbd_data.10916b8b4567.00000000000013c7/head//3,a5c4386b/rbd_data.10a76b8b4567.00000000000001c4/head//3,2bd4386b/rbd_data.10b46b8b4567.0000000000001357/head//3,75e5386b/rbd_data.108d6b8b4567.0000000000003316/head//3,9707386b/rbd_data.10986b8b4567.0000000000001dec/head//3]
2014-06-12 14:51:55.088463 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  requesting scrubmap from osd.29
2014-06-12 14:51:55.088486 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:55.088504 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:55.088513 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:55.088519 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:55.088524 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:55.088530 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_map_chunk [e2c9086b//0//-1,5217386b//0//-1)
2014-06-12 14:51:55.088537 7f9d60bd2700 10 collection_list_partial 3.6b_head start:e2c9086b//0//-1 is_max:0
2014-06-12 14:51:55.089028 7f9d60bd2700 20 objects: [e07b086b/rbd_data.10b26b8b4567.0000000000003b18/head//3,5710186b/rbd_data.10ad6b8b4567.000000000000247e/head//3,fae1186b/rbd_data.10946b8b4567.0000000000002a53/head//3,2f32186b/rbd_data.10a86b8b4567.00000000000016a3/head//3,8f23186b/rbd_data.10b16b8b4567.0000000000003424/head//3,ad04186b/rbd_data.10996b8b4567.00000000000022e5/head//3,35e4186b/rbd_data.109a6b8b4567.000000000000142e/head//3,6c17186b/rbd_data.10ac6b8b4567.00000000000035b8/head//3,d957186b/rbd_data.10a26b8b4567.00000000000021e7/head//3,7f09186b/rbd_data.10b06b8b4567.0000000000003923/head//3,af3c186b/rbd_data.10b26b8b4567.0000000000001f3d/head//3,c29d186b/rbd_data.10a36b8b4567.0000000000002e8c/head//3,baaf186b/rbd_data.10ab6b8b4567.000000000000316b/head//3,b203286b/rbd_data.10a46b8b4567.0000000000000c91/head//3,dd87286b/rbd_data.108f6b8b4567.00000000000013dd/head//3,e7c8286b/rbd_data.10b46b8b4567.0000000000002aa3/head//3,f4ac286b/rbd_data.108f6b8b4567.00000000000016d8/head//3,226d286b/rbd_data.10966b8b4567.00000000000020b0/head//3,492e286b/rbd_data.10a16b8b4567.0000000000001476/head//3,9682386b/rbd_data.10a46b8b4567.00000000000003dc/head//3,ca34386b/rbd_data.10916b8b4567.00000000000013c7/head//3,a5c4386b/rbd_data.10a76b8b4567.00000000000001c4/head//3,2bd4386b/rbd_data.10b46b8b4567.0000000000001357/head//3,75e5386b/rbd_data.108d6b8b4567.0000000000003316/head//3,9707386b/rbd_data.10986b8b4567.0000000000001dec/head//3,5217386b/rbd_data.10936b8b4567.0000000000000369/head//3,2058386b/rbd_data.10ac6b8b4567.0000000000002160/head//3,cdb8386b/rbd_data.10b36b8b4567.0000000000001dd8/head//3,e2ad386b/rbd_data.109c6b8b4567.000000000000287d/head//3,38bd386b/rbd_data.10aa6b8b4567.0000000000002b5e/head//3,1d8f386b/rbd_data.108e6b8b4567.0000000000003a1f/head//3,5191486b/rbd_data.10b06b8b4567.00000000000029aa/head//3,bbd1486b/rbd_data.108e6b8b4567.0000000000002410/head//3,1d13486b/rbd_data.10b26b8b4567.00000000000003c3/head//3,76d5486b/rbd_data.109f6b8b4567.0000000000003445/head//3,5928486b/rbd_data.10a76b8b4567.0000000000001f53/head//3,d94b486b/rbd_data.109a6b8b4567.000000000000101c/head//3,f6db486b/rbd_data.10aa6b8b4567.0000000000002bc7/head//3,550e486b/rbd_data.10ac6b8b4567.000000000000127c/head//3,9f0e486b/rbd_data.10aa6b8b4567.000000000000370c/head//3,14bf486b/rbd_data.10a16b8b4567.0000000000001949/head//3,ad11586b/rbd_data.10a56b8b4567.0000000000001e55/head//3,f076586b/rbd_data.10a66b8b4567.0000000000000698/head//3,1087586b/rbd_data.10a76b8b4567.0000000000000f8e/head//3,877b586b/rbd_data.10a76b8b4567.0000000000002e95/head//3,58ac586b/rbd_data.10b06b8b4567.000000000000377f/head//3,aad7686b/rbd_data.10a16b8b4567.00000000000015f7/head//3,a488686b/rbd_data.10906b8b4567.00000000000016f4/head//3,5589686b/rbd_data.10af6b8b4567.00000000000029c4/head//3,c3cb686b/rbd_data.10a16b8b4567.0000000000001796/head//3,73e0786b/rbd_data.109c6b8b4567.0000000000002ee4/head//3,1201786b/rbd_data.10976b8b4567.0000000000002b50/head//3,7212786b/rbd_data.10a16b8b4567.00000000000024ae/head//3,f132786b/rbd_data.10a06b8b4567.000000000000371a/head//3,1a72786b/rbd_data.10a96b8b4567.000000000000066e/head//3,1895786b/rbd_data.109c6b8b4567.0000000000002a88/head//3,e776786b/rbd_data.10a36b8b4567.0000000000000ade/head//3,4ed6786b/rbd_data.109b6b8b4567.0000000000002220/head//3,c649786b/rbd_data.10a46b8b4567.0000000000002cd2/head//3,12ca786b/rbd_data.10ae6b8b4567.0000000000001ae2/head//3,237b786b/rbd_data.10a66b8b4567.00000000000007e1/head//3,1ddb786b/rbd_data.10a16b8b4567.00000000000005d2/head//3,4fdd786b/rbd_data.10a86b8b4567.00000000000034b2/head//3,ad1f786b/rbd_data.10a26b8b4567.0000000000000e64/head//3]
2014-06-12 14:51:55.089100 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:55.089116 7f9d60bd2700 10 stat 3.6b_head/e07b086b/rbd_data.10b26b8b4567.0000000000003b18/head//3
2014-06-12 14:51:55.089232 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid e07b086b/rbd_data.10b26b8b4567.0000000000003b18/head//3
2014-06-12 14:51:55.118957 7f9d60bd2700 10 getattrs 3.6b_head/e07b086b/rbd_data.10b26b8b4567.0000000000003b18/head//3 = 0
2014-06-12 14:51:55.118966 7f9d60bd2700 10 stat 3.6b_head/5710186b/rbd_data.10ad6b8b4567.000000000000247e/head//3
2014-06-12 14:51:55.119073 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5710186b/rbd_data.10ad6b8b4567.000000000000247e/head//3
2014-06-12 14:51:55.122084 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:55.133690 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:55.143081 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:55.152501 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:55.153173 7f9d60bd2700 10 getattrs 3.6b_head/5710186b/rbd_data.10ad6b8b4567.000000000000247e/head//3 = 0
2014-06-12 14:51:55.153183 7f9d60bd2700 10 stat 3.6b_head/fae1186b/rbd_data.10946b8b4567.0000000000002a53/head//3
2014-06-12 14:51:55.153307 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid fae1186b/rbd_data.10946b8b4567.0000000000002a53/head//3
2014-06-12 14:51:55.170999 7f9d60bd2700 10 getattrs 3.6b_head/fae1186b/rbd_data.10946b8b4567.0000000000002a53/head//3 = 0
2014-06-12 14:51:55.171008 7f9d60bd2700 10 stat 3.6b_head/2f32186b/rbd_data.10a86b8b4567.00000000000016a3/head//3
2014-06-12 14:51:55.171113 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 2f32186b/rbd_data.10a86b8b4567.00000000000016a3/head//3
2014-06-12 14:51:55.180988 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:55.193586 7f9d60bd2700 10 getattrs 3.6b_head/2f32186b/rbd_data.10a86b8b4567.00000000000016a3/head//3 = 0
2014-06-12 14:51:55.193597 7f9d60bd2700 10 stat 3.6b_head/8f23186b/rbd_data.10b16b8b4567.0000000000003424/head//3
2014-06-12 14:51:55.193715 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 8f23186b/rbd_data.10b16b8b4567.0000000000003424/head//3
2014-06-12 14:51:55.218086 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:55.230679 7f9d60bd2700 10 getattrs 3.6b_head/8f23186b/rbd_data.10b16b8b4567.0000000000003424/head//3 = 0
2014-06-12 14:51:55.230686 7f9d60bd2700 10 stat 3.6b_head/ad04186b/rbd_data.10996b8b4567.00000000000022e5/head//3
2014-06-12 14:51:55.230779 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ad04186b/rbd_data.10996b8b4567.00000000000022e5/head//3
2014-06-12 14:51:55.253858 7f9d60bd2700 10 getattrs 3.6b_head/ad04186b/rbd_data.10996b8b4567.00000000000022e5/head//3 = 0
2014-06-12 14:51:55.253875 7f9d60bd2700 10 stat 3.6b_head/35e4186b/rbd_data.109a6b8b4567.000000000000142e/head//3
2014-06-12 14:51:55.253981 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 35e4186b/rbd_data.109a6b8b4567.000000000000142e/head//3
2014-06-12 14:51:55.277756 7f9d60bd2700 10 getattrs 3.6b_head/35e4186b/rbd_data.109a6b8b4567.000000000000142e/head//3 = 0
2014-06-12 14:51:55.277765 7f9d60bd2700 10 stat 3.6b_head/6c17186b/rbd_data.10ac6b8b4567.00000000000035b8/head//3
2014-06-12 14:51:55.277857 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6c17186b/rbd_data.10ac6b8b4567.00000000000035b8/head//3
2014-06-12 14:51:55.280518 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:55.280539 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:51:55.280544 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:51:55.280548 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:51:55.280552 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:51:55.280597 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:55.280602 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:55.280603 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:55.280652 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:55.281460 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:55.299143 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:55.312967 7f9d60bd2700 10 getattrs 3.6b_head/6c17186b/rbd_data.10ac6b8b4567.00000000000035b8/head//3 = 0
2014-06-12 14:51:55.312977 7f9d60bd2700 10 stat 3.6b_head/d957186b/rbd_data.10a26b8b4567.00000000000021e7/head//3
2014-06-12 14:51:55.313083 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid d957186b/rbd_data.10a26b8b4567.00000000000021e7/head//3
2014-06-12 14:51:55.316219 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:55.316226 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:55.316230 7f9d74bfa700 20 osd.2 173 _dispatch 0x3175c700 pg_stats_ack(1 pgs tid 4874) v1
2014-06-12 14:51:55.316235 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:51:55.316245 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:55.316249 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:55.332138 7f9d60bd2700 10 getattrs 3.6b_head/d957186b/rbd_data.10a26b8b4567.00000000000021e7/head//3 = 0
2014-06-12 14:51:55.332145 7f9d60bd2700 10 stat 3.6b_head/7f09186b/rbd_data.10b06b8b4567.0000000000003923/head//3
2014-06-12 14:51:55.332242 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7f09186b/rbd_data.10b06b8b4567.0000000000003923/head//3
2014-06-12 14:51:55.342288 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:55.357452 7f9d60bd2700 10 getattrs 3.6b_head/7f09186b/rbd_data.10b06b8b4567.0000000000003923/head//3 = 0
2014-06-12 14:51:55.357467 7f9d60bd2700 10 stat 3.6b_head/af3c186b/rbd_data.10b26b8b4567.0000000000001f3d/head//3
2014-06-12 14:51:55.357588 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid af3c186b/rbd_data.10b26b8b4567.0000000000001f3d/head//3
2014-06-12 14:51:55.375627 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:55.387640 7f9d60bd2700 10 getattrs 3.6b_head/af3c186b/rbd_data.10b26b8b4567.0000000000001f3d/head//3 = 0
2014-06-12 14:51:55.387651 7f9d60bd2700 10 stat 3.6b_head/c29d186b/rbd_data.10a36b8b4567.0000000000002e8c/head//3
2014-06-12 14:51:55.387778 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c29d186b/rbd_data.10a36b8b4567.0000000000002e8c/head//3
2014-06-12 14:51:55.420193 7f9d60bd2700 10 getattrs 3.6b_head/c29d186b/rbd_data.10a36b8b4567.0000000000002e8c/head//3 = 0
2014-06-12 14:51:55.420206 7f9d60bd2700 10 stat 3.6b_head/baaf186b/rbd_data.10ab6b8b4567.000000000000316b/head//3
2014-06-12 14:51:55.420340 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid baaf186b/rbd_data.10ab6b8b4567.000000000000316b/head//3
2014-06-12 14:51:55.455237 7f9d60bd2700 10 getattrs 3.6b_head/baaf186b/rbd_data.10ab6b8b4567.000000000000316b/head//3 = 0
2014-06-12 14:51:55.455246 7f9d60bd2700 10 stat 3.6b_head/b203286b/rbd_data.10a46b8b4567.0000000000000c91/head//3
2014-06-12 14:51:55.455341 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid b203286b/rbd_data.10a46b8b4567.0000000000000c91/head//3
2014-06-12 14:51:55.484270 7f9d60bd2700 10 getattrs 3.6b_head/b203286b/rbd_data.10a46b8b4567.0000000000000c91/head//3 = 0
2014-06-12 14:51:55.484279 7f9d60bd2700 10 stat 3.6b_head/dd87286b/rbd_data.108f6b8b4567.00000000000013dd/head//3
2014-06-12 14:51:55.484425 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid dd87286b/rbd_data.108f6b8b4567.00000000000013dd/head//3
2014-06-12 14:51:55.484578 7f9d60bd2700 10 getattrs 3.6b_head/dd87286b/rbd_data.108f6b8b4567.00000000000013dd/head//3 = 0
2014-06-12 14:51:55.484589 7f9d60bd2700 10 stat 3.6b_head/e7c8286b/rbd_data.10b46b8b4567.0000000000002aa3/head//3
2014-06-12 14:51:55.484752 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid e7c8286b/rbd_data.10b46b8b4567.0000000000002aa3/head//3
2014-06-12 14:51:55.491636 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:55.491683 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:55.514090 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:55.517802 7f9d60bd2700 10 getattrs 3.6b_head/e7c8286b/rbd_data.10b46b8b4567.0000000000002aa3/head//3 = 0
2014-06-12 14:51:55.517822 7f9d60bd2700 10 stat 3.6b_head/f4ac286b/rbd_data.108f6b8b4567.00000000000016d8/head//3
2014-06-12 14:51:55.518018 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid f4ac286b/rbd_data.108f6b8b4567.00000000000016d8/head//3
2014-06-12 14:51:55.519127 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:55.535638 7f9d60bd2700 10 getattrs 3.6b_head/f4ac286b/rbd_data.108f6b8b4567.00000000000016d8/head//3 = 0
2014-06-12 14:51:55.535649 7f9d60bd2700 10 stat 3.6b_head/226d286b/rbd_data.10966b8b4567.00000000000020b0/head//3
2014-06-12 14:51:55.535767 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 226d286b/rbd_data.10966b8b4567.00000000000020b0/head//3
2014-06-12 14:51:55.564658 7f9d60bd2700 10 getattrs 3.6b_head/226d286b/rbd_data.10966b8b4567.00000000000020b0/head//3 = 0
2014-06-12 14:51:55.564672 7f9d60bd2700 10 stat 3.6b_head/492e286b/rbd_data.10a16b8b4567.0000000000001476/head//3
2014-06-12 14:51:55.564796 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 492e286b/rbd_data.10a16b8b4567.0000000000001476/head//3
2014-06-12 14:51:55.589399 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:55.589486 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:55.600308 7f9d60bd2700 10 getattrs 3.6b_head/492e286b/rbd_data.10a16b8b4567.0000000000001476/head//3 = 0
2014-06-12 14:51:55.600316 7f9d60bd2700 10 stat 3.6b_head/9682386b/rbd_data.10a46b8b4567.00000000000003dc/head//3
2014-06-12 14:51:55.600415 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9682386b/rbd_data.10a46b8b4567.00000000000003dc/head//3
2014-06-12 14:51:55.636728 7f9d60bd2700 10 getattrs 3.6b_head/9682386b/rbd_data.10a46b8b4567.00000000000003dc/head//3 = 0
2014-06-12 14:51:55.636739 7f9d60bd2700 10 stat 3.6b_head/ca34386b/rbd_data.10916b8b4567.00000000000013c7/head//3
2014-06-12 14:51:55.636827 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ca34386b/rbd_data.10916b8b4567.00000000000013c7/head//3
2014-06-12 14:51:55.641212 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:55.650909 7f9d5501e700  5 -- op tracker -- , seq: 748614, time: 2014-06-12 14:51:55.650826, event: header_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:55.650919 7f9d5501e700  5 -- op tracker -- , seq: 748614, time: 2014-06-12 14:51:55.650828, event: throttled, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:55.650924 7f9d5501e700  5 -- op tracker -- , seq: 748614, time: 2014-06-12 14:51:55.650878, event: all_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:55.650930 7f9d5501e700  5 -- op tracker -- , seq: 748614, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:55.650934 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:55.650939 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:55.650945 7f9d5501e700 15 osd.2 173 enqueue_op 0x3850ec00 prio 127 cost 10717 latency 0.000119 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:55.651017 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:55.672040 7f9d60bd2700 10 getattrs 3.6b_head/ca34386b/rbd_data.10916b8b4567.00000000000013c7/head//3 = 0
2014-06-12 14:51:55.672050 7f9d60bd2700 10 stat 3.6b_head/a5c4386b/rbd_data.10a76b8b4567.00000000000001c4/head//3
2014-06-12 14:51:55.672170 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a5c4386b/rbd_data.10a76b8b4567.00000000000001c4/head//3
2014-06-12 14:51:55.724230 7f9d60bd2700 10 getattrs 3.6b_head/a5c4386b/rbd_data.10a76b8b4567.00000000000001c4/head//3 = 0
2014-06-12 14:51:55.724253 7f9d60bd2700 10 stat 3.6b_head/2bd4386b/rbd_data.10b46b8b4567.0000000000001357/head//3
2014-06-12 14:51:55.724389 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 2bd4386b/rbd_data.10b46b8b4567.0000000000001357/head//3
2014-06-12 14:51:55.728999 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:55.760311 7f9d60bd2700 10 getattrs 3.6b_head/2bd4386b/rbd_data.10b46b8b4567.0000000000001357/head//3 = 0
2014-06-12 14:51:55.760324 7f9d60bd2700 10 stat 3.6b_head/75e5386b/rbd_data.108d6b8b4567.0000000000003316/head//3
2014-06-12 14:51:55.760464 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 75e5386b/rbd_data.108d6b8b4567.0000000000003316/head//3
2014-06-12 14:51:55.774344 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:55.774529 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:55.784203 7f9d60bd2700 10 getattrs 3.6b_head/75e5386b/rbd_data.108d6b8b4567.0000000000003316/head//3 = 0
2014-06-12 14:51:55.784210 7f9d60bd2700 10 stat 3.6b_head/9707386b/rbd_data.10986b8b4567.0000000000001dec/head//3
2014-06-12 14:51:55.784291 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9707386b/rbd_data.10986b8b4567.0000000000001dec/head//3
2014-06-12 14:51:55.784585 7f9d60bd2700 10 getattrs 3.6b_head/9707386b/rbd_data.10986b8b4567.0000000000001dec/head//3 = 0
2014-06-12 14:51:55.784592 7f9d60bd2700 10 collection_getattrs 3.6b_head
2014-06-12 14:51:55.784640 7f9d60bd2700 10 collection_getattrs 3.6b_head = 0
2014-06-12 14:51:55.784645 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:55.784679 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:55.784685 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:55.784693 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:55.784696 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:55.784731 7f9d61bd4700 10 osd.2 173 dequeue_op 0x3850ec00 prio 127 cost 10717 latency 0.133904 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]
2014-06-12 14:51:55.784781 7f9d61bd4700  5 -- op tracker -- , seq: 748614, time: 2014-06-12 14:51:55.784774, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:55.784812 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] handle_message: 0x3850ec00
2014-06-12 14:51:55.784829 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:55.784849 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:55.784868 7f9d61bd4700  5 -- op tracker -- , seq: 748614, time: 2014-06-12 14:51:55.784863, event: started, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:55.784897 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  got 29 scrub map
2014-06-12 14:51:55.785003 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] map version is 173'2887
2014-06-12 14:51:55.785025 7f9d61bd4700 10 osd.2 173 dequeue_op 0x3850ec00 finish
2014-06-12 14:51:55.785029 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:55.785036 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:55.785037 7f9d61bd4700  5 -- op tracker -- , seq: 748614, time: 2014-06-12 14:51:55.785031, event: done, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:55.785051 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:55.785062 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:55.785080 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:55.785086 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:55.785102 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:55.785123 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:55.785136 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub replica 29 has 25 items
2014-06-12 14:51:55.785153 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e07b086b/rbd_data.10b26b8b4567.0000000000003b18/head//3, auth == maps.end()
2014-06-12 14:51:55.785163 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj e07b086b/rbd_data.10b26b8b4567.0000000000003b18/head//3
2014-06-12 14:51:55.785173 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5710186b/rbd_data.10ad6b8b4567.000000000000247e/head//3, auth == maps.end()
2014-06-12 14:51:55.785180 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5710186b/rbd_data.10ad6b8b4567.000000000000247e/head//3
2014-06-12 14:51:55.785188 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fae1186b/rbd_data.10946b8b4567.0000000000002a53/head//3, auth == maps.end()
2014-06-12 14:51:55.785197 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj fae1186b/rbd_data.10946b8b4567.0000000000002a53/head//3
2014-06-12 14:51:55.785206 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2f32186b/rbd_data.10a86b8b4567.00000000000016a3/head//3, auth == maps.end()
2014-06-12 14:51:55.785214 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 2f32186b/rbd_data.10a86b8b4567.00000000000016a3/head//3
2014-06-12 14:51:55.785222 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8f23186b/rbd_data.10b16b8b4567.0000000000003424/head//3, auth == maps.end()
2014-06-12 14:51:55.785230 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 8f23186b/rbd_data.10b16b8b4567.0000000000003424/head//3
2014-06-12 14:51:55.785239 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ad04186b/rbd_data.10996b8b4567.00000000000022e5/head//3, auth == maps.end()
2014-06-12 14:51:55.785247 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ad04186b/rbd_data.10996b8b4567.00000000000022e5/head//3
2014-06-12 14:51:55.785256 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 35e4186b/rbd_data.109a6b8b4567.000000000000142e/head//3, auth == maps.end()
2014-06-12 14:51:55.785264 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 35e4186b/rbd_data.109a6b8b4567.000000000000142e/head//3
2014-06-12 14:51:55.785273 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6c17186b/rbd_data.10ac6b8b4567.00000000000035b8/head//3, auth == maps.end()
2014-06-12 14:51:55.785281 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6c17186b/rbd_data.10ac6b8b4567.00000000000035b8/head//3
2014-06-12 14:51:55.785290 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d957186b/rbd_data.10a26b8b4567.00000000000021e7/head//3, auth == maps.end()
2014-06-12 14:51:55.785298 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj d957186b/rbd_data.10a26b8b4567.00000000000021e7/head//3
2014-06-12 14:51:55.785306 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7f09186b/rbd_data.10b06b8b4567.0000000000003923/head//3, auth == maps.end()
2014-06-12 14:51:55.785314 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7f09186b/rbd_data.10b06b8b4567.0000000000003923/head//3
2014-06-12 14:51:55.785322 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj af3c186b/rbd_data.10b26b8b4567.0000000000001f3d/head//3, auth == maps.end()
2014-06-12 14:51:55.785329 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj af3c186b/rbd_data.10b26b8b4567.0000000000001f3d/head//3
2014-06-12 14:51:55.785338 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c29d186b/rbd_data.10a36b8b4567.0000000000002e8c/head//3, auth == maps.end()
2014-06-12 14:51:55.785346 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c29d186b/rbd_data.10a36b8b4567.0000000000002e8c/head//3
2014-06-12 14:51:55.785355 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj baaf186b/rbd_data.10ab6b8b4567.000000000000316b/head//3, auth == maps.end()
2014-06-12 14:51:55.785363 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj baaf186b/rbd_data.10ab6b8b4567.000000000000316b/head//3
2014-06-12 14:51:55.785372 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b203286b/rbd_data.10a46b8b4567.0000000000000c91/head//3, auth == maps.end()
2014-06-12 14:51:55.785380 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj b203286b/rbd_data.10a46b8b4567.0000000000000c91/head//3
2014-06-12 14:51:55.785389 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dd87286b/rbd_data.108f6b8b4567.00000000000013dd/head//3, auth == maps.end()
2014-06-12 14:51:55.785397 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj dd87286b/rbd_data.108f6b8b4567.00000000000013dd/head//3
2014-06-12 14:51:55.785406 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e7c8286b/rbd_data.10b46b8b4567.0000000000002aa3/head//3, auth == maps.end()
2014-06-12 14:51:55.785414 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj e7c8286b/rbd_data.10b46b8b4567.0000000000002aa3/head//3
2014-06-12 14:51:55.785422 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f4ac286b/rbd_data.108f6b8b4567.00000000000016d8/head//3, auth == maps.end()
2014-06-12 14:51:55.785430 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj f4ac286b/rbd_data.108f6b8b4567.00000000000016d8/head//3
2014-06-12 14:51:55.785439 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 226d286b/rbd_data.10966b8b4567.00000000000020b0/head//3, auth == maps.end()
2014-06-12 14:51:55.785447 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 226d286b/rbd_data.10966b8b4567.00000000000020b0/head//3
2014-06-12 14:51:55.785456 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 492e286b/rbd_data.10a16b8b4567.0000000000001476/head//3, auth == maps.end()
2014-06-12 14:51:55.785464 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 492e286b/rbd_data.10a16b8b4567.0000000000001476/head//3
2014-06-12 14:51:55.785473 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9682386b/rbd_data.10a46b8b4567.00000000000003dc/head//3, auth == maps.end()
2014-06-12 14:51:55.785481 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9682386b/rbd_data.10a46b8b4567.00000000000003dc/head//3
2014-06-12 14:51:55.785490 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ca34386b/rbd_data.10916b8b4567.00000000000013c7/head//3, auth == maps.end()
2014-06-12 14:51:55.785498 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ca34386b/rbd_data.10916b8b4567.00000000000013c7/head//3
2014-06-12 14:51:55.785507 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a5c4386b/rbd_data.10a76b8b4567.00000000000001c4/head//3, auth == maps.end()
2014-06-12 14:51:55.785515 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a5c4386b/rbd_data.10a76b8b4567.00000000000001c4/head//3
2014-06-12 14:51:55.785523 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2bd4386b/rbd_data.10b46b8b4567.0000000000001357/head//3, auth == maps.end()
2014-06-12 14:51:55.785531 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 2bd4386b/rbd_data.10b46b8b4567.0000000000001357/head//3
2014-06-12 14:51:55.785539 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 75e5386b/rbd_data.108d6b8b4567.0000000000003316/head//3, auth == maps.end()
2014-06-12 14:51:55.785547 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 75e5386b/rbd_data.108d6b8b4567.0000000000003316/head//3
2014-06-12 14:51:55.785555 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9707386b/rbd_data.10986b8b4567.0000000000001dec/head//3, auth == maps.end()
2014-06-12 14:51:55.785564 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9707386b/rbd_data.10986b8b4567.0000000000001dec/head//3
2014-06-12 14:51:55.785574 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] 
2014-06-12 14:51:55.785580 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub
2014-06-12 14:51:55.785592 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9707386b/rbd_data.10986b8b4567.0000000000001dec/head//3 9707386b/rbd_data.10986b8b4567.0000000000001dec/head//3(173'1239 client.4289.0:62230 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1239)
2014-06-12 14:51:55.785604 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  75e5386b/rbd_data.108d6b8b4567.0000000000003316/head//3 75e5386b/rbd_data.108d6b8b4567.0000000000003316/head//3(173'2001 client.4278.0:102627 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2001)
2014-06-12 14:51:55.785619 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  2bd4386b/rbd_data.10b46b8b4567.0000000000001357/head//3 2bd4386b/rbd_data.10b46b8b4567.0000000000001357/head//3(173'862 client.4317.0:44229 wrlock_by=unknown.0.0:0 dirty s 4194304 uv862)
2014-06-12 14:51:55.785638 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a5c4386b/rbd_data.10a76b8b4567.00000000000001c4/head//3 a5c4386b/rbd_data.10a76b8b4567.00000000000001c4/head//3(173'2436 client.4304.0:119514 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2436)
2014-06-12 14:51:55.785649 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ca34386b/rbd_data.10916b8b4567.00000000000013c7/head//3 ca34386b/rbd_data.10916b8b4567.00000000000013c7/head//3(173'930 client.4282.0:46206 wrlock_by=unknown.0.0:0 dirty s 4194304 uv930)
2014-06-12 14:51:55.785659 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9682386b/rbd_data.10a46b8b4567.00000000000003dc/head//3 9682386b/rbd_data.10a46b8b4567.00000000000003dc/head//3(173'2486 client.4301.0:124316 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2486)
2014-06-12 14:51:55.785678 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  492e286b/rbd_data.10a16b8b4567.0000000000001476/head//3 492e286b/rbd_data.10a16b8b4567.0000000000001476/head//3(173'914 client.4298.0:47399 wrlock_by=unknown.0.0:0 dirty s 4194304 uv914)
2014-06-12 14:51:55.785688 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  226d286b/rbd_data.10966b8b4567.00000000000020b0/head//3 226d286b/rbd_data.10966b8b4567.00000000000020b0/head//3(173'1301 client.4287.0:65898 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1301)
2014-06-12 14:51:55.785698 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  f4ac286b/rbd_data.108f6b8b4567.00000000000016d8/head//3 f4ac286b/rbd_data.108f6b8b4567.00000000000016d8/head//3(173'1021 client.4280.0:50275 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1021)
2014-06-12 14:51:55.785709 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  e7c8286b/rbd_data.10b46b8b4567.0000000000002aa3/head//3 e7c8286b/rbd_data.10b46b8b4567.0000000000002aa3/head//3(173'1878 client.4317.0:86404 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1878)
2014-06-12 14:51:55.785719 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  dd87286b/rbd_data.108f6b8b4567.00000000000013dd/head//3 dd87286b/rbd_data.108f6b8b4567.00000000000013dd/head//3(173'901 client.4280.0:44763 wrlock_by=unknown.0.0:0 dirty s 4194304 uv901)
2014-06-12 14:51:55.785729 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  b203286b/rbd_data.10a46b8b4567.0000000000000c91/head//3 b203286b/rbd_data.10a46b8b4567.0000000000000c91/head//3(173'2741 client.4301.0:138748 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2741)
2014-06-12 14:51:55.785739 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  baaf186b/rbd_data.10ab6b8b4567.000000000000316b/head//3 baaf186b/rbd_data.10ab6b8b4567.000000000000316b/head//3(173'1978 client.4308.0:99126 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1978)
2014-06-12 14:51:55.785749 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c29d186b/rbd_data.10a36b8b4567.0000000000002e8c/head//3 c29d186b/rbd_data.10a36b8b4567.0000000000002e8c/head//3(173'1805 client.4300.0:91279 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1805)
2014-06-12 14:51:55.785758 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  af3c186b/rbd_data.10b26b8b4567.0000000000001f3d/head//3 af3c186b/rbd_data.10b26b8b4567.0000000000001f3d/head//3(173'1206 client.4315.0:65964 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1206)
2014-06-12 14:51:55.785768 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7f09186b/rbd_data.10b06b8b4567.0000000000003923/head//3 7f09186b/rbd_data.10b06b8b4567.0000000000003923/head//3(173'2278 client.4313.0:114680 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2278)
2014-06-12 14:51:55.785778 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  d957186b/rbd_data.10a26b8b4567.00000000000021e7/head//3 d957186b/rbd_data.10a26b8b4567.00000000000021e7/head//3(173'1548 client.4299.0:72169 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1548)
2014-06-12 14:51:55.785788 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6c17186b/rbd_data.10ac6b8b4567.00000000000035b8/head//3 6c17186b/rbd_data.10ac6b8b4567.00000000000035b8/head//3(173'2205 client.4309.0:105802 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2205)
2014-06-12 14:51:55.785798 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  35e4186b/rbd_data.109a6b8b4567.000000000000142e/head//3 35e4186b/rbd_data.109a6b8b4567.000000000000142e/head//3(173'892 client.4291.0:45547 wrlock_by=unknown.0.0:0 dirty s 4194304 uv892)
2014-06-12 14:51:55.785809 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ad04186b/rbd_data.10996b8b4567.00000000000022e5/head//3 ad04186b/rbd_data.10996b8b4567.00000000000022e5/head//3(173'1371 client.4290.0:70886 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1371)
2014-06-12 14:51:55.785819 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  8f23186b/rbd_data.10b16b8b4567.0000000000003424/head//3 8f23186b/rbd_data.10b16b8b4567.0000000000003424/head//3(173'2172 client.4314.0:104901 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2172)
2014-06-12 14:51:55.785829 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  2f32186b/rbd_data.10a86b8b4567.00000000000016a3/head//3 2f32186b/rbd_data.10a86b8b4567.00000000000016a3/head//3(173'951 client.4305.0:49781 wrlock_by=unknown.0.0:0 dirty s 4194304 uv951)
2014-06-12 14:51:55.785839 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  fae1186b/rbd_data.10946b8b4567.0000000000002a53/head//3 fae1186b/rbd_data.10946b8b4567.0000000000002a53/head//3(173'1680 client.4285.0:84808 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1680)
2014-06-12 14:51:55.785848 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5710186b/rbd_data.10ad6b8b4567.000000000000247e/head//3 5710186b/rbd_data.10ad6b8b4567.000000000000247e/head//3(173'1527 client.4310.0:74675 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1527)
2014-06-12 14:51:55.785858 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  e07b086b/rbd_data.10b26b8b4567.0000000000003b18/head//3 e07b086b/rbd_data.10b26b8b4567.0000000000003b18/head//3(173'2242 client.4315.0:116485 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2242)
2014-06-12 14:51:55.785866 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:55.785875 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:55.785882 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:55.785884 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:55.785885 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:55.785915 7f9d60bd2700 10 collection_list_partial 3.6b_head start:5217386b//0//-1 is_max:0
2014-06-12 14:51:55.786093 7f9d60bd2700 20 objects: [2058386b/rbd_data.10ac6b8b4567.0000000000002160/head//3,cdb8386b/rbd_data.10b36b8b4567.0000000000001dd8/head//3,e2ad386b/rbd_data.109c6b8b4567.000000000000287d/head//3,38bd386b/rbd_data.10aa6b8b4567.0000000000002b5e/head//3,1d8f386b/rbd_data.108e6b8b4567.0000000000003a1f/head//3,5191486b/rbd_data.10b06b8b4567.00000000000029aa/head//3,bbd1486b/rbd_data.108e6b8b4567.0000000000002410/head//3,1d13486b/rbd_data.10b26b8b4567.00000000000003c3/head//3,76d5486b/rbd_data.109f6b8b4567.0000000000003445/head//3,5928486b/rbd_data.10a76b8b4567.0000000000001f53/head//3,d94b486b/rbd_data.109a6b8b4567.000000000000101c/head//3,f6db486b/rbd_data.10aa6b8b4567.0000000000002bc7/head//3,550e486b/rbd_data.10ac6b8b4567.000000000000127c/head//3,9f0e486b/rbd_data.10aa6b8b4567.000000000000370c/head//3,14bf486b/rbd_data.10a16b8b4567.0000000000001949/head//3,ad11586b/rbd_data.10a56b8b4567.0000000000001e55/head//3,f076586b/rbd_data.10a66b8b4567.0000000000000698/head//3,1087586b/rbd_data.10a76b8b4567.0000000000000f8e/head//3,877b586b/rbd_data.10a76b8b4567.0000000000002e95/head//3,58ac586b/rbd_data.10b06b8b4567.000000000000377f/head//3,aad7686b/rbd_data.10a16b8b4567.00000000000015f7/head//3,a488686b/rbd_data.10906b8b4567.00000000000016f4/head//3,5589686b/rbd_data.10af6b8b4567.00000000000029c4/head//3,c3cb686b/rbd_data.10a16b8b4567.0000000000001796/head//3,73e0786b/rbd_data.109c6b8b4567.0000000000002ee4/head//3]
2014-06-12 14:51:55.786561 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  requesting scrubmap from osd.29
2014-06-12 14:51:55.786583 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:55.786596 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:55.786603 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:55.786615 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:55.786620 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:55.786625 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_map_chunk [5217386b//0//-1,1201786b//0//-1)
2014-06-12 14:51:55.786631 7f9d60bd2700 10 collection_list_partial 3.6b_head start:5217386b//0//-1 is_max:0
2014-06-12 14:51:55.787039 7f9d60bd2700 20 objects: [2058386b/rbd_data.10ac6b8b4567.0000000000002160/head//3,cdb8386b/rbd_data.10b36b8b4567.0000000000001dd8/head//3,e2ad386b/rbd_data.109c6b8b4567.000000000000287d/head//3,38bd386b/rbd_data.10aa6b8b4567.0000000000002b5e/head//3,1d8f386b/rbd_data.108e6b8b4567.0000000000003a1f/head//3,5191486b/rbd_data.10b06b8b4567.00000000000029aa/head//3,bbd1486b/rbd_data.108e6b8b4567.0000000000002410/head//3,1d13486b/rbd_data.10b26b8b4567.00000000000003c3/head//3,76d5486b/rbd_data.109f6b8b4567.0000000000003445/head//3,5928486b/rbd_data.10a76b8b4567.0000000000001f53/head//3,d94b486b/rbd_data.109a6b8b4567.000000000000101c/head//3,f6db486b/rbd_data.10aa6b8b4567.0000000000002bc7/head//3,550e486b/rbd_data.10ac6b8b4567.000000000000127c/head//3,9f0e486b/rbd_data.10aa6b8b4567.000000000000370c/head//3,14bf486b/rbd_data.10a16b8b4567.0000000000001949/head//3,ad11586b/rbd_data.10a56b8b4567.0000000000001e55/head//3,f076586b/rbd_data.10a66b8b4567.0000000000000698/head//3,1087586b/rbd_data.10a76b8b4567.0000000000000f8e/head//3,877b586b/rbd_data.10a76b8b4567.0000000000002e95/head//3,58ac586b/rbd_data.10b06b8b4567.000000000000377f/head//3,aad7686b/rbd_data.10a16b8b4567.00000000000015f7/head//3,a488686b/rbd_data.10906b8b4567.00000000000016f4/head//3,5589686b/rbd_data.10af6b8b4567.00000000000029c4/head//3,c3cb686b/rbd_data.10a16b8b4567.0000000000001796/head//3,73e0786b/rbd_data.109c6b8b4567.0000000000002ee4/head//3,1201786b/rbd_data.10976b8b4567.0000000000002b50/head//3,7212786b/rbd_data.10a16b8b4567.00000000000024ae/head//3,f132786b/rbd_data.10a06b8b4567.000000000000371a/head//3,1a72786b/rbd_data.10a96b8b4567.000000000000066e/head//3,1895786b/rbd_data.109c6b8b4567.0000000000002a88/head//3,e776786b/rbd_data.10a36b8b4567.0000000000000ade/head//3,4ed6786b/rbd_data.109b6b8b4567.0000000000002220/head//3,c649786b/rbd_data.10a46b8b4567.0000000000002cd2/head//3,12ca786b/rbd_data.10ae6b8b4567.0000000000001ae2/head//3,237b786b/rbd_data.10a66b8b4567.00000000000007e1/head//3,1ddb786b/rbd_data.10a16b8b4567.00000000000005d2/head//3,4fdd786b/rbd_data.10a86b8b4567.00000000000034b2/head//3,ad1f786b/rbd_data.10a26b8b4567.0000000000000e64/head//3,124f786b/rbd_data.10966b8b4567.000000000000163a/head//3,aebf786b/rbd_data.109f6b8b4567.0000000000001ed0/head//3,e610886b/rbd_data.10a56b8b4567.0000000000000fc0/head//3,dc55886b/rbd_data.10946b8b4567.0000000000000778/head//3,2c86886b/rbd_data.10966b8b4567.00000000000008d2/head//3,778886b/rbd_data.109e6b8b4567.0000000000003aed/head//3,4ec8886b/rbd_data.10af6b8b4567.00000000000021ad/head//3,9269886b/rbd_data.10a06b8b4567.0000000000001a49/head//3,3fbd886b/rbd_data.109b6b8b4567.0000000000000991/head//3,7ffe886b/rbd_data.10b06b8b4567.0000000000002387/head//3,da20986b/rbd_data.10a76b8b4567.0000000000002436/head//3,e2e0986b/rbd_data.10a76b8b4567.000000000000066b/head//3,d3e0986b/rbd_data.109b6b8b4567.0000000000001d62/head//3,1622986b/rbd_data.10a16b8b4567.0000000000001130/head//3,1852986b/rbd_data.109b6b8b4567.0000000000001c96/head//3,1f72986b/rbd_data.109a6b8b4567.0000000000002b71/head//3,9574986b/rbd_data.10a56b8b4567.00000000000012ea/head//3,ba76986b/rbd_data.10a86b8b4567.0000000000002547/head//3,2386986b/rbd_data.10966b8b4567.000000000000246c/head//3,8ae6986b/rbd_data.10b06b8b4567.0000000000003881/head//3,807986b/rbd_data.10ad6b8b4567.0000000000003a00/head//3,71d7986b/rbd_data.10956b8b4567.00000000000001bc/head//3,462a986b/rbd_data.10ad6b8b4567.0000000000000ee5/head//3,a7ba986b/rbd_data.10b46b8b4567.0000000000001a25/head//3,640b986b/rbd_data.109a6b8b4567.0000000000001e70/head//3,694b986b/rbd_data.10ac6b8b4567.00000000000010be/head//3]
2014-06-12 14:51:55.787099 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:55.787112 7f9d60bd2700 10 stat 3.6b_head/2058386b/rbd_data.10ac6b8b4567.0000000000002160/head//3
2014-06-12 14:51:55.787202 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 2058386b/rbd_data.10ac6b8b4567.0000000000002160/head//3
2014-06-12 14:51:55.821983 7f9d60bd2700 10 getattrs 3.6b_head/2058386b/rbd_data.10ac6b8b4567.0000000000002160/head//3 = 0
2014-06-12 14:51:55.821990 7f9d60bd2700 10 stat 3.6b_head/cdb8386b/rbd_data.10b36b8b4567.0000000000001dd8/head//3
2014-06-12 14:51:55.822086 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid cdb8386b/rbd_data.10b36b8b4567.0000000000001dd8/head//3
2014-06-12 14:51:55.851889 7f9d60bd2700 10 getattrs 3.6b_head/cdb8386b/rbd_data.10b36b8b4567.0000000000001dd8/head//3 = 0
2014-06-12 14:51:55.851901 7f9d60bd2700 10 stat 3.6b_head/e2ad386b/rbd_data.109c6b8b4567.000000000000287d/head//3
2014-06-12 14:51:55.851998 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid e2ad386b/rbd_data.109c6b8b4567.000000000000287d/head//3
2014-06-12 14:51:55.868427 7f9d60bd2700 10 getattrs 3.6b_head/e2ad386b/rbd_data.109c6b8b4567.000000000000287d/head//3 = 0
2014-06-12 14:51:55.868436 7f9d60bd2700 10 stat 3.6b_head/38bd386b/rbd_data.10aa6b8b4567.0000000000002b5e/head//3
2014-06-12 14:51:55.868536 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 38bd386b/rbd_data.10aa6b8b4567.0000000000002b5e/head//3
2014-06-12 14:51:55.873193 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:55.884065 7f9d60bd2700 10 getattrs 3.6b_head/38bd386b/rbd_data.10aa6b8b4567.0000000000002b5e/head//3 = 0
2014-06-12 14:51:55.884073 7f9d60bd2700 10 stat 3.6b_head/1d8f386b/rbd_data.108e6b8b4567.0000000000003a1f/head//3
2014-06-12 14:51:55.884173 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1d8f386b/rbd_data.108e6b8b4567.0000000000003a1f/head//3
2014-06-12 14:51:55.901527 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:55.901814 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:55.904716 7f9d60bd2700 10 getattrs 3.6b_head/1d8f386b/rbd_data.108e6b8b4567.0000000000003a1f/head//3 = 0
2014-06-12 14:51:55.904724 7f9d60bd2700 10 stat 3.6b_head/5191486b/rbd_data.10b06b8b4567.00000000000029aa/head//3
2014-06-12 14:51:55.904833 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5191486b/rbd_data.10b06b8b4567.00000000000029aa/head//3
2014-06-12 14:51:55.926996 7f9d60bd2700 10 getattrs 3.6b_head/5191486b/rbd_data.10b06b8b4567.00000000000029aa/head//3 = 0
2014-06-12 14:51:55.927012 7f9d60bd2700 10 stat 3.6b_head/bbd1486b/rbd_data.108e6b8b4567.0000000000002410/head//3
2014-06-12 14:51:55.927114 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid bbd1486b/rbd_data.108e6b8b4567.0000000000002410/head//3
2014-06-12 14:51:55.938295 7f9d60bd2700 10 getattrs 3.6b_head/bbd1486b/rbd_data.108e6b8b4567.0000000000002410/head//3 = 0
2014-06-12 14:51:55.938313 7f9d60bd2700 10 stat 3.6b_head/1d13486b/rbd_data.10b26b8b4567.00000000000003c3/head//3
2014-06-12 14:51:55.938495 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1d13486b/rbd_data.10b26b8b4567.00000000000003c3/head//3
2014-06-12 14:51:55.988180 7f9d60bd2700 10 getattrs 3.6b_head/1d13486b/rbd_data.10b26b8b4567.00000000000003c3/head//3 = 0
2014-06-12 14:51:55.988194 7f9d60bd2700 10 stat 3.6b_head/76d5486b/rbd_data.109f6b8b4567.0000000000003445/head//3
2014-06-12 14:51:55.988300 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 76d5486b/rbd_data.109f6b8b4567.0000000000003445/head//3
2014-06-12 14:51:56.009088 7f9d60bd2700 10 getattrs 3.6b_head/76d5486b/rbd_data.109f6b8b4567.0000000000003445/head//3 = 0
2014-06-12 14:51:56.009109 7f9d60bd2700 10 stat 3.6b_head/5928486b/rbd_data.10a76b8b4567.0000000000001f53/head//3
2014-06-12 14:51:56.009298 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5928486b/rbd_data.10a76b8b4567.0000000000001f53/head//3
2014-06-12 14:51:56.009970 7f9d60bd2700 10 getattrs 3.6b_head/5928486b/rbd_data.10a76b8b4567.0000000000001f53/head//3 = 0
2014-06-12 14:51:56.009984 7f9d60bd2700 10 stat 3.6b_head/d94b486b/rbd_data.109a6b8b4567.000000000000101c/head//3
2014-06-12 14:51:56.010163 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid d94b486b/rbd_data.109a6b8b4567.000000000000101c/head//3
2014-06-12 14:51:56.024915 7f9d60bd2700 10 getattrs 3.6b_head/d94b486b/rbd_data.109a6b8b4567.000000000000101c/head//3 = 0
2014-06-12 14:51:56.024924 7f9d60bd2700 10 stat 3.6b_head/f6db486b/rbd_data.10aa6b8b4567.0000000000002bc7/head//3
2014-06-12 14:51:56.025024 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid f6db486b/rbd_data.10aa6b8b4567.0000000000002bc7/head//3
2014-06-12 14:51:56.040875 7f9d60bd2700 10 getattrs 3.6b_head/f6db486b/rbd_data.10aa6b8b4567.0000000000002bc7/head//3 = 0
2014-06-12 14:51:56.040895 7f9d60bd2700 10 stat 3.6b_head/550e486b/rbd_data.10ac6b8b4567.000000000000127c/head//3
2014-06-12 14:51:56.041076 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 550e486b/rbd_data.10ac6b8b4567.000000000000127c/head//3
2014-06-12 14:51:56.054133 7f9d60bd2700 10 getattrs 3.6b_head/550e486b/rbd_data.10ac6b8b4567.000000000000127c/head//3 = 0
2014-06-12 14:51:56.054144 7f9d60bd2700 10 stat 3.6b_head/9f0e486b/rbd_data.10aa6b8b4567.000000000000370c/head//3
2014-06-12 14:51:56.054255 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9f0e486b/rbd_data.10aa6b8b4567.000000000000370c/head//3
2014-06-12 14:51:56.065301 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:56.067396 7f9d60bd2700 10 getattrs 3.6b_head/9f0e486b/rbd_data.10aa6b8b4567.000000000000370c/head//3 = 0
2014-06-12 14:51:56.067406 7f9d60bd2700 10 stat 3.6b_head/14bf486b/rbd_data.10a16b8b4567.0000000000001949/head//3
2014-06-12 14:51:56.067517 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 14bf486b/rbd_data.10a16b8b4567.0000000000001949/head//3
2014-06-12 14:51:56.098175 7f9d60bd2700 10 getattrs 3.6b_head/14bf486b/rbd_data.10a16b8b4567.0000000000001949/head//3 = 0
2014-06-12 14:51:56.098189 7f9d60bd2700 10 stat 3.6b_head/ad11586b/rbd_data.10a56b8b4567.0000000000001e55/head//3
2014-06-12 14:51:56.098384 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ad11586b/rbd_data.10a56b8b4567.0000000000001e55/head//3
2014-06-12 14:51:56.131134 7f9d60bd2700 10 getattrs 3.6b_head/ad11586b/rbd_data.10a56b8b4567.0000000000001e55/head//3 = 0
2014-06-12 14:51:56.131145 7f9d60bd2700 10 stat 3.6b_head/f076586b/rbd_data.10a66b8b4567.0000000000000698/head//3
2014-06-12 14:51:56.131270 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid f076586b/rbd_data.10a66b8b4567.0000000000000698/head//3
2014-06-12 14:51:56.168238 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:56.198025 7f9d60bd2700 10 getattrs 3.6b_head/f076586b/rbd_data.10a66b8b4567.0000000000000698/head//3 = 0
2014-06-12 14:51:56.198046 7f9d60bd2700 10 stat 3.6b_head/1087586b/rbd_data.10a76b8b4567.0000000000000f8e/head//3
2014-06-12 14:51:56.198196 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1087586b/rbd_data.10a76b8b4567.0000000000000f8e/head//3
2014-06-12 14:51:56.247542 7f9d60bd2700 10 getattrs 3.6b_head/1087586b/rbd_data.10a76b8b4567.0000000000000f8e/head//3 = 0
2014-06-12 14:51:56.247568 7f9d60bd2700 10 stat 3.6b_head/877b586b/rbd_data.10a76b8b4567.0000000000002e95/head//3
2014-06-12 14:51:56.247769 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 877b586b/rbd_data.10a76b8b4567.0000000000002e95/head//3
2014-06-12 14:51:56.275066 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:56.275503 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:56.280679 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:56.280706 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:56.280711 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:56.280713 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:56.280734 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:56.283474 7f9d60bd2700 10 getattrs 3.6b_head/877b586b/rbd_data.10a76b8b4567.0000000000002e95/head//3 = 0
2014-06-12 14:51:56.283481 7f9d60bd2700 10 stat 3.6b_head/58ac586b/rbd_data.10b06b8b4567.000000000000377f/head//3
2014-06-12 14:51:56.283590 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 58ac586b/rbd_data.10b06b8b4567.000000000000377f/head//3
2014-06-12 14:51:56.301700 7f9d5501e700  5 -- op tracker -- , seq: 748615, time: 2014-06-12 14:51:56.301590, event: header_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:56.301709 7f9d5501e700  5 -- op tracker -- , seq: 748615, time: 2014-06-12 14:51:56.301593, event: throttled, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:56.301726 7f9d5501e700  5 -- op tracker -- , seq: 748615, time: 2014-06-12 14:51:56.301658, event: all_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:56.301731 7f9d5501e700  5 -- op tracker -- , seq: 748615, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:56.301735 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:56.301739 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:56.301745 7f9d5501e700 15 osd.2 173 enqueue_op 0x3850ed00 prio 127 cost 10717 latency 0.000154 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:56.301769 7f9d67be0700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:56.312732 7f9d60bd2700 10 getattrs 3.6b_head/58ac586b/rbd_data.10b06b8b4567.000000000000377f/head//3 = 0
2014-06-12 14:51:56.312740 7f9d60bd2700 10 stat 3.6b_head/aad7686b/rbd_data.10a16b8b4567.00000000000015f7/head//3
2014-06-12 14:51:56.312847 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid aad7686b/rbd_data.10a16b8b4567.00000000000015f7/head//3
2014-06-12 14:51:56.327006 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:56.342623 7f9d60bd2700 10 getattrs 3.6b_head/aad7686b/rbd_data.10a16b8b4567.00000000000015f7/head//3 = 0
2014-06-12 14:51:56.342640 7f9d60bd2700 10 stat 3.6b_head/a488686b/rbd_data.10906b8b4567.00000000000016f4/head//3
2014-06-12 14:51:56.342763 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a488686b/rbd_data.10906b8b4567.00000000000016f4/head//3
2014-06-12 14:51:56.367080 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:56.371998 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:56.372273 7f9d60bd2700 10 getattrs 3.6b_head/a488686b/rbd_data.10906b8b4567.00000000000016f4/head//3 = 0
2014-06-12 14:51:56.372280 7f9d60bd2700 10 stat 3.6b_head/5589686b/rbd_data.10af6b8b4567.00000000000029c4/head//3
2014-06-12 14:51:56.372380 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5589686b/rbd_data.10af6b8b4567.00000000000029c4/head//3
2014-06-12 14:51:56.382677 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:56.392162 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:56.393543 7f9d60bd2700 10 getattrs 3.6b_head/5589686b/rbd_data.10af6b8b4567.00000000000029c4/head//3 = 0
2014-06-12 14:51:56.393551 7f9d60bd2700 10 stat 3.6b_head/c3cb686b/rbd_data.10a16b8b4567.0000000000001796/head//3
2014-06-12 14:51:56.393664 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c3cb686b/rbd_data.10a16b8b4567.0000000000001796/head//3
2014-06-12 14:51:56.411623 7f9d60bd2700 10 getattrs 3.6b_head/c3cb686b/rbd_data.10a16b8b4567.0000000000001796/head//3 = 0
2014-06-12 14:51:56.411632 7f9d60bd2700 10 stat 3.6b_head/73e0786b/rbd_data.109c6b8b4567.0000000000002ee4/head//3
2014-06-12 14:51:56.411725 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 73e0786b/rbd_data.109c6b8b4567.0000000000002ee4/head//3
2014-06-12 14:51:56.432531 7f9d60bd2700 10 getattrs 3.6b_head/73e0786b/rbd_data.109c6b8b4567.0000000000002ee4/head//3 = 0
2014-06-12 14:51:56.432543 7f9d60bd2700 10 collection_getattrs 3.6b_head
2014-06-12 14:51:56.432599 7f9d60bd2700 10 collection_getattrs 3.6b_head = 0
2014-06-12 14:51:56.432603 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:56.432632 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:56.432641 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:56.432650 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:56.432653 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:56.432683 7f9d67be0700 10 osd.2 173 dequeue_op 0x3850ed00 prio 127 cost 10717 latency 0.131092 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]
2014-06-12 14:51:56.432734 7f9d67be0700  5 -- op tracker -- , seq: 748615, time: 2014-06-12 14:51:56.432726, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:56.432764 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] handle_message: 0x3850ed00
2014-06-12 14:51:56.432783 7f9d67be0700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:56.432807 7f9d67be0700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:56.432826 7f9d67be0700  5 -- op tracker -- , seq: 748615, time: 2014-06-12 14:51:56.432820, event: started, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:56.432840 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  got 29 scrub map
2014-06-12 14:51:56.432945 7f9d67be0700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] map version is 173'2887
2014-06-12 14:51:56.432965 7f9d67be0700 10 osd.2 173 dequeue_op 0x3850ed00 finish
2014-06-12 14:51:56.432977 7f9d67be0700  5 -- op tracker -- , seq: 748615, time: 2014-06-12 14:51:56.432971, event: done, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:56.433004 7f9d67be0700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:56.433009 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:56.433005 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:56.433018 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:56.433040 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:56.433053 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:56.433094 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:56.433101 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:56.433107 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub replica 29 has 25 items
2014-06-12 14:51:56.433128 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2058386b/rbd_data.10ac6b8b4567.0000000000002160/head//3, auth == maps.end()
2014-06-12 14:51:56.433145 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 2058386b/rbd_data.10ac6b8b4567.0000000000002160/head//3
2014-06-12 14:51:56.433158 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj cdb8386b/rbd_data.10b36b8b4567.0000000000001dd8/head//3, auth == maps.end()
2014-06-12 14:51:56.433167 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj cdb8386b/rbd_data.10b36b8b4567.0000000000001dd8/head//3
2014-06-12 14:51:56.433183 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e2ad386b/rbd_data.109c6b8b4567.000000000000287d/head//3, auth == maps.end()
2014-06-12 14:51:56.433191 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj e2ad386b/rbd_data.109c6b8b4567.000000000000287d/head//3
2014-06-12 14:51:56.433201 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 38bd386b/rbd_data.10aa6b8b4567.0000000000002b5e/head//3, auth == maps.end()
2014-06-12 14:51:56.433209 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 38bd386b/rbd_data.10aa6b8b4567.0000000000002b5e/head//3
2014-06-12 14:51:56.433218 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1d8f386b/rbd_data.108e6b8b4567.0000000000003a1f/head//3, auth == maps.end()
2014-06-12 14:51:56.433227 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1d8f386b/rbd_data.108e6b8b4567.0000000000003a1f/head//3
2014-06-12 14:51:56.433235 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5191486b/rbd_data.10b06b8b4567.00000000000029aa/head//3, auth == maps.end()
2014-06-12 14:51:56.433244 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5191486b/rbd_data.10b06b8b4567.00000000000029aa/head//3
2014-06-12 14:51:56.433253 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bbd1486b/rbd_data.108e6b8b4567.0000000000002410/head//3, auth == maps.end()
2014-06-12 14:51:56.433261 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj bbd1486b/rbd_data.108e6b8b4567.0000000000002410/head//3
2014-06-12 14:51:56.433270 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1d13486b/rbd_data.10b26b8b4567.00000000000003c3/head//3, auth == maps.end()
2014-06-12 14:51:56.433278 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1d13486b/rbd_data.10b26b8b4567.00000000000003c3/head//3
2014-06-12 14:51:56.433287 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 76d5486b/rbd_data.109f6b8b4567.0000000000003445/head//3, auth == maps.end()
2014-06-12 14:51:56.433296 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 76d5486b/rbd_data.109f6b8b4567.0000000000003445/head//3
2014-06-12 14:51:56.433305 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5928486b/rbd_data.10a76b8b4567.0000000000001f53/head//3, auth == maps.end()
2014-06-12 14:51:56.433313 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5928486b/rbd_data.10a76b8b4567.0000000000001f53/head//3
2014-06-12 14:51:56.433325 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d94b486b/rbd_data.109a6b8b4567.000000000000101c/head//3, auth == maps.end()
2014-06-12 14:51:56.433335 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj d94b486b/rbd_data.109a6b8b4567.000000000000101c/head//3
2014-06-12 14:51:56.433344 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f6db486b/rbd_data.10aa6b8b4567.0000000000002bc7/head//3, auth == maps.end()
2014-06-12 14:51:56.433354 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj f6db486b/rbd_data.10aa6b8b4567.0000000000002bc7/head//3
2014-06-12 14:51:56.433364 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 550e486b/rbd_data.10ac6b8b4567.000000000000127c/head//3, auth == maps.end()
2014-06-12 14:51:56.433373 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 550e486b/rbd_data.10ac6b8b4567.000000000000127c/head//3
2014-06-12 14:51:56.433382 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9f0e486b/rbd_data.10aa6b8b4567.000000000000370c/head//3, auth == maps.end()
2014-06-12 14:51:56.433390 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9f0e486b/rbd_data.10aa6b8b4567.000000000000370c/head//3
2014-06-12 14:51:56.433399 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 14bf486b/rbd_data.10a16b8b4567.0000000000001949/head//3, auth == maps.end()
2014-06-12 14:51:56.433408 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 14bf486b/rbd_data.10a16b8b4567.0000000000001949/head//3
2014-06-12 14:51:56.433417 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ad11586b/rbd_data.10a56b8b4567.0000000000001e55/head//3, auth == maps.end()
2014-06-12 14:51:56.433425 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ad11586b/rbd_data.10a56b8b4567.0000000000001e55/head//3
2014-06-12 14:51:56.433434 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f076586b/rbd_data.10a66b8b4567.0000000000000698/head//3, auth == maps.end()
2014-06-12 14:51:56.433442 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj f076586b/rbd_data.10a66b8b4567.0000000000000698/head//3
2014-06-12 14:51:56.433453 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1087586b/rbd_data.10a76b8b4567.0000000000000f8e/head//3, auth == maps.end()
2014-06-12 14:51:56.433464 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1087586b/rbd_data.10a76b8b4567.0000000000000f8e/head//3
2014-06-12 14:51:56.433473 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 877b586b/rbd_data.10a76b8b4567.0000000000002e95/head//3, auth == maps.end()
2014-06-12 14:51:56.433483 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 877b586b/rbd_data.10a76b8b4567.0000000000002e95/head//3
2014-06-12 14:51:56.433493 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 58ac586b/rbd_data.10b06b8b4567.000000000000377f/head//3, auth == maps.end()
2014-06-12 14:51:56.433502 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 58ac586b/rbd_data.10b06b8b4567.000000000000377f/head//3
2014-06-12 14:51:56.433513 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aad7686b/rbd_data.10a16b8b4567.00000000000015f7/head//3, auth == maps.end()
2014-06-12 14:51:56.433521 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj aad7686b/rbd_data.10a16b8b4567.00000000000015f7/head//3
2014-06-12 14:51:56.433531 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a488686b/rbd_data.10906b8b4567.00000000000016f4/head//3, auth == maps.end()
2014-06-12 14:51:56.433540 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a488686b/rbd_data.10906b8b4567.00000000000016f4/head//3
2014-06-12 14:51:56.433550 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5589686b/rbd_data.10af6b8b4567.00000000000029c4/head//3, auth == maps.end()
2014-06-12 14:51:56.433559 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5589686b/rbd_data.10af6b8b4567.00000000000029c4/head//3
2014-06-12 14:51:56.433567 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c3cb686b/rbd_data.10a16b8b4567.0000000000001796/head//3, auth == maps.end()
2014-06-12 14:51:56.433576 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c3cb686b/rbd_data.10a16b8b4567.0000000000001796/head//3
2014-06-12 14:51:56.433585 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 73e0786b/rbd_data.109c6b8b4567.0000000000002ee4/head//3, auth == maps.end()
2014-06-12 14:51:56.433593 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 73e0786b/rbd_data.109c6b8b4567.0000000000002ee4/head//3
2014-06-12 14:51:56.433603 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] 
2014-06-12 14:51:56.433610 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub
2014-06-12 14:51:56.433635 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  73e0786b/rbd_data.109c6b8b4567.0000000000002ee4/head//3 73e0786b/rbd_data.109c6b8b4567.0000000000002ee4/head//3(173'1956 client.4293.0:94506 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1956)
2014-06-12 14:51:56.433650 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c3cb686b/rbd_data.10a16b8b4567.0000000000001796/head//3 c3cb686b/rbd_data.10a16b8b4567.0000000000001796/head//3(173'1012 client.4298.0:53024 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1012)
2014-06-12 14:51:56.433662 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5589686b/rbd_data.10af6b8b4567.00000000000029c4/head//3 5589686b/rbd_data.10af6b8b4567.00000000000029c4/head//3(173'1704 client.4312.0:86962 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1704)
2014-06-12 14:51:56.433673 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a488686b/rbd_data.10906b8b4567.00000000000016f4/head//3 a488686b/rbd_data.10906b8b4567.00000000000016f4/head//3(173'977 client.4281.0:50437 wrlock_by=unknown.0.0:0 dirty s 4194304 uv977)
2014-06-12 14:51:56.433684 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  aad7686b/rbd_data.10a16b8b4567.00000000000015f7/head//3 aad7686b/rbd_data.10a16b8b4567.00000000000015f7/head//3(173'995 client.4298.0:50175 wrlock_by=unknown.0.0:0 dirty s 4194304 uv995)
2014-06-12 14:51:56.433697 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  58ac586b/rbd_data.10b06b8b4567.000000000000377f/head//3 58ac586b/rbd_data.10b06b8b4567.000000000000377f/head//3(173'2178 client.4313.0:111374 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2178)
2014-06-12 14:51:56.433708 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  877b586b/rbd_data.10a76b8b4567.0000000000002e95/head//3 877b586b/rbd_data.10a76b8b4567.0000000000002e95/head//3(173'1907 client.4304.0:93537 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1907)
2014-06-12 14:51:56.433721 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1087586b/rbd_data.10a76b8b4567.0000000000000f8e/head//3 1087586b/rbd_data.10a76b8b4567.0000000000000f8e/head//3(173'2861 client.4304.0:143182 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2861)
2014-06-12 14:51:56.433732 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  f076586b/rbd_data.10a66b8b4567.0000000000000698/head//3 f076586b/rbd_data.10a66b8b4567.0000000000000698/head//3(173'2537 client.4303.0:128593 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2537)
2014-06-12 14:51:56.433744 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ad11586b/rbd_data.10a56b8b4567.0000000000001e55/head//3 ad11586b/rbd_data.10a56b8b4567.0000000000001e55/head//3(173'1257 client.4302.0:64598 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1257)
2014-06-12 14:51:56.433756 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  14bf486b/rbd_data.10a16b8b4567.0000000000001949/head//3 14bf486b/rbd_data.10a16b8b4567.0000000000001949/head//3(173'1091 client.4298.0:56060 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1091)
2014-06-12 14:51:56.433767 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9f0e486b/rbd_data.10aa6b8b4567.000000000000370c/head//3 9f0e486b/rbd_data.10aa6b8b4567.000000000000370c/head//3(173'2167 client.4307.0:111407 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2167)
2014-06-12 14:51:56.433780 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  550e486b/rbd_data.10ac6b8b4567.000000000000127c/head//3 550e486b/rbd_data.10ac6b8b4567.000000000000127c/head//3(173'836 client.4309.0:42341 wrlock_by=unknown.0.0:0 dirty s 4194304 uv836)
2014-06-12 14:51:56.433792 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  f6db486b/rbd_data.10aa6b8b4567.0000000000002bc7/head//3 f6db486b/rbd_data.10aa6b8b4567.0000000000002bc7/head//3(173'1786 client.4307.0:90445 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1786)
2014-06-12 14:51:56.433804 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  d94b486b/rbd_data.109a6b8b4567.000000000000101c/head//3 d94b486b/rbd_data.109a6b8b4567.000000000000101c/head//3(173'729 client.4291.0:38194 wrlock_by=unknown.0.0:0 dirty s 4194304 uv729)
2014-06-12 14:51:56.433819 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5928486b/rbd_data.10a76b8b4567.0000000000001f53/head//3 5928486b/rbd_data.10a76b8b4567.0000000000001f53/head//3(173'1365 client.4304.0:66301 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1365)
2014-06-12 14:51:56.433832 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  76d5486b/rbd_data.109f6b8b4567.0000000000003445/head//3 76d5486b/rbd_data.109f6b8b4567.0000000000003445/head//3(173'1996 client.4296.0:102006 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1996)
2014-06-12 14:51:56.433844 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1d13486b/rbd_data.10b26b8b4567.00000000000003c3/head//3 1d13486b/rbd_data.10b26b8b4567.00000000000003c3/head//3(173'2493 client.4315.0:123082 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2493)
2014-06-12 14:51:56.433856 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  bbd1486b/rbd_data.108e6b8b4567.0000000000002410/head//3 bbd1486b/rbd_data.108e6b8b4567.0000000000002410/head//3(173'1518 client.4279.0:74568 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1518)
2014-06-12 14:51:56.433869 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5191486b/rbd_data.10b06b8b4567.00000000000029aa/head//3 5191486b/rbd_data.10b06b8b4567.00000000000029aa/head//3(173'1641 client.4313.0:85628 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1641)
2014-06-12 14:51:56.433881 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1d8f386b/rbd_data.108e6b8b4567.0000000000003a1f/head//3 1d8f386b/rbd_data.108e6b8b4567.0000000000003a1f/head//3(173'2297 client.4279.0:114223 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2297)
2014-06-12 14:51:56.433892 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  38bd386b/rbd_data.10aa6b8b4567.0000000000002b5e/head//3 38bd386b/rbd_data.10aa6b8b4567.0000000000002b5e/head//3(173'1773 client.4307.0:89690 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1773)
2014-06-12 14:51:56.433904 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  e2ad386b/rbd_data.109c6b8b4567.000000000000287d/head//3 e2ad386b/rbd_data.109c6b8b4567.000000000000287d/head//3(173'1725 client.4293.0:82204 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1725)
2014-06-12 14:51:56.433916 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  cdb8386b/rbd_data.10b36b8b4567.0000000000001dd8/head//3 cdb8386b/rbd_data.10b36b8b4567.0000000000001dd8/head//3(173'1220 client.4316.0:64598 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1220)
2014-06-12 14:51:56.433928 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  2058386b/rbd_data.10ac6b8b4567.0000000000002160/head//3 2058386b/rbd_data.10ac6b8b4567.0000000000002160/head//3(173'1391 client.4309.0:68016 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1391)
2014-06-12 14:51:56.433938 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:56.433949 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:56.433957 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:56.433959 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:56.433961 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:56.433999 7f9d60bd2700 10 collection_list_partial 3.6b_head start:1201786b//0//-1 is_max:0
2014-06-12 14:51:56.434221 7f9d60bd2700 20 objects: [7212786b/rbd_data.10a16b8b4567.00000000000024ae/head//3,f132786b/rbd_data.10a06b8b4567.000000000000371a/head//3,1a72786b/rbd_data.10a96b8b4567.000000000000066e/head//3,1895786b/rbd_data.109c6b8b4567.0000000000002a88/head//3,e776786b/rbd_data.10a36b8b4567.0000000000000ade/head//3,4ed6786b/rbd_data.109b6b8b4567.0000000000002220/head//3,c649786b/rbd_data.10a46b8b4567.0000000000002cd2/head//3,12ca786b/rbd_data.10ae6b8b4567.0000000000001ae2/head//3,237b786b/rbd_data.10a66b8b4567.00000000000007e1/head//3,1ddb786b/rbd_data.10a16b8b4567.00000000000005d2/head//3,4fdd786b/rbd_data.10a86b8b4567.00000000000034b2/head//3,ad1f786b/rbd_data.10a26b8b4567.0000000000000e64/head//3,124f786b/rbd_data.10966b8b4567.000000000000163a/head//3,aebf786b/rbd_data.109f6b8b4567.0000000000001ed0/head//3,e610886b/rbd_data.10a56b8b4567.0000000000000fc0/head//3,dc55886b/rbd_data.10946b8b4567.0000000000000778/head//3,2c86886b/rbd_data.10966b8b4567.00000000000008d2/head//3,778886b/rbd_data.109e6b8b4567.0000000000003aed/head//3,4ec8886b/rbd_data.10af6b8b4567.00000000000021ad/head//3,9269886b/rbd_data.10a06b8b4567.0000000000001a49/head//3,3fbd886b/rbd_data.109b6b8b4567.0000000000000991/head//3,7ffe886b/rbd_data.10b06b8b4567.0000000000002387/head//3,da20986b/rbd_data.10a76b8b4567.0000000000002436/head//3,e2e0986b/rbd_data.10a76b8b4567.000000000000066b/head//3,d3e0986b/rbd_data.109b6b8b4567.0000000000001d62/head//3]
2014-06-12 14:51:56.434709 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  requesting scrubmap from osd.29
2014-06-12 14:51:56.434737 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:56.434755 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:56.434763 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:56.434769 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:56.434774 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:56.434780 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_map_chunk [1201786b//0//-1,1622986b//0//-1)
2014-06-12 14:51:56.434787 7f9d60bd2700 10 collection_list_partial 3.6b_head start:1201786b//0//-1 is_max:0
2014-06-12 14:51:56.435266 7f9d60bd2700 20 objects: [7212786b/rbd_data.10a16b8b4567.00000000000024ae/head//3,f132786b/rbd_data.10a06b8b4567.000000000000371a/head//3,1a72786b/rbd_data.10a96b8b4567.000000000000066e/head//3,1895786b/rbd_data.109c6b8b4567.0000000000002a88/head//3,e776786b/rbd_data.10a36b8b4567.0000000000000ade/head//3,4ed6786b/rbd_data.109b6b8b4567.0000000000002220/head//3,c649786b/rbd_data.10a46b8b4567.0000000000002cd2/head//3,12ca786b/rbd_data.10ae6b8b4567.0000000000001ae2/head//3,237b786b/rbd_data.10a66b8b4567.00000000000007e1/head//3,1ddb786b/rbd_data.10a16b8b4567.00000000000005d2/head//3,4fdd786b/rbd_data.10a86b8b4567.00000000000034b2/head//3,ad1f786b/rbd_data.10a26b8b4567.0000000000000e64/head//3,124f786b/rbd_data.10966b8b4567.000000000000163a/head//3,aebf786b/rbd_data.109f6b8b4567.0000000000001ed0/head//3,e610886b/rbd_data.10a56b8b4567.0000000000000fc0/head//3,dc55886b/rbd_data.10946b8b4567.0000000000000778/head//3,2c86886b/rbd_data.10966b8b4567.00000000000008d2/head//3,778886b/rbd_data.109e6b8b4567.0000000000003aed/head//3,4ec8886b/rbd_data.10af6b8b4567.00000000000021ad/head//3,9269886b/rbd_data.10a06b8b4567.0000000000001a49/head//3,3fbd886b/rbd_data.109b6b8b4567.0000000000000991/head//3,7ffe886b/rbd_data.10b06b8b4567.0000000000002387/head//3,da20986b/rbd_data.10a76b8b4567.0000000000002436/head//3,e2e0986b/rbd_data.10a76b8b4567.000000000000066b/head//3,d3e0986b/rbd_data.109b6b8b4567.0000000000001d62/head//3,1622986b/rbd_data.10a16b8b4567.0000000000001130/head//3,1852986b/rbd_data.109b6b8b4567.0000000000001c96/head//3,1f72986b/rbd_data.109a6b8b4567.0000000000002b71/head//3,9574986b/rbd_data.10a56b8b4567.00000000000012ea/head//3,ba76986b/rbd_data.10a86b8b4567.0000000000002547/head//3,2386986b/rbd_data.10966b8b4567.000000000000246c/head//3,8ae6986b/rbd_data.10b06b8b4567.0000000000003881/head//3,807986b/rbd_data.10ad6b8b4567.0000000000003a00/head//3,71d7986b/rbd_data.10956b8b4567.00000000000001bc/head//3,462a986b/rbd_data.10ad6b8b4567.0000000000000ee5/head//3,a7ba986b/rbd_data.10b46b8b4567.0000000000001a25/head//3,640b986b/rbd_data.109a6b8b4567.0000000000001e70/head//3,694b986b/rbd_data.10ac6b8b4567.00000000000010be/head//3,f7bf986b/rbd_data.10b46b8b4567.00000000000002dd/head//3,6f90a86b/rbd_data.10b46b8b4567.0000000000001745/head//3,5431a86b/rbd_data.10966b8b4567.0000000000003181/head//3,23e1a86b/rbd_data.10966b8b4567.00000000000015a1/head//3,e6d4a86b/rbd_data.10a86b8b4567.0000000000002b33/head//3,6cf5a86b/rbd_data.10a56b8b4567.0000000000002e62/head//3,c87a86b/rbd_data.10ae6b8b4567.00000000000017b9/head//3,acb8a86b/rbd_data.10976b8b4567.0000000000003914/head//3,dcc8a86b/rbd_data.109b6b8b4567.00000000000005a8/head//3,f799a86b/rbd_data.10af6b8b4567.0000000000001656/head//3,6b99a86b/rbd_data.10a36b8b4567.0000000000003450/head//3,42d9a86b/rbd_data.109e6b8b4567.0000000000002580/head//3,c2f9a86b/rbd_data.10a66b8b4567.0000000000000fda/head//3,ab5ea86b/rbd_data.10b36b8b4567.0000000000003956/head//3,8fb1b86b/rbd_data.10ac6b8b4567.0000000000001c07/head//3,98f7b86b/rbd_data.10a76b8b4567.0000000000000875/head//3,1698b86b/rbd_data.10ac6b8b4567.0000000000001b01/head//3,1ca8b86b/rbd_data.10b46b8b4567.0000000000002391/head//3,fae8b86b/rbd_data.10af6b8b4567.0000000000002783/head//3,8db9b86b/rbd_data.10ae6b8b4567.000000000000110c/head//3,d07bb86b/rbd_data.10af6b8b4567.00000000000011b9/head//3,f0d1c86b/rbd_data.10ab6b8b4567.00000000000037fa/head//3,2383c86b/rbd_data.10986b8b4567.0000000000002a8c/head//3,6c14c86b/rbd_data.10916b8b4567.000000000000017d/head//3,8624c86b/rbd_data.109f6b8b4567.000000000000275c/head//3,b546c86b/rbd_data.10af6b8b4567.0000000000003b24/head//3]
2014-06-12 14:51:56.435342 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:56.435357 7f9d60bd2700 10 stat 3.6b_head/7212786b/rbd_data.10a16b8b4567.00000000000024ae/head//3
2014-06-12 14:51:56.435470 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7212786b/rbd_data.10a16b8b4567.00000000000024ae/head//3
2014-06-12 14:51:56.465944 7f9d60bd2700 10 getattrs 3.6b_head/7212786b/rbd_data.10a16b8b4567.00000000000024ae/head//3 = 0
2014-06-12 14:51:56.465955 7f9d60bd2700 10 stat 3.6b_head/f132786b/rbd_data.10a06b8b4567.000000000000371a/head//3
2014-06-12 14:51:56.466048 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid f132786b/rbd_data.10a06b8b4567.000000000000371a/head//3
2014-06-12 14:51:56.502195 7f9d60bd2700 10 getattrs 3.6b_head/f132786b/rbd_data.10a06b8b4567.000000000000371a/head//3 = 0
2014-06-12 14:51:56.502203 7f9d60bd2700 10 stat 3.6b_head/1a72786b/rbd_data.10a96b8b4567.000000000000066e/head//3
2014-06-12 14:51:56.502303 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1a72786b/rbd_data.10a96b8b4567.000000000000066e/head//3
2014-06-12 14:51:56.522851 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:56.523680 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:56.533129 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:56.547944 7f9d60bd2700 10 getattrs 3.6b_head/1a72786b/rbd_data.10a96b8b4567.000000000000066e/head//3 = 0
2014-06-12 14:51:56.547954 7f9d60bd2700 10 stat 3.6b_head/1895786b/rbd_data.109c6b8b4567.0000000000002a88/head//3
2014-06-12 14:51:56.548062 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1895786b/rbd_data.109c6b8b4567.0000000000002a88/head//3
2014-06-12 14:51:56.570585 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:56.581122 7f9d60bd2700 10 getattrs 3.6b_head/1895786b/rbd_data.109c6b8b4567.0000000000002a88/head//3 = 0
2014-06-12 14:51:56.581130 7f9d60bd2700 10 stat 3.6b_head/e776786b/rbd_data.10a36b8b4567.0000000000000ade/head//3
2014-06-12 14:51:56.581226 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid e776786b/rbd_data.10a36b8b4567.0000000000000ade/head//3
2014-06-12 14:51:56.637824 7f9d60bd2700 10 getattrs 3.6b_head/e776786b/rbd_data.10a36b8b4567.0000000000000ade/head//3 = 0
2014-06-12 14:51:56.637843 7f9d60bd2700 10 stat 3.6b_head/4ed6786b/rbd_data.109b6b8b4567.0000000000002220/head//3
2014-06-12 14:51:56.637937 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 4ed6786b/rbd_data.109b6b8b4567.0000000000002220/head//3
2014-06-12 14:51:56.644380 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:56.662451 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:56.665923 7f9d60bd2700 10 getattrs 3.6b_head/4ed6786b/rbd_data.109b6b8b4567.0000000000002220/head//3 = 0
2014-06-12 14:51:56.665933 7f9d60bd2700 10 stat 3.6b_head/c649786b/rbd_data.10a46b8b4567.0000000000002cd2/head//3
2014-06-12 14:51:56.666047 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c649786b/rbd_data.10a46b8b4567.0000000000002cd2/head//3
2014-06-12 14:51:56.685475 7f9d60bd2700 10 getattrs 3.6b_head/c649786b/rbd_data.10a46b8b4567.0000000000002cd2/head//3 = 0
2014-06-12 14:51:56.685484 7f9d60bd2700 10 stat 3.6b_head/12ca786b/rbd_data.10ae6b8b4567.0000000000001ae2/head//3
2014-06-12 14:51:56.685586 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 12ca786b/rbd_data.10ae6b8b4567.0000000000001ae2/head//3
2014-06-12 14:51:56.686001 7f9d60bd2700 10 getattrs 3.6b_head/12ca786b/rbd_data.10ae6b8b4567.0000000000001ae2/head//3 = 0
2014-06-12 14:51:56.686008 7f9d60bd2700 10 stat 3.6b_head/237b786b/rbd_data.10a66b8b4567.00000000000007e1/head//3
2014-06-12 14:51:56.686099 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 237b786b/rbd_data.10a66b8b4567.00000000000007e1/head//3
2014-06-12 14:51:56.701313 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:56.717871 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:56.717918 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:56.727112 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:56.746602 7f9d60bd2700 10 getattrs 3.6b_head/237b786b/rbd_data.10a66b8b4567.00000000000007e1/head//3 = 0
2014-06-12 14:51:56.746617 7f9d60bd2700 10 stat 3.6b_head/1ddb786b/rbd_data.10a16b8b4567.00000000000005d2/head//3
2014-06-12 14:51:56.746723 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1ddb786b/rbd_data.10a16b8b4567.00000000000005d2/head//3
2014-06-12 14:51:56.768781 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:56.786106 7f9d60bd2700 10 getattrs 3.6b_head/1ddb786b/rbd_data.10a16b8b4567.00000000000005d2/head//3 = 0
2014-06-12 14:51:56.786120 7f9d60bd2700 10 stat 3.6b_head/4fdd786b/rbd_data.10a86b8b4567.00000000000034b2/head//3
2014-06-12 14:51:56.786246 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 4fdd786b/rbd_data.10a86b8b4567.00000000000034b2/head//3
2014-06-12 14:51:56.802705 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:56.809931 7f9d60bd2700 10 getattrs 3.6b_head/4fdd786b/rbd_data.10a86b8b4567.00000000000034b2/head//3 = 0
2014-06-12 14:51:56.809942 7f9d60bd2700 10 stat 3.6b_head/ad1f786b/rbd_data.10a26b8b4567.0000000000000e64/head//3
2014-06-12 14:51:56.810058 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ad1f786b/rbd_data.10a26b8b4567.0000000000000e64/head//3
2014-06-12 14:51:56.833289 7f9d60bd2700 10 getattrs 3.6b_head/ad1f786b/rbd_data.10a26b8b4567.0000000000000e64/head//3 = 0
2014-06-12 14:51:56.833300 7f9d60bd2700 10 stat 3.6b_head/124f786b/rbd_data.10966b8b4567.000000000000163a/head//3
2014-06-12 14:51:56.833404 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 124f786b/rbd_data.10966b8b4567.000000000000163a/head//3
2014-06-12 14:51:56.845448 7f9d60bd2700 10 getattrs 3.6b_head/124f786b/rbd_data.10966b8b4567.000000000000163a/head//3 = 0
2014-06-12 14:51:56.845460 7f9d60bd2700 10 stat 3.6b_head/aebf786b/rbd_data.109f6b8b4567.0000000000001ed0/head//3
2014-06-12 14:51:56.845575 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid aebf786b/rbd_data.109f6b8b4567.0000000000001ed0/head//3
2014-06-12 14:51:56.855846 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:56.866024 7f9d60bd2700 10 getattrs 3.6b_head/aebf786b/rbd_data.109f6b8b4567.0000000000001ed0/head//3 = 0
2014-06-12 14:51:56.866034 7f9d60bd2700 10 stat 3.6b_head/e610886b/rbd_data.10a56b8b4567.0000000000000fc0/head//3
2014-06-12 14:51:56.866155 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid e610886b/rbd_data.10a56b8b4567.0000000000000fc0/head//3
2014-06-12 14:51:56.866192 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:56.869556 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:56.886031 7f9d60bd2700 10 getattrs 3.6b_head/e610886b/rbd_data.10a56b8b4567.0000000000000fc0/head//3 = 0
2014-06-12 14:51:56.886041 7f9d60bd2700 10 stat 3.6b_head/dc55886b/rbd_data.10946b8b4567.0000000000000778/head//3
2014-06-12 14:51:56.886166 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid dc55886b/rbd_data.10946b8b4567.0000000000000778/head//3
2014-06-12 14:51:56.908293 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:56.912433 7f9d60bd2700 10 getattrs 3.6b_head/dc55886b/rbd_data.10946b8b4567.0000000000000778/head//3 = 0
2014-06-12 14:51:56.912444 7f9d60bd2700 10 stat 3.6b_head/2c86886b/rbd_data.10966b8b4567.00000000000008d2/head//3
2014-06-12 14:51:56.912571 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 2c86886b/rbd_data.10966b8b4567.00000000000008d2/head//3
2014-06-12 14:51:56.965412 7f9d60bd2700 10 getattrs 3.6b_head/2c86886b/rbd_data.10966b8b4567.00000000000008d2/head//3 = 0
2014-06-12 14:51:56.965443 7f9d60bd2700 10 stat 3.6b_head/778886b/rbd_data.109e6b8b4567.0000000000003aed/head//3
2014-06-12 14:51:56.965564 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 778886b/rbd_data.109e6b8b4567.0000000000003aed/head//3
2014-06-12 14:51:56.965674 7f9d60bd2700 10 getattrs 3.6b_head/778886b/rbd_data.109e6b8b4567.0000000000003aed/head//3 = 0
2014-06-12 14:51:56.965679 7f9d60bd2700 10 stat 3.6b_head/4ec8886b/rbd_data.10af6b8b4567.00000000000021ad/head//3
2014-06-12 14:51:56.965758 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 4ec8886b/rbd_data.10af6b8b4567.00000000000021ad/head//3
2014-06-12 14:51:56.965823 7f9d60bd2700 10 getattrs 3.6b_head/4ec8886b/rbd_data.10af6b8b4567.00000000000021ad/head//3 = 0
2014-06-12 14:51:56.965828 7f9d60bd2700 10 stat 3.6b_head/9269886b/rbd_data.10a06b8b4567.0000000000001a49/head//3
2014-06-12 14:51:56.965905 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9269886b/rbd_data.10a06b8b4567.0000000000001a49/head//3
2014-06-12 14:51:56.966238 7f9d60bd2700 10 getattrs 3.6b_head/9269886b/rbd_data.10a06b8b4567.0000000000001a49/head//3 = 0
2014-06-12 14:51:56.966247 7f9d60bd2700 10 stat 3.6b_head/3fbd886b/rbd_data.109b6b8b4567.0000000000000991/head//3
2014-06-12 14:51:56.966333 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 3fbd886b/rbd_data.109b6b8b4567.0000000000000991/head//3
2014-06-12 14:51:56.966789 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:56.976375 7f9d60bd2700 10 getattrs 3.6b_head/3fbd886b/rbd_data.109b6b8b4567.0000000000000991/head//3 = 0
2014-06-12 14:51:56.976384 7f9d60bd2700 10 stat 3.6b_head/7ffe886b/rbd_data.10b06b8b4567.0000000000002387/head//3
2014-06-12 14:51:56.976515 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7ffe886b/rbd_data.10b06b8b4567.0000000000002387/head//3
2014-06-12 14:51:57.006047 7f9d60bd2700 10 getattrs 3.6b_head/7ffe886b/rbd_data.10b06b8b4567.0000000000002387/head//3 = 0
2014-06-12 14:51:57.006067 7f9d60bd2700 10 stat 3.6b_head/da20986b/rbd_data.10a76b8b4567.0000000000002436/head//3
2014-06-12 14:51:57.006188 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid da20986b/rbd_data.10a76b8b4567.0000000000002436/head//3
2014-06-12 14:51:57.021947 7f9d60bd2700 10 getattrs 3.6b_head/da20986b/rbd_data.10a76b8b4567.0000000000002436/head//3 = 0
2014-06-12 14:51:57.021956 7f9d60bd2700 10 stat 3.6b_head/e2e0986b/rbd_data.10a76b8b4567.000000000000066b/head//3
2014-06-12 14:51:57.022051 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid e2e0986b/rbd_data.10a76b8b4567.000000000000066b/head//3
2014-06-12 14:51:57.024256 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:57.024312 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:51:57.030870 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:57.061622 7f9d60bd2700 10 getattrs 3.6b_head/e2e0986b/rbd_data.10a76b8b4567.000000000000066b/head//3 = 0
2014-06-12 14:51:57.061662 7f9d60bd2700 10 stat 3.6b_head/d3e0986b/rbd_data.109b6b8b4567.0000000000001d62/head//3
2014-06-12 14:51:57.061824 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid d3e0986b/rbd_data.109b6b8b4567.0000000000001d62/head//3
2014-06-12 14:51:57.067337 7f9d5501e700  5 -- op tracker -- , seq: 748616, time: 2014-06-12 14:51:57.067257, event: header_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.067350 7f9d5501e700  5 -- op tracker -- , seq: 748616, time: 2014-06-12 14:51:57.067259, event: throttled, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.067368 7f9d5501e700  5 -- op tracker -- , seq: 748616, time: 2014-06-12 14:51:57.067303, event: all_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.067374 7f9d5501e700  5 -- op tracker -- , seq: 748616, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.067378 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:57.067384 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:57.067403 7f9d5501e700 15 osd.2 173 enqueue_op 0x3850f500 prio 127 cost 10717 latency 0.000146 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:57.067477 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:57.080208 7f9d60bd2700 10 getattrs 3.6b_head/d3e0986b/rbd_data.109b6b8b4567.0000000000001d62/head//3 = 0
2014-06-12 14:51:57.080226 7f9d60bd2700 10 collection_getattrs 3.6b_head
2014-06-12 14:51:57.080296 7f9d60bd2700 10 collection_getattrs 3.6b_head = 0
2014-06-12 14:51:57.080301 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:57.080329 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:57.080354 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:57.080370 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:57.080375 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:57.080402 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x3850f500 prio 127 cost 10717 latency 0.013144 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]
2014-06-12 14:51:57.080462 7f9d6a3e5700  5 -- op tracker -- , seq: 748616, time: 2014-06-12 14:51:57.080454, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.080480 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] handle_message: 0x3850f500
2014-06-12 14:51:57.080497 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:57.080518 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:57.080536 7f9d6a3e5700  5 -- op tracker -- , seq: 748616, time: 2014-06-12 14:51:57.080531, event: started, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.080557 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  got 29 scrub map
2014-06-12 14:51:57.080664 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] map version is 173'2887
2014-06-12 14:51:57.080684 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x3850f500 finish
2014-06-12 14:51:57.080690 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:57.080696 7f9d6a3e5700  5 -- op tracker -- , seq: 748616, time: 2014-06-12 14:51:57.080690, event: done, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.080700 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:57.080721 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:57.080727 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:57.080724 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:57.080751 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:57.080792 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:57.080805 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:57.080817 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub replica 29 has 25 items
2014-06-12 14:51:57.080852 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7212786b/rbd_data.10a16b8b4567.00000000000024ae/head//3, auth == maps.end()
2014-06-12 14:51:57.080871 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7212786b/rbd_data.10a16b8b4567.00000000000024ae/head//3
2014-06-12 14:51:57.080892 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f132786b/rbd_data.10a06b8b4567.000000000000371a/head//3, auth == maps.end()
2014-06-12 14:51:57.080911 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj f132786b/rbd_data.10a06b8b4567.000000000000371a/head//3
2014-06-12 14:51:57.080930 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1a72786b/rbd_data.10a96b8b4567.000000000000066e/head//3, auth == maps.end()
2014-06-12 14:51:57.080952 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1a72786b/rbd_data.10a96b8b4567.000000000000066e/head//3
2014-06-12 14:51:57.080972 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1895786b/rbd_data.109c6b8b4567.0000000000002a88/head//3, auth == maps.end()
2014-06-12 14:51:57.080991 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1895786b/rbd_data.109c6b8b4567.0000000000002a88/head//3
2014-06-12 14:51:57.081011 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e776786b/rbd_data.10a36b8b4567.0000000000000ade/head//3, auth == maps.end()
2014-06-12 14:51:57.081031 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj e776786b/rbd_data.10a36b8b4567.0000000000000ade/head//3
2014-06-12 14:51:57.081051 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4ed6786b/rbd_data.109b6b8b4567.0000000000002220/head//3, auth == maps.end()
2014-06-12 14:51:57.081070 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 4ed6786b/rbd_data.109b6b8b4567.0000000000002220/head//3
2014-06-12 14:51:57.081090 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c649786b/rbd_data.10a46b8b4567.0000000000002cd2/head//3, auth == maps.end()
2014-06-12 14:51:57.081109 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c649786b/rbd_data.10a46b8b4567.0000000000002cd2/head//3
2014-06-12 14:51:57.081127 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 12ca786b/rbd_data.10ae6b8b4567.0000000000001ae2/head//3, auth == maps.end()
2014-06-12 14:51:57.081146 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 12ca786b/rbd_data.10ae6b8b4567.0000000000001ae2/head//3
2014-06-12 14:51:57.081166 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 237b786b/rbd_data.10a66b8b4567.00000000000007e1/head//3, auth == maps.end()
2014-06-12 14:51:57.081185 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 237b786b/rbd_data.10a66b8b4567.00000000000007e1/head//3
2014-06-12 14:51:57.081205 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1ddb786b/rbd_data.10a16b8b4567.00000000000005d2/head//3, auth == maps.end()
2014-06-12 14:51:57.081225 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1ddb786b/rbd_data.10a16b8b4567.00000000000005d2/head//3
2014-06-12 14:51:57.081244 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4fdd786b/rbd_data.10a86b8b4567.00000000000034b2/head//3, auth == maps.end()
2014-06-12 14:51:57.081263 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 4fdd786b/rbd_data.10a86b8b4567.00000000000034b2/head//3
2014-06-12 14:51:57.081284 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ad1f786b/rbd_data.10a26b8b4567.0000000000000e64/head//3, auth == maps.end()
2014-06-12 14:51:57.081303 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ad1f786b/rbd_data.10a26b8b4567.0000000000000e64/head//3
2014-06-12 14:51:57.081320 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 124f786b/rbd_data.10966b8b4567.000000000000163a/head//3, auth == maps.end()
2014-06-12 14:51:57.081339 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 124f786b/rbd_data.10966b8b4567.000000000000163a/head//3
2014-06-12 14:51:57.081358 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj aebf786b/rbd_data.109f6b8b4567.0000000000001ed0/head//3, auth == maps.end()
2014-06-12 14:51:57.081378 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj aebf786b/rbd_data.109f6b8b4567.0000000000001ed0/head//3
2014-06-12 14:51:57.081398 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e610886b/rbd_data.10a56b8b4567.0000000000000fc0/head//3, auth == maps.end()
2014-06-12 14:51:57.081416 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj e610886b/rbd_data.10a56b8b4567.0000000000000fc0/head//3
2014-06-12 14:51:57.081436 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dc55886b/rbd_data.10946b8b4567.0000000000000778/head//3, auth == maps.end()
2014-06-12 14:51:57.081456 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj dc55886b/rbd_data.10946b8b4567.0000000000000778/head//3
2014-06-12 14:51:57.081475 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2c86886b/rbd_data.10966b8b4567.00000000000008d2/head//3, auth == maps.end()
2014-06-12 14:51:57.081495 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 2c86886b/rbd_data.10966b8b4567.00000000000008d2/head//3
2014-06-12 14:51:57.081515 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 778886b/rbd_data.109e6b8b4567.0000000000003aed/head//3, auth == maps.end()
2014-06-12 14:51:57.081534 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 778886b/rbd_data.109e6b8b4567.0000000000003aed/head//3
2014-06-12 14:51:57.081553 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 4ec8886b/rbd_data.10af6b8b4567.00000000000021ad/head//3, auth == maps.end()
2014-06-12 14:51:57.081571 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 4ec8886b/rbd_data.10af6b8b4567.00000000000021ad/head//3
2014-06-12 14:51:57.081592 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9269886b/rbd_data.10a06b8b4567.0000000000001a49/head//3, auth == maps.end()
2014-06-12 14:51:57.081610 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9269886b/rbd_data.10a06b8b4567.0000000000001a49/head//3
2014-06-12 14:51:57.081644 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3fbd886b/rbd_data.109b6b8b4567.0000000000000991/head//3, auth == maps.end()
2014-06-12 14:51:57.081663 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 3fbd886b/rbd_data.109b6b8b4567.0000000000000991/head//3
2014-06-12 14:51:57.081684 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7ffe886b/rbd_data.10b06b8b4567.0000000000002387/head//3, auth == maps.end()
2014-06-12 14:51:57.081703 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7ffe886b/rbd_data.10b06b8b4567.0000000000002387/head//3
2014-06-12 14:51:57.081723 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj da20986b/rbd_data.10a76b8b4567.0000000000002436/head//3, auth == maps.end()
2014-06-12 14:51:57.081742 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj da20986b/rbd_data.10a76b8b4567.0000000000002436/head//3
2014-06-12 14:51:57.081763 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e2e0986b/rbd_data.10a76b8b4567.000000000000066b/head//3, auth == maps.end()
2014-06-12 14:51:57.081782 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj e2e0986b/rbd_data.10a76b8b4567.000000000000066b/head//3
2014-06-12 14:51:57.081802 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d3e0986b/rbd_data.109b6b8b4567.0000000000001d62/head//3, auth == maps.end()
2014-06-12 14:51:57.081820 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj d3e0986b/rbd_data.109b6b8b4567.0000000000001d62/head//3
2014-06-12 14:51:57.081841 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] 
2014-06-12 14:51:57.081856 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub
2014-06-12 14:51:57.081878 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  d3e0986b/rbd_data.109b6b8b4567.0000000000001d62/head//3 d3e0986b/rbd_data.109b6b8b4567.0000000000001d62/head//3(173'1316 client.4292.0:60635 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1316)
2014-06-12 14:51:57.081900 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  e2e0986b/rbd_data.10a76b8b4567.000000000000066b/head//3 e2e0986b/rbd_data.10a76b8b4567.000000000000066b/head//3(173'2542 client.4304.0:126838 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2542)
2014-06-12 14:51:57.081922 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  da20986b/rbd_data.10a76b8b4567.0000000000002436/head//3 da20986b/rbd_data.10a76b8b4567.0000000000002436/head//3(173'1541 client.4304.0:74877 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1541)
2014-06-12 14:51:57.081945 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7ffe886b/rbd_data.10b06b8b4567.0000000000002387/head//3 7ffe886b/rbd_data.10b06b8b4567.0000000000002387/head//3(173'1461 client.4313.0:74261 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1461)
2014-06-12 14:51:57.081967 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  3fbd886b/rbd_data.109b6b8b4567.0000000000000991/head//3 3fbd886b/rbd_data.109b6b8b4567.0000000000000991/head//3(173'2685 client.4292.0:131071 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2685)
2014-06-12 14:51:57.081991 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9269886b/rbd_data.10a06b8b4567.0000000000001a49/head//3 9269886b/rbd_data.10a06b8b4567.0000000000001a49/head//3(173'1044 client.4297.0:56132 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1044)
2014-06-12 14:51:57.082014 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  4ec8886b/rbd_data.10af6b8b4567.00000000000021ad/head//3 4ec8886b/rbd_data.10af6b8b4567.00000000000021ad/head//3(173'1437 client.4312.0:72102 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1437)
2014-06-12 14:51:57.082036 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  778886b/rbd_data.109e6b8b4567.0000000000003aed/head//3 778886b/rbd_data.109e6b8b4567.0000000000003aed/head//3(173'2274 client.4295.0:115805 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2274)
2014-06-12 14:51:57.082060 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  2c86886b/rbd_data.10966b8b4567.00000000000008d2/head//3 2c86886b/rbd_data.10966b8b4567.00000000000008d2/head//3(173'2650 client.4287.0:128580 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2650)
2014-06-12 14:51:57.082083 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  dc55886b/rbd_data.10946b8b4567.0000000000000778/head//3 dc55886b/rbd_data.10946b8b4567.0000000000000778/head//3(173'2565 client.4285.0:127729 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2565)
2014-06-12 14:51:57.082105 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  e610886b/rbd_data.10a56b8b4567.0000000000000fc0/head//3 e610886b/rbd_data.10a56b8b4567.0000000000000fc0/head//3(173'2871 client.4302.0:145314 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2871)
2014-06-12 14:51:57.082128 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  aebf786b/rbd_data.109f6b8b4567.0000000000001ed0/head//3 aebf786b/rbd_data.109f6b8b4567.0000000000001ed0/head//3(173'1166 client.4296.0:63554 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1166)
2014-06-12 14:51:57.082150 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  124f786b/rbd_data.10966b8b4567.000000000000163a/head//3 124f786b/rbd_data.10966b8b4567.000000000000163a/head//3(173'960 client.4287.0:48676 wrlock_by=unknown.0.0:0 dirty s 4194304 uv960)
2014-06-12 14:51:57.082173 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ad1f786b/rbd_data.10a26b8b4567.0000000000000e64/head//3 ad1f786b/rbd_data.10a26b8b4567.0000000000000e64/head//3(173'2843 client.4299.0:143811 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2843)
2014-06-12 14:51:57.082196 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  4fdd786b/rbd_data.10a86b8b4567.00000000000034b2/head//3 4fdd786b/rbd_data.10a86b8b4567.00000000000034b2/head//3(173'2024 client.4305.0:103884 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2024)
2014-06-12 14:51:57.082219 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1ddb786b/rbd_data.10a16b8b4567.00000000000005d2/head//3 1ddb786b/rbd_data.10a16b8b4567.00000000000005d2/head//3(173'2519 client.4298.0:127839 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2519)
2014-06-12 14:51:57.082242 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  237b786b/rbd_data.10a66b8b4567.00000000000007e1/head//3 237b786b/rbd_data.10a66b8b4567.00000000000007e1/head//3(173'2587 client.4303.0:130812 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2587)
2014-06-12 14:51:57.082264 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  12ca786b/rbd_data.10ae6b8b4567.0000000000001ae2/head//3 12ca786b/rbd_data.10ae6b8b4567.0000000000001ae2/head//3(173'1103 client.4311.0:57253 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1103)
2014-06-12 14:51:57.082289 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c649786b/rbd_data.10a46b8b4567.0000000000002cd2/head//3 c649786b/rbd_data.10a46b8b4567.0000000000002cd2/head//3(173'1839 client.4301.0:90836 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1839)
2014-06-12 14:51:57.082311 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  4ed6786b/rbd_data.109b6b8b4567.0000000000002220/head//3 4ed6786b/rbd_data.109b6b8b4567.0000000000002220/head//3(173'1492 client.4292.0:68608 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1492)
2014-06-12 14:51:57.082334 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  e776786b/rbd_data.10a36b8b4567.0000000000000ade/head//3 e776786b/rbd_data.10a36b8b4567.0000000000000ade/head//3(173'2728 client.4300.0:132904 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2728)
2014-06-12 14:51:57.082356 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1895786b/rbd_data.109c6b8b4567.0000000000002a88/head//3 1895786b/rbd_data.109c6b8b4567.0000000000002a88/head//3(173'1798 client.4293.0:86233 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1798)
2014-06-12 14:51:57.082378 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1a72786b/rbd_data.10a96b8b4567.000000000000066e/head//3 1a72786b/rbd_data.10a96b8b4567.000000000000066e/head//3(173'2551 client.4306.0:128334 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2551)
2014-06-12 14:51:57.082401 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  f132786b/rbd_data.10a06b8b4567.000000000000371a/head//3 f132786b/rbd_data.10a06b8b4567.000000000000371a/head//3(173'2124 client.4297.0:107987 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2124)
2014-06-12 14:51:57.082424 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7212786b/rbd_data.10a16b8b4567.00000000000024ae/head//3 7212786b/rbd_data.10a16b8b4567.00000000000024ae/head//3(173'1479 client.4298.0:76288 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1479)
2014-06-12 14:51:57.082442 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:57.082469 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:57.082483 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:57.082486 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:57.082488 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:57.082546 7f9d60bd2700 10 collection_list_partial 3.6b_head start:1622986b//0//-1 is_max:0
2014-06-12 14:51:57.082762 7f9d60bd2700 20 objects: [1852986b/rbd_data.109b6b8b4567.0000000000001c96/head//3,1f72986b/rbd_data.109a6b8b4567.0000000000002b71/head//3,9574986b/rbd_data.10a56b8b4567.00000000000012ea/head//3,ba76986b/rbd_data.10a86b8b4567.0000000000002547/head//3,2386986b/rbd_data.10966b8b4567.000000000000246c/head//3,8ae6986b/rbd_data.10b06b8b4567.0000000000003881/head//3,807986b/rbd_data.10ad6b8b4567.0000000000003a00/head//3,71d7986b/rbd_data.10956b8b4567.00000000000001bc/head//3,462a986b/rbd_data.10ad6b8b4567.0000000000000ee5/head//3,a7ba986b/rbd_data.10b46b8b4567.0000000000001a25/head//3,640b986b/rbd_data.109a6b8b4567.0000000000001e70/head//3,694b986b/rbd_data.10ac6b8b4567.00000000000010be/head//3,f7bf986b/rbd_data.10b46b8b4567.00000000000002dd/head//3,6f90a86b/rbd_data.10b46b8b4567.0000000000001745/head//3,5431a86b/rbd_data.10966b8b4567.0000000000003181/head//3,23e1a86b/rbd_data.10966b8b4567.00000000000015a1/head//3,e6d4a86b/rbd_data.10a86b8b4567.0000000000002b33/head//3,6cf5a86b/rbd_data.10a56b8b4567.0000000000002e62/head//3,c87a86b/rbd_data.10ae6b8b4567.00000000000017b9/head//3,acb8a86b/rbd_data.10976b8b4567.0000000000003914/head//3,dcc8a86b/rbd_data.109b6b8b4567.00000000000005a8/head//3,f799a86b/rbd_data.10af6b8b4567.0000000000001656/head//3,6b99a86b/rbd_data.10a36b8b4567.0000000000003450/head//3,42d9a86b/rbd_data.109e6b8b4567.0000000000002580/head//3,c2f9a86b/rbd_data.10a66b8b4567.0000000000000fda/head//3]
2014-06-12 14:51:57.083266 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  requesting scrubmap from osd.29
2014-06-12 14:51:57.083290 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:57.083312 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:57.083325 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:57.083336 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:57.083347 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:57.083357 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_map_chunk [1622986b//0//-1,ab5ea86b//0//-1)
2014-06-12 14:51:57.083370 7f9d60bd2700 10 collection_list_partial 3.6b_head start:1622986b//0//-1 is_max:0
2014-06-12 14:51:57.083891 7f9d60bd2700 20 objects: [1852986b/rbd_data.109b6b8b4567.0000000000001c96/head//3,1f72986b/rbd_data.109a6b8b4567.0000000000002b71/head//3,9574986b/rbd_data.10a56b8b4567.00000000000012ea/head//3,ba76986b/rbd_data.10a86b8b4567.0000000000002547/head//3,2386986b/rbd_data.10966b8b4567.000000000000246c/head//3,8ae6986b/rbd_data.10b06b8b4567.0000000000003881/head//3,807986b/rbd_data.10ad6b8b4567.0000000000003a00/head//3,71d7986b/rbd_data.10956b8b4567.00000000000001bc/head//3,462a986b/rbd_data.10ad6b8b4567.0000000000000ee5/head//3,a7ba986b/rbd_data.10b46b8b4567.0000000000001a25/head//3,640b986b/rbd_data.109a6b8b4567.0000000000001e70/head//3,694b986b/rbd_data.10ac6b8b4567.00000000000010be/head//3,f7bf986b/rbd_data.10b46b8b4567.00000000000002dd/head//3,6f90a86b/rbd_data.10b46b8b4567.0000000000001745/head//3,5431a86b/rbd_data.10966b8b4567.0000000000003181/head//3,23e1a86b/rbd_data.10966b8b4567.00000000000015a1/head//3,e6d4a86b/rbd_data.10a86b8b4567.0000000000002b33/head//3,6cf5a86b/rbd_data.10a56b8b4567.0000000000002e62/head//3,c87a86b/rbd_data.10ae6b8b4567.00000000000017b9/head//3,acb8a86b/rbd_data.10976b8b4567.0000000000003914/head//3,dcc8a86b/rbd_data.109b6b8b4567.00000000000005a8/head//3,f799a86b/rbd_data.10af6b8b4567.0000000000001656/head//3,6b99a86b/rbd_data.10a36b8b4567.0000000000003450/head//3,42d9a86b/rbd_data.109e6b8b4567.0000000000002580/head//3,c2f9a86b/rbd_data.10a66b8b4567.0000000000000fda/head//3,ab5ea86b/rbd_data.10b36b8b4567.0000000000003956/head//3,8fb1b86b/rbd_data.10ac6b8b4567.0000000000001c07/head//3,98f7b86b/rbd_data.10a76b8b4567.0000000000000875/head//3,1698b86b/rbd_data.10ac6b8b4567.0000000000001b01/head//3,1ca8b86b/rbd_data.10b46b8b4567.0000000000002391/head//3,fae8b86b/rbd_data.10af6b8b4567.0000000000002783/head//3,8db9b86b/rbd_data.10ae6b8b4567.000000000000110c/head//3,d07bb86b/rbd_data.10af6b8b4567.00000000000011b9/head//3,f0d1c86b/rbd_data.10ab6b8b4567.00000000000037fa/head//3,2383c86b/rbd_data.10986b8b4567.0000000000002a8c/head//3,6c14c86b/rbd_data.10916b8b4567.000000000000017d/head//3,8624c86b/rbd_data.109f6b8b4567.000000000000275c/head//3,b546c86b/rbd_data.10af6b8b4567.0000000000003b24/head//3,a8a6c86b/rbd_data.10966b8b4567.0000000000002484/head//3,24e6c86b/rbd_data.10a86b8b4567.00000000000030a3/head//3,1567c86b/rbd_data.10a66b8b4567.0000000000000255/head//3,ec77c86b/rbd_data.10b26b8b4567.0000000000001b8c/head//3,f5a8c86b/rbd_data.108e6b8b4567.000000000000054a/head//3,f75bc86b/rbd_data.10a06b8b4567.00000000000031b3/head//3,dd2ec86b/rbd_data.10b26b8b4567.000000000000196f/head//3,6f2ec86b/rbd_data.10a06b8b4567.0000000000002dbd/head//3,391fc86b/rbd_data.10946b8b4567.0000000000003110/head//3,adc2d86b/rbd_data.10af6b8b4567.000000000000041e/head//3,98e3d86b/rbd_data.109b6b8b4567.000000000000115e/head//3,3fe4d86b/rbd_data.10906b8b4567.0000000000003b05/head//3,bb78d86b/rbd_data.10a96b8b4567.00000000000024f7/head//3,5439d86b/rbd_data.10906b8b4567.000000000000122a/head//3,b68ad86b/rbd_data.10a96b8b4567.0000000000000cf4/head//3,bfcad86b/rbd_data.10986b8b4567.00000000000026c3/head//3,90bbd86b/rbd_data.10aa6b8b4567.0000000000003348/head//3,dc0ed86b/rbd_data.10906b8b4567.00000000000021c6/head//3,f0c0e86b/rbd_data.10906b8b4567.000000000000024c/head//3,de74e86b/rbd_data.10b36b8b4567.000000000000075d/head//3,80e4e86b/rbd_data.10a66b8b4567.00000000000007ec/head//3,7755e86b/rbd_data.10926b8b4567.0000000000001e80/head//3,c6e5e86b/rbd_data.10966b8b4567.0000000000000f07/head//3,f1a7e86b/rbd_data.10a66b8b4567.0000000000001b31/head//3,512de86b/rbd_data.109a6b8b4567.000000000000248b/head//3,c44de86b/rbd_data.10a36b8b4567.0000000000003aad/head//3]
2014-06-12 14:51:57.083977 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:57.083996 7f9d60bd2700 10 stat 3.6b_head/1852986b/rbd_data.109b6b8b4567.0000000000001c96/head//3
2014-06-12 14:51:57.084126 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1852986b/rbd_data.109b6b8b4567.0000000000001c96/head//3
2014-06-12 14:51:57.099006 7f9d60bd2700 10 getattrs 3.6b_head/1852986b/rbd_data.109b6b8b4567.0000000000001c96/head//3 = 0
2014-06-12 14:51:57.099015 7f9d60bd2700 10 stat 3.6b_head/1f72986b/rbd_data.109a6b8b4567.0000000000002b71/head//3
2014-06-12 14:51:57.099104 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1f72986b/rbd_data.109a6b8b4567.0000000000002b71/head//3
2014-06-12 14:51:57.122169 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:57.130338 7f9d60bd2700 10 getattrs 3.6b_head/1f72986b/rbd_data.109a6b8b4567.0000000000002b71/head//3 = 0
2014-06-12 14:51:57.130351 7f9d60bd2700 10 stat 3.6b_head/9574986b/rbd_data.10a56b8b4567.00000000000012ea/head//3
2014-06-12 14:51:57.130457 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9574986b/rbd_data.10a56b8b4567.00000000000012ea/head//3
2014-06-12 14:51:57.133765 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:57.143165 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:57.147592 7f9d60bd2700 10 getattrs 3.6b_head/9574986b/rbd_data.10a56b8b4567.00000000000012ea/head//3 = 0
2014-06-12 14:51:57.147601 7f9d60bd2700 10 stat 3.6b_head/ba76986b/rbd_data.10a86b8b4567.0000000000002547/head//3
2014-06-12 14:51:57.147707 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ba76986b/rbd_data.10a86b8b4567.0000000000002547/head//3
2014-06-12 14:51:57.148083 7f9d60bd2700 10 getattrs 3.6b_head/ba76986b/rbd_data.10a86b8b4567.0000000000002547/head//3 = 0
2014-06-12 14:51:57.148092 7f9d60bd2700 10 stat 3.6b_head/2386986b/rbd_data.10966b8b4567.000000000000246c/head//3
2014-06-12 14:51:57.148205 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 2386986b/rbd_data.10966b8b4567.000000000000246c/head//3
2014-06-12 14:51:57.152582 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:57.172736 7f9d60bd2700 10 getattrs 3.6b_head/2386986b/rbd_data.10966b8b4567.000000000000246c/head//3 = 0
2014-06-12 14:51:57.172748 7f9d60bd2700 10 stat 3.6b_head/8ae6986b/rbd_data.10b06b8b4567.0000000000003881/head//3
2014-06-12 14:51:57.172876 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 8ae6986b/rbd_data.10b06b8b4567.0000000000003881/head//3
2014-06-12 14:51:57.181072 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:57.208257 7f9d60bd2700 10 getattrs 3.6b_head/8ae6986b/rbd_data.10b06b8b4567.0000000000003881/head//3 = 0
2014-06-12 14:51:57.208270 7f9d60bd2700 10 stat 3.6b_head/807986b/rbd_data.10ad6b8b4567.0000000000003a00/head//3
2014-06-12 14:51:57.208414 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 807986b/rbd_data.10ad6b8b4567.0000000000003a00/head//3
2014-06-12 14:51:57.218172 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:57.234950 7f9d60bd2700 10 getattrs 3.6b_head/807986b/rbd_data.10ad6b8b4567.0000000000003a00/head//3 = 0
2014-06-12 14:51:57.234967 7f9d60bd2700 10 stat 3.6b_head/71d7986b/rbd_data.10956b8b4567.00000000000001bc/head//3
2014-06-12 14:51:57.235096 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 71d7986b/rbd_data.10956b8b4567.00000000000001bc/head//3
2014-06-12 14:51:57.271235 7f9d60bd2700 10 getattrs 3.6b_head/71d7986b/rbd_data.10956b8b4567.00000000000001bc/head//3 = 0
2014-06-12 14:51:57.271246 7f9d60bd2700 10 stat 3.6b_head/462a986b/rbd_data.10ad6b8b4567.0000000000000ee5/head//3
2014-06-12 14:51:57.271376 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 462a986b/rbd_data.10ad6b8b4567.0000000000000ee5/head//3
2014-06-12 14:51:57.275266 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:57.275322 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:57.280789 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:57.280814 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:57.280819 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:57.280821 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:57.280859 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:57.281544 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:57.299210 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:57.299339 7f9d60bd2700 10 getattrs 3.6b_head/462a986b/rbd_data.10ad6b8b4567.0000000000000ee5/head//3 = 0
2014-06-12 14:51:57.299351 7f9d60bd2700 10 stat 3.6b_head/a7ba986b/rbd_data.10b46b8b4567.0000000000001a25/head//3
2014-06-12 14:51:57.299462 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a7ba986b/rbd_data.10b46b8b4567.0000000000001a25/head//3
2014-06-12 14:51:57.317279 7f9d60bd2700 10 getattrs 3.6b_head/a7ba986b/rbd_data.10b46b8b4567.0000000000001a25/head//3 = 0
2014-06-12 14:51:57.317298 7f9d60bd2700 10 stat 3.6b_head/640b986b/rbd_data.109a6b8b4567.0000000000001e70/head//3
2014-06-12 14:51:57.317406 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 640b986b/rbd_data.109a6b8b4567.0000000000001e70/head//3
2014-06-12 14:51:57.333093 7f9d60bd2700 10 getattrs 3.6b_head/640b986b/rbd_data.109a6b8b4567.0000000000001e70/head//3 = 0
2014-06-12 14:51:57.333116 7f9d60bd2700 10 stat 3.6b_head/694b986b/rbd_data.10ac6b8b4567.00000000000010be/head//3
2014-06-12 14:51:57.333245 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 694b986b/rbd_data.10ac6b8b4567.00000000000010be/head//3
2014-06-12 14:51:57.342372 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:57.351901 7f9d60bd2700 10 getattrs 3.6b_head/694b986b/rbd_data.10ac6b8b4567.00000000000010be/head//3 = 0
2014-06-12 14:51:57.351919 7f9d60bd2700 10 stat 3.6b_head/f7bf986b/rbd_data.10b46b8b4567.00000000000002dd/head//3
2014-06-12 14:51:57.352066 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid f7bf986b/rbd_data.10b46b8b4567.00000000000002dd/head//3
2014-06-12 14:51:57.375698 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:57.415397 7f9d60bd2700 10 getattrs 3.6b_head/f7bf986b/rbd_data.10b46b8b4567.00000000000002dd/head//3 = 0
2014-06-12 14:51:57.415414 7f9d60bd2700 10 stat 3.6b_head/6f90a86b/rbd_data.10b46b8b4567.0000000000001745/head//3
2014-06-12 14:51:57.415568 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6f90a86b/rbd_data.10b46b8b4567.0000000000001745/head//3
2014-06-12 14:51:57.415701 7f9d60bd2700 10 getattrs 3.6b_head/6f90a86b/rbd_data.10b46b8b4567.0000000000001745/head//3 = 0
2014-06-12 14:51:57.415710 7f9d60bd2700 10 stat 3.6b_head/5431a86b/rbd_data.10966b8b4567.0000000000003181/head//3
2014-06-12 14:51:57.415839 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 5431a86b/rbd_data.10966b8b4567.0000000000003181/head//3
2014-06-12 14:51:57.441895 7f9d60bd2700 10 getattrs 3.6b_head/5431a86b/rbd_data.10966b8b4567.0000000000003181/head//3 = 0
2014-06-12 14:51:57.441907 7f9d60bd2700 10 stat 3.6b_head/23e1a86b/rbd_data.10966b8b4567.00000000000015a1/head//3
2014-06-12 14:51:57.442037 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 23e1a86b/rbd_data.10966b8b4567.00000000000015a1/head//3
2014-06-12 14:51:57.442441 7f9d60bd2700 10 getattrs 3.6b_head/23e1a86b/rbd_data.10966b8b4567.00000000000015a1/head//3 = 0
2014-06-12 14:51:57.442450 7f9d60bd2700 10 stat 3.6b_head/e6d4a86b/rbd_data.10a86b8b4567.0000000000002b33/head//3
2014-06-12 14:51:57.442573 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid e6d4a86b/rbd_data.10a86b8b4567.0000000000002b33/head//3
2014-06-12 14:51:57.475743 7f9d60bd2700 10 getattrs 3.6b_head/e6d4a86b/rbd_data.10a86b8b4567.0000000000002b33/head//3 = 0
2014-06-12 14:51:57.475754 7f9d60bd2700 10 stat 3.6b_head/6cf5a86b/rbd_data.10a56b8b4567.0000000000002e62/head//3
2014-06-12 14:51:57.475886 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6cf5a86b/rbd_data.10a56b8b4567.0000000000002e62/head//3
2014-06-12 14:51:57.487466 7f9d60bd2700 10 getattrs 3.6b_head/6cf5a86b/rbd_data.10a56b8b4567.0000000000002e62/head//3 = 0
2014-06-12 14:51:57.487478 7f9d60bd2700 10 stat 3.6b_head/c87a86b/rbd_data.10ae6b8b4567.00000000000017b9/head//3
2014-06-12 14:51:57.487616 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c87a86b/rbd_data.10ae6b8b4567.00000000000017b9/head//3
2014-06-12 14:51:57.488063 7f9d60bd2700 10 getattrs 3.6b_head/c87a86b/rbd_data.10ae6b8b4567.00000000000017b9/head//3 = 0
2014-06-12 14:51:57.488075 7f9d60bd2700 10 stat 3.6b_head/acb8a86b/rbd_data.10976b8b4567.0000000000003914/head//3
2014-06-12 14:51:57.488210 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid acb8a86b/rbd_data.10976b8b4567.0000000000003914/head//3
2014-06-12 14:51:57.505715 7f9d60bd2700 10 getattrs 3.6b_head/acb8a86b/rbd_data.10976b8b4567.0000000000003914/head//3 = 0
2014-06-12 14:51:57.505724 7f9d60bd2700 10 stat 3.6b_head/dcc8a86b/rbd_data.109b6b8b4567.00000000000005a8/head//3
2014-06-12 14:51:57.505819 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid dcc8a86b/rbd_data.109b6b8b4567.00000000000005a8/head//3
2014-06-12 14:51:57.514176 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:57.519196 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:57.543628 7f9d60bd2700 10 getattrs 3.6b_head/dcc8a86b/rbd_data.109b6b8b4567.00000000000005a8/head//3 = 0
2014-06-12 14:51:57.543644 7f9d60bd2700 10 stat 3.6b_head/f799a86b/rbd_data.10af6b8b4567.0000000000001656/head//3
2014-06-12 14:51:57.543831 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid f799a86b/rbd_data.10af6b8b4567.0000000000001656/head//3
2014-06-12 14:51:57.556512 7f9d60bd2700 10 getattrs 3.6b_head/f799a86b/rbd_data.10af6b8b4567.0000000000001656/head//3 = 0
2014-06-12 14:51:57.556519 7f9d60bd2700 10 stat 3.6b_head/6b99a86b/rbd_data.10a36b8b4567.0000000000003450/head//3
2014-06-12 14:51:57.556614 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6b99a86b/rbd_data.10a36b8b4567.0000000000003450/head//3
2014-06-12 14:51:57.562537 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:57.562690 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:57.587667 7f9d60bd2700 10 getattrs 3.6b_head/6b99a86b/rbd_data.10a36b8b4567.0000000000003450/head//3 = 0
2014-06-12 14:51:57.587675 7f9d60bd2700 10 stat 3.6b_head/42d9a86b/rbd_data.109e6b8b4567.0000000000002580/head//3
2014-06-12 14:51:57.587778 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 42d9a86b/rbd_data.109e6b8b4567.0000000000002580/head//3
2014-06-12 14:51:57.604750 7f9d60bd2700 10 getattrs 3.6b_head/42d9a86b/rbd_data.109e6b8b4567.0000000000002580/head//3 = 0
2014-06-12 14:51:57.604764 7f9d60bd2700 10 stat 3.6b_head/c2f9a86b/rbd_data.10a66b8b4567.0000000000000fda/head//3
2014-06-12 14:51:57.604861 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c2f9a86b/rbd_data.10a66b8b4567.0000000000000fda/head//3
2014-06-12 14:51:57.605479 7f9d60bd2700 10 getattrs 3.6b_head/c2f9a86b/rbd_data.10a66b8b4567.0000000000000fda/head//3 = 0
2014-06-12 14:51:57.605497 7f9d60bd2700 10 collection_getattrs 3.6b_head
2014-06-12 14:51:57.605543 7f9d60bd2700 10 collection_getattrs 3.6b_head = 0
2014-06-12 14:51:57.605546 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:57.605583 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:57.605590 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:57.605596 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:57.605600 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:57.641310 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:57.653928 7f9d5501e700  5 -- op tracker -- , seq: 748617, time: 2014-06-12 14:51:57.653864, event: header_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.653937 7f9d5501e700  5 -- op tracker -- , seq: 748617, time: 2014-06-12 14:51:57.653866, event: throttled, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.653943 7f9d5501e700  5 -- op tracker -- , seq: 748617, time: 2014-06-12 14:51:57.653902, event: all_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.653948 7f9d5501e700  5 -- op tracker -- , seq: 748617, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.653951 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:57.653956 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:57.653962 7f9d5501e700 15 osd.2 173 enqueue_op 0x3850f000 prio 127 cost 10717 latency 0.000098 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:57.654025 7f9d61bd4700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:57.654040 7f9d61bd4700 10 osd.2 173 dequeue_op 0x3850f000 prio 127 cost 10717 latency 0.000175 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]
2014-06-12 14:51:57.654084 7f9d61bd4700  5 -- op tracker -- , seq: 748617, time: 2014-06-12 14:51:57.654076, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.654106 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] handle_message: 0x3850f000
2014-06-12 14:51:57.654123 7f9d61bd4700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:57.654140 7f9d61bd4700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:57.654157 7f9d61bd4700  5 -- op tracker -- , seq: 748617, time: 2014-06-12 14:51:57.654151, event: started, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.654164 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  got 29 scrub map
2014-06-12 14:51:57.654272 7f9d61bd4700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] map version is 173'2887
2014-06-12 14:51:57.654293 7f9d61bd4700 10 osd.2 173 dequeue_op 0x3850f000 finish
2014-06-12 14:51:57.654303 7f9d61bd4700  5 -- op tracker -- , seq: 748617, time: 2014-06-12 14:51:57.654297, event: done, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:57.654315 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:57.654325 7f9d61bd4700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:57.654330 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:57.654326 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:57.654347 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:57.654360 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:57.654405 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:57.654413 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:57.654419 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub replica 29 has 25 items
2014-06-12 14:51:57.654439 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1852986b/rbd_data.109b6b8b4567.0000000000001c96/head//3, auth == maps.end()
2014-06-12 14:51:57.654457 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1852986b/rbd_data.109b6b8b4567.0000000000001c96/head//3
2014-06-12 14:51:57.654470 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1f72986b/rbd_data.109a6b8b4567.0000000000002b71/head//3, auth == maps.end()
2014-06-12 14:51:57.654480 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1f72986b/rbd_data.109a6b8b4567.0000000000002b71/head//3
2014-06-12 14:51:57.654488 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9574986b/rbd_data.10a56b8b4567.00000000000012ea/head//3, auth == maps.end()
2014-06-12 14:51:57.654496 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9574986b/rbd_data.10a56b8b4567.00000000000012ea/head//3
2014-06-12 14:51:57.654505 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ba76986b/rbd_data.10a86b8b4567.0000000000002547/head//3, auth == maps.end()
2014-06-12 14:51:57.654512 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ba76986b/rbd_data.10a86b8b4567.0000000000002547/head//3
2014-06-12 14:51:57.654520 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2386986b/rbd_data.10966b8b4567.000000000000246c/head//3, auth == maps.end()
2014-06-12 14:51:57.654528 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 2386986b/rbd_data.10966b8b4567.000000000000246c/head//3
2014-06-12 14:51:57.654536 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8ae6986b/rbd_data.10b06b8b4567.0000000000003881/head//3, auth == maps.end()
2014-06-12 14:51:57.654543 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 8ae6986b/rbd_data.10b06b8b4567.0000000000003881/head//3
2014-06-12 14:51:57.654551 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 807986b/rbd_data.10ad6b8b4567.0000000000003a00/head//3, auth == maps.end()
2014-06-12 14:51:57.654559 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 807986b/rbd_data.10ad6b8b4567.0000000000003a00/head//3
2014-06-12 14:51:57.654567 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 71d7986b/rbd_data.10956b8b4567.00000000000001bc/head//3, auth == maps.end()
2014-06-12 14:51:57.654576 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 71d7986b/rbd_data.10956b8b4567.00000000000001bc/head//3
2014-06-12 14:51:57.654586 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 462a986b/rbd_data.10ad6b8b4567.0000000000000ee5/head//3, auth == maps.end()
2014-06-12 14:51:57.654595 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 462a986b/rbd_data.10ad6b8b4567.0000000000000ee5/head//3
2014-06-12 14:51:57.654604 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a7ba986b/rbd_data.10b46b8b4567.0000000000001a25/head//3, auth == maps.end()
2014-06-12 14:51:57.654613 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a7ba986b/rbd_data.10b46b8b4567.0000000000001a25/head//3
2014-06-12 14:51:57.654621 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 640b986b/rbd_data.109a6b8b4567.0000000000001e70/head//3, auth == maps.end()
2014-06-12 14:51:57.654629 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 640b986b/rbd_data.109a6b8b4567.0000000000001e70/head//3
2014-06-12 14:51:57.654639 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 694b986b/rbd_data.10ac6b8b4567.00000000000010be/head//3, auth == maps.end()
2014-06-12 14:51:57.654648 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 694b986b/rbd_data.10ac6b8b4567.00000000000010be/head//3
2014-06-12 14:51:57.654656 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f7bf986b/rbd_data.10b46b8b4567.00000000000002dd/head//3, auth == maps.end()
2014-06-12 14:51:57.654665 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj f7bf986b/rbd_data.10b46b8b4567.00000000000002dd/head//3
2014-06-12 14:51:57.654673 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6f90a86b/rbd_data.10b46b8b4567.0000000000001745/head//3, auth == maps.end()
2014-06-12 14:51:57.654682 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6f90a86b/rbd_data.10b46b8b4567.0000000000001745/head//3
2014-06-12 14:51:57.654691 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 5431a86b/rbd_data.10966b8b4567.0000000000003181/head//3, auth == maps.end()
2014-06-12 14:51:57.654700 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 5431a86b/rbd_data.10966b8b4567.0000000000003181/head//3
2014-06-12 14:51:57.654708 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 23e1a86b/rbd_data.10966b8b4567.00000000000015a1/head//3, auth == maps.end()
2014-06-12 14:51:57.654717 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 23e1a86b/rbd_data.10966b8b4567.00000000000015a1/head//3
2014-06-12 14:51:57.654726 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj e6d4a86b/rbd_data.10a86b8b4567.0000000000002b33/head//3, auth == maps.end()
2014-06-12 14:51:57.654734 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj e6d4a86b/rbd_data.10a86b8b4567.0000000000002b33/head//3
2014-06-12 14:51:57.654744 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6cf5a86b/rbd_data.10a56b8b4567.0000000000002e62/head//3, auth == maps.end()
2014-06-12 14:51:57.654753 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6cf5a86b/rbd_data.10a56b8b4567.0000000000002e62/head//3
2014-06-12 14:51:57.654762 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c87a86b/rbd_data.10ae6b8b4567.00000000000017b9/head//3, auth == maps.end()
2014-06-12 14:51:57.654770 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c87a86b/rbd_data.10ae6b8b4567.00000000000017b9/head//3
2014-06-12 14:51:57.654779 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj acb8a86b/rbd_data.10976b8b4567.0000000000003914/head//3, auth == maps.end()
2014-06-12 14:51:57.654788 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj acb8a86b/rbd_data.10976b8b4567.0000000000003914/head//3
2014-06-12 14:51:57.654796 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dcc8a86b/rbd_data.109b6b8b4567.00000000000005a8/head//3, auth == maps.end()
2014-06-12 14:51:57.654805 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj dcc8a86b/rbd_data.109b6b8b4567.00000000000005a8/head//3
2014-06-12 14:51:57.654815 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f799a86b/rbd_data.10af6b8b4567.0000000000001656/head//3, auth == maps.end()
2014-06-12 14:51:57.654822 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj f799a86b/rbd_data.10af6b8b4567.0000000000001656/head//3
2014-06-12 14:51:57.654832 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6b99a86b/rbd_data.10a36b8b4567.0000000000003450/head//3, auth == maps.end()
2014-06-12 14:51:57.654840 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6b99a86b/rbd_data.10a36b8b4567.0000000000003450/head//3
2014-06-12 14:51:57.654848 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 42d9a86b/rbd_data.109e6b8b4567.0000000000002580/head//3, auth == maps.end()
2014-06-12 14:51:57.654857 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 42d9a86b/rbd_data.109e6b8b4567.0000000000002580/head//3
2014-06-12 14:51:57.654865 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c2f9a86b/rbd_data.10a66b8b4567.0000000000000fda/head//3, auth == maps.end()
2014-06-12 14:51:57.654874 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c2f9a86b/rbd_data.10a66b8b4567.0000000000000fda/head//3
2014-06-12 14:51:57.654884 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] 
2014-06-12 14:51:57.654891 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub
2014-06-12 14:51:57.654902 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c2f9a86b/rbd_data.10a66b8b4567.0000000000000fda/head//3 c2f9a86b/rbd_data.10a66b8b4567.0000000000000fda/head//3(173'2887 client.4303.0:144621 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2887)
2014-06-12 14:51:57.654915 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  42d9a86b/rbd_data.109e6b8b4567.0000000000002580/head//3 42d9a86b/rbd_data.109e6b8b4567.0000000000002580/head//3(173'1532 client.4295.0:77764 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1532)
2014-06-12 14:51:57.654926 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6b99a86b/rbd_data.10a36b8b4567.0000000000003450/head//3 6b99a86b/rbd_data.10a36b8b4567.0000000000003450/head//3(173'2007 client.4300.0:102054 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2007)
2014-06-12 14:51:57.654938 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  f799a86b/rbd_data.10af6b8b4567.0000000000001656/head//3 f799a86b/rbd_data.10af6b8b4567.0000000000001656/head//3(173'1030 client.4312.0:52219 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1030)
2014-06-12 14:51:57.654949 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  dcc8a86b/rbd_data.109b6b8b4567.00000000000005a8/head//3 dcc8a86b/rbd_data.109b6b8b4567.00000000000005a8/head//3(173'2514 client.4292.0:124029 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2514)
2014-06-12 14:51:57.654960 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  acb8a86b/rbd_data.10976b8b4567.0000000000003914/head//3 acb8a86b/rbd_data.10976b8b4567.0000000000003914/head//3(173'2334 client.4288.0:110311 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2334)
2014-06-12 14:51:57.654969 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c87a86b/rbd_data.10ae6b8b4567.00000000000017b9/head//3 c87a86b/rbd_data.10ae6b8b4567.00000000000017b9/head//3(173'990 client.4311.0:51772 wrlock_by=unknown.0.0:0 dirty s 4194304 uv990)
2014-06-12 14:51:57.654980 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6cf5a86b/rbd_data.10a56b8b4567.0000000000002e62/head//3 6cf5a86b/rbd_data.10a56b8b4567.0000000000002e62/head//3(173'1939 client.4302.0:94524 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1939)
2014-06-12 14:51:57.654992 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  e6d4a86b/rbd_data.10a86b8b4567.0000000000002b33/head//3 e6d4a86b/rbd_data.10a86b8b4567.0000000000002b33/head//3(173'1666 client.4305.0:87016 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1666)
2014-06-12 14:51:57.655003 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  23e1a86b/rbd_data.10966b8b4567.00000000000015a1/head//3 23e1a86b/rbd_data.10966b8b4567.00000000000015a1/head//3(173'937 client.4287.0:47680 wrlock_by=unknown.0.0:0 dirty s 4194304 uv937)
2014-06-12 14:51:57.655015 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  5431a86b/rbd_data.10966b8b4567.0000000000003181/head//3 5431a86b/rbd_data.10966b8b4567.0000000000003181/head//3(173'1947 client.4287.0:96173 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1947)
2014-06-12 14:51:57.655026 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6f90a86b/rbd_data.10b46b8b4567.0000000000001745/head//3 6f90a86b/rbd_data.10b46b8b4567.0000000000001745/head//3(173'1067 client.4317.0:51503 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1067)
2014-06-12 14:51:57.655037 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  f7bf986b/rbd_data.10b46b8b4567.00000000000002dd/head//3 f7bf986b/rbd_data.10b46b8b4567.00000000000002dd/head//3(173'2476 client.4317.0:121146 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2476)
2014-06-12 14:51:57.655048 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  694b986b/rbd_data.10ac6b8b4567.00000000000010be/head//3 694b986b/rbd_data.10ac6b8b4567.00000000000010be/head//3(173'757 client.4309.0:39356 wrlock_by=unknown.0.0:0 dirty s 4194304 uv757)
2014-06-12 14:51:57.655059 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  640b986b/rbd_data.109a6b8b4567.0000000000001e70/head//3 640b986b/rbd_data.109a6b8b4567.0000000000001e70/head//3(173'1232 client.4291.0:63475 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1232)
2014-06-12 14:51:57.655070 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a7ba986b/rbd_data.10b46b8b4567.0000000000001a25/head//3 a7ba986b/rbd_data.10b46b8b4567.0000000000001a25/head//3(173'1191 client.4317.0:56502 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1191)
2014-06-12 14:51:57.655081 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  462a986b/rbd_data.10ad6b8b4567.0000000000000ee5/head//3 462a986b/rbd_data.10ad6b8b4567.0000000000000ee5/head//3(173'2880 client.4310.0:142285 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2880)
2014-06-12 14:51:57.655092 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  71d7986b/rbd_data.10956b8b4567.00000000000001bc/head//3 71d7986b/rbd_data.10956b8b4567.00000000000001bc/head//3(173'2432 client.4286.0:120666 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2432)
2014-06-12 14:51:57.655103 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  807986b/rbd_data.10ad6b8b4567.0000000000003a00/head//3 807986b/rbd_data.10ad6b8b4567.0000000000003a00/head//3(173'2268 client.4310.0:114082 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2268)
2014-06-12 14:51:57.655115 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  8ae6986b/rbd_data.10b06b8b4567.0000000000003881/head//3 8ae6986b/rbd_data.10b06b8b4567.0000000000003881/head//3(173'2251 client.4313.0:113411 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2251)
2014-06-12 14:51:57.655126 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  2386986b/rbd_data.10966b8b4567.000000000000246c/head//3 2386986b/rbd_data.10966b8b4567.000000000000246c/head//3(173'1408 client.4287.0:71903 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1408)
2014-06-12 14:51:57.655137 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ba76986b/rbd_data.10a86b8b4567.0000000000002547/head//3 ba76986b/rbd_data.10a86b8b4567.0000000000002547/head//3(173'1484 client.4305.0:75726 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1484)
2014-06-12 14:51:57.655148 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9574986b/rbd_data.10a56b8b4567.00000000000012ea/head//3 9574986b/rbd_data.10a56b8b4567.00000000000012ea/head//3(173'845 client.4302.0:43975 wrlock_by=unknown.0.0:0 dirty s 4194304 uv845)
2014-06-12 14:51:57.655160 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1f72986b/rbd_data.109a6b8b4567.0000000000002b71/head//3 1f72986b/rbd_data.109a6b8b4567.0000000000002b71/head//3(173'1698 client.4291.0:86018 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1698)
2014-06-12 14:51:57.655170 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1852986b/rbd_data.109b6b8b4567.0000000000001c96/head//3 1852986b/rbd_data.109b6b8b4567.0000000000001c96/head//3(173'1282 client.4292.0:59249 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1282)
2014-06-12 14:51:57.655178 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:57.655188 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:57.655196 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:57.655198 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:57.655199 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:57.655234 7f9d60bd2700 10 collection_list_partial 3.6b_head start:ab5ea86b//0//-1 is_max:0
2014-06-12 14:51:57.655447 7f9d60bd2700 20 objects: [8fb1b86b/rbd_data.10ac6b8b4567.0000000000001c07/head//3,98f7b86b/rbd_data.10a76b8b4567.0000000000000875/head//3,1698b86b/rbd_data.10ac6b8b4567.0000000000001b01/head//3,1ca8b86b/rbd_data.10b46b8b4567.0000000000002391/head//3,fae8b86b/rbd_data.10af6b8b4567.0000000000002783/head//3,8db9b86b/rbd_data.10ae6b8b4567.000000000000110c/head//3,d07bb86b/rbd_data.10af6b8b4567.00000000000011b9/head//3,f0d1c86b/rbd_data.10ab6b8b4567.00000000000037fa/head//3,2383c86b/rbd_data.10986b8b4567.0000000000002a8c/head//3,6c14c86b/rbd_data.10916b8b4567.000000000000017d/head//3,8624c86b/rbd_data.109f6b8b4567.000000000000275c/head//3,b546c86b/rbd_data.10af6b8b4567.0000000000003b24/head//3,a8a6c86b/rbd_data.10966b8b4567.0000000000002484/head//3,24e6c86b/rbd_data.10a86b8b4567.00000000000030a3/head//3,1567c86b/rbd_data.10a66b8b4567.0000000000000255/head//3,ec77c86b/rbd_data.10b26b8b4567.0000000000001b8c/head//3,f5a8c86b/rbd_data.108e6b8b4567.000000000000054a/head//3,f75bc86b/rbd_data.10a06b8b4567.00000000000031b3/head//3,dd2ec86b/rbd_data.10b26b8b4567.000000000000196f/head//3,6f2ec86b/rbd_data.10a06b8b4567.0000000000002dbd/head//3,391fc86b/rbd_data.10946b8b4567.0000000000003110/head//3,adc2d86b/rbd_data.10af6b8b4567.000000000000041e/head//3,98e3d86b/rbd_data.109b6b8b4567.000000000000115e/head//3,3fe4d86b/rbd_data.10906b8b4567.0000000000003b05/head//3,bb78d86b/rbd_data.10a96b8b4567.00000000000024f7/head//3]
2014-06-12 14:51:57.655920 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  requesting scrubmap from osd.29
2014-06-12 14:51:57.655939 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:57.655951 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:57.655959 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:57.655965 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:57.655970 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:57.655974 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_map_chunk [ab5ea86b//0//-1,5439d86b//0//-1)
2014-06-12 14:51:57.655981 7f9d60bd2700 10 collection_list_partial 3.6b_head start:ab5ea86b//0//-1 is_max:0
2014-06-12 14:51:57.656358 7f9d60bd2700 20 objects: [8fb1b86b/rbd_data.10ac6b8b4567.0000000000001c07/head//3,98f7b86b/rbd_data.10a76b8b4567.0000000000000875/head//3,1698b86b/rbd_data.10ac6b8b4567.0000000000001b01/head//3,1ca8b86b/rbd_data.10b46b8b4567.0000000000002391/head//3,fae8b86b/rbd_data.10af6b8b4567.0000000000002783/head//3,8db9b86b/rbd_data.10ae6b8b4567.000000000000110c/head//3,d07bb86b/rbd_data.10af6b8b4567.00000000000011b9/head//3,f0d1c86b/rbd_data.10ab6b8b4567.00000000000037fa/head//3,2383c86b/rbd_data.10986b8b4567.0000000000002a8c/head//3,6c14c86b/rbd_data.10916b8b4567.000000000000017d/head//3,8624c86b/rbd_data.109f6b8b4567.000000000000275c/head//3,b546c86b/rbd_data.10af6b8b4567.0000000000003b24/head//3,a8a6c86b/rbd_data.10966b8b4567.0000000000002484/head//3,24e6c86b/rbd_data.10a86b8b4567.00000000000030a3/head//3,1567c86b/rbd_data.10a66b8b4567.0000000000000255/head//3,ec77c86b/rbd_data.10b26b8b4567.0000000000001b8c/head//3,f5a8c86b/rbd_data.108e6b8b4567.000000000000054a/head//3,f75bc86b/rbd_data.10a06b8b4567.00000000000031b3/head//3,dd2ec86b/rbd_data.10b26b8b4567.000000000000196f/head//3,6f2ec86b/rbd_data.10a06b8b4567.0000000000002dbd/head//3,391fc86b/rbd_data.10946b8b4567.0000000000003110/head//3,adc2d86b/rbd_data.10af6b8b4567.000000000000041e/head//3,98e3d86b/rbd_data.109b6b8b4567.000000000000115e/head//3,3fe4d86b/rbd_data.10906b8b4567.0000000000003b05/head//3,bb78d86b/rbd_data.10a96b8b4567.00000000000024f7/head//3,5439d86b/rbd_data.10906b8b4567.000000000000122a/head//3,b68ad86b/rbd_data.10a96b8b4567.0000000000000cf4/head//3,bfcad86b/rbd_data.10986b8b4567.00000000000026c3/head//3,90bbd86b/rbd_data.10aa6b8b4567.0000000000003348/head//3,dc0ed86b/rbd_data.10906b8b4567.00000000000021c6/head//3,f0c0e86b/rbd_data.10906b8b4567.000000000000024c/head//3,de74e86b/rbd_data.10b36b8b4567.000000000000075d/head//3,80e4e86b/rbd_data.10a66b8b4567.00000000000007ec/head//3,7755e86b/rbd_data.10926b8b4567.0000000000001e80/head//3,c6e5e86b/rbd_data.10966b8b4567.0000000000000f07/head//3,f1a7e86b/rbd_data.10a66b8b4567.0000000000001b31/head//3,512de86b/rbd_data.109a6b8b4567.000000000000248b/head//3,c44de86b/rbd_data.10a36b8b4567.0000000000003aad/head//3,547de86b/rbd_data.10a66b8b4567.0000000000001590/head//3,bde86b/rbd_data.10926b8b4567.0000000000000f16/head//3,ec4ee86b/rbd_data.108d6b8b4567.000000000000284d/head//3,559ee86b/rbd_data.10926b8b4567.0000000000000d45/head//3,3ceee86b/rbd_data.10ac6b8b4567.0000000000000a24/head//3,a64fe86b/rbd_data.10a16b8b4567.0000000000003938/head//3,fc70f86b/rbd_data.10956b8b4567.0000000000002b93/head//3,d101f86b/rbd_data.10956b8b4567.0000000000001101/head//3,6331f86b/rbd_data.10b06b8b4567.000000000000367c/head//3,9b4f86b/rbd_data.10a96b8b4567.000000000000140a/head//3,3397f86b/rbd_data.10a56b8b4567.0000000000001f47/head//3,a15af86b/rbd_data.10a96b8b4567.0000000000001afc/head//3,95aaf86b/rbd_data.10906b8b4567.0000000000001fc7/head//3,352cf86b/rbd_data.108d6b8b4567.0000000000000fd8/head//3,24bcf86b/rbd_data.108f6b8b4567.0000000000001379/head//3,d0ecf86b/rbd_data.10a96b8b4567.00000000000026b4/head//3,70eff86b/rbd_data.109c6b8b4567.0000000000000a0f/head//3]
2014-06-12 14:51:57.656423 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:57.656439 7f9d60bd2700 10 stat 3.6b_head/8fb1b86b/rbd_data.10ac6b8b4567.0000000000001c07/head//3
2014-06-12 14:51:57.656541 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 8fb1b86b/rbd_data.10ac6b8b4567.0000000000001c07/head//3
2014-06-12 14:51:57.667549 7f9d60bd2700 10 getattrs 3.6b_head/8fb1b86b/rbd_data.10ac6b8b4567.0000000000001c07/head//3 = 0
2014-06-12 14:51:57.667558 7f9d60bd2700 10 stat 3.6b_head/98f7b86b/rbd_data.10a76b8b4567.0000000000000875/head//3
2014-06-12 14:51:57.667657 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 98f7b86b/rbd_data.10a76b8b4567.0000000000000875/head//3
2014-06-12 14:51:57.702019 7f9d60bd2700 10 getattrs 3.6b_head/98f7b86b/rbd_data.10a76b8b4567.0000000000000875/head//3 = 0
2014-06-12 14:51:57.702036 7f9d60bd2700 10 stat 3.6b_head/1698b86b/rbd_data.10ac6b8b4567.0000000000001b01/head//3
2014-06-12 14:51:57.702137 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1698b86b/rbd_data.10ac6b8b4567.0000000000001b01/head//3
2014-06-12 14:51:57.702485 7f9d60bd2700 10 getattrs 3.6b_head/1698b86b/rbd_data.10ac6b8b4567.0000000000001b01/head//3 = 0
2014-06-12 14:51:57.702492 7f9d60bd2700 10 stat 3.6b_head/1ca8b86b/rbd_data.10b46b8b4567.0000000000002391/head//3
2014-06-12 14:51:57.702582 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1ca8b86b/rbd_data.10b46b8b4567.0000000000002391/head//3
2014-06-12 14:51:57.715109 7f9d60bd2700 10 getattrs 3.6b_head/1ca8b86b/rbd_data.10b46b8b4567.0000000000002391/head//3 = 0
2014-06-12 14:51:57.715118 7f9d60bd2700 10 stat 3.6b_head/fae8b86b/rbd_data.10af6b8b4567.0000000000002783/head//3
2014-06-12 14:51:57.715216 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid fae8b86b/rbd_data.10af6b8b4567.0000000000002783/head//3
2014-06-12 14:51:57.727595 7f9d60bd2700 10 getattrs 3.6b_head/fae8b86b/rbd_data.10af6b8b4567.0000000000002783/head//3 = 0
2014-06-12 14:51:57.727604 7f9d60bd2700 10 stat 3.6b_head/8db9b86b/rbd_data.10ae6b8b4567.000000000000110c/head//3
2014-06-12 14:51:57.727716 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 8db9b86b/rbd_data.10ae6b8b4567.000000000000110c/head//3
2014-06-12 14:51:57.729080 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:57.761944 7f9d60bd2700 10 getattrs 3.6b_head/8db9b86b/rbd_data.10ae6b8b4567.000000000000110c/head//3 = 0
2014-06-12 14:51:57.761956 7f9d60bd2700 10 stat 3.6b_head/d07bb86b/rbd_data.10af6b8b4567.00000000000011b9/head//3
2014-06-12 14:51:57.762059 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid d07bb86b/rbd_data.10af6b8b4567.00000000000011b9/head//3
2014-06-12 14:51:57.798198 7f9d60bd2700 10 getattrs 3.6b_head/d07bb86b/rbd_data.10af6b8b4567.00000000000011b9/head//3 = 0
2014-06-12 14:51:57.798208 7f9d60bd2700 10 stat 3.6b_head/f0d1c86b/rbd_data.10ab6b8b4567.00000000000037fa/head//3
2014-06-12 14:51:57.798301 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid f0d1c86b/rbd_data.10ab6b8b4567.00000000000037fa/head//3
2014-06-12 14:51:57.817040 7f9d60bd2700 10 getattrs 3.6b_head/f0d1c86b/rbd_data.10ab6b8b4567.00000000000037fa/head//3 = 0
2014-06-12 14:51:57.817049 7f9d60bd2700 10 stat 3.6b_head/2383c86b/rbd_data.10986b8b4567.0000000000002a8c/head//3
2014-06-12 14:51:57.817151 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 2383c86b/rbd_data.10986b8b4567.0000000000002a8c/head//3
2014-06-12 14:51:57.830126 7f9d60bd2700 10 getattrs 3.6b_head/2383c86b/rbd_data.10986b8b4567.0000000000002a8c/head//3 = 0
2014-06-12 14:51:57.830143 7f9d60bd2700 10 stat 3.6b_head/6c14c86b/rbd_data.10916b8b4567.000000000000017d/head//3
2014-06-12 14:51:57.830302 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6c14c86b/rbd_data.10916b8b4567.000000000000017d/head//3
2014-06-12 14:51:57.873273 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:57.880795 7f9d60bd2700 10 getattrs 3.6b_head/6c14c86b/rbd_data.10916b8b4567.000000000000017d/head//3 = 0
2014-06-12 14:51:57.880817 7f9d60bd2700 10 stat 3.6b_head/8624c86b/rbd_data.109f6b8b4567.000000000000275c/head//3
2014-06-12 14:51:57.880932 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 8624c86b/rbd_data.109f6b8b4567.000000000000275c/head//3
2014-06-12 14:51:57.894267 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:57.894287 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:51:57.894820 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:57.894880 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:51:57.894933 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:57.895009 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:57.895157 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:57.895195 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:57.895222 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:57.895263 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:51:57.895333 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:57.895377 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:57.895436 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:57.895482 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:57.895528 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:57.895576 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:57.895622 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:57.895669 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:57.895715 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:57.895761 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:51:57.895807 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:57.895852 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:51:57.895897 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:57.895942 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:57.900815 7f9d60bd2700 10 getattrs 3.6b_head/8624c86b/rbd_data.109f6b8b4567.000000000000275c/head//3 = 0
2014-06-12 14:51:57.900823 7f9d60bd2700 10 stat 3.6b_head/b546c86b/rbd_data.10af6b8b4567.0000000000003b24/head//3
2014-06-12 14:51:57.900910 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid b546c86b/rbd_data.10af6b8b4567.0000000000003b24/head//3
2014-06-12 14:51:57.925910 7f9d60bd2700 10 getattrs 3.6b_head/b546c86b/rbd_data.10af6b8b4567.0000000000003b24/head//3 = 0
2014-06-12 14:51:57.925919 7f9d60bd2700 10 stat 3.6b_head/a8a6c86b/rbd_data.10966b8b4567.0000000000002484/head//3
2014-06-12 14:51:57.926023 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a8a6c86b/rbd_data.10966b8b4567.0000000000002484/head//3
2014-06-12 14:51:57.962093 7f9d60bd2700 10 getattrs 3.6b_head/a8a6c86b/rbd_data.10966b8b4567.0000000000002484/head//3 = 0
2014-06-12 14:51:57.962107 7f9d60bd2700 10 stat 3.6b_head/24e6c86b/rbd_data.10a86b8b4567.00000000000030a3/head//3
2014-06-12 14:51:57.962218 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 24e6c86b/rbd_data.10a86b8b4567.00000000000030a3/head//3
2014-06-12 14:51:57.962337 7f9d60bd2700 10 getattrs 3.6b_head/24e6c86b/rbd_data.10a86b8b4567.00000000000030a3/head//3 = 0
2014-06-12 14:51:57.962343 7f9d60bd2700 10 stat 3.6b_head/1567c86b/rbd_data.10a66b8b4567.0000000000000255/head//3
2014-06-12 14:51:57.962449 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 1567c86b/rbd_data.10a66b8b4567.0000000000000255/head//3
2014-06-12 14:51:58.001451 7f9d60bd2700 10 getattrs 3.6b_head/1567c86b/rbd_data.10a66b8b4567.0000000000000255/head//3 = 0
2014-06-12 14:51:58.001472 7f9d60bd2700 10 stat 3.6b_head/ec77c86b/rbd_data.10b26b8b4567.0000000000001b8c/head//3
2014-06-12 14:51:58.001572 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ec77c86b/rbd_data.10b26b8b4567.0000000000001b8c/head//3
2014-06-12 14:51:58.001700 7f9d60bd2700 10 getattrs 3.6b_head/ec77c86b/rbd_data.10b26b8b4567.0000000000001b8c/head//3 = 0
2014-06-12 14:51:58.001706 7f9d60bd2700 10 stat 3.6b_head/f5a8c86b/rbd_data.108e6b8b4567.000000000000054a/head//3
2014-06-12 14:51:58.001788 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid f5a8c86b/rbd_data.108e6b8b4567.000000000000054a/head//3
2014-06-12 14:51:58.065369 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:58.072587 7f9d60bd2700 10 getattrs 3.6b_head/f5a8c86b/rbd_data.108e6b8b4567.000000000000054a/head//3 = 0
2014-06-12 14:51:58.072599 7f9d60bd2700 10 stat 3.6b_head/f75bc86b/rbd_data.10a06b8b4567.00000000000031b3/head//3
2014-06-12 14:51:58.072821 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid f75bc86b/rbd_data.10a06b8b4567.00000000000031b3/head//3
2014-06-12 14:51:58.111074 7f9d60bd2700 10 getattrs 3.6b_head/f75bc86b/rbd_data.10a06b8b4567.00000000000031b3/head//3 = 0
2014-06-12 14:51:58.111081 7f9d60bd2700 10 stat 3.6b_head/dd2ec86b/rbd_data.10b26b8b4567.000000000000196f/head//3
2014-06-12 14:51:58.111190 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid dd2ec86b/rbd_data.10b26b8b4567.000000000000196f/head//3
2014-06-12 14:51:58.123227 7f9d60bd2700 10 getattrs 3.6b_head/dd2ec86b/rbd_data.10b26b8b4567.000000000000196f/head//3 = 0
2014-06-12 14:51:58.123233 7f9d60bd2700 10 stat 3.6b_head/6f2ec86b/rbd_data.10a06b8b4567.0000000000002dbd/head//3
2014-06-12 14:51:58.123334 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6f2ec86b/rbd_data.10a06b8b4567.0000000000002dbd/head//3
2014-06-12 14:51:58.138937 7f9d60bd2700 10 getattrs 3.6b_head/6f2ec86b/rbd_data.10a06b8b4567.0000000000002dbd/head//3 = 0
2014-06-12 14:51:58.138945 7f9d60bd2700 10 stat 3.6b_head/391fc86b/rbd_data.10946b8b4567.0000000000003110/head//3
2014-06-12 14:51:58.139032 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 391fc86b/rbd_data.10946b8b4567.0000000000003110/head//3
2014-06-12 14:51:58.163642 7f9d60bd2700 10 getattrs 3.6b_head/391fc86b/rbd_data.10946b8b4567.0000000000003110/head//3 = 0
2014-06-12 14:51:58.163652 7f9d60bd2700 10 stat 3.6b_head/adc2d86b/rbd_data.10af6b8b4567.000000000000041e/head//3
2014-06-12 14:51:58.163740 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid adc2d86b/rbd_data.10af6b8b4567.000000000000041e/head//3
2014-06-12 14:51:58.168322 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:58.181046 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:58.181106 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:51:58.202030 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:58.202324 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:51:58.217851 7f9d60bd2700 10 getattrs 3.6b_head/adc2d86b/rbd_data.10af6b8b4567.000000000000041e/head//3 = 0
2014-06-12 14:51:58.217880 7f9d60bd2700 10 stat 3.6b_head/98e3d86b/rbd_data.109b6b8b4567.000000000000115e/head//3
2014-06-12 14:51:58.217993 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 98e3d86b/rbd_data.109b6b8b4567.000000000000115e/head//3
2014-06-12 14:51:58.250687 7f9d60bd2700 10 getattrs 3.6b_head/98e3d86b/rbd_data.109b6b8b4567.000000000000115e/head//3 = 0
2014-06-12 14:51:58.250694 7f9d60bd2700 10 stat 3.6b_head/3fe4d86b/rbd_data.10906b8b4567.0000000000003b05/head//3
2014-06-12 14:51:58.250797 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 3fe4d86b/rbd_data.10906b8b4567.0000000000003b05/head//3
2014-06-12 14:51:58.280818 7f9d60bd2700 10 getattrs 3.6b_head/3fe4d86b/rbd_data.10906b8b4567.0000000000003b05/head//3 = 0
2014-06-12 14:51:58.280825 7f9d60bd2700 10 stat 3.6b_head/bb78d86b/rbd_data.10a96b8b4567.00000000000024f7/head//3
2014-06-12 14:51:58.280883 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:58.280900 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:58.280903 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:58.280904 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:58.280906 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:58.280938 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid bb78d86b/rbd_data.10a96b8b4567.00000000000024f7/head//3
2014-06-12 14:51:58.297266 7f9d60bd2700 10 getattrs 3.6b_head/bb78d86b/rbd_data.10a96b8b4567.00000000000024f7/head//3 = 0
2014-06-12 14:51:58.297275 7f9d60bd2700 10 collection_getattrs 3.6b_head
2014-06-12 14:51:58.297328 7f9d60bd2700 10 collection_getattrs 3.6b_head = 0
2014-06-12 14:51:58.297330 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:58.297367 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:58.297373 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:58.297388 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:58.297391 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:58.303213 7f9d5501e700  5 -- op tracker -- , seq: 748618, time: 2014-06-12 14:51:58.303147, event: header_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.303228 7f9d5501e700  5 -- op tracker -- , seq: 748618, time: 2014-06-12 14:51:58.303149, event: throttled, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.303247 7f9d5501e700  5 -- op tracker -- , seq: 748618, time: 2014-06-12 14:51:58.303185, event: all_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.303253 7f9d5501e700  5 -- op tracker -- , seq: 748618, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.303256 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:58.303261 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:58.303267 7f9d5501e700 15 osd.2 173 enqueue_op 0x3850eb00 prio 127 cost 10717 latency 0.000120 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:58.303310 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:58.303338 7f9d673df700 10 osd.2 173 dequeue_op 0x3850eb00 prio 127 cost 10717 latency 0.000191 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]
2014-06-12 14:51:58.303375 7f9d673df700  5 -- op tracker -- , seq: 748618, time: 2014-06-12 14:51:58.303369, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.303392 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] handle_message: 0x3850eb00
2014-06-12 14:51:58.303408 7f9d673df700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:58.303425 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:58.303442 7f9d673df700  5 -- op tracker -- , seq: 748618, time: 2014-06-12 14:51:58.303437, event: started, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.303450 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  got 29 scrub map
2014-06-12 14:51:58.303555 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] map version is 173'2887
2014-06-12 14:51:58.303576 7f9d673df700 10 osd.2 173 dequeue_op 0x3850eb00 finish
2014-06-12 14:51:58.303587 7f9d673df700  5 -- op tracker -- , seq: 748618, time: 2014-06-12 14:51:58.303581, event: done, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.303599 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:58.303607 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:58.303611 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:51:58.303612 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:58.303635 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:58.303647 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:58.303692 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:58.303699 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:58.303705 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub replica 29 has 25 items
2014-06-12 14:51:58.303724 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8fb1b86b/rbd_data.10ac6b8b4567.0000000000001c07/head//3, auth == maps.end()
2014-06-12 14:51:58.303735 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 8fb1b86b/rbd_data.10ac6b8b4567.0000000000001c07/head//3
2014-06-12 14:51:58.303749 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 98f7b86b/rbd_data.10a76b8b4567.0000000000000875/head//3, auth == maps.end()
2014-06-12 14:51:58.303758 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 98f7b86b/rbd_data.10a76b8b4567.0000000000000875/head//3
2014-06-12 14:51:58.303767 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1698b86b/rbd_data.10ac6b8b4567.0000000000001b01/head//3, auth == maps.end()
2014-06-12 14:51:58.303777 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1698b86b/rbd_data.10ac6b8b4567.0000000000001b01/head//3
2014-06-12 14:51:58.303785 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1ca8b86b/rbd_data.10b46b8b4567.0000000000002391/head//3, auth == maps.end()
2014-06-12 14:51:58.303794 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1ca8b86b/rbd_data.10b46b8b4567.0000000000002391/head//3
2014-06-12 14:51:58.303803 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fae8b86b/rbd_data.10af6b8b4567.0000000000002783/head//3, auth == maps.end()
2014-06-12 14:51:58.303812 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj fae8b86b/rbd_data.10af6b8b4567.0000000000002783/head//3
2014-06-12 14:51:58.303821 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8db9b86b/rbd_data.10ae6b8b4567.000000000000110c/head//3, auth == maps.end()
2014-06-12 14:51:58.303830 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 8db9b86b/rbd_data.10ae6b8b4567.000000000000110c/head//3
2014-06-12 14:51:58.303839 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d07bb86b/rbd_data.10af6b8b4567.00000000000011b9/head//3, auth == maps.end()
2014-06-12 14:51:58.303846 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj d07bb86b/rbd_data.10af6b8b4567.00000000000011b9/head//3
2014-06-12 14:51:58.303854 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f0d1c86b/rbd_data.10ab6b8b4567.00000000000037fa/head//3, auth == maps.end()
2014-06-12 14:51:58.303862 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj f0d1c86b/rbd_data.10ab6b8b4567.00000000000037fa/head//3
2014-06-12 14:51:58.303870 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 2383c86b/rbd_data.10986b8b4567.0000000000002a8c/head//3, auth == maps.end()
2014-06-12 14:51:58.303878 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 2383c86b/rbd_data.10986b8b4567.0000000000002a8c/head//3
2014-06-12 14:51:58.303886 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6c14c86b/rbd_data.10916b8b4567.000000000000017d/head//3, auth == maps.end()
2014-06-12 14:51:58.303893 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6c14c86b/rbd_data.10916b8b4567.000000000000017d/head//3
2014-06-12 14:51:58.303901 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 8624c86b/rbd_data.109f6b8b4567.000000000000275c/head//3, auth == maps.end()
2014-06-12 14:51:58.303909 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 8624c86b/rbd_data.109f6b8b4567.000000000000275c/head//3
2014-06-12 14:51:58.303917 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b546c86b/rbd_data.10af6b8b4567.0000000000003b24/head//3, auth == maps.end()
2014-06-12 14:51:58.303926 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj b546c86b/rbd_data.10af6b8b4567.0000000000003b24/head//3
2014-06-12 14:51:58.303937 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a8a6c86b/rbd_data.10966b8b4567.0000000000002484/head//3, auth == maps.end()
2014-06-12 14:51:58.303945 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a8a6c86b/rbd_data.10966b8b4567.0000000000002484/head//3
2014-06-12 14:51:58.303954 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 24e6c86b/rbd_data.10a86b8b4567.00000000000030a3/head//3, auth == maps.end()
2014-06-12 14:51:58.303963 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 24e6c86b/rbd_data.10a86b8b4567.00000000000030a3/head//3
2014-06-12 14:51:58.303972 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 1567c86b/rbd_data.10a66b8b4567.0000000000000255/head//3, auth == maps.end()
2014-06-12 14:51:58.303981 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 1567c86b/rbd_data.10a66b8b4567.0000000000000255/head//3
2014-06-12 14:51:58.303991 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ec77c86b/rbd_data.10b26b8b4567.0000000000001b8c/head//3, auth == maps.end()
2014-06-12 14:51:58.304000 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ec77c86b/rbd_data.10b26b8b4567.0000000000001b8c/head//3
2014-06-12 14:51:58.304010 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f5a8c86b/rbd_data.108e6b8b4567.000000000000054a/head//3, auth == maps.end()
2014-06-12 14:51:58.304022 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj f5a8c86b/rbd_data.108e6b8b4567.000000000000054a/head//3
2014-06-12 14:51:58.304030 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f75bc86b/rbd_data.10a06b8b4567.00000000000031b3/head//3, auth == maps.end()
2014-06-12 14:51:58.304039 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj f75bc86b/rbd_data.10a06b8b4567.00000000000031b3/head//3
2014-06-12 14:51:58.304048 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dd2ec86b/rbd_data.10b26b8b4567.000000000000196f/head//3, auth == maps.end()
2014-06-12 14:51:58.304057 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj dd2ec86b/rbd_data.10b26b8b4567.000000000000196f/head//3
2014-06-12 14:51:58.304067 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6f2ec86b/rbd_data.10a06b8b4567.0000000000002dbd/head//3, auth == maps.end()
2014-06-12 14:51:58.304075 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6f2ec86b/rbd_data.10a06b8b4567.0000000000002dbd/head//3
2014-06-12 14:51:58.304085 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 391fc86b/rbd_data.10946b8b4567.0000000000003110/head//3, auth == maps.end()
2014-06-12 14:51:58.304093 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 391fc86b/rbd_data.10946b8b4567.0000000000003110/head//3
2014-06-12 14:51:58.304102 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj adc2d86b/rbd_data.10af6b8b4567.000000000000041e/head//3, auth == maps.end()
2014-06-12 14:51:58.304111 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj adc2d86b/rbd_data.10af6b8b4567.000000000000041e/head//3
2014-06-12 14:51:58.304120 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 98e3d86b/rbd_data.109b6b8b4567.000000000000115e/head//3, auth == maps.end()
2014-06-12 14:51:58.304129 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 98e3d86b/rbd_data.109b6b8b4567.000000000000115e/head//3
2014-06-12 14:51:58.304137 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3fe4d86b/rbd_data.10906b8b4567.0000000000003b05/head//3, auth == maps.end()
2014-06-12 14:51:58.304146 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 3fe4d86b/rbd_data.10906b8b4567.0000000000003b05/head//3
2014-06-12 14:51:58.304154 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bb78d86b/rbd_data.10a96b8b4567.00000000000024f7/head//3, auth == maps.end()
2014-06-12 14:51:58.304163 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj bb78d86b/rbd_data.10a96b8b4567.00000000000024f7/head//3
2014-06-12 14:51:58.304174 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] 
2014-06-12 14:51:58.304181 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub
2014-06-12 14:51:58.304192 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  bb78d86b/rbd_data.10a96b8b4567.00000000000024f7/head//3 bb78d86b/rbd_data.10a96b8b4567.00000000000024f7/head//3(173'1604 client.4306.0:78274 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1604)
2014-06-12 14:51:58.304205 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  3fe4d86b/rbd_data.10906b8b4567.0000000000003b05/head//3 3fe4d86b/rbd_data.10906b8b4567.0000000000003b05/head//3(173'2237 client.4281.0:115707 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2237)
2014-06-12 14:51:58.304216 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  98e3d86b/rbd_data.109b6b8b4567.000000000000115e/head//3 98e3d86b/rbd_data.109b6b8b4567.000000000000115e/head//3(173'829 client.4292.0:39304 wrlock_by=unknown.0.0:0 dirty s 4194304 uv829)
2014-06-12 14:51:58.304228 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  adc2d86b/rbd_data.10af6b8b4567.000000000000041e/head//3 adc2d86b/rbd_data.10af6b8b4567.000000000000041e/head//3(173'2498 client.4312.0:126681 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2498)
2014-06-12 14:51:58.304240 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  391fc86b/rbd_data.10946b8b4567.0000000000003110/head//3 391fc86b/rbd_data.10946b8b4567.0000000000003110/head//3(173'1969 client.4285.0:96960 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1969)
2014-06-12 14:51:58.304251 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6f2ec86b/rbd_data.10a06b8b4567.0000000000002dbd/head//3 6f2ec86b/rbd_data.10a06b8b4567.0000000000002dbd/head//3(173'1832 client.4297.0:90850 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1832)
2014-06-12 14:51:58.304262 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  dd2ec86b/rbd_data.10b26b8b4567.000000000000196f/head//3 dd2ec86b/rbd_data.10b26b8b4567.000000000000196f/head//3(173'1035 client.4315.0:55139 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1035)
2014-06-12 14:51:58.304274 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  f75bc86b/rbd_data.10a06b8b4567.00000000000031b3/head//3 f75bc86b/rbd_data.10a06b8b4567.00000000000031b3/head//3(173'1987 client.4297.0:98075 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1987)
2014-06-12 14:51:58.304284 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  f5a8c86b/rbd_data.108e6b8b4567.000000000000054a/head//3 f5a8c86b/rbd_data.108e6b8b4567.000000000000054a/head//3(173'2507 client.4279.0:125005 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2507)
2014-06-12 14:51:58.304295 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ec77c86b/rbd_data.10b26b8b4567.0000000000001b8c/head//3 ec77c86b/rbd_data.10b26b8b4567.0000000000001b8c/head//3(173'1108 client.4315.0:59183 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1108)
2014-06-12 14:51:58.304306 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1567c86b/rbd_data.10a66b8b4567.0000000000000255/head//3 1567c86b/rbd_data.10a66b8b4567.0000000000000255/head//3(173'2465 client.4303.0:121597 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2465)
2014-06-12 14:51:58.304318 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  24e6c86b/rbd_data.10a86b8b4567.00000000000030a3/head//3 24e6c86b/rbd_data.10a86b8b4567.00000000000030a3/head//3(173'1886 client.4305.0:96402 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1886)
2014-06-12 14:51:58.304330 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a8a6c86b/rbd_data.10966b8b4567.0000000000002484/head//3 a8a6c86b/rbd_data.10966b8b4567.0000000000002484/head//3(173'1428 client.4287.0:72035 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1428)
2014-06-12 14:51:58.304341 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  b546c86b/rbd_data.10af6b8b4567.0000000000003b24/head//3 b546c86b/rbd_data.10af6b8b4567.0000000000003b24/head//3(173'2381 client.4312.0:119355 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2381)
2014-06-12 14:51:58.304352 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  8624c86b/rbd_data.109f6b8b4567.000000000000275c/head//3 8624c86b/rbd_data.109f6b8b4567.000000000000275c/head//3(173'1452 client.4296.0:77872 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1452)
2014-06-12 14:51:58.304363 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6c14c86b/rbd_data.10916b8b4567.000000000000017d/head//3 6c14c86b/rbd_data.10916b8b4567.000000000000017d/head//3(173'2421 client.4282.0:120630 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2421)
2014-06-12 14:51:58.304374 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  2383c86b/rbd_data.10986b8b4567.0000000000002a8c/head//3 2383c86b/rbd_data.10986b8b4567.0000000000002a8c/head//3(173'1710 client.4289.0:84652 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1710)
2014-06-12 14:51:58.304385 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  f0d1c86b/rbd_data.10ab6b8b4567.00000000000037fa/head//3 f0d1c86b/rbd_data.10ab6b8b4567.00000000000037fa/head//3(173'2282 client.4308.0:111601 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2282)
2014-06-12 14:51:58.304396 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  d07bb86b/rbd_data.10af6b8b4567.00000000000011b9/head//3 d07bb86b/rbd_data.10af6b8b4567.00000000000011b9/head//3(173'854 client.4312.0:43740 wrlock_by=unknown.0.0:0 dirty s 4194304 uv854)
2014-06-12 14:51:58.304407 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  8db9b86b/rbd_data.10ae6b8b4567.000000000000110c/head//3 8db9b86b/rbd_data.10ae6b8b4567.000000000000110c/head//3(173'752 client.4311.0:39691 wrlock_by=unknown.0.0:0 dirty s 4194304 uv752)
2014-06-12 14:51:58.304418 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  fae8b86b/rbd_data.10af6b8b4567.0000000000002783/head//3 fae8b86b/rbd_data.10af6b8b4567.0000000000002783/head//3(173'1574 client.4312.0:82786 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1574)
2014-06-12 14:51:58.304430 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1ca8b86b/rbd_data.10b46b8b4567.0000000000002391/head//3 1ca8b86b/rbd_data.10b46b8b4567.0000000000002391/head//3(173'1558 client.4317.0:73273 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1558)
2014-06-12 14:51:58.304441 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  1698b86b/rbd_data.10ac6b8b4567.0000000000001b01/head//3 1698b86b/rbd_data.10ac6b8b4567.0000000000001b01/head//3(173'1155 client.4309.0:57465 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1155)
2014-06-12 14:51:58.304452 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  98f7b86b/rbd_data.10a76b8b4567.0000000000000875/head//3 98f7b86b/rbd_data.10a76b8b4567.0000000000000875/head//3(173'2624 client.4304.0:130392 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2624)
2014-06-12 14:51:58.304463 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  8fb1b86b/rbd_data.10ac6b8b4567.0000000000001c07/head//3 8fb1b86b/rbd_data.10ac6b8b4567.0000000000001c07/head//3(173'1197 client.4309.0:58995 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1197)
2014-06-12 14:51:58.304472 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:58.304482 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:58.304488 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:58.304490 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:58.304491 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:58.304525 7f9d60bd2700 10 collection_list_partial 3.6b_head start:5439d86b//0//-1 is_max:0
2014-06-12 14:51:58.304732 7f9d60bd2700 20 objects: [b68ad86b/rbd_data.10a96b8b4567.0000000000000cf4/head//3,bfcad86b/rbd_data.10986b8b4567.00000000000026c3/head//3,90bbd86b/rbd_data.10aa6b8b4567.0000000000003348/head//3,dc0ed86b/rbd_data.10906b8b4567.00000000000021c6/head//3,f0c0e86b/rbd_data.10906b8b4567.000000000000024c/head//3,de74e86b/rbd_data.10b36b8b4567.000000000000075d/head//3,80e4e86b/rbd_data.10a66b8b4567.00000000000007ec/head//3,7755e86b/rbd_data.10926b8b4567.0000000000001e80/head//3,c6e5e86b/rbd_data.10966b8b4567.0000000000000f07/head//3,f1a7e86b/rbd_data.10a66b8b4567.0000000000001b31/head//3,512de86b/rbd_data.109a6b8b4567.000000000000248b/head//3,c44de86b/rbd_data.10a36b8b4567.0000000000003aad/head//3,547de86b/rbd_data.10a66b8b4567.0000000000001590/head//3,bde86b/rbd_data.10926b8b4567.0000000000000f16/head//3,ec4ee86b/rbd_data.108d6b8b4567.000000000000284d/head//3,559ee86b/rbd_data.10926b8b4567.0000000000000d45/head//3,3ceee86b/rbd_data.10ac6b8b4567.0000000000000a24/head//3,a64fe86b/rbd_data.10a16b8b4567.0000000000003938/head//3,fc70f86b/rbd_data.10956b8b4567.0000000000002b93/head//3,d101f86b/rbd_data.10956b8b4567.0000000000001101/head//3,6331f86b/rbd_data.10b06b8b4567.000000000000367c/head//3,9b4f86b/rbd_data.10a96b8b4567.000000000000140a/head//3,3397f86b/rbd_data.10a56b8b4567.0000000000001f47/head//3,a15af86b/rbd_data.10a96b8b4567.0000000000001afc/head//3,95aaf86b/rbd_data.10906b8b4567.0000000000001fc7/head//3]
2014-06-12 14:51:58.305189 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  requesting scrubmap from osd.29
2014-06-12 14:51:58.305213 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:58.305226 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:58.305234 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:58.305239 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:58.305244 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:58.305249 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_map_chunk [5439d86b//0//-1,352cf86b//0//-1)
2014-06-12 14:51:58.305255 7f9d60bd2700 10 collection_list_partial 3.6b_head start:5439d86b//0//-1 is_max:0
2014-06-12 14:51:58.305429 7f9d60bd2700 20 objects: [b68ad86b/rbd_data.10a96b8b4567.0000000000000cf4/head//3,bfcad86b/rbd_data.10986b8b4567.00000000000026c3/head//3,90bbd86b/rbd_data.10aa6b8b4567.0000000000003348/head//3,dc0ed86b/rbd_data.10906b8b4567.00000000000021c6/head//3,f0c0e86b/rbd_data.10906b8b4567.000000000000024c/head//3,de74e86b/rbd_data.10b36b8b4567.000000000000075d/head//3,80e4e86b/rbd_data.10a66b8b4567.00000000000007ec/head//3,7755e86b/rbd_data.10926b8b4567.0000000000001e80/head//3,c6e5e86b/rbd_data.10966b8b4567.0000000000000f07/head//3,f1a7e86b/rbd_data.10a66b8b4567.0000000000001b31/head//3,512de86b/rbd_data.109a6b8b4567.000000000000248b/head//3,c44de86b/rbd_data.10a36b8b4567.0000000000003aad/head//3,547de86b/rbd_data.10a66b8b4567.0000000000001590/head//3,bde86b/rbd_data.10926b8b4567.0000000000000f16/head//3,ec4ee86b/rbd_data.108d6b8b4567.000000000000284d/head//3,559ee86b/rbd_data.10926b8b4567.0000000000000d45/head//3,3ceee86b/rbd_data.10ac6b8b4567.0000000000000a24/head//3,a64fe86b/rbd_data.10a16b8b4567.0000000000003938/head//3,fc70f86b/rbd_data.10956b8b4567.0000000000002b93/head//3,d101f86b/rbd_data.10956b8b4567.0000000000001101/head//3,6331f86b/rbd_data.10b06b8b4567.000000000000367c/head//3,9b4f86b/rbd_data.10a96b8b4567.000000000000140a/head//3,3397f86b/rbd_data.10a56b8b4567.0000000000001f47/head//3,a15af86b/rbd_data.10a96b8b4567.0000000000001afc/head//3,95aaf86b/rbd_data.10906b8b4567.0000000000001fc7/head//3,352cf86b/rbd_data.108d6b8b4567.0000000000000fd8/head//3,24bcf86b/rbd_data.108f6b8b4567.0000000000001379/head//3,d0ecf86b/rbd_data.10a96b8b4567.00000000000026b4/head//3,70eff86b/rbd_data.109c6b8b4567.0000000000000a0f/head//3]
2014-06-12 14:51:58.305481 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scan_list scanning 25 objects
2014-06-12 14:51:58.305495 7f9d60bd2700 10 stat 3.6b_head/b68ad86b/rbd_data.10a96b8b4567.0000000000000cf4/head//3
2014-06-12 14:51:58.305598 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid b68ad86b/rbd_data.10a96b8b4567.0000000000000cf4/head//3
2014-06-12 14:51:58.327093 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:58.355561 7f9d60bd2700 10 getattrs 3.6b_head/b68ad86b/rbd_data.10a96b8b4567.0000000000000cf4/head//3 = 0
2014-06-12 14:51:58.355575 7f9d60bd2700 10 stat 3.6b_head/bfcad86b/rbd_data.10986b8b4567.00000000000026c3/head//3
2014-06-12 14:51:58.355709 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid bfcad86b/rbd_data.10986b8b4567.00000000000026c3/head//3
2014-06-12 14:51:58.366603 7f9d60bd2700 10 getattrs 3.6b_head/bfcad86b/rbd_data.10986b8b4567.00000000000026c3/head//3 = 0
2014-06-12 14:51:58.366614 7f9d60bd2700 10 stat 3.6b_head/90bbd86b/rbd_data.10aa6b8b4567.0000000000003348/head//3
2014-06-12 14:51:58.366824 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 90bbd86b/rbd_data.10aa6b8b4567.0000000000003348/head//3
2014-06-12 14:51:58.372081 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:58.382757 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:51:58.385348 7f9d60bd2700 10 getattrs 3.6b_head/90bbd86b/rbd_data.10aa6b8b4567.0000000000003348/head//3 = 0
2014-06-12 14:51:58.385358 7f9d60bd2700 10 stat 3.6b_head/dc0ed86b/rbd_data.10906b8b4567.00000000000021c6/head//3
2014-06-12 14:51:58.385470 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid dc0ed86b/rbd_data.10906b8b4567.00000000000021c6/head//3
2014-06-12 14:51:58.392247 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:58.397055 7f9d60bd2700 10 getattrs 3.6b_head/dc0ed86b/rbd_data.10906b8b4567.00000000000021c6/head//3 = 0
2014-06-12 14:51:58.397064 7f9d60bd2700 10 stat 3.6b_head/f0c0e86b/rbd_data.10906b8b4567.000000000000024c/head//3
2014-06-12 14:51:58.397171 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid f0c0e86b/rbd_data.10906b8b4567.000000000000024c/head//3
2014-06-12 14:51:58.433084 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:51:58.469225 7f9d60bd2700 10 getattrs 3.6b_head/f0c0e86b/rbd_data.10906b8b4567.000000000000024c/head//3 = 0
2014-06-12 14:51:58.469240 7f9d60bd2700 10 stat 3.6b_head/de74e86b/rbd_data.10b36b8b4567.000000000000075d/head//3
2014-06-12 14:51:58.469383 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid de74e86b/rbd_data.10b36b8b4567.000000000000075d/head//3
2014-06-12 14:51:58.506986 7f9d60bd2700 10 getattrs 3.6b_head/de74e86b/rbd_data.10b36b8b4567.000000000000075d/head//3 = 0
2014-06-12 14:51:58.507003 7f9d60bd2700 10 stat 3.6b_head/80e4e86b/rbd_data.10a66b8b4567.00000000000007ec/head//3
2014-06-12 14:51:58.507147 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 80e4e86b/rbd_data.10a66b8b4567.00000000000007ec/head//3
2014-06-12 14:51:58.522937 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:51:58.523765 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:58.529335 7f9d60bd2700 10 getattrs 3.6b_head/80e4e86b/rbd_data.10a66b8b4567.00000000000007ec/head//3 = 0
2014-06-12 14:51:58.529348 7f9d60bd2700 10 stat 3.6b_head/7755e86b/rbd_data.10926b8b4567.0000000000001e80/head//3
2014-06-12 14:51:58.529478 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 7755e86b/rbd_data.10926b8b4567.0000000000001e80/head//3
2014-06-12 14:51:58.533209 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:58.556492 7f9d60bd2700 10 getattrs 3.6b_head/7755e86b/rbd_data.10926b8b4567.0000000000001e80/head//3 = 0
2014-06-12 14:51:58.556503 7f9d60bd2700 10 stat 3.6b_head/c6e5e86b/rbd_data.10966b8b4567.0000000000000f07/head//3
2014-06-12 14:51:58.556638 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c6e5e86b/rbd_data.10966b8b4567.0000000000000f07/head//3
2014-06-12 14:51:58.570670 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:51:58.606569 7f9d60bd2700 10 getattrs 3.6b_head/c6e5e86b/rbd_data.10966b8b4567.0000000000000f07/head//3 = 0
2014-06-12 14:51:58.606581 7f9d60bd2700 10 stat 3.6b_head/f1a7e86b/rbd_data.10a66b8b4567.0000000000001b31/head//3
2014-06-12 14:51:58.606696 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid f1a7e86b/rbd_data.10a66b8b4567.0000000000001b31/head//3
2014-06-12 14:51:58.619245 7f9d60bd2700 10 getattrs 3.6b_head/f1a7e86b/rbd_data.10a66b8b4567.0000000000001b31/head//3 = 0
2014-06-12 14:51:58.619254 7f9d60bd2700 10 stat 3.6b_head/512de86b/rbd_data.109a6b8b4567.000000000000248b/head//3
2014-06-12 14:51:58.619361 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 512de86b/rbd_data.109a6b8b4567.000000000000248b/head//3
2014-06-12 14:51:58.634108 7f9d60bd2700 10 getattrs 3.6b_head/512de86b/rbd_data.109a6b8b4567.000000000000248b/head//3 = 0
2014-06-12 14:51:58.634117 7f9d60bd2700 10 stat 3.6b_head/c44de86b/rbd_data.10a36b8b4567.0000000000003aad/head//3
2014-06-12 14:51:58.634221 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid c44de86b/rbd_data.10a36b8b4567.0000000000003aad/head//3
2014-06-12 14:51:58.644463 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:51:58.662535 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:58.666223 7f9d60bd2700 10 getattrs 3.6b_head/c44de86b/rbd_data.10a36b8b4567.0000000000003aad/head//3 = 0
2014-06-12 14:51:58.666233 7f9d60bd2700 10 stat 3.6b_head/547de86b/rbd_data.10a66b8b4567.0000000000001590/head//3
2014-06-12 14:51:58.666339 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 547de86b/rbd_data.10a66b8b4567.0000000000001590/head//3
2014-06-12 14:51:58.695768 7f9d60bd2700 10 getattrs 3.6b_head/547de86b/rbd_data.10a66b8b4567.0000000000001590/head//3 = 0
2014-06-12 14:51:58.695779 7f9d60bd2700 10 stat 3.6b_head/bde86b/rbd_data.10926b8b4567.0000000000000f16/head//3
2014-06-12 14:51:58.695878 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid bde86b/rbd_data.10926b8b4567.0000000000000f16/head//3
2014-06-12 14:51:58.701402 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:58.727199 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:51:58.737703 7f9d60bd2700 10 getattrs 3.6b_head/bde86b/rbd_data.10926b8b4567.0000000000000f16/head//3 = 0
2014-06-12 14:51:58.737713 7f9d60bd2700 10 stat 3.6b_head/ec4ee86b/rbd_data.108d6b8b4567.000000000000284d/head//3
2014-06-12 14:51:58.737834 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid ec4ee86b/rbd_data.108d6b8b4567.000000000000284d/head//3
2014-06-12 14:51:58.768851 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:58.769375 7f9d60bd2700 10 getattrs 3.6b_head/ec4ee86b/rbd_data.108d6b8b4567.000000000000284d/head//3 = 0
2014-06-12 14:51:58.769385 7f9d60bd2700 10 stat 3.6b_head/559ee86b/rbd_data.10926b8b4567.0000000000000d45/head//3
2014-06-12 14:51:58.769494 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 559ee86b/rbd_data.10926b8b4567.0000000000000d45/head//3
2014-06-12 14:51:58.785644 7f9d60bd2700 10 getattrs 3.6b_head/559ee86b/rbd_data.10926b8b4567.0000000000000d45/head//3 = 0
2014-06-12 14:51:58.785655 7f9d60bd2700 10 stat 3.6b_head/3ceee86b/rbd_data.10ac6b8b4567.0000000000000a24/head//3
2014-06-12 14:51:58.785769 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 3ceee86b/rbd_data.10ac6b8b4567.0000000000000a24/head//3
2014-06-12 14:51:58.802788 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:58.812175 7f9d60bd2700 10 getattrs 3.6b_head/3ceee86b/rbd_data.10ac6b8b4567.0000000000000a24/head//3 = 0
2014-06-12 14:51:58.812205 7f9d60bd2700 10 stat 3.6b_head/a64fe86b/rbd_data.10a16b8b4567.0000000000003938/head//3
2014-06-12 14:51:58.812308 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a64fe86b/rbd_data.10a16b8b4567.0000000000003938/head//3
2014-06-12 14:51:58.829875 7f9d60bd2700 10 getattrs 3.6b_head/a64fe86b/rbd_data.10a16b8b4567.0000000000003938/head//3 = 0
2014-06-12 14:51:58.829921 7f9d60bd2700 10 stat 3.6b_head/fc70f86b/rbd_data.10956b8b4567.0000000000002b93/head//3
2014-06-12 14:51:58.830152 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid fc70f86b/rbd_data.10956b8b4567.0000000000002b93/head//3
2014-06-12 14:51:58.855933 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:51:58.858903 7f9d60bd2700 10 getattrs 3.6b_head/fc70f86b/rbd_data.10956b8b4567.0000000000002b93/head//3 = 0
2014-06-12 14:51:58.858915 7f9d60bd2700 10 stat 3.6b_head/d101f86b/rbd_data.10956b8b4567.0000000000001101/head//3
2014-06-12 14:51:58.859022 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid d101f86b/rbd_data.10956b8b4567.0000000000001101/head//3
2014-06-12 14:51:58.866268 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:58.869648 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:51:58.872893 7f9d60bd2700 10 getattrs 3.6b_head/d101f86b/rbd_data.10956b8b4567.0000000000001101/head//3 = 0
2014-06-12 14:51:58.872903 7f9d60bd2700 10 stat 3.6b_head/6331f86b/rbd_data.10b06b8b4567.000000000000367c/head//3
2014-06-12 14:51:58.872993 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 6331f86b/rbd_data.10b06b8b4567.000000000000367c/head//3
2014-06-12 14:51:58.887929 7f9d60bd2700 10 getattrs 3.6b_head/6331f86b/rbd_data.10b06b8b4567.000000000000367c/head//3 = 0
2014-06-12 14:51:58.887943 7f9d60bd2700 10 stat 3.6b_head/9b4f86b/rbd_data.10a96b8b4567.000000000000140a/head//3
2014-06-12 14:51:58.888031 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 9b4f86b/rbd_data.10a96b8b4567.000000000000140a/head//3
2014-06-12 14:51:58.888111 7f9d60bd2700 10 getattrs 3.6b_head/9b4f86b/rbd_data.10a96b8b4567.000000000000140a/head//3 = 0
2014-06-12 14:51:58.888116 7f9d60bd2700 10 stat 3.6b_head/3397f86b/rbd_data.10a56b8b4567.0000000000001f47/head//3
2014-06-12 14:51:58.888194 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 3397f86b/rbd_data.10a56b8b4567.0000000000001f47/head//3
2014-06-12 14:51:58.898964 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:58.899057 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:51:58.908372 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:58.911167 7f9d60bd2700 10 getattrs 3.6b_head/3397f86b/rbd_data.10a56b8b4567.0000000000001f47/head//3 = 0
2014-06-12 14:51:58.911178 7f9d60bd2700 10 stat 3.6b_head/a15af86b/rbd_data.10a96b8b4567.0000000000001afc/head//3
2014-06-12 14:51:58.911279 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid a15af86b/rbd_data.10a96b8b4567.0000000000001afc/head//3
2014-06-12 14:51:58.943495 7f9d60bd2700 10 getattrs 3.6b_head/a15af86b/rbd_data.10a96b8b4567.0000000000001afc/head//3 = 0
2014-06-12 14:51:58.943504 7f9d60bd2700 10 stat 3.6b_head/95aaf86b/rbd_data.10906b8b4567.0000000000001fc7/head//3
2014-06-12 14:51:58.943598 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 95aaf86b/rbd_data.10906b8b4567.0000000000001fc7/head//3
2014-06-12 14:51:58.943989 7f9d60bd2700 10 getattrs 3.6b_head/95aaf86b/rbd_data.10906b8b4567.0000000000001fc7/head//3 = 0
2014-06-12 14:51:58.943998 7f9d60bd2700 10 collection_getattrs 3.6b_head
2014-06-12 14:51:58.944053 7f9d60bd2700 10 collection_getattrs 3.6b_head = 0
2014-06-12 14:51:58.944057 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:58.944087 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:58.944094 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:58.944101 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:58.944103 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:58.966866 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:58.982323 7f9d5501e700  5 -- op tracker -- , seq: 748619, time: 2014-06-12 14:51:58.982250, event: header_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.982333 7f9d5501e700  5 -- op tracker -- , seq: 748619, time: 2014-06-12 14:51:58.982252, event: throttled, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.982339 7f9d5501e700  5 -- op tracker -- , seq: 748619, time: 2014-06-12 14:51:58.982293, event: all_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.982345 7f9d5501e700  5 -- op tracker -- , seq: 748619, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.982349 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:58.982354 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:58.982361 7f9d5501e700 15 osd.2 173 enqueue_op 0x3850ea00 prio 127 cost 10717 latency 0.000110 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:58.982421 7f9d66bde700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:58.982443 7f9d66bde700 10 osd.2 173 dequeue_op 0x3850ea00 prio 127 cost 10717 latency 0.000193 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]
2014-06-12 14:51:58.982483 7f9d66bde700  5 -- op tracker -- , seq: 748619, time: 2014-06-12 14:51:58.982476, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.982500 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] handle_message: 0x3850ea00
2014-06-12 14:51:58.982516 7f9d66bde700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:58.982533 7f9d66bde700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:58.982550 7f9d66bde700  5 -- op tracker -- , seq: 748619, time: 2014-06-12 14:51:58.982545, event: started, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.982578 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  got 29 scrub map
2014-06-12 14:51:58.982683 7f9d66bde700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] map version is 173'2887
2014-06-12 14:51:58.982704 7f9d66bde700 10 osd.2 173 dequeue_op 0x3850ea00 finish
2014-06-12 14:51:58.982715 7f9d66bde700  5 -- op tracker -- , seq: 748619, time: 2014-06-12 14:51:58.982709, event: done, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:58.982737 7f9d66bde700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:58.982744 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:58.982755 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:51:58.982757 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:58.982780 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:58.982792 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:58.982832 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:58.982840 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub   osd.2 has 25 items
2014-06-12 14:51:58.982847 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub replica 29 has 25 items
2014-06-12 14:51:58.982868 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj b68ad86b/rbd_data.10a96b8b4567.0000000000000cf4/head//3, auth == maps.end()
2014-06-12 14:51:58.982892 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj b68ad86b/rbd_data.10a96b8b4567.0000000000000cf4/head//3
2014-06-12 14:51:58.982907 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bfcad86b/rbd_data.10986b8b4567.00000000000026c3/head//3, auth == maps.end()
2014-06-12 14:51:58.982916 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj bfcad86b/rbd_data.10986b8b4567.00000000000026c3/head//3
2014-06-12 14:51:58.982926 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 90bbd86b/rbd_data.10aa6b8b4567.0000000000003348/head//3, auth == maps.end()
2014-06-12 14:51:58.982935 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 90bbd86b/rbd_data.10aa6b8b4567.0000000000003348/head//3
2014-06-12 14:51:58.982948 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj dc0ed86b/rbd_data.10906b8b4567.00000000000021c6/head//3, auth == maps.end()
2014-06-12 14:51:58.982957 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj dc0ed86b/rbd_data.10906b8b4567.00000000000021c6/head//3
2014-06-12 14:51:58.982969 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f0c0e86b/rbd_data.10906b8b4567.000000000000024c/head//3, auth == maps.end()
2014-06-12 14:51:58.982979 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj f0c0e86b/rbd_data.10906b8b4567.000000000000024c/head//3
2014-06-12 14:51:58.982989 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj de74e86b/rbd_data.10b36b8b4567.000000000000075d/head//3, auth == maps.end()
2014-06-12 14:51:58.983000 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj de74e86b/rbd_data.10b36b8b4567.000000000000075d/head//3
2014-06-12 14:51:58.983009 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 80e4e86b/rbd_data.10a66b8b4567.00000000000007ec/head//3, auth == maps.end()
2014-06-12 14:51:58.983019 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 80e4e86b/rbd_data.10a66b8b4567.00000000000007ec/head//3
2014-06-12 14:51:58.983029 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 7755e86b/rbd_data.10926b8b4567.0000000000001e80/head//3, auth == maps.end()
2014-06-12 14:51:58.983038 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 7755e86b/rbd_data.10926b8b4567.0000000000001e80/head//3
2014-06-12 14:51:58.983047 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c6e5e86b/rbd_data.10966b8b4567.0000000000000f07/head//3, auth == maps.end()
2014-06-12 14:51:58.983057 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c6e5e86b/rbd_data.10966b8b4567.0000000000000f07/head//3
2014-06-12 14:51:58.983067 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj f1a7e86b/rbd_data.10a66b8b4567.0000000000001b31/head//3, auth == maps.end()
2014-06-12 14:51:58.983075 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj f1a7e86b/rbd_data.10a66b8b4567.0000000000001b31/head//3
2014-06-12 14:51:58.983085 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 512de86b/rbd_data.109a6b8b4567.000000000000248b/head//3, auth == maps.end()
2014-06-12 14:51:58.983093 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 512de86b/rbd_data.109a6b8b4567.000000000000248b/head//3
2014-06-12 14:51:58.983103 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj c44de86b/rbd_data.10a36b8b4567.0000000000003aad/head//3, auth == maps.end()
2014-06-12 14:51:58.983111 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj c44de86b/rbd_data.10a36b8b4567.0000000000003aad/head//3
2014-06-12 14:51:58.983120 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 547de86b/rbd_data.10a66b8b4567.0000000000001590/head//3, auth == maps.end()
2014-06-12 14:51:58.983129 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 547de86b/rbd_data.10a66b8b4567.0000000000001590/head//3
2014-06-12 14:51:58.983138 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj bde86b/rbd_data.10926b8b4567.0000000000000f16/head//3, auth == maps.end()
2014-06-12 14:51:58.983147 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj bde86b/rbd_data.10926b8b4567.0000000000000f16/head//3
2014-06-12 14:51:58.983157 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj ec4ee86b/rbd_data.108d6b8b4567.000000000000284d/head//3, auth == maps.end()
2014-06-12 14:51:58.983167 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj ec4ee86b/rbd_data.108d6b8b4567.000000000000284d/head//3
2014-06-12 14:51:58.983177 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 559ee86b/rbd_data.10926b8b4567.0000000000000d45/head//3, auth == maps.end()
2014-06-12 14:51:58.983187 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 559ee86b/rbd_data.10926b8b4567.0000000000000d45/head//3
2014-06-12 14:51:58.983197 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3ceee86b/rbd_data.10ac6b8b4567.0000000000000a24/head//3, auth == maps.end()
2014-06-12 14:51:58.983207 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 3ceee86b/rbd_data.10ac6b8b4567.0000000000000a24/head//3
2014-06-12 14:51:58.983217 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a64fe86b/rbd_data.10a16b8b4567.0000000000003938/head//3, auth == maps.end()
2014-06-12 14:51:58.983227 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a64fe86b/rbd_data.10a16b8b4567.0000000000003938/head//3
2014-06-12 14:51:58.983239 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj fc70f86b/rbd_data.10956b8b4567.0000000000002b93/head//3, auth == maps.end()
2014-06-12 14:51:58.983249 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj fc70f86b/rbd_data.10956b8b4567.0000000000002b93/head//3
2014-06-12 14:51:58.983259 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d101f86b/rbd_data.10956b8b4567.0000000000001101/head//3, auth == maps.end()
2014-06-12 14:51:58.983267 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj d101f86b/rbd_data.10956b8b4567.0000000000001101/head//3
2014-06-12 14:51:58.983278 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 6331f86b/rbd_data.10b06b8b4567.000000000000367c/head//3, auth == maps.end()
2014-06-12 14:51:58.983289 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 6331f86b/rbd_data.10b06b8b4567.000000000000367c/head//3
2014-06-12 14:51:58.983300 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 9b4f86b/rbd_data.10a96b8b4567.000000000000140a/head//3, auth == maps.end()
2014-06-12 14:51:58.983310 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 9b4f86b/rbd_data.10a96b8b4567.000000000000140a/head//3
2014-06-12 14:51:58.983322 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 3397f86b/rbd_data.10a56b8b4567.0000000000001f47/head//3, auth == maps.end()
2014-06-12 14:51:58.983332 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 3397f86b/rbd_data.10a56b8b4567.0000000000001f47/head//3
2014-06-12 14:51:58.983342 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj a15af86b/rbd_data.10a96b8b4567.0000000000001afc/head//3, auth == maps.end()
2014-06-12 14:51:58.983352 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj a15af86b/rbd_data.10a96b8b4567.0000000000001afc/head//3
2014-06-12 14:51:58.983363 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 95aaf86b/rbd_data.10906b8b4567.0000000000001fc7/head//3, auth == maps.end()
2014-06-12 14:51:58.983374 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 95aaf86b/rbd_data.10906b8b4567.0000000000001fc7/head//3
2014-06-12 14:51:58.983385 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] 
2014-06-12 14:51:58.983394 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub
2014-06-12 14:51:58.983406 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  95aaf86b/rbd_data.10906b8b4567.0000000000001fc7/head//3 95aaf86b/rbd_data.10906b8b4567.0000000000001fc7/head//3(173'1248 client.4281.0:66187 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1248)
2014-06-12 14:51:58.983421 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a15af86b/rbd_data.10a96b8b4567.0000000000001afc/head//3 a15af86b/rbd_data.10a96b8b4567.0000000000001afc/head//3(173'1227 client.4306.0:59854 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1227)
2014-06-12 14:51:58.983434 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  3397f86b/rbd_data.10a56b8b4567.0000000000001f47/head//3 3397f86b/rbd_data.10a56b8b4567.0000000000001f47/head//3(173'1293 client.4302.0:66252 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1293)
2014-06-12 14:51:58.983448 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  9b4f86b/rbd_data.10a96b8b4567.000000000000140a/head//3 9b4f86b/rbd_data.10a96b8b4567.000000000000140a/head//3(173'942 client.4306.0:47098 wrlock_by=unknown.0.0:0 dirty s 4194304 uv942)
2014-06-12 14:51:58.983460 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  6331f86b/rbd_data.10b06b8b4567.000000000000367c/head//3 6331f86b/rbd_data.10b06b8b4567.000000000000367c/head//3(173'2153 client.4313.0:109537 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2153)
2014-06-12 14:51:58.983473 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  d101f86b/rbd_data.10956b8b4567.0000000000001101/head//3 d101f86b/rbd_data.10956b8b4567.0000000000001101/head//3(173'734 client.4286.0:39723 wrlock_by=unknown.0.0:0 dirty s 4194304 uv734)
2014-06-12 14:51:58.983486 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  fc70f86b/rbd_data.10956b8b4567.0000000000002b93/head//3 fc70f86b/rbd_data.10956b8b4567.0000000000002b93/head//3(173'1811 client.4286.0:88412 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1811)
2014-06-12 14:51:58.983499 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  a64fe86b/rbd_data.10a16b8b4567.0000000000003938/head//3 a64fe86b/rbd_data.10a16b8b4567.0000000000003938/head//3(173'2292 client.4298.0:114294 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2292)
2014-06-12 14:51:58.983512 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  3ceee86b/rbd_data.10ac6b8b4567.0000000000000a24/head//3 3ceee86b/rbd_data.10ac6b8b4567.0000000000000a24/head//3(173'2656 client.4309.0:132399 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2656)
2014-06-12 14:51:58.983526 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  559ee86b/rbd_data.10926b8b4567.0000000000000d45/head//3 559ee86b/rbd_data.10926b8b4567.0000000000000d45/head//3(173'2781 client.4283.0:138406 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2781)
2014-06-12 14:51:58.983539 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  ec4ee86b/rbd_data.108d6b8b4567.000000000000284d/head//3 ec4ee86b/rbd_data.108d6b8b4567.000000000000284d/head//3(173'1567 client.4278.0:82818 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1567)
2014-06-12 14:51:58.983552 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  bde86b/rbd_data.10926b8b4567.0000000000000f16/head//3 bde86b/rbd_data.10926b8b4567.0000000000000f16/head//3(173'2833 client.4283.0:141569 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2833)
2014-06-12 14:51:58.983565 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  547de86b/rbd_data.10a66b8b4567.0000000000001590/head//3 547de86b/rbd_data.10a66b8b4567.0000000000001590/head//3(173'964 client.4303.0:47368 wrlock_by=unknown.0.0:0 dirty s 4194304 uv964)
2014-06-12 14:51:58.983578 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c44de86b/rbd_data.10a36b8b4567.0000000000003aad/head//3 c44de86b/rbd_data.10a36b8b4567.0000000000003aad/head//3(173'2200 client.4300.0:113362 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2200)
2014-06-12 14:51:58.983590 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  512de86b/rbd_data.109a6b8b4567.000000000000248b/head//3 512de86b/rbd_data.109a6b8b4567.000000000000248b/head//3(173'1468 client.4291.0:73850 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1468)
2014-06-12 14:51:58.983604 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  f1a7e86b/rbd_data.10a66b8b4567.0000000000001b31/head//3 f1a7e86b/rbd_data.10a66b8b4567.0000000000001b31/head//3(173'1163 client.4303.0:57040 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1163)
2014-06-12 14:51:58.983617 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  c6e5e86b/rbd_data.10966b8b4567.0000000000000f07/head//3 c6e5e86b/rbd_data.10966b8b4567.0000000000000f07/head//3(173'2813 client.4287.0:138709 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2813)
2014-06-12 14:51:58.983630 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  7755e86b/rbd_data.10926b8b4567.0000000000001e80/head//3 7755e86b/rbd_data.10926b8b4567.0000000000001e80/head//3(173'1271 client.4283.0:64032 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1271)
2014-06-12 14:51:58.983643 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  80e4e86b/rbd_data.10a66b8b4567.00000000000007ec/head//3 80e4e86b/rbd_data.10a66b8b4567.00000000000007ec/head//3(173'2596 client.4303.0:130893 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2596)
2014-06-12 14:51:58.983654 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  de74e86b/rbd_data.10b36b8b4567.000000000000075d/head//3 de74e86b/rbd_data.10b36b8b4567.000000000000075d/head//3(173'2578 client.4316.0:130777 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2578)
2014-06-12 14:51:58.983667 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  f0c0e86b/rbd_data.10906b8b4567.000000000000024c/head//3 f0c0e86b/rbd_data.10906b8b4567.000000000000024c/head//3(173'2446 client.4281.0:119929 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2446)
2014-06-12 14:51:58.983680 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  dc0ed86b/rbd_data.10906b8b4567.00000000000021c6/head//3 dc0ed86b/rbd_data.10906b8b4567.00000000000021c6/head//3(173'1330 client.4281.0:69751 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1330)
2014-06-12 14:51:58.983692 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  90bbd86b/rbd_data.10aa6b8b4567.0000000000003348/head//3 90bbd86b/rbd_data.10aa6b8b4567.0000000000003348/head//3(173'2084 client.4307.0:104681 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2084)
2014-06-12 14:51:58.983705 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  bfcad86b/rbd_data.10986b8b4567.00000000000026c3/head//3 bfcad86b/rbd_data.10986b8b4567.00000000000026c3/head//3(173'1553 client.4289.0:78055 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1553)
2014-06-12 14:51:58.983719 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  b68ad86b/rbd_data.10a96b8b4567.0000000000000cf4/head//3 b68ad86b/rbd_data.10a96b8b4567.0000000000000cf4/head//3(173'2774 client.4306.0:139603 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2774)
2014-06-12 14:51:58.983729 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:58.983740 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:58.983749 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:58.983751 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:58.983752 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state NEW_CHUNK
2014-06-12 14:51:58.983792 7f9d60bd2700 10 collection_list_partial 3.6b_head start:352cf86b//0//-1 is_max:0
2014-06-12 14:51:58.983892 7f9d60bd2700 20 objects: [24bcf86b/rbd_data.108f6b8b4567.0000000000001379/head//3,d0ecf86b/rbd_data.10a96b8b4567.00000000000026b4/head//3,70eff86b/rbd_data.109c6b8b4567.0000000000000a0f/head//3]
2014-06-12 14:51:58.984300 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  requesting scrubmap from osd.29
2014-06-12 14:51:58.984330 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:58.984345 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_PUSHES
2014-06-12 14:51:58.984354 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_LAST_UPDATE
2014-06-12 14:51:58.984360 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state BUILD_MAP
2014-06-12 14:51:58.984366 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map
2014-06-12 14:51:58.984371 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_map_chunk [352cf86b//0//-1,MAX)
2014-06-12 14:51:58.984378 7f9d60bd2700 10 collection_list_partial 3.6b_head start:352cf86b//0//-1 is_max:0
2014-06-12 14:51:58.984429 7f9d60bd2700 20 objects: [24bcf86b/rbd_data.108f6b8b4567.0000000000001379/head//3,d0ecf86b/rbd_data.10a96b8b4567.00000000000026b4/head//3,70eff86b/rbd_data.109c6b8b4567.0000000000000a0f/head//3]
2014-06-12 14:51:58.984444 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scan_list scanning 3 objects
2014-06-12 14:51:58.984454 7f9d60bd2700 10 stat 3.6b_head/24bcf86b/rbd_data.108f6b8b4567.0000000000001379/head//3
2014-06-12 14:51:58.984558 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 24bcf86b/rbd_data.108f6b8b4567.0000000000001379/head//3
2014-06-12 14:51:58.996135 7f9d60bd2700 10 getattrs 3.6b_head/24bcf86b/rbd_data.108f6b8b4567.0000000000001379/head//3 = 0
2014-06-12 14:51:58.996145 7f9d60bd2700 10 stat 3.6b_head/d0ecf86b/rbd_data.10a96b8b4567.00000000000026b4/head//3
2014-06-12 14:51:58.996328 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid d0ecf86b/rbd_data.10a96b8b4567.00000000000026b4/head//3
2014-06-12 14:51:59.004383 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:59.004528 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:59.009316 7f9d60bd2700 10 getattrs 3.6b_head/d0ecf86b/rbd_data.10a96b8b4567.00000000000026b4/head//3 = 0
2014-06-12 14:51:59.009325 7f9d60bd2700 10 stat 3.6b_head/70eff86b/rbd_data.109c6b8b4567.0000000000000a0f/head//3
2014-06-12 14:51:59.009434 7f9d60bd2700 10 lookup_strip_header done  cid 3.6b_head oid 70eff86b/rbd_data.109c6b8b4567.0000000000000a0f/head//3
2014-06-12 14:51:59.009543 7f9d60bd2700 10 getattrs 3.6b_head/70eff86b/rbd_data.109c6b8b4567.0000000000000a0f/head//3 = 0
2014-06-12 14:51:59.009549 7f9d60bd2700 10 collection_getattrs 3.6b_head
2014-06-12 14:51:59.009590 7f9d60bd2700 10 collection_getattrs 3.6b_head = 0
2014-06-12 14:51:59.009592 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] build_scrub_map_chunk done.
2014-06-12 14:51:59.009639 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:59.009649 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] wait for replicas to build scrub map
2014-06-12 14:51:59.009657 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:59.009660 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:59.070487 7f9d5501e700  5 -- op tracker -- , seq: 748620, time: 2014-06-12 14:51:59.070415, event: header_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:59.070504 7f9d5501e700  5 -- op tracker -- , seq: 748620, time: 2014-06-12 14:51:59.070417, event: throttled, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:59.070514 7f9d5501e700  5 -- op tracker -- , seq: 748620, time: 2014-06-12 14:51:59.070450, event: all_read, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:59.070525 7f9d5501e700  5 -- op tracker -- , seq: 748620, time: 0.000000, event: dispatched, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:59.070532 7f9d5501e700 10 osd.2 173 handle_replica_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 epoch 173
2014-06-12 14:51:59.070540 7f9d5501e700 20 osd.2 173 should_share_map osd.29 172.16.96.202:6870/5024 173
2014-06-12 14:51:59.070549 7f9d5501e700 15 osd.2 173 enqueue_op 0x330c6800 prio 127 cost 1323 latency 0.000134 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:59.070605 7f9d6a3e5700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:51:59.070618 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x330c6800 prio 127 cost 1323 latency 0.000202 osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10 pg pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]
2014-06-12 14:51:59.070638 7f9d6a3e5700  5 -- op tracker -- , seq: 748620, time: 2014-06-12 14:51:59.070635, event: reached_pg, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:59.070681 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] handle_message: 0x330c6800
2014-06-12 14:51:59.070699 7f9d6a3e5700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] do_sub_op osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v10
2014-06-12 14:51:59.070716 7f9d6a3e5700  7 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] sub_op_scrub_map
2014-06-12 14:51:59.070734 7f9d6a3e5700  5 -- op tracker -- , seq: 748620, time: 2014-06-12 14:51:59.070728, event: started, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:59.070741 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  got 29 scrub map
2014-06-12 14:51:59.070772 7f9d6a3e5700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] map version is 173'2887
2014-06-12 14:51:59.070790 7f9d6a3e5700 10 osd.2 173 dequeue_op 0x330c6800 finish
2014-06-12 14:51:59.070801 7f9d6a3e5700  5 -- op tracker -- , seq: 748620, time: 2014-06-12 14:51:59.070795, event: done, op: osd_sub_op(unknown.0.0:0 3.6b 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
2014-06-12 14:51:59.070832 7f9d6a3e5700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:51:59.070849 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:59.070827 7f9d60bd2700 12 OSD::disk_tp worker wq OSD::ScrubWQ start processing 0x2ecc000 (1 active)
2014-06-12 14:51:59.070855 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state WAIT_REPLICAS
2014-06-12 14:51:59.070890 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state COMPARE_MAPS
2014-06-12 14:51:59.070897 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub_compare_maps has maps, analyzing
2014-06-12 14:51:59.070905 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  comparing replica scrub maps
2014-06-12 14:51:59.070917 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub   osd.2 has 3 items
2014-06-12 14:51:59.070924 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub replica 29 has 3 items
2014-06-12 14:51:59.070934 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 24bcf86b/rbd_data.108f6b8b4567.0000000000001379/head//3, auth == maps.end()
2014-06-12 14:51:59.070950 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 24bcf86b/rbd_data.108f6b8b4567.0000000000001379/head//3
2014-06-12 14:51:59.070963 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj d0ecf86b/rbd_data.10a96b8b4567.00000000000026b4/head//3, auth == maps.end()
2014-06-12 14:51:59.070972 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj d0ecf86b/rbd_data.10a96b8b4567.00000000000026b4/head//3
2014-06-12 14:51:59.070981 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 2 for obj 70eff86b/rbd_data.109c6b8b4567.0000000000000a0f/head//3, auth == maps.end()
2014-06-12 14:51:59.070989 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] be_select_auth_object: selecting osd 29 for obj 70eff86b/rbd_data.109c6b8b4567.0000000000000a0f/head//3
2014-06-12 14:51:59.070999 7f9d60bd2700  2 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] 
2014-06-12 14:51:59.071005 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub
2014-06-12 14:51:59.071017 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  70eff86b/rbd_data.109c6b8b4567.0000000000000a0f/head//3 70eff86b/rbd_data.109c6b8b4567.0000000000000a0f/head//3(173'2674 client.4293.0:134065 wrlock_by=unknown.0.0:0 dirty s 4194304 uv2674)
2014-06-12 14:51:59.071033 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  d0ecf86b/rbd_data.10a96b8b4567.00000000000026b4/head//3 d0ecf86b/rbd_data.10a96b8b4567.00000000000026b4/head//3(173'1689 client.4306.0:81523 wrlock_by=unknown.0.0:0 dirty s 4194304 uv1689)
2014-06-12 14:51:59.071046 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub  24bcf86b/rbd_data.108f6b8b4567.0000000000001379/head//3 24bcf86b/rbd_data.108f6b8b4567.0000000000001379/head//3(173'877 client.4280.0:43984 wrlock_by=unknown.0.0:0 dirty s 4194304 uv877)
2014-06-12 14:51:59.071056 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] _scrub (scrub) finish
2014-06-12 14:51:59.071063 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing]  requeue_ops 
2014-06-12 14:51:59.071069 7f9d60bd2700 20 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub state FINISH
2014-06-12 14:51:59.071075 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] scrub got 303/315 objects, 0/0 clones, 303/315 dirty, 0/0 omap, 0/0 hit_set_archive, 1270874112/1321205760 bytes.
2014-06-12 14:51:59.071088 7f9d60bd2700  0 log [ERR] : 3.6b scrub stat mismatch, got 303/315 objects, 0/0 clones, 303/315 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts, 1270874112/1321205760 bytes.
2014-06-12 14:51:59.071090 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+scrubbing] process_inconsistent() checking authoritative
2014-06-12 14:51:59.071098 7f9d60bd2700  0 log [ERR] : 3.6b scrub 1 errors
2014-06-12 14:51:59.071148 7f9d60bd2700 10 log is not dirty
2014-06-12 14:51:59.071153 7f9d60bd2700  5 queue_transactions existing osr(3.6b 0x276cc60)/0x276cc60
2014-06-12 14:51:59.071157 7f9d60bd2700  5 queue_transactions (trailing journal)  
2014-06-12 14:51:59.071158 7f9d60bd2700  5 queue_op 0x19ed6060 seq 2679 osr(3.6b 0x276cc60) 826 bytes   (queue has 1 ops and 826 bytes)
2014-06-12 14:51:59.071164 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+inconsistent] publish_stats_to_osd 173:3788
2014-06-12 14:51:59.071178 7f9d60bd2700 20 osd.2 173 dec_scrubs_active 1 -> 0 (max 1, pending 0)
2014-06-12 14:51:59.071179 7f9d60bd2700 15 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+inconsistent]  requeue_ops 
2014-06-12 14:51:59.071188 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+inconsistent] scrub requesting unreserve from osd.29
2014-06-12 14:51:59.071190 7f9d7915f700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x53c5560 (1 active)
2014-06-12 14:51:59.071199 7f9d7915f700  5 _do_op 0x19ed6060 seq 2679 osr(3.6b 0x276cc60)/0x276cc60 start
2014-06-12 14:51:59.071202 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:59.071204 7f9d7915f700 10 _do_transaction on 0x234c840
2014-06-12 14:51:59.071210 7f9d7915f700 15 _omap_setkeys meta/16ef7597/infos/head//-1
2014-06-12 14:51:59.071234 7f9d60bd2700 10 osd.2 pg_epoch: 173 pg[3.6b( v 173'2887 (0'0,173'2887] local-les=161 n=315 ec=23 les/c 161/161 160/160/133) [2,29] r=0 lpr=160 crt=0'0 lcod 173'2886 mlcod 173'2886 active+clean+inconsistent] share_pg_info
2014-06-12 14:51:59.071272 7f9d60bd2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:51:59.071284 7f9d7915f700 10 lookup_strip_header done  cid meta oid 16ef7597/infos/head//-1
2014-06-12 14:51:59.071289 7f9d60bd2700 15 OSD::disk_tp worker wq OSD::ScrubWQ done processing 0x2ecc000 (0 active)
2014-06-12 14:51:59.071291 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:51:59.071311 7f9d7915f700 20 set_header setting 3 cid meta oid 16ef7597/infos/head//-1 parent seq 0
2014-06-12 14:51:59.071335 7f9d7915f700  5 submit_transaction r = 0
2014-06-12 14:51:59.071392 7f9d7915f700 10 _do_op 0x19ed6060 seq 2679 r = 0, finisher 0x57a49a0 0
2014-06-12 14:51:59.071405 7f9d7915f700 10 _finish_op 0x19ed6060 seq 2679 osr(3.6b 0x276cc60)/0x276cc60
2014-06-12 14:51:59.071413 7f9d7915f700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x53c5560 (0 active)
2014-06-12 14:51:59.071417 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:59.122249 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:59.143251 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:51:59.152665 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:59.181153 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:59.218258 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:51:59.280974 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:51:59.281001 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:51:59.281006 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:51:59.281008 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:51:59.281045 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:51:59.281630 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:51:59.299283 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:59.342447 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:59.375792 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:59.514262 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:51:59.519293 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:51:59.618665 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:59.619045 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:51:59.641409 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:51:59.654394 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:51:59.690130 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:59.690215 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:51:59.729159 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:51:59.775550 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:59.775776 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:51:59.873373 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:00.065467 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:00.168410 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:00.233377 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:00.233502 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:00.281099 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:00.281120 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:52:00.281124 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:52:00.281129 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:52:00.281132 7f9d84175700 10 osd.2 173 send_pg_stats - 1 pgs updated
2014-06-12 14:52:00.281179 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:00.281183 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:00.281185 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:00.281217 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:00.303701 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:00.327183 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:00.333643 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:00.333650 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:00.333653 7f9d74bfa700 20 osd.2 173 _dispatch 0x62c8700 pg_stats_ack(1 pgs tid 4875) v1
2014-06-12 14:52:00.333658 7f9d74bfa700 10 osd.2 173 handle_pg_stats_ack 
2014-06-12 14:52:00.333666 7f9d74bfa700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:00.333669 7f9d74bfa700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:00.372164 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:00.382836 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:00.392333 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:00.433183 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:00.523034 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:00.523849 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:00.533303 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:00.570770 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:00.599582 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:00.599680 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:00.644564 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:00.662618 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:00.701485 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:00.727273 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:00.768932 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:00.802869 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:00.856017 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:00.866345 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:00.869724 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:00.908450 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:00.966962 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:00.982843 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:01.063044 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:01.063170 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:01.070938 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:01.071364 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:01.071488 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:01.122333 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:01.124808 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:01.124870 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:01.143325 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:01.152754 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:01.181250 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:01.218357 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:01.281273 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:01.281327 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.7 >= max 0.5 = no, load too high
2014-06-12 14:52:01.281345 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:52:01.281343 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:01.281348 7f9d84175700 10 osd.2 173 sched_scrub 3.6c high load at 2014-06-11 13:02:32.502351: 92968.8 < max (604800 seconds)
2014-06-12 14:52:01.281369 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:01.281372 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:01.281373 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:01.281711 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:01.299363 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:01.342527 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:01.375884 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:01.392409 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:01.392505 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:01.394876 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:01.394892 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:01.395564 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:01.395618 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:01.395668 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:01.395711 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:01.395766 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:01.395833 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:01.395878 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:01.395938 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:01.395984 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:01.396031 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:01.396078 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:01.396123 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:01.396169 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:01.396212 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:01.396256 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:01.396306 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:01.396353 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:01.396398 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:01.396444 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:01.396490 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:01.396544 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:01.396589 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:01.514328 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:01.519391 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:01.563642 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:01.563706 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:01.641483 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:01.654465 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:01.729258 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:01.873478 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:01.934073 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:01.934496 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:02.065563 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:02.168493 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:02.281465 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:02.281489 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:02.281493 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:02.281496 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:02.281534 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:02.303799 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:02.327275 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:02.372259 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:02.382914 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:02.392418 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:02.433261 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:02.523138 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:02.523935 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:02.533400 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:02.570870 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:02.644663 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:02.662699 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:02.701567 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:02.727370 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:02.769043 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:02.802969 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:02.856106 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:02.866440 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:02.869806 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:02.900256 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:02.900432 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:02.908543 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:02.967059 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:02.982942 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:03.071037 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:03.071446 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:03.071567 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:03.093195 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:03.093294 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:03.104944 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:03.105018 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:03.122425 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:03.143425 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:03.152849 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:03.175720 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:03.175789 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:03.181345 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:03.218455 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:03.276384 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:03.276467 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:03.281582 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:03.281649 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:03.281660 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.7 >= max 0.5 = no, load too high
2014-06-12 14:52:03.281674 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:52:03.281678 7f9d84175700 10 osd.2 173 sched_scrub 3.6c high load at 2014-06-11 13:02:32.502351: 92970.8 < max (604800 seconds)
2014-06-12 14:52:03.281716 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:03.281720 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:03.281722 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:03.281789 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:03.299445 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:03.342632 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:03.375965 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:03.400949 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:03.401009 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:03.481739 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:03.481821 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:03.502625 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:03.502869 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:03.514429 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:03.519490 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:03.594011 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:03.594113 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:03.641580 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:03.654557 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:03.719704 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:03.719754 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:03.729358 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:03.790883 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:03.791081 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:03.873577 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:04.065654 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:04.168572 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:04.281786 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:04.281809 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:04.281812 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:04.281814 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:04.281855 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:04.303883 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:04.327359 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:04.372355 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:04.383008 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:04.392507 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:04.433362 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:04.523228 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:04.524017 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:04.533473 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:04.570967 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:04.644764 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:04.662787 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:04.701648 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:04.727453 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:04.769126 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:04.803048 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:04.834868 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:04.834999 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:04.856183 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:04.866521 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:04.869881 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:04.908624 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:04.967142 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:04.983041 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:05.071137 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:05.071530 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:05.071646 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:05.101706 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:05.101834 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:05.122506 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:05.143523 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:05.152942 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:05.181440 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:05.218554 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:05.281871 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:05.281880 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:05.281899 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:52:05.281904 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:52:05.281911 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:52:05.281924 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:05.281929 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:05.281932 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:05.281963 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:05.299516 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:05.342726 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:05.376036 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:05.495625 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:05.495651 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:05.496336 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:05.496425 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:05.496452 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:05.496473 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:05.496497 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:05.496519 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:05.496573 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:05.496642 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:05.496679 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:05.496735 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:05.496787 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:05.496810 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:05.496832 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:05.496856 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:05.496882 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:05.496909 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:05.496938 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:05.496962 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:05.496989 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:05.497014 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:05.497039 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:05.497064 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:05.514514 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:05.519573 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:05.641665 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:05.654657 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:05.729454 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:05.873664 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:06.065728 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:06.168655 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:06.282022 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:06.282075 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:06.282077 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.73 >= max 0.5 = no, load too high
2014-06-12 14:52:06.282094 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:52:06.282097 7f9d84175700 10 osd.2 173 sched_scrub 3.6c high load at 2014-06-11 13:02:32.502351: 92973.8 < max (604800 seconds)
2014-06-12 14:52:06.282118 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:06.282121 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:06.282122 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:06.303977 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:06.327459 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:06.372453 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:06.383087 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:06.392590 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:06.425604 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:06.425674 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:06.433434 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:06.523326 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:06.524101 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:06.533566 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:06.571051 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:06.644844 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:06.662883 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:06.676312 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:06.676544 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:06.701722 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:06.727553 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:06.769223 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:06.803145 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:06.856266 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:06.864533 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:06.864958 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:06.866618 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:06.869961 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:06.908706 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:06.967239 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:06.983140 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:07.071236 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:07.071605 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:07.071726 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:07.122589 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:07.143605 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:07.153038 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:07.181534 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:07.205584 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:07.205656 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:07.218655 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:07.281959 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:07.282188 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:07.282240 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.73 >= max 0.5 = no, load too high
2014-06-12 14:52:07.282254 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:52:07.282258 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:07.282258 7f9d84175700 10 osd.2 173 sched_scrub 3.6c high load at 2014-06-11 13:02:32.502351: 92974.8 < max (604800 seconds)
2014-06-12 14:52:07.282278 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:07.282280 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:07.282282 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:07.299592 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:07.342810 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:07.376114 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:07.514597 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:07.519672 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:07.641749 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:07.654756 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:07.729548 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:07.873751 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:08.065799 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:08.168744 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:08.203388 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:08.203525 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:08.282371 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:08.282422 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.73 >= max 0.5 = no, load too high
2014-06-12 14:52:08.282436 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:52:08.282441 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:08.282439 7f9d84175700 10 osd.2 173 sched_scrub 3.6c high load at 2014-06-11 13:02:32.502351: 92975.8 < max (604800 seconds)
2014-06-12 14:52:08.282459 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:08.282462 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:08.282464 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:08.304063 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:08.327555 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:08.372538 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:08.376929 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:08.377059 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:08.383183 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:08.392678 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:08.433530 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:08.491465 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:08.491710 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:08.523424 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:08.524192 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:08.533661 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:08.571151 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:08.644929 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:08.662981 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:08.701801 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:08.727652 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:08.769317 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:08.782453 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:08.782731 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:08.803242 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:08.856349 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:08.866715 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:08.870037 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:08.908778 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:08.935485 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:08.935692 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:08.967337 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:08.983232 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:09.019985 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:09.020146 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:09.071335 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:09.071687 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:09.071807 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:09.122672 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:09.143689 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:09.153134 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:09.177139 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:09.177200 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:09.181635 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:09.202305 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:09.202766 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:09.218754 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:09.282059 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:09.282539 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:09.282563 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:09.282567 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:09.282569 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:09.282609 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:09.299674 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:09.342891 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:09.376201 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:09.494952 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:09.495024 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:09.514696 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:09.519773 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:09.641848 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:09.654855 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:09.729647 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:09.873826 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:09.926395 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:09.926510 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:09.995818 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:09.995895 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:10.065872 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:10.168839 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:10.282659 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:10.282678 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:52:10.282682 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:52:10.282686 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:52:10.282724 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.73 >= max 0.5 = no, load too high
2014-06-12 14:52:10.282738 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:52:10.282737 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:10.282742 7f9d84175700 10 osd.2 173 sched_scrub 3.6c high load at 2014-06-11 13:02:32.502351: 92977.8 < max (604800 seconds)
2014-06-12 14:52:10.282794 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:10.282798 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:10.282800 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:10.302933 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:10.302996 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:10.304142 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:10.327655 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:10.372633 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:10.383280 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:10.392779 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:10.433610 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:10.483171 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:10.483323 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:10.504111 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:10.504235 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:10.523523 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:10.524288 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:10.533743 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:10.571248 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:10.645026 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:10.663067 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:10.701909 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:10.727750 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:10.769412 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:10.803338 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:10.856435 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:10.866795 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:10.870119 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:10.908855 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:10.965184 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:10.965265 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:10.967410 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:10.983314 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:10.983940 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:10.984179 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:11.071432 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:11.071770 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:11.071885 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:11.122747 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:11.143765 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:11.153228 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:11.181707 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:11.218853 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:11.282141 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:11.282873 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:11.282926 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.67 >= max 0.5 = no, load too high
2014-06-12 14:52:11.282940 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:52:11.282944 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:11.282943 7f9d84175700 10 osd.2 173 sched_scrub 3.6c high load at 2014-06-11 13:02:32.502351: 92978.8 < max (604800 seconds)
2014-06-12 14:52:11.282963 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:11.282966 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:11.282968 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:11.299754 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:11.342972 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:11.376282 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:11.396434 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:11.396453 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:11.397098 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:11.397176 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:11.397232 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:11.397283 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:11.397330 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:11.397399 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:11.397474 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:11.397526 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:11.397573 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:11.397594 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:11.397628 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:11.397651 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:11.397672 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:11.397693 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:11.397714 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:11.397740 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:11.397830 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:11.397875 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:11.397901 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:11.397929 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:11.397959 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:11.397986 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:11.514795 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:11.519857 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:11.627218 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:11.627271 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:11.641923 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:11.654941 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:11.696675 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:11.696733 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:11.729723 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:11.873909 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:11.877740 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:11.877999 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:11.906098 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:11.906180 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:12.003517 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:12.003614 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:12.065944 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:12.168934 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:12.204919 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:12.205057 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:12.283056 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:12.283080 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:12.283084 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:12.283087 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:12.283125 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:12.304226 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:12.327753 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:12.372730 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:12.383374 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:12.392878 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:12.433706 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:12.523623 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:12.524373 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:12.533825 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:12.571347 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:12.645124 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:12.663161 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:12.677720 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:12.677808 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:12.701983 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:12.727834 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:12.769482 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:12.803434 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:12.856517 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:12.866892 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:12.870218 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:12.908935 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:12.967487 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:12.983413 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:13.071530 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:13.071853 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:13.071958 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:13.097201 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:13.097220 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:13.097946 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:13.098034 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:13.098094 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:13.098123 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:13.098148 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:13.098171 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:13.098191 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:13.098222 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:13.098263 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:13.098340 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:13.098369 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:13.098399 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:13.098426 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:13.098453 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:13.098480 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:13.098506 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:13.098533 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:13.098559 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:13.098590 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:13.098614 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:13.098638 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:13.098666 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:13.122829 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:13.143848 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:13.153323 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:13.181788 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:13.192275 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:13.192566 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:13.218941 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:13.282241 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:13.283167 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:13.283190 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:13.283194 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:13.283197 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:13.283236 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:13.299836 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:13.327964 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:13.328056 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:13.343051 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:13.376364 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:13.514880 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:13.519941 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:13.641999 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:13.655024 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:13.729805 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:13.828822 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:13.828918 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:13.873992 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:14.066025 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:14.169015 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:14.198033 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:14.198052 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:14.198775 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:14.198854 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:14.198906 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:14.198966 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:14.199016 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:14.199060 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:14.199130 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:14.199176 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:14.199229 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:14.199276 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:14.199323 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:14.199363 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:14.199409 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:14.199455 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:14.199498 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:14.199546 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:14.199590 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:14.199636 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:14.199682 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:14.199724 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:14.199767 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:14.199912 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:14.236204 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:14.236334 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:14.283286 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:14.283309 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:14.283314 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:14.283317 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:14.283357 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:14.304309 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:14.320686 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:14.320797 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:14.327837 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:14.372827 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:14.378236 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:14.378338 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:14.383471 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:14.392976 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:14.433788 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:14.484704 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:14.485087 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:14.523722 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:14.524457 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:14.533922 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:14.571444 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:14.597961 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:14.598013 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:14.645206 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:14.663260 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:14.702052 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:14.727915 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:14.769576 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:14.778493 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:14.778853 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:14.803530 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:14.856605 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:14.866972 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:14.870297 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:14.909015 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:14.967585 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:14.983509 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:15.065960 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:15.066456 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:15.071629 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:15.071928 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:15.072036 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:15.122911 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:15.143931 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:15.153420 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:15.181862 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:15.219041 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:15.282324 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:15.283405 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:15.283425 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:52:15.283429 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:52:15.283433 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:52:15.283443 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:15.283447 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:15.283448 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:15.283483 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:15.298858 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:15.298877 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:15.299606 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:15.299678 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:15.299730 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:15.299774 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:15.299825 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:15.299908 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:15.299951 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:15.300008 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:15.300059 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:15.300103 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:15.300152 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:15.300198 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:15.300243 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:15.300290 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:15.300333 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:15.300379 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:15.300424 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:15.300471 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:15.300513 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:15.300563 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:15.300609 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:15.300649 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:15.300691 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:15.343131 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:15.376444 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:15.504201 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:15.504630 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:15.514976 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:15.520038 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:15.642074 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:15.655115 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:15.729901 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:15.874070 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:16.021471 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:16.021567 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:16.066101 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:16.169110 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:16.283514 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:16.283537 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:16.283548 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:16.283549 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:16.283584 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:16.304380 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:16.327910 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:16.372920 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:16.383550 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:16.393058 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:16.433862 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:16.479218 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:16.479378 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:16.522226 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:16.522314 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:16.523805 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:16.524542 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:16.534003 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:16.571543 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:16.604778 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:16.605028 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:16.606948 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:16.607038 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:16.645306 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:16.663355 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:16.702120 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:16.727999 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:16.767218 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:16.767295 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:16.769673 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:16.803625 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:16.856702 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:16.867053 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:16.870380 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:16.905608 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:16.906086 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:16.909096 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:16.967681 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:16.983606 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:17.071727 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:17.072003 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:17.072118 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:17.122995 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:17.144025 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:17.153518 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:17.181941 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:17.219117 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:17.282411 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:17.283638 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:17.283694 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.61 >= max 0.5 = no, load too high
2014-06-12 14:52:17.283711 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:52:17.283709 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:17.283714 7f9d84175700 10 osd.2 173 sched_scrub 3.6c high load at 2014-06-11 13:02:32.502351: 92984.8 < max (604800 seconds)
2014-06-12 14:52:17.283746 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:17.283749 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:17.283750 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:17.299986 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:17.329611 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:17.329695 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:17.343226 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:17.376524 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:17.499134 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:17.499194 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:17.515076 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:17.520122 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:17.642159 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:17.655214 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:17.729976 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:17.874155 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:17.879060 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:17.879294 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:17.985464 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:17.985963 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:18.066198 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:18.169190 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:18.179979 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:18.180331 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:18.283839 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:18.283864 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:18.283869 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:18.283871 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:18.283912 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:18.304456 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:18.327989 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:18.373026 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:18.383654 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:18.393160 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:18.433936 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:18.492942 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:18.493139 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:18.523903 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:18.524639 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:18.534100 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:18.571642 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:18.645405 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:18.663451 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:18.702185 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:18.728081 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:18.769744 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:18.803719 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:18.856798 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:18.867150 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:18.870461 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:18.909169 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:18.967772 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:18.983704 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:19.071811 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:19.072082 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:19.072190 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:19.123076 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:19.144107 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:19.153626 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:19.182012 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:19.206255 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:19.206469 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:19.219213 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:19.282485 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:19.283934 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:19.283951 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:19.283954 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:19.283956 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:19.283999 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:19.300066 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:19.343305 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:19.376608 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:19.505378 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:19.505474 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:19.515171 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:19.520206 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:19.536946 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:19.537029 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:19.642262 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:19.655311 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:19.730048 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:19.874241 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:20.022984 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:20.023126 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:20.066274 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:20.169285 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:20.267579 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:20.267693 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:20.284046 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:20.284065 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:52:20.284069 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:52:20.284073 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:52:20.284112 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.61 >= max 0.5 = no, load too high
2014-06-12 14:52:20.284125 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:52:20.284123 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:20.284129 7f9d84175700 10 osd.2 173 sched_scrub 3.6c high load at 2014-06-11 13:02:32.502351: 92987.8 < max (604800 seconds)
2014-06-12 14:52:20.284184 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:20.284188 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:20.284189 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:20.304542 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:20.328072 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:20.373122 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:20.383730 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:20.393243 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:20.434012 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:20.524001 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:20.524723 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:20.534172 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:20.571739 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:20.599693 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:20.599713 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:20.600546 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:20.600631 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:20.600668 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:20.600747 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:20.600821 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:20.600880 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:20.600911 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:20.600942 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:20.600966 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:20.600991 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:20.601016 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:20.601044 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:20.601070 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:20.601095 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:20.601121 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:20.601149 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:20.601180 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:20.601205 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:20.601229 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:20.601254 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:20.601280 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:20.601307 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:20.645524 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:20.663536 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:20.702269 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:20.728163 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:20.769841 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:20.779704 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:20.780089 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:20.803815 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:20.830531 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:20.830598 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:20.856897 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:20.867246 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:20.870545 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:20.909240 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:20.967851 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:20.983790 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:21.071894 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:21.072162 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:21.072270 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:21.123160 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:21.144186 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:21.153728 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:21.182085 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:21.219312 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:21.237562 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:21.237660 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:21.282566 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:21.284278 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:21.284330 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:21.284331 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.56 >= max 0.5 = no, load too high
2014-06-12 14:52:21.284345 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:52:21.284348 7f9d84175700 10 osd.2 173 sched_scrub 3.6c high load at 2014-06-11 13:02:32.502351: 92988.8 < max (604800 seconds)
2014-06-12 14:52:21.284368 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:21.284371 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:21.284373 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:21.300146 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:21.343386 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:21.376702 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:21.515272 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:21.520291 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:21.642344 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:21.655411 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:21.680698 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:21.680776 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:21.730121 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:21.738359 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:21.738650 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:21.874340 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:21.907638 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:21.907731 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:22.066365 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:22.106991 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:22.107349 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:22.169380 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:22.284462 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:22.284514 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.56 >= max 0.5 = no, load too high
2014-06-12 14:52:22.284528 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:52:22.284532 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:22.284531 7f9d84175700 10 osd.2 173 sched_scrub 3.6c high load at 2014-06-11 13:02:32.502351: 92989.8 < max (604800 seconds)
2014-06-12 14:52:22.284553 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:22.284555 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:22.284557 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:22.300618 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:22.300638 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:22.301318 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:22.301384 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:22.301416 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:22.301460 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:22.301483 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:22.301544 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:22.301598 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:22.301664 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:22.301811 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:22.301838 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:22.301863 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:22.301901 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:22.301925 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:22.301954 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:22.301978 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:22.302005 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:22.302029 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:22.302053 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:22.302076 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:22.302100 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:22.302124 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:22.302149 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:22.304614 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:22.328157 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:22.373218 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:22.383811 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:22.393341 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:22.434099 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:22.480403 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:22.480895 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:22.524099 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:22.524808 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:22.534252 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:22.571838 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:22.645626 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:22.663632 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:22.702357 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:22.728247 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:22.769912 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:22.800068 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:22.800157 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:22.803896 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:22.856998 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:22.867341 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:22.870631 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:22.909312 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:22.967934 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:22.983872 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:23.071987 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:23.072244 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:23.072340 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:23.123243 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:23.144270 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:23.153800 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:23.182182 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:23.219411 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:23.282662 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:23.284645 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:23.284668 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:23.284672 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:23.284675 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:23.284715 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:23.300230 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:23.343479 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:23.376798 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:23.515370 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:23.520380 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:23.523776 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:23.524236 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:23.642445 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:23.655508 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:23.730203 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:23.794043 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:23.794221 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:23.874423 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:23.885975 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:23.886409 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:24.066447 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:24.169473 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:24.284765 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:24.284788 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:24.284793 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:24.284795 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:24.284834 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:24.304707 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:24.328236 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:24.368314 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:24.368626 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:24.373318 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:24.383892 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:24.393439 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:24.434181 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:24.524196 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:24.524890 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:24.534345 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:24.571939 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:24.645709 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:24.663711 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:24.702439 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:24.728335 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:24.769991 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:24.803992 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:24.806066 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:24.806230 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:24.857097 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:24.867436 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:24.870711 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:24.909388 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:24.968016 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:24.983973 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:25.072069 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:25.072319 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:25.072419 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:25.123324 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:25.144359 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:25.153902 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:25.182262 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:25.219508 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:25.239205 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:25.239302 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:25.282745 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:25.284883 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:25.284902 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:52:25.284906 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:52:25.284910 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:52:25.284920 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:25.284923 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:25.284925 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:25.284958 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:25.300311 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:25.343559 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:25.376893 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:25.515470 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:25.520462 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:25.607640 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:25.607749 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:25.642527 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:25.655606 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:25.701080 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:25.701155 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:25.730298 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:25.874503 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:26.066531 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:26.169545 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:26.202013 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:26.202098 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:26.285000 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:26.285024 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:26.285029 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:26.285031 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:26.285031 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:26.304790 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:26.328322 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:26.373415 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:26.383974 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:26.393542 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:26.401422 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:26.401442 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:26.402295 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:26.402384 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:26.402425 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:26.402447 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:26.402484 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:26.402525 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:26.402611 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:26.402645 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:26.402667 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:26.402712 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:26.402734 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:26.402755 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:26.402778 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:26.402820 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:26.402846 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:26.402871 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:26.402895 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:26.402916 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:26.402939 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:26.402963 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:26.402987 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:26.403011 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:26.434259 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:26.524292 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:26.524976 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:26.534428 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:26.572028 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:26.645794 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:26.663795 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:26.702545 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:26.728419 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:26.731328 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:26.731525 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:26.770087 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:26.804082 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:26.857198 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:26.867530 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:26.870794 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:26.909464 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:26.968096 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:26.984056 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:27.072151 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:27.072399 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:27.072499 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:27.123405 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:27.144443 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:27.153974 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:27.182360 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:27.208292 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:27.208604 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:27.219594 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:27.282846 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:27.285122 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:27.285146 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:27.285150 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:27.285153 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:27.285191 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:27.300391 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:27.343638 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:27.376989 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:27.386732 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:27.387096 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:27.515552 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:27.520554 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:27.581331 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:27.581419 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:27.642610 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:27.655689 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:27.706477 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:27.706576 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:27.730378 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:27.874591 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:27.894524 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:27.894737 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:28.066611 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:28.082099 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:28.082181 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:28.169639 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:28.207014 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:28.207232 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:28.285244 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:28.285267 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:28.285272 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:28.285274 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:28.285311 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:28.304889 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:28.328405 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:28.373513 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:28.381027 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:28.381120 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:28.384067 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:28.393645 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:28.434357 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:28.502955 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:28.503031 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:28.524392 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:28.525064 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:28.534524 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:28.572109 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:28.645877 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:28.663879 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:28.702631 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:28.728505 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:28.739671 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:28.739949 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:28.770170 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:28.804176 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:28.857297 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:28.867631 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:28.870881 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:28.909547 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:28.968176 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:28.984138 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:29.072236 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:29.072474 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:29.072577 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:29.087518 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:29.087641 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:29.123491 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:29.144534 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:29.154075 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:29.182440 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:29.219693 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:29.282930 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:29.285362 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:29.285386 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:29.285390 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:29.285392 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:29.285431 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:29.300472 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:29.343718 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:29.377085 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:29.424474 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:29.424791 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:29.515651 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:29.520654 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:29.642693 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:29.655772 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:29.669117 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:29.669466 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:29.730474 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:29.874673 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:30.066693 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:30.081780 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:30.081861 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:30.169721 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:30.232247 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:30.232348 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:30.285482 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:30.285501 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:52:30.285505 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:52:30.285508 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:52:30.285519 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:30.285523 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:30.285525 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:30.285560 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:30.304992 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:30.328491 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:30.373598 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:30.382962 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:30.383366 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:30.384149 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:30.393744 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:30.434438 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:30.524493 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:30.525147 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:30.534604 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:30.572193 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:30.645954 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:30.663962 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:30.702711 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:30.728600 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:30.770266 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:30.804254 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:30.857381 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:30.867712 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:30.870981 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:30.908247 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:30.908301 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:30.909630 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:30.968258 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:30.984219 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:31.072320 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:31.072546 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:31.072658 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:31.102386 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:31.102407 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:31.102972 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:31.103026 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:31.103039 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:31.103082 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:31.103113 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:31.103139 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:31.103160 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:31.103205 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:31.103258 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:31.103310 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:31.103353 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:31.103420 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:31.103478 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:31.103491 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:31.103518 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:31.103538 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:31.103572 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:31.103600 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:31.103624 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:31.103648 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:31.103673 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:31.103790 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:31.123575 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:31.144614 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:31.154154 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:31.182522 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:31.219787 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:31.283022 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:31.285599 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:31.285640 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:31.285648 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:31.285651 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:31.285680 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:31.300554 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:31.308977 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:31.309240 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:31.343799 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:31.370126 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:31.370337 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:31.377165 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:31.515738 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:31.520728 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:31.642794 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:31.655854 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:31.730555 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:31.874763 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:31.932964 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:31.933000 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:32.066769 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:32.169803 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:32.285723 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:32.285751 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:32.285755 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:32.285756 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:32.285793 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:32.305076 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:32.328575 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:32.373694 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:32.382369 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:32.382494 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:32.384226 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:32.393829 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:32.434521 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:32.524596 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:32.525234 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:32.534685 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:32.572295 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:32.588340 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:32.588476 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:32.603823 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:32.603874 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:32.646037 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:32.664048 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:32.702792 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:32.728692 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:32.770339 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:32.804354 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:32.857465 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:32.867786 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:32.871064 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:32.883115 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:32.883168 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:32.909710 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:32.925241 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:32.925366 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:32.968346 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:32.984307 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:33.072409 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:33.072618 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:33.072740 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:33.123658 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:33.144698 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:33.154231 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:33.182603 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:33.194816 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:33.195298 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:33.219871 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:33.283107 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:33.285823 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:33.285847 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:33.285850 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:33.285852 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:33.285889 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:33.300651 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:33.343879 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:33.377261 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:33.507378 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:33.507468 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:33.515826 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:33.520800 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:33.642883 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:33.655936 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:33.730654 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:33.874852 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:34.008226 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:34.008272 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:34.066846 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:34.169878 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:34.270607 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:34.270814 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:34.285928 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:34.285953 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:34.285958 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:34.285960 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:34.285980 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:34.304744 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:34.304791 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:34.305146 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:34.328651 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:34.373780 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:34.384306 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:34.393913 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:34.434623 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:34.524683 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:34.525317 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:34.534765 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:34.572379 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:34.583898 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:34.583972 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:34.625996 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:34.626105 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:34.640168 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:34.640270 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:34.646122 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:34.664130 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:34.702899 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:34.728791 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:34.770421 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:34.804434 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:34.857541 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:34.867864 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:34.871148 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:34.909794 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:34.968433 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:34.984391 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:35.072496 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:35.072692 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:35.072829 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:35.123734 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:35.144783 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:35.154312 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:35.182684 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:35.219971 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:35.283193 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:35.286025 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:35.286041 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:52:35.286044 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:52:35.286047 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:52:35.286056 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:35.286059 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:35.286060 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:35.286067 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:35.300747 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:35.343952 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:35.377341 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:35.433606 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:35.433704 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:35.488931 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:35.489107 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:35.515927 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:35.520893 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:35.642977 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:35.656022 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:35.708573 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:35.708950 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:35.730742 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:35.803089 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:35.803110 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:35.803591 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:35.803703 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:35.803727 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:35.803757 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:35.803790 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:35.803840 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:35.803853 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:35.803866 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:35.803880 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:35.803891 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:35.803902 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:35.803913 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:35.803924 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:35.803935 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:35.803945 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:35.803956 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:35.803966 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:35.803977 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:35.803989 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:35.804000 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:35.804011 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:35.804023 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:35.874941 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:36.066930 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:36.169954 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:36.283648 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:36.283738 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:36.286132 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:36.286157 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:36.286161 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:36.286164 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:36.286200 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:36.305224 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:36.328721 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:36.373863 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:36.384386 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:36.393997 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:36.434722 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:36.524774 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:36.525402 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:36.534847 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:36.572463 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:36.609803 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:36.609881 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:36.646206 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:36.664205 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:36.695466 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:36.695569 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:36.702990 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:36.728875 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:36.770507 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:36.784373 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:36.784435 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:36.804530 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:36.808583 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:36.808641 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:36.857640 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:36.867947 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:36.871230 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:36.909898 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:36.968516 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:36.984468 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:37.072583 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:37.072773 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:37.072901 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:37.123817 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:37.144869 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:37.154395 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:37.182766 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:37.189383 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:37.189453 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:37.205598 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:37.205662 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:37.220071 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:37.283282 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:37.286231 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:37.286254 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:37.286258 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:37.286260 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:37.286300 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:37.300827 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:37.344036 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:37.377435 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:37.516003 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:37.520962 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:37.643066 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:37.656105 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:37.730824 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:37.875037 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:38.067013 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:38.103687 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:38.103709 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:38.104458 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:38.104519 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:38.104574 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:38.104615 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:38.104742 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:38.104777 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:38.104806 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:38.104827 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:38.104852 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:38.104879 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:38.104904 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:38.104927 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:38.104951 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:38.104975 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:38.105005 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:38.105028 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:38.105052 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:38.105081 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:38.105108 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:38.105132 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:38.105156 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:38.105183 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:38.126664 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:38.126690 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:38.140917 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:38.141025 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:38.170035 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:38.286333 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:38.286392 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.52 >= max 0.5 = no, load too high
2014-06-12 14:52:38.286406 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:52:38.286407 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:38.286410 7f9d84175700 10 osd.2 173 sched_scrub 3.6c high load at 2014-06-11 13:02:32.502351: 93005.8 < max (604800 seconds)
2014-06-12 14:52:38.286430 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:38.286433 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:38.286435 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:38.305323 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:38.328799 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:38.334344 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:38.334428 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:38.371291 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:38.371709 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:38.373937 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:38.384468 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:38.394080 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:38.396191 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:38.396288 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:38.434806 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:38.524875 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:38.525503 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:38.534922 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:38.572547 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:38.609136 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:38.609486 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:38.646293 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:38.664279 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:38.684558 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:38.685017 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:38.703072 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:38.728972 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:38.770588 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:38.804627 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:38.857724 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:38.868040 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:38.871331 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:38.890084 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:38.890495 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:38.909996 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:38.968596 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:38.984567 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:39.072668 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:39.072856 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:39.072980 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:39.123904 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:39.144972 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:39.154479 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:39.182847 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:39.220170 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:39.283377 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:39.286510 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:39.286563 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.52 >= max 0.5 = no, load too high
2014-06-12 14:52:39.286577 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=0
2014-06-12 14:52:39.286580 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:39.286581 7f9d84175700 10 osd.2 173 sched_scrub 3.6c high load at 2014-06-11 13:02:32.502351: 93006.8 < max (604800 seconds)
2014-06-12 14:52:39.286601 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:39.286603 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:39.286605 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:39.300908 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:39.344117 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:39.377531 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:39.516100 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:39.521050 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:39.643154 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:39.656188 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:39.730906 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:39.785701 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:39.785785 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:39.841578 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:39.841912 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:39.875138 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:40.035144 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:40.035227 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:40.067115 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:40.096892 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:40.096946 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:40.110171 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:40.110638 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:40.170133 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:40.285432 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:40.285549 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:40.286682 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:40.286702 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:52:40.286706 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:52:40.286710 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:52:40.286721 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:40.286725 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:40.286727 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:40.286759 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:40.305406 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:40.328884 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:40.374029 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:40.384562 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:40.394180 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:40.434889 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:40.524973 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:40.525589 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:40.535014 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:40.572631 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:40.597631 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:40.597729 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:40.646375 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:40.664366 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:40.703153 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:40.729073 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:40.770667 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:40.804706 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:40.857811 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:40.868134 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:40.871429 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:40.909101 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:40.909219 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:40.910072 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:40.968694 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:40.984651 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:41.027309 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:41.027364 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:41.072751 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:41.072929 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:41.073062 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:41.123987 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:41.145055 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:41.154561 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:41.182927 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:41.220253 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:41.283462 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:41.286801 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:41.286829 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:41.286834 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:41.286837 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:41.286873 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:41.300991 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:41.344197 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:41.377625 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:41.385794 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:41.386014 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:41.409941 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:41.409990 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:41.485917 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:41.486045 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:41.516200 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:41.521122 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:41.643237 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:41.656271 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:41.730988 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:41.810796 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:41.810903 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:41.875249 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:41.906448 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:41.906487 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:42.067186 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:42.170221 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:42.204470 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:42.204491 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:42.205056 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:42.205121 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:42.205136 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:42.205150 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:42.205191 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:42.205221 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:42.205242 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:42.205256 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:42.205290 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:42.205324 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:42.205370 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:42.205424 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:42.205511 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:42.205539 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:42.205579 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:42.205605 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:42.205654 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:42.205675 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:42.205701 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:42.205724 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:42.205748 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:42.205772 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:42.286912 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:42.286940 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:42.286945 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:42.286947 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:42.286978 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:42.305489 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:42.328966 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:42.374109 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:42.384643 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:42.390864 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:42.390904 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:42.394264 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:42.434970 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:42.525065 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:42.525669 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:42.535095 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:42.572715 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:42.646461 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:42.664457 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:42.703233 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:42.728023 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:42.728075 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:42.729154 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:42.770739 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:42.804787 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:42.857895 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:42.868215 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:42.871520 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:42.910148 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:42.968777 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:42.984737 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:43.072837 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:43.073005 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:43.073142 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:43.124073 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:43.145157 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:43.154646 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:43.183024 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:43.220352 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:43.283559 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:43.287022 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:43.287048 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:43.287053 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:43.287055 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:43.287093 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:43.301072 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:43.309722 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:43.309953 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:43.342431 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:43.342498 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:43.344276 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:43.377723 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:43.511490 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:43.511631 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:43.516299 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:43.521187 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:43.643338 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:43.656355 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:43.672002 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:43.672360 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:43.731069 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:43.875346 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:44.011927 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:44.011953 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:44.067264 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:44.098226 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:44.098503 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:44.170307 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:44.286545 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:44.286609 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:44.287121 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:44.287174 7f9d84175700 20 osd.2 173 scrub_should_schedule loadavg 0.48 < max 0.5 = yes
2014-06-12 14:52:44.287188 7f9d84175700 20 osd.2 173 sched_scrub load_is_low=1
2014-06-12 14:52:44.287189 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:44.287194 7f9d84175700 10 osd.2 173 sched_scrub scrubbing 3.6c at 2014-06-11 13:02:32.502351
2014-06-12 14:52:44.287205 7f9d84175700 20 osd.2 173 inc_scrubs_pending 0 -> 1 (max 1, active 0)
2014-06-12 14:52:44.287208 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] sched_scrub: reserved locally, reserving replicas
2014-06-12 14:52:44.287237 7f9d84175700 10 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] scrub requesting reserve from osd.28
2014-06-12 14:52:44.287251 7f9d84175700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:44.287268 7f9d84175700 20 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] sched_scrub: reserved 2, waiting for replicas
2014-06-12 14:52:44.287276 7f9d84175700 20 osd.2 173 sched_scrub done
2014-06-12 14:52:44.287278 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:44.287279 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:44.287917 7f9d5b6bc700  5 -- op tracker -- , seq: 748621, time: 2014-06-12 14:52:44.287846, event: header_read, op: osd_sub_op_reply(unknown.0.0:0 3.6c 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:52:44.287935 7f9d5b6bc700  5 -- op tracker -- , seq: 748621, time: 2014-06-12 14:52:44.287848, event: throttled, op: osd_sub_op_reply(unknown.0.0:0 3.6c 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:52:44.287945 7f9d5b6bc700  5 -- op tracker -- , seq: 748621, time: 2014-06-12 14:52:44.287869, event: all_read, op: osd_sub_op_reply(unknown.0.0:0 3.6c 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:52:44.287955 7f9d5b6bc700  5 -- op tracker -- , seq: 748621, time: 0.000000, event: dispatched, op: osd_sub_op_reply(unknown.0.0:0 3.6c 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:52:44.287961 7f9d5b6bc700 10 osd.2 173 handle_replica_op osd_sub_op_reply(unknown.0.0:0 3.6c 0//0//3 [scrub-reserve] ack, result = 0) v2 epoch 173
2014-06-12 14:52:44.287968 7f9d5b6bc700 20 osd.2 173 should_share_map osd.28 172.16.96.202:6863/4507 173
2014-06-12 14:52:44.287976 7f9d5b6bc700 15 osd.2 173 enqueue_op 0xe20a800 prio 127 cost 1 latency 0.000130 osd_sub_op_reply(unknown.0.0:0 3.6c 0//0//3 [scrub-reserve] ack, result = 0) v2
2014-06-12 14:52:44.288038 7f9d673df700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active)
2014-06-12 14:52:44.288054 7f9d673df700 10 osd.2 173 dequeue_op 0xe20a800 prio 127 cost 1 latency 0.000207 osd_sub_op_reply(unknown.0.0:0 3.6c 0//0//3 [scrub-reserve] ack, result = 0) v2 pg pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean]
2014-06-12 14:52:44.288076 7f9d673df700  5 -- op tracker -- , seq: 748621, time: 2014-06-12 14:52:44.288072, event: reached_pg, op: osd_sub_op_reply(unknown.0.0:0 3.6c 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:52:44.288100 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] handle_message: 0xe20a800
2014-06-12 14:52:44.288113 7f9d673df700  7 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] sub_op_scrub_reserve_reply
2014-06-12 14:52:44.288124 7f9d673df700  5 -- op tracker -- , seq: 748621, time: 2014-06-12 14:52:44.288121, event: started, op: osd_sub_op_reply(unknown.0.0:0 3.6c 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:52:44.288138 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean]  osd.28 scrub reserve = fail
2014-06-12 14:52:44.288146 7f9d673df700 20 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] sched_scrub: failed, a peer declined
2014-06-12 14:52:44.288153 7f9d673df700 20 osd.2 173 dec_scrubs_pending 1 -> 0 (max 1, active 0)
2014-06-12 14:52:44.288155 7f9d673df700 10 osd.2 pg_epoch: 173 pg[3.6c( v 173'2591 (0'0,173'2591] local-les=157 n=275 ec=23 les/c 157/157 156/156/133) [2,28] r=0 lpr=156 crt=0'0 lcod 173'2590 mlcod 173'2590 active+clean] scrub requesting unreserve from osd.28
2014-06-12 14:52:44.288166 7f9d673df700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:44.288182 7f9d673df700 10 osd.2 173 dequeue_op 0xe20a800 finish
2014-06-12 14:52:44.288188 7f9d673df700  5 -- op tracker -- , seq: 748621, time: 2014-06-12 14:52:44.288185, event: done, op: osd_sub_op_reply(unknown.0.0:0 3.6c 0//0//3 [scrub-reserve] ack, result = 0)
2014-06-12 14:52:44.288266 7f9d673df700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)
2014-06-12 14:52:44.288275 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:44.310200 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:44.310259 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:44.329068 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:44.374191 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:44.384739 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:44.394349 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:44.428902 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:44.429062 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:44.435069 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:44.525153 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:44.525747 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:44.535194 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:44.572800 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:44.646545 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:44.664551 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:44.703343 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:44.729253 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:44.770821 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:44.804873 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:44.857978 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:44.868292 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:44.871603 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:44.910234 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:44.968857 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:44.984835 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:44.986426 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:44.986614 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:45.072923 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:45.073087 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:45.073217 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:45.124156 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:45.145255 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:45.154728 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:45.183118 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:45.220434 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:45.283646 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:45.287354 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:45.287376 7f9d84175700  7 osd.2 173 do_mon_report
2014-06-12 14:52:45.287380 7f9d84175700 10 osd.2 173 send_alive up_thru currently 160 want 160
2014-06-12 14:52:45.287385 7f9d84175700 20 osd.2 173 send_pg_stats
2014-06-12 14:52:45.287396 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:45.287400 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:45.287402 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:45.287432 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:45.301147 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:45.344355 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:45.377808 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:45.407067 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:45.407108 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:45.516401 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:45.521254 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:45.643440 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:45.656440 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:45.731168 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:45.875436 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:45.935946 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:45.936009 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:46.067338 7f9d81970700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:46.087072 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:46.087224 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:46.170391 7f9d7a962700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:46.242823 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:46.242895 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:46.287491 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:46.287519 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:46.287526 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:46.287530 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:46.287557 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:46.288345 7f9d673df700 20 OSD::op_tp worker waiting
2014-06-12 14:52:46.329168 7f9d82972700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:46.374274 7f9d7895e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:46.384836 7f9d603d1700 20 OSD::command_tp worker waiting
2014-06-12 14:52:46.394434 7f9d62bd6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:46.435153 7f9d67be0700 20 OSD::op_tp worker waiting
2014-06-12 14:52:46.525252 7f9d663dd700 20 OSD::op_tp worker waiting
2014-06-12 14:52:46.525825 7f9d83173700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:46.535275 7f9d7a161700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:46.572887 7f9d65bdc700 20 OSD::op_tp worker waiting
2014-06-12 14:52:46.646630 7f9d683e1700 20 OSD::op_tp worker waiting
2014-06-12 14:52:46.664650 7f9d7f16b700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:46.703420 7f9d7e169700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:46.729353 7f9d6b3e7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:46.770913 7f9d7815d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:46.804968 7f9d82171700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:46.810175 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:46.810257 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:46.858065 7f9d6abe6700 20 OSD::op_tp worker waiting
2014-06-12 14:52:46.868379 7f9d7e96a700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:46.871690 7f9d693e3700 20 OSD::op_tp worker waiting
2014-06-12 14:52:46.905142 7f9d5fbd0700 20 osd.2 173 update_osd_stat osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:46.905164 7f9d5fbd0700  5 osd.2 173 heartbeat: osd_stat(19433 MB used, 912 GB avail, 931 GB total, peers [1,20,21,22,23,24,25,26,27,28,29]/[] op hist [])
2014-06-12 14:52:46.905842 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:46.905900 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5d62000 already has epoch 173
2014-06-12 14:52:46.906008 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:46.906060 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7760 already has epoch 173
2014-06-12 14:52:46.906084 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:46.906099 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5cf7600 already has epoch 173
2014-06-12 14:52:46.906120 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:46.906174 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60dec60 already has epoch 173
2014-06-12 14:52:46.906215 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:46.906241 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:46.906265 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:46.906289 7f9d71bf4700 20 osd.2 173 share_map_peer 0x60de2c0 already has epoch 173
2014-06-12 14:52:46.906317 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:46.906340 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:46.906367 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:46.906396 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f2b080 already has epoch 173
2014-06-12 14:52:46.906420 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:46.906442 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:46.906488 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:46.906514 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5c31760 already has epoch 173
2014-06-12 14:52:46.906538 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:46.906562 7f9d71bf4700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:46.910315 7f9d8116f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:46.968955 7f9d7d167700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:46.984935 7f9d66bde700 20 OSD::op_tp worker waiting
2014-06-12 14:52:47.073008 7f9d6a3e5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:47.073165 7f9d60bd2700 20 OSD::disk_tp worker waiting
2014-06-12 14:52:47.073297 7f9d7915f700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:47.124237 7f9d69be4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:47.145339 7f9d64bda700 20 OSD::op_tp worker waiting
2014-06-12 14:52:47.154800 7f9d8016d700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:47.183201 7f9d79960700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:47.220532 7f9d623d5700 20 OSD::op_tp worker waiting
2014-06-12 14:52:47.283738 7f9d653db700 20 OSD::op_tp worker waiting
2014-06-12 14:52:47.287608 7f9d84175700  5 osd.2 173 tick
2014-06-12 14:52:47.287634 7f9d84175700 20 osd.2 173 scrub_random_backoff lost coin flip, randomly backing off
2014-06-12 14:52:47.287639 7f9d84175700 10 osd.2 173 do_waiters -- start
2014-06-12 14:52:47.287642 7f9d84175700 10 osd.2 173 do_waiters -- finish
2014-06-12 14:52:47.287680 7f9d613d3700 20 OSD::recovery_tp worker waiting
2014-06-12 14:52:47.301238 7f9d7d968700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:47.344433 7f9d7c165700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:47.377881 7f9d8096e700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:47.512631 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:47.512748 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5de9760 already has epoch 173
2014-06-12 14:52:47.516487 7f9d63bd8700 20 OSD::op_tp worker waiting
2014-06-12 14:52:47.521326 7f9d68be2700 20 OSD::op_tp worker waiting
2014-06-12 14:52:47.598956 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:47.599072 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f902c0 already has epoch 173
2014-06-12 14:52:47.636559 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:47.636616 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5a01080 already has epoch 173
2014-06-12 14:52:47.643530 7f9d643d9700 20 OSD::op_tp worker waiting
2014-06-12 14:52:47.656538 7f9d61bd4700 20 OSD::op_tp worker waiting
2014-06-12 14:52:47.731265 7f9d7c966700 20 KeyValueStore::op_tp worker waiting
2014-06-12 14:52:47.875537 7f9d633d7700 20 OSD::op_tp worker waiting
2014-06-12 14:52:47.929969 7f9d70bf2700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:47.930080 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x5f2a2c0 already has epoch 173
2014-06-12 14:52:47.943633 7f9d70bf2700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:47.943802 7f9d6f3ef700 20 osd.2 173 share_map_peer 0x60de580 already has epoch 173
2014-06-12 14:52:48.067437 7f9d81970700 20 KeyValueStore::op_tp worker waiting